2020-01-15T23:57:48.415 INFO:root:teuthology version: 1.0.0-7a27738 2020-01-15T23:57:48.415 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.ISFcgs.tmp -- --verbose --lock --description rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{more-async-partial-recovery.yaml} backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/fastclose.yaml msgr/async-v1only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/mapgap.yaml thrashosds-health.yaml workloads/set-chunk-promote-flush.yaml} --name sage-2020-01-15_21:23:44-rados-wip-sage-testing-2020-01-15-1209-distro-basic-smithi --block --owner scheduled_sage@teuthology --archive /home/teuthworker/archive/sage-2020-01-15_21:23:44-rados-wip-sage-testing-2020-01-15-1209-distro-basic-smithi/4671691 2020-01-15T23:57:48.454 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-01-15T23:57:48.521 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/sage-2020-01-15_21:23:44-rados-wip-sage-testing-2020-01-15-1209-distro-basic-smithi/4671691 branch: wip-sage-testing-2020-01-15-1209 description: rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{more-async-partial-recovery.yaml} backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/fastclose.yaml msgr/async-v1only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/mapgap.yaml thrashosds-health.yaml workloads/set-chunk-promote-flush.yaml} email: null first_in_suite: false job_id: '4671691' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: sage-2020-01-15_21:23:44-rados-wip-sage-testing-2020-01-15-1209-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 30 os_type: ubuntu os_version: '18.04' overrides: admin_socket: branch: wip-sage-testing-2020-01-15-1209 ceph: conf: global: mon client directed command retry: 5 ms bind msgr2: false ms inject socket failures: 2500 ms tcp read timeout: 5 ms type: async osd_async_recovery_min_cost: 1 osd_max_pg_log_entries: 2 osd_min_pg_log_entries: 1 osd_object_clean_region_max_num_intervals: 1000 osd_pool_default_min_size: 2 osd_pool_default_size: 3 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd map cache size: 1 osd max backfills: 6 osd memory target: 1610612736 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub during recovery: false osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true fs: xfs log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - osd_map_cache_size - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow requests mon_bind_msgr2: false sha1: cb3c85e74500283e01cc592f7e2858e323186910 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: cb3c85e74500283e01cc592f7e2858e323186910 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 min_in: 4 workunit: branch: wip-sage-testing-2020-01-15-1209 sha1: cb3c85e74500283e01cc592f7e2858e323186910 owner: scheduled_sage@teuthology priority: 90 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 sha1: cb3c85e74500283e01cc592f7e2858e323186910 suite: rados suite_branch: wip-sage-testing-2020-01-15-1209 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-sage-testing-2020-01-15-1209/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: cb3c85e74500283e01cc592f7e2858e323186910 tasks: - install: null - ceph: null - exec: mon.a: - while ! ceph balancer status ; do sleep 1 ; done - ceph balancer mode crush-compat - ceph balancer on - thrashosds: chance_pgnum_grow: 0.25 chance_pgnum_shrink: 0.25 chance_pgpnum_fix: 0.25 chance_test_map_discontinuity: 2 timeout: 1800 - exec: client.0: - sudo ceph osd pool create low_tier 4 - rados: clients: - client.0 low_tier_pool: low_tier objects: 50 op_weights: read: 100 tier_flush: 10 tier_promote: 10 write: 100 ops: 1500 set_chunk: true teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.2508 2020-01-15T23:57:48.522 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-sage-testing-2020-01-15-1209/qa; will attempt to use it 2020-01-15T23:57:48.522 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-sage-testing-2020-01-15-1209/qa/tasks 2020-01-15T23:57:48.522 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-01-15T23:57:48.525 INFO:teuthology.task.internal:Checking packages... 2020-01-15T23:57:48.537 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash 'cb3c85e74500283e01cc592f7e2858e323186910' 2020-01-15T23:57:48.538 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-01-15T23:57:48.538 INFO:teuthology.packaging:ref: None 2020-01-15T23:57:48.538 INFO:teuthology.packaging:tag: None 2020-01-15T23:57:48.538 INFO:teuthology.packaging:branch: wip-sage-testing-2020-01-15-1209 2020-01-15T23:57:48.538 INFO:teuthology.packaging:sha1: cb3c85e74500283e01cc592f7e2858e323186910 2020-01-15T23:57:48.538 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=wip-sage-testing-2020-01-15-1209 2020-01-15T23:57:48.696 INFO:teuthology.task.internal:Found packages for ceph version 15.0.0-9353-gcb3c85e-1bionic 2020-01-15T23:57:48.696 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-01-15T23:57:48.720 INFO:teuthology.task.internal:no buildpackages task found 2020-01-15T23:57:48.720 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-01-15T23:57:48.777 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-01-15T23:57:48.777 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-01-15T23:57:48.854 DEBUG:teuthology.lock.ops:lock_many request: {'count': 2, 'locked_by': 'scheduled_sage@teuthology', 'description': '/home/teuthworker/archive/sage-2020-01-15_21:23:44-rados-wip-sage-testing-2020-01-15-1209-distro-basic-smithi/4671691', 'machine_type': 'smithi'} 2020-01-15T23:57:48.878 DEBUG:teuthology.lock.ops:locked ubuntu@smithi123.front.sepia.ceph.com, ubuntu@smithi145.front.sepia.ceph.com 2020-01-15T23:57:48.960 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2020-01-15T23:57:49.157 INFO:teuthology.provision.fog.smithi145:Scheduling deploy of ubuntu 18.04 2020-01-15T23:57:49.170 INFO:teuthology.provision.fog.smithi123:Scheduling deploy of ubuntu 18.04 2020-01-15T23:57:49.473 INFO:teuthology.orchestra.console:Power off smithi123 2020-01-15T23:57:49.473 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-01-15T23:57:49.492 INFO:teuthology.orchestra.console:Power off smithi145 2020-01-15T23:57:49.493 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-01-15T23:57:49.501 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-01-15T23:57:49.512 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-01-15T23:57:49.624 INFO:teuthology.orchestra.console:Power off for smithi123 completed 2020-01-15T23:57:49.637 INFO:teuthology.orchestra.console:Power off for smithi145 completed 2020-01-15T23:57:49.725 INFO:teuthology.orchestra.console:Power on smithi123 2020-01-15T23:57:49.725 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-01-15T23:57:49.738 INFO:teuthology.orchestra.console:Power on smithi145 2020-01-15T23:57:49.738 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-01-15T23:57:49.745 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-01-15T23:57:49.757 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-01-15T23:57:53.766 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-01-15T23:57:53.777 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-01-15T23:57:57.879 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-01-15T23:57:57.887 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-01-15T23:57:58.088 INFO:teuthology.orchestra.console:Power on for smithi123 completed 2020-01-15T23:57:58.097 INFO:teuthology.orchestra.console:Power on for smithi145 completed 2020-01-15T23:57:58.190 INFO:teuthology.provision.fog.smithi123:Waiting for deploy to finish 2020-01-15T23:57:58.198 INFO:teuthology.provision.fog.smithi145:Waiting for deploy to finish 2020-01-16T00:00:44.361 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi145.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:00:53.434 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi145.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:01:02.489 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi145.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:01:11.545 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi145.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:01:14.510 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi123.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:01:20.601 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi145.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:01:23.575 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi123.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:01:29.659 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi145.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:01:32.631 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi123.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:01:38.713 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi145.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:01:41.688 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi123.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:01:50.746 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi123.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:01:59.807 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi123.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:02:08.891 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi123.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:02:16.282 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi145.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:02:16.288 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-01-16T00:02:17.947 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi123.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:02:22.290 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi145.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:02:28.322 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi145.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:02:28.332 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-01-16T00:02:34.335 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi145.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:02:34.346 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-01-16T00:02:40.347 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi145.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:02:40.357 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-01-16T00:02:46.361 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi145.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:02:46.470 INFO:teuthology.orchestra.run.smithi145:> hostname 2020-01-16T00:02:47.118 INFO:teuthology.orchestra.run.smithi145.stdout:smithi145.front.sepia.ceph.com 2020-01-16T00:02:47.119 INFO:teuthology.orchestra.run.smithi145:> grep smithi145.front.sepia.ceph.com /etc/hosts 2020-01-16T00:02:47.126 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145 smithi145.front.sepia.ceph.com smithi145 2020-01-16T00:02:47.126 INFO:teuthology.orchestra.run.smithi145:> sudo hostname smithi145 2020-01-16T00:02:47.248 INFO:teuthology.orchestra.run.smithi145:> sudo sed -i -e 's/smithi145.front.sepia.ceph.com/smithi145/g' /etc/hosts 2020-01-16T00:02:47.262 INFO:teuthology.orchestra.run.smithi145:> sudo sed -i -e 's/172.21.15.145/172.21.15.145/g' /etc/hosts 2020-01-16T00:02:47.317 INFO:teuthology.orchestra.run.smithi145:> sudo sed -i -e 's/smithi145.front.sepia.ceph.com/smithi145/g' /etc/hostname 2020-01-16T00:02:47.387 INFO:teuthology.orchestra.run.smithi145:> sudo hostname smithi145 2020-01-16T00:02:47.441 INFO:teuthology.provision.fog.smithi145:Deploy complete! 2020-01-16T00:02:47.443 DEBUG:teuthology.parallel:result is None 2020-01-16T00:02:55.451 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi123.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:02:55.473 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-01-16T00:03:01.478 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi123.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:03:01.489 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-01-16T00:03:07.494 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi123.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:03:07.504 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-01-16T00:03:13.506 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi123.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:03:13.516 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-01-16T00:03:19.518 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi123.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:03:19.528 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-01-16T00:03:25.530 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi123.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:03:25.687 INFO:teuthology.orchestra.run.smithi123:> hostname 2020-01-16T00:03:26.297 INFO:teuthology.orchestra.run.smithi123.stdout:smithi123.front.sepia.ceph.com 2020-01-16T00:03:26.297 INFO:teuthology.orchestra.run.smithi123:> grep smithi123.front.sepia.ceph.com /etc/hosts 2020-01-16T00:03:26.304 INFO:teuthology.orchestra.run.smithi123.stdout:172.21.15.123 smithi123.front.sepia.ceph.com smithi123 2020-01-16T00:03:26.304 INFO:teuthology.orchestra.run.smithi123:> sudo hostname smithi123 2020-01-16T00:03:26.428 INFO:teuthology.orchestra.run.smithi123:> sudo sed -i -e 's/smithi123.front.sepia.ceph.com/smithi123/g' /etc/hosts 2020-01-16T00:03:26.442 INFO:teuthology.orchestra.run.smithi123:> sudo sed -i -e 's/172.21.15.123/172.21.15.123/g' /etc/hosts 2020-01-16T00:03:26.497 INFO:teuthology.orchestra.run.smithi123:> sudo sed -i -e 's/smithi123.front.sepia.ceph.com/smithi123/g' /etc/hostname 2020-01-16T00:03:26.556 INFO:teuthology.orchestra.run.smithi123:> sudo hostname smithi123 2020-01-16T00:03:26.611 INFO:teuthology.provision.fog.smithi123:Deploy complete! 2020-01-16T00:03:26.613 DEBUG:teuthology.parallel:result is None 2020-01-16T00:03:26.763 INFO:teuthology.lock.ops:Checking smithi123.front.sepia.ceph.com 2020-01-16T00:03:26.764 INFO:teuthology.lock.ops:New key found. Updating... 2020-01-16T00:03:26.777 INFO:teuthology.lock.ops:Checking smithi145.front.sepia.ceph.com 2020-01-16T00:03:26.777 INFO:teuthology.lock.ops:New key found. Updating... 2020-01-16T00:03:26.792 INFO:teuthology.lock.ops:Updating [smithi123.front.sepia.ceph.com]: set os type and version on server 2020-01-16T00:03:26.792 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-01-16T00:03:26.793 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi123.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:03:26.863 INFO:teuthology.orchestra.run.smithi123:> true 2020-01-16T00:03:27.042 INFO:teuthology.orchestra.run.smithi123:> uname -m 2020-01-16T00:03:27.090 INFO:teuthology.orchestra.run.smithi123.stdout:x86_64 2020-01-16T00:03:27.090 INFO:teuthology.orchestra.run.smithi123:> cat /etc/os-release 2020-01-16T00:03:27.136 INFO:teuthology.orchestra.run.smithi123.stdout:NAME="Ubuntu" 2020-01-16T00:03:27.137 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION="18.04.3 LTS (Bionic Beaver)" 2020-01-16T00:03:27.137 INFO:teuthology.orchestra.run.smithi123.stdout:ID=ubuntu 2020-01-16T00:03:27.137 INFO:teuthology.orchestra.run.smithi123.stdout:ID_LIKE=debian 2020-01-16T00:03:27.137 INFO:teuthology.orchestra.run.smithi123.stdout:PRETTY_NAME="Ubuntu 18.04.3 LTS" 2020-01-16T00:03:27.138 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION_ID="18.04" 2020-01-16T00:03:27.138 INFO:teuthology.orchestra.run.smithi123.stdout:HOME_URL="https://www.ubuntu.com/" 2020-01-16T00:03:27.138 INFO:teuthology.orchestra.run.smithi123.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-01-16T00:03:27.138 INFO:teuthology.orchestra.run.smithi123.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-01-16T00:03:27.138 INFO:teuthology.orchestra.run.smithi123.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-01-16T00:03:27.138 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION_CODENAME=bionic 2020-01-16T00:03:27.139 INFO:teuthology.orchestra.run.smithi123.stdout:UBUNTU_CODENAME=bionic 2020-01-16T00:03:27.139 INFO:teuthology.lock.ops:Updating smithi123.front.sepia.ceph.com on lock server 2020-01-16T00:03:27.157 INFO:teuthology.lock.ops:Updating [smithi145.front.sepia.ceph.com]: set os type and version on server 2020-01-16T00:03:27.157 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-01-16T00:03:27.157 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi145.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:03:27.221 INFO:teuthology.orchestra.run.smithi145:> true 2020-01-16T00:03:27.447 INFO:teuthology.orchestra.run.smithi145:> uname -m 2020-01-16T00:03:27.496 INFO:teuthology.orchestra.run.smithi145.stdout:x86_64 2020-01-16T00:03:27.496 INFO:teuthology.orchestra.run.smithi145:> cat /etc/os-release 2020-01-16T00:03:27.548 INFO:teuthology.orchestra.run.smithi145.stdout:NAME="Ubuntu" 2020-01-16T00:03:27.549 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION="18.04.3 LTS (Bionic Beaver)" 2020-01-16T00:03:27.549 INFO:teuthology.orchestra.run.smithi145.stdout:ID=ubuntu 2020-01-16T00:03:27.549 INFO:teuthology.orchestra.run.smithi145.stdout:ID_LIKE=debian 2020-01-16T00:03:27.549 INFO:teuthology.orchestra.run.smithi145.stdout:PRETTY_NAME="Ubuntu 18.04.3 LTS" 2020-01-16T00:03:27.549 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION_ID="18.04" 2020-01-16T00:03:27.549 INFO:teuthology.orchestra.run.smithi145.stdout:HOME_URL="https://www.ubuntu.com/" 2020-01-16T00:03:27.550 INFO:teuthology.orchestra.run.smithi145.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-01-16T00:03:27.550 INFO:teuthology.orchestra.run.smithi145.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-01-16T00:03:27.550 INFO:teuthology.orchestra.run.smithi145.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-01-16T00:03:27.550 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION_CODENAME=bionic 2020-01-16T00:03:27.550 INFO:teuthology.orchestra.run.smithi145.stdout:UBUNTU_CODENAME=bionic 2020-01-16T00:03:27.550 INFO:teuthology.lock.ops:Updating smithi145.front.sepia.ceph.com on lock server 2020-01-16T00:03:27.567 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far 2020-01-16T00:03:27.599 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi123.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDAtTFcxpKk/Gf1EokgFNST3fRhvfOR2OgUUPB039HuVj1o29IATfDCJB+NEPfrjnv6ngyyYlv+e53hHnJBusfP/J7Fds5US04wbUVrRluJLK2I7tGotRl7rUeXh4LwZQsQ462Rk3Z3uu0F+QOdu6mxy7SRHGthDMCktNG/JJNnVhP2uyEZGBKwLuQcWw5EaVtwkgU75/uxsytPvSJ23T5Imo10Ez1wrOicyDh8D6NWs0dhFra7wG2AnjSogEMGCWOBo/mrNin14hdV+fi+kakbXlWywcBixbzo1DtvvXzE4Nd+y3AdUOl6oEFkF0d8aSRk+Yd+4cndtwIBLBvNm1Ad smithi145.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQClIIpQOIOKY4e1T8vM7W7KgPjkdMmoU9OgSSroJKKygQTZD9LPRr9vmMvD8exQep8CkGi+r+w5HCyZxbAnimaI2l0R/HdmKDqpUTboHS7Rp7AjyHLNXvVXHNPrMX66lebTjNFZ/6/24+AKOOVXQm4M7OdKN6GdYpEt8sHgEJrfRfiRLKDqY6M8F50C73z5XFmW6RhSXAkuuNFxeZiZxziJo5PJe181LSO6vBLKkNJ0LPGcYlczMFesXwOWVd0nPLAz5UuYv8kIH2IDuOZlBiFEVwIlWbIluN9UH1DfbFMimzDViBgv7tJBxbUbgqc48CEjVbBJlv/lQT/On6YK3zDr 2020-01-16T00:03:27.599 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-01-16T00:03:27.660 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-01-16T00:03:27.670 INFO:teuthology.task.internal:Saving configuration 2020-01-16T00:03:27.688 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-01-16T00:03:27.721 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-01-16T00:03:27.737 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2020-01-15 23:57:48.862442', u'locked_by': u'scheduled_sage@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi123.front.sepia.ceph.com', u'os_version': u'18.04', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'ubuntu', u'arch': u'x86_64', u'ssh_pub_key': u'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAID473kCFCqr28mPCJbFt74RaDy7Y22TX14TmKUujP3Ow', u'description': u'/home/teuthworker/archive/sage-2020-01-15_21:23:44-rados-wip-sage-testing-2020-01-15-1209-distro-basic-smithi/4671691'} 2020-01-16T00:03:27.752 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2020-01-15 23:57:48.862466', u'locked_by': u'scheduled_sage@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi145.front.sepia.ceph.com', u'os_version': u'18.04', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'ubuntu', u'arch': u'x86_64', u'ssh_pub_key': u'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJ81cYEfQBUo92WwfBmBabP49JMv3B09VcN9Ryw2ix0m', u'description': u'/home/teuthworker/archive/sage-2020-01-15_21:23:44-rados-wip-sage-testing-2020-01-15-1209-distro-basic-smithi/4671691'} 2020-01-16T00:03:27.752 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-01-16T00:03:27.762 INFO:teuthology.task.internal:roles: ubuntu@smithi123.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2020-01-16T00:03:27.762 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2020-01-16T00:03:27.762 INFO:teuthology.run_tasks:Running task console_log... 2020-01-16T00:03:27.836 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2020-01-16T00:03:27.837 INFO:teuthology.run_tasks:Running task internal.connect... 2020-01-16T00:03:27.849 INFO:teuthology.task.internal:Opening connections... 2020-01-16T00:03:27.849 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi123.front.sepia.ceph.com 2020-01-16T00:03:27.851 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi123.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:03:27.926 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi145.front.sepia.ceph.com 2020-01-16T00:03:27.926 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi145.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:03:27.999 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-01-16T00:03:28.010 INFO:teuthology.orchestra.run.smithi123:> uname -m 2020-01-16T00:03:28.122 INFO:teuthology.orchestra.run.smithi123.stdout:x86_64 2020-01-16T00:03:28.122 INFO:teuthology.orchestra.run.smithi123:> cat /etc/os-release 2020-01-16T00:03:28.173 INFO:teuthology.orchestra.run.smithi123.stdout:NAME="Ubuntu" 2020-01-16T00:03:28.173 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION="18.04.3 LTS (Bionic Beaver)" 2020-01-16T00:03:28.173 INFO:teuthology.orchestra.run.smithi123.stdout:ID=ubuntu 2020-01-16T00:03:28.173 INFO:teuthology.orchestra.run.smithi123.stdout:ID_LIKE=debian 2020-01-16T00:03:28.173 INFO:teuthology.orchestra.run.smithi123.stdout:PRETTY_NAME="Ubuntu 18.04.3 LTS" 2020-01-16T00:03:28.173 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION_ID="18.04" 2020-01-16T00:03:28.174 INFO:teuthology.orchestra.run.smithi123.stdout:HOME_URL="https://www.ubuntu.com/" 2020-01-16T00:03:28.174 INFO:teuthology.orchestra.run.smithi123.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-01-16T00:03:28.174 INFO:teuthology.orchestra.run.smithi123.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-01-16T00:03:28.174 INFO:teuthology.orchestra.run.smithi123.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-01-16T00:03:28.174 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION_CODENAME=bionic 2020-01-16T00:03:28.174 INFO:teuthology.orchestra.run.smithi123.stdout:UBUNTU_CODENAME=bionic 2020-01-16T00:03:28.175 INFO:teuthology.lock.ops:Updating smithi123.front.sepia.ceph.com on lock server 2020-01-16T00:03:28.194 INFO:teuthology.orchestra.run.smithi145:> uname -m 2020-01-16T00:03:28.200 INFO:teuthology.orchestra.run.smithi145.stdout:x86_64 2020-01-16T00:03:28.200 INFO:teuthology.orchestra.run.smithi145:> cat /etc/os-release 2020-01-16T00:03:28.248 INFO:teuthology.orchestra.run.smithi145.stdout:NAME="Ubuntu" 2020-01-16T00:03:28.249 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION="18.04.3 LTS (Bionic Beaver)" 2020-01-16T00:03:28.249 INFO:teuthology.orchestra.run.smithi145.stdout:ID=ubuntu 2020-01-16T00:03:28.249 INFO:teuthology.orchestra.run.smithi145.stdout:ID_LIKE=debian 2020-01-16T00:03:28.249 INFO:teuthology.orchestra.run.smithi145.stdout:PRETTY_NAME="Ubuntu 18.04.3 LTS" 2020-01-16T00:03:28.249 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION_ID="18.04" 2020-01-16T00:03:28.249 INFO:teuthology.orchestra.run.smithi145.stdout:HOME_URL="https://www.ubuntu.com/" 2020-01-16T00:03:28.250 INFO:teuthology.orchestra.run.smithi145.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-01-16T00:03:28.250 INFO:teuthology.orchestra.run.smithi145.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-01-16T00:03:28.250 INFO:teuthology.orchestra.run.smithi145.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-01-16T00:03:28.250 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION_CODENAME=bionic 2020-01-16T00:03:28.250 INFO:teuthology.orchestra.run.smithi145.stdout:UBUNTU_CODENAME=bionic 2020-01-16T00:03:28.250 INFO:teuthology.lock.ops:Updating smithi145.front.sepia.ceph.com on lock server 2020-01-16T00:03:28.265 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-01-16T00:03:28.283 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-01-16T00:03:28.296 INFO:teuthology.task.internal:Checking for old test directory... 2020-01-16T00:03:28.296 INFO:teuthology.orchestra.run.smithi123:> test '!' -e /home/ubuntu/cephtest 2020-01-16T00:03:28.298 INFO:teuthology.orchestra.run.smithi145:> test '!' -e /home/ubuntu/cephtest 2020-01-16T00:03:28.303 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-01-16T00:03:28.383 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-01-16T00:03:28.384 INFO:teuthology.orchestra.run.smithi123:> test -z $(ls -A /var/lib/ceph) 2020-01-16T00:03:28.386 INFO:teuthology.orchestra.run.smithi145:> test -z $(ls -A /var/lib/ceph) 2020-01-16T00:03:28.401 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-01-16T00:03:28.444 INFO:teuthology.run_tasks:Running task kernel... 2020-01-16T00:03:28.461 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'mon.c': {'sha1': 'distro', 'kdb': True}, 'client.0': {'sha1': 'distro', 'kdb': True}, 'osd.1': {'sha1': 'distro', 'kdb': True}, 'osd.0': {'sha1': 'distro', 'kdb': True}, 'osd.3': {'sha1': 'distro', 'kdb': True}, 'osd.2': {'sha1': 'distro', 'kdb': True}, 'osd.5': {'sha1': 'distro', 'kdb': True}, 'osd.4': {'sha1': 'distro', 'kdb': True}, 'osd.7': {'sha1': 'distro', 'kdb': True}, 'osd.6': {'sha1': 'distro', 'kdb': True}, 'mon.b': {'sha1': 'distro', 'kdb': True}, 'mgr.y': {'sha1': 'distro', 'kdb': True}, 'mgr.x': {'sha1': 'distro', 'kdb': True}, 'client.1': {'sha1': 'distro', 'kdb': True}} 2020-01-16T00:03:28.461 INFO:teuthology.task.kernel:config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'mon.b': {'sha1': 'distro', 'kdb': True}}, timeout 300 2020-01-16T00:03:28.462 INFO:teuthology.orchestra.run.smithi123:> uname -r 2020-01-16T00:03:28.467 INFO:teuthology.orchestra.run.smithi123.stdout:4.15.0-72-generic 2020-01-16T00:03:28.468 INFO:teuthology.task.kernel:Running kernel on smithi123: 4.15.0-72-generic 2020-01-16T00:03:28.468 INFO:teuthology.orchestra.run.smithi123:> sudo apt-get clean 2020-01-16T00:03:28.572 INFO:teuthology.orchestra.run.smithi123:> sudo apt-get update 2020-01-16T00:03:28.851 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-01-16T00:03:28.851 INFO:teuthology.orchestra.run.smithi123.stdout:Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2020-01-16T00:03:28.921 INFO:teuthology.orchestra.run.smithi123.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2020-01-16T00:03:29.197 INFO:teuthology.orchestra.run.smithi123.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2020-01-16T00:03:29.234 INFO:teuthology.orchestra.run.smithi123.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [417 kB] 2020-01-16T00:03:29.359 INFO:teuthology.orchestra.run.smithi123.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [629 kB] 2020-01-16T00:03:29.522 INFO:teuthology.orchestra.run.smithi123.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [606 kB] 2020-01-16T00:03:29.615 INFO:teuthology.orchestra.run.smithi123.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [196 kB] 2020-01-16T00:03:29.634 INFO:teuthology.orchestra.run.smithi123.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [17.0 kB] 2020-01-16T00:03:29.635 INFO:teuthology.orchestra.run.smithi123.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [5,032 B] 2020-01-16T00:03:29.636 INFO:teuthology.orchestra.run.smithi123.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [633 kB] 2020-01-16T00:03:29.641 INFO:teuthology.orchestra.run.smithi123.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [830 kB] 2020-01-16T00:03:29.692 INFO:teuthology.orchestra.run.smithi123.stdout:Get:13 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [607 kB] 2020-01-16T00:03:29.726 INFO:teuthology.orchestra.run.smithi123.stdout:Get:14 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [213 kB] 2020-01-16T00:03:29.728 INFO:teuthology.orchestra.run.smithi123.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [290 kB] 2020-01-16T00:03:29.737 INFO:teuthology.orchestra.run.smithi123.stdout:Get:16 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [6,116 B] 2020-01-16T00:03:29.737 INFO:teuthology.orchestra.run.smithi123.stdout:Get:17 http://security.ubuntu.com/ubuntu bionic-security/multiverse i386 Packages [4,284 B] 2020-01-16T00:03:29.743 INFO:teuthology.orchestra.run.smithi123.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [26.0 kB] 2020-01-16T00:03:29.745 INFO:teuthology.orchestra.run.smithi123.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [6,976 B] 2020-01-16T00:03:29.746 INFO:teuthology.orchestra.run.smithi123.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,000 kB] 2020-01-16T00:03:29.811 INFO:teuthology.orchestra.run.smithi123.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,038 kB] 2020-01-16T00:03:29.835 INFO:teuthology.orchestra.run.smithi123.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [321 kB] 2020-01-16T00:03:29.845 INFO:teuthology.orchestra.run.smithi123.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [9,500 B] 2020-01-16T00:03:29.845 INFO:teuthology.orchestra.run.smithi123.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [7,472 B] 2020-01-16T00:03:29.845 INFO:teuthology.orchestra.run.smithi123.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [4,540 B] 2020-01-16T00:03:29.845 INFO:teuthology.orchestra.run.smithi123.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic-backports/universe i386 Packages [4,036 B] 2020-01-16T00:03:29.891 INFO:teuthology.orchestra.run.smithi123.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [4,044 B] 2020-01-16T00:03:29.892 INFO:teuthology.orchestra.run.smithi123.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic-backports/universe Translation-en [1,900 B] 2020-01-16T00:03:30.588 INFO:teuthology.orchestra.run.smithi123.stdout:Fetched 7,127 kB in 2s (3,918 kB/s) 2020-01-16T00:03:31.922 INFO:teuthology.orchestra.run.smithi123.stdout:Reading package lists... 2020-01-16T00:03:31.966 INFO:teuthology.orchestra.run.smithi123:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-01-16T00:03:32.067 INFO:teuthology.orchestra.run.smithi123.stdout:Reading package lists... 2020-01-16T00:03:32.308 INFO:teuthology.orchestra.run.smithi123.stdout:Building dependency tree... 2020-01-16T00:03:32.308 INFO:teuthology.orchestra.run.smithi123.stdout:Reading state information... 2020-01-16T00:03:32.380 INFO:teuthology.orchestra.run.smithi123.stderr:E: Unable to locate package linux-image-current-generic 2020-01-16T00:03:32.382 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-01-16T00:03:32.382 INFO:teuthology.orchestra.run.smithi123:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-01-16T00:03:32.471 INFO:teuthology.orchestra.run.smithi123.stdout:Reading package lists... 2020-01-16T00:03:32.718 INFO:teuthology.orchestra.run.smithi123.stdout:Building dependency tree... 2020-01-16T00:03:32.719 INFO:teuthology.orchestra.run.smithi123.stdout:Reading state information... 2020-01-16T00:03:32.879 INFO:teuthology.orchestra.run.smithi123.stdout:The following additional packages will be installed: 2020-01-16T00:03:32.880 INFO:teuthology.orchestra.run.smithi123.stdout: linux-generic linux-headers-4.15.0-74 linux-headers-4.15.0-74-generic 2020-01-16T00:03:32.880 INFO:teuthology.orchestra.run.smithi123.stdout: linux-headers-generic linux-image-4.15.0-74-generic 2020-01-16T00:03:32.880 INFO:teuthology.orchestra.run.smithi123.stdout: linux-modules-4.15.0-74-generic linux-modules-extra-4.15.0-74-generic 2020-01-16T00:03:32.881 INFO:teuthology.orchestra.run.smithi123.stdout:Suggested packages: 2020-01-16T00:03:32.881 INFO:teuthology.orchestra.run.smithi123.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2020-01-16T00:03:32.882 INFO:teuthology.orchestra.run.smithi123.stdout:Recommended packages: 2020-01-16T00:03:32.882 INFO:teuthology.orchestra.run.smithi123.stdout: thermald 2020-01-16T00:03:32.913 INFO:teuthology.orchestra.run.smithi123.stdout:The following NEW packages will be installed: 2020-01-16T00:03:32.914 INFO:teuthology.orchestra.run.smithi123.stdout: linux-headers-4.15.0-74 linux-headers-4.15.0-74-generic 2020-01-16T00:03:32.914 INFO:teuthology.orchestra.run.smithi123.stdout: linux-image-4.15.0-74-generic linux-modules-4.15.0-74-generic 2020-01-16T00:03:32.914 INFO:teuthology.orchestra.run.smithi123.stdout: linux-modules-extra-4.15.0-74-generic 2020-01-16T00:03:32.915 INFO:teuthology.orchestra.run.smithi123.stdout:The following packages will be upgraded: 2020-01-16T00:03:32.916 INFO:teuthology.orchestra.run.smithi123.stdout: linux-generic linux-headers-generic linux-image-generic 2020-01-16T00:03:33.097 INFO:teuthology.orchestra.run.smithi123.stdout:3 upgraded, 5 newly installed, 0 to remove and 28 not upgraded. 2020-01-16T00:03:33.097 INFO:teuthology.orchestra.run.smithi123.stdout:Need to get 65.9 MB of archives. 2020-01-16T00:03:33.098 INFO:teuthology.orchestra.run.smithi123.stdout:After this operation, 332 MB of additional disk space will be used. 2020-01-16T00:03:33.098 INFO:teuthology.orchestra.run.smithi123.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-74-generic amd64 4.15.0-74.84 [13.0 MB] 2020-01-16T00:03:34.117 INFO:teuthology.orchestra.run.smithi123.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-74-generic amd64 4.15.0-74.84 [7,993 kB] 2020-01-16T00:03:34.311 INFO:teuthology.orchestra.run.smithi123.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-74-generic amd64 4.15.0-74.84 [32.7 MB] 2020-01-16T00:03:35.262 INFO:teuthology.orchestra.run.smithi123.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.74.76 [1,864 B] 2020-01-16T00:03:35.262 INFO:teuthology.orchestra.run.smithi123.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.74.76 [2,388 B] 2020-01-16T00:03:35.263 INFO:teuthology.orchestra.run.smithi123.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-74 all 4.15.0-74.84 [11.0 MB] 2020-01-16T00:03:35.542 INFO:teuthology.orchestra.run.smithi123.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-74-generic amd64 4.15.0-74.84 [1,122 kB] 2020-01-16T00:03:35.602 INFO:teuthology.orchestra.run.smithi123.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.74.76 [2,348 B] 2020-01-16T00:03:36.079 INFO:teuthology.orchestra.run.smithi123.stdout:Fetched 65.9 MB in 3s (24.7 MB/s) 2020-01-16T00:03:36.244 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package linux-modules-4.15.0-74-generic. 2020-01-16T00:03:36.280 INFO:teuthology.orchestra.run.smithi123.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 101858 files and directories currently installed.) 2020-01-16T00:03:36.282 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../0-linux-modules-4.15.0-74-generic_4.15.0-74.84_amd64.deb ... 2020-01-16T00:03:36.357 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking linux-modules-4.15.0-74-generic (4.15.0-74.84) ... 2020-01-16T00:03:38.016 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package linux-image-4.15.0-74-generic. 2020-01-16T00:03:38.030 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../1-linux-image-4.15.0-74-generic_4.15.0-74.84_amd64.deb ... 2020-01-16T00:03:38.133 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking linux-image-4.15.0-74-generic (4.15.0-74.84) ... 2020-01-16T00:03:38.652 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-74-generic. 2020-01-16T00:03:38.664 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-74-generic_4.15.0-74.84_amd64.deb ... 2020-01-16T00:03:38.702 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking linux-modules-extra-4.15.0-74-generic (4.15.0-74.84) ... 2020-01-16T00:03:42.426 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../3-linux-generic_4.15.0.74.76_amd64.deb ... 2020-01-16T00:03:42.538 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking linux-generic (4.15.0.74.76) over (4.15.0.72.74) ... 2020-01-16T00:03:42.845 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.74.76_amd64.deb ... 2020-01-16T00:03:42.975 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking linux-image-generic (4.15.0.74.76) over (4.15.0.72.74) ... 2020-01-16T00:03:43.344 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package linux-headers-4.15.0-74. 2020-01-16T00:03:43.358 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../5-linux-headers-4.15.0-74_4.15.0-74.84_all.deb ... 2020-01-16T00:03:43.394 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking linux-headers-4.15.0-74 (4.15.0-74.84) ... 2020-01-16T00:03:48.192 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package linux-headers-4.15.0-74-generic. 2020-01-16T00:03:48.210 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../6-linux-headers-4.15.0-74-generic_4.15.0-74.84_amd64.deb ... 2020-01-16T00:03:48.250 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking linux-headers-4.15.0-74-generic (4.15.0-74.84) ... 2020-01-16T00:03:50.560 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.74.76_amd64.deb ... 2020-01-16T00:03:50.682 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking linux-headers-generic (4.15.0.74.76) over (4.15.0.72.74) ... 2020-01-16T00:03:51.028 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up linux-headers-4.15.0-74 (4.15.0-74.84) ... 2020-01-16T00:03:51.178 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up linux-modules-4.15.0-74-generic (4.15.0-74.84) ... 2020-01-16T00:03:51.739 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up linux-image-4.15.0-74-generic (4.15.0-74.84) ... 2020-01-16T00:03:52.234 INFO:teuthology.orchestra.run.smithi123.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-74-generic 2020-01-16T00:03:52.235 INFO:teuthology.orchestra.run.smithi123.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-74-generic 2020-01-16T00:03:52.366 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up linux-modules-extra-4.15.0-74-generic (4.15.0-74.84) ... 2020-01-16T00:03:53.236 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up linux-headers-4.15.0-74-generic (4.15.0-74.84) ... 2020-01-16T00:03:53.660 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up linux-image-generic (4.15.0.74.76) ... 2020-01-16T00:03:53.811 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up linux-headers-generic (4.15.0.74.76) ... 2020-01-16T00:03:53.937 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up linux-generic (4.15.0.74.76) ... 2020-01-16T00:03:54.063 INFO:teuthology.orchestra.run.smithi123.stdout:Processing triggers for linux-image-4.15.0-74-generic (4.15.0-74.84) ... 2020-01-16T00:03:54.259 INFO:teuthology.orchestra.run.smithi123.stdout:/etc/kernel/postinst.d/initramfs-tools: 2020-01-16T00:03:54.259 INFO:teuthology.orchestra.run.smithi123.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-74-generic 2020-01-16T00:04:08.850 INFO:teuthology.orchestra.run.smithi123.stdout:/etc/kernel/postinst.d/zz-update-grub: 2020-01-16T00:04:08.851 INFO:teuthology.orchestra.run.smithi123.stdout:Sourcing file `/etc/default/grub' 2020-01-16T00:04:08.878 INFO:teuthology.orchestra.run.smithi123.stdout:Generating grub configuration file ... 2020-01-16T00:04:09.202 INFO:teuthology.orchestra.run.smithi123.stdout:Found linux image: /boot/vmlinuz-4.15.0-74-generic 2020-01-16T00:04:09.248 INFO:teuthology.orchestra.run.smithi123.stdout:Found initrd image: /boot/initrd.img-4.15.0-74-generic 2020-01-16T00:04:09.541 INFO:teuthology.orchestra.run.smithi123.stdout:Found linux image: /boot/vmlinuz-4.15.0-72-generic 2020-01-16T00:04:09.566 INFO:teuthology.orchestra.run.smithi123.stdout:Found initrd image: /boot/initrd.img-4.15.0-72-generic 2020-01-16T00:04:11.031 INFO:teuthology.orchestra.run.smithi123.stdout:done 2020-01-16T00:04:11.420 INFO:teuthology.orchestra.run.smithi123:> dpkg -s linux-image-generic 2020-01-16T00:04:11.442 INFO:teuthology.orchestra.run.smithi123.stdout:Package: linux-image-generic 2020-01-16T00:04:11.443 INFO:teuthology.orchestra.run.smithi123.stdout:Status: install ok installed 2020-01-16T00:04:11.443 INFO:teuthology.orchestra.run.smithi123.stdout:Priority: optional 2020-01-16T00:04:11.443 INFO:teuthology.orchestra.run.smithi123.stdout:Section: kernel 2020-01-16T00:04:11.443 INFO:teuthology.orchestra.run.smithi123.stdout:Installed-Size: 15 2020-01-16T00:04:11.444 INFO:teuthology.orchestra.run.smithi123.stdout:Maintainer: Ubuntu Kernel Team 2020-01-16T00:04:11.444 INFO:teuthology.orchestra.run.smithi123.stdout:Architecture: amd64 2020-01-16T00:04:11.444 INFO:teuthology.orchestra.run.smithi123.stdout:Source: linux-meta 2020-01-16T00:04:11.444 INFO:teuthology.orchestra.run.smithi123.stdout:Version: 4.15.0.74.76 2020-01-16T00:04:11.445 INFO:teuthology.orchestra.run.smithi123.stdout:Depends: linux-image-4.15.0-74-generic, linux-modules-extra-4.15.0-74-generic, linux-firmware, intel-microcode, amd64-microcode 2020-01-16T00:04:11.445 INFO:teuthology.orchestra.run.smithi123.stdout:Recommends: thermald 2020-01-16T00:04:11.445 INFO:teuthology.orchestra.run.smithi123.stdout:Description: Generic Linux kernel image 2020-01-16T00:04:11.445 INFO:teuthology.orchestra.run.smithi123.stdout: This package will always depend on the latest generic kernel image 2020-01-16T00:04:11.445 INFO:teuthology.orchestra.run.smithi123.stdout: available. 2020-01-16T00:04:11.446 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-72-generic Expected: 4.15.0-74-generic 2020-01-16T00:04:11.446 INFO:teuthology.orchestra.run.smithi145:> uname -r 2020-01-16T00:04:11.452 INFO:teuthology.orchestra.run.smithi145.stdout:4.15.0-72-generic 2020-01-16T00:04:11.453 INFO:teuthology.task.kernel:Running kernel on smithi145: 4.15.0-72-generic 2020-01-16T00:04:11.453 INFO:teuthology.orchestra.run.smithi145:> sudo apt-get clean 2020-01-16T00:04:11.612 INFO:teuthology.orchestra.run.smithi145:> sudo apt-get update 2020-01-16T00:04:11.754 INFO:teuthology.orchestra.run.smithi145.stdout:Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2020-01-16T00:04:11.880 INFO:teuthology.orchestra.run.smithi145.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-01-16T00:04:11.960 INFO:teuthology.orchestra.run.smithi145.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2020-01-16T00:04:12.013 INFO:teuthology.orchestra.run.smithi145.stdout:Get:4 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [606 kB] 2020-01-16T00:04:12.100 INFO:teuthology.orchestra.run.smithi145.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [417 kB] 2020-01-16T00:04:12.112 INFO:teuthology.orchestra.run.smithi145.stdout:Get:6 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [196 kB] 2020-01-16T00:04:12.114 INFO:teuthology.orchestra.run.smithi145.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [17.0 kB] 2020-01-16T00:04:12.115 INFO:teuthology.orchestra.run.smithi145.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [5,032 B] 2020-01-16T00:04:12.115 INFO:teuthology.orchestra.run.smithi145.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [607 kB] 2020-01-16T00:04:12.134 INFO:teuthology.orchestra.run.smithi145.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [633 kB] 2020-01-16T00:04:12.141 INFO:teuthology.orchestra.run.smithi145.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [213 kB] 2020-01-16T00:04:12.144 INFO:teuthology.orchestra.run.smithi145.stdout:Get:12 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [6,116 B] 2020-01-16T00:04:12.145 INFO:teuthology.orchestra.run.smithi145.stdout:Get:13 http://security.ubuntu.com/ubuntu bionic-security/multiverse i386 Packages [4,284 B] 2020-01-16T00:04:12.239 INFO:teuthology.orchestra.run.smithi145.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2020-01-16T00:04:12.357 INFO:teuthology.orchestra.run.smithi145.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [629 kB] 2020-01-16T00:04:12.552 INFO:teuthology.orchestra.run.smithi145.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [830 kB] 2020-01-16T00:04:12.644 INFO:teuthology.orchestra.run.smithi145.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [290 kB] 2020-01-16T00:04:12.665 INFO:teuthology.orchestra.run.smithi145.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [26.0 kB] 2020-01-16T00:04:12.666 INFO:teuthology.orchestra.run.smithi145.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [6,976 B] 2020-01-16T00:04:12.667 INFO:teuthology.orchestra.run.smithi145.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,038 kB] 2020-01-16T00:04:12.720 INFO:teuthology.orchestra.run.smithi145.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,000 kB] 2020-01-16T00:04:12.754 INFO:teuthology.orchestra.run.smithi145.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [321 kB] 2020-01-16T00:04:12.763 INFO:teuthology.orchestra.run.smithi145.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [9,500 B] 2020-01-16T00:04:12.764 INFO:teuthology.orchestra.run.smithi145.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [7,472 B] 2020-01-16T00:04:12.764 INFO:teuthology.orchestra.run.smithi145.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [4,540 B] 2020-01-16T00:04:12.764 INFO:teuthology.orchestra.run.smithi145.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic-backports/universe i386 Packages [4,036 B] 2020-01-16T00:04:12.806 INFO:teuthology.orchestra.run.smithi145.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [4,044 B] 2020-01-16T00:04:12.806 INFO:teuthology.orchestra.run.smithi145.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic-backports/universe Translation-en [1,900 B] 2020-01-16T00:04:13.272 INFO:teuthology.orchestra.run.smithi145.stdout:Fetched 7,127 kB in 1s (4,892 kB/s) 2020-01-16T00:04:14.627 INFO:teuthology.orchestra.run.smithi145.stdout:Reading package lists... 2020-01-16T00:04:14.671 INFO:teuthology.orchestra.run.smithi145:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-01-16T00:04:14.767 INFO:teuthology.orchestra.run.smithi145.stdout:Reading package lists... 2020-01-16T00:04:15.017 INFO:teuthology.orchestra.run.smithi145.stdout:Building dependency tree... 2020-01-16T00:04:15.017 INFO:teuthology.orchestra.run.smithi145.stdout:Reading state information... 2020-01-16T00:04:15.090 INFO:teuthology.orchestra.run.smithi145.stderr:E: Unable to locate package linux-image-current-generic 2020-01-16T00:04:15.092 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-01-16T00:04:15.092 INFO:teuthology.orchestra.run.smithi145:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-01-16T00:04:15.172 INFO:teuthology.orchestra.run.smithi145.stdout:Reading package lists... 2020-01-16T00:04:15.409 INFO:teuthology.orchestra.run.smithi145.stdout:Building dependency tree... 2020-01-16T00:04:15.409 INFO:teuthology.orchestra.run.smithi145.stdout:Reading state information... 2020-01-16T00:04:15.566 INFO:teuthology.orchestra.run.smithi145.stdout:The following additional packages will be installed: 2020-01-16T00:04:15.566 INFO:teuthology.orchestra.run.smithi145.stdout: linux-generic linux-headers-4.15.0-74 linux-headers-4.15.0-74-generic 2020-01-16T00:04:15.566 INFO:teuthology.orchestra.run.smithi145.stdout: linux-headers-generic linux-image-4.15.0-74-generic 2020-01-16T00:04:15.567 INFO:teuthology.orchestra.run.smithi145.stdout: linux-modules-4.15.0-74-generic linux-modules-extra-4.15.0-74-generic 2020-01-16T00:04:15.568 INFO:teuthology.orchestra.run.smithi145.stdout:Suggested packages: 2020-01-16T00:04:15.568 INFO:teuthology.orchestra.run.smithi145.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2020-01-16T00:04:15.568 INFO:teuthology.orchestra.run.smithi145.stdout:Recommended packages: 2020-01-16T00:04:15.568 INFO:teuthology.orchestra.run.smithi145.stdout: thermald 2020-01-16T00:04:15.601 INFO:teuthology.orchestra.run.smithi145.stdout:The following NEW packages will be installed: 2020-01-16T00:04:15.601 INFO:teuthology.orchestra.run.smithi145.stdout: linux-headers-4.15.0-74 linux-headers-4.15.0-74-generic 2020-01-16T00:04:15.602 INFO:teuthology.orchestra.run.smithi145.stdout: linux-image-4.15.0-74-generic linux-modules-4.15.0-74-generic 2020-01-16T00:04:15.602 INFO:teuthology.orchestra.run.smithi145.stdout: linux-modules-extra-4.15.0-74-generic 2020-01-16T00:04:15.602 INFO:teuthology.orchestra.run.smithi145.stdout:The following packages will be upgraded: 2020-01-16T00:04:15.603 INFO:teuthology.orchestra.run.smithi145.stdout: linux-generic linux-headers-generic linux-image-generic 2020-01-16T00:04:15.779 INFO:teuthology.orchestra.run.smithi145.stdout:3 upgraded, 5 newly installed, 0 to remove and 28 not upgraded. 2020-01-16T00:04:15.779 INFO:teuthology.orchestra.run.smithi145.stdout:Need to get 65.9 MB of archives. 2020-01-16T00:04:15.780 INFO:teuthology.orchestra.run.smithi145.stdout:After this operation, 332 MB of additional disk space will be used. 2020-01-16T00:04:15.780 INFO:teuthology.orchestra.run.smithi145.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-74-generic amd64 4.15.0-74.84 [13.0 MB] 2020-01-16T00:04:16.802 INFO:teuthology.orchestra.run.smithi145.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-74-generic amd64 4.15.0-74.84 [7,993 kB] 2020-01-16T00:04:16.992 INFO:teuthology.orchestra.run.smithi145.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-74-generic amd64 4.15.0-74.84 [32.7 MB] 2020-01-16T00:04:17.904 INFO:teuthology.orchestra.run.smithi145.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.74.76 [1,864 B] 2020-01-16T00:04:17.904 INFO:teuthology.orchestra.run.smithi145.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.74.76 [2,388 B] 2020-01-16T00:04:17.904 INFO:teuthology.orchestra.run.smithi145.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-74 all 4.15.0-74.84 [11.0 MB] 2020-01-16T00:04:18.226 INFO:teuthology.orchestra.run.smithi145.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-74-generic amd64 4.15.0-74.84 [1,122 kB] 2020-01-16T00:04:18.236 INFO:teuthology.orchestra.run.smithi145.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.74.76 [2,348 B] 2020-01-16T00:04:18.732 INFO:teuthology.orchestra.run.smithi145.stdout:Fetched 65.9 MB in 3s (25.1 MB/s) 2020-01-16T00:04:18.897 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package linux-modules-4.15.0-74-generic. 2020-01-16T00:04:18.934 INFO:teuthology.orchestra.run.smithi145.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 101858 files and directories currently installed.) 2020-01-16T00:04:18.937 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../0-linux-modules-4.15.0-74-generic_4.15.0-74.84_amd64.deb ... 2020-01-16T00:04:18.986 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking linux-modules-4.15.0-74-generic (4.15.0-74.84) ... 2020-01-16T00:04:20.562 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package linux-image-4.15.0-74-generic. 2020-01-16T00:04:20.575 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../1-linux-image-4.15.0-74-generic_4.15.0-74.84_amd64.deb ... 2020-01-16T00:04:20.665 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking linux-image-4.15.0-74-generic (4.15.0-74.84) ... 2020-01-16T00:04:21.173 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-74-generic. 2020-01-16T00:04:21.184 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-74-generic_4.15.0-74.84_amd64.deb ... 2020-01-16T00:04:21.223 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking linux-modules-extra-4.15.0-74-generic (4.15.0-74.84) ... 2020-01-16T00:04:24.945 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../3-linux-generic_4.15.0.74.76_amd64.deb ... 2020-01-16T00:04:25.069 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking linux-generic (4.15.0.74.76) over (4.15.0.72.74) ... 2020-01-16T00:04:25.401 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.74.76_amd64.deb ... 2020-01-16T00:04:25.523 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking linux-image-generic (4.15.0.74.76) over (4.15.0.72.74) ... 2020-01-16T00:04:25.876 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package linux-headers-4.15.0-74. 2020-01-16T00:04:25.887 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../5-linux-headers-4.15.0-74_4.15.0-74.84_all.deb ... 2020-01-16T00:04:25.917 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking linux-headers-4.15.0-74 (4.15.0-74.84) ... 2020-01-16T00:04:30.823 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package linux-headers-4.15.0-74-generic. 2020-01-16T00:04:30.845 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../6-linux-headers-4.15.0-74-generic_4.15.0-74.84_amd64.deb ... 2020-01-16T00:04:30.889 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking linux-headers-4.15.0-74-generic (4.15.0-74.84) ... 2020-01-16T00:04:33.198 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.74.76_amd64.deb ... 2020-01-16T00:04:33.320 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking linux-headers-generic (4.15.0.74.76) over (4.15.0.72.74) ... 2020-01-16T00:04:33.640 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up linux-headers-4.15.0-74 (4.15.0-74.84) ... 2020-01-16T00:04:33.800 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up linux-modules-4.15.0-74-generic (4.15.0-74.84) ... 2020-01-16T00:04:34.343 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up linux-image-4.15.0-74-generic (4.15.0-74.84) ... 2020-01-16T00:04:34.799 INFO:teuthology.orchestra.run.smithi145.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-74-generic 2020-01-16T00:04:34.799 INFO:teuthology.orchestra.run.smithi145.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-74-generic 2020-01-16T00:04:34.937 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up linux-modules-extra-4.15.0-74-generic (4.15.0-74.84) ... 2020-01-16T00:04:35.430 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up linux-headers-4.15.0-74-generic (4.15.0-74.84) ... 2020-01-16T00:04:35.556 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up linux-image-generic (4.15.0.74.76) ... 2020-01-16T00:04:35.683 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up linux-headers-generic (4.15.0.74.76) ... 2020-01-16T00:04:35.825 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up linux-generic (4.15.0.74.76) ... 2020-01-16T00:04:35.952 INFO:teuthology.orchestra.run.smithi145.stdout:Processing triggers for linux-image-4.15.0-74-generic (4.15.0-74.84) ... 2020-01-16T00:04:36.173 INFO:teuthology.orchestra.run.smithi145.stdout:/etc/kernel/postinst.d/initramfs-tools: 2020-01-16T00:04:36.173 INFO:teuthology.orchestra.run.smithi145.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-74-generic 2020-01-16T00:04:50.991 INFO:teuthology.orchestra.run.smithi145.stdout:/etc/kernel/postinst.d/zz-update-grub: 2020-01-16T00:04:50.991 INFO:teuthology.orchestra.run.smithi145.stdout:Sourcing file `/etc/default/grub' 2020-01-16T00:04:51.011 INFO:teuthology.orchestra.run.smithi145.stdout:Generating grub configuration file ... 2020-01-16T00:04:51.355 INFO:teuthology.orchestra.run.smithi145.stdout:Found linux image: /boot/vmlinuz-4.15.0-74-generic 2020-01-16T00:04:51.381 INFO:teuthology.orchestra.run.smithi145.stdout:Found initrd image: /boot/initrd.img-4.15.0-74-generic 2020-01-16T00:04:51.687 INFO:teuthology.orchestra.run.smithi145.stdout:Found linux image: /boot/vmlinuz-4.15.0-72-generic 2020-01-16T00:04:51.719 INFO:teuthology.orchestra.run.smithi145.stdout:Found initrd image: /boot/initrd.img-4.15.0-72-generic 2020-01-16T00:04:52.790 INFO:teuthology.orchestra.run.smithi145.stdout:done 2020-01-16T00:04:53.189 INFO:teuthology.orchestra.run.smithi145:> dpkg -s linux-image-generic 2020-01-16T00:04:53.212 INFO:teuthology.orchestra.run.smithi145.stdout:Package: linux-image-generic 2020-01-16T00:04:53.212 INFO:teuthology.orchestra.run.smithi145.stdout:Status: install ok installed 2020-01-16T00:04:53.213 INFO:teuthology.orchestra.run.smithi145.stdout:Priority: optional 2020-01-16T00:04:53.213 INFO:teuthology.orchestra.run.smithi145.stdout:Section: kernel 2020-01-16T00:04:53.213 INFO:teuthology.orchestra.run.smithi145.stdout:Installed-Size: 15 2020-01-16T00:04:53.213 INFO:teuthology.orchestra.run.smithi145.stdout:Maintainer: Ubuntu Kernel Team 2020-01-16T00:04:53.213 INFO:teuthology.orchestra.run.smithi145.stdout:Architecture: amd64 2020-01-16T00:04:53.213 INFO:teuthology.orchestra.run.smithi145.stdout:Source: linux-meta 2020-01-16T00:04:53.213 INFO:teuthology.orchestra.run.smithi145.stdout:Version: 4.15.0.74.76 2020-01-16T00:04:53.214 INFO:teuthology.orchestra.run.smithi145.stdout:Depends: linux-image-4.15.0-74-generic, linux-modules-extra-4.15.0-74-generic, linux-firmware, intel-microcode, amd64-microcode 2020-01-16T00:04:53.214 INFO:teuthology.orchestra.run.smithi145.stdout:Recommends: thermald 2020-01-16T00:04:53.214 INFO:teuthology.orchestra.run.smithi145.stdout:Description: Generic Linux kernel image 2020-01-16T00:04:53.214 INFO:teuthology.orchestra.run.smithi145.stdout: This package will always depend on the latest generic kernel image 2020-01-16T00:04:53.214 INFO:teuthology.orchestra.run.smithi145.stdout: available. 2020-01-16T00:04:53.214 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-72-generic Expected: 4.15.0-74-generic 2020-01-16T00:04:53.215 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2020-01-16T00:04:53.215 DEBUG:teuthology.task.kernel:src is distro, skipping download 2020-01-16T00:04:53.215 DEBUG:teuthology.task.kernel:src is distro, skipping download 2020-01-16T00:04:53.215 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2020-01-16T00:04:53.215 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi123.front.sepia.ceph.com, path=None, version=distro) 2020-01-16T00:04:53.215 INFO:teuthology.orchestra.run.smithi123:> sudo apt-get clean 2020-01-16T00:04:53.318 INFO:teuthology.orchestra.run.smithi123:> sudo apt-get update 2020-01-16T00:04:53.576 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-01-16T00:04:53.576 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:2 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-01-16T00:04:53.647 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-01-16T00:04:53.727 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-01-16T00:04:55.364 INFO:teuthology.orchestra.run.smithi123.stdout:Reading package lists... 2020-01-16T00:04:55.413 INFO:teuthology.orchestra.run.smithi123:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-01-16T00:04:55.494 INFO:teuthology.orchestra.run.smithi123.stdout:Reading package lists... 2020-01-16T00:04:55.727 INFO:teuthology.orchestra.run.smithi123.stdout:Building dependency tree... 2020-01-16T00:04:55.728 INFO:teuthology.orchestra.run.smithi123.stdout:Reading state information... 2020-01-16T00:04:55.806 INFO:teuthology.orchestra.run.smithi123.stderr:E: Unable to locate package linux-image-current-generic 2020-01-16T00:04:55.807 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-01-16T00:04:55.808 INFO:teuthology.orchestra.run.smithi123:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-01-16T00:04:55.883 INFO:teuthology.orchestra.run.smithi123.stdout:Reading package lists... 2020-01-16T00:04:56.132 INFO:teuthology.orchestra.run.smithi123.stdout:Building dependency tree... 2020-01-16T00:04:56.133 INFO:teuthology.orchestra.run.smithi123.stdout:Reading state information... 2020-01-16T00:04:56.323 INFO:teuthology.orchestra.run.smithi123.stdout:linux-image-generic is already the newest version (4.15.0.74.76). 2020-01-16T00:04:56.324 INFO:teuthology.orchestra.run.smithi123.stdout:0 upgraded, 0 newly installed, 0 to remove and 28 not upgraded. 2020-01-16T00:04:56.325 INFO:teuthology.orchestra.run.smithi123:> dpkg -s linux-image-generic 2020-01-16T00:04:56.349 INFO:teuthology.orchestra.run.smithi123.stdout:Package: linux-image-generic 2020-01-16T00:04:56.349 INFO:teuthology.orchestra.run.smithi123.stdout:Status: install ok installed 2020-01-16T00:04:56.350 INFO:teuthology.orchestra.run.smithi123.stdout:Priority: optional 2020-01-16T00:04:56.350 INFO:teuthology.orchestra.run.smithi123.stdout:Section: kernel 2020-01-16T00:04:56.350 INFO:teuthology.orchestra.run.smithi123.stdout:Installed-Size: 15 2020-01-16T00:04:56.350 INFO:teuthology.orchestra.run.smithi123.stdout:Maintainer: Ubuntu Kernel Team 2020-01-16T00:04:56.350 INFO:teuthology.orchestra.run.smithi123.stdout:Architecture: amd64 2020-01-16T00:04:56.350 INFO:teuthology.orchestra.run.smithi123.stdout:Source: linux-meta 2020-01-16T00:04:56.351 INFO:teuthology.orchestra.run.smithi123.stdout:Version: 4.15.0.74.76 2020-01-16T00:04:56.351 INFO:teuthology.orchestra.run.smithi123.stdout:Depends: linux-image-4.15.0-74-generic, linux-modules-extra-4.15.0-74-generic, linux-firmware, intel-microcode, amd64-microcode 2020-01-16T00:04:56.351 INFO:teuthology.orchestra.run.smithi123.stdout:Recommends: thermald 2020-01-16T00:04:56.351 INFO:teuthology.orchestra.run.smithi123.stdout:Description: Generic Linux kernel image 2020-01-16T00:04:56.351 INFO:teuthology.orchestra.run.smithi123.stdout: This package will always depend on the latest generic kernel image 2020-01-16T00:04:56.351 INFO:teuthology.orchestra.run.smithi123.stdout: available. 2020-01-16T00:04:56.352 INFO:teuthology.orchestra.run.smithi123:> mktemp 2020-01-16T00:04:56.398 INFO:teuthology.orchestra.run.smithi123.stdout:/tmp/tmp.Y6CutcVFtu 2020-01-16T00:04:56.399 INFO:teuthology.orchestra.run.smithi123:> sudo cp /boot/grub/grub.cfg /tmp/tmp.Y6CutcVFtu 2020-01-16T00:04:56.455 INFO:teuthology.orchestra.run.smithi123:> sudo chmod 0666 /tmp/tmp.Y6CutcVFtu 2020-01-16T00:04:56.565 DEBUG:teuthology.orchestra.remote:smithi123:/tmp/tmp.Y6CutcVFtu is 8KB 2020-01-16T00:04:56.591 INFO:teuthology.orchestra.run.smithi123:> rm -fr /tmp/tmp.Y6CutcVFtu 2020-01-16T00:04:56.597 INFO:teuthology.orchestra.run.smithi123:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2020-01-16T00:04:56.651 INFO:teuthology.orchestra.run.smithi123:> sudo sh -c 'cat > /etc/grub.d/01_ceph_kernel' && sudo chmod 755 /etc/grub.d/01_ceph_kernel 2020-01-16T00:04:56.749 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-74-generic 2020-01-16T00:04:56.749 INFO:teuthology.orchestra.run.smithi123:> sudo update-grub 2020-01-16T00:04:57.929 INFO:teuthology.orchestra.run.smithi123.stderr:Sourcing file `/etc/default/grub' 2020-01-16T00:04:57.946 INFO:teuthology.orchestra.run.smithi123.stderr:Generating grub configuration file ... 2020-01-16T00:04:58.170 INFO:teuthology.orchestra.run.smithi123.stderr:Found linux image: /boot/vmlinuz-4.15.0-74-generic 2020-01-16T00:04:58.184 INFO:teuthology.orchestra.run.smithi123.stderr:Found initrd image: /boot/initrd.img-4.15.0-74-generic 2020-01-16T00:04:58.481 INFO:teuthology.orchestra.run.smithi123.stderr:Found linux image: /boot/vmlinuz-4.15.0-72-generic 2020-01-16T00:04:58.495 INFO:teuthology.orchestra.run.smithi123.stderr:Found initrd image: /boot/initrd.img-4.15.0-72-generic 2020-01-16T00:04:59.696 INFO:teuthology.orchestra.run.smithi123.stderr:done 2020-01-16T00:04:59.698 INFO:teuthology.orchestra.run.smithi123:> sudo shutdown -r now 2020-01-16T00:04:59.700 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2020-01-16T00:04:59.700 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi145.front.sepia.ceph.com, path=None, version=distro) 2020-01-16T00:04:59.700 INFO:teuthology.orchestra.run.smithi145:> sudo apt-get clean 2020-01-16T00:04:59.767 INFO:teuthology.orchestra.run.smithi145:> sudo apt-get update 2020-01-16T00:05:00.030 INFO:teuthology.orchestra.run.smithi145.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-01-16T00:05:00.030 INFO:teuthology.orchestra.run.smithi145.stdout:Hit:2 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-01-16T00:05:00.099 INFO:teuthology.orchestra.run.smithi145.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-01-16T00:05:00.180 INFO:teuthology.orchestra.run.smithi145.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-01-16T00:05:01.985 INFO:teuthology.orchestra.run.smithi145.stdout:Reading package lists... 2020-01-16T00:05:02.032 INFO:teuthology.orchestra.run.smithi145:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-01-16T00:05:02.123 INFO:teuthology.orchestra.run.smithi145.stdout:Reading package lists... 2020-01-16T00:05:02.358 INFO:teuthology.orchestra.run.smithi145.stdout:Building dependency tree... 2020-01-16T00:05:02.358 INFO:teuthology.orchestra.run.smithi145.stdout:Reading state information... 2020-01-16T00:05:02.436 INFO:teuthology.orchestra.run.smithi145.stderr:E: Unable to locate package linux-image-current-generic 2020-01-16T00:05:02.438 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-01-16T00:05:02.439 INFO:teuthology.orchestra.run.smithi145:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-01-16T00:05:02.510 INFO:teuthology.orchestra.run.smithi145.stdout:Reading package lists... 2020-01-16T00:05:02.732 INFO:teuthology.orchestra.run.smithi145.stdout:Building dependency tree... 2020-01-16T00:05:02.732 INFO:teuthology.orchestra.run.smithi145.stdout:Reading state information... 2020-01-16T00:05:02.914 INFO:teuthology.orchestra.run.smithi145.stdout:linux-image-generic is already the newest version (4.15.0.74.76). 2020-01-16T00:05:02.915 INFO:teuthology.orchestra.run.smithi145.stdout:0 upgraded, 0 newly installed, 0 to remove and 28 not upgraded. 2020-01-16T00:05:02.918 INFO:teuthology.orchestra.run.smithi145:> dpkg -s linux-image-generic 2020-01-16T00:05:02.981 INFO:teuthology.orchestra.run.smithi145.stdout:Package: linux-image-generic 2020-01-16T00:05:02.982 INFO:teuthology.orchestra.run.smithi145.stdout:Status: install ok installed 2020-01-16T00:05:02.982 INFO:teuthology.orchestra.run.smithi145.stdout:Priority: optional 2020-01-16T00:05:02.982 INFO:teuthology.orchestra.run.smithi145.stdout:Section: kernel 2020-01-16T00:05:02.982 INFO:teuthology.orchestra.run.smithi145.stdout:Installed-Size: 15 2020-01-16T00:05:02.982 INFO:teuthology.orchestra.run.smithi145.stdout:Maintainer: Ubuntu Kernel Team 2020-01-16T00:05:02.982 INFO:teuthology.orchestra.run.smithi145.stdout:Architecture: amd64 2020-01-16T00:05:02.982 INFO:teuthology.orchestra.run.smithi145.stdout:Source: linux-meta 2020-01-16T00:05:02.983 INFO:teuthology.orchestra.run.smithi145.stdout:Version: 4.15.0.74.76 2020-01-16T00:05:02.983 INFO:teuthology.orchestra.run.smithi145.stdout:Depends: linux-image-4.15.0-74-generic, linux-modules-extra-4.15.0-74-generic, linux-firmware, intel-microcode, amd64-microcode 2020-01-16T00:05:02.983 INFO:teuthology.orchestra.run.smithi145.stdout:Recommends: thermald 2020-01-16T00:05:02.983 INFO:teuthology.orchestra.run.smithi145.stdout:Description: Generic Linux kernel image 2020-01-16T00:05:02.983 INFO:teuthology.orchestra.run.smithi145.stdout: This package will always depend on the latest generic kernel image 2020-01-16T00:05:02.983 INFO:teuthology.orchestra.run.smithi145.stdout: available. 2020-01-16T00:05:02.984 INFO:teuthology.orchestra.run.smithi145:> mktemp 2020-01-16T00:05:03.030 INFO:teuthology.orchestra.run.smithi145.stdout:/tmp/tmp.EpEfLdEGeM 2020-01-16T00:05:03.031 INFO:teuthology.orchestra.run.smithi145:> sudo cp /boot/grub/grub.cfg /tmp/tmp.EpEfLdEGeM 2020-01-16T00:05:03.086 INFO:teuthology.orchestra.run.smithi145:> sudo chmod 0666 /tmp/tmp.EpEfLdEGeM 2020-01-16T00:05:03.206 DEBUG:teuthology.orchestra.remote:smithi145:/tmp/tmp.EpEfLdEGeM is 8KB 2020-01-16T00:05:03.217 INFO:teuthology.orchestra.run.smithi145:> rm -fr /tmp/tmp.EpEfLdEGeM 2020-01-16T00:05:03.224 INFO:teuthology.orchestra.run.smithi145:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2020-01-16T00:05:03.278 INFO:teuthology.orchestra.run.smithi145:> sudo sh -c 'cat > /etc/grub.d/01_ceph_kernel' && sudo chmod 755 /etc/grub.d/01_ceph_kernel 2020-01-16T00:05:03.376 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-74-generic 2020-01-16T00:05:03.377 INFO:teuthology.orchestra.run.smithi145:> sudo update-grub 2020-01-16T00:05:04.579 INFO:teuthology.orchestra.run.smithi145.stderr:Sourcing file `/etc/default/grub' 2020-01-16T00:05:04.586 INFO:teuthology.orchestra.run.smithi145.stderr:Generating grub configuration file ... 2020-01-16T00:05:04.827 INFO:teuthology.orchestra.run.smithi145.stderr:Found linux image: /boot/vmlinuz-4.15.0-74-generic 2020-01-16T00:05:04.835 INFO:teuthology.orchestra.run.smithi145.stderr:Found initrd image: /boot/initrd.img-4.15.0-74-generic 2020-01-16T00:05:05.124 INFO:teuthology.orchestra.run.smithi145.stderr:Found linux image: /boot/vmlinuz-4.15.0-72-generic 2020-01-16T00:05:05.137 INFO:teuthology.orchestra.run.smithi145.stderr:Found initrd image: /boot/initrd.img-4.15.0-72-generic 2020-01-16T00:05:05.694 INFO:teuthology.orchestra.run.smithi145.stderr:done 2020-01-16T00:05:05.696 INFO:teuthology.orchestra.run.smithi145:> sudo shutdown -r now 2020-01-16T00:05:05.698 INFO:teuthology.misc:Re-opening connections... 2020-01-16T00:05:05.698 INFO:teuthology.misc:trying to connect to ubuntu@smithi123.front.sepia.ceph.com 2020-01-16T00:05:05.698 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-01-16T00:05:05.699 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi123.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:06:05.700 DEBUG:teuthology.orchestra.remote:timed out 2020-01-16T00:06:05.700 INFO:teuthology.misc:trying to connect to ubuntu@smithi145.front.sepia.ceph.com 2020-01-16T00:06:05.700 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-01-16T00:06:05.701 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi145.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:06:37.286 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.145 2020-01-16T00:06:37.286 DEBUG:teuthology.misc:waited 91.5881130695 2020-01-16T00:06:38.286 INFO:teuthology.misc:trying to connect to ubuntu@smithi123.front.sepia.ceph.com 2020-01-16T00:06:38.287 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-01-16T00:06:38.287 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi123.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:06:53.670 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.123 2020-01-16T00:06:53.670 INFO:teuthology.misc:trying to connect to ubuntu@smithi145.front.sepia.ceph.com 2020-01-16T00:06:53.670 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-01-16T00:06:53.671 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi145.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:06:53.764 INFO:teuthology.orchestra.run.smithi145:> true 2020-01-16T00:06:54.380 DEBUG:teuthology.misc:waited 108.682029009 2020-01-16T00:06:55.381 INFO:teuthology.misc:trying to connect to ubuntu@smithi123.front.sepia.ceph.com 2020-01-16T00:06:55.381 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-01-16T00:06:55.382 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi123.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:06:55.569 INFO:teuthology.orchestra.run.smithi123:> true 2020-01-16T00:06:56.121 DEBUG:teuthology.misc:waited 110.422656059 2020-01-16T00:06:57.122 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2020-01-16T00:06:57.122 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.15.0-74-generic"... 2020-01-16T00:06:57.122 INFO:teuthology.orchestra.run.smithi123:> uname -r 2020-01-16T00:06:57.128 INFO:teuthology.orchestra.run.smithi123.stdout:4.15.0-74-generic 2020-01-16T00:06:57.128 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-74-generic vs 4.15.0-74-generic 2020-01-16T00:06:57.129 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2020-01-16T00:06:57.129 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2020-01-16T00:06:57.129 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.15.0-74-generic"... 2020-01-16T00:06:57.129 INFO:teuthology.orchestra.run.smithi145:> uname -r 2020-01-16T00:06:57.135 INFO:teuthology.orchestra.run.smithi145.stdout:4.15.0-74-generic 2020-01-16T00:06:57.135 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-74-generic vs 4.15.0-74-generic 2020-01-16T00:06:57.135 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2020-01-16T00:06:58.136 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2020-01-16T00:06:58.137 INFO:teuthology.orchestra.run.smithi123:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-01-16T00:06:58.263 INFO:teuthology.orchestra.run.smithi123.stdout:ttyS1 2020-01-16T00:06:58.296 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2020-01-16T00:06:58.296 INFO:teuthology.orchestra.run.smithi145:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-01-16T00:06:58.331 INFO:teuthology.orchestra.run.smithi145.stdout:ttyS1 2020-01-16T00:06:58.360 INFO:teuthology.run_tasks:Running task internal.base... 2020-01-16T00:06:58.382 INFO:teuthology.task.internal:Creating test directory... 2020-01-16T00:06:58.382 INFO:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-01-16T00:06:58.384 INFO:teuthology.orchestra.run.smithi145:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-01-16T00:06:58.408 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2020-01-16T00:06:58.419 INFO:teuthology.run_tasks:Running task internal.archive... 2020-01-16T00:06:58.441 INFO:teuthology.task.internal:Creating archive directory... 2020-01-16T00:06:58.442 INFO:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-01-16T00:06:58.444 INFO:teuthology.orchestra.run.smithi145:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-01-16T00:06:58.459 INFO:teuthology.run_tasks:Running task internal.coredump... 2020-01-16T00:06:58.474 INFO:teuthology.task.internal:Enabling coredump saving... 2020-01-16T00:06:58.475 INFO:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-01-16T00:06:58.503 INFO:teuthology.orchestra.run.smithi145:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-01-16T00:06:58.522 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-01-16T00:06:58.524 INFO:teuthology.orchestra.run.smithi145.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-01-16T00:06:58.529 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-01-16T00:06:58.530 INFO:teuthology.orchestra.run.smithi145.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-01-16T00:06:58.532 INFO:teuthology.run_tasks:Running task internal.sudo... 2020-01-16T00:06:58.543 INFO:teuthology.task.internal:Configuring sudo... 2020-01-16T00:06:58.544 INFO:teuthology.orchestra.run.smithi123:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-01-16T00:06:58.575 INFO:teuthology.orchestra.run.smithi145:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-01-16T00:06:58.589 INFO:teuthology.run_tasks:Running task internal.syslog... 2020-01-16T00:06:58.601 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2020-01-16T00:06:58.601 INFO:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-01-16T00:06:58.631 INFO:teuthology.orchestra.run.smithi145:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-01-16T00:06:58.640 INFO:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-01-16T00:06:58.684 INFO:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-01-16T00:06:58.731 INFO:teuthology.orchestra.run.smithi123:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2020-01-16T00:06:58.820 INFO:teuthology.orchestra.run.smithi145:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-01-16T00:06:58.826 INFO:teuthology.orchestra.run.smithi145:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-01-16T00:06:58.876 INFO:teuthology.orchestra.run.smithi145:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2020-01-16T00:06:58.965 INFO:teuthology.orchestra.run.smithi123:> sudo service rsyslog restart 2020-01-16T00:06:58.967 INFO:teuthology.orchestra.run.smithi145:> sudo service rsyslog restart 2020-01-16T00:06:59.048 INFO:teuthology.run_tasks:Running task internal.timer... 2020-01-16T00:06:59.059 INFO:teuthology.task.internal:Starting timer... 2020-01-16T00:06:59.059 INFO:teuthology.run_tasks:Running task pcp... 2020-01-16T00:06:59.091 INFO:teuthology.run_tasks:Running task selinux... 2020-01-16T00:06:59.103 DEBUG:teuthology.task.selinux:Excluding smithi123: OS 'ubuntu' does not support SELinux 2020-01-16T00:06:59.103 DEBUG:teuthology.task.selinux:Excluding smithi145: OS 'ubuntu' does not support SELinux 2020-01-16T00:06:59.103 DEBUG:teuthology.task.selinux:Getting current SELinux state 2020-01-16T00:06:59.103 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2020-01-16T00:06:59.104 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2020-01-16T00:06:59.104 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2020-01-16T00:06:59.115 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated; assuming it is current 2020-01-16T00:06:59.116 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2020-01-16T00:06:59.131 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2020-01-16T00:06:59.134 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi123.front.sepia.ceph.com,smithi145.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2020-01-16T00:06:59.477 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2020-01-16T00:07:00.629 INFO:teuthology.task.ansible.out: [WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2020-01-16T00:07:03.289 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2020-01-16T00:07:03.314 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:03 +0000 (0:00:00.066) 0:00:00.066 ****** 2020-01-16T00:07:03.330 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:03 +0000 (0:00:00.015) 0:00:00.081 ****** 2020-01-16T00:07:06.004 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi145.front.sepia.ceph.com] 2020-01-16T00:07:06.100 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:06 +0000 (0:00:02.770) 0:00:02.852 ****** 2020-01-16T00:07:06.193 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] 2020-01-16T00:07:06.287 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:06 +0000 (0:00:00.186) 0:00:03.038 ****** 2020-01-16T00:07:06.384 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2020-01-16T00:07:06.386 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-01-16T00:07:06.425 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:06 +0000 (0:00:00.137) 0:00:03.176 ****** 2020-01-16T00:07:06.430 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-01-16T00:07:06.471 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:06 +0000 (0:00:00.045) 0:00:03.222 ****** 2020-01-16T00:07:06.883 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2020-01-16T00:07:06.883 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-01-16T00:07:06.926 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:06 +0000 (0:00:00.455) 0:00:03.678 ****** 2020-01-16T00:07:06.934 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-01-16T00:07:06.984 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:06 +0000 (0:00:00.057) 0:00:03.736 ****** 2020-01-16T00:07:07.269 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2020-01-16T00:07:07.270 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.077628", "end": "2020-01-16 00:07:07.531312", "rc": 0, "start": "2020-01-16 00:07:07.453684", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.073427", "end": "2020-01-16 00:07:07.365191", "rc": 0, "start": "2020-01-16 00:07:07.291764", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2020-01-16T00:07:07.309 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:07 +0000 (0:00:00.325) 0:00:04.061 ****** 2020-01-16T00:07:07.342 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:07 +0000 (0:00:00.033) 0:00:04.094 ****** 2020-01-16T00:07:07.554 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2020-01-16T00:07:07.556 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-01-16T00:07:07.596 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:07 +0000 (0:00:00.253) 0:00:04.347 ****** 2020-01-16T00:07:07.602 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-01-16T00:07:07.640 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:07 +0000 (0:00:00.044) 0:00:04.392 ****** 2020-01-16T00:07:08.371 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2020-01-16T00:07:08.372 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1579133227.61-118155456466220/source", "state": "file", "uid": 0} 2020-01-16T00:07:08.412 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:08 +0000 (0:00:00.771) 0:00:05.164 ****** 2020-01-16T00:07:08.418 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1579133227.66-1904684997391/source", "state": "file", "uid": 0} 2020-01-16T00:07:08.467 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:08 +0000 (0:00:00.054) 0:00:05.218 ****** 2020-01-16T00:07:08.732 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2020-01-16T00:07:08.733 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-01-16T00:07:08.755 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-01-16T00:07:08.875 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-01-16T00:07:08.898 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-01-16T00:07:08.996 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-01-16T00:07:09.018 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-01-16T00:07:09.141 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-01-16T00:07:09.165 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-01-16T00:07:09.283 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-01-16T00:07:09.295 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-01-16T00:07:09.404 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-01-16T00:07:09.446 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-01-16T00:07:09.530 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-01-16T00:07:09.592 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-01-16T00:07:09.723 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** 2020-01-16T00:07:09.724 INFO:teuthology.task.ansible.out:skipping: no hosts matched 2020-01-16T00:07:09.729 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2020-01-16T00:07:09.915 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:09 +0000 (0:00:01.448) 0:00:06.667 ****** 2020-01-16T00:07:09.949 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:09 +0000 (0:00:00.033) 0:00:06.700 ****** 2020-01-16T00:07:10.632 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2020-01-16T00:07:10.634 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] 2020-01-16T00:07:10.646 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] 2020-01-16T00:07:10.768 INFO:teuthology.task.ansible.out: [WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2020-01-16T00:07:10.769 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:10 +0000 (0:00:00.818) 0:00:07.519 ****** 2020-01-16T00:07:10.809 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:10 +0000 (0:00:00.040) 0:00:07.560 ****** 2020-01-16T00:07:11.134 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2020-01-16T00:07:11.135 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.233326", "end": "2020-01-16 00:07:11.112337", "rc": 0, "start": "2020-01-16 00:07:10.879011", "stderr": "", "stderr_lines": [], "stdout": "19360304df051f6c2a8374fc9e8fc34ccf5740c9", "stdout_lines": ["19360304df051f6c2a8374fc9e8fc34ccf5740c9"]} 2020-01-16T00:07:11.199 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:11 +0000 (0:00:00.390) 0:00:07.950 ****** 2020-01-16T00:07:11.206 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.209766", "end": "2020-01-16 00:07:11.130408", "rc": 0, "start": "2020-01-16 00:07:10.920642", "stderr": "", "stderr_lines": [], "stdout": "19360304df051f6c2a8374fc9e8fc34ccf5740c9", "stdout_lines": ["19360304df051f6c2a8374fc9e8fc34ccf5740c9"]} 2020-01-16T00:07:11.255 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:11 +0000 (0:00:00.056) 0:00:08.007 ****** 2020-01-16T00:07:11.313 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2020-01-16T00:07:11.314 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.004294", "end": "2020-01-16 00:07:11.571248", "failed_when_result": false, "rc": 0, "start": "2020-01-16 00:07:11.566954", "stderr": "", "stderr_lines": [], "stdout": "c84d080fc97e482f88d240859322d5765ac90366", "stdout_lines": ["c84d080fc97e482f88d240859322d5765ac90366"]} 2020-01-16T00:07:11.355 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:11 +0000 (0:00:00.099) 0:00:08.106 ****** 2020-01-16T00:07:11.383 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-01-16T00:07:11.384 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:11.426 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:11 +0000 (0:00:00.070) 0:00:08.177 ****** 2020-01-16T00:07:11.431 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2020-01-16T00:07:11.432 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.011257", "end": "2020-01-16 00:07:11.499987", "failed_when_result": false, "rc": 0, "start": "2020-01-16 00:07:11.488730", "stderr": "", "stderr_lines": [], "stdout": "c84d080fc97e482f88d240859322d5765ac90366", "stdout_lines": ["c84d080fc97e482f88d240859322d5765ac90366"]} 2020-01-16T00:07:11.486 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:11 +0000 (0:00:00.060) 0:00:08.238 ****** 2020-01-16T00:07:11.493 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-01-16T00:07:11.494 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:11.546 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:11 +0000 (0:00:00.060) 0:00:08.298 ****** 2020-01-16T00:07:11.551 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-01-16T00:07:11.553 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:11.596 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:11 +0000 (0:00:00.049) 0:00:08.347 ****** 2020-01-16T00:07:11.601 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2020-01-16T00:07:11.602 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:11.645 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:11 +0000 (0:00:00.049) 0:00:08.396 ****** 2020-01-16T00:07:11.650 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-01-16T00:07:11.651 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:11.692 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:11 +0000 (0:00:00.047) 0:00:08.443 ****** 2020-01-16T00:07:11.697 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-01-16T00:07:11.698 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:11.746 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:11 +0000 (0:00:00.053) 0:00:08.497 ****** 2020-01-16T00:07:11.751 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2020-01-16T00:07:11.752 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:11.802 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:11 +0000 (0:00:00.056) 0:00:08.553 ****** 2020-01-16T00:07:11.808 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2020-01-16T00:07:11.808 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:11.848 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:11 +0000 (0:00:00.046) 0:00:08.600 ****** 2020-01-16T00:07:11.853 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-01-16T00:07:11.855 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:11.894 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:11 +0000 (0:00:00.045) 0:00:08.645 ****** 2020-01-16T00:07:11.919 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:11.960 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:11 +0000 (0:00:00.066) 0:00:08.711 ****** 2020-01-16T00:07:11.976 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-01-16T00:07:11.977 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-01-16T00:07:12.023 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:12 +0000 (0:00:00.063) 0:00:08.775 ****** 2020-01-16T00:07:12.107 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-01-16T00:07:12.149 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:12 +0000 (0:00:00.125) 0:00:08.900 ****** 2020-01-16T00:07:12.197 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-01-16T00:07:12.198 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2020-01-16T00:07:12.329 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2020-01-16T00:07:12.341 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2020-01-16T00:07:12.472 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2020-01-16T00:07:12.489 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2020-01-16T00:07:12.604 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2020-01-16T00:07:12.637 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1105} 2020-01-16T00:07:12.763 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1105} 2020-01-16T00:07:12.775 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1106} 2020-01-16T00:07:12.918 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1106} 2020-01-16T00:07:12.930 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1107} 2020-01-16T00:07:13.070 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1108} 2020-01-16T00:07:13.072 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1107} 2020-01-16T00:07:13.224 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2020-01-16T00:07:13.226 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1108} 2020-01-16T00:07:13.370 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2020-01-16T00:07:13.372 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2020-01-16T00:07:13.519 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2020-01-16T00:07:13.520 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2020-01-16T00:07:13.671 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2020-01-16T00:07:13.672 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2020-01-16T00:07:13.819 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2020-01-16T00:07:13.820 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2020-01-16T00:07:13.970 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "ite 2020-01-16T00:07:13.974 INFO:teuthology.task.ansible.out:m", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2020-01-16T00:07:14.109 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2020-01-16T00:07:14.116 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2020-01-16T00:07:14.253 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2020-01-16T00:07:14.254 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2020-01-16T00:07:14.394 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcw 2020-01-16T00:07:14.395 INFO:teuthology.task.ansible.out:JjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2020-01-16T00:07:14.540 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2020-01-16T00:07:14.541 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2020-01-16T00:07:14.689 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/gmeno", "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "move_home": false, "name": "gmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2020-01-16T00:07:14.690 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2020-01-16T00:07:14.839 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/gmeno", "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "move_home": false, "name": "gmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2020-01-16T00:07:14.840 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2020-01-16T00:07:14.960 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2020-01-16T00:07:14.982 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2020-01-16T00:07:15.107 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2020-01-16T00:07:15.132 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1122} 2020-01-16T00:07:15.247 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1122} 2020-01-16T00:07:15.283 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2020-01-16T00:07:15.392 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2020-01-16T00:07:15.435 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2020-01-16T00:07:15.532 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2020-01-16T00:07:15.587 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2020-01-16T00:07:15.653 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2020-01-16T00:07:15.742 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2020-01-16T00:07:15.798 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2020-01-16T00:07:15.863 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2020-01-16T00:07:15.947 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2020-01-16T00:07:16.013 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2020-01-16T00:07:16.083 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2020-01-16T00:07:16.159 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2020-01-16T00:07:16.234 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2020-01-16T00:07:16.300 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2020-01-16T00:07:16.384 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2020-01-16T00:07:16.457 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2020-01-16T00:07:16.530 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2020-01-16T00:07:16.614 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2020-01-16T00:07:16.678 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2020-01-16T00:07:16.770 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2020-01-16T00:07:16.823 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2020-01-16T00:07:16.907 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2020-01-16T00:07:16.966 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2020-01-16T00:07:17.064 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2020-01-16T00:07:17.119 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2020-01-16T00:07:17.223 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2020-01-16T00:07:17.256 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2020-01-16T00:07:17.378 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2020-01-16T00:07:17.397 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2020-01-16T00:07:17.527 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2020-01-16T00:07:17.528 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2020-01-16T00:07:17.656 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2020-01-16T00:07:17.658 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2020-01-16T00:07:17.791 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-01-16T00:07:17.813 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-01-16T00:07:17.934 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2020-01-16T00:07:17.936 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2020-01-16T00:07:18.078 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2020-01-16T00:07:18.079 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2020-01-16T00:07:18.226 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2020-01-16T00:07:18.238 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2020-01-16T00:07:18.386 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2020-01-16T00:07:18.398 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2020-01-16T00:07:18.530 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2020-01-16T00:07:18.562 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2020-01-16T00:07:18.668 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2020-01-16T00:07:18.711 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2020-01-16T00:07:18.816 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2020-01-16T00:07:18.869 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2020-01-16T00:07:18.954 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-01-16T00:07:18.997 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-01-16T00:07:19.083 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2020-01-16T00:07:19.147 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2020-01-16T00:07:19.231 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2020-01-16T00:07:19.286 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2020-01-16T00:07:19.387 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2020-01-16T00:07:19.410 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2020-01-16T00:07:19.550 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2020-01-16T00:07:19.581 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2020-01-16T00:07:19.707 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2020-01-16T00:07:19.708 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2020-01-16T00:07:19.838 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2020-01-16T00:07:19.870 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2020-01-16T00:07:19.964 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2020-01-16T00:07:19.998 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2020-01-16T00:07:20.105 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2020-01-16T00:07:20.129 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2020-01-16T00:07:20.264 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2020-01-16T00:07:20.276 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2020-01-16T00:07:20.411 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2020-01-16T00:07:20.433 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2020-01-16T00:07:20.550 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2020-01-16T00:07:20.583 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2020-01-16T00:07:20.698 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2020-01-16T00:07:20.731 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2020-01-16T00:07:20.828 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2020-01-16T00:07:20.881 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2020-01-16T00:07:20.979 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2020-01-16T00:07:21.011 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2020-01-16T00:07:21.143 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2020-01-16T00:07:21.166 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2020-01-16T00:07:21.304 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2020-01-16T00:07:21.326 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2020-01-16T00:07:21.437 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-01-16T00:07:21.471 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-01-16T00:07:21.579 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2020-01-16T00:07:21.627 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2020-01-16T00:07:21.721 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2020-01-16T00:07:21.785 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2020-01-16T00:07:21.879 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2020-01-16T00:07:21.953 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2020-01-16T00:07:22.019 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2020-01-16T00:07:22.082 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2020-01-16T00:07:22.178 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2020-01-16T00:07:22.221 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2020-01-16T00:07:22.311 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2020-01-16T00:07:22.371 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2020-01-16T00:07:22.453 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2020-01-16T00:07:22.520 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2020-01-16T00:07:22.605 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2020-01-16T00:07:22.667 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2020-01-16T00:07:22.773 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2020-01-16T00:07:22.813 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2020-01-16T00:07:22.898 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2020-01-16T00:07:22.953 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2020-01-16T00:07:23.035 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2020-01-16T00:07:23.100 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2020-01-16T00:07:23.190 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2020-01-16T00:07:23.234 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2020-01-16T00:07:23.352 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2020-01-16T00:07:23.499 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2020-01-16T00:07:23.501 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2020-01-16T00:07:23.634 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2020-01-16T00:07:23.656 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2020-01-16T00:07:23.780 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2020-01-16T00:07:23.814 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2020-01-16T00:07:23.930 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2020-01-16T00:07:23.963 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2020-01-16T00:07:24.094 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2020-01-16T00:07:24.111 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2020-01-16T00:07:24.216 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2020-01-16T00:07:24.248 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2020-01-16T00:07:24.374 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2020-01-16T00:07:24.385 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2020-01-16T00:07:24.509 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2020-01-16T00:07:24.532 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2020-01-16T00:07:24.648 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2020-01-16T00:07:24.691 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2020-01-16T00:07:24.807 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2020-01-16T00:07:24.844 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2020-01-16T00:07:24.938 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2020-01-16T00:07:24.971 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2020-01-16T00:07:25.067 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2020-01-16T00:07:25.121 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2020-01-16T00:07:25.205 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2020-01-16T00:07:25.290 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2020-01-16T00:07:25.354 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2020-01-16T00:07:25.427 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2020-01-16T00:07:25.507 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2020-01-16T00:07:25.600 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2020-01-16T00:07:25.641 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2020-01-16T00:07:25.738 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2020-01-16T00:07:25.772 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2020-01-16T00:07:25.884 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2020-01-16T00:07:25.906 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2020-01-16T00:07:26.021 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2020-01-16T00:07:26.053 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2020-01-16T00:07:26.169 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2020-01-16T00:07:26.222 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2020-01-16T00:07:26.316 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2020-01-16T00:07:26.359 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2020-01-16T00:07:26.459 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2020-01-16T00:07:26.501 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2020-01-16T00:07:26.608 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2020-01-16T00:07:26.640 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2020-01-16T00:07:26.745 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2020-01-16T00:07:26.798 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2020-01-16T00:07:26.898 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2020-01-16T00:07:26.941 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-01-16T00:07:27.047 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-01-16T00:07:27.097 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2020-01-16T00:07:27.280 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2020-01-16T00:07:27.333 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2020-01-16T00:07:27.428 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2020-01-16T00:07:27.496 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2020-01-16T00:07:27.580 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2020-01-16T00:07:28.021 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2020-01-16T00:07:28.226 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2020-01-16T00:07:28.729 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 RYbu6kdXmsBtrtAkph0lIg b4a266ab604d320ab175070a427f4ce486f65cde25c5ac248e8316e3fd2f2435', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 RYbu6kdXmsBtrtAkph0lIg b4a266ab604d320ab175070a427f4ce486f65cde25c5ac248e8316e3fd2f2435"}, "name": "xiubli", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2020-01-16T00:07:28.734 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 RYbu6kdXmsBtrtAkph0lIg b4a266ab604d320ab175070a427f4ce486f65cde25c5ac248e8316e3fd2f2435', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 RYbu6kdXmsBtrtAkph0lIg b4a266ab604d320ab175070a427f4ce486f65cde25c5ac248e8316e3fd2f2435"}, "name": "xiubli", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2020-01-16T00:07:29.260 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "name": "jafaj", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2020-01-16T00:07:29.266 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "name": "jafaj", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2020-01-16T00:07:29.467 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:29 +0000 (0:00:17.318) 0:00:26.219 ****** 2020-01-16T00:07:29.502 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:29 +0000 (0:00:00.034) 0:00:26.253 ****** 2020-01-16T00:07:29.506 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-01-16T00:07:29.550 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:29 +0000 (0:00:00.048) 0:00:26.301 ****** 2020-01-16T00:07:29.602 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:29 +0000 (0:00:00.052) 0:00:26.354 ****** 2020-01-16T00:07:29.608 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-01-16T00:07:29.613 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ss 2020-01-16T00:07:29.614 INFO:teuthology.task.ansible.out:h-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN 2020-01-16T00:07:29.616 INFO:teuthology.task.ansible.out:2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499ea 2020-01-16T00:07:29.617 INFO:teuthology.task.ansible.out:f767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d 2020-01-16T00:07:29.618 INFO:teuthology.task.ansible.out:85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d 2020-01-16T00:07:29.620 INFO:teuthology.task.ansible.out:86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868" 2020-01-16T00:07:29.621 INFO:teuthology.task.ansible.out:}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad 2020-01-16T00:07:29.623 INFO:teuthology.task.ansible.out:7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpav 2020-01-16T00:07:29.623 INFO:teuthology.task.ansible.out:ani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280 2020-01-16T00:07:29.624 INFO:teuthology.task.ansible.out:b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkr 2020-01-16T00:07:29.626 INFO:teuthology.task.ansible.out:Q 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, {"name": "toabctl", "ovpn": 2020-01-16T00:07:29.627 INFO:teuthology.task.ansible.out: "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "xiubli@fedora1 RYbu6kdXmsBtrtAkph0lIg b4a266ab604d320ab175070a427f4ce486f65cde25c5ac248e8316e3fd2f2435"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}]}, "changed": false} 2020-01-16T00:07:29.676 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:29 +0000 (0:00:00.069) 0:00:26.423 ****** 2020-01-16T00:07:29.683 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ss 2020-01-16T00:07:29.685 INFO:teuthology.task.ansible.out:h-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN 2020-01-16T00:07:29.686 INFO:teuthology.task.ansible.out:2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499ea 2020-01-16T00:07:29.688 INFO:teuthology.task.ansible.out:f767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d 2020-01-16T00:07:29.689 INFO:teuthology.task.ansible.out:85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d 2020-01-16T00:07:29.691 INFO:teuthology.task.ansible.out:86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868" 2020-01-16T00:07:29.692 INFO:teuthology.task.ansible.out:}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d 2020-01-16T00:07:29.692 INFO:teuthology.task.ansible.out:49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad 2020-01-16T00:07:29.694 INFO:teuthology.task.ansible.out:7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280 2020-01-16T00:07:29.695 INFO:teuthology.task.ansible.out:b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkr 2020-01-16T00:07:29.697 INFO:teuthology.task.ansible.out:Q 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, {"name": "toabctl", "ovpn": 2020-01-16T00:07:29.698 INFO:teuthology.task.ansible.out: "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "xiubli@fedora1 RYbu6kdXmsBtrtAkph0lIg b4a266ab604d320ab175070a427f4ce486f65cde25c5ac248e8316e3fd2f2435"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}]}, "changed": false} 2020-01-16T00:07:29.737 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:29 +0000 (0:00:00.064) 0:00:26.488 ****** 2020-01-16T00:07:30.575 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2020-01-16T00:07:30.577 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com -> localhost] => {"after": "19360304df051f6c2a8374fc9e8fc34ccf5740c9", "attempts": 1, "before": "19360304df051f6c2a8374fc9e8fc34ccf5740c9", "changed": false, "remote_url_changed": false} 2020-01-16T00:07:30.618 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:30 +0000 (0:00:00.880) 0:00:27.369 ****** 2020-01-16T00:07:30.625 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com -> localhost] => {"after": "19360304df051f6c2a8374fc9e8fc34ccf5740c9", "attempts": 1, "before": "19360304df051f6c2a8374fc9e8fc34ccf5740c9", "changed": false, "remote_url_changed": false} 2020-01-16T00:07:30.668 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:30 +0000 (0:00:00.049) 0:00:27.419 ****** 2020-01-16T00:07:30.787 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-01-16T00:07:30.789 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2020-01-16T00:07:30.861 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2020-01-16T00:07:30.948 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2020-01-16T00:07:31.031 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2020-01-16T00:07:31.106 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2020-01-16T00:07:31.181 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2020-01-16T00:07:31.235 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZtIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "va 2020-01-16T00:07:31.236 INFO:teuthology.task.ansible.out:lidate_certs": true} 2020-01-16T00:07:31.308 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZtIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "va 2020-01-16T00:07:31.310 INFO:teuthology.task.ansible.out:lidate_certs": true} 2020-01-16T00:07:31.382 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2020-01-16T00:07:31.467 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2020-01-16T00:07:31.510 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZtt 2020-01-16T00:07:31.511 INFO:teuthology.task.ansible.out:Edz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2020-01-16T00:07:31.625 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZtt 2020-01-16T00:07:31.627 INFO:teuthology.task.ansible.out:Edz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2020-01-16T00:07:31.668 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2020-01-16T00:07:31.770 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2020-01-16T00:07:31.834 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2020-01-16T00:07:31.845 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:31.905 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2020-01-16T00:07:31.916 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:32.000 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uT 2020-01-16T00:07:32.002 INFO:teuthology.task.ansible.out:fMorbnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2020-01-16T00:07:32.012 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:32.030 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:32.040 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:32.082 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uT 2020-01-16T00:07:32.084 INFO:teuthology.task.ansible.out:fMorbnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:32.095 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:32.116 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:32.196 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2020-01-16T00:07:32.251 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2020-01-16T00:07:32.318 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+ 2020-01-16T00:07:32.320 INFO:teuthology.task.ansible.out:2vtSoAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2020-01-16T00:07:32.372 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+ 2020-01-16T00:07:32.373 INFO:teuthology.task.ansible.out:2vtSoAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2020-01-16T00:07:32.458 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2020-01-16T00:07:32.543 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2020-01-16T00:07:32.585 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2020-01-16T00:07:32.682 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2020-01-16T00:07:32.735 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvWnKfzTd574C4GXg1mr4pbJOInKv4qMRf0DwksTzqtXStlhT+PaO2C8p48U2Xjz5hmPK9J+iHA2xKwI6lKI194wUo3tW283euRyTCN7LlbV6tgD9k78hDaeA+53MHYmwbxjHEqAadqrLU+68LfmL7gaaUSToP/Ap+hdcDC7H9lulMcxkUeahEiDfG2s1mD8IUM6XVuqZM9WEARKpF1oSEBKhMMob9YJZPpJWipuAGkPvDA0yHK1lB58Yr2x0yaezVjQc7V2dwaY1gXR6AscU5WzlwWNNwXHqPdyyWRaGx/9Fk00yfzX4V7ZqTvKdMLKwB7EnamduTMPsaoKBUBJCh gmeno@gmeno-virtual-machine", "key_options": null, "keyfile": "/home/gmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gmeno", "validate_certs": true} 2020-01-16T00:07:32.834 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvWnKfzTd574C4GXg1mr4pbJOInKv4qMRf0DwksTzqtXStlhT+PaO2C8p48U2Xjz5hmPK9J+iHA2xKwI6lKI194wUo3tW283euRyTCN7LlbV6tgD9k78hDaeA+53MHYmwbxjHEqAadqrLU+68LfmL7gaaUSToP/Ap+hdcDC7H9lulMcxkUeahEiDfG2s1mD8IUM6XVuqZM9WEARKpF1oSEBKhMMob9YJZPpJWipuAGkPvDA0yHK1lB58Yr2x0yaezVjQc7V2dwaY1gXR6AscU5WzlwWNNwXHqPdyyWRaGx/9Fk00yfzX4V7ZqTvKdMLKwB7EnamduTMPsaoKBUBJCh gmeno@gmeno-virtual-machine", "key_options": null, "keyfile": "/home/gmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gmeno", "validate_certs": true} 2020-01-16T00:07:32.899 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2020-01-16T00:07:32.969 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2020-01-16T00:07:33.085 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2020-01-16T00:07:33.108 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2020-01-16T00:07:33.244 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2020-01-16T00:07:33.246 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2020-01-16T00:07:33.394 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1 2020-01-16T00:07:33.395 INFO:teuthology.task.ansible.out:Z9CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467 2020-01-16T00:07:33.397 INFO:teuthology.task.ansible.out:\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2020-01-16T00:07:33.522 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkR 2020-01-16T00:07:33.524 INFO:teuthology.task.ansible.out:hoBYB0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2020-01-16T00:07:33.554 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkR 2020-01-16T00:07:33.555 INFO:teuthology.task.ansible.out:hoBYB0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2020-01-16T00:07:33.712 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2020-01-16T00:07:33.714 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2020-01-16T00:07:33.855 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2020-01-16T00:07:33.857 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2020-01-16T00:07:33.982 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwm 2020-01-16T00:07:33.983 INFO:teuthology.task.ansible.out:ih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2020-01-16T00:07:33.984 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwm 2020-01-16T00:07:33.986 INFO:teuthology.task.ansible.out:ih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2020-01-16T00:07:34.117 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2020-01-16T00:07:34.128 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2020-01-16T00:07:34.252 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2020-01-16T00:07:34.277 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2020-01-16T00:07:34.376 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2020-01-16T00:07:34.433 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2020-01-16T00:07:34.527 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2020-01-16T00:07:34.580 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2020-01-16T00:07:34.663 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2020-01-16T00:07:34.740 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2020-01-16T00:07:34.812 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2 2020-01-16T00:07:34.813 INFO:teuthology.task.ansible.out:CfQvUxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2020-01-16T00:07:34.907 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2 2020-01-16T00:07:34.909 INFO:teuthology.task.ansible.out:CfQvUxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2020-01-16T00:07:34.970 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2020-01-16T00:07:35.057 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2020-01-16T00:07:35.090 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2020-01-16T00:07:35.206 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2020-01-16T00:07:35.228 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2020-01-16T00:07:35.343 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2020-01-16T00:07:35.388 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2020-01-16T00:07:35.492 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2020-01-16T00:07:35.524 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2020-01-16T00:07:35.659 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2020-01-16T00:07:35.670 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2020-01-16T00:07:35.672 INFO:teuthology.task.ansible.out: 2020-01-16T00:07:35.829 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYG 2020-01-16T00:07:35.831 INFO:teuthology.task.ansible.out:T9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-01-16T00:07:35.965 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2020-01-16T00:07:35.997 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-01-16T00:07:36.124 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH/YT1mGZg50qDdngGQ4Ea+jfJGZ582J6pQbYkrvuGsyATMzNwk73sAbn7mgbpw7nkW/ol/vwahXHZ5heC056BFnqiZ7tjoDeotDx/1XxFtkbyIaVflUeD1vBww38v6IslLHcVhPilauPVZSfVzEHI7JB0a+9OwNrEH3+dt1B6vR+WRy2gHG/aEJffPMv4yv/0FgRWy3cEiUvGlLq0KVVTbsqqLQhw/u5k8bBIoibBtZLFAMTrjLYp9Dv8dxJROvmey6O2TuX6gCY1rLGTAArYULpXH73OLIqaksJKc3srgoJrTSZPO6bo3BQeLz4ODJ7uvTpvQIhacBpp/dq1ln7p owasserm@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCiqChxNdielMPBchs0u6bxWstqU9oeY2K9qf1PNwEIrwDYTrpDjsdoVnS750tOBjWY0wr9Gf9ATOJXi89151oao9GaUueQ1rTgmu7GtPlKsuBUVNMFzLEa0mqneqDA4AfknisFt/LnL8fLM4BddT25tmQ4oUGS7F17p4kXqwboK/DsJciCMWWc/5AT4NVwdnYFzIA6ta25/bB94kpktaZ6HDw7G8q/mK62Yza3EZfq8SP7p0sPL7oVsInQ9AutcVonmALgzPa9t1/MPGaXUIqARaWtwe4MSlNEIs4afLFyteBVeq+dgU01yeyjFivwVmuS+aPFTm6sjGGe8/gOsFDIJiVertOqi663m7q1WYGlVANfif7uL2+uJ9yrvOmrFh5s/CJMJBRG1SLfK1j/ivFwvPAQc/uMqEIRzrVZOsuNLPO4zHPWzmBAPUteb+2BqSdUFYMlBxb3lmhFFffroDqnOK1nbdAfmyKiosUizQgX2Gs2wjjtejuhS9dpYkNu+sBOujsxa2mZUouyWf4fXyWGVjthrCZaM+FyeSDpJrgwgRiSKNB8aRxB10ZNo9WZEy6eH2adp0Xf6v2KsKvryDndqBEkBI8zLbGrPVHGSEqiSBWc+NeAB06uSscJd0pHU3hDg0qW0EaxOXaial9qrBiwcFVKEr0g7LDURQ6KZsZy9w== owasserm@owasserm.redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2020-01-16T00:07:36.146 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2020-01-16T00:07:36.265 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6 2020-01-16T00:07:36.267 INFO:teuthology.task.ansible.out:AKY1OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2020-01-16T00:07:36.291 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH/YT1mGZg50qDdngGQ4Ea+jfJGZ582J6pQbYkrvuGsyATMzNwk73sAbn7mgbpw7nkW/ol/vwahXHZ5heC056BFnqiZ7tjoDeotDx/1XxFtkbyIaVflUeD1vBww38v6IslLHcVhPilauPVZSfVzEHI7JB0a+9OwNrEH3+dt1B6vR+WRy2gHG/aEJffPMv4yv/0FgRWy3cEiUvGlLq0KVVTbsqqLQhw/u5k8bBIoibBtZLFAMTrjLYp9Dv8dxJROvmey6O2TuX6gCY1rLGTAArYULpXH73OLIqaksJKc3srgoJrTSZPO6bo3BQeLz4ODJ7uvTpvQIhacBpp/dq1ln7p owasserm@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCiqChxNdielMPBchs0u6bxWstqU9oeY2K9qf1PNwEIrwDYTrpDjsdoVnS750tOBjWY0wr9Gf9ATOJXi89151oao9GaUueQ1rTgmu7GtPlKsuBUVNMFzLEa0mqneqDA4AfknisFt/LnL8fLM4BddT25tmQ4oUGS7F17p4kXqwboK/DsJciCMWWc/5AT4NVwdnYFzIA6ta25/bB94kpktaZ6HDw7G8q/mK62Yza3EZfq8SP7p0sPL7oVsInQ9AutcVonmALgzPa9t1/MPGaXUIqARaWtwe4MSlNEIs4afLFyteBVeq+dgU01yeyjFivwVmuS+aPFTm6sjGGe8/gOsFDIJiVertOqi663m7q1WYGlVANfif7uL2+uJ9yrvOmrFh5s/CJMJBRG1SLfK1j/ivFwvPAQc/uMqEIRzrVZOsuNLPO4zHPWzmBAPUteb+2BqSdUFYMlBxb3lmhFFffroDqnOK1nbdAfmyKiosUizQgX2Gs2wjjtejuhS9dpYkNu+sBOujsxa2mZUouyWf4fXyWGVjthrCZaM+FyeSDpJrgwgRiSKNB8aRxB10ZNo9WZEy6eH2adp0Xf6v2KsKvryDndqBEkBI8zLbGrPVHGSEqiSBWc+NeAB06uSscJd0pHU3hDg0qW0EaxOXaial9qrBiwcFVKEr0g7LDURQ6KZsZy9w== owasserm@owasserm.redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2020-01-16T00:07:36.418 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2Ruq 2020-01-16T00:07:36.420 INFO:teuthology.task.ansible.out:iZyKz2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2020-01-16T00:07:36.483 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6 2020-01-16T00:07:36.484 INFO:teuthology.task.ansible.out:AKY1OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2020-01-16T00:07:36.575 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2020-01-16T00:07:36.620 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2Ruq 2020-01-16T00:07:36.621 INFO:teuthology.task.ansible.out:iZyKz2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2020-01-16T00:07:36.714 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2020-01-16T00:07:36.767 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2020-01-16T00:07:36.872 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2020-01-16T00:07:36.916 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2020-01-16T00:07:37.011 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options" 2020-01-16T00:07:37.013 INFO:teuthology.task.ansible.out:: null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-01-16T00:07:37.070 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2020-01-16T00:07:37.198 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options" 2020-01-16T00:07:37.200 INFO:teuthology.task.ansible.out:: null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-01-16T00:07:37.315 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+z 2020-01-16T00:07:37.317 INFO:teuthology.task.ansible.out:mfjEfdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbB 2020-01-16T00:07:37.318 INFO:teuthology.task.ansible.out:mS+dTzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2020-01-16T00:07:37.327 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+z 2020-01-16T00:07:37.328 INFO:teuthology.task.ansible.out:mfjEfdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbB 2020-01-16T00:07:37.331 INFO:teuthology.task.ansible.out:mS+dTzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2020-01-16T00:07:37.458 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2020-01-16T00:07:37.480 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2020-01-16T00:07:37.597 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2020-01-16T00:07:37.598 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2020-01-16T00:07:37.750 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2020-01-16T00:07:37.836 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2020-01-16T00:07:37.868 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nf 2020-01-16T00:07:37.869 INFO:teuthology.task.ansible.out:pT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2020-01-16T00:07:37.963 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2020-01-16T00:07:38.016 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2020-01-16T00:07:38.100 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2020-01-16T00:07:38.153 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com", "key_options": null, "keyfile": "/home/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2020-01-16T00:07:38.240 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com", "key_options": null, "keyfile": "/home/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2020-01-16T00:07:38.296 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2020-01-16T00:07:38.387 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2020-01-16T00:07:38.451 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEudQMGBPqfZBV8bXpdQOgVTDP6GFsr5oL4gdbHFMuJCGe0Sfq8LTaSEBQx0MH7tXc7JJ1fU51U7DSt0kfy/PFeV1Y5XnNATiZu/SzdO3jHe06/8CCr3DLnlEmzXASq+tqDJbY1eV011d/TX1jiC8Qd1VNbpdZYExEDsDzwJXnWdh6rjTL4anGjIgQWcmU0XdbUJXvRP4dGjVvZVHZnflAkxLPBbAx/8BBOuBmv6vP4hNs4poi4VLwR6S6Ha1fgMJmsn6StsCVuOvKBJFuGvaulHgHgdoDJ1ryUFikkXXCWoNx4IbwtpQIkSxWMuAovlwwk9G8ozSs+RrFzCavE0f omw@omw-dev", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2020-01-16T00:07:38.521 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEudQMGBPqfZBV8bXpdQOgVTDP6GFsr5oL4gdbHFMuJCGe0Sfq8LTaSEBQx0MH7tXc7JJ1fU51U7DSt0kfy/PFeV1Y5XnNATiZu/SzdO3jHe06/8CCr3DLnlEmzXASq+tqDJbY1eV011d/TX1jiC8Qd1VNbpdZYExEDsDzwJXnWdh6rjTL4anGjIgQWcmU0XdbUJXvRP4dGjVvZVHZnflAkxLPBbAx/8BBOuBmv6vP4hNs4poi4VLwR6S6Ha1fgMJmsn6StsCVuOvKBJFuGvaulHgHgdoDJ1ryUFikkXXCWoNx4IbwtpQIkSxWMuAovlwwk9G8ozSs+RrFzCavE0f omw@omw-dev", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2020-01-16T00:07:38.583 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user 2020-01-16T00:07:38.584 INFO:teuthology.task.ansible.out:": "gabrioux", "validate_certs": true} 2020-01-16T00:07:38.638 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user 2020-01-16T00:07:38.640 INFO:teuthology.task.ansible.out:": "gabrioux", "validate_certs": true} 2020-01-16T00:07:38.722 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2020-01-16T00:07:38.789 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2020-01-16T00:07:38.879 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2020-01-16T00:07:38.922 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2020-01-16T00:07:39.021 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS 2020-01-16T00:07:39.022 INFO:teuthology.task.ansible.out:33Kkke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2020-01-16T00:07:39.050 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS 2020-01-16T00:07:39.052 INFO:teuthology.task.ansible.out:33Kkke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2020-01-16T00:07:39.160 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2020-01-16T00:07:39.182 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2020-01-16T00:07:39.300 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2020-01-16T00:07:39.322 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2020-01-16T00:07:39.427 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2020-01-16T00:07:39.449 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2020-01-16T00:07:39.567 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-01-16T00:07:39.589 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-01-16T00:07:39.696 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2020-01-16T00:07:39.729 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2020-01-16T00:07:39.841 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2020-01-16T00:07:39.876 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2020-01-16T00:07:39.972 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2020-01-16T00:07:40.039 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2020-01-16T00:07:40.124 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2020-01-16T00:07:40.160 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2020-01-16T00:07:40.276 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2020-01-16T00:07:40.298 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2020-01-16T00:07:40.400 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2020-01-16T00:07:40.498 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2020-01-16T00:07:40.551 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2020-01-16T00:07:40.632 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2020-01-16T00:07:40.675 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2020-01-16T00:07:40.793 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2020-01-16T00:07:40.809 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2020-01-16T00:07:40.938 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2020-01-16T00:07:40.960 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2020-01-16T00:07:41.091 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2020-01-16T00:07:41.102 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9/Io6M3q8yQqoPNPKAPJ6Fzq18DMfmNtxcD8GUWNo0xcMxd7XOlarvYc7INDQjkpzx6s+aqMme55pSUZuN5plW+nuFmRPqBaIxyeCnNjzCch0wbtEZe5dzT270FhzM1Tu4W0dw+WD6zrGjXayLNs2uxNp3UrWtMTBRzd0IfxJ1R2aJtiCE/HNxIZn+bhS/OmEPV1izz23MvKXD+nRaeJK6Aqrg1zbnpoNskz2fZ/AnmhnAMc4EHl/NUyW1mi4g81KcQMwE8poWmw1NpZje9gWFcISVClLvvhk/A6at/v90P7PisX+/ulTA/QYJYvGBjgbNtURYu/PWzYBOAEZQGfj shilpa@dhcp35-184.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqRSG3i8QY7t9sp6I8vDZ7LVdk+DDrMhI1QmvU2z+YQhUGDBOvvO6YP0gUXPDyalyj0sJXiwLWPmkrOsbnR96YCUcIt7Ba6R0d0w85JxcJ1WyPcsg2UyOBwq4A+s7VAFZm6FEWzlaubixsNp04gKDqBjSjaqA1Kwx8/Tg39byay8Hn3VJeZ1SkRukvhykz2e6gl7ydfuJQVmEyO3pTrd5157ZqgvQRABT10wc3vR2m4eC65hU/v0ci7HMY9oVtGEdWVBapWMuiocrOPCRWF5zrMJwW4vzPOCHa/47hBEIWfOWJidUVzAawnM3nOe7PykvvDK/iP8B6JpxKTE74eLeh shilpa@fedora.lab.eng.blr.redhat.com", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2020-01-16T00:07:41.231 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9/Io6M3q8yQqoPNPKAPJ6Fzq18DMfmNtxcD8GUWNo0xcMxd7XOlarvYc7INDQjkpzx6s+aqMme55pSUZuN5plW+nuFmRPqBaIxyeCnNjzCch0wbtEZe5dzT270FhzM1Tu4W0dw+WD6zrGjXayLNs2uxNp3UrWtMTBRzd0IfxJ1R2aJtiCE/HNxIZn+bhS/OmEPV1izz23MvKXD+nRaeJK6Aqrg1zbnpoNskz2fZ/AnmhnAMc4EHl/NUyW1mi4g81KcQMwE8poWmw1NpZje9gWFcISVClLvvhk/A6at/v90P7PisX+/ulTA/QYJYvGBjgbNtURYu/PWzYBOAEZQGfj shilpa@dhcp35-184.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqRSG3i8QY7t9sp6I8vDZ7LVdk+DDrMhI1QmvU2z+YQhUGDBOvvO6YP0gUXPDyalyj0sJXiwLWPmkrOsbnR96YCUcIt7Ba6R0d0w85JxcJ1WyPcsg2UyOBwq4A+s7VAFZm6FEWzlaubixsNp04gKDqBjSjaqA1Kwx8/Tg39byay8Hn3VJeZ1SkRukvhykz2e6gl7ydfuJQVmEyO3pTrd5157ZqgvQRABT10wc3vR2m4eC65hU/v0ci7HMY9oVtGEdWVBapWMuiocrOPCRWF5zrMJwW4vzPOCHa/47hBEIWfOWJidUVzAawnM3nOe7PykvvDK/iP8B6JpxKTE74eLeh shilpa@fedora.lab.eng.blr.redhat.com", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2020-01-16T00:07:41.233 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2020-01-16T00:07:41.235 INFO:teuthology.task.ansible.out: 2020-01-16T00:07:41.379 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2020-01-16T00:07:41.397 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2020-01-16T00:07:41.516 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2020-01-16T00:07:41.538 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2020-01-16T00:07:41.662 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2020-01-16T00:07:41.671 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2020-01-16T00:07:41.799 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2020-01-16T00:07:41.806 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2020-01-16T00:07:41.946 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2020-01-16T00:07:41.958 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2020-01-16T00:07:42.097 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2020-01-16T00:07:42.104 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2020-01-16T00:07:42.234 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2020-01-16T00:07:42.245 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2020-01-16T00:07:42.366 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2020-01-16T00:07:42.377 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2020-01-16T00:07:42.509 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2020-01-16T00:07:42.517 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2020-01-16T00:07:42.675 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0M 2020-01-16T00:07:42.677 INFO:teuthology.task.ansible.out:OIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.localdomain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2020-01-16T00:07:42.813 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfb 2020-01-16T00:07:42.814 INFO:teuthology.task.ansible.out:smXq2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2020-01-16T00:07:42.814 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9A 2020-01-16T00:07:42.816 INFO:teuthology.task.ansible.out:LTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.localdomain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2020-01-16T00:07:42.952 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz1UnJy64hXJKl1OsqRPlZ84m/fyHE8/Hqimy5s8sWcvRs2tL5oK30zFataqv1BC7r46+9t1eGUIRAuBg6QPuJnJCnPzS8yTVJ6DqSDGE2NAnpcMK5fSb+qgQ2adHFkjzZ4eKZbnDJMjYBtpUEp1Ve2ncdq5YCy6hGLIstkGq331ULBc/Lm3ZHBwy5I51S8eldkuP975otBYSPZaaN25QDncZzrW36zb3VwoGUAANEfHC1kLgTTFjx0bdsAGYCduXzyyXls8qSv9gbzcRzZZoH5B7tj8VDBoHszPnqGjGqwtWaelifYE/Svp+8fJKLlzRUM+XGNZoNcRl1/1XVhYV/ deepika@deepika", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2020-01-16T00:07:42.995 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfb 2020-01-16T00:07:42.996 INFO:teuthology.task.ansible.out:smXq2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2020-01-16T00:07:43.090 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2020-01-16T00:07:43.112 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz1UnJy64hXJKl1OsqRPlZ84m/fyHE8/Hqimy5s8sWcvRs2tL5oK30zFataqv1BC7r46+9t1eGUIRAuBg6QPuJnJCnPzS8yTVJ6DqSDGE2NAnpcMK5fSb+qgQ2adHFkjzZ4eKZbnDJMjYBtpUEp1Ve2ncdq5YCy6hGLIstkGq331ULBc/Lm3ZHBwy5I51S8eldkuP975otBYSPZaaN25QDncZzrW36zb3VwoGUAANEfHC1kLgTTFjx0bdsAGYCduXzyyXls8qSv9gbzcRzZZoH5B7tj8VDBoHszPnqGjGqwtWaelifYE/Svp+8fJKLlzRUM+XGNZoNcRl1/1XVhYV/ deepika@deepika", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2020-01-16T00:07:43.222 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2020-01-16T00:07:43.259 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2020-01-16T00:07:43.371 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2020-01-16T00:07:43.406 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2020-01-16T00:07:43.503 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2020-01-16T00:07:43.510 INFO:teuthology.task.ansible.out: 2020-01-16T00:07:43.525 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2020-01-16T00:07:43.660 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2020-01-16T00:07:43.682 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2020-01-16T00:07:43.794 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2020-01-16T00:07:43.845 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2020-01-16T00:07:43.929 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2020-01-16T00:07:43.992 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2020-01-16T00:07:44.061 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2020-01-16T00:07:44.124 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2020-01-16T00:07:44.210 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2020-01-16T00:07:44.260 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2020-01-16T00:07:44.360 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2020-01-16T00:07:44.382 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2020-01-16T00:07:44.491 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2020-01-16T00:07:44.534 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2020-01-16T00:07:44.642 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2020-01-16T00:07:44.684 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2020-01-16T00:07:44.785 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2020-01-16T00:07:44.838 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2020-01-16T00:07:44.927 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_key 2020-01-16T00:07:44.928 INFO:teuthology.task.ansible.out:s", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-01-16T00:07:44.966 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2020-01-16T00:07:45.084 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_key 2020-01-16T00:07:45.086 INFO:teuthology.task.ansible.out:s", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-01-16T00:07:45.096 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlnlBzRdWRqns/9VrYuF9tLILUjeGd9z8Tf/dPZmxAt3nnZqpTvI2xXemQpyt/2lsD0U24EuTc1Y/dBsgqvXtZOw51TZE8RK26hvCCaKGfcKONjvR7goHAmkYPHK9+F6m7fgYkRzA0+ZW4HcWNh/xxhhL2G1EGfz5bzIoGQJHSdzc6VeVIsY7HiYEMTxvxdsgV72Kjq97BdmHm56RBzoH24oN0UzNZS8Syx5SuWj9dA3IyFBPwjLlozSk6U2Ofd7qaXuQKl807eD9g6RhRldzHPS+i0EK3EM6Jvho5Y08k1KqV4gIEwMdzDbD3SH0xq+nUPFKcDTUFNp+tkg10GxqN zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2020-01-16T00:07:45.098 INFO:teuthology.task.ansible.out: 2020-01-16T00:07:45.223 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2020-01-16T00:07:45.247 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlnlBzRdWRqns/9VrYuF9tLILUjeGd9z8Tf/dPZmxAt3nnZqpTvI2xXemQpyt/2lsD0U24EuTc1Y/dBsgqvXtZOw51TZE8RK26hvCCaKGfcKONjvR7goHAmkYPHK9+F6m7fgYkRzA0+ZW4HcWNh/xxhhL2G1EGfz5bzIoGQJHSdzc6VeVIsY7HiYEMTxvxdsgV72Kjq97BdmHm56RBzoH24oN0UzNZS8Syx5SuWj9dA3IyFBPwjLlozSk6U2Ofd7qaXuQKl807eD9g6RhRldzHPS+i0EK3EM6Jvho5Y08k1KqV4gIEwMdzDbD3SH0xq+nUPFKcDTUFNp+tkg10GxqN zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2020-01-16T00:07:45.248 INFO:teuthology.task.ansible.out: 2020-01-16T00:07:45.354 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2020-01-16T00:07:45.427 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2020-01-16T00:07:45.491 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2020-01-16T00:07:45.590 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2020-01-16T00:07:45.632 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 RYbu6kdXmsBtrtAkph0lIg b4a266ab604d320ab175070a427f4ce486f65cde25c5ac248e8316e3fd2f2435', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 RYbu6kdXmsBtrtAkph0lIg b4a266ab604d320ab175070a427f4ce486f65cde25c5ac248e8316e3fd2f2435"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzVa77kg0qGGE3IpMZgSYJ+UwQiDEOg3Lji470V/sQ/R7g6/v7FZpaYqGn1wG3XeUp2gGACxNKLQfGV9EGU66NvfQYqn9A8dXRpTpZuGWN8EcKK98YrSAq0s0WzeiEzVT86RHzVHIqL6/Zn4B1p9sIeY9Hb+bfKOdtYkRBDoFELtiMRzHPFaiIJFnXl+EhH8wceYqXVNNvCuOUBMnOiDYLDtujvYdPt+ff7rvXuOzbwcAKoINFM1C4FtoUEYriKlsxJ2rr7AblfDm/WIrCZcqRBU4JcudtvZOtHhvN4uL6WgzuzuhIQFyePAeIHFUn6AiwKC/MApq7z1Qj+N3pN/Nz xiubli@fedora1", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2020-01-16T00:07:45.748 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2020-01-16T00:07:45.759 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2020-01-16T00:07:45.867 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:45 +0000 (0:00:15.199) 0:00:42.618 ****** 2020-01-16T00:07:45.915 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 RYbu6kdXmsBtrtAkph0lIg b4a266ab604d320ab175070a427f4ce486f65cde25c5ac248e8316e3fd2f2435', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 RYbu6kdXmsBtrtAkph0lIg b4a266ab604d320ab175070a427f4ce486f65cde25c5ac248e8316e3fd2f2435"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzVa77kg0qGGE3IpMZgSYJ+UwQiDEOg3Lji470V/sQ/R7g6/v7FZpaYqGn1wG3XeUp2gGACxNKLQfGV9EGU66NvfQYqn9A8dXRpTpZuGWN8EcKK98YrSAq0s0WzeiEzVT86RHzVHIqL6/Zn4B1p9sIeY9Hb+bfKOdtYkRBDoFELtiMRzHPFaiIJFnXl+EhH8wceYqXVNNvCuOUBMnOiDYLDtujvYdPt+ff7rvXuOzbwcAKoINFM1C4FtoUEYriKlsxJ2rr7AblfDm/WIrCZcqRBU4JcudtvZOtHhvN4uL6WgzuzuhIQFyePAeIHFUn6AiwKC/MApq7z1Qj+N3pN/Nz xiubli@fedora1", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2020-01-16T00:07:45.927 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-01-16T00:07:45.928 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:45.938 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:45.949 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:45.961 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "skip_reason": "Condi 2020-01-16T00:07:45.962 INFO:teuthology.task.ansible.out:tional result was False"} skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:45.973 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:45.974 INFO:teuthology.task.ansible.out: 2020-01-16T00:07:45.985 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:45.986 INFO:teuthology.task.ansible.out: 2020-01-16T00:07:45.988 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:45.989 INFO:teuthology.task.ansible.out: 2020-01-16T00:07:46.042 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-01-16T00:07:46.044 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2020-01-16T00:07:46.138 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:46 +0000 (0:00:00.271) 0:00:42.890 ****** 2020-01-16T00:07:46.143 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-01-16T00:07:46.144 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2020-01-16T00:07:46.145 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.182 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.193 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.193 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.204 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.219 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.227 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.228 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.239 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.289 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemerson", "validate_certs": true} 2020-01-16T00:07:46.397 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2020-01-16T00:07:46.419 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n" 2020-01-16T00:07:46.421 INFO:teuthology.task.ansible.out:}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.452 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2020-01-16T00:07:46.542 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemerson", "validate_certs": true} 2020-01-16T00:07:46.571 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2020-01-16T00:07:46.582 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.604 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.615 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.627 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.639 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.639 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.650 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.662 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.673 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.687 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.688 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2020-01-16T00:07:46.701 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.812 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.822 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.834 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.835 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.846 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.857 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.858 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2020-01-16T00:07:46.860 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.880 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.891 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.892 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.914 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.915 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.942 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.942 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.964 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.964 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.975 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.976 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.987 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:46.999 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.022 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.033 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.035 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.046 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.047 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.049 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.050 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.062 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.069 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.099 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.103 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45 2020-01-16T00:07:47.106 INFO:teuthology.task.ansible.out:dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.126 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.127 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.129 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.130 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.140 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.141 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.152 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.154 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.170 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.176 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.178 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.179 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.190 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.190 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.192 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.202 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.203 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.204 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.215 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.215 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.226 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.228 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.229 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.240 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.241 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.242 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.253 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.254 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.255 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.266 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.266 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.268 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.278 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.278 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.280 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.290 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.291 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.302 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.303 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.305 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.315 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.326 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.327 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.350 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.351 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.353 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.354 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.373 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.374 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.375 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.386 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.387 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.388 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.399 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.400 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.411 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.411 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.413 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.423 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.424 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.435 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.436 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.437 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.447 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.448 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.449 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.459 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.460 INFO:teuthology.task.ansible.out: 2020-01-16T00:07:47.461 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.463 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.478 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.506 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.506 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.508 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.508 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.509 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.511 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.521 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.523 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.534 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.537 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.545 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.545 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.556 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.557 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.558 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.569 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.570 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.580 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.581 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.592 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.593 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.594 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 RYbu6kdXmsBtrtAkph0lIg b4a266ab604d320ab175070a427f4ce486f65cde25c5ac248e8316e3fd2f2435', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 RYbu6kdXmsBtrtAkph0lIg b4a266ab604d320ab175070a427f4ce486f65cde25c5ac248e8316e3fd2f2435"}, "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.605 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.606 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.624 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.638 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.665 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:47 +0000 (0:00:01.527) 0:00:44.417 ****** 2020-01-16T00:07:47.672 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.673 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.675 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.675 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.676 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.688 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.689 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.699 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.700 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2020-01-16T00:07:47.700 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.745 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:47 +0000 (0:00:00.079) 0:00:44.496 ****** 2020-01-16T00:07:47.750 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-01-16T00:07:47.751 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.752 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.753 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.754 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.755 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.757 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.773 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.782 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.793 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.803 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.937 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.938 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 RYbu6kdXmsBtrtAkph0lIg b4a266ab604d320ab175070a427f4ce486f65cde25c5ac248e8316e3fd2f2435', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 RYbu6kdXmsBtrtAkph0lIg b4a266ab604d320ab175070a427f4ce486f65cde25c5ac248e8316e3fd2f2435"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.938 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:47.992 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:47 +0000 (0:00:00.247) 0:00:44.744 ****** 2020-01-16T00:07:48.018 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2020-01-16T00:07:48.019 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:48.061 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:48 +0000 (0:00:00.069) 0:00:44.813 ****** 2020-01-16T00:07:48.066 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi123.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2020-01-16T00:07:48.182 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2020-01-16T00:07:48.226 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2020-01-16T00:07:48.306 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2020-01-16T00:07:48.361 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2020-01-16T00:07:48.449 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2020-01-16T00:07:48.497 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2020-01-16T00:07:48.575 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2020-01-16T00:07:48.629 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2020-01-16T00:07:48.699 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2020-01-16T00:07:48.753 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2020-01-16T00:07:48.827 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2020-01-16T00:07:48.927 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2020-01-16T00:07:48.949 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2020-01-16T00:07:49.053 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2020-01-16T00:07:49.075 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2020-01-16T00:07:49.191 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2020-01-16T00:07:49.206 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2020-01-16T00:07:49.325 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2020-01-16T00:07:49.327 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2020-01-16T00:07:49.463 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2020-01-16T00:07:49.484 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2020-01-16T00:07:49.579 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2020-01-16T00:07:49.622 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2020-01-16T00:07:49.725 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2020-01-16T00:07:49.747 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2020-01-16T00:07:49.855 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2020-01-16T00:07:49.867 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2020-01-16T00:07:49.981 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2020-01-16T00:07:50.014 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2020-01-16T00:07:50.110 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2020-01-16T00:07:50.165 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2020-01-16T00:07:50.249 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2020-01-16T00:07:50.312 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2020-01-16T00:07:50.386 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2020-01-16T00:07:50.457 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2020-01-16T00:07:50.535 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2020-01-16T00:07:50.589 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2020-01-16T00:07:50.675 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2020-01-16T00:07:50.719 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2020-01-16T00:07:50.798 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2020-01-16T00:07:50.864 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2020-01-16T00:07:50.923 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2020-01-16T00:07:50.986 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2020-01-16T00:07:51.080 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2020-01-16T00:07:51.134 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2020-01-16T00:07:51.219 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2020-01-16T00:07:51.262 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2020-01-16T00:07:51.356 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2020-01-16T00:07:51.399 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2020-01-16T00:07:51.484 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2020-01-16T00:07:51.547 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2020-01-16T00:07:51.621 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2020-01-16T00:07:51.684 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2020-01-16T00:07:51.745 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2020-01-16T00:07:51.827 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2020-01-16T00:07:51.878 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2020-01-16T00:07:51.962 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2020-01-16T00:07:52.033 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2020-01-16T00:07:52.105 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2020-01-16T00:07:52.164 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2020-01-16T00:07:52.238 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2020-01-16T00:07:52.291 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2020-01-16T00:07:52.376 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2020-01-16T00:07:52.440 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2020-01-16T00:07:52.567 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2020-01-16T00:07:52.891 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "adeza", "name": "adeza", "remove": false, "state": "absent"} 2020-01-16T00:07:53.028 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2020-01-16T00:07:53.167 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2020-01-16T00:07:53.210 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "adeza", "name": "adeza", "remove": false, "state": "absent"} 2020-01-16T00:07:53.298 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2020-01-16T00:07:53.365 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2020-01-16T00:07:53.423 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2020-01-16T00:07:53.497 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2020-01-16T00:07:53.541 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2020-01-16T00:07:53.645 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2020-01-16T00:07:53.693 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2020-01-16T00:07:53.790 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2020-01-16T00:07:53.812 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2020-01-16T00:07:53.949 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2020-01-16T00:07:53.961 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2020-01-16T00:07:54.032 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:54 +0000 (0:00:05.970) 0:00:50.783 ****** 2020-01-16T00:07:54.094 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2020-01-16T00:07:54.211 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2020-01-16T00:07:54.367 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2020-01-16T00:07:54.430 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-01-16T00:07:54.432 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => {"changed": true, "checksum": "57a80abb6bd2cfda924e34ebf4650ffb7c549276", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "dcd56ec72ab960a85198ee963f616ac5", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1579133274.06-57993114977250/source", "state": "file", "uid": 0} 2020-01-16T00:07:54.478 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:54 +0000 (0:00:00.445) 0:00:51.229 ****** 2020-01-16T00:07:54.512 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:54 +0000 (0:00:00.034) 0:00:51.263 ****** 2020-01-16T00:07:54.517 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** ok: [smithi123.front.sepia.ceph.com] => { "msg": "Host smithi123.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2020-01-16T00:07:54.559 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:54 +0000 (0:00:00.047) 0:00:51.311 ****** 2020-01-16T00:07:54.595 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-01-16T00:07:54.597 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2020-01-16T00:07:54.653 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:54 +0000 (0:00:00.093) 0:00:51.404 ****** 2020-01-16T00:07:54.678 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-01-16T00:07:54.680 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:54.717 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:54 +0000 (0:00:00.063) 0:00:51.468 ****** 2020-01-16T00:07:54.742 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-01-16T00:07:54.744 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:54.781 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:54 +0000 (0:00:00.064) 0:00:51.533 ****** 2020-01-16T00:07:54.810 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-01-16T00:07:54.812 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:54.850 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:54 +0000 (0:00:00.068) 0:00:51.601 ****** 2020-01-16T00:07:54.878 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:54.919 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:54 +0000 (0:00:00.069) 0:00:51.670 ****** 2020-01-16T00:07:54.926 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "checksum": "57a80abb6bd2cfda924e34ebf4650ffb7c549276", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "dcd56ec72ab960a85198ee963f616ac5", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1579133274.54-274365115601462/source", "state": "file", "uid": 0} 2020-01-16T00:07:54.964 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:54 +0000 (0:00:00.045) 0:00:51.716 ****** 2020-01-16T00:07:54.968 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-01-16T00:07:54.969 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:55.008 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:55 +0000 (0:00:00.043) 0:00:51.759 ****** 2020-01-16T00:07:55.017 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** ok: [smithi145.front.sepia.ceph.com] => { "msg": "Host smithi145.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2020-01-16T00:07:55.054 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:55 +0000 (0:00:00.045) 0:00:51.805 ****** 2020-01-16T00:07:55.058 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-01-16T00:07:55.058 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:55.100 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:55 +0000 (0:00:00.046) 0:00:51.852 ****** 2020-01-16T00:07:55.106 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-01-16T00:07:55.106 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2020-01-16T00:07:55.149 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:55 +0000 (0:00:00.045) 0:00:51.897 ****** 2020-01-16T00:07:55.150 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-01-16T00:07:55.151 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:55.192 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:55 +0000 (0:00:00.046) 0:00:51.944 ****** 2020-01-16T00:07:55.198 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-01-16T00:07:55.199 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:55.244 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:55 +0000 (0:00:00.051) 0:00:51.995 ****** 2020-01-16T00:07:55.248 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-01-16T00:07:55.249 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:55.295 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:55 +0000 (0:00:00.051) 0:00:52.046 ****** 2020-01-16T00:07:55.300 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-01-16T00:07:55.302 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:55.347 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:55 +0000 (0:00:00.052) 0:00:52.098 ****** 2020-01-16T00:07:55.358 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** skipping: [smithi123.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi123.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-01-16T00:07:55.396 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:55 +0000 (0:00:00.048) 0:00:52.147 ****** 2020-01-16T00:07:55.401 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-01-16T00:07:55.406 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:55.444 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:55 +0000 (0:00:00.048) 0:00:52.196 ****** 2020-01-16T00:07:55.449 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-01-16T00:07:55.450 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:55.493 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:55 +0000 (0:00:00.048) 0:00:52.245 ****** 2020-01-16T00:07:55.498 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-01-16T00:07:55.500 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:55.545 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:55 +0000 (0:00:00.051) 0:00:52.296 ****** 2020-01-16T00:07:55.550 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-01-16T00:07:55.551 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:55.598 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:55 +0000 (0:00:00.053) 0:00:52.350 ****** 2020-01-16T00:07:55.604 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-01-16T00:07:55.604 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:55.655 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:55 +0000 (0:00:00.057) 0:00:52.407 ****** 2020-01-16T00:07:55.660 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-01-16T00:07:55.661 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-01-16T00:07:55.712 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:55 +0000 (0:00:00.057) 0:00:52.464 ****** 2020-01-16T00:07:55.720 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-01-16T00:07:55.721 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:55.765 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:55 +0000 (0:00:00.053) 0:00:52.517 ****** 2020-01-16T00:07:55.770 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-01-16T00:07:55.772 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:55.819 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:55 +0000 (0:00:00.053) 0:00:52.570 ****** 2020-01-16T00:07:55.824 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-01-16T00:07:55.826 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:55.865 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:55 +0000 (0:00:00.046) 0:00:52.617 ****** 2020-01-16T00:07:55.870 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** skipping: [smithi123.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-01-16T00:07:55.913 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:55 +0000 (0:00:00.048) 0:00:52.665 ****** 2020-01-16T00:07:55.919 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-01-16T00:07:55.920 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:55.962 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:55 +0000 (0:00:00.049) 0:00:52.714 ****** 2020-01-16T00:07:55.970 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-01-16T00:07:55.972 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:56.018 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:56 +0000 (0:00:00.052) 0:00:52.766 ****** 2020-01-16T00:07:56.020 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-01-16T00:07:56.022 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi145.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-01-16T00:07:56.066 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:56 +0000 (0:00:00.051) 0:00:52.817 ****** 2020-01-16T00:07:56.071 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-01-16T00:07:56.072 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:56.116 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:56 +0000 (0:00:00.049) 0:00:52.867 ****** 2020-01-16T00:07:56.121 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-01-16T00:07:56.123 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:56.162 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:56 +0000 (0:00:00.046) 0:00:52.914 ****** 2020-01-16T00:07:56.167 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-01-16T00:07:56.169 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:56.210 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:56 +0000 (0:00:00.047) 0:00:52.962 ****** 2020-01-16T00:07:56.216 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-01-16T00:07:56.216 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:56.258 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:56 +0000 (0:00:00.046) 0:00:53.008 ****** 2020-01-16T00:07:56.262 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-01-16T00:07:56.262 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:56.305 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:56 +0000 (0:00:00.047) 0:00:53.056 ****** 2020-01-16T00:07:56.311 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-01-16T00:07:56.313 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-01-16T00:07:56.352 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:56 +0000 (0:00:00.046) 0:00:53.103 ****** 2020-01-16T00:07:56.357 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-01-16T00:07:56.357 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-01-16T00:07:56.400 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:56 +0000 (0:00:00.048) 0:00:53.151 ****** 2020-01-16T00:07:56.405 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-01-16T00:07:56.407 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:56.444 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:56 +0000 (0:00:00.044) 0:00:53.196 ****** 2020-01-16T00:07:56.449 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-01-16T00:07:56.450 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:56.490 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:56 +0000 (0:00:00.045) 0:00:53.241 ****** 2020-01-16T00:07:56.495 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-01-16T00:07:56.496 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-01-16T00:07:56.535 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:56 +0000 (0:00:00.045) 0:00:53.286 ****** 2020-01-16T00:07:56.536 INFO:teuthology.task.ansible.out: 2020-01-16T00:07:56.542 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-01-16T00:07:56.543 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:56.586 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:56 +0000 (0:00:00.050) 0:00:53.337 ****** 2020-01-16T00:07:56.591 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-01-16T00:07:56.593 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:56.633 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:56 +0000 (0:00:00.046) 0:00:53.384 ****** 2020-01-16T00:07:56.641 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:56.685 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:56 +0000 (0:00:00.051) 0:00:53.436 ****** 2020-01-16T00:07:56.693 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:56.729 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:56 +0000 (0:00:00.044) 0:00:53.481 ****** 2020-01-16T00:07:56.734 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2020-01-16T00:07:56.734 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:56.777 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:56 +0000 (0:00:00.047) 0:00:53.529 ****** 2020-01-16T00:07:56.784 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-01-16T00:07:56.785 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:56.824 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:56 +0000 (0:00:00.046) 0:00:53.575 ****** 2020-01-16T00:07:56.829 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2020-01-16T00:07:56.829 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:56.870 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:56 +0000 (0:00:00.046) 0:00:53.622 ****** 2020-01-16T00:07:56.877 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-01-16T00:07:56.879 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:56.920 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:56 +0000 (0:00:00.049) 0:00:53.672 ****** 2020-01-16T00:07:56.926 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:56.969 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:56 +0000 (0:00:00.049) 0:00:53.721 ****** 2020-01-16T00:07:56.976 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-01-16T00:07:56.977 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-01-16T00:07:57.019 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:57 +0000 (0:00:00.049) 0:00:53.770 ****** 2020-01-16T00:07:57.024 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-01-16T00:07:57.024 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-01-16T00:07:57.064 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:57 +0000 (0:00:00.045) 0:00:53.816 ****** 2020-01-16T00:07:57.069 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-01-16T00:07:57.071 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:57.109 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:57 +0000 (0:00:00.044) 0:00:53.860 ****** 2020-01-16T00:07:57.113 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-01-16T00:07:57.170 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:57 +0000 (0:00:00.061) 0:00:53.922 ****** 2020-01-16T00:07:57.176 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-01-16T00:07:57.177 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:57.216 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:57 +0000 (0:00:00.045) 0:00:53.967 ****** 2020-01-16T00:07:57.220 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-01-16T00:07:57.220 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:57.267 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:57 +0000 (0:00:00.051) 0:00:54.019 ****** 2020-01-16T00:07:57.274 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-01-16T00:07:57.274 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:57.321 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:57 +0000 (0:00:00.053) 0:00:54.072 ****** 2020-01-16T00:07:57.325 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-01-16T00:07:57.326 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:57.375 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:57 +0000 (0:00:00.054) 0:00:54.127 ****** 2020-01-16T00:07:57.386 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:57.435 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:57 +0000 (0:00:00.059) 0:00:54.187 ****** 2020-01-16T00:07:57.443 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-01-16T00:07:57.444 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/18/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/18/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:57.446 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/18/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/18/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:57.488 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:57 +0000 (0:00:00.052) 0:00:54.239 ****** 2020-01-16T00:07:57.494 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2020-01-16T00:07:57.496 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:57.541 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:57 +0000 (0:00:00.052) 0:00:54.292 ****** 2020-01-16T00:07:57.545 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-01-16T00:07:57.545 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} 2020-01-16T00:07:57.547 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2020-01-16T00:07:57.594 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:57 +0000 (0:00:00.053) 0:00:54.346 ****** 2020-01-16T00:07:57.601 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-01-16T00:07:57.603 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:57.652 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:57 +0000 (0:00:00.057) 0:00:54.403 ****** 2020-01-16T00:07:57.658 INFO:teuthology.task.ansible.out: TASK [common : Temporarily comment mirrorlist so we can update yum cache] ****** 2020-01-16T00:07:57.660 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/18/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/18/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:57.661 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/18/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/18/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:57.703 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:57 +0000 (0:00:00.051) 0:00:54.455 ****** 2020-01-16T00:07:57.709 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-01-16T00:07:57.710 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-01-16T00:07:57.751 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:57 +0000 (0:00:00.047) 0:00:54.503 ****** 2020-01-16T00:07:57.756 INFO:teuthology.task.ansible.out: TASK [common : Update yum cache] *********************************************** 2020-01-16T00:07:57.757 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:57.799 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:57 +0000 (0:00:00.047) 0:00:54.551 ****** 2020-01-16T00:07:57.805 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-01-16T00:07:57.853 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:57 +0000 (0:00:00.053) 0:00:54.604 ****** 2020-01-16T00:07:57.858 INFO:teuthology.task.ansible.out: TASK [common : Uncomment mirrorlist] ******************************************* 2020-01-16T00:07:57.859 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/18/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/18/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:57.861 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/18/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/18/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:57.901 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:57 +0000 (0:00:00.047) 0:00:54.652 ****** 2020-01-16T00:07:57.906 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-01-16T00:07:57.908 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:57.946 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:57 +0000 (0:00:00.044) 0:00:54.697 ****** 2020-01-16T00:07:57.972 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-01-16T00:07:57.974 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:58.013 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:58 +0000 (0:00:00.066) 0:00:54.764 ****** 2020-01-16T00:07:58.050 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-01-16T00:07:58.050 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/18/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/18/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:58.052 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-01-16T00:07:58.052 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.014879", "end": "2020-01-16 00:07:58.280474", "rc": 0, "start": "2020-01-16 00:07:58.265595", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-01-16T00:07:58.053 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** skipping: [smithi145.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/18/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/18/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:58.094 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:58 +0000 (0:00:00.081) 0:00:54.845 ****** 2020-01-16T00:07:58.128 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:58 +0000 (0:00:00.034) 0:00:54.880 ****** 2020-01-16T00:07:58.132 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-01-16T00:07:58.134 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:58.175 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:58 +0000 (0:00:00.046) 0:00:54.927 ****** 2020-01-16T00:07:58.181 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* skipping: [smithi145.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} 2020-01-16T00:07:58.182 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2020-01-16T00:07:58.222 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:58 +0000 (0:00:00.046) 0:00:54.973 ****** 2020-01-16T00:07:58.226 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-01-16T00:07:58.227 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:58.282 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:58 +0000 (0:00:00.060) 0:00:55.034 ****** 2020-01-16T00:07:58.288 INFO:teuthology.task.ansible.out: TASK [common : Temporarily comment mirrorlist so we can update yum cache] ****** 2020-01-16T00:07:58.288 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/18/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/18/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:58.290 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/18/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/18/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:58.328 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:58 +0000 (0:00:00.045) 0:00:55.079 ****** 2020-01-16T00:07:58.333 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-01-16T00:07:58.334 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:58.374 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:58 +0000 (0:00:00.045) 0:00:55.125 ****** 2020-01-16T00:07:58.379 INFO:teuthology.task.ansible.out: TASK [common : Update yum cache] *********************************************** 2020-01-16T00:07:58.381 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:58.421 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:58 +0000 (0:00:00.047) 0:00:55.173 ****** 2020-01-16T00:07:58.427 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-01-16T00:07:58.427 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:58.482 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:58 +0000 (0:00:00.060) 0:00:55.233 ****** 2020-01-16T00:07:58.491 INFO:teuthology.task.ansible.out: TASK [common : Uncomment mirrorlist] ******************************************* skipping: [smithi145.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/18/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/18/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/18/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/18/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:58.537 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:58 +0000 (0:00:00.054) 0:00:55.288 ****** 2020-01-16T00:07:58.542 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-01-16T00:07:58.544 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:58.588 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:58 +0000 (0:00:00.050) 0:00:55.339 ****** 2020-01-16T00:07:58.615 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-01-16T00:07:58.617 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:58.656 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:58 +0000 (0:00:00.067) 0:00:55.407 ****** 2020-01-16T00:07:58.682 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-01-16T00:07:58.683 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.014703", "end": "2020-01-16 00:07:58.750216", "rc": 0, "start": "2020-01-16 00:07:58.735513", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-01-16T00:07:58.684 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:58.725 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:58 +0000 (0:00:00.069) 0:00:55.476 ****** 2020-01-16T00:07:58.764 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:58 +0000 (0:00:00.038) 0:00:55.515 ****** 2020-01-16T00:07:58.800 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-01-16T00:07:58.801 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:58.843 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:58 +0000 (0:00:00.077) 0:00:55.593 ****** 2020-01-16T00:07:58.868 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-01-16T00:07:58.870 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:58.918 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:58 +0000 (0:00:00.076) 0:00:55.669 ****** 2020-01-16T00:07:58.944 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-01-16T00:07:58.946 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:58.987 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:58 +0000 (0:00:00.069) 0:00:55.739 ****** 2020-01-16T00:07:59.014 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-01-16T00:07:59.015 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:59.053 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:59 +0000 (0:00:00.065) 0:00:55.804 ****** 2020-01-16T00:07:59.079 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-01-16T00:07:59.081 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:59.119 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:59 +0000 (0:00:00.066) 0:00:55.871 ****** 2020-01-16T00:07:59.145 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-01-16T00:07:59.147 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:59.187 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:59 +0000 (0:00:00.067) 0:00:55.938 ****** 2020-01-16T00:07:59.215 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-01-16T00:07:59.217 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:59.271 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:59 +0000 (0:00:00.084) 0:00:56.023 ****** 2020-01-16T00:07:59.278 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** changed: [smithi123.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.410838", "end": "2020-01-16 00:07:59.500465", "rc": 0, "start": "2020-01-16 00:07:59.089627", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-01-16T00:07:59.323 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:59 +0000 (0:00:00.051) 0:00:56.074 ****** 2020-01-16T00:07:59.351 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-01-16T00:07:59.352 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:59.395 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:59 +0000 (0:00:00.071) 0:00:56.146 ****** 2020-01-16T00:07:59.437 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-01-16T00:07:59.439 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.034709", "end": "2020-01-16 00:07:59.500370", "rc": 0, "start": "2020-01-16 00:07:59.465661", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-01-16T00:07:59.482 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:59 +0000 (0:00:00.086) 0:00:56.233 ****** 2020-01-16T00:07:59.509 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-01-16T00:07:59.510 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:07:59.549 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:59 +0000 (0:00:00.066) 0:00:56.300 ****** 2020-01-16T00:07:59.564 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-01-16T00:07:59.566 INFO:teuthology.task.ansible.out: [WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi123.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.043374", "end": "2020-01-16 00:07:59.800712", "rc": 0, "start": "2020-01-16 00:07:59.757338", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-01-16T00:07:59.604 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:59 +0000 (0:00:00.054) 0:00:56.355 ****** 2020-01-16T00:07:59.736 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.045862", "end": "2020-01-16 00:07:59.804609", "rc": 0, "start": "2020-01-16 00:07:59.758747", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-01-16T00:07:59.777 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:07:59 +0000 (0:00:00.173) 0:00:56.528 ****** 2020-01-16T00:08:03.147 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** [WARNING]: Could not find aptitude. Using apt-get instead ok: [smithi123.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1579133010, "cache_updated": false, "changed": false} 2020-01-16T00:08:03.193 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:03 +0000 (0:00:03.415) 0:00:59.944 ****** 2020-01-16T00:08:03.277 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1579133053, "cache_updated": false, "changed": false} 2020-01-16T00:08:03.320 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:03 +0000 (0:00:00.127) 0:01:00.072 ****** 2020-01-16T00:08:04.060 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-01-16T00:08:04.062 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"cache_update_time": 1579133010, "cache_updated": false, "changed": false} 2020-01-16T00:08:04.105 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:04 +0000 (0:00:00.784) 0:01:00.856 ****** 2020-01-16T00:08:04.141 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-01-16T00:08:04.142 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:04.185 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:04 +0000 (0:00:00.079) 0:01:00.936 ****** 2020-01-16T00:08:04.190 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-01-16T00:08:04.191 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"cache_update_time": 1579133053, "cache_updated": false, "changed": false} 2020-01-16T00:08:04.237 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:04 +0000 (0:00:00.052) 0:01:00.988 ****** 2020-01-16T00:08:04.263 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-01-16T00:08:04.264 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:04.305 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:04 +0000 (0:00:00.068) 0:01:01.057 ****** 2020-01-16T00:08:04.668 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-01-16T00:08:04.669 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2020-01-16T00:08:04.728 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:04 +0000 (0:00:00.423) 0:01:01.480 ****** 2020-01-16T00:08:04.734 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2020-01-16T00:08:04.774 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:04 +0000 (0:00:00.045) 0:01:01.525 ****** 2020-01-16T00:08:05.621 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2020-01-16T00:08:05.623 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"cache_update_time": 1579133010, "cache_updated": false, "changed": false} 2020-01-16T00:08:05.676 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:05 +0000 (0:00:00.902) 0:01:02.428 ****** 2020-01-16T00:08:05.736 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"cache_update_time": 1579133053, "cache_updated": false, "changed": false} 2020-01-16T00:08:05.813 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:05 +0000 (0:00:00.136) 0:01:02.564 ****** 2020-01-16T00:08:06.058 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-01-16T00:08:06.059 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2020-01-16T00:08:06.146 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2020-01-16T00:08:06.383 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2020-01-16T00:08:06.497 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2020-01-16T00:08:06.688 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2020-01-16T00:08:06.743 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:06 +0000 (0:00:00.930) 0:01:03.495 ****** 2020-01-16T00:08:06.796 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2020-01-16T00:08:06.841 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:06 +0000 (0:00:00.097) 0:01:03.592 ****** 2020-01-16T00:08:06.876 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-01-16T00:08:06.878 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2020-01-16T00:08:06.918 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:06 +0000 (0:00:00.077) 0:01:03.670 ****** 2020-01-16T00:08:06.977 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2020-01-16T00:08:07.026 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:07 +0000 (0:00:00.105) 0:01:03.776 ****** 2020-01-16T00:08:07.227 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-01-16T00:08:07.229 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 12725, "state": "file", "uid": 0} 2020-01-16T00:08:07.337 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 12725, "state": "file", "uid": 0} 2020-01-16T00:08:07.518 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2020-01-16T00:08:07.651 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2020-01-16T00:08:07.802 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2020-01-16T00:08:07.845 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:07 +0000 (0:00:00.819) 0:01:04.597 ****** 2020-01-16T00:08:07.882 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-01-16T00:08:07.883 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2020-01-16T00:08:07.922 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:07 +0000 (0:00:00.076) 0:01:04.673 ****** 2020-01-16T00:08:07.947 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-01-16T00:08:07.948 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2020-01-16T00:08:07.951 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-01-16T00:08:07.952 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:07.990 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:07 +0000 (0:00:00.068) 0:01:04.742 ****** 2020-01-16T00:08:08.027 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:08 +0000 (0:00:00.037) 0:01:04.779 ****** 2020-01-16T00:08:08.032 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-01-16T00:08:08.034 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-01-16T00:08:08.074 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:08 +0000 (0:00:00.046) 0:01:04.825 ****** 2020-01-16T00:08:08.079 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-01-16T00:08:08.080 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2020-01-16T00:08:08.134 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:08 +0000 (0:00:00.060) 0:01:04.886 ****** 2020-01-16T00:08:08.139 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-01-16T00:08:08.140 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-01-16T00:08:08.141 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-01-16T00:08:08.159 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi123.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-01-16T00:08:08.192 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:08 +0000 (0:00:00.057) 0:01:04.943 ****** 2020-01-16T00:08:08.197 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-01-16T00:08:08.199 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:08.239 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:08 +0000 (0:00:00.047) 0:01:04.991 ****** 2020-01-16T00:08:08.244 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-01-16T00:08:08.245 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:08.285 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:08 +0000 (0:00:00.045) 0:01:05.036 ****** 2020-01-16T00:08:08.294 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* skipping: [smithi145.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-01-16T00:08:08.331 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:08 +0000 (0:00:00.045) 0:01:05.082 ****** 2020-01-16T00:08:08.379 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-01-16T00:08:08.380 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-01-16T00:08:08.381 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-01-16T00:08:08.391 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-01-16T00:08:08.402 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-01-16T00:08:08.444 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:08 +0000 (0:00:00.113) 0:01:05.196 ****** 2020-01-16T00:08:08.480 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-01-16T00:08:08.482 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:08.526 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:08 +0000 (0:00:00.082) 0:01:05.278 ****** 2020-01-16T00:08:09.072 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu)] ***************** ok: [smithi123.front.sepia.ceph.com] => {"cache_update_time": 1579133010, "cache_updated": false, "changed": false} 2020-01-16T00:08:09.123 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:09 +0000 (0:00:00.596) 0:01:05.875 ****** 2020-01-16T00:08:09.420 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"cache_update_time": 1579133053, "cache_updated": false, "changed": false} 2020-01-16T00:08:09.422 INFO:teuthology.task.ansible.out: 2020-01-16T00:08:09.471 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:09 +0000 (0:00:00.347) 0:01:06.222 ****** 2020-01-16T00:08:09.476 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-01-16T00:08:09.477 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2020-01-16T00:08:09.518 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:09 +0000 (0:00:00.046) 0:01:06.269 ****** 2020-01-16T00:08:09.808 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2020-01-16T00:08:09.849 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:09 +0000 (0:00:00.330) 0:01:06.600 ****** 2020-01-16T00:08:09.854 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-01-16T00:08:09.855 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2020-01-16T00:08:09.904 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:09 +0000 (0:00:00.055) 0:01:06.656 ****** 2020-01-16T00:08:10.027 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-01-16T00:08:10.028 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-01-16T00:08:10.069 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:10 +0000 (0:00:00.162) 0:01:06.818 ****** 2020-01-16T00:08:10.092 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-01-16T00:08:10.094 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:10.132 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:10 +0000 (0:00:00.065) 0:01:06.884 ****** 2020-01-16T00:08:10.159 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-01-16T00:08:10.160 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-01-16T00:08:10.213 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:10 +0000 (0:00:00.080) 0:01:06.964 ****** 2020-01-16T00:08:10.220 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-01-16T00:08:10.226 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2020-01-16T00:08:10.264 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:10 +0000 (0:00:00.050) 0:01:07.015 ****** 2020-01-16T00:08:10.270 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-01-16T00:08:10.271 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:10.319 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:10 +0000 (0:00:00.054) 0:01:07.070 ****** 2020-01-16T00:08:10.390 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-01-16T00:08:10.391 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-01-16T00:08:10.432 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:10 +0000 (0:00:00.113) 0:01:07.184 ****** 2020-01-16T00:08:10.470 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-01-16T00:08:10.471 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:10.511 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:10 +0000 (0:00:00.078) 0:01:07.262 ****** 2020-01-16T00:08:10.538 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-01-16T00:08:10.540 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-01-16T00:08:10.582 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:10 +0000 (0:00:00.070) 0:01:07.333 ****** 2020-01-16T00:08:10.609 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:10.651 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:10 +0000 (0:00:00.068) 0:01:07.402 ****** 2020-01-16T00:08:10.668 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-01-16T00:08:10.671 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"changed": false, "checksum": "9c3d1d79583cf76a6d81bb2ec150bc4af5ca3c5c", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1166, "state": "file", "uid": 0} 2020-01-16T00:08:10.713 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:10 +0000 (0:00:00.062) 0:01:07.465 ****** 2020-01-16T00:08:11.003 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "9c3d1d79583cf76a6d81bb2ec150bc4af5ca3c5c", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1166, "state": "file", "uid": 0} 2020-01-16T00:08:11.054 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:11 +0000 (0:00:00.340) 0:01:07.805 ****** 2020-01-16T00:08:11.358 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-01-16T00:08:11.360 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2020-01-16 00:06:54 UTC", "ActiveEnterTimestampMonotonic": "17761799", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "basic.target -.mount time-sync.target remote-fs.target systemd-tmpfiles-setup.service systemd-journald.socket var-run.mount network.target local-fs.target nss-lookup.target system.slice sysinit.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2020-01-16 00:06:54 UTC", "AssertTimestampMonotonic": "17761081", "Before": "xdm.service multi-user.target getty@tty1.service shutdown.target plymouth-quit.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2020-01-16 00:06:54 UTC", "ConditionTimestampMonotonic": "17761081", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nagios-nrpe-ser 2020-01-16T00:08:11.362 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1260", "ExecMainStartTimestamp": "Thu 2020-01-16 00:06:54 UTC", "ExecMainStartTimestampMonotonic": "17761761", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Thu 2020-01-16 00:06:54 UTC] ; stop_time=[n/a] ; pid=1260 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2020-01-16 00:06:54 UTC", "InactiveExitTimestampMonotonic": "17761799", "InvocationID": "5e4f85b3a0e241b999f5329477a794bc", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "i 2020-01-16T00:08:11.363 INFO:teuthology.task.ansible.out:nfinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128031", "LimitNPROCSoft": "128031", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128031", "LimitSIGPENDINGSoft": "128031", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1260", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", 2020-01-16T00:08:11.365 INFO:teuthology.task.ansible.out: "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2020-01-16 00:06:54 UTC", "StateChangeTimestampMonotonic": "17761799", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2020-01-16 00:06:54 UTC", "WatchdogTimestampMonotonic": "17761798", "WatchdogUSec": "0"}} 2020-01-16T00:08:11.406 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:11 +0000 (0:00:00.349) 0:01:08.154 ****** 2020-01-16T00:08:11.412 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2020-01-16 00:06:40 UTC", "ActiveEnterTimestampMonotonic": "18281921", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-tmpfiles-setup.service sysinit.target basic.target system.slice systemd-journald.socket time-sync.target var-run.mount local-fs.target remote-fs.target -.mount network.target nss-lookup.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2020-01-16 00:06:40 UTC", "AssertTimestampMonotonic": "18280779", "Before": "xdm.service getty@tty1.service shutdown.target multi-user.target plymouth-quit.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2020-01-16 00:06:40 UTC", "ConditionTimestampMonotonic": "18280777", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nagios-nrpe-ser 2020-01-16T00:08:11.414 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1266", "ExecMainStartTimestamp": "Thu 2020-01-16 00:06:40 UTC", "ExecMainStartTimestampMonotonic": "18281874", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Thu 2020-01-16 00:06:40 UTC] ; stop_time=[n/a] ; pid=1266 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2020-01-16 00:06:40 UTC", "InactiveExitTimestampMonotonic": "18281921", "InvocationID": "977ced3bafed447d8d3c25fd74d49fa7", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "i 2020-01-16T00:08:11.415 INFO:teuthology.task.ansible.out:nfinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128031", "LimitNPROCSoft": "128031", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128031", "LimitSIGPENDINGSoft": "128031", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1266", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", 2020-01-16T00:08:11.416 INFO:teuthology.task.ansible.out: "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2020-01-16 00:06:40 UTC", "StateChangeTimestampMonotonic": "18281921", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2020-01-16 00:06:40 UTC", "WatchdogTimestampMonotonic": "18281920", "WatchdogUSec": "0"}} 2020-01-16T00:08:11.456 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:11 +0000 (0:00:00.052) 0:01:08.207 ****** 2020-01-16T00:08:11.460 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2020-01-16T00:08:11.460 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:11.507 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:11 +0000 (0:00:00.050) 0:01:08.258 ****** 2020-01-16T00:08:11.511 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:11.576 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:11 +0000 (0:00:00.069) 0:01:08.327 ****** 2020-01-16T00:08:11.581 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:11.627 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:11 +0000 (0:00:00.050) 0:01:08.378 ****** 2020-01-16T00:08:11.633 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:11.680 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:11 +0000 (0:00:00.053) 0:01:08.431 ****** 2020-01-16T00:08:11.685 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-01-16T00:08:11.686 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:11.729 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:11 +0000 (0:00:00.049) 0:01:08.480 ****** 2020-01-16T00:08:11.737 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:11.778 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:11 +0000 (0:00:00.049) 0:01:08.530 ****** 2020-01-16T00:08:11.784 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2020-01-16T00:08:11.784 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:11.824 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:11 +0000 (0:00:00.045) 0:01:08.575 ****** 2020-01-16T00:08:11.830 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:11.870 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:11 +0000 (0:00:00.046) 0:01:08.622 ****** 2020-01-16T00:08:11.876 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-01-16T00:08:11.878 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:11.918 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:11 +0000 (0:00:00.047) 0:01:08.669 ****** 2020-01-16T00:08:11.924 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:11.964 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:11 +0000 (0:00:00.046) 0:01:08.716 ****** 2020-01-16T00:08:11.970 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-01-16T00:08:11.972 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:12.011 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:12 +0000 (0:00:00.046) 0:01:08.763 ****** 2020-01-16T00:08:12.017 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:12.057 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:12 +0000 (0:00:00.046) 0:01:08.809 ****** 2020-01-16T00:08:12.063 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-01-16T00:08:12.063 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:12.103 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:12 +0000 (0:00:00.045) 0:01:08.854 ****** 2020-01-16T00:08:12.110 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:12.157 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:12 +0000 (0:00:00.053) 0:01:08.908 ****** 2020-01-16T00:08:12.163 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2020-01-16T00:08:12.163 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:12.203 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:12 +0000 (0:00:00.046) 0:01:08.955 ****** 2020-01-16T00:08:12.209 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:12.249 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:12 +0000 (0:00:00.045) 0:01:09.000 ****** 2020-01-16T00:08:12.256 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:12.313 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:12 +0000 (0:00:00.063) 0:01:09.064 ****** 2020-01-16T00:08:12.322 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:12.375 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:12 +0000 (0:00:00.061) 0:01:09.126 ****** 2020-01-16T00:08:12.380 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2020-01-16T00:08:12.381 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:12.422 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:12 +0000 (0:00:00.048) 0:01:09.174 ****** 2020-01-16T00:08:12.429 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:12.472 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:12 +0000 (0:00:00.049) 0:01:09.223 ****** 2020-01-16T00:08:12.479 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-01-16T00:08:12.480 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2020-01-16T00:08:12.535 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:12 +0000 (0:00:00.062) 0:01:09.286 ****** 2020-01-16T00:08:12.543 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2020-01-16T00:08:12.590 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:12 +0000 (0:00:00.055) 0:01:09.342 ****** 2020-01-16T00:08:12.597 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-01-16T00:08:12.598 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-virtualenv", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "python-configobj", "python-gevent", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "btrfs-tools", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "python-nose", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "python-numpy", "python-matplotlib", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2020-01-16T00:08:12.642 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:12 +0000 (0:00:00.051) 0:01:09.394 ****** 2020-01-16T00:08:12.648 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-virtualenv", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "python-configobj", "python-gevent", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "btrfs-tools", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "python-nose", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "python-numpy", "python-matplotlib", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2020-01-16T00:08:12.722 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:12 +0000 (0:00:00.072) 0:01:09.466 ****** TASK [testnode : Including major version specific variables.] ****************** ok: [smithi123.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2020-01-16T00:08:12.770 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:12 +0000 (0:00:00.054) 0:01:09.521 ****** 2020-01-16T00:08:12.777 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2020-01-16T00:08:12.829 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:12 +0000 (0:00:00.055) 0:01:09.577 ****** 2020-01-16T00:08:12.831 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-01-16T00:08:12.832 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-01-16T00:08:12.875 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:12 +0000 (0:00:00.049) 0:01:09.627 ****** 2020-01-16T00:08:12.882 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-01-16T00:08:12.922 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:12 +0000 (0:00:00.046) 0:01:09.674 ****** 2020-01-16T00:08:13.021 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-01-16T00:08:13.023 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-01-16T00:08:13.062 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:13 +0000 (0:00:00.139) 0:01:09.814 ****** 2020-01-16T00:08:13.068 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-01-16T00:08:13.108 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:13 +0000 (0:00:00.045) 0:01:09.860 ****** 2020-01-16T00:08:13.196 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-01-16T00:08:13.198 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"changed": false, "gid": 1205, "name": "ubuntu", "state": "present", "system": false} 2020-01-16T00:08:13.240 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:13 +0000 (0:00:00.131) 0:01:09.992 ****** 2020-01-16T00:08:13.257 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "gid": 1205, "name": "ubuntu", "state": "present", "system": false} 2020-01-16T00:08:13.299 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:13 +0000 (0:00:00.058) 0:01:10.050 ****** 2020-01-16T00:08:13.304 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-01-16T00:08:13.305 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:13.345 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:13 +0000 (0:00:00.046) 0:01:10.097 ****** 2020-01-16T00:08:13.352 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:13.394 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:13 +0000 (0:00:00.048) 0:01:10.145 ****** 2020-01-16T00:08:13.404 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:13.439 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:13 +0000 (0:00:00.045) 0:01:10.191 ****** 2020-01-16T00:08:13.445 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:13.486 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:13 +0000 (0:00:00.046) 0:01:10.237 ****** 2020-01-16T00:08:13.605 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-01-16T00:08:13.606 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2020-01-16T00:08:13.650 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:13 +0000 (0:00:00.163) 0:01:10.401 ****** 2020-01-16T00:08:13.657 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2020-01-16T00:08:13.699 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:13 +0000 (0:00:00.049) 0:01:10.450 ****** 2020-01-16T00:08:14.080 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-01-16T00:08:14.082 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2020-01-16T00:08:14.140 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:14 +0000 (0:00:00.441) 0:01:10.892 ****** 2020-01-16T00:08:14.148 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2020-01-16T00:08:14.196 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:14 +0000 (0:00:00.055) 0:01:10.947 ****** 2020-01-16T00:08:14.202 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-01-16T00:08:14.203 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi123"}, "changed": false} 2020-01-16T00:08:14.252 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:14 +0000 (0:00:00.055) 0:01:11.003 ****** 2020-01-16T00:08:14.258 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi145"}, "changed": false} 2020-01-16T00:08:14.299 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:14 +0000 (0:00:00.047) 0:01:11.051 ****** 2020-01-16T00:08:15.175 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2020-01-16T00:08:15.175 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi123.front.sepia.ceph.com", "ansible_hostname": "smithi123", "ansible_nodename": "smithi123"}, "changed": true, "name": "smithi123"} 2020-01-16T00:08:15.177 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi145.front.sepia.ceph.com", "ansible_hostname": "smithi145", "ansible_nodename": "smithi145"}, "changed": true, "name": "smithi145"} 2020-01-16T00:08:15.231 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:15 +0000 (0:00:00.931) 0:01:11.982 ****** 2020-01-16T00:08:15.278 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:15 +0000 (0:00:00.046) 0:01:12.029 ****** 2020-01-16T00:08:15.576 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* ok: [smithi123.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2020-01-16T00:08:15.616 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:15 +0000 (0:00:00.338) 0:01:12.368 ****** 2020-01-16T00:08:15.625 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2020-01-16T00:08:15.680 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:15 +0000 (0:00:00.063) 0:01:12.432 ****** 2020-01-16T00:08:15.969 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2020-01-16T00:08:15.971 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2020-01-16T00:08:16.032 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:16 +0000 (0:00:00.351) 0:01:12.784 ****** 2020-01-16T00:08:16.040 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2020-01-16T00:08:16.084 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:16 +0000 (0:00:00.051) 0:01:12.836 ****** 2020-01-16T00:08:16.602 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-01-16T00:08:16.602 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-01-16T00:08:16.653 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:16 +0000 (0:00:00.568) 0:01:13.404 ****** 2020-01-16T00:08:16.660 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-01-16T00:08:16.704 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:16 +0000 (0:00:00.051) 0:01:13.455 ****** 2020-01-16T00:08:16.709 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-01-16T00:08:16.711 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:16.754 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:16 +0000 (0:00:00.049) 0:01:13.505 ****** 2020-01-16T00:08:16.759 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:16.800 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:16 +0000 (0:00:00.046) 0:01:13.551 ****** 2020-01-16T00:08:16.805 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-01-16T00:08:16.806 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:16.846 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:16 +0000 (0:00:00.045) 0:01:13.597 ****** 2020-01-16T00:08:16.852 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:16.896 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:16 +0000 (0:00:00.049) 0:01:13.647 ****** 2020-01-16T00:08:16.901 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-01-16T00:08:16.901 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:16.944 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:16 +0000 (0:00:00.047) 0:01:13.695 ****** 2020-01-16T00:08:16.949 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:17.012 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:17 +0000 (0:00:00.066) 0:01:13.762 ****** 2020-01-16T00:08:17.017 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-01-16T00:08:17.018 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:17.064 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:17 +0000 (0:00:00.052) 0:01:13.814 ****** 2020-01-16T00:08:17.068 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:17.114 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:17 +0000 (0:00:00.051) 0:01:13.865 ****** 2020-01-16T00:08:17.119 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-01-16T00:08:17.120 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:17.167 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:17 +0000 (0:00:00.052) 0:01:13.918 ****** 2020-01-16T00:08:17.173 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:17.216 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:17 +0000 (0:00:00.048) 0:01:13.967 ****** 2020-01-16T00:08:17.221 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-01-16T00:08:17.223 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:17.262 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:17 +0000 (0:00:00.046) 0:01:14.014 ****** 2020-01-16T00:08:17.268 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:17.309 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:17 +0000 (0:00:00.046) 0:01:14.060 ****** 2020-01-16T00:08:17.314 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-01-16T00:08:17.315 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:17.368 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:17 +0000 (0:00:00.058) 0:01:14.119 ****** 2020-01-16T00:08:17.376 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:17.428 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:17 +0000 (0:00:00.059) 0:01:14.179 ****** 2020-01-16T00:08:17.432 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-01-16T00:08:17.474 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:17 +0000 (0:00:00.046) 0:01:14.226 ****** 2020-01-16T00:08:17.543 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:17 +0000 (0:00:00.068) 0:01:14.295 ****** 2020-01-16T00:08:17.548 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-01-16T00:08:17.607 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:17 +0000 (0:00:00.063) 0:01:14.358 ****** 2020-01-16T00:08:17.667 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:17 +0000 (0:00:00.052) 0:01:14.411 ****** TASK [testnode : Enable copr repos] ******************************************** 2020-01-16T00:08:17.714 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:17 +0000 (0:00:00.053) 0:01:14.465 ****** 2020-01-16T00:08:17.759 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:17 +0000 (0:00:00.045) 0:01:14.510 ****** 2020-01-16T00:08:17.764 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2020-01-16T00:08:17.766 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:17.805 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:17 +0000 (0:00:00.046) 0:01:14.557 ****** 2020-01-16T00:08:17.811 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:17.855 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:17 +0000 (0:00:00.049) 0:01:14.606 ****** 2020-01-16T00:08:17.859 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-01-16T00:08:17.861 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi123.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-01-16T00:08:17.909 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:17 +0000 (0:00:00.054) 0:01:14.661 ****** 2020-01-16T00:08:17.919 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-01-16T00:08:17.961 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:17 +0000 (0:00:00.051) 0:01:14.713 ****** 2020-01-16T00:08:17.967 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-01-16T00:08:17.969 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:18.010 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:18 +0000 (0:00:00.048) 0:01:14.761 ****** 2020-01-16T00:08:18.014 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:18.055 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:18 +0000 (0:00:00.045) 0:01:14.807 ****** 2020-01-16T00:08:18.061 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-01-16T00:08:18.062 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:18.104 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:18 +0000 (0:00:00.048) 0:01:14.856 ****** 2020-01-16T00:08:18.111 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:18.156 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:18 +0000 (0:00:00.051) 0:01:14.907 ****** 2020-01-16T00:08:18.164 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:18.204 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:18 +0000 (0:00:00.047) 0:01:14.955 ****** 2020-01-16T00:08:18.210 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:18.255 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:18 +0000 (0:00:00.051) 0:01:15.007 ****** 2020-01-16T00:08:18.260 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-01-16T00:08:18.261 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:18.303 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:18 +0000 (0:00:00.047) 0:01:15.054 ****** 2020-01-16T00:08:18.309 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:18.356 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:18 +0000 (0:00:00.053) 0:01:15.107 ****** 2020-01-16T00:08:18.363 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-01-16T00:08:18.364 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:18.405 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:18 +0000 (0:00:00.048) 0:01:15.156 ****** 2020-01-16T00:08:18.411 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:18.451 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:18 +0000 (0:00:00.046) 0:01:15.203 ****** 2020-01-16T00:08:18.457 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-01-16T00:08:18.458 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:18.500 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:18 +0000 (0:00:00.048) 0:01:15.252 ****** 2020-01-16T00:08:18.507 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:18.552 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:18 +0000 (0:00:00.052) 0:01:15.304 ****** 2020-01-16T00:08:18.558 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-01-16T00:08:18.558 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:18.601 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:18 +0000 (0:00:00.048) 0:01:15.352 ****** 2020-01-16T00:08:18.607 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:18.651 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:18 +0000 (0:00:00.050) 0:01:15.402 ****** 2020-01-16T00:08:18.658 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:18.699 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:18 +0000 (0:00:00.048) 0:01:15.451 ****** 2020-01-16T00:08:18.706 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:18.746 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:18 +0000 (0:00:00.046) 0:01:15.497 ****** 2020-01-16T00:08:18.751 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-01-16T00:08:18.752 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:18.794 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:18 +0000 (0:00:00.048) 0:01:15.546 ****** 2020-01-16T00:08:18.801 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:18.841 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:18 +0000 (0:00:00.046) 0:01:15.592 ****** 2020-01-16T00:08:18.846 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-01-16T00:08:18.847 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:18.897 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:18 +0000 (0:00:00.049) 0:01:15.642 ****** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:18.936 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:18 +0000 (0:00:00.045) 0:01:15.687 ****** 2020-01-16T00:08:18.940 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-01-16T00:08:18.941 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:18.982 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:18 +0000 (0:00:00.046) 0:01:15.734 ****** 2020-01-16T00:08:18.988 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:19.030 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:19 +0000 (0:00:00.047) 0:01:15.782 ****** 2020-01-16T00:08:19.035 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-01-16T00:08:19.035 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:19.081 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:19 +0000 (0:00:00.050) 0:01:15.832 ****** 2020-01-16T00:08:19.086 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:19.126 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:19 +0000 (0:00:00.045) 0:01:15.878 ****** 2020-01-16T00:08:19.131 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-01-16T00:08:19.132 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:19.174 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:19 +0000 (0:00:00.047) 0:01:15.926 ****** 2020-01-16T00:08:19.180 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:19.223 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:19 +0000 (0:00:00.048) 0:01:15.974 ****** 2020-01-16T00:08:19.227 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-01-16T00:08:19.229 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:19.269 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:19 +0000 (0:00:00.046) 0:01:16.020 ****** 2020-01-16T00:08:19.275 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:19.316 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:19 +0000 (0:00:00.047) 0:01:16.068 ****** 2020-01-16T00:08:19.321 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-01-16T00:08:19.322 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:19.367 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:19 +0000 (0:00:00.050) 0:01:16.119 ****** 2020-01-16T00:08:19.373 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:19.429 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:19 +0000 (0:00:00.061) 0:01:16.180 ****** 2020-01-16T00:08:19.530 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-01-16T00:08:19.531 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.003492", "end": "2020-01-16 00:08:19.735023", "rc": 0, "start": "2020-01-16 00:08:19.731531", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-01-16T00:08:19.594 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:19 +0000 (0:00:00.163) 0:01:16.344 ****** 2020-01-16T00:08:19.599 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005217", "end": "2020-01-16 00:08:19.625792", "rc": 0, "start": "2020-01-16 00:08:19.620575", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-01-16T00:08:19.661 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:19 +0000 (0:00:00.069) 0:01:16.413 ****** 2020-01-16T00:08:19.666 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-01-16T00:08:19.709 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:19 +0000 (0:00:00.048) 0:01:16.461 ****** 2020-01-16T00:08:19.760 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:19 +0000 (0:00:00.049) 0:01:16.510 ****** 2020-01-16T00:08:20.041 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** ok: [smithi123.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2020-01-16T00:08:20.080 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:20 +0000 (0:00:00.321) 0:01:16.832 ****** 2020-01-16T00:08:20.088 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2020-01-16T00:08:20.132 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:20 +0000 (0:00:00.052) 0:01:16.884 ****** 2020-01-16T00:08:20.137 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-01-16T00:08:20.139 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:20.181 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:20 +0000 (0:00:00.048) 0:01:16.932 ****** 2020-01-16T00:08:20.187 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:20.227 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:20 +0000 (0:00:00.046) 0:01:16.979 ****** 2020-01-16T00:08:21.451 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2020-01-16T00:08:21.452 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2020-01-16T00:08:21.650 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2020-01-16T00:08:22.387 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2020-01-16T00:08:22.430 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:22 +0000 (0:00:02.202) 0:01:19.181 ****** 2020-01-16T00:08:22.510 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2020-01-16T00:08:22.550 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:22 +0000 (0:00:00.120) 0:01:19.302 ****** 2020-01-16T00:08:23.412 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-01-16T00:08:23.412 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"cache_update_time": 1579133010, "cache_updated": false, "changed": false} 2020-01-16T00:08:23.454 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:23 +0000 (0:00:00.903) 0:01:20.205 ****** 2020-01-16T00:08:23.460 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"cache_update_time": 1579133053, "cache_updated": false, "changed": false} 2020-01-16T00:08:23.503 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:23 +0000 (0:00:00.049) 0:01:20.255 ****** 2020-01-16T00:08:23.508 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-01-16T00:08:23.550 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:23 +0000 (0:00:00.046) 0:01:20.301 ****** 2020-01-16T00:08:23.599 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:23 +0000 (0:00:00.046) 0:01:20.347 ****** 2020-01-16T00:08:23.694 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2020-01-16T00:08:23.696 INFO:teuthology.task.ansible.out: [WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo changed: [smithi123.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.022169", "end": "2020-01-16 00:08:23.919105", "rc": 0, "start": "2020-01-16 00:08:23.896936", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-01-16T00:08:23.740 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:23 +0000 (0:00:00.144) 0:01:20.492 ****** 2020-01-16T00:08:23.768 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.033332", "end": "2020-01-16 00:08:23.832206", "rc": 0, "start": "2020-01-16 00:08:23.798874", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-01-16T00:08:23.820 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:23 +0000 (0:00:00.080) 0:01:20.572 ****** 2020-01-16T00:08:26.747 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2020-01-16T00:08:26.747 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"cache_update_time": 1579133010, "cache_updated": false, "changed": false} 2020-01-16T00:08:26.803 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:26 +0000 (0:00:02.982) 0:01:23.554 ****** 2020-01-16T00:08:26.809 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"cache_update_time": 1579133053, "cache_updated": false, "changed": false} 2020-01-16T00:08:26.850 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:26 +0000 (0:00:00.046) 0:01:23.601 ****** 2020-01-16T00:08:36.502 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** changed: [smithi123.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 28 not upgraded.\nAfter this operation, 47.5 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database .. 2020-01-16T00:08:36.504 INFO:teuthology.task.ansible.out:. 95%\r(Reading database ... 100%\r(Reading database ... 137378 files and directories currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.21) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.21) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.21) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.21) ...\r\nRemoving librbd1 (12.2.12-0ubuntu0.18.04.3) ...\r\nRemoving librados2 (12.2.12-0ubuntu0.18.04.3) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 28 not upgraded.", "After this operation, 47.5 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading d 2020-01-16T00:08:36.505 INFO:teuthology.task.ansible.out:atabase ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137378 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.21) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.21) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.21) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.21) ...", "Removing librbd1 (12.2.12-0ubuntu0.18.04.3) ...", "Removing librados2 (12.2.12-0ubuntu0.18.04.3) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ..."]} 2020-01-16T00:08:36.543 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:36 +0000 (0:00:09.693) 0:01:33.295 ****** 2020-01-16T00:08:36.755 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 28 not upgraded.\nAfter this operation, 47.5 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137378 files and directorie 2020-01-16T00:08:36.757 INFO:teuthology.task.ansible.out:s currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.21) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.21) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.21) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.21) ...\r\nRemoving librbd1 (12.2.12-0ubuntu0.18.04.3) ...\r\nRemoving librados2 (12.2.12-0ubuntu0.18.04.3) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 28 not upgraded.", "After this operation, 47.5 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Readin 2020-01-16T00:08:36.759 INFO:teuthology.task.ansible.out:g database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137378 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.21) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.21) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.21) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.21) ...", "Removing librbd1 (12.2.12-0ubuntu0.18.04.3) ...", "Removing librados2 (12.2.12-0ubuntu0.18.04.3) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ..."]} 2020-01-16T00:08:36.799 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:36 +0000 (0:00:00.254) 0:01:33.550 ****** 2020-01-16T00:08:37.426 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* ok: [smithi123.front.sepia.ceph.com] => {"cache_update_time": 1579133010, "cache_updated": false, "changed": false} 2020-01-16T00:08:37.469 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:37 +0000 (0:00:00.670) 0:01:34.220 ****** 2020-01-16T00:08:37.471 INFO:teuthology.task.ansible.out: 2020-01-16T00:08:37.498 INFO:teuthology.task.ansible.out: 2020-01-16T00:08:37.499 INFO:teuthology.task.ansible.out:TASK [testnode : Upgrade non aarch64 packages] ********************************* skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:37.537 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:37 +0000 (0:00:00.068) 0:01:34.288 ****** 2020-01-16T00:08:37.538 INFO:teuthology.task.ansible.out: 2020-01-16T00:08:37.691 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-01-16T00:08:37.693 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"cache_update_time": 1579133053, "cache_updated": false, "changed": false} 2020-01-16T00:08:37.733 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:37 +0000 (0:00:00.195) 0:01:34.484 ****** 2020-01-16T00:08:37.769 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-01-16T00:08:37.770 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:37.813 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:37 +0000 (0:00:00.080) 0:01:34.564 ****** 2020-01-16T00:08:47.138 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-01-16T00:08:47.140 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => {"cache_update_time": 1579133010, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.5).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n python-nose-doc samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 python3-nose qemu-block-extra qemu-system-common\n qemu-system-x86 qemu-utils\n0 upgraded, 7 newly installed, 0 to remove and 28 not upgraded.\nNeed to get 10.5 MB of archives.\nAfter this operation, 48.1 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.12-0ubuntu0.18.04.3 [2702 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.12-0ubuntu0.18.04.3 [920 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.21 [39.8 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.21 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-nose all 1.3.7-3 [115 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.21 [5197 kB]\nGet:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.21 [868 kB]\nFetched 10.5 MB in 1s (9919 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Re 2020-01-16T00:08:47.141 INFO:teuthology.task.ansible.out:ading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137236 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.12-0ubuntu0.18.04.3_amd64.deb ...\r\nUnpacking librados2 (12.2.12-0ubuntu0.18.04.3) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.12-0ubuntu0.18.04.3_amd64.deb ...\r\nUnpacking librbd1 (12.2.12-0ubuntu0.18.04.3) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.21) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.21) ...\r\nSelecting previously unselected package python3-nose.\r\nPreparing to unpack .../4-python3-nose_1.3.7-3_all.deb ...\r\nUnpacking python3-nose (1.3.7-3) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../5-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.21) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../6-qemu-utils_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.21) ...\r\nSetting up librados2 (12.2.12-0ubuntu0.18.04.3) ...\r\nSetting up python3-nose (1.3.7-3) ...\r\nSetting up librbd1 (12.2.12-0ubuntu0.18.04.3) ...\r\nSetting up qemu-block- 2020-01-16T00:08:47.143 INFO:teuthology.task.ansible.out:extra:amd64 (1:2.11+dfsg-1ubuntu7.21) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.21) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.21) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.21) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.5).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " python-nose-doc samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 python3-nose qemu-block-extra qemu-system-common", " qemu-system-x86 qemu-utils", "0 upgraded, 7 newly installed, 0 to remove and 28 not upgraded.", "Need to get 10.5 MB of archives.", "After this operation, 48.1 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.12-0ubuntu0.18.04.3 [2702 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.12-0ubuntu0.18.04.3 [920 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.21 [39.8 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.21 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-nose all 1.3.7-3 [115 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.21 [5197 kB]", "Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.21 [868 kB]", "Fetched 10.5 MB in 1s (9919 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading databa 2020-01-16T00:08:47.144 INFO:teuthology.task.ansible.out:se ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137236 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.12-0ubuntu0.18.04.3_amd64.deb ...", "Unpacking librados2 (12.2.12-0ubuntu0.18.04.3) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.12-0ubuntu0.18.04.3_amd64.deb ...", "Unpacking librbd1 (12.2.12-0ubuntu0.18.04.3) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.21) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.21) ...", "Selecting previously unselected package python3-nose.", "Preparing to unpack .../4-python3-nose_1.3.7-3_all.deb ...", "Unpacking python3-nose (1.3.7-3) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../5-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.21) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../6-qemu-utils_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.21) ...", "Setting up librados2 (12.2.12-0ubuntu0.18.04.3) ...", "Setting 2020-01-16T00:08:47.145 INFO:teuthology.task.ansible.out:up python3-nose (1.3.7-3) ...", "Setting up librbd1 (12.2.12-0ubuntu0.18.04.3) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.21) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.21) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.21) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.21) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ..."]} 2020-01-16T00:08:47.179 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:47 +0000 (0:00:09.366) 0:01:43.931 ****** 2020-01-16T00:08:47.515 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"cache_update_time": 1579133053, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.5).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n python-nose-doc samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 python3-nose qemu-block-extra qemu-system-common\n qemu-system-x86 qemu-utils\n0 upgraded, 7 newly installed, 0 to remove and 28 not upgraded.\nNeed to get 10.5 MB of archives.\nAfter this operation, 48.1 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.12-0ubuntu0.18.04.3 [2702 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.12-0ubuntu0.18.04.3 [920 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.21 [39.8 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.21 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-nose all 1.3.7-3 [115 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.21 [5197 kB]\nGet:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.21 [868 kB]\nFetched 10.5 MB in 1s (9785 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Re 2020-01-16T00:08:47.516 INFO:teuthology.task.ansible.out:ading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137236 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.12-0ubuntu0.18.04.3_amd64.deb ...\r\nUnpacking librados2 (12.2.12-0ubuntu0.18.04.3) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.12-0ubuntu0.18.04.3_amd64.deb ...\r\nUnpacking librbd1 (12.2.12-0ubuntu0.18.04.3) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.21) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.21) ...\r\nSelecting previously unselected package python3-nose.\r\nPreparing to unpack .../4-python3-nose_1.3.7-3_all.deb ...\r\nUnpacking python3-nose (1.3.7-3) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../5-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.21) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../6-qemu-utils_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.21) ...\r\nSetting up librados2 (12.2.12-0ubuntu0.18.04.3) ...\r\nSetting up python3-nose (1.3.7-3) ...\r\nSetting up librbd1 (12.2.12-0ubuntu0.18.04.3) ...\r\nSetting up qemu-block- 2020-01-16T00:08:47.518 INFO:teuthology.task.ansible.out:extra:amd64 (1:2.11+dfsg-1ubuntu7.21) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.21) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.21) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.21) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.5).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " python-nose-doc samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 python3-nose qemu-block-extra qemu-system-common", " qemu-system-x86 qemu-utils", "0 upgraded, 7 newly installed, 0 to remove and 28 not upgraded.", "Need to get 10.5 MB of archives.", "After this operation, 48.1 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.12-0ubuntu0.18.04.3 [2702 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.12-0ubuntu0.18.04.3 [920 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.21 [39.8 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.21 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-nose all 1.3.7-3 [115 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.21 [5197 kB]", "Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.21 [868 kB]", "Fetched 10.5 MB in 1s (9785 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading databa 2020-01-16T00:08:47.519 INFO:teuthology.task.ansible.out:se ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137236 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.12-0ubuntu0.18.04.3_amd64.deb ...", "Unpacking librados2 (12.2.12-0ubuntu0.18.04.3) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.12-0ubuntu0.18.04.3_amd64.deb ...", "Unpacking l 2020-01-16T00:08:47.519 INFO:teuthology.task.ansible.out:ibrbd1 (12.2.12-0ubuntu0.18.04.3) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.21) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.21) ...", "Selecting previously unselected package python3-nose.", "Preparing to unpack .../4-python3-nose_1.3.7-3_all.deb ...", "Unpacking python3-nose (1.3.7-3) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../5-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.21) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../6-qemu-utils_1%3a2.11+dfsg-1ubuntu7.21_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.21) ...", "Setting up librados2 (12.2.12-0ubuntu0.18.04.3) ...", "Setting 2020-01-16T00:08:47.521 INFO:teuthology.task.ansible.out:up python3-nose (1.3.7-3) ...", "Setting up librbd1 (12.2.12-0ubuntu0.18.04.3) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.21) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.21) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.21) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.21) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ..."]} 2020-01-16T00:08:47.560 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:47 +0000 (0:00:00.380) 0:01:44.311 ****** 2020-01-16T00:08:48.032 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-01-16T00:08:48.033 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"cache_update_time": 1579133010, "cache_updated": false, "changed": false} 2020-01-16T00:08:48.073 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:48 +0000 (0:00:00.513) 0:01:44.825 ****** 2020-01-16T00:08:48.203 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-01-16T00:08:48.205 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2020-01-16T00:08:48.246 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:48 +0000 (0:00:00.173) 0:01:44.998 ****** 2020-01-16T00:08:48.272 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-01-16T00:08:48.273 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:48.312 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:48 +0000 (0:00:00.065) 0:01:45.064 ****** 2020-01-16T00:08:48.443 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ ok: [smithi123.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2020-01-16T00:08:48.484 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:48 +0000 (0:00:00.171) 0:01:45.235 ****** 2020-01-16T00:08:48.490 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-01-16T00:08:48.491 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"cache_update_time": 1579133053, "cache_updated": false, "changed": false} 2020-01-16T00:08:48.531 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:48 +0000 (0:00:00.047) 0:01:45.283 ****** 2020-01-16T00:08:48.671 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-01-16T00:08:48.672 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2020-01-16T00:08:48.712 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:48 +0000 (0:00:00.180) 0:01:45.463 ****** 2020-01-16T00:08:48.737 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-01-16T00:08:48.740 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:48.779 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:48 +0000 (0:00:00.066) 0:01:45.530 ****** 2020-01-16T00:08:48.805 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-01-16T00:08:48.806 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2020-01-16T00:08:48.849 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:48 +0000 (0:00:00.069) 0:01:45.600 ****** 2020-01-16T00:08:48.917 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-01-16T00:08:48.918 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2020-01-16T00:08:48.959 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:48 +0000 (0:00:00.110) 0:01:45.710 ****** 2020-01-16T00:08:48.995 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-01-16T00:08:48.996 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1205, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-01-16T00:08:49.044 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:49 +0000 (0:00:00.084) 0:01:45.795 ****** 2020-01-16T00:08:49.186 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-01-16T00:08:49.187 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-01-16T00:08:49.232 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:49 +0000 (0:00:00.187) 0:01:45.983 ****** 2020-01-16T00:08:49.311 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-01-16T00:08:49.313 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2020-01-16T00:08:49.358 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:49 +0000 (0:00:00.126) 0:01:46.110 ****** 2020-01-16T00:08:49.503 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-01-16T00:08:49.506 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Thu 2020-01-16 00:06:55 UTC", "ActiveEnterTimestampMonotonic": "18315206", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "-.mount sysinit.target systemd-tmpfiles-setup.service remote-fs.target systemd-journald.socket nss-lookup.target network.target system.slice basic.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2020-01-16 00:06:54 UTC", "AssertTimestampMonotonic": "17761949", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2020-01-16 00:06:54 UTC", "ConditionTimestampMonotonic": "17761949", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2020-01-16T00:08:49.508 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1338", "ExecMainStartTimestamp": "Thu 2020-01-16 00:06:55 UTC", "ExecMainStartTimestampMonotonic": "18315196", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2020-01-16 00:06:54 UTC", "InactiveExitTimestampMonotonic": "17762897", "InvocationID": "d0d709ed7f944347a8aaaa7e1bf36d8c", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2020-01-16T00:08:49.509 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128031", "LimitNPROCSoft": "128031", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128031", "LimitSIGPENDINGSoft": "128031", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1338", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": 2020-01-16T00:08:49.510 INFO:teuthology.task.ansible.out: "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2020-01-16 00:06:55 UTC", "StateChangeTimestampMonotonic": "18315206", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2020-01-16 00:06:55 UTC", "WatchdogTimestampMonotonic": "18315205", "WatchdogUSec": "0"}} 2020-01-16T00:08:49.552 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:49 +0000 (0:00:00.193) 0:01:46.304 ****** 2020-01-16T00:08:49.558 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-01-16T00:08:49.560 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1205, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-01-16T00:08:49.601 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:49 +0000 (0:00:00.048) 0:01:46.353 ****** 2020-01-16T00:08:49.608 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-01-16T00:08:49.609 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:49.660 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:49 +0000 (0:00:00.058) 0:01:46.412 ****** 2020-01-16T00:08:49.687 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-01-16T00:08:49.688 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:49.731 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:49 +0000 (0:00:00.070) 0:01:46.482 ****** 2020-01-16T00:08:49.736 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-01-16T00:08:49.737 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-01-16T00:08:49.780 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:49 +0000 (0:00:00.048) 0:01:46.531 ****** 2020-01-16T00:08:49.781 INFO:teuthology.task.ansible.out: 2020-01-16T00:08:49.787 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-01-16T00:08:49.787 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:49.832 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:49 +0000 (0:00:00.052) 0:01:46.583 ****** 2020-01-16T00:08:49.854 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-01-16T00:08:49.855 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:49.894 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:49 +0000 (0:00:00.062) 0:01:46.646 ****** 2020-01-16T00:08:49.920 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-01-16T00:08:49.922 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:49.959 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:49 +0000 (0:00:00.064) 0:01:46.711 ****** 2020-01-16T00:08:49.985 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-01-16T00:08:49.986 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:50.025 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:50 +0000 (0:00:00.065) 0:01:46.776 ****** 2020-01-16T00:08:50.052 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-01-16T00:08:50.053 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:50.095 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:50 +0000 (0:00:00.070) 0:01:46.847 ****** 2020-01-16T00:08:50.102 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-01-16T00:08:50.104 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Thu 2020-01-16 00:06:41 UTC", "ActiveEnterTimestampMonotonic": "18865097", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "-.mount systemd-tmpfiles-setup.service system.slice remote-fs.target basic.target network.target sysinit.target systemd-journald.socket nss-lookup.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2020-01-16 00:06:40 UTC", "AssertTimestampMonotonic": "18278923", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2020-01-16 00:06:40 UTC", "ConditionTimestampMonotonic": "18278923", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2020-01-16T00:08:50.106 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1408", "ExecMainStartTimestamp": "Thu 2020-01-16 00:06:41 UTC", "ExecMainStartTimestampMonotonic": "18865086", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2020-01-16 00:06:40 UTC", "InactiveExitTimestampMonotonic": "18279700", "InvocationID": "98b7b01be0874e6f84433a225a7ad2fe", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2020-01-16T00:08:50.107 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128031", "LimitNPROCSoft": "128031", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128031", "LimitSIGPENDINGSoft": "128031", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1408", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": 2020-01-16T00:08:50.109 INFO:teuthology.task.ansible.out: "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2020-01-16 00:06:41 UTC", "StateChangeTimestampMonotonic": "18865097", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2020-01-16 00:06:41 UTC", "WatchdogTimestampMonotonic": "18865096", "WatchdogUSec": "0"}} 2020-01-16T00:08:50.146 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:50 +0000 (0:00:00.050) 0:01:46.898 ****** 2020-01-16T00:08:50.152 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:50.191 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:50 +0000 (0:00:00.044) 0:01:46.943 ****** 2020-01-16T00:08:50.196 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-01-16T00:08:50.198 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:50.239 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:50 +0000 (0:00:00.047) 0:01:46.990 ****** 2020-01-16T00:08:50.244 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-01-16T00:08:50.245 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:50.286 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:50 +0000 (0:00:00.047) 0:01:47.037 ****** 2020-01-16T00:08:50.291 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-01-16T00:08:50.292 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:50.332 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:50 +0000 (0:00:00.046) 0:01:47.084 ****** 2020-01-16T00:08:50.337 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-01-16T00:08:50.338 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:50.377 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:50 +0000 (0:00:00.044) 0:01:47.128 ****** 2020-01-16T00:08:50.382 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-01-16T00:08:50.383 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:50.424 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:50 +0000 (0:00:00.046) 0:01:47.175 ****** 2020-01-16T00:08:50.429 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-01-16T00:08:50.430 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:50.468 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:50 +0000 (0:00:00.044) 0:01:47.220 ****** 2020-01-16T00:08:50.473 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-01-16T00:08:50.475 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:50.513 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:50 +0000 (0:00:00.044) 0:01:47.264 ****** 2020-01-16T00:08:50.518 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-01-16T00:08:50.519 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:50.557 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:50 +0000 (0:00:00.043) 0:01:47.308 ****** 2020-01-16T00:08:50.561 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-01-16T00:08:50.563 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:50.605 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:50 +0000 (0:00:00.048) 0:01:47.357 ****** 2020-01-16T00:08:50.609 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-01-16T00:08:50.611 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:50.654 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:50 +0000 (0:00:00.044) 0:01:47.401 ****** 2020-01-16T00:08:50.655 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-01-16T00:08:50.655 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:50.695 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:50 +0000 (0:00:00.045) 0:01:47.447 ****** 2020-01-16T00:08:50.700 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-01-16T00:08:50.702 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:50.748 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:50 +0000 (0:00:00.045) 0:01:47.492 ****** TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:50.786 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:50 +0000 (0:00:00.045) 0:01:47.538 ****** 2020-01-16T00:08:50.791 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-01-16T00:08:50.793 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:50.832 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:50 +0000 (0:00:00.045) 0:01:47.583 ****** 2020-01-16T00:08:50.836 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-01-16T00:08:50.838 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:50.877 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:50 +0000 (0:00:00.045) 0:01:47.629 ****** 2020-01-16T00:08:50.885 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:50.923 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:50 +0000 (0:00:00.045) 0:01:47.674 ****** 2020-01-16T00:08:50.928 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-01-16T00:08:50.929 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:50.968 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:50 +0000 (0:00:00.045) 0:01:47.719 ****** 2020-01-16T00:08:50.973 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-01-16T00:08:50.973 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:51.013 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:51 +0000 (0:00:00.044) 0:01:47.764 ****** 2020-01-16T00:08:51.018 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-01-16T00:08:51.021 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:51.060 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:51 +0000 (0:00:00.046) 0:01:47.811 ****** 2020-01-16T00:08:51.065 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-01-16T00:08:51.067 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:51.108 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:51 +0000 (0:00:00.047) 0:01:47.859 ****** 2020-01-16T00:08:51.113 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-01-16T00:08:51.114 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:51.153 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:51 +0000 (0:00:00.045) 0:01:47.905 ****** 2020-01-16T00:08:51.158 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-01-16T00:08:51.158 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:51.198 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:51 +0000 (0:00:00.045) 0:01:47.950 ****** 2020-01-16T00:08:51.203 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-01-16T00:08:51.205 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:51.244 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:51 +0000 (0:00:00.045) 0:01:47.996 ****** 2020-01-16T00:08:51.249 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-01-16T00:08:51.251 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:51.295 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:51 +0000 (0:00:00.046) 0:01:48.042 ****** 2020-01-16T00:08:51.298 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-01-16T00:08:51.299 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:51.339 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:51 +0000 (0:00:00.048) 0:01:48.091 ****** 2020-01-16T00:08:51.344 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-01-16T00:08:51.346 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:51.385 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:51 +0000 (0:00:00.045) 0:01:48.137 ****** 2020-01-16T00:08:51.390 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-01-16T00:08:51.392 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:51.432 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:51 +0000 (0:00:00.046) 0:01:48.184 ****** 2020-01-16T00:08:51.437 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-01-16T00:08:51.439 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:51.478 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:51 +0000 (0:00:00.045) 0:01:48.230 ****** 2020-01-16T00:08:51.483 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-01-16T00:08:51.485 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:51.527 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:51 +0000 (0:00:00.049) 0:01:48.279 ****** 2020-01-16T00:08:51.532 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-01-16T00:08:51.533 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:51.577 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:51 +0000 (0:00:00.045) 0:01:48.324 ****** 2020-01-16T00:08:51.578 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-01-16T00:08:51.578 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:51.619 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:51 +0000 (0:00:00.045) 0:01:48.370 ****** 2020-01-16T00:08:51.624 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-01-16T00:08:51.626 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:51.664 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:51 +0000 (0:00:00.045) 0:01:48.416 ****** 2020-01-16T00:08:51.669 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-01-16T00:08:51.671 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:51.717 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:51 +0000 (0:00:00.052) 0:01:48.468 ****** 2020-01-16T00:08:51.722 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-01-16T00:08:51.723 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:51.769 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:51 +0000 (0:00:00.046) 0:01:48.515 ****** TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:51.817 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:51 +0000 (0:00:00.053) 0:01:48.568 ****** 2020-01-16T00:08:51.842 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-01-16T00:08:51.844 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:51.882 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:51 +0000 (0:00:00.064) 0:01:48.633 ****** 2020-01-16T00:08:51.897 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-01-16T00:08:51.898 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2020-01-16T00:08:51.943 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:51 +0000 (0:00:00.061) 0:01:48.695 ****** 2020-01-16T00:08:51.948 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-01-16T00:08:51.950 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:51.989 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:51 +0000 (0:00:00.045) 0:01:48.740 ****** 2020-01-16T00:08:52.013 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-01-16T00:08:52.015 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:52.054 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:52 +0000 (0:00:00.065) 0:01:48.805 ****** 2020-01-16T00:08:52.079 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-01-16T00:08:52.081 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:52.120 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:52 +0000 (0:00:00.066) 0:01:48.871 ****** 2020-01-16T00:08:52.146 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-01-16T00:08:52.148 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:52.186 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:52 +0000 (0:00:00.065) 0:01:48.937 ****** 2020-01-16T00:08:52.211 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-01-16T00:08:52.213 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:52.255 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:52 +0000 (0:00:00.068) 0:01:49.006 ****** 2020-01-16T00:08:52.269 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-01-16T00:08:52.270 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2020-01-16T00:08:52.310 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:52 +0000 (0:00:00.054) 0:01:49.061 ****** 2020-01-16T00:08:52.317 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-01-16T00:08:52.319 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:52.360 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:52 +0000 (0:00:00.048) 0:01:49.109 ****** 2020-01-16T00:08:52.362 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:52.407 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:52 +0000 (0:00:00.049) 0:01:49.158 ****** 2020-01-16T00:08:52.506 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-01-16T00:08:52.507 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2020-01-16T00:08:52.546 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:52 +0000 (0:00:00.138) 0:01:49.297 ****** 2020-01-16T00:08:52.741 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-01-16T00:08:52.742 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2020-01-16T00:08:52.782 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:52 +0000 (0:00:00.236) 0:01:49.534 ****** 2020-01-16T00:08:52.872 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-01-16T00:08:52.873 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2020-01-16T00:08:52.912 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:52 +0000 (0:00:00.129) 0:01:49.663 ****** 2020-01-16T00:08:52.938 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-01-16T00:08:52.939 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:52.977 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:52 +0000 (0:00:00.064) 0:01:49.728 ****** 2020-01-16T00:08:53.097 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* ok: [smithi123.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2020-01-16T00:08:53.139 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:53 +0000 (0:00:00.161) 0:01:49.890 ****** 2020-01-16T00:08:53.322 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-01-16T00:08:53.323 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2020-01-16T00:08:53.363 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:53 +0000 (0:00:00.223) 0:01:50.114 ****** 2020-01-16T00:08:53.451 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-01-16T00:08:53.452 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2020-01-16T00:08:53.494 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:53 +0000 (0:00:00.131) 0:01:50.246 ****** 2020-01-16T00:08:53.635 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-01-16T00:08:53.636 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-01-16T00:08:53.693 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:53 +0000 (0:00:00.199) 0:01:50.445 ****** 2020-01-16T00:08:53.699 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2020-01-16T00:08:53.700 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2020-01-16T00:08:53.758 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:53 +0000 (0:00:00.064) 0:01:50.509 ****** 2020-01-16T00:08:53.763 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-01-16T00:08:53.764 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:53.804 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:53 +0000 (0:00:00.045) 0:01:50.555 ****** 2020-01-16T00:08:53.829 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:53.870 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:53 +0000 (0:00:00.066) 0:01:50.621 ****** 2020-01-16T00:08:53.896 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-01-16T00:08:53.897 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:53.935 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:53 +0000 (0:00:00.065) 0:01:50.687 ****** 2020-01-16T00:08:53.963 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-01-16T00:08:53.963 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:54.001 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:53 +0000 (0:00:00.066) 0:01:50.753 ****** 2020-01-16T00:08:54.028 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-01-16T00:08:54.029 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:54.068 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:54 +0000 (0:00:00.066) 0:01:50.820 ****** 2020-01-16T00:08:54.084 INFO:teuthology.task.ansible.out: 2020-01-16T00:08:54.086 INFO:teuthology.task.ansible.out:TASK [testnode : Enable kernel modules to load at boot time.] ****************** ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2020-01-16T00:08:54.125 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:54 +0000 (0:00:00.056) 0:01:50.877 ****** 2020-01-16T00:08:54.127 INFO:teuthology.task.ansible.out: 2020-01-16T00:08:54.131 INFO:teuthology.task.ansible.out: 2020-01-16T00:08:54.133 INFO:teuthology.task.ansible.out:TASK [testnode : set_fact] ***************************************************** ok: [smithi123.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-01-16T00:08:54.178 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:54 +0000 (0:00:00.053) 0:01:50.930 ****** 2020-01-16T00:08:54.180 INFO:teuthology.task.ansible.out: 2020-01-16T00:08:54.227 INFO:teuthology.task.ansible.out: 2020-01-16T00:08:54.229 INFO:teuthology.task.ansible.out:TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi123.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-01-16T00:08:54.240 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2020-01-16T00:08:54.242 INFO:teuthology.task.ansible.out: skipping: [smithi123.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2020-01-16T00:08:54.252 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2020-01-16T00:08:54.254 INFO:teuthology.task.ansible.out: 2020-01-16T00:08:54.263 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2020-01-16T00:08:54.265 INFO:teuthology.task.ansible.out: 2020-01-16T00:08:54.275 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2020-01-16T00:08:54.277 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** ok: [smithi145.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-01-16T00:08:54.317 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:54 +0000 (0:00:00.138) 0:01:51.069 ****** 2020-01-16T00:08:54.319 INFO:teuthology.task.ansible.out: 2020-01-16T00:08:54.325 INFO:teuthology.task.ansible.out: 2020-01-16T00:08:54.326 INFO:teuthology.task.ansible.out:TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi123.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2020-01-16T00:08:54.328 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} skipping: [smithi123.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2020-01-16T00:08:54.329 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-01-16T00:08:54.330 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} skipping: [smithi123.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-01-16T00:08:54.342 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-01-16T00:08:54.343 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Check for NVMe drive] ***************************************** 2020-01-16T00:08:54.344 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-01-16T00:08:54.383 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:54 +0000 (0:00:00.065) 0:01:51.134 ****** 2020-01-16T00:08:54.384 INFO:teuthology.task.ansible.out: 2020-01-16T00:08:54.389 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-01-16T00:08:54.391 INFO:teuthology.task.ansible.out: 2020-01-16T00:08:54.438 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:54 +0000 (0:00:00.055) 0:01:51.190 ****** 2020-01-16T00:08:54.445 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-01-16T00:08:54.446 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:54.486 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:54 +0000 (0:00:00.048) 0:01:51.238 ****** 2020-01-16T00:08:54.513 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-01-16T00:08:54.514 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:54.560 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:54 +0000 (0:00:00.073) 0:01:51.312 ****** 2020-01-16T00:08:54.565 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-01-16T00:08:54.566 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003066", "end": "2020-01-16 00:08:54.547448", "rc": 0, "start": "2020-01-16 00:08:54.544382", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-01-16T00:08:54.609 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:54 +0000 (0:00:00.048) 0:01:51.361 ****** 2020-01-16T00:08:54.615 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-01-16T00:08:54.616 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:54.655 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:54 +0000 (0:00:00.046) 0:01:51.407 ****** 2020-01-16T00:08:54.661 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-01-16T00:08:54.661 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:54.705 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:54 +0000 (0:00:00.049) 0:01:51.457 ****** 2020-01-16T00:08:54.710 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-01-16T00:08:54.710 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:54.752 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:54 +0000 (0:00:00.047) 0:01:51.504 ****** 2020-01-16T00:08:54.757 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-01-16T00:08:54.758 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:54.798 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:54 +0000 (0:00:00.045) 0:01:51.550 ****** 2020-01-16T00:08:54.803 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-01-16T00:08:54.804 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-01-16T00:08:54.844 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:54 +0000 (0:00:00.045) 0:01:51.595 ****** 2020-01-16T00:08:54.848 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-01-16T00:08:54.850 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:54.888 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:54 +0000 (0:00:00.044) 0:01:51.640 ****** 2020-01-16T00:08:54.894 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-01-16T00:08:54.894 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-01-16T00:08:54.896 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2020-01-16T00:08:54.906 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2020-01-16T00:08:54.936 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2020-01-16T00:08:54.937 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2020-01-16T00:08:54.939 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2020-01-16T00:08:54.939 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2020-01-16T00:08:54.950 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2020-01-16T00:08:54.950 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2020-01-16T00:08:54.971 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-01-16T00:08:54.972 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-01-16T00:08:54.983 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-01-16T00:08:55.004 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-01-16T00:08:55.005 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-01-16T00:08:55.006 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-01-16T00:08:55.049 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:55 +0000 (0:00:00.160) 0:01:51.801 ****** 2020-01-16T00:08:55.173 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-01-16T00:08:55.175 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.002849", "end": "2020-01-16 00:08:55.157440", "rc": 0, "start": "2020-01-16 00:08:55.154591", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-01-16T00:08:55.216 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:55 +0000 (0:00:00.166) 0:01:51.967 ****** 2020-01-16T00:08:55.251 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-01-16T00:08:55.253 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:55.302 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:55 +0000 (0:00:00.086) 0:01:52.054 ****** 2020-01-16T00:08:55.328 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-01-16T00:08:55.330 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:55.369 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:55 +0000 (0:00:00.066) 0:01:52.120 ****** 2020-01-16T00:08:55.394 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-01-16T00:08:55.396 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:55.435 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:55 +0000 (0:00:00.065) 0:01:52.186 ****** 2020-01-16T00:08:55.761 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-01-16T00:08:55.762 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"cache_update_time": 1579133010, "cache_updated": false, "changed": false} 2020-01-16T00:08:55.802 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:55 +0000 (0:00:00.367) 0:01:52.554 ****** 2020-01-16T00:08:55.828 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-01-16T00:08:55.829 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:55.867 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:55 +0000 (0:00:00.064) 0:01:52.619 ****** 2020-01-16T00:08:55.904 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-01-16T00:08:55.905 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item={u'block_used': 13723482, u'uuid': u'831992a6-90a2-4452-b219-e8f28acb0e82', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 226371626, u'size_available': 927218180096, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60905894, u'device': u'/dev/sda1', u'inode_used': 149082, u'block_size': 4096, u'options': u'rw,relatime,errors=remount-ro,data=ordered'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226371626, "block_size": 4096, "block_total": 240095108, "block_used": 13723482, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905894, "inode_total": 61054976, "inode_used": 149082, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927218180096, "size_total": 983429562368, "uuid": "831992a6-90a2-4452-b219-e8f28acb0e82"}} 2020-01-16T00:08:55.947 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:55 +0000 (0:00:00.079) 0:01:52.698 ****** 2020-01-16T00:08:56.076 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-01-16T00:08:56.078 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.007821", "end": "2020-01-16 00:08:56.287713", "rc": 0, "start": "2020-01-16 00:08:56.279892", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-01-16T00:08:56.116 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:56 +0000 (0:00:00.169) 0:01:52.867 ****** 2020-01-16T00:08:56.152 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-01-16T00:08:56.153 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'block_used': 13723482, u'uuid': u'831992a6-90a2-4452-b219-e8f28acb0e82', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 226371626, u'size_available': 927218180096, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60905894, u'device': u'/dev/sda1', u'inode_used': 149082, u'block_size': 4096, u'options': u'rw,relatime,errors=remount-ro,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226371626, "block_size": 4096, "block_total": 240095108, "block_used": 13723482, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905894, "inode_total": 61054976, "inode_used": 149082, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927218180096, "size_total": 983429562368, "uuid": "831992a6-90a2-4452-b219-e8f28acb0e82"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:56.195 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:56 +0000 (0:00:00.078) 0:01:52.946 ****** 2020-01-16T00:08:56.220 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-01-16T00:08:56.259 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:56 +0000 (0:00:00.064) 0:01:53.010 ****** 2020-01-16T00:08:56.306 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-01-16T00:08:56.308 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5L6SZ', u'wwn-0x5000c500918d7a6f'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5L6SZ', u'wwn': u'0x5000c500918d7a6f', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'cfq', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'831992a6-90a2-4452-b219-e8f28acb0e82', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5L6SZ-part1', u'wwn-0x5000c500918d7a6f-part1'], u'uuids': [u'831992a6-90a2-4452-b219-e8f28acb0e82']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5L6SZ", "wwn-0x5000c500918d7a6f"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5L6SZ-part1", "wwn-0x5000c500918d7a6f-part1"], "labels": [], "masters": [], "uuids": ["831992a6-90a2-4452-b219-e8f28acb0e82"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "831992a6-90a2-4452-b219-e8f28acb0e82"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5L6SZ", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "vir 2020-01-16T00:08:56.310 INFO:teuthology.task.ansible.out:tual": 1, "wwn": "0x5000c500918d7a6f"}}, "skip_reason": "Conditional result was False"} TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-01-16T00:08:56.311 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"cache_update_time": 1579133053, "cache_updated": false, "changed": false} 2020-01-16T00:08:56.355 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:56 +0000 (0:00:00.095) 0:01:53.106 ****** 2020-01-16T00:08:56.360 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-01-16T00:08:56.363 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:56.364 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:56.366 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:56.368 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:56.370 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:56.381 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-01-16T00:08:56.383 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Zap all non-root disks] *************************************** skipping: [smithi123.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:56.421 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:56 +0000 (0:00:00.065) 0:01:53.172 ****** 2020-01-16T00:08:56.427 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:56.428 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:56.461 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-01-16T00:08:56.462 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={u'block_used': 13723478, u'uuid': u'831992a6-90a2-4452-b219-e8f28acb0e82', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 226371630, u'size_available': 927218196480, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60905894, u'device': u'/dev/sda1', u'inode_used': 149082, u'block_size': 4096, u'options': u'rw,relatime,errors=remount-ro,data=ordered'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226371630, "block_size": 4096, "block_total": 240095108, "block_used": 13723478, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905894, "inode_total": 61054976, "inode_used": 149082, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927218196480, "size_total": 983429562368, "uuid": "831992a6-90a2-4452-b219-e8f28acb0e82"}} 2020-01-16T00:08:56.517 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:56 +0000 (0:00:00.096) 0:01:53.269 ****** 2020-01-16T00:08:56.647 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-01-16T00:08:56.649 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009822", "end": "2020-01-16 00:08:56.695371", "rc": 0, "start": "2020-01-16 00:08:56.685549", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-01-16T00:08:56.692 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:56 +0000 (0:00:00.174) 0:01:53.444 ****** 2020-01-16T00:08:56.738 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-01-16T00:08:56.739 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'block_used': 13723478, u'uuid': u'831992a6-90a2-4452-b219-e8f28acb0e82', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 226371630, u'size_available': 927218196480, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60905894, u'device': u'/dev/sda1', u'inode_used': 149082, u'block_size': 4096, u'options': u'rw,relatime,errors=remount-ro,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226371630, "block_size": 4096, "block_total": 240095108, "block_used": 13723478, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905894, "inode_total": 61054976, "inode_used": 149082, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927218196480, "size_total": 983429562368, "uuid": "831992a6-90a2-4452-b219-e8f28acb0e82"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:56.783 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:56 +0000 (0:00:00.090) 0:01:53.535 ****** 2020-01-16T00:08:56.809 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-01-16T00:08:56.849 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:08:56 +0000 (0:00:00.065) 0:01:53.600 ****** 2020-01-16T00:08:56.896 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-01-16T00:08:56.897 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XLDP', u'wwn-0x5000c5009293fa4d'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5XLDP', u'wwn': u'0x5000c5009293fa4d', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'cfq', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'831992a6-90a2-4452-b219-e8f28acb0e82', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XLDP-part1', u'wwn-0x5000c5009293fa4d-part1'], u'uuids': [u'831992a6-90a2-4452-b219-e8f28acb0e82']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XLDP", "wwn-0x5000c5009293fa4d"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XLDP-part1", "wwn-0x5000c5009293fa4d-part1"], "labels": [], "masters": [], "uuids": ["831992a6-90a2-4452-b219-e8f28acb0e82"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "831992a6-90a2-4452-b219-e8f28acb0e82"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XLDP", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "vir 2020-01-16T00:08:56.898 INFO:teuthology.task.ansible.out:tual": 1, "wwn": "0x5000c5009293fa4d"}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:56.909 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:56.921 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:56.933 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:56.945 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:56.947 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:56.959 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:56.970 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:56.982 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:08:58.269 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-fkpFhk-mpaB-s58D-0hlG-rzym-j2s3-BEwDWS', u'nvme-INTEL_SSDPEDMD400G4_PHFT620400YK400BGN', u'nvme-nvme.8086-50484654363230343030594b34303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.737034", "end": "2020-01-16 00:08:58.460194", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-fkpFhk-mpaB-s58D-0hlG-rzym-j2s3-BEwDWS", "nvme-INTEL_SSDPEDMD400G4_PHFT620400YK400BGN", "nvme-nvme.8086-50484654363230343030594b34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-01-16 00:08:56.723160", "stderr": "", "stderr_lines": [], "stdout": "Creating new GP 2020-01-16T00:08:58.271 INFO:teuthology.task.ansible.out:T entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-01-16T00:08:59.414 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-2ztsNE-wFd3-fuSV-7wWt-glYZ-uRES-B9heb0', u'nvme-INTEL_SSDPEDMD400G4_PHFT620500N9400BGN', u'nvme-nvme.8086-504846543632303530304e3934303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:02.319537", "end": "2020-01-16 00:08:59.456227", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-2ztsNE-wFd3-fuSV-7wWt-glYZ-uRES-B9heb0", "nvme-INTEL_SSDPEDMD400G4_PHFT620500N9400BGN", "nvme-nvme.8086-504846543632303530304e3934303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-01-16 00:08:57.136690", "stderr": "", "stderr_lines": [], "stdout": "Creating new GP 2020-01-16T00:08:59.416 INFO:teuthology.task.ansible.out:T entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-01-16T00:08:59.519 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-e7gHGppcfRS2gGx0eTSk3NLhU1tpdc3QWjoyjZasojjQxzafalGEF4OtFdVazCSC'], u'uuids': [u'fa1ff632-13cf-4bf1-8512-f8f430b5bc68']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.125586", "end": "2020-01-16 00:08:59.715729", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-e7gHGppcfRS2gGx0eTSk3NLhU1tpdc3QWjoyjZasojjQxzafalGEF4OtFdVazCSC"], "labels": [], "masters": [], "uuids": ["fa1ff632-13cf-4bf1-8512-f8f430b5bc68"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-01-16 00:08:58.590143", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-01-16T00:09:00.672 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-FAZ6JSmsjcwGRZVRKTjjxt6ztpvZTWUfXarDgJSG5KsuT12ogWAVXEQcO889rweJ'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.117669", "end": "2020-01-16 00:09:00.700388", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-FAZ6JSmsjcwGRZVRKTjjxt6ztpvZTWUfXarDgJSG5KsuT12ogWAVXEQcO889rweJ"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-01-16 00:08:59.582719", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-01-16T00:09:00.737 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-e7gHGppcfRS2gGx0eTSk3NLhU1tpdc3QpMnIzjEzZhK7pVrzJByUsoZgzdQ15gXh'], u'uuids': [u'9d22c34c-c366-4816-b4fd-f53b989a37b9']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.105529", "end": "2020-01-16 00:09:00.941555", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-e7gHGppcfRS2gGx0eTSk3NLhU1tpdc3QpMnIzjEzZhK7pVrzJByUsoZgzdQ15gXh"], "labels": [], "masters": [], "uuids": ["9d22c34c-c366-4816-b4fd-f53b989a37b9"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-01-16 00:08:59.836026", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-01-16T00:09:01.906 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-FAZ6JSmsjcwGRZVRKTjjxt6ztpvZTWUfPzbKXiKJVp1mGm1hyXYVWoNr64BsWlf8'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.101887", "end": "2020-01-16 00:09:01.933007", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-FAZ6JSmsjcwGRZVRKTjjxt6ztpvZTWUfPzbKXiKJVp1mGm1hyXYVWoNr64BsWlf8"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-01-16 00:09:00.831120", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-01-16T00:09:01.974 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-e7gHGppcfRS2gGx0eTSk3NLhU1tpdc3Qu1wOe2vuZGLnSSJy1lYsf9WSw5wmQkHq'], u'uuids': [u'c5ef2306-b1ed-4c6a-ad0d-882587056fcf']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.111876", "end": "2020-01-16 00:09:02.174890", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-e7gHGppcfRS2gGx0eTSk3NLhU1tpdc3Qu1wOe2vuZGLnSSJy1lYsf9WSw5wmQkHq"], "labels": [], "masters": [], "uuids": ["c5ef2306-b1ed-4c6a-ad0d-882587056fcf"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-01-16 00:09:01.063014", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-01-16T00:09:03.115 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-FAZ6JSmsjcwGRZVRKTjjxt6ztpvZTWUfRtap4piWononWNovT21Vw1iRZGqDGibD'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.113349", "end": "2020-01-16 00:09:03.157939", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-FAZ6JSmsjcwGRZVRKTjjxt6ztpvZTWUfRtap4piWononWNovT21Vw1iRZGqDGibD"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-01-16 00:09:02.044590", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-01-16T00:09:03.219 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-e7gHGppcfRS2gGx0eTSk3NLhU1tpdc3Q55kM1ncux8IBdSNoYFyOOtnEggjfFAGY'], u'uuids': [u'43deedda-6031-43d4-9e8a-d80239c1d37c']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.117886", "end": "2020-01-16 00:09:03.416040", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-e7gHGppcfRS2gGx0eTSk3NLhU1tpdc3Q55kM1ncux8IBdSNoYFyOOtnEggjfFAGY"], "labels": [], "masters": [], "uuids": ["43deedda-6031-43d4-9e8a-d80239c1d37c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-01-16 00:09:02.298154", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-01-16T00:09:04.339 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-FAZ6JSmsjcwGRZVRKTjjxt6ztpvZTWUfv3Rnra3CLeH5Y2dNmKdpAHvJPh4Ih4u3'], u'uuids': [u'eb9f8eef-b58e-425c-9c70-1535ebc25d5b']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.099541", "end": "2020-01-16 00:09:04.365954", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-FAZ6JSmsjcwGRZVRKTjjxt6ztpvZTWUfv3Rnra3CLeH5Y2dNmKdpAHvJPh4Ih4u3"], "labels": [], "masters": [], "uuids": ["eb9f8eef-b58e-425c-9c70-1535ebc25d5b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-01-16 00:09:03.266413", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-01-16T00:09:04.444 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-e7gHGppcfRS2gGx0eTSk3NLhU1tpdc3Q2lR5DnlUl1cv26sbL4swSJ1dos2CeiUf'], u'uuids': [u'f5e04456-cb1e-4458-8c12-799d6e427123']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.105273", "end": "2020-01-16 00:09:04.647449", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-e7gHGppcfRS2gGx0eTSk3NLhU1tpdc3Q2lR5DnlUl1cv26sbL4swSJ1dos2CeiUf"], "labels": [], "masters": [], "uuids": ["f5e04456-cb1e-4458-8c12-799d6e427123"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-01-16 00:09:03.542176", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-01-16T00:09:04.506 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:04 +0000 (0:00:07.657) 0:02:01.258 ****** 2020-01-16T00:09:04.554 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-01-16T00:09:04.555 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5L6SZ', u'wwn-0x5000c500918d7a6f'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5L6SZ', u'wwn': u'0x5000c500918d7a6f', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'cfq', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'831992a6-90a2-4452-b219-e8f28acb0e82', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5L6SZ-part1', u'wwn-0x5000c500918d7a6f-part1'], u'uuids': [u'831992a6-90a2-4452-b219-e8f28acb0e82']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5L6SZ", "wwn-0x5000c500918d7a6f"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5L6SZ-part1", "wwn-0x5000c500918d7a6f-part1"], "labels": [], "masters": [], "uuids": ["831992a6-90a2-4452-b219-e8f28acb0e82"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "831992a6-90a2-4452-b219-e8f28acb0e82"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5L6SZ", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "vir 2020-01-16T00:09:04.557 INFO:teuthology.task.ansible.out:tual": 1, "wwn": "0x5000c500918d7a6f"}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:04.557 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:04.569 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:04.582 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:04.583 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:04.595 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:04.607 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:04.619 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:04.620 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:04.788 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-fkpFhk-mpaB-s58D-0hlG-rzym-j2s3-BEwDWS', u'nvme-INTEL_SSDPEDMD400G4_PHFT620400YK400BGN', u'nvme-nvme.8086-50484654363230343030594b34303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.048760", "end": "2020-01-16 00:09:04.983358", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-fkpFhk-mpaB-s58D-0hlG-rzym-j2s3-BEwDWS", "nvme-INTEL_SSDPEDMD400G4_PHFT620400YK400BGN", "nvme-nvme.8086-50484654363230343030594b34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-01-16 00:09:04.934598", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 11 2020-01-16T00:09:04.790 INFO:teuthology.task.ansible.out:0 MiB) copied, 0.0329443 s, 3.5 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0329443 s, 3.5 GB/s"], "stdout": "", "stdout_lines": []} 2020-01-16T00:09:05.050 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-e7gHGppcfRS2gGx0eTSk3NLhU1tpdc3QWjoyjZasojjQxzafalGEF4OtFdVazCSC'], u'uuids': [u'fa1ff632-13cf-4bf1-8512-f8f430b5bc68']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.152096", "end": "2020-01-16 00:09:05.256133", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-e7gHGppcfRS2gGx0eTSk3NLhU1tpdc3QWjoyjZasojjQxzafalGEF4OtFdVazCSC"], "labels": [], "masters": [], "uuids": ["fa1ff632-13cf-4bf1-8512-f8f430b5bc68"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-01-16 00:09:05.104037", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.147901 s, 780 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.147901 s, 780 MB/s"], "stdout": "", "stdout_lines": []} 2020-01-16T00:09:05.321 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-e7gHGppcfRS2gGx0eTSk3NLhU1tpdc3QpMnIzjEzZhK7pVrzJByUsoZgzdQ15gXh'], u'uuids': [u'9d22c34c-c366-4816-b4fd-f53b989a37b9']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.157907", "end": "2020-01-16 00:09:05.514146", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-e7gHGppcfRS2gGx0eTSk3NLhU1tpdc3QpMnIzjEzZhK7pVrzJByUsoZgzdQ15gXh"], "labels": [], "masters": [], "uuids": ["9d22c34c-c366-4816-b4fd-f53b989a37b9"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-01-16 00:09:05.356239", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.153582 s, 751 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.153582 s, 751 MB/s"], "stdout": "", "stdout_lines": []} 2020-01-16T00:09:05.591 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-01-16T00:09:05.592 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-FAZ6JSmsjcwGRZVRKTjjxt6ztpvZTWUfuN8Kbd8c0UgKk4tkfA6h5pq7hbboDnQ1'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.138049", "end": "2020-01-16 00:09:05.624373", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-FAZ6JSmsjcwGRZVRKTjjxt6ztpvZTWUfuN8Kbd8c0UgKk4tkfA6h5pq7hbboDnQ1"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-01-16 00:09:04.486324", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-01-16T00:09:05.608 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-01-16T00:09:05.608 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-e7gHGppcfRS2gGx0eTSk3NLhU1tpdc3Qu1wOe2vuZGLnSSJy1lYsf9WSw5wmQkHq'], u'uuids': [u'c5ef2306-b1ed-4c6a-ad0d-882587056fcf']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.154163", "end": "2020-01-16 00:09:05.804447", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-e7gHGppcfRS2gGx0eTSk3NLhU1tpdc3Qu1wOe2vuZGLnSSJy1lYsf9WSw5wmQkHq"], "labels": [], "masters": [], "uuids": ["c5ef2306-b1ed-4c6a-ad0d-882587056fcf"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-01-16 00:09:05.650284", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.149958 s, 769 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.149958 s, 769 MB/s"], "stdout": "", "stdout_lines": []} 2020-01-16T00:09:05.646 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:05 +0000 (0:00:01.140) 0:02:02.398 ****** 2020-01-16T00:09:05.684 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XLDP', u'wwn-0x5000c5009293fa4d'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5XLDP', u'wwn': u'0x5000c5009293fa4d', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'cfq', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'831992a6-90a2-4452-b219-e8f28acb0e82', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XLDP-part1', u'wwn-0x5000c5009293fa4d-part1'], u'uuids': [u'831992a6-90a2-4452-b219-e8f28acb0e82']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XLDP", "wwn-0x5000c5009293fa4d"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XLDP-part1", "wwn-0x5000c5009293fa4d-part1"], "labels": [], "masters": [], "uuids": ["831992a6-90a2-4452-b219-e8f28acb0e82"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "831992a6-90a2-4452-b219-e8f28acb0e82"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XLDP", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "vir 2020-01-16T00:09:05.686 INFO:teuthology.task.ansible.out:tual": 1, "wwn": "0x5000c5009293fa4d"}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:05.696 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:05.708 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:05.710 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:05.721 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:05.733 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:05.735 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:05.747 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:05.758 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'0', u'model': None, u'size': u'0.00 Bytes', u'scheduler_mode': u'none', u'rotational': u'1', u'sectors': u'0', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'loop4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:05.886 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-e7gHGppcfRS2gGx0eTSk3NLhU1tpdc3Q55kM1ncux8IBdSNoYFyOOtnEggjfFAGY'], u'uuids': [u'43deedda-6031-43d4-9e8a-d80239c1d37c']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.149109", "end": "2020-01-16 00:09:06.083532", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-e7gHGppcfRS2gGx0eTSk3NLhU1tpdc3Q55kM1ncux8IBdSNoYFyOOtnEggjfFAGY"], "labels": [], "masters": [], "uuids": ["43deedda-6031-43d4-9e8a-d80239c1d37c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-01-16 00:09:05.934423", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.144885 s, 796 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.144885 s, 796 MB/s"], "stdout": "", "stdout_lines": []} 2020-01-16T00:09:05.930 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-2ztsNE-wFd3-fuSV-7wWt-glYZ-uRES-B9heb0', u'nvme-INTEL_SSDPEDMD400G4_PHFT620500N9400BGN', u'nvme-nvme.8086-504846543632303530304e3934303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.053951", "end": "2020-01-16 00:09:05.957026", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-2ztsNE-wFd3-fuSV-7wWt-glYZ-uRES-B9heb0", "nvme-INTEL_SSDPEDMD400G4_PHFT620500N9400BGN", "nvme-nvme.8086-504846543632303530304e3934303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-01-16 00:09:05.903075", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 11 2020-01-16T00:09:05.932 INFO:teuthology.task.ansible.out:0 MiB) copied, 0.0330205 s, 3.5 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0330205 s, 3.5 GB/s"], "stdout": "", "stdout_lines": []} 2020-01-16T00:09:06.161 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-e7gHGppcfRS2gGx0eTSk3NLhU1tpdc3Q2lR5DnlUl1cv26sbL4swSJ1dos2CeiUf'], u'uuids': [u'f5e04456-cb1e-4458-8c12-799d6e427123']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.150389", "end": "2020-01-16 00:09:06.352475", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-e7gHGppcfRS2gGx0eTSk3NLhU1tpdc3Q2lR5DnlUl1cv26sbL4swSJ1dos2CeiUf"], "labels": [], "masters": [], "uuids": ["f5e04456-cb1e-4458-8c12-799d6e427123"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-01-16 00:09:06.202086", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.14792 s, 780 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.14792 s, 780 MB/s"], "stdout": "", "stdout_lines": []} 2020-01-16T00:09:06.216 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:06 +0000 (0:00:00.570) 0:02:02.968 ****** 2020-01-16T00:09:06.223 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-FAZ6JSmsjcwGRZVRKTjjxt6ztpvZTWUfXarDgJSG5KsuT12ogWAVXEQcO889rweJ'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.172266", "end": "2020-01-16 00:09:06.241982", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-FAZ6JSmsjcwGRZVRKTjjxt6ztpvZTWUfXarDgJSG5KsuT12ogWAVXEQcO889rweJ"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-01-16 00:09:06.069716", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.168515 s, 684 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.168515 s, 684 MB/s"], "stdout": "", "stdout_lines": []} 2020-01-16T00:09:06.476 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-01-16T00:09:06.477 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.123926", "end": "2020-01-16 00:09:06.671865", "rc": 0, "start": "2020-01-16 00:09:06.547939", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-01-16T00:09:06.516 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:06 +0000 (0:00:00.299) 0:02:03.268 ****** 2020-01-16T00:09:06.522 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-01-16T00:09:06.524 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-FAZ6JSmsjcwGRZVRKTjjxt6ztpvZTWUfPzbKXiKJVp1mGm1hyXYVWoNr64BsWlf8'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.158161", "end": "2020-01-16 00:09:06.520752", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-FAZ6JSmsjcwGRZVRKTjjxt6ztpvZTWUfPzbKXiKJVp1mGm1hyXYVWoNr64BsWlf8"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-01-16 00:09:06.362591", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.153895 s, 749 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.153895 s, 749 MB/s"], "stdout": "", "stdout_lines": []} 2020-01-16T00:09:06.526 INFO:teuthology.task.ansible.out: 2020-01-16T00:09:06.712 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-01-16T00:09:06.713 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.057495", "end": "2020-01-16 00:09:06.916796", "rc": 0, "start": "2020-01-16 00:09:06.859301", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-01-16T00:09:06.755 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:06 +0000 (0:00:00.238) 0:02:03.506 ****** 2020-01-16T00:09:06.761 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-01-16T00:09:06.763 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-FAZ6JSmsjcwGRZVRKTjjxt6ztpvZTWUfRtap4piWononWNovT21Vw1iRZGqDGibD'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.151677", "end": "2020-01-16 00:09:06.790443", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-FAZ6JSmsjcwGRZVRKTjjxt6ztpvZTWUfRtap4piWononWNovT21Vw1iRZGqDGibD"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-01-16 00:09:06.638766", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.149402 s, 772 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.149402 s, 772 MB/s"], "stdout": "", "stdout_lines": []} 2020-01-16T00:09:06.776 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-01-16T00:09:06.817 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:06 +0000 (0:00:00.062) 0:02:03.569 ****** 2020-01-16T00:09:06.844 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-01-16T00:09:06.845 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:06.888 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:06 +0000 (0:00:00.070) 0:02:03.639 ****** 2020-01-16T00:09:06.914 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-01-16T00:09:06.915 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:06.955 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:06 +0000 (0:00:00.066) 0:02:03.706 ****** 2020-01-16T00:09:06.981 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-01-16T00:09:06.982 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:07.026 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:07 +0000 (0:00:00.070) 0:02:03.777 ****** 2020-01-16T00:09:07.032 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-01-16T00:09:07.041 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-FAZ6JSmsjcwGRZVRKTjjxt6ztpvZTWUfv3Rnra3CLeH5Y2dNmKdpAHvJPh4Ih4u3'], u'uuids': [u'eb9f8eef-b58e-425c-9c70-1535ebc25d5b']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.146475", "end": "2020-01-16 00:09:07.061832", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-FAZ6JSmsjcwGRZVRKTjjxt6ztpvZTWUfv3Rnra3CLeH5Y2dNmKdpAHvJPh4Ih4u3"], "labels": [], "masters": [], "uuids": ["eb9f8eef-b58e-425c-9c70-1535ebc25d5b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-01-16 00:09:06.915357", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.143579 s, 803 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.143579 s, 803 MB/s"], "stdout": "", "stdout_lines": []} 2020-01-16T00:09:07.053 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-01-16T00:09:07.054 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:07.092 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:07 +0000 (0:00:00.066) 0:02:03.843 ****** 2020-01-16T00:09:07.212 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-01-16T00:09:07.213 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-01-16T00:09:07.255 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:07 +0000 (0:00:00.163) 0:02:04.006 ****** 2020-01-16T00:09:07.304 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-01-16T00:09:07.306 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={u'block_used': 13723482, u'uuid': u'831992a6-90a2-4452-b219-e8f28acb0e82', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 226371626, u'size_available': 927218180096, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60905894, u'device': u'/dev/sda1', u'inode_used': 149082, u'block_size': 4096, u'options': u'rw,relatime,errors=remount-ro,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226371626, "block_size": 4096, "block_total": 240095108, "block_used": 13723482, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905894, "inode_total": 61054976, "inode_used": 149082, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927218180096, "size_total": 983429562368, "uuid": "831992a6-90a2-4452-b219-e8f28acb0e82"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:07.353 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:07 +0000 (0:00:00.098) 0:02:04.105 ****** 2020-01-16T00:09:07.359 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-01-16T00:09:07.361 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-FAZ6JSmsjcwGRZVRKTjjxt6ztpvZTWUfuN8Kbd8c0UgKk4tkfA6h5pq7hbboDnQ1'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.158309", "end": "2020-01-16 00:09:07.348369", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-FAZ6JSmsjcwGRZVRKTjjxt6ztpvZTWUfuN8Kbd8c0UgKk4tkfA6h5pq7hbboDnQ1"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-01-16 00:09:07.190060", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.15528 s, 743 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.15528 s, 743 MB/s"], "stdout": "", "stdout_lines": []} 2020-01-16T00:09:07.377 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-01-16T00:09:07.378 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:07.419 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:07 +0000 (0:00:00.065) 0:02:04.170 ****** 2020-01-16T00:09:07.469 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:07 +0000 (0:00:00.047) 0:02:04.218 ****** 2020-01-16T00:09:07.474 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-01-16T00:09:07.475 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:07.518 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:07 +0000 (0:00:00.050) 0:02:04.269 ****** 2020-01-16T00:09:07.544 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-01-16T00:09:07.544 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:07.585 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:07 +0000 (0:00:00.067) 0:02:04.336 ****** 2020-01-16T00:09:07.611 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-01-16T00:09:07.612 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:07.653 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:07 +0000 (0:00:00.068) 0:02:04.405 ****** 2020-01-16T00:09:07.690 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-01-16T00:09:07.691 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:07.732 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:07 +0000 (0:00:00.078) 0:02:04.484 ****** 2020-01-16T00:09:07.738 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-01-16T00:09:07.740 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.115707", "end": "2020-01-16 00:09:07.747935", "rc": 0, "start": "2020-01-16 00:09:07.632228", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-01-16T00:09:07.782 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:07 +0000 (0:00:00.049) 0:02:04.533 ****** 2020-01-16T00:09:07.787 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-01-16T00:09:07.788 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:07.831 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:07 +0000 (0:00:00.048) 0:02:04.581 ****** 2020-01-16T00:09:07.994 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-01-16T00:09:07.995 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.068992", "end": "2020-01-16 00:09:08.025652", "rc": 0, "start": "2020-01-16 00:09:07.956660", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-01-16T00:09:08.043 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:08 +0000 (0:00:00.213) 0:02:04.795 ****** 2020-01-16T00:09:08.070 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-01-16T00:09:08.118 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:08 +0000 (0:00:00.075) 0:02:04.870 ****** 2020-01-16T00:09:08.144 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-01-16T00:09:08.146 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:08.189 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:08 +0000 (0:00:00.070) 0:02:04.941 ****** 2020-01-16T00:09:08.216 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-01-16T00:09:08.218 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:08.284 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:08 +0000 (0:00:00.089) 0:02:05.030 ****** 2020-01-16T00:09:08.315 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-01-16T00:09:08.317 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:08.364 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:08 +0000 (0:00:00.085) 0:02:05.116 ****** 2020-01-16T00:09:08.390 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-01-16T00:09:08.392 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:08.433 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:08 +0000 (0:00:00.068) 0:02:05.185 ****** 2020-01-16T00:09:08.446 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** changed: [smithi123.front.sepia.ceph.com] => (item={'value': {u'pvs': u'/dev/nvme0n1'}, 'key': u'vg_nvme'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-01-16T00:09:08.498 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:08 +0000 (0:00:00.064) 0:02:05.249 ****** 2020-01-16T00:09:08.567 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-01-16T00:09:08.568 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-01-16T00:09:08.608 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:08 +0000 (0:00:00.110) 0:02:05.360 ****** 2020-01-16T00:09:08.645 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-01-16T00:09:08.646 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={u'block_used': 13723478, u'uuid': u'831992a6-90a2-4452-b219-e8f28acb0e82', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 226371630, u'size_available': 927218196480, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60905894, u'device': u'/dev/sda1', u'inode_used': 149082, u'block_size': 4096, u'options': u'rw,relatime,errors=remount-ro,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226371630, "block_size": 4096, "block_total": 240095108, "block_used": 13723478, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905894, "inode_total": 61054976, "inode_used": 149082, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927218196480, "size_total": 983429562368, "uuid": "831992a6-90a2-4452-b219-e8f28acb0e82"}, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:08.693 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:08 +0000 (0:00:00.084) 0:02:05.444 ****** 2020-01-16T00:09:08.729 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-01-16T00:09:08.731 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:08.799 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:08 +0000 (0:00:00.105) 0:02:05.550 ****** 2020-01-16T00:09:08.827 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-01-16T00:09:08.828 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:08.870 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:08 +0000 (0:00:00.070) 0:02:05.621 ****** 2020-01-16T00:09:08.871 INFO:teuthology.task.ansible.out: 2020-01-16T00:09:08.907 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-01-16T00:09:08.908 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:08.948 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:08 +0000 (0:00:00.077) 0:02:05.699 ****** 2020-01-16T00:09:08.973 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-01-16T00:09:08.975 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:09.014 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:09 +0000 (0:00:00.066) 0:02:05.765 ****** 2020-01-16T00:09:09.040 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-01-16T00:09:09.041 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} TASK [testnode : Determine desired logical volume percentage size] ************* skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:09.081 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:09 +0000 (0:00:00.066) 0:02:05.832 ****** 2020-01-16T00:09:09.110 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-01-16T00:09:09.111 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:09.150 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:09 +0000 (0:00:00.069) 0:02:05.902 ****** 2020-01-16T00:09:09.152 INFO:teuthology.task.ansible.out: 2020-01-16T00:09:09.281 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-01-16T00:09:09.282 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-01-16T00:09:09.449 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-01-16T00:09:09.451 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'pvs': u'/dev/nvme0n1'}, 'key': u'vg_nvme'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-01-16T00:09:09.494 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:09 +0000 (0:00:00.343) 0:02:06.245 ****** 2020-01-16T00:09:09.526 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-01-16T00:09:09.527 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-01-16T00:09:09.752 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-01-16T00:09:09.753 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-01-16T00:09:09.971 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-01-16T00:09:10.031 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:10 +0000 (0:00:00.531) 0:02:06.777 ****** 2020-01-16T00:09:10.035 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-01-16T00:09:10.167 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-01-16T00:09:10.168 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-01-16T00:09:10.221 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:10 +0000 (0:00:00.195) 0:02:06.973 ****** 2020-01-16T00:09:10.258 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-01-16T00:09:10.260 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} TASK [testnode : Write /scratch_devs] ****************************************** skipping: [smithi123.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:10.387 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-01-16T00:09:10.495 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-01-16T00:09:10.516 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-01-16T00:09:10.518 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-01-16T00:09:10.644 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-01-16T00:09:10.707 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-01-16T00:09:10.709 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-01-16T00:09:10.752 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:10 +0000 (0:00:00.530) 0:02:07.504 ****** 2020-01-16T00:09:10.778 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-01-16T00:09:10.779 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-01-16T00:09:10.823 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:10 +0000 (0:00:00.070) 0:02:07.575 ****** 2020-01-16T00:09:10.891 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-01-16T00:09:10.893 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-01-16T00:09:10.934 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:10 +0000 (0:00:00.111) 0:02:07.686 ****** 2020-01-16T00:09:10.953 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-01-16T00:09:10.954 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2020-01-16T00:09:10.998 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:10 +0000 (0:00:00.064) 0:02:07.750 ****** 2020-01-16T00:09:11.003 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-01-16T00:09:11.004 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:11.046 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-01-16T00:09:11.048 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-01-16T00:09:11.094 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:11 +0000 (0:00:00.096) 0:02:07.846 ****** 2020-01-16T00:09:11.104 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-01-16T00:09:11.105 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-01-16T00:09:11.222 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-01-16T00:09:11.338 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-01-16T00:09:11.457 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-01-16T00:09:11.508 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:11 +0000 (0:00:00.411) 0:02:08.258 ****** 2020-01-16T00:09:11.577 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2020-01-16T00:09:11.578 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => {"changed": true} 2020-01-16T00:09:11.640 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:11 +0000 (0:00:00.133) 0:02:08.392 ****** 2020-01-16T00:09:11.659 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-01-16T00:09:11.660 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2020-01-16T00:09:11.703 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:11 +0000 (0:00:00.062) 0:02:08.454 ****** 2020-01-16T00:09:11.739 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-01-16T00:09:11.741 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-01-16T00:09:11.781 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:11 +0000 (0:00:00.078) 0:02:08.533 ****** 2020-01-16T00:09:12.013 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2020-01-16T00:09:12.015 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true} 2020-01-16T00:09:12.055 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:12 +0000 (0:00:00.273) 0:02:08.806 ****** 2020-01-16T00:09:12.133 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2020-01-16T00:09:12.134 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-01-16T00:09:12.177 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:12 +0000 (0:00:00.122) 0:02:08.929 ****** 2020-01-16T00:09:12.256 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-01-16T00:09:12.326 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:12 +0000 (0:00:00.148) 0:02:09.078 ****** 2020-01-16T00:09:12.331 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-01-16T00:09:12.339 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"changed": false, "gid": 1205, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2020-01-16T00:09:12.372 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:12 +0000 (0:00:00.045) 0:02:09.123 ****** 2020-01-16T00:09:12.462 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "gid": 1205, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2020-01-16T00:09:12.502 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:12 +0000 (0:00:00.130) 0:02:09.254 ****** 2020-01-16T00:09:12.508 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2020-01-16T00:09:12.509 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2020-01-16T00:09:12.560 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:12 +0000 (0:00:00.054) 0:02:09.308 ****** 2020-01-16T00:09:12.646 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2020-01-16T00:09:12.692 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:12 +0000 (0:00:00.134) 0:02:09.443 ****** 2020-01-16T00:09:12.895 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2020-01-16T00:09:12.896 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1205, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2020-01-16T00:09:12.937 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:12 +0000 (0:00:00.245) 0:02:09.689 ****** 2020-01-16T00:09:13.027 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1205, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2020-01-16T00:09:13.074 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:13 +0000 (0:00:00.136) 0:02:09.825 ****** 2020-01-16T00:09:13.278 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-01-16T00:09:13.280 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2020-01-16T00:09:13.321 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:13 +0000 (0:00:00.247) 0:02:10.072 ****** 2020-01-16T00:09:13.411 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2020-01-16T00:09:13.454 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:13 +0000 (0:00:00.132) 0:02:10.205 ****** 2020-01-16T00:09:14.267 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2020-01-16T00:09:14.268 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"cache_update_time": 1579133010, "cache_updated": false, "changed": false} 2020-01-16T00:09:14.309 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:14 +0000 (0:00:00.855) 0:02:11.061 ****** 2020-01-16T00:09:14.399 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"cache_update_time": 1579133053, "cache_updated": false, "changed": false} 2020-01-16T00:09:14.439 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:14 +0000 (0:00:00.129) 0:02:11.191 ****** 2020-01-16T00:09:14.508 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-01-16T00:09:14.509 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.047571", "end": "2020-01-16 00:09:14.706394", "rc": 0, "start": "2020-01-16 00:09:14.658823", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2020-01-16T00:09:14.550 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:14 +0000 (0:00:00.111) 0:02:11.302 ****** 2020-01-16T00:09:14.598 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-01-16T00:09:14.599 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:14.639 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:14 +0000 (0:00:00.088) 0:02:11.391 ****** 2020-01-16T00:09:14.645 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-01-16T00:09:14.646 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.044900", "end": "2020-01-16 00:09:14.648857", "rc": 0, "start": "2020-01-16 00:09:14.603957", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2020-01-16T00:09:14.688 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:14 +0000 (0:00:00.048) 0:02:11.440 ****** 2020-01-16T00:09:14.694 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2020-01-16T00:09:14.695 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:14.742 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:14 +0000 (0:00:00.053) 0:02:11.494 ****** 2020-01-16T00:09:14.748 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-01-16T00:09:14.749 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:14.796 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:14 +0000 (0:00:00.053) 0:02:11.547 ****** 2020-01-16T00:09:14.835 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2020-01-16T00:09:14.837 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:14.880 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:14 +0000 (0:00:00.084) 0:02:11.632 ****** 2020-01-16T00:09:15.625 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-01-16T00:09:15.626 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"cache_update_time": 1579133010, "cache_updated": false, "changed": false} 2020-01-16T00:09:15.665 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:15 +0000 (0:00:00.784) 0:02:12.416 ****** 2020-01-16T00:09:15.691 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-01-16T00:09:15.693 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:15.732 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:15 +0000 (0:00:00.066) 0:02:12.483 ****** 2020-01-16T00:09:15.736 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-01-16T00:09:15.737 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"cache_update_time": 1579133053, "cache_updated": false, "changed": false} 2020-01-16T00:09:15.778 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:15 +0000 (0:00:00.045) 0:02:12.529 ****** 2020-01-16T00:09:15.803 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-01-16T00:09:15.805 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:15.842 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:15 +0000 (0:00:00.064) 0:02:12.594 ****** 2020-01-16T00:09:16.035 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-01-16T00:09:16.036 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2020-01-16T00:09:16.085 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:16 +0000 (0:00:00.242) 0:02:12.836 ****** 2020-01-16T00:09:16.112 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-01-16T00:09:16.113 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:16.151 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:16 +0000 (0:00:00.066) 0:02:12.902 ****** 2020-01-16T00:09:16.166 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-01-16T00:09:16.167 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2020-01-16T00:09:16.207 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:16 +0000 (0:00:00.056) 0:02:12.959 ****** 2020-01-16T00:09:16.234 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-01-16T00:09:16.235 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:16.274 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:16 +0000 (0:00:00.066) 0:02:13.026 ****** 2020-01-16T00:09:16.365 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-01-16T00:09:16.367 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2020-01-16 00:07:02 UTC", "ActiveEnterTimestampMonotonic": "25605532", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket -.mount sysinit.target network.target systemd-tmpfiles-setup.service system.slice basic.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2020-01-16 00:06:53 UTC", "AssertTimestampMonotonic": "17268069", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2020-01-16 00:06:53 UTC", "ConditionTimestampMonotonic": "17268068", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target systemd-timesyncd.service", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2020-01-16T00:09:16.368 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1962", "ExecMainStartTimestamp": "Thu 2020-01-16 00:07:02 UTC", "ExecMainStartTimestampMonotonic": "25605519", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2020-01-16 00:06:53 UTC", "InactiveExitTimestampMonotonic": "17268849", "InvocationID": "aad07a75d9944fb48525c0ae29770b40", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128031", "LimitNPROCSoft": "128031", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2020-01-16T00:09:16.370 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128031", "LimitSIGPENDINGSoft": "128031", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1962", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2020-01-16 00:07:02 UTC", "StateChangeTimestampMonoton 2020-01-16T00:09:16.372 INFO:teuthology.task.ansible.out:ic": "25605532", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2020-01-16 00:07:02 UTC", "WatchdogTimestampMonotonic": "25605531", "WatchdogUSec": "0"}} 2020-01-16T00:09:16.408 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:16 +0000 (0:00:00.133) 0:02:13.159 ****** 2020-01-16T00:09:16.434 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-01-16T00:09:16.435 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:16.475 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:16 +0000 (0:00:00.067) 0:02:13.227 ****** 2020-01-16T00:09:16.494 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-01-16T00:09:16.497 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2020-01-16 00:06:48 UTC", "ActiveEnterTimestampMonotonic": "26089251", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "network.target system.slice sysinit.target systemd-tmpfiles-setup.service systemd-journald.socket basic.target -.mount", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2020-01-16 00:06:40 UTC", "AssertTimestampMonotonic": "17800059", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2020-01-16 00:06:40 UTC", "ConditionTimestampMonotonic": "17800059", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2020-01-16T00:09:16.498 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1713", "ExecMainStartTimestamp": "Thu 2020-01-16 00:06:48 UTC", "ExecMainStartTimestampMonotonic": "26089238", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2020-01-16 00:06:40 UTC", "InactiveExitTimestampMonotonic": "17801167", "InvocationID": "590dfa37419f47b29946b0a952b37aa4", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128031", "LimitNPROCSoft": "128031", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2020-01-16T00:09:16.500 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128031", "LimitSIGPENDINGSoft": "128031", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1713", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2020-01-16 00:06:48 UTC", "StateChangeTimestampMonoton 2020-01-16T00:09:16.502 INFO:teuthology.task.ansible.out:ic": "26089251", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2020-01-16 00:06:48 UTC", "WatchdogTimestampMonotonic": "26089249", "WatchdogUSec": "0"}} 2020-01-16T00:09:16.538 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:16 +0000 (0:00:00.062) 0:02:13.289 ****** 2020-01-16T00:09:16.564 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-01-16T00:09:16.565 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:16.603 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:16 +0000 (0:00:00.065) 0:02:13.355 ****** 2020-01-16T00:09:17.313 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-01-16T00:09:17.314 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"cache_update_time": 1579133010, "cache_updated": false, "changed": false} 2020-01-16T00:09:17.357 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:17 +0000 (0:00:00.753) 0:02:14.108 ****** 2020-01-16T00:09:17.385 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-01-16T00:09:17.387 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:17.426 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:17 +0000 (0:00:00.069) 0:02:14.178 ****** 2020-01-16T00:09:17.516 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-01-16T00:09:17.517 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"cache_update_time": 1579133053, "cache_updated": false, "changed": false} 2020-01-16T00:09:17.561 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:17 +0000 (0:00:00.134) 0:02:14.313 ****** 2020-01-16T00:09:17.566 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-01-16T00:09:17.567 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"changed": false, "gid": 1205, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2020-01-16T00:09:17.615 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:17 +0000 (0:00:00.053) 0:02:14.366 ****** 2020-01-16T00:09:17.620 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-01-16T00:09:17.621 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:17.672 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:17 +0000 (0:00:00.057) 0:02:14.424 ****** 2020-01-16T00:09:17.781 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-01-16T00:09:17.782 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "gid": 1205, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2020-01-16T00:09:17.827 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:17 +0000 (0:00:00.154) 0:02:14.578 ****** 2020-01-16T00:09:17.936 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2020-01-16T00:09:17.937 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1205, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2020-01-16T00:09:17.979 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:17 +0000 (0:00:00.152) 0:02:14.730 ****** 2020-01-16T00:09:18.174 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1205, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2020-01-16T00:09:18.225 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:18 +0000 (0:00:00.246) 0:02:14.977 ****** 2020-01-16T00:09:18.969 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2020-01-16T00:09:18.969 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip2", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-01-16T00:09:19.012 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:19 +0000 (0:00:00.786) 0:02:15.763 ****** 2020-01-16T00:09:19.028 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip2", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-01-16T00:09:19.071 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:19 +0000 (0:00:00.059) 0:02:15.823 ****** 2020-01-16T00:09:19.535 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-01-16T00:09:19.537 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false} 2020-01-16T00:09:19.583 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:19 +0000 (0:00:00.511) 0:02:16.334 ****** 2020-01-16T00:09:19.599 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"changed": false} 2020-01-16T00:09:19.643 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:19 +0000 (0:00:00.060) 0:02:16.394 ****** 2020-01-16T00:09:19.648 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-01-16T00:09:19.648 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-01-16T00:09:19.687 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:19 +0000 (0:00:00.044) 0:02:16.438 ****** 2020-01-16T00:09:19.691 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-01-16T00:09:19.734 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:19 +0000 (0:00:00.045) 0:02:16.484 ****** 2020-01-16T00:09:19.833 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-01-16T00:09:19.834 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1579132956.4173982, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1579132954.95413, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 107167, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1579132954.95413, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "4209737475", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2020-01-16T00:09:19.874 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:19 +0000 (0:00:00.142) 0:02:16.626 ****** 2020-01-16T00:09:19.891 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1579132995.74189, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1579132994.3026097, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 107167, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1579132994.3026097, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "4209737475", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2020-01-16T00:09:19.935 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:19 +0000 (0:00:00.060) 0:02:16.686 ****** 2020-01-16T00:09:20.223 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-01-16T00:09:20.224 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2020-01-16T00:09:20.263 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:20 +0000 (0:00:00.327) 0:02:17.014 ****** 2020-01-16T00:09:20.268 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2020-01-16T00:09:20.308 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:20 +0000 (0:00:00.045) 0:02:17.060 ****** 2020-01-16T00:09:20.313 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-01-16T00:09:20.313 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-01-16T00:09:20.354 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:20 +0000 (0:00:00.046) 0:02:17.106 ****** 2020-01-16T00:09:20.360 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-01-16T00:09:20.411 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:20 +0000 (0:00:00.056) 0:02:17.162 ****** 2020-01-16T00:09:20.417 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2020-01-16T00:09:20.418 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:20.461 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:20 +0000 (0:00:00.050) 0:02:17.212 ****** 2020-01-16T00:09:20.467 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:20.510 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:20 +0000 (0:00:00.048) 0:02:17.261 ****** 2020-01-16T00:09:20.581 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2020-01-16T00:09:20.582 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-01-16T00:09:20.625 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:20 +0000 (0:00:00.115) 0:02:17.376 ****** 2020-01-16T00:09:20.632 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-01-16T00:09:20.676 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:20 +0000 (0:00:00.051) 0:02:17.427 ****** 2020-01-16T00:09:20.756 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-01-16T00:09:20.757 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-01-16T00:09:20.803 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:20 +0000 (0:00:00.125) 0:02:17.553 ****** 2020-01-16T00:09:20.819 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-01-16T00:09:20.868 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:20 +0000 (0:00:00.065) 0:02:17.619 ****** 2020-01-16T00:09:20.877 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2020-01-16T00:09:20.911 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:20.924 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:20 +0000 (0:00:00.056) 0:02:17.675 ****** 2020-01-16T00:09:20.930 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:20.984 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:20 +0000 (0:00:00.060) 0:02:17.736 ****** 2020-01-16T00:09:20.990 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-01-16T00:09:20.990 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2020-01-16T00:09:20.992 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi123.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2020-01-16T00:09:20.993 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi123.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-01-16T00:09:21.040 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:21 +0000 (0:00:00.055) 0:02:17.792 ****** 2020-01-16T00:09:21.045 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2020-01-16T00:09:21.046 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-01-16T00:09:21.048 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2020-01-16T00:09:21.049 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2020-01-16T00:09:21.050 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-01-16T00:09:21.095 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:21 +0000 (0:00:00.054) 0:02:17.847 ****** 2020-01-16T00:09:21.100 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-01-16T00:09:21.101 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:21.102 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:21.104 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno2', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi123.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:21.105 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'lo', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:21.151 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:21 +0000 (0:00:00.056) 0:02:17.903 ****** 2020-01-16T00:09:21.157 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:21.158 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'lo', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:21.160 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno2', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:21.160 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:21.162 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:21.203 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:21 +0000 (0:00:00.051) 0:02:17.955 ****** 2020-01-16T00:09:21.208 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-01-16T00:09:21.209 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:21.253 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:21 +0000 (0:00:00.049) 0:02:18.004 ****** 2020-01-16T00:09:21.259 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:21.302 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:21 +0000 (0:00:00.049) 0:02:18.053 ****** 2020-01-16T00:09:21.307 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-01-16T00:09:21.309 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:21.351 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:21 +0000 (0:00:00.048) 0:02:18.102 ****** 2020-01-16T00:09:21.357 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:21.401 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:21 +0000 (0:00:00.049) 0:02:18.152 ****** 2020-01-16T00:09:21.405 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-01-16T00:09:21.407 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:21.449 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:21 +0000 (0:00:00.048) 0:02:18.200 ****** 2020-01-16T00:09:21.454 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:21.498 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:21 +0000 (0:00:00.049) 0:02:18.249 ****** 2020-01-16T00:09:21.502 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-01-16T00:09:21.504 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:21.545 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:21 +0000 (0:00:00.044) 0:02:18.294 ****** 2020-01-16T00:09:21.549 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:21.599 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:21 +0000 (0:00:00.055) 0:02:18.350 ****** 2020-01-16T00:09:21.604 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-01-16T00:09:21.605 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:21.649 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:21 +0000 (0:00:00.050) 0:02:18.401 ****** 2020-01-16T00:09:21.656 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:21.699 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:21 +0000 (0:00:00.050) 0:02:18.451 ****** 2020-01-16T00:09:21.704 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-01-16T00:09:21.705 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:21.747 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:21 +0000 (0:00:00.048) 0:02:18.499 ****** 2020-01-16T00:09:21.753 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:21.802 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:21 +0000 (0:00:00.055) 0:02:18.554 ****** 2020-01-16T00:09:21.807 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-01-16T00:09:21.808 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:21.860 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:21 +0000 (0:00:00.057) 0:02:18.612 ****** 2020-01-16T00:09:21.866 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:21.909 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:21 +0000 (0:00:00.048) 0:02:18.660 ****** 2020-01-16T00:09:21.913 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-01-16T00:09:21.915 INFO:teuthology.task.ansible.out:skipping: [smithi123.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:21.955 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:21 +0000 (0:00:00.046) 0:02:18.706 ****** 2020-01-16T00:09:21.961 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-01-16T00:09:22.002 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:21 +0000 (0:00:00.047) 0:02:18.754 ****** 2020-01-16T00:09:22.126 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi123.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.011103", "end": "2020-01-16 00:09:22.300271", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-01-16 00:09:22.289168", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-01-16T00:09:22.162 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.011030", "end": "2020-01-16 00:09:22.171642", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-01-16 00:09:22.160612", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-01-16T00:09:22.246 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006035", "end": "2020-01-16 00:09:22.433546", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-01-16 00:09:22.427511", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-01-16T00:09:22.267 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006086", "end": "2020-01-16 00:09:22.294976", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-01-16 00:09:22.288890", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-01-16T00:09:22.361 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005456", "end": "2020-01-16 00:09:22.553589", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-01-16 00:09:22.548133", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2020-01-16T00:09:22.383 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005416", "end": "2020-01-16 00:09:22.408921", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-01-16 00:09:22.403505", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2020-01-16T00:09:22.510 INFO:teuthology.task.ansible.out:ok: [smithi123.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.010898", "end": "2020-01-16 00:09:22.691801", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-01-16 00:09:22.680903", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-01-16T00:09:22.566 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:22 +0000 (0:00:00.563) 0:02:19.317 ****** 2020-01-16T00:09:22.592 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-01-16T00:09:22.641 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:22 +0000 (0:00:00.074) 0:02:19.392 ****** 2020-01-16T00:09:22.719 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-01-16T00:09:22.719 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.227155", "end": "2020-01-16 00:09:22.747791", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-01-16 00:09:22.520636", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-01-16T00:09:22.760 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:22 +0000 (0:00:00.119) 0:02:19.511 ****** 2020-01-16T00:09:22.765 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-01-16T00:09:22.767 INFO:teuthology.task.ansible.out:changed: [smithi123.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2020-01-16T00:09:22.809 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-01-16T00:09:22.874 INFO:teuthology.task.ansible.out:Thursday 16 January 2020 00:09:22 +0000 (0:00:00.114) 0:02:19.626 ****** 2020-01-16T00:09:23.003 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-01-16T00:09:23.004 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2020-01-16T00:09:23.116 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2020-01-16T00:09:23.119 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2020-01-16T00:09:23.121 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2020-01-16T00:09:23.122 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* smithi123.front.sepia.ceph.com : ok=113 changed=32 unreachable=0 failed=0 skipped=169 rescued=0 ignored=0 smithi145.front.sepia.ceph.com : ok=113 changed=32 unreachable=0 failed=0 skipped=169 rescued=0 ignored=0 Thursday 16 January 2020 00:09:23 +0000 (0:00:00.247) 0:02:19.873 ****** =============================================================================== 2020-01-16T00:09:23.124 INFO:teuthology.task.ansible.out:users : Create all admin users with sudo access. ----------------------- 17.32s users : Update authorized_keys using the keys repo --------------------- 15.20s testnode : Ensure packages are not present. ----------------------------- 9.69s testnode : Install packages --------------------------------------------- 9.37s testnode : Zap all non-root disks --------------------------------------- 7.66s users : Remove revoked users -------------------------------------------- 5.97s common : Update apt cache ----------------------------------------------- 3.42s testnode : Update apt cache. -------------------------------------------- 2.98s Gathering Facts --------------------------------------------------------- 2.77s testnode : Install apt keys --------------------------------------------- 2.20s users : Update authorized_keys for each user with literal keys ---------- 1.53s ansible-managed : Add authorized keys for the ansible user. ------------- 1.45s testnode : Set the system's hostname ------------------------------------ 0.93s common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/. --- 0.93s testnode : Install python-apt ------------------------------------------- 0.90s common : Make sure smartmontools is installed --------------------------- 0.90s users : Clone the keys repo --------------------------------------------- 0.88s testnode : Ensure perl-doc is installed on apt systems. ----------------- 0.86s 2020-01-16T00:09:23.124 INFO:teuthology.task.ansible.out:common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/. --- 0.82s Gathering Facts --------------------------------------------------------- 0.82s 2020-01-16T00:09:23.389 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi123.front.sepia.ceph.com'), Remote(name='ubuntu@smithi145.front.sepia.ceph.com')] 2020-01-16T00:09:23.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-01-16T00:09:23.392 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi123.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:09:23.467 INFO:teuthology.orchestra.run.smithi123:> true 2020-01-16T00:09:23.672 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-01-16T00:09:23.673 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi145.front.sepia.ceph.com', 'timeout': 60} 2020-01-16T00:09:23.754 INFO:teuthology.orchestra.run.smithi145:> true 2020-01-16T00:09:23.954 INFO:teuthology.run_tasks:Running task clock... 2020-01-16T00:09:23.966 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2020-01-16T00:09:23.966 INFO:teuthology.orchestra.run.smithi123:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-01-16T00:09:24.044 INFO:teuthology.orchestra.run.smithi123.stdout:16 Jan 00:09:24 ntpd[7756]: ntpd 4.2.8p10@1.3728-o (1): Starting 2020-01-16T00:09:24.044 INFO:teuthology.orchestra.run.smithi123.stdout:16 Jan 00:09:24 ntpd[7756]: Command line: ntpd -gq 2020-01-16T00:09:24.045 INFO:teuthology.orchestra.run.smithi123.stdout:16 Jan 00:09:24 ntpd[7756]: proto: precision = 0.120 usec (-23) 2020-01-16T00:09:24.049 INFO:teuthology.orchestra.run.smithi123.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-01-16T00:09:24.050 INFO:teuthology.orchestra.run.smithi123.stderr:restrict ::: KOD does nothing without LIMITED. 2020-01-16T00:09:24.050 INFO:teuthology.orchestra.run.smithi123.stdout:16 Jan 00:09:24 ntpd[7756]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-01-16T00:09:24.050 INFO:teuthology.orchestra.run.smithi123.stdout:16 Jan 00:09:24 ntpd[7756]: restrict ::: KOD does nothing without LIMITED. 2020-01-16T00:09:24.050 INFO:teuthology.orchestra.run.smithi123.stdout:16 Jan 00:09:24 ntpd[7756]: Listen and drop on 0 v6wildcard [::]:123 2020-01-16T00:09:24.051 INFO:teuthology.orchestra.run.smithi123.stdout:16 Jan 00:09:24 ntpd[7756]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2020-01-16T00:09:24.052 INFO:teuthology.orchestra.run.smithi123.stdout:16 Jan 00:09:24 ntpd[7756]: Listen normally on 2 lo 127.0.0.1:123 2020-01-16T00:09:24.052 INFO:teuthology.orchestra.run.smithi123.stdout:16 Jan 00:09:24 ntpd[7756]: Listen normally on 3 enp3s0f1 172.21.15.123:123 2020-01-16T00:09:24.052 INFO:teuthology.orchestra.run.smithi123.stdout:16 Jan 00:09:24 ntpd[7756]: Listen normally on 4 lo [::1]:123 2020-01-16T00:09:24.053 INFO:teuthology.orchestra.run.smithi123.stdout:16 Jan 00:09:24 ntpd[7756]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d3f1%5]:123 2020-01-16T00:09:24.053 INFO:teuthology.orchestra.run.smithi123.stdout:16 Jan 00:09:24 ntpd[7756]: Listening on routing socket on fd #22 for interface updates 2020-01-16T00:09:26.046 INFO:teuthology.orchestra.run.smithi123.stderr:16 Jan 00:09:26 ntpd[7756]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-01-16T00:09:26.046 INFO:teuthology.orchestra.run.smithi123.stderr:16 Jan 00:09:26 ntpd[7756]: can't open /var/log/ntpstats/rawstats.20200116: Permission denied 2020-01-16T00:09:26.046 INFO:teuthology.orchestra.run.smithi123.stderr:16 Jan 00:09:26 ntpd[7756]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-01-16T00:09:26.046 INFO:teuthology.orchestra.run.smithi123.stderr:16 Jan 00:09:26 ntpd[7756]: can't open /var/log/ntpstats/peerstats.20200116: Permission denied 2020-01-16T00:09:28.046 INFO:teuthology.orchestra.run.smithi123.stderr:16 Jan 00:09:28 ntpd[7756]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-01-16T00:09:28.047 INFO:teuthology.orchestra.run.smithi123.stderr:16 Jan 00:09:28 ntpd[7756]: can't open /var/log/ntpstats/rawstats.20200116: Permission denied 2020-01-16T00:09:28.047 INFO:teuthology.orchestra.run.smithi123.stderr:16 Jan 00:09:28 ntpd[7756]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-01-16T00:09:28.047 INFO:teuthology.orchestra.run.smithi123.stderr:16 Jan 00:09:28 ntpd[7756]: can't open /var/log/ntpstats/peerstats.20200116: Permission denied 2020-01-16T00:09:30.047 INFO:teuthology.orchestra.run.smithi123.stderr:16 Jan 00:09:30 ntpd[7756]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-01-16T00:09:30.048 INFO:teuthology.orchestra.run.smithi123.stderr:16 Jan 00:09:30 ntpd[7756]: can't open /var/log/ntpstats/rawstats.20200116: Permission denied 2020-01-16T00:09:30.048 INFO:teuthology.orchestra.run.smithi123.stderr:16 Jan 00:09:30 ntpd[7756]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-01-16T00:09:30.048 INFO:teuthology.orchestra.run.smithi123.stderr:16 Jan 00:09:30 ntpd[7756]: can't open /var/log/ntpstats/peerstats.20200116: Permission denied 2020-01-16T00:09:32.049 INFO:teuthology.orchestra.run.smithi123.stderr:16 Jan 00:09:32 ntpd[7756]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-01-16T00:09:32.050 INFO:teuthology.orchestra.run.smithi123.stderr:16 Jan 00:09:32 ntpd[7756]: can't open /var/log/ntpstats/rawstats.20200116: Permission denied 2020-01-16T00:09:32.050 INFO:teuthology.orchestra.run.smithi123.stderr:16 Jan 00:09:32 ntpd[7756]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-01-16T00:09:32.050 INFO:teuthology.orchestra.run.smithi123.stderr:16 Jan 00:09:32 ntpd[7756]: can't open /var/log/ntpstats/peerstats.20200116: Permission denied 2020-01-16T00:09:32.050 INFO:teuthology.orchestra.run.smithi123.stderr:16 Jan 00:09:32 ntpd[7756]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2020-01-16T00:09:32.050 INFO:teuthology.orchestra.run.smithi123.stderr:16 Jan 00:09:32 ntpd[7756]: can't open /var/log/ntpstats/loopstats.20200116: Permission denied 2020-01-16T00:09:32.050 INFO:teuthology.orchestra.run.smithi123.stdout:16 Jan 00:09:32 ntpd[7756]: ntpd: time set -0.211346 s 2020-01-16T00:09:32.051 INFO:teuthology.orchestra.run.smithi123.stdout:ntpd: time set -0.211346s 2020-01-16T00:09:32.108 INFO:teuthology.orchestra.run.smithi123.stdout: remote refid st t when poll reach delay offset jitter 2020-01-16T00:09:32.108 INFO:teuthology.orchestra.run.smithi123.stdout:============================================================================== 2020-01-16T00:09:32.108 INFO:teuthology.orchestra.run.smithi123.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-01-16T00:09:32.109 INFO:teuthology.orchestra.run.smithi123.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-01-16T00:09:32.109 INFO:teuthology.orchestra.run.smithi123.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-01-16T00:09:32.109 INFO:teuthology.orchestra.run.smithi123.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-01-16T00:09:32.110 INFO:teuthology.orchestra.run.smithi145:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-01-16T00:09:32.162 INFO:teuthology.orchestra.run.smithi145.stdout:16 Jan 00:09:32 ntpd[7606]: ntpd 4.2.8p10@1.3728-o (1): Starting 2020-01-16T00:09:32.162 INFO:teuthology.orchestra.run.smithi145.stdout:16 Jan 00:09:32 ntpd[7606]: Command line: ntpd -gq 2020-01-16T00:09:32.163 INFO:teuthology.orchestra.run.smithi145.stdout:16 Jan 00:09:32 ntpd[7606]: proto: precision = 0.087 usec (-23) 2020-01-16T00:09:32.163 INFO:teuthology.orchestra.run.smithi145.stdout:16 Jan 00:09:32 ntpd[7606]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-01-16T00:09:32.164 INFO:teuthology.orchestra.run.smithi145.stdout:16 Jan 00:09:32 ntpd[7606]: restrict ::: KOD does nothing without LIMITED. 2020-01-16T00:09:32.164 INFO:teuthology.orchestra.run.smithi145.stdout:16 Jan 00:09:32 ntpd[7606]: Listen and drop on 0 v6wildcard [::]:123 2020-01-16T00:09:32.164 INFO:teuthology.orchestra.run.smithi145.stdout:16 Jan 00:09:32 ntpd[7606]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2020-01-16T00:09:32.164 INFO:teuthology.orchestra.run.smithi145.stdout:16 Jan 00:09:32 ntpd[7606]: Listen normally on 2 lo 127.0.0.1:123 2020-01-16T00:09:32.164 INFO:teuthology.orchestra.run.smithi145.stdout:16 Jan 00:09:32 ntpd[7606]: Listen normally on 3 enp3s0f1 172.21.15.145:123 2020-01-16T00:09:32.164 INFO:teuthology.orchestra.run.smithi145.stdout:16 Jan 00:09:32 ntpd[7606]: Listen normally on 4 lo [::1]:123 2020-01-16T00:09:32.164 INFO:teuthology.orchestra.run.smithi145.stdout:16 Jan 00:09:32 ntpd[7606]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7791%5]:123 2020-01-16T00:09:32.164 INFO:teuthology.orchestra.run.smithi145.stdout:16 Jan 00:09:32 ntpd[7606]: Listening on routing socket on fd #22 for interface updates 2020-01-16T00:09:32.165 INFO:teuthology.orchestra.run.smithi145.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-01-16T00:09:32.165 INFO:teuthology.orchestra.run.smithi145.stderr:restrict ::: KOD does nothing without LIMITED. 2020-01-16T00:09:33.163 INFO:teuthology.orchestra.run.smithi145.stderr:16 Jan 00:09:33 ntpd[7606]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-01-16T00:09:33.163 INFO:teuthology.orchestra.run.smithi145.stderr:16 Jan 00:09:33 ntpd[7606]: can't open /var/log/ntpstats/rawstats.20200116: Permission denied 2020-01-16T00:09:33.163 INFO:teuthology.orchestra.run.smithi145.stderr:16 Jan 00:09:33 ntpd[7606]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-01-16T00:09:33.163 INFO:teuthology.orchestra.run.smithi145.stderr:16 Jan 00:09:33 ntpd[7606]: can't open /var/log/ntpstats/peerstats.20200116: Permission denied 2020-01-16T00:09:35.164 INFO:teuthology.orchestra.run.smithi145.stderr:16 Jan 00:09:35 ntpd[7606]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-01-16T00:09:35.165 INFO:teuthology.orchestra.run.smithi145.stderr:16 Jan 00:09:35 ntpd[7606]: can't open /var/log/ntpstats/rawstats.20200116: Permission denied 2020-01-16T00:09:35.165 INFO:teuthology.orchestra.run.smithi145.stderr:16 Jan 00:09:35 ntpd[7606]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-01-16T00:09:35.165 INFO:teuthology.orchestra.run.smithi145.stderr:16 Jan 00:09:35 ntpd[7606]: can't open /var/log/ntpstats/peerstats.20200116: Permission denied 2020-01-16T00:09:37.165 INFO:teuthology.orchestra.run.smithi145.stderr:16 Jan 00:09:37 ntpd[7606]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-01-16T00:09:37.165 INFO:teuthology.orchestra.run.smithi145.stderr:16 Jan 00:09:37 ntpd[7606]: can't open /var/log/ntpstats/rawstats.20200116: Permission denied 2020-01-16T00:09:37.166 INFO:teuthology.orchestra.run.smithi145.stderr:16 Jan 00:09:37 ntpd[7606]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-01-16T00:09:37.166 INFO:teuthology.orchestra.run.smithi145.stderr:16 Jan 00:09:37 ntpd[7606]: can't open /var/log/ntpstats/peerstats.20200116: Permission denied 2020-01-16T00:09:39.166 INFO:teuthology.orchestra.run.smithi145.stderr:16 Jan 00:09:39 ntpd[7606]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-01-16T00:09:39.166 INFO:teuthology.orchestra.run.smithi145.stderr:16 Jan 00:09:39 ntpd[7606]: can't open /var/log/ntpstats/rawstats.20200116: Permission denied 2020-01-16T00:09:39.166 INFO:teuthology.orchestra.run.smithi145.stderr:16 Jan 00:09:39 ntpd[7606]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-01-16T00:09:39.166 INFO:teuthology.orchestra.run.smithi145.stderr:16 Jan 00:09:39 ntpd[7606]: can't open /var/log/ntpstats/peerstats.20200116: Permission denied 2020-01-16T00:09:39.167 INFO:teuthology.orchestra.run.smithi145.stderr:16 Jan 00:09:39 ntpd[7606]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2020-01-16T00:09:39.167 INFO:teuthology.orchestra.run.smithi145.stderr:16 Jan 00:09:39 ntpd[7606]: can't open /var/log/ntpstats/loopstats.20200116: Permission denied 2020-01-16T00:09:39.167 INFO:teuthology.orchestra.run.smithi145.stdout:16 Jan 00:09:39 ntpd[7606]: ntpd: time set -0.044749 s 2020-01-16T00:09:39.167 INFO:teuthology.orchestra.run.smithi145.stdout:ntpd: time set -0.044749s 2020-01-16T00:09:39.256 INFO:teuthology.orchestra.run.smithi145.stdout: remote refid st t when poll reach delay offset jitter 2020-01-16T00:09:39.257 INFO:teuthology.orchestra.run.smithi145.stdout:============================================================================== 2020-01-16T00:09:39.257 INFO:teuthology.orchestra.run.smithi145.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-01-16T00:09:39.257 INFO:teuthology.orchestra.run.smithi145.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-01-16T00:09:39.257 INFO:teuthology.orchestra.run.smithi145.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-01-16T00:09:39.257 INFO:teuthology.orchestra.run.smithi145.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-01-16T00:09:39.258 INFO:teuthology.run_tasks:Running task install... 2020-01-16T00:09:39.268 DEBUG:teuthology.task.install:project ceph 2020-01-16T00:09:39.268 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': 'cb3c85e74500283e01cc592f7e2858e323186910'}} 2020-01-16T00:09:39.268 DEBUG:teuthology.task.install:config {'sha1': 'cb3c85e74500283e01cc592f7e2858e323186910'} 2020-01-16T00:09:39.269 INFO:teuthology.task.install:Using flavor: basic 2020-01-16T00:09:39.281 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2020-01-16T00:09:39.281 INFO:teuthology.task.install:extra packages: [] 2020-01-16T00:09:39.281 INFO:teuthology.orchestra.run.smithi123:> sudo lsb_release -is 2020-01-16T00:09:39.335 INFO:teuthology.orchestra.run.smithi123.stdout:Ubuntu 2020-01-16T00:09:39.340 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-01-16T00:09:39.341 INFO:teuthology.orchestra.run.smithi145:> sudo lsb_release -is 2020-01-16T00:09:39.341 INFO:teuthology.orchestra.run.smithi123:> sudo apt-key list | grep Ceph 2020-01-16T00:09:39.394 INFO:teuthology.orchestra.run.smithi145.stdout:Ubuntu 2020-01-16T00:09:39.399 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-01-16T00:09:39.400 INFO:teuthology.orchestra.run.smithi145:> sudo apt-key list | grep Ceph 2020-01-16T00:09:39.432 INFO:teuthology.orchestra.run.smithi123.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2020-01-16T00:09:39.479 INFO:teuthology.orchestra.run.smithi145.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2020-01-16T00:09:39.486 INFO:teuthology.orchestra.run.smithi123.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2020-01-16T00:09:39.486 INFO:teuthology.orchestra.run.smithi123.stdout:uid [ unknown] Ceph.com (release key) 2020-01-16T00:09:39.487 INFO:teuthology.orchestra.run.smithi123:> sudo lsb_release -is 2020-01-16T00:09:39.537 INFO:teuthology.orchestra.run.smithi145.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2020-01-16T00:09:39.537 INFO:teuthology.orchestra.run.smithi145.stdout:uid [ unknown] Ceph.com (release key) 2020-01-16T00:09:39.538 INFO:teuthology.orchestra.run.smithi145:> sudo lsb_release -is 2020-01-16T00:09:39.542 INFO:teuthology.orchestra.run.smithi123.stdout:Ubuntu 2020-01-16T00:09:39.547 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-01-16T00:09:39.547 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2020-01-16T00:09:39.548 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=cb3c85e74500283e01cc592f7e2858e323186910 2020-01-16T00:09:39.598 INFO:teuthology.orchestra.run.smithi145.stdout:Ubuntu 2020-01-16T00:09:39.599 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-01-16T00:09:39.599 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2020-01-16T00:09:39.600 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=cb3c85e74500283e01cc592f7e2858e323186910 2020-01-16T00:09:39.729 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default/ 2020-01-16T00:09:39.730 INFO:teuthology.task.install.deb:Package version is 15.0.0-9353-gcb3c85e-1bionic 2020-01-16T00:09:39.758 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default/ 2020-01-16T00:09:39.758 INFO:teuthology.task.install.deb:Package version is 15.0.0-9353-gcb3c85e-1bionic 2020-01-16T00:09:39.873 INFO:teuthology.orchestra.run.smithi123:> sudo sh -c 'cat > /etc/apt/sources.list.d/ceph.list' 2020-01-16T00:09:39.875 INFO:teuthology.orchestra.run.smithi145:> sudo sh -c 'cat > /etc/apt/sources.list.d/ceph.list' 2020-01-16T00:09:39.922 INFO:teuthology.orchestra.run.smithi123:> sudo apt-get update 2020-01-16T00:09:39.925 INFO:teuthology.orchestra.run.smithi145:> sudo apt-get update 2020-01-16T00:09:40.119 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-01-16T00:09:40.180 INFO:teuthology.orchestra.run.smithi123.stdout:Ign:2 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic InRelease 2020-01-16T00:09:40.185 INFO:teuthology.orchestra.run.smithi145.stdout:Ign:1 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic InRelease 2020-01-16T00:09:40.201 INFO:teuthology.orchestra.run.smithi123.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic Release [7,663 B] 2020-01-16T00:09:40.206 INFO:teuthology.orchestra.run.smithi145.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic Release [7,663 B] 2020-01-16T00:09:40.214 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-01-16T00:09:40.224 INFO:teuthology.orchestra.run.smithi123.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic Release.gpg 2020-01-16T00:09:40.229 INFO:teuthology.orchestra.run.smithi145.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-01-16T00:09:40.230 INFO:teuthology.orchestra.run.smithi145.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic Release.gpg 2020-01-16T00:09:40.233 INFO:teuthology.orchestra.run.smithi145.stdout:Hit:5 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-01-16T00:09:40.246 INFO:teuthology.orchestra.run.smithi123.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main i386 Packages [2,593 B] 2020-01-16T00:09:40.251 INFO:teuthology.orchestra.run.smithi145.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main i386 Packages [2,593 B] 2020-01-16T00:09:40.284 INFO:teuthology.orchestra.run.smithi123.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 Packages [15.3 kB] 2020-01-16T00:09:40.289 INFO:teuthology.orchestra.run.smithi145.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 Packages [15.3 kB] 2020-01-16T00:09:40.294 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-01-16T00:09:40.297 INFO:teuthology.orchestra.run.smithi145.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-01-16T00:09:40.375 INFO:teuthology.orchestra.run.smithi123.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-01-16T00:09:40.378 INFO:teuthology.orchestra.run.smithi145.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-01-16T00:09:40.719 INFO:teuthology.orchestra.run.smithi123.stdout:Fetched 25.6 kB in 1s (42.6 kB/s) 2020-01-16T00:09:40.817 INFO:teuthology.orchestra.run.smithi145.stdout:Fetched 25.6 kB in 1s (36.7 kB/s) 2020-01-16T00:09:41.868 INFO:teuthology.orchestra.run.smithi123.stdout:Reading package lists... 2020-01-16T00:09:41.914 INFO:teuthology.orchestra.run.smithi123:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.0.0-9353-gcb3c85e-1bionic cephadm=15.0.0-9353-gcb3c85e-1bionic ceph-mds=15.0.0-9353-gcb3c85e-1bionic ceph-mgr=15.0.0-9353-gcb3c85e-1bionic ceph-common=15.0.0-9353-gcb3c85e-1bionic ceph-fuse=15.0.0-9353-gcb3c85e-1bionic ceph-test=15.0.0-9353-gcb3c85e-1bionic radosgw=15.0.0-9353-gcb3c85e-1bionic python3-rados=15.0.0-9353-gcb3c85e-1bionic python3-rgw=15.0.0-9353-gcb3c85e-1bionic python3-cephfs=15.0.0-9353-gcb3c85e-1bionic python3-rbd=15.0.0-9353-gcb3c85e-1bionic libcephfs2=15.0.0-9353-gcb3c85e-1bionic libcephfs-dev=15.0.0-9353-gcb3c85e-1bionic librados2=15.0.0-9353-gcb3c85e-1bionic librbd1=15.0.0-9353-gcb3c85e-1bionic rbd-fuse=15.0.0-9353-gcb3c85e-1bionic 2020-01-16T00:09:41.968 INFO:teuthology.orchestra.run.smithi145.stdout:Reading package lists... 2020-01-16T00:09:41.994 INFO:teuthology.orchestra.run.smithi123.stdout:Reading package lists... 2020-01-16T00:09:42.012 INFO:teuthology.orchestra.run.smithi145:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.0.0-9353-gcb3c85e-1bionic cephadm=15.0.0-9353-gcb3c85e-1bionic ceph-mds=15.0.0-9353-gcb3c85e-1bionic ceph-mgr=15.0.0-9353-gcb3c85e-1bionic ceph-common=15.0.0-9353-gcb3c85e-1bionic ceph-fuse=15.0.0-9353-gcb3c85e-1bionic ceph-test=15.0.0-9353-gcb3c85e-1bionic radosgw=15.0.0-9353-gcb3c85e-1bionic python3-rados=15.0.0-9353-gcb3c85e-1bionic python3-rgw=15.0.0-9353-gcb3c85e-1bionic python3-cephfs=15.0.0-9353-gcb3c85e-1bionic python3-rbd=15.0.0-9353-gcb3c85e-1bionic libcephfs2=15.0.0-9353-gcb3c85e-1bionic libcephfs-dev=15.0.0-9353-gcb3c85e-1bionic librados2=15.0.0-9353-gcb3c85e-1bionic librbd1=15.0.0-9353-gcb3c85e-1bionic rbd-fuse=15.0.0-9353-gcb3c85e-1bionic 2020-01-16T00:09:42.088 INFO:teuthology.orchestra.run.smithi145.stdout:Reading package lists... 2020-01-16T00:09:42.233 INFO:teuthology.orchestra.run.smithi123.stdout:Building dependency tree... 2020-01-16T00:09:42.234 INFO:teuthology.orchestra.run.smithi123.stdout:Reading state information... 2020-01-16T00:09:42.325 INFO:teuthology.orchestra.run.smithi145.stdout:Building dependency tree... 2020-01-16T00:09:42.326 INFO:teuthology.orchestra.run.smithi145.stdout:Reading state information... 2020-01-16T00:09:42.384 INFO:teuthology.orchestra.run.smithi123.stdout:The following additional packages will be installed: 2020-01-16T00:09:42.384 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2020-01-16T00:09:42.384 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-rook ceph-mon 2020-01-16T00:09:42.386 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-osd jq libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 2020-01-16T00:09:42.386 INFO:teuthology.orchestra.run.smithi123.stdout: librabbitmq4 libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 2020-01-16T00:09:42.386 INFO:teuthology.orchestra.run.smithi123.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2020-01-16T00:09:42.386 INFO:teuthology.orchestra.run.smithi123.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-01-16T00:09:42.387 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2020-01-16T00:09:42.387 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2020-01-16T00:09:42.387 INFO:teuthology.orchestra.run.smithi123.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2020-01-16T00:09:42.387 INFO:teuthology.orchestra.run.smithi123.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2020-01-16T00:09:42.387 INFO:teuthology.orchestra.run.smithi123.stdout: python3-mako python3-markupsafe python3-matplotlib python3-monotonic 2020-01-16T00:09:42.387 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy python3-oauth2client python3-olefile python3-openssl 2020-01-16T00:09:42.387 INFO:teuthology.orchestra.run.smithi123.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2020-01-16T00:09:42.388 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pil python3-pluggy python3-prettytable python3-py python3-pyasn1 2020-01-16T00:09:42.388 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2020-01-16T00:09:42.388 INFO:teuthology.orchestra.run.smithi123.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2020-01-16T00:09:42.388 INFO:teuthology.orchestra.run.smithi123.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-01-16T00:09:42.388 INFO:teuthology.orchestra.run.smithi123.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-01-16T00:09:42.388 INFO:teuthology.orchestra.run.smithi123.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-01-16T00:09:42.388 INFO:teuthology.orchestra.run.smithi123.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-01-16T00:09:42.388 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2020-01-16T00:09:42.389 INFO:teuthology.orchestra.run.smithi123.stdout:Suggested packages: 2020-01-16T00:09:42.389 INFO:teuthology.orchestra.run.smithi123.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2020-01-16T00:09:42.389 INFO:teuthology.orchestra.run.smithi123.stdout: python-pytest-xdist python-mock python-cryptography-doc 2020-01-16T00:09:42.389 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi python3-crypto 2020-01-16T00:09:42.390 INFO:teuthology.orchestra.run.smithi123.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc dvipng 2020-01-16T00:09:42.390 INFO:teuthology.orchestra.run.smithi123.stdout: ffmpeg gir1.2-gtk-3.0 ghostscript inkscape ipython3 python-matplotlib-doc 2020-01-16T00:09:42.390 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cairocffi python3-gi-cairo python3-gobject python3-pyqt4 python3-sip 2020-01-16T00:09:42.390 INFO:teuthology.orchestra.run.smithi123.stdout: python3-tornado texlive-extra-utils texlive-latex-extra ttf-staypuft 2020-01-16T00:09:42.390 INFO:teuthology.orchestra.run.smithi123.stdout: python-numpy-doc python3-dev python3-numpy-dbg python-openssl-doc 2020-01-16T00:09:42.390 INFO:teuthology.orchestra.run.smithi123.stdout: python3-openssl-dbg httpd-wsgi libapache2-mod-python libapache2-mod-scgi 2020-01-16T00:09:42.390 INFO:teuthology.orchestra.run.smithi123.stdout: libjs-mochikit python-pecan-doc python-pil-doc python3-pil-dbg 2020-01-16T00:09:42.390 INFO:teuthology.orchestra.run.smithi123.stdout: python-pyinotify-doc python-pyparsing-doc python-scipy-doc 2020-01-16T00:09:42.391 INFO:teuthology.orchestra.run.smithi123.stdout: python-setuptools-doc python3-dap python-sklearn-doc python-sqlalchemy-doc 2020-01-16T00:09:42.391 INFO:teuthology.orchestra.run.smithi123.stdout: python3-psycopg2 python3-pymysql python3-fdb tix python3-tk-dbg 2020-01-16T00:09:42.391 INFO:teuthology.orchestra.run.smithi123.stdout: python-waitress-doc python-webob-doc python-webtest-doc python3-termcolor 2020-01-16T00:09:42.391 INFO:teuthology.orchestra.run.smithi123.stdout: python3-watchdog python-werkzeug-doc 2020-01-16T00:09:42.479 INFO:teuthology.orchestra.run.smithi145.stdout:The following additional packages will be installed: 2020-01-16T00:09:42.480 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2020-01-16T00:09:42.480 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-rook ceph-mon 2020-01-16T00:09:42.481 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-osd jq libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 2020-01-16T00:09:42.481 INFO:teuthology.orchestra.run.smithi145.stdout: librabbitmq4 libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 2020-01-16T00:09:42.481 INFO:teuthology.orchestra.run.smithi145.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2020-01-16T00:09:42.481 INFO:teuthology.orchestra.run.smithi145.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-01-16T00:09:42.482 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2020-01-16T00:09:42.482 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2020-01-16T00:09:42.482 INFO:teuthology.orchestra.run.smithi145.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2020-01-16T00:09:42.482 INFO:teuthology.orchestra.run.smithi145.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2020-01-16T00:09:42.482 INFO:teuthology.orchestra.run.smithi145.stdout: python3-mako python3-markupsafe python3-matplotlib python3-monotonic 2020-01-16T00:09:42.482 INFO:teuthology.orchestra.run.smithi145.stdout: python3-numpy python3-oauth2client python3-olefile python3-openssl 2020-01-16T00:09:42.482 INFO:teuthology.orchestra.run.smithi145.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2020-01-16T00:09:42.483 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pil python3-pluggy python3-prettytable python3-py python3-pyasn1 2020-01-16T00:09:42.483 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2020-01-16T00:09:42.483 INFO:teuthology.orchestra.run.smithi145.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2020-01-16T00:09:42.483 INFO:teuthology.orchestra.run.smithi145.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-01-16T00:09:42.483 INFO:teuthology.orchestra.run.smithi145.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-01-16T00:09:42.483 INFO:teuthology.orchestra.run.smithi145.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-01-16T00:09:42.483 INFO:teuthology.orchestra.run.smithi145.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-01-16T00:09:42.483 INFO:teuthology.orchestra.run.smithi145.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2020-01-16T00:09:42.486 INFO:teuthology.orchestra.run.smithi123.stdout:The following NEW packages will be installed: 2020-01-16T00:09:42.486 INFO:teuthology.orchestra.run.smithi123.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2020-01-16T00:09:42.486 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2020-01-16T00:09:42.486 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-rook ceph-mon 2020-01-16T00:09:42.486 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-osd ceph-test cephadm jq libcephfs-dev libcephfs2 libjq1 libleveldb1v5 2020-01-16T00:09:42.486 INFO:teuthology.orchestra.run.smithi123.stdout: liblttng-ust0 liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 2020-01-16T00:09:42.487 INFO:teuthology.orchestra.run.smithi123.stdout: librgw2 libsnappy1v5 libxslt1.1 python-attr python-funcsigs 2020-01-16T00:09:42.487 INFO:teuthology.orchestra.run.smithi123.stdout: python-pastedeploy-tpl python-pluggy python-py python-pytest 2020-01-16T00:09:42.487 INFO:teuthology.orchestra.run.smithi123.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-01-16T00:09:42.487 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2020-01-16T00:09:42.487 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2020-01-16T00:09:42.487 INFO:teuthology.orchestra.run.smithi123.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2020-01-16T00:09:42.487 INFO:teuthology.orchestra.run.smithi123.stdout: python3-httplib2 python3-joblib python3-jwt python3-kubernetes 2020-01-16T00:09:42.488 INFO:teuthology.orchestra.run.smithi123.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2020-01-16T00:09:42.488 INFO:teuthology.orchestra.run.smithi123.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client 2020-01-16T00:09:42.488 INFO:teuthology.orchestra.run.smithi123.stdout: python3-olefile python3-openssl python3-paste python3-pastedeploy 2020-01-16T00:09:42.488 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pastescript python3-pecan python3-pil python3-pluggy 2020-01-16T00:09:42.488 INFO:teuthology.orchestra.run.smithi123.stdout: python3-prettytable python3-py python3-pyasn1 python3-pyasn1-modules 2020-01-16T00:09:42.488 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyinotify python3-pyparsing python3-pytest python3-rados python3-rbd 2020-01-16T00:09:42.488 INFO:teuthology.orchestra.run.smithi123.stdout: python3-repoze.lru python3-rgw python3-routes python3-rsa python3-scipy 2020-01-16T00:09:42.488 INFO:teuthology.orchestra.run.smithi123.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-01-16T00:09:42.489 INFO:teuthology.orchestra.run.smithi123.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-01-16T00:09:42.489 INFO:teuthology.orchestra.run.smithi123.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-01-16T00:09:42.489 INFO:teuthology.orchestra.run.smithi123.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-01-16T00:09:42.489 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2020-01-16T00:09:42.489 INFO:teuthology.orchestra.run.smithi123.stdout:The following packages will be upgraded: 2020-01-16T00:09:42.489 INFO:teuthology.orchestra.run.smithi123.stdout: librados2 librbd1 2020-01-16T00:09:42.489 INFO:teuthology.orchestra.run.smithi145.stdout:Suggested packages: 2020-01-16T00:09:42.489 INFO:teuthology.orchestra.run.smithi145.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2020-01-16T00:09:42.490 INFO:teuthology.orchestra.run.smithi145.stdout: python-pytest-xdist python-mock python-cryptography-doc 2020-01-16T00:09:42.490 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi python3-crypto 2020-01-16T00:09:42.490 INFO:teuthology.orchestra.run.smithi145.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc dvipng 2020-01-16T00:09:42.490 INFO:teuthology.orchestra.run.smithi145.stdout: ffmpeg gir1.2-gtk-3.0 ghostscript inkscape ipython3 python-matplotlib-doc 2020-01-16T00:09:42.490 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cairocffi python3-gi-cairo python3-gobject python3-pyqt4 python3-sip 2020-01-16T00:09:42.490 INFO:teuthology.orchestra.run.smithi145.stdout: python3-tornado texlive-extra-utils texlive-latex-extra ttf-staypuft 2020-01-16T00:09:42.490 INFO:teuthology.orchestra.run.smithi145.stdout: python-numpy-doc python3-dev python3-numpy-dbg python-openssl-doc 2020-01-16T00:09:42.490 INFO:teuthology.orchestra.run.smithi145.stdout: python3-openssl-dbg httpd-wsgi libapache2-mod-python libapache2-mod-scgi 2020-01-16T00:09:42.491 INFO:teuthology.orchestra.run.smithi145.stdout: libjs-mochikit python-pecan-doc python-pil-doc python3-pil-dbg 2020-01-16T00:09:42.491 INFO:teuthology.orchestra.run.smithi145.stdout: python-pyinotify-doc python-pyparsing-doc python-scipy-doc 2020-01-16T00:09:42.491 INFO:teuthology.orchestra.run.smithi145.stdout: python-setuptools-doc python3-dap python-sklearn-doc python-sqlalchemy-doc 2020-01-16T00:09:42.491 INFO:teuthology.orchestra.run.smithi145.stdout: python3-psycopg2 python3-pymysql python3-fdb tix python3-tk-dbg 2020-01-16T00:09:42.491 INFO:teuthology.orchestra.run.smithi145.stdout: python-waitress-doc python-webob-doc python-webtest-doc python3-termcolor 2020-01-16T00:09:42.491 INFO:teuthology.orchestra.run.smithi145.stdout: python3-watchdog python-werkzeug-doc 2020-01-16T00:09:42.585 INFO:teuthology.orchestra.run.smithi145.stdout:The following NEW packages will be installed: 2020-01-16T00:09:42.585 INFO:teuthology.orchestra.run.smithi145.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2020-01-16T00:09:42.586 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2020-01-16T00:09:42.586 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-rook ceph-mon 2020-01-16T00:09:42.586 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-osd ceph-test cephadm jq libcephfs-dev libcephfs2 libjq1 libleveldb1v5 2020-01-16T00:09:42.586 INFO:teuthology.orchestra.run.smithi145.stdout: liblttng-ust0 liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 2020-01-16T00:09:42.586 INFO:teuthology.orchestra.run.smithi145.stdout: librgw2 libsnappy1v5 libxslt1.1 python-attr python-funcsigs 2020-01-16T00:09:42.586 INFO:teuthology.orchestra.run.smithi145.stdout: python-pastedeploy-tpl python-pluggy python-py python-pytest 2020-01-16T00:09:42.586 INFO:teuthology.orchestra.run.smithi145.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-01-16T00:09:42.586 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2020-01-16T00:09:42.587 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2020-01-16T00:09:42.587 INFO:teuthology.orchestra.run.smithi145.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2020-01-16T00:09:42.587 INFO:teuthology.orchestra.run.smithi145.stdout: python3-httplib2 python3-joblib python3-jwt python3-kubernetes 2020-01-16T00:09:42.587 INFO:teuthology.orchestra.run.smithi145.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2020-01-16T00:09:42.587 INFO:teuthology.orchestra.run.smithi145.stdout: python3-matplotlib python3-monotonic python3-numpy python3-oauth2client 2020-01-16T00:09:42.587 INFO:teuthology.orchestra.run.smithi145.stdout: python3-olefile python3-openssl python3-paste python3-pastedeploy 2020-01-16T00:09:42.587 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pastescript python3-pecan python3-pil python3-pluggy 2020-01-16T00:09:42.587 INFO:teuthology.orchestra.run.smithi145.stdout: python3-prettytable python3-py python3-pyasn1 python3-pyasn1-modules 2020-01-16T00:09:42.588 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyinotify python3-pyparsing python3-pytest python3-rados python3-rbd 2020-01-16T00:09:42.588 INFO:teuthology.orchestra.run.smithi145.stdout: python3-repoze.lru python3-rgw python3-routes python3-rsa python3-scipy 2020-01-16T00:09:42.588 INFO:teuthology.orchestra.run.smithi145.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-01-16T00:09:42.588 INFO:teuthology.orchestra.run.smithi145.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-01-16T00:09:42.588 INFO:teuthology.orchestra.run.smithi145.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-01-16T00:09:42.588 INFO:teuthology.orchestra.run.smithi145.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-01-16T00:09:42.588 INFO:teuthology.orchestra.run.smithi145.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2020-01-16T00:09:42.589 INFO:teuthology.orchestra.run.smithi145.stdout:The following packages will be upgraded: 2020-01-16T00:09:42.589 INFO:teuthology.orchestra.run.smithi145.stdout: librados2 librbd1 2020-01-16T00:09:42.636 INFO:teuthology.orchestra.run.smithi123.stdout:2 upgraded, 106 newly installed, 0 to remove and 28 not upgraded. 2020-01-16T00:09:42.636 INFO:teuthology.orchestra.run.smithi123.stdout:Need to get 151 MB of archives. 2020-01-16T00:09:42.636 INFO:teuthology.orchestra.run.smithi123.stdout:After this operation, 703 MB of additional disk space will be used. 2020-01-16T00:09:42.636 INFO:teuthology.orchestra.run.smithi123.stdout:Get:1 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 15.0.0-9353-gcb3c85e-1bionic [1,581 kB] 2020-01-16T00:09:42.676 INFO:teuthology.orchestra.run.smithi123.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2020-01-16T00:09:42.732 INFO:teuthology.orchestra.run.smithi145.stdout:2 upgraded, 106 newly installed, 0 to remove and 28 not upgraded. 2020-01-16T00:09:42.733 INFO:teuthology.orchestra.run.smithi145.stdout:Need to get 151 MB of archives. 2020-01-16T00:09:42.733 INFO:teuthology.orchestra.run.smithi145.stdout:After this operation, 703 MB of additional disk space will be used. 2020-01-16T00:09:42.733 INFO:teuthology.orchestra.run.smithi145.stdout:Get:1 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 15.0.0-9353-gcb3c85e-1bionic [1,581 kB] 2020-01-16T00:09:42.756 INFO:teuthology.orchestra.run.smithi123.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2020-01-16T00:09:42.774 INFO:teuthology.orchestra.run.smithi145.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2020-01-16T00:09:42.791 INFO:teuthology.orchestra.run.smithi123.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 15.0.0-9353-gcb3c85e-1bionic [3,050 kB] 2020-01-16T00:09:42.856 INFO:teuthology.orchestra.run.smithi145.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2020-01-16T00:09:42.895 INFO:teuthology.orchestra.run.smithi123.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 15.0.0-9353-gcb3c85e-1bionic [432 kB] 2020-01-16T00:09:42.910 INFO:teuthology.orchestra.run.smithi123.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 15.0.0-9353-gcb3c85e-1bionic [331 kB] 2020-01-16T00:09:42.920 INFO:teuthology.orchestra.run.smithi123.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 15.0.0-9353-gcb3c85e-1bionic [20.5 kB] 2020-01-16T00:09:42.921 INFO:teuthology.orchestra.run.smithi123.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 15.0.0-9353-gcb3c85e-1bionic [118 kB] 2020-01-16T00:09:42.925 INFO:teuthology.orchestra.run.smithi123.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 15.0.0-9353-gcb3c85e-1bionic [18.3 kB] 2020-01-16T00:09:42.926 INFO:teuthology.orchestra.run.smithi123.stdout:Get:10 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 15.0.0-9353-gcb3c85e-1bionic [276 kB] 2020-01-16T00:09:42.934 INFO:teuthology.orchestra.run.smithi123.stdout:Get:11 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 15.0.0-9353-gcb3c85e-1bionic [4,618 kB] 2020-01-16T00:09:43.025 INFO:teuthology.orchestra.run.smithi123.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2020-01-16T00:09:43.039 INFO:teuthology.orchestra.run.smithi123.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2020-01-16T00:09:43.062 INFO:teuthology.orchestra.run.smithi123.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2020-01-16T00:09:43.110 INFO:teuthology.orchestra.run.smithi123.stdout:Get:15 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 15.0.0-9353-gcb3c85e-1bionic [99.2 kB] 2020-01-16T00:09:43.113 INFO:teuthology.orchestra.run.smithi145.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2020-01-16T00:09:43.114 INFO:teuthology.orchestra.run.smithi123.stdout:Get:16 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 15.0.0-9353-gcb3c85e-1bionic [334 kB] 2020-01-16T00:09:43.125 INFO:teuthology.orchestra.run.smithi145.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2020-01-16T00:09:43.126 INFO:teuthology.orchestra.run.smithi123.stdout:Get:17 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 15.0.0-9353-gcb3c85e-1bionic [18.0 MB] 2020-01-16T00:09:43.135 INFO:teuthology.orchestra.run.smithi145.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2020-01-16T00:09:43.168 INFO:teuthology.orchestra.run.smithi123.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2020-01-16T00:09:43.171 INFO:teuthology.orchestra.run.smithi123.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2020-01-16T00:09:43.199 INFO:teuthology.orchestra.run.smithi123.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2020-01-16T00:09:43.206 INFO:teuthology.orchestra.run.smithi123.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2020-01-16T00:09:43.217 INFO:teuthology.orchestra.run.smithi123.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2020-01-16T00:09:43.221 INFO:teuthology.orchestra.run.smithi123.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2020-01-16T00:09:43.239 INFO:teuthology.orchestra.run.smithi145.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2020-01-16T00:09:43.242 INFO:teuthology.orchestra.run.smithi145.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2020-01-16T00:09:43.243 INFO:teuthology.orchestra.run.smithi123.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2020-01-16T00:09:43.245 INFO:teuthology.orchestra.run.smithi123.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2020-01-16T00:09:43.253 INFO:teuthology.orchestra.run.smithi123.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.3 [221 kB] 2020-01-16T00:09:43.271 INFO:teuthology.orchestra.run.smithi145.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2020-01-16T00:09:43.276 INFO:teuthology.orchestra.run.smithi123.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2020-01-16T00:09:43.279 INFO:teuthology.orchestra.run.smithi123.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2020-01-16T00:09:43.287 INFO:teuthology.orchestra.run.smithi145.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2020-01-16T00:09:43.290 INFO:teuthology.orchestra.run.smithi145.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2020-01-16T00:09:43.295 INFO:teuthology.orchestra.run.smithi145.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2020-01-16T00:09:43.319 INFO:teuthology.orchestra.run.smithi145.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2020-01-16T00:09:43.320 INFO:teuthology.orchestra.run.smithi145.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2020-01-16T00:09:43.325 INFO:teuthology.orchestra.run.smithi145.stdout:Get:15 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 15.0.0-9353-gcb3c85e-1bionic [3,050 kB] 2020-01-16T00:09:43.328 INFO:teuthology.orchestra.run.smithi145.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.3 [221 kB] 2020-01-16T00:09:43.349 INFO:teuthology.orchestra.run.smithi123.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2020-01-16T00:09:43.350 INFO:teuthology.orchestra.run.smithi123.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2020-01-16T00:09:43.352 INFO:teuthology.orchestra.run.smithi123.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2020-01-16T00:09:43.353 INFO:teuthology.orchestra.run.smithi123.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2020-01-16T00:09:43.353 INFO:teuthology.orchestra.run.smithi123.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2020-01-16T00:09:43.356 INFO:teuthology.orchestra.run.smithi145.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2020-01-16T00:09:43.358 INFO:teuthology.orchestra.run.smithi145.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2020-01-16T00:09:43.360 INFO:teuthology.orchestra.run.smithi145.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2020-01-16T00:09:43.436 INFO:teuthology.orchestra.run.smithi145.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2020-01-16T00:09:43.438 INFO:teuthology.orchestra.run.smithi145.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2020-01-16T00:09:43.439 INFO:teuthology.orchestra.run.smithi145.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2020-01-16T00:09:43.439 INFO:teuthology.orchestra.run.smithi145.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2020-01-16T00:09:43.517 INFO:teuthology.orchestra.run.smithi123.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2020-01-16T00:09:43.603 INFO:teuthology.orchestra.run.smithi145.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2020-01-16T00:09:43.788 INFO:teuthology.orchestra.run.smithi123.stdout:Get:35 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 15.0.0-9353-gcb3c85e-1bionic [5,125 kB] 2020-01-16T00:09:43.808 INFO:teuthology.orchestra.run.smithi123.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2020-01-16T00:09:43.853 INFO:teuthology.orchestra.run.smithi123.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2020-01-16T00:09:43.869 INFO:teuthology.orchestra.run.smithi123.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2020-01-16T00:09:43.885 INFO:teuthology.orchestra.run.smithi145.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2020-01-16T00:09:43.906 INFO:teuthology.orchestra.run.smithi145.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2020-01-16T00:09:43.966 INFO:teuthology.orchestra.run.smithi145.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2020-01-16T00:09:43.974 INFO:teuthology.orchestra.run.smithi123.stdout:Get:39 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 15.0.0-9353-gcb3c85e-1bionic [1,396 kB] 2020-01-16T00:09:44.027 INFO:teuthology.orchestra.run.smithi123.stdout:Get:40 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 15.0.0-9353-gcb3c85e-1bionic [5,934 kB] 2020-01-16T00:09:44.082 INFO:teuthology.orchestra.run.smithi123.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2020-01-16T00:09:44.082 INFO:teuthology.orchestra.run.smithi123.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2020-01-16T00:09:44.083 INFO:teuthology.orchestra.run.smithi123.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2020-01-16T00:09:44.086 INFO:teuthology.orchestra.run.smithi123.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2020-01-16T00:09:44.101 INFO:teuthology.orchestra.run.smithi123.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2020-01-16T00:09:44.125 INFO:teuthology.orchestra.run.smithi123.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2020-01-16T00:09:44.126 INFO:teuthology.orchestra.run.smithi123.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2020-01-16T00:09:44.131 INFO:teuthology.orchestra.run.smithi123.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2020-01-16T00:09:44.186 INFO:teuthology.orchestra.run.smithi145.stdout:Get:28 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 15.0.0-9353-gcb3c85e-1bionic [432 kB] 2020-01-16T00:09:44.199 INFO:teuthology.orchestra.run.smithi145.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2020-01-16T00:09:44.200 INFO:teuthology.orchestra.run.smithi145.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2020-01-16T00:09:44.203 INFO:teuthology.orchestra.run.smithi145.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2020-01-16T00:09:44.208 INFO:teuthology.orchestra.run.smithi145.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2020-01-16T00:09:44.224 INFO:teuthology.orchestra.run.smithi145.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2020-01-16T00:09:44.245 INFO:teuthology.orchestra.run.smithi123.stdout:Get:49 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 15.0.0-9353-gcb3c85e-1bionic [22.7 MB] 2020-01-16T00:09:44.253 INFO:teuthology.orchestra.run.smithi145.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2020-01-16T00:09:44.255 INFO:teuthology.orchestra.run.smithi145.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2020-01-16T00:09:44.258 INFO:teuthology.orchestra.run.smithi145.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2020-01-16T00:09:44.277 INFO:teuthology.orchestra.run.smithi123.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2020-01-16T00:09:44.278 INFO:teuthology.orchestra.run.smithi123.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2020-01-16T00:09:44.314 INFO:teuthology.orchestra.run.smithi145.stdout:Get:37 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 15.0.0-9353-gcb3c85e-1bionic [331 kB] 2020-01-16T00:09:44.384 INFO:teuthology.orchestra.run.smithi145.stdout:Get:38 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 15.0.0-9353-gcb3c85e-1bionic [20.5 kB] 2020-01-16T00:09:44.385 INFO:teuthology.orchestra.run.smithi145.stdout:Get:39 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 15.0.0-9353-gcb3c85e-1bionic [118 kB] 2020-01-16T00:09:44.394 INFO:teuthology.orchestra.run.smithi145.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2020-01-16T00:09:44.395 INFO:teuthology.orchestra.run.smithi145.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2020-01-16T00:09:44.428 INFO:teuthology.orchestra.run.smithi145.stdout:Get:42 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 15.0.0-9353-gcb3c85e-1bionic [18.3 kB] 2020-01-16T00:09:44.444 INFO:teuthology.orchestra.run.smithi145.stdout:Get:43 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 15.0.0-9353-gcb3c85e-1bionic [276 kB] 2020-01-16T00:09:44.512 INFO:teuthology.orchestra.run.smithi145.stdout:Get:44 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 15.0.0-9353-gcb3c85e-1bionic [4,618 kB] 2020-01-16T00:09:44.558 INFO:teuthology.orchestra.run.smithi123.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2020-01-16T00:09:44.592 INFO:teuthology.orchestra.run.smithi123.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2020-01-16T00:09:44.594 INFO:teuthology.orchestra.run.smithi123.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2020-01-16T00:09:44.636 INFO:teuthology.orchestra.run.smithi123.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2020-01-16T00:09:44.638 INFO:teuthology.orchestra.run.smithi123.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2020-01-16T00:09:44.638 INFO:teuthology.orchestra.run.smithi123.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2020-01-16T00:09:44.638 INFO:teuthology.orchestra.run.smithi123.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.1 [28.6 kB] 2020-01-16T00:09:44.653 INFO:teuthology.orchestra.run.smithi145.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2020-01-16T00:09:44.694 INFO:teuthology.orchestra.run.smithi145.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2020-01-16T00:09:44.694 INFO:teuthology.orchestra.run.smithi145.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2020-01-16T00:09:44.735 INFO:teuthology.orchestra.run.smithi145.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2020-01-16T00:09:44.736 INFO:teuthology.orchestra.run.smithi145.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2020-01-16T00:09:44.737 INFO:teuthology.orchestra.run.smithi145.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2020-01-16T00:09:44.737 INFO:teuthology.orchestra.run.smithi145.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.1 [28.6 kB] 2020-01-16T00:09:44.800 INFO:teuthology.orchestra.run.smithi123.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2020-01-16T00:09:44.898 INFO:teuthology.orchestra.run.smithi145.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2020-01-16T00:09:45.057 INFO:teuthology.orchestra.run.smithi123.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2020-01-16T00:09:45.085 INFO:teuthology.orchestra.run.smithi123.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2020-01-16T00:09:45.096 INFO:teuthology.orchestra.run.smithi123.stdout:Get:62 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 15.0.0-9353-gcb3c85e-1bionic [3,760 B] 2020-01-16T00:09:45.096 INFO:teuthology.orchestra.run.smithi123.stdout:Get:63 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 15.0.0-9353-gcb3c85e-1bionic [511 kB] 2020-01-16T00:09:45.109 INFO:teuthology.orchestra.run.smithi123.stdout:Get:64 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 15.0.0-9353-gcb3c85e-1bionic [1,829 kB] 2020-01-16T00:09:45.138 INFO:teuthology.orchestra.run.smithi123.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2020-01-16T00:09:45.139 INFO:teuthology.orchestra.run.smithi145.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2020-01-16T00:09:45.152 INFO:teuthology.orchestra.run.smithi145.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2020-01-16T00:09:45.178 INFO:teuthology.orchestra.run.smithi123.stdout:Get:66 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 15.0.0-9353-gcb3c85e-1bionic [24.1 kB] 2020-01-16T00:09:45.179 INFO:teuthology.orchestra.run.smithi123.stdout:Get:67 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 15.0.0-9353-gcb3c85e-1bionic [19.8 kB] 2020-01-16T00:09:45.180 INFO:teuthology.orchestra.run.smithi123.stdout:Get:68 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 15.0.0-9353-gcb3c85e-1bionic [3,415 kB] 2020-01-16T00:09:45.219 INFO:teuthology.orchestra.run.smithi145.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2020-01-16T00:09:45.309 INFO:teuthology.orchestra.run.smithi123.stdout:Get:69 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-cloud all 15.0.0-9353-gcb3c85e-1bionic [34.1 kB] 2020-01-16T00:09:45.310 INFO:teuthology.orchestra.run.smithi123.stdout:Get:70 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 15.0.0-9353-gcb3c85e-1bionic [7,889 kB] 2020-01-16T00:09:45.390 INFO:teuthology.orchestra.run.smithi123.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2020-01-16T00:09:45.405 INFO:teuthology.orchestra.run.smithi123.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2020-01-16T00:09:45.435 INFO:teuthology.orchestra.run.smithi123.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2020-01-16T00:09:45.438 INFO:teuthology.orchestra.run.smithi123.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2020-01-16T00:09:45.448 INFO:teuthology.orchestra.run.smithi145.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2020-01-16T00:09:45.454 INFO:teuthology.orchestra.run.smithi123.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2020-01-16T00:09:45.458 INFO:teuthology.orchestra.run.smithi145.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2020-01-16T00:09:45.465 INFO:teuthology.orchestra.run.smithi123.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2020-01-16T00:09:45.472 INFO:teuthology.orchestra.run.smithi123.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2020-01-16T00:09:45.475 INFO:teuthology.orchestra.run.smithi123.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2020-01-16T00:09:45.484 INFO:teuthology.orchestra.run.smithi145.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2020-01-16T00:09:45.486 INFO:teuthology.orchestra.run.smithi145.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2020-01-16T00:09:45.500 INFO:teuthology.orchestra.run.smithi123.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2020-01-16T00:09:45.504 INFO:teuthology.orchestra.run.smithi145.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2020-01-16T00:09:45.509 INFO:teuthology.orchestra.run.smithi123.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2020-01-16T00:09:45.516 INFO:teuthology.orchestra.run.smithi145.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2020-01-16T00:09:45.525 INFO:teuthology.orchestra.run.smithi145.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2020-01-16T00:09:45.529 INFO:teuthology.orchestra.run.smithi145.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2020-01-16T00:09:45.551 INFO:teuthology.orchestra.run.smithi123.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2020-01-16T00:09:45.552 INFO:teuthology.orchestra.run.smithi123.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2020-01-16T00:09:45.553 INFO:teuthology.orchestra.run.smithi145.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2020-01-16T00:09:45.553 INFO:teuthology.orchestra.run.smithi123.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2020-01-16T00:09:45.553 INFO:teuthology.orchestra.run.smithi123.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2020-01-16T00:09:45.555 INFO:teuthology.orchestra.run.smithi123.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2020-01-16T00:09:45.560 INFO:teuthology.orchestra.run.smithi145.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2020-01-16T00:09:45.605 INFO:teuthology.orchestra.run.smithi123.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 15.0.0-9353-gcb3c85e-1bionic [3,876 B] 2020-01-16T00:09:45.605 INFO:teuthology.orchestra.run.smithi123.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 15.0.0-9353-gcb3c85e-1bionic [13.3 kB] 2020-01-16T00:09:45.605 INFO:teuthology.orchestra.run.smithi123.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 15.0.0-9353-gcb3c85e-1bionic [41.5 MB] 2020-01-16T00:09:45.609 INFO:teuthology.orchestra.run.smithi145.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2020-01-16T00:09:45.610 INFO:teuthology.orchestra.run.smithi145.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2020-01-16T00:09:45.611 INFO:teuthology.orchestra.run.smithi145.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2020-01-16T00:09:45.611 INFO:teuthology.orchestra.run.smithi145.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2020-01-16T00:09:45.614 INFO:teuthology.orchestra.run.smithi145.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2020-01-16T00:09:45.618 INFO:teuthology.orchestra.run.smithi145.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2020-01-16T00:09:45.632 INFO:teuthology.orchestra.run.smithi123.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2020-01-16T00:09:45.633 INFO:teuthology.orchestra.run.smithi123.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2020-01-16T00:09:45.689 INFO:teuthology.orchestra.run.smithi145.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2020-01-16T00:09:45.768 INFO:teuthology.orchestra.run.smithi145.stdout:Get:73 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 15.0.0-9353-gcb3c85e-1bionic [99.2 kB] 2020-01-16T00:09:45.773 INFO:teuthology.orchestra.run.smithi145.stdout:Get:74 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 15.0.0-9353-gcb3c85e-1bionic [334 kB] 2020-01-16T00:09:45.795 INFO:teuthology.orchestra.run.smithi123.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2020-01-16T00:09:45.848 INFO:teuthology.orchestra.run.smithi145.stdout:Get:75 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 15.0.0-9353-gcb3c85e-1bionic [18.0 MB] 2020-01-16T00:09:45.851 INFO:teuthology.orchestra.run.smithi145.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2020-01-16T00:09:45.876 INFO:teuthology.orchestra.run.smithi123.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2020-01-16T00:09:45.932 INFO:teuthology.orchestra.run.smithi145.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2020-01-16T00:09:46.057 INFO:teuthology.orchestra.run.smithi123.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.1 [1,091 kB] 2020-01-16T00:09:46.098 INFO:teuthology.orchestra.run.smithi145.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.1 [1,091 kB] 2020-01-16T00:09:46.358 INFO:teuthology.orchestra.run.smithi123.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2020-01-16T00:09:46.365 INFO:teuthology.orchestra.run.smithi123.stdout:Get:95 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2020-01-16T00:09:46.365 INFO:teuthology.orchestra.run.smithi123.stdout:Get:96 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2020-01-16T00:09:46.379 INFO:teuthology.orchestra.run.smithi145.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2020-01-16T00:09:46.384 INFO:teuthology.orchestra.run.smithi145.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2020-01-16T00:09:46.386 INFO:teuthology.orchestra.run.smithi145.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2020-01-16T00:09:46.532 INFO:teuthology.orchestra.run.smithi123.stdout:Get:97 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2020-01-16T00:09:46.532 INFO:teuthology.orchestra.run.smithi123.stdout:Get:98 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2020-01-16T00:09:46.532 INFO:teuthology.orchestra.run.smithi123.stdout:Get:99 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pil amd64 5.1.0-1 [328 kB] 2020-01-16T00:09:46.546 INFO:teuthology.orchestra.run.smithi123.stdout:Get:100 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2020-01-16T00:09:46.546 INFO:teuthology.orchestra.run.smithi123.stdout:Get:101 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2020-01-16T00:09:46.548 INFO:teuthology.orchestra.run.smithi123.stdout:Get:102 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2020-01-16T00:09:46.548 INFO:teuthology.orchestra.run.smithi123.stdout:Get:103 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2020-01-16T00:09:46.551 INFO:teuthology.orchestra.run.smithi123.stdout:Get:104 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2020-01-16T00:09:46.552 INFO:teuthology.orchestra.run.smithi145.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2020-01-16T00:09:46.553 INFO:teuthology.orchestra.run.smithi123.stdout:Get:105 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2020-01-16T00:09:46.553 INFO:teuthology.orchestra.run.smithi145.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2020-01-16T00:09:46.554 INFO:teuthology.orchestra.run.smithi145.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pil amd64 5.1.0-1 [328 kB] 2020-01-16T00:09:46.558 INFO:teuthology.orchestra.run.smithi145.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2020-01-16T00:09:46.558 INFO:teuthology.orchestra.run.smithi145.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2020-01-16T00:09:46.559 INFO:teuthology.orchestra.run.smithi145.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2020-01-16T00:09:46.560 INFO:teuthology.orchestra.run.smithi145.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2020-01-16T00:09:46.563 INFO:teuthology.orchestra.run.smithi145.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2020-01-16T00:09:46.565 INFO:teuthology.orchestra.run.smithi145.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2020-01-16T00:09:47.161 INFO:teuthology.orchestra.run.smithi123.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 15.0.0-9353-gcb3c85e-1bionic [17.2 kB] 2020-01-16T00:09:47.161 INFO:teuthology.orchestra.run.smithi123.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 15.0.0-9353-gcb3c85e-1bionic [5,509 kB] 2020-01-16T00:09:47.355 INFO:teuthology.orchestra.run.smithi123.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 15.0.0-9353-gcb3c85e-1bionic [61.9 kB] 2020-01-16T00:09:48.056 INFO:teuthology.orchestra.run.smithi123.stdout:Fetched 151 MB in 5s (31.1 MB/s) 2020-01-16T00:09:48.179 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-uritemplate. 2020-01-16T00:09:48.229 INFO:teuthology.orchestra.run.smithi123.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137449 files and directories currently installed.) 2020-01-16T00:09:48.233 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2020-01-16T00:09:48.256 INFO:teuthology.orchestra.run.smithi145.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 15.0.0-9353-gcb3c85e-1bionic [5,125 kB] 2020-01-16T00:09:48.278 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2020-01-16T00:09:48.589 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2020-01-16T00:09:48.604 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2020-01-16T00:09:48.632 INFO:teuthology.orchestra.run.smithi145.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 15.0.0-9353-gcb3c85e-1bionic [1,396 kB] 2020-01-16T00:09:48.689 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2020-01-16T00:09:48.732 INFO:teuthology.orchestra.run.smithi145.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 15.0.0-9353-gcb3c85e-1bionic [5,934 kB] 2020-01-16T00:09:49.040 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../002-librbd1_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:09:49.142 INFO:teuthology.orchestra.run.smithi145.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 15.0.0-9353-gcb3c85e-1bionic [22.7 MB] 2020-01-16T00:09:49.168 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking librbd1 (15.0.0-9353-gcb3c85e-1bionic) over (12.2.12-0ubuntu0.18.04.3) ... 2020-01-16T00:09:49.643 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../003-librados2_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:09:49.738 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking librados2 (15.0.0-9353-gcb3c85e-1bionic) over (12.2.12-0ubuntu0.18.04.3) ... 2020-01-16T00:09:50.375 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libcephfs2. 2020-01-16T00:09:50.389 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../004-libcephfs2_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:09:50.434 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libcephfs2 (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:09:50.563 INFO:teuthology.orchestra.run.smithi145.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 15.0.0-9353-gcb3c85e-1bionic [3,760 B] 2020-01-16T00:09:50.564 INFO:teuthology.orchestra.run.smithi145.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 15.0.0-9353-gcb3c85e-1bionic [511 kB] 2020-01-16T00:09:50.589 INFO:teuthology.orchestra.run.smithi145.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 15.0.0-9353-gcb3c85e-1bionic [1,829 kB] 2020-01-16T00:09:50.697 INFO:teuthology.orchestra.run.smithi145.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 15.0.0-9353-gcb3c85e-1bionic [24.1 kB] 2020-01-16T00:09:50.697 INFO:teuthology.orchestra.run.smithi145.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 15.0.0-9353-gcb3c85e-1bionic [19.8 kB] 2020-01-16T00:09:50.697 INFO:teuthology.orchestra.run.smithi145.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 15.0.0-9353-gcb3c85e-1bionic [3,415 kB] 2020-01-16T00:09:50.728 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-rados. 2020-01-16T00:09:50.741 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../005-python3-rados_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:09:50.778 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-rados (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:09:50.891 INFO:teuthology.orchestra.run.smithi145.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-cloud all 15.0.0-9353-gcb3c85e-1bionic [34.1 kB] 2020-01-16T00:09:50.892 INFO:teuthology.orchestra.run.smithi145.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 15.0.0-9353-gcb3c85e-1bionic [7,889 kB] 2020-01-16T00:09:51.105 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-ceph-argparse. 2020-01-16T00:09:51.121 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../006-python3-ceph-argparse_15.0.0-9353-gcb3c85e-1bionic_all.deb ... 2020-01-16T00:09:51.155 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-ceph-argparse (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:09:51.332 INFO:teuthology.orchestra.run.smithi145.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 15.0.0-9353-gcb3c85e-1bionic [3,876 B] 2020-01-16T00:09:51.333 INFO:teuthology.orchestra.run.smithi145.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 15.0.0-9353-gcb3c85e-1bionic [13.3 kB] 2020-01-16T00:09:51.333 INFO:teuthology.orchestra.run.smithi145.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 15.0.0-9353-gcb3c85e-1bionic [41.5 MB] 2020-01-16T00:09:51.441 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-cephfs. 2020-01-16T00:09:51.454 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../007-python3-cephfs_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:09:51.491 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-cephfs (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:09:51.802 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-ceph-common. 2020-01-16T00:09:51.818 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../008-python3-ceph-common_15.0.0-9353-gcb3c85e-1bionic_all.deb ... 2020-01-16T00:09:51.860 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-ceph-common (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:09:52.172 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-prettytable. 2020-01-16T00:09:52.187 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2020-01-16T00:09:52.222 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2020-01-16T00:09:52.441 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-rbd. 2020-01-16T00:09:52.455 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../010-python3-rbd_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:09:52.491 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-rbd (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:09:52.852 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package librabbitmq4:amd64. 2020-01-16T00:09:52.866 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2020-01-16T00:09:52.911 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-01-16T00:09:53.238 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package librdkafka1:amd64. 2020-01-16T00:09:53.253 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2020-01-16T00:09:53.296 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2020-01-16T00:09:53.384 INFO:teuthology.orchestra.run.smithi145.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 15.0.0-9353-gcb3c85e-1bionic [17.2 kB] 2020-01-16T00:09:53.385 INFO:teuthology.orchestra.run.smithi145.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 15.0.0-9353-gcb3c85e-1bionic [5,509 kB] 2020-01-16T00:09:53.591 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package librgw2. 2020-01-16T00:09:53.607 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../013-librgw2_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:09:53.631 INFO:teuthology.orchestra.run.smithi145.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2020-01-15-1209/cb3c85e74500283e01cc592f7e2858e323186910/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 15.0.0-9353-gcb3c85e-1bionic [61.9 kB] 2020-01-16T00:09:53.641 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking librgw2 (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:09:54.260 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-rgw. 2020-01-16T00:09:54.274 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../014-python3-rgw_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:09:54.315 INFO:teuthology.orchestra.run.smithi145.stdout:Fetched 151 MB in 11s (13.7 MB/s) 2020-01-16T00:09:54.318 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-rgw (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:09:54.436 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-uritemplate. 2020-01-16T00:09:54.487 INFO:teuthology.orchestra.run.smithi145.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137449 files and directories currently installed.) 2020-01-16T00:09:54.492 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2020-01-16T00:09:54.568 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2020-01-16T00:09:54.629 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2020-01-16T00:09:54.644 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2020-01-16T00:09:54.688 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2020-01-16T00:09:54.862 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2020-01-16T00:09:54.878 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2020-01-16T00:09:54.962 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2020-01-16T00:09:54.965 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2020-01-16T00:09:54.981 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2020-01-16T00:09:55.015 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2020-01-16T00:09:55.312 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../002-librbd1_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:09:55.385 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package liboath0. 2020-01-16T00:09:55.398 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2020-01-16T00:09:55.435 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking liboath0 (2.6.1-1) ... 2020-01-16T00:09:55.475 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking librbd1 (15.0.0-9353-gcb3c85e-1bionic) over (12.2.12-0ubuntu0.18.04.3) ... 2020-01-16T00:09:55.730 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libradosstriper1. 2020-01-16T00:09:55.744 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../018-libradosstriper1_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:09:55.787 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libradosstriper1 (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:09:55.935 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../003-librados2_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:09:56.045 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking librados2 (15.0.0-9353-gcb3c85e-1bionic) over (12.2.12-0ubuntu0.18.04.3) ... 2020-01-16T00:09:56.182 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-common. 2020-01-16T00:09:56.198 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../019-ceph-common_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:09:56.232 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-common (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:09:56.666 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libcephfs2. 2020-01-16T00:09:56.682 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../004-libcephfs2_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:09:56.724 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libcephfs2 (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:09:57.026 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-rados. 2020-01-16T00:09:57.042 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../005-python3-rados_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:09:57.077 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-rados (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:09:57.396 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-ceph-argparse. 2020-01-16T00:09:57.409 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../006-python3-ceph-argparse_15.0.0-9353-gcb3c85e-1bionic_all.deb ... 2020-01-16T00:09:57.446 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-ceph-argparse (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:09:57.757 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-cephfs. 2020-01-16T00:09:57.771 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../007-python3-cephfs_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:09:57.815 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-cephfs (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:09:57.979 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-base. 2020-01-16T00:09:57.993 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../020-ceph-base_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:09:58.046 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-base (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:09:58.135 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-ceph-common. 2020-01-16T00:09:58.148 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../008-python3-ceph-common_15.0.0-9353-gcb3c85e-1bionic_all.deb ... 2020-01-16T00:09:58.185 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-ceph-common (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:09:58.446 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-prettytable. 2020-01-16T00:09:58.462 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2020-01-16T00:09:58.496 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2020-01-16T00:09:58.766 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-rbd. 2020-01-16T00:09:58.780 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../010-python3-rbd_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:09:58.824 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-rbd (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:09:58.917 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-cffi-backend. 2020-01-16T00:09:58.932 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../021-python3-cffi-backend_1.11.5-1_amd64.deb ... 2020-01-16T00:09:58.967 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2020-01-16T00:09:59.177 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package librabbitmq4:amd64. 2020-01-16T00:09:59.191 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2020-01-16T00:09:59.235 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-01-16T00:09:59.270 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-bcrypt. 2020-01-16T00:09:59.283 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../022-python3-bcrypt_3.1.4-2_amd64.deb ... 2020-01-16T00:09:59.320 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2020-01-16T00:09:59.514 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package librdkafka1:amd64. 2020-01-16T00:09:59.529 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2020-01-16T00:09:59.563 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2020-01-16T00:09:59.672 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-cherrypy3. 2020-01-16T00:09:59.686 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../023-python3-cherrypy3_8.9.1-2_all.deb ... 2020-01-16T00:09:59.731 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2020-01-16T00:09:59.891 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package librgw2. 2020-01-16T00:09:59.905 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../013-librgw2_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:09:59.941 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking librgw2 (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:00.085 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-jwt. 2020-01-16T00:10:00.098 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../024-python3-jwt_1.5.3+ds1-1_all.deb ... 2020-01-16T00:10:00.134 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2020-01-16T00:10:00.429 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-asn1crypto. 2020-01-16T00:10:00.443 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../025-python3-asn1crypto_0.24.0-1_all.deb ... 2020-01-16T00:10:00.487 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2020-01-16T00:10:00.578 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-rgw. 2020-01-16T00:10:00.593 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../014-python3-rgw_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:10:00.636 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-rgw (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:00.815 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-cryptography. 2020-01-16T00:10:00.829 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../026-python3-cryptography_2.1.4-1ubuntu1.3_amd64.deb ... 2020-01-16T00:10:00.873 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.3) ... 2020-01-16T00:10:00.930 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2020-01-16T00:10:00.944 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2020-01-16T00:10:00.980 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2020-01-16T00:10:01.235 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-openssl. 2020-01-16T00:10:01.241 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2020-01-16T00:10:01.251 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../027-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2020-01-16T00:10:01.257 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2020-01-16T00:10:01.293 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2020-01-16T00:10:01.300 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2020-01-16T00:10:01.636 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package liboath0. 2020-01-16T00:10:01.638 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-logutils. 2020-01-16T00:10:01.651 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2020-01-16T00:10:01.653 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../028-python3-logutils_0.3.3-5_all.deb ... 2020-01-16T00:10:01.686 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking liboath0 (2.6.1-1) ... 2020-01-16T00:10:01.696 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-logutils (0.3.3-5) ... 2020-01-16T00:10:01.980 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libradosstriper1. 2020-01-16T00:10:01.997 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../018-libradosstriper1_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:10:02.007 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-markupsafe. 2020-01-16T00:10:02.021 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../029-python3-markupsafe_1.0-1build1_amd64.deb ... 2020-01-16T00:10:02.039 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libradosstriper1 (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:02.065 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2020-01-16T00:10:02.385 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-mako. 2020-01-16T00:10:02.401 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../030-python3-mako_1.0.7+ds1-1_all.deb ... 2020-01-16T00:10:02.444 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2020-01-16T00:10:02.476 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-common. 2020-01-16T00:10:02.489 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../019-ceph-common_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:10:02.526 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-common (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:02.754 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-simplegeneric. 2020-01-16T00:10:02.770 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../031-python3-simplegeneric_0.8.1-1_all.deb ... 2020-01-16T00:10:02.812 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2020-01-16T00:10:03.115 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-singledispatch. 2020-01-16T00:10:03.129 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../032-python3-singledispatch_3.4.0.3-2_all.deb ... 2020-01-16T00:10:03.173 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2020-01-16T00:10:03.510 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-webob. 2020-01-16T00:10:03.524 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../033-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2020-01-16T00:10:03.568 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2020-01-16T00:10:03.895 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-bs4. 2020-01-16T00:10:03.909 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../034-python3-bs4_4.6.0-1_all.deb ... 2020-01-16T00:10:03.954 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2020-01-16T00:10:04.206 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-base. 2020-01-16T00:10:04.220 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../020-ceph-base_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:10:04.265 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-waitress. 2020-01-16T00:10:04.273 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-base (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:04.280 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../035-python3-waitress_1.0.1-1_all.deb ... 2020-01-16T00:10:04.323 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-waitress (1.0.1-1) ... 2020-01-16T00:10:04.651 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-tempita. 2020-01-16T00:10:04.665 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../036-python3-tempita_0.5.2-2_all.deb ... 2020-01-16T00:10:04.709 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-tempita (0.5.2-2) ... 2020-01-16T00:10:05.062 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-paste. 2020-01-16T00:10:05.078 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../037-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2020-01-16T00:10:05.120 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-01-16T00:10:05.127 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-cffi-backend. 2020-01-16T00:10:05.142 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../021-python3-cffi-backend_1.11.5-1_amd64.deb ... 2020-01-16T00:10:05.185 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2020-01-16T00:10:05.467 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2020-01-16T00:10:05.483 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../038-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2020-01-16T00:10:05.496 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-bcrypt. 2020-01-16T00:10:05.511 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../022-python3-bcrypt_3.1.4-2_amd64.deb ... 2020-01-16T00:10:05.525 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2020-01-16T00:10:05.555 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2020-01-16T00:10:05.824 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-cherrypy3. 2020-01-16T00:10:05.837 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../023-python3-cherrypy3_8.9.1-2_all.deb ... 2020-01-16T00:10:05.842 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-pastedeploy. 2020-01-16T00:10:05.855 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../039-python3-pastedeploy_1.5.2-4_all.deb ... 2020-01-16T00:10:05.866 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2020-01-16T00:10:05.892 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2020-01-16T00:10:06.178 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-jwt. 2020-01-16T00:10:06.191 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../024-python3-jwt_1.5.3+ds1-1_all.deb ... 2020-01-16T00:10:06.213 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-webtest. 2020-01-16T00:10:06.226 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../040-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2020-01-16T00:10:06.227 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2020-01-16T00:10:06.271 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2020-01-16T00:10:06.609 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-pecan. 2020-01-16T00:10:06.624 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../041-python3-pecan_1.2.1-2_all.deb ... 2020-01-16T00:10:06.667 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-pecan (1.2.1-2) ... 2020-01-16T00:10:06.772 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-asn1crypto. 2020-01-16T00:10:06.785 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../025-python3-asn1crypto_0.24.0-1_all.deb ... 2020-01-16T00:10:06.822 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2020-01-16T00:10:07.038 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-werkzeug. 2020-01-16T00:10:07.053 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../042-python3-werkzeug_0.14.1+dfsg1-1_all.deb ... 2020-01-16T00:10:07.091 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1) ... 2020-01-16T00:10:07.144 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-cryptography. 2020-01-16T00:10:07.159 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../026-python3-cryptography_2.1.4-1ubuntu1.3_amd64.deb ... 2020-01-16T00:10:07.202 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.3) ... 2020-01-16T00:10:07.453 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-mgr. 2020-01-16T00:10:07.470 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../043-ceph-mgr_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:10:07.512 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-mgr (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:07.572 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-openssl. 2020-01-16T00:10:07.589 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../027-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2020-01-16T00:10:07.631 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2020-01-16T00:10:07.967 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-logutils. 2020-01-16T00:10:07.980 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../028-python3-logutils_0.3.3-5_all.deb ... 2020-01-16T00:10:07.995 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-mon. 2020-01-16T00:10:08.014 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../044-ceph-mon_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:10:08.025 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-logutils (0.3.3-5) ... 2020-01-16T00:10:08.053 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-mon (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:08.320 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-markupsafe. 2020-01-16T00:10:08.335 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../029-python3-markupsafe_1.0-1build1_amd64.deb ... 2020-01-16T00:10:08.378 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2020-01-16T00:10:08.681 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-mako. 2020-01-16T00:10:08.694 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../030-python3-mako_1.0.7+ds1-1_all.deb ... 2020-01-16T00:10:08.731 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2020-01-16T00:10:08.971 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-osd. 2020-01-16T00:10:08.986 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../045-ceph-osd_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:10:09.029 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-osd (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:09.042 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-simplegeneric. 2020-01-16T00:10:09.056 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../031-python3-simplegeneric_0.8.1-1_all.deb ... 2020-01-16T00:10:09.100 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2020-01-16T00:10:09.420 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-singledispatch. 2020-01-16T00:10:09.433 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../032-python3-singledispatch_3.4.0.3-2_all.deb ... 2020-01-16T00:10:09.470 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2020-01-16T00:10:09.781 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-webob. 2020-01-16T00:10:09.795 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../033-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2020-01-16T00:10:09.839 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2020-01-16T00:10:10.158 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-bs4. 2020-01-16T00:10:10.174 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../034-python3-bs4_4.6.0-1_all.deb ... 2020-01-16T00:10:10.217 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2020-01-16T00:10:10.562 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-waitress. 2020-01-16T00:10:10.577 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../035-python3-waitress_1.0.1-1_all.deb ... 2020-01-16T00:10:10.620 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-waitress (1.0.1-1) ... 2020-01-16T00:10:10.973 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-tempita. 2020-01-16T00:10:10.989 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../036-python3-tempita_0.5.2-2_all.deb ... 2020-01-16T00:10:11.031 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-tempita (0.5.2-2) ... 2020-01-16T00:10:11.032 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph. 2020-01-16T00:10:11.046 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../046-ceph_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:10:11.091 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:11.334 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-paste. 2020-01-16T00:10:11.348 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../037-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2020-01-16T00:10:11.401 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-01-16T00:10:11.416 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-fuse. 2020-01-16T00:10:11.430 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../047-ceph-fuse_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:10:11.474 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-fuse (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:11.850 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2020-01-16T00:10:11.862 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-mds. 2020-01-16T00:10:11.863 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../038-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2020-01-16T00:10:11.878 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../048-ceph-mds_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:10:11.919 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2020-01-16T00:10:11.921 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-mds (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:12.213 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-pastedeploy. 2020-01-16T00:10:12.229 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../039-python3-pastedeploy_1.5.2-4_all.deb ... 2020-01-16T00:10:12.272 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2020-01-16T00:10:12.417 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package cephadm. 2020-01-16T00:10:12.430 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../049-cephadm_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:10:12.467 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking cephadm (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:12.632 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-webtest. 2020-01-16T00:10:12.647 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../040-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2020-01-16T00:10:12.690 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2020-01-16T00:10:12.811 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2020-01-16T00:10:12.825 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../050-ceph-mgr-cephadm_15.0.0-9353-gcb3c85e-1bionic_all.deb ... 2020-01-16T00:10:12.869 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-mgr-cephadm (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:13.032 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-pecan. 2020-01-16T00:10:13.047 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../041-python3-pecan_1.2.1-2_all.deb ... 2020-01-16T00:10:13.090 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-pecan (1.2.1-2) ... 2020-01-16T00:10:13.172 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-repoze.lru. 2020-01-16T00:10:13.185 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../051-python3-repoze.lru_0.7-1_all.deb ... 2020-01-16T00:10:13.230 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2020-01-16T00:10:13.447 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-werkzeug. 2020-01-16T00:10:13.460 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../042-python3-werkzeug_0.14.1+dfsg1-1_all.deb ... 2020-01-16T00:10:13.505 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1) ... 2020-01-16T00:10:13.533 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-routes. 2020-01-16T00:10:13.548 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../052-python3-routes_2.4.1-1_all.deb ... 2020-01-16T00:10:13.606 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-routes (2.4.1-1) ... 2020-01-16T00:10:13.894 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-mgr. 2020-01-16T00:10:13.908 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../043-ceph-mgr_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:10:13.927 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2020-01-16T00:10:13.941 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../053-ceph-mgr-dashboard_15.0.0-9353-gcb3c85e-1bionic_all.deb ... 2020-01-16T00:10:13.952 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-mgr (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:13.985 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-mgr-dashboard (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:14.478 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-mon. 2020-01-16T00:10:14.492 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../044-ceph-mon_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:10:14.539 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-mon (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:14.868 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2020-01-16T00:10:14.883 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../054-ceph-mgr-diskprediction-cloud_15.0.0-9353-gcb3c85e-1bionic_all.deb ... 2020-01-16T00:10:14.926 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:15.421 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-osd. 2020-01-16T00:10:15.435 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../045-ceph-osd_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:10:15.480 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-osd (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:15.871 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-numpy. 2020-01-16T00:10:15.887 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../055-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2020-01-16T00:10:15.929 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2020-01-16T00:10:16.488 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-decorator. 2020-01-16T00:10:16.504 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../056-python3-decorator_4.1.2-1_all.deb ... 2020-01-16T00:10:16.548 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-decorator (4.1.2-1) ... 2020-01-16T00:10:16.892 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-scipy. 2020-01-16T00:10:16.908 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../057-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2020-01-16T00:10:16.953 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2020-01-16T00:10:17.976 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph. 2020-01-16T00:10:17.989 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../046-ceph_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:10:18.035 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:18.128 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-sklearn-lib. 2020-01-16T00:10:18.146 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../058-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2020-01-16T00:10:18.187 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2020-01-16T00:10:18.401 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-fuse. 2020-01-16T00:10:18.416 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../047-ceph-fuse_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:10:18.468 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-fuse (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:18.657 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-joblib. 2020-01-16T00:10:18.672 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../059-python3-joblib_0.11-1_all.deb ... 2020-01-16T00:10:18.706 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-joblib (0.11-1) ... 2020-01-16T00:10:18.867 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-mds. 2020-01-16T00:10:18.880 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../048-ceph-mds_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:10:18.925 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-mds (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:19.060 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-sklearn. 2020-01-16T00:10:19.073 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../060-python3-sklearn_0.19.1-3_all.deb ... 2020-01-16T00:10:19.118 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2020-01-16T00:10:19.453 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package cephadm. 2020-01-16T00:10:19.467 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../049-cephadm_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:10:19.511 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking cephadm (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:19.624 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2020-01-16T00:10:19.640 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../061-ceph-mgr-diskprediction-local_15.0.0-9353-gcb3c85e-1bionic_all.deb ... 2020-01-16T00:10:19.674 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-mgr-diskprediction-local (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:19.814 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2020-01-16T00:10:19.828 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../050-ceph-mgr-cephadm_15.0.0-9353-gcb3c85e-1bionic_all.deb ... 2020-01-16T00:10:19.872 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-mgr-cephadm (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:20.200 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-repoze.lru. 2020-01-16T00:10:20.215 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../051-python3-repoze.lru_0.7-1_all.deb ... 2020-01-16T00:10:20.258 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2020-01-16T00:10:20.544 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-routes. 2020-01-16T00:10:20.559 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../052-python3-routes_2.4.1-1_all.deb ... 2020-01-16T00:10:20.611 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-routes (2.4.1-1) ... 2020-01-16T00:10:20.760 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-dateutil. 2020-01-16T00:10:20.776 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../062-python3-dateutil_2.6.1-1_all.deb ... 2020-01-16T00:10:20.827 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2020-01-16T00:10:20.931 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2020-01-16T00:10:20.944 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../053-ceph-mgr-dashboard_15.0.0-9353-gcb3c85e-1bionic_all.deb ... 2020-01-16T00:10:20.989 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-mgr-dashboard (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:21.147 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-monotonic. 2020-01-16T00:10:21.161 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../063-python3-monotonic_1.1-2_all.deb ... 2020-01-16T00:10:21.222 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-monotonic (1.1-2) ... 2020-01-16T00:10:21.483 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-fasteners. 2020-01-16T00:10:21.497 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../064-python3-fasteners_0.12.0-3_all.deb ... 2020-01-16T00:10:21.532 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2020-01-16T00:10:21.768 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-httplib2. 2020-01-16T00:10:21.785 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../065-python3-httplib2_0.9.2+dfsg-1ubuntu0.1_all.deb ... 2020-01-16T00:10:21.827 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.1) ... 2020-01-16T00:10:21.896 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2020-01-16T00:10:21.912 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../054-ceph-mgr-diskprediction-cloud_15.0.0-9353-gcb3c85e-1bionic_all.deb ... 2020-01-16T00:10:21.956 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:22.113 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-pyasn1. 2020-01-16T00:10:22.130 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../066-python3-pyasn1_0.4.2-3_all.deb ... 2020-01-16T00:10:22.171 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2020-01-16T00:10:22.266 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-numpy. 2020-01-16T00:10:22.280 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../055-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2020-01-16T00:10:22.324 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2020-01-16T00:10:22.524 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-pyasn1-modules. 2020-01-16T00:10:22.540 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../067-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2020-01-16T00:10:22.582 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2020-01-16T00:10:22.894 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-rsa. 2020-01-16T00:10:22.910 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../068-python3-rsa_3.4.2-1_all.deb ... 2020-01-16T00:10:22.955 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-decorator. 2020-01-16T00:10:22.960 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-rsa (3.4.2-1) ... 2020-01-16T00:10:22.972 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../056-python3-decorator_4.1.2-1_all.deb ... 2020-01-16T00:10:23.013 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-decorator (4.1.2-1) ... 2020-01-16T00:10:23.222 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-sqlalchemy. 2020-01-16T00:10:23.236 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../069-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2020-01-16T00:10:23.272 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-01-16T00:10:23.351 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-scipy. 2020-01-16T00:10:23.367 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../057-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2020-01-16T00:10:23.410 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2020-01-16T00:10:23.668 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-oauth2client. 2020-01-16T00:10:23.685 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../070-python3-oauth2client_4.1.2-3_all.deb ... 2020-01-16T00:10:23.717 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2020-01-16T00:10:24.038 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-setuptools. 2020-01-16T00:10:24.054 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../071-python3-setuptools_39.0.1-2_all.deb ... 2020-01-16T00:10:24.096 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2020-01-16T00:10:24.474 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-websocket. 2020-01-16T00:10:24.491 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../072-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2020-01-16T00:10:24.502 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-sklearn-lib. 2020-01-16T00:10:24.518 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../058-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2020-01-16T00:10:24.532 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2020-01-16T00:10:24.560 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2020-01-16T00:10:24.843 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-kubernetes. 2020-01-16T00:10:24.858 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../073-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2020-01-16T00:10:24.901 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2020-01-16T00:10:25.072 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-joblib. 2020-01-16T00:10:25.089 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../059-python3-joblib_0.11-1_all.deb ... 2020-01-16T00:10:25.139 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-joblib (0.11-1) ... 2020-01-16T00:10:25.331 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2020-01-16T00:10:25.345 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../074-ceph-mgr-k8sevents_15.0.0-9353-gcb3c85e-1bionic_all.deb ... 2020-01-16T00:10:25.389 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-mgr-k8sevents (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:25.500 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-sklearn. 2020-01-16T00:10:25.515 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../060-python3-sklearn_0.19.1-3_all.deb ... 2020-01-16T00:10:25.559 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2020-01-16T00:10:25.683 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-mgr-rook. 2020-01-16T00:10:25.697 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../075-ceph-mgr-rook_15.0.0-9353-gcb3c85e-1bionic_all.deb ... 2020-01-16T00:10:25.741 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-mgr-rook (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:26.027 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libonig4:amd64. 2020-01-16T00:10:26.041 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../076-libonig4_6.7.0-1_amd64.deb ... 2020-01-16T00:10:26.077 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2020-01-16T00:10:26.099 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2020-01-16T00:10:26.116 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../061-ceph-mgr-diskprediction-local_15.0.0-9353-gcb3c85e-1bionic_all.deb ... 2020-01-16T00:10:26.149 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-mgr-diskprediction-local (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:26.414 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libjq1:amd64. 2020-01-16T00:10:26.428 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../077-libjq1_1.5+dfsg-2_amd64.deb ... 2020-01-16T00:10:26.471 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2020-01-16T00:10:26.741 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package jq. 2020-01-16T00:10:26.758 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../078-jq_1.5+dfsg-2_amd64.deb ... 2020-01-16T00:10:26.799 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking jq (1.5+dfsg-2) ... 2020-01-16T00:10:27.052 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package socat. 2020-01-16T00:10:27.066 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../079-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2020-01-16T00:10:27.110 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2020-01-16T00:10:27.285 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-dateutil. 2020-01-16T00:10:27.300 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../062-python3-dateutil_2.6.1-1_all.deb ... 2020-01-16T00:10:27.343 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2020-01-16T00:10:27.430 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libxslt1.1:amd64. 2020-01-16T00:10:27.444 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../080-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2020-01-16T00:10:27.557 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-01-16T00:10:27.671 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-monotonic. 2020-01-16T00:10:27.687 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../063-python3-monotonic_1.1-2_all.deb ... 2020-01-16T00:10:27.730 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-monotonic (1.1-2) ... 2020-01-16T00:10:27.835 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package xmlstarlet. 2020-01-16T00:10:27.849 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../081-xmlstarlet_1.6.1-2_amd64.deb ... 2020-01-16T00:10:27.918 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2020-01-16T00:10:28.066 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-fasteners. 2020-01-16T00:10:28.080 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../064-python3-fasteners_0.12.0-3_all.deb ... 2020-01-16T00:10:28.124 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2020-01-16T00:10:28.266 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package ceph-test. 2020-01-16T00:10:28.281 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../082-ceph-test_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:10:28.324 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking ceph-test (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:28.427 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-httplib2. 2020-01-16T00:10:28.442 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../065-python3-httplib2_0.9.2+dfsg-1ubuntu0.1_all.deb ... 2020-01-16T00:10:28.486 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.1) ... 2020-01-16T00:10:28.772 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-pyasn1. 2020-01-16T00:10:28.788 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../066-python3-pyasn1_0.4.2-3_all.deb ... 2020-01-16T00:10:28.830 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2020-01-16T00:10:29.175 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-pyasn1-modules. 2020-01-16T00:10:29.190 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../067-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2020-01-16T00:10:29.233 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2020-01-16T00:10:29.586 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-rsa. 2020-01-16T00:10:29.600 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../068-python3-rsa_3.4.2-1_all.deb ... 2020-01-16T00:10:29.644 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-rsa (3.4.2-1) ... 2020-01-16T00:10:29.939 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-sqlalchemy. 2020-01-16T00:10:29.955 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../069-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2020-01-16T00:10:29.998 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-01-16T00:10:30.427 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-oauth2client. 2020-01-16T00:10:30.443 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../070-python3-oauth2client_4.1.2-3_all.deb ... 2020-01-16T00:10:30.485 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2020-01-16T00:10:30.813 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-setuptools. 2020-01-16T00:10:30.827 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../071-python3-setuptools_39.0.1-2_all.deb ... 2020-01-16T00:10:30.871 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2020-01-16T00:10:31.292 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-websocket. 2020-01-16T00:10:31.307 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../072-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2020-01-16T00:10:31.350 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2020-01-16T00:10:31.445 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package libcephfs-dev. 2020-01-16T00:10:31.461 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../083-libcephfs-dev_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:10:31.494 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking libcephfs-dev (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:31.670 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-kubernetes. 2020-01-16T00:10:31.684 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../073-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2020-01-16T00:10:31.728 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2020-01-16T00:10:31.756 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-attr. 2020-01-16T00:10:31.769 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../084-python-attr_17.4.0-2_all.deb ... 2020-01-16T00:10:31.814 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-attr (17.4.0-2) ... 2020-01-16T00:10:32.133 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-funcsigs. 2020-01-16T00:10:32.148 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../085-python-funcsigs_1.0.2-4_all.deb ... 2020-01-16T00:10:32.149 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2020-01-16T00:10:32.163 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../074-ceph-mgr-k8sevents_15.0.0-9353-gcb3c85e-1bionic_all.deb ... 2020-01-16T00:10:32.191 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2020-01-16T00:10:32.207 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-mgr-k8sevents (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:32.478 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-pluggy. 2020-01-16T00:10:32.484 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-mgr-rook. 2020-01-16T00:10:32.494 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../086-python-pluggy_0.6.0-1_all.deb ... 2020-01-16T00:10:32.501 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../075-ceph-mgr-rook_15.0.0-9353-gcb3c85e-1bionic_all.deb ... 2020-01-16T00:10:32.536 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-pluggy (0.6.0-1) ... 2020-01-16T00:10:32.568 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-mgr-rook (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:32.847 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-py. 2020-01-16T00:10:32.861 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../087-python-py_1.5.2-1_all.deb ... 2020-01-16T00:10:32.895 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libonig4:amd64. 2020-01-16T00:10:32.905 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-py (1.5.2-1) ... 2020-01-16T00:10:32.911 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../076-libonig4_6.7.0-1_amd64.deb ... 2020-01-16T00:10:32.962 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2020-01-16T00:10:33.217 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python-pytest. 2020-01-16T00:10:33.230 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../088-python-pytest_3.3.2-2_all.deb ... 2020-01-16T00:10:33.275 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python-pytest (3.3.2-2) ... 2020-01-16T00:10:33.315 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libjq1:amd64. 2020-01-16T00:10:33.329 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../077-libjq1_1.5+dfsg-2_amd64.deb ... 2020-01-16T00:10:33.373 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2020-01-16T00:10:33.645 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-attr. 2020-01-16T00:10:33.651 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package jq. 2020-01-16T00:10:33.660 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../089-python3-attr_17.4.0-2_all.deb ... 2020-01-16T00:10:33.667 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../078-jq_1.5+dfsg-2_amd64.deb ... 2020-01-16T00:10:33.703 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-attr (17.4.0-2) ... 2020-01-16T00:10:33.709 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking jq (1.5+dfsg-2) ... 2020-01-16T00:10:33.954 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package socat. 2020-01-16T00:10:33.968 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../079-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2020-01-16T00:10:34.012 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2020-01-16T00:10:34.014 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-cycler. 2020-01-16T00:10:34.029 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../090-python3-cycler_0.10.0-1_all.deb ... 2020-01-16T00:10:34.073 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-cycler (0.10.0-1) ... 2020-01-16T00:10:34.384 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-webencodings. 2020-01-16T00:10:34.399 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../091-python3-webencodings_0.5-2_all.deb ... 2020-01-16T00:10:34.432 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libxslt1.1:amd64. 2020-01-16T00:10:34.442 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-webencodings (0.5-2) ... 2020-01-16T00:10:34.447 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../080-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2020-01-16T00:10:34.568 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-01-16T00:10:34.745 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-html5lib. 2020-01-16T00:10:34.761 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../092-python3-html5lib_0.999999999-1_all.deb ... 2020-01-16T00:10:34.803 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2020-01-16T00:10:34.854 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package xmlstarlet. 2020-01-16T00:10:34.870 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../081-xmlstarlet_1.6.1-2_amd64.deb ... 2020-01-16T00:10:34.912 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2020-01-16T00:10:35.149 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-lxml:amd64. 2020-01-16T00:10:35.162 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../093-python3-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2020-01-16T00:10:35.206 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-01-16T00:10:35.242 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-test. 2020-01-16T00:10:35.255 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../082-ceph-test_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:10:35.299 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-test (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:35.651 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-pyparsing. 2020-01-16T00:10:35.666 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../094-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2020-01-16T00:10:35.709 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2020-01-16T00:10:36.029 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-tz. 2020-01-16T00:10:36.044 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../095-python3-tz_2018.3-2_all.deb ... 2020-01-16T00:10:36.087 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-tz (2018.3-2) ... 2020-01-16T00:10:36.382 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-matplotlib. 2020-01-16T00:10:36.397 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../096-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2020-01-16T00:10:36.432 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2020-01-16T00:10:37.087 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-olefile. 2020-01-16T00:10:37.104 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../097-python3-olefile_0.45.1-1_all.deb ... 2020-01-16T00:10:37.145 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-olefile (0.45.1-1) ... 2020-01-16T00:10:37.465 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-pastescript. 2020-01-16T00:10:37.480 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../098-python3-pastescript_2.0.2-2_all.deb ... 2020-01-16T00:10:37.523 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2020-01-16T00:10:37.885 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-pil:amd64. 2020-01-16T00:10:37.900 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../099-python3-pil_5.1.0-1_amd64.deb ... 2020-01-16T00:10:37.943 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-pil:amd64 (5.1.0-1) ... 2020-01-16T00:10:38.338 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-pluggy. 2020-01-16T00:10:38.353 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../100-python3-pluggy_0.6.0-1_all.deb ... 2020-01-16T00:10:38.396 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2020-01-16T00:10:38.645 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libcephfs-dev. 2020-01-16T00:10:38.661 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../083-libcephfs-dev_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:10:38.703 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libcephfs-dev (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:38.716 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-py. 2020-01-16T00:10:38.733 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../101-python3-py_1.5.2-1_all.deb ... 2020-01-16T00:10:38.774 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-py (1.5.2-1) ... 2020-01-16T00:10:39.031 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python-attr. 2020-01-16T00:10:39.045 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../084-python-attr_17.4.0-2_all.deb ... 2020-01-16T00:10:39.090 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python-attr (17.4.0-2) ... 2020-01-16T00:10:39.110 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-pyinotify. 2020-01-16T00:10:39.125 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../102-python3-pyinotify_0.9.6-1_all.deb ... 2020-01-16T00:10:39.194 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2020-01-16T00:10:39.376 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python-funcsigs. 2020-01-16T00:10:39.393 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../085-python-funcsigs_1.0.2-4_all.deb ... 2020-01-16T00:10:39.434 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2020-01-16T00:10:39.538 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-pytest. 2020-01-16T00:10:39.553 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../103-python3-pytest_3.3.2-2_all.deb ... 2020-01-16T00:10:39.596 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-pytest (3.3.2-2) ... 2020-01-16T00:10:39.720 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python-pluggy. 2020-01-16T00:10:39.737 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../086-python-pluggy_0.6.0-1_all.deb ... 2020-01-16T00:10:39.778 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python-pluggy (0.6.0-1) ... 2020-01-16T00:10:39.925 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-simplejson. 2020-01-16T00:10:39.941 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../104-python3-simplejson_3.13.2-1_amd64.deb ... 2020-01-16T00:10:39.983 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2020-01-16T00:10:40.090 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python-py. 2020-01-16T00:10:40.104 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../087-python-py_1.5.2-1_all.deb ... 2020-01-16T00:10:40.156 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python-py (1.5.2-1) ... 2020-01-16T00:10:40.311 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package python3-tk:amd64. 2020-01-16T00:10:40.325 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../105-python3-tk_3.6.9-1~18.04_amd64.deb ... 2020-01-16T00:10:40.369 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2020-01-16T00:10:40.551 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python-pytest. 2020-01-16T00:10:40.567 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../088-python-pytest_3.3.2-2_all.deb ... 2020-01-16T00:10:40.609 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python-pytest (3.3.2-2) ... 2020-01-16T00:10:40.823 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package radosgw. 2020-01-16T00:10:40.837 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../106-radosgw_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:10:40.881 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking radosgw (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:40.963 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-attr. 2020-01-16T00:10:40.980 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../089-python3-attr_17.4.0-2_all.deb ... 2020-01-16T00:10:41.029 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-attr (17.4.0-2) ... 2020-01-16T00:10:41.324 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-cycler. 2020-01-16T00:10:41.337 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../090-python3-cycler_0.10.0-1_all.deb ... 2020-01-16T00:10:41.382 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-cycler (0.10.0-1) ... 2020-01-16T00:10:41.598 INFO:teuthology.orchestra.run.smithi123.stdout:Selecting previously unselected package rbd-fuse. 2020-01-16T00:10:41.608 INFO:teuthology.orchestra.run.smithi123.stdout:Preparing to unpack .../107-rbd-fuse_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:10:41.651 INFO:teuthology.orchestra.run.smithi123.stdout:Unpacking rbd-fuse (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:41.668 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-webencodings. 2020-01-16T00:10:41.685 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../091-python3-webencodings_0.5-2_all.deb ... 2020-01-16T00:10:41.726 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-webencodings (0.5-2) ... 2020-01-16T00:10:41.950 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-logutils (0.3.3-5) ... 2020-01-16T00:10:42.062 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-html5lib. 2020-01-16T00:10:42.079 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../092-python3-html5lib_0.999999999-1_all.deb ... 2020-01-16T00:10:42.121 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2020-01-16T00:10:42.264 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1) ... 2020-01-16T00:10:42.466 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-lxml:amd64. 2020-01-16T00:10:42.480 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../093-python3-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2020-01-16T00:10:42.524 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-01-16T00:10:42.624 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2020-01-16T00:10:42.767 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-01-16T00:10:42.952 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-pyparsing. 2020-01-16T00:10:42.966 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../094-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2020-01-16T00:10:43.010 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2020-01-16T00:10:43.330 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-tz. 2020-01-16T00:10:43.344 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../095-python3-tz_2018.3-2_all.deb ... 2020-01-16T00:10:43.369 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-pil:amd64 (5.1.0-1) ... 2020-01-16T00:10:43.413 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-tz (2018.3-2) ... 2020-01-16T00:10:43.716 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-matplotlib. 2020-01-16T00:10:43.731 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../096-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2020-01-16T00:10:43.745 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2020-01-16T00:10:43.774 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2020-01-16T00:10:44.030 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-webencodings (0.5-2) ... 2020-01-16T00:10:44.324 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-attr (17.4.0-2) ... 2020-01-16T00:10:44.430 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-olefile. 2020-01-16T00:10:44.444 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../097-python3-olefile_0.45.1-1_all.deb ... 2020-01-16T00:10:44.488 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-olefile (0.45.1-1) ... 2020-01-16T00:10:44.515 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2020-01-16T00:10:44.641 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-py (1.5.2-1) ... 2020-01-16T00:10:44.800 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-pastescript. 2020-01-16T00:10:44.813 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../098-python3-pastescript_2.0.2-2_all.deb ... 2020-01-16T00:10:44.858 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2020-01-16T00:10:44.985 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-waitress (1.0.1-1) ... 2020-01-16T00:10:45.236 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-pil:amd64. 2020-01-16T00:10:45.251 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../099-python3-pil_5.1.0-1_amd64.deb ... 2020-01-16T00:10:45.294 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-pil:amd64 (5.1.0-1) ... 2020-01-16T00:10:45.327 INFO:teuthology.orchestra.run.smithi123.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2020-01-16T00:10:45.604 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-olefile (0.45.1-1) ... 2020-01-16T00:10:45.648 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-pluggy. 2020-01-16T00:10:45.664 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../100-python3-pluggy_0.6.0-1_all.deb ... 2020-01-16T00:10:45.706 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2020-01-16T00:10:45.992 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-py. 2020-01-16T00:10:46.008 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../101-python3-py_1.5.2-1_all.deb ... 2020-01-16T00:10:46.050 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-py (1.5.2-1) ... 2020-01-16T00:10:46.297 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-ceph-argparse (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:46.387 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-pyinotify. 2020-01-16T00:10:46.402 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../102-python3-pyinotify_0.9.6-1_all.deb ... 2020-01-16T00:10:46.470 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2020-01-16T00:10:46.598 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-tempita (0.5.2-2) ... 2020-01-16T00:10:46.782 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-pytest. 2020-01-16T00:10:46.796 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../103-python3-pytest_3.3.2-2_all.deb ... 2020-01-16T00:10:46.840 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-pytest (3.3.2-2) ... 2020-01-16T00:10:46.899 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2020-01-16T00:10:47.025 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2020-01-16T00:10:47.193 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-simplejson. 2020-01-16T00:10:47.207 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../104-python3-simplejson_3.13.2-1_amd64.deb ... 2020-01-16T00:10:47.251 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2020-01-16T00:10:47.580 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-tk:amd64. 2020-01-16T00:10:47.594 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../105-python3-tk_3.6.9-1~18.04_amd64.deb ... 2020-01-16T00:10:47.654 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2020-01-16T00:10:47.928 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2020-01-16T00:10:48.125 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package radosgw. 2020-01-16T00:10:48.141 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../106-radosgw_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:10:48.183 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking radosgw (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:48.220 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2020-01-16T00:10:48.505 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-py (1.5.2-1) ... 2020-01-16T00:10:48.732 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-simplejson (3.13.2-1) ... 2020-01-16T00:10:48.911 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package rbd-fuse. 2020-01-16T00:10:48.925 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../107-rbd-fuse_15.0.0-9353-gcb3c85e-1bionic_amd64.deb ... 2020-01-16T00:10:48.969 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking rbd-fuse (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:49.083 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2020-01-16T00:10:49.209 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-pluggy (0.6.0-1) ... 2020-01-16T00:10:49.280 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-logutils (0.3.3-5) ... 2020-01-16T00:10:49.410 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2020-01-16T00:10:49.608 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1) ... 2020-01-16T00:10:49.677 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2020-01-16T00:10:49.959 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2020-01-16T00:10:49.962 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-cycler (0.10.0-1) ... 2020-01-16T00:10:50.085 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-01-16T00:10:50.238 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2020-01-16T00:10:50.598 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-bs4 (4.6.0-1) ... 2020-01-16T00:10:50.680 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-pil:amd64 (5.1.0-1) ... 2020-01-16T00:10:50.958 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-pluggy (0.6.0-1) ... 2020-01-16T00:10:51.064 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2020-01-16T00:10:51.259 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up cephadm (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:51.374 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-webencodings (0.5-2) ... 2020-01-16T00:10:51.667 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python-attr (17.4.0-2) ... 2020-01-16T00:10:51.859 INFO:teuthology.orchestra.run.smithi123.stdout:Adding system user cephadm....done 2020-01-16T00:10:51.884 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2020-01-16T00:10:52.010 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-py (1.5.2-1) ... 2020-01-16T00:10:52.023 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2020-01-16T00:10:52.333 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-01-16T00:10:52.354 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-waitress (1.0.1-1) ... 2020-01-16T00:10:52.459 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2020-01-16T00:10:52.703 INFO:teuthology.orchestra.run.smithi145.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2020-01-16T00:10:52.810 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-prettytable (0.7.2-3) ... 2020-01-16T00:10:53.007 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-olefile (0.45.1-1) ... 2020-01-16T00:10:53.086 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2020-01-16T00:10:53.187 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2020-01-16T00:10:53.316 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-ceph-argparse (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:53.530 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-ceph-common (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:53.609 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-tempita (0.5.2-2) ... 2020-01-16T00:10:53.832 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-monotonic (1.1-2) ... 2020-01-16T00:10:53.910 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2020-01-16T00:10:54.036 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2020-01-16T00:10:54.133 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2020-01-16T00:10:54.259 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-html5lib (0.999999999-1) ... 2020-01-16T00:10:54.610 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-repoze.lru (0.7-1) ... 2020-01-16T00:10:54.895 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.1) ... 2020-01-16T00:10:54.981 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2020-01-16T00:10:55.181 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2020-01-16T00:10:55.299 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2020-01-16T00:10:55.505 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-attr (17.4.0-2) ... 2020-01-16T00:10:55.592 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python-py (1.5.2-1) ... 2020-01-16T00:10:55.815 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-funcsigs (1.0.2-4) ... 2020-01-16T00:10:55.818 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-simplejson (3.13.2-1) ... 2020-01-16T00:10:55.999 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-01-16T00:10:56.136 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2020-01-16T00:10:56.279 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python-pluggy (0.6.0-1) ... 2020-01-16T00:10:56.480 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2020-01-16T00:10:56.534 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2020-01-16T00:10:56.624 INFO:teuthology.orchestra.run.smithi123.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2020-01-16T00:10:56.765 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2020-01-16T00:10:56.861 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up liboath0 (2.6.1-1) ... 2020-01-16T00:10:57.012 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-01-16T00:10:57.091 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-cycler (0.10.0-1) ... 2020-01-16T00:10:57.121 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2020-01-16T00:10:57.356 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2020-01-16T00:10:57.383 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2020-01-16T00:10:57.482 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-setuptools (39.0.1-2) ... 2020-01-16T00:10:57.743 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-bs4 (4.6.0-1) ... 2020-01-16T00:10:57.875 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2020-01-16T00:10:58.001 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-decorator (4.1.2-1) ... 2020-01-16T00:10:58.069 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-pluggy (0.6.0-1) ... 2020-01-16T00:10:58.293 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-tz (2018.3-2) ... 2020-01-16T00:10:58.381 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up cephadm (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:10:58.561 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2020-01-16T00:10:58.837 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2020-01-16T00:10:59.099 INFO:teuthology.orchestra.run.smithi145.stdout:Adding system user cephadm....done 2020-01-16T00:10:59.155 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-joblib (0.11-1) ... 2020-01-16T00:10:59.279 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2020-01-16T00:10:59.532 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-routes (2.4.1-1) ... 2020-01-16T00:10:59.589 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-01-16T00:10:59.715 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2020-01-16T00:10:59.841 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2020-01-16T00:11:00.058 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-prettytable (0.7.2-3) ... 2020-01-16T00:11:00.142 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-dateutil (2.6.1-1) ... 2020-01-16T00:11:00.376 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2020-01-16T00:11:00.460 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.3) ... 2020-01-16T00:11:00.502 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2020-01-16T00:11:00.812 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-fasteners (0.12.0-3) ... 2020-01-16T00:11:00.853 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-ceph-common (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:01.096 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up jq (1.5+dfsg-2) ... 2020-01-16T00:11:01.180 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-monotonic (1.1-2) ... 2020-01-16T00:11:01.214 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2020-01-16T00:11:01.465 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2020-01-16T00:11:01.591 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-html5lib (0.999999999-1) ... 2020-01-16T00:11:01.942 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-repoze.lru (0.7-1) ... 2020-01-16T00:11:02.243 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.1) ... 2020-01-16T00:11:02.369 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-01-16T00:11:02.553 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2020-01-16T00:11:02.654 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2020-01-16T00:11:02.904 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-attr (17.4.0-2) ... 2020-01-16T00:11:02.972 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2020-01-16T00:11:03.098 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-pytest (3.3.2-2) ... 2020-01-16T00:11:03.213 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python-funcsigs (1.0.2-4) ... 2020-01-16T00:11:03.451 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-01-16T00:11:03.457 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python-pytest (3.3.2-2) ... 2020-01-16T00:11:03.709 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2020-01-16T00:11:03.835 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up xmlstarlet (1.6.1-2) ... 2020-01-16T00:11:03.874 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2020-01-16T00:11:03.961 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up librados2 (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:03.964 INFO:teuthology.orchestra.run.smithi145.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2020-01-16T00:11:04.062 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-rsa (3.4.2-1) ... 2020-01-16T00:11:04.218 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up liboath0 (2.6.1-1) ... 2020-01-16T00:11:04.346 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libcephfs2 (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:04.394 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-01-16T00:11:04.464 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libcephfs-dev (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:04.529 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2020-01-16T00:11:04.615 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-rados (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:04.741 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2020-01-16T00:11:04.796 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2020-01-16T00:11:04.923 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-setuptools (39.0.1-2) ... 2020-01-16T00:11:05.042 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2020-01-16T00:11:05.333 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2020-01-16T00:11:05.343 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2020-01-16T00:11:05.459 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-decorator (4.1.2-1) ... 2020-01-16T00:11:05.669 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2020-01-16T00:11:05.743 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-tz (2018.3-2) ... 2020-01-16T00:11:06.052 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2020-01-16T00:11:06.354 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2020-01-16T00:11:06.489 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-sklearn (0.19.1-3) ... 2020-01-16T00:11:06.663 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-joblib (0.11-1) ... 2020-01-16T00:11:07.057 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-routes (2.4.1-1) ... 2020-01-16T00:11:07.343 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2020-01-16T00:11:07.374 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2020-01-16T00:11:07.653 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up libradosstriper1 (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:07.684 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-dateutil (2.6.1-1) ... 2020-01-16T00:11:07.774 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-cephfs (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:08.035 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.3) ... 2020-01-16T00:11:08.155 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up librgw2 (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:08.291 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-fuse (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:08.412 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-fasteners (0.12.0-3) ... 2020-01-16T00:11:08.512 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-01-16T00:11:08.512 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-01-16T00:11:08.730 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up jq (1.5+dfsg-2) ... 2020-01-16T00:11:08.771 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up librbd1 (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:08.856 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2020-01-16T00:11:08.872 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-pecan (1.2.1-2) ... 2020-01-16T00:11:08.961 INFO:teuthology.orchestra.run.smithi123.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2020-01-16T00:11:08.994 INFO:teuthology.orchestra.run.smithi123.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2020-01-16T00:11:09.294 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2020-01-16T00:11:09.960 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-pastescript (2.0.2-2) ... 2020-01-16T00:11:10.028 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-01-16T00:11:10.286 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-rgw (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:10.354 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2020-01-16T00:11:10.387 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up rbd-fuse (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:10.515 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up python3-rbd (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:10.641 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-common (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:10.673 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2020-01-16T00:11:10.799 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-pytest (3.3.2-2) ... 2020-01-16T00:11:11.150 INFO:teuthology.orchestra.run.smithi123.stdout:Adding group ceph....done 2020-01-16T00:11:11.167 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python-pytest (3.3.2-2) ... 2020-01-16T00:11:11.426 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2020-01-16T00:11:11.511 INFO:teuthology.orchestra.run.smithi123.stdout:Adding system user ceph....done 2020-01-16T00:11:11.552 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up xmlstarlet (1.6.1-2) ... 2020-01-16T00:11:11.678 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up librados2 (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:11.715 INFO:teuthology.orchestra.run.smithi123.stdout:Setting system user ceph properties....done 2020-01-16T00:11:11.821 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-rsa (3.4.2-1) ... 2020-01-16T00:11:11.867 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2020-01-16T00:11:12.089 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2020-01-16T00:11:12.147 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libcephfs2 (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:12.273 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libcephfs-dev (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:12.376 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up radosgw (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:12.399 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-rados (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:12.526 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2020-01-16T00:11:12.631 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-01-16T00:11:12.631 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-01-16T00:11:12.835 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2020-01-16T00:11:13.011 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-test (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:13.128 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-base (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:13.443 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2020-01-16T00:11:13.444 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2020-01-16T00:11:13.745 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-mgr (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:13.762 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2020-01-16T00:11:13.947 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-01-16T00:11:13.947 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-01-16T00:11:14.221 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-mgr-diskprediction-local (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:14.339 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-osd (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:14.607 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-sklearn (0.19.1-3) ... 2020-01-16T00:11:14.794 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-01-16T00:11:14.794 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-01-16T00:11:15.066 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-mgr-cephadm (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:15.201 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-mgr-diskprediction-cloud (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:15.335 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-mgr-rook (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:15.470 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-mds (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:15.470 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2020-01-16T00:11:15.679 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-01-16T00:11:15.679 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-01-16T00:11:15.781 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libradosstriper1 (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:15.910 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-cephfs (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:15.954 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-mon (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:16.155 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-01-16T00:11:16.155 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-01-16T00:11:16.315 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up librgw2 (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:16.422 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-mgr-k8sevents (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:16.451 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-fuse (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:16.683 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-01-16T00:11:16.683 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-01-16T00:11:16.887 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph-mgr-dashboard (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:16.958 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up librbd1 (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:17.085 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-pecan (1.2.1-2) ... 2020-01-16T00:11:17.174 INFO:teuthology.orchestra.run.smithi145.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2020-01-16T00:11:17.214 INFO:teuthology.orchestra.run.smithi145.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2020-01-16T00:11:17.224 INFO:teuthology.orchestra.run.smithi123.stdout:Setting up ceph (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:17.351 INFO:teuthology.orchestra.run.smithi123.stdout:Processing triggers for systemd (237-3ubuntu10.33) ... 2020-01-16T00:11:17.531 INFO:teuthology.orchestra.run.smithi123.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2020-01-16T00:11:17.558 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2020-01-16T00:11:18.236 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-pastescript (2.0.2-2) ... 2020-01-16T00:11:18.548 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-rgw (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:18.680 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up rbd-fuse (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:18.831 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-rbd (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:18.947 INFO:teuthology.orchestra.run.smithi123.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2020-01-16T00:11:18.957 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-common (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:19.000 INFO:teuthology.orchestra.run.smithi123.stdout:ureadahead will be reprofiled on next reboot 2020-01-16T00:11:19.059 INFO:teuthology.orchestra.run.smithi123.stdout:Processing triggers for libc-bin (2.27-3ubuntu1) ... 2020-01-16T00:11:19.493 INFO:teuthology.orchestra.run.smithi145.stdout:Adding group ceph....done 2020-01-16T00:11:19.498 INFO:teuthology.orchestra.run.smithi123.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2020-01-16T00:11:19.500 DEBUG:teuthology.parallel:result is None 2020-01-16T00:11:19.920 INFO:teuthology.orchestra.run.smithi145.stdout:Adding system user ceph....done 2020-01-16T00:11:20.114 INFO:teuthology.orchestra.run.smithi145.stdout:Setting system user ceph properties....done 2020-01-16T00:11:20.275 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2020-01-16T00:11:20.496 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2020-01-16T00:11:20.795 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up radosgw (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:21.057 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-01-16T00:11:21.058 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-01-16T00:11:21.488 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-test (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:21.623 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-base (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:21.963 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2020-01-16T00:11:22.258 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-mgr (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:22.480 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-01-16T00:11:22.480 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-01-16T00:11:22.776 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-mgr-diskprediction-local (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:22.911 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-osd (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:23.372 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-01-16T00:11:23.372 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-01-16T00:11:23.664 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-mgr-cephadm (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:23.812 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-mgr-diskprediction-cloud (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:23.973 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-mgr-rook (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:24.123 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-mds (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:24.339 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-01-16T00:11:24.339 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-01-16T00:11:24.630 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-mon (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:24.849 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-01-16T00:11:24.849 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-01-16T00:11:25.127 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-mgr-k8sevents (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:25.253 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-mgr-dashboard (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:25.396 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph (15.0.0-9353-gcb3c85e-1bionic) ... 2020-01-16T00:11:25.523 INFO:teuthology.orchestra.run.smithi145.stdout:Processing triggers for systemd (237-3ubuntu10.33) ... 2020-01-16T00:11:25.738 INFO:teuthology.orchestra.run.smithi145.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2020-01-16T00:11:27.167 INFO:teuthology.orchestra.run.smithi145.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2020-01-16T00:11:27.213 INFO:teuthology.orchestra.run.smithi145.stdout:ureadahead will be reprofiled on next reboot 2020-01-16T00:11:27.268 INFO:teuthology.orchestra.run.smithi145.stdout:Processing triggers for libc-bin (2.27-3ubuntu1) ... 2020-01-16T00:11:27.700 INFO:teuthology.orchestra.run.smithi145.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2020-01-16T00:11:27.702 DEBUG:teuthology.parallel:result is None 2020-01-16T00:11:27.702 INFO:teuthology.orchestra.run.smithi123:> sudo lsb_release -is 2020-01-16T00:11:27.758 INFO:teuthology.orchestra.run.smithi123.stdout:Ubuntu 2020-01-16T00:11:27.764 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-01-16T00:11:27.764 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=cb3c85e74500283e01cc592f7e2858e323186910 2020-01-16T00:11:27.959 INFO:teuthology.orchestra.run.smithi123:> dpkg-query -W -f '${Version}' ceph 2020-01-16T00:11:27.984 INFO:teuthology.orchestra.run.smithi123.stdout:15.0.0-9353-gcb3c85e-1bionic 2020-01-16T00:11:27.985 INFO:teuthology.packaging:The installed version of ceph is 15.0.0-9353-gcb3c85e-1bionic 2020-01-16T00:11:27.985 INFO:teuthology.task.install:The correct ceph version 15.0.0-9353-gcb3c85e-1bionic is installed. 2020-01-16T00:11:27.985 INFO:teuthology.orchestra.run.smithi145:> sudo lsb_release -is 2020-01-16T00:11:28.041 INFO:teuthology.orchestra.run.smithi145.stdout:Ubuntu 2020-01-16T00:11:28.047 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-01-16T00:11:28.047 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=cb3c85e74500283e01cc592f7e2858e323186910 2020-01-16T00:11:28.210 INFO:teuthology.orchestra.run.smithi145:> dpkg-query -W -f '${Version}' ceph 2020-01-16T00:11:28.235 INFO:teuthology.orchestra.run.smithi145.stdout:15.0.0-9353-gcb3c85e-1bionic 2020-01-16T00:11:28.235 INFO:teuthology.packaging:The installed version of ceph is 15.0.0-9353-gcb3c85e-1bionic 2020-01-16T00:11:28.235 INFO:teuthology.task.install:The correct ceph version 15.0.0-9353-gcb3c85e-1bionic is installed. 2020-01-16T00:11:28.235 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2020-01-16T00:11:28.236 INFO:teuthology.orchestra.run.smithi123:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2020-01-16T00:11:28.252 INFO:teuthology.orchestra.run.smithi145:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2020-01-16T00:11:28.295 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2020-01-16T00:11:28.295 INFO:teuthology.orchestra.run.smithi123:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2020-01-16T00:11:28.311 INFO:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-01-16T00:11:28.368 INFO:teuthology.orchestra.run.smithi145:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2020-01-16T00:11:28.383 INFO:teuthology.orchestra.run.smithi145:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-01-16T00:11:28.438 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2020-01-16T00:11:28.439 INFO:teuthology.orchestra.run.smithi123:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2020-01-16T00:11:28.488 INFO:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-01-16T00:11:28.543 INFO:teuthology.orchestra.run.smithi145:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2020-01-16T00:11:28.591 INFO:teuthology.orchestra.run.smithi145:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-01-16T00:11:28.646 INFO:teuthology.run_tasks:Running task ceph... 2020-01-16T00:11:28.741 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2020-01-16T00:11:28.741 INFO:teuthology.orchestra.run.smithi123:> sudo chmod 777 /var/log/ceph 2020-01-16T00:11:28.743 INFO:teuthology.orchestra.run.smithi145:> sudo chmod 777 /var/log/ceph 2020-01-16T00:11:28.758 INFO:tasks.ceph:Disabling ceph logrotate... 2020-01-16T00:11:28.758 INFO:teuthology.orchestra.run.smithi123:> sudo rm -f -- /etc/logrotate.d/ceph 2020-01-16T00:11:28.799 INFO:teuthology.orchestra.run.smithi145:> sudo rm -f -- /etc/logrotate.d/ceph 2020-01-16T00:11:28.814 INFO:tasks.ceph:Creating extra log directories... 2020-01-16T00:11:28.814 INFO:teuthology.orchestra.run.smithi123:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-01-16T00:11:28.855 INFO:teuthology.orchestra.run.smithi145:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-01-16T00:11:28.871 INFO:tasks.ceph:Creating ceph cluster ceph... 2020-01-16T00:11:28.872 INFO:tasks.ceph:config {'mon_bind_addrvec': True, 'block_journal': None, 'fs': 'xfs', 'log_whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow requests'], 'skip_mgr_daemons': False, 'cpu_profile': set([]), 'mkfs_options': None, 'tmpfs_journal': None, 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mount_options': None, 'conf': {'global': {'osd_object_clean_region_max_num_intervals': 1000, 'ms type': 'async', 'ms tcp read timeout': 5, 'osd_pool_default_size': 3, 'osd_pool_default_min_size': 2, 'osd_max_pg_log_entries': 2, 'ms bind msgr2': False, 'mon client directed command retry': 5, 'osd_async_recovery_min_cost': 1, 'ms inject socket failures': 2500, 'osd_min_pg_log_entries': 1}, 'mon': {'debug mon': 20, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'debug ms': 1, 'mon osdmap full prune min': 15, 'mon osdmap full prune interval': 2, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'mon osd full ratio': 0.9, 'osd op queue cut off': 'debug_random', 'mon osd backfillfull_ratio': 0.85, 'debug journal': 20, 'osd scrub max interval': 120, 'debug osd': 25, 'debug bluestore': 20, 'debug bluefs': 20, 'osd debug verify missing on start': True, 'debug ms': 1, 'debug rocksdb': 10, 'osd shutdown pgref assert': True, 'bdev async discard': True, 'bluestore fsck on mount': True, 'osd max backfills': 6, 'mon osd nearfull ratio': 0.8, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd debug verify cached snaps': True, 'osd map cache size': 1, 'osd op queue': 'debug_random', 'osd scrub during recovery': False, 'bluestore block size': 96636764160, 'osd scrub min interval': 60, 'debug filestore': 20, 'bdev enable discard': True, 'osd failsafe full ratio': 0.95}}} 2020-01-16T00:11:28.872 INFO:tasks.ceph:ctx.config {'kernel': {'sha1': 'distro', 'kdb': True}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'sha1': 'distro', 'kdb': True}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'exec': {'mon.a': ['while ! ceph balancer status ; do sleep 1 ; done', 'ceph balancer mode crush-compat', 'ceph balancer on']}}, {'thrashosds': {'chance_test_map_discontinuity': 2, 'chance_pgnum_grow': 0.25, 'chance_pgpnum_fix': 0.25, 'timeout': 1800, 'chance_pgnum_shrink': 0.25}}, {'exec': {'client.0': ['sudo ceph osd pool create low_tier 4']}}, {'rados': {'op_weights': {'read': 100, 'tier_flush': 10, 'write': 100, 'tier_promote': 10}, 'objects': 50, 'set_chunk': True, 'ops': 1500, 'clients': ['client.0'], 'low_tier_pool': 'low_tier'}}], 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage-testing-2020-01-15-1209/qa', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.2508', 'owner': 'scheduled_sage@teuthology', 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 30}}], 'targets': {'smithi123.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDAtTFcxpKk/Gf1EokgFNST3fRhvfOR2OgUUPB039HuVj1o29IATfDCJB+NEPfrjnv6ngyyYlv+e53hHnJBusfP/J7Fds5US04wbUVrRluJLK2I7tGotRl7rUeXh4LwZQsQ462Rk3Z3uu0F+QOdu6mxy7SRHGthDMCktNG/JJNnVhP2uyEZGBKwLuQcWw5EaVtwkgU75/uxsytPvSJ23T5Imo10Ez1wrOicyDh8D6NWs0dhFra7wG2AnjSogEMGCWOBo/mrNin14hdV+fi+kakbXlWywcBixbzo1DtvvXzE4Nd+y3AdUOl6oEFkF0d8aSRk+Yd+4cndtwIBLBvNm1Ad', 'smithi145.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQClIIpQOIOKY4e1T8vM7W7KgPjkdMmoU9OgSSroJKKygQTZD9LPRr9vmMvD8exQep8CkGi+r+w5HCyZxbAnimaI2l0R/HdmKDqpUTboHS7Rp7AjyHLNXvVXHNPrMX66lebTjNFZ/6/24+AKOOVXQm4M7OdKN6GdYpEt8sHgEJrfRfiRLKDqY6M8F50C73z5XFmW6RhSXAkuuNFxeZiZxziJo5PJe181LSO6vBLKkNJ0LPGcYlczMFesXwOWVd0nPLAz5UuYv8kIH2IDuOZlBiFEVwIlWbIluN9UH1DfbFMimzDViBgv7tJBxbUbgqc48CEjVbBJlv/lQT/On6YK3zDr'}, 'job_id': '4671691', 'tube': 'smithi', 'suite_branch': 'wip-sage-testing-2020-01-15-1209', 'priority': 90, 'os_version': '18.04', 'first_in_suite': False, 'branch': 'wip-sage-testing-2020-01-15-1209', 'suite': 'rados', 'email': None, 'archive_path': '/home/teuthworker/archive/sage-2020-01-15_21:23:44-rados-wip-sage-testing-2020-01-15-1209-distro-basic-smithi/4671691', 'description': 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{more-async-partial-recovery.yaml} backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/fastclose.yaml msgr/async-v1only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/mapgap.yaml thrashosds-health.yaml workloads/set-chunk-promote-flush.yaml}', 'nuke-on-error': True, 'repo': 'git://git.ceph.com/ceph-ci.git', 'last_in_suite': False, 'machine_type': 'smithi', 'teuthology_branch': 'master', 'sha1': 'cb3c85e74500283e01cc592f7e2858e323186910', 'name': 'sage-2020-01-15_21:23:44-rados-wip-sage-testing-2020-01-15-1209-distro-basic-smithi', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']], 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': 'cb3c85e74500283e01cc592f7e2858e323186910', 'os_type': 'ubuntu', 'overrides': {'ceph': {'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow requests'], 'mon_bind_msgr2': False, 'conf': {'global': {'osd_object_clean_region_max_num_intervals': 1000, 'ms type': 'async', 'ms tcp read timeout': 5, 'osd_pool_default_size': 3, 'osd_pool_default_min_size': 2, 'osd_max_pg_log_entries': 2, 'ms bind msgr2': False, 'mon client directed command retry': 5, 'osd_async_recovery_min_cost': 1, 'ms inject socket failures': 2500, 'osd_min_pg_log_entries': 1}, 'mon': {'debug mon': 20, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'debug ms': 1, 'mon osdmap full prune min': 15, 'mon osdmap full prune interval': 2, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'mon osd full ratio': 0.9, 'osd op queue cut off': 'debug_random', 'mon osd backfillfull_ratio': 0.85, 'debug journal': 20, 'osd scrub max interval': 120, 'debug osd': 25, 'debug bluestore': 20, 'debug bluefs': 20, 'osd debug verify missing on start': True, 'debug ms': 1, 'debug rocksdb': 10, 'osd shutdown pgref assert': True, 'bdev async discard': True, 'bluestore fsck on mount': True, 'osd max backfills': 6, 'mon osd nearfull ratio': 0.8, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd debug verify cached snaps': True, 'osd map cache size': 1, 'osd op queue': 'debug_random', 'osd scrub during recovery': False, 'bluestore block size': 96636764160, 'osd scrub min interval': 60, 'debug filestore': 20, 'bdev enable discard': True, 'osd failsafe full ratio': 0.95}}, 'fs': 'xfs', 'sha1': 'cb3c85e74500283e01cc592f7e2858e323186910'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': 'cb3c85e74500283e01cc592f7e2858e323186910'}}, 'admin_socket': {'branch': 'wip-sage-testing-2020-01-15-1209'}, 'thrashosds': {'bdev_inject_crash_probability': 0.5, 'min_in': 4, 'bdev_inject_crash': 2}, 'workunit': {'sha1': 'cb3c85e74500283e01cc592f7e2858e323186910', 'branch': 'wip-sage-testing-2020-01-15-1209'}}} 2020-01-16T00:11:28.872 INFO:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2020-01-16T00:11:28.911 INFO:teuthology.orchestra.run.smithi145:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2020-01-16T00:11:28.918 INFO:teuthology.orchestra.run.smithi123:> sudo install -d -m0777 -- /var/run/ceph 2020-01-16T00:11:28.959 INFO:teuthology.orchestra.run.smithi145:> sudo install -d -m0777 -- /var/run/ceph 2020-01-16T00:11:29.021 DEBUG:teuthology.orchestra.remote:smithi123:/scratch_devs is 72B 2020-01-16T00:11:29.032 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2020-01-16T00:11:29.032 INFO:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-01-16T00:11:29.039 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-1 2020-01-16T00:11:29.039 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-01-16T00:11:29.040 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 6h/6d Inode: 433 Links: 1 2020-01-16T00:11:29.040 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-01-16T00:11:29.040 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2020-01-16 00:09:09.621076741 +0000 2020-01-16T00:11:29.040 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2020-01-16 00:09:09.509132744 +0000 2020-01-16T00:11:29.040 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2020-01-16 00:09:09.509132744 +0000 2020-01-16T00:11:29.040 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: - 2020-01-16T00:11:29.047 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2020-01-16T00:11:29.047 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2020-01-16T00:11:29.047 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000379865 s, 1.3 MB/s 2020-01-16T00:11:29.051 INFO:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-01-16T00:11:29.100 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-01-16T00:11:29.100 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-01-16T00:11:29.100 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 6h/6d Inode: 439 Links: 1 2020-01-16T00:11:29.100 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-01-16T00:11:29.101 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2020-01-16 00:09:09.848962736 +0000 2020-01-16T00:11:29.101 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2020-01-16 00:09:09.749012738 +0000 2020-01-16T00:11:29.101 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2020-01-16 00:09:09.749012738 +0000 2020-01-16T00:11:29.101 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: - 2020-01-16T00:11:29.109 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2020-01-16T00:11:29.109 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2020-01-16T00:11:29.109 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000321817 s, 1.6 MB/s 2020-01-16T00:11:29.113 INFO:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-01-16T00:11:29.161 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-3 2020-01-16T00:11:29.161 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-01-16T00:11:29.161 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 6h/6d Inode: 445 Links: 1 2020-01-16T00:11:29.161 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-01-16T00:11:29.161 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2020-01-16 00:09:10.080846731 +0000 2020-01-16T00:11:29.162 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2020-01-16 00:09:09.980896732 +0000 2020-01-16T00:11:29.163 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2020-01-16 00:09:09.980896732 +0000 2020-01-16T00:11:29.163 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: - 2020-01-16T00:11:29.169 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2020-01-16T00:11:29.169 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2020-01-16T00:11:29.169 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000377325 s, 1.4 MB/s 2020-01-16T00:11:29.173 INFO:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-01-16T00:11:29.220 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-4 2020-01-16T00:11:29.220 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-01-16T00:11:29.220 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 6h/6d Inode: 451 Links: 1 2020-01-16T00:11:29.220 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-01-16T00:11:29.220 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2020-01-16 00:11:08.571565857 +0000 2020-01-16T00:11:29.221 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2020-01-16 00:09:10.188792729 +0000 2020-01-16T00:11:29.221 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2020-01-16 00:09:10.188792729 +0000 2020-01-16T00:11:29.221 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: - 2020-01-16T00:11:29.229 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2020-01-16T00:11:29.229 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2020-01-16T00:11:29.229 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000395717 s, 1.3 MB/s 2020-01-16T00:11:29.233 INFO:tasks.ceph:fs option selected, checking for scratch devs 2020-01-16T00:11:29.233 INFO:tasks.ceph:found devs: ['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2020-01-16T00:11:29.234 WARNING:teuthology.misc:Entering get_wwn_id_map, a deprecated function that will be removed 2020-01-16T00:11:29.234 INFO:tasks.ceph:dev map: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'} 2020-01-16T00:11:29.242 DEBUG:teuthology.orchestra.remote:smithi145:/scratch_devs is 72B 2020-01-16T00:11:29.252 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2020-01-16T00:11:29.252 INFO:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-01-16T00:11:29.259 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-1 2020-01-16T00:11:29.259 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-01-16T00:11:29.260 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 6h/6d Inode: 443 Links: 1 2020-01-16T00:11:29.260 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-01-16T00:11:29.260 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2020-01-16 00:09:10.197055090 +0000 2020-01-16T00:11:29.260 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2020-01-16 00:09:10.089109092 +0000 2020-01-16T00:11:29.260 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2020-01-16 00:09:10.089109092 +0000 2020-01-16T00:11:29.260 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: - 2020-01-16T00:11:29.269 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2020-01-16T00:11:29.269 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2020-01-16T00:11:29.269 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.000451393 s, 1.1 MB/s 2020-01-16T00:11:29.274 INFO:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-01-16T00:11:29.327 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-01-16T00:11:29.327 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-01-16T00:11:29.327 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 6h/6d Inode: 449 Links: 1 2020-01-16T00:11:29.327 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-01-16T00:11:29.327 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2020-01-16 00:09:10.312997087 +0000 2020-01-16T00:11:29.327 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2020-01-16 00:09:10.308999087 +0000 2020-01-16T00:11:29.328 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2020-01-16 00:09:10.308999087 +0000 2020-01-16T00:11:29.328 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: - 2020-01-16T00:11:29.336 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2020-01-16T00:11:29.336 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2020-01-16T00:11:29.336 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.000364004 s, 1.4 MB/s 2020-01-16T00:11:29.340 INFO:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-01-16T00:11:29.392 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-3 2020-01-16T00:11:29.392 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-01-16T00:11:29.392 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 6h/6d Inode: 455 Links: 1 2020-01-16T00:11:29.392 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-01-16T00:11:29.392 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2020-01-16 00:09:10.652827080 +0000 2020-01-16T00:11:29.392 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2020-01-16 00:09:10.556875082 +0000 2020-01-16T00:11:29.392 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2020-01-16 00:09:10.556875082 +0000 2020-01-16T00:11:29.393 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: - 2020-01-16T00:11:29.401 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2020-01-16T00:11:29.401 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2020-01-16T00:11:29.401 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.000427327 s, 1.2 MB/s 2020-01-16T00:11:29.405 INFO:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-01-16T00:11:29.455 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-4 2020-01-16T00:11:29.455 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-01-16T00:11:29.455 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 6h/6d Inode: 461 Links: 1 2020-01-16T00:11:29.455 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-01-16T00:11:29.455 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2020-01-16 00:11:16.753078833 +0000 2020-01-16T00:11:29.455 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2020-01-16 00:09:10.760773078 +0000 2020-01-16T00:11:29.456 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2020-01-16 00:09:10.760773078 +0000 2020-01-16T00:11:29.456 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: - 2020-01-16T00:11:29.464 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2020-01-16T00:11:29.465 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2020-01-16T00:11:29.465 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.000400169 s, 1.3 MB/s 2020-01-16T00:11:29.469 INFO:tasks.ceph:fs option selected, checking for scratch devs 2020-01-16T00:11:29.469 INFO:tasks.ceph:found devs: ['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2020-01-16T00:11:29.469 WARNING:teuthology.misc:Entering get_wwn_id_map, a deprecated function that will be removed 2020-01-16T00:11:29.469 INFO:tasks.ceph:dev map: {'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'} 2020-01-16T00:11:29.469 INFO:tasks.ceph:Generating config... 2020-01-16T00:11:29.471 INFO:tasks.ceph:[global] osd_object_clean_region_max_num_intervals = 1000 2020-01-16T00:11:29.471 INFO:tasks.ceph:[global] ms type = async 2020-01-16T00:11:29.472 INFO:tasks.ceph:[global] ms tcp read timeout = 5 2020-01-16T00:11:29.472 INFO:tasks.ceph:[global] osd_pool_default_size = 3 2020-01-16T00:11:29.472 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2020-01-16T00:11:29.472 INFO:tasks.ceph:[global] osd_max_pg_log_entries = 2 2020-01-16T00:11:29.472 INFO:tasks.ceph:[global] ms bind msgr2 = False 2020-01-16T00:11:29.472 INFO:tasks.ceph:[global] mon client directed command retry = 5 2020-01-16T00:11:29.472 INFO:tasks.ceph:[global] osd_async_recovery_min_cost = 1 2020-01-16T00:11:29.473 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2020-01-16T00:11:29.473 INFO:tasks.ceph:[global] osd_min_pg_log_entries = 1 2020-01-16T00:11:29.473 INFO:tasks.ceph:[mon] debug mon = 20 2020-01-16T00:11:29.473 INFO:tasks.ceph:[mon] debug paxos = 20 2020-01-16T00:11:29.473 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2020-01-16T00:11:29.473 INFO:tasks.ceph:[mon] debug ms = 1 2020-01-16T00:11:29.473 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2020-01-16T00:11:29.474 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2020-01-16T00:11:29.474 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2020-01-16T00:11:29.474 INFO:tasks.ceph:[mon] paxos service trim min = 10 2020-01-16T00:11:29.474 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2020-01-16T00:11:29.474 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2020-01-16T00:11:29.474 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2020-01-16T00:11:29.474 INFO:tasks.ceph:[osd] debug journal = 20 2020-01-16T00:11:29.475 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2020-01-16T00:11:29.475 INFO:tasks.ceph:[osd] debug osd = 25 2020-01-16T00:11:29.475 INFO:tasks.ceph:[osd] debug bluestore = 20 2020-01-16T00:11:29.475 INFO:tasks.ceph:[osd] debug bluefs = 20 2020-01-16T00:11:29.475 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2020-01-16T00:11:29.475 INFO:tasks.ceph:[osd] debug ms = 1 2020-01-16T00:11:29.475 INFO:tasks.ceph:[osd] debug rocksdb = 10 2020-01-16T00:11:29.475 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2020-01-16T00:11:29.476 INFO:tasks.ceph:[osd] bdev async discard = True 2020-01-16T00:11:29.476 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2020-01-16T00:11:29.476 INFO:tasks.ceph:[osd] osd max backfills = 6 2020-01-16T00:11:29.476 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2020-01-16T00:11:29.476 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2020-01-16T00:11:29.476 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2020-01-16T00:11:29.476 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2020-01-16T00:11:29.476 INFO:tasks.ceph:[osd] osd map cache size = 1 2020-01-16T00:11:29.477 INFO:tasks.ceph:[osd] osd op queue = debug_random 2020-01-16T00:11:29.477 INFO:tasks.ceph:[osd] osd scrub during recovery = False 2020-01-16T00:11:29.477 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2020-01-16T00:11:29.477 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2020-01-16T00:11:29.477 INFO:tasks.ceph:[osd] debug filestore = 20 2020-01-16T00:11:29.477 INFO:tasks.ceph:[osd] bdev enable discard = True 2020-01-16T00:11:29.477 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2020-01-16T00:11:29.477 INFO:tasks.ceph:Setting up mon.a... 2020-01-16T00:11:29.478 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2020-01-16T00:11:29.509 INFO:teuthology.orchestra.run.smithi123.stdout:creating /etc/ceph/ceph.keyring 2020-01-16T00:11:29.511 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2020-01-16T00:11:29.594 INFO:teuthology.orchestra.run.smithi123:> sudo chmod 0644 /etc/ceph/ceph.keyring 2020-01-16T00:11:29.652 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.123:6789]'), ('mon.c', '[v1:172.21.15.123:6790]'), ('mon.b', '[v1:172.21.15.145:6789]')] 2020-01-16T00:11:29.652 INFO:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --addv a '[v1:172.21.15.123:6789]' --addv c '[v1:172.21.15.123:6790]' --addv b '[v1:172.21.15.145:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2020-01-16T00:11:29.726 INFO:teuthology.orchestra.run.smithi123.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2020-01-16T00:11:29.726 INFO:teuthology.orchestra.run.smithi123.stdout:monmaptool: generated fsid 864e7191-e3db-4201-b88a-a4873d1eb7fe 2020-01-16T00:11:29.726 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 0 2020-01-16T00:11:29.726 INFO:teuthology.orchestra.run.smithi123.stdout:fsid 864e7191-e3db-4201-b88a-a4873d1eb7fe 2020-01-16T00:11:29.726 INFO:teuthology.orchestra.run.smithi123.stdout:last_changed 2020-01-16T00:11:29.727601+0000 2020-01-16T00:11:29.726 INFO:teuthology.orchestra.run.smithi123.stdout:created 2020-01-16T00:11:29.727601+0000 2020-01-16T00:11:29.727 INFO:teuthology.orchestra.run.smithi123.stdout:min_mon_release 0 (unknown) 2020-01-16T00:11:29.727 INFO:teuthology.orchestra.run.smithi123.stdout:0: v1:172.21.15.123:6789/0 mon.a 2020-01-16T00:11:29.727 INFO:teuthology.orchestra.run.smithi123.stdout:1: v1:172.21.15.145:6789/0 mon.b 2020-01-16T00:11:29.727 INFO:teuthology.orchestra.run.smithi123.stdout:2: v1:172.21.15.123:6790/0 mon.c 2020-01-16T00:11:29.727 INFO:teuthology.orchestra.run.smithi123.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2020-01-16T00:11:29.728 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 864e7191-e3db-4201-b88a-a4873d1eb7fe... 2020-01-16T00:11:29.729 INFO:teuthology.orchestra.run.smithi123:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-01-16T00:11:29.771 INFO:teuthology.orchestra.run.smithi145:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-01-16T00:11:29.802 INFO:teuthology.orchestra.run.smithi123.stdout:[global] 2020-01-16T00:11:29.803 INFO:teuthology.orchestra.run.smithi123.stdout: chdir = "" 2020-01-16T00:11:29.803 INFO:teuthology.orchestra.run.smithi123.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-01-16T00:11:29.803 INFO:teuthology.orchestra.run.smithi123.stdout: auth supported = cephx 2020-01-16T00:11:29.803 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:11:29.803 INFO:teuthology.orchestra.run.smithi123.stdout: filestore xattr use omap = true 2020-01-16T00:11:29.803 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:11:29.803 INFO:teuthology.orchestra.run.smithi123.stdout: mon clock drift allowed = 1.000 2020-01-16T00:11:29.804 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:11:29.804 INFO:teuthology.orchestra.run.smithi123.stdout: osd crush chooseleaf type = 0 2020-01-16T00:11:29.804 INFO:teuthology.orchestra.run.smithi123.stdout: auth debug = true 2020-01-16T00:11:29.804 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:11:29.804 INFO:teuthology.orchestra.run.smithi123.stdout: ms die on old message = true 2020-01-16T00:11:29.804 INFO:teuthology.orchestra.run.smithi123.stdout: ms die on bug = true 2020-01-16T00:11:29.804 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:11:29.805 INFO:teuthology.orchestra.run.smithi123.stdout: mon max pg per osd = 10000 # >= luminous 2020-01-16T00:11:29.805 INFO:teuthology.orchestra.run.smithi123.stdout: mon pg warn max object skew = 0 2020-01-16T00:11:29.805 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:11:29.805 INFO:teuthology.orchestra.run.smithi123.stdout: # disable pg_autoscaler by default for new pools 2020-01-16T00:11:29.805 INFO:teuthology.orchestra.run.smithi123.stdout: osd_pool_default_pg_autoscale_mode = off 2020-01-16T00:11:29.805 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:11:29.806 INFO:teuthology.orchestra.run.smithi123.stdout: osd pool default size = 2 2020-01-16T00:11:29.806 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:11:29.806 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd allow primary affinity = true 2020-01-16T00:11:29.806 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd allow pg remap = true 2020-01-16T00:11:29.806 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on legacy crush tunables = false 2020-01-16T00:11:29.806 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on crush straw calc version zero = false 2020-01-16T00:11:29.806 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on no sortbitwise = false 2020-01-16T00:11:29.806 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on osd down out interval zero = false 2020-01-16T00:11:29.807 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on too few osds = false 2020-01-16T00:11:29.807 INFO:teuthology.orchestra.run.smithi123.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-01-16T00:11:29.807 INFO:teuthology.orchestra.run.smithi123.stdout: mon_warn_on_pool_no_redundancy = false 2020-01-16T00:11:29.807 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:11:29.807 INFO:teuthology.orchestra.run.smithi123.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-01-16T00:11:29.807 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:11:29.807 INFO:teuthology.orchestra.run.smithi123.stdout: osd default data pool replay window = 5 2020-01-16T00:11:29.808 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:11:29.808 INFO:teuthology.orchestra.run.smithi123.stdout: mon allow pool delete = true 2020-01-16T00:11:29.808 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:11:29.808 INFO:teuthology.orchestra.run.smithi123.stdout: mon cluster log file level = debug 2020-01-16T00:11:29.808 INFO:teuthology.orchestra.run.smithi123.stdout: debug asserts on shutdown = true 2020-01-16T00:11:29.808 INFO:teuthology.orchestra.run.smithi123.stdout: mon host = "[v1:172.21.15.123:6789],[v1:172.21.15.123:6790],[v1:172.21.15.145:6789]" 2020-01-16T00:11:29.808 INFO:teuthology.orchestra.run.smithi123.stdout: osd_object_clean_region_max_num_intervals = 1000 2020-01-16T00:11:29.809 INFO:teuthology.orchestra.run.smithi123.stdout: ms type = async 2020-01-16T00:11:29.809 INFO:teuthology.orchestra.run.smithi123.stdout: ms tcp read timeout = 5 2020-01-16T00:11:29.809 INFO:teuthology.orchestra.run.smithi123.stdout: osd_pool_default_size = 3 2020-01-16T00:11:29.809 INFO:teuthology.orchestra.run.smithi123.stdout: osd_pool_default_min_size = 2 2020-01-16T00:11:29.809 INFO:teuthology.orchestra.run.smithi123.stdout: osd_max_pg_log_entries = 2 2020-01-16T00:11:29.810 INFO:teuthology.orchestra.run.smithi123.stdout: ms bind msgr2 = False 2020-01-16T00:11:29.810 INFO:teuthology.orchestra.run.smithi123.stdout: mon client directed command retry = 5 2020-01-16T00:11:29.810 INFO:teuthology.orchestra.run.smithi123.stdout: osd_async_recovery_min_cost = 1 2020-01-16T00:11:29.810 INFO:teuthology.orchestra.run.smithi123.stdout: ms inject socket failures = 2500 2020-01-16T00:11:29.810 INFO:teuthology.orchestra.run.smithi123.stdout: osd_min_pg_log_entries = 1 2020-01-16T00:11:29.810 INFO:teuthology.orchestra.run.smithi123.stdout: fsid = 864e7191-e3db-4201-b88a-a4873d1eb7fe 2020-01-16T00:11:29.810 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:11:29.811 INFO:teuthology.orchestra.run.smithi123.stdout:[osd] 2020-01-16T00:11:29.811 INFO:teuthology.orchestra.run.smithi123.stdout: osd journal size = 100 2020-01-16T00:11:29.811 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:11:29.811 INFO:teuthology.orchestra.run.smithi123.stdout: osd scrub load threshold = 5.0 2020-01-16T00:11:29.811 INFO:teuthology.orchestra.run.smithi123.stdout: osd scrub max interval = 120 2020-01-16T00:11:29.811 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:11:29.812 INFO:teuthology.orchestra.run.smithi123.stdout: osd recover clone overlap = true 2020-01-16T00:11:29.812 INFO:teuthology.orchestra.run.smithi123.stdout: osd recovery max chunk = 1048576 2020-01-16T00:11:29.812 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:11:29.812 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug shutdown = true 2020-01-16T00:11:29.812 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug op order = true 2020-01-16T00:11:29.812 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug verify stray on activate = true 2020-01-16T00:11:29.812 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:11:29.813 INFO:teuthology.orchestra.run.smithi123.stdout: osd open classes on start = true 2020-01-16T00:11:29.813 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug pg log writeout = true 2020-01-16T00:11:29.813 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:11:29.813 INFO:teuthology.orchestra.run.smithi123.stdout: osd deep scrub update digest min age = 30 2020-01-16T00:11:29.813 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:11:29.813 INFO:teuthology.orchestra.run.smithi123.stdout: osd map max advance = 10 2020-01-16T00:11:29.814 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:11:29.814 INFO:teuthology.orchestra.run.smithi123.stdout: journal zero on create = true 2020-01-16T00:11:29.814 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:11:29.814 INFO:teuthology.orchestra.run.smithi123.stdout: filestore ondisk finisher threads = 3 2020-01-16T00:11:29.814 INFO:teuthology.orchestra.run.smithi123.stdout: filestore apply finisher threads = 3 2020-01-16T00:11:29.814 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:11:29.814 INFO:teuthology.orchestra.run.smithi123.stdout: bdev debug aio = true 2020-01-16T00:11:29.814 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug misdirected ops = true 2020-01-16T00:11:29.815 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd full ratio = 0.9 2020-01-16T00:11:29.815 INFO:teuthology.orchestra.run.smithi123.stdout: osd op queue cut off = debug_random 2020-01-16T00:11:29.815 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd backfillfull_ratio = 0.85 2020-01-16T00:11:29.815 INFO:teuthology.orchestra.run.smithi123.stdout: debug journal = 20 2020-01-16T00:11:29.815 INFO:teuthology.orchestra.run.smithi123.stdout: debug osd = 25 2020-01-16T00:11:29.816 INFO:teuthology.orchestra.run.smithi123.stdout: debug bluestore = 20 2020-01-16T00:11:29.816 INFO:teuthology.orchestra.run.smithi123.stdout: debug bluefs = 20 2020-01-16T00:11:29.816 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug verify missing on start = True 2020-01-16T00:11:29.816 INFO:teuthology.orchestra.run.smithi123.stdout: debug ms = 1 2020-01-16T00:11:29.816 INFO:teuthology.orchestra.run.smithi123.stdout: debug rocksdb = 10 2020-01-16T00:11:29.816 INFO:teuthology.orchestra.run.smithi123.stdout: osd shutdown pgref assert = True 2020-01-16T00:11:29.816 INFO:teuthology.orchestra.run.smithi123.stdout: bdev async discard = True 2020-01-16T00:11:29.817 INFO:teuthology.orchestra.run.smithi123.stdout: bluestore fsck on mount = True 2020-01-16T00:11:29.817 INFO:teuthology.orchestra.run.smithi123.stdout: osd max backfills = 6 2020-01-16T00:11:29.817 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd nearfull ratio = 0.8 2020-01-16T00:11:29.817 INFO:teuthology.orchestra.run.smithi123.stdout: osd memory target = 1610612736 2020-01-16T00:11:29.817 INFO:teuthology.orchestra.run.smithi123.stdout: osd objectstore = bluestore 2020-01-16T00:11:29.817 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug verify cached snaps = True 2020-01-16T00:11:29.818 INFO:teuthology.orchestra.run.smithi123.stdout: osd map cache size = 1 2020-01-16T00:11:29.818 INFO:teuthology.orchestra.run.smithi123.stdout: osd op queue = debug_random 2020-01-16T00:11:29.818 INFO:teuthology.orchestra.run.smithi123.stdout: osd scrub during recovery = False 2020-01-16T00:11:29.818 INFO:teuthology.orchestra.run.smithi123.stdout: bluestore block size = 96636764160 2020-01-16T00:11:29.818 INFO:teuthology.orchestra.run.smithi123.stdout: osd scrub min interval = 60 2020-01-16T00:11:29.818 INFO:teuthology.orchestra.run.smithi123.stdout: debug filestore = 20 2020-01-16T00:11:29.818 INFO:teuthology.orchestra.run.smithi123.stdout: bdev enable discard = True 2020-01-16T00:11:29.819 INFO:teuthology.orchestra.run.smithi123.stdout: osd failsafe full ratio = 0.95 2020-01-16T00:11:29.819 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:11:29.819 INFO:teuthology.orchestra.run.smithi123.stdout:[mgr] 2020-01-16T00:11:29.819 INFO:teuthology.orchestra.run.smithi123.stdout: debug ms = 1 2020-01-16T00:11:29.819 INFO:teuthology.orchestra.run.smithi123.stdout: debug mgr = 20 2020-01-16T00:11:29.819 INFO:teuthology.orchestra.run.smithi123.stdout: debug mon = 20 2020-01-16T00:11:29.819 INFO:teuthology.orchestra.run.smithi123.stdout: debug auth = 20 2020-01-16T00:11:29.820 INFO:teuthology.orchestra.run.smithi123.stdout: mon reweight min pgs per osd = 4 2020-01-16T00:11:29.820 INFO:teuthology.orchestra.run.smithi123.stdout: mon reweight min bytes per osd = 10 2020-01-16T00:11:29.820 INFO:teuthology.orchestra.run.smithi123.stdout: mgr/telemetry/nag = false 2020-01-16T00:11:29.820 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:11:29.820 INFO:teuthology.orchestra.run.smithi123.stdout:[mon] 2020-01-16T00:11:29.820 INFO:teuthology.orchestra.run.smithi123.stdout: debug ms = 1 2020-01-16T00:11:29.820 INFO:teuthology.orchestra.run.smithi123.stdout: debug mon = 20 2020-01-16T00:11:29.821 INFO:teuthology.orchestra.run.smithi123.stdout: debug paxos = 20 2020-01-16T00:11:29.821 INFO:teuthology.orchestra.run.smithi123.stdout: debug auth = 20 2020-01-16T00:11:29.821 INFO:teuthology.orchestra.run.smithi123.stdout: mon data avail warn = 5 2020-01-16T00:11:29.821 INFO:teuthology.orchestra.run.smithi123.stdout: mon mgr mkfs grace = 240 2020-01-16T00:11:29.821 INFO:teuthology.orchestra.run.smithi123.stdout: mon reweight min pgs per osd = 4 2020-01-16T00:11:29.821 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd reporter subtree level = osd 2020-01-16T00:11:29.821 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd prime pg temp = true 2020-01-16T00:11:29.822 INFO:teuthology.orchestra.run.smithi123.stdout: mon reweight min bytes per osd = 10 2020-01-16T00:11:29.822 INFO:teuthology.orchestra.run.smithi123.stdout: mon min osdmap epochs = 50 2020-01-16T00:11:29.822 INFO:teuthology.orchestra.run.smithi123.stdout: mon osdmap full prune min = 15 2020-01-16T00:11:29.822 INFO:teuthology.orchestra.run.smithi123.stdout: mon osdmap full prune interval = 2 2020-01-16T00:11:29.822 INFO:teuthology.orchestra.run.smithi123.stdout: mon osdmap full prune txsize = 2 2020-01-16T00:11:29.822 INFO:teuthology.orchestra.run.smithi123.stdout: paxos service trim min = 10 2020-01-16T00:11:29.822 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:11:29.823 INFO:teuthology.orchestra.run.smithi123.stdout:[client] 2020-01-16T00:11:29.823 INFO:teuthology.orchestra.run.smithi123.stdout: rgw cache enabled = true 2020-01-16T00:11:29.823 INFO:teuthology.orchestra.run.smithi123.stdout: rgw enable ops log = true 2020-01-16T00:11:29.823 INFO:teuthology.orchestra.run.smithi123.stdout: rgw enable usage log = true 2020-01-16T00:11:29.823 INFO:teuthology.orchestra.run.smithi123.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-01-16T00:11:29.823 INFO:teuthology.orchestra.run.smithi123.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-01-16T00:11:29.823 INFO:teuthology.orchestra.run.smithi123.stdout:[mon.a] 2020-01-16T00:11:29.824 INFO:teuthology.orchestra.run.smithi123.stdout:[mon.c] 2020-01-16T00:11:29.824 INFO:teuthology.orchestra.run.smithi123.stdout:[mon.b] 2020-01-16T00:11:29.825 INFO:teuthology.orchestra.run.smithi145.stdout:[global] 2020-01-16T00:11:29.825 INFO:teuthology.orchestra.run.smithi145.stdout: chdir = "" 2020-01-16T00:11:29.826 INFO:teuthology.orchestra.run.smithi145.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-01-16T00:11:29.826 INFO:teuthology.orchestra.run.smithi145.stdout: auth supported = cephx 2020-01-16T00:11:29.826 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-01-16T00:11:29.826 INFO:teuthology.orchestra.run.smithi145.stdout: filestore xattr use omap = true 2020-01-16T00:11:29.826 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-01-16T00:11:29.826 INFO:teuthology.orchestra.run.smithi145.stdout: mon clock drift allowed = 1.000 2020-01-16T00:11:29.827 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-01-16T00:11:29.827 INFO:teuthology.orchestra.run.smithi145.stdout: osd crush chooseleaf type = 0 2020-01-16T00:11:29.827 INFO:teuthology.orchestra.run.smithi145.stdout: auth debug = true 2020-01-16T00:11:29.827 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-01-16T00:11:29.827 INFO:teuthology.orchestra.run.smithi145.stdout: ms die on old message = true 2020-01-16T00:11:29.827 INFO:teuthology.orchestra.run.smithi145.stdout: ms die on bug = true 2020-01-16T00:11:29.827 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-01-16T00:11:29.828 INFO:teuthology.orchestra.run.smithi145.stdout: mon max pg per osd = 10000 # >= luminous 2020-01-16T00:11:29.828 INFO:teuthology.orchestra.run.smithi145.stdout: mon pg warn max object skew = 0 2020-01-16T00:11:29.828 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-01-16T00:11:29.828 INFO:teuthology.orchestra.run.smithi145.stdout: # disable pg_autoscaler by default for new pools 2020-01-16T00:11:29.828 INFO:teuthology.orchestra.run.smithi145.stdout: osd_pool_default_pg_autoscale_mode = off 2020-01-16T00:11:29.828 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-01-16T00:11:29.828 INFO:teuthology.orchestra.run.smithi145.stdout: osd pool default size = 2 2020-01-16T00:11:29.828 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-01-16T00:11:29.829 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd allow primary affinity = true 2020-01-16T00:11:29.829 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd allow pg remap = true 2020-01-16T00:11:29.829 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on legacy crush tunables = false 2020-01-16T00:11:29.829 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on crush straw calc version zero = false 2020-01-16T00:11:29.829 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on no sortbitwise = false 2020-01-16T00:11:29.829 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on osd down out interval zero = false 2020-01-16T00:11:29.829 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on too few osds = false 2020-01-16T00:11:29.830 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-01-16T00:11:29.830 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_pool_no_redundancy = false 2020-01-16T00:11:29.830 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-01-16T00:11:29.831 INFO:teuthology.orchestra.run.smithi145.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-01-16T00:11:29.831 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-01-16T00:11:29.831 INFO:teuthology.orchestra.run.smithi145.stdout: osd default data pool replay window = 5 2020-01-16T00:11:29.831 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-01-16T00:11:29.831 INFO:teuthology.orchestra.run.smithi145.stdout: mon allow pool delete = true 2020-01-16T00:11:29.831 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-01-16T00:11:29.831 INFO:teuthology.orchestra.run.smithi145.stdout: mon cluster log file level = debug 2020-01-16T00:11:29.832 INFO:teuthology.orchestra.run.smithi145.stdout: debug asserts on shutdown = true 2020-01-16T00:11:29.832 INFO:teuthology.orchestra.run.smithi145.stdout: mon host = "[v1:172.21.15.123:6789],[v1:172.21.15.123:6790],[v1:172.21.15.145:6789]" 2020-01-16T00:11:29.832 INFO:teuthology.orchestra.run.smithi145.stdout: osd_object_clean_region_max_num_intervals = 1000 2020-01-16T00:11:29.832 INFO:teuthology.orchestra.run.smithi145.stdout: ms type = async 2020-01-16T00:11:29.832 INFO:teuthology.orchestra.run.smithi145.stdout: ms tcp read timeout = 5 2020-01-16T00:11:29.832 INFO:teuthology.orchestra.run.smithi145.stdout: osd_pool_default_size = 3 2020-01-16T00:11:29.832 INFO:teuthology.orchestra.run.smithi145.stdout: osd_pool_default_min_size = 2 2020-01-16T00:11:29.833 INFO:teuthology.orchestra.run.smithi145.stdout: osd_max_pg_log_entries = 2 2020-01-16T00:11:29.833 INFO:teuthology.orchestra.run.smithi145.stdout: ms bind msgr2 = False 2020-01-16T00:11:29.833 INFO:teuthology.orchestra.run.smithi145.stdout: mon client directed command retry = 5 2020-01-16T00:11:29.833 INFO:teuthology.orchestra.run.smithi145.stdout: osd_async_recovery_min_cost = 1 2020-01-16T00:11:29.833 INFO:teuthology.orchestra.run.smithi145.stdout: ms inject socket failures = 2500 2020-01-16T00:11:29.833 INFO:teuthology.orchestra.run.smithi145.stdout: osd_min_pg_log_entries = 1 2020-01-16T00:11:29.833 INFO:teuthology.orchestra.run.smithi145.stdout: fsid = 864e7191-e3db-4201-b88a-a4873d1eb7fe 2020-01-16T00:11:29.833 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-01-16T00:11:29.834 INFO:teuthology.orchestra.run.smithi145.stdout:[osd] 2020-01-16T00:11:29.834 INFO:teuthology.orchestra.run.smithi145.stdout: osd journal size = 100 2020-01-16T00:11:29.834 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-01-16T00:11:29.834 INFO:teuthology.orchestra.run.smithi145.stdout: osd scrub load threshold = 5.0 2020-01-16T00:11:29.834 INFO:teuthology.orchestra.run.smithi145.stdout: osd scrub max interval = 120 2020-01-16T00:11:29.834 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-01-16T00:11:29.834 INFO:teuthology.orchestra.run.smithi145.stdout: osd recover clone overlap = true 2020-01-16T00:11:29.835 INFO:teuthology.orchestra.run.smithi145.stdout: osd recovery max chunk = 1048576 2020-01-16T00:11:29.835 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-01-16T00:11:29.835 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug shutdown = true 2020-01-16T00:11:29.835 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug op order = true 2020-01-16T00:11:29.835 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug verify stray on activate = true 2020-01-16T00:11:29.835 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-01-16T00:11:29.835 INFO:teuthology.orchestra.run.smithi145.stdout: osd open classes on start = true 2020-01-16T00:11:29.835 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug pg log writeout = true 2020-01-16T00:11:29.836 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-01-16T00:11:29.836 INFO:teuthology.orchestra.run.smithi145.stdout: osd deep scrub update digest min age = 30 2020-01-16T00:11:29.836 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-01-16T00:11:29.836 INFO:teuthology.orchestra.run.smithi145.stdout: osd map max advance = 10 2020-01-16T00:11:29.836 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-01-16T00:11:29.837 INFO:teuthology.orchestra.run.smithi145.stdout: journal zero on create = true 2020-01-16T00:11:29.837 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-01-16T00:11:29.837 INFO:teuthology.orchestra.run.smithi145.stdout: filestore ondisk finisher threads = 3 2020-01-16T00:11:29.837 INFO:teuthology.orchestra.run.smithi145.stdout: filestore apply finisher threads = 3 2020-01-16T00:11:29.837 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-01-16T00:11:29.837 INFO:teuthology.orchestra.run.smithi145.stdout: bdev debug aio = true 2020-01-16T00:11:29.837 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug misdirected ops = true 2020-01-16T00:11:29.838 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd full ratio = 0.9 2020-01-16T00:11:29.838 INFO:teuthology.orchestra.run.smithi145.stdout: osd op queue cut off = debug_random 2020-01-16T00:11:29.838 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd backfillfull_ratio = 0.85 2020-01-16T00:11:29.838 INFO:teuthology.orchestra.run.smithi145.stdout: debug journal = 20 2020-01-16T00:11:29.838 INFO:teuthology.orchestra.run.smithi145.stdout: debug osd = 25 2020-01-16T00:11:29.838 INFO:teuthology.orchestra.run.smithi145.stdout: debug bluestore = 20 2020-01-16T00:11:29.838 INFO:teuthology.orchestra.run.smithi145.stdout: debug bluefs = 20 2020-01-16T00:11:29.838 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug verify missing on start = True 2020-01-16T00:11:29.839 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2020-01-16T00:11:29.839 INFO:teuthology.orchestra.run.smithi145.stdout: debug rocksdb = 10 2020-01-16T00:11:29.839 INFO:teuthology.orchestra.run.smithi145.stdout: osd shutdown pgref assert = True 2020-01-16T00:11:29.839 INFO:teuthology.orchestra.run.smithi145.stdout: bdev async discard = True 2020-01-16T00:11:29.839 INFO:teuthology.orchestra.run.smithi145.stdout: bluestore fsck on mount = True 2020-01-16T00:11:29.839 INFO:teuthology.orchestra.run.smithi145.stdout: osd max backfills = 6 2020-01-16T00:11:29.839 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd nearfull ratio = 0.8 2020-01-16T00:11:29.840 INFO:teuthology.orchestra.run.smithi145.stdout: osd memory target = 1610612736 2020-01-16T00:11:29.840 INFO:teuthology.orchestra.run.smithi145.stdout: osd objectstore = bluestore 2020-01-16T00:11:29.840 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug verify cached snaps = True 2020-01-16T00:11:29.840 INFO:teuthology.orchestra.run.smithi145.stdout: osd map cache size = 1 2020-01-16T00:11:29.840 INFO:teuthology.orchestra.run.smithi145.stdout: osd op queue = debug_random 2020-01-16T00:11:29.840 INFO:teuthology.orchestra.run.smithi145.stdout: osd scrub during recovery = False 2020-01-16T00:11:29.840 INFO:teuthology.orchestra.run.smithi145.stdout: bluestore block size = 96636764160 2020-01-16T00:11:29.840 INFO:teuthology.orchestra.run.smithi145.stdout: osd scrub min interval = 60 2020-01-16T00:11:29.841 INFO:teuthology.orchestra.run.smithi145.stdout: debug filestore = 20 2020-01-16T00:11:29.841 INFO:teuthology.orchestra.run.smithi145.stdout: bdev enable discard = True 2020-01-16T00:11:29.841 INFO:teuthology.orchestra.run.smithi145.stdout: osd failsafe full ratio = 0.95 2020-01-16T00:11:29.841 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-01-16T00:11:29.841 INFO:teuthology.orchestra.run.smithi145.stdout:[mgr] 2020-01-16T00:11:29.841 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2020-01-16T00:11:29.841 INFO:teuthology.orchestra.run.smithi145.stdout: debug mgr = 20 2020-01-16T00:11:29.842 INFO:teuthology.orchestra.run.smithi145.stdout: debug mon = 20 2020-01-16T00:11:29.842 INFO:teuthology.orchestra.run.smithi145.stdout: debug auth = 20 2020-01-16T00:11:29.842 INFO:teuthology.orchestra.run.smithi145.stdout: mon reweight min pgs per osd = 4 2020-01-16T00:11:29.842 INFO:teuthology.orchestra.run.smithi145.stdout: mon reweight min bytes per osd = 10 2020-01-16T00:11:29.842 INFO:teuthology.orchestra.run.smithi145.stdout: mgr/telemetry/nag = false 2020-01-16T00:11:29.842 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-01-16T00:11:29.842 INFO:teuthology.orchestra.run.smithi145.stdout:[mon] 2020-01-16T00:11:29.842 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2020-01-16T00:11:29.843 INFO:teuthology.orchestra.run.smithi145.stdout: debug mon = 20 2020-01-16T00:11:29.843 INFO:teuthology.orchestra.run.smithi145.stdout: debug paxos = 20 2020-01-16T00:11:29.843 INFO:teuthology.orchestra.run.smithi145.stdout: debug auth = 20 2020-01-16T00:11:29.843 INFO:teuthology.orchestra.run.smithi145.stdout: mon data avail warn = 5 2020-01-16T00:11:29.843 INFO:teuthology.orchestra.run.smithi145.stdout: mon mgr mkfs grace = 240 2020-01-16T00:11:29.843 INFO:teuthology.orchestra.run.smithi145.stdout: mon reweight min pgs per osd = 4 2020-01-16T00:11:29.843 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd reporter subtree level = osd 2020-01-16T00:11:29.844 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd prime pg temp = true 2020-01-16T00:11:29.844 INFO:teuthology.orchestra.run.smithi145.stdout: mon reweight min bytes per osd = 10 2020-01-16T00:11:29.844 INFO:teuthology.orchestra.run.smithi145.stdout: mon min osdmap epochs = 50 2020-01-16T00:11:29.844 INFO:teuthology.orchestra.run.smithi145.stdout: mon osdmap full prune min = 15 2020-01-16T00:11:29.844 INFO:teuthology.orchestra.run.smithi145.stdout: mon osdmap full prune interval = 2 2020-01-16T00:11:29.844 INFO:teuthology.orchestra.run.smithi145.stdout: mon osdmap full prune txsize = 2 2020-01-16T00:11:29.844 INFO:teuthology.orchestra.run.smithi145.stdout: paxos service trim min = 10 2020-01-16T00:11:29.845 INFO:teuthology.orchestra.run.smithi145.stdout: 2020-01-16T00:11:29.845 INFO:teuthology.orchestra.run.smithi145.stdout:[client] 2020-01-16T00:11:29.845 INFO:teuthology.orchestra.run.smithi145.stdout: rgw cache enabled = true 2020-01-16T00:11:29.845 INFO:teuthology.orchestra.run.smithi145.stdout: rgw enable ops log = true 2020-01-16T00:11:29.845 INFO:teuthology.orchestra.run.smithi145.stdout: rgw enable usage log = true 2020-01-16T00:11:29.845 INFO:teuthology.orchestra.run.smithi145.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-01-16T00:11:29.845 INFO:teuthology.orchestra.run.smithi145.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-01-16T00:11:29.845 INFO:teuthology.orchestra.run.smithi145.stdout:[mon.a] 2020-01-16T00:11:29.846 INFO:teuthology.orchestra.run.smithi145.stdout:[mon.c] 2020-01-16T00:11:29.846 INFO:teuthology.orchestra.run.smithi145.stdout:[mon.b] 2020-01-16T00:11:29.846 INFO:tasks.ceph:Creating admin key on mon.a... 2020-01-16T00:11:29.847 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2020-01-16T00:11:29.913 INFO:tasks.ceph:Copying monmap to all nodes... 2020-01-16T00:11:29.962 DEBUG:teuthology.orchestra.remote:smithi123:/etc/ceph/ceph.keyring is 206B 2020-01-16T00:11:29.980 DEBUG:teuthology.orchestra.remote:smithi123:/home/ubuntu/cephtest/ceph.monmap is 274B 2020-01-16T00:11:29.990 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi123.front.sepia.ceph.com 2020-01-16T00:11:29.990 INFO:teuthology.orchestra.run.smithi123:> sudo sh -c 'cat > /etc/ceph/ceph.keyring' && sudo chmod 0644 /etc/ceph/ceph.keyring 2020-01-16T00:11:30.046 INFO:teuthology.orchestra.run.smithi123:> cat > /home/ubuntu/cephtest/ceph.monmap 2020-01-16T00:11:30.135 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi145.front.sepia.ceph.com 2020-01-16T00:11:30.135 INFO:teuthology.orchestra.run.smithi145:> sudo sh -c 'cat > /etc/ceph/ceph.keyring' && sudo chmod 0644 /etc/ceph/ceph.keyring 2020-01-16T00:11:30.194 INFO:teuthology.orchestra.run.smithi145:> cat > /home/ubuntu/cephtest/ceph.monmap 2020-01-16T00:11:30.282 INFO:tasks.ceph:Setting up mon nodes... 2020-01-16T00:11:30.282 INFO:tasks.ceph:Setting up mgr nodes... 2020-01-16T00:11:30.282 INFO:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2020-01-16T00:11:30.325 INFO:teuthology.orchestra.run.smithi123.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2020-01-16T00:11:30.328 INFO:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2020-01-16T00:11:30.374 INFO:teuthology.orchestra.run.smithi145.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2020-01-16T00:11:30.376 INFO:tasks.ceph:Setting up mds nodes... 2020-01-16T00:11:30.377 INFO:tasks.ceph_client:Setting up client nodes... 2020-01-16T00:11:30.377 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2020-01-16T00:11:30.410 INFO:teuthology.orchestra.run.smithi123.stdout:creating /etc/ceph/ceph.client.0.keyring 2020-01-16T00:11:30.423 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2020-01-16T00:11:30.465 INFO:teuthology.orchestra.run.smithi145.stdout:creating /etc/ceph/ceph.client.1.keyring 2020-01-16T00:11:30.470 INFO:tasks.ceph:Running mkfs on osd nodes... 2020-01-16T00:11:30.470 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi123.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi145.front.sepia.ceph.com'): {'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'}} 2020-01-16T00:11:30.470 INFO:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2020-01-16T00:11:30.485 INFO:tasks.ceph:{'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'} 2020-01-16T00:11:30.486 INFO:tasks.ceph:{} 2020-01-16T00:11:30.486 INFO:tasks.ceph:osd.0 2020-01-16T00:11:30.486 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi123.front.sepia.ceph.com 2020-01-16T00:11:30.486 INFO:teuthology.orchestra.run.smithi123:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2020-01-16T00:11:31.028 INFO:teuthology.orchestra.run.smithi123.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2020-01-16T00:11:31.028 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 attr=2, projid32bit=1 2020-01-16T00:11:31.029 INFO:teuthology.orchestra.run.smithi123.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-01-16T00:11:31.029 INFO:teuthology.orchestra.run.smithi123.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-01-16T00:11:31.029 INFO:teuthology.orchestra.run.smithi123.stdout: = sunit=0 swidth=0 blks 2020-01-16T00:11:31.029 INFO:teuthology.orchestra.run.smithi123.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-01-16T00:11:31.029 INFO:teuthology.orchestra.run.smithi123.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-01-16T00:11:31.029 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-01-16T00:11:31.030 INFO:teuthology.orchestra.run.smithi123.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-01-16T00:11:31.030 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi123.front.sepia.ceph.com -o noatime 2020-01-16T00:11:31.031 INFO:teuthology.orchestra.run.smithi123:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0 2020-01-16T00:11:31.109 INFO:teuthology.orchestra.run.smithi123:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2020-01-16T00:11:31.120 INFO:teuthology.orchestra.run.smithi123.stderr:sudo: /sbin/restorecon: command not found 2020-01-16T00:11:31.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-01-16T00:11:31.120 INFO:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2020-01-16T00:11:31.177 INFO:tasks.ceph:{'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'} 2020-01-16T00:11:31.177 INFO:tasks.ceph:{} 2020-01-16T00:11:31.177 INFO:tasks.ceph:osd.1 2020-01-16T00:11:31.178 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi123.front.sepia.ceph.com 2020-01-16T00:11:31.178 INFO:teuthology.orchestra.run.smithi123:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-01-16T00:11:31.711 INFO:teuthology.orchestra.run.smithi123.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-01-16T00:11:31.711 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 attr=2, projid32bit=1 2020-01-16T00:11:31.711 INFO:teuthology.orchestra.run.smithi123.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-01-16T00:11:31.712 INFO:teuthology.orchestra.run.smithi123.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-01-16T00:11:31.712 INFO:teuthology.orchestra.run.smithi123.stdout: = sunit=0 swidth=0 blks 2020-01-16T00:11:31.712 INFO:teuthology.orchestra.run.smithi123.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-01-16T00:11:31.712 INFO:teuthology.orchestra.run.smithi123.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-01-16T00:11:31.713 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-01-16T00:11:31.713 INFO:teuthology.orchestra.run.smithi123.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-01-16T00:11:31.714 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi123.front.sepia.ceph.com -o noatime 2020-01-16T00:11:31.714 INFO:teuthology.orchestra.run.smithi123:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-1 2020-01-16T00:11:31.763 INFO:teuthology.orchestra.run.smithi123:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2020-01-16T00:11:31.775 INFO:teuthology.orchestra.run.smithi123.stderr:sudo: /sbin/restorecon: command not found 2020-01-16T00:11:31.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-01-16T00:11:31.775 INFO:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2020-01-16T00:11:31.833 INFO:tasks.ceph:{'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'} 2020-01-16T00:11:31.833 INFO:tasks.ceph:{} 2020-01-16T00:11:31.833 INFO:tasks.ceph:osd.2 2020-01-16T00:11:31.833 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi123.front.sepia.ceph.com 2020-01-16T00:11:31.833 INFO:teuthology.orchestra.run.smithi123:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-01-16T00:11:32.380 INFO:teuthology.orchestra.run.smithi123.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-01-16T00:11:32.381 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 attr=2, projid32bit=1 2020-01-16T00:11:32.381 INFO:teuthology.orchestra.run.smithi123.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-01-16T00:11:32.381 INFO:teuthology.orchestra.run.smithi123.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-01-16T00:11:32.381 INFO:teuthology.orchestra.run.smithi123.stdout: = sunit=0 swidth=0 blks 2020-01-16T00:11:32.381 INFO:teuthology.orchestra.run.smithi123.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-01-16T00:11:32.381 INFO:teuthology.orchestra.run.smithi123.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-01-16T00:11:32.382 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-01-16T00:11:32.382 INFO:teuthology.orchestra.run.smithi123.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-01-16T00:11:32.383 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi123.front.sepia.ceph.com -o noatime 2020-01-16T00:11:32.383 INFO:teuthology.orchestra.run.smithi123:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2 2020-01-16T00:11:32.430 INFO:teuthology.orchestra.run.smithi123:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2020-01-16T00:11:32.438 INFO:teuthology.orchestra.run.smithi123.stderr:sudo: /sbin/restorecon: command not found 2020-01-16T00:11:32.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-01-16T00:11:32.441 INFO:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2020-01-16T00:11:32.496 INFO:tasks.ceph:{'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'} 2020-01-16T00:11:32.496 INFO:tasks.ceph:{} 2020-01-16T00:11:32.496 INFO:tasks.ceph:osd.3 2020-01-16T00:11:32.496 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi123.front.sepia.ceph.com 2020-01-16T00:11:32.497 INFO:teuthology.orchestra.run.smithi123:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-01-16T00:11:33.061 INFO:teuthology.orchestra.run.smithi123.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-01-16T00:11:33.061 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 attr=2, projid32bit=1 2020-01-16T00:11:33.061 INFO:teuthology.orchestra.run.smithi123.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-01-16T00:11:33.061 INFO:teuthology.orchestra.run.smithi123.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-01-16T00:11:33.061 INFO:teuthology.orchestra.run.smithi123.stdout: = sunit=0 swidth=0 blks 2020-01-16T00:11:33.062 INFO:teuthology.orchestra.run.smithi123.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-01-16T00:11:33.062 INFO:teuthology.orchestra.run.smithi123.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-01-16T00:11:33.062 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-01-16T00:11:33.062 INFO:teuthology.orchestra.run.smithi123.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-01-16T00:11:33.063 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi123.front.sepia.ceph.com -o noatime 2020-01-16T00:11:33.063 INFO:teuthology.orchestra.run.smithi123:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-3 2020-01-16T00:11:33.135 INFO:teuthology.orchestra.run.smithi123:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2020-01-16T00:11:33.152 INFO:teuthology.orchestra.run.smithi123.stderr:sudo: /sbin/restorecon: command not found 2020-01-16T00:11:33.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-01-16T00:11:33.152 INFO:teuthology.orchestra.run.smithi123:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-01-16T00:11:33.253 INFO:teuthology.orchestra.run.smithi123.stderr:2020-01-16T00:11:33.251+0000 7efc7c1d5d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2020-01-16T00:11:33.254 INFO:teuthology.orchestra.run.smithi123.stderr:2020-01-16T00:11:33.255+0000 7efc7c1d5d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2020-01-16T00:11:33.255 INFO:teuthology.orchestra.run.smithi123.stderr:2020-01-16T00:11:33.255+0000 7efc7c1d5d80 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2020-01-16T00:11:33.255 INFO:teuthology.orchestra.run.smithi123.stderr:2020-01-16T00:11:33.255+0000 7efc7c1d5d80 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2020-01-16T00:11:33.255 INFO:teuthology.orchestra.run.smithi123.stderr:2020-01-16T00:11:33.255+0000 7efc7c1d5d80 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2020-01-16T00:11:33.265 INFO:teuthology.orchestra.run.smithi123.stderr:2020-01-16T00:11:33.263+0000 7efc7c1d5d80 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2020-01-16T00:11:35.721 INFO:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2020-01-16T00:11:35.737 INFO:teuthology.orchestra.run.smithi123:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-01-16T00:11:35.820 INFO:teuthology.orchestra.run.smithi123.stderr:2020-01-16T00:11:35.819+0000 7ff39f30dd80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2020-01-16T00:11:35.820 INFO:teuthology.orchestra.run.smithi123.stderr:2020-01-16T00:11:35.819+0000 7ff39f30dd80 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2020-01-16T00:11:35.820 INFO:teuthology.orchestra.run.smithi123.stderr:2020-01-16T00:11:35.819+0000 7ff39f30dd80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2020-01-16T00:11:35.820 INFO:teuthology.orchestra.run.smithi123.stderr:2020-01-16T00:11:35.819+0000 7ff39f30dd80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2020-01-16T00:11:35.820 INFO:teuthology.orchestra.run.smithi123.stderr:2020-01-16T00:11:35.819+0000 7ff39f30dd80 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2020-01-16T00:11:35.821 INFO:teuthology.orchestra.run.smithi123.stderr:2020-01-16T00:11:35.819+0000 7ff39f30dd80 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2020-01-16T00:11:38.267 INFO:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2020-01-16T00:11:38.325 INFO:teuthology.orchestra.run.smithi123:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-01-16T00:11:38.407 INFO:teuthology.orchestra.run.smithi123.stderr:2020-01-16T00:11:38.407+0000 7fc514294d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2020-01-16T00:11:38.408 INFO:teuthology.orchestra.run.smithi123.stderr:2020-01-16T00:11:38.407+0000 7fc514294d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2020-01-16T00:11:38.408 INFO:teuthology.orchestra.run.smithi123.stderr:2020-01-16T00:11:38.407+0000 7fc514294d80 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2020-01-16T00:11:38.408 INFO:teuthology.orchestra.run.smithi123.stderr:2020-01-16T00:11:38.407+0000 7fc514294d80 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2020-01-16T00:11:38.408 INFO:teuthology.orchestra.run.smithi123.stderr:2020-01-16T00:11:38.407+0000 7fc514294d80 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2020-01-16T00:11:38.408 INFO:teuthology.orchestra.run.smithi123.stderr:2020-01-16T00:11:38.407+0000 7fc514294d80 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2020-01-16T00:11:40.864 INFO:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2020-01-16T00:11:40.921 INFO:teuthology.orchestra.run.smithi123:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-01-16T00:11:41.000 INFO:teuthology.orchestra.run.smithi123.stderr:2020-01-16T00:11:40.999+0000 7ff432691d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2020-01-16T00:11:41.000 INFO:teuthology.orchestra.run.smithi123.stderr:2020-01-16T00:11:40.999+0000 7ff432691d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2020-01-16T00:11:41.000 INFO:teuthology.orchestra.run.smithi123.stderr:2020-01-16T00:11:40.999+0000 7ff432691d80 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2020-01-16T00:11:41.001 INFO:teuthology.orchestra.run.smithi123.stderr:2020-01-16T00:11:40.999+0000 7ff432691d80 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2020-01-16T00:11:41.001 INFO:teuthology.orchestra.run.smithi123.stderr:2020-01-16T00:11:40.999+0000 7ff432691d80 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2020-01-16T00:11:41.001 INFO:teuthology.orchestra.run.smithi123.stderr:2020-01-16T00:11:40.999+0000 7ff432691d80 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2020-01-16T00:11:43.432 INFO:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2020-01-16T00:11:43.488 INFO:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2020-01-16T00:11:43.504 INFO:tasks.ceph:{'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'} 2020-01-16T00:11:43.504 INFO:tasks.ceph:{} 2020-01-16T00:11:43.504 INFO:tasks.ceph:osd.4 2020-01-16T00:11:43.504 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi145.front.sepia.ceph.com 2020-01-16T00:11:43.504 INFO:teuthology.orchestra.run.smithi145:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2020-01-16T00:11:44.037 INFO:teuthology.orchestra.run.smithi145.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2020-01-16T00:11:44.037 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 attr=2, projid32bit=1 2020-01-16T00:11:44.037 INFO:teuthology.orchestra.run.smithi145.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-01-16T00:11:44.038 INFO:teuthology.orchestra.run.smithi145.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-01-16T00:11:44.038 INFO:teuthology.orchestra.run.smithi145.stdout: = sunit=0 swidth=0 blks 2020-01-16T00:11:44.038 INFO:teuthology.orchestra.run.smithi145.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-01-16T00:11:44.038 INFO:teuthology.orchestra.run.smithi145.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-01-16T00:11:44.038 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-01-16T00:11:44.038 INFO:teuthology.orchestra.run.smithi145.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-01-16T00:11:44.039 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi145.front.sepia.ceph.com -o noatime 2020-01-16T00:11:44.039 INFO:teuthology.orchestra.run.smithi145:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-4 2020-01-16T00:11:44.113 INFO:teuthology.orchestra.run.smithi145:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2020-01-16T00:11:44.124 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: /sbin/restorecon: command not found 2020-01-16T00:11:44.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-01-16T00:11:44.126 INFO:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2020-01-16T00:11:44.183 INFO:tasks.ceph:{'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'} 2020-01-16T00:11:44.184 INFO:tasks.ceph:{} 2020-01-16T00:11:44.184 INFO:tasks.ceph:osd.5 2020-01-16T00:11:44.184 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi145.front.sepia.ceph.com 2020-01-16T00:11:44.184 INFO:teuthology.orchestra.run.smithi145:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-01-16T00:11:44.707 INFO:teuthology.orchestra.run.smithi145.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-01-16T00:11:44.707 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 attr=2, projid32bit=1 2020-01-16T00:11:44.707 INFO:teuthology.orchestra.run.smithi145.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-01-16T00:11:44.708 INFO:teuthology.orchestra.run.smithi145.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-01-16T00:11:44.708 INFO:teuthology.orchestra.run.smithi145.stdout: = sunit=0 swidth=0 blks 2020-01-16T00:11:44.708 INFO:teuthology.orchestra.run.smithi145.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-01-16T00:11:44.708 INFO:teuthology.orchestra.run.smithi145.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-01-16T00:11:44.708 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-01-16T00:11:44.708 INFO:teuthology.orchestra.run.smithi145.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-01-16T00:11:44.709 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi145.front.sepia.ceph.com -o noatime 2020-01-16T00:11:44.709 INFO:teuthology.orchestra.run.smithi145:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-5 2020-01-16T00:11:44.756 INFO:teuthology.orchestra.run.smithi145:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2020-01-16T00:11:44.766 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: /sbin/restorecon: command not found 2020-01-16T00:11:44.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-01-16T00:11:44.767 INFO:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2020-01-16T00:11:44.828 INFO:tasks.ceph:{'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'} 2020-01-16T00:11:44.828 INFO:tasks.ceph:{} 2020-01-16T00:11:44.828 INFO:tasks.ceph:osd.6 2020-01-16T00:11:44.828 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi145.front.sepia.ceph.com 2020-01-16T00:11:44.829 INFO:teuthology.orchestra.run.smithi145:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-01-16T00:11:45.345 INFO:teuthology.orchestra.run.smithi145.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-01-16T00:11:45.346 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 attr=2, projid32bit=1 2020-01-16T00:11:45.346 INFO:teuthology.orchestra.run.smithi145.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-01-16T00:11:45.346 INFO:teuthology.orchestra.run.smithi145.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-01-16T00:11:45.346 INFO:teuthology.orchestra.run.smithi145.stdout: = sunit=0 swidth=0 blks 2020-01-16T00:11:45.346 INFO:teuthology.orchestra.run.smithi145.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-01-16T00:11:45.346 INFO:teuthology.orchestra.run.smithi145.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-01-16T00:11:45.346 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-01-16T00:11:45.347 INFO:teuthology.orchestra.run.smithi145.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-01-16T00:11:45.347 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi145.front.sepia.ceph.com -o noatime 2020-01-16T00:11:45.347 INFO:teuthology.orchestra.run.smithi145:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-6 2020-01-16T00:11:45.394 INFO:teuthology.orchestra.run.smithi145:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2020-01-16T00:11:45.447 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: /sbin/restorecon: command not found 2020-01-16T00:11:45.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-01-16T00:11:45.448 INFO:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2020-01-16T00:11:45.507 INFO:tasks.ceph:{'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'} 2020-01-16T00:11:45.507 INFO:tasks.ceph:{} 2020-01-16T00:11:45.507 INFO:tasks.ceph:osd.7 2020-01-16T00:11:45.507 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi145.front.sepia.ceph.com 2020-01-16T00:11:45.507 INFO:teuthology.orchestra.run.smithi145:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-01-16T00:11:46.040 INFO:teuthology.orchestra.run.smithi145.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-01-16T00:11:46.040 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 attr=2, projid32bit=1 2020-01-16T00:11:46.040 INFO:teuthology.orchestra.run.smithi145.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-01-16T00:11:46.040 INFO:teuthology.orchestra.run.smithi145.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-01-16T00:11:46.041 INFO:teuthology.orchestra.run.smithi145.stdout: = sunit=0 swidth=0 blks 2020-01-16T00:11:46.041 INFO:teuthology.orchestra.run.smithi145.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-01-16T00:11:46.047 INFO:teuthology.orchestra.run.smithi145.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-01-16T00:11:46.047 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-01-16T00:11:46.047 INFO:teuthology.orchestra.run.smithi145.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-01-16T00:11:46.048 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi145.front.sepia.ceph.com -o noatime 2020-01-16T00:11:46.049 INFO:teuthology.orchestra.run.smithi145:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-7 2020-01-16T00:11:46.090 INFO:teuthology.orchestra.run.smithi145:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2020-01-16T00:11:46.143 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: /sbin/restorecon: command not found 2020-01-16T00:11:46.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-01-16T00:11:46.145 INFO:teuthology.orchestra.run.smithi145:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-01-16T00:11:46.234 INFO:teuthology.orchestra.run.smithi145.stderr:2020-01-16T00:11:46.233+0000 7fafa4ce0d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2020-01-16T00:11:46.235 INFO:teuthology.orchestra.run.smithi145.stderr:2020-01-16T00:11:46.233+0000 7fafa4ce0d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2020-01-16T00:11:46.236 INFO:teuthology.orchestra.run.smithi145.stderr:2020-01-16T00:11:46.233+0000 7fafa4ce0d80 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2020-01-16T00:11:46.236 INFO:teuthology.orchestra.run.smithi145.stderr:2020-01-16T00:11:46.233+0000 7fafa4ce0d80 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2020-01-16T00:11:46.236 INFO:teuthology.orchestra.run.smithi145.stderr:2020-01-16T00:11:46.233+0000 7fafa4ce0d80 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2020-01-16T00:11:46.243 INFO:teuthology.orchestra.run.smithi145.stderr:2020-01-16T00:11:46.241+0000 7fafa4ce0d80 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2020-01-16T00:11:48.673 INFO:teuthology.orchestra.run.smithi145:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2020-01-16T00:11:48.731 INFO:teuthology.orchestra.run.smithi145:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-01-16T00:11:48.810 INFO:teuthology.orchestra.run.smithi145.stderr:2020-01-16T00:11:48.805+0000 7fbc19aeed80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2020-01-16T00:11:48.810 INFO:teuthology.orchestra.run.smithi145.stderr:2020-01-16T00:11:48.805+0000 7fbc19aeed80 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2020-01-16T00:11:48.810 INFO:teuthology.orchestra.run.smithi145.stderr:2020-01-16T00:11:48.805+0000 7fbc19aeed80 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2020-01-16T00:11:48.810 INFO:teuthology.orchestra.run.smithi145.stderr:2020-01-16T00:11:48.805+0000 7fbc19aeed80 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2020-01-16T00:11:48.811 INFO:teuthology.orchestra.run.smithi145.stderr:2020-01-16T00:11:48.805+0000 7fbc19aeed80 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2020-01-16T00:11:48.811 INFO:teuthology.orchestra.run.smithi145.stderr:2020-01-16T00:11:48.809+0000 7fbc19aeed80 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2020-01-16T00:11:51.251 INFO:teuthology.orchestra.run.smithi145:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2020-01-16T00:11:51.308 INFO:teuthology.orchestra.run.smithi145:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-01-16T00:11:51.390 INFO:teuthology.orchestra.run.smithi145.stderr:2020-01-16T00:11:51.389+0000 7f437e403d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2020-01-16T00:11:51.390 INFO:teuthology.orchestra.run.smithi145.stderr:2020-01-16T00:11:51.389+0000 7f437e403d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2020-01-16T00:11:51.390 INFO:teuthology.orchestra.run.smithi145.stderr:2020-01-16T00:11:51.389+0000 7f437e403d80 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2020-01-16T00:11:51.390 INFO:teuthology.orchestra.run.smithi145.stderr:2020-01-16T00:11:51.389+0000 7f437e403d80 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2020-01-16T00:11:51.390 INFO:teuthology.orchestra.run.smithi145.stderr:2020-01-16T00:11:51.389+0000 7f437e403d80 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2020-01-16T00:11:51.391 INFO:teuthology.orchestra.run.smithi145.stderr:2020-01-16T00:11:51.389+0000 7f437e403d80 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2020-01-16T00:11:53.823 INFO:teuthology.orchestra.run.smithi145:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2020-01-16T00:11:53.880 INFO:teuthology.orchestra.run.smithi145:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-01-16T00:11:53.962 INFO:teuthology.orchestra.run.smithi145.stderr:2020-01-16T00:11:53.961+0000 7f13f8a0ad80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2020-01-16T00:11:53.962 INFO:teuthology.orchestra.run.smithi145.stderr:2020-01-16T00:11:53.961+0000 7f13f8a0ad80 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2020-01-16T00:11:53.962 INFO:teuthology.orchestra.run.smithi145.stderr:2020-01-16T00:11:53.961+0000 7f13f8a0ad80 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2020-01-16T00:11:53.963 INFO:teuthology.orchestra.run.smithi145.stderr:2020-01-16T00:11:53.961+0000 7f13f8a0ad80 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2020-01-16T00:11:53.963 INFO:teuthology.orchestra.run.smithi145.stderr:2020-01-16T00:11:53.961+0000 7f13f8a0ad80 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2020-01-16T00:11:53.963 INFO:teuthology.orchestra.run.smithi145.stderr:2020-01-16T00:11:53.961+0000 7f13f8a0ad80 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2020-01-16T00:11:56.434 INFO:teuthology.orchestra.run.smithi145:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2020-01-16T00:11:56.491 INFO:tasks.ceph:Reading keys from all nodes... 2020-01-16T00:11:56.491 INFO:teuthology.orchestra.run.smithi123:> mktemp 2020-01-16T00:11:56.498 INFO:teuthology.orchestra.run.smithi123.stdout:/tmp/tmp.t2cLoW4NyI 2020-01-16T00:11:56.498 INFO:teuthology.orchestra.run.smithi123:> sudo cp /var/lib/ceph/mgr/ceph-y/keyring /tmp/tmp.t2cLoW4NyI 2020-01-16T00:11:56.556 INFO:teuthology.orchestra.run.smithi123:> sudo chmod 0666 /tmp/tmp.t2cLoW4NyI 2020-01-16T00:11:56.661 DEBUG:teuthology.orchestra.remote:smithi123:/tmp/tmp.t2cLoW4NyI is 56B 2020-01-16T00:11:56.672 INFO:teuthology.orchestra.run.smithi123:> rm -fr /tmp/tmp.t2cLoW4NyI 2020-01-16T00:11:56.678 INFO:teuthology.orchestra.run.smithi123:> mktemp 2020-01-16T00:11:56.726 INFO:teuthology.orchestra.run.smithi123.stdout:/tmp/tmp.LLkYIwwHfR 2020-01-16T00:11:56.727 INFO:teuthology.orchestra.run.smithi123:> sudo cp /var/lib/ceph/osd/ceph-0/keyring /tmp/tmp.LLkYIwwHfR 2020-01-16T00:11:56.784 INFO:teuthology.orchestra.run.smithi123:> sudo chmod 0666 /tmp/tmp.LLkYIwwHfR 2020-01-16T00:11:56.889 DEBUG:teuthology.orchestra.remote:smithi123:/tmp/tmp.LLkYIwwHfR is 56B 2020-01-16T00:11:56.899 INFO:teuthology.orchestra.run.smithi123:> rm -fr /tmp/tmp.LLkYIwwHfR 2020-01-16T00:11:56.904 INFO:teuthology.orchestra.run.smithi123:> mktemp 2020-01-16T00:11:56.950 INFO:teuthology.orchestra.run.smithi123.stdout:/tmp/tmp.QYOuhLzwDD 2020-01-16T00:11:56.951 INFO:teuthology.orchestra.run.smithi123:> sudo cp /var/lib/ceph/osd/ceph-1/keyring /tmp/tmp.QYOuhLzwDD 2020-01-16T00:11:57.008 INFO:teuthology.orchestra.run.smithi123:> sudo chmod 0666 /tmp/tmp.QYOuhLzwDD 2020-01-16T00:11:57.113 DEBUG:teuthology.orchestra.remote:smithi123:/tmp/tmp.QYOuhLzwDD is 56B 2020-01-16T00:11:57.123 INFO:teuthology.orchestra.run.smithi123:> rm -fr /tmp/tmp.QYOuhLzwDD 2020-01-16T00:11:57.129 INFO:teuthology.orchestra.run.smithi123:> mktemp 2020-01-16T00:11:57.178 INFO:teuthology.orchestra.run.smithi123.stdout:/tmp/tmp.2NrCgGzp4b 2020-01-16T00:11:57.179 INFO:teuthology.orchestra.run.smithi123:> sudo cp /var/lib/ceph/osd/ceph-2/keyring /tmp/tmp.2NrCgGzp4b 2020-01-16T00:11:57.236 INFO:teuthology.orchestra.run.smithi123:> sudo chmod 0666 /tmp/tmp.2NrCgGzp4b 2020-01-16T00:11:57.345 DEBUG:teuthology.orchestra.remote:smithi123:/tmp/tmp.2NrCgGzp4b is 56B 2020-01-16T00:11:57.355 INFO:teuthology.orchestra.run.smithi123:> rm -fr /tmp/tmp.2NrCgGzp4b 2020-01-16T00:11:57.361 INFO:teuthology.orchestra.run.smithi123:> mktemp 2020-01-16T00:11:57.410 INFO:teuthology.orchestra.run.smithi123.stdout:/tmp/tmp.WZsGWZeErR 2020-01-16T00:11:57.411 INFO:teuthology.orchestra.run.smithi123:> sudo cp /var/lib/ceph/osd/ceph-3/keyring /tmp/tmp.WZsGWZeErR 2020-01-16T00:11:57.468 INFO:teuthology.orchestra.run.smithi123:> sudo chmod 0666 /tmp/tmp.WZsGWZeErR 2020-01-16T00:11:57.573 DEBUG:teuthology.orchestra.remote:smithi123:/tmp/tmp.WZsGWZeErR is 56B 2020-01-16T00:11:57.624 INFO:teuthology.orchestra.run.smithi123:> rm -fr /tmp/tmp.WZsGWZeErR 2020-01-16T00:11:57.635 INFO:teuthology.orchestra.run.smithi145:> mktemp 2020-01-16T00:11:57.641 INFO:teuthology.orchestra.run.smithi145.stdout:/tmp/tmp.zIuiqdjLrL 2020-01-16T00:11:57.641 INFO:teuthology.orchestra.run.smithi145:> sudo cp /var/lib/ceph/mgr/ceph-x/keyring /tmp/tmp.zIuiqdjLrL 2020-01-16T00:11:57.698 INFO:teuthology.orchestra.run.smithi145:> sudo chmod 0666 /tmp/tmp.zIuiqdjLrL 2020-01-16T00:11:57.812 DEBUG:teuthology.orchestra.remote:smithi145:/tmp/tmp.zIuiqdjLrL is 56B 2020-01-16T00:11:57.861 INFO:teuthology.orchestra.run.smithi145:> rm -fr /tmp/tmp.zIuiqdjLrL 2020-01-16T00:11:57.883 INFO:teuthology.orchestra.run.smithi145:> mktemp 2020-01-16T00:11:57.930 INFO:teuthology.orchestra.run.smithi145.stdout:/tmp/tmp.MSdB438Qsq 2020-01-16T00:11:57.930 INFO:teuthology.orchestra.run.smithi145:> sudo cp /var/lib/ceph/osd/ceph-4/keyring /tmp/tmp.MSdB438Qsq 2020-01-16T00:11:57.987 INFO:teuthology.orchestra.run.smithi145:> sudo chmod 0666 /tmp/tmp.MSdB438Qsq 2020-01-16T00:11:58.095 DEBUG:teuthology.orchestra.remote:smithi145:/tmp/tmp.MSdB438Qsq is 56B 2020-01-16T00:11:58.106 INFO:teuthology.orchestra.run.smithi145:> rm -fr /tmp/tmp.MSdB438Qsq 2020-01-16T00:11:58.112 INFO:teuthology.orchestra.run.smithi145:> mktemp 2020-01-16T00:11:58.162 INFO:teuthology.orchestra.run.smithi145.stdout:/tmp/tmp.Gkm5ty6eiy 2020-01-16T00:11:58.162 INFO:teuthology.orchestra.run.smithi145:> sudo cp /var/lib/ceph/osd/ceph-5/keyring /tmp/tmp.Gkm5ty6eiy 2020-01-16T00:11:58.219 INFO:teuthology.orchestra.run.smithi145:> sudo chmod 0666 /tmp/tmp.Gkm5ty6eiy 2020-01-16T00:11:58.328 DEBUG:teuthology.orchestra.remote:smithi145:/tmp/tmp.Gkm5ty6eiy is 56B 2020-01-16T00:11:58.338 INFO:teuthology.orchestra.run.smithi145:> rm -fr /tmp/tmp.Gkm5ty6eiy 2020-01-16T00:11:58.345 INFO:teuthology.orchestra.run.smithi145:> mktemp 2020-01-16T00:11:58.393 INFO:teuthology.orchestra.run.smithi145.stdout:/tmp/tmp.EUHsiDTchJ 2020-01-16T00:11:58.394 INFO:teuthology.orchestra.run.smithi145:> sudo cp /var/lib/ceph/osd/ceph-6/keyring /tmp/tmp.EUHsiDTchJ 2020-01-16T00:11:58.452 INFO:teuthology.orchestra.run.smithi145:> sudo chmod 0666 /tmp/tmp.EUHsiDTchJ 2020-01-16T00:11:58.578 DEBUG:teuthology.orchestra.remote:smithi145:/tmp/tmp.EUHsiDTchJ is 56B 2020-01-16T00:11:58.629 INFO:teuthology.orchestra.run.smithi145:> rm -fr /tmp/tmp.EUHsiDTchJ 2020-01-16T00:11:58.635 INFO:teuthology.orchestra.run.smithi145:> mktemp 2020-01-16T00:11:58.682 INFO:teuthology.orchestra.run.smithi145.stdout:/tmp/tmp.FhKrBrxXaT 2020-01-16T00:11:58.682 INFO:teuthology.orchestra.run.smithi145:> sudo cp /var/lib/ceph/osd/ceph-7/keyring /tmp/tmp.FhKrBrxXaT 2020-01-16T00:11:58.739 INFO:teuthology.orchestra.run.smithi145:> sudo chmod 0666 /tmp/tmp.FhKrBrxXaT 2020-01-16T00:11:58.848 DEBUG:teuthology.orchestra.remote:smithi145:/tmp/tmp.FhKrBrxXaT is 56B 2020-01-16T00:11:58.858 INFO:teuthology.orchestra.run.smithi145:> rm -fr /tmp/tmp.FhKrBrxXaT 2020-01-16T00:11:58.874 DEBUG:teuthology.orchestra.remote:smithi123:/etc/ceph/ceph.client.0.keyring is 59B 2020-01-16T00:11:58.937 DEBUG:teuthology.orchestra.remote:smithi145:/etc/ceph/ceph.client.1.keyring is 59B 2020-01-16T00:11:58.950 INFO:tasks.ceph:Adding keys to all mons... 2020-01-16T00:11:58.950 INFO:teuthology.orchestra.run.smithi123:> sudo tee -a /etc/ceph/ceph.keyring 2020-01-16T00:11:58.952 INFO:teuthology.orchestra.run.smithi145:> sudo tee -a /etc/ceph/ceph.keyring 2020-01-16T00:11:58.996 INFO:teuthology.orchestra.run.smithi145.stdout:[mgr.y] 2020-01-16T00:11:58.996 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQAyqh9eu5uJExAA6IuXpZkcIe2NceT8R7UjBA== 2020-01-16T00:11:58.997 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.0] 2020-01-16T00:11:58.997 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQA1qh9e/OozDxAAdSOVPo5PstdLu2qGQ0GgVA== 2020-01-16T00:11:58.997 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.1] 2020-01-16T00:11:58.997 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQA3qh9emlPuMBAAHqYNE6ZZxsRiFTVhpGUaAw== 2020-01-16T00:11:58.997 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.2] 2020-01-16T00:11:58.997 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQA6qh9eKR9lGBAA1dpOVoToSJZ6C4aRfuKITg== 2020-01-16T00:11:58.997 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.3] 2020-01-16T00:11:58.998 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQA9qh9eAk8XABAAMdocMTX8D7iPXAf5VobQfA== 2020-01-16T00:11:58.998 INFO:teuthology.orchestra.run.smithi145.stdout:[mgr.x] 2020-01-16T00:11:58.998 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQAyqh9e8yFnFhAANRNHvWP5Mzfo6bbFrM+p0w== 2020-01-16T00:11:58.998 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.4] 2020-01-16T00:11:58.999 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQBCqh9ep5MUDhAAA44oxmRX+jrZjXYaxKK1cA== 2020-01-16T00:11:58.999 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.5] 2020-01-16T00:11:58.999 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQBEqh9eiaE8MBAA5y4WdOZAs9eBXb8wzz4hqA== 2020-01-16T00:11:58.999 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.6] 2020-01-16T00:11:58.999 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQBHqh9eg/xQFxAAB3sEDzI7MtdYPx+mcapiDQ== 2020-01-16T00:11:58.999 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.7] 2020-01-16T00:11:58.999 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQBJqh9eKu10ORAAUvhq3HEocTbG9P6m6KPcvA== 2020-01-16T00:11:59.000 INFO:teuthology.orchestra.run.smithi145.stdout:[client.0] 2020-01-16T00:11:59.000 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQAyqh9eXcCUGBAACcEBQ7xm3gj9DqVwqT+hNQ== 2020-01-16T00:11:59.000 INFO:teuthology.orchestra.run.smithi145.stdout:[client.1] 2020-01-16T00:11:59.000 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQAyqh9eMR1IGxAApWCMcUgRrbhdmt2O91xlAA== 2020-01-16T00:11:59.001 INFO:teuthology.orchestra.run.smithi123.stdout:[mgr.y] 2020-01-16T00:11:59.001 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAyqh9eu5uJExAA6IuXpZkcIe2NceT8R7UjBA== 2020-01-16T00:11:59.002 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.0] 2020-01-16T00:11:59.002 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQA1qh9e/OozDxAAdSOVPo5PstdLu2qGQ0GgVA== 2020-01-16T00:11:59.002 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.1] 2020-01-16T00:11:59.002 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQA3qh9emlPuMBAAHqYNE6ZZxsRiFTVhpGUaAw== 2020-01-16T00:11:59.002 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.2] 2020-01-16T00:11:59.002 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQA6qh9eKR9lGBAA1dpOVoToSJZ6C4aRfuKITg== 2020-01-16T00:11:59.003 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.3] 2020-01-16T00:11:59.003 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQA9qh9eAk8XABAAMdocMTX8D7iPXAf5VobQfA== 2020-01-16T00:11:59.003 INFO:teuthology.orchestra.run.smithi123.stdout:[mgr.x] 2020-01-16T00:11:59.003 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAyqh9e8yFnFhAANRNHvWP5Mzfo6bbFrM+p0w== 2020-01-16T00:11:59.003 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.4] 2020-01-16T00:11:59.003 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQBCqh9ep5MUDhAAA44oxmRX+jrZjXYaxKK1cA== 2020-01-16T00:11:59.003 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.5] 2020-01-16T00:11:59.003 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQBEqh9eiaE8MBAA5y4WdOZAs9eBXb8wzz4hqA== 2020-01-16T00:11:59.004 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.6] 2020-01-16T00:11:59.004 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQBHqh9eg/xQFxAAB3sEDzI7MtdYPx+mcapiDQ== 2020-01-16T00:11:59.004 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.7] 2020-01-16T00:11:59.004 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQBJqh9eKu10ORAAUvhq3HEocTbG9P6m6KPcvA== 2020-01-16T00:11:59.004 INFO:teuthology.orchestra.run.smithi123.stdout:[client.0] 2020-01-16T00:11:59.004 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAyqh9eXcCUGBAACcEBQ7xm3gj9DqVwqT+hNQ== 2020-01-16T00:11:59.004 INFO:teuthology.orchestra.run.smithi123.stdout:[client.1] 2020-01-16T00:11:59.004 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAyqh9eMR1IGxAApWCMcUgRrbhdmt2O91xlAA== 2020-01-16T00:11:59.005 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2020-01-16T00:11:59.046 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2020-01-16T00:11:59.088 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-01-16T00:11:59.131 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-01-16T00:11:59.312 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-01-16T00:11:59.314 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-01-16T00:11:59.354 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-01-16T00:11:59.398 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-01-16T00:11:59.440 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-01-16T00:11:59.483 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-01-16T00:11:59.525 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2020-01-16T00:11:59.566 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2020-01-16T00:11:59.609 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-01-16T00:11:59.655 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-01-16T00:11:59.697 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-01-16T00:11:59.738 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-01-16T00:11:59.781 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-01-16T00:11:59.823 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-01-16T00:11:59.864 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-01-16T00:11:59.906 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-01-16T00:11:59.948 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mds allow --cap osd 'allow rwx' --cap mgr 'allow r' --cap mon 'allow rw' 2020-01-16T00:11:59.993 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mds allow --cap osd 'allow rwx' --cap mgr 'allow r' --cap mon 'allow rw' 2020-01-16T00:12:00.038 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mds allow --cap osd 'allow rwx' --cap mgr 'allow r' --cap mon 'allow rw' 2020-01-16T00:12:00.079 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mds allow --cap osd 'allow rwx' --cap mgr 'allow r' --cap mon 'allow rw' 2020-01-16T00:12:00.124 INFO:tasks.ceph:Running mkfs on mon nodes... 2020-01-16T00:12:00.124 INFO:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2020-01-16T00:12:00.139 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2020-01-16T00:12:00.253 INFO:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2020-01-16T00:12:00.268 INFO:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2020-01-16T00:12:00.324 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2020-01-16T00:12:00.437 INFO:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2020-01-16T00:12:00.453 INFO:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2020-01-16T00:12:00.468 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2020-01-16T00:12:00.584 INFO:teuthology.orchestra.run.smithi145:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2020-01-16T00:12:00.599 INFO:teuthology.orchestra.run.smithi123:> rm -- /home/ubuntu/cephtest/ceph.monmap 2020-01-16T00:12:00.603 INFO:teuthology.orchestra.run.smithi145:> rm -- /home/ubuntu/cephtest/ceph.monmap 2020-01-16T00:12:00.646 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2020-01-16T00:12:00.646 INFO:tasks.ceph.mon.a:Restarting daemon 2020-01-16T00:12:00.647 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2020-01-16T00:12:00.650 INFO:tasks.ceph.mon.a:Started 2020-01-16T00:12:00.650 INFO:tasks.ceph.mon.c:Restarting daemon 2020-01-16T00:12:00.650 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2020-01-16T00:12:00.652 INFO:tasks.ceph.mon.c:Started 2020-01-16T00:12:00.653 INFO:tasks.ceph.mon.b:Restarting daemon 2020-01-16T00:12:00.653 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2020-01-16T00:12:00.689 INFO:tasks.ceph.mon.b:Started 2020-01-16T00:12:00.690 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2020-01-16T00:12:00.690 INFO:tasks.ceph.mgr.y:Restarting daemon 2020-01-16T00:12:00.690 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2020-01-16T00:12:00.692 INFO:tasks.ceph.mgr.y:Started 2020-01-16T00:12:00.692 INFO:tasks.ceph.mgr.x:Restarting daemon 2020-01-16T00:12:00.693 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2020-01-16T00:12:00.694 INFO:tasks.ceph.mgr.x:Started 2020-01-16T00:12:00.695 INFO:tasks.ceph:Setting crush tunables to default 2020-01-16T00:12:00.695 INFO:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph osd crush tunables default 2020-01-16T00:12:06.263 INFO:teuthology.orchestra.run.smithi123.stderr:adjusted tunables profile to default 2020-01-16T00:12:06.280 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2020-01-16T00:12:06.280 INFO:teuthology.orchestra.run.smithi123:> mktemp 2020-01-16T00:12:06.284 INFO:teuthology.orchestra.run.smithi123.stdout:/tmp/tmp.y4OzBlkVrQ 2020-01-16T00:12:06.284 INFO:teuthology.orchestra.run.smithi123:> sudo cp /var/lib/ceph/osd/ceph-0/fsid /tmp/tmp.y4OzBlkVrQ 2020-01-16T00:12:06.340 INFO:teuthology.orchestra.run.smithi123:> sudo chmod 0666 /tmp/tmp.y4OzBlkVrQ 2020-01-16T00:12:06.449 DEBUG:teuthology.orchestra.remote:smithi123:/tmp/tmp.y4OzBlkVrQ is 37B 2020-01-16T00:12:06.459 INFO:teuthology.orchestra.run.smithi123:> rm -fr /tmp/tmp.y4OzBlkVrQ 2020-01-16T00:12:06.464 INFO:teuthology.orchestra.run.smithi123:> mktemp 2020-01-16T00:12:06.510 INFO:teuthology.orchestra.run.smithi123.stdout:/tmp/tmp.ans2HAY2Zn 2020-01-16T00:12:06.511 INFO:teuthology.orchestra.run.smithi123:> sudo cp /var/lib/ceph/osd/ceph-1/fsid /tmp/tmp.ans2HAY2Zn 2020-01-16T00:12:06.567 INFO:teuthology.orchestra.run.smithi123:> sudo chmod 0666 /tmp/tmp.ans2HAY2Zn 2020-01-16T00:12:06.672 DEBUG:teuthology.orchestra.remote:smithi123:/tmp/tmp.ans2HAY2Zn is 37B 2020-01-16T00:12:06.683 INFO:teuthology.orchestra.run.smithi123:> rm -fr /tmp/tmp.ans2HAY2Zn 2020-01-16T00:12:06.689 INFO:teuthology.orchestra.run.smithi123:> mktemp 2020-01-16T00:12:06.738 INFO:teuthology.orchestra.run.smithi123.stdout:/tmp/tmp.c4ozCy1ekI 2020-01-16T00:12:06.739 INFO:teuthology.orchestra.run.smithi123:> sudo cp /var/lib/ceph/osd/ceph-2/fsid /tmp/tmp.c4ozCy1ekI 2020-01-16T00:12:06.796 INFO:teuthology.orchestra.run.smithi123:> sudo chmod 0666 /tmp/tmp.c4ozCy1ekI 2020-01-16T00:12:06.895 DEBUG:teuthology.orchestra.remote:smithi123:/tmp/tmp.c4ozCy1ekI is 37B 2020-01-16T00:12:06.904 INFO:teuthology.orchestra.run.smithi123:> rm -fr /tmp/tmp.c4ozCy1ekI 2020-01-16T00:12:06.909 INFO:teuthology.orchestra.run.smithi123:> mktemp 2020-01-16T00:12:06.958 INFO:teuthology.orchestra.run.smithi123.stdout:/tmp/tmp.qFIv8XspVJ 2020-01-16T00:12:06.958 INFO:teuthology.orchestra.run.smithi123:> sudo cp /var/lib/ceph/osd/ceph-3/fsid /tmp/tmp.qFIv8XspVJ 2020-01-16T00:12:07.018 INFO:teuthology.orchestra.run.smithi123:> sudo chmod 0666 /tmp/tmp.qFIv8XspVJ 2020-01-16T00:12:07.123 DEBUG:teuthology.orchestra.remote:smithi123:/tmp/tmp.qFIv8XspVJ is 37B 2020-01-16T00:12:07.133 INFO:teuthology.orchestra.run.smithi123:> rm -fr /tmp/tmp.qFIv8XspVJ 2020-01-16T00:12:07.139 INFO:teuthology.orchestra.run.smithi145:> mktemp 2020-01-16T00:12:07.145 INFO:teuthology.orchestra.run.smithi145.stdout:/tmp/tmp.3qVrYxIl5S 2020-01-16T00:12:07.145 INFO:teuthology.orchestra.run.smithi145:> sudo cp /var/lib/ceph/osd/ceph-4/fsid /tmp/tmp.3qVrYxIl5S 2020-01-16T00:12:07.202 INFO:teuthology.orchestra.run.smithi145:> sudo chmod 0666 /tmp/tmp.3qVrYxIl5S 2020-01-16T00:12:07.307 DEBUG:teuthology.orchestra.remote:smithi145:/tmp/tmp.3qVrYxIl5S is 37B 2020-01-16T00:12:07.316 INFO:teuthology.orchestra.run.smithi145:> rm -fr /tmp/tmp.3qVrYxIl5S 2020-01-16T00:12:07.327 INFO:teuthology.orchestra.run.smithi145:> mktemp 2020-01-16T00:12:07.373 INFO:teuthology.orchestra.run.smithi145.stdout:/tmp/tmp.vfbfiB2F2X 2020-01-16T00:12:07.374 INFO:teuthology.orchestra.run.smithi145:> sudo cp /var/lib/ceph/osd/ceph-5/fsid /tmp/tmp.vfbfiB2F2X 2020-01-16T00:12:07.430 INFO:teuthology.orchestra.run.smithi145:> sudo chmod 0666 /tmp/tmp.vfbfiB2F2X 2020-01-16T00:12:07.535 DEBUG:teuthology.orchestra.remote:smithi145:/tmp/tmp.vfbfiB2F2X is 37B 2020-01-16T00:12:07.548 INFO:teuthology.orchestra.run.smithi145:> rm -fr /tmp/tmp.vfbfiB2F2X 2020-01-16T00:12:07.553 INFO:teuthology.orchestra.run.smithi145:> mktemp 2020-01-16T00:12:07.601 INFO:teuthology.orchestra.run.smithi145.stdout:/tmp/tmp.kSKCHLXmRU 2020-01-16T00:12:07.602 INFO:teuthology.orchestra.run.smithi145:> sudo cp /var/lib/ceph/osd/ceph-6/fsid /tmp/tmp.kSKCHLXmRU 2020-01-16T00:12:07.658 INFO:teuthology.orchestra.run.smithi145:> sudo chmod 0666 /tmp/tmp.kSKCHLXmRU 2020-01-16T00:12:07.763 DEBUG:teuthology.orchestra.remote:smithi145:/tmp/tmp.kSKCHLXmRU is 37B 2020-01-16T00:12:07.772 INFO:teuthology.orchestra.run.smithi145:> rm -fr /tmp/tmp.kSKCHLXmRU 2020-01-16T00:12:07.778 INFO:teuthology.orchestra.run.smithi145:> mktemp 2020-01-16T00:12:07.827 INFO:teuthology.orchestra.run.smithi145.stdout:/tmp/tmp.v7uMpWh0qu 2020-01-16T00:12:07.827 INFO:teuthology.orchestra.run.smithi145:> sudo cp /var/lib/ceph/osd/ceph-7/fsid /tmp/tmp.v7uMpWh0qu 2020-01-16T00:12:07.883 INFO:teuthology.orchestra.run.smithi145:> sudo chmod 0666 /tmp/tmp.v7uMpWh0qu 2020-01-16T00:12:07.987 DEBUG:teuthology.orchestra.remote:smithi145:/tmp/tmp.v7uMpWh0qu is 37B 2020-01-16T00:12:07.997 INFO:teuthology.orchestra.run.smithi145:> rm -fr /tmp/tmp.v7uMpWh0qu 2020-01-16T00:12:08.003 INFO:teuthology.orchestra.run.smithi145:> sudo ceph --cluster ceph osd new 4b8d1e40-95e3-4adc-9929-cb7e6cc8b766 0 2020-01-16T00:12:08.254 INFO:teuthology.orchestra.run.smithi145.stdout:0 2020-01-16T00:12:08.267 INFO:teuthology.orchestra.run.smithi145:> sudo ceph --cluster ceph osd new 2e373d15-6348-4caa-aa9f-b46037376a3b 1 2020-01-16T00:12:08.469 INFO:teuthology.orchestra.run.smithi145.stdout:1 2020-01-16T00:12:08.485 INFO:teuthology.orchestra.run.smithi145:> sudo ceph --cluster ceph osd new 5736ff2c-076a-4d41-8e11-06e2fd1f5f94 2 2020-01-16T00:12:08.699 INFO:teuthology.orchestra.run.smithi145.stdout:2 2020-01-16T00:12:08.713 INFO:teuthology.orchestra.run.smithi145:> sudo ceph --cluster ceph osd new 554f53a8-c5b1-43c9-a7d6-93629d84f1ea 3 2020-01-16T00:12:08.929 INFO:teuthology.orchestra.run.smithi145.stdout:3 2020-01-16T00:12:08.941 INFO:teuthology.orchestra.run.smithi145:> sudo ceph --cluster ceph osd new 77945419-c04b-4071-ab87-256d77640dc0 4 2020-01-16T00:12:09.143 INFO:teuthology.orchestra.run.smithi145.stdout:4 2020-01-16T00:12:09.154 INFO:teuthology.orchestra.run.smithi145:> sudo ceph --cluster ceph osd new d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4 5 2020-01-16T00:12:09.354 INFO:teuthology.orchestra.run.smithi145.stdout:5 2020-01-16T00:12:09.366 INFO:teuthology.orchestra.run.smithi145:> sudo ceph --cluster ceph osd new e9d342cd-5c2d-40a0-b8ea-24588f280389 6 2020-01-16T00:12:09.571 INFO:teuthology.orchestra.run.smithi145.stdout:6 2020-01-16T00:12:09.586 INFO:teuthology.orchestra.run.smithi145:> sudo ceph --cluster ceph osd new 6a301b66-384a-40d8-adc8-96fff49e982e 7 2020-01-16T00:12:09.789 INFO:teuthology.orchestra.run.smithi145.stdout:7 2020-01-16T00:12:09.801 INFO:tasks.ceph.osd.0:Restarting daemon 2020-01-16T00:12:09.802 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2020-01-16T00:12:09.805 INFO:tasks.ceph.osd.0:Started 2020-01-16T00:12:09.805 INFO:tasks.ceph.osd.1:Restarting daemon 2020-01-16T00:12:09.805 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2020-01-16T00:12:09.808 INFO:tasks.ceph.osd.1:Started 2020-01-16T00:12:09.808 INFO:tasks.ceph.osd.2:Restarting daemon 2020-01-16T00:12:09.809 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2020-01-16T00:12:09.811 INFO:tasks.ceph.osd.2:Started 2020-01-16T00:12:09.811 INFO:tasks.ceph.osd.3:Restarting daemon 2020-01-16T00:12:09.811 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2020-01-16T00:12:09.814 INFO:tasks.ceph.osd.3:Started 2020-01-16T00:12:09.815 INFO:tasks.ceph.osd.4:Restarting daemon 2020-01-16T00:12:09.815 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2020-01-16T00:12:09.827 INFO:tasks.ceph.osd.4:Started 2020-01-16T00:12:09.828 INFO:tasks.ceph.osd.5:Restarting daemon 2020-01-16T00:12:09.828 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2020-01-16T00:12:09.834 INFO:tasks.ceph.osd.5:Started 2020-01-16T00:12:09.834 INFO:tasks.ceph.osd.6:Restarting daemon 2020-01-16T00:12:09.836 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2020-01-16T00:12:09.844 INFO:tasks.ceph.osd.6:Started 2020-01-16T00:12:09.844 INFO:tasks.ceph.osd.7:Restarting daemon 2020-01-16T00:12:09.844 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2020-01-16T00:12:09.847 INFO:tasks.ceph.osd.7:Started 2020-01-16T00:12:09.847 INFO:tasks.ceph:Waiting for OSDs to come up 2020-01-16T00:12:10.170 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:12:10.169+0000 7f4bc7136d80 -1 Falling back to public interface 2020-01-16T00:12:10.191 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:10.191+0000 7f6d02046d80 -1 Falling back to public interface 2020-01-16T00:12:10.411 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:12:10.411+0000 7fba4bc11d80 -1 Falling back to public interface 2020-01-16T00:12:10.427 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:12:10.427+0000 7f469a55fd80 -1 Falling back to public interface 2020-01-16T00:12:10.450 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:10.449+0000 7f61efa2ad80 -1 Falling back to public interface 2020-01-16T00:12:10.450 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:12:10.451+0000 7fd8c23f5d80 -1 Falling back to public interface 2020-01-16T00:12:10.461 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:12:10.461+0000 7f9859febd80 -1 Falling back to public interface 2020-01-16T00:12:10.473 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:10.473+0000 7f7d01992d80 -1 Falling back to public interface 2020-01-16T00:12:10.659 INFO:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2020-01-16T00:12:10.845 INFO:teuthology.misc.health.smithi123.stdout: 2020-01-16T00:12:10.845 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":10,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:12:09.785449+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:12:10.855 DEBUG:teuthology.misc:0 of 8 OSDs are up 2020-01-16T00:12:11.343 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:12:11.341+0000 7f4bc7136d80 -1 osd.4 0 log_to_monitors {default=true} 2020-01-16T00:12:11.387 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:11.387+0000 7f6d02046d80 -1 osd.2 0 log_to_monitors {default=true} 2020-01-16T00:12:11.570 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:12:11.571+0000 7fba4bc11d80 -1 osd.1 0 log_to_monitors {default=true} 2020-01-16T00:12:11.587 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:12:11.587+0000 7f469a55fd80 -1 osd.0 0 log_to_monitors {default=true} 2020-01-16T00:12:11.589 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:11.589+0000 7f61efa2ad80 -1 osd.5 0 log_to_monitors {default=true} 2020-01-16T00:12:11.617 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:12:11.617+0000 7f9859febd80 -1 osd.7 0 log_to_monitors {default=true} 2020-01-16T00:12:11.627 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:12:11.627+0000 7fd8c23f5d80 -1 osd.3 0 log_to_monitors {default=true} 2020-01-16T00:12:11.675 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:11.673+0000 7f7d01992d80 -1 osd.6 0 log_to_monitors {default=true} 2020-01-16T00:12:12.963 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:12:12.961+0000 7f983f22f700 -1 osd.7 0 waiting for initial osdmap 2020-01-16T00:12:12.963 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:12.961+0000 7f61d4c6e700 -1 osd.5 0 waiting for initial osdmap 2020-01-16T00:12:12.964 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:12:12.963+0000 7fba30e55700 -1 osd.1 0 waiting for initial osdmap 2020-01-16T00:12:12.964 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:12.963+0000 7f6ce728a700 -1 osd.2 0 waiting for initial osdmap 2020-01-16T00:12:12.964 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:12:12.963+0000 7fd8a7639700 -1 osd.3 0 waiting for initial osdmap 2020-01-16T00:12:12.965 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:12:12.965+0000 7f4bac37a700 -1 osd.4 0 waiting for initial osdmap 2020-01-16T00:12:12.965 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:12.965+0000 7f7ce6bd6700 -1 osd.6 0 waiting for initial osdmap 2020-01-16T00:12:12.966 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:12:12.967+0000 7f467f7a3700 -1 osd.0 0 waiting for initial osdmap 2020-01-16T00:12:17.662 INFO:teuthology.orchestra.run.smithi123:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2020-01-16T00:12:17.902 INFO:teuthology.misc.health.smithi123.stdout: 2020-01-16T00:12:17.902 INFO:teuthology.misc.health.smithi123.stdout:{"epoch":14,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:12:14.980690+0000","last_up_change":"2020-01-16T00:12:13.968282+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists","up"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:12:12.621038+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:12:12.564547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:12:12.383046+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:12:12.619027+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:12:12.389288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:12:12.614969+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:12:12.704423+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:12:12.656476+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:12:17.922 DEBUG:teuthology.misc:8 of 8 OSDs are up 2020-01-16T00:12:17.922 INFO:tasks.ceph:Creating RBD pool 2020-01-16T00:12:17.922 INFO:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph osd pool create rbd 8 2020-01-16T00:12:19.019 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'rbd' created 2020-01-16T00:12:19.031 INFO:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2020-01-16T00:12:20.031 INFO:teuthology.orchestra.run.smithi123.stderr:enabled application 'rbd' on pool 'rbd' 2020-01-16T00:12:20.042 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2020-01-16T00:12:20.043 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:12:20.043 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2020-01-16T00:12:20.283 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:12:20.283 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":16,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:12:20.011771+0000","last_up_change":"2020-01-16T00:12:13.968282+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists","up"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:12:12.621038+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:12:12.564547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:12:12.383046+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:12:12.619027+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:12:12.389288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:12:12.614969+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:12:12.704423+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:12:12.656476+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:12:20.296 INFO:tasks.ceph.ceph_manager.ceph:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool,creating', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2020-01-16T00:12:18.200931+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 3, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 2, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 32769, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'16', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2020-01-16T00:12:20.296 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-01-16T00:12:20.546 INFO:teuthology.orchestra.run.smithi123.stdout:pg_num: 8 2020-01-16T00:12:20.563 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2020-01-16T00:12:20.564 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2020-01-16T00:12:20.564 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2020-01-16T00:12:20.836 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:12:20.850 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":4,"active_gid":4109,"active_name":"y","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6800","nonce":11747}]},"active_addr":"172.21.15.123:6800/11747","active_change":"2020-01-16T00:12:09.941947+0000","active_mgr_features":4540138292836696063,"available":true,"standbys":[{"gid":4115,"name":"x","mgr_features":4540138292836696063,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"ansible","can_run":true,"error_string":"","module_options":{"ca_bundle":{"name":"ca_bundle","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_location":{"name":"server_location","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_server":{"name":"verify_server","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"inventory_cache_timeout":{"name":"inventory_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"service_cache_timeout":{"name":"service_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"str","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"str","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"str","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"deepsea","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_eauth":{"name":"salt_api_eauth","type":"str","level":"advanced","flags":0,"default_value":"sharedsecret","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_password":{"name":"salt_api_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_url":{"name":"salt_api_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_username":{"name":"salt_api_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator_cli","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["ansible","cephadm","deepsea","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"ansible","can_run":true,"error_string":"","module_options":{"ca_bundle":{"name":"ca_bundle","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_location":{"name":"server_location","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_server":{"name":"verify_server","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"inventory_cache_timeout":{"name":"inventory_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"service_cache_timeout":{"name":"service_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"str","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"str","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"str","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"deepsea","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_eauth":{"name":"salt_api_eauth","type":"str","level":"advanced","flags":0,"default_value":"sharedsecret","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_password":{"name":"salt_api_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_url":{"name":"salt_api_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_username":{"name":"salt_api_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator_cli","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["ansible","cephadm","deepsea","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator_cli","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.123:0","nonce":1569820937}]}]}} 2020-01-16T00:12:20.861 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2020-01-16T00:12:20.861 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2020-01-16T00:12:20.862 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:12:21.109 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:12:21.109 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":17,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:12:21.029381+0000","last_up_change":"2020-01-16T00:12:13.968282+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists","up"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:12:12.621038+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:12:12.564547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:12:12.383046+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:12:12.619027+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:12:12.389288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:12:12.614969+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:12:12.704423+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:12:12.656476+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:12:21.129 INFO:tasks.ceph.ceph_manager.ceph:all up! 2020-01-16T00:12:21.129 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:12:21.365 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:12:21.365 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":17,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:12:21.029381+0000","last_up_change":"2020-01-16T00:12:13.968282+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists","up"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:12:12.621038+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:12:12.564547+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:12:12.383046+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:12:12.619027+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:12:12.389288+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:12:12.614969+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:12:12.704423+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:12:12.656476+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:12:21.378 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2020-01-16T00:12:21.529 INFO:teuthology.orchestra.run.smithi123.stdout:55834574851 2020-01-16T00:12:21.530 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2020-01-16T00:12:21.686 INFO:teuthology.orchestra.run.smithi123.stdout:55834574851 2020-01-16T00:12:21.687 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2020-01-16T00:12:21.841 INFO:teuthology.orchestra.run.smithi123.stdout:55834574851 2020-01-16T00:12:21.842 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2020-01-16T00:12:21.995 INFO:teuthology.orchestra.run.smithi123.stdout:55834574851 2020-01-16T00:12:21.996 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2020-01-16T00:12:22.160 INFO:teuthology.orchestra.run.smithi123.stdout:55834574852 2020-01-16T00:12:22.160 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2020-01-16T00:12:22.325 INFO:teuthology.orchestra.run.smithi123.stdout:55834574851 2020-01-16T00:12:22.325 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2020-01-16T00:12:22.493 INFO:teuthology.orchestra.run.smithi123.stdout:55834574852 2020-01-16T00:12:22.493 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2020-01-16T00:12:22.653 INFO:teuthology.orchestra.run.smithi123.stdout:55834574851 2020-01-16T00:12:22.653 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2020-01-16T00:12:22.898 INFO:teuthology.orchestra.run.smithi123.stdout:55834574851 2020-01-16T00:12:22.911 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.0 2020-01-16T00:12:22.912 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2020-01-16T00:12:23.157 INFO:teuthology.orchestra.run.smithi123.stdout:55834574851 2020-01-16T00:12:23.175 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.1 2020-01-16T00:12:23.175 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2020-01-16T00:12:23.420 INFO:teuthology.orchestra.run.smithi123.stdout:55834574851 2020-01-16T00:12:23.433 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.2 2020-01-16T00:12:23.433 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2020-01-16T00:12:23.669 INFO:teuthology.orchestra.run.smithi123.stdout:55834574850 2020-01-16T00:12:23.689 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.3 2020-01-16T00:12:24.690 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2020-01-16T00:12:24.946 INFO:teuthology.orchestra.run.smithi123.stdout:55834574851 2020-01-16T00:12:24.965 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.3 2020-01-16T00:12:24.965 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2020-01-16T00:12:25.207 INFO:teuthology.orchestra.run.smithi123.stdout:55834574852 2020-01-16T00:12:25.220 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.4 2020-01-16T00:12:25.221 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2020-01-16T00:12:25.475 INFO:teuthology.orchestra.run.smithi123.stdout:55834574851 2020-01-16T00:12:25.488 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.5 2020-01-16T00:12:25.489 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2020-01-16T00:12:25.736 INFO:teuthology.orchestra.run.smithi123.stdout:55834574852 2020-01-16T00:12:25.749 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.6 2020-01-16T00:12:25.749 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2020-01-16T00:12:25.993 INFO:teuthology.orchestra.run.smithi123.stdout:55834574851 2020-01-16T00:12:26.006 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.7 2020-01-16T00:12:26.006 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2020-01-16T00:12:26.006 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-01-16T00:12:26.242 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:12:26.244 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2020-01-16T00:12:26.255 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2020-01-16T00:12:25.953338+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":24,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8389824,"kb_used_data":1152,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746584896,"statfs":{"total":773094113280,"available":764502933504,"internally_reserved":8589934592,"allocated":1179648,"data_stored":350752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.034899+0000","last_change":"2020-01-16T00:12:20.034125+0000","last_active":"2020-01-16T00:12:20.034899+0000","last_peered":"2020-01-16T00:12:20.034899+0000","last_clean":"2020-01-16T00:12:20.034899+0000","last_became_active":"2020-01-16T00:12:20.033645+0000","last_became_peered":"2020-01-16T00:12:20.033645+0000","last_unstale":"2020-01-16T00:12:20.034899+0000","last_undegraded":"2020-01-16T00:12:20.034899+0000","last_fullsized":"2020-01-16T00:12:20.034899+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.033656+0000","last_change":"2020-01-16T00:12:20.033023+0000","last_active":"2020-01-16T00:12:20.033656+0000","last_peered":"2020-01-16T00:12:20.033656+0000","last_clean":"2020-01-16T00:12:20.033656+0000","last_became_active":"2020-01-16T00:12:20.031576+0000","last_became_peered":"2020-01-16T00:12:20.031576+0000","last_unstale":"2020-01-16T00:12:20.033656+0000","last_undegraded":"2020-01-16T00:12:20.033656+0000","last_fullsized":"2020-01-16T00:12:20.033656+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.031703+0000","last_change":"2020-01-16T00:12:20.031201+0000","last_active":"2020-01-16T00:12:20.031703+0000","last_peered":"2020-01-16T00:12:20.031703+0000","last_clean":"2020-01-16T00:12:20.031703+0000","last_became_active":"2020-01-16T00:12:20.030752+0000","last_became_peered":"2020-01-16T00:12:20.030752+0000","last_unstale":"2020-01-16T00:12:20.031703+0000","last_undegraded":"2020-01-16T00:12:20.031703+0000","last_fullsized":"2020-01-16T00:12:20.031703+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.034773+0000","last_change":"2020-01-16T00:12:20.033205+0000","last_active":"2020-01-16T00:12:20.034773+0000","last_peered":"2020-01-16T00:12:20.034773+0000","last_clean":"2020-01-16T00:12:20.034773+0000","last_became_active":"2020-01-16T00:12:20.030792+0000","last_became_peered":"2020-01-16T00:12:20.030792+0000","last_unstale":"2020-01-16T00:12:20.034773+0000","last_undegraded":"2020-01-16T00:12:20.034773+0000","last_fullsized":"2020-01-16T00:12:20.034773+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.033641+0000","last_change":"2020-01-16T00:12:20.032658+0000","last_active":"2020-01-16T00:12:20.033641+0000","last_peered":"2020-01-16T00:12:20.033641+0000","last_clean":"2020-01-16T00:12:20.033641+0000","last_became_active":"2020-01-16T00:12:20.030794+0000","last_became_peered":"2020-01-16T00:12:20.030794+0000","last_unstale":"2020-01-16T00:12:20.033641+0000","last_undegraded":"2020-01-16T00:12:20.033641+0000","last_fullsized":"2020-01-16T00:12:20.033641+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.032720+0000","last_change":"2020-01-16T00:12:20.032010+0000","last_active":"2020-01-16T00:12:20.032720+0000","last_peered":"2020-01-16T00:12:20.032720+0000","last_clean":"2020-01-16T00:12:20.032720+0000","last_became_active":"2020-01-16T00:12:20.031560+0000","last_became_peered":"2020-01-16T00:12:20.031560+0000","last_unstale":"2020-01-16T00:12:20.032720+0000","last_undegraded":"2020-01-16T00:12:20.032720+0000","last_fullsized":"2020-01-16T00:12:20.032720+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.033296+0000","last_change":"2020-01-16T00:12:20.032733+0000","last_active":"2020-01-16T00:12:20.033296+0000","last_peered":"2020-01-16T00:12:20.033296+0000","last_clean":"2020-01-16T00:12:20.033296+0000","last_became_active":"2020-01-16T00:12:20.032284+0000","last_became_peered":"2020-01-16T00:12:20.032284+0000","last_unstale":"2020-01-16T00:12:20.033296+0000","last_undegraded":"2020-01-16T00:12:20.033296+0000","last_fullsized":"2020-01-16T00:12:20.033296+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.032829+0000","last_change":"2020-01-16T00:12:20.031741+0000","last_active":"2020-01-16T00:12:20.032829+0000","last_peered":"2020-01-16T00:12:20.032829+0000","last_clean":"2020-01-16T00:12:20.032829+0000","last_became_active":"2020-01-16T00:12:20.030776+0000","last_became_peered":"2020-01-16T00:12:20.030776+0000","last_unstale":"2020-01-16T00:12:20.032829+0000","last_undegraded":"2020-01-16T00:12:20.032829+0000","last_fullsized":"2020-01-16T00:12:20.032829+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}]}} 2020-01-16T00:12:26.256 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-01-16T00:12:26.496 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:12:26.498 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2020-01-16T00:12:26.509 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2020-01-16T00:12:25.953338+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":24,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8389824,"kb_used_data":1152,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746584896,"statfs":{"total":773094113280,"available":764502933504,"internally_reserved":8589934592,"allocated":1179648,"data_stored":350752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.034899+0000","last_change":"2020-01-16T00:12:20.034125+0000","last_active":"2020-01-16T00:12:20.034899+0000","last_peered":"2020-01-16T00:12:20.034899+0000","last_clean":"2020-01-16T00:12:20.034899+0000","last_became_active":"2020-01-16T00:12:20.033645+0000","last_became_peered":"2020-01-16T00:12:20.033645+0000","last_unstale":"2020-01-16T00:12:20.034899+0000","last_undegraded":"2020-01-16T00:12:20.034899+0000","last_fullsized":"2020-01-16T00:12:20.034899+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.033656+0000","last_change":"2020-01-16T00:12:20.033023+0000","last_active":"2020-01-16T00:12:20.033656+0000","last_peered":"2020-01-16T00:12:20.033656+0000","last_clean":"2020-01-16T00:12:20.033656+0000","last_became_active":"2020-01-16T00:12:20.031576+0000","last_became_peered":"2020-01-16T00:12:20.031576+0000","last_unstale":"2020-01-16T00:12:20.033656+0000","last_undegraded":"2020-01-16T00:12:20.033656+0000","last_fullsized":"2020-01-16T00:12:20.033656+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.031703+0000","last_change":"2020-01-16T00:12:20.031201+0000","last_active":"2020-01-16T00:12:20.031703+0000","last_peered":"2020-01-16T00:12:20.031703+0000","last_clean":"2020-01-16T00:12:20.031703+0000","last_became_active":"2020-01-16T00:12:20.030752+0000","last_became_peered":"2020-01-16T00:12:20.030752+0000","last_unstale":"2020-01-16T00:12:20.031703+0000","last_undegraded":"2020-01-16T00:12:20.031703+0000","last_fullsized":"2020-01-16T00:12:20.031703+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.034773+0000","last_change":"2020-01-16T00:12:20.033205+0000","last_active":"2020-01-16T00:12:20.034773+0000","last_peered":"2020-01-16T00:12:20.034773+0000","last_clean":"2020-01-16T00:12:20.034773+0000","last_became_active":"2020-01-16T00:12:20.030792+0000","last_became_peered":"2020-01-16T00:12:20.030792+0000","last_unstale":"2020-01-16T00:12:20.034773+0000","last_undegraded":"2020-01-16T00:12:20.034773+0000","last_fullsized":"2020-01-16T00:12:20.034773+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.033641+0000","last_change":"2020-01-16T00:12:20.032658+0000","last_active":"2020-01-16T00:12:20.033641+0000","last_peered":"2020-01-16T00:12:20.033641+0000","last_clean":"2020-01-16T00:12:20.033641+0000","last_became_active":"2020-01-16T00:12:20.030794+0000","last_became_peered":"2020-01-16T00:12:20.030794+0000","last_unstale":"2020-01-16T00:12:20.033641+0000","last_undegraded":"2020-01-16T00:12:20.033641+0000","last_fullsized":"2020-01-16T00:12:20.033641+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.032720+0000","last_change":"2020-01-16T00:12:20.032010+0000","last_active":"2020-01-16T00:12:20.032720+0000","last_peered":"2020-01-16T00:12:20.032720+0000","last_clean":"2020-01-16T00:12:20.032720+0000","last_became_active":"2020-01-16T00:12:20.031560+0000","last_became_peered":"2020-01-16T00:12:20.031560+0000","last_unstale":"2020-01-16T00:12:20.032720+0000","last_undegraded":"2020-01-16T00:12:20.032720+0000","last_fullsized":"2020-01-16T00:12:20.032720+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.033296+0000","last_change":"2020-01-16T00:12:20.032733+0000","last_active":"2020-01-16T00:12:20.033296+0000","last_peered":"2020-01-16T00:12:20.033296+0000","last_clean":"2020-01-16T00:12:20.033296+0000","last_became_active":"2020-01-16T00:12:20.032284+0000","last_became_peered":"2020-01-16T00:12:20.032284+0000","last_unstale":"2020-01-16T00:12:20.033296+0000","last_undegraded":"2020-01-16T00:12:20.033296+0000","last_fullsized":"2020-01-16T00:12:20.033296+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.032829+0000","last_change":"2020-01-16T00:12:20.031741+0000","last_active":"2020-01-16T00:12:20.032829+0000","last_peered":"2020-01-16T00:12:20.032829+0000","last_clean":"2020-01-16T00:12:20.032829+0000","last_became_active":"2020-01-16T00:12:20.030776+0000","last_became_peered":"2020-01-16T00:12:20.030776+0000","last_unstale":"2020-01-16T00:12:20.032829+0000","last_undegraded":"2020-01-16T00:12:20.032829+0000","last_fullsized":"2020-01-16T00:12:20.032829+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}]}} 2020-01-16T00:12:26.510 INFO:tasks.ceph.ceph_manager.ceph:clean! 2020-01-16T00:12:26.510 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2020-01-16T00:12:26.510 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2020-01-16T00:12:26.511 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2020-01-16T00:12:26.785 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:12:26.798 INFO:teuthology.orchestra.run.smithi123.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2020-01-16T00:12:26.798 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2020-01-16T00:12:26.799 INFO:teuthology.run_tasks:Running task exec... 2020-01-16T00:12:26.810 INFO:teuthology.task.exec:Executing custom commands... 2020-01-16T00:12:26.810 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi123.front.sepia.ceph.com 2020-01-16T00:12:26.810 INFO:teuthology.orchestra.run.smithi123:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'while ! ceph balancer status ; do sleep 1 ; done' 2020-01-16T00:12:27.053 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:27.053 INFO:teuthology.orchestra.run.smithi123.stdout: "active": false, 2020-01-16T00:12:27.053 INFO:teuthology.orchestra.run.smithi123.stdout: "last_optimize_duration": "", 2020-01-16T00:12:27.054 INFO:teuthology.orchestra.run.smithi123.stdout: "last_optimize_started": "", 2020-01-16T00:12:27.054 INFO:teuthology.orchestra.run.smithi123.stdout: "mode": "none", 2020-01-16T00:12:27.054 INFO:teuthology.orchestra.run.smithi123.stdout: "optimize_result": "", 2020-01-16T00:12:27.054 INFO:teuthology.orchestra.run.smithi123.stdout: "plans": [] 2020-01-16T00:12:27.054 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:27.066 INFO:teuthology.orchestra.run.smithi123:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer mode crush-compat' 2020-01-16T00:12:27.795 INFO:teuthology.orchestra.run.smithi123:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer on' 2020-01-16T00:12:28.066 INFO:teuthology.run_tasks:Running task thrashosds... 2020-01-16T00:12:28.077 INFO:tasks.thrashosds:config is {'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'optrack_toggle_delay': 2.0, 'chance_test_map_discontinuity': 2, 'chance_pgpnum_fix': 0.25, 'timeout': 1800, 'chance_pgnum_grow': 0.25, 'sighup_delay': 0.1, 'chance_pgnum_shrink': 0.25} 2020-01-16T00:12:28.078 INFO:tasks.thrashosds:overrides is {'ceph': {'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow requests'], 'mon_bind_msgr2': False, 'conf': {'global': {'osd_object_clean_region_max_num_intervals': 1000, 'ms type': 'async', 'ms tcp read timeout': 5, 'osd_pool_default_size': 3, 'osd_pool_default_min_size': 2, 'osd_max_pg_log_entries': 2, 'ms bind msgr2': False, 'mon client directed command retry': 5, 'osd_async_recovery_min_cost': 1, 'ms inject socket failures': 2500, 'osd_min_pg_log_entries': 1}, 'mon': {'debug mon': 20, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'debug ms': 1, 'mon osdmap full prune min': 15, 'mon osdmap full prune interval': 2, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'mon osd full ratio': 0.9, 'osd op queue cut off': 'debug_random', 'mon osd backfillfull_ratio': 0.85, 'debug journal': 20, 'osd scrub max interval': 120, 'debug osd': 25, 'debug bluestore': 20, 'debug bluefs': 20, 'osd debug verify missing on start': True, 'debug ms': 1, 'debug rocksdb': 10, 'osd shutdown pgref assert': True, 'bdev async discard': True, 'bluestore fsck on mount': True, 'osd max backfills': 6, 'mon osd nearfull ratio': 0.8, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd debug verify cached snaps': True, 'osd map cache size': 1, 'osd op queue': 'debug_random', 'osd scrub during recovery': False, 'bluestore block size': 96636764160, 'osd scrub min interval': 60, 'debug filestore': 20, 'bdev enable discard': True, 'osd failsafe full ratio': 0.95}}, 'fs': 'xfs', 'sha1': 'cb3c85e74500283e01cc592f7e2858e323186910'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': 'cb3c85e74500283e01cc592f7e2858e323186910'}}, 'admin_socket': {'branch': 'wip-sage-testing-2020-01-15-1209'}, 'thrashosds': {'bdev_inject_crash_probability': 0.5, 'min_in': 4, 'bdev_inject_crash': 2}, 'workunit': {'sha1': 'cb3c85e74500283e01cc592f7e2858e323186910', 'branch': 'wip-sage-testing-2020-01-15-1209'}} 2020-01-16T00:12:28.078 INFO:tasks.thrashosds:config is {'dump_ops_enable': 'true', 'bdev_inject_crash': 2, 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash_probability': 0.5, 'optrack_toggle_delay': 2.0, 'chance_test_map_discontinuity': 2, 'chance_pgpnum_fix': 0.25, 'timeout': 1800, 'min_in': 4, 'chance_pgnum_grow': 0.25, 'sighup_delay': 0.1, 'chance_pgnum_shrink': 0.25} 2020-01-16T00:12:28.078 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2020-01-16T00:12:28.346 INFO:tasks.thrashosds:Beginning thrashosds... 2020-01-16T00:12:28.347 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2020-01-16T00:12:28.347 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-01-16T00:12:28.579 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:12:28.582 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2020-01-16T00:12:28.593 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2020-01-16T00:12:27.953785+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":24,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8389824,"kb_used_data":1152,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746584896,"statfs":{"total":773094113280,"available":764502933504,"internally_reserved":8589934592,"allocated":1179648,"data_stored":350752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.034899+0000","last_change":"2020-01-16T00:12:20.034125+0000","last_active":"2020-01-16T00:12:20.034899+0000","last_peered":"2020-01-16T00:12:20.034899+0000","last_clean":"2020-01-16T00:12:20.034899+0000","last_became_active":"2020-01-16T00:12:20.033645+0000","last_became_peered":"2020-01-16T00:12:20.033645+0000","last_unstale":"2020-01-16T00:12:20.034899+0000","last_undegraded":"2020-01-16T00:12:20.034899+0000","last_fullsized":"2020-01-16T00:12:20.034899+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.033656+0000","last_change":"2020-01-16T00:12:20.033023+0000","last_active":"2020-01-16T00:12:20.033656+0000","last_peered":"2020-01-16T00:12:20.033656+0000","last_clean":"2020-01-16T00:12:20.033656+0000","last_became_active":"2020-01-16T00:12:20.031576+0000","last_became_peered":"2020-01-16T00:12:20.031576+0000","last_unstale":"2020-01-16T00:12:20.033656+0000","last_undegraded":"2020-01-16T00:12:20.033656+0000","last_fullsized":"2020-01-16T00:12:20.033656+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.031703+0000","last_change":"2020-01-16T00:12:20.031201+0000","last_active":"2020-01-16T00:12:20.031703+0000","last_peered":"2020-01-16T00:12:20.031703+0000","last_clean":"2020-01-16T00:12:20.031703+0000","last_became_active":"2020-01-16T00:12:20.030752+0000","last_became_peered":"2020-01-16T00:12:20.030752+0000","last_unstale":"2020-01-16T00:12:20.031703+0000","last_undegraded":"2020-01-16T00:12:20.031703+0000","last_fullsized":"2020-01-16T00:12:20.031703+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.034773+0000","last_change":"2020-01-16T00:12:20.033205+0000","last_active":"2020-01-16T00:12:20.034773+0000","last_peered":"2020-01-16T00:12:20.034773+0000","last_clean":"2020-01-16T00:12:20.034773+0000","last_became_active":"2020-01-16T00:12:20.030792+0000","last_became_peered":"2020-01-16T00:12:20.030792+0000","last_unstale":"2020-01-16T00:12:20.034773+0000","last_undegraded":"2020-01-16T00:12:20.034773+0000","last_fullsized":"2020-01-16T00:12:20.034773+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.033641+0000","last_change":"2020-01-16T00:12:20.032658+0000","last_active":"2020-01-16T00:12:20.033641+0000","last_peered":"2020-01-16T00:12:20.033641+0000","last_clean":"2020-01-16T00:12:20.033641+0000","last_became_active":"2020-01-16T00:12:20.030794+0000","last_became_peered":"2020-01-16T00:12:20.030794+0000","last_unstale":"2020-01-16T00:12:20.033641+0000","last_undegraded":"2020-01-16T00:12:20.033641+0000","last_fullsized":"2020-01-16T00:12:20.033641+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.032720+0000","last_change":"2020-01-16T00:12:20.032010+0000","last_active":"2020-01-16T00:12:20.032720+0000","last_peered":"2020-01-16T00:12:20.032720+0000","last_clean":"2020-01-16T00:12:20.032720+0000","last_became_active":"2020-01-16T00:12:20.031560+0000","last_became_peered":"2020-01-16T00:12:20.031560+0000","last_unstale":"2020-01-16T00:12:20.032720+0000","last_undegraded":"2020-01-16T00:12:20.032720+0000","last_fullsized":"2020-01-16T00:12:20.032720+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.033296+0000","last_change":"2020-01-16T00:12:20.032733+0000","last_active":"2020-01-16T00:12:20.033296+0000","last_peered":"2020-01-16T00:12:20.033296+0000","last_clean":"2020-01-16T00:12:20.033296+0000","last_became_active":"2020-01-16T00:12:20.032284+0000","last_became_peered":"2020-01-16T00:12:20.032284+0000","last_unstale":"2020-01-16T00:12:20.033296+0000","last_undegraded":"2020-01-16T00:12:20.033296+0000","last_fullsized":"2020-01-16T00:12:20.033296+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.032829+0000","last_change":"2020-01-16T00:12:20.031741+0000","last_active":"2020-01-16T00:12:20.032829+0000","last_peered":"2020-01-16T00:12:20.032829+0000","last_clean":"2020-01-16T00:12:20.032829+0000","last_became_active":"2020-01-16T00:12:20.030776+0000","last_became_peered":"2020-01-16T00:12:20.030776+0000","last_unstale":"2020-01-16T00:12:20.032829+0000","last_undegraded":"2020-01-16T00:12:20.032829+0000","last_fullsized":"2020-01-16T00:12:20.032829+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}]}} 2020-01-16T00:12:28.594 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-01-16T00:12:28.835 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:12:28.838 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2020-01-16T00:12:28.847 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2020-01-16T00:12:27.953785+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":24,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8389824,"kb_used_data":1152,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746584896,"statfs":{"total":773094113280,"available":764502933504,"internally_reserved":8589934592,"allocated":1179648,"data_stored":350752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.034899+0000","last_change":"2020-01-16T00:12:20.034125+0000","last_active":"2020-01-16T00:12:20.034899+0000","last_peered":"2020-01-16T00:12:20.034899+0000","last_clean":"2020-01-16T00:12:20.034899+0000","last_became_active":"2020-01-16T00:12:20.033645+0000","last_became_peered":"2020-01-16T00:12:20.033645+0000","last_unstale":"2020-01-16T00:12:20.034899+0000","last_undegraded":"2020-01-16T00:12:20.034899+0000","last_fullsized":"2020-01-16T00:12:20.034899+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.033656+0000","last_change":"2020-01-16T00:12:20.033023+0000","last_active":"2020-01-16T00:12:20.033656+0000","last_peered":"2020-01-16T00:12:20.033656+0000","last_clean":"2020-01-16T00:12:20.033656+0000","last_became_active":"2020-01-16T00:12:20.031576+0000","last_became_peered":"2020-01-16T00:12:20.031576+0000","last_unstale":"2020-01-16T00:12:20.033656+0000","last_undegraded":"2020-01-16T00:12:20.033656+0000","last_fullsized":"2020-01-16T00:12:20.033656+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.031703+0000","last_change":"2020-01-16T00:12:20.031201+0000","last_active":"2020-01-16T00:12:20.031703+0000","last_peered":"2020-01-16T00:12:20.031703+0000","last_clean":"2020-01-16T00:12:20.031703+0000","last_became_active":"2020-01-16T00:12:20.030752+0000","last_became_peered":"2020-01-16T00:12:20.030752+0000","last_unstale":"2020-01-16T00:12:20.031703+0000","last_undegraded":"2020-01-16T00:12:20.031703+0000","last_fullsized":"2020-01-16T00:12:20.031703+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.034773+0000","last_change":"2020-01-16T00:12:20.033205+0000","last_active":"2020-01-16T00:12:20.034773+0000","last_peered":"2020-01-16T00:12:20.034773+0000","last_clean":"2020-01-16T00:12:20.034773+0000","last_became_active":"2020-01-16T00:12:20.030792+0000","last_became_peered":"2020-01-16T00:12:20.030792+0000","last_unstale":"2020-01-16T00:12:20.034773+0000","last_undegraded":"2020-01-16T00:12:20.034773+0000","last_fullsized":"2020-01-16T00:12:20.034773+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.033641+0000","last_change":"2020-01-16T00:12:20.032658+0000","last_active":"2020-01-16T00:12:20.033641+0000","last_peered":"2020-01-16T00:12:20.033641+0000","last_clean":"2020-01-16T00:12:20.033641+0000","last_became_active":"2020-01-16T00:12:20.030794+0000","last_became_peered":"2020-01-16T00:12:20.030794+0000","last_unstale":"2020-01-16T00:12:20.033641+0000","last_undegraded":"2020-01-16T00:12:20.033641+0000","last_fullsized":"2020-01-16T00:12:20.033641+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.032720+0000","last_change":"2020-01-16T00:12:20.032010+0000","last_active":"2020-01-16T00:12:20.032720+0000","last_peered":"2020-01-16T00:12:20.032720+0000","last_clean":"2020-01-16T00:12:20.032720+0000","last_became_active":"2020-01-16T00:12:20.031560+0000","last_became_peered":"2020-01-16T00:12:20.031560+0000","last_unstale":"2020-01-16T00:12:20.032720+0000","last_undegraded":"2020-01-16T00:12:20.032720+0000","last_fullsized":"2020-01-16T00:12:20.032720+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.033296+0000","last_change":"2020-01-16T00:12:20.032733+0000","last_active":"2020-01-16T00:12:20.033296+0000","last_peered":"2020-01-16T00:12:20.033296+0000","last_clean":"2020-01-16T00:12:20.033296+0000","last_became_active":"2020-01-16T00:12:20.032284+0000","last_became_peered":"2020-01-16T00:12:20.032284+0000","last_unstale":"2020-01-16T00:12:20.033296+0000","last_undegraded":"2020-01-16T00:12:20.033296+0000","last_fullsized":"2020-01-16T00:12:20.033296+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.032829+0000","last_change":"2020-01-16T00:12:20.031741+0000","last_active":"2020-01-16T00:12:20.032829+0000","last_peered":"2020-01-16T00:12:20.032829+0000","last_clean":"2020-01-16T00:12:20.032829+0000","last_became_active":"2020-01-16T00:12:20.030776+0000","last_became_peered":"2020-01-16T00:12:20.030776+0000","last_unstale":"2020-01-16T00:12:20.032829+0000","last_undegraded":"2020-01-16T00:12:20.032829+0000","last_fullsized":"2020-01-16T00:12:20.032829+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}]}} 2020-01-16T00:12:28.849 INFO:tasks.ceph.ceph_manager.ceph:clean! 2020-01-16T00:12:28.849 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2020-01-16T00:12:29.087 INFO:teuthology.orchestra.run.smithi123.stdout:epoch 18 2020-01-16T00:12:29.088 INFO:teuthology.orchestra.run.smithi123.stdout:fsid 864e7191-e3db-4201-b88a-a4873d1eb7fe 2020-01-16T00:12:29.088 INFO:teuthology.orchestra.run.smithi123.stdout:created 2020-01-16T00:12:06.009499+0000 2020-01-16T00:12:29.088 INFO:teuthology.orchestra.run.smithi123.stdout:modified 2020-01-16T00:12:27.761473+0000 2020-01-16T00:12:29.088 INFO:teuthology.orchestra.run.smithi123.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2020-01-16T00:12:29.088 INFO:teuthology.orchestra.run.smithi123.stdout:crush_version 5 2020-01-16T00:12:29.088 INFO:teuthology.orchestra.run.smithi123.stdout:full_ratio 0.95 2020-01-16T00:12:29.088 INFO:teuthology.orchestra.run.smithi123.stdout:backfillfull_ratio 0.9 2020-01-16T00:12:29.089 INFO:teuthology.orchestra.run.smithi123.stdout:nearfull_ratio 0.85 2020-01-16T00:12:29.089 INFO:teuthology.orchestra.run.smithi123.stdout:require_min_compat_client jewel 2020-01-16T00:12:29.089 INFO:teuthology.orchestra.run.smithi123.stdout:min_compat_client jewel 2020-01-16T00:12:29.089 INFO:teuthology.orchestra.run.smithi123.stdout:require_osd_release octopus 2020-01-16T00:12:29.089 INFO:teuthology.orchestra.run.smithi123.stdout:pool 1 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 17 flags hashpspool stripe_width 0 application rbd 2020-01-16T00:12:29.089 INFO:teuthology.orchestra.run.smithi123.stdout:max_osd 8 2020-01-16T00:12:29.090 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.123:6809/11906 v1:172.21.15.123:6810/11906 exists,up 4b8d1e40-95e3-4adc-9929-cb7e6cc8b766 2020-01-16T00:12:29.090 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1 up in weight 1 up_from 13 up_thru 15 down_at 0 last_clean_interval [0,0) v1:172.21.15.123:6805/11905 v1:172.21.15.123:6806/11905 exists,up 2e373d15-6348-4caa-aa9f-b46037376a3b 2020-01-16T00:12:29.090 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.123:6801/11907 v1:172.21.15.123:6802/11907 exists,up 5736ff2c-076a-4d41-8e11-06e2fd1f5f94 2020-01-16T00:12:29.090 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3 up in weight 1 up_from 13 up_thru 15 down_at 0 last_clean_interval [0,0) v1:172.21.15.123:6813/11917 v1:172.21.15.123:6814/11917 exists,up 554f53a8-c5b1-43c9-a7d6-93629d84f1ea 2020-01-16T00:12:29.090 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4 up in weight 1 up_from 13 up_thru 15 down_at 0 last_clean_interval [0,0) v1:172.21.15.145:6800/11851 v1:172.21.15.145:6801/11851 exists,up 77945419-c04b-4071-ab87-256d77640dc0 2020-01-16T00:12:29.090 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5 up in weight 1 up_from 13 up_thru 15 down_at 0 last_clean_interval [0,0) v1:172.21.15.145:6804/11860 v1:172.21.15.145:6805/11860 exists,up d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4 2020-01-16T00:12:29.090 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6 up in weight 1 up_from 13 up_thru 15 down_at 0 last_clean_interval [0,0) v1:172.21.15.145:6812/11867 v1:172.21.15.145:6813/11867 exists,up e9d342cd-5c2d-40a0-b8ea-24588f280389 2020-01-16T00:12:29.090 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7 up in weight 1 up_from 13 up_thru 15 down_at 0 last_clean_interval [0,0) v1:172.21.15.145:6808/11868 v1:172.21.15.145:6809/11868 exists,up 6a301b66-384a-40d8-adc8-96fff49e982e 2020-01-16T00:12:29.104 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.123:6809/11906 v1:172.21.15.123:6810/11906 exists,up 4b8d1e40-95e3-4adc-9929-cb7e6cc8b766', 'osd.1 up in weight 1 up_from 13 up_thru 15 down_at 0 last_clean_interval [0,0) v1:172.21.15.123:6805/11905 v1:172.21.15.123:6806/11905 exists,up 2e373d15-6348-4caa-aa9f-b46037376a3b', 'osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.123:6801/11907 v1:172.21.15.123:6802/11907 exists,up 5736ff2c-076a-4d41-8e11-06e2fd1f5f94', 'osd.3 up in weight 1 up_from 13 up_thru 15 down_at 0 last_clean_interval [0,0) v1:172.21.15.123:6813/11917 v1:172.21.15.123:6814/11917 exists,up 554f53a8-c5b1-43c9-a7d6-93629d84f1ea', 'osd.4 up in weight 1 up_from 13 up_thru 15 down_at 0 last_clean_interval [0,0) v1:172.21.15.145:6800/11851 v1:172.21.15.145:6801/11851 exists,up 77945419-c04b-4071-ab87-256d77640dc0', 'osd.5 up in weight 1 up_from 13 up_thru 15 down_at 0 last_clean_interval [0,0) v1:172.21.15.145:6804/11860 v1:172.21.15.145:6805/11860 exists,up d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4', 'osd.6 up in weight 1 up_from 13 up_thru 15 down_at 0 last_clean_interval [0,0) v1:172.21.15.145:6812/11867 v1:172.21.15.145:6813/11867 exists,up e9d342cd-5c2d-40a0-b8ea-24588f280389', 'osd.7 up in weight 1 up_from 13 up_thru 15 down_at 0 last_clean_interval [0,0) v1:172.21.15.145:6808/11868 v1:172.21.15.145:6809/11868 exists,up 6a301b66-384a-40d8-adc8-96fff49e982e'] 2020-01-16T00:12:29.104 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2020-01-16T00:12:29.209 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:29.209 INFO:teuthology.orchestra.run.smithi123.stdout: "name": "mon.a", 2020-01-16T00:12:29.209 INFO:teuthology.orchestra.run.smithi123.stdout: "cluster": "ceph", 2020-01-16T00:12:29.209 INFO:teuthology.orchestra.run.smithi123.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2020-01-16T00:12:29.209 INFO:teuthology.orchestra.run.smithi123.stdout: "admin_socket_mode": "", 2020-01-16T00:12:29.209 INFO:teuthology.orchestra.run.smithi123.stdout: "allow_ansi": "Terminal", 2020-01-16T00:12:29.210 INFO:teuthology.orchestra.run.smithi123.stdout: "auth_client_required": "cephx, none", 2020-01-16T00:12:29.210 INFO:teuthology.orchestra.run.smithi123.stdout: "auth_cluster_required": "cephx", 2020-01-16T00:12:29.210 INFO:teuthology.orchestra.run.smithi123.stdout: "auth_debug": "true", 2020-01-16T00:12:29.210 INFO:teuthology.orchestra.run.smithi123.stdout: "auth_mon_ticket_ttl": "43200.000000", 2020-01-16T00:12:29.210 INFO:teuthology.orchestra.run.smithi123.stdout: "auth_service_required": "cephx", 2020-01-16T00:12:29.210 INFO:teuthology.orchestra.run.smithi123.stdout: "auth_service_ticket_ttl": "3600.000000", 2020-01-16T00:12:29.210 INFO:teuthology.orchestra.run.smithi123.stdout: "auth_supported": "cephx", 2020-01-16T00:12:29.211 INFO:teuthology.orchestra.run.smithi123.stdout: "bdev_aio": "true", 2020-01-16T00:12:29.211 INFO:teuthology.orchestra.run.smithi123.stdout: "bdev_aio_max_queue_depth": "1024", 2020-01-16T00:12:29.213 INFO:teuthology.orchestra.run.smithi123.stdout: "bdev_aio_poll_ms": "250", 2020-01-16T00:12:29.213 INFO:teuthology.orchestra.run.smithi123.stdout: "bdev_aio_reap_max": "16", 2020-01-16T00:12:29.214 INFO:teuthology.orchestra.run.smithi123.stdout: "bdev_async_discard": "false", 2020-01-16T00:12:29.214 INFO:teuthology.orchestra.run.smithi123.stdout: "bdev_block_size": "4096", 2020-01-16T00:12:29.214 INFO:teuthology.orchestra.run.smithi123.stdout: "bdev_debug_aio": "false", 2020-01-16T00:12:29.214 INFO:teuthology.orchestra.run.smithi123.stdout: "bdev_debug_aio_log_age": "5.000000", 2020-01-16T00:12:29.214 INFO:teuthology.orchestra.run.smithi123.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2020-01-16T00:12:29.214 INFO:teuthology.orchestra.run.smithi123.stdout: "bdev_debug_inflight_ios": "false", 2020-01-16T00:12:29.214 INFO:teuthology.orchestra.run.smithi123.stdout: "bdev_enable_discard": "false", 2020-01-16T00:12:29.215 INFO:teuthology.orchestra.run.smithi123.stdout: "bdev_inject_crash": "0", 2020-01-16T00:12:29.215 INFO:teuthology.orchestra.run.smithi123.stdout: "bdev_inject_crash_flush_delay": "2", 2020-01-16T00:12:29.215 INFO:teuthology.orchestra.run.smithi123.stdout: "bdev_nvme_retry_count": "-1", 2020-01-16T00:12:29.215 INFO:teuthology.orchestra.run.smithi123.stdout: "bdev_nvme_unbind_from_kernel": "false", 2020-01-16T00:12:29.215 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_alloc_size": "1048576", 2020-01-16T00:12:29.215 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_allocator": "bitmap", 2020-01-16T00:12:29.215 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_buffered_io": "true", 2020-01-16T00:12:29.215 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_compact_log_sync": "false", 2020-01-16T00:12:29.216 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2020-01-16T00:12:29.216 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_log_compact_min_size": "16777216", 2020-01-16T00:12:29.216 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_log_replay_check_allocations": "true", 2020-01-16T00:12:29.216 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_max_log_runway": "4194304", 2020-01-16T00:12:29.216 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_max_prefetch": "1048576", 2020-01-16T00:12:29.216 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_min_flush_size": "524288", 2020-01-16T00:12:29.216 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_min_log_runway": "1048576", 2020-01-16T00:12:29.217 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_preextend_wal_files": "true", 2020-01-16T00:12:29.217 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_shared_alloc_size": "65536", 2020-01-16T00:12:29.217 INFO:teuthology.orchestra.run.smithi123.stdout: "bluefs_sync_write": "false", 2020-01-16T00:12:29.217 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2020-01-16T00:12:29.217 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2020-01-16T00:12:29.217 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_allocator": "bitmap", 2020-01-16T00:12:29.217 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2020-01-16T00:12:29.217 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2020-01-16T00:12:29.218 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2020-01-16T00:12:29.218 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_bitmapallocator_span_size": "1024", 2020-01-16T00:12:29.218 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_blobid_prealloc": "10240", 2020-01-16T00:12:29.218 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_block_create": "true", 2020-01-16T00:12:29.218 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_block_db_create": "false", 2020-01-16T00:12:29.218 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_block_db_path": "", 2020-01-16T00:12:29.218 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_block_db_size": "0", 2020-01-16T00:12:29.219 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_block_path": "", 2020-01-16T00:12:29.219 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_block_preallocate_file": "false", 2020-01-16T00:12:29.219 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_block_size": "107374182400", 2020-01-16T00:12:29.219 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_block_wal_create": "false", 2020-01-16T00:12:29.219 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_block_wal_path": "", 2020-01-16T00:12:29.219 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_block_wal_size": "100663296", 2020-01-16T00:12:29.219 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_bluefs": "true", 2020-01-16T00:12:29.219 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2020-01-16T00:12:29.220 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_bluefs_balance_interval": "1.000000", 2020-01-16T00:12:29.220 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_bluefs_db_compatibility": "true", 2020-01-16T00:12:29.220 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_bluefs_env_mirror": "false", 2020-01-16T00:12:29.220 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_bluefs_gift_ratio": "0.020000", 2020-01-16T00:12:29.220 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_bluefs_max_free": "10737418240", 2020-01-16T00:12:29.220 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_bluefs_max_ratio": "0.900000", 2020-01-16T00:12:29.220 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_bluefs_min": "1073741824", 2020-01-16T00:12:29.220 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_bluefs_min_free": "1073741824", 2020-01-16T00:12:29.221 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_bluefs_min_ratio": "0.020000", 2020-01-16T00:12:29.221 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_bluefs_reclaim_ratio": "0.200000", 2020-01-16T00:12:29.221 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_autotune": "true", 2020-01-16T00:12:29.221 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_autotune_interval": "5.000000", 2020-01-16T00:12:29.221 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_kv_ratio": "0.400000", 2020-01-16T00:12:29.221 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_meta_ratio": "0.400000", 2020-01-16T00:12:29.221 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_size": "0", 2020-01-16T00:12:29.222 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_size_hdd": "1073741824", 2020-01-16T00:12:29.222 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_size_ssd": "3221225472", 2020-01-16T00:12:29.222 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_trim_interval": "0.050000", 2020-01-16T00:12:29.222 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_trim_max_skip_pinned": "64", 2020-01-16T00:12:29.222 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_cache_type": "2q", 2020-01-16T00:12:29.222 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_clone_cow": "true", 2020-01-16T00:12:29.222 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_compression_algorithm": "snappy", 2020-01-16T00:12:29.222 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_compression_max_blob_size": "0", 2020-01-16T00:12:29.223 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_compression_max_blob_size_hdd": "524288", 2020-01-16T00:12:29.223 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2020-01-16T00:12:29.223 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_compression_min_blob_size": "0", 2020-01-16T00:12:29.223 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_compression_min_blob_size_hdd": "131072", 2020-01-16T00:12:29.223 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_compression_min_blob_size_ssd": "8192", 2020-01-16T00:12:29.241 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_compression_mode": "none", 2020-01-16T00:12:29.242 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_compression_required_ratio": "0.875000", 2020-01-16T00:12:29.242 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_csum_type": "crc32c", 2020-01-16T00:12:29.242 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_debug_enforce_settings": "default", 2020-01-16T00:12:29.242 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_debug_freelist": "false", 2020-01-16T00:12:29.242 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_debug_fsck_abort": "false", 2020-01-16T00:12:29.242 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_debug_inject_bug21040": "false", 2020-01-16T00:12:29.243 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2020-01-16T00:12:29.243 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_debug_inject_read_err": "false", 2020-01-16T00:12:29.243 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_debug_misc": "false", 2020-01-16T00:12:29.243 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_debug_no_reuse_blocks": "false", 2020-01-16T00:12:29.243 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_debug_omit_block_device_write": "false", 2020-01-16T00:12:29.243 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_debug_omit_kv_commit": "false", 2020-01-16T00:12:29.243 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2020-01-16T00:12:29.243 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_debug_prefill": "0.000000", 2020-01-16T00:12:29.244 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_debug_prefragment_max": "1048576", 2020-01-16T00:12:29.244 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_debug_random_read_err": "0.000000", 2020-01-16T00:12:29.244 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2020-01-16T00:12:29.244 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_debug_small_allocations": "0", 2020-01-16T00:12:29.244 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2020-01-16T00:12:29.244 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_default_buffered_read": "true", 2020-01-16T00:12:29.244 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_default_buffered_write": "false", 2020-01-16T00:12:29.244 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_deferred_batch_ops": "0", 2020-01-16T00:12:29.245 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2020-01-16T00:12:29.245 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2020-01-16T00:12:29.245 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2020-01-16T00:12:29.245 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_extent_map_shard_max_size": "1200", 2020-01-16T00:12:29.245 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_extent_map_shard_min_size": "150", 2020-01-16T00:12:29.245 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_extent_map_shard_target_size": "500", 2020-01-16T00:12:29.245 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2020-01-16T00:12:29.246 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_freelist_blocks_per_key": "128", 2020-01-16T00:12:29.246 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2020-01-16T00:12:29.246 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2020-01-16T00:12:29.246 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_fsck_on_mkfs": "true", 2020-01-16T00:12:29.246 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2020-01-16T00:12:29.246 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_fsck_on_mount": "false", 2020-01-16T00:12:29.246 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_fsck_on_mount_deep": "false", 2020-01-16T00:12:29.246 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_fsck_on_umount": "false", 2020-01-16T00:12:29.247 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_fsck_on_umount_deep": "false", 2020-01-16T00:12:29.247 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_fsck_quick_fix_on_mount": "true", 2020-01-16T00:12:29.247 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_fsck_quick_fix_threads": "2", 2020-01-16T00:12:29.247 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2020-01-16T00:12:29.247 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_gc_enable_blob_threshold": "0", 2020-01-16T00:12:29.247 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_gc_enable_total_threshold": "0", 2020-01-16T00:12:29.247 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_ignore_data_csum": "false", 2020-01-16T00:12:29.248 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_ioring": "false", 2020-01-16T00:12:29.248 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_kvbackend": "rocksdb", 2020-01-16T00:12:29.248 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_log_collection_list_age": "60.000000", 2020-01-16T00:12:29.248 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2020-01-16T00:12:29.248 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_log_op_age": "5.000000", 2020-01-16T00:12:29.248 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_max_alloc_size": "0", 2020-01-16T00:12:29.248 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_max_blob_size": "0", 2020-01-16T00:12:29.248 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_max_blob_size_hdd": "524288", 2020-01-16T00:12:29.249 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_max_blob_size_ssd": "65536", 2020-01-16T00:12:29.249 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_max_defer_interval": "3.000000", 2020-01-16T00:12:29.249 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_max_deferred_txc": "32", 2020-01-16T00:12:29.249 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_min_alloc_size": "0", 2020-01-16T00:12:29.249 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_min_alloc_size_hdd": "4096", 2020-01-16T00:12:29.249 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_min_alloc_size_ssd": "4096", 2020-01-16T00:12:29.249 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_nid_prealloc": "1024", 2020-01-16T00:12:29.249 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_prefer_deferred_size": "0", 2020-01-16T00:12:29.250 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_prefer_deferred_size_hdd": "32768", 2020-01-16T00:12:29.250 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2020-01-16T00:12:29.250 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_retry_disk_reads": "3", 2020-01-16T00:12:29.250 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_rocksdb_cf": "false", 2020-01-16T00:12:29.250 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_rocksdb_cfs": "M= P= L=", 2020-01-16T00:12:29.250 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_rocksdb_options": "compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2", 2020-01-16T00:12:29.250 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_spdk_coremask": "0x1", 2020-01-16T00:12:29.251 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_spdk_io_sleep": "5", 2020-01-16T00:12:29.251 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_spdk_max_io_completion": "0", 2020-01-16T00:12:29.251 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_spdk_mem": "512", 2020-01-16T00:12:29.251 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_sync_submit_transaction": "false", 2020-01-16T00:12:29.251 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_throttle_bytes": "67108864", 2020-01-16T00:12:29.251 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_throttle_cost_per_io": "0", 2020-01-16T00:12:29.251 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2020-01-16T00:12:29.251 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2020-01-16T00:12:29.252 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2020-01-16T00:12:29.252 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_throttle_trace_rate": "0.000000", 2020-01-16T00:12:29.252 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_tracing": "false", 2020-01-16T00:12:29.252 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_volume_selection_policy": "rocksdb_original", 2020-01-16T00:12:29.252 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_volume_selection_reserved": "0", 2020-01-16T00:12:29.252 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2020-01-16T00:12:29.252 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2020-01-16T00:12:29.253 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_warn_on_legacy_statfs": "true", 2020-01-16T00:12:29.253 INFO:teuthology.orchestra.run.smithi123.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2020-01-16T00:12:29.253 INFO:teuthology.orchestra.run.smithi123.stdout: "cephadm_path": "/usr/sbin/cephadm", 2020-01-16T00:12:29.253 INFO:teuthology.orchestra.run.smithi123.stdout: "cephx_cluster_require_signatures": "false", 2020-01-16T00:12:29.253 INFO:teuthology.orchestra.run.smithi123.stdout: "cephx_cluster_require_version": "1", 2020-01-16T00:12:29.253 INFO:teuthology.orchestra.run.smithi123.stdout: "cephx_require_signatures": "false", 2020-01-16T00:12:29.253 INFO:teuthology.orchestra.run.smithi123.stdout: "cephx_require_version": "1", 2020-01-16T00:12:29.268 INFO:teuthology.orchestra.run.smithi123.stdout: "cephx_service_require_signatures": "false", 2020-01-16T00:12:29.268 INFO:teuthology.orchestra.run.smithi123.stdout: "cephx_service_require_version": "1", 2020-01-16T00:12:29.268 INFO:teuthology.orchestra.run.smithi123.stdout: "cephx_sign_messages": "true", 2020-01-16T00:12:29.269 INFO:teuthology.orchestra.run.smithi123.stdout: "chdir": "", 2020-01-16T00:12:29.269 INFO:teuthology.orchestra.run.smithi123.stdout: "client_acl_type": "", 2020-01-16T00:12:29.269 INFO:teuthology.orchestra.run.smithi123.stdout: "client_cache_mid": "0.750000", 2020-01-16T00:12:29.269 INFO:teuthology.orchestra.run.smithi123.stdout: "client_cache_size": "16384", 2020-01-16T00:12:29.269 INFO:teuthology.orchestra.run.smithi123.stdout: "client_caps_release_delay": "5", 2020-01-16T00:12:29.269 INFO:teuthology.orchestra.run.smithi123.stdout: "client_check_pool_perm": "true", 2020-01-16T00:12:29.269 INFO:teuthology.orchestra.run.smithi123.stdout: "client_debug_force_sync_read": "false", 2020-01-16T00:12:29.270 INFO:teuthology.orchestra.run.smithi123.stdout: "client_debug_getattr_caps": "false", 2020-01-16T00:12:29.270 INFO:teuthology.orchestra.run.smithi123.stdout: "client_debug_inject_tick_delay": "0", 2020-01-16T00:12:29.270 INFO:teuthology.orchestra.run.smithi123.stdout: "client_die_on_failed_dentry_invalidate": "true", 2020-01-16T00:12:29.270 INFO:teuthology.orchestra.run.smithi123.stdout: "client_die_on_failed_remount": "false", 2020-01-16T00:12:29.270 INFO:teuthology.orchestra.run.smithi123.stdout: "client_dirsize_rbytes": "true", 2020-01-16T00:12:29.270 INFO:teuthology.orchestra.run.smithi123.stdout: "client_force_lazyio": "false", 2020-01-16T00:12:29.270 INFO:teuthology.orchestra.run.smithi123.stdout: "client_inject_fixed_oldest_tid": "false", 2020-01-16T00:12:29.270 INFO:teuthology.orchestra.run.smithi123.stdout: "client_inject_release_failure": "false", 2020-01-16T00:12:29.271 INFO:teuthology.orchestra.run.smithi123.stdout: "client_max_inline_size": "4096", 2020-01-16T00:12:29.271 INFO:teuthology.orchestra.run.smithi123.stdout: "client_mds_namespace": "", 2020-01-16T00:12:29.271 INFO:teuthology.orchestra.run.smithi123.stdout: "client_metadata": "", 2020-01-16T00:12:29.271 INFO:teuthology.orchestra.run.smithi123.stdout: "client_mount_gid": "-1", 2020-01-16T00:12:29.271 INFO:teuthology.orchestra.run.smithi123.stdout: "client_mount_timeout": "300.000000", 2020-01-16T00:12:29.271 INFO:teuthology.orchestra.run.smithi123.stdout: "client_mount_uid": "-1", 2020-01-16T00:12:29.271 INFO:teuthology.orchestra.run.smithi123.stdout: "client_mountpoint": "/", 2020-01-16T00:12:29.272 INFO:teuthology.orchestra.run.smithi123.stdout: "client_notify_timeout": "10", 2020-01-16T00:12:29.272 INFO:teuthology.orchestra.run.smithi123.stdout: "client_oc": "true", 2020-01-16T00:12:29.272 INFO:teuthology.orchestra.run.smithi123.stdout: "client_oc_max_dirty": "104857600", 2020-01-16T00:12:29.272 INFO:teuthology.orchestra.run.smithi123.stdout: "client_oc_max_dirty_age": "5.000000", 2020-01-16T00:12:29.272 INFO:teuthology.orchestra.run.smithi123.stdout: "client_oc_max_objects": "1000", 2020-01-16T00:12:29.272 INFO:teuthology.orchestra.run.smithi123.stdout: "client_oc_size": "209715200", 2020-01-16T00:12:29.272 INFO:teuthology.orchestra.run.smithi123.stdout: "client_oc_target_dirty": "8388608", 2020-01-16T00:12:29.272 INFO:teuthology.orchestra.run.smithi123.stdout: "client_permissions": "true", 2020-01-16T00:12:29.273 INFO:teuthology.orchestra.run.smithi123.stdout: "client_quota_df": "true", 2020-01-16T00:12:29.273 INFO:teuthology.orchestra.run.smithi123.stdout: "client_readahead_max_bytes": "0", 2020-01-16T00:12:29.273 INFO:teuthology.orchestra.run.smithi123.stdout: "client_readahead_max_periods": "4", 2020-01-16T00:12:29.273 INFO:teuthology.orchestra.run.smithi123.stdout: "client_readahead_min": "131072", 2020-01-16T00:12:29.273 INFO:teuthology.orchestra.run.smithi123.stdout: "client_reconnect_stale": "false", 2020-01-16T00:12:29.273 INFO:teuthology.orchestra.run.smithi123.stdout: "client_snapdir": ".snap", 2020-01-16T00:12:29.273 INFO:teuthology.orchestra.run.smithi123.stdout: "client_tick_interval": "1.000000", 2020-01-16T00:12:29.274 INFO:teuthology.orchestra.run.smithi123.stdout: "client_trace": "", 2020-01-16T00:12:29.274 INFO:teuthology.orchestra.run.smithi123.stdout: "client_try_dentry_invalidate": "false", 2020-01-16T00:12:29.274 INFO:teuthology.orchestra.run.smithi123.stdout: "client_use_faked_inos": "false", 2020-01-16T00:12:29.274 INFO:teuthology.orchestra.run.smithi123.stdout: "client_use_random_mds": "false", 2020-01-16T00:12:29.274 INFO:teuthology.orchestra.run.smithi123.stdout: "clog_to_graylog": "false", 2020-01-16T00:12:29.274 INFO:teuthology.orchestra.run.smithi123.stdout: "clog_to_graylog_host": "127.0.0.1", 2020-01-16T00:12:29.275 INFO:teuthology.orchestra.run.smithi123.stdout: "clog_to_graylog_port": "12201", 2020-01-16T00:12:29.275 INFO:teuthology.orchestra.run.smithi123.stdout: "clog_to_monitors": "default=true", 2020-01-16T00:12:29.275 INFO:teuthology.orchestra.run.smithi123.stdout: "clog_to_syslog": "false", 2020-01-16T00:12:29.275 INFO:teuthology.orchestra.run.smithi123.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2020-01-16T00:12:29.275 INFO:teuthology.orchestra.run.smithi123.stdout: "clog_to_syslog_level": "info", 2020-01-16T00:12:29.275 INFO:teuthology.orchestra.run.smithi123.stdout: "cluster_addr": "-", 2020-01-16T00:12:29.275 INFO:teuthology.orchestra.run.smithi123.stdout: "cluster_network": "", 2020-01-16T00:12:29.275 INFO:teuthology.orchestra.run.smithi123.stdout: "cluster_network_interface": "", 2020-01-16T00:12:29.276 INFO:teuthology.orchestra.run.smithi123.stdout: "colors": "Terminal", 2020-01-16T00:12:29.276 INFO:teuthology.orchestra.run.smithi123.stdout: "compressor_zlib_isal": "false", 2020-01-16T00:12:29.276 INFO:teuthology.orchestra.run.smithi123.stdout: "compressor_zlib_level": "5", 2020-01-16T00:12:29.276 INFO:teuthology.orchestra.run.smithi123.stdout: "container_image": "ceph/daemon-base:latest-master-devel", 2020-01-16T00:12:29.276 INFO:teuthology.orchestra.run.smithi123.stdout: "continuation_prompt": ">", 2020-01-16T00:12:29.276 INFO:teuthology.orchestra.run.smithi123.stdout: "crash_dir": "/var/lib/ceph/crash", 2020-01-16T00:12:29.276 INFO:teuthology.orchestra.run.smithi123.stdout: "crimson_osd_obc_lru_size": "10", 2020-01-16T00:12:29.276 INFO:teuthology.orchestra.run.smithi123.stdout: "crush_location": "", 2020-01-16T00:12:29.277 INFO:teuthology.orchestra.run.smithi123.stdout: "crush_location_hook": "", 2020-01-16T00:12:29.277 INFO:teuthology.orchestra.run.smithi123.stdout: "crush_location_hook_timeout": "10", 2020-01-16T00:12:29.277 INFO:teuthology.orchestra.run.smithi123.stdout: "daemonize": "false", 2020-01-16T00:12:29.277 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_allow_any_pool_priority": "false", 2020-01-16T00:12:29.277 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_asok": "1/5", 2020-01-16T00:12:29.277 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_asok_assert_abort": "false", 2020-01-16T00:12:29.277 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_asserts_on_shutdown": "true", 2020-01-16T00:12:29.278 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_auth": "20/20", 2020-01-16T00:12:29.278 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_bdev": "1/3", 2020-01-16T00:12:29.278 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_bluefs": "1/5", 2020-01-16T00:12:29.278 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_bluestore": "1/5", 2020-01-16T00:12:29.278 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_buffer": "0/1", 2020-01-16T00:12:29.279 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_civetweb": "1/10", 2020-01-16T00:12:29.279 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_client": "0/5", 2020-01-16T00:12:29.279 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_compressor": "1/5", 2020-01-16T00:12:29.279 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_context": "0/1", 2020-01-16T00:12:29.280 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_crush": "1/1", 2020-01-16T00:12:29.280 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_crypto": "1/5", 2020-01-16T00:12:29.280 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_deliberately_leak_memory": "false", 2020-01-16T00:12:29.280 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_disable_randomized_ping": "false", 2020-01-16T00:12:29.280 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_dpdk": "1/5", 2020-01-16T00:12:29.280 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_eventtrace": "1/5", 2020-01-16T00:12:29.280 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_filer": "0/1", 2020-01-16T00:12:29.281 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_filestore": "1/3", 2020-01-16T00:12:29.281 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_finisher": "1/1", 2020-01-16T00:12:29.281 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_fuse": "1/5", 2020-01-16T00:12:29.281 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_heartbeat_testing_span": "0", 2020-01-16T00:12:29.281 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_heartbeatmap": "1/5", 2020-01-16T00:12:29.281 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_immutable_obj_cache": "0/5", 2020-01-16T00:12:29.282 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_javaclient": "1/5", 2020-01-16T00:12:29.282 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_journal": "1/3", 2020-01-16T00:12:29.282 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_journaler": "0/5", 2020-01-16T00:12:29.282 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_kstore": "1/5", 2020-01-16T00:12:29.282 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_leveldb": "4/5", 2020-01-16T00:12:29.282 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_lockdep": "0/1", 2020-01-16T00:12:29.282 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_mds": "1/5", 2020-01-16T00:12:29.282 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_mds_balancer": "1/5", 2020-01-16T00:12:29.283 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_mds_locker": "1/5", 2020-01-16T00:12:29.283 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_mds_log": "1/5", 2020-01-16T00:12:29.283 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_mds_log_expire": "1/5", 2020-01-16T00:12:29.283 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_mds_migrator": "1/5", 2020-01-16T00:12:29.283 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_memdb": "4/5", 2020-01-16T00:12:29.283 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_mgr": "1/5", 2020-01-16T00:12:29.283 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_mgrc": "1/5", 2020-01-16T00:12:29.283 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_mon": "20/20", 2020-01-16T00:12:29.284 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_monc": "0/10", 2020-01-16T00:12:29.284 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_ms": "1/1", 2020-01-16T00:12:29.284 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_none": "0/5", 2020-01-16T00:12:29.284 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_objclass": "0/5", 2020-01-16T00:12:29.284 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_objectcacher": "0/5", 2020-01-16T00:12:29.284 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_objecter": "0/1", 2020-01-16T00:12:29.284 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_optracker": "0/5", 2020-01-16T00:12:29.284 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_osd": "1/5", 2020-01-16T00:12:29.285 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_paxos": "20/20", 2020-01-16T00:12:29.285 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_perfcounter": "1/5", 2020-01-16T00:12:29.285 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_prioritycache": "1/5", 2020-01-16T00:12:29.285 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_rados": "0/5", 2020-01-16T00:12:29.285 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_rbd": "0/5", 2020-01-16T00:12:29.285 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_rbd_mirror": "0/5", 2020-01-16T00:12:29.285 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_rbd_replay": "0/5", 2020-01-16T00:12:29.286 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_refs": "0/0", 2020-01-16T00:12:29.286 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_reserver": "1/1", 2020-01-16T00:12:29.286 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_rgw": "1/5", 2020-01-16T00:12:29.286 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_rgw_sync": "1/5", 2020-01-16T00:12:29.286 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_rocksdb": "4/5", 2020-01-16T00:12:29.286 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_shell": "false", 2020-01-16T00:12:29.287 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_striper": "0/1", 2020-01-16T00:12:29.287 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_test": "0/5", 2020-01-16T00:12:29.287 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_throttle": "1/1", 2020-01-16T00:12:29.287 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_timer": "0/1", 2020-01-16T00:12:29.287 INFO:teuthology.orchestra.run.smithi123.stdout: "debug_tp": "0/5", 2020-01-16T00:12:29.287 INFO:teuthology.orchestra.run.smithi123.stdout: "device_failure_prediction_mode": "none", 2020-01-16T00:12:29.287 INFO:teuthology.orchestra.run.smithi123.stdout: "echo": "false", 2020-01-16T00:12:29.287 INFO:teuthology.orchestra.run.smithi123.stdout: "editor": "vim", 2020-01-16T00:12:29.288 INFO:teuthology.orchestra.run.smithi123.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2020-01-16T00:12:29.288 INFO:teuthology.orchestra.run.smithi123.stdout: "erasure_code_dir": "/usr/lib/ceph/erasure-code", 2020-01-16T00:12:29.288 INFO:teuthology.orchestra.run.smithi123.stdout: "err_to_graylog": "false", 2020-01-16T00:12:29.288 INFO:teuthology.orchestra.run.smithi123.stdout: "err_to_stderr": "true", 2020-01-16T00:12:29.288 INFO:teuthology.orchestra.run.smithi123.stdout: "err_to_syslog": "false", 2020-01-16T00:12:29.288 INFO:teuthology.orchestra.run.smithi123.stdout: "event_tracing": "false", 2020-01-16T00:12:29.288 INFO:teuthology.orchestra.run.smithi123.stdout: "fake_statfs_for_testing": "0", 2020-01-16T00:12:29.288 INFO:teuthology.orchestra.run.smithi123.stdout: "fatal_signal_handlers": "true", 2020-01-16T00:12:29.289 INFO:teuthology.orchestra.run.smithi123.stdout: "feedback_to_output": "false", 2020-01-16T00:12:29.289 INFO:teuthology.orchestra.run.smithi123.stdout: "filer_max_purge_ops": "10", 2020-01-16T00:12:29.289 INFO:teuthology.orchestra.run.smithi123.stdout: "filer_max_truncate_ops": "128", 2020-01-16T00:12:29.290 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_apply_finisher_threads": "1", 2020-01-16T00:12:29.290 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_blackhole": "false", 2020-01-16T00:12:29.290 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_btrfs_clone_range": "true", 2020-01-16T00:12:29.290 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_btrfs_snap": "true", 2020-01-16T00:12:29.290 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_caller_concurrency": "10", 2020-01-16T00:12:29.291 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_collect_device_partition_information": "true", 2020-01-16T00:12:29.291 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_commit_timeout": "600.000000", 2020-01-16T00:12:29.291 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_debug_inject_read_err": "false", 2020-01-16T00:12:29.291 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_debug_omap_check": "false", 2020-01-16T00:12:29.291 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_debug_random_read_err": "0.000000", 2020-01-16T00:12:29.291 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_debug_verify_split": "false", 2020-01-16T00:12:29.291 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_dump_file": "", 2020-01-16T00:12:29.291 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2020-01-16T00:12:29.292 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_expected_throughput_ops": "200.000000", 2020-01-16T00:12:29.292 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_fadvise": "true", 2020-01-16T00:12:29.292 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_fail_eio": "true", 2020-01-16T00:12:29.292 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_fd_cache_shards": "16", 2020-01-16T00:12:29.292 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_fd_cache_size": "128", 2020-01-16T00:12:29.292 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_fiemap": "false", 2020-01-16T00:12:29.292 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_fiemap_threshold": "4096", 2020-01-16T00:12:29.293 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_fsync_flushes_journal_data": "false", 2020-01-16T00:12:29.293 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_index_retry_probability": "0.000000", 2020-01-16T00:12:29.293 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_inject_stall": "0", 2020-01-16T00:12:29.293 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_journal_parallel": "false", 2020-01-16T00:12:29.293 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_journal_trailing": "false", 2020-01-16T00:12:29.293 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_journal_writeahead": "false", 2020-01-16T00:12:29.293 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_kill_at": "0", 2020-01-16T00:12:29.293 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_max_alloc_hint_size": "1048576", 2020-01-16T00:12:29.294 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_max_inline_xattr_size": "0", 2020-01-16T00:12:29.294 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2020-01-16T00:12:29.294 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_max_inline_xattr_size_other": "512", 2020-01-16T00:12:29.294 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2020-01-16T00:12:29.294 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_max_inline_xattrs": "0", 2020-01-16T00:12:29.294 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2020-01-16T00:12:29.294 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_max_inline_xattrs_other": "2", 2020-01-16T00:12:29.294 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_max_inline_xattrs_xfs": "10", 2020-01-16T00:12:29.295 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_max_sync_interval": "5.000000", 2020-01-16T00:12:29.295 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_max_xattr_value_size": "0", 2020-01-16T00:12:29.296 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2020-01-16T00:12:29.297 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_max_xattr_value_size_other": "1024", 2020-01-16T00:12:29.297 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2020-01-16T00:12:29.297 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_merge_threshold": "-10", 2020-01-16T00:12:29.297 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_min_sync_interval": "0.010000", 2020-01-16T00:12:29.297 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_odsync_write": "false", 2020-01-16T00:12:29.297 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_omap_backend": "rocksdb", 2020-01-16T00:12:29.298 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_omap_backend_path": "", 2020-01-16T00:12:29.298 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_omap_header_cache_size": "1024", 2020-01-16T00:12:29.298 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_ondisk_finisher_threads": "1", 2020-01-16T00:12:29.298 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_op_thread_suicide_timeout": "180", 2020-01-16T00:12:29.298 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_op_thread_timeout": "60", 2020-01-16T00:12:29.298 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_op_threads": "2", 2020-01-16T00:12:29.298 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_punch_hole": "false", 2020-01-16T00:12:29.298 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2020-01-16T00:12:29.299 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2020-01-16T00:12:29.299 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2020-01-16T00:12:29.299 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_queue_high_threshhold": "0.900000", 2020-01-16T00:12:29.299 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_queue_low_threshhold": "0.300000", 2020-01-16T00:12:29.299 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_queue_max_bytes": "104857600", 2020-01-16T00:12:29.299 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2020-01-16T00:12:29.299 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2020-01-16T00:12:29.300 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2020-01-16T00:12:29.300 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_queue_max_ops": "50", 2020-01-16T00:12:29.300 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2020-01-16T00:12:29.300 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_seek_data_hole": "false", 2020-01-16T00:12:29.300 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_sloppy_crc": "false", 2020-01-16T00:12:29.300 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_sloppy_crc_block_size": "65536", 2020-01-16T00:12:29.300 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_splice": "false", 2020-01-16T00:12:29.301 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_split_multiple": "2", 2020-01-16T00:12:29.301 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_split_rand_factor": "20", 2020-01-16T00:12:29.301 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_update_to": "1000", 2020-01-16T00:12:29.301 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2020-01-16T00:12:29.301 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2020-01-16T00:12:29.301 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2020-01-16T00:12:29.301 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2020-01-16T00:12:29.302 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2020-01-16T00:12:29.302 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2020-01-16T00:12:29.302 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_wbthrottle_enable": "true", 2020-01-16T00:12:29.302 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2020-01-16T00:12:29.302 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2020-01-16T00:12:29.302 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2020-01-16T00:12:29.302 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2020-01-16T00:12:29.303 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2020-01-16T00:12:29.303 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2020-01-16T00:12:29.303 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_xfs_extsize": "false", 2020-01-16T00:12:29.303 INFO:teuthology.orchestra.run.smithi123.stdout: "filestore_zfs_snap": "false", 2020-01-16T00:12:29.303 INFO:teuthology.orchestra.run.smithi123.stdout: "fio_dir": "/tmp/fio", 2020-01-16T00:12:29.303 INFO:teuthology.orchestra.run.smithi123.stdout: "fsid": "864e7191-e3db-4201-b88a-a4873d1eb7fe", 2020-01-16T00:12:29.303 INFO:teuthology.orchestra.run.smithi123.stdout: "fuse_allow_other": "true", 2020-01-16T00:12:29.303 INFO:teuthology.orchestra.run.smithi123.stdout: "fuse_atomic_o_trunc": "true", 2020-01-16T00:12:29.304 INFO:teuthology.orchestra.run.smithi123.stdout: "fuse_big_writes": "false", 2020-01-16T00:12:29.304 INFO:teuthology.orchestra.run.smithi123.stdout: "fuse_debug": "false", 2020-01-16T00:12:29.304 INFO:teuthology.orchestra.run.smithi123.stdout: "fuse_default_permissions": "false", 2020-01-16T00:12:29.304 INFO:teuthology.orchestra.run.smithi123.stdout: "fuse_disable_pagecache": "false", 2020-01-16T00:12:29.304 INFO:teuthology.orchestra.run.smithi123.stdout: "fuse_max_write": "0", 2020-01-16T00:12:29.304 INFO:teuthology.orchestra.run.smithi123.stdout: "fuse_multithreaded": "true", 2020-01-16T00:12:29.304 INFO:teuthology.orchestra.run.smithi123.stdout: "fuse_require_active_mds": "true", 2020-01-16T00:12:29.304 INFO:teuthology.orchestra.run.smithi123.stdout: "fuse_set_user_groups": "true", 2020-01-16T00:12:29.305 INFO:teuthology.orchestra.run.smithi123.stdout: "fuse_syncfs_on_mksnap": "true", 2020-01-16T00:12:29.305 INFO:teuthology.orchestra.run.smithi123.stdout: "fuse_use_invalidate_cb": "true", 2020-01-16T00:12:29.305 INFO:teuthology.orchestra.run.smithi123.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2020-01-16T00:12:29.305 INFO:teuthology.orchestra.run.smithi123.stdout: "gss_target_name": "ceph", 2020-01-16T00:12:29.305 INFO:teuthology.orchestra.run.smithi123.stdout: "heartbeat_file": "", 2020-01-16T00:12:29.305 INFO:teuthology.orchestra.run.smithi123.stdout: "heartbeat_inject_failure": "0", 2020-01-16T00:12:29.305 INFO:teuthology.orchestra.run.smithi123.stdout: "heartbeat_interval": "5", 2020-01-16T00:12:29.306 INFO:teuthology.orchestra.run.smithi123.stdout: "host": "", 2020-01-16T00:12:29.306 INFO:teuthology.orchestra.run.smithi123.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2020-01-16T00:12:29.306 INFO:teuthology.orchestra.run.smithi123.stdout: "immutable_object_cache_max_inflight_ops": "128", 2020-01-16T00:12:29.306 INFO:teuthology.orchestra.run.smithi123.stdout: "immutable_object_cache_max_size": "1073741824", 2020-01-16T00:12:29.306 INFO:teuthology.orchestra.run.smithi123.stdout: "immutable_object_cache_path": "/tmp", 2020-01-16T00:12:29.306 INFO:teuthology.orchestra.run.smithi123.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2020-01-16T00:12:29.306 INFO:teuthology.orchestra.run.smithi123.stdout: "immutable_object_cache_watermark": "0.100000", 2020-01-16T00:12:29.306 INFO:teuthology.orchestra.run.smithi123.stdout: "inject_early_sigterm": "false", 2020-01-16T00:12:29.307 INFO:teuthology.orchestra.run.smithi123.stdout: "journal_aio": "true", 2020-01-16T00:12:29.307 INFO:teuthology.orchestra.run.smithi123.stdout: "journal_align_min_size": "65536", 2020-01-16T00:12:29.307 INFO:teuthology.orchestra.run.smithi123.stdout: "journal_block_align": "true", 2020-01-16T00:12:29.307 INFO:teuthology.orchestra.run.smithi123.stdout: "journal_block_size": "4096", 2020-01-16T00:12:29.307 INFO:teuthology.orchestra.run.smithi123.stdout: "journal_dio": "true", 2020-01-16T00:12:29.307 INFO:teuthology.orchestra.run.smithi123.stdout: "journal_discard": "false", 2020-01-16T00:12:29.307 INFO:teuthology.orchestra.run.smithi123.stdout: "journal_force_aio": "false", 2020-01-16T00:12:29.308 INFO:teuthology.orchestra.run.smithi123.stdout: "journal_ignore_corruption": "false", 2020-01-16T00:12:29.308 INFO:teuthology.orchestra.run.smithi123.stdout: "journal_max_write_bytes": "10485760", 2020-01-16T00:12:29.308 INFO:teuthology.orchestra.run.smithi123.stdout: "journal_max_write_entries": "100", 2020-01-16T00:12:29.308 INFO:teuthology.orchestra.run.smithi123.stdout: "journal_replay_from": "0", 2020-01-16T00:12:29.308 INFO:teuthology.orchestra.run.smithi123.stdout: "journal_throttle_high_multiple": "0.000000", 2020-01-16T00:12:29.308 INFO:teuthology.orchestra.run.smithi123.stdout: "journal_throttle_high_threshhold": "0.900000", 2020-01-16T00:12:29.308 INFO:teuthology.orchestra.run.smithi123.stdout: "journal_throttle_low_threshhold": "0.600000", 2020-01-16T00:12:29.308 INFO:teuthology.orchestra.run.smithi123.stdout: "journal_throttle_max_multiple": "0.000000", 2020-01-16T00:12:29.309 INFO:teuthology.orchestra.run.smithi123.stdout: "journal_write_header_frequency": "0", 2020-01-16T00:12:29.309 INFO:teuthology.orchestra.run.smithi123.stdout: "journal_zero_on_create": "false", 2020-01-16T00:12:29.309 INFO:teuthology.orchestra.run.smithi123.stdout: "journaler_prefetch_periods": "10", 2020-01-16T00:12:29.309 INFO:teuthology.orchestra.run.smithi123.stdout: "journaler_prezero_periods": "5", 2020-01-16T00:12:29.309 INFO:teuthology.orchestra.run.smithi123.stdout: "journaler_write_head_interval": "15", 2020-01-16T00:12:29.309 INFO:teuthology.orchestra.run.smithi123.stdout: "key": "", 2020-01-16T00:12:29.309 INFO:teuthology.orchestra.run.smithi123.stdout: "keyfile": "", 2020-01-16T00:12:29.310 INFO:teuthology.orchestra.run.smithi123.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2020-01-16T00:12:29.310 INFO:teuthology.orchestra.run.smithi123.stdout: "kstore_backend": "rocksdb", 2020-01-16T00:12:29.310 INFO:teuthology.orchestra.run.smithi123.stdout: "kstore_default_stripe_size": "65536", 2020-01-16T00:12:29.310 INFO:teuthology.orchestra.run.smithi123.stdout: "kstore_fsck_on_mount": "false", 2020-01-16T00:12:29.310 INFO:teuthology.orchestra.run.smithi123.stdout: "kstore_fsck_on_mount_deep": "true", 2020-01-16T00:12:29.310 INFO:teuthology.orchestra.run.smithi123.stdout: "kstore_max_bytes": "67108864", 2020-01-16T00:12:29.310 INFO:teuthology.orchestra.run.smithi123.stdout: "kstore_max_ops": "512", 2020-01-16T00:12:29.311 INFO:teuthology.orchestra.run.smithi123.stdout: "kstore_nid_prealloc": "1024", 2020-01-16T00:12:29.311 INFO:teuthology.orchestra.run.smithi123.stdout: "kstore_onode_map_size": "1024", 2020-01-16T00:12:29.311 INFO:teuthology.orchestra.run.smithi123.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2020-01-16T00:12:29.311 INFO:teuthology.orchestra.run.smithi123.stdout: "kstore_sync_submit_transaction": "false", 2020-01-16T00:12:29.311 INFO:teuthology.orchestra.run.smithi123.stdout: "kstore_sync_transaction": "false", 2020-01-16T00:12:29.311 INFO:teuthology.orchestra.run.smithi123.stdout: "leveldb_block_size": "65536", 2020-01-16T00:12:29.312 INFO:teuthology.orchestra.run.smithi123.stdout: "leveldb_bloom_size": "0", 2020-01-16T00:12:29.312 INFO:teuthology.orchestra.run.smithi123.stdout: "leveldb_cache_size": "536870912", 2020-01-16T00:12:29.312 INFO:teuthology.orchestra.run.smithi123.stdout: "leveldb_compact_on_mount": "false", 2020-01-16T00:12:29.312 INFO:teuthology.orchestra.run.smithi123.stdout: "leveldb_compression": "false", 2020-01-16T00:12:29.312 INFO:teuthology.orchestra.run.smithi123.stdout: "leveldb_log": "", 2020-01-16T00:12:29.312 INFO:teuthology.orchestra.run.smithi123.stdout: "leveldb_log_to_ceph_log": "true", 2020-01-16T00:12:29.312 INFO:teuthology.orchestra.run.smithi123.stdout: "leveldb_max_open_files": "0", 2020-01-16T00:12:29.312 INFO:teuthology.orchestra.run.smithi123.stdout: "leveldb_paranoid": "false", 2020-01-16T00:12:29.313 INFO:teuthology.orchestra.run.smithi123.stdout: "leveldb_write_buffer_size": "33554432", 2020-01-16T00:12:29.313 INFO:teuthology.orchestra.run.smithi123.stdout: "lockdep": "false", 2020-01-16T00:12:29.313 INFO:teuthology.orchestra.run.smithi123.stdout: "lockdep_force_backtrace": "false", 2020-01-16T00:12:29.313 INFO:teuthology.orchestra.run.smithi123.stdout: "log_coarse_timestamps": "true", 2020-01-16T00:12:29.313 INFO:teuthology.orchestra.run.smithi123.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2020-01-16T00:12:29.313 INFO:teuthology.orchestra.run.smithi123.stdout: "log_flush_on_exit": "false", 2020-01-16T00:12:29.313 INFO:teuthology.orchestra.run.smithi123.stdout: "log_graylog_host": "127.0.0.1", 2020-01-16T00:12:29.313 INFO:teuthology.orchestra.run.smithi123.stdout: "log_graylog_port": "12201", 2020-01-16T00:12:29.314 INFO:teuthology.orchestra.run.smithi123.stdout: "log_max_new": "1000", 2020-01-16T00:12:29.314 INFO:teuthology.orchestra.run.smithi123.stdout: "log_max_recent": "10000", 2020-01-16T00:12:29.314 INFO:teuthology.orchestra.run.smithi123.stdout: "log_stderr_prefix": "", 2020-01-16T00:12:29.314 INFO:teuthology.orchestra.run.smithi123.stdout: "log_stop_at_utilization": "0.970000", 2020-01-16T00:12:29.314 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_file": "true", 2020-01-16T00:12:29.314 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_graylog": "false", 2020-01-16T00:12:29.314 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_stderr": "false", 2020-01-16T00:12:29.315 INFO:teuthology.orchestra.run.smithi123.stdout: "log_to_syslog": "false", 2020-01-16T00:12:29.315 INFO:teuthology.orchestra.run.smithi123.stdout: "max_completion_items": "50", 2020-01-16T00:12:29.315 INFO:teuthology.orchestra.run.smithi123.stdout: "max_rotating_auth_attempts": "10", 2020-01-16T00:12:29.315 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_action_on_write_error": "1", 2020-01-16T00:12:29.315 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_bal_export_pin": "true", 2020-01-16T00:12:29.315 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_bal_fragment_dirs": "true", 2020-01-16T00:12:29.315 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2020-01-16T00:12:29.315 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_bal_fragment_interval": "5", 2020-01-16T00:12:29.316 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_bal_fragment_size_max": "100000", 2020-01-16T00:12:29.316 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_bal_idle_threshold": "0.000000", 2020-01-16T00:12:29.316 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_bal_interval": "10", 2020-01-16T00:12:29.316 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_bal_max": "-1", 2020-01-16T00:12:29.316 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_bal_max_until": "-1", 2020-01-16T00:12:29.316 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_bal_merge_size": "50", 2020-01-16T00:12:29.316 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_bal_midchunk": "0.300000", 2020-01-16T00:12:29.316 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_bal_min_rebalance": "0.100000", 2020-01-16T00:12:29.317 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_bal_min_start": "0.200000", 2020-01-16T00:12:29.317 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_bal_minchunk": "0.001000", 2020-01-16T00:12:29.317 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_bal_mode": "0", 2020-01-16T00:12:29.317 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_bal_need_max": "1.200000", 2020-01-16T00:12:29.317 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_bal_need_min": "0.800000", 2020-01-16T00:12:29.317 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_bal_replicate_threshold": "8000.000000", 2020-01-16T00:12:29.317 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_bal_sample_interval": "3.000000", 2020-01-16T00:12:29.317 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_bal_split_bits": "3", 2020-01-16T00:12:29.318 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_bal_split_rd": "25000.000000", 2020-01-16T00:12:29.318 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_bal_split_size": "10000", 2020-01-16T00:12:29.318 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_bal_split_wr": "10000.000000", 2020-01-16T00:12:29.318 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_bal_target_decay": "10.000000", 2020-01-16T00:12:29.318 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2020-01-16T00:12:29.318 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_beacon_grace": "15.000000", 2020-01-16T00:12:29.319 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_beacon_interval": "4.000000", 2020-01-16T00:12:29.319 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_cache_memory_limit": "4294967296", 2020-01-16T00:12:29.319 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_cache_mid": "0.700000", 2020-01-16T00:12:29.319 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_cache_release_free_interval": "10", 2020-01-16T00:12:29.319 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_cache_reservation": "0.050000", 2020-01-16T00:12:29.319 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_cache_trim_decay_rate": "1.000000", 2020-01-16T00:12:29.319 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_cache_trim_interval": "1", 2020-01-16T00:12:29.319 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_cache_trim_threshold": "65536", 2020-01-16T00:12:29.320 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2020-01-16T00:12:29.320 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_client_prealloc_inos": "1000", 2020-01-16T00:12:29.320 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2020-01-16T00:12:29.320 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_damage_table_max_entries": "10000", 2020-01-16T00:12:29.320 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2020-01-16T00:12:29.320 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_debug_auth_pins": "false", 2020-01-16T00:12:29.321 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_debug_frag": "false", 2020-01-16T00:12:29.321 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_debug_scatterstat": "false", 2020-01-16T00:12:29.321 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_debug_subtrees": "false", 2020-01-16T00:12:29.321 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_decay_halflife": "5.000000", 2020-01-16T00:12:29.321 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_default_dir_hash": "2", 2020-01-16T00:12:29.321 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_defer_session_stale": "true", 2020-01-16T00:12:29.322 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_dir_keys_per_op": "16384", 2020-01-16T00:12:29.322 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_dir_max_commit_size": "10", 2020-01-16T00:12:29.322 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_dirstat_min_interval": "1.000000", 2020-01-16T00:12:29.322 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_dump_cache_after_rejoin": "false", 2020-01-16T00:12:29.322 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_dump_cache_on_map": "false", 2020-01-16T00:12:29.322 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_dump_cache_threshold_file": "0", 2020-01-16T00:12:29.322 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2020-01-16T00:12:29.323 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_early_reply": "true", 2020-01-16T00:12:29.323 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_enable_op_tracker": "true", 2020-01-16T00:12:29.323 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_enforce_unique_name": "true", 2020-01-16T00:12:29.324 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_forward_all_requests_to_auth": "false", 2020-01-16T00:12:29.324 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_freeze_tree_timeout": "30.000000", 2020-01-16T00:12:29.324 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2020-01-16T00:12:29.325 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_health_cache_threshold": "1.500000", 2020-01-16T00:12:29.325 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_health_summarize_threshold": "10", 2020-01-16T00:12:29.325 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_heartbeat_grace": "15.000000", 2020-01-16T00:12:29.325 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_inject_migrator_session_race": "false", 2020-01-16T00:12:29.325 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2020-01-16T00:12:29.325 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_join_fs": "", 2020-01-16T00:12:29.325 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_journal_format": "1", 2020-01-16T00:12:29.326 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_kill_create_at": "0", 2020-01-16T00:12:29.326 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_kill_export_at": "0", 2020-01-16T00:12:29.326 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_kill_import_at": "0", 2020-01-16T00:12:29.326 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_kill_journal_at": "0", 2020-01-16T00:12:29.326 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_kill_journal_expire_at": "0", 2020-01-16T00:12:29.326 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_kill_journal_replay_at": "0", 2020-01-16T00:12:29.326 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_kill_link_at": "0", 2020-01-16T00:12:29.326 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_kill_mdstable_at": "0", 2020-01-16T00:12:29.327 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_kill_openc_at": "0", 2020-01-16T00:12:29.327 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_kill_rename_at": "0", 2020-01-16T00:12:29.327 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_log_events_per_segment": "1024", 2020-01-16T00:12:29.327 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_log_max_events": "-1", 2020-01-16T00:12:29.327 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_log_max_segments": "128", 2020-01-16T00:12:29.327 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_log_pause": "false", 2020-01-16T00:12:29.327 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_log_segment_size": "0", 2020-01-16T00:12:29.327 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_log_skip_corrupt_events": "false", 2020-01-16T00:12:29.328 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_max_caps_per_client": "1048576", 2020-01-16T00:12:29.328 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_max_completed_flushes": "100000", 2020-01-16T00:12:29.328 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_max_completed_requests": "100000", 2020-01-16T00:12:29.328 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_max_export_size": "20971520", 2020-01-16T00:12:29.328 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_max_file_recover": "32", 2020-01-16T00:12:29.328 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_max_purge_files": "64", 2020-01-16T00:12:29.328 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_max_purge_ops": "8192", 2020-01-16T00:12:29.328 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2020-01-16T00:12:29.329 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_max_retries_on_remount_failure": "5", 2020-01-16T00:12:29.329 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_max_scrub_ops_in_progress": "5", 2020-01-16T00:12:29.329 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_max_snaps_per_dir": "100", 2020-01-16T00:12:29.329 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_max_xattr_pairs_size": "65536", 2020-01-16T00:12:29.329 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_min_caps_per_client": "100", 2020-01-16T00:12:29.329 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_mon_shutdown_timeout": "5.000000", 2020-01-16T00:12:29.329 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_numa_node": "-1", 2020-01-16T00:12:29.329 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_op_complaint_time": "30.000000", 2020-01-16T00:12:29.330 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_op_history_duration": "600", 2020-01-16T00:12:29.330 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_op_history_size": "20", 2020-01-16T00:12:29.330 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_op_log_threshold": "5", 2020-01-16T00:12:29.330 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2020-01-16T00:12:29.330 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_recall_global_max_decay_threshold": "65536", 2020-01-16T00:12:29.330 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_recall_max_caps": "5000", 2020-01-16T00:12:29.330 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_recall_max_decay_rate": "2.500000", 2020-01-16T00:12:29.330 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_recall_max_decay_threshold": "16384", 2020-01-16T00:12:29.331 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_recall_warning_decay_rate": "60.000000", 2020-01-16T00:12:29.331 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_recall_warning_threshold": "32768", 2020-01-16T00:12:29.331 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_reconnect_timeout": "45.000000", 2020-01-16T00:12:29.331 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_replay_interval": "1.000000", 2020-01-16T00:12:29.331 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_replay_unsafe_with_closed_session": "false", 2020-01-16T00:12:29.331 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_request_load_average_decay_rate": "60.000000", 2020-01-16T00:12:29.332 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_root_ino_gid": "0", 2020-01-16T00:12:29.332 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_root_ino_uid": "0", 2020-01-16T00:12:29.332 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_scatter_nudge_interval": "5.000000", 2020-01-16T00:12:29.332 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_session_blacklist_on_evict": "true", 2020-01-16T00:12:29.332 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_session_blacklist_on_timeout": "true", 2020-01-16T00:12:29.332 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2020-01-16T00:12:29.332 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_session_cache_liveness_magnitude": "10", 2020-01-16T00:12:29.332 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_sessionmap_keys_per_op": "1024", 2020-01-16T00:12:29.333 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_shutdown_check": "0", 2020-01-16T00:12:29.333 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_skip_ino": "0", 2020-01-16T00:12:29.333 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_snap_max_uid": "4294967294", 2020-01-16T00:12:29.333 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_snap_min_uid": "0", 2020-01-16T00:12:29.333 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_snap_rstat": "false", 2020-01-16T00:12:29.333 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_task_status_update_interval": "2.000000", 2020-01-16T00:12:29.333 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_thrash_exports": "0", 2020-01-16T00:12:29.334 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_thrash_fragments": "0", 2020-01-16T00:12:29.334 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_tick_interval": "5.000000", 2020-01-16T00:12:29.334 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_verify_backtrace": "1", 2020-01-16T00:12:29.334 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_verify_scatter": "false", 2020-01-16T00:12:29.334 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_wipe_ino_prealloc": "false", 2020-01-16T00:12:29.334 INFO:teuthology.orchestra.run.smithi123.stdout: "mds_wipe_sessions": "false", 2020-01-16T00:12:29.335 INFO:teuthology.orchestra.run.smithi123.stdout: "mempool_debug": "false", 2020-01-16T00:12:29.335 INFO:teuthology.orchestra.run.smithi123.stdout: "memstore_debug_omit_block_device_write": "false", 2020-01-16T00:12:29.335 INFO:teuthology.orchestra.run.smithi123.stdout: "memstore_device_bytes": "1073741824", 2020-01-16T00:12:29.335 INFO:teuthology.orchestra.run.smithi123.stdout: "memstore_page_set": "false", 2020-01-16T00:12:29.335 INFO:teuthology.orchestra.run.smithi123.stdout: "memstore_page_size": "65536", 2020-01-16T00:12:29.335 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr_client_bytes": "134217728", 2020-01-16T00:12:29.335 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr_client_messages": "512", 2020-01-16T00:12:29.335 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2020-01-16T00:12:29.336 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr_connect_retry_interval": "1.000000", 2020-01-16T00:12:29.336 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2020-01-16T00:12:29.336 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2020-01-16T00:12:29.336 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr_initial_modules": "restful iostat", 2020-01-16T00:12:29.336 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr_mds_bytes": "134217728", 2020-01-16T00:12:29.336 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr_mds_messages": "128", 2020-01-16T00:12:29.336 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2020-01-16T00:12:29.336 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr_mon_bytes": "134217728", 2020-01-16T00:12:29.337 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr_mon_messages": "128", 2020-01-16T00:12:29.337 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr_osd_bytes": "536870912", 2020-01-16T00:12:29.337 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr_osd_messages": "8192", 2020-01-16T00:12:29.337 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr_service_beacon_grace": "60.000000", 2020-01-16T00:12:29.337 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr_stats_period": "5", 2020-01-16T00:12:29.337 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr_stats_threshold": "5", 2020-01-16T00:12:29.337 INFO:teuthology.orchestra.run.smithi123.stdout: "mgr_tick_period": "2", 2020-01-16T00:12:29.338 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_accept_timeout_factor": "2.000000", 2020-01-16T00:12:29.338 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_allow_pool_delete": "true", 2020-01-16T00:12:29.338 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2020-01-16T00:12:29.338 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2020-01-16T00:12:29.338 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_client_bytes": "104857600", 2020-01-16T00:12:29.338 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_client_directed_command_retry": "5", 2020-01-16T00:12:29.338 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_client_hunt_interval": "3.000000", 2020-01-16T00:12:29.339 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2020-01-16T00:12:29.339 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2020-01-16T00:12:29.339 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2020-01-16T00:12:29.339 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_client_hunt_parallel": "3", 2020-01-16T00:12:29.339 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_client_max_log_entries_per_message": "1000", 2020-01-16T00:12:29.339 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_client_ping_interval": "10.000000", 2020-01-16T00:12:29.339 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_client_ping_timeout": "30.000000", 2020-01-16T00:12:29.340 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_clock_drift_allowed": "1.000000", 2020-01-16T00:12:29.340 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2020-01-16T00:12:29.340 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2020-01-16T00:12:29.340 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_cluster_log_file_level": "debug", 2020-01-16T00:12:29.340 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_cluster_log_to_file": "true", 2020-01-16T00:12:29.340 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_cluster_log_to_graylog": "false", 2020-01-16T00:12:29.340 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2020-01-16T00:12:29.340 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_cluster_log_to_graylog_port": "12201", 2020-01-16T00:12:29.341 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_cluster_log_to_stderr": "false", 2020-01-16T00:12:29.341 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_cluster_log_to_syslog": "default=false", 2020-01-16T00:12:29.341 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2020-01-16T00:12:29.341 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_cluster_log_to_syslog_level": "info", 2020-01-16T00:12:29.341 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_compact_on_bootstrap": "false", 2020-01-16T00:12:29.341 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_compact_on_start": "false", 2020-01-16T00:12:29.342 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_compact_on_trim": "true", 2020-01-16T00:12:29.342 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_config_key_max_entry_size": "65536", 2020-01-16T00:12:29.342 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_cpu_threads": "4", 2020-01-16T00:12:29.342 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_crush_min_required_version": "hammer", 2020-01-16T00:12:29.342 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_daemon_bytes": "419430400", 2020-01-16T00:12:29.342 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2020-01-16T00:12:29.342 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_data_avail_crit": "5", 2020-01-16T00:12:29.343 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_data_avail_warn": "5", 2020-01-16T00:12:29.343 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_data_size_warn": "16106127360", 2020-01-16T00:12:29.343 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_debug_block_osdmap_trim": "false", 2020-01-16T00:12:29.343 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_debug_deprecated_as_obsolete": "false", 2020-01-16T00:12:29.343 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_debug_dump_json": "false", 2020-01-16T00:12:29.344 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2020-01-16T00:12:29.344 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_debug_dump_transactions": "false", 2020-01-16T00:12:29.344 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_debug_extra_checks": "false", 2020-01-16T00:12:29.344 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_debug_no_initial_persistent_features": "false", 2020-01-16T00:12:29.344 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2020-01-16T00:12:29.344 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_debug_no_require_nautilus": "false", 2020-01-16T00:12:29.344 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_debug_no_require_octopus": "false", 2020-01-16T00:12:29.344 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2020-01-16T00:12:29.345 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_delta_reset_interval": "10.000000", 2020-01-16T00:12:29.345 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_dns_srv_name": "ceph-mon", 2020-01-16T00:12:29.345 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_election_timeout": "5.000000", 2020-01-16T00:12:29.345 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_enable_op_tracker": "true", 2020-01-16T00:12:29.345 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_fake_pool_delete": "false", 2020-01-16T00:12:29.345 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_force_quorum_join": "false", 2020-01-16T00:12:29.345 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_globalid_prealloc": "10000", 2020-01-16T00:12:29.345 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_health_log_update_period": "5", 2020-01-16T00:12:29.346 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_health_max_detail": "50", 2020-01-16T00:12:29.346 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_health_to_clog": "true", 2020-01-16T00:12:29.346 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_health_to_clog_interval": "3600", 2020-01-16T00:12:29.346 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2020-01-16T00:12:29.346 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_host": "[v1:172.21.15.123:6789],[v1:172.21.15.123:6790],[v1:172.21.15.145:6789]", 2020-01-16T00:12:29.346 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_initial_members": "", 2020-01-16T00:12:29.346 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2020-01-16T00:12:29.346 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2020-01-16T00:12:29.347 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_inject_transaction_delay_max": "10.000000", 2020-01-16T00:12:29.347 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2020-01-16T00:12:29.347 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_keyvaluedb": "rocksdb", 2020-01-16T00:12:29.347 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_lease": "5.000000", 2020-01-16T00:12:29.347 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2020-01-16T00:12:29.347 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_lease_renew_interval_factor": "0.600000", 2020-01-16T00:12:29.347 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_log_max_summary": "50", 2020-01-16T00:12:29.348 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_max_log_entries_per_event": "4096", 2020-01-16T00:12:29.348 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_max_log_epochs": "500", 2020-01-16T00:12:29.348 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_max_mdsmap_epochs": "500", 2020-01-16T00:12:29.348 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_max_mgrmap_epochs": "500", 2020-01-16T00:12:29.348 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_max_osd": "10000", 2020-01-16T00:12:29.348 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_max_pg_per_osd": "10000", 2020-01-16T00:12:29.348 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_max_pool_pg_num": "65536", 2020-01-16T00:12:29.348 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_max_snap_prune_per_epoch": "100", 2020-01-16T00:12:29.349 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_mds_blacklist_interval": "86400.000000", 2020-01-16T00:12:29.349 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_mds_force_trim_to": "0", 2020-01-16T00:12:29.349 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_mds_skip_sanity": "false", 2020-01-16T00:12:29.349 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_memory_autotune": "true", 2020-01-16T00:12:29.349 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_memory_target": "2147483648", 2020-01-16T00:12:29.349 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_mgr_beacon_grace": "30", 2020-01-16T00:12:29.349 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_mgr_blacklist_interval": "86400.000000", 2020-01-16T00:12:29.350 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_mgr_digest_period": "5", 2020-01-16T00:12:29.350 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_mgr_inactive_grace": "60", 2020-01-16T00:12:29.350 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_mgr_mkfs_grace": "240", 2020-01-16T00:12:29.350 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2020-01-16T00:12:29.350 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_min_osdmap_epochs": "50", 2020-01-16T00:12:29.350 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_op_complaint_time": "30", 2020-01-16T00:12:29.350 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_op_history_duration": "600", 2020-01-16T00:12:29.350 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_op_history_size": "20", 2020-01-16T00:12:29.351 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_op_history_slow_op_size": "20", 2020-01-16T00:12:29.351 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_op_history_slow_op_threshold": "10", 2020-01-16T00:12:29.351 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_op_log_threshold": "5", 2020-01-16T00:12:29.351 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_osd_adjust_down_out_interval": "true", 2020-01-16T00:12:29.351 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2020-01-16T00:12:29.352 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2020-01-16T00:12:29.352 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_osd_auto_mark_in": "false", 2020-01-16T00:12:29.352 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_osd_auto_mark_new_in": "true", 2020-01-16T00:12:29.352 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2020-01-16T00:12:29.353 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_osd_blacklist_default_expire": "3600.000000", 2020-01-16T00:12:29.353 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_osd_cache_size": "500", 2020-01-16T00:12:29.353 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_osd_cache_size_min": "134217728", 2020-01-16T00:12:29.353 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_osd_crush_smoke_test": "true", 2020-01-16T00:12:29.353 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_osd_destroyed_out_interval": "600", 2020-01-16T00:12:29.353 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_osd_down_out_interval": "600", 2020-01-16T00:12:29.353 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_osd_down_out_subtree_limit": "rack", 2020-01-16T00:12:29.354 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2020-01-16T00:12:29.354 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_osd_force_trim_to": "0", 2020-01-16T00:12:29.354 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_osd_full_ratio": "0.950000", 2020-01-16T00:12:29.354 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_osd_initial_require_min_compat_client": "jewel", 2020-01-16T00:12:29.354 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_osd_laggy_halflife": "3600", 2020-01-16T00:12:29.354 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_osd_laggy_max_interval": "300", 2020-01-16T00:12:29.354 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_osd_laggy_weight": "0.300000", 2020-01-16T00:12:29.354 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2020-01-16T00:12:29.355 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_osd_max_creating_pgs": "1024", 2020-01-16T00:12:29.355 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_osd_max_initial_pgs": "1024", 2020-01-16T00:12:29.355 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_osd_min_down_reporters": "2", 2020-01-16T00:12:29.355 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_osd_min_in_ratio": "0.750000", 2020-01-16T00:12:29.355 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_osd_min_up_ratio": "0.300000", 2020-01-16T00:12:29.355 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_osd_nearfull_ratio": "0.850000", 2020-01-16T00:12:29.356 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_osd_prime_pg_temp": "true", 2020-01-16T00:12:29.356 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2020-01-16T00:12:29.356 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2020-01-16T00:12:29.356 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_osd_report_timeout": "900", 2020-01-16T00:12:29.356 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_osd_reporter_subtree_level": "osd", 2020-01-16T00:12:29.356 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2020-01-16T00:12:29.356 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_osd_warn_op_age": "32.000000", 2020-01-16T00:12:29.356 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_osdmap_full_prune_enabled": "true", 2020-01-16T00:12:29.357 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_osdmap_full_prune_interval": "2", 2020-01-16T00:12:29.357 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_osdmap_full_prune_min": "15", 2020-01-16T00:12:29.357 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_osdmap_full_prune_txsize": "2", 2020-01-16T00:12:29.357 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2020-01-16T00:12:29.357 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_pg_stuck_threshold": "60", 2020-01-16T00:12:29.357 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2020-01-16T00:12:29.357 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_pg_warn_min_objects": "10000", 2020-01-16T00:12:29.357 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_pg_warn_min_per_osd": "0", 2020-01-16T00:12:29.358 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_pg_warn_min_pool_objects": "1000", 2020-01-16T00:12:29.358 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_pool_quota_crit_threshold": "0", 2020-01-16T00:12:29.358 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_pool_quota_warn_threshold": "0", 2020-01-16T00:12:29.358 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_probe_timeout": "2.000000", 2020-01-16T00:12:29.358 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_reweight_max_change": "0.050000", 2020-01-16T00:12:29.358 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_reweight_max_osds": "4", 2020-01-16T00:12:29.358 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_reweight_min_bytes_per_osd": "10", 2020-01-16T00:12:29.359 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_reweight_min_pgs_per_osd": "4", 2020-01-16T00:12:29.359 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2020-01-16T00:12:29.359 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2020-01-16T00:12:29.359 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2020-01-16T00:12:29.359 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_scrub_interval": "86400", 2020-01-16T00:12:29.359 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_scrub_max_keys": "100", 2020-01-16T00:12:29.359 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_scrub_timeout": "300", 2020-01-16T00:12:29.359 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_session_timeout": "300", 2020-01-16T00:12:29.360 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_smart_report_timeout": "5", 2020-01-16T00:12:29.360 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_stat_smooth_intervals": "6", 2020-01-16T00:12:29.360 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_subscribe_interval": "86400.000000", 2020-01-16T00:12:29.360 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_sync_debug": "false", 2020-01-16T00:12:29.360 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_sync_max_payload_keys": "2000", 2020-01-16T00:12:29.361 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_sync_max_payload_size": "1048576", 2020-01-16T00:12:29.362 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_sync_provider_kill_at": "0", 2020-01-16T00:12:29.362 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_sync_requester_kill_at": "0", 2020-01-16T00:12:29.362 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_sync_timeout": "60.000000", 2020-01-16T00:12:29.362 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_target_pg_per_osd": "100", 2020-01-16T00:12:29.362 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_tick_interval": "5", 2020-01-16T00:12:29.362 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_timecheck_interval": "300.000000", 2020-01-16T00:12:29.362 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_timecheck_skew_interval": "30.000000", 2020-01-16T00:12:29.363 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2020-01-16T00:12:29.363 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2020-01-16T00:12:29.363 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2020-01-16T00:12:29.363 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_warn_on_misplaced": "false", 2020-01-16T00:12:29.363 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2020-01-16T00:12:29.363 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2020-01-16T00:12:29.363 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_warn_on_pool_no_app": "true", 2020-01-16T00:12:29.364 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_warn_on_pool_no_redundancy": "false", 2020-01-16T00:12:29.364 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2020-01-16T00:12:29.364 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2020-01-16T00:12:29.364 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2020-01-16T00:12:29.364 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_warn_on_too_few_osds": "false", 2020-01-16T00:12:29.364 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2020-01-16T00:12:29.364 INFO:teuthology.orchestra.run.smithi123.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2020-01-16T00:12:29.365 INFO:teuthology.orchestra.run.smithi123.stdout: "monmap": "", 2020-01-16T00:12:29.365 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_async_max_op_threads": "5", 2020-01-16T00:12:29.365 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_async_op_threads": "3", 2020-01-16T00:12:29.365 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_async_rdma_buffer_size": "131072", 2020-01-16T00:12:29.365 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_async_rdma_cm": "false", 2020-01-16T00:12:29.365 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_async_rdma_device_name": "", 2020-01-16T00:12:29.366 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_async_rdma_dscp": "96", 2020-01-16T00:12:29.366 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_async_rdma_enable_hugepage": "false", 2020-01-16T00:12:29.366 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_async_rdma_gid_idx": "0", 2020-01-16T00:12:29.366 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_async_rdma_local_gid": "", 2020-01-16T00:12:29.367 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_async_rdma_polling_us": "1000", 2020-01-16T00:12:29.367 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_async_rdma_port_num": "1", 2020-01-16T00:12:29.367 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_async_rdma_receive_buffers": "32768", 2020-01-16T00:12:29.367 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_async_rdma_receive_queue_len": "4096", 2020-01-16T00:12:29.367 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_async_rdma_roce_ver": "1", 2020-01-16T00:12:29.368 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_async_rdma_send_buffers": "1024", 2020-01-16T00:12:29.368 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_async_rdma_sl": "3", 2020-01-16T00:12:29.368 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_async_rdma_support_srq": "true", 2020-01-16T00:12:29.368 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_async_rdma_type": "ib", 2020-01-16T00:12:29.368 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_bind_before_connect": "false", 2020-01-16T00:12:29.369 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_bind_ipv4": "true", 2020-01-16T00:12:29.369 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_bind_ipv6": "false", 2020-01-16T00:12:29.369 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_bind_msgr1": "true", 2020-01-16T00:12:29.369 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_bind_msgr2": "false", 2020-01-16T00:12:29.370 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_bind_port_max": "7300", 2020-01-16T00:12:29.370 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_bind_port_min": "6800", 2020-01-16T00:12:29.370 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_bind_prefer_ipv4": "false", 2020-01-16T00:12:29.370 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_bind_retry_count": "3", 2020-01-16T00:12:29.371 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_bind_retry_delay": "5", 2020-01-16T00:12:29.371 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_blackhole_client": "false", 2020-01-16T00:12:29.371 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_blackhole_mds": "false", 2020-01-16T00:12:29.371 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_blackhole_mgr": "false", 2020-01-16T00:12:29.371 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_blackhole_mon": "false", 2020-01-16T00:12:29.372 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_blackhole_osd": "false", 2020-01-16T00:12:29.372 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_client_mode": "crc secure", 2020-01-16T00:12:29.372 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_cluster_mode": "crc secure", 2020-01-16T00:12:29.372 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_cluster_type": "", 2020-01-16T00:12:29.372 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_connection_idle_timeout": "900", 2020-01-16T00:12:29.373 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_connection_ready_timeout": "10", 2020-01-16T00:12:29.373 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_crc_data": "true", 2020-01-16T00:12:29.373 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_crc_header": "true", 2020-01-16T00:12:29.373 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_die_on_bad_msg": "false", 2020-01-16T00:12:29.373 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_die_on_bug": "true", 2020-01-16T00:12:29.374 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_die_on_old_message": "true", 2020-01-16T00:12:29.374 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_die_on_skipped_message": "false", 2020-01-16T00:12:29.374 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_die_on_unhandled_msg": "false", 2020-01-16T00:12:29.374 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_dispatch_throttle_bytes": "104857600", 2020-01-16T00:12:29.374 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_dpdk_coremask": "0xF", 2020-01-16T00:12:29.375 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_dpdk_debug_allow_loopback": "false", 2020-01-16T00:12:29.375 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2020-01-16T00:12:29.375 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_dpdk_host_ipv4_addr": "", 2020-01-16T00:12:29.375 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_dpdk_hugepages": "", 2020-01-16T00:12:29.375 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_dpdk_hw_flow_control": "true", 2020-01-16T00:12:29.375 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2020-01-16T00:12:29.376 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_dpdk_lro": "true", 2020-01-16T00:12:29.376 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_dpdk_memory_channel": "4", 2020-01-16T00:12:29.376 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2020-01-16T00:12:29.376 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_dpdk_pmd": "", 2020-01-16T00:12:29.376 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_dpdk_port_id": "0", 2020-01-16T00:12:29.376 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2020-01-16T00:12:29.376 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_dump_corrupt_message_level": "1", 2020-01-16T00:12:29.376 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_dump_on_send": "false", 2020-01-16T00:12:29.377 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_initial_backoff": "0.200000", 2020-01-16T00:12:29.377 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_inject_delay_max": "1.000000", 2020-01-16T00:12:29.377 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_inject_delay_msg_type": "", 2020-01-16T00:12:29.377 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_inject_delay_probability": "0.000000", 2020-01-16T00:12:29.377 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_inject_delay_type": "", 2020-01-16T00:12:29.377 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_inject_internal_delays": "0.000000", 2020-01-16T00:12:29.377 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_inject_socket_failures": "2500", 2020-01-16T00:12:29.377 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_learn_addr_from_peer": "true", 2020-01-16T00:12:29.378 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_max_accept_failures": "4", 2020-01-16T00:12:29.378 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_max_backoff": "15.000000", 2020-01-16T00:12:29.378 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_mon_client_mode": "secure crc", 2020-01-16T00:12:29.378 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_mon_cluster_mode": "secure crc", 2020-01-16T00:12:29.378 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_mon_service_mode": "secure crc", 2020-01-16T00:12:29.378 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2020-01-16T00:12:29.378 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_pq_min_cost": "65536", 2020-01-16T00:12:29.379 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_public_type": "", 2020-01-16T00:12:29.379 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_service_mode": "crc secure", 2020-01-16T00:12:29.379 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_tcp_listen_backlog": "512", 2020-01-16T00:12:29.379 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_tcp_nodelay": "true", 2020-01-16T00:12:29.379 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_tcp_prefetch_max_size": "4096", 2020-01-16T00:12:29.379 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_tcp_rcvbuf": "0", 2020-01-16T00:12:29.379 INFO:teuthology.orchestra.run.smithi123.stdout: "ms_type": "async", 2020-01-16T00:12:29.379 INFO:teuthology.orchestra.run.smithi123.stdout: "no_config_file": "false", 2020-01-16T00:12:29.380 INFO:teuthology.orchestra.run.smithi123.stdout: "objecter_completion_locks_per_session": "32", 2020-01-16T00:12:29.380 INFO:teuthology.orchestra.run.smithi123.stdout: "objecter_debug_inject_relock_delay": "false", 2020-01-16T00:12:29.380 INFO:teuthology.orchestra.run.smithi123.stdout: "objecter_inflight_op_bytes": "104857600", 2020-01-16T00:12:29.380 INFO:teuthology.orchestra.run.smithi123.stdout: "objecter_inflight_ops": "1024", 2020-01-16T00:12:29.380 INFO:teuthology.orchestra.run.smithi123.stdout: "objecter_inject_no_watch_ping": "false", 2020-01-16T00:12:29.380 INFO:teuthology.orchestra.run.smithi123.stdout: "objecter_retry_writes_after_first_reply": "false", 2020-01-16T00:12:29.380 INFO:teuthology.orchestra.run.smithi123.stdout: "objecter_tick_interval": "5.000000", 2020-01-16T00:12:29.380 INFO:teuthology.orchestra.run.smithi123.stdout: "objecter_timeout": "10.000000", 2020-01-16T00:12:29.381 INFO:teuthology.orchestra.run.smithi123.stdout: "objectstore_blackhole": "false", 2020-01-16T00:12:29.381 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_agent_delay_time": "5.000000", 2020-01-16T00:12:29.381 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_agent_hist_halflife": "1000", 2020-01-16T00:12:29.381 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_agent_max_low_ops": "2", 2020-01-16T00:12:29.381 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_agent_max_ops": "4", 2020-01-16T00:12:29.381 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_agent_min_evict_effort": "0.100000", 2020-01-16T00:12:29.381 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_agent_quantize_effort": "0.100000", 2020-01-16T00:12:29.382 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_agent_slop": "0.020000", 2020-01-16T00:12:29.382 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_allow_recovery_below_min_size": "true", 2020-01-16T00:12:29.382 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_async_recovery_min_cost": "1", 2020-01-16T00:12:29.382 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_auto_mark_unfound_lost": "false", 2020-01-16T00:12:29.382 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_backfill_retry_interval": "30.000000", 2020-01-16T00:12:29.382 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_backfill_scan_max": "512", 2020-01-16T00:12:29.382 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_backfill_scan_min": "64", 2020-01-16T00:12:29.382 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_backoff_on_degraded": "false", 2020-01-16T00:12:29.383 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_backoff_on_peering": "false", 2020-01-16T00:12:29.383 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_backoff_on_unfound": "true", 2020-01-16T00:12:29.383 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_beacon_report_interval": "300", 2020-01-16T00:12:29.383 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_bench_duration": "30", 2020-01-16T00:12:29.383 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_bench_large_size_max_throughput": "104857600", 2020-01-16T00:12:29.383 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_bench_max_block_size": "67108864", 2020-01-16T00:12:29.384 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_bench_small_size_max_iops": "100", 2020-01-16T00:12:29.384 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_blkin_trace_all": "false", 2020-01-16T00:12:29.384 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2020-01-16T00:12:29.384 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2020-01-16T00:12:29.384 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_check_for_log_corruption": "false", 2020-01-16T00:12:29.385 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_check_max_object_name_len_on_startup": "true", 2020-01-16T00:12:29.385 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas", 2020-01-16T00:12:29.385 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_class_dir": "/usr/lib/rados-classes", 2020-01-16T00:12:29.385 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas", 2020-01-16T00:12:29.385 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_class_update_on_start": "true", 2020-01-16T00:12:29.386 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_client_message_cap": "100", 2020-01-16T00:12:29.386 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_client_message_size_cap": "524288000", 2020-01-16T00:12:29.386 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_client_op_priority": "63", 2020-01-16T00:12:29.386 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_client_watch_timeout": "30", 2020-01-16T00:12:29.386 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_command_max_records": "256", 2020-01-16T00:12:29.386 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_command_thread_suicide_timeout": "900", 2020-01-16T00:12:29.386 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_command_thread_timeout": "600", 2020-01-16T00:12:29.386 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_copyfrom_max_chunk": "8388608", 2020-01-16T00:12:29.387 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_crush_chooseleaf_type": "0", 2020-01-16T00:12:29.387 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_crush_initial_weight": "-1.000000", 2020-01-16T00:12:29.387 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_crush_update_on_start": "true", 2020-01-16T00:12:29.387 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_crush_update_weight_set": "true", 2020-01-16T00:12:29.387 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2020-01-16T00:12:29.387 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2020-01-16T00:12:29.387 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2020-01-16T00:12:29.387 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_debug_drop_ping_duration": "0", 2020-01-16T00:12:29.388 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_debug_drop_ping_probability": "0.000000", 2020-01-16T00:12:29.388 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_debug_feed_pullee": "-1", 2020-01-16T00:12:29.388 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_debug_inject_copyfrom_error": "false", 2020-01-16T00:12:29.388 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2020-01-16T00:12:29.388 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2020-01-16T00:12:29.388 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_debug_misdirected_ops": "false", 2020-01-16T00:12:29.388 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_debug_no_acting_change": "false", 2020-01-16T00:12:29.389 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_debug_no_purge_strays": "false", 2020-01-16T00:12:29.389 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_debug_op_order": "false", 2020-01-16T00:12:29.389 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_debug_pg_log_writeout": "false", 2020-01-16T00:12:29.389 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_debug_pretend_recovery_active": "false", 2020-01-16T00:12:29.389 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_debug_random_push_read_error": "0.000000", 2020-01-16T00:12:29.389 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2020-01-16T00:12:29.389 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_debug_shutdown": "false", 2020-01-16T00:12:29.389 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2020-01-16T00:12:29.390 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2020-01-16T00:12:29.390 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_debug_verify_cached_snaps": "false", 2020-01-16T00:12:29.390 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_debug_verify_missing_on_start": "false", 2020-01-16T00:12:29.390 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_debug_verify_snaps": "false", 2020-01-16T00:12:29.390 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_debug_verify_stray_on_activate": "false", 2020-01-16T00:12:29.390 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_deep_scrub_interval": "604800.000000", 2020-01-16T00:12:29.390 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_deep_scrub_keys": "1024", 2020-01-16T00:12:29.391 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2020-01-16T00:12:29.391 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2020-01-16T00:12:29.391 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2020-01-16T00:12:29.391 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_deep_scrub_stride": "524288", 2020-01-16T00:12:29.391 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2020-01-16T00:12:29.391 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_default_data_pool_replay_window": "5", 2020-01-16T00:12:29.391 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_default_notify_timeout": "30", 2020-01-16T00:12:29.392 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_delete_sleep": "0.000000", 2020-01-16T00:12:29.392 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_delete_sleep_hdd": "5.000000", 2020-01-16T00:12:29.392 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_delete_sleep_hybrid": "2.000000", 2020-01-16T00:12:29.392 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_delete_sleep_ssd": "0.000000", 2020-01-16T00:12:29.392 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_discard_disconnected_ops": "true", 2020-01-16T00:12:29.392 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_enable_op_tracker": "true", 2020-01-16T00:12:29.392 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_erasure_code_plugins": "jerasure lrc isa", 2020-01-16T00:12:29.393 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_failsafe_full_ratio": "0.970000", 2020-01-16T00:12:29.393 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_fast_fail_on_connection_refused": "true", 2020-01-16T00:12:29.393 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_fast_info": "true", 2020-01-16T00:12:29.393 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_fast_shutdown": "true", 2020-01-16T00:12:29.393 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_find_best_info_ignore_history_les": "false", 2020-01-16T00:12:29.393 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_force_auth_primary_missing_objects": "100", 2020-01-16T00:12:29.393 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2020-01-16T00:12:29.393 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_function_tracing": "false", 2020-01-16T00:12:29.394 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_heartbeat_grace": "20", 2020-01-16T00:12:29.394 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_heartbeat_interval": "6", 2020-01-16T00:12:29.394 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2020-01-16T00:12:29.394 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_heartbeat_min_peers": "10", 2020-01-16T00:12:29.394 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_heartbeat_min_size": "2000", 2020-01-16T00:12:29.394 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_heartbeat_stale": "600", 2020-01-16T00:12:29.394 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2020-01-16T00:12:29.394 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_hit_set_max_size": "100000", 2020-01-16T00:12:29.395 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_hit_set_min_size": "1000", 2020-01-16T00:12:29.395 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_hit_set_namespace": ".ceph-internal", 2020-01-16T00:12:29.395 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_ignore_stale_divergent_priors": "false", 2020-01-16T00:12:29.395 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2020-01-16T00:12:29.395 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_inject_failure_on_pg_removal": "false", 2020-01-16T00:12:29.395 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2020-01-16T00:12:29.395 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_journal_flush_on_shutdown": "true", 2020-01-16T00:12:29.396 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_journal_size": "5120", 2020-01-16T00:12:29.396 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_kill_backfill_at": "0", 2020-01-16T00:12:29.396 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_loop_before_reset_tphandle": "64", 2020-01-16T00:12:29.396 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_map_cache_size": "50", 2020-01-16T00:12:29.396 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_map_dedup": "true", 2020-01-16T00:12:29.396 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_map_message_max": "40", 2020-01-16T00:12:29.396 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_map_message_max_bytes": "10485760", 2020-01-16T00:12:29.396 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_map_share_max_epochs": "40", 2020-01-16T00:12:29.397 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_max_attr_name_len": "100", 2020-01-16T00:12:29.397 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_max_attr_size": "0", 2020-01-16T00:12:29.397 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_max_backfills": "1", 2020-01-16T00:12:29.397 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_max_markdown_count": "5", 2020-01-16T00:12:29.397 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_max_markdown_period": "600", 2020-01-16T00:12:29.397 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_max_object_name_len": "2048", 2020-01-16T00:12:29.397 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_max_object_namespace_len": "256", 2020-01-16T00:12:29.398 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_max_object_size": "134217728", 2020-01-16T00:12:29.398 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2020-01-16T00:12:29.398 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_max_omap_entries_per_request": "1024", 2020-01-16T00:12:29.398 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_max_pg_blocked_by": "16", 2020-01-16T00:12:29.398 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_max_pg_log_entries": "2", 2020-01-16T00:12:29.399 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2020-01-16T00:12:29.399 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_max_pgls": "1024", 2020-01-16T00:12:29.399 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_max_push_cost": "8388608", 2020-01-16T00:12:29.399 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_max_push_objects": "10", 2020-01-16T00:12:29.399 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_max_scrubs": "1", 2020-01-16T00:12:29.400 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2020-01-16T00:12:29.400 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_max_trimming_pgs": "2", 2020-01-16T00:12:29.400 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_max_write_op_reply_len": "32", 2020-01-16T00:12:29.400 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_max_write_size": "90", 2020-01-16T00:12:29.400 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2020-01-16T00:12:29.401 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mclock_scheduler_background_best_effort_lim": "999999", 2020-01-16T00:12:29.401 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mclock_scheduler_background_best_effort_res": "1", 2020-01-16T00:12:29.401 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2020-01-16T00:12:29.401 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mclock_scheduler_background_recovery_lim": "999999", 2020-01-16T00:12:29.401 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mclock_scheduler_background_recovery_res": "1", 2020-01-16T00:12:29.402 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2020-01-16T00:12:29.402 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mclock_scheduler_client_lim": "999999", 2020-01-16T00:12:29.402 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mclock_scheduler_client_res": "1", 2020-01-16T00:12:29.402 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mclock_scheduler_client_wgt": "1", 2020-01-16T00:12:29.402 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_memory_base": "805306368", 2020-01-16T00:12:29.403 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_memory_cache_min": "134217728", 2020-01-16T00:12:29.403 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_memory_cache_resize_interval": "1.000000", 2020-01-16T00:12:29.403 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_memory_expected_fragmentation": "0.150000", 2020-01-16T00:12:29.403 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_memory_target": "4294967296", 2020-01-16T00:12:29.404 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2020-01-16T00:12:29.404 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_min_pg_log_entries": "1", 2020-01-16T00:12:29.404 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_min_recovery_priority": "0", 2020-01-16T00:12:29.404 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mon_ack_timeout": "30.000000", 2020-01-16T00:12:29.404 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mon_heartbeat_interval": "30", 2020-01-16T00:12:29.405 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2020-01-16T00:12:29.405 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mon_report_interval": "5", 2020-01-16T00:12:29.405 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mon_report_max_in_flight": "2", 2020-01-16T00:12:29.405 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_mon_shutdown_timeout": "5.000000", 2020-01-16T00:12:29.406 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_num_cache_shards": "32", 2020-01-16T00:12:29.406 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_num_op_tracker_shard": "32", 2020-01-16T00:12:29.406 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_numa_auto_affinity": "true", 2020-01-16T00:12:29.406 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_numa_node": "-1", 2020-01-16T00:12:29.406 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_numa_prefer_iface": "true", 2020-01-16T00:12:29.407 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_object_clean_region_max_num_intervals": "1000", 2020-01-16T00:12:29.407 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_objecter_finishers": "1", 2020-01-16T00:12:29.407 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_objectstore": "bluestore", 2020-01-16T00:12:29.408 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_objectstore_fuse": "false", 2020-01-16T00:12:29.408 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_objectstore_tracing": "false", 2020-01-16T00:12:29.408 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_op_complaint_time": "30.000000", 2020-01-16T00:12:29.408 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_op_history_duration": "600", 2020-01-16T00:12:29.408 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_op_history_size": "20", 2020-01-16T00:12:29.408 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_op_history_slow_op_size": "20", 2020-01-16T00:12:29.409 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2020-01-16T00:12:29.409 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_op_log_threshold": "5", 2020-01-16T00:12:29.409 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_op_num_shards": "0", 2020-01-16T00:12:29.409 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_op_num_shards_hdd": "5", 2020-01-16T00:12:29.410 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_op_num_shards_ssd": "8", 2020-01-16T00:12:29.410 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_op_num_threads_per_shard": "0", 2020-01-16T00:12:29.410 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2020-01-16T00:12:29.410 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2020-01-16T00:12:29.410 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2020-01-16T00:12:29.410 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_op_pq_min_cost": "65536", 2020-01-16T00:12:29.410 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_op_queue": "wpq", 2020-01-16T00:12:29.411 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_op_queue_cut_off": "high", 2020-01-16T00:12:29.411 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_op_thread_suicide_timeout": "150", 2020-01-16T00:12:29.411 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_op_thread_timeout": "15", 2020-01-16T00:12:29.411 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_open_classes_on_start": "true", 2020-01-16T00:12:29.411 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_os_flags": "0", 2020-01-16T00:12:29.411 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_peering_op_priority": "255", 2020-01-16T00:12:29.411 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pg_delete_cost": "1048576", 2020-01-16T00:12:29.412 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pg_delete_priority": "5", 2020-01-16T00:12:29.412 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2020-01-16T00:12:29.412 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2020-01-16T00:12:29.412 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pg_log_dups_tracked": "3000", 2020-01-16T00:12:29.412 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pg_log_trim_max": "10000", 2020-01-16T00:12:29.413 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pg_log_trim_min": "100", 2020-01-16T00:12:29.413 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2020-01-16T00:12:29.413 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pg_object_context_cache_count": "64", 2020-01-16T00:12:29.413 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pg_stat_report_interval_max": "500", 2020-01-16T00:12:29.414 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2020-01-16T00:12:29.414 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pool_default_cache_min_evict_age": "0", 2020-01-16T00:12:29.414 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pool_default_cache_min_flush_age": "0", 2020-01-16T00:12:29.414 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2020-01-16T00:12:29.414 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2020-01-16T00:12:29.414 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2020-01-16T00:12:29.414 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pool_default_crush_rule": "-1", 2020-01-16T00:12:29.414 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pool_default_ec_fast_read": "false", 2020-01-16T00:12:29.415 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd", 2020-01-16T00:12:29.415 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pool_default_flag_hashpspool": "true", 2020-01-16T00:12:29.415 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pool_default_flag_nodelete": "false", 2020-01-16T00:12:29.415 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pool_default_flag_nopgchange": "false", 2020-01-16T00:12:29.415 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pool_default_flag_nosizechange": "false", 2020-01-16T00:12:29.415 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pool_default_flags": "0", 2020-01-16T00:12:29.415 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2020-01-16T00:12:29.415 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pool_default_min_size": "2", 2020-01-16T00:12:29.416 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2020-01-16T00:12:29.416 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pool_default_pg_num": "16", 2020-01-16T00:12:29.416 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pool_default_pgp_num": "0", 2020-01-16T00:12:29.416 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2020-01-16T00:12:29.416 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pool_default_size": "3", 2020-01-16T00:12:29.416 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pool_default_type": "replicated", 2020-01-16T00:12:29.416 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2020-01-16T00:12:29.416 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_pool_use_gmt_hitset": "true", 2020-01-16T00:12:29.417 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_push_per_object_cost": "1000", 2020-01-16T00:12:29.417 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_read_ec_check_for_errors": "false", 2020-01-16T00:12:29.417 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_recover_clone_overlap": "true", 2020-01-16T00:12:29.417 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_recover_clone_overlap_limit": "10", 2020-01-16T00:12:29.417 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_recovery_cost": "20971520", 2020-01-16T00:12:29.417 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_recovery_delay_start": "0.000000", 2020-01-16T00:12:29.417 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_recovery_max_active": "0", 2020-01-16T00:12:29.418 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_recovery_max_active_hdd": "3", 2020-01-16T00:12:29.418 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_recovery_max_active_ssd": "10", 2020-01-16T00:12:29.418 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_recovery_max_chunk": "8388608", 2020-01-16T00:12:29.418 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2020-01-16T00:12:29.418 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_recovery_max_single_start": "1", 2020-01-16T00:12:29.418 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_recovery_op_priority": "3", 2020-01-16T00:12:29.418 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_recovery_op_warn_multiple": "16", 2020-01-16T00:12:29.418 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_recovery_priority": "5", 2020-01-16T00:12:29.419 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_recovery_retry_interval": "30.000000", 2020-01-16T00:12:29.419 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_recovery_sleep": "0.000000", 2020-01-16T00:12:29.419 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_recovery_sleep_hdd": "0.100000", 2020-01-16T00:12:29.419 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2020-01-16T00:12:29.419 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_recovery_sleep_ssd": "0.000000", 2020-01-16T00:12:29.419 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_repair_during_recovery": "false", 2020-01-16T00:12:29.419 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_requested_scrub_priority": "120", 2020-01-16T00:12:29.420 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_rollback_to_cluster_snap": "", 2020-01-16T00:12:29.420 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_scrub_auto_repair": "false", 2020-01-16T00:12:29.420 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_scrub_auto_repair_num_errors": "5", 2020-01-16T00:12:29.420 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_scrub_backoff_ratio": "0.660000", 2020-01-16T00:12:29.420 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_scrub_begin_hour": "0", 2020-01-16T00:12:29.420 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_scrub_begin_week_day": "0", 2020-01-16T00:12:29.420 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_scrub_chunk_max": "25", 2020-01-16T00:12:29.421 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_scrub_chunk_min": "5", 2020-01-16T00:12:29.421 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_scrub_cost": "52428800", 2020-01-16T00:12:29.421 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_scrub_during_recovery": "false", 2020-01-16T00:12:29.421 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_scrub_end_hour": "24", 2020-01-16T00:12:29.421 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_scrub_end_week_day": "7", 2020-01-16T00:12:29.421 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_scrub_extended_sleep": "0.000000", 2020-01-16T00:12:29.421 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2020-01-16T00:12:29.422 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_scrub_invalid_stats": "true", 2020-01-16T00:12:29.422 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_scrub_load_threshold": "0.500000", 2020-01-16T00:12:29.422 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_scrub_max_interval": "604800.000000", 2020-01-16T00:12:29.422 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_scrub_max_preemptions": "5", 2020-01-16T00:12:29.422 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_scrub_min_interval": "86400.000000", 2020-01-16T00:12:29.422 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_scrub_priority": "5", 2020-01-16T00:12:29.422 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_scrub_sleep": "0.000000", 2020-01-16T00:12:29.423 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_shutdown_pgref_assert": "false", 2020-01-16T00:12:29.423 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_skip_data_digest": "false", 2020-01-16T00:12:29.423 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_smart_report_timeout": "5", 2020-01-16T00:12:29.423 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_snap_trim_cost": "1048576", 2020-01-16T00:12:29.423 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_snap_trim_priority": "5", 2020-01-16T00:12:29.423 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_snap_trim_sleep": "0.000000", 2020-01-16T00:12:29.424 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2020-01-16T00:12:29.424 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2020-01-16T00:12:29.424 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2020-01-16T00:12:29.424 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_stats_ack_timeout_decay": "0.900000", 2020-01-16T00:12:29.424 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_stats_ack_timeout_factor": "2.000000", 2020-01-16T00:12:29.424 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_target_transaction_size": "30", 2020-01-16T00:12:29.424 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_tier_default_cache_hit_set_count": "4", 2020-01-16T00:12:29.425 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2020-01-16T00:12:29.425 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2020-01-16T00:12:29.425 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2020-01-16T00:12:29.425 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2020-01-16T00:12:29.425 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2020-01-16T00:12:29.425 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2020-01-16T00:12:29.425 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_tier_default_cache_mode": "writeback", 2020-01-16T00:12:29.426 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2020-01-16T00:12:29.426 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_tier_promote_max_objects_sec": "25", 2020-01-16T00:12:29.426 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_tracing": "false", 2020-01-16T00:12:29.426 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_use_stale_snap": "false", 2020-01-16T00:12:29.426 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2020-01-16T00:12:29.426 INFO:teuthology.orchestra.run.smithi123.stdout: "osdc_blkin_trace_all": "false", 2020-01-16T00:12:29.426 INFO:teuthology.orchestra.run.smithi123.stdout: "paxos_kill_at": "0", 2020-01-16T00:12:29.427 INFO:teuthology.orchestra.run.smithi123.stdout: "paxos_max_join_drift": "10", 2020-01-16T00:12:29.427 INFO:teuthology.orchestra.run.smithi123.stdout: "paxos_min": "500", 2020-01-16T00:12:29.427 INFO:teuthology.orchestra.run.smithi123.stdout: "paxos_min_wait": "0.050000", 2020-01-16T00:12:29.427 INFO:teuthology.orchestra.run.smithi123.stdout: "paxos_propose_interval": "1.000000", 2020-01-16T00:12:29.427 INFO:teuthology.orchestra.run.smithi123.stdout: "paxos_service_trim_max": "500", 2020-01-16T00:12:29.427 INFO:teuthology.orchestra.run.smithi123.stdout: "paxos_service_trim_min": "10", 2020-01-16T00:12:29.427 INFO:teuthology.orchestra.run.smithi123.stdout: "paxos_stash_full_interval": "25", 2020-01-16T00:12:29.427 INFO:teuthology.orchestra.run.smithi123.stdout: "paxos_trim_max": "500", 2020-01-16T00:12:29.428 INFO:teuthology.orchestra.run.smithi123.stdout: "paxos_trim_min": "250", 2020-01-16T00:12:29.428 INFO:teuthology.orchestra.run.smithi123.stdout: "perf": "true", 2020-01-16T00:12:29.428 INFO:teuthology.orchestra.run.smithi123.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2020-01-16T00:12:29.428 INFO:teuthology.orchestra.run.smithi123.stdout: "plugin_crypto_accelerator": "crypto_isal", 2020-01-16T00:12:29.428 INFO:teuthology.orchestra.run.smithi123.stdout: "plugin_dir": "/usr/lib/ceph", 2020-01-16T00:12:29.428 INFO:teuthology.orchestra.run.smithi123.stdout: "prompt": "\u001b[01;33mCephFS:~\u001b[96m/\u001b[0m\u001b[01;33m>>>\u001b[00m ", 2020-01-16T00:12:29.428 INFO:teuthology.orchestra.run.smithi123.stdout: "public_addr": "-", 2020-01-16T00:12:29.428 INFO:teuthology.orchestra.run.smithi123.stdout: "public_addrv": "", 2020-01-16T00:12:29.429 INFO:teuthology.orchestra.run.smithi123.stdout: "public_bind_addr": "-", 2020-01-16T00:12:29.429 INFO:teuthology.orchestra.run.smithi123.stdout: "public_network": "", 2020-01-16T00:12:29.429 INFO:teuthology.orchestra.run.smithi123.stdout: "public_network_interface": "", 2020-01-16T00:12:29.429 INFO:teuthology.orchestra.run.smithi123.stdout: "qat_compressor_enabled": "false", 2020-01-16T00:12:29.429 INFO:teuthology.orchestra.run.smithi123.stdout: "quiet": "false", 2020-01-16T00:12:29.430 INFO:teuthology.orchestra.run.smithi123.stdout: "rados_mon_op_timeout": "0.000000", 2020-01-16T00:12:29.430 INFO:teuthology.orchestra.run.smithi123.stdout: "rados_osd_op_timeout": "0.000000", 2020-01-16T00:12:29.430 INFO:teuthology.orchestra.run.smithi123.stdout: "rados_tracing": "false", 2020-01-16T00:12:29.430 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_atime_update_interval": "60", 2020-01-16T00:12:29.431 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2020-01-16T00:12:29.431 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_balance_parent_reads": "false", 2020-01-16T00:12:29.431 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_balance_snap_reads": "false", 2020-01-16T00:12:29.431 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_blacklist_expire_seconds": "0", 2020-01-16T00:12:29.431 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_blacklist_on_break_lock": "true", 2020-01-16T00:12:29.432 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_blkin_trace_all": "false", 2020-01-16T00:12:29.432 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_cache": "true", 2020-01-16T00:12:29.432 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_cache_block_writes_upfront": "false", 2020-01-16T00:12:29.432 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_cache_max_dirty": "25165824", 2020-01-16T00:12:29.432 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_cache_max_dirty_age": "1.000000", 2020-01-16T00:12:29.433 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_cache_max_dirty_object": "0", 2020-01-16T00:12:29.433 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_cache_policy": "writearound", 2020-01-16T00:12:29.433 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_cache_size": "33554432", 2020-01-16T00:12:29.433 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_cache_target_dirty": "16777216", 2020-01-16T00:12:29.433 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_cache_writethrough_until_flush": "true", 2020-01-16T00:12:29.434 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_clone_copy_on_read": "false", 2020-01-16T00:12:29.434 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_concurrent_management_ops": "10", 2020-01-16T00:12:29.434 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_default_clone_format": "auto", 2020-01-16T00:12:29.435 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_default_data_pool": "", 2020-01-16T00:12:29.435 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_default_features": "61", 2020-01-16T00:12:29.435 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_default_format": "2", 2020-01-16T00:12:29.435 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_default_map_options": "", 2020-01-16T00:12:29.435 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_default_order": "22", 2020-01-16T00:12:29.436 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_default_pool": "rbd", 2020-01-16T00:12:29.436 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_default_stripe_count": "0", 2020-01-16T00:12:29.436 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_default_stripe_unit": "0", 2020-01-16T00:12:29.436 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_disable_zero_copy_writes": "true", 2020-01-16T00:12:29.436 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_discard_granularity_bytes": "65536", 2020-01-16T00:12:29.437 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_discard_on_zeroed_write_same": "true", 2020-01-16T00:12:29.437 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_enable_alloc_hint": "true", 2020-01-16T00:12:29.437 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_io_scheduler": "simple", 2020-01-16T00:12:29.437 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2020-01-16T00:12:29.438 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_journal_commit_age": "5.000000", 2020-01-16T00:12:29.438 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2020-01-16T00:12:29.438 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_journal_max_payload_bytes": "16384", 2020-01-16T00:12:29.438 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_journal_object_flush_age": "0.000000", 2020-01-16T00:12:29.438 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_journal_object_flush_bytes": "1048576", 2020-01-16T00:12:29.439 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_journal_object_flush_interval": "0", 2020-01-16T00:12:29.439 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2020-01-16T00:12:29.439 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2020-01-16T00:12:29.439 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_journal_order": "24", 2020-01-16T00:12:29.439 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_journal_pool": "", 2020-01-16T00:12:29.440 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_journal_splay_width": "4", 2020-01-16T00:12:29.440 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_localize_parent_reads": "false", 2020-01-16T00:12:29.441 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_localize_snap_reads": "false", 2020-01-16T00:12:29.441 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2020-01-16T00:12:29.441 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2020-01-16T00:12:29.441 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2020-01-16T00:12:29.441 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2020-01-16T00:12:29.442 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2020-01-16T00:12:29.442 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2020-01-16T00:12:29.442 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_mirror_image_policy_type": "simple", 2020-01-16T00:12:29.442 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2020-01-16T00:12:29.442 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_mirror_image_state_check_interval": "30", 2020-01-16T00:12:29.442 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2020-01-16T00:12:29.442 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2020-01-16T00:12:29.442 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2020-01-16T00:12:29.443 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2020-01-16T00:12:29.443 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2020-01-16T00:12:29.443 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_mirror_memory_autotune": "true", 2020-01-16T00:12:29.443 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_mirror_memory_base": "805306368", 2020-01-16T00:12:29.443 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2020-01-16T00:12:29.443 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_mirror_memory_cache_min": "134217728", 2020-01-16T00:12:29.443 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2020-01-16T00:12:29.443 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2020-01-16T00:12:29.444 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_mirror_memory_target": "4294967296", 2020-01-16T00:12:29.444 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_mirror_perf_stats_prio": "5", 2020-01-16T00:12:29.444 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2020-01-16T00:12:29.444 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2020-01-16T00:12:29.444 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_mirroring_delete_delay": "0", 2020-01-16T00:12:29.444 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_mirroring_max_mirroring_snapshots": "3", 2020-01-16T00:12:29.444 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_mirroring_replay_delay": "0", 2020-01-16T00:12:29.444 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2020-01-16T00:12:29.445 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2020-01-16T00:12:29.445 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_move_to_trash_on_remove": "false", 2020-01-16T00:12:29.445 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2020-01-16T00:12:29.445 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_mtime_update_interval": "60", 2020-01-16T00:12:29.445 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_non_blocking_aio": "true", 2020-01-16T00:12:29.445 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_op_thread_timeout": "60", 2020-01-16T00:12:29.445 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_op_threads": "1", 2020-01-16T00:12:29.446 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_parent_cache_enabled": "false", 2020-01-16T00:12:29.446 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_qos_bps_burst": "0", 2020-01-16T00:12:29.446 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_qos_bps_limit": "0", 2020-01-16T00:12:29.446 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_qos_iops_burst": "0", 2020-01-16T00:12:29.446 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_qos_iops_limit": "0", 2020-01-16T00:12:29.446 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_qos_read_bps_burst": "0", 2020-01-16T00:12:29.447 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_qos_read_bps_limit": "0", 2020-01-16T00:12:29.447 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_qos_read_iops_burst": "0", 2020-01-16T00:12:29.447 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_qos_read_iops_limit": "0", 2020-01-16T00:12:29.447 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_qos_schedule_tick_min": "50", 2020-01-16T00:12:29.447 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_qos_write_bps_burst": "0", 2020-01-16T00:12:29.447 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_qos_write_bps_limit": "0", 2020-01-16T00:12:29.447 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_qos_write_iops_burst": "0", 2020-01-16T00:12:29.447 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_qos_write_iops_limit": "0", 2020-01-16T00:12:29.448 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2020-01-16T00:12:29.448 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_readahead_max_bytes": "524288", 2020-01-16T00:12:29.448 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_readahead_trigger_requests": "10", 2020-01-16T00:12:29.448 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_request_timed_out_seconds": "30", 2020-01-16T00:12:29.448 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_rwl_enabled": "false", 2020-01-16T00:12:29.448 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_rwl_log_periodic_stats": "false", 2020-01-16T00:12:29.448 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_rwl_path": "/tmp", 2020-01-16T00:12:29.448 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_rwl_size": "1073741824", 2020-01-16T00:12:29.449 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_skip_partial_discard": "true", 2020-01-16T00:12:29.449 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2020-01-16T00:12:29.449 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_tracing": "false", 2020-01-16T00:12:29.449 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_validate_names": "true", 2020-01-16T00:12:29.449 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd_validate_pool": "true", 2020-01-16T00:12:29.449 INFO:teuthology.orchestra.run.smithi123.stdout: "restapi_base_url": "", 2020-01-16T00:12:29.449 INFO:teuthology.orchestra.run.smithi123.stdout: "restapi_log_level": "", 2020-01-16T00:12:29.449 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_acl_grants_max_num": "100", 2020-01-16T00:12:29.450 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_admin_entry": "admin", 2020-01-16T00:12:29.450 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_barbican_url": "", 2020-01-16T00:12:29.450 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2020-01-16T00:12:29.450 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_bucket_default_quota_max_size": "-1", 2020-01-16T00:12:29.450 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_bucket_index_max_aio": "128", 2020-01-16T00:12:29.450 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_bucket_quota_cache_size": "10000", 2020-01-16T00:12:29.451 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_bucket_quota_soft_threshold": "0.950000", 2020-01-16T00:12:29.451 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_bucket_quota_ttl": "600", 2020-01-16T00:12:29.451 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_cache_enabled": "true", 2020-01-16T00:12:29.451 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_cache_expiry_interval": "900", 2020-01-16T00:12:29.451 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_cache_lru_size": "10000", 2020-01-16T00:12:29.451 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_content_length_compat": "false", 2020-01-16T00:12:29.451 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_copy_obj_progress": "true", 2020-01-16T00:12:29.451 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2020-01-16T00:12:29.452 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_cors_rules_max_num": "100", 2020-01-16T00:12:29.452 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_cross_domain_policy": "", 2020-01-16T00:12:29.452 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_crypt_default_encryption_key": "", 2020-01-16T00:12:29.452 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_crypt_require_ssl": "true", 2020-01-16T00:12:29.452 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2020-01-16T00:12:29.452 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2020-01-16T00:12:29.452 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_crypt_suppress_logs": "true", 2020-01-16T00:12:29.452 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_crypt_vault_addr": "", 2020-01-16T00:12:29.453 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_crypt_vault_auth": "token", 2020-01-16T00:12:29.453 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_crypt_vault_namespace": "", 2020-01-16T00:12:29.453 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_crypt_vault_prefix": "", 2020-01-16T00:12:29.453 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_crypt_vault_secret_engine": "transit", 2020-01-16T00:12:29.453 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_crypt_vault_token_file": "", 2020-01-16T00:12:29.454 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_curl_low_speed_limit": "1024", 2020-01-16T00:12:29.454 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_curl_low_speed_time": "300", 2020-01-16T00:12:29.454 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_curl_wait_timeout_ms": "1000", 2020-01-16T00:12:29.454 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2020-01-16T00:12:29.454 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_data_log_changes_size": "1000", 2020-01-16T00:12:29.454 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_data_log_num_shards": "128", 2020-01-16T00:12:29.454 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_data_log_obj_prefix": "data_log", 2020-01-16T00:12:29.454 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_data_log_window": "30", 2020-01-16T00:12:29.455 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_data_notify_interval_msec": "200", 2020-01-16T00:12:29.455 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_default_realm_info_oid": "default.realm", 2020-01-16T00:12:29.455 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_default_region_info_oid": "default.region", 2020-01-16T00:12:29.455 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_default_zone_info_oid": "default.zone", 2020-01-16T00:12:29.455 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2020-01-16T00:12:29.455 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_defer_to_bucket_acls": "", 2020-01-16T00:12:29.455 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_delete_multi_obj_max_num": "1000", 2020-01-16T00:12:29.456 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_dmclock_admin_lim": "0.000000", 2020-01-16T00:12:29.456 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_dmclock_admin_res": "100.000000", 2020-01-16T00:12:29.456 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2020-01-16T00:12:29.456 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_dmclock_auth_lim": "0.000000", 2020-01-16T00:12:29.456 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_dmclock_auth_res": "200.000000", 2020-01-16T00:12:29.456 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2020-01-16T00:12:29.456 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_dmclock_data_lim": "0.000000", 2020-01-16T00:12:29.456 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_dmclock_data_res": "500.000000", 2020-01-16T00:12:29.456 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_dmclock_data_wgt": "500.000000", 2020-01-16T00:12:29.457 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2020-01-16T00:12:29.457 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_dmclock_metadata_res": "500.000000", 2020-01-16T00:12:29.457 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2020-01-16T00:12:29.457 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_dns_name": "", 2020-01-16T00:12:29.457 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_dns_s3website_name": "", 2020-01-16T00:12:29.457 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_dynamic_resharding": "true", 2020-01-16T00:12:29.457 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, pubsub", 2020-01-16T00:12:29.458 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_enable_gc_threads": "true", 2020-01-16T00:12:29.458 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_enable_lc_threads": "true", 2020-01-16T00:12:29.458 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_enable_ops_log": "false", 2020-01-16T00:12:29.458 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_enable_quota_threads": "true", 2020-01-16T00:12:29.458 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_enable_static_website": "false", 2020-01-16T00:12:29.458 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_enable_usage_log": "false", 2020-01-16T00:12:29.458 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_enforce_swift_acls": "true", 2020-01-16T00:12:29.458 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_exit_timeout_secs": "120", 2020-01-16T00:12:29.459 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_expose_bucket": "false", 2020-01-16T00:12:29.459 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_extended_http_attrs": "", 2020-01-16T00:12:29.459 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_fcgi_socket_backlog": "1024", 2020-01-16T00:12:29.459 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_frontends": "beast port=7480", 2020-01-16T00:12:29.459 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_gc_max_concurrent_io": "10", 2020-01-16T00:12:29.459 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_gc_max_deferred": "50", 2020-01-16T00:12:29.459 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2020-01-16T00:12:29.459 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_gc_max_objs": "32", 2020-01-16T00:12:29.460 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_gc_max_queue_size": "134213632", 2020-01-16T00:12:29.460 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_gc_max_trim_chunk": "16", 2020-01-16T00:12:29.460 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_gc_obj_min_wait": "7200", 2020-01-16T00:12:29.460 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_gc_processor_max_time": "3600", 2020-01-16T00:12:29.461 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_gc_processor_period": "3600", 2020-01-16T00:12:29.461 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_get_obj_max_req_size": "4194304", 2020-01-16T00:12:29.461 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_get_obj_window_size": "16777216", 2020-01-16T00:12:29.461 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_healthcheck_disabling_path": "", 2020-01-16T00:12:29.461 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_host": "", 2020-01-16T00:12:29.461 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_ignore_get_invalid_range": "false", 2020-01-16T00:12:29.461 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_init_timeout": "300", 2020-01-16T00:12:29.462 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2020-01-16T00:12:29.462 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_keystone_accepted_admin_roles": "", 2020-01-16T00:12:29.462 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2020-01-16T00:12:29.462 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_keystone_admin_domain": "", 2020-01-16T00:12:29.462 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_keystone_admin_password": "", 2020-01-16T00:12:29.462 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_keystone_admin_password_path": "", 2020-01-16T00:12:29.462 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_keystone_admin_project": "", 2020-01-16T00:12:29.462 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_keystone_admin_tenant": "", 2020-01-16T00:12:29.463 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_keystone_admin_token": "", 2020-01-16T00:12:29.463 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_keystone_admin_token_path": "", 2020-01-16T00:12:29.463 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_keystone_admin_user": "", 2020-01-16T00:12:29.463 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_keystone_api_version": "2", 2020-01-16T00:12:29.463 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_keystone_barbican_domain": "", 2020-01-16T00:12:29.464 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_keystone_barbican_password": "", 2020-01-16T00:12:29.464 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_keystone_barbican_project": "", 2020-01-16T00:12:29.464 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_keystone_barbican_tenant": "", 2020-01-16T00:12:29.464 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_keystone_barbican_user": "", 2020-01-16T00:12:29.464 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_keystone_implicit_tenants": "false", 2020-01-16T00:12:29.464 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_keystone_token_cache_size": "10000", 2020-01-16T00:12:29.464 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_keystone_url": "", 2020-01-16T00:12:29.464 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_keystone_verify_ssl": "true", 2020-01-16T00:12:29.465 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_lc_debug_interval": "-1", 2020-01-16T00:12:29.465 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_lc_lock_max_time": "60", 2020-01-16T00:12:29.465 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_lc_max_objs": "32", 2020-01-16T00:12:29.465 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_lc_max_rules": "1000", 2020-01-16T00:12:29.465 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_lc_thread_delay": "0", 2020-01-16T00:12:29.465 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2020-01-16T00:12:29.465 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_ldap_dnattr": "uid", 2020-01-16T00:12:29.465 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2020-01-16T00:12:29.466 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_ldap_searchfilter": "", 2020-01-16T00:12:29.466 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2020-01-16T00:12:29.466 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_ldap_uri": "ldaps://", 2020-01-16T00:12:29.466 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2020-01-16T00:12:29.466 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_list_bucket_min_readahead": "1000", 2020-01-16T00:12:29.466 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_list_buckets_max_chunk": "1000", 2020-01-16T00:12:29.466 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_log_http_headers": "", 2020-01-16T00:12:29.466 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_log_nonexistent_bucket": "false", 2020-01-16T00:12:29.467 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2020-01-16T00:12:29.467 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_log_object_name_utc": "false", 2020-01-16T00:12:29.467 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_max_attr_name_len": "0", 2020-01-16T00:12:29.467 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_max_attr_size": "0", 2020-01-16T00:12:29.467 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_max_attrs_num_in_req": "0", 2020-01-16T00:12:29.467 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_max_chunk_size": "4194304", 2020-01-16T00:12:29.467 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_max_concurrent_requests": "1024", 2020-01-16T00:12:29.468 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_max_dynamic_shards": "1999", 2020-01-16T00:12:29.468 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_max_listing_results": "1000", 2020-01-16T00:12:29.468 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_max_notify_retries": "3", 2020-01-16T00:12:29.468 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_max_objs_per_shard": "100000", 2020-01-16T00:12:29.468 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_max_put_param_size": "1048576", 2020-01-16T00:12:29.468 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_max_put_size": "5368709120", 2020-01-16T00:12:29.468 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_max_slo_entries": "1000", 2020-01-16T00:12:29.468 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_md_log_max_shards": "64", 2020-01-16T00:12:29.469 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_md_notify_interval_msec": "200", 2020-01-16T00:12:29.469 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_mime_types_file": "/etc/mime.types", 2020-01-16T00:12:29.469 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_mp_lock_max_time": "600", 2020-01-16T00:12:29.469 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_multipart_min_part_size": "5242880", 2020-01-16T00:12:29.469 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_multipart_part_upload_limit": "10000", 2020-01-16T00:12:29.469 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_nfs_fhcache_partitions": "3", 2020-01-16T00:12:29.469 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_nfs_fhcache_size": "2017", 2020-01-16T00:12:29.469 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2020-01-16T00:12:29.470 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_nfs_lru_lanes": "5", 2020-01-16T00:12:29.470 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_nfs_max_gc": "300", 2020-01-16T00:12:29.470 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_nfs_namespace_expire_secs": "300", 2020-01-16T00:12:29.470 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_nfs_s3_fast_attrs": "false", 2020-01-16T00:12:29.471 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_nfs_write_completion_interval_s": "10", 2020-01-16T00:12:29.471 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_num_async_rados_threads": "32", 2020-01-16T00:12:29.471 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_num_control_oids": "8", 2020-01-16T00:12:29.471 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_num_rados_handles": "1", 2020-01-16T00:12:29.471 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_obj_stripe_size": "4194304", 2020-01-16T00:12:29.472 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_obj_tombstone_cache_size": "1000", 2020-01-16T00:12:29.472 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_objexp_chunk_size": "100", 2020-01-16T00:12:29.472 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_objexp_gc_interval": "600", 2020-01-16T00:12:29.472 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_objexp_hints_num_shards": "127", 2020-01-16T00:12:29.472 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_olh_pending_timeout_sec": "3600", 2020-01-16T00:12:29.472 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_op_thread_suicide_timeout": "0", 2020-01-16T00:12:29.472 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_op_thread_timeout": "600", 2020-01-16T00:12:29.473 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_op_tracing": "false", 2020-01-16T00:12:29.473 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_opa_token": "", 2020-01-16T00:12:29.473 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_opa_url": "", 2020-01-16T00:12:29.473 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_opa_verify_ssl": "true", 2020-01-16T00:12:29.473 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_ops_log_data_backlog": "5242880", 2020-01-16T00:12:29.473 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_ops_log_rados": "true", 2020-01-16T00:12:29.473 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_ops_log_socket_path": "", 2020-01-16T00:12:29.473 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_override_bucket_index_max_shards": "0", 2020-01-16T00:12:29.474 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2020-01-16T00:12:29.474 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_period_push_interval": "2.000000", 2020-01-16T00:12:29.474 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_period_push_interval_max": "30.000000", 2020-01-16T00:12:29.474 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_period_root_pool": ".rgw.root", 2020-01-16T00:12:29.474 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_port": "", 2020-01-16T00:12:29.474 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_print_continue": "true", 2020-01-16T00:12:29.474 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_print_prohibited_content_length": "false", 2020-01-16T00:12:29.474 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_put_obj_max_window_size": "67108864", 2020-01-16T00:12:29.475 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_put_obj_min_window_size": "16777216", 2020-01-16T00:12:29.475 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2020-01-16T00:12:29.475 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_rados_pool_pg_num_min": "8", 2020-01-16T00:12:29.475 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_rados_pool_recovery_priority": "5", 2020-01-16T00:12:29.475 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_rados_tracing": "false", 2020-01-16T00:12:29.475 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_realm": "", 2020-01-16T00:12:29.475 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_realm_root_pool": ".rgw.root", 2020-01-16T00:12:29.475 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_region": "", 2020-01-16T00:12:29.476 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_region_root_pool": ".rgw.root", 2020-01-16T00:12:29.476 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_relaxed_region_enforcement": "false", 2020-01-16T00:12:29.476 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_relaxed_s3_bucket_names": "false", 2020-01-16T00:12:29.476 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2020-01-16T00:12:29.476 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_request_uri": "", 2020-01-16T00:12:29.476 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_reshard_batch_size": "64", 2020-01-16T00:12:29.476 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_reshard_bucket_lock_duration": "360", 2020-01-16T00:12:29.477 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_reshard_max_aio": "128", 2020-01-16T00:12:29.477 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_reshard_num_logs": "16", 2020-01-16T00:12:29.477 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_reshard_thread_interval": "600", 2020-01-16T00:12:29.477 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_resolve_cname": "false", 2020-01-16T00:12:29.477 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_rest_getusage_op_compat": "false", 2020-01-16T00:12:29.477 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_run_sync_thread": "true", 2020-01-16T00:12:29.477 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_s3_auth_order": "sts, external, local", 2020-01-16T00:12:29.477 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_s3_auth_use_keystone": "false", 2020-01-16T00:12:29.478 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_s3_auth_use_ldap": "false", 2020-01-16T00:12:29.478 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_s3_auth_use_rados": "true", 2020-01-16T00:12:29.478 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_s3_auth_use_sts": "false", 2020-01-16T00:12:29.479 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_s3_success_create_obj_status": "0", 2020-01-16T00:12:29.479 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_safe_max_objects_per_shard": "102400", 2020-01-16T00:12:29.479 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_scheduler_type": "throttler", 2020-01-16T00:12:29.479 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_script_uri": "", 2020-01-16T00:12:29.479 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_service_provider_name": "", 2020-01-16T00:12:29.480 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_shard_warning_threshold": "90.000000", 2020-01-16T00:12:29.480 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_socket_path": "", 2020-01-16T00:12:29.480 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_sts_client_id": "", 2020-01-16T00:12:29.480 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_sts_client_secret": "", 2020-01-16T00:12:29.480 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_sts_entry": "sts", 2020-01-16T00:12:29.480 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_sts_key": "sts", 2020-01-16T00:12:29.480 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_sts_max_session_duration": "43200", 2020-01-16T00:12:29.480 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_sts_token_introspection_url": "", 2020-01-16T00:12:29.481 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_swift_account_in_url": "false", 2020-01-16T00:12:29.481 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_swift_auth_entry": "auth", 2020-01-16T00:12:29.481 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_swift_auth_url": "", 2020-01-16T00:12:29.481 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_swift_custom_header": "", 2020-01-16T00:12:29.481 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_swift_enforce_content_length": "false", 2020-01-16T00:12:29.481 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_swift_need_stats": "true", 2020-01-16T00:12:29.482 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_swift_tenant_name": "", 2020-01-16T00:12:29.482 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_swift_token_expiration": "86400", 2020-01-16T00:12:29.482 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_swift_url": "", 2020-01-16T00:12:29.482 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_swift_url_prefix": "swift", 2020-01-16T00:12:29.482 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_swift_versioning_enabled": "false", 2020-01-16T00:12:29.482 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2020-01-16T00:12:29.482 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_sync_lease_period": "120", 2020-01-16T00:12:29.482 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2020-01-16T00:12:29.483 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_sync_log_trim_interval": "1200", 2020-01-16T00:12:29.483 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_sync_log_trim_max_buckets": "16", 2020-01-16T00:12:29.483 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2020-01-16T00:12:29.483 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2020-01-16T00:12:29.483 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_sync_trace_history_size": "4096", 2020-01-16T00:12:29.483 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_sync_trace_per_node_log_size": "32", 2020-01-16T00:12:29.483 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2020-01-16T00:12:29.484 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_thread_pool_size": "512", 2020-01-16T00:12:29.484 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_torrent_comment": "", 2020-01-16T00:12:29.484 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_torrent_createby": "", 2020-01-16T00:12:29.484 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_torrent_encoding": "", 2020-01-16T00:12:29.484 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_torrent_flag": "false", 2020-01-16T00:12:29.484 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_torrent_origin": "", 2020-01-16T00:12:29.484 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_torrent_sha_unit": "524288", 2020-01-16T00:12:29.484 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_torrent_tracker": "", 2020-01-16T00:12:29.485 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_trust_forwarded_https": "false", 2020-01-16T00:12:29.485 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_usage_log_flush_threshold": "1024", 2020-01-16T00:12:29.485 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_usage_log_tick_interval": "30", 2020-01-16T00:12:29.485 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_usage_max_shards": "32", 2020-01-16T00:12:29.485 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_usage_max_user_shards": "1", 2020-01-16T00:12:29.485 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_use_opa_authz": "false", 2020-01-16T00:12:29.485 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_user_default_quota_max_objects": "-1", 2020-01-16T00:12:29.485 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_user_default_quota_max_size": "-1", 2020-01-16T00:12:29.486 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_user_max_buckets": "1000", 2020-01-16T00:12:29.486 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2020-01-16T00:12:29.486 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_user_quota_sync_idle_users": "false", 2020-01-16T00:12:29.486 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_user_quota_sync_interval": "86400", 2020-01-16T00:12:29.486 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_user_quota_sync_wait_time": "86400", 2020-01-16T00:12:29.486 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_user_unique_email": "true", 2020-01-16T00:12:29.486 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_verify_ssl": "true", 2020-01-16T00:12:29.486 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_website_routing_rules_max_num": "50", 2020-01-16T00:12:29.487 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_zone": "", 2020-01-16T00:12:29.487 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_zone_root_pool": ".rgw.root", 2020-01-16T00:12:29.487 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_zonegroup": "", 2020-01-16T00:12:29.487 INFO:teuthology.orchestra.run.smithi123.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2020-01-16T00:12:29.487 INFO:teuthology.orchestra.run.smithi123.stdout: "rocksdb_block_size": "4096", 2020-01-16T00:12:29.487 INFO:teuthology.orchestra.run.smithi123.stdout: "rocksdb_bloom_bits_per_key": "20", 2020-01-16T00:12:29.487 INFO:teuthology.orchestra.run.smithi123.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2020-01-16T00:12:29.487 INFO:teuthology.orchestra.run.smithi123.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "true", 2020-01-16T00:12:29.488 INFO:teuthology.orchestra.run.smithi123.stdout: "rocksdb_cache_row_ratio": "0.000000", 2020-01-16T00:12:29.488 INFO:teuthology.orchestra.run.smithi123.stdout: "rocksdb_cache_shard_bits": "4", 2020-01-16T00:12:29.488 INFO:teuthology.orchestra.run.smithi123.stdout: "rocksdb_cache_size": "536870912", 2020-01-16T00:12:29.488 INFO:teuthology.orchestra.run.smithi123.stdout: "rocksdb_cache_type": "binned_lru", 2020-01-16T00:12:29.488 INFO:teuthology.orchestra.run.smithi123.stdout: "rocksdb_collect_compaction_stats": "false", 2020-01-16T00:12:29.488 INFO:teuthology.orchestra.run.smithi123.stdout: "rocksdb_collect_extended_stats": "false", 2020-01-16T00:12:29.488 INFO:teuthology.orchestra.run.smithi123.stdout: "rocksdb_collect_memory_stats": "false", 2020-01-16T00:12:29.488 INFO:teuthology.orchestra.run.smithi123.stdout: "rocksdb_delete_range_threshold": "1024", 2020-01-16T00:12:29.489 INFO:teuthology.orchestra.run.smithi123.stdout: "rocksdb_index_type": "binary_search", 2020-01-16T00:12:29.489 INFO:teuthology.orchestra.run.smithi123.stdout: "rocksdb_log_to_ceph_log": "true", 2020-01-16T00:12:29.489 INFO:teuthology.orchestra.run.smithi123.stdout: "rocksdb_metadata_block_size": "4096", 2020-01-16T00:12:29.489 INFO:teuthology.orchestra.run.smithi123.stdout: "rocksdb_partition_filters": "false", 2020-01-16T00:12:29.489 INFO:teuthology.orchestra.run.smithi123.stdout: "rocksdb_perf": "false", 2020-01-16T00:12:29.489 INFO:teuthology.orchestra.run.smithi123.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2020-01-16T00:12:29.489 INFO:teuthology.orchestra.run.smithi123.stdout: "rotating_keys_bootstrap_timeout": "30", 2020-01-16T00:12:29.489 INFO:teuthology.orchestra.run.smithi123.stdout: "rotating_keys_renewal_timeout": "10", 2020-01-16T00:12:29.490 INFO:teuthology.orchestra.run.smithi123.stdout: "run_dir": "/var/run/ceph", 2020-01-16T00:12:29.490 INFO:teuthology.orchestra.run.smithi123.stdout: "setgroup": "", 2020-01-16T00:12:29.490 INFO:teuthology.orchestra.run.smithi123.stdout: "setuser": "", 2020-01-16T00:12:29.491 INFO:teuthology.orchestra.run.smithi123.stdout: "setuser_match_path": "", 2020-01-16T00:12:29.491 INFO:teuthology.orchestra.run.smithi123.stdout: "target_max_misplaced_ratio": "0.050000", 2020-01-16T00:12:29.491 INFO:teuthology.orchestra.run.smithi123.stdout: "thp": "false", 2020-01-16T00:12:29.491 INFO:teuthology.orchestra.run.smithi123.stdout: "threadpool_default_timeout": "60", 2020-01-16T00:12:29.491 INFO:teuthology.orchestra.run.smithi123.stdout: "threadpool_empty_queue_max_wait": "2", 2020-01-16T00:12:29.491 INFO:teuthology.orchestra.run.smithi123.stdout: "throttler_perf_counter": "true", 2020-01-16T00:12:29.492 INFO:teuthology.orchestra.run.smithi123.stdout: "timing": "false" 2020-01-16T00:12:29.492 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:29.494 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2020-01-16T00:12:29.631 INFO:teuthology.orchestra.run.smithi123.stdout:mon.a: {} 2020-01-16T00:12:29.645 INFO:teuthology.orchestra.run.smithi123.stdout:mon.b: {} 2020-01-16T00:12:29.658 INFO:teuthology.orchestra.run.smithi123.stdout:mon.c: {} 2020-01-16T00:12:29.660 INFO:teuthology.orchestra.run.smithi123.stderr:mon.a: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2020-01-16T00:12:29.660 INFO:teuthology.orchestra.run.smithi123.stderr:mon.b: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2020-01-16T00:12:29.660 INFO:teuthology.orchestra.run.smithi123.stderr:mon.c: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2020-01-16T00:12:29.672 INFO:teuthology.orchestra.run.smithi123:> type ceph-objectstore-tool 2020-01-16T00:12:29.678 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2020-01-16T00:12:29.678 INFO:teuthology.orchestra.run.smithi145:> type ceph-objectstore-tool 2020-01-16T00:12:29.684 INFO:teuthology.orchestra.run.smithi145.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2020-01-16T00:12:29.684 INFO:teuthology.run_tasks:Running task exec... 2020-01-16T00:12:29.699 INFO:teuthology.task.exec:Executing custom commands... 2020-01-16T00:12:29.699 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi123.front.sepia.ceph.com 2020-01-16T00:12:29.699 INFO:teuthology.orchestra.run.smithi123:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool create low_tier 4' 2020-01-16T00:12:29.699 INFO:tasks.thrashosds.thrasher:starting do_thrash 2020-01-16T00:12:29.699 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4, 7, 6] 2020-01-16T00:12:29.700 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2020-01-16T00:12:29.700 INFO:tasks.thrashosds.thrasher:Shrinking pool rbd 2020-01-16T00:12:29.700 INFO:tasks.ceph.ceph_manager.ceph:contract_pool rbd by 10 min 8 2020-01-16T00:12:29.700 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-01-16T00:12:29.701 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2020-01-16T00:12:29.701 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2020-01-16T00:12:29.701 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-01-16T00:12:29.702 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2020-01-16T00:12:29.702 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-01-16T00:12:29.702 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2020-01-16T00:12:29.702 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2020-01-16T00:12:29.728 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:12:29.729+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:12:29.803 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:29.803+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:12:29.886 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:29.886 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:12:29.887 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:12:29.887 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:29.899 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-01-16T00:12:29.906 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:12:29.907 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:29.907+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:12:29.940 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:12:29.973 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:12:29.995 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:12:29.997 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2020-01-16T00:12:30.004 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:30.005+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:12:30.006 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:12:30.008 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2020-01-16T00:12:29.954284+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":24,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8389824,"kb_used_data":1152,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746584896,"statfs":{"total":773094113280,"available":764502933504,"internally_reserved":8589934592,"allocated":1179648,"data_stored":350752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.034899+0000","last_change":"2020-01-16T00:12:20.034125+0000","last_active":"2020-01-16T00:12:20.034899+0000","last_peered":"2020-01-16T00:12:20.034899+0000","last_clean":"2020-01-16T00:12:20.034899+0000","last_became_active":"2020-01-16T00:12:20.033645+0000","last_became_peered":"2020-01-16T00:12:20.033645+0000","last_unstale":"2020-01-16T00:12:20.034899+0000","last_undegraded":"2020-01-16T00:12:20.034899+0000","last_fullsized":"2020-01-16T00:12:20.034899+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.033656+0000","last_change":"2020-01-16T00:12:20.033023+0000","last_active":"2020-01-16T00:12:20.033656+0000","last_peered":"2020-01-16T00:12:20.033656+0000","last_clean":"2020-01-16T00:12:20.033656+0000","last_became_active":"2020-01-16T00:12:20.031576+0000","last_became_peered":"2020-01-16T00:12:20.031576+0000","last_unstale":"2020-01-16T00:12:20.033656+0000","last_undegraded":"2020-01-16T00:12:20.033656+0000","last_fullsized":"2020-01-16T00:12:20.033656+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.031703+0000","last_change":"2020-01-16T00:12:20.031201+0000","last_active":"2020-01-16T00:12:20.031703+0000","last_peered":"2020-01-16T00:12:20.031703+0000","last_clean":"2020-01-16T00:12:20.031703+0000","last_became_active":"2020-01-16T00:12:20.030752+0000","last_became_peered":"2020-01-16T00:12:20.030752+0000","last_unstale":"2020-01-16T00:12:20.031703+0000","last_undegraded":"2020-01-16T00:12:20.031703+0000","last_fullsized":"2020-01-16T00:12:20.031703+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.034773+0000","last_change":"2020-01-16T00:12:20.033205+0000","last_active":"2020-01-16T00:12:20.034773+0000","last_peered":"2020-01-16T00:12:20.034773+0000","last_clean":"2020-01-16T00:12:20.034773+0000","last_became_active":"2020-01-16T00:12:20.030792+0000","last_became_peered":"2020-01-16T00:12:20.030792+0000","last_unstale":"2020-01-16T00:12:20.034773+0000","last_undegraded":"2020-01-16T00:12:20.034773+0000","last_fullsized":"2020-01-16T00:12:20.034773+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.033641+0000","last_change":"2020-01-16T00:12:20.032658+0000","last_active":"2020-01-16T00:12:20.033641+0000","last_peered":"2020-01-16T00:12:20.033641+0000","last_clean":"2020-01-16T00:12:20.033641+0000","last_became_active":"2020-01-16T00:12:20.030794+0000","last_became_peered":"2020-01-16T00:12:20.030794+0000","last_unstale":"2020-01-16T00:12:20.033641+0000","last_undegraded":"2020-01-16T00:12:20.033641+0000","last_fullsized":"2020-01-16T00:12:20.033641+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.032720+0000","last_change":"2020-01-16T00:12:20.032010+0000","last_active":"2020-01-16T00:12:20.032720+0000","last_peered":"2020-01-16T00:12:20.032720+0000","last_clean":"2020-01-16T00:12:20.032720+0000","last_became_active":"2020-01-16T00:12:20.031560+0000","last_became_peered":"2020-01-16T00:12:20.031560+0000","last_unstale":"2020-01-16T00:12:20.032720+0000","last_undegraded":"2020-01-16T00:12:20.032720+0000","last_fullsized":"2020-01-16T00:12:20.032720+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.033296+0000","last_change":"2020-01-16T00:12:20.032733+0000","last_active":"2020-01-16T00:12:20.033296+0000","last_peered":"2020-01-16T00:12:20.033296+0000","last_clean":"2020-01-16T00:12:20.033296+0000","last_became_active":"2020-01-16T00:12:20.032284+0000","last_became_peered":"2020-01-16T00:12:20.032284+0000","last_unstale":"2020-01-16T00:12:20.033296+0000","last_undegraded":"2020-01-16T00:12:20.033296+0000","last_fullsized":"2020-01-16T00:12:20.033296+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"10","reported_epoch":"16","state":"active+clean","last_fresh":"2020-01-16T00:12:20.032829+0000","last_change":"2020-01-16T00:12:20.031741+0000","last_active":"2020-01-16T00:12:20.032829+0000","last_peered":"2020-01-16T00:12:20.032829+0000","last_clean":"2020-01-16T00:12:20.032829+0000","last_became_active":"2020-01-16T00:12:20.030776+0000","last_became_peered":"2020-01-16T00:12:20.030776+0000","last_unstale":"2020-01-16T00:12:20.032829+0000","last_undegraded":"2020-01-16T00:12:20.032829+0000","last_fullsized":"2020-01-16T00:12:20.032829+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048728,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323112,"statfs":{"total":96636764160,"available":95562866688,"internally_reserved":1073741824,"allocated":147456,"data_stored":43844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}]}} 2020-01-16T00:12:30.010 INFO:tasks.ceph.ceph_manager.ceph:would drop below min_pgs, proj -2, currently 8 2020-01-16T00:12:30.028 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:30.029 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:12:30.038 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-01-16T00:12:30.054 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5: {} 2020-01-16T00:12:30.082 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:12:30.105 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:30.105+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:12:30.120 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:12:30.122 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-01-16T00:12:30.122 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-01-16T00:12:30.122 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-01-16T00:12:30.123 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-01-16T00:12:30.123 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-01-16T00:12:30.123 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-01-16T00:12:30.123 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-01-16T00:12:30.123 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-01-16T00:12:30.167 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:30.177 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-01-16T00:12:30.206 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:12:30.205+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:12:30.307 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:30.305+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:30.334 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:30.346 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-01-16T00:12:30.408 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:30.409+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:12:30.446 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:30.455 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-01-16T00:12:30.509 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:12:30.511+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:12:30.567 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:30.571 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-01-16T00:12:30.613 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:30.615+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:12:30.711 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:30.709+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:12:30.724 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:30.733 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-01-16T00:12:30.796 INFO:teuthology.orchestra.run.smithi123.stderr:noscrub is set 2020-01-16T00:12:30.796 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'low_tier' created 2020-01-16T00:12:30.808 INFO:teuthology.run_tasks:Running task rados... 2020-01-16T00:12:30.820 INFO:tasks.rados:Beginning rados... 2020-01-16T00:12:30.820 DEBUG:teuthology.run_tasks:Unwinding manager rados 2020-01-16T00:12:30.833 INFO:tasks.rados:joining rados 2020-01-16T00:12:30.833 INFO:tasks.rados:clients are ['client.0', 'client.1'] 2020-01-16T00:12:30.833 INFO:tasks.rados:starting run 0 out of 1 2020-01-16T00:12:30.833 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2020-01-16T00:12:30.834 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2020-01-16T00:12:30.835 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:12:30.835+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:12:30.842 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:30.853 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-01-16T00:12:30.942 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:30.937+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:31.014 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:31.025 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-01-16T00:12:31.037 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:31.037+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:31.126 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:31.139 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:12:31.139+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:12:31.139 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-01-16T00:12:31.240 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:12:31.241+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:12:31.296 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:31.304 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-01-16T00:12:31.348 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:31.351+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:12:31.411 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:31.421 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-01-16T00:12:31.465 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:12:31.467+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:12:31.530 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:31.540 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-01-16T00:12:31.584 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:12:31.585+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:12:31.641 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:31.646 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:31.645+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:31.650 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-01-16T00:12:31.749 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:31.749+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:12:31.798 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:31.808 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-01-16T00:12:31.813 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'unique_pool_0' created 2020-01-16T00:12:31.825 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2020-01-16T00:12:31.874 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:31.871+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:12:31.951 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:31.949+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:31.954 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:31.964 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-01-16T00:12:32.052 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:12:32.053+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:12:32.116 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:32.125 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-01-16T00:12:32.138 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-01-16T00:12:32.188 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:12:32.191+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:12:32.254 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:32.255+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:12:32.277 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:32.280 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:12:32.286 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-01-16T00:12:32.299 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:12:32.316 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:12:32.337 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:12:32.355 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:32.355+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:12:32.357 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:12:32.380 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5: {} 2020-01-16T00:12:32.385 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:32.395 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-01-16T00:12:32.402 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:12:32.425 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:12:32.428 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-01-16T00:12:32.428 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-01-16T00:12:32.428 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-01-16T00:12:32.428 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-01-16T00:12:32.429 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-01-16T00:12:32.429 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-01-16T00:12:32.429 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-01-16T00:12:32.429 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-01-16T00:12:32.456 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:32.457+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:32.494 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:32.494 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:12:32.494 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:12:32.494 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:12:32.495 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:32.503 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-01-16T00:12:32.557 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:12:32.559+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:12:32.645 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:32.645 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:12:32.645 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:12:32.645 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [] 2020-01-16T00:12:32.646 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:32.653 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-01-16T00:12:32.699 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:12:32.701+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:12:32.753 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:32.753 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:12:32.753 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:12:32.753 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:32.759 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:12:32.759+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:12:32.762 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-01-16T00:12:32.828 INFO:teuthology.orchestra.run.smithi123.stderr:enabled application 'rados' on pool 'unique_pool_0' 2020-01-16T00:12:32.834 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2020-01-16T00:12:32.885 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:12:32.887+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:12:32.905 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:32.905 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:12:32.906 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:12:32.906 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:12:32.906 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:32.916 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-01-16T00:12:32.967 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:12:32.965+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:12:33.019 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:33.020 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:12:33.020 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:12:33.020 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [] 2020-01-16T00:12:33.020 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:33.028 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-01-16T00:12:33.071 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:12:33.073+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:12:33.137 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:33.137 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:12:33.138 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:12:33.138 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:33.147 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-01-16T00:12:33.168 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:12:33.169+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:12:33.254 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:33.254 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:12:33.255 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:12:33.255 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:12:33.255 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:33.264 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-01-16T00:12:33.309 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:12:33.311+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:12:33.370 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:33.370 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:12:33.370 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:12:33.370 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [] 2020-01-16T00:12:33.370 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:33.371 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:33.369+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:12:33.380 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-01-16T00:12:33.471 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:12:33.471+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:12:33.535 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:33.535 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:12:33.535 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:12:33.535 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:33.549 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-01-16T00:12:33.593 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:12:33.595+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:12:33.673 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:12:33.675+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:12:33.700 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:33.701 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:12:33.701 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:12:33.701 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:12:33.701 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:33.710 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-01-16T00:12:33.797 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:33.799+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:12:33.833 INFO:teuthology.orchestra.run.smithi123.stderr:nodeep-scrub is set 2020-01-16T00:12:33.842 INFO:teuthology.orchestra.run.smithi123:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --low_tier_pool low_tier --max-ops 1500 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op write_excl 50 --op delete 10 --pool unique_pool_0 2020-01-16T00:12:33.859 INFO:tasks.rados.rados.0.smithi123.stdout:adding op weight read -> 100 2020-01-16T00:12:33.859 INFO:tasks.rados.rados.0.smithi123.stdout:adding op weight write -> 50 2020-01-16T00:12:33.859 INFO:tasks.rados.rados.0.smithi123.stdout:adding op weight write_excl -> 50 2020-01-16T00:12:33.859 INFO:tasks.rados.rados.0.smithi123.stdout:adding op weight delete -> 10 2020-01-16T00:12:33.859 INFO:tasks.rados.rados.0.smithi123.stdout:ceph version 15.0.0-9353-gcb3c85e (cb3c85e74500283e01cc592f7e2858e323186910) octopus (dev) 2020-01-16T00:12:33.859 INFO:tasks.rados.rados.0.smithi123.stdout:Configuration: 2020-01-16T00:12:33.859 INFO:tasks.rados.rados.0.smithi123.stdout: Number of operations: 1500 2020-01-16T00:12:33.860 INFO:tasks.rados.rados.0.smithi123.stdout: Number of objects: 50 2020-01-16T00:12:33.860 INFO:tasks.rados.rados.0.smithi123.stdout: Max in flight operations: 16 2020-01-16T00:12:33.860 INFO:tasks.rados.rados.0.smithi123.stdout: Object size (in bytes): 4000000 2020-01-16T00:12:33.860 INFO:tasks.rados.rados.0.smithi123.stdout: Write stride min: 400000 2020-01-16T00:12:33.860 INFO:tasks.rados.rados.0.smithi123.stdout: Write stride max: 800000 2020-01-16T00:12:33.864 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:33.864 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:12:33.864 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:12:33.865 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [] 2020-01-16T00:12:33.865 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:33.875 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:33.873+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:33.875 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-01-16T00:12:33.976 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:12:33.979+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:12:34.032 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:34.033 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:12:34.033 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:12:34.033 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:34.042 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-01-16T00:12:34.077 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:12:34.077+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:12:34.178 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:12:34.177+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:12:34.195 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:34.195 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:12:34.195 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:12:34.195 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:12:34.195 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:34.204 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-01-16T00:12:34.278 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:12:34.279+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:12:34.314 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:34.314 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:12:34.315 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:12:34.315 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [] 2020-01-16T00:12:34.315 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:34.324 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-01-16T00:12:34.379 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:34.377+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:12:34.440 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-01-16T00:12:34.480 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:34.481+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:12:34.492 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:34.492 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:12:34.493 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:12:34.493 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:34.508 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-01-16T00:12:34.590 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:12:34.591 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:34.593+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:12:34.602 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:12:34.614 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:34.614 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:12:34.614 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:12:34.615 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:12:34.615 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:34.622 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:12:34.625 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-01-16T00:12:34.648 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:12:34.670 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:12:34.691 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:12:34.691+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:12:34.696 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5: {} 2020-01-16T00:12:34.716 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:12:34.727 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:34.736 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:12:34.737 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-01-16T00:12:34.737 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-01-16T00:12:34.738 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-01-16T00:12:34.738 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-01-16T00:12:34.738 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-01-16T00:12:34.738 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-01-16T00:12:34.738 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-01-16T00:12:34.738 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-01-16T00:12:34.738 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-01-16T00:12:34.792 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:12:34.793+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:12:34.838 INFO:tasks.rados.rados.0.smithi123.stdout:1: write initial oid 1 2020-01-16T00:12:34.839 INFO:tasks.rados.rados.0.smithi123.stdout:1: seq_num 0 ranges {416807=475249,1542129=474678} 2020-01-16T00:12:34.847 INFO:tasks.rados.rados.0.smithi123.stdout:1: writing smithi12313934-1 from 416807 to 892056 tid 1 2020-01-16T00:12:34.854 INFO:tasks.rados.rados.0.smithi123.stdout:1: writing smithi12313934-1 from 1542129 to 2016807 tid 2 2020-01-16T00:12:34.854 INFO:tasks.rados.rados.0.smithi123.stdout:2: write initial oid 2 2020-01-16T00:12:34.854 INFO:tasks.rados.rados.0.smithi123.stdout:2: seq_num 1 ranges {416807=475249,1542129=474678} 2020-01-16T00:12:34.858 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:34.859 INFO:tasks.rados.rados.0.smithi123.stdout:2: writing smithi12313934-2 from 416807 to 892056 tid 1 2020-01-16T00:12:34.866 INFO:tasks.rados.rados.0.smithi123.stdout:2: writing smithi12313934-2 from 1542129 to 2016807 tid 2 2020-01-16T00:12:34.866 INFO:tasks.rados.rados.0.smithi123.stdout:3: write initial oid 3 2020-01-16T00:12:34.866 INFO:tasks.rados.rados.0.smithi123.stdout:3: seq_num 2 ranges {433614=550498,1600611=433003} 2020-01-16T00:12:34.869 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-01-16T00:12:34.871 INFO:tasks.rados.rados.0.smithi123.stdout:3: writing smithi12313934-3 from 433614 to 984112 tid 1 2020-01-16T00:12:34.877 INFO:tasks.rados.rados.0.smithi123.stdout:3: writing smithi12313934-3 from 1600611 to 2033614 tid 2 2020-01-16T00:12:34.878 INFO:tasks.rados.rados.0.smithi123.stdout:4: write initial oid 4 2020-01-16T00:12:34.878 INFO:tasks.rados.rados.0.smithi123.stdout:4: seq_num 3 ranges {450421=625747,1659093=391328} 2020-01-16T00:12:34.884 INFO:tasks.rados.rados.0.smithi123.stdout:4: writing smithi12313934-4 from 450421 to 1076168 tid 1 2020-01-16T00:12:34.889 INFO:tasks.rados.rados.0.smithi123.stdout:4: writing smithi12313934-4 from 1659093 to 2050421 tid 2 2020-01-16T00:12:34.890 INFO:tasks.rados.rados.0.smithi123.stdout:5: write initial oid 5 2020-01-16T00:12:34.890 INFO:tasks.rados.rados.0.smithi123.stdout:5: seq_num 4 ranges {467228=700996,1717575=349653} 2020-01-16T00:12:34.896 INFO:tasks.rados.rados.0.smithi123.stdout:5: writing smithi12313934-5 from 467228 to 1168224 tid 1 2020-01-16T00:12:34.902 INFO:tasks.rados.rados.0.smithi123.stdout:5: writing smithi12313934-5 from 1717575 to 2067228 tid 2 2020-01-16T00:12:34.902 INFO:tasks.rados.rados.0.smithi123.stdout:6: write initial oid 6 2020-01-16T00:12:34.902 INFO:tasks.rados.rados.0.smithi123.stdout:6: seq_num 5 ranges {484035=776245,2059704=24331} 2020-01-16T00:12:34.909 INFO:tasks.rados.rados.0.smithi123.stdout:6: writing smithi12313934-6 from 484035 to 1260280 tid 1 2020-01-16T00:12:34.911 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:34.913+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:12:34.914 INFO:tasks.rados.rados.0.smithi123.stdout:6: writing smithi12313934-6 from 2059704 to 2084035 tid 2 2020-01-16T00:12:34.914 INFO:tasks.rados.rados.0.smithi123.stdout:7: write initial oid 7 2020-01-16T00:12:34.914 INFO:tasks.rados.rados.0.smithi123.stdout:7: seq_num 6 ranges {500842=451494,1718186=382656} 2020-01-16T00:12:34.919 INFO:tasks.rados.rados.0.smithi123.stdout:7: writing smithi12313934-7 from 500842 to 952336 tid 1 2020-01-16T00:12:34.926 INFO:tasks.rados.rados.0.smithi123.stdout:7: writing smithi12313934-7 from 1718186 to 2100842 tid 2 2020-01-16T00:12:34.926 INFO:tasks.rados.rados.0.smithi123.stdout:8: write initial oid 8 2020-01-16T00:12:34.926 INFO:tasks.rados.rados.0.smithi123.stdout:8: seq_num 7 ranges {517649=526743,1776668=340981} 2020-01-16T00:12:34.932 INFO:tasks.rados.rados.0.smithi123.stdout:8: writing smithi12313934-8 from 517649 to 1044392 tid 1 2020-01-16T00:12:34.938 INFO:tasks.rados.rados.0.smithi123.stdout:8: writing smithi12313934-8 from 1776668 to 2117649 tid 2 2020-01-16T00:12:34.938 INFO:tasks.rados.rados.0.smithi123.stdout:9: write initial oid 9 2020-01-16T00:12:34.938 INFO:tasks.rados.rados.0.smithi123.stdout:9: seq_num 8 ranges {534456=718345,1951503=182953} 2020-01-16T00:12:34.945 INFO:tasks.rados.rados.0.smithi123.stdout:9: writing smithi12313934-9 from 534456 to 1252801 tid 1 2020-01-16T00:12:34.950 INFO:tasks.rados.rados.0.smithi123.stdout:9: writing smithi12313934-9 from 1951503 to 2134456 tid 2 2020-01-16T00:12:34.951 INFO:tasks.rados.rados.0.smithi123.stdout:10: write initial oid 10 2020-01-16T00:12:34.951 INFO:tasks.rados.rados.0.smithi123.stdout:10: seq_num 9 ranges {551263=793594,1893632=257631} 2020-01-16T00:12:34.958 INFO:tasks.rados.rados.0.smithi123.stdout:10: writing smithi12313934-10 from 551263 to 1344857 tid 1 2020-01-16T00:12:34.963 INFO:tasks.rados.rados.0.smithi123.stdout:10: writing smithi12313934-10 from 1893632 to 2151263 tid 2 2020-01-16T00:12:34.963 INFO:tasks.rados.rados.0.smithi123.stdout:11: write initial oid 11 2020-01-16T00:12:34.963 INFO:tasks.rados.rados.0.smithi123.stdout:11: seq_num 10 ranges {568070=468843,1552114=615956} 2020-01-16T00:12:34.968 INFO:tasks.rados.rados.0.smithi123.stdout:11: writing smithi12313934-11 from 568070 to 1036913 tid 1 2020-01-16T00:12:34.975 INFO:tasks.rados.rados.0.smithi123.stdout:11: writing smithi12313934-11 from 1552114 to 2168070 tid 2 2020-01-16T00:12:34.975 INFO:tasks.rados.rados.0.smithi123.stdout:12: write initial oid 12 2020-01-16T00:12:34.975 INFO:tasks.rados.rados.0.smithi123.stdout:12: seq_num 11 ranges {584877=544092,1610596=562003} 2020-01-16T00:12:34.981 INFO:tasks.rados.rados.0.smithi123.stdout:12: writing smithi12313934-12 from 584877 to 1128969 tid 1 2020-01-16T00:12:34.989 INFO:tasks.rados.rados.0.smithi123.stdout:12: writing smithi12313934-12 from 1610596 to 2172599 tid 2 2020-01-16T00:12:34.989 INFO:tasks.rados.rados.0.smithi123.stdout:13: write initial oid 13 2020-01-16T00:12:34.989 INFO:tasks.rados.rados.0.smithi123.stdout:13: seq_num 12 ranges {601684=619341,1669078=532606} 2020-01-16T00:12:34.994 INFO:tasks.rados.rados.0.smithi123.stdout:13: writing smithi12313934-13 from 601684 to 1221025 tid 1 2020-01-16T00:12:34.994 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:12:34.993+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:12:35.000 INFO:tasks.rados.rados.0.smithi123.stdout:13: writing smithi12313934-13 from 1669078 to 2201684 tid 2 2020-01-16T00:12:35.000 INFO:tasks.rados.rados.0.smithi123.stdout:14: write initial oid 14 2020-01-16T00:12:35.000 INFO:tasks.rados.rados.0.smithi123.stdout:14: seq_num 13 ranges {618491=694590,2011207=207284} 2020-01-16T00:12:35.007 INFO:tasks.rados.rados.0.smithi123.stdout:14: writing smithi12313934-14 from 618491 to 1313081 tid 1 2020-01-16T00:12:35.011 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4, 7, 6] 2020-01-16T00:12:35.011 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2020-01-16T00:12:35.011 INFO:tasks.thrashosds.thrasher:inject_pause on 4 2020-01-16T00:12:35.011 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2020-01-16T00:12:35.011 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2020-01-16T00:12:35.012 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3 2020-01-16T00:12:35.013 INFO:tasks.rados.rados.0.smithi123.stdout:14: writing smithi12313934-14 from 2011207 to 2218491 tid 2 2020-01-16T00:12:35.013 INFO:tasks.rados.rados.0.smithi123.stdout:15: write initial oid 15 2020-01-16T00:12:35.013 INFO:tasks.rados.rados.0.smithi123.stdout:15: seq_num 14 ranges {635298=769839,2069689=165609} 2020-01-16T00:12:35.014 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:35.020 INFO:tasks.rados.rados.0.smithi123.stdout:15: writing smithi12313934-15 from 635298 to 1405137 tid 1 2020-01-16T00:12:35.025 INFO:tasks.rados.rados.0.smithi123.stdout:15: writing smithi12313934-15 from 2069689 to 2235298 tid 2 2020-01-16T00:12:35.025 INFO:tasks.rados.rados.0.smithi123.stdout:16: write initial oid 16 2020-01-16T00:12:35.026 INFO:tasks.rados.rados.0.smithi123.stdout:16: seq_num 15 ranges {652105=445088,1728171=452988} 2020-01-16T00:12:35.027 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-01-16T00:12:35.031 INFO:tasks.rados.rados.0.smithi123.stdout:16: writing smithi12313934-16 from 652105 to 1097193 tid 1 2020-01-16T00:12:35.039 INFO:tasks.rados.rados.0.smithi123.stdout:16: writing smithi12313934-16 from 1728171 to 2181159 tid 2 2020-01-16T00:12:35.039 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.039 INFO:tasks.rados.rados.0.smithi123.stdout:1: finishing write tid 1 to smithi12313934-1 2020-01-16T00:12:35.039 INFO:tasks.rados.rados.0.smithi123.stdout:1: finishing write tid 2 to smithi12313934-1 2020-01-16T00:12:35.040 INFO:tasks.rados.rados.0.smithi123.stdout:1: finishing write tid 4 to smithi12313934-1 2020-01-16T00:12:35.040 INFO:tasks.rados.rados.0.smithi123.stdout:1: finishing write tid 5 to smithi12313934-1 2020-01-16T00:12:35.040 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2020-01-16T00:12:35.040 INFO:tasks.rados.rados.0.smithi123.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2020-01-16T00:12:35.041 INFO:tasks.rados.rados.0.smithi123.stdout:2: finishing write tid 1 to smithi12313934-2 2020-01-16T00:12:35.041 INFO:tasks.rados.rados.0.smithi123.stdout:2: finishing write tid 2 to smithi12313934-2 2020-01-16T00:12:35.041 INFO:tasks.rados.rados.0.smithi123.stdout:2: finishing write tid 4 to smithi12313934-2 2020-01-16T00:12:35.041 INFO:tasks.rados.rados.0.smithi123.stdout:1: done (15 left) 2020-01-16T00:12:35.041 INFO:tasks.rados.rados.0.smithi123.stdout:17: write initial oid 17 2020-01-16T00:12:35.042 INFO:tasks.rados.rados.0.smithi123.stdout:17: seq_num 16 ranges {668912=636690,1903006=365906} 2020-01-16T00:12:35.045 INFO:tasks.rados.rados.0.smithi123.stdout:17: writing smithi12313934-17 from 668912 to 1305602 tid 1 2020-01-16T00:12:35.051 INFO:tasks.rados.rados.0.smithi123.stdout:17: writing smithi12313934-17 from 1903006 to 2268912 tid 2 2020-01-16T00:12:35.052 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.052 INFO:tasks.rados.rados.0.smithi123.stdout:2: finishing write tid 5 to smithi12313934-2 2020-01-16T00:12:35.052 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2020-01-16T00:12:35.052 INFO:tasks.rados.rados.0.smithi123.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2020-01-16T00:12:35.053 INFO:tasks.rados.rados.0.smithi123.stdout:2: done (15 left) 2020-01-16T00:12:35.053 INFO:tasks.rados.rados.0.smithi123.stdout:18: write initial oid 18 2020-01-16T00:12:35.053 INFO:tasks.rados.rados.0.smithi123.stdout:18: seq_num 17 ranges {685719=711939,1845135=440584} 2020-01-16T00:12:35.059 INFO:tasks.rados.rados.0.smithi123.stdout:18: writing smithi12313934-18 from 685719 to 1397658 tid 1 2020-01-16T00:12:35.064 INFO:tasks.rados.rados.0.smithi123.stdout:18: writing smithi12313934-18 from 1845135 to 2285719 tid 2 2020-01-16T00:12:35.065 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.065 INFO:tasks.rados.rados.0.smithi123.stdout:3: finishing write tid 1 to smithi12313934-3 2020-01-16T00:12:35.065 INFO:tasks.rados.rados.0.smithi123.stdout:3: finishing write tid 2 to smithi12313934-3 2020-01-16T00:12:35.065 INFO:tasks.rados.rados.0.smithi123.stdout:3: finishing write tid 4 to smithi12313934-3 2020-01-16T00:12:35.065 INFO:tasks.rados.rados.0.smithi123.stdout:3: finishing write tid 5 to smithi12313934-3 2020-01-16T00:12:35.065 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2020-01-16T00:12:35.065 INFO:tasks.rados.rados.0.smithi123.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2020-01-16T00:12:35.066 INFO:tasks.rados.rados.0.smithi123.stdout:4: finishing write tid 1 to smithi12313934-4 2020-01-16T00:12:35.066 INFO:tasks.rados.rados.0.smithi123.stdout:4: finishing write tid 2 to smithi12313934-4 2020-01-16T00:12:35.066 INFO:tasks.rados.rados.0.smithi123.stdout:3: done (15 left) 2020-01-16T00:12:35.066 INFO:tasks.rados.rados.0.smithi123.stdout:19: write initial oid 19 2020-01-16T00:12:35.066 INFO:tasks.rados.rados.0.smithi123.stdout:19: seq_num 18 ranges {702526=787188,1903617=398909} 2020-01-16T00:12:35.073 INFO:tasks.rados.rados.0.smithi123.stdout:19: writing smithi12313934-19 from 702526 to 1489714 tid 1 2020-01-16T00:12:35.078 INFO:tasks.rados.rados.0.smithi123.stdout:19: writing smithi12313934-19 from 1903617 to 2302526 tid 2 2020-01-16T00:12:35.078 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.078 INFO:tasks.rados.rados.0.smithi123.stdout:4: finishing write tid 4 to smithi12313934-4 2020-01-16T00:12:35.078 INFO:tasks.rados.rados.0.smithi123.stdout:4: finishing write tid 5 to smithi12313934-4 2020-01-16T00:12:35.078 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2020-01-16T00:12:35.078 INFO:tasks.rados.rados.0.smithi123.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2020-01-16T00:12:35.079 INFO:tasks.rados.rados.0.smithi123.stdout:5: finishing write tid 1 to smithi12313934-5 2020-01-16T00:12:35.079 INFO:tasks.rados.rados.0.smithi123.stdout:5: finishing write tid 2 to smithi12313934-5 2020-01-16T00:12:35.079 INFO:tasks.rados.rados.0.smithi123.stdout:4: done (15 left) 2020-01-16T00:12:35.079 INFO:tasks.rados.rados.0.smithi123.stdout:20: write initial oid 20 2020-01-16T00:12:35.079 INFO:tasks.rados.rados.0.smithi123.stdout:20: seq_num 19 ranges {719333=462437,1962099=357234} 2020-01-16T00:12:35.088 INFO:tasks.rados.rados.0.smithi123.stdout:20: writing smithi12313934-20 from 719333 to 1181770 tid 1 2020-01-16T00:12:35.103 INFO:tasks.rados.rados.0.smithi123.stdout:20: writing smithi12313934-20 from 1962099 to 2319333 tid 2 2020-01-16T00:12:35.103 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.103 INFO:tasks.rados.rados.0.smithi123.stdout:5: finishing write tid 4 to smithi12313934-5 2020-01-16T00:12:35.103 INFO:tasks.rados.rados.0.smithi123.stdout:5: finishing write tid 5 to smithi12313934-5 2020-01-16T00:12:35.103 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2020-01-16T00:12:35.104 INFO:tasks.rados.rados.0.smithi123.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2020-01-16T00:12:35.104 INFO:tasks.rados.rados.0.smithi123.stdout:6: finishing write tid 1 to smithi12313934-6 2020-01-16T00:12:35.104 INFO:tasks.rados.rados.0.smithi123.stdout:6: finishing write tid 2 to smithi12313934-6 2020-01-16T00:12:35.104 INFO:tasks.rados.rados.0.smithi123.stdout:6: finishing write tid 4 to smithi12313934-6 2020-01-16T00:12:35.104 INFO:tasks.rados.rados.0.smithi123.stdout:6: finishing write tid 5 to smithi12313934-6 2020-01-16T00:12:35.104 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 6 v 3 (ObjNum 5 snap 0 seq_num 5) dirty exists 2020-01-16T00:12:35.104 INFO:tasks.rados.rados.0.smithi123.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2020-01-16T00:12:35.104 INFO:tasks.rados.rados.0.smithi123.stdout:7: finishing write tid 1 to smithi12313934-7 2020-01-16T00:12:35.104 INFO:tasks.rados.rados.0.smithi123.stdout:7: finishing write tid 2 to smithi12313934-7 2020-01-16T00:12:35.105 INFO:tasks.rados.rados.0.smithi123.stdout:7: finishing write tid 4 to smithi12313934-7 2020-01-16T00:12:35.105 INFO:tasks.rados.rados.0.smithi123.stdout:7: finishing write tid 5 to smithi12313934-7 2020-01-16T00:12:35.105 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 7 v 6 (ObjNum 6 snap 0 seq_num 6) dirty exists 2020-01-16T00:12:35.105 INFO:tasks.rados.rados.0.smithi123.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2020-01-16T00:12:35.105 INFO:tasks.rados.rados.0.smithi123.stdout:8: finishing write tid 1 to smithi12313934-8 2020-01-16T00:12:35.105 INFO:tasks.rados.rados.0.smithi123.stdout:8: finishing write tid 2 to smithi12313934-8 2020-01-16T00:12:35.105 INFO:tasks.rados.rados.0.smithi123.stdout:8: finishing write tid 4 to smithi12313934-8 2020-01-16T00:12:35.105 INFO:tasks.rados.rados.0.smithi123.stdout:8: finishing write tid 5 to smithi12313934-8 2020-01-16T00:12:35.106 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 8 v 6 (ObjNum 7 snap 0 seq_num 7) dirty exists 2020-01-16T00:12:35.106 INFO:tasks.rados.rados.0.smithi123.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2020-01-16T00:12:35.106 INFO:tasks.rados.rados.0.smithi123.stdout:5: done (15 left) 2020-01-16T00:12:35.106 INFO:tasks.rados.rados.0.smithi123.stdout:6: done (14 left) 2020-01-16T00:12:35.106 INFO:tasks.rados.rados.0.smithi123.stdout:7: done (13 left) 2020-01-16T00:12:35.106 INFO:tasks.rados.rados.0.smithi123.stdout:8: done (12 left) 2020-01-16T00:12:35.106 INFO:tasks.rados.rados.0.smithi123.stdout:21: write initial oid 21 2020-01-16T00:12:35.106 INFO:tasks.rados.rados.0.smithi123.stdout:21: seq_num 20 ranges {736140=537686,2020581=315559} 2020-01-16T00:12:35.107 INFO:tasks.rados.rados.0.smithi123.stdout:21: writing smithi12313934-21 from 736140 to 1273826 tid 1 2020-01-16T00:12:35.107 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:12:35.103+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:12:35.107 INFO:tasks.rados.rados.0.smithi123.stdout:21: writing smithi12313934-21 from 2020581 to 2336140 tid 2 2020-01-16T00:12:35.107 INFO:tasks.rados.rados.0.smithi123.stdout:22: write initial oid 22 2020-01-16T00:12:35.108 INFO:tasks.rados.rados.0.smithi123.stdout:22: seq_num 21 ranges {752947=612935,1962710=390237} 2020-01-16T00:12:35.118 INFO:tasks.rados.rados.0.smithi123.stdout:22: writing smithi12313934-22 from 752947 to 1365882 tid 1 2020-01-16T00:12:35.118 INFO:tasks.rados.rados.0.smithi123.stdout:22: writing smithi12313934-22 from 1962710 to 2352947 tid 2 2020-01-16T00:12:35.118 INFO:tasks.rados.rados.0.smithi123.stdout:23: write initial oid 23 2020-01-16T00:12:35.118 INFO:tasks.rados.rados.0.smithi123.stdout:23: seq_num 22 ranges {769754=688184,2021192=348562} 2020-01-16T00:12:35.125 INFO:tasks.rados.rados.0.smithi123.stdout:23: writing smithi12313934-23 from 769754 to 1457938 tid 1 2020-01-16T00:12:35.131 INFO:tasks.rados.rados.0.smithi123.stdout:23: writing smithi12313934-23 from 2021192 to 2369754 tid 2 2020-01-16T00:12:35.131 INFO:tasks.rados.rados.0.smithi123.stdout:24: write initial oid 24 2020-01-16T00:12:35.132 INFO:tasks.rados.rados.0.smithi123.stdout:24: seq_num 23 ranges {786561=479786,1796027=467664} 2020-01-16T00:12:35.132 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:35.132 INFO:teuthology.orchestra.run.smithi145.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2020-01-16T00:12:35.132 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:35.138 INFO:tasks.rados.rados.0.smithi123.stdout:24: writing smithi12313934-24 from 786561 to 1266347 tid 1 2020-01-16T00:12:35.144 INFO:tasks.rados.rados.0.smithi123.stdout:24: writing smithi12313934-24 from 1796027 to 2263691 tid 2 2020-01-16T00:12:35.144 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.144 INFO:tasks.rados.rados.0.smithi123.stdout:9: finishing write tid 1 to smithi12313934-9 2020-01-16T00:12:35.144 INFO:tasks.rados.rados.0.smithi123.stdout:9: finishing write tid 2 to smithi12313934-9 2020-01-16T00:12:35.145 INFO:tasks.rados.rados.0.smithi123.stdout:9: finishing write tid 4 to smithi12313934-9 2020-01-16T00:12:35.145 INFO:tasks.rados.rados.0.smithi123.stdout:9: finishing write tid 5 to smithi12313934-9 2020-01-16T00:12:35.145 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 9 v 3 (ObjNum 8 snap 0 seq_num 8) dirty exists 2020-01-16T00:12:35.145 INFO:tasks.rados.rados.0.smithi123.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2020-01-16T00:12:35.145 INFO:tasks.rados.rados.0.smithi123.stdout:10: finishing write tid 1 to smithi12313934-10 2020-01-16T00:12:35.145 INFO:tasks.rados.rados.0.smithi123.stdout:9: done (15 left) 2020-01-16T00:12:35.145 INFO:tasks.rados.rados.0.smithi123.stdout:25: write initial oid 25 2020-01-16T00:12:35.146 INFO:tasks.rados.rados.0.smithi123.stdout:25: seq_num 24 ranges {403368=555035,1454509=727675} 2020-01-16T00:12:35.149 INFO:tasks.rados.rados.0.smithi123.stdout:25: writing smithi12313934-25 from 403368 to 958403 tid 1 2020-01-16T00:12:35.156 INFO:tasks.rados.rados.0.smithi123.stdout:25: writing smithi12313934-25 from 1454509 to 2182184 tid 2 2020-01-16T00:12:35.157 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.157 INFO:tasks.rados.rados.0.smithi123.stdout:10: finishing write tid 2 to smithi12313934-10 2020-01-16T00:12:35.157 INFO:tasks.rados.rados.0.smithi123.stdout:10: finishing write tid 4 to smithi12313934-10 2020-01-16T00:12:35.157 INFO:tasks.rados.rados.0.smithi123.stdout:10: finishing write tid 5 to smithi12313934-10 2020-01-16T00:12:35.157 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 10 v 3 (ObjNum 9 snap 0 seq_num 9) dirty exists 2020-01-16T00:12:35.157 INFO:tasks.rados.rados.0.smithi123.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2020-01-16T00:12:35.157 INFO:tasks.rados.rados.0.smithi123.stdout:11: finishing write tid 1 to smithi12313934-11 2020-01-16T00:12:35.157 INFO:tasks.rados.rados.0.smithi123.stdout:11: finishing write tid 2 to smithi12313934-11 2020-01-16T00:12:35.158 INFO:tasks.rados.rados.0.smithi123.stdout:11: finishing write tid 4 to smithi12313934-11 2020-01-16T00:12:35.158 INFO:tasks.rados.rados.0.smithi123.stdout:10: done (15 left) 2020-01-16T00:12:35.158 INFO:tasks.rados.rados.0.smithi123.stdout:26: write initial oid 26 2020-01-16T00:12:35.158 INFO:tasks.rados.rados.0.smithi123.stdout:26: seq_num 25 ranges {420175=630284,1796638=471333} 2020-01-16T00:12:35.162 INFO:tasks.rados.rados.0.smithi123.stdout:26: writing smithi12313934-26 from 420175 to 1050459 tid 1 2020-01-16T00:12:35.178 INFO:tasks.rados.rados.0.smithi123.stdout:26: writing smithi12313934-26 from 1796638 to 2267971 tid 2 2020-01-16T00:12:35.179 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.179 INFO:tasks.rados.rados.0.smithi123.stdout:11: finishing write tid 5 to smithi12313934-11 2020-01-16T00:12:35.179 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 11 v 6 (ObjNum 10 snap 0 seq_num 10) dirty exists 2020-01-16T00:12:35.179 INFO:tasks.rados.rados.0.smithi123.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2020-01-16T00:12:35.179 INFO:tasks.rados.rados.0.smithi123.stdout:11: done (15 left) 2020-01-16T00:12:35.179 INFO:tasks.rados.rados.0.smithi123.stdout:27: write initial oid 27 2020-01-16T00:12:35.179 INFO:tasks.rados.rados.0.smithi123.stdout:27: seq_num 26 ranges {436982=705533,1855120=581862} 2020-01-16T00:12:35.179 INFO:tasks.rados.rados.0.smithi123.stdout:27: writing smithi12313934-27 from 436982 to 1142515 tid 1 2020-01-16T00:12:35.180 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:35.191 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-01-16T00:12:35.192 INFO:tasks.rados.rados.0.smithi123.stdout:27: writing smithi12313934-27 from 1855120 to 2436982 tid 2 2020-01-16T00:12:35.192 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.192 INFO:tasks.rados.rados.0.smithi123.stdout:12: finishing write tid 1 to smithi12313934-12 2020-01-16T00:12:35.192 INFO:tasks.rados.rados.0.smithi123.stdout:12: finishing write tid 2 to smithi12313934-12 2020-01-16T00:12:35.192 INFO:tasks.rados.rados.0.smithi123.stdout:12: finishing write tid 4 to smithi12313934-12 2020-01-16T00:12:35.192 INFO:tasks.rados.rados.0.smithi123.stdout:12: finishing write tid 5 to smithi12313934-12 2020-01-16T00:12:35.193 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 12 v 9 (ObjNum 11 snap 0 seq_num 11) dirty exists 2020-01-16T00:12:35.193 INFO:tasks.rados.rados.0.smithi123.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2020-01-16T00:12:35.193 INFO:tasks.rados.rados.0.smithi123.stdout:13: finishing write tid 1 to smithi12313934-13 2020-01-16T00:12:35.193 INFO:tasks.rados.rados.0.smithi123.stdout:13: finishing write tid 2 to smithi12313934-13 2020-01-16T00:12:35.193 INFO:tasks.rados.rados.0.smithi123.stdout:13: finishing write tid 4 to smithi12313934-13 2020-01-16T00:12:35.193 INFO:tasks.rados.rados.0.smithi123.stdout:13: finishing write tid 5 to smithi12313934-13 2020-01-16T00:12:35.193 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 3 (ObjNum 12 snap 0 seq_num 12) dirty exists 2020-01-16T00:12:35.194 INFO:tasks.rados.rados.0.smithi123.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2020-01-16T00:12:35.194 INFO:tasks.rados.rados.0.smithi123.stdout:14: finishing write tid 1 to smithi12313934-14 2020-01-16T00:12:35.194 INFO:tasks.rados.rados.0.smithi123.stdout:12: done (15 left) 2020-01-16T00:12:35.194 INFO:tasks.rados.rados.0.smithi123.stdout:13: done (14 left) 2020-01-16T00:12:35.194 INFO:tasks.rados.rados.0.smithi123.stdout:28: write initial oid 28 2020-01-16T00:12:35.194 INFO:tasks.rados.rados.0.smithi123.stdout:28: seq_num 27 ranges {453789=780782,1913602=475002} 2020-01-16T00:12:35.194 INFO:tasks.rados.rados.0.smithi123.stdout:28: writing smithi12313934-28 from 453789 to 1234571 tid 1 2020-01-16T00:12:35.201 INFO:tasks.rados.rados.0.smithi123.stdout:28: writing smithi12313934-28 from 1913602 to 2388604 tid 2 2020-01-16T00:12:35.202 INFO:tasks.rados.rados.0.smithi123.stdout:29: write initial oid 29 2020-01-16T00:12:35.202 INFO:tasks.rados.rados.0.smithi123.stdout:29: seq_num 28 ranges {470596=456031,1572084=618660} 2020-01-16T00:12:35.203 INFO:tasks.rados.rados.0.smithi123.stdout:29: writing smithi12313934-29 from 470596 to 926627 tid 1 2020-01-16T00:12:35.203 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:12:35.203+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:12:35.209 INFO:tasks.rados.rados.0.smithi123.stdout:29: writing smithi12313934-29 from 1572084 to 2190744 tid 2 2020-01-16T00:12:35.209 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.210 INFO:tasks.rados.rados.0.smithi123.stdout:14: finishing write tid 2 to smithi12313934-14 2020-01-16T00:12:35.210 INFO:tasks.rados.rados.0.smithi123.stdout:14: finishing write tid 4 to smithi12313934-14 2020-01-16T00:12:35.210 INFO:tasks.rados.rados.0.smithi123.stdout:14: finishing write tid 5 to smithi12313934-14 2020-01-16T00:12:35.210 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 14 v 3 (ObjNum 13 snap 0 seq_num 13) dirty exists 2020-01-16T00:12:35.210 INFO:tasks.rados.rados.0.smithi123.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2020-01-16T00:12:35.210 INFO:tasks.rados.rados.0.smithi123.stdout:15: finishing write tid 1 to smithi12313934-15 2020-01-16T00:12:35.210 INFO:tasks.rados.rados.0.smithi123.stdout:15: finishing write tid 2 to smithi12313934-15 2020-01-16T00:12:35.210 INFO:tasks.rados.rados.0.smithi123.stdout:15: finishing write tid 4 to smithi12313934-15 2020-01-16T00:12:35.211 INFO:tasks.rados.rados.0.smithi123.stdout:14: done (15 left) 2020-01-16T00:12:35.211 INFO:tasks.rados.rados.0.smithi123.stdout:30: write initial oid 30 2020-01-16T00:12:35.211 INFO:tasks.rados.rados.0.smithi123.stdout:30: seq_num 29 ranges {487403=531280,1514213=478671} 2020-01-16T00:12:35.217 INFO:tasks.rados.rados.0.smithi123.stdout:30: writing smithi12313934-30 from 487403 to 1018683 tid 1 2020-01-16T00:12:35.222 INFO:tasks.rados.rados.0.smithi123.stdout:30: writing smithi12313934-30 from 1514213 to 1992884 tid 2 2020-01-16T00:12:35.223 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.223 INFO:tasks.rados.rados.0.smithi123.stdout:15: finishing write tid 5 to smithi12313934-15 2020-01-16T00:12:35.223 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 15 v 3 (ObjNum 14 snap 0 seq_num 14) dirty exists 2020-01-16T00:12:35.223 INFO:tasks.rados.rados.0.smithi123.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2020-01-16T00:12:35.223 INFO:tasks.rados.rados.0.smithi123.stdout:16: finishing write tid 1 to smithi12313934-16 2020-01-16T00:12:35.223 INFO:tasks.rados.rados.0.smithi123.stdout:16: finishing write tid 2 to smithi12313934-16 2020-01-16T00:12:35.223 INFO:tasks.rados.rados.0.smithi123.stdout:15: done (15 left) 2020-01-16T00:12:35.224 INFO:tasks.rados.rados.0.smithi123.stdout:31: write initial oid 31 2020-01-16T00:12:35.224 INFO:tasks.rados.rados.0.smithi123.stdout:31: seq_num 30 ranges {504210=606529,1572695=622329} 2020-01-16T00:12:35.226 INFO:tasks.rados.rados.0.smithi123.stdout:31: writing smithi12313934-31 from 504210 to 1110739 tid 1 2020-01-16T00:12:35.237 INFO:tasks.rados.rados.0.smithi123.stdout:31: writing smithi12313934-31 from 1572695 to 2195024 tid 2 2020-01-16T00:12:35.237 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.237 INFO:tasks.rados.rados.0.smithi123.stdout:16: finishing write tid 4 to smithi12313934-16 2020-01-16T00:12:35.237 INFO:tasks.rados.rados.0.smithi123.stdout:16: finishing write tid 5 to smithi12313934-16 2020-01-16T00:12:35.238 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 16 v 6 (ObjNum 15 snap 0 seq_num 15) dirty exists 2020-01-16T00:12:35.238 INFO:tasks.rados.rados.0.smithi123.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2020-01-16T00:12:35.238 INFO:tasks.rados.rados.0.smithi123.stdout:17: finishing write tid 1 to smithi12313934-17 2020-01-16T00:12:35.238 INFO:tasks.rados.rados.0.smithi123.stdout:17: finishing write tid 2 to smithi12313934-17 2020-01-16T00:12:35.238 INFO:tasks.rados.rados.0.smithi123.stdout:17: finishing write tid 4 to smithi12313934-17 2020-01-16T00:12:35.238 INFO:tasks.rados.rados.0.smithi123.stdout:16: done (15 left) 2020-01-16T00:12:35.238 INFO:tasks.rados.rados.0.smithi123.stdout:32: write initial oid 32 2020-01-16T00:12:35.238 INFO:tasks.rados.rados.0.smithi123.stdout:32: seq_num 31 ranges {521017=798131,1747530=482340} 2020-01-16T00:12:35.240 INFO:tasks.rados.rados.0.smithi123.stdout:32: writing smithi12313934-32 from 521017 to 1319148 tid 1 2020-01-16T00:12:35.245 INFO:tasks.rados.rados.0.smithi123.stdout:32: writing smithi12313934-32 from 1747530 to 2229870 tid 2 2020-01-16T00:12:35.246 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.246 INFO:tasks.rados.rados.0.smithi123.stdout:17: finishing write tid 5 to smithi12313934-17 2020-01-16T00:12:35.246 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 17 v 6 (ObjNum 16 snap 0 seq_num 16) dirty exists 2020-01-16T00:12:35.246 INFO:tasks.rados.rados.0.smithi123.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2020-01-16T00:12:35.246 INFO:tasks.rados.rados.0.smithi123.stdout:18: finishing write tid 1 to smithi12313934-18 2020-01-16T00:12:35.246 INFO:tasks.rados.rados.0.smithi123.stdout:18: finishing write tid 2 to smithi12313934-18 2020-01-16T00:12:35.246 INFO:tasks.rados.rados.0.smithi123.stdout:18: finishing write tid 4 to smithi12313934-18 2020-01-16T00:12:35.247 INFO:tasks.rados.rados.0.smithi123.stdout:18: finishing write tid 5 to smithi12313934-18 2020-01-16T00:12:35.247 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 18 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2020-01-16T00:12:35.247 INFO:tasks.rados.rados.0.smithi123.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2020-01-16T00:12:35.247 INFO:tasks.rados.rados.0.smithi123.stdout:19: finishing write tid 1 to smithi12313934-19 2020-01-16T00:12:35.247 INFO:tasks.rados.rados.0.smithi123.stdout:17: done (15 left) 2020-01-16T00:12:35.247 INFO:tasks.rados.rados.0.smithi123.stdout:18: done (14 left) 2020-01-16T00:12:35.248 INFO:tasks.rados.rados.0.smithi123.stdout:33: write initial oid 33 2020-01-16T00:12:35.248 INFO:tasks.rados.rados.0.smithi123.stdout:33: seq_num 32 ranges {537824=473380,1806012=625998} 2020-01-16T00:12:35.251 INFO:tasks.rados.rados.0.smithi123.stdout:33: writing smithi12313934-33 from 537824 to 1011204 tid 1 2020-01-16T00:12:35.258 INFO:tasks.rados.rados.0.smithi123.stdout:33: writing smithi12313934-33 from 1806012 to 2432010 tid 2 2020-01-16T00:12:35.259 INFO:tasks.rados.rados.0.smithi123.stdout:34: write initial oid 34 2020-01-16T00:12:35.259 INFO:tasks.rados.rados.0.smithi123.stdout:34: seq_num 33 ranges {554631=548629,1748141=486009} 2020-01-16T00:12:35.265 INFO:tasks.rados.rados.0.smithi123.stdout:34: writing smithi12313934-34 from 554631 to 1103260 tid 1 2020-01-16T00:12:35.271 INFO:tasks.rados.rados.0.smithi123.stdout:34: writing smithi12313934-34 from 1748141 to 2234150 tid 2 2020-01-16T00:12:35.271 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.272 INFO:tasks.rados.rados.0.smithi123.stdout:19: finishing write tid 2 to smithi12313934-19 2020-01-16T00:12:35.272 INFO:tasks.rados.rados.0.smithi123.stdout:19: finishing write tid 4 to smithi12313934-19 2020-01-16T00:12:35.272 INFO:tasks.rados.rados.0.smithi123.stdout:19: finishing write tid 5 to smithi12313934-19 2020-01-16T00:12:35.272 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 19 v 9 (ObjNum 18 snap 0 seq_num 18) dirty exists 2020-01-16T00:12:35.272 INFO:tasks.rados.rados.0.smithi123.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2020-01-16T00:12:35.272 INFO:tasks.rados.rados.0.smithi123.stdout:20: finishing write tid 1 to smithi12313934-20 2020-01-16T00:12:35.272 INFO:tasks.rados.rados.0.smithi123.stdout:20: finishing write tid 2 to smithi12313934-20 2020-01-16T00:12:35.273 INFO:tasks.rados.rados.0.smithi123.stdout:19: done (15 left) 2020-01-16T00:12:35.273 INFO:tasks.rados.rados.0.smithi123.stdout:35: write initial oid 35 2020-01-16T00:12:35.273 INFO:tasks.rados.rados.0.smithi123.stdout:35: seq_num 34 ranges {571438=623878,1806623=629667} 2020-01-16T00:12:35.277 INFO:tasks.rados.rados.0.smithi123.stdout:35: writing smithi12313934-35 from 571438 to 1195316 tid 1 2020-01-16T00:12:35.285 INFO:tasks.rados.rados.0.smithi123.stdout:35: writing smithi12313934-35 from 1806623 to 2436290 tid 2 2020-01-16T00:12:35.285 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.285 INFO:tasks.rados.rados.0.smithi123.stdout:20: finishing write tid 4 to smithi12313934-20 2020-01-16T00:12:35.285 INFO:tasks.rados.rados.0.smithi123.stdout:20: finishing write tid 5 to smithi12313934-20 2020-01-16T00:12:35.286 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 20 v 9 (ObjNum 19 snap 0 seq_num 19) dirty exists 2020-01-16T00:12:35.286 INFO:tasks.rados.rados.0.smithi123.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2020-01-16T00:12:35.286 INFO:tasks.rados.rados.0.smithi123.stdout:21: finishing write tid 1 to smithi12313934-21 2020-01-16T00:12:35.286 INFO:tasks.rados.rados.0.smithi123.stdout:21: finishing write tid 2 to smithi12313934-21 2020-01-16T00:12:35.286 INFO:tasks.rados.rados.0.smithi123.stdout:21: finishing write tid 4 to smithi12313934-21 2020-01-16T00:12:35.286 INFO:tasks.rados.rados.0.smithi123.stdout:21: finishing write tid 5 to smithi12313934-21 2020-01-16T00:12:35.286 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 21 v 3 (ObjNum 20 snap 0 seq_num 20) dirty exists 2020-01-16T00:12:35.286 INFO:tasks.rados.rados.0.smithi123.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2020-01-16T00:12:35.287 INFO:tasks.rados.rados.0.smithi123.stdout:22: finishing write tid 1 to smithi12313934-22 2020-01-16T00:12:35.287 INFO:tasks.rados.rados.0.smithi123.stdout:20: done (15 left) 2020-01-16T00:12:35.287 INFO:tasks.rados.rados.0.smithi123.stdout:21: done (14 left) 2020-01-16T00:12:35.287 INFO:tasks.rados.rados.0.smithi123.stdout:36: write initial oid 36 2020-01-16T00:12:35.287 INFO:tasks.rados.rados.0.smithi123.stdout:36: seq_num 35 ranges {588245=699127,1865105=489678} 2020-01-16T00:12:35.292 INFO:tasks.rados.rados.0.smithi123.stdout:36: writing smithi12313934-36 from 588245 to 1287372 tid 1 2020-01-16T00:12:35.298 INFO:tasks.rados.rados.0.smithi123.stdout:36: writing smithi12313934-36 from 1865105 to 2354783 tid 2 2020-01-16T00:12:35.298 INFO:tasks.rados.rados.0.smithi123.stdout:37: write initial oid 37 2020-01-16T00:12:35.299 INFO:tasks.rados.rados.0.smithi123.stdout:37: seq_num 36 ranges {605052=774376,1923587=633336} 2020-01-16T00:12:35.304 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:35.305+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:35.304 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:35.305 INFO:tasks.rados.rados.0.smithi123.stdout:37: writing smithi12313934-37 from 605052 to 1379428 tid 1 2020-01-16T00:12:35.312 INFO:tasks.rados.rados.0.smithi123.stdout:37: writing smithi12313934-37 from 1923587 to 2556923 tid 2 2020-01-16T00:12:35.312 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.312 INFO:tasks.rados.rados.0.smithi123.stdout:22: finishing write tid 2 to smithi12313934-22 2020-01-16T00:12:35.312 INFO:tasks.rados.rados.0.smithi123.stdout:22: finishing write tid 4 to smithi12313934-22 2020-01-16T00:12:35.313 INFO:tasks.rados.rados.0.smithi123.stdout:22: finishing write tid 5 to smithi12313934-22 2020-01-16T00:12:35.313 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 22 v 3 (ObjNum 21 snap 0 seq_num 21) dirty exists 2020-01-16T00:12:35.313 INFO:tasks.rados.rados.0.smithi123.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2020-01-16T00:12:35.313 INFO:tasks.rados.rados.0.smithi123.stdout:23: finishing write tid 1 to smithi12313934-23 2020-01-16T00:12:35.313 INFO:tasks.rados.rados.0.smithi123.stdout:23: finishing write tid 2 to smithi12313934-23 2020-01-16T00:12:35.313 INFO:tasks.rados.rados.0.smithi123.stdout:22: done (15 left) 2020-01-16T00:12:35.314 INFO:tasks.rados.rados.0.smithi123.stdout:38: write initial oid 38 2020-01-16T00:12:35.314 INFO:tasks.rados.rados.0.smithi123.stdout:38: seq_num 37 ranges {621859=449625,1865716=756143} 2020-01-16T00:12:35.315 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-01-16T00:12:35.317 INFO:tasks.rados.rados.0.smithi123.stdout:38: writing smithi12313934-38 from 621859 to 1071484 tid 1 2020-01-16T00:12:35.326 INFO:tasks.rados.rados.0.smithi123.stdout:38: writing smithi12313934-38 from 1865716 to 2621859 tid 2 2020-01-16T00:12:35.327 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.327 INFO:tasks.rados.rados.0.smithi123.stdout:23: finishing write tid 4 to smithi12313934-23 2020-01-16T00:12:35.327 INFO:tasks.rados.rados.0.smithi123.stdout:23: finishing write tid 5 to smithi12313934-23 2020-01-16T00:12:35.327 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 23 v 9 (ObjNum 22 snap 0 seq_num 22) dirty exists 2020-01-16T00:12:35.328 INFO:tasks.rados.rados.0.smithi123.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2020-01-16T00:12:35.328 INFO:tasks.rados.rados.0.smithi123.stdout:24: finishing write tid 1 to smithi12313934-24 2020-01-16T00:12:35.328 INFO:tasks.rados.rados.0.smithi123.stdout:23: done (15 left) 2020-01-16T00:12:35.328 INFO:tasks.rados.rados.0.smithi123.stdout:39: write initial oid 39 2020-01-16T00:12:35.328 INFO:tasks.rados.rados.0.smithi123.stdout:39: seq_num 38 ranges {638666=524874,1924198=637005} 2020-01-16T00:12:35.332 INFO:tasks.rados.rados.0.smithi123.stdout:39: writing smithi12313934-39 from 638666 to 1163540 tid 1 2020-01-16T00:12:35.342 INFO:tasks.rados.rados.0.smithi123.stdout:39: writing smithi12313934-39 from 1924198 to 2561203 tid 2 2020-01-16T00:12:35.343 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.343 INFO:tasks.rados.rados.0.smithi123.stdout:24: finishing write tid 2 to smithi12313934-24 2020-01-16T00:12:35.343 INFO:tasks.rados.rados.0.smithi123.stdout:24: finishing write tid 4 to smithi12313934-24 2020-01-16T00:12:35.343 INFO:tasks.rados.rados.0.smithi123.stdout:24: finishing write tid 5 to smithi12313934-24 2020-01-16T00:12:35.343 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 24 v 3 (ObjNum 23 snap 0 seq_num 23) dirty exists 2020-01-16T00:12:35.343 INFO:tasks.rados.rados.0.smithi123.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2020-01-16T00:12:35.344 INFO:tasks.rados.rados.0.smithi123.stdout:25: finishing write tid 1 to smithi12313934-25 2020-01-16T00:12:35.344 INFO:tasks.rados.rados.0.smithi123.stdout:25: finishing write tid 2 to smithi12313934-25 2020-01-16T00:12:35.344 INFO:tasks.rados.rados.0.smithi123.stdout:25: finishing write tid 4 to smithi12313934-25 2020-01-16T00:12:35.344 INFO:tasks.rados.rados.0.smithi123.stdout:25: finishing write tid 5 to smithi12313934-25 2020-01-16T00:12:35.345 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 25 v 12 (ObjNum 24 snap 0 seq_num 24) dirty exists 2020-01-16T00:12:35.345 INFO:tasks.rados.rados.0.smithi123.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2020-01-16T00:12:35.345 INFO:tasks.rados.rados.0.smithi123.stdout:26: finishing write tid 1 to smithi12313934-26 2020-01-16T00:12:35.345 INFO:tasks.rados.rados.0.smithi123.stdout:26: finishing write tid 2 to smithi12313934-26 2020-01-16T00:12:35.346 INFO:tasks.rados.rados.0.smithi123.stdout:26: finishing write tid 4 to smithi12313934-26 2020-01-16T00:12:35.346 INFO:tasks.rados.rados.0.smithi123.stdout:26: finishing write tid 5 to smithi12313934-26 2020-01-16T00:12:35.346 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 26 v 9 (ObjNum 25 snap 0 seq_num 25) dirty exists 2020-01-16T00:12:35.346 INFO:tasks.rados.rados.0.smithi123.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2020-01-16T00:12:35.346 INFO:tasks.rados.rados.0.smithi123.stdout:27: finishing write tid 1 to smithi12313934-27 2020-01-16T00:12:35.346 INFO:tasks.rados.rados.0.smithi123.stdout:27: finishing write tid 2 to smithi12313934-27 2020-01-16T00:12:35.347 INFO:tasks.rados.rados.0.smithi123.stdout:27: finishing write tid 4 to smithi12313934-27 2020-01-16T00:12:35.347 INFO:tasks.rados.rados.0.smithi123.stdout:27: finishing write tid 5 to smithi12313934-27 2020-01-16T00:12:35.347 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 27 v 6 (ObjNum 26 snap 0 seq_num 26) dirty exists 2020-01-16T00:12:35.347 INFO:tasks.rados.rados.0.smithi123.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2020-01-16T00:12:35.347 INFO:tasks.rados.rados.0.smithi123.stdout:28: finishing write tid 1 to smithi12313934-28 2020-01-16T00:12:35.347 INFO:tasks.rados.rados.0.smithi123.stdout:28: finishing write tid 2 to smithi12313934-28 2020-01-16T00:12:35.347 INFO:tasks.rados.rados.0.smithi123.stdout:28: finishing write tid 4 to smithi12313934-28 2020-01-16T00:12:35.348 INFO:tasks.rados.rados.0.smithi123.stdout:28: finishing write tid 5 to smithi12313934-28 2020-01-16T00:12:35.348 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 28 v 6 (ObjNum 27 snap 0 seq_num 27) dirty exists 2020-01-16T00:12:35.348 INFO:tasks.rados.rados.0.smithi123.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2020-01-16T00:12:35.348 INFO:tasks.rados.rados.0.smithi123.stdout:29: finishing write tid 1 to smithi12313934-29 2020-01-16T00:12:35.348 INFO:tasks.rados.rados.0.smithi123.stdout:29: finishing write tid 2 to smithi12313934-29 2020-01-16T00:12:35.348 INFO:tasks.rados.rados.0.smithi123.stdout:29: finishing write tid 4 to smithi12313934-29 2020-01-16T00:12:35.348 INFO:tasks.rados.rados.0.smithi123.stdout:29: finishing write tid 5 to smithi12313934-29 2020-01-16T00:12:35.349 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 29 v 6 (ObjNum 28 snap 0 seq_num 28) dirty exists 2020-01-16T00:12:35.349 INFO:tasks.rados.rados.0.smithi123.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2020-01-16T00:12:35.349 INFO:tasks.rados.rados.0.smithi123.stdout:30: finishing write tid 1 to smithi12313934-30 2020-01-16T00:12:35.349 INFO:tasks.rados.rados.0.smithi123.stdout:24: done (15 left) 2020-01-16T00:12:35.349 INFO:tasks.rados.rados.0.smithi123.stdout:25: done (14 left) 2020-01-16T00:12:35.349 INFO:tasks.rados.rados.0.smithi123.stdout:26: done (13 left) 2020-01-16T00:12:35.349 INFO:tasks.rados.rados.0.smithi123.stdout:27: done (12 left) 2020-01-16T00:12:35.350 INFO:tasks.rados.rados.0.smithi123.stdout:28: done (11 left) 2020-01-16T00:12:35.350 INFO:tasks.rados.rados.0.smithi123.stdout:29: done (10 left) 2020-01-16T00:12:35.350 INFO:tasks.rados.rados.0.smithi123.stdout:40: write initial oid 40 2020-01-16T00:12:35.350 INFO:tasks.rados.rados.0.smithi123.stdout:40: seq_num 39 ranges {655473=716476,2099033=556440} 2020-01-16T00:12:35.350 INFO:tasks.rados.rados.0.smithi123.stdout:40: writing smithi12313934-40 from 655473 to 1371949 tid 1 2020-01-16T00:12:35.355 INFO:tasks.rados.rados.0.smithi123.stdout:40: writing smithi12313934-40 from 2099033 to 2655473 tid 2 2020-01-16T00:12:35.356 INFO:tasks.rados.rados.0.smithi123.stdout:41: write initial oid 41 2020-01-16T00:12:35.356 INFO:tasks.rados.rados.0.smithi123.stdout:41: seq_num 40 ranges {672280=791725,2157515=514765} 2020-01-16T00:12:35.363 INFO:tasks.rados.rados.0.smithi123.stdout:41: writing smithi12313934-41 from 672280 to 1464005 tid 1 2020-01-16T00:12:35.370 INFO:tasks.rados.rados.0.smithi123.stdout:41: writing smithi12313934-41 from 2157515 to 2672280 tid 2 2020-01-16T00:12:35.376 INFO:tasks.rados.rados.0.smithi123.stdout:42: write initial oid 42 2020-01-16T00:12:35.376 INFO:tasks.rados.rados.0.smithi123.stdout:42: seq_num 41 ranges {689087=466974,1699644=784332} 2020-01-16T00:12:35.376 INFO:tasks.rados.rados.0.smithi123.stdout:42: writing smithi12313934-42 from 689087 to 1156061 tid 1 2020-01-16T00:12:35.384 INFO:tasks.rados.rados.0.smithi123.stdout:42: writing smithi12313934-42 from 1699644 to 2483976 tid 2 2020-01-16T00:12:35.384 INFO:tasks.rados.rados.0.smithi123.stdout:43: write initial oid 43 2020-01-16T00:12:35.384 INFO:tasks.rados.rados.0.smithi123.stdout:43: seq_num 42 ranges {705894=542223,1758126=644343} 2020-01-16T00:12:35.391 INFO:tasks.rados.rados.0.smithi123.stdout:43: writing smithi12313934-43 from 705894 to 1248117 tid 1 2020-01-16T00:12:35.397 INFO:tasks.rados.rados.0.smithi123.stdout:43: writing smithi12313934-43 from 1758126 to 2402469 tid 2 2020-01-16T00:12:35.397 INFO:tasks.rados.rados.0.smithi123.stdout:44: write initial oid 44 2020-01-16T00:12:35.397 INFO:tasks.rados.rados.0.smithi123.stdout:44: seq_num 43 ranges {722701=617472,1816608=788001} 2020-01-16T00:12:35.410 INFO:tasks.rados.rados.0.smithi123.stdout:44: writing smithi12313934-44 from 722701 to 1340173 tid 1 2020-01-16T00:12:35.411 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:12:35.409+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:12:35.421 INFO:tasks.rados.rados.0.smithi123.stdout:44: writing smithi12313934-44 from 1816608 to 2604609 tid 2 2020-01-16T00:12:35.422 INFO:tasks.rados.rados.0.smithi123.stdout:45: write initial oid 45 2020-01-16T00:12:35.422 INFO:tasks.rados.rados.0.smithi123.stdout:45: seq_num 44 ranges {739508=692721,1875090=648012} 2020-01-16T00:12:35.422 INFO:tasks.rados.rados.0.smithi123.stdout:45: writing smithi12313934-45 from 739508 to 1432229 tid 1 2020-01-16T00:12:35.426 INFO:tasks.rados.rados.0.smithi123.stdout:45: writing smithi12313934-45 from 1875090 to 2523102 tid 2 2020-01-16T00:12:35.427 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.427 INFO:tasks.rados.rados.0.smithi123.stdout:30: finishing write tid 2 to smithi12313934-30 2020-01-16T00:12:35.427 INFO:tasks.rados.rados.0.smithi123.stdout:30: finishing write tid 4 to smithi12313934-30 2020-01-16T00:12:35.427 INFO:tasks.rados.rados.0.smithi123.stdout:30: finishing write tid 5 to smithi12313934-30 2020-01-16T00:12:35.427 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 30 v 9 (ObjNum 29 snap 0 seq_num 29) dirty exists 2020-01-16T00:12:35.427 INFO:tasks.rados.rados.0.smithi123.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2020-01-16T00:12:35.427 INFO:tasks.rados.rados.0.smithi123.stdout:31: finishing write tid 1 to smithi12313934-31 2020-01-16T00:12:35.428 INFO:tasks.rados.rados.0.smithi123.stdout:30: done (15 left) 2020-01-16T00:12:35.428 INFO:tasks.rados.rados.0.smithi123.stdout:46: write initial oid 46 2020-01-16T00:12:35.428 INFO:tasks.rados.rados.0.smithi123.stdout:46: seq_num 45 ranges {756315=767970,1933572=791670} 2020-01-16T00:12:35.434 INFO:tasks.rados.rados.0.smithi123.stdout:46: writing smithi12313934-46 from 756315 to 1524285 tid 1 2020-01-16T00:12:35.442 INFO:tasks.rados.rados.0.smithi123.stdout:46: writing smithi12313934-46 from 1933572 to 2725242 tid 2 2020-01-16T00:12:35.442 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.442 INFO:tasks.rados.rados.0.smithi123.stdout:31: finishing write tid 2 to smithi12313934-31 2020-01-16T00:12:35.442 INFO:tasks.rados.rados.0.smithi123.stdout:31: finishing write tid 4 to smithi12313934-31 2020-01-16T00:12:35.443 INFO:tasks.rados.rados.0.smithi123.stdout:31: finishing write tid 5 to smithi12313934-31 2020-01-16T00:12:35.443 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 31 v 9 (ObjNum 30 snap 0 seq_num 30) dirty exists 2020-01-16T00:12:35.443 INFO:tasks.rados.rados.0.smithi123.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2020-01-16T00:12:35.443 INFO:tasks.rados.rados.0.smithi123.stdout:32: finishing write tid 1 to smithi12313934-32 2020-01-16T00:12:35.443 INFO:tasks.rados.rados.0.smithi123.stdout:32: finishing write tid 2 to smithi12313934-32 2020-01-16T00:12:35.443 INFO:tasks.rados.rados.0.smithi123.stdout:31: done (15 left) 2020-01-16T00:12:35.443 INFO:tasks.rados.rados.0.smithi123.stdout:47: write initial oid 47 2020-01-16T00:12:35.444 INFO:tasks.rados.rados.0.smithi123.stdout:47: seq_num 46 ranges {773122=559572,1992054=651681} 2020-01-16T00:12:35.449 INFO:tasks.rados.rados.0.smithi123.stdout:47: writing smithi12313934-47 from 773122 to 1332694 tid 1 2020-01-16T00:12:35.457 INFO:tasks.rados.rados.0.smithi123.stdout:47: writing smithi12313934-47 from 1992054 to 2643735 tid 2 2020-01-16T00:12:35.457 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.457 INFO:tasks.rados.rados.0.smithi123.stdout:32: finishing write tid 4 to smithi12313934-32 2020-01-16T00:12:35.457 INFO:tasks.rados.rados.0.smithi123.stdout:32: finishing write tid 5 to smithi12313934-32 2020-01-16T00:12:35.458 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 32 v 9 (ObjNum 31 snap 0 seq_num 31) dirty exists 2020-01-16T00:12:35.458 INFO:tasks.rados.rados.0.smithi123.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2020-01-16T00:12:35.458 INFO:tasks.rados.rados.0.smithi123.stdout:33: finishing write tid 1 to smithi12313934-33 2020-01-16T00:12:35.458 INFO:tasks.rados.rados.0.smithi123.stdout:33: finishing write tid 2 to smithi12313934-33 2020-01-16T00:12:35.458 INFO:tasks.rados.rados.0.smithi123.stdout:32: done (15 left) 2020-01-16T00:12:35.458 INFO:tasks.rados.rados.0.smithi123.stdout:48: write initial oid 48 2020-01-16T00:12:35.458 INFO:tasks.rados.rados.0.smithi123.stdout:48: seq_num 47 ranges {789929=634821,2050536=739393} 2020-01-16T00:12:35.465 INFO:tasks.rados.rados.0.smithi123.stdout:48: writing smithi12313934-48 from 789929 to 1424750 tid 1 2020-01-16T00:12:35.466 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:35.473 INFO:tasks.rados.rados.0.smithi123.stdout:48: writing smithi12313934-48 from 2050536 to 2789929 tid 2 2020-01-16T00:12:35.473 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.473 INFO:tasks.rados.rados.0.smithi123.stdout:33: finishing write tid 4 to smithi12313934-33 2020-01-16T00:12:35.473 INFO:tasks.rados.rados.0.smithi123.stdout:33: finishing write tid 5 to smithi12313934-33 2020-01-16T00:12:35.473 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 33 v 12 (ObjNum 32 snap 0 seq_num 32) dirty exists 2020-01-16T00:12:35.474 INFO:tasks.rados.rados.0.smithi123.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2020-01-16T00:12:35.474 INFO:tasks.rados.rados.0.smithi123.stdout:34: finishing write tid 1 to smithi12313934-34 2020-01-16T00:12:35.474 INFO:tasks.rados.rados.0.smithi123.stdout:34: finishing write tid 2 to smithi12313934-34 2020-01-16T00:12:35.474 INFO:tasks.rados.rados.0.smithi123.stdout:34: finishing write tid 4 to smithi12313934-34 2020-01-16T00:12:35.474 INFO:tasks.rados.rados.0.smithi123.stdout:33: done (15 left) 2020-01-16T00:12:35.474 INFO:tasks.rados.rados.0.smithi123.stdout:49: write initial oid 49 2020-01-16T00:12:35.475 INFO:tasks.rados.rados.0.smithi123.stdout:49: seq_num 48 ranges {406736=710070,1709018=655350} 2020-01-16T00:12:35.477 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-01-16T00:12:35.478 INFO:tasks.rados.rados.0.smithi123.stdout:49: writing smithi12313934-49 from 406736 to 1116806 tid 1 2020-01-16T00:12:35.486 INFO:tasks.rados.rados.0.smithi123.stdout:49: writing smithi12313934-49 from 1709018 to 2364368 tid 2 2020-01-16T00:12:35.486 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.486 INFO:tasks.rados.rados.0.smithi123.stdout:34: finishing write tid 5 to smithi12313934-34 2020-01-16T00:12:35.486 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 34 v 6 (ObjNum 33 snap 0 seq_num 33) dirty exists 2020-01-16T00:12:35.486 INFO:tasks.rados.rados.0.smithi123.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2020-01-16T00:12:35.486 INFO:tasks.rados.rados.0.smithi123.stdout:35: finishing write tid 1 to smithi12313934-35 2020-01-16T00:12:35.487 INFO:tasks.rados.rados.0.smithi123.stdout:35: finishing write tid 2 to smithi12313934-35 2020-01-16T00:12:35.487 INFO:tasks.rados.rados.0.smithi123.stdout:35: finishing write tid 4 to smithi12313934-35 2020-01-16T00:12:35.487 INFO:tasks.rados.rados.0.smithi123.stdout:35: finishing write tid 5 to smithi12313934-35 2020-01-16T00:12:35.487 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 35 v 12 (ObjNum 34 snap 0 seq_num 34) dirty exists 2020-01-16T00:12:35.487 INFO:tasks.rados.rados.0.smithi123.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2020-01-16T00:12:35.487 INFO:tasks.rados.rados.0.smithi123.stdout:36: finishing write tid 1 to smithi12313934-36 2020-01-16T00:12:35.487 INFO:tasks.rados.rados.0.smithi123.stdout:34: done (15 left) 2020-01-16T00:12:35.487 INFO:tasks.rados.rados.0.smithi123.stdout:35: done (14 left) 2020-01-16T00:12:35.488 INFO:tasks.rados.rados.0.smithi123.stdout:50: write initial oid 50 2020-01-16T00:12:35.488 INFO:tasks.rados.rados.0.smithi123.stdout:50: seq_num 49 ranges {423543=785319,1767500=799008} 2020-01-16T00:12:35.492 INFO:tasks.rados.rados.0.smithi123.stdout:50: writing smithi12313934-50 from 423543 to 1208862 tid 1 2020-01-16T00:12:35.504 INFO:tasks.rados.rados.0.smithi123.stdout:50: writing smithi12313934-50 from 1767500 to 2566508 tid 2 2020-01-16T00:12:35.504 INFO:tasks.rados.rados.0.smithi123.stdout:51: copy oid 2 target oid 2 low_tier 2020-01-16T00:12:35.504 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 2 to 2 low_tier tgt_pool_name low_tier 2020-01-16T00:12:35.505 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.505 INFO:tasks.rados.rados.0.smithi123.stdout:36: finishing write tid 2 to smithi12313934-36 2020-01-16T00:12:35.505 INFO:tasks.rados.rados.0.smithi123.stdout:36: finishing write tid 4 to smithi12313934-36 2020-01-16T00:12:35.505 INFO:tasks.rados.rados.0.smithi123.stdout:36: finishing write tid 5 to smithi12313934-36 2020-01-16T00:12:35.505 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 36 v 6 (ObjNum 35 snap 0 seq_num 35) dirty exists 2020-01-16T00:12:35.505 INFO:tasks.rados.rados.0.smithi123.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2020-01-16T00:12:35.505 INFO:tasks.rados.rados.0.smithi123.stdout:37: finishing write tid 1 to smithi12313934-37 2020-01-16T00:12:35.506 INFO:tasks.rados.rados.0.smithi123.stdout:37: finishing write tid 2 to smithi12313934-37 2020-01-16T00:12:35.506 INFO:tasks.rados.rados.0.smithi123.stdout:37: finishing write tid 4 to smithi12313934-37 2020-01-16T00:12:35.506 INFO:tasks.rados.rados.0.smithi123.stdout:36: done (15 left) 2020-01-16T00:12:35.506 INFO:tasks.rados.rados.0.smithi123.stdout:52: copy oid 3 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.506 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 3 to 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2020-01-16T00:12:35.506 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.506 INFO:tasks.rados.rados.0.smithi123.stdout:37: finishing write tid 5 to smithi12313934-37 2020-01-16T00:12:35.506 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 37 v 15 (ObjNum 36 snap 0 seq_num 36) dirty exists 2020-01-16T00:12:35.507 INFO:tasks.rados.rados.0.smithi123.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2020-01-16T00:12:35.507 INFO:tasks.rados.rados.0.smithi123.stdout:38: finishing write tid 1 to smithi12313934-38 2020-01-16T00:12:35.507 INFO:tasks.rados.rados.0.smithi123.stdout:38: finishing write tid 2 to smithi12313934-38 2020-01-16T00:12:35.507 INFO:tasks.rados.rados.0.smithi123.stdout:37: done (15 left) 2020-01-16T00:12:35.507 INFO:tasks.rados.rados.0.smithi123.stdout:53: copy oid 4 target oid 4 low_tier 2020-01-16T00:12:35.507 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 4 to 4 low_tier tgt_pool_name low_tier 2020-01-16T00:12:35.507 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.507 INFO:tasks.rados.rados.0.smithi123.stdout:38: finishing write tid 4 to smithi12313934-38 2020-01-16T00:12:35.508 INFO:tasks.rados.rados.0.smithi123.stdout:38: finishing write tid 5 to smithi12313934-38 2020-01-16T00:12:35.508 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 38 v 3 (ObjNum 37 snap 0 seq_num 37) dirty exists 2020-01-16T00:12:35.508 INFO:tasks.rados.rados.0.smithi123.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2020-01-16T00:12:35.508 INFO:tasks.rados.rados.0.smithi123.stdout:39: finishing write tid 1 to smithi12313934-39 2020-01-16T00:12:35.508 INFO:tasks.rados.rados.0.smithi123.stdout:38: done (15 left) 2020-01-16T00:12:35.508 INFO:tasks.rados.rados.0.smithi123.stdout:54: copy oid 5 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.508 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 5 to 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2020-01-16T00:12:35.509 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.509 INFO:tasks.rados.rados.0.smithi123.stdout:39: finishing write tid 2 to smithi12313934-39 2020-01-16T00:12:35.509 INFO:tasks.rados.rados.0.smithi123.stdout:39: finishing write tid 4 to smithi12313934-39 2020-01-16T00:12:35.509 INFO:tasks.rados.rados.0.smithi123.stdout:39: finishing write tid 5 to smithi12313934-39 2020-01-16T00:12:35.509 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 39 v 6 (ObjNum 38 snap 0 seq_num 38) dirty exists 2020-01-16T00:12:35.509 INFO:tasks.rados.rados.0.smithi123.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2020-01-16T00:12:35.509 INFO:tasks.rados.rados.0.smithi123.stdout:40: finishing write tid 1 to smithi12313934-40 2020-01-16T00:12:35.510 INFO:tasks.rados.rados.0.smithi123.stdout:39: done (15 left) 2020-01-16T00:12:35.510 INFO:tasks.rados.rados.0.smithi123.stdout:55: copy oid 6 target oid 6 low_tier 2020-01-16T00:12:35.510 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 6 to 6 low_tier tgt_pool_name low_tier 2020-01-16T00:12:35.510 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.510 INFO:tasks.rados.rados.0.smithi123.stdout:40: finishing write tid 2 to smithi12313934-40 2020-01-16T00:12:35.511 INFO:tasks.rados.rados.0.smithi123.stdout:40: finishing write tid 4 to smithi12313934-40 2020-01-16T00:12:35.511 INFO:tasks.rados.rados.0.smithi123.stdout:40: finishing write tid 5 to smithi12313934-40 2020-01-16T00:12:35.511 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 40 v 9 (ObjNum 39 snap 0 seq_num 39) dirty exists 2020-01-16T00:12:35.511 INFO:tasks.rados.rados.0.smithi123.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2020-01-16T00:12:35.511 INFO:tasks.rados.rados.0.smithi123.stdout:41: finishing write tid 1 to smithi12313934-41 2020-01-16T00:12:35.525 INFO:tasks.rados.rados.0.smithi123.stdout:41: finishing write tid 2 to smithi12313934-41 2020-01-16T00:12:35.525 INFO:tasks.rados.rados.0.smithi123.stdout:40: done (15 left) 2020-01-16T00:12:35.525 INFO:tasks.rados.rados.0.smithi123.stdout:56: copy oid 7 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.526 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 7 to 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2020-01-16T00:12:35.526 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.526 INFO:tasks.rados.rados.0.smithi123.stdout:41: finishing write tid 4 to smithi12313934-41 2020-01-16T00:12:35.531 INFO:tasks.rados.rados.0.smithi123.stdout:41: finishing write tid 5 to smithi12313934-41 2020-01-16T00:12:35.531 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 41 v 12 (ObjNum 40 snap 0 seq_num 40) dirty exists 2020-01-16T00:12:35.531 INFO:tasks.rados.rados.0.smithi123.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2020-01-16T00:12:35.531 INFO:tasks.rados.rados.0.smithi123.stdout:42: finishing write tid 1 to smithi12313934-42 2020-01-16T00:12:35.531 INFO:tasks.rados.rados.0.smithi123.stdout:42: finishing write tid 2 to smithi12313934-42 2020-01-16T00:12:35.531 INFO:tasks.rados.rados.0.smithi123.stdout:42: finishing write tid 4 to smithi12313934-42 2020-01-16T00:12:35.531 INFO:tasks.rados.rados.0.smithi123.stdout:41: done (15 left) 2020-01-16T00:12:35.532 INFO:tasks.rados.rados.0.smithi123.stdout:57: copy oid 8 target oid 8 low_tier 2020-01-16T00:12:35.532 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 8 to 8 low_tier tgt_pool_name low_tier 2020-01-16T00:12:35.532 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.532 INFO:tasks.rados.rados.0.smithi123.stdout:42: finishing write tid 5 to smithi12313934-42 2020-01-16T00:12:35.532 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 42 v 6 (ObjNum 41 snap 0 seq_num 41) dirty exists 2020-01-16T00:12:35.532 INFO:tasks.rados.rados.0.smithi123.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2020-01-16T00:12:35.532 INFO:tasks.rados.rados.0.smithi123.stdout:43: finishing write tid 1 to smithi12313934-43 2020-01-16T00:12:35.533 INFO:tasks.rados.rados.0.smithi123.stdout:42: done (15 left) 2020-01-16T00:12:35.533 INFO:tasks.rados.rados.0.smithi123.stdout:58: copy oid 9 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.533 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 9 to 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2020-01-16T00:12:35.533 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.533 INFO:tasks.rados.rados.0.smithi123.stdout:43: finishing write tid 2 to smithi12313934-43 2020-01-16T00:12:35.533 INFO:tasks.rados.rados.0.smithi123.stdout:43: finishing write tid 4 to smithi12313934-43 2020-01-16T00:12:35.533 INFO:tasks.rados.rados.0.smithi123.stdout:43: finishing write tid 5 to smithi12313934-43 2020-01-16T00:12:35.534 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 43 v 12 (ObjNum 42 snap 0 seq_num 42) dirty exists 2020-01-16T00:12:35.534 INFO:tasks.rados.rados.0.smithi123.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2020-01-16T00:12:35.534 INFO:tasks.rados.rados.0.smithi123.stdout:44: finishing write tid 1 to smithi12313934-44 2020-01-16T00:12:35.534 INFO:tasks.rados.rados.0.smithi123.stdout:43: done (15 left) 2020-01-16T00:12:35.534 INFO:tasks.rados.rados.0.smithi123.stdout:59: copy oid 10 target oid 10 low_tier 2020-01-16T00:12:35.534 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 10 to 10 low_tier tgt_pool_name low_tier 2020-01-16T00:12:35.534 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.534 INFO:tasks.rados.rados.0.smithi123.stdout:44: finishing write tid 2 to smithi12313934-44 2020-01-16T00:12:35.535 INFO:tasks.rados.rados.0.smithi123.stdout:44: finishing write tid 4 to smithi12313934-44 2020-01-16T00:12:35.535 INFO:tasks.rados.rados.0.smithi123.stdout:44: finishing write tid 5 to smithi12313934-44 2020-01-16T00:12:35.535 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 44 v 15 (ObjNum 43 snap 0 seq_num 43) dirty exists 2020-01-16T00:12:35.535 INFO:tasks.rados.rados.0.smithi123.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2020-01-16T00:12:35.535 INFO:tasks.rados.rados.0.smithi123.stdout:45: finishing write tid 1 to smithi12313934-45 2020-01-16T00:12:35.535 INFO:tasks.rados.rados.0.smithi123.stdout:44: done (15 left) 2020-01-16T00:12:35.535 INFO:tasks.rados.rados.0.smithi123.stdout:60: copy oid 11 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.535 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 11 to 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2020-01-16T00:12:35.536 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.536 INFO:tasks.rados.rados.0.smithi123.stdout:45: finishing write tid 2 to smithi12313934-45 2020-01-16T00:12:35.536 INFO:tasks.rados.rados.0.smithi123.stdout:45: finishing write tid 4 to smithi12313934-45 2020-01-16T00:12:35.536 INFO:tasks.rados.rados.0.smithi123.stdout:45: finishing write tid 5 to smithi12313934-45 2020-01-16T00:12:35.536 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 45 v 6 (ObjNum 44 snap 0 seq_num 44) dirty exists 2020-01-16T00:12:35.536 INFO:tasks.rados.rados.0.smithi123.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2020-01-16T00:12:35.536 INFO:tasks.rados.rados.0.smithi123.stdout:46: finishing write tid 1 to smithi12313934-46 2020-01-16T00:12:35.537 INFO:tasks.rados.rados.0.smithi123.stdout:45: done (15 left) 2020-01-16T00:12:35.537 INFO:tasks.rados.rados.0.smithi123.stdout:61: copy oid 12 target oid 12 low_tier 2020-01-16T00:12:35.537 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 12 to 12 low_tier tgt_pool_name low_tier 2020-01-16T00:12:35.537 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.537 INFO:tasks.rados.rados.0.smithi123.stdout:46: finishing write tid 2 to smithi12313934-46 2020-01-16T00:12:35.537 INFO:tasks.rados.rados.0.smithi123.stdout:46: finishing write tid 4 to smithi12313934-46 2020-01-16T00:12:35.537 INFO:tasks.rados.rados.0.smithi123.stdout:46: finishing write tid 5 to smithi12313934-46 2020-01-16T00:12:35.537 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 46 v 18 (ObjNum 45 snap 0 seq_num 45) dirty exists 2020-01-16T00:12:35.538 INFO:tasks.rados.rados.0.smithi123.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2020-01-16T00:12:35.538 INFO:tasks.rados.rados.0.smithi123.stdout:47: finishing write tid 1 to smithi12313934-47 2020-01-16T00:12:35.538 INFO:tasks.rados.rados.0.smithi123.stdout:47: finishing write tid 2 to smithi12313934-47 2020-01-16T00:12:35.538 INFO:tasks.rados.rados.0.smithi123.stdout:46: done (15 left) 2020-01-16T00:12:35.538 INFO:tasks.rados.rados.0.smithi123.stdout:62: copy oid 13 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.538 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 13 to 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2020-01-16T00:12:35.538 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.539 INFO:tasks.rados.rados.0.smithi123.stdout:47: finishing write tid 4 to smithi12313934-47 2020-01-16T00:12:35.539 INFO:tasks.rados.rados.0.smithi123.stdout:47: finishing write tid 5 to smithi12313934-47 2020-01-16T00:12:35.539 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 47 v 15 (ObjNum 46 snap 0 seq_num 46) dirty exists 2020-01-16T00:12:35.539 INFO:tasks.rados.rados.0.smithi123.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2020-01-16T00:12:35.539 INFO:tasks.rados.rados.0.smithi123.stdout:48: finishing write tid 1 to smithi12313934-48 2020-01-16T00:12:35.539 INFO:tasks.rados.rados.0.smithi123.stdout:48: finishing write tid 2 to smithi12313934-48 2020-01-16T00:12:35.539 INFO:tasks.rados.rados.0.smithi123.stdout:48: finishing write tid 4 to smithi12313934-48 2020-01-16T00:12:35.549 INFO:tasks.rados.rados.0.smithi123.stdout:47: done (15 left) 2020-01-16T00:12:35.549 INFO:tasks.rados.rados.0.smithi123.stdout:63: copy oid 14 target oid 14 low_tier 2020-01-16T00:12:35.549 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 14 to 14 low_tier tgt_pool_name low_tier 2020-01-16T00:12:35.549 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.550 INFO:tasks.rados.rados.0.smithi123.stdout:48: finishing write tid 5 to smithi12313934-48 2020-01-16T00:12:35.550 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 48 v 9 (ObjNum 47 snap 0 seq_num 47) dirty exists 2020-01-16T00:12:35.550 INFO:tasks.rados.rados.0.smithi123.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2020-01-16T00:12:35.550 INFO:tasks.rados.rados.0.smithi123.stdout:49: finishing write tid 1 to smithi12313934-49 2020-01-16T00:12:35.550 INFO:tasks.rados.rados.0.smithi123.stdout:49: finishing write tid 2 to smithi12313934-49 2020-01-16T00:12:35.550 INFO:tasks.rados.rados.0.smithi123.stdout:49: finishing write tid 4 to smithi12313934-49 2020-01-16T00:12:35.551 INFO:tasks.rados.rados.0.smithi123.stdout:49: finishing write tid 5 to smithi12313934-49 2020-01-16T00:12:35.551 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 49 v 9 (ObjNum 48 snap 0 seq_num 48) dirty exists 2020-01-16T00:12:35.551 INFO:tasks.rados.rados.0.smithi123.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2020-01-16T00:12:35.551 INFO:tasks.rados.rados.0.smithi123.stdout:50: finishing write tid 1 to smithi12313934-50 2020-01-16T00:12:35.551 INFO:tasks.rados.rados.0.smithi123.stdout:48: done (15 left) 2020-01-16T00:12:35.551 INFO:tasks.rados.rados.0.smithi123.stdout:49: done (14 left) 2020-01-16T00:12:35.552 INFO:tasks.rados.rados.0.smithi123.stdout:64: copy oid 15 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.552 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 15 to 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2020-01-16T00:12:35.552 INFO:tasks.rados.rados.0.smithi123.stdout:65: copy oid 16 target oid 16 low_tier 2020-01-16T00:12:35.552 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 16 to 16 low_tier tgt_pool_name low_tier 2020-01-16T00:12:35.552 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.555 INFO:tasks.rados.rados.0.smithi123.stdout:50: finishing write tid 2 to smithi12313934-50 2020-01-16T00:12:35.555 INFO:tasks.rados.rados.0.smithi123.stdout:50: finishing write tid 4 to smithi12313934-50 2020-01-16T00:12:35.555 INFO:tasks.rados.rados.0.smithi123.stdout:50: finishing write tid 5 to smithi12313934-50 2020-01-16T00:12:35.555 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 50 v 12 (ObjNum 49 snap 0 seq_num 49) dirty exists 2020-01-16T00:12:35.555 INFO:tasks.rados.rados.0.smithi123.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2020-01-16T00:12:35.555 INFO:tasks.rados.rados.0.smithi123.stdout:50: done (15 left) 2020-01-16T00:12:35.555 INFO:tasks.rados.rados.0.smithi123.stdout:66: copy oid 17 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.556 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 17 to 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2020-01-16T00:12:35.556 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.556 INFO:tasks.rados.rados.0.smithi123.stdout:55: finishing copy op to oid 6 low_tier 2020-01-16T00:12:35.556 INFO:tasks.rados.rados.0.smithi123.stdout:60: finishing copy op to oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.556 INFO:tasks.rados.rados.0.smithi123.stdout:55: done (15 left) 2020-01-16T00:12:35.556 INFO:tasks.rados.rados.0.smithi123.stdout:60: done (14 left) 2020-01-16T00:12:35.556 INFO:tasks.rados.rados.0.smithi123.stdout:67: copy oid 18 target oid 18 low_tier 2020-01-16T00:12:35.557 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 18 to 18 low_tier tgt_pool_name low_tier 2020-01-16T00:12:35.562 INFO:tasks.rados.rados.0.smithi123.stdout:68: copy oid 19 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.562 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 19 to 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2020-01-16T00:12:35.562 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.562 INFO:tasks.rados.rados.0.smithi123.stdout:63: finishing copy op to oid 14 low_tier 2020-01-16T00:12:35.562 INFO:tasks.rados.rados.0.smithi123.stdout:63: done (15 left) 2020-01-16T00:12:35.562 INFO:tasks.rados.rados.0.smithi123.stdout:69: copy oid 20 target oid 20 low_tier 2020-01-16T00:12:35.563 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 20 to 20 low_tier tgt_pool_name low_tier 2020-01-16T00:12:35.563 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.564 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:35.553+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:12:35.565 INFO:tasks.rados.rados.0.smithi123.stdout:51: finishing copy op to oid 2 low_tier 2020-01-16T00:12:35.565 INFO:tasks.rados.rados.0.smithi123.stdout:51: done (15 left) 2020-01-16T00:12:35.565 INFO:tasks.rados.rados.0.smithi123.stdout:70: copy oid 21 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.565 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 21 to 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2020-01-16T00:12:35.565 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.565 INFO:tasks.rados.rados.0.smithi123.stdout:52: finishing copy op to oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.565 INFO:tasks.rados.rados.0.smithi123.stdout:52: done (15 left) 2020-01-16T00:12:35.566 INFO:tasks.rados.rados.0.smithi123.stdout:71: copy oid 22 target oid 22 low_tier 2020-01-16T00:12:35.566 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 22 to 22 low_tier tgt_pool_name low_tier 2020-01-16T00:12:35.566 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.571 INFO:tasks.rados.rados.0.smithi123.stdout:54: finishing copy op to oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.572 INFO:tasks.rados.rados.0.smithi123.stdout:54: done (15 left) 2020-01-16T00:12:35.572 INFO:tasks.rados.rados.0.smithi123.stdout:72: copy oid 23 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.572 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 23 to 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2020-01-16T00:12:35.572 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.572 INFO:tasks.rados.rados.0.smithi123.stdout:53: finishing copy op to oid 4 low_tier 2020-01-16T00:12:35.572 INFO:tasks.rados.rados.0.smithi123.stdout:53: done (15 left) 2020-01-16T00:12:35.572 INFO:tasks.rados.rados.0.smithi123.stdout:73: copy oid 24 target oid 24 low_tier 2020-01-16T00:12:35.573 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 24 to 24 low_tier tgt_pool_name low_tier 2020-01-16T00:12:35.573 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.573 INFO:tasks.rados.rados.0.smithi123.stdout:61: finishing copy op to oid 12 low_tier 2020-01-16T00:12:35.573 INFO:tasks.rados.rados.0.smithi123.stdout:61: done (15 left) 2020-01-16T00:12:35.573 INFO:tasks.rados.rados.0.smithi123.stdout:74: copy oid 25 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.573 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 25 to 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2020-01-16T00:12:35.573 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.574 INFO:tasks.rados.rados.0.smithi123.stdout:59: finishing copy op to oid 10 low_tier 2020-01-16T00:12:35.574 INFO:tasks.rados.rados.0.smithi123.stdout:59: done (15 left) 2020-01-16T00:12:35.574 INFO:tasks.rados.rados.0.smithi123.stdout:75: copy oid 26 target oid 26 low_tier 2020-01-16T00:12:35.574 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 26 to 26 low_tier tgt_pool_name low_tier 2020-01-16T00:12:35.575 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.575 INFO:tasks.rados.rados.0.smithi123.stdout:56: finishing copy op to oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.575 INFO:tasks.rados.rados.0.smithi123.stdout:56: done (15 left) 2020-01-16T00:12:35.575 INFO:tasks.rados.rados.0.smithi123.stdout:76: copy oid 27 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.575 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 27 to 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2020-01-16T00:12:35.576 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.593 INFO:tasks.rados.rados.0.smithi123.stdout:66: finishing copy op to oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.594 INFO:tasks.rados.rados.0.smithi123.stdout:66: done (15 left) 2020-01-16T00:12:35.594 INFO:tasks.rados.rados.0.smithi123.stdout:77: copy oid 28 target oid 28 low_tier 2020-01-16T00:12:35.594 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 28 to 28 low_tier tgt_pool_name low_tier 2020-01-16T00:12:35.594 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.595 INFO:tasks.rados.rados.0.smithi123.stdout:65: finishing copy op to oid 16 low_tier 2020-01-16T00:12:35.595 INFO:tasks.rados.rados.0.smithi123.stdout:65: done (15 left) 2020-01-16T00:12:35.595 INFO:tasks.rados.rados.0.smithi123.stdout:78: copy oid 29 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.595 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 29 to 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2020-01-16T00:12:35.596 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.596 INFO:tasks.rados.rados.0.smithi123.stdout:57: finishing copy op to oid 8 low_tier 2020-01-16T00:12:35.596 INFO:tasks.rados.rados.0.smithi123.stdout:57: done (15 left) 2020-01-16T00:12:35.596 INFO:tasks.rados.rados.0.smithi123.stdout:79: copy oid 30 target oid 30 low_tier 2020-01-16T00:12:35.596 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 30 to 30 low_tier tgt_pool_name low_tier 2020-01-16T00:12:35.596 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.601 INFO:tasks.rados.rados.0.smithi123.stdout:74: finishing copy op to oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.601 INFO:tasks.rados.rados.0.smithi123.stdout:74: done (15 left) 2020-01-16T00:12:35.601 INFO:tasks.rados.rados.0.smithi123.stdout:80: copy oid 31 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.602 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 31 to 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2020-01-16T00:12:35.602 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.604 INFO:tasks.rados.rados.0.smithi123.stdout:69: finishing copy op to oid 20 low_tier 2020-01-16T00:12:35.604 INFO:tasks.rados.rados.0.smithi123.stdout:69: done (15 left) 2020-01-16T00:12:35.604 INFO:tasks.rados.rados.0.smithi123.stdout:81: copy oid 32 target oid 32 low_tier 2020-01-16T00:12:35.605 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 32 to 32 low_tier tgt_pool_name low_tier 2020-01-16T00:12:35.605 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.606 INFO:tasks.rados.rados.0.smithi123.stdout:64: finishing copy op to oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.606 INFO:tasks.rados.rados.0.smithi123.stdout:64: done (15 left) 2020-01-16T00:12:35.607 INFO:tasks.rados.rados.0.smithi123.stdout:82: copy oid 33 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.607 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 33 to 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2020-01-16T00:12:35.607 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.607 INFO:tasks.rados.rados.0.smithi123.stdout:58: finishing copy op to oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.607 INFO:tasks.rados.rados.0.smithi123.stdout:58: done (15 left) 2020-01-16T00:12:35.607 INFO:tasks.rados.rados.0.smithi123.stdout:83: copy oid 34 target oid 34 low_tier 2020-01-16T00:12:35.607 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 34 to 34 low_tier tgt_pool_name low_tier 2020-01-16T00:12:35.608 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.608 INFO:tasks.rados.rados.0.smithi123.stdout:62: finishing copy op to oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.608 INFO:tasks.rados.rados.0.smithi123.stdout:62: done (15 left) 2020-01-16T00:12:35.608 INFO:tasks.rados.rados.0.smithi123.stdout:84: copy oid 35 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.608 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 35 to 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2020-01-16T00:12:35.608 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.608 INFO:tasks.rados.rados.0.smithi123.stdout:68: finishing copy op to oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.608 INFO:tasks.rados.rados.0.smithi123.stdout:68: done (15 left) 2020-01-16T00:12:35.609 INFO:tasks.rados.rados.0.smithi123.stdout:85: copy oid 36 target oid 36 low_tier 2020-01-16T00:12:35.609 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 36 to 36 low_tier tgt_pool_name low_tier 2020-01-16T00:12:35.609 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.609 INFO:tasks.rados.rados.0.smithi123.stdout:73: finishing copy op to oid 24 low_tier 2020-01-16T00:12:35.609 INFO:tasks.rados.rados.0.smithi123.stdout:73: done (15 left) 2020-01-16T00:12:35.610 INFO:tasks.rados.rados.0.smithi123.stdout:86: copy oid 37 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.610 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 37 to 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2020-01-16T00:12:35.610 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.611 INFO:tasks.rados.rados.0.smithi123.stdout:70: finishing copy op to oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.612 INFO:tasks.rados.rados.0.smithi123.stdout:70: done (15 left) 2020-01-16T00:12:35.612 INFO:tasks.rados.rados.0.smithi123.stdout:87: copy oid 38 target oid 38 low_tier 2020-01-16T00:12:35.612 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 38 to 38 low_tier tgt_pool_name low_tier 2020-01-16T00:12:35.612 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.614 INFO:tasks.rados.rados.0.smithi123.stdout:72: finishing copy op to oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.614 INFO:tasks.rados.rados.0.smithi123.stdout:75: finishing copy op to oid 26 low_tier 2020-01-16T00:12:35.614 INFO:tasks.rados.rados.0.smithi123.stdout:77: finishing copy op to oid 28 low_tier 2020-01-16T00:12:35.614 INFO:tasks.rados.rados.0.smithi123.stdout:72: done (15 left) 2020-01-16T00:12:35.614 INFO:tasks.rados.rados.0.smithi123.stdout:75: done (14 left) 2020-01-16T00:12:35.615 INFO:tasks.rados.rados.0.smithi123.stdout:77: done (13 left) 2020-01-16T00:12:35.615 INFO:tasks.rados.rados.0.smithi123.stdout:88: copy oid 39 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.615 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 39 to 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2020-01-16T00:12:35.615 INFO:tasks.rados.rados.0.smithi123.stdout:89: copy oid 40 target oid 40 low_tier 2020-01-16T00:12:35.615 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 40 to 40 low_tier tgt_pool_name low_tier 2020-01-16T00:12:35.615 INFO:tasks.rados.rados.0.smithi123.stdout:90: copy oid 41 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.615 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 41 to 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2020-01-16T00:12:35.616 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.618 INFO:tasks.rados.rados.0.smithi123.stdout:67: finishing copy op to oid 18 low_tier 2020-01-16T00:12:35.618 INFO:tasks.rados.rados.0.smithi123.stdout:67: done (15 left) 2020-01-16T00:12:35.618 INFO:tasks.rados.rados.0.smithi123.stdout:91: copy oid 42 target oid 42 low_tier 2020-01-16T00:12:35.619 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 42 to 42 low_tier tgt_pool_name low_tier 2020-01-16T00:12:35.619 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.619 INFO:tasks.rados.rados.0.smithi123.stdout:71: finishing copy op to oid 22 low_tier 2020-01-16T00:12:35.619 INFO:tasks.rados.rados.0.smithi123.stdout:71: done (15 left) 2020-01-16T00:12:35.619 INFO:tasks.rados.rados.0.smithi123.stdout:92: copy oid 43 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.619 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 43 to 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2020-01-16T00:12:35.619 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.620 INFO:tasks.rados.rados.0.smithi123.stdout:76: finishing copy op to oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.620 INFO:tasks.rados.rados.0.smithi123.stdout:76: done (15 left) 2020-01-16T00:12:35.620 INFO:tasks.rados.rados.0.smithi123.stdout:93: copy oid 44 target oid 44 low_tier 2020-01-16T00:12:35.620 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 44 to 44 low_tier tgt_pool_name low_tier 2020-01-16T00:12:35.620 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.628 INFO:tasks.rados.rados.0.smithi123.stdout:78: finishing copy op to oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.628 INFO:tasks.rados.rados.0.smithi123.stdout:78: done (15 left) 2020-01-16T00:12:35.629 INFO:tasks.rados.rados.0.smithi123.stdout:94: copy oid 45 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.629 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 45 to 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2020-01-16T00:12:35.629 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.636 INFO:tasks.rados.rados.0.smithi123.stdout:79: finishing copy op to oid 30 low_tier 2020-01-16T00:12:35.636 INFO:tasks.rados.rados.0.smithi123.stdout:79: done (15 left) 2020-01-16T00:12:35.636 INFO:tasks.rados.rados.0.smithi123.stdout:95: copy oid 46 target oid 46 low_tier 2020-01-16T00:12:35.636 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 46 to 46 low_tier tgt_pool_name low_tier 2020-01-16T00:12:35.636 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.642 INFO:tasks.rados.rados.0.smithi123.stdout:84: finishing copy op to oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.643 INFO:tasks.rados.rados.0.smithi123.stdout:84: done (15 left) 2020-01-16T00:12:35.643 INFO:tasks.rados.rados.0.smithi123.stdout:96: copy oid 47 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.643 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 47 to 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2020-01-16T00:12:35.643 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.654 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:12:35.653+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:12:35.669 INFO:tasks.rados.rados.0.smithi123.stdout:80: finishing copy op to oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.670 INFO:tasks.rados.rados.0.smithi123.stdout:80: done (15 left) 2020-01-16T00:12:35.670 INFO:tasks.rados.rados.0.smithi123.stdout:97: copy oid 48 target oid 48 low_tier 2020-01-16T00:12:35.670 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 48 to 48 low_tier tgt_pool_name low_tier 2020-01-16T00:12:35.670 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.682 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:35.684 INFO:tasks.rados.rados.0.smithi123.stdout:81: finishing copy op to oid 32 low_tier 2020-01-16T00:12:35.684 INFO:tasks.rados.rados.0.smithi123.stdout:81: done (15 left) 2020-01-16T00:12:35.684 INFO:tasks.rados.rados.0.smithi123.stdout:98: copy oid 49 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.685 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 49 to 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2020-01-16T00:12:35.685 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.691 INFO:tasks.rados.rados.0.smithi123.stdout:82: finishing copy op to oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.691 INFO:tasks.rados.rados.0.smithi123.stdout:82: done (15 left) 2020-01-16T00:12:35.691 INFO:tasks.rados.rados.0.smithi123.stdout:99: copy oid 50 target oid 50 low_tier 2020-01-16T00:12:35.691 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 50 to 50 low_tier tgt_pool_name low_tier 2020-01-16T00:12:35.691 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.692 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-01-16T00:12:35.699 INFO:tasks.rados.rados.0.smithi123.stdout:83: finishing copy op to oid 34 low_tier 2020-01-16T00:12:35.699 INFO:tasks.rados.rados.0.smithi123.stdout:83: done (15 left) 2020-01-16T00:12:35.699 INFO:tasks.rados.rados.0.smithi123.stdout:100: copy oid 1 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.699 INFO:tasks.rados.rados.0.smithi123.stdout:copy op oid 1 to 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_pool_name low_tier 2020-01-16T00:12:35.699 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.702 INFO:tasks.rados.rados.0.smithi123.stdout:86: finishing copy op to oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.703 INFO:tasks.rados.rados.0.smithi123.stdout:86: done (15 left) 2020-01-16T00:12:35.703 INFO:tasks.rados.rados.0.smithi123.stdout:101: set_chunk oid 2 offset: 476160 length: 1240064 target oid 2 low_tier tgt_offset: 476160 2020-01-16T00:12:35.703 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.715 INFO:tasks.rados.rados.0.smithi123.stdout:85: finishing copy op to oid 36 low_tier 2020-01-16T00:12:35.715 INFO:tasks.rados.rados.0.smithi123.stdout:85: done (15 left) 2020-01-16T00:12:35.715 INFO:tasks.rados.rados.0.smithi123.stdout:102: set_chunk oid 3 offset: 312320 length: 603136 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 312320 2020-01-16T00:12:35.715 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.718 INFO:tasks.rados.rados.0.smithi123.stdout:88: finishing copy op to oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.719 INFO:tasks.rados.rados.0.smithi123.stdout:94: finishing copy op to oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.720 INFO:tasks.rados.rados.0.smithi123.stdout:101: finishing set_chunk to oid 2 2020-01-16T00:12:35.720 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 2 v 13 (ObjNum 1 snap 0 seq_num 1) dirty exists 2020-01-16T00:12:35.720 INFO:tasks.rados.rados.0.smithi123.stdout:88: done (15 left) 2020-01-16T00:12:35.720 INFO:tasks.rados.rados.0.smithi123.stdout:94: done (14 left) 2020-01-16T00:12:35.720 INFO:tasks.rados.rados.0.smithi123.stdout:101: done (13 left) 2020-01-16T00:12:35.720 INFO:tasks.rados.rados.0.smithi123.stdout:103: set_chunk oid 4 offset: 1020416 length: 515584 target oid 4 low_tier tgt_offset: 1020416 2020-01-16T00:12:35.720 INFO:tasks.rados.rados.0.smithi123.stdout:104: set_chunk oid 5 offset: 1779712 length: 28672 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1779712 2020-01-16T00:12:35.720 INFO:tasks.rados.rados.0.smithi123.stdout:105: set_chunk oid 6 offset: 878592 length: 645632 target oid 6 low_tier tgt_offset: 878592 2020-01-16T00:12:35.721 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.724 INFO:tasks.rados.rados.0.smithi123.stdout:87: finishing copy op to oid 38 low_tier 2020-01-16T00:12:35.724 INFO:tasks.rados.rados.0.smithi123.stdout:96: finishing copy op to oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.724 INFO:tasks.rados.rados.0.smithi123.stdout:91: finishing copy op to oid 42 low_tier 2020-01-16T00:12:35.724 INFO:tasks.rados.rados.0.smithi123.stdout:87: done (15 left) 2020-01-16T00:12:35.724 INFO:tasks.rados.rados.0.smithi123.stdout:91: done (14 left) 2020-01-16T00:12:35.725 INFO:tasks.rados.rados.0.smithi123.stdout:96: done (13 left) 2020-01-16T00:12:35.725 INFO:tasks.rados.rados.0.smithi123.stdout:106: set_chunk oid 7 offset: 160768 length: 1134080 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 160768 2020-01-16T00:12:35.725 INFO:tasks.rados.rados.0.smithi123.stdout:107: set_chunk oid 8 offset: 1589248 length: 433664 target oid 8 low_tier tgt_offset: 1589248 2020-01-16T00:12:35.725 INFO:tasks.rados.rados.0.smithi123.stdout:108: set_chunk oid 9 offset: 1072128 length: 499200 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1072128 2020-01-16T00:12:35.725 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.725 INFO:tasks.rados.rados.0.smithi123.stdout:90: finishing copy op to oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.725 INFO:tasks.rados.rados.0.smithi123.stdout:90: done (15 left) 2020-01-16T00:12:35.726 INFO:tasks.rados.rados.0.smithi123.stdout:109: set_chunk oid 10 offset: 453120 length: 360448 target oid 10 low_tier tgt_offset: 453120 2020-01-16T00:12:35.726 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.726 INFO:tasks.rados.rados.0.smithi123.stdout:97: finishing copy op to oid 48 low_tier 2020-01-16T00:12:35.726 INFO:tasks.rados.rados.0.smithi123.stdout:97: done (15 left) 2020-01-16T00:12:35.726 INFO:tasks.rados.rados.0.smithi123.stdout:110: set_chunk oid 11 offset: 279552 length: 1427968 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 279552 2020-01-16T00:12:35.727 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.727 INFO:tasks.rados.rados.0.smithi123.stdout:89: finishing copy op to oid 40 low_tier 2020-01-16T00:12:35.727 INFO:tasks.rados.rados.0.smithi123.stdout:89: done (15 left) 2020-01-16T00:12:35.727 INFO:tasks.rados.rados.0.smithi123.stdout:111: set_chunk oid 12 offset: 1033216 length: 672768 target oid 12 low_tier tgt_offset: 1033216 2020-01-16T00:12:35.727 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.727 INFO:tasks.rados.rados.0.smithi123.stdout:104: finishing set_chunk to oid 5 2020-01-16T00:12:35.727 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 5 v 16 (ObjNum 4 snap 0 seq_num 4) dirty exists 2020-01-16T00:12:35.727 INFO:tasks.rados.rados.0.smithi123.stdout:104: done (15 left) 2020-01-16T00:12:35.728 INFO:tasks.rados.rados.0.smithi123.stdout:112: set_chunk oid 13 offset: 916992 length: 596480 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 916992 2020-01-16T00:12:35.728 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.728 INFO:tasks.rados.rados.0.smithi123.stdout:95: finishing copy op to oid 46 low_tier 2020-01-16T00:12:35.728 INFO:tasks.rados.rados.0.smithi123.stdout:95: done (15 left) 2020-01-16T00:12:35.728 INFO:tasks.rados.rados.0.smithi123.stdout:113: set_chunk oid 14 offset: 470528 length: 1206272 target oid 14 low_tier tgt_offset: 470528 2020-01-16T00:12:35.728 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.728 INFO:tasks.rados.rados.0.smithi123.stdout:102: finishing set_chunk to oid 3 2020-01-16T00:12:35.728 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 3 v 7 (ObjNum 2 snap 0 seq_num 2) dirty exists 2020-01-16T00:12:35.729 INFO:tasks.rados.rados.0.smithi123.stdout:105: finishing set_chunk to oid 6 2020-01-16T00:12:35.729 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 6 v 10 (ObjNum 5 snap 0 seq_num 5) dirty exists 2020-01-16T00:12:35.729 INFO:tasks.rados.rados.0.smithi123.stdout:109: finishing set_chunk to oid 10 2020-01-16T00:12:35.729 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 10 v 10 (ObjNum 9 snap 0 seq_num 9) dirty exists 2020-01-16T00:12:35.729 INFO:tasks.rados.rados.0.smithi123.stdout:102: done (15 left) 2020-01-16T00:12:35.729 INFO:tasks.rados.rados.0.smithi123.stdout:105: done (14 left) 2020-01-16T00:12:35.729 INFO:tasks.rados.rados.0.smithi123.stdout:109: done (13 left) 2020-01-16T00:12:35.730 INFO:tasks.rados.rados.0.smithi123.stdout:114: set_chunk oid 15 offset: 331264 length: 198656 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 331264 2020-01-16T00:12:35.730 INFO:tasks.rados.rados.0.smithi123.stdout:115: set_chunk oid 16 offset: 833536 length: 364544 target oid 16 low_tier tgt_offset: 833536 2020-01-16T00:12:35.730 INFO:tasks.rados.rados.0.smithi123.stdout:116: set_chunk oid 17 offset: 330752 length: 378368 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 330752 2020-01-16T00:12:35.731 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.731 INFO:tasks.rados.rados.0.smithi123.stdout:106: finishing set_chunk to oid 7 2020-01-16T00:12:35.731 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 7 v 14 (ObjNum 6 snap 0 seq_num 6) dirty exists 2020-01-16T00:12:35.731 INFO:tasks.rados.rados.0.smithi123.stdout:106: done (15 left) 2020-01-16T00:12:35.731 INFO:tasks.rados.rados.0.smithi123.stdout:117: set_chunk oid 18 offset: 953856 length: 471040 target oid 18 low_tier tgt_offset: 953856 2020-01-16T00:12:35.731 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.731 INFO:tasks.rados.rados.0.smithi123.stdout:107: finishing set_chunk to oid 8 2020-01-16T00:12:35.731 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 8 v 8 (ObjNum 7 snap 0 seq_num 7) dirty exists 2020-01-16T00:12:35.732 INFO:tasks.rados.rados.0.smithi123.stdout:107: done (15 left) 2020-01-16T00:12:35.732 INFO:tasks.rados.rados.0.smithi123.stdout:118: set_chunk oid 19 offset: 592384 length: 1625600 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 592384 2020-01-16T00:12:35.732 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.838 INFO:tasks.rados.rados.0.smithi123.stdout:114: finishing set_chunk to oid 15 2020-01-16T00:12:35.839 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 15 v 13 (ObjNum 14 snap 0 seq_num 14) dirty exists 2020-01-16T00:12:35.839 INFO:tasks.rados.rados.0.smithi123.stdout:114: done (15 left) 2020-01-16T00:12:35.839 INFO:tasks.rados.rados.0.smithi123.stdout:119: set_chunk oid 20 offset: 59392 length: 257024 target oid 20 low_tier tgt_offset: 59392 2020-01-16T00:12:35.839 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.839 INFO:tasks.rados.rados.0.smithi123.stdout:98: finishing copy op to oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.839 INFO:tasks.rados.rados.0.smithi123.stdout:99: finishing copy op to oid 50 low_tier 2020-01-16T00:12:35.839 INFO:tasks.rados.rados.0.smithi123.stdout:98: done (15 left) 2020-01-16T00:12:35.840 INFO:tasks.rados.rados.0.smithi123.stdout:99: done (14 left) 2020-01-16T00:12:35.840 INFO:tasks.rados.rados.0.smithi123.stdout:120: set_chunk oid 21 offset: 2100224 length: 34304 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 2100224 2020-01-16T00:12:35.840 INFO:tasks.rados.rados.0.smithi123.stdout:121: set_chunk oid 22 offset: 1822720 length: 105472 target oid 22 low_tier tgt_offset: 1822720 2020-01-16T00:12:35.840 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.840 INFO:tasks.rados.rados.0.smithi123.stdout:103: finishing set_chunk to oid 4 2020-01-16T00:12:35.840 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 4 v 13 (ObjNum 3 snap 0 seq_num 3) dirty exists 2020-01-16T00:12:35.840 INFO:tasks.rados.rados.0.smithi123.stdout:117: finishing set_chunk to oid 18 2020-01-16T00:12:35.841 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 18 v 19 (ObjNum 17 snap 0 seq_num 17) dirty exists 2020-01-16T00:12:35.841 INFO:tasks.rados.rados.0.smithi123.stdout:103: done (15 left) 2020-01-16T00:12:35.841 INFO:tasks.rados.rados.0.smithi123.stdout:117: done (14 left) 2020-01-16T00:12:35.841 INFO:tasks.rados.rados.0.smithi123.stdout:122: set_chunk oid 23 offset: 808960 length: 1406976 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 808960 2020-01-16T00:12:35.841 INFO:tasks.rados.rados.0.smithi123.stdout:123: set_chunk oid 24 offset: 30720 length: 1585664 target oid 24 low_tier tgt_offset: 30720 2020-01-16T00:12:35.841 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.841 INFO:tasks.rados.rados.0.smithi123.stdout:116: finishing set_chunk to oid 17 2020-01-16T00:12:35.842 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 17 v 14 (ObjNum 16 snap 0 seq_num 16) dirty exists 2020-01-16T00:12:35.842 INFO:tasks.rados.rados.0.smithi123.stdout:116: done (15 left) 2020-01-16T00:12:35.842 INFO:tasks.rados.rados.0.smithi123.stdout:124: set_chunk oid 25 offset: 84992 length: 680448 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 84992 2020-01-16T00:12:35.842 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.842 INFO:tasks.rados.rados.0.smithi123.stdout:112: finishing set_chunk to oid 13 2020-01-16T00:12:35.842 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 7 (ObjNum 12 snap 0 seq_num 12) dirty exists 2020-01-16T00:12:35.842 INFO:tasks.rados.rados.0.smithi123.stdout:112: done (15 left) 2020-01-16T00:12:35.843 INFO:tasks.rados.rados.0.smithi123.stdout:125: set_chunk oid 26 offset: 1494016 length: 315392 target oid 26 low_tier tgt_offset: 1494016 2020-01-16T00:12:35.843 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.843 INFO:tasks.rados.rados.0.smithi123.stdout:118: finishing set_chunk to oid 19 2020-01-16T00:12:35.843 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 19 v 15 (ObjNum 18 snap 0 seq_num 18) dirty exists 2020-01-16T00:12:35.843 INFO:tasks.rados.rados.0.smithi123.stdout:118: done (15 left) 2020-01-16T00:12:35.843 INFO:tasks.rados.rados.0.smithi123.stdout:126: set_chunk oid 27 offset: 113152 length: 1699840 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 113152 2020-01-16T00:12:35.843 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.844 INFO:tasks.rados.rados.0.smithi123.stdout:113: finishing set_chunk to oid 14 2020-01-16T00:12:35.844 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 14 v 16 (ObjNum 13 snap 0 seq_num 13) dirty exists 2020-01-16T00:12:35.844 INFO:tasks.rados.rados.0.smithi123.stdout:113: done (15 left) 2020-01-16T00:12:35.844 INFO:tasks.rados.rados.0.smithi123.stdout:127: set_chunk oid 28 offset: 1498624 length: 783872 target oid 28 low_tier tgt_offset: 1498624 2020-01-16T00:12:35.844 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.844 INFO:tasks.rados.rados.0.smithi123.stdout:110: finishing set_chunk to oid 11 2020-01-16T00:12:35.844 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 11 v 11 (ObjNum 10 snap 0 seq_num 10) dirty exists 2020-01-16T00:12:35.845 INFO:tasks.rados.rados.0.smithi123.stdout:111: finishing set_chunk to oid 12 2020-01-16T00:12:35.845 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 12 v 12 (ObjNum 11 snap 0 seq_num 11) dirty exists 2020-01-16T00:12:35.845 INFO:tasks.rados.rados.0.smithi123.stdout:110: done (15 left) 2020-01-16T00:12:35.845 INFO:tasks.rados.rados.0.smithi123.stdout:111: done (14 left) 2020-01-16T00:12:35.845 INFO:tasks.rados.rados.0.smithi123.stdout:128: set_chunk oid 29 offset: 1186304 length: 657408 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1186304 2020-01-16T00:12:35.845 INFO:tasks.rados.rados.0.smithi123.stdout:129: set_chunk oid 30 offset: 1542656 length: 711168 target oid 30 low_tier tgt_offset: 1542656 2020-01-16T00:12:35.845 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.845 INFO:tasks.rados.rados.0.smithi123.stdout:120: finishing set_chunk to oid 21 2020-01-16T00:12:35.846 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 21 v 7 (ObjNum 20 snap 0 seq_num 20) dirty exists 2020-01-16T00:12:35.846 INFO:tasks.rados.rados.0.smithi123.stdout:120: done (15 left) 2020-01-16T00:12:35.846 INFO:tasks.rados.rados.0.smithi123.stdout:130: set_chunk oid 31 offset: 263680 length: 1668096 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 263680 2020-01-16T00:12:35.846 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.846 INFO:tasks.rados.rados.0.smithi123.stdout:123: finishing set_chunk to oid 24 2020-01-16T00:12:35.846 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 24 v 10 (ObjNum 23 snap 0 seq_num 23) dirty exists 2020-01-16T00:12:35.846 INFO:tasks.rados.rados.0.smithi123.stdout:125: finishing set_chunk to oid 26 2020-01-16T00:12:35.847 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 26 v 20 (ObjNum 25 snap 0 seq_num 25) dirty exists 2020-01-16T00:12:35.847 INFO:tasks.rados.rados.0.smithi123.stdout:92: finishing copy op to oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.847 INFO:tasks.rados.rados.0.smithi123.stdout:121: finishing set_chunk to oid 22 2020-01-16T00:12:35.847 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 22 v 4 (ObjNum 21 snap 0 seq_num 21) dirty exists 2020-01-16T00:12:35.847 INFO:tasks.rados.rados.0.smithi123.stdout:93: finishing copy op to oid 44 low_tier 2020-01-16T00:12:35.847 INFO:tasks.rados.rados.0.smithi123.stdout:108: finishing set_chunk to oid 9 2020-01-16T00:12:35.847 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 9 v 10 (ObjNum 8 snap 0 seq_num 8) dirty exists 2020-01-16T00:12:35.848 INFO:tasks.rados.rados.0.smithi123.stdout:100: finishing copy op to oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:35.848 INFO:tasks.rados.rados.0.smithi123.stdout:92: done (15 left) 2020-01-16T00:12:35.848 INFO:tasks.rados.rados.0.smithi123.stdout:93: done (14 left) 2020-01-16T00:12:35.848 INFO:tasks.rados.rados.0.smithi123.stdout:100: done (13 left) 2020-01-16T00:12:35.848 INFO:tasks.rados.rados.0.smithi123.stdout:108: done (12 left) 2020-01-16T00:12:35.848 INFO:tasks.rados.rados.0.smithi123.stdout:121: done (11 left) 2020-01-16T00:12:35.848 INFO:tasks.rados.rados.0.smithi123.stdout:123: done (10 left) 2020-01-16T00:12:35.848 INFO:tasks.rados.rados.0.smithi123.stdout:125: done (9 left) 2020-01-16T00:12:35.849 INFO:tasks.rados.rados.0.smithi123.stdout:131: set_chunk oid 32 offset: 605696 length: 877568 target oid 32 low_tier tgt_offset: 605696 2020-01-16T00:12:35.849 INFO:tasks.rados.rados.0.smithi123.stdout:132: set_chunk oid 33 offset: 658944 length: 1355776 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 658944 2020-01-16T00:12:35.849 INFO:tasks.rados.rados.0.smithi123.stdout:133: set_chunk oid 34 offset: 901632 length: 1154560 target oid 34 low_tier tgt_offset: 901632 2020-01-16T00:12:35.849 INFO:tasks.rados.rados.0.smithi123.stdout:134: set_chunk oid 35 offset: 1823744 length: 153600 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1823744 2020-01-16T00:12:35.849 INFO:tasks.rados.rados.0.smithi123.stdout:135: set_chunk oid 36 offset: 643584 length: 327680 target oid 36 low_tier tgt_offset: 643584 2020-01-16T00:12:35.849 INFO:tasks.rados.rados.0.smithi123.stdout:136: set_chunk oid 37 offset: 574464 length: 1014784 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 574464 2020-01-16T00:12:35.849 INFO:tasks.rados.rados.0.smithi123.stdout:137: set_chunk oid 38 offset: 1915392 length: 653312 target oid 38 low_tier tgt_offset: 1915392 2020-01-16T00:12:35.850 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.850 INFO:tasks.rados.rados.0.smithi123.stdout:115: finishing set_chunk to oid 16 2020-01-16T00:12:35.850 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 16 v 17 (ObjNum 15 snap 0 seq_num 15) dirty exists 2020-01-16T00:12:35.850 INFO:tasks.rados.rados.0.smithi123.stdout:115: done (15 left) 2020-01-16T00:12:35.850 INFO:tasks.rados.rados.0.smithi123.stdout:138: set_chunk oid 39 offset: 122368 length: 11264 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 122368 2020-01-16T00:12:35.850 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.850 INFO:tasks.rados.rados.0.smithi123.stdout:128: finishing set_chunk to oid 29 2020-01-16T00:12:35.851 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 29 v 11 (ObjNum 28 snap 0 seq_num 28) dirty exists 2020-01-16T00:12:35.851 INFO:tasks.rados.rados.0.smithi123.stdout:128: done (15 left) 2020-01-16T00:12:35.851 INFO:tasks.rados.rados.0.smithi123.stdout:139: set_chunk oid 40 offset: 1926656 length: 67072 target oid 40 low_tier tgt_offset: 1926656 2020-01-16T00:12:35.851 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.851 INFO:tasks.rados.rados.0.smithi123.stdout:119: finishing set_chunk to oid 20 2020-01-16T00:12:35.851 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 20 v 18 (ObjNum 19 snap 0 seq_num 19) dirty exists 2020-01-16T00:12:35.851 INFO:tasks.rados.rados.0.smithi123.stdout:119: done (15 left) 2020-01-16T00:12:35.851 INFO:tasks.rados.rados.0.smithi123.stdout:140: set_chunk oid 41 offset: 925184 length: 293376 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 925184 2020-01-16T00:12:35.852 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.852 INFO:tasks.rados.rados.0.smithi123.stdout:129: finishing set_chunk to oid 30 2020-01-16T00:12:35.852 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 30 v 12 (ObjNum 29 snap 0 seq_num 29) dirty exists 2020-01-16T00:12:35.852 INFO:tasks.rados.rados.0.smithi123.stdout:129: done (15 left) 2020-01-16T00:12:35.852 INFO:tasks.rados.rados.0.smithi123.stdout:141: set_chunk oid 42 offset: 1364992 length: 1145344 target oid 42 low_tier tgt_offset: 1364992 2020-01-16T00:12:35.852 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.852 INFO:tasks.rados.rados.0.smithi123.stdout:132: finishing set_chunk to oid 33 2020-01-16T00:12:35.852 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 33 v 21 (ObjNum 32 snap 0 seq_num 32) dirty exists 2020-01-16T00:12:35.853 INFO:tasks.rados.rados.0.smithi123.stdout:132: done (15 left) 2020-01-16T00:12:35.853 INFO:tasks.rados.rados.0.smithi123.stdout:142: set_chunk oid 43 offset: 2331136 length: 192000 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 2331136 2020-01-16T00:12:35.853 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.853 INFO:tasks.rados.rados.0.smithi123.stdout:124: finishing set_chunk to oid 25 2020-01-16T00:12:35.853 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 25 v 19 (ObjNum 24 snap 0 seq_num 24) dirty exists 2020-01-16T00:12:35.853 INFO:tasks.rados.rados.0.smithi123.stdout:124: done (15 left) 2020-01-16T00:12:35.853 INFO:tasks.rados.rados.0.smithi123.stdout:143: set_chunk oid 44 offset: 169472 length: 818176 target oid 44 low_tier tgt_offset: 169472 2020-01-16T00:12:35.853 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.854 INFO:tasks.rados.rados.0.smithi123.stdout:135: finishing set_chunk to oid 36 2020-01-16T00:12:35.854 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 36 v 8 (ObjNum 35 snap 0 seq_num 35) dirty exists 2020-01-16T00:12:35.854 INFO:tasks.rados.rados.0.smithi123.stdout:135: done (15 left) 2020-01-16T00:12:35.854 INFO:tasks.rados.rados.0.smithi123.stdout:144: set_chunk oid 45 offset: 640512 length: 403968 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 640512 2020-01-16T00:12:35.854 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.854 INFO:tasks.rados.rados.0.smithi123.stdout:127: finishing set_chunk to oid 28 2020-01-16T00:12:35.854 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 28 v 14 (ObjNum 27 snap 0 seq_num 27) dirty exists 2020-01-16T00:12:35.854 INFO:tasks.rados.rados.0.smithi123.stdout:126: finishing set_chunk to oid 27 2020-01-16T00:12:35.855 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 27 v 11 (ObjNum 26 snap 0 seq_num 26) dirty exists 2020-01-16T00:12:35.855 INFO:tasks.rados.rados.0.smithi123.stdout:136: finishing set_chunk to oid 37 2020-01-16T00:12:35.855 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 37 v 22 (ObjNum 36 snap 0 seq_num 36) dirty exists 2020-01-16T00:12:35.855 INFO:tasks.rados.rados.0.smithi123.stdout:130: finishing set_chunk to oid 31 2020-01-16T00:12:35.855 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 31 v 15 (ObjNum 30 snap 0 seq_num 30) dirty exists 2020-01-16T00:12:35.855 INFO:tasks.rados.rados.0.smithi123.stdout:131: finishing set_chunk to oid 32 2020-01-16T00:12:35.855 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 32 v 12 (ObjNum 31 snap 0 seq_num 31) dirty exists 2020-01-16T00:12:35.856 INFO:tasks.rados.rados.0.smithi123.stdout:126: done (15 left) 2020-01-16T00:12:35.856 INFO:tasks.rados.rados.0.smithi123.stdout:127: done (14 left) 2020-01-16T00:12:35.856 INFO:tasks.rados.rados.0.smithi123.stdout:130: done (13 left) 2020-01-16T00:12:35.856 INFO:tasks.rados.rados.0.smithi123.stdout:131: done (12 left) 2020-01-16T00:12:35.856 INFO:tasks.rados.rados.0.smithi123.stdout:136: done (11 left) 2020-01-16T00:12:35.856 INFO:tasks.rados.rados.0.smithi123.stdout:145: set_chunk oid 46 offset: 56832 length: 1408000 target oid 46 low_tier tgt_offset: 56832 2020-01-16T00:12:35.856 INFO:tasks.rados.rados.0.smithi123.stdout:146: set_chunk oid 47 offset: 2092032 length: 38400 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 2092032 2020-01-16T00:12:35.857 INFO:tasks.rados.rados.0.smithi123.stdout:122: finishing set_chunk to oid 23 2020-01-16T00:12:35.857 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 23 v 15 (ObjNum 22 snap 0 seq_num 22) dirty exists 2020-01-16T00:12:35.857 INFO:tasks.rados.rados.0.smithi123.stdout:122: done (12 left) 2020-01-16T00:12:35.857 INFO:tasks.rados.rados.0.smithi123.stdout:147: set_chunk oid 48 offset: 264704 length: 1399296 target oid 48 low_tier tgt_offset: 264704 2020-01-16T00:12:35.857 INFO:tasks.rados.rados.0.smithi123.stdout:148: set_chunk oid 49 offset: 453632 length: 1075200 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 453632 2020-01-16T00:12:35.857 INFO:tasks.rados.rados.0.smithi123.stdout:149: set_chunk oid 50 offset: 1613312 length: 498688 target oid 50 low_tier tgt_offset: 1613312 2020-01-16T00:12:35.857 INFO:tasks.rados.rados.0.smithi123.stdout:150: set_chunk oid 1 offset: 1046016 length: 729088 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1046016 2020-01-16T00:12:35.857 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.858 INFO:tasks.rados.rados.0.smithi123.stdout:134: finishing set_chunk to oid 35 2020-01-16T00:12:35.858 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 35 v 16 (ObjNum 34 snap 0 seq_num 34) dirty exists 2020-01-16T00:12:35.858 INFO:tasks.rados.rados.0.smithi123.stdout:134: done (15 left) 2020-01-16T00:12:35.858 INFO:tasks.rados.rados.0.smithi123.stdout:151: set_chunk oid 2 offset: 276992 length: 1268736 target oid 2 low_tier tgt_offset: 276992 2020-01-16T00:12:35.858 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.858 INFO:tasks.rados.rados.0.smithi123.stdout:133: finishing set_chunk to oid 34 2020-01-16T00:12:35.858 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 34 v 8 (ObjNum 33 snap 0 seq_num 33) dirty exists 2020-01-16T00:12:35.858 INFO:tasks.rados.rados.0.smithi123.stdout:133: done (15 left) 2020-01-16T00:12:35.859 INFO:tasks.rados.rados.0.smithi123.stdout:152: set_chunk oid 3 offset: 12288 length: 479744 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 12288 2020-01-16T00:12:35.859 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.859 INFO:tasks.rados.rados.0.smithi123.stdout:137: finishing set_chunk to oid 38 2020-01-16T00:12:35.859 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 38 v 10 (ObjNum 37 snap 0 seq_num 37) dirty exists 2020-01-16T00:12:35.859 INFO:tasks.rados.rados.0.smithi123.stdout:141: finishing set_chunk to oid 42 2020-01-16T00:12:35.859 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 42 v 11 (ObjNum 41 snap 0 seq_num 41) dirty exists 2020-01-16T00:12:35.859 INFO:tasks.rados.rados.0.smithi123.stdout:137: done (15 left) 2020-01-16T00:12:35.859 INFO:tasks.rados.rados.0.smithi123.stdout:141: done (14 left) 2020-01-16T00:12:35.860 INFO:tasks.rados.rados.0.smithi123.stdout:153: set_chunk oid 4 offset: 1110528 length: 221184 target oid 4 low_tier tgt_offset: 1110528 2020-01-16T00:12:35.860 INFO:tasks.rados.rados.0.smithi123.stdout:154: set_chunk oid 5 offset: 102400 length: 870912 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 102400 2020-01-16T00:12:35.860 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.860 INFO:tasks.rados.rados.0.smithi123.stdout:140: finishing set_chunk to oid 41 2020-01-16T00:12:35.860 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 41 v 16 (ObjNum 40 snap 0 seq_num 40) dirty exists 2020-01-16T00:12:35.860 INFO:tasks.rados.rados.0.smithi123.stdout:140: done (15 left) 2020-01-16T00:12:35.860 INFO:tasks.rados.rados.0.smithi123.stdout:155: set_chunk oid 6 offset: 915968 length: 908800 target oid 6 low_tier tgt_offset: 915968 2020-01-16T00:12:35.861 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.861 INFO:tasks.rados.rados.0.smithi123.stdout:145: finishing set_chunk to oid 46 2020-01-16T00:12:35.861 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 46 v 23 (ObjNum 45 snap 0 seq_num 45) dirty exists 2020-01-16T00:12:35.861 INFO:tasks.rados.rados.0.smithi123.stdout:145: done (15 left) 2020-01-16T00:12:35.861 INFO:tasks.rados.rados.0.smithi123.stdout:156: set_chunk oid 7 offset: 55296 length: 1907200 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 55296 2020-01-16T00:12:35.861 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.862 INFO:tasks.rados.rados.0.smithi123.stdout:144: finishing set_chunk to oid 45 2020-01-16T00:12:35.862 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 45 v 11 (ObjNum 44 snap 0 seq_num 44) dirty exists 2020-01-16T00:12:35.862 INFO:tasks.rados.rados.0.smithi123.stdout:144: done (15 left) 2020-01-16T00:12:35.862 INFO:tasks.rados.rados.0.smithi123.stdout:157: set_chunk oid 8 offset: 285696 length: 965632 target oid 8 low_tier tgt_offset: 285696 2020-01-16T00:12:35.862 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.862 INFO:tasks.rados.rados.0.smithi123.stdout:138: finishing set_chunk to oid 39 2020-01-16T00:12:35.862 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 39 v 17 (ObjNum 38 snap 0 seq_num 38) dirty exists 2020-01-16T00:12:35.862 INFO:tasks.rados.rados.0.smithi123.stdout:138: done (15 left) 2020-01-16T00:12:35.863 INFO:tasks.rados.rados.0.smithi123.stdout:158: set_chunk oid 9 offset: 764416 length: 547840 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 764416 2020-01-16T00:12:35.863 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.863 INFO:tasks.rados.rados.0.smithi123.stdout:139: finishing set_chunk to oid 40 2020-01-16T00:12:35.863 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 40 v 18 (ObjNum 39 snap 0 seq_num 39) dirty exists 2020-01-16T00:12:35.863 INFO:tasks.rados.rados.0.smithi123.stdout:139: done (15 left) 2020-01-16T00:12:35.863 INFO:tasks.rados.rados.0.smithi123.stdout:159: set_chunk oid 10 offset: 425984 length: 849408 target oid 10 low_tier tgt_offset: 425984 2020-01-16T00:12:35.863 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.863 INFO:tasks.rados.rados.0.smithi123.stdout:142: finishing set_chunk to oid 43 2020-01-16T00:12:35.864 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 43 v 19 (ObjNum 42 snap 0 seq_num 42) dirty exists 2020-01-16T00:12:35.864 INFO:tasks.rados.rados.0.smithi123.stdout:142: done (15 left) 2020-01-16T00:12:35.864 INFO:tasks.rados.rados.0.smithi123.stdout:160: set_chunk oid 11 offset: 156672 length: 44032 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 156672 2020-01-16T00:12:35.864 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.864 INFO:tasks.rados.rados.0.smithi123.stdout:150: finishing set_chunk to oid 1 2020-01-16T00:12:35.864 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 1 v 24 (ObjNum 0 snap 0 seq_num 0) dirty exists 2020-01-16T00:12:35.864 INFO:tasks.rados.rados.0.smithi123.stdout:153: finishing set_chunk to oid 4 2020-01-16T00:12:35.865 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1110528 length: 221184 chunk_info offset: 1020416 length 515584 2020-01-16T00:12:35.865 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 4 v 13 (ObjNum 3 snap 0 seq_num 3) dirty exists 2020-01-16T00:12:35.865 INFO:tasks.rados.rados.0.smithi123.stdout:150: done (15 left) 2020-01-16T00:12:35.865 INFO:tasks.rados.rados.0.smithi123.stdout:153: done (14 left) 2020-01-16T00:12:35.865 INFO:tasks.rados.rados.0.smithi123.stdout:161: set_chunk oid 12 offset: 672256 length: 927232 target oid 12 low_tier tgt_offset: 672256 2020-01-16T00:12:35.865 INFO:tasks.rados.rados.0.smithi123.stdout:162: set_chunk oid 13 offset: 1541120 length: 378368 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1541120 2020-01-16T00:12:35.865 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.865 INFO:tasks.rados.rados.0.smithi123.stdout:155: finishing set_chunk to oid 6 2020-01-16T00:12:35.866 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 915968 length: 908800 chunk_info offset: 878592 length 645632 2020-01-16T00:12:35.866 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 6 v 10 (ObjNum 5 snap 0 seq_num 5) dirty exists 2020-01-16T00:12:35.866 INFO:tasks.rados.rados.0.smithi123.stdout:147: finishing set_chunk to oid 48 2020-01-16T00:12:35.866 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 48 v 12 (ObjNum 47 snap 0 seq_num 47) dirty exists 2020-01-16T00:12:35.866 INFO:tasks.rados.rados.0.smithi123.stdout:148: finishing set_chunk to oid 49 2020-01-16T00:12:35.866 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 49 v 12 (ObjNum 48 snap 0 seq_num 48) dirty exists 2020-01-16T00:12:35.866 INFO:tasks.rados.rados.0.smithi123.stdout:143: finishing set_chunk to oid 44 2020-01-16T00:12:35.866 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 44 v 20 (ObjNum 43 snap 0 seq_num 43) dirty exists 2020-01-16T00:12:35.867 INFO:tasks.rados.rados.0.smithi123.stdout:146: finishing set_chunk to oid 47 2020-01-16T00:12:35.867 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 47 v 20 (ObjNum 46 snap 0 seq_num 46) dirty exists 2020-01-16T00:12:35.867 INFO:tasks.rados.rados.0.smithi123.stdout:152: finishing set_chunk to oid 3 2020-01-16T00:12:35.867 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 12288 length: 479744 chunk_info offset: 312320 length 603136 2020-01-16T00:12:35.867 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 3 v 7 (ObjNum 2 snap 0 seq_num 2) dirty exists 2020-01-16T00:12:35.867 INFO:tasks.rados.rados.0.smithi123.stdout:154: finishing set_chunk to oid 5 2020-01-16T00:12:35.867 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 5 v 21 (ObjNum 4 snap 0 seq_num 4) dirty exists 2020-01-16T00:12:35.868 INFO:tasks.rados.rados.0.smithi123.stdout:143: done (15 left) 2020-01-16T00:12:35.868 INFO:tasks.rados.rados.0.smithi123.stdout:146: done (14 left) 2020-01-16T00:12:35.868 INFO:tasks.rados.rados.0.smithi123.stdout:147: done (13 left) 2020-01-16T00:12:35.868 INFO:tasks.rados.rados.0.smithi123.stdout:148: done (12 left) 2020-01-16T00:12:35.868 INFO:tasks.rados.rados.0.smithi123.stdout:152: done (11 left) 2020-01-16T00:12:35.868 INFO:tasks.rados.rados.0.smithi123.stdout:154: done (10 left) 2020-01-16T00:12:35.868 INFO:tasks.rados.rados.0.smithi123.stdout:155: done (9 left) 2020-01-16T00:12:35.869 INFO:tasks.rados.rados.0.smithi123.stdout:163: set_chunk oid 14 offset: 170496 length: 1257984 target oid 14 low_tier tgt_offset: 170496 2020-01-16T00:12:35.869 INFO:tasks.rados.rados.0.smithi123.stdout:164: set_chunk oid 15 offset: 1708032 length: 408576 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1708032 2020-01-16T00:12:35.869 INFO:tasks.rados.rados.0.smithi123.stdout:165: set_chunk oid 16 offset: 821248 length: 828928 target oid 16 low_tier tgt_offset: 821248 2020-01-16T00:12:35.869 INFO:tasks.rados.rados.0.smithi123.stdout:149: finishing set_chunk to oid 50 2020-01-16T00:12:35.869 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 50 v 16 (ObjNum 49 snap 0 seq_num 49) dirty exists 2020-01-16T00:12:35.869 INFO:tasks.rados.rados.0.smithi123.stdout:151: finishing set_chunk to oid 2 2020-01-16T00:12:35.869 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 276992 length: 1268736 chunk_info offset: 476160 length 1240064 2020-01-16T00:12:35.870 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 2 v 13 (ObjNum 1 snap 0 seq_num 1) dirty exists 2020-01-16T00:12:35.870 INFO:tasks.rados.rados.0.smithi123.stdout:149: done (11 left) 2020-01-16T00:12:35.870 INFO:tasks.rados.rados.0.smithi123.stdout:151: done (10 left) 2020-01-16T00:12:35.870 INFO:tasks.rados.rados.0.smithi123.stdout:166: set_chunk oid 17 offset: 226816 length: 857600 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 226816 2020-01-16T00:12:35.870 INFO:tasks.rados.rados.0.smithi123.stdout:167: set_chunk oid 18 offset: 997888 length: 982016 target oid 18 low_tier tgt_offset: 997888 2020-01-16T00:12:35.870 INFO:tasks.rados.rados.0.smithi123.stdout:168: set_chunk oid 19 offset: 331264 length: 1575936 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 331264 2020-01-16T00:12:35.870 INFO:tasks.rados.rados.0.smithi123.stdout:169: set_chunk oid 20 offset: 46592 length: 647680 target oid 20 low_tier tgt_offset: 46592 2020-01-16T00:12:35.870 INFO:tasks.rados.rados.0.smithi123.stdout:170: set_chunk oid 21 offset: 346112 length: 838656 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 346112 2020-01-16T00:12:35.871 INFO:tasks.rados.rados.0.smithi123.stdout:171: set_chunk oid 22 offset: 923648 length: 683520 target oid 22 low_tier tgt_offset: 923648 2020-01-16T00:12:35.871 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.872 INFO:tasks.rados.rados.0.smithi123.stdout:159: finishing set_chunk to oid 10 2020-01-16T00:12:35.872 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 425984 length: 849408 chunk_info offset: 453120 length 360448 2020-01-16T00:12:35.873 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 10 v 10 (ObjNum 9 snap 0 seq_num 9) dirty exists 2020-01-16T00:12:35.873 INFO:tasks.rados.rados.0.smithi123.stdout:157: finishing set_chunk to oid 8 2020-01-16T00:12:35.873 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 8 v 10 (ObjNum 7 snap 0 seq_num 7) dirty exists 2020-01-16T00:12:35.873 INFO:tasks.rados.rados.0.smithi123.stdout:156: finishing set_chunk to oid 7 2020-01-16T00:12:35.873 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 55296 length: 1907200 chunk_info offset: 160768 length 1134080 2020-01-16T00:12:35.873 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 7 v 14 (ObjNum 6 snap 0 seq_num 6) dirty exists 2020-01-16T00:12:35.873 INFO:tasks.rados.rados.0.smithi123.stdout:156: done (15 left) 2020-01-16T00:12:35.874 INFO:tasks.rados.rados.0.smithi123.stdout:157: done (14 left) 2020-01-16T00:12:35.874 INFO:tasks.rados.rados.0.smithi123.stdout:159: done (13 left) 2020-01-16T00:12:35.874 INFO:tasks.rados.rados.0.smithi123.stdout:172: set_chunk oid 23 offset: 995840 length: 69632 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 995840 2020-01-16T00:12:35.874 INFO:tasks.rados.rados.0.smithi123.stdout:173: set_chunk oid 24 offset: 738816 length: 1440768 target oid 24 low_tier tgt_offset: 738816 2020-01-16T00:12:35.874 INFO:tasks.rados.rados.0.smithi123.stdout:174: set_chunk oid 25 offset: 773632 length: 1419776 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 773632 2020-01-16T00:12:35.874 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.874 INFO:tasks.rados.rados.0.smithi123.stdout:160: finishing set_chunk to oid 11 2020-01-16T00:12:35.875 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 11 v 14 (ObjNum 10 snap 0 seq_num 10) dirty exists 2020-01-16T00:12:35.875 INFO:tasks.rados.rados.0.smithi123.stdout:160: done (15 left) 2020-01-16T00:12:35.875 INFO:tasks.rados.rados.0.smithi123.stdout:175: set_chunk oid 26 offset: 963072 length: 376320 target oid 26 low_tier tgt_offset: 963072 2020-01-16T00:12:35.875 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.875 INFO:tasks.rados.rados.0.smithi123.stdout:162: finishing set_chunk to oid 13 2020-01-16T00:12:35.875 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 9 (ObjNum 12 snap 0 seq_num 12) dirty exists 2020-01-16T00:12:35.875 INFO:tasks.rados.rados.0.smithi123.stdout:162: done (15 left) 2020-01-16T00:12:35.876 INFO:tasks.rados.rados.0.smithi123.stdout:176: set_chunk oid 27 offset: 6144 length: 717824 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 6144 2020-01-16T00:12:35.876 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.876 INFO:tasks.rados.rados.0.smithi123.stdout:158: finishing set_chunk to oid 9 2020-01-16T00:12:35.876 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 764416 length: 547840 chunk_info offset: 1072128 length 499200 2020-01-16T00:12:35.876 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 9 v 10 (ObjNum 8 snap 0 seq_num 8) dirty exists 2020-01-16T00:12:35.876 INFO:tasks.rados.rados.0.smithi123.stdout:158: done (15 left) 2020-01-16T00:12:35.876 INFO:tasks.rados.rados.0.smithi123.stdout:177: set_chunk oid 28 offset: 1754112 length: 623616 target oid 28 low_tier tgt_offset: 1754112 2020-01-16T00:12:35.876 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.877 INFO:tasks.rados.rados.0.smithi123.stdout:166: finishing set_chunk to oid 17 2020-01-16T00:12:35.877 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 226816 length: 857600 chunk_info offset: 330752 length 378368 2020-01-16T00:12:35.877 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 17 v 14 (ObjNum 16 snap 0 seq_num 16) dirty exists 2020-01-16T00:12:35.877 INFO:tasks.rados.rados.0.smithi123.stdout:166: done (15 left) 2020-01-16T00:12:35.877 INFO:tasks.rados.rados.0.smithi123.stdout:178: set_chunk oid 29 offset: 849920 length: 1152000 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 849920 2020-01-16T00:12:35.877 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.877 INFO:tasks.rados.rados.0.smithi123.stdout:163: finishing set_chunk to oid 14 2020-01-16T00:12:35.878 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 170496 length: 1257984 chunk_info offset: 470528 length 1206272 2020-01-16T00:12:35.878 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 14 v 16 (ObjNum 13 snap 0 seq_num 13) dirty exists 2020-01-16T00:12:35.878 INFO:tasks.rados.rados.0.smithi123.stdout:163: done (15 left) 2020-01-16T00:12:35.878 INFO:tasks.rados.rados.0.smithi123.stdout:179: set_chunk oid 30 offset: 1087488 length: 253952 target oid 30 low_tier tgt_offset: 1087488 2020-01-16T00:12:35.878 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.878 INFO:tasks.rados.rados.0.smithi123.stdout:161: finishing set_chunk to oid 12 2020-01-16T00:12:35.878 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 672256 length: 927232 chunk_info offset: 1033216 length 672768 2020-01-16T00:12:35.879 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 12 v 12 (ObjNum 11 snap 0 seq_num 11) dirty exists 2020-01-16T00:12:35.879 INFO:tasks.rados.rados.0.smithi123.stdout:161: done (15 left) 2020-01-16T00:12:35.879 INFO:tasks.rados.rados.0.smithi123.stdout:180: set_chunk oid 31 offset: 1076224 length: 333824 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1076224 2020-01-16T00:12:35.879 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.879 INFO:tasks.rados.rados.0.smithi123.stdout:171: finishing set_chunk to oid 22 2020-01-16T00:12:35.879 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 22 v 5 (ObjNum 21 snap 0 seq_num 21) dirty exists 2020-01-16T00:12:35.879 INFO:tasks.rados.rados.0.smithi123.stdout:165: finishing set_chunk to oid 16 2020-01-16T00:12:35.879 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 821248 length: 828928 chunk_info offset: 833536 length 364544 2020-01-16T00:12:35.880 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 16 v 17 (ObjNum 15 snap 0 seq_num 15) dirty exists 2020-01-16T00:12:35.880 INFO:tasks.rados.rados.0.smithi123.stdout:169: finishing set_chunk to oid 20 2020-01-16T00:12:35.880 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 46592 length: 647680 chunk_info offset: 59392 length 257024 2020-01-16T00:12:35.880 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 20 v 18 (ObjNum 19 snap 0 seq_num 19) dirty exists 2020-01-16T00:12:35.880 INFO:tasks.rados.rados.0.smithi123.stdout:165: done (15 left) 2020-01-16T00:12:35.880 INFO:tasks.rados.rados.0.smithi123.stdout:169: done (14 left) 2020-01-16T00:12:35.880 INFO:tasks.rados.rados.0.smithi123.stdout:171: done (13 left) 2020-01-16T00:12:35.881 INFO:tasks.rados.rados.0.smithi123.stdout:181: set_chunk oid 32 offset: 114176 length: 1733632 target oid 32 low_tier tgt_offset: 114176 2020-01-16T00:12:35.881 INFO:tasks.rados.rados.0.smithi123.stdout:182: set_chunk oid 33 offset: 209408 length: 650240 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 209408 2020-01-16T00:12:35.881 INFO:tasks.rados.rados.0.smithi123.stdout:183: set_chunk oid 34 offset: 925696 length: 971776 target oid 34 low_tier tgt_offset: 925696 2020-01-16T00:12:35.881 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.881 INFO:tasks.rados.rados.0.smithi123.stdout:164: finishing set_chunk to oid 15 2020-01-16T00:12:35.881 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 15 v 17 (ObjNum 14 snap 0 seq_num 14) dirty exists 2020-01-16T00:12:35.881 INFO:tasks.rados.rados.0.smithi123.stdout:164: done (15 left) 2020-01-16T00:12:35.882 INFO:tasks.rados.rados.0.smithi123.stdout:184: set_chunk oid 35 offset: 926720 length: 423424 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 926720 2020-01-16T00:12:35.882 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.882 INFO:tasks.rados.rados.0.smithi123.stdout:173: finishing set_chunk to oid 24 2020-01-16T00:12:35.882 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 738816 length: 1440768 chunk_info offset: 30720 length 1585664 2020-01-16T00:12:35.882 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 24 v 10 (ObjNum 23 snap 0 seq_num 23) dirty exists 2020-01-16T00:12:35.882 INFO:tasks.rados.rados.0.smithi123.stdout:173: done (15 left) 2020-01-16T00:12:35.882 INFO:tasks.rados.rados.0.smithi123.stdout:185: set_chunk oid 36 offset: 857600 length: 917504 target oid 36 low_tier tgt_offset: 857600 2020-01-16T00:12:35.882 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.883 INFO:tasks.rados.rados.0.smithi123.stdout:178: finishing set_chunk to oid 29 2020-01-16T00:12:35.883 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 849920 length: 1152000 chunk_info offset: 1186304 length 657408 2020-01-16T00:12:35.883 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 29 v 11 (ObjNum 28 snap 0 seq_num 28) dirty exists 2020-01-16T00:12:35.883 INFO:tasks.rados.rados.0.smithi123.stdout:178: done (15 left) 2020-01-16T00:12:35.883 INFO:tasks.rados.rados.0.smithi123.stdout:186: set_chunk oid 37 offset: 1125376 length: 1407488 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1125376 2020-01-16T00:12:35.883 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.883 INFO:tasks.rados.rados.0.smithi123.stdout:179: finishing set_chunk to oid 30 2020-01-16T00:12:35.883 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 30 v 15 (ObjNum 29 snap 0 seq_num 29) dirty exists 2020-01-16T00:12:35.884 INFO:tasks.rados.rados.0.smithi123.stdout:179: done (15 left) 2020-01-16T00:12:35.884 INFO:tasks.rados.rados.0.smithi123.stdout:187: set_chunk oid 38 offset: 76800 length: 1861632 target oid 38 low_tier tgt_offset: 76800 2020-01-16T00:12:35.884 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.884 INFO:tasks.rados.rados.0.smithi123.stdout:174: finishing set_chunk to oid 25 2020-01-16T00:12:35.884 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 25 v 24 (ObjNum 24 snap 0 seq_num 24) dirty exists 2020-01-16T00:12:35.884 INFO:tasks.rados.rados.0.smithi123.stdout:174: done (15 left) 2020-01-16T00:12:35.884 INFO:tasks.rados.rados.0.smithi123.stdout:188: set_chunk oid 39 offset: 1570304 length: 12288 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1570304 2020-01-16T00:12:35.884 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.885 INFO:tasks.rados.rados.0.smithi123.stdout:172: finishing set_chunk to oid 23 2020-01-16T00:12:35.885 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 995840 length: 69632 chunk_info offset: 808960 length 1406976 2020-01-16T00:12:35.885 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 23 v 15 (ObjNum 22 snap 0 seq_num 22) dirty exists 2020-01-16T00:12:35.885 INFO:tasks.rados.rados.0.smithi123.stdout:172: done (15 left) 2020-01-16T00:12:35.885 INFO:tasks.rados.rados.0.smithi123.stdout:189: set_chunk oid 40 offset: 790528 length: 688128 target oid 40 low_tier tgt_offset: 790528 2020-01-16T00:12:35.885 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.885 INFO:tasks.rados.rados.0.smithi123.stdout:167: finishing set_chunk to oid 18 2020-01-16T00:12:35.886 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 997888 length: 982016 chunk_info offset: 953856 length 471040 2020-01-16T00:12:35.886 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 18 v 19 (ObjNum 17 snap 0 seq_num 17) dirty exists 2020-01-16T00:12:35.886 INFO:tasks.rados.rados.0.smithi123.stdout:167: done (15 left) 2020-01-16T00:12:35.886 INFO:tasks.rados.rados.0.smithi123.stdout:190: set_chunk oid 41 offset: 625664 length: 927744 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 625664 2020-01-16T00:12:35.886 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.886 INFO:tasks.rados.rados.0.smithi123.stdout:175: finishing set_chunk to oid 26 2020-01-16T00:12:35.886 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 26 v 26 (ObjNum 25 snap 0 seq_num 25) dirty exists 2020-01-16T00:12:35.886 INFO:tasks.rados.rados.0.smithi123.stdout:175: done (15 left) 2020-01-16T00:12:35.887 INFO:tasks.rados.rados.0.smithi123.stdout:191: set_chunk oid 42 offset: 0 length: 2468352 target oid 42 low_tier tgt_offset: 0 2020-01-16T00:12:35.887 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.887 INFO:tasks.rados.rados.0.smithi123.stdout:170: finishing set_chunk to oid 21 2020-01-16T00:12:35.887 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 21 v 9 (ObjNum 20 snap 0 seq_num 20) dirty exists 2020-01-16T00:12:35.887 INFO:tasks.rados.rados.0.smithi123.stdout:177: finishing set_chunk to oid 28 2020-01-16T00:12:35.887 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1754112 length: 623616 chunk_info offset: 1498624 length 783872 2020-01-16T00:12:35.887 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 28 v 14 (ObjNum 27 snap 0 seq_num 27) dirty exists 2020-01-16T00:12:35.887 INFO:tasks.rados.rados.0.smithi123.stdout:176: finishing set_chunk to oid 27 2020-01-16T00:12:35.888 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 6144 length: 717824 chunk_info offset: 113152 length 1699840 2020-01-16T00:12:35.888 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 27 v 11 (ObjNum 26 snap 0 seq_num 26) dirty exists 2020-01-16T00:12:35.888 INFO:tasks.rados.rados.0.smithi123.stdout:168: finishing set_chunk to oid 19 2020-01-16T00:12:35.888 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 331264 length: 1575936 chunk_info offset: 592384 length 1625600 2020-01-16T00:12:35.888 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 19 v 15 (ObjNum 18 snap 0 seq_num 18) dirty exists 2020-01-16T00:12:35.888 INFO:tasks.rados.rados.0.smithi123.stdout:168: done (15 left) 2020-01-16T00:12:35.888 INFO:tasks.rados.rados.0.smithi123.stdout:170: done (14 left) 2020-01-16T00:12:35.888 INFO:tasks.rados.rados.0.smithi123.stdout:176: done (13 left) 2020-01-16T00:12:35.889 INFO:tasks.rados.rados.0.smithi123.stdout:177: done (12 left) 2020-01-16T00:12:35.889 INFO:tasks.rados.rados.0.smithi123.stdout:192: set_chunk oid 43 offset: 1384448 length: 188416 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1384448 2020-01-16T00:12:35.889 INFO:tasks.rados.rados.0.smithi123.stdout:193: set_chunk oid 44 offset: 1635840 length: 423424 target oid 44 low_tier tgt_offset: 1635840 2020-01-16T00:12:35.889 INFO:tasks.rados.rados.0.smithi123.stdout:194: set_chunk oid 45 offset: 263680 length: 837632 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 263680 2020-01-16T00:12:35.889 INFO:tasks.rados.rados.0.smithi123.stdout:195: set_chunk oid 46 offset: 911872 length: 1691136 target oid 46 low_tier tgt_offset: 911872 2020-01-16T00:12:35.889 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.889 INFO:tasks.rados.rados.0.smithi123.stdout:180: finishing set_chunk to oid 31 2020-01-16T00:12:35.890 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1076224 length: 333824 chunk_info offset: 263680 length 1668096 2020-01-16T00:12:35.890 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 31 v 15 (ObjNum 30 snap 0 seq_num 30) dirty exists 2020-01-16T00:12:35.890 INFO:tasks.rados.rados.0.smithi123.stdout:180: done (15 left) 2020-01-16T00:12:35.890 INFO:tasks.rados.rados.0.smithi123.stdout:196: set_chunk oid 47 offset: 1584128 length: 846848 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1584128 2020-01-16T00:12:35.890 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.890 INFO:tasks.rados.rados.0.smithi123.stdout:182: finishing set_chunk to oid 33 2020-01-16T00:12:35.890 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 209408 length: 650240 chunk_info offset: 658944 length 1355776 2020-01-16T00:12:35.890 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 33 v 21 (ObjNum 32 snap 0 seq_num 32) dirty exists 2020-01-16T00:12:35.891 INFO:tasks.rados.rados.0.smithi123.stdout:184: finishing set_chunk to oid 35 2020-01-16T00:12:35.891 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 35 v 20 (ObjNum 34 snap 0 seq_num 34) dirty exists 2020-01-16T00:12:35.891 INFO:tasks.rados.rados.0.smithi123.stdout:182: done (15 left) 2020-01-16T00:12:35.891 INFO:tasks.rados.rados.0.smithi123.stdout:184: done (14 left) 2020-01-16T00:12:35.891 INFO:tasks.rados.rados.0.smithi123.stdout:197: set_chunk oid 48 offset: 698368 length: 81920 target oid 48 low_tier tgt_offset: 698368 2020-01-16T00:12:35.891 INFO:tasks.rados.rados.0.smithi123.stdout:198: set_chunk oid 49 offset: 1352704 length: 493568 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1352704 2020-01-16T00:12:35.891 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.892 INFO:tasks.rados.rados.0.smithi123.stdout:186: finishing set_chunk to oid 37 2020-01-16T00:12:35.892 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1125376 length: 1407488 chunk_info offset: 574464 length 1014784 2020-01-16T00:12:35.892 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 37 v 22 (ObjNum 36 snap 0 seq_num 36) dirty exists 2020-01-16T00:12:35.892 INFO:tasks.rados.rados.0.smithi123.stdout:186: done (15 left) 2020-01-16T00:12:35.892 INFO:tasks.rados.rados.0.smithi123.stdout:199: set_chunk oid 50 offset: 1664512 length: 275456 target oid 50 low_tier tgt_offset: 1664512 2020-01-16T00:12:35.892 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.893 INFO:tasks.rados.rados.0.smithi123.stdout:181: finishing set_chunk to oid 32 2020-01-16T00:12:35.893 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 114176 length: 1733632 chunk_info offset: 605696 length 877568 2020-01-16T00:12:35.893 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 32 v 12 (ObjNum 31 snap 0 seq_num 31) dirty exists 2020-01-16T00:12:35.893 INFO:tasks.rados.rados.0.smithi123.stdout:181: done (15 left) 2020-01-16T00:12:35.893 INFO:tasks.rados.rados.0.smithi123.stdout:200: set_chunk oid 1 offset: 955392 length: 627200 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 955392 2020-01-16T00:12:35.893 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.894 INFO:tasks.rados.rados.0.smithi123.stdout:185: finishing set_chunk to oid 36 2020-01-16T00:12:35.894 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 857600 length: 917504 chunk_info offset: 643584 length 327680 2020-01-16T00:12:35.894 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 36 v 8 (ObjNum 35 snap 0 seq_num 35) dirty exists 2020-01-16T00:12:35.894 INFO:tasks.rados.rados.0.smithi123.stdout:195: finishing set_chunk to oid 46 2020-01-16T00:12:35.894 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 911872 length: 1691136 chunk_info offset: 56832 length 1408000 2020-01-16T00:12:35.894 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 46 v 23 (ObjNum 45 snap 0 seq_num 45) dirty exists 2020-01-16T00:12:35.894 INFO:tasks.rados.rados.0.smithi123.stdout:185: done (15 left) 2020-01-16T00:12:35.895 INFO:tasks.rados.rados.0.smithi123.stdout:195: done (14 left) 2020-01-16T00:12:35.895 INFO:tasks.rados.rados.0.smithi123.stdout:201: set_chunk oid 2 offset: 887808 length: 89088 target oid 2 low_tier tgt_offset: 887808 2020-01-16T00:12:35.895 INFO:tasks.rados.rados.0.smithi123.stdout:202: set_chunk oid 3 offset: 77824 length: 863232 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 77824 2020-01-16T00:12:35.895 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.895 INFO:tasks.rados.rados.0.smithi123.stdout:190: finishing set_chunk to oid 41 2020-01-16T00:12:35.895 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 625664 length: 927744 chunk_info offset: 925184 length 293376 2020-01-16T00:12:35.895 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 41 v 16 (ObjNum 40 snap 0 seq_num 40) dirty exists 2020-01-16T00:12:35.896 INFO:tasks.rados.rados.0.smithi123.stdout:190: done (15 left) 2020-01-16T00:12:35.896 INFO:tasks.rados.rados.0.smithi123.stdout:203: set_chunk oid 4 offset: 11264 length: 1718784 target oid 4 low_tier tgt_offset: 11264 2020-01-16T00:12:35.896 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.896 INFO:tasks.rados.rados.0.smithi123.stdout:183: finishing set_chunk to oid 34 2020-01-16T00:12:35.896 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 925696 length: 971776 chunk_info offset: 901632 length 1154560 2020-01-16T00:12:35.896 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 34 v 8 (ObjNum 33 snap 0 seq_num 33) dirty exists 2020-01-16T00:12:35.896 INFO:tasks.rados.rados.0.smithi123.stdout:183: done (15 left) 2020-01-16T00:12:35.897 INFO:tasks.rados.rados.0.smithi123.stdout:204: set_chunk oid 5 offset: 391168 length: 483328 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 391168 2020-01-16T00:12:35.897 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.897 INFO:tasks.rados.rados.0.smithi123.stdout:187: finishing set_chunk to oid 38 2020-01-16T00:12:35.897 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 76800 length: 1861632 chunk_info offset: 1915392 length 653312 2020-01-16T00:12:35.897 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 38 v 10 (ObjNum 37 snap 0 seq_num 37) dirty exists 2020-01-16T00:12:35.897 INFO:tasks.rados.rados.0.smithi123.stdout:187: done (15 left) 2020-01-16T00:12:35.897 INFO:tasks.rados.rados.0.smithi123.stdout:205: set_chunk oid 6 offset: 710656 length: 584704 target oid 6 low_tier tgt_offset: 710656 2020-01-16T00:12:35.898 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.898 INFO:tasks.rados.rados.0.smithi123.stdout:203: finishing set_chunk to oid 4 2020-01-16T00:12:35.898 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 11264 length: 1718784 chunk_info offset: 1020416 length 515584 2020-01-16T00:12:35.898 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 4 v 13 (ObjNum 3 snap 0 seq_num 3) dirty exists 2020-01-16T00:12:35.898 INFO:tasks.rados.rados.0.smithi123.stdout:203: done (15 left) 2020-01-16T00:12:35.898 INFO:tasks.rados.rados.0.smithi123.stdout:206: set_chunk oid 7 offset: 634368 length: 1176064 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 634368 2020-01-16T00:12:35.898 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.899 INFO:tasks.rados.rados.0.smithi123.stdout:202: finishing set_chunk to oid 3 2020-01-16T00:12:35.899 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 77824 length: 863232 chunk_info offset: 312320 length 603136 2020-01-16T00:12:35.899 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 3 v 7 (ObjNum 2 snap 0 seq_num 2) dirty exists 2020-01-16T00:12:35.899 INFO:tasks.rados.rados.0.smithi123.stdout:202: done (15 left) 2020-01-16T00:12:35.899 INFO:tasks.rados.rados.0.smithi123.stdout:207: set_chunk oid 8 offset: 1940480 length: 73728 target oid 8 low_tier tgt_offset: 1940480 2020-01-16T00:12:35.899 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.900 INFO:tasks.rados.rados.0.smithi123.stdout:188: finishing set_chunk to oid 39 2020-01-16T00:12:35.900 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 39 v 22 (ObjNum 38 snap 0 seq_num 38) dirty exists 2020-01-16T00:12:35.900 INFO:tasks.rados.rados.0.smithi123.stdout:189: finishing set_chunk to oid 40 2020-01-16T00:12:35.900 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 40 v 23 (ObjNum 39 snap 0 seq_num 39) dirty exists 2020-01-16T00:12:35.900 INFO:tasks.rados.rados.0.smithi123.stdout:194: finishing set_chunk to oid 45 2020-01-16T00:12:35.900 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 263680 length: 837632 chunk_info offset: 640512 length 403968 2020-01-16T00:12:35.900 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 45 v 11 (ObjNum 44 snap 0 seq_num 44) dirty exists 2020-01-16T00:12:35.901 INFO:tasks.rados.rados.0.smithi123.stdout:188: done (15 left) 2020-01-16T00:12:35.901 INFO:tasks.rados.rados.0.smithi123.stdout:189: done (14 left) 2020-01-16T00:12:35.901 INFO:tasks.rados.rados.0.smithi123.stdout:194: done (13 left) 2020-01-16T00:12:35.901 INFO:tasks.rados.rados.0.smithi123.stdout:208: set_chunk oid 9 offset: 514560 length: 177152 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 514560 2020-01-16T00:12:35.901 INFO:tasks.rados.rados.0.smithi123.stdout:209: set_chunk oid 10 offset: 1757696 length: 131072 target oid 10 low_tier tgt_offset: 1757696 2020-01-16T00:12:35.901 INFO:tasks.rados.rados.0.smithi123.stdout:210: set_chunk oid 11 offset: 937984 length: 1012736 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 937984 2020-01-16T00:12:35.901 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.901 INFO:tasks.rados.rados.0.smithi123.stdout:192: finishing set_chunk to oid 43 2020-01-16T00:12:35.902 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 43 v 24 (ObjNum 42 snap 0 seq_num 42) dirty exists 2020-01-16T00:12:35.902 INFO:tasks.rados.rados.0.smithi123.stdout:192: done (15 left) 2020-01-16T00:12:35.902 INFO:tasks.rados.rados.0.smithi123.stdout:211: set_chunk oid 12 offset: 820224 length: 645632 target oid 12 low_tier tgt_offset: 820224 2020-01-16T00:12:35.902 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.902 INFO:tasks.rados.rados.0.smithi123.stdout:198: finishing set_chunk to oid 49 2020-01-16T00:12:35.902 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1352704 length: 493568 chunk_info offset: 453632 length 1075200 2020-01-16T00:12:35.902 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 49 v 12 (ObjNum 48 snap 0 seq_num 48) dirty exists 2020-01-16T00:12:35.903 INFO:tasks.rados.rados.0.smithi123.stdout:193: finishing set_chunk to oid 44 2020-01-16T00:12:35.903 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 44 v 25 (ObjNum 43 snap 0 seq_num 43) dirty exists 2020-01-16T00:12:35.903 INFO:tasks.rados.rados.0.smithi123.stdout:193: done (15 left) 2020-01-16T00:12:35.903 INFO:tasks.rados.rados.0.smithi123.stdout:198: done (14 left) 2020-01-16T00:12:35.903 INFO:tasks.rados.rados.0.smithi123.stdout:212: set_chunk oid 13 offset: 184832 length: 903168 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 184832 2020-01-16T00:12:35.903 INFO:tasks.rados.rados.0.smithi123.stdout:213: set_chunk oid 14 offset: 55296 length: 299008 target oid 14 low_tier tgt_offset: 55296 2020-01-16T00:12:35.903 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.903 INFO:tasks.rados.rados.0.smithi123.stdout:191: finishing set_chunk to oid 42 2020-01-16T00:12:35.904 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 0 length: 2468352 chunk_info offset: 1364992 length 1145344 2020-01-16T00:12:35.904 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 42 v 11 (ObjNum 41 snap 0 seq_num 41) dirty exists 2020-01-16T00:12:35.904 INFO:tasks.rados.rados.0.smithi123.stdout:201: finishing set_chunk to oid 2 2020-01-16T00:12:35.904 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 887808 length: 89088 chunk_info offset: 476160 length 1240064 2020-01-16T00:12:35.904 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 2 v 13 (ObjNum 1 snap 0 seq_num 1) dirty exists 2020-01-16T00:12:35.904 INFO:tasks.rados.rados.0.smithi123.stdout:191: done (15 left) 2020-01-16T00:12:35.904 INFO:tasks.rados.rados.0.smithi123.stdout:201: done (14 left) 2020-01-16T00:12:35.905 INFO:tasks.rados.rados.0.smithi123.stdout:214: set_chunk oid 15 offset: 6656 length: 939008 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 6656 2020-01-16T00:12:35.905 INFO:tasks.rados.rados.0.smithi123.stdout:215: set_chunk oid 16 offset: 384000 length: 1445888 target oid 16 low_tier tgt_offset: 384000 2020-01-16T00:12:35.905 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.905 INFO:tasks.rados.rados.0.smithi123.stdout:197: finishing set_chunk to oid 48 2020-01-16T00:12:35.905 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 698368 length: 81920 chunk_info offset: 264704 length 1399296 2020-01-16T00:12:35.905 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 48 v 12 (ObjNum 47 snap 0 seq_num 47) dirty exists 2020-01-16T00:12:35.906 INFO:tasks.rados.rados.0.smithi123.stdout:197: done (15 left) 2020-01-16T00:12:35.906 INFO:tasks.rados.rados.0.smithi123.stdout:216: set_chunk oid 17 offset: 476160 length: 51200 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 476160 2020-01-16T00:12:35.906 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.906 INFO:tasks.rados.rados.0.smithi123.stdout:196: finishing set_chunk to oid 47 2020-01-16T00:12:35.906 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1584128 length: 846848 chunk_info offset: 2092032 length 38400 2020-01-16T00:12:35.906 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 47 v 20 (ObjNum 46 snap 0 seq_num 46) dirty exists 2020-01-16T00:12:35.906 INFO:tasks.rados.rados.0.smithi123.stdout:204: finishing set_chunk to oid 5 2020-01-16T00:12:35.907 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 391168 length: 483328 chunk_info offset: 102400 length 870912 2020-01-16T00:12:35.907 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 5 v 21 (ObjNum 4 snap 0 seq_num 4) dirty exists 2020-01-16T00:12:35.907 INFO:tasks.rados.rados.0.smithi123.stdout:196: done (15 left) 2020-01-16T00:12:35.907 INFO:tasks.rados.rados.0.smithi123.stdout:204: done (14 left) 2020-01-16T00:12:35.907 INFO:tasks.rados.rados.0.smithi123.stdout:217: set_chunk oid 18 offset: 1978368 length: 184320 target oid 18 low_tier tgt_offset: 1978368 2020-01-16T00:12:35.907 INFO:tasks.rados.rados.0.smithi123.stdout:218: set_chunk oid 19 offset: 669184 length: 1231360 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 669184 2020-01-16T00:12:35.907 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.908 INFO:tasks.rados.rados.0.smithi123.stdout:200: finishing set_chunk to oid 1 2020-01-16T00:12:35.908 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 955392 length: 627200 chunk_info offset: 1046016 length 729088 2020-01-16T00:12:35.908 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 1 v 24 (ObjNum 0 snap 0 seq_num 0) dirty exists 2020-01-16T00:12:35.908 INFO:tasks.rados.rados.0.smithi123.stdout:200: done (15 left) 2020-01-16T00:12:35.908 INFO:tasks.rados.rados.0.smithi123.stdout:219: set_chunk oid 20 offset: 1278464 length: 442368 target oid 20 low_tier tgt_offset: 1278464 2020-01-16T00:12:35.908 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.908 INFO:tasks.rados.rados.0.smithi123.stdout:206: finishing set_chunk to oid 7 2020-01-16T00:12:35.909 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 634368 length: 1176064 chunk_info offset: 160768 length 1134080 2020-01-16T00:12:35.909 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 7 v 14 (ObjNum 6 snap 0 seq_num 6) dirty exists 2020-01-16T00:12:35.909 INFO:tasks.rados.rados.0.smithi123.stdout:206: done (15 left) 2020-01-16T00:12:35.909 INFO:tasks.rados.rados.0.smithi123.stdout:220: set_chunk oid 21 offset: 1149952 length: 727552 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1149952 2020-01-16T00:12:35.909 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.909 INFO:tasks.rados.rados.0.smithi123.stdout:207: finishing set_chunk to oid 8 2020-01-16T00:12:35.909 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1940480 length: 73728 chunk_info offset: 1589248 length 433664 2020-01-16T00:12:35.910 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 8 v 10 (ObjNum 7 snap 0 seq_num 7) dirty exists 2020-01-16T00:12:35.910 INFO:tasks.rados.rados.0.smithi123.stdout:207: done (15 left) 2020-01-16T00:12:35.910 INFO:tasks.rados.rados.0.smithi123.stdout:221: set_chunk oid 22 offset: 1002496 length: 712704 target oid 22 low_tier tgt_offset: 1002496 2020-01-16T00:12:35.910 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.910 INFO:tasks.rados.rados.0.smithi123.stdout:205: finishing set_chunk to oid 6 2020-01-16T00:12:35.910 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 710656 length: 584704 chunk_info offset: 878592 length 645632 2020-01-16T00:12:35.911 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 6 v 10 (ObjNum 5 snap 0 seq_num 5) dirty exists 2020-01-16T00:12:35.911 INFO:tasks.rados.rados.0.smithi123.stdout:205: done (15 left) 2020-01-16T00:12:35.911 INFO:tasks.rados.rados.0.smithi123.stdout:222: set_chunk oid 23 offset: 279040 length: 1426432 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 279040 2020-01-16T00:12:35.911 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.911 INFO:tasks.rados.rados.0.smithi123.stdout:199: finishing set_chunk to oid 50 2020-01-16T00:12:35.911 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1664512 length: 275456 chunk_info offset: 1613312 length 498688 2020-01-16T00:12:35.911 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 50 v 16 (ObjNum 49 snap 0 seq_num 49) dirty exists 2020-01-16T00:12:35.911 INFO:tasks.rados.rados.0.smithi123.stdout:199: done (15 left) 2020-01-16T00:12:35.912 INFO:tasks.rados.rados.0.smithi123.stdout:223: set_chunk oid 24 offset: 224768 length: 1041408 target oid 24 low_tier tgt_offset: 224768 2020-01-16T00:12:35.912 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.912 INFO:tasks.rados.rados.0.smithi123.stdout:208: finishing set_chunk to oid 9 2020-01-16T00:12:35.912 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 9 v 16 (ObjNum 8 snap 0 seq_num 8) dirty exists 2020-01-16T00:12:35.912 INFO:tasks.rados.rados.0.smithi123.stdout:209: finishing set_chunk to oid 10 2020-01-16T00:12:35.912 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 10 v 16 (ObjNum 9 snap 0 seq_num 9) dirty exists 2020-01-16T00:12:35.912 INFO:tasks.rados.rados.0.smithi123.stdout:208: done (15 left) 2020-01-16T00:12:35.912 INFO:tasks.rados.rados.0.smithi123.stdout:209: done (14 left) 2020-01-16T00:12:35.913 INFO:tasks.rados.rados.0.smithi123.stdout:224: set_chunk oid 25 offset: 722944 length: 987648 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 722944 2020-01-16T00:12:35.913 INFO:tasks.rados.rados.0.smithi123.stdout:225: set_chunk oid 26 offset: 1337344 length: 737792 target oid 26 low_tier tgt_offset: 1337344 2020-01-16T00:12:35.913 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.913 INFO:tasks.rados.rados.0.smithi123.stdout:212: finishing set_chunk to oid 13 2020-01-16T00:12:35.913 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 184832 length: 903168 chunk_info offset: 916992 length 596480 2020-01-16T00:12:35.913 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 9 (ObjNum 12 snap 0 seq_num 12) dirty exists 2020-01-16T00:12:35.914 INFO:tasks.rados.rados.0.smithi123.stdout:212: done (15 left) 2020-01-16T00:12:35.914 INFO:tasks.rados.rados.0.smithi123.stdout:226: set_chunk oid 27 offset: 192512 length: 77312 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 192512 2020-01-16T00:12:35.914 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.914 INFO:tasks.rados.rados.0.smithi123.stdout:210: finishing set_chunk to oid 11 2020-01-16T00:12:35.914 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 937984 length: 1012736 chunk_info offset: 279552 length 1427968 2020-01-16T00:12:35.914 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 11 v 14 (ObjNum 10 snap 0 seq_num 10) dirty exists 2020-01-16T00:12:35.914 INFO:tasks.rados.rados.0.smithi123.stdout:210: done (15 left) 2020-01-16T00:12:35.915 INFO:tasks.rados.rados.0.smithi123.stdout:227: set_chunk oid 28 offset: 1010176 length: 1184256 target oid 28 low_tier tgt_offset: 1010176 2020-01-16T00:12:35.915 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.915 INFO:tasks.rados.rados.0.smithi123.stdout:218: finishing set_chunk to oid 19 2020-01-16T00:12:35.915 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 669184 length: 1231360 chunk_info offset: 592384 length 1625600 2020-01-16T00:12:35.915 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 19 v 15 (ObjNum 18 snap 0 seq_num 18) dirty exists 2020-01-16T00:12:35.915 INFO:tasks.rados.rados.0.smithi123.stdout:218: done (15 left) 2020-01-16T00:12:35.915 INFO:tasks.rados.rados.0.smithi123.stdout:228: set_chunk oid 29 offset: 609280 length: 1707520 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 609280 2020-01-16T00:12:35.915 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.916 INFO:tasks.rados.rados.0.smithi123.stdout:216: finishing set_chunk to oid 17 2020-01-16T00:12:35.916 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 476160 length: 51200 chunk_info offset: 330752 length 378368 2020-01-16T00:12:35.916 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 17 v 14 (ObjNum 16 snap 0 seq_num 16) dirty exists 2020-01-16T00:12:35.916 INFO:tasks.rados.rados.0.smithi123.stdout:213: finishing set_chunk to oid 14 2020-01-16T00:12:35.916 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 14 v 26 (ObjNum 13 snap 0 seq_num 13) dirty exists 2020-01-16T00:12:35.916 INFO:tasks.rados.rados.0.smithi123.stdout:213: done (15 left) 2020-01-16T00:12:35.916 INFO:tasks.rados.rados.0.smithi123.stdout:216: done (14 left) 2020-01-16T00:12:35.916 INFO:tasks.rados.rados.0.smithi123.stdout:229: set_chunk oid 30 offset: 59904 length: 604672 target oid 30 low_tier tgt_offset: 59904 2020-01-16T00:12:35.917 INFO:tasks.rados.rados.0.smithi123.stdout:230: set_chunk oid 31 offset: 707584 length: 1102336 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 707584 2020-01-16T00:12:35.917 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.917 INFO:tasks.rados.rados.0.smithi123.stdout:214: finishing set_chunk to oid 15 2020-01-16T00:12:35.917 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 6656 length: 939008 chunk_info offset: 331264 length 198656 2020-01-16T00:12:35.917 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 15 v 17 (ObjNum 14 snap 0 seq_num 14) dirty exists 2020-01-16T00:12:35.917 INFO:tasks.rados.rados.0.smithi123.stdout:214: done (15 left) 2020-01-16T00:12:35.917 INFO:tasks.rados.rados.0.smithi123.stdout:231: set_chunk oid 32 offset: 2444288 length: 71168 target oid 32 low_tier tgt_offset: 2444288 2020-01-16T00:12:35.918 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.918 INFO:tasks.rados.rados.0.smithi123.stdout:217: finishing set_chunk to oid 18 2020-01-16T00:12:35.918 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 18 v 31 (ObjNum 17 snap 0 seq_num 17) dirty exists 2020-01-16T00:12:35.918 INFO:tasks.rados.rados.0.smithi123.stdout:222: finishing set_chunk to oid 23 2020-01-16T00:12:35.918 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 279040 length: 1426432 chunk_info offset: 808960 length 1406976 2020-01-16T00:12:35.918 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 23 v 15 (ObjNum 22 snap 0 seq_num 22) dirty exists 2020-01-16T00:12:35.919 INFO:tasks.rados.rados.0.smithi123.stdout:217: done (15 left) 2020-01-16T00:12:35.919 INFO:tasks.rados.rados.0.smithi123.stdout:222: done (14 left) 2020-01-16T00:12:35.919 INFO:tasks.rados.rados.0.smithi123.stdout:232: set_chunk oid 33 offset: 32768 length: 2050560 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 32768 2020-01-16T00:12:35.919 INFO:tasks.rados.rados.0.smithi123.stdout:233: set_chunk oid 34 offset: 615936 length: 180224 target oid 34 low_tier tgt_offset: 615936 2020-01-16T00:12:35.919 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.919 INFO:tasks.rados.rados.0.smithi123.stdout:211: finishing set_chunk to oid 12 2020-01-16T00:12:35.919 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 820224 length: 645632 chunk_info offset: 1033216 length 672768 2020-01-16T00:12:35.920 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 12 v 12 (ObjNum 11 snap 0 seq_num 11) dirty exists 2020-01-16T00:12:35.920 INFO:tasks.rados.rados.0.smithi123.stdout:211: done (15 left) 2020-01-16T00:12:35.920 INFO:tasks.rados.rados.0.smithi123.stdout:234: set_chunk oid 35 offset: 1769472 length: 453120 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1769472 2020-01-16T00:12:35.920 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.920 INFO:tasks.rados.rados.0.smithi123.stdout:225: finishing set_chunk to oid 26 2020-01-16T00:12:35.920 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1337344 length: 737792 chunk_info offset: 963072 length 376320 2020-01-16T00:12:35.920 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 26 v 26 (ObjNum 25 snap 0 seq_num 25) dirty exists 2020-01-16T00:12:35.921 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1337344 length: 737792 chunk_info offset: 1494016 length 315392 2020-01-16T00:12:35.921 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 26 v 26 (ObjNum 25 snap 0 seq_num 25) dirty exists 2020-01-16T00:12:35.921 INFO:tasks.rados.rados.0.smithi123.stdout:220: finishing set_chunk to oid 21 2020-01-16T00:12:35.921 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1149952 length: 727552 chunk_info offset: 346112 length 838656 2020-01-16T00:12:35.921 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 21 v 9 (ObjNum 20 snap 0 seq_num 20) dirty exists 2020-01-16T00:12:35.921 INFO:tasks.rados.rados.0.smithi123.stdout:221: finishing set_chunk to oid 22 2020-01-16T00:12:35.921 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1002496 length: 712704 chunk_info offset: 923648 length 683520 2020-01-16T00:12:35.922 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 22 v 5 (ObjNum 21 snap 0 seq_num 21) dirty exists 2020-01-16T00:12:35.922 INFO:tasks.rados.rados.0.smithi123.stdout:220: done (15 left) 2020-01-16T00:12:35.922 INFO:tasks.rados.rados.0.smithi123.stdout:221: done (14 left) 2020-01-16T00:12:35.922 INFO:tasks.rados.rados.0.smithi123.stdout:225: done (13 left) 2020-01-16T00:12:35.922 INFO:tasks.rados.rados.0.smithi123.stdout:235: set_chunk oid 36 offset: 743936 length: 1061888 target oid 36 low_tier tgt_offset: 743936 2020-01-16T00:12:35.923 INFO:tasks.rados.rados.0.smithi123.stdout:236: set_chunk oid 37 offset: 10240 length: 1890816 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 10240 2020-01-16T00:12:35.923 INFO:tasks.rados.rados.0.smithi123.stdout:237: set_chunk oid 38 offset: 586752 length: 1668096 target oid 38 low_tier tgt_offset: 586752 2020-01-16T00:12:35.923 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.923 INFO:tasks.rados.rados.0.smithi123.stdout:227: finishing set_chunk to oid 28 2020-01-16T00:12:35.923 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1010176 length: 1184256 chunk_info offset: 1498624 length 783872 2020-01-16T00:12:35.923 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 28 v 14 (ObjNum 27 snap 0 seq_num 27) dirty exists 2020-01-16T00:12:35.923 INFO:tasks.rados.rados.0.smithi123.stdout:226: finishing set_chunk to oid 27 2020-01-16T00:12:35.924 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 192512 length: 77312 chunk_info offset: 113152 length 1699840 2020-01-16T00:12:35.924 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 27 v 11 (ObjNum 26 snap 0 seq_num 26) dirty exists 2020-01-16T00:12:35.924 INFO:tasks.rados.rados.0.smithi123.stdout:226: done (15 left) 2020-01-16T00:12:35.924 INFO:tasks.rados.rados.0.smithi123.stdout:227: done (14 left) 2020-01-16T00:12:35.924 INFO:tasks.rados.rados.0.smithi123.stdout:238: set_chunk oid 39 offset: 1510400 length: 629760 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1510400 2020-01-16T00:12:35.924 INFO:tasks.rados.rados.0.smithi123.stdout:239: set_chunk oid 40 offset: 733696 length: 1799168 target oid 40 low_tier tgt_offset: 733696 2020-01-16T00:12:35.925 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.925 INFO:tasks.rados.rados.0.smithi123.stdout:223: finishing set_chunk to oid 24 2020-01-16T00:12:35.925 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 224768 length: 1041408 chunk_info offset: 30720 length 1585664 2020-01-16T00:12:35.925 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 24 v 10 (ObjNum 23 snap 0 seq_num 23) dirty exists 2020-01-16T00:12:35.925 INFO:tasks.rados.rados.0.smithi123.stdout:228: finishing set_chunk to oid 29 2020-01-16T00:12:35.925 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 609280 length: 1707520 chunk_info offset: 1186304 length 657408 2020-01-16T00:12:35.925 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 29 v 11 (ObjNum 28 snap 0 seq_num 28) dirty exists 2020-01-16T00:12:35.925 INFO:tasks.rados.rados.0.smithi123.stdout:223: done (15 left) 2020-01-16T00:12:35.926 INFO:tasks.rados.rados.0.smithi123.stdout:228: done (14 left) 2020-01-16T00:12:35.926 INFO:tasks.rados.rados.0.smithi123.stdout:240: set_chunk oid 41 offset: 769536 length: 804864 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 769536 2020-01-16T00:12:35.926 INFO:tasks.rados.rados.0.smithi123.stdout:241: set_chunk oid 42 offset: 1089536 length: 949760 target oid 42 low_tier tgt_offset: 1089536 2020-01-16T00:12:35.926 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.926 INFO:tasks.rados.rados.0.smithi123.stdout:215: finishing set_chunk to oid 16 2020-01-16T00:12:35.926 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 384000 length: 1445888 chunk_info offset: 833536 length 364544 2020-01-16T00:12:35.926 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 16 v 17 (ObjNum 15 snap 0 seq_num 15) dirty exists 2020-01-16T00:12:35.927 INFO:tasks.rados.rados.0.smithi123.stdout:215: done (15 left) 2020-01-16T00:12:35.927 INFO:tasks.rados.rados.0.smithi123.stdout:242: set_chunk oid 43 offset: 1277952 length: 776192 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1277952 2020-01-16T00:12:35.927 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.927 INFO:tasks.rados.rados.0.smithi123.stdout:233: finishing set_chunk to oid 34 2020-01-16T00:12:35.927 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 34 v 12 (ObjNum 33 snap 0 seq_num 33) dirty exists 2020-01-16T00:12:35.927 INFO:tasks.rados.rados.0.smithi123.stdout:229: finishing set_chunk to oid 30 2020-01-16T00:12:35.927 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 30 v 18 (ObjNum 29 snap 0 seq_num 29) dirty exists 2020-01-16T00:12:35.928 INFO:tasks.rados.rados.0.smithi123.stdout:231: finishing set_chunk to oid 32 2020-01-16T00:12:35.928 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 32 v 18 (ObjNum 31 snap 0 seq_num 31) dirty exists 2020-01-16T00:12:35.928 INFO:tasks.rados.rados.0.smithi123.stdout:219: finishing set_chunk to oid 20 2020-01-16T00:12:35.928 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 20 v 28 (ObjNum 19 snap 0 seq_num 19) dirty exists 2020-01-16T00:12:35.928 INFO:tasks.rados.rados.0.smithi123.stdout:219: done (15 left) 2020-01-16T00:12:35.928 INFO:tasks.rados.rados.0.smithi123.stdout:229: done (14 left) 2020-01-16T00:12:35.928 INFO:tasks.rados.rados.0.smithi123.stdout:231: done (13 left) 2020-01-16T00:12:35.928 INFO:tasks.rados.rados.0.smithi123.stdout:233: done (12 left) 2020-01-16T00:12:35.929 INFO:tasks.rados.rados.0.smithi123.stdout:243: set_chunk oid 44 offset: 1466368 length: 829952 target oid 44 low_tier tgt_offset: 1466368 2020-01-16T00:12:35.929 INFO:tasks.rados.rados.0.smithi123.stdout:244: set_chunk oid 45 offset: 818688 length: 1325056 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 818688 2020-01-16T00:12:35.929 INFO:tasks.rados.rados.0.smithi123.stdout:245: set_chunk oid 46 offset: 282624 length: 322048 target oid 46 low_tier tgt_offset: 282624 2020-01-16T00:12:35.929 INFO:tasks.rados.rados.0.smithi123.stdout:246: set_chunk oid 47 offset: 123904 length: 1034752 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 123904 2020-01-16T00:12:35.929 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.929 INFO:tasks.rados.rados.0.smithi123.stdout:232: finishing set_chunk to oid 33 2020-01-16T00:12:35.929 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 32768 length: 2050560 chunk_info offset: 658944 length 1355776 2020-01-16T00:12:35.929 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 33 v 21 (ObjNum 32 snap 0 seq_num 32) dirty exists 2020-01-16T00:12:35.930 INFO:tasks.rados.rados.0.smithi123.stdout:232: done (15 left) 2020-01-16T00:12:35.930 INFO:tasks.rados.rados.0.smithi123.stdout:247: set_chunk oid 48 offset: 843776 length: 765440 target oid 48 low_tier tgt_offset: 843776 2020-01-16T00:12:35.930 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.930 INFO:tasks.rados.rados.0.smithi123.stdout:230: finishing set_chunk to oid 31 2020-01-16T00:12:35.930 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 707584 length: 1102336 chunk_info offset: 263680 length 1668096 2020-01-16T00:12:35.930 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 31 v 15 (ObjNum 30 snap 0 seq_num 30) dirty exists 2020-01-16T00:12:35.930 INFO:tasks.rados.rados.0.smithi123.stdout:230: done (15 left) 2020-01-16T00:12:35.931 INFO:tasks.rados.rados.0.smithi123.stdout:248: set_chunk oid 49 offset: 1658880 length: 398336 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1658880 2020-01-16T00:12:35.931 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.931 INFO:tasks.rados.rados.0.smithi123.stdout:236: finishing set_chunk to oid 37 2020-01-16T00:12:35.931 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 10240 length: 1890816 chunk_info offset: 574464 length 1014784 2020-01-16T00:12:35.931 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 37 v 22 (ObjNum 36 snap 0 seq_num 36) dirty exists 2020-01-16T00:12:35.931 INFO:tasks.rados.rados.0.smithi123.stdout:236: done (15 left) 2020-01-16T00:12:35.931 INFO:tasks.rados.rados.0.smithi123.stdout:249: set_chunk oid 50 offset: 24576 length: 2594816 target oid 50 low_tier tgt_offset: 24576 2020-01-16T00:12:35.931 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.932 INFO:tasks.rados.rados.0.smithi123.stdout:224: finishing set_chunk to oid 25 2020-01-16T00:12:35.932 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 722944 length: 987648 chunk_info offset: 84992 length 680448 2020-01-16T00:12:35.932 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 25 v 24 (ObjNum 24 snap 0 seq_num 24) dirty exists 2020-01-16T00:12:35.932 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 722944 length: 987648 chunk_info offset: 773632 length 1419776 2020-01-16T00:12:35.932 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 25 v 24 (ObjNum 24 snap 0 seq_num 24) dirty exists 2020-01-16T00:12:35.932 INFO:tasks.rados.rados.0.smithi123.stdout:224: done (15 left) 2020-01-16T00:12:35.932 INFO:tasks.rados.rados.0.smithi123.stdout:250: set_chunk oid 1 offset: 1072640 length: 860160 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1072640 2020-01-16T00:12:35.933 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.933 INFO:tasks.rados.rados.0.smithi123.stdout:234: finishing set_chunk to oid 35 2020-01-16T00:12:35.933 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1769472 length: 453120 chunk_info offset: 1823744 length 153600 2020-01-16T00:12:35.933 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 35 v 20 (ObjNum 34 snap 0 seq_num 34) dirty exists 2020-01-16T00:12:35.933 INFO:tasks.rados.rados.0.smithi123.stdout:234: done (15 left) 2020-01-16T00:12:35.933 INFO:tasks.rados.rados.0.smithi123.stdout:251: set_chunk oid 2 offset: 1523712 length: 399360 target oid 2 low_tier tgt_offset: 1523712 2020-01-16T00:12:35.933 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.933 INFO:tasks.rados.rados.0.smithi123.stdout:240: finishing set_chunk to oid 41 2020-01-16T00:12:35.934 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 769536 length: 804864 chunk_info offset: 925184 length 293376 2020-01-16T00:12:35.934 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 41 v 16 (ObjNum 40 snap 0 seq_num 40) dirty exists 2020-01-16T00:12:35.934 INFO:tasks.rados.rados.0.smithi123.stdout:240: done (15 left) 2020-01-16T00:12:35.934 INFO:tasks.rados.rados.0.smithi123.stdout:252: set_chunk oid 3 offset: 1156096 length: 722944 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1156096 2020-01-16T00:12:35.934 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.934 INFO:tasks.rados.rados.0.smithi123.stdout:238: finishing set_chunk to oid 39 2020-01-16T00:12:35.934 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1510400 length: 629760 chunk_info offset: 1570304 length 12288 2020-01-16T00:12:35.935 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 39 v 22 (ObjNum 38 snap 0 seq_num 38) dirty exists 2020-01-16T00:12:35.935 INFO:tasks.rados.rados.0.smithi123.stdout:238: done (15 left) 2020-01-16T00:12:35.935 INFO:tasks.rados.rados.0.smithi123.stdout:253: set_chunk oid 4 offset: 803328 length: 176128 target oid 4 low_tier tgt_offset: 803328 2020-01-16T00:12:35.935 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.935 INFO:tasks.rados.rados.0.smithi123.stdout:235: finishing set_chunk to oid 36 2020-01-16T00:12:35.935 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 743936 length: 1061888 chunk_info offset: 643584 length 327680 2020-01-16T00:12:35.935 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 36 v 8 (ObjNum 35 snap 0 seq_num 35) dirty exists 2020-01-16T00:12:35.935 INFO:tasks.rados.rados.0.smithi123.stdout:235: done (15 left) 2020-01-16T00:12:35.936 INFO:tasks.rados.rados.0.smithi123.stdout:254: set_chunk oid 5 offset: 943616 length: 670208 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 943616 2020-01-16T00:12:35.936 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.936 INFO:tasks.rados.rados.0.smithi123.stdout:239: finishing set_chunk to oid 40 2020-01-16T00:12:35.936 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 733696 length: 1799168 chunk_info offset: 790528 length 688128 2020-01-16T00:12:35.936 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 40 v 23 (ObjNum 39 snap 0 seq_num 39) dirty exists 2020-01-16T00:12:35.936 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 733696 length: 1799168 chunk_info offset: 1926656 length 67072 2020-01-16T00:12:35.936 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 40 v 23 (ObjNum 39 snap 0 seq_num 39) dirty exists 2020-01-16T00:12:35.936 INFO:tasks.rados.rados.0.smithi123.stdout:239: done (15 left) 2020-01-16T00:12:35.937 INFO:tasks.rados.rados.0.smithi123.stdout:255: set_chunk oid 6 offset: 1143808 length: 849920 target oid 6 low_tier tgt_offset: 1143808 2020-01-16T00:12:35.937 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.937 INFO:tasks.rados.rados.0.smithi123.stdout:245: finishing set_chunk to oid 46 2020-01-16T00:12:35.937 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 282624 length: 322048 chunk_info offset: 56832 length 1408000 2020-01-16T00:12:35.937 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 46 v 23 (ObjNum 45 snap 0 seq_num 45) dirty exists 2020-01-16T00:12:35.938 INFO:tasks.rados.rados.0.smithi123.stdout:245: done (15 left) 2020-01-16T00:12:35.938 INFO:tasks.rados.rados.0.smithi123.stdout:256: set_chunk oid 7 offset: 674816 length: 219136 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 674816 2020-01-16T00:12:35.938 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.938 INFO:tasks.rados.rados.0.smithi123.stdout:244: finishing set_chunk to oid 45 2020-01-16T00:12:35.938 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 818688 length: 1325056 chunk_info offset: 640512 length 403968 2020-01-16T00:12:35.938 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 45 v 11 (ObjNum 44 snap 0 seq_num 44) dirty exists 2020-01-16T00:12:35.938 INFO:tasks.rados.rados.0.smithi123.stdout:244: done (15 left) 2020-01-16T00:12:35.939 INFO:tasks.rados.rados.0.smithi123.stdout:257: set_chunk oid 8 offset: 345088 length: 984576 target oid 8 low_tier tgt_offset: 345088 2020-01-16T00:12:35.939 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.939 INFO:tasks.rados.rados.0.smithi123.stdout:237: finishing set_chunk to oid 38 2020-01-16T00:12:35.939 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 586752 length: 1668096 chunk_info offset: 1915392 length 653312 2020-01-16T00:12:35.939 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 38 v 10 (ObjNum 37 snap 0 seq_num 37) dirty exists 2020-01-16T00:12:35.939 INFO:tasks.rados.rados.0.smithi123.stdout:242: finishing set_chunk to oid 43 2020-01-16T00:12:35.939 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1277952 length: 776192 chunk_info offset: 1384448 length 188416 2020-01-16T00:12:35.939 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 43 v 24 (ObjNum 42 snap 0 seq_num 42) dirty exists 2020-01-16T00:12:35.940 INFO:tasks.rados.rados.0.smithi123.stdout:243: finishing set_chunk to oid 44 2020-01-16T00:12:35.940 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1466368 length: 829952 chunk_info offset: 1635840 length 423424 2020-01-16T00:12:35.940 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 44 v 25 (ObjNum 43 snap 0 seq_num 43) dirty exists 2020-01-16T00:12:35.940 INFO:tasks.rados.rados.0.smithi123.stdout:250: finishing set_chunk to oid 1 2020-01-16T00:12:35.940 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1072640 length: 860160 chunk_info offset: 1046016 length 729088 2020-01-16T00:12:35.940 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 1 v 24 (ObjNum 0 snap 0 seq_num 0) dirty exists 2020-01-16T00:12:35.940 INFO:tasks.rados.rados.0.smithi123.stdout:241: finishing set_chunk to oid 42 2020-01-16T00:12:35.940 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1089536 length: 949760 chunk_info offset: 1364992 length 1145344 2020-01-16T00:12:35.941 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 42 v 11 (ObjNum 41 snap 0 seq_num 41) dirty exists 2020-01-16T00:12:35.941 INFO:tasks.rados.rados.0.smithi123.stdout:246: finishing set_chunk to oid 47 2020-01-16T00:12:35.941 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 47 v 30 (ObjNum 46 snap 0 seq_num 46) dirty exists 2020-01-16T00:12:35.941 INFO:tasks.rados.rados.0.smithi123.stdout:247: finishing set_chunk to oid 48 2020-01-16T00:12:35.941 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 843776 length: 765440 chunk_info offset: 264704 length 1399296 2020-01-16T00:12:35.941 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 48 v 12 (ObjNum 47 snap 0 seq_num 47) dirty exists 2020-01-16T00:12:35.941 INFO:tasks.rados.rados.0.smithi123.stdout:248: finishing set_chunk to oid 49 2020-01-16T00:12:35.941 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 49 v 18 (ObjNum 48 snap 0 seq_num 48) dirty exists 2020-01-16T00:12:35.942 INFO:tasks.rados.rados.0.smithi123.stdout:251: finishing set_chunk to oid 2 2020-01-16T00:12:35.942 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1523712 length: 399360 chunk_info offset: 476160 length 1240064 2020-01-16T00:12:35.942 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 2 v 13 (ObjNum 1 snap 0 seq_num 1) dirty exists 2020-01-16T00:12:35.942 INFO:tasks.rados.rados.0.smithi123.stdout:237: done (15 left) 2020-01-16T00:12:35.942 INFO:tasks.rados.rados.0.smithi123.stdout:241: done (14 left) 2020-01-16T00:12:35.942 INFO:tasks.rados.rados.0.smithi123.stdout:242: done (13 left) 2020-01-16T00:12:35.942 INFO:tasks.rados.rados.0.smithi123.stdout:243: done (12 left) 2020-01-16T00:12:35.942 INFO:tasks.rados.rados.0.smithi123.stdout:246: done (11 left) 2020-01-16T00:12:35.943 INFO:tasks.rados.rados.0.smithi123.stdout:247: done (10 left) 2020-01-16T00:12:35.943 INFO:tasks.rados.rados.0.smithi123.stdout:248: done (9 left) 2020-01-16T00:12:35.943 INFO:tasks.rados.rados.0.smithi123.stdout:250: done (8 left) 2020-01-16T00:12:35.943 INFO:tasks.rados.rados.0.smithi123.stdout:251: done (7 left) 2020-01-16T00:12:35.943 INFO:tasks.rados.rados.0.smithi123.stdout:258: set_chunk oid 9 offset: 324096 length: 533504 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 324096 2020-01-16T00:12:35.943 INFO:tasks.rados.rados.0.smithi123.stdout:259: set_chunk oid 10 offset: 330240 length: 551424 target oid 10 low_tier tgt_offset: 330240 2020-01-16T00:12:35.943 INFO:tasks.rados.rados.0.smithi123.stdout:260: set_chunk oid 11 offset: 607744 length: 634880 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 607744 2020-01-16T00:12:35.944 INFO:tasks.rados.rados.0.smithi123.stdout:261: set_chunk oid 12 offset: 946176 length: 1011200 target oid 12 low_tier tgt_offset: 946176 2020-01-16T00:12:35.944 INFO:tasks.rados.rados.0.smithi123.stdout:262: set_chunk oid 13 offset: 4096 length: 737792 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 4096 2020-01-16T00:12:35.944 INFO:tasks.rados.rados.0.smithi123.stdout:263: set_chunk oid 14 offset: 478208 length: 749056 target oid 14 low_tier tgt_offset: 478208 2020-01-16T00:12:35.944 INFO:tasks.rados.rados.0.smithi123.stdout:264: set_chunk oid 15 offset: 1168384 length: 976896 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1168384 2020-01-16T00:12:35.944 INFO:tasks.rados.rados.0.smithi123.stdout:265: set_chunk oid 16 offset: 517632 length: 45056 target oid 16 low_tier tgt_offset: 517632 2020-01-16T00:12:35.944 INFO:tasks.rados.rados.0.smithi123.stdout:266: set_chunk oid 17 offset: 1497600 length: 145920 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1497600 2020-01-16T00:12:35.944 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.944 INFO:tasks.rados.rados.0.smithi123.stdout:249: finishing set_chunk to oid 50 2020-01-16T00:12:35.945 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 24576 length: 2594816 chunk_info offset: 1613312 length 498688 2020-01-16T00:12:35.945 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 50 v 16 (ObjNum 49 snap 0 seq_num 49) dirty exists 2020-01-16T00:12:35.945 INFO:tasks.rados.rados.0.smithi123.stdout:256: finishing set_chunk to oid 7 2020-01-16T00:12:35.945 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 674816 length: 219136 chunk_info offset: 160768 length 1134080 2020-01-16T00:12:35.945 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 7 v 14 (ObjNum 6 snap 0 seq_num 6) dirty exists 2020-01-16T00:12:35.945 INFO:tasks.rados.rados.0.smithi123.stdout:254: finishing set_chunk to oid 5 2020-01-16T00:12:35.945 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 943616 length: 670208 chunk_info offset: 102400 length 870912 2020-01-16T00:12:35.946 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 5 v 21 (ObjNum 4 snap 0 seq_num 4) dirty exists 2020-01-16T00:12:35.946 INFO:tasks.rados.rados.0.smithi123.stdout:253: finishing set_chunk to oid 4 2020-01-16T00:12:35.946 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 4 v 25 (ObjNum 3 snap 0 seq_num 3) dirty exists 2020-01-16T00:12:35.946 INFO:tasks.rados.rados.0.smithi123.stdout:255: finishing set_chunk to oid 6 2020-01-16T00:12:35.946 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1143808 length: 849920 chunk_info offset: 878592 length 645632 2020-01-16T00:12:35.946 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 6 v 10 (ObjNum 5 snap 0 seq_num 5) dirty exists 2020-01-16T00:12:35.946 INFO:tasks.rados.rados.0.smithi123.stdout:249: done (15 left) 2020-01-16T00:12:35.946 INFO:tasks.rados.rados.0.smithi123.stdout:253: done (14 left) 2020-01-16T00:12:35.947 INFO:tasks.rados.rados.0.smithi123.stdout:254: done (13 left) 2020-01-16T00:12:35.947 INFO:tasks.rados.rados.0.smithi123.stdout:255: done (12 left) 2020-01-16T00:12:35.947 INFO:tasks.rados.rados.0.smithi123.stdout:256: done (11 left) 2020-01-16T00:12:35.947 INFO:tasks.rados.rados.0.smithi123.stdout:267: set_chunk oid 18 offset: 581632 length: 1234944 target oid 18 low_tier tgt_offset: 581632 2020-01-16T00:12:35.947 INFO:tasks.rados.rados.0.smithi123.stdout:268: set_chunk oid 19 offset: 291328 length: 483840 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 291328 2020-01-16T00:12:35.947 INFO:tasks.rados.rados.0.smithi123.stdout:269: set_chunk oid 20 offset: 385536 length: 861184 target oid 20 low_tier tgt_offset: 385536 2020-01-16T00:12:35.947 INFO:tasks.rados.rados.0.smithi123.stdout:270: set_chunk oid 21 offset: 78848 length: 485888 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 78848 2020-01-16T00:12:35.947 INFO:tasks.rados.rados.0.smithi123.stdout:271: set_chunk oid 22 offset: 2161152 length: 4096 target oid 22 low_tier tgt_offset: 2161152 2020-01-16T00:12:35.948 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.948 INFO:tasks.rados.rados.0.smithi123.stdout:258: finishing set_chunk to oid 9 2020-01-16T00:12:35.948 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 324096 length: 533504 chunk_info offset: 514560 length 177152 2020-01-16T00:12:35.948 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 9 v 16 (ObjNum 8 snap 0 seq_num 8) dirty exists 2020-01-16T00:12:35.948 INFO:tasks.rados.rados.0.smithi123.stdout:258: done (15 left) 2020-01-16T00:12:35.948 INFO:tasks.rados.rados.0.smithi123.stdout:272: set_chunk oid 23 offset: 901120 length: 1019392 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 901120 2020-01-16T00:12:35.948 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.948 INFO:tasks.rados.rados.0.smithi123.stdout:263: finishing set_chunk to oid 14 2020-01-16T00:12:35.949 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 478208 length: 749056 chunk_info offset: 470528 length 1206272 2020-01-16T00:12:35.949 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 14 v 26 (ObjNum 13 snap 0 seq_num 13) dirty exists 2020-01-16T00:12:35.949 INFO:tasks.rados.rados.0.smithi123.stdout:263: done (15 left) 2020-01-16T00:12:35.949 INFO:tasks.rados.rados.0.smithi123.stdout:273: set_chunk oid 24 offset: 219648 length: 1546752 target oid 24 low_tier tgt_offset: 219648 2020-01-16T00:12:35.949 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.949 INFO:tasks.rados.rados.0.smithi123.stdout:262: finishing set_chunk to oid 13 2020-01-16T00:12:35.949 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 13 (ObjNum 12 snap 0 seq_num 12) dirty exists 2020-01-16T00:12:35.949 INFO:tasks.rados.rados.0.smithi123.stdout:252: finishing set_chunk to oid 3 2020-01-16T00:12:35.950 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 3 v 13 (ObjNum 2 snap 0 seq_num 2) dirty exists 2020-01-16T00:12:35.950 INFO:tasks.rados.rados.0.smithi123.stdout:257: finishing set_chunk to oid 8 2020-01-16T00:12:35.950 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 345088 length: 984576 chunk_info offset: 285696 length 965632 2020-01-16T00:12:35.950 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 8 v 10 (ObjNum 7 snap 0 seq_num 7) dirty exists 2020-01-16T00:12:35.950 INFO:tasks.rados.rados.0.smithi123.stdout:260: finishing set_chunk to oid 11 2020-01-16T00:12:35.950 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 607744 length: 634880 chunk_info offset: 279552 length 1427968 2020-01-16T00:12:35.950 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 11 v 14 (ObjNum 10 snap 0 seq_num 10) dirty exists 2020-01-16T00:12:35.950 INFO:tasks.rados.rados.0.smithi123.stdout:252: done (15 left) 2020-01-16T00:12:35.951 INFO:tasks.rados.rados.0.smithi123.stdout:257: done (14 left) 2020-01-16T00:12:35.951 INFO:tasks.rados.rados.0.smithi123.stdout:260: done (13 left) 2020-01-16T00:12:35.951 INFO:tasks.rados.rados.0.smithi123.stdout:262: done (12 left) 2020-01-16T00:12:35.951 INFO:tasks.rados.rados.0.smithi123.stdout:274: set_chunk oid 25 offset: 1082368 length: 608768 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1082368 2020-01-16T00:12:35.951 INFO:tasks.rados.rados.0.smithi123.stdout:275: set_chunk oid 26 offset: 661504 length: 1265152 target oid 26 low_tier tgt_offset: 661504 2020-01-16T00:12:35.951 INFO:tasks.rados.rados.0.smithi123.stdout:276: set_chunk oid 27 offset: 1416192 length: 189952 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1416192 2020-01-16T00:12:35.951 INFO:tasks.rados.rados.0.smithi123.stdout:277: set_chunk oid 28 offset: 1038848 length: 497664 target oid 28 low_tier tgt_offset: 1038848 2020-01-16T00:12:35.952 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.952 INFO:tasks.rados.rados.0.smithi123.stdout:266: finishing set_chunk to oid 17 2020-01-16T00:12:35.952 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 17 v 26 (ObjNum 16 snap 0 seq_num 16) dirty exists 2020-01-16T00:12:35.952 INFO:tasks.rados.rados.0.smithi123.stdout:266: done (15 left) 2020-01-16T00:12:35.952 INFO:tasks.rados.rados.0.smithi123.stdout:278: set_chunk oid 29 offset: 1016320 length: 286720 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1016320 2020-01-16T00:12:35.952 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.952 INFO:tasks.rados.rados.0.smithi123.stdout:261: finishing set_chunk to oid 12 2020-01-16T00:12:35.952 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 946176 length: 1011200 chunk_info offset: 1033216 length 672768 2020-01-16T00:12:35.953 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 12 v 12 (ObjNum 11 snap 0 seq_num 11) dirty exists 2020-01-16T00:12:35.953 INFO:tasks.rados.rados.0.smithi123.stdout:261: done (15 left) 2020-01-16T00:12:35.953 INFO:tasks.rados.rados.0.smithi123.stdout:279: set_chunk oid 30 offset: 1076224 length: 252928 target oid 30 low_tier tgt_offset: 1076224 2020-01-16T00:12:35.953 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.953 INFO:tasks.rados.rados.0.smithi123.stdout:265: finishing set_chunk to oid 16 2020-01-16T00:12:35.953 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 16 v 32 (ObjNum 15 snap 0 seq_num 15) dirty exists 2020-01-16T00:12:35.953 INFO:tasks.rados.rados.0.smithi123.stdout:265: done (15 left) 2020-01-16T00:12:35.954 INFO:tasks.rados.rados.0.smithi123.stdout:280: set_chunk oid 31 offset: 1091072 length: 338944 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1091072 2020-01-16T00:12:35.954 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.954 INFO:tasks.rados.rados.0.smithi123.stdout:264: finishing set_chunk to oid 15 2020-01-16T00:12:35.954 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1168384 length: 976896 chunk_info offset: 1708032 length 408576 2020-01-16T00:12:35.954 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 15 v 17 (ObjNum 14 snap 0 seq_num 14) dirty exists 2020-01-16T00:12:35.954 INFO:tasks.rados.rados.0.smithi123.stdout:264: done (15 left) 2020-01-16T00:12:35.954 INFO:tasks.rados.rados.0.smithi123.stdout:281: set_chunk oid 32 offset: 412160 length: 724992 target oid 32 low_tier tgt_offset: 412160 2020-01-16T00:12:35.955 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.955 INFO:tasks.rados.rados.0.smithi123.stdout:267: finishing set_chunk to oid 18 2020-01-16T00:12:35.955 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 581632 length: 1234944 chunk_info offset: 953856 length 471040 2020-01-16T00:12:35.955 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 18 v 31 (ObjNum 17 snap 0 seq_num 17) dirty exists 2020-01-16T00:12:35.955 INFO:tasks.rados.rados.0.smithi123.stdout:267: done (15 left) 2020-01-16T00:12:35.955 INFO:tasks.rados.rados.0.smithi123.stdout:282: set_chunk oid 33 offset: 286720 length: 1025536 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 286720 2020-01-16T00:12:35.955 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.955 INFO:tasks.rados.rados.0.smithi123.stdout:277: finishing set_chunk to oid 28 2020-01-16T00:12:35.956 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1038848 length: 497664 chunk_info offset: 1498624 length 783872 2020-01-16T00:12:35.956 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 28 v 14 (ObjNum 27 snap 0 seq_num 27) dirty exists 2020-01-16T00:12:35.956 INFO:tasks.rados.rados.0.smithi123.stdout:277: done (15 left) 2020-01-16T00:12:35.956 INFO:tasks.rados.rados.0.smithi123.stdout:283: set_chunk oid 34 offset: 1198592 length: 1285632 target oid 34 low_tier tgt_offset: 1198592 2020-01-16T00:12:35.956 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.956 INFO:tasks.rados.rados.0.smithi123.stdout:276: finishing set_chunk to oid 27 2020-01-16T00:12:35.957 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1416192 length: 189952 chunk_info offset: 113152 length 1699840 2020-01-16T00:12:35.957 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 27 v 11 (ObjNum 26 snap 0 seq_num 26) dirty exists 2020-01-16T00:12:35.957 INFO:tasks.rados.rados.0.smithi123.stdout:276: done (15 left) 2020-01-16T00:12:35.957 INFO:tasks.rados.rados.0.smithi123.stdout:284: set_chunk oid 35 offset: 2168320 length: 226816 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 2168320 2020-01-16T00:12:35.957 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.957 INFO:tasks.rados.rados.0.smithi123.stdout:259: finishing set_chunk to oid 10 2020-01-16T00:12:35.957 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 330240 length: 551424 chunk_info offset: 453120 length 360448 2020-01-16T00:12:35.958 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 10 v 16 (ObjNum 9 snap 0 seq_num 9) dirty exists 2020-01-16T00:12:35.958 INFO:tasks.rados.rados.0.smithi123.stdout:259: done (15 left) 2020-01-16T00:12:35.958 INFO:tasks.rados.rados.0.smithi123.stdout:285: set_chunk oid 36 offset: 141824 length: 2272768 target oid 36 low_tier tgt_offset: 141824 2020-01-16T00:12:35.958 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.958 INFO:tasks.rados.rados.0.smithi123.stdout:268: finishing set_chunk to oid 19 2020-01-16T00:12:35.958 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 291328 length: 483840 chunk_info offset: 592384 length 1625600 2020-01-16T00:12:35.958 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 19 v 15 (ObjNum 18 snap 0 seq_num 18) dirty exists 2020-01-16T00:12:35.959 INFO:tasks.rados.rados.0.smithi123.stdout:268: done (15 left) 2020-01-16T00:12:35.959 INFO:tasks.rados.rados.0.smithi123.stdout:286: set_chunk oid 37 offset: 297984 length: 975360 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 297984 2020-01-16T00:12:35.959 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.959 INFO:tasks.rados.rados.0.smithi123.stdout:269: finishing set_chunk to oid 20 2020-01-16T00:12:35.959 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 20 v 33 (ObjNum 19 snap 0 seq_num 19) dirty exists 2020-01-16T00:12:35.959 INFO:tasks.rados.rados.0.smithi123.stdout:274: finishing set_chunk to oid 25 2020-01-16T00:12:35.959 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1082368 length: 608768 chunk_info offset: 773632 length 1419776 2020-01-16T00:12:35.959 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 25 v 24 (ObjNum 24 snap 0 seq_num 24) dirty exists 2020-01-16T00:12:35.960 INFO:tasks.rados.rados.0.smithi123.stdout:271: finishing set_chunk to oid 22 2020-01-16T00:12:35.960 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 22 v 7 (ObjNum 21 snap 0 seq_num 21) dirty exists 2020-01-16T00:12:35.960 INFO:tasks.rados.rados.0.smithi123.stdout:270: finishing set_chunk to oid 21 2020-01-16T00:12:35.960 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 78848 length: 485888 chunk_info offset: 346112 length 838656 2020-01-16T00:12:35.960 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 21 v 9 (ObjNum 20 snap 0 seq_num 20) dirty exists 2020-01-16T00:12:35.960 INFO:tasks.rados.rados.0.smithi123.stdout:273: finishing set_chunk to oid 24 2020-01-16T00:12:35.960 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 219648 length: 1546752 chunk_info offset: 30720 length 1585664 2020-01-16T00:12:35.961 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 24 v 10 (ObjNum 23 snap 0 seq_num 23) dirty exists 2020-01-16T00:12:35.961 INFO:tasks.rados.rados.0.smithi123.stdout:278: finishing set_chunk to oid 29 2020-01-16T00:12:35.961 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1016320 length: 286720 chunk_info offset: 1186304 length 657408 2020-01-16T00:12:35.961 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 29 v 11 (ObjNum 28 snap 0 seq_num 28) dirty exists 2020-01-16T00:12:35.961 INFO:tasks.rados.rados.0.smithi123.stdout:269: done (15 left) 2020-01-16T00:12:35.961 INFO:tasks.rados.rados.0.smithi123.stdout:270: done (14 left) 2020-01-16T00:12:35.961 INFO:tasks.rados.rados.0.smithi123.stdout:271: done (13 left) 2020-01-16T00:12:35.961 INFO:tasks.rados.rados.0.smithi123.stdout:273: done (12 left) 2020-01-16T00:12:35.962 INFO:tasks.rados.rados.0.smithi123.stdout:274: done (11 left) 2020-01-16T00:12:35.962 INFO:tasks.rados.rados.0.smithi123.stdout:278: done (10 left) 2020-01-16T00:12:35.962 INFO:tasks.rados.rados.0.smithi123.stdout:287: set_chunk oid 38 offset: 1180160 length: 1026048 target oid 38 low_tier tgt_offset: 1180160 2020-01-16T00:12:35.962 INFO:tasks.rados.rados.0.smithi123.stdout:288: set_chunk oid 39 offset: 763392 length: 413696 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 763392 2020-01-16T00:12:35.962 INFO:tasks.rados.rados.0.smithi123.stdout:289: set_chunk oid 40 offset: 833024 length: 525312 target oid 40 low_tier tgt_offset: 833024 2020-01-16T00:12:35.962 INFO:tasks.rados.rados.0.smithi123.stdout:290: set_chunk oid 41 offset: 1808384 length: 557056 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1808384 2020-01-16T00:12:35.962 INFO:tasks.rados.rados.0.smithi123.stdout:291: set_chunk oid 42 offset: 883712 length: 1169408 target oid 42 low_tier tgt_offset: 883712 2020-01-16T00:12:35.963 INFO:tasks.rados.rados.0.smithi123.stdout:292: set_chunk oid 43 offset: 1161216 length: 1473024 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1161216 2020-01-16T00:12:35.963 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.963 INFO:tasks.rados.rados.0.smithi123.stdout:279: finishing set_chunk to oid 30 2020-01-16T00:12:35.963 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1076224 length: 252928 chunk_info offset: 1087488 length 253952 2020-01-16T00:12:35.963 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 30 v 18 (ObjNum 29 snap 0 seq_num 29) dirty exists 2020-01-16T00:12:35.963 INFO:tasks.rados.rados.0.smithi123.stdout:279: done (15 left) 2020-01-16T00:12:35.963 INFO:tasks.rados.rados.0.smithi123.stdout:293: set_chunk oid 44 offset: 239104 length: 1904128 target oid 44 low_tier tgt_offset: 239104 2020-01-16T00:12:35.963 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.964 INFO:tasks.rados.rados.0.smithi123.stdout:283: finishing set_chunk to oid 34 2020-01-16T00:12:35.964 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1198592 length: 1285632 chunk_info offset: 901632 length 1154560 2020-01-16T00:12:35.964 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 34 v 12 (ObjNum 33 snap 0 seq_num 33) dirty exists 2020-01-16T00:12:35.964 INFO:tasks.rados.rados.0.smithi123.stdout:283: done (15 left) 2020-01-16T00:12:35.964 INFO:tasks.rados.rados.0.smithi123.stdout:294: set_chunk oid 45 offset: 773632 length: 1245184 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 773632 2020-01-16T00:12:35.964 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.964 INFO:tasks.rados.rados.0.smithi123.stdout:275: finishing set_chunk to oid 26 2020-01-16T00:12:35.964 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 661504 length: 1265152 chunk_info offset: 963072 length 376320 2020-01-16T00:12:35.965 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 26 v 26 (ObjNum 25 snap 0 seq_num 25) dirty exists 2020-01-16T00:12:35.965 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 661504 length: 1265152 chunk_info offset: 1494016 length 315392 2020-01-16T00:12:35.965 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 26 v 26 (ObjNum 25 snap 0 seq_num 25) dirty exists 2020-01-16T00:12:35.965 INFO:tasks.rados.rados.0.smithi123.stdout:272: finishing set_chunk to oid 23 2020-01-16T00:12:35.965 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 901120 length: 1019392 chunk_info offset: 808960 length 1406976 2020-01-16T00:12:35.965 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 23 v 15 (ObjNum 22 snap 0 seq_num 22) dirty exists 2020-01-16T00:12:35.965 INFO:tasks.rados.rados.0.smithi123.stdout:272: done (15 left) 2020-01-16T00:12:35.965 INFO:tasks.rados.rados.0.smithi123.stdout:275: done (14 left) 2020-01-16T00:12:35.966 INFO:tasks.rados.rados.0.smithi123.stdout:295: set_chunk oid 46 offset: 28160 length: 2592768 target oid 46 low_tier tgt_offset: 28160 2020-01-16T00:12:35.966 INFO:tasks.rados.rados.0.smithi123.stdout:296: set_chunk oid 47 offset: 749568 length: 1699328 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 749568 2020-01-16T00:12:35.966 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.966 INFO:tasks.rados.rados.0.smithi123.stdout:281: finishing set_chunk to oid 32 2020-01-16T00:12:35.966 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 412160 length: 724992 chunk_info offset: 605696 length 877568 2020-01-16T00:12:35.966 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 32 v 0 (ObjNum 31 snap 0 seq_num 31) dirty exists 2020-01-16T00:12:35.966 INFO:tasks.rados.rados.0.smithi123.stdout:280: finishing set_chunk to oid 31 2020-01-16T00:12:35.966 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1091072 length: 338944 chunk_info offset: 263680 length 1668096 2020-01-16T00:12:35.967 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 31 v 0 (ObjNum 30 snap 0 seq_num 30) dirty exists 2020-01-16T00:12:35.967 INFO:tasks.rados.rados.0.smithi123.stdout:280: done (15 left) 2020-01-16T00:12:35.967 INFO:tasks.rados.rados.0.smithi123.stdout:281: done (14 left) 2020-01-16T00:12:35.967 INFO:tasks.rados.rados.0.smithi123.stdout:297: set_chunk oid 48 offset: 1871872 length: 474624 target oid 48 low_tier tgt_offset: 1871872 2020-01-16T00:12:35.967 INFO:tasks.rados.rados.0.smithi123.stdout:298: set_chunk oid 49 offset: 776192 length: 1499648 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 776192 2020-01-16T00:12:35.967 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.967 INFO:tasks.rados.rados.0.smithi123.stdout:285: finishing set_chunk to oid 36 2020-01-16T00:12:35.967 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 141824 length: 2272768 chunk_info offset: 643584 length 327680 2020-01-16T00:12:35.968 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 36 v 8 (ObjNum 35 snap 0 seq_num 35) dirty exists 2020-01-16T00:12:35.968 INFO:tasks.rados.rados.0.smithi123.stdout:285: done (15 left) 2020-01-16T00:12:35.968 INFO:tasks.rados.rados.0.smithi123.stdout:299: set_chunk oid 50 offset: 1515520 length: 1107456 target oid 50 low_tier tgt_offset: 1515520 2020-01-16T00:12:35.968 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.968 INFO:tasks.rados.rados.0.smithi123.stdout:287: finishing set_chunk to oid 38 2020-01-16T00:12:35.968 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1180160 length: 1026048 chunk_info offset: 1915392 length 653312 2020-01-16T00:12:35.968 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 38 v 10 (ObjNum 37 snap 0 seq_num 37) dirty exists 2020-01-16T00:12:35.969 INFO:tasks.rados.rados.0.smithi123.stdout:287: done (15 left) 2020-01-16T00:12:35.969 INFO:tasks.rados.rados.0.smithi123.stdout:300: set_chunk oid 1 offset: 1474048 length: 436736 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1474048 2020-01-16T00:12:35.969 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.969 INFO:tasks.rados.rados.0.smithi123.stdout:291: finishing set_chunk to oid 42 2020-01-16T00:12:35.969 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 883712 length: 1169408 chunk_info offset: 1364992 length 1145344 2020-01-16T00:12:35.969 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 42 v 11 (ObjNum 41 snap 0 seq_num 41) dirty exists 2020-01-16T00:12:35.969 INFO:tasks.rados.rados.0.smithi123.stdout:290: finishing set_chunk to oid 41 2020-01-16T00:12:35.970 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 41 v 28 (ObjNum 40 snap 0 seq_num 40) dirty exists 2020-01-16T00:12:35.970 INFO:tasks.rados.rados.0.smithi123.stdout:290: done (15 left) 2020-01-16T00:12:35.970 INFO:tasks.rados.rados.0.smithi123.stdout:291: done (14 left) 2020-01-16T00:12:35.970 INFO:tasks.rados.rados.0.smithi123.stdout:301: set_chunk oid 2 offset: 373760 length: 1170944 target oid 2 low_tier tgt_offset: 373760 2020-01-16T00:12:35.970 INFO:tasks.rados.rados.0.smithi123.stdout:302: set_chunk oid 3 offset: 868864 length: 950272 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 868864 2020-01-16T00:12:35.970 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.970 INFO:tasks.rados.rados.0.smithi123.stdout:282: finishing set_chunk to oid 33 2020-01-16T00:12:35.970 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 286720 length: 1025536 chunk_info offset: 658944 length 1355776 2020-01-16T00:12:35.971 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 33 v 21 (ObjNum 32 snap 0 seq_num 32) dirty exists 2020-01-16T00:12:35.971 INFO:tasks.rados.rados.0.smithi123.stdout:284: finishing set_chunk to oid 35 2020-01-16T00:12:35.971 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 35 v 28 (ObjNum 34 snap 0 seq_num 34) dirty exists 2020-01-16T00:12:35.971 INFO:tasks.rados.rados.0.smithi123.stdout:282: done (15 left) 2020-01-16T00:12:35.971 INFO:tasks.rados.rados.0.smithi123.stdout:284: done (14 left) 2020-01-16T00:12:35.971 INFO:tasks.rados.rados.0.smithi123.stdout:303: set_chunk oid 4 offset: 749568 length: 1125376 target oid 4 low_tier tgt_offset: 749568 2020-01-16T00:12:35.971 INFO:tasks.rados.rados.0.smithi123.stdout:304: set_chunk oid 5 offset: 730624 length: 700416 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 730624 2020-01-16T00:12:35.971 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.972 INFO:tasks.rados.rados.0.smithi123.stdout:294: finishing set_chunk to oid 45 2020-01-16T00:12:35.972 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 773632 length: 1245184 chunk_info offset: 640512 length 403968 2020-01-16T00:12:35.972 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 45 v 11 (ObjNum 44 snap 0 seq_num 44) dirty exists 2020-01-16T00:12:35.972 INFO:tasks.rados.rados.0.smithi123.stdout:294: done (15 left) 2020-01-16T00:12:35.972 INFO:tasks.rados.rados.0.smithi123.stdout:305: set_chunk oid 6 offset: 1244672 length: 788480 target oid 6 low_tier tgt_offset: 1244672 2020-01-16T00:12:35.972 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.972 INFO:tasks.rados.rados.0.smithi123.stdout:288: finishing set_chunk to oid 39 2020-01-16T00:12:35.972 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 39 v 32 (ObjNum 38 snap 0 seq_num 38) dirty exists 2020-01-16T00:12:35.973 INFO:tasks.rados.rados.0.smithi123.stdout:289: finishing set_chunk to oid 40 2020-01-16T00:12:35.973 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 833024 length: 525312 chunk_info offset: 790528 length 688128 2020-01-16T00:12:35.973 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 40 v 23 (ObjNum 39 snap 0 seq_num 39) dirty exists 2020-01-16T00:12:35.973 INFO:tasks.rados.rados.0.smithi123.stdout:288: done (15 left) 2020-01-16T00:12:35.973 INFO:tasks.rados.rados.0.smithi123.stdout:289: done (14 left) 2020-01-16T00:12:35.973 INFO:tasks.rados.rados.0.smithi123.stdout:306: set_chunk oid 7 offset: 637440 length: 1375232 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 637440 2020-01-16T00:12:35.973 INFO:tasks.rados.rados.0.smithi123.stdout:307: set_chunk oid 8 offset: 812544 length: 1063424 target oid 8 low_tier tgt_offset: 812544 2020-01-16T00:12:35.973 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.974 INFO:tasks.rados.rados.0.smithi123.stdout:292: finishing set_chunk to oid 43 2020-01-16T00:12:35.974 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1161216 length: 1473024 chunk_info offset: 1384448 length 188416 2020-01-16T00:12:35.974 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 43 v 24 (ObjNum 42 snap 0 seq_num 42) dirty exists 2020-01-16T00:12:35.974 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1161216 length: 1473024 chunk_info offset: 2331136 length 192000 2020-01-16T00:12:35.974 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 43 v 24 (ObjNum 42 snap 0 seq_num 42) dirty exists 2020-01-16T00:12:35.974 INFO:tasks.rados.rados.0.smithi123.stdout:292: done (15 left) 2020-01-16T00:12:35.974 INFO:tasks.rados.rados.0.smithi123.stdout:308: set_chunk oid 9 offset: 781312 length: 2048 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 781312 2020-01-16T00:12:35.974 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.975 INFO:tasks.rados.rados.0.smithi123.stdout:296: finishing set_chunk to oid 47 2020-01-16T00:12:35.975 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 749568 length: 1699328 chunk_info offset: 123904 length 1034752 2020-01-16T00:12:35.975 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 47 v 30 (ObjNum 46 snap 0 seq_num 46) dirty exists 2020-01-16T00:12:35.975 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 749568 length: 1699328 chunk_info offset: 2092032 length 38400 2020-01-16T00:12:35.975 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 47 v 30 (ObjNum 46 snap 0 seq_num 46) dirty exists 2020-01-16T00:12:35.975 INFO:tasks.rados.rados.0.smithi123.stdout:296: done (15 left) 2020-01-16T00:12:35.975 INFO:tasks.rados.rados.0.smithi123.stdout:309: set_chunk oid 10 offset: 1087488 length: 573440 target oid 10 low_tier tgt_offset: 1087488 2020-01-16T00:12:35.975 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.976 INFO:tasks.rados.rados.0.smithi123.stdout:286: finishing set_chunk to oid 37 2020-01-16T00:12:35.976 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 297984 length: 975360 chunk_info offset: 574464 length 1014784 2020-01-16T00:12:35.976 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 37 v 22 (ObjNum 36 snap 0 seq_num 36) dirty exists 2020-01-16T00:12:35.976 INFO:tasks.rados.rados.0.smithi123.stdout:298: finishing set_chunk to oid 49 2020-01-16T00:12:35.976 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 776192 length: 1499648 chunk_info offset: 453632 length 1075200 2020-01-16T00:12:35.976 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 49 v 18 (ObjNum 48 snap 0 seq_num 48) dirty exists 2020-01-16T00:12:35.976 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 776192 length: 1499648 chunk_info offset: 1658880 length 398336 2020-01-16T00:12:35.977 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 49 v 18 (ObjNum 48 snap 0 seq_num 48) dirty exists 2020-01-16T00:12:35.977 INFO:tasks.rados.rados.0.smithi123.stdout:293: finishing set_chunk to oid 44 2020-01-16T00:12:35.977 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 239104 length: 1904128 chunk_info offset: 169472 length 818176 2020-01-16T00:12:35.977 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 44 v 25 (ObjNum 43 snap 0 seq_num 43) dirty exists 2020-01-16T00:12:35.977 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 239104 length: 1904128 chunk_info offset: 1635840 length 423424 2020-01-16T00:12:35.977 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 44 v 25 (ObjNum 43 snap 0 seq_num 43) dirty exists 2020-01-16T00:12:35.977 INFO:tasks.rados.rados.0.smithi123.stdout:302: finishing set_chunk to oid 3 2020-01-16T00:12:35.977 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 868864 length: 950272 chunk_info offset: 312320 length 603136 2020-01-16T00:12:35.978 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 3 v 13 (ObjNum 2 snap 0 seq_num 2) dirty exists 2020-01-16T00:12:35.978 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 868864 length: 950272 chunk_info offset: 1156096 length 722944 2020-01-16T00:12:35.978 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 3 v 13 (ObjNum 2 snap 0 seq_num 2) dirty exists 2020-01-16T00:12:35.978 INFO:tasks.rados.rados.0.smithi123.stdout:286: done (15 left) 2020-01-16T00:12:35.979 INFO:tasks.rados.rados.0.smithi123.stdout:293: done (14 left) 2020-01-16T00:12:35.979 INFO:tasks.rados.rados.0.smithi123.stdout:298: done (13 left) 2020-01-16T00:12:35.979 INFO:tasks.rados.rados.0.smithi123.stdout:302: done (12 left) 2020-01-16T00:12:35.979 INFO:tasks.rados.rados.0.smithi123.stdout:310: set_chunk oid 11 offset: 1440256 length: 1536 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1440256 2020-01-16T00:12:35.979 INFO:tasks.rados.rados.0.smithi123.stdout:311: set_chunk oid 12 offset: 171008 length: 1731584 target oid 12 low_tier tgt_offset: 171008 2020-01-16T00:12:35.979 INFO:tasks.rados.rados.0.smithi123.stdout:312: set_chunk oid 13 offset: 403456 length: 1629184 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 403456 2020-01-16T00:12:35.979 INFO:tasks.rados.rados.0.smithi123.stdout:313: set_chunk oid 14 offset: 1329664 length: 578560 target oid 14 low_tier tgt_offset: 1329664 2020-01-16T00:12:35.979 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.980 INFO:tasks.rados.rados.0.smithi123.stdout:297: finishing set_chunk to oid 48 2020-01-16T00:12:35.980 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 48 v 21 (ObjNum 47 snap 0 seq_num 47) dirty exists 2020-01-16T00:12:35.980 INFO:tasks.rados.rados.0.smithi123.stdout:297: done (15 left) 2020-01-16T00:12:35.980 INFO:tasks.rados.rados.0.smithi123.stdout:314: set_chunk oid 15 offset: 1613312 length: 576000 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1613312 2020-01-16T00:12:35.980 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.980 INFO:tasks.rados.rados.0.smithi123.stdout:301: finishing set_chunk to oid 2 2020-01-16T00:12:35.980 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 373760 length: 1170944 chunk_info offset: 476160 length 1240064 2020-01-16T00:12:35.980 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 2 v 13 (ObjNum 1 snap 0 seq_num 1) dirty exists 2020-01-16T00:12:35.981 INFO:tasks.rados.rados.0.smithi123.stdout:295: finishing set_chunk to oid 46 2020-01-16T00:12:35.981 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 28160 length: 2592768 chunk_info offset: 56832 length 1408000 2020-01-16T00:12:35.981 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 46 v 23 (ObjNum 45 snap 0 seq_num 45) dirty exists 2020-01-16T00:12:35.981 INFO:tasks.rados.rados.0.smithi123.stdout:295: done (15 left) 2020-01-16T00:12:35.981 INFO:tasks.rados.rados.0.smithi123.stdout:301: done (14 left) 2020-01-16T00:12:35.981 INFO:tasks.rados.rados.0.smithi123.stdout:315: set_chunk oid 16 offset: 1227776 length: 248320 target oid 16 low_tier tgt_offset: 1227776 2020-01-16T00:12:35.981 INFO:tasks.rados.rados.0.smithi123.stdout:316: set_chunk oid 17 offset: 648704 length: 386048 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 648704 2020-01-16T00:12:35.981 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.982 INFO:tasks.rados.rados.0.smithi123.stdout:306: finishing set_chunk to oid 7 2020-01-16T00:12:35.982 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 637440 length: 1375232 chunk_info offset: 160768 length 1134080 2020-01-16T00:12:35.982 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 7 v 14 (ObjNum 6 snap 0 seq_num 6) dirty exists 2020-01-16T00:12:35.982 INFO:tasks.rados.rados.0.smithi123.stdout:306: done (15 left) 2020-01-16T00:12:35.982 INFO:tasks.rados.rados.0.smithi123.stdout:317: set_chunk oid 18 offset: 171520 length: 1418752 target oid 18 low_tier tgt_offset: 171520 2020-01-16T00:12:35.982 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.982 INFO:tasks.rados.rados.0.smithi123.stdout:300: finishing set_chunk to oid 1 2020-01-16T00:12:35.982 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1474048 length: 436736 chunk_info offset: 1046016 length 729088 2020-01-16T00:12:35.983 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 1 v 24 (ObjNum 0 snap 0 seq_num 0) dirty exists 2020-01-16T00:12:35.983 INFO:tasks.rados.rados.0.smithi123.stdout:304: finishing set_chunk to oid 5 2020-01-16T00:12:35.983 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 730624 length: 700416 chunk_info offset: 102400 length 870912 2020-01-16T00:12:35.983 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 5 v 21 (ObjNum 4 snap 0 seq_num 4) dirty exists 2020-01-16T00:12:35.983 INFO:tasks.rados.rados.0.smithi123.stdout:300: done (15 left) 2020-01-16T00:12:35.983 INFO:tasks.rados.rados.0.smithi123.stdout:304: done (14 left) 2020-01-16T00:12:35.983 INFO:tasks.rados.rados.0.smithi123.stdout:318: set_chunk oid 19 offset: 246272 length: 427008 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 246272 2020-01-16T00:12:35.983 INFO:tasks.rados.rados.0.smithi123.stdout:319: set_chunk oid 20 offset: 1973248 length: 263680 target oid 20 low_tier tgt_offset: 1973248 2020-01-16T00:12:35.984 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.984 INFO:tasks.rados.rados.0.smithi123.stdout:303: finishing set_chunk to oid 4 2020-01-16T00:12:35.984 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 749568 length: 1125376 chunk_info offset: 803328 length 176128 2020-01-16T00:12:35.984 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 4 v 25 (ObjNum 3 snap 0 seq_num 3) dirty exists 2020-01-16T00:12:35.984 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 749568 length: 1125376 chunk_info offset: 1020416 length 515584 2020-01-16T00:12:35.984 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 4 v 25 (ObjNum 3 snap 0 seq_num 3) dirty exists 2020-01-16T00:12:35.984 INFO:tasks.rados.rados.0.smithi123.stdout:305: finishing set_chunk to oid 6 2020-01-16T00:12:35.985 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1244672 length: 788480 chunk_info offset: 878592 length 645632 2020-01-16T00:12:35.985 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 6 v 10 (ObjNum 5 snap 0 seq_num 5) dirty exists 2020-01-16T00:12:35.985 INFO:tasks.rados.rados.0.smithi123.stdout:307: finishing set_chunk to oid 8 2020-01-16T00:12:35.985 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 812544 length: 1063424 chunk_info offset: 285696 length 965632 2020-01-16T00:12:35.985 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 8 v 10 (ObjNum 7 snap 0 seq_num 7) dirty exists 2020-01-16T00:12:35.985 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 812544 length: 1063424 chunk_info offset: 1589248 length 433664 2020-01-16T00:12:35.985 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 8 v 10 (ObjNum 7 snap 0 seq_num 7) dirty exists 2020-01-16T00:12:35.985 INFO:tasks.rados.rados.0.smithi123.stdout:303: done (15 left) 2020-01-16T00:12:35.986 INFO:tasks.rados.rados.0.smithi123.stdout:305: done (14 left) 2020-01-16T00:12:35.986 INFO:tasks.rados.rados.0.smithi123.stdout:307: done (13 left) 2020-01-16T00:12:35.986 INFO:tasks.rados.rados.0.smithi123.stdout:320: set_chunk oid 21 offset: 1099264 length: 348672 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1099264 2020-01-16T00:12:35.986 INFO:tasks.rados.rados.0.smithi123.stdout:321: set_chunk oid 22 offset: 922112 length: 1132544 target oid 22 low_tier tgt_offset: 922112 2020-01-16T00:12:35.986 INFO:tasks.rados.rados.0.smithi123.stdout:322: set_chunk oid 23 offset: 297984 length: 886784 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 297984 2020-01-16T00:12:35.986 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.986 INFO:tasks.rados.rados.0.smithi123.stdout:299: finishing set_chunk to oid 50 2020-01-16T00:12:35.986 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1515520 length: 1107456 chunk_info offset: 1613312 length 498688 2020-01-16T00:12:35.987 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 50 v 16 (ObjNum 49 snap 0 seq_num 49) dirty exists 2020-01-16T00:12:35.987 INFO:tasks.rados.rados.0.smithi123.stdout:299: done (15 left) 2020-01-16T00:12:35.987 INFO:tasks.rados.rados.0.smithi123.stdout:323: set_chunk oid 24 offset: 2011648 length: 186368 target oid 24 low_tier tgt_offset: 2011648 2020-01-16T00:12:35.987 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.987 INFO:tasks.rados.rados.0.smithi123.stdout:309: finishing set_chunk to oid 10 2020-01-16T00:12:35.987 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 10 v 22 (ObjNum 9 snap 0 seq_num 9) dirty exists 2020-01-16T00:12:35.987 INFO:tasks.rados.rados.0.smithi123.stdout:309: done (15 left) 2020-01-16T00:12:35.987 INFO:tasks.rados.rados.0.smithi123.stdout:324: set_chunk oid 25 offset: 1406976 length: 638976 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1406976 2020-01-16T00:12:35.988 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.988 INFO:tasks.rados.rados.0.smithi123.stdout:313: finishing set_chunk to oid 14 2020-01-16T00:12:35.988 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1329664 length: 578560 chunk_info offset: 470528 length 1206272 2020-01-16T00:12:35.988 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 14 v 26 (ObjNum 13 snap 0 seq_num 13) dirty exists 2020-01-16T00:12:35.988 INFO:tasks.rados.rados.0.smithi123.stdout:313: done (15 left) 2020-01-16T00:12:35.988 INFO:tasks.rados.rados.0.smithi123.stdout:325: set_chunk oid 26 offset: 408064 length: 728576 target oid 26 low_tier tgt_offset: 408064 2020-01-16T00:12:35.988 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.988 INFO:tasks.rados.rados.0.smithi123.stdout:312: finishing set_chunk to oid 13 2020-01-16T00:12:35.989 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 403456 length: 1629184 chunk_info offset: 4096 length 737792 2020-01-16T00:12:35.989 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 13 (ObjNum 12 snap 0 seq_num 12) dirty exists 2020-01-16T00:12:35.989 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 403456 length: 1629184 chunk_info offset: 916992 length 596480 2020-01-16T00:12:35.989 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 13 (ObjNum 12 snap 0 seq_num 12) dirty exists 2020-01-16T00:12:35.989 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 403456 length: 1629184 chunk_info offset: 1541120 length 378368 2020-01-16T00:12:35.989 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 13 (ObjNum 12 snap 0 seq_num 12) dirty exists 2020-01-16T00:12:35.989 INFO:tasks.rados.rados.0.smithi123.stdout:312: done (15 left) 2020-01-16T00:12:35.989 INFO:tasks.rados.rados.0.smithi123.stdout:326: set_chunk oid 27 offset: 373248 length: 650752 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 373248 2020-01-16T00:12:35.990 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.990 INFO:tasks.rados.rados.0.smithi123.stdout:318: finishing set_chunk to oid 19 2020-01-16T00:12:35.990 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 246272 length: 427008 chunk_info offset: 592384 length 1625600 2020-01-16T00:12:35.990 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 19 v 15 (ObjNum 18 snap 0 seq_num 18) dirty exists 2020-01-16T00:12:35.990 INFO:tasks.rados.rados.0.smithi123.stdout:318: done (15 left) 2020-01-16T00:12:35.990 INFO:tasks.rados.rados.0.smithi123.stdout:327: set_chunk oid 28 offset: 222208 length: 1338880 target oid 28 low_tier tgt_offset: 222208 2020-01-16T00:12:35.990 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.990 INFO:tasks.rados.rados.0.smithi123.stdout:308: finishing set_chunk to oid 9 2020-01-16T00:12:35.991 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 9 v 22 (ObjNum 8 snap 0 seq_num 8) dirty exists 2020-01-16T00:12:35.991 INFO:tasks.rados.rados.0.smithi123.stdout:316: finishing set_chunk to oid 17 2020-01-16T00:12:35.991 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 648704 length: 386048 chunk_info offset: 330752 length 378368 2020-01-16T00:12:35.991 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 17 v 26 (ObjNum 16 snap 0 seq_num 16) dirty exists 2020-01-16T00:12:35.991 INFO:tasks.rados.rados.0.smithi123.stdout:308: done (15 left) 2020-01-16T00:12:35.991 INFO:tasks.rados.rados.0.smithi123.stdout:316: done (14 left) 2020-01-16T00:12:35.991 INFO:tasks.rados.rados.0.smithi123.stdout:328: set_chunk oid 29 offset: 564736 length: 820736 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 564736 2020-01-16T00:12:35.991 INFO:tasks.rados.rados.0.smithi123.stdout:329: set_chunk oid 30 offset: 906752 length: 762880 target oid 30 low_tier tgt_offset: 906752 2020-01-16T00:12:35.992 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.992 INFO:tasks.rados.rados.0.smithi123.stdout:317: finishing set_chunk to oid 18 2020-01-16T00:12:35.992 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 171520 length: 1418752 chunk_info offset: 953856 length 471040 2020-01-16T00:12:35.992 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 18 v 31 (ObjNum 17 snap 0 seq_num 17) dirty exists 2020-01-16T00:12:35.992 INFO:tasks.rados.rados.0.smithi123.stdout:317: done (15 left) 2020-01-16T00:12:35.992 INFO:tasks.rados.rados.0.smithi123.stdout:330: set_chunk oid 31 offset: 787968 length: 1338368 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 787968 2020-01-16T00:12:35.992 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.992 INFO:tasks.rados.rados.0.smithi123.stdout:314: finishing set_chunk to oid 15 2020-01-16T00:12:35.993 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1613312 length: 576000 chunk_info offset: 1708032 length 408576 2020-01-16T00:12:35.993 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 15 v 17 (ObjNum 14 snap 0 seq_num 14) dirty exists 2020-01-16T00:12:35.993 INFO:tasks.rados.rados.0.smithi123.stdout:314: done (15 left) 2020-01-16T00:12:35.993 INFO:tasks.rados.rados.0.smithi123.stdout:331: set_chunk oid 32 offset: 2337280 length: 154112 target oid 32 low_tier tgt_offset: 2337280 2020-01-16T00:12:35.993 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.993 INFO:tasks.rados.rados.0.smithi123.stdout:315: finishing set_chunk to oid 16 2020-01-16T00:12:35.994 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 16 v 37 (ObjNum 15 snap 0 seq_num 15) dirty exists 2020-01-16T00:12:35.994 INFO:tasks.rados.rados.0.smithi123.stdout:319: finishing set_chunk to oid 20 2020-01-16T00:12:35.994 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 20 v 38 (ObjNum 19 snap 0 seq_num 19) dirty exists 2020-01-16T00:12:35.994 INFO:tasks.rados.rados.0.smithi123.stdout:315: done (15 left) 2020-01-16T00:12:35.994 INFO:tasks.rados.rados.0.smithi123.stdout:319: done (14 left) 2020-01-16T00:12:35.994 INFO:tasks.rados.rados.0.smithi123.stdout:332: set_chunk oid 33 offset: 1158144 length: 1111552 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1158144 2020-01-16T00:12:35.994 INFO:tasks.rados.rados.0.smithi123.stdout:333: set_chunk oid 34 offset: 1649152 length: 41472 target oid 34 low_tier tgt_offset: 1649152 2020-01-16T00:12:35.995 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.995 INFO:tasks.rados.rados.0.smithi123.stdout:321: finishing set_chunk to oid 22 2020-01-16T00:12:35.995 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 922112 length: 1132544 chunk_info offset: 923648 length 683520 2020-01-16T00:12:35.995 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 22 v 7 (ObjNum 21 snap 0 seq_num 21) dirty exists 2020-01-16T00:12:35.995 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 922112 length: 1132544 chunk_info offset: 1822720 length 105472 2020-01-16T00:12:35.995 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 22 v 7 (ObjNum 21 snap 0 seq_num 21) dirty exists 2020-01-16T00:12:35.995 INFO:tasks.rados.rados.0.smithi123.stdout:322: finishing set_chunk to oid 23 2020-01-16T00:12:35.996 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 297984 length: 886784 chunk_info offset: 808960 length 1406976 2020-01-16T00:12:35.996 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 23 v 15 (ObjNum 22 snap 0 seq_num 22) dirty exists 2020-01-16T00:12:35.996 INFO:tasks.rados.rados.0.smithi123.stdout:323: finishing set_chunk to oid 24 2020-01-16T00:12:35.996 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 24 v 22 (ObjNum 23 snap 0 seq_num 23) dirty exists 2020-01-16T00:12:35.996 INFO:tasks.rados.rados.0.smithi123.stdout:321: done (15 left) 2020-01-16T00:12:35.996 INFO:tasks.rados.rados.0.smithi123.stdout:322: done (14 left) 2020-01-16T00:12:35.996 INFO:tasks.rados.rados.0.smithi123.stdout:323: done (13 left) 2020-01-16T00:12:35.996 INFO:tasks.rados.rados.0.smithi123.stdout:334: set_chunk oid 35 offset: 1848320 length: 246784 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1848320 2020-01-16T00:12:35.997 INFO:tasks.rados.rados.0.smithi123.stdout:335: set_chunk oid 36 offset: 831488 length: 1432064 target oid 36 low_tier tgt_offset: 831488 2020-01-16T00:12:35.997 INFO:tasks.rados.rados.0.smithi123.stdout:336: set_chunk oid 37 offset: 1686528 length: 34816 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1686528 2020-01-16T00:12:35.997 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.997 INFO:tasks.rados.rados.0.smithi123.stdout:310: finishing set_chunk to oid 11 2020-01-16T00:12:35.997 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1440256 length: 1536 chunk_info offset: 279552 length 1427968 2020-01-16T00:12:35.997 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 11 v 14 (ObjNum 10 snap 0 seq_num 10) dirty exists 2020-01-16T00:12:35.997 INFO:tasks.rados.rados.0.smithi123.stdout:328: finishing set_chunk to oid 29 2020-01-16T00:12:35.998 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 564736 length: 820736 chunk_info offset: 1186304 length 657408 2020-01-16T00:12:35.998 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 29 v 11 (ObjNum 28 snap 0 seq_num 28) dirty exists 2020-01-16T00:12:35.999 INFO:tasks.rados.rados.0.smithi123.stdout:310: done (15 left) 2020-01-16T00:12:35.999 INFO:tasks.rados.rados.0.smithi123.stdout:328: done (14 left) 2020-01-16T00:12:35.999 INFO:tasks.rados.rados.0.smithi123.stdout:337: set_chunk oid 38 offset: 928256 length: 830464 target oid 38 low_tier tgt_offset: 928256 2020-01-16T00:12:35.999 INFO:tasks.rados.rados.0.smithi123.stdout:338: set_chunk oid 39 offset: 1216512 length: 996864 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1216512 2020-01-16T00:12:35.999 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:35.999 INFO:tasks.rados.rados.0.smithi123.stdout:311: finishing set_chunk to oid 12 2020-01-16T00:12:35.999 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 171008 length: 1731584 chunk_info offset: 1033216 length 672768 2020-01-16T00:12:36.000 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 12 v 12 (ObjNum 11 snap 0 seq_num 11) dirty exists 2020-01-16T00:12:36.000 INFO:tasks.rados.rados.0.smithi123.stdout:329: finishing set_chunk to oid 30 2020-01-16T00:12:36.000 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 906752 length: 762880 chunk_info offset: 1087488 length 253952 2020-01-16T00:12:36.000 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 30 v 18 (ObjNum 29 snap 0 seq_num 29) dirty exists 2020-01-16T00:12:36.000 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 906752 length: 762880 chunk_info offset: 1542656 length 711168 2020-01-16T00:12:36.000 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 30 v 18 (ObjNum 29 snap 0 seq_num 29) dirty exists 2020-01-16T00:12:36.000 INFO:tasks.rados.rados.0.smithi123.stdout:311: done (15 left) 2020-01-16T00:12:36.000 INFO:tasks.rados.rados.0.smithi123.stdout:329: done (14 left) 2020-01-16T00:12:36.001 INFO:tasks.rados.rados.0.smithi123.stdout:339: set_chunk oid 40 offset: 1244160 length: 1083392 target oid 40 low_tier tgt_offset: 1244160 2020-01-16T00:12:36.001 INFO:tasks.rados.rados.0.smithi123.stdout:340: set_chunk oid 41 offset: 241152 length: 1600512 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 241152 2020-01-16T00:12:36.001 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.001 INFO:tasks.rados.rados.0.smithi123.stdout:324: finishing set_chunk to oid 25 2020-01-16T00:12:36.001 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1406976 length: 638976 chunk_info offset: 773632 length 1419776 2020-01-16T00:12:36.001 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 25 v 24 (ObjNum 24 snap 0 seq_num 24) dirty exists 2020-01-16T00:12:36.001 INFO:tasks.rados.rados.0.smithi123.stdout:320: finishing set_chunk to oid 21 2020-01-16T00:12:36.002 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1099264 length: 348672 chunk_info offset: 346112 length 838656 2020-01-16T00:12:36.002 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 21 v 9 (ObjNum 20 snap 0 seq_num 20) dirty exists 2020-01-16T00:12:36.002 INFO:tasks.rados.rados.0.smithi123.stdout:320: done (15 left) 2020-01-16T00:12:36.002 INFO:tasks.rados.rados.0.smithi123.stdout:324: done (14 left) 2020-01-16T00:12:36.002 INFO:tasks.rados.rados.0.smithi123.stdout:341: set_chunk oid 42 offset: 1043968 length: 624640 target oid 42 low_tier tgt_offset: 1043968 2020-01-16T00:12:36.002 INFO:tasks.rados.rados.0.smithi123.stdout:342: set_chunk oid 43 offset: 82432 length: 268288 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 82432 2020-01-16T00:12:36.002 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.002 INFO:tasks.rados.rados.0.smithi123.stdout:325: finishing set_chunk to oid 26 2020-01-16T00:12:36.003 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 408064 length: 728576 chunk_info offset: 963072 length 376320 2020-01-16T00:12:36.003 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 26 v 26 (ObjNum 25 snap 0 seq_num 25) dirty exists 2020-01-16T00:12:36.003 INFO:tasks.rados.rados.0.smithi123.stdout:325: done (15 left) 2020-01-16T00:12:36.003 INFO:tasks.rados.rados.0.smithi123.stdout:343: set_chunk oid 44 offset: 457216 length: 740864 target oid 44 low_tier tgt_offset: 457216 2020-01-16T00:12:36.003 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.003 INFO:tasks.rados.rados.0.smithi123.stdout:327: finishing set_chunk to oid 28 2020-01-16T00:12:36.003 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 222208 length: 1338880 chunk_info offset: 1498624 length 783872 2020-01-16T00:12:36.004 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 28 v 14 (ObjNum 27 snap 0 seq_num 27) dirty exists 2020-01-16T00:12:36.004 INFO:tasks.rados.rados.0.smithi123.stdout:326: finishing set_chunk to oid 27 2020-01-16T00:12:36.004 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 373248 length: 650752 chunk_info offset: 113152 length 1699840 2020-01-16T00:12:36.004 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 27 v 11 (ObjNum 26 snap 0 seq_num 26) dirty exists 2020-01-16T00:12:36.004 INFO:tasks.rados.rados.0.smithi123.stdout:330: finishing set_chunk to oid 31 2020-01-16T00:12:36.004 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 787968 length: 1338368 chunk_info offset: 263680 length 1668096 2020-01-16T00:12:36.004 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 31 v 15 (ObjNum 30 snap 0 seq_num 30) dirty exists 2020-01-16T00:12:36.004 INFO:tasks.rados.rados.0.smithi123.stdout:331: finishing set_chunk to oid 32 2020-01-16T00:12:36.005 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 2337280 length: 154112 chunk_info offset: 2444288 length 71168 2020-01-16T00:12:36.005 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 32 v 18 (ObjNum 31 snap 0 seq_num 31) dirty exists 2020-01-16T00:12:36.005 INFO:tasks.rados.rados.0.smithi123.stdout:326: done (15 left) 2020-01-16T00:12:36.005 INFO:tasks.rados.rados.0.smithi123.stdout:327: done (14 left) 2020-01-16T00:12:36.005 INFO:tasks.rados.rados.0.smithi123.stdout:330: done (13 left) 2020-01-16T00:12:36.005 INFO:tasks.rados.rados.0.smithi123.stdout:331: done (12 left) 2020-01-16T00:12:36.005 INFO:tasks.rados.rados.0.smithi123.stdout:344: set_chunk oid 45 offset: 232960 length: 1368064 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 232960 2020-01-16T00:12:36.005 INFO:tasks.rados.rados.0.smithi123.stdout:345: set_chunk oid 46 offset: 844288 length: 517632 target oid 46 low_tier tgt_offset: 844288 2020-01-16T00:12:36.006 INFO:tasks.rados.rados.0.smithi123.stdout:346: set_chunk oid 47 offset: 1076736 length: 1150464 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1076736 2020-01-16T00:12:36.006 INFO:tasks.rados.rados.0.smithi123.stdout:347: set_chunk oid 48 offset: 63488 length: 1670656 target oid 48 low_tier tgt_offset: 63488 2020-01-16T00:12:36.006 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.006 INFO:tasks.rados.rados.0.smithi123.stdout:338: finishing set_chunk to oid 39 2020-01-16T00:12:36.006 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1216512 length: 996864 chunk_info offset: 1570304 length 12288 2020-01-16T00:12:36.006 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 39 v 32 (ObjNum 38 snap 0 seq_num 38) dirty exists 2020-01-16T00:12:36.006 INFO:tasks.rados.rados.0.smithi123.stdout:338: done (15 left) 2020-01-16T00:12:36.006 INFO:tasks.rados.rados.0.smithi123.stdout:348: set_chunk oid 49 offset: 1113600 length: 185856 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1113600 2020-01-16T00:12:36.007 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.007 INFO:tasks.rados.rados.0.smithi123.stdout:333: finishing set_chunk to oid 34 2020-01-16T00:12:36.007 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1649152 length: 41472 chunk_info offset: 901632 length 1154560 2020-01-16T00:12:36.007 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 34 v 12 (ObjNum 33 snap 0 seq_num 33) dirty exists 2020-01-16T00:12:36.007 INFO:tasks.rados.rados.0.smithi123.stdout:333: done (15 left) 2020-01-16T00:12:36.007 INFO:tasks.rados.rados.0.smithi123.stdout:349: set_chunk oid 50 offset: 814592 length: 1879552 target oid 50 low_tier tgt_offset: 814592 2020-01-16T00:12:36.007 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.007 INFO:tasks.rados.rados.0.smithi123.stdout:340: finishing set_chunk to oid 41 2020-01-16T00:12:36.008 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 241152 length: 1600512 chunk_info offset: 925184 length 293376 2020-01-16T00:12:36.008 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 41 v 28 (ObjNum 40 snap 0 seq_num 40) dirty exists 2020-01-16T00:12:36.008 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 241152 length: 1600512 chunk_info offset: 1808384 length 557056 2020-01-16T00:12:36.008 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 41 v 28 (ObjNum 40 snap 0 seq_num 40) dirty exists 2020-01-16T00:12:36.008 INFO:tasks.rados.rados.0.smithi123.stdout:340: done (15 left) 2020-01-16T00:12:36.008 INFO:tasks.rados.rados.0.smithi123.stdout:350: set_chunk oid 1 offset: 44544 length: 740352 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 44544 2020-01-16T00:12:36.008 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.009 INFO:tasks.rados.rados.0.smithi123.stdout:337: finishing set_chunk to oid 38 2020-01-16T00:12:36.009 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 38 v 22 (ObjNum 37 snap 0 seq_num 37) dirty exists 2020-01-16T00:12:36.009 INFO:tasks.rados.rados.0.smithi123.stdout:337: done (15 left) 2020-01-16T00:12:36.009 INFO:tasks.rados.rados.0.smithi123.stdout:351: set_chunk oid 2 offset: 418816 length: 928768 target oid 2 low_tier tgt_offset: 418816 2020-01-16T00:12:36.009 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.009 INFO:tasks.rados.rados.0.smithi123.stdout:334: finishing set_chunk to oid 35 2020-01-16T00:12:36.009 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1848320 length: 246784 chunk_info offset: 1823744 length 153600 2020-01-16T00:12:36.009 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 35 v 28 (ObjNum 34 snap 0 seq_num 34) dirty exists 2020-01-16T00:12:36.010 INFO:tasks.rados.rados.0.smithi123.stdout:341: finishing set_chunk to oid 42 2020-01-16T00:12:36.010 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1043968 length: 624640 chunk_info offset: 1364992 length 1145344 2020-01-16T00:12:36.010 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 42 v 11 (ObjNum 41 snap 0 seq_num 41) dirty exists 2020-01-16T00:12:36.010 INFO:tasks.rados.rados.0.smithi123.stdout:334: done (15 left) 2020-01-16T00:12:36.010 INFO:tasks.rados.rados.0.smithi123.stdout:341: done (14 left) 2020-01-16T00:12:36.010 INFO:tasks.rados.rados.0.smithi123.stdout:352: set_chunk oid 3 offset: 817664 length: 1156608 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 817664 2020-01-16T00:12:36.011 INFO:tasks.rados.rados.0.smithi123.stdout:353: set_chunk oid 4 offset: 757760 length: 290816 target oid 4 low_tier tgt_offset: 757760 2020-01-16T00:12:36.011 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.011 INFO:tasks.rados.rados.0.smithi123.stdout:332: finishing set_chunk to oid 33 2020-01-16T00:12:36.011 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1158144 length: 1111552 chunk_info offset: 658944 length 1355776 2020-01-16T00:12:36.011 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 33 v 21 (ObjNum 32 snap 0 seq_num 32) dirty exists 2020-01-16T00:12:36.011 INFO:tasks.rados.rados.0.smithi123.stdout:332: done (15 left) 2020-01-16T00:12:36.011 INFO:tasks.rados.rados.0.smithi123.stdout:354: set_chunk oid 5 offset: 131584 length: 1413632 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 131584 2020-01-16T00:12:36.012 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.012 INFO:tasks.rados.rados.0.smithi123.stdout:344: finishing set_chunk to oid 45 2020-01-16T00:12:36.012 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 232960 length: 1368064 chunk_info offset: 640512 length 403968 2020-01-16T00:12:36.012 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 45 v 11 (ObjNum 44 snap 0 seq_num 44) dirty exists 2020-01-16T00:12:36.012 INFO:tasks.rados.rados.0.smithi123.stdout:335: finishing set_chunk to oid 36 2020-01-16T00:12:36.013 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 831488 length: 1432064 chunk_info offset: 643584 length 327680 2020-01-16T00:12:36.013 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 36 v 8 (ObjNum 35 snap 0 seq_num 35) dirty exists 2020-01-16T00:12:36.013 INFO:tasks.rados.rados.0.smithi123.stdout:335: done (15 left) 2020-01-16T00:12:36.013 INFO:tasks.rados.rados.0.smithi123.stdout:344: done (14 left) 2020-01-16T00:12:36.013 INFO:tasks.rados.rados.0.smithi123.stdout:355: set_chunk oid 6 offset: 177664 length: 1021440 target oid 6 low_tier tgt_offset: 177664 2020-01-16T00:12:36.013 INFO:tasks.rados.rados.0.smithi123.stdout:347: finishing set_chunk to oid 48 2020-01-16T00:12:36.013 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 63488 length: 1670656 chunk_info offset: 264704 length 1399296 2020-01-16T00:12:36.013 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 48 v 21 (ObjNum 47 snap 0 seq_num 47) dirty exists 2020-01-16T00:12:36.014 INFO:tasks.rados.rados.0.smithi123.stdout:347: done (14 left) 2020-01-16T00:12:36.014 INFO:tasks.rados.rados.0.smithi123.stdout:356: set_chunk oid 7 offset: 516608 length: 26112 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 516608 2020-01-16T00:12:36.014 INFO:tasks.rados.rados.0.smithi123.stdout:357: set_chunk oid 8 offset: 371712 length: 1710080 target oid 8 low_tier tgt_offset: 371712 2020-01-16T00:12:36.014 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.014 INFO:tasks.rados.rados.0.smithi123.stdout:339: finishing set_chunk to oid 40 2020-01-16T00:12:36.014 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1244160 length: 1083392 chunk_info offset: 790528 length 688128 2020-01-16T00:12:36.015 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 40 v 23 (ObjNum 39 snap 0 seq_num 39) dirty exists 2020-01-16T00:12:36.015 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1244160 length: 1083392 chunk_info offset: 1926656 length 67072 2020-01-16T00:12:36.016 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 40 v 23 (ObjNum 39 snap 0 seq_num 39) dirty exists 2020-01-16T00:12:36.016 INFO:tasks.rados.rados.0.smithi123.stdout:339: done (15 left) 2020-01-16T00:12:36.016 INFO:tasks.rados.rados.0.smithi123.stdout:358: set_chunk oid 9 offset: 696832 length: 406528 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 696832 2020-01-16T00:12:36.016 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.016 INFO:tasks.rados.rados.0.smithi123.stdout:348: finishing set_chunk to oid 49 2020-01-16T00:12:36.016 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1113600 length: 185856 chunk_info offset: 453632 length 1075200 2020-01-16T00:12:36.017 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 49 v 18 (ObjNum 48 snap 0 seq_num 48) dirty exists 2020-01-16T00:12:36.017 INFO:tasks.rados.rados.0.smithi123.stdout:348: done (15 left) 2020-01-16T00:12:36.017 INFO:tasks.rados.rados.0.smithi123.stdout:359: set_chunk oid 10 offset: 271360 length: 188928 target oid 10 low_tier tgt_offset: 271360 2020-01-16T00:12:36.017 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.017 INFO:tasks.rados.rados.0.smithi123.stdout:342: finishing set_chunk to oid 43 2020-01-16T00:12:36.018 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 43 v 39 (ObjNum 42 snap 0 seq_num 42) dirty exists 2020-01-16T00:12:36.018 INFO:tasks.rados.rados.0.smithi123.stdout:342: done (15 left) 2020-01-16T00:12:36.018 INFO:tasks.rados.rados.0.smithi123.stdout:360: set_chunk oid 11 offset: 204800 length: 1570816 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 204800 2020-01-16T00:12:36.018 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.018 INFO:tasks.rados.rados.0.smithi123.stdout:336: finishing set_chunk to oid 37 2020-01-16T00:12:36.018 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 37 v 46 (ObjNum 36 snap 0 seq_num 36) dirty exists 2020-01-16T00:12:36.019 INFO:tasks.rados.rados.0.smithi123.stdout:345: finishing set_chunk to oid 46 2020-01-16T00:12:36.019 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 844288 length: 517632 chunk_info offset: 56832 length 1408000 2020-01-16T00:12:36.019 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 46 v 23 (ObjNum 45 snap 0 seq_num 45) dirty exists 2020-01-16T00:12:36.019 INFO:tasks.rados.rados.0.smithi123.stdout:346: finishing set_chunk to oid 47 2020-01-16T00:12:36.019 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1076736 length: 1150464 chunk_info offset: 123904 length 1034752 2020-01-16T00:12:36.020 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 47 v 30 (ObjNum 46 snap 0 seq_num 46) dirty exists 2020-01-16T00:12:36.020 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1076736 length: 1150464 chunk_info offset: 2092032 length 38400 2020-01-16T00:12:36.020 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 47 v 30 (ObjNum 46 snap 0 seq_num 46) dirty exists 2020-01-16T00:12:36.020 INFO:tasks.rados.rados.0.smithi123.stdout:336: done (15 left) 2020-01-16T00:12:36.020 INFO:tasks.rados.rados.0.smithi123.stdout:345: done (14 left) 2020-01-16T00:12:36.020 INFO:tasks.rados.rados.0.smithi123.stdout:346: done (13 left) 2020-01-16T00:12:36.021 INFO:tasks.rados.rados.0.smithi123.stdout:361: set_chunk oid 12 offset: 676352 length: 111104 target oid 12 low_tier tgt_offset: 676352 2020-01-16T00:12:36.021 INFO:tasks.rados.rados.0.smithi123.stdout:362: set_chunk oid 13 offset: 391680 length: 1129984 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 391680 2020-01-16T00:12:36.021 INFO:tasks.rados.rados.0.smithi123.stdout:363: set_chunk oid 14 offset: 279040 length: 752128 target oid 14 low_tier tgt_offset: 279040 2020-01-16T00:12:36.021 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.021 INFO:tasks.rados.rados.0.smithi123.stdout:349: finishing set_chunk to oid 50 2020-01-16T00:12:36.021 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 814592 length: 1879552 chunk_info offset: 1613312 length 498688 2020-01-16T00:12:36.022 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 50 v 16 (ObjNum 49 snap 0 seq_num 49) dirty exists 2020-01-16T00:12:36.022 INFO:tasks.rados.rados.0.smithi123.stdout:343: finishing set_chunk to oid 44 2020-01-16T00:12:36.022 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 457216 length: 740864 chunk_info offset: 169472 length 818176 2020-01-16T00:12:36.022 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 44 v 25 (ObjNum 43 snap 0 seq_num 43) dirty exists 2020-01-16T00:12:36.022 INFO:tasks.rados.rados.0.smithi123.stdout:350: finishing set_chunk to oid 1 2020-01-16T00:12:36.023 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 1 v 48 (ObjNum 0 snap 0 seq_num 0) dirty exists 2020-01-16T00:12:36.023 INFO:tasks.rados.rados.0.smithi123.stdout:343: done (15 left) 2020-01-16T00:12:36.023 INFO:tasks.rados.rados.0.smithi123.stdout:349: done (14 left) 2020-01-16T00:12:36.023 INFO:tasks.rados.rados.0.smithi123.stdout:350: done (13 left) 2020-01-16T00:12:36.023 INFO:tasks.rados.rados.0.smithi123.stdout:364: set_chunk oid 15 offset: 752640 length: 650240 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 752640 2020-01-16T00:12:36.023 INFO:tasks.rados.rados.0.smithi123.stdout:365: set_chunk oid 16 offset: 228352 length: 126976 target oid 16 low_tier tgt_offset: 228352 2020-01-16T00:12:36.024 INFO:tasks.rados.rados.0.smithi123.stdout:366: set_chunk oid 17 offset: 361472 length: 1112576 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 361472 2020-01-16T00:12:36.024 INFO:tasks.rados.rados.0.smithi123.stdout:351: finishing set_chunk to oid 2 2020-01-16T00:12:36.024 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 418816 length: 928768 chunk_info offset: 476160 length 1240064 2020-01-16T00:12:36.024 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 2 v 13 (ObjNum 1 snap 0 seq_num 1) dirty exists 2020-01-16T00:12:36.024 INFO:tasks.rados.rados.0.smithi123.stdout:351: done (15 left) 2020-01-16T00:12:36.024 INFO:tasks.rados.rados.0.smithi123.stdout:367: set_chunk oid 18 offset: 244736 length: 1378304 target oid 18 low_tier tgt_offset: 244736 2020-01-16T00:12:36.025 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.025 INFO:tasks.rados.rados.0.smithi123.stdout:355: finishing set_chunk to oid 6 2020-01-16T00:12:36.025 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 177664 length: 1021440 chunk_info offset: 878592 length 645632 2020-01-16T00:12:36.025 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 6 v 10 (ObjNum 5 snap 0 seq_num 5) dirty exists 2020-01-16T00:12:36.025 INFO:tasks.rados.rados.0.smithi123.stdout:355: done (15 left) 2020-01-16T00:12:36.026 INFO:tasks.rados.rados.0.smithi123.stdout:368: set_chunk oid 19 offset: 1207296 length: 433152 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1207296 2020-01-16T00:12:36.026 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.026 INFO:tasks.rados.rados.0.smithi123.stdout:352: finishing set_chunk to oid 3 2020-01-16T00:12:36.026 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 817664 length: 1156608 chunk_info offset: 312320 length 603136 2020-01-16T00:12:36.026 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 3 v 13 (ObjNum 2 snap 0 seq_num 2) dirty exists 2020-01-16T00:12:36.026 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 817664 length: 1156608 chunk_info offset: 1156096 length 722944 2020-01-16T00:12:36.027 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 3 v 13 (ObjNum 2 snap 0 seq_num 2) dirty exists 2020-01-16T00:12:36.027 INFO:tasks.rados.rados.0.smithi123.stdout:352: done (15 left) 2020-01-16T00:12:36.027 INFO:tasks.rados.rados.0.smithi123.stdout:369: set_chunk oid 20 offset: 391680 length: 1416192 target oid 20 low_tier tgt_offset: 391680 2020-01-16T00:12:36.027 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.027 INFO:tasks.rados.rados.0.smithi123.stdout:358: finishing set_chunk to oid 9 2020-01-16T00:12:36.027 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 696832 length: 406528 chunk_info offset: 781312 length 2048 2020-01-16T00:12:36.028 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 9 v 22 (ObjNum 8 snap 0 seq_num 8) dirty exists 2020-01-16T00:12:36.028 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 696832 length: 406528 chunk_info offset: 1072128 length 499200 2020-01-16T00:12:36.028 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 9 v 22 (ObjNum 8 snap 0 seq_num 8) dirty exists 2020-01-16T00:12:36.028 INFO:tasks.rados.rados.0.smithi123.stdout:357: finishing set_chunk to oid 8 2020-01-16T00:12:36.028 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 371712 length: 1710080 chunk_info offset: 285696 length 965632 2020-01-16T00:12:36.028 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 8 v 10 (ObjNum 7 snap 0 seq_num 7) dirty exists 2020-01-16T00:12:36.029 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 371712 length: 1710080 chunk_info offset: 1589248 length 433664 2020-01-16T00:12:36.029 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 8 v 10 (ObjNum 7 snap 0 seq_num 7) dirty exists 2020-01-16T00:12:36.029 INFO:tasks.rados.rados.0.smithi123.stdout:357: done (15 left) 2020-01-16T00:12:36.029 INFO:tasks.rados.rados.0.smithi123.stdout:358: done (14 left) 2020-01-16T00:12:36.029 INFO:tasks.rados.rados.0.smithi123.stdout:370: set_chunk oid 21 offset: 1643008 length: 45056 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1643008 2020-01-16T00:12:36.030 INFO:tasks.rados.rados.0.smithi123.stdout:371: set_chunk oid 22 offset: 333312 length: 1050624 target oid 22 low_tier tgt_offset: 333312 2020-01-16T00:12:36.030 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.030 INFO:tasks.rados.rados.0.smithi123.stdout:353: finishing set_chunk to oid 4 2020-01-16T00:12:36.030 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 757760 length: 290816 chunk_info offset: 803328 length 176128 2020-01-16T00:12:36.030 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 4 v 25 (ObjNum 3 snap 0 seq_num 3) dirty exists 2020-01-16T00:12:36.030 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 757760 length: 290816 chunk_info offset: 1020416 length 515584 2020-01-16T00:12:36.030 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 4 v 25 (ObjNum 3 snap 0 seq_num 3) dirty exists 2020-01-16T00:12:36.031 INFO:tasks.rados.rados.0.smithi123.stdout:353: done (15 left) 2020-01-16T00:12:36.031 INFO:tasks.rados.rados.0.smithi123.stdout:372: set_chunk oid 23 offset: 294400 length: 114176 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 294400 2020-01-16T00:12:36.031 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.031 INFO:tasks.rados.rados.0.smithi123.stdout:360: finishing set_chunk to oid 11 2020-01-16T00:12:36.031 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 204800 length: 1570816 chunk_info offset: 279552 length 1427968 2020-01-16T00:12:36.031 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 11 v 14 (ObjNum 10 snap 0 seq_num 10) dirty exists 2020-01-16T00:12:36.031 INFO:tasks.rados.rados.0.smithi123.stdout:360: done (15 left) 2020-01-16T00:12:36.031 INFO:tasks.rados.rados.0.smithi123.stdout:373: set_chunk oid 24 offset: 612864 length: 347136 target oid 24 low_tier tgt_offset: 612864 2020-01-16T00:12:36.032 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.032 INFO:tasks.rados.rados.0.smithi123.stdout:361: finishing set_chunk to oid 12 2020-01-16T00:12:36.032 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 12 v 27 (ObjNum 11 snap 0 seq_num 11) dirty exists 2020-01-16T00:12:36.032 INFO:tasks.rados.rados.0.smithi123.stdout:361: done (15 left) 2020-01-16T00:12:36.032 INFO:tasks.rados.rados.0.smithi123.stdout:374: set_chunk oid 25 offset: 1735168 length: 337408 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1735168 2020-01-16T00:12:36.032 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.032 INFO:tasks.rados.rados.0.smithi123.stdout:373: finishing set_chunk to oid 24 2020-01-16T00:12:36.032 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 612864 length: 347136 chunk_info offset: 30720 length 1585664 2020-01-16T00:12:36.033 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 24 v 22 (ObjNum 23 snap 0 seq_num 23) dirty exists 2020-01-16T00:12:36.033 INFO:tasks.rados.rados.0.smithi123.stdout:373: done (15 left) 2020-01-16T00:12:36.033 INFO:tasks.rados.rados.0.smithi123.stdout:375: set_chunk oid 26 offset: 891904 length: 1305600 target oid 26 low_tier tgt_offset: 891904 2020-01-16T00:12:36.033 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.033 INFO:tasks.rados.rados.0.smithi123.stdout:356: finishing set_chunk to oid 7 2020-01-16T00:12:36.033 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 516608 length: 26112 chunk_info offset: 160768 length 1134080 2020-01-16T00:12:36.033 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 7 v 14 (ObjNum 6 snap 0 seq_num 6) dirty exists 2020-01-16T00:12:36.034 INFO:tasks.rados.rados.0.smithi123.stdout:356: done (15 left) 2020-01-16T00:12:36.034 INFO:tasks.rados.rados.0.smithi123.stdout:376: set_chunk oid 27 offset: 256512 length: 1965568 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 256512 2020-01-16T00:12:36.034 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.034 INFO:tasks.rados.rados.0.smithi123.stdout:359: finishing set_chunk to oid 10 2020-01-16T00:12:36.034 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 271360 length: 188928 chunk_info offset: 453120 length 360448 2020-01-16T00:12:36.034 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 10 v 22 (ObjNum 9 snap 0 seq_num 9) dirty exists 2020-01-16T00:12:36.034 INFO:tasks.rados.rados.0.smithi123.stdout:359: done (15 left) 2020-01-16T00:12:36.035 INFO:tasks.rados.rados.0.smithi123.stdout:377: set_chunk oid 28 offset: 1055232 length: 1124352 target oid 28 low_tier tgt_offset: 1055232 2020-01-16T00:12:36.035 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.035 INFO:tasks.rados.rados.0.smithi123.stdout:362: finishing set_chunk to oid 13 2020-01-16T00:12:36.035 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 391680 length: 1129984 chunk_info offset: 4096 length 737792 2020-01-16T00:12:36.035 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 13 (ObjNum 12 snap 0 seq_num 12) dirty exists 2020-01-16T00:12:36.035 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 391680 length: 1129984 chunk_info offset: 916992 length 596480 2020-01-16T00:12:36.035 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 13 (ObjNum 12 snap 0 seq_num 12) dirty exists 2020-01-16T00:12:36.036 INFO:tasks.rados.rados.0.smithi123.stdout:362: done (15 left) 2020-01-16T00:12:36.036 INFO:tasks.rados.rados.0.smithi123.stdout:378: set_chunk oid 29 offset: 1719808 length: 571904 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1719808 2020-01-16T00:12:36.036 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.036 INFO:tasks.rados.rados.0.smithi123.stdout:367: finishing set_chunk to oid 18 2020-01-16T00:12:36.036 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 244736 length: 1378304 chunk_info offset: 953856 length 471040 2020-01-16T00:12:36.036 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 18 v 31 (ObjNum 17 snap 0 seq_num 17) dirty exists 2020-01-16T00:12:36.036 INFO:tasks.rados.rados.0.smithi123.stdout:367: done (15 left) 2020-01-16T00:12:36.036 INFO:tasks.rados.rados.0.smithi123.stdout:379: set_chunk oid 30 offset: 758784 length: 456704 target oid 30 low_tier tgt_offset: 758784 2020-01-16T00:12:36.037 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.037 INFO:tasks.rados.rados.0.smithi123.stdout:366: finishing set_chunk to oid 17 2020-01-16T00:12:36.037 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 361472 length: 1112576 chunk_info offset: 330752 length 378368 2020-01-16T00:12:36.037 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 17 v 26 (ObjNum 16 snap 0 seq_num 16) dirty exists 2020-01-16T00:12:36.037 INFO:tasks.rados.rados.0.smithi123.stdout:368: finishing set_chunk to oid 19 2020-01-16T00:12:36.037 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1207296 length: 433152 chunk_info offset: 592384 length 1625600 2020-01-16T00:12:36.037 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 19 v 15 (ObjNum 18 snap 0 seq_num 18) dirty exists 2020-01-16T00:12:36.038 INFO:tasks.rados.rados.0.smithi123.stdout:363: finishing set_chunk to oid 14 2020-01-16T00:12:36.038 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 279040 length: 752128 chunk_info offset: 55296 length 299008 2020-01-16T00:12:36.038 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 14 v 26 (ObjNum 13 snap 0 seq_num 13) dirty exists 2020-01-16T00:12:36.038 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 279040 length: 752128 chunk_info offset: 470528 length 1206272 2020-01-16T00:12:36.038 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 14 v 26 (ObjNum 13 snap 0 seq_num 13) dirty exists 2020-01-16T00:12:36.038 INFO:tasks.rados.rados.0.smithi123.stdout:375: finishing set_chunk to oid 26 2020-01-16T00:12:36.038 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 891904 length: 1305600 chunk_info offset: 963072 length 376320 2020-01-16T00:12:36.038 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 26 v 26 (ObjNum 25 snap 0 seq_num 25) dirty exists 2020-01-16T00:12:36.039 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 891904 length: 1305600 chunk_info offset: 1494016 length 315392 2020-01-16T00:12:36.039 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 26 v 26 (ObjNum 25 snap 0 seq_num 25) dirty exists 2020-01-16T00:12:36.039 INFO:tasks.rados.rados.0.smithi123.stdout:363: done (15 left) 2020-01-16T00:12:36.039 INFO:tasks.rados.rados.0.smithi123.stdout:366: done (14 left) 2020-01-16T00:12:36.039 INFO:tasks.rados.rados.0.smithi123.stdout:368: done (13 left) 2020-01-16T00:12:36.039 INFO:tasks.rados.rados.0.smithi123.stdout:375: done (12 left) 2020-01-16T00:12:36.039 INFO:tasks.rados.rados.0.smithi123.stdout:380: set_chunk oid 31 offset: 1219072 length: 462848 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1219072 2020-01-16T00:12:36.039 INFO:tasks.rados.rados.0.smithi123.stdout:381: set_chunk oid 32 offset: 48128 length: 1748992 target oid 32 low_tier tgt_offset: 48128 2020-01-16T00:12:36.040 INFO:tasks.rados.rados.0.smithi123.stdout:382: set_chunk oid 33 offset: 291840 length: 1646080 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 291840 2020-01-16T00:12:36.040 INFO:tasks.rados.rados.0.smithi123.stdout:383: set_chunk oid 34 offset: 172032 length: 1383936 target oid 34 low_tier tgt_offset: 172032 2020-01-16T00:12:36.040 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.040 INFO:tasks.rados.rados.0.smithi123.stdout:372: finishing set_chunk to oid 23 2020-01-16T00:12:36.040 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 23 v 35 (ObjNum 22 snap 0 seq_num 22) dirty exists 2020-01-16T00:12:36.040 INFO:tasks.rados.rados.0.smithi123.stdout:354: finishing set_chunk to oid 5 2020-01-16T00:12:36.041 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 131584 length: 1413632 chunk_info offset: 102400 length 870912 2020-01-16T00:12:36.041 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 5 v 21 (ObjNum 4 snap 0 seq_num 4) dirty exists 2020-01-16T00:12:36.041 INFO:tasks.rados.rados.0.smithi123.stdout:354: done (15 left) 2020-01-16T00:12:36.041 INFO:tasks.rados.rados.0.smithi123.stdout:372: done (14 left) 2020-01-16T00:12:36.041 INFO:tasks.rados.rados.0.smithi123.stdout:384: set_chunk oid 35 offset: 1263616 length: 899584 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1263616 2020-01-16T00:12:36.041 INFO:tasks.rados.rados.0.smithi123.stdout:385: set_chunk oid 36 offset: 399360 length: 603648 target oid 36 low_tier tgt_offset: 399360 2020-01-16T00:12:36.041 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.042 INFO:tasks.rados.rados.0.smithi123.stdout:370: finishing set_chunk to oid 21 2020-01-16T00:12:36.042 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 21 v 17 (ObjNum 20 snap 0 seq_num 20) dirty exists 2020-01-16T00:12:36.042 INFO:tasks.rados.rados.0.smithi123.stdout:371: finishing set_chunk to oid 22 2020-01-16T00:12:36.042 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 333312 length: 1050624 chunk_info offset: 923648 length 683520 2020-01-16T00:12:36.042 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 22 v 7 (ObjNum 21 snap 0 seq_num 21) dirty exists 2020-01-16T00:12:36.042 INFO:tasks.rados.rados.0.smithi123.stdout:365: finishing set_chunk to oid 16 2020-01-16T00:12:36.042 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 16 v 42 (ObjNum 15 snap 0 seq_num 15) dirty exists 2020-01-16T00:12:36.043 INFO:tasks.rados.rados.0.smithi123.stdout:369: finishing set_chunk to oid 20 2020-01-16T00:12:36.043 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 391680 length: 1416192 chunk_info offset: 385536 length 861184 2020-01-16T00:12:36.044 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 20 v 38 (ObjNum 19 snap 0 seq_num 19) dirty exists 2020-01-16T00:12:36.044 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 391680 length: 1416192 chunk_info offset: 1278464 length 442368 2020-01-16T00:12:36.044 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 20 v 38 (ObjNum 19 snap 0 seq_num 19) dirty exists 2020-01-16T00:12:36.044 INFO:tasks.rados.rados.0.smithi123.stdout:364: finishing set_chunk to oid 15 2020-01-16T00:12:36.044 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 15 v 33 (ObjNum 14 snap 0 seq_num 14) dirty exists 2020-01-16T00:12:36.044 INFO:tasks.rados.rados.0.smithi123.stdout:364: done (15 left) 2020-01-16T00:12:36.045 INFO:tasks.rados.rados.0.smithi123.stdout:365: done (14 left) 2020-01-16T00:12:36.045 INFO:tasks.rados.rados.0.smithi123.stdout:369: done (13 left) 2020-01-16T00:12:36.045 INFO:tasks.rados.rados.0.smithi123.stdout:370: done (12 left) 2020-01-16T00:12:36.045 INFO:tasks.rados.rados.0.smithi123.stdout:371: done (11 left) 2020-01-16T00:12:36.045 INFO:tasks.rados.rados.0.smithi123.stdout:386: set_chunk oid 37 offset: 412672 length: 12800 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 412672 2020-01-16T00:12:36.045 INFO:tasks.rados.rados.0.smithi123.stdout:387: set_chunk oid 38 offset: 949760 length: 931840 target oid 38 low_tier tgt_offset: 949760 2020-01-16T00:12:36.046 INFO:tasks.rados.rados.0.smithi123.stdout:388: set_chunk oid 39 offset: 500224 length: 171008 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 500224 2020-01-16T00:12:36.046 INFO:tasks.rados.rados.0.smithi123.stdout:389: set_chunk oid 40 offset: 710144 length: 1118720 target oid 40 low_tier tgt_offset: 710144 2020-01-16T00:12:36.046 INFO:tasks.rados.rados.0.smithi123.stdout:390: set_chunk oid 41 offset: 1332736 length: 44544 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1332736 2020-01-16T00:12:36.046 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.046 INFO:tasks.rados.rados.0.smithi123.stdout:374: finishing set_chunk to oid 25 2020-01-16T00:12:36.046 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1735168 length: 337408 chunk_info offset: 773632 length 1419776 2020-01-16T00:12:36.047 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 25 v 24 (ObjNum 24 snap 0 seq_num 24) dirty exists 2020-01-16T00:12:36.047 INFO:tasks.rados.rados.0.smithi123.stdout:374: done (15 left) 2020-01-16T00:12:36.047 INFO:tasks.rados.rados.0.smithi123.stdout:391: set_chunk oid 42 offset: 1257984 length: 517632 target oid 42 low_tier tgt_offset: 1257984 2020-01-16T00:12:36.047 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.048 INFO:tasks.rados.rados.0.smithi123.stdout:376: finishing set_chunk to oid 27 2020-01-16T00:12:36.048 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 256512 length: 1965568 chunk_info offset: 113152 length 1699840 2020-01-16T00:12:36.048 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 27 v 11 (ObjNum 26 snap 0 seq_num 26) dirty exists 2020-01-16T00:12:36.048 INFO:tasks.rados.rados.0.smithi123.stdout:376: done (15 left) 2020-01-16T00:12:36.048 INFO:tasks.rados.rados.0.smithi123.stdout:392: set_chunk oid 43 offset: 930304 length: 524288 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 930304 2020-01-16T00:12:36.048 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.049 INFO:tasks.rados.rados.0.smithi123.stdout:378: finishing set_chunk to oid 29 2020-01-16T00:12:36.049 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1719808 length: 571904 chunk_info offset: 1186304 length 657408 2020-01-16T00:12:36.049 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 29 v 11 (ObjNum 28 snap 0 seq_num 28) dirty exists 2020-01-16T00:12:36.049 INFO:tasks.rados.rados.0.smithi123.stdout:379: finishing set_chunk to oid 30 2020-01-16T00:12:36.049 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 758784 length: 456704 chunk_info offset: 1087488 length 253952 2020-01-16T00:12:36.049 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 30 v 18 (ObjNum 29 snap 0 seq_num 29) dirty exists 2020-01-16T00:12:36.049 INFO:tasks.rados.rados.0.smithi123.stdout:382: finishing set_chunk to oid 33 2020-01-16T00:12:36.049 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 291840 length: 1646080 chunk_info offset: 658944 length 1355776 2020-01-16T00:12:36.050 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 33 v 21 (ObjNum 32 snap 0 seq_num 32) dirty exists 2020-01-16T00:12:36.050 INFO:tasks.rados.rados.0.smithi123.stdout:378: done (15 left) 2020-01-16T00:12:36.050 INFO:tasks.rados.rados.0.smithi123.stdout:379: done (14 left) 2020-01-16T00:12:36.050 INFO:tasks.rados.rados.0.smithi123.stdout:382: done (13 left) 2020-01-16T00:12:36.050 INFO:tasks.rados.rados.0.smithi123.stdout:393: set_chunk oid 44 offset: 239104 length: 2037248 target oid 44 low_tier tgt_offset: 239104 2020-01-16T00:12:36.050 INFO:tasks.rados.rados.0.smithi123.stdout:394: set_chunk oid 45 offset: 1445888 length: 278528 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1445888 2020-01-16T00:12:36.051 INFO:tasks.rados.rados.0.smithi123.stdout:395: set_chunk oid 46 offset: 48128 length: 2481664 target oid 46 low_tier tgt_offset: 48128 2020-01-16T00:12:36.051 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.051 INFO:tasks.rados.rados.0.smithi123.stdout:377: finishing set_chunk to oid 28 2020-01-16T00:12:36.051 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1055232 length: 1124352 chunk_info offset: 1498624 length 783872 2020-01-16T00:12:36.051 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 28 v 14 (ObjNum 27 snap 0 seq_num 27) dirty exists 2020-01-16T00:12:36.051 INFO:tasks.rados.rados.0.smithi123.stdout:380: finishing set_chunk to oid 31 2020-01-16T00:12:36.051 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1219072 length: 462848 chunk_info offset: 263680 length 1668096 2020-01-16T00:12:36.052 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 31 v 15 (ObjNum 30 snap 0 seq_num 30) dirty exists 2020-01-16T00:12:36.052 INFO:tasks.rados.rados.0.smithi123.stdout:381: finishing set_chunk to oid 32 2020-01-16T00:12:36.052 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 48128 length: 1748992 chunk_info offset: 605696 length 877568 2020-01-16T00:12:36.052 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 32 v 18 (ObjNum 31 snap 0 seq_num 31) dirty exists 2020-01-16T00:12:36.052 INFO:tasks.rados.rados.0.smithi123.stdout:377: done (15 left) 2020-01-16T00:12:36.052 INFO:tasks.rados.rados.0.smithi123.stdout:380: done (14 left) 2020-01-16T00:12:36.052 INFO:tasks.rados.rados.0.smithi123.stdout:381: done (13 left) 2020-01-16T00:12:36.052 INFO:tasks.rados.rados.0.smithi123.stdout:396: set_chunk oid 47 offset: 83968 length: 24064 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 83968 2020-01-16T00:12:36.053 INFO:tasks.rados.rados.0.smithi123.stdout:397: set_chunk oid 48 offset: 1616896 length: 1153024 target oid 48 low_tier tgt_offset: 1616896 2020-01-16T00:12:36.053 INFO:tasks.rados.rados.0.smithi123.stdout:398: set_chunk oid 49 offset: 70656 length: 122880 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 70656 2020-01-16T00:12:36.053 INFO:tasks.rados.rados.0.smithi123.stdout:384: finishing set_chunk to oid 35 2020-01-16T00:12:36.053 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1263616 length: 899584 chunk_info offset: 926720 length 423424 2020-01-16T00:12:36.053 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 35 v 28 (ObjNum 34 snap 0 seq_num 34) dirty exists 2020-01-16T00:12:36.053 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1263616 length: 899584 chunk_info offset: 1823744 length 153600 2020-01-16T00:12:36.053 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 35 v 28 (ObjNum 34 snap 0 seq_num 34) dirty exists 2020-01-16T00:12:36.054 INFO:tasks.rados.rados.0.smithi123.stdout:390: finishing set_chunk to oid 41 2020-01-16T00:12:36.054 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 41 v 36 (ObjNum 40 snap 0 seq_num 40) dirty exists 2020-01-16T00:12:36.054 INFO:tasks.rados.rados.0.smithi123.stdout:385: finishing set_chunk to oid 36 2020-01-16T00:12:36.054 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 399360 length: 603648 chunk_info offset: 643584 length 327680 2020-01-16T00:12:36.054 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 36 v 8 (ObjNum 35 snap 0 seq_num 35) dirty exists 2020-01-16T00:12:36.054 INFO:tasks.rados.rados.0.smithi123.stdout:384: done (15 left) 2020-01-16T00:12:36.054 INFO:tasks.rados.rados.0.smithi123.stdout:385: done (14 left) 2020-01-16T00:12:36.054 INFO:tasks.rados.rados.0.smithi123.stdout:390: done (13 left) 2020-01-16T00:12:36.055 INFO:tasks.rados.rados.0.smithi123.stdout:399: set_chunk oid 50 offset: 195584 length: 359424 target oid 50 low_tier tgt_offset: 195584 2020-01-16T00:12:36.055 INFO:tasks.rados.rados.0.smithi123.stdout:383: finishing set_chunk to oid 34 2020-01-16T00:12:36.055 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 172032 length: 1383936 chunk_info offset: 615936 length 180224 2020-01-16T00:12:36.055 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 34 v 12 (ObjNum 33 snap 0 seq_num 33) dirty exists 2020-01-16T00:12:36.055 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 172032 length: 1383936 chunk_info offset: 901632 length 1154560 2020-01-16T00:12:36.055 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 34 v 12 (ObjNum 33 snap 0 seq_num 33) dirty exists 2020-01-16T00:12:36.055 INFO:tasks.rados.rados.0.smithi123.stdout:383: done (13 left) 2020-01-16T00:12:36.055 INFO:tasks.rados.rados.0.smithi123.stdout:400: set_chunk oid 1 offset: 221184 length: 1665024 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 221184 2020-01-16T00:12:36.056 INFO:tasks.rados.rados.0.smithi123.stdout:401: set_chunk oid 2 offset: 155136 length: 452608 target oid 2 low_tier tgt_offset: 155136 2020-01-16T00:12:36.056 INFO:tasks.rados.rados.0.smithi123.stdout:402: set_chunk oid 3 offset: 96256 length: 1916416 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 96256 2020-01-16T00:12:36.056 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.056 INFO:tasks.rados.rados.0.smithi123.stdout:386: finishing set_chunk to oid 37 2020-01-16T00:12:36.056 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 37 v 52 (ObjNum 36 snap 0 seq_num 36) dirty exists 2020-01-16T00:12:36.056 INFO:tasks.rados.rados.0.smithi123.stdout:386: done (15 left) 2020-01-16T00:12:36.056 INFO:tasks.rados.rados.0.smithi123.stdout:403: set_chunk oid 4 offset: 1281536 length: 274944 target oid 4 low_tier tgt_offset: 1281536 2020-01-16T00:12:36.057 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.057 INFO:tasks.rados.rados.0.smithi123.stdout:388: finishing set_chunk to oid 39 2020-01-16T00:12:36.057 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 39 v 42 (ObjNum 38 snap 0 seq_num 38) dirty exists 2020-01-16T00:12:36.057 INFO:tasks.rados.rados.0.smithi123.stdout:388: done (15 left) 2020-01-16T00:12:36.057 INFO:tasks.rados.rados.0.smithi123.stdout:404: set_chunk oid 5 offset: 290816 length: 37376 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 290816 2020-01-16T00:12:36.057 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.057 INFO:tasks.rados.rados.0.smithi123.stdout:389: finishing set_chunk to oid 40 2020-01-16T00:12:36.057 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 710144 length: 1118720 chunk_info offset: 790528 length 688128 2020-01-16T00:12:36.058 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 40 v 23 (ObjNum 39 snap 0 seq_num 39) dirty exists 2020-01-16T00:12:36.058 INFO:tasks.rados.rados.0.smithi123.stdout:389: done (15 left) 2020-01-16T00:12:36.058 INFO:tasks.rados.rados.0.smithi123.stdout:405: set_chunk oid 6 offset: 904704 length: 214528 target oid 6 low_tier tgt_offset: 904704 2020-01-16T00:12:36.058 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.058 INFO:tasks.rados.rados.0.smithi123.stdout:392: finishing set_chunk to oid 43 2020-01-16T00:12:36.058 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 930304 length: 524288 chunk_info offset: 1384448 length 188416 2020-01-16T00:12:36.058 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 43 v 39 (ObjNum 42 snap 0 seq_num 42) dirty exists 2020-01-16T00:12:36.058 INFO:tasks.rados.rados.0.smithi123.stdout:392: done (15 left) 2020-01-16T00:12:36.059 INFO:tasks.rados.rados.0.smithi123.stdout:406: set_chunk oid 7 offset: 151040 length: 493056 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 151040 2020-01-16T00:12:36.059 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.059 INFO:tasks.rados.rados.0.smithi123.stdout:387: finishing set_chunk to oid 38 2020-01-16T00:12:36.059 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 949760 length: 931840 chunk_info offset: 928256 length 830464 2020-01-16T00:12:36.059 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 38 v 22 (ObjNum 37 snap 0 seq_num 37) dirty exists 2020-01-16T00:12:36.059 INFO:tasks.rados.rados.0.smithi123.stdout:391: finishing set_chunk to oid 42 2020-01-16T00:12:36.059 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1257984 length: 517632 chunk_info offset: 1364992 length 1145344 2020-01-16T00:12:36.059 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 42 v 11 (ObjNum 41 snap 0 seq_num 41) dirty exists 2020-01-16T00:12:36.060 INFO:tasks.rados.rados.0.smithi123.stdout:387: done (15 left) 2020-01-16T00:12:36.060 INFO:tasks.rados.rados.0.smithi123.stdout:391: done (14 left) 2020-01-16T00:12:36.060 INFO:tasks.rados.rados.0.smithi123.stdout:407: set_chunk oid 8 offset: 1030656 length: 436736 target oid 8 low_tier tgt_offset: 1030656 2020-01-16T00:12:36.060 INFO:tasks.rados.rados.0.smithi123.stdout:408: set_chunk oid 9 offset: 336896 length: 1717248 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 336896 2020-01-16T00:12:36.060 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.060 INFO:tasks.rados.rados.0.smithi123.stdout:393: finishing set_chunk to oid 44 2020-01-16T00:12:36.060 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 239104 length: 2037248 chunk_info offset: 169472 length 818176 2020-01-16T00:12:36.061 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 44 v 25 (ObjNum 43 snap 0 seq_num 43) dirty exists 2020-01-16T00:12:36.061 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 239104 length: 2037248 chunk_info offset: 1635840 length 423424 2020-01-16T00:12:36.061 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 44 v 25 (ObjNum 43 snap 0 seq_num 43) dirty exists 2020-01-16T00:12:36.061 INFO:tasks.rados.rados.0.smithi123.stdout:393: done (15 left) 2020-01-16T00:12:36.061 INFO:tasks.rados.rados.0.smithi123.stdout:409: set_chunk oid 10 offset: 975872 length: 762880 target oid 10 low_tier tgt_offset: 975872 2020-01-16T00:12:36.061 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.062 INFO:tasks.rados.rados.0.smithi123.stdout:405: finishing set_chunk to oid 6 2020-01-16T00:12:36.062 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 904704 length: 214528 chunk_info offset: 878592 length 645632 2020-01-16T00:12:36.062 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 6 v 10 (ObjNum 5 snap 0 seq_num 5) dirty exists 2020-01-16T00:12:36.062 INFO:tasks.rados.rados.0.smithi123.stdout:405: done (15 left) 2020-01-16T00:12:36.062 INFO:tasks.rados.rados.0.smithi123.stdout:410: set_chunk oid 11 offset: 373760 length: 707072 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 373760 2020-01-16T00:12:36.062 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.062 INFO:tasks.rados.rados.0.smithi123.stdout:395: finishing set_chunk to oid 46 2020-01-16T00:12:36.063 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 48128 length: 2481664 chunk_info offset: 56832 length 1408000 2020-01-16T00:12:36.063 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 46 v 23 (ObjNum 45 snap 0 seq_num 45) dirty exists 2020-01-16T00:12:36.063 INFO:tasks.rados.rados.0.smithi123.stdout:395: done (15 left) 2020-01-16T00:12:36.063 INFO:tasks.rados.rados.0.smithi123.stdout:411: set_chunk oid 12 offset: 111104 length: 300032 target oid 12 low_tier tgt_offset: 111104 2020-01-16T00:12:36.063 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.063 INFO:tasks.rados.rados.0.smithi123.stdout:403: finishing set_chunk to oid 4 2020-01-16T00:12:36.063 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1281536 length: 274944 chunk_info offset: 1020416 length 515584 2020-01-16T00:12:36.063 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 4 v 25 (ObjNum 3 snap 0 seq_num 3) dirty exists 2020-01-16T00:12:36.064 INFO:tasks.rados.rados.0.smithi123.stdout:403: done (15 left) 2020-01-16T00:12:36.064 INFO:tasks.rados.rados.0.smithi123.stdout:412: set_chunk oid 13 offset: 101888 length: 2032640 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 101888 2020-01-16T00:12:36.064 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.064 INFO:tasks.rados.rados.0.smithi123.stdout:401: finishing set_chunk to oid 2 2020-01-16T00:12:36.064 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 155136 length: 452608 chunk_info offset: 476160 length 1240064 2020-01-16T00:12:36.064 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 2 v 13 (ObjNum 1 snap 0 seq_num 1) dirty exists 2020-01-16T00:12:36.064 INFO:tasks.rados.rados.0.smithi123.stdout:401: done (15 left) 2020-01-16T00:12:36.064 INFO:tasks.rados.rados.0.smithi123.stdout:413: set_chunk oid 14 offset: 1798144 length: 391168 target oid 14 low_tier tgt_offset: 1798144 2020-01-16T00:12:36.065 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.065 INFO:tasks.rados.rados.0.smithi123.stdout:406: finishing set_chunk to oid 7 2020-01-16T00:12:36.065 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 151040 length: 493056 chunk_info offset: 160768 length 1134080 2020-01-16T00:12:36.065 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 7 v 14 (ObjNum 6 snap 0 seq_num 6) dirty exists 2020-01-16T00:12:36.065 INFO:tasks.rados.rados.0.smithi123.stdout:406: done (15 left) 2020-01-16T00:12:36.066 INFO:tasks.rados.rados.0.smithi123.stdout:414: set_chunk oid 15 offset: 741376 length: 122880 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 741376 2020-01-16T00:12:36.066 INFO:tasks.rados.rados.0.smithi123.stdout:400: finishing set_chunk to oid 1 2020-01-16T00:12:36.066 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 221184 length: 1665024 chunk_info offset: 44544 length 740352 2020-01-16T00:12:36.066 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 1 v 48 (ObjNum 0 snap 0 seq_num 0) dirty exists 2020-01-16T00:12:36.066 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 221184 length: 1665024 chunk_info offset: 1046016 length 729088 2020-01-16T00:12:36.066 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 1 v 48 (ObjNum 0 snap 0 seq_num 0) dirty exists 2020-01-16T00:12:36.067 INFO:tasks.rados.rados.0.smithi123.stdout:399: finishing set_chunk to oid 50 2020-01-16T00:12:36.067 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 50 v 36 (ObjNum 49 snap 0 seq_num 49) dirty exists 2020-01-16T00:12:36.067 INFO:tasks.rados.rados.0.smithi123.stdout:396: finishing set_chunk to oid 47 2020-01-16T00:12:36.067 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 47 v 45 (ObjNum 46 snap 0 seq_num 46) dirty exists 2020-01-16T00:12:36.067 INFO:tasks.rados.rados.0.smithi123.stdout:404: finishing set_chunk to oid 5 2020-01-16T00:12:36.067 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 290816 length: 37376 chunk_info offset: 102400 length 870912 2020-01-16T00:12:36.067 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 5 v 21 (ObjNum 4 snap 0 seq_num 4) dirty exists 2020-01-16T00:12:36.068 INFO:tasks.rados.rados.0.smithi123.stdout:396: done (15 left) 2020-01-16T00:12:36.068 INFO:tasks.rados.rados.0.smithi123.stdout:399: done (14 left) 2020-01-16T00:12:36.068 INFO:tasks.rados.rados.0.smithi123.stdout:400: done (13 left) 2020-01-16T00:12:36.068 INFO:tasks.rados.rados.0.smithi123.stdout:404: done (12 left) 2020-01-16T00:12:36.068 INFO:tasks.rados.rados.0.smithi123.stdout:415: set_chunk oid 16 offset: 856576 length: 535552 target oid 16 low_tier tgt_offset: 856576 2020-01-16T00:12:36.068 INFO:tasks.rados.rados.0.smithi123.stdout:416: set_chunk oid 17 offset: 247296 length: 1451520 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 247296 2020-01-16T00:12:36.069 INFO:tasks.rados.rados.0.smithi123.stdout:394: finishing set_chunk to oid 45 2020-01-16T00:12:36.069 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 45 v 26 (ObjNum 44 snap 0 seq_num 44) dirty exists 2020-01-16T00:12:36.069 INFO:tasks.rados.rados.0.smithi123.stdout:398: finishing set_chunk to oid 49 2020-01-16T00:12:36.069 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 49 v 27 (ObjNum 48 snap 0 seq_num 48) dirty exists 2020-01-16T00:12:36.069 INFO:tasks.rados.rados.0.smithi123.stdout:409: finishing set_chunk to oid 10 2020-01-16T00:12:36.069 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 975872 length: 762880 chunk_info offset: 1087488 length 573440 2020-01-16T00:12:36.069 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 10 v 22 (ObjNum 9 snap 0 seq_num 9) dirty exists 2020-01-16T00:12:36.069 INFO:tasks.rados.rados.0.smithi123.stdout:402: finishing set_chunk to oid 3 2020-01-16T00:12:36.070 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 96256 length: 1916416 chunk_info offset: 312320 length 603136 2020-01-16T00:12:36.070 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 3 v 13 (ObjNum 2 snap 0 seq_num 2) dirty exists 2020-01-16T00:12:36.070 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 96256 length: 1916416 chunk_info offset: 1156096 length 722944 2020-01-16T00:12:36.070 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 3 v 13 (ObjNum 2 snap 0 seq_num 2) dirty exists 2020-01-16T00:12:36.070 INFO:tasks.rados.rados.0.smithi123.stdout:407: finishing set_chunk to oid 8 2020-01-16T00:12:36.070 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1030656 length: 436736 chunk_info offset: 285696 length 965632 2020-01-16T00:12:36.070 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 8 v 10 (ObjNum 7 snap 0 seq_num 7) dirty exists 2020-01-16T00:12:36.071 INFO:tasks.rados.rados.0.smithi123.stdout:394: done (13 left) 2020-01-16T00:12:36.071 INFO:tasks.rados.rados.0.smithi123.stdout:398: done (12 left) 2020-01-16T00:12:36.071 INFO:tasks.rados.rados.0.smithi123.stdout:402: done (11 left) 2020-01-16T00:12:36.071 INFO:tasks.rados.rados.0.smithi123.stdout:407: done (10 left) 2020-01-16T00:12:36.071 INFO:tasks.rados.rados.0.smithi123.stdout:409: done (9 left) 2020-01-16T00:12:36.071 INFO:tasks.rados.rados.0.smithi123.stdout:417: set_chunk oid 18 offset: 214016 length: 280064 target oid 18 low_tier tgt_offset: 214016 2020-01-16T00:12:36.072 INFO:tasks.rados.rados.0.smithi123.stdout:418: set_chunk oid 19 offset: 739328 length: 689152 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 739328 2020-01-16T00:12:36.072 INFO:tasks.rados.rados.0.smithi123.stdout:419: set_chunk oid 20 offset: 529920 length: 94720 target oid 20 low_tier tgt_offset: 529920 2020-01-16T00:12:36.072 INFO:tasks.rados.rados.0.smithi123.stdout:397: finishing set_chunk to oid 48 2020-01-16T00:12:36.072 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1616896 length: 1153024 chunk_info offset: 264704 length 1399296 2020-01-16T00:12:36.072 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 48 v 21 (ObjNum 47 snap 0 seq_num 47) dirty exists 2020-01-16T00:12:36.072 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1616896 length: 1153024 chunk_info offset: 1871872 length 474624 2020-01-16T00:12:36.073 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 48 v 21 (ObjNum 47 snap 0 seq_num 47) dirty exists 2020-01-16T00:12:36.073 INFO:tasks.rados.rados.0.smithi123.stdout:408: finishing set_chunk to oid 9 2020-01-16T00:12:36.073 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 336896 length: 1717248 chunk_info offset: 514560 length 177152 2020-01-16T00:12:36.073 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 9 v 22 (ObjNum 8 snap 0 seq_num 8) dirty exists 2020-01-16T00:12:36.073 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 336896 length: 1717248 chunk_info offset: 781312 length 2048 2020-01-16T00:12:36.073 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 9 v 22 (ObjNum 8 snap 0 seq_num 8) dirty exists 2020-01-16T00:12:36.073 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 336896 length: 1717248 chunk_info offset: 1072128 length 499200 2020-01-16T00:12:36.073 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 9 v 22 (ObjNum 8 snap 0 seq_num 8) dirty exists 2020-01-16T00:12:36.074 INFO:tasks.rados.rados.0.smithi123.stdout:397: done (11 left) 2020-01-16T00:12:36.074 INFO:tasks.rados.rados.0.smithi123.stdout:408: done (10 left) 2020-01-16T00:12:36.074 INFO:tasks.rados.rados.0.smithi123.stdout:420: set_chunk oid 21 offset: 790528 length: 994816 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 790528 2020-01-16T00:12:36.074 INFO:tasks.rados.rados.0.smithi123.stdout:421: set_chunk oid 22 offset: 549888 length: 105472 target oid 22 low_tier tgt_offset: 549888 2020-01-16T00:12:36.074 INFO:tasks.rados.rados.0.smithi123.stdout:422: set_chunk oid 23 offset: 865280 length: 470016 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 865280 2020-01-16T00:12:36.074 INFO:tasks.rados.rados.0.smithi123.stdout:423: set_chunk oid 24 offset: 155136 length: 601088 target oid 24 low_tier tgt_offset: 155136 2020-01-16T00:12:36.074 INFO:tasks.rados.rados.0.smithi123.stdout:424: set_chunk oid 25 offset: 120832 length: 1619456 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 120832 2020-01-16T00:12:36.075 INFO:tasks.rados.rados.0.smithi123.stdout:425: set_chunk oid 26 offset: 541696 length: 1556992 target oid 26 low_tier tgt_offset: 541696 2020-01-16T00:12:36.075 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.075 INFO:tasks.rados.rados.0.smithi123.stdout:413: finishing set_chunk to oid 14 2020-01-16T00:12:36.076 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 14 v 46 (ObjNum 13 snap 0 seq_num 13) dirty exists 2020-01-16T00:12:36.076 INFO:tasks.rados.rados.0.smithi123.stdout:413: done (15 left) 2020-01-16T00:12:36.076 INFO:tasks.rados.rados.0.smithi123.stdout:426: set_chunk oid 27 offset: 783360 length: 846848 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 783360 2020-01-16T00:12:36.076 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.076 INFO:tasks.rados.rados.0.smithi123.stdout:412: finishing set_chunk to oid 13 2020-01-16T00:12:36.076 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 101888 length: 2032640 chunk_info offset: 4096 length 737792 2020-01-16T00:12:36.076 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 13 (ObjNum 12 snap 0 seq_num 12) dirty exists 2020-01-16T00:12:36.077 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 101888 length: 2032640 chunk_info offset: 916992 length 596480 2020-01-16T00:12:36.077 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 13 (ObjNum 12 snap 0 seq_num 12) dirty exists 2020-01-16T00:12:36.077 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 101888 length: 2032640 chunk_info offset: 1541120 length 378368 2020-01-16T00:12:36.077 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 13 (ObjNum 12 snap 0 seq_num 12) dirty exists 2020-01-16T00:12:36.077 INFO:tasks.rados.rados.0.smithi123.stdout:412: done (15 left) 2020-01-16T00:12:36.077 INFO:tasks.rados.rados.0.smithi123.stdout:427: set_chunk oid 28 offset: 815616 length: 1115136 target oid 28 low_tier tgt_offset: 815616 2020-01-16T00:12:36.077 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.078 INFO:tasks.rados.rados.0.smithi123.stdout:410: finishing set_chunk to oid 11 2020-01-16T00:12:36.078 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 373760 length: 707072 chunk_info offset: 279552 length 1427968 2020-01-16T00:12:36.078 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 11 v 14 (ObjNum 10 snap 0 seq_num 10) dirty exists 2020-01-16T00:12:36.078 INFO:tasks.rados.rados.0.smithi123.stdout:410: done (15 left) 2020-01-16T00:12:36.078 INFO:tasks.rados.rados.0.smithi123.stdout:428: set_chunk oid 29 offset: 503808 length: 918528 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 503808 2020-01-16T00:12:36.078 INFO:tasks.rados.rados.0.smithi123.stdout:418: finishing set_chunk to oid 19 2020-01-16T00:12:36.078 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 739328 length: 689152 chunk_info offset: 592384 length 1625600 2020-01-16T00:12:36.078 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 19 v 15 (ObjNum 18 snap 0 seq_num 18) dirty exists 2020-01-16T00:12:36.079 INFO:tasks.rados.rados.0.smithi123.stdout:418: done (15 left) 2020-01-16T00:12:36.079 INFO:tasks.rados.rados.0.smithi123.stdout:429: set_chunk oid 30 offset: 1156608 length: 117760 target oid 30 low_tier tgt_offset: 1156608 2020-01-16T00:12:36.079 INFO:tasks.rados.rados.0.smithi123.stdout:411: finishing set_chunk to oid 12 2020-01-16T00:12:36.079 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 12 v 30 (ObjNum 11 snap 0 seq_num 11) dirty exists 2020-01-16T00:12:36.079 INFO:tasks.rados.rados.0.smithi123.stdout:411: done (15 left) 2020-01-16T00:12:36.079 INFO:tasks.rados.rados.0.smithi123.stdout:430: set_chunk oid 31 offset: 275968 length: 1394688 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 275968 2020-01-16T00:12:36.079 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.079 INFO:tasks.rados.rados.0.smithi123.stdout:417: finishing set_chunk to oid 18 2020-01-16T00:12:36.080 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 18 v 55 (ObjNum 17 snap 0 seq_num 17) dirty exists 2020-01-16T00:12:36.080 INFO:tasks.rados.rados.0.smithi123.stdout:417: done (15 left) 2020-01-16T00:12:36.080 INFO:tasks.rados.rados.0.smithi123.stdout:431: set_chunk oid 32 offset: 524800 length: 1588224 target oid 32 low_tier tgt_offset: 524800 2020-01-16T00:12:36.080 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.080 INFO:tasks.rados.rados.0.smithi123.stdout:421: finishing set_chunk to oid 22 2020-01-16T00:12:36.080 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 22 v 10 (ObjNum 21 snap 0 seq_num 21) dirty exists 2020-01-16T00:12:36.080 INFO:tasks.rados.rados.0.smithi123.stdout:420: finishing set_chunk to oid 21 2020-01-16T00:12:36.080 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 790528 length: 994816 chunk_info offset: 346112 length 838656 2020-01-16T00:12:36.081 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 21 v 17 (ObjNum 20 snap 0 seq_num 20) dirty exists 2020-01-16T00:12:36.081 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 790528 length: 994816 chunk_info offset: 1643008 length 45056 2020-01-16T00:12:36.081 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 21 v 17 (ObjNum 20 snap 0 seq_num 20) dirty exists 2020-01-16T00:12:36.081 INFO:tasks.rados.rados.0.smithi123.stdout:420: done (15 left) 2020-01-16T00:12:36.081 INFO:tasks.rados.rados.0.smithi123.stdout:421: done (14 left) 2020-01-16T00:12:36.081 INFO:tasks.rados.rados.0.smithi123.stdout:432: set_chunk oid 33 offset: 444928 length: 1911296 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 444928 2020-01-16T00:12:36.081 INFO:tasks.rados.rados.0.smithi123.stdout:433: set_chunk oid 34 offset: 827392 length: 1490944 target oid 34 low_tier tgt_offset: 827392 2020-01-16T00:12:36.082 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.082 INFO:tasks.rados.rados.0.smithi123.stdout:415: finishing set_chunk to oid 16 2020-01-16T00:12:36.082 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 856576 length: 535552 chunk_info offset: 833536 length 364544 2020-01-16T00:12:36.082 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 16 v 42 (ObjNum 15 snap 0 seq_num 15) dirty exists 2020-01-16T00:12:36.082 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 856576 length: 535552 chunk_info offset: 1227776 length 248320 2020-01-16T00:12:36.082 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 16 v 42 (ObjNum 15 snap 0 seq_num 15) dirty exists 2020-01-16T00:12:36.082 INFO:tasks.rados.rados.0.smithi123.stdout:415: done (15 left) 2020-01-16T00:12:36.083 INFO:tasks.rados.rados.0.smithi123.stdout:434: set_chunk oid 35 offset: 827392 length: 1368576 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 827392 2020-01-16T00:12:36.083 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.083 INFO:tasks.rados.rados.0.smithi123.stdout:419: finishing set_chunk to oid 20 2020-01-16T00:12:36.083 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 529920 length: 94720 chunk_info offset: 385536 length 861184 2020-01-16T00:12:36.083 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 20 v 38 (ObjNum 19 snap 0 seq_num 19) dirty exists 2020-01-16T00:12:36.083 INFO:tasks.rados.rados.0.smithi123.stdout:419: done (15 left) 2020-01-16T00:12:36.083 INFO:tasks.rados.rados.0.smithi123.stdout:435: set_chunk oid 36 offset: 408064 length: 876032 target oid 36 low_tier tgt_offset: 408064 2020-01-16T00:12:36.083 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.084 INFO:tasks.rados.rados.0.smithi123.stdout:414: finishing set_chunk to oid 15 2020-01-16T00:12:36.084 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 741376 length: 122880 chunk_info offset: 752640 length 650240 2020-01-16T00:12:36.084 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 15 v 33 (ObjNum 14 snap 0 seq_num 14) dirty exists 2020-01-16T00:12:36.084 INFO:tasks.rados.rados.0.smithi123.stdout:414: done (15 left) 2020-01-16T00:12:36.084 INFO:tasks.rados.rados.0.smithi123.stdout:436: set_chunk oid 37 offset: 1891328 length: 405504 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1891328 2020-01-16T00:12:36.084 INFO:tasks.rados.rados.0.smithi123.stdout:424: finishing set_chunk to oid 25 2020-01-16T00:12:36.084 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 120832 length: 1619456 chunk_info offset: 84992 length 680448 2020-01-16T00:12:36.084 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 25 v 24 (ObjNum 24 snap 0 seq_num 24) dirty exists 2020-01-16T00:12:36.085 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 120832 length: 1619456 chunk_info offset: 773632 length 1419776 2020-01-16T00:12:36.085 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 25 v 24 (ObjNum 24 snap 0 seq_num 24) dirty exists 2020-01-16T00:12:36.085 INFO:tasks.rados.rados.0.smithi123.stdout:424: done (15 left) 2020-01-16T00:12:36.085 INFO:tasks.rados.rados.0.smithi123.stdout:437: set_chunk oid 38 offset: 803840 length: 931840 target oid 38 low_tier tgt_offset: 803840 2020-01-16T00:12:36.085 INFO:tasks.rados.rados.0.smithi123.stdout:423: finishing set_chunk to oid 24 2020-01-16T00:12:36.085 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 155136 length: 601088 chunk_info offset: 30720 length 1585664 2020-01-16T00:12:36.085 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 24 v 22 (ObjNum 23 snap 0 seq_num 23) dirty exists 2020-01-16T00:12:36.086 INFO:tasks.rados.rados.0.smithi123.stdout:425: finishing set_chunk to oid 26 2020-01-16T00:12:36.086 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 541696 length: 1556992 chunk_info offset: 963072 length 376320 2020-01-16T00:12:36.086 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 26 v 26 (ObjNum 25 snap 0 seq_num 25) dirty exists 2020-01-16T00:12:36.086 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 541696 length: 1556992 chunk_info offset: 1494016 length 315392 2020-01-16T00:12:36.086 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 26 v 26 (ObjNum 25 snap 0 seq_num 25) dirty exists 2020-01-16T00:12:36.086 INFO:tasks.rados.rados.0.smithi123.stdout:423: done (15 left) 2020-01-16T00:12:36.087 INFO:tasks.rados.rados.0.smithi123.stdout:425: done (14 left) 2020-01-16T00:12:36.087 INFO:tasks.rados.rados.0.smithi123.stdout:438: set_chunk oid 39 offset: 1060352 length: 964096 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1060352 2020-01-16T00:12:36.087 INFO:tasks.rados.rados.0.smithi123.stdout:439: set_chunk oid 40 offset: 1830400 length: 708608 target oid 40 low_tier tgt_offset: 1830400 2020-01-16T00:12:36.087 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.087 INFO:tasks.rados.rados.0.smithi123.stdout:427: finishing set_chunk to oid 28 2020-01-16T00:12:36.087 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 815616 length: 1115136 chunk_info offset: 1498624 length 783872 2020-01-16T00:12:36.087 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 28 v 14 (ObjNum 27 snap 0 seq_num 27) dirty exists 2020-01-16T00:12:36.088 INFO:tasks.rados.rados.0.smithi123.stdout:426: finishing set_chunk to oid 27 2020-01-16T00:12:36.088 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 783360 length: 846848 chunk_info offset: 113152 length 1699840 2020-01-16T00:12:36.088 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 27 v 11 (ObjNum 26 snap 0 seq_num 26) dirty exists 2020-01-16T00:12:36.088 INFO:tasks.rados.rados.0.smithi123.stdout:430: finishing set_chunk to oid 31 2020-01-16T00:12:36.088 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 275968 length: 1394688 chunk_info offset: 263680 length 1668096 2020-01-16T00:12:36.088 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 31 v 15 (ObjNum 30 snap 0 seq_num 30) dirty exists 2020-01-16T00:12:36.088 INFO:tasks.rados.rados.0.smithi123.stdout:426: done (15 left) 2020-01-16T00:12:36.088 INFO:tasks.rados.rados.0.smithi123.stdout:427: done (14 left) 2020-01-16T00:12:36.089 INFO:tasks.rados.rados.0.smithi123.stdout:430: done (13 left) 2020-01-16T00:12:36.089 INFO:tasks.rados.rados.0.smithi123.stdout:440: set_chunk oid 41 offset: 60416 length: 1306624 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 60416 2020-01-16T00:12:36.089 INFO:tasks.rados.rados.0.smithi123.stdout:441: set_chunk oid 42 offset: 81408 length: 1716224 target oid 42 low_tier tgt_offset: 81408 2020-01-16T00:12:36.089 INFO:tasks.rados.rados.0.smithi123.stdout:442: set_chunk oid 43 offset: 1175552 length: 1271296 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1175552 2020-01-16T00:12:36.089 INFO:tasks.rados.rados.0.smithi123.stdout:428: finishing set_chunk to oid 29 2020-01-16T00:12:36.089 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 503808 length: 918528 chunk_info offset: 1186304 length 657408 2020-01-16T00:12:36.089 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 29 v 11 (ObjNum 28 snap 0 seq_num 28) dirty exists 2020-01-16T00:12:36.090 INFO:tasks.rados.rados.0.smithi123.stdout:433: finishing set_chunk to oid 34 2020-01-16T00:12:36.090 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 827392 length: 1490944 chunk_info offset: 901632 length 1154560 2020-01-16T00:12:36.090 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 34 v 12 (ObjNum 33 snap 0 seq_num 33) dirty exists 2020-01-16T00:12:36.090 INFO:tasks.rados.rados.0.smithi123.stdout:428: done (15 left) 2020-01-16T00:12:36.090 INFO:tasks.rados.rados.0.smithi123.stdout:433: done (14 left) 2020-01-16T00:12:36.090 INFO:tasks.rados.rados.0.smithi123.stdout:443: set_chunk oid 44 offset: 2134528 length: 413696 target oid 44 low_tier tgt_offset: 2134528 2020-01-16T00:12:36.090 INFO:tasks.rados.rados.0.smithi123.stdout:444: set_chunk oid 45 offset: 682496 length: 1383936 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 682496 2020-01-16T00:12:36.090 INFO:tasks.rados.rados.0.smithi123.stdout:416: finishing set_chunk to oid 17 2020-01-16T00:12:36.091 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 247296 length: 1451520 chunk_info offset: 330752 length 378368 2020-01-16T00:12:36.091 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 17 v 26 (ObjNum 16 snap 0 seq_num 16) dirty exists 2020-01-16T00:12:36.091 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 247296 length: 1451520 chunk_info offset: 1497600 length 145920 2020-01-16T00:12:36.091 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 17 v 26 (ObjNum 16 snap 0 seq_num 16) dirty exists 2020-01-16T00:12:36.091 INFO:tasks.rados.rados.0.smithi123.stdout:422: finishing set_chunk to oid 23 2020-01-16T00:12:36.091 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 865280 length: 470016 chunk_info offset: 808960 length 1406976 2020-01-16T00:12:36.091 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 23 v 35 (ObjNum 22 snap 0 seq_num 22) dirty exists 2020-01-16T00:12:36.091 INFO:tasks.rados.rados.0.smithi123.stdout:435: finishing set_chunk to oid 36 2020-01-16T00:12:36.092 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 408064 length: 876032 chunk_info offset: 643584 length 327680 2020-01-16T00:12:36.092 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 36 v 8 (ObjNum 35 snap 0 seq_num 35) dirty exists 2020-01-16T00:12:36.092 INFO:tasks.rados.rados.0.smithi123.stdout:416: done (15 left) 2020-01-16T00:12:36.092 INFO:tasks.rados.rados.0.smithi123.stdout:422: done (14 left) 2020-01-16T00:12:36.092 INFO:tasks.rados.rados.0.smithi123.stdout:435: done (13 left) 2020-01-16T00:12:36.092 INFO:tasks.rados.rados.0.smithi123.stdout:445: set_chunk oid 46 offset: 346624 length: 1594368 target oid 46 low_tier tgt_offset: 346624 2020-01-16T00:12:36.092 INFO:tasks.rados.rados.0.smithi123.stdout:446: set_chunk oid 47 offset: 2636288 length: 12800 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 2636288 2020-01-16T00:12:36.092 INFO:tasks.rados.rados.0.smithi123.stdout:447: set_chunk oid 48 offset: 1316352 length: 359424 target oid 48 low_tier tgt_offset: 1316352 2020-01-16T00:12:36.093 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.093 INFO:tasks.rados.rados.0.smithi123.stdout:429: finishing set_chunk to oid 30 2020-01-16T00:12:36.093 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1156608 length: 117760 chunk_info offset: 1087488 length 253952 2020-01-16T00:12:36.093 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 30 v 18 (ObjNum 29 snap 0 seq_num 29) dirty exists 2020-01-16T00:12:36.093 INFO:tasks.rados.rados.0.smithi123.stdout:429: done (15 left) 2020-01-16T00:12:36.093 INFO:tasks.rados.rados.0.smithi123.stdout:448: set_chunk oid 49 offset: 1801216 length: 863232 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1801216 2020-01-16T00:12:36.093 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.094 INFO:tasks.rados.rados.0.smithi123.stdout:434: finishing set_chunk to oid 35 2020-01-16T00:12:36.094 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 827392 length: 1368576 chunk_info offset: 926720 length 423424 2020-01-16T00:12:36.094 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 35 v 28 (ObjNum 34 snap 0 seq_num 34) dirty exists 2020-01-16T00:12:36.094 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 827392 length: 1368576 chunk_info offset: 1823744 length 153600 2020-01-16T00:12:36.094 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 35 v 28 (ObjNum 34 snap 0 seq_num 34) dirty exists 2020-01-16T00:12:36.094 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 827392 length: 1368576 chunk_info offset: 2168320 length 226816 2020-01-16T00:12:36.094 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 35 v 28 (ObjNum 34 snap 0 seq_num 34) dirty exists 2020-01-16T00:12:36.094 INFO:tasks.rados.rados.0.smithi123.stdout:434: done (15 left) 2020-01-16T00:12:36.095 INFO:tasks.rados.rados.0.smithi123.stdout:449: set_chunk oid 50 offset: 2422272 length: 237568 target oid 50 low_tier tgt_offset: 2422272 2020-01-16T00:12:36.095 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.095 INFO:tasks.rados.rados.0.smithi123.stdout:432: finishing set_chunk to oid 33 2020-01-16T00:12:36.095 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 444928 length: 1911296 chunk_info offset: 658944 length 1355776 2020-01-16T00:12:36.095 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 33 v 21 (ObjNum 32 snap 0 seq_num 32) dirty exists 2020-01-16T00:12:36.095 INFO:tasks.rados.rados.0.smithi123.stdout:432: done (15 left) 2020-01-16T00:12:36.095 INFO:tasks.rados.rados.0.smithi123.stdout:450: set_chunk oid 1 offset: 829952 length: 152576 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 829952 2020-01-16T00:12:36.096 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.096 INFO:tasks.rados.rados.0.smithi123.stdout:436: finishing set_chunk to oid 37 2020-01-16T00:12:36.096 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 37 v 58 (ObjNum 36 snap 0 seq_num 36) dirty exists 2020-01-16T00:12:36.096 INFO:tasks.rados.rados.0.smithi123.stdout:436: done (15 left) 2020-01-16T00:12:36.096 INFO:tasks.rados.rados.0.smithi123.stdout:451: set_chunk oid 2 offset: 919040 length: 468992 target oid 2 low_tier tgt_offset: 919040 2020-01-16T00:12:36.096 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.097 INFO:tasks.rados.rados.0.smithi123.stdout:438: finishing set_chunk to oid 39 2020-01-16T00:12:36.097 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1060352 length: 964096 chunk_info offset: 763392 length 413696 2020-01-16T00:12:36.097 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 39 v 42 (ObjNum 38 snap 0 seq_num 38) dirty exists 2020-01-16T00:12:36.097 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1060352 length: 964096 chunk_info offset: 1570304 length 12288 2020-01-16T00:12:36.097 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 39 v 42 (ObjNum 38 snap 0 seq_num 38) dirty exists 2020-01-16T00:12:36.097 INFO:tasks.rados.rados.0.smithi123.stdout:438: done (15 left) 2020-01-16T00:12:36.097 INFO:tasks.rados.rados.0.smithi123.stdout:452: set_chunk oid 3 offset: 33280 length: 202752 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 33280 2020-01-16T00:12:36.097 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.098 INFO:tasks.rados.rados.0.smithi123.stdout:439: finishing set_chunk to oid 40 2020-01-16T00:12:36.098 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1830400 length: 708608 chunk_info offset: 1926656 length 67072 2020-01-16T00:12:36.098 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 40 v 23 (ObjNum 39 snap 0 seq_num 39) dirty exists 2020-01-16T00:12:36.098 INFO:tasks.rados.rados.0.smithi123.stdout:439: done (15 left) 2020-01-16T00:12:36.098 INFO:tasks.rados.rados.0.smithi123.stdout:453: set_chunk oid 4 offset: 32768 length: 1219072 target oid 4 low_tier tgt_offset: 32768 2020-01-16T00:12:36.098 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.098 INFO:tasks.rados.rados.0.smithi123.stdout:442: finishing set_chunk to oid 43 2020-01-16T00:12:36.099 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1175552 length: 1271296 chunk_info offset: 1384448 length 188416 2020-01-16T00:12:36.099 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 43 v 39 (ObjNum 42 snap 0 seq_num 42) dirty exists 2020-01-16T00:12:36.099 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1175552 length: 1271296 chunk_info offset: 2331136 length 192000 2020-01-16T00:12:36.099 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 43 v 39 (ObjNum 42 snap 0 seq_num 42) dirty exists 2020-01-16T00:12:36.099 INFO:tasks.rados.rados.0.smithi123.stdout:442: done (15 left) 2020-01-16T00:12:36.099 INFO:tasks.rados.rados.0.smithi123.stdout:454: set_chunk oid 5 offset: 922112 length: 958464 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 922112 2020-01-16T00:12:36.099 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.099 INFO:tasks.rados.rados.0.smithi123.stdout:431: finishing set_chunk to oid 32 2020-01-16T00:12:36.100 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 524800 length: 1588224 chunk_info offset: 605696 length 877568 2020-01-16T00:12:36.100 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 32 v 18 (ObjNum 31 snap 0 seq_num 31) dirty exists 2020-01-16T00:12:36.100 INFO:tasks.rados.rados.0.smithi123.stdout:431: done (15 left) 2020-01-16T00:12:36.100 INFO:tasks.rados.rados.0.smithi123.stdout:455: set_chunk oid 6 offset: 721920 length: 587264 target oid 6 low_tier tgt_offset: 721920 2020-01-16T00:12:36.100 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.100 INFO:tasks.rados.rados.0.smithi123.stdout:440: finishing set_chunk to oid 41 2020-01-16T00:12:36.100 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 60416 length: 1306624 chunk_info offset: 925184 length 293376 2020-01-16T00:12:36.100 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 41 v 36 (ObjNum 40 snap 0 seq_num 40) dirty exists 2020-01-16T00:12:36.101 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 60416 length: 1306624 chunk_info offset: 1332736 length 44544 2020-01-16T00:12:36.101 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 41 v 36 (ObjNum 40 snap 0 seq_num 40) dirty exists 2020-01-16T00:12:36.101 INFO:tasks.rados.rados.0.smithi123.stdout:440: done (15 left) 2020-01-16T00:12:36.101 INFO:tasks.rados.rados.0.smithi123.stdout:456: set_chunk oid 7 offset: 570880 length: 565248 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 570880 2020-01-16T00:12:36.101 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.101 INFO:tasks.rados.rados.0.smithi123.stdout:437: finishing set_chunk to oid 38 2020-01-16T00:12:36.101 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 803840 length: 931840 chunk_info offset: 928256 length 830464 2020-01-16T00:12:36.101 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 38 v 22 (ObjNum 37 snap 0 seq_num 37) dirty exists 2020-01-16T00:12:36.102 INFO:tasks.rados.rados.0.smithi123.stdout:437: done (15 left) 2020-01-16T00:12:36.102 INFO:tasks.rados.rados.0.smithi123.stdout:457: set_chunk oid 8 offset: 500736 length: 1264640 target oid 8 low_tier tgt_offset: 500736 2020-01-16T00:12:36.102 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.102 INFO:tasks.rados.rados.0.smithi123.stdout:443: finishing set_chunk to oid 44 2020-01-16T00:12:36.102 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 44 v 50 (ObjNum 43 snap 0 seq_num 43) dirty exists 2020-01-16T00:12:36.102 INFO:tasks.rados.rados.0.smithi123.stdout:443: done (15 left) 2020-01-16T00:12:36.102 INFO:tasks.rados.rados.0.smithi123.stdout:458: set_chunk oid 9 offset: 349696 length: 761344 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 349696 2020-01-16T00:12:36.103 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.103 INFO:tasks.rados.rados.0.smithi123.stdout:445: finishing set_chunk to oid 46 2020-01-16T00:12:36.103 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 346624 length: 1594368 chunk_info offset: 56832 length 1408000 2020-01-16T00:12:36.103 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 46 v 23 (ObjNum 45 snap 0 seq_num 45) dirty exists 2020-01-16T00:12:36.103 INFO:tasks.rados.rados.0.smithi123.stdout:445: done (15 left) 2020-01-16T00:12:36.103 INFO:tasks.rados.rados.0.smithi123.stdout:459: set_chunk oid 10 offset: 873472 length: 746496 target oid 10 low_tier tgt_offset: 873472 2020-01-16T00:12:36.103 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.103 INFO:tasks.rados.rados.0.smithi123.stdout:444: finishing set_chunk to oid 45 2020-01-16T00:12:36.104 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 682496 length: 1383936 chunk_info offset: 640512 length 403968 2020-01-16T00:12:36.104 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 45 v 26 (ObjNum 44 snap 0 seq_num 44) dirty exists 2020-01-16T00:12:36.104 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 682496 length: 1383936 chunk_info offset: 1445888 length 278528 2020-01-16T00:12:36.104 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 45 v 26 (ObjNum 44 snap 0 seq_num 44) dirty exists 2020-01-16T00:12:36.104 INFO:tasks.rados.rados.0.smithi123.stdout:444: done (15 left) 2020-01-16T00:12:36.104 INFO:tasks.rados.rados.0.smithi123.stdout:460: set_chunk oid 11 offset: 801280 length: 987136 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 801280 2020-01-16T00:12:36.104 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.104 INFO:tasks.rados.rados.0.smithi123.stdout:451: finishing set_chunk to oid 2 2020-01-16T00:12:36.105 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 919040 length: 468992 chunk_info offset: 476160 length 1240064 2020-01-16T00:12:36.105 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 2 v 13 (ObjNum 1 snap 0 seq_num 1) dirty exists 2020-01-16T00:12:36.105 INFO:tasks.rados.rados.0.smithi123.stdout:451: done (15 left) 2020-01-16T00:12:36.105 INFO:tasks.rados.rados.0.smithi123.stdout:461: set_chunk oid 12 offset: 787968 length: 1234944 target oid 12 low_tier tgt_offset: 787968 2020-01-16T00:12:36.105 INFO:tasks.rados.rados.0.smithi123.stdout:441: finishing set_chunk to oid 42 2020-01-16T00:12:36.105 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 81408 length: 1716224 chunk_info offset: 1364992 length 1145344 2020-01-16T00:12:36.105 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 42 v 11 (ObjNum 41 snap 0 seq_num 41) dirty exists 2020-01-16T00:12:36.106 INFO:tasks.rados.rados.0.smithi123.stdout:452: finishing set_chunk to oid 3 2020-01-16T00:12:36.106 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 3 v 21 (ObjNum 2 snap 0 seq_num 2) dirty exists 2020-01-16T00:12:36.106 INFO:tasks.rados.rados.0.smithi123.stdout:450: finishing set_chunk to oid 1 2020-01-16T00:12:36.106 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 1 v 60 (ObjNum 0 snap 0 seq_num 0) dirty exists 2020-01-16T00:12:36.106 INFO:tasks.rados.rados.0.smithi123.stdout:448: finishing set_chunk to oid 49 2020-01-16T00:12:36.106 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1801216 length: 863232 chunk_info offset: 1658880 length 398336 2020-01-16T00:12:36.107 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 49 v 27 (ObjNum 48 snap 0 seq_num 48) dirty exists 2020-01-16T00:12:36.107 INFO:tasks.rados.rados.0.smithi123.stdout:441: done (15 left) 2020-01-16T00:12:36.107 INFO:tasks.rados.rados.0.smithi123.stdout:448: done (14 left) 2020-01-16T00:12:36.107 INFO:tasks.rados.rados.0.smithi123.stdout:450: done (13 left) 2020-01-16T00:12:36.107 INFO:tasks.rados.rados.0.smithi123.stdout:452: done (12 left) 2020-01-16T00:12:36.107 INFO:tasks.rados.rados.0.smithi123.stdout:462: set_chunk oid 13 offset: 738816 length: 927744 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 738816 2020-01-16T00:12:36.107 INFO:tasks.rados.rados.0.smithi123.stdout:463: set_chunk oid 14 offset: 173056 length: 15360 target oid 14 low_tier tgt_offset: 173056 2020-01-16T00:12:36.107 INFO:tasks.rados.rados.0.smithi123.stdout:464: set_chunk oid 15 offset: 444928 length: 178176 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 444928 2020-01-16T00:12:36.108 INFO:tasks.rados.rados.0.smithi123.stdout:465: set_chunk oid 16 offset: 1323520 length: 457216 target oid 16 low_tier tgt_offset: 1323520 2020-01-16T00:12:36.108 INFO:tasks.rados.rados.0.smithi123.stdout:447: finishing set_chunk to oid 48 2020-01-16T00:12:36.108 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1316352 length: 359424 chunk_info offset: 264704 length 1399296 2020-01-16T00:12:36.108 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 48 v 21 (ObjNum 47 snap 0 seq_num 47) dirty exists 2020-01-16T00:12:36.108 INFO:tasks.rados.rados.0.smithi123.stdout:447: done (15 left) 2020-01-16T00:12:36.108 INFO:tasks.rados.rados.0.smithi123.stdout:466: set_chunk oid 17 offset: 69632 length: 1705984 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 69632 2020-01-16T00:12:36.108 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.109 INFO:tasks.rados.rados.0.smithi123.stdout:446: finishing set_chunk to oid 47 2020-01-16T00:12:36.109 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 47 v 50 (ObjNum 46 snap 0 seq_num 46) dirty exists 2020-01-16T00:12:36.109 INFO:tasks.rados.rados.0.smithi123.stdout:453: finishing set_chunk to oid 4 2020-01-16T00:12:36.109 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 32768 length: 1219072 chunk_info offset: 803328 length 176128 2020-01-16T00:12:36.109 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 4 v 25 (ObjNum 3 snap 0 seq_num 3) dirty exists 2020-01-16T00:12:36.109 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 32768 length: 1219072 chunk_info offset: 1020416 length 515584 2020-01-16T00:12:36.109 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 4 v 25 (ObjNum 3 snap 0 seq_num 3) dirty exists 2020-01-16T00:12:36.109 INFO:tasks.rados.rados.0.smithi123.stdout:446: done (15 left) 2020-01-16T00:12:36.110 INFO:tasks.rados.rados.0.smithi123.stdout:453: done (14 left) 2020-01-16T00:12:36.110 INFO:tasks.rados.rados.0.smithi123.stdout:467: set_chunk oid 18 offset: 863744 length: 775168 target oid 18 low_tier tgt_offset: 863744 2020-01-16T00:12:36.110 INFO:tasks.rados.rados.0.smithi123.stdout:468: set_chunk oid 19 offset: 146432 length: 221696 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 146432 2020-01-16T00:12:36.110 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.110 INFO:tasks.rados.rados.0.smithi123.stdout:449: finishing set_chunk to oid 50 2020-01-16T00:12:36.110 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 50 v 40 (ObjNum 49 snap 0 seq_num 49) dirty exists 2020-01-16T00:12:36.110 INFO:tasks.rados.rados.0.smithi123.stdout:449: done (15 left) 2020-01-16T00:12:36.110 INFO:tasks.rados.rados.0.smithi123.stdout:469: set_chunk oid 20 offset: 1547776 length: 243200 target oid 20 low_tier tgt_offset: 1547776 2020-01-16T00:12:36.111 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.111 INFO:tasks.rados.rados.0.smithi123.stdout:458: finishing set_chunk to oid 9 2020-01-16T00:12:36.111 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 349696 length: 761344 chunk_info offset: 514560 length 177152 2020-01-16T00:12:36.111 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 9 v 22 (ObjNum 8 snap 0 seq_num 8) dirty exists 2020-01-16T00:12:36.111 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 349696 length: 761344 chunk_info offset: 781312 length 2048 2020-01-16T00:12:36.111 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 9 v 22 (ObjNum 8 snap 0 seq_num 8) dirty exists 2020-01-16T00:12:36.111 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 349696 length: 761344 chunk_info offset: 1072128 length 499200 2020-01-16T00:12:36.111 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 9 v 22 (ObjNum 8 snap 0 seq_num 8) dirty exists 2020-01-16T00:12:36.112 INFO:tasks.rados.rados.0.smithi123.stdout:458: done (15 left) 2020-01-16T00:12:36.112 INFO:tasks.rados.rados.0.smithi123.stdout:470: set_chunk oid 21 offset: 225280 length: 1706496 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 225280 2020-01-16T00:12:36.112 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.112 INFO:tasks.rados.rados.0.smithi123.stdout:454: finishing set_chunk to oid 5 2020-01-16T00:12:36.112 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 922112 length: 958464 chunk_info offset: 102400 length 870912 2020-01-16T00:12:36.112 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 5 v 21 (ObjNum 4 snap 0 seq_num 4) dirty exists 2020-01-16T00:12:36.112 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 922112 length: 958464 chunk_info offset: 1779712 length 28672 2020-01-16T00:12:36.113 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 5 v 21 (ObjNum 4 snap 0 seq_num 4) dirty exists 2020-01-16T00:12:36.113 INFO:tasks.rados.rados.0.smithi123.stdout:454: done (15 left) 2020-01-16T00:12:36.113 INFO:tasks.rados.rados.0.smithi123.stdout:471: set_chunk oid 22 offset: 93184 length: 1159168 target oid 22 low_tier tgt_offset: 93184 2020-01-16T00:12:36.113 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.113 INFO:tasks.rados.rados.0.smithi123.stdout:456: finishing set_chunk to oid 7 2020-01-16T00:12:36.113 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 570880 length: 565248 chunk_info offset: 160768 length 1134080 2020-01-16T00:12:36.113 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 7 v 14 (ObjNum 6 snap 0 seq_num 6) dirty exists 2020-01-16T00:12:36.113 INFO:tasks.rados.rados.0.smithi123.stdout:457: finishing set_chunk to oid 8 2020-01-16T00:12:36.114 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 500736 length: 1264640 chunk_info offset: 285696 length 965632 2020-01-16T00:12:36.114 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 8 v 10 (ObjNum 7 snap 0 seq_num 7) dirty exists 2020-01-16T00:12:36.114 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 500736 length: 1264640 chunk_info offset: 1589248 length 433664 2020-01-16T00:12:36.114 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 8 v 10 (ObjNum 7 snap 0 seq_num 7) dirty exists 2020-01-16T00:12:36.114 INFO:tasks.rados.rados.0.smithi123.stdout:455: finishing set_chunk to oid 6 2020-01-16T00:12:36.114 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 721920 length: 587264 chunk_info offset: 878592 length 645632 2020-01-16T00:12:36.114 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 6 v 10 (ObjNum 5 snap 0 seq_num 5) dirty exists 2020-01-16T00:12:36.114 INFO:tasks.rados.rados.0.smithi123.stdout:455: done (15 left) 2020-01-16T00:12:36.115 INFO:tasks.rados.rados.0.smithi123.stdout:456: done (14 left) 2020-01-16T00:12:36.115 INFO:tasks.rados.rados.0.smithi123.stdout:457: done (13 left) 2020-01-16T00:12:36.115 INFO:tasks.rados.rados.0.smithi123.stdout:472: set_chunk oid 23 offset: 640000 length: 671744 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 640000 2020-01-16T00:12:36.115 INFO:tasks.rados.rados.0.smithi123.stdout:473: set_chunk oid 24 offset: 1095680 length: 570880 target oid 24 low_tier tgt_offset: 1095680 2020-01-16T00:12:36.115 INFO:tasks.rados.rados.0.smithi123.stdout:474: set_chunk oid 25 offset: 552448 length: 719360 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 552448 2020-01-16T00:12:36.115 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.115 INFO:tasks.rados.rados.0.smithi123.stdout:459: finishing set_chunk to oid 10 2020-01-16T00:12:36.116 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 873472 length: 746496 chunk_info offset: 1087488 length 573440 2020-01-16T00:12:36.116 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 10 v 22 (ObjNum 9 snap 0 seq_num 9) dirty exists 2020-01-16T00:12:36.116 INFO:tasks.rados.rados.0.smithi123.stdout:459: done (15 left) 2020-01-16T00:12:36.116 INFO:tasks.rados.rados.0.smithi123.stdout:475: set_chunk oid 26 offset: 538112 length: 1760256 target oid 26 low_tier tgt_offset: 538112 2020-01-16T00:12:36.116 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.116 INFO:tasks.rados.rados.0.smithi123.stdout:462: finishing set_chunk to oid 13 2020-01-16T00:12:36.116 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 738816 length: 927744 chunk_info offset: 4096 length 737792 2020-01-16T00:12:36.117 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 13 (ObjNum 12 snap 0 seq_num 12) dirty exists 2020-01-16T00:12:36.117 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 738816 length: 927744 chunk_info offset: 916992 length 596480 2020-01-16T00:12:36.117 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 13 (ObjNum 12 snap 0 seq_num 12) dirty exists 2020-01-16T00:12:36.117 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 738816 length: 927744 chunk_info offset: 1541120 length 378368 2020-01-16T00:12:36.117 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 13 (ObjNum 12 snap 0 seq_num 12) dirty exists 2020-01-16T00:12:36.117 INFO:tasks.rados.rados.0.smithi123.stdout:462: done (15 left) 2020-01-16T00:12:36.117 INFO:tasks.rados.rados.0.smithi123.stdout:476: set_chunk oid 27 offset: 759296 length: 732160 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 759296 2020-01-16T00:12:36.118 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.118 INFO:tasks.rados.rados.0.smithi123.stdout:468: finishing set_chunk to oid 19 2020-01-16T00:12:36.118 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 19 v 43 (ObjNum 18 snap 0 seq_num 18) dirty exists 2020-01-16T00:12:36.118 INFO:tasks.rados.rados.0.smithi123.stdout:468: done (15 left) 2020-01-16T00:12:36.118 INFO:tasks.rados.rados.0.smithi123.stdout:477: set_chunk oid 28 offset: 2269696 length: 6144 target oid 28 low_tier tgt_offset: 2269696 2020-01-16T00:12:36.118 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.118 INFO:tasks.rados.rados.0.smithi123.stdout:465: finishing set_chunk to oid 16 2020-01-16T00:12:36.118 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1323520 length: 457216 chunk_info offset: 1227776 length 248320 2020-01-16T00:12:36.119 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 16 v 42 (ObjNum 15 snap 0 seq_num 15) dirty exists 2020-01-16T00:12:36.119 INFO:tasks.rados.rados.0.smithi123.stdout:465: done (15 left) 2020-01-16T00:12:36.119 INFO:tasks.rados.rados.0.smithi123.stdout:478: set_chunk oid 29 offset: 827904 length: 652288 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 827904 2020-01-16T00:12:36.119 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.119 INFO:tasks.rados.rados.0.smithi123.stdout:466: finishing set_chunk to oid 17 2020-01-16T00:12:36.119 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 69632 length: 1705984 chunk_info offset: 330752 length 378368 2020-01-16T00:12:36.119 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 17 v 26 (ObjNum 16 snap 0 seq_num 16) dirty exists 2020-01-16T00:12:36.120 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 69632 length: 1705984 chunk_info offset: 1497600 length 145920 2020-01-16T00:12:36.120 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 17 v 26 (ObjNum 16 snap 0 seq_num 16) dirty exists 2020-01-16T00:12:36.120 INFO:tasks.rados.rados.0.smithi123.stdout:466: done (15 left) 2020-01-16T00:12:36.120 INFO:tasks.rados.rados.0.smithi123.stdout:479: set_chunk oid 30 offset: 1428480 length: 820224 target oid 30 low_tier tgt_offset: 1428480 2020-01-16T00:12:36.120 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.120 INFO:tasks.rados.rados.0.smithi123.stdout:464: finishing set_chunk to oid 15 2020-01-16T00:12:36.120 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 444928 length: 178176 chunk_info offset: 331264 length 198656 2020-01-16T00:12:36.120 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 15 v 33 (ObjNum 14 snap 0 seq_num 14) dirty exists 2020-01-16T00:12:36.121 INFO:tasks.rados.rados.0.smithi123.stdout:464: done (15 left) 2020-01-16T00:12:36.121 INFO:tasks.rados.rados.0.smithi123.stdout:480: set_chunk oid 31 offset: 454656 length: 180736 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 454656 2020-01-16T00:12:36.121 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.121 INFO:tasks.rados.rados.0.smithi123.stdout:467: finishing set_chunk to oid 18 2020-01-16T00:12:36.121 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 863744 length: 775168 chunk_info offset: 953856 length 471040 2020-01-16T00:12:36.121 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 18 v 55 (ObjNum 17 snap 0 seq_num 17) dirty exists 2020-01-16T00:12:36.121 INFO:tasks.rados.rados.0.smithi123.stdout:467: done (15 left) 2020-01-16T00:12:36.121 INFO:tasks.rados.rados.0.smithi123.stdout:481: set_chunk oid 32 offset: 980480 length: 1033728 target oid 32 low_tier tgt_offset: 980480 2020-01-16T00:12:36.122 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.122 INFO:tasks.rados.rados.0.smithi123.stdout:469: finishing set_chunk to oid 20 2020-01-16T00:12:36.122 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1547776 length: 243200 chunk_info offset: 1278464 length 442368 2020-01-16T00:12:36.122 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 20 v 38 (ObjNum 19 snap 0 seq_num 19) dirty exists 2020-01-16T00:12:36.122 INFO:tasks.rados.rados.0.smithi123.stdout:470: finishing set_chunk to oid 21 2020-01-16T00:12:36.122 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 225280 length: 1706496 chunk_info offset: 346112 length 838656 2020-01-16T00:12:36.122 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 21 v 17 (ObjNum 20 snap 0 seq_num 20) dirty exists 2020-01-16T00:12:36.123 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 225280 length: 1706496 chunk_info offset: 1643008 length 45056 2020-01-16T00:12:36.123 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 21 v 17 (ObjNum 20 snap 0 seq_num 20) dirty exists 2020-01-16T00:12:36.123 INFO:tasks.rados.rados.0.smithi123.stdout:471: finishing set_chunk to oid 22 2020-01-16T00:12:36.123 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 93184 length: 1159168 chunk_info offset: 549888 length 105472 2020-01-16T00:12:36.123 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 22 v 10 (ObjNum 21 snap 0 seq_num 21) dirty exists 2020-01-16T00:12:36.123 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 93184 length: 1159168 chunk_info offset: 923648 length 683520 2020-01-16T00:12:36.123 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 22 v 10 (ObjNum 21 snap 0 seq_num 21) dirty exists 2020-01-16T00:12:36.123 INFO:tasks.rados.rados.0.smithi123.stdout:474: finishing set_chunk to oid 25 2020-01-16T00:12:36.124 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 552448 length: 719360 chunk_info offset: 84992 length 680448 2020-01-16T00:12:36.124 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 25 v 24 (ObjNum 24 snap 0 seq_num 24) dirty exists 2020-01-16T00:12:36.124 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 552448 length: 719360 chunk_info offset: 773632 length 1419776 2020-01-16T00:12:36.124 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 25 v 24 (ObjNum 24 snap 0 seq_num 24) dirty exists 2020-01-16T00:12:36.124 INFO:tasks.rados.rados.0.smithi123.stdout:469: done (15 left) 2020-01-16T00:12:36.124 INFO:tasks.rados.rados.0.smithi123.stdout:470: done (14 left) 2020-01-16T00:12:36.124 INFO:tasks.rados.rados.0.smithi123.stdout:471: done (13 left) 2020-01-16T00:12:36.124 INFO:tasks.rados.rados.0.smithi123.stdout:474: done (12 left) 2020-01-16T00:12:36.125 INFO:tasks.rados.rados.0.smithi123.stdout:482: set_chunk oid 33 offset: 418816 length: 1734144 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 418816 2020-01-16T00:12:36.125 INFO:tasks.rados.rados.0.smithi123.stdout:483: set_chunk oid 34 offset: 242176 length: 1713152 target oid 34 low_tier tgt_offset: 242176 2020-01-16T00:12:36.125 INFO:tasks.rados.rados.0.smithi123.stdout:460: finishing set_chunk to oid 11 2020-01-16T00:12:36.125 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 801280 length: 987136 chunk_info offset: 279552 length 1427968 2020-01-16T00:12:36.125 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 11 v 14 (ObjNum 10 snap 0 seq_num 10) dirty exists 2020-01-16T00:12:36.125 INFO:tasks.rados.rados.0.smithi123.stdout:461: finishing set_chunk to oid 12 2020-01-16T00:12:36.125 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 787968 length: 1234944 chunk_info offset: 1033216 length 672768 2020-01-16T00:12:36.126 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 12 v 30 (ObjNum 11 snap 0 seq_num 11) dirty exists 2020-01-16T00:12:36.126 INFO:tasks.rados.rados.0.smithi123.stdout:460: done (13 left) 2020-01-16T00:12:36.126 INFO:tasks.rados.rados.0.smithi123.stdout:461: done (12 left) 2020-01-16T00:12:36.126 INFO:tasks.rados.rados.0.smithi123.stdout:484: set_chunk oid 35 offset: 375808 length: 1355776 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 375808 2020-01-16T00:12:36.126 INFO:tasks.rados.rados.0.smithi123.stdout:485: set_chunk oid 36 offset: 1672704 length: 400896 target oid 36 low_tier tgt_offset: 1672704 2020-01-16T00:12:36.126 INFO:tasks.rados.rados.0.smithi123.stdout:486: set_chunk oid 37 offset: 818688 length: 420352 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 818688 2020-01-16T00:12:36.126 INFO:tasks.rados.rados.0.smithi123.stdout:487: set_chunk oid 38 offset: 2125312 length: 106496 target oid 38 low_tier tgt_offset: 2125312 2020-01-16T00:12:36.126 INFO:tasks.rados.rados.0.smithi123.stdout:473: finishing set_chunk to oid 24 2020-01-16T00:12:36.127 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1095680 length: 570880 chunk_info offset: 30720 length 1585664 2020-01-16T00:12:36.127 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 24 v 22 (ObjNum 23 snap 0 seq_num 23) dirty exists 2020-01-16T00:12:36.127 INFO:tasks.rados.rados.0.smithi123.stdout:478: finishing set_chunk to oid 29 2020-01-16T00:12:36.127 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 827904 length: 652288 chunk_info offset: 1186304 length 657408 2020-01-16T00:12:36.127 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 29 v 11 (ObjNum 28 snap 0 seq_num 28) dirty exists 2020-01-16T00:12:36.127 INFO:tasks.rados.rados.0.smithi123.stdout:473: done (15 left) 2020-01-16T00:12:36.128 INFO:tasks.rados.rados.0.smithi123.stdout:478: done (14 left) 2020-01-16T00:12:36.128 INFO:tasks.rados.rados.0.smithi123.stdout:488: set_chunk oid 39 offset: 2089984 length: 199168 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 2089984 2020-01-16T00:12:36.128 INFO:tasks.rados.rados.0.smithi123.stdout:489: set_chunk oid 40 offset: 692224 length: 1627136 target oid 40 low_tier tgt_offset: 692224 2020-01-16T00:12:36.128 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.128 INFO:tasks.rados.rados.0.smithi123.stdout:463: finishing set_chunk to oid 14 2020-01-16T00:12:36.128 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 173056 length: 15360 chunk_info offset: 55296 length 299008 2020-01-16T00:12:36.128 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 14 v 46 (ObjNum 13 snap 0 seq_num 13) dirty exists 2020-01-16T00:12:36.128 INFO:tasks.rados.rados.0.smithi123.stdout:463: done (15 left) 2020-01-16T00:12:36.129 INFO:tasks.rados.rados.0.smithi123.stdout:490: set_chunk oid 41 offset: 1843200 length: 175616 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1843200 2020-01-16T00:12:36.129 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.129 INFO:tasks.rados.rados.0.smithi123.stdout:476: finishing set_chunk to oid 27 2020-01-16T00:12:36.129 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 759296 length: 732160 chunk_info offset: 113152 length 1699840 2020-01-16T00:12:36.129 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 27 v 11 (ObjNum 26 snap 0 seq_num 26) dirty exists 2020-01-16T00:12:36.129 INFO:tasks.rados.rados.0.smithi123.stdout:475: finishing set_chunk to oid 26 2020-01-16T00:12:36.129 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 538112 length: 1760256 chunk_info offset: 963072 length 376320 2020-01-16T00:12:36.130 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 26 v 26 (ObjNum 25 snap 0 seq_num 25) dirty exists 2020-01-16T00:12:36.130 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 538112 length: 1760256 chunk_info offset: 1494016 length 315392 2020-01-16T00:12:36.130 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 26 v 26 (ObjNum 25 snap 0 seq_num 25) dirty exists 2020-01-16T00:12:36.130 INFO:tasks.rados.rados.0.smithi123.stdout:475: done (15 left) 2020-01-16T00:12:36.130 INFO:tasks.rados.rados.0.smithi123.stdout:476: done (14 left) 2020-01-16T00:12:36.130 INFO:tasks.rados.rados.0.smithi123.stdout:491: set_chunk oid 42 offset: 1627136 length: 663040 target oid 42 low_tier tgt_offset: 1627136 2020-01-16T00:12:36.130 INFO:tasks.rados.rados.0.smithi123.stdout:492: set_chunk oid 43 offset: 450560 length: 1651200 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 450560 2020-01-16T00:12:36.130 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.131 INFO:tasks.rados.rados.0.smithi123.stdout:477: finishing set_chunk to oid 28 2020-01-16T00:12:36.131 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 2269696 length: 6144 chunk_info offset: 1498624 length 783872 2020-01-16T00:12:36.131 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 28 v 14 (ObjNum 27 snap 0 seq_num 27) dirty exists 2020-01-16T00:12:36.131 INFO:tasks.rados.rados.0.smithi123.stdout:477: done (15 left) 2020-01-16T00:12:36.131 INFO:tasks.rados.rados.0.smithi123.stdout:493: set_chunk oid 44 offset: 1068032 length: 349696 target oid 44 low_tier tgt_offset: 1068032 2020-01-16T00:12:36.131 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.131 INFO:tasks.rados.rados.0.smithi123.stdout:472: finishing set_chunk to oid 23 2020-01-16T00:12:36.131 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 640000 length: 671744 chunk_info offset: 808960 length 1406976 2020-01-16T00:12:36.132 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 23 v 35 (ObjNum 22 snap 0 seq_num 22) dirty exists 2020-01-16T00:12:36.132 INFO:tasks.rados.rados.0.smithi123.stdout:472: done (15 left) 2020-01-16T00:12:36.132 INFO:tasks.rados.rados.0.smithi123.stdout:494: set_chunk oid 45 offset: 21504 length: 1860608 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 21504 2020-01-16T00:12:36.132 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.132 INFO:tasks.rados.rados.0.smithi123.stdout:479: finishing set_chunk to oid 30 2020-01-16T00:12:36.132 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1428480 length: 820224 chunk_info offset: 1542656 length 711168 2020-01-16T00:12:36.132 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 30 v 18 (ObjNum 29 snap 0 seq_num 29) dirty exists 2020-01-16T00:12:36.132 INFO:tasks.rados.rados.0.smithi123.stdout:481: finishing set_chunk to oid 32 2020-01-16T00:12:36.133 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 980480 length: 1033728 chunk_info offset: 605696 length 877568 2020-01-16T00:12:36.133 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 32 v 18 (ObjNum 31 snap 0 seq_num 31) dirty exists 2020-01-16T00:12:36.133 INFO:tasks.rados.rados.0.smithi123.stdout:479: done (15 left) 2020-01-16T00:12:36.133 INFO:tasks.rados.rados.0.smithi123.stdout:481: done (14 left) 2020-01-16T00:12:36.133 INFO:tasks.rados.rados.0.smithi123.stdout:495: set_chunk oid 46 offset: 1922048 length: 118272 target oid 46 low_tier tgt_offset: 1922048 2020-01-16T00:12:36.133 INFO:tasks.rados.rados.0.smithi123.stdout:496: set_chunk oid 47 offset: 854528 length: 566784 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 854528 2020-01-16T00:12:36.133 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.134 INFO:tasks.rados.rados.0.smithi123.stdout:482: finishing set_chunk to oid 33 2020-01-16T00:12:36.134 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 418816 length: 1734144 chunk_info offset: 658944 length 1355776 2020-01-16T00:12:36.134 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 33 v 21 (ObjNum 32 snap 0 seq_num 32) dirty exists 2020-01-16T00:12:36.134 INFO:tasks.rados.rados.0.smithi123.stdout:482: done (15 left) 2020-01-16T00:12:36.134 INFO:tasks.rados.rados.0.smithi123.stdout:497: set_chunk oid 48 offset: 1327616 length: 1041408 target oid 48 low_tier tgt_offset: 1327616 2020-01-16T00:12:36.134 INFO:tasks.rados.rados.0.smithi123.stdout:480: finishing set_chunk to oid 31 2020-01-16T00:12:36.134 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 454656 length: 180736 chunk_info offset: 263680 length 1668096 2020-01-16T00:12:36.134 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 31 v 15 (ObjNum 30 snap 0 seq_num 30) dirty exists 2020-01-16T00:12:36.135 INFO:tasks.rados.rados.0.smithi123.stdout:486: finishing set_chunk to oid 37 2020-01-16T00:12:36.135 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 818688 length: 420352 chunk_info offset: 574464 length 1014784 2020-01-16T00:12:36.135 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 37 v 58 (ObjNum 36 snap 0 seq_num 36) dirty exists 2020-01-16T00:12:36.135 INFO:tasks.rados.rados.0.smithi123.stdout:480: done (15 left) 2020-01-16T00:12:36.135 INFO:tasks.rados.rados.0.smithi123.stdout:486: done (14 left) 2020-01-16T00:12:36.135 INFO:tasks.rados.rados.0.smithi123.stdout:498: set_chunk oid 49 offset: 441856 length: 1858560 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 441856 2020-01-16T00:12:36.135 INFO:tasks.rados.rados.0.smithi123.stdout:499: set_chunk oid 50 offset: 2076160 length: 222208 target oid 50 low_tier tgt_offset: 2076160 2020-01-16T00:12:36.136 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.136 INFO:tasks.rados.rados.0.smithi123.stdout:483: finishing set_chunk to oid 34 2020-01-16T00:12:36.136 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 242176 length: 1713152 chunk_info offset: 615936 length 180224 2020-01-16T00:12:36.136 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 34 v 12 (ObjNum 33 snap 0 seq_num 33) dirty exists 2020-01-16T00:12:36.136 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 242176 length: 1713152 chunk_info offset: 901632 length 1154560 2020-01-16T00:12:36.136 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 34 v 12 (ObjNum 33 snap 0 seq_num 33) dirty exists 2020-01-16T00:12:36.136 INFO:tasks.rados.rados.0.smithi123.stdout:485: finishing set_chunk to oid 36 2020-01-16T00:12:36.136 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 36 v 22 (ObjNum 35 snap 0 seq_num 35) dirty exists 2020-01-16T00:12:36.137 INFO:tasks.rados.rados.0.smithi123.stdout:483: done (15 left) 2020-01-16T00:12:36.137 INFO:tasks.rados.rados.0.smithi123.stdout:485: done (14 left) 2020-01-16T00:12:36.137 INFO:tasks.rados.rados.0.smithi123.stdout:500: set_chunk oid 1 offset: 73728 length: 430080 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 73728 2020-01-16T00:12:36.137 INFO:tasks.rados.rados.0.smithi123.stdout:501: set_chunk oid 2 offset: 1326080 length: 125440 target oid 2 low_tier tgt_offset: 1326080 2020-01-16T00:12:36.137 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.137 INFO:tasks.rados.rados.0.smithi123.stdout:490: finishing set_chunk to oid 41 2020-01-16T00:12:36.138 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1843200 length: 175616 chunk_info offset: 1808384 length 557056 2020-01-16T00:12:36.138 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 41 v 36 (ObjNum 40 snap 0 seq_num 40) dirty exists 2020-01-16T00:12:36.138 INFO:tasks.rados.rados.0.smithi123.stdout:490: done (15 left) 2020-01-16T00:12:36.138 INFO:tasks.rados.rados.0.smithi123.stdout:502: set_chunk oid 3 offset: 805376 length: 707584 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 805376 2020-01-16T00:12:36.138 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.138 INFO:tasks.rados.rados.0.smithi123.stdout:494: finishing set_chunk to oid 45 2020-01-16T00:12:36.138 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 21504 length: 1860608 chunk_info offset: 640512 length 403968 2020-01-16T00:12:36.138 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 45 v 26 (ObjNum 44 snap 0 seq_num 44) dirty exists 2020-01-16T00:12:36.139 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 21504 length: 1860608 chunk_info offset: 1445888 length 278528 2020-01-16T00:12:36.139 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 45 v 26 (ObjNum 44 snap 0 seq_num 44) dirty exists 2020-01-16T00:12:36.139 INFO:tasks.rados.rados.0.smithi123.stdout:494: done (15 left) 2020-01-16T00:12:36.139 INFO:tasks.rados.rados.0.smithi123.stdout:503: set_chunk oid 4 offset: 289792 length: 890880 target oid 4 low_tier tgt_offset: 289792 2020-01-16T00:12:36.139 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.139 INFO:tasks.rados.rados.0.smithi123.stdout:487: finishing set_chunk to oid 38 2020-01-16T00:12:36.139 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 2125312 length: 106496 chunk_info offset: 1915392 length 653312 2020-01-16T00:12:36.140 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 38 v 22 (ObjNum 37 snap 0 seq_num 37) dirty exists 2020-01-16T00:12:36.140 INFO:tasks.rados.rados.0.smithi123.stdout:487: done (15 left) 2020-01-16T00:12:36.140 INFO:tasks.rados.rados.0.smithi123.stdout:504: set_chunk oid 5 offset: 81408 length: 1128448 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 81408 2020-01-16T00:12:36.140 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.140 INFO:tasks.rados.rados.0.smithi123.stdout:491: finishing set_chunk to oid 42 2020-01-16T00:12:36.140 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1627136 length: 663040 chunk_info offset: 1364992 length 1145344 2020-01-16T00:12:36.140 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 42 v 11 (ObjNum 41 snap 0 seq_num 41) dirty exists 2020-01-16T00:12:36.140 INFO:tasks.rados.rados.0.smithi123.stdout:491: done (15 left) 2020-01-16T00:12:36.141 INFO:tasks.rados.rados.0.smithi123.stdout:505: set_chunk oid 6 offset: 1079296 length: 941056 target oid 6 low_tier tgt_offset: 1079296 2020-01-16T00:12:36.141 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.141 INFO:tasks.rados.rados.0.smithi123.stdout:484: finishing set_chunk to oid 35 2020-01-16T00:12:36.141 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 375808 length: 1355776 chunk_info offset: 926720 length 423424 2020-01-16T00:12:36.141 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 35 v 28 (ObjNum 34 snap 0 seq_num 34) dirty exists 2020-01-16T00:12:36.141 INFO:tasks.rados.rados.0.smithi123.stdout:495: finishing set_chunk to oid 46 2020-01-16T00:12:36.141 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 46 v 65 (ObjNum 45 snap 0 seq_num 45) dirty exists 2020-01-16T00:12:36.141 INFO:tasks.rados.rados.0.smithi123.stdout:484: done (15 left) 2020-01-16T00:12:36.142 INFO:tasks.rados.rados.0.smithi123.stdout:495: done (14 left) 2020-01-16T00:12:36.142 INFO:tasks.rados.rados.0.smithi123.stdout:506: set_chunk oid 7 offset: 1207296 length: 720896 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1207296 2020-01-16T00:12:36.142 INFO:tasks.rados.rados.0.smithi123.stdout:507: set_chunk oid 8 offset: 137216 length: 1441792 target oid 8 low_tier tgt_offset: 137216 2020-01-16T00:12:36.142 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.142 INFO:tasks.rados.rados.0.smithi123.stdout:497: finishing set_chunk to oid 48 2020-01-16T00:12:36.142 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1327616 length: 1041408 chunk_info offset: 264704 length 1399296 2020-01-16T00:12:36.142 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 48 v 21 (ObjNum 47 snap 0 seq_num 47) dirty exists 2020-01-16T00:12:36.142 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1327616 length: 1041408 chunk_info offset: 1871872 length 474624 2020-01-16T00:12:36.143 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 48 v 21 (ObjNum 47 snap 0 seq_num 47) dirty exists 2020-01-16T00:12:36.143 INFO:tasks.rados.rados.0.smithi123.stdout:499: finishing set_chunk to oid 50 2020-01-16T00:12:36.143 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 2076160 length: 222208 chunk_info offset: 1613312 length 498688 2020-01-16T00:12:36.143 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 50 v 40 (ObjNum 49 snap 0 seq_num 49) dirty exists 2020-01-16T00:12:36.143 INFO:tasks.rados.rados.0.smithi123.stdout:496: finishing set_chunk to oid 47 2020-01-16T00:12:36.143 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 854528 length: 566784 chunk_info offset: 123904 length 1034752 2020-01-16T00:12:36.143 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 47 v 50 (ObjNum 46 snap 0 seq_num 46) dirty exists 2020-01-16T00:12:36.144 INFO:tasks.rados.rados.0.smithi123.stdout:498: finishing set_chunk to oid 49 2020-01-16T00:12:36.144 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 441856 length: 1858560 chunk_info offset: 453632 length 1075200 2020-01-16T00:12:36.144 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 49 v 27 (ObjNum 48 snap 0 seq_num 48) dirty exists 2020-01-16T00:12:36.144 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 441856 length: 1858560 chunk_info offset: 1658880 length 398336 2020-01-16T00:12:36.144 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 49 v 27 (ObjNum 48 snap 0 seq_num 48) dirty exists 2020-01-16T00:12:36.144 INFO:tasks.rados.rados.0.smithi123.stdout:488: finishing set_chunk to oid 39 2020-01-16T00:12:36.144 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 39 v 52 (ObjNum 38 snap 0 seq_num 38) dirty exists 2020-01-16T00:12:36.144 INFO:tasks.rados.rados.0.smithi123.stdout:500: finishing set_chunk to oid 1 2020-01-16T00:12:36.145 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 73728 length: 430080 chunk_info offset: 44544 length 740352 2020-01-16T00:12:36.145 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 1 v 60 (ObjNum 0 snap 0 seq_num 0) dirty exists 2020-01-16T00:12:36.145 INFO:tasks.rados.rados.0.smithi123.stdout:488: done (15 left) 2020-01-16T00:12:36.145 INFO:tasks.rados.rados.0.smithi123.stdout:496: done (14 left) 2020-01-16T00:12:36.145 INFO:tasks.rados.rados.0.smithi123.stdout:497: done (13 left) 2020-01-16T00:12:36.145 INFO:tasks.rados.rados.0.smithi123.stdout:498: done (12 left) 2020-01-16T00:12:36.145 INFO:tasks.rados.rados.0.smithi123.stdout:499: done (11 left) 2020-01-16T00:12:36.145 INFO:tasks.rados.rados.0.smithi123.stdout:500: done (10 left) 2020-01-16T00:12:36.146 INFO:tasks.rados.rados.0.smithi123.stdout:508: set_chunk oid 9 offset: 543232 length: 1582080 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 543232 2020-01-16T00:12:36.146 INFO:tasks.rados.rados.0.smithi123.stdout:509: set_chunk oid 10 offset: 1445376 length: 497664 target oid 10 low_tier tgt_offset: 1445376 2020-01-16T00:12:36.146 INFO:tasks.rados.rados.0.smithi123.stdout:510: set_chunk oid 11 offset: 276992 length: 1611776 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 276992 2020-01-16T00:12:36.146 INFO:tasks.rados.rados.0.smithi123.stdout:511: set_chunk oid 12 offset: 1411072 length: 95744 target oid 12 low_tier tgt_offset: 1411072 2020-01-16T00:12:36.146 INFO:tasks.rados.rados.0.smithi123.stdout:512: set_chunk oid 13 offset: 946688 length: 1119744 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 946688 2020-01-16T00:12:36.146 INFO:tasks.rados.rados.0.smithi123.stdout:489: finishing set_chunk to oid 40 2020-01-16T00:12:36.146 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 692224 length: 1627136 chunk_info offset: 790528 length 688128 2020-01-16T00:12:36.147 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 40 v 23 (ObjNum 39 snap 0 seq_num 39) dirty exists 2020-01-16T00:12:36.147 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 692224 length: 1627136 chunk_info offset: 1926656 length 67072 2020-01-16T00:12:36.147 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 40 v 23 (ObjNum 39 snap 0 seq_num 39) dirty exists 2020-01-16T00:12:36.147 INFO:tasks.rados.rados.0.smithi123.stdout:502: finishing set_chunk to oid 3 2020-01-16T00:12:36.147 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 805376 length: 707584 chunk_info offset: 312320 length 603136 2020-01-16T00:12:36.147 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 3 v 21 (ObjNum 2 snap 0 seq_num 2) dirty exists 2020-01-16T00:12:36.148 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 805376 length: 707584 chunk_info offset: 1156096 length 722944 2020-01-16T00:12:36.148 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 3 v 21 (ObjNum 2 snap 0 seq_num 2) dirty exists 2020-01-16T00:12:36.148 INFO:tasks.rados.rados.0.smithi123.stdout:492: finishing set_chunk to oid 43 2020-01-16T00:12:36.148 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 450560 length: 1651200 chunk_info offset: 1384448 length 188416 2020-01-16T00:12:36.148 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 43 v 39 (ObjNum 42 snap 0 seq_num 42) dirty exists 2020-01-16T00:12:36.148 INFO:tasks.rados.rados.0.smithi123.stdout:489: done (14 left) 2020-01-16T00:12:36.148 INFO:tasks.rados.rados.0.smithi123.stdout:492: done (13 left) 2020-01-16T00:12:36.148 INFO:tasks.rados.rados.0.smithi123.stdout:502: done (12 left) 2020-01-16T00:12:36.149 INFO:tasks.rados.rados.0.smithi123.stdout:513: set_chunk oid 14 offset: 1253888 length: 170496 target oid 14 low_tier tgt_offset: 1253888 2020-01-16T00:12:36.149 INFO:tasks.rados.rados.0.smithi123.stdout:514: set_chunk oid 15 offset: 421376 length: 1074176 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 421376 2020-01-16T00:12:36.149 INFO:tasks.rados.rados.0.smithi123.stdout:503: finishing set_chunk to oid 4 2020-01-16T00:12:36.149 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 289792 length: 890880 chunk_info offset: 803328 length 176128 2020-01-16T00:12:36.149 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 4 v 25 (ObjNum 3 snap 0 seq_num 3) dirty exists 2020-01-16T00:12:36.149 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 289792 length: 890880 chunk_info offset: 1020416 length 515584 2020-01-16T00:12:36.149 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 4 v 25 (ObjNum 3 snap 0 seq_num 3) dirty exists 2020-01-16T00:12:36.150 INFO:tasks.rados.rados.0.smithi123.stdout:503: done (13 left) 2020-01-16T00:12:36.150 INFO:tasks.rados.rados.0.smithi123.stdout:515: set_chunk oid 16 offset: 937984 length: 294400 target oid 16 low_tier tgt_offset: 937984 2020-01-16T00:12:36.150 INFO:tasks.rados.rados.0.smithi123.stdout:516: set_chunk oid 17 offset: 312832 length: 1673216 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 312832 2020-01-16T00:12:36.150 INFO:tasks.rados.rados.0.smithi123.stdout:517: set_chunk oid 18 offset: 919552 length: 704512 target oid 18 low_tier tgt_offset: 919552 2020-01-16T00:12:36.150 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.150 INFO:tasks.rados.rados.0.smithi123.stdout:504: finishing set_chunk to oid 5 2020-01-16T00:12:36.150 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 81408 length: 1128448 chunk_info offset: 102400 length 870912 2020-01-16T00:12:36.150 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 5 v 21 (ObjNum 4 snap 0 seq_num 4) dirty exists 2020-01-16T00:12:36.151 INFO:tasks.rados.rados.0.smithi123.stdout:504: done (15 left) 2020-01-16T00:12:36.151 INFO:tasks.rados.rados.0.smithi123.stdout:518: set_chunk oid 19 offset: 174080 length: 53760 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 174080 2020-01-16T00:12:36.151 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.151 INFO:tasks.rados.rados.0.smithi123.stdout:501: finishing set_chunk to oid 2 2020-01-16T00:12:36.151 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1326080 length: 125440 chunk_info offset: 476160 length 1240064 2020-01-16T00:12:36.151 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 2 v 13 (ObjNum 1 snap 0 seq_num 1) dirty exists 2020-01-16T00:12:36.151 INFO:tasks.rados.rados.0.smithi123.stdout:505: finishing set_chunk to oid 6 2020-01-16T00:12:36.151 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1079296 length: 941056 chunk_info offset: 878592 length 645632 2020-01-16T00:12:36.152 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 6 v 10 (ObjNum 5 snap 0 seq_num 5) dirty exists 2020-01-16T00:12:36.152 INFO:tasks.rados.rados.0.smithi123.stdout:493: finishing set_chunk to oid 44 2020-01-16T00:12:36.152 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 44 v 55 (ObjNum 43 snap 0 seq_num 43) dirty exists 2020-01-16T00:12:36.152 INFO:tasks.rados.rados.0.smithi123.stdout:493: done (15 left) 2020-01-16T00:12:36.152 INFO:tasks.rados.rados.0.smithi123.stdout:501: done (14 left) 2020-01-16T00:12:36.152 INFO:tasks.rados.rados.0.smithi123.stdout:505: done (13 left) 2020-01-16T00:12:36.152 INFO:tasks.rados.rados.0.smithi123.stdout:519: set_chunk oid 20 offset: 269824 length: 393728 target oid 20 low_tier tgt_offset: 269824 2020-01-16T00:12:36.153 INFO:tasks.rados.rados.0.smithi123.stdout:520: set_chunk oid 21 offset: 515584 length: 1544704 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 515584 2020-01-16T00:12:36.153 INFO:tasks.rados.rados.0.smithi123.stdout:521: set_chunk oid 22 offset: 284672 length: 83968 target oid 22 low_tier tgt_offset: 284672 2020-01-16T00:12:36.153 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.153 INFO:tasks.rados.rados.0.smithi123.stdout:507: finishing set_chunk to oid 8 2020-01-16T00:12:36.153 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 137216 length: 1441792 chunk_info offset: 285696 length 965632 2020-01-16T00:12:36.153 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 8 v 10 (ObjNum 7 snap 0 seq_num 7) dirty exists 2020-01-16T00:12:36.153 INFO:tasks.rados.rados.0.smithi123.stdout:507: done (15 left) 2020-01-16T00:12:36.153 INFO:tasks.rados.rados.0.smithi123.stdout:522: set_chunk oid 23 offset: 804352 length: 1430528 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 804352 2020-01-16T00:12:36.154 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.154 INFO:tasks.rados.rados.0.smithi123.stdout:506: finishing set_chunk to oid 7 2020-01-16T00:12:36.154 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1207296 length: 720896 chunk_info offset: 160768 length 1134080 2020-01-16T00:12:36.154 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 7 v 14 (ObjNum 6 snap 0 seq_num 6) dirty exists 2020-01-16T00:12:36.154 INFO:tasks.rados.rados.0.smithi123.stdout:513: finishing set_chunk to oid 14 2020-01-16T00:12:36.154 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1253888 length: 170496 chunk_info offset: 470528 length 1206272 2020-01-16T00:12:36.154 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 14 v 46 (ObjNum 13 snap 0 seq_num 13) dirty exists 2020-01-16T00:12:36.154 INFO:tasks.rados.rados.0.smithi123.stdout:506: done (15 left) 2020-01-16T00:12:36.155 INFO:tasks.rados.rados.0.smithi123.stdout:513: done (14 left) 2020-01-16T00:12:36.155 INFO:tasks.rados.rados.0.smithi123.stdout:523: set_chunk oid 24 offset: 583168 length: 1126400 target oid 24 low_tier tgt_offset: 583168 2020-01-16T00:12:36.155 INFO:tasks.rados.rados.0.smithi123.stdout:524: set_chunk oid 25 offset: 219648 length: 236544 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 219648 2020-01-16T00:12:36.155 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.155 INFO:tasks.rados.rados.0.smithi123.stdout:512: finishing set_chunk to oid 13 2020-01-16T00:12:36.155 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 946688 length: 1119744 chunk_info offset: 916992 length 596480 2020-01-16T00:12:36.155 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 13 (ObjNum 12 snap 0 seq_num 12) dirty exists 2020-01-16T00:12:36.156 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 946688 length: 1119744 chunk_info offset: 1541120 length 378368 2020-01-16T00:12:36.156 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 13 (ObjNum 12 snap 0 seq_num 12) dirty exists 2020-01-16T00:12:36.156 INFO:tasks.rados.rados.0.smithi123.stdout:512: done (15 left) 2020-01-16T00:12:36.156 INFO:tasks.rados.rados.0.smithi123.stdout:525: set_chunk oid 26 offset: 1965056 length: 33280 target oid 26 low_tier tgt_offset: 1965056 2020-01-16T00:12:36.156 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.156 INFO:tasks.rados.rados.0.smithi123.stdout:509: finishing set_chunk to oid 10 2020-01-16T00:12:36.156 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1445376 length: 497664 chunk_info offset: 1087488 length 573440 2020-01-16T00:12:36.156 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 10 v 22 (ObjNum 9 snap 0 seq_num 9) dirty exists 2020-01-16T00:12:36.157 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1445376 length: 497664 chunk_info offset: 1757696 length 131072 2020-01-16T00:12:36.157 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 10 v 22 (ObjNum 9 snap 0 seq_num 9) dirty exists 2020-01-16T00:12:36.157 INFO:tasks.rados.rados.0.smithi123.stdout:508: finishing set_chunk to oid 9 2020-01-16T00:12:36.157 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 543232 length: 1582080 chunk_info offset: 514560 length 177152 2020-01-16T00:12:36.157 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 9 v 22 (ObjNum 8 snap 0 seq_num 8) dirty exists 2020-01-16T00:12:36.157 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 543232 length: 1582080 chunk_info offset: 781312 length 2048 2020-01-16T00:12:36.158 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 9 v 22 (ObjNum 8 snap 0 seq_num 8) dirty exists 2020-01-16T00:12:36.158 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 543232 length: 1582080 chunk_info offset: 1072128 length 499200 2020-01-16T00:12:36.158 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 9 v 22 (ObjNum 8 snap 0 seq_num 8) dirty exists 2020-01-16T00:12:36.158 INFO:tasks.rados.rados.0.smithi123.stdout:514: finishing set_chunk to oid 15 2020-01-16T00:12:36.158 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 421376 length: 1074176 chunk_info offset: 331264 length 198656 2020-01-16T00:12:36.158 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 15 v 33 (ObjNum 14 snap 0 seq_num 14) dirty exists 2020-01-16T00:12:36.158 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 421376 length: 1074176 chunk_info offset: 752640 length 650240 2020-01-16T00:12:36.158 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 15 v 33 (ObjNum 14 snap 0 seq_num 14) dirty exists 2020-01-16T00:12:36.159 INFO:tasks.rados.rados.0.smithi123.stdout:508: done (15 left) 2020-01-16T00:12:36.159 INFO:tasks.rados.rados.0.smithi123.stdout:509: done (14 left) 2020-01-16T00:12:36.159 INFO:tasks.rados.rados.0.smithi123.stdout:514: done (13 left) 2020-01-16T00:12:36.159 INFO:tasks.rados.rados.0.smithi123.stdout:526: set_chunk oid 27 offset: 1710080 length: 238080 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1710080 2020-01-16T00:12:36.159 INFO:tasks.rados.rados.0.smithi123.stdout:527: set_chunk oid 28 offset: 2364416 length: 18944 target oid 28 low_tier tgt_offset: 2364416 2020-01-16T00:12:36.159 INFO:tasks.rados.rados.0.smithi123.stdout:528: set_chunk oid 29 offset: 2000896 length: 161280 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 2000896 2020-01-16T00:12:36.159 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.160 INFO:tasks.rados.rados.0.smithi123.stdout:516: finishing set_chunk to oid 17 2020-01-16T00:12:36.160 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 312832 length: 1673216 chunk_info offset: 330752 length 378368 2020-01-16T00:12:36.160 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 17 v 26 (ObjNum 16 snap 0 seq_num 16) dirty exists 2020-01-16T00:12:36.160 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 312832 length: 1673216 chunk_info offset: 1497600 length 145920 2020-01-16T00:12:36.160 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 17 v 26 (ObjNum 16 snap 0 seq_num 16) dirty exists 2020-01-16T00:12:36.160 INFO:tasks.rados.rados.0.smithi123.stdout:518: finishing set_chunk to oid 19 2020-01-16T00:12:36.160 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 174080 length: 53760 chunk_info offset: 146432 length 221696 2020-01-16T00:12:36.160 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 19 v 43 (ObjNum 18 snap 0 seq_num 18) dirty exists 2020-01-16T00:12:36.161 INFO:tasks.rados.rados.0.smithi123.stdout:516: done (15 left) 2020-01-16T00:12:36.161 INFO:tasks.rados.rados.0.smithi123.stdout:518: done (14 left) 2020-01-16T00:12:36.161 INFO:tasks.rados.rados.0.smithi123.stdout:529: set_chunk oid 30 offset: 823808 length: 1645056 target oid 30 low_tier tgt_offset: 823808 2020-01-16T00:12:36.161 INFO:tasks.rados.rados.0.smithi123.stdout:530: set_chunk oid 31 offset: 1280512 length: 36352 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1280512 2020-01-16T00:12:36.161 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.161 INFO:tasks.rados.rados.0.smithi123.stdout:510: finishing set_chunk to oid 11 2020-01-16T00:12:36.161 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 276992 length: 1611776 chunk_info offset: 279552 length 1427968 2020-01-16T00:12:36.161 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 11 v 14 (ObjNum 10 snap 0 seq_num 10) dirty exists 2020-01-16T00:12:36.162 INFO:tasks.rados.rados.0.smithi123.stdout:510: done (15 left) 2020-01-16T00:12:36.162 INFO:tasks.rados.rados.0.smithi123.stdout:531: set_chunk oid 32 offset: 345600 length: 2071552 target oid 32 low_tier tgt_offset: 345600 2020-01-16T00:12:36.162 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.162 INFO:tasks.rados.rados.0.smithi123.stdout:511: finishing set_chunk to oid 12 2020-01-16T00:12:36.162 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1411072 length: 95744 chunk_info offset: 1033216 length 672768 2020-01-16T00:12:36.162 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 12 v 30 (ObjNum 11 snap 0 seq_num 11) dirty exists 2020-01-16T00:12:36.163 INFO:tasks.rados.rados.0.smithi123.stdout:522: finishing set_chunk to oid 23 2020-01-16T00:12:36.163 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 804352 length: 1430528 chunk_info offset: 808960 length 1406976 2020-01-16T00:12:36.163 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 23 v 35 (ObjNum 22 snap 0 seq_num 22) dirty exists 2020-01-16T00:12:36.163 INFO:tasks.rados.rados.0.smithi123.stdout:511: done (15 left) 2020-01-16T00:12:36.163 INFO:tasks.rados.rados.0.smithi123.stdout:522: done (14 left) 2020-01-16T00:12:36.164 INFO:tasks.rados.rados.0.smithi123.stdout:532: set_chunk oid 33 offset: 437248 length: 577536 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 437248 2020-01-16T00:12:36.164 INFO:tasks.rados.rados.0.smithi123.stdout:533: set_chunk oid 34 offset: 1342464 length: 130560 target oid 34 low_tier tgt_offset: 1342464 2020-01-16T00:12:36.164 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.164 INFO:tasks.rados.rados.0.smithi123.stdout:521: finishing set_chunk to oid 22 2020-01-16T00:12:36.164 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 22 v 12 (ObjNum 21 snap 0 seq_num 21) dirty exists 2020-01-16T00:12:36.164 INFO:tasks.rados.rados.0.smithi123.stdout:523: finishing set_chunk to oid 24 2020-01-16T00:12:36.164 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 583168 length: 1126400 chunk_info offset: 30720 length 1585664 2020-01-16T00:12:36.165 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 24 v 22 (ObjNum 23 snap 0 seq_num 23) dirty exists 2020-01-16T00:12:36.165 INFO:tasks.rados.rados.0.smithi123.stdout:515: finishing set_chunk to oid 16 2020-01-16T00:12:36.165 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 937984 length: 294400 chunk_info offset: 833536 length 364544 2020-01-16T00:12:36.165 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 16 v 42 (ObjNum 15 snap 0 seq_num 15) dirty exists 2020-01-16T00:12:36.165 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 937984 length: 294400 chunk_info offset: 1227776 length 248320 2020-01-16T00:12:36.165 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 16 v 42 (ObjNum 15 snap 0 seq_num 15) dirty exists 2020-01-16T00:12:36.165 INFO:tasks.rados.rados.0.smithi123.stdout:517: finishing set_chunk to oid 18 2020-01-16T00:12:36.166 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 919552 length: 704512 chunk_info offset: 953856 length 471040 2020-01-16T00:12:36.166 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 18 v 55 (ObjNum 17 snap 0 seq_num 17) dirty exists 2020-01-16T00:12:36.166 INFO:tasks.rados.rados.0.smithi123.stdout:515: done (15 left) 2020-01-16T00:12:36.166 INFO:tasks.rados.rados.0.smithi123.stdout:517: done (14 left) 2020-01-16T00:12:36.166 INFO:tasks.rados.rados.0.smithi123.stdout:521: done (13 left) 2020-01-16T00:12:36.166 INFO:tasks.rados.rados.0.smithi123.stdout:523: done (12 left) 2020-01-16T00:12:36.166 INFO:tasks.rados.rados.0.smithi123.stdout:534: set_chunk oid 35 offset: 1010176 length: 1537536 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1010176 2020-01-16T00:12:36.166 INFO:tasks.rados.rados.0.smithi123.stdout:535: set_chunk oid 36 offset: 2366976 length: 203776 target oid 36 low_tier tgt_offset: 2366976 2020-01-16T00:12:36.167 INFO:tasks.rados.rados.0.smithi123.stdout:536: set_chunk oid 37 offset: 1131008 length: 1090560 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1131008 2020-01-16T00:12:36.167 INFO:tasks.rados.rados.0.smithi123.stdout:537: set_chunk oid 38 offset: 1850880 length: 339456 target oid 38 low_tier tgt_offset: 1850880 2020-01-16T00:12:36.167 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.167 INFO:tasks.rados.rados.0.smithi123.stdout:525: finishing set_chunk to oid 26 2020-01-16T00:12:36.167 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 26 v 68 (ObjNum 25 snap 0 seq_num 25) dirty exists 2020-01-16T00:12:36.167 INFO:tasks.rados.rados.0.smithi123.stdout:525: done (15 left) 2020-01-16T00:12:36.167 INFO:tasks.rados.rados.0.smithi123.stdout:538: set_chunk oid 39 offset: 539136 length: 2092544 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 539136 2020-01-16T00:12:36.168 INFO:tasks.rados.rados.0.smithi123.stdout:526: finishing set_chunk to oid 27 2020-01-16T00:12:36.168 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1710080 length: 238080 chunk_info offset: 113152 length 1699840 2020-01-16T00:12:36.168 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 27 v 11 (ObjNum 26 snap 0 seq_num 26) dirty exists 2020-01-16T00:12:36.168 INFO:tasks.rados.rados.0.smithi123.stdout:526: done (15 left) 2020-01-16T00:12:36.168 INFO:tasks.rados.rados.0.smithi123.stdout:539: set_chunk oid 40 offset: 1227776 length: 149504 target oid 40 low_tier tgt_offset: 1227776 2020-01-16T00:12:36.168 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.169 INFO:tasks.rados.rados.0.smithi123.stdout:519: finishing set_chunk to oid 20 2020-01-16T00:12:36.169 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 269824 length: 393728 chunk_info offset: 59392 length 257024 2020-01-16T00:12:36.169 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 20 v 38 (ObjNum 19 snap 0 seq_num 19) dirty exists 2020-01-16T00:12:36.169 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 269824 length: 393728 chunk_info offset: 385536 length 861184 2020-01-16T00:12:36.169 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 20 v 38 (ObjNum 19 snap 0 seq_num 19) dirty exists 2020-01-16T00:12:36.169 INFO:tasks.rados.rados.0.smithi123.stdout:524: finishing set_chunk to oid 25 2020-01-16T00:12:36.169 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 219648 length: 236544 chunk_info offset: 84992 length 680448 2020-01-16T00:12:36.169 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 25 v 24 (ObjNum 24 snap 0 seq_num 24) dirty exists 2020-01-16T00:12:36.170 INFO:tasks.rados.rados.0.smithi123.stdout:519: done (15 left) 2020-01-16T00:12:36.170 INFO:tasks.rados.rados.0.smithi123.stdout:524: done (14 left) 2020-01-16T00:12:36.170 INFO:tasks.rados.rados.0.smithi123.stdout:540: set_chunk oid 41 offset: 451072 length: 431616 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 451072 2020-01-16T00:12:36.170 INFO:tasks.rados.rados.0.smithi123.stdout:541: set_chunk oid 42 offset: 460288 length: 1388544 target oid 42 low_tier tgt_offset: 460288 2020-01-16T00:12:36.170 INFO:tasks.rados.rados.0.smithi123.stdout:520: finishing set_chunk to oid 21 2020-01-16T00:12:36.170 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 515584 length: 1544704 chunk_info offset: 346112 length 838656 2020-01-16T00:12:36.170 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 21 v 17 (ObjNum 20 snap 0 seq_num 20) dirty exists 2020-01-16T00:12:36.171 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 515584 length: 1544704 chunk_info offset: 1643008 length 45056 2020-01-16T00:12:36.171 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 21 v 17 (ObjNum 20 snap 0 seq_num 20) dirty exists 2020-01-16T00:12:36.171 INFO:tasks.rados.rados.0.smithi123.stdout:528: finishing set_chunk to oid 29 2020-01-16T00:12:36.171 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 29 v 35 (ObjNum 28 snap 0 seq_num 28) dirty exists 2020-01-16T00:12:36.171 INFO:tasks.rados.rados.0.smithi123.stdout:527: finishing set_chunk to oid 28 2020-01-16T00:12:36.171 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 28 v 46 (ObjNum 27 snap 0 seq_num 27) dirty exists 2020-01-16T00:12:36.171 INFO:tasks.rados.rados.0.smithi123.stdout:520: done (15 left) 2020-01-16T00:12:36.171 INFO:tasks.rados.rados.0.smithi123.stdout:527: done (14 left) 2020-01-16T00:12:36.172 INFO:tasks.rados.rados.0.smithi123.stdout:528: done (13 left) 2020-01-16T00:12:36.172 INFO:tasks.rados.rados.0.smithi123.stdout:542: set_chunk oid 43 offset: 1094144 length: 882688 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1094144 2020-01-16T00:12:36.172 INFO:tasks.rados.rados.0.smithi123.stdout:543: set_chunk oid 44 offset: 1639424 length: 1038848 target oid 44 low_tier tgt_offset: 1639424 2020-01-16T00:12:36.172 INFO:tasks.rados.rados.0.smithi123.stdout:529: finishing set_chunk to oid 30 2020-01-16T00:12:36.172 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 823808 length: 1645056 chunk_info offset: 1087488 length 253952 2020-01-16T00:12:36.172 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 30 v 18 (ObjNum 29 snap 0 seq_num 29) dirty exists 2020-01-16T00:12:36.172 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 823808 length: 1645056 chunk_info offset: 1542656 length 711168 2020-01-16T00:12:36.172 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 30 v 18 (ObjNum 29 snap 0 seq_num 29) dirty exists 2020-01-16T00:12:36.173 INFO:tasks.rados.rados.0.smithi123.stdout:529: done (14 left) 2020-01-16T00:12:36.173 INFO:tasks.rados.rados.0.smithi123.stdout:544: set_chunk oid 45 offset: 1887232 length: 593920 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1887232 2020-01-16T00:12:36.173 INFO:tasks.rados.rados.0.smithi123.stdout:545: set_chunk oid 46 offset: 823808 length: 364544 target oid 46 low_tier tgt_offset: 823808 2020-01-16T00:12:36.173 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.173 INFO:tasks.rados.rados.0.smithi123.stdout:531: finishing set_chunk to oid 32 2020-01-16T00:12:36.173 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 345600 length: 2071552 chunk_info offset: 605696 length 877568 2020-01-16T00:12:36.173 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 32 v 18 (ObjNum 31 snap 0 seq_num 31) dirty exists 2020-01-16T00:12:36.174 INFO:tasks.rados.rados.0.smithi123.stdout:531: done (15 left) 2020-01-16T00:12:36.174 INFO:tasks.rados.rados.0.smithi123.stdout:546: set_chunk oid 47 offset: 1116672 length: 962560 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1116672 2020-01-16T00:12:36.174 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.174 INFO:tasks.rados.rados.0.smithi123.stdout:533: finishing set_chunk to oid 34 2020-01-16T00:12:36.174 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1342464 length: 130560 chunk_info offset: 901632 length 1154560 2020-01-16T00:12:36.174 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 34 v 12 (ObjNum 33 snap 0 seq_num 33) dirty exists 2020-01-16T00:12:36.174 INFO:tasks.rados.rados.0.smithi123.stdout:533: done (15 left) 2020-01-16T00:12:36.174 INFO:tasks.rados.rados.0.smithi123.stdout:547: set_chunk oid 48 offset: 353792 length: 566272 target oid 48 low_tier tgt_offset: 353792 2020-01-16T00:12:36.175 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.175 INFO:tasks.rados.rados.0.smithi123.stdout:535: finishing set_chunk to oid 36 2020-01-16T00:12:36.175 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 36 v 24 (ObjNum 35 snap 0 seq_num 35) dirty exists 2020-01-16T00:12:36.175 INFO:tasks.rados.rados.0.smithi123.stdout:530: finishing set_chunk to oid 31 2020-01-16T00:12:36.175 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1280512 length: 36352 chunk_info offset: 263680 length 1668096 2020-01-16T00:12:36.175 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 31 v 15 (ObjNum 30 snap 0 seq_num 30) dirty exists 2020-01-16T00:12:36.175 INFO:tasks.rados.rados.0.smithi123.stdout:530: done (15 left) 2020-01-16T00:12:36.176 INFO:tasks.rados.rados.0.smithi123.stdout:535: done (14 left) 2020-01-16T00:12:36.176 INFO:tasks.rados.rados.0.smithi123.stdout:548: set_chunk oid 49 offset: 74752 length: 2561536 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 74752 2020-01-16T00:12:36.176 INFO:tasks.rados.rados.0.smithi123.stdout:549: set_chunk oid 50 offset: 1314304 length: 488960 target oid 50 low_tier tgt_offset: 1314304 2020-01-16T00:12:36.176 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.176 INFO:tasks.rados.rados.0.smithi123.stdout:534: finishing set_chunk to oid 35 2020-01-16T00:12:36.176 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1010176 length: 1537536 chunk_info offset: 926720 length 423424 2020-01-16T00:12:36.176 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 35 v 28 (ObjNum 34 snap 0 seq_num 34) dirty exists 2020-01-16T00:12:36.176 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1010176 length: 1537536 chunk_info offset: 1823744 length 153600 2020-01-16T00:12:36.177 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 35 v 28 (ObjNum 34 snap 0 seq_num 34) dirty exists 2020-01-16T00:12:36.177 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1010176 length: 1537536 chunk_info offset: 2168320 length 226816 2020-01-16T00:12:36.177 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 35 v 28 (ObjNum 34 snap 0 seq_num 34) dirty exists 2020-01-16T00:12:36.177 INFO:tasks.rados.rados.0.smithi123.stdout:534: done (15 left) 2020-01-16T00:12:36.177 INFO:tasks.rados.rados.0.smithi123.stdout:550: set_chunk oid 1 offset: 523264 length: 57344 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 523264 2020-01-16T00:12:36.177 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.177 INFO:tasks.rados.rados.0.smithi123.stdout:538: finishing set_chunk to oid 39 2020-01-16T00:12:36.177 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 539136 length: 2092544 chunk_info offset: 500224 length 171008 2020-01-16T00:12:36.178 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 39 v 52 (ObjNum 38 snap 0 seq_num 38) dirty exists 2020-01-16T00:12:36.178 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 539136 length: 2092544 chunk_info offset: 763392 length 413696 2020-01-16T00:12:36.178 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 39 v 52 (ObjNum 38 snap 0 seq_num 38) dirty exists 2020-01-16T00:12:36.178 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 539136 length: 2092544 chunk_info offset: 1570304 length 12288 2020-01-16T00:12:36.178 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 39 v 52 (ObjNum 38 snap 0 seq_num 38) dirty exists 2020-01-16T00:12:36.179 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 539136 length: 2092544 chunk_info offset: 2089984 length 199168 2020-01-16T00:12:36.179 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 39 v 52 (ObjNum 38 snap 0 seq_num 38) dirty exists 2020-01-16T00:12:36.179 INFO:tasks.rados.rados.0.smithi123.stdout:538: done (15 left) 2020-01-16T00:12:36.179 INFO:tasks.rados.rados.0.smithi123.stdout:551: set_chunk oid 2 offset: 66048 length: 966144 target oid 2 low_tier tgt_offset: 66048 2020-01-16T00:12:36.179 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.179 INFO:tasks.rados.rados.0.smithi123.stdout:532: finishing set_chunk to oid 33 2020-01-16T00:12:36.179 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 437248 length: 577536 chunk_info offset: 658944 length 1355776 2020-01-16T00:12:36.179 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 33 v 21 (ObjNum 32 snap 0 seq_num 32) dirty exists 2020-01-16T00:12:36.180 INFO:tasks.rados.rados.0.smithi123.stdout:532: done (15 left) 2020-01-16T00:12:36.180 INFO:tasks.rados.rados.0.smithi123.stdout:552: set_chunk oid 3 offset: 1616384 length: 113152 target oid 3 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1616384 2020-01-16T00:12:36.180 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.180 INFO:tasks.rados.rados.0.smithi123.stdout:539: finishing set_chunk to oid 40 2020-01-16T00:12:36.180 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1227776 length: 149504 chunk_info offset: 790528 length 688128 2020-01-16T00:12:36.180 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 40 v 23 (ObjNum 39 snap 0 seq_num 39) dirty exists 2020-01-16T00:12:36.180 INFO:tasks.rados.rados.0.smithi123.stdout:539: done (15 left) 2020-01-16T00:12:36.180 INFO:tasks.rados.rados.0.smithi123.stdout:553: set_chunk oid 4 offset: 188928 length: 1134080 target oid 4 low_tier tgt_offset: 188928 2020-01-16T00:12:36.181 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.181 INFO:tasks.rados.rados.0.smithi123.stdout:540: finishing set_chunk to oid 41 2020-01-16T00:12:36.181 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 41 v 48 (ObjNum 40 snap 0 seq_num 40) dirty exists 2020-01-16T00:12:36.181 INFO:tasks.rados.rados.0.smithi123.stdout:540: done (15 left) 2020-01-16T00:12:36.181 INFO:tasks.rados.rados.0.smithi123.stdout:554: set_chunk oid 5 offset: 496128 length: 206848 target oid 5 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 496128 2020-01-16T00:12:36.181 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.181 INFO:tasks.rados.rados.0.smithi123.stdout:536: finishing set_chunk to oid 37 2020-01-16T00:12:36.182 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1131008 length: 1090560 chunk_info offset: 574464 length 1014784 2020-01-16T00:12:36.182 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 37 v 58 (ObjNum 36 snap 0 seq_num 36) dirty exists 2020-01-16T00:12:36.182 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1131008 length: 1090560 chunk_info offset: 1686528 length 34816 2020-01-16T00:12:36.182 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 37 v 58 (ObjNum 36 snap 0 seq_num 36) dirty exists 2020-01-16T00:12:36.182 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1131008 length: 1090560 chunk_info offset: 1891328 length 405504 2020-01-16T00:12:36.182 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 37 v 58 (ObjNum 36 snap 0 seq_num 36) dirty exists 2020-01-16T00:12:36.182 INFO:tasks.rados.rados.0.smithi123.stdout:536: done (15 left) 2020-01-16T00:12:36.182 INFO:tasks.rados.rados.0.smithi123.stdout:555: set_chunk oid 6 offset: 898560 length: 198656 target oid 6 low_tier tgt_offset: 898560 2020-01-16T00:12:36.183 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.183 INFO:tasks.rados.rados.0.smithi123.stdout:537: finishing set_chunk to oid 38 2020-01-16T00:12:36.183 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1850880 length: 339456 chunk_info offset: 1915392 length 653312 2020-01-16T00:12:36.183 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 38 v 22 (ObjNum 37 snap 0 seq_num 37) dirty exists 2020-01-16T00:12:36.183 INFO:tasks.rados.rados.0.smithi123.stdout:537: done (15 left) 2020-01-16T00:12:36.183 INFO:tasks.rados.rados.0.smithi123.stdout:556: set_chunk oid 7 offset: 198656 length: 772608 target oid 7 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 198656 2020-01-16T00:12:36.183 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.184 INFO:tasks.rados.rados.0.smithi123.stdout:542: finishing set_chunk to oid 43 2020-01-16T00:12:36.184 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1094144 length: 882688 chunk_info offset: 1384448 length 188416 2020-01-16T00:12:36.184 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 43 v 39 (ObjNum 42 snap 0 seq_num 42) dirty exists 2020-01-16T00:12:36.184 INFO:tasks.rados.rados.0.smithi123.stdout:542: done (15 left) 2020-01-16T00:12:36.184 INFO:tasks.rados.rados.0.smithi123.stdout:557: set_chunk oid 8 offset: 258560 length: 963072 target oid 8 low_tier tgt_offset: 258560 2020-01-16T00:12:36.184 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.184 INFO:tasks.rados.rados.0.smithi123.stdout:541: finishing set_chunk to oid 42 2020-01-16T00:12:36.184 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 460288 length: 1388544 chunk_info offset: 1364992 length 1145344 2020-01-16T00:12:36.185 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 42 v 11 (ObjNum 41 snap 0 seq_num 41) dirty exists 2020-01-16T00:12:36.185 INFO:tasks.rados.rados.0.smithi123.stdout:552: finishing set_chunk to oid 3 2020-01-16T00:12:36.185 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1616384 length: 113152 chunk_info offset: 1156096 length 722944 2020-01-16T00:12:36.185 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 3 v 21 (ObjNum 2 snap 0 seq_num 2) dirty exists 2020-01-16T00:12:36.185 INFO:tasks.rados.rados.0.smithi123.stdout:541: done (15 left) 2020-01-16T00:12:36.185 INFO:tasks.rados.rados.0.smithi123.stdout:552: done (14 left) 2020-01-16T00:12:36.185 INFO:tasks.rados.rados.0.smithi123.stdout:558: set_chunk oid 9 offset: 60416 length: 1515520 target oid 9 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 60416 2020-01-16T00:12:36.186 INFO:tasks.rados.rados.0.smithi123.stdout:559: set_chunk oid 10 offset: 374784 length: 694784 target oid 10 low_tier tgt_offset: 374784 2020-01-16T00:12:36.186 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.186 INFO:tasks.rados.rados.0.smithi123.stdout:543: finishing set_chunk to oid 44 2020-01-16T00:12:36.186 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1639424 length: 1038848 chunk_info offset: 1635840 length 423424 2020-01-16T00:12:36.186 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 44 v 55 (ObjNum 43 snap 0 seq_num 43) dirty exists 2020-01-16T00:12:36.186 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1639424 length: 1038848 chunk_info offset: 2134528 length 413696 2020-01-16T00:12:36.186 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 44 v 55 (ObjNum 43 snap 0 seq_num 43) dirty exists 2020-01-16T00:12:36.186 INFO:tasks.rados.rados.0.smithi123.stdout:544: finishing set_chunk to oid 45 2020-01-16T00:12:36.187 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 45 v 35 (ObjNum 44 snap 0 seq_num 44) dirty exists 2020-01-16T00:12:36.187 INFO:tasks.rados.rados.0.smithi123.stdout:543: done (15 left) 2020-01-16T00:12:36.187 INFO:tasks.rados.rados.0.smithi123.stdout:544: done (14 left) 2020-01-16T00:12:36.187 INFO:tasks.rados.rados.0.smithi123.stdout:560: set_chunk oid 11 offset: 98816 length: 379392 target oid 11 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 98816 2020-01-16T00:12:36.187 INFO:tasks.rados.rados.0.smithi123.stdout:547: finishing set_chunk to oid 48 2020-01-16T00:12:36.187 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 353792 length: 566272 chunk_info offset: 264704 length 1399296 2020-01-16T00:12:36.187 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 48 v 21 (ObjNum 47 snap 0 seq_num 47) dirty exists 2020-01-16T00:12:36.187 INFO:tasks.rados.rados.0.smithi123.stdout:547: done (14 left) 2020-01-16T00:12:36.188 INFO:tasks.rados.rados.0.smithi123.stdout:561: set_chunk oid 12 offset: 58880 length: 995328 target oid 12 low_tier tgt_offset: 58880 2020-01-16T00:12:36.188 INFO:tasks.rados.rados.0.smithi123.stdout:562: set_chunk oid 13 offset: 214016 length: 1964544 target oid 13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 214016 2020-01-16T00:12:36.188 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.188 INFO:tasks.rados.rados.0.smithi123.stdout:551: finishing set_chunk to oid 2 2020-01-16T00:12:36.188 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 66048 length: 966144 chunk_info offset: 476160 length 1240064 2020-01-16T00:12:36.189 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 2 v 13 (ObjNum 1 snap 0 seq_num 1) dirty exists 2020-01-16T00:12:36.189 INFO:tasks.rados.rados.0.smithi123.stdout:551: done (15 left) 2020-01-16T00:12:36.189 INFO:tasks.rados.rados.0.smithi123.stdout:563: set_chunk oid 14 offset: 1126912 length: 977408 target oid 14 low_tier tgt_offset: 1126912 2020-01-16T00:12:36.189 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.189 INFO:tasks.rados.rados.0.smithi123.stdout:545: finishing set_chunk to oid 46 2020-01-16T00:12:36.189 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 823808 length: 364544 chunk_info offset: 56832 length 1408000 2020-01-16T00:12:36.189 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 46 v 65 (ObjNum 45 snap 0 seq_num 45) dirty exists 2020-01-16T00:12:36.189 INFO:tasks.rados.rados.0.smithi123.stdout:546: finishing set_chunk to oid 47 2020-01-16T00:12:36.190 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1116672 length: 962560 chunk_info offset: 123904 length 1034752 2020-01-16T00:12:36.190 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 47 v 50 (ObjNum 46 snap 0 seq_num 46) dirty exists 2020-01-16T00:12:36.190 INFO:tasks.rados.rados.0.smithi123.stdout:549: finishing set_chunk to oid 50 2020-01-16T00:12:36.190 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1314304 length: 488960 chunk_info offset: 1613312 length 498688 2020-01-16T00:12:36.190 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 50 v 40 (ObjNum 49 snap 0 seq_num 49) dirty exists 2020-01-16T00:12:36.190 INFO:tasks.rados.rados.0.smithi123.stdout:548: finishing set_chunk to oid 49 2020-01-16T00:12:36.190 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 74752 length: 2561536 chunk_info offset: 70656 length 122880 2020-01-16T00:12:36.190 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 49 v 27 (ObjNum 48 snap 0 seq_num 48) dirty exists 2020-01-16T00:12:36.191 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 74752 length: 2561536 chunk_info offset: 453632 length 1075200 2020-01-16T00:12:36.191 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 49 v 27 (ObjNum 48 snap 0 seq_num 48) dirty exists 2020-01-16T00:12:36.191 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 74752 length: 2561536 chunk_info offset: 1658880 length 398336 2020-01-16T00:12:36.191 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 49 v 27 (ObjNum 48 snap 0 seq_num 48) dirty exists 2020-01-16T00:12:36.191 INFO:tasks.rados.rados.0.smithi123.stdout:545: done (15 left) 2020-01-16T00:12:36.191 INFO:tasks.rados.rados.0.smithi123.stdout:546: done (14 left) 2020-01-16T00:12:36.191 INFO:tasks.rados.rados.0.smithi123.stdout:548: done (13 left) 2020-01-16T00:12:36.192 INFO:tasks.rados.rados.0.smithi123.stdout:549: done (12 left) 2020-01-16T00:12:36.192 INFO:tasks.rados.rados.0.smithi123.stdout:564: set_chunk oid 15 offset: 341504 length: 1600512 target oid 15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 341504 2020-01-16T00:12:36.192 INFO:tasks.rados.rados.0.smithi123.stdout:565: set_chunk oid 16 offset: 51712 length: 1907200 target oid 16 low_tier tgt_offset: 51712 2020-01-16T00:12:36.192 INFO:tasks.rados.rados.0.smithi123.stdout:566: set_chunk oid 17 offset: 726528 length: 113152 target oid 17 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 726528 2020-01-16T00:12:36.192 INFO:tasks.rados.rados.0.smithi123.stdout:567: set_chunk oid 18 offset: 1016832 length: 55296 target oid 18 low_tier tgt_offset: 1016832 2020-01-16T00:12:36.192 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.192 INFO:tasks.rados.rados.0.smithi123.stdout:557: finishing set_chunk to oid 8 2020-01-16T00:12:36.192 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 258560 length: 963072 chunk_info offset: 285696 length 965632 2020-01-16T00:12:36.193 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 8 v 10 (ObjNum 7 snap 0 seq_num 7) dirty exists 2020-01-16T00:12:36.193 INFO:tasks.rados.rados.0.smithi123.stdout:557: done (15 left) 2020-01-16T00:12:36.193 INFO:tasks.rados.rados.0.smithi123.stdout:568: set_chunk oid 19 offset: 1225728 length: 925696 target oid 19 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1225728 2020-01-16T00:12:36.193 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.193 INFO:tasks.rados.rados.0.smithi123.stdout:556: finishing set_chunk to oid 7 2020-01-16T00:12:36.193 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 198656 length: 772608 chunk_info offset: 160768 length 1134080 2020-01-16T00:12:36.193 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 7 v 14 (ObjNum 6 snap 0 seq_num 6) dirty exists 2020-01-16T00:12:36.193 INFO:tasks.rados.rados.0.smithi123.stdout:556: done (15 left) 2020-01-16T00:12:36.194 INFO:tasks.rados.rados.0.smithi123.stdout:569: set_chunk oid 20 offset: 483328 length: 726016 target oid 20 low_tier tgt_offset: 483328 2020-01-16T00:12:36.194 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.194 INFO:tasks.rados.rados.0.smithi123.stdout:550: finishing set_chunk to oid 1 2020-01-16T00:12:36.194 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 523264 length: 57344 chunk_info offset: 44544 length 740352 2020-01-16T00:12:36.194 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 1 v 60 (ObjNum 0 snap 0 seq_num 0) dirty exists 2020-01-16T00:12:36.194 INFO:tasks.rados.rados.0.smithi123.stdout:553: finishing set_chunk to oid 4 2020-01-16T00:12:36.194 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 188928 length: 1134080 chunk_info offset: 803328 length 176128 2020-01-16T00:12:36.195 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 4 v 25 (ObjNum 3 snap 0 seq_num 3) dirty exists 2020-01-16T00:12:36.195 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 188928 length: 1134080 chunk_info offset: 1020416 length 515584 2020-01-16T00:12:36.195 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 4 v 25 (ObjNum 3 snap 0 seq_num 3) dirty exists 2020-01-16T00:12:36.195 INFO:tasks.rados.rados.0.smithi123.stdout:550: done (15 left) 2020-01-16T00:12:36.195 INFO:tasks.rados.rados.0.smithi123.stdout:553: done (14 left) 2020-01-16T00:12:36.195 INFO:tasks.rados.rados.0.smithi123.stdout:570: set_chunk oid 21 offset: 58368 length: 1251328 target oid 21 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 58368 2020-01-16T00:12:36.195 INFO:tasks.rados.rados.0.smithi123.stdout:559: finishing set_chunk to oid 10 2020-01-16T00:12:36.195 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 374784 length: 694784 chunk_info offset: 453120 length 360448 2020-01-16T00:12:36.196 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 10 v 22 (ObjNum 9 snap 0 seq_num 9) dirty exists 2020-01-16T00:12:36.196 INFO:tasks.rados.rados.0.smithi123.stdout:559: done (14 left) 2020-01-16T00:12:36.196 INFO:tasks.rados.rados.0.smithi123.stdout:571: set_chunk oid 22 offset: 1022464 length: 1074176 target oid 22 low_tier tgt_offset: 1022464 2020-01-16T00:12:36.196 INFO:tasks.rados.rados.0.smithi123.stdout:562: finishing set_chunk to oid 13 2020-01-16T00:12:36.196 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 214016 length: 1964544 chunk_info offset: 4096 length 737792 2020-01-16T00:12:36.196 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 13 (ObjNum 12 snap 0 seq_num 12) dirty exists 2020-01-16T00:12:36.196 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 214016 length: 1964544 chunk_info offset: 916992 length 596480 2020-01-16T00:12:36.196 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 13 (ObjNum 12 snap 0 seq_num 12) dirty exists 2020-01-16T00:12:36.197 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 214016 length: 1964544 chunk_info offset: 1541120 length 378368 2020-01-16T00:12:36.197 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 13 (ObjNum 12 snap 0 seq_num 12) dirty exists 2020-01-16T00:12:36.197 INFO:tasks.rados.rados.0.smithi123.stdout:562: done (14 left) 2020-01-16T00:12:36.197 INFO:tasks.rados.rados.0.smithi123.stdout:572: set_chunk oid 23 offset: 356352 length: 737792 target oid 23 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 356352 2020-01-16T00:12:36.197 INFO:tasks.rados.rados.0.smithi123.stdout:573: set_chunk oid 24 offset: 151040 length: 1329664 target oid 24 low_tier tgt_offset: 151040 2020-01-16T00:12:36.197 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.197 INFO:tasks.rados.rados.0.smithi123.stdout:563: finishing set_chunk to oid 14 2020-01-16T00:12:36.198 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1126912 length: 977408 chunk_info offset: 470528 length 1206272 2020-01-16T00:12:36.198 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 14 v 46 (ObjNum 13 snap 0 seq_num 13) dirty exists 2020-01-16T00:12:36.198 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1126912 length: 977408 chunk_info offset: 1798144 length 391168 2020-01-16T00:12:36.198 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 14 v 46 (ObjNum 13 snap 0 seq_num 13) dirty exists 2020-01-16T00:12:36.198 INFO:tasks.rados.rados.0.smithi123.stdout:563: done (15 left) 2020-01-16T00:12:36.198 INFO:tasks.rados.rados.0.smithi123.stdout:574: set_chunk oid 25 offset: 2219520 length: 158208 target oid 25 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 2219520 2020-01-16T00:12:36.198 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.199 INFO:tasks.rados.rados.0.smithi123.stdout:558: finishing set_chunk to oid 9 2020-01-16T00:12:36.199 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 60416 length: 1515520 chunk_info offset: 514560 length 177152 2020-01-16T00:12:36.199 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 9 v 22 (ObjNum 8 snap 0 seq_num 8) dirty exists 2020-01-16T00:12:36.199 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 60416 length: 1515520 chunk_info offset: 781312 length 2048 2020-01-16T00:12:36.199 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 9 v 22 (ObjNum 8 snap 0 seq_num 8) dirty exists 2020-01-16T00:12:36.199 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 60416 length: 1515520 chunk_info offset: 1072128 length 499200 2020-01-16T00:12:36.200 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 9 v 22 (ObjNum 8 snap 0 seq_num 8) dirty exists 2020-01-16T00:12:36.200 INFO:tasks.rados.rados.0.smithi123.stdout:558: done (15 left) 2020-01-16T00:12:36.200 INFO:tasks.rados.rados.0.smithi123.stdout:575: set_chunk oid 26 offset: 171008 length: 1206272 target oid 26 low_tier tgt_offset: 171008 2020-01-16T00:12:36.200 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.200 INFO:tasks.rados.rados.0.smithi123.stdout:555: finishing set_chunk to oid 6 2020-01-16T00:12:36.200 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 898560 length: 198656 chunk_info offset: 878592 length 645632 2020-01-16T00:12:36.200 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 6 v 10 (ObjNum 5 snap 0 seq_num 5) dirty exists 2020-01-16T00:12:36.200 INFO:tasks.rados.rados.0.smithi123.stdout:555: done (15 left) 2020-01-16T00:12:36.201 INFO:tasks.rados.rados.0.smithi123.stdout:576: set_chunk oid 27 offset: 105984 length: 337920 target oid 27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 105984 2020-01-16T00:12:36.201 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.201 INFO:tasks.rados.rados.0.smithi123.stdout:566: finishing set_chunk to oid 17 2020-01-16T00:12:36.201 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 17 v 50 (ObjNum 16 snap 0 seq_num 16) dirty exists 2020-01-16T00:12:36.201 INFO:tasks.rados.rados.0.smithi123.stdout:566: done (15 left) 2020-01-16T00:12:36.201 INFO:tasks.rados.rados.0.smithi123.stdout:577: set_chunk oid 28 offset: 397824 length: 386048 target oid 28 low_tier tgt_offset: 397824 2020-01-16T00:12:36.201 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.201 INFO:tasks.rados.rados.0.smithi123.stdout:568: finishing set_chunk to oid 19 2020-01-16T00:12:36.202 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1225728 length: 925696 chunk_info offset: 592384 length 1625600 2020-01-16T00:12:36.202 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 19 v 43 (ObjNum 18 snap 0 seq_num 18) dirty exists 2020-01-16T00:12:36.202 INFO:tasks.rados.rados.0.smithi123.stdout:568: done (15 left) 2020-01-16T00:12:36.202 INFO:tasks.rados.rados.0.smithi123.stdout:578: set_chunk oid 29 offset: 1560576 length: 397312 target oid 29 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1560576 2020-01-16T00:12:36.202 INFO:tasks.rados.rados.0.smithi123.stdout:554: finishing set_chunk to oid 5 2020-01-16T00:12:36.202 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 496128 length: 206848 chunk_info offset: 102400 length 870912 2020-01-16T00:12:36.202 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 5 v 21 (ObjNum 4 snap 0 seq_num 4) dirty exists 2020-01-16T00:12:36.204 INFO:tasks.rados.rados.0.smithi123.stdout:554: done (15 left) 2020-01-16T00:12:36.204 INFO:tasks.rados.rados.0.smithi123.stdout:579: set_chunk oid 30 offset: 2178560 length: 267776 target oid 30 low_tier tgt_offset: 2178560 2020-01-16T00:12:36.205 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.205 INFO:tasks.rados.rados.0.smithi123.stdout:567: finishing set_chunk to oid 18 2020-01-16T00:12:36.205 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1016832 length: 55296 chunk_info offset: 953856 length 471040 2020-01-16T00:12:36.205 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 18 v 55 (ObjNum 17 snap 0 seq_num 17) dirty exists 2020-01-16T00:12:36.205 INFO:tasks.rados.rados.0.smithi123.stdout:567: done (15 left) 2020-01-16T00:12:36.205 INFO:tasks.rados.rados.0.smithi123.stdout:580: set_chunk oid 31 offset: 291328 length: 2107392 target oid 31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 291328 2020-01-16T00:12:36.205 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.206 INFO:tasks.rados.rados.0.smithi123.stdout:560: finishing set_chunk to oid 11 2020-01-16T00:12:36.206 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 98816 length: 379392 chunk_info offset: 156672 length 44032 2020-01-16T00:12:36.206 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 11 v 14 (ObjNum 10 snap 0 seq_num 10) dirty exists 2020-01-16T00:12:36.206 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 98816 length: 379392 chunk_info offset: 279552 length 1427968 2020-01-16T00:12:36.206 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 11 v 14 (ObjNum 10 snap 0 seq_num 10) dirty exists 2020-01-16T00:12:36.206 INFO:tasks.rados.rados.0.smithi123.stdout:560: done (15 left) 2020-01-16T00:12:36.206 INFO:tasks.rados.rados.0.smithi123.stdout:581: set_chunk oid 32 offset: 981504 length: 248832 target oid 32 low_tier tgt_offset: 981504 2020-01-16T00:12:36.206 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.207 INFO:tasks.rados.rados.0.smithi123.stdout:561: finishing set_chunk to oid 12 2020-01-16T00:12:36.207 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 58880 length: 995328 chunk_info offset: 111104 length 300032 2020-01-16T00:12:36.207 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 12 v 30 (ObjNum 11 snap 0 seq_num 11) dirty exists 2020-01-16T00:12:36.207 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 58880 length: 995328 chunk_info offset: 676352 length 111104 2020-01-16T00:12:36.207 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 12 v 30 (ObjNum 11 snap 0 seq_num 11) dirty exists 2020-01-16T00:12:36.207 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 58880 length: 995328 chunk_info offset: 1033216 length 672768 2020-01-16T00:12:36.207 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 12 v 30 (ObjNum 11 snap 0 seq_num 11) dirty exists 2020-01-16T00:12:36.208 INFO:tasks.rados.rados.0.smithi123.stdout:561: done (15 left) 2020-01-16T00:12:36.208 INFO:tasks.rados.rados.0.smithi123.stdout:582: set_chunk oid 33 offset: 914944 length: 228864 target oid 33 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 914944 2020-01-16T00:12:36.208 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.208 INFO:tasks.rados.rados.0.smithi123.stdout:564: finishing set_chunk to oid 15 2020-01-16T00:12:36.208 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 341504 length: 1600512 chunk_info offset: 331264 length 198656 2020-01-16T00:12:36.208 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 15 v 33 (ObjNum 14 snap 0 seq_num 14) dirty exists 2020-01-16T00:12:36.208 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 341504 length: 1600512 chunk_info offset: 752640 length 650240 2020-01-16T00:12:36.208 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 15 v 33 (ObjNum 14 snap 0 seq_num 14) dirty exists 2020-01-16T00:12:36.209 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 341504 length: 1600512 chunk_info offset: 1708032 length 408576 2020-01-16T00:12:36.209 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 15 v 33 (ObjNum 14 snap 0 seq_num 14) dirty exists 2020-01-16T00:12:36.209 INFO:tasks.rados.rados.0.smithi123.stdout:564: done (15 left) 2020-01-16T00:12:36.209 INFO:tasks.rados.rados.0.smithi123.stdout:583: set_chunk oid 34 offset: 508928 length: 1074176 target oid 34 low_tier tgt_offset: 508928 2020-01-16T00:12:36.209 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.209 INFO:tasks.rados.rados.0.smithi123.stdout:570: finishing set_chunk to oid 21 2020-01-16T00:12:36.210 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 58368 length: 1251328 chunk_info offset: 346112 length 838656 2020-01-16T00:12:36.210 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 21 v 17 (ObjNum 20 snap 0 seq_num 20) dirty exists 2020-01-16T00:12:36.210 INFO:tasks.rados.rados.0.smithi123.stdout:571: finishing set_chunk to oid 22 2020-01-16T00:12:36.210 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1022464 length: 1074176 chunk_info offset: 923648 length 683520 2020-01-16T00:12:36.210 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 22 v 12 (ObjNum 21 snap 0 seq_num 21) dirty exists 2020-01-16T00:12:36.210 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1022464 length: 1074176 chunk_info offset: 1822720 length 105472 2020-01-16T00:12:36.210 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 22 v 12 (ObjNum 21 snap 0 seq_num 21) dirty exists 2020-01-16T00:12:36.211 INFO:tasks.rados.rados.0.smithi123.stdout:570: done (15 left) 2020-01-16T00:12:36.211 INFO:tasks.rados.rados.0.smithi123.stdout:571: done (14 left) 2020-01-16T00:12:36.211 INFO:tasks.rados.rados.0.smithi123.stdout:584: set_chunk oid 35 offset: 818688 length: 510464 target oid 35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 818688 2020-01-16T00:12:36.211 INFO:tasks.rados.rados.0.smithi123.stdout:585: set_chunk oid 36 offset: 528896 length: 355328 target oid 36 low_tier tgt_offset: 528896 2020-01-16T00:12:36.211 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.211 INFO:tasks.rados.rados.0.smithi123.stdout:573: finishing set_chunk to oid 24 2020-01-16T00:12:36.211 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 151040 length: 1329664 chunk_info offset: 30720 length 1585664 2020-01-16T00:12:36.211 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 24 v 22 (ObjNum 23 snap 0 seq_num 23) dirty exists 2020-01-16T00:12:36.212 INFO:tasks.rados.rados.0.smithi123.stdout:578: finishing set_chunk to oid 29 2020-01-16T00:12:36.212 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1560576 length: 397312 chunk_info offset: 1186304 length 657408 2020-01-16T00:12:36.212 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 29 v 35 (ObjNum 28 snap 0 seq_num 28) dirty exists 2020-01-16T00:12:36.212 INFO:tasks.rados.rados.0.smithi123.stdout:573: done (15 left) 2020-01-16T00:12:36.212 INFO:tasks.rados.rados.0.smithi123.stdout:578: done (14 left) 2020-01-16T00:12:36.212 INFO:tasks.rados.rados.0.smithi123.stdout:586: set_chunk oid 37 offset: 717312 length: 909312 target oid 37 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 717312 2020-01-16T00:12:36.212 INFO:tasks.rados.rados.0.smithi123.stdout:587: set_chunk oid 38 offset: 69632 length: 773120 target oid 38 low_tier tgt_offset: 69632 2020-01-16T00:12:36.213 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.213 INFO:tasks.rados.rados.0.smithi123.stdout:565: finishing set_chunk to oid 16 2020-01-16T00:12:36.213 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 51712 length: 1907200 chunk_info offset: 228352 length 126976 2020-01-16T00:12:36.213 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 16 v 42 (ObjNum 15 snap 0 seq_num 15) dirty exists 2020-01-16T00:12:36.213 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 51712 length: 1907200 chunk_info offset: 517632 length 45056 2020-01-16T00:12:36.213 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 16 v 42 (ObjNum 15 snap 0 seq_num 15) dirty exists 2020-01-16T00:12:36.213 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 51712 length: 1907200 chunk_info offset: 833536 length 364544 2020-01-16T00:12:36.213 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 16 v 42 (ObjNum 15 snap 0 seq_num 15) dirty exists 2020-01-16T00:12:36.214 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 51712 length: 1907200 chunk_info offset: 1227776 length 248320 2020-01-16T00:12:36.214 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 16 v 42 (ObjNum 15 snap 0 seq_num 15) dirty exists 2020-01-16T00:12:36.214 INFO:tasks.rados.rados.0.smithi123.stdout:575: finishing set_chunk to oid 26 2020-01-16T00:12:36.214 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 171008 length: 1206272 chunk_info offset: 963072 length 376320 2020-01-16T00:12:36.214 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 26 v 68 (ObjNum 25 snap 0 seq_num 25) dirty exists 2020-01-16T00:12:36.214 INFO:tasks.rados.rados.0.smithi123.stdout:565: done (15 left) 2020-01-16T00:12:36.214 INFO:tasks.rados.rados.0.smithi123.stdout:575: done (14 left) 2020-01-16T00:12:36.214 INFO:tasks.rados.rados.0.smithi123.stdout:588: set_chunk oid 39 offset: 669696 length: 1332736 target oid 39 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 669696 2020-01-16T00:12:36.215 INFO:tasks.rados.rados.0.smithi123.stdout:569: finishing set_chunk to oid 20 2020-01-16T00:12:36.215 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 483328 length: 726016 chunk_info offset: 385536 length 861184 2020-01-16T00:12:36.215 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 20 v 38 (ObjNum 19 snap 0 seq_num 19) dirty exists 2020-01-16T00:12:36.215 INFO:tasks.rados.rados.0.smithi123.stdout:569: done (14 left) 2020-01-16T00:12:36.215 INFO:tasks.rados.rados.0.smithi123.stdout:589: set_chunk oid 40 offset: 756736 length: 307712 target oid 40 low_tier tgt_offset: 756736 2020-01-16T00:12:36.215 INFO:tasks.rados.rados.0.smithi123.stdout:574: finishing set_chunk to oid 25 2020-01-16T00:12:36.215 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 25 v 64 (ObjNum 24 snap 0 seq_num 24) dirty exists 2020-01-16T00:12:36.216 INFO:tasks.rados.rados.0.smithi123.stdout:574: done (14 left) 2020-01-16T00:12:36.216 INFO:tasks.rados.rados.0.smithi123.stdout:590: set_chunk oid 41 offset: 158208 length: 2224640 target oid 41 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 158208 2020-01-16T00:12:36.216 INFO:tasks.rados.rados.0.smithi123.stdout:591: set_chunk oid 42 offset: 70656 length: 727552 target oid 42 low_tier tgt_offset: 70656 2020-01-16T00:12:36.216 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.216 INFO:tasks.rados.rados.0.smithi123.stdout:572: finishing set_chunk to oid 23 2020-01-16T00:12:36.216 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 356352 length: 737792 chunk_info offset: 294400 length 114176 2020-01-16T00:12:36.216 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 23 v 35 (ObjNum 22 snap 0 seq_num 22) dirty exists 2020-01-16T00:12:36.216 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 356352 length: 737792 chunk_info offset: 808960 length 1406976 2020-01-16T00:12:36.217 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 23 v 35 (ObjNum 22 snap 0 seq_num 22) dirty exists 2020-01-16T00:12:36.217 INFO:tasks.rados.rados.0.smithi123.stdout:572: done (15 left) 2020-01-16T00:12:36.217 INFO:tasks.rados.rados.0.smithi123.stdout:592: set_chunk oid 43 offset: 589824 length: 849920 target oid 43 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 589824 2020-01-16T00:12:36.217 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.217 INFO:tasks.rados.rados.0.smithi123.stdout:579: finishing set_chunk to oid 30 2020-01-16T00:12:36.217 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 2178560 length: 267776 chunk_info offset: 1542656 length 711168 2020-01-16T00:12:36.217 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 30 v 18 (ObjNum 29 snap 0 seq_num 29) dirty exists 2020-01-16T00:12:36.217 INFO:tasks.rados.rados.0.smithi123.stdout:579: done (15 left) 2020-01-16T00:12:36.218 INFO:tasks.rados.rados.0.smithi123.stdout:593: set_chunk oid 44 offset: 1375232 length: 972288 target oid 44 low_tier tgt_offset: 1375232 2020-01-16T00:12:36.218 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.218 INFO:tasks.rados.rados.0.smithi123.stdout:576: finishing set_chunk to oid 27 2020-01-16T00:12:36.218 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 105984 length: 337920 chunk_info offset: 113152 length 1699840 2020-01-16T00:12:36.218 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 27 v 11 (ObjNum 26 snap 0 seq_num 26) dirty exists 2020-01-16T00:12:36.218 INFO:tasks.rados.rados.0.smithi123.stdout:576: done (15 left) 2020-01-16T00:12:36.218 INFO:tasks.rados.rados.0.smithi123.stdout:594: set_chunk oid 45 offset: 857088 length: 1654784 target oid 45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 857088 2020-01-16T00:12:36.219 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.219 INFO:tasks.rados.rados.0.smithi123.stdout:582: finishing set_chunk to oid 33 2020-01-16T00:12:36.219 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 914944 length: 228864 chunk_info offset: 658944 length 1355776 2020-01-16T00:12:36.219 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 33 v 21 (ObjNum 32 snap 0 seq_num 32) dirty exists 2020-01-16T00:12:36.219 INFO:tasks.rados.rados.0.smithi123.stdout:582: done (15 left) 2020-01-16T00:12:36.219 INFO:tasks.rados.rados.0.smithi123.stdout:595: set_chunk oid 46 offset: 204288 length: 1196032 target oid 46 low_tier tgt_offset: 204288 2020-01-16T00:12:36.220 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.220 INFO:tasks.rados.rados.0.smithi123.stdout:583: finishing set_chunk to oid 34 2020-01-16T00:12:36.220 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 508928 length: 1074176 chunk_info offset: 615936 length 180224 2020-01-16T00:12:36.220 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 34 v 12 (ObjNum 33 snap 0 seq_num 33) dirty exists 2020-01-16T00:12:36.220 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 508928 length: 1074176 chunk_info offset: 901632 length 1154560 2020-01-16T00:12:36.220 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 34 v 12 (ObjNum 33 snap 0 seq_num 33) dirty exists 2020-01-16T00:12:36.220 INFO:tasks.rados.rados.0.smithi123.stdout:583: done (15 left) 2020-01-16T00:12:36.220 INFO:tasks.rados.rados.0.smithi123.stdout:596: set_chunk oid 47 offset: 1908224 length: 99328 target oid 47 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1908224 2020-01-16T00:12:36.221 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.221 INFO:tasks.rados.rados.0.smithi123.stdout:585: finishing set_chunk to oid 36 2020-01-16T00:12:36.221 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 528896 length: 355328 chunk_info offset: 643584 length 327680 2020-01-16T00:12:36.221 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 36 v 24 (ObjNum 35 snap 0 seq_num 35) dirty exists 2020-01-16T00:12:36.221 INFO:tasks.rados.rados.0.smithi123.stdout:585: done (15 left) 2020-01-16T00:12:36.221 INFO:tasks.rados.rados.0.smithi123.stdout:597: set_chunk oid 48 offset: 195584 length: 842240 target oid 48 low_tier tgt_offset: 195584 2020-01-16T00:12:36.221 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.222 INFO:tasks.rados.rados.0.smithi123.stdout:590: finishing set_chunk to oid 41 2020-01-16T00:12:36.222 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 158208 length: 2224640 chunk_info offset: 451072 length 431616 2020-01-16T00:12:36.222 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 41 v 48 (ObjNum 40 snap 0 seq_num 40) dirty exists 2020-01-16T00:12:36.222 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 158208 length: 2224640 chunk_info offset: 925184 length 293376 2020-01-16T00:12:36.222 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 41 v 48 (ObjNum 40 snap 0 seq_num 40) dirty exists 2020-01-16T00:12:36.222 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 158208 length: 2224640 chunk_info offset: 1332736 length 44544 2020-01-16T00:12:36.222 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 41 v 48 (ObjNum 40 snap 0 seq_num 40) dirty exists 2020-01-16T00:12:36.222 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 158208 length: 2224640 chunk_info offset: 1808384 length 557056 2020-01-16T00:12:36.223 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 41 v 48 (ObjNum 40 snap 0 seq_num 40) dirty exists 2020-01-16T00:12:36.223 INFO:tasks.rados.rados.0.smithi123.stdout:577: finishing set_chunk to oid 28 2020-01-16T00:12:36.223 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 28 v 50 (ObjNum 27 snap 0 seq_num 27) dirty exists 2020-01-16T00:12:36.223 INFO:tasks.rados.rados.0.smithi123.stdout:580: finishing set_chunk to oid 31 2020-01-16T00:12:36.223 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 291328 length: 2107392 chunk_info offset: 263680 length 1668096 2020-01-16T00:12:36.223 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 31 v 15 (ObjNum 30 snap 0 seq_num 30) dirty exists 2020-01-16T00:12:36.223 INFO:tasks.rados.rados.0.smithi123.stdout:581: finishing set_chunk to oid 32 2020-01-16T00:12:36.223 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 981504 length: 248832 chunk_info offset: 605696 length 877568 2020-01-16T00:12:36.224 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 32 v 18 (ObjNum 31 snap 0 seq_num 31) dirty exists 2020-01-16T00:12:36.224 INFO:tasks.rados.rados.0.smithi123.stdout:577: done (15 left) 2020-01-16T00:12:36.224 INFO:tasks.rados.rados.0.smithi123.stdout:580: done (14 left) 2020-01-16T00:12:36.224 INFO:tasks.rados.rados.0.smithi123.stdout:581: done (13 left) 2020-01-16T00:12:36.224 INFO:tasks.rados.rados.0.smithi123.stdout:590: done (12 left) 2020-01-16T00:12:36.224 INFO:tasks.rados.rados.0.smithi123.stdout:598: set_chunk oid 49 offset: 329728 length: 1223680 target oid 49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 329728 2020-01-16T00:12:36.224 INFO:tasks.rados.rados.0.smithi123.stdout:599: set_chunk oid 50 offset: 1400832 length: 1298432 target oid 50 low_tier tgt_offset: 1400832 2020-01-16T00:12:36.224 INFO:tasks.rados.rados.0.smithi123.stdout:600: set_chunk oid 1 offset: 1216000 length: 481792 target oid 1 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm tgt_offset: 1216000 2020-01-16T00:12:36.225 INFO:tasks.rados.rados.0.smithi123.stdout:601 oid_not_in_use 35 oid_redirect_not_in_use 0 2020-01-16T00:12:36.225 INFO:tasks.rados.rados.0.smithi123.stdout:600 wait for manifest initialization 2020-01-16T00:12:36.225 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 15 2020-01-16T00:12:36.225 INFO:tasks.rados.rados.0.smithi123.stdout:586: finishing set_chunk to oid 37 2020-01-16T00:12:36.225 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 717312 length: 909312 chunk_info offset: 574464 length 1014784 2020-01-16T00:12:36.225 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 37 v 58 (ObjNum 36 snap 0 seq_num 36) dirty exists 2020-01-16T00:12:36.225 INFO:tasks.rados.rados.0.smithi123.stdout:584: finishing set_chunk to oid 35 2020-01-16T00:12:36.226 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 818688 length: 510464 chunk_info offset: 926720 length 423424 2020-01-16T00:12:36.226 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 35 v 28 (ObjNum 34 snap 0 seq_num 34) dirty exists 2020-01-16T00:12:36.226 INFO:tasks.rados.rados.0.smithi123.stdout:584: done (14 left) 2020-01-16T00:12:36.226 INFO:tasks.rados.rados.0.smithi123.stdout:586: done (13 left) 2020-01-16T00:12:36.226 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 13 2020-01-16T00:12:36.226 INFO:tasks.rados.rados.0.smithi123.stdout:595: finishing set_chunk to oid 46 2020-01-16T00:12:36.226 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 204288 length: 1196032 chunk_info offset: 56832 length 1408000 2020-01-16T00:12:36.226 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 46 v 65 (ObjNum 45 snap 0 seq_num 45) dirty exists 2020-01-16T00:12:36.227 INFO:tasks.rados.rados.0.smithi123.stdout:595: done (12 left) 2020-01-16T00:12:36.227 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 12 2020-01-16T00:12:36.227 INFO:tasks.rados.rados.0.smithi123.stdout:594: finishing set_chunk to oid 45 2020-01-16T00:12:36.227 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 857088 length: 1654784 chunk_info offset: 640512 length 403968 2020-01-16T00:12:36.227 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 45 v 35 (ObjNum 44 snap 0 seq_num 44) dirty exists 2020-01-16T00:12:36.227 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 857088 length: 1654784 chunk_info offset: 1445888 length 278528 2020-01-16T00:12:36.227 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 45 v 35 (ObjNum 44 snap 0 seq_num 44) dirty exists 2020-01-16T00:12:36.228 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 857088 length: 1654784 chunk_info offset: 1887232 length 593920 2020-01-16T00:12:36.228 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 45 v 35 (ObjNum 44 snap 0 seq_num 44) dirty exists 2020-01-16T00:12:36.228 INFO:tasks.rados.rados.0.smithi123.stdout:588: finishing set_chunk to oid 39 2020-01-16T00:12:36.228 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 669696 length: 1332736 chunk_info offset: 500224 length 171008 2020-01-16T00:12:36.228 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 39 v 52 (ObjNum 38 snap 0 seq_num 38) dirty exists 2020-01-16T00:12:36.228 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 669696 length: 1332736 chunk_info offset: 763392 length 413696 2020-01-16T00:12:36.228 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 39 v 52 (ObjNum 38 snap 0 seq_num 38) dirty exists 2020-01-16T00:12:36.228 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 669696 length: 1332736 chunk_info offset: 1570304 length 12288 2020-01-16T00:12:36.229 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 39 v 52 (ObjNum 38 snap 0 seq_num 38) dirty exists 2020-01-16T00:12:36.229 INFO:tasks.rados.rados.0.smithi123.stdout:589: finishing set_chunk to oid 40 2020-01-16T00:12:36.229 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 756736 length: 307712 chunk_info offset: 790528 length 688128 2020-01-16T00:12:36.229 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 40 v 23 (ObjNum 39 snap 0 seq_num 39) dirty exists 2020-01-16T00:12:36.229 INFO:tasks.rados.rados.0.smithi123.stdout:592: finishing set_chunk to oid 43 2020-01-16T00:12:36.229 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 589824 length: 849920 chunk_info offset: 1384448 length 188416 2020-01-16T00:12:36.230 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 43 v 39 (ObjNum 42 snap 0 seq_num 42) dirty exists 2020-01-16T00:12:36.230 INFO:tasks.rados.rados.0.smithi123.stdout:587: finishing set_chunk to oid 38 2020-01-16T00:12:36.230 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 38 v 37 (ObjNum 37 snap 0 seq_num 37) dirty exists 2020-01-16T00:12:36.230 INFO:tasks.rados.rados.0.smithi123.stdout:587: done (11 left) 2020-01-16T00:12:36.230 INFO:tasks.rados.rados.0.smithi123.stdout:588: done (10 left) 2020-01-16T00:12:36.230 INFO:tasks.rados.rados.0.smithi123.stdout:589: done (9 left) 2020-01-16T00:12:36.230 INFO:tasks.rados.rados.0.smithi123.stdout:592: done (8 left) 2020-01-16T00:12:36.230 INFO:tasks.rados.rados.0.smithi123.stdout:594: done (7 left) 2020-01-16T00:12:36.231 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 7 2020-01-16T00:12:36.231 INFO:tasks.rados.rados.0.smithi123.stdout:600: finishing set_chunk to oid 1 2020-01-16T00:12:36.231 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1216000 length: 481792 chunk_info offset: 1046016 length 729088 2020-01-16T00:12:36.231 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 1 v 60 (ObjNum 0 snap 0 seq_num 0) dirty exists 2020-01-16T00:12:36.231 INFO:tasks.rados.rados.0.smithi123.stdout:600: done (6 left) 2020-01-16T00:12:36.231 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 6 2020-01-16T00:12:36.231 INFO:tasks.rados.rados.0.smithi123.stdout:598: finishing set_chunk to oid 49 2020-01-16T00:12:36.231 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 329728 length: 1223680 chunk_info offset: 453632 length 1075200 2020-01-16T00:12:36.232 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 49 v 27 (ObjNum 48 snap 0 seq_num 48) dirty exists 2020-01-16T00:12:36.232 INFO:tasks.rados.rados.0.smithi123.stdout:598: done (5 left) 2020-01-16T00:12:36.232 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 5 2020-01-16T00:12:36.232 INFO:tasks.rados.rados.0.smithi123.stdout:591: finishing set_chunk to oid 42 2020-01-16T00:12:36.232 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 42 v 38 (ObjNum 41 snap 0 seq_num 41) dirty exists 2020-01-16T00:12:36.232 INFO:tasks.rados.rados.0.smithi123.stdout:591: done (4 left) 2020-01-16T00:12:36.232 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 4 2020-01-16T00:12:36.233 INFO:tasks.rados.rados.0.smithi123.stdout:599: finishing set_chunk to oid 50 2020-01-16T00:12:36.233 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1400832 length: 1298432 chunk_info offset: 1613312 length 498688 2020-01-16T00:12:36.233 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 50 v 40 (ObjNum 49 snap 0 seq_num 49) dirty exists 2020-01-16T00:12:36.233 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1400832 length: 1298432 chunk_info offset: 2422272 length 237568 2020-01-16T00:12:36.233 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 50 v 40 (ObjNum 49 snap 0 seq_num 49) dirty exists 2020-01-16T00:12:36.233 INFO:tasks.rados.rados.0.smithi123.stdout:599: done (3 left) 2020-01-16T00:12:36.233 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 3 2020-01-16T00:12:36.233 INFO:tasks.rados.rados.0.smithi123.stdout:593: finishing set_chunk to oid 44 2020-01-16T00:12:36.234 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1375232 length: 972288 chunk_info offset: 1068032 length 349696 2020-01-16T00:12:36.234 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 44 v 55 (ObjNum 43 snap 0 seq_num 43) dirty exists 2020-01-16T00:12:36.234 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1375232 length: 972288 chunk_info offset: 1635840 length 423424 2020-01-16T00:12:36.234 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 44 v 55 (ObjNum 43 snap 0 seq_num 43) dirty exists 2020-01-16T00:12:36.234 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 1375232 length: 972288 chunk_info offset: 2134528 length 413696 2020-01-16T00:12:36.234 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 44 v 55 (ObjNum 43 snap 0 seq_num 43) dirty exists 2020-01-16T00:12:36.234 INFO:tasks.rados.rados.0.smithi123.stdout:597: finishing set_chunk to oid 48 2020-01-16T00:12:36.234 INFO:tasks.rados.rados.0.smithi123.stdout: range is overlapped offset: 195584 length: 842240 chunk_info offset: 264704 length 1399296 2020-01-16T00:12:36.235 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 48 v 21 (ObjNum 47 snap 0 seq_num 47) dirty exists 2020-01-16T00:12:36.235 INFO:tasks.rados.rados.0.smithi123.stdout:593: done (2 left) 2020-01-16T00:12:36.235 INFO:tasks.rados.rados.0.smithi123.stdout:597: done (1 left) 2020-01-16T00:12:36.235 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 1 2020-01-16T00:12:36.235 INFO:tasks.rados.rados.0.smithi123.stdout:596: finishing set_chunk to oid 47 2020-01-16T00:12:36.235 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 47 v 65 (ObjNum 46 snap 0 seq_num 46) dirty exists 2020-01-16T00:12:36.235 INFO:tasks.rados.rados.0.smithi123.stdout:596: done (0 left) 2020-01-16T00:12:36.236 INFO:tasks.rados.rados.0.smithi123.stdout:601 oid_not_in_use 50 oid_redirect_not_in_use 0 2020-01-16T00:12:36.236 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 51 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:36.236 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 52 low_tier 2020-01-16T00:12:36.236 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 53 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:36.236 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 54 low_tier 2020-01-16T00:12:36.236 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 55 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:36.236 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 56 low_tier 2020-01-16T00:12:36.236 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 57 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:36.237 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 58 low_tier 2020-01-16T00:12:36.237 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 59 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:36.237 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 60 low_tier 2020-01-16T00:12:36.237 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 61 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:36.237 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 62 low_tier 2020-01-16T00:12:36.237 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 63 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:36.237 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 64 low_tier 2020-01-16T00:12:36.237 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 65 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:36.238 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 66 low_tier 2020-01-16T00:12:36.238 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 67 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:36.238 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 68 low_tier 2020-01-16T00:12:36.238 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 69 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:36.238 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 70 low_tier 2020-01-16T00:12:36.238 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 71 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:36.238 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 72 low_tier 2020-01-16T00:12:36.239 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 73 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:36.239 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 74 low_tier 2020-01-16T00:12:36.239 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 75 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:36.239 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 76 low_tier 2020-01-16T00:12:36.239 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 77 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:36.239 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 78 low_tier 2020-01-16T00:12:36.239 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 79 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:36.240 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 80 low_tier 2020-01-16T00:12:36.240 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 81 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:36.240 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 82 low_tier 2020-01-16T00:12:36.240 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 83 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:36.240 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 84 low_tier 2020-01-16T00:12:36.240 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 85 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:36.241 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 86 low_tier 2020-01-16T00:12:36.241 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 87 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:36.241 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 88 low_tier 2020-01-16T00:12:36.241 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 89 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:36.241 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 90 low_tier 2020-01-16T00:12:36.241 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 91 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:36.241 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 92 low_tier 2020-01-16T00:12:36.242 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 93 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:36.242 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 94 low_tier 2020-01-16T00:12:36.242 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 95 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:36.242 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 96 low_tier 2020-01-16T00:12:36.242 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 97 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:36.242 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 98 low_tier 2020-01-16T00:12:36.242 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 99 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm 2020-01-16T00:12:36.242 INFO:tasks.rados.rados.0.smithi123.stdout: redirect_not_in_use: 100 low_tier 2020-01-16T00:12:36.243 INFO:tasks.rados.rados.0.smithi123.stdout:601: read oid 35 snap -1 2020-01-16T00:12:36.243 INFO:tasks.rados.rados.0.smithi123.stdout:601: expect (ObjNum 34 snap 0 seq_num 34) 2020-01-16T00:12:36.243 INFO:tasks.rados.rados.0.smithi123.stdout:602: read oid 3 snap -1 2020-01-16T00:12:36.243 INFO:tasks.rados.rados.0.smithi123.stdout:602: expect (ObjNum 2 snap 0 seq_num 2) 2020-01-16T00:12:36.243 INFO:tasks.rados.rados.0.smithi123.stdout:603: read oid 22 snap -1 2020-01-16T00:12:36.243 INFO:tasks.rados.rados.0.smithi123.stdout:603: expect (ObjNum 21 snap 0 seq_num 21) 2020-01-16T00:12:36.243 INFO:tasks.rados.rados.0.smithi123.stdout:604: read oid 7 snap -1 2020-01-16T00:12:36.243 INFO:tasks.rados.rados.0.smithi123.stdout:604: expect (ObjNum 6 snap 0 seq_num 6) 2020-01-16T00:12:36.244 INFO:tasks.rados.rados.0.smithi123.stdout:605: write oid 39 current snap is 0 2020-01-16T00:12:36.244 INFO:tasks.rados.rados.0.smithi123.stdout:605: seq_num 50 ranges {440350=460568,1309629=542666,2323973=516377} 2020-01-16T00:12:36.244 INFO:tasks.rados.rados.0.smithi123.stdout:605: writing smithi12313934-39 from 440350 to 900918 tid 1 2020-01-16T00:12:36.244 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2020-01-16T00:12:36.247 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:12:36.245+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:12:36.247 INFO:tasks.rados.rados.0.smithi123.stdout:605: writing smithi12313934-39 from 1309629 to 1852295 tid 2 2020-01-16T00:12:36.248 INFO:tasks.rados.rados.0.smithi123.stdout:605: writing smithi12313934-39 from 2323973 to 2840350 tid 3 2020-01-16T00:12:36.248 INFO:tasks.rados.rados.0.smithi123.stdout:606: delete oid 8 current snap is 0 2020-01-16T00:12:36.248 INFO:tasks.rados.rados.0.smithi123.stdout:601: expect (ObjNum 34 snap 0 seq_num 34) 2020-01-16T00:12:36.248 INFO:tasks.rados.rados.0.smithi123.stdout:602: expect (ObjNum 2 snap 0 seq_num 2) 2020-01-16T00:12:36.248 INFO:tasks.rados.rados.0.smithi123.stdout:605: finishing write tid 1 to smithi12313934-39 2020-01-16T00:12:36.248 INFO:tasks.rados.rados.0.smithi123.stdout:605: finishing write tid 2 to smithi12313934-39 2020-01-16T00:12:36.248 INFO:tasks.rados.rados.0.smithi123.stdout:605: finishing write tid 3 to smithi12313934-39 2020-01-16T00:12:36.248 INFO:tasks.rados.rados.0.smithi123.stdout:603: expect (ObjNum 21 snap 0 seq_num 21) 2020-01-16T00:12:36.249 INFO:tasks.rados.rados.0.smithi123.stdout:605: finishing write tid 5 to smithi12313934-39 2020-01-16T00:12:36.249 INFO:tasks.rados.rados.0.smithi123.stdout:604: expect (ObjNum 6 snap 0 seq_num 6) 2020-01-16T00:12:36.249 INFO:tasks.rados.rados.0.smithi123.stdout:605: finishing write tid 6 to smithi12313934-39 2020-01-16T00:12:36.249 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 39 v 70 (ObjNum 50 snap 0 seq_num 50) dirty exists 2020-01-16T00:12:36.249 INFO:tasks.rados.rados.0.smithi123.stdout:605: left oid 39 (ObjNum 50 snap 0 seq_num 50) 2020-01-16T00:12:36.249 INFO:tasks.rados.rados.0.smithi123.stdout:601: done (5 left) 2020-01-16T00:12:36.249 INFO:tasks.rados.rados.0.smithi123.stdout:602: done (4 left) 2020-01-16T00:12:36.250 INFO:tasks.rados.rados.0.smithi123.stdout:603: done (3 left) 2020-01-16T00:12:36.250 INFO:tasks.rados.rados.0.smithi123.stdout:604: done (2 left) 2020-01-16T00:12:36.250 INFO:tasks.rados.rados.0.smithi123.stdout:605: done (1 left) 2020-01-16T00:12:36.250 INFO:tasks.rados.rados.0.smithi123.stdout:606: done (0 left) 2020-01-16T00:12:36.250 INFO:tasks.rados.rados.0.smithi123.stdout:607: read oid 41 snap -1 2020-01-16T00:12:36.251 INFO:tasks.rados.rados.0.smithi123.stdout:607: expect (ObjNum 40 snap 0 seq_num 40) 2020-01-16T00:12:36.251 INFO:tasks.rados.rados.0.smithi123.stdout:608: write oid 38 current snap is 0 2020-01-16T00:12:36.251 INFO:tasks.rados.rados.0.smithi123.stdout:608: seq_num 52 ranges {473964=611066,1826593=546335,2778819=95145} 2020-01-16T00:12:36.251 INFO:tasks.rados.rados.0.smithi123.stdout:608: writing smithi12313934-38 from 473964 to 1085030 tid 1 2020-01-16T00:12:36.251 INFO:tasks.rados.rados.0.smithi123.stdout:608: writing smithi12313934-38 from 1826593 to 2372928 tid 2 2020-01-16T00:12:36.251 INFO:tasks.rados.rados.0.smithi123.stdout:608: writing smithi12313934-38 from 2778819 to 2873964 tid 3 2020-01-16T00:12:36.251 INFO:tasks.rados.rados.0.smithi123.stdout:609: read oid 13 snap -1 2020-01-16T00:12:36.251 INFO:tasks.rados.rados.0.smithi123.stdout:609: expect (ObjNum 12 snap 0 seq_num 12) 2020-01-16T00:12:36.252 INFO:tasks.rados.rados.0.smithi123.stdout:610: write oid 2 current snap is 0 2020-01-16T00:12:36.252 INFO:tasks.rados.rados.0.smithi123.stdout:607: expect (ObjNum 40 snap 0 seq_num 40) 2020-01-16T00:12:36.252 INFO:tasks.rados.rados.0.smithi123.stdout:608: finishing write tid 1 to smithi12313934-38 2020-01-16T00:12:36.252 INFO:tasks.rados.rados.0.smithi123.stdout:608: finishing write tid 2 to smithi12313934-38 2020-01-16T00:12:36.252 INFO:tasks.rados.rados.0.smithi123.stdout:609: expect (ObjNum 12 snap 0 seq_num 12) 2020-01-16T00:12:36.306 INFO:tasks.rados.rados.0.smithi123.stdout:608: finishing write tid 3 to smithi12313934-38 2020-01-16T00:12:36.306 INFO:tasks.rados.rados.0.smithi123.stdout:608: finishing write tid 5 to smithi12313934-38 2020-01-16T00:12:36.307 INFO:tasks.rados.rados.0.smithi123.stdout:608: finishing write tid 6 to smithi12313934-38 2020-01-16T00:12:36.307 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 38 v 45 (ObjNum 52 snap 0 seq_num 52) dirty exists 2020-01-16T00:12:36.307 INFO:tasks.rados.rados.0.smithi123.stdout:608: left oid 38 (ObjNum 52 snap 0 seq_num 52) 2020-01-16T00:12:36.307 INFO:tasks.rados.rados.0.smithi123.stdout:610: seq_num 53 ranges {490771=686315,1885075=406346} 2020-01-16T00:12:36.314 INFO:tasks.rados.rados.0.smithi123.stdout:610: writing smithi12313934-2 from 490771 to 1177086 tid 1 2020-01-16T00:12:36.321 INFO:tasks.rados.rados.0.smithi123.stdout:610: writing smithi12313934-2 from 1885075 to 2291421 tid 2 2020-01-16T00:12:36.321 INFO:tasks.rados.rados.0.smithi123.stdout:607: done (3 left) 2020-01-16T00:12:36.321 INFO:tasks.rados.rados.0.smithi123.stdout:608: done (2 left) 2020-01-16T00:12:36.321 INFO:tasks.rados.rados.0.smithi123.stdout:609: done (1 left) 2020-01-16T00:12:36.321 INFO:tasks.rados.rados.0.smithi123.stdout:611: write oid 33 current snap is 0 2020-01-16T00:12:36.322 INFO:tasks.rados.rados.0.smithi123.stdout:611: seq_num 54 ranges {507578=477917,1543557=550004,2833665=73913} 2020-01-16T00:12:36.327 INFO:tasks.rados.rados.0.smithi123.stdout:611: writing smithi12313934-33 from 507578 to 985495 tid 1 2020-01-16T00:12:36.334 INFO:tasks.rados.rados.0.smithi123.stdout:611: writing smithi12313934-33 from 1543557 to 2093561 tid 2 2020-01-16T00:12:36.339 INFO:tasks.rados.rados.0.smithi123.stdout:611: writing smithi12313934-33 from 2833665 to 2907578 tid 3 2020-01-16T00:12:36.340 INFO:tasks.rados.rados.0.smithi123.stdout:612: read oid 15 snap -1 2020-01-16T00:12:36.340 INFO:tasks.rados.rados.0.smithi123.stdout:612: expect (ObjNum 14 snap 0 seq_num 14) 2020-01-16T00:12:36.340 INFO:tasks.rados.rados.0.smithi123.stdout:613: delete oid 8 current snap is 0 2020-01-16T00:12:36.340 INFO:tasks.rados.rados.0.smithi123.stdout:610: finishing write tid 1 to smithi12313934-2 2020-01-16T00:12:36.340 INFO:tasks.rados.rados.0.smithi123.stdout:610: finishing write tid 2 to smithi12313934-2 2020-01-16T00:12:36.341 INFO:tasks.rados.rados.0.smithi123.stdout:610: finishing write tid 4 to smithi12313934-2 2020-01-16T00:12:36.341 INFO:tasks.rados.rados.0.smithi123.stdout:610: finishing write tid 5 to smithi12313934-2 2020-01-16T00:12:36.341 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 2 v 59 (ObjNum 53 snap 0 seq_num 53) dirty exists 2020-01-16T00:12:36.341 INFO:tasks.rados.rados.0.smithi123.stdout:610: left oid 2 (ObjNum 53 snap 0 seq_num 53) 2020-01-16T00:12:36.342 INFO:tasks.rados.rados.0.smithi123.stdout:610: done (3 left) 2020-01-16T00:12:36.342 INFO:tasks.rados.rados.0.smithi123.stdout:613: done (2 left) 2020-01-16T00:12:36.342 INFO:tasks.rados.rados.0.smithi123.stdout:614: write (excl) oid 50 current snap is 0 2020-01-16T00:12:36.342 INFO:tasks.rados.rados.0.smithi123.stdout:614: seq_num 56 ranges {541192=628415,1660521=553673,2888511=52681} 2020-01-16T00:12:36.346 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:12:36.347+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:12:36.349 INFO:tasks.rados.rados.0.smithi123.stdout:614: writing smithi12313934-50 from 541192 to 1169607 tid 1 2020-01-16T00:12:36.356 INFO:tasks.rados.rados.0.smithi123.stdout:614: writing smithi12313934-50 from 1660521 to 2214194 tid 2 2020-01-16T00:12:36.361 INFO:tasks.rados.rados.0.smithi123.stdout:614: writing smithi12313934-50 from 2888511 to 2941192 tid 3 2020-01-16T00:12:36.361 INFO:tasks.rados.rados.0.smithi123.stdout:615: delete oid 9 current snap is 0 2020-01-16T00:12:36.361 INFO:tasks.rados.rados.0.smithi123.stdout:611: finishing write tid 1 to smithi12313934-33 2020-01-16T00:12:36.362 INFO:tasks.rados.rados.0.smithi123.stdout:611: finishing write tid 2 to smithi12313934-33 2020-01-16T00:12:36.362 INFO:tasks.rados.rados.0.smithi123.stdout:611: finishing write tid 3 to smithi12313934-33 2020-01-16T00:12:36.362 INFO:tasks.rados.rados.0.smithi123.stdout:611: finishing write tid 5 to smithi12313934-33 2020-01-16T00:12:36.362 INFO:tasks.rados.rados.0.smithi123.stdout:611: finishing write tid 6 to smithi12313934-33 2020-01-16T00:12:36.362 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 33 v 85 (ObjNum 54 snap 0 seq_num 54) dirty exists 2020-01-16T00:12:36.362 INFO:tasks.rados.rados.0.smithi123.stdout:611: left oid 33 (ObjNum 54 snap 0 seq_num 54) 2020-01-16T00:12:36.364 INFO:tasks.rados.rados.0.smithi123.stdout:612: expect (ObjNum 14 snap 0 seq_num 14) 2020-01-16T00:12:36.375 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:36.385 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-01-16T00:12:36.437 INFO:tasks.rados.rados.0.smithi123.stdout:614: finishing write tid 1 to smithi12313934-50 2020-01-16T00:12:36.438 INFO:tasks.rados.rados.0.smithi123.stdout:614: finishing write tid 2 to smithi12313934-50 2020-01-16T00:12:36.438 INFO:tasks.rados.rados.0.smithi123.stdout:614: finishing write tid 3 to smithi12313934-50 2020-01-16T00:12:36.438 INFO:tasks.rados.rados.0.smithi123.stdout:614: finishing write tid 5 to smithi12313934-50 2020-01-16T00:12:36.438 INFO:tasks.rados.rados.0.smithi123.stdout:614: finishing write tid 6 to smithi12313934-50 2020-01-16T00:12:36.438 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 50 v 58 (ObjNum 56 snap 0 seq_num 56) dirty exists 2020-01-16T00:12:36.438 INFO:tasks.rados.rados.0.smithi123.stdout:614: left oid 50 (ObjNum 56 snap 0 seq_num 56) 2020-01-16T00:12:36.438 INFO:tasks.rados.rados.0.smithi123.stdout:611: done (3 left) 2020-01-16T00:12:36.439 INFO:tasks.rados.rados.0.smithi123.stdout:612: done (2 left) 2020-01-16T00:12:36.439 INFO:tasks.rados.rados.0.smithi123.stdout:614: done (1 left) 2020-01-16T00:12:36.439 INFO:tasks.rados.rados.0.smithi123.stdout:615: done (0 left) 2020-01-16T00:12:36.439 INFO:tasks.rados.rados.0.smithi123.stdout:616: read oid 20 snap -1 2020-01-16T00:12:36.439 INFO:tasks.rados.rados.0.smithi123.stdout:616: expect (ObjNum 19 snap 0 seq_num 19) 2020-01-16T00:12:36.439 INFO:tasks.rados.rados.0.smithi123.stdout:617: write (excl) oid 48 current snap is 0 2020-01-16T00:12:36.439 INFO:tasks.rados.rados.0.smithi123.stdout:617: seq_num 58 ranges {574806=778913,2061132=557342} 2020-01-16T00:12:36.447 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:12:36.445+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:12:36.454 INFO:tasks.rados.rados.0.smithi123.stdout:617: writing smithi12313934-48 from 574806 to 1353719 tid 1 2020-01-16T00:12:36.462 INFO:tasks.rados.rados.0.smithi123.stdout:617: writing smithi12313934-48 from 2061132 to 2618474 tid 2 2020-01-16T00:12:36.462 INFO:tasks.rados.rados.0.smithi123.stdout:618: read oid 1 snap -1 2020-01-16T00:12:36.462 INFO:tasks.rados.rados.0.smithi123.stdout:618: expect (ObjNum 0 snap 0 seq_num 0) 2020-01-16T00:12:36.462 INFO:tasks.rados.rados.0.smithi123.stdout:619: write (excl) oid 15 current snap is 0 2020-01-16T00:12:36.462 INFO:tasks.rados.rados.0.smithi123.stdout:619: seq_num 59 ranges {591613=454162,1719614=417353,2570780=420833} 2020-01-16T00:12:36.468 INFO:tasks.rados.rados.0.smithi123.stdout:619: writing smithi12313934-15 from 591613 to 1045775 tid 1 2020-01-16T00:12:36.474 INFO:tasks.rados.rados.0.smithi123.stdout:619: writing smithi12313934-15 from 1719614 to 2136967 tid 2 2020-01-16T00:12:36.481 INFO:tasks.rados.rados.0.smithi123.stdout:619: writing smithi12313934-15 from 2570780 to 2991613 tid 3 2020-01-16T00:12:36.481 INFO:tasks.rados.rados.0.smithi123.stdout:620: read oid 21 snap -1 2020-01-16T00:12:36.481 INFO:tasks.rados.rados.0.smithi123.stdout:620: expect (ObjNum 20 snap 0 seq_num 20) 2020-01-16T00:12:36.481 INFO:tasks.rados.rados.0.smithi123.stdout:616: expect (ObjNum 19 snap 0 seq_num 19) 2020-01-16T00:12:36.494 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:36.506 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-01-16T00:12:36.548 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:12:36.549+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:12:36.557 INFO:tasks.rados.rados.0.smithi123.stdout:619: finishing write tid 1 to smithi12313934-15 2020-01-16T00:12:36.558 INFO:tasks.rados.rados.0.smithi123.stdout:617: finishing write tid 1 to smithi12313934-48 2020-01-16T00:12:36.558 INFO:tasks.rados.rados.0.smithi123.stdout:619: finishing write tid 2 to smithi12313934-15 2020-01-16T00:12:36.558 INFO:tasks.rados.rados.0.smithi123.stdout:618: expect (ObjNum 0 snap 0 seq_num 0) 2020-01-16T00:12:36.618 INFO:tasks.rados.rados.0.smithi123.stdout:619: finishing write tid 3 to smithi12313934-15 2020-01-16T00:12:36.619 INFO:tasks.rados.rados.0.smithi123.stdout:619: finishing write tid 5 to smithi12313934-15 2020-01-16T00:12:36.619 INFO:tasks.rados.rados.0.smithi123.stdout:619: finishing write tid 6 to smithi12313934-15 2020-01-16T00:12:36.619 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 15 v 62 (ObjNum 59 snap 0 seq_num 59) dirty exists 2020-01-16T00:12:36.619 INFO:tasks.rados.rados.0.smithi123.stdout:619: left oid 15 (ObjNum 59 snap 0 seq_num 59) 2020-01-16T00:12:36.619 INFO:tasks.rados.rados.0.smithi123.stdout:617: finishing write tid 2 to smithi12313934-48 2020-01-16T00:12:36.619 INFO:tasks.rados.rados.0.smithi123.stdout:617: finishing write tid 4 to smithi12313934-48 2020-01-16T00:12:36.619 INFO:tasks.rados.rados.0.smithi123.stdout:617: finishing write tid 5 to smithi12313934-48 2020-01-16T00:12:36.620 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 48 v 50 (ObjNum 58 snap 0 seq_num 58) dirty exists 2020-01-16T00:12:36.620 INFO:tasks.rados.rados.0.smithi123.stdout:617: left oid 48 (ObjNum 58 snap 0 seq_num 58) 2020-01-16T00:12:36.620 INFO:tasks.rados.rados.0.smithi123.stdout:620: expect (ObjNum 20 snap 0 seq_num 20) 2020-01-16T00:12:36.648 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:12:36.651+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:12:36.655 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:36.665 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-01-16T00:12:36.693 INFO:tasks.rados.rados.0.smithi123.stdout:621: read oid 19 snap -1 2020-01-16T00:12:36.694 INFO:tasks.rados.rados.0.smithi123.stdout:621: expect (ObjNum 18 snap 0 seq_num 18) 2020-01-16T00:12:36.694 INFO:tasks.rados.rados.0.smithi123.stdout:616: done (5 left) 2020-01-16T00:12:36.694 INFO:tasks.rados.rados.0.smithi123.stdout:617: done (4 left) 2020-01-16T00:12:36.694 INFO:tasks.rados.rados.0.smithi123.stdout:618: done (3 left) 2020-01-16T00:12:36.694 INFO:tasks.rados.rados.0.smithi123.stdout:619: done (2 left) 2020-01-16T00:12:36.694 INFO:tasks.rados.rados.0.smithi123.stdout:620: done (1 left) 2020-01-16T00:12:36.695 INFO:tasks.rados.rados.0.smithi123.stdout:622: write (excl) oid 16 current snap is 0 2020-01-16T00:12:36.695 INFO:tasks.rados.rados.0.smithi123.stdout:622: seq_num 60 ranges {608420=529411,1778096=561011,2881850=126570} 2020-01-16T00:12:36.708 INFO:tasks.rados.rados.0.smithi123.stdout:622: writing smithi12313934-16 from 608420 to 1137831 tid 1 2020-01-16T00:12:36.716 INFO:tasks.rados.rados.0.smithi123.stdout:622: writing smithi12313934-16 from 1778096 to 2339107 tid 2 2020-01-16T00:12:36.718 INFO:tasks.rados.rados.0.smithi123.stdout:622: writing smithi12313934-16 from 2881850 to 3008420 tid 3 2020-01-16T00:12:36.719 INFO:tasks.rados.rados.0.smithi123.stdout:623: write oid 27 current snap is 0 2020-01-16T00:12:36.719 INFO:tasks.rados.rados.0.smithi123.stdout:623: seq_num 61 ranges {625227=721013,1952931=704669} 2020-01-16T00:12:36.726 INFO:tasks.rados.rados.0.smithi123.stdout:623: writing smithi12313934-27 from 625227 to 1346240 tid 1 2020-01-16T00:12:36.736 INFO:tasks.rados.rados.0.smithi123.stdout:623: writing smithi12313934-27 from 1952931 to 2657600 tid 2 2020-01-16T00:12:36.736 INFO:tasks.rados.rados.0.smithi123.stdout:624: write (excl) oid 24 current snap is 0 2020-01-16T00:12:36.736 INFO:tasks.rados.rados.0.smithi123.stdout:624: seq_num 62 ranges {642034=796262,1895060=564680} 2020-01-16T00:12:36.743 INFO:tasks.rados.rados.0.smithi123.stdout:624: writing smithi12313934-24 from 642034 to 1438296 tid 1 2020-01-16T00:12:36.749 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-01-16T00:12:36.750 INFO:tasks.rados.rados.0.smithi123.stdout:624: writing smithi12313934-24 from 1895060 to 2459740 tid 2 2020-01-16T00:12:36.751 INFO:tasks.rados.rados.0.smithi123.stdout:625: read oid 9 snap -1 2020-01-16T00:12:36.751 INFO:tasks.rados.rados.0.smithi123.stdout:625: expect deleted 2020-01-16T00:12:36.751 INFO:tasks.rados.rados.0.smithi123.stdout:626: write oid 2 current snap is 0 2020-01-16T00:12:36.751 INFO:tasks.rados.rados.0.smithi123.stdout:626: seq_num 63 ranges {658841=471511,1553542=708338,2964119=94722} 2020-01-16T00:12:36.751 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:36.751+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:12:36.757 INFO:tasks.rados.rados.0.smithi123.stdout:626: writing smithi12313934-2 from 658841 to 1130352 tid 1 2020-01-16T00:12:36.764 INFO:tasks.rados.rados.0.smithi123.stdout:626: writing smithi12313934-2 from 1553542 to 2261880 tid 2 2020-01-16T00:12:36.770 INFO:tasks.rados.rados.0.smithi123.stdout:626: writing smithi12313934-2 from 2964119 to 3058841 tid 3 2020-01-16T00:12:36.770 INFO:tasks.rados.rados.0.smithi123.stdout:627: write (excl) oid 8 current snap is 0 2020-01-16T00:12:36.771 INFO:tasks.rados.rados.0.smithi123.stdout:627: seq_num 64 ranges {675648=546760,2012024=568349} 2020-01-16T00:12:36.778 INFO:tasks.rados.rados.0.smithi123.stdout:627: writing smithi12313934-8 from 675648 to 1222408 tid 1 2020-01-16T00:12:36.783 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:36.790 INFO:tasks.rados.rados.0.smithi123.stdout:627: writing smithi12313934-8 from 2012024 to 2580373 tid 2 2020-01-16T00:12:36.791 INFO:tasks.rados.rados.0.smithi123.stdout:628: read oid 47 snap -1 2020-01-16T00:12:36.791 INFO:tasks.rados.rados.0.smithi123.stdout:628: expect (ObjNum 46 snap 0 seq_num 46) 2020-01-16T00:12:36.791 INFO:tasks.rados.rados.0.smithi123.stdout:629: read oid 5 snap -1 2020-01-16T00:12:36.791 INFO:tasks.rados.rados.0.smithi123.stdout:629: expect (ObjNum 4 snap 0 seq_num 4) 2020-01-16T00:12:36.791 INFO:tasks.rados.rados.0.smithi123.stdout:630: read oid 7 snap -1 2020-01-16T00:12:36.791 INFO:tasks.rados.rados.0.smithi123.stdout:630: expect (ObjNum 6 snap 0 seq_num 6) 2020-01-16T00:12:36.791 INFO:tasks.rados.rados.0.smithi123.stdout:631: write oid 30 current snap is 0 2020-01-16T00:12:36.792 INFO:tasks.rados.rados.0.smithi123.stdout:631: seq_num 65 ranges {692455=622009,2070506=712007} 2020-01-16T00:12:36.796 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-01-16T00:12:36.796 INFO:tasks.rados.rados.0.smithi123.stdout:631: writing smithi12313934-30 from 692455 to 1314464 tid 1 2020-01-16T00:12:36.806 INFO:tasks.rados.rados.0.smithi123.stdout:631: writing smithi12313934-30 from 2070506 to 2782513 tid 2 2020-01-16T00:12:36.806 INFO:tasks.rados.rados.0.smithi123.stdout:632: read oid 34 snap -1 2020-01-16T00:12:36.806 INFO:tasks.rados.rados.0.smithi123.stdout:632: expect (ObjNum 33 snap 0 seq_num 33) 2020-01-16T00:12:36.806 INFO:tasks.rados.rados.0.smithi123.stdout:633: write oid 1 current snap is 0 2020-01-16T00:12:36.806 INFO:tasks.rados.rados.0.smithi123.stdout:633: seq_num 66 ranges {709262=697258,2012635=572018,3046388=62874} 2020-01-16T00:12:36.814 INFO:tasks.rados.rados.0.smithi123.stdout:633: writing smithi12313934-1 from 709262 to 1406520 tid 1 2020-01-16T00:12:36.841 INFO:tasks.rados.rados.0.smithi123.stdout:633: writing smithi12313934-1 from 2012635 to 2584653 tid 2 2020-01-16T00:12:36.841 INFO:tasks.rados.rados.0.smithi123.stdout:633: writing smithi12313934-1 from 3046388 to 3109262 tid 3 2020-01-16T00:12:36.841 INFO:tasks.rados.rados.0.smithi123.stdout:634: write oid 50 current snap is 0 2020-01-16T00:12:36.842 INFO:tasks.rados.rados.0.smithi123.stdout:634: seq_num 67 ranges {726069=772507,2071117=715676} 2020-01-16T00:12:36.842 INFO:tasks.rados.rados.0.smithi123.stdout:634: writing smithi12313934-50 from 726069 to 1498576 tid 1 2020-01-16T00:12:36.860 INFO:tasks.rados.rados.0.smithi123.stdout:634: writing smithi12313934-50 from 2071117 to 2786793 tid 2 2020-01-16T00:12:36.861 INFO:tasks.rados.rados.0.smithi123.stdout:635: read oid 37 snap -1 2020-01-16T00:12:36.861 INFO:tasks.rados.rados.0.smithi123.stdout:635: expect (ObjNum 36 snap 0 seq_num 36) 2020-01-16T00:12:36.861 INFO:tasks.rados.rados.0.smithi123.stdout:636: read oid 23 snap -1 2020-01-16T00:12:36.861 INFO:tasks.rados.rados.0.smithi123.stdout:636: expect (ObjNum 22 snap 0 seq_num 22) 2020-01-16T00:12:36.862 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.862 INFO:tasks.rados.rados.0.smithi123.stdout:621: expect (ObjNum 18 snap 0 seq_num 18) 2020-01-16T00:12:36.862 INFO:teuthology.orchestra.run.smithi123.stderr:noscrub is unset 2020-01-16T00:12:36.863 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:36.861+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:12:36.901 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:36.911 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-01-16T00:12:36.926 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:12:36.926 INFO:tasks.rados.rados.0.smithi123.stdout:622: finishing write tid 1 to smithi12313934-16 2020-01-16T00:12:36.926 INFO:tasks.rados.rados.0.smithi123.stdout:622: finishing write tid 2 to smithi12313934-16 2020-01-16T00:12:36.932 INFO:tasks.rados.rados.0.smithi123.stdout:622: finishing write tid 3 to smithi12313934-16 2020-01-16T00:12:36.932 INFO:tasks.rados.rados.0.smithi123.stdout:622: finishing write tid 5 to smithi12313934-16 2020-01-16T00:12:36.932 INFO:tasks.rados.rados.0.smithi123.stdout:622: finishing write tid 6 to smithi12313934-16 2020-01-16T00:12:36.932 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 16 v 71 (ObjNum 60 snap 0 seq_num 60) dirty exists 2020-01-16T00:12:36.932 INFO:tasks.rados.rados.0.smithi123.stdout:622: left oid 16 (ObjNum 60 snap 0 seq_num 60) 2020-01-16T00:12:36.932 INFO:tasks.rados.rados.0.smithi123.stdout:623: finishing write tid 1 to smithi12313934-27 2020-01-16T00:12:36.933 INFO:tasks.rados.rados.0.smithi123.stdout:624: finishing write tid 1 to smithi12313934-24 2020-01-16T00:12:36.933 INFO:tasks.rados.rados.0.smithi123.stdout:623: finishing write tid 2 to smithi12313934-27 2020-01-16T00:12:36.933 INFO:tasks.rados.rados.0.smithi123.stdout:623: finishing write tid 4 to smithi12313934-27 2020-01-16T00:12:36.933 INFO:tasks.rados.rados.0.smithi123.stdout:623: finishing write tid 5 to smithi12313934-27 2020-01-16T00:12:36.933 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 27 v 46 (ObjNum 61 snap 0 seq_num 61) dirty exists 2020-01-16T00:12:36.933 INFO:tasks.rados.rados.0.smithi123.stdout:623: left oid 27 (ObjNum 61 snap 0 seq_num 61) 2020-01-16T00:12:36.933 INFO:tasks.rados.rados.0.smithi123.stdout:624: finishing write tid 2 to smithi12313934-24 2020-01-16T00:12:36.936 INFO:tasks.rados.rados.0.smithi123.stdout:621: done (15 left) 2020-01-16T00:12:36.936 INFO:tasks.rados.rados.0.smithi123.stdout:622: done (14 left) 2020-01-16T00:12:36.936 INFO:tasks.rados.rados.0.smithi123.stdout:623: done (13 left) 2020-01-16T00:12:36.936 INFO:tasks.rados.rados.0.smithi123.stdout:625: done (12 left) 2020-01-16T00:12:36.936 INFO:tasks.rados.rados.0.smithi123.stdout:637: read oid 40 snap -1 2020-01-16T00:12:36.937 INFO:tasks.rados.rados.0.smithi123.stdout:637: expect (ObjNum 39 snap 0 seq_num 39) 2020-01-16T00:12:36.937 INFO:tasks.rados.rados.0.smithi123.stdout:638: read oid 27 snap -1 2020-01-16T00:12:36.937 INFO:tasks.rados.rados.0.smithi123.stdout:638: expect (ObjNum 61 snap 0 seq_num 61) 2020-01-16T00:12:36.937 INFO:tasks.rados.rados.0.smithi123.stdout:639: read oid 38 snap -1 2020-01-16T00:12:36.937 INFO:tasks.rados.rados.0.smithi123.stdout:639: expect (ObjNum 52 snap 0 seq_num 52) 2020-01-16T00:12:36.938 INFO:tasks.rados.rados.0.smithi123.stdout:640: write oid 46 current snap is 0 2020-01-16T00:12:36.938 INFO:tasks.rados.rados.0.smithi123.stdout:640: seq_num 68 ranges {742876=447756,1729599=575687,3101234=41642} 2020-01-16T00:12:36.938 INFO:tasks.rados.rados.0.smithi123.stdout:640: writing smithi12313934-46 from 742876 to 1190632 tid 1 2020-01-16T00:12:36.939 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:12:36.943 INFO:tasks.rados.rados.0.smithi123.stdout:640: writing smithi12313934-46 from 1729599 to 2305286 tid 2 2020-01-16T00:12:36.963 INFO:tasks.rados.rados.0.smithi123.stdout:640: writing smithi12313934-46 from 3101234 to 3142876 tid 3 2020-01-16T00:12:36.964 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:36.964 INFO:tasks.rados.rados.0.smithi123.stdout:626: finishing write tid 1 to smithi12313934-2 2020-01-16T00:12:36.964 INFO:tasks.rados.rados.0.smithi123.stdout:624: finishing write tid 4 to smithi12313934-24 2020-01-16T00:12:36.964 INFO:tasks.rados.rados.0.smithi123.stdout:624: finishing write tid 5 to smithi12313934-24 2020-01-16T00:12:36.965 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 24 v 44 (ObjNum 62 snap 0 seq_num 62) dirty exists 2020-01-16T00:12:36.965 INFO:tasks.rados.rados.0.smithi123.stdout:624: left oid 24 (ObjNum 62 snap 0 seq_num 62) 2020-01-16T00:12:36.965 INFO:tasks.rados.rados.0.smithi123.stdout:626: finishing write tid 2 to smithi12313934-2 2020-01-16T00:12:36.965 INFO:tasks.rados.rados.0.smithi123.stdout:627: finishing write tid 1 to smithi12313934-8 2020-01-16T00:12:36.965 INFO:tasks.rados.rados.0.smithi123.stdout:626: finishing write tid 3 to smithi12313934-2 2020-01-16T00:12:36.966 INFO:tasks.rados.rados.0.smithi123.stdout:626: finishing write tid 5 to smithi12313934-2 2020-01-16T00:12:36.966 INFO:tasks.rados.rados.0.smithi123.stdout:626: finishing write tid 6 to smithi12313934-2 2020-01-16T00:12:36.966 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 2 v 66 (ObjNum 63 snap 0 seq_num 63) dirty exists 2020-01-16T00:12:36.966 INFO:tasks.rados.rados.0.smithi123.stdout:626: left oid 2 (ObjNum 63 snap 0 seq_num 63) 2020-01-16T00:12:36.967 INFO:tasks.rados.rados.0.smithi123.stdout:627: finishing write tid 2 to smithi12313934-8 2020-01-16T00:12:36.967 INFO:tasks.rados.rados.0.smithi123.stdout:627: finishing write tid 4 to smithi12313934-8 2020-01-16T00:12:36.967 INFO:tasks.rados.rados.0.smithi123.stdout:627: finishing write tid 5 to smithi12313934-8 2020-01-16T00:12:36.967 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 8 v 33 (ObjNum 64 snap 0 seq_num 64) dirty exists 2020-01-16T00:12:36.967 INFO:tasks.rados.rados.0.smithi123.stdout:627: left oid 8 (ObjNum 64 snap 0 seq_num 64) 2020-01-16T00:12:36.968 INFO:tasks.rados.rados.0.smithi123.stdout:630: expect (ObjNum 6 snap 0 seq_num 6) 2020-01-16T00:12:36.968 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:36.963+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:12:36.969 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:12:36.993 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:12:37.017 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:12:37.038 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5: {} 2020-01-16T00:12:37.039 INFO:tasks.rados.rados.0.smithi123.stdout:629: expect (ObjNum 4 snap 0 seq_num 4) 2020-01-16T00:12:37.060 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:12:37.062 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:12:37.063+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:12:37.069 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:37.069 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:12:37.070 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:12:37.070 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [] 2020-01-16T00:12:37.070 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:37.079 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-01-16T00:12:37.082 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:12:37.083 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-01-16T00:12:37.083 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-01-16T00:12:37.084 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-01-16T00:12:37.084 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-01-16T00:12:37.084 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-01-16T00:12:37.086 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-01-16T00:12:37.086 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-01-16T00:12:37.086 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-01-16T00:12:37.101 INFO:tasks.rados.rados.0.smithi123.stdout:624: done (15 left) 2020-01-16T00:12:37.101 INFO:tasks.rados.rados.0.smithi123.stdout:626: done (14 left) 2020-01-16T00:12:37.101 INFO:tasks.rados.rados.0.smithi123.stdout:627: done (13 left) 2020-01-16T00:12:37.101 INFO:tasks.rados.rados.0.smithi123.stdout:629: done (12 left) 2020-01-16T00:12:37.102 INFO:tasks.rados.rados.0.smithi123.stdout:630: done (11 left) 2020-01-16T00:12:37.102 INFO:tasks.rados.rados.0.smithi123.stdout:641: write (excl) oid 19 current snap is 0 2020-01-16T00:12:37.102 INFO:tasks.rados.rados.0.smithi123.stdout:641: seq_num 69 ranges {759683=639358,1904434=719345,3128657=31026} 2020-01-16T00:12:37.108 INFO:tasks.rados.rados.0.smithi123.stdout:641: writing smithi12313934-19 from 759683 to 1399041 tid 1 2020-01-16T00:12:37.116 INFO:tasks.rados.rados.0.smithi123.stdout:641: writing smithi12313934-19 from 1904434 to 2623779 tid 2 2020-01-16T00:12:37.119 INFO:tasks.rados.rados.0.smithi123.stdout:641: writing smithi12313934-19 from 3128657 to 3159683 tid 3 2020-01-16T00:12:37.120 INFO:tasks.rados.rados.0.smithi123.stdout:642: read oid 49 snap -1 2020-01-16T00:12:37.120 INFO:tasks.rados.rados.0.smithi123.stdout:642: expect (ObjNum 48 snap 0 seq_num 48) 2020-01-16T00:12:37.120 INFO:tasks.rados.rados.0.smithi123.stdout:643: write oid 5 current snap is 0 2020-01-16T00:12:37.121 INFO:tasks.rados.rados.0.smithi123.stdout:643: seq_num 70 ranges {776490=714607,2246563=579356} 2020-01-16T00:12:37.128 INFO:tasks.rados.rados.0.smithi123.stdout:643: writing smithi12313934-5 from 776490 to 1491097 tid 1 2020-01-16T00:12:37.137 INFO:tasks.rados.rados.0.smithi123.stdout:643: writing smithi12313934-5 from 2246563 to 2825919 tid 2 2020-01-16T00:12:37.137 INFO:tasks.rados.rados.0.smithi123.stdout:644: read oid 45 snap -1 2020-01-16T00:12:37.137 INFO:tasks.rados.rados.0.smithi123.stdout:644: expect (ObjNum 44 snap 0 seq_num 44) 2020-01-16T00:12:37.138 INFO:tasks.rados.rados.0.smithi123.stdout:645: write oid 17 current snap is 0 2020-01-16T00:12:37.138 INFO:tasks.rados.rados.0.smithi123.stdout:645: seq_num 71 ranges {793297=789856,2305045=723014} 2020-01-16T00:12:37.145 INFO:tasks.rados.rados.0.smithi123.stdout:645: writing smithi12313934-17 from 793297 to 1583153 tid 1 2020-01-16T00:12:37.154 INFO:tasks.rados.rados.0.smithi123.stdout:645: writing smithi12313934-17 from 2305045 to 3028059 tid 2 2020-01-16T00:12:37.154 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:37.154 INFO:tasks.rados.rados.0.smithi123.stdout:631: finishing write tid 1 to smithi12313934-30 2020-01-16T00:12:37.155 INFO:tasks.rados.rados.0.smithi123.stdout:631: finishing write tid 2 to smithi12313934-30 2020-01-16T00:12:37.155 INFO:tasks.rados.rados.0.smithi123.stdout:631: finishing write tid 4 to smithi12313934-30 2020-01-16T00:12:37.155 INFO:tasks.rados.rados.0.smithi123.stdout:631: finishing write tid 5 to smithi12313934-30 2020-01-16T00:12:37.155 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 30 v 48 (ObjNum 65 snap 0 seq_num 65) dirty exists 2020-01-16T00:12:37.155 INFO:tasks.rados.rados.0.smithi123.stdout:631: left oid 30 (ObjNum 65 snap 0 seq_num 65) 2020-01-16T00:12:37.156 INFO:tasks.rados.rados.0.smithi123.stdout:633: finishing write tid 1 to smithi12313934-1 2020-01-16T00:12:37.156 INFO:tasks.rados.rados.0.smithi123.stdout:628: expect (ObjNum 46 snap 0 seq_num 46) 2020-01-16T00:12:37.163 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:12:37.165+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:12:37.191 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:37.192 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:12:37.192 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:12:37.192 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:37.199 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-01-16T00:12:37.240 INFO:tasks.rados.rados.0.smithi123.stdout:632: expect (ObjNum 33 snap 0 seq_num 33) 2020-01-16T00:12:37.264 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:37.265+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:37.320 INFO:tasks.rados.rados.0.smithi123.stdout:633: finishing write tid 2 to smithi12313934-1 2020-01-16T00:12:37.320 INFO:tasks.rados.rados.0.smithi123.stdout:633: finishing write tid 3 to smithi12313934-1 2020-01-16T00:12:37.320 INFO:tasks.rados.rados.0.smithi123.stdout:633: finishing write tid 5 to smithi12313934-1 2020-01-16T00:12:37.320 INFO:tasks.rados.rados.0.smithi123.stdout:634: finishing write tid 1 to smithi12313934-50 2020-01-16T00:12:37.320 INFO:tasks.rados.rados.0.smithi123.stdout:633: finishing write tid 6 to smithi12313934-1 2020-01-16T00:12:37.321 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 1 v 91 (ObjNum 66 snap 0 seq_num 66) dirty exists 2020-01-16T00:12:37.321 INFO:tasks.rados.rados.0.smithi123.stdout:633: left oid 1 (ObjNum 66 snap 0 seq_num 66) 2020-01-16T00:12:37.321 INFO:tasks.rados.rados.0.smithi123.stdout:634: finishing write tid 2 to smithi12313934-50 2020-01-16T00:12:37.321 INFO:tasks.rados.rados.0.smithi123.stdout:634: finishing write tid 4 to smithi12313934-50 2020-01-16T00:12:37.321 INFO:tasks.rados.rados.0.smithi123.stdout:634: finishing write tid 5 to smithi12313934-50 2020-01-16T00:12:37.321 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 50 v 66 (ObjNum 67 snap 0 seq_num 67) dirty exists 2020-01-16T00:12:37.321 INFO:tasks.rados.rados.0.smithi123.stdout:634: left oid 50 (ObjNum 67 snap 0 seq_num 67) 2020-01-16T00:12:37.322 INFO:tasks.rados.rados.0.smithi123.stdout:636: expect (ObjNum 22 snap 0 seq_num 22) 2020-01-16T00:12:37.363 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:37.364 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:12:37.364 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:12:37.364 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:12:37.364 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:37.366 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:37.367+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:12:37.373 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-01-16T00:12:37.402 INFO:tasks.rados.rados.0.smithi123.stdout:635: expect (ObjNum 36 snap 0 seq_num 36) 2020-01-16T00:12:37.467 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:12:37.467+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:12:37.483 INFO:tasks.rados.rados.0.smithi123.stdout:639: expect (ObjNum 52 snap 0 seq_num 52) 2020-01-16T00:12:37.509 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:37.510 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:12:37.510 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:12:37.510 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:12:37.510 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.510 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:885 3.0 3:0b70bb20:::smithi12313934-46:head [write 742876~447756] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:37.510 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:36.940663+0000", 2020-01-16T00:12:37.511 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.57025816100000004, 2020-01-16T00:12:37.511 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012667668, 2020-01-16T00:12:37.511 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:37.511 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:37.512 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:37.512 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:37.512 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:37.512 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 885 2020-01-16T00:12:37.512 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.512 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:37.512 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.513 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:37.513 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.940663+0000", 2020-01-16T00:12:37.513 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:37.513 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.513 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.513 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:37.513 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.940663+0000", 2020-01-16T00:12:37.513 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:37.514 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.514 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.514 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:37.514 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.940661+0000", 2020-01-16T00:12:37.514 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000152392 2020-01-16T00:12:37.514 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.515 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.515 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:37.515 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.940813+0000", 2020-01-16T00:12:37.516 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7827000000000001e-05 2020-01-16T00:12:37.517 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.517 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.517 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:37.517 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.940831+0000", 2020-01-16T00:12:37.517 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9649999999999997e-06 2020-01-16T00:12:37.517 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.517 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.518 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:37.518 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.940840+0000", 2020-01-16T00:12:37.518 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4238000000000003e-05 2020-01-16T00:12:37.518 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.518 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.518 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:37.518 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.940914+0000", 2020-01-16T00:12:37.518 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019430200000000001 2020-01-16T00:12:37.519 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.519 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.519 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:37.519 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.941108+0000", 2020-01-16T00:12:37.519 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.004627511 2020-01-16T00:12:37.519 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.520 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.520 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:37.520 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.945736+0000", 2020-01-16T00:12:37.520 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000121834 2020-01-16T00:12:37.520 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.520 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.520 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:37.520 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.945858+0000", 2020-01-16T00:12:37.521 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000196589 2020-01-16T00:12:37.521 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.521 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.521 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:12:37.521 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.946054+0000", 2020-01-16T00:12:37.521 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0038495209999999998 2020-01-16T00:12:37.521 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.522 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.522 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:37.522 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.949904+0000", 2020-01-16T00:12:37.522 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001149148 2020-01-16T00:12:37.522 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.522 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.522 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:37.522 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.951053+0000", 2020-01-16T00:12:37.523 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002184812 2020-01-16T00:12:37.523 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.523 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.523 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:37.523 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.953238+0000", 2020-01-16T00:12:37.523 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8943000000000002e-05 2020-01-16T00:12:37.523 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.523 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.524 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:37.524 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.953287+0000", 2020-01-16T00:12:37.524 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3714000000000002e-05 2020-01-16T00:12:37.526 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.526 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.526 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:37.526 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.953330+0000", 2020-01-16T00:12:37.526 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012667668 2020-01-16T00:12:37.527 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.527 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:37.527 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.527 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.527 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.527 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:886 3.0 3:0b70bb20:::smithi12313934-46:head [write 1729599~575687] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:37.527 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:36.947656+0000", 2020-01-16T00:12:37.527 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.56326482499999997, 2020-01-16T00:12:37.528 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0066569680000000001, 2020-01-16T00:12:37.528 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:37.528 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:37.528 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:37.528 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:37.528 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:37.528 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 886 2020-01-16T00:12:37.528 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.529 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:37.529 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.529 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:37.529 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.947656+0000", 2020-01-16T00:12:37.529 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:37.529 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.530 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.530 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:37.530 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.947656+0000", 2020-01-16T00:12:37.532 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:37.533 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.533 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.533 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:37.533 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.947653+0000", 2020-01-16T00:12:37.533 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000219333 2020-01-16T00:12:37.533 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.533 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.534 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:37.541 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.947873+0000", 2020-01-16T00:12:37.541 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7187e-05 2020-01-16T00:12:37.541 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.541 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.541 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:37.542 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.947890+0000", 2020-01-16T00:12:37.542 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7179999999999996e-06 2020-01-16T00:12:37.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.542 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.542 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:37.542 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.947898+0000", 2020-01-16T00:12:37.542 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00052636000000000002 2020-01-16T00:12:37.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.543 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.543 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:37.543 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.948424+0000", 2020-01-16T00:12:37.543 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112526 2020-01-16T00:12:37.543 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.543 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.543 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:37.543 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.948537+0000", 2020-01-16T00:12:37.544 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000182122 2020-01-16T00:12:37.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.544 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.544 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:12:37.544 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.948719+0000", 2020-01-16T00:12:37.544 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0016688460000000001 2020-01-16T00:12:37.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.544 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.545 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:37.545 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.950388+0000", 2020-01-16T00:12:37.545 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001173648 2020-01-16T00:12:37.545 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.545 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.545 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:37.545 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.951561+0000", 2020-01-16T00:12:37.546 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0026285179999999998 2020-01-16T00:12:37.546 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.546 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.546 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:37.546 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.954190+0000", 2020-01-16T00:12:37.546 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3638000000000002e-05 2020-01-16T00:12:37.546 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.547 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.547 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:37.547 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.954263+0000", 2020-01-16T00:12:37.547 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9633999999999999e-05 2020-01-16T00:12:37.547 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.547 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.547 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:37.547 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.954313+0000", 2020-01-16T00:12:37.548 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0066569680000000001 2020-01-16T00:12:37.548 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.548 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:37.548 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.548 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.548 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.548 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:887 3.0 3:0b70bb20:::smithi12313934-46:head [write 3101234~41642] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:37.548 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:36.952515+0000", 2020-01-16T00:12:37.549 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.55840628800000003, 2020-01-16T00:12:37.549 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011573231, 2020-01-16T00:12:37.549 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:37.549 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:37.549 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:37.549 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:37.549 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:37.550 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 887 2020-01-16T00:12:37.550 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.550 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:37.550 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.551 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:37.551 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.952515+0000", 2020-01-16T00:12:37.551 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:37.551 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.551 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.551 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:37.551 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.952515+0000", 2020-01-16T00:12:37.551 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:37.552 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.552 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.552 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:37.552 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.952512+0000", 2020-01-16T00:12:37.552 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9529000000000001e-05 2020-01-16T00:12:37.552 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.552 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.552 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:37.553 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.952542+0000", 2020-01-16T00:12:37.553 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1565e-05 2020-01-16T00:12:37.553 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.553 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.553 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:37.553 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.952553+0000", 2020-01-16T00:12:37.553 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8160000000000001e-06 2020-01-16T00:12:37.554 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.554 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.554 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:37.554 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.952558+0000", 2020-01-16T00:12:37.554 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0346000000000002e-05 2020-01-16T00:12:37.554 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.554 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.554 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:37.555 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.952608+0000", 2020-01-16T00:12:37.555 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00033374899999999999 2020-01-16T00:12:37.555 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.555 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.555 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:37.555 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.952942+0000", 2020-01-16T00:12:37.555 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0090136859999999999 2020-01-16T00:12:37.555 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.556 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.556 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:37.556 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.961956+0000", 2020-01-16T00:12:37.556 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010887099999999999 2020-01-16T00:12:37.556 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.556 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.556 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:37.556 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.962065+0000", 2020-01-16T00:12:37.557 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014016199999999999 2020-01-16T00:12:37.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.557 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.557 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:12:37.557 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.962205+0000", 2020-01-16T00:12:37.557 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001098038 2020-01-16T00:12:37.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.557 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.558 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:37.558 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.963303+0000", 2020-01-16T00:12:37.558 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000224284 2020-01-16T00:12:37.558 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.558 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.558 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:37.558 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.963527+0000", 2020-01-16T00:12:37.558 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00050208000000000002 2020-01-16T00:12:37.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.559 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.559 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:37.559 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.964029+0000", 2020-01-16T00:12:37.559 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3380999999999998e-05 2020-01-16T00:12:37.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.559 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.559 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:37.560 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.964063+0000", 2020-01-16T00:12:37.560 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5182999999999999e-05 2020-01-16T00:12:37.560 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.560 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.561 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:37.561 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.964088+0000", 2020-01-16T00:12:37.561 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011573231 2020-01-16T00:12:37.561 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.561 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:37.561 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.561 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.561 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.562 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:888 3.0 3:0b70bb20:::smithi12313934-46:head [setxattr _header (56) in=7b,truncate 3142876] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:37.562 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:36.952566+0000", 2020-01-16T00:12:37.562 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.55835445699999997, 2020-01-16T00:12:37.562 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011670758, 2020-01-16T00:12:37.562 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:37.562 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:37.562 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:37.562 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:37.563 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:37.563 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 888 2020-01-16T00:12:37.563 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.563 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:37.563 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.563 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:37.563 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.952566+0000", 2020-01-16T00:12:37.564 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:37.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.564 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.564 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:37.564 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.952566+0000", 2020-01-16T00:12:37.564 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:37.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.564 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.565 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:37.565 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.952566+0000", 2020-01-16T00:12:37.565 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1170000000000003e-06 2020-01-16T00:12:37.565 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.565 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.565 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:37.565 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.952571+0000", 2020-01-16T00:12:37.565 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5130000000000004e-06 2020-01-16T00:12:37.566 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.566 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.566 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:37.566 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.952576+0000", 2020-01-16T00:12:37.566 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9249999999999997e-06 2020-01-16T00:12:37.566 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.566 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.566 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:37.567 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.952580+0000", 2020-01-16T00:12:37.567 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00042332699999999998 2020-01-16T00:12:37.567 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.567 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.567 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:37.567 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.953004+0000", 2020-01-16T00:12:37.567 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4640000000000004e-05 2020-01-16T00:12:37.567 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.568 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.568 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:37.568 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.953078+0000", 2020-01-16T00:12:37.568 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0096024760000000004 2020-01-16T00:12:37.568 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.568 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.568 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:37.568 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.962681+0000", 2020-01-16T00:12:37.569 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6589999999999995e-05 2020-01-16T00:12:37.569 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.569 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.569 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:37.569 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.962777+0000", 2020-01-16T00:12:37.569 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000125776 2020-01-16T00:12:37.569 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.569 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.570 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:12:37.570 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.962903+0000", 2020-01-16T00:12:37.570 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00047934799999999999 2020-01-16T00:12:37.570 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.570 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.571 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:37.571 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.963382+0000", 2020-01-16T00:12:37.572 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017676899999999999 2020-01-16T00:12:37.572 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.572 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.573 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:37.573 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.963559+0000", 2020-01-16T00:12:37.573 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00061822899999999998 2020-01-16T00:12:37.573 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.573 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.574 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:37.574 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.964177+0000", 2020-01-16T00:12:37.574 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8666e-05 2020-01-16T00:12:37.574 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.574 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.575 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:37.575 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.964206+0000", 2020-01-16T00:12:37.576 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1062999999999997e-05 2020-01-16T00:12:37.576 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.577 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.577 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:37.577 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.964237+0000", 2020-01-16T00:12:37.577 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011670758 2020-01-16T00:12:37.577 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.577 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:37.577 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.578 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.578 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.578 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:889 3.0 3:0b70bb20:::smithi12313934-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:12:37.578 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:36.952611+0000", 2020-01-16T00:12:37.578 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.558309905, 2020-01-16T00:12:37.578 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011796638999999999, 2020-01-16T00:12:37.578 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:37.578 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:37.579 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:37.579 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:37.579 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:37.579 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 889 2020-01-16T00:12:37.579 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.579 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:37.579 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.580 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:37.580 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.952611+0000", 2020-01-16T00:12:37.580 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:37.580 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.580 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.580 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:37.580 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.952611+0000", 2020-01-16T00:12:37.580 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:12:37.581 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.581 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.581 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:37.581 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.952610+0000", 2020-01-16T00:12:37.581 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6690000000000003e-06 2020-01-16T00:12:37.581 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.581 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.582 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:37.582 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.952616+0000", 2020-01-16T00:12:37.582 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3060000000000003e-06 2020-01-16T00:12:37.582 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.582 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.582 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:37.582 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.952625+0000", 2020-01-16T00:12:37.582 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0548e-05 2020-01-16T00:12:37.583 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.583 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.583 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:37.583 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.952635+0000", 2020-01-16T00:12:37.583 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00076067300000000001 2020-01-16T00:12:37.583 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.583 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.583 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:37.584 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.953396+0000", 2020-01-16T00:12:37.584 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3973e-05 2020-01-16T00:12:37.584 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.584 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.584 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:37.584 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.953450+0000", 2020-01-16T00:12:37.584 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.009733986 2020-01-16T00:12:37.585 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.585 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.585 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:37.585 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.963184+0000", 2020-01-16T00:12:37.585 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9312999999999999e-05 2020-01-16T00:12:37.585 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.585 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.585 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:37.586 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.963243+0000", 2020-01-16T00:12:37.586 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001031959 2020-01-16T00:12:37.586 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.586 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.586 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:37.586 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.964275+0000", 2020-01-16T00:12:37.586 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4951999999999997e-05 2020-01-16T00:12:37.587 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.587 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.587 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:37.587 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.964340+0000", 2020-01-16T00:12:37.587 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7318999999999997e-05 2020-01-16T00:12:37.587 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.587 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.587 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:37.588 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.964408+0000", 2020-01-16T00:12:37.588 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011796638999999999 2020-01-16T00:12:37.588 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.588 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:37.588 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.588 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.588 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.588 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop_reply(osd.1.0:146 3.0 e24/20 ondisk, result = 0)", 2020-01-16T00:12:37.589 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:36.952710+0000", 2020-01-16T00:12:37.589 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.55821086799999997, 2020-01-16T00:12:37.589 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00048535400000000002, 2020-01-16T00:12:37.589 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:37.589 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:37.589 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:37.589 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.590 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:37.590 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.952710+0000", 2020-01-16T00:12:37.590 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:37.590 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.590 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.590 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:37.590 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.952710+0000", 2020-01-16T00:12:37.591 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:37.591 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.591 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.591 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:37.591 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.952709+0000", 2020-01-16T00:12:37.591 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0239999999999997e-06 2020-01-16T00:12:37.591 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.591 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.592 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:37.592 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.952717+0000", 2020-01-16T00:12:37.592 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0635000000000001e-05 2020-01-16T00:12:37.592 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.592 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.592 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:37.592 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.952727+0000", 2020-01-16T00:12:37.592 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1699999999999996e-06 2020-01-16T00:12:37.593 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.593 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.593 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:37.593 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.952732+0000", 2020-01-16T00:12:37.593 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00038483000000000001 2020-01-16T00:12:37.593 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.593 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.594 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:37.594 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.953117+0000", 2020-01-16T00:12:37.594 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0559999999999999e-06 2020-01-16T00:12:37.594 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.594 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.594 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:37.594 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.953124+0000", 2020-01-16T00:12:37.594 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0994999999999994e-05 2020-01-16T00:12:37.595 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.595 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.595 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:37.595 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.953195+0000", 2020-01-16T00:12:37.595 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00048535400000000002 2020-01-16T00:12:37.595 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.595 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:37.596 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.596 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.596 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.596 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop_reply(client.4488.0:885 3.0 e24/20 ondisk, result = 0)", 2020-01-16T00:12:37.596 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:36.952785+0000", 2020-01-16T00:12:37.596 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.55813563300000002, 2020-01-16T00:12:37.596 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00055096300000000002, 2020-01-16T00:12:37.596 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:37.597 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:37.597 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:37.597 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.597 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:37.597 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.952785+0000", 2020-01-16T00:12:37.597 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:37.597 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.597 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.598 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:37.598 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.952785+0000", 2020-01-16T00:12:37.598 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:37.598 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.598 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.598 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:37.599 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.952783+0000", 2020-01-16T00:12:37.599 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6930000000000007e-06 2020-01-16T00:12:37.599 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.599 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.599 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:37.599 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.952791+0000", 2020-01-16T00:12:37.599 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1595e-05 2020-01-16T00:12:37.600 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.600 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.600 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:37.600 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.952803+0000", 2020-01-16T00:12:37.600 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0069999999999997e-06 2020-01-16T00:12:37.600 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.600 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.600 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:37.601 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.952811+0000", 2020-01-16T00:12:37.601 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00041309499999999999 2020-01-16T00:12:37.601 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.601 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.601 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:37.601 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.953224+0000", 2020-01-16T00:12:37.601 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6470000000000001e-06 2020-01-16T00:12:37.601 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.602 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.602 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:37.602 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.953230+0000", 2020-01-16T00:12:37.602 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010576799999999999 2020-01-16T00:12:37.602 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.602 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.602 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:37.602 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.953336+0000", 2020-01-16T00:12:37.603 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00055096300000000002 2020-01-16T00:12:37.603 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.603 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:37.603 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.603 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.603 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.603 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop_reply(osd.1.0:149 3.0 e24/20 ondisk, result = 0)", 2020-01-16T00:12:37.603 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:36.962345+0000", 2020-01-16T00:12:37.604 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.548576281, 2020-01-16T00:12:37.604 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00095974199999999997, 2020-01-16T00:12:37.604 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:37.604 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:37.604 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:37.604 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.604 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:37.605 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.962345+0000", 2020-01-16T00:12:37.605 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:37.605 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.605 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.605 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:37.605 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.962345+0000", 2020-01-16T00:12:37.606 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:37.606 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.606 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.606 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:37.606 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.962343+0000", 2020-01-16T00:12:37.606 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.618e-06 2020-01-16T00:12:37.606 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.607 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.607 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:37.607 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.962350+0000", 2020-01-16T00:12:37.607 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0299999999999996e-06 2020-01-16T00:12:37.607 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.607 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.607 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:37.607 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.962357+0000", 2020-01-16T00:12:37.608 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1829999999999996e-06 2020-01-16T00:12:37.608 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.608 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.608 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:37.608 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.962361+0000", 2020-01-16T00:12:37.608 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00091143099999999998 2020-01-16T00:12:37.608 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.609 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.609 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:37.609 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.963273+0000", 2020-01-16T00:12:37.609 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1199999999999999e-06 2020-01-16T00:12:37.609 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.609 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.609 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:37.609 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.963279+0000", 2020-01-16T00:12:37.610 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5585999999999999e-05 2020-01-16T00:12:37.610 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.610 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.610 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:37.610 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.963304+0000", 2020-01-16T00:12:37.610 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00095974199999999997 2020-01-16T00:12:37.610 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.610 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:37.611 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.611 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.611 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.611 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop_reply(osd.1.0:149 3.0 e24/20 ondisk, result = 0)", 2020-01-16T00:12:37.611 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:36.962671+0000", 2020-01-16T00:12:37.611 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.54824990299999998, 2020-01-16T00:12:37.611 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00069477799999999995, 2020-01-16T00:12:37.612 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:37.612 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:37.612 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:37.612 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.612 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:37.612 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.962671+0000", 2020-01-16T00:12:37.612 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:37.612 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.613 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.613 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:37.613 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.962671+0000", 2020-01-16T00:12:37.613 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:37.613 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.613 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.613 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:37.614 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.962669+0000", 2020-01-16T00:12:37.614 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8430000000000004e-06 2020-01-16T00:12:37.614 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.614 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.614 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:37.614 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.962676+0000", 2020-01-16T00:12:37.614 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0244999999999999e-05 2020-01-16T00:12:37.614 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.615 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.615 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:37.615 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.962686+0000", 2020-01-16T00:12:37.615 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0509999999999997e-06 2020-01-16T00:12:37.615 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.615 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.615 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:37.616 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.962695+0000", 2020-01-16T00:12:37.616 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00062385499999999998 2020-01-16T00:12:37.616 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.616 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.616 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:37.616 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.963318+0000", 2020-01-16T00:12:37.616 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.529e-06 2020-01-16T00:12:37.617 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.617 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.617 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:37.617 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.963322+0000", 2020-01-16T00:12:37.617 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3801000000000003e-05 2020-01-16T00:12:37.617 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.617 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.617 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:37.618 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.963366+0000", 2020-01-16T00:12:37.618 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00069477799999999995 2020-01-16T00:12:37.618 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.618 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:37.618 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.618 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.618 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.618 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:890 3.a e24/20 3:5c245412:::smithi12313934-19:head v 24'67, mlcod=23'65)", 2020-01-16T00:12:37.619 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.114365+0000", 2020-01-16T00:12:37.619 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.39655568400000002, 2020-01-16T00:12:37.619 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030368589999999998, 2020-01-16T00:12:37.619 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:37.619 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:37.619 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:37.619 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.619 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:37.620 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.114365+0000", 2020-01-16T00:12:37.620 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:37.620 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.620 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.620 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:37.620 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.114365+0000", 2020-01-16T00:12:37.621 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999962 2020-01-16T00:12:37.621 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.621 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.621 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:37.621 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.114361+0000", 2020-01-16T00:12:37.621 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00057728200000000003 2020-01-16T00:12:37.621 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.621 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.622 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:37.622 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.114938+0000", 2020-01-16T00:12:37.622 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.4630999999999998e-05 2020-01-16T00:12:37.622 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.622 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.622 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:37.622 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.114963+0000", 2020-01-16T00:12:37.623 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1165999999999999e-05 2020-01-16T00:12:37.623 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.623 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.623 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:37.623 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.114974+0000", 2020-01-16T00:12:37.623 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5667000000000006e-05 2020-01-16T00:12:37.623 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.623 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.624 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:37.624 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.115050+0000", 2020-01-16T00:12:37.624 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1856999999999997e-05 2020-01-16T00:12:37.624 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.624 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.624 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:37.625 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.115082+0000", 2020-01-16T00:12:37.625 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002272806 2020-01-16T00:12:37.625 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.625 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.625 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:37.625 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.117355+0000", 2020-01-16T00:12:37.625 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7493000000000001e-05 2020-01-16T00:12:37.625 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.626 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.626 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:37.626 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.117402+0000", 2020-01-16T00:12:37.626 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030368589999999998 2020-01-16T00:12:37.626 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.626 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:37.626 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.626 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.627 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.627 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:891 3.a e24/20 3:5c245412:::smithi12313934-19:head v 24'68, mlcod=23'66)", 2020-01-16T00:12:37.627 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.121381+0000", 2020-01-16T00:12:37.627 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.38953951199999998, 2020-01-16T00:12:37.627 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0027381419999999998, 2020-01-16T00:12:37.627 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:37.627 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:37.627 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:37.628 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.628 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:37.628 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.121381+0000", 2020-01-16T00:12:37.628 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:37.628 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.628 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.628 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:37.629 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.121381+0000", 2020-01-16T00:12:37.629 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:37.629 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.629 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.629 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:37.629 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.121380+0000", 2020-01-16T00:12:37.629 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00065680800000000004 2020-01-16T00:12:37.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.630 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.630 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:37.630 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.122036+0000", 2020-01-16T00:12:37.630 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5084e-05 2020-01-16T00:12:37.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.630 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.630 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:37.631 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.122052+0000", 2020-01-16T00:12:37.631 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.489e-06 2020-01-16T00:12:37.631 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.631 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.631 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:37.631 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.122058+0000", 2020-01-16T00:12:37.631 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2003999999999997e-05 2020-01-16T00:12:37.632 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.632 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.632 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:37.632 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.122130+0000", 2020-01-16T00:12:37.632 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2775000000000003e-05 2020-01-16T00:12:37.632 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.632 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.632 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:37.633 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.122173+0000", 2020-01-16T00:12:37.633 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001913496 2020-01-16T00:12:37.633 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.633 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.633 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:37.633 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.124086+0000", 2020-01-16T00:12:37.633 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3185000000000002e-05 2020-01-16T00:12:37.633 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.634 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.634 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:37.635 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.124119+0000", 2020-01-16T00:12:37.635 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0027381419999999998 2020-01-16T00:12:37.635 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.635 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:37.636 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.636 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.636 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.636 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:892 3.a e24/20 3:5c245412:::smithi12313934-19:head v 24'69, mlcod=23'66)", 2020-01-16T00:12:37.636 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.123576+0000", 2020-01-16T00:12:37.636 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.38734521500000002, 2020-01-16T00:12:37.637 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00082074100000000005, 2020-01-16T00:12:37.637 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:37.637 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:37.637 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:37.637 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.637 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:37.637 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123576+0000", 2020-01-16T00:12:37.637 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:37.638 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.638 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.638 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:37.638 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123576+0000", 2020-01-16T00:12:37.638 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:37.638 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.638 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.638 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:37.639 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123574+0000", 2020-01-16T00:12:37.639 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6468000000000001e-05 2020-01-16T00:12:37.639 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.639 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.639 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:37.639 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123601+0000", 2020-01-16T00:12:37.639 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7280000000000001e-05 2020-01-16T00:12:37.639 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.640 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.640 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:37.640 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123618+0000", 2020-01-16T00:12:37.640 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8390000000000001e-06 2020-01-16T00:12:37.640 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.640 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.640 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:37.641 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123625+0000", 2020-01-16T00:12:37.641 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8664e-05 2020-01-16T00:12:37.641 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.641 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.641 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:37.641 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123674+0000", 2020-01-16T00:12:37.641 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0934999999999999e-05 2020-01-16T00:12:37.641 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.642 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.642 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:37.642 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123695+0000", 2020-01-16T00:12:37.642 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00068258299999999995 2020-01-16T00:12:37.642 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.642 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.643 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:37.643 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.124377+0000", 2020-01-16T00:12:37.643 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9211e-05 2020-01-16T00:12:37.643 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.643 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.643 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:37.643 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.124396+0000", 2020-01-16T00:12:37.643 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00082074100000000005 2020-01-16T00:12:37.644 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.644 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:37.644 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.644 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.644 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.644 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:893 3.a e24/20 3:5c245412:::smithi12313934-19:head v 24'70, mlcod=23'66)", 2020-01-16T00:12:37.644 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.124354+0000", 2020-01-16T00:12:37.644 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.38656696000000002, 2020-01-16T00:12:37.645 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000580337, 2020-01-16T00:12:37.645 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:37.645 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:37.645 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:37.645 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.645 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:37.645 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.124354+0000", 2020-01-16T00:12:37.646 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:37.646 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.646 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.646 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:37.646 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.124354+0000", 2020-01-16T00:12:37.646 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:37.646 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.647 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.647 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:37.647 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.124352+0000", 2020-01-16T00:12:37.647 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.164e-05 2020-01-16T00:12:37.648 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.648 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.648 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:37.648 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.124364+0000", 2020-01-16T00:12:37.648 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.456e-05 2020-01-16T00:12:37.648 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.649 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.649 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:37.649 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.124378+0000", 2020-01-16T00:12:37.649 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3270000000000001e-06 2020-01-16T00:12:37.649 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.649 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.649 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:37.650 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.124387+0000", 2020-01-16T00:12:37.650 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6955e-05 2020-01-16T00:12:37.650 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.650 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.650 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:37.650 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.124434+0000", 2020-01-16T00:12:37.651 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7869999999999999e-05 2020-01-16T00:12:37.651 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.651 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.651 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:37.651 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.124461+0000", 2020-01-16T00:12:37.651 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00045349799999999999 2020-01-16T00:12:37.651 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.652 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.652 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:37.652 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.124915+0000", 2020-01-16T00:12:37.652 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9324e-05 2020-01-16T00:12:37.652 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.652 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.652 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:37.653 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.124934+0000", 2020-01-16T00:12:37.653 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000580337 2020-01-16T00:12:37.653 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.653 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:37.653 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.653 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.654 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.654 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:905 3.4 3:22ad9a20:::smithi12313934-17:head [write 793297~789856] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:37.654 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.148691+0000", 2020-01-16T00:12:37.654 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.36222982199999998, 2020-01-16T00:12:37.654 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011577993, 2020-01-16T00:12:37.654 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:37.655 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:37.655 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:37.655 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:37.655 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:37.655 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 905 2020-01-16T00:12:37.655 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.655 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:37.656 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.656 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:37.656 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.148691+0000", 2020-01-16T00:12:37.656 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:37.656 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.656 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.656 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:37.657 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.148691+0000", 2020-01-16T00:12:37.657 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:12:37.657 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.657 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.657 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:37.658 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.148688+0000", 2020-01-16T00:12:37.658 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021271800000000001 2020-01-16T00:12:37.658 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.658 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.658 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:37.658 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.148901+0000", 2020-01-16T00:12:37.658 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9321e-05 2020-01-16T00:12:37.659 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.659 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.659 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:37.659 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.148920+0000", 2020-01-16T00:12:37.659 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2639e-05 2020-01-16T00:12:37.659 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.660 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.660 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:37.660 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.148933+0000", 2020-01-16T00:12:37.660 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5141000000000005e-05 2020-01-16T00:12:37.660 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.660 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.660 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:37.661 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.148998+0000", 2020-01-16T00:12:37.661 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00027670499999999997 2020-01-16T00:12:37.661 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.661 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.661 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:37.661 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.149275+0000", 2020-01-16T00:12:37.661 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0037401520000000001 2020-01-16T00:12:37.662 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.662 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.662 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:37.662 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.153015+0000", 2020-01-16T00:12:37.662 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012262199999999999 2020-01-16T00:12:37.662 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.662 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.663 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:37.663 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.153137+0000", 2020-01-16T00:12:37.663 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017710800000000001 2020-01-16T00:12:37.663 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.663 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.663 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:12:37.663 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.153314+0000", 2020-01-16T00:12:37.664 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001890685 2020-01-16T00:12:37.664 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.664 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.664 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:37.664 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.155205+0000", 2020-01-16T00:12:37.664 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.004901702 2020-01-16T00:12:37.664 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.665 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.665 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:37.665 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.160107+0000", 2020-01-16T00:12:37.665 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6640000000000001e-05 2020-01-16T00:12:37.665 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.665 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.666 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:37.666 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.160153+0000", 2020-01-16T00:12:37.666 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9250999999999999e-05 2020-01-16T00:12:37.666 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.666 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.666 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:37.667 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.160213+0000", 2020-01-16T00:12:37.667 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6283999999999998e-05 2020-01-16T00:12:37.667 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.667 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.667 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:37.667 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.160269+0000", 2020-01-16T00:12:37.668 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011577993 2020-01-16T00:12:37.668 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.668 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:37.668 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.668 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.668 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.668 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:906 3.4 3:22ad9a20:::smithi12313934-17:head [write 2305045~723014] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:37.669 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.157390+0000", 2020-01-16T00:12:37.669 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.35353038399999998, 2020-01-16T00:12:37.669 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033718429999999998, 2020-01-16T00:12:37.669 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:37.669 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:37.669 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:37.669 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:37.670 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:37.670 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 906 2020-01-16T00:12:37.670 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.670 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:37.670 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.670 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:37.671 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.157390+0000", 2020-01-16T00:12:37.671 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:37.671 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.671 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.671 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:37.671 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.157390+0000", 2020-01-16T00:12:37.671 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2020-01-16T00:12:37.672 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.672 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.672 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:37.672 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.157387+0000", 2020-01-16T00:12:37.672 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00031434799999999999 2020-01-16T00:12:37.672 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.672 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.673 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:37.673 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.157701+0000", 2020-01-16T00:12:37.673 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3920000000000001e-05 2020-01-16T00:12:37.673 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.673 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.673 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:37.673 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.157725+0000", 2020-01-16T00:12:37.674 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.291e-05 2020-01-16T00:12:37.674 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.674 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.674 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:37.674 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.157738+0000", 2020-01-16T00:12:37.674 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0689000000000001e-05 2020-01-16T00:12:37.674 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.675 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.675 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:37.675 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.157799+0000", 2020-01-16T00:12:37.675 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016271499999999999 2020-01-16T00:12:37.675 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.675 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.676 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:37.676 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.157962+0000", 2020-01-16T00:12:37.676 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022799699999999999 2020-01-16T00:12:37.676 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.676 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.676 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:12:37.676 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.158190+0000", 2020-01-16T00:12:37.677 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0022564870000000002 2020-01-16T00:12:37.677 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.677 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.677 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:37.677 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.160446+0000", 2020-01-16T00:12:37.677 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020755799999999999 2020-01-16T00:12:37.677 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.678 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.678 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:37.678 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.160654+0000", 2020-01-16T00:12:37.678 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9265999999999999e-05 2020-01-16T00:12:37.678 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.678 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.679 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:37.679 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.160703+0000", 2020-01-16T00:12:37.679 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2889000000000003e-05 2020-01-16T00:12:37.679 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.679 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.679 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:37.679 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.160736+0000", 2020-01-16T00:12:37.680 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6444e-05 2020-01-16T00:12:37.680 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.680 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.680 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:37.680 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.160762+0000", 2020-01-16T00:12:37.680 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033718429999999998 2020-01-16T00:12:37.681 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.681 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:37.681 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.681 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.681 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.681 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:907 3.4 3:22ad9a20:::smithi12313934-17:head [setxattr _header (56) in=7b,truncate 3193297] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:37.681 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.157758+0000", 2020-01-16T00:12:37.682 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.35316331400000001, 2020-01-16T00:12:37.682 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031654679999999998, 2020-01-16T00:12:37.682 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:37.682 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:37.682 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:37.682 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:37.683 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:37.683 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 907 2020-01-16T00:12:37.683 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.683 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:37.683 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.683 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:37.683 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.157758+0000", 2020-01-16T00:12:37.684 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:37.684 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.684 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.684 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:37.684 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.157758+0000", 2020-01-16T00:12:37.684 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:37.684 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.684 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.685 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:37.685 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.157756+0000", 2020-01-16T00:12:37.685 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0148e-05 2020-01-16T00:12:37.685 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.685 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.685 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:37.686 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.157766+0000", 2020-01-16T00:12:37.686 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2588e-05 2020-01-16T00:12:37.686 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.686 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.686 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:37.686 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.157778+0000", 2020-01-16T00:12:37.686 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2039999999999997e-06 2020-01-16T00:12:37.687 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.687 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.687 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:37.687 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.157787+0000", 2020-01-16T00:12:37.687 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0016271390000000001 2020-01-16T00:12:37.687 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.687 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.688 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:37.688 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.159414+0000", 2020-01-16T00:12:37.688 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016733000000000001 2020-01-16T00:12:37.688 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.688 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.688 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:37.688 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.159581+0000", 2020-01-16T00:12:37.689 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019569100000000001 2020-01-16T00:12:37.689 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.689 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.689 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:12:37.689 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.159777+0000", 2020-01-16T00:12:37.689 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001020831 2020-01-16T00:12:37.690 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.690 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.690 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:37.690 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.160798+0000", 2020-01-16T00:12:37.690 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9953999999999997e-05 2020-01-16T00:12:37.690 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.690 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.691 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:37.691 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.160838+0000", 2020-01-16T00:12:37.691 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3544e-05 2020-01-16T00:12:37.691 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.691 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.691 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:37.691 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.160861+0000", 2020-01-16T00:12:37.692 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0534999999999998e-05 2020-01-16T00:12:37.692 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.692 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.692 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:37.692 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.160892+0000", 2020-01-16T00:12:37.692 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1331999999999997e-05 2020-01-16T00:12:37.693 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.693 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.693 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:37.693 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.160923+0000", 2020-01-16T00:12:37.693 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031654679999999998 2020-01-16T00:12:37.694 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.694 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:37.694 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.694 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.694 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.694 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:908 3.4 3:22ad9a20:::smithi12313934-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:12:37.694 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.157799+0000", 2020-01-16T00:12:37.695 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.35312221999999999, 2020-01-16T00:12:37.695 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0032981730000000002, 2020-01-16T00:12:37.695 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:37.695 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:37.695 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:37.695 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:37.695 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:37.696 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 908 2020-01-16T00:12:37.696 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.696 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:37.696 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.696 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:37.696 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.157799+0000", 2020-01-16T00:12:37.696 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:37.697 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.697 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.697 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:37.697 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.157799+0000", 2020-01-16T00:12:37.697 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:12:37.697 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.698 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.698 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:37.698 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.157798+0000", 2020-01-16T00:12:37.698 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.322e-06 2020-01-16T00:12:37.698 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.698 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.698 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:37.699 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.157803+0000", 2020-01-16T00:12:37.699 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2619999999999999e-05 2020-01-16T00:12:37.699 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.699 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.699 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:37.699 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.157816+0000", 2020-01-16T00:12:37.699 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3139999999999998e-06 2020-01-16T00:12:37.700 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.700 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.700 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:37.700 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.157823+0000", 2020-01-16T00:12:37.700 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002479948 2020-01-16T00:12:37.700 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.700 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.701 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:37.701 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.160303+0000", 2020-01-16T00:12:37.701 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013291099999999999 2020-01-16T00:12:37.701 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.701 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.701 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:37.701 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.160436+0000", 2020-01-16T00:12:37.702 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00052049799999999999 2020-01-16T00:12:37.702 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.702 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.702 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:37.702 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.160956+0000", 2020-01-16T00:12:37.702 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7356999999999998e-05 2020-01-16T00:12:37.702 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.703 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.703 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:37.703 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.161034+0000", 2020-01-16T00:12:37.703 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3145000000000006e-05 2020-01-16T00:12:37.703 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.703 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.704 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:37.704 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.161097+0000", 2020-01-16T00:12:37.704 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0032981730000000002 2020-01-16T00:12:37.704 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.704 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:37.704 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.704 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.705 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.705 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop_reply(client.4488.0:905 3.4 e24/20 ondisk, result = 0)", 2020-01-16T00:12:37.705 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.157854+0000", 2020-01-16T00:12:37.705 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.35306723000000001, 2020-01-16T00:12:37.705 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0022550550000000002, 2020-01-16T00:12:37.705 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:37.705 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:37.706 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:37.706 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.706 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:37.706 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.157854+0000", 2020-01-16T00:12:37.706 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:37.706 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.706 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.707 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:37.707 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.157854+0000", 2020-01-16T00:12:37.707 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:37.707 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.707 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.707 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:37.707 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.157852+0000", 2020-01-16T00:12:37.708 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9640000000000003e-06 2020-01-16T00:12:37.708 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.708 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.708 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:37.708 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.157860+0000", 2020-01-16T00:12:37.708 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2112000000000001e-05 2020-01-16T00:12:37.708 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.708 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.709 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:37.709 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.157882+0000", 2020-01-16T00:12:37.709 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6120000000000001e-06 2020-01-16T00:12:37.709 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.709 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.709 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:37.710 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.157890+0000", 2020-01-16T00:12:37.710 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021973019999999999 2020-01-16T00:12:37.710 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.710 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.710 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:37.710 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.160087+0000", 2020-01-16T00:12:37.710 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6879999999999994e-06 2020-01-16T00:12:37.711 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.711 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.711 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:37.711 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.160097+0000", 2020-01-16T00:12:37.711 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1821000000000001e-05 2020-01-16T00:12:37.711 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.711 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.712 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:37.712 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.160109+0000", 2020-01-16T00:12:37.712 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0022550550000000002 2020-01-16T00:12:37.712 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.712 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:37.712 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.712 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.713 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.713 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop_reply(client.4488.0:905 3.4 e24/20 ondisk, result = 0)", 2020-01-16T00:12:37.713 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.158633+0000", 2020-01-16T00:12:37.713 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.35228787499999997, 2020-01-16T00:12:37.713 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0016436980000000001, 2020-01-16T00:12:37.713 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:37.714 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:37.714 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:37.714 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.714 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:37.714 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.158633+0000", 2020-01-16T00:12:37.714 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:37.714 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.715 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.715 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:37.715 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.158633+0000", 2020-01-16T00:12:37.715 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:37.715 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.715 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.715 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:37.716 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.158631+0000", 2020-01-16T00:12:37.716 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7120000000000006e-06 2020-01-16T00:12:37.716 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.716 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.716 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:37.716 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.158640+0000", 2020-01-16T00:12:37.716 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3126000000000001e-05 2020-01-16T00:12:37.717 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.717 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.717 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:37.717 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.158673+0000", 2020-01-16T00:12:37.717 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2660000000000007e-06 2020-01-16T00:12:37.717 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.717 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.718 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:37.718 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.158682+0000", 2020-01-16T00:12:37.718 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0014550469999999999 2020-01-16T00:12:37.718 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.718 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.718 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:37.718 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.160137+0000", 2020-01-16T00:12:37.719 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6290000000000004e-06 2020-01-16T00:12:37.719 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.719 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.719 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:37.719 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.160144+0000", 2020-01-16T00:12:37.719 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013259 2020-01-16T00:12:37.721 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.721 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.722 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:37.722 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.160277+0000", 2020-01-16T00:12:37.722 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0016436980000000001 2020-01-16T00:12:37.722 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.722 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:37.722 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.723 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.723 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.723 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop_reply(client.4488.0:907 3.4 e24/20 ondisk, result = 0)", 2020-01-16T00:12:37.723 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.160645+0000", 2020-01-16T00:12:37.723 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.35027580200000002, 2020-01-16T00:12:37.723 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00028585399999999999, 2020-01-16T00:12:37.723 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:37.723 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:37.724 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:37.724 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.724 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:37.724 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.160645+0000", 2020-01-16T00:12:37.724 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:37.724 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.725 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.725 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:37.725 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.160645+0000", 2020-01-16T00:12:37.725 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:12:37.725 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.725 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.726 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:37.726 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.160644+0000", 2020-01-16T00:12:37.726 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9339999999999999e-06 2020-01-16T00:12:37.726 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.726 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.726 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:37.726 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.160648+0000", 2020-01-16T00:12:37.727 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3720000000000004e-06 2020-01-16T00:12:37.727 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.727 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.727 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:37.727 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.160654+0000", 2020-01-16T00:12:37.727 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3960000000000002e-06 2020-01-16T00:12:37.727 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.728 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.728 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:37.728 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.160657+0000", 2020-01-16T00:12:37.728 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000197231 2020-01-16T00:12:37.728 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.728 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.728 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:37.729 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.160854+0000", 2020-01-16T00:12:37.729 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4340000000000001e-06 2020-01-16T00:12:37.729 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.729 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.729 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:37.729 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.160858+0000", 2020-01-16T00:12:37.729 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3244999999999994e-05 2020-01-16T00:12:37.730 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:37.730 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:37.730 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:37.730 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.160931+0000", 2020-01-16T00:12:37.730 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00028585399999999999 2020-01-16T00:12:37.730 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.731 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:37.731 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.731 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:37.731 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:37.731 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:37.733 INFO:tasks.rados.rados.0.smithi123.stdout:638: expect (ObjNum 61 snap 0 seq_num 61) 2020-01-16T00:12:37.733 INFO:tasks.rados.rados.0.smithi123.stdout:640: finishing write tid 1 to smithi12313934-46 2020-01-16T00:12:37.733 INFO:tasks.rados.rados.0.smithi123.stdout:640: finishing write tid 2 to smithi12313934-46 2020-01-16T00:12:37.733 INFO:tasks.rados.rados.0.smithi123.stdout:637: expect (ObjNum 39 snap 0 seq_num 39) 2020-01-16T00:12:37.733 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:37.735+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:12:37.734 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-01-16T00:12:37.807 INFO:tasks.rados.rados.0.smithi123.stdout:640: finishing write tid 3 to smithi12313934-46 2020-01-16T00:12:37.807 INFO:tasks.rados.rados.0.smithi123.stdout:640: finishing write tid 5 to smithi12313934-46 2020-01-16T00:12:37.807 INFO:tasks.rados.rados.0.smithi123.stdout:640: finishing write tid 6 to smithi12313934-46 2020-01-16T00:12:37.807 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 46 v 98 (ObjNum 68 snap 0 seq_num 68) dirty exists 2020-01-16T00:12:37.807 INFO:tasks.rados.rados.0.smithi123.stdout:640: left oid 46 (ObjNum 68 snap 0 seq_num 68) 2020-01-16T00:12:37.808 INFO:tasks.rados.rados.0.smithi123.stdout:641: finishing write tid 1 to smithi12313934-19 2020-01-16T00:12:37.808 INFO:tasks.rados.rados.0.smithi123.stdout:641: finishing write tid 2 to smithi12313934-19 2020-01-16T00:12:37.808 INFO:tasks.rados.rados.0.smithi123.stdout:628: done (15 left) 2020-01-16T00:12:37.808 INFO:tasks.rados.rados.0.smithi123.stdout:631: done (14 left) 2020-01-16T00:12:37.808 INFO:tasks.rados.rados.0.smithi123.stdout:632: done (13 left) 2020-01-16T00:12:37.808 INFO:tasks.rados.rados.0.smithi123.stdout:633: done (12 left) 2020-01-16T00:12:37.808 INFO:tasks.rados.rados.0.smithi123.stdout:634: done (11 left) 2020-01-16T00:12:37.809 INFO:tasks.rados.rados.0.smithi123.stdout:635: done (10 left) 2020-01-16T00:12:37.809 INFO:tasks.rados.rados.0.smithi123.stdout:636: done (9 left) 2020-01-16T00:12:37.810 INFO:tasks.rados.rados.0.smithi123.stdout:637: done (8 left) 2020-01-16T00:12:37.810 INFO:tasks.rados.rados.0.smithi123.stdout:638: done (7 left) 2020-01-16T00:12:37.810 INFO:tasks.rados.rados.0.smithi123.stdout:639: done (6 left) 2020-01-16T00:12:37.810 INFO:tasks.rados.rados.0.smithi123.stdout:640: done (5 left) 2020-01-16T00:12:37.810 INFO:tasks.rados.rados.0.smithi123.stdout:646: write oid 27 current snap is 0 2020-01-16T00:12:37.811 INFO:tasks.rados.rados.0.smithi123.stdout:646: seq_num 72 ranges {410104=465105,1563527=583025,2810926=399178} 2020-01-16T00:12:37.819 INFO:tasks.rados.rados.0.smithi123.stdout:646: writing smithi12313934-27 from 410104 to 875209 tid 1 2020-01-16T00:12:37.827 INFO:tasks.rados.rados.0.smithi123.stdout:646: writing smithi12313934-27 from 1563527 to 2146552 tid 2 2020-01-16T00:12:37.834 INFO:tasks.rados.rados.0.smithi123.stdout:646: writing smithi12313934-27 from 2810926 to 3210104 tid 3 2020-01-16T00:12:37.834 INFO:tasks.rados.rados.0.smithi123.stdout:647: write (excl) oid 28 current snap is 0 2020-01-16T00:12:37.834 INFO:tasks.rados.rados.0.smithi123.stdout:647: seq_num 73 ranges {426911=540354,1622009=726683,3121996=104915} 2020-01-16T00:12:37.834 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:12:37.835+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:12:37.839 INFO:tasks.rados.rados.0.smithi123.stdout:647: writing smithi12313934-28 from 426911 to 967265 tid 1 2020-01-16T00:12:37.847 INFO:tasks.rados.rados.0.smithi123.stdout:647: writing smithi12313934-28 from 1622009 to 2348692 tid 2 2020-01-16T00:12:37.853 INFO:tasks.rados.rados.0.smithi123.stdout:647: writing smithi12313934-28 from 3121996 to 3226911 tid 3 2020-01-16T00:12:37.853 INFO:tasks.rados.rados.0.smithi123.stdout:648: read oid 15 snap -1 2020-01-16T00:12:37.853 INFO:tasks.rados.rados.0.smithi123.stdout:648: expect (ObjNum 59 snap 0 seq_num 59) 2020-01-16T00:12:37.853 INFO:tasks.rados.rados.0.smithi123.stdout:649: write (excl) oid 12 current snap is 0 2020-01-16T00:12:37.853 INFO:tasks.rados.rados.0.smithi123.stdout:649: seq_num 74 ranges {443718=615603,1564138=470341,2633066=610652} 2020-01-16T00:12:37.859 INFO:tasks.rados.rados.0.smithi123.stdout:649: writing smithi12313934-12 from 443718 to 1059321 tid 1 2020-01-16T00:12:37.865 INFO:tasks.rados.rados.0.smithi123.stdout:649: writing smithi12313934-12 from 1564138 to 2034479 tid 2 2020-01-16T00:12:37.873 INFO:tasks.rados.rados.0.smithi123.stdout:649: writing smithi12313934-12 from 2633066 to 3243718 tid 3 2020-01-16T00:12:37.873 INFO:tasks.rados.rados.0.smithi123.stdout:650: read oid 3 snap -1 2020-01-16T00:12:37.873 INFO:tasks.rados.rados.0.smithi123.stdout:650: expect (ObjNum 2 snap 0 seq_num 2) 2020-01-16T00:12:37.873 INFO:tasks.rados.rados.0.smithi123.stdout:651: read oid 22 snap -1 2020-01-16T00:12:37.873 INFO:tasks.rados.rados.0.smithi123.stdout:651: expect (ObjNum 21 snap 0 seq_num 21) 2020-01-16T00:12:37.873 INFO:tasks.rados.rados.0.smithi123.stdout:652: write (excl) oid 48 current snap is 0 2020-01-16T00:12:37.873 INFO:tasks.rados.rados.0.smithi123.stdout:652: seq_num 75 ranges {460525=690852,1622620=730352,3060489=200036} 2020-01-16T00:12:37.879 INFO:tasks.rados.rados.0.smithi123.stdout:652: writing smithi12313934-48 from 460525 to 1151377 tid 1 2020-01-16T00:12:37.886 INFO:tasks.rados.rados.0.smithi123.stdout:652: writing smithi12313934-48 from 1622620 to 2352972 tid 2 2020-01-16T00:12:37.892 INFO:tasks.rados.rados.0.smithi123.stdout:652: writing smithi12313934-48 from 3060489 to 3260525 tid 3 2020-01-16T00:12:37.892 INFO:tasks.rados.rados.0.smithi123.stdout:653: write (excl) oid 4 current snap is 0 2020-01-16T00:12:37.892 INFO:tasks.rados.rados.0.smithi123.stdout:653: seq_num 76 ranges {477332=766101,1681102=474010,2687912=589420} 2020-01-16T00:12:37.894 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:37.894 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:12:37.894 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:12:37.894 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:37.899 INFO:tasks.rados.rados.0.smithi123.stdout:653: writing smithi12313934-4 from 477332 to 1243433 tid 1 2020-01-16T00:12:37.904 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-01-16T00:12:37.905 INFO:tasks.rados.rados.0.smithi123.stdout:653: writing smithi12313934-4 from 1681102 to 2155112 tid 2 2020-01-16T00:12:37.912 INFO:tasks.rados.rados.0.smithi123.stdout:653: writing smithi12313934-4 from 2687912 to 3277332 tid 3 2020-01-16T00:12:37.912 INFO:tasks.rados.rados.0.smithi123.stdout:654: read oid 14 snap -1 2020-01-16T00:12:37.912 INFO:tasks.rados.rados.0.smithi123.stdout:654: expect (ObjNum 13 snap 0 seq_num 13) 2020-01-16T00:12:37.912 INFO:tasks.rados.rados.0.smithi123.stdout:655: read oid 7 snap -1 2020-01-16T00:12:37.913 INFO:tasks.rados.rados.0.smithi123.stdout:655: expect (ObjNum 6 snap 0 seq_num 6) 2020-01-16T00:12:37.913 INFO:tasks.rados.rados.0.smithi123.stdout:656: read oid 21 snap -1 2020-01-16T00:12:37.913 INFO:tasks.rados.rados.0.smithi123.stdout:656: expect (ObjNum 20 snap 0 seq_num 20) 2020-01-16T00:12:37.913 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:37.913 INFO:tasks.rados.rados.0.smithi123.stdout:641: finishing write tid 3 to smithi12313934-19 2020-01-16T00:12:37.913 INFO:tasks.rados.rados.0.smithi123.stdout:641: finishing write tid 5 to smithi12313934-19 2020-01-16T00:12:37.914 INFO:tasks.rados.rados.0.smithi123.stdout:641: finishing write tid 6 to smithi12313934-19 2020-01-16T00:12:37.914 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 19 v 70 (ObjNum 69 snap 0 seq_num 69) dirty exists 2020-01-16T00:12:37.914 INFO:tasks.rados.rados.0.smithi123.stdout:641: left oid 19 (ObjNum 69 snap 0 seq_num 69) 2020-01-16T00:12:37.914 INFO:tasks.rados.rados.0.smithi123.stdout:643: finishing write tid 1 to smithi12313934-5 2020-01-16T00:12:37.914 INFO:tasks.rados.rados.0.smithi123.stdout:641: done (15 left) 2020-01-16T00:12:37.914 INFO:tasks.rados.rados.0.smithi123.stdout:657: write (excl) oid 31 current snap is 0 2020-01-16T00:12:37.915 INFO:tasks.rados.rados.0.smithi123.stdout:657: seq_num 77 ranges {494139=557703,1455937=734021,2948041=346098} 2020-01-16T00:12:37.919 INFO:tasks.rados.rados.0.smithi123.stdout:657: writing smithi12313934-31 from 494139 to 1051842 tid 1 2020-01-16T00:12:37.928 INFO:tasks.rados.rados.0.smithi123.stdout:657: writing smithi12313934-31 from 1455937 to 2189958 tid 2 2020-01-16T00:12:37.934 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:12:37.935+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:12:37.937 INFO:tasks.rados.rados.0.smithi123.stdout:657: writing smithi12313934-31 from 2948041 to 3294139 tid 3 2020-01-16T00:12:37.937 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:37.937 INFO:tasks.rados.rados.0.smithi123.stdout:642: expect (ObjNum 48 snap 0 seq_num 48) 2020-01-16T00:12:38.009 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:38.009 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:12:38.010 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:12:38.010 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:12:38.010 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:38.017 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-01-16T00:12:38.021 INFO:tasks.rados.rados.0.smithi123.stdout:643: finishing write tid 2 to smithi12313934-5 2020-01-16T00:12:38.021 INFO:tasks.rados.rados.0.smithi123.stdout:643: finishing write tid 4 to smithi12313934-5 2020-01-16T00:12:38.021 INFO:tasks.rados.rados.0.smithi123.stdout:643: finishing write tid 5 to smithi12313934-5 2020-01-16T00:12:38.022 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 5 v 76 (ObjNum 70 snap 0 seq_num 70) dirty exists 2020-01-16T00:12:38.022 INFO:tasks.rados.rados.0.smithi123.stdout:643: left oid 5 (ObjNum 70 snap 0 seq_num 70) 2020-01-16T00:12:38.022 INFO:tasks.rados.rados.0.smithi123.stdout:642: done (15 left) 2020-01-16T00:12:38.022 INFO:tasks.rados.rados.0.smithi123.stdout:643: done (14 left) 2020-01-16T00:12:38.022 INFO:tasks.rados.rados.0.smithi123.stdout:658: write oid 18 current snap is 0 2020-01-16T00:12:38.022 INFO:tasks.rados.rados.0.smithi123.stdout:658: seq_num 78 ranges {510946=632952,1798066=477679,2742758=457217} 2020-01-16T00:12:38.029 INFO:tasks.rados.rados.0.smithi123.stdout:658: writing smithi12313934-18 from 510946 to 1143898 tid 1 2020-01-16T00:12:38.035 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:12:38.033+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:12:38.037 INFO:tasks.rados.rados.0.smithi123.stdout:658: writing smithi12313934-18 from 1798066 to 2275745 tid 2 2020-01-16T00:12:38.044 INFO:tasks.rados.rados.0.smithi123.stdout:658: writing smithi12313934-18 from 2742758 to 3199975 tid 3 2020-01-16T00:12:38.044 INFO:tasks.rados.rados.0.smithi123.stdout:659: read oid 43 snap -1 2020-01-16T00:12:38.045 INFO:tasks.rados.rados.0.smithi123.stdout:659: expect (ObjNum 42 snap 0 seq_num 42) 2020-01-16T00:12:38.045 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:38.045 INFO:tasks.rados.rados.0.smithi123.stdout:644: expect (ObjNum 44 snap 0 seq_num 44) 2020-01-16T00:12:38.126 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:38.126 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:12:38.126 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:12:38.127 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:12:38.127 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.127 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.1.13:21 2.1 e24/19 2:bafac7f5:::smithi12313934-46 low_tier:head v 24'24, mlcod=24'22)", 2020-01-16T00:12:38.128 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:36.954506+0000", 2020-01-16T00:12:38.128 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 1.173333534, 2020-01-16T00:12:38.128 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0065224649999999999, 2020-01-16T00:12:38.128 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:38.128 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:38.128 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:38.128 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.129 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:38.129 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.954506+0000", 2020-01-16T00:12:38.129 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:38.129 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.129 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.129 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:38.130 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.954506+0000", 2020-01-16T00:12:38.130 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:38.130 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.130 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.130 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:38.130 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.954504+0000", 2020-01-16T00:12:38.132 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00036003199999999998 2020-01-16T00:12:38.133 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.133 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.133 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:38.133 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.954864+0000", 2020-01-16T00:12:38.133 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6718000000000002e-05 2020-01-16T00:12:38.134 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.134 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.134 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:38.134 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.954881+0000", 2020-01-16T00:12:38.135 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8170000000000001e-06 2020-01-16T00:12:38.135 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.135 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.135 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:38.135 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.954888+0000", 2020-01-16T00:12:38.136 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1836000000000003e-05 2020-01-16T00:12:38.136 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.136 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.136 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.136 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.954940+0000", 2020-01-16T00:12:38.137 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8199e-05 2020-01-16T00:12:38.137 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.137 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.137 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:38.137 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.954958+0000", 2020-01-16T00:12:38.137 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0060089929999999998 2020-01-16T00:12:38.138 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.138 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.138 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:38.138 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.960967+0000", 2020-01-16T00:12:38.139 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1935999999999998e-05 2020-01-16T00:12:38.139 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.139 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.139 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:38.140 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.961029+0000", 2020-01-16T00:12:38.140 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0065224649999999999 2020-01-16T00:12:38.140 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.140 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:38.140 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.140 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.141 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.141 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.1.13:22 2.1 e24/19 2:bafac7f5:::smithi12313934-46 low_tier:head v 24'25, mlcod=24'22)", 2020-01-16T00:12:38.141 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:36.957132+0000", 2020-01-16T00:12:38.141 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 1.1707075140000001, 2020-01-16T00:12:38.142 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041674609999999999, 2020-01-16T00:12:38.142 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:38.142 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:38.142 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:38.142 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.143 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:38.143 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.957132+0000", 2020-01-16T00:12:38.143 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:38.143 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.143 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.143 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:38.144 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.957132+0000", 2020-01-16T00:12:38.144 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:38.144 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.144 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.145 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:38.145 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.957130+0000", 2020-01-16T00:12:38.145 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4346999999999999e-05 2020-01-16T00:12:38.145 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.145 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.146 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:38.146 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.957194+0000", 2020-01-16T00:12:38.146 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6614999999999999e-05 2020-01-16T00:12:38.146 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.146 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.146 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:38.146 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.957211+0000", 2020-01-16T00:12:38.147 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6529999999999994e-06 2020-01-16T00:12:38.147 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.147 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.147 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:38.147 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.957218+0000", 2020-01-16T00:12:38.148 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9471999999999997e-05 2020-01-16T00:12:38.148 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.148 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.148 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.148 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.957258+0000", 2020-01-16T00:12:38.149 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8435000000000001e-05 2020-01-16T00:12:38.149 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.149 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.149 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:38.149 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.957286+0000", 2020-01-16T00:12:38.149 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039801250000000002 2020-01-16T00:12:38.150 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.150 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.150 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:38.150 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.961266+0000", 2020-01-16T00:12:38.150 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3266999999999998e-05 2020-01-16T00:12:38.151 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.151 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.151 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:38.151 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.961300+0000", 2020-01-16T00:12:38.151 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041674609999999999 2020-01-16T00:12:38.151 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.152 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:38.152 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.152 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.152 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.152 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.4.0:110 3.3 e24/20 3:cd231094:::smithi12313934-49:head v 24'40, mlcod=23'38)", 2020-01-16T00:12:38.152 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.130258+0000", 2020-01-16T00:12:38.152 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.99758166400000003, 2020-01-16T00:12:38.153 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0062283110000000003, 2020-01-16T00:12:38.153 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:38.153 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:38.153 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:38.153 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.153 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:38.154 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.130258+0000", 2020-01-16T00:12:38.154 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:38.154 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.154 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.154 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:38.155 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.130258+0000", 2020-01-16T00:12:38.155 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:12:38.155 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.155 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.156 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:38.156 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.130257+0000", 2020-01-16T00:12:38.156 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001916636 2020-01-16T00:12:38.156 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.156 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.156 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:38.157 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.132174+0000", 2020-01-16T00:12:38.157 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5815e-05 2020-01-16T00:12:38.157 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.157 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.157 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:38.157 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.132190+0000", 2020-01-16T00:12:38.157 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2159999999999995e-06 2020-01-16T00:12:38.158 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.158 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.158 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:38.158 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.132199+0000", 2020-01-16T00:12:38.158 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2299999999999995e-05 2020-01-16T00:12:38.158 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.159 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.159 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.159 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.132281+0000", 2020-01-16T00:12:38.159 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6641999999999999e-05 2020-01-16T00:12:38.160 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.160 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.160 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:38.160 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.132318+0000", 2020-01-16T00:12:38.160 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.004106129 2020-01-16T00:12:38.161 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.161 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.161 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:38.161 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.136424+0000", 2020-01-16T00:12:38.162 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2360000000000006e-05 2020-01-16T00:12:38.162 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.162 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.163 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:38.163 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.136486+0000", 2020-01-16T00:12:38.163 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0062283110000000003 2020-01-16T00:12:38.163 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.164 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:38.164 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.164 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.164 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.165 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.4.0:111 3.3 e24/20 3:c204aa4f:::smithi12313934-45:head v 24'41, mlcod=23'39)", 2020-01-16T00:12:38.165 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.143122+0000", 2020-01-16T00:12:38.166 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.98471828400000005, 2020-01-16T00:12:38.166 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.004384092, 2020-01-16T00:12:38.166 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:38.166 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:38.167 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:38.167 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.167 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:38.167 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.143122+0000", 2020-01-16T00:12:38.168 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:38.168 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.168 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.169 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:38.169 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.143122+0000", 2020-01-16T00:12:38.169 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999809 2020-01-16T00:12:38.169 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.170 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.170 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:38.170 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.143103+0000", 2020-01-16T00:12:38.171 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013691770000000001 2020-01-16T00:12:38.171 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.171 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.172 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:38.172 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.144472+0000", 2020-01-16T00:12:38.172 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7095000000000001e-05 2020-01-16T00:12:38.173 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.173 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.173 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:38.173 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.144499+0000", 2020-01-16T00:12:38.173 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8910000000000007e-06 2020-01-16T00:12:38.173 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.174 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.174 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:38.174 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.144509+0000", 2020-01-16T00:12:38.174 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2083999999999996e-05 2020-01-16T00:12:38.174 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.174 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.174 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.175 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.144581+0000", 2020-01-16T00:12:38.175 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4900999999999997e-05 2020-01-16T00:12:38.175 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.175 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.175 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:38.176 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.144616+0000", 2020-01-16T00:12:38.176 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0028443679999999999 2020-01-16T00:12:38.176 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.176 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.176 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:38.176 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.147460+0000", 2020-01-16T00:12:38.177 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5447e-05 2020-01-16T00:12:38.177 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.177 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.177 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:38.177 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.147506+0000", 2020-01-16T00:12:38.178 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.004384092 2020-01-16T00:12:38.178 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.178 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:38.178 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.178 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.178 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.179 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.5.13:36 2.0 e24/19 2:20e40cec:::smithi12313934-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 24'16, mlcod=23'14)", 2020-01-16T00:12:38.179 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.834879+0000", 2020-01-16T00:12:38.179 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.29296054900000001, 2020-01-16T00:12:38.179 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0075184850000000001, 2020-01-16T00:12:38.179 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:38.179 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:38.179 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:38.180 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.180 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:38.180 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.834879+0000", 2020-01-16T00:12:38.180 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:38.180 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.180 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.181 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:38.181 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.834879+0000", 2020-01-16T00:12:38.181 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:38.181 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.181 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.181 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:38.181 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.834877+0000", 2020-01-16T00:12:38.182 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031393089999999998 2020-01-16T00:12:38.182 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.182 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.182 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:38.182 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.838016+0000", 2020-01-16T00:12:38.182 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2144e-05 2020-01-16T00:12:38.183 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.183 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.183 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:38.183 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.838038+0000", 2020-01-16T00:12:38.184 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2058e-05 2020-01-16T00:12:38.184 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.184 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.184 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:38.185 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.838050+0000", 2020-01-16T00:12:38.185 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8533000000000003e-05 2020-01-16T00:12:38.185 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.185 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.186 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.186 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.838119+0000", 2020-01-16T00:12:38.187 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3591000000000003e-05 2020-01-16T00:12:38.187 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.187 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.187 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:38.188 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.838162+0000", 2020-01-16T00:12:38.188 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041910380000000002 2020-01-16T00:12:38.188 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.188 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.189 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:38.189 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.842353+0000", 2020-01-16T00:12:38.189 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4433e-05 2020-01-16T00:12:38.190 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.190 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.190 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:38.190 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.842398+0000", 2020-01-16T00:12:38.191 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0075184850000000001 2020-01-16T00:12:38.191 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.191 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:38.191 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.192 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.192 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.192 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.5.13:40 2.0 e24/19 2:20e40cec:::smithi12313934-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 24'17, mlcod=23'15)", 2020-01-16T00:12:38.193 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.849607+0000", 2020-01-16T00:12:38.193 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.27823242399999998, 2020-01-16T00:12:38.193 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083979039999999994, 2020-01-16T00:12:38.194 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:38.194 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:38.194 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:38.194 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.195 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:38.195 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.849607+0000", 2020-01-16T00:12:38.195 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:38.195 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.196 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.196 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:38.197 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.849607+0000", 2020-01-16T00:12:38.197 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:38.197 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.197 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.198 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:38.198 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.849605+0000", 2020-01-16T00:12:38.198 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0037534560000000001 2020-01-16T00:12:38.198 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.199 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.199 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:38.199 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.853358+0000", 2020-01-16T00:12:38.199 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9181000000000002e-05 2020-01-16T00:12:38.200 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.200 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.200 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:38.200 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.853377+0000", 2020-01-16T00:12:38.201 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0040000000000002e-06 2020-01-16T00:12:38.201 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.201 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.201 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:38.202 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.853384+0000", 2020-01-16T00:12:38.202 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2583000000000001e-05 2020-01-16T00:12:38.202 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.202 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.202 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.203 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.853437+0000", 2020-01-16T00:12:38.203 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8212999999999999e-05 2020-01-16T00:12:38.203 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.203 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.204 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:38.204 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.853465+0000", 2020-01-16T00:12:38.204 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0044847589999999996 2020-01-16T00:12:38.204 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.204 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.205 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:38.205 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.857950+0000", 2020-01-16T00:12:38.205 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5309999999999997e-05 2020-01-16T00:12:38.205 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.206 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.206 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:38.206 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.858005+0000", 2020-01-16T00:12:38.207 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083979039999999994 2020-01-16T00:12:38.207 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.207 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:38.208 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.208 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.208 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.208 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.5.13:42 2.0 e24/19 2:313458df:::smithi12313934-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 24'18, mlcod=24'16)", 2020-01-16T00:12:38.209 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.940683+0000", 2020-01-16T00:12:38.209 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.18715658499999999, 2020-01-16T00:12:38.209 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0097026960000000002, 2020-01-16T00:12:38.209 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:38.209 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:38.210 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:38.210 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.210 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:38.210 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.940683+0000", 2020-01-16T00:12:38.211 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:38.211 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.211 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.211 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:38.212 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.940683+0000", 2020-01-16T00:12:38.212 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:12:38.212 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.212 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.213 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:38.213 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.940680+0000", 2020-01-16T00:12:38.213 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003126775 2020-01-16T00:12:38.214 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.214 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.214 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:38.214 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.943807+0000", 2020-01-16T00:12:38.214 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5426999999999999e-05 2020-01-16T00:12:38.215 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.215 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.215 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:38.215 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.943833+0000", 2020-01-16T00:12:38.215 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7409999999999998e-06 2020-01-16T00:12:38.216 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.216 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.216 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:38.216 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.943841+0000", 2020-01-16T00:12:38.217 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011645399999999999 2020-01-16T00:12:38.217 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.218 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.218 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.218 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.943958+0000", 2020-01-16T00:12:38.218 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7291000000000003e-05 2020-01-16T00:12:38.219 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.219 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.219 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:38.219 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.944025+0000", 2020-01-16T00:12:38.220 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063108449999999998 2020-01-16T00:12:38.220 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.220 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.220 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:38.221 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.950336+0000", 2020-01-16T00:12:38.221 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9874999999999999e-05 2020-01-16T00:12:38.221 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.221 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.222 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:38.222 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.950386+0000", 2020-01-16T00:12:38.222 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0097026960000000002 2020-01-16T00:12:38.222 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.222 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:38.223 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.223 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.223 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.223 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.5.13:43 2.0 e24/19 2:313458df:::smithi12313934-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 24'19, mlcod=24'17)", 2020-01-16T00:12:38.224 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.954776+0000", 2020-01-16T00:12:38.224 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.17306419200000001, 2020-01-16T00:12:38.224 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063434609999999999, 2020-01-16T00:12:38.224 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:38.224 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:38.225 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:38.225 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.225 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:38.225 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.954776+0000", 2020-01-16T00:12:38.226 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:38.226 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.226 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.226 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:38.227 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.954776+0000", 2020-01-16T00:12:38.227 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:38.228 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.228 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.228 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:38.228 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.954773+0000", 2020-01-16T00:12:38.228 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030654699999999998 2020-01-16T00:12:38.229 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.229 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.229 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:38.230 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.957839+0000", 2020-01-16T00:12:38.230 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7915e-05 2020-01-16T00:12:38.230 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.230 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.230 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:38.230 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.957856+0000", 2020-01-16T00:12:38.230 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3810000000000001e-06 2020-01-16T00:12:38.231 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.231 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.231 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:38.231 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.957864+0000", 2020-01-16T00:12:38.231 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3652000000000007e-05 2020-01-16T00:12:38.231 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.231 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.232 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.232 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.957927+0000", 2020-01-16T00:12:38.232 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0889000000000002e-05 2020-01-16T00:12:38.232 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.232 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.232 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:38.233 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.957958+0000", 2020-01-16T00:12:38.233 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031031019999999999 2020-01-16T00:12:38.233 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.233 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.233 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:38.233 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.961061+0000", 2020-01-16T00:12:38.234 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7602000000000002e-05 2020-01-16T00:12:38.234 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.234 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.234 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:38.234 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.961119+0000", 2020-01-16T00:12:38.234 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063434609999999999 2020-01-16T00:12:38.235 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.235 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:38.235 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.235 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.235 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:38.235 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:38.239 INFO:tasks.rados.rados.0.smithi123.stdout:645: finishing write tid 1 to smithi12313934-17 2020-01-16T00:12:38.239 INFO:tasks.rados.rados.0.smithi123.stdout:645: finishing write tid 2 to smithi12313934-17 2020-01-16T00:12:38.240 INFO:tasks.rados.rados.0.smithi123.stdout:645: finishing write tid 4 to smithi12313934-17 2020-01-16T00:12:38.240 INFO:tasks.rados.rados.0.smithi123.stdout:645: finishing write tid 5 to smithi12313934-17 2020-01-16T00:12:38.240 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 17 v 58 (ObjNum 71 snap 0 seq_num 71) dirty exists 2020-01-16T00:12:38.241 INFO:tasks.rados.rados.0.smithi123.stdout:645: left oid 17 (ObjNum 71 snap 0 seq_num 71) 2020-01-16T00:12:38.241 INFO:tasks.rados.rados.0.smithi123.stdout:644: done (15 left) 2020-01-16T00:12:38.241 INFO:tasks.rados.rados.0.smithi123.stdout:645: done (14 left) 2020-01-16T00:12:38.241 INFO:tasks.rados.rados.0.smithi123.stdout:660: write oid 41 current snap is 0 2020-01-16T00:12:38.242 INFO:tasks.rados.rados.0.smithi123.stdout:660: seq_num 79 ranges {527753=708201,1856548=737690,3286534=41219} 2020-01-16T00:12:38.242 INFO:tasks.rados.rados.0.smithi123.stdout:660: writing smithi12313934-41 from 527753 to 1235954 tid 1 2020-01-16T00:12:38.242 INFO:tasks.rados.rados.0.smithi123.stdout:660: writing smithi12313934-41 from 1856548 to 2594238 tid 2 2020-01-16T00:12:38.242 INFO:tasks.rados.rados.0.smithi123.stdout:660: writing smithi12313934-41 from 3286534 to 3327753 tid 3 2020-01-16T00:12:38.243 INFO:tasks.rados.rados.0.smithi123.stdout:661: read oid 6 snap -1 2020-01-16T00:12:38.243 INFO:tasks.rados.rados.0.smithi123.stdout:661: expect (ObjNum 5 snap 0 seq_num 5) 2020-01-16T00:12:38.243 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:38.244 INFO:tasks.rados.rados.0.smithi123.stdout:646: finishing write tid 1 to smithi12313934-27 2020-01-16T00:12:38.244 INFO:tasks.rados.rados.0.smithi123.stdout:646: finishing write tid 2 to smithi12313934-27 2020-01-16T00:12:38.244 INFO:tasks.rados.rados.0.smithi123.stdout:646: finishing write tid 3 to smithi12313934-27 2020-01-16T00:12:38.244 INFO:tasks.rados.rados.0.smithi123.stdout:646: finishing write tid 5 to smithi12313934-27 2020-01-16T00:12:38.245 INFO:tasks.rados.rados.0.smithi123.stdout:646: finishing write tid 6 to smithi12313934-27 2020-01-16T00:12:38.245 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 27 v 52 (ObjNum 72 snap 0 seq_num 72) dirty exists 2020-01-16T00:12:38.245 INFO:tasks.rados.rados.0.smithi123.stdout:646: left oid 27 (ObjNum 72 snap 0 seq_num 72) 2020-01-16T00:12:38.246 INFO:tasks.rados.rados.0.smithi123.stdout:648: expect (ObjNum 59 snap 0 seq_num 59) 2020-01-16T00:12:38.246 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-01-16T00:12:38.247 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:12:38.237+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:12:38.289 INFO:tasks.rados.rados.0.smithi123.stdout:647: finishing write tid 1 to smithi12313934-28 2020-01-16T00:12:38.289 INFO:tasks.rados.rados.0.smithi123.stdout:647: finishing write tid 2 to smithi12313934-28 2020-01-16T00:12:38.289 INFO:tasks.rados.rados.0.smithi123.stdout:646: done (15 left) 2020-01-16T00:12:38.289 INFO:tasks.rados.rados.0.smithi123.stdout:648: done (14 left) 2020-01-16T00:12:38.290 INFO:tasks.rados.rados.0.smithi123.stdout:662: write oid 9 current snap is 0 2020-01-16T00:12:38.290 INFO:tasks.rados.rados.0.smithi123.stdout:662: seq_num 80 ranges {544560=783450,1915030=481348,2797604=479761} 2020-01-16T00:12:38.299 INFO:tasks.rados.rados.0.smithi123.stdout:662: writing smithi12313934-9 from 544560 to 1328010 tid 1 2020-01-16T00:12:38.306 INFO:tasks.rados.rados.0.smithi123.stdout:662: writing smithi12313934-9 from 1915030 to 2396378 tid 2 2020-01-16T00:12:38.312 INFO:tasks.rados.rados.0.smithi123.stdout:662: writing smithi12313934-9 from 2797604 to 3277365 tid 3 2020-01-16T00:12:38.312 INFO:tasks.rados.rados.0.smithi123.stdout:663: read oid 27 snap -1 2020-01-16T00:12:38.313 INFO:tasks.rados.rados.0.smithi123.stdout:663: expect (ObjNum 72 snap 0 seq_num 72) 2020-01-16T00:12:38.313 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:38.313 INFO:tasks.rados.rados.0.smithi123.stdout:647: finishing write tid 3 to smithi12313934-28 2020-01-16T00:12:38.313 INFO:tasks.rados.rados.0.smithi123.stdout:647: finishing write tid 5 to smithi12313934-28 2020-01-16T00:12:38.313 INFO:tasks.rados.rados.0.smithi123.stdout:647: finishing write tid 6 to smithi12313934-28 2020-01-16T00:12:38.313 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 28 v 71 (ObjNum 73 snap 0 seq_num 73) dirty exists 2020-01-16T00:12:38.314 INFO:tasks.rados.rados.0.smithi123.stdout:647: left oid 28 (ObjNum 73 snap 0 seq_num 73) 2020-01-16T00:12:38.314 INFO:tasks.rados.rados.0.smithi123.stdout:649: finishing write tid 1 to smithi12313934-12 2020-01-16T00:12:38.314 INFO:tasks.rados.rados.0.smithi123.stdout:647: done (15 left) 2020-01-16T00:12:38.314 INFO:tasks.rados.rados.0.smithi123.stdout:664: read oid 44 snap -1 2020-01-16T00:12:38.314 INFO:tasks.rados.rados.0.smithi123.stdout:664: expect (ObjNum 43 snap 0 seq_num 43) 2020-01-16T00:12:38.314 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:38.314 INFO:tasks.rados.rados.0.smithi123.stdout:651: expect (ObjNum 21 snap 0 seq_num 21) 2020-01-16T00:12:38.338 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:12:38.337+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:12:38.386 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:38.386 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:12:38.386 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:12:38.386 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:38.388 INFO:tasks.rados.rados.0.smithi123.stdout:650: expect (ObjNum 2 snap 0 seq_num 2) 2020-01-16T00:12:38.394 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-01-16T00:12:38.450 INFO:tasks.rados.rados.0.smithi123.stdout:649: finishing write tid 2 to smithi12313934-12 2020-01-16T00:12:38.450 INFO:tasks.rados.rados.0.smithi123.stdout:650: done (15 left) 2020-01-16T00:12:38.450 INFO:tasks.rados.rados.0.smithi123.stdout:651: done (14 left) 2020-01-16T00:12:38.450 INFO:tasks.rados.rados.0.smithi123.stdout:665: write oid 40 current snap is 0 2020-01-16T00:12:38.451 INFO:tasks.rados.rados.0.smithi123.stdout:665: seq_num 81 ranges {561367=458699,1573512=741359,2941380=419987} 2020-01-16T00:12:38.451 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:12:38.451+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:12:38.464 INFO:tasks.rados.rados.0.smithi123.stdout:665: writing smithi12313934-40 from 561367 to 1020066 tid 1 2020-01-16T00:12:38.471 INFO:tasks.rados.rados.0.smithi123.stdout:665: writing smithi12313934-40 from 1573512 to 2314871 tid 2 2020-01-16T00:12:38.478 INFO:tasks.rados.rados.0.smithi123.stdout:665: writing smithi12313934-40 from 2941380 to 3361367 tid 3 2020-01-16T00:12:38.479 INFO:tasks.rados.rados.0.smithi123.stdout:666: read oid 22 snap -1 2020-01-16T00:12:38.479 INFO:tasks.rados.rados.0.smithi123.stdout:666: expect (ObjNum 21 snap 0 seq_num 21) 2020-01-16T00:12:38.479 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:38.479 INFO:tasks.rados.rados.0.smithi123.stdout:649: finishing write tid 3 to smithi12313934-12 2020-01-16T00:12:38.479 INFO:tasks.rados.rados.0.smithi123.stdout:649: finishing write tid 5 to smithi12313934-12 2020-01-16T00:12:38.479 INFO:tasks.rados.rados.0.smithi123.stdout:649: finishing write tid 6 to smithi12313934-12 2020-01-16T00:12:38.480 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 12 v 44 (ObjNum 74 snap 0 seq_num 74) dirty exists 2020-01-16T00:12:38.480 INFO:tasks.rados.rados.0.smithi123.stdout:649: left oid 12 (ObjNum 74 snap 0 seq_num 74) 2020-01-16T00:12:38.480 INFO:tasks.rados.rados.0.smithi123.stdout:649: done (15 left) 2020-01-16T00:12:38.480 INFO:tasks.rados.rados.0.smithi123.stdout:667: read oid 33 snap -1 2020-01-16T00:12:38.480 INFO:tasks.rados.rados.0.smithi123.stdout:667: expect (ObjNum 54 snap 0 seq_num 54) 2020-01-16T00:12:38.480 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:38.480 INFO:tasks.rados.rados.0.smithi123.stdout:652: finishing write tid 1 to smithi12313934-48 2020-01-16T00:12:38.481 INFO:tasks.rados.rados.0.smithi123.stdout:652: finishing write tid 2 to smithi12313934-48 2020-01-16T00:12:38.481 INFO:tasks.rados.rados.0.smithi123.stdout:652: finishing write tid 3 to smithi12313934-48 2020-01-16T00:12:38.481 INFO:tasks.rados.rados.0.smithi123.stdout:652: finishing write tid 5 to smithi12313934-48 2020-01-16T00:12:38.481 INFO:tasks.rados.rados.0.smithi123.stdout:652: finishing write tid 6 to smithi12313934-48 2020-01-16T00:12:38.481 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 48 v 55 (ObjNum 75 snap 0 seq_num 75) dirty exists 2020-01-16T00:12:38.481 INFO:tasks.rados.rados.0.smithi123.stdout:652: left oid 48 (ObjNum 75 snap 0 seq_num 75) 2020-01-16T00:12:38.482 INFO:tasks.rados.rados.0.smithi123.stdout:653: finishing write tid 1 to smithi12313934-4 2020-01-16T00:12:38.482 INFO:tasks.rados.rados.0.smithi123.stdout:652: done (15 left) 2020-01-16T00:12:38.482 INFO:tasks.rados.rados.0.smithi123.stdout:668: write (excl) oid 35 current snap is 0 2020-01-16T00:12:38.482 INFO:tasks.rados.rados.0.smithi123.stdout:668: seq_num 82 ranges {578174=533948,1515641=485017,2736097=502305} 2020-01-16T00:12:38.486 INFO:tasks.rados.rados.0.smithi123.stdout:668: writing smithi12313934-35 from 578174 to 1112122 tid 1 2020-01-16T00:12:38.492 INFO:tasks.rados.rados.0.smithi123.stdout:668: writing smithi12313934-35 from 1515641 to 2000658 tid 2 2020-01-16T00:12:38.500 INFO:tasks.rados.rados.0.smithi123.stdout:668: writing smithi12313934-35 from 2736097 to 3238402 tid 3 2020-01-16T00:12:38.500 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:38.500 INFO:tasks.rados.rados.0.smithi123.stdout:655: expect (ObjNum 6 snap 0 seq_num 6) 2020-01-16T00:12:38.541 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:38.543+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:12:38.551 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:38.552 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:12:38.552 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:12:38.552 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:12:38.552 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:38.561 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-01-16T00:12:38.568 INFO:tasks.rados.rados.0.smithi123.stdout:656: expect (ObjNum 20 snap 0 seq_num 20) 2020-01-16T00:12:38.641 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:12:38.641+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:12:38.643 INFO:tasks.rados.rados.0.smithi123.stdout:653: finishing write tid 2 to smithi12313934-4 2020-01-16T00:12:38.643 INFO:tasks.rados.rados.0.smithi123.stdout:653: finishing write tid 3 to smithi12313934-4 2020-01-16T00:12:38.643 INFO:tasks.rados.rados.0.smithi123.stdout:653: finishing write tid 5 to smithi12313934-4 2020-01-16T00:12:38.643 INFO:tasks.rados.rados.0.smithi123.stdout:653: finishing write tid 6 to smithi12313934-4 2020-01-16T00:12:38.644 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 4 v 64 (ObjNum 76 snap 0 seq_num 76) dirty exists 2020-01-16T00:12:38.644 INFO:tasks.rados.rados.0.smithi123.stdout:653: left oid 4 (ObjNum 76 snap 0 seq_num 76) 2020-01-16T00:12:38.644 INFO:tasks.rados.rados.0.smithi123.stdout:654: expect (ObjNum 13 snap 0 seq_num 13) 2020-01-16T00:12:38.673 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:38.674 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:12:38.674 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:12:38.674 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:12:38.674 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.674 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.5.0:146 3.c e24/20 3:3d6543e6:::smithi12313934-28:head v 24'67, mlcod=24'65)", 2020-01-16T00:12:38.675 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.849191+0000", 2020-01-16T00:12:38.675 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.82312529000000001, 2020-01-16T00:12:38.675 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0088840970000000005, 2020-01-16T00:12:38.675 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:38.675 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:38.675 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:38.676 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.676 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:38.676 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.849191+0000", 2020-01-16T00:12:38.676 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:38.676 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.676 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.676 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:38.676 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.849191+0000", 2020-01-16T00:12:38.677 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:38.677 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.677 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.677 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:38.677 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.849188+0000", 2020-01-16T00:12:38.677 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0024105039999999999 2020-01-16T00:12:38.678 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.678 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.678 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:38.678 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.851599+0000", 2020-01-16T00:12:38.678 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0900999999999998e-05 2020-01-16T00:12:38.678 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.678 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.679 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:38.679 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.851660+0000", 2020-01-16T00:12:38.679 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0473e-05 2020-01-16T00:12:38.679 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.679 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.679 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:38.679 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.851670+0000", 2020-01-16T00:12:38.679 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8696999999999997e-05 2020-01-16T00:12:38.680 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.680 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.680 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.680 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.851729+0000", 2020-01-16T00:12:38.680 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3631999999999999e-05 2020-01-16T00:12:38.680 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.680 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.681 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:38.681 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.851763+0000", 2020-01-16T00:12:38.681 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0062692039999999996 2020-01-16T00:12:38.681 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.681 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.681 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:38.681 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.858032+0000", 2020-01-16T00:12:38.682 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2969999999999997e-05 2020-01-16T00:12:38.682 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.682 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.682 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:38.682 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.858075+0000", 2020-01-16T00:12:38.682 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0088840970000000005 2020-01-16T00:12:38.682 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.682 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:38.683 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.683 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.683 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.683 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:922 3.8 3:113947f2:::smithi12313934-12:head [write 443718~615603,stat] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:38.683 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.863202+0000", 2020-01-16T00:12:38.683 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.80911370699999996, 2020-01-16T00:12:38.683 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017152697000000001, 2020-01-16T00:12:38.684 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:38.684 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:38.684 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:38.684 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:38.684 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:38.684 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 922 2020-01-16T00:12:38.684 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.684 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:38.685 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.685 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:38.685 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863202+0000", 2020-01-16T00:12:38.685 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:38.685 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.686 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.686 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:38.686 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863202+0000", 2020-01-16T00:12:38.686 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:38.686 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.686 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.686 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:38.687 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863200+0000", 2020-01-16T00:12:38.687 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00038569199999999998 2020-01-16T00:12:38.687 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.687 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.687 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:38.687 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863585+0000", 2020-01-16T00:12:38.687 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5685e-05 2020-01-16T00:12:38.687 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.688 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.688 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:38.688 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863601+0000", 2020-01-16T00:12:38.688 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.4734000000000001e-05 2020-01-16T00:12:38.688 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.688 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.689 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:38.689 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863626+0000", 2020-01-16T00:12:38.689 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1037e-05 2020-01-16T00:12:38.689 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.689 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.689 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.689 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863667+0000", 2020-01-16T00:12:38.690 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016041499999999999 2020-01-16T00:12:38.690 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.690 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.690 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:38.690 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863827+0000", 2020-01-16T00:12:38.690 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0094286430000000004 2020-01-16T00:12:38.690 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.691 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.691 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.691 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.873256+0000", 2020-01-16T00:12:38.691 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2867999999999994e-05 2020-01-16T00:12:38.691 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.691 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.691 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:38.691 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.873329+0000", 2020-01-16T00:12:38.692 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021174789999999998 2020-01-16T00:12:38.692 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.692 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.692 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.692 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.875446+0000", 2020-01-16T00:12:38.692 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011244200000000001 2020-01-16T00:12:38.692 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.693 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.693 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:38.693 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.875559+0000", 2020-01-16T00:12:38.693 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021772600000000001 2020-01-16T00:12:38.693 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.693 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.693 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1,7", 2020-01-16T00:12:38.694 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.875776+0000", 2020-01-16T00:12:38.694 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0028978860000000001 2020-01-16T00:12:38.694 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.694 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.694 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:38.694 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.878674+0000", 2020-01-16T00:12:38.694 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.324e-05 2020-01-16T00:12:38.694 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.695 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.695 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:38.695 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.878687+0000", 2020-01-16T00:12:38.695 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0014896180000000001 2020-01-16T00:12:38.695 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.696 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.696 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:38.696 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880177+0000", 2020-01-16T00:12:38.696 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6881000000000001e-05 2020-01-16T00:12:38.696 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.696 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.696 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:38.696 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880224+0000", 2020-01-16T00:12:38.697 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013096700000000001 2020-01-16T00:12:38.697 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.697 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.697 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:38.697 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880355+0000", 2020-01-16T00:12:38.698 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017152697000000001 2020-01-16T00:12:38.698 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.698 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:38.698 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.698 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.698 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.699 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:915 3.c e24/20 3:3d6543e6:::smithi12313934-28:head v 24'69, mlcod=24'67)", 2020-01-16T00:12:38.699 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.865476+0000", 2020-01-16T00:12:38.699 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.80684036199999998, 2020-01-16T00:12:38.699 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040973210000000001, 2020-01-16T00:12:38.700 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:38.700 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:38.700 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:38.700 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.700 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:38.701 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.865476+0000", 2020-01-16T00:12:38.701 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:38.701 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.701 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.702 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:38.702 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.865476+0000", 2020-01-16T00:12:38.702 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:12:38.702 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.702 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.702 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:38.703 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.865473+0000", 2020-01-16T00:12:38.703 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013264819999999999 2020-01-16T00:12:38.703 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.703 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.703 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:38.704 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.866799+0000", 2020-01-16T00:12:38.704 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.4854999999999998e-05 2020-01-16T00:12:38.704 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.704 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.704 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:38.704 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.866824+0000", 2020-01-16T00:12:38.704 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1210000000000001e-06 2020-01-16T00:12:38.704 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.705 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.705 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:38.705 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.866831+0000", 2020-01-16T00:12:38.705 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0987e-05 2020-01-16T00:12:38.705 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.706 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.706 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.706 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.866862+0000", 2020-01-16T00:12:38.706 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6844999999999999e-05 2020-01-16T00:12:38.706 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.706 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.706 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:38.707 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.866879+0000", 2020-01-16T00:12:38.707 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0026476830000000001 2020-01-16T00:12:38.707 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.707 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.707 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:38.707 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869527+0000", 2020-01-16T00:12:38.707 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6155000000000001e-05 2020-01-16T00:12:38.707 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.708 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.708 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:38.708 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869573+0000", 2020-01-16T00:12:38.708 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040973210000000001 2020-01-16T00:12:38.708 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.708 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:38.708 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.709 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.709 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.709 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:923 3.8 3:113947f2:::smithi12313934-12:head [write 1564138~470341] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:38.709 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.869292+0000", 2020-01-16T00:12:38.710 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.80302337700000004, 2020-01-16T00:12:38.710 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017661620999999999, 2020-01-16T00:12:38.710 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:38.710 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:38.711 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:38.711 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:38.711 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:38.711 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 923 2020-01-16T00:12:38.711 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.712 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:38.712 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.712 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:38.712 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869292+0000", 2020-01-16T00:12:38.713 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:38.713 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.713 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.714 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:38.715 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869292+0000", 2020-01-16T00:12:38.715 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:38.715 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.715 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.716 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:38.716 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869291+0000", 2020-01-16T00:12:38.716 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00049842899999999999 2020-01-16T00:12:38.716 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.717 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.717 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:38.717 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869789+0000", 2020-01-16T00:12:38.718 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1014999999999999e-05 2020-01-16T00:12:38.718 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.719 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.719 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:38.719 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869830+0000", 2020-01-16T00:12:38.719 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7880000000000007e-06 2020-01-16T00:12:38.720 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.720 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.720 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:38.720 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869838+0000", 2020-01-16T00:12:38.720 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8346999999999997e-05 2020-01-16T00:12:38.720 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.720 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.721 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.721 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869876+0000", 2020-01-16T00:12:38.721 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9726999999999999e-05 2020-01-16T00:12:38.721 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.721 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.721 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:38.721 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869936+0000", 2020-01-16T00:12:38.722 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003434917 2020-01-16T00:12:38.722 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.722 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.722 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.722 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.873371+0000", 2020-01-16T00:12:38.722 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118808 2020-01-16T00:12:38.722 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.723 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.723 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:38.723 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.873490+0000", 2020-01-16T00:12:38.723 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036712910000000001 2020-01-16T00:12:38.723 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.723 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.723 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.723 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877161+0000", 2020-01-16T00:12:38.724 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9909999999999999e-05 2020-01-16T00:12:38.724 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.724 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.724 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:38.724 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877241+0000", 2020-01-16T00:12:38.724 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031619970000000002 2020-01-16T00:12:38.724 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.724 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.725 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.725 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880403+0000", 2020-01-16T00:12:38.725 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013071099999999999 2020-01-16T00:12:38.725 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.725 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.725 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:38.725 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880534+0000", 2020-01-16T00:12:38.726 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00032823500000000002 2020-01-16T00:12:38.726 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.726 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.726 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1,7", 2020-01-16T00:12:38.726 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880862+0000", 2020-01-16T00:12:38.726 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00513862 2020-01-16T00:12:38.727 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.727 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.727 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:38.727 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886001+0000", 2020-01-16T00:12:38.727 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8869000000000001e-05 2020-01-16T00:12:38.727 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.727 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.727 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:38.728 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886030+0000", 2020-01-16T00:12:38.728 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00081600200000000003 2020-01-16T00:12:38.728 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.728 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.728 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:38.729 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886846+0000", 2020-01-16T00:12:38.729 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4095999999999998e-05 2020-01-16T00:12:38.729 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.729 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.729 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:38.729 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886900+0000", 2020-01-16T00:12:38.730 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.448e-05 2020-01-16T00:12:38.730 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.730 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.730 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:38.730 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886954+0000", 2020-01-16T00:12:38.730 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017661620999999999 2020-01-16T00:12:38.730 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.731 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:38.731 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.731 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.731 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.731 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:924 3.8 3:113947f2:::smithi12313934-12:head [write 2633066~610652] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:38.731 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.876207+0000", 2020-01-16T00:12:38.731 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.79610922299999998, 2020-01-16T00:12:38.731 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011069638, 2020-01-16T00:12:38.732 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:38.732 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:38.732 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:38.733 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:38.733 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:38.733 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 924 2020-01-16T00:12:38.733 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.733 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:38.733 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.734 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:38.734 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876207+0000", 2020-01-16T00:12:38.734 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:38.734 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.734 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.734 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:38.734 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876207+0000", 2020-01-16T00:12:38.735 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:38.735 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.735 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.735 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:38.735 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876204+0000", 2020-01-16T00:12:38.735 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000157842 2020-01-16T00:12:38.735 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.735 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.736 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:38.736 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876362+0000", 2020-01-16T00:12:38.736 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0181e-05 2020-01-16T00:12:38.736 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.737 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.737 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:38.737 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876372+0000", 2020-01-16T00:12:38.737 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2409999999999997e-06 2020-01-16T00:12:38.737 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.737 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.737 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:38.738 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876377+0000", 2020-01-16T00:12:38.738 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00089908200000000005 2020-01-16T00:12:38.738 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.738 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.738 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.738 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877276+0000", 2020-01-16T00:12:38.738 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2391000000000006e-05 2020-01-16T00:12:38.739 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.739 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.739 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:38.739 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877338+0000", 2020-01-16T00:12:38.739 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0062765850000000003 2020-01-16T00:12:38.740 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.740 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.741 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.741 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.883615+0000", 2020-01-16T00:12:38.742 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011738700000000001 2020-01-16T00:12:38.742 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.742 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.742 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:38.742 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.883732+0000", 2020-01-16T00:12:38.742 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023677 2020-01-16T00:12:38.742 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.743 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.743 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1,7", 2020-01-16T00:12:38.743 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.883969+0000", 2020-01-16T00:12:38.743 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0028104979999999998 2020-01-16T00:12:38.743 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.743 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.743 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:38.744 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886779+0000", 2020-01-16T00:12:38.746 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00025251899999999999 2020-01-16T00:12:38.746 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.746 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.746 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:38.746 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.887032+0000", 2020-01-16T00:12:38.746 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018013099999999999 2020-01-16T00:12:38.747 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.747 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.747 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:38.747 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.887212+0000", 2020-01-16T00:12:38.747 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9646e-05 2020-01-16T00:12:38.747 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.747 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.748 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:38.748 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.887242+0000", 2020-01-16T00:12:38.748 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4551000000000002e-05 2020-01-16T00:12:38.748 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.748 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.748 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:38.748 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.887276+0000", 2020-01-16T00:12:38.748 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011069638 2020-01-16T00:12:38.749 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.749 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:38.749 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.749 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.749 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.749 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:925 3.8 3:113947f2:::smithi12313934-12:head [setxattr _header (56) in=7b,truncate 3243718] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:38.749 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.876384+0000", 2020-01-16T00:12:38.750 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.79593217599999999, 2020-01-16T00:12:38.750 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011601155, 2020-01-16T00:12:38.750 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:38.750 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:38.750 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:38.750 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:38.751 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:38.751 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 925 2020-01-16T00:12:38.751 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.751 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:38.751 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.751 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:38.751 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876384+0000", 2020-01-16T00:12:38.752 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:38.752 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.752 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.752 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:38.752 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876384+0000", 2020-01-16T00:12:38.752 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:38.752 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.752 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.753 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:38.753 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876383+0000", 2020-01-16T00:12:38.753 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.946e-06 2020-01-16T00:12:38.753 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.753 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.753 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:38.753 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876387+0000", 2020-01-16T00:12:38.753 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7179999999999998e-06 2020-01-16T00:12:38.754 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.754 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.754 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:38.754 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876391+0000", 2020-01-16T00:12:38.754 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3980000000000002e-06 2020-01-16T00:12:38.754 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.754 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.755 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:38.755 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876393+0000", 2020-01-16T00:12:38.755 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00097104099999999996 2020-01-16T00:12:38.755 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.755 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.755 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.755 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877364+0000", 2020-01-16T00:12:38.756 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6619000000000001e-05 2020-01-16T00:12:38.756 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.756 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.756 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:38.756 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877421+0000", 2020-01-16T00:12:38.756 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0078613510000000008 2020-01-16T00:12:38.756 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.757 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.757 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.757 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.885282+0000", 2020-01-16T00:12:38.757 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015531400000000001 2020-01-16T00:12:38.757 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.757 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.758 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:38.758 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.885437+0000", 2020-01-16T00:12:38.758 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019132099999999999 2020-01-16T00:12:38.758 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.758 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.758 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1,7", 2020-01-16T00:12:38.758 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.885629+0000", 2020-01-16T00:12:38.759 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0011787270000000001 2020-01-16T00:12:38.759 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.759 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.759 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:38.759 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886807+0000", 2020-01-16T00:12:38.760 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000739092 2020-01-16T00:12:38.760 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.760 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.760 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:38.761 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.887547+0000", 2020-01-16T00:12:38.761 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00037348900000000001 2020-01-16T00:12:38.761 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.761 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.762 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:38.762 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.887920+0000", 2020-01-16T00:12:38.762 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9629999999999999e-05 2020-01-16T00:12:38.763 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.763 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.763 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:38.763 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.887950+0000", 2020-01-16T00:12:38.764 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5172999999999997e-05 2020-01-16T00:12:38.764 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.764 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.764 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:38.764 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.887985+0000", 2020-01-16T00:12:38.764 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011601155 2020-01-16T00:12:38.764 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.765 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:38.765 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.765 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.765 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.765 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:926 3.8 3:113947f2:::smithi12313934-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:12:38.765 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.876396+0000", 2020-01-16T00:12:38.765 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.79592030700000005, 2020-01-16T00:12:38.765 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011788966999999999, 2020-01-16T00:12:38.766 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:38.766 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:38.766 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:38.766 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:38.766 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:38.766 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 926 2020-01-16T00:12:38.766 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.767 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:38.767 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.767 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:38.767 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876396+0000", 2020-01-16T00:12:38.767 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:38.767 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.767 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.767 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:38.768 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876396+0000", 2020-01-16T00:12:38.768 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:38.768 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.768 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.768 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:38.768 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876395+0000", 2020-01-16T00:12:38.768 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9369999999999998e-06 2020-01-16T00:12:38.768 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.769 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.769 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:38.769 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876397+0000", 2020-01-16T00:12:38.769 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2839999999999998e-06 2020-01-16T00:12:38.769 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.769 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.769 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:38.770 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876400+0000", 2020-01-16T00:12:38.770 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2060000000000001e-06 2020-01-16T00:12:38.770 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.770 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.770 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:38.770 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876403+0000", 2020-01-16T00:12:38.770 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010315179999999999 2020-01-16T00:12:38.770 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.771 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.771 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.771 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877434+0000", 2020-01-16T00:12:38.771 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0519999999999997e-05 2020-01-16T00:12:38.771 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.771 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.771 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:38.771 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877485+0000", 2020-01-16T00:12:38.772 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0084118030000000007 2020-01-16T00:12:38.772 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.772 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.772 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.772 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.885897+0000", 2020-01-16T00:12:38.772 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6395000000000001e-05 2020-01-16T00:12:38.772 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.772 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.773 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:38.773 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.885993+0000", 2020-01-16T00:12:38.773 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020290849999999999 2020-01-16T00:12:38.773 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.773 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.773 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.773 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.888022+0000", 2020-01-16T00:12:38.773 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7325999999999998e-05 2020-01-16T00:12:38.774 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.774 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.774 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:38.774 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.888099+0000", 2020-01-16T00:12:38.774 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5185999999999994e-05 2020-01-16T00:12:38.774 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.774 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.774 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:38.775 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.888185+0000", 2020-01-16T00:12:38.775 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011788966999999999 2020-01-16T00:12:38.775 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.775 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:38.775 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.775 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.775 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.776 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:927 3.b 3:df203f2a:::smithi12313934-3:head [read 0~2033614,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:38.776 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.876405+0000", 2020-01-16T00:12:38.776 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.79591078400000004, 2020-01-16T00:12:38.776 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.005475728, 2020-01-16T00:12:38.776 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:38.776 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:38.776 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:38.777 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:38.777 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:38.777 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 927 2020-01-16T00:12:38.777 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.777 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:38.777 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.777 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:38.777 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876405+0000", 2020-01-16T00:12:38.778 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:38.778 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.778 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.778 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:38.778 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876405+0000", 2020-01-16T00:12:38.779 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:38.779 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.779 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.780 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:38.780 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876405+0000", 2020-01-16T00:12:38.780 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7439999999999999e-06 2020-01-16T00:12:38.780 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.780 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.780 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:38.780 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876408+0000", 2020-01-16T00:12:38.781 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7379999999999999e-06 2020-01-16T00:12:38.781 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.781 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.781 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:38.781 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876410+0000", 2020-01-16T00:12:38.781 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8619999999999999e-06 2020-01-16T00:12:38.782 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.782 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.782 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:38.782 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876412+0000", 2020-01-16T00:12:38.782 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8871000000000001e-05 2020-01-16T00:12:38.782 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.782 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.783 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.783 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876451+0000", 2020-01-16T00:12:38.783 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9624999999999997e-05 2020-01-16T00:12:38.783 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.783 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.783 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:38.783 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876541+0000", 2020-01-16T00:12:38.784 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0053401639999999997 2020-01-16T00:12:38.784 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.784 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.784 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:38.784 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.881881+0000", 2020-01-16T00:12:38.784 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.005475728 2020-01-16T00:12:38.784 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.784 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:38.785 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.785 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.785 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.785 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:928 3.b 3:df203f2a:::smithi12313934-3:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:38.785 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.876426+0000", 2020-01-16T00:12:38.785 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.79588983800000002, 2020-01-16T00:12:38.785 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0070152909999999999, 2020-01-16T00:12:38.786 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:38.786 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:38.786 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:38.786 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:38.786 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:38.786 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 928 2020-01-16T00:12:38.786 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.786 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:38.787 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.787 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:38.787 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876426+0000", 2020-01-16T00:12:38.787 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:38.787 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.787 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.788 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:38.789 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876426+0000", 2020-01-16T00:12:38.789 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:38.789 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.790 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.790 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:38.790 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876426+0000", 2020-01-16T00:12:38.790 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.18e-06 2020-01-16T00:12:38.790 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.790 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.790 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:38.790 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876429+0000", 2020-01-16T00:12:38.791 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2399999999999998e-06 2020-01-16T00:12:38.791 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.791 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.791 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:38.791 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876434+0000", 2020-01-16T00:12:38.792 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1330000000000001e-06 2020-01-16T00:12:38.792 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.792 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.792 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:38.792 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876441+0000", 2020-01-16T00:12:38.792 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0064132030000000001 2020-01-16T00:12:38.792 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.792 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.793 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.793 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.882854+0000", 2020-01-16T00:12:38.793 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016152500000000001 2020-01-16T00:12:38.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.793 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.793 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:38.793 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.883016+0000", 2020-01-16T00:12:38.794 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00042550800000000001 2020-01-16T00:12:38.794 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.794 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.794 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:38.794 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.883441+0000", 2020-01-16T00:12:38.794 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0070152909999999999 2020-01-16T00:12:38.794 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.795 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:38.795 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.795 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.795 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.795 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:929 3.b 3:df203f2a:::smithi12313934-3:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:38.795 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.876448+0000", 2020-01-16T00:12:38.795 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.79586760499999998, 2020-01-16T00:12:38.795 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0076937330000000003, 2020-01-16T00:12:38.796 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:38.796 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:38.796 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:38.796 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:38.796 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:38.796 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 929 2020-01-16T00:12:38.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.797 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:38.797 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.797 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:38.797 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876448+0000", 2020-01-16T00:12:38.797 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:38.797 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.797 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.798 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:38.798 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876448+0000", 2020-01-16T00:12:38.798 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:38.798 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.798 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.799 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:38.799 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876448+0000", 2020-01-16T00:12:38.799 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9470000000000002e-06 2020-01-16T00:12:38.799 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.799 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.799 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:38.799 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876453+0000", 2020-01-16T00:12:38.800 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9840000000000001e-06 2020-01-16T00:12:38.800 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.800 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.800 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:38.800 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876458+0000", 2020-01-16T00:12:38.800 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1879999999999998e-06 2020-01-16T00:12:38.801 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.801 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.801 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:38.801 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876461+0000", 2020-01-16T00:12:38.801 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0070321639999999996 2020-01-16T00:12:38.801 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.802 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.802 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.802 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.883493+0000", 2020-01-16T00:12:38.802 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021568600000000001 2020-01-16T00:12:38.803 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.803 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.803 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:38.803 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.883709+0000", 2020-01-16T00:12:38.803 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043327600000000002 2020-01-16T00:12:38.803 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.803 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.804 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:38.804 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.884142+0000", 2020-01-16T00:12:38.804 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0076937330000000003 2020-01-16T00:12:38.804 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.805 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:38.805 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.805 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.805 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.806 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:933 3.2 3:4f8fe446:::smithi12313934-48:head [write 460525~690852,stat] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:38.806 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.883885+0000", 2020-01-16T00:12:38.806 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.78843082200000003, 2020-01-16T00:12:38.806 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.004940262, 2020-01-16T00:12:38.806 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:38.806 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:38.806 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:38.807 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:38.807 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:38.807 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 933 2020-01-16T00:12:38.807 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.807 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:38.807 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.807 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:38.808 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.883885+0000", 2020-01-16T00:12:38.808 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:38.808 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.808 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.808 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:38.809 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.883885+0000", 2020-01-16T00:12:38.809 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:12:38.809 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.809 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.809 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:38.810 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.883882+0000", 2020-01-16T00:12:38.810 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00024325799999999999 2020-01-16T00:12:38.810 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.810 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.811 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:38.811 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.884126+0000", 2020-01-16T00:12:38.811 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3713999999999999e-05 2020-01-16T00:12:38.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.811 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.812 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:38.812 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.884139+0000", 2020-01-16T00:12:38.812 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9119999999999998e-06 2020-01-16T00:12:38.813 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.813 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.813 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:38.813 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.884147+0000", 2020-01-16T00:12:38.813 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6669999999999998e-05 2020-01-16T00:12:38.814 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.814 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.814 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.814 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.884204+0000", 2020-01-16T00:12:38.815 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000114109 2020-01-16T00:12:38.815 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.815 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.815 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:38.816 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.884318+0000", 2020-01-16T00:12:38.816 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017802200000000001 2020-01-16T00:12:38.816 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.816 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.816 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:12:38.816 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.884496+0000", 2020-01-16T00:12:38.817 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0027106579999999999 2020-01-16T00:12:38.817 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.817 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.817 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:38.817 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.887207+0000", 2020-01-16T00:12:38.818 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001427967 2020-01-16T00:12:38.818 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.818 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.818 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:38.818 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.888635+0000", 2020-01-16T00:12:38.819 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011501 2020-01-16T00:12:38.819 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.819 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.819 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:38.819 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.888750+0000", 2020-01-16T00:12:38.820 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0114999999999998e-05 2020-01-16T00:12:38.820 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.820 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.820 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:38.820 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.888790+0000", 2020-01-16T00:12:38.820 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5608000000000002e-05 2020-01-16T00:12:38.821 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.821 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.821 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:38.821 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.888825+0000", 2020-01-16T00:12:38.821 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.004940262 2020-01-16T00:12:38.822 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.822 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:38.822 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.822 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.823 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.823 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:934 3.2 3:4f8fe446:::smithi12313934-48:head [write 1622620~730352] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:38.823 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.890610+0000", 2020-01-16T00:12:38.823 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.78170566399999997, 2020-01-16T00:12:38.823 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041613839999999997, 2020-01-16T00:12:38.823 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:38.823 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:38.824 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:38.824 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:38.824 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:38.824 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 934 2020-01-16T00:12:38.824 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.824 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:38.824 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.825 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:38.825 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.890610+0000", 2020-01-16T00:12:38.825 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:38.825 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.825 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.825 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:38.825 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.890610+0000", 2020-01-16T00:12:38.825 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:38.826 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.826 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.826 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:38.826 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.890608+0000", 2020-01-16T00:12:38.826 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000233947 2020-01-16T00:12:38.826 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.826 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.827 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:38.827 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.890842+0000", 2020-01-16T00:12:38.827 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4816e-05 2020-01-16T00:12:38.827 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.827 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.827 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:38.827 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.890857+0000", 2020-01-16T00:12:38.827 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9320000000000008e-06 2020-01-16T00:12:38.828 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.828 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.828 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:38.828 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.890867+0000", 2020-01-16T00:12:38.828 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3040000000000002e-05 2020-01-16T00:12:38.828 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.828 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.828 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.829 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.890900+0000", 2020-01-16T00:12:38.829 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8905000000000004e-05 2020-01-16T00:12:38.829 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.829 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.829 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:38.829 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.890979+0000", 2020-01-16T00:12:38.829 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012959199999999999 2020-01-16T00:12:38.830 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.830 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.830 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:12:38.830 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.891108+0000", 2020-01-16T00:12:38.830 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0019003519999999999 2020-01-16T00:12:38.830 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.831 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.831 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:38.831 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.893009+0000", 2020-01-16T00:12:38.831 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0016470460000000001 2020-01-16T00:12:38.831 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.831 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.832 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:38.832 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.894656+0000", 2020-01-16T00:12:38.832 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1043000000000005e-05 2020-01-16T00:12:38.832 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.832 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.832 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:38.832 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.894717+0000", 2020-01-16T00:12:38.833 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0583999999999997e-05 2020-01-16T00:12:38.833 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.833 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.833 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:38.834 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.894748+0000", 2020-01-16T00:12:38.834 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.4088e-05 2020-01-16T00:12:38.834 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.834 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.834 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:38.834 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.894772+0000", 2020-01-16T00:12:38.835 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041613839999999997 2020-01-16T00:12:38.835 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.835 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:38.835 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.835 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.835 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.835 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:935 3.2 3:4f8fe446:::smithi12313934-48:head [write 3060489~200036] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:38.836 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.895775+0000", 2020-01-16T00:12:38.836 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.77654119099999996, 2020-01-16T00:12:38.836 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012912735, 2020-01-16T00:12:38.836 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:38.836 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:38.837 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:38.837 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:38.837 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:38.837 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 935 2020-01-16T00:12:38.837 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.838 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:38.838 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.838 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:38.838 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895775+0000", 2020-01-16T00:12:38.839 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:38.839 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.839 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.839 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:38.839 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895775+0000", 2020-01-16T00:12:38.839 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:38.840 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.840 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.840 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:38.840 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895773+0000", 2020-01-16T00:12:38.840 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3350000000000007e-05 2020-01-16T00:12:38.840 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.841 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.841 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:38.841 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895856+0000", 2020-01-16T00:12:38.841 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.418e-05 2020-01-16T00:12:38.841 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.841 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.841 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:38.842 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895871+0000", 2020-01-16T00:12:38.842 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2750000000000001e-06 2020-01-16T00:12:38.842 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.842 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.842 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:38.842 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895877+0000", 2020-01-16T00:12:38.842 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7113000000000002e-05 2020-01-16T00:12:38.843 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.843 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.843 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.843 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895914+0000", 2020-01-16T00:12:38.844 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00064030700000000001 2020-01-16T00:12:38.844 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.844 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.844 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:38.844 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.896554+0000", 2020-01-16T00:12:38.844 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0098022809999999995 2020-01-16T00:12:38.844 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.845 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.845 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.845 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.906357+0000", 2020-01-16T00:12:38.845 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112898 2020-01-16T00:12:38.845 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.845 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.845 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:38.846 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.906469+0000", 2020-01-16T00:12:38.846 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022189399999999999 2020-01-16T00:12:38.846 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.846 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.846 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:12:38.846 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.906691+0000", 2020-01-16T00:12:38.846 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018153869999999999 2020-01-16T00:12:38.846 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.847 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.847 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:38.847 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908507+0000", 2020-01-16T00:12:38.847 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6517999999999999e-05 2020-01-16T00:12:38.847 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.847 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.847 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:38.848 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908533+0000", 2020-01-16T00:12:38.848 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1899e-05 2020-01-16T00:12:38.848 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.848 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.848 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:38.848 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908575+0000", 2020-01-16T00:12:38.848 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2222999999999998e-05 2020-01-16T00:12:38.848 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.849 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.849 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:38.849 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908627+0000", 2020-01-16T00:12:38.849 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0019999999999998e-05 2020-01-16T00:12:38.849 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.849 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.849 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:38.850 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908687+0000", 2020-01-16T00:12:38.850 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012912735 2020-01-16T00:12:38.850 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.850 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:38.850 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.851 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.851 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.851 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:936 3.2 3:4f8fe446:::smithi12313934-48:head [setxattr _header (56) in=7b,truncate 3260525] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:38.852 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.895886+0000", 2020-01-16T00:12:38.852 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.77642948700000003, 2020-01-16T00:12:38.852 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014382986, 2020-01-16T00:12:38.852 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:38.852 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:38.852 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:38.852 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:38.852 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:38.853 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 936 2020-01-16T00:12:38.853 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.853 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:38.853 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.853 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:38.853 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895886+0000", 2020-01-16T00:12:38.853 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:38.854 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.854 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.854 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:38.854 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895886+0000", 2020-01-16T00:12:38.854 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:12:38.854 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.854 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.854 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:38.855 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895885+0000", 2020-01-16T00:12:38.855 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0980000000000001e-06 2020-01-16T00:12:38.855 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.855 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.855 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:38.855 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895891+0000", 2020-01-16T00:12:38.855 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.959e-06 2020-01-16T00:12:38.855 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.856 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.856 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:38.856 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895897+0000", 2020-01-16T00:12:38.856 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9920000000000001e-06 2020-01-16T00:12:38.856 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.856 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.856 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:38.856 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895901+0000", 2020-01-16T00:12:38.857 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00068615199999999999 2020-01-16T00:12:38.857 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.857 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.857 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.857 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.896587+0000", 2020-01-16T00:12:38.857 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2934e-05 2020-01-16T00:12:38.857 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.858 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.858 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:38.858 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.896620+0000", 2020-01-16T00:12:38.858 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010751811 2020-01-16T00:12:38.858 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.858 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.859 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.859 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.907371+0000", 2020-01-16T00:12:38.859 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011422899999999999 2020-01-16T00:12:38.859 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.859 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.859 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:38.859 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.907486+0000", 2020-01-16T00:12:38.859 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000185172 2020-01-16T00:12:38.860 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.860 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.860 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:12:38.860 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.907671+0000", 2020-01-16T00:12:38.860 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001265164 2020-01-16T00:12:38.860 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.860 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.860 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:38.861 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908936+0000", 2020-01-16T00:12:38.861 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000297247 2020-01-16T00:12:38.861 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.861 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.861 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:38.861 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.909233+0000", 2020-01-16T00:12:38.861 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000907528 2020-01-16T00:12:38.862 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.862 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.862 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:38.862 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910141+0000", 2020-01-16T00:12:38.862 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3384000000000002e-05 2020-01-16T00:12:38.862 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.862 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.863 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:38.863 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910204+0000", 2020-01-16T00:12:38.863 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5229999999999997e-05 2020-01-16T00:12:38.863 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.863 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.863 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:38.863 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910269+0000", 2020-01-16T00:12:38.863 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014382986 2020-01-16T00:12:38.864 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.864 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:38.864 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.864 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.864 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.864 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:937 3.2 3:4f8fe446:::smithi12313934-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:12:38.864 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.895918+0000", 2020-01-16T00:12:38.865 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.77639785900000002, 2020-01-16T00:12:38.865 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014610741999999999, 2020-01-16T00:12:38.865 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:38.865 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:38.865 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:38.865 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:38.865 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:38.865 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 937 2020-01-16T00:12:38.866 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.866 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:38.866 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.866 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:38.866 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895918+0000", 2020-01-16T00:12:38.866 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:38.866 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.866 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.867 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:38.867 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895918+0000", 2020-01-16T00:12:38.867 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:12:38.867 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.867 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.867 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:38.867 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895917+0000", 2020-01-16T00:12:38.867 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8840000000000004e-06 2020-01-16T00:12:38.868 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.868 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.868 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:38.868 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895921+0000", 2020-01-16T00:12:38.868 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1579999999999999e-06 2020-01-16T00:12:38.868 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.868 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.868 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:38.869 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895928+0000", 2020-01-16T00:12:38.869 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4139999999999996e-06 2020-01-16T00:12:38.869 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.869 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.869 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:38.869 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895933+0000", 2020-01-16T00:12:38.869 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000702218 2020-01-16T00:12:38.870 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.870 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.870 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.870 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.896635+0000", 2020-01-16T00:12:38.870 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6421e-05 2020-01-16T00:12:38.870 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.870 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.870 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:38.871 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.896661+0000", 2020-01-16T00:12:38.871 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011578879 2020-01-16T00:12:38.871 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.871 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.871 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.871 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908240+0000", 2020-01-16T00:12:38.871 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5567000000000003e-05 2020-01-16T00:12:38.871 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.872 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.872 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:38.872 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908316+0000", 2020-01-16T00:12:38.872 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0019937560000000002 2020-01-16T00:12:38.872 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.872 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.872 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.872 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910310+0000", 2020-01-16T00:12:38.873 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000109566 2020-01-16T00:12:38.873 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.873 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.873 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:38.873 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910419+0000", 2020-01-16T00:12:38.873 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000109629 2020-01-16T00:12:38.873 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.873 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.874 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:38.874 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910529+0000", 2020-01-16T00:12:38.874 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014610741999999999 2020-01-16T00:12:38.874 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.874 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:38.874 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.874 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.874 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.875 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.5.0:158 3.c e24/20 3:3877922e:::smithi12313934-31:head v 24'72, mlcod=24'70)", 2020-01-16T00:12:38.875 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.931147+0000", 2020-01-16T00:12:38.875 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.74116866100000001, 2020-01-16T00:12:38.875 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0060017630000000002, 2020-01-16T00:12:38.875 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:38.875 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:38.875 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:38.876 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.876 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:38.876 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931147+0000", 2020-01-16T00:12:38.876 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:38.876 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.876 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.876 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:38.876 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931147+0000", 2020-01-16T00:12:38.877 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:38.877 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.877 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.877 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:38.877 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931145+0000", 2020-01-16T00:12:38.877 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0015459009999999999 2020-01-16T00:12:38.878 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.878 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.878 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:38.878 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932691+0000", 2020-01-16T00:12:38.878 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4831000000000001e-05 2020-01-16T00:12:38.878 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.878 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.878 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:38.879 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932705+0000", 2020-01-16T00:12:38.879 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3569999999999997e-06 2020-01-16T00:12:38.879 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.879 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.879 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:38.879 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932711+0000", 2020-01-16T00:12:38.879 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6208000000000003e-05 2020-01-16T00:12:38.879 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.880 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.880 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.880 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932747+0000", 2020-01-16T00:12:38.880 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7938999999999999e-05 2020-01-16T00:12:38.880 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.880 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.880 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:38.880 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932765+0000", 2020-01-16T00:12:38.881 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0043314 2020-01-16T00:12:38.881 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.881 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.881 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:38.881 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.937096+0000", 2020-01-16T00:12:38.881 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2614999999999997e-05 2020-01-16T00:12:38.881 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.881 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.882 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:38.882 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.937149+0000", 2020-01-16T00:12:38.882 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0060017630000000002 2020-01-16T00:12:38.882 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.882 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:38.882 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.882 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.882 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.883 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:970 3.8 3:1e8f7c6f:::smithi12313934-6:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:38.883 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.159615+0000", 2020-01-16T00:12:38.883 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.51270132599999996, 2020-01-16T00:12:38.883 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0057821019999999999, 2020-01-16T00:12:38.883 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:38.883 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:38.883 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:38.884 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:38.884 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:38.884 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 970 2020-01-16T00:12:38.884 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.884 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:38.884 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.884 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:38.884 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159615+0000", 2020-01-16T00:12:38.885 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:38.885 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.885 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.885 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:38.885 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159615+0000", 2020-01-16T00:12:38.885 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999914 2020-01-16T00:12:38.885 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.885 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.886 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:38.886 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159606+0000", 2020-01-16T00:12:38.886 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6858999999999999e-05 2020-01-16T00:12:38.886 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.886 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.886 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:38.886 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159623+0000", 2020-01-16T00:12:38.887 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6963999999999999e-05 2020-01-16T00:12:38.887 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.887 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.887 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:38.887 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159640+0000", 2020-01-16T00:12:38.887 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0510000000000002e-06 2020-01-16T00:12:38.887 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.887 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.888 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:38.888 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159646+0000", 2020-01-16T00:12:38.888 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000101834 2020-01-16T00:12:38.888 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.888 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.888 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.888 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159748+0000", 2020-01-16T00:12:38.888 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00037551000000000001 2020-01-16T00:12:38.889 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.889 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.889 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:38.889 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.160123+0000", 2020-01-16T00:12:38.889 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0046367609999999997 2020-01-16T00:12:38.889 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.889 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.890 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.890 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.164760+0000", 2020-01-16T00:12:38.890 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2069000000000004e-05 2020-01-16T00:12:38.890 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.890 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.890 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:38.890 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.164832+0000", 2020-01-16T00:12:38.891 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00033586500000000002 2020-01-16T00:12:38.891 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.891 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.891 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.891 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.165168+0000", 2020-01-16T00:12:38.891 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0454000000000002e-05 2020-01-16T00:12:38.891 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.891 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.892 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:38.892 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.165218+0000", 2020-01-16T00:12:38.892 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000178178 2020-01-16T00:12:38.892 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.892 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.892 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:38.892 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.165397+0000", 2020-01-16T00:12:38.892 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0057821019999999999 2020-01-16T00:12:38.893 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.893 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:38.893 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.893 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.893 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.894 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:971 3.8 3:1e8f7c6f:::smithi12313934-6:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:38.894 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.159657+0000", 2020-01-16T00:12:38.894 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.51265868999999997, 2020-01-16T00:12:38.894 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0059336909999999996, 2020-01-16T00:12:38.894 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:38.894 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:38.894 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:38.894 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:38.895 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:38.895 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 971 2020-01-16T00:12:38.895 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.895 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:38.895 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.895 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:38.895 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159657+0000", 2020-01-16T00:12:38.895 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:38.896 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.896 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.896 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:38.896 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159657+0000", 2020-01-16T00:12:38.896 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:38.896 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.896 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.896 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:38.897 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159657+0000", 2020-01-16T00:12:38.897 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9390000000000002e-06 2020-01-16T00:12:38.897 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.897 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.897 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:38.897 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159660+0000", 2020-01-16T00:12:38.897 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7389999999999996e-06 2020-01-16T00:12:38.898 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.898 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.898 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:38.898 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159665+0000", 2020-01-16T00:12:38.898 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2473999999999999e-05 2020-01-16T00:12:38.898 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.898 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.898 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:38.899 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159678+0000", 2020-01-16T00:12:38.899 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00054627300000000001 2020-01-16T00:12:38.899 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.899 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.899 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.899 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.160224+0000", 2020-01-16T00:12:38.899 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8795000000000002e-05 2020-01-16T00:12:38.899 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.900 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.900 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:38.900 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.160283+0000", 2020-01-16T00:12:38.900 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0045895440000000001 2020-01-16T00:12:38.900 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.900 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.900 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.900 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.164872+0000", 2020-01-16T00:12:38.901 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8940999999999997e-05 2020-01-16T00:12:38.901 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.901 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.901 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:38.901 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.164931+0000", 2020-01-16T00:12:38.901 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00048151800000000001 2020-01-16T00:12:38.901 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.901 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.902 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.902 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.165413+0000", 2020-01-16T00:12:38.902 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.117e-05 2020-01-16T00:12:38.902 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.902 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.902 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:38.902 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.165464+0000", 2020-01-16T00:12:38.902 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012686699999999999 2020-01-16T00:12:38.903 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.903 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.903 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:38.903 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.165591+0000", 2020-01-16T00:12:38.903 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0059336909999999996 2020-01-16T00:12:38.903 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.903 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:38.903 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.904 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.904 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.904 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:972 3.8 3:1e8f7c6f:::smithi12313934-6:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:38.905 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.159726+0000", 2020-01-16T00:12:38.905 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.51258946900000002, 2020-01-16T00:12:38.905 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0061844259999999998, 2020-01-16T00:12:38.905 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:38.905 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:38.905 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:38.906 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:38.906 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:38.906 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 972 2020-01-16T00:12:38.906 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.906 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:38.906 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.906 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:38.907 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159726+0000", 2020-01-16T00:12:38.907 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:38.907 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.907 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.907 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:38.907 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159726+0000", 2020-01-16T00:12:38.907 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:38.908 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.908 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.908 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:38.908 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159726+0000", 2020-01-16T00:12:38.908 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0910000000000003e-06 2020-01-16T00:12:38.909 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.909 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.909 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:38.909 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159730+0000", 2020-01-16T00:12:38.909 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.6910000000000003e-06 2020-01-16T00:12:38.909 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.909 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.910 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:38.910 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159738+0000", 2020-01-16T00:12:38.910 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1479999999999999e-06 2020-01-16T00:12:38.910 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.910 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.910 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:38.911 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159743+0000", 2020-01-16T00:12:38.911 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00057154700000000005 2020-01-16T00:12:38.911 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.911 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.911 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.911 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.160314+0000", 2020-01-16T00:12:38.911 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8436999999999997e-05 2020-01-16T00:12:38.911 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.912 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.912 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:38.912 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.160363+0000", 2020-01-16T00:12:38.912 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0045992810000000002 2020-01-16T00:12:38.912 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.912 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.912 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.912 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.164962+0000", 2020-01-16T00:12:38.913 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8581000000000001e-05 2020-01-16T00:12:38.913 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.913 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.913 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:38.913 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.165010+0000", 2020-01-16T00:12:38.913 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00060742200000000004 2020-01-16T00:12:38.913 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.913 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.914 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.914 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.165618+0000", 2020-01-16T00:12:38.914 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6708999999999999e-05 2020-01-16T00:12:38.914 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.914 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.914 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:38.914 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.165715+0000", 2020-01-16T00:12:38.914 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019620999999999999 2020-01-16T00:12:38.915 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.915 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.915 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:38.915 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.165911+0000", 2020-01-16T00:12:38.915 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0061844259999999998 2020-01-16T00:12:38.915 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.915 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:38.916 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.916 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.916 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.916 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.4.13:24 2.1 e24/19 2:8b431f80:::smithi12313934-40 low_tier:head v 24'26, mlcod=24'24)", 2020-01-16T00:12:38.916 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.486538+0000", 2020-01-16T00:12:38.916 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.18577787000000001, 2020-01-16T00:12:38.916 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0043336379999999999, 2020-01-16T00:12:38.916 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:38.917 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:38.917 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:38.917 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.917 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:38.917 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.486538+0000", 2020-01-16T00:12:38.917 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:38.917 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.917 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.918 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:38.918 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.486538+0000", 2020-01-16T00:12:38.918 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999962 2020-01-16T00:12:38.918 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.918 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.918 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:38.918 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.486534+0000", 2020-01-16T00:12:38.918 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023787 2020-01-16T00:12:38.919 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.919 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.919 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:38.919 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.486772+0000", 2020-01-16T00:12:38.919 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.1219000000000001e-05 2020-01-16T00:12:38.919 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.919 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.919 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:38.920 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.486793+0000", 2020-01-16T00:12:38.920 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5920000000000001e-06 2020-01-16T00:12:38.920 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.920 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.920 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:38.920 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.486802+0000", 2020-01-16T00:12:38.920 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6994e-05 2020-01-16T00:12:38.920 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.921 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.921 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:38.921 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.486899+0000", 2020-01-16T00:12:38.921 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5993e-05 2020-01-16T00:12:38.921 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.921 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.921 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:38.921 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.486925+0000", 2020-01-16T00:12:38.922 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0038910469999999999 2020-01-16T00:12:38.922 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.922 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.922 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:38.922 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.490816+0000", 2020-01-16T00:12:38.922 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5559000000000001e-05 2020-01-16T00:12:38.922 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:38.922 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:38.923 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:38.923 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.490872+0000", 2020-01-16T00:12:38.923 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0043336379999999999 2020-01-16T00:12:38.923 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.923 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:38.923 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.923 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:38.924 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:38.924 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:38.924 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2020-01-16T00:12:38.926 INFO:tasks.rados.rados.0.smithi123.stdout:657: finishing write tid 1 to smithi12313934-31 2020-01-16T00:12:38.926 INFO:tasks.rados.rados.0.smithi123.stdout:657: finishing write tid 2 to smithi12313934-31 2020-01-16T00:12:38.927 INFO:tasks.rados.rados.0.smithi123.stdout:657: finishing write tid 3 to smithi12313934-31 2020-01-16T00:12:38.927 INFO:tasks.rados.rados.0.smithi123.stdout:657: finishing write tid 5 to smithi12313934-31 2020-01-16T00:12:38.927 INFO:tasks.rados.rados.0.smithi123.stdout:657: finishing write tid 6 to smithi12313934-31 2020-01-16T00:12:38.927 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 31 v 78 (ObjNum 77 snap 0 seq_num 77) dirty exists 2020-01-16T00:12:38.927 INFO:tasks.rados.rados.0.smithi123.stdout:657: left oid 31 (ObjNum 77 snap 0 seq_num 77) 2020-01-16T00:12:38.927 INFO:tasks.rados.rados.0.smithi123.stdout:658: finishing write tid 1 to smithi12313934-18 2020-01-16T00:12:38.928 INFO:tasks.rados.rados.0.smithi123.stdout:658: finishing write tid 2 to smithi12313934-18 2020-01-16T00:12:38.928 INFO:tasks.rados.rados.0.smithi123.stdout:658: finishing write tid 3 to smithi12313934-18 2020-01-16T00:12:38.928 INFO:tasks.rados.rados.0.smithi123.stdout:658: finishing write tid 5 to smithi12313934-18 2020-01-16T00:12:38.928 INFO:tasks.rados.rados.0.smithi123.stdout:658: finishing write tid 6 to smithi12313934-18 2020-01-16T00:12:38.928 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 18 v 103 (ObjNum 78 snap 0 seq_num 78) dirty exists 2020-01-16T00:12:38.928 INFO:tasks.rados.rados.0.smithi123.stdout:658: left oid 18 (ObjNum 78 snap 0 seq_num 78) 2020-01-16T00:12:38.928 INFO:tasks.rados.rados.0.smithi123.stdout:659: expect (ObjNum 42 snap 0 seq_num 42) 2020-01-16T00:12:38.928 INFO:tasks.rados.rados.0.smithi123.stdout:660: finishing write tid 1 to smithi12313934-41 2020-01-16T00:12:38.929 INFO:tasks.rados.rados.0.smithi123.stdout:660: finishing write tid 2 to smithi12313934-41 2020-01-16T00:12:38.929 INFO:tasks.rados.rados.0.smithi123.stdout:660: finishing write tid 3 to smithi12313934-41 2020-01-16T00:12:38.929 INFO:tasks.rados.rados.0.smithi123.stdout:660: finishing write tid 5 to smithi12313934-41 2020-01-16T00:12:38.929 INFO:tasks.rados.rados.0.smithi123.stdout:660: finishing write tid 6 to smithi12313934-41 2020-01-16T00:12:38.929 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 41 v 74 (ObjNum 79 snap 0 seq_num 79) dirty exists 2020-01-16T00:12:38.929 INFO:tasks.rados.rados.0.smithi123.stdout:660: left oid 41 (ObjNum 79 snap 0 seq_num 79) 2020-01-16T00:12:38.930 INFO:tasks.rados.rados.0.smithi123.stdout:661: expect (ObjNum 5 snap 0 seq_num 5) 2020-01-16T00:12:38.930 INFO:tasks.rados.rados.0.smithi123.stdout:662: finishing write tid 1 to smithi12313934-9 2020-01-16T00:12:38.930 INFO:tasks.rados.rados.0.smithi123.stdout:662: finishing write tid 2 to smithi12313934-9 2020-01-16T00:12:38.930 INFO:tasks.rados.rados.0.smithi123.stdout:662: finishing write tid 3 to smithi12313934-9 2020-01-16T00:12:38.930 INFO:tasks.rados.rados.0.smithi123.stdout:662: finishing write tid 5 to smithi12313934-9 2020-01-16T00:12:38.930 INFO:tasks.rados.rados.0.smithi123.stdout:662: finishing write tid 6 to smithi12313934-9 2020-01-16T00:12:38.930 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 9 v 56 (ObjNum 80 snap 0 seq_num 80) dirty exists 2020-01-16T00:12:38.930 INFO:tasks.rados.rados.0.smithi123.stdout:662: left oid 9 (ObjNum 80 snap 0 seq_num 80) 2020-01-16T00:12:38.931 INFO:tasks.rados.rados.0.smithi123.stdout:663: expect (ObjNum 72 snap 0 seq_num 72) 2020-01-16T00:12:38.931 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:38.925+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:38.931 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-01-16T00:12:39.026 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:12:39.027+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:12:39.027 INFO:tasks.rados.rados.0.smithi123.stdout:664: expect (ObjNum 43 snap 0 seq_num 43) 2020-01-16T00:12:39.089 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:39.089 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:12:39.090 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:12:39.090 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:39.094 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-01-16T00:12:39.098 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-01-16T00:12:39.110 INFO:tasks.rados.rados.0.smithi123.stdout:665: finishing write tid 1 to smithi12313934-40 2020-01-16T00:12:39.110 INFO:tasks.rados.rados.0.smithi123.stdout:665: finishing write tid 2 to smithi12313934-40 2020-01-16T00:12:39.110 INFO:tasks.rados.rados.0.smithi123.stdout:667: expect (ObjNum 54 snap 0 seq_num 54) 2020-01-16T00:12:39.127 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:12:39.129+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:12:39.211 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:12:39.221 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:39.222 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:12:39.222 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:12:39.222 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:12:39.222 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:39.227 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-01-16T00:12:39.230 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:12:39.231 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:12:39.231+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:12:39.248 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:12:39.256 INFO:tasks.rados.rados.0.smithi123.stdout:666: expect (ObjNum 21 snap 0 seq_num 21) 2020-01-16T00:12:39.267 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:12:39.289 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:12:39.311 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5: {} 2020-01-16T00:12:39.328 INFO:tasks.rados.rados.0.smithi123.stdout:668: finishing write tid 1 to smithi12313934-35 2020-01-16T00:12:39.328 INFO:tasks.rados.rados.0.smithi123.stdout:665: finishing write tid 3 to smithi12313934-40 2020-01-16T00:12:39.328 INFO:tasks.rados.rados.0.smithi123.stdout:665: finishing write tid 5 to smithi12313934-40 2020-01-16T00:12:39.328 INFO:tasks.rados.rados.0.smithi123.stdout:653: done (15 left) 2020-01-16T00:12:39.329 INFO:tasks.rados.rados.0.smithi123.stdout:654: done (14 left) 2020-01-16T00:12:39.329 INFO:tasks.rados.rados.0.smithi123.stdout:655: done (13 left) 2020-01-16T00:12:39.329 INFO:tasks.rados.rados.0.smithi123.stdout:656: done (12 left) 2020-01-16T00:12:39.329 INFO:tasks.rados.rados.0.smithi123.stdout:657: done (11 left) 2020-01-16T00:12:39.329 INFO:tasks.rados.rados.0.smithi123.stdout:658: done (10 left) 2020-01-16T00:12:39.329 INFO:tasks.rados.rados.0.smithi123.stdout:659: done (9 left) 2020-01-16T00:12:39.329 INFO:tasks.rados.rados.0.smithi123.stdout:660: done (8 left) 2020-01-16T00:12:39.329 INFO:tasks.rados.rados.0.smithi123.stdout:661: done (7 left) 2020-01-16T00:12:39.330 INFO:tasks.rados.rados.0.smithi123.stdout:662: done (6 left) 2020-01-16T00:12:39.330 INFO:tasks.rados.rados.0.smithi123.stdout:663: done (5 left) 2020-01-16T00:12:39.330 INFO:tasks.rados.rados.0.smithi123.stdout:664: done (4 left) 2020-01-16T00:12:39.330 INFO:tasks.rados.rados.0.smithi123.stdout:666: done (3 left) 2020-01-16T00:12:39.330 INFO:tasks.rados.rados.0.smithi123.stdout:667: done (2 left) 2020-01-16T00:12:39.330 INFO:tasks.rados.rados.0.smithi123.stdout:669: read oid 42 snap -1 2020-01-16T00:12:39.330 INFO:tasks.rados.rados.0.smithi123.stdout:669: expect (ObjNum 41 snap 0 seq_num 41) 2020-01-16T00:12:39.330 INFO:tasks.rados.rados.0.smithi123.stdout:670: read oid 32 snap -1 2020-01-16T00:12:39.331 INFO:tasks.rados.rados.0.smithi123.stdout:670: expect (ObjNum 31 snap 0 seq_num 31) 2020-01-16T00:12:39.331 INFO:tasks.rados.rados.0.smithi123.stdout:671: write (excl) oid 1 current snap is 0 2020-01-16T00:12:39.331 INFO:tasks.rados.rados.0.smithi123.stdout:671: seq_num 83 ranges {594981=609197,1974123=745028,3279873=115108} 2020-01-16T00:12:39.331 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:39.332 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:12:39.332 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:39.333+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:39.338 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-01-16T00:12:39.343 INFO:tasks.rados.rados.0.smithi123.stdout:671: writing smithi12313934-1 from 594981 to 1204178 tid 1 2020-01-16T00:12:39.352 INFO:tasks.rados.rados.0.smithi123.stdout:671: writing smithi12313934-1 from 1974123 to 2719151 tid 2 2020-01-16T00:12:39.357 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:12:39.357 INFO:tasks.rados.rados.0.smithi123.stdout:671: writing smithi12313934-1 from 3279873 to 3394981 tid 3 2020-01-16T00:12:39.357 INFO:tasks.rados.rados.0.smithi123.stdout:672: read oid 46 snap -1 2020-01-16T00:12:39.357 INFO:tasks.rados.rados.0.smithi123.stdout:672: expect (ObjNum 68 snap 0 seq_num 68) 2020-01-16T00:12:39.358 INFO:tasks.rados.rados.0.smithi123.stdout:673: write oid 30 current snap is 0 2020-01-16T00:12:39.358 INFO:tasks.rados.rados.0.smithi123.stdout:673: seq_num 84 ranges {611788=400799,1748958=488686,2907296=504492} 2020-01-16T00:12:39.358 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-01-16T00:12:39.359 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-01-16T00:12:39.359 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-01-16T00:12:39.359 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-01-16T00:12:39.359 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-01-16T00:12:39.359 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-01-16T00:12:39.359 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-01-16T00:12:39.359 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-01-16T00:12:39.362 INFO:tasks.rados.rados.0.smithi123.stdout:673: writing smithi12313934-30 from 611788 to 1012587 tid 1 2020-01-16T00:12:39.370 INFO:tasks.rados.rados.0.smithi123.stdout:673: writing smithi12313934-30 from 1748958 to 2237644 tid 2 2020-01-16T00:12:39.377 INFO:tasks.rados.rados.0.smithi123.stdout:673: writing smithi12313934-30 from 2907296 to 3411788 tid 3 2020-01-16T00:12:39.377 INFO:tasks.rados.rados.0.smithi123.stdout:674: write (excl) oid 25 current snap is 0 2020-01-16T00:12:39.378 INFO:tasks.rados.rados.0.smithi123.stdout:674: seq_num 85 ranges {628595=476048,1807440=632344,2934719=452474} 2020-01-16T00:12:39.383 INFO:tasks.rados.rados.0.smithi123.stdout:674: writing smithi12313934-25 from 628595 to 1104643 tid 1 2020-01-16T00:12:39.391 INFO:tasks.rados.rados.0.smithi123.stdout:674: writing smithi12313934-25 from 1807440 to 2439784 tid 2 2020-01-16T00:12:39.396 INFO:tasks.rados.rados.0.smithi123.stdout:674: writing smithi12313934-25 from 2934719 to 3387193 tid 3 2020-01-16T00:12:39.396 INFO:tasks.rados.rados.0.smithi123.stdout:675: write (excl) oid 3 current snap is 0 2020-01-16T00:12:39.397 INFO:tasks.rados.rados.0.smithi123.stdout:675: seq_num 86 ranges {645402=551297,1749569=492355,2845789=599613} 2020-01-16T00:12:39.402 INFO:tasks.rados.rados.0.smithi123.stdout:675: writing smithi12313934-3 from 645402 to 1196699 tid 1 2020-01-16T00:12:39.408 INFO:tasks.rados.rados.0.smithi123.stdout:675: writing smithi12313934-3 from 1749569 to 2241924 tid 2 2020-01-16T00:12:39.416 INFO:tasks.rados.rados.0.smithi123.stdout:675: writing smithi12313934-3 from 2845789 to 3445402 tid 3 2020-01-16T00:12:39.416 INFO:tasks.rados.rados.0.smithi123.stdout:676: write oid 27 current snap is 0 2020-01-16T00:12:39.417 INFO:tasks.rados.rados.0.smithi123.stdout:676: seq_num 87 ranges {662209=626546,1808051=636013,2873212=588997} 2020-01-16T00:12:39.430 INFO:tasks.rados.rados.0.smithi123.stdout:676: writing smithi12313934-27 from 662209 to 1288755 tid 1 2020-01-16T00:12:39.430 INFO:tasks.rados.rados.0.smithi123.stdout:676: writing smithi12313934-27 from 1808051 to 2444064 tid 2 2020-01-16T00:12:39.435 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:12:39.433+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:12:39.436 INFO:tasks.rados.rados.0.smithi123.stdout:676: writing smithi12313934-27 from 2873212 to 3462209 tid 3 2020-01-16T00:12:39.436 INFO:tasks.rados.rados.0.smithi123.stdout:677: read oid 5 snap -1 2020-01-16T00:12:39.436 INFO:tasks.rados.rados.0.smithi123.stdout:677: expect (ObjNum 70 snap 0 seq_num 70) 2020-01-16T00:12:39.436 INFO:tasks.rados.rados.0.smithi123.stdout:678: write (excl) oid 37 current snap is 0 2020-01-16T00:12:39.436 INFO:tasks.rados.rados.0.smithi123.stdout:678: seq_num 88 ranges {679016=701795,1866533=496024,2900635=402643} 2020-01-16T00:12:39.444 INFO:tasks.rados.rados.0.smithi123.stdout:678: writing smithi12313934-37 from 679016 to 1380811 tid 1 2020-01-16T00:12:39.450 INFO:tasks.rados.rados.0.smithi123.stdout:678: writing smithi12313934-37 from 1866533 to 2362557 tid 2 2020-01-16T00:12:39.454 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:39.456 INFO:tasks.rados.rados.0.smithi123.stdout:678: writing smithi12313934-37 from 2900635 to 3303278 tid 3 2020-01-16T00:12:39.456 INFO:tasks.rados.rados.0.smithi123.stdout:679: write (excl) oid 7 current snap is 0 2020-01-16T00:12:39.456 INFO:tasks.rados.rados.0.smithi123.stdout:679: seq_num 89 ranges {695823=777044,1925015=639682,3328058=167765} 2020-01-16T00:12:39.464 INFO:tasks.rados.rados.0.smithi123.stdout:679: writing smithi12313934-7 from 695823 to 1472867 tid 1 2020-01-16T00:12:39.465 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-01-16T00:12:39.470 INFO:tasks.rados.rados.0.smithi123.stdout:679: writing smithi12313934-7 from 1925015 to 2564697 tid 2 2020-01-16T00:12:39.476 INFO:tasks.rados.rados.0.smithi123.stdout:679: writing smithi12313934-7 from 3328058 to 3495823 tid 3 2020-01-16T00:12:39.477 INFO:tasks.rados.rados.0.smithi123.stdout:680: write (excl) oid 39 current snap is 0 2020-01-16T00:12:39.477 INFO:tasks.rados.rados.0.smithi123.stdout:680: seq_num 90 ranges {712630=452293,1583497=499693,2555481=425187} 2020-01-16T00:12:39.482 INFO:tasks.rados.rados.0.smithi123.stdout:680: writing smithi12313934-39 from 712630 to 1164923 tid 1 2020-01-16T00:12:39.487 INFO:tasks.rados.rados.0.smithi123.stdout:680: writing smithi12313934-39 from 1583497 to 2083190 tid 2 2020-01-16T00:12:39.493 INFO:tasks.rados.rados.0.smithi123.stdout:680: writing smithi12313934-39 from 2555481 to 2980668 tid 3 2020-01-16T00:12:39.493 INFO:tasks.rados.rados.0.smithi123.stdout:681: read oid 48 snap -1 2020-01-16T00:12:39.493 INFO:tasks.rados.rados.0.smithi123.stdout:681: expect (ObjNum 75 snap 0 seq_num 75) 2020-01-16T00:12:39.493 INFO:tasks.rados.rados.0.smithi123.stdout:682: write (excl) oid 9 current snap is 0 2020-01-16T00:12:39.493 INFO:tasks.rados.rados.0.smithi123.stdout:682: seq_num 91 ranges {729437=527542,1925626=643351,3266551=262886} 2020-01-16T00:12:39.499 INFO:tasks.rados.rados.0.smithi123.stdout:682: writing smithi12313934-9 from 729437 to 1256979 tid 1 2020-01-16T00:12:39.507 INFO:tasks.rados.rados.0.smithi123.stdout:682: writing smithi12313934-9 from 1925626 to 2568977 tid 2 2020-01-16T00:12:39.513 INFO:tasks.rados.rados.0.smithi123.stdout:682: writing smithi12313934-9 from 3266551 to 3529437 tid 3 2020-01-16T00:12:39.513 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:39.513 INFO:tasks.rados.rados.0.smithi123.stdout:665: finishing write tid 6 to smithi12313934-40 2020-01-16T00:12:39.514 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 40 v 79 (ObjNum 81 snap 0 seq_num 81) dirty exists 2020-01-16T00:12:39.514 INFO:tasks.rados.rados.0.smithi123.stdout:665: left oid 40 (ObjNum 81 snap 0 seq_num 81) 2020-01-16T00:12:39.514 INFO:tasks.rados.rados.0.smithi123.stdout:665: done (15 left) 2020-01-16T00:12:39.514 INFO:tasks.rados.rados.0.smithi123.stdout:683: read oid 33 snap -1 2020-01-16T00:12:39.514 INFO:tasks.rados.rados.0.smithi123.stdout:683: expect (ObjNum 54 snap 0 seq_num 54) 2020-01-16T00:12:39.514 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:39.514 INFO:tasks.rados.rados.0.smithi123.stdout:668: finishing write tid 2 to smithi12313934-35 2020-01-16T00:12:39.514 INFO:tasks.rados.rados.0.smithi123.stdout:668: finishing write tid 3 to smithi12313934-35 2020-01-16T00:12:39.515 INFO:tasks.rados.rados.0.smithi123.stdout:668: finishing write tid 5 to smithi12313934-35 2020-01-16T00:12:39.515 INFO:tasks.rados.rados.0.smithi123.stdout:668: finishing write tid 6 to smithi12313934-35 2020-01-16T00:12:39.515 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 35 v 68 (ObjNum 82 snap 0 seq_num 82) dirty exists 2020-01-16T00:12:39.515 INFO:tasks.rados.rados.0.smithi123.stdout:668: left oid 35 (ObjNum 82 snap 0 seq_num 82) 2020-01-16T00:12:39.515 INFO:tasks.rados.rados.0.smithi123.stdout:668: done (15 left) 2020-01-16T00:12:39.515 INFO:tasks.rados.rados.0.smithi123.stdout:684: write oid 45 current snap is 0 2020-01-16T00:12:39.515 INFO:tasks.rados.rados.0.smithi123.stdout:684: seq_num 92 ranges {746244=719144,2100461=503362,3126680=419564} 2020-01-16T00:12:39.521 INFO:tasks.rados.rados.0.smithi123.stdout:684: writing smithi12313934-45 from 746244 to 1465388 tid 1 2020-01-16T00:12:39.529 INFO:tasks.rados.rados.0.smithi123.stdout:684: writing smithi12313934-45 from 2100461 to 2603823 tid 2 2020-01-16T00:12:39.533 INFO:tasks.rados.rados.0.smithi123.stdout:684: writing smithi12313934-45 from 3126680 to 3546244 tid 3 2020-01-16T00:12:39.533 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:39.533 INFO:tasks.rados.rados.0.smithi123.stdout:671: finishing write tid 1 to smithi12313934-1 2020-01-16T00:12:39.533 INFO:tasks.rados.rados.0.smithi123.stdout:669: expect (ObjNum 41 snap 0 seq_num 41) 2020-01-16T00:12:39.551 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:39.553+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:39.611 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:39.617 INFO:tasks.rados.rados.0.smithi123.stdout:670: expect (ObjNum 31 snap 0 seq_num 31) 2020-01-16T00:12:39.619 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-01-16T00:12:39.637 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:12:39.639+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:12:39.691 INFO:tasks.rados.rados.0.smithi123.stdout:673: finishing write tid 1 to smithi12313934-30 2020-01-16T00:12:39.691 INFO:tasks.rados.rados.0.smithi123.stdout:672: expect (ObjNum 68 snap 0 seq_num 68) 2020-01-16T00:12:39.738 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:12:39.739+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:12:39.773 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:39.794 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-01-16T00:12:39.829 INFO:tasks.rados.rados.0.smithi123.stdout:671: finishing write tid 2 to smithi12313934-1 2020-01-16T00:12:39.829 INFO:tasks.rados.rados.0.smithi123.stdout:671: finishing write tid 3 to smithi12313934-1 2020-01-16T00:12:39.829 INFO:tasks.rados.rados.0.smithi123.stdout:671: finishing write tid 5 to smithi12313934-1 2020-01-16T00:12:39.829 INFO:tasks.rados.rados.0.smithi123.stdout:671: finishing write tid 6 to smithi12313934-1 2020-01-16T00:12:39.830 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 1 v 108 (ObjNum 83 snap 0 seq_num 83) dirty exists 2020-01-16T00:12:39.830 INFO:tasks.rados.rados.0.smithi123.stdout:671: left oid 1 (ObjNum 83 snap 0 seq_num 83) 2020-01-16T00:12:39.830 INFO:tasks.rados.rados.0.smithi123.stdout:673: finishing write tid 2 to smithi12313934-30 2020-01-16T00:12:39.830 INFO:tasks.rados.rados.0.smithi123.stdout:673: finishing write tid 3 to smithi12313934-30 2020-01-16T00:12:39.830 INFO:tasks.rados.rados.0.smithi123.stdout:673: finishing write tid 5 to smithi12313934-30 2020-01-16T00:12:39.830 INFO:tasks.rados.rados.0.smithi123.stdout:673: finishing write tid 6 to smithi12313934-30 2020-01-16T00:12:39.830 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 30 v 53 (ObjNum 84 snap 0 seq_num 84) dirty exists 2020-01-16T00:12:39.830 INFO:tasks.rados.rados.0.smithi123.stdout:673: left oid 30 (ObjNum 84 snap 0 seq_num 84) 2020-01-16T00:12:39.831 INFO:tasks.rados.rados.0.smithi123.stdout:674: finishing write tid 1 to smithi12313934-25 2020-01-16T00:12:39.831 INFO:tasks.rados.rados.0.smithi123.stdout:674: finishing write tid 2 to smithi12313934-25 2020-01-16T00:12:39.831 INFO:tasks.rados.rados.0.smithi123.stdout:675: finishing write tid 1 to smithi12313934-3 2020-01-16T00:12:39.831 INFO:tasks.rados.rados.0.smithi123.stdout:675: finishing write tid 2 to smithi12313934-3 2020-01-16T00:12:39.831 INFO:tasks.rados.rados.0.smithi123.stdout:675: finishing write tid 3 to smithi12313934-3 2020-01-16T00:12:39.831 INFO:tasks.rados.rados.0.smithi123.stdout:669: done (15 left) 2020-01-16T00:12:39.831 INFO:tasks.rados.rados.0.smithi123.stdout:670: done (14 left) 2020-01-16T00:12:39.831 INFO:tasks.rados.rados.0.smithi123.stdout:671: done (13 left) 2020-01-16T00:12:39.832 INFO:tasks.rados.rados.0.smithi123.stdout:672: done (12 left) 2020-01-16T00:12:39.832 INFO:tasks.rados.rados.0.smithi123.stdout:673: done (11 left) 2020-01-16T00:12:39.832 INFO:tasks.rados.rados.0.smithi123.stdout:685: write (excl) oid 26 current snap is 0 2020-01-16T00:12:39.832 INFO:tasks.rados.rados.0.smithi123.stdout:685: seq_num 93 ranges {763051=794393,2158943=647020,3437750=125301} 2020-01-16T00:12:39.843 INFO:tasks.rados.rados.0.smithi123.stdout:685: writing smithi12313934-26 from 763051 to 1557444 tid 1 2020-01-16T00:12:39.847 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:12:39.849+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:12:39.851 INFO:tasks.rados.rados.0.smithi123.stdout:685: writing smithi12313934-26 from 2158943 to 2805963 tid 2 2020-01-16T00:12:39.856 INFO:tasks.rados.rados.0.smithi123.stdout:685: writing smithi12313934-26 from 3437750 to 3563051 tid 3 2020-01-16T00:12:39.856 INFO:tasks.rados.rados.0.smithi123.stdout:686: read oid 12 snap -1 2020-01-16T00:12:39.856 INFO:tasks.rados.rados.0.smithi123.stdout:686: expect (ObjNum 74 snap 0 seq_num 74) 2020-01-16T00:12:39.856 INFO:tasks.rados.rados.0.smithi123.stdout:687: read oid 38 snap -1 2020-01-16T00:12:39.856 INFO:tasks.rados.rados.0.smithi123.stdout:687: expect (ObjNum 52 snap 0 seq_num 52) 2020-01-16T00:12:39.857 INFO:tasks.rados.rados.0.smithi123.stdout:688: write (excl) oid 36 current snap is 0 2020-01-16T00:12:39.857 INFO:tasks.rados.rados.0.smithi123.stdout:688: seq_num 94 ranges {779858=469642,1817425=507031,2781526=586628} 2020-01-16T00:12:39.861 INFO:teuthology.orchestra.run.smithi123.stderr:nodeep-scrub is unset 2020-01-16T00:12:39.863 INFO:tasks.rados.rados.0.smithi123.stdout:688: writing smithi12313934-36 from 779858 to 1249500 tid 1 2020-01-16T00:12:39.870 INFO:tasks.rados.rados.0.smithi123.stdout:688: writing smithi12313934-36 from 1817425 to 2324456 tid 2 2020-01-16T00:12:39.882 INFO:tasks.rados.rados.0.smithi123.stdout:688: writing smithi12313934-36 from 2781526 to 3368154 tid 3 2020-01-16T00:12:39.882 INFO:tasks.rados.rados.0.smithi123.stdout:689: read oid 1 snap -1 2020-01-16T00:12:39.883 INFO:tasks.rados.rados.0.smithi123.stdout:689: expect (ObjNum 83 snap 0 seq_num 83) 2020-01-16T00:12:39.883 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:39.883 INFO:tasks.rados.rados.0.smithi123.stdout:675: finishing write tid 5 to smithi12313934-3 2020-01-16T00:12:39.883 INFO:tasks.rados.rados.0.smithi123.stdout:675: finishing write tid 6 to smithi12313934-3 2020-01-16T00:12:39.883 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 3 v 37 (ObjNum 86 snap 0 seq_num 86) dirty exists 2020-01-16T00:12:39.883 INFO:tasks.rados.rados.0.smithi123.stdout:675: left oid 3 (ObjNum 86 snap 0 seq_num 86) 2020-01-16T00:12:39.883 INFO:tasks.rados.rados.0.smithi123.stdout:674: finishing write tid 3 to smithi12313934-25 2020-01-16T00:12:39.884 INFO:tasks.rados.rados.0.smithi123.stdout:675: done (15 left) 2020-01-16T00:12:39.884 INFO:tasks.rados.rados.0.smithi123.stdout:690: delete oid 31 current snap is 0 2020-01-16T00:12:39.884 INFO:tasks.rados.rados.0.smithi123.stdout:674: finishing write tid 5 to smithi12313934-25 2020-01-16T00:12:39.884 INFO:tasks.rados.rados.0.smithi123.stdout:674: finishing write tid 6 to smithi12313934-25 2020-01-16T00:12:39.884 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 25 v 82 (ObjNum 85 snap 0 seq_num 85) dirty exists 2020-01-16T00:12:39.884 INFO:tasks.rados.rados.0.smithi123.stdout:674: left oid 25 (ObjNum 85 snap 0 seq_num 85) 2020-01-16T00:12:39.884 INFO:tasks.rados.rados.0.smithi123.stdout:676: finishing write tid 1 to smithi12313934-27 2020-01-16T00:12:39.884 INFO:tasks.rados.rados.0.smithi123.stdout:676: finishing write tid 2 to smithi12313934-27 2020-01-16T00:12:39.885 INFO:tasks.rados.rados.0.smithi123.stdout:677: expect (ObjNum 70 snap 0 seq_num 70) 2020-01-16T00:12:39.916 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:39.925 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-01-16T00:12:39.940 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:12:39.943+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:12:40.019 INFO:tasks.rados.rados.0.smithi123.stdout:678: finishing write tid 1 to smithi12313934-37 2020-01-16T00:12:40.019 INFO:tasks.rados.rados.0.smithi123.stdout:678: finishing write tid 2 to smithi12313934-37 2020-01-16T00:12:40.020 INFO:tasks.rados.rados.0.smithi123.stdout:674: done (15 left) 2020-01-16T00:12:40.020 INFO:tasks.rados.rados.0.smithi123.stdout:677: done (14 left) 2020-01-16T00:12:40.020 INFO:tasks.rados.rados.0.smithi123.stdout:690: done (13 left) 2020-01-16T00:12:40.020 INFO:tasks.rados.rados.0.smithi123.stdout:691: read oid 29 snap -1 2020-01-16T00:12:40.020 INFO:tasks.rados.rados.0.smithi123.stdout:691: expect (ObjNum 28 snap 0 seq_num 28) 2020-01-16T00:12:40.020 INFO:tasks.rados.rados.0.smithi123.stdout:692: write (excl) oid 18 current snap is 0 2020-01-16T00:12:40.020 INFO:tasks.rados.rados.0.smithi123.stdout:692: seq_num 96 ranges {413472=620140,1818036=510700,3120019=493453} 2020-01-16T00:12:40.030 INFO:tasks.rados.rados.0.smithi123.stdout:692: writing smithi12313934-18 from 413472 to 1033612 tid 1 2020-01-16T00:12:40.039 INFO:tasks.rados.rados.0.smithi123.stdout:692: writing smithi12313934-18 from 1818036 to 2328736 tid 2 2020-01-16T00:12:40.041 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:40.043+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:12:40.047 INFO:tasks.rados.rados.0.smithi123.stdout:692: writing smithi12313934-18 from 3120019 to 3613472 tid 3 2020-01-16T00:12:40.048 INFO:tasks.rados.rados.0.smithi123.stdout:693: write (excl) oid 46 current snap is 0 2020-01-16T00:12:40.048 INFO:tasks.rados.rados.0.smithi123.stdout:693: seq_num 97 ranges {430279=695389,1876518=654358,3031089=536797} 2020-01-16T00:12:40.053 INFO:tasks.rados.rados.0.smithi123.stdout:693: writing smithi12313934-46 from 430279 to 1125668 tid 1 2020-01-16T00:12:40.061 INFO:tasks.rados.rados.0.smithi123.stdout:693: writing smithi12313934-46 from 1876518 to 2530876 tid 2 2020-01-16T00:12:40.068 INFO:tasks.rados.rados.0.smithi123.stdout:693: writing smithi12313934-46 from 3031089 to 3567886 tid 3 2020-01-16T00:12:40.068 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:40.069 INFO:tasks.rados.rados.0.smithi123.stdout:678: finishing write tid 3 to smithi12313934-37 2020-01-16T00:12:40.069 INFO:tasks.rados.rados.0.smithi123.stdout:678: finishing write tid 5 to smithi12313934-37 2020-01-16T00:12:40.069 INFO:tasks.rados.rados.0.smithi123.stdout:678: finishing write tid 6 to smithi12313934-37 2020-01-16T00:12:40.069 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 37 v 112 (ObjNum 88 snap 0 seq_num 88) dirty exists 2020-01-16T00:12:40.069 INFO:tasks.rados.rados.0.smithi123.stdout:678: left oid 37 (ObjNum 88 snap 0 seq_num 88) 2020-01-16T00:12:40.069 INFO:tasks.rados.rados.0.smithi123.stdout:676: finishing write tid 3 to smithi12313934-27 2020-01-16T00:12:40.069 INFO:tasks.rados.rados.0.smithi123.stdout:676: finishing write tid 5 to smithi12313934-27 2020-01-16T00:12:40.070 INFO:tasks.rados.rados.0.smithi123.stdout:676: finishing write tid 6 to smithi12313934-27 2020-01-16T00:12:40.070 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 27 v 63 (ObjNum 87 snap 0 seq_num 87) dirty exists 2020-01-16T00:12:40.070 INFO:tasks.rados.rados.0.smithi123.stdout:676: left oid 27 (ObjNum 87 snap 0 seq_num 87) 2020-01-16T00:12:40.070 INFO:tasks.rados.rados.0.smithi123.stdout:679: finishing write tid 1 to smithi12313934-7 2020-01-16T00:12:40.070 INFO:tasks.rados.rados.0.smithi123.stdout:676: done (15 left) 2020-01-16T00:12:40.070 INFO:tasks.rados.rados.0.smithi123.stdout:678: done (14 left) 2020-01-16T00:12:40.070 INFO:tasks.rados.rados.0.smithi123.stdout:694: read oid 28 snap -1 2020-01-16T00:12:40.071 INFO:tasks.rados.rados.0.smithi123.stdout:694: expect (ObjNum 73 snap 0 seq_num 73) 2020-01-16T00:12:40.071 INFO:tasks.rados.rados.0.smithi123.stdout:695: write (excl) oid 14 current snap is 0 2020-01-16T00:12:40.071 INFO:tasks.rados.rados.0.smithi123.stdout:695: seq_num 98 ranges {447086=770638,1935000=798016,3458512=188574} 2020-01-16T00:12:40.075 INFO:tasks.rados.rados.0.smithi123.stdout:695: writing smithi12313934-14 from 447086 to 1217724 tid 1 2020-01-16T00:12:40.081 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:40.085 INFO:tasks.rados.rados.0.smithi123.stdout:695: writing smithi12313934-14 from 1935000 to 2733016 tid 2 2020-01-16T00:12:40.091 INFO:tasks.rados.rados.0.smithi123.stdout:695: writing smithi12313934-14 from 3458512 to 3647086 tid 3 2020-01-16T00:12:40.091 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:40.091 INFO:tasks.rados.rados.0.smithi123.stdout:679: finishing write tid 2 to smithi12313934-7 2020-01-16T00:12:40.091 INFO:tasks.rados.rados.0.smithi123.stdout:679: finishing write tid 3 to smithi12313934-7 2020-01-16T00:12:40.091 INFO:tasks.rados.rados.0.smithi123.stdout:679: finishing write tid 5 to smithi12313934-7 2020-01-16T00:12:40.091 INFO:tasks.rados.rados.0.smithi123.stdout:679: finishing write tid 6 to smithi12313934-7 2020-01-16T00:12:40.092 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 7 v 79 (ObjNum 89 snap 0 seq_num 89) dirty exists 2020-01-16T00:12:40.092 INFO:tasks.rados.rados.0.smithi123.stdout:679: left oid 7 (ObjNum 89 snap 0 seq_num 89) 2020-01-16T00:12:40.092 INFO:tasks.rados.rados.0.smithi123.stdout:680: finishing write tid 1 to smithi12313934-39 2020-01-16T00:12:40.092 INFO:tasks.rados.rados.0.smithi123.stdout:679: done (15 left) 2020-01-16T00:12:40.092 INFO:tasks.rados.rados.0.smithi123.stdout:696: read oid 13 snap -1 2020-01-16T00:12:40.092 INFO:tasks.rados.rados.0.smithi123.stdout:696: expect (ObjNum 12 snap 0 seq_num 12) 2020-01-16T00:12:40.092 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:40.093 INFO:tasks.rados.rados.0.smithi123.stdout:680: finishing write tid 2 to smithi12313934-39 2020-01-16T00:12:40.093 INFO:tasks.rados.rados.0.smithi123.stdout:680: finishing write tid 3 to smithi12313934-39 2020-01-16T00:12:40.093 INFO:tasks.rados.rados.0.smithi123.stdout:680: finishing write tid 5 to smithi12313934-39 2020-01-16T00:12:40.093 INFO:tasks.rados.rados.0.smithi123.stdout:680: finishing write tid 6 to smithi12313934-39 2020-01-16T00:12:40.093 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 39 v 83 (ObjNum 90 snap 0 seq_num 90) dirty exists 2020-01-16T00:12:40.093 INFO:tasks.rados.rados.0.smithi123.stdout:680: left oid 39 (ObjNum 90 snap 0 seq_num 90) 2020-01-16T00:12:40.093 INFO:tasks.rados.rados.0.smithi123.stdout:681: expect (ObjNum 75 snap 0 seq_num 75) 2020-01-16T00:12:40.095 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-01-16T00:12:40.142 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4, 7, 6] 2020-01-16T00:12:40.142 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2020-01-16T00:12:40.143 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2020-01-16T00:12:40.143 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-01-16T00:12:40.144 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:12:40.143+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:12:40.190 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:40.200 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-01-16T00:12:40.243 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:12:40.243+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:12:40.281 INFO:tasks.rados.rados.0.smithi123.stdout:682: finishing write tid 1 to smithi12313934-9 2020-01-16T00:12:40.281 INFO:tasks.rados.rados.0.smithi123.stdout:682: finishing write tid 2 to smithi12313934-9 2020-01-16T00:12:40.281 INFO:tasks.rados.rados.0.smithi123.stdout:680: done (15 left) 2020-01-16T00:12:40.281 INFO:tasks.rados.rados.0.smithi123.stdout:681: done (14 left) 2020-01-16T00:12:40.282 INFO:tasks.rados.rados.0.smithi123.stdout:697: read oid 6 snap -1 2020-01-16T00:12:40.282 INFO:tasks.rados.rados.0.smithi123.stdout:697: expect (ObjNum 5 snap 0 seq_num 5) 2020-01-16T00:12:40.282 INFO:tasks.rados.rados.0.smithi123.stdout:698: read oid 3 snap -1 2020-01-16T00:12:40.282 INFO:tasks.rados.rados.0.smithi123.stdout:698: expect (ObjNum 86 snap 0 seq_num 86) 2020-01-16T00:12:40.282 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:40.282 INFO:tasks.rados.rados.0.smithi123.stdout:682: finishing write tid 3 to smithi12313934-9 2020-01-16T00:12:40.282 INFO:tasks.rados.rados.0.smithi123.stdout:682: finishing write tid 5 to smithi12313934-9 2020-01-16T00:12:40.283 INFO:tasks.rados.rados.0.smithi123.stdout:682: finishing write tid 6 to smithi12313934-9 2020-01-16T00:12:40.283 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 9 v 67 (ObjNum 91 snap 0 seq_num 91) dirty exists 2020-01-16T00:12:40.283 INFO:tasks.rados.rados.0.smithi123.stdout:682: left oid 9 (ObjNum 91 snap 0 seq_num 91) 2020-01-16T00:12:40.283 INFO:tasks.rados.rados.0.smithi123.stdout:683: expect (ObjNum 54 snap 0 seq_num 54) 2020-01-16T00:12:40.304 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:40.314 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-01-16T00:12:40.360 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:12:40.361+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:12:40.384 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:12:40.386 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2020-01-16T00:12:40.398 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2020-01-16T00:12:39.956661+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19966716,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":19,"num_read_kb":9,"num_write":37,"num_write_kb":8943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":84,"acting":84,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":59,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8390224,"kb_used_data":1552,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746584496,"statfs":{"total":773094113280,"available":764502523904,"internally_reserved":8589934592,"allocated":1589248,"data_stored":537988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000731"},"pg_stats":[{"pgid":"3.f","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2020-01-16T00:12:34.697874+0000","last_change":"2020-01-16T00:12:32.829755+0000","last_active":"2020-01-16T00:12:34.697874+0000","last_peered":"2020-01-16T00:12:34.697874+0000","last_clean":"2020-01-16T00:12:34.697874+0000","last_became_active":"2020-01-16T00:12:32.829304+0000","last_became_peered":"2020-01-16T00:12:32.829304+0000","last_unstale":"2020-01-16T00:12:34.697874+0000","last_undegraded":"2020-01-16T00:12:34.697874+0000","last_fullsized":"2020-01-16T00:12:34.697874+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2020-01-16T00:12:34.697762+0000","last_change":"2020-01-16T00:12:32.829508+0000","last_active":"2020-01-16T00:12:34.697762+0000","last_peered":"2020-01-16T00:12:34.697762+0000","last_clean":"2020-01-16T00:12:34.697762+0000","last_became_active":"2020-01-16T00:12:32.829047+0000","last_became_peered":"2020-01-16T00:12:32.829047+0000","last_unstale":"2020-01-16T00:12:34.697762+0000","last_undegraded":"2020-01-16T00:12:34.697762+0000","last_fullsized":"2020-01-16T00:12:34.697762+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"23'3","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2020-01-16T00:12:34.911637+0000","last_change":"2020-01-16T00:12:32.834607+0000","last_active":"2020-01-16T00:12:34.911637+0000","last_peered":"2020-01-16T00:12:34.911637+0000","last_clean":"2020-01-16T00:12:34.911637+0000","last_became_active":"2020-01-16T00:12:32.834155+0000","last_became_peered":"2020-01-16T00:12:32.834155+0000","last_unstale":"2020-01-16T00:12:34.911637+0000","last_undegraded":"2020-01-16T00:12:34.911637+0000","last_fullsized":"2020-01-16T00:12:34.911637+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2067228,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2020-01-16T00:12:33.829423+0000","last_change":"2020-01-16T00:12:32.834625+0000","last_active":"2020-01-16T00:12:33.829423+0000","last_peered":"2020-01-16T00:12:33.829423+0000","last_clean":"2020-01-16T00:12:33.829423+0000","last_became_active":"2020-01-16T00:12:32.834210+0000","last_became_peered":"2020-01-16T00:12:32.834210+0000","last_unstale":"2020-01-16T00:12:33.829423+0000","last_undegraded":"2020-01-16T00:12:33.829423+0000","last_fullsized":"2020-01-16T00:12:33.829423+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"23'6","reported_seq":"19","reported_epoch":"23","state":"active+clean","last_fresh":"2020-01-16T00:12:34.944140+0000","last_change":"2020-01-16T00:12:32.833522+0000","last_active":"2020-01-16T00:12:34.944140+0000","last_peered":"2020-01-16T00:12:34.944140+0000","last_clean":"2020-01-16T00:12:34.944140+0000","last_became_active":"2020-01-16T00:12:32.833051+0000","last_became_peered":"2020-01-16T00:12:32.833051+0000","last_unstale":"2020-01-16T00:12:34.944140+0000","last_undegraded":"2020-01-16T00:12:34.944140+0000","last_fullsized":"2020-01-16T00:12:34.944140+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4151263,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"23'6","reported_seq":"19","reported_epoch":"23","state":"active+clean","last_fresh":"2020-01-16T00:12:34.932111+0000","last_change":"2020-01-16T00:12:32.834658+0000","last_active":"2020-01-16T00:12:34.932111+0000","last_peered":"2020-01-16T00:12:34.932111+0000","last_clean":"2020-01-16T00:12:34.932111+0000","last_became_active":"2020-01-16T00:12:32.832936+0000","last_became_peered":"2020-01-16T00:12:32.832936+0000","last_unstale":"2020-01-16T00:12:34.932111+0000","last_undegraded":"2020-01-16T00:12:34.932111+0000","last_fullsized":"2020-01-16T00:12:34.932111+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4117649,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2020-01-16T00:12:34.697172+0000","last_change":"2020-01-16T00:12:32.832821+0000","last_active":"2020-01-16T00:12:34.697172+0000","last_peered":"2020-01-16T00:12:34.697172+0000","last_clean":"2020-01-16T00:12:34.697172+0000","last_became_active":"2020-01-16T00:12:32.832272+0000","last_became_peered":"2020-01-16T00:12:32.832272+0000","last_unstale":"2020-01-16T00:12:34.697172+0000","last_undegraded":"2020-01-16T00:12:34.697172+0000","last_fullsized":"2020-01-16T00:12:34.697172+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"23'3","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2020-01-16T00:12:34.919811+0000","last_change":"2020-01-16T00:12:32.830727+0000","last_active":"2020-01-16T00:12:34.919811+0000","last_peered":"2020-01-16T00:12:34.919811+0000","last_clean":"2020-01-16T00:12:34.919811+0000","last_became_active":"2020-01-16T00:12:32.830301+0000","last_became_peered":"2020-01-16T00:12:32.830301+0000","last_unstale":"2020-01-16T00:12:34.919811+0000","last_undegraded":"2020-01-16T00:12:34.919811+0000","last_fullsized":"2020-01-16T00:12:34.919811+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2084035,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2020-01-16T00:12:34.697209+0000","last_change":"2020-01-16T00:12:32.830011+0000","last_active":"2020-01-16T00:12:34.697209+0000","last_peered":"2020-01-16T00:12:34.697209+0000","last_clean":"2020-01-16T00:12:34.697209+0000","last_became_active":"2020-01-16T00:12:32.829571+0000","last_became_peered":"2020-01-16T00:12:32.829571+0000","last_unstale":"2020-01-16T00:12:34.697209+0000","last_undegraded":"2020-01-16T00:12:34.697209+0000","last_fullsized":"2020-01-16T00:12:34.697209+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"16","reported_epoch":"22","state":"active+clean","last_fresh":"2020-01-16T00:12:34.698174+0000","last_change":"2020-01-16T00:12:20.031741+0000","last_active":"2020-01-16T00:12:34.698174+0000","last_peered":"2020-01-16T00:12:34.698174+0000","last_clean":"2020-01-16T00:12:34.698174+0000","last_became_active":"2020-01-16T00:12:20.030776+0000","last_became_peered":"2020-01-16T00:12:20.030776+0000","last_unstale":"2020-01-16T00:12:34.698174+0000","last_undegraded":"2020-01-16T00:12:34.698174+0000","last_fullsized":"2020-01-16T00:12:34.698174+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2020-01-16T00:12:34.698096+0000","last_change":"2020-01-16T00:12:32.832234+0000","last_active":"2020-01-16T00:12:34.698096+0000","last_peered":"2020-01-16T00:12:34.698096+0000","last_clean":"2020-01-16T00:12:34.698096+0000","last_became_active":"2020-01-16T00:12:32.831847+0000","last_became_peered":"2020-01-16T00:12:32.831847+0000","last_unstale":"2020-01-16T00:12:34.698096+0000","last_undegraded":"2020-01-16T00:12:34.698096+0000","last_fullsized":"2020-01-16T00:12:34.698096+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"16","reported_epoch":"22","state":"active+clean","last_fresh":"2020-01-16T00:12:34.697131+0000","last_change":"2020-01-16T00:12:20.032733+0000","last_active":"2020-01-16T00:12:34.697131+0000","last_peered":"2020-01-16T00:12:34.697131+0000","last_clean":"2020-01-16T00:12:34.697131+0000","last_became_active":"2020-01-16T00:12:20.032284+0000","last_became_peered":"2020-01-16T00:12:20.032284+0000","last_unstale":"2020-01-16T00:12:34.697131+0000","last_undegraded":"2020-01-16T00:12:34.697131+0000","last_fullsized":"2020-01-16T00:12:34.697131+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"16","reported_epoch":"22","state":"active+clean","last_fresh":"2020-01-16T00:12:33.831991+0000","last_change":"2020-01-16T00:12:20.032010+0000","last_active":"2020-01-16T00:12:33.831991+0000","last_peered":"2020-01-16T00:12:33.831991+0000","last_clean":"2020-01-16T00:12:33.831991+0000","last_became_active":"2020-01-16T00:12:20.031560+0000","last_became_peered":"2020-01-16T00:12:20.031560+0000","last_unstale":"2020-01-16T00:12:33.831991+0000","last_undegraded":"2020-01-16T00:12:33.831991+0000","last_fullsized":"2020-01-16T00:12:33.831991+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"12","reported_epoch":"22","state":"active+clean","last_fresh":"2020-01-16T00:12:34.697879+0000","last_change":"2020-01-16T00:12:31.817123+0000","last_active":"2020-01-16T00:12:34.697879+0000","last_peered":"2020-01-16T00:12:34.697879+0000","last_clean":"2020-01-16T00:12:34.697879+0000","last_became_active":"2020-01-16T00:12:31.816785+0000","last_became_peered":"2020-01-16T00:12:31.816785+0000","last_unstale":"2020-01-16T00:12:34.697879+0000","last_undegraded":"2020-01-16T00:12:34.697879+0000","last_fullsized":"2020-01-16T00:12:34.697879+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:30.783643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:30.783643+0000","last_clean_scrub_stamp":"2020-01-16T00:12:30.783643+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2020-01-16T00:12:34.696816+0000","last_change":"2020-01-16T00:12:32.836948+0000","last_active":"2020-01-16T00:12:34.696816+0000","last_peered":"2020-01-16T00:12:34.696816+0000","last_clean":"2020-01-16T00:12:34.696816+0000","last_became_active":"2020-01-16T00:12:32.836523+0000","last_became_peered":"2020-01-16T00:12:32.836523+0000","last_unstale":"2020-01-16T00:12:34.696816+0000","last_undegraded":"2020-01-16T00:12:34.696816+0000","last_fullsized":"2020-01-16T00:12:34.696816+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"16","reported_epoch":"22","state":"active+clean","last_fresh":"2020-01-16T00:12:34.697766+0000","last_change":"2020-01-16T00:12:20.032658+0000","last_active":"2020-01-16T00:12:34.697766+0000","last_peered":"2020-01-16T00:12:34.697766+0000","last_clean":"2020-01-16T00:12:34.697766+0000","last_became_active":"2020-01-16T00:12:20.030794+0000","last_became_peered":"2020-01-16T00:12:20.030794+0000","last_unstale":"2020-01-16T00:12:34.697766+0000","last_undegraded":"2020-01-16T00:12:34.697766+0000","last_fullsized":"2020-01-16T00:12:34.697766+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"12","reported_epoch":"22","state":"active+clean","last_fresh":"2020-01-16T00:12:33.831647+0000","last_change":"2020-01-16T00:12:31.817506+0000","last_active":"2020-01-16T00:12:33.831647+0000","last_peered":"2020-01-16T00:12:33.831647+0000","last_clean":"2020-01-16T00:12:33.831647+0000","last_became_active":"2020-01-16T00:12:31.817077+0000","last_became_peered":"2020-01-16T00:12:31.817077+0000","last_unstale":"2020-01-16T00:12:33.831647+0000","last_undegraded":"2020-01-16T00:12:33.831647+0000","last_fullsized":"2020-01-16T00:12:33.831647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:30.783643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:30.783643+0000","last_clean_scrub_stamp":"2020-01-16T00:12:30.783643+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"23'3","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2020-01-16T00:12:34.861527+0000","last_change":"2020-01-16T00:12:32.830294+0000","last_active":"2020-01-16T00:12:34.861527+0000","last_peered":"2020-01-16T00:12:34.861527+0000","last_clean":"2020-01-16T00:12:34.861527+0000","last_became_active":"2020-01-16T00:12:32.829870+0000","last_became_peered":"2020-01-16T00:12:32.829870+0000","last_unstale":"2020-01-16T00:12:34.861527+0000","last_undegraded":"2020-01-16T00:12:34.861527+0000","last_fullsized":"2020-01-16T00:12:34.861527+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2016807,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"16","reported_epoch":"22","state":"active+clean","last_fresh":"2020-01-16T00:12:34.697729+0000","last_change":"2020-01-16T00:12:20.033205+0000","last_active":"2020-01-16T00:12:34.697729+0000","last_peered":"2020-01-16T00:12:34.697729+0000","last_clean":"2020-01-16T00:12:34.697729+0000","last_became_active":"2020-01-16T00:12:20.030792+0000","last_became_peered":"2020-01-16T00:12:20.030792+0000","last_unstale":"2020-01-16T00:12:34.697729+0000","last_undegraded":"2020-01-16T00:12:34.697729+0000","last_fullsized":"2020-01-16T00:12:34.697729+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"12","reported_epoch":"22","state":"active+clean","last_fresh":"2020-01-16T00:12:33.829103+0000","last_change":"2020-01-16T00:12:31.815364+0000","last_active":"2020-01-16T00:12:33.829103+0000","last_peered":"2020-01-16T00:12:33.829103+0000","last_clean":"2020-01-16T00:12:33.829103+0000","last_became_active":"2020-01-16T00:12:31.814890+0000","last_became_peered":"2020-01-16T00:12:31.814890+0000","last_unstale":"2020-01-16T00:12:33.829103+0000","last_undegraded":"2020-01-16T00:12:33.829103+0000","last_fullsized":"2020-01-16T00:12:33.829103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:30.783643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:30.783643+0000","last_clean_scrub_stamp":"2020-01-16T00:12:30.783643+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"11","reported_epoch":"22","state":"active+clean","last_fresh":"2020-01-16T00:12:34.696917+0000","last_change":"2020-01-16T00:12:32.834986+0000","last_active":"2020-01-16T00:12:34.696917+0000","last_peered":"2020-01-16T00:12:34.696917+0000","last_clean":"2020-01-16T00:12:34.696917+0000","last_became_active":"2020-01-16T00:12:32.834537+0000","last_became_peered":"2020-01-16T00:12:32.834537+0000","last_unstale":"2020-01-16T00:12:34.696917+0000","last_undegraded":"2020-01-16T00:12:34.696917+0000","last_fullsized":"2020-01-16T00:12:34.696917+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"16","reported_epoch":"22","state":"active+clean","last_fresh":"2020-01-16T00:12:33.829052+0000","last_change":"2020-01-16T00:12:20.031201+0000","last_active":"2020-01-16T00:12:33.829052+0000","last_peered":"2020-01-16T00:12:33.829052+0000","last_clean":"2020-01-16T00:12:33.829052+0000","last_became_active":"2020-01-16T00:12:20.030752+0000","last_became_peered":"2020-01-16T00:12:20.030752+0000","last_unstale":"2020-01-16T00:12:33.829052+0000","last_undegraded":"2020-01-16T00:12:33.829052+0000","last_fullsized":"2020-01-16T00:12:33.829052+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"12","reported_epoch":"22","state":"active+clean","last_fresh":"2020-01-16T00:12:33.829317+0000","last_change":"2020-01-16T00:12:31.815506+0000","last_active":"2020-01-16T00:12:33.829317+0000","last_peered":"2020-01-16T00:12:33.829317+0000","last_clean":"2020-01-16T00:12:33.829317+0000","last_became_active":"2020-01-16T00:12:31.815111+0000","last_became_peered":"2020-01-16T00:12:31.815111+0000","last_unstale":"2020-01-16T00:12:33.829317+0000","last_undegraded":"2020-01-16T00:12:33.829317+0000","last_fullsized":"2020-01-16T00:12:33.829317+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:30.783643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:30.783643+0000","last_clean_scrub_stamp":"2020-01-16T00:12:30.783643+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"23'1","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2020-01-16T00:12:34.840300+0000","last_change":"2020-01-16T00:12:32.832002+0000","last_active":"2020-01-16T00:12:34.840300+0000","last_peered":"2020-01-16T00:12:34.840300+0000","last_clean":"2020-01-16T00:12:34.840300+0000","last_became_active":"2020-01-16T00:12:32.831577+0000","last_became_peered":"2020-01-16T00:12:32.831577+0000","last_unstale":"2020-01-16T00:12:34.840300+0000","last_undegraded":"2020-01-16T00:12:34.840300+0000","last_fullsized":"2020-01-16T00:12:34.840300+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1344857,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"23'3","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2020-01-16T00:12:34.896196+0000","last_change":"2020-01-16T00:12:32.834894+0000","last_active":"2020-01-16T00:12:34.896196+0000","last_peered":"2020-01-16T00:12:34.896196+0000","last_clean":"2020-01-16T00:12:34.896196+0000","last_became_active":"2020-01-16T00:12:32.834414+0000","last_became_peered":"2020-01-16T00:12:32.834414+0000","last_unstale":"2020-01-16T00:12:34.896196+0000","last_undegraded":"2020-01-16T00:12:34.896196+0000","last_fullsized":"2020-01-16T00:12:34.896196+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2050421,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":995,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"16","reported_epoch":"22","state":"active+clean","last_fresh":"2020-01-16T00:12:34.698833+0000","last_change":"2020-01-16T00:12:20.033023+0000","last_active":"2020-01-16T00:12:34.698833+0000","last_peered":"2020-01-16T00:12:34.698833+0000","last_clean":"2020-01-16T00:12:34.698833+0000","last_became_active":"2020-01-16T00:12:20.031576+0000","last_became_peered":"2020-01-16T00:12:20.031576+0000","last_unstale":"2020-01-16T00:12:34.698833+0000","last_undegraded":"2020-01-16T00:12:34.698833+0000","last_fullsized":"2020-01-16T00:12:34.698833+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"23'3","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2020-01-16T00:12:34.956955+0000","last_change":"2020-01-16T00:12:32.832313+0000","last_active":"2020-01-16T00:12:34.956955+0000","last_peered":"2020-01-16T00:12:34.956955+0000","last_clean":"2020-01-16T00:12:34.956955+0000","last_became_active":"2020-01-16T00:12:32.831843+0000","last_became_peered":"2020-01-16T00:12:32.831843+0000","last_unstale":"2020-01-16T00:12:34.956955+0000","last_undegraded":"2020-01-16T00:12:34.956955+0000","last_fullsized":"2020-01-16T00:12:34.956955+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2134456,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"16","reported_epoch":"22","state":"active+clean","last_fresh":"2020-01-16T00:12:34.698248+0000","last_change":"2020-01-16T00:12:20.034125+0000","last_active":"2020-01-16T00:12:34.698248+0000","last_peered":"2020-01-16T00:12:34.698248+0000","last_clean":"2020-01-16T00:12:34.698248+0000","last_became_active":"2020-01-16T00:12:20.033645+0000","last_became_peered":"2020-01-16T00:12:20.033645+0000","last_unstale":"2020-01-16T00:12:34.698248+0000","last_undegraded":"2020-01-16T00:12:34.698248+0000","last_fullsized":"2020-01-16T00:12:34.698248+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":19966716,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":19,"num_read_kb":9,"num_write":37,"num_write_kb":8943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28459008,"data_stored":28218156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":48,"acting":48,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":2,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":12,"acting":12,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048772,"kb_used_data":188,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323068,"statfs":{"total":96636764160,"available":95562821632,"internally_reserved":1073741824,"allocated":192512,"data_stored":64666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048772,"kb_used_data":188,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323068,"statfs":{"total":96636764160,"available":95562821632,"internally_reserved":1073741824,"allocated":192512,"data_stored":64666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048784,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323056,"statfs":{"total":96636764160,"available":95562809344,"internally_reserved":1073741824,"allocated":204800,"data_stored":69831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048772,"kb_used_data":188,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323068,"statfs":{"total":96636764160,"available":95562821632,"internally_reserved":1073741824,"allocated":192512,"data_stored":64666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048784,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323056,"statfs":{"total":96636764160,"available":95562809344,"internally_reserved":1073741824,"allocated":204800,"data_stored":69831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048784,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323056,"statfs":{"total":96636764160,"available":95562809344,"internally_reserved":1073741824,"allocated":204800,"data_stored":69831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048772,"kb_used_data":188,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323068,"statfs":{"total":96636764160,"available":95562821632,"internally_reserved":1073741824,"allocated":192512,"data_stored":64666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048784,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323056,"statfs":{"total":96636764160,"available":95562809344,"internally_reserved":1073741824,"allocated":204800,"data_stored":69831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}]}} 2020-01-16T00:12:40.401 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pgp_num 16 2020-01-16T00:12:40.410 INFO:tasks.rados.rados.0.smithi123.stdout:684: finishing write tid 1 to smithi12313934-45 2020-01-16T00:12:40.410 INFO:tasks.rados.rados.0.smithi123.stdout:684: finishing write tid 2 to smithi12313934-45 2020-01-16T00:12:40.410 INFO:tasks.rados.rados.0.smithi123.stdout:684: finishing write tid 3 to smithi12313934-45 2020-01-16T00:12:40.411 INFO:tasks.rados.rados.0.smithi123.stdout:684: finishing write tid 5 to smithi12313934-45 2020-01-16T00:12:40.411 INFO:tasks.rados.rados.0.smithi123.stdout:684: finishing write tid 6 to smithi12313934-45 2020-01-16T00:12:40.411 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 45 v 46 (ObjNum 92 snap 0 seq_num 92) dirty exists 2020-01-16T00:12:40.411 INFO:tasks.rados.rados.0.smithi123.stdout:684: left oid 45 (ObjNum 92 snap 0 seq_num 92) 2020-01-16T00:12:40.411 INFO:tasks.rados.rados.0.smithi123.stdout:686: expect (ObjNum 74 snap 0 seq_num 74) 2020-01-16T00:12:40.423 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:40.433 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-01-16T00:12:40.476 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:40.477+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:40.546 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:12:40.547+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:12:40.554 INFO:tasks.rados.rados.0.smithi123.stdout:687: expect (ObjNum 52 snap 0 seq_num 52) 2020-01-16T00:12:40.584 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:40.594 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-01-16T00:12:40.658 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:12:40.657+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:12:40.684 INFO:tasks.rados.rados.0.smithi123.stdout:685: finishing write tid 1 to smithi12313934-26 2020-01-16T00:12:40.684 INFO:tasks.rados.rados.0.smithi123.stdout:685: finishing write tid 2 to smithi12313934-26 2020-01-16T00:12:40.684 INFO:tasks.rados.rados.0.smithi123.stdout:688: finishing write tid 1 to smithi12313934-36 2020-01-16T00:12:40.684 INFO:tasks.rados.rados.0.smithi123.stdout:688: finishing write tid 2 to smithi12313934-36 2020-01-16T00:12:40.684 INFO:tasks.rados.rados.0.smithi123.stdout:688: finishing write tid 3 to smithi12313934-36 2020-01-16T00:12:40.685 INFO:tasks.rados.rados.0.smithi123.stdout:688: finishing write tid 5 to smithi12313934-36 2020-01-16T00:12:40.685 INFO:tasks.rados.rados.0.smithi123.stdout:688: finishing write tid 6 to smithi12313934-36 2020-01-16T00:12:40.685 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 36 v 32 (ObjNum 94 snap 0 seq_num 94) dirty exists 2020-01-16T00:12:40.685 INFO:tasks.rados.rados.0.smithi123.stdout:688: left oid 36 (ObjNum 94 snap 0 seq_num 94) 2020-01-16T00:12:40.685 INFO:tasks.rados.rados.0.smithi123.stdout:685: finishing write tid 3 to smithi12313934-26 2020-01-16T00:12:40.686 INFO:tasks.rados.rados.0.smithi123.stdout:685: finishing write tid 5 to smithi12313934-26 2020-01-16T00:12:40.686 INFO:tasks.rados.rados.0.smithi123.stdout:685: finishing write tid 6 to smithi12313934-26 2020-01-16T00:12:40.686 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 26 v 117 (ObjNum 93 snap 0 seq_num 93) dirty exists 2020-01-16T00:12:40.686 INFO:tasks.rados.rados.0.smithi123.stdout:685: left oid 26 (ObjNum 93 snap 0 seq_num 93) 2020-01-16T00:12:40.686 INFO:tasks.rados.rados.0.smithi123.stdout:682: done (15 left) 2020-01-16T00:12:40.686 INFO:tasks.rados.rados.0.smithi123.stdout:683: done (14 left) 2020-01-16T00:12:40.687 INFO:tasks.rados.rados.0.smithi123.stdout:684: done (13 left) 2020-01-16T00:12:40.687 INFO:tasks.rados.rados.0.smithi123.stdout:685: done (12 left) 2020-01-16T00:12:40.687 INFO:tasks.rados.rados.0.smithi123.stdout:686: done (11 left) 2020-01-16T00:12:40.687 INFO:tasks.rados.rados.0.smithi123.stdout:687: done (10 left) 2020-01-16T00:12:40.687 INFO:tasks.rados.rados.0.smithi123.stdout:688: done (9 left) 2020-01-16T00:12:40.687 INFO:tasks.rados.rados.0.smithi123.stdout:699: write (excl) oid 38 current snap is 0 2020-01-16T00:12:40.688 INFO:tasks.rados.rados.0.smithi123.stdout:699: seq_num 99 ranges {463893=562240,1593482=658027,2685935=559341} 2020-01-16T00:12:40.698 INFO:tasks.rados.rados.0.smithi123.stdout:699: writing smithi12313934-38 from 463893 to 1026133 tid 1 2020-01-16T00:12:40.705 INFO:tasks.rados.rados.0.smithi123.stdout:699: writing smithi12313934-38 from 1593482 to 2251509 tid 2 2020-01-16T00:12:40.710 INFO:tasks.rados.rados.0.smithi123.stdout:699: writing smithi12313934-38 from 2685935 to 3245276 tid 3 2020-01-16T00:12:40.710 INFO:tasks.rados.rados.0.smithi123.stdout:700: write oid 19 current snap is 0 2020-01-16T00:12:40.711 INFO:tasks.rados.rados.0.smithi123.stdout:700: seq_num 100 ranges {480700=637489,1651964=401685,2713358=770613} 2020-01-16T00:12:40.713 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:40.717 INFO:tasks.rados.rados.0.smithi123.stdout:700: writing smithi12313934-19 from 480700 to 1118189 tid 1 2020-01-16T00:12:40.723 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-01-16T00:12:40.723 INFO:tasks.rados.rados.0.smithi123.stdout:700: writing smithi12313934-19 from 1651964 to 2053649 tid 2 2020-01-16T00:12:40.730 INFO:tasks.rados.rados.0.smithi123.stdout:700: writing smithi12313934-19 from 2713358 to 3483971 tid 3 2020-01-16T00:12:40.731 INFO:tasks.rados.rados.0.smithi123.stdout:701: read oid 16 snap -1 2020-01-16T00:12:40.731 INFO:tasks.rados.rados.0.smithi123.stdout:701: expect (ObjNum 60 snap 0 seq_num 60) 2020-01-16T00:12:40.731 INFO:tasks.rados.rados.0.smithi123.stdout:702: write oid 47 current snap is 0 2020-01-16T00:12:40.731 INFO:tasks.rados.rados.0.smithi123.stdout:702: seq_num 101 ranges {497507=712738,1710446=661696,3140781=556726} 2020-01-16T00:12:40.737 INFO:tasks.rados.rados.0.smithi123.stdout:702: writing smithi12313934-47 from 497507 to 1210245 tid 1 2020-01-16T00:12:40.743 INFO:tasks.rados.rados.0.smithi123.stdout:702: writing smithi12313934-47 from 1710446 to 2372142 tid 2 2020-01-16T00:12:40.748 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:12:40.747+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:12:40.752 INFO:tasks.rados.rados.0.smithi123.stdout:702: writing smithi12313934-47 from 3140781 to 3697507 tid 3 2020-01-16T00:12:40.752 INFO:tasks.rados.rados.0.smithi123.stdout:703: write oid 25 current snap is 0 2020-01-16T00:12:40.753 INFO:tasks.rados.rados.0.smithi123.stdout:703: seq_num 102 ranges {514314=787987,1768928=405354,2768204=509510} 2020-01-16T00:12:40.757 INFO:tasks.rados.rados.0.smithi123.stdout:703: writing smithi12313934-25 from 514314 to 1302301 tid 1 2020-01-16T00:12:40.774 INFO:tasks.rados.rados.0.smithi123.stdout:703: writing smithi12313934-25 from 1768928 to 2174282 tid 2 2020-01-16T00:12:40.775 INFO:tasks.rados.rados.0.smithi123.stdout:703: writing smithi12313934-25 from 2768204 to 3277714 tid 3 2020-01-16T00:12:40.775 INFO:tasks.rados.rados.0.smithi123.stdout:704: delete oid 4 current snap is 0 2020-01-16T00:12:40.775 INFO:tasks.rados.rados.0.smithi123.stdout:689: expect (ObjNum 83 snap 0 seq_num 83) 2020-01-16T00:12:40.849 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:12:40.851+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:12:40.868 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:40.878 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-01-16T00:12:40.879 INFO:teuthology.orchestra.run.smithi123.stderr:set pool 3 pgp_num to 16 2020-01-16T00:12:40.950 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:40.951+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:12:40.967 INFO:tasks.rados.rados.0.smithi123.stdout:691: expect (ObjNum 28 snap 0 seq_num 28) 2020-01-16T00:12:40.979 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:40.989 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-01-16T00:12:41.044 INFO:tasks.rados.rados.0.smithi123.stdout:692: finishing write tid 1 to smithi12313934-18 2020-01-16T00:12:41.044 INFO:tasks.rados.rados.0.smithi123.stdout:692: finishing write tid 2 to smithi12313934-18 2020-01-16T00:12:41.044 INFO:tasks.rados.rados.0.smithi123.stdout:692: finishing write tid 3 to smithi12313934-18 2020-01-16T00:12:41.044 INFO:tasks.rados.rados.0.smithi123.stdout:692: finishing write tid 5 to smithi12313934-18 2020-01-16T00:12:41.045 INFO:tasks.rados.rados.0.smithi123.stdout:692: finishing write tid 6 to smithi12313934-18 2020-01-16T00:12:41.045 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 18 v 122 (ObjNum 96 snap 0 seq_num 96) dirty exists 2020-01-16T00:12:41.045 INFO:tasks.rados.rados.0.smithi123.stdout:692: left oid 18 (ObjNum 96 snap 0 seq_num 96) 2020-01-16T00:12:41.045 INFO:tasks.rados.rados.0.smithi123.stdout:693: finishing write tid 1 to smithi12313934-46 2020-01-16T00:12:41.045 INFO:tasks.rados.rados.0.smithi123.stdout:693: finishing write tid 2 to smithi12313934-46 2020-01-16T00:12:41.045 INFO:tasks.rados.rados.0.smithi123.stdout:695: finishing write tid 1 to smithi12313934-14 2020-01-16T00:12:41.045 INFO:tasks.rados.rados.0.smithi123.stdout:693: finishing write tid 3 to smithi12313934-46 2020-01-16T00:12:41.046 INFO:tasks.rados.rados.0.smithi123.stdout:693: finishing write tid 5 to smithi12313934-46 2020-01-16T00:12:41.046 INFO:tasks.rados.rados.0.smithi123.stdout:693: finishing write tid 6 to smithi12313934-46 2020-01-16T00:12:41.046 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 46 v 127 (ObjNum 97 snap 0 seq_num 97) dirty exists 2020-01-16T00:12:41.046 INFO:tasks.rados.rados.0.smithi123.stdout:693: left oid 46 (ObjNum 97 snap 0 seq_num 97) 2020-01-16T00:12:41.046 INFO:tasks.rados.rados.0.smithi123.stdout:694: expect (ObjNum 73 snap 0 seq_num 73) 2020-01-16T00:12:41.050 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:41.051+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:12:41.095 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:41.108 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-01-16T00:12:41.179 INFO:tasks.rados.rados.0.smithi123.stdout:695: finishing write tid 2 to smithi12313934-14 2020-01-16T00:12:41.179 INFO:tasks.rados.rados.0.smithi123.stdout:695: finishing write tid 3 to smithi12313934-14 2020-01-16T00:12:41.179 INFO:tasks.rados.rados.0.smithi123.stdout:695: finishing write tid 5 to smithi12313934-14 2020-01-16T00:12:41.179 INFO:tasks.rados.rados.0.smithi123.stdout:695: finishing write tid 6 to smithi12313934-14 2020-01-16T00:12:41.179 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 14 v 87 (ObjNum 98 snap 0 seq_num 98) dirty exists 2020-01-16T00:12:41.180 INFO:tasks.rados.rados.0.smithi123.stdout:695: left oid 14 (ObjNum 98 snap 0 seq_num 98) 2020-01-16T00:12:41.180 INFO:tasks.rados.rados.0.smithi123.stdout:696: expect (ObjNum 12 snap 0 seq_num 12) 2020-01-16T00:12:41.180 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:12:41.179+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:12:41.240 INFO:tasks.rados.rados.0.smithi123.stdout:697: expect (ObjNum 5 snap 0 seq_num 5) 2020-01-16T00:12:41.252 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:41.253 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:12:41.255+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:12:41.261 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-01-16T00:12:41.301 INFO:tasks.rados.rados.0.smithi123.stdout:698: expect (ObjNum 86 snap 0 seq_num 86) 2020-01-16T00:12:41.353 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:12:41.355+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:12:41.372 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-01-16T00:12:41.422 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:41.431 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-01-16T00:12:41.441 INFO:tasks.rados.rados.0.smithi123.stdout:699: finishing write tid 1 to smithi12313934-38 2020-01-16T00:12:41.441 INFO:tasks.rados.rados.0.smithi123.stdout:699: finishing write tid 2 to smithi12313934-38 2020-01-16T00:12:41.441 INFO:tasks.rados.rados.0.smithi123.stdout:700: finishing write tid 1 to smithi12313934-19 2020-01-16T00:12:41.442 INFO:tasks.rados.rados.0.smithi123.stdout:700: finishing write tid 2 to smithi12313934-19 2020-01-16T00:12:41.442 INFO:tasks.rados.rados.0.smithi123.stdout:699: finishing write tid 3 to smithi12313934-38 2020-01-16T00:12:41.442 INFO:tasks.rados.rados.0.smithi123.stdout:699: finishing write tid 5 to smithi12313934-38 2020-01-16T00:12:41.442 INFO:tasks.rados.rados.0.smithi123.stdout:699: finishing write tid 6 to smithi12313934-38 2020-01-16T00:12:41.442 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 38 v 61 (ObjNum 99 snap 0 seq_num 99) dirty exists 2020-01-16T00:12:41.442 INFO:tasks.rados.rados.0.smithi123.stdout:699: left oid 38 (ObjNum 99 snap 0 seq_num 99) 2020-01-16T00:12:41.442 INFO:tasks.rados.rados.0.smithi123.stdout:700: finishing write tid 3 to smithi12313934-19 2020-01-16T00:12:41.442 INFO:tasks.rados.rados.0.smithi123.stdout:689: done (14 left) 2020-01-16T00:12:41.443 INFO:tasks.rados.rados.0.smithi123.stdout:691: done (13 left) 2020-01-16T00:12:41.443 INFO:tasks.rados.rados.0.smithi123.stdout:692: done (12 left) 2020-01-16T00:12:41.443 INFO:tasks.rados.rados.0.smithi123.stdout:693: done (11 left) 2020-01-16T00:12:41.443 INFO:tasks.rados.rados.0.smithi123.stdout:694: done (10 left) 2020-01-16T00:12:41.443 INFO:tasks.rados.rados.0.smithi123.stdout:695: done (9 left) 2020-01-16T00:12:41.443 INFO:tasks.rados.rados.0.smithi123.stdout:696: done (8 left) 2020-01-16T00:12:41.443 INFO:tasks.rados.rados.0.smithi123.stdout:697: done (7 left) 2020-01-16T00:12:41.444 INFO:tasks.rados.rados.0.smithi123.stdout:698: done (6 left) 2020-01-16T00:12:41.444 INFO:tasks.rados.rados.0.smithi123.stdout:699: done (5 left) 2020-01-16T00:12:41.444 INFO:tasks.rados.rados.0.smithi123.stdout:704: done (4 left) 2020-01-16T00:12:41.444 INFO:tasks.rados.rados.0.smithi123.stdout:705: write (excl) oid 17 current snap is 0 2020-01-16T00:12:41.444 INFO:tasks.rados.rados.0.smithi123.stdout:705: seq_num 104 ranges {547928=538485,1769539=409023,2706697=532054} 2020-01-16T00:12:41.449 INFO:tasks.rados.rados.0.smithi123.stdout:705: writing smithi12313934-17 from 547928 to 1086413 tid 1 2020-01-16T00:12:41.454 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:41.453+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:41.456 INFO:tasks.rados.rados.0.smithi123.stdout:705: writing smithi12313934-17 from 1769539 to 2178562 tid 2 2020-01-16T00:12:41.463 INFO:tasks.rados.rados.0.smithi123.stdout:705: writing smithi12313934-17 from 2706697 to 3238751 tid 3 2020-01-16T00:12:41.463 INFO:tasks.rados.rados.0.smithi123.stdout:706: read oid 48 snap -1 2020-01-16T00:12:41.463 INFO:tasks.rados.rados.0.smithi123.stdout:706: expect (ObjNum 75 snap 0 seq_num 75) 2020-01-16T00:12:41.463 INFO:tasks.rados.rados.0.smithi123.stdout:700: finishing write tid 5 to smithi12313934-19 2020-01-16T00:12:41.463 INFO:tasks.rados.rados.0.smithi123.stdout:700: finishing write tid 6 to smithi12313934-19 2020-01-16T00:12:41.463 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 19 v 83 (ObjNum 100 snap 0 seq_num 100) dirty exists 2020-01-16T00:12:41.463 INFO:tasks.rados.rados.0.smithi123.stdout:700: left oid 19 (ObjNum 100 snap 0 seq_num 100) 2020-01-16T00:12:41.464 INFO:tasks.rados.rados.0.smithi123.stdout:701: expect (ObjNum 60 snap 0 seq_num 60) 2020-01-16T00:12:41.502 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:12:41.521 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:12:41.538 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:12:41.539 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:41.539 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:12:41.540 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:12:41.540 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:41.548 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-01-16T00:12:41.558 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:12:41.559 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:12:41.559+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:12:41.609 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:12:41.609 INFO:tasks.rados.rados.0.smithi123.stdout:702: finishing write tid 1 to smithi12313934-47 2020-01-16T00:12:41.609 INFO:tasks.rados.rados.0.smithi123.stdout:702: finishing write tid 2 to smithi12313934-47 2020-01-16T00:12:41.609 INFO:tasks.rados.rados.0.smithi123.stdout:702: finishing write tid 3 to smithi12313934-47 2020-01-16T00:12:41.610 INFO:tasks.rados.rados.0.smithi123.stdout:702: finishing write tid 5 to smithi12313934-47 2020-01-16T00:12:41.610 INFO:tasks.rados.rados.0.smithi123.stdout:702: finishing write tid 6 to smithi12313934-47 2020-01-16T00:12:41.610 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 47 v 86 (ObjNum 101 snap 0 seq_num 101) dirty exists 2020-01-16T00:12:41.610 INFO:tasks.rados.rados.0.smithi123.stdout:702: left oid 47 (ObjNum 101 snap 0 seq_num 101) 2020-01-16T00:12:41.610 INFO:tasks.rados.rados.0.smithi123.stdout:703: finishing write tid 1 to smithi12313934-25 2020-01-16T00:12:41.610 INFO:tasks.rados.rados.0.smithi123.stdout:703: finishing write tid 2 to smithi12313934-25 2020-01-16T00:12:41.611 INFO:tasks.rados.rados.0.smithi123.stdout:703: finishing write tid 3 to smithi12313934-25 2020-01-16T00:12:41.611 INFO:tasks.rados.rados.0.smithi123.stdout:703: finishing write tid 5 to smithi12313934-25 2020-01-16T00:12:41.611 INFO:tasks.rados.rados.0.smithi123.stdout:703: finishing write tid 6 to smithi12313934-25 2020-01-16T00:12:41.611 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 25 v 90 (ObjNum 102 snap 0 seq_num 102) dirty exists 2020-01-16T00:12:41.611 INFO:tasks.rados.rados.0.smithi123.stdout:703: left oid 25 (ObjNum 102 snap 0 seq_num 102) 2020-01-16T00:12:41.611 INFO:tasks.rados.rados.0.smithi123.stdout:705: finishing write tid 1 to smithi12313934-17 2020-01-16T00:12:41.611 INFO:tasks.rados.rados.0.smithi123.stdout:705: finishing write tid 2 to smithi12313934-17 2020-01-16T00:12:41.612 INFO:tasks.rados.rados.0.smithi123.stdout:705: finishing write tid 3 to smithi12313934-17 2020-01-16T00:12:41.612 INFO:tasks.rados.rados.0.smithi123.stdout:705: finishing write tid 5 to smithi12313934-17 2020-01-16T00:12:41.612 INFO:tasks.rados.rados.0.smithi123.stdout:700: done (5 left) 2020-01-16T00:12:41.612 INFO:tasks.rados.rados.0.smithi123.stdout:701: done (4 left) 2020-01-16T00:12:41.612 INFO:tasks.rados.rados.0.smithi123.stdout:702: done (3 left) 2020-01-16T00:12:41.612 INFO:tasks.rados.rados.0.smithi123.stdout:703: done (2 left) 2020-01-16T00:12:41.612 INFO:tasks.rados.rados.0.smithi123.stdout:707: read oid 30 snap -1 2020-01-16T00:12:41.613 INFO:tasks.rados.rados.0.smithi123.stdout:707: expect (ObjNum 84 snap 0 seq_num 84) 2020-01-16T00:12:41.613 INFO:tasks.rados.rados.0.smithi123.stdout:708: write oid 37 current snap is 0 2020-01-16T00:12:41.613 INFO:tasks.rados.rados.0.smithi123.stdout:708: seq_num 105 ranges {564735=613734,1828021=669034,3134120=630615} 2020-01-16T00:12:41.618 INFO:tasks.rados.rados.0.smithi123.stdout:708: writing smithi12313934-37 from 564735 to 1178469 tid 1 2020-01-16T00:12:41.622 INFO:tasks.rados.rados.0.smithi123.stdout:708: writing smithi12313934-37 from 1828021 to 2497055 tid 2 2020-01-16T00:12:41.630 INFO:tasks.rados.rados.0.smithi123.stdout:708: writing smithi12313934-37 from 3134120 to 3764735 tid 3 2020-01-16T00:12:41.630 INFO:tasks.rados.rados.0.smithi123.stdout:709: write oid 45 current snap is 0 2020-01-16T00:12:41.630 INFO:tasks.rados.rados.0.smithi123.stdout:709: seq_num 106 ranges {581542=688983,1886503=412692,2761543=670951} 2020-01-16T00:12:41.634 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5: {} 2020-01-16T00:12:41.637 INFO:tasks.rados.rados.0.smithi123.stdout:709: writing smithi12313934-45 from 581542 to 1270525 tid 1 2020-01-16T00:12:41.642 INFO:tasks.rados.rados.0.smithi123.stdout:709: writing smithi12313934-45 from 1886503 to 2299195 tid 2 2020-01-16T00:12:41.650 INFO:tasks.rados.rados.0.smithi123.stdout:709: writing smithi12313934-45 from 2761543 to 3432494 tid 3 2020-01-16T00:12:41.650 INFO:tasks.rados.rados.0.smithi123.stdout:710: read oid 28 snap -1 2020-01-16T00:12:41.651 INFO:tasks.rados.rados.0.smithi123.stdout:710: expect (ObjNum 73 snap 0 seq_num 73) 2020-01-16T00:12:41.651 INFO:tasks.rados.rados.0.smithi123.stdout:711: write oid 26 current snap is 0 2020-01-16T00:12:41.651 INFO:tasks.rados.rados.0.smithi123.stdout:711: seq_num 107 ranges {598349=480585,1544985=672703,2905319=482223} 2020-01-16T00:12:41.656 INFO:tasks.rados.rados.0.smithi123.stdout:711: writing smithi12313934-26 from 598349 to 1078934 tid 1 2020-01-16T00:12:41.656 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:12:41.657+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:12:41.660 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:12:41.662 INFO:tasks.rados.rados.0.smithi123.stdout:711: writing smithi12313934-26 from 1544985 to 2217688 tid 2 2020-01-16T00:12:41.669 INFO:tasks.rados.rados.0.smithi123.stdout:711: writing smithi12313934-26 from 2905319 to 3387542 tid 3 2020-01-16T00:12:41.669 INFO:tasks.rados.rados.0.smithi123.stdout:712: write (excl) oid 19 current snap is 0 2020-01-16T00:12:41.669 INFO:tasks.rados.rados.0.smithi123.stdout:712: seq_num 108 ranges {615156=555834,1603467=416361,2816389=693495} 2020-01-16T00:12:41.675 INFO:tasks.rados.rados.0.smithi123.stdout:712: writing smithi12313934-19 from 615156 to 1170990 tid 1 2020-01-16T00:12:41.680 INFO:tasks.rados.rados.0.smithi123.stdout:712: writing smithi12313934-19 from 1603467 to 2019828 tid 2 2020-01-16T00:12:41.681 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:41.681 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:12:41.681 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:12:41.682 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:12:41.682 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:41.682 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:12:41.684 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-01-16T00:12:41.684 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-01-16T00:12:41.684 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-01-16T00:12:41.684 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-01-16T00:12:41.685 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-01-16T00:12:41.685 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-01-16T00:12:41.685 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-01-16T00:12:41.685 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-01-16T00:12:41.689 INFO:tasks.rados.rados.0.smithi123.stdout:712: writing smithi12313934-19 from 2816389 to 3509884 tid 3 2020-01-16T00:12:41.689 INFO:tasks.rados.rados.0.smithi123.stdout:713: delete oid 34 current snap is 0 2020-01-16T00:12:41.689 INFO:tasks.rados.rados.0.smithi123.stdout:705: finishing write tid 6 to smithi12313934-17 2020-01-16T00:12:41.689 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 17 v 74 (ObjNum 104 snap 0 seq_num 104) dirty exists 2020-01-16T00:12:41.690 INFO:tasks.rados.rados.0.smithi123.stdout:705: left oid 17 (ObjNum 104 snap 0 seq_num 104) 2020-01-16T00:12:41.690 INFO:tasks.rados.rados.0.smithi123.stdout:706: expect (ObjNum 75 snap 0 seq_num 75) 2020-01-16T00:12:41.693 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-01-16T00:12:41.757 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:41.757+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:12:41.839 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:41.839 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:12:41.840 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:12:41.840 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:12:41.840 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.840 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.1.13:21 2.1 e24/19 2:bafac7f5:::smithi12313934-46 low_tier:head v 24'24, mlcod=24'22)", 2020-01-16T00:12:41.840 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:36.954506+0000", 2020-01-16T00:12:41.840 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.886056043, 2020-01-16T00:12:41.840 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0065224649999999999, 2020-01-16T00:12:41.840 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:41.841 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:41.841 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:41.841 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.841 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:41.841 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.954506+0000", 2020-01-16T00:12:41.841 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:41.841 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.842 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.842 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:41.842 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.954506+0000", 2020-01-16T00:12:41.842 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:41.842 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.842 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.842 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:41.842 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.954504+0000", 2020-01-16T00:12:41.843 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00036003199999999998 2020-01-16T00:12:41.843 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.843 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.843 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:41.843 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.954864+0000", 2020-01-16T00:12:41.843 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6718000000000002e-05 2020-01-16T00:12:41.843 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.844 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.844 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:41.844 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.954881+0000", 2020-01-16T00:12:41.844 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8170000000000001e-06 2020-01-16T00:12:41.844 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.844 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.844 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:41.844 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.954888+0000", 2020-01-16T00:12:41.845 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1836000000000003e-05 2020-01-16T00:12:41.845 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.845 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.845 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:41.845 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.954940+0000", 2020-01-16T00:12:41.845 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8199e-05 2020-01-16T00:12:41.845 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.845 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.846 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:41.846 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.954958+0000", 2020-01-16T00:12:41.846 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0060089929999999998 2020-01-16T00:12:41.846 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.846 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.846 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:41.846 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.960967+0000", 2020-01-16T00:12:41.847 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1935999999999998e-05 2020-01-16T00:12:41.847 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.847 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.847 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:41.847 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.961029+0000", 2020-01-16T00:12:41.847 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0065224649999999999 2020-01-16T00:12:41.847 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:41.847 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:41.848 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:41.848 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.848 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.848 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.1.13:22 2.1 e24/19 2:bafac7f5:::smithi12313934-46 low_tier:head v 24'25, mlcod=24'22)", 2020-01-16T00:12:41.848 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:36.957132+0000", 2020-01-16T00:12:41.848 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.8834300229999998, 2020-01-16T00:12:41.848 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041674609999999999, 2020-01-16T00:12:41.848 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:41.849 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:41.849 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:41.849 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.849 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:41.849 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.957132+0000", 2020-01-16T00:12:41.849 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:41.849 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.850 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.850 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:41.850 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.957132+0000", 2020-01-16T00:12:41.850 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:41.850 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.850 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.850 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:41.851 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.957130+0000", 2020-01-16T00:12:41.851 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4346999999999999e-05 2020-01-16T00:12:41.851 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.851 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.851 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:41.851 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.957194+0000", 2020-01-16T00:12:41.851 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6614999999999999e-05 2020-01-16T00:12:41.851 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.852 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.852 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:41.852 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.957211+0000", 2020-01-16T00:12:41.852 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6529999999999994e-06 2020-01-16T00:12:41.852 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.852 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.852 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:41.853 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.957218+0000", 2020-01-16T00:12:41.853 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9471999999999997e-05 2020-01-16T00:12:41.853 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.853 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.853 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:41.853 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.957258+0000", 2020-01-16T00:12:41.853 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8435000000000001e-05 2020-01-16T00:12:41.853 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.854 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.854 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:41.854 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.957286+0000", 2020-01-16T00:12:41.854 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039801250000000002 2020-01-16T00:12:41.854 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.854 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.854 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:41.855 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.961266+0000", 2020-01-16T00:12:41.855 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3266999999999998e-05 2020-01-16T00:12:41.855 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.855 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.855 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:41.855 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.961300+0000", 2020-01-16T00:12:41.855 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041674609999999999 2020-01-16T00:12:41.855 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:41.856 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:41.856 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:41.856 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.856 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.856 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.4.0:110 3.3 e24/20 3:cd231094:::smithi12313934-49:head v 24'40, mlcod=23'38)", 2020-01-16T00:12:41.856 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.130258+0000", 2020-01-16T00:12:41.856 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.7103041729999999, 2020-01-16T00:12:41.856 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0062283110000000003, 2020-01-16T00:12:41.857 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:41.857 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:41.857 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:41.857 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.857 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:41.857 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.130258+0000", 2020-01-16T00:12:41.857 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:41.857 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.858 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.858 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:41.858 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.130258+0000", 2020-01-16T00:12:41.858 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:12:41.858 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.858 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.858 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:41.859 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.130257+0000", 2020-01-16T00:12:41.859 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001916636 2020-01-16T00:12:41.859 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.859 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.859 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:41.859 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.132174+0000", 2020-01-16T00:12:41.859 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5815e-05 2020-01-16T00:12:41.859 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.860 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.860 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:41.860 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.132190+0000", 2020-01-16T00:12:41.860 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2159999999999995e-06 2020-01-16T00:12:41.860 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.860 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.860 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:41.860 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.132199+0000", 2020-01-16T00:12:41.861 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2299999999999995e-05 2020-01-16T00:12:41.861 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.861 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.861 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:41.861 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.132281+0000", 2020-01-16T00:12:41.861 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6641999999999999e-05 2020-01-16T00:12:41.861 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.862 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.862 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:41.862 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.132318+0000", 2020-01-16T00:12:41.862 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.004106129 2020-01-16T00:12:41.862 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.862 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.863 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:41.863 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.136424+0000", 2020-01-16T00:12:41.863 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2360000000000006e-05 2020-01-16T00:12:41.863 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.863 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.863 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:41.863 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.136486+0000", 2020-01-16T00:12:41.863 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0062283110000000003 2020-01-16T00:12:41.864 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:41.864 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:41.864 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:41.864 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.864 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.864 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.4.0:111 3.3 e24/20 3:c204aa4f:::smithi12313934-45:head v 24'41, mlcod=23'39)", 2020-01-16T00:12:41.864 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.143122+0000", 2020-01-16T00:12:41.865 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.6974407930000002, 2020-01-16T00:12:41.865 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.004384092, 2020-01-16T00:12:41.865 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:41.865 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:41.865 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:41.865 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.865 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:41.865 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.143122+0000", 2020-01-16T00:12:41.866 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:41.866 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.866 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.866 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:41.866 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.143122+0000", 2020-01-16T00:12:41.866 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999809 2020-01-16T00:12:41.866 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.867 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.867 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:41.867 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.143103+0000", 2020-01-16T00:12:41.867 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013691770000000001 2020-01-16T00:12:41.867 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.867 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.867 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:41.867 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.144472+0000", 2020-01-16T00:12:41.868 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7095000000000001e-05 2020-01-16T00:12:41.868 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.868 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.868 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:41.868 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.144499+0000", 2020-01-16T00:12:41.868 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8910000000000007e-06 2020-01-16T00:12:41.868 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.868 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.869 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:41.869 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.144509+0000", 2020-01-16T00:12:41.869 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2083999999999996e-05 2020-01-16T00:12:41.869 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.869 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.869 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:41.869 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.144581+0000", 2020-01-16T00:12:41.869 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4900999999999997e-05 2020-01-16T00:12:41.870 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.870 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.870 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:41.870 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.144616+0000", 2020-01-16T00:12:41.870 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0028443679999999999 2020-01-16T00:12:41.870 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.870 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.871 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:41.871 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.147460+0000", 2020-01-16T00:12:41.871 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5447e-05 2020-01-16T00:12:41.871 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.871 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.871 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:41.871 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.147506+0000", 2020-01-16T00:12:41.871 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.004384092 2020-01-16T00:12:41.872 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:41.872 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:41.872 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:41.872 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.872 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.872 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.5.13:36 2.0 e24/19 2:20e40cec:::smithi12313934-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 24'16, mlcod=23'14)", 2020-01-16T00:12:41.872 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.834879+0000", 2020-01-16T00:12:41.872 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.0056830579999998, 2020-01-16T00:12:41.873 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0075184850000000001, 2020-01-16T00:12:41.873 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:41.873 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:41.873 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:41.873 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.873 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:41.873 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.834879+0000", 2020-01-16T00:12:41.873 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:41.874 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.874 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.874 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:41.874 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.834879+0000", 2020-01-16T00:12:41.874 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:41.874 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.874 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.875 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:41.875 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.834877+0000", 2020-01-16T00:12:41.875 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031393089999999998 2020-01-16T00:12:41.875 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.875 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.875 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:41.875 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.838016+0000", 2020-01-16T00:12:41.875 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2144e-05 2020-01-16T00:12:41.876 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.876 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.876 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:41.876 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.838038+0000", 2020-01-16T00:12:41.876 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2058e-05 2020-01-16T00:12:41.876 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.876 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.876 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:41.877 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.838050+0000", 2020-01-16T00:12:41.877 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8533000000000003e-05 2020-01-16T00:12:41.877 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.877 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.877 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:41.877 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.838119+0000", 2020-01-16T00:12:41.877 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3591000000000003e-05 2020-01-16T00:12:41.878 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.878 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.878 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:41.878 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.838162+0000", 2020-01-16T00:12:41.878 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041910380000000002 2020-01-16T00:12:41.878 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.878 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.878 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:41.879 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.842353+0000", 2020-01-16T00:12:41.879 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4433e-05 2020-01-16T00:12:41.879 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.879 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.879 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:41.879 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.842398+0000", 2020-01-16T00:12:41.879 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0075184850000000001 2020-01-16T00:12:41.880 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:41.880 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:41.880 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:41.880 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.880 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.880 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.5.13:40 2.0 e24/19 2:20e40cec:::smithi12313934-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 24'17, mlcod=23'15)", 2020-01-16T00:12:41.880 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.849607+0000", 2020-01-16T00:12:41.881 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 3.9909549329999998, 2020-01-16T00:12:41.881 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083979039999999994, 2020-01-16T00:12:41.881 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:41.881 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:41.881 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:41.881 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.881 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:41.882 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.849607+0000", 2020-01-16T00:12:41.882 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:41.882 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.882 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.882 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:41.882 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.849607+0000", 2020-01-16T00:12:41.882 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:41.882 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.883 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.883 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:41.883 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.849605+0000", 2020-01-16T00:12:41.883 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0037534560000000001 2020-01-16T00:12:41.883 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.883 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.883 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:41.883 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.853358+0000", 2020-01-16T00:12:41.884 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9181000000000002e-05 2020-01-16T00:12:41.884 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.884 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.884 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:41.884 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.853377+0000", 2020-01-16T00:12:41.884 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0040000000000002e-06 2020-01-16T00:12:41.884 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.884 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.885 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:41.885 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.853384+0000", 2020-01-16T00:12:41.885 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2583000000000001e-05 2020-01-16T00:12:41.885 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.885 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.885 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:41.885 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.853437+0000", 2020-01-16T00:12:41.886 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8212999999999999e-05 2020-01-16T00:12:41.886 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.886 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.886 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:41.886 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.853465+0000", 2020-01-16T00:12:41.886 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0044847589999999996 2020-01-16T00:12:41.886 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.886 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.887 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:41.887 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.857950+0000", 2020-01-16T00:12:41.887 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5309999999999997e-05 2020-01-16T00:12:41.887 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.887 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.887 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:41.887 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.858005+0000", 2020-01-16T00:12:41.887 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083979039999999994 2020-01-16T00:12:41.888 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:41.888 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:41.888 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:41.888 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.888 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.888 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.5.13:42 2.0 e24/19 2:313458df:::smithi12313934-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 24'18, mlcod=24'16)", 2020-01-16T00:12:41.888 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.940683+0000", 2020-01-16T00:12:41.889 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 3.8998790940000001, 2020-01-16T00:12:41.889 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0097026960000000002, 2020-01-16T00:12:41.889 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:41.889 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:41.889 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:41.889 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.890 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:41.890 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.940683+0000", 2020-01-16T00:12:41.890 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:41.890 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.890 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.890 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:41.891 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.940683+0000", 2020-01-16T00:12:41.891 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:12:41.891 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.891 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.891 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:41.891 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.940680+0000", 2020-01-16T00:12:41.891 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003126775 2020-01-16T00:12:41.891 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.892 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.892 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:41.892 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.943807+0000", 2020-01-16T00:12:41.892 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5426999999999999e-05 2020-01-16T00:12:41.892 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.892 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.892 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:41.893 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.943833+0000", 2020-01-16T00:12:41.893 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7409999999999998e-06 2020-01-16T00:12:41.893 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.893 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.893 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:41.893 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.943841+0000", 2020-01-16T00:12:41.893 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011645399999999999 2020-01-16T00:12:41.893 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.894 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.894 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:41.894 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.943958+0000", 2020-01-16T00:12:41.894 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7291000000000003e-05 2020-01-16T00:12:41.894 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.894 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.894 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:41.894 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.944025+0000", 2020-01-16T00:12:41.895 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063108449999999998 2020-01-16T00:12:41.895 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.895 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.895 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:41.895 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.950336+0000", 2020-01-16T00:12:41.895 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9874999999999999e-05 2020-01-16T00:12:41.895 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.896 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.896 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:41.896 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.950386+0000", 2020-01-16T00:12:41.896 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0097026960000000002 2020-01-16T00:12:41.896 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:41.896 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:41.896 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:41.897 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.897 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.897 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.5.13:43 2.0 e24/19 2:313458df:::smithi12313934-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 24'19, mlcod=24'17)", 2020-01-16T00:12:41.897 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.954776+0000", 2020-01-16T00:12:41.897 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 3.8857867009999998, 2020-01-16T00:12:41.897 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063434609999999999, 2020-01-16T00:12:41.897 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:41.897 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:41.898 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:41.898 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.898 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:41.898 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.954776+0000", 2020-01-16T00:12:41.898 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:41.898 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.898 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.899 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:41.899 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.954776+0000", 2020-01-16T00:12:41.899 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:41.899 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.899 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.899 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:41.899 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.954773+0000", 2020-01-16T00:12:41.899 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030654699999999998 2020-01-16T00:12:41.900 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.900 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.900 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:41.900 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.957839+0000", 2020-01-16T00:12:41.900 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7915e-05 2020-01-16T00:12:41.900 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.900 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.900 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:41.901 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.957856+0000", 2020-01-16T00:12:41.901 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3810000000000001e-06 2020-01-16T00:12:41.901 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.901 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.901 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:41.901 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.957864+0000", 2020-01-16T00:12:41.901 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3652000000000007e-05 2020-01-16T00:12:41.902 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.902 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.902 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:41.902 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.957927+0000", 2020-01-16T00:12:41.902 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0889000000000002e-05 2020-01-16T00:12:41.902 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.902 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.902 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:41.903 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.957958+0000", 2020-01-16T00:12:41.903 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031031019999999999 2020-01-16T00:12:41.903 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.903 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.903 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:41.903 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.961061+0000", 2020-01-16T00:12:41.903 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7602000000000002e-05 2020-01-16T00:12:41.903 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.904 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.904 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:41.904 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.961119+0000", 2020-01-16T00:12:41.904 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063434609999999999 2020-01-16T00:12:41.904 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:41.904 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:41.904 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:41.904 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.905 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.905 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.4.13:24 2.1 e24/19 2:8b431f80:::smithi12313934-40 low_tier:head v 24'26, mlcod=24'24)", 2020-01-16T00:12:41.905 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.487871+0000", 2020-01-16T00:12:41.905 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 3.3526912219999998, 2020-01-16T00:12:41.905 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.004284866, 2020-01-16T00:12:41.905 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:41.905 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:41.906 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:41.906 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.906 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:41.906 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.487871+0000", 2020-01-16T00:12:41.906 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:41.906 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.906 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.907 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:41.907 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.487871+0000", 2020-01-16T00:12:41.907 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999948 2020-01-16T00:12:41.907 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.907 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.907 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:41.907 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.487866+0000", 2020-01-16T00:12:41.908 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00048013100000000001 2020-01-16T00:12:41.908 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.908 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.908 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:41.908 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.488346+0000", 2020-01-16T00:12:41.908 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2366000000000001e-05 2020-01-16T00:12:41.908 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.908 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.909 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:41.909 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.488378+0000", 2020-01-16T00:12:41.909 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3856000000000001e-05 2020-01-16T00:12:41.909 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.909 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.909 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:41.909 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.488392+0000", 2020-01-16T00:12:41.909 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2500000000000001e-05 2020-01-16T00:12:41.910 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.910 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.910 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:41.910 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.488455+0000", 2020-01-16T00:12:41.910 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7166999999999999e-05 2020-01-16T00:12:41.910 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.910 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.910 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:41.911 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.488482+0000", 2020-01-16T00:12:41.911 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036303860000000002 2020-01-16T00:12:41.911 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.911 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.911 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:41.911 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492112+0000", 2020-01-16T00:12:41.911 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.375e-05 2020-01-16T00:12:41.912 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.912 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.912 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:41.912 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492156+0000", 2020-01-16T00:12:41.912 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.004284866 2020-01-16T00:12:41.912 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:41.913 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:41.913 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:41.913 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.913 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.913 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.4.13:25 2.1 e24/19 2:8b431f80:::smithi12313934-40 low_tier:head v 24'27, mlcod=24'24)", 2020-01-16T00:12:41.913 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.488958+0000", 2020-01-16T00:12:41.913 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 3.3516040180000002, 2020-01-16T00:12:41.913 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003593742, 2020-01-16T00:12:41.914 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:41.914 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:41.914 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:41.914 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.914 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:41.914 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.488958+0000", 2020-01-16T00:12:41.914 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:41.915 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.915 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.915 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:41.915 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.488958+0000", 2020-01-16T00:12:41.915 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:12:41.915 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.915 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.915 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:41.916 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.488955+0000", 2020-01-16T00:12:41.916 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4795e-05 2020-01-16T00:12:41.916 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.916 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.916 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:41.916 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.489010+0000", 2020-01-16T00:12:41.916 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9344e-05 2020-01-16T00:12:41.916 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.917 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.917 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:41.917 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.489029+0000", 2020-01-16T00:12:41.917 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1071e-05 2020-01-16T00:12:41.917 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.917 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.917 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:41.918 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.489040+0000", 2020-01-16T00:12:41.918 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00024036899999999999 2020-01-16T00:12:41.918 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.918 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.918 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:41.918 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.489281+0000", 2020-01-16T00:12:41.918 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8859000000000001e-05 2020-01-16T00:12:41.918 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.919 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.919 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:41.919 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.489330+0000", 2020-01-16T00:12:41.919 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031488990000000001 2020-01-16T00:12:41.919 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.919 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.919 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:41.919 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492479+0000", 2020-01-16T00:12:41.920 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3497999999999993e-05 2020-01-16T00:12:41.920 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.920 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.920 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:41.920 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492552+0000", 2020-01-16T00:12:41.920 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003593742 2020-01-16T00:12:41.920 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:41.920 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:41.921 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:41.921 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.921 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.921 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:1158 3.3 e26/20 3:c204aa4f:::smithi12313934-45:head v 26'47, mlcod=24'45)", 2020-01-16T00:12:41.921 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:41.642470+0000", 2020-01-16T00:12:41.921 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.19809253800000001, 2020-01-16T00:12:41.921 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0027686830000000001, 2020-01-16T00:12:41.922 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:41.922 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:41.922 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:41.922 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.922 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:41.922 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.642470+0000", 2020-01-16T00:12:41.922 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:41.922 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.923 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.923 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:41.923 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.642470+0000", 2020-01-16T00:12:41.923 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:12:41.923 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.923 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.923 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:41.923 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.642467+0000", 2020-01-16T00:12:41.924 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00077819300000000005 2020-01-16T00:12:41.924 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.924 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.924 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:41.924 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.643245+0000", 2020-01-16T00:12:41.924 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7768e-05 2020-01-16T00:12:41.924 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.924 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.925 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:41.925 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.643263+0000", 2020-01-16T00:12:41.925 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2879999999999998e-06 2020-01-16T00:12:41.925 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.925 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.925 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:41.925 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.643269+0000", 2020-01-16T00:12:41.926 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4140000000000001e-05 2020-01-16T00:12:41.926 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.926 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.926 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:41.926 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.643313+0000", 2020-01-16T00:12:41.926 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0186999999999998e-05 2020-01-16T00:12:41.926 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.926 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.927 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:41.927 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.643334+0000", 2020-01-16T00:12:41.927 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001865263 2020-01-16T00:12:41.927 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.927 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.927 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:41.927 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.645199+0000", 2020-01-16T00:12:41.927 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9629000000000003e-05 2020-01-16T00:12:41.928 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.928 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.928 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:41.928 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.645238+0000", 2020-01-16T00:12:41.928 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0027686830000000001 2020-01-16T00:12:41.928 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:41.928 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:41.928 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:41.929 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.929 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.929 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:1159 3.3 e26/20 3:c204aa4f:::smithi12313934-45:head v 26'48, mlcod=24'46)", 2020-01-16T00:12:41.929 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:41.648126+0000", 2020-01-16T00:12:41.929 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.19243584999999999, 2020-01-16T00:12:41.929 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0044647230000000003, 2020-01-16T00:12:41.929 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:41.930 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:41.930 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:41.930 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.930 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:41.930 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.648126+0000", 2020-01-16T00:12:41.930 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:41.930 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.930 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.931 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:41.931 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.648126+0000", 2020-01-16T00:12:41.931 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:41.931 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.931 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.931 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:41.931 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.648125+0000", 2020-01-16T00:12:41.931 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00038849000000000002 2020-01-16T00:12:41.932 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.932 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.932 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:41.933 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.648514+0000", 2020-01-16T00:12:41.933 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0025e-05 2020-01-16T00:12:41.933 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.934 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.934 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:41.934 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.648524+0000", 2020-01-16T00:12:41.934 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0359999999999997e-06 2020-01-16T00:12:41.935 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.935 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.935 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:41.935 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.648529+0000", 2020-01-16T00:12:41.935 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2657000000000003e-05 2020-01-16T00:12:41.935 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.935 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.935 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:41.936 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.648561+0000", 2020-01-16T00:12:41.936 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9491e-05 2020-01-16T00:12:41.936 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.936 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.936 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:41.936 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.648581+0000", 2020-01-16T00:12:41.936 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039626219999999998 2020-01-16T00:12:41.936 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.937 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.937 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:41.937 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.652543+0000", 2020-01-16T00:12:41.937 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7793999999999997e-05 2020-01-16T00:12:41.937 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.937 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.937 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:41.938 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.652591+0000", 2020-01-16T00:12:41.938 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0044647230000000003 2020-01-16T00:12:41.938 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:41.938 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:41.938 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:41.938 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.938 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.938 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:1160 3.3 e26/20 3:c204aa4f:::smithi12313934-45:head v 26'49, mlcod=26'47)", 2020-01-16T00:12:41.939 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:41.655373+0000", 2020-01-16T00:12:41.939 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.18518880200000001, 2020-01-16T00:12:41.939 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003742408, 2020-01-16T00:12:41.939 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:41.939 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:41.939 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:41.939 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.939 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:41.940 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.655373+0000", 2020-01-16T00:12:41.940 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:41.940 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.940 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.940 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:41.940 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.655373+0000", 2020-01-16T00:12:41.940 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:41.940 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.941 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.941 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:41.941 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.655371+0000", 2020-01-16T00:12:41.941 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0016852219999999999 2020-01-16T00:12:41.941 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.941 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.941 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:41.942 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.657057+0000", 2020-01-16T00:12:41.942 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6586e-05 2020-01-16T00:12:41.942 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.942 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.942 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:41.942 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.657073+0000", 2020-01-16T00:12:41.942 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5209999999999999e-06 2020-01-16T00:12:41.942 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.943 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.943 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:41.943 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.657080+0000", 2020-01-16T00:12:41.943 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3679e-05 2020-01-16T00:12:41.943 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.943 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.943 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:41.943 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.657123+0000", 2020-01-16T00:12:41.944 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0659e-05 2020-01-16T00:12:41.944 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.944 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.944 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:41.944 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.657144+0000", 2020-01-16T00:12:41.944 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0019336500000000001 2020-01-16T00:12:41.944 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.945 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.945 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:41.945 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.659078+0000", 2020-01-16T00:12:41.945 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8247000000000002e-05 2020-01-16T00:12:41.945 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.945 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.945 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:41.945 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.659116+0000", 2020-01-16T00:12:41.946 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003742408 2020-01-16T00:12:41.946 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:41.946 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:41.946 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:41.946 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.946 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.946 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:1161 3.3 e26/20 3:c204aa4f:::smithi12313934-45:head v 26'50, mlcod=26'47)", 2020-01-16T00:12:41.946 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:41.658246+0000", 2020-01-16T00:12:41.947 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.182316121, 2020-01-16T00:12:41.947 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0011861160000000001, 2020-01-16T00:12:41.947 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:41.947 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:41.947 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:41.947 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.947 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:41.947 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.658246+0000", 2020-01-16T00:12:41.948 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:41.948 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.948 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.948 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:41.948 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.658246+0000", 2020-01-16T00:12:41.948 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:41.948 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.948 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.949 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:41.949 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.658244+0000", 2020-01-16T00:12:41.949 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2130000000000001e-05 2020-01-16T00:12:41.949 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.949 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.949 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:41.949 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.658256+0000", 2020-01-16T00:12:41.950 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4409e-05 2020-01-16T00:12:41.950 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.950 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.950 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:41.950 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.658271+0000", 2020-01-16T00:12:41.950 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2046e-05 2020-01-16T00:12:41.950 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.950 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.951 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:41.951 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.658283+0000", 2020-01-16T00:12:41.951 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2694000000000001e-05 2020-01-16T00:12:41.951 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.951 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.951 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:41.951 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.658335+0000", 2020-01-16T00:12:41.951 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0273e-05 2020-01-16T00:12:41.952 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.952 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.952 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:41.952 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.658366+0000", 2020-01-16T00:12:41.952 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001027388 2020-01-16T00:12:41.952 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.952 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:41.953 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.659393+0000", 2020-01-16T00:12:41.953 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9331e-05 2020-01-16T00:12:41.953 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:41.953 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:41.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:41.953 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.659432+0000", 2020-01-16T00:12:41.953 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0011861160000000001 2020-01-16T00:12:41.954 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:41.954 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:41.954 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:41.954 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:41.954 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:41.954 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:41.955 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2020-01-16T00:12:41.956 INFO:tasks.rados.rados.0.smithi123.stdout:707: expect (ObjNum 84 snap 0 seq_num 84) 2020-01-16T00:12:41.956 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:12:41.955+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:12:41.957 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-01-16T00:12:42.057 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:42.059+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:12:42.078 INFO:tasks.rados.rados.0.smithi123.stdout:708: finishing write tid 1 to smithi12313934-37 2020-01-16T00:12:42.078 INFO:tasks.rados.rados.0.smithi123.stdout:708: finishing write tid 2 to smithi12313934-37 2020-01-16T00:12:42.078 INFO:tasks.rados.rados.0.smithi123.stdout:708: finishing write tid 3 to smithi12313934-37 2020-01-16T00:12:42.078 INFO:tasks.rados.rados.0.smithi123.stdout:708: finishing write tid 5 to smithi12313934-37 2020-01-16T00:12:42.079 INFO:tasks.rados.rados.0.smithi123.stdout:708: finishing write tid 6 to smithi12313934-37 2020-01-16T00:12:42.079 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 37 v 131 (ObjNum 105 snap 0 seq_num 105) dirty exists 2020-01-16T00:12:42.079 INFO:tasks.rados.rados.0.smithi123.stdout:708: left oid 37 (ObjNum 105 snap 0 seq_num 105) 2020-01-16T00:12:42.079 INFO:tasks.rados.rados.0.smithi123.stdout:709: finishing write tid 1 to smithi12313934-45 2020-01-16T00:12:42.079 INFO:tasks.rados.rados.0.smithi123.stdout:709: finishing write tid 2 to smithi12313934-45 2020-01-16T00:12:42.079 INFO:tasks.rados.rados.0.smithi123.stdout:705: done (8 left) 2020-01-16T00:12:42.079 INFO:tasks.rados.rados.0.smithi123.stdout:706: done (7 left) 2020-01-16T00:12:42.079 INFO:tasks.rados.rados.0.smithi123.stdout:707: done (6 left) 2020-01-16T00:12:42.080 INFO:tasks.rados.rados.0.smithi123.stdout:708: done (5 left) 2020-01-16T00:12:42.080 INFO:tasks.rados.rados.0.smithi123.stdout:713: done (4 left) 2020-01-16T00:12:42.080 INFO:tasks.rados.rados.0.smithi123.stdout:714: write (excl) oid 3 current snap is 0 2020-01-16T00:12:42.080 INFO:tasks.rados.rados.0.smithi123.stdout:714: seq_num 110 ranges {648770=706332,2120431=420030,3271235=432392} 2020-01-16T00:12:42.085 INFO:tasks.rados.rados.0.smithi123.stdout:714: writing smithi12313934-3 from 648770 to 1355102 tid 1 2020-01-16T00:12:42.092 INFO:tasks.rados.rados.0.smithi123.stdout:714: writing smithi12313934-3 from 2120431 to 2540461 tid 2 2020-01-16T00:12:42.101 INFO:tasks.rados.rados.0.smithi123.stdout:714: writing smithi12313934-3 from 3271235 to 3703627 tid 3 2020-01-16T00:12:42.102 INFO:tasks.rados.rados.0.smithi123.stdout:715: delete oid 7 current snap is 0 2020-01-16T00:12:42.102 INFO:tasks.rados.rados.0.smithi123.stdout:709: finishing write tid 3 to smithi12313934-45 2020-01-16T00:12:42.102 INFO:tasks.rados.rados.0.smithi123.stdout:710: expect (ObjNum 73 snap 0 seq_num 73) 2020-01-16T00:12:42.114 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:42.114 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:12:42.114 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:12:42.115 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:42.123 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-01-16T00:12:42.162 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:12:42.161+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:12:42.240 INFO:tasks.rados.rados.0.smithi123.stdout:709: finishing write tid 5 to smithi12313934-45 2020-01-16T00:12:42.240 INFO:tasks.rados.rados.0.smithi123.stdout:709: finishing write tid 6 to smithi12313934-45 2020-01-16T00:12:42.240 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 45 v 50 (ObjNum 106 snap 0 seq_num 106) dirty exists 2020-01-16T00:12:42.240 INFO:tasks.rados.rados.0.smithi123.stdout:709: left oid 45 (ObjNum 106 snap 0 seq_num 106) 2020-01-16T00:12:42.241 INFO:tasks.rados.rados.0.smithi123.stdout:711: finishing write tid 1 to smithi12313934-26 2020-01-16T00:12:42.241 INFO:tasks.rados.rados.0.smithi123.stdout:711: finishing write tid 2 to smithi12313934-26 2020-01-16T00:12:42.241 INFO:tasks.rados.rados.0.smithi123.stdout:712: finishing write tid 1 to smithi12313934-19 2020-01-16T00:12:42.241 INFO:tasks.rados.rados.0.smithi123.stdout:711: finishing write tid 3 to smithi12313934-26 2020-01-16T00:12:42.241 INFO:tasks.rados.rados.0.smithi123.stdout:711: finishing write tid 5 to smithi12313934-26 2020-01-16T00:12:42.241 INFO:tasks.rados.rados.0.smithi123.stdout:711: finishing write tid 6 to smithi12313934-26 2020-01-16T00:12:42.241 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 26 v 136 (ObjNum 107 snap 0 seq_num 107) dirty exists 2020-01-16T00:12:42.241 INFO:tasks.rados.rados.0.smithi123.stdout:711: left oid 26 (ObjNum 107 snap 0 seq_num 107) 2020-01-16T00:12:42.242 INFO:tasks.rados.rados.0.smithi123.stdout:712: finishing write tid 2 to smithi12313934-19 2020-01-16T00:12:42.242 INFO:tasks.rados.rados.0.smithi123.stdout:712: finishing write tid 3 to smithi12313934-19 2020-01-16T00:12:42.242 INFO:tasks.rados.rados.0.smithi123.stdout:712: finishing write tid 5 to smithi12313934-19 2020-01-16T00:12:42.242 INFO:tasks.rados.rados.0.smithi123.stdout:709: done (5 left) 2020-01-16T00:12:42.242 INFO:tasks.rados.rados.0.smithi123.stdout:710: done (4 left) 2020-01-16T00:12:42.242 INFO:tasks.rados.rados.0.smithi123.stdout:711: done (3 left) 2020-01-16T00:12:42.242 INFO:tasks.rados.rados.0.smithi123.stdout:715: done (2 left) 2020-01-16T00:12:42.243 INFO:tasks.rados.rados.0.smithi123.stdout:716: read oid 39 snap -1 2020-01-16T00:12:42.243 INFO:tasks.rados.rados.0.smithi123.stdout:716: expect (ObjNum 90 snap 0 seq_num 90) 2020-01-16T00:12:42.243 INFO:tasks.rados.rados.0.smithi123.stdout:717: write (excl) oid 25 current snap is 0 2020-01-16T00:12:42.243 INFO:tasks.rados.rados.0.smithi123.stdout:717: seq_num 112 ranges {682384=456830,1721042=423699,2809728=454936,3731357=151027} 2020-01-16T00:12:42.254 INFO:tasks.rados.rados.0.smithi123.stdout:717: writing smithi12313934-25 from 682384 to 1139214 tid 1 2020-01-16T00:12:42.260 INFO:tasks.rados.rados.0.smithi123.stdout:717: writing smithi12313934-25 from 1721042 to 2144741 tid 2 2020-01-16T00:12:42.262 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:42.261+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:42.266 INFO:tasks.rados.rados.0.smithi123.stdout:717: writing smithi12313934-25 from 2809728 to 3264664 tid 3 2020-01-16T00:12:42.270 INFO:tasks.rados.rados.0.smithi123.stdout:717: writing smithi12313934-25 from 3731357 to 3882384 tid 4 2020-01-16T00:12:42.270 INFO:tasks.rados.rados.0.smithi123.stdout:718: write oid 21 current snap is 0 2020-01-16T00:12:42.270 INFO:tasks.rados.rados.0.smithi123.stdout:718: seq_num 113 ranges {699191=532079,1779524=567357,2837151=666208} 2020-01-16T00:12:42.271 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:42.271 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:12:42.271 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:12:42.271 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:12:42.272 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:42.277 INFO:tasks.rados.rados.0.smithi123.stdout:718: writing smithi12313934-21 from 699191 to 1231270 tid 1 2020-01-16T00:12:42.280 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-01-16T00:12:42.283 INFO:tasks.rados.rados.0.smithi123.stdout:718: writing smithi12313934-21 from 1779524 to 2346881 tid 2 2020-01-16T00:12:42.290 INFO:tasks.rados.rados.0.smithi123.stdout:718: writing smithi12313934-21 from 2837151 to 3503359 tid 3 2020-01-16T00:12:42.290 INFO:tasks.rados.rados.0.smithi123.stdout:719: write oid 22 current snap is 0 2020-01-16T00:12:42.291 INFO:tasks.rados.rados.0.smithi123.stdout:719: seq_num 114 ranges {715998=607328,1838006=427368,2864574=477480,3863835=52163} 2020-01-16T00:12:42.298 INFO:tasks.rados.rados.0.smithi123.stdout:719: writing smithi12313934-22 from 715998 to 1323326 tid 1 2020-01-16T00:12:42.304 INFO:tasks.rados.rados.0.smithi123.stdout:719: writing smithi12313934-22 from 1838006 to 2265374 tid 2 2020-01-16T00:12:42.319 INFO:tasks.rados.rados.0.smithi123.stdout:719: writing smithi12313934-22 from 2864574 to 3342054 tid 3 2020-01-16T00:12:42.319 INFO:tasks.rados.rados.0.smithi123.stdout:719: writing smithi12313934-22 from 3863835 to 3915998 tid 4 2020-01-16T00:12:42.319 INFO:tasks.rados.rados.0.smithi123.stdout:720: write (excl) oid 20 current snap is 0 2020-01-16T00:12:42.320 INFO:tasks.rados.rados.0.smithi123.stdout:720: seq_num 115 ranges {732805=798930,2296488=571026,3291997=405105} 2020-01-16T00:12:42.325 INFO:tasks.rados.rados.0.smithi123.stdout:720: writing smithi12313934-20 from 732805 to 1531735 tid 1 2020-01-16T00:12:42.335 INFO:tasks.rados.rados.0.smithi123.stdout:720: writing smithi12313934-20 from 2296488 to 2867514 tid 2 2020-01-16T00:12:42.341 INFO:tasks.rados.rados.0.smithi123.stdout:720: writing smithi12313934-20 from 3291997 to 3697102 tid 3 2020-01-16T00:12:42.341 INFO:tasks.rados.rados.0.smithi123.stdout:721: write (excl) oid 10 current snap is 0 2020-01-16T00:12:42.341 INFO:tasks.rados.rados.0.smithi123.stdout:721: seq_num 116 ranges {749612=474179,1954970=431037,2919420=616377} 2020-01-16T00:12:42.347 INFO:tasks.rados.rados.0.smithi123.stdout:721: writing smithi12313934-10 from 749612 to 1223791 tid 1 2020-01-16T00:12:42.354 INFO:tasks.rados.rados.0.smithi123.stdout:721: writing smithi12313934-10 from 1954970 to 2386007 tid 2 2020-01-16T00:12:42.361 INFO:tasks.rados.rados.0.smithi123.stdout:721: writing smithi12313934-10 from 2919420 to 3535797 tid 3 2020-01-16T00:12:42.361 INFO:tasks.rados.rados.0.smithi123.stdout:722: read oid 26 snap -1 2020-01-16T00:12:42.361 INFO:tasks.rados.rados.0.smithi123.stdout:722: expect (ObjNum 107 snap 0 seq_num 107) 2020-01-16T00:12:42.361 INFO:tasks.rados.rados.0.smithi123.stdout:723: write oid 14 current snap is 0 2020-01-16T00:12:42.361 INFO:tasks.rados.rados.0.smithi123.stdout:723: seq_num 117 ranges {766419=549428,2013452=574695,3346843=427649} 2020-01-16T00:12:42.366 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:42.365+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:42.368 INFO:tasks.rados.rados.0.smithi123.stdout:723: writing smithi12313934-14 from 766419 to 1315847 tid 1 2020-01-16T00:12:42.387 INFO:tasks.rados.rados.0.smithi123.stdout:723: writing smithi12313934-14 from 2013452 to 2588147 tid 2 2020-01-16T00:12:42.388 INFO:tasks.rados.rados.0.smithi123.stdout:723: writing smithi12313934-14 from 3346843 to 3774492 tid 3 2020-01-16T00:12:42.388 INFO:tasks.rados.rados.0.smithi123.stdout:724: read oid 42 snap -1 2020-01-16T00:12:42.388 INFO:tasks.rados.rados.0.smithi123.stdout:724: expect (ObjNum 41 snap 0 seq_num 41) 2020-01-16T00:12:42.388 INFO:tasks.rados.rados.0.smithi123.stdout:725: read oid 47 snap -1 2020-01-16T00:12:42.389 INFO:tasks.rados.rados.0.smithi123.stdout:725: expect (ObjNum 101 snap 0 seq_num 101) 2020-01-16T00:12:42.389 INFO:tasks.rados.rados.0.smithi123.stdout:726: delete oid 17 current snap is 0 2020-01-16T00:12:42.389 INFO:tasks.rados.rados.0.smithi123.stdout:712: finishing write tid 6 to smithi12313934-19 2020-01-16T00:12:42.389 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 19 v 87 (ObjNum 108 snap 0 seq_num 108) dirty exists 2020-01-16T00:12:42.389 INFO:tasks.rados.rados.0.smithi123.stdout:712: left oid 19 (ObjNum 108 snap 0 seq_num 108) 2020-01-16T00:12:42.390 INFO:tasks.rados.rados.0.smithi123.stdout:714: finishing write tid 1 to smithi12313934-3 2020-01-16T00:12:42.390 INFO:tasks.rados.rados.0.smithi123.stdout:714: finishing write tid 2 to smithi12313934-3 2020-01-16T00:12:42.390 INFO:tasks.rados.rados.0.smithi123.stdout:714: finishing write tid 3 to smithi12313934-3 2020-01-16T00:12:42.390 INFO:tasks.rados.rados.0.smithi123.stdout:714: finishing write tid 5 to smithi12313934-3 2020-01-16T00:12:42.390 INFO:tasks.rados.rados.0.smithi123.stdout:714: finishing write tid 6 to smithi12313934-3 2020-01-16T00:12:42.391 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 3 v 41 (ObjNum 110 snap 0 seq_num 110) dirty exists 2020-01-16T00:12:42.391 INFO:tasks.rados.rados.0.smithi123.stdout:714: left oid 3 (ObjNum 110 snap 0 seq_num 110) 2020-01-16T00:12:42.391 INFO:tasks.rados.rados.0.smithi123.stdout:716: expect (ObjNum 90 snap 0 seq_num 90) 2020-01-16T00:12:42.404 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:42.404 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:12:42.404 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:12:42.405 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:12:42.405 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.405 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.5.0:146 3.c e24/20 3:3d6543e6:::smithi12313934-28:head v 24'67, mlcod=24'65)", 2020-01-16T00:12:42.405 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.849191+0000", 2020-01-16T00:12:42.405 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.5555362739999996, 2020-01-16T00:12:42.405 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0088840970000000005, 2020-01-16T00:12:42.405 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:42.406 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:42.406 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:42.406 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.406 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:42.406 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.849191+0000", 2020-01-16T00:12:42.406 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:42.406 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.406 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.407 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:42.407 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.849191+0000", 2020-01-16T00:12:42.407 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:42.407 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.407 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.407 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:42.407 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.849188+0000", 2020-01-16T00:12:42.408 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0024105039999999999 2020-01-16T00:12:42.408 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.413 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.413 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:42.413 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.851599+0000", 2020-01-16T00:12:42.413 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0900999999999998e-05 2020-01-16T00:12:42.414 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.414 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.414 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:42.414 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.851660+0000", 2020-01-16T00:12:42.414 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0473e-05 2020-01-16T00:12:42.414 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.414 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.415 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:42.415 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.851670+0000", 2020-01-16T00:12:42.415 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8696999999999997e-05 2020-01-16T00:12:42.415 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.415 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.415 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.415 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.851729+0000", 2020-01-16T00:12:42.415 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3631999999999999e-05 2020-01-16T00:12:42.416 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.416 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.416 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:42.416 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.851763+0000", 2020-01-16T00:12:42.416 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0062692039999999996 2020-01-16T00:12:42.416 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.416 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.416 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:42.417 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.858032+0000", 2020-01-16T00:12:42.417 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2969999999999997e-05 2020-01-16T00:12:42.417 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.417 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.417 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:42.417 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.858075+0000", 2020-01-16T00:12:42.417 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0088840970000000005 2020-01-16T00:12:42.417 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.418 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:42.418 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.418 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.418 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.418 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:922 3.8 3:113947f2:::smithi12313934-12:head [write 443718~615603,stat] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:42.418 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.863202+0000", 2020-01-16T00:12:42.419 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.5415246910000002, 2020-01-16T00:12:42.419 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017152697000000001, 2020-01-16T00:12:42.419 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:42.419 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:42.419 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:42.420 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:42.420 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:42.420 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 922 2020-01-16T00:12:42.420 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.420 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:42.420 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.421 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:42.421 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863202+0000", 2020-01-16T00:12:42.421 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:42.421 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.421 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.421 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:42.421 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863202+0000", 2020-01-16T00:12:42.422 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:42.422 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.422 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.422 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:42.422 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863200+0000", 2020-01-16T00:12:42.422 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00038569199999999998 2020-01-16T00:12:42.422 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.422 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.423 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:42.423 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863585+0000", 2020-01-16T00:12:42.423 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5685e-05 2020-01-16T00:12:42.423 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.424 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.424 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:42.424 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863601+0000", 2020-01-16T00:12:42.424 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.4734000000000001e-05 2020-01-16T00:12:42.424 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.424 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.424 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:42.424 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863626+0000", 2020-01-16T00:12:42.425 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1037e-05 2020-01-16T00:12:42.425 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.425 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.425 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.425 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863667+0000", 2020-01-16T00:12:42.425 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016041499999999999 2020-01-16T00:12:42.425 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.425 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.426 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:42.426 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863827+0000", 2020-01-16T00:12:42.426 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0094286430000000004 2020-01-16T00:12:42.426 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.426 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.426 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.426 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.873256+0000", 2020-01-16T00:12:42.427 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2867999999999994e-05 2020-01-16T00:12:42.427 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.427 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.427 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:42.427 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.873329+0000", 2020-01-16T00:12:42.427 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021174789999999998 2020-01-16T00:12:42.427 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.428 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.428 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.428 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.875446+0000", 2020-01-16T00:12:42.428 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011244200000000001 2020-01-16T00:12:42.428 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.428 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.428 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:42.428 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.875559+0000", 2020-01-16T00:12:42.429 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021772600000000001 2020-01-16T00:12:42.429 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.429 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.429 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1,7", 2020-01-16T00:12:42.429 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.875776+0000", 2020-01-16T00:12:42.429 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0028978860000000001 2020-01-16T00:12:42.429 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.430 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.430 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:42.430 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.878674+0000", 2020-01-16T00:12:42.430 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.324e-05 2020-01-16T00:12:42.430 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.430 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.430 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:42.430 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.878687+0000", 2020-01-16T00:12:42.431 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0014896180000000001 2020-01-16T00:12:42.431 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.431 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.431 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:42.431 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880177+0000", 2020-01-16T00:12:42.431 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6881000000000001e-05 2020-01-16T00:12:42.432 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.432 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.432 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:42.432 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880224+0000", 2020-01-16T00:12:42.432 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013096700000000001 2020-01-16T00:12:42.432 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.432 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.433 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:42.433 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880355+0000", 2020-01-16T00:12:42.433 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017152697000000001 2020-01-16T00:12:42.433 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.433 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:42.433 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.433 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.434 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.434 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:923 3.8 3:113947f2:::smithi12313934-12:head [write 1564138~470341] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:42.434 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.869292+0000", 2020-01-16T00:12:42.434 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.5354343610000001, 2020-01-16T00:12:42.434 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017661620999999999, 2020-01-16T00:12:42.434 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:42.434 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:42.435 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:42.435 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:42.435 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:42.435 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 923 2020-01-16T00:12:42.435 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.435 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:42.435 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.435 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:42.436 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869292+0000", 2020-01-16T00:12:42.436 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:42.436 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.436 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.436 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:42.436 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869292+0000", 2020-01-16T00:12:42.436 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:42.436 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.437 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.437 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:42.437 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869291+0000", 2020-01-16T00:12:42.437 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00049842899999999999 2020-01-16T00:12:42.437 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.437 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.437 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:42.438 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869789+0000", 2020-01-16T00:12:42.438 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1014999999999999e-05 2020-01-16T00:12:42.438 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.438 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.438 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:42.438 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869830+0000", 2020-01-16T00:12:42.438 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7880000000000007e-06 2020-01-16T00:12:42.438 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.439 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.441 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:42.441 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869838+0000", 2020-01-16T00:12:42.441 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8346999999999997e-05 2020-01-16T00:12:42.441 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.441 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.441 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.441 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869876+0000", 2020-01-16T00:12:42.441 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9726999999999999e-05 2020-01-16T00:12:42.442 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.442 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.442 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:42.442 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869936+0000", 2020-01-16T00:12:42.442 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003434917 2020-01-16T00:12:42.442 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.442 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.442 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.443 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.873371+0000", 2020-01-16T00:12:42.443 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118808 2020-01-16T00:12:42.443 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.443 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.444 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:42.444 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.873490+0000", 2020-01-16T00:12:42.445 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036712910000000001 2020-01-16T00:12:42.445 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.445 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.445 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.445 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877161+0000", 2020-01-16T00:12:42.445 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9909999999999999e-05 2020-01-16T00:12:42.445 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.446 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.446 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:42.446 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877241+0000", 2020-01-16T00:12:42.446 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031619970000000002 2020-01-16T00:12:42.446 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.446 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.446 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.446 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880403+0000", 2020-01-16T00:12:42.447 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013071099999999999 2020-01-16T00:12:42.447 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.447 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.447 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:42.447 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880534+0000", 2020-01-16T00:12:42.447 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00032823500000000002 2020-01-16T00:12:42.447 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.447 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.448 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1,7", 2020-01-16T00:12:42.448 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880862+0000", 2020-01-16T00:12:42.448 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00513862 2020-01-16T00:12:42.448 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.448 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.448 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:42.448 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886001+0000", 2020-01-16T00:12:42.448 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8869000000000001e-05 2020-01-16T00:12:42.449 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.449 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.449 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:42.449 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886030+0000", 2020-01-16T00:12:42.449 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00081600200000000003 2020-01-16T00:12:42.450 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.450 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.450 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:42.450 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886846+0000", 2020-01-16T00:12:42.450 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4095999999999998e-05 2020-01-16T00:12:42.450 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.450 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.450 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:42.451 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886900+0000", 2020-01-16T00:12:42.451 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.448e-05 2020-01-16T00:12:42.451 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.451 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.451 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:42.451 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886954+0000", 2020-01-16T00:12:42.458 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017661620999999999 2020-01-16T00:12:42.458 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.458 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:42.458 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.458 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.458 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.459 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:924 3.8 3:113947f2:::smithi12313934-12:head [write 2633066~610652] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:42.459 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.876207+0000", 2020-01-16T00:12:42.459 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.5285202069999997, 2020-01-16T00:12:42.459 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011069638, 2020-01-16T00:12:42.459 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:42.459 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:42.460 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:42.460 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:42.460 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:42.460 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 924 2020-01-16T00:12:42.460 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.460 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:42.460 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.461 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:42.461 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876207+0000", 2020-01-16T00:12:42.461 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:42.461 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.461 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.461 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:42.461 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876207+0000", 2020-01-16T00:12:42.461 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:42.462 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.462 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.462 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:42.462 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876204+0000", 2020-01-16T00:12:42.462 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000157842 2020-01-16T00:12:42.462 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.462 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.462 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:42.463 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876362+0000", 2020-01-16T00:12:42.463 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0181e-05 2020-01-16T00:12:42.463 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.463 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.463 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:42.463 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876372+0000", 2020-01-16T00:12:42.463 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2409999999999997e-06 2020-01-16T00:12:42.464 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.464 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.464 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:42.464 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876377+0000", 2020-01-16T00:12:42.464 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00089908200000000005 2020-01-16T00:12:42.464 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.465 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.465 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.465 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877276+0000", 2020-01-16T00:12:42.465 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2391000000000006e-05 2020-01-16T00:12:42.465 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.465 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.465 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:42.465 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877338+0000", 2020-01-16T00:12:42.466 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0062765850000000003 2020-01-16T00:12:42.466 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.466 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.466 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.466 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.883615+0000", 2020-01-16T00:12:42.466 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011738700000000001 2020-01-16T00:12:42.466 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.466 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.467 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:42.467 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.883732+0000", 2020-01-16T00:12:42.467 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023677 2020-01-16T00:12:42.467 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.467 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.467 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1,7", 2020-01-16T00:12:42.467 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.883969+0000", 2020-01-16T00:12:42.468 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0028104979999999998 2020-01-16T00:12:42.468 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.468 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.468 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:42.468 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886779+0000", 2020-01-16T00:12:42.468 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00025251899999999999 2020-01-16T00:12:42.468 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.469 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.469 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:42.469 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.887032+0000", 2020-01-16T00:12:42.469 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018013099999999999 2020-01-16T00:12:42.469 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.469 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.469 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:42.469 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.887212+0000", 2020-01-16T00:12:42.470 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9646e-05 2020-01-16T00:12:42.470 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.470 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.470 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:42.470 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.887242+0000", 2020-01-16T00:12:42.470 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4551000000000002e-05 2020-01-16T00:12:42.470 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.471 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.471 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:42.471 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.887276+0000", 2020-01-16T00:12:42.471 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011069638 2020-01-16T00:12:42.471 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.471 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:42.471 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.471 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.471 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.472 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:925 3.8 3:113947f2:::smithi12313934-12:head [setxattr _header (56) in=7b,truncate 3243718] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:42.472 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.876384+0000", 2020-01-16T00:12:42.472 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.5283431600000004, 2020-01-16T00:12:42.472 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011601155, 2020-01-16T00:12:42.472 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:42.472 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:42.472 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:42.473 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:42.473 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:42.473 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 925 2020-01-16T00:12:42.473 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.473 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:42.473 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.473 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:42.473 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876384+0000", 2020-01-16T00:12:42.474 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:42.474 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.474 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.474 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:42.474 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876384+0000", 2020-01-16T00:12:42.474 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:42.475 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.475 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.475 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:42.475 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876383+0000", 2020-01-16T00:12:42.475 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.946e-06 2020-01-16T00:12:42.475 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.476 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.476 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:42.476 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876387+0000", 2020-01-16T00:12:42.476 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7179999999999998e-06 2020-01-16T00:12:42.476 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.476 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.476 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:42.476 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876391+0000", 2020-01-16T00:12:42.477 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3980000000000002e-06 2020-01-16T00:12:42.477 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.477 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.477 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:42.477 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876393+0000", 2020-01-16T00:12:42.477 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00097104099999999996 2020-01-16T00:12:42.477 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.477 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.478 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.478 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877364+0000", 2020-01-16T00:12:42.478 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6619000000000001e-05 2020-01-16T00:12:42.478 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.478 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.478 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:42.478 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877421+0000", 2020-01-16T00:12:42.478 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0078613510000000008 2020-01-16T00:12:42.479 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.479 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.479 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.479 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.885282+0000", 2020-01-16T00:12:42.479 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015531400000000001 2020-01-16T00:12:42.479 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.479 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.479 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:42.480 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.885437+0000", 2020-01-16T00:12:42.480 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019132099999999999 2020-01-16T00:12:42.480 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.480 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.480 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1,7", 2020-01-16T00:12:42.480 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.885629+0000", 2020-01-16T00:12:42.480 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0011787270000000001 2020-01-16T00:12:42.480 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.481 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.481 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:42.481 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886807+0000", 2020-01-16T00:12:42.481 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000739092 2020-01-16T00:12:42.481 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.481 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.481 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:42.482 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.887547+0000", 2020-01-16T00:12:42.482 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00037348900000000001 2020-01-16T00:12:42.482 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.482 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.482 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:42.482 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.887920+0000", 2020-01-16T00:12:42.482 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9629999999999999e-05 2020-01-16T00:12:42.482 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.483 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.483 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:42.483 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.887950+0000", 2020-01-16T00:12:42.483 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5172999999999997e-05 2020-01-16T00:12:42.483 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.483 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.483 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:42.483 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.887985+0000", 2020-01-16T00:12:42.484 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011601155 2020-01-16T00:12:42.484 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.484 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:42.484 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.484 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.484 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.484 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:926 3.8 3:113947f2:::smithi12313934-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:12:42.484 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.876396+0000", 2020-01-16T00:12:42.485 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.5283312909999998, 2020-01-16T00:12:42.485 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011788966999999999, 2020-01-16T00:12:42.485 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:42.485 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:42.485 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:42.485 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:42.485 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:42.485 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 926 2020-01-16T00:12:42.486 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.486 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:42.486 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.486 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:42.486 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876396+0000", 2020-01-16T00:12:42.486 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:42.486 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.486 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.487 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:42.487 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876396+0000", 2020-01-16T00:12:42.487 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:42.487 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.487 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.487 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:42.488 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876395+0000", 2020-01-16T00:12:42.488 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9369999999999998e-06 2020-01-16T00:12:42.488 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.488 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.488 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:42.488 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876397+0000", 2020-01-16T00:12:42.488 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2839999999999998e-06 2020-01-16T00:12:42.488 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.489 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.489 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:42.489 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876400+0000", 2020-01-16T00:12:42.489 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2060000000000001e-06 2020-01-16T00:12:42.489 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.489 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.489 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:42.489 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876403+0000", 2020-01-16T00:12:42.490 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010315179999999999 2020-01-16T00:12:42.490 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.490 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.490 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.490 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877434+0000", 2020-01-16T00:12:42.490 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0519999999999997e-05 2020-01-16T00:12:42.490 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.490 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.491 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:42.491 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877485+0000", 2020-01-16T00:12:42.491 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0084118030000000007 2020-01-16T00:12:42.491 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.491 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.491 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.491 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.885897+0000", 2020-01-16T00:12:42.491 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6395000000000001e-05 2020-01-16T00:12:42.492 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.492 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.492 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:42.492 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.885993+0000", 2020-01-16T00:12:42.492 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020290849999999999 2020-01-16T00:12:42.492 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.492 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.492 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.493 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.888022+0000", 2020-01-16T00:12:42.493 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7325999999999998e-05 2020-01-16T00:12:42.493 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.493 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.493 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:42.493 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.888099+0000", 2020-01-16T00:12:42.493 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5185999999999994e-05 2020-01-16T00:12:42.494 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.494 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.494 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:42.494 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.888185+0000", 2020-01-16T00:12:42.494 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011788966999999999 2020-01-16T00:12:42.494 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.494 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:42.494 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.495 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.495 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.495 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:928 3.b 3:df203f2a:::smithi12313934-3:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:42.495 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.876426+0000", 2020-01-16T00:12:42.495 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.5283008220000003, 2020-01-16T00:12:42.495 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0070152909999999999, 2020-01-16T00:12:42.495 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:42.495 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:42.496 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:42.496 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:42.496 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:42.496 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 928 2020-01-16T00:12:42.496 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.496 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:42.496 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.496 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:42.497 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876426+0000", 2020-01-16T00:12:42.497 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:42.497 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.497 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.497 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:42.497 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876426+0000", 2020-01-16T00:12:42.497 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:42.498 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.498 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.498 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:42.498 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876426+0000", 2020-01-16T00:12:42.498 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.18e-06 2020-01-16T00:12:42.498 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.498 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.499 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:42.499 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876429+0000", 2020-01-16T00:12:42.499 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2399999999999998e-06 2020-01-16T00:12:42.499 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.499 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.499 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:42.499 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876434+0000", 2020-01-16T00:12:42.499 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1330000000000001e-06 2020-01-16T00:12:42.500 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.500 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.500 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:42.500 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876441+0000", 2020-01-16T00:12:42.500 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0064132030000000001 2020-01-16T00:12:42.500 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.500 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.501 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.501 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.882854+0000", 2020-01-16T00:12:42.501 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016152500000000001 2020-01-16T00:12:42.501 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.501 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.501 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:42.501 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.883016+0000", 2020-01-16T00:12:42.502 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00042550800000000001 2020-01-16T00:12:42.502 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.502 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.502 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:42.502 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.883441+0000", 2020-01-16T00:12:42.502 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0070152909999999999 2020-01-16T00:12:42.502 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.502 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:42.503 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.503 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.503 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.503 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:929 3.b 3:df203f2a:::smithi12313934-3:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:42.503 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.876448+0000", 2020-01-16T00:12:42.503 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.5282785890000001, 2020-01-16T00:12:42.504 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0076937330000000003, 2020-01-16T00:12:42.504 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:42.504 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:42.504 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:42.505 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:42.505 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:42.505 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 929 2020-01-16T00:12:42.505 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.505 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:42.505 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.505 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:42.506 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876448+0000", 2020-01-16T00:12:42.506 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:42.506 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.506 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.506 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:42.506 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876448+0000", 2020-01-16T00:12:42.506 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:42.507 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.507 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.507 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:42.507 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876448+0000", 2020-01-16T00:12:42.507 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9470000000000002e-06 2020-01-16T00:12:42.507 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.507 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.507 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:42.508 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876453+0000", 2020-01-16T00:12:42.508 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9840000000000001e-06 2020-01-16T00:12:42.508 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.508 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.508 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:42.508 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876458+0000", 2020-01-16T00:12:42.508 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1879999999999998e-06 2020-01-16T00:12:42.508 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.509 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.509 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:42.509 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876461+0000", 2020-01-16T00:12:42.509 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0070321639999999996 2020-01-16T00:12:42.509 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.509 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.509 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.510 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.883493+0000", 2020-01-16T00:12:42.510 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021568600000000001 2020-01-16T00:12:42.510 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.510 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.510 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:42.510 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.883709+0000", 2020-01-16T00:12:42.510 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043327600000000002 2020-01-16T00:12:42.510 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.511 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.511 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:42.511 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.884142+0000", 2020-01-16T00:12:42.511 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0076937330000000003 2020-01-16T00:12:42.511 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.511 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:42.511 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.512 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.512 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.512 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:935 3.2 3:4f8fe446:::smithi12313934-48:head [write 3060489~200036] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:42.512 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.895775+0000", 2020-01-16T00:12:42.512 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.5089521750000001, 2020-01-16T00:12:42.512 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012912735, 2020-01-16T00:12:42.512 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:42.512 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:42.513 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:42.513 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:42.513 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:42.513 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 935 2020-01-16T00:12:42.513 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.513 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:42.513 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.513 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:42.514 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895775+0000", 2020-01-16T00:12:42.514 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:42.514 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.514 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.514 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:42.514 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895775+0000", 2020-01-16T00:12:42.514 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:42.515 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.515 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.515 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:42.515 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895773+0000", 2020-01-16T00:12:42.515 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3350000000000007e-05 2020-01-16T00:12:42.515 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.515 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.515 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:42.516 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895856+0000", 2020-01-16T00:12:42.516 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.418e-05 2020-01-16T00:12:42.516 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.516 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.516 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:42.516 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895871+0000", 2020-01-16T00:12:42.516 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2750000000000001e-06 2020-01-16T00:12:42.517 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.517 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.517 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:42.517 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895877+0000", 2020-01-16T00:12:42.517 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7113000000000002e-05 2020-01-16T00:12:42.517 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.517 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.517 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.518 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895914+0000", 2020-01-16T00:12:42.518 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00064030700000000001 2020-01-16T00:12:42.518 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.518 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.518 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:42.518 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.896554+0000", 2020-01-16T00:12:42.518 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0098022809999999995 2020-01-16T00:12:42.518 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.519 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.519 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.519 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.906357+0000", 2020-01-16T00:12:42.519 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112898 2020-01-16T00:12:42.519 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.519 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.519 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:42.520 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.906469+0000", 2020-01-16T00:12:42.520 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022189399999999999 2020-01-16T00:12:42.520 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.520 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.520 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:12:42.520 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.906691+0000", 2020-01-16T00:12:42.520 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018153869999999999 2020-01-16T00:12:42.520 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.521 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.521 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:42.521 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908507+0000", 2020-01-16T00:12:42.521 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6517999999999999e-05 2020-01-16T00:12:42.521 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.521 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.521 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:42.521 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908533+0000", 2020-01-16T00:12:42.522 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1899e-05 2020-01-16T00:12:42.522 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.522 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.522 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:42.522 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908575+0000", 2020-01-16T00:12:42.522 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2222999999999998e-05 2020-01-16T00:12:42.522 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.523 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.523 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:42.523 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908627+0000", 2020-01-16T00:12:42.523 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0019999999999998e-05 2020-01-16T00:12:42.523 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.523 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.523 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:42.523 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908687+0000", 2020-01-16T00:12:42.524 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012912735 2020-01-16T00:12:42.524 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.524 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:42.524 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.524 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.524 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.524 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:936 3.2 3:4f8fe446:::smithi12313934-48:head [setxattr _header (56) in=7b,truncate 3260525] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:42.525 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.895886+0000", 2020-01-16T00:12:42.525 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.5088404710000001, 2020-01-16T00:12:42.525 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014382986, 2020-01-16T00:12:42.525 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:42.525 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:42.525 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:42.525 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:42.525 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:42.526 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 936 2020-01-16T00:12:42.526 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.526 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:42.526 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.526 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:42.526 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895886+0000", 2020-01-16T00:12:42.526 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:42.526 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.527 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.527 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:42.527 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895886+0000", 2020-01-16T00:12:42.527 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:12:42.527 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.527 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.527 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:42.528 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895885+0000", 2020-01-16T00:12:42.528 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0980000000000001e-06 2020-01-16T00:12:42.528 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.528 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.528 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:42.528 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895891+0000", 2020-01-16T00:12:42.528 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.959e-06 2020-01-16T00:12:42.528 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.529 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.529 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:42.529 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895897+0000", 2020-01-16T00:12:42.529 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9920000000000001e-06 2020-01-16T00:12:42.529 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.529 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.529 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:42.530 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895901+0000", 2020-01-16T00:12:42.530 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00068615199999999999 2020-01-16T00:12:42.530 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.530 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.530 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.530 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.896587+0000", 2020-01-16T00:12:42.530 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2934e-05 2020-01-16T00:12:42.530 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.531 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.531 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:42.531 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.896620+0000", 2020-01-16T00:12:42.531 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010751811 2020-01-16T00:12:42.531 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.531 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.531 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.531 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.907371+0000", 2020-01-16T00:12:42.532 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011422899999999999 2020-01-16T00:12:42.532 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.532 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.532 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:42.532 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.907486+0000", 2020-01-16T00:12:42.532 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000185172 2020-01-16T00:12:42.532 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.533 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.533 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:12:42.533 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.907671+0000", 2020-01-16T00:12:42.533 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001265164 2020-01-16T00:12:42.533 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.533 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.533 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:42.533 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908936+0000", 2020-01-16T00:12:42.534 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000297247 2020-01-16T00:12:42.534 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.534 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.534 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:42.534 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.909233+0000", 2020-01-16T00:12:42.534 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000907528 2020-01-16T00:12:42.534 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.534 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.535 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:42.535 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910141+0000", 2020-01-16T00:12:42.535 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3384000000000002e-05 2020-01-16T00:12:42.535 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.535 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.535 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:42.535 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910204+0000", 2020-01-16T00:12:42.536 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5229999999999997e-05 2020-01-16T00:12:42.536 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.536 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.536 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:42.536 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910269+0000", 2020-01-16T00:12:42.536 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014382986 2020-01-16T00:12:42.536 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.536 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:42.537 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.537 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.537 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.537 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:937 3.2 3:4f8fe446:::smithi12313934-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:12:42.537 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.895918+0000", 2020-01-16T00:12:42.537 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.5088088429999997, 2020-01-16T00:12:42.537 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014610741999999999, 2020-01-16T00:12:42.538 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:42.538 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:42.538 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:42.538 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:42.538 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:42.538 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 937 2020-01-16T00:12:42.538 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.538 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:42.539 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.539 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:42.539 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895918+0000", 2020-01-16T00:12:42.539 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:42.539 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.539 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.539 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:42.539 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895918+0000", 2020-01-16T00:12:42.540 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:12:42.540 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.540 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.540 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:42.540 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895917+0000", 2020-01-16T00:12:42.540 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8840000000000004e-06 2020-01-16T00:12:42.540 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.540 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.541 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:42.541 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895921+0000", 2020-01-16T00:12:42.541 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1579999999999999e-06 2020-01-16T00:12:42.541 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.541 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.541 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:42.541 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895928+0000", 2020-01-16T00:12:42.542 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4139999999999996e-06 2020-01-16T00:12:42.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.542 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.542 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:42.542 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895933+0000", 2020-01-16T00:12:42.542 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000702218 2020-01-16T00:12:42.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.542 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.543 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.543 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.896635+0000", 2020-01-16T00:12:42.543 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6421e-05 2020-01-16T00:12:42.543 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.543 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.543 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:42.543 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.896661+0000", 2020-01-16T00:12:42.543 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011578879 2020-01-16T00:12:42.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.544 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.544 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.544 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908240+0000", 2020-01-16T00:12:42.544 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5567000000000003e-05 2020-01-16T00:12:42.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.544 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.545 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:42.545 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908316+0000", 2020-01-16T00:12:42.545 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0019937560000000002 2020-01-16T00:12:42.545 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.545 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.545 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.545 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910310+0000", 2020-01-16T00:12:42.545 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000109566 2020-01-16T00:12:42.546 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.546 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.546 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:42.546 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910419+0000", 2020-01-16T00:12:42.546 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000109629 2020-01-16T00:12:42.546 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.546 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.546 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:42.547 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910529+0000", 2020-01-16T00:12:42.547 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014610741999999999 2020-01-16T00:12:42.547 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.547 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:42.547 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.547 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.547 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.548 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.5.0:158 3.c e24/20 3:3877922e:::smithi12313934-31:head v 24'72, mlcod=24'70)", 2020-01-16T00:12:42.548 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.931147+0000", 2020-01-16T00:12:42.548 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.473579645, 2020-01-16T00:12:42.548 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0060017630000000002, 2020-01-16T00:12:42.548 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:42.548 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:42.548 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:42.548 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.549 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:42.549 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931147+0000", 2020-01-16T00:12:42.549 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:42.549 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.549 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.549 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:42.549 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931147+0000", 2020-01-16T00:12:42.550 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:42.550 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.550 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.550 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:42.550 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931145+0000", 2020-01-16T00:12:42.550 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0015459009999999999 2020-01-16T00:12:42.550 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.550 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.551 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:42.551 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932691+0000", 2020-01-16T00:12:42.551 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4831000000000001e-05 2020-01-16T00:12:42.551 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.551 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.551 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:42.551 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932705+0000", 2020-01-16T00:12:42.552 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3569999999999997e-06 2020-01-16T00:12:42.552 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.552 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.552 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:42.552 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932711+0000", 2020-01-16T00:12:42.552 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6208000000000003e-05 2020-01-16T00:12:42.552 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.552 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.553 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.553 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932747+0000", 2020-01-16T00:12:42.553 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7938999999999999e-05 2020-01-16T00:12:42.553 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.553 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.553 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:42.553 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932765+0000", 2020-01-16T00:12:42.553 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0043314 2020-01-16T00:12:42.554 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.554 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.554 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:42.554 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.937096+0000", 2020-01-16T00:12:42.554 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2614999999999997e-05 2020-01-16T00:12:42.554 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.554 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.555 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:42.555 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.937149+0000", 2020-01-16T00:12:42.555 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0060017630000000002 2020-01-16T00:12:42.555 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.555 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:42.555 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.555 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.555 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.556 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:970 3.8 3:1e8f7c6f:::smithi12313934-6:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:42.556 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.159615+0000", 2020-01-16T00:12:42.556 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.2451123099999997, 2020-01-16T00:12:42.556 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0057821019999999999, 2020-01-16T00:12:42.556 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:42.556 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:42.556 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:42.556 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:42.557 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:42.557 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 970 2020-01-16T00:12:42.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.557 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:42.557 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.557 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:42.557 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159615+0000", 2020-01-16T00:12:42.557 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:42.558 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.558 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.558 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:42.558 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159615+0000", 2020-01-16T00:12:42.558 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999914 2020-01-16T00:12:42.558 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.558 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.559 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:42.559 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159606+0000", 2020-01-16T00:12:42.559 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6858999999999999e-05 2020-01-16T00:12:42.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.559 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.559 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:42.559 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159623+0000", 2020-01-16T00:12:42.559 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6963999999999999e-05 2020-01-16T00:12:42.560 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.560 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.560 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:42.560 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159640+0000", 2020-01-16T00:12:42.560 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0510000000000002e-06 2020-01-16T00:12:42.560 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.560 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.560 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:42.561 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159646+0000", 2020-01-16T00:12:42.561 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000101834 2020-01-16T00:12:42.561 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.561 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.561 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.561 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159748+0000", 2020-01-16T00:12:42.561 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00037551000000000001 2020-01-16T00:12:42.562 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.562 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.562 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:42.562 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.160123+0000", 2020-01-16T00:12:42.562 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0046367609999999997 2020-01-16T00:12:42.562 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.562 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.562 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.563 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.164760+0000", 2020-01-16T00:12:42.563 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2069000000000004e-05 2020-01-16T00:12:42.563 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.563 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.563 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:42.563 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.164832+0000", 2020-01-16T00:12:42.563 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00033586500000000002 2020-01-16T00:12:42.563 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.564 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.564 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.564 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.165168+0000", 2020-01-16T00:12:42.564 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0454000000000002e-05 2020-01-16T00:12:42.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.564 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.564 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:42.565 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.165218+0000", 2020-01-16T00:12:42.565 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000178178 2020-01-16T00:12:42.565 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.565 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.565 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:42.565 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.165397+0000", 2020-01-16T00:12:42.565 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0057821019999999999 2020-01-16T00:12:42.565 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.566 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:42.566 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.566 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.566 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.566 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:971 3.8 3:1e8f7c6f:::smithi12313934-6:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:42.566 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.159657+0000", 2020-01-16T00:12:42.566 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.2450696739999998, 2020-01-16T00:12:42.567 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0059336909999999996, 2020-01-16T00:12:42.567 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:42.567 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:42.567 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:42.567 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:42.567 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:42.567 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 971 2020-01-16T00:12:42.567 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.568 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:42.568 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.568 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:42.568 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159657+0000", 2020-01-16T00:12:42.568 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:42.568 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.568 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.568 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:42.569 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159657+0000", 2020-01-16T00:12:42.569 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:42.569 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.569 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.569 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:42.569 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159657+0000", 2020-01-16T00:12:42.569 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9390000000000002e-06 2020-01-16T00:12:42.570 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.570 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.570 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:42.570 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159660+0000", 2020-01-16T00:12:42.570 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7389999999999996e-06 2020-01-16T00:12:42.570 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.570 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.570 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:42.571 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159665+0000", 2020-01-16T00:12:42.571 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2473999999999999e-05 2020-01-16T00:12:42.571 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.571 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.571 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:42.571 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159678+0000", 2020-01-16T00:12:42.571 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00054627300000000001 2020-01-16T00:12:42.571 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.572 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.572 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.572 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.160224+0000", 2020-01-16T00:12:42.572 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8795000000000002e-05 2020-01-16T00:12:42.572 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.572 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.572 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:42.573 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.160283+0000", 2020-01-16T00:12:42.573 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0045895440000000001 2020-01-16T00:12:42.573 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.573 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.573 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.573 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.164872+0000", 2020-01-16T00:12:42.573 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8940999999999997e-05 2020-01-16T00:12:42.574 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.574 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.574 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:42.574 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.164931+0000", 2020-01-16T00:12:42.574 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00048151800000000001 2020-01-16T00:12:42.574 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.574 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.574 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.575 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.165413+0000", 2020-01-16T00:12:42.575 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.117e-05 2020-01-16T00:12:42.575 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.575 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.575 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:42.575 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.165464+0000", 2020-01-16T00:12:42.575 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012686699999999999 2020-01-16T00:12:42.575 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.576 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.576 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:42.576 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.165591+0000", 2020-01-16T00:12:42.576 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0059336909999999996 2020-01-16T00:12:42.576 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.576 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:42.576 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.577 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.577 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.577 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:972 3.8 3:1e8f7c6f:::smithi12313934-6:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:42.577 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.159726+0000", 2020-01-16T00:12:42.577 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.2450004530000003, 2020-01-16T00:12:42.577 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0061844259999999998, 2020-01-16T00:12:42.577 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:42.577 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:42.578 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:42.578 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:42.578 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:42.578 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 972 2020-01-16T00:12:42.578 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.578 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:42.578 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.578 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:42.579 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159726+0000", 2020-01-16T00:12:42.579 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:42.579 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.579 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.579 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:42.579 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159726+0000", 2020-01-16T00:12:42.579 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:42.580 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.580 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.580 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:42.580 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159726+0000", 2020-01-16T00:12:42.580 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0910000000000003e-06 2020-01-16T00:12:42.580 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.580 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.580 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:42.581 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159730+0000", 2020-01-16T00:12:42.581 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.6910000000000003e-06 2020-01-16T00:12:42.581 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.581 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.581 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:42.581 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159738+0000", 2020-01-16T00:12:42.581 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1479999999999999e-06 2020-01-16T00:12:42.582 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.582 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.582 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:42.582 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159743+0000", 2020-01-16T00:12:42.582 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00057154700000000005 2020-01-16T00:12:42.582 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.582 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.582 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.583 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.160314+0000", 2020-01-16T00:12:42.583 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8436999999999997e-05 2020-01-16T00:12:42.583 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.583 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.583 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:42.583 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.160363+0000", 2020-01-16T00:12:42.583 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0045992810000000002 2020-01-16T00:12:42.583 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.584 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.584 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.584 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.164962+0000", 2020-01-16T00:12:42.584 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8581000000000001e-05 2020-01-16T00:12:42.584 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.584 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.584 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:42.584 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.165010+0000", 2020-01-16T00:12:42.585 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00060742200000000004 2020-01-16T00:12:42.585 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.585 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.585 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.585 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.165618+0000", 2020-01-16T00:12:42.585 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6708999999999999e-05 2020-01-16T00:12:42.585 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.586 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.586 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:42.586 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.165715+0000", 2020-01-16T00:12:42.586 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019620999999999999 2020-01-16T00:12:42.586 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.586 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.586 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:42.586 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.165911+0000", 2020-01-16T00:12:42.587 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0061844259999999998 2020-01-16T00:12:42.587 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.587 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:42.587 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.587 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.587 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.587 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1149 3.2 3:4f8fe446:::smithi12313934-48:head [read 0~3260525,checksum] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:12:42.587 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:41.603914+0000", 2020-01-16T00:12:42.588 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.80081318999999995, 2020-01-16T00:12:42.588 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0069101639999999999, 2020-01-16T00:12:42.588 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:42.588 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:42.588 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:42.588 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:42.588 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:42.589 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1149 2020-01-16T00:12:42.589 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.589 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:42.589 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.589 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:42.589 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.603914+0000", 2020-01-16T00:12:42.589 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:42.589 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.590 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.590 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:42.590 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.603914+0000", 2020-01-16T00:12:42.590 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:42.590 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.590 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.590 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:42.591 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.603913+0000", 2020-01-16T00:12:42.591 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9479999999999999e-06 2020-01-16T00:12:42.591 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.591 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.591 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:42.591 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.603918+0000", 2020-01-16T00:12:42.591 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4470000000000003e-06 2020-01-16T00:12:42.591 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.592 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.592 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:42.592 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.603924+0000", 2020-01-16T00:12:42.592 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8700000000000002e-06 2020-01-16T00:12:42.592 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.592 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.592 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:42.592 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.603928+0000", 2020-01-16T00:12:42.593 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0050935329999999999 2020-01-16T00:12:42.593 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.593 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.593 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.593 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.609022+0000", 2020-01-16T00:12:42.593 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012769600000000001 2020-01-16T00:12:42.593 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.594 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.594 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:42.594 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.609149+0000", 2020-01-16T00:12:42.594 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0016743299999999999 2020-01-16T00:12:42.594 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.594 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.594 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:42.594 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.610824+0000", 2020-01-16T00:12:42.595 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0069101639999999999 2020-01-16T00:12:42.595 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.595 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:42.595 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.595 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.595 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.595 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1151 3.7 3:efab040c:::smithi12313934-30:head [read 0~3411788,checksum] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:12:42.595 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:41.603971+0000", 2020-01-16T00:12:42.596 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.800756307, 2020-01-16T00:12:42.596 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0082599330000000006, 2020-01-16T00:12:42.596 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:42.596 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:42.596 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:42.596 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:42.596 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:42.597 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1151 2020-01-16T00:12:42.597 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.597 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:42.597 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.597 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:42.597 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.603971+0000", 2020-01-16T00:12:42.597 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:42.597 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.598 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.598 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:42.598 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.603971+0000", 2020-01-16T00:12:42.598 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:12:42.598 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.598 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.598 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:42.599 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.603970+0000", 2020-01-16T00:12:42.599 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7449999999999995e-06 2020-01-16T00:12:42.599 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.599 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.599 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:42.599 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.603977+0000", 2020-01-16T00:12:42.599 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7689999999999999e-06 2020-01-16T00:12:42.600 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.600 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.600 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:42.600 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.603984+0000", 2020-01-16T00:12:42.600 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7489999999999999e-06 2020-01-16T00:12:42.600 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.600 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.601 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:42.601 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.603987+0000", 2020-01-16T00:12:42.601 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0050570839999999999 2020-01-16T00:12:42.601 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.601 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.601 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.601 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.609044+0000", 2020-01-16T00:12:42.602 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000142452 2020-01-16T00:12:42.602 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.602 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.602 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:42.602 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.609186+0000", 2020-01-16T00:12:42.602 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030440089999999999 2020-01-16T00:12:42.602 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.602 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.603 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:42.603 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.612230+0000", 2020-01-16T00:12:42.603 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0082599330000000006 2020-01-16T00:12:42.603 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.603 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:42.603 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.603 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.604 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.604 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1152 3.7 3:efab040c:::smithi12313934-30:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:12:42.604 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:41.603990+0000", 2020-01-16T00:12:42.604 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.80073667199999998, 2020-01-16T00:12:42.604 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093224899999999992, 2020-01-16T00:12:42.604 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:42.604 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:42.605 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:42.605 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:42.605 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:42.605 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1152 2020-01-16T00:12:42.605 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.605 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:42.605 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.606 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:42.606 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.603990+0000", 2020-01-16T00:12:42.606 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:42.606 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.606 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.606 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:42.606 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.603990+0000", 2020-01-16T00:12:42.607 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:42.607 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.607 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.607 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:42.607 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.603990+0000", 2020-01-16T00:12:42.607 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4040000000000003e-06 2020-01-16T00:12:42.607 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.607 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.608 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:42.608 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.603995+0000", 2020-01-16T00:12:42.608 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8899999999999999e-06 2020-01-16T00:12:42.608 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.608 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.608 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:42.608 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.603998+0000", 2020-01-16T00:12:42.609 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0849999999999996e-06 2020-01-16T00:12:42.609 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.609 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.609 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:42.609 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.604003+0000", 2020-01-16T00:12:42.609 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0082858129999999995 2020-01-16T00:12:42.609 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.610 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.610 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.610 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.612289+0000", 2020-01-16T00:12:42.610 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016920300000000001 2020-01-16T00:12:42.610 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.610 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.610 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:42.611 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.612458+0000", 2020-01-16T00:12:42.611 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00085449099999999997 2020-01-16T00:12:42.611 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.611 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.611 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:42.611 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.613313+0000", 2020-01-16T00:12:42.611 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093224899999999992 2020-01-16T00:12:42.611 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.612 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:42.612 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.612 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.612 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.612 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1177 3.b 3:df203f2a:::smithi12313934-3:head [write 648770~706332,stat] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:12:42.612 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:42.089533+0000", 2020-01-16T00:12:42.612 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.31519394499999998, 2020-01-16T00:12:42.613 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0068691790000000004, 2020-01-16T00:12:42.613 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:42.613 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:42.613 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:42.613 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:42.613 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:42.613 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1177 2020-01-16T00:12:42.614 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.614 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:42.614 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.614 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:42.614 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.089533+0000", 2020-01-16T00:12:42.614 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:42.614 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.615 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.615 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:42.615 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.089533+0000", 2020-01-16T00:12:42.615 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999938 2020-01-16T00:12:42.615 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.615 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.615 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:42.615 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.089527+0000", 2020-01-16T00:12:42.616 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00039742100000000002 2020-01-16T00:12:42.616 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.616 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.616 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:42.616 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.089924+0000", 2020-01-16T00:12:42.616 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1369999999999999e-05 2020-01-16T00:12:42.616 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.617 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.617 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:42.617 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.089966+0000", 2020-01-16T00:12:42.617 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5207e-05 2020-01-16T00:12:42.617 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.617 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.617 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:42.618 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.089981+0000", 2020-01-16T00:12:42.618 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000104066 2020-01-16T00:12:42.618 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.618 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.618 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.618 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.090085+0000", 2020-01-16T00:12:42.618 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000234594 2020-01-16T00:12:42.618 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.619 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.619 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:42.619 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.090320+0000", 2020-01-16T00:12:42.619 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034746400000000002 2020-01-16T00:12:42.619 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.619 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.619 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,4", 2020-01-16T00:12:42.620 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.090667+0000", 2020-01-16T00:12:42.620 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0038092809999999999 2020-01-16T00:12:42.620 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.620 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.620 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:42.620 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.094476+0000", 2020-01-16T00:12:42.620 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00081967599999999998 2020-01-16T00:12:42.620 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.621 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.621 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:42.621 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.095296+0000", 2020-01-16T00:12:42.621 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00099452900000000007 2020-01-16T00:12:42.621 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.621 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.621 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:42.622 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.096291+0000", 2020-01-16T00:12:42.622 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8214999999999997e-05 2020-01-16T00:12:42.622 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.622 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.622 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:42.622 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.096349+0000", 2020-01-16T00:12:42.622 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3359000000000002e-05 2020-01-16T00:12:42.623 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.623 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.623 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:42.623 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.096402+0000", 2020-01-16T00:12:42.623 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0068691790000000004 2020-01-16T00:12:42.623 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.623 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:42.623 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.624 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.624 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.624 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1181 3.b 3:df203f2a:::smithi12313934-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-01-16T00:12:42.624 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:42.103293+0000", 2020-01-16T00:12:42.624 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.30143340699999999, 2020-01-16T00:12:42.624 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0058994340000000003, 2020-01-16T00:12:42.624 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:42.625 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:42.625 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:42.625 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:42.625 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:42.625 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1181 2020-01-16T00:12:42.625 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.625 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:42.626 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.626 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:42.626 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.103293+0000", 2020-01-16T00:12:42.626 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:42.626 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.626 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.626 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:42.627 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.103293+0000", 2020-01-16T00:12:42.627 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:42.627 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.627 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.627 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:42.627 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.103293+0000", 2020-01-16T00:12:42.627 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8560000000000002e-06 2020-01-16T00:12:42.628 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.628 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.628 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:42.628 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.103303+0000", 2020-01-16T00:12:42.628 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7009999999999999e-06 2020-01-16T00:12:42.628 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.628 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.628 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:42.629 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.103309+0000", 2020-01-16T00:12:42.629 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0869e-05 2020-01-16T00:12:42.629 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.629 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.629 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:42.629 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.103320+0000", 2020-01-16T00:12:42.629 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0027600799999999998 2020-01-16T00:12:42.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.630 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.630 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.630 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.106080+0000", 2020-01-16T00:12:42.630 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013417 2020-01-16T00:12:42.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.630 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.631 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:42.631 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.106214+0000", 2020-01-16T00:12:42.631 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002706122 2020-01-16T00:12:42.631 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.631 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.631 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:42.631 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.108921+0000", 2020-01-16T00:12:42.631 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014078100000000001 2020-01-16T00:12:42.632 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.632 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.632 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:42.632 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.109061+0000", 2020-01-16T00:12:42.632 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000131488 2020-01-16T00:12:42.632 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:42.632 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:42.633 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:42.633 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.109193+0000", 2020-01-16T00:12:42.633 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0058994340000000003 2020-01-16T00:12:42.633 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.633 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:42.633 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.633 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:42.634 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:42.634 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:42.637 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-01-16T00:12:42.637 INFO:tasks.rados.rados.0.smithi123.stdout:717: finishing write tid 1 to smithi12313934-25 2020-01-16T00:12:42.638 INFO:tasks.rados.rados.0.smithi123.stdout:717: finishing write tid 2 to smithi12313934-25 2020-01-16T00:12:42.638 INFO:tasks.rados.rados.0.smithi123.stdout:717: finishing write tid 3 to smithi12313934-25 2020-01-16T00:12:42.638 INFO:tasks.rados.rados.0.smithi123.stdout:717: finishing write tid 4 to smithi12313934-25 2020-01-16T00:12:42.638 INFO:tasks.rados.rados.0.smithi123.stdout:717: finishing write tid 6 to smithi12313934-25 2020-01-16T00:12:42.638 INFO:tasks.rados.rados.0.smithi123.stdout:717: finishing write tid 7 to smithi12313934-25 2020-01-16T00:12:42.638 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 25 v 95 (ObjNum 112 snap 0 seq_num 112) dirty exists 2020-01-16T00:12:42.638 INFO:tasks.rados.rados.0.smithi123.stdout:717: left oid 25 (ObjNum 112 snap 0 seq_num 112) 2020-01-16T00:12:42.639 INFO:tasks.rados.rados.0.smithi123.stdout:718: finishing write tid 1 to smithi12313934-21 2020-01-16T00:12:42.639 INFO:tasks.rados.rados.0.smithi123.stdout:718: finishing write tid 2 to smithi12313934-21 2020-01-16T00:12:42.639 INFO:tasks.rados.rados.0.smithi123.stdout:718: finishing write tid 3 to smithi12313934-21 2020-01-16T00:12:42.639 INFO:tasks.rados.rados.0.smithi123.stdout:718: finishing write tid 5 to smithi12313934-21 2020-01-16T00:12:42.639 INFO:tasks.rados.rados.0.smithi123.stdout:718: finishing write tid 6 to smithi12313934-21 2020-01-16T00:12:42.639 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 21 v 33 (ObjNum 113 snap 0 seq_num 113) dirty exists 2020-01-16T00:12:42.639 INFO:tasks.rados.rados.0.smithi123.stdout:718: left oid 21 (ObjNum 113 snap 0 seq_num 113) 2020-01-16T00:12:42.640 INFO:tasks.rados.rados.0.smithi123.stdout:712: done (12 left) 2020-01-16T00:12:42.640 INFO:tasks.rados.rados.0.smithi123.stdout:714: done (11 left) 2020-01-16T00:12:42.640 INFO:tasks.rados.rados.0.smithi123.stdout:716: done (10 left) 2020-01-16T00:12:42.640 INFO:tasks.rados.rados.0.smithi123.stdout:717: done (9 left) 2020-01-16T00:12:42.640 INFO:tasks.rados.rados.0.smithi123.stdout:718: done (8 left) 2020-01-16T00:12:42.640 INFO:tasks.rados.rados.0.smithi123.stdout:726: done (7 left) 2020-01-16T00:12:42.640 INFO:tasks.rados.rados.0.smithi123.stdout:727: read oid 15 snap -1 2020-01-16T00:12:42.641 INFO:tasks.rados.rados.0.smithi123.stdout:727: expect (ObjNum 59 snap 0 seq_num 59) 2020-01-16T00:12:42.641 INFO:tasks.rados.rados.0.smithi123.stdout:728: read oid 17 snap -1 2020-01-16T00:12:42.641 INFO:tasks.rados.rados.0.smithi123.stdout:728: expect deleted 2020-01-16T00:12:42.641 INFO:tasks.rados.rados.0.smithi123.stdout:719: finishing write tid 1 to smithi12313934-22 2020-01-16T00:12:42.641 INFO:tasks.rados.rados.0.smithi123.stdout:729: read oid 1 snap -1 2020-01-16T00:12:42.641 INFO:tasks.rados.rados.0.smithi123.stdout:729: expect (ObjNum 83 snap 0 seq_num 83) 2020-01-16T00:12:42.641 INFO:tasks.rados.rados.0.smithi123.stdout:730: write oid 45 current snap is 0 2020-01-16T00:12:42.641 INFO:tasks.rados.rados.0.smithi123.stdout:730: seq_num 119 ranges {400033=699926,1614063=385970} 2020-01-16T00:12:42.642 INFO:tasks.rados.rados.0.smithi123.stdout:730: writing smithi12313934-45 from 400033 to 1099959 tid 1 2020-01-16T00:12:42.642 INFO:tasks.rados.rados.0.smithi123.stdout:730: writing smithi12313934-45 from 1614063 to 2000033 tid 2 2020-01-16T00:12:42.642 INFO:tasks.rados.rados.0.smithi123.stdout:731: read oid 19 snap -1 2020-01-16T00:12:42.642 INFO:tasks.rados.rados.0.smithi123.stdout:731: expect (ObjNum 108 snap 0 seq_num 108) 2020-01-16T00:12:42.642 INFO:tasks.rados.rados.0.smithi123.stdout:732: write (excl) oid 31 current snap is 0 2020-01-16T00:12:42.642 INFO:tasks.rados.rados.0.smithi123.stdout:732: seq_num 120 ranges {416840=775175,1672545=344295} 2020-01-16T00:12:42.642 INFO:tasks.rados.rados.0.smithi123.stdout:732: writing smithi12313934-31 from 416840 to 1192015 tid 1 2020-01-16T00:12:42.643 INFO:tasks.rados.rados.0.smithi123.stdout:732: writing smithi12313934-31 from 1672545 to 2016840 tid 2 2020-01-16T00:12:42.643 INFO:tasks.rados.rados.0.smithi123.stdout:733: read oid 29 snap -1 2020-01-16T00:12:42.643 INFO:tasks.rados.rados.0.smithi123.stdout:733: expect (ObjNum 28 snap 0 seq_num 28) 2020-01-16T00:12:42.643 INFO:tasks.rados.rados.0.smithi123.stdout:734: read oid 6 snap -1 2020-01-16T00:12:42.643 INFO:tasks.rados.rados.0.smithi123.stdout:734: expect (ObjNum 5 snap 0 seq_num 5) 2020-01-16T00:12:42.643 INFO:tasks.rados.rados.0.smithi123.stdout:735: read oid 2 snap -1 2020-01-16T00:12:42.643 INFO:tasks.rados.rados.0.smithi123.stdout:735: expect (ObjNum 63 snap 0 seq_num 63) 2020-01-16T00:12:42.644 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:42.644 INFO:tasks.rados.rados.0.smithi123.stdout:719: finishing write tid 2 to smithi12313934-22 2020-01-16T00:12:42.644 INFO:tasks.rados.rados.0.smithi123.stdout:719: finishing write tid 3 to smithi12313934-22 2020-01-16T00:12:42.644 INFO:tasks.rados.rados.0.smithi123.stdout:719: finishing write tid 4 to smithi12313934-22 2020-01-16T00:12:42.644 INFO:tasks.rados.rados.0.smithi123.stdout:719: finishing write tid 6 to smithi12313934-22 2020-01-16T00:12:42.644 INFO:tasks.rados.rados.0.smithi123.stdout:719: finishing write tid 7 to smithi12313934-22 2020-01-16T00:12:42.644 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 22 v 19 (ObjNum 114 snap 0 seq_num 114) dirty exists 2020-01-16T00:12:42.644 INFO:tasks.rados.rados.0.smithi123.stdout:719: left oid 22 (ObjNum 114 snap 0 seq_num 114) 2020-01-16T00:12:42.645 INFO:tasks.rados.rados.0.smithi123.stdout:720: finishing write tid 1 to smithi12313934-20 2020-01-16T00:12:42.645 INFO:tasks.rados.rados.0.smithi123.stdout:720: finishing write tid 2 to smithi12313934-20 2020-01-16T00:12:42.645 INFO:tasks.rados.rados.0.smithi123.stdout:720: finishing write tid 3 to smithi12313934-20 2020-01-16T00:12:42.645 INFO:tasks.rados.rados.0.smithi123.stdout:720: finishing write tid 5 to smithi12313934-20 2020-01-16T00:12:42.645 INFO:tasks.rados.rados.0.smithi123.stdout:720: finishing write tid 6 to smithi12313934-20 2020-01-16T00:12:42.645 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 20 v 99 (ObjNum 115 snap 0 seq_num 115) dirty exists 2020-01-16T00:12:42.645 INFO:tasks.rados.rados.0.smithi123.stdout:720: left oid 20 (ObjNum 115 snap 0 seq_num 115) 2020-01-16T00:12:42.646 INFO:tasks.rados.rados.0.smithi123.stdout:721: finishing write tid 1 to smithi12313934-10 2020-01-16T00:12:42.646 INFO:tasks.rados.rados.0.smithi123.stdout:722: expect (ObjNum 107 snap 0 seq_num 107) 2020-01-16T00:12:42.646 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:42.637+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:42.736 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:42.735+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:12:42.784 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:42.784 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:12:42.784 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:12:42.784 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:42.793 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-01-16T00:12:42.830 INFO:tasks.rados.rados.0.smithi123.stdout:721: finishing write tid 2 to smithi12313934-10 2020-01-16T00:12:42.831 INFO:tasks.rados.rados.0.smithi123.stdout:721: finishing write tid 3 to smithi12313934-10 2020-01-16T00:12:42.831 INFO:tasks.rados.rados.0.smithi123.stdout:721: finishing write tid 5 to smithi12313934-10 2020-01-16T00:12:42.831 INFO:tasks.rados.rados.0.smithi123.stdout:721: finishing write tid 6 to smithi12313934-10 2020-01-16T00:12:42.831 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 10 v 55 (ObjNum 116 snap 0 seq_num 116) dirty exists 2020-01-16T00:12:42.831 INFO:tasks.rados.rados.0.smithi123.stdout:721: left oid 10 (ObjNum 116 snap 0 seq_num 116) 2020-01-16T00:12:42.831 INFO:tasks.rados.rados.0.smithi123.stdout:723: finishing write tid 1 to smithi12313934-14 2020-01-16T00:12:42.831 INFO:tasks.rados.rados.0.smithi123.stdout:723: finishing write tid 2 to smithi12313934-14 2020-01-16T00:12:42.832 INFO:tasks.rados.rados.0.smithi123.stdout:719: done (15 left) 2020-01-16T00:12:42.832 INFO:tasks.rados.rados.0.smithi123.stdout:720: done (14 left) 2020-01-16T00:12:42.832 INFO:tasks.rados.rados.0.smithi123.stdout:721: done (13 left) 2020-01-16T00:12:42.832 INFO:tasks.rados.rados.0.smithi123.stdout:722: done (12 left) 2020-01-16T00:12:42.832 INFO:tasks.rados.rados.0.smithi123.stdout:736: read oid 20 snap -1 2020-01-16T00:12:42.832 INFO:tasks.rados.rados.0.smithi123.stdout:736: expect (ObjNum 115 snap 0 seq_num 115) 2020-01-16T00:12:42.832 INFO:tasks.rados.rados.0.smithi123.stdout:737: read oid 26 snap -1 2020-01-16T00:12:42.833 INFO:tasks.rados.rados.0.smithi123.stdout:737: expect (ObjNum 107 snap 0 seq_num 107) 2020-01-16T00:12:42.833 INFO:tasks.rados.rados.0.smithi123.stdout:738: write oid 12 current snap is 0 2020-01-16T00:12:42.833 INFO:tasks.rados.rados.0.smithi123.stdout:738: seq_num 121 ranges {433647=450424,1331027=582033} 2020-01-16T00:12:42.835 INFO:tasks.rados.rados.0.smithi123.stdout:738: writing smithi12313934-12 from 433647 to 884071 tid 1 2020-01-16T00:12:42.836 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:42.837+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:42.842 INFO:tasks.rados.rados.0.smithi123.stdout:738: writing smithi12313934-12 from 1331027 to 1913060 tid 2 2020-01-16T00:12:42.842 INFO:tasks.rados.rados.0.smithi123.stdout:739: read oid 49 snap -1 2020-01-16T00:12:42.842 INFO:tasks.rados.rados.0.smithi123.stdout:739: expect (ObjNum 48 snap 0 seq_num 48) 2020-01-16T00:12:42.842 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:42.842 INFO:tasks.rados.rados.0.smithi123.stdout:723: finishing write tid 3 to smithi12313934-14 2020-01-16T00:12:42.842 INFO:tasks.rados.rados.0.smithi123.stdout:723: finishing write tid 5 to smithi12313934-14 2020-01-16T00:12:42.843 INFO:tasks.rados.rados.0.smithi123.stdout:723: finishing write tid 6 to smithi12313934-14 2020-01-16T00:12:42.843 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 14 v 91 (ObjNum 117 snap 0 seq_num 117) dirty exists 2020-01-16T00:12:42.843 INFO:tasks.rados.rados.0.smithi123.stdout:723: left oid 14 (ObjNum 117 snap 0 seq_num 117) 2020-01-16T00:12:42.843 INFO:tasks.rados.rados.0.smithi123.stdout:724: expect (ObjNum 41 snap 0 seq_num 41) 2020-01-16T00:12:42.890 INFO:teuthology.orchestra.run.smithi123.stderr:noscrub is set 2020-01-16T00:12:42.928 INFO:tasks.rados.rados.0.smithi123.stdout:725: expect (ObjNum 101 snap 0 seq_num 101) 2020-01-16T00:12:42.934 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:42.934 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:12:42.934 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:12:42.935 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:12:42.935 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:42.937 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:12:42.939+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:12:42.943 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-01-16T00:12:43.037 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:43.037+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:12:43.071 INFO:tasks.rados.rados.0.smithi123.stdout:723: done (15 left) 2020-01-16T00:12:43.071 INFO:tasks.rados.rados.0.smithi123.stdout:724: done (14 left) 2020-01-16T00:12:43.072 INFO:tasks.rados.rados.0.smithi123.stdout:725: done (13 left) 2020-01-16T00:12:43.072 INFO:tasks.rados.rados.0.smithi123.stdout:728: done (12 left) 2020-01-16T00:12:43.072 INFO:tasks.rados.rados.0.smithi123.stdout:740: read oid 16 snap -1 2020-01-16T00:12:43.072 INFO:tasks.rados.rados.0.smithi123.stdout:740: expect (ObjNum 60 snap 0 seq_num 60) 2020-01-16T00:12:43.073 INFO:tasks.rados.rados.0.smithi123.stdout:741: read oid 42 snap -1 2020-01-16T00:12:43.073 INFO:tasks.rados.rados.0.smithi123.stdout:741: expect (ObjNum 41 snap 0 seq_num 41) 2020-01-16T00:12:43.073 INFO:tasks.rados.rados.0.smithi123.stdout:742: read oid 25 snap -1 2020-01-16T00:12:43.073 INFO:tasks.rados.rados.0.smithi123.stdout:742: expect (ObjNum 112 snap 0 seq_num 112) 2020-01-16T00:12:43.073 INFO:tasks.rados.rados.0.smithi123.stdout:743: write oid 37 current snap is 0 2020-01-16T00:12:43.074 INFO:tasks.rados.rados.0.smithi123.stdout:743: seq_num 122 ranges {450454=642026,1505862=544592} 2020-01-16T00:12:43.077 INFO:tasks.rados.rados.0.smithi123.stdout:743: writing smithi12313934-37 from 450454 to 1092480 tid 1 2020-01-16T00:12:43.083 INFO:tasks.rados.rados.0.smithi123.stdout:743: writing smithi12313934-37 from 1505862 to 2050454 tid 2 2020-01-16T00:12:43.083 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:43.083 INFO:tasks.rados.rados.0.smithi123.stdout:727: expect (ObjNum 59 snap 0 seq_num 59) 2020-01-16T00:12:43.098 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:43.098 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:12:43.099 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:12:43.099 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:12:43.099 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.099 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:8 2.1 2:aa80ea11:::smithi12313934-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [read 70656~122880 [fadvise_sequential+fadvise_dontneed]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e24)", 2020-01-16T00:12:43.100 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.123508+0000", 2020-01-16T00:12:43.100 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.9765106670000003, 2020-01-16T00:12:43.100 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0028394589999999999, 2020-01-16T00:12:43.100 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:43.100 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:43.100 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:43.100 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.101 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:43.101 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123508+0000", 2020-01-16T00:12:43.101 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:43.101 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.101 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.102 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:43.102 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123508+0000", 2020-01-16T00:12:43.102 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:12:43.102 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.102 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.102 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:43.102 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123505+0000", 2020-01-16T00:12:43.103 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1678000000000001e-05 2020-01-16T00:12:43.103 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.103 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.103 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:43.103 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123517+0000", 2020-01-16T00:12:43.104 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5393999999999999e-05 2020-01-16T00:12:43.104 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.104 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.104 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:43.104 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123562+0000", 2020-01-16T00:12:43.104 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1204e-05 2020-01-16T00:12:43.105 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.105 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.105 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.105 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123573+0000", 2020-01-16T00:12:43.105 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7509000000000005e-05 2020-01-16T00:12:43.106 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.106 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.106 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:43.106 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123661+0000", 2020-01-16T00:12:43.106 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000199786 2020-01-16T00:12:43.106 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.106 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.107 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:43.107 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123860+0000", 2020-01-16T00:12:43.107 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0024868519999999999 2020-01-16T00:12:43.107 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.107 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.107 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:43.107 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.126347+0000", 2020-01-16T00:12:43.107 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0028394589999999999 2020-01-16T00:12:43.108 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.108 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:43.108 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.108 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.108 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.108 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:9 2.1 2:aa80ea11:::smithi12313934-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [read 453632~1075200 [fadvise_sequential+fadvise_dontneed]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e24)", 2020-01-16T00:12:43.109 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.123601+0000", 2020-01-16T00:12:43.109 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.9764175079999999, 2020-01-16T00:12:43.109 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041686819999999999, 2020-01-16T00:12:43.109 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:43.110 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:43.110 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:43.110 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.110 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:43.110 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123601+0000", 2020-01-16T00:12:43.111 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:43.111 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.111 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.111 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:43.112 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123601+0000", 2020-01-16T00:12:43.112 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:12:43.112 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.112 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.113 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:43.113 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123600+0000", 2020-01-16T00:12:43.113 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0259999999999996e-06 2020-01-16T00:12:43.113 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.113 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.114 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:43.114 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123609+0000", 2020-01-16T00:12:43.114 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2979999999999992e-06 2020-01-16T00:12:43.114 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.114 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.114 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:43.115 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123617+0000", 2020-01-16T00:12:43.115 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3850000000000002e-06 2020-01-16T00:12:43.115 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.115 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.115 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.115 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123626+0000", 2020-01-16T00:12:43.115 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0027674969999999998 2020-01-16T00:12:43.115 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.116 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.116 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:43.116 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.126393+0000", 2020-01-16T00:12:43.116 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013459499999999999 2020-01-16T00:12:43.116 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.116 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.116 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:43.116 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.126528+0000", 2020-01-16T00:12:43.117 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012418900000000001 2020-01-16T00:12:43.117 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.117 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.117 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:43.117 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.127770+0000", 2020-01-16T00:12:43.117 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041686819999999999 2020-01-16T00:12:43.117 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.118 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:43.118 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.118 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.118 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.118 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:10 2.1 2:aa80ea11:::smithi12313934-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [read 1658880~398336 [fadvise_sequential+fadvise_dontneed]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e24)", 2020-01-16T00:12:43.118 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.123646+0000", 2020-01-16T00:12:43.118 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.9763722279999998, 2020-01-16T00:12:43.118 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052215789999999996, 2020-01-16T00:12:43.119 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:43.119 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:43.119 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:43.119 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.119 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:43.119 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123646+0000", 2020-01-16T00:12:43.119 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:43.119 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.120 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.120 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:43.120 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123646+0000", 2020-01-16T00:12:43.120 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:12:43.120 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.120 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.120 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:43.121 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123645+0000", 2020-01-16T00:12:43.121 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7880000000000003e-06 2020-01-16T00:12:43.121 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.121 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.121 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:43.121 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123651+0000", 2020-01-16T00:12:43.121 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.218e-05 2020-01-16T00:12:43.121 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.122 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.122 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:43.122 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123663+0000", 2020-01-16T00:12:43.122 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0930999999999999e-05 2020-01-16T00:12:43.122 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.122 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.122 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.123 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123674+0000", 2020-01-16T00:12:43.123 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041493830000000004 2020-01-16T00:12:43.123 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.123 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.123 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:43.123 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.127824+0000", 2020-01-16T00:12:43.123 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00025807199999999998 2020-01-16T00:12:43.123 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.124 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.124 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:43.124 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.128082+0000", 2020-01-16T00:12:43.124 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00078618000000000004 2020-01-16T00:12:43.124 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.124 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.124 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:43.124 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.128868+0000", 2020-01-16T00:12:43.125 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052215789999999996 2020-01-16T00:12:43.125 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.125 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:43.125 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.125 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.125 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.125 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.1.0:152 3.4 e24/20 3:22ad9a20:::smithi12313934-17:head v 24'55, mlcod=23'53)", 2020-01-16T00:12:43.126 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.151668+0000", 2020-01-16T00:12:43.126 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.9483502919999998, 2020-01-16T00:12:43.126 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0026942580000000002, 2020-01-16T00:12:43.126 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:43.126 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:43.126 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:43.126 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.127 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:43.127 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.151668+0000", 2020-01-16T00:12:43.127 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:43.127 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.127 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.127 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:43.127 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.151668+0000", 2020-01-16T00:12:43.127 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:43.128 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.128 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.128 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:43.128 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.151666+0000", 2020-01-16T00:12:43.128 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020745400000000001 2020-01-16T00:12:43.128 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.128 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.128 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:43.129 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.151874+0000", 2020-01-16T00:12:43.129 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4854e-05 2020-01-16T00:12:43.129 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.129 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.129 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:43.129 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.151888+0000", 2020-01-16T00:12:43.129 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8589999999999999e-06 2020-01-16T00:12:43.130 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.130 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.130 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.130 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.151895+0000", 2020-01-16T00:12:43.130 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2868999999999996e-05 2020-01-16T00:12:43.130 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.130 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.131 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:43.131 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.151978+0000", 2020-01-16T00:12:43.131 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6750999999999998e-05 2020-01-16T00:12:43.131 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.131 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.131 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:43.131 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.152005+0000", 2020-01-16T00:12:43.132 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0023119740000000001 2020-01-16T00:12:43.132 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.132 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.132 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:43.132 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.154317+0000", 2020-01-16T00:12:43.133 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5549e-05 2020-01-16T00:12:43.133 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.133 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.133 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:43.133 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.154362+0000", 2020-01-16T00:12:43.134 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0026942580000000002 2020-01-16T00:12:43.134 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.134 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:43.134 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.134 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.134 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.134 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:909 3.5 e24/20 3:ad8ab15f:::smithi12313934-27:head v 24'47, mlcod=23'45)", 2020-01-16T00:12:43.135 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.825287+0000", 2020-01-16T00:12:43.135 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.2747315129999999, 2020-01-16T00:12:43.135 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0027923050000000001, 2020-01-16T00:12:43.135 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:43.135 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:43.135 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:43.135 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.135 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:43.136 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.825287+0000", 2020-01-16T00:12:43.136 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:43.136 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.136 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.136 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:43.136 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.825287+0000", 2020-01-16T00:12:43.136 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:43.136 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.137 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.137 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:43.137 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.825285+0000", 2020-01-16T00:12:43.137 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00077494 2020-01-16T00:12:43.137 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.138 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.138 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:43.138 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.826060+0000", 2020-01-16T00:12:43.138 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5159000000000001e-05 2020-01-16T00:12:43.138 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.139 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.139 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:43.139 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.826085+0000", 2020-01-16T00:12:43.139 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0479000000000001e-05 2020-01-16T00:12:43.139 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.139 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.139 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.139 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.826095+0000", 2020-01-16T00:12:43.140 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9318999999999999e-05 2020-01-16T00:12:43.140 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.140 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.140 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:43.140 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.826154+0000", 2020-01-16T00:12:43.140 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8455000000000001e-05 2020-01-16T00:12:43.140 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.141 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.141 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:43.141 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.826183+0000", 2020-01-16T00:12:43.141 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001849093 2020-01-16T00:12:43.141 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.141 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.141 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:43.141 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.828032+0000", 2020-01-16T00:12:43.142 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7256000000000003e-05 2020-01-16T00:12:43.142 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.142 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.142 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:43.142 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.828079+0000", 2020-01-16T00:12:43.142 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0027923050000000001 2020-01-16T00:12:43.142 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.143 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:43.143 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.143 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.143 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.143 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:910 3.5 e24/20 3:ad8ab15f:::smithi12313934-27:head v 24'49, mlcod=23'46)", 2020-01-16T00:12:43.143 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.845784+0000", 2020-01-16T00:12:43.143 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.2542349540000002, 2020-01-16T00:12:43.143 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0032848759999999999, 2020-01-16T00:12:43.144 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:43.144 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:43.144 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:43.144 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.144 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:43.144 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.845784+0000", 2020-01-16T00:12:43.144 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:43.144 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.145 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.145 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:43.145 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.845784+0000", 2020-01-16T00:12:43.145 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:12:43.145 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.145 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.145 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:43.146 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.845782+0000", 2020-01-16T00:12:43.146 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001034952 2020-01-16T00:12:43.146 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.146 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.146 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:43.146 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.846817+0000", 2020-01-16T00:12:43.146 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4044e-05 2020-01-16T00:12:43.146 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.147 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.147 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:43.147 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.846831+0000", 2020-01-16T00:12:43.147 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8999999999999997e-06 2020-01-16T00:12:43.147 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.148 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.148 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.148 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.846836+0000", 2020-01-16T00:12:43.148 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0311000000000001e-05 2020-01-16T00:12:43.148 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.148 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.148 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:43.149 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.846877+0000", 2020-01-16T00:12:43.149 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.745e-05 2020-01-16T00:12:43.149 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.149 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.149 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:43.149 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.846904+0000", 2020-01-16T00:12:43.150 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002122667 2020-01-16T00:12:43.150 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.150 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.150 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:43.151 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.849027+0000", 2020-01-16T00:12:43.151 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1607999999999999e-05 2020-01-16T00:12:43.151 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.151 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.151 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:43.151 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.849068+0000", 2020-01-16T00:12:43.151 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0032848759999999999 2020-01-16T00:12:43.152 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.152 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:43.152 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.152 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.152 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.152 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:938 3.4 e24/20 3:2bdd9e88:::smithi12313934-4:head v 24'60, mlcod=24'58)", 2020-01-16T00:12:43.152 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.912054+0000", 2020-01-16T00:12:43.152 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.187964536, 2020-01-16T00:12:43.153 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030936610000000002, 2020-01-16T00:12:43.153 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:43.153 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:43.153 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:43.153 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.153 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:43.153 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.912054+0000", 2020-01-16T00:12:43.154 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:43.154 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.154 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.154 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:43.154 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.912054+0000", 2020-01-16T00:12:43.154 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:43.154 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.155 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.155 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:43.155 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.912053+0000", 2020-01-16T00:12:43.155 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019864899999999999 2020-01-16T00:12:43.155 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.155 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.155 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:43.155 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.912251+0000", 2020-01-16T00:12:43.156 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1235e-05 2020-01-16T00:12:43.156 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.156 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.156 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:43.156 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.912262+0000", 2020-01-16T00:12:43.156 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3989999999999997e-06 2020-01-16T00:12:43.156 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.157 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.157 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.157 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.912267+0000", 2020-01-16T00:12:43.157 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4737000000000003e-05 2020-01-16T00:12:43.157 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.157 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.157 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:43.157 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.912302+0000", 2020-01-16T00:12:43.158 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2785e-05 2020-01-16T00:12:43.158 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.158 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.158 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:43.158 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.912324+0000", 2020-01-16T00:12:43.158 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0027574140000000001 2020-01-16T00:12:43.158 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.159 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.159 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:43.159 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915082+0000", 2020-01-16T00:12:43.159 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5845000000000004e-05 2020-01-16T00:12:43.159 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.159 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.159 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:43.159 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915148+0000", 2020-01-16T00:12:43.160 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030936610000000002 2020-01-16T00:12:43.160 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.160 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:43.160 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.160 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.160 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.160 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.1.13:28 2.3 e24/19 2:de10f4fa:::smithi12313934-4 low_tier:head v 24'20, mlcod=23'18)", 2020-01-16T00:12:43.160 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.915072+0000", 2020-01-16T00:12:43.161 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.1849462649999998, 2020-01-16T00:12:43.161 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003575015, 2020-01-16T00:12:43.161 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:43.161 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:43.161 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:43.161 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.161 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:43.161 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915072+0000", 2020-01-16T00:12:43.162 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:43.162 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.162 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.162 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:43.162 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915072+0000", 2020-01-16T00:12:43.162 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:43.162 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.163 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.163 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:43.163 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915070+0000", 2020-01-16T00:12:43.163 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000111706 2020-01-16T00:12:43.163 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.163 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.163 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:43.163 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915182+0000", 2020-01-16T00:12:43.164 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1409999999999999e-05 2020-01-16T00:12:43.164 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.164 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.164 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:43.164 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915193+0000", 2020-01-16T00:12:43.164 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0713999999999999e-05 2020-01-16T00:12:43.164 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.165 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.165 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.165 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915204+0000", 2020-01-16T00:12:43.165 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0732000000000002e-05 2020-01-16T00:12:43.165 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.165 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.165 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:43.165 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915245+0000", 2020-01-16T00:12:43.166 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7824e-05 2020-01-16T00:12:43.166 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.166 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.166 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:43.166 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915272+0000", 2020-01-16T00:12:43.166 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033395159999999998 2020-01-16T00:12:43.166 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.166 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.167 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:43.167 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.918612+0000", 2020-01-16T00:12:43.167 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5345000000000001e-05 2020-01-16T00:12:43.167 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.167 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.167 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:43.167 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.918647+0000", 2020-01-16T00:12:43.168 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003575015 2020-01-16T00:12:43.168 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.168 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:43.168 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.168 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.168 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.168 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.4.0:112 3.9 e24/20 3:9218e7f1:::smithi12313934-14:head v 24'72, mlcod=23'70)", 2020-01-16T00:12:43.168 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.925123+0000", 2020-01-16T00:12:43.169 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.174895266, 2020-01-16T00:12:43.169 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.004957024, 2020-01-16T00:12:43.169 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:43.169 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:43.169 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:43.169 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.169 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:43.170 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.925123+0000", 2020-01-16T00:12:43.170 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:43.170 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.170 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.170 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:43.170 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.925123+0000", 2020-01-16T00:12:43.170 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:43.171 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.171 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.171 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:43.171 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.925121+0000", 2020-01-16T00:12:43.171 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0024713119999999998 2020-01-16T00:12:43.171 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.171 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.172 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:43.172 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927593+0000", 2020-01-16T00:12:43.172 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7319999999999993e-06 2020-01-16T00:12:43.172 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.172 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.172 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:43.172 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927603+0000", 2020-01-16T00:12:43.172 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6600000000000003e-06 2020-01-16T00:12:43.173 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.173 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.173 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.173 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927607+0000", 2020-01-16T00:12:43.173 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7138999999999998e-05 2020-01-16T00:12:43.173 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.173 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.173 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:43.174 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927644+0000", 2020-01-16T00:12:43.174 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7068999999999999e-05 2020-01-16T00:12:43.174 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.174 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.174 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:43.174 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927661+0000", 2020-01-16T00:12:43.174 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0023781840000000002 2020-01-16T00:12:43.174 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.175 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.175 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:43.175 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.930040+0000", 2020-01-16T00:12:43.175 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0679000000000001e-05 2020-01-16T00:12:43.175 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.175 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.175 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:43.176 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.930080+0000", 2020-01-16T00:12:43.176 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.004957024 2020-01-16T00:12:43.176 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.176 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:43.176 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.176 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.176 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.176 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.1.0:162 3.0 e24/20 3:00c146d5:::smithi12313934-18:head v 24'99, mlcod=24'97)", 2020-01-16T00:12:43.177 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.039167+0000", 2020-01-16T00:12:43.177 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.0608519809999999, 2020-01-16T00:12:43.177 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0049106159999999996, 2020-01-16T00:12:43.177 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:43.177 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:43.177 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:43.178 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.178 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:43.178 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.039167+0000", 2020-01-16T00:12:43.178 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:43.178 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.178 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.178 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:43.179 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.039167+0000", 2020-01-16T00:12:43.179 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999957 2020-01-16T00:12:43.179 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.179 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.179 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:43.180 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.039162+0000", 2020-01-16T00:12:43.180 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000551155 2020-01-16T00:12:43.180 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.180 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.180 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:43.180 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.039713+0000", 2020-01-16T00:12:43.180 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1465999999999999e-05 2020-01-16T00:12:43.181 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.181 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.181 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:43.181 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.039755+0000", 2020-01-16T00:12:43.181 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7122999999999999e-05 2020-01-16T00:12:43.181 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.181 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.181 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.182 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.039772+0000", 2020-01-16T00:12:43.182 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6485999999999997e-05 2020-01-16T00:12:43.182 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.182 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.182 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:43.182 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.039868+0000", 2020-01-16T00:12:43.183 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1465999999999996e-05 2020-01-16T00:12:43.183 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.183 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.183 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:43.184 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.039940+0000", 2020-01-16T00:12:43.184 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040611019999999996 2020-01-16T00:12:43.184 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.184 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.184 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:43.184 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.044001+0000", 2020-01-16T00:12:43.185 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6151999999999999e-05 2020-01-16T00:12:43.185 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.185 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.185 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:43.185 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.044077+0000", 2020-01-16T00:12:43.185 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0049106159999999996 2020-01-16T00:12:43.185 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.186 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:43.186 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.186 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.186 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.186 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:957 3.0 e24/20 3:00c146d5:::smithi12313934-18:head v 24'100, mlcod=24'97)", 2020-01-16T00:12:43.186 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.042105+0000", 2020-01-16T00:12:43.186 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.0579132309999997, 2020-01-16T00:12:43.187 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0046019770000000002, 2020-01-16T00:12:43.187 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:43.187 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:43.187 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:43.187 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.187 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:43.187 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.042105+0000", 2020-01-16T00:12:43.187 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:43.188 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.188 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.188 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:43.188 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.042105+0000", 2020-01-16T00:12:43.188 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999962 2020-01-16T00:12:43.188 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.188 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.188 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:43.189 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.042101+0000", 2020-01-16T00:12:43.189 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00041373799999999997 2020-01-16T00:12:43.189 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.189 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.189 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:43.189 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.042515+0000", 2020-01-16T00:12:43.189 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.858e-05 2020-01-16T00:12:43.190 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.190 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.190 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:43.190 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.042554+0000", 2020-01-16T00:12:43.190 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5400000000000002e-05 2020-01-16T00:12:43.190 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.190 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.191 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.191 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.042569+0000", 2020-01-16T00:12:43.191 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1859999999999994e-05 2020-01-16T00:12:43.191 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.191 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.191 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:43.191 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.042631+0000", 2020-01-16T00:12:43.191 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7620999999999999e-05 2020-01-16T00:12:43.192 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.192 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.192 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:43.192 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.042658+0000", 2020-01-16T00:12:43.192 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040138750000000001 2020-01-16T00:12:43.192 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.192 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.193 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:43.193 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.046672+0000", 2020-01-16T00:12:43.193 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4952e-05 2020-01-16T00:12:43.193 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.193 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.193 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:43.193 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.046707+0000", 2020-01-16T00:12:43.194 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0046019770000000002 2020-01-16T00:12:43.194 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.194 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:43.194 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.194 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.194 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.194 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:958 3.0 e24/20 3:00c146d5:::smithi12313934-18:head v 24'101, mlcod=24'97)", 2020-01-16T00:12:43.195 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.044214+0000", 2020-01-16T00:12:43.195 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.0558044549999996, 2020-01-16T00:12:43.195 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033269810000000001, 2020-01-16T00:12:43.195 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:43.195 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:43.195 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:43.195 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.196 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:43.196 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.044214+0000", 2020-01-16T00:12:43.196 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:43.196 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.196 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.196 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:43.196 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.044214+0000", 2020-01-16T00:12:43.196 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:43.197 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.197 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.197 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:43.197 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.044212+0000", 2020-01-16T00:12:43.197 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016133 2020-01-16T00:12:43.197 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.197 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.197 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:43.198 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.044374+0000", 2020-01-16T00:12:43.198 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0190000000000006e-06 2020-01-16T00:12:43.198 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.198 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.198 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:43.198 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.044382+0000", 2020-01-16T00:12:43.198 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1409999999999998e-06 2020-01-16T00:12:43.199 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.199 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.199 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.199 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.044386+0000", 2020-01-16T00:12:43.199 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5098999999999997e-05 2020-01-16T00:12:43.199 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.200 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.200 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:43.200 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.044421+0000", 2020-01-16T00:12:43.200 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.4280000000000001e-05 2020-01-16T00:12:43.200 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.200 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.200 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:43.200 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.044445+0000", 2020-01-16T00:12:43.201 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030470100000000002 2020-01-16T00:12:43.201 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.201 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.201 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:43.201 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.047492+0000", 2020-01-16T00:12:43.201 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8952999999999997e-05 2020-01-16T00:12:43.201 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.202 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.202 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:43.202 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.047541+0000", 2020-01-16T00:12:43.202 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033269810000000001 2020-01-16T00:12:43.202 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.202 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:43.202 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.202 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.203 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.203 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:959 3.0 e24/20 3:00c146d5:::smithi12313934-18:head v 24'102, mlcod=24'98)", 2020-01-16T00:12:43.203 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.047434+0000", 2020-01-16T00:12:43.203 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.0525849840000001, 2020-01-16T00:12:43.203 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0038721860000000001, 2020-01-16T00:12:43.203 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:43.203 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:43.204 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:43.204 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.204 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:43.204 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.047434+0000", 2020-01-16T00:12:43.204 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:43.204 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.204 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.204 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:43.205 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.047434+0000", 2020-01-16T00:12:43.205 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:43.205 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.205 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.205 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:43.205 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.047432+0000", 2020-01-16T00:12:43.205 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00027367899999999998 2020-01-16T00:12:43.206 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.206 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.206 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:43.206 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.047705+0000", 2020-01-16T00:12:43.206 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1781e-05 2020-01-16T00:12:43.206 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.206 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.206 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:43.207 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.047717+0000", 2020-01-16T00:12:43.207 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5249999999999996e-06 2020-01-16T00:12:43.207 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.207 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.207 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.207 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.047723+0000", 2020-01-16T00:12:43.207 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7784999999999998e-05 2020-01-16T00:12:43.208 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.208 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.208 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:43.208 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.047760+0000", 2020-01-16T00:12:43.208 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2864999999999999e-05 2020-01-16T00:12:43.208 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.208 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.208 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:43.209 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.047783+0000", 2020-01-16T00:12:43.209 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0034841450000000001 2020-01-16T00:12:43.209 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.209 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.209 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:43.209 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.051268+0000", 2020-01-16T00:12:43.209 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.82e-05 2020-01-16T00:12:43.210 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.210 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.210 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:43.210 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.051306+0000", 2020-01-16T00:12:43.210 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0038721860000000001 2020-01-16T00:12:43.210 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.210 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:43.211 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.211 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.211 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.211 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:973 3.5 e24/20 3:aef1ce98:::smithi12313934-9:head v 24'53, mlcod=24'51)", 2020-01-16T00:12:43.211 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.305124+0000", 2020-01-16T00:12:43.211 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.7948941449999998, 2020-01-16T00:12:43.211 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036133279999999999, 2020-01-16T00:12:43.211 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:43.212 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:43.212 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:43.212 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.212 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:43.212 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.305124+0000", 2020-01-16T00:12:43.212 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:43.212 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.213 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.213 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:43.213 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.305124+0000", 2020-01-16T00:12:43.213 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999857 2020-01-16T00:12:43.213 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.213 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.213 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:43.213 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.305110+0000", 2020-01-16T00:12:43.214 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013042819999999999 2020-01-16T00:12:43.214 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.214 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.214 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:43.214 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.306414+0000", 2020-01-16T00:12:43.214 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2325000000000001e-05 2020-01-16T00:12:43.214 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.215 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.215 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:43.215 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.306437+0000", 2020-01-16T00:12:43.215 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1420000000000001e-05 2020-01-16T00:12:43.215 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.215 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.215 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.215 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.306448+0000", 2020-01-16T00:12:43.216 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5415999999999997e-05 2020-01-16T00:12:43.216 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.216 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.216 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:43.216 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.306534+0000", 2020-01-16T00:12:43.216 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4882000000000002e-05 2020-01-16T00:12:43.216 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.216 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.217 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:43.217 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.306568+0000", 2020-01-16T00:12:43.217 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021100450000000001 2020-01-16T00:12:43.217 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.217 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.217 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:43.217 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.308678+0000", 2020-01-16T00:12:43.218 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9237999999999998e-05 2020-01-16T00:12:43.218 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.218 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.218 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:43.218 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.308738+0000", 2020-01-16T00:12:43.218 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036133279999999999 2020-01-16T00:12:43.218 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.219 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:43.219 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.219 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.219 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.219 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.4.0:114 3.9 e24/20 3:9fc687b2:::smithi12313934-44:head v 24'74, mlcod=24'72)", 2020-01-16T00:12:43.219 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:12:43.219 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.7747397569999999, 2020-01-16T00:12:43.219 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010399429, 2020-01-16T00:12:43.220 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:43.220 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:43.220 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:43.220 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.220 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:43.220 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:12:43.220 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:43.220 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.221 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.221 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:43.221 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:12:43.221 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:12:43.221 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.221 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.221 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:43.221 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325276+0000", 2020-01-16T00:12:43.222 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0046127069999999997 2020-01-16T00:12:43.222 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.222 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.222 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:43.222 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329889+0000", 2020-01-16T00:12:43.222 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3519999999999998e-05 2020-01-16T00:12:43.222 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.223 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.223 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:43.223 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329912+0000", 2020-01-16T00:12:43.223 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4070000000000004e-06 2020-01-16T00:12:43.223 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.223 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.223 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.224 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329920+0000", 2020-01-16T00:12:43.224 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8817000000000003e-05 2020-01-16T00:12:43.224 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.224 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.224 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:43.224 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329978+0000", 2020-01-16T00:12:43.224 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9721e-05 2020-01-16T00:12:43.224 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.225 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.225 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:43.225 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329998+0000", 2020-01-16T00:12:43.225 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056328139999999999 2020-01-16T00:12:43.225 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.225 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.225 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:43.226 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.335631+0000", 2020-01-16T00:12:43.226 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7170999999999999e-05 2020-01-16T00:12:43.226 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.226 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.226 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:43.226 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.335678+0000", 2020-01-16T00:12:43.226 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010399429 2020-01-16T00:12:43.226 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.227 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:43.227 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.227 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.227 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.227 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:24 2.1 2:8b431f80:::smithi12313934-40 low_tier:head [write 790528~688128] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:12:43.227 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.482303+0000", 2020-01-16T00:12:43.227 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.6177156610000001, 2020-01-16T00:12:43.227 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010071853, 2020-01-16T00:12:43.228 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:43.228 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:43.228 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:43.228 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.228 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:43.228 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.482303+0000", 2020-01-16T00:12:43.228 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:43.229 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.229 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.229 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:43.229 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.482303+0000", 2020-01-16T00:12:43.229 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:43.230 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.230 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.230 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:43.230 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.482301+0000", 2020-01-16T00:12:43.230 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031173960000000001 2020-01-16T00:12:43.230 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.230 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.230 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:43.231 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485418+0000", 2020-01-16T00:12:43.231 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9300000000000001e-05 2020-01-16T00:12:43.231 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.231 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.231 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:43.232 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485448+0000", 2020-01-16T00:12:43.232 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2611e-05 2020-01-16T00:12:43.232 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.232 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.232 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.232 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485460+0000", 2020-01-16T00:12:43.232 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112487 2020-01-16T00:12:43.233 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.233 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.233 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:43.233 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485573+0000", 2020-01-16T00:12:43.233 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022169299999999999 2020-01-16T00:12:43.233 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.233 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.234 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:43.234 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485794+0000", 2020-01-16T00:12:43.234 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00036287500000000002 2020-01-16T00:12:43.234 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.234 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.234 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:12:43.235 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.486157+0000", 2020-01-16T00:12:43.235 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040260310000000002 2020-01-16T00:12:43.235 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.235 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.235 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:43.236 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.490183+0000", 2020-01-16T00:12:43.236 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00081083000000000001 2020-01-16T00:12:43.236 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.236 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.236 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:43.236 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.490994+0000", 2020-01-16T00:12:43.236 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001287746 2020-01-16T00:12:43.236 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.237 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.237 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:43.237 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492282+0000", 2020-01-16T00:12:43.237 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8810999999999998e-05 2020-01-16T00:12:43.237 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.237 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.237 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:43.238 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492321+0000", 2020-01-16T00:12:43.238 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4095000000000003e-05 2020-01-16T00:12:43.238 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.238 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.238 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:43.238 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492375+0000", 2020-01-16T00:12:43.238 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010071853 2020-01-16T00:12:43.238 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.239 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:43.239 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.239 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.239 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.239 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:25 2.1 2:8b431f80:::smithi12313934-40 low_tier:head [write 1926656~67072] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:12:43.239 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.485481+0000", 2020-01-16T00:12:43.239 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.6145371959999997, 2020-01-16T00:12:43.240 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0081599020000000001, 2020-01-16T00:12:43.240 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:43.240 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:43.240 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:43.240 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.240 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:43.240 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485481+0000", 2020-01-16T00:12:43.240 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:43.241 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.241 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.241 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:43.241 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485481+0000", 2020-01-16T00:12:43.241 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:43.241 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.241 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.242 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:43.242 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485479+0000", 2020-01-16T00:12:43.242 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012423800000000001 2020-01-16T00:12:43.242 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.242 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.242 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:43.242 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485603+0000", 2020-01-16T00:12:43.243 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8366999999999999e-05 2020-01-16T00:12:43.243 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.243 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.243 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:43.243 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485621+0000", 2020-01-16T00:12:43.243 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0207000000000001e-05 2020-01-16T00:12:43.243 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.243 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.244 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.244 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485632+0000", 2020-01-16T00:12:43.244 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002868532 2020-01-16T00:12:43.244 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.244 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.244 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:43.244 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.488500+0000", 2020-01-16T00:12:43.245 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011340000000000001 2020-01-16T00:12:43.245 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.245 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.245 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:43.245 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.488614+0000", 2020-01-16T00:12:43.245 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019056300000000001 2020-01-16T00:12:43.245 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.245 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.246 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:12:43.246 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.488804+0000", 2020-01-16T00:12:43.246 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00153302 2020-01-16T00:12:43.246 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.246 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.246 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:43.246 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.490337+0000", 2020-01-16T00:12:43.247 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00089946100000000005 2020-01-16T00:12:43.247 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.247 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.247 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:43.247 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.491237+0000", 2020-01-16T00:12:43.247 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0022862820000000002 2020-01-16T00:12:43.247 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.247 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.248 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:43.248 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.493523+0000", 2020-01-16T00:12:43.248 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5609999999999998e-05 2020-01-16T00:12:43.248 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.248 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.248 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:43.248 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.493578+0000", 2020-01-16T00:12:43.249 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2770999999999999e-05 2020-01-16T00:12:43.249 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.249 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.249 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:43.249 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.493641+0000", 2020-01-16T00:12:43.250 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0081599020000000001 2020-01-16T00:12:43.251 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.251 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:43.251 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.251 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.251 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.252 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:1216 3.9 e26/20 3:9218e7f1:::smithi12313934-14:head v 26'88, mlcod=25'86)", 2020-01-16T00:12:43.252 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:42.373228+0000", 2020-01-16T00:12:43.252 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.72679072600000005, 2020-01-16T00:12:43.252 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0028688559999999999, 2020-01-16T00:12:43.253 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:43.253 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:43.253 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:43.253 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.253 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:43.254 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.373228+0000", 2020-01-16T00:12:43.254 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:43.254 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.254 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.255 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:43.255 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.373228+0000", 2020-01-16T00:12:43.255 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:43.255 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.256 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.256 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:43.256 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.373226+0000", 2020-01-16T00:12:43.257 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000653968 2020-01-16T00:12:43.257 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.257 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.257 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:43.257 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.373879+0000", 2020-01-16T00:12:43.258 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6801999999999999e-05 2020-01-16T00:12:43.258 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.258 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.258 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:43.259 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.373896+0000", 2020-01-16T00:12:43.259 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.083e-06 2020-01-16T00:12:43.259 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.259 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.259 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.260 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.373902+0000", 2020-01-16T00:12:43.260 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9879999999999998e-05 2020-01-16T00:12:43.260 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.260 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.261 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:43.261 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.373942+0000", 2020-01-16T00:12:43.261 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0975000000000002e-05 2020-01-16T00:12:43.261 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.262 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.262 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:43.262 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.373963+0000", 2020-01-16T00:12:43.262 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020747809999999999 2020-01-16T00:12:43.263 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.263 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.263 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:43.263 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.376038+0000", 2020-01-16T00:12:43.263 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8627e-05 2020-01-16T00:12:43.264 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.264 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.264 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:43.264 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.376097+0000", 2020-01-16T00:12:43.265 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0028688559999999999 2020-01-16T00:12:43.265 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.265 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:43.265 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.265 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.266 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.266 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:1217 3.9 e26/20 3:9218e7f1:::smithi12313934-14:head v 26'89, mlcod=25'87)", 2020-01-16T00:12:43.266 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:42.381109+0000", 2020-01-16T00:12:43.266 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.71890938400000004, 2020-01-16T00:12:43.267 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030510870000000001, 2020-01-16T00:12:43.267 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:43.267 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:43.267 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:43.267 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.268 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:43.268 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.381109+0000", 2020-01-16T00:12:43.268 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:43.268 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.269 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.269 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:43.269 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.381109+0000", 2020-01-16T00:12:43.270 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:12:43.270 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.270 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.270 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:43.270 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.381106+0000", 2020-01-16T00:12:43.271 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00068075700000000004 2020-01-16T00:12:43.271 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.271 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.271 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:43.272 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.381787+0000", 2020-01-16T00:12:43.272 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.1463000000000001e-05 2020-01-16T00:12:43.272 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.272 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.273 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:43.273 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.381809+0000", 2020-01-16T00:12:43.273 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0137e-05 2020-01-16T00:12:43.273 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.273 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.274 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.274 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.381819+0000", 2020-01-16T00:12:43.274 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7674999999999999e-05 2020-01-16T00:12:43.274 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.275 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.275 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:43.275 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.381876+0000", 2020-01-16T00:12:43.275 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8028000000000001e-05 2020-01-16T00:12:43.276 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.276 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.276 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:43.276 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.381895+0000", 2020-01-16T00:12:43.277 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0022136180000000001 2020-01-16T00:12:43.277 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.277 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.277 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:43.277 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.384108+0000", 2020-01-16T00:12:43.278 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2068999999999999e-05 2020-01-16T00:12:43.278 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.278 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.278 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:43.279 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.384160+0000", 2020-01-16T00:12:43.279 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030510870000000001 2020-01-16T00:12:43.279 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.279 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:43.279 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.280 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.280 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.280 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:1218 3.9 e26/20 3:9218e7f1:::smithi12313934-14:head v 26'90, mlcod=26'88)", 2020-01-16T00:12:43.280 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:42.387885+0000", 2020-01-16T00:12:43.280 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.712133405, 2020-01-16T00:12:43.281 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002635639, 2020-01-16T00:12:43.281 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:43.281 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:43.281 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:43.282 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.282 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:43.282 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.387885+0000", 2020-01-16T00:12:43.282 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:43.283 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.283 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.283 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:43.283 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.387885+0000", 2020-01-16T00:12:43.283 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:43.284 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.284 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.284 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:43.284 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.387884+0000", 2020-01-16T00:12:43.285 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000402381 2020-01-16T00:12:43.285 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.285 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.285 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:43.285 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.388286+0000", 2020-01-16T00:12:43.286 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9734000000000001e-05 2020-01-16T00:12:43.286 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.286 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.286 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:43.287 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.388306+0000", 2020-01-16T00:12:43.287 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4819999999999999e-06 2020-01-16T00:12:43.287 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.287 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.288 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.288 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.388312+0000", 2020-01-16T00:12:43.288 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0093000000000002e-05 2020-01-16T00:12:43.288 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.288 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.289 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:43.289 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.388372+0000", 2020-01-16T00:12:43.289 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.4037e-05 2020-01-16T00:12:43.289 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.289 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.290 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:43.290 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.388396+0000", 2020-01-16T00:12:43.290 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020895779999999999 2020-01-16T00:12:43.290 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.291 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.291 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:43.291 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.390486+0000", 2020-01-16T00:12:43.291 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4845000000000003e-05 2020-01-16T00:12:43.291 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:43.292 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:43.292 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:43.292 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.390521+0000", 2020-01-16T00:12:43.292 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002635639 2020-01-16T00:12:43.293 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.293 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:43.293 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.293 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:43.294 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:43.294 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:43.296 INFO:tasks.rados.rados.0.smithi123.stdout:729: expect (ObjNum 83 snap 0 seq_num 83) 2020-01-16T00:12:43.296 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:12:43.295+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:12:43.296 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-01-16T00:12:43.396 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:12:43.395+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:12:43.401 INFO:tasks.rados.rados.0.smithi123.stdout:730: finishing write tid 1 to smithi12313934-45 2020-01-16T00:12:43.401 INFO:tasks.rados.rados.0.smithi123.stdout:730: finishing write tid 2 to smithi12313934-45 2020-01-16T00:12:43.401 INFO:tasks.rados.rados.0.smithi123.stdout:732: finishing write tid 1 to smithi12313934-31 2020-01-16T00:12:43.402 INFO:tasks.rados.rados.0.smithi123.stdout:734: expect (ObjNum 5 snap 0 seq_num 5) 2020-01-16T00:12:43.410 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:43.410 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:12:43.410 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:12:43.410 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:43.420 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-01-16T00:12:43.466 INFO:tasks.rados.rados.0.smithi123.stdout:730: finishing write tid 4 to smithi12313934-45 2020-01-16T00:12:43.467 INFO:tasks.rados.rados.0.smithi123.stdout:730: finishing write tid 5 to smithi12313934-45 2020-01-16T00:12:43.467 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 45 v 59 (ObjNum 119 snap 0 seq_num 119) dirty exists 2020-01-16T00:12:43.467 INFO:tasks.rados.rados.0.smithi123.stdout:730: left oid 45 (ObjNum 119 snap 0 seq_num 119) 2020-01-16T00:12:43.467 INFO:tasks.rados.rados.0.smithi123.stdout:732: finishing write tid 2 to smithi12313934-31 2020-01-16T00:12:43.467 INFO:tasks.rados.rados.0.smithi123.stdout:732: finishing write tid 4 to smithi12313934-31 2020-01-16T00:12:43.468 INFO:tasks.rados.rados.0.smithi123.stdout:732: finishing write tid 5 to smithi12313934-31 2020-01-16T00:12:43.468 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 31 v 82 (ObjNum 120 snap 0 seq_num 120) dirty exists 2020-01-16T00:12:43.468 INFO:tasks.rados.rados.0.smithi123.stdout:732: left oid 31 (ObjNum 120 snap 0 seq_num 120) 2020-01-16T00:12:43.468 INFO:tasks.rados.rados.0.smithi123.stdout:733: expect (ObjNum 28 snap 0 seq_num 28) 2020-01-16T00:12:43.496 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:12:43.497+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:12:43.524 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:43.524 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:12:43.525 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:12:43.525 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:12:43.525 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:43.534 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-01-16T00:12:43.537 INFO:tasks.rados.rados.0.smithi123.stdout:731: expect (ObjNum 108 snap 0 seq_num 108) 2020-01-16T00:12:43.623 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:43.625+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:43.689 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:43.690 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:12:43.690 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:12:43.690 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:12:43.690 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.690 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:910 3.5 3:ad8ab15f:::smithi12313934-27:head [write 1563527~583025] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:43.691 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.830607+0000", 2020-01-16T00:12:43.691 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.8563344180000003, 2020-01-16T00:12:43.691 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019080092999999999, 2020-01-16T00:12:43.691 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:43.691 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:43.691 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:43.691 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:43.692 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:43.692 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 910 2020-01-16T00:12:43.692 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.692 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:43.692 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.692 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:43.692 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.830607+0000", 2020-01-16T00:12:43.692 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:43.693 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.693 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.693 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:43.693 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.830607+0000", 2020-01-16T00:12:43.693 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:43.693 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.693 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.694 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:43.694 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.830604+0000", 2020-01-16T00:12:43.694 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00078346100000000005 2020-01-16T00:12:43.694 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.694 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.694 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:43.694 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831388+0000", 2020-01-16T00:12:43.694 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.6936000000000001e-05 2020-01-16T00:12:43.695 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.695 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.695 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:43.695 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831415+0000", 2020-01-16T00:12:43.695 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0737000000000001e-05 2020-01-16T00:12:43.695 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.695 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.695 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.696 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831425+0000", 2020-01-16T00:12:43.696 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1964e-05 2020-01-16T00:12:43.696 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.696 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.696 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.696 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831467+0000", 2020-01-16T00:12:43.696 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028524600000000001 2020-01-16T00:12:43.697 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.697 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.697 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:43.697 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831753+0000", 2020-01-16T00:12:43.697 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012909141000000001 2020-01-16T00:12:43.697 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.697 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.697 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.698 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.844662+0000", 2020-01-16T00:12:43.698 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022242 2020-01-16T00:12:43.698 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.698 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.698 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:43.698 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.844884+0000", 2020-01-16T00:12:43.698 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015169700000000001 2020-01-16T00:12:43.699 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.699 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.699 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,3", 2020-01-16T00:12:43.699 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.845036+0000", 2020-01-16T00:12:43.699 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0027572859999999999 2020-01-16T00:12:43.699 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.699 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.699 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:43.700 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.847793+0000", 2020-01-16T00:12:43.700 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00084882100000000004 2020-01-16T00:12:43.700 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.700 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.700 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:43.700 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.848642+0000", 2020-01-16T00:12:43.700 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00065781900000000003 2020-01-16T00:12:43.700 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.701 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.701 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:43.701 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.849300+0000", 2020-01-16T00:12:43.701 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0001717 2020-01-16T00:12:43.701 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.701 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.701 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:43.713 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.849472+0000", 2020-01-16T00:12:43.713 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000215348 2020-01-16T00:12:43.713 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.713 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.713 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:43.713 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.849687+0000", 2020-01-16T00:12:43.714 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019080092999999999 2020-01-16T00:12:43.714 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.714 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:43.714 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.714 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.714 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.714 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:911 3.5 3:ad8ab15f:::smithi12313934-27:head [write 2810926~399178] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:43.715 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.836707+0000", 2020-01-16T00:12:43.715 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.8502337889999998, 2020-01-16T00:12:43.715 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.024736662999999999, 2020-01-16T00:12:43.715 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:43.715 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:43.715 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:43.715 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:43.715 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:43.716 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 911 2020-01-16T00:12:43.716 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.716 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:43.716 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.716 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:43.716 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.836707+0000", 2020-01-16T00:12:43.716 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:43.716 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.717 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.717 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:43.717 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.836707+0000", 2020-01-16T00:12:43.717 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:12:43.717 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.717 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.717 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:43.718 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.836704+0000", 2020-01-16T00:12:43.718 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00077069200000000001 2020-01-16T00:12:43.718 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.718 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.718 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:43.718 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837474+0000", 2020-01-16T00:12:43.718 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7440000000000001e-05 2020-01-16T00:12:43.718 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.719 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.719 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:43.719 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837512+0000", 2020-01-16T00:12:43.719 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8246000000000001e-05 2020-01-16T00:12:43.719 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.719 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.719 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.719 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837530+0000", 2020-01-16T00:12:43.720 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000105405 2020-01-16T00:12:43.720 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.720 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.720 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.720 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837635+0000", 2020-01-16T00:12:43.720 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6244999999999998e-05 2020-01-16T00:12:43.720 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.721 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.721 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:43.721 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837722+0000", 2020-01-16T00:12:43.721 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0089724320000000007 2020-01-16T00:12:43.721 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.721 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.721 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.721 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.846694+0000", 2020-01-16T00:12:43.722 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042899099999999999 2020-01-16T00:12:43.722 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.722 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.722 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:43.722 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.847123+0000", 2020-01-16T00:12:43.722 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011739893 2020-01-16T00:12:43.722 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.723 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.723 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.723 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.858863+0000", 2020-01-16T00:12:43.723 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1312999999999999e-05 2020-01-16T00:12:43.723 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.723 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.723 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:43.723 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.858944+0000", 2020-01-16T00:12:43.724 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015682900000000001 2020-01-16T00:12:43.724 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.724 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.724 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,3", 2020-01-16T00:12:43.724 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.859101+0000", 2020-01-16T00:12:43.724 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001604442 2020-01-16T00:12:43.724 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.724 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.725 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:43.725 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.860706+0000", 2020-01-16T00:12:43.725 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000377568 2020-01-16T00:12:43.725 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.725 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.725 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:43.725 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.861083+0000", 2020-01-16T00:12:43.725 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022656400000000001 2020-01-16T00:12:43.726 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.726 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.726 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:43.726 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.861310+0000", 2020-01-16T00:12:43.726 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7865000000000002e-05 2020-01-16T00:12:43.726 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.726 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.726 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:43.727 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.861368+0000", 2020-01-16T00:12:43.727 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6549000000000002e-05 2020-01-16T00:12:43.727 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.727 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.727 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:43.727 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.861444+0000", 2020-01-16T00:12:43.727 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.024736662999999999 2020-01-16T00:12:43.728 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.728 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:43.728 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.728 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.728 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.728 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:912 3.5 3:ad8ab15f:::smithi12313934-27:head [setxattr _header (56) in=7b,truncate 3210104] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:43.728 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.837560+0000", 2020-01-16T00:12:43.728 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.8493810770000003, 2020-01-16T00:12:43.729 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.024152410999999999, 2020-01-16T00:12:43.729 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:43.729 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:43.729 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:43.729 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:43.729 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:43.729 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 912 2020-01-16T00:12:43.729 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.730 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:43.730 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.730 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:43.730 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837560+0000", 2020-01-16T00:12:43.730 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:43.730 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.730 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.730 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:43.731 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837560+0000", 2020-01-16T00:12:43.731 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:43.731 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.731 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.731 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:43.731 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837558+0000", 2020-01-16T00:12:43.731 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8468999999999999e-05 2020-01-16T00:12:43.731 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.732 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.732 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:43.732 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837576+0000", 2020-01-16T00:12:43.732 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8437e-05 2020-01-16T00:12:43.732 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.732 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.732 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:43.732 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837595+0000", 2020-01-16T00:12:43.733 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2782999999999999e-05 2020-01-16T00:12:43.733 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.733 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.733 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.733 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837608+0000", 2020-01-16T00:12:43.733 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000174883 2020-01-16T00:12:43.733 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.733 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.734 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.734 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837782+0000", 2020-01-16T00:12:43.734 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8194000000000001e-05 2020-01-16T00:12:43.734 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.734 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.734 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:43.734 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837821+0000", 2020-01-16T00:12:43.735 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0094026500000000002 2020-01-16T00:12:43.735 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.735 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.735 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.735 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.847223+0000", 2020-01-16T00:12:43.735 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000120973 2020-01-16T00:12:43.735 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.735 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.736 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:43.736 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.847344+0000", 2020-01-16T00:12:43.736 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012601658999999999 2020-01-16T00:12:43.736 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.736 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.736 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.736 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.859946+0000", 2020-01-16T00:12:43.737 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012089 2020-01-16T00:12:43.737 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.737 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.737 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:43.737 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.860067+0000", 2020-01-16T00:12:43.737 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020346700000000001 2020-01-16T00:12:43.737 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.737 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.738 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,3", 2020-01-16T00:12:43.738 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.860270+0000", 2020-01-16T00:12:43.738 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00057751500000000002 2020-01-16T00:12:43.738 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.738 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.738 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:43.738 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.860848+0000", 2020-01-16T00:12:43.738 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00067389800000000003 2020-01-16T00:12:43.739 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.739 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.739 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:43.739 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.861522+0000", 2020-01-16T00:12:43.739 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2540999999999996e-05 2020-01-16T00:12:43.739 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.739 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.739 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:43.740 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.861584+0000", 2020-01-16T00:12:43.740 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8773e-05 2020-01-16T00:12:43.740 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.740 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.740 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:43.740 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.861653+0000", 2020-01-16T00:12:43.740 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9493000000000001e-05 2020-01-16T00:12:43.740 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.741 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.741 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:43.741 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.861713+0000", 2020-01-16T00:12:43.741 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.024152410999999999 2020-01-16T00:12:43.741 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.741 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:43.741 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.741 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.742 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.742 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:913 3.5 3:ad8ab15f:::smithi12313934-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:12:43.742 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.837629+0000", 2020-01-16T00:12:43.742 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.8493122590000004, 2020-01-16T00:12:43.742 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.024296113000000001, 2020-01-16T00:12:43.742 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:43.742 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:43.742 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:43.743 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:43.743 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:43.743 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 913 2020-01-16T00:12:43.743 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.743 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:43.743 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.743 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:43.743 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837629+0000", 2020-01-16T00:12:43.744 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:43.744 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.744 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.744 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:43.744 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837629+0000", 2020-01-16T00:12:43.744 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:43.744 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.744 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.745 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:43.745 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837627+0000", 2020-01-16T00:12:43.745 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.329e-06 2020-01-16T00:12:43.745 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.745 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.745 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:43.745 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837637+0000", 2020-01-16T00:12:43.746 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7397e-05 2020-01-16T00:12:43.746 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.746 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.746 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:43.746 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837654+0000", 2020-01-16T00:12:43.746 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6073999999999999e-05 2020-01-16T00:12:43.746 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.746 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.747 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.747 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837670+0000", 2020-01-16T00:12:43.747 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000171781 2020-01-16T00:12:43.747 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.747 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.747 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.748 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837842+0000", 2020-01-16T00:12:43.748 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1575000000000002e-05 2020-01-16T00:12:43.748 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.748 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.749 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:43.749 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837873+0000", 2020-01-16T00:12:43.749 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0095469749999999992 2020-01-16T00:12:43.749 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.749 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.750 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.750 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.847420+0000", 2020-01-16T00:12:43.750 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5698000000000003e-05 2020-01-16T00:12:43.750 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.750 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.751 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:43.751 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.847456+0000", 2020-01-16T00:12:43.751 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013066217 2020-01-16T00:12:43.751 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.752 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.752 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.752 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.860522+0000", 2020-01-16T00:12:43.752 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.2278000000000001e-05 2020-01-16T00:12:43.752 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.753 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.753 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:43.754 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.860605+0000", 2020-01-16T00:12:43.754 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001147779 2020-01-16T00:12:43.755 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.755 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.755 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.755 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.861752+0000", 2020-01-16T00:12:43.755 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4598000000000006e-05 2020-01-16T00:12:43.756 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.756 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.756 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:43.756 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.861827+0000", 2020-01-16T00:12:43.757 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8060999999999995e-05 2020-01-16T00:12:43.757 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.757 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.757 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:43.757 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.861925+0000", 2020-01-16T00:12:43.757 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.024296113000000001 2020-01-16T00:12:43.758 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.758 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:43.758 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.758 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.758 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.758 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:914 3.c 3:3d6543e6:::smithi12313934-28:head [write 426911~540354,stat] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:43.759 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.842336+0000", 2020-01-16T00:12:43.759 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.8446055369999996, 2020-01-16T00:12:43.759 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021922343, 2020-01-16T00:12:43.759 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:43.759 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:43.759 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:43.759 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:43.759 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:43.760 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 914 2020-01-16T00:12:43.760 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.760 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:43.760 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.760 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:43.760 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.842336+0000", 2020-01-16T00:12:43.760 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:43.761 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.761 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.761 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:43.761 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.842336+0000", 2020-01-16T00:12:43.761 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:12:43.761 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.761 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.762 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:43.762 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.842333+0000", 2020-01-16T00:12:43.762 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00063242099999999998 2020-01-16T00:12:43.762 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.762 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.762 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:43.763 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.842965+0000", 2020-01-16T00:12:43.763 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.8354999999999999e-05 2020-01-16T00:12:43.763 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.763 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.763 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:43.763 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.842994+0000", 2020-01-16T00:12:43.763 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1077000000000001e-05 2020-01-16T00:12:43.764 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.764 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.764 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.764 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843005+0000", 2020-01-16T00:12:43.764 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000107552 2020-01-16T00:12:43.764 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.765 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.765 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.765 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843112+0000", 2020-01-16T00:12:43.766 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00037837800000000002 2020-01-16T00:12:43.766 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.766 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.766 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:43.766 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843491+0000", 2020-01-16T00:12:43.766 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0092250490000000008 2020-01-16T00:12:43.766 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.766 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.767 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.767 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.852716+0000", 2020-01-16T00:12:43.767 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018407699999999999 2020-01-16T00:12:43.767 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.767 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.767 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:43.767 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.852900+0000", 2020-01-16T00:12:43.767 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0078355379999999995 2020-01-16T00:12:43.768 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.768 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.768 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.768 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.860735+0000", 2020-01-16T00:12:43.768 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012386100000000001 2020-01-16T00:12:43.768 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.768 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.769 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:43.769 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.860859+0000", 2020-01-16T00:12:43.769 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014735099999999999 2020-01-16T00:12:43.769 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.769 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.769 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2020-01-16T00:12:43.769 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.861006+0000", 2020-01-16T00:12:43.770 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0027785040000000002 2020-01-16T00:12:43.770 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.770 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.770 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:43.770 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.863785+0000", 2020-01-16T00:12:43.770 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000127083 2020-01-16T00:12:43.770 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.770 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.771 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:43.771 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.863912+0000", 2020-01-16T00:12:43.771 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000179106 2020-01-16T00:12:43.771 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.771 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.771 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:43.771 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.864091+0000", 2020-01-16T00:12:43.771 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7744999999999998e-05 2020-01-16T00:12:43.772 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.772 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.772 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:43.772 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.864139+0000", 2020-01-16T00:12:43.772 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011914200000000001 2020-01-16T00:12:43.772 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.772 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.772 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:43.773 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.864258+0000", 2020-01-16T00:12:43.773 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021922343 2020-01-16T00:12:43.773 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.773 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:43.773 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.773 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.773 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.774 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:915 3.c 3:3d6543e6:::smithi12313934-28:head [write 1622009~726683] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:43.774 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.850793+0000", 2020-01-16T00:12:43.774 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.8361483319999996, 2020-01-16T00:12:43.774 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018526131000000001, 2020-01-16T00:12:43.774 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:43.774 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:43.774 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:43.774 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:43.775 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:43.775 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 915 2020-01-16T00:12:43.775 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.775 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:43.775 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.775 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:43.775 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.850793+0000", 2020-01-16T00:12:43.776 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:43.776 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.776 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.776 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:43.776 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.850793+0000", 2020-01-16T00:12:43.776 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999957 2020-01-16T00:12:43.777 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.777 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.777 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:43.777 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.850788+0000", 2020-01-16T00:12:43.777 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0022032459999999999 2020-01-16T00:12:43.778 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.778 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.778 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:43.778 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.852992+0000", 2020-01-16T00:12:43.778 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6056999999999999e-05 2020-01-16T00:12:43.778 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.779 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.779 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:43.779 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.853008+0000", 2020-01-16T00:12:43.779 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5619999999999993e-06 2020-01-16T00:12:43.779 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.779 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.779 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.779 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.853017+0000", 2020-01-16T00:12:43.780 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9026000000000002e-05 2020-01-16T00:12:43.780 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.780 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.780 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.780 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.853056+0000", 2020-01-16T00:12:43.780 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.1551000000000005e-05 2020-01-16T00:12:43.780 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.781 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.781 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:43.781 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.853148+0000", 2020-01-16T00:12:43.781 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0091288280000000003 2020-01-16T00:12:43.781 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.781 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.782 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.782 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.862277+0000", 2020-01-16T00:12:43.782 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000101439 2020-01-16T00:12:43.782 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.782 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.782 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:43.782 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.862378+0000", 2020-01-16T00:12:43.782 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001920597 2020-01-16T00:12:43.783 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.783 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.783 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.783 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.864299+0000", 2020-01-16T00:12:43.783 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000129355 2020-01-16T00:12:43.783 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.783 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.783 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:43.784 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.864428+0000", 2020-01-16T00:12:43.784 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000226999 2020-01-16T00:12:43.784 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.784 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.784 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2020-01-16T00:12:43.784 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.864655+0000", 2020-01-16T00:12:43.784 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0036207840000000002 2020-01-16T00:12:43.784 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.785 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.785 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:43.785 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.868276+0000", 2020-01-16T00:12:43.785 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4215999999999996e-05 2020-01-16T00:12:43.785 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.785 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.785 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:43.786 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.868350+0000", 2020-01-16T00:12:43.786 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00083996500000000003 2020-01-16T00:12:43.786 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.786 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.786 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:43.786 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.869190+0000", 2020-01-16T00:12:43.786 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0606999999999998e-05 2020-01-16T00:12:43.786 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.787 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.787 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:43.787 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.869251+0000", 2020-01-16T00:12:43.787 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8396e-05 2020-01-16T00:12:43.787 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.787 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.787 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:43.787 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.869319+0000", 2020-01-16T00:12:43.788 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018526131000000001 2020-01-16T00:12:43.788 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.788 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:43.788 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.788 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.788 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.788 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:916 3.c 3:3d6543e6:::smithi12313934-28:head [write 3121996~104915] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:43.789 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.855949+0000", 2020-01-16T00:12:43.789 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.8309921920000001, 2020-01-16T00:12:43.789 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013528775, 2020-01-16T00:12:43.789 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:43.789 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:43.789 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:43.789 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:43.790 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:43.790 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 916 2020-01-16T00:12:43.790 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.790 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:43.790 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.790 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:43.790 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.855949+0000", 2020-01-16T00:12:43.791 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:43.791 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.791 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.791 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:43.791 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.855949+0000", 2020-01-16T00:12:43.791 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:43.791 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.791 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.792 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:43.792 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.855947+0000", 2020-01-16T00:12:43.792 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013682200000000001 2020-01-16T00:12:43.792 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.792 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.792 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:43.792 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856084+0000", 2020-01-16T00:12:43.793 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7864999999999999e-05 2020-01-16T00:12:43.793 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.793 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.793 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:43.793 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856102+0000", 2020-01-16T00:12:43.793 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.8699999999999998e-06 2020-01-16T00:12:43.793 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.793 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.794 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.794 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856111+0000", 2020-01-16T00:12:43.794 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.299e-05 2020-01-16T00:12:43.794 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.794 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.794 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.794 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856154+0000", 2020-01-16T00:12:43.794 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0636999999999994e-05 2020-01-16T00:12:43.795 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.795 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.795 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:43.795 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856234+0000", 2020-01-16T00:12:43.795 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0061827920000000003 2020-01-16T00:12:43.795 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.796 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.796 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.797 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.862417+0000", 2020-01-16T00:12:43.797 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010285099999999999 2020-01-16T00:12:43.797 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.798 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.798 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:43.798 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.862520+0000", 2020-01-16T00:12:43.798 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0038991479999999999 2020-01-16T00:12:43.799 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.799 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.799 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.799 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.866419+0000", 2020-01-16T00:12:43.799 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010513700000000001 2020-01-16T00:12:43.800 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.800 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.800 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:43.800 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.866524+0000", 2020-01-16T00:12:43.800 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014972299999999999 2020-01-16T00:12:43.801 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.801 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.801 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2020-01-16T00:12:43.801 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.866674+0000", 2020-01-16T00:12:43.802 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001618415 2020-01-16T00:12:43.802 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.802 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.802 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:43.802 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.868292+0000", 2020-01-16T00:12:43.803 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000297736 2020-01-16T00:12:43.803 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.803 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.803 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:43.803 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.868590+0000", 2020-01-16T00:12:43.804 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000798283 2020-01-16T00:12:43.804 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.804 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.804 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:43.805 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.869388+0000", 2020-01-16T00:12:43.805 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5621000000000002e-05 2020-01-16T00:12:43.805 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.805 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.805 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:43.806 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.869434+0000", 2020-01-16T00:12:43.806 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3894999999999999e-05 2020-01-16T00:12:43.806 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.806 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.806 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:43.806 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.869478+0000", 2020-01-16T00:12:43.806 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013528775 2020-01-16T00:12:43.807 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.807 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:43.807 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.807 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.807 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.807 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:917 3.c 3:3d6543e6:::smithi12313934-28:head [setxattr _header (56) in=7b,truncate 3226911] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:43.807 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.856125+0000", 2020-01-16T00:12:43.808 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.8308159819999998, 2020-01-16T00:12:43.808 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013497781, 2020-01-16T00:12:43.808 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:43.808 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:43.808 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:43.808 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:43.808 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:43.808 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 917 2020-01-16T00:12:43.809 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.809 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:43.809 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.809 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:43.809 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856125+0000", 2020-01-16T00:12:43.809 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:43.809 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.810 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.810 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:43.810 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856125+0000", 2020-01-16T00:12:43.810 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:43.811 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.811 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.811 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:43.811 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856124+0000", 2020-01-16T00:12:43.811 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8859999999999998e-06 2020-01-16T00:12:43.811 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.812 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.812 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:43.812 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856131+0000", 2020-01-16T00:12:43.812 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6570000000000003e-06 2020-01-16T00:12:43.812 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.813 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.813 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:43.813 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856140+0000", 2020-01-16T00:12:43.813 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5420000000000002e-06 2020-01-16T00:12:43.814 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.814 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.814 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.814 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856146+0000", 2020-01-16T00:12:43.814 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012714399999999999 2020-01-16T00:12:43.815 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.816 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.816 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.816 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856273+0000", 2020-01-16T00:12:43.817 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000122113 2020-01-16T00:12:43.817 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.817 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.817 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:43.817 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856395+0000", 2020-01-16T00:12:43.817 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.006160708 2020-01-16T00:12:43.818 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.818 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.818 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.818 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.862556+0000", 2020-01-16T00:12:43.818 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7220999999999999e-05 2020-01-16T00:12:43.818 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.819 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.819 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:43.819 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.862623+0000", 2020-01-16T00:12:43.819 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0046116050000000004 2020-01-16T00:12:43.819 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.819 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.819 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.820 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.867235+0000", 2020-01-16T00:12:43.820 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019234499999999999 2020-01-16T00:12:43.820 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.820 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.820 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:43.820 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.867427+0000", 2020-01-16T00:12:43.820 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022820699999999999 2020-01-16T00:12:43.821 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.821 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.821 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2020-01-16T00:12:43.821 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.867656+0000", 2020-01-16T00:12:43.821 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00065250900000000001 2020-01-16T00:12:43.821 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.822 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.822 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:43.822 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.868308+0000", 2020-01-16T00:12:43.822 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00043943999999999998 2020-01-16T00:12:43.822 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.822 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.822 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:43.823 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.868748+0000", 2020-01-16T00:12:43.823 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00078105600000000002 2020-01-16T00:12:43.823 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.823 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.823 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:43.823 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.869529+0000", 2020-01-16T00:12:43.823 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2976000000000003e-05 2020-01-16T00:12:43.824 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.824 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.824 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:43.824 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.869572+0000", 2020-01-16T00:12:43.824 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1439e-05 2020-01-16T00:12:43.824 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.824 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.825 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:43.825 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.869623+0000", 2020-01-16T00:12:43.825 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013497781 2020-01-16T00:12:43.825 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.825 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:43.825 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.825 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.826 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.826 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:918 3.c 3:3d6543e6:::smithi12313934-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:12:43.826 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.856157+0000", 2020-01-16T00:12:43.826 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.830784156, 2020-01-16T00:12:43.826 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013685233, 2020-01-16T00:12:43.826 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:43.826 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:43.827 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:43.827 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:43.827 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:43.827 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 918 2020-01-16T00:12:43.827 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.827 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:43.827 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.828 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:43.828 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856157+0000", 2020-01-16T00:12:43.828 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:43.828 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.828 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.828 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:43.828 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856157+0000", 2020-01-16T00:12:43.829 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:43.829 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.829 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.829 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:43.829 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856156+0000", 2020-01-16T00:12:43.829 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7450000000000003e-06 2020-01-16T00:12:43.829 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.830 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.830 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:43.830 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856161+0000", 2020-01-16T00:12:43.830 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5880000000000004e-06 2020-01-16T00:12:43.830 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.830 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.830 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:43.831 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856171+0000", 2020-01-16T00:12:43.831 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1620000000000002e-06 2020-01-16T00:12:43.831 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.831 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.831 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.831 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856178+0000", 2020-01-16T00:12:43.831 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000256796 2020-01-16T00:12:43.832 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.832 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.832 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.832 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856435+0000", 2020-01-16T00:12:43.832 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.8992000000000005e-05 2020-01-16T00:12:43.832 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.832 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.833 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:43.833 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856514+0000", 2020-01-16T00:12:43.833 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0061401370000000004 2020-01-16T00:12:43.833 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.833 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.833 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.834 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.862654+0000", 2020-01-16T00:12:43.834 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2589000000000007e-05 2020-01-16T00:12:43.834 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.834 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.834 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:43.834 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.862716+0000", 2020-01-16T00:12:43.834 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0053446409999999998 2020-01-16T00:12:43.835 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.835 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.835 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.835 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.868061+0000", 2020-01-16T00:12:43.835 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019486699999999999 2020-01-16T00:12:43.835 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.835 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.836 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:43.836 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.868256+0000", 2020-01-16T00:12:43.836 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014152170000000001 2020-01-16T00:12:43.836 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.836 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.836 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.836 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.869671+0000", 2020-01-16T00:12:43.837 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.2387999999999998e-05 2020-01-16T00:12:43.837 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.837 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.837 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:43.837 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.869753+0000", 2020-01-16T00:12:43.837 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.8926999999999997e-05 2020-01-16T00:12:43.837 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.838 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.838 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:43.838 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.869842+0000", 2020-01-16T00:12:43.838 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013685233 2020-01-16T00:12:43.838 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.838 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:43.838 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.838 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.839 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.839 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.4.13:16 2.3 2:e19dd0d6:::smithi12313934-14 low_tier:head [read 1798144~391168 [fadvise_sequential+fadvise_dontneed]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e24)", 2020-01-16T00:12:43.839 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.915333+0000", 2020-01-16T00:12:43.839 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.7716086019999997, 2020-01-16T00:12:43.839 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0086441409999999993, 2020-01-16T00:12:43.839 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:43.839 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:43.840 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:43.840 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.840 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:43.840 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.915333+0000", 2020-01-16T00:12:43.840 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:43.840 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.840 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.841 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:43.841 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.915333+0000", 2020-01-16T00:12:43.841 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:43.841 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.841 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.841 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:43.841 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.915332+0000", 2020-01-16T00:12:43.842 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8210000000000004e-06 2020-01-16T00:12:43.842 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.842 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.842 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:43.842 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.915339+0000", 2020-01-16T00:12:43.842 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.8259999999999993e-06 2020-01-16T00:12:43.842 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.842 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.843 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:43.843 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.915347+0000", 2020-01-16T00:12:43.843 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7119999999999999e-06 2020-01-16T00:12:43.843 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.843 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.843 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.844 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.915352+0000", 2020-01-16T00:12:43.844 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0078058870000000001 2020-01-16T00:12:43.844 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.844 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.844 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.844 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.923157+0000", 2020-01-16T00:12:43.844 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4171999999999994e-05 2020-01-16T00:12:43.844 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.845 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.845 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:43.845 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.923232+0000", 2020-01-16T00:12:43.845 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00074513400000000005 2020-01-16T00:12:43.845 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.845 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.845 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:43.846 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.923977+0000", 2020-01-16T00:12:43.846 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0086441409999999993 2020-01-16T00:12:43.846 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.846 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:43.846 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.846 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.846 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.847 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:952 3.c 3:3877922e:::smithi12313934-31:head [write 494139~557703,stat] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:43.847 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.923660+0000", 2020-01-16T00:12:43.847 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.7632813780000003, 2020-01-16T00:12:43.847 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016369574000000001, 2020-01-16T00:12:43.847 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:43.847 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:43.847 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:43.848 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:43.848 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:43.848 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 952 2020-01-16T00:12:43.848 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.848 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:43.848 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.849 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:43.849 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.923660+0000", 2020-01-16T00:12:43.849 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:43.849 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.849 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.849 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:43.849 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.923660+0000", 2020-01-16T00:12:43.850 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:43.850 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.850 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.850 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:43.850 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.923659+0000", 2020-01-16T00:12:43.851 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0015925539999999999 2020-01-16T00:12:43.851 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.851 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.851 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:43.851 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.925251+0000", 2020-01-16T00:12:43.851 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4825e-05 2020-01-16T00:12:43.851 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.851 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.852 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:43.852 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.925276+0000", 2020-01-16T00:12:43.852 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.217e-06 2020-01-16T00:12:43.852 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.852 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.852 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.852 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.925285+0000", 2020-01-16T00:12:43.853 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8985000000000004e-05 2020-01-16T00:12:43.853 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.853 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.853 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.853 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.925354+0000", 2020-01-16T00:12:43.853 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022915199999999999 2020-01-16T00:12:43.854 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.854 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.854 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:43.854 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.925583+0000", 2020-01-16T00:12:43.854 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0072044309999999999 2020-01-16T00:12:43.854 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.855 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.855 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.855 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.932788+0000", 2020-01-16T00:12:43.855 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2225999999999994e-05 2020-01-16T00:12:43.855 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.855 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.855 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:43.856 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.932880+0000", 2020-01-16T00:12:43.856 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0040176580000000003 2020-01-16T00:12:43.856 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.856 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.856 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.856 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.936898+0000", 2020-01-16T00:12:43.856 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9100999999999999e-05 2020-01-16T00:12:43.856 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.857 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.857 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:43.857 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.936997+0000", 2020-01-16T00:12:43.857 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00021074099999999999 2020-01-16T00:12:43.857 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.857 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.858 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2020-01-16T00:12:43.858 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.937207+0000", 2020-01-16T00:12:43.858 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001814979 2020-01-16T00:12:43.858 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.858 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.858 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:43.858 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939022+0000", 2020-01-16T00:12:43.858 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014921699999999999 2020-01-16T00:12:43.859 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.859 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.859 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:43.859 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939172+0000", 2020-01-16T00:12:43.859 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00070305200000000002 2020-01-16T00:12:43.859 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.860 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.860 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:43.860 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939875+0000", 2020-01-16T00:12:43.860 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.9981999999999994e-05 2020-01-16T00:12:43.860 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.860 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.860 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:43.861 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939955+0000", 2020-01-16T00:12:43.861 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4770000000000004e-05 2020-01-16T00:12:43.861 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.861 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.861 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:43.861 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.940029+0000", 2020-01-16T00:12:43.861 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016369574000000001 2020-01-16T00:12:43.862 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.862 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:43.862 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.862 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.862 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.862 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:953 3.c 3:3877922e:::smithi12313934-31:head [write 1455937~734021] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:43.862 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.931235+0000", 2020-01-16T00:12:43.863 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.7557061699999998, 2020-01-16T00:12:43.863 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.022666925000000001, 2020-01-16T00:12:43.863 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:43.863 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:43.863 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:43.863 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:43.863 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:43.864 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 953 2020-01-16T00:12:43.864 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.864 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:43.864 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.864 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:43.864 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.931235+0000", 2020-01-16T00:12:43.864 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:43.865 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.865 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.865 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:43.865 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.931235+0000", 2020-01-16T00:12:43.865 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:43.865 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.866 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.866 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:43.866 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.931233+0000", 2020-01-16T00:12:43.866 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0013294019999999999 2020-01-16T00:12:43.866 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.866 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.866 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:43.867 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.932562+0000", 2020-01-16T00:12:43.867 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0194e-05 2020-01-16T00:12:43.867 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.867 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.867 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:43.867 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.932582+0000", 2020-01-16T00:12:43.867 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.199e-06 2020-01-16T00:12:43.868 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.868 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.868 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.868 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.932589+0000", 2020-01-16T00:12:43.868 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033658000000000001 2020-01-16T00:12:43.868 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.868 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.869 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.869 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.932926+0000", 2020-01-16T00:12:43.869 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000174749 2020-01-16T00:12:43.869 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.869 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.869 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:43.869 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933101+0000", 2020-01-16T00:12:43.870 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0054526330000000001 2020-01-16T00:12:43.870 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.870 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.870 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.870 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.938553+0000", 2020-01-16T00:12:43.870 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00026010999999999997 2020-01-16T00:12:43.870 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.870 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.871 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:43.871 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.938813+0000", 2020-01-16T00:12:43.871 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012019494 2020-01-16T00:12:43.871 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.871 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.871 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.872 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.950833+0000", 2020-01-16T00:12:43.872 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6416999999999997e-05 2020-01-16T00:12:43.872 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.872 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.872 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:43.872 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.950909+0000", 2020-01-16T00:12:43.872 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000152855 2020-01-16T00:12:43.872 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.873 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.873 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2020-01-16T00:12:43.873 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.951062+0000", 2020-01-16T00:12:43.873 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0025050680000000001 2020-01-16T00:12:43.873 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.873 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.873 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:43.874 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953567+0000", 2020-01-16T00:12:43.874 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017207999999999999 2020-01-16T00:12:43.874 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.874 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.874 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:43.874 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953739+0000", 2020-01-16T00:12:43.874 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1077999999999999e-05 2020-01-16T00:12:43.875 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.875 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.875 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:43.875 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953820+0000", 2020-01-16T00:12:43.875 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1516e-05 2020-01-16T00:12:43.875 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.875 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.876 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:43.876 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953862+0000", 2020-01-16T00:12:43.876 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0030999999999997e-05 2020-01-16T00:12:43.876 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.876 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.876 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:43.876 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953902+0000", 2020-01-16T00:12:43.877 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.022666925000000001 2020-01-16T00:12:43.877 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.877 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:43.877 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.877 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.877 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.877 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:954 3.c 3:3877922e:::smithi12313934-31:head [write 2948041~346098] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:43.878 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.940459+0000", 2020-01-16T00:12:43.878 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.7464825079999997, 2020-01-16T00:12:43.878 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023904965, 2020-01-16T00:12:43.878 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:43.878 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:43.878 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:43.878 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:43.879 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:43.879 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 954 2020-01-16T00:12:43.879 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.879 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:43.879 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.879 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:43.879 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.940459+0000", 2020-01-16T00:12:43.880 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:43.880 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.880 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.880 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:43.880 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.940459+0000", 2020-01-16T00:12:43.880 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:43.880 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.880 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.881 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:43.881 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.940457+0000", 2020-01-16T00:12:43.881 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00067923600000000005 2020-01-16T00:12:43.881 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.881 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.881 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:43.882 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941136+0000", 2020-01-16T00:12:43.882 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6631999999999999e-05 2020-01-16T00:12:43.882 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.882 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.882 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:43.882 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941152+0000", 2020-01-16T00:12:43.882 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9340000000000003e-06 2020-01-16T00:12:43.882 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.883 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.883 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.883 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941158+0000", 2020-01-16T00:12:43.883 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8878000000000001e-05 2020-01-16T00:12:43.883 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.883 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.883 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.884 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941217+0000", 2020-01-16T00:12:43.884 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1224999999999997e-05 2020-01-16T00:12:43.884 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.884 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.884 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:43.884 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941289+0000", 2020-01-16T00:12:43.884 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011199931999999999 2020-01-16T00:12:43.885 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.885 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.885 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.885 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.952488+0000", 2020-01-16T00:12:43.885 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000240067 2020-01-16T00:12:43.885 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.885 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.885 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:43.886 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.952729+0000", 2020-01-16T00:12:43.886 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0094184660000000003 2020-01-16T00:12:43.886 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.886 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.886 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.886 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.962147+0000", 2020-01-16T00:12:43.886 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6094000000000006e-05 2020-01-16T00:12:43.887 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.887 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.887 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:43.887 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.962233+0000", 2020-01-16T00:12:43.887 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022739400000000001 2020-01-16T00:12:43.887 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.888 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.888 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2020-01-16T00:12:43.888 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.962460+0000", 2020-01-16T00:12:43.888 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014799710000000001 2020-01-16T00:12:43.888 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.888 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.888 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:43.889 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.963940+0000", 2020-01-16T00:12:43.889 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000279182 2020-01-16T00:12:43.889 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.889 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.889 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:43.889 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.964220+0000", 2020-01-16T00:12:43.889 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4141999999999997e-05 2020-01-16T00:12:43.890 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.890 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.890 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:43.890 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.964284+0000", 2020-01-16T00:12:43.890 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3547999999999997e-05 2020-01-16T00:12:43.890 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.893 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.893 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:43.893 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.964327+0000", 2020-01-16T00:12:43.893 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6341000000000003e-05 2020-01-16T00:12:43.893 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.893 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.894 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:43.894 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.964364+0000", 2020-01-16T00:12:43.894 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023904965 2020-01-16T00:12:43.894 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.894 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:43.894 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.894 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.895 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.895 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:955 3.c 3:3877922e:::smithi12313934-31:head [setxattr _header (56) in=7b,truncate 3294139] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:43.895 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.941169+0000", 2020-01-16T00:12:43.895 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.7457724609999996, 2020-01-16T00:12:43.895 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023400259999999999, 2020-01-16T00:12:43.896 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:43.896 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:43.896 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:43.896 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:43.896 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:43.896 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 955 2020-01-16T00:12:43.896 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.896 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:43.897 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.897 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:43.897 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941169+0000", 2020-01-16T00:12:43.897 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:43.897 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.897 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.897 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:43.898 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941169+0000", 2020-01-16T00:12:43.898 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:43.898 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.898 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.898 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:43.898 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941168+0000", 2020-01-16T00:12:43.899 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.6700000000000006e-06 2020-01-16T00:12:43.899 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.899 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.899 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:43.899 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941177+0000", 2020-01-16T00:12:43.899 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6420000000000001e-06 2020-01-16T00:12:43.899 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.899 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.900 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:43.900 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941184+0000", 2020-01-16T00:12:43.900 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5019999999999998e-06 2020-01-16T00:12:43.900 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.900 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.900 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.901 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941190+0000", 2020-01-16T00:12:43.901 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013520700000000001 2020-01-16T00:12:43.901 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.901 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.901 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.901 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941326+0000", 2020-01-16T00:12:43.901 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1218999999999998e-05 2020-01-16T00:12:43.902 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.902 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.902 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:43.902 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941387+0000", 2020-01-16T00:12:43.902 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011403931000000001 2020-01-16T00:12:43.902 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.902 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.903 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.903 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.952791+0000", 2020-01-16T00:12:43.903 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6416999999999998e-05 2020-01-16T00:12:43.903 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.903 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.903 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:43.903 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.952857+0000", 2020-01-16T00:12:43.903 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010427334999999999 2020-01-16T00:12:43.904 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.904 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.904 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.904 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.963284+0000", 2020-01-16T00:12:43.904 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.9720000000000002e-05 2020-01-16T00:12:43.904 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.904 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.905 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:43.905 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.963374+0000", 2020-01-16T00:12:43.905 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000131137 2020-01-16T00:12:43.905 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.905 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.905 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2020-01-16T00:12:43.905 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.963505+0000", 2020-01-16T00:12:43.906 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00065508899999999998 2020-01-16T00:12:43.906 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.906 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.906 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:43.906 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.964160+0000", 2020-01-16T00:12:43.906 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000279265 2020-01-16T00:12:43.906 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.907 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.907 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:43.907 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.964440+0000", 2020-01-16T00:12:43.907 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5080999999999997e-05 2020-01-16T00:12:43.907 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.907 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.907 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:43.908 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.964485+0000", 2020-01-16T00:12:43.908 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0565999999999998e-05 2020-01-16T00:12:43.908 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.908 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.908 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:43.908 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.964525+0000", 2020-01-16T00:12:43.909 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3662999999999999e-05 2020-01-16T00:12:43.909 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.909 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.909 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:43.909 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.964569+0000", 2020-01-16T00:12:43.909 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023400259999999999 2020-01-16T00:12:43.909 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.910 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:43.910 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.910 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.910 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.910 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:956 3.c 3:3877922e:::smithi12313934-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:12:43.910 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.941200+0000", 2020-01-16T00:12:43.910 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.7457416090000004, 2020-01-16T00:12:43.911 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023677922000000001, 2020-01-16T00:12:43.911 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:43.911 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:43.911 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:43.911 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:43.911 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:43.911 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 956 2020-01-16T00:12:43.912 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.912 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:43.912 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.912 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:43.912 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941200+0000", 2020-01-16T00:12:43.912 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:43.912 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.913 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.913 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:43.913 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941200+0000", 2020-01-16T00:12:43.913 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:43.913 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.913 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.913 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:43.914 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941199+0000", 2020-01-16T00:12:43.914 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9330000000000003e-06 2020-01-16T00:12:43.914 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.914 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.914 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:43.914 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941203+0000", 2020-01-16T00:12:43.914 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1578000000000002e-05 2020-01-16T00:12:43.914 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.915 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.915 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:43.915 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941234+0000", 2020-01-16T00:12:43.915 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2530000000000004e-06 2020-01-16T00:12:43.915 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.915 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.916 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.916 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941240+0000", 2020-01-16T00:12:43.916 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016458900000000001 2020-01-16T00:12:43.916 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.916 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.916 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.916 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941404+0000", 2020-01-16T00:12:43.916 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3485000000000002e-05 2020-01-16T00:12:43.917 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.917 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.917 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:43.917 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941438+0000", 2020-01-16T00:12:43.917 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011489497 2020-01-16T00:12:43.917 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.917 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.918 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.918 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.952927+0000", 2020-01-16T00:12:43.918 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4110999999999999e-05 2020-01-16T00:12:43.918 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.918 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.918 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:43.918 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.952961+0000", 2020-01-16T00:12:43.919 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010832766000000001 2020-01-16T00:12:43.919 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.919 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.919 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.919 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.963794+0000", 2020-01-16T00:12:43.919 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012301900000000001 2020-01-16T00:12:43.920 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.920 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.920 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:43.920 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.963917+0000", 2020-01-16T00:12:43.920 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00069668799999999999 2020-01-16T00:12:43.920 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.920 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.921 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.921 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.964614+0000", 2020-01-16T00:12:43.921 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012451500000000001 2020-01-16T00:12:43.921 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.921 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.921 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:43.921 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.964738+0000", 2020-01-16T00:12:43.922 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000139128 2020-01-16T00:12:43.922 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.922 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.922 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:43.922 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.964878+0000", 2020-01-16T00:12:43.922 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023677922000000001 2020-01-16T00:12:43.922 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.923 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:43.923 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.923 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.923 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.923 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:979 3.5 3:ad8ab15f:::smithi12313934-27:head [read 0~3210104,checksum] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:43.923 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315954+0000", 2020-01-16T00:12:43.923 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.3709874879999999, 2020-01-16T00:12:43.924 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010639968, 2020-01-16T00:12:43.924 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:43.924 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:43.924 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:43.924 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:43.924 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:43.924 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 979 2020-01-16T00:12:43.925 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.925 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:43.925 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.925 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:43.925 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315954+0000", 2020-01-16T00:12:43.925 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:43.925 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.926 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.926 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:43.926 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315954+0000", 2020-01-16T00:12:43.926 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:43.926 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.926 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.926 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:43.927 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315953+0000", 2020-01-16T00:12:43.927 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.1200000000000008e-06 2020-01-16T00:12:43.927 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.927 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.927 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:43.927 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315962+0000", 2020-01-16T00:12:43.928 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.488e-05 2020-01-16T00:12:43.928 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.928 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.928 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:43.928 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315977+0000", 2020-01-16T00:12:43.928 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6283e-05 2020-01-16T00:12:43.929 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.929 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.929 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.929 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315993+0000", 2020-01-16T00:12:43.929 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0079730960000000007 2020-01-16T00:12:43.929 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.930 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.930 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.930 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.323966+0000", 2020-01-16T00:12:43.930 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020471399999999999 2020-01-16T00:12:43.930 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.930 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.931 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:43.931 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.324171+0000", 2020-01-16T00:12:43.931 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0024230290000000002 2020-01-16T00:12:43.931 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.931 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.931 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:43.931 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.326594+0000", 2020-01-16T00:12:43.932 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010639968 2020-01-16T00:12:43.932 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.932 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:43.932 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.932 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.932 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.932 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:980 3.5 3:ad8ab15f:::smithi12313934-27:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:43.933 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.316004+0000", 2020-01-16T00:12:43.933 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.3709370759999997, 2020-01-16T00:12:43.933 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011373866, 2020-01-16T00:12:43.933 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:43.933 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:43.933 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:43.933 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:43.934 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:43.934 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 980 2020-01-16T00:12:43.934 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.934 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:43.934 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.934 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:43.934 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316004+0000", 2020-01-16T00:12:43.935 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:43.935 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.935 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.935 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:43.935 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316004+0000", 2020-01-16T00:12:43.935 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:43.935 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.936 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.936 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:43.936 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316003+0000", 2020-01-16T00:12:43.936 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.9419999999999998e-06 2020-01-16T00:12:43.936 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.936 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.936 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:43.936 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316010+0000", 2020-01-16T00:12:43.937 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4960000000000001e-05 2020-01-16T00:12:43.937 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.937 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.937 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:43.937 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316025+0000", 2020-01-16T00:12:43.937 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2890000000000003e-06 2020-01-16T00:12:43.938 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.938 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.938 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.938 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316034+0000", 2020-01-16T00:12:43.938 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010653673000000001 2020-01-16T00:12:43.938 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.938 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.939 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.939 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.326688+0000", 2020-01-16T00:12:43.939 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000231511 2020-01-16T00:12:43.939 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.939 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.939 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:43.940 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.326919+0000", 2020-01-16T00:12:43.940 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00045858800000000001 2020-01-16T00:12:43.940 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.940 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.940 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:43.940 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.327378+0000", 2020-01-16T00:12:43.941 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011373866 2020-01-16T00:12:43.941 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.941 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:43.941 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.941 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.941 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.942 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.1.13:47 2.3 2:c6fec7d0:::smithi12313934-26 low_tier:head [write 963072~376320] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e26)", 2020-01-16T00:12:43.942 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:41.673013+0000", 2020-01-16T00:12:43.942 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 2.013927936, 2020-01-16T00:12:43.942 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0095009329999999996, 2020-01-16T00:12:43.942 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:43.942 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:43.942 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:43.943 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.943 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:43.943 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.673013+0000", 2020-01-16T00:12:43.943 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:43.943 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.943 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.943 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:43.944 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.673013+0000", 2020-01-16T00:12:43.944 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:43.944 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.944 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.944 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:43.944 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.673011+0000", 2020-01-16T00:12:43.944 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000590852 2020-01-16T00:12:43.945 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.945 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.945 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:43.945 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.673602+0000", 2020-01-16T00:12:43.945 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9289999999999998e-05 2020-01-16T00:12:43.945 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.945 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.946 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:43.946 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.673641+0000", 2020-01-16T00:12:43.946 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0559e-05 2020-01-16T00:12:43.946 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.946 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.946 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.946 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.673652+0000", 2020-01-16T00:12:43.947 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5334999999999997e-05 2020-01-16T00:12:43.947 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.947 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.947 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.947 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.673717+0000", 2020-01-16T00:12:43.947 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000109041 2020-01-16T00:12:43.947 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.948 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.948 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:43.948 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.673826+0000", 2020-01-16T00:12:43.948 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015066 2020-01-16T00:12:43.948 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.948 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.948 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:12:43.949 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.673977+0000", 2020-01-16T00:12:43.949 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0082881929999999993 2020-01-16T00:12:43.949 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.949 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.949 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:43.949 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.682265+0000", 2020-01-16T00:12:43.949 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.9645000000000002e-05 2020-01-16T00:12:43.950 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.950 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.950 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:43.950 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.682334+0000", 2020-01-16T00:12:43.950 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5344000000000001e-05 2020-01-16T00:12:43.951 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.951 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.951 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:43.951 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.682350+0000", 2020-01-16T00:12:43.951 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6589999999999997e-05 2020-01-16T00:12:43.951 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.951 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.952 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:43.952 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.682426+0000", 2020-01-16T00:12:43.952 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7816000000000001e-05 2020-01-16T00:12:43.952 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.952 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.952 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:43.952 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.682514+0000", 2020-01-16T00:12:43.952 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0095009329999999996 2020-01-16T00:12:43.953 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.953 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:43.953 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.953 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.953 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.953 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.1.13:48 2.3 2:c6fec7d0:::smithi12313934-26 low_tier:head [write 1494016~315392] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e26)", 2020-01-16T00:12:43.954 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:41.673668+0000", 2020-01-16T00:12:43.954 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 2.0132728150000001, 2020-01-16T00:12:43.954 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0090818140000000006, 2020-01-16T00:12:43.954 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:43.954 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:43.954 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:43.954 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.955 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:43.955 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.673668+0000", 2020-01-16T00:12:43.955 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:43.955 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.955 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.955 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:43.955 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.673668+0000", 2020-01-16T00:12:43.956 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:43.956 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.956 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.956 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:43.956 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.673667+0000", 2020-01-16T00:12:43.956 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00031002999999999998 2020-01-16T00:12:43.957 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.957 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.957 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:43.957 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.673977+0000", 2020-01-16T00:12:43.957 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8076000000000001e-05 2020-01-16T00:12:43.957 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.957 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.958 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:43.958 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.673995+0000", 2020-01-16T00:12:43.958 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9789999999999998e-06 2020-01-16T00:12:43.958 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.958 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.958 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.959 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.674001+0000", 2020-01-16T00:12:43.959 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0017961819999999999 2020-01-16T00:12:43.959 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.959 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.959 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.959 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.675797+0000", 2020-01-16T00:12:43.959 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013250299999999999 2020-01-16T00:12:43.960 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.960 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.960 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:43.960 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.675930+0000", 2020-01-16T00:12:43.960 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022845999999999999 2020-01-16T00:12:43.960 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.961 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.961 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:12:43.961 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.676158+0000", 2020-01-16T00:12:43.961 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0063733770000000004 2020-01-16T00:12:43.961 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.961 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.961 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:43.962 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.682532+0000", 2020-01-16T00:12:43.962 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3147999999999999e-05 2020-01-16T00:12:43.962 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.962 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.962 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:43.962 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.682595+0000", 2020-01-16T00:12:43.962 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3842999999999999e-05 2020-01-16T00:12:43.962 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.963 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.963 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:43.963 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.682649+0000", 2020-01-16T00:12:43.963 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2639000000000003e-05 2020-01-16T00:12:43.963 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.963 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.963 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:43.964 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.682691+0000", 2020-01-16T00:12:43.964 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8822e-05 2020-01-16T00:12:43.964 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.964 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.964 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:43.964 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.682750+0000", 2020-01-16T00:12:43.964 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0090818140000000006 2020-01-16T00:12:43.965 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.965 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:43.965 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.965 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.965 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.965 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.1.13:49 2.3 2:c6fec7d0:::smithi12313934-26 low_tier:head [write 1965056~33280] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e26)", 2020-01-16T00:12:43.965 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:41.674014+0000", 2020-01-16T00:12:43.966 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 2.0129267350000002, 2020-01-16T00:12:43.966 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0090726620000000004, 2020-01-16T00:12:43.966 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:43.966 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:43.966 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:43.966 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.966 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:43.967 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.674014+0000", 2020-01-16T00:12:43.967 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:43.967 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.967 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.967 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:43.967 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.674014+0000", 2020-01-16T00:12:43.967 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:43.968 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.968 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.968 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:43.968 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.674014+0000", 2020-01-16T00:12:43.968 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9543000000000002e-05 2020-01-16T00:12:43.968 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.968 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.969 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:43.969 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.674043+0000", 2020-01-16T00:12:43.969 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2354999999999999e-05 2020-01-16T00:12:43.969 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.969 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.969 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:43.969 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.674055+0000", 2020-01-16T00:12:43.970 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0391000000000001e-05 2020-01-16T00:12:43.970 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.970 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.970 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:43.970 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.674086+0000", 2020-01-16T00:12:43.970 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003699777 2020-01-16T00:12:43.970 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.971 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.971 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:43.971 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.677786+0000", 2020-01-16T00:12:43.971 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000101382 2020-01-16T00:12:43.971 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.971 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.971 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:43.972 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.677887+0000", 2020-01-16T00:12:43.972 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000205923 2020-01-16T00:12:43.972 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.972 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.972 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:12:43.972 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.678093+0000", 2020-01-16T00:12:43.972 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.004722955 2020-01-16T00:12:43.973 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.973 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.973 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:43.973 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.682816+0000", 2020-01-16T00:12:43.973 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4629999999999998e-05 2020-01-16T00:12:43.973 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.973 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.974 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:43.974 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.682860+0000", 2020-01-16T00:12:43.974 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000144676 2020-01-16T00:12:43.974 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.974 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.974 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:43.974 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.683005+0000", 2020-01-16T00:12:43.975 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4332999999999997e-05 2020-01-16T00:12:43.975 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.975 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.975 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:43.975 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.683049+0000", 2020-01-16T00:12:43.975 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7638000000000001e-05 2020-01-16T00:12:43.975 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:43.976 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:43.976 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:43.976 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.683087+0000", 2020-01-16T00:12:43.976 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0090726620000000004 2020-01-16T00:12:43.976 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.976 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:43.976 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.977 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:43.977 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:43.977 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:43.977 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-01-16T00:12:43.979 INFO:tasks.rados.rados.0.smithi123.stdout:735: expect (ObjNum 63 snap 0 seq_num 63) 2020-01-16T00:12:43.979 INFO:tasks.rados.rados.0.smithi123.stdout:737: expect (ObjNum 107 snap 0 seq_num 107) 2020-01-16T00:12:43.980 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:43.981+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:12:43.980 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-01-16T00:12:44.080 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:44.081+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:44.110 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:12:44.129 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:12:44.138 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:44.138 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:12:44.138 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:12:44.138 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:44.147 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-01-16T00:12:44.149 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:12:44.156 INFO:tasks.rados.rados.0.smithi123.stdout:738: finishing write tid 1 to smithi12313934-12 2020-01-16T00:12:44.156 INFO:tasks.rados.rados.0.smithi123.stdout:738: finishing write tid 2 to smithi12313934-12 2020-01-16T00:12:44.156 INFO:tasks.rados.rados.0.smithi123.stdout:738: finishing write tid 4 to smithi12313934-12 2020-01-16T00:12:44.156 INFO:tasks.rados.rados.0.smithi123.stdout:738: finishing write tid 5 to smithi12313934-12 2020-01-16T00:12:44.156 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 12 v 48 (ObjNum 121 snap 0 seq_num 121) dirty exists 2020-01-16T00:12:44.156 INFO:tasks.rados.rados.0.smithi123.stdout:738: left oid 12 (ObjNum 121 snap 0 seq_num 121) 2020-01-16T00:12:44.156 INFO:tasks.rados.rados.0.smithi123.stdout:736: expect (ObjNum 115 snap 0 seq_num 115) 2020-01-16T00:12:44.169 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:12:44.191 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:12:44.192 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:44.193+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:12:44.213 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5: {} 2020-01-16T00:12:44.242 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:12:44.261 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:12:44.265 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-01-16T00:12:44.265 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-01-16T00:12:44.265 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-01-16T00:12:44.265 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-01-16T00:12:44.265 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-01-16T00:12:44.265 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-01-16T00:12:44.266 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-01-16T00:12:44.266 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-01-16T00:12:44.282 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:12:44.281+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:12:44.302 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:44.311 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-01-16T00:12:44.323 INFO:tasks.rados.rados.0.smithi123.stdout:739: expect (ObjNum 48 snap 0 seq_num 48) 2020-01-16T00:12:44.401 INFO:tasks.rados.rados.0.smithi123.stdout:741: expect (ObjNum 41 snap 0 seq_num 41) 2020-01-16T00:12:44.403 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:12:44.405+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:12:44.467 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:44.476 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-01-16T00:12:44.477 INFO:tasks.rados.rados.0.smithi123.stdout:740: expect (ObjNum 60 snap 0 seq_num 60) 2020-01-16T00:12:44.520 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:12:44.521+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:12:44.585 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:12:44.587+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:12:44.598 INFO:tasks.rados.rados.0.smithi123.stdout:743: finishing write tid 1 to smithi12313934-37 2020-01-16T00:12:44.599 INFO:tasks.rados.rados.0.smithi123.stdout:743: finishing write tid 2 to smithi12313934-37 2020-01-16T00:12:44.599 INFO:tasks.rados.rados.0.smithi123.stdout:743: finishing write tid 4 to smithi12313934-37 2020-01-16T00:12:44.599 INFO:tasks.rados.rados.0.smithi123.stdout:743: finishing write tid 5 to smithi12313934-37 2020-01-16T00:12:44.599 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 37 v 139 (ObjNum 122 snap 0 seq_num 122) dirty exists 2020-01-16T00:12:44.599 INFO:tasks.rados.rados.0.smithi123.stdout:743: left oid 37 (ObjNum 122 snap 0 seq_num 122) 2020-01-16T00:12:44.600 INFO:tasks.rados.rados.0.smithi123.stdout:742: expect (ObjNum 112 snap 0 seq_num 112) 2020-01-16T00:12:44.629 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:44.638 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-01-16T00:12:44.686 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:12:44.687+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:12:44.745 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:44.758 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-01-16T00:12:44.807 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:44.809+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:12:44.861 INFO:tasks.rados.rados.0.smithi123.stdout:727: done (15 left) 2020-01-16T00:12:44.861 INFO:tasks.rados.rados.0.smithi123.stdout:729: done (14 left) 2020-01-16T00:12:44.862 INFO:tasks.rados.rados.0.smithi123.stdout:730: done (13 left) 2020-01-16T00:12:44.862 INFO:tasks.rados.rados.0.smithi123.stdout:731: done (12 left) 2020-01-16T00:12:44.862 INFO:tasks.rados.rados.0.smithi123.stdout:732: done (11 left) 2020-01-16T00:12:44.862 INFO:tasks.rados.rados.0.smithi123.stdout:733: done (10 left) 2020-01-16T00:12:44.862 INFO:tasks.rados.rados.0.smithi123.stdout:734: done (9 left) 2020-01-16T00:12:44.862 INFO:tasks.rados.rados.0.smithi123.stdout:735: done (8 left) 2020-01-16T00:12:44.862 INFO:tasks.rados.rados.0.smithi123.stdout:736: done (7 left) 2020-01-16T00:12:44.863 INFO:tasks.rados.rados.0.smithi123.stdout:737: done (6 left) 2020-01-16T00:12:44.863 INFO:tasks.rados.rados.0.smithi123.stdout:738: done (5 left) 2020-01-16T00:12:44.864 INFO:tasks.rados.rados.0.smithi123.stdout:739: done (4 left) 2020-01-16T00:12:44.864 INFO:tasks.rados.rados.0.smithi123.stdout:740: done (3 left) 2020-01-16T00:12:44.864 INFO:tasks.rados.rados.0.smithi123.stdout:741: done (2 left) 2020-01-16T00:12:44.864 INFO:tasks.rados.rados.0.smithi123.stdout:742: done (1 left) 2020-01-16T00:12:44.864 INFO:tasks.rados.rados.0.smithi123.stdout:743: done (0 left) 2020-01-16T00:12:44.864 INFO:tasks.rados.rados.0.smithi123.stdout:744: write (excl) oid 22 current snap is 0 2020-01-16T00:12:44.864 INFO:tasks.rados.rados.0.smithi123.stdout:744: seq_num 123 ranges {467261=717275,1847991=219270} 2020-01-16T00:12:44.867 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:44.875 INFO:tasks.rados.rados.0.smithi123.stdout:744: writing smithi12313934-22 from 467261 to 1184536 tid 1 2020-01-16T00:12:44.877 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-01-16T00:12:44.881 INFO:tasks.rados.rados.0.smithi123.stdout:744: writing smithi12313934-22 from 1847991 to 2067261 tid 2 2020-01-16T00:12:44.881 INFO:tasks.rados.rados.0.smithi123.stdout:745: read oid 2 snap -1 2020-01-16T00:12:44.881 INFO:tasks.rados.rados.0.smithi123.stdout:745: expect (ObjNum 63 snap 0 seq_num 63) 2020-01-16T00:12:44.881 INFO:tasks.rados.rados.0.smithi123.stdout:746: write oid 13 current snap is 0 2020-01-16T00:12:44.881 INFO:tasks.rados.rados.0.smithi123.stdout:746: seq_num 124 ranges {484068=792524,1906473=177595} 2020-01-16T00:12:44.888 INFO:tasks.rados.rados.0.smithi123.stdout:746: writing smithi12313934-13 from 484068 to 1276592 tid 1 2020-01-16T00:12:44.893 INFO:tasks.rados.rados.0.smithi123.stdout:746: writing smithi12313934-13 from 1906473 to 2084068 tid 2 2020-01-16T00:12:44.893 INFO:tasks.rados.rados.0.smithi123.stdout:747: write (excl) oid 10 current snap is 0 2020-01-16T00:12:44.894 INFO:tasks.rados.rados.0.smithi123.stdout:747: seq_num 125 ranges {500875=467773,1564955=535920} 2020-01-16T00:12:44.899 INFO:tasks.rados.rados.0.smithi123.stdout:747: writing smithi12313934-10 from 500875 to 968648 tid 1 2020-01-16T00:12:44.903 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2020-01-16T00:12:44.906 INFO:tasks.rados.rados.0.smithi123.stdout:747: writing smithi12313934-10 from 1564955 to 2100875 tid 2 2020-01-16T00:12:44.906 INFO:tasks.rados.rados.0.smithi123.stdout:748: write (excl) oid 33 current snap is 0 2020-01-16T00:12:44.906 INFO:tasks.rados.rados.0.smithi123.stdout:748: seq_num 126 ranges {517682=543022,1623437=494245} 2020-01-16T00:12:44.913 INFO:tasks.rados.rados.0.smithi123.stdout:748: writing smithi12313934-33 from 517682 to 1060704 tid 1 2020-01-16T00:12:44.920 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:44.921+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:44.920 INFO:tasks.rados.rados.0.smithi123.stdout:748: writing smithi12313934-33 from 1623437 to 2117682 tid 2 2020-01-16T00:12:44.920 INFO:tasks.rados.rados.0.smithi123.stdout:749: write (excl) oid 42 current snap is 0 2020-01-16T00:12:44.920 INFO:tasks.rados.rados.0.smithi123.stdout:749: seq_num 127 ranges {534489=618271,1565566=568923} 2020-01-16T00:12:44.926 INFO:tasks.rados.rados.0.smithi123.stdout:749: writing smithi12313934-42 from 534489 to 1152760 tid 1 2020-01-16T00:12:44.933 INFO:tasks.rados.rados.0.smithi123.stdout:749: writing smithi12313934-42 from 1565566 to 2134489 tid 2 2020-01-16T00:12:44.933 INFO:tasks.rados.rados.0.smithi123.stdout:750: read oid 47 snap -1 2020-01-16T00:12:44.933 INFO:tasks.rados.rados.0.smithi123.stdout:750: expect (ObjNum 101 snap 0 seq_num 101) 2020-01-16T00:12:44.934 INFO:tasks.rados.rados.0.smithi123.stdout:751: read oid 15 snap -1 2020-01-16T00:12:44.934 INFO:tasks.rados.rados.0.smithi123.stdout:751: expect (ObjNum 59 snap 0 seq_num 59) 2020-01-16T00:12:44.934 INFO:tasks.rados.rados.0.smithi123.stdout:752: read oid 36 snap -1 2020-01-16T00:12:44.934 INFO:tasks.rados.rados.0.smithi123.stdout:752: expect (ObjNum 94 snap 0 seq_num 94) 2020-01-16T00:12:44.934 INFO:tasks.rados.rados.0.smithi123.stdout:753: read oid 50 snap -1 2020-01-16T00:12:44.934 INFO:tasks.rados.rados.0.smithi123.stdout:753: expect (ObjNum 67 snap 0 seq_num 67) 2020-01-16T00:12:44.934 INFO:tasks.rados.rados.0.smithi123.stdout:754: read oid 29 snap -1 2020-01-16T00:12:44.935 INFO:tasks.rados.rados.0.smithi123.stdout:754: expect (ObjNum 28 snap 0 seq_num 28) 2020-01-16T00:12:44.935 INFO:tasks.rados.rados.0.smithi123.stdout:755: write (excl) oid 25 current snap is 0 2020-01-16T00:12:44.935 INFO:tasks.rados.rados.0.smithi123.stdout:755: seq_num 128 ranges {551296=693520,2024048=127248} 2020-01-16T00:12:44.941 INFO:tasks.rados.rados.0.smithi123.stdout:755: writing smithi12313934-25 from 551296 to 1244816 tid 1 2020-01-16T00:12:44.947 INFO:tasks.rados.rados.0.smithi123.stdout:755: writing smithi12313934-25 from 2024048 to 2151296 tid 2 2020-01-16T00:12:44.947 INFO:tasks.rados.rados.0.smithi123.stdout:756: write (excl) oid 14 current snap is 0 2020-01-16T00:12:44.947 INFO:tasks.rados.rados.0.smithi123.stdout:756: seq_num 129 ranges {568103=768769,2082530=85573} 2020-01-16T00:12:44.955 INFO:tasks.rados.rados.0.smithi123.stdout:756: writing smithi12313934-14 from 568103 to 1336872 tid 1 2020-01-16T00:12:44.961 INFO:tasks.rados.rados.0.smithi123.stdout:756: writing smithi12313934-14 from 2082530 to 2168103 tid 2 2020-01-16T00:12:44.961 INFO:tasks.rados.rados.0.smithi123.stdout:757: read oid 12 snap -1 2020-01-16T00:12:44.961 INFO:tasks.rados.rados.0.smithi123.stdout:757: expect (ObjNum 121 snap 0 seq_num 121) 2020-01-16T00:12:44.961 INFO:tasks.rados.rados.0.smithi123.stdout:758: read oid 1 snap -1 2020-01-16T00:12:44.961 INFO:tasks.rados.rados.0.smithi123.stdout:758: expect (ObjNum 83 snap 0 seq_num 83) 2020-01-16T00:12:44.962 INFO:tasks.rados.rados.0.smithi123.stdout:759: write (excl) oid 38 current snap is 0 2020-01-16T00:12:44.962 INFO:tasks.rados.rados.0.smithi123.stdout:759: seq_num 130 ranges {584910=560371,1857365=327545} 2020-01-16T00:12:44.967 INFO:tasks.rados.rados.0.smithi123.stdout:759: writing smithi12313934-38 from 584910 to 1145281 tid 1 2020-01-16T00:12:44.974 INFO:tasks.rados.rados.0.smithi123.stdout:759: writing smithi12313934-38 from 1857365 to 2184910 tid 2 2020-01-16T00:12:44.974 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:44.974 INFO:tasks.rados.rados.0.smithi123.stdout:744: finishing write tid 1 to smithi12313934-22 2020-01-16T00:12:44.975 INFO:tasks.rados.rados.0.smithi123.stdout:744: finishing write tid 2 to smithi12313934-22 2020-01-16T00:12:44.975 INFO:tasks.rados.rados.0.smithi123.stdout:744: finishing write tid 4 to smithi12313934-22 2020-01-16T00:12:44.975 INFO:tasks.rados.rados.0.smithi123.stdout:744: finishing write tid 5 to smithi12313934-22 2020-01-16T00:12:44.975 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 22 v 22 (ObjNum 123 snap 0 seq_num 123) dirty exists 2020-01-16T00:12:44.975 INFO:tasks.rados.rados.0.smithi123.stdout:744: left oid 22 (ObjNum 123 snap 0 seq_num 123) 2020-01-16T00:12:44.975 INFO:tasks.rados.rados.0.smithi123.stdout:745: expect (ObjNum 63 snap 0 seq_num 63) 2020-01-16T00:12:45.001 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:12:45.001+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:12:45.027 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:45.034 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-01-16T00:12:45.121 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:12:45.121+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:12:45.152 INFO:tasks.rados.rados.0.smithi123.stdout:746: finishing write tid 1 to smithi12313934-13 2020-01-16T00:12:45.153 INFO:tasks.rados.rados.0.smithi123.stdout:746: finishing write tid 2 to smithi12313934-13 2020-01-16T00:12:45.153 INFO:tasks.rados.rados.0.smithi123.stdout:747: finishing write tid 1 to smithi12313934-10 2020-01-16T00:12:45.153 INFO:tasks.rados.rados.0.smithi123.stdout:746: finishing write tid 4 to smithi12313934-13 2020-01-16T00:12:45.153 INFO:tasks.rados.rados.0.smithi123.stdout:746: finishing write tid 5 to smithi12313934-13 2020-01-16T00:12:45.153 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 36 (ObjNum 124 snap 0 seq_num 124) dirty exists 2020-01-16T00:12:45.153 INFO:tasks.rados.rados.0.smithi123.stdout:746: left oid 13 (ObjNum 124 snap 0 seq_num 124) 2020-01-16T00:12:45.153 INFO:tasks.rados.rados.0.smithi123.stdout:747: finishing write tid 2 to smithi12313934-10 2020-01-16T00:12:45.154 INFO:tasks.rados.rados.0.smithi123.stdout:748: finishing write tid 1 to smithi12313934-33 2020-01-16T00:12:45.154 INFO:tasks.rados.rados.0.smithi123.stdout:747: finishing write tid 4 to smithi12313934-10 2020-01-16T00:12:45.154 INFO:tasks.rados.rados.0.smithi123.stdout:747: finishing write tid 5 to smithi12313934-10 2020-01-16T00:12:45.154 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 10 v 63 (ObjNum 125 snap 0 seq_num 125) dirty exists 2020-01-16T00:12:45.154 INFO:tasks.rados.rados.0.smithi123.stdout:747: left oid 10 (ObjNum 125 snap 0 seq_num 125) 2020-01-16T00:12:45.154 INFO:tasks.rados.rados.0.smithi123.stdout:748: finishing write tid 2 to smithi12313934-33 2020-01-16T00:12:45.154 INFO:tasks.rados.rados.0.smithi123.stdout:749: finishing write tid 1 to smithi12313934-42 2020-01-16T00:12:45.155 INFO:tasks.rados.rados.0.smithi123.stdout:744: done (15 left) 2020-01-16T00:12:45.155 INFO:tasks.rados.rados.0.smithi123.stdout:745: done (14 left) 2020-01-16T00:12:45.155 INFO:tasks.rados.rados.0.smithi123.stdout:746: done (13 left) 2020-01-16T00:12:45.155 INFO:tasks.rados.rados.0.smithi123.stdout:747: done (12 left) 2020-01-16T00:12:45.155 INFO:tasks.rados.rados.0.smithi123.stdout:760: read oid 23 snap -1 2020-01-16T00:12:45.155 INFO:tasks.rados.rados.0.smithi123.stdout:760: expect (ObjNum 22 snap 0 seq_num 22) 2020-01-16T00:12:45.156 INFO:tasks.rados.rados.0.smithi123.stdout:761: write (excl) oid 8 current snap is 0 2020-01-16T00:12:45.156 INFO:tasks.rados.rados.0.smithi123.stdout:761: seq_num 131 ranges {601717=635620,1799494=402223} 2020-01-16T00:12:45.166 INFO:tasks.rados.rados.0.smithi123.stdout:761: writing smithi12313934-8 from 601717 to 1237337 tid 1 2020-01-16T00:12:45.173 INFO:tasks.rados.rados.0.smithi123.stdout:761: writing smithi12313934-8 from 1799494 to 2201717 tid 2 2020-01-16T00:12:45.173 INFO:tasks.rados.rados.0.smithi123.stdout:762: read oid 28 snap -1 2020-01-16T00:12:45.173 INFO:tasks.rados.rados.0.smithi123.stdout:762: expect (ObjNum 73 snap 0 seq_num 73) 2020-01-16T00:12:45.173 INFO:tasks.rados.rados.0.smithi123.stdout:763: read oid 45 snap -1 2020-01-16T00:12:45.173 INFO:tasks.rados.rados.0.smithi123.stdout:763: expect (ObjNum 119 snap 0 seq_num 119) 2020-01-16T00:12:45.173 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:45.173 INFO:tasks.rados.rados.0.smithi123.stdout:754: expect (ObjNum 28 snap 0 seq_num 28) 2020-01-16T00:12:45.189 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:45.198 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-01-16T00:12:45.203 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:12:45.203+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:12:45.247 INFO:tasks.rados.rados.0.smithi123.stdout:752: expect (ObjNum 94 snap 0 seq_num 94) 2020-01-16T00:12:45.304 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:45.307+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:12:45.346 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:45.355 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-01-16T00:12:45.400 INFO:tasks.rados.rados.0.smithi123.stdout:749: finishing write tid 2 to smithi12313934-42 2020-01-16T00:12:45.400 INFO:tasks.rados.rados.0.smithi123.stdout:748: finishing write tid 4 to smithi12313934-33 2020-01-16T00:12:45.400 INFO:tasks.rados.rados.0.smithi123.stdout:748: finishing write tid 5 to smithi12313934-33 2020-01-16T00:12:45.400 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 33 v 144 (ObjNum 126 snap 0 seq_num 126) dirty exists 2020-01-16T00:12:45.400 INFO:tasks.rados.rados.0.smithi123.stdout:748: left oid 33 (ObjNum 126 snap 0 seq_num 126) 2020-01-16T00:12:45.400 INFO:tasks.rados.rados.0.smithi123.stdout:748: done (15 left) 2020-01-16T00:12:45.401 INFO:tasks.rados.rados.0.smithi123.stdout:752: done (14 left) 2020-01-16T00:12:45.401 INFO:tasks.rados.rados.0.smithi123.stdout:754: done (13 left) 2020-01-16T00:12:45.401 INFO:tasks.rados.rados.0.smithi123.stdout:764: read oid 22 snap -1 2020-01-16T00:12:45.401 INFO:tasks.rados.rados.0.smithi123.stdout:764: expect (ObjNum 123 snap 0 seq_num 123) 2020-01-16T00:12:45.401 INFO:tasks.rados.rados.0.smithi123.stdout:751: expect (ObjNum 59 snap 0 seq_num 59) 2020-01-16T00:12:45.405 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:45.405+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:12:45.465 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:45.473 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-01-16T00:12:45.516 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:12:45.519+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:12:45.517 INFO:tasks.rados.rados.0.smithi123.stdout:749: finishing write tid 4 to smithi12313934-42 2020-01-16T00:12:45.518 INFO:tasks.rados.rados.0.smithi123.stdout:749: finishing write tid 5 to smithi12313934-42 2020-01-16T00:12:45.518 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 42 v 65 (ObjNum 127 snap 0 seq_num 127) dirty exists 2020-01-16T00:12:45.518 INFO:tasks.rados.rados.0.smithi123.stdout:749: left oid 42 (ObjNum 127 snap 0 seq_num 127) 2020-01-16T00:12:45.518 INFO:tasks.rados.rados.0.smithi123.stdout:750: expect (ObjNum 101 snap 0 seq_num 101) 2020-01-16T00:12:45.606 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:12:45.607+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:12:45.617 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:45.626 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-01-16T00:12:45.659 INFO:tasks.rados.rados.0.smithi123.stdout:755: finishing write tid 1 to smithi12313934-25 2020-01-16T00:12:45.659 INFO:tasks.rados.rados.0.smithi123.stdout:755: finishing write tid 2 to smithi12313934-25 2020-01-16T00:12:45.659 INFO:tasks.rados.rados.0.smithi123.stdout:755: finishing write tid 4 to smithi12313934-25 2020-01-16T00:12:45.659 INFO:tasks.rados.rados.0.smithi123.stdout:755: finishing write tid 5 to smithi12313934-25 2020-01-16T00:12:45.659 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 25 v 102 (ObjNum 128 snap 0 seq_num 128) dirty exists 2020-01-16T00:12:45.659 INFO:tasks.rados.rados.0.smithi123.stdout:755: left oid 25 (ObjNum 128 snap 0 seq_num 128) 2020-01-16T00:12:45.660 INFO:tasks.rados.rados.0.smithi123.stdout:753: expect (ObjNum 67 snap 0 seq_num 67) 2020-01-16T00:12:45.711 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:45.709+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:12:45.762 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:45.778 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-01-16T00:12:45.825 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:12:45.827+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:12:45.832 INFO:tasks.rados.rados.0.smithi123.stdout:756: finishing write tid 1 to smithi12313934-14 2020-01-16T00:12:45.832 INFO:tasks.rados.rados.0.smithi123.stdout:749: done (13 left) 2020-01-16T00:12:45.832 INFO:tasks.rados.rados.0.smithi123.stdout:750: done (12 left) 2020-01-16T00:12:45.832 INFO:tasks.rados.rados.0.smithi123.stdout:751: done (11 left) 2020-01-16T00:12:45.832 INFO:tasks.rados.rados.0.smithi123.stdout:753: done (10 left) 2020-01-16T00:12:45.832 INFO:tasks.rados.rados.0.smithi123.stdout:755: done (9 left) 2020-01-16T00:12:45.832 INFO:tasks.rados.rados.0.smithi123.stdout:765: read oid 7 snap -1 2020-01-16T00:12:45.833 INFO:tasks.rados.rados.0.smithi123.stdout:765: expect deleted 2020-01-16T00:12:45.833 INFO:tasks.rados.rados.0.smithi123.stdout:766: write oid 31 current snap is 0 2020-01-16T00:12:45.833 INFO:tasks.rados.rados.0.smithi123.stdout:766: seq_num 132 ranges {618524=710869,1857976=360548} 2020-01-16T00:12:45.841 INFO:tasks.rados.rados.0.smithi123.stdout:766: writing smithi12313934-31 from 618524 to 1329393 tid 1 2020-01-16T00:12:45.847 INFO:tasks.rados.rados.0.smithi123.stdout:766: writing smithi12313934-31 from 1857976 to 2218524 tid 2 2020-01-16T00:12:45.847 INFO:tasks.rados.rados.0.smithi123.stdout:767: write oid 47 current snap is 0 2020-01-16T00:12:45.847 INFO:tasks.rados.rados.0.smithi123.stdout:767: seq_num 133 ranges {635331=786118,1916458=318873} 2020-01-16T00:12:45.856 INFO:tasks.rados.rados.0.smithi123.stdout:767: writing smithi12313934-47 from 635331 to 1421449 tid 1 2020-01-16T00:12:45.861 INFO:tasks.rados.rados.0.smithi123.stdout:767: writing smithi12313934-47 from 1916458 to 2235331 tid 2 2020-01-16T00:12:45.862 INFO:tasks.rados.rados.0.smithi123.stdout:768: read oid 16 snap -1 2020-01-16T00:12:45.862 INFO:tasks.rados.rados.0.smithi123.stdout:768: expect (ObjNum 60 snap 0 seq_num 60) 2020-01-16T00:12:45.862 INFO:tasks.rados.rados.0.smithi123.stdout:769: write oid 5 current snap is 0 2020-01-16T00:12:45.862 INFO:tasks.rados.rados.0.smithi123.stdout:769: seq_num 134 ranges {652138=461367,1574940=677198} 2020-01-16T00:12:45.869 INFO:tasks.rados.rados.0.smithi123.stdout:769: writing smithi12313934-5 from 652138 to 1113505 tid 1 2020-01-16T00:12:45.879 INFO:tasks.rados.rados.0.smithi123.stdout:769: writing smithi12313934-5 from 1574940 to 2252138 tid 2 2020-01-16T00:12:45.879 INFO:tasks.rados.rados.0.smithi123.stdout:770: read oid 11 snap -1 2020-01-16T00:12:45.879 INFO:tasks.rados.rados.0.smithi123.stdout:770: expect (ObjNum 10 snap 0 seq_num 10) 2020-01-16T00:12:45.879 INFO:tasks.rados.rados.0.smithi123.stdout:771: read oid 13 snap -1 2020-01-16T00:12:45.879 INFO:tasks.rados.rados.0.smithi123.stdout:771: expect (ObjNum 124 snap 0 seq_num 124) 2020-01-16T00:12:45.879 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:45.879 INFO:tasks.rados.rados.0.smithi123.stdout:757: expect (ObjNum 121 snap 0 seq_num 121) 2020-01-16T00:12:45.888 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4, 7, 6] 2020-01-16T00:12:45.888 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2020-01-16T00:12:45.888 INFO:tasks.thrashosds.thrasher:inject_pause on 5 2020-01-16T00:12:45.888 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2020-01-16T00:12:45.888 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2020-01-16T00:12:45.889 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3 2020-01-16T00:12:45.895 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:45.902 INFO:teuthology.orchestra.run.smithi123.stderr:nodeep-scrub is set 2020-01-16T00:12:45.908 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-01-16T00:12:45.935 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:12:45.937+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:12:45.986 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:45.986 INFO:teuthology.orchestra.run.smithi145.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2020-01-16T00:12:45.986 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:45.996 INFO:tasks.rados.rados.0.smithi123.stdout:756: finishing write tid 2 to smithi12313934-14 2020-01-16T00:12:45.996 INFO:tasks.rados.rados.0.smithi123.stdout:756: finishing write tid 4 to smithi12313934-14 2020-01-16T00:12:45.996 INFO:tasks.rados.rados.0.smithi123.stdout:756: finishing write tid 5 to smithi12313934-14 2020-01-16T00:12:45.996 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 14 v 94 (ObjNum 129 snap 0 seq_num 129) dirty exists 2020-01-16T00:12:45.996 INFO:tasks.rados.rados.0.smithi123.stdout:756: left oid 14 (ObjNum 129 snap 0 seq_num 129) 2020-01-16T00:12:45.997 INFO:tasks.rados.rados.0.smithi123.stdout:758: expect (ObjNum 83 snap 0 seq_num 83) 2020-01-16T00:12:46.011 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:12:46.011+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:12:46.050 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:46.058 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-01-16T00:12:46.144 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:12:46.147+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:12:46.190 INFO:tasks.rados.rados.0.smithi123.stdout:759: finishing write tid 1 to smithi12313934-38 2020-01-16T00:12:46.190 INFO:tasks.rados.rados.0.smithi123.stdout:759: finishing write tid 2 to smithi12313934-38 2020-01-16T00:12:46.190 INFO:tasks.rados.rados.0.smithi123.stdout:759: finishing write tid 4 to smithi12313934-38 2020-01-16T00:12:46.190 INFO:tasks.rados.rados.0.smithi123.stdout:759: finishing write tid 5 to smithi12313934-38 2020-01-16T00:12:46.191 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 38 v 69 (ObjNum 130 snap 0 seq_num 130) dirty exists 2020-01-16T00:12:46.191 INFO:tasks.rados.rados.0.smithi123.stdout:759: left oid 38 (ObjNum 130 snap 0 seq_num 130) 2020-01-16T00:12:46.191 INFO:tasks.rados.rados.0.smithi123.stdout:760: expect (ObjNum 22 snap 0 seq_num 22) 2020-01-16T00:12:46.205 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:46.214 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:46.213+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:46.215 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-01-16T00:12:46.256 INFO:tasks.rados.rados.0.smithi123.stdout:761: finishing write tid 1 to smithi12313934-8 2020-01-16T00:12:46.257 INFO:tasks.rados.rados.0.smithi123.stdout:761: finishing write tid 2 to smithi12313934-8 2020-01-16T00:12:46.257 INFO:tasks.rados.rados.0.smithi123.stdout:761: finishing write tid 4 to smithi12313934-8 2020-01-16T00:12:46.257 INFO:tasks.rados.rados.0.smithi123.stdout:761: finishing write tid 5 to smithi12313934-8 2020-01-16T00:12:46.257 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 8 v 44 (ObjNum 131 snap 0 seq_num 131) dirty exists 2020-01-16T00:12:46.257 INFO:tasks.rados.rados.0.smithi123.stdout:761: left oid 8 (ObjNum 131 snap 0 seq_num 131) 2020-01-16T00:12:46.257 INFO:tasks.rados.rados.0.smithi123.stdout:763: expect (ObjNum 119 snap 0 seq_num 119) 2020-01-16T00:12:46.274 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-01-16T00:12:46.312 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:46.313+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:46.353 INFO:tasks.rados.rados.0.smithi123.stdout:762: expect (ObjNum 73 snap 0 seq_num 73) 2020-01-16T00:12:46.357 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:46.366 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-01-16T00:12:46.413 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:46.413+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:12:46.424 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:12:46.443 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:12:46.461 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:12:46.466 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:46.474 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-01-16T00:12:46.482 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:12:46.489 INFO:tasks.rados.rados.0.smithi123.stdout:756: done (15 left) 2020-01-16T00:12:46.489 INFO:tasks.rados.rados.0.smithi123.stdout:757: done (14 left) 2020-01-16T00:12:46.489 INFO:tasks.rados.rados.0.smithi123.stdout:758: done (13 left) 2020-01-16T00:12:46.489 INFO:tasks.rados.rados.0.smithi123.stdout:759: done (12 left) 2020-01-16T00:12:46.489 INFO:tasks.rados.rados.0.smithi123.stdout:760: done (11 left) 2020-01-16T00:12:46.490 INFO:tasks.rados.rados.0.smithi123.stdout:761: done (10 left) 2020-01-16T00:12:46.490 INFO:tasks.rados.rados.0.smithi123.stdout:762: done (9 left) 2020-01-16T00:12:46.490 INFO:tasks.rados.rados.0.smithi123.stdout:763: done (8 left) 2020-01-16T00:12:46.490 INFO:tasks.rados.rados.0.smithi123.stdout:765: done (7 left) 2020-01-16T00:12:46.490 INFO:tasks.rados.rados.0.smithi123.stdout:772: read oid 28 snap -1 2020-01-16T00:12:46.490 INFO:tasks.rados.rados.0.smithi123.stdout:772: expect (ObjNum 73 snap 0 seq_num 73) 2020-01-16T00:12:46.490 INFO:tasks.rados.rados.0.smithi123.stdout:773: read oid 24 snap -1 2020-01-16T00:12:46.490 INFO:tasks.rados.rados.0.smithi123.stdout:773: expect (ObjNum 62 snap 0 seq_num 62) 2020-01-16T00:12:46.491 INFO:tasks.rados.rados.0.smithi123.stdout:774: delete oid 25 current snap is 0 2020-01-16T00:12:46.491 INFO:tasks.rados.rados.0.smithi123.stdout:764: expect (ObjNum 123 snap 0 seq_num 123) 2020-01-16T00:12:46.509 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:12:46.514 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:12:46.515+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:12:46.537 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5: {} 2020-01-16T00:12:46.558 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:12:46.572 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:46.572 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:12:46.572 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:12:46.572 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:12:46.572 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.572 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.5.0:146 3.c e24/20 3:3d6543e6:::smithi12313934-28:head v 24'67, mlcod=24'65)", 2020-01-16T00:12:46.572 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.849191+0000", 2020-01-16T00:12:46.573 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 8.722081846, 2020-01-16T00:12:46.573 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0088840970000000005, 2020-01-16T00:12:46.573 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:46.573 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:46.573 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:46.573 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.573 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:46.574 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.849191+0000", 2020-01-16T00:12:46.574 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:46.574 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.574 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.574 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:46.574 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.849191+0000", 2020-01-16T00:12:46.574 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:46.575 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.575 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.575 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:46.575 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.849188+0000", 2020-01-16T00:12:46.575 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0024105039999999999 2020-01-16T00:12:46.575 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.575 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.575 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:46.576 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.851599+0000", 2020-01-16T00:12:46.576 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0900999999999998e-05 2020-01-16T00:12:46.576 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.576 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.576 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:46.577 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.851660+0000", 2020-01-16T00:12:46.577 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0473e-05 2020-01-16T00:12:46.577 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.577 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.577 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:46.577 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.851670+0000", 2020-01-16T00:12:46.577 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8696999999999997e-05 2020-01-16T00:12:46.578 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.578 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.578 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:46.578 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.851729+0000", 2020-01-16T00:12:46.578 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3631999999999999e-05 2020-01-16T00:12:46.578 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.578 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.579 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:46.579 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.851763+0000", 2020-01-16T00:12:46.579 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0062692039999999996 2020-01-16T00:12:46.579 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.579 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.579 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:46.579 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.858032+0000", 2020-01-16T00:12:46.580 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2969999999999997e-05 2020-01-16T00:12:46.580 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.580 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.580 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:46.580 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.858075+0000", 2020-01-16T00:12:46.580 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0088840970000000005 2020-01-16T00:12:46.581 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.581 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:46.581 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.581 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.581 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.581 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:922 3.8 3:113947f2:::smithi12313934-12:head [write 443718~615603,stat] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:46.582 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.863202+0000", 2020-01-16T00:12:46.582 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 8.7080702629999998, 2020-01-16T00:12:46.582 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017152697000000001, 2020-01-16T00:12:46.582 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:46.582 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:46.583 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:46.583 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:46.583 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:46.583 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 922 2020-01-16T00:12:46.583 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.583 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:46.584 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.584 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:46.584 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863202+0000", 2020-01-16T00:12:46.584 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:46.584 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.585 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.585 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:46.585 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863202+0000", 2020-01-16T00:12:46.585 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:46.585 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.586 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.586 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:46.586 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863200+0000", 2020-01-16T00:12:46.586 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00038569199999999998 2020-01-16T00:12:46.587 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.587 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.587 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:46.587 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863585+0000", 2020-01-16T00:12:46.588 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5685e-05 2020-01-16T00:12:46.588 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.591 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.600 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:46.600 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863601+0000", 2020-01-16T00:12:46.601 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.4734000000000001e-05 2020-01-16T00:12:46.601 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.601 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.602 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:46.602 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863626+0000", 2020-01-16T00:12:46.602 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1037e-05 2020-01-16T00:12:46.602 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.602 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.602 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:46.602 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863667+0000", 2020-01-16T00:12:46.603 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016041499999999999 2020-01-16T00:12:46.603 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.603 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.603 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:46.603 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863827+0000", 2020-01-16T00:12:46.603 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0094286430000000004 2020-01-16T00:12:46.604 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.604 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.604 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:46.604 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.873256+0000", 2020-01-16T00:12:46.604 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2867999999999994e-05 2020-01-16T00:12:46.604 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.605 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.605 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:46.605 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.873329+0000", 2020-01-16T00:12:46.606 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021174789999999998 2020-01-16T00:12:46.606 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.606 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.606 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:46.606 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.875446+0000", 2020-01-16T00:12:46.606 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011244200000000001 2020-01-16T00:12:46.607 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.607 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.607 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:46.607 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.875559+0000", 2020-01-16T00:12:46.608 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021772600000000001 2020-01-16T00:12:46.608 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.610 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.610 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1,7", 2020-01-16T00:12:46.610 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.875776+0000", 2020-01-16T00:12:46.611 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0028978860000000001 2020-01-16T00:12:46.611 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.612 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.612 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:46.612 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.878674+0000", 2020-01-16T00:12:46.612 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.324e-05 2020-01-16T00:12:46.612 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.612 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.613 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:46.613 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.878687+0000", 2020-01-16T00:12:46.613 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0014896180000000001 2020-01-16T00:12:46.613 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.613 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.613 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:46.614 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880177+0000", 2020-01-16T00:12:46.614 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6881000000000001e-05 2020-01-16T00:12:46.614 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.614 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.614 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:46.614 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880224+0000", 2020-01-16T00:12:46.615 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013096700000000001 2020-01-16T00:12:46.615 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.615 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.615 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:46.615 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880355+0000", 2020-01-16T00:12:46.615 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017152697000000001 2020-01-16T00:12:46.615 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.616 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:46.616 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.616 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.616 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.616 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:923 3.8 3:113947f2:::smithi12313934-12:head [write 1564138~470341] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:46.616 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.869292+0000", 2020-01-16T00:12:46.616 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 8.7019799330000005, 2020-01-16T00:12:46.616 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017661620999999999, 2020-01-16T00:12:46.617 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:46.617 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:46.617 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:46.617 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:46.617 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:46.617 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 923 2020-01-16T00:12:46.618 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.618 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:46.618 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.619 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:46.619 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869292+0000", 2020-01-16T00:12:46.619 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:46.619 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.620 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.620 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:46.620 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869292+0000", 2020-01-16T00:12:46.620 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:46.620 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.620 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.621 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:46.621 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869291+0000", 2020-01-16T00:12:46.621 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00049842899999999999 2020-01-16T00:12:46.621 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.621 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.621 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:46.622 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869789+0000", 2020-01-16T00:12:46.622 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1014999999999999e-05 2020-01-16T00:12:46.622 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.622 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.623 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:46.623 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869830+0000", 2020-01-16T00:12:46.623 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7880000000000007e-06 2020-01-16T00:12:46.623 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.623 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.624 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:46.624 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869838+0000", 2020-01-16T00:12:46.624 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8346999999999997e-05 2020-01-16T00:12:46.624 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.624 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.625 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:46.625 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869876+0000", 2020-01-16T00:12:46.625 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9726999999999999e-05 2020-01-16T00:12:46.625 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.625 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.625 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:46.626 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869936+0000", 2020-01-16T00:12:46.626 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003434917 2020-01-16T00:12:46.626 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.626 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.626 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:46.626 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.873371+0000", 2020-01-16T00:12:46.627 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118808 2020-01-16T00:12:46.627 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.627 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.627 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:46.627 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.873490+0000", 2020-01-16T00:12:46.628 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036712910000000001 2020-01-16T00:12:46.628 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.628 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.628 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:46.628 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877161+0000", 2020-01-16T00:12:46.628 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9909999999999999e-05 2020-01-16T00:12:46.629 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.629 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.629 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:46.629 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877241+0000", 2020-01-16T00:12:46.629 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031619970000000002 2020-01-16T00:12:46.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.630 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.630 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:46.630 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880403+0000", 2020-01-16T00:12:46.630 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013071099999999999 2020-01-16T00:12:46.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.630 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.631 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:46.631 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880534+0000", 2020-01-16T00:12:46.631 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00032823500000000002 2020-01-16T00:12:46.631 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.631 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.631 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1,7", 2020-01-16T00:12:46.631 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880862+0000", 2020-01-16T00:12:46.632 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00513862 2020-01-16T00:12:46.632 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.632 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.632 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:46.632 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886001+0000", 2020-01-16T00:12:46.632 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8869000000000001e-05 2020-01-16T00:12:46.633 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.633 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.633 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:46.633 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886030+0000", 2020-01-16T00:12:46.634 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00081600200000000003 2020-01-16T00:12:46.635 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.635 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.635 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:46.636 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886846+0000", 2020-01-16T00:12:46.636 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4095999999999998e-05 2020-01-16T00:12:46.636 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.636 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.636 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:46.636 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886900+0000", 2020-01-16T00:12:46.636 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.448e-05 2020-01-16T00:12:46.637 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.637 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.637 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:46.637 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886954+0000", 2020-01-16T00:12:46.637 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017661620999999999 2020-01-16T00:12:46.637 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.637 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:46.637 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.638 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.638 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.638 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:924 3.8 3:113947f2:::smithi12313934-12:head [write 2633066~610652] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:46.638 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.876207+0000", 2020-01-16T00:12:46.638 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 8.6950657790000001, 2020-01-16T00:12:46.638 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011069638, 2020-01-16T00:12:46.638 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:46.638 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:46.639 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:46.639 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:46.639 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:46.639 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 924 2020-01-16T00:12:46.639 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.639 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:46.639 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.639 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:46.640 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876207+0000", 2020-01-16T00:12:46.640 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:46.640 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.640 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.640 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:46.640 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876207+0000", 2020-01-16T00:12:46.640 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:46.640 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.641 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.641 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:46.641 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876204+0000", 2020-01-16T00:12:46.641 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000157842 2020-01-16T00:12:46.641 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.641 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.641 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:46.642 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876362+0000", 2020-01-16T00:12:46.642 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0181e-05 2020-01-16T00:12:46.642 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.642 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.642 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:46.642 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876372+0000", 2020-01-16T00:12:46.642 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2409999999999997e-06 2020-01-16T00:12:46.642 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.643 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.643 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:46.643 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876377+0000", 2020-01-16T00:12:46.643 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00089908200000000005 2020-01-16T00:12:46.643 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.643 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.643 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:46.643 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877276+0000", 2020-01-16T00:12:46.644 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2391000000000006e-05 2020-01-16T00:12:46.644 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.644 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.644 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:46.644 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877338+0000", 2020-01-16T00:12:46.644 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0062765850000000003 2020-01-16T00:12:46.644 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.644 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.645 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:46.645 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.883615+0000", 2020-01-16T00:12:46.645 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011738700000000001 2020-01-16T00:12:46.645 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.645 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.645 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:46.645 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.883732+0000", 2020-01-16T00:12:46.645 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023677 2020-01-16T00:12:46.646 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.646 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.646 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1,7", 2020-01-16T00:12:46.646 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.883969+0000", 2020-01-16T00:12:46.646 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0028104979999999998 2020-01-16T00:12:46.646 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.646 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.646 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:46.647 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886779+0000", 2020-01-16T00:12:46.647 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00025251899999999999 2020-01-16T00:12:46.647 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.647 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.647 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:46.647 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.887032+0000", 2020-01-16T00:12:46.647 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018013099999999999 2020-01-16T00:12:46.647 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.648 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.648 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:46.648 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.887212+0000", 2020-01-16T00:12:46.648 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9646e-05 2020-01-16T00:12:46.648 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.648 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.648 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:46.648 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.887242+0000", 2020-01-16T00:12:46.649 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4551000000000002e-05 2020-01-16T00:12:46.649 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.649 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.649 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:46.649 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.887276+0000", 2020-01-16T00:12:46.649 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011069638 2020-01-16T00:12:46.649 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.650 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:46.650 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.650 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.650 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.650 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:925 3.8 3:113947f2:::smithi12313934-12:head [setxattr _header (56) in=7b,truncate 3243718] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:46.650 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.876384+0000", 2020-01-16T00:12:46.650 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 8.6948887320000008, 2020-01-16T00:12:46.650 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011601155, 2020-01-16T00:12:46.651 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:46.651 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:46.651 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:46.651 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:46.651 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:46.651 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 925 2020-01-16T00:12:46.651 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.651 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:46.652 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.652 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:46.652 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876384+0000", 2020-01-16T00:12:46.652 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:46.652 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.652 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.652 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:46.652 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876384+0000", 2020-01-16T00:12:46.653 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:46.653 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.653 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.653 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:46.653 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876383+0000", 2020-01-16T00:12:46.653 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.946e-06 2020-01-16T00:12:46.653 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.654 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.654 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:46.654 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876387+0000", 2020-01-16T00:12:46.654 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7179999999999998e-06 2020-01-16T00:12:46.654 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.654 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.654 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:46.654 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876391+0000", 2020-01-16T00:12:46.655 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3980000000000002e-06 2020-01-16T00:12:46.655 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.655 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.655 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:46.655 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876393+0000", 2020-01-16T00:12:46.655 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00097104099999999996 2020-01-16T00:12:46.655 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.655 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.656 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:46.656 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877364+0000", 2020-01-16T00:12:46.656 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6619000000000001e-05 2020-01-16T00:12:46.656 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.656 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.656 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:46.656 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877421+0000", 2020-01-16T00:12:46.656 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0078613510000000008 2020-01-16T00:12:46.657 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.657 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.657 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:46.657 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.885282+0000", 2020-01-16T00:12:46.657 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015531400000000001 2020-01-16T00:12:46.657 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.657 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.657 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:46.658 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.885437+0000", 2020-01-16T00:12:46.658 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019132099999999999 2020-01-16T00:12:46.658 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.658 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.658 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1,7", 2020-01-16T00:12:46.658 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.885629+0000", 2020-01-16T00:12:46.658 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0011787270000000001 2020-01-16T00:12:46.658 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.659 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.659 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:46.659 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886807+0000", 2020-01-16T00:12:46.659 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000739092 2020-01-16T00:12:46.659 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.659 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.659 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:46.659 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.887547+0000", 2020-01-16T00:12:46.660 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00037348900000000001 2020-01-16T00:12:46.660 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.660 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.660 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:46.660 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.887920+0000", 2020-01-16T00:12:46.660 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9629999999999999e-05 2020-01-16T00:12:46.660 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.660 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.661 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:46.661 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.887950+0000", 2020-01-16T00:12:46.661 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5172999999999997e-05 2020-01-16T00:12:46.661 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.661 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.661 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:46.661 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.887985+0000", 2020-01-16T00:12:46.662 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011601155 2020-01-16T00:12:46.662 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.662 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:46.662 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.662 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.662 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.662 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:926 3.8 3:113947f2:::smithi12313934-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:12:46.662 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.876396+0000", 2020-01-16T00:12:46.663 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 8.6948768629999993, 2020-01-16T00:12:46.663 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011788966999999999, 2020-01-16T00:12:46.663 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:46.663 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:46.663 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:46.663 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:46.663 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:46.663 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 926 2020-01-16T00:12:46.664 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.664 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:46.664 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.664 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:46.664 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876396+0000", 2020-01-16T00:12:46.664 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:46.664 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.664 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.665 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:46.665 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876396+0000", 2020-01-16T00:12:46.665 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:46.665 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.665 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.665 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:46.665 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876395+0000", 2020-01-16T00:12:46.665 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9369999999999998e-06 2020-01-16T00:12:46.666 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.666 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.666 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:46.666 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876397+0000", 2020-01-16T00:12:46.666 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2839999999999998e-06 2020-01-16T00:12:46.666 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.666 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.667 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:46.667 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876400+0000", 2020-01-16T00:12:46.667 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2060000000000001e-06 2020-01-16T00:12:46.667 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.667 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.667 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:46.667 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876403+0000", 2020-01-16T00:12:46.667 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010315179999999999 2020-01-16T00:12:46.668 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.668 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.668 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:46.668 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877434+0000", 2020-01-16T00:12:46.668 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0519999999999997e-05 2020-01-16T00:12:46.668 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.668 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.668 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:46.669 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877485+0000", 2020-01-16T00:12:46.669 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0084118030000000007 2020-01-16T00:12:46.669 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.669 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.669 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:46.669 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.885897+0000", 2020-01-16T00:12:46.669 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6395000000000001e-05 2020-01-16T00:12:46.669 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.670 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.670 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:46.670 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.885993+0000", 2020-01-16T00:12:46.670 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020290849999999999 2020-01-16T00:12:46.670 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.670 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.670 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:46.670 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.888022+0000", 2020-01-16T00:12:46.671 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7325999999999998e-05 2020-01-16T00:12:46.671 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.671 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.671 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:46.671 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.888099+0000", 2020-01-16T00:12:46.671 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5185999999999994e-05 2020-01-16T00:12:46.671 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.671 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.672 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:46.672 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.888185+0000", 2020-01-16T00:12:46.672 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011788966999999999 2020-01-16T00:12:46.672 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.672 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:46.672 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.672 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.672 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.673 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:928 3.b 3:df203f2a:::smithi12313934-3:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:46.673 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.876426+0000", 2020-01-16T00:12:46.673 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 8.6948463940000007, 2020-01-16T00:12:46.673 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0070152909999999999, 2020-01-16T00:12:46.673 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:46.673 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:46.673 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:46.673 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:46.674 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:46.674 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 928 2020-01-16T00:12:46.674 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.674 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:46.674 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.674 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:46.674 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876426+0000", 2020-01-16T00:12:46.674 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:46.675 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.675 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.675 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:46.675 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876426+0000", 2020-01-16T00:12:46.675 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:46.675 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.675 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.676 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:46.676 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876426+0000", 2020-01-16T00:12:46.676 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.18e-06 2020-01-16T00:12:46.676 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.676 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.676 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:46.676 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876429+0000", 2020-01-16T00:12:46.676 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2399999999999998e-06 2020-01-16T00:12:46.677 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.677 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.677 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:46.677 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876434+0000", 2020-01-16T00:12:46.677 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1330000000000001e-06 2020-01-16T00:12:46.677 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.677 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.677 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:46.678 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876441+0000", 2020-01-16T00:12:46.678 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0064132030000000001 2020-01-16T00:12:46.678 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.678 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.678 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:46.678 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.882854+0000", 2020-01-16T00:12:46.678 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016152500000000001 2020-01-16T00:12:46.678 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.679 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.679 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:46.679 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.883016+0000", 2020-01-16T00:12:46.679 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00042550800000000001 2020-01-16T00:12:46.679 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.679 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.679 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:46.679 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.883441+0000", 2020-01-16T00:12:46.680 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0070152909999999999 2020-01-16T00:12:46.680 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.680 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:46.680 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.680 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.680 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.680 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:929 3.b 3:df203f2a:::smithi12313934-3:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:46.680 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.876448+0000", 2020-01-16T00:12:46.681 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 8.6948241609999997, 2020-01-16T00:12:46.681 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0076937330000000003, 2020-01-16T00:12:46.681 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:46.681 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:46.681 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:46.681 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:46.681 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:46.681 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 929 2020-01-16T00:12:46.682 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.682 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:46.682 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.682 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:46.682 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876448+0000", 2020-01-16T00:12:46.682 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:46.682 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.683 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.683 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:46.683 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876448+0000", 2020-01-16T00:12:46.683 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:46.683 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.683 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.683 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:46.683 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876448+0000", 2020-01-16T00:12:46.684 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9470000000000002e-06 2020-01-16T00:12:46.684 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.684 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.684 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:46.684 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876453+0000", 2020-01-16T00:12:46.684 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9840000000000001e-06 2020-01-16T00:12:46.684 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.684 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.685 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:46.685 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876458+0000", 2020-01-16T00:12:46.685 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1879999999999998e-06 2020-01-16T00:12:46.685 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.685 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.685 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:46.685 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.876461+0000", 2020-01-16T00:12:46.685 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0070321639999999996 2020-01-16T00:12:46.686 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.686 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.686 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:46.686 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.883493+0000", 2020-01-16T00:12:46.686 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021568600000000001 2020-01-16T00:12:46.686 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.686 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.686 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:46.687 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.883709+0000", 2020-01-16T00:12:46.687 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043327600000000002 2020-01-16T00:12:46.687 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.687 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.687 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:46.687 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.884142+0000", 2020-01-16T00:12:46.687 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0076937330000000003 2020-01-16T00:12:46.688 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.688 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:46.688 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.688 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.688 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.688 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:935 3.2 3:4f8fe446:::smithi12313934-48:head [write 3060489~200036] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:46.688 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.895775+0000", 2020-01-16T00:12:46.688 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 8.6754977469999996, 2020-01-16T00:12:46.689 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012912735, 2020-01-16T00:12:46.689 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:46.689 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:46.689 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:46.689 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:46.689 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:46.689 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 935 2020-01-16T00:12:46.690 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.690 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:46.690 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.690 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:46.690 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895775+0000", 2020-01-16T00:12:46.690 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:46.690 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.690 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.691 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:46.691 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895775+0000", 2020-01-16T00:12:46.691 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:46.691 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.691 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.691 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:46.691 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895773+0000", 2020-01-16T00:12:46.691 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3350000000000007e-05 2020-01-16T00:12:46.692 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.692 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.692 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:46.692 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895856+0000", 2020-01-16T00:12:46.692 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.418e-05 2020-01-16T00:12:46.692 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.692 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.692 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:46.693 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895871+0000", 2020-01-16T00:12:46.693 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2750000000000001e-06 2020-01-16T00:12:46.693 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.693 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.693 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:46.693 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895877+0000", 2020-01-16T00:12:46.693 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7113000000000002e-05 2020-01-16T00:12:46.693 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.694 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.694 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:46.694 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895914+0000", 2020-01-16T00:12:46.694 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00064030700000000001 2020-01-16T00:12:46.694 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.694 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.694 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:46.694 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.896554+0000", 2020-01-16T00:12:46.695 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0098022809999999995 2020-01-16T00:12:46.695 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.695 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.695 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:46.695 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.906357+0000", 2020-01-16T00:12:46.695 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112898 2020-01-16T00:12:46.695 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.695 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.696 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:46.696 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.906469+0000", 2020-01-16T00:12:46.696 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022189399999999999 2020-01-16T00:12:46.696 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.696 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.696 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:12:46.696 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.906691+0000", 2020-01-16T00:12:46.697 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018153869999999999 2020-01-16T00:12:46.697 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.697 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.697 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:46.697 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908507+0000", 2020-01-16T00:12:46.697 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6517999999999999e-05 2020-01-16T00:12:46.697 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.697 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.698 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:46.698 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908533+0000", 2020-01-16T00:12:46.698 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1899e-05 2020-01-16T00:12:46.698 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.698 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.698 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:46.698 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908575+0000", 2020-01-16T00:12:46.699 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2222999999999998e-05 2020-01-16T00:12:46.699 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.699 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.699 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:46.699 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908627+0000", 2020-01-16T00:12:46.699 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0019999999999998e-05 2020-01-16T00:12:46.699 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.700 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.700 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:46.700 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908687+0000", 2020-01-16T00:12:46.700 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012912735 2020-01-16T00:12:46.700 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.700 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:46.700 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.700 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.701 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.701 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:936 3.2 3:4f8fe446:::smithi12313934-48:head [setxattr _header (56) in=7b,truncate 3260525] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:46.701 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.895886+0000", 2020-01-16T00:12:46.701 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 8.6753860429999996, 2020-01-16T00:12:46.701 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014382986, 2020-01-16T00:12:46.701 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:46.701 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:46.701 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:46.702 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:46.702 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:46.702 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 936 2020-01-16T00:12:46.702 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.702 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:46.702 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.702 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:46.703 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895886+0000", 2020-01-16T00:12:46.703 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:46.703 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.703 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.703 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:46.703 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895886+0000", 2020-01-16T00:12:46.703 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:12:46.703 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.704 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.704 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:46.704 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895885+0000", 2020-01-16T00:12:46.704 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0980000000000001e-06 2020-01-16T00:12:46.704 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.704 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.704 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:46.704 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895891+0000", 2020-01-16T00:12:46.705 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.959e-06 2020-01-16T00:12:46.705 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.705 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.705 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:46.705 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895897+0000", 2020-01-16T00:12:46.705 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9920000000000001e-06 2020-01-16T00:12:46.705 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.705 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.706 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:46.706 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895901+0000", 2020-01-16T00:12:46.706 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00068615199999999999 2020-01-16T00:12:46.706 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.706 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.706 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:46.706 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.896587+0000", 2020-01-16T00:12:46.707 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2934e-05 2020-01-16T00:12:46.707 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.707 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.707 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:46.707 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.896620+0000", 2020-01-16T00:12:46.707 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010751811 2020-01-16T00:12:46.707 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.707 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.708 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:46.708 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.907371+0000", 2020-01-16T00:12:46.708 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011422899999999999 2020-01-16T00:12:46.708 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.708 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.708 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:46.708 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.907486+0000", 2020-01-16T00:12:46.708 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000185172 2020-01-16T00:12:46.709 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.709 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.709 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:12:46.709 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.907671+0000", 2020-01-16T00:12:46.709 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001265164 2020-01-16T00:12:46.709 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.709 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.709 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:46.710 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908936+0000", 2020-01-16T00:12:46.710 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000297247 2020-01-16T00:12:46.710 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.710 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.710 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:46.710 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.909233+0000", 2020-01-16T00:12:46.710 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000907528 2020-01-16T00:12:46.711 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.711 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.711 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:46.711 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910141+0000", 2020-01-16T00:12:46.711 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3384000000000002e-05 2020-01-16T00:12:46.711 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.711 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.711 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:46.712 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910204+0000", 2020-01-16T00:12:46.712 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5229999999999997e-05 2020-01-16T00:12:46.712 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.712 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.712 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:46.712 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910269+0000", 2020-01-16T00:12:46.712 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014382986 2020-01-16T00:12:46.713 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.713 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:46.713 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.713 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.713 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.713 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:937 3.2 3:4f8fe446:::smithi12313934-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:12:46.713 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.895918+0000", 2020-01-16T00:12:46.713 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 8.6753544149999993, 2020-01-16T00:12:46.714 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014610741999999999, 2020-01-16T00:12:46.714 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:46.714 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:46.714 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:46.714 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:46.714 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:46.714 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 937 2020-01-16T00:12:46.715 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.715 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:46.715 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.715 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:46.715 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895918+0000", 2020-01-16T00:12:46.715 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:46.715 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.715 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.716 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:46.716 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895918+0000", 2020-01-16T00:12:46.716 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:12:46.716 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.716 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.716 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:46.716 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895917+0000", 2020-01-16T00:12:46.716 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8840000000000004e-06 2020-01-16T00:12:46.717 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.717 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.717 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:46.717 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895921+0000", 2020-01-16T00:12:46.717 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1579999999999999e-06 2020-01-16T00:12:46.717 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.717 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.718 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:46.718 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895928+0000", 2020-01-16T00:12:46.718 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4139999999999996e-06 2020-01-16T00:12:46.718 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.718 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.718 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:46.718 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895933+0000", 2020-01-16T00:12:46.718 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000702218 2020-01-16T00:12:46.719 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.719 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.719 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:46.719 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.896635+0000", 2020-01-16T00:12:46.719 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6421e-05 2020-01-16T00:12:46.719 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.719 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.719 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:46.720 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.896661+0000", 2020-01-16T00:12:46.720 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011578879 2020-01-16T00:12:46.720 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.720 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.720 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:46.720 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908240+0000", 2020-01-16T00:12:46.720 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5567000000000003e-05 2020-01-16T00:12:46.720 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.721 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.721 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:46.721 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908316+0000", 2020-01-16T00:12:46.721 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0019937560000000002 2020-01-16T00:12:46.721 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.721 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.721 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:46.722 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910310+0000", 2020-01-16T00:12:46.722 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000109566 2020-01-16T00:12:46.722 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.722 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.722 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:46.722 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910419+0000", 2020-01-16T00:12:46.722 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000109629 2020-01-16T00:12:46.722 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.723 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.723 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:46.723 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910529+0000", 2020-01-16T00:12:46.723 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014610741999999999 2020-01-16T00:12:46.723 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.723 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:46.723 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.723 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.724 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.724 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.5.0:158 3.c e24/20 3:3877922e:::smithi12313934-31:head v 24'72, mlcod=24'70)", 2020-01-16T00:12:46.724 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.931147+0000", 2020-01-16T00:12:46.724 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 8.6401252169999996, 2020-01-16T00:12:46.724 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0060017630000000002, 2020-01-16T00:12:46.724 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:46.724 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:46.724 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:46.725 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.725 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:46.725 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931147+0000", 2020-01-16T00:12:46.725 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:46.725 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.725 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.726 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:46.726 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931147+0000", 2020-01-16T00:12:46.726 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:46.726 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.726 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.726 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:46.726 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931145+0000", 2020-01-16T00:12:46.727 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0015459009999999999 2020-01-16T00:12:46.727 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.727 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.727 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:46.727 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932691+0000", 2020-01-16T00:12:46.727 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4831000000000001e-05 2020-01-16T00:12:46.727 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.727 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.728 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:46.728 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932705+0000", 2020-01-16T00:12:46.728 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3569999999999997e-06 2020-01-16T00:12:46.728 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.728 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.728 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:46.728 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932711+0000", 2020-01-16T00:12:46.728 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6208000000000003e-05 2020-01-16T00:12:46.729 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.729 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.729 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:46.729 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932747+0000", 2020-01-16T00:12:46.729 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7938999999999999e-05 2020-01-16T00:12:46.729 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.729 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.729 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:46.730 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932765+0000", 2020-01-16T00:12:46.730 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0043314 2020-01-16T00:12:46.730 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.730 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.730 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:46.730 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.937096+0000", 2020-01-16T00:12:46.730 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2614999999999997e-05 2020-01-16T00:12:46.730 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.731 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.731 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:46.731 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.937149+0000", 2020-01-16T00:12:46.731 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0060017630000000002 2020-01-16T00:12:46.731 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.731 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:46.731 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.731 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.732 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.732 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:972 3.8 3:1e8f7c6f:::smithi12313934-6:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:46.732 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.159726+0000", 2020-01-16T00:12:46.732 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 8.4115460249999998, 2020-01-16T00:12:46.732 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0061844259999999998, 2020-01-16T00:12:46.732 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:46.732 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:46.733 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:46.733 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:46.733 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:46.733 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 972 2020-01-16T00:12:46.733 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.733 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:46.733 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.733 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:46.734 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159726+0000", 2020-01-16T00:12:46.734 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:46.734 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.734 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.734 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:46.734 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159726+0000", 2020-01-16T00:12:46.734 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:46.734 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.735 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.735 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:46.735 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159726+0000", 2020-01-16T00:12:46.735 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0910000000000003e-06 2020-01-16T00:12:46.735 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.735 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.735 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:46.735 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159730+0000", 2020-01-16T00:12:46.736 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.6910000000000003e-06 2020-01-16T00:12:46.736 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.736 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.736 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:46.736 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159738+0000", 2020-01-16T00:12:46.736 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1479999999999999e-06 2020-01-16T00:12:46.736 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.736 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.737 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:46.737 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.159743+0000", 2020-01-16T00:12:46.737 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00057154700000000005 2020-01-16T00:12:46.737 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.737 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.737 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:46.737 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.160314+0000", 2020-01-16T00:12:46.737 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8436999999999997e-05 2020-01-16T00:12:46.738 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.738 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.738 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:46.738 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.160363+0000", 2020-01-16T00:12:46.738 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0045992810000000002 2020-01-16T00:12:46.738 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.738 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.738 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:46.739 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.164962+0000", 2020-01-16T00:12:46.739 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8581000000000001e-05 2020-01-16T00:12:46.739 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.739 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.739 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:46.739 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.165010+0000", 2020-01-16T00:12:46.739 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00060742200000000004 2020-01-16T00:12:46.739 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.740 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.740 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:46.740 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.165618+0000", 2020-01-16T00:12:46.740 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6708999999999999e-05 2020-01-16T00:12:46.740 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.740 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.740 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:46.741 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.165715+0000", 2020-01-16T00:12:46.741 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019620999999999999 2020-01-16T00:12:46.741 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.741 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.741 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:46.741 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.165911+0000", 2020-01-16T00:12:46.741 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0061844259999999998 2020-01-16T00:12:46.741 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.742 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:46.742 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.742 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.742 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.742 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1149 3.2 3:4f8fe446:::smithi12313934-48:head [read 0~3260525,checksum] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:12:46.742 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:41.603914+0000", 2020-01-16T00:12:46.742 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.9673587619999999, 2020-01-16T00:12:46.742 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0069101639999999999, 2020-01-16T00:12:46.743 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:46.743 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:46.743 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:46.743 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:46.743 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:46.743 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1149 2020-01-16T00:12:46.743 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.743 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:46.744 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.744 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:46.744 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.603914+0000", 2020-01-16T00:12:46.744 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:46.744 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.744 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.744 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:46.745 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.603914+0000", 2020-01-16T00:12:46.745 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:46.745 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.745 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.745 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:46.745 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.603913+0000", 2020-01-16T00:12:46.745 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9479999999999999e-06 2020-01-16T00:12:46.745 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.746 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.746 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:46.746 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.603918+0000", 2020-01-16T00:12:46.746 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4470000000000003e-06 2020-01-16T00:12:46.746 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.746 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.746 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:46.746 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.603924+0000", 2020-01-16T00:12:46.747 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8700000000000002e-06 2020-01-16T00:12:46.747 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.747 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.747 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:46.747 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.603928+0000", 2020-01-16T00:12:46.747 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0050935329999999999 2020-01-16T00:12:46.747 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.747 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.748 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:46.748 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.609022+0000", 2020-01-16T00:12:46.748 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012769600000000001 2020-01-16T00:12:46.748 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.748 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.748 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:46.748 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.609149+0000", 2020-01-16T00:12:46.748 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0016743299999999999 2020-01-16T00:12:46.749 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.749 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.749 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:46.749 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.610824+0000", 2020-01-16T00:12:46.749 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0069101639999999999 2020-01-16T00:12:46.749 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.749 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:46.750 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.750 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.750 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.750 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1151 3.7 3:efab040c:::smithi12313934-30:head [read 0~3411788,checksum] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:12:46.750 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:41.603971+0000", 2020-01-16T00:12:46.750 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.9673018789999999, 2020-01-16T00:12:46.750 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0082599330000000006, 2020-01-16T00:12:46.750 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:46.751 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:46.751 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:46.751 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:46.751 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:46.751 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1151 2020-01-16T00:12:46.751 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.751 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:46.751 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.752 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:46.752 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.603971+0000", 2020-01-16T00:12:46.752 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:46.752 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.752 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.752 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:46.752 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.603971+0000", 2020-01-16T00:12:46.752 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:12:46.753 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.753 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.753 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:46.753 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.603970+0000", 2020-01-16T00:12:46.753 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7449999999999995e-06 2020-01-16T00:12:46.753 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.753 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.754 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:46.754 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.603977+0000", 2020-01-16T00:12:46.754 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7689999999999999e-06 2020-01-16T00:12:46.754 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.754 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.754 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:46.754 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.603984+0000", 2020-01-16T00:12:46.754 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7489999999999999e-06 2020-01-16T00:12:46.755 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.755 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.755 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:46.755 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.603987+0000", 2020-01-16T00:12:46.755 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0050570839999999999 2020-01-16T00:12:46.755 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.755 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.756 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:46.756 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.609044+0000", 2020-01-16T00:12:46.756 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000142452 2020-01-16T00:12:46.756 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.756 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.756 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:46.756 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.609186+0000", 2020-01-16T00:12:46.756 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030440089999999999 2020-01-16T00:12:46.757 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.757 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.757 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:46.757 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.612230+0000", 2020-01-16T00:12:46.757 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0082599330000000006 2020-01-16T00:12:46.757 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.757 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:46.757 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.758 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.758 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.758 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1152 3.7 3:efab040c:::smithi12313934-30:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:12:46.758 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:41.603990+0000", 2020-01-16T00:12:46.758 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.9672822439999997, 2020-01-16T00:12:46.758 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093224899999999992, 2020-01-16T00:12:46.758 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:46.759 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:46.759 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:46.759 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:46.759 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:46.759 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1152 2020-01-16T00:12:46.759 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.759 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:46.759 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.760 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:46.760 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.603990+0000", 2020-01-16T00:12:46.760 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:46.760 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.760 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.760 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:46.760 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.603990+0000", 2020-01-16T00:12:46.760 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:46.761 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.761 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.761 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:46.761 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.603990+0000", 2020-01-16T00:12:46.761 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4040000000000003e-06 2020-01-16T00:12:46.761 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.761 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.761 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:46.762 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.603995+0000", 2020-01-16T00:12:46.762 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8899999999999999e-06 2020-01-16T00:12:46.762 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.762 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.762 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:46.762 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.603998+0000", 2020-01-16T00:12:46.762 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0849999999999996e-06 2020-01-16T00:12:46.762 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.763 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.763 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:46.763 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.604003+0000", 2020-01-16T00:12:46.763 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0082858129999999995 2020-01-16T00:12:46.763 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.763 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.763 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:46.764 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.612289+0000", 2020-01-16T00:12:46.764 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016920300000000001 2020-01-16T00:12:46.764 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.764 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.764 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:46.765 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.612458+0000", 2020-01-16T00:12:46.765 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00085449099999999997 2020-01-16T00:12:46.765 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.765 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.765 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:46.765 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.613313+0000", 2020-01-16T00:12:46.765 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093224899999999992 2020-01-16T00:12:46.766 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.766 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:46.766 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.766 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.766 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.766 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1177 3.b 3:df203f2a:::smithi12313934-3:head [write 648770~706332,stat] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:12:46.766 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:42.089533+0000", 2020-01-16T00:12:46.766 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.4817395170000003, 2020-01-16T00:12:46.767 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0068691790000000004, 2020-01-16T00:12:46.767 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:46.767 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:46.767 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:46.767 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:46.767 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:46.767 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1177 2020-01-16T00:12:46.767 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.768 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:46.768 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.768 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:46.768 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.089533+0000", 2020-01-16T00:12:46.768 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:46.768 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.768 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.769 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:46.769 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.089533+0000", 2020-01-16T00:12:46.769 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999938 2020-01-16T00:12:46.769 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.769 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.769 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:46.769 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.089527+0000", 2020-01-16T00:12:46.769 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00039742100000000002 2020-01-16T00:12:46.770 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.770 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.770 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:46.770 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.089924+0000", 2020-01-16T00:12:46.770 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1369999999999999e-05 2020-01-16T00:12:46.770 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.770 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.771 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:46.771 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.089966+0000", 2020-01-16T00:12:46.771 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5207e-05 2020-01-16T00:12:46.771 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.771 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.771 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:46.771 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.089981+0000", 2020-01-16T00:12:46.771 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000104066 2020-01-16T00:12:46.772 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.772 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.772 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:46.772 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.090085+0000", 2020-01-16T00:12:46.772 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000234594 2020-01-16T00:12:46.772 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.772 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.772 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:46.773 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.090320+0000", 2020-01-16T00:12:46.773 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034746400000000002 2020-01-16T00:12:46.773 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.773 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.773 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,4", 2020-01-16T00:12:46.773 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.090667+0000", 2020-01-16T00:12:46.773 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0038092809999999999 2020-01-16T00:12:46.773 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.774 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.774 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:46.774 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.094476+0000", 2020-01-16T00:12:46.774 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00081967599999999998 2020-01-16T00:12:46.774 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.774 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.774 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:46.775 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.095296+0000", 2020-01-16T00:12:46.775 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00099452900000000007 2020-01-16T00:12:46.775 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.775 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.775 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:46.775 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.096291+0000", 2020-01-16T00:12:46.775 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8214999999999997e-05 2020-01-16T00:12:46.775 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.776 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.776 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:46.776 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.096349+0000", 2020-01-16T00:12:46.776 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3359000000000002e-05 2020-01-16T00:12:46.776 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.776 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.776 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:46.776 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.096402+0000", 2020-01-16T00:12:46.777 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0068691790000000004 2020-01-16T00:12:46.777 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.777 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:46.777 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.777 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.777 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.777 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1249 3.7 3:e8af1ebf:::smithi12313934-29:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:12:46.777 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:42.603441+0000", 2020-01-16T00:12:46.778 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 3.9678313009999999, 2020-01-16T00:12:46.778 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0065309820000000003, 2020-01-16T00:12:46.778 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:46.778 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:46.778 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:46.778 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:46.778 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:46.778 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1249 2020-01-16T00:12:46.779 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.779 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:46.779 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.779 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:46.779 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.603441+0000", 2020-01-16T00:12:46.779 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:46.779 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.780 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.780 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:46.780 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.603441+0000", 2020-01-16T00:12:46.780 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:46.780 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.780 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.780 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:46.780 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.603440+0000", 2020-01-16T00:12:46.781 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9153000000000001e-05 2020-01-16T00:12:46.781 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.781 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.781 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:46.781 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.603459+0000", 2020-01-16T00:12:46.781 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5519999999999993e-06 2020-01-16T00:12:46.781 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.781 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.782 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:46.782 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.603469+0000", 2020-01-16T00:12:46.782 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3890000000000001e-05 2020-01-16T00:12:46.782 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.782 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.782 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:46.782 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.603482+0000", 2020-01-16T00:12:46.782 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0032714229999999999 2020-01-16T00:12:46.783 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.783 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.783 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:46.783 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.606754+0000", 2020-01-16T00:12:46.783 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000116247 2020-01-16T00:12:46.783 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.783 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.783 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:46.784 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.606870+0000", 2020-01-16T00:12:46.784 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031019569999999998 2020-01-16T00:12:46.784 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.784 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.784 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:46.784 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.609972+0000", 2020-01-16T00:12:46.784 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0065309820000000003 2020-01-16T00:12:46.784 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.785 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:46.785 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.785 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.785 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.785 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1250 3.7 3:e8af1ebf:::smithi12313934-29:head [read 0~2470596,checksum] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:12:46.785 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:42.603510+0000", 2020-01-16T00:12:46.785 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 3.9677624360000001, 2020-01-16T00:12:46.785 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0087977060000000006, 2020-01-16T00:12:46.786 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:46.786 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:46.786 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:46.786 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:46.786 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:46.786 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1250 2020-01-16T00:12:46.786 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.787 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:46.787 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.787 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:46.787 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.603510+0000", 2020-01-16T00:12:46.787 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:46.787 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.787 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.787 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:46.788 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.603510+0000", 2020-01-16T00:12:46.788 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:12:46.788 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.788 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.788 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:46.788 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.603509+0000", 2020-01-16T00:12:46.788 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0569999999999993e-06 2020-01-16T00:12:46.788 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.789 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.789 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:46.789 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.603517+0000", 2020-01-16T00:12:46.789 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0504e-05 2020-01-16T00:12:46.789 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.789 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.789 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:46.789 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.603527+0000", 2020-01-16T00:12:46.790 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4000000000000003e-06 2020-01-16T00:12:46.790 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.790 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.790 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:46.790 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.603535+0000", 2020-01-16T00:12:46.790 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0064815130000000004 2020-01-16T00:12:46.790 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.790 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.791 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:46.791 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.610016+0000", 2020-01-16T00:12:46.791 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014758600000000001 2020-01-16T00:12:46.791 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.791 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.791 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:46.791 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.610164+0000", 2020-01-16T00:12:46.791 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021437019999999999 2020-01-16T00:12:46.792 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.792 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.792 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:46.792 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.612308+0000", 2020-01-16T00:12:46.792 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0087977060000000006 2020-01-16T00:12:46.792 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.792 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:46.792 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.793 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.793 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1263 3.8 3:113947f2:::smithi12313934-12:head [write 433647~450424] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:12:46.793 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:42.839750+0000", 2020-01-16T00:12:46.793 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 3.7315226319999999, 2020-01-16T00:12:46.793 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095742199999999996, 2020-01-16T00:12:46.793 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:46.793 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:46.794 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:12:46.794 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:12:46.794 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:46.794 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1263 2020-01-16T00:12:46.794 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.794 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:46.794 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.795 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:46.795 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.839750+0000", 2020-01-16T00:12:46.795 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:46.795 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.795 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.795 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:46.795 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.839750+0000", 2020-01-16T00:12:46.795 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999943 2020-01-16T00:12:46.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.796 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.796 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:46.796 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.839744+0000", 2020-01-16T00:12:46.796 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00030987000000000001 2020-01-16T00:12:46.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.796 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.796 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:46.797 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.840054+0000", 2020-01-16T00:12:46.797 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0200000000000001e-05 2020-01-16T00:12:46.797 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.797 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.797 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:46.797 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.840094+0000", 2020-01-16T00:12:46.797 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5480000000000001e-05 2020-01-16T00:12:46.798 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.798 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.798 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:46.798 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.840110+0000", 2020-01-16T00:12:46.798 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7233999999999999e-05 2020-01-16T00:12:46.798 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.798 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.799 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:46.799 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.840197+0000", 2020-01-16T00:12:46.799 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012238700000000001 2020-01-16T00:12:46.799 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.799 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.799 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:46.799 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.840319+0000", 2020-01-16T00:12:46.799 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017505999999999999 2020-01-16T00:12:46.800 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.800 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.800 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1,7", 2020-01-16T00:12:46.800 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.840495+0000", 2020-01-16T00:12:46.800 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001580347 2020-01-16T00:12:46.800 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.800 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.801 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:46.801 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.842075+0000", 2020-01-16T00:12:46.801 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0037933849999999998 2020-01-16T00:12:46.801 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.801 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.801 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:46.801 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.845868+0000", 2020-01-16T00:12:46.801 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033479339999999999 2020-01-16T00:12:46.802 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.802 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.802 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:46.802 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.849216+0000", 2020-01-16T00:12:46.802 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7949999999999999e-05 2020-01-16T00:12:46.802 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.802 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.803 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:46.803 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.849274+0000", 2020-01-16T00:12:46.803 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9889000000000003e-05 2020-01-16T00:12:46.803 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:46.803 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:46.803 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:46.803 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:42.849324+0000", 2020-01-16T00:12:46.803 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095742199999999996 2020-01-16T00:12:46.804 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.804 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:46.804 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.804 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:46.804 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:46.804 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:46.808 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:12:46.809 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-01-16T00:12:46.809 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-01-16T00:12:46.809 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-01-16T00:12:46.809 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-01-16T00:12:46.810 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-01-16T00:12:46.810 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-01-16T00:12:46.810 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-01-16T00:12:46.810 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-01-16T00:12:46.810 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-01-16T00:12:46.810 INFO:tasks.rados.rados.0.smithi123.stdout:766: finishing write tid 1 to smithi12313934-31 2020-01-16T00:12:46.810 INFO:tasks.rados.rados.0.smithi123.stdout:766: finishing write tid 2 to smithi12313934-31 2020-01-16T00:12:46.811 INFO:tasks.rados.rados.0.smithi123.stdout:766: finishing write tid 4 to smithi12313934-31 2020-01-16T00:12:46.811 INFO:tasks.rados.rados.0.smithi123.stdout:766: finishing write tid 5 to smithi12313934-31 2020-01-16T00:12:46.811 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 31 v 85 (ObjNum 132 snap 0 seq_num 132) dirty exists 2020-01-16T00:12:46.811 INFO:tasks.rados.rados.0.smithi123.stdout:766: left oid 31 (ObjNum 132 snap 0 seq_num 132) 2020-01-16T00:12:46.811 INFO:tasks.rados.rados.0.smithi123.stdout:764: done (9 left) 2020-01-16T00:12:46.811 INFO:tasks.rados.rados.0.smithi123.stdout:766: done (8 left) 2020-01-16T00:12:46.811 INFO:tasks.rados.rados.0.smithi123.stdout:774: done (7 left) 2020-01-16T00:12:46.811 INFO:tasks.rados.rados.0.smithi123.stdout:775: read oid 36 snap -1 2020-01-16T00:12:46.812 INFO:tasks.rados.rados.0.smithi123.stdout:775: expect (ObjNum 94 snap 0 seq_num 94) 2020-01-16T00:12:46.812 INFO:tasks.rados.rados.0.smithi123.stdout:776: read oid 29 snap -1 2020-01-16T00:12:46.812 INFO:tasks.rados.rados.0.smithi123.stdout:776: expect (ObjNum 28 snap 0 seq_num 28) 2020-01-16T00:12:46.812 INFO:tasks.rados.rados.0.smithi123.stdout:777: write (excl) oid 35 current snap is 0 2020-01-16T00:12:46.812 INFO:tasks.rados.rados.0.smithi123.stdout:777: seq_num 136 ranges {685752=611865,1975551=310201} 2020-01-16T00:12:46.812 INFO:tasks.rados.rados.0.smithi123.stdout:777: writing smithi12313934-35 from 685752 to 1297617 tid 1 2020-01-16T00:12:46.812 INFO:tasks.rados.rados.0.smithi123.stdout:777: writing smithi12313934-35 from 1975551 to 2285752 tid 2 2020-01-16T00:12:46.812 INFO:tasks.rados.rados.0.smithi123.stdout:778: write oid 6 current snap is 0 2020-01-16T00:12:46.813 INFO:tasks.rados.rados.0.smithi123.stdout:778: seq_num 137 ranges {702559=403467,1750386=495032} 2020-01-16T00:12:46.813 INFO:tasks.rados.rados.0.smithi123.stdout:778: writing smithi12313934-6 from 702559 to 1106026 tid 1 2020-01-16T00:12:46.813 INFO:tasks.rados.rados.0.smithi123.stdout:778: writing smithi12313934-6 from 1750386 to 2245418 tid 2 2020-01-16T00:12:46.813 INFO:tasks.rados.rados.0.smithi123.stdout:779: read oid 32 snap -1 2020-01-16T00:12:46.813 INFO:tasks.rados.rados.0.smithi123.stdout:779: expect (ObjNum 31 snap 0 seq_num 31) 2020-01-16T00:12:46.813 INFO:tasks.rados.rados.0.smithi123.stdout:780: read oid 2 snap -1 2020-01-16T00:12:46.813 INFO:tasks.rados.rados.0.smithi123.stdout:780: expect (ObjNum 63 snap 0 seq_num 63) 2020-01-16T00:12:46.814 INFO:tasks.rados.rados.0.smithi123.stdout:781: write oid 40 current snap is 0 2020-01-16T00:12:46.814 INFO:tasks.rados.rados.0.smithi123.stdout:781: seq_num 138 ranges {719366=478716,1808868=510498} 2020-01-16T00:12:46.814 INFO:tasks.rados.rados.0.smithi123.stdout:781: writing smithi12313934-40 from 719366 to 1198082 tid 1 2020-01-16T00:12:46.814 INFO:tasks.rados.rados.0.smithi123.stdout:781: writing smithi12313934-40 from 1808868 to 2319366 tid 2 2020-01-16T00:12:46.814 INFO:tasks.rados.rados.0.smithi123.stdout:782: write oid 41 current snap is 0 2020-01-16T00:12:46.814 INFO:tasks.rados.rados.0.smithi123.stdout:782: seq_num 139 ranges {736173=553965,1867350=468823} 2020-01-16T00:12:46.814 INFO:tasks.rados.rados.0.smithi123.stdout:782: writing smithi12313934-41 from 736173 to 1290138 tid 1 2020-01-16T00:12:46.814 INFO:tasks.rados.rados.0.smithi123.stdout:782: writing smithi12313934-41 from 1867350 to 2336173 tid 2 2020-01-16T00:12:46.815 INFO:tasks.rados.rados.0.smithi123.stdout:783: write (excl) oid 7 current snap is 0 2020-01-16T00:12:46.815 INFO:tasks.rados.rados.0.smithi123.stdout:783: seq_num 140 ranges {752980=629214,1809479=543501} 2020-01-16T00:12:46.815 INFO:tasks.rados.rados.0.smithi123.stdout:783: writing smithi12313934-7 from 752980 to 1382194 tid 1 2020-01-16T00:12:46.815 INFO:tasks.rados.rados.0.smithi123.stdout:783: writing smithi12313934-7 from 1809479 to 2352980 tid 2 2020-01-16T00:12:46.815 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:46.815 INFO:tasks.rados.rados.0.smithi123.stdout:767: finishing write tid 1 to smithi12313934-47 2020-01-16T00:12:46.815 INFO:tasks.rados.rados.0.smithi123.stdout:767: finishing write tid 2 to smithi12313934-47 2020-01-16T00:12:46.815 INFO:tasks.rados.rados.0.smithi123.stdout:767: finishing write tid 4 to smithi12313934-47 2020-01-16T00:12:46.816 INFO:tasks.rados.rados.0.smithi123.stdout:767: finishing write tid 5 to smithi12313934-47 2020-01-16T00:12:46.816 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 47 v 105 (ObjNum 133 snap 0 seq_num 133) dirty exists 2020-01-16T00:12:46.816 INFO:tasks.rados.rados.0.smithi123.stdout:767: left oid 47 (ObjNum 133 snap 0 seq_num 133) 2020-01-16T00:12:46.816 INFO:tasks.rados.rados.0.smithi123.stdout:768: expect (ObjNum 60 snap 0 seq_num 60) 2020-01-16T00:12:46.816 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:46.807+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:12:46.817 INFO:tasks.rados.rados.0.smithi123.stdout:769: finishing write tid 1 to smithi12313934-5 2020-01-16T00:12:46.817 INFO:tasks.rados.rados.0.smithi123.stdout:769: finishing write tid 2 to smithi12313934-5 2020-01-16T00:12:46.817 INFO:tasks.rados.rados.0.smithi123.stdout:771: expect (ObjNum 124 snap 0 seq_num 124) 2020-01-16T00:12:46.905 INFO:tasks.rados.rados.0.smithi123.stdout:769: finishing write tid 4 to smithi12313934-5 2020-01-16T00:12:46.905 INFO:tasks.rados.rados.0.smithi123.stdout:769: finishing write tid 5 to smithi12313934-5 2020-01-16T00:12:46.905 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 5 v 109 (ObjNum 134 snap 0 seq_num 134) dirty exists 2020-01-16T00:12:46.905 INFO:tasks.rados.rados.0.smithi123.stdout:769: left oid 5 (ObjNum 134 snap 0 seq_num 134) 2020-01-16T00:12:46.905 INFO:tasks.rados.rados.0.smithi123.stdout:770: expect (ObjNum 10 snap 0 seq_num 10) 2020-01-16T00:12:46.907 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:46.909+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:12:46.932 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:46.933 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:12:46.933 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:12:46.933 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:46.941 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-01-16T00:12:46.971 INFO:tasks.rados.rados.0.smithi123.stdout:772: expect (ObjNum 73 snap 0 seq_num 73) 2020-01-16T00:12:47.008 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:12:47.011+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:12:47.039 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:47.039 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:12:47.040 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:12:47.040 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:12:47.040 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:47.049 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-01-16T00:12:47.104 INFO:tasks.rados.rados.0.smithi123.stdout:773: expect (ObjNum 62 snap 0 seq_num 62) 2020-01-16T00:12:47.109 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:12:47.111+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:12:47.191 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:47.191 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:12:47.191 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:12:47.191 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:12:47.192 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.192 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:9 2.1 2:aa80ea11:::smithi12313934-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [read 453632~1075200 [fadvise_sequential+fadvise_dontneed]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e24)", 2020-01-16T00:12:47.192 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.123601+0000", 2020-01-16T00:12:47.192 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 10.069303943, 2020-01-16T00:12:47.192 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041686819999999999, 2020-01-16T00:12:47.192 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:47.192 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:47.193 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:47.193 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.193 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:47.193 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123601+0000", 2020-01-16T00:12:47.193 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:47.193 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.193 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.193 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:47.194 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123601+0000", 2020-01-16T00:12:47.194 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:12:47.194 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.194 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.194 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:47.194 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123600+0000", 2020-01-16T00:12:47.194 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0259999999999996e-06 2020-01-16T00:12:47.195 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.195 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.195 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:47.195 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123609+0000", 2020-01-16T00:12:47.195 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2979999999999992e-06 2020-01-16T00:12:47.195 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.195 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.195 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:47.196 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123617+0000", 2020-01-16T00:12:47.196 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3850000000000002e-06 2020-01-16T00:12:47.196 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.196 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.196 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:47.196 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123626+0000", 2020-01-16T00:12:47.196 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0027674969999999998 2020-01-16T00:12:47.196 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.197 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.197 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:47.197 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.126393+0000", 2020-01-16T00:12:47.197 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013459499999999999 2020-01-16T00:12:47.197 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.197 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.197 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:47.198 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.126528+0000", 2020-01-16T00:12:47.198 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012418900000000001 2020-01-16T00:12:47.198 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.198 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.198 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:47.199 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.127770+0000", 2020-01-16T00:12:47.199 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041686819999999999 2020-01-16T00:12:47.199 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.199 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:47.199 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.199 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.199 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.200 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:10 2.1 2:aa80ea11:::smithi12313934-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [read 1658880~398336 [fadvise_sequential+fadvise_dontneed]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e24)", 2020-01-16T00:12:47.200 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.123646+0000", 2020-01-16T00:12:47.200 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 10.069258662999999, 2020-01-16T00:12:47.200 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052215789999999996, 2020-01-16T00:12:47.200 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:47.200 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:47.200 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:47.201 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.201 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:47.201 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123646+0000", 2020-01-16T00:12:47.201 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:47.201 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.201 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.201 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:47.201 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123646+0000", 2020-01-16T00:12:47.202 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:12:47.202 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.202 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.202 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:47.202 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123645+0000", 2020-01-16T00:12:47.202 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7880000000000003e-06 2020-01-16T00:12:47.202 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.203 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.203 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:47.203 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123651+0000", 2020-01-16T00:12:47.203 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.218e-05 2020-01-16T00:12:47.203 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.203 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.203 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:47.203 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123663+0000", 2020-01-16T00:12:47.204 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0930999999999999e-05 2020-01-16T00:12:47.204 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.204 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.204 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:47.204 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123674+0000", 2020-01-16T00:12:47.204 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041493830000000004 2020-01-16T00:12:47.204 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.205 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.205 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:47.205 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.127824+0000", 2020-01-16T00:12:47.205 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00025807199999999998 2020-01-16T00:12:47.205 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.205 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.205 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:47.205 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.128082+0000", 2020-01-16T00:12:47.206 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00078618000000000004 2020-01-16T00:12:47.206 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.206 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.206 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:47.206 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.128868+0000", 2020-01-16T00:12:47.206 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052215789999999996 2020-01-16T00:12:47.206 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.206 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:47.207 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.207 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.207 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.207 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:910 3.5 e24/20 3:ad8ab15f:::smithi12313934-27:head v 24'49, mlcod=23'46)", 2020-01-16T00:12:47.207 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.845784+0000", 2020-01-16T00:12:47.207 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 9.3471213889999998, 2020-01-16T00:12:47.207 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0032848759999999999, 2020-01-16T00:12:47.208 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:47.208 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:47.208 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:47.208 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.208 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:47.208 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.845784+0000", 2020-01-16T00:12:47.208 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:47.208 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.209 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.209 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:47.209 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.845784+0000", 2020-01-16T00:12:47.209 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:12:47.209 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.209 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.209 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:47.209 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.845782+0000", 2020-01-16T00:12:47.210 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001034952 2020-01-16T00:12:47.210 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.210 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.210 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:47.210 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.846817+0000", 2020-01-16T00:12:47.210 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4044e-05 2020-01-16T00:12:47.210 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.211 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.211 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:47.211 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.846831+0000", 2020-01-16T00:12:47.211 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8999999999999997e-06 2020-01-16T00:12:47.211 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.211 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.211 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:47.211 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.846836+0000", 2020-01-16T00:12:47.212 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0311000000000001e-05 2020-01-16T00:12:47.212 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.212 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.212 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:47.212 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.846877+0000", 2020-01-16T00:12:47.212 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.745e-05 2020-01-16T00:12:47.212 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.212 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.213 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:47.213 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.846904+0000", 2020-01-16T00:12:47.213 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002122667 2020-01-16T00:12:47.213 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.213 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.213 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:47.213 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.849027+0000", 2020-01-16T00:12:47.214 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1607999999999999e-05 2020-01-16T00:12:47.214 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.214 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.214 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:47.214 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.849068+0000", 2020-01-16T00:12:47.214 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0032848759999999999 2020-01-16T00:12:47.214 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.214 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:47.215 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.215 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.215 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.215 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.1.13:28 2.3 e24/19 2:de10f4fa:::smithi12313934-4 low_tier:head v 24'20, mlcod=23'18)", 2020-01-16T00:12:47.215 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.915072+0000", 2020-01-16T00:12:47.215 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 9.2778326999999994, 2020-01-16T00:12:47.215 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003575015, 2020-01-16T00:12:47.216 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:47.216 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:47.216 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:47.216 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.216 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:47.216 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915072+0000", 2020-01-16T00:12:47.216 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:47.216 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.217 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.217 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:47.217 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915072+0000", 2020-01-16T00:12:47.217 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:47.217 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.217 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.217 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:47.218 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915070+0000", 2020-01-16T00:12:47.218 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000111706 2020-01-16T00:12:47.218 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.218 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.218 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:47.219 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915182+0000", 2020-01-16T00:12:47.219 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1409999999999999e-05 2020-01-16T00:12:47.219 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.219 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.219 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:47.219 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915193+0000", 2020-01-16T00:12:47.219 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0713999999999999e-05 2020-01-16T00:12:47.220 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.220 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.220 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:47.220 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915204+0000", 2020-01-16T00:12:47.220 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0732000000000002e-05 2020-01-16T00:12:47.220 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.220 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.220 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:47.221 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915245+0000", 2020-01-16T00:12:47.221 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7824e-05 2020-01-16T00:12:47.221 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.221 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.221 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:47.221 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915272+0000", 2020-01-16T00:12:47.221 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033395159999999998 2020-01-16T00:12:47.221 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.222 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.222 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:47.222 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.918612+0000", 2020-01-16T00:12:47.222 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5345000000000001e-05 2020-01-16T00:12:47.222 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.222 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.222 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:47.223 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.918647+0000", 2020-01-16T00:12:47.223 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003575015 2020-01-16T00:12:47.223 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.223 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:47.223 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.223 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.223 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.223 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.4.0:112 3.9 e24/20 3:9218e7f1:::smithi12313934-14:head v 24'72, mlcod=23'70)", 2020-01-16T00:12:47.224 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.925123+0000", 2020-01-16T00:12:47.224 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 9.2677817010000005, 2020-01-16T00:12:47.224 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.004957024, 2020-01-16T00:12:47.224 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:47.224 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:47.224 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:47.224 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.231 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:47.231 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.925123+0000", 2020-01-16T00:12:47.231 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:47.231 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.231 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.232 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:47.232 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.925123+0000", 2020-01-16T00:12:47.232 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:47.232 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.232 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.232 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:47.232 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.925121+0000", 2020-01-16T00:12:47.232 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0024713119999999998 2020-01-16T00:12:47.233 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.233 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.233 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:47.233 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927593+0000", 2020-01-16T00:12:47.233 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7319999999999993e-06 2020-01-16T00:12:47.233 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.233 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.234 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:47.234 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927603+0000", 2020-01-16T00:12:47.234 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6600000000000003e-06 2020-01-16T00:12:47.234 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.234 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.234 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:47.234 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927607+0000", 2020-01-16T00:12:47.234 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7138999999999998e-05 2020-01-16T00:12:47.235 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.235 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.235 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:47.235 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927644+0000", 2020-01-16T00:12:47.235 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7068999999999999e-05 2020-01-16T00:12:47.235 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.235 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.236 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:47.236 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927661+0000", 2020-01-16T00:12:47.236 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0023781840000000002 2020-01-16T00:12:47.236 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.236 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.236 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:47.236 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.930040+0000", 2020-01-16T00:12:47.237 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0679000000000001e-05 2020-01-16T00:12:47.237 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.237 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.237 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:47.237 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.930080+0000", 2020-01-16T00:12:47.237 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.004957024 2020-01-16T00:12:47.237 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.238 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:47.238 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.238 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.238 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.238 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.1.0:162 3.0 e24/20 3:00c146d5:::smithi12313934-18:head v 24'99, mlcod=24'97)", 2020-01-16T00:12:47.238 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.039167+0000", 2020-01-16T00:12:47.238 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 9.1537384159999995, 2020-01-16T00:12:47.238 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0049106159999999996, 2020-01-16T00:12:47.239 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:47.239 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:47.239 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:47.239 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.239 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:47.239 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.039167+0000", 2020-01-16T00:12:47.239 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:47.239 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.240 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.240 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:47.240 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.039167+0000", 2020-01-16T00:12:47.240 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999957 2020-01-16T00:12:47.240 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.240 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.240 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:47.241 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.039162+0000", 2020-01-16T00:12:47.241 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000551155 2020-01-16T00:12:47.241 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.241 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.241 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:47.241 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.039713+0000", 2020-01-16T00:12:47.241 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1465999999999999e-05 2020-01-16T00:12:47.241 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.242 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.242 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:47.242 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.039755+0000", 2020-01-16T00:12:47.242 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7122999999999999e-05 2020-01-16T00:12:47.242 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.242 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.242 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:47.243 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.039772+0000", 2020-01-16T00:12:47.243 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6485999999999997e-05 2020-01-16T00:12:47.243 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.243 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.243 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:47.243 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.039868+0000", 2020-01-16T00:12:47.243 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1465999999999996e-05 2020-01-16T00:12:47.243 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.244 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.244 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:47.244 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.039940+0000", 2020-01-16T00:12:47.244 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040611019999999996 2020-01-16T00:12:47.244 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.244 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.244 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:47.245 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.044001+0000", 2020-01-16T00:12:47.245 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6151999999999999e-05 2020-01-16T00:12:47.245 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.245 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.245 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:47.245 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.044077+0000", 2020-01-16T00:12:47.245 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0049106159999999996 2020-01-16T00:12:47.246 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.246 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:47.246 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.246 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.246 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.246 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:957 3.0 e24/20 3:00c146d5:::smithi12313934-18:head v 24'100, mlcod=24'97)", 2020-01-16T00:12:47.246 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.042105+0000", 2020-01-16T00:12:47.246 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 9.1507996659999993, 2020-01-16T00:12:47.247 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0046019770000000002, 2020-01-16T00:12:47.247 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:47.247 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:47.247 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:47.247 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.247 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:47.247 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.042105+0000", 2020-01-16T00:12:47.248 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:47.248 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.248 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.248 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:47.248 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.042105+0000", 2020-01-16T00:12:47.248 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999962 2020-01-16T00:12:47.248 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.249 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.249 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:47.249 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.042101+0000", 2020-01-16T00:12:47.249 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00041373799999999997 2020-01-16T00:12:47.249 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.249 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.249 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:47.249 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.042515+0000", 2020-01-16T00:12:47.250 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.858e-05 2020-01-16T00:12:47.250 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.250 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.250 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:47.250 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.042554+0000", 2020-01-16T00:12:47.250 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5400000000000002e-05 2020-01-16T00:12:47.250 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.251 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.251 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:47.251 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.042569+0000", 2020-01-16T00:12:47.251 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1859999999999994e-05 2020-01-16T00:12:47.251 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.251 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.252 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:47.252 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.042631+0000", 2020-01-16T00:12:47.252 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7620999999999999e-05 2020-01-16T00:12:47.252 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.252 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.252 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:47.252 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.042658+0000", 2020-01-16T00:12:47.252 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040138750000000001 2020-01-16T00:12:47.253 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.253 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.253 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:47.253 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.046672+0000", 2020-01-16T00:12:47.253 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4952e-05 2020-01-16T00:12:47.253 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.253 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.254 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:47.254 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.046707+0000", 2020-01-16T00:12:47.254 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0046019770000000002 2020-01-16T00:12:47.254 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.254 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:47.254 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.254 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.254 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.255 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:958 3.0 e24/20 3:00c146d5:::smithi12313934-18:head v 24'101, mlcod=24'97)", 2020-01-16T00:12:47.255 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.044214+0000", 2020-01-16T00:12:47.255 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 9.1486908899999992, 2020-01-16T00:12:47.255 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033269810000000001, 2020-01-16T00:12:47.255 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:47.255 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:47.255 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:47.256 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.256 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:47.256 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.044214+0000", 2020-01-16T00:12:47.256 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:47.256 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.256 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.257 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:47.257 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.044214+0000", 2020-01-16T00:12:47.257 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:47.257 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.257 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.257 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:47.257 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.044212+0000", 2020-01-16T00:12:47.258 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016133 2020-01-16T00:12:47.258 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.258 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.258 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:47.258 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.044374+0000", 2020-01-16T00:12:47.258 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0190000000000006e-06 2020-01-16T00:12:47.258 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.258 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.259 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:47.259 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.044382+0000", 2020-01-16T00:12:47.259 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1409999999999998e-06 2020-01-16T00:12:47.259 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.259 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.259 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:47.259 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.044386+0000", 2020-01-16T00:12:47.260 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5098999999999997e-05 2020-01-16T00:12:47.260 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.260 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.260 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:47.260 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.044421+0000", 2020-01-16T00:12:47.260 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.4280000000000001e-05 2020-01-16T00:12:47.260 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.260 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.261 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:47.261 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.044445+0000", 2020-01-16T00:12:47.261 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030470100000000002 2020-01-16T00:12:47.261 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.261 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.261 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:47.261 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.047492+0000", 2020-01-16T00:12:47.261 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8952999999999997e-05 2020-01-16T00:12:47.262 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.262 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.262 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:47.262 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.047541+0000", 2020-01-16T00:12:47.262 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033269810000000001 2020-01-16T00:12:47.262 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.262 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:47.263 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.263 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.263 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.263 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:959 3.0 e24/20 3:00c146d5:::smithi12313934-18:head v 24'102, mlcod=24'98)", 2020-01-16T00:12:47.263 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.047434+0000", 2020-01-16T00:12:47.263 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 9.1454714189999997, 2020-01-16T00:12:47.263 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0038721860000000001, 2020-01-16T00:12:47.263 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:47.264 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:47.264 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:47.264 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.264 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:47.264 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.047434+0000", 2020-01-16T00:12:47.264 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:47.264 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.264 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.265 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:47.265 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.047434+0000", 2020-01-16T00:12:47.265 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:47.265 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.265 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.265 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:47.265 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.047432+0000", 2020-01-16T00:12:47.266 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00027367899999999998 2020-01-16T00:12:47.266 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.266 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.266 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:47.266 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.047705+0000", 2020-01-16T00:12:47.266 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1781e-05 2020-01-16T00:12:47.266 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.266 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.267 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:47.267 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.047717+0000", 2020-01-16T00:12:47.267 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5249999999999996e-06 2020-01-16T00:12:47.267 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.267 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.267 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:47.267 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.047723+0000", 2020-01-16T00:12:47.267 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7784999999999998e-05 2020-01-16T00:12:47.268 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.268 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.268 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:47.268 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.047760+0000", 2020-01-16T00:12:47.268 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2864999999999999e-05 2020-01-16T00:12:47.268 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.268 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.268 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:47.269 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.047783+0000", 2020-01-16T00:12:47.269 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0034841450000000001 2020-01-16T00:12:47.269 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.269 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.269 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:47.269 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.051268+0000", 2020-01-16T00:12:47.269 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.82e-05 2020-01-16T00:12:47.270 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.270 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.270 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:47.270 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.051306+0000", 2020-01-16T00:12:47.270 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0038721860000000001 2020-01-16T00:12:47.270 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.270 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:47.270 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.271 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.271 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.271 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:973 3.5 e24/20 3:aef1ce98:::smithi12313934-9:head v 24'53, mlcod=24'51)", 2020-01-16T00:12:47.271 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.305124+0000", 2020-01-16T00:12:47.271 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 8.8877805799999994, 2020-01-16T00:12:47.271 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036133279999999999, 2020-01-16T00:12:47.271 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:47.271 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:47.272 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:47.272 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.272 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:47.272 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.305124+0000", 2020-01-16T00:12:47.272 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:47.272 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.272 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.273 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:47.273 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.305124+0000", 2020-01-16T00:12:47.273 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999857 2020-01-16T00:12:47.273 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.273 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.273 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:47.273 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.305110+0000", 2020-01-16T00:12:47.273 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013042819999999999 2020-01-16T00:12:47.274 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.274 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.274 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:47.274 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.306414+0000", 2020-01-16T00:12:47.274 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2325000000000001e-05 2020-01-16T00:12:47.274 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.274 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.275 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:47.275 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.306437+0000", 2020-01-16T00:12:47.275 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1420000000000001e-05 2020-01-16T00:12:47.275 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.275 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.275 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:47.275 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.306448+0000", 2020-01-16T00:12:47.275 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5415999999999997e-05 2020-01-16T00:12:47.276 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.276 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.276 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:47.276 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.306534+0000", 2020-01-16T00:12:47.276 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4882000000000002e-05 2020-01-16T00:12:47.276 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.276 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.276 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:47.277 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.306568+0000", 2020-01-16T00:12:47.277 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021100450000000001 2020-01-16T00:12:47.277 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.277 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.277 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:47.277 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.308678+0000", 2020-01-16T00:12:47.277 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9237999999999998e-05 2020-01-16T00:12:47.278 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.278 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.278 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:47.278 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.308738+0000", 2020-01-16T00:12:47.278 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036133279999999999 2020-01-16T00:12:47.278 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.278 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:47.278 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.279 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.279 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.279 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.4.0:114 3.9 e24/20 3:9fc687b2:::smithi12313934-44:head v 24'74, mlcod=24'72)", 2020-01-16T00:12:47.279 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:12:47.279 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 8.8676261919999995, 2020-01-16T00:12:47.279 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010399429, 2020-01-16T00:12:47.279 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:47.280 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:47.280 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:47.280 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.280 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:47.280 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:12:47.280 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:47.280 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.280 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.281 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:47.281 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:12:47.281 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:12:47.281 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.281 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.281 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:47.281 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325276+0000", 2020-01-16T00:12:47.281 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0046127069999999997 2020-01-16T00:12:47.282 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.282 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.282 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:47.282 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329889+0000", 2020-01-16T00:12:47.282 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3519999999999998e-05 2020-01-16T00:12:47.282 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.282 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.283 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:47.283 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329912+0000", 2020-01-16T00:12:47.283 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4070000000000004e-06 2020-01-16T00:12:47.283 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.283 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.283 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:47.283 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329920+0000", 2020-01-16T00:12:47.283 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8817000000000003e-05 2020-01-16T00:12:47.284 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.284 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.284 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:47.284 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329978+0000", 2020-01-16T00:12:47.284 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9721e-05 2020-01-16T00:12:47.284 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.284 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.284 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:47.285 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329998+0000", 2020-01-16T00:12:47.285 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056328139999999999 2020-01-16T00:12:47.285 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.285 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.285 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:47.285 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.335631+0000", 2020-01-16T00:12:47.285 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7170999999999999e-05 2020-01-16T00:12:47.286 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.286 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.286 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:47.286 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.335678+0000", 2020-01-16T00:12:47.286 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010399429 2020-01-16T00:12:47.286 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.286 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:47.286 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.287 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.287 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.287 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:24 2.1 2:8b431f80:::smithi12313934-40 low_tier:head [write 790528~688128] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:12:47.287 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.482303+0000", 2020-01-16T00:12:47.287 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 8.7106020960000006, 2020-01-16T00:12:47.287 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010071853, 2020-01-16T00:12:47.287 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:47.288 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:47.288 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:47.288 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.288 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:47.288 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.482303+0000", 2020-01-16T00:12:47.288 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:47.288 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.289 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.289 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:47.289 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.482303+0000", 2020-01-16T00:12:47.289 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:47.289 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.289 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.289 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:47.289 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.482301+0000", 2020-01-16T00:12:47.290 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031173960000000001 2020-01-16T00:12:47.290 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.290 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.290 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:47.290 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485418+0000", 2020-01-16T00:12:47.290 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9300000000000001e-05 2020-01-16T00:12:47.290 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.290 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.291 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:47.291 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485448+0000", 2020-01-16T00:12:47.291 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2611e-05 2020-01-16T00:12:47.291 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.291 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.291 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:47.291 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485460+0000", 2020-01-16T00:12:47.292 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112487 2020-01-16T00:12:47.292 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.292 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.292 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:47.292 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485573+0000", 2020-01-16T00:12:47.293 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022169299999999999 2020-01-16T00:12:47.293 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.293 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.294 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:47.294 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485794+0000", 2020-01-16T00:12:47.295 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00036287500000000002 2020-01-16T00:12:47.295 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.296 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.296 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:12:47.296 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.486157+0000", 2020-01-16T00:12:47.296 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040260310000000002 2020-01-16T00:12:47.296 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.296 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.296 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:47.297 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.490183+0000", 2020-01-16T00:12:47.297 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00081083000000000001 2020-01-16T00:12:47.297 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.297 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.297 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:47.297 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.490994+0000", 2020-01-16T00:12:47.297 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001287746 2020-01-16T00:12:47.297 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.298 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.298 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:47.298 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492282+0000", 2020-01-16T00:12:47.298 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8810999999999998e-05 2020-01-16T00:12:47.298 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.298 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.298 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:47.299 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492321+0000", 2020-01-16T00:12:47.299 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4095000000000003e-05 2020-01-16T00:12:47.299 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.299 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.299 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:47.299 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492375+0000", 2020-01-16T00:12:47.299 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010071853 2020-01-16T00:12:47.300 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.300 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:47.300 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.300 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.300 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.300 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:25 2.1 2:8b431f80:::smithi12313934-40 low_tier:head [write 1926656~67072] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:12:47.300 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.485481+0000", 2020-01-16T00:12:47.301 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 8.7074236309999993, 2020-01-16T00:12:47.301 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0081599020000000001, 2020-01-16T00:12:47.301 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:47.301 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:47.301 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:47.301 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.301 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:47.301 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485481+0000", 2020-01-16T00:12:47.302 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:47.302 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.302 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.302 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:47.302 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485481+0000", 2020-01-16T00:12:47.302 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:47.302 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.303 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.303 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:47.304 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485479+0000", 2020-01-16T00:12:47.304 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012423800000000001 2020-01-16T00:12:47.304 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.304 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.305 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:47.305 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485603+0000", 2020-01-16T00:12:47.305 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8366999999999999e-05 2020-01-16T00:12:47.305 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.305 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.305 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:47.305 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485621+0000", 2020-01-16T00:12:47.306 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0207000000000001e-05 2020-01-16T00:12:47.306 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.306 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.306 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:47.306 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485632+0000", 2020-01-16T00:12:47.306 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002868532 2020-01-16T00:12:47.306 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.306 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.307 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:47.307 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.488500+0000", 2020-01-16T00:12:47.307 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011340000000000001 2020-01-16T00:12:47.307 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.307 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.307 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:47.307 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.488614+0000", 2020-01-16T00:12:47.307 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019056300000000001 2020-01-16T00:12:47.308 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.308 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.308 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:12:47.308 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.488804+0000", 2020-01-16T00:12:47.308 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00153302 2020-01-16T00:12:47.308 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.308 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.308 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:47.309 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.490337+0000", 2020-01-16T00:12:47.309 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00089946100000000005 2020-01-16T00:12:47.309 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.309 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.309 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:47.309 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.491237+0000", 2020-01-16T00:12:47.309 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0022862820000000002 2020-01-16T00:12:47.310 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.310 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.310 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:47.310 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.493523+0000", 2020-01-16T00:12:47.310 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5609999999999998e-05 2020-01-16T00:12:47.310 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.310 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.310 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:47.311 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.493578+0000", 2020-01-16T00:12:47.311 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2770999999999999e-05 2020-01-16T00:12:47.311 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.311 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.311 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:47.311 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.493641+0000", 2020-01-16T00:12:47.311 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0081599020000000001 2020-01-16T00:12:47.311 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.312 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:47.312 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.312 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.312 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.313 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:1279 3.0 e26/20 3:05b2bf53:::smithi12313934-37:head v 26'137, mlcod=26'135)", 2020-01-16T00:12:47.313 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:43.082160+0000", 2020-01-16T00:12:47.313 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.1107448980000001, 2020-01-16T00:12:47.313 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0091277130000000008, 2020-01-16T00:12:47.313 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:47.313 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:47.314 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:47.314 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.314 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:47.314 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.082160+0000", 2020-01-16T00:12:47.314 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:47.314 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.314 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.315 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:47.315 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.082160+0000", 2020-01-16T00:12:47.315 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999957 2020-01-16T00:12:47.315 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.315 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.316 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:47.316 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.082156+0000", 2020-01-16T00:12:47.316 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00046771999999999997 2020-01-16T00:12:47.316 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.316 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.316 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:47.316 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.082623+0000", 2020-01-16T00:12:47.317 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3099999999999998e-05 2020-01-16T00:12:47.317 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.317 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.317 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:47.317 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.082656+0000", 2020-01-16T00:12:47.317 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2063e-05 2020-01-16T00:12:47.317 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.317 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.318 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:47.318 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.082668+0000", 2020-01-16T00:12:47.318 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021980900000000002 2020-01-16T00:12:47.318 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.318 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.318 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:47.318 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.084867+0000", 2020-01-16T00:12:47.319 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00033252299999999999 2020-01-16T00:12:47.319 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.319 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.319 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:47.319 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.085199+0000", 2020-01-16T00:12:47.319 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0060364019999999997 2020-01-16T00:12:47.319 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.319 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.320 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:47.320 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.091235+0000", 2020-01-16T00:12:47.320 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2255e-05 2020-01-16T00:12:47.320 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.320 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.320 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:47.320 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.091288+0000", 2020-01-16T00:12:47.320 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0091277130000000008 2020-01-16T00:12:47.321 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.321 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:47.321 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.321 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.321 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.321 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:1280 3.0 e27/20 3:05b2bf53:::smithi12313934-37:head v 27'138, mlcod=26'135)", 2020-01-16T00:12:47.321 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:43.088262+0000", 2020-01-16T00:12:47.322 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.1046433179999999, 2020-01-16T00:12:47.322 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036090330000000002, 2020-01-16T00:12:47.322 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:47.322 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:47.322 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:47.322 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.322 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:47.322 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.088262+0000", 2020-01-16T00:12:47.323 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:47.323 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.323 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.323 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:47.323 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.088262+0000", 2020-01-16T00:12:47.323 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:47.323 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.324 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.324 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:47.324 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.088261+0000", 2020-01-16T00:12:47.324 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000173606 2020-01-16T00:12:47.324 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.324 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.324 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:47.324 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.088435+0000", 2020-01-16T00:12:47.325 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4881e-05 2020-01-16T00:12:47.325 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.325 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.325 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:47.325 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.088450+0000", 2020-01-16T00:12:47.325 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2459999999999992e-06 2020-01-16T00:12:47.325 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.325 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.326 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:47.326 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.088459+0000", 2020-01-16T00:12:47.326 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001007066 2020-01-16T00:12:47.326 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.326 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.326 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:47.326 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.089466+0000", 2020-01-16T00:12:47.326 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6548999999999999e-05 2020-01-16T00:12:47.327 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.327 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.327 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:47.327 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.089502+0000", 2020-01-16T00:12:47.327 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0023328870000000001 2020-01-16T00:12:47.327 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.327 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.327 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:47.328 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.091835+0000", 2020-01-16T00:12:47.328 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.54e-05 2020-01-16T00:12:47.328 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.328 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.328 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:47.328 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.091871+0000", 2020-01-16T00:12:47.328 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036090330000000002 2020-01-16T00:12:47.328 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.329 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:47.329 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.329 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.329 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.329 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:52 2.1 2:9200d056:::smithi12313934-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 926720~423424] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e28)", 2020-01-16T00:12:47.329 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:46.629755+0000", 2020-01-16T00:12:47.329 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.56315024000000002, 2020-01-16T00:12:47.330 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063571069999999999, 2020-01-16T00:12:47.330 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:47.330 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:47.330 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:47.330 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.330 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:47.330 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.629755+0000", 2020-01-16T00:12:47.330 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:47.331 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.331 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.331 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:47.331 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.629755+0000", 2020-01-16T00:12:47.331 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:47.331 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.331 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.331 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:47.332 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.629753+0000", 2020-01-16T00:12:47.332 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000127501 2020-01-16T00:12:47.332 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.332 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.332 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:47.332 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.629880+0000", 2020-01-16T00:12:47.332 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2235000000000001e-05 2020-01-16T00:12:47.332 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.333 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.333 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:47.333 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.629892+0000", 2020-01-16T00:12:47.333 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6879999999999998e-06 2020-01-16T00:12:47.333 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.333 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.333 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:47.333 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.629897+0000", 2020-01-16T00:12:47.334 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000103049 2020-01-16T00:12:47.334 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.334 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.334 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:47.334 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630000+0000", 2020-01-16T00:12:47.334 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000250289 2020-01-16T00:12:47.334 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.334 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.335 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:47.335 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630250+0000", 2020-01-16T00:12:47.335 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019101 2020-01-16T00:12:47.335 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.335 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.335 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:12:47.335 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630441+0000", 2020-01-16T00:12:47.335 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0026126909999999999 2020-01-16T00:12:47.336 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.336 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.336 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:47.336 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.633054+0000", 2020-01-16T00:12:47.336 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0026195569999999998 2020-01-16T00:12:47.336 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.336 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.336 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:47.337 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.635674+0000", 2020-01-16T00:12:47.337 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020174 2020-01-16T00:12:47.337 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.337 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.337 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:47.337 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.635875+0000", 2020-01-16T00:12:47.337 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013694400000000001 2020-01-16T00:12:47.337 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.338 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.338 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:47.338 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.636012+0000", 2020-01-16T00:12:47.338 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9442000000000001e-05 2020-01-16T00:12:47.338 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.338 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.338 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:47.338 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.636112+0000", 2020-01-16T00:12:47.339 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063571069999999999 2020-01-16T00:12:47.339 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.339 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:47.339 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.339 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.339 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.339 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:53 2.1 2:9200d056:::smithi12313934-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 1823744~153600] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e28)", 2020-01-16T00:12:47.340 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:46.630287+0000", 2020-01-16T00:12:47.340 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.56261795299999995, 2020-01-16T00:12:47.340 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0059987900000000004, 2020-01-16T00:12:47.340 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:47.340 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:47.340 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:47.340 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.340 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:47.341 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630287+0000", 2020-01-16T00:12:47.341 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:47.341 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.341 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.341 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:47.341 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630287+0000", 2020-01-16T00:12:47.341 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:12:47.341 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.342 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.342 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:47.342 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630286+0000", 2020-01-16T00:12:47.342 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7387999999999999e-05 2020-01-16T00:12:47.342 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.342 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.342 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:47.342 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630353+0000", 2020-01-16T00:12:47.343 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.219e-05 2020-01-16T00:12:47.343 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.343 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.343 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:47.343 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630366+0000", 2020-01-16T00:12:47.343 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9020000000000001e-06 2020-01-16T00:12:47.343 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.344 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.344 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:47.344 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630369+0000", 2020-01-16T00:12:47.344 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001257942 2020-01-16T00:12:47.344 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.344 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.345 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:47.345 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.631627+0000", 2020-01-16T00:12:47.345 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000110399 2020-01-16T00:12:47.345 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.345 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.345 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:47.345 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.631738+0000", 2020-01-16T00:12:47.345 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020549199999999999 2020-01-16T00:12:47.346 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.346 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.346 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:12:47.346 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.631943+0000", 2020-01-16T00:12:47.346 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0015335290000000001 2020-01-16T00:12:47.346 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.346 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.346 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:47.347 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.633477+0000", 2020-01-16T00:12:47.347 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002315176 2020-01-16T00:12:47.347 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.347 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.347 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:47.347 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.635792+0000", 2020-01-16T00:12:47.347 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00042626900000000001 2020-01-16T00:12:47.347 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.348 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.348 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:47.348 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.636218+0000", 2020-01-16T00:12:47.348 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9233000000000002e-05 2020-01-16T00:12:47.348 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.348 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.348 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:47.348 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.636258+0000", 2020-01-16T00:12:47.349 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8257999999999999e-05 2020-01-16T00:12:47.349 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.349 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.349 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:47.349 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.636286+0000", 2020-01-16T00:12:47.349 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0059987900000000004 2020-01-16T00:12:47.349 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.349 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:47.350 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.350 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.350 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.350 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:54 2.1 2:9200d056:::smithi12313934-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 2168320~226816] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e28)", 2020-01-16T00:12:47.350 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:46.630988+0000", 2020-01-16T00:12:47.350 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.56191705700000005, 2020-01-16T00:12:47.350 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0062526489999999999, 2020-01-16T00:12:47.350 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:47.351 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:47.351 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:47.351 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.351 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:47.351 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630988+0000", 2020-01-16T00:12:47.351 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:47.351 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.351 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.352 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:47.352 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630988+0000", 2020-01-16T00:12:47.352 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:47.352 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.352 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.352 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:47.352 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630987+0000", 2020-01-16T00:12:47.352 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5952000000000006e-05 2020-01-16T00:12:47.353 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.353 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.353 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:47.353 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.631073+0000", 2020-01-16T00:12:47.353 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.172e-05 2020-01-16T00:12:47.353 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.353 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.354 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:47.354 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.631084+0000", 2020-01-16T00:12:47.354 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9619999999999993e-06 2020-01-16T00:12:47.354 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.354 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.354 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:47.354 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.631092+0000", 2020-01-16T00:12:47.354 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002514877 2020-01-16T00:12:47.355 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.355 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.355 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:47.355 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.633607+0000", 2020-01-16T00:12:47.355 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1843999999999999e-05 2020-01-16T00:12:47.355 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.355 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.355 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:47.356 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.633669+0000", 2020-01-16T00:12:47.356 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010064999999999999 2020-01-16T00:12:47.356 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.356 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.356 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:12:47.356 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.633770+0000", 2020-01-16T00:12:47.356 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0025793230000000001 2020-01-16T00:12:47.356 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.357 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.357 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:47.357 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.636349+0000", 2020-01-16T00:12:47.357 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013228400000000001 2020-01-16T00:12:47.357 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.357 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.357 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:47.357 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.636481+0000", 2020-01-16T00:12:47.358 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00069413899999999998 2020-01-16T00:12:47.358 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.358 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.358 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:47.358 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.637175+0000", 2020-01-16T00:12:47.359 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5463999999999998e-05 2020-01-16T00:12:47.359 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.359 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.359 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:47.359 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.637211+0000", 2020-01-16T00:12:47.359 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9675e-05 2020-01-16T00:12:47.359 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.360 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.360 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:47.360 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.637241+0000", 2020-01-16T00:12:47.360 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0062526489999999999 2020-01-16T00:12:47.360 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.360 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:47.360 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.360 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.361 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.361 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:38 2.1 2:8b431f80:::smithi12313934-40 low_tier:head [write 790528~688128] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e28)", 2020-01-16T00:12:47.361 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:46.654720+0000", 2020-01-16T00:12:47.361 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.53818462199999995, 2020-01-16T00:12:47.361 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0061792729999999999, 2020-01-16T00:12:47.361 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:47.361 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:47.361 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:47.362 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.362 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:47.362 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.654720+0000", 2020-01-16T00:12:47.362 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:47.362 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.362 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.362 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:47.362 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.654720+0000", 2020-01-16T00:12:47.363 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:47.363 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.363 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.363 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:47.363 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.654719+0000", 2020-01-16T00:12:47.363 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0016889749999999999 2020-01-16T00:12:47.363 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.364 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.364 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:47.364 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.656408+0000", 2020-01-16T00:12:47.364 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1197e-05 2020-01-16T00:12:47.364 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.364 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.365 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:47.365 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.656419+0000", 2020-01-16T00:12:47.365 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3610000000000003e-06 2020-01-16T00:12:47.365 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.365 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.365 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:47.365 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.656423+0000", 2020-01-16T00:12:47.365 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9363999999999998e-05 2020-01-16T00:12:47.366 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.366 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.366 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:47.366 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.656463+0000", 2020-01-16T00:12:47.366 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011794 2020-01-16T00:12:47.366 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.366 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.366 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:47.367 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.656581+0000", 2020-01-16T00:12:47.367 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000115374 2020-01-16T00:12:47.367 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.367 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.367 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:12:47.367 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.656696+0000", 2020-01-16T00:12:47.367 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0038889340000000001 2020-01-16T00:12:47.368 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.368 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.368 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:47.368 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.660585+0000", 2020-01-16T00:12:47.368 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000108517 2020-01-16T00:12:47.368 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.368 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.368 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:47.369 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.660694+0000", 2020-01-16T00:12:47.369 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.1643000000000003e-05 2020-01-16T00:12:47.369 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.369 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.369 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:47.369 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.660785+0000", 2020-01-16T00:12:47.369 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7000000000000003e-05 2020-01-16T00:12:47.369 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.370 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.370 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:47.370 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.660842+0000", 2020-01-16T00:12:47.370 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7374000000000003e-05 2020-01-16T00:12:47.370 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.370 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.370 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:47.371 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.660900+0000", 2020-01-16T00:12:47.371 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0061792729999999999 2020-01-16T00:12:47.371 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.371 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:47.372 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.372 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.372 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.372 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:39 2.1 2:8b431f80:::smithi12313934-40 low_tier:head [write 1926656~67072] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e28)", 2020-01-16T00:12:47.372 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:46.656432+0000", 2020-01-16T00:12:47.372 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.53647258399999997, 2020-01-16T00:12:47.372 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0046525220000000001, 2020-01-16T00:12:47.373 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:47.373 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:47.373 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:47.373 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.373 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:47.373 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.656432+0000", 2020-01-16T00:12:47.373 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:47.374 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.374 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.374 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:47.374 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.656432+0000", 2020-01-16T00:12:47.374 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:47.374 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.374 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.374 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:47.375 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.656432+0000", 2020-01-16T00:12:47.375 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0343999999999999e-05 2020-01-16T00:12:47.375 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.375 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.375 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:47.375 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.656462+0000", 2020-01-16T00:12:47.375 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3470000000000005e-06 2020-01-16T00:12:47.375 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.375 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.376 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:47.376 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.656471+0000", 2020-01-16T00:12:47.376 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.157e-06 2020-01-16T00:12:47.376 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.376 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.376 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:47.376 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.656475+0000", 2020-01-16T00:12:47.377 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001008236 2020-01-16T00:12:47.377 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.377 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.377 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:47.377 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.657483+0000", 2020-01-16T00:12:47.377 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010872 2020-01-16T00:12:47.377 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.377 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.378 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:47.378 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.657592+0000", 2020-01-16T00:12:47.378 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017760900000000001 2020-01-16T00:12:47.378 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.378 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.378 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:12:47.378 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.657769+0000", 2020-01-16T00:12:47.379 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031973909999999999 2020-01-16T00:12:47.379 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.379 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.379 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:47.379 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.660967+0000", 2020-01-16T00:12:47.379 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5121e-05 2020-01-16T00:12:47.379 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.379 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.380 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:47.381 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.660982+0000", 2020-01-16T00:12:47.381 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1785999999999997e-05 2020-01-16T00:12:47.381 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.381 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.381 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:47.381 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.661013+0000", 2020-01-16T00:12:47.381 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1900999999999997e-05 2020-01-16T00:12:47.382 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.382 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.382 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:47.382 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.661055+0000", 2020-01-16T00:12:47.382 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.953e-05 2020-01-16T00:12:47.382 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:47.382 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:47.383 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:47.383 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.661085+0000", 2020-01-16T00:12:47.383 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0046525220000000001 2020-01-16T00:12:47.383 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.383 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:47.383 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.383 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:47.383 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:47.384 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:47.385 INFO:tasks.rados.rados.0.smithi123.stdout:776: expect (ObjNum 28 snap 0 seq_num 28) 2020-01-16T00:12:47.385 INFO:tasks.rados.rados.0.smithi123.stdout:775: expect (ObjNum 94 snap 0 seq_num 94) 2020-01-16T00:12:47.385 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:47.385+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:12:47.386 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-01-16T00:12:47.452 INFO:tasks.rados.rados.0.smithi123.stdout:777: finishing write tid 1 to smithi12313934-35 2020-01-16T00:12:47.452 INFO:tasks.rados.rados.0.smithi123.stdout:777: finishing write tid 2 to smithi12313934-35 2020-01-16T00:12:47.452 INFO:tasks.rados.rados.0.smithi123.stdout:778: finishing write tid 1 to smithi12313934-6 2020-01-16T00:12:47.452 INFO:tasks.rados.rados.0.smithi123.stdout:777: finishing write tid 4 to smithi12313934-35 2020-01-16T00:12:47.452 INFO:tasks.rados.rados.0.smithi123.stdout:777: finishing write tid 5 to smithi12313934-35 2020-01-16T00:12:47.452 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 35 v 79 (ObjNum 136 snap 0 seq_num 136) dirty exists 2020-01-16T00:12:47.453 INFO:tasks.rados.rados.0.smithi123.stdout:777: left oid 35 (ObjNum 136 snap 0 seq_num 136) 2020-01-16T00:12:47.453 INFO:tasks.rados.rados.0.smithi123.stdout:778: finishing write tid 2 to smithi12313934-6 2020-01-16T00:12:47.453 INFO:tasks.rados.rados.0.smithi123.stdout:778: finishing write tid 4 to smithi12313934-6 2020-01-16T00:12:47.453 INFO:tasks.rados.rados.0.smithi123.stdout:778: finishing write tid 5 to smithi12313934-6 2020-01-16T00:12:47.453 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 6 v 53 (ObjNum 137 snap 0 seq_num 137) dirty exists 2020-01-16T00:12:47.453 INFO:tasks.rados.rados.0.smithi123.stdout:778: left oid 6 (ObjNum 137 snap 0 seq_num 137) 2020-01-16T00:12:47.453 INFO:tasks.rados.rados.0.smithi123.stdout:779: expect (ObjNum 31 snap 0 seq_num 31) 2020-01-16T00:12:47.486 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:12:47.485+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:12:47.498 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:47.498 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:12:47.498 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:12:47.498 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:47.515 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-01-16T00:12:47.528 INFO:tasks.rados.rados.0.smithi123.stdout:780: expect (ObjNum 63 snap 0 seq_num 63) 2020-01-16T00:12:47.587 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:12:47.587+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:12:47.665 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:47.666 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:12:47.666 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:12:47.666 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:12:47.666 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:47.675 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-01-16T00:12:47.686 INFO:tasks.rados.rados.0.smithi123.stdout:781: finishing write tid 1 to smithi12313934-40 2020-01-16T00:12:47.687 INFO:tasks.rados.rados.0.smithi123.stdout:781: finishing write tid 2 to smithi12313934-40 2020-01-16T00:12:47.687 INFO:tasks.rados.rados.0.smithi123.stdout:782: finishing write tid 1 to smithi12313934-41 2020-01-16T00:12:47.687 INFO:tasks.rados.rados.0.smithi123.stdout:781: finishing write tid 4 to smithi12313934-40 2020-01-16T00:12:47.687 INFO:tasks.rados.rados.0.smithi123.stdout:781: finishing write tid 5 to smithi12313934-40 2020-01-16T00:12:47.687 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 40 v 98 (ObjNum 138 snap 0 seq_num 138) dirty exists 2020-01-16T00:12:47.687 INFO:tasks.rados.rados.0.smithi123.stdout:781: left oid 40 (ObjNum 138 snap 0 seq_num 138) 2020-01-16T00:12:47.687 INFO:tasks.rados.rados.0.smithi123.stdout:782: finishing write tid 2 to smithi12313934-41 2020-01-16T00:12:47.687 INFO:tasks.rados.rados.0.smithi123.stdout:782: finishing write tid 4 to smithi12313934-41 2020-01-16T00:12:47.688 INFO:tasks.rados.rados.0.smithi123.stdout:782: finishing write tid 5 to smithi12313934-41 2020-01-16T00:12:47.688 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 41 v 91 (ObjNum 139 snap 0 seq_num 139) dirty exists 2020-01-16T00:12:47.688 INFO:tasks.rados.rados.0.smithi123.stdout:782: left oid 41 (ObjNum 139 snap 0 seq_num 139) 2020-01-16T00:12:47.688 INFO:tasks.rados.rados.0.smithi123.stdout:783: finishing write tid 1 to smithi12313934-7 2020-01-16T00:12:47.688 INFO:tasks.rados.rados.0.smithi123.stdout:783: finishing write tid 2 to smithi12313934-7 2020-01-16T00:12:47.688 INFO:tasks.rados.rados.0.smithi123.stdout:783: finishing write tid 4 to smithi12313934-7 2020-01-16T00:12:47.688 INFO:tasks.rados.rados.0.smithi123.stdout:783: finishing write tid 5 to smithi12313934-7 2020-01-16T00:12:47.689 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 7 v 94 (ObjNum 140 snap 0 seq_num 140) dirty exists 2020-01-16T00:12:47.689 INFO:tasks.rados.rados.0.smithi123.stdout:783: left oid 7 (ObjNum 140 snap 0 seq_num 140) 2020-01-16T00:12:47.689 INFO:tasks.rados.rados.0.smithi123.stdout:767: done (15 left) 2020-01-16T00:12:47.689 INFO:tasks.rados.rados.0.smithi123.stdout:768: done (14 left) 2020-01-16T00:12:47.689 INFO:tasks.rados.rados.0.smithi123.stdout:769: done (13 left) 2020-01-16T00:12:47.689 INFO:tasks.rados.rados.0.smithi123.stdout:770: done (12 left) 2020-01-16T00:12:47.689 INFO:tasks.rados.rados.0.smithi123.stdout:771: done (11 left) 2020-01-16T00:12:47.689 INFO:tasks.rados.rados.0.smithi123.stdout:772: done (10 left) 2020-01-16T00:12:47.690 INFO:tasks.rados.rados.0.smithi123.stdout:773: done (9 left) 2020-01-16T00:12:47.690 INFO:tasks.rados.rados.0.smithi123.stdout:775: done (8 left) 2020-01-16T00:12:47.690 INFO:tasks.rados.rados.0.smithi123.stdout:776: done (7 left) 2020-01-16T00:12:47.690 INFO:tasks.rados.rados.0.smithi123.stdout:777: done (6 left) 2020-01-16T00:12:47.690 INFO:tasks.rados.rados.0.smithi123.stdout:778: done (5 left) 2020-01-16T00:12:47.690 INFO:tasks.rados.rados.0.smithi123.stdout:779: done (4 left) 2020-01-16T00:12:47.690 INFO:tasks.rados.rados.0.smithi123.stdout:780: done (3 left) 2020-01-16T00:12:47.690 INFO:tasks.rados.rados.0.smithi123.stdout:781: done (2 left) 2020-01-16T00:12:47.691 INFO:tasks.rados.rados.0.smithi123.stdout:782: done (1 left) 2020-01-16T00:12:47.691 INFO:tasks.rados.rados.0.smithi123.stdout:783: done (0 left) 2020-01-16T00:12:47.691 INFO:tasks.rados.rados.0.smithi123.stdout:784: read oid 21 snap -1 2020-01-16T00:12:47.691 INFO:tasks.rados.rados.0.smithi123.stdout:784: expect (ObjNum 113 snap 0 seq_num 113) 2020-01-16T00:12:47.691 INFO:tasks.rados.rados.0.smithi123.stdout:785: write oid 3 current snap is 0 2020-01-16T00:12:47.691 INFO:tasks.rados.rados.0.smithi123.stdout:785: seq_num 141 ranges {769787=704463,2267961=101826} 2020-01-16T00:12:47.693 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:47.695+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:12:47.704 INFO:tasks.rados.rados.0.smithi123.stdout:785: writing smithi12313934-3 from 769787 to 1474250 tid 1 2020-01-16T00:12:47.709 INFO:tasks.rados.rados.0.smithi123.stdout:785: writing smithi12313934-3 from 2267961 to 2369787 tid 2 2020-01-16T00:12:47.709 INFO:tasks.rados.rados.0.smithi123.stdout:786: read oid 42 snap -1 2020-01-16T00:12:47.709 INFO:tasks.rados.rados.0.smithi123.stdout:786: expect (ObjNum 127 snap 0 seq_num 127) 2020-01-16T00:12:47.709 INFO:tasks.rados.rados.0.smithi123.stdout:784: expect (ObjNum 113 snap 0 seq_num 113) 2020-01-16T00:12:47.786 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:47.788 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:12:47.789 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:12:47.789 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:12:47.789 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.789 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:910 3.5 3:ad8ab15f:::smithi12313934-27:head [write 1563527~583025] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:47.789 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.830607+0000", 2020-01-16T00:12:47.789 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 9.9513728750000006, 2020-01-16T00:12:47.789 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019080092999999999, 2020-01-16T00:12:47.790 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:47.790 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:47.790 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:47.790 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:47.790 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:47.790 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 910 2020-01-16T00:12:47.790 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.791 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:47.791 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.791 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:47.791 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.830607+0000", 2020-01-16T00:12:47.791 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:47.791 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.791 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.791 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:47.792 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.830607+0000", 2020-01-16T00:12:47.792 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:47.792 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.792 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.792 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:47.792 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.830604+0000", 2020-01-16T00:12:47.792 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00078346100000000005 2020-01-16T00:12:47.792 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.793 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.793 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:47.793 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831388+0000", 2020-01-16T00:12:47.793 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.6936000000000001e-05 2020-01-16T00:12:47.793 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.793 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.793 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:47.794 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831415+0000", 2020-01-16T00:12:47.794 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0737000000000001e-05 2020-01-16T00:12:47.794 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.794 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.794 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:47.794 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831425+0000", 2020-01-16T00:12:47.794 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1964e-05 2020-01-16T00:12:47.794 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.795 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.795 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:47.795 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831467+0000", 2020-01-16T00:12:47.795 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028524600000000001 2020-01-16T00:12:47.795 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.795 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.795 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:47.795 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831753+0000", 2020-01-16T00:12:47.796 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012909141000000001 2020-01-16T00:12:47.796 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.802 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.802 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:47.802 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.844662+0000", 2020-01-16T00:12:47.802 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022242 2020-01-16T00:12:47.802 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.803 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.803 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:47.803 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.844884+0000", 2020-01-16T00:12:47.803 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015169700000000001 2020-01-16T00:12:47.803 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.803 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.804 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,3", 2020-01-16T00:12:47.804 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.845036+0000", 2020-01-16T00:12:47.804 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0027572859999999999 2020-01-16T00:12:47.804 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.804 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.804 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:47.804 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.847793+0000", 2020-01-16T00:12:47.805 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00084882100000000004 2020-01-16T00:12:47.805 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.805 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.805 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:47.805 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.848642+0000", 2020-01-16T00:12:47.805 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00065781900000000003 2020-01-16T00:12:47.805 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.806 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.806 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:47.806 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.849300+0000", 2020-01-16T00:12:47.806 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0001717 2020-01-16T00:12:47.806 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.807 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.807 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:47.807 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.849472+0000", 2020-01-16T00:12:47.807 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000215348 2020-01-16T00:12:47.807 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.807 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.807 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:47.807 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.849687+0000", 2020-01-16T00:12:47.808 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019080092999999999 2020-01-16T00:12:47.808 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:47.808 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:47.808 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:47.808 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.808 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.808 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:911 3.5 3:ad8ab15f:::smithi12313934-27:head [write 2810926~399178] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:47.809 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.836707+0000", 2020-01-16T00:12:47.809 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 9.945272246, 2020-01-16T00:12:47.809 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.024736662999999999, 2020-01-16T00:12:47.809 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:47.809 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:47.810 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:47.810 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:47.810 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:47.810 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 911 2020-01-16T00:12:47.811 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.811 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:47.811 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.811 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:47.811 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.836707+0000", 2020-01-16T00:12:47.811 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:47.811 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.811 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.812 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:47.812 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.836707+0000", 2020-01-16T00:12:47.812 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:12:47.812 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.812 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.812 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:47.813 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.836704+0000", 2020-01-16T00:12:47.813 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00077069200000000001 2020-01-16T00:12:47.813 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.813 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.813 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:47.813 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837474+0000", 2020-01-16T00:12:47.813 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7440000000000001e-05 2020-01-16T00:12:47.814 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.814 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.814 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:47.814 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837512+0000", 2020-01-16T00:12:47.814 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8246000000000001e-05 2020-01-16T00:12:47.814 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.814 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.814 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:47.815 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837530+0000", 2020-01-16T00:12:47.815 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000105405 2020-01-16T00:12:47.815 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.815 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.815 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:47.815 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837635+0000", 2020-01-16T00:12:47.815 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6244999999999998e-05 2020-01-16T00:12:47.815 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.816 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.816 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:47.816 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837722+0000", 2020-01-16T00:12:47.816 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0089724320000000007 2020-01-16T00:12:47.816 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.816 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.816 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:47.822 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.846694+0000", 2020-01-16T00:12:47.822 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042899099999999999 2020-01-16T00:12:47.822 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.822 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.822 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:47.822 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.847123+0000", 2020-01-16T00:12:47.823 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011739893 2020-01-16T00:12:47.823 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.823 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.823 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:47.823 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.858863+0000", 2020-01-16T00:12:47.823 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1312999999999999e-05 2020-01-16T00:12:47.824 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.824 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.824 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:47.824 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.858944+0000", 2020-01-16T00:12:47.824 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015682900000000001 2020-01-16T00:12:47.824 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.824 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.825 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,3", 2020-01-16T00:12:47.825 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.859101+0000", 2020-01-16T00:12:47.825 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001604442 2020-01-16T00:12:47.825 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.825 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.825 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:47.825 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.860706+0000", 2020-01-16T00:12:47.826 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000377568 2020-01-16T00:12:47.826 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.826 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.826 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:47.826 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.861083+0000", 2020-01-16T00:12:47.826 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022656400000000001 2020-01-16T00:12:47.826 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.826 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.827 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:47.827 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.861310+0000", 2020-01-16T00:12:47.827 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7865000000000002e-05 2020-01-16T00:12:47.827 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.827 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.827 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:47.828 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.861368+0000", 2020-01-16T00:12:47.828 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6549000000000002e-05 2020-01-16T00:12:47.828 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.828 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.828 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:47.828 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.861444+0000", 2020-01-16T00:12:47.828 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.024736662999999999 2020-01-16T00:12:47.828 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:47.829 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:47.829 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:47.829 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.829 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.829 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:912 3.5 3:ad8ab15f:::smithi12313934-27:head [setxattr _header (56) in=7b,truncate 3210104] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:47.837 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.837560+0000", 2020-01-16T00:12:47.838 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 9.9444195339999997, 2020-01-16T00:12:47.840 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.024152410999999999, 2020-01-16T00:12:47.841 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:47.841 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:47.841 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:47.841 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:47.842 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:47.842 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 912 2020-01-16T00:12:47.842 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.842 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:47.842 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.843 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:47.843 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837560+0000", 2020-01-16T00:12:47.843 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:47.843 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.843 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.844 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:47.844 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837560+0000", 2020-01-16T00:12:47.844 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:47.844 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.844 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.845 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:47.845 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837558+0000", 2020-01-16T00:12:47.845 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8468999999999999e-05 2020-01-16T00:12:47.845 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.845 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.845 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:47.846 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837576+0000", 2020-01-16T00:12:47.846 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8437e-05 2020-01-16T00:12:47.846 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.846 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.846 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:47.846 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837595+0000", 2020-01-16T00:12:47.846 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2782999999999999e-05 2020-01-16T00:12:47.847 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.847 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.847 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:47.847 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837608+0000", 2020-01-16T00:12:47.847 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000174883 2020-01-16T00:12:47.847 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.847 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.847 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:47.848 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837782+0000", 2020-01-16T00:12:47.848 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8194000000000001e-05 2020-01-16T00:12:47.849 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.849 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.849 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:47.850 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837821+0000", 2020-01-16T00:12:47.850 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0094026500000000002 2020-01-16T00:12:47.850 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.850 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.850 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:47.850 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.847223+0000", 2020-01-16T00:12:47.850 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000120973 2020-01-16T00:12:47.851 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.851 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.851 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:47.851 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.847344+0000", 2020-01-16T00:12:47.851 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012601658999999999 2020-01-16T00:12:47.851 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.852 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.852 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:47.852 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.859946+0000", 2020-01-16T00:12:47.852 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012089 2020-01-16T00:12:47.852 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.852 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.852 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:47.853 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.860067+0000", 2020-01-16T00:12:47.853 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020346700000000001 2020-01-16T00:12:47.853 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.853 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.853 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,3", 2020-01-16T00:12:47.853 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.860270+0000", 2020-01-16T00:12:47.853 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00057751500000000002 2020-01-16T00:12:47.854 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.854 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.854 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:47.854 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.860848+0000", 2020-01-16T00:12:47.854 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00067389800000000003 2020-01-16T00:12:47.854 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.855 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.855 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:47.855 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.861522+0000", 2020-01-16T00:12:47.855 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2540999999999996e-05 2020-01-16T00:12:47.855 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.855 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.855 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:47.856 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.861584+0000", 2020-01-16T00:12:47.856 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8773e-05 2020-01-16T00:12:47.856 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.856 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.856 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:47.856 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.861653+0000", 2020-01-16T00:12:47.856 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9493000000000001e-05 2020-01-16T00:12:47.857 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.857 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.857 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:47.857 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.861713+0000", 2020-01-16T00:12:47.857 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.024152410999999999 2020-01-16T00:12:47.857 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:47.857 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:47.858 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:47.858 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.858 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.858 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:913 3.5 3:ad8ab15f:::smithi12313934-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:12:47.858 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.837629+0000", 2020-01-16T00:12:47.858 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 9.9443507160000006, 2020-01-16T00:12:47.858 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.024296113000000001, 2020-01-16T00:12:47.858 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:47.859 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:47.859 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:47.859 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:47.859 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:47.859 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 913 2020-01-16T00:12:47.860 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.860 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:47.860 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.860 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:47.860 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837629+0000", 2020-01-16T00:12:47.860 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:47.860 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.860 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.861 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:47.861 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837629+0000", 2020-01-16T00:12:47.861 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:47.861 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.861 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.862 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:47.862 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837627+0000", 2020-01-16T00:12:47.862 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.329e-06 2020-01-16T00:12:47.862 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.862 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.862 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:47.863 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837637+0000", 2020-01-16T00:12:47.863 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7397e-05 2020-01-16T00:12:47.863 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.863 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.863 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:47.863 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837654+0000", 2020-01-16T00:12:47.863 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6073999999999999e-05 2020-01-16T00:12:47.864 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.864 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.864 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:47.864 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837670+0000", 2020-01-16T00:12:47.864 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000171781 2020-01-16T00:12:47.865 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.865 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.865 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:47.865 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837842+0000", 2020-01-16T00:12:47.865 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1575000000000002e-05 2020-01-16T00:12:47.866 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.866 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.866 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:47.866 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837873+0000", 2020-01-16T00:12:47.867 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0095469749999999992 2020-01-16T00:12:47.867 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.867 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.867 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:47.867 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.847420+0000", 2020-01-16T00:12:47.868 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5698000000000003e-05 2020-01-16T00:12:47.868 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.868 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.868 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:47.868 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.847456+0000", 2020-01-16T00:12:47.869 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013066217 2020-01-16T00:12:47.869 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.870 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.870 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:47.870 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.860522+0000", 2020-01-16T00:12:47.870 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.2278000000000001e-05 2020-01-16T00:12:47.870 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.871 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.871 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:47.871 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.860605+0000", 2020-01-16T00:12:47.872 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001147779 2020-01-16T00:12:47.872 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.872 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.873 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:47.873 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.861752+0000", 2020-01-16T00:12:47.873 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4598000000000006e-05 2020-01-16T00:12:47.873 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.873 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.873 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:47.874 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.861827+0000", 2020-01-16T00:12:47.874 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8060999999999995e-05 2020-01-16T00:12:47.874 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.874 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.874 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:47.874 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.861925+0000", 2020-01-16T00:12:47.874 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.024296113000000001 2020-01-16T00:12:47.875 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:47.875 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:47.875 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:47.875 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.875 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.875 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:914 3.c 3:3d6543e6:::smithi12313934-28:head [write 426911~540354,stat] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:47.875 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.842336+0000", 2020-01-16T00:12:47.875 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 9.9396439940000008, 2020-01-16T00:12:47.876 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021922343, 2020-01-16T00:12:47.876 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:47.876 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:47.876 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:47.876 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:47.876 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:47.876 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 914 2020-01-16T00:12:47.877 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.877 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:47.877 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.877 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:47.877 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.842336+0000", 2020-01-16T00:12:47.877 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:47.877 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.877 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.878 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:47.878 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.842336+0000", 2020-01-16T00:12:47.878 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:12:47.878 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.878 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.878 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:47.878 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.842333+0000", 2020-01-16T00:12:47.879 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00063242099999999998 2020-01-16T00:12:47.879 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.879 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.879 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:47.879 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.842965+0000", 2020-01-16T00:12:47.879 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.8354999999999999e-05 2020-01-16T00:12:47.879 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.880 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.880 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:47.880 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.842994+0000", 2020-01-16T00:12:47.880 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1077000000000001e-05 2020-01-16T00:12:47.880 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.880 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.881 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:47.881 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843005+0000", 2020-01-16T00:12:47.881 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000107552 2020-01-16T00:12:47.881 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.882 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.882 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:47.882 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843112+0000", 2020-01-16T00:12:47.883 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00037837800000000002 2020-01-16T00:12:47.883 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.883 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.883 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:47.884 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843491+0000", 2020-01-16T00:12:47.884 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0092250490000000008 2020-01-16T00:12:47.884 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.884 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.885 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:47.885 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.852716+0000", 2020-01-16T00:12:47.886 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018407699999999999 2020-01-16T00:12:47.886 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.886 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.886 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:47.886 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.852900+0000", 2020-01-16T00:12:47.887 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0078355379999999995 2020-01-16T00:12:47.887 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.887 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.887 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:47.887 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.860735+0000", 2020-01-16T00:12:47.887 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012386100000000001 2020-01-16T00:12:47.888 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.888 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.888 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:47.888 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.860859+0000", 2020-01-16T00:12:47.888 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014735099999999999 2020-01-16T00:12:47.888 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.889 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.889 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2020-01-16T00:12:47.889 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.861006+0000", 2020-01-16T00:12:47.889 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0027785040000000002 2020-01-16T00:12:47.889 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.889 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.889 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:47.890 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.863785+0000", 2020-01-16T00:12:47.890 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000127083 2020-01-16T00:12:47.890 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.890 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.890 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:47.890 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.863912+0000", 2020-01-16T00:12:47.890 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000179106 2020-01-16T00:12:47.891 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.891 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.891 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:47.891 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.864091+0000", 2020-01-16T00:12:47.891 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7744999999999998e-05 2020-01-16T00:12:47.891 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.891 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.892 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:47.892 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.864139+0000", 2020-01-16T00:12:47.892 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011914200000000001 2020-01-16T00:12:47.892 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.892 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.892 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:47.893 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.864258+0000", 2020-01-16T00:12:47.893 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021922343 2020-01-16T00:12:47.893 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:47.893 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:47.893 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:47.893 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.893 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.894 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:915 3.c 3:3d6543e6:::smithi12313934-28:head [write 1622009~726683] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:47.894 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.850793+0000", 2020-01-16T00:12:47.894 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 9.9311867889999998, 2020-01-16T00:12:47.894 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018526131000000001, 2020-01-16T00:12:47.894 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:47.894 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:47.895 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:47.895 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:47.895 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:47.895 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 915 2020-01-16T00:12:47.895 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.895 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:47.896 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.896 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:47.896 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.850793+0000", 2020-01-16T00:12:47.896 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:47.896 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.896 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.897 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:47.897 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.850793+0000", 2020-01-16T00:12:47.897 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999957 2020-01-16T00:12:47.897 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.897 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.897 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:47.897 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.850788+0000", 2020-01-16T00:12:47.898 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0022032459999999999 2020-01-16T00:12:47.898 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.898 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.898 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:47.898 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.852992+0000", 2020-01-16T00:12:47.898 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6056999999999999e-05 2020-01-16T00:12:47.898 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.899 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.899 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:47.899 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.853008+0000", 2020-01-16T00:12:47.899 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5619999999999993e-06 2020-01-16T00:12:47.899 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.899 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.899 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:47.900 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.853017+0000", 2020-01-16T00:12:47.900 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9026000000000002e-05 2020-01-16T00:12:47.900 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.900 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.900 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:47.900 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.853056+0000", 2020-01-16T00:12:47.900 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.1551000000000005e-05 2020-01-16T00:12:47.901 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.901 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.901 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:47.901 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.853148+0000", 2020-01-16T00:12:47.901 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0091288280000000003 2020-01-16T00:12:47.901 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.901 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.902 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:47.902 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.862277+0000", 2020-01-16T00:12:47.902 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000101439 2020-01-16T00:12:47.902 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.902 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.902 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:47.902 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.862378+0000", 2020-01-16T00:12:47.903 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001920597 2020-01-16T00:12:47.903 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.903 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.903 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:47.903 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.864299+0000", 2020-01-16T00:12:47.903 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000129355 2020-01-16T00:12:47.904 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.904 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.904 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:47.904 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.864428+0000", 2020-01-16T00:12:47.904 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000226999 2020-01-16T00:12:47.904 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.905 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.905 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2020-01-16T00:12:47.905 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.864655+0000", 2020-01-16T00:12:47.905 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0036207840000000002 2020-01-16T00:12:47.905 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.905 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.906 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:47.906 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.868276+0000", 2020-01-16T00:12:47.906 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4215999999999996e-05 2020-01-16T00:12:47.906 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.906 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.906 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:47.906 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.868350+0000", 2020-01-16T00:12:47.907 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00083996500000000003 2020-01-16T00:12:47.907 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.907 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.907 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:47.907 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.869190+0000", 2020-01-16T00:12:47.907 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0606999999999998e-05 2020-01-16T00:12:47.907 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.908 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.908 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:47.908 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.869251+0000", 2020-01-16T00:12:47.908 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8396e-05 2020-01-16T00:12:47.908 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.908 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.908 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:47.909 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.869319+0000", 2020-01-16T00:12:47.909 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018526131000000001 2020-01-16T00:12:47.909 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:47.909 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:47.910 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:47.910 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.910 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.910 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:916 3.c 3:3d6543e6:::smithi12313934-28:head [write 3121996~104915] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:47.910 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.855949+0000", 2020-01-16T00:12:47.910 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 9.9260306489999994, 2020-01-16T00:12:47.911 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013528775, 2020-01-16T00:12:47.911 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:47.911 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:47.911 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:47.911 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:47.911 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:47.911 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 916 2020-01-16T00:12:47.912 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.912 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:47.912 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.912 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:47.912 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.855949+0000", 2020-01-16T00:12:47.912 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:47.912 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.913 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.913 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:47.913 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.855949+0000", 2020-01-16T00:12:47.913 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:47.913 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.913 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.913 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:47.914 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.855947+0000", 2020-01-16T00:12:47.914 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013682200000000001 2020-01-16T00:12:47.914 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.914 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.914 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:47.914 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856084+0000", 2020-01-16T00:12:47.914 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7864999999999999e-05 2020-01-16T00:12:47.915 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.915 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.915 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:47.915 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856102+0000", 2020-01-16T00:12:47.915 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.8699999999999998e-06 2020-01-16T00:12:47.915 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.916 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.916 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:47.916 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856111+0000", 2020-01-16T00:12:47.916 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.299e-05 2020-01-16T00:12:47.916 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.916 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.916 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:47.917 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856154+0000", 2020-01-16T00:12:47.917 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0636999999999994e-05 2020-01-16T00:12:47.917 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.917 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.917 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:47.917 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856234+0000", 2020-01-16T00:12:47.917 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0061827920000000003 2020-01-16T00:12:47.917 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.918 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.918 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:47.918 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.862417+0000", 2020-01-16T00:12:47.918 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010285099999999999 2020-01-16T00:12:47.918 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.918 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.919 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:47.919 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.862520+0000", 2020-01-16T00:12:47.919 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0038991479999999999 2020-01-16T00:12:47.919 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.919 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.919 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:47.919 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.866419+0000", 2020-01-16T00:12:47.920 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010513700000000001 2020-01-16T00:12:47.920 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.920 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.920 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:47.920 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.866524+0000", 2020-01-16T00:12:47.920 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014972299999999999 2020-01-16T00:12:47.920 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.921 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.921 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2020-01-16T00:12:47.921 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.866674+0000", 2020-01-16T00:12:47.921 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001618415 2020-01-16T00:12:47.921 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.921 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.921 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:47.922 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.868292+0000", 2020-01-16T00:12:47.922 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000297736 2020-01-16T00:12:47.922 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.922 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.922 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:47.922 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.868590+0000", 2020-01-16T00:12:47.922 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000798283 2020-01-16T00:12:47.923 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.923 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.923 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:47.923 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.869388+0000", 2020-01-16T00:12:47.923 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5621000000000002e-05 2020-01-16T00:12:47.923 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.924 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.924 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:47.924 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.869434+0000", 2020-01-16T00:12:47.924 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3894999999999999e-05 2020-01-16T00:12:47.924 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.924 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.924 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:47.925 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.869478+0000", 2020-01-16T00:12:47.925 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013528775 2020-01-16T00:12:47.925 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:47.925 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:47.925 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:47.925 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.926 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.926 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:917 3.c 3:3d6543e6:::smithi12313934-28:head [setxattr _header (56) in=7b,truncate 3226911] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:47.926 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.856125+0000", 2020-01-16T00:12:47.926 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 9.9258544390000001, 2020-01-16T00:12:47.926 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013497781, 2020-01-16T00:12:47.926 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:47.926 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:47.926 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:47.927 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:47.927 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:47.927 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 917 2020-01-16T00:12:47.927 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.927 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:47.927 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.927 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:47.928 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856125+0000", 2020-01-16T00:12:47.928 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:47.928 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.928 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.928 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:47.928 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856125+0000", 2020-01-16T00:12:47.928 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:47.929 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.929 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.929 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:47.929 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856124+0000", 2020-01-16T00:12:47.929 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8859999999999998e-06 2020-01-16T00:12:47.929 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.930 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.930 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:47.930 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856131+0000", 2020-01-16T00:12:47.930 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6570000000000003e-06 2020-01-16T00:12:47.930 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.930 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.930 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:47.931 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856140+0000", 2020-01-16T00:12:47.931 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5420000000000002e-06 2020-01-16T00:12:47.931 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.931 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.931 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:47.931 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856146+0000", 2020-01-16T00:12:47.931 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012714399999999999 2020-01-16T00:12:47.932 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.932 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.932 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:47.932 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856273+0000", 2020-01-16T00:12:47.932 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000122113 2020-01-16T00:12:47.932 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.932 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.933 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:47.933 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856395+0000", 2020-01-16T00:12:47.933 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.006160708 2020-01-16T00:12:47.933 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.933 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.933 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:47.934 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.862556+0000", 2020-01-16T00:12:47.934 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7220999999999999e-05 2020-01-16T00:12:47.934 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.934 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.934 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:47.934 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.862623+0000", 2020-01-16T00:12:47.934 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0046116050000000004 2020-01-16T00:12:47.934 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.935 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.935 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:47.935 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.867235+0000", 2020-01-16T00:12:47.935 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019234499999999999 2020-01-16T00:12:47.935 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.935 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.936 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:47.936 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.867427+0000", 2020-01-16T00:12:47.936 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022820699999999999 2020-01-16T00:12:47.936 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.936 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.936 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2020-01-16T00:12:47.937 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.867656+0000", 2020-01-16T00:12:47.937 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00065250900000000001 2020-01-16T00:12:47.937 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.937 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.937 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:47.937 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.868308+0000", 2020-01-16T00:12:47.937 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00043943999999999998 2020-01-16T00:12:47.938 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.938 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.938 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:47.938 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.868748+0000", 2020-01-16T00:12:47.938 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00078105600000000002 2020-01-16T00:12:47.938 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.938 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.939 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:47.939 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.869529+0000", 2020-01-16T00:12:47.939 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2976000000000003e-05 2020-01-16T00:12:47.939 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.939 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.939 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:47.939 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.869572+0000", 2020-01-16T00:12:47.940 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1439e-05 2020-01-16T00:12:47.940 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.940 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.940 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:47.940 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.869623+0000", 2020-01-16T00:12:47.940 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013497781 2020-01-16T00:12:47.940 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:47.941 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:47.941 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:47.941 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.941 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.941 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:918 3.c 3:3d6543e6:::smithi12313934-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:12:47.941 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.856157+0000", 2020-01-16T00:12:47.942 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 9.9258226129999994, 2020-01-16T00:12:47.942 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013685233, 2020-01-16T00:12:47.942 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:47.942 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:47.942 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:47.942 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:47.942 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:47.943 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 918 2020-01-16T00:12:47.943 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.943 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:47.943 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.943 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:47.943 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856157+0000", 2020-01-16T00:12:47.943 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:47.944 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.944 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.944 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:47.944 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856157+0000", 2020-01-16T00:12:47.944 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:47.944 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.944 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.945 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:47.945 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856156+0000", 2020-01-16T00:12:47.945 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7450000000000003e-06 2020-01-16T00:12:47.945 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.945 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.945 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:47.946 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856161+0000", 2020-01-16T00:12:47.946 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5880000000000004e-06 2020-01-16T00:12:47.946 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.946 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.946 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:47.946 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856171+0000", 2020-01-16T00:12:47.946 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1620000000000002e-06 2020-01-16T00:12:47.947 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.947 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.947 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:47.947 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856178+0000", 2020-01-16T00:12:47.947 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000256796 2020-01-16T00:12:47.947 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.947 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.948 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:47.948 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856435+0000", 2020-01-16T00:12:47.948 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.8992000000000005e-05 2020-01-16T00:12:47.948 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.948 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.948 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:47.948 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856514+0000", 2020-01-16T00:12:47.949 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0061401370000000004 2020-01-16T00:12:47.949 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.949 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.949 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:47.949 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.862654+0000", 2020-01-16T00:12:47.949 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2589000000000007e-05 2020-01-16T00:12:47.950 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.950 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.950 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:47.950 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.862716+0000", 2020-01-16T00:12:47.950 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0053446409999999998 2020-01-16T00:12:47.950 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.951 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.951 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:47.951 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.868061+0000", 2020-01-16T00:12:47.951 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019486699999999999 2020-01-16T00:12:47.951 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.951 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.952 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:47.952 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.868256+0000", 2020-01-16T00:12:47.952 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014152170000000001 2020-01-16T00:12:47.952 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.952 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.952 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:47.952 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.869671+0000", 2020-01-16T00:12:47.953 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.2387999999999998e-05 2020-01-16T00:12:47.953 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.953 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.953 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:47.953 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.869753+0000", 2020-01-16T00:12:47.953 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.8926999999999997e-05 2020-01-16T00:12:47.953 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.954 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.954 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:47.954 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.869842+0000", 2020-01-16T00:12:47.954 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013685233 2020-01-16T00:12:47.954 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:47.954 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:47.954 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:47.955 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.955 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.955 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.4.13:16 2.3 2:e19dd0d6:::smithi12313934-14 low_tier:head [read 1798144~391168 [fadvise_sequential+fadvise_dontneed]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e24)", 2020-01-16T00:12:47.955 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.915333+0000", 2020-01-16T00:12:47.955 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 9.8666470589999999, 2020-01-16T00:12:47.955 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0086441409999999993, 2020-01-16T00:12:47.955 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:47.956 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:47.956 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:47.956 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.956 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:47.956 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.915333+0000", 2020-01-16T00:12:47.956 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:47.957 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.957 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.957 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:47.957 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.915333+0000", 2020-01-16T00:12:47.957 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:47.957 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.957 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.958 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:47.958 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.915332+0000", 2020-01-16T00:12:47.958 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8210000000000004e-06 2020-01-16T00:12:47.959 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.959 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.959 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:47.959 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.915339+0000", 2020-01-16T00:12:47.959 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.8259999999999993e-06 2020-01-16T00:12:47.960 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.960 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.960 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:47.960 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.915347+0000", 2020-01-16T00:12:47.961 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7119999999999999e-06 2020-01-16T00:12:47.961 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.961 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.961 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:47.961 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.915352+0000", 2020-01-16T00:12:47.961 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0078058870000000001 2020-01-16T00:12:47.961 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.962 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.962 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:47.962 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.923157+0000", 2020-01-16T00:12:47.962 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4171999999999994e-05 2020-01-16T00:12:47.962 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.962 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.962 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:47.963 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.923232+0000", 2020-01-16T00:12:47.963 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00074513400000000005 2020-01-16T00:12:47.963 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.963 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.963 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:47.963 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.923977+0000", 2020-01-16T00:12:47.964 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0086441409999999993 2020-01-16T00:12:47.964 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:47.964 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:47.964 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:47.964 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.964 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.964 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:952 3.c 3:3877922e:::smithi12313934-31:head [write 494139~557703,stat] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:47.965 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.923660+0000", 2020-01-16T00:12:47.965 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 9.8583198349999996, 2020-01-16T00:12:47.965 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016369574000000001, 2020-01-16T00:12:47.965 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:47.965 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:47.965 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:47.966 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:47.966 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:47.966 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 952 2020-01-16T00:12:47.966 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.966 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:47.966 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.966 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:47.967 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.923660+0000", 2020-01-16T00:12:47.967 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:47.967 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.967 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.967 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:47.967 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.923660+0000", 2020-01-16T00:12:47.968 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:47.968 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.968 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.968 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:47.968 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.923659+0000", 2020-01-16T00:12:47.968 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0015925539999999999 2020-01-16T00:12:47.968 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.969 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.969 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:47.970 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.925251+0000", 2020-01-16T00:12:47.970 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4825e-05 2020-01-16T00:12:47.970 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.970 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.970 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:47.970 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.925276+0000", 2020-01-16T00:12:47.970 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.217e-06 2020-01-16T00:12:47.971 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.971 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.971 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:47.971 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.925285+0000", 2020-01-16T00:12:47.971 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8985000000000004e-05 2020-01-16T00:12:47.971 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.971 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.972 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:47.972 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.925354+0000", 2020-01-16T00:12:47.972 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022915199999999999 2020-01-16T00:12:47.972 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.972 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.972 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:47.972 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.925583+0000", 2020-01-16T00:12:47.972 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0072044309999999999 2020-01-16T00:12:47.973 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.973 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.973 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:47.973 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.932788+0000", 2020-01-16T00:12:47.974 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2225999999999994e-05 2020-01-16T00:12:47.974 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.974 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.974 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:47.974 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.932880+0000", 2020-01-16T00:12:47.975 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0040176580000000003 2020-01-16T00:12:47.975 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.975 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.975 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:47.976 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.936898+0000", 2020-01-16T00:12:47.976 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9100999999999999e-05 2020-01-16T00:12:47.976 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.976 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.976 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:47.977 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.936997+0000", 2020-01-16T00:12:47.977 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00021074099999999999 2020-01-16T00:12:47.977 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.978 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.978 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2020-01-16T00:12:47.978 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.937207+0000", 2020-01-16T00:12:47.978 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001814979 2020-01-16T00:12:47.978 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.978 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.978 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:47.979 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939022+0000", 2020-01-16T00:12:47.979 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014921699999999999 2020-01-16T00:12:47.979 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.979 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.979 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:47.979 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939172+0000", 2020-01-16T00:12:47.980 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00070305200000000002 2020-01-16T00:12:47.980 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.980 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.980 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:47.980 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939875+0000", 2020-01-16T00:12:47.980 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.9981999999999994e-05 2020-01-16T00:12:47.980 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.980 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.981 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:47.981 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939955+0000", 2020-01-16T00:12:47.981 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4770000000000004e-05 2020-01-16T00:12:47.981 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.981 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.981 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:47.982 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.940029+0000", 2020-01-16T00:12:47.982 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016369574000000001 2020-01-16T00:12:47.982 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:47.982 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:47.982 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:47.982 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.983 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.983 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:953 3.c 3:3877922e:::smithi12313934-31:head [write 1455937~734021] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:47.983 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.931235+0000", 2020-01-16T00:12:47.983 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 9.8507446269999992, 2020-01-16T00:12:47.984 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.022666925000000001, 2020-01-16T00:12:47.984 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:47.985 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:47.985 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:47.985 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:47.985 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:47.985 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 953 2020-01-16T00:12:47.986 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.986 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:47.986 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.986 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:47.986 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.931235+0000", 2020-01-16T00:12:47.987 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:47.987 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.987 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.987 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:47.987 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.931235+0000", 2020-01-16T00:12:47.988 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:47.988 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.988 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.988 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:47.988 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.931233+0000", 2020-01-16T00:12:47.988 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0013294019999999999 2020-01-16T00:12:47.988 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.989 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.989 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:47.989 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.932562+0000", 2020-01-16T00:12:47.989 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0194e-05 2020-01-16T00:12:47.989 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.989 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.989 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:47.990 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.932582+0000", 2020-01-16T00:12:47.990 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.199e-06 2020-01-16T00:12:47.990 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.990 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.990 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:47.991 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.932589+0000", 2020-01-16T00:12:47.991 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033658000000000001 2020-01-16T00:12:47.991 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.992 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.993 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:47.993 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.932926+0000", 2020-01-16T00:12:47.993 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000174749 2020-01-16T00:12:47.993 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.993 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.993 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:47.994 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933101+0000", 2020-01-16T00:12:47.994 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0054526330000000001 2020-01-16T00:12:47.994 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.994 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.994 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:47.994 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.938553+0000", 2020-01-16T00:12:47.994 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00026010999999999997 2020-01-16T00:12:47.995 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.995 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.995 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:47.995 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.938813+0000", 2020-01-16T00:12:47.996 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012019494 2020-01-16T00:12:47.996 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.996 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.996 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:47.997 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.950833+0000", 2020-01-16T00:12:47.998 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6416999999999997e-05 2020-01-16T00:12:47.998 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.998 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.998 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:47.999 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.950909+0000", 2020-01-16T00:12:47.999 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000152855 2020-01-16T00:12:47.999 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:47.999 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:47.999 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2020-01-16T00:12:48.000 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.951062+0000", 2020-01-16T00:12:48.000 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0025050680000000001 2020-01-16T00:12:48.001 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.001 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.001 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:48.001 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953567+0000", 2020-01-16T00:12:48.001 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017207999999999999 2020-01-16T00:12:48.002 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.002 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.002 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:48.002 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953739+0000", 2020-01-16T00:12:48.003 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1077999999999999e-05 2020-01-16T00:12:48.003 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.003 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.003 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:48.004 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953820+0000", 2020-01-16T00:12:48.004 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1516e-05 2020-01-16T00:12:48.004 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.004 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.004 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:48.005 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953862+0000", 2020-01-16T00:12:48.005 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0030999999999997e-05 2020-01-16T00:12:48.005 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.005 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.006 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:48.006 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953902+0000", 2020-01-16T00:12:48.006 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.022666925000000001 2020-01-16T00:12:48.006 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.006 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:48.007 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.007 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.007 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.008 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:954 3.c 3:3877922e:::smithi12313934-31:head [write 2948041~346098] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:48.008 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.940459+0000", 2020-01-16T00:12:48.008 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 9.8415209650000008, 2020-01-16T00:12:48.008 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023904965, 2020-01-16T00:12:48.009 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:48.009 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:48.009 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:48.009 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:48.010 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:48.010 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 954 2020-01-16T00:12:48.011 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.011 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:48.011 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.012 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:48.012 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.940459+0000", 2020-01-16T00:12:48.012 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:48.012 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.013 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.013 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:48.013 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.940459+0000", 2020-01-16T00:12:48.013 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:48.014 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.014 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.014 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:48.014 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.940457+0000", 2020-01-16T00:12:48.014 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00067923600000000005 2020-01-16T00:12:48.015 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.015 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.015 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:48.015 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941136+0000", 2020-01-16T00:12:48.015 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6631999999999999e-05 2020-01-16T00:12:48.015 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.016 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.016 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:48.016 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941152+0000", 2020-01-16T00:12:48.016 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9340000000000003e-06 2020-01-16T00:12:48.016 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.016 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.017 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:48.017 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941158+0000", 2020-01-16T00:12:48.017 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8878000000000001e-05 2020-01-16T00:12:48.017 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.017 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.018 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.018 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941217+0000", 2020-01-16T00:12:48.018 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1224999999999997e-05 2020-01-16T00:12:48.018 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.019 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.019 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:48.019 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941289+0000", 2020-01-16T00:12:48.019 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011199931999999999 2020-01-16T00:12:48.019 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.019 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.020 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.020 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.952488+0000", 2020-01-16T00:12:48.020 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000240067 2020-01-16T00:12:48.020 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.020 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.020 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:48.020 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.952729+0000", 2020-01-16T00:12:48.020 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0094184660000000003 2020-01-16T00:12:48.021 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.021 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.021 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.021 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.962147+0000", 2020-01-16T00:12:48.021 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6094000000000006e-05 2020-01-16T00:12:48.022 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.022 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.022 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:48.022 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.962233+0000", 2020-01-16T00:12:48.022 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022739400000000001 2020-01-16T00:12:48.022 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.022 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.023 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2020-01-16T00:12:48.023 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.962460+0000", 2020-01-16T00:12:48.024 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014799710000000001 2020-01-16T00:12:48.024 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.024 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.024 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:48.024 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.963940+0000", 2020-01-16T00:12:48.024 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000279182 2020-01-16T00:12:48.024 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.025 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.025 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:48.025 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.964220+0000", 2020-01-16T00:12:48.025 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4141999999999997e-05 2020-01-16T00:12:48.025 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.025 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.025 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:48.025 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.964284+0000", 2020-01-16T00:12:48.026 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3547999999999997e-05 2020-01-16T00:12:48.026 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.026 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.026 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:48.026 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.964327+0000", 2020-01-16T00:12:48.026 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6341000000000003e-05 2020-01-16T00:12:48.026 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.026 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.027 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:48.027 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.964364+0000", 2020-01-16T00:12:48.027 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023904965 2020-01-16T00:12:48.027 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.027 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:48.027 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.027 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.027 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.028 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:955 3.c 3:3877922e:::smithi12313934-31:head [setxattr _header (56) in=7b,truncate 3294139] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:48.028 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.941169+0000", 2020-01-16T00:12:48.028 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 9.8408109180000007, 2020-01-16T00:12:48.028 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023400259999999999, 2020-01-16T00:12:48.028 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:48.028 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:48.028 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:48.028 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:48.029 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:48.029 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 955 2020-01-16T00:12:48.029 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.030 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:48.030 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.030 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:48.031 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941169+0000", 2020-01-16T00:12:48.031 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:48.031 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.031 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.031 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:48.031 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941169+0000", 2020-01-16T00:12:48.031 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:48.032 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.032 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.032 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:48.032 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941168+0000", 2020-01-16T00:12:48.032 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.6700000000000006e-06 2020-01-16T00:12:48.032 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.032 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.032 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:48.033 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941177+0000", 2020-01-16T00:12:48.033 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6420000000000001e-06 2020-01-16T00:12:48.033 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.033 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.033 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:48.033 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941184+0000", 2020-01-16T00:12:48.033 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5019999999999998e-06 2020-01-16T00:12:48.033 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.034 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.034 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:48.034 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941190+0000", 2020-01-16T00:12:48.034 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013520700000000001 2020-01-16T00:12:48.034 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.034 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.034 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.035 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941326+0000", 2020-01-16T00:12:48.035 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1218999999999998e-05 2020-01-16T00:12:48.035 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.035 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.035 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:48.035 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941387+0000", 2020-01-16T00:12:48.035 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011403931000000001 2020-01-16T00:12:48.035 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.036 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.036 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.036 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.952791+0000", 2020-01-16T00:12:48.036 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6416999999999998e-05 2020-01-16T00:12:48.036 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.036 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.036 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:48.036 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.952857+0000", 2020-01-16T00:12:48.037 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010427334999999999 2020-01-16T00:12:48.037 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.037 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.037 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.037 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.963284+0000", 2020-01-16T00:12:48.037 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.9720000000000002e-05 2020-01-16T00:12:48.037 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.037 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.038 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:48.038 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.963374+0000", 2020-01-16T00:12:48.038 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000131137 2020-01-16T00:12:48.038 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.038 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.038 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2020-01-16T00:12:48.038 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.963505+0000", 2020-01-16T00:12:48.038 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00065508899999999998 2020-01-16T00:12:48.039 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.039 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.039 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:48.039 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.964160+0000", 2020-01-16T00:12:48.039 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000279265 2020-01-16T00:12:48.039 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.039 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.040 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:48.040 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.964440+0000", 2020-01-16T00:12:48.040 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5080999999999997e-05 2020-01-16T00:12:48.040 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.040 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.040 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:48.040 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.964485+0000", 2020-01-16T00:12:48.041 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0565999999999998e-05 2020-01-16T00:12:48.041 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.041 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.041 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:48.041 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.964525+0000", 2020-01-16T00:12:48.041 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3662999999999999e-05 2020-01-16T00:12:48.042 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.042 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.042 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:48.042 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.964569+0000", 2020-01-16T00:12:48.042 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023400259999999999 2020-01-16T00:12:48.042 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.042 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:48.043 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.043 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.043 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.043 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:956 3.c 3:3877922e:::smithi12313934-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:12:48.043 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.941200+0000", 2020-01-16T00:12:48.043 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 9.8407800660000007, 2020-01-16T00:12:48.043 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023677922000000001, 2020-01-16T00:12:48.043 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:48.044 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:48.044 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:48.044 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:48.044 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:48.044 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 956 2020-01-16T00:12:48.044 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.044 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:48.044 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.045 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:48.045 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941200+0000", 2020-01-16T00:12:48.045 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:48.045 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.045 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.045 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:48.045 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941200+0000", 2020-01-16T00:12:48.045 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:48.046 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.046 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.046 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:48.046 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941199+0000", 2020-01-16T00:12:48.046 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9330000000000003e-06 2020-01-16T00:12:48.046 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.046 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.046 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:48.047 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941203+0000", 2020-01-16T00:12:48.047 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1578000000000002e-05 2020-01-16T00:12:48.047 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.047 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.047 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:48.047 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941234+0000", 2020-01-16T00:12:48.047 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2530000000000004e-06 2020-01-16T00:12:48.047 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.048 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.048 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:48.048 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941240+0000", 2020-01-16T00:12:48.048 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016458900000000001 2020-01-16T00:12:48.048 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.048 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.048 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.048 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941404+0000", 2020-01-16T00:12:48.049 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3485000000000002e-05 2020-01-16T00:12:48.049 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.049 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.049 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:48.049 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941438+0000", 2020-01-16T00:12:48.049 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011489497 2020-01-16T00:12:48.050 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.050 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.050 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.050 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.952927+0000", 2020-01-16T00:12:48.050 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4110999999999999e-05 2020-01-16T00:12:48.050 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.051 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.051 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:48.051 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.952961+0000", 2020-01-16T00:12:48.051 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010832766000000001 2020-01-16T00:12:48.051 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.051 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.051 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.051 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.963794+0000", 2020-01-16T00:12:48.052 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012301900000000001 2020-01-16T00:12:48.052 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.052 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.052 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:48.052 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.963917+0000", 2020-01-16T00:12:48.052 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00069668799999999999 2020-01-16T00:12:48.052 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.052 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.053 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.053 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.964614+0000", 2020-01-16T00:12:48.053 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012451500000000001 2020-01-16T00:12:48.053 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.053 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.053 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:48.053 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.964738+0000", 2020-01-16T00:12:48.053 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000139128 2020-01-16T00:12:48.054 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.054 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.054 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:48.054 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.964878+0000", 2020-01-16T00:12:48.054 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023677922000000001 2020-01-16T00:12:48.054 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.054 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:48.054 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.055 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.055 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.055 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:979 3.5 3:ad8ab15f:::smithi12313934-27:head [read 0~3210104,checksum] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:48.055 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315954+0000", 2020-01-16T00:12:48.055 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 9.4660259450000002, 2020-01-16T00:12:48.055 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010639968, 2020-01-16T00:12:48.055 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:48.055 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:48.056 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:48.056 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:48.056 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:48.056 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 979 2020-01-16T00:12:48.056 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.056 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:48.056 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.056 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:48.057 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315954+0000", 2020-01-16T00:12:48.057 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:48.057 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.057 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.057 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:48.057 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315954+0000", 2020-01-16T00:12:48.057 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:48.057 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.058 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.058 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:48.058 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315953+0000", 2020-01-16T00:12:48.058 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.1200000000000008e-06 2020-01-16T00:12:48.058 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.058 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.058 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:48.059 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315962+0000", 2020-01-16T00:12:48.059 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.488e-05 2020-01-16T00:12:48.059 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.059 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.059 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:48.059 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315977+0000", 2020-01-16T00:12:48.059 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6283e-05 2020-01-16T00:12:48.059 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.060 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.060 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:48.060 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315993+0000", 2020-01-16T00:12:48.060 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0079730960000000007 2020-01-16T00:12:48.061 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.061 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.061 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.061 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.323966+0000", 2020-01-16T00:12:48.061 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020471399999999999 2020-01-16T00:12:48.061 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.061 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.061 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:48.062 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.324171+0000", 2020-01-16T00:12:48.062 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0024230290000000002 2020-01-16T00:12:48.062 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.062 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.062 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:48.062 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.326594+0000", 2020-01-16T00:12:48.062 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010639968 2020-01-16T00:12:48.063 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.063 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:48.063 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.063 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.063 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.063 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:980 3.5 3:ad8ab15f:::smithi12313934-27:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:48.063 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.316004+0000", 2020-01-16T00:12:48.064 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 9.4659755329999999, 2020-01-16T00:12:48.064 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011373866, 2020-01-16T00:12:48.064 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:48.064 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:48.064 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:48.064 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:48.064 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:48.065 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 980 2020-01-16T00:12:48.065 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.065 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:48.065 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.065 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:48.065 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316004+0000", 2020-01-16T00:12:48.065 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:48.065 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.066 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.066 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:48.066 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316004+0000", 2020-01-16T00:12:48.066 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:48.066 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.066 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.066 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:48.066 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316003+0000", 2020-01-16T00:12:48.067 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.9419999999999998e-06 2020-01-16T00:12:48.067 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.067 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.067 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:48.067 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316010+0000", 2020-01-16T00:12:48.067 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4960000000000001e-05 2020-01-16T00:12:48.067 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.067 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.068 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:48.068 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316025+0000", 2020-01-16T00:12:48.068 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2890000000000003e-06 2020-01-16T00:12:48.068 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.068 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.068 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:48.068 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316034+0000", 2020-01-16T00:12:48.068 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010653673000000001 2020-01-16T00:12:48.069 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.069 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.069 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.069 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.326688+0000", 2020-01-16T00:12:48.069 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000231511 2020-01-16T00:12:48.069 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.069 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.069 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:48.070 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.326919+0000", 2020-01-16T00:12:48.070 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00045858800000000001 2020-01-16T00:12:48.070 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.070 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.070 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:48.070 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.327378+0000", 2020-01-16T00:12:48.070 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011373866 2020-01-16T00:12:48.070 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.071 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:48.071 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.071 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.071 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.071 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.1.13:47 2.3 2:c6fec7d0:::smithi12313934-26 low_tier:head [write 963072~376320] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e26)", 2020-01-16T00:12:48.071 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:41.673013+0000", 2020-01-16T00:12:48.071 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 6.1089663930000002, 2020-01-16T00:12:48.072 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0095009329999999996, 2020-01-16T00:12:48.072 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:48.072 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:48.072 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:48.072 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.072 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:48.072 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.673013+0000", 2020-01-16T00:12:48.073 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:48.073 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.073 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.073 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:48.073 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.673013+0000", 2020-01-16T00:12:48.073 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:48.073 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.074 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.074 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:48.074 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.673011+0000", 2020-01-16T00:12:48.074 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000590852 2020-01-16T00:12:48.074 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.074 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.074 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:48.074 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.673602+0000", 2020-01-16T00:12:48.075 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9289999999999998e-05 2020-01-16T00:12:48.075 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.075 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.075 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:48.075 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.673641+0000", 2020-01-16T00:12:48.075 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0559e-05 2020-01-16T00:12:48.075 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.075 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.076 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:48.076 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.673652+0000", 2020-01-16T00:12:48.076 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5334999999999997e-05 2020-01-16T00:12:48.076 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.076 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.076 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.076 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.673717+0000", 2020-01-16T00:12:48.076 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000109041 2020-01-16T00:12:48.077 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.077 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.082 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:48.082 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.673826+0000", 2020-01-16T00:12:48.082 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015066 2020-01-16T00:12:48.082 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.082 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.082 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:12:48.082 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.673977+0000", 2020-01-16T00:12:48.083 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0082881929999999993 2020-01-16T00:12:48.083 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.083 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.083 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:48.083 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.682265+0000", 2020-01-16T00:12:48.083 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.9645000000000002e-05 2020-01-16T00:12:48.083 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.083 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.084 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:48.084 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.682334+0000", 2020-01-16T00:12:48.084 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5344000000000001e-05 2020-01-16T00:12:48.084 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.084 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.084 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:48.084 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.682350+0000", 2020-01-16T00:12:48.084 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6589999999999997e-05 2020-01-16T00:12:48.085 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.085 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.085 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:48.085 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.682426+0000", 2020-01-16T00:12:48.085 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7816000000000001e-05 2020-01-16T00:12:48.085 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.085 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.085 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:48.086 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.682514+0000", 2020-01-16T00:12:48.086 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0095009329999999996 2020-01-16T00:12:48.086 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.086 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:48.086 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.086 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.086 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.086 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.1.13:48 2.3 2:c6fec7d0:::smithi12313934-26 low_tier:head [write 1494016~315392] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e26)", 2020-01-16T00:12:48.087 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:41.673668+0000", 2020-01-16T00:12:48.087 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 6.1083112719999999, 2020-01-16T00:12:48.087 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0090818140000000006, 2020-01-16T00:12:48.087 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:48.087 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:48.087 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:48.087 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.087 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:48.088 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.673668+0000", 2020-01-16T00:12:48.088 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:48.088 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.088 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.088 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:48.088 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.673668+0000", 2020-01-16T00:12:48.088 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:48.088 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.089 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.089 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:48.089 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.673667+0000", 2020-01-16T00:12:48.089 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00031002999999999998 2020-01-16T00:12:48.089 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.089 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.089 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:48.090 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.673977+0000", 2020-01-16T00:12:48.090 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8076000000000001e-05 2020-01-16T00:12:48.090 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.090 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.090 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:48.090 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.673995+0000", 2020-01-16T00:12:48.090 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9789999999999998e-06 2020-01-16T00:12:48.090 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.091 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.091 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:48.091 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.674001+0000", 2020-01-16T00:12:48.091 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0017961819999999999 2020-01-16T00:12:48.091 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.091 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.091 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.091 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.675797+0000", 2020-01-16T00:12:48.092 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013250299999999999 2020-01-16T00:12:48.092 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.092 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.092 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:48.092 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.675930+0000", 2020-01-16T00:12:48.092 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022845999999999999 2020-01-16T00:12:48.092 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.092 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.093 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:12:48.093 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.676158+0000", 2020-01-16T00:12:48.093 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0063733770000000004 2020-01-16T00:12:48.093 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.093 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.093 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:48.094 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.682532+0000", 2020-01-16T00:12:48.094 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3147999999999999e-05 2020-01-16T00:12:48.094 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.094 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.094 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:48.094 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.682595+0000", 2020-01-16T00:12:48.094 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3842999999999999e-05 2020-01-16T00:12:48.094 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.095 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.095 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:48.095 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.682649+0000", 2020-01-16T00:12:48.095 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2639000000000003e-05 2020-01-16T00:12:48.095 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.095 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.095 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:48.095 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.682691+0000", 2020-01-16T00:12:48.096 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8822e-05 2020-01-16T00:12:48.096 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.096 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.096 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:48.096 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.682750+0000", 2020-01-16T00:12:48.096 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0090818140000000006 2020-01-16T00:12:48.096 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.096 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:48.097 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.097 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.097 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.097 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.1.13:49 2.3 2:c6fec7d0:::smithi12313934-26 low_tier:head [write 1965056~33280] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e26)", 2020-01-16T00:12:48.097 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:41.674014+0000", 2020-01-16T00:12:48.097 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 6.107965192, 2020-01-16T00:12:48.097 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0090726620000000004, 2020-01-16T00:12:48.098 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:48.098 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:48.098 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:48.098 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.098 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:48.098 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.674014+0000", 2020-01-16T00:12:48.098 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:48.099 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.099 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.099 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:48.099 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.674014+0000", 2020-01-16T00:12:48.099 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:48.099 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.099 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.099 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:48.100 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.674014+0000", 2020-01-16T00:12:48.100 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9543000000000002e-05 2020-01-16T00:12:48.100 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.100 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.100 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:48.100 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.674043+0000", 2020-01-16T00:12:48.100 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2354999999999999e-05 2020-01-16T00:12:48.101 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.101 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.101 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:48.101 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.674055+0000", 2020-01-16T00:12:48.101 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0391000000000001e-05 2020-01-16T00:12:48.101 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.101 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.101 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:48.102 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.674086+0000", 2020-01-16T00:12:48.102 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003699777 2020-01-16T00:12:48.102 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.102 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.102 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.102 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.677786+0000", 2020-01-16T00:12:48.102 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000101382 2020-01-16T00:12:48.102 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.103 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.103 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:48.103 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.677887+0000", 2020-01-16T00:12:48.103 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000205923 2020-01-16T00:12:48.103 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.103 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.103 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:12:48.103 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.678093+0000", 2020-01-16T00:12:48.104 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.004722955 2020-01-16T00:12:48.104 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.104 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.104 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:48.104 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.682816+0000", 2020-01-16T00:12:48.104 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4629999999999998e-05 2020-01-16T00:12:48.104 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.104 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.105 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:48.105 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.682860+0000", 2020-01-16T00:12:48.105 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000144676 2020-01-16T00:12:48.105 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.105 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.105 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:48.105 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.683005+0000", 2020-01-16T00:12:48.105 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4332999999999997e-05 2020-01-16T00:12:48.106 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.106 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.106 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:48.106 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.683049+0000", 2020-01-16T00:12:48.106 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7638000000000001e-05 2020-01-16T00:12:48.106 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.106 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.106 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:48.107 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.683087+0000", 2020-01-16T00:12:48.107 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0090726620000000004 2020-01-16T00:12:48.107 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.107 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:48.107 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.107 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.107 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:48.107 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:48.108 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2020-01-16T00:12:48.112 INFO:tasks.rados.rados.0.smithi123.stdout:787: read oid 1 snap -1 2020-01-16T00:12:48.112 INFO:tasks.rados.rados.0.smithi123.stdout:787: expect (ObjNum 83 snap 0 seq_num 83) 2020-01-16T00:12:48.112 INFO:tasks.rados.rados.0.smithi123.stdout:784: done (3 left) 2020-01-16T00:12:48.112 INFO:tasks.rados.rados.0.smithi123.stdout:788: read oid 12 snap -1 2020-01-16T00:12:48.113 INFO:tasks.rados.rados.0.smithi123.stdout:788: expect (ObjNum 121 snap 0 seq_num 121) 2020-01-16T00:12:48.113 INFO:tasks.rados.rados.0.smithi123.stdout:789: write oid 45 current snap is 0 2020-01-16T00:12:48.113 INFO:tasks.rados.rados.0.smithi123.stdout:789: seq_num 142 ranges {786594=779712,2326443=60151} 2020-01-16T00:12:48.113 INFO:tasks.rados.rados.0.smithi123.stdout:789: writing smithi12313934-45 from 786594 to 1566306 tid 1 2020-01-16T00:12:48.113 INFO:tasks.rados.rados.0.smithi123.stdout:789: writing smithi12313934-45 from 2326443 to 2386594 tid 2 2020-01-16T00:12:48.113 INFO:tasks.rados.rados.0.smithi123.stdout:790: read oid 22 snap -1 2020-01-16T00:12:48.113 INFO:tasks.rados.rados.0.smithi123.stdout:790: expect (ObjNum 123 snap 0 seq_num 123) 2020-01-16T00:12:48.113 INFO:tasks.rados.rados.0.smithi123.stdout:791: write (excl) oid 32 current snap is 0 2020-01-16T00:12:48.114 INFO:tasks.rados.rados.0.smithi123.stdout:791: seq_num 143 ranges {403401=454961,1584925=506039} 2020-01-16T00:12:48.114 INFO:tasks.rados.rados.0.smithi123.stdout:791: writing smithi12313934-32 from 403401 to 858362 tid 1 2020-01-16T00:12:48.114 INFO:tasks.rados.rados.0.smithi123.stdout:791: writing smithi12313934-32 from 1584925 to 2090964 tid 2 2020-01-16T00:12:48.114 INFO:tasks.rados.rados.0.smithi123.stdout:792: read oid 37 snap -1 2020-01-16T00:12:48.114 INFO:tasks.rados.rados.0.smithi123.stdout:792: expect (ObjNum 122 snap 0 seq_num 122) 2020-01-16T00:12:48.114 INFO:tasks.rados.rados.0.smithi123.stdout:785: finishing write tid 1 to smithi12313934-3 2020-01-16T00:12:48.114 INFO:tasks.rados.rados.0.smithi123.stdout:793: read oid 35 snap -1 2020-01-16T00:12:48.114 INFO:tasks.rados.rados.0.smithi123.stdout:793: expect (ObjNum 136 snap 0 seq_num 136) 2020-01-16T00:12:48.115 INFO:tasks.rados.rados.0.smithi123.stdout:794: read oid 9 snap -1 2020-01-16T00:12:48.115 INFO:tasks.rados.rados.0.smithi123.stdout:794: expect (ObjNum 91 snap 0 seq_num 91) 2020-01-16T00:12:48.115 INFO:tasks.rados.rados.0.smithi123.stdout:795: read oid 36 snap -1 2020-01-16T00:12:48.115 INFO:tasks.rados.rados.0.smithi123.stdout:795: expect (ObjNum 94 snap 0 seq_num 94) 2020-01-16T00:12:48.115 INFO:tasks.rados.rados.0.smithi123.stdout:796: write oid 20 current snap is 0 2020-01-16T00:12:48.115 INFO:tasks.rados.rados.0.smithi123.stdout:796: seq_num 144 ranges {420208=530210,1527054=766050} 2020-01-16T00:12:48.115 INFO:tasks.rados.rados.0.smithi123.stdout:796: writing smithi12313934-20 from 420208 to 950418 tid 1 2020-01-16T00:12:48.116 INFO:tasks.rados.rados.0.smithi123.stdout:796: writing smithi12313934-20 from 1527054 to 2293104 tid 2 2020-01-16T00:12:48.116 INFO:tasks.rados.rados.0.smithi123.stdout:797: read oid 48 snap -1 2020-01-16T00:12:48.116 INFO:tasks.rados.rados.0.smithi123.stdout:797: expect (ObjNum 75 snap 0 seq_num 75) 2020-01-16T00:12:48.116 INFO:tasks.rados.rados.0.smithi123.stdout:798: read oid 17 snap -1 2020-01-16T00:12:48.116 INFO:tasks.rados.rados.0.smithi123.stdout:798: expect deleted 2020-01-16T00:12:48.116 INFO:tasks.rados.rados.0.smithi123.stdout:799: write oid 24 current snap is 0 2020-01-16T00:12:48.116 INFO:tasks.rados.rados.0.smithi123.stdout:799: seq_num 145 ranges {437015=721812,1701889=509708} 2020-01-16T00:12:48.117 INFO:tasks.rados.rados.0.smithi123.stdout:799: writing smithi12313934-24 from 437015 to 1158827 tid 1 2020-01-16T00:12:48.117 INFO:tasks.rados.rados.0.smithi123.stdout:799: writing smithi12313934-24 from 1701889 to 2211597 tid 2 2020-01-16T00:12:48.117 INFO:tasks.rados.rados.0.smithi123.stdout:800: write oid 34 current snap is 0 2020-01-16T00:12:48.117 INFO:tasks.rados.rados.0.smithi123.stdout:800: seq_num 146 ranges {453822=797061,1760371=653366} 2020-01-16T00:12:48.117 INFO:tasks.rados.rados.0.smithi123.stdout:800: writing smithi12313934-34 from 453822 to 1250883 tid 1 2020-01-16T00:12:48.117 INFO:tasks.rados.rados.0.smithi123.stdout:800: writing smithi12313934-34 from 1760371 to 2413737 tid 2 2020-01-16T00:12:48.117 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:48.117 INFO:tasks.rados.rados.0.smithi123.stdout:785: finishing write tid 2 to smithi12313934-3 2020-01-16T00:12:48.118 INFO:tasks.rados.rados.0.smithi123.stdout:785: finishing write tid 4 to smithi12313934-3 2020-01-16T00:12:48.118 INFO:tasks.rados.rados.0.smithi123.stdout:785: finishing write tid 5 to smithi12313934-3 2020-01-16T00:12:48.118 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 3 v 47 (ObjNum 141 snap 0 seq_num 141) dirty exists 2020-01-16T00:12:48.118 INFO:tasks.rados.rados.0.smithi123.stdout:785: left oid 3 (ObjNum 141 snap 0 seq_num 141) 2020-01-16T00:12:48.118 INFO:tasks.rados.rados.0.smithi123.stdout:786: expect (ObjNum 127 snap 0 seq_num 127) 2020-01-16T00:12:48.118 INFO:tasks.rados.rados.0.smithi123.stdout:788: expect (ObjNum 121 snap 0 seq_num 121) 2020-01-16T00:12:48.119 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:48.109+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:48.119 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-01-16T00:12:48.140 INFO:tasks.rados.rados.0.smithi123.stdout:787: expect (ObjNum 83 snap 0 seq_num 83) 2020-01-16T00:12:48.210 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:48.209+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:48.272 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:48.272 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:12:48.272 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:12:48.273 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:48.282 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-01-16T00:12:48.311 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:48.311+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:12:48.327 INFO:tasks.rados.rados.0.smithi123.stdout:789: finishing write tid 1 to smithi12313934-45 2020-01-16T00:12:48.328 INFO:tasks.rados.rados.0.smithi123.stdout:789: finishing write tid 2 to smithi12313934-45 2020-01-16T00:12:48.328 INFO:tasks.rados.rados.0.smithi123.stdout:790: expect (ObjNum 123 snap 0 seq_num 123) 2020-01-16T00:12:48.411 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:12:48.411+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:12:48.434 INFO:tasks.rados.rados.0.smithi123.stdout:791: finishing write tid 1 to smithi12313934-32 2020-01-16T00:12:48.434 INFO:tasks.rados.rados.0.smithi123.stdout:789: finishing write tid 4 to smithi12313934-45 2020-01-16T00:12:48.434 INFO:tasks.rados.rados.0.smithi123.stdout:789: finishing write tid 5 to smithi12313934-45 2020-01-16T00:12:48.434 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 45 v 67 (ObjNum 142 snap 0 seq_num 142) dirty exists 2020-01-16T00:12:48.435 INFO:tasks.rados.rados.0.smithi123.stdout:789: left oid 45 (ObjNum 142 snap 0 seq_num 142) 2020-01-16T00:12:48.435 INFO:tasks.rados.rados.0.smithi123.stdout:795: expect (ObjNum 94 snap 0 seq_num 94) 2020-01-16T00:12:48.435 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:48.435 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:12:48.435 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:12:48.435 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:12:48.436 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:48.444 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-01-16T00:12:48.513 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:12:48.515+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:12:48.547 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:48.547 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:12:48.547 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:12:48.548 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:12:48.548 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.548 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:895 3.3 3:cd231094:::smithi12313934-49:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:48.548 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.122390+0000", 2020-01-16T00:12:48.548 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 11.425383261, 2020-01-16T00:12:48.548 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014295501, 2020-01-16T00:12:48.548 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:48.549 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:48.549 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:48.549 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:48.549 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:48.549 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 895 2020-01-16T00:12:48.549 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.550 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:48.550 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.550 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:48.550 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122390+0000", 2020-01-16T00:12:48.550 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:48.550 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.550 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.550 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:48.551 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122390+0000", 2020-01-16T00:12:48.551 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999957 2020-01-16T00:12:48.551 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.551 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.551 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:48.551 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122386+0000", 2020-01-16T00:12:48.551 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0656999999999999e-05 2020-01-16T00:12:48.551 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.552 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.552 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:48.552 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122407+0000", 2020-01-16T00:12:48.552 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9102999999999999e-05 2020-01-16T00:12:48.552 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.552 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.552 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:48.552 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122436+0000", 2020-01-16T00:12:48.553 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.774e-05 2020-01-16T00:12:48.553 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.553 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.553 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:48.553 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122483+0000", 2020-01-16T00:12:48.553 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5185000000000004e-05 2020-01-16T00:12:48.553 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.554 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.554 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.554 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122579+0000", 2020-01-16T00:12:48.554 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000304085 2020-01-16T00:12:48.554 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.554 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.554 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:48.554 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122883+0000", 2020-01-16T00:12:48.555 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0092077970000000002 2020-01-16T00:12:48.555 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.555 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.555 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.555 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132091+0000", 2020-01-16T00:12:48.555 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000170025 2020-01-16T00:12:48.555 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.555 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.556 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:48.556 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132261+0000", 2020-01-16T00:12:48.556 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0040130979999999997 2020-01-16T00:12:48.556 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.556 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.556 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.556 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136274+0000", 2020-01-16T00:12:48.557 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014227900000000001 2020-01-16T00:12:48.557 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.557 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.557 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:48.557 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136416+0000", 2020-01-16T00:12:48.557 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00026995200000000001 2020-01-16T00:12:48.557 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.557 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.558 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:48.558 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136686+0000", 2020-01-16T00:12:48.558 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014295501 2020-01-16T00:12:48.558 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.558 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:48.558 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.558 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.559 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.559 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:896 3.3 3:cd231094:::smithi12313934-49:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:48.559 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.122539+0000", 2020-01-16T00:12:48.559 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 11.425234261, 2020-01-16T00:12:48.559 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014440173000000001, 2020-01-16T00:12:48.560 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:48.560 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:48.560 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:48.560 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:48.560 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:48.560 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 896 2020-01-16T00:12:48.561 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.561 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:48.561 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.561 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:48.561 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122539+0000", 2020-01-16T00:12:48.561 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:48.561 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.562 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.562 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:48.562 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122539+0000", 2020-01-16T00:12:48.562 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:48.562 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.562 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.563 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:48.563 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122537+0000", 2020-01-16T00:12:48.563 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2432999999999999e-05 2020-01-16T00:12:48.563 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.563 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.563 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:48.563 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122550+0000", 2020-01-16T00:12:48.564 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9369999999999999e-05 2020-01-16T00:12:48.564 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.564 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.564 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:48.564 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122569+0000", 2020-01-16T00:12:48.564 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1747e-05 2020-01-16T00:12:48.564 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.565 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.565 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:48.565 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122581+0000", 2020-01-16T00:12:48.565 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00041555800000000001 2020-01-16T00:12:48.565 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.565 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.565 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.566 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122996+0000", 2020-01-16T00:12:48.566 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016840299999999999 2020-01-16T00:12:48.566 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.566 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.566 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:48.566 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.123165+0000", 2020-01-16T00:12:48.566 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0091641359999999998 2020-01-16T00:12:48.566 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.567 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.567 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.567 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132329+0000", 2020-01-16T00:12:48.568 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013433400000000001 2020-01-16T00:12:48.568 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.568 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.568 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:48.568 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132463+0000", 2020-01-16T00:12:48.568 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0042524110000000002 2020-01-16T00:12:48.568 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.569 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.569 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.569 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136716+0000", 2020-01-16T00:12:48.569 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4591999999999993e-05 2020-01-16T00:12:48.569 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.569 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.569 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:48.569 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136790+0000", 2020-01-16T00:12:48.570 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018927500000000001 2020-01-16T00:12:48.570 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.570 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.570 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:48.570 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136980+0000", 2020-01-16T00:12:48.570 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014440173000000001 2020-01-16T00:12:48.570 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.570 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:48.571 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.571 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.571 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.571 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:897 3.3 3:cd231094:::smithi12313934-49:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:48.571 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.122613+0000", 2020-01-16T00:12:48.571 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 11.425161117, 2020-01-16T00:12:48.571 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014606846, 2020-01-16T00:12:48.572 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:48.572 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:48.572 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:48.572 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:48.572 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:48.581 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 897 2020-01-16T00:12:48.581 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.581 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:48.581 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.581 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:48.581 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122613+0000", 2020-01-16T00:12:48.581 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:48.582 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.582 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.582 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:48.582 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122613+0000", 2020-01-16T00:12:48.582 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:48.582 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.582 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.583 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:48.583 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122611+0000", 2020-01-16T00:12:48.583 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3670000000000003e-06 2020-01-16T00:12:48.583 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.583 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.583 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:48.583 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122620+0000", 2020-01-16T00:12:48.583 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5193e-05 2020-01-16T00:12:48.584 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.584 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.584 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:48.584 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122635+0000", 2020-01-16T00:12:48.585 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0696999999999999e-05 2020-01-16T00:12:48.585 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.585 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.585 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:48.585 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122646+0000", 2020-01-16T00:12:48.586 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00059961400000000005 2020-01-16T00:12:48.586 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.586 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.586 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.587 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.123246+0000", 2020-01-16T00:12:48.587 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011189200000000001 2020-01-16T00:12:48.587 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.587 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.587 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:48.587 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.123358+0000", 2020-01-16T00:12:48.587 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0091363550000000005 2020-01-16T00:12:48.588 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.588 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.588 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.588 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132494+0000", 2020-01-16T00:12:48.588 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2956999999999998e-05 2020-01-16T00:12:48.588 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.588 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.589 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:48.589 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132587+0000", 2020-01-16T00:12:48.589 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0044061769999999998 2020-01-16T00:12:48.589 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.589 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.589 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.589 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136993+0000", 2020-01-16T00:12:48.589 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7822999999999998e-05 2020-01-16T00:12:48.590 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.590 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.590 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:48.590 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.137051+0000", 2020-01-16T00:12:48.590 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000168498 2020-01-16T00:12:48.590 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.590 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.590 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:48.591 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.137219+0000", 2020-01-16T00:12:48.591 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014606846 2020-01-16T00:12:48.591 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.591 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:48.591 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.591 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.591 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.592 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:943 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:48.592 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:12:48.592 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 10.633295143, 2020-01-16T00:12:48.592 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018847004000000001, 2020-01-16T00:12:48.592 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:48.592 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:48.592 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:48.592 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:48.593 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:48.593 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 943 2020-01-16T00:12:48.593 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.593 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:48.593 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.593 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:48.593 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:12:48.593 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:48.594 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.594 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.594 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:48.594 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:12:48.594 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:12:48.594 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.594 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.595 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:48.595 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914475+0000", 2020-01-16T00:12:48.595 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.508e-05 2020-01-16T00:12:48.595 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.595 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.595 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:48.595 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914490+0000", 2020-01-16T00:12:48.595 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.857e-05 2020-01-16T00:12:48.596 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.596 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.596 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:48.596 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914509+0000", 2020-01-16T00:12:48.596 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.46e-06 2020-01-16T00:12:48.596 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.596 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.596 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:48.597 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914515+0000", 2020-01-16T00:12:48.597 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2612999999999999e-05 2020-01-16T00:12:48.597 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.597 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.597 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.597 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914568+0000", 2020-01-16T00:12:48.597 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028312299999999998 2020-01-16T00:12:48.597 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.598 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.598 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:48.598 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914851+0000", 2020-01-16T00:12:48.598 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012282533 2020-01-16T00:12:48.598 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.598 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.598 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.598 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927133+0000", 2020-01-16T00:12:48.599 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4839999999999995e-05 2020-01-16T00:12:48.599 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.599 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.599 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:48.599 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927208+0000", 2020-01-16T00:12:48.599 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0047425610000000002 2020-01-16T00:12:48.599 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.599 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.600 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.600 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.931951+0000", 2020-01-16T00:12:48.600 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3628999999999994e-05 2020-01-16T00:12:48.600 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.600 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.600 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:48.600 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.932045+0000", 2020-01-16T00:12:48.601 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012810180000000001 2020-01-16T00:12:48.601 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.601 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.601 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:48.601 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933326+0000", 2020-01-16T00:12:48.601 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018847004000000001 2020-01-16T00:12:48.601 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.601 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:48.602 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.602 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.602 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.602 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:944 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:48.602 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:12:48.602 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 10.633245586999999, 2020-01-16T00:12:48.602 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019086901999999999, 2020-01-16T00:12:48.602 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:48.603 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:48.603 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:48.603 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:48.603 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:48.603 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 944 2020-01-16T00:12:48.603 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.603 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:48.603 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.604 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:48.604 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:12:48.604 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:48.604 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.604 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.604 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:48.604 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:12:48.604 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:48.605 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.605 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.605 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:48.605 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914527+0000", 2020-01-16T00:12:48.605 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7509999999999999e-06 2020-01-16T00:12:48.605 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.605 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.605 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:48.606 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914532+0000", 2020-01-16T00:12:48.606 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5149999999999997e-06 2020-01-16T00:12:48.606 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.606 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.606 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:48.606 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914538+0000", 2020-01-16T00:12:48.606 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1890000000000007e-06 2020-01-16T00:12:48.606 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.607 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.607 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:48.607 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914546+0000", 2020-01-16T00:12:48.607 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000354951 2020-01-16T00:12:48.607 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.607 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.607 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.607 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914901+0000", 2020-01-16T00:12:48.608 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1594000000000002e-05 2020-01-16T00:12:48.608 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.608 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.608 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:48.608 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914942+0000", 2020-01-16T00:12:48.608 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012300070999999999 2020-01-16T00:12:48.608 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.609 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.609 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.609 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927242+0000", 2020-01-16T00:12:48.609 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4076999999999997e-05 2020-01-16T00:12:48.609 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.609 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.609 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:48.610 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927296+0000", 2020-01-16T00:12:48.610 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0060609180000000002 2020-01-16T00:12:48.610 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.610 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.610 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.610 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933357+0000", 2020-01-16T00:12:48.611 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4488999999999999e-05 2020-01-16T00:12:48.611 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.611 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.611 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:48.611 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933422+0000", 2020-01-16T00:12:48.611 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019318300000000001 2020-01-16T00:12:48.611 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.612 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.612 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:48.612 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933615+0000", 2020-01-16T00:12:48.612 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019086901999999999 2020-01-16T00:12:48.612 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.612 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:48.612 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.612 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.613 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.613 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:945 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:48.613 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:12:48.613 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 10.633218381000001, 2020-01-16T00:12:48.613 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019342986, 2020-01-16T00:12:48.613 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:48.613 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:48.613 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:48.614 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:48.614 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:48.614 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 945 2020-01-16T00:12:48.614 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.614 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:48.614 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.614 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:48.615 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:12:48.615 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:48.615 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.615 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.615 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:48.615 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:12:48.615 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:48.615 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.616 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.616 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:48.616 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:12:48.616 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4580000000000002e-06 2020-01-16T00:12:48.616 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.616 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.616 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:48.616 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914558+0000", 2020-01-16T00:12:48.617 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7829999999999998e-06 2020-01-16T00:12:48.617 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.618 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.618 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:48.618 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914563+0000", 2020-01-16T00:12:48.619 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0001e-05 2020-01-16T00:12:48.619 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.619 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.619 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:48.619 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914573+0000", 2020-01-16T00:12:48.620 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00040365600000000001 2020-01-16T00:12:48.620 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.620 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.620 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.620 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914976+0000", 2020-01-16T00:12:48.620 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1390000000000001e-05 2020-01-16T00:12:48.620 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.620 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.621 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:48.621 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.915028+0000", 2020-01-16T00:12:48.621 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012288843000000001 2020-01-16T00:12:48.621 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.621 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.621 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.621 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927317+0000", 2020-01-16T00:12:48.621 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7871000000000002e-05 2020-01-16T00:12:48.622 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.622 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.622 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:48.622 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927375+0000", 2020-01-16T00:12:48.622 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0062580589999999998 2020-01-16T00:12:48.622 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.622 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.623 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.623 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933633+0000", 2020-01-16T00:12:48.623 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7086999999999997e-05 2020-01-16T00:12:48.623 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.623 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.623 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:48.623 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933690+0000", 2020-01-16T00:12:48.623 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020855799999999999 2020-01-16T00:12:48.624 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.624 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.624 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:48.624 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933898+0000", 2020-01-16T00:12:48.624 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019342986 2020-01-16T00:12:48.624 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.625 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:48.625 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.625 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.625 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.625 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:963 3.9 3:9557b633:::smithi12313934-43:head [read 0~2705894,checksum] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:48.625 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.046589+0000", 2020-01-16T00:12:48.625 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 10.501184533, 2020-01-16T00:12:48.625 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014014016000000001, 2020-01-16T00:12:48.626 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:48.626 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:48.626 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:48.626 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:48.626 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:48.626 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 963 2020-01-16T00:12:48.626 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.626 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:48.627 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.627 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:48.627 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046589+0000", 2020-01-16T00:12:48.627 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:48.627 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.627 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.627 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:48.627 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046589+0000", 2020-01-16T00:12:48.628 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:48.628 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.628 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.628 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:48.628 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046588+0000", 2020-01-16T00:12:48.628 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7469999999999994e-06 2020-01-16T00:12:48.628 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.628 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.629 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:48.629 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046596+0000", 2020-01-16T00:12:48.629 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3445999999999998e-05 2020-01-16T00:12:48.629 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.629 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.629 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:48.629 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046630+0000", 2020-01-16T00:12:48.630 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7779999999999999e-05 2020-01-16T00:12:48.630 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.630 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.630 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:48.630 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046648+0000", 2020-01-16T00:12:48.630 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00075081300000000002 2020-01-16T00:12:48.630 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.630 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.631 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.631 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.047398+0000", 2020-01-16T00:12:48.631 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000116698 2020-01-16T00:12:48.631 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.632 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.632 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:48.632 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.047515+0000", 2020-01-16T00:12:48.632 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0056630689999999997 2020-01-16T00:12:48.632 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.633 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.633 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.633 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.053178+0000", 2020-01-16T00:12:48.633 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012084299999999999 2020-01-16T00:12:48.633 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.633 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.634 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:48.634 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.053299+0000", 2020-01-16T00:12:48.634 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0056921510000000003 2020-01-16T00:12:48.634 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.634 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.635 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.635 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.058991+0000", 2020-01-16T00:12:48.635 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000118606 2020-01-16T00:12:48.635 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.636 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.636 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:48.636 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.059110+0000", 2020-01-16T00:12:48.636 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001493306 2020-01-16T00:12:48.636 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.637 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.637 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:48.637 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.060603+0000", 2020-01-16T00:12:48.637 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014014016000000001 2020-01-16T00:12:48.637 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.638 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:48.638 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.638 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.638 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.638 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:964 3.9 3:9557b633:::smithi12313934-43:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:48.639 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.046668+0000", 2020-01-16T00:12:48.639 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 10.501105929, 2020-01-16T00:12:48.639 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014425670999999999, 2020-01-16T00:12:48.639 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:48.640 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:48.640 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:48.640 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:48.640 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:48.640 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 964 2020-01-16T00:12:48.641 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.641 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:48.641 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.641 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:48.641 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046668+0000", 2020-01-16T00:12:48.641 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:48.641 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.641 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.642 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:48.642 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046668+0000", 2020-01-16T00:12:48.642 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:48.642 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.642 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.642 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:48.642 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046666+0000", 2020-01-16T00:12:48.643 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0879999999999999e-05 2020-01-16T00:12:48.643 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.643 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.643 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:48.643 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046677+0000", 2020-01-16T00:12:48.644 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5767e-05 2020-01-16T00:12:48.644 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.644 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.644 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:48.644 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046693+0000", 2020-01-16T00:12:48.644 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1399999999999999e-05 2020-01-16T00:12:48.644 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.645 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.645 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:48.645 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046704+0000", 2020-01-16T00:12:48.645 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00089331800000000004 2020-01-16T00:12:48.645 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.645 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.645 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.646 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.047597+0000", 2020-01-16T00:12:48.646 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8628000000000005e-05 2020-01-16T00:12:48.646 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.646 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.646 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:48.646 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.047696+0000", 2020-01-16T00:12:48.646 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0056536290000000003 2020-01-16T00:12:48.647 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.647 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.647 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.647 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.053349+0000", 2020-01-16T00:12:48.647 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.6946000000000003e-05 2020-01-16T00:12:48.647 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.647 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.647 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:48.648 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.053446+0000", 2020-01-16T00:12:48.648 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0072058909999999999 2020-01-16T00:12:48.648 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.648 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.648 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.648 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.060652+0000", 2020-01-16T00:12:48.648 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000135597 2020-01-16T00:12:48.648 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.649 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.649 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:48.649 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.060788+0000", 2020-01-16T00:12:48.649 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030548499999999998 2020-01-16T00:12:48.649 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.649 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.649 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:48.650 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.061093+0000", 2020-01-16T00:12:48.650 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014425670999999999 2020-01-16T00:12:48.650 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.650 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:48.650 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.650 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.650 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.651 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:981 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:48.651 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:12:48.651 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 10.232441488999999, 2020-01-16T00:12:48.651 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021222494000000001, 2020-01-16T00:12:48.651 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:48.651 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:48.651 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:48.652 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:48.652 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:48.652 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 981 2020-01-16T00:12:48.652 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.652 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:48.652 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.652 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:48.652 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:12:48.653 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:48.653 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.653 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.653 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:48.653 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:12:48.653 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999948 2020-01-16T00:12:48.653 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.653 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.654 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:48.654 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315327+0000", 2020-01-16T00:12:48.654 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.508e-05 2020-01-16T00:12:48.654 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.654 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.654 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:48.654 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315352+0000", 2020-01-16T00:12:48.654 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1183999999999998e-05 2020-01-16T00:12:48.655 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.655 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.655 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:48.655 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315383+0000", 2020-01-16T00:12:48.655 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7180000000000002e-05 2020-01-16T00:12:48.655 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.655 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.656 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:48.656 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315401+0000", 2020-01-16T00:12:48.656 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017290599999999999 2020-01-16T00:12:48.656 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.656 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.657 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.657 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315574+0000", 2020-01-16T00:12:48.657 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00047908100000000002 2020-01-16T00:12:48.657 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.658 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.658 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:48.659 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316053+0000", 2020-01-16T00:12:48.659 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011808947 2020-01-16T00:12:48.659 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.659 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.659 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.659 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.327862+0000", 2020-01-16T00:12:48.659 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010067499999999999 2020-01-16T00:12:48.659 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.660 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.660 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:48.660 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.327962+0000", 2020-01-16T00:12:48.660 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0074745139999999998 2020-01-16T00:12:48.660 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.660 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.660 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.660 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.335437+0000", 2020-01-16T00:12:48.661 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000103403 2020-01-16T00:12:48.661 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.661 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.661 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:48.661 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.335540+0000", 2020-01-16T00:12:48.661 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001014544 2020-01-16T00:12:48.661 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.661 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.662 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:48.662 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336555+0000", 2020-01-16T00:12:48.662 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021222494000000001 2020-01-16T00:12:48.662 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.662 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:48.662 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.662 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.663 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.663 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:982 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:48.663 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:12:48.663 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 10.232338965, 2020-01-16T00:12:48.663 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.022087721000000001, 2020-01-16T00:12:48.663 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:48.663 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:48.663 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:48.664 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:48.664 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:48.664 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 982 2020-01-16T00:12:48.664 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.664 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:48.664 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.664 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:48.664 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:12:48.665 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:48.665 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.665 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.665 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:48.665 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:12:48.665 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:48.666 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.666 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.666 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:48.666 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315433+0000", 2020-01-16T00:12:48.666 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1347e-05 2020-01-16T00:12:48.666 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.666 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.666 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:48.667 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315445+0000", 2020-01-16T00:12:48.667 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4525999999999999e-05 2020-01-16T00:12:48.667 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.667 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.667 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:48.667 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315459+0000", 2020-01-16T00:12:48.667 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.024e-05 2020-01-16T00:12:48.668 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.668 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.668 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:48.668 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315470+0000", 2020-01-16T00:12:48.668 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000736468 2020-01-16T00:12:48.668 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.668 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.668 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.669 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316206+0000", 2020-01-16T00:12:48.669 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000132967 2020-01-16T00:12:48.669 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.669 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.669 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:48.669 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316339+0000", 2020-01-16T00:12:48.669 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011701717 2020-01-16T00:12:48.670 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.670 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.670 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.670 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328041+0000", 2020-01-16T00:12:48.670 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000120339 2020-01-16T00:12:48.670 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.670 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.670 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:48.671 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328161+0000", 2020-01-16T00:12:48.671 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0084341260000000001 2020-01-16T00:12:48.671 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.671 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.671 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.671 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336595+0000", 2020-01-16T00:12:48.671 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5247999999999993e-05 2020-01-16T00:12:48.671 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.672 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.672 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:48.672 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336690+0000", 2020-01-16T00:12:48.672 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00083204000000000004 2020-01-16T00:12:48.672 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.672 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.672 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:48.672 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337522+0000", 2020-01-16T00:12:48.673 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.022087721000000001 2020-01-16T00:12:48.673 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.673 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:48.673 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.673 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.673 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.673 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:983 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:48.674 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:12:48.674 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 10.232219255, 2020-01-16T00:12:48.674 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023363387999999999, 2020-01-16T00:12:48.674 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:48.674 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:48.674 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:48.674 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:48.674 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:48.675 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 983 2020-01-16T00:12:48.675 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.675 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:48.675 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.675 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:48.675 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:12:48.675 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:48.676 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.676 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.676 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:48.676 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:12:48.676 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:48.676 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.677 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.677 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:48.677 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315553+0000", 2020-01-16T00:12:48.677 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1760000000000004e-06 2020-01-16T00:12:48.677 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.677 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.677 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:48.678 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315560+0000", 2020-01-16T00:12:48.678 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3587000000000001e-05 2020-01-16T00:12:48.678 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.678 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.678 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:48.678 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315574+0000", 2020-01-16T00:12:48.678 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7800000000000003e-06 2020-01-16T00:12:48.678 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.679 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.679 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:48.679 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315581+0000", 2020-01-16T00:12:48.679 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000823024 2020-01-16T00:12:48.679 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.679 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.679 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.679 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316404+0000", 2020-01-16T00:12:48.680 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2413e-05 2020-01-16T00:12:48.680 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.680 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.680 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:48.680 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316466+0000", 2020-01-16T00:12:48.680 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011738561999999999 2020-01-16T00:12:48.680 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.681 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.681 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.681 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328205+0000", 2020-01-16T00:12:48.681 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6783999999999997e-05 2020-01-16T00:12:48.681 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.681 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.681 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:48.681 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328262+0000", 2020-01-16T00:12:48.682 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0093022560000000001 2020-01-16T00:12:48.682 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.682 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.682 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.682 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337564+0000", 2020-01-16T00:12:48.682 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000183151 2020-01-16T00:12:48.682 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.682 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.683 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:48.683 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337747+0000", 2020-01-16T00:12:48.683 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011707449999999999 2020-01-16T00:12:48.683 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.683 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.683 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:48.683 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.338918+0000", 2020-01-16T00:12:48.683 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023363387999999999 2020-01-16T00:12:48.684 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.684 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:48.684 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.684 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.684 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.684 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:986 3.9 3:93dfe6f1:::smithi12313934-40:head [write 2941380~419987] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:48.684 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:12:48.685 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 10.066919478999999, 2020-01-16T00:12:48.685 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016978400000000001, 2020-01-16T00:12:48.685 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:48.685 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:48.685 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:48.686 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:48.686 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:48.687 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 986 2020-01-16T00:12:48.687 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.687 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:48.687 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.687 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:48.687 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:12:48.688 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:48.688 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.688 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.688 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:48.688 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:12:48.688 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:48.688 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.689 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.689 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:48.689 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480852+0000", 2020-01-16T00:12:48.689 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042503899999999999 2020-01-16T00:12:48.689 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.689 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.690 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:48.690 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481277+0000", 2020-01-16T00:12:48.690 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3097e-05 2020-01-16T00:12:48.690 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.690 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.690 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:48.690 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481290+0000", 2020-01-16T00:12:48.690 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8460000000000002e-06 2020-01-16T00:12:48.691 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.691 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.691 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:48.691 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481297+0000", 2020-01-16T00:12:48.691 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8443999999999999e-05 2020-01-16T00:12:48.691 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.691 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.692 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.692 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481346+0000", 2020-01-16T00:12:48.692 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022115399999999999 2020-01-16T00:12:48.692 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.692 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.692 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:48.693 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481567+0000", 2020-01-16T00:12:48.693 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012531502 2020-01-16T00:12:48.693 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.693 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.693 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.693 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494098+0000", 2020-01-16T00:12:48.693 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.4462999999999993e-05 2020-01-16T00:12:48.693 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.694 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.694 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:48.694 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494193+0000", 2020-01-16T00:12:48.694 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017600999999999999 2020-01-16T00:12:48.694 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.694 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.694 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:12:48.694 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494369+0000", 2020-01-16T00:12:48.695 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0031739619999999998 2020-01-16T00:12:48.695 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.695 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.695 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:48.695 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497543+0000", 2020-01-16T00:12:48.695 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000175107 2020-01-16T00:12:48.695 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.695 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.696 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:48.696 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497718+0000", 2020-01-16T00:12:48.696 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5207000000000003e-05 2020-01-16T00:12:48.696 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.696 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.696 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:48.696 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497763+0000", 2020-01-16T00:12:48.697 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7042999999999998e-05 2020-01-16T00:12:48.697 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.697 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.697 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:48.697 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497800+0000", 2020-01-16T00:12:48.697 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2628e-05 2020-01-16T00:12:48.697 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.698 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.698 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:48.698 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497833+0000", 2020-01-16T00:12:48.698 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016978400000000001 2020-01-16T00:12:48.698 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.698 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:48.698 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.698 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.699 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.699 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:987 3.9 3:93dfe6f1:::smithi12313934-40:head [setxattr _header (56) in=7b,truncate 3361367] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:48.699 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.481307+0000", 2020-01-16T00:12:48.699 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 10.066467112, 2020-01-16T00:12:48.699 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016794989, 2020-01-16T00:12:48.699 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:48.699 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:48.700 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:48.700 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:48.700 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:48.700 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 987 2020-01-16T00:12:48.700 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.701 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:48.701 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.701 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:48.701 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481307+0000", 2020-01-16T00:12:48.701 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:48.701 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.701 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.702 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:48.702 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481307+0000", 2020-01-16T00:12:48.702 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:48.702 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.702 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.702 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:48.702 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481306+0000", 2020-01-16T00:12:48.702 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.113e-06 2020-01-16T00:12:48.703 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.703 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.703 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:48.703 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481311+0000", 2020-01-16T00:12:48.703 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1520000000000002e-06 2020-01-16T00:12:48.703 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.703 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.703 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:48.704 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481315+0000", 2020-01-16T00:12:48.704 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.7640000000000001e-06 2020-01-16T00:12:48.704 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.704 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.704 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:48.704 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481318+0000", 2020-01-16T00:12:48.704 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00029012500000000002 2020-01-16T00:12:48.704 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.705 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.705 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.705 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481608+0000", 2020-01-16T00:12:48.705 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6239000000000002e-05 2020-01-16T00:12:48.705 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.705 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.705 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:48.705 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481654+0000", 2020-01-16T00:12:48.706 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014559534000000001 2020-01-16T00:12:48.706 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.706 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.706 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.706 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.496214+0000", 2020-01-16T00:12:48.706 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000225932 2020-01-16T00:12:48.706 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.706 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.707 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:48.707 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.496439+0000", 2020-01-16T00:12:48.707 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00031099600000000002 2020-01-16T00:12:48.707 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.707 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.707 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:12:48.707 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.496750+0000", 2020-01-16T00:12:48.708 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00098608599999999995 2020-01-16T00:12:48.708 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.708 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.708 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:48.708 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497737+0000", 2020-01-16T00:12:48.708 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022081499999999999 2020-01-16T00:12:48.708 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.708 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.709 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:48.709 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497957+0000", 2020-01-16T00:12:48.709 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7071000000000003e-05 2020-01-16T00:12:48.709 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.709 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.709 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:48.710 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498014+0000", 2020-01-16T00:12:48.710 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1111e-05 2020-01-16T00:12:48.710 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.710 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.710 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:48.710 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498056+0000", 2020-01-16T00:12:48.711 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5974999999999999e-05 2020-01-16T00:12:48.711 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.711 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.712 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:48.712 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498102+0000", 2020-01-16T00:12:48.712 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016794989 2020-01-16T00:12:48.712 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.712 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:48.712 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.713 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.713 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.713 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:988 3.9 3:93dfe6f1:::smithi12313934-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:12:48.713 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:12:48.713 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 10.066434015, 2020-01-16T00:12:48.713 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017221692, 2020-01-16T00:12:48.713 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:48.713 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:48.714 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:48.714 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:48.714 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:48.714 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 988 2020-01-16T00:12:48.714 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.714 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:48.714 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.714 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:48.715 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:12:48.715 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:48.715 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.715 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.715 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:48.715 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:12:48.715 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:48.716 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.716 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.716 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:48.716 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481339+0000", 2020-01-16T00:12:48.716 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3089999999999997e-06 2020-01-16T00:12:48.716 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.716 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.716 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:48.717 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481343+0000", 2020-01-16T00:12:48.717 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2230000000000004e-06 2020-01-16T00:12:48.717 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.717 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.717 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:48.717 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481350+0000", 2020-01-16T00:12:48.717 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.319e-06 2020-01-16T00:12:48.717 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.718 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.718 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:48.718 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481356+0000", 2020-01-16T00:12:48.718 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000321544 2020-01-16T00:12:48.718 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.718 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.718 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.718 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481678+0000", 2020-01-16T00:12:48.719 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6375999999999999e-05 2020-01-16T00:12:48.719 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.719 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.719 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:48.719 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481714+0000", 2020-01-16T00:12:48.719 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015557666 2020-01-16T00:12:48.719 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.719 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.720 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.720 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497272+0000", 2020-01-16T00:12:48.720 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012791300000000001 2020-01-16T00:12:48.720 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.720 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.720 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:48.720 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497399+0000", 2020-01-16T00:12:48.720 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00076944600000000002 2020-01-16T00:12:48.721 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.721 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.721 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.721 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498169+0000", 2020-01-16T00:12:48.721 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018306799999999999 2020-01-16T00:12:48.721 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.721 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.721 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:48.722 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498352+0000", 2020-01-16T00:12:48.722 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020935999999999999 2020-01-16T00:12:48.722 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.722 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.722 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:48.722 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498561+0000", 2020-01-16T00:12:48.722 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017221692 2020-01-16T00:12:48.722 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.723 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:48.723 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.723 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.723 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.723 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1208 3.3 3:c478bfe8:::smithi12313934-10:head [write 749612~474179,stat] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:12:48.723 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.350952+0000", 2020-01-16T00:12:48.723 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 6.1968211780000004, 2020-01-16T00:12:48.723 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014967729000000001, 2020-01-16T00:12:48.724 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:48.724 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:48.724 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:48.724 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:48.724 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:48.724 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1208 2020-01-16T00:12:48.724 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.724 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:48.725 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.725 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:48.725 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.350952+0000", 2020-01-16T00:12:48.725 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:48.725 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.725 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.725 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:48.726 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.350952+0000", 2020-01-16T00:12:48.726 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:12:48.726 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.726 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.726 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:48.726 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.350949+0000", 2020-01-16T00:12:48.726 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042894 2020-01-16T00:12:48.726 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.727 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.727 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:48.727 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.351377+0000", 2020-01-16T00:12:48.727 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3921e-05 2020-01-16T00:12:48.727 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.727 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.727 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:48.727 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.351401+0000", 2020-01-16T00:12:48.728 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1087000000000001e-05 2020-01-16T00:12:48.728 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.728 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.728 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:48.728 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.351412+0000", 2020-01-16T00:12:48.728 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8127000000000001e-05 2020-01-16T00:12:48.728 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.728 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.729 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.729 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.351471+0000", 2020-01-16T00:12:48.729 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000230268 2020-01-16T00:12:48.729 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.729 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.729 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:48.729 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.351701+0000", 2020-01-16T00:12:48.729 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0051228369999999999 2020-01-16T00:12:48.730 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.730 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.730 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.730 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.356824+0000", 2020-01-16T00:12:48.730 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015984800000000001 2020-01-16T00:12:48.730 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.731 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.731 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:48.731 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.356984+0000", 2020-01-16T00:12:48.731 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0038163770000000001 2020-01-16T00:12:48.731 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.731 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.731 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.732 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.360800+0000", 2020-01-16T00:12:48.732 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.9878999999999999e-05 2020-01-16T00:12:48.732 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.732 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.732 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:48.732 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.360880+0000", 2020-01-16T00:12:48.733 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000145041 2020-01-16T00:12:48.733 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.733 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.733 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2020-01-16T00:12:48.733 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.361025+0000", 2020-01-16T00:12:48.733 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0020743039999999999 2020-01-16T00:12:48.733 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.733 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.734 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:48.734 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.363099+0000", 2020-01-16T00:12:48.734 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00072876500000000003 2020-01-16T00:12:48.734 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.734 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.734 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:48.734 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.363828+0000", 2020-01-16T00:12:48.734 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00189153 2020-01-16T00:12:48.735 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.735 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.735 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:48.735 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.365719+0000", 2020-01-16T00:12:48.735 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3948000000000005e-05 2020-01-16T00:12:48.735 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.735 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.735 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:48.736 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.365783+0000", 2020-01-16T00:12:48.736 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000136833 2020-01-16T00:12:48.736 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.736 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.736 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:48.736 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.365920+0000", 2020-01-16T00:12:48.736 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014967729000000001 2020-01-16T00:12:48.736 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.737 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:48.737 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.737 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.737 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.737 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1237 3.3 3:c204aa4f:::smithi12313934-45:head [write 400033~699926] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:12:48.737 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.584946+0000", 2020-01-16T00:12:48.737 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.9628272960000004, 2020-01-16T00:12:48.738 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014287855, 2020-01-16T00:12:48.738 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:48.738 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:48.738 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:48.738 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:48.738 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:48.738 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1237 2020-01-16T00:12:48.738 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.739 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:48.739 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.739 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:48.739 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.584946+0000", 2020-01-16T00:12:48.739 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:48.739 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.739 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.739 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:48.740 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.584946+0000", 2020-01-16T00:12:48.740 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999752 2020-01-16T00:12:48.740 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.740 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.740 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:48.740 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.584921+0000", 2020-01-16T00:12:48.740 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00077243400000000001 2020-01-16T00:12:48.740 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.741 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.741 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:48.741 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.585694+0000", 2020-01-16T00:12:48.741 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3083000000000001e-05 2020-01-16T00:12:48.741 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.741 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.742 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:48.742 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.585737+0000", 2020-01-16T00:12:48.742 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1854000000000001e-05 2020-01-16T00:12:48.742 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.742 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.743 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:48.743 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.585759+0000", 2020-01-16T00:12:48.743 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016127000000000001 2020-01-16T00:12:48.743 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.743 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.743 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.743 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.585920+0000", 2020-01-16T00:12:48.744 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024934700000000002 2020-01-16T00:12:48.744 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.744 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.744 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:48.746 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.586169+0000", 2020-01-16T00:12:48.746 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00036508999999999999 2020-01-16T00:12:48.746 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.746 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.747 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2020-01-16T00:12:48.747 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.586534+0000", 2020-01-16T00:12:48.747 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012168910999999999 2020-01-16T00:12:48.747 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.747 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.748 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:48.748 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598703+0000", 2020-01-16T00:12:48.748 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5302000000000004e-05 2020-01-16T00:12:48.748 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.748 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.748 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:48.749 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598789+0000", 2020-01-16T00:12:48.749 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00027301099999999998 2020-01-16T00:12:48.749 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.749 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.750 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:48.750 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599062+0000", 2020-01-16T00:12:48.750 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.4237e-05 2020-01-16T00:12:48.750 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.750 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.750 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:48.750 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599146+0000", 2020-01-16T00:12:48.751 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.8319000000000006e-05 2020-01-16T00:12:48.751 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.751 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.751 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:48.751 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599234+0000", 2020-01-16T00:12:48.751 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014287855 2020-01-16T00:12:48.752 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.752 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:48.752 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.752 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.752 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.752 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1239 3.3 3:c204aa4f:::smithi12313934-45:head [setxattr _header (56) in=7b,truncate 2000033] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:12:48.752 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:12:48.752 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.9573601959999998, 2020-01-16T00:12:48.753 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019161661999999999, 2020-01-16T00:12:48.753 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:48.753 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:48.753 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:48.753 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:48.753 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:48.753 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1239 2020-01-16T00:12:48.754 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.754 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:48.754 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.754 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:48.754 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:12:48.754 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:48.754 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.754 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.755 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:48.755 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:12:48.755 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:12:48.755 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.755 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.755 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:48.755 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590411+0000", 2020-01-16T00:12:48.756 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3879999999999999e-05 2020-01-16T00:12:48.756 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.756 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.756 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:48.756 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590425+0000", 2020-01-16T00:12:48.756 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6542999999999998e-05 2020-01-16T00:12:48.756 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.756 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.757 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:48.757 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590471+0000", 2020-01-16T00:12:48.757 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8907999999999999e-05 2020-01-16T00:12:48.757 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.757 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.757 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:48.757 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590490+0000", 2020-01-16T00:12:48.757 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0045691960000000002 2020-01-16T00:12:48.758 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.758 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.758 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.758 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.595059+0000", 2020-01-16T00:12:48.758 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0035638940000000002 2020-01-16T00:12:48.758 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.758 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.759 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:48.759 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598623+0000", 2020-01-16T00:12:48.759 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0077558310000000004 2020-01-16T00:12:48.759 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.759 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.759 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.759 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606379+0000", 2020-01-16T00:12:48.759 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012171999999999999 2020-01-16T00:12:48.760 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.760 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.760 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:48.760 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606501+0000", 2020-01-16T00:12:48.760 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013144200000000001 2020-01-16T00:12:48.760 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.760 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.761 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2020-01-16T00:12:48.761 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606632+0000", 2020-01-16T00:12:48.761 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001288565 2020-01-16T00:12:48.761 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.761 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.761 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:48.761 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607921+0000", 2020-01-16T00:12:48.762 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033399299999999999 2020-01-16T00:12:48.762 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.762 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.762 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:48.762 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.608255+0000", 2020-01-16T00:12:48.762 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012280870000000001 2020-01-16T00:12:48.762 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.763 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.763 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:48.763 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609483+0000", 2020-01-16T00:12:48.763 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2286000000000001e-05 2020-01-16T00:12:48.763 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.763 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.763 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:48.763 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609525+0000", 2020-01-16T00:12:48.764 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0121999999999999e-05 2020-01-16T00:12:48.764 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.764 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.764 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:48.764 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609575+0000", 2020-01-16T00:12:48.764 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019161661999999999 2020-01-16T00:12:48.764 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.765 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:48.765 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.765 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.765 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.765 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1240 3.3 3:c204aa4f:::smithi12313934-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-01-16T00:12:48.765 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:12:48.765 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.9572600070000004, 2020-01-16T00:12:48.766 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019268828000000002, 2020-01-16T00:12:48.766 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:48.766 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:48.766 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:48.766 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:48.766 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:48.767 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1240 2020-01-16T00:12:48.767 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.767 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:48.767 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.767 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:48.767 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:12:48.767 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:48.767 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.768 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.768 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:48.768 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:12:48.768 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:48.768 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.768 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.768 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:48.769 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590512+0000", 2020-01-16T00:12:48.769 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0525e-05 2020-01-16T00:12:48.769 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.769 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.769 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:48.769 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590522+0000", 2020-01-16T00:12:48.769 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6418000000000001e-05 2020-01-16T00:12:48.769 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.770 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.770 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:48.770 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590539+0000", 2020-01-16T00:12:48.770 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0679999999999999e-05 2020-01-16T00:12:48.770 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.770 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.770 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:48.771 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590549+0000", 2020-01-16T00:12:48.771 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0083626789999999996 2020-01-16T00:12:48.771 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.771 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.771 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.771 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598912+0000", 2020-01-16T00:12:48.771 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013344 2020-01-16T00:12:48.772 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.772 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.772 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:48.772 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599045+0000", 2020-01-16T00:12:48.772 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0084002269999999997 2020-01-16T00:12:48.773 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.773 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.773 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.773 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607446+0000", 2020-01-16T00:12:48.774 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000109873 2020-01-16T00:12:48.774 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.774 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.774 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:48.775 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607555+0000", 2020-01-16T00:12:48.775 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0020573309999999999 2020-01-16T00:12:48.775 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.775 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.776 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.776 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609613+0000", 2020-01-16T00:12:48.776 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6351e-05 2020-01-16T00:12:48.776 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.776 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.777 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:48.777 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609699+0000", 2020-01-16T00:12:48.777 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3374999999999994e-05 2020-01-16T00:12:48.777 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.778 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.778 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:48.778 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609782+0000", 2020-01-16T00:12:48.778 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019268828000000002 2020-01-16T00:12:48.779 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.779 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:48.779 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.779 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.779 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.780 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1436 3.3 3:c204aa4f:::smithi12313934-45:head [setxattr _header (56) in=7b,truncate 2386594] snapc 0=[] ondisk+write+known_if_redirected e28)", 2020-01-16T00:12:48.780 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:47.900544+0000", 2020-01-16T00:12:48.780 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.64722932600000005, 2020-01-16T00:12:48.780 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016402521, 2020-01-16T00:12:48.781 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:48.781 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:48.781 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:48.781 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:48.782 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:48.782 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1436 2020-01-16T00:12:48.782 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.782 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:48.782 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.783 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:48.783 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900544+0000", 2020-01-16T00:12:48.783 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:48.783 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.784 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.784 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:48.784 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900544+0000", 2020-01-16T00:12:48.784 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:48.784 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.785 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.785 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:48.785 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900542+0000", 2020-01-16T00:12:48.785 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3862999999999999e-05 2020-01-16T00:12:48.786 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.786 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.786 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:48.786 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900556+0000", 2020-01-16T00:12:48.787 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5204e-05 2020-01-16T00:12:48.787 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.787 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.787 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:48.787 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900571+0000", 2020-01-16T00:12:48.788 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.092e-05 2020-01-16T00:12:48.788 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.788 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.788 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:48.789 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900582+0000", 2020-01-16T00:12:48.789 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002786026 2020-01-16T00:12:48.789 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.789 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.789 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.790 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.903368+0000", 2020-01-16T00:12:48.790 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0027123659999999999 2020-01-16T00:12:48.790 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.790 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.791 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:48.791 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.906081+0000", 2020-01-16T00:12:48.791 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0082707790000000007 2020-01-16T00:12:48.791 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.791 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.792 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.792 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.914352+0000", 2020-01-16T00:12:48.792 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020580300000000001 2020-01-16T00:12:48.792 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.793 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.793 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:48.793 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.914557+0000", 2020-01-16T00:12:48.793 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017729900000000001 2020-01-16T00:12:48.794 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.794 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.794 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2020-01-16T00:12:48.794 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.914735+0000", 2020-01-16T00:12:48.794 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00074632099999999999 2020-01-16T00:12:48.794 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.795 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.795 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:48.795 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.915481+0000", 2020-01-16T00:12:48.795 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012279599999999999 2020-01-16T00:12:48.795 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.795 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.795 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:48.795 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.915604+0000", 2020-01-16T00:12:48.796 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012619339999999999 2020-01-16T00:12:48.796 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.796 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.796 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:48.796 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.916866+0000", 2020-01-16T00:12:48.796 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1162000000000003e-05 2020-01-16T00:12:48.796 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.796 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.797 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:48.797 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.916907+0000", 2020-01-16T00:12:48.797 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9987999999999997e-05 2020-01-16T00:12:48.797 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.797 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.797 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:48.797 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.916947+0000", 2020-01-16T00:12:48.798 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016402521 2020-01-16T00:12:48.798 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.798 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:48.798 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.798 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.798 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.798 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1437 3.3 3:c204aa4f:::smithi12313934-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e28)", 2020-01-16T00:12:48.798 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:47.900594+0000", 2020-01-16T00:12:48.799 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.64717946299999995, 2020-01-16T00:12:48.799 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016745237999999999, 2020-01-16T00:12:48.799 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:48.799 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:48.799 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:48.799 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:48.799 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:48.800 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1437 2020-01-16T00:12:48.800 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.800 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:48.800 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.800 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:48.800 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900594+0000", 2020-01-16T00:12:48.800 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:48.800 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.801 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.801 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:48.801 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900594+0000", 2020-01-16T00:12:48.801 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:48.801 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.801 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.801 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:48.801 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900593+0000", 2020-01-16T00:12:48.802 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5059999999999995e-06 2020-01-16T00:12:48.802 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.802 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.802 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:48.802 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900601+0000", 2020-01-16T00:12:48.802 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5077e-05 2020-01-16T00:12:48.802 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.803 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.803 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:48.803 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900616+0000", 2020-01-16T00:12:48.803 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0183000000000001e-05 2020-01-16T00:12:48.803 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.803 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.804 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:48.804 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900627+0000", 2020-01-16T00:12:48.804 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0063667660000000003 2020-01-16T00:12:48.804 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.804 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.804 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.804 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.906993+0000", 2020-01-16T00:12:48.805 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8988e-05 2020-01-16T00:12:48.805 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.805 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.805 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:48.805 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.907032+0000", 2020-01-16T00:12:48.805 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0080128850000000008 2020-01-16T00:12:48.805 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.805 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.806 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.806 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.915045+0000", 2020-01-16T00:12:48.806 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013200899999999999 2020-01-16T00:12:48.806 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.806 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.806 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:48.806 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.915177+0000", 2020-01-16T00:12:48.808 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0018373090000000001 2020-01-16T00:12:48.809 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.809 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.809 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:48.809 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.917015+0000", 2020-01-16T00:12:48.809 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016701699999999999 2020-01-16T00:12:48.809 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.809 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.809 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:48.810 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.917182+0000", 2020-01-16T00:12:48.810 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015782400000000001 2020-01-16T00:12:48.810 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:48.810 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:48.810 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:48.810 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.917339+0000", 2020-01-16T00:12:48.810 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016745237999999999 2020-01-16T00:12:48.811 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.811 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:48.811 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.811 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:48.811 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:48.811 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:48.812 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-01-16T00:12:48.813 INFO:tasks.rados.rados.0.smithi123.stdout:793: expect (ObjNum 136 snap 0 seq_num 136) 2020-01-16T00:12:48.814 INFO:tasks.rados.rados.0.smithi123.stdout:792: expect (ObjNum 122 snap 0 seq_num 122) 2020-01-16T00:12:48.814 INFO:tasks.rados.rados.0.smithi123.stdout:796: finishing write tid 1 to smithi12313934-20 2020-01-16T00:12:48.814 INFO:tasks.rados.rados.0.smithi123.stdout:791: finishing write tid 2 to smithi12313934-32 2020-01-16T00:12:48.814 INFO:tasks.rados.rados.0.smithi123.stdout:791: finishing write tid 4 to smithi12313934-32 2020-01-16T00:12:48.814 INFO:tasks.rados.rados.0.smithi123.stdout:791: finishing write tid 5 to smithi12313934-32 2020-01-16T00:12:48.814 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 32 v 70 (ObjNum 143 snap 0 seq_num 143) dirty exists 2020-01-16T00:12:48.814 INFO:tasks.rados.rados.0.smithi123.stdout:791: left oid 32 (ObjNum 143 snap 0 seq_num 143) 2020-01-16T00:12:48.815 INFO:tasks.rados.rados.0.smithi123.stdout:794: expect (ObjNum 91 snap 0 seq_num 91) 2020-01-16T00:12:48.815 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:48.815+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:12:48.815 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-01-16T00:12:48.914 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:48.915+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:12:48.922 INFO:tasks.rados.rados.0.smithi123.stdout:796: finishing write tid 2 to smithi12313934-20 2020-01-16T00:12:48.922 INFO:tasks.rados.rados.0.smithi123.stdout:796: finishing write tid 4 to smithi12313934-20 2020-01-16T00:12:48.922 INFO:tasks.rados.rados.0.smithi123.stdout:797: expect (ObjNum 75 snap 0 seq_num 75) 2020-01-16T00:12:48.923 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:48.923 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:12:48.923 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:12:48.923 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:48.933 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-01-16T00:12:48.948 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:12:48.965 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:12:48.983 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:12:49.002 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:12:49.015 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:49.017+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:12:49.026 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:12:49.027 INFO:teuthology.orchestra.run.smithi123.stderr:noscrub is unset 2020-01-16T00:12:49.048 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5: {} 2020-01-16T00:12:49.053 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:49.054 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:12:49.054 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:12:49.054 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:12:49.054 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:49.063 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-01-16T00:12:49.070 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:12:49.093 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:12:49.095 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-01-16T00:12:49.095 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-01-16T00:12:49.095 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-01-16T00:12:49.095 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-01-16T00:12:49.096 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-01-16T00:12:49.096 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-01-16T00:12:49.096 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-01-16T00:12:49.096 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-01-16T00:12:49.101 INFO:tasks.rados.rados.0.smithi123.stdout:796: finishing write tid 5 to smithi12313934-20 2020-01-16T00:12:49.101 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 20 v 113 (ObjNum 144 snap 0 seq_num 144) dirty exists 2020-01-16T00:12:49.101 INFO:tasks.rados.rados.0.smithi123.stdout:796: left oid 20 (ObjNum 144 snap 0 seq_num 144) 2020-01-16T00:12:49.102 INFO:tasks.rados.rados.0.smithi123.stdout:799: finishing write tid 1 to smithi12313934-24 2020-01-16T00:12:49.103 INFO:tasks.rados.rados.0.smithi123.stdout:785: done (15 left) 2020-01-16T00:12:49.103 INFO:tasks.rados.rados.0.smithi123.stdout:786: done (14 left) 2020-01-16T00:12:49.103 INFO:tasks.rados.rados.0.smithi123.stdout:787: done (13 left) 2020-01-16T00:12:49.104 INFO:tasks.rados.rados.0.smithi123.stdout:788: done (12 left) 2020-01-16T00:12:49.104 INFO:tasks.rados.rados.0.smithi123.stdout:789: done (11 left) 2020-01-16T00:12:49.104 INFO:tasks.rados.rados.0.smithi123.stdout:790: done (10 left) 2020-01-16T00:12:49.104 INFO:tasks.rados.rados.0.smithi123.stdout:791: done (9 left) 2020-01-16T00:12:49.104 INFO:tasks.rados.rados.0.smithi123.stdout:792: done (8 left) 2020-01-16T00:12:49.105 INFO:tasks.rados.rados.0.smithi123.stdout:793: done (7 left) 2020-01-16T00:12:49.105 INFO:tasks.rados.rados.0.smithi123.stdout:794: done (6 left) 2020-01-16T00:12:49.105 INFO:tasks.rados.rados.0.smithi123.stdout:795: done (5 left) 2020-01-16T00:12:49.105 INFO:tasks.rados.rados.0.smithi123.stdout:796: done (4 left) 2020-01-16T00:12:49.106 INFO:tasks.rados.rados.0.smithi123.stdout:797: done (3 left) 2020-01-16T00:12:49.106 INFO:tasks.rados.rados.0.smithi123.stdout:798: done (2 left) 2020-01-16T00:12:49.106 INFO:tasks.rados.rados.0.smithi123.stdout:801: write oid 26 current snap is 0 2020-01-16T00:12:49.106 INFO:tasks.rados.rados.0.smithi123.stdout:801: seq_num 147 ranges {470629=472310,1418853=513377} 2020-01-16T00:12:49.108 INFO:tasks.rados.rados.0.smithi123.stdout:801: writing smithi12313934-26 from 470629 to 942939 tid 1 2020-01-16T00:12:49.116 INFO:tasks.rados.rados.0.smithi123.stdout:801: writing smithi12313934-26 from 1418853 to 1932230 tid 2 2020-01-16T00:12:49.116 INFO:tasks.rados.rados.0.smithi123.stdout:802: read oid 22 snap -1 2020-01-16T00:12:49.117 INFO:tasks.rados.rados.0.smithi123.stdout:802: expect (ObjNum 123 snap 0 seq_num 123) 2020-01-16T00:12:49.117 INFO:tasks.rados.rados.0.smithi123.stdout:803: read oid 44 snap -1 2020-01-16T00:12:49.117 INFO:tasks.rados.rados.0.smithi123.stdout:803: expect (ObjNum 43 snap 0 seq_num 43) 2020-01-16T00:12:49.117 INFO:tasks.rados.rados.0.smithi123.stdout:804: write (excl) oid 12 current snap is 0 2020-01-16T00:12:49.118 INFO:tasks.rados.rados.0.smithi123.stdout:804: seq_num 148 ranges {487436=547559,1760982=657035} 2020-01-16T00:12:49.122 INFO:tasks.rados.rados.0.smithi123.stdout:804: writing smithi12313934-12 from 487436 to 1034995 tid 1 2020-01-16T00:12:49.148 INFO:tasks.rados.rados.0.smithi123.stdout:804: writing smithi12313934-12 from 1760982 to 2418017 tid 2 2020-01-16T00:12:49.148 INFO:tasks.rados.rados.0.smithi123.stdout:805: read oid 17 snap -1 2020-01-16T00:12:49.149 INFO:tasks.rados.rados.0.smithi123.stdout:805: expect deleted 2020-01-16T00:12:49.149 INFO:tasks.rados.rados.0.smithi123.stdout:806: write oid 29 current snap is 0 2020-01-16T00:12:49.149 INFO:tasks.rados.rados.0.smithi123.stdout:806: seq_num 149 ranges {504243=622808,1819464=517046} 2020-01-16T00:12:49.150 INFO:tasks.rados.rados.0.smithi123.stdout:806: writing smithi12313934-29 from 504243 to 1127051 tid 1 2020-01-16T00:12:49.150 INFO:tasks.rados.rados.0.smithi123.stdout:806: writing smithi12313934-29 from 1819464 to 2336510 tid 2 2020-01-16T00:12:49.150 INFO:tasks.rados.rados.0.smithi123.stdout:807: read oid 20 snap -1 2020-01-16T00:12:49.150 INFO:tasks.rados.rados.0.smithi123.stdout:807: expect (ObjNum 144 snap 0 seq_num 144) 2020-01-16T00:12:49.152 INFO:tasks.rados.rados.0.smithi123.stdout:808: read oid 30 snap -1 2020-01-16T00:12:49.152 INFO:tasks.rados.rados.0.smithi123.stdout:808: expect (ObjNum 84 snap 0 seq_num 84) 2020-01-16T00:12:49.152 INFO:tasks.rados.rados.0.smithi123.stdout:809: read oid 13 snap -1 2020-01-16T00:12:49.153 INFO:tasks.rados.rados.0.smithi123.stdout:809: expect (ObjNum 124 snap 0 seq_num 124) 2020-01-16T00:12:49.153 INFO:tasks.rados.rados.0.smithi123.stdout:810: write oid 8 current snap is 0 2020-01-16T00:12:49.154 INFO:tasks.rados.rados.0.smithi123.stdout:810: seq_num 150 ranges {521050=698057,1877946=643104} 2020-01-16T00:12:49.155 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:49.153+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:49.155 INFO:tasks.rados.rados.0.smithi123.stdout:810: writing smithi12313934-8 from 521050 to 1219107 tid 1 2020-01-16T00:12:49.163 INFO:tasks.rados.rados.0.smithi123.stdout:810: writing smithi12313934-8 from 1877946 to 2521050 tid 2 2020-01-16T00:12:49.164 INFO:tasks.rados.rados.0.smithi123.stdout:811: read oid 42 snap -1 2020-01-16T00:12:49.164 INFO:tasks.rados.rados.0.smithi123.stdout:811: expect (ObjNum 127 snap 0 seq_num 127) 2020-01-16T00:12:49.164 INFO:tasks.rados.rados.0.smithi123.stdout:812: write oid 43 current snap is 0 2020-01-16T00:12:49.164 INFO:tasks.rados.rados.0.smithi123.stdout:812: seq_num 151 ranges {537857=773306,1936428=520715} 2020-01-16T00:12:49.167 INFO:tasks.rados.rados.0.smithi123.stdout:812: writing smithi12313934-43 from 537857 to 1311163 tid 1 2020-01-16T00:12:49.173 INFO:tasks.rados.rados.0.smithi123.stdout:812: writing smithi12313934-43 from 1936428 to 2457143 tid 2 2020-01-16T00:12:49.174 INFO:tasks.rados.rados.0.smithi123.stdout:813: read oid 5 snap -1 2020-01-16T00:12:49.174 INFO:tasks.rados.rados.0.smithi123.stdout:813: expect (ObjNum 134 snap 0 seq_num 134) 2020-01-16T00:12:49.174 INFO:tasks.rados.rados.0.smithi123.stdout:814: write oid 15 current snap is 0 2020-01-16T00:12:49.174 INFO:tasks.rados.rados.0.smithi123.stdout:814: seq_num 152 ranges {554664=448555,1478557=664373} 2020-01-16T00:12:49.179 INFO:tasks.rados.rados.0.smithi123.stdout:814: writing smithi12313934-15 from 554664 to 1003219 tid 1 2020-01-16T00:12:49.193 INFO:tasks.rados.rados.0.smithi123.stdout:814: writing smithi12313934-15 from 1478557 to 2142930 tid 2 2020-01-16T00:12:49.193 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:49.194 INFO:tasks.rados.rados.0.smithi123.stdout:799: finishing write tid 2 to smithi12313934-24 2020-01-16T00:12:49.194 INFO:tasks.rados.rados.0.smithi123.stdout:800: finishing write tid 1 to smithi12313934-34 2020-01-16T00:12:49.194 INFO:tasks.rados.rados.0.smithi123.stdout:799: finishing write tid 4 to smithi12313934-24 2020-01-16T00:12:49.194 INFO:tasks.rados.rados.0.smithi123.stdout:799: finishing write tid 5 to smithi12313934-24 2020-01-16T00:12:49.194 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 24 v 58 (ObjNum 145 snap 0 seq_num 145) dirty exists 2020-01-16T00:12:49.194 INFO:tasks.rados.rados.0.smithi123.stdout:799: left oid 24 (ObjNum 145 snap 0 seq_num 145) 2020-01-16T00:12:49.194 INFO:tasks.rados.rados.0.smithi123.stdout:800: finishing write tid 2 to smithi12313934-34 2020-01-16T00:12:49.194 INFO:tasks.rados.rados.0.smithi123.stdout:799: done (15 left) 2020-01-16T00:12:49.195 INFO:tasks.rados.rados.0.smithi123.stdout:815: read oid 11 snap -1 2020-01-16T00:12:49.195 INFO:tasks.rados.rados.0.smithi123.stdout:815: expect (ObjNum 10 snap 0 seq_num 10) 2020-01-16T00:12:49.195 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:49.195 INFO:tasks.rados.rados.0.smithi123.stdout:800: finishing write tid 4 to smithi12313934-34 2020-01-16T00:12:49.195 INFO:tasks.rados.rados.0.smithi123.stdout:800: finishing write tid 5 to smithi12313934-34 2020-01-16T00:12:49.195 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 34 v 36 (ObjNum 146 snap 0 seq_num 146) dirty exists 2020-01-16T00:12:49.196 INFO:tasks.rados.rados.0.smithi123.stdout:800: left oid 34 (ObjNum 146 snap 0 seq_num 146) 2020-01-16T00:12:49.196 INFO:tasks.rados.rados.0.smithi123.stdout:800: done (15 left) 2020-01-16T00:12:49.196 INFO:tasks.rados.rados.0.smithi123.stdout:816: read oid 40 snap -1 2020-01-16T00:12:49.196 INFO:tasks.rados.rados.0.smithi123.stdout:816: expect (ObjNum 138 snap 0 seq_num 138) 2020-01-16T00:12:49.197 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:49.197 INFO:tasks.rados.rados.0.smithi123.stdout:801: finishing write tid 1 to smithi12313934-26 2020-01-16T00:12:49.197 INFO:tasks.rados.rados.0.smithi123.stdout:801: finishing write tid 2 to smithi12313934-26 2020-01-16T00:12:49.197 INFO:tasks.rados.rados.0.smithi123.stdout:801: finishing write tid 4 to smithi12313934-26 2020-01-16T00:12:49.197 INFO:tasks.rados.rados.0.smithi123.stdout:801: finishing write tid 5 to smithi12313934-26 2020-01-16T00:12:49.198 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 26 v 147 (ObjNum 147 snap 0 seq_num 147) dirty exists 2020-01-16T00:12:49.198 INFO:tasks.rados.rados.0.smithi123.stdout:801: left oid 26 (ObjNum 147 snap 0 seq_num 147) 2020-01-16T00:12:49.198 INFO:tasks.rados.rados.0.smithi123.stdout:801: done (15 left) 2020-01-16T00:12:49.198 INFO:tasks.rados.rados.0.smithi123.stdout:817: write (excl) oid 45 current snap is 0 2020-01-16T00:12:49.198 INFO:tasks.rados.rados.0.smithi123.stdout:817: seq_num 153 ranges {571471=640157,1653392=524384} 2020-01-16T00:12:49.199 INFO:tasks.rados.rados.0.smithi123.stdout:817: writing smithi12313934-45 from 571471 to 1211628 tid 1 2020-01-16T00:12:49.201 INFO:tasks.rados.rados.0.smithi123.stdout:817: writing smithi12313934-45 from 1653392 to 2177776 tid 2 2020-01-16T00:12:49.201 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:49.201 INFO:tasks.rados.rados.0.smithi123.stdout:802: expect (ObjNum 123 snap 0 seq_num 123) 2020-01-16T00:12:49.216 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:12:49.217+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:12:49.225 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:49.239 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-01-16T00:12:49.323 INFO:tasks.rados.rados.0.smithi123.stdout:804: finishing write tid 1 to smithi12313934-12 2020-01-16T00:12:49.324 INFO:tasks.rados.rados.0.smithi123.stdout:803: expect (ObjNum 43 snap 0 seq_num 43) 2020-01-16T00:12:49.327 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:49.329+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:12:49.394 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:49.404 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-01-16T00:12:49.406 INFO:tasks.rados.rados.0.smithi123.stdout:804: finishing write tid 2 to smithi12313934-12 2020-01-16T00:12:49.406 INFO:tasks.rados.rados.0.smithi123.stdout:804: finishing write tid 4 to smithi12313934-12 2020-01-16T00:12:49.406 INFO:tasks.rados.rados.0.smithi123.stdout:804: finishing write tid 5 to smithi12313934-12 2020-01-16T00:12:49.406 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 12 v 56 (ObjNum 148 snap 0 seq_num 148) dirty exists 2020-01-16T00:12:49.407 INFO:tasks.rados.rados.0.smithi123.stdout:804: left oid 12 (ObjNum 148 snap 0 seq_num 148) 2020-01-16T00:12:49.407 INFO:tasks.rados.rados.0.smithi123.stdout:806: finishing write tid 1 to smithi12313934-29 2020-01-16T00:12:49.407 INFO:tasks.rados.rados.0.smithi123.stdout:802: done (15 left) 2020-01-16T00:12:49.407 INFO:tasks.rados.rados.0.smithi123.stdout:803: done (14 left) 2020-01-16T00:12:49.407 INFO:tasks.rados.rados.0.smithi123.stdout:804: done (13 left) 2020-01-16T00:12:49.407 INFO:tasks.rados.rados.0.smithi123.stdout:805: done (12 left) 2020-01-16T00:12:49.407 INFO:tasks.rados.rados.0.smithi123.stdout:818: read oid 37 snap -1 2020-01-16T00:12:49.408 INFO:tasks.rados.rados.0.smithi123.stdout:818: expect (ObjNum 122 snap 0 seq_num 122) 2020-01-16T00:12:49.408 INFO:tasks.rados.rados.0.smithi123.stdout:819: read oid 12 snap -1 2020-01-16T00:12:49.408 INFO:tasks.rados.rados.0.smithi123.stdout:819: expect (ObjNum 148 snap 0 seq_num 148) 2020-01-16T00:12:49.408 INFO:tasks.rados.rados.0.smithi123.stdout:820: write oid 14 current snap is 0 2020-01-16T00:12:49.408 INFO:tasks.rados.rados.0.smithi123.stdout:820: seq_num 154 ranges {588278=715406,1711874=668042} 2020-01-16T00:12:49.422 INFO:tasks.rados.rados.0.smithi123.stdout:820: writing smithi12313934-14 from 588278 to 1303684 tid 1 2020-01-16T00:12:49.428 INFO:tasks.rados.rados.0.smithi123.stdout:820: writing smithi12313934-14 from 1711874 to 2379916 tid 2 2020-01-16T00:12:49.428 INFO:tasks.rados.rados.0.smithi123.stdout:821: read oid 49 snap -1 2020-01-16T00:12:49.428 INFO:tasks.rados.rados.0.smithi123.stdout:821: expect (ObjNum 48 snap 0 seq_num 48) 2020-01-16T00:12:49.429 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:49.429 INFO:tasks.rados.rados.0.smithi123.stdout:809: expect (ObjNum 124 snap 0 seq_num 124) 2020-01-16T00:12:49.447 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:49.449+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:12:49.520 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:12:49.523+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:12:49.524 INFO:tasks.rados.rados.0.smithi123.stdout:806: finishing write tid 2 to smithi12313934-29 2020-01-16T00:12:49.524 INFO:tasks.rados.rados.0.smithi123.stdout:807: expect (ObjNum 144 snap 0 seq_num 144) 2020-01-16T00:12:49.546 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:49.556 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-01-16T00:12:49.621 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:12:49.623+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:12:49.663 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:49.665 INFO:tasks.rados.rados.0.smithi123.stdout:808: expect (ObjNum 84 snap 0 seq_num 84) 2020-01-16T00:12:49.673 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-01-16T00:12:49.722 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:12:49.723+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:12:49.778 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:49.787 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-01-16T00:12:49.831 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:12:49.831+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:12:49.839 INFO:tasks.rados.rados.0.smithi123.stdout:810: finishing write tid 1 to smithi12313934-8 2020-01-16T00:12:49.839 INFO:tasks.rados.rados.0.smithi123.stdout:806: finishing write tid 4 to smithi12313934-29 2020-01-16T00:12:49.840 INFO:tasks.rados.rados.0.smithi123.stdout:806: finishing write tid 5 to smithi12313934-29 2020-01-16T00:12:49.840 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 29 v 62 (ObjNum 149 snap 0 seq_num 149) dirty exists 2020-01-16T00:12:49.840 INFO:tasks.rados.rados.0.smithi123.stdout:806: left oid 29 (ObjNum 149 snap 0 seq_num 149) 2020-01-16T00:12:49.840 INFO:tasks.rados.rados.0.smithi123.stdout:810: finishing write tid 2 to smithi12313934-8 2020-01-16T00:12:49.840 INFO:tasks.rados.rados.0.smithi123.stdout:806: done (15 left) 2020-01-16T00:12:49.840 INFO:tasks.rados.rados.0.smithi123.stdout:807: done (14 left) 2020-01-16T00:12:49.840 INFO:tasks.rados.rados.0.smithi123.stdout:808: done (13 left) 2020-01-16T00:12:49.840 INFO:tasks.rados.rados.0.smithi123.stdout:809: done (12 left) 2020-01-16T00:12:49.841 INFO:tasks.rados.rados.0.smithi123.stdout:822: read oid 16 snap -1 2020-01-16T00:12:49.841 INFO:tasks.rados.rados.0.smithi123.stdout:822: expect (ObjNum 60 snap 0 seq_num 60) 2020-01-16T00:12:49.841 INFO:tasks.rados.rados.0.smithi123.stdout:823: write oid 29 current snap is 0 2020-01-16T00:12:49.841 INFO:tasks.rados.rados.0.smithi123.stdout:823: seq_num 155 ranges {605085=790655,2170356=434729} 2020-01-16T00:12:49.856 INFO:tasks.rados.rados.0.smithi123.stdout:823: writing smithi12313934-29 from 605085 to 1395740 tid 1 2020-01-16T00:12:49.863 INFO:tasks.rados.rados.0.smithi123.stdout:823: writing smithi12313934-29 from 2170356 to 2605085 tid 2 2020-01-16T00:12:49.863 INFO:tasks.rados.rados.0.smithi123.stdout:824: read oid 27 snap -1 2020-01-16T00:12:49.863 INFO:tasks.rados.rados.0.smithi123.stdout:824: expect (ObjNum 87 snap 0 seq_num 87) 2020-01-16T00:12:49.863 INFO:tasks.rados.rados.0.smithi123.stdout:825: write (excl) oid 3 current snap is 0 2020-01-16T00:12:49.864 INFO:tasks.rados.rados.0.smithi123.stdout:825: seq_num 156 ranges {621892=465904,1712485=671711} 2020-01-16T00:12:49.869 INFO:tasks.rados.rados.0.smithi123.stdout:825: writing smithi12313934-3 from 621892 to 1087796 tid 1 2020-01-16T00:12:49.877 INFO:tasks.rados.rados.0.smithi123.stdout:825: writing smithi12313934-3 from 1712485 to 2384196 tid 2 2020-01-16T00:12:49.877 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:49.877 INFO:tasks.rados.rados.0.smithi123.stdout:810: finishing write tid 4 to smithi12313934-8 2020-01-16T00:12:49.878 INFO:tasks.rados.rados.0.smithi123.stdout:810: finishing write tid 5 to smithi12313934-8 2020-01-16T00:12:49.878 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 8 v 50 (ObjNum 150 snap 0 seq_num 150) dirty exists 2020-01-16T00:12:49.878 INFO:tasks.rados.rados.0.smithi123.stdout:810: left oid 8 (ObjNum 150 snap 0 seq_num 150) 2020-01-16T00:12:49.878 INFO:tasks.rados.rados.0.smithi123.stdout:811: expect (ObjNum 127 snap 0 seq_num 127) 2020-01-16T00:12:49.924 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:12:49.927+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:12:49.947 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:49.956 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-01-16T00:12:49.965 INFO:tasks.rados.rados.0.smithi123.stdout:812: finishing write tid 1 to smithi12313934-43 2020-01-16T00:12:49.966 INFO:tasks.rados.rados.0.smithi123.stdout:812: finishing write tid 2 to smithi12313934-43 2020-01-16T00:12:49.966 INFO:tasks.rados.rados.0.smithi123.stdout:812: finishing write tid 4 to smithi12313934-43 2020-01-16T00:12:49.966 INFO:tasks.rados.rados.0.smithi123.stdout:812: finishing write tid 5 to smithi12313934-43 2020-01-16T00:12:49.966 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 43 v 101 (ObjNum 151 snap 0 seq_num 151) dirty exists 2020-01-16T00:12:49.966 INFO:tasks.rados.rados.0.smithi123.stdout:812: left oid 43 (ObjNum 151 snap 0 seq_num 151) 2020-01-16T00:12:49.966 INFO:tasks.rados.rados.0.smithi123.stdout:810: done (15 left) 2020-01-16T00:12:49.967 INFO:tasks.rados.rados.0.smithi123.stdout:811: done (14 left) 2020-01-16T00:12:49.967 INFO:tasks.rados.rados.0.smithi123.stdout:812: done (13 left) 2020-01-16T00:12:49.967 INFO:tasks.rados.rados.0.smithi123.stdout:826: write (excl) oid 2 current snap is 0 2020-01-16T00:12:49.967 INFO:tasks.rados.rados.0.smithi123.stdout:826: seq_num 157 ranges {638699=541153,1770967=531722} 2020-01-16T00:12:49.981 INFO:tasks.rados.rados.0.smithi123.stdout:826: writing smithi12313934-2 from 638699 to 1179852 tid 1 2020-01-16T00:12:49.989 INFO:tasks.rados.rados.0.smithi123.stdout:826: writing smithi12313934-2 from 1770967 to 2302689 tid 2 2020-01-16T00:12:49.989 INFO:tasks.rados.rados.0.smithi123.stdout:827: write oid 43 current snap is 0 2020-01-16T00:12:49.989 INFO:tasks.rados.rados.0.smithi123.stdout:827: seq_num 158 ranges {655506=616402,1829449=675380} 2020-01-16T00:12:49.997 INFO:tasks.rados.rados.0.smithi123.stdout:827: writing smithi12313934-43 from 655506 to 1271908 tid 1 2020-01-16T00:12:50.006 INFO:tasks.rados.rados.0.smithi123.stdout:827: writing smithi12313934-43 from 1829449 to 2504829 tid 2 2020-01-16T00:12:50.006 INFO:tasks.rados.rados.0.smithi123.stdout:828: write oid 41 current snap is 0 2020-01-16T00:12:50.006 INFO:tasks.rados.rados.0.smithi123.stdout:828: seq_num 159 ranges {672313=691651,1887931=419038} 2020-01-16T00:12:50.015 INFO:tasks.rados.rados.0.smithi123.stdout:828: writing smithi12313934-41 from 672313 to 1363964 tid 1 2020-01-16T00:12:50.021 INFO:tasks.rados.rados.0.smithi123.stdout:828: writing smithi12313934-41 from 1887931 to 2306969 tid 2 2020-01-16T00:12:50.021 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:50.021 INFO:tasks.rados.rados.0.smithi123.stdout:813: expect (ObjNum 134 snap 0 seq_num 134) 2020-01-16T00:12:50.024 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:50.025+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:12:50.079 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:50.088 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-01-16T00:12:50.131 INFO:tasks.rados.rados.0.smithi123.stdout:814: finishing write tid 1 to smithi12313934-15 2020-01-16T00:12:50.131 INFO:tasks.rados.rados.0.smithi123.stdout:814: finishing write tid 2 to smithi12313934-15 2020-01-16T00:12:50.131 INFO:tasks.rados.rados.0.smithi123.stdout:814: finishing write tid 4 to smithi12313934-15 2020-01-16T00:12:50.131 INFO:tasks.rados.rados.0.smithi123.stdout:814: finishing write tid 5 to smithi12313934-15 2020-01-16T00:12:50.132 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 15 v 88 (ObjNum 152 snap 0 seq_num 152) dirty exists 2020-01-16T00:12:50.132 INFO:tasks.rados.rados.0.smithi123.stdout:814: left oid 15 (ObjNum 152 snap 0 seq_num 152) 2020-01-16T00:12:50.132 INFO:tasks.rados.rados.0.smithi123.stdout:815: expect (ObjNum 10 snap 0 seq_num 10) 2020-01-16T00:12:50.132 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:12:50.131+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:12:50.194 INFO:tasks.rados.rados.0.smithi123.stdout:816: expect (ObjNum 138 snap 0 seq_num 138) 2020-01-16T00:12:50.234 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:50.233+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:50.237 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:50.246 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-01-16T00:12:50.312 INFO:tasks.rados.rados.0.smithi123.stdout:817: finishing write tid 1 to smithi12313934-45 2020-01-16T00:12:50.312 INFO:tasks.rados.rados.0.smithi123.stdout:817: finishing write tid 2 to smithi12313934-45 2020-01-16T00:12:50.312 INFO:tasks.rados.rados.0.smithi123.stdout:817: finishing write tid 4 to smithi12313934-45 2020-01-16T00:12:50.312 INFO:tasks.rados.rados.0.smithi123.stdout:817: finishing write tid 5 to smithi12313934-45 2020-01-16T00:12:50.312 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 45 v 71 (ObjNum 153 snap 0 seq_num 153) dirty exists 2020-01-16T00:12:50.312 INFO:tasks.rados.rados.0.smithi123.stdout:817: left oid 45 (ObjNum 153 snap 0 seq_num 153) 2020-01-16T00:12:50.312 INFO:tasks.rados.rados.0.smithi123.stdout:819: expect (ObjNum 148 snap 0 seq_num 148) 2020-01-16T00:12:50.333 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:12:50.335+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:12:50.347 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:50.356 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-01-16T00:12:50.444 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:50.447+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:12:50.456 INFO:tasks.rados.rados.0.smithi123.stdout:818: expect (ObjNum 122 snap 0 seq_num 122) 2020-01-16T00:12:50.499 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:50.514 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-01-16T00:12:50.556 INFO:tasks.rados.rados.0.smithi123.stdout:813: done (15 left) 2020-01-16T00:12:50.556 INFO:tasks.rados.rados.0.smithi123.stdout:814: done (14 left) 2020-01-16T00:12:50.556 INFO:tasks.rados.rados.0.smithi123.stdout:815: done (13 left) 2020-01-16T00:12:50.556 INFO:tasks.rados.rados.0.smithi123.stdout:816: done (12 left) 2020-01-16T00:12:50.556 INFO:tasks.rados.rados.0.smithi123.stdout:817: done (11 left) 2020-01-16T00:12:50.557 INFO:tasks.rados.rados.0.smithi123.stdout:818: done (10 left) 2020-01-16T00:12:50.557 INFO:tasks.rados.rados.0.smithi123.stdout:819: done (9 left) 2020-01-16T00:12:50.557 INFO:tasks.rados.rados.0.smithi123.stdout:829: read oid 26 snap -1 2020-01-16T00:12:50.557 INFO:tasks.rados.rados.0.smithi123.stdout:829: expect (ObjNum 147 snap 0 seq_num 147) 2020-01-16T00:12:50.557 INFO:tasks.rados.rados.0.smithi123.stdout:830: read oid 6 snap -1 2020-01-16T00:12:50.557 INFO:tasks.rados.rados.0.smithi123.stdout:830: expect (ObjNum 137 snap 0 seq_num 137) 2020-01-16T00:12:50.557 INFO:tasks.rados.rados.0.smithi123.stdout:831: read oid 32 snap -1 2020-01-16T00:12:50.563 INFO:tasks.rados.rados.0.smithi123.stdout:831: expect (ObjNum 143 snap 0 seq_num 143) 2020-01-16T00:12:50.564 INFO:tasks.rados.rados.0.smithi123.stdout:832: read oid 44 snap -1 2020-01-16T00:12:50.564 INFO:tasks.rados.rados.0.smithi123.stdout:832: expect (ObjNum 43 snap 0 seq_num 43) 2020-01-16T00:12:50.564 INFO:tasks.rados.rados.0.smithi123.stdout:833: read oid 39 snap -1 2020-01-16T00:12:50.564 INFO:tasks.rados.rados.0.smithi123.stdout:833: expect (ObjNum 90 snap 0 seq_num 90) 2020-01-16T00:12:50.564 INFO:tasks.rados.rados.0.smithi123.stdout:834: write (excl) oid 42 current snap is 0 2020-01-16T00:12:50.568 INFO:tasks.rados.rados.0.smithi123.stdout:834: seq_num 160 ranges {689120=483253,1946413=679049} 2020-01-16T00:12:50.569 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:12:50.555+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:12:50.570 INFO:tasks.rados.rados.0.smithi123.stdout:834: writing smithi12313934-42 from 689120 to 1172373 tid 1 2020-01-16T00:12:50.586 INFO:tasks.rados.rados.0.smithi123.stdout:834: writing smithi12313934-42 from 1946413 to 2625462 tid 2 2020-01-16T00:12:50.586 INFO:tasks.rados.rados.0.smithi123.stdout:835: write (excl) oid 36 current snap is 0 2020-01-16T00:12:50.586 INFO:tasks.rados.rados.0.smithi123.stdout:835: seq_num 161 ranges {705927=558502,2004895=422707} 2020-01-16T00:12:50.587 INFO:tasks.rados.rados.0.smithi123.stdout:835: writing smithi12313934-36 from 705927 to 1264429 tid 1 2020-01-16T00:12:50.595 INFO:tasks.rados.rados.0.smithi123.stdout:835: writing smithi12313934-36 from 2004895 to 2427602 tid 2 2020-01-16T00:12:50.595 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:50.595 INFO:tasks.rados.rados.0.smithi123.stdout:820: finishing write tid 1 to smithi12313934-14 2020-01-16T00:12:50.595 INFO:tasks.rados.rados.0.smithi123.stdout:821: expect (ObjNum 48 snap 0 seq_num 48) 2020-01-16T00:12:50.635 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:50.633+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:50.670 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:50.678 INFO:tasks.rados.rados.0.smithi123.stdout:820: finishing write tid 2 to smithi12313934-14 2020-01-16T00:12:50.678 INFO:tasks.rados.rados.0.smithi123.stdout:820: finishing write tid 4 to smithi12313934-14 2020-01-16T00:12:50.679 INFO:tasks.rados.rados.0.smithi123.stdout:821: done (15 left) 2020-01-16T00:12:50.679 INFO:tasks.rados.rados.0.smithi123.stdout:836: write oid 7 current snap is 0 2020-01-16T00:12:50.679 INFO:tasks.rados.rados.0.smithi123.stdout:836: seq_num 162 ranges {722734=633751,2063377=659357} 2020-01-16T00:12:50.679 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-01-16T00:12:50.685 INFO:tasks.rados.rados.0.smithi123.stdout:836: writing smithi12313934-7 from 722734 to 1356485 tid 1 2020-01-16T00:12:50.694 INFO:tasks.rados.rados.0.smithi123.stdout:836: writing smithi12313934-7 from 2063377 to 2722734 tid 2 2020-01-16T00:12:50.694 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:50.695 INFO:tasks.rados.rados.0.smithi123.stdout:820: finishing write tid 5 to smithi12313934-14 2020-01-16T00:12:50.695 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 14 v 104 (ObjNum 154 snap 0 seq_num 154) dirty exists 2020-01-16T00:12:50.695 INFO:tasks.rados.rados.0.smithi123.stdout:820: left oid 14 (ObjNum 154 snap 0 seq_num 154) 2020-01-16T00:12:50.695 INFO:tasks.rados.rados.0.smithi123.stdout:822: expect (ObjNum 60 snap 0 seq_num 60) 2020-01-16T00:12:50.736 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:12:50.737+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:12:50.780 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:50.788 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-01-16T00:12:50.829 INFO:tasks.rados.rados.0.smithi123.stdout:823: finishing write tid 1 to smithi12313934-29 2020-01-16T00:12:50.829 INFO:tasks.rados.rados.0.smithi123.stdout:823: finishing write tid 2 to smithi12313934-29 2020-01-16T00:12:50.829 INFO:tasks.rados.rados.0.smithi123.stdout:823: finishing write tid 4 to smithi12313934-29 2020-01-16T00:12:50.829 INFO:tasks.rados.rados.0.smithi123.stdout:823: finishing write tid 5 to smithi12313934-29 2020-01-16T00:12:50.829 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 29 v 65 (ObjNum 155 snap 0 seq_num 155) dirty exists 2020-01-16T00:12:50.829 INFO:tasks.rados.rados.0.smithi123.stdout:823: left oid 29 (ObjNum 155 snap 0 seq_num 155) 2020-01-16T00:12:50.829 INFO:tasks.rados.rados.0.smithi123.stdout:825: finishing write tid 1 to smithi12313934-3 2020-01-16T00:12:50.830 INFO:tasks.rados.rados.0.smithi123.stdout:820: done (15 left) 2020-01-16T00:12:50.830 INFO:tasks.rados.rados.0.smithi123.stdout:822: done (14 left) 2020-01-16T00:12:50.830 INFO:tasks.rados.rados.0.smithi123.stdout:823: done (13 left) 2020-01-16T00:12:50.830 INFO:tasks.rados.rados.0.smithi123.stdout:837: read oid 25 snap -1 2020-01-16T00:12:50.830 INFO:tasks.rados.rados.0.smithi123.stdout:837: expect deleted 2020-01-16T00:12:50.830 INFO:tasks.rados.rados.0.smithi123.stdout:838: write oid 48 current snap is 0 2020-01-16T00:12:50.830 INFO:tasks.rados.rados.0.smithi123.stdout:838: seq_num 163 ranges {739541=709000,2121859=426376} 2020-01-16T00:12:50.837 INFO:tasks.rados.rados.0.smithi123.stdout:838: writing smithi12313934-48 from 739541 to 1448541 tid 1 2020-01-16T00:12:50.842 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:50.839+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:12:50.845 INFO:tasks.rados.rados.0.smithi123.stdout:838: writing smithi12313934-48 from 2121859 to 2548235 tid 2 2020-01-16T00:12:50.846 INFO:tasks.rados.rados.0.smithi123.stdout:839: read oid 4 snap -1 2020-01-16T00:12:50.846 INFO:tasks.rados.rados.0.smithi123.stdout:839: expect deleted 2020-01-16T00:12:50.846 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:50.846 INFO:tasks.rados.rados.0.smithi123.stdout:825: finishing write tid 2 to smithi12313934-3 2020-01-16T00:12:50.847 INFO:tasks.rados.rados.0.smithi123.stdout:825: finishing write tid 4 to smithi12313934-3 2020-01-16T00:12:50.847 INFO:tasks.rados.rados.0.smithi123.stdout:825: finishing write tid 5 to smithi12313934-3 2020-01-16T00:12:50.847 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 3 v 53 (ObjNum 156 snap 0 seq_num 156) dirty exists 2020-01-16T00:12:50.848 INFO:tasks.rados.rados.0.smithi123.stdout:825: left oid 3 (ObjNum 156 snap 0 seq_num 156) 2020-01-16T00:12:50.848 INFO:tasks.rados.rados.0.smithi123.stdout:824: expect (ObjNum 87 snap 0 seq_num 87) 2020-01-16T00:12:50.935 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:50.939 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:50.937+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:12:50.943 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-01-16T00:12:50.996 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4, 7, 6] 2020-01-16T00:12:50.996 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2020-01-16T00:12:50.997 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2020-01-16T00:12:51.037 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2020-01-16T00:12:51.039 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:12:51.041+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:12:51.065 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:51.073 INFO:tasks.rados.rados.0.smithi123.stdout:826: finishing write tid 1 to smithi12313934-2 2020-01-16T00:12:51.073 INFO:tasks.rados.rados.0.smithi123.stdout:827: finishing write tid 1 to smithi12313934-43 2020-01-16T00:12:51.073 INFO:tasks.rados.rados.0.smithi123.stdout:826: finishing write tid 2 to smithi12313934-2 2020-01-16T00:12:51.073 INFO:tasks.rados.rados.0.smithi123.stdout:826: finishing write tid 4 to smithi12313934-2 2020-01-16T00:12:51.073 INFO:tasks.rados.rados.0.smithi123.stdout:826: finishing write tid 5 to smithi12313934-2 2020-01-16T00:12:51.074 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 2 v 98 (ObjNum 157 snap 0 seq_num 157) dirty exists 2020-01-16T00:12:51.074 INFO:tasks.rados.rados.0.smithi123.stdout:826: left oid 2 (ObjNum 157 snap 0 seq_num 157) 2020-01-16T00:12:51.074 INFO:tasks.rados.rados.0.smithi123.stdout:827: finishing write tid 2 to smithi12313934-43 2020-01-16T00:12:51.074 INFO:tasks.rados.rados.0.smithi123.stdout:827: finishing write tid 4 to smithi12313934-43 2020-01-16T00:12:51.074 INFO:tasks.rados.rados.0.smithi123.stdout:827: finishing write tid 5 to smithi12313934-43 2020-01-16T00:12:51.074 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 43 v 107 (ObjNum 158 snap 0 seq_num 158) dirty exists 2020-01-16T00:12:51.074 INFO:tasks.rados.rados.0.smithi123.stdout:827: left oid 43 (ObjNum 158 snap 0 seq_num 158) 2020-01-16T00:12:51.074 INFO:tasks.rados.rados.0.smithi123.stdout:828: finishing write tid 1 to smithi12313934-41 2020-01-16T00:12:51.075 INFO:tasks.rados.rados.0.smithi123.stdout:828: finishing write tid 2 to smithi12313934-41 2020-01-16T00:12:51.075 INFO:tasks.rados.rados.0.smithi123.stdout:828: finishing write tid 4 to smithi12313934-41 2020-01-16T00:12:51.075 INFO:tasks.rados.rados.0.smithi123.stdout:828: finishing write tid 5 to smithi12313934-41 2020-01-16T00:12:51.075 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 41 v 102 (ObjNum 159 snap 0 seq_num 159) dirty exists 2020-01-16T00:12:51.075 INFO:tasks.rados.rados.0.smithi123.stdout:828: left oid 41 (ObjNum 159 snap 0 seq_num 159) 2020-01-16T00:12:51.075 INFO:tasks.rados.rados.0.smithi123.stdout:824: done (15 left) 2020-01-16T00:12:51.075 INFO:tasks.rados.rados.0.smithi123.stdout:825: done (14 left) 2020-01-16T00:12:51.075 INFO:tasks.rados.rados.0.smithi123.stdout:826: done (13 left) 2020-01-16T00:12:51.076 INFO:tasks.rados.rados.0.smithi123.stdout:827: done (12 left) 2020-01-16T00:12:51.076 INFO:tasks.rados.rados.0.smithi123.stdout:828: done (11 left) 2020-01-16T00:12:51.076 INFO:tasks.rados.rados.0.smithi123.stdout:840: read oid 11 snap -1 2020-01-16T00:12:51.076 INFO:tasks.rados.rados.0.smithi123.stdout:840: expect (ObjNum 10 snap 0 seq_num 10) 2020-01-16T00:12:51.076 INFO:tasks.rados.rados.0.smithi123.stdout:841: write oid 30 current snap is 0 2020-01-16T00:12:51.076 INFO:tasks.rados.rados.0.smithi123.stdout:841: seq_num 164 ranges {756348=784249,2063988=686387} 2020-01-16T00:12:51.078 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-01-16T00:12:51.081 INFO:tasks.rados.rados.0.smithi123.stdout:841: writing smithi12313934-30 from 756348 to 1540597 tid 1 2020-01-16T00:12:51.089 INFO:tasks.rados.rados.0.smithi123.stdout:841: writing smithi12313934-30 from 2063988 to 2750375 tid 2 2020-01-16T00:12:51.089 INFO:tasks.rados.rados.0.smithi123.stdout:842: write (excl) oid 9 current snap is 0 2020-01-16T00:12:51.090 INFO:tasks.rados.rados.0.smithi123.stdout:842: seq_num 165 ranges {773155=459498,1722470=430045,2648676=124479} 2020-01-16T00:12:51.095 INFO:tasks.rados.rados.0.smithi123.stdout:842: writing smithi12313934-9 from 773155 to 1232653 tid 1 2020-01-16T00:12:51.101 INFO:tasks.rados.rados.0.smithi123.stdout:842: writing smithi12313934-9 from 1722470 to 2152515 tid 2 2020-01-16T00:12:51.105 INFO:tasks.rados.rados.0.smithi123.stdout:842: writing smithi12313934-9 from 2648676 to 2773155 tid 3 2020-01-16T00:12:51.105 INFO:tasks.rados.rados.0.smithi123.stdout:843: write oid 1 current snap is 0 2020-01-16T00:12:51.105 INFO:tasks.rados.rados.0.smithi123.stdout:843: seq_num 166 ranges {789962=534747,1780952=690056} 2020-01-16T00:12:51.109 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-01-16T00:12:51.113 INFO:tasks.rados.rados.0.smithi123.stdout:843: writing smithi12313934-1 from 789962 to 1324709 tid 1 2020-01-16T00:12:51.120 INFO:tasks.rados.rados.0.smithi123.stdout:843: writing smithi12313934-1 from 1780952 to 2471008 tid 2 2020-01-16T00:12:51.121 INFO:tasks.rados.rados.0.smithi123.stdout:844: write oid 49 current snap is 0 2020-01-16T00:12:51.121 INFO:tasks.rados.rados.0.smithi123.stdout:844: seq_num 167 ranges {406769=609996,1439434=433714,2303522=503247} 2020-01-16T00:12:51.133 INFO:tasks.rados.rados.0.smithi123.stdout:844: writing smithi12313934-49 from 406769 to 1016765 tid 1 2020-01-16T00:12:51.133 INFO:tasks.rados.rados.0.smithi123.stdout:844: writing smithi12313934-49 from 1439434 to 1873148 tid 2 2020-01-16T00:12:51.137 INFO:tasks.rados.rados.0.smithi123.stdout:844: writing smithi12313934-49 from 2303522 to 2806769 tid 3 2020-01-16T00:12:51.138 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:51.138 INFO:tasks.rados.rados.0.smithi123.stdout:830: expect (ObjNum 137 snap 0 seq_num 137) 2020-01-16T00:12:51.140 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:12:51.141+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:12:51.210 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:51.222 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-01-16T00:12:51.241 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:12:51.241+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:12:51.251 INFO:tasks.rados.rados.0.smithi123.stdout:829: expect (ObjNum 147 snap 0 seq_num 147) 2020-01-16T00:12:51.276 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:12:51.297 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:12:51.316 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:12:51.325 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:12:51.325 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:51.326 INFO:teuthology.orchestra.run.smithi123.stdout: "epoch": 29, 2020-01-16T00:12:51.326 INFO:teuthology.orchestra.run.smithi123.stdout: "fsid": "864e7191-e3db-4201-b88a-a4873d1eb7fe", 2020-01-16T00:12:51.326 INFO:teuthology.orchestra.run.smithi123.stdout: "created": "2020-01-16T00:12:06.009499+0000", 2020-01-16T00:12:51.326 INFO:teuthology.orchestra.run.smithi123.stdout: "modified": "2020-01-16T00:12:49.011884+0000", 2020-01-16T00:12:51.326 INFO:teuthology.orchestra.run.smithi123.stdout: "last_up_change": "2020-01-16T00:12:13.968282+0000", 2020-01-16T00:12:51.326 INFO:teuthology.orchestra.run.smithi123.stdout: "last_in_change": "2020-01-16T00:12:13.968282+0000", 2020-01-16T00:12:51.326 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-01-16T00:12:51.326 INFO:teuthology.orchestra.run.smithi123.stdout: "flags_num": 5804032, 2020-01-16T00:12:51.327 INFO:teuthology.orchestra.run.smithi123.stdout: "flags_set": [ 2020-01-16T00:12:51.327 INFO:teuthology.orchestra.run.smithi123.stdout: "nodeep-scrub", 2020-01-16T00:12:51.327 INFO:teuthology.orchestra.run.smithi123.stdout: "pglog_hardlimit", 2020-01-16T00:12:51.327 INFO:teuthology.orchestra.run.smithi123.stdout: "purged_snapdirs", 2020-01-16T00:12:51.327 INFO:teuthology.orchestra.run.smithi123.stdout: "recovery_deletes", 2020-01-16T00:12:51.327 INFO:teuthology.orchestra.run.smithi123.stdout: "sortbitwise" 2020-01-16T00:12:51.327 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2020-01-16T00:12:51.327 INFO:teuthology.orchestra.run.smithi123.stdout: "crush_version": 5, 2020-01-16T00:12:51.328 INFO:teuthology.orchestra.run.smithi123.stdout: "full_ratio": 0.94999998807907104, 2020-01-16T00:12:51.328 INFO:teuthology.orchestra.run.smithi123.stdout: "backfillfull_ratio": 0.89999997615814209, 2020-01-16T00:12:51.328 INFO:teuthology.orchestra.run.smithi123.stdout: "nearfull_ratio": 0.85000002384185791, 2020-01-16T00:12:51.328 INFO:teuthology.orchestra.run.smithi123.stdout: "cluster_snapshot": "", 2020-01-16T00:12:51.328 INFO:teuthology.orchestra.run.smithi123.stdout: "pool_max": 3, 2020-01-16T00:12:51.328 INFO:teuthology.orchestra.run.smithi123.stdout: "max_osd": 8, 2020-01-16T00:12:51.328 INFO:teuthology.orchestra.run.smithi123.stdout: "require_min_compat_client": "jewel", 2020-01-16T00:12:51.329 INFO:teuthology.orchestra.run.smithi123.stdout: "min_compat_client": "jewel", 2020-01-16T00:12:51.329 INFO:teuthology.orchestra.run.smithi123.stdout: "require_osd_release": "octopus", 2020-01-16T00:12:51.329 INFO:teuthology.orchestra.run.smithi123.stdout: "pools": [ 2020-01-16T00:12:51.329 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.329 INFO:teuthology.orchestra.run.smithi123.stdout: "pool": 1, 2020-01-16T00:12:51.329 INFO:teuthology.orchestra.run.smithi123.stdout: "pool_name": "rbd", 2020-01-16T00:12:51.329 INFO:teuthology.orchestra.run.smithi123.stdout: "create_time": "2020-01-16T00:12:18.200931+0000", 2020-01-16T00:12:51.330 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2020-01-16T00:12:51.330 INFO:teuthology.orchestra.run.smithi123.stdout: "flags_names": "hashpspool", 2020-01-16T00:12:51.330 INFO:teuthology.orchestra.run.smithi123.stdout: "type": 1, 2020-01-16T00:12:51.330 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 3, 2020-01-16T00:12:51.331 INFO:teuthology.orchestra.run.smithi123.stdout: "min_size": 2, 2020-01-16T00:12:51.331 INFO:teuthology.orchestra.run.smithi123.stdout: "crush_rule": 0, 2020-01-16T00:12:51.331 INFO:teuthology.orchestra.run.smithi123.stdout: "object_hash": 2, 2020-01-16T00:12:51.331 INFO:teuthology.orchestra.run.smithi123.stdout: "pg_autoscale_mode": "off", 2020-01-16T00:12:51.331 INFO:teuthology.orchestra.run.smithi123.stdout: "pg_num": 8, 2020-01-16T00:12:51.332 INFO:teuthology.orchestra.run.smithi123.stdout: "pg_placement_num": 8, 2020-01-16T00:12:51.332 INFO:teuthology.orchestra.run.smithi123.stdout: "pg_placement_num_target": 8, 2020-01-16T00:12:51.332 INFO:teuthology.orchestra.run.smithi123.stdout: "pg_num_target": 8, 2020-01-16T00:12:51.333 INFO:teuthology.orchestra.run.smithi123.stdout: "pg_num_pending": 8, 2020-01-16T00:12:51.333 INFO:teuthology.orchestra.run.smithi123.stdout: "last_pg_merge_meta": { 2020-01-16T00:12:51.333 INFO:teuthology.orchestra.run.smithi123.stdout: "source_pgid": "0.0", 2020-01-16T00:12:51.333 INFO:teuthology.orchestra.run.smithi123.stdout: "ready_epoch": 0, 2020-01-16T00:12:51.333 INFO:teuthology.orchestra.run.smithi123.stdout: "last_epoch_started": 0, 2020-01-16T00:12:51.333 INFO:teuthology.orchestra.run.smithi123.stdout: "last_epoch_clean": 0, 2020-01-16T00:12:51.333 INFO:teuthology.orchestra.run.smithi123.stdout: "source_version": "0'0", 2020-01-16T00:12:51.334 INFO:teuthology.orchestra.run.smithi123.stdout: "target_version": "0'0" 2020-01-16T00:12:51.334 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.334 INFO:teuthology.orchestra.run.smithi123.stdout: "last_change": "17", 2020-01-16T00:12:51.334 INFO:teuthology.orchestra.run.smithi123.stdout: "last_force_op_resend": "0", 2020-01-16T00:12:51.334 INFO:teuthology.orchestra.run.smithi123.stdout: "last_force_op_resend_prenautilus": "0", 2020-01-16T00:12:51.334 INFO:teuthology.orchestra.run.smithi123.stdout: "last_force_op_resend_preluminous": "0", 2020-01-16T00:12:51.334 INFO:teuthology.orchestra.run.smithi123.stdout: "auid": 0, 2020-01-16T00:12:51.334 INFO:teuthology.orchestra.run.smithi123.stdout: "snap_mode": "selfmanaged", 2020-01-16T00:12:51.335 INFO:teuthology.orchestra.run.smithi123.stdout: "snap_seq": 0, 2020-01-16T00:12:51.335 INFO:teuthology.orchestra.run.smithi123.stdout: "snap_epoch": 0, 2020-01-16T00:12:51.335 INFO:teuthology.orchestra.run.smithi123.stdout: "pool_snaps": [], 2020-01-16T00:12:51.335 INFO:teuthology.orchestra.run.smithi123.stdout: "removed_snaps": "[]", 2020-01-16T00:12:51.335 INFO:teuthology.orchestra.run.smithi123.stdout: "quota_max_bytes": 0, 2020-01-16T00:12:51.335 INFO:teuthology.orchestra.run.smithi123.stdout: "quota_max_objects": 0, 2020-01-16T00:12:51.335 INFO:teuthology.orchestra.run.smithi123.stdout: "tiers": [], 2020-01-16T00:12:51.336 INFO:teuthology.orchestra.run.smithi123.stdout: "tier_of": -1, 2020-01-16T00:12:51.336 INFO:teuthology.orchestra.run.smithi123.stdout: "read_tier": -1, 2020-01-16T00:12:51.336 INFO:teuthology.orchestra.run.smithi123.stdout: "write_tier": -1, 2020-01-16T00:12:51.336 INFO:teuthology.orchestra.run.smithi123.stdout: "cache_mode": "none", 2020-01-16T00:12:51.337 INFO:teuthology.orchestra.run.smithi123.stdout: "target_max_bytes": 0, 2020-01-16T00:12:51.337 INFO:teuthology.orchestra.run.smithi123.stdout: "target_max_objects": 0, 2020-01-16T00:12:51.337 INFO:teuthology.orchestra.run.smithi123.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-01-16T00:12:51.337 INFO:teuthology.orchestra.run.smithi123.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-01-16T00:12:51.337 INFO:teuthology.orchestra.run.smithi123.stdout: "cache_target_full_ratio_micro": 800000, 2020-01-16T00:12:51.337 INFO:teuthology.orchestra.run.smithi123.stdout: "cache_min_flush_age": 0, 2020-01-16T00:12:51.338 INFO:teuthology.orchestra.run.smithi123.stdout: "cache_min_evict_age": 0, 2020-01-16T00:12:51.338 INFO:teuthology.orchestra.run.smithi123.stdout: "erasure_code_profile": "", 2020-01-16T00:12:51.338 INFO:teuthology.orchestra.run.smithi123.stdout: "hit_set_params": { 2020-01-16T00:12:51.338 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "none" 2020-01-16T00:12:51.338 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.338 INFO:teuthology.orchestra.run.smithi123.stdout: "hit_set_period": 0, 2020-01-16T00:12:51.338 INFO:teuthology.orchestra.run.smithi123.stdout: "hit_set_count": 0, 2020-01-16T00:12:51.339 INFO:teuthology.orchestra.run.smithi123.stdout: "use_gmt_hitset": true, 2020-01-16T00:12:51.339 INFO:teuthology.orchestra.run.smithi123.stdout: "min_read_recency_for_promote": 0, 2020-01-16T00:12:51.339 INFO:teuthology.orchestra.run.smithi123.stdout: "min_write_recency_for_promote": 0, 2020-01-16T00:12:51.339 INFO:teuthology.orchestra.run.smithi123.stdout: "hit_set_grade_decay_rate": 0, 2020-01-16T00:12:51.339 INFO:teuthology.orchestra.run.smithi123.stdout: "hit_set_search_last_n": 0, 2020-01-16T00:12:51.339 INFO:teuthology.orchestra.run.smithi123.stdout: "grade_table": [], 2020-01-16T00:12:51.339 INFO:teuthology.orchestra.run.smithi123.stdout: "stripe_width": 0, 2020-01-16T00:12:51.339 INFO:teuthology.orchestra.run.smithi123.stdout: "expected_num_objects": 0, 2020-01-16T00:12:51.340 INFO:teuthology.orchestra.run.smithi123.stdout: "fast_read": false, 2020-01-16T00:12:51.340 INFO:teuthology.orchestra.run.smithi123.stdout: "options": {}, 2020-01-16T00:12:51.340 INFO:teuthology.orchestra.run.smithi123.stdout: "application_metadata": { 2020-01-16T00:12:51.340 INFO:teuthology.orchestra.run.smithi123.stdout: "rbd": {} 2020-01-16T00:12:51.340 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.340 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.340 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.341 INFO:teuthology.orchestra.run.smithi123.stdout: "pool": 2, 2020-01-16T00:12:51.341 INFO:teuthology.orchestra.run.smithi123.stdout: "pool_name": "low_tier", 2020-01-16T00:12:51.341 INFO:teuthology.orchestra.run.smithi123.stdout: "create_time": "2020-01-16T00:12:30.019477+0000", 2020-01-16T00:12:51.341 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2020-01-16T00:12:51.341 INFO:teuthology.orchestra.run.smithi123.stdout: "flags_names": "hashpspool", 2020-01-16T00:12:51.341 INFO:teuthology.orchestra.run.smithi123.stdout: "type": 1, 2020-01-16T00:12:51.342 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 3, 2020-01-16T00:12:51.342 INFO:teuthology.orchestra.run.smithi123.stdout: "min_size": 2, 2020-01-16T00:12:51.342 INFO:teuthology.orchestra.run.smithi123.stdout: "crush_rule": 0, 2020-01-16T00:12:51.342 INFO:teuthology.orchestra.run.smithi123.stdout: "object_hash": 2, 2020-01-16T00:12:51.342 INFO:teuthology.orchestra.run.smithi123.stdout: "pg_autoscale_mode": "off", 2020-01-16T00:12:51.342 INFO:teuthology.orchestra.run.smithi123.stdout: "pg_num": 4, 2020-01-16T00:12:51.342 INFO:teuthology.orchestra.run.smithi123.stdout: "pg_placement_num": 4, 2020-01-16T00:12:51.343 INFO:teuthology.orchestra.run.smithi123.stdout: "pg_placement_num_target": 4, 2020-01-16T00:12:51.343 INFO:teuthology.orchestra.run.smithi123.stdout: "pg_num_target": 4, 2020-01-16T00:12:51.343 INFO:teuthology.orchestra.run.smithi123.stdout: "pg_num_pending": 4, 2020-01-16T00:12:51.343 INFO:teuthology.orchestra.run.smithi123.stdout: "last_pg_merge_meta": { 2020-01-16T00:12:51.343 INFO:teuthology.orchestra.run.smithi123.stdout: "source_pgid": "0.0", 2020-01-16T00:12:51.343 INFO:teuthology.orchestra.run.smithi123.stdout: "ready_epoch": 0, 2020-01-16T00:12:51.343 INFO:teuthology.orchestra.run.smithi123.stdout: "last_epoch_started": 0, 2020-01-16T00:12:51.344 INFO:teuthology.orchestra.run.smithi123.stdout: "last_epoch_clean": 0, 2020-01-16T00:12:51.344 INFO:teuthology.orchestra.run.smithi123.stdout: "source_version": "0'0", 2020-01-16T00:12:51.344 INFO:teuthology.orchestra.run.smithi123.stdout: "target_version": "0'0" 2020-01-16T00:12:51.344 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.344 INFO:teuthology.orchestra.run.smithi123.stdout: "last_change": "21", 2020-01-16T00:12:51.344 INFO:teuthology.orchestra.run.smithi123.stdout: "last_force_op_resend": "0", 2020-01-16T00:12:51.345 INFO:teuthology.orchestra.run.smithi123.stdout: "last_force_op_resend_prenautilus": "0", 2020-01-16T00:12:51.345 INFO:teuthology.orchestra.run.smithi123.stdout: "last_force_op_resend_preluminous": "0", 2020-01-16T00:12:51.345 INFO:teuthology.orchestra.run.smithi123.stdout: "auid": 0, 2020-01-16T00:12:51.345 INFO:teuthology.orchestra.run.smithi123.stdout: "snap_mode": "selfmanaged", 2020-01-16T00:12:51.345 INFO:teuthology.orchestra.run.smithi123.stdout: "snap_seq": 0, 2020-01-16T00:12:51.345 INFO:teuthology.orchestra.run.smithi123.stdout: "snap_epoch": 0, 2020-01-16T00:12:51.346 INFO:teuthology.orchestra.run.smithi123.stdout: "pool_snaps": [], 2020-01-16T00:12:51.346 INFO:teuthology.orchestra.run.smithi123.stdout: "removed_snaps": "[]", 2020-01-16T00:12:51.346 INFO:teuthology.orchestra.run.smithi123.stdout: "quota_max_bytes": 0, 2020-01-16T00:12:51.346 INFO:teuthology.orchestra.run.smithi123.stdout: "quota_max_objects": 0, 2020-01-16T00:12:51.346 INFO:teuthology.orchestra.run.smithi123.stdout: "tiers": [], 2020-01-16T00:12:51.346 INFO:teuthology.orchestra.run.smithi123.stdout: "tier_of": -1, 2020-01-16T00:12:51.347 INFO:teuthology.orchestra.run.smithi123.stdout: "read_tier": -1, 2020-01-16T00:12:51.347 INFO:teuthology.orchestra.run.smithi123.stdout: "write_tier": -1, 2020-01-16T00:12:51.347 INFO:teuthology.orchestra.run.smithi123.stdout: "cache_mode": "none", 2020-01-16T00:12:51.347 INFO:teuthology.orchestra.run.smithi123.stdout: "target_max_bytes": 0, 2020-01-16T00:12:51.348 INFO:teuthology.orchestra.run.smithi123.stdout: "target_max_objects": 0, 2020-01-16T00:12:51.348 INFO:teuthology.orchestra.run.smithi123.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-01-16T00:12:51.348 INFO:teuthology.orchestra.run.smithi123.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-01-16T00:12:51.348 INFO:teuthology.orchestra.run.smithi123.stdout: "cache_target_full_ratio_micro": 800000, 2020-01-16T00:12:51.349 INFO:teuthology.orchestra.run.smithi123.stdout: "cache_min_flush_age": 0, 2020-01-16T00:12:51.349 INFO:teuthology.orchestra.run.smithi123.stdout: "cache_min_evict_age": 0, 2020-01-16T00:12:51.349 INFO:teuthology.orchestra.run.smithi123.stdout: "erasure_code_profile": "", 2020-01-16T00:12:51.349 INFO:teuthology.orchestra.run.smithi123.stdout: "hit_set_params": { 2020-01-16T00:12:51.350 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "none" 2020-01-16T00:12:51.350 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.350 INFO:teuthology.orchestra.run.smithi123.stdout: "hit_set_period": 0, 2020-01-16T00:12:51.350 INFO:teuthology.orchestra.run.smithi123.stdout: "hit_set_count": 0, 2020-01-16T00:12:51.350 INFO:teuthology.orchestra.run.smithi123.stdout: "use_gmt_hitset": true, 2020-01-16T00:12:51.351 INFO:teuthology.orchestra.run.smithi123.stdout: "min_read_recency_for_promote": 0, 2020-01-16T00:12:51.351 INFO:teuthology.orchestra.run.smithi123.stdout: "min_write_recency_for_promote": 0, 2020-01-16T00:12:51.351 INFO:teuthology.orchestra.run.smithi123.stdout: "hit_set_grade_decay_rate": 0, 2020-01-16T00:12:51.351 INFO:teuthology.orchestra.run.smithi123.stdout: "hit_set_search_last_n": 0, 2020-01-16T00:12:51.352 INFO:teuthology.orchestra.run.smithi123.stdout: "grade_table": [], 2020-01-16T00:12:51.352 INFO:teuthology.orchestra.run.smithi123.stdout: "stripe_width": 0, 2020-01-16T00:12:51.352 INFO:teuthology.orchestra.run.smithi123.stdout: "expected_num_objects": 0, 2020-01-16T00:12:51.352 INFO:teuthology.orchestra.run.smithi123.stdout: "fast_read": false, 2020-01-16T00:12:51.353 INFO:teuthology.orchestra.run.smithi123.stdout: "options": {}, 2020-01-16T00:12:51.353 INFO:teuthology.orchestra.run.smithi123.stdout: "application_metadata": {} 2020-01-16T00:12:51.353 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.353 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.353 INFO:teuthology.orchestra.run.smithi123.stdout: "pool": 3, 2020-01-16T00:12:51.354 INFO:teuthology.orchestra.run.smithi123.stdout: "pool_name": "unique_pool_0", 2020-01-16T00:12:51.354 INFO:teuthology.orchestra.run.smithi123.stdout: "create_time": "2020-01-16T00:12:31.098841+0000", 2020-01-16T00:12:51.354 INFO:teuthology.orchestra.run.smithi123.stdout: "flags": 1, 2020-01-16T00:12:51.354 INFO:teuthology.orchestra.run.smithi123.stdout: "flags_names": "hashpspool", 2020-01-16T00:12:51.355 INFO:teuthology.orchestra.run.smithi123.stdout: "type": 1, 2020-01-16T00:12:51.355 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 3, 2020-01-16T00:12:51.355 INFO:teuthology.orchestra.run.smithi123.stdout: "min_size": 2, 2020-01-16T00:12:51.358 INFO:teuthology.orchestra.run.smithi123.stdout: "crush_rule": 0, 2020-01-16T00:12:51.358 INFO:teuthology.orchestra.run.smithi123.stdout: "object_hash": 2, 2020-01-16T00:12:51.360 INFO:teuthology.orchestra.run.smithi123.stdout: "pg_autoscale_mode": "off", 2020-01-16T00:12:51.361 INFO:teuthology.orchestra.run.smithi123.stdout: "pg_num": 16, 2020-01-16T00:12:51.361 INFO:teuthology.orchestra.run.smithi123.stdout: "pg_placement_num": 16, 2020-01-16T00:12:51.361 INFO:teuthology.orchestra.run.smithi123.stdout: "pg_placement_num_target": 16, 2020-01-16T00:12:51.361 INFO:teuthology.orchestra.run.smithi123.stdout: "pg_num_target": 16, 2020-01-16T00:12:51.361 INFO:teuthology.orchestra.run.smithi123.stdout: "pg_num_pending": 16, 2020-01-16T00:12:51.361 INFO:teuthology.orchestra.run.smithi123.stdout: "last_pg_merge_meta": { 2020-01-16T00:12:51.362 INFO:teuthology.orchestra.run.smithi123.stdout: "source_pgid": "0.0", 2020-01-16T00:12:51.362 INFO:teuthology.orchestra.run.smithi123.stdout: "ready_epoch": 0, 2020-01-16T00:12:51.362 INFO:teuthology.orchestra.run.smithi123.stdout: "last_epoch_started": 0, 2020-01-16T00:12:51.362 INFO:teuthology.orchestra.run.smithi123.stdout: "last_epoch_clean": 0, 2020-01-16T00:12:51.362 INFO:teuthology.orchestra.run.smithi123.stdout: "source_version": "0'0", 2020-01-16T00:12:51.362 INFO:teuthology.orchestra.run.smithi123.stdout: "target_version": "0'0" 2020-01-16T00:12:51.363 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.363 INFO:teuthology.orchestra.run.smithi123.stdout: "last_change": "26", 2020-01-16T00:12:51.363 INFO:teuthology.orchestra.run.smithi123.stdout: "last_force_op_resend": "0", 2020-01-16T00:12:51.363 INFO:teuthology.orchestra.run.smithi123.stdout: "last_force_op_resend_prenautilus": "0", 2020-01-16T00:12:51.363 INFO:teuthology.orchestra.run.smithi123.stdout: "last_force_op_resend_preluminous": "0", 2020-01-16T00:12:51.363 INFO:teuthology.orchestra.run.smithi123.stdout: "auid": 0, 2020-01-16T00:12:51.363 INFO:teuthology.orchestra.run.smithi123.stdout: "snap_mode": "selfmanaged", 2020-01-16T00:12:51.364 INFO:teuthology.orchestra.run.smithi123.stdout: "snap_seq": 0, 2020-01-16T00:12:51.364 INFO:teuthology.orchestra.run.smithi123.stdout: "snap_epoch": 0, 2020-01-16T00:12:51.364 INFO:teuthology.orchestra.run.smithi123.stdout: "pool_snaps": [], 2020-01-16T00:12:51.364 INFO:teuthology.orchestra.run.smithi123.stdout: "removed_snaps": "[]", 2020-01-16T00:12:51.364 INFO:teuthology.orchestra.run.smithi123.stdout: "quota_max_bytes": 0, 2020-01-16T00:12:51.364 INFO:teuthology.orchestra.run.smithi123.stdout: "quota_max_objects": 0, 2020-01-16T00:12:51.364 INFO:teuthology.orchestra.run.smithi123.stdout: "tiers": [], 2020-01-16T00:12:51.364 INFO:teuthology.orchestra.run.smithi123.stdout: "tier_of": -1, 2020-01-16T00:12:51.365 INFO:teuthology.orchestra.run.smithi123.stdout: "read_tier": -1, 2020-01-16T00:12:51.365 INFO:teuthology.orchestra.run.smithi123.stdout: "write_tier": -1, 2020-01-16T00:12:51.365 INFO:teuthology.orchestra.run.smithi123.stdout: "cache_mode": "none", 2020-01-16T00:12:51.365 INFO:teuthology.orchestra.run.smithi123.stdout: "target_max_bytes": 0, 2020-01-16T00:12:51.365 INFO:teuthology.orchestra.run.smithi123.stdout: "target_max_objects": 0, 2020-01-16T00:12:51.366 INFO:teuthology.orchestra.run.smithi123.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-01-16T00:12:51.366 INFO:teuthology.orchestra.run.smithi123.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-01-16T00:12:51.366 INFO:teuthology.orchestra.run.smithi123.stdout: "cache_target_full_ratio_micro": 800000, 2020-01-16T00:12:51.366 INFO:teuthology.orchestra.run.smithi123.stdout: "cache_min_flush_age": 0, 2020-01-16T00:12:51.366 INFO:teuthology.orchestra.run.smithi123.stdout: "cache_min_evict_age": 0, 2020-01-16T00:12:51.367 INFO:teuthology.orchestra.run.smithi123.stdout: "erasure_code_profile": "", 2020-01-16T00:12:51.367 INFO:teuthology.orchestra.run.smithi123.stdout: "hit_set_params": { 2020-01-16T00:12:51.367 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "none" 2020-01-16T00:12:51.367 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.367 INFO:teuthology.orchestra.run.smithi123.stdout: "hit_set_period": 0, 2020-01-16T00:12:51.367 INFO:teuthology.orchestra.run.smithi123.stdout: "hit_set_count": 0, 2020-01-16T00:12:51.367 INFO:teuthology.orchestra.run.smithi123.stdout: "use_gmt_hitset": true, 2020-01-16T00:12:51.367 INFO:teuthology.orchestra.run.smithi123.stdout: "min_read_recency_for_promote": 0, 2020-01-16T00:12:51.368 INFO:teuthology.orchestra.run.smithi123.stdout: "min_write_recency_for_promote": 0, 2020-01-16T00:12:51.368 INFO:teuthology.orchestra.run.smithi123.stdout: "hit_set_grade_decay_rate": 0, 2020-01-16T00:12:51.368 INFO:teuthology.orchestra.run.smithi123.stdout: "hit_set_search_last_n": 0, 2020-01-16T00:12:51.368 INFO:teuthology.orchestra.run.smithi123.stdout: "grade_table": [], 2020-01-16T00:12:51.368 INFO:teuthology.orchestra.run.smithi123.stdout: "stripe_width": 0, 2020-01-16T00:12:51.368 INFO:teuthology.orchestra.run.smithi123.stdout: "expected_num_objects": 0, 2020-01-16T00:12:51.368 INFO:teuthology.orchestra.run.smithi123.stdout: "fast_read": false, 2020-01-16T00:12:51.368 INFO:teuthology.orchestra.run.smithi123.stdout: "options": {}, 2020-01-16T00:12:51.369 INFO:teuthology.orchestra.run.smithi123.stdout: "application_metadata": { 2020-01-16T00:12:51.369 INFO:teuthology.orchestra.run.smithi123.stdout: "rados": {} 2020-01-16T00:12:51.369 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.369 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.369 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2020-01-16T00:12:51.369 INFO:teuthology.orchestra.run.smithi123.stdout: "osds": [ 2020-01-16T00:12:51.369 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.370 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": 0, 2020-01-16T00:12:51.370 INFO:teuthology.orchestra.run.smithi123.stdout: "uuid": "4b8d1e40-95e3-4adc-9929-cb7e6cc8b766", 2020-01-16T00:12:51.370 INFO:teuthology.orchestra.run.smithi123.stdout: "up": 1, 2020-01-16T00:12:51.370 INFO:teuthology.orchestra.run.smithi123.stdout: "in": 1, 2020-01-16T00:12:51.370 INFO:teuthology.orchestra.run.smithi123.stdout: "weight": 1, 2020-01-16T00:12:51.370 INFO:teuthology.orchestra.run.smithi123.stdout: "primary_affinity": 1, 2020-01-16T00:12:51.370 INFO:teuthology.orchestra.run.smithi123.stdout: "last_clean_begin": 0, 2020-01-16T00:12:51.370 INFO:teuthology.orchestra.run.smithi123.stdout: "last_clean_end": 0, 2020-01-16T00:12:51.371 INFO:teuthology.orchestra.run.smithi123.stdout: "up_from": 13, 2020-01-16T00:12:51.371 INFO:teuthology.orchestra.run.smithi123.stdout: "up_thru": 20, 2020-01-16T00:12:51.371 INFO:teuthology.orchestra.run.smithi123.stdout: "down_at": 0, 2020-01-16T00:12:51.371 INFO:teuthology.orchestra.run.smithi123.stdout: "lost_at": 0, 2020-01-16T00:12:51.371 INFO:teuthology.orchestra.run.smithi123.stdout: "public_addrs": { 2020-01-16T00:12:51.372 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2020-01-16T00:12:51.372 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.372 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2020-01-16T00:12:51.372 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.123:6809", 2020-01-16T00:12:51.372 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 11906 2020-01-16T00:12:51.373 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.373 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.373 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.373 INFO:teuthology.orchestra.run.smithi123.stdout: "cluster_addrs": { 2020-01-16T00:12:51.374 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2020-01-16T00:12:51.374 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.374 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2020-01-16T00:12:51.374 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.123:6810", 2020-01-16T00:12:51.374 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 11906 2020-01-16T00:12:51.374 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.374 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.375 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.375 INFO:teuthology.orchestra.run.smithi123.stdout: "heartbeat_back_addrs": { 2020-01-16T00:12:51.375 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2020-01-16T00:12:51.375 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.375 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2020-01-16T00:12:51.375 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.123:6812", 2020-01-16T00:12:51.375 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 11906 2020-01-16T00:12:51.375 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.376 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.376 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.376 INFO:teuthology.orchestra.run.smithi123.stdout: "heartbeat_front_addrs": { 2020-01-16T00:12:51.376 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2020-01-16T00:12:51.376 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.376 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2020-01-16T00:12:51.377 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.123:6811", 2020-01-16T00:12:51.377 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 11906 2020-01-16T00:12:51.377 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.377 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.377 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.377 INFO:teuthology.orchestra.run.smithi123.stdout: "public_addr": "172.21.15.123:6809/11906", 2020-01-16T00:12:51.377 INFO:teuthology.orchestra.run.smithi123.stdout: "cluster_addr": "172.21.15.123:6810/11906", 2020-01-16T00:12:51.377 INFO:teuthology.orchestra.run.smithi123.stdout: "heartbeat_back_addr": "172.21.15.123:6812/11906", 2020-01-16T00:12:51.378 INFO:teuthology.orchestra.run.smithi123.stdout: "heartbeat_front_addr": "172.21.15.123:6811/11906", 2020-01-16T00:12:51.378 INFO:teuthology.orchestra.run.smithi123.stdout: "state": [ 2020-01-16T00:12:51.378 INFO:teuthology.orchestra.run.smithi123.stdout: "exists", 2020-01-16T00:12:51.378 INFO:teuthology.orchestra.run.smithi123.stdout: "up" 2020-01-16T00:12:51.378 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.378 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.378 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.379 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": 1, 2020-01-16T00:12:51.379 INFO:teuthology.orchestra.run.smithi123.stdout: "uuid": "2e373d15-6348-4caa-aa9f-b46037376a3b", 2020-01-16T00:12:51.379 INFO:teuthology.orchestra.run.smithi123.stdout: "up": 1, 2020-01-16T00:12:51.379 INFO:teuthology.orchestra.run.smithi123.stdout: "in": 1, 2020-01-16T00:12:51.379 INFO:teuthology.orchestra.run.smithi123.stdout: "weight": 1, 2020-01-16T00:12:51.379 INFO:teuthology.orchestra.run.smithi123.stdout: "primary_affinity": 1, 2020-01-16T00:12:51.380 INFO:teuthology.orchestra.run.smithi123.stdout: "last_clean_begin": 0, 2020-01-16T00:12:51.380 INFO:teuthology.orchestra.run.smithi123.stdout: "last_clean_end": 0, 2020-01-16T00:12:51.380 INFO:teuthology.orchestra.run.smithi123.stdout: "up_from": 13, 2020-01-16T00:12:51.380 INFO:teuthology.orchestra.run.smithi123.stdout: "up_thru": 20, 2020-01-16T00:12:51.380 INFO:teuthology.orchestra.run.smithi123.stdout: "down_at": 0, 2020-01-16T00:12:51.380 INFO:teuthology.orchestra.run.smithi123.stdout: "lost_at": 0, 2020-01-16T00:12:51.380 INFO:teuthology.orchestra.run.smithi123.stdout: "public_addrs": { 2020-01-16T00:12:51.381 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2020-01-16T00:12:51.381 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.381 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2020-01-16T00:12:51.381 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.123:6805", 2020-01-16T00:12:51.381 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 11905 2020-01-16T00:12:51.381 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.381 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.381 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.382 INFO:teuthology.orchestra.run.smithi123.stdout: "cluster_addrs": { 2020-01-16T00:12:51.382 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2020-01-16T00:12:51.382 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.382 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2020-01-16T00:12:51.382 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.123:6806", 2020-01-16T00:12:51.382 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 11905 2020-01-16T00:12:51.382 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.383 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.383 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.383 INFO:teuthology.orchestra.run.smithi123.stdout: "heartbeat_back_addrs": { 2020-01-16T00:12:51.383 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2020-01-16T00:12:51.383 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.383 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2020-01-16T00:12:51.384 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.123:6808", 2020-01-16T00:12:51.384 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 11905 2020-01-16T00:12:51.384 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.384 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.385 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.385 INFO:teuthology.orchestra.run.smithi123.stdout: "heartbeat_front_addrs": { 2020-01-16T00:12:51.385 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2020-01-16T00:12:51.385 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.385 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2020-01-16T00:12:51.385 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.123:6807", 2020-01-16T00:12:51.386 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 11905 2020-01-16T00:12:51.386 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.386 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.386 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.386 INFO:teuthology.orchestra.run.smithi123.stdout: "public_addr": "172.21.15.123:6805/11905", 2020-01-16T00:12:51.386 INFO:teuthology.orchestra.run.smithi123.stdout: "cluster_addr": "172.21.15.123:6806/11905", 2020-01-16T00:12:51.387 INFO:teuthology.orchestra.run.smithi123.stdout: "heartbeat_back_addr": "172.21.15.123:6808/11905", 2020-01-16T00:12:51.387 INFO:teuthology.orchestra.run.smithi123.stdout: "heartbeat_front_addr": "172.21.15.123:6807/11905", 2020-01-16T00:12:51.387 INFO:teuthology.orchestra.run.smithi123.stdout: "state": [ 2020-01-16T00:12:51.387 INFO:teuthology.orchestra.run.smithi123.stdout: "exists", 2020-01-16T00:12:51.387 INFO:teuthology.orchestra.run.smithi123.stdout: "up" 2020-01-16T00:12:51.388 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.388 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.388 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.388 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": 2, 2020-01-16T00:12:51.388 INFO:teuthology.orchestra.run.smithi123.stdout: "uuid": "5736ff2c-076a-4d41-8e11-06e2fd1f5f94", 2020-01-16T00:12:51.389 INFO:teuthology.orchestra.run.smithi123.stdout: "up": 1, 2020-01-16T00:12:51.389 INFO:teuthology.orchestra.run.smithi123.stdout: "in": 1, 2020-01-16T00:12:51.389 INFO:teuthology.orchestra.run.smithi123.stdout: "weight": 1, 2020-01-16T00:12:51.389 INFO:teuthology.orchestra.run.smithi123.stdout: "primary_affinity": 1, 2020-01-16T00:12:51.390 INFO:teuthology.orchestra.run.smithi123.stdout: "last_clean_begin": 0, 2020-01-16T00:12:51.390 INFO:teuthology.orchestra.run.smithi123.stdout: "last_clean_end": 0, 2020-01-16T00:12:51.390 INFO:teuthology.orchestra.run.smithi123.stdout: "up_from": 13, 2020-01-16T00:12:51.391 INFO:teuthology.orchestra.run.smithi123.stdout: "up_thru": 19, 2020-01-16T00:12:51.392 INFO:teuthology.orchestra.run.smithi123.stdout: "down_at": 0, 2020-01-16T00:12:51.392 INFO:teuthology.orchestra.run.smithi123.stdout: "lost_at": 0, 2020-01-16T00:12:51.392 INFO:teuthology.orchestra.run.smithi123.stdout: "public_addrs": { 2020-01-16T00:12:51.392 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2020-01-16T00:12:51.392 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.392 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2020-01-16T00:12:51.392 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.123:6801", 2020-01-16T00:12:51.392 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 11907 2020-01-16T00:12:51.393 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.393 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.393 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.393 INFO:teuthology.orchestra.run.smithi123.stdout: "cluster_addrs": { 2020-01-16T00:12:51.393 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2020-01-16T00:12:51.393 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.393 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2020-01-16T00:12:51.394 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.123:6802", 2020-01-16T00:12:51.394 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 11907 2020-01-16T00:12:51.394 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.394 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.394 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.394 INFO:teuthology.orchestra.run.smithi123.stdout: "heartbeat_back_addrs": { 2020-01-16T00:12:51.395 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2020-01-16T00:12:51.395 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.395 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2020-01-16T00:12:51.395 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.123:6804", 2020-01-16T00:12:51.395 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 11907 2020-01-16T00:12:51.395 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.396 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.396 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.396 INFO:teuthology.orchestra.run.smithi123.stdout: "heartbeat_front_addrs": { 2020-01-16T00:12:51.397 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2020-01-16T00:12:51.397 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.397 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2020-01-16T00:12:51.397 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.123:6803", 2020-01-16T00:12:51.397 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 11907 2020-01-16T00:12:51.397 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.397 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.398 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.398 INFO:teuthology.orchestra.run.smithi123.stdout: "public_addr": "172.21.15.123:6801/11907", 2020-01-16T00:12:51.398 INFO:teuthology.orchestra.run.smithi123.stdout: "cluster_addr": "172.21.15.123:6802/11907", 2020-01-16T00:12:51.398 INFO:teuthology.orchestra.run.smithi123.stdout: "heartbeat_back_addr": "172.21.15.123:6804/11907", 2020-01-16T00:12:51.398 INFO:teuthology.orchestra.run.smithi123.stdout: "heartbeat_front_addr": "172.21.15.123:6803/11907", 2020-01-16T00:12:51.398 INFO:teuthology.orchestra.run.smithi123.stdout: "state": [ 2020-01-16T00:12:51.398 INFO:teuthology.orchestra.run.smithi123.stdout: "exists", 2020-01-16T00:12:51.398 INFO:teuthology.orchestra.run.smithi123.stdout: "up" 2020-01-16T00:12:51.399 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.399 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.399 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.399 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": 3, 2020-01-16T00:12:51.399 INFO:teuthology.orchestra.run.smithi123.stdout: "uuid": "554f53a8-c5b1-43c9-a7d6-93629d84f1ea", 2020-01-16T00:12:51.399 INFO:teuthology.orchestra.run.smithi123.stdout: "up": 1, 2020-01-16T00:12:51.400 INFO:teuthology.orchestra.run.smithi123.stdout: "in": 1, 2020-01-16T00:12:51.400 INFO:teuthology.orchestra.run.smithi123.stdout: "weight": 1, 2020-01-16T00:12:51.400 INFO:teuthology.orchestra.run.smithi123.stdout: "primary_affinity": 1, 2020-01-16T00:12:51.400 INFO:teuthology.orchestra.run.smithi123.stdout: "last_clean_begin": 0, 2020-01-16T00:12:51.400 INFO:teuthology.orchestra.run.smithi123.stdout: "last_clean_end": 0, 2020-01-16T00:12:51.400 INFO:teuthology.orchestra.run.smithi123.stdout: "up_from": 13, 2020-01-16T00:12:51.400 INFO:teuthology.orchestra.run.smithi123.stdout: "up_thru": 20, 2020-01-16T00:12:51.400 INFO:teuthology.orchestra.run.smithi123.stdout: "down_at": 0, 2020-01-16T00:12:51.401 INFO:teuthology.orchestra.run.smithi123.stdout: "lost_at": 0, 2020-01-16T00:12:51.401 INFO:teuthology.orchestra.run.smithi123.stdout: "public_addrs": { 2020-01-16T00:12:51.401 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2020-01-16T00:12:51.401 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.401 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2020-01-16T00:12:51.401 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.123:6813", 2020-01-16T00:12:51.402 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 11917 2020-01-16T00:12:51.403 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.403 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.403 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.403 INFO:teuthology.orchestra.run.smithi123.stdout: "cluster_addrs": { 2020-01-16T00:12:51.403 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2020-01-16T00:12:51.404 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.404 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2020-01-16T00:12:51.404 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.123:6814", 2020-01-16T00:12:51.404 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 11917 2020-01-16T00:12:51.404 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.404 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.404 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.405 INFO:teuthology.orchestra.run.smithi123.stdout: "heartbeat_back_addrs": { 2020-01-16T00:12:51.405 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2020-01-16T00:12:51.405 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.405 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2020-01-16T00:12:51.405 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.123:6816", 2020-01-16T00:12:51.405 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 11917 2020-01-16T00:12:51.405 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.406 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.406 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.406 INFO:teuthology.orchestra.run.smithi123.stdout: "heartbeat_front_addrs": { 2020-01-16T00:12:51.406 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2020-01-16T00:12:51.406 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.406 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2020-01-16T00:12:51.407 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.123:6815", 2020-01-16T00:12:51.407 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 11917 2020-01-16T00:12:51.407 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.407 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.408 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.408 INFO:teuthology.orchestra.run.smithi123.stdout: "public_addr": "172.21.15.123:6813/11917", 2020-01-16T00:12:51.408 INFO:teuthology.orchestra.run.smithi123.stdout: "cluster_addr": "172.21.15.123:6814/11917", 2020-01-16T00:12:51.409 INFO:teuthology.orchestra.run.smithi123.stdout: "heartbeat_back_addr": "172.21.15.123:6816/11917", 2020-01-16T00:12:51.409 INFO:teuthology.orchestra.run.smithi123.stdout: "heartbeat_front_addr": "172.21.15.123:6815/11917", 2020-01-16T00:12:51.409 INFO:teuthology.orchestra.run.smithi123.stdout: "state": [ 2020-01-16T00:12:51.409 INFO:teuthology.orchestra.run.smithi123.stdout: "exists", 2020-01-16T00:12:51.409 INFO:teuthology.orchestra.run.smithi123.stdout: "up" 2020-01-16T00:12:51.410 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.410 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.410 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.410 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": 4, 2020-01-16T00:12:51.410 INFO:teuthology.orchestra.run.smithi123.stdout: "uuid": "77945419-c04b-4071-ab87-256d77640dc0", 2020-01-16T00:12:51.410 INFO:teuthology.orchestra.run.smithi123.stdout: "up": 1, 2020-01-16T00:12:51.410 INFO:teuthology.orchestra.run.smithi123.stdout: "in": 1, 2020-01-16T00:12:51.411 INFO:teuthology.orchestra.run.smithi123.stdout: "weight": 1, 2020-01-16T00:12:51.411 INFO:teuthology.orchestra.run.smithi123.stdout: "primary_affinity": 1, 2020-01-16T00:12:51.411 INFO:teuthology.orchestra.run.smithi123.stdout: "last_clean_begin": 0, 2020-01-16T00:12:51.411 INFO:teuthology.orchestra.run.smithi123.stdout: "last_clean_end": 0, 2020-01-16T00:12:51.411 INFO:teuthology.orchestra.run.smithi123.stdout: "up_from": 13, 2020-01-16T00:12:51.411 INFO:teuthology.orchestra.run.smithi123.stdout: "up_thru": 20, 2020-01-16T00:12:51.411 INFO:teuthology.orchestra.run.smithi123.stdout: "down_at": 0, 2020-01-16T00:12:51.412 INFO:teuthology.orchestra.run.smithi123.stdout: "lost_at": 0, 2020-01-16T00:12:51.412 INFO:teuthology.orchestra.run.smithi123.stdout: "public_addrs": { 2020-01-16T00:12:51.412 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2020-01-16T00:12:51.412 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.412 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2020-01-16T00:12:51.412 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.145:6800", 2020-01-16T00:12:51.412 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 11851 2020-01-16T00:12:51.413 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.413 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.413 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.413 INFO:teuthology.orchestra.run.smithi123.stdout: "cluster_addrs": { 2020-01-16T00:12:51.413 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2020-01-16T00:12:51.413 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.413 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2020-01-16T00:12:51.414 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.145:6801", 2020-01-16T00:12:51.414 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 11851 2020-01-16T00:12:51.415 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.415 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.415 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.415 INFO:teuthology.orchestra.run.smithi123.stdout: "heartbeat_back_addrs": { 2020-01-16T00:12:51.415 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2020-01-16T00:12:51.415 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.415 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2020-01-16T00:12:51.415 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.145:6803", 2020-01-16T00:12:51.416 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 11851 2020-01-16T00:12:51.416 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.416 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.416 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.416 INFO:teuthology.orchestra.run.smithi123.stdout: "heartbeat_front_addrs": { 2020-01-16T00:12:51.416 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2020-01-16T00:12:51.416 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.417 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2020-01-16T00:12:51.417 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.145:6802", 2020-01-16T00:12:51.417 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 11851 2020-01-16T00:12:51.417 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.417 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.417 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.417 INFO:teuthology.orchestra.run.smithi123.stdout: "public_addr": "172.21.15.145:6800/11851", 2020-01-16T00:12:51.418 INFO:teuthology.orchestra.run.smithi123.stdout: "cluster_addr": "172.21.15.145:6801/11851", 2020-01-16T00:12:51.418 INFO:teuthology.orchestra.run.smithi123.stdout: "heartbeat_back_addr": "172.21.15.145:6803/11851", 2020-01-16T00:12:51.418 INFO:teuthology.orchestra.run.smithi123.stdout: "heartbeat_front_addr": "172.21.15.145:6802/11851", 2020-01-16T00:12:51.418 INFO:teuthology.orchestra.run.smithi123.stdout: "state": [ 2020-01-16T00:12:51.418 INFO:teuthology.orchestra.run.smithi123.stdout: "exists", 2020-01-16T00:12:51.418 INFO:teuthology.orchestra.run.smithi123.stdout: "up" 2020-01-16T00:12:51.418 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.419 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.419 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.419 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": 5, 2020-01-16T00:12:51.419 INFO:teuthology.orchestra.run.smithi123.stdout: "uuid": "d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4", 2020-01-16T00:12:51.419 INFO:teuthology.orchestra.run.smithi123.stdout: "up": 1, 2020-01-16T00:12:51.419 INFO:teuthology.orchestra.run.smithi123.stdout: "in": 1, 2020-01-16T00:12:51.419 INFO:teuthology.orchestra.run.smithi123.stdout: "weight": 1, 2020-01-16T00:12:51.419 INFO:teuthology.orchestra.run.smithi123.stdout: "primary_affinity": 1, 2020-01-16T00:12:51.420 INFO:teuthology.orchestra.run.smithi123.stdout: "last_clean_begin": 0, 2020-01-16T00:12:51.420 INFO:teuthology.orchestra.run.smithi123.stdout: "last_clean_end": 0, 2020-01-16T00:12:51.420 INFO:teuthology.orchestra.run.smithi123.stdout: "up_from": 13, 2020-01-16T00:12:51.420 INFO:teuthology.orchestra.run.smithi123.stdout: "up_thru": 20, 2020-01-16T00:12:51.420 INFO:teuthology.orchestra.run.smithi123.stdout: "down_at": 0, 2020-01-16T00:12:51.420 INFO:teuthology.orchestra.run.smithi123.stdout: "lost_at": 0, 2020-01-16T00:12:51.420 INFO:teuthology.orchestra.run.smithi123.stdout: "public_addrs": { 2020-01-16T00:12:51.421 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2020-01-16T00:12:51.421 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.421 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2020-01-16T00:12:51.421 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.145:6804", 2020-01-16T00:12:51.421 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 11860 2020-01-16T00:12:51.421 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.421 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.421 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.422 INFO:teuthology.orchestra.run.smithi123.stdout: "cluster_addrs": { 2020-01-16T00:12:51.422 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2020-01-16T00:12:51.422 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.422 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2020-01-16T00:12:51.422 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.145:6805", 2020-01-16T00:12:51.422 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 11860 2020-01-16T00:12:51.422 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.423 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.423 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.423 INFO:teuthology.orchestra.run.smithi123.stdout: "heartbeat_back_addrs": { 2020-01-16T00:12:51.423 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2020-01-16T00:12:51.423 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.423 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2020-01-16T00:12:51.423 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.145:6807", 2020-01-16T00:12:51.423 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 11860 2020-01-16T00:12:51.424 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.424 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.424 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.424 INFO:teuthology.orchestra.run.smithi123.stdout: "heartbeat_front_addrs": { 2020-01-16T00:12:51.424 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2020-01-16T00:12:51.424 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.424 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2020-01-16T00:12:51.424 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.145:6806", 2020-01-16T00:12:51.425 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 11860 2020-01-16T00:12:51.425 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.425 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.425 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.425 INFO:teuthology.orchestra.run.smithi123.stdout: "public_addr": "172.21.15.145:6804/11860", 2020-01-16T00:12:51.425 INFO:teuthology.orchestra.run.smithi123.stdout: "cluster_addr": "172.21.15.145:6805/11860", 2020-01-16T00:12:51.425 INFO:teuthology.orchestra.run.smithi123.stdout: "heartbeat_back_addr": "172.21.15.145:6807/11860", 2020-01-16T00:12:51.426 INFO:teuthology.orchestra.run.smithi123.stdout: "heartbeat_front_addr": "172.21.15.145:6806/11860", 2020-01-16T00:12:51.426 INFO:teuthology.orchestra.run.smithi123.stdout: "state": [ 2020-01-16T00:12:51.426 INFO:teuthology.orchestra.run.smithi123.stdout: "exists", 2020-01-16T00:12:51.426 INFO:teuthology.orchestra.run.smithi123.stdout: "up" 2020-01-16T00:12:51.426 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.426 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.426 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.427 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": 6, 2020-01-16T00:12:51.427 INFO:teuthology.orchestra.run.smithi123.stdout: "uuid": "e9d342cd-5c2d-40a0-b8ea-24588f280389", 2020-01-16T00:12:51.427 INFO:teuthology.orchestra.run.smithi123.stdout: "up": 1, 2020-01-16T00:12:51.427 INFO:teuthology.orchestra.run.smithi123.stdout: "in": 1, 2020-01-16T00:12:51.427 INFO:teuthology.orchestra.run.smithi123.stdout: "weight": 1, 2020-01-16T00:12:51.427 INFO:teuthology.orchestra.run.smithi123.stdout: "primary_affinity": 1, 2020-01-16T00:12:51.427 INFO:teuthology.orchestra.run.smithi123.stdout: "last_clean_begin": 0, 2020-01-16T00:12:51.427 INFO:teuthology.orchestra.run.smithi123.stdout: "last_clean_end": 0, 2020-01-16T00:12:51.428 INFO:teuthology.orchestra.run.smithi123.stdout: "up_from": 13, 2020-01-16T00:12:51.428 INFO:teuthology.orchestra.run.smithi123.stdout: "up_thru": 20, 2020-01-16T00:12:51.428 INFO:teuthology.orchestra.run.smithi123.stdout: "down_at": 0, 2020-01-16T00:12:51.429 INFO:teuthology.orchestra.run.smithi123.stdout: "lost_at": 0, 2020-01-16T00:12:51.429 INFO:teuthology.orchestra.run.smithi123.stdout: "public_addrs": { 2020-01-16T00:12:51.429 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2020-01-16T00:12:51.429 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.430 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2020-01-16T00:12:51.430 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.145:6812", 2020-01-16T00:12:51.430 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 11867 2020-01-16T00:12:51.430 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.431 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.431 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.431 INFO:teuthology.orchestra.run.smithi123.stdout: "cluster_addrs": { 2020-01-16T00:12:51.432 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2020-01-16T00:12:51.432 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.433 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2020-01-16T00:12:51.433 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.145:6813", 2020-01-16T00:12:51.433 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 11867 2020-01-16T00:12:51.433 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.435 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.436 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.436 INFO:teuthology.orchestra.run.smithi123.stdout: "heartbeat_back_addrs": { 2020-01-16T00:12:51.436 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2020-01-16T00:12:51.436 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.436 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2020-01-16T00:12:51.436 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.145:6815", 2020-01-16T00:12:51.436 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 11867 2020-01-16T00:12:51.437 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.437 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.437 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.437 INFO:teuthology.orchestra.run.smithi123.stdout: "heartbeat_front_addrs": { 2020-01-16T00:12:51.437 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2020-01-16T00:12:51.438 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.438 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2020-01-16T00:12:51.438 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.145:6814", 2020-01-16T00:12:51.438 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 11867 2020-01-16T00:12:51.438 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.438 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.439 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.439 INFO:teuthology.orchestra.run.smithi123.stdout: "public_addr": "172.21.15.145:6812/11867", 2020-01-16T00:12:51.439 INFO:teuthology.orchestra.run.smithi123.stdout: "cluster_addr": "172.21.15.145:6813/11867", 2020-01-16T00:12:51.439 INFO:teuthology.orchestra.run.smithi123.stdout: "heartbeat_back_addr": "172.21.15.145:6815/11867", 2020-01-16T00:12:51.439 INFO:teuthology.orchestra.run.smithi123.stdout: "heartbeat_front_addr": "172.21.15.145:6814/11867", 2020-01-16T00:12:51.439 INFO:teuthology.orchestra.run.smithi123.stdout: "state": [ 2020-01-16T00:12:51.440 INFO:teuthology.orchestra.run.smithi123.stdout: "exists", 2020-01-16T00:12:51.440 INFO:teuthology.orchestra.run.smithi123.stdout: "up" 2020-01-16T00:12:51.440 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.440 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.440 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.440 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": 7, 2020-01-16T00:12:51.441 INFO:teuthology.orchestra.run.smithi123.stdout: "uuid": "6a301b66-384a-40d8-adc8-96fff49e982e", 2020-01-16T00:12:51.441 INFO:teuthology.orchestra.run.smithi123.stdout: "up": 1, 2020-01-16T00:12:51.441 INFO:teuthology.orchestra.run.smithi123.stdout: "in": 1, 2020-01-16T00:12:51.441 INFO:teuthology.orchestra.run.smithi123.stdout: "weight": 1, 2020-01-16T00:12:51.441 INFO:teuthology.orchestra.run.smithi123.stdout: "primary_affinity": 1, 2020-01-16T00:12:51.442 INFO:teuthology.orchestra.run.smithi123.stdout: "last_clean_begin": 0, 2020-01-16T00:12:51.442 INFO:teuthology.orchestra.run.smithi123.stdout: "last_clean_end": 0, 2020-01-16T00:12:51.442 INFO:teuthology.orchestra.run.smithi123.stdout: "up_from": 13, 2020-01-16T00:12:51.442 INFO:teuthology.orchestra.run.smithi123.stdout: "up_thru": 20, 2020-01-16T00:12:51.442 INFO:teuthology.orchestra.run.smithi123.stdout: "down_at": 0, 2020-01-16T00:12:51.442 INFO:teuthology.orchestra.run.smithi123.stdout: "lost_at": 0, 2020-01-16T00:12:51.443 INFO:teuthology.orchestra.run.smithi123.stdout: "public_addrs": { 2020-01-16T00:12:51.443 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2020-01-16T00:12:51.443 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.443 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2020-01-16T00:12:51.443 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.145:6808", 2020-01-16T00:12:51.444 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 11868 2020-01-16T00:12:51.444 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.444 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.444 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.445 INFO:teuthology.orchestra.run.smithi123.stdout: "cluster_addrs": { 2020-01-16T00:12:51.445 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2020-01-16T00:12:51.445 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.445 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2020-01-16T00:12:51.446 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.145:6809", 2020-01-16T00:12:51.446 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 11868 2020-01-16T00:12:51.446 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.447 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.447 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.447 INFO:teuthology.orchestra.run.smithi123.stdout: "heartbeat_back_addrs": { 2020-01-16T00:12:51.448 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2020-01-16T00:12:51.448 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.449 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2020-01-16T00:12:51.449 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.145:6811", 2020-01-16T00:12:51.450 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 11868 2020-01-16T00:12:51.450 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.450 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.450 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.450 INFO:teuthology.orchestra.run.smithi123.stdout: "heartbeat_front_addrs": { 2020-01-16T00:12:51.450 INFO:teuthology.orchestra.run.smithi123.stdout: "addrvec": [ 2020-01-16T00:12:51.451 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.451 INFO:teuthology.orchestra.run.smithi123.stdout: "type": "v1", 2020-01-16T00:12:51.451 INFO:teuthology.orchestra.run.smithi123.stdout: "addr": "172.21.15.145:6810", 2020-01-16T00:12:51.451 INFO:teuthology.orchestra.run.smithi123.stdout: "nonce": 11868 2020-01-16T00:12:51.451 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.451 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.451 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.452 INFO:teuthology.orchestra.run.smithi123.stdout: "public_addr": "172.21.15.145:6808/11868", 2020-01-16T00:12:51.452 INFO:teuthology.orchestra.run.smithi123.stdout: "cluster_addr": "172.21.15.145:6809/11868", 2020-01-16T00:12:51.452 INFO:teuthology.orchestra.run.smithi123.stdout: "heartbeat_back_addr": "172.21.15.145:6811/11868", 2020-01-16T00:12:51.452 INFO:teuthology.orchestra.run.smithi123.stdout: "heartbeat_front_addr": "172.21.15.145:6810/11868", 2020-01-16T00:12:51.452 INFO:teuthology.orchestra.run.smithi123.stdout: "state": [ 2020-01-16T00:12:51.452 INFO:teuthology.orchestra.run.smithi123.stdout: "exists", 2020-01-16T00:12:51.452 INFO:teuthology.orchestra.run.smithi123.stdout: "up" 2020-01-16T00:12:51.452 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.453 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.453 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2020-01-16T00:12:51.453 INFO:teuthology.orchestra.run.smithi123.stdout: "osd_xinfo": [ 2020-01-16T00:12:51.453 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.453 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": 0, 2020-01-16T00:12:51.453 INFO:teuthology.orchestra.run.smithi123.stdout: "down_stamp": "0.000000", 2020-01-16T00:12:51.453 INFO:teuthology.orchestra.run.smithi123.stdout: "laggy_probability": 0, 2020-01-16T00:12:51.453 INFO:teuthology.orchestra.run.smithi123.stdout: "laggy_interval": 0, 2020-01-16T00:12:51.454 INFO:teuthology.orchestra.run.smithi123.stdout: "features": 4540138292836696063, 2020-01-16T00:12:51.454 INFO:teuthology.orchestra.run.smithi123.stdout: "old_weight": 0, 2020-01-16T00:12:51.454 INFO:teuthology.orchestra.run.smithi123.stdout: "last_purged_snaps_scrub": "2020-01-16T00:12:12.621038+0000", 2020-01-16T00:12:51.454 INFO:teuthology.orchestra.run.smithi123.stdout: "dead_epoch": 0 2020-01-16T00:12:51.454 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.454 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.454 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": 1, 2020-01-16T00:12:51.455 INFO:teuthology.orchestra.run.smithi123.stdout: "down_stamp": "0.000000", 2020-01-16T00:12:51.455 INFO:teuthology.orchestra.run.smithi123.stdout: "laggy_probability": 0, 2020-01-16T00:12:51.455 INFO:teuthology.orchestra.run.smithi123.stdout: "laggy_interval": 0, 2020-01-16T00:12:51.455 INFO:teuthology.orchestra.run.smithi123.stdout: "features": 4540138292836696063, 2020-01-16T00:12:51.455 INFO:teuthology.orchestra.run.smithi123.stdout: "old_weight": 0, 2020-01-16T00:12:51.455 INFO:teuthology.orchestra.run.smithi123.stdout: "last_purged_snaps_scrub": "2020-01-16T00:12:12.564547+0000", 2020-01-16T00:12:51.455 INFO:teuthology.orchestra.run.smithi123.stdout: "dead_epoch": 0 2020-01-16T00:12:51.456 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.456 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.456 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": 2, 2020-01-16T00:12:51.456 INFO:teuthology.orchestra.run.smithi123.stdout: "down_stamp": "0.000000", 2020-01-16T00:12:51.456 INFO:teuthology.orchestra.run.smithi123.stdout: "laggy_probability": 0, 2020-01-16T00:12:51.456 INFO:teuthology.orchestra.run.smithi123.stdout: "laggy_interval": 0, 2020-01-16T00:12:51.456 INFO:teuthology.orchestra.run.smithi123.stdout: "features": 4540138292836696063, 2020-01-16T00:12:51.456 INFO:teuthology.orchestra.run.smithi123.stdout: "old_weight": 0, 2020-01-16T00:12:51.457 INFO:teuthology.orchestra.run.smithi123.stdout: "last_purged_snaps_scrub": "2020-01-16T00:12:12.383046+0000", 2020-01-16T00:12:51.457 INFO:teuthology.orchestra.run.smithi123.stdout: "dead_epoch": 0 2020-01-16T00:12:51.457 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.457 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.457 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": 3, 2020-01-16T00:12:51.457 INFO:teuthology.orchestra.run.smithi123.stdout: "down_stamp": "0.000000", 2020-01-16T00:12:51.457 INFO:teuthology.orchestra.run.smithi123.stdout: "laggy_probability": 0, 2020-01-16T00:12:51.458 INFO:teuthology.orchestra.run.smithi123.stdout: "laggy_interval": 0, 2020-01-16T00:12:51.458 INFO:teuthology.orchestra.run.smithi123.stdout: "features": 4540138292836696063, 2020-01-16T00:12:51.458 INFO:teuthology.orchestra.run.smithi123.stdout: "old_weight": 0, 2020-01-16T00:12:51.459 INFO:teuthology.orchestra.run.smithi123.stdout: "last_purged_snaps_scrub": "2020-01-16T00:12:12.619027+0000", 2020-01-16T00:12:51.459 INFO:teuthology.orchestra.run.smithi123.stdout: "dead_epoch": 0 2020-01-16T00:12:51.459 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.459 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.459 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": 4, 2020-01-16T00:12:51.459 INFO:teuthology.orchestra.run.smithi123.stdout: "down_stamp": "0.000000", 2020-01-16T00:12:51.459 INFO:teuthology.orchestra.run.smithi123.stdout: "laggy_probability": 0, 2020-01-16T00:12:51.460 INFO:teuthology.orchestra.run.smithi123.stdout: "laggy_interval": 0, 2020-01-16T00:12:51.460 INFO:teuthology.orchestra.run.smithi123.stdout: "features": 4540138292836696063, 2020-01-16T00:12:51.460 INFO:teuthology.orchestra.run.smithi123.stdout: "old_weight": 0, 2020-01-16T00:12:51.460 INFO:teuthology.orchestra.run.smithi123.stdout: "last_purged_snaps_scrub": "2020-01-16T00:12:12.389288+0000", 2020-01-16T00:12:51.460 INFO:teuthology.orchestra.run.smithi123.stdout: "dead_epoch": 0 2020-01-16T00:12:51.460 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.460 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.461 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": 5, 2020-01-16T00:12:51.461 INFO:teuthology.orchestra.run.smithi123.stdout: "down_stamp": "0.000000", 2020-01-16T00:12:51.461 INFO:teuthology.orchestra.run.smithi123.stdout: "laggy_probability": 0, 2020-01-16T00:12:51.461 INFO:teuthology.orchestra.run.smithi123.stdout: "laggy_interval": 0, 2020-01-16T00:12:51.461 INFO:teuthology.orchestra.run.smithi123.stdout: "features": 4540138292836696063, 2020-01-16T00:12:51.461 INFO:teuthology.orchestra.run.smithi123.stdout: "old_weight": 0, 2020-01-16T00:12:51.461 INFO:teuthology.orchestra.run.smithi123.stdout: "last_purged_snaps_scrub": "2020-01-16T00:12:12.614969+0000", 2020-01-16T00:12:51.461 INFO:teuthology.orchestra.run.smithi123.stdout: "dead_epoch": 0 2020-01-16T00:12:51.462 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.462 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.462 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": 6, 2020-01-16T00:12:51.462 INFO:teuthology.orchestra.run.smithi123.stdout: "down_stamp": "0.000000", 2020-01-16T00:12:51.462 INFO:teuthology.orchestra.run.smithi123.stdout: "laggy_probability": 0, 2020-01-16T00:12:51.462 INFO:teuthology.orchestra.run.smithi123.stdout: "laggy_interval": 0, 2020-01-16T00:12:51.462 INFO:teuthology.orchestra.run.smithi123.stdout: "features": 4540138292836696063, 2020-01-16T00:12:51.462 INFO:teuthology.orchestra.run.smithi123.stdout: "old_weight": 0, 2020-01-16T00:12:51.463 INFO:teuthology.orchestra.run.smithi123.stdout: "last_purged_snaps_scrub": "2020-01-16T00:12:12.704423+0000", 2020-01-16T00:12:51.463 INFO:teuthology.orchestra.run.smithi123.stdout: "dead_epoch": 0 2020-01-16T00:12:51.463 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.463 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.463 INFO:teuthology.orchestra.run.smithi123.stdout: "osd": 7, 2020-01-16T00:12:51.463 INFO:teuthology.orchestra.run.smithi123.stdout: "down_stamp": "0.000000", 2020-01-16T00:12:51.463 INFO:teuthology.orchestra.run.smithi123.stdout: "laggy_probability": 0, 2020-01-16T00:12:51.464 INFO:teuthology.orchestra.run.smithi123.stdout: "laggy_interval": 0, 2020-01-16T00:12:51.464 INFO:teuthology.orchestra.run.smithi123.stdout: "features": 4540138292836696063, 2020-01-16T00:12:51.464 INFO:teuthology.orchestra.run.smithi123.stdout: "old_weight": 0, 2020-01-16T00:12:51.464 INFO:teuthology.orchestra.run.smithi123.stdout: "last_purged_snaps_scrub": "2020-01-16T00:12:12.656476+0000", 2020-01-16T00:12:51.464 INFO:teuthology.orchestra.run.smithi123.stdout: "dead_epoch": 0 2020-01-16T00:12:51.464 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.464 INFO:teuthology.orchestra.run.smithi123.stdout: ], 2020-01-16T00:12:51.464 INFO:teuthology.orchestra.run.smithi123.stdout: "pg_upmap": [], 2020-01-16T00:12:51.465 INFO:teuthology.orchestra.run.smithi123.stdout: "pg_upmap_items": [], 2020-01-16T00:12:51.465 INFO:teuthology.orchestra.run.smithi123.stdout: "pg_temp": [], 2020-01-16T00:12:51.465 INFO:teuthology.orchestra.run.smithi123.stdout: "primary_temp": [], 2020-01-16T00:12:51.465 INFO:teuthology.orchestra.run.smithi123.stdout: "blacklist": {}, 2020-01-16T00:12:51.465 INFO:teuthology.orchestra.run.smithi123.stdout: "erasure_code_profiles": { 2020-01-16T00:12:51.465 INFO:teuthology.orchestra.run.smithi123.stdout: "default": { 2020-01-16T00:12:51.465 INFO:teuthology.orchestra.run.smithi123.stdout: "crush-failure-domain": "osd", 2020-01-16T00:12:51.466 INFO:teuthology.orchestra.run.smithi123.stdout: "k": "2", 2020-01-16T00:12:51.466 INFO:teuthology.orchestra.run.smithi123.stdout: "m": "1", 2020-01-16T00:12:51.468 INFO:teuthology.orchestra.run.smithi123.stdout: "plugin": "jerasure", 2020-01-16T00:12:51.469 INFO:teuthology.orchestra.run.smithi123.stdout: "ruleset-failure-domain": "osd", 2020-01-16T00:12:51.469 INFO:teuthology.orchestra.run.smithi123.stdout: "technique": "reed_sol_van" 2020-01-16T00:12:51.469 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.469 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.469 INFO:teuthology.orchestra.run.smithi123.stdout: "removed_snaps_queue": [], 2020-01-16T00:12:51.469 INFO:teuthology.orchestra.run.smithi123.stdout: "new_removed_snaps": [], 2020-01-16T00:12:51.469 INFO:teuthology.orchestra.run.smithi123.stdout: "new_purged_snaps": [], 2020-01-16T00:12:51.470 INFO:teuthology.orchestra.run.smithi123.stdout: "crush_node_flags": {}, 2020-01-16T00:12:51.470 INFO:teuthology.orchestra.run.smithi123.stdout: "device_class_flags": {} 2020-01-16T00:12:51.470 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:51.473 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:12:51.473 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:12:51.474 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5: {} 2020-01-16T00:12:51.474 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:12:51.474 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:12:51.474 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:51.474 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:12:51.475 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:12:51.475 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:12:51.475 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.475 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:9 2.1 2:aa80ea11:::smithi12313934-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [read 453632~1075200 [fadvise_sequential+fadvise_dontneed]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e24)", 2020-01-16T00:12:51.475 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.123601+0000", 2020-01-16T00:12:51.475 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 14.234365930999999, 2020-01-16T00:12:51.475 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041686819999999999, 2020-01-16T00:12:51.475 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:51.476 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:51.476 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:51.476 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.476 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:51.476 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123601+0000", 2020-01-16T00:12:51.476 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:51.477 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.477 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.477 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:51.477 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123601+0000", 2020-01-16T00:12:51.477 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:12:51.477 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.477 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.478 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:51.478 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123600+0000", 2020-01-16T00:12:51.478 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0259999999999996e-06 2020-01-16T00:12:51.478 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.478 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.478 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:51.479 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123609+0000", 2020-01-16T00:12:51.479 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2979999999999992e-06 2020-01-16T00:12:51.479 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.479 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.479 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:51.479 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123617+0000", 2020-01-16T00:12:51.479 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3850000000000002e-06 2020-01-16T00:12:51.479 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.480 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.480 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:51.480 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123626+0000", 2020-01-16T00:12:51.480 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0027674969999999998 2020-01-16T00:12:51.480 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.480 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.480 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:51.480 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.126393+0000", 2020-01-16T00:12:51.481 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013459499999999999 2020-01-16T00:12:51.481 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.481 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.481 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:51.481 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.126528+0000", 2020-01-16T00:12:51.481 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012418900000000001 2020-01-16T00:12:51.482 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.482 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.482 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:51.482 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.127770+0000", 2020-01-16T00:12:51.482 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041686819999999999 2020-01-16T00:12:51.482 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.482 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.483 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.483 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.483 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.483 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:10 2.1 2:aa80ea11:::smithi12313934-49 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [read 1658880~398336 [fadvise_sequential+fadvise_dontneed]] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e24)", 2020-01-16T00:12:51.483 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.123646+0000", 2020-01-16T00:12:51.483 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 14.234320651000001, 2020-01-16T00:12:51.484 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052215789999999996, 2020-01-16T00:12:51.484 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:51.484 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:12:51.484 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:51.484 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.484 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:51.484 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123646+0000", 2020-01-16T00:12:51.485 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:51.485 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.485 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.485 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:51.485 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123646+0000", 2020-01-16T00:12:51.485 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:12:51.485 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.486 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.486 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:51.486 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123645+0000", 2020-01-16T00:12:51.486 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7880000000000003e-06 2020-01-16T00:12:51.486 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.486 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.486 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:51.486 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123651+0000", 2020-01-16T00:12:51.487 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.218e-05 2020-01-16T00:12:51.487 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.487 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.487 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:51.488 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123663+0000", 2020-01-16T00:12:51.488 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0930999999999999e-05 2020-01-16T00:12:51.488 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.488 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.488 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:51.488 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.123674+0000", 2020-01-16T00:12:51.488 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041493830000000004 2020-01-16T00:12:51.489 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.489 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.489 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:51.489 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.127824+0000", 2020-01-16T00:12:51.489 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00025807199999999998 2020-01-16T00:12:51.489 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.489 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.489 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:51.490 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.128082+0000", 2020-01-16T00:12:51.490 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00078618000000000004 2020-01-16T00:12:51.490 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.490 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.490 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:51.490 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.128868+0000", 2020-01-16T00:12:51.490 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052215789999999996 2020-01-16T00:12:51.490 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.491 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.491 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.491 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.491 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.491 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.4.0:112 3.9 e24/20 3:9218e7f1:::smithi12313934-14:head v 24'72, mlcod=23'70)", 2020-01-16T00:12:51.491 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.925123+0000", 2020-01-16T00:12:51.491 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 13.432843689, 2020-01-16T00:12:51.492 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.004957024, 2020-01-16T00:12:51.492 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:51.492 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:51.492 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:51.492 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.492 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:51.492 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.925123+0000", 2020-01-16T00:12:51.492 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:51.493 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.493 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.493 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:51.493 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.925123+0000", 2020-01-16T00:12:51.493 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:51.493 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.493 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.494 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:51.494 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.925121+0000", 2020-01-16T00:12:51.494 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0024713119999999998 2020-01-16T00:12:51.494 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.494 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.494 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:51.494 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927593+0000", 2020-01-16T00:12:51.495 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7319999999999993e-06 2020-01-16T00:12:51.495 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.495 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.495 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:51.495 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927603+0000", 2020-01-16T00:12:51.495 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6600000000000003e-06 2020-01-16T00:12:51.495 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.496 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.496 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:51.496 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927607+0000", 2020-01-16T00:12:51.496 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7138999999999998e-05 2020-01-16T00:12:51.496 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.496 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.496 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:51.496 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927644+0000", 2020-01-16T00:12:51.497 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7068999999999999e-05 2020-01-16T00:12:51.497 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.497 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.497 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:51.497 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927661+0000", 2020-01-16T00:12:51.497 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0023781840000000002 2020-01-16T00:12:51.498 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.498 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.498 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:51.498 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.930040+0000", 2020-01-16T00:12:51.498 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0679000000000001e-05 2020-01-16T00:12:51.498 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.498 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.499 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:51.499 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.930080+0000", 2020-01-16T00:12:51.499 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.004957024 2020-01-16T00:12:51.499 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.499 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.499 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.500 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.500 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.500 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.1.0:162 3.0 e24/20 3:00c146d5:::smithi12313934-18:head v 24'99, mlcod=24'97)", 2020-01-16T00:12:51.500 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.039167+0000", 2020-01-16T00:12:51.500 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 13.318800403999999, 2020-01-16T00:12:51.500 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0049106159999999996, 2020-01-16T00:12:51.501 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:51.501 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:51.501 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:51.501 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.501 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:51.502 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.039167+0000", 2020-01-16T00:12:51.502 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:51.502 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.502 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.503 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:51.503 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.039167+0000", 2020-01-16T00:12:51.503 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999957 2020-01-16T00:12:51.503 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.503 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.504 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:51.504 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.039162+0000", 2020-01-16T00:12:51.504 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000551155 2020-01-16T00:12:51.504 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.504 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.504 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:51.505 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.039713+0000", 2020-01-16T00:12:51.505 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1465999999999999e-05 2020-01-16T00:12:51.505 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.505 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.505 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:51.506 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.039755+0000", 2020-01-16T00:12:51.506 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7122999999999999e-05 2020-01-16T00:12:51.506 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.506 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.506 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:51.506 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.039772+0000", 2020-01-16T00:12:51.506 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6485999999999997e-05 2020-01-16T00:12:51.506 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.507 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.507 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:51.507 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.039868+0000", 2020-01-16T00:12:51.507 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1465999999999996e-05 2020-01-16T00:12:51.507 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.507 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.508 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:51.508 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.039940+0000", 2020-01-16T00:12:51.508 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040611019999999996 2020-01-16T00:12:51.508 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.508 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.508 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:51.508 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.044001+0000", 2020-01-16T00:12:51.509 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6151999999999999e-05 2020-01-16T00:12:51.509 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.509 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.509 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:51.509 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.044077+0000", 2020-01-16T00:12:51.509 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0049106159999999996 2020-01-16T00:12:51.509 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.509 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.510 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.510 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.510 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.510 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:957 3.0 e24/20 3:00c146d5:::smithi12313934-18:head v 24'100, mlcod=24'97)", 2020-01-16T00:12:51.510 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.042105+0000", 2020-01-16T00:12:51.510 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 13.315861654000001, 2020-01-16T00:12:51.510 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0046019770000000002, 2020-01-16T00:12:51.510 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:51.511 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:51.511 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:51.511 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.511 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:51.511 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.042105+0000", 2020-01-16T00:12:51.511 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:51.511 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.511 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.512 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:51.512 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.042105+0000", 2020-01-16T00:12:51.512 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999962 2020-01-16T00:12:51.512 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.512 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.512 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:51.512 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.042101+0000", 2020-01-16T00:12:51.513 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00041373799999999997 2020-01-16T00:12:51.513 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.514 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.514 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:51.514 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.042515+0000", 2020-01-16T00:12:51.514 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.858e-05 2020-01-16T00:12:51.514 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.514 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.514 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:51.515 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.042554+0000", 2020-01-16T00:12:51.515 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5400000000000002e-05 2020-01-16T00:12:51.515 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.515 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.515 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:51.515 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.042569+0000", 2020-01-16T00:12:51.515 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1859999999999994e-05 2020-01-16T00:12:51.515 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.516 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.516 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:51.516 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.042631+0000", 2020-01-16T00:12:51.516 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7620999999999999e-05 2020-01-16T00:12:51.516 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.516 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.516 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:51.517 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.042658+0000", 2020-01-16T00:12:51.517 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040138750000000001 2020-01-16T00:12:51.517 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.517 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.517 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:51.517 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.046672+0000", 2020-01-16T00:12:51.517 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4952e-05 2020-01-16T00:12:51.517 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.518 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.518 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:51.518 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.046707+0000", 2020-01-16T00:12:51.518 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0046019770000000002 2020-01-16T00:12:51.518 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.518 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.518 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.518 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.519 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.519 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:959 3.0 e24/20 3:00c146d5:::smithi12313934-18:head v 24'102, mlcod=24'98)", 2020-01-16T00:12:51.519 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.047434+0000", 2020-01-16T00:12:51.519 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 13.310533406999999, 2020-01-16T00:12:51.519 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0038721860000000001, 2020-01-16T00:12:51.519 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:51.519 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:51.519 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:51.520 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.520 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:51.520 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.047434+0000", 2020-01-16T00:12:51.520 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:51.520 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.520 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.520 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:51.521 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.047434+0000", 2020-01-16T00:12:51.521 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:51.521 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.521 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.521 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:51.521 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.047432+0000", 2020-01-16T00:12:51.521 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00027367899999999998 2020-01-16T00:12:51.521 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.522 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.522 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:51.522 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.047705+0000", 2020-01-16T00:12:51.522 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1781e-05 2020-01-16T00:12:51.522 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.522 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.522 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:51.522 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.047717+0000", 2020-01-16T00:12:51.523 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5249999999999996e-06 2020-01-16T00:12:51.523 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.523 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.523 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:51.523 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.047723+0000", 2020-01-16T00:12:51.523 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7784999999999998e-05 2020-01-16T00:12:51.523 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.523 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.524 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:51.524 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.047760+0000", 2020-01-16T00:12:51.524 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2864999999999999e-05 2020-01-16T00:12:51.524 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.524 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.524 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:51.524 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.047783+0000", 2020-01-16T00:12:51.525 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0034841450000000001 2020-01-16T00:12:51.525 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.525 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.525 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:51.525 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.051268+0000", 2020-01-16T00:12:51.525 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.82e-05 2020-01-16T00:12:51.525 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.525 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.526 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:51.526 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.051306+0000", 2020-01-16T00:12:51.526 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0038721860000000001 2020-01-16T00:12:51.526 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.526 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.526 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.526 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.526 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.527 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:973 3.5 e24/20 3:aef1ce98:::smithi12313934-9:head v 24'53, mlcod=24'51)", 2020-01-16T00:12:51.527 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.305124+0000", 2020-01-16T00:12:51.527 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 13.052842568000001, 2020-01-16T00:12:51.527 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036133279999999999, 2020-01-16T00:12:51.527 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:51.527 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:51.527 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:51.527 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.528 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:51.528 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.305124+0000", 2020-01-16T00:12:51.528 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:51.528 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.528 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.528 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:51.528 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.305124+0000", 2020-01-16T00:12:51.529 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999857 2020-01-16T00:12:51.529 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.529 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.529 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:51.529 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.305110+0000", 2020-01-16T00:12:51.529 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013042819999999999 2020-01-16T00:12:51.530 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.530 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.530 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:51.530 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.306414+0000", 2020-01-16T00:12:51.530 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2325000000000001e-05 2020-01-16T00:12:51.530 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.530 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.531 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:51.531 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.306437+0000", 2020-01-16T00:12:51.531 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1420000000000001e-05 2020-01-16T00:12:51.531 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.531 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.531 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:51.531 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.306448+0000", 2020-01-16T00:12:51.531 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5415999999999997e-05 2020-01-16T00:12:51.532 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.532 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.532 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:51.532 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.306534+0000", 2020-01-16T00:12:51.532 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4882000000000002e-05 2020-01-16T00:12:51.532 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.532 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.533 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:51.533 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.306568+0000", 2020-01-16T00:12:51.533 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021100450000000001 2020-01-16T00:12:51.533 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.533 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.533 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:51.534 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.308678+0000", 2020-01-16T00:12:51.534 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9237999999999998e-05 2020-01-16T00:12:51.534 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.534 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.534 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:51.534 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.308738+0000", 2020-01-16T00:12:51.534 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036133279999999999 2020-01-16T00:12:51.535 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.535 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.535 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.535 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.535 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.535 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.4.0:114 3.9 e24/20 3:9fc687b2:::smithi12313934-44:head v 24'74, mlcod=24'72)", 2020-01-16T00:12:51.535 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:12:51.535 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 13.032688179999999, 2020-01-16T00:12:51.536 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010399429, 2020-01-16T00:12:51.536 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:51.536 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:51.536 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:51.536 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.536 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:51.536 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:12:51.536 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:51.537 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.537 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.537 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:51.537 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:12:51.537 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:12:51.537 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.537 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.538 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:51.538 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325276+0000", 2020-01-16T00:12:51.538 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0046127069999999997 2020-01-16T00:12:51.538 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.538 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.538 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:51.538 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329889+0000", 2020-01-16T00:12:51.539 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3519999999999998e-05 2020-01-16T00:12:51.539 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.539 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.539 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:51.539 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329912+0000", 2020-01-16T00:12:51.539 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4070000000000004e-06 2020-01-16T00:12:51.540 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.540 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.540 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:51.540 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329920+0000", 2020-01-16T00:12:51.540 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8817000000000003e-05 2020-01-16T00:12:51.540 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.541 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.541 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:51.541 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329978+0000", 2020-01-16T00:12:51.541 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9721e-05 2020-01-16T00:12:51.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.542 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.542 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:51.542 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329998+0000", 2020-01-16T00:12:51.542 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056328139999999999 2020-01-16T00:12:51.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.542 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.543 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:51.543 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.335631+0000", 2020-01-16T00:12:51.543 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7170999999999999e-05 2020-01-16T00:12:51.543 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.543 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.543 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:51.544 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.335678+0000", 2020-01-16T00:12:51.544 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010399429 2020-01-16T00:12:51.544 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.544 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.544 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.545 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.545 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:24 2.1 2:8b431f80:::smithi12313934-40 low_tier:head [write 790528~688128] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:12:51.545 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.482303+0000", 2020-01-16T00:12:51.545 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 12.875664084, 2020-01-16T00:12:51.545 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010071853, 2020-01-16T00:12:51.546 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:51.546 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:51.546 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:51.546 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.546 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:51.546 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.482303+0000", 2020-01-16T00:12:51.547 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:51.547 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.547 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.547 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:51.548 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.482303+0000", 2020-01-16T00:12:51.548 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:51.548 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.548 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.548 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:51.548 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.482301+0000", 2020-01-16T00:12:51.548 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031173960000000001 2020-01-16T00:12:51.549 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.550 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.550 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:51.550 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485418+0000", 2020-01-16T00:12:51.550 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9300000000000001e-05 2020-01-16T00:12:51.550 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.551 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.551 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:51.551 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485448+0000", 2020-01-16T00:12:51.551 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2611e-05 2020-01-16T00:12:51.551 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.551 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.551 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:51.552 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485460+0000", 2020-01-16T00:12:51.552 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112487 2020-01-16T00:12:51.552 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.552 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.552 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:51.552 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485573+0000", 2020-01-16T00:12:51.553 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022169299999999999 2020-01-16T00:12:51.553 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.553 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.553 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:51.553 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485794+0000", 2020-01-16T00:12:51.553 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00036287500000000002 2020-01-16T00:12:51.553 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.554 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.554 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:12:51.554 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.486157+0000", 2020-01-16T00:12:51.554 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040260310000000002 2020-01-16T00:12:51.554 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.554 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.555 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:51.555 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.490183+0000", 2020-01-16T00:12:51.555 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00081083000000000001 2020-01-16T00:12:51.555 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.555 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.555 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:51.555 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.490994+0000", 2020-01-16T00:12:51.556 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001287746 2020-01-16T00:12:51.556 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.556 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.556 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:51.556 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492282+0000", 2020-01-16T00:12:51.556 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8810999999999998e-05 2020-01-16T00:12:51.556 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.557 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.557 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:51.557 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492321+0000", 2020-01-16T00:12:51.557 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4095000000000003e-05 2020-01-16T00:12:51.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.557 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.557 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:51.558 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492375+0000", 2020-01-16T00:12:51.558 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010071853 2020-01-16T00:12:51.558 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.558 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.558 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.558 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.558 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.559 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:25 2.1 2:8b431f80:::smithi12313934-40 low_tier:head [write 1926656~67072] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:12:51.559 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.485481+0000", 2020-01-16T00:12:51.559 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 12.872485619000001, 2020-01-16T00:12:51.559 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0081599020000000001, 2020-01-16T00:12:51.559 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:51.559 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:51.559 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:51.560 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.560 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:51.560 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485481+0000", 2020-01-16T00:12:51.560 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:51.560 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.560 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.560 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:51.561 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485481+0000", 2020-01-16T00:12:51.561 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:51.561 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.561 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.561 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:51.561 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485479+0000", 2020-01-16T00:12:51.562 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012423800000000001 2020-01-16T00:12:51.562 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.562 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.562 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:51.562 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485603+0000", 2020-01-16T00:12:51.562 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8366999999999999e-05 2020-01-16T00:12:51.562 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.563 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.563 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:51.563 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485621+0000", 2020-01-16T00:12:51.563 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0207000000000001e-05 2020-01-16T00:12:51.563 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.563 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.563 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:51.563 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485632+0000", 2020-01-16T00:12:51.564 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002868532 2020-01-16T00:12:51.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.564 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.564 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:51.564 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.488500+0000", 2020-01-16T00:12:51.564 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011340000000000001 2020-01-16T00:12:51.565 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.565 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.565 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:51.565 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.488614+0000", 2020-01-16T00:12:51.565 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019056300000000001 2020-01-16T00:12:51.565 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.565 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.566 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:12:51.566 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.488804+0000", 2020-01-16T00:12:51.566 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00153302 2020-01-16T00:12:51.566 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.566 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.566 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:51.567 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.490337+0000", 2020-01-16T00:12:51.567 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00089946100000000005 2020-01-16T00:12:51.567 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.567 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.567 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:51.567 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.491237+0000", 2020-01-16T00:12:51.567 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0022862820000000002 2020-01-16T00:12:51.568 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.568 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.568 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:51.568 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.493523+0000", 2020-01-16T00:12:51.568 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5609999999999998e-05 2020-01-16T00:12:51.568 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.569 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.569 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:51.569 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.493578+0000", 2020-01-16T00:12:51.569 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2770999999999999e-05 2020-01-16T00:12:51.569 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.569 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.570 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:51.570 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.493641+0000", 2020-01-16T00:12:51.570 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0081599020000000001 2020-01-16T00:12:51.570 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.570 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.570 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.571 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.571 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.571 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:1279 3.0 e26/20 3:05b2bf53:::smithi12313934-37:head v 26'137, mlcod=26'135)", 2020-01-16T00:12:51.571 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:43.082160+0000", 2020-01-16T00:12:51.571 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 8.2758068859999998, 2020-01-16T00:12:51.571 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0091277130000000008, 2020-01-16T00:12:51.571 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:51.572 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:51.572 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:51.572 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.572 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:51.572 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.082160+0000", 2020-01-16T00:12:51.572 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:51.572 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.573 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.573 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:51.573 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.082160+0000", 2020-01-16T00:12:51.573 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999957 2020-01-16T00:12:51.573 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.573 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.573 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:51.574 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.082156+0000", 2020-01-16T00:12:51.574 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00046771999999999997 2020-01-16T00:12:51.574 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.574 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.574 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:51.574 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.082623+0000", 2020-01-16T00:12:51.574 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3099999999999998e-05 2020-01-16T00:12:51.575 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.575 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.575 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:51.575 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.082656+0000", 2020-01-16T00:12:51.575 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2063e-05 2020-01-16T00:12:51.575 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.576 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.576 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:51.576 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.082668+0000", 2020-01-16T00:12:51.576 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021980900000000002 2020-01-16T00:12:51.576 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.576 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.576 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:51.577 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.084867+0000", 2020-01-16T00:12:51.577 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00033252299999999999 2020-01-16T00:12:51.577 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.577 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.577 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:51.577 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.085199+0000", 2020-01-16T00:12:51.578 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0060364019999999997 2020-01-16T00:12:51.578 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.578 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.578 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:51.578 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.091235+0000", 2020-01-16T00:12:51.579 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2255e-05 2020-01-16T00:12:51.579 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.579 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.579 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:51.579 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.091288+0000", 2020-01-16T00:12:51.579 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0091277130000000008 2020-01-16T00:12:51.580 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.580 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.580 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.580 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.580 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.580 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:1280 3.0 e27/20 3:05b2bf53:::smithi12313934-37:head v 27'138, mlcod=26'135)", 2020-01-16T00:12:51.580 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:43.088262+0000", 2020-01-16T00:12:51.581 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 8.2697053060000005, 2020-01-16T00:12:51.581 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036090330000000002, 2020-01-16T00:12:51.581 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:51.581 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:51.581 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:51.581 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.582 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:51.582 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.088262+0000", 2020-01-16T00:12:51.582 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:51.582 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.582 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.582 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:51.583 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.088262+0000", 2020-01-16T00:12:51.583 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:51.583 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.583 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.583 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:51.583 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.088261+0000", 2020-01-16T00:12:51.583 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000173606 2020-01-16T00:12:51.584 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.584 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.584 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:51.584 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.088435+0000", 2020-01-16T00:12:51.584 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4881e-05 2020-01-16T00:12:51.584 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.584 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.585 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:51.585 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.088450+0000", 2020-01-16T00:12:51.585 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2459999999999992e-06 2020-01-16T00:12:51.585 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.585 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.585 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:51.585 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.088459+0000", 2020-01-16T00:12:51.586 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001007066 2020-01-16T00:12:51.586 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.586 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.586 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:51.586 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.089466+0000", 2020-01-16T00:12:51.586 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6548999999999999e-05 2020-01-16T00:12:51.586 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.587 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.587 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:51.587 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.089502+0000", 2020-01-16T00:12:51.587 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0023328870000000001 2020-01-16T00:12:51.587 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.587 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.587 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:51.588 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.091835+0000", 2020-01-16T00:12:51.588 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.54e-05 2020-01-16T00:12:51.588 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.588 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.588 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:51.588 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.091871+0000", 2020-01-16T00:12:51.588 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036090330000000002 2020-01-16T00:12:51.589 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.589 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.589 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.589 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.589 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.589 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:52 2.1 2:9200d056:::smithi12313934-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 926720~423424] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e28)", 2020-01-16T00:12:51.590 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:46.629755+0000", 2020-01-16T00:12:51.590 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.7282122280000003, 2020-01-16T00:12:51.590 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063571069999999999, 2020-01-16T00:12:51.590 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:51.590 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:51.590 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:51.590 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.591 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:51.591 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.629755+0000", 2020-01-16T00:12:51.591 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:51.591 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.591 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.591 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:51.591 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.629755+0000", 2020-01-16T00:12:51.592 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:51.592 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.592 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.592 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:51.592 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.629753+0000", 2020-01-16T00:12:51.592 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000127501 2020-01-16T00:12:51.593 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.593 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.593 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:51.593 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.629880+0000", 2020-01-16T00:12:51.593 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2235000000000001e-05 2020-01-16T00:12:51.593 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.594 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.594 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:51.594 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.629892+0000", 2020-01-16T00:12:51.594 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6879999999999998e-06 2020-01-16T00:12:51.594 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.594 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.594 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:51.595 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.629897+0000", 2020-01-16T00:12:51.595 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000103049 2020-01-16T00:12:51.595 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.595 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.595 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:51.595 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630000+0000", 2020-01-16T00:12:51.596 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000250289 2020-01-16T00:12:51.596 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.596 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.596 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:51.596 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630250+0000", 2020-01-16T00:12:51.596 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019101 2020-01-16T00:12:51.597 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.597 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.597 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:12:51.597 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630441+0000", 2020-01-16T00:12:51.597 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0026126909999999999 2020-01-16T00:12:51.597 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.597 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.598 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:51.598 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.633054+0000", 2020-01-16T00:12:51.598 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0026195569999999998 2020-01-16T00:12:51.598 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.598 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.598 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:51.598 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.635674+0000", 2020-01-16T00:12:51.599 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020174 2020-01-16T00:12:51.599 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.599 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.599 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:51.599 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.635875+0000", 2020-01-16T00:12:51.599 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013694400000000001 2020-01-16T00:12:51.599 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.600 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.600 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:51.600 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.636012+0000", 2020-01-16T00:12:51.600 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9442000000000001e-05 2020-01-16T00:12:51.600 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.600 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.601 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:51.601 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.636112+0000", 2020-01-16T00:12:51.601 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063571069999999999 2020-01-16T00:12:51.601 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.601 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.601 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.601 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.602 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.602 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:53 2.1 2:9200d056:::smithi12313934-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 1823744~153600] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e28)", 2020-01-16T00:12:51.602 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:46.630287+0000", 2020-01-16T00:12:51.602 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.7276799409999999, 2020-01-16T00:12:51.602 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0059987900000000004, 2020-01-16T00:12:51.602 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:51.602 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:51.603 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:51.603 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.603 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:51.603 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630287+0000", 2020-01-16T00:12:51.603 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:51.603 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.603 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.604 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:51.604 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630287+0000", 2020-01-16T00:12:51.604 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:12:51.604 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.604 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.604 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:51.605 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630286+0000", 2020-01-16T00:12:51.605 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7387999999999999e-05 2020-01-16T00:12:51.605 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.605 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.605 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:51.605 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630353+0000", 2020-01-16T00:12:51.605 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.219e-05 2020-01-16T00:12:51.606 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.606 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.606 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:51.606 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630366+0000", 2020-01-16T00:12:51.606 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9020000000000001e-06 2020-01-16T00:12:51.606 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.606 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.606 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:51.607 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630369+0000", 2020-01-16T00:12:51.607 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001257942 2020-01-16T00:12:51.607 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.607 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.607 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:51.607 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.631627+0000", 2020-01-16T00:12:51.608 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000110399 2020-01-16T00:12:51.608 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.608 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.608 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:51.608 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.631738+0000", 2020-01-16T00:12:51.608 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020549199999999999 2020-01-16T00:12:51.608 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.609 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.609 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:12:51.609 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.631943+0000", 2020-01-16T00:12:51.609 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0015335290000000001 2020-01-16T00:12:51.609 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.609 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.609 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:51.610 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.633477+0000", 2020-01-16T00:12:51.610 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002315176 2020-01-16T00:12:51.610 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.610 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.610 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:51.610 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.635792+0000", 2020-01-16T00:12:51.610 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00042626900000000001 2020-01-16T00:12:51.611 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.611 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.611 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:51.611 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.636218+0000", 2020-01-16T00:12:51.611 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9233000000000002e-05 2020-01-16T00:12:51.611 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.612 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.612 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:51.612 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.636258+0000", 2020-01-16T00:12:51.612 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8257999999999999e-05 2020-01-16T00:12:51.612 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.612 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.612 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:51.613 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.636286+0000", 2020-01-16T00:12:51.613 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0059987900000000004 2020-01-16T00:12:51.613 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.613 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.613 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.613 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.613 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.614 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:54 2.1 2:9200d056:::smithi12313934-35 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 2168320~226816] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e28)", 2020-01-16T00:12:51.614 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:46.630988+0000", 2020-01-16T00:12:51.614 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.7269790450000002, 2020-01-16T00:12:51.614 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0062526489999999999, 2020-01-16T00:12:51.615 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:51.615 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:51.615 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:51.615 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.615 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:51.615 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630988+0000", 2020-01-16T00:12:51.616 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:51.616 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.616 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.616 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:51.616 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630988+0000", 2020-01-16T00:12:51.616 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:51.616 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.617 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.617 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:51.617 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630987+0000", 2020-01-16T00:12:51.617 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5952000000000006e-05 2020-01-16T00:12:51.617 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.617 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.618 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:51.618 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.631073+0000", 2020-01-16T00:12:51.618 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.172e-05 2020-01-16T00:12:51.618 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.618 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.618 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:51.618 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.631084+0000", 2020-01-16T00:12:51.619 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9619999999999993e-06 2020-01-16T00:12:51.619 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.619 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.619 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:51.619 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.631092+0000", 2020-01-16T00:12:51.619 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002514877 2020-01-16T00:12:51.619 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.620 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.620 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:51.620 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.633607+0000", 2020-01-16T00:12:51.620 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1843999999999999e-05 2020-01-16T00:12:51.620 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.620 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.620 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:51.621 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.633669+0000", 2020-01-16T00:12:51.621 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010064999999999999 2020-01-16T00:12:51.621 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.621 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.621 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:12:51.621 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.633770+0000", 2020-01-16T00:12:51.621 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0025793230000000001 2020-01-16T00:12:51.622 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.622 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.622 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:51.622 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.636349+0000", 2020-01-16T00:12:51.622 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013228400000000001 2020-01-16T00:12:51.622 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.623 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.623 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:51.623 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.636481+0000", 2020-01-16T00:12:51.623 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00069413899999999998 2020-01-16T00:12:51.623 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.623 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.623 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:51.623 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.637175+0000", 2020-01-16T00:12:51.624 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5463999999999998e-05 2020-01-16T00:12:51.624 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.624 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.624 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:51.624 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.637211+0000", 2020-01-16T00:12:51.624 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9675e-05 2020-01-16T00:12:51.624 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.625 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.625 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:51.625 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.637241+0000", 2020-01-16T00:12:51.625 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0062526489999999999 2020-01-16T00:12:51.625 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.625 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.625 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.626 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.626 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.626 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:38 2.1 2:8b431f80:::smithi12313934-40 low_tier:head [write 790528~688128] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e28)", 2020-01-16T00:12:51.626 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:46.654720+0000", 2020-01-16T00:12:51.626 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.7032466099999999, 2020-01-16T00:12:51.626 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0061792729999999999, 2020-01-16T00:12:51.626 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:51.627 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:51.627 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:51.627 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.627 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:51.627 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.654720+0000", 2020-01-16T00:12:51.627 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:51.627 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.628 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.628 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:51.628 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.654720+0000", 2020-01-16T00:12:51.628 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:51.628 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.628 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.628 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:51.629 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.654719+0000", 2020-01-16T00:12:51.629 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0016889749999999999 2020-01-16T00:12:51.629 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.629 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.629 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:51.629 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.656408+0000", 2020-01-16T00:12:51.629 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1197e-05 2020-01-16T00:12:51.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.630 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.630 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:51.630 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.656419+0000", 2020-01-16T00:12:51.630 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3610000000000003e-06 2020-01-16T00:12:51.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.630 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.631 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:51.631 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.656423+0000", 2020-01-16T00:12:51.631 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9363999999999998e-05 2020-01-16T00:12:51.631 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.631 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.631 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:51.631 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.656463+0000", 2020-01-16T00:12:51.632 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011794 2020-01-16T00:12:51.632 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.632 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.632 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:51.632 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.656581+0000", 2020-01-16T00:12:51.632 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000115374 2020-01-16T00:12:51.632 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.633 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.633 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:12:51.633 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.656696+0000", 2020-01-16T00:12:51.633 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0038889340000000001 2020-01-16T00:12:51.633 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.633 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.633 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:51.634 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.660585+0000", 2020-01-16T00:12:51.634 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000108517 2020-01-16T00:12:51.634 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.634 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.634 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:51.634 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.660694+0000", 2020-01-16T00:12:51.634 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.1643000000000003e-05 2020-01-16T00:12:51.634 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.635 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.635 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:51.635 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.660785+0000", 2020-01-16T00:12:51.635 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7000000000000003e-05 2020-01-16T00:12:51.635 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.635 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.635 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:51.636 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.660842+0000", 2020-01-16T00:12:51.636 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7374000000000003e-05 2020-01-16T00:12:51.636 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.636 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.636 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:51.636 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.660900+0000", 2020-01-16T00:12:51.636 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0061792729999999999 2020-01-16T00:12:51.637 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.637 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.637 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.637 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.637 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.637 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:39 2.1 2:8b431f80:::smithi12313934-40 low_tier:head [write 1926656~67072] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e28)", 2020-01-16T00:12:51.637 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:46.656432+0000", 2020-01-16T00:12:51.638 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.7015345719999999, 2020-01-16T00:12:51.638 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0046525220000000001, 2020-01-16T00:12:51.638 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:51.638 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:51.638 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:51.639 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.639 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:51.640 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.656432+0000", 2020-01-16T00:12:51.640 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:51.640 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.640 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.640 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:51.640 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.656432+0000", 2020-01-16T00:12:51.641 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:51.641 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.641 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.641 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:51.641 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.656432+0000", 2020-01-16T00:12:51.641 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0343999999999999e-05 2020-01-16T00:12:51.641 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.642 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.642 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:51.642 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.656462+0000", 2020-01-16T00:12:51.642 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3470000000000005e-06 2020-01-16T00:12:51.642 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.642 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.642 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:51.643 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.656471+0000", 2020-01-16T00:12:51.643 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.157e-06 2020-01-16T00:12:51.643 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.643 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.643 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:51.643 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.656475+0000", 2020-01-16T00:12:51.643 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001008236 2020-01-16T00:12:51.643 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.644 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.644 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:51.644 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.657483+0000", 2020-01-16T00:12:51.644 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010872 2020-01-16T00:12:51.644 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.644 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.644 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:51.645 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.657592+0000", 2020-01-16T00:12:51.645 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017760900000000001 2020-01-16T00:12:51.645 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.645 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.645 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:12:51.645 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.657769+0000", 2020-01-16T00:12:51.645 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031973909999999999 2020-01-16T00:12:51.646 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.646 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.646 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:51.646 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.660967+0000", 2020-01-16T00:12:51.646 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5121e-05 2020-01-16T00:12:51.646 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.646 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.646 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:51.647 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.660982+0000", 2020-01-16T00:12:51.647 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1785999999999997e-05 2020-01-16T00:12:51.647 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.647 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.647 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:51.647 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.661013+0000", 2020-01-16T00:12:51.648 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1900999999999997e-05 2020-01-16T00:12:51.648 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.648 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.648 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:51.648 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.661055+0000", 2020-01-16T00:12:51.648 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.953e-05 2020-01-16T00:12:51.649 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.650 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.650 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:51.650 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.661085+0000", 2020-01-16T00:12:51.650 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0046525220000000001 2020-01-16T00:12:51.650 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.650 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.650 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.650 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.651 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.651 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:1441 3.5 e28/20 3:a269d636:::smithi12313934-32:head v 28'68, mlcod=24'66)", 2020-01-16T00:12:51.651 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:47.908644+0000", 2020-01-16T00:12:51.651 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 3.4493226699999999, 2020-01-16T00:12:51.651 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0051439329999999998, 2020-01-16T00:12:51.651 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:51.651 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:51.652 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:51.652 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.652 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:51.652 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.908644+0000", 2020-01-16T00:12:51.652 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:51.652 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.652 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.652 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:51.653 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.908644+0000", 2020-01-16T00:12:51.653 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2020-01-16T00:12:51.653 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.653 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.653 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:51.653 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.908641+0000", 2020-01-16T00:12:51.653 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036604150000000002 2020-01-16T00:12:51.654 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.654 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.654 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:51.654 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.912301+0000", 2020-01-16T00:12:51.654 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6002999999999999e-05 2020-01-16T00:12:51.654 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.654 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.655 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:51.655 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.912317+0000", 2020-01-16T00:12:51.655 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2259999999999996e-06 2020-01-16T00:12:51.655 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.655 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.655 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:51.655 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.912323+0000", 2020-01-16T00:12:51.656 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6557999999999997e-05 2020-01-16T00:12:51.656 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.656 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.656 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:51.656 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.912369+0000", 2020-01-16T00:12:51.656 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6648999999999998e-05 2020-01-16T00:12:51.656 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.657 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.657 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:51.657 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.912396+0000", 2020-01-16T00:12:51.657 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013523249999999999 2020-01-16T00:12:51.657 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.657 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.658 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:51.658 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.913748+0000", 2020-01-16T00:12:51.658 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9956999999999997e-05 2020-01-16T00:12:51.658 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.658 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.658 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:51.658 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.913788+0000", 2020-01-16T00:12:51.659 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0051439329999999998 2020-01-16T00:12:51.659 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.659 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.659 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.659 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.659 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.659 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:47 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e28)", 2020-01-16T00:12:51.659 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:47.941839+0000", 2020-01-16T00:12:51.660 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 3.4161277650000001, 2020-01-16T00:12:51.660 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.008400154, 2020-01-16T00:12:51.660 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:51.660 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:51.660 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:51.660 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.660 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:51.661 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.941839+0000", 2020-01-16T00:12:51.661 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:51.661 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.661 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.661 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:51.661 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.941839+0000", 2020-01-16T00:12:51.661 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:51.661 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.662 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.662 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:51.662 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.941837+0000", 2020-01-16T00:12:51.662 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00046009399999999998 2020-01-16T00:12:51.662 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.662 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.662 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:51.663 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942297+0000", 2020-01-16T00:12:51.663 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8628999999999999e-05 2020-01-16T00:12:51.663 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.663 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.663 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:51.663 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942335+0000", 2020-01-16T00:12:51.664 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1729999999999998e-06 2020-01-16T00:12:51.664 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.664 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.664 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:51.664 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942343+0000", 2020-01-16T00:12:51.664 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4845999999999997e-05 2020-01-16T00:12:51.664 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.664 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.665 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:51.665 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942387+0000", 2020-01-16T00:12:51.665 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6398999999999999e-05 2020-01-16T00:12:51.665 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.665 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.665 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:51.665 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942454+0000", 2020-01-16T00:12:51.666 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010786 2020-01-16T00:12:51.666 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.666 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.666 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:12:51.666 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942562+0000", 2020-01-16T00:12:51.666 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0064317760000000002 2020-01-16T00:12:51.666 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.666 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.667 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:51.667 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.948993+0000", 2020-01-16T00:12:51.667 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0011048849999999999 2020-01-16T00:12:51.667 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.667 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.667 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:51.668 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.950098+0000", 2020-01-16T00:12:51.668 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7064000000000003e-05 2020-01-16T00:12:51.668 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.668 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.668 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:51.668 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.950135+0000", 2020-01-16T00:12:51.668 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6589000000000003e-05 2020-01-16T00:12:51.669 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.669 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.669 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:51.669 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.950192+0000", 2020-01-16T00:12:51.669 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7286000000000001e-05 2020-01-16T00:12:51.669 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.669 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.670 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:51.670 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.950239+0000", 2020-01-16T00:12:51.670 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.008400154 2020-01-16T00:12:51.670 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.670 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.670 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.670 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.670 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.671 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:48 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e28)", 2020-01-16T00:12:51.671 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:47.942354+0000", 2020-01-16T00:12:51.672 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 3.4156133510000002, 2020-01-16T00:12:51.672 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0080319870000000008, 2020-01-16T00:12:51.673 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:12:51.673 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:51.673 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:12:51.673 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.673 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:12:51.673 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942354+0000", 2020-01-16T00:12:51.673 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:12:51.673 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.674 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.674 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:12:51.674 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942354+0000", 2020-01-16T00:12:51.674 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:51.674 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.674 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.674 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:12:51.674 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942353+0000", 2020-01-16T00:12:51.675 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5825000000000002e-05 2020-01-16T00:12:51.675 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.675 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.675 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:12:51.675 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942409+0000", 2020-01-16T00:12:51.675 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5929999999999996e-06 2020-01-16T00:12:51.675 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.676 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.676 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:12:51.676 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942414+0000", 2020-01-16T00:12:51.676 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4199999999999999e-06 2020-01-16T00:12:51.676 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.676 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.676 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:12:51.676 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942418+0000", 2020-01-16T00:12:51.677 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0027594500000000001 2020-01-16T00:12:51.677 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.677 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.677 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:12:51.677 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.945177+0000", 2020-01-16T00:12:51.677 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.4040000000000002e-05 2020-01-16T00:12:51.677 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.678 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.678 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:12:51.678 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.945261+0000", 2020-01-16T00:12:51.678 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012906699999999999 2020-01-16T00:12:51.678 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.678 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.679 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:12:51.679 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.945390+0000", 2020-01-16T00:12:51.679 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0047161060000000003 2020-01-16T00:12:51.679 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.679 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.679 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:12:51.679 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.950106+0000", 2020-01-16T00:12:51.679 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016987400000000001 2020-01-16T00:12:51.680 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.680 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.680 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:51.680 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.950276+0000", 2020-01-16T00:12:51.680 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6329000000000002e-05 2020-01-16T00:12:51.680 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.680 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.681 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:51.681 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.950303+0000", 2020-01-16T00:12:51.681 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1443000000000003e-05 2020-01-16T00:12:51.681 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.681 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.681 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:12:51.681 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.950344+0000", 2020-01-16T00:12:51.681 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1486000000000002e-05 2020-01-16T00:12:51.682 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:12:51.682 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:12:51.682 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:12:51.682 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.950386+0000", 2020-01-16T00:12:51.682 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0080319870000000008 2020-01-16T00:12:51.682 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.682 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.683 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.683 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:12:51.683 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:12:51.683 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:51.683 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-01-16T00:12:51.684 INFO:tasks.rados.rados.0.smithi123.stdout:832: expect (ObjNum 43 snap 0 seq_num 43) 2020-01-16T00:12:51.684 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-01-16T00:12:51.684 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-01-16T00:12:51.684 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-01-16T00:12:51.684 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-01-16T00:12:51.685 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-01-16T00:12:51.685 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-01-16T00:12:51.685 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-01-16T00:12:51.685 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-01-16T00:12:51.685 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:51.471+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:12:51.686 INFO:tasks.thrashosds.thrasher:j is {u'nearfull_ratio': 0.8500000238418579, u'backfillfull_ratio': 0.8999999761581421, u'last_in_change': u'2020-01-16T00:12:13.968282+0000', u'full_ratio': 0.949999988079071, u'cluster_snapshot': u'', u'pg_upmap': [], u'new_purged_snaps': [], u'erasure_code_profiles': {u'default': {u'plugin': u'jerasure', u'crush-failure-domain': u'osd', u'k': u'2', u'technique': u'reed_sol_van', u'm': u'1', u'ruleset-failure-domain': u'osd'}}, u'osds': [{u'heartbeat_back_addr': u'172.21.15.123:6812/11906', u'uuid': u'4b8d1e40-95e3-4adc-9929-cb7e6cc8b766', u'weight': 1, u'up_from': 13, u'cluster_addr': u'172.21.15.123:6810/11906', u'heartbeat_front_addr': u'172.21.15.123:6811/11906', u'down_at': 0, u'public_addrs': {u'addrvec': [{u'nonce': 11906, u'type': u'v1', u'addr': u'172.21.15.123:6809'}]}, u'up': 1, u'cluster_addrs': {u'addrvec': [{u'nonce': 11906, u'type': u'v1', u'addr': u'172.21.15.123:6810'}]}, u'lost_at': 0, u'primary_affinity': 1, u'state': [u'exists', u'up'], u'last_clean_begin': 0, u'last_clean_end': 0, u'in': 1, u'public_addr': u'172.21.15.123:6809/11906', u'up_thru': 20, u'heartbeat_back_addrs': {u'addrvec': [{u'nonce': 11906, u'type': u'v1', u'addr': u'172.21.15.123:6812'}]}, u'heartbeat_front_addrs': {u'addrvec': [{u'nonce': 11906, u'type': u'v1', u'addr': u'172.21.15.123:6811'}]}, u'osd': 0}, {u'heartbeat_back_addr': u'172.21.15.123:6808/11905', u'uuid': u'2e373d15-6348-4caa-aa9f-b46037376a3b', u'weight': 1, u'up_from': 13, u'cluster_addr': u'172.21.15.123:6806/11905', u'heartbeat_front_addr': u'172.21.15.123:6807/11905', u'down_at': 0, u'public_addrs': {u'addrvec': [{u'nonce': 11905, u'type': u'v1', u'addr': u'172.21.15.123:6805'}]}, u'up': 1, u'cluster_addrs': {u'addrvec': [{u'nonce': 11905, u'type': u'v1', u'addr': u'172.21.15.123:6806'}]}, u'lost_at': 0, u'primary_affinity': 1, u'state': [u'exists', u'up'], u'last_clean_begin': 0, u'last_clean_end': 0, u'in': 1, u'public_addr': u'172.21.15.123:6805/11905', u'up_thru': 20, u'heartbeat_back_addrs': {u'addrvec': [{u'nonce': 11905, u'type': u'v1', u'addr': u'172.21.15.123:6808'}]}, u'heartbeat_front_addrs': {u'addrvec': [{u'nonce': 11905, u'type': u'v1', u'addr': u'172.21.15.123:6807'}]}, u'osd': 1}, {u'heartbeat_back_addr': u'172.21.15.123:6804/11907', u'uuid': u'5736ff2c-076a-4d41-8e11-06e2fd1f5f94', u'weight': 1, u'up_from': 13, u'cluster_addr': u'172.21.15.123:6802/11907', u'heartbeat_front_addr': u'172.21.15.123:6803/11907', u'down_at': 0, u'public_addrs': {u'addrvec': [{u'nonce': 11907, u'type': u'v1', u'addr': u'172.21.15.123:6801'}]}, u'up': 1, u'cluster_addrs': {u'addrvec': [{u'nonce': 11907, u'type': u'v1', u'addr': u'172.21.15.123:6802'}]}, u'lost_at': 0, u'primary_affinity': 1, u'state': [u'exists', u'up'], u'last_clean_begin': 0, u'last_clean_end': 0, u'in': 1, u'public_addr': u'172.21.15.123:6801/11907', u'up_thru': 19, u'heartbeat_back_addrs': {u'addrvec': [{u'nonce': 11907, u'type': u'v1', u'addr': u'172.21.15.123:6804'}]}, u'heartbeat_front_addrs': {u'addrvec': [{u'nonce': 11907, u'type': u'v1', u'addr': u'172.21.15.123:6803'}]}, u'osd': 2}, {u'heartbeat_back_addr': u'172.21.15.123:6816/11917', u'uuid': u'554f53a8-c5b1-43c9-a7d6-93629d84f1ea', u'weight': 1, u'up_from': 13, u'cluster_addr': u'172.21.15.123:6814/11917', u'heartbeat_front_addr': u'172.21.15.123:6815/11917', u'down_at': 0, u'public_addrs': {u'addrvec': [{u'nonce': 11917, u'type': u'v1', u'addr': u'172.21.15.123:6813'}]}, u'up': 1, u'cluster_addrs': {u'addrvec': [{u'nonce': 11917, u'type': u'v1', u'addr': u'172.21.15.123:6814'}]}, u'lost_at': 0, u'primary_affinity': 1, u'state': [u'exists', u'up'], u'last_clean_begin': 0, u'last_clean_end': 0, u'in': 1, u'public_addr': u'172.21.15.123:6813/11917', u'up_thru': 20, u'heartbeat_back_addrs': {u'addrvec': [{u'nonce': 11917, u'type': u'v1', u'addr': u'172.21.15.123:6816'}]}, u'heartbeat_front_addrs': {u'addrvec': [{u'nonce': 11917, u'type': u'v1', u'addr': u'172.21.15.123:6815'}]}, u'osd': 3}, {u'heartbeat_back_addr': u'172.21.15.145:6803/11851', u'uuid': u'77945419-c04b-4071-ab87-256d77640dc0', u'weight': 1, u'up_from': 13, u'cluster_addr': u'172.21.15.145:6801/11851', u'heartbeat_front_addr': u'172.21.15.145:6802/11851', u'down_at': 0, u'public_addrs': {u'addrvec': [{u'nonce': 11851, u'type': u'v1', u'addr': u'172.21.15.145:6800'}]}, u'up': 1, u'cluster_addrs': {u'addrvec': [{u'nonce': 11851, u'type': u'v1', u'addr': u'172.21.15.145:6801'}]}, u'lost_at': 0, u'primary_affinity': 1, u'state': [u'exists', u'up'], u'last_clean_begin': 0, u'last_clean_end': 0, u'in': 1, u'public_addr': u'172.21.15.145:6800/11851', u'up_thru': 20, u'heartbeat_back_addrs': {u'addrvec': [{u'nonce': 11851, u'type': u'v1', u'addr': u'172.21.15.145:6803'}]}, u'heartbeat_front_addrs': {u'addrvec': [{u'nonce': 11851, u'type': u'v1', u'addr': u'172.21.15.145:6802'}]}, u'osd': 4}, {u'heartbeat_back_addr': u'172.21.15.145:6807/11860', u'uuid': u'd5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4', u'weight': 1, u'up_from': 13, u'cluster_addr': u'172.21.15.145:6805/11860', u'heartbeat_front_addr': u'172.21.15.145:6806/11860', u'down_at': 0, u'public_addrs': {u'addrvec': [{u'nonce': 11860, u'type': u'v1', u'addr': u'172.21.15.145:6804'}]}, u'up': 1, u'cluster_addrs': {u'addrvec': [{u'nonce': 11860, u'type': u'v1', u'addr': u'172.21.15.145:6805'}]}, u'lost_at': 0, u'primary_affinity': 1, u'state': [u'exists', u'up'], u'last_clean_begin': 0, u'last_clean_end': 0, u'in': 1, u'public_addr': u'172.21.15.145:6804/11860', u'up_thru': 20, u'heartbeat_back_addrs': {u'addrvec': [{u'nonce': 11860, u'type': u'v1', u'addr': u'172.21.15.145:6807'}]}, u'heartbeat_front_addrs': {u'addrvec': [{u'nonce': 11860, u'type': u'v1', u'addr': u'172.21.15.145:6806'}]}, u'osd': 5}, {u'heartbeat_back_addr': u'172.21.15.145:6815/11867', u'uuid': u'e9d342cd-5c2d-40a0-b8ea-24588f280389', u'weight': 1, u'up_from': 13, u'cluster_addr': u'172.21.15.145:6813/11867', u'heartbeat_front_addr': u'172.21.15.145:6814/11867', u'down_at': 0, u'public_addrs': {u'addrvec': [{u'nonce': 11867, u'type': u'v1', u'addr': u'172.21.15.145:6812'}]}, u'up': 1, u'cluster_addrs': {u'addrvec': [{u'nonce': 11867, u'type': u'v1', u'addr': u'172.21.15.145:6813'}]}, u'lost_at': 0, u'primary_affinity': 1, u'state': [u'exists', u'up'], u'last_clean_begin': 0, u'last_clean_end': 0, u'in': 1, u'public_addr': u'172.21.15.145:6812/11867', u'up_thru': 20, u'heartbeat_back_addrs': {u'addrvec': [{u'nonce': 11867, u'type': u'v1', u'addr': u'172.21.15.145:6815'}]}, u'heartbeat_front_addrs': {u'addrvec': [{u'nonce': 11867, u'type': u'v1', u'addr': u'172.21.15.145:6814'}]}, u'osd': 6}, {u'heartbeat_back_addr': u'172.21.15.145:6811/11868', u'uuid': u'6a301b66-384a-40d8-adc8-96fff49e982e', u'weight': 1, u'up_from': 13, u'cluster_addr': u'172.21.15.145:6809/11868', u'heartbeat_front_addr': u'172.21.15.145:6810/11868', u'down_at': 0, u'public_addrs': {u'addrvec': [{u'nonce': 11868, u'type': u'v1', u'addr': u'172.21.15.145:6808'}]}, u'up': 1, u'cluster_addrs': {u'addrvec': [{u'nonce': 11868, u'type': u'v1', u'addr': u'172.21.15.145:6809'}]}, u'lost_at': 0, u'primary_affinity': 1, u'state': [u'exists', u'up'], u'last_clean_begin': 0, u'last_clean_end': 0, u'in': 1, u'public_addr': u'172.21.15.145:6808/11868', u'up_thru': 20, u'heartbeat_back_addrs': {u'addrvec': [{u'nonce': 11868, u'type': u'v1', u'addr': u'172.21.15.145:6811'}]}, u'heartbeat_front_addrs': {u'addrvec': [{u'nonce': 11868, u'type': u'v1', u'addr': u'172.21.15.145:6810'}]}, u'osd': 7}], u'crush_node_flags': {}, u'epoch': 29, u'pg_temp': [], u'require_min_compat_client': u'jewel', u'crush_version': 5, u'primary_temp': [], u'pool_max': 3, u'device_class_flags': {}, u'max_osd': 8, u'require_osd_release': u'octopus', u'min_compat_client': u'jewel', u'osd_xinfo': [{u'laggy_probability': 0, u'laggy_interval': 0, u'features': 4540138292836696063, u'old_weight': 0, u'dead_epoch': 0, u'down_stamp': u'0.000000', u'osd': 0, u'last_purged_snaps_scrub': u'2020-01-16T00:12:12.621038+0000'}, {u'laggy_probability': 0, u'laggy_interval': 0, u'features': 4540138292836696063, u'old_weight': 0, u'dead_epoch': 0, u'down_stamp': u'0.000000', u'osd': 1, u'last_purged_snaps_scrub': u'2020-01-16T00:12:12.564547+0000'}, {u'laggy_probability': 0, u'laggy_interval': 0, u'features': 4540138292836696063, u'old_weight': 0, u'dead_epoch': 0, u'down_stamp': u'0.000000', u'osd': 2, u'last_purged_snaps_scrub': u'2020-01-16T00:12:12.383046+0000'}, {u'laggy_probability': 0, u'laggy_interval': 0, u'features': 4540138292836696063, u'old_weight': 0, u'dead_epoch': 0, u'down_stamp': u'0.000000', u'osd': 3, u'last_purged_snaps_scrub': u'2020-01-16T00:12:12.619027+0000'}, {u'laggy_probability': 0, u'laggy_interval': 0, u'features': 4540138292836696063, u'old_weight': 0, u'dead_epoch': 0, u'down_stamp': u'0.000000', u'osd': 4, u'last_purged_snaps_scrub': u'2020-01-16T00:12:12.389288+0000'}, {u'laggy_probability': 0, u'laggy_interval': 0, u'features': 4540138292836696063, u'old_weight': 0, u'dead_epoch': 0, u'down_stamp': u'0.000000', u'osd': 5, u'last_purged_snaps_scrub': u'2020-01-16T00:12:12.614969+0000'}, {u'laggy_probability': 0, u'laggy_interval': 0, u'features': 4540138292836696063, u'old_weight': 0, u'dead_epoch': 0, u'down_stamp': u'0.000000', u'osd': 6, u'last_purged_snaps_scrub': u'2020-01-16T00:12:12.704423+0000'}, {u'laggy_probability': 0, u'laggy_interval': 0, u'features': 4540138292836696063, u'old_weight': 0, u'dead_epoch': 0, u'down_stamp': u'0.000000', u'osd': 7, u'last_purged_snaps_scrub': u'2020-01-16T00:12:12.656476+0000'}], u'blacklist': {}, u'last_up_change': u'2020-01-16T00:12:13.968282+0000', u'flags_set': [u'nodeep-scrub', u'pglog_hardlimit', u'purged_snapdirs', u'recovery_deletes', u'sortbitwise'], u'fsid': u'864e7191-e3db-4201-b88a-a4873d1eb7fe', u'flags_num': 5804032, u'created': u'2020-01-16T00:12:06.009499+0000', u'modified': u'2020-01-16T00:12:49.011884+0000', u'flags': u'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', u'pg_upmap_items': [], u'pools': [{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2020-01-16T00:12:18.200931+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 3, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 2, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'17', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 4, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2020-01-16T00:12:30.019477+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 3, u'pg_num_pending': 4, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 4, u'pg_num': 4, u'type': 1, u'grade_table': [], u'pool_name': u'low_tier', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 4, u'cache_mode': u'none', u'min_size': 2, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 2, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'21', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 16, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2020-01-16T00:12:31.098841+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 3, u'pg_num_pending': 16, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 16, u'pg_num': 16, u'type': 1, u'grade_table': [], u'pool_name': u'unique_pool_0', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 16, u'cache_mode': u'none', u'min_size': 2, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 3, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rados': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'26', u'min_write_recency_for_promote': 0, u'read_tier': -1}], u'new_removed_snaps': [], u'removed_snaps_queue': []} 2020-01-16T00:12:51.687 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-01-16T00:12:51.688 INFO:tasks.rados.rados.0.smithi123.stdout:834: finishing write tid 1 to smithi12313934-42 2020-01-16T00:12:51.688 INFO:tasks.rados.rados.0.smithi123.stdout:833: expect (ObjNum 90 snap 0 seq_num 90) 2020-01-16T00:12:51.689 INFO:tasks.rados.rados.0.smithi123.stdout:831: expect (ObjNum 143 snap 0 seq_num 143) 2020-01-16T00:12:51.689 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:12:51.689+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:12:51.757 INFO:tasks.rados.rados.0.smithi123.stdout:834: finishing write tid 2 to smithi12313934-42 2020-01-16T00:12:51.757 INFO:tasks.rados.rados.0.smithi123.stdout:835: finishing write tid 1 to smithi12313934-36 2020-01-16T00:12:51.757 INFO:tasks.rados.rados.0.smithi123.stdout:834: finishing write tid 4 to smithi12313934-42 2020-01-16T00:12:51.757 INFO:tasks.rados.rados.0.smithi123.stdout:834: finishing write tid 5 to smithi12313934-42 2020-01-16T00:12:51.757 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 42 v 73 (ObjNum 160 snap 0 seq_num 160) dirty exists 2020-01-16T00:12:51.757 INFO:tasks.rados.rados.0.smithi123.stdout:834: left oid 42 (ObjNum 160 snap 0 seq_num 160) 2020-01-16T00:12:51.758 INFO:tasks.rados.rados.0.smithi123.stdout:835: finishing write tid 2 to smithi12313934-36 2020-01-16T00:12:51.758 INFO:tasks.rados.rados.0.smithi123.stdout:835: finishing write tid 4 to smithi12313934-36 2020-01-16T00:12:51.758 INFO:tasks.rados.rados.0.smithi123.stdout:835: finishing write tid 5 to smithi12313934-36 2020-01-16T00:12:51.758 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 36 v 40 (ObjNum 161 snap 0 seq_num 161) dirty exists 2020-01-16T00:12:51.758 INFO:tasks.rados.rados.0.smithi123.stdout:835: left oid 36 (ObjNum 161 snap 0 seq_num 161) 2020-01-16T00:12:51.758 INFO:tasks.rados.rados.0.smithi123.stdout:836: finishing write tid 1 to smithi12313934-7 2020-01-16T00:12:51.758 INFO:tasks.rados.rados.0.smithi123.stdout:836: finishing write tid 2 to smithi12313934-7 2020-01-16T00:12:51.759 INFO:tasks.rados.rados.0.smithi123.stdout:836: finishing write tid 4 to smithi12313934-7 2020-01-16T00:12:51.759 INFO:tasks.rados.rados.0.smithi123.stdout:836: finishing write tid 5 to smithi12313934-7 2020-01-16T00:12:51.759 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 7 v 105 (ObjNum 162 snap 0 seq_num 162) dirty exists 2020-01-16T00:12:51.759 INFO:tasks.rados.rados.0.smithi123.stdout:836: left oid 7 (ObjNum 162 snap 0 seq_num 162) 2020-01-16T00:12:51.759 INFO:tasks.rados.rados.0.smithi123.stdout:829: done (15 left) 2020-01-16T00:12:51.759 INFO:tasks.rados.rados.0.smithi123.stdout:830: done (14 left) 2020-01-16T00:12:51.759 INFO:tasks.rados.rados.0.smithi123.stdout:831: done (13 left) 2020-01-16T00:12:51.760 INFO:tasks.rados.rados.0.smithi123.stdout:832: done (12 left) 2020-01-16T00:12:51.760 INFO:tasks.rados.rados.0.smithi123.stdout:833: done (11 left) 2020-01-16T00:12:51.760 INFO:tasks.rados.rados.0.smithi123.stdout:834: done (10 left) 2020-01-16T00:12:51.760 INFO:tasks.rados.rados.0.smithi123.stdout:835: done (9 left) 2020-01-16T00:12:51.761 INFO:tasks.rados.rados.0.smithi123.stdout:836: done (8 left) 2020-01-16T00:12:51.761 INFO:tasks.rados.rados.0.smithi123.stdout:837: done (7 left) 2020-01-16T00:12:51.761 INFO:tasks.rados.rados.0.smithi123.stdout:839: done (6 left) 2020-01-16T00:12:51.761 INFO:tasks.rados.rados.0.smithi123.stdout:845: read oid 26 snap -1 2020-01-16T00:12:51.761 INFO:tasks.rados.rados.0.smithi123.stdout:845: expect (ObjNum 147 snap 0 seq_num 147) 2020-01-16T00:12:51.762 INFO:tasks.rados.rados.0.smithi123.stdout:846: read oid 10 snap -1 2020-01-16T00:12:51.762 INFO:tasks.rados.rados.0.smithi123.stdout:846: expect (ObjNum 125 snap 0 seq_num 125) 2020-01-16T00:12:51.762 INFO:tasks.rados.rados.0.smithi123.stdout:847: read oid 46 snap -1 2020-01-16T00:12:51.762 INFO:tasks.rados.rados.0.smithi123.stdout:847: expect (ObjNum 97 snap 0 seq_num 97) 2020-01-16T00:12:51.762 INFO:tasks.rados.rados.0.smithi123.stdout:848: write oid 18 current snap is 0 2020-01-16T00:12:51.762 INFO:tasks.rados.rados.0.smithi123.stdout:848: seq_num 168 ranges {423576=401598,1497916=693725} 2020-01-16T00:12:51.769 INFO:tasks.rados.rados.0.smithi123.stdout:848: writing smithi12313934-18 from 423576 to 825174 tid 1 2020-01-16T00:12:51.776 INFO:teuthology.orchestra.run.smithi123.stderr:nodeep-scrub is unset 2020-01-16T00:12:51.778 INFO:tasks.rados.rados.0.smithi123.stdout:848: writing smithi12313934-18 from 1497916 to 2191641 tid 2 2020-01-16T00:12:51.779 INFO:tasks.rados.rados.0.smithi123.stdout:838: finishing write tid 1 to smithi12313934-48 2020-01-16T00:12:51.779 INFO:tasks.rados.rados.0.smithi123.stdout:838: finishing write tid 2 to smithi12313934-48 2020-01-16T00:12:51.779 INFO:tasks.rados.rados.0.smithi123.stdout:838: finishing write tid 4 to smithi12313934-48 2020-01-16T00:12:51.779 INFO:tasks.rados.rados.0.smithi123.stdout:838: finishing write tid 5 to smithi12313934-48 2020-01-16T00:12:51.779 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 48 v 77 (ObjNum 163 snap 0 seq_num 163) dirty exists 2020-01-16T00:12:51.779 INFO:tasks.rados.rados.0.smithi123.stdout:838: left oid 48 (ObjNum 163 snap 0 seq_num 163) 2020-01-16T00:12:51.780 INFO:tasks.rados.rados.0.smithi123.stdout:840: expect (ObjNum 10 snap 0 seq_num 10) 2020-01-16T00:12:51.789 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:51.789+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:12:51.807 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:51.807 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:12:51.808 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:12:51.808 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:51.816 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-01-16T00:12:51.845 INFO:tasks.rados.rados.0.smithi123.stdout:841: finishing write tid 1 to smithi12313934-30 2020-01-16T00:12:51.845 INFO:tasks.rados.rados.0.smithi123.stdout:841: finishing write tid 2 to smithi12313934-30 2020-01-16T00:12:51.845 INFO:tasks.rados.rados.0.smithi123.stdout:841: finishing write tid 4 to smithi12313934-30 2020-01-16T00:12:51.845 INFO:tasks.rados.rados.0.smithi123.stdout:841: finishing write tid 5 to smithi12313934-30 2020-01-16T00:12:51.846 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 30 v 68 (ObjNum 164 snap 0 seq_num 164) dirty exists 2020-01-16T00:12:51.846 INFO:tasks.rados.rados.0.smithi123.stdout:841: left oid 30 (ObjNum 164 snap 0 seq_num 164) 2020-01-16T00:12:51.846 INFO:tasks.rados.rados.0.smithi123.stdout:842: finishing write tid 1 to smithi12313934-9 2020-01-16T00:12:51.846 INFO:tasks.rados.rados.0.smithi123.stdout:842: finishing write tid 2 to smithi12313934-9 2020-01-16T00:12:51.847 INFO:tasks.rados.rados.0.smithi123.stdout:849: read oid 36 snap -1 2020-01-16T00:12:51.847 INFO:tasks.rados.rados.0.smithi123.stdout:849: expect (ObjNum 161 snap 0 seq_num 161) 2020-01-16T00:12:51.847 INFO:tasks.rados.rados.0.smithi123.stdout:838: done (10 left) 2020-01-16T00:12:51.847 INFO:tasks.rados.rados.0.smithi123.stdout:840: done (9 left) 2020-01-16T00:12:51.848 INFO:tasks.rados.rados.0.smithi123.stdout:841: done (8 left) 2020-01-16T00:12:51.848 INFO:tasks.rados.rados.0.smithi123.stdout:850: read oid 41 snap -1 2020-01-16T00:12:51.848 INFO:tasks.rados.rados.0.smithi123.stdout:850: expect (ObjNum 159 snap 0 seq_num 159) 2020-01-16T00:12:51.848 INFO:tasks.rados.rados.0.smithi123.stdout:851: read oid 13 snap -1 2020-01-16T00:12:51.849 INFO:tasks.rados.rados.0.smithi123.stdout:851: expect (ObjNum 124 snap 0 seq_num 124) 2020-01-16T00:12:51.850 INFO:tasks.rados.rados.0.smithi123.stdout:852: delete oid 35 current snap is 0 2020-01-16T00:12:51.850 INFO:tasks.rados.rados.0.smithi123.stdout:842: finishing write tid 3 to smithi12313934-9 2020-01-16T00:12:51.850 INFO:tasks.rados.rados.0.smithi123.stdout:842: finishing write tid 5 to smithi12313934-9 2020-01-16T00:12:51.850 INFO:tasks.rados.rados.0.smithi123.stdout:842: finishing write tid 6 to smithi12313934-9 2020-01-16T00:12:51.851 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 9 v 74 (ObjNum 165 snap 0 seq_num 165) dirty exists 2020-01-16T00:12:51.851 INFO:tasks.rados.rados.0.smithi123.stdout:842: left oid 9 (ObjNum 165 snap 0 seq_num 165) 2020-01-16T00:12:51.851 INFO:tasks.rados.rados.0.smithi123.stdout:843: finishing write tid 1 to smithi12313934-1 2020-01-16T00:12:51.851 INFO:tasks.rados.rados.0.smithi123.stdout:843: finishing write tid 2 to smithi12313934-1 2020-01-16T00:12:51.852 INFO:tasks.rados.rados.0.smithi123.stdout:843: finishing write tid 4 to smithi12313934-1 2020-01-16T00:12:51.852 INFO:tasks.rados.rados.0.smithi123.stdout:843: finishing write tid 5 to smithi12313934-1 2020-01-16T00:12:51.852 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 1 v 150 (ObjNum 166 snap 0 seq_num 166) dirty exists 2020-01-16T00:12:51.852 INFO:tasks.rados.rados.0.smithi123.stdout:843: left oid 1 (ObjNum 166 snap 0 seq_num 166) 2020-01-16T00:12:51.853 INFO:tasks.rados.rados.0.smithi123.stdout:844: finishing write tid 1 to smithi12313934-49 2020-01-16T00:12:51.853 INFO:tasks.rados.rados.0.smithi123.stdout:844: finishing write tid 2 to smithi12313934-49 2020-01-16T00:12:51.853 INFO:tasks.rados.rados.0.smithi123.stdout:844: finishing write tid 3 to smithi12313934-49 2020-01-16T00:12:51.854 INFO:tasks.rados.rados.0.smithi123.stdout:844: finishing write tid 5 to smithi12313934-49 2020-01-16T00:12:51.854 INFO:tasks.rados.rados.0.smithi123.stdout:844: finishing write tid 6 to smithi12313934-49 2020-01-16T00:12:51.854 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 49 v 75 (ObjNum 167 snap 0 seq_num 167) dirty exists 2020-01-16T00:12:51.854 INFO:tasks.rados.rados.0.smithi123.stdout:844: left oid 49 (ObjNum 167 snap 0 seq_num 167) 2020-01-16T00:12:51.854 INFO:tasks.rados.rados.0.smithi123.stdout:845: expect (ObjNum 147 snap 0 seq_num 147) 2020-01-16T00:12:51.890 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:12:51.891+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:12:51.922 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:12:51.923 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2020-01-16T00:12:51.933 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2020-01-16T00:12:51.767969+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":247917780,"num_objects":97,"num_object_clones":0,"num_object_copies":291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":1344,"num_read_kb":909470,"num_write":933,"num_write_kb":429088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":42,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":947,"ondisk_log_size":947,"up":84,"acting":84,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":84,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":9065856,"kb_used_data":638896,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":745908864,"statfs":{"total":773094113280,"available":763810676736,"internally_reserved":8589934592,"allocated":654229504,"data_stored":650996676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19125376,"num_objects":-2,"num_object_clones":0,"num_object_copies":-6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":681,"num_read_kb":551936,"num_write":305,"num_write_kb":98349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.810322"},"pg_stats":[{"pgid":"3.f","version":"27'22","reported_seq":"61","reported_epoch":"29","state":"active+clean","last_fresh":"2020-01-16T00:12:49.123501+0000","last_change":"2020-01-16T00:12:32.829755+0000","last_active":"2020-01-16T00:12:49.123501+0000","last_peered":"2020-01-16T00:12:49.123501+0000","last_clean":"2020-01-16T00:12:49.123501+0000","last_became_active":"2020-01-16T00:12:32.829304+0000","last_became_peered":"2020-01-16T00:12:32.829304+0000","last_unstale":"2020-01-16T00:12:49.123501+0000","last_undegraded":"2020-01-16T00:12:49.123501+0000","last_fullsized":"2020-01-16T00:12:49.123501+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2067261,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":56,"num_read_kb":38601,"num_write":19,"num_write_kb":4387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"28'36","reported_seq":"76","reported_epoch":"28","state":"active+clean","last_fresh":"2020-01-16T00:12:47.957330+0000","last_change":"2020-01-16T00:12:32.829508+0000","last_active":"2020-01-16T00:12:47.957330+0000","last_peered":"2020-01-16T00:12:47.957330+0000","last_clean":"2020-01-16T00:12:47.957330+0000","last_became_active":"2020-01-16T00:12:32.829047+0000","last_became_peered":"2020-01-16T00:12:32.829047+0000","last_unstale":"2020-01-16T00:12:47.957330+0000","last_undegraded":"2020-01-16T00:12:47.957330+0000","last_fullsized":"2020-01-16T00:12:47.957330+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6353013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":49,"num_read_kb":29374,"num_write":23,"num_write_kb":7191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"28'113","reported_seq":"193","reported_epoch":"29","state":"active+clean","last_fresh":"2020-01-16T00:12:49.847529+0000","last_change":"2020-01-16T00:12:32.834607+0000","last_active":"2020-01-16T00:12:49.847529+0000","last_peered":"2020-01-16T00:12:49.847529+0000","last_clean":"2020-01-16T00:12:49.847529+0000","last_became_active":"2020-01-16T00:12:32.834155+0000","last_became_peered":"2020-01-16T00:12:32.834155+0000","last_unstale":"2020-01-16T00:12:49.847529+0000","last_undegraded":"2020-01-16T00:12:49.847529+0000","last_fullsized":"2020-01-16T00:12:49.847529+0000","mapping_epoch":20,"log_start":"27'100","ondisk_log_start":"27'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9916097,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":147,"num_read_kb":126960,"num_write":90,"num_write_kb":26945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":5,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"29'88","reported_seq":"148","reported_epoch":"29","state":"active+clean","last_fresh":"2020-01-16T00:12:49.195178+0000","last_change":"2020-01-16T00:12:32.834625+0000","last_active":"2020-01-16T00:12:49.195178+0000","last_peered":"2020-01-16T00:12:49.195178+0000","last_clean":"2020-01-16T00:12:49.195178+0000","last_became_active":"2020-01-16T00:12:32.834210+0000","last_became_peered":"2020-01-16T00:12:32.834210+0000","last_unstale":"2020-01-16T00:12:49.195178+0000","last_undegraded":"2020-01-16T00:12:49.195178+0000","last_fullsized":"2020-01-16T00:12:49.195178+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":88,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11126168,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":95,"num_read_kb":63018,"num_write":63,"num_write_kb":20075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":4,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"29'53","reported_seq":"90","reported_epoch":"29","state":"active+clean","last_fresh":"2020-01-16T00:12:49.885922+0000","last_change":"2020-01-16T00:12:32.833522+0000","last_active":"2020-01-16T00:12:49.885922+0000","last_peered":"2020-01-16T00:12:49.885922+0000","last_clean":"2020-01-16T00:12:49.885922+0000","last_became_active":"2020-01-16T00:12:32.833051+0000","last_became_peered":"2020-01-16T00:12:32.833051+0000","last_unstale":"2020-01-16T00:12:49.885922+0000","last_undegraded":"2020-01-16T00:12:49.885922+0000","last_fullsized":"2020-01-16T00:12:49.885922+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":53,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5142942,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":18667,"num_write":44,"num_write_kb":13113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"28'94","reported_seq":"155","reported_epoch":"28","state":"active+clean","last_fresh":"2020-01-16T00:12:46.681756+0000","last_change":"2020-01-16T00:12:32.834658+0000","last_active":"2020-01-16T00:12:46.681756+0000","last_peered":"2020-01-16T00:12:46.681756+0000","last_clean":"2020-01-16T00:12:46.681756+0000","last_became_active":"2020-01-16T00:12:32.832936+0000","last_became_peered":"2020-01-16T00:12:32.832936+0000","last_unstale":"2020-01-16T00:12:46.681756+0000","last_undegraded":"2020-01-16T00:12:46.681756+0000","last_fullsized":"2020-01-16T00:12:46.681756+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":94,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11563150,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":93,"num_read_kb":46343,"num_write":67,"num_write_kb":21818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":4,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"29'104","reported_seq":"161","reported_epoch":"29","state":"active+clean","last_fresh":"2020-01-16T00:12:49.440033+0000","last_change":"2020-01-16T00:12:32.832821+0000","last_active":"2020-01-16T00:12:49.440033+0000","last_peered":"2020-01-16T00:12:49.440033+0000","last_clean":"2020-01-16T00:12:49.440033+0000","last_became_active":"2020-01-16T00:12:32.832272+0000","last_became_peered":"2020-01-16T00:12:32.832272+0000","last_unstale":"2020-01-16T00:12:49.440033+0000","last_undegraded":"2020-01-16T00:12:49.440033+0000","last_fullsized":"2020-01-16T00:12:49.440033+0000","mapping_epoch":20,"log_start":"29'100","ondisk_log_start":"29'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13680832,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":81,"num_read_kb":55330,"num_write":78,"num_write_kb":23835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":5,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"29'56","reported_seq":"110","reported_epoch":"29","state":"active+clean","last_fresh":"2020-01-16T00:12:49.412755+0000","last_change":"2020-01-16T00:12:32.830727+0000","last_active":"2020-01-16T00:12:49.412755+0000","last_peered":"2020-01-16T00:12:49.412755+0000","last_clean":"2020-01-16T00:12:49.412755+0000","last_became_active":"2020-01-16T00:12:32.830301+0000","last_became_peered":"2020-01-16T00:12:32.830301+0000","last_unstale":"2020-01-16T00:12:49.412755+0000","last_undegraded":"2020-01-16T00:12:49.412755+0000","last_fullsized":"2020-01-16T00:12:49.412755+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6958065,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":90,"num_read_kb":66007,"num_write":35,"num_write_kb":10776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"29'65","reported_seq":"115","reported_epoch":"29","state":"active+clean","last_fresh":"2020-01-16T00:12:49.871491+0000","last_change":"2020-01-16T00:12:32.830011+0000","last_active":"2020-01-16T00:12:49.871491+0000","last_peered":"2020-01-16T00:12:49.871491+0000","last_clean":"2020-01-16T00:12:49.871491+0000","last_became_active":"2020-01-16T00:12:32.829571+0000","last_became_peered":"2020-01-16T00:12:32.829571+0000","last_unstale":"2020-01-16T00:12:49.871491+0000","last_undegraded":"2020-01-16T00:12:49.871491+0000","last_fullsized":"2020-01-16T00:12:49.871491+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":65,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8453888,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":72,"num_read_kb":54927,"num_write":44,"num_write_kb":14541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"22","reported_epoch":"28","state":"active+clean","last_fresh":"2020-01-16T00:12:45.902800+0000","last_change":"2020-01-16T00:12:20.031741+0000","last_active":"2020-01-16T00:12:45.902800+0000","last_peered":"2020-01-16T00:12:45.902800+0000","last_clean":"2020-01-16T00:12:45.902800+0000","last_became_active":"2020-01-16T00:12:20.030776+0000","last_became_peered":"2020-01-16T00:12:20.030776+0000","last_unstale":"2020-01-16T00:12:45.902800+0000","last_undegraded":"2020-01-16T00:12:45.902800+0000","last_fullsized":"2020-01-16T00:12:45.902800+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"17","reported_epoch":"28","state":"active+clean","last_fresh":"2020-01-16T00:12:45.959940+0000","last_change":"2020-01-16T00:12:32.832234+0000","last_active":"2020-01-16T00:12:45.959940+0000","last_peered":"2020-01-16T00:12:45.959940+0000","last_clean":"2020-01-16T00:12:45.959940+0000","last_became_active":"2020-01-16T00:12:32.831847+0000","last_became_peered":"2020-01-16T00:12:32.831847+0000","last_unstale":"2020-01-16T00:12:45.959940+0000","last_undegraded":"2020-01-16T00:12:45.959940+0000","last_fullsized":"2020-01-16T00:12:45.959940+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"22","reported_epoch":"28","state":"active+clean","last_fresh":"2020-01-16T00:12:45.902384+0000","last_change":"2020-01-16T00:12:20.032733+0000","last_active":"2020-01-16T00:12:45.902384+0000","last_peered":"2020-01-16T00:12:45.902384+0000","last_clean":"2020-01-16T00:12:45.902384+0000","last_became_active":"2020-01-16T00:12:20.032284+0000","last_became_peered":"2020-01-16T00:12:20.032284+0000","last_unstale":"2020-01-16T00:12:45.902384+0000","last_undegraded":"2020-01-16T00:12:45.902384+0000","last_fullsized":"2020-01-16T00:12:45.902384+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"22","reported_epoch":"28","state":"active+clean","last_fresh":"2020-01-16T00:12:45.959051+0000","last_change":"2020-01-16T00:12:20.032010+0000","last_active":"2020-01-16T00:12:45.959051+0000","last_peered":"2020-01-16T00:12:45.959051+0000","last_clean":"2020-01-16T00:12:45.959051+0000","last_became_active":"2020-01-16T00:12:20.031560+0000","last_became_peered":"2020-01-16T00:12:20.031560+0000","last_unstale":"2020-01-16T00:12:45.959051+0000","last_undegraded":"2020-01-16T00:12:45.959051+0000","last_fullsized":"2020-01-16T00:12:45.959051+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"29'39","reported_seq":"100","reported_epoch":"29","state":"active+clean","last_fresh":"2020-01-16T00:12:49.212058+0000","last_change":"2020-01-16T00:12:31.817123+0000","last_active":"2020-01-16T00:12:49.212058+0000","last_peered":"2020-01-16T00:12:49.212058+0000","last_clean":"2020-01-16T00:12:49.212058+0000","last_became_active":"2020-01-16T00:12:31.816785+0000","last_became_peered":"2020-01-16T00:12:31.816785+0000","last_unstale":"2020-01-16T00:12:49.212058+0000","last_undegraded":"2020-01-16T00:12:49.212058+0000","last_fullsized":"2020-01-16T00:12:49.212058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:30.783643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:30.783643+0000","last_clean_scrub_stamp":"2020-01-16T00:12:30.783643+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33008486,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":43,"num_read_kb":17379,"num_write":39,"num_write_kb":51192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"27'36","reported_seq":"80","reported_epoch":"29","state":"active+clean","last_fresh":"2020-01-16T00:12:49.149269+0000","last_change":"2020-01-16T00:12:32.836948+0000","last_active":"2020-01-16T00:12:49.149269+0000","last_peered":"2020-01-16T00:12:49.149269+0000","last_clean":"2020-01-16T00:12:49.149269+0000","last_became_active":"2020-01-16T00:12:32.836523+0000","last_became_peered":"2020-01-16T00:12:32.836523+0000","last_unstale":"2020-01-16T00:12:49.149269+0000","last_undegraded":"2020-01-16T00:12:49.149269+0000","last_fullsized":"2020-01-16T00:12:49.149269+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5663926,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":65,"num_read_kb":51740,"num_write":23,"num_write_kb":7349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"22","reported_epoch":"28","state":"active+clean","last_fresh":"2020-01-16T00:12:45.959243+0000","last_change":"2020-01-16T00:12:20.032658+0000","last_active":"2020-01-16T00:12:45.959243+0000","last_peered":"2020-01-16T00:12:45.959243+0000","last_clean":"2020-01-16T00:12:45.959243+0000","last_became_active":"2020-01-16T00:12:20.030794+0000","last_became_peered":"2020-01-16T00:12:20.030794+0000","last_unstale":"2020-01-16T00:12:45.959243+0000","last_undegraded":"2020-01-16T00:12:45.959243+0000","last_fullsized":"2020-01-16T00:12:45.959243+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"28'49","reported_seq":"104","reported_epoch":"28","state":"active+clean","last_fresh":"2020-01-16T00:12:47.950344+0000","last_change":"2020-01-16T00:12:31.817506+0000","last_active":"2020-01-16T00:12:47.950344+0000","last_peered":"2020-01-16T00:12:47.950344+0000","last_clean":"2020-01-16T00:12:47.950344+0000","last_became_active":"2020-01-16T00:12:31.817077+0000","last_became_peered":"2020-01-16T00:12:31.817077+0000","last_unstale":"2020-01-16T00:12:47.950344+0000","last_undegraded":"2020-01-16T00:12:47.950344+0000","last_fullsized":"2020-01-16T00:12:47.950344+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:30.783643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:30.783643+0000","last_clean_scrub_stamp":"2020-01-16T00:12:30.783643+0000","log_size":49,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37445501,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":37,"num_read_kb":19216,"num_write":49,"num_write_kb":57156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"29'147","reported_seq":"229","reported_epoch":"29","state":"active+clean","last_fresh":"2020-01-16T00:12:49.412024+0000","last_change":"2020-01-16T00:12:32.830294+0000","last_active":"2020-01-16T00:12:49.412024+0000","last_peered":"2020-01-16T00:12:49.412024+0000","last_clean":"2020-01-16T00:12:49.412024+0000","last_became_active":"2020-01-16T00:12:32.829870+0000","last_became_peered":"2020-01-16T00:12:32.829870+0000","last_unstale":"2020-01-16T00:12:49.412024+0000","last_undegraded":"2020-01-16T00:12:49.412024+0000","last_fullsized":"2020-01-16T00:12:49.412024+0000","mapping_epoch":20,"log_start":"24'100","ondisk_log_start":"24'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17277497,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":142,"num_read_kb":102444,"num_write":107,"num_write_kb":34088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"22","reported_epoch":"28","state":"active+clean","last_fresh":"2020-01-16T00:12:45.959270+0000","last_change":"2020-01-16T00:12:20.033205+0000","last_active":"2020-01-16T00:12:45.959270+0000","last_peered":"2020-01-16T00:12:45.959270+0000","last_clean":"2020-01-16T00:12:45.959270+0000","last_became_active":"2020-01-16T00:12:20.030792+0000","last_became_peered":"2020-01-16T00:12:20.030792+0000","last_unstale":"2020-01-16T00:12:45.959270+0000","last_undegraded":"2020-01-16T00:12:45.959270+0000","last_fullsized":"2020-01-16T00:12:45.959270+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"29'37","reported_seq":"89","reported_epoch":"29","state":"active+clean","last_fresh":"2020-01-16T00:12:49.161643+0000","last_change":"2020-01-16T00:12:31.815364+0000","last_active":"2020-01-16T00:12:49.161643+0000","last_peered":"2020-01-16T00:12:49.161643+0000","last_clean":"2020-01-16T00:12:49.161643+0000","last_became_active":"2020-01-16T00:12:31.814890+0000","last_became_peered":"2020-01-16T00:12:31.814890+0000","last_unstale":"2020-01-16T00:12:49.161643+0000","last_undegraded":"2020-01-16T00:12:49.161643+0000","last_fullsized":"2020-01-16T00:12:49.161643+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:30.783643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:30.783643+0000","last_clean_scrub_stamp":"2020-01-16T00:12:30.783643+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30319399,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":34,"num_read_kb":16408,"num_write":37,"num_write_kb":43221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"27'69","reported_seq":"128","reported_epoch":"29","state":"active+clean","last_fresh":"2020-01-16T00:12:49.166498+0000","last_change":"2020-01-16T00:12:32.834986+0000","last_active":"2020-01-16T00:12:49.166498+0000","last_peered":"2020-01-16T00:12:49.166498+0000","last_clean":"2020-01-16T00:12:49.166498+0000","last_became_active":"2020-01-16T00:12:32.834537+0000","last_became_peered":"2020-01-16T00:12:32.834537+0000","last_unstale":"2020-01-16T00:12:49.166498+0000","last_undegraded":"2020-01-16T00:12:49.166498+0000","last_fullsized":"2020-01-16T00:12:49.166498+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":69,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7579924,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":105,"num_read_kb":95141,"num_write":46,"num_write_kb":17493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"22","reported_epoch":"28","state":"active+clean","last_fresh":"2020-01-16T00:12:45.958772+0000","last_change":"2020-01-16T00:12:20.031201+0000","last_active":"2020-01-16T00:12:45.958772+0000","last_peered":"2020-01-16T00:12:45.958772+0000","last_clean":"2020-01-16T00:12:45.958772+0000","last_became_active":"2020-01-16T00:12:20.030752+0000","last_became_peered":"2020-01-16T00:12:20.030752+0000","last_unstale":"2020-01-16T00:12:45.958772+0000","last_undegraded":"2020-01-16T00:12:45.958772+0000","last_fullsized":"2020-01-16T00:12:45.958772+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"27'19","reported_seq":"58","reported_epoch":"28","state":"active+clean","last_fresh":"2020-01-16T00:12:45.959034+0000","last_change":"2020-01-16T00:12:31.815506+0000","last_active":"2020-01-16T00:12:45.959034+0000","last_peered":"2020-01-16T00:12:45.959034+0000","last_clean":"2020-01-16T00:12:45.959034+0000","last_became_active":"2020-01-16T00:12:31.815111+0000","last_became_peered":"2020-01-16T00:12:31.815111+0000","last_unstale":"2020-01-16T00:12:45.959034+0000","last_undegraded":"2020-01-16T00:12:45.959034+0000","last_fullsized":"2020-01-16T00:12:45.959034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:30.783643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:30.783643+0000","last_clean_scrub_stamp":"2020-01-16T00:12:30.783643+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19831996,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":21,"num_read_kb":10085,"num_write":19,"num_write_kb":29369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"29'71","reported_seq":"116","reported_epoch":"29","state":"active+clean","last_fresh":"2020-01-16T00:12:49.436742+0000","last_change":"2020-01-16T00:12:32.832002+0000","last_active":"2020-01-16T00:12:49.436742+0000","last_peered":"2020-01-16T00:12:49.436742+0000","last_clean":"2020-01-16T00:12:49.436742+0000","last_became_active":"2020-01-16T00:12:32.831577+0000","last_became_peered":"2020-01-16T00:12:32.831577+0000","last_unstale":"2020-01-16T00:12:49.436742+0000","last_undegraded":"2020-01-16T00:12:49.436742+0000","last_fullsized":"2020-01-16T00:12:49.436742+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":71,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7479082,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":56,"num_read_kb":25736,"num_write":52,"num_write_kb":16344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"28'79","reported_seq":"121","reported_epoch":"28","state":"active+clean","last_fresh":"2020-01-16T00:12:47.919485+0000","last_change":"2020-01-16T00:12:32.834894+0000","last_active":"2020-01-16T00:12:47.919485+0000","last_peered":"2020-01-16T00:12:47.919485+0000","last_clean":"2020-01-16T00:12:47.919485+0000","last_became_active":"2020-01-16T00:12:32.834414+0000","last_became_peered":"2020-01-16T00:12:32.834414+0000","last_unstale":"2020-01-16T00:12:47.919485+0000","last_undegraded":"2020-01-16T00:12:47.919485+0000","last_fullsized":"2020-01-16T00:12:47.919485+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":79,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4655506,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":52,"num_read_kb":28265,"num_write":51,"num_write_kb":14887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":4,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"22","reported_epoch":"28","state":"active+clean","last_fresh":"2020-01-16T00:12:45.959488+0000","last_change":"2020-01-16T00:12:20.033023+0000","last_active":"2020-01-16T00:12:45.959488+0000","last_peered":"2020-01-16T00:12:45.959488+0000","last_clean":"2020-01-16T00:12:45.959488+0000","last_became_active":"2020-01-16T00:12:20.031576+0000","last_became_peered":"2020-01-16T00:12:20.031576+0000","last_unstale":"2020-01-16T00:12:45.959488+0000","last_undegraded":"2020-01-16T00:12:45.959488+0000","last_fullsized":"2020-01-16T00:12:45.959488+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"28'70","reported_seq":"118","reported_epoch":"29","state":"active+clean","last_fresh":"2020-01-16T00:12:49.878476+0000","last_change":"2020-01-16T00:12:32.832313+0000","last_active":"2020-01-16T00:12:49.878476+0000","last_peered":"2020-01-16T00:12:49.878476+0000","last_clean":"2020-01-16T00:12:49.878476+0000","last_became_active":"2020-01-16T00:12:32.831843+0000","last_became_peered":"2020-01-16T00:12:32.831843+0000","last_unstale":"2020-01-16T00:12:49.878476+0000","last_undegraded":"2020-01-16T00:12:49.878476+0000","last_fullsized":"2020-01-16T00:12:49.878476+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":70,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9395047,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":64,"num_read_kb":43829,"num_write":47,"num_write_kb":15308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"22","reported_epoch":"28","state":"active+clean","last_fresh":"2020-01-16T00:12:45.959391+0000","last_change":"2020-01-16T00:12:20.034125+0000","last_active":"2020-01-16T00:12:45.959391+0000","last_peered":"2020-01-16T00:12:45.959391+0000","last_clean":"2020-01-16T00:12:45.959391+0000","last_became_active":"2020-01-16T00:12:20.033645+0000","last_became_peered":"2020-01-16T00:12:20.033645+0000","last_unstale":"2020-01-16T00:12:45.959391+0000","last_undegraded":"2020-01-16T00:12:45.959391+0000","last_fullsized":"2020-01-16T00:12:45.959391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":127312398,"num_objects":47,"num_object_clones":0,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":1209,"num_read_kb":846382,"num_write":789,"num_write_kb":248150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":42,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":290795520,"data_stored":289372953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":803,"ondisk_log_size":803,"up":48,"acting":48,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":2,"num_pg":4,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":135,"num_read_kb":63088,"num_write":144,"num_write_kb":180938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":362127360,"data_stored":361816146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144,"ondisk_log_size":144,"up":12,"acting":12,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1148144,"kb_used_data":98016,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93223696,"statfs":{"total":96636764160,"available":95461064704,"internally_reserved":1073741824,"allocated":100368384,"data_stored":99933897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1120724,"kb_used_data":68056,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93251116,"statfs":{"total":96636764160,"available":95489142784,"internally_reserved":1073741824,"allocated":69689344,"data_stored":69226815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1139940,"kb_used_data":85552,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93231900,"statfs":{"total":96636764160,"available":95469465600,"internally_reserved":1073741824,"allocated":87605248,"data_stored":87198874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1147112,"kb_used_data":90732,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93224728,"statfs":{"total":96636764160,"available":95462121472,"internally_reserved":1073741824,"allocated":92909568,"data_stored":92566577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1153268,"kb_used_data":100444,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93218572,"statfs":{"total":96636764160,"available":95455817728,"internally_reserved":1073741824,"allocated":102854656,"data_stored":102451045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1130132,"kb_used_data":78348,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93241708,"statfs":{"total":96636764160,"available":95479508992,"internally_reserved":1073741824,"allocated":80228352,"data_stored":79816143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574858,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1094580,"kb_used_data":36408,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93277260,"statfs":{"total":96636764160,"available":95515914240,"internally_reserved":1073741824,"allocated":37281792,"data_stored":36922640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1131956,"kb_used_data":81340,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93239884,"statfs":{"total":96636764160,"available":95477641216,"internally_reserved":1073741824,"allocated":83292160,"data_stored":82880685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}]}} 2020-01-16T00:12:51.936 INFO:tasks.thrashosds.thrasher:Setting 3.7 to [1, 5, 3] 2020-01-16T00:12:51.936 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.7', '1', '5', '3'] 2020-01-16T00:12:51.936 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 3.7 1 5 3 2020-01-16T00:12:51.966 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:51.966 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:12:51.966 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:12:51.967 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:12:51.967 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:51.975 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-01-16T00:12:51.997 INFO:tasks.rados.rados.0.smithi123.stdout:847: expect (ObjNum 97 snap 0 seq_num 97) 2020-01-16T00:12:51.997 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:51.999+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:12:52.087 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:52.087 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:12:52.087 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:12:52.087 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:12:52.088 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.088 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:910 3.5 3:ad8ab15f:::smithi12313934-27:head [write 1563527~583025] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:52.088 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.830607+0000", 2020-01-16T00:12:52.088 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.254466333, 2020-01-16T00:12:52.088 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019080092999999999, 2020-01-16T00:12:52.088 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.088 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:52.089 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.089 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.089 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.089 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 910 2020-01-16T00:12:52.089 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.089 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.089 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.090 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.090 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.830607+0000", 2020-01-16T00:12:52.090 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.090 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.090 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.090 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.090 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.830607+0000", 2020-01-16T00:12:52.091 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:52.091 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.091 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.091 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.091 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.830604+0000", 2020-01-16T00:12:52.091 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00078346100000000005 2020-01-16T00:12:52.092 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.092 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.092 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.092 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831388+0000", 2020-01-16T00:12:52.092 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.6936000000000001e-05 2020-01-16T00:12:52.092 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.092 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.093 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.093 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831415+0000", 2020-01-16T00:12:52.093 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0737000000000001e-05 2020-01-16T00:12:52.093 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.093 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.093 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.093 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831425+0000", 2020-01-16T00:12:52.093 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1964e-05 2020-01-16T00:12:52.094 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.094 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.094 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.094 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831467+0000", 2020-01-16T00:12:52.094 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028524600000000001 2020-01-16T00:12:52.094 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.094 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.094 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:52.095 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831753+0000", 2020-01-16T00:12:52.095 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012909141000000001 2020-01-16T00:12:52.096 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.096 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.096 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.096 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.844662+0000", 2020-01-16T00:12:52.096 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022242 2020-01-16T00:12:52.097 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.097 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.097 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.097 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.844884+0000", 2020-01-16T00:12:52.097 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015169700000000001 2020-01-16T00:12:52.097 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.097 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.098 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,3", 2020-01-16T00:12:52.098 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.845036+0000", 2020-01-16T00:12:52.098 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0027572859999999999 2020-01-16T00:12:52.098 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.098 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.099 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:52.099 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.847793+0000", 2020-01-16T00:12:52.099 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00084882100000000004 2020-01-16T00:12:52.099 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.099 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.099 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:52.099 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.848642+0000", 2020-01-16T00:12:52.100 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00065781900000000003 2020-01-16T00:12:52.100 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.100 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.100 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:52.100 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.849300+0000", 2020-01-16T00:12:52.100 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0001717 2020-01-16T00:12:52.100 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.101 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.101 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:52.101 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.849472+0000", 2020-01-16T00:12:52.101 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000215348 2020-01-16T00:12:52.101 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.102 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.102 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.102 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.849687+0000", 2020-01-16T00:12:52.102 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019080092999999999 2020-01-16T00:12:52.102 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.102 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.102 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.103 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.103 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.103 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:911 3.5 3:ad8ab15f:::smithi12313934-27:head [write 2810926~399178] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:52.103 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.836707+0000", 2020-01-16T00:12:52.103 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.248365703999999, 2020-01-16T00:12:52.103 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.024736662999999999, 2020-01-16T00:12:52.104 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.104 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:52.104 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.104 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.104 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.104 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 911 2020-01-16T00:12:52.104 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.104 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.105 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.105 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.105 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.836707+0000", 2020-01-16T00:12:52.105 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.105 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.106 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.106 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.106 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.836707+0000", 2020-01-16T00:12:52.106 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:12:52.106 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.107 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.107 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.107 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.836704+0000", 2020-01-16T00:12:52.107 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00077069200000000001 2020-01-16T00:12:52.107 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.108 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.108 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.108 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837474+0000", 2020-01-16T00:12:52.108 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7440000000000001e-05 2020-01-16T00:12:52.108 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.109 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.109 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.109 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837512+0000", 2020-01-16T00:12:52.109 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8246000000000001e-05 2020-01-16T00:12:52.109 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.110 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.110 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.110 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837530+0000", 2020-01-16T00:12:52.110 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000105405 2020-01-16T00:12:52.110 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.110 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.110 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.111 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837635+0000", 2020-01-16T00:12:52.111 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6244999999999998e-05 2020-01-16T00:12:52.111 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.111 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.111 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:52.111 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837722+0000", 2020-01-16T00:12:52.112 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0089724320000000007 2020-01-16T00:12:52.112 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.112 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.112 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.112 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.846694+0000", 2020-01-16T00:12:52.112 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042899099999999999 2020-01-16T00:12:52.112 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.113 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.113 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:52.113 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.847123+0000", 2020-01-16T00:12:52.113 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011739893 2020-01-16T00:12:52.113 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.113 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.113 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.113 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.858863+0000", 2020-01-16T00:12:52.114 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1312999999999999e-05 2020-01-16T00:12:52.114 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.114 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.114 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.114 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.858944+0000", 2020-01-16T00:12:52.114 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015682900000000001 2020-01-16T00:12:52.114 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.115 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.115 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,3", 2020-01-16T00:12:52.115 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.859101+0000", 2020-01-16T00:12:52.115 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001604442 2020-01-16T00:12:52.115 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.115 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.115 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:52.116 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.860706+0000", 2020-01-16T00:12:52.116 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000377568 2020-01-16T00:12:52.116 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.116 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.116 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:52.117 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.861083+0000", 2020-01-16T00:12:52.117 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022656400000000001 2020-01-16T00:12:52.117 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.117 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.117 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:52.117 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.861310+0000", 2020-01-16T00:12:52.117 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7865000000000002e-05 2020-01-16T00:12:52.118 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.118 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.118 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:52.118 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.861368+0000", 2020-01-16T00:12:52.118 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6549000000000002e-05 2020-01-16T00:12:52.118 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.118 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.118 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.119 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.861444+0000", 2020-01-16T00:12:52.119 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.024736662999999999 2020-01-16T00:12:52.119 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.119 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.119 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.119 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.119 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.120 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:912 3.5 3:ad8ab15f:::smithi12313934-27:head [setxattr _header (56) in=7b,truncate 3210104] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:52.120 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.837560+0000", 2020-01-16T00:12:52.120 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.247512992000001, 2020-01-16T00:12:52.120 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.024152410999999999, 2020-01-16T00:12:52.120 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.121 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:52.121 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.121 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.121 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.121 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 912 2020-01-16T00:12:52.122 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.122 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.122 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.123 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.131 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837560+0000", 2020-01-16T00:12:52.132 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.132 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.132 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.133 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.133 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837560+0000", 2020-01-16T00:12:52.133 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:52.133 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.134 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.134 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.134 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837558+0000", 2020-01-16T00:12:52.134 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8468999999999999e-05 2020-01-16T00:12:52.134 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.135 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.135 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.135 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837576+0000", 2020-01-16T00:12:52.135 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8437e-05 2020-01-16T00:12:52.135 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.136 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.136 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.136 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837595+0000", 2020-01-16T00:12:52.136 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2782999999999999e-05 2020-01-16T00:12:52.136 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.136 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.136 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.137 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837608+0000", 2020-01-16T00:12:52.137 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000174883 2020-01-16T00:12:52.137 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.137 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.137 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.137 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837782+0000", 2020-01-16T00:12:52.137 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8194000000000001e-05 2020-01-16T00:12:52.137 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.138 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.138 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:52.138 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837821+0000", 2020-01-16T00:12:52.138 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0094026500000000002 2020-01-16T00:12:52.138 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.138 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.138 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.139 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.847223+0000", 2020-01-16T00:12:52.139 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000120973 2020-01-16T00:12:52.139 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.139 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.139 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:52.139 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.847344+0000", 2020-01-16T00:12:52.139 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012601658999999999 2020-01-16T00:12:52.140 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.140 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.140 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.140 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.859946+0000", 2020-01-16T00:12:52.140 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012089 2020-01-16T00:12:52.140 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.141 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.141 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.141 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.860067+0000", 2020-01-16T00:12:52.141 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020346700000000001 2020-01-16T00:12:52.141 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.141 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.142 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,3", 2020-01-16T00:12:52.142 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.860270+0000", 2020-01-16T00:12:52.142 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00057751500000000002 2020-01-16T00:12:52.142 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.142 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.143 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:52.143 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.860848+0000", 2020-01-16T00:12:52.143 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00067389800000000003 2020-01-16T00:12:52.143 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.143 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.143 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:52.143 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.861522+0000", 2020-01-16T00:12:52.144 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2540999999999996e-05 2020-01-16T00:12:52.144 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.144 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.144 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:52.144 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.861584+0000", 2020-01-16T00:12:52.144 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8773e-05 2020-01-16T00:12:52.144 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.145 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.145 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:52.145 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.861653+0000", 2020-01-16T00:12:52.146 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9493000000000001e-05 2020-01-16T00:12:52.146 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.146 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.146 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.146 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.861713+0000", 2020-01-16T00:12:52.146 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.024152410999999999 2020-01-16T00:12:52.146 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.147 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.147 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.147 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.147 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.147 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:913 3.5 3:ad8ab15f:::smithi12313934-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:12:52.147 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.837629+0000", 2020-01-16T00:12:52.148 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.247444174, 2020-01-16T00:12:52.148 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.024296113000000001, 2020-01-16T00:12:52.148 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.148 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:52.148 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.148 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.148 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.149 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 913 2020-01-16T00:12:52.149 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.149 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.149 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.149 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.149 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837629+0000", 2020-01-16T00:12:52.149 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.149 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.150 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.150 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.150 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837629+0000", 2020-01-16T00:12:52.150 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:52.150 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.150 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.151 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.151 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837627+0000", 2020-01-16T00:12:52.151 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.329e-06 2020-01-16T00:12:52.151 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.151 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.151 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.151 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837637+0000", 2020-01-16T00:12:52.152 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7397e-05 2020-01-16T00:12:52.152 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.152 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.152 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.152 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837654+0000", 2020-01-16T00:12:52.152 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6073999999999999e-05 2020-01-16T00:12:52.153 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.153 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.153 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.153 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837670+0000", 2020-01-16T00:12:52.153 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000171781 2020-01-16T00:12:52.153 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.154 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.154 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.154 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837842+0000", 2020-01-16T00:12:52.154 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1575000000000002e-05 2020-01-16T00:12:52.154 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.154 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.154 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:52.154 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.837873+0000", 2020-01-16T00:12:52.155 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0095469749999999992 2020-01-16T00:12:52.155 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.155 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.155 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.155 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.847420+0000", 2020-01-16T00:12:52.156 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5698000000000003e-05 2020-01-16T00:12:52.156 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.156 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.156 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:52.156 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.847456+0000", 2020-01-16T00:12:52.157 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013066217 2020-01-16T00:12:52.157 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.157 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.157 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.157 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.860522+0000", 2020-01-16T00:12:52.157 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.2278000000000001e-05 2020-01-16T00:12:52.157 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.157 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.158 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:52.158 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.860605+0000", 2020-01-16T00:12:52.158 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001147779 2020-01-16T00:12:52.158 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.158 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.158 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.158 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.861752+0000", 2020-01-16T00:12:52.159 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4598000000000006e-05 2020-01-16T00:12:52.159 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.159 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.159 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.159 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.861827+0000", 2020-01-16T00:12:52.159 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8060999999999995e-05 2020-01-16T00:12:52.159 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.159 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.160 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.160 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.861925+0000", 2020-01-16T00:12:52.160 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.024296113000000001 2020-01-16T00:12:52.160 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.160 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.160 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.160 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.161 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.161 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:914 3.c 3:3d6543e6:::smithi12313934-28:head [write 426911~540354,stat] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:52.161 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.842336+0000", 2020-01-16T00:12:52.161 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.242737452, 2020-01-16T00:12:52.161 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021922343, 2020-01-16T00:12:52.161 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.161 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:52.162 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.162 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.162 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.162 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 914 2020-01-16T00:12:52.162 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.162 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.163 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.163 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.163 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.842336+0000", 2020-01-16T00:12:52.163 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.163 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.163 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.164 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.164 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.842336+0000", 2020-01-16T00:12:52.164 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:12:52.164 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.164 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.164 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.164 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.842333+0000", 2020-01-16T00:12:52.165 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00063242099999999998 2020-01-16T00:12:52.165 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.165 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.165 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.165 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.842965+0000", 2020-01-16T00:12:52.165 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.8354999999999999e-05 2020-01-16T00:12:52.165 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.166 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.166 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.166 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.842994+0000", 2020-01-16T00:12:52.166 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1077000000000001e-05 2020-01-16T00:12:52.166 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.167 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.167 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.167 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843005+0000", 2020-01-16T00:12:52.167 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000107552 2020-01-16T00:12:52.168 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.168 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.168 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.168 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843112+0000", 2020-01-16T00:12:52.168 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00037837800000000002 2020-01-16T00:12:52.169 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.169 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.169 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:52.169 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843491+0000", 2020-01-16T00:12:52.170 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0092250490000000008 2020-01-16T00:12:52.170 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.170 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.170 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.170 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.852716+0000", 2020-01-16T00:12:52.170 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018407699999999999 2020-01-16T00:12:52.171 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.171 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.171 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:52.171 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.852900+0000", 2020-01-16T00:12:52.171 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0078355379999999995 2020-01-16T00:12:52.172 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.172 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.172 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.172 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.860735+0000", 2020-01-16T00:12:52.172 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012386100000000001 2020-01-16T00:12:52.172 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.173 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.173 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.173 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.860859+0000", 2020-01-16T00:12:52.173 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014735099999999999 2020-01-16T00:12:52.173 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.174 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.174 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2020-01-16T00:12:52.174 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.861006+0000", 2020-01-16T00:12:52.174 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0027785040000000002 2020-01-16T00:12:52.174 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.174 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.174 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:52.174 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.863785+0000", 2020-01-16T00:12:52.175 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000127083 2020-01-16T00:12:52.175 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.175 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.175 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:52.175 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.863912+0000", 2020-01-16T00:12:52.175 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000179106 2020-01-16T00:12:52.175 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.175 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.176 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:52.176 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.864091+0000", 2020-01-16T00:12:52.176 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7744999999999998e-05 2020-01-16T00:12:52.176 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.176 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.176 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:52.176 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.864139+0000", 2020-01-16T00:12:52.177 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011914200000000001 2020-01-16T00:12:52.177 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.177 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.177 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.177 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.864258+0000", 2020-01-16T00:12:52.177 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021922343 2020-01-16T00:12:52.177 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.177 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.178 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.178 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.178 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.178 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:915 3.c 3:3d6543e6:::smithi12313934-28:head [write 1622009~726683] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:52.178 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.850793+0000", 2020-01-16T00:12:52.178 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.234280246999999, 2020-01-16T00:12:52.179 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018526131000000001, 2020-01-16T00:12:52.179 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.179 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:52.179 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.179 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.179 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.179 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 915 2020-01-16T00:12:52.180 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.180 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.180 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.180 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.180 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.850793+0000", 2020-01-16T00:12:52.181 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.181 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.181 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.181 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.181 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.850793+0000", 2020-01-16T00:12:52.182 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999957 2020-01-16T00:12:52.182 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.182 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.182 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.182 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.850788+0000", 2020-01-16T00:12:52.182 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0022032459999999999 2020-01-16T00:12:52.183 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.183 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.183 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.183 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.852992+0000", 2020-01-16T00:12:52.183 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6056999999999999e-05 2020-01-16T00:12:52.184 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.184 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.184 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.184 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.853008+0000", 2020-01-16T00:12:52.184 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5619999999999993e-06 2020-01-16T00:12:52.184 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.185 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.185 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.185 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.853017+0000", 2020-01-16T00:12:52.185 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9026000000000002e-05 2020-01-16T00:12:52.185 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.185 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.185 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.185 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.853056+0000", 2020-01-16T00:12:52.186 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.1551000000000005e-05 2020-01-16T00:12:52.186 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.186 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.186 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:52.186 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.853148+0000", 2020-01-16T00:12:52.186 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0091288280000000003 2020-01-16T00:12:52.186 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.187 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.187 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.187 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.862277+0000", 2020-01-16T00:12:52.187 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000101439 2020-01-16T00:12:52.187 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.187 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.187 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:52.187 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.862378+0000", 2020-01-16T00:12:52.188 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001920597 2020-01-16T00:12:52.188 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.188 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.188 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.188 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.864299+0000", 2020-01-16T00:12:52.188 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000129355 2020-01-16T00:12:52.188 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.188 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.189 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.189 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.864428+0000", 2020-01-16T00:12:52.189 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000226999 2020-01-16T00:12:52.189 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.189 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.189 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2020-01-16T00:12:52.189 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.864655+0000", 2020-01-16T00:12:52.190 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0036207840000000002 2020-01-16T00:12:52.190 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.190 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.190 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:52.190 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.868276+0000", 2020-01-16T00:12:52.190 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4215999999999996e-05 2020-01-16T00:12:52.190 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.190 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.191 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:52.191 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.868350+0000", 2020-01-16T00:12:52.191 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00083996500000000003 2020-01-16T00:12:52.191 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.191 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.192 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:52.192 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.869190+0000", 2020-01-16T00:12:52.192 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0606999999999998e-05 2020-01-16T00:12:52.192 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.193 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.193 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:52.193 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.869251+0000", 2020-01-16T00:12:52.193 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8396e-05 2020-01-16T00:12:52.194 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.194 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.194 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.195 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.869319+0000", 2020-01-16T00:12:52.195 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018526131000000001 2020-01-16T00:12:52.195 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.195 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.195 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.195 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.196 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.196 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:916 3.c 3:3d6543e6:::smithi12313934-28:head [write 3121996~104915] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:52.196 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.855949+0000", 2020-01-16T00:12:52.196 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.229124107000001, 2020-01-16T00:12:52.196 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013528775, 2020-01-16T00:12:52.196 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.196 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:52.197 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.197 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.197 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.197 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 916 2020-01-16T00:12:52.197 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.197 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.197 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.198 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.198 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.855949+0000", 2020-01-16T00:12:52.198 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.198 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.198 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.198 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.198 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.855949+0000", 2020-01-16T00:12:52.198 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:52.199 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.199 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.199 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.199 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.855947+0000", 2020-01-16T00:12:52.199 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013682200000000001 2020-01-16T00:12:52.199 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.200 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.200 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.200 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856084+0000", 2020-01-16T00:12:52.200 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7864999999999999e-05 2020-01-16T00:12:52.200 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.200 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.200 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.201 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856102+0000", 2020-01-16T00:12:52.201 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.8699999999999998e-06 2020-01-16T00:12:52.201 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.201 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.201 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.201 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856111+0000", 2020-01-16T00:12:52.201 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.299e-05 2020-01-16T00:12:52.202 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.202 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.202 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.202 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856154+0000", 2020-01-16T00:12:52.202 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0636999999999994e-05 2020-01-16T00:12:52.202 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.203 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.203 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:52.203 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856234+0000", 2020-01-16T00:12:52.203 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0061827920000000003 2020-01-16T00:12:52.203 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.204 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.204 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.204 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.862417+0000", 2020-01-16T00:12:52.204 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010285099999999999 2020-01-16T00:12:52.204 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.204 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.205 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:52.205 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.862520+0000", 2020-01-16T00:12:52.205 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0038991479999999999 2020-01-16T00:12:52.205 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.206 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.206 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.206 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.866419+0000", 2020-01-16T00:12:52.206 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010513700000000001 2020-01-16T00:12:52.206 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.206 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.207 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.207 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.866524+0000", 2020-01-16T00:12:52.207 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014972299999999999 2020-01-16T00:12:52.207 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.207 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.207 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2020-01-16T00:12:52.207 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.866674+0000", 2020-01-16T00:12:52.208 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001618415 2020-01-16T00:12:52.208 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.208 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.208 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:52.208 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.868292+0000", 2020-01-16T00:12:52.208 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000297736 2020-01-16T00:12:52.208 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.209 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.209 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:52.209 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.868590+0000", 2020-01-16T00:12:52.209 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000798283 2020-01-16T00:12:52.209 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.209 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.210 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:52.210 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.869388+0000", 2020-01-16T00:12:52.210 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5621000000000002e-05 2020-01-16T00:12:52.210 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.210 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.210 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:52.210 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.869434+0000", 2020-01-16T00:12:52.211 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3894999999999999e-05 2020-01-16T00:12:52.211 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.211 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.211 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.211 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.869478+0000", 2020-01-16T00:12:52.211 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013528775 2020-01-16T00:12:52.211 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.211 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.212 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.212 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.212 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.212 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:917 3.c 3:3d6543e6:::smithi12313934-28:head [setxattr _header (56) in=7b,truncate 3226911] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:52.212 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.856125+0000", 2020-01-16T00:12:52.212 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.228947896999999, 2020-01-16T00:12:52.212 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013497781, 2020-01-16T00:12:52.213 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.213 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:52.213 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.213 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.213 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.214 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 917 2020-01-16T00:12:52.214 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.214 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.214 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.214 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.215 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856125+0000", 2020-01-16T00:12:52.215 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.215 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.215 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.215 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.215 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856125+0000", 2020-01-16T00:12:52.215 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:52.215 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.216 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.216 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.216 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856124+0000", 2020-01-16T00:12:52.216 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8859999999999998e-06 2020-01-16T00:12:52.217 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.217 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.217 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.217 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856131+0000", 2020-01-16T00:12:52.217 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6570000000000003e-06 2020-01-16T00:12:52.217 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.217 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.218 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.218 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856140+0000", 2020-01-16T00:12:52.218 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5420000000000002e-06 2020-01-16T00:12:52.218 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.218 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.218 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.218 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856146+0000", 2020-01-16T00:12:52.219 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012714399999999999 2020-01-16T00:12:52.219 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.219 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.219 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.219 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856273+0000", 2020-01-16T00:12:52.219 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000122113 2020-01-16T00:12:52.219 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.219 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.220 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:52.220 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856395+0000", 2020-01-16T00:12:52.220 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.006160708 2020-01-16T00:12:52.220 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.220 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.220 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.220 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.862556+0000", 2020-01-16T00:12:52.220 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7220999999999999e-05 2020-01-16T00:12:52.221 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.221 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.221 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:52.221 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.862623+0000", 2020-01-16T00:12:52.221 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0046116050000000004 2020-01-16T00:12:52.221 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.221 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.222 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.222 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.867235+0000", 2020-01-16T00:12:52.222 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019234499999999999 2020-01-16T00:12:52.222 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.222 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.222 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.222 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.867427+0000", 2020-01-16T00:12:52.222 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022820699999999999 2020-01-16T00:12:52.223 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.223 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.223 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2020-01-16T00:12:52.223 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.867656+0000", 2020-01-16T00:12:52.224 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00065250900000000001 2020-01-16T00:12:52.224 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.224 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.224 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:52.224 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.868308+0000", 2020-01-16T00:12:52.224 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00043943999999999998 2020-01-16T00:12:52.225 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.225 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.225 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:52.225 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.868748+0000", 2020-01-16T00:12:52.225 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00078105600000000002 2020-01-16T00:12:52.225 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.225 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.226 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:52.226 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.869529+0000", 2020-01-16T00:12:52.226 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2976000000000003e-05 2020-01-16T00:12:52.226 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.226 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.226 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:52.227 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.869572+0000", 2020-01-16T00:12:52.227 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1439e-05 2020-01-16T00:12:52.227 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.227 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.227 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.227 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.869623+0000", 2020-01-16T00:12:52.228 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013497781 2020-01-16T00:12:52.228 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.228 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.228 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.228 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.228 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.228 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:918 3.c 3:3d6543e6:::smithi12313934-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:12:52.228 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.856157+0000", 2020-01-16T00:12:52.229 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.228916071, 2020-01-16T00:12:52.229 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013685233, 2020-01-16T00:12:52.229 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.229 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:52.229 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.229 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.229 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.230 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 918 2020-01-16T00:12:52.230 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.230 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.230 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.230 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.230 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856157+0000", 2020-01-16T00:12:52.230 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.231 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.231 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.231 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.231 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856157+0000", 2020-01-16T00:12:52.231 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:52.231 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.231 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.231 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.232 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856156+0000", 2020-01-16T00:12:52.232 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7450000000000003e-06 2020-01-16T00:12:52.232 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.232 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.232 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.232 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856161+0000", 2020-01-16T00:12:52.232 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5880000000000004e-06 2020-01-16T00:12:52.233 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.233 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.233 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.233 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856171+0000", 2020-01-16T00:12:52.233 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1620000000000002e-06 2020-01-16T00:12:52.233 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.233 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.234 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.234 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856178+0000", 2020-01-16T00:12:52.234 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000256796 2020-01-16T00:12:52.234 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.234 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.235 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.235 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856435+0000", 2020-01-16T00:12:52.235 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.8992000000000005e-05 2020-01-16T00:12:52.235 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.235 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.236 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:52.236 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.856514+0000", 2020-01-16T00:12:52.236 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0061401370000000004 2020-01-16T00:12:52.236 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.236 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.236 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.236 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.862654+0000", 2020-01-16T00:12:52.237 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2589000000000007e-05 2020-01-16T00:12:52.237 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.237 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.237 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:52.237 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.862716+0000", 2020-01-16T00:12:52.237 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0053446409999999998 2020-01-16T00:12:52.238 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.238 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.238 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.238 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.868061+0000", 2020-01-16T00:12:52.238 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019486699999999999 2020-01-16T00:12:52.238 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.238 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.239 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:52.239 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.868256+0000", 2020-01-16T00:12:52.239 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014152170000000001 2020-01-16T00:12:52.239 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.239 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.239 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.239 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.869671+0000", 2020-01-16T00:12:52.240 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.2387999999999998e-05 2020-01-16T00:12:52.240 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.240 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.240 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.240 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.869753+0000", 2020-01-16T00:12:52.240 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.8926999999999997e-05 2020-01-16T00:12:52.240 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.241 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.241 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.241 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.869842+0000", 2020-01-16T00:12:52.241 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013685233 2020-01-16T00:12:52.241 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.241 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.241 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.242 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.242 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.242 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:952 3.c 3:3877922e:::smithi12313934-31:head [write 494139~557703,stat] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:52.242 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.923660+0000", 2020-01-16T00:12:52.242 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.161413293000001, 2020-01-16T00:12:52.242 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016369574000000001, 2020-01-16T00:12:52.242 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.243 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:52.243 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.243 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.243 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.243 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 952 2020-01-16T00:12:52.243 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.243 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.243 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.244 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.244 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.923660+0000", 2020-01-16T00:12:52.244 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.244 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.244 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.245 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.245 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.923660+0000", 2020-01-16T00:12:52.245 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:52.245 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.246 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.246 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.246 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.923659+0000", 2020-01-16T00:12:52.246 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0015925539999999999 2020-01-16T00:12:52.246 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.246 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.247 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.247 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.925251+0000", 2020-01-16T00:12:52.247 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4825e-05 2020-01-16T00:12:52.247 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.247 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.247 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.247 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.925276+0000", 2020-01-16T00:12:52.248 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.217e-06 2020-01-16T00:12:52.248 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.248 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.248 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.248 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.925285+0000", 2020-01-16T00:12:52.248 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8985000000000004e-05 2020-01-16T00:12:52.248 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.249 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.250 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.250 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.925354+0000", 2020-01-16T00:12:52.250 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022915199999999999 2020-01-16T00:12:52.250 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.250 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.250 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:52.250 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.925583+0000", 2020-01-16T00:12:52.251 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0072044309999999999 2020-01-16T00:12:52.251 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.251 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.251 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.251 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.932788+0000", 2020-01-16T00:12:52.251 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2225999999999994e-05 2020-01-16T00:12:52.251 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.251 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.252 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:52.252 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.932880+0000", 2020-01-16T00:12:52.252 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0040176580000000003 2020-01-16T00:12:52.252 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.252 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.252 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.252 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.936898+0000", 2020-01-16T00:12:52.252 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9100999999999999e-05 2020-01-16T00:12:52.253 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.253 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.253 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.253 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.936997+0000", 2020-01-16T00:12:52.253 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00021074099999999999 2020-01-16T00:12:52.253 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.253 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.253 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2020-01-16T00:12:52.254 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.937207+0000", 2020-01-16T00:12:52.254 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001814979 2020-01-16T00:12:52.254 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.254 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.254 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:52.254 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939022+0000", 2020-01-16T00:12:52.255 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014921699999999999 2020-01-16T00:12:52.255 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.255 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.255 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:52.255 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939172+0000", 2020-01-16T00:12:52.255 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00070305200000000002 2020-01-16T00:12:52.255 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.256 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.256 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:52.256 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939875+0000", 2020-01-16T00:12:52.256 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.9981999999999994e-05 2020-01-16T00:12:52.256 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.256 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.256 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:52.257 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939955+0000", 2020-01-16T00:12:52.257 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4770000000000004e-05 2020-01-16T00:12:52.257 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.257 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.257 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.257 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.940029+0000", 2020-01-16T00:12:52.257 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016369574000000001 2020-01-16T00:12:52.257 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.258 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.258 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.258 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.258 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.258 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:953 3.c 3:3877922e:::smithi12313934-31:head [write 1455937~734021] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:52.258 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.931235+0000", 2020-01-16T00:12:52.258 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.153838085, 2020-01-16T00:12:52.259 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.022666925000000001, 2020-01-16T00:12:52.259 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.259 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:52.259 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.259 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.259 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.259 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 953 2020-01-16T00:12:52.259 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.260 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.260 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.260 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.260 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.931235+0000", 2020-01-16T00:12:52.260 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.260 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.260 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.260 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.261 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.931235+0000", 2020-01-16T00:12:52.261 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:52.261 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.261 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.261 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.261 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.931233+0000", 2020-01-16T00:12:52.261 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0013294019999999999 2020-01-16T00:12:52.261 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.262 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.262 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.262 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.932562+0000", 2020-01-16T00:12:52.262 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0194e-05 2020-01-16T00:12:52.262 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.262 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.262 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.262 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.932582+0000", 2020-01-16T00:12:52.263 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.199e-06 2020-01-16T00:12:52.263 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.263 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.263 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.263 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.932589+0000", 2020-01-16T00:12:52.263 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033658000000000001 2020-01-16T00:12:52.263 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.263 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.264 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.264 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.932926+0000", 2020-01-16T00:12:52.264 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000174749 2020-01-16T00:12:52.264 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.264 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.264 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:52.264 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933101+0000", 2020-01-16T00:12:52.265 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0054526330000000001 2020-01-16T00:12:52.265 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.265 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.265 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.265 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.938553+0000", 2020-01-16T00:12:52.265 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00026010999999999997 2020-01-16T00:12:52.266 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.266 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.266 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:52.266 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.938813+0000", 2020-01-16T00:12:52.266 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012019494 2020-01-16T00:12:52.266 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.266 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.266 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.267 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.950833+0000", 2020-01-16T00:12:52.267 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6416999999999997e-05 2020-01-16T00:12:52.267 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.267 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.267 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.267 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.950909+0000", 2020-01-16T00:12:52.267 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000152855 2020-01-16T00:12:52.267 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.268 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.268 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2020-01-16T00:12:52.268 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.951062+0000", 2020-01-16T00:12:52.268 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0025050680000000001 2020-01-16T00:12:52.268 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.268 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.268 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:52.269 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953567+0000", 2020-01-16T00:12:52.269 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017207999999999999 2020-01-16T00:12:52.269 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.269 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.269 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:52.269 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953739+0000", 2020-01-16T00:12:52.270 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1077999999999999e-05 2020-01-16T00:12:52.270 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.270 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.270 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:52.270 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953820+0000", 2020-01-16T00:12:52.270 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1516e-05 2020-01-16T00:12:52.270 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.270 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.271 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:52.271 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953862+0000", 2020-01-16T00:12:52.271 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0030999999999997e-05 2020-01-16T00:12:52.271 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.271 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.271 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.271 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953902+0000", 2020-01-16T00:12:52.272 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.022666925000000001 2020-01-16T00:12:52.272 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.272 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.272 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.272 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.273 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.273 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:954 3.c 3:3877922e:::smithi12313934-31:head [write 2948041~346098] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:52.273 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.940459+0000", 2020-01-16T00:12:52.273 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.144614423, 2020-01-16T00:12:52.273 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023904965, 2020-01-16T00:12:52.273 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.273 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:52.274 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.274 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.274 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.274 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 954 2020-01-16T00:12:52.274 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.274 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.274 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.274 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.275 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.940459+0000", 2020-01-16T00:12:52.275 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.275 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.275 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.275 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.275 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.940459+0000", 2020-01-16T00:12:52.276 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:52.276 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.276 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.276 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.276 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.940457+0000", 2020-01-16T00:12:52.276 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00067923600000000005 2020-01-16T00:12:52.276 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.277 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.277 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.277 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941136+0000", 2020-01-16T00:12:52.277 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6631999999999999e-05 2020-01-16T00:12:52.277 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.277 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.277 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.277 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941152+0000", 2020-01-16T00:12:52.278 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9340000000000003e-06 2020-01-16T00:12:52.278 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.278 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.278 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.278 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941158+0000", 2020-01-16T00:12:52.278 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8878000000000001e-05 2020-01-16T00:12:52.278 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.278 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.279 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.279 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941217+0000", 2020-01-16T00:12:52.279 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1224999999999997e-05 2020-01-16T00:12:52.279 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.279 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.279 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:52.279 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941289+0000", 2020-01-16T00:12:52.279 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011199931999999999 2020-01-16T00:12:52.280 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.280 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.280 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.280 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.952488+0000", 2020-01-16T00:12:52.280 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000240067 2020-01-16T00:12:52.280 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.280 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.280 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:52.281 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.952729+0000", 2020-01-16T00:12:52.281 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0094184660000000003 2020-01-16T00:12:52.281 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.281 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.281 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.281 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.962147+0000", 2020-01-16T00:12:52.281 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6094000000000006e-05 2020-01-16T00:12:52.281 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.282 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.282 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.282 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.962233+0000", 2020-01-16T00:12:52.282 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022739400000000001 2020-01-16T00:12:52.282 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.282 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.282 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2020-01-16T00:12:52.282 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.962460+0000", 2020-01-16T00:12:52.283 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014799710000000001 2020-01-16T00:12:52.283 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.283 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.284 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:52.285 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.963940+0000", 2020-01-16T00:12:52.285 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000279182 2020-01-16T00:12:52.285 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.285 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.285 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:52.285 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.964220+0000", 2020-01-16T00:12:52.286 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4141999999999997e-05 2020-01-16T00:12:52.286 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.286 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.286 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:52.286 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.964284+0000", 2020-01-16T00:12:52.286 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3547999999999997e-05 2020-01-16T00:12:52.286 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.287 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.287 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:52.287 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.964327+0000", 2020-01-16T00:12:52.287 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6341000000000003e-05 2020-01-16T00:12:52.287 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.287 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.287 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.287 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.964364+0000", 2020-01-16T00:12:52.288 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023904965 2020-01-16T00:12:52.288 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.288 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.288 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.288 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.288 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.288 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:955 3.c 3:3877922e:::smithi12313934-31:head [setxattr _header (56) in=7b,truncate 3294139] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:52.288 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.941169+0000", 2020-01-16T00:12:52.289 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.143904376, 2020-01-16T00:12:52.289 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023400259999999999, 2020-01-16T00:12:52.289 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.289 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:52.289 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.289 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.289 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.289 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 955 2020-01-16T00:12:52.290 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.290 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.290 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.290 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.290 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941169+0000", 2020-01-16T00:12:52.290 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.290 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.290 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.291 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.291 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941169+0000", 2020-01-16T00:12:52.291 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:52.291 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.291 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.291 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.291 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941168+0000", 2020-01-16T00:12:52.291 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.6700000000000006e-06 2020-01-16T00:12:52.292 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.292 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.292 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.292 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941177+0000", 2020-01-16T00:12:52.292 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6420000000000001e-06 2020-01-16T00:12:52.292 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.292 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.292 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.293 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941184+0000", 2020-01-16T00:12:52.293 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5019999999999998e-06 2020-01-16T00:12:52.293 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.293 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.293 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.293 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941190+0000", 2020-01-16T00:12:52.293 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013520700000000001 2020-01-16T00:12:52.293 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.294 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.294 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.294 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941326+0000", 2020-01-16T00:12:52.294 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1218999999999998e-05 2020-01-16T00:12:52.294 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.294 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.294 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:52.294 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941387+0000", 2020-01-16T00:12:52.295 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011403931000000001 2020-01-16T00:12:52.295 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.295 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.295 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.295 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.952791+0000", 2020-01-16T00:12:52.295 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6416999999999998e-05 2020-01-16T00:12:52.296 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.296 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.296 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:52.296 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.952857+0000", 2020-01-16T00:12:52.296 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010427334999999999 2020-01-16T00:12:52.296 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.296 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.297 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.297 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.963284+0000", 2020-01-16T00:12:52.297 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.9720000000000002e-05 2020-01-16T00:12:52.297 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.297 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.297 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.297 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.963374+0000", 2020-01-16T00:12:52.297 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000131137 2020-01-16T00:12:52.298 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.298 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.298 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2020-01-16T00:12:52.298 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.963505+0000", 2020-01-16T00:12:52.298 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00065508899999999998 2020-01-16T00:12:52.298 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.298 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.298 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:52.299 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.964160+0000", 2020-01-16T00:12:52.299 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000279265 2020-01-16T00:12:52.299 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.299 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.299 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:52.299 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.964440+0000", 2020-01-16T00:12:52.299 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5080999999999997e-05 2020-01-16T00:12:52.299 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.300 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.300 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:52.300 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.964485+0000", 2020-01-16T00:12:52.300 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0565999999999998e-05 2020-01-16T00:12:52.300 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.300 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.300 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:52.301 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.964525+0000", 2020-01-16T00:12:52.301 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3662999999999999e-05 2020-01-16T00:12:52.301 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.301 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.301 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.301 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.964569+0000", 2020-01-16T00:12:52.301 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023400259999999999 2020-01-16T00:12:52.302 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.302 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.302 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.302 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.302 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.302 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:956 3.c 3:3877922e:::smithi12313934-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:12:52.302 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.941200+0000", 2020-01-16T00:12:52.303 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.143873524, 2020-01-16T00:12:52.303 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023677922000000001, 2020-01-16T00:12:52.303 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.303 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:52.303 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.303 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.303 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.303 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 956 2020-01-16T00:12:52.304 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.304 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.304 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.304 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.304 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941200+0000", 2020-01-16T00:12:52.304 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.304 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.305 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.305 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.305 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941200+0000", 2020-01-16T00:12:52.305 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:52.305 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.305 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.305 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.306 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941199+0000", 2020-01-16T00:12:52.306 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9330000000000003e-06 2020-01-16T00:12:52.306 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.306 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.306 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.307 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941203+0000", 2020-01-16T00:12:52.307 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1578000000000002e-05 2020-01-16T00:12:52.307 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.307 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.307 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.307 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941234+0000", 2020-01-16T00:12:52.308 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2530000000000004e-06 2020-01-16T00:12:52.308 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.308 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.308 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.308 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941240+0000", 2020-01-16T00:12:52.309 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016458900000000001 2020-01-16T00:12:52.309 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.309 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.309 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.309 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941404+0000", 2020-01-16T00:12:52.309 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3485000000000002e-05 2020-01-16T00:12:52.309 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.309 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.310 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:52.310 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.941438+0000", 2020-01-16T00:12:52.310 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011489497 2020-01-16T00:12:52.310 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.310 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.310 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.310 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.952927+0000", 2020-01-16T00:12:52.310 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4110999999999999e-05 2020-01-16T00:12:52.311 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.311 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.311 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:52.311 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.952961+0000", 2020-01-16T00:12:52.311 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010832766000000001 2020-01-16T00:12:52.311 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.311 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.311 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.312 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.963794+0000", 2020-01-16T00:12:52.312 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012301900000000001 2020-01-16T00:12:52.312 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.312 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.312 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:52.312 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.963917+0000", 2020-01-16T00:12:52.312 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00069668799999999999 2020-01-16T00:12:52.312 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.313 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.313 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.313 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.964614+0000", 2020-01-16T00:12:52.313 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012451500000000001 2020-01-16T00:12:52.313 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.313 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.313 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.313 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.964738+0000", 2020-01-16T00:12:52.314 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000139128 2020-01-16T00:12:52.314 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.314 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.314 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.314 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.964878+0000", 2020-01-16T00:12:52.314 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023677922000000001 2020-01-16T00:12:52.314 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.314 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.315 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.315 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.315 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.315 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:979 3.5 3:ad8ab15f:::smithi12313934-27:head [read 0~3210104,checksum] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:52.315 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315954+0000", 2020-01-16T00:12:52.315 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 13.769119402999999, 2020-01-16T00:12:52.315 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010639968, 2020-01-16T00:12:52.316 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.316 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:52.316 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.316 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.316 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.316 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 979 2020-01-16T00:12:52.317 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.317 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.317 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.317 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.317 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315954+0000", 2020-01-16T00:12:52.317 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.317 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.317 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.318 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.318 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315954+0000", 2020-01-16T00:12:52.318 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:52.318 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.318 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.319 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.319 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315953+0000", 2020-01-16T00:12:52.319 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.1200000000000008e-06 2020-01-16T00:12:52.319 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.319 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.319 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.319 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315962+0000", 2020-01-16T00:12:52.319 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.488e-05 2020-01-16T00:12:52.320 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.320 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.320 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.320 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315977+0000", 2020-01-16T00:12:52.320 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6283e-05 2020-01-16T00:12:52.320 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.320 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.320 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.321 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315993+0000", 2020-01-16T00:12:52.321 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0079730960000000007 2020-01-16T00:12:52.321 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.321 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.321 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.321 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.323966+0000", 2020-01-16T00:12:52.321 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020471399999999999 2020-01-16T00:12:52.322 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.322 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.322 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.322 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.324171+0000", 2020-01-16T00:12:52.322 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0024230290000000002 2020-01-16T00:12:52.322 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.322 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.322 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.323 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.326594+0000", 2020-01-16T00:12:52.323 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010639968 2020-01-16T00:12:52.323 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.323 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.323 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.323 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.323 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.323 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:980 3.5 3:ad8ab15f:::smithi12313934-27:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:52.324 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.316004+0000", 2020-01-16T00:12:52.324 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 13.769068990999999, 2020-01-16T00:12:52.324 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011373866, 2020-01-16T00:12:52.324 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.324 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:52.324 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.325 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.325 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.325 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 980 2020-01-16T00:12:52.325 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.325 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.325 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.325 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.326 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316004+0000", 2020-01-16T00:12:52.326 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.326 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.326 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.327 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.327 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316004+0000", 2020-01-16T00:12:52.327 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:52.327 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.327 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.327 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.327 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316003+0000", 2020-01-16T00:12:52.327 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.9419999999999998e-06 2020-01-16T00:12:52.328 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.328 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.328 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.328 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316010+0000", 2020-01-16T00:12:52.328 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4960000000000001e-05 2020-01-16T00:12:52.329 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.329 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.329 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.329 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316025+0000", 2020-01-16T00:12:52.329 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2890000000000003e-06 2020-01-16T00:12:52.329 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.330 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.330 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.330 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316034+0000", 2020-01-16T00:12:52.330 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010653673000000001 2020-01-16T00:12:52.330 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.330 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.330 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.330 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.326688+0000", 2020-01-16T00:12:52.331 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000231511 2020-01-16T00:12:52.331 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.331 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.331 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.331 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.326919+0000", 2020-01-16T00:12:52.331 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00045858800000000001 2020-01-16T00:12:52.331 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.331 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.332 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.332 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.327378+0000", 2020-01-16T00:12:52.332 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011373866 2020-01-16T00:12:52.332 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.332 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.332 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.332 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.332 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.333 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.1.13:47 2.3 2:c6fec7d0:::smithi12313934-26 low_tier:head [write 963072~376320] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e26)", 2020-01-16T00:12:52.333 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:41.673013+0000", 2020-01-16T00:12:52.333 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 10.412059851, 2020-01-16T00:12:52.333 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0095009329999999996, 2020-01-16T00:12:52.333 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.333 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:52.333 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.333 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.334 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.334 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.673013+0000", 2020-01-16T00:12:52.334 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.334 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.334 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.334 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.334 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.673013+0000", 2020-01-16T00:12:52.334 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:52.335 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.335 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.335 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.335 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.673011+0000", 2020-01-16T00:12:52.335 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000590852 2020-01-16T00:12:52.335 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.335 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.335 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.336 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.673602+0000", 2020-01-16T00:12:52.336 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9289999999999998e-05 2020-01-16T00:12:52.336 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.336 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.336 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.336 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.673641+0000", 2020-01-16T00:12:52.337 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0559e-05 2020-01-16T00:12:52.337 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.337 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.337 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.337 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.673652+0000", 2020-01-16T00:12:52.337 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5334999999999997e-05 2020-01-16T00:12:52.338 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.338 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.338 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.338 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.673717+0000", 2020-01-16T00:12:52.338 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000109041 2020-01-16T00:12:52.338 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.338 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.338 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.339 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.673826+0000", 2020-01-16T00:12:52.339 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015066 2020-01-16T00:12:52.339 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.339 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.339 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:12:52.339 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.673977+0000", 2020-01-16T00:12:52.339 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0082881929999999993 2020-01-16T00:12:52.339 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.340 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.340 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:52.340 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.682265+0000", 2020-01-16T00:12:52.340 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.9645000000000002e-05 2020-01-16T00:12:52.340 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.340 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.340 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:52.340 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.682334+0000", 2020-01-16T00:12:52.341 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5344000000000001e-05 2020-01-16T00:12:52.341 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.341 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.341 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:52.341 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.682350+0000", 2020-01-16T00:12:52.341 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6589999999999997e-05 2020-01-16T00:12:52.342 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.342 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.342 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:52.342 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.682426+0000", 2020-01-16T00:12:52.342 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7816000000000001e-05 2020-01-16T00:12:52.342 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.342 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.342 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.343 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:41.682514+0000", 2020-01-16T00:12:52.343 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0095009329999999996 2020-01-16T00:12:52.343 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.343 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.343 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.343 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.343 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.343 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1442 3.5 3:a269d636:::smithi12313934-32:head [write 1584925~506039] snapc 0=[] ondisk+write+known_if_redirected e28)", 2020-01-16T00:12:52.344 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:47.913242+0000", 2020-01-16T00:12:52.344 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 4.1718312859999997, 2020-01-16T00:12:52.344 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0094839199999999999, 2020-01-16T00:12:52.344 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.344 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:52.344 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.344 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.344 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.345 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1442 2020-01-16T00:12:52.345 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.345 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.345 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.345 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.345 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.913242+0000", 2020-01-16T00:12:52.345 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.346 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.346 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.346 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.346 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.913242+0000", 2020-01-16T00:12:52.346 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:52.346 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.347 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.347 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.347 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.913240+0000", 2020-01-16T00:12:52.347 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00062529100000000004 2020-01-16T00:12:52.347 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.348 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.348 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.348 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.913865+0000", 2020-01-16T00:12:52.348 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4123e-05 2020-01-16T00:12:52.348 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.348 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.348 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.349 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.913879+0000", 2020-01-16T00:12:52.349 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0419999999999999e-06 2020-01-16T00:12:52.349 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.349 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.349 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.349 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.913885+0000", 2020-01-16T00:12:52.349 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4945999999999999e-05 2020-01-16T00:12:52.350 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.350 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.350 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.350 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.913940+0000", 2020-01-16T00:12:52.350 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000126277 2020-01-16T00:12:52.351 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.351 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.351 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.351 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.914067+0000", 2020-01-16T00:12:52.351 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019388200000000001 2020-01-16T00:12:52.351 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.351 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.351 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,3", 2020-01-16T00:12:52.352 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.914261+0000", 2020-01-16T00:12:52.352 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0074386319999999997 2020-01-16T00:12:52.352 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.352 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.352 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:52.352 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.921699+0000", 2020-01-16T00:12:52.352 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00085258199999999995 2020-01-16T00:12:52.352 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.353 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.353 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:52.353 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.922552+0000", 2020-01-16T00:12:52.353 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0529999999999998e-05 2020-01-16T00:12:52.353 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.353 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.353 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:52.354 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.922612+0000", 2020-01-16T00:12:52.354 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5325000000000003e-05 2020-01-16T00:12:52.354 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.354 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.354 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:52.354 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.922668+0000", 2020-01-16T00:12:52.354 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8094999999999999e-05 2020-01-16T00:12:52.354 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.355 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.355 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.355 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.922726+0000", 2020-01-16T00:12:52.355 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0094839199999999999 2020-01-16T00:12:52.355 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.355 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.355 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.355 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.356 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.356 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1444 3.5 3:a269d636:::smithi12313934-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e28)", 2020-01-16T00:12:52.356 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:47.913943+0000", 2020-01-16T00:12:52.356 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 4.171130336, 2020-01-16T00:12:52.356 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0094638180000000006, 2020-01-16T00:12:52.356 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.357 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:52.357 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.357 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.357 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.357 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1444 2020-01-16T00:12:52.358 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.358 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.358 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.359 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.359 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.913943+0000", 2020-01-16T00:12:52.359 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.359 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.359 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.360 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.360 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.913943+0000", 2020-01-16T00:12:52.360 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:52.360 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.360 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.360 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.360 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.913941+0000", 2020-01-16T00:12:52.360 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4161e-05 2020-01-16T00:12:52.361 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.361 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.361 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.361 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.913955+0000", 2020-01-16T00:12:52.361 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0210999999999999e-05 2020-01-16T00:12:52.361 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.361 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.362 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.362 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.913965+0000", 2020-01-16T00:12:52.362 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2429999999999996e-06 2020-01-16T00:12:52.362 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.362 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.362 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.362 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.913971+0000", 2020-01-16T00:12:52.362 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0025181219999999998 2020-01-16T00:12:52.363 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.363 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.363 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.363 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.916489+0000", 2020-01-16T00:12:52.363 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.4636999999999996e-05 2020-01-16T00:12:52.363 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.363 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.363 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:52.364 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.916573+0000", 2020-01-16T00:12:52.364 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0065288619999999999 2020-01-16T00:12:52.364 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.364 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.364 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.364 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.923102+0000", 2020-01-16T00:12:52.364 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013211599999999999 2020-01-16T00:12:52.365 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.365 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.365 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.365 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.923234+0000", 2020-01-16T00:12:52.365 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017227799999999999 2020-01-16T00:12:52.365 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.365 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.366 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.366 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.923407+0000", 2020-01-16T00:12:52.366 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0094638180000000006 2020-01-16T00:12:52.366 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.366 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.366 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.367 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.367 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.367 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1453 3.5 3:aef1ce98:::smithi12313934-9:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e28)", 2020-01-16T00:12:52.367 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:47.914015+0000", 2020-01-16T00:12:52.367 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 4.1710581710000003, 2020-01-16T00:12:52.367 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0099596319999999995, 2020-01-16T00:12:52.367 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.368 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:52.368 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.368 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.368 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.368 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1453 2020-01-16T00:12:52.368 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.368 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.368 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.369 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.369 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.914015+0000", 2020-01-16T00:12:52.369 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.369 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.369 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.369 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.370 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.914015+0000", 2020-01-16T00:12:52.370 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:52.370 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.370 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.370 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.370 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.914015+0000", 2020-01-16T00:12:52.370 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.322e-06 2020-01-16T00:12:52.371 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.371 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.371 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.371 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.914017+0000", 2020-01-16T00:12:52.371 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4570000000000001e-06 2020-01-16T00:12:52.371 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.371 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.371 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.372 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.914020+0000", 2020-01-16T00:12:52.372 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3819999999999998e-06 2020-01-16T00:12:52.372 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.372 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.372 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.372 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.914023+0000", 2020-01-16T00:12:52.372 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0094306679999999997 2020-01-16T00:12:52.373 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.373 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.373 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.373 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.923453+0000", 2020-01-16T00:12:52.373 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000117725 2020-01-16T00:12:52.373 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.373 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.373 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.374 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.923571+0000", 2020-01-16T00:12:52.374 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00040342299999999998 2020-01-16T00:12:52.374 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.374 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.374 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.374 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.923975+0000", 2020-01-16T00:12:52.374 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0099596319999999995 2020-01-16T00:12:52.375 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.375 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.375 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.375 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.375 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.375 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:52.377 INFO:tasks.rados.rados.0.smithi123.stdout:846: expect (ObjNum 125 snap 0 seq_num 125) 2020-01-16T00:12:52.377 INFO:tasks.rados.rados.0.smithi123.stdout:848: finishing write tid 1 to smithi12313934-18 2020-01-16T00:12:52.377 INFO:tasks.rados.rados.0.smithi123.stdout:848: finishing write tid 2 to smithi12313934-18 2020-01-16T00:12:52.378 INFO:tasks.rados.rados.0.smithi123.stdout:848: finishing write tid 4 to smithi12313934-18 2020-01-16T00:12:52.378 INFO:tasks.rados.rados.0.smithi123.stdout:848: finishing write tid 5 to smithi12313934-18 2020-01-16T00:12:52.378 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 18 v 153 (ObjNum 168 snap 0 seq_num 168) dirty exists 2020-01-16T00:12:52.378 INFO:tasks.rados.rados.0.smithi123.stdout:848: left oid 18 (ObjNum 168 snap 0 seq_num 168) 2020-01-16T00:12:52.378 INFO:tasks.rados.rados.0.smithi123.stdout:849: expect (ObjNum 161 snap 0 seq_num 161) 2020-01-16T00:12:52.381 INFO:teuthology.orchestra.run.smithi123.stderr:Error EPERM: min_compat_client jewel < luminous, which is required for pg-upmap. Try 'ceph osd set-require-min-compat-client luminous' before using the new interface 2020-01-16T00:12:52.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-01-16T00:12:52.382 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:12:52.379+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:12:52.383 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-01-16T00:12:52.383 INFO:tasks.thrashosds.thrasher:Failed to rm-pg-upmap, ignoring 2020-01-16T00:12:52.444 INFO:tasks.rados.rados.0.smithi123.stdout:850: expect (ObjNum 159 snap 0 seq_num 159) 2020-01-16T00:12:52.478 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:12:52.479+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:12:52.501 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:52.502 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:12:52.502 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:12:52.502 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:52.513 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-01-16T00:12:52.579 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:52.579+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:12:52.605 INFO:tasks.rados.rados.0.smithi123.stdout:851: expect (ObjNum 124 snap 0 seq_num 124) 2020-01-16T00:12:52.625 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:52.625 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:12:52.625 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:12:52.625 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:12:52.626 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:52.634 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-01-16T00:12:52.679 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:52.679+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:12:52.693 INFO:tasks.rados.rados.0.smithi123.stdout:842: done (10 left) 2020-01-16T00:12:52.693 INFO:tasks.rados.rados.0.smithi123.stdout:843: done (9 left) 2020-01-16T00:12:52.693 INFO:tasks.rados.rados.0.smithi123.stdout:844: done (8 left) 2020-01-16T00:12:52.693 INFO:tasks.rados.rados.0.smithi123.stdout:845: done (7 left) 2020-01-16T00:12:52.694 INFO:tasks.rados.rados.0.smithi123.stdout:846: done (6 left) 2020-01-16T00:12:52.694 INFO:tasks.rados.rados.0.smithi123.stdout:847: done (5 left) 2020-01-16T00:12:52.694 INFO:tasks.rados.rados.0.smithi123.stdout:848: done (4 left) 2020-01-16T00:12:52.694 INFO:tasks.rados.rados.0.smithi123.stdout:849: done (3 left) 2020-01-16T00:12:52.694 INFO:tasks.rados.rados.0.smithi123.stdout:850: done (2 left) 2020-01-16T00:12:52.694 INFO:tasks.rados.rados.0.smithi123.stdout:851: done (1 left) 2020-01-16T00:12:52.694 INFO:tasks.rados.rados.0.smithi123.stdout:852: done (0 left) 2020-01-16T00:12:52.695 INFO:tasks.rados.rados.0.smithi123.stdout:853: write oid 2 current snap is 0 2020-01-16T00:12:52.695 INFO:tasks.rados.rados.0.smithi123.stdout:853: seq_num 170 ranges {457190=552096,1614880=581041,2785791=71399} 2020-01-16T00:12:52.702 INFO:tasks.rados.rados.0.smithi123.stdout:853: writing smithi12313934-2 from 457190 to 1009286 tid 1 2020-01-16T00:12:52.711 INFO:tasks.rados.rados.0.smithi123.stdout:853: writing smithi12313934-2 from 1614880 to 2195921 tid 2 2020-01-16T00:12:52.715 INFO:tasks.rados.rados.0.smithi123.stdout:853: writing smithi12313934-2 from 2785791 to 2857190 tid 3 2020-01-16T00:12:52.715 INFO:tasks.rados.rados.0.smithi123.stdout:854: write oid 24 current snap is 0 2020-01-16T00:12:52.715 INFO:tasks.rados.rados.0.smithi123.stdout:854: seq_num 171 ranges {473997=627345,1673362=441052,2529567=344430} 2020-01-16T00:12:52.721 INFO:tasks.rados.rados.0.smithi123.stdout:854: writing smithi12313934-24 from 473997 to 1101342 tid 1 2020-01-16T00:12:52.726 INFO:tasks.rados.rados.0.smithi123.stdout:854: writing smithi12313934-24 from 1673362 to 2114414 tid 2 2020-01-16T00:12:52.731 INFO:tasks.rados.rados.0.smithi123.stdout:854: writing smithi12313934-24 from 2529567 to 2873997 tid 3 2020-01-16T00:12:52.731 INFO:tasks.rados.rados.0.smithi123.stdout:855: read oid 18 snap -1 2020-01-16T00:12:52.731 INFO:tasks.rados.rados.0.smithi123.stdout:855: expect (ObjNum 168 snap 0 seq_num 168) 2020-01-16T00:12:52.732 INFO:tasks.rados.rados.0.smithi123.stdout:856: delete oid 3 current snap is 0 2020-01-16T00:12:52.732 INFO:tasks.rados.rados.0.smithi123.stdout:853: finishing write tid 1 to smithi12313934-2 2020-01-16T00:12:52.732 INFO:tasks.rados.rados.0.smithi123.stdout:853: finishing write tid 2 to smithi12313934-2 2020-01-16T00:12:52.732 INFO:tasks.rados.rados.0.smithi123.stdout:853: finishing write tid 3 to smithi12313934-2 2020-01-16T00:12:52.732 INFO:tasks.rados.rados.0.smithi123.stdout:854: finishing write tid 1 to smithi12313934-24 2020-01-16T00:12:52.732 INFO:tasks.rados.rados.0.smithi123.stdout:853: finishing write tid 5 to smithi12313934-2 2020-01-16T00:12:52.732 INFO:tasks.rados.rados.0.smithi123.stdout:853: finishing write tid 6 to smithi12313934-2 2020-01-16T00:12:52.733 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 2 v 111 (ObjNum 170 snap 0 seq_num 170) dirty exists 2020-01-16T00:12:52.733 INFO:tasks.rados.rados.0.smithi123.stdout:853: left oid 2 (ObjNum 170 snap 0 seq_num 170) 2020-01-16T00:12:52.733 INFO:tasks.rados.rados.0.smithi123.stdout:854: finishing write tid 2 to smithi12313934-24 2020-01-16T00:12:52.735 INFO:tasks.rados.rados.0.smithi123.stdout:853: done (3 left) 2020-01-16T00:12:52.735 INFO:tasks.rados.rados.0.smithi123.stdout:856: done (2 left) 2020-01-16T00:12:52.736 INFO:tasks.rados.rados.0.smithi123.stdout:857: write (excl) oid 10 current snap is 0 2020-01-16T00:12:52.736 INFO:tasks.rados.rados.0.smithi123.stdout:857: seq_num 173 ranges {507611=777843,2073973=444721} 2020-01-16T00:12:52.741 INFO:tasks.rados.rados.0.smithi123.stdout:857: writing smithi12313934-10 from 507611 to 1285454 tid 1 2020-01-16T00:12:52.749 INFO:tasks.rados.rados.0.smithi123.stdout:857: writing smithi12313934-10 from 2073973 to 2518694 tid 2 2020-01-16T00:12:52.749 INFO:tasks.rados.rados.0.smithi123.stdout:858: read oid 47 snap -1 2020-01-16T00:12:52.749 INFO:tasks.rados.rados.0.smithi123.stdout:858: expect (ObjNum 133 snap 0 seq_num 133) 2020-01-16T00:12:52.749 INFO:tasks.rados.rados.0.smithi123.stdout:859: read oid 13 snap -1 2020-01-16T00:12:52.749 INFO:tasks.rados.rados.0.smithi123.stdout:859: expect (ObjNum 124 snap 0 seq_num 124) 2020-01-16T00:12:52.750 INFO:tasks.rados.rados.0.smithi123.stdout:860: read oid 46 snap -1 2020-01-16T00:12:52.750 INFO:tasks.rados.rados.0.smithi123.stdout:860: expect (ObjNum 97 snap 0 seq_num 97) 2020-01-16T00:12:52.750 INFO:tasks.rados.rados.0.smithi123.stdout:861: read oid 23 snap -1 2020-01-16T00:12:52.750 INFO:tasks.rados.rados.0.smithi123.stdout:861: expect (ObjNum 22 snap 0 seq_num 22) 2020-01-16T00:12:52.750 INFO:tasks.rados.rados.0.smithi123.stdout:862: write oid 25 current snap is 0 2020-01-16T00:12:52.750 INFO:tasks.rados.rados.0.smithi123.stdout:862: seq_num 174 ranges {524418=453092,1732455=588379,2779130=145288} 2020-01-16T00:12:52.752 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:52.752 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:12:52.752 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:12:52.752 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:12:52.752 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.752 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:895 3.3 3:cd231094:::smithi12313934-49:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:52.753 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.122390+0000", 2020-01-16T00:12:52.753 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 15.628725165000001, 2020-01-16T00:12:52.753 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014295501, 2020-01-16T00:12:52.753 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.753 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:52.753 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.753 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.753 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.754 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 895 2020-01-16T00:12:52.754 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.754 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.754 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.754 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.754 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122390+0000", 2020-01-16T00:12:52.754 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.754 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.755 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.755 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.755 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122390+0000", 2020-01-16T00:12:52.755 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999957 2020-01-16T00:12:52.755 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.755 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.755 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.756 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122386+0000", 2020-01-16T00:12:52.756 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0656999999999999e-05 2020-01-16T00:12:52.756 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.756 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.756 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.756 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122407+0000", 2020-01-16T00:12:52.756 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9102999999999999e-05 2020-01-16T00:12:52.756 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.757 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.757 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.757 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122436+0000", 2020-01-16T00:12:52.757 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.774e-05 2020-01-16T00:12:52.757 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.757 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.757 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.757 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122483+0000", 2020-01-16T00:12:52.758 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5185000000000004e-05 2020-01-16T00:12:52.758 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.758 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.758 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.758 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122579+0000", 2020-01-16T00:12:52.758 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000304085 2020-01-16T00:12:52.758 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.759 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.759 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:52.759 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122883+0000", 2020-01-16T00:12:52.759 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0092077970000000002 2020-01-16T00:12:52.759 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.759 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.759 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.760 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132091+0000", 2020-01-16T00:12:52.760 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000170025 2020-01-16T00:12:52.760 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.760 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.760 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:52.760 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132261+0000", 2020-01-16T00:12:52.760 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0040130979999999997 2020-01-16T00:12:52.760 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.761 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.761 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.761 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136274+0000", 2020-01-16T00:12:52.761 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014227900000000001 2020-01-16T00:12:52.761 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.762 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.762 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.762 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136416+0000", 2020-01-16T00:12:52.762 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00026995200000000001 2020-01-16T00:12:52.762 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.762 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.762 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.762 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136686+0000", 2020-01-16T00:12:52.763 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014295501 2020-01-16T00:12:52.763 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.763 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.763 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.763 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.763 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.763 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:896 3.3 3:cd231094:::smithi12313934-49:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:52.763 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.122539+0000", 2020-01-16T00:12:52.764 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 15.628576165, 2020-01-16T00:12:52.764 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014440173000000001, 2020-01-16T00:12:52.764 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.764 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:52.764 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.764 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.764 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.764 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 896 2020-01-16T00:12:52.765 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.765 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.765 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.765 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.765 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122539+0000", 2020-01-16T00:12:52.765 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.765 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.765 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.766 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.766 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122539+0000", 2020-01-16T00:12:52.767 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:52.767 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.767 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.767 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.767 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122537+0000", 2020-01-16T00:12:52.767 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2432999999999999e-05 2020-01-16T00:12:52.767 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.767 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.768 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.768 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122550+0000", 2020-01-16T00:12:52.768 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9369999999999999e-05 2020-01-16T00:12:52.768 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.768 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.768 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.768 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122569+0000", 2020-01-16T00:12:52.769 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1747e-05 2020-01-16T00:12:52.769 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.769 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.769 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.769 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122581+0000", 2020-01-16T00:12:52.769 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00041555800000000001 2020-01-16T00:12:52.769 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.770 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.770 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.770 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122996+0000", 2020-01-16T00:12:52.770 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016840299999999999 2020-01-16T00:12:52.770 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.770 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.770 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:52.770 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.123165+0000", 2020-01-16T00:12:52.771 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0091641359999999998 2020-01-16T00:12:52.771 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.771 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.771 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.771 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132329+0000", 2020-01-16T00:12:52.771 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013433400000000001 2020-01-16T00:12:52.772 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.772 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.772 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:52.772 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132463+0000", 2020-01-16T00:12:52.772 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0042524110000000002 2020-01-16T00:12:52.772 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.772 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.773 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.773 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136716+0000", 2020-01-16T00:12:52.773 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4591999999999993e-05 2020-01-16T00:12:52.773 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.773 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.773 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.773 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136790+0000", 2020-01-16T00:12:52.774 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018927500000000001 2020-01-16T00:12:52.774 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.774 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.774 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.774 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136980+0000", 2020-01-16T00:12:52.774 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014440173000000001 2020-01-16T00:12:52.774 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.774 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.775 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.775 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.775 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.775 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:897 3.3 3:cd231094:::smithi12313934-49:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:52.775 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.122613+0000", 2020-01-16T00:12:52.775 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 15.628503021, 2020-01-16T00:12:52.776 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014606846, 2020-01-16T00:12:52.776 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.776 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:52.776 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.776 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.776 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.776 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 897 2020-01-16T00:12:52.776 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.777 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.777 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.777 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.777 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122613+0000", 2020-01-16T00:12:52.778 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.778 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.778 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.778 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.778 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122613+0000", 2020-01-16T00:12:52.778 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:52.778 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.778 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.779 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.779 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122611+0000", 2020-01-16T00:12:52.779 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3670000000000003e-06 2020-01-16T00:12:52.779 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.779 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.780 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.780 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122620+0000", 2020-01-16T00:12:52.780 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5193e-05 2020-01-16T00:12:52.780 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.780 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.780 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.780 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122635+0000", 2020-01-16T00:12:52.780 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0696999999999999e-05 2020-01-16T00:12:52.781 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.781 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.781 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.781 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122646+0000", 2020-01-16T00:12:52.781 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00059961400000000005 2020-01-16T00:12:52.781 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.781 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.782 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.782 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.123246+0000", 2020-01-16T00:12:52.782 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011189200000000001 2020-01-16T00:12:52.782 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.782 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.783 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:52.783 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.123358+0000", 2020-01-16T00:12:52.783 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0091363550000000005 2020-01-16T00:12:52.783 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.783 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.783 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.783 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132494+0000", 2020-01-16T00:12:52.783 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2956999999999998e-05 2020-01-16T00:12:52.784 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.784 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.784 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:52.784 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132587+0000", 2020-01-16T00:12:52.784 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0044061769999999998 2020-01-16T00:12:52.784 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.784 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.784 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.785 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136993+0000", 2020-01-16T00:12:52.785 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7822999999999998e-05 2020-01-16T00:12:52.785 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.785 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.785 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.785 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.137051+0000", 2020-01-16T00:12:52.785 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000168498 2020-01-16T00:12:52.786 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.786 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.786 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.786 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.137219+0000", 2020-01-16T00:12:52.786 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014606846 2020-01-16T00:12:52.786 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.786 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.786 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.787 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.787 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.787 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:943 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:52.787 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:12:52.787 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.836637047, 2020-01-16T00:12:52.787 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018847004000000001, 2020-01-16T00:12:52.787 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.787 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:52.788 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.788 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.788 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.788 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 943 2020-01-16T00:12:52.788 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.788 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.788 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.789 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.789 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:12:52.789 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.789 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.789 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.790 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.790 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:12:52.790 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:12:52.790 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.790 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.790 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.790 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914475+0000", 2020-01-16T00:12:52.791 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.508e-05 2020-01-16T00:12:52.791 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.791 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.791 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.791 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914490+0000", 2020-01-16T00:12:52.791 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.857e-05 2020-01-16T00:12:52.791 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.791 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.792 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.792 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914509+0000", 2020-01-16T00:12:52.792 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.46e-06 2020-01-16T00:12:52.792 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.792 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.793 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.793 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914515+0000", 2020-01-16T00:12:52.793 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2612999999999999e-05 2020-01-16T00:12:52.793 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.793 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.793 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.793 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914568+0000", 2020-01-16T00:12:52.793 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028312299999999998 2020-01-16T00:12:52.794 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.794 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.794 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:52.794 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914851+0000", 2020-01-16T00:12:52.794 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012282533 2020-01-16T00:12:52.794 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.794 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.795 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.795 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927133+0000", 2020-01-16T00:12:52.795 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4839999999999995e-05 2020-01-16T00:12:52.795 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.795 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.795 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:52.795 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927208+0000", 2020-01-16T00:12:52.795 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0047425610000000002 2020-01-16T00:12:52.796 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.796 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.796 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.796 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.931951+0000", 2020-01-16T00:12:52.796 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3628999999999994e-05 2020-01-16T00:12:52.796 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.796 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.796 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.796 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.932045+0000", 2020-01-16T00:12:52.797 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012810180000000001 2020-01-16T00:12:52.797 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.797 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.797 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.797 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933326+0000", 2020-01-16T00:12:52.797 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018847004000000001 2020-01-16T00:12:52.798 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.798 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.798 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.798 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.798 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.798 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:944 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:52.798 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:12:52.798 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.836587491, 2020-01-16T00:12:52.799 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019086901999999999, 2020-01-16T00:12:52.799 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.799 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:52.799 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.799 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.799 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.799 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 944 2020-01-16T00:12:52.800 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.800 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.800 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.800 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.800 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:12:52.800 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.800 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.801 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.801 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.801 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:12:52.801 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:52.801 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.801 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.801 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.801 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914527+0000", 2020-01-16T00:12:52.802 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7509999999999999e-06 2020-01-16T00:12:52.802 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.802 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.802 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.802 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914532+0000", 2020-01-16T00:12:52.803 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5149999999999997e-06 2020-01-16T00:12:52.803 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.803 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.803 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.803 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914538+0000", 2020-01-16T00:12:52.803 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1890000000000007e-06 2020-01-16T00:12:52.803 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.803 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.804 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.804 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914546+0000", 2020-01-16T00:12:52.804 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000354951 2020-01-16T00:12:52.804 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.804 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.804 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.804 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914901+0000", 2020-01-16T00:12:52.805 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1594000000000002e-05 2020-01-16T00:12:52.805 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.805 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.805 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:52.805 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914942+0000", 2020-01-16T00:12:52.805 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012300070999999999 2020-01-16T00:12:52.805 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.805 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.806 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.806 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927242+0000", 2020-01-16T00:12:52.806 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4076999999999997e-05 2020-01-16T00:12:52.806 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.806 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.806 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:52.806 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927296+0000", 2020-01-16T00:12:52.807 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0060609180000000002 2020-01-16T00:12:52.807 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.807 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.807 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.807 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933357+0000", 2020-01-16T00:12:52.807 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4488999999999999e-05 2020-01-16T00:12:52.807 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.808 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.808 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.808 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933422+0000", 2020-01-16T00:12:52.808 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019318300000000001 2020-01-16T00:12:52.808 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.808 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.808 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.809 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933615+0000", 2020-01-16T00:12:52.809 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019086901999999999 2020-01-16T00:12:52.809 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.809 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.809 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.809 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.809 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.810 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:945 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:52.810 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:12:52.810 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.836560284999999, 2020-01-16T00:12:52.810 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019342986, 2020-01-16T00:12:52.810 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.810 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:52.810 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.811 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.811 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.811 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 945 2020-01-16T00:12:52.811 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.811 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.811 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.811 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.811 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:12:52.812 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.812 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.812 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.812 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.812 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:12:52.812 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:52.812 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.812 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.813 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.813 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:12:52.813 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4580000000000002e-06 2020-01-16T00:12:52.813 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.813 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.814 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.814 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914558+0000", 2020-01-16T00:12:52.814 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7829999999999998e-06 2020-01-16T00:12:52.814 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.814 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.814 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.814 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914563+0000", 2020-01-16T00:12:52.814 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0001e-05 2020-01-16T00:12:52.815 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.815 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.815 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.815 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914573+0000", 2020-01-16T00:12:52.815 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00040365600000000001 2020-01-16T00:12:52.815 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.815 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.815 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.816 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914976+0000", 2020-01-16T00:12:52.816 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1390000000000001e-05 2020-01-16T00:12:52.816 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.816 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.816 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:52.816 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.915028+0000", 2020-01-16T00:12:52.816 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012288843000000001 2020-01-16T00:12:52.816 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.817 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.817 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.817 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927317+0000", 2020-01-16T00:12:52.817 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7871000000000002e-05 2020-01-16T00:12:52.817 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.817 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.817 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:52.817 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927375+0000", 2020-01-16T00:12:52.818 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0062580589999999998 2020-01-16T00:12:52.818 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.818 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.818 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.818 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933633+0000", 2020-01-16T00:12:52.818 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7086999999999997e-05 2020-01-16T00:12:52.818 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.819 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.819 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.819 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933690+0000", 2020-01-16T00:12:52.819 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020855799999999999 2020-01-16T00:12:52.819 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.819 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.819 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.819 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933898+0000", 2020-01-16T00:12:52.820 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019342986 2020-01-16T00:12:52.820 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.820 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.820 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.820 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.820 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.820 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:963 3.9 3:9557b633:::smithi12313934-43:head [read 0~2705894,checksum] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:52.821 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.046589+0000", 2020-01-16T00:12:52.821 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.704526437, 2020-01-16T00:12:52.821 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014014016000000001, 2020-01-16T00:12:52.821 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.821 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:52.821 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.821 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.822 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.822 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 963 2020-01-16T00:12:52.822 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.822 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.822 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.822 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.822 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046589+0000", 2020-01-16T00:12:52.822 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.823 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.823 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.823 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.823 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046589+0000", 2020-01-16T00:12:52.823 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:52.824 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.824 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.824 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.824 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046588+0000", 2020-01-16T00:12:52.824 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7469999999999994e-06 2020-01-16T00:12:52.824 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.824 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.824 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.825 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046596+0000", 2020-01-16T00:12:52.825 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3445999999999998e-05 2020-01-16T00:12:52.825 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.825 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.825 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.825 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046630+0000", 2020-01-16T00:12:52.825 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7779999999999999e-05 2020-01-16T00:12:52.825 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.826 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.826 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.826 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046648+0000", 2020-01-16T00:12:52.826 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00075081300000000002 2020-01-16T00:12:52.826 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.826 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.826 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.826 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.047398+0000", 2020-01-16T00:12:52.827 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000116698 2020-01-16T00:12:52.827 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.827 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.827 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:52.827 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.047515+0000", 2020-01-16T00:12:52.827 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0056630689999999997 2020-01-16T00:12:52.827 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.828 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.828 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.828 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.053178+0000", 2020-01-16T00:12:52.828 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012084299999999999 2020-01-16T00:12:52.828 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.828 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.828 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:52.828 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.053299+0000", 2020-01-16T00:12:52.829 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0056921510000000003 2020-01-16T00:12:52.829 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.829 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.829 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.829 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.058991+0000", 2020-01-16T00:12:52.829 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000118606 2020-01-16T00:12:52.829 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.829 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.830 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.830 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.059110+0000", 2020-01-16T00:12:52.830 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001493306 2020-01-16T00:12:52.830 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.830 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.830 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.831 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.060603+0000", 2020-01-16T00:12:52.831 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014014016000000001 2020-01-16T00:12:52.831 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.831 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.831 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.831 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.831 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.831 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:964 3.9 3:9557b633:::smithi12313934-43:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:52.832 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.046668+0000", 2020-01-16T00:12:52.832 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.704447833, 2020-01-16T00:12:52.832 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014425670999999999, 2020-01-16T00:12:52.832 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.832 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:52.832 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.832 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.833 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.833 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 964 2020-01-16T00:12:52.833 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.833 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.833 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.833 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.834 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046668+0000", 2020-01-16T00:12:52.834 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.834 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.834 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.834 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.834 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046668+0000", 2020-01-16T00:12:52.835 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:52.835 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.835 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.835 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.835 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046666+0000", 2020-01-16T00:12:52.836 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0879999999999999e-05 2020-01-16T00:12:52.836 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.836 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.836 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.836 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046677+0000", 2020-01-16T00:12:52.836 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5767e-05 2020-01-16T00:12:52.836 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.837 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.837 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.837 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046693+0000", 2020-01-16T00:12:52.837 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1399999999999999e-05 2020-01-16T00:12:52.837 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.837 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.837 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.837 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046704+0000", 2020-01-16T00:12:52.838 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00089331800000000004 2020-01-16T00:12:52.838 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.838 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.838 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.838 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.047597+0000", 2020-01-16T00:12:52.838 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8628000000000005e-05 2020-01-16T00:12:52.838 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.838 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.839 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:52.839 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.047696+0000", 2020-01-16T00:12:52.839 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0056536290000000003 2020-01-16T00:12:52.839 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.839 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.839 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.839 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.053349+0000", 2020-01-16T00:12:52.840 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.6946000000000003e-05 2020-01-16T00:12:52.840 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.840 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.840 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:52.840 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.053446+0000", 2020-01-16T00:12:52.840 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0072058909999999999 2020-01-16T00:12:52.840 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.841 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.841 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.841 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.060652+0000", 2020-01-16T00:12:52.841 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000135597 2020-01-16T00:12:52.841 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.841 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.841 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.841 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.060788+0000", 2020-01-16T00:12:52.842 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030548499999999998 2020-01-16T00:12:52.842 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.842 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.842 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.842 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.061093+0000", 2020-01-16T00:12:52.842 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014425670999999999 2020-01-16T00:12:52.842 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.842 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.843 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.843 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.843 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.843 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:981 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:52.843 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:12:52.843 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.435783392999999, 2020-01-16T00:12:52.843 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021222494000000001, 2020-01-16T00:12:52.844 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.844 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:52.844 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.844 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.844 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.844 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 981 2020-01-16T00:12:52.845 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.845 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.845 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.845 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.845 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:12:52.845 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.845 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.845 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.846 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.846 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:12:52.846 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999948 2020-01-16T00:12:52.846 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.846 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.846 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.846 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315327+0000", 2020-01-16T00:12:52.847 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.508e-05 2020-01-16T00:12:52.847 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.847 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.847 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.847 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315352+0000", 2020-01-16T00:12:52.847 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1183999999999998e-05 2020-01-16T00:12:52.847 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.847 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.847 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.848 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315383+0000", 2020-01-16T00:12:52.848 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7180000000000002e-05 2020-01-16T00:12:52.848 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.848 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.848 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.848 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315401+0000", 2020-01-16T00:12:52.848 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017290599999999999 2020-01-16T00:12:52.848 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.849 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.849 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.849 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315574+0000", 2020-01-16T00:12:52.849 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00047908100000000002 2020-01-16T00:12:52.849 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.849 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.849 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:52.850 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316053+0000", 2020-01-16T00:12:52.850 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011808947 2020-01-16T00:12:52.850 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.850 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.850 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.850 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.327862+0000", 2020-01-16T00:12:52.850 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010067499999999999 2020-01-16T00:12:52.851 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.851 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.851 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:52.851 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.327962+0000", 2020-01-16T00:12:52.851 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0074745139999999998 2020-01-16T00:12:52.851 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.851 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.851 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.852 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.335437+0000", 2020-01-16T00:12:52.852 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000103403 2020-01-16T00:12:52.852 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.852 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.852 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.852 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.335540+0000", 2020-01-16T00:12:52.852 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001014544 2020-01-16T00:12:52.852 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.853 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.853 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.853 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336555+0000", 2020-01-16T00:12:52.853 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021222494000000001 2020-01-16T00:12:52.853 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.853 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.853 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.853 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.854 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.854 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:982 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:52.854 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:12:52.854 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.435680869, 2020-01-16T00:12:52.854 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.022087721000000001, 2020-01-16T00:12:52.855 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.855 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:52.855 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.855 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.855 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.855 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 982 2020-01-16T00:12:52.855 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.855 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.856 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.856 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.856 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:12:52.856 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.856 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.856 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.856 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.856 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:12:52.857 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:52.857 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.857 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.857 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.857 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315433+0000", 2020-01-16T00:12:52.857 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1347e-05 2020-01-16T00:12:52.857 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.857 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.858 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.858 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315445+0000", 2020-01-16T00:12:52.858 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4525999999999999e-05 2020-01-16T00:12:52.858 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.858 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.858 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.858 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315459+0000", 2020-01-16T00:12:52.858 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.024e-05 2020-01-16T00:12:52.859 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.859 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.859 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.859 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315470+0000", 2020-01-16T00:12:52.859 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000736468 2020-01-16T00:12:52.859 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.859 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.859 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.860 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316206+0000", 2020-01-16T00:12:52.860 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000132967 2020-01-16T00:12:52.860 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.860 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.860 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:52.860 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316339+0000", 2020-01-16T00:12:52.861 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011701717 2020-01-16T00:12:52.861 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.861 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.862 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.862 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328041+0000", 2020-01-16T00:12:52.862 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000120339 2020-01-16T00:12:52.862 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.862 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.862 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:52.862 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328161+0000", 2020-01-16T00:12:52.863 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0084341260000000001 2020-01-16T00:12:52.863 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.863 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.863 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.863 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336595+0000", 2020-01-16T00:12:52.863 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5247999999999993e-05 2020-01-16T00:12:52.864 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.864 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.864 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.864 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336690+0000", 2020-01-16T00:12:52.864 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00083204000000000004 2020-01-16T00:12:52.864 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.864 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.864 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.865 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337522+0000", 2020-01-16T00:12:52.865 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.022087721000000001 2020-01-16T00:12:52.865 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.865 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.865 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.865 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.865 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.865 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:983 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:52.866 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:12:52.866 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.435561159000001, 2020-01-16T00:12:52.866 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023363387999999999, 2020-01-16T00:12:52.866 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.866 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:52.866 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.866 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.866 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.867 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 983 2020-01-16T00:12:52.867 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.867 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.867 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.867 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.867 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:12:52.867 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.867 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.868 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.868 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.868 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:12:52.868 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:52.868 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.868 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.868 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.869 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315553+0000", 2020-01-16T00:12:52.869 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1760000000000004e-06 2020-01-16T00:12:52.869 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.869 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.869 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.869 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315560+0000", 2020-01-16T00:12:52.869 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3587000000000001e-05 2020-01-16T00:12:52.869 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.870 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.870 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.870 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315574+0000", 2020-01-16T00:12:52.870 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7800000000000003e-06 2020-01-16T00:12:52.870 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.870 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.871 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.871 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315581+0000", 2020-01-16T00:12:52.871 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000823024 2020-01-16T00:12:52.871 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.871 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.871 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.871 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316404+0000", 2020-01-16T00:12:52.871 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2413e-05 2020-01-16T00:12:52.872 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.872 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.872 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:52.872 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316466+0000", 2020-01-16T00:12:52.872 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011738561999999999 2020-01-16T00:12:52.872 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.872 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.872 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.873 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328205+0000", 2020-01-16T00:12:52.873 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6783999999999997e-05 2020-01-16T00:12:52.873 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.873 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.873 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:52.873 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328262+0000", 2020-01-16T00:12:52.873 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0093022560000000001 2020-01-16T00:12:52.873 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.874 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.874 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.874 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337564+0000", 2020-01-16T00:12:52.874 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000183151 2020-01-16T00:12:52.874 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.875 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.875 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.875 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337747+0000", 2020-01-16T00:12:52.875 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011707449999999999 2020-01-16T00:12:52.875 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.875 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.875 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.876 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.338918+0000", 2020-01-16T00:12:52.876 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023363387999999999 2020-01-16T00:12:52.876 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.876 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.876 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.876 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.876 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.876 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:986 3.9 3:93dfe6f1:::smithi12313934-40:head [write 2941380~419987] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:52.877 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:12:52.877 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.270261382999999, 2020-01-16T00:12:52.877 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016978400000000001, 2020-01-16T00:12:52.877 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.877 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:52.877 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.877 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.877 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.878 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 986 2020-01-16T00:12:52.878 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.878 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.878 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.878 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.878 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:12:52.878 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.878 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.879 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.879 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.879 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:12:52.879 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:52.879 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.879 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.879 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.880 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480852+0000", 2020-01-16T00:12:52.880 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042503899999999999 2020-01-16T00:12:52.880 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.880 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.880 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.880 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481277+0000", 2020-01-16T00:12:52.880 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3097e-05 2020-01-16T00:12:52.880 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.881 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.881 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.881 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481290+0000", 2020-01-16T00:12:52.881 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8460000000000002e-06 2020-01-16T00:12:52.881 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.881 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.881 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.881 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481297+0000", 2020-01-16T00:12:52.882 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8443999999999999e-05 2020-01-16T00:12:52.882 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.882 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.882 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.882 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481346+0000", 2020-01-16T00:12:52.882 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022115399999999999 2020-01-16T00:12:52.882 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.882 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.883 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:52.883 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481567+0000", 2020-01-16T00:12:52.883 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012531502 2020-01-16T00:12:52.883 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.883 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.883 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.883 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494098+0000", 2020-01-16T00:12:52.883 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.4462999999999993e-05 2020-01-16T00:12:52.884 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.884 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.884 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.884 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494193+0000", 2020-01-16T00:12:52.884 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017600999999999999 2020-01-16T00:12:52.884 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.884 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.884 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:12:52.885 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494369+0000", 2020-01-16T00:12:52.885 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0031739619999999998 2020-01-16T00:12:52.885 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.885 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.885 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:52.885 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497543+0000", 2020-01-16T00:12:52.886 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000175107 2020-01-16T00:12:52.886 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.886 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.886 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:52.886 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497718+0000", 2020-01-16T00:12:52.886 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5207000000000003e-05 2020-01-16T00:12:52.887 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.887 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.888 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:52.888 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497763+0000", 2020-01-16T00:12:52.888 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7042999999999998e-05 2020-01-16T00:12:52.888 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.888 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.888 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:52.888 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497800+0000", 2020-01-16T00:12:52.889 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2628e-05 2020-01-16T00:12:52.889 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.889 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.889 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.889 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497833+0000", 2020-01-16T00:12:52.889 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016978400000000001 2020-01-16T00:12:52.889 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.890 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.890 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.890 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.890 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.890 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:987 3.9 3:93dfe6f1:::smithi12313934-40:head [setxattr _header (56) in=7b,truncate 3361367] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:52.890 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.481307+0000", 2020-01-16T00:12:52.890 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.269809016, 2020-01-16T00:12:52.891 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016794989, 2020-01-16T00:12:52.891 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.891 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:52.891 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.891 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.891 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.891 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 987 2020-01-16T00:12:52.892 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.892 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.892 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.892 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.892 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481307+0000", 2020-01-16T00:12:52.892 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.892 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.892 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.893 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.893 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481307+0000", 2020-01-16T00:12:52.893 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:52.893 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.893 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.893 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.895 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481306+0000", 2020-01-16T00:12:52.895 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.113e-06 2020-01-16T00:12:52.896 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.896 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.897 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.897 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481311+0000", 2020-01-16T00:12:52.897 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1520000000000002e-06 2020-01-16T00:12:52.897 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.897 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.897 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.898 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481315+0000", 2020-01-16T00:12:52.898 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.7640000000000001e-06 2020-01-16T00:12:52.898 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.898 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.898 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.898 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481318+0000", 2020-01-16T00:12:52.899 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00029012500000000002 2020-01-16T00:12:52.899 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.899 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.899 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.899 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481608+0000", 2020-01-16T00:12:52.899 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6239000000000002e-05 2020-01-16T00:12:52.900 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.900 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.900 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:52.900 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481654+0000", 2020-01-16T00:12:52.900 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014559534000000001 2020-01-16T00:12:52.900 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.901 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.901 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.901 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.496214+0000", 2020-01-16T00:12:52.901 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000225932 2020-01-16T00:12:52.901 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.901 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.901 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.902 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.496439+0000", 2020-01-16T00:12:52.902 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00031099600000000002 2020-01-16T00:12:52.902 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.902 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.902 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:12:52.902 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.496750+0000", 2020-01-16T00:12:52.903 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00098608599999999995 2020-01-16T00:12:52.903 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.903 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.903 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:52.903 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497737+0000", 2020-01-16T00:12:52.903 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022081499999999999 2020-01-16T00:12:52.903 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.904 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.904 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:52.904 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497957+0000", 2020-01-16T00:12:52.904 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7071000000000003e-05 2020-01-16T00:12:52.904 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.904 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.905 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:52.905 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498014+0000", 2020-01-16T00:12:52.905 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1111e-05 2020-01-16T00:12:52.906 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.906 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.906 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:52.906 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498056+0000", 2020-01-16T00:12:52.906 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5974999999999999e-05 2020-01-16T00:12:52.906 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.906 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.906 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.907 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498102+0000", 2020-01-16T00:12:52.907 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016794989 2020-01-16T00:12:52.907 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.907 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.907 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.907 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.907 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.907 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:988 3.9 3:93dfe6f1:::smithi12313934-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:12:52.908 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:12:52.908 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.269775919000001, 2020-01-16T00:12:52.908 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017221692, 2020-01-16T00:12:52.908 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.908 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:52.908 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.908 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.909 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.909 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 988 2020-01-16T00:12:52.909 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.909 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.909 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.909 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.909 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:12:52.910 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.910 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.910 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.910 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.910 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:12:52.910 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:52.911 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.911 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.911 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.911 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481339+0000", 2020-01-16T00:12:52.911 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3089999999999997e-06 2020-01-16T00:12:52.912 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.912 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.912 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.912 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481343+0000", 2020-01-16T00:12:52.912 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2230000000000004e-06 2020-01-16T00:12:52.912 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.912 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.912 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.913 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481350+0000", 2020-01-16T00:12:52.913 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.319e-06 2020-01-16T00:12:52.913 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.913 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.913 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.913 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481356+0000", 2020-01-16T00:12:52.913 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000321544 2020-01-16T00:12:52.913 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.914 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.914 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.914 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481678+0000", 2020-01-16T00:12:52.914 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6375999999999999e-05 2020-01-16T00:12:52.914 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.914 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.914 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:52.914 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481714+0000", 2020-01-16T00:12:52.915 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015557666 2020-01-16T00:12:52.915 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.915 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.915 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.915 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497272+0000", 2020-01-16T00:12:52.915 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012791300000000001 2020-01-16T00:12:52.915 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.916 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.916 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:52.916 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497399+0000", 2020-01-16T00:12:52.916 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00076944600000000002 2020-01-16T00:12:52.916 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.916 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.916 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.916 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498169+0000", 2020-01-16T00:12:52.917 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018306799999999999 2020-01-16T00:12:52.917 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.917 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.917 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.917 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498352+0000", 2020-01-16T00:12:52.917 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020935999999999999 2020-01-16T00:12:52.917 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.917 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.918 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.918 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498561+0000", 2020-01-16T00:12:52.918 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017221692 2020-01-16T00:12:52.918 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.918 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.918 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.918 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.918 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.919 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1208 3.3 3:c478bfe8:::smithi12313934-10:head [write 749612~474179,stat] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:12:52.919 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.350952+0000", 2020-01-16T00:12:52.919 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 10.400163082000001, 2020-01-16T00:12:52.919 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014967729000000001, 2020-01-16T00:12:52.919 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.919 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:52.919 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.920 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.920 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.920 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1208 2020-01-16T00:12:52.920 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.920 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.920 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.920 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.920 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.350952+0000", 2020-01-16T00:12:52.921 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.921 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.921 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.921 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.921 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.350952+0000", 2020-01-16T00:12:52.921 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:12:52.921 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.921 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.922 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.922 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.350949+0000", 2020-01-16T00:12:52.922 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042894 2020-01-16T00:12:52.922 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.922 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.923 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.923 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.351377+0000", 2020-01-16T00:12:52.923 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3921e-05 2020-01-16T00:12:52.923 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.923 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.923 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.923 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.351401+0000", 2020-01-16T00:12:52.923 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1087000000000001e-05 2020-01-16T00:12:52.924 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.924 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.924 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.924 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.351412+0000", 2020-01-16T00:12:52.924 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8127000000000001e-05 2020-01-16T00:12:52.924 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.924 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.925 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.925 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.351471+0000", 2020-01-16T00:12:52.925 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000230268 2020-01-16T00:12:52.925 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.925 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.925 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:52.925 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.351701+0000", 2020-01-16T00:12:52.925 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0051228369999999999 2020-01-16T00:12:52.926 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.926 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.926 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.926 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.356824+0000", 2020-01-16T00:12:52.926 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015984800000000001 2020-01-16T00:12:52.926 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.926 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.926 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:52.927 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.356984+0000", 2020-01-16T00:12:52.927 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0038163770000000001 2020-01-16T00:12:52.927 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.927 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.927 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.927 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.360800+0000", 2020-01-16T00:12:52.927 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.9878999999999999e-05 2020-01-16T00:12:52.927 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.928 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.928 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.928 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.360880+0000", 2020-01-16T00:12:52.928 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000145041 2020-01-16T00:12:52.928 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.928 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.928 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2020-01-16T00:12:52.928 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.361025+0000", 2020-01-16T00:12:52.929 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0020743039999999999 2020-01-16T00:12:52.929 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.929 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.929 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:52.929 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.363099+0000", 2020-01-16T00:12:52.929 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00072876500000000003 2020-01-16T00:12:52.929 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.929 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.930 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:52.930 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.363828+0000", 2020-01-16T00:12:52.930 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00189153 2020-01-16T00:12:52.930 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.930 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.930 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:52.930 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.365719+0000", 2020-01-16T00:12:52.930 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3948000000000005e-05 2020-01-16T00:12:52.931 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.931 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.931 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:52.931 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.365783+0000", 2020-01-16T00:12:52.931 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000136833 2020-01-16T00:12:52.931 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.931 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.931 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.932 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.365920+0000", 2020-01-16T00:12:52.932 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014967729000000001 2020-01-16T00:12:52.932 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.932 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.932 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.933 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.933 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.933 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1237 3.3 3:c204aa4f:::smithi12313934-45:head [write 400033~699926] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:12:52.933 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.584946+0000", 2020-01-16T00:12:52.933 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 10.166169200000001, 2020-01-16T00:12:52.933 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014287855, 2020-01-16T00:12:52.933 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.934 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:52.934 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.934 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.934 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.934 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1237 2020-01-16T00:12:52.934 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.934 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.934 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.935 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.935 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.584946+0000", 2020-01-16T00:12:52.935 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.935 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.935 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.935 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.935 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.584946+0000", 2020-01-16T00:12:52.935 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999752 2020-01-16T00:12:52.936 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.936 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.936 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.936 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.584921+0000", 2020-01-16T00:12:52.936 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00077243400000000001 2020-01-16T00:12:52.936 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.936 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.936 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.937 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.585694+0000", 2020-01-16T00:12:52.937 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3083000000000001e-05 2020-01-16T00:12:52.937 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.937 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.937 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.937 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.585737+0000", 2020-01-16T00:12:52.937 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1854000000000001e-05 2020-01-16T00:12:52.937 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.938 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.938 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.938 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.585759+0000", 2020-01-16T00:12:52.938 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016127000000000001 2020-01-16T00:12:52.938 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.938 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.938 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.938 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.585920+0000", 2020-01-16T00:12:52.939 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024934700000000002 2020-01-16T00:12:52.939 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.939 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.939 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.939 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.586169+0000", 2020-01-16T00:12:52.939 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00036508999999999999 2020-01-16T00:12:52.939 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.939 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.940 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2020-01-16T00:12:52.940 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.586534+0000", 2020-01-16T00:12:52.940 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012168910999999999 2020-01-16T00:12:52.940 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.940 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.940 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:52.940 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598703+0000", 2020-01-16T00:12:52.940 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5302000000000004e-05 2020-01-16T00:12:52.941 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.941 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.941 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:52.941 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598789+0000", 2020-01-16T00:12:52.941 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00027301099999999998 2020-01-16T00:12:52.941 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.941 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.942 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:52.942 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599062+0000", 2020-01-16T00:12:52.942 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.4237e-05 2020-01-16T00:12:52.942 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.942 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.942 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:52.943 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599146+0000", 2020-01-16T00:12:52.943 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.8319000000000006e-05 2020-01-16T00:12:52.943 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.943 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.943 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.943 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599234+0000", 2020-01-16T00:12:52.943 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014287855 2020-01-16T00:12:52.944 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.944 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.944 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.944 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.944 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.944 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1239 3.3 3:c204aa4f:::smithi12313934-45:head [setxattr _header (56) in=7b,truncate 2000033] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:12:52.944 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:12:52.944 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 10.1607021, 2020-01-16T00:12:52.945 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019161661999999999, 2020-01-16T00:12:52.945 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.945 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:52.945 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.945 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.945 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.945 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1239 2020-01-16T00:12:52.945 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.946 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.946 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.946 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.946 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:12:52.946 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.946 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.946 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.946 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.947 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:12:52.947 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:12:52.947 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.947 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.947 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.947 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590411+0000", 2020-01-16T00:12:52.947 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3879999999999999e-05 2020-01-16T00:12:52.948 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.948 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.948 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.948 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590425+0000", 2020-01-16T00:12:52.948 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6542999999999998e-05 2020-01-16T00:12:52.948 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.948 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.948 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.949 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590471+0000", 2020-01-16T00:12:52.949 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8907999999999999e-05 2020-01-16T00:12:52.949 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.949 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.949 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.949 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590490+0000", 2020-01-16T00:12:52.950 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0045691960000000002 2020-01-16T00:12:52.950 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.950 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.950 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.950 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.595059+0000", 2020-01-16T00:12:52.950 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0035638940000000002 2020-01-16T00:12:52.950 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.950 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.951 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:52.951 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598623+0000", 2020-01-16T00:12:52.951 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0077558310000000004 2020-01-16T00:12:52.951 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.951 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.951 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.951 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606379+0000", 2020-01-16T00:12:52.951 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012171999999999999 2020-01-16T00:12:52.952 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.952 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.952 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.952 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606501+0000", 2020-01-16T00:12:52.952 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013144200000000001 2020-01-16T00:12:52.952 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.952 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.953 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2020-01-16T00:12:52.953 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606632+0000", 2020-01-16T00:12:52.953 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001288565 2020-01-16T00:12:52.953 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.953 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.954 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:52.954 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607921+0000", 2020-01-16T00:12:52.954 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033399299999999999 2020-01-16T00:12:52.954 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.954 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.954 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:52.954 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.608255+0000", 2020-01-16T00:12:52.954 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012280870000000001 2020-01-16T00:12:52.955 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.955 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.955 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:52.955 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609483+0000", 2020-01-16T00:12:52.955 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2286000000000001e-05 2020-01-16T00:12:52.955 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.955 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.955 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:52.956 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609525+0000", 2020-01-16T00:12:52.956 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0121999999999999e-05 2020-01-16T00:12:52.956 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.956 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.956 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.956 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609575+0000", 2020-01-16T00:12:52.956 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019161661999999999 2020-01-16T00:12:52.957 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.957 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.957 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.957 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.957 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.957 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1240 3.3 3:c204aa4f:::smithi12313934-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-01-16T00:12:52.957 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:12:52.958 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 10.160601911000001, 2020-01-16T00:12:52.958 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019268828000000002, 2020-01-16T00:12:52.958 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.958 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:52.958 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.958 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.958 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.959 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1240 2020-01-16T00:12:52.959 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.959 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.959 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.959 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.959 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:12:52.959 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.959 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.960 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.960 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.960 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:12:52.960 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:52.960 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.960 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.960 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.960 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590512+0000", 2020-01-16T00:12:52.961 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0525e-05 2020-01-16T00:12:52.961 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.961 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.961 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.961 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590522+0000", 2020-01-16T00:12:52.961 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6418000000000001e-05 2020-01-16T00:12:52.961 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.961 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.962 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.962 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590539+0000", 2020-01-16T00:12:52.962 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0679999999999999e-05 2020-01-16T00:12:52.962 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.962 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.962 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.962 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590549+0000", 2020-01-16T00:12:52.962 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0083626789999999996 2020-01-16T00:12:52.963 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.963 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.963 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.963 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598912+0000", 2020-01-16T00:12:52.963 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013344 2020-01-16T00:12:52.964 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.964 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.964 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:52.964 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599045+0000", 2020-01-16T00:12:52.964 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0084002269999999997 2020-01-16T00:12:52.964 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.964 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.964 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.965 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607446+0000", 2020-01-16T00:12:52.965 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000109873 2020-01-16T00:12:52.965 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.965 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.965 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:52.965 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607555+0000", 2020-01-16T00:12:52.965 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0020573309999999999 2020-01-16T00:12:52.965 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.966 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.966 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.966 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609613+0000", 2020-01-16T00:12:52.966 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6351e-05 2020-01-16T00:12:52.966 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.966 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.966 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.966 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609699+0000", 2020-01-16T00:12:52.967 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3374999999999994e-05 2020-01-16T00:12:52.967 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.967 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.967 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.967 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609782+0000", 2020-01-16T00:12:52.967 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019268828000000002 2020-01-16T00:12:52.967 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.967 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.968 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.968 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.968 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.968 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1436 3.3 3:c204aa4f:::smithi12313934-45:head [setxattr _header (56) in=7b,truncate 2386594] snapc 0=[] ondisk+write+known_if_redirected e28)", 2020-01-16T00:12:52.968 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:47.900544+0000", 2020-01-16T00:12:52.968 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 4.8505712299999999, 2020-01-16T00:12:52.968 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016402521, 2020-01-16T00:12:52.969 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.969 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:52.969 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.969 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.969 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.969 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1436 2020-01-16T00:12:52.969 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.969 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.970 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.970 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.970 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900544+0000", 2020-01-16T00:12:52.970 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.970 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.970 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.970 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.970 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900544+0000", 2020-01-16T00:12:52.971 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:52.971 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.971 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.971 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.971 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900542+0000", 2020-01-16T00:12:52.971 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3862999999999999e-05 2020-01-16T00:12:52.971 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.972 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.972 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.972 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900556+0000", 2020-01-16T00:12:52.972 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5204e-05 2020-01-16T00:12:52.972 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.972 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.972 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.973 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900571+0000", 2020-01-16T00:12:52.973 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.092e-05 2020-01-16T00:12:52.973 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.973 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.974 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.974 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900582+0000", 2020-01-16T00:12:52.974 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002786026 2020-01-16T00:12:52.974 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.974 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.974 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.974 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.903368+0000", 2020-01-16T00:12:52.975 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0027123659999999999 2020-01-16T00:12:52.975 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.975 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.975 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:52.975 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.906081+0000", 2020-01-16T00:12:52.975 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0082707790000000007 2020-01-16T00:12:52.975 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.976 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.976 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.976 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.914352+0000", 2020-01-16T00:12:52.976 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020580300000000001 2020-01-16T00:12:52.976 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.976 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.976 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.976 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.914557+0000", 2020-01-16T00:12:52.977 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017729900000000001 2020-01-16T00:12:52.977 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.977 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.977 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2020-01-16T00:12:52.977 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.914735+0000", 2020-01-16T00:12:52.977 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00074632099999999999 2020-01-16T00:12:52.977 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.977 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.978 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:52.978 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.915481+0000", 2020-01-16T00:12:52.978 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012279599999999999 2020-01-16T00:12:52.978 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.978 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.978 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:52.978 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.915604+0000", 2020-01-16T00:12:52.978 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012619339999999999 2020-01-16T00:12:52.979 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.979 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.979 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:52.979 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.916866+0000", 2020-01-16T00:12:52.979 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1162000000000003e-05 2020-01-16T00:12:52.979 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.979 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.979 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:52.980 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.916907+0000", 2020-01-16T00:12:52.980 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9987999999999997e-05 2020-01-16T00:12:52.980 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.980 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.980 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.980 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.916947+0000", 2020-01-16T00:12:52.980 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016402521 2020-01-16T00:12:52.981 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.981 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.981 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.981 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.981 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.981 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1437 3.3 3:c204aa4f:::smithi12313934-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e28)", 2020-01-16T00:12:52.981 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:47.900594+0000", 2020-01-16T00:12:52.981 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 4.8505213669999998, 2020-01-16T00:12:52.982 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016745237999999999, 2020-01-16T00:12:52.982 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:52.982 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:52.982 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:52.982 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:52.982 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:52.983 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1437 2020-01-16T00:12:52.983 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.983 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:52.983 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.983 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:52.983 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900594+0000", 2020-01-16T00:12:52.984 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:52.984 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.984 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.984 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:52.984 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900594+0000", 2020-01-16T00:12:52.985 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:52.985 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.985 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.985 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:52.985 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900593+0000", 2020-01-16T00:12:52.985 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5059999999999995e-06 2020-01-16T00:12:52.985 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.986 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.986 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:52.986 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900601+0000", 2020-01-16T00:12:52.986 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5077e-05 2020-01-16T00:12:52.986 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.986 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.986 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:52.986 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900616+0000", 2020-01-16T00:12:52.987 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0183000000000001e-05 2020-01-16T00:12:52.987 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.987 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.987 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:52.987 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900627+0000", 2020-01-16T00:12:52.987 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0063667660000000003 2020-01-16T00:12:52.987 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.987 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.988 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.988 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.906993+0000", 2020-01-16T00:12:52.988 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8988e-05 2020-01-16T00:12:52.988 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.988 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.988 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:52.988 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.907032+0000", 2020-01-16T00:12:52.989 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0080128850000000008 2020-01-16T00:12:52.989 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.989 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.989 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.989 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.915045+0000", 2020-01-16T00:12:52.989 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013200899999999999 2020-01-16T00:12:52.989 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.989 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.990 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:52.990 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.915177+0000", 2020-01-16T00:12:52.990 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0018373090000000001 2020-01-16T00:12:52.990 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.990 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.990 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:52.990 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.917015+0000", 2020-01-16T00:12:52.990 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016701699999999999 2020-01-16T00:12:52.991 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.991 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.991 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:52.991 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.917182+0000", 2020-01-16T00:12:52.991 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015782400000000001 2020-01-16T00:12:52.991 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:52.991 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:52.992 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:52.992 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.917339+0000", 2020-01-16T00:12:52.992 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016745237999999999 2020-01-16T00:12:52.992 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.992 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.992 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.992 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:52.992 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:52.993 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:52.996 INFO:tasks.rados.rados.0.smithi123.stdout:862: writing smithi12313934-25 from 524418 to 977510 tid 1 2020-01-16T00:12:52.996 INFO:tasks.rados.rados.0.smithi123.stdout:862: writing smithi12313934-25 from 1732455 to 2320834 tid 2 2020-01-16T00:12:52.997 INFO:tasks.rados.rados.0.smithi123.stdout:862: writing smithi12313934-25 from 2779130 to 2924418 tid 3 2020-01-16T00:12:52.997 INFO:tasks.rados.rados.0.smithi123.stdout:863: read oid 31 snap -1 2020-01-16T00:12:52.997 INFO:tasks.rados.rados.0.smithi123.stdout:863: expect (ObjNum 132 snap 0 seq_num 132) 2020-01-16T00:12:52.997 INFO:tasks.rados.rados.0.smithi123.stdout:864: write oid 35 current snap is 0 2020-01-16T00:12:52.997 INFO:tasks.rados.rados.0.smithi123.stdout:864: seq_num 175 ranges {541225=644694,1907290=448390} 2020-01-16T00:12:52.997 INFO:tasks.rados.rados.0.smithi123.stdout:864: writing smithi12313934-35 from 541225 to 1185919 tid 1 2020-01-16T00:12:52.997 INFO:tasks.rados.rados.0.smithi123.stdout:864: writing smithi12313934-35 from 1907290 to 2355680 tid 2 2020-01-16T00:12:52.998 INFO:tasks.rados.rados.0.smithi123.stdout:865: write (excl) oid 29 current snap is 0 2020-01-16T00:12:52.998 INFO:tasks.rados.rados.0.smithi123.stdout:865: seq_num 176 ranges {558032=719943,1849419=592048} 2020-01-16T00:12:52.998 INFO:tasks.rados.rados.0.smithi123.stdout:865: writing smithi12313934-29 from 558032 to 1277975 tid 1 2020-01-16T00:12:52.998 INFO:tasks.rados.rados.0.smithi123.stdout:865: writing smithi12313934-29 from 1849419 to 2441467 tid 2 2020-01-16T00:12:52.998 INFO:tasks.rados.rados.0.smithi123.stdout:866: read oid 36 snap -1 2020-01-16T00:12:52.998 INFO:tasks.rados.rados.0.smithi123.stdout:866: expect (ObjNum 161 snap 0 seq_num 161) 2020-01-16T00:12:52.998 INFO:tasks.rados.rados.0.smithi123.stdout:867: read oid 40 snap -1 2020-01-16T00:12:52.999 INFO:tasks.rados.rados.0.smithi123.stdout:867: expect (ObjNum 138 snap 0 seq_num 138) 2020-01-16T00:12:52.999 INFO:tasks.rados.rados.0.smithi123.stdout:868: read oid 1 snap -1 2020-01-16T00:12:52.999 INFO:tasks.rados.rados.0.smithi123.stdout:868: expect (ObjNum 166 snap 0 seq_num 166) 2020-01-16T00:12:52.999 INFO:tasks.rados.rados.0.smithi123.stdout:869: write oid 50 current snap is 0 2020-01-16T00:12:52.999 INFO:tasks.rados.rados.0.smithi123.stdout:869: seq_num 177 ranges {574839=795192,1907901=452059} 2020-01-16T00:12:52.999 INFO:tasks.rados.rados.0.smithi123.stdout:869: writing smithi12313934-50 from 574839 to 1370031 tid 1 2020-01-16T00:12:52.999 INFO:tasks.rados.rados.0.smithi123.stdout:869: writing smithi12313934-50 from 1907901 to 2359960 tid 2 2020-01-16T00:12:52.999 INFO:tasks.rados.rados.0.smithi123.stdout:870: read oid 6 snap -1 2020-01-16T00:12:53.000 INFO:tasks.rados.rados.0.smithi123.stdout:870: expect (ObjNum 137 snap 0 seq_num 137) 2020-01-16T00:12:53.000 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:53.000 INFO:tasks.rados.rados.0.smithi123.stdout:855: expect (ObjNum 168 snap 0 seq_num 168) 2020-01-16T00:12:53.000 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:12:52.995+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:12:53.000 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-01-16T00:12:53.001 INFO:tasks.rados.rados.0.smithi123.stdout:857: finishing write tid 1 to smithi12313934-10 2020-01-16T00:12:53.001 INFO:tasks.rados.rados.0.smithi123.stdout:854: finishing write tid 3 to smithi12313934-24 2020-01-16T00:12:53.002 INFO:tasks.rados.rados.0.smithi123.stdout:854: finishing write tid 5 to smithi12313934-24 2020-01-16T00:12:53.002 INFO:tasks.rados.rados.0.smithi123.stdout:854: finishing write tid 6 to smithi12313934-24 2020-01-16T00:12:53.002 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 24 v 73 (ObjNum 171 snap 0 seq_num 171) dirty exists 2020-01-16T00:12:53.002 INFO:tasks.rados.rados.0.smithi123.stdout:854: left oid 24 (ObjNum 171 snap 0 seq_num 171) 2020-01-16T00:12:53.002 INFO:tasks.rados.rados.0.smithi123.stdout:859: expect (ObjNum 124 snap 0 seq_num 124) 2020-01-16T00:12:53.084 INFO:tasks.rados.rados.0.smithi123.stdout:861: expect (ObjNum 22 snap 0 seq_num 22) 2020-01-16T00:12:53.095 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:53.097+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:12:53.118 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:53.118 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:12:53.118 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:12:53.118 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:53.127 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-01-16T00:12:53.152 INFO:tasks.rados.rados.0.smithi123.stdout:857: finishing write tid 2 to smithi12313934-10 2020-01-16T00:12:53.152 INFO:tasks.rados.rados.0.smithi123.stdout:857: finishing write tid 4 to smithi12313934-10 2020-01-16T00:12:53.152 INFO:tasks.rados.rados.0.smithi123.stdout:857: finishing write tid 5 to smithi12313934-10 2020-01-16T00:12:53.152 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 10 v 78 (ObjNum 173 snap 0 seq_num 173) dirty exists 2020-01-16T00:12:53.152 INFO:tasks.rados.rados.0.smithi123.stdout:857: left oid 10 (ObjNum 173 snap 0 seq_num 173) 2020-01-16T00:12:53.153 INFO:tasks.rados.rados.0.smithi123.stdout:858: expect (ObjNum 133 snap 0 seq_num 133) 2020-01-16T00:12:53.196 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:53.199+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:12:53.239 INFO:tasks.rados.rados.0.smithi123.stdout:862: finishing write tid 1 to smithi12313934-25 2020-01-16T00:12:53.240 INFO:tasks.rados.rados.0.smithi123.stdout:860: expect (ObjNum 97 snap 0 seq_num 97) 2020-01-16T00:12:53.283 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:53.284 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:12:53.284 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:12:53.284 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:12:53.284 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:53.292 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-01-16T00:12:53.297 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:12:53.299+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:12:53.400 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:53.401+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:53.405 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:53.406 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:12:53.406 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:12:53.406 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:12:53.406 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.406 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:898 3.d 3:bd9fc72f:::smithi12313934-5:head [write 776490~714607] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:53.406 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.131050+0000", 2020-01-16T00:12:53.406 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 16.273590670000001, 2020-01-16T00:12:53.406 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0076965840000000002, 2020-01-16T00:12:53.407 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:53.407 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:53.407 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:53.407 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:53.407 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:53.407 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 898 2020-01-16T00:12:53.407 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.408 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:53.408 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.408 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:53.408 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.131050+0000", 2020-01-16T00:12:53.408 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:53.408 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.409 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.409 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:53.409 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.131050+0000", 2020-01-16T00:12:53.409 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:12:53.409 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.409 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.409 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:53.410 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.131046+0000", 2020-01-16T00:12:53.410 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0013709010000000001 2020-01-16T00:12:53.410 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.410 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.410 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:53.411 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132417+0000", 2020-01-16T00:12:53.411 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1613000000000001e-05 2020-01-16T00:12:53.411 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.411 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.411 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:53.411 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132438+0000", 2020-01-16T00:12:53.411 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0764000000000001e-05 2020-01-16T00:12:53.411 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.412 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.412 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:53.412 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132449+0000", 2020-01-16T00:12:53.412 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5473e-05 2020-01-16T00:12:53.412 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.412 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.413 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:53.413 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132535+0000", 2020-01-16T00:12:53.413 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013572000000000001 2020-01-16T00:12:53.413 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.413 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.413 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:53.413 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132670+0000", 2020-01-16T00:12:53.417 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000124719 2020-01-16T00:12:53.418 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.418 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.418 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:12:53.418 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132795+0000", 2020-01-16T00:12:53.421 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023122680000000001 2020-01-16T00:12:53.421 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.421 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.421 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:53.423 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.135107+0000", 2020-01-16T00:12:53.423 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00094608099999999998 2020-01-16T00:12:53.423 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.423 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.423 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:53.423 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136053+0000", 2020-01-16T00:12:53.423 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0026099780000000002 2020-01-16T00:12:53.424 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.424 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.424 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:53.424 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.138663+0000", 2020-01-16T00:12:53.424 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4024999999999999e-05 2020-01-16T00:12:53.424 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.424 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.425 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:53.425 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.138707+0000", 2020-01-16T00:12:53.425 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9035000000000001e-05 2020-01-16T00:12:53.425 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.425 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.425 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:53.425 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.138746+0000", 2020-01-16T00:12:53.426 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0076965840000000002 2020-01-16T00:12:53.426 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.426 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:53.426 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.426 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.426 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.426 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:899 3.d 3:bd9fc72f:::smithi12313934-5:head [write 2246563~579356] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:53.426 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.138816+0000", 2020-01-16T00:12:53.427 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 16.265824799000001, 2020-01-16T00:12:53.427 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0073434679999999997, 2020-01-16T00:12:53.427 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:53.427 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:53.427 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:53.427 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:53.427 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:53.428 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 899 2020-01-16T00:12:53.428 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.428 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:53.428 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.428 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:53.428 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.138816+0000", 2020-01-16T00:12:53.428 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:53.428 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.429 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.437 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:53.437 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.138816+0000", 2020-01-16T00:12:53.437 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:53.438 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.438 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.438 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:53.438 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.138813+0000", 2020-01-16T00:12:53.438 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0029589059999999999 2020-01-16T00:12:53.438 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.438 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.438 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:53.439 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.141772+0000", 2020-01-16T00:12:53.439 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3594999999999999e-05 2020-01-16T00:12:53.439 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.439 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.439 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:53.439 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.141806+0000", 2020-01-16T00:12:53.439 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9459999999999993e-06 2020-01-16T00:12:53.440 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.440 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.440 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:53.440 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.141816+0000", 2020-01-16T00:12:53.440 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4413000000000003e-05 2020-01-16T00:12:53.440 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.440 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.440 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:53.441 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.141860+0000", 2020-01-16T00:12:53.441 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6380999999999998e-05 2020-01-16T00:12:53.441 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.441 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.441 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:53.441 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.141947+0000", 2020-01-16T00:12:53.441 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013747 2020-01-16T00:12:53.441 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.442 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.442 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:12:53.442 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.142084+0000", 2020-01-16T00:12:53.442 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002304281 2020-01-16T00:12:53.442 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.442 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.442 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:53.443 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.144388+0000", 2020-01-16T00:12:53.443 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000348754 2020-01-16T00:12:53.443 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.443 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.443 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:53.443 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.144737+0000", 2020-01-16T00:12:53.443 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012783429999999999 2020-01-16T00:12:53.443 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.444 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.444 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:53.444 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.146015+0000", 2020-01-16T00:12:53.444 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5207000000000001e-05 2020-01-16T00:12:53.444 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.444 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.444 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:53.444 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.146081+0000", 2020-01-16T00:12:53.445 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.8583000000000003e-05 2020-01-16T00:12:53.445 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.445 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.445 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:53.445 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.146159+0000", 2020-01-16T00:12:53.445 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0073434679999999997 2020-01-16T00:12:53.445 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.447 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:53.447 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.448 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.448 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.448 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.5.13:36 2.0 2:20e40cec:::smithi12313934-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 113152~1699840] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:12:53.448 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:12:53.448 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 15.572701598, 2020-01-16T00:12:53.448 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011736636, 2020-01-16T00:12:53.448 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:53.449 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:53.449 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:53.449 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.449 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:53.449 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:12:53.449 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:53.449 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.450 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.450 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:53.450 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:12:53.450 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999952 2020-01-16T00:12:53.450 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.450 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.450 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:53.451 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831934+0000", 2020-01-16T00:12:53.451 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001034325 2020-01-16T00:12:53.451 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.451 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.451 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:53.451 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.832968+0000", 2020-01-16T00:12:53.451 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.506e-05 2020-01-16T00:12:53.451 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.452 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.452 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:53.452 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833004+0000", 2020-01-16T00:12:53.452 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3406e-05 2020-01-16T00:12:53.452 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.452 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.452 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:53.453 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833017+0000", 2020-01-16T00:12:53.453 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015477000000000001 2020-01-16T00:12:53.453 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.453 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.453 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:53.453 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833172+0000", 2020-01-16T00:12:53.453 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033750700000000002 2020-01-16T00:12:53.454 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.454 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.454 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:53.454 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833509+0000", 2020-01-16T00:12:53.454 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00055861400000000003 2020-01-16T00:12:53.454 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.454 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.455 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:12:53.455 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.834068+0000", 2020-01-16T00:12:53.455 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.007474545 2020-01-16T00:12:53.455 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.455 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.456 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:53.456 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.841542+0000", 2020-01-16T00:12:53.456 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00056319499999999995 2020-01-16T00:12:53.456 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.456 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.456 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:53.456 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.842106+0000", 2020-01-16T00:12:53.456 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001371017 2020-01-16T00:12:53.457 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.457 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.457 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:53.457 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843477+0000", 2020-01-16T00:12:53.457 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000123938 2020-01-16T00:12:53.457 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.457 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.458 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:53.458 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843601+0000", 2020-01-16T00:12:53.458 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5012000000000006e-05 2020-01-16T00:12:53.458 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.458 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.458 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:53.458 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843676+0000", 2020-01-16T00:12:53.458 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011736636 2020-01-16T00:12:53.459 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.459 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:53.459 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.459 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.459 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.459 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.5.13:40 2.0 2:20e40cec:::smithi12313934-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 113152~1699840] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:12:53.459 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.847376+0000", 2020-01-16T00:12:53.460 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 15.557264869999999, 2020-01-16T00:12:53.460 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010871093, 2020-01-16T00:12:53.460 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:53.460 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:53.460 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:53.460 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.460 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:53.460 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.847376+0000", 2020-01-16T00:12:53.461 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:53.461 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.461 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.461 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:53.461 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.847376+0000", 2020-01-16T00:12:53.461 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:12:53.461 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.462 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.462 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:53.462 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.847373+0000", 2020-01-16T00:12:53.462 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00077394699999999998 2020-01-16T00:12:53.462 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.462 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.462 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:53.462 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.848147+0000", 2020-01-16T00:12:53.463 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.173e-05 2020-01-16T00:12:53.463 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.463 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.463 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:53.463 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.848168+0000", 2020-01-16T00:12:53.463 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2738999999999999e-05 2020-01-16T00:12:53.463 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.464 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.464 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:53.464 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.848181+0000", 2020-01-16T00:12:53.464 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4627999999999999e-05 2020-01-16T00:12:53.464 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.464 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.464 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:53.464 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.848236+0000", 2020-01-16T00:12:53.465 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000141311 2020-01-16T00:12:53.465 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.465 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.465 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:53.465 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.848377+0000", 2020-01-16T00:12:53.465 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015462700000000001 2020-01-16T00:12:53.465 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.466 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.466 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:12:53.466 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.848532+0000", 2020-01-16T00:12:53.466 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0039933540000000002 2020-01-16T00:12:53.466 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.466 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.466 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:53.466 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.852525+0000", 2020-01-16T00:12:53.467 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0051016129999999996 2020-01-16T00:12:53.467 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.467 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.467 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:53.467 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.857627+0000", 2020-01-16T00:12:53.467 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00044689099999999999 2020-01-16T00:12:53.467 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.468 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.468 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:53.468 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.858073+0000", 2020-01-16T00:12:53.468 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8947e-05 2020-01-16T00:12:53.468 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.468 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.468 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:53.469 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.858172+0000", 2020-01-16T00:12:53.469 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4369999999999994e-05 2020-01-16T00:12:53.469 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.469 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.469 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:53.469 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.858247+0000", 2020-01-16T00:12:53.469 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010871093 2020-01-16T00:12:53.469 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.470 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:53.470 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.470 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.470 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.470 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.5.13:42 2.0 2:313458df:::smithi12313934-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 263680~1668096] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:12:53.470 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.939100+0000", 2020-01-16T00:12:53.470 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 15.465540962, 2020-01-16T00:12:53.471 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010934255, 2020-01-16T00:12:53.471 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:53.471 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:53.471 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:53.471 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.471 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:53.471 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939100+0000", 2020-01-16T00:12:53.471 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:53.472 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.472 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.472 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:53.472 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939100+0000", 2020-01-16T00:12:53.472 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:53.472 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.472 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.472 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:53.473 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939097+0000", 2020-01-16T00:12:53.473 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00049374800000000002 2020-01-16T00:12:53.473 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.473 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.473 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:53.473 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939591+0000", 2020-01-16T00:12:53.473 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0840999999999999e-05 2020-01-16T00:12:53.474 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.474 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.474 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:53.474 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939612+0000", 2020-01-16T00:12:53.474 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4749999999999998e-06 2020-01-16T00:12:53.474 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.474 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.475 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:53.475 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939618+0000", 2020-01-16T00:12:53.475 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6371e-05 2020-01-16T00:12:53.475 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.475 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.475 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:53.475 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939674+0000", 2020-01-16T00:12:53.476 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2e-05 2020-01-16T00:12:53.476 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.476 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.476 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:53.476 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939766+0000", 2020-01-16T00:12:53.476 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020133999999999999 2020-01-16T00:12:53.476 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.476 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.477 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:12:53.477 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939968+0000", 2020-01-16T00:12:53.477 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0027942560000000002 2020-01-16T00:12:53.477 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.477 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.477 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:53.477 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.942762+0000", 2020-01-16T00:12:53.478 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.004998466 2020-01-16T00:12:53.478 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.478 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.478 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:53.478 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.947760+0000", 2020-01-16T00:12:53.478 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0021783330000000002 2020-01-16T00:12:53.478 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.478 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.479 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:53.479 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.949939+0000", 2020-01-16T00:12:53.479 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2391999999999997e-05 2020-01-16T00:12:53.479 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.479 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.479 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:53.479 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.949981+0000", 2020-01-16T00:12:53.480 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2629e-05 2020-01-16T00:12:53.480 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.480 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.480 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:53.480 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.950034+0000", 2020-01-16T00:12:53.480 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010934255 2020-01-16T00:12:53.480 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.480 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:53.481 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.481 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.481 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.481 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.5.13:43 2.0 2:313458df:::smithi12313934-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 263680~1668096] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:12:53.481 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.953019+0000", 2020-01-16T00:12:53.481 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 15.451621166000001, 2020-01-16T00:12:53.481 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0085200620000000001, 2020-01-16T00:12:53.482 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:53.482 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:53.482 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:53.482 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.482 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:53.482 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953019+0000", 2020-01-16T00:12:53.482 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:53.482 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.483 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.483 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:53.483 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953019+0000", 2020-01-16T00:12:53.483 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:12:53.483 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.483 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.483 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:53.484 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953016+0000", 2020-01-16T00:12:53.484 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00058927999999999997 2020-01-16T00:12:53.484 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.484 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.484 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:53.484 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953606+0000", 2020-01-16T00:12:53.484 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0846999999999999e-05 2020-01-16T00:12:53.484 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.485 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.485 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:53.485 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953627+0000", 2020-01-16T00:12:53.485 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7020000000000004e-06 2020-01-16T00:12:53.485 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.485 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.485 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:53.486 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953633+0000", 2020-01-16T00:12:53.486 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.2762000000000005e-05 2020-01-16T00:12:53.486 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.486 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.486 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:53.486 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953716+0000", 2020-01-16T00:12:53.486 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000139238 2020-01-16T00:12:53.487 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.487 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.487 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:53.487 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953855+0000", 2020-01-16T00:12:53.487 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020386300000000001 2020-01-16T00:12:53.487 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.487 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.487 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:12:53.488 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.954059+0000", 2020-01-16T00:12:53.488 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0031229399999999998 2020-01-16T00:12:53.488 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.488 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.488 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:53.488 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.957182+0000", 2020-01-16T00:12:53.488 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0034781980000000001 2020-01-16T00:12:53.489 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.489 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.489 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:53.489 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.960660+0000", 2020-01-16T00:12:53.489 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00076596200000000002 2020-01-16T00:12:53.489 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.489 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.489 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:53.490 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.961426+0000", 2020-01-16T00:12:53.490 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1937e-05 2020-01-16T00:12:53.490 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.490 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.490 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:53.490 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.961488+0000", 2020-01-16T00:12:53.490 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1285000000000001e-05 2020-01-16T00:12:53.491 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.491 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.491 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:53.491 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.961539+0000", 2020-01-16T00:12:53.491 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0085200620000000001 2020-01-16T00:12:53.491 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.491 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:53.491 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.492 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.492 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.492 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(osd.4.0:114 3.9 e24/20 3:9fc687b2:::smithi12313934-44:head v 24'74, mlcod=24'72)", 2020-01-16T00:12:53.492 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.324586+0000", 2020-01-16T00:12:53.492 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 15.080054587999999, 2020-01-16T00:12:53.492 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.007288157, 2020-01-16T00:12:53.492 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:53.492 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:53.493 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:53.493 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.493 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:53.493 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.324586+0000", 2020-01-16T00:12:53.493 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:53.493 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.493 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.493 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:53.494 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.324586+0000", 2020-01-16T00:12:53.494 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999938 2020-01-16T00:12:53.494 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.494 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.494 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:53.494 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.324580+0000", 2020-01-16T00:12:53.494 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001520373 2020-01-16T00:12:53.494 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.495 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.495 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:53.495 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.326100+0000", 2020-01-16T00:12:53.495 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8723000000000002e-05 2020-01-16T00:12:53.495 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.495 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.495 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:53.496 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.326139+0000", 2020-01-16T00:12:53.496 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8349e-05 2020-01-16T00:12:53.496 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.496 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.496 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:53.496 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.326157+0000", 2020-01-16T00:12:53.496 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000110313 2020-01-16T00:12:53.496 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.497 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.497 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:53.497 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.326268+0000", 2020-01-16T00:12:53.497 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.9811e-05 2020-01-16T00:12:53.497 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.497 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.497 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:53.497 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.326337+0000", 2020-01-16T00:12:53.498 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0054869039999999999 2020-01-16T00:12:53.498 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.498 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.498 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:53.498 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.331824+0000", 2020-01-16T00:12:53.498 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9693e-05 2020-01-16T00:12:53.498 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.498 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.499 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:53.499 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.331874+0000", 2020-01-16T00:12:53.499 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.007288157 2020-01-16T00:12:53.499 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.499 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:53.499 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.499 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.499 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.500 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1186 3.d 3:b1521a3e:::smithi12313934-25:head [write 682384~456830,stat] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:12:53.500 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.258156+0000", 2020-01-16T00:12:53.500 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 11.146484998, 2020-01-16T00:12:53.500 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0092969679999999992, 2020-01-16T00:12:53.500 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:53.500 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:53.500 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:53.501 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:53.501 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:53.501 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1186 2020-01-16T00:12:53.501 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.501 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:53.501 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.501 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:53.501 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.258156+0000", 2020-01-16T00:12:53.502 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:53.502 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.502 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.502 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:53.502 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.258156+0000", 2020-01-16T00:12:53.502 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999943 2020-01-16T00:12:53.502 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.503 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.503 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:53.503 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.258150+0000", 2020-01-16T00:12:53.503 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001866517 2020-01-16T00:12:53.503 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.503 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.503 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:53.503 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.260016+0000", 2020-01-16T00:12:53.504 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4024999999999999e-05 2020-01-16T00:12:53.504 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.504 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.504 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:53.504 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.260070+0000", 2020-01-16T00:12:53.504 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4295e-05 2020-01-16T00:12:53.504 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.504 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.505 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:53.505 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.260085+0000", 2020-01-16T00:12:53.505 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013625599999999999 2020-01-16T00:12:53.505 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.505 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.505 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:53.505 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.260221+0000", 2020-01-16T00:12:53.505 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000230015 2020-01-16T00:12:53.506 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.506 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.506 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:53.506 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.260451+0000", 2020-01-16T00:12:53.506 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00031965999999999999 2020-01-16T00:12:53.506 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.506 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.508 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:12:53.508 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.260770+0000", 2020-01-16T00:12:53.509 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0031788649999999999 2020-01-16T00:12:53.509 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.509 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.509 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:53.509 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.263949+0000", 2020-01-16T00:12:53.509 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0022620090000000002 2020-01-16T00:12:53.509 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.510 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.510 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:53.510 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.266211+0000", 2020-01-16T00:12:53.510 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00097717199999999989 2020-01-16T00:12:53.510 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.510 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.510 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:53.510 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.267188+0000", 2020-01-16T00:12:53.511 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000128594 2020-01-16T00:12:53.511 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.511 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.511 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:53.511 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.267317+0000", 2020-01-16T00:12:53.511 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013541200000000001 2020-01-16T00:12:53.511 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.511 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.512 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:53.512 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.267452+0000", 2020-01-16T00:12:53.512 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0092969679999999992 2020-01-16T00:12:53.512 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.512 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:53.512 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.512 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.512 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.513 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1187 3.d 3:b1521a3e:::smithi12313934-25:head [write 1721042~423699] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:12:53.513 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.263828+0000", 2020-01-16T00:12:53.513 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 11.140812525999999, 2020-01-16T00:12:53.513 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011305709000000001, 2020-01-16T00:12:53.513 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:53.513 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:53.513 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:53.514 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:53.514 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:53.514 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1187 2020-01-16T00:12:53.514 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.514 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:53.514 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.514 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:53.514 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.263828+0000", 2020-01-16T00:12:53.515 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:53.515 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.515 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.515 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:53.515 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.263828+0000", 2020-01-16T00:12:53.515 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:12:53.515 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.515 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.516 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:53.516 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.263824+0000", 2020-01-16T00:12:53.516 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00048485899999999997 2020-01-16T00:12:53.516 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.516 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.516 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:53.516 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264309+0000", 2020-01-16T00:12:53.516 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5622999999999999e-05 2020-01-16T00:12:53.517 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.517 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.517 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:53.517 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264325+0000", 2020-01-16T00:12:53.517 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2530000000000004e-06 2020-01-16T00:12:53.517 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.517 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.518 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:53.518 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264330+0000", 2020-01-16T00:12:53.518 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000102252 2020-01-16T00:12:53.518 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.518 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.518 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:53.518 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264432+0000", 2020-01-16T00:12:53.518 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022361000000000001 2020-01-16T00:12:53.519 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.519 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.519 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:53.519 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264656+0000", 2020-01-16T00:12:53.519 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0028688849999999998 2020-01-16T00:12:53.519 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.519 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.520 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:53.520 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.267525+0000", 2020-01-16T00:12:53.520 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00023640900000000001 2020-01-16T00:12:53.520 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.520 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.520 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:53.520 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.267761+0000", 2020-01-16T00:12:53.520 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00035667000000000001 2020-01-16T00:12:53.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.521 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.521 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:12:53.521 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.268118+0000", 2020-01-16T00:12:53.521 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023898040000000001 2020-01-16T00:12:53.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.521 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.522 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:53.522 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.270507+0000", 2020-01-16T00:12:53.522 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014919690000000001 2020-01-16T00:12:53.522 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.522 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.522 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:53.522 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.271999+0000", 2020-01-16T00:12:53.522 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0029764510000000002 2020-01-16T00:12:53.523 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.523 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.523 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:53.523 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.274976+0000", 2020-01-16T00:12:53.523 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5409999999999997e-05 2020-01-16T00:12:53.523 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.523 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.523 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:53.524 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.275061+0000", 2020-01-16T00:12:53.524 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2452000000000004e-05 2020-01-16T00:12:53.524 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.524 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.524 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:53.524 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.275134+0000", 2020-01-16T00:12:53.524 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011305709000000001 2020-01-16T00:12:53.525 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.525 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:53.525 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.525 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.525 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.525 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1226 3.d 3:b76659c4:::smithi12313934-47:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:12:53.525 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.386627+0000", 2020-01-16T00:12:53.526 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 11.018013352000001, 2020-01-16T00:12:53.526 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0076727160000000004, 2020-01-16T00:12:53.526 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:53.526 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:53.526 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:53.526 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:53.526 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:53.526 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1226 2020-01-16T00:12:53.527 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.527 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:53.527 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.527 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:53.527 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.386627+0000", 2020-01-16T00:12:53.527 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:53.527 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.528 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.528 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:53.528 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.386627+0000", 2020-01-16T00:12:53.528 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:53.528 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.528 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.528 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:53.528 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.386627+0000", 2020-01-16T00:12:53.529 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0120000000000001e-06 2020-01-16T00:12:53.529 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.529 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.529 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:53.529 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.386630+0000", 2020-01-16T00:12:53.529 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3579999999999996e-06 2020-01-16T00:12:53.530 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.530 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.530 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:53.530 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.386634+0000", 2020-01-16T00:12:53.530 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3739999999999998e-06 2020-01-16T00:12:53.530 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.530 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.531 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:53.531 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.386637+0000", 2020-01-16T00:12:53.531 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0067046830000000003 2020-01-16T00:12:53.531 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.531 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.531 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:53.531 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.393342+0000", 2020-01-16T00:12:53.531 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00023329199999999999 2020-01-16T00:12:53.532 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.532 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.532 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:53.532 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.393575+0000", 2020-01-16T00:12:53.532 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00072454099999999999 2020-01-16T00:12:53.532 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.532 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.532 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:53.533 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.394300+0000", 2020-01-16T00:12:53.533 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0076727160000000004 2020-01-16T00:12:53.533 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.533 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:53.533 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.533 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.533 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.534 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1259 3.d 3:b374ba21:::smithi12313934-20:head [read 0~3932805,checksum] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:12:53.534 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.833857+0000", 2020-01-16T00:12:53.534 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 10.570783781999999, 2020-01-16T00:12:53.534 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0094623380000000007, 2020-01-16T00:12:53.534 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:53.534 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:53.534 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:53.534 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:53.535 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:53.535 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1259 2020-01-16T00:12:53.535 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.535 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:53.535 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.535 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:53.535 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.833857+0000", 2020-01-16T00:12:53.535 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:53.536 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.536 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.536 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:53.536 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.833857+0000", 2020-01-16T00:12:53.536 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:53.536 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.536 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.536 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:53.537 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.833856+0000", 2020-01-16T00:12:53.537 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.9659999999999999e-06 2020-01-16T00:12:53.537 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.537 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.537 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:53.537 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.833863+0000", 2020-01-16T00:12:53.537 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4699999999999996e-06 2020-01-16T00:12:53.538 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.538 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.538 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:53.538 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.833870+0000", 2020-01-16T00:12:53.538 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0039999999999999e-06 2020-01-16T00:12:53.538 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.538 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.538 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:53.539 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.833875+0000", 2020-01-16T00:12:53.539 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0069094480000000003 2020-01-16T00:12:53.539 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.539 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.539 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:53.539 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.840785+0000", 2020-01-16T00:12:53.539 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020800699999999999 2020-01-16T00:12:53.539 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.540 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.540 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:53.540 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.840993+0000", 2020-01-16T00:12:53.540 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023263960000000001 2020-01-16T00:12:53.540 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.540 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.540 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:53.540 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.843319+0000", 2020-01-16T00:12:53.541 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0094623380000000007 2020-01-16T00:12:53.541 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.541 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:53.541 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.541 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.541 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.541 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1272 3.d 3:b3dd5d45:::smithi12313934-16:head [read 0~3008420,checksum] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:12:53.541 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074658+0000", 2020-01-16T00:12:53.542 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 10.329982941999999, 2020-01-16T00:12:53.542 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0083000039999999997, 2020-01-16T00:12:53.542 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:53.542 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:53.542 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:53.542 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:53.542 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:53.543 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1272 2020-01-16T00:12:53.543 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.543 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:53.543 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.543 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:53.543 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074658+0000", 2020-01-16T00:12:53.543 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:53.543 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.544 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.544 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:53.544 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074658+0000", 2020-01-16T00:12:53.544 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:53.544 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.544 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.544 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:53.544 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074657+0000", 2020-01-16T00:12:53.545 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7270000000000007e-06 2020-01-16T00:12:53.545 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.545 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.545 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:53.545 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074664+0000", 2020-01-16T00:12:53.545 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1094000000000002e-05 2020-01-16T00:12:53.545 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.545 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.546 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:53.546 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074685+0000", 2020-01-16T00:12:53.546 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0260000000000004e-06 2020-01-16T00:12:53.546 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.546 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.546 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:53.546 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074691+0000", 2020-01-16T00:12:53.547 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0061756880000000004 2020-01-16T00:12:53.547 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.547 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.547 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:53.547 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.080867+0000", 2020-01-16T00:12:53.547 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015871700000000001 2020-01-16T00:12:53.547 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.548 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.548 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:53.548 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.081026+0000", 2020-01-16T00:12:53.548 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0019317519999999999 2020-01-16T00:12:53.548 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.548 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.548 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:53.548 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.082958+0000", 2020-01-16T00:12:53.549 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0083000039999999997 2020-01-16T00:12:53.549 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.549 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:53.549 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.549 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.549 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.549 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1276 3.d 3:b1521a3e:::smithi12313934-25:head [read 0~3882384,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:12:53.550 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:12:53.550 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 10.329942028, 2020-01-16T00:12:53.550 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015839157999999999, 2020-01-16T00:12:53.550 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:53.550 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:53.550 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:53.550 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:53.550 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:53.551 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1276 2020-01-16T00:12:53.551 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.551 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:53.551 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.551 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:53.551 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:12:53.551 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:53.551 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.552 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.552 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:53.552 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:12:53.552 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:53.552 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.552 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.552 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:53.552 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:12:53.553 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5429999999999999e-06 2020-01-16T00:12:53.553 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.553 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.553 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:53.553 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074704+0000", 2020-01-16T00:12:53.553 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0839999999999992e-06 2020-01-16T00:12:53.553 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.554 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.554 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:53.554 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074712+0000", 2020-01-16T00:12:53.554 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.173e-06 2020-01-16T00:12:53.554 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.554 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.554 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:53.555 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074718+0000", 2020-01-16T00:12:53.555 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0083034700000000003 2020-01-16T00:12:53.555 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.555 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.555 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:53.555 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.083022+0000", 2020-01-16T00:12:53.555 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028456400000000001 2020-01-16T00:12:53.556 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.556 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.556 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:53.556 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.083306+0000", 2020-01-16T00:12:53.556 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0072312640000000003 2020-01-16T00:12:53.556 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.556 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.556 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:53.557 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090538+0000", 2020-01-16T00:12:53.557 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015839157999999999 2020-01-16T00:12:53.557 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.557 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:53.557 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.557 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.557 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.557 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1277 3.d 3:b1521a3e:::smithi12313934-25:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:12:53.558 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:12:53.558 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 10.329898438000001, 2020-01-16T00:12:53.558 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016646459999999998, 2020-01-16T00:12:53.558 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:53.558 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:53.558 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:53.558 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:53.558 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:53.559 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1277 2020-01-16T00:12:53.559 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.559 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:53.559 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.559 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:53.559 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:12:53.559 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:53.560 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.560 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.560 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:53.560 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:12:53.560 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:53.560 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.560 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.560 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:53.561 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074741+0000", 2020-01-16T00:12:53.561 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0776e-05 2020-01-16T00:12:53.561 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.561 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.561 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:53.561 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074752+0000", 2020-01-16T00:12:53.561 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7600000000000002e-06 2020-01-16T00:12:53.561 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.562 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.562 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:53.562 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074759+0000", 2020-01-16T00:12:53.562 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6840000000000001e-06 2020-01-16T00:12:53.562 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.562 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.562 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:53.563 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074765+0000", 2020-01-16T00:12:53.563 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015815216 2020-01-16T00:12:53.563 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.563 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.563 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:53.563 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090580+0000", 2020-01-16T00:12:53.563 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000182404 2020-01-16T00:12:53.563 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.564 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.564 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:53.564 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090763+0000", 2020-01-16T00:12:53.564 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00062579300000000003 2020-01-16T00:12:53.564 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.564 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.564 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:53.564 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091389+0000", 2020-01-16T00:12:53.565 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016646459999999998 2020-01-16T00:12:53.565 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.565 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:53.565 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.565 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.565 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.565 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1278 3.d 3:b1521a3e:::smithi12313934-25:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:12:53.565 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:12:53.566 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 10.329868725000001, 2020-01-16T00:12:53.566 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017994915, 2020-01-16T00:12:53.566 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:53.566 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:53.566 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:53.566 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:53.566 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:53.566 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1278 2020-01-16T00:12:53.567 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.567 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:53.567 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.567 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:53.567 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:12:53.567 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:53.567 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.568 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.568 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:53.568 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:12:53.568 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:53.568 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.568 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.568 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:53.568 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074771+0000", 2020-01-16T00:12:53.569 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4569999999999998e-06 2020-01-16T00:12:53.569 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.569 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.569 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:53.569 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074775+0000", 2020-01-16T00:12:53.569 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0330000000000003e-06 2020-01-16T00:12:53.569 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.569 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.570 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:53.570 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074782+0000", 2020-01-16T00:12:53.570 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9470000000000002e-06 2020-01-16T00:12:53.570 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.570 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.570 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:53.570 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074787+0000", 2020-01-16T00:12:53.570 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01666134 2020-01-16T00:12:53.571 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.571 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.571 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:53.571 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091449+0000", 2020-01-16T00:12:53.571 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000247731 2020-01-16T00:12:53.571 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.571 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.571 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:53.572 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091696+0000", 2020-01-16T00:12:53.572 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010703170000000001 2020-01-16T00:12:53.572 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.572 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.572 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:53.572 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.092767+0000", 2020-01-16T00:12:53.572 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017994915 2020-01-16T00:12:53.573 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.573 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:53.573 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.573 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.573 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.573 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1419 3.e 3:7cfd7007:::smithi12313934-21:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e28)", 2020-01-16T00:12:53.573 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:47.689893+0000", 2020-01-16T00:12:53.573 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.7147478630000004, 2020-01-16T00:12:53.574 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0074137259999999998, 2020-01-16T00:12:53.574 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:53.574 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:53.574 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:53.574 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:53.574 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:53.574 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1419 2020-01-16T00:12:53.574 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.575 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:53.575 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.575 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:53.575 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.689893+0000", 2020-01-16T00:12:53.575 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:53.575 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.575 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.576 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:53.576 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.689893+0000", 2020-01-16T00:12:53.576 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:53.576 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.576 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.576 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:53.576 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.689891+0000", 2020-01-16T00:12:53.577 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.8100000000000004e-06 2020-01-16T00:12:53.577 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.577 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.577 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:53.577 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.689900+0000", 2020-01-16T00:12:53.577 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.509e-05 2020-01-16T00:12:53.577 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.577 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.578 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:53.578 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.689915+0000", 2020-01-16T00:12:53.578 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.73e-06 2020-01-16T00:12:53.578 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.578 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.578 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:53.578 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.689925+0000", 2020-01-16T00:12:53.578 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.006916312 2020-01-16T00:12:53.579 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.579 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.579 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:53.579 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.696841+0000", 2020-01-16T00:12:53.579 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011041599999999999 2020-01-16T00:12:53.579 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.579 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.580 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:53.580 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.696951+0000", 2020-01-16T00:12:53.580 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00035496099999999999 2020-01-16T00:12:53.580 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.580 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.580 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:53.580 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.697306+0000", 2020-01-16T00:12:53.580 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0074137259999999998 2020-01-16T00:12:53.581 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.581 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:53.581 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.581 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.581 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.581 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1420 3.e 3:7cfd7007:::smithi12313934-21:head [read 0~3899191,checksum] snapc 0=[] ondisk+read+known_if_redirected e28)", 2020-01-16T00:12:53.581 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:47.689967+0000", 2020-01-16T00:12:53.581 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.7146734400000003, 2020-01-16T00:12:53.582 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010893273, 2020-01-16T00:12:53.582 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:53.582 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:53.582 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:53.582 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:53.582 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:53.582 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1420 2020-01-16T00:12:53.582 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.583 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:53.583 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.583 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:53.583 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.689967+0000", 2020-01-16T00:12:53.583 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:53.583 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.583 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.584 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:53.584 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.689967+0000", 2020-01-16T00:12:53.584 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:53.584 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.584 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.584 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:53.584 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.689965+0000", 2020-01-16T00:12:53.584 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.696e-05 2020-01-16T00:12:53.585 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.585 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.585 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:53.585 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.689982+0000", 2020-01-16T00:12:53.585 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9613e-05 2020-01-16T00:12:53.585 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.585 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.585 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:53.586 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.690002+0000", 2020-01-16T00:12:53.586 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3003000000000001e-05 2020-01-16T00:12:53.586 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.586 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.586 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:53.586 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.690015+0000", 2020-01-16T00:12:53.586 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0073546410000000003 2020-01-16T00:12:53.586 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.587 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.587 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:53.587 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.697370+0000", 2020-01-16T00:12:53.587 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00031806899999999998 2020-01-16T00:12:53.587 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.587 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.587 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:53.588 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.697688+0000", 2020-01-16T00:12:53.588 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003172673 2020-01-16T00:12:53.588 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.588 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.588 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:53.588 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.700860+0000", 2020-01-16T00:12:53.588 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010893273 2020-01-16T00:12:53.588 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.589 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:53.589 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.589 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.589 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.589 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.4.13:40 2.0 2:01b4bead:::smithi12313934-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 640512~403968] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e28)", 2020-01-16T00:12:53.589 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:47.904891+0000", 2020-01-16T00:12:53.589 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.4997497920000002, 2020-01-16T00:12:53.589 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0078047129999999996, 2020-01-16T00:12:53.590 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:53.590 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:53.590 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:53.590 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.590 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:53.590 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.904891+0000", 2020-01-16T00:12:53.590 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:53.591 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.591 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.591 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:53.591 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.904891+0000", 2020-01-16T00:12:53.591 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:53.591 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.591 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.591 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:53.592 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.904889+0000", 2020-01-16T00:12:53.592 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017825 2020-01-16T00:12:53.592 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.592 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.592 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:53.592 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.905067+0000", 2020-01-16T00:12:53.592 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4921e-05 2020-01-16T00:12:53.592 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.593 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.593 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:53.593 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.905082+0000", 2020-01-16T00:12:53.593 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5339999999999999e-06 2020-01-16T00:12:53.593 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.593 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.593 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:53.593 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.905087+0000", 2020-01-16T00:12:53.594 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011594 2020-01-16T00:12:53.594 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.594 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.594 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:53.594 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.905203+0000", 2020-01-16T00:12:53.594 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00026064000000000003 2020-01-16T00:12:53.594 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.594 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.595 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:53.595 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.905464+0000", 2020-01-16T00:12:53.595 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00044418700000000002 2020-01-16T00:12:53.595 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.595 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.595 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:12:53.595 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.905908+0000", 2020-01-16T00:12:53.596 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0055870989999999999 2020-01-16T00:12:53.596 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.596 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.596 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:53.596 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.911495+0000", 2020-01-16T00:12:53.596 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3438e-05 2020-01-16T00:12:53.596 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.596 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.597 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:53.597 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.911539+0000", 2020-01-16T00:12:53.597 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001073112 2020-01-16T00:12:53.597 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.597 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.597 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:53.597 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.912612+0000", 2020-01-16T00:12:53.597 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2938e-05 2020-01-16T00:12:53.598 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.598 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.598 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:53.598 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.912665+0000", 2020-01-16T00:12:53.598 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0811e-05 2020-01-16T00:12:53.598 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.598 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.599 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:53.599 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.912695+0000", 2020-01-16T00:12:53.599 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0078047129999999996 2020-01-16T00:12:53.599 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.599 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:53.599 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.599 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.600 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.600 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.4.13:41 2.0 2:01b4bead:::smithi12313934-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 1445888~278528] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e28)", 2020-01-16T00:12:53.600 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:47.905100+0000", 2020-01-16T00:12:53.600 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.4995405890000004, 2020-01-16T00:12:53.600 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0079266059999999992, 2020-01-16T00:12:53.600 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:53.600 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:53.600 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:53.601 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.601 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:53.601 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.905100+0000", 2020-01-16T00:12:53.601 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:53.601 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.601 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.601 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:53.602 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.905100+0000", 2020-01-16T00:12:53.602 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:53.602 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.602 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.602 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:53.602 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.905099+0000", 2020-01-16T00:12:53.602 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3053000000000005e-05 2020-01-16T00:12:53.603 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.603 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.603 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:53.603 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.905192+0000", 2020-01-16T00:12:53.603 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2110000000000003e-06 2020-01-16T00:12:53.603 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.603 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.603 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:53.604 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.905198+0000", 2020-01-16T00:12:53.604 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4429999999999999e-06 2020-01-16T00:12:53.604 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.604 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.604 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:53.604 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.905202+0000", 2020-01-16T00:12:53.604 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003197858 2020-01-16T00:12:53.605 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.605 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.605 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:53.605 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.908400+0000", 2020-01-16T00:12:53.605 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012500200000000001 2020-01-16T00:12:53.605 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.605 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.605 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:53.606 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.908525+0000", 2020-01-16T00:12:53.606 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000183332 2020-01-16T00:12:53.606 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.606 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.606 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:12:53.606 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.908708+0000", 2020-01-16T00:12:53.606 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0027965339999999998 2020-01-16T00:12:53.606 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.607 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.607 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:53.607 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.911505+0000", 2020-01-16T00:12:53.607 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012356649999999999 2020-01-16T00:12:53.607 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.607 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.607 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:53.608 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.912740+0000", 2020-01-16T00:12:53.608 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022262900000000001 2020-01-16T00:12:53.608 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.608 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.608 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:53.608 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.912963+0000", 2020-01-16T00:12:53.608 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4484999999999999e-05 2020-01-16T00:12:53.608 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.609 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.609 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:53.609 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.912997+0000", 2020-01-16T00:12:53.609 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9139999999999999e-05 2020-01-16T00:12:53.609 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.609 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.609 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:53.609 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.913027+0000", 2020-01-16T00:12:53.610 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0079266059999999992 2020-01-16T00:12:53.610 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.610 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:53.610 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.610 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.610 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.610 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.4.13:42 2.0 2:01b4bead:::smithi12313934-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 1887232~499362] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e28)", 2020-01-16T00:12:53.611 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:47.906201+0000", 2020-01-16T00:12:53.611 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.4984397459999999, 2020-01-16T00:12:53.611 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0073482859999999999, 2020-01-16T00:12:53.611 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:53.611 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:53.611 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:53.611 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.611 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:53.612 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.906201+0000", 2020-01-16T00:12:53.612 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:53.612 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.612 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.612 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:53.612 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.906201+0000", 2020-01-16T00:12:53.612 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:53.612 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.613 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.613 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:53.613 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.906199+0000", 2020-01-16T00:12:53.613 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022299900000000001 2020-01-16T00:12:53.613 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.613 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.613 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:53.614 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.906422+0000", 2020-01-16T00:12:53.614 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0536999999999999e-05 2020-01-16T00:12:53.614 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.614 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.614 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:53.614 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.906432+0000", 2020-01-16T00:12:53.614 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8006000000000002e-05 2020-01-16T00:12:53.614 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.615 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.615 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:53.615 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.906470+0000", 2020-01-16T00:12:53.615 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003113209 2020-01-16T00:12:53.615 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.615 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.615 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:53.615 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.909584+0000", 2020-01-16T00:12:53.616 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000164413 2020-01-16T00:12:53.616 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.616 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.616 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:53.616 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.909748+0000", 2020-01-16T00:12:53.616 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00031923500000000002 2020-01-16T00:12:53.616 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.616 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.617 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:12:53.617 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.910067+0000", 2020-01-16T00:12:53.617 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001845998 2020-01-16T00:12:53.617 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.617 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.617 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:53.617 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.911913+0000", 2020-01-16T00:12:53.618 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001285567 2020-01-16T00:12:53.618 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.618 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.618 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:53.618 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.913199+0000", 2020-01-16T00:12:53.618 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00027355599999999999 2020-01-16T00:12:53.618 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.618 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.619 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:53.619 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.913472+0000", 2020-01-16T00:12:53.619 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9681999999999997e-05 2020-01-16T00:12:53.619 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.619 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.619 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:53.619 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.913512+0000", 2020-01-16T00:12:53.619 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7060000000000001e-05 2020-01-16T00:12:53.620 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:53.620 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:53.620 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:53.620 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.913549+0000", 2020-01-16T00:12:53.620 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0073482859999999999 2020-01-16T00:12:53.620 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.620 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:53.620 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.621 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:53.621 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:53.621 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:53.622 INFO:tasks.rados.rados.0.smithi123.stdout:862: finishing write tid 2 to smithi12313934-25 2020-01-16T00:12:53.623 INFO:tasks.rados.rados.0.smithi123.stdout:862: finishing write tid 3 to smithi12313934-25 2020-01-16T00:12:53.623 INFO:tasks.rados.rados.0.smithi123.stdout:862: finishing write tid 5 to smithi12313934-25 2020-01-16T00:12:53.623 INFO:tasks.rados.rados.0.smithi123.stdout:862: finishing write tid 6 to smithi12313934-25 2020-01-16T00:12:53.623 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 25 v 117 (ObjNum 174 snap 0 seq_num 174) dirty exists 2020-01-16T00:12:53.623 INFO:tasks.rados.rados.0.smithi123.stdout:862: left oid 25 (ObjNum 174 snap 0 seq_num 174) 2020-01-16T00:12:53.623 INFO:tasks.rados.rados.0.smithi123.stdout:863: expect (ObjNum 132 snap 0 seq_num 132) 2020-01-16T00:12:53.623 INFO:tasks.rados.rados.0.smithi123.stdout:864: finishing write tid 1 to smithi12313934-35 2020-01-16T00:12:53.624 INFO:tasks.rados.rados.0.smithi123.stdout:864: finishing write tid 2 to smithi12313934-35 2020-01-16T00:12:53.624 INFO:tasks.rados.rados.0.smithi123.stdout:864: finishing write tid 4 to smithi12313934-35 2020-01-16T00:12:53.624 INFO:tasks.rados.rados.0.smithi123.stdout:864: finishing write tid 5 to smithi12313934-35 2020-01-16T00:12:53.624 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 35 v 83 (ObjNum 175 snap 0 seq_num 175) dirty exists 2020-01-16T00:12:53.624 INFO:tasks.rados.rados.0.smithi123.stdout:864: left oid 35 (ObjNum 175 snap 0 seq_num 175) 2020-01-16T00:12:53.624 INFO:tasks.rados.rados.0.smithi123.stdout:865: finishing write tid 1 to smithi12313934-29 2020-01-16T00:12:53.624 INFO:tasks.rados.rados.0.smithi123.stdout:866: expect (ObjNum 161 snap 0 seq_num 161) 2020-01-16T00:12:53.625 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-01-16T00:12:53.626 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:12:53.623+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:12:53.662 INFO:tasks.rados.rados.0.smithi123.stdout:865: finishing write tid 2 to smithi12313934-29 2020-01-16T00:12:53.662 INFO:tasks.rados.rados.0.smithi123.stdout:868: expect (ObjNum 166 snap 0 seq_num 166) 2020-01-16T00:12:53.686 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-01-16T00:12:53.723 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:12:53.725+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:12:53.782 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:53.783 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:12:53.783 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:12:53.783 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:53.790 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2020-01-16T00:12:53.793 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-01-16T00:12:53.821 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:12:53.824 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:12:53.827+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:12:53.840 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:12:53.842 INFO:tasks.rados.rados.0.smithi123.stdout:867: expect (ObjNum 138 snap 0 seq_num 138) 2020-01-16T00:12:53.861 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:12:53.879 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:12:53.903 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:12:53.922 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5: {} 2020-01-16T00:12:53.925 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:12:53.927+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:12:53.942 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:12:53.948 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:53.959 INFO:tasks.rados.rados.0.smithi123.stdout:865: finishing write tid 4 to smithi12313934-29 2020-01-16T00:12:53.959 INFO:tasks.rados.rados.0.smithi123.stdout:865: finishing write tid 5 to smithi12313934-29 2020-01-16T00:12:53.959 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 29 v 77 (ObjNum 176 snap 0 seq_num 176) dirty exists 2020-01-16T00:12:53.960 INFO:tasks.rados.rados.0.smithi123.stdout:865: left oid 29 (ObjNum 176 snap 0 seq_num 176) 2020-01-16T00:12:53.960 INFO:tasks.rados.rados.0.smithi123.stdout:869: finishing write tid 1 to smithi12313934-50 2020-01-16T00:12:53.960 INFO:tasks.rados.rados.0.smithi123.stdout:870: expect (ObjNum 137 snap 0 seq_num 137) 2020-01-16T00:12:53.961 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-01-16T00:12:53.964 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:12:53.966 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-01-16T00:12:53.966 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-01-16T00:12:53.967 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-01-16T00:12:53.967 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-01-16T00:12:53.967 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-01-16T00:12:53.967 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-01-16T00:12:53.967 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-01-16T00:12:53.968 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-01-16T00:12:54.027 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:54.025+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:54.058 INFO:tasks.rados.rados.0.smithi123.stdout:869: finishing write tid 2 to smithi12313934-50 2020-01-16T00:12:54.059 INFO:tasks.rados.rados.0.smithi123.stdout:869: finishing write tid 4 to smithi12313934-50 2020-01-16T00:12:54.059 INFO:tasks.rados.rados.0.smithi123.stdout:869: finishing write tid 5 to smithi12313934-50 2020-01-16T00:12:54.059 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 50 v 91 (ObjNum 177 snap 0 seq_num 177) dirty exists 2020-01-16T00:12:54.059 INFO:tasks.rados.rados.0.smithi123.stdout:869: left oid 50 (ObjNum 177 snap 0 seq_num 177) 2020-01-16T00:12:54.059 INFO:tasks.rados.rados.0.smithi123.stdout:854: done (15 left) 2020-01-16T00:12:54.060 INFO:tasks.rados.rados.0.smithi123.stdout:855: done (14 left) 2020-01-16T00:12:54.060 INFO:tasks.rados.rados.0.smithi123.stdout:857: done (13 left) 2020-01-16T00:12:54.060 INFO:tasks.rados.rados.0.smithi123.stdout:858: done (12 left) 2020-01-16T00:12:54.060 INFO:tasks.rados.rados.0.smithi123.stdout:859: done (11 left) 2020-01-16T00:12:54.061 INFO:tasks.rados.rados.0.smithi123.stdout:860: done (10 left) 2020-01-16T00:12:54.061 INFO:tasks.rados.rados.0.smithi123.stdout:861: done (9 left) 2020-01-16T00:12:54.061 INFO:tasks.rados.rados.0.smithi123.stdout:862: done (8 left) 2020-01-16T00:12:54.061 INFO:tasks.rados.rados.0.smithi123.stdout:863: done (7 left) 2020-01-16T00:12:54.062 INFO:tasks.rados.rados.0.smithi123.stdout:864: done (6 left) 2020-01-16T00:12:54.062 INFO:tasks.rados.rados.0.smithi123.stdout:865: done (5 left) 2020-01-16T00:12:54.062 INFO:tasks.rados.rados.0.smithi123.stdout:866: done (4 left) 2020-01-16T00:12:54.062 INFO:tasks.rados.rados.0.smithi123.stdout:867: done (3 left) 2020-01-16T00:12:54.062 INFO:tasks.rados.rados.0.smithi123.stdout:868: done (2 left) 2020-01-16T00:12:54.063 INFO:tasks.rados.rados.0.smithi123.stdout:869: done (1 left) 2020-01-16T00:12:54.063 INFO:tasks.rados.rados.0.smithi123.stdout:870: done (0 left) 2020-01-16T00:12:54.063 INFO:tasks.rados.rados.0.smithi123.stdout:871: read oid 35 snap -1 2020-01-16T00:12:54.065 INFO:tasks.rados.rados.0.smithi123.stdout:871: expect (ObjNum 175 snap 0 seq_num 175) 2020-01-16T00:12:54.065 INFO:tasks.rados.rados.0.smithi123.stdout:872: write oid 8 current snap is 0 2020-01-16T00:12:54.065 INFO:tasks.rados.rados.0.smithi123.stdout:872: seq_num 178 ranges {591646=470441,1566383=595717,2888822=102824} 2020-01-16T00:12:54.068 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:54.070 INFO:tasks.rados.rados.0.smithi123.stdout:872: writing smithi12313934-8 from 591646 to 1062087 tid 1 2020-01-16T00:12:54.079 INFO:tasks.rados.rados.0.smithi123.stdout:872: writing smithi12313934-8 from 1566383 to 2162100 tid 2 2020-01-16T00:12:54.079 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-01-16T00:12:54.082 INFO:tasks.rados.rados.0.smithi123.stdout:872: writing smithi12313934-8 from 2888822 to 2991646 tid 3 2020-01-16T00:12:54.083 INFO:tasks.rados.rados.0.smithi123.stdout:873: read oid 37 snap -1 2020-01-16T00:12:54.083 INFO:tasks.rados.rados.0.smithi123.stdout:873: expect (ObjNum 122 snap 0 seq_num 122) 2020-01-16T00:12:54.083 INFO:tasks.rados.rados.0.smithi123.stdout:874: read oid 47 snap -1 2020-01-16T00:12:54.083 INFO:tasks.rados.rados.0.smithi123.stdout:874: expect (ObjNum 133 snap 0 seq_num 133) 2020-01-16T00:12:54.084 INFO:tasks.rados.rados.0.smithi123.stdout:875: read oid 16 snap -1 2020-01-16T00:12:54.084 INFO:tasks.rados.rados.0.smithi123.stdout:875: expect (ObjNum 60 snap 0 seq_num 60) 2020-01-16T00:12:54.084 INFO:tasks.rados.rados.0.smithi123.stdout:876: read oid 50 snap -1 2020-01-16T00:12:54.085 INFO:tasks.rados.rados.0.smithi123.stdout:876: expect (ObjNum 177 snap 0 seq_num 177) 2020-01-16T00:12:54.085 INFO:tasks.rados.rados.0.smithi123.stdout:877: write oid 26 current snap is 0 2020-01-16T00:12:54.085 INFO:tasks.rados.rados.0.smithi123.stdout:877: seq_num 179 ranges {608453=545690,1624865=455728,2632598=375855} 2020-01-16T00:12:54.094 INFO:tasks.rados.rados.0.smithi123.stdout:877: writing smithi12313934-26 from 608453 to 1154143 tid 1 2020-01-16T00:12:54.094 INFO:tasks.rados.rados.0.smithi123.stdout:877: writing smithi12313934-26 from 1624865 to 2080593 tid 2 2020-01-16T00:12:54.100 INFO:tasks.rados.rados.0.smithi123.stdout:877: writing smithi12313934-26 from 2632598 to 3008453 tid 3 2020-01-16T00:12:54.100 INFO:tasks.rados.rados.0.smithi123.stdout:878: read oid 17 snap -1 2020-01-16T00:12:54.100 INFO:tasks.rados.rados.0.smithi123.stdout:878: expect deleted 2020-01-16T00:12:54.100 INFO:tasks.rados.rados.0.smithi123.stdout:879: write oid 19 current snap is 0 2020-01-16T00:12:54.100 INFO:tasks.rados.rados.0.smithi123.stdout:879: seq_num 180 ranges {625260=620939,1683347=599386,2943668=81592} 2020-01-16T00:12:54.107 INFO:tasks.rados.rados.0.smithi123.stdout:879: writing smithi12313934-19 from 625260 to 1246199 tid 1 2020-01-16T00:12:54.114 INFO:tasks.rados.rados.0.smithi123.stdout:879: writing smithi12313934-19 from 1683347 to 2282733 tid 2 2020-01-16T00:12:54.122 INFO:tasks.rados.rados.0.smithi123.stdout:879: writing smithi12313934-19 from 2943668 to 3025260 tid 3 2020-01-16T00:12:54.122 INFO:tasks.rados.rados.0.smithi123.stdout:880: read oid 21 snap -1 2020-01-16T00:12:54.122 INFO:tasks.rados.rados.0.smithi123.stdout:880: expect (ObjNum 113 snap 0 seq_num 113) 2020-01-16T00:12:54.123 INFO:tasks.rados.rados.0.smithi123.stdout:881: write oid 11 current snap is 0 2020-01-16T00:12:54.123 INFO:tasks.rados.rados.0.smithi123.stdout:881: seq_num 181 ranges {642067=696188,2025476=459397,2971091=70976} 2020-01-16T00:12:54.126 INFO:tasks.rados.rados.0.smithi123.stdout:881: writing smithi12313934-11 from 642067 to 1338255 tid 1 2020-01-16T00:12:54.127 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:54.129+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:12:54.133 INFO:tasks.rados.rados.0.smithi123.stdout:881: writing smithi12313934-11 from 2025476 to 2484873 tid 2 2020-01-16T00:12:54.137 INFO:tasks.rados.rados.0.smithi123.stdout:881: writing smithi12313934-11 from 2971091 to 3042067 tid 3 2020-01-16T00:12:54.138 INFO:tasks.rados.rados.0.smithi123.stdout:882: read oid 32 snap -1 2020-01-16T00:12:54.138 INFO:tasks.rados.rados.0.smithi123.stdout:882: expect (ObjNum 143 snap 0 seq_num 143) 2020-01-16T00:12:54.138 INFO:tasks.rados.rados.0.smithi123.stdout:883: read oid 40 snap -1 2020-01-16T00:12:54.138 INFO:tasks.rados.rados.0.smithi123.stdout:883: expect (ObjNum 138 snap 0 seq_num 138) 2020-01-16T00:12:54.138 INFO:tasks.rados.rados.0.smithi123.stdout:884: write (excl) oid 1 current snap is 0 2020-01-16T00:12:54.138 INFO:tasks.rados.rados.0.smithi123.stdout:884: seq_num 182 ranges {658874=771437,2083958=603055} 2020-01-16T00:12:54.147 INFO:tasks.rados.rados.0.smithi123.stdout:884: writing smithi12313934-1 from 658874 to 1430311 tid 1 2020-01-16T00:12:54.169 INFO:tasks.rados.rados.0.smithi123.stdout:884: writing smithi12313934-1 from 2083958 to 2687013 tid 2 2020-01-16T00:12:54.169 INFO:tasks.rados.rados.0.smithi123.stdout:885: write oid 41 current snap is 0 2020-01-16T00:12:54.169 INFO:tasks.rados.rados.0.smithi123.stdout:885: seq_num 183 ranges {675681=563039,1858793=746713,3025937=49744} 2020-01-16T00:12:54.169 INFO:tasks.rados.rados.0.smithi123.stdout:885: writing smithi12313934-41 from 675681 to 1238720 tid 1 2020-01-16T00:12:54.170 INFO:tasks.rados.rados.0.smithi123.stdout:885: writing smithi12313934-41 from 1858793 to 2605506 tid 2 2020-01-16T00:12:54.173 INFO:tasks.rados.rados.0.smithi123.stdout:885: writing smithi12313934-41 from 3025937 to 3075681 tid 3 2020-01-16T00:12:54.173 INFO:tasks.rados.rados.0.smithi123.stdout:886: write (excl) oid 45 current snap is 0 2020-01-16T00:12:54.173 INFO:tasks.rados.rados.0.smithi123.stdout:886: seq_num 184 ranges {692488=638288,1917275=606724} 2020-01-16T00:12:54.181 INFO:tasks.rados.rados.0.smithi123.stdout:886: writing smithi12313934-45 from 692488 to 1330776 tid 1 2020-01-16T00:12:54.188 INFO:tasks.rados.rados.0.smithi123.stdout:886: writing smithi12313934-45 from 1917275 to 2523999 tid 2 2020-01-16T00:12:54.188 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:54.188 INFO:tasks.rados.rados.0.smithi123.stdout:871: expect (ObjNum 175 snap 0 seq_num 175) 2020-01-16T00:12:54.195 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:54.204 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-01-16T00:12:54.256 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:54.251+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:12:54.305 INFO:tasks.rados.rados.0.smithi123.stdout:872: finishing write tid 1 to smithi12313934-8 2020-01-16T00:12:54.306 INFO:tasks.rados.rados.0.smithi123.stdout:871: done (15 left) 2020-01-16T00:12:54.306 INFO:tasks.rados.rados.0.smithi123.stdout:887: delete oid 27 current snap is 0 2020-01-16T00:12:54.306 INFO:tasks.rados.rados.0.smithi123.stdout:872: finishing write tid 2 to smithi12313934-8 2020-01-16T00:12:54.306 INFO:tasks.rados.rados.0.smithi123.stdout:872: finishing write tid 3 to smithi12313934-8 2020-01-16T00:12:54.306 INFO:tasks.rados.rados.0.smithi123.stdout:872: finishing write tid 5 to smithi12313934-8 2020-01-16T00:12:54.306 INFO:tasks.rados.rados.0.smithi123.stdout:872: finishing write tid 6 to smithi12313934-8 2020-01-16T00:12:54.307 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 8 v 58 (ObjNum 178 snap 0 seq_num 178) dirty exists 2020-01-16T00:12:54.307 INFO:tasks.rados.rados.0.smithi123.stdout:872: left oid 8 (ObjNum 178 snap 0 seq_num 178) 2020-01-16T00:12:54.307 INFO:tasks.rados.rados.0.smithi123.stdout:873: expect (ObjNum 122 snap 0 seq_num 122) 2020-01-16T00:12:54.329 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:12:54.329+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:12:54.368 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:54.378 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-01-16T00:12:54.414 INFO:tasks.rados.rados.0.smithi123.stdout:874: expect (ObjNum 133 snap 0 seq_num 133) 2020-01-16T00:12:54.460 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:12:54.463+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:12:54.508 INFO:tasks.rados.rados.0.smithi123.stdout:872: done (15 left) 2020-01-16T00:12:54.509 INFO:tasks.rados.rados.0.smithi123.stdout:873: done (14 left) 2020-01-16T00:12:54.509 INFO:tasks.rados.rados.0.smithi123.stdout:874: done (13 left) 2020-01-16T00:12:54.509 INFO:tasks.rados.rados.0.smithi123.stdout:887: done (12 left) 2020-01-16T00:12:54.509 INFO:tasks.rados.rados.0.smithi123.stdout:888: write (excl) oid 18 current snap is 0 2020-01-16T00:12:54.509 INFO:tasks.rados.rados.0.smithi123.stdout:888: seq_num 186 ranges {726102=788786,1917886=610393,3108206=17896} 2020-01-16T00:12:54.517 INFO:tasks.rados.rados.0.smithi123.stdout:888: writing smithi12313934-18 from 726102 to 1514888 tid 1 2020-01-16T00:12:54.517 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:54.523 INFO:tasks.rados.rados.0.smithi123.stdout:888: writing smithi12313934-18 from 1917886 to 2528279 tid 2 2020-01-16T00:12:54.529 INFO:tasks.rados.rados.0.smithi123.stdout:888: writing smithi12313934-18 from 3108206 to 3126102 tid 3 2020-01-16T00:12:54.530 INFO:tasks.rados.rados.0.smithi123.stdout:889: read oid 28 snap -1 2020-01-16T00:12:54.530 INFO:tasks.rados.rados.0.smithi123.stdout:889: expect (ObjNum 73 snap 0 seq_num 73) 2020-01-16T00:12:54.530 INFO:tasks.rados.rados.0.smithi123.stdout:890: read oid 29 snap -1 2020-01-16T00:12:54.530 INFO:tasks.rados.rados.0.smithi123.stdout:890: expect (ObjNum 176 snap 0 seq_num 176) 2020-01-16T00:12:54.530 INFO:tasks.rados.rados.0.smithi123.stdout:891: write (excl) oid 3 current snap is 0 2020-01-16T00:12:54.530 INFO:tasks.rados.rados.0.smithi123.stdout:891: seq_num 187 ranges {742909=464035,1976368=754051} 2020-01-16T00:12:54.531 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-01-16T00:12:54.533 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:54.533+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:54.534 INFO:tasks.rados.rados.0.smithi123.stdout:891: writing smithi12313934-3 from 742909 to 1206944 tid 1 2020-01-16T00:12:54.544 INFO:tasks.rados.rados.0.smithi123.stdout:891: writing smithi12313934-3 from 1976368 to 2730419 tid 2 2020-01-16T00:12:54.544 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:54.544 INFO:tasks.rados.rados.0.smithi123.stdout:877: finishing write tid 1 to smithi12313934-26 2020-01-16T00:12:54.544 INFO:tasks.rados.rados.0.smithi123.stdout:877: finishing write tid 2 to smithi12313934-26 2020-01-16T00:12:54.545 INFO:tasks.rados.rados.0.smithi123.stdout:876: expect (ObjNum 177 snap 0 seq_num 177) 2020-01-16T00:12:54.631 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:54.635 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:12:54.635+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:12:54.640 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-01-16T00:12:54.734 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:12:54.735+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:12:54.787 INFO:tasks.rados.rados.0.smithi123.stdout:875: expect (ObjNum 60 snap 0 seq_num 60) 2020-01-16T00:12:54.789 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:54.799 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-01-16T00:12:54.836 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:54.837+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:54.867 INFO:teuthology.orchestra.run.smithi123.stderr:noscrub is set 2020-01-16T00:12:54.938 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:12:54.941+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:12:54.940 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:54.940 INFO:tasks.rados.rados.0.smithi123.stdout:877: finishing write tid 3 to smithi12313934-26 2020-01-16T00:12:54.941 INFO:tasks.rados.rados.0.smithi123.stdout:877: finishing write tid 5 to smithi12313934-26 2020-01-16T00:12:54.941 INFO:tasks.rados.rados.0.smithi123.stdout:875: done (15 left) 2020-01-16T00:12:54.941 INFO:tasks.rados.rados.0.smithi123.stdout:876: done (14 left) 2020-01-16T00:12:54.941 INFO:tasks.rados.rados.0.smithi123.stdout:878: done (13 left) 2020-01-16T00:12:54.941 INFO:tasks.rados.rados.0.smithi123.stdout:892: write (excl) oid 4 current snap is 0 2020-01-16T00:12:54.941 INFO:tasks.rados.rados.0.smithi123.stdout:892: seq_num 188 ranges {759716=539284,2034850=614062} 2020-01-16T00:12:54.948 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-01-16T00:12:54.949 INFO:tasks.rados.rados.0.smithi123.stdout:892: writing smithi12313934-4 from 759716 to 1299000 tid 1 2020-01-16T00:12:54.958 INFO:tasks.rados.rados.0.smithi123.stdout:892: writing smithi12313934-4 from 2034850 to 2648912 tid 2 2020-01-16T00:12:54.959 INFO:tasks.rados.rados.0.smithi123.stdout:893: read oid 31 snap -1 2020-01-16T00:12:54.959 INFO:tasks.rados.rados.0.smithi123.stdout:893: expect (ObjNum 132 snap 0 seq_num 132) 2020-01-16T00:12:54.959 INFO:tasks.rados.rados.0.smithi123.stdout:894: read oid 30 snap -1 2020-01-16T00:12:54.959 INFO:tasks.rados.rados.0.smithi123.stdout:894: expect (ObjNum 164 snap 0 seq_num 164) 2020-01-16T00:12:54.959 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:54.959 INFO:tasks.rados.rados.0.smithi123.stdout:877: finishing write tid 6 to smithi12313934-26 2020-01-16T00:12:54.959 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 26 v 158 (ObjNum 179 snap 0 seq_num 179) dirty exists 2020-01-16T00:12:54.960 INFO:tasks.rados.rados.0.smithi123.stdout:877: left oid 26 (ObjNum 179 snap 0 seq_num 179) 2020-01-16T00:12:54.960 INFO:tasks.rados.rados.0.smithi123.stdout:879: finishing write tid 1 to smithi12313934-19 2020-01-16T00:12:54.960 INFO:tasks.rados.rados.0.smithi123.stdout:879: finishing write tid 2 to smithi12313934-19 2020-01-16T00:12:54.960 INFO:tasks.rados.rados.0.smithi123.stdout:879: finishing write tid 3 to smithi12313934-19 2020-01-16T00:12:54.960 INFO:tasks.rados.rados.0.smithi123.stdout:879: finishing write tid 5 to smithi12313934-19 2020-01-16T00:12:54.960 INFO:tasks.rados.rados.0.smithi123.stdout:879: finishing write tid 6 to smithi12313934-19 2020-01-16T00:12:54.960 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 19 v 115 (ObjNum 180 snap 0 seq_num 180) dirty exists 2020-01-16T00:12:54.960 INFO:tasks.rados.rados.0.smithi123.stdout:879: left oid 19 (ObjNum 180 snap 0 seq_num 180) 2020-01-16T00:12:54.961 INFO:tasks.rados.rados.0.smithi123.stdout:880: expect (ObjNum 113 snap 0 seq_num 113) 2020-01-16T00:12:55.038 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:55.037+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:12:55.064 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:55.074 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-01-16T00:12:55.139 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:55.141+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:12:55.160 INFO:tasks.rados.rados.0.smithi123.stdout:881: finishing write tid 1 to smithi12313934-11 2020-01-16T00:12:55.160 INFO:tasks.rados.rados.0.smithi123.stdout:881: finishing write tid 2 to smithi12313934-11 2020-01-16T00:12:55.160 INFO:tasks.rados.rados.0.smithi123.stdout:881: finishing write tid 3 to smithi12313934-11 2020-01-16T00:12:55.160 INFO:tasks.rados.rados.0.smithi123.stdout:881: finishing write tid 5 to smithi12313934-11 2020-01-16T00:12:55.161 INFO:tasks.rados.rados.0.smithi123.stdout:881: finishing write tid 6 to smithi12313934-11 2020-01-16T00:12:55.161 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 11 v 60 (ObjNum 181 snap 0 seq_num 181) dirty exists 2020-01-16T00:12:55.161 INFO:tasks.rados.rados.0.smithi123.stdout:881: left oid 11 (ObjNum 181 snap 0 seq_num 181) 2020-01-16T00:12:55.161 INFO:tasks.rados.rados.0.smithi123.stdout:877: done (15 left) 2020-01-16T00:12:55.161 INFO:tasks.rados.rados.0.smithi123.stdout:879: done (14 left) 2020-01-16T00:12:55.161 INFO:tasks.rados.rados.0.smithi123.stdout:880: done (13 left) 2020-01-16T00:12:55.161 INFO:tasks.rados.rados.0.smithi123.stdout:881: done (12 left) 2020-01-16T00:12:55.161 INFO:tasks.rados.rados.0.smithi123.stdout:895: write oid 48 current snap is 0 2020-01-16T00:12:55.162 INFO:tasks.rados.rados.0.smithi123.stdout:895: seq_num 189 ranges {776523=614533,1976979=757720} 2020-01-16T00:12:55.167 INFO:tasks.rados.rados.0.smithi123.stdout:895: writing smithi12313934-48 from 776523 to 1391056 tid 1 2020-01-16T00:12:55.176 INFO:tasks.rados.rados.0.smithi123.stdout:895: writing smithi12313934-48 from 1976979 to 2734699 tid 2 2020-01-16T00:12:55.176 INFO:tasks.rados.rados.0.smithi123.stdout:896: read oid 2 snap -1 2020-01-16T00:12:55.176 INFO:tasks.rados.rados.0.smithi123.stdout:896: expect (ObjNum 170 snap 0 seq_num 170) 2020-01-16T00:12:55.176 INFO:tasks.rados.rados.0.smithi123.stdout:897: write oid 17 current snap is 0 2020-01-16T00:12:55.176 INFO:tasks.rados.rados.0.smithi123.stdout:897: seq_num 190 ranges {793330=689782,2035461=617731,3101545=91785} 2020-01-16T00:12:55.185 INFO:tasks.rados.rados.0.smithi123.stdout:897: writing smithi12313934-17 from 793330 to 1483112 tid 1 2020-01-16T00:12:55.193 INFO:tasks.rados.rados.0.smithi123.stdout:897: writing smithi12313934-17 from 2035461 to 2653192 tid 2 2020-01-16T00:12:55.196 INFO:tasks.rados.rados.0.smithi123.stdout:897: writing smithi12313934-17 from 3101545 to 3193330 tid 3 2020-01-16T00:12:55.196 INFO:tasks.rados.rados.0.smithi123.stdout:898: read oid 15 snap -1 2020-01-16T00:12:55.196 INFO:tasks.rados.rados.0.smithi123.stdout:898: expect (ObjNum 152 snap 0 seq_num 152) 2020-01-16T00:12:55.197 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:55.197 INFO:tasks.rados.rados.0.smithi123.stdout:883: expect (ObjNum 138 snap 0 seq_num 138) 2020-01-16T00:12:55.223 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:55.233 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-01-16T00:12:55.277 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:55.278+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:12:55.332 INFO:tasks.rados.rados.0.smithi123.stdout:884: finishing write tid 1 to smithi12313934-1 2020-01-16T00:12:55.333 INFO:tasks.rados.rados.0.smithi123.stdout:882: expect (ObjNum 143 snap 0 seq_num 143) 2020-01-16T00:12:55.340 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:12:55.338+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:12:55.382 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:55.391 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-01-16T00:12:55.441 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:55.441+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:55.442 INFO:tasks.rados.rados.0.smithi123.stdout:885: finishing write tid 1 to smithi12313934-41 2020-01-16T00:12:55.442 INFO:tasks.rados.rados.0.smithi123.stdout:882: done (15 left) 2020-01-16T00:12:55.443 INFO:tasks.rados.rados.0.smithi123.stdout:883: done (14 left) 2020-01-16T00:12:55.443 INFO:tasks.rados.rados.0.smithi123.stdout:899: write (excl) oid 11 current snap is 0 2020-01-16T00:12:55.443 INFO:tasks.rados.rados.0.smithi123.stdout:899: seq_num 191 ranges {410137=481384,1410296=761389,2728968=481169} 2020-01-16T00:12:55.454 INFO:tasks.rados.rados.0.smithi123.stdout:899: writing smithi12313934-11 from 410137 to 891521 tid 1 2020-01-16T00:12:55.461 INFO:tasks.rados.rados.0.smithi123.stdout:899: writing smithi12313934-11 from 1410296 to 2171685 tid 2 2020-01-16T00:12:55.469 INFO:tasks.rados.rados.0.smithi123.stdout:899: writing smithi12313934-11 from 2728968 to 3210137 tid 3 2020-01-16T00:12:55.469 INFO:tasks.rados.rados.0.smithi123.stdout:900: read oid 40 snap -1 2020-01-16T00:12:55.469 INFO:tasks.rados.rados.0.smithi123.stdout:900: expect (ObjNum 138 snap 0 seq_num 138) 2020-01-16T00:12:55.470 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:55.470 INFO:tasks.rados.rados.0.smithi123.stdout:884: finishing write tid 2 to smithi12313934-1 2020-01-16T00:12:55.470 INFO:tasks.rados.rados.0.smithi123.stdout:884: finishing write tid 4 to smithi12313934-1 2020-01-16T00:12:55.470 INFO:tasks.rados.rados.0.smithi123.stdout:884: finishing write tid 5 to smithi12313934-1 2020-01-16T00:12:55.470 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 1 v 162 (ObjNum 182 snap 0 seq_num 182) dirty exists 2020-01-16T00:12:55.470 INFO:tasks.rados.rados.0.smithi123.stdout:884: left oid 1 (ObjNum 182 snap 0 seq_num 182) 2020-01-16T00:12:55.470 INFO:tasks.rados.rados.0.smithi123.stdout:885: finishing write tid 2 to smithi12313934-41 2020-01-16T00:12:55.471 INFO:tasks.rados.rados.0.smithi123.stdout:885: finishing write tid 3 to smithi12313934-41 2020-01-16T00:12:55.471 INFO:tasks.rados.rados.0.smithi123.stdout:885: finishing write tid 5 to smithi12313934-41 2020-01-16T00:12:55.471 INFO:tasks.rados.rados.0.smithi123.stdout:885: finishing write tid 6 to smithi12313934-41 2020-01-16T00:12:55.471 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 41 v 119 (ObjNum 183 snap 0 seq_num 183) dirty exists 2020-01-16T00:12:55.471 INFO:tasks.rados.rados.0.smithi123.stdout:885: left oid 41 (ObjNum 183 snap 0 seq_num 183) 2020-01-16T00:12:55.471 INFO:tasks.rados.rados.0.smithi123.stdout:886: finishing write tid 1 to smithi12313934-45 2020-01-16T00:12:55.471 INFO:tasks.rados.rados.0.smithi123.stdout:886: finishing write tid 2 to smithi12313934-45 2020-01-16T00:12:55.471 INFO:tasks.rados.rados.0.smithi123.stdout:886: finishing write tid 4 to smithi12313934-45 2020-01-16T00:12:55.472 INFO:tasks.rados.rados.0.smithi123.stdout:886: finishing write tid 5 to smithi12313934-45 2020-01-16T00:12:55.472 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 45 v 81 (ObjNum 184 snap 0 seq_num 184) dirty exists 2020-01-16T00:12:55.472 INFO:tasks.rados.rados.0.smithi123.stdout:886: left oid 45 (ObjNum 184 snap 0 seq_num 184) 2020-01-16T00:12:55.472 INFO:tasks.rados.rados.0.smithi123.stdout:884: done (15 left) 2020-01-16T00:12:55.472 INFO:tasks.rados.rados.0.smithi123.stdout:885: done (14 left) 2020-01-16T00:12:55.472 INFO:tasks.rados.rados.0.smithi123.stdout:886: done (13 left) 2020-01-16T00:12:55.472 INFO:tasks.rados.rados.0.smithi123.stdout:901: read oid 25 snap -1 2020-01-16T00:12:55.472 INFO:tasks.rados.rados.0.smithi123.stdout:901: expect (ObjNum 174 snap 0 seq_num 174) 2020-01-16T00:12:55.473 INFO:tasks.rados.rados.0.smithi123.stdout:902: read oid 6 snap -1 2020-01-16T00:12:55.473 INFO:tasks.rados.rados.0.smithi123.stdout:902: expect (ObjNum 137 snap 0 seq_num 137) 2020-01-16T00:12:55.473 INFO:tasks.rados.rados.0.smithi123.stdout:903: read oid 37 snap -1 2020-01-16T00:12:55.473 INFO:tasks.rados.rados.0.smithi123.stdout:903: expect (ObjNum 122 snap 0 seq_num 122) 2020-01-16T00:12:55.473 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:55.473 INFO:tasks.rados.rados.0.smithi123.stdout:888: finishing write tid 1 to smithi12313934-18 2020-01-16T00:12:55.473 INFO:tasks.rados.rados.0.smithi123.stdout:890: expect (ObjNum 176 snap 0 seq_num 176) 2020-01-16T00:12:55.540 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:55.543 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:55.545+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:55.550 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-01-16T00:12:55.644 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:12:55.645+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:12:55.658 INFO:tasks.rados.rados.0.smithi123.stdout:891: finishing write tid 1 to smithi12313934-3 2020-01-16T00:12:55.659 INFO:tasks.rados.rados.0.smithi123.stdout:888: finishing write tid 2 to smithi12313934-18 2020-01-16T00:12:55.659 INFO:tasks.rados.rados.0.smithi123.stdout:888: finishing write tid 3 to smithi12313934-18 2020-01-16T00:12:55.659 INFO:tasks.rados.rados.0.smithi123.stdout:888: finishing write tid 5 to smithi12313934-18 2020-01-16T00:12:55.659 INFO:tasks.rados.rados.0.smithi123.stdout:888: finishing write tid 6 to smithi12313934-18 2020-01-16T00:12:55.659 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 18 v 167 (ObjNum 186 snap 0 seq_num 186) dirty exists 2020-01-16T00:12:55.659 INFO:tasks.rados.rados.0.smithi123.stdout:888: left oid 18 (ObjNum 186 snap 0 seq_num 186) 2020-01-16T00:12:55.659 INFO:tasks.rados.rados.0.smithi123.stdout:889: expect (ObjNum 73 snap 0 seq_num 73) 2020-01-16T00:12:55.700 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:55.710 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-01-16T00:12:55.745 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:55.745+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:55.798 INFO:tasks.rados.rados.0.smithi123.stdout:891: finishing write tid 2 to smithi12313934-3 2020-01-16T00:12:55.798 INFO:tasks.rados.rados.0.smithi123.stdout:891: finishing write tid 4 to smithi12313934-3 2020-01-16T00:12:55.798 INFO:tasks.rados.rados.0.smithi123.stdout:891: finishing write tid 5 to smithi12313934-3 2020-01-16T00:12:55.798 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 3 v 61 (ObjNum 187 snap 0 seq_num 187) dirty exists 2020-01-16T00:12:55.799 INFO:tasks.rados.rados.0.smithi123.stdout:891: left oid 3 (ObjNum 187 snap 0 seq_num 187) 2020-01-16T00:12:55.799 INFO:tasks.rados.rados.0.smithi123.stdout:892: finishing write tid 1 to smithi12313934-4 2020-01-16T00:12:55.799 INFO:tasks.rados.rados.0.smithi123.stdout:892: finishing write tid 2 to smithi12313934-4 2020-01-16T00:12:55.799 INFO:tasks.rados.rados.0.smithi123.stdout:888: done (15 left) 2020-01-16T00:12:55.799 INFO:tasks.rados.rados.0.smithi123.stdout:889: done (14 left) 2020-01-16T00:12:55.799 INFO:tasks.rados.rados.0.smithi123.stdout:890: done (13 left) 2020-01-16T00:12:55.799 INFO:tasks.rados.rados.0.smithi123.stdout:891: done (12 left) 2020-01-16T00:12:55.799 INFO:tasks.rados.rados.0.smithi123.stdout:904: delete oid 3 current snap is 0 2020-01-16T00:12:55.800 INFO:tasks.rados.rados.0.smithi123.stdout:892: finishing write tid 4 to smithi12313934-4 2020-01-16T00:12:55.800 INFO:tasks.rados.rados.0.smithi123.stdout:892: finishing write tid 5 to smithi12313934-4 2020-01-16T00:12:55.800 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 4 v 86 (ObjNum 188 snap 0 seq_num 188) dirty exists 2020-01-16T00:12:55.800 INFO:tasks.rados.rados.0.smithi123.stdout:892: left oid 4 (ObjNum 188 snap 0 seq_num 188) 2020-01-16T00:12:55.800 INFO:tasks.rados.rados.0.smithi123.stdout:894: expect (ObjNum 164 snap 0 seq_num 164) 2020-01-16T00:12:55.822 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:55.831 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-01-16T00:12:55.875 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:12:55.877+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:12:55.946 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:55.948 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:55.946+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:12:55.955 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-01-16T00:12:55.971 INFO:tasks.rados.rados.0.smithi123.stdout:893: expect (ObjNum 132 snap 0 seq_num 132) 2020-01-16T00:12:55.983 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-01-16T00:12:56.054 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:12:56.054+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:12:56.061 INFO:tasks.rados.rados.0.smithi123.stdout:895: finishing write tid 1 to smithi12313934-48 2020-01-16T00:12:56.061 INFO:tasks.rados.rados.0.smithi123.stdout:895: finishing write tid 2 to smithi12313934-48 2020-01-16T00:12:56.062 INFO:tasks.rados.rados.0.smithi123.stdout:896: expect (ObjNum 170 snap 0 seq_num 170) 2020-01-16T00:12:56.111 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:56.121 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-01-16T00:12:56.123 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:12:56.143 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:12:56.170 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:12:56.170 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:56.169+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:56.188 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:12:56.214 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:12:56.230 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:56.235 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5: {} 2020-01-16T00:12:56.240 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-01-16T00:12:56.260 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:12:56.268 INFO:tasks.rados.rados.0.smithi123.stdout:897: finishing write tid 1 to smithi12313934-17 2020-01-16T00:12:56.268 INFO:tasks.rados.rados.0.smithi123.stdout:895: finishing write tid 4 to smithi12313934-48 2020-01-16T00:12:56.269 INFO:tasks.rados.rados.0.smithi123.stdout:895: finishing write tid 5 to smithi12313934-48 2020-01-16T00:12:56.269 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 48 v 81 (ObjNum 189 snap 0 seq_num 189) dirty exists 2020-01-16T00:12:56.269 INFO:tasks.rados.rados.0.smithi123.stdout:895: left oid 48 (ObjNum 189 snap 0 seq_num 189) 2020-01-16T00:12:56.269 INFO:tasks.rados.rados.0.smithi123.stdout:897: finishing write tid 2 to smithi12313934-17 2020-01-16T00:12:56.269 INFO:tasks.rados.rados.0.smithi123.stdout:897: finishing write tid 3 to smithi12313934-17 2020-01-16T00:12:56.269 INFO:tasks.rados.rados.0.smithi123.stdout:897: finishing write tid 5 to smithi12313934-17 2020-01-16T00:12:56.269 INFO:tasks.rados.rados.0.smithi123.stdout:897: finishing write tid 6 to smithi12313934-17 2020-01-16T00:12:56.270 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 17 v 90 (ObjNum 190 snap 0 seq_num 190) dirty exists 2020-01-16T00:12:56.270 INFO:tasks.rados.rados.0.smithi123.stdout:897: left oid 17 (ObjNum 190 snap 0 seq_num 190) 2020-01-16T00:12:56.270 INFO:tasks.rados.rados.0.smithi123.stdout:892: done (12 left) 2020-01-16T00:12:56.270 INFO:tasks.rados.rados.0.smithi123.stdout:893: done (11 left) 2020-01-16T00:12:56.270 INFO:tasks.rados.rados.0.smithi123.stdout:894: done (10 left) 2020-01-16T00:12:56.270 INFO:tasks.rados.rados.0.smithi123.stdout:895: done (9 left) 2020-01-16T00:12:56.270 INFO:tasks.rados.rados.0.smithi123.stdout:896: done (8 left) 2020-01-16T00:12:56.271 INFO:tasks.rados.rados.0.smithi123.stdout:897: done (7 left) 2020-01-16T00:12:56.271 INFO:tasks.rados.rados.0.smithi123.stdout:904: done (6 left) 2020-01-16T00:12:56.271 INFO:tasks.rados.rados.0.smithi123.stdout:905: write oid 41 current snap is 0 2020-01-16T00:12:56.271 INFO:tasks.rados.rados.0.smithi123.stdout:905: seq_num 193 ranges {443751=631882,1810907=765058,3067461=176290} 2020-01-16T00:12:56.274 INFO:tasks.rados.rados.0.smithi123.stdout:905: writing smithi12313934-41 from 443751 to 1075633 tid 1 2020-01-16T00:12:56.282 INFO:tasks.rados.rados.0.smithi123.stdout:905: writing smithi12313934-41 from 1810907 to 2575965 tid 2 2020-01-16T00:12:56.284 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:12:56.286 INFO:tasks.rados.rados.0.smithi123.stdout:905: writing smithi12313934-41 from 3067461 to 3243751 tid 3 2020-01-16T00:12:56.287 INFO:tasks.rados.rados.0.smithi123.stdout:906: read oid 29 snap -1 2020-01-16T00:12:56.287 INFO:tasks.rados.rados.0.smithi123.stdout:906: expect (ObjNum 176 snap 0 seq_num 176) 2020-01-16T00:12:56.287 INFO:tasks.rados.rados.0.smithi123.stdout:907: write (excl) oid 39 current snap is 0 2020-01-16T00:12:56.287 INFO:tasks.rados.rados.0.smithi123.stdout:907: seq_num 194 ranges {460558=707131,1869389=508716,3094884=165674} 2020-01-16T00:12:56.287 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-01-16T00:12:56.287 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-01-16T00:12:56.288 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-01-16T00:12:56.288 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-01-16T00:12:56.288 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-01-16T00:12:56.288 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-01-16T00:12:56.288 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-01-16T00:12:56.288 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-01-16T00:12:56.289 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:56.289+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:56.293 INFO:tasks.rados.rados.0.smithi123.stdout:907: writing smithi12313934-39 from 460558 to 1167689 tid 1 2020-01-16T00:12:56.300 INFO:tasks.rados.rados.0.smithi123.stdout:907: writing smithi12313934-39 from 1869389 to 2378105 tid 2 2020-01-16T00:12:56.305 INFO:tasks.rados.rados.0.smithi123.stdout:907: writing smithi12313934-39 from 3094884 to 3260558 tid 3 2020-01-16T00:12:56.305 INFO:tasks.rados.rados.0.smithi123.stdout:908: write (excl) oid 8 current snap is 0 2020-01-16T00:12:56.305 INFO:tasks.rados.rados.0.smithi123.stdout:908: seq_num 195 ranges {477365=782380,1927871=768727,3122307=155058} 2020-01-16T00:12:56.312 INFO:tasks.rados.rados.0.smithi123.stdout:908: writing smithi12313934-8 from 477365 to 1259745 tid 1 2020-01-16T00:12:56.326 INFO:tasks.rados.rados.0.smithi123.stdout:908: writing smithi12313934-8 from 1927871 to 2696598 tid 2 2020-01-16T00:12:56.326 INFO:tasks.rados.rados.0.smithi123.stdout:908: writing smithi12313934-8 from 3122307 to 3277365 tid 3 2020-01-16T00:12:56.327 INFO:tasks.rados.rados.0.smithi123.stdout:909: read oid 32 snap -1 2020-01-16T00:12:56.327 INFO:tasks.rados.rados.0.smithi123.stdout:909: expect (ObjNum 143 snap 0 seq_num 143) 2020-01-16T00:12:56.327 INFO:tasks.rados.rados.0.smithi123.stdout:910: write (excl) oid 49 current snap is 0 2020-01-16T00:12:56.327 INFO:tasks.rados.rados.0.smithi123.stdout:910: seq_num 196 ranges {494172=457629,1586353=512385,2749730=544442} 2020-01-16T00:12:56.334 INFO:tasks.rados.rados.0.smithi123.stdout:910: writing smithi12313934-49 from 494172 to 951801 tid 1 2020-01-16T00:12:56.335 INFO:tasks.rados.rados.0.smithi123.stdout:910: writing smithi12313934-49 from 1586353 to 2098738 tid 2 2020-01-16T00:12:56.342 INFO:tasks.rados.rados.0.smithi123.stdout:910: writing smithi12313934-49 from 2749730 to 3294172 tid 3 2020-01-16T00:12:56.342 INFO:tasks.rados.rados.0.smithi123.stdout:911: write (excl) oid 18 current snap is 0 2020-01-16T00:12:56.342 INFO:tasks.rados.rados.0.smithi123.stdout:911: seq_num 197 ranges {510979=532878,1528482=772396,3060800=250179} 2020-01-16T00:12:56.348 INFO:tasks.rados.rados.0.smithi123.stdout:911: writing smithi12313934-18 from 510979 to 1043857 tid 1 2020-01-16T00:12:56.355 INFO:tasks.rados.rados.0.smithi123.stdout:911: writing smithi12313934-18 from 1528482 to 2300878 tid 2 2020-01-16T00:12:56.355 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:56.357+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:12:56.361 INFO:tasks.rados.rados.0.smithi123.stdout:911: writing smithi12313934-18 from 3060800 to 3310979 tid 3 2020-01-16T00:12:56.361 INFO:tasks.rados.rados.0.smithi123.stdout:912: read oid 21 snap -1 2020-01-16T00:12:56.361 INFO:tasks.rados.rados.0.smithi123.stdout:912: expect (ObjNum 113 snap 0 seq_num 113) 2020-01-16T00:12:56.361 INFO:tasks.rados.rados.0.smithi123.stdout:913: write oid 5 current snap is 0 2020-01-16T00:12:56.361 INFO:tasks.rados.rados.0.smithi123.stdout:913: seq_num 198 ranges {527786=724480,1703317=516054,2804576=435035} 2020-01-16T00:12:56.369 INFO:tasks.rados.rados.0.smithi123.stdout:913: writing smithi12313934-5 from 527786 to 1252266 tid 1 2020-01-16T00:12:56.375 INFO:tasks.rados.rados.0.smithi123.stdout:913: writing smithi12313934-5 from 1703317 to 2219371 tid 2 2020-01-16T00:12:56.381 INFO:tasks.rados.rados.0.smithi123.stdout:913: writing smithi12313934-5 from 2804576 to 3239611 tid 3 2020-01-16T00:12:56.381 INFO:tasks.rados.rados.0.smithi123.stdout:914: write oid 20 current snap is 0 2020-01-16T00:12:56.381 INFO:tasks.rados.rados.0.smithi123.stdout:914: seq_num 199 ranges {544593=799729,1761799=776065,2948352=396241} 2020-01-16T00:12:56.388 INFO:tasks.rados.rados.0.smithi123.stdout:914: writing smithi12313934-20 from 544593 to 1344322 tid 1 2020-01-16T00:12:56.395 INFO:tasks.rados.rados.0.smithi123.stdout:914: writing smithi12313934-20 from 1761799 to 2537864 tid 2 2020-01-16T00:12:56.400 INFO:tasks.rados.rados.0.smithi123.stdout:914: writing smithi12313934-20 from 2948352 to 3344593 tid 3 2020-01-16T00:12:56.400 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:56.400 INFO:tasks.rados.rados.0.smithi123.stdout:898: expect (ObjNum 152 snap 0 seq_num 152) 2020-01-16T00:12:56.407 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:56.407 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:12:56.407 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:12:56.407 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:56.416 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-01-16T00:12:56.459 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:56.461+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:12:56.534 INFO:tasks.rados.rados.0.smithi123.stdout:899: finishing write tid 1 to smithi12313934-11 2020-01-16T00:12:56.534 INFO:tasks.rados.rados.0.smithi123.stdout:899: finishing write tid 2 to smithi12313934-11 2020-01-16T00:12:56.535 INFO:tasks.rados.rados.0.smithi123.stdout:903: expect (ObjNum 122 snap 0 seq_num 122) 2020-01-16T00:12:56.557 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:12:56.558+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:12:56.567 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:56.568 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:12:56.568 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:12:56.568 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:12:56.568 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:56.577 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-01-16T00:12:56.632 INFO:tasks.rados.rados.0.smithi123.stdout:902: expect (ObjNum 137 snap 0 seq_num 137) 2020-01-16T00:12:56.658 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:12:56.658+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:12:56.684 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:56.684 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:12:56.685 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:12:56.685 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:12:56.685 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.685 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:895 3.3 3:cd231094:::smithi12313934-49:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:56.685 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.122390+0000", 2020-01-16T00:12:56.685 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 19.560743955, 2020-01-16T00:12:56.685 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014295501, 2020-01-16T00:12:56.686 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:56.686 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:56.686 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:56.686 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:56.686 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:56.686 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 895 2020-01-16T00:12:56.686 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.687 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:56.687 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.687 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:56.687 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122390+0000", 2020-01-16T00:12:56.687 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:56.687 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.687 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.687 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:56.688 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122390+0000", 2020-01-16T00:12:56.688 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999957 2020-01-16T00:12:56.688 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.688 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.688 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:56.688 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122386+0000", 2020-01-16T00:12:56.688 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0656999999999999e-05 2020-01-16T00:12:56.688 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.689 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.689 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:56.689 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122407+0000", 2020-01-16T00:12:56.689 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9102999999999999e-05 2020-01-16T00:12:56.689 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.689 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.689 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:56.689 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122436+0000", 2020-01-16T00:12:56.690 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.774e-05 2020-01-16T00:12:56.690 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.690 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.690 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:56.690 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122483+0000", 2020-01-16T00:12:56.690 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5185000000000004e-05 2020-01-16T00:12:56.690 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.691 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.691 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.691 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122579+0000", 2020-01-16T00:12:56.691 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000304085 2020-01-16T00:12:56.691 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.691 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.692 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:56.692 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122883+0000", 2020-01-16T00:12:56.692 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0092077970000000002 2020-01-16T00:12:56.692 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.692 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.692 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.692 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132091+0000", 2020-01-16T00:12:56.693 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000170025 2020-01-16T00:12:56.693 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.693 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.693 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:56.693 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132261+0000", 2020-01-16T00:12:56.693 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0040130979999999997 2020-01-16T00:12:56.693 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.693 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.694 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.694 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136274+0000", 2020-01-16T00:12:56.694 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014227900000000001 2020-01-16T00:12:56.694 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.694 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.694 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:56.694 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136416+0000", 2020-01-16T00:12:56.695 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00026995200000000001 2020-01-16T00:12:56.695 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.695 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.695 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:56.695 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136686+0000", 2020-01-16T00:12:56.695 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014295501 2020-01-16T00:12:56.695 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.695 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:56.696 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.696 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.696 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.696 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:896 3.3 3:cd231094:::smithi12313934-49:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:56.696 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.122539+0000", 2020-01-16T00:12:56.696 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 19.560594954999999, 2020-01-16T00:12:56.696 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014440173000000001, 2020-01-16T00:12:56.696 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:56.697 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:56.697 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:56.697 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:56.697 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:56.697 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 896 2020-01-16T00:12:56.697 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.697 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:56.698 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.698 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:56.698 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122539+0000", 2020-01-16T00:12:56.698 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:56.698 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.698 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.698 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:56.698 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122539+0000", 2020-01-16T00:12:56.699 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:56.699 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.699 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.699 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:56.699 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122537+0000", 2020-01-16T00:12:56.699 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2432999999999999e-05 2020-01-16T00:12:56.699 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.699 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.700 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:56.700 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122550+0000", 2020-01-16T00:12:56.700 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9369999999999999e-05 2020-01-16T00:12:56.700 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.700 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.700 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:56.700 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122569+0000", 2020-01-16T00:12:56.701 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1747e-05 2020-01-16T00:12:56.701 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.701 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.701 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:56.701 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122581+0000", 2020-01-16T00:12:56.701 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00041555800000000001 2020-01-16T00:12:56.702 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.702 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.702 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.702 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122996+0000", 2020-01-16T00:12:56.702 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016840299999999999 2020-01-16T00:12:56.702 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.703 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.703 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:56.703 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.123165+0000", 2020-01-16T00:12:56.703 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0091641359999999998 2020-01-16T00:12:56.703 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.703 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.703 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.704 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132329+0000", 2020-01-16T00:12:56.704 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013433400000000001 2020-01-16T00:12:56.704 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.704 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.704 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:56.704 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132463+0000", 2020-01-16T00:12:56.704 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0042524110000000002 2020-01-16T00:12:56.704 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.705 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.705 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.705 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136716+0000", 2020-01-16T00:12:56.705 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4591999999999993e-05 2020-01-16T00:12:56.705 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.705 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.705 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:56.705 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136790+0000", 2020-01-16T00:12:56.706 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018927500000000001 2020-01-16T00:12:56.706 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.706 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.706 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:56.706 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136980+0000", 2020-01-16T00:12:56.706 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014440173000000001 2020-01-16T00:12:56.706 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.706 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:56.707 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.707 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.707 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.707 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:897 3.3 3:cd231094:::smithi12313934-49:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:56.707 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.122613+0000", 2020-01-16T00:12:56.707 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 19.560521811000001, 2020-01-16T00:12:56.707 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014606846, 2020-01-16T00:12:56.708 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:56.708 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:56.708 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:56.708 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:56.708 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:56.708 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 897 2020-01-16T00:12:56.708 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.708 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:56.709 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.709 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:56.709 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122613+0000", 2020-01-16T00:12:56.709 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:56.709 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.709 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.709 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:56.709 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122613+0000", 2020-01-16T00:12:56.710 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:56.710 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.710 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.710 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:56.710 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122611+0000", 2020-01-16T00:12:56.710 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3670000000000003e-06 2020-01-16T00:12:56.710 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.711 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.711 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:56.711 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122620+0000", 2020-01-16T00:12:56.711 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5193e-05 2020-01-16T00:12:56.711 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.711 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.711 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:56.712 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122635+0000", 2020-01-16T00:12:56.712 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0696999999999999e-05 2020-01-16T00:12:56.712 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.712 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.712 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:56.712 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122646+0000", 2020-01-16T00:12:56.713 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00059961400000000005 2020-01-16T00:12:56.713 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.713 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.713 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.713 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.123246+0000", 2020-01-16T00:12:56.713 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011189200000000001 2020-01-16T00:12:56.713 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.714 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.714 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:56.714 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.123358+0000", 2020-01-16T00:12:56.714 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0091363550000000005 2020-01-16T00:12:56.714 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.714 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.714 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.714 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132494+0000", 2020-01-16T00:12:56.715 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2956999999999998e-05 2020-01-16T00:12:56.715 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.715 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.715 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:56.715 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132587+0000", 2020-01-16T00:12:56.715 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0044061769999999998 2020-01-16T00:12:56.715 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.715 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.716 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.716 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136993+0000", 2020-01-16T00:12:56.716 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7822999999999998e-05 2020-01-16T00:12:56.716 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.716 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.716 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:56.716 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.137051+0000", 2020-01-16T00:12:56.717 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000168498 2020-01-16T00:12:56.717 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.717 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.717 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:56.717 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.137219+0000", 2020-01-16T00:12:56.717 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014606846 2020-01-16T00:12:56.717 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.717 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:56.718 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.718 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.718 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.718 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:943 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:56.718 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:12:56.718 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 18.768655837000001, 2020-01-16T00:12:56.718 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018847004000000001, 2020-01-16T00:12:56.718 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:56.719 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:56.719 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:56.719 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:56.719 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:56.719 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 943 2020-01-16T00:12:56.719 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.719 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:56.720 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.720 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:56.720 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:12:56.720 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:56.720 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.720 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.720 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:56.720 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:12:56.721 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:12:56.721 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.721 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.721 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:56.721 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914475+0000", 2020-01-16T00:12:56.721 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.508e-05 2020-01-16T00:12:56.721 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.721 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.722 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:56.722 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914490+0000", 2020-01-16T00:12:56.722 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.857e-05 2020-01-16T00:12:56.722 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.722 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.723 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:56.723 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914509+0000", 2020-01-16T00:12:56.723 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.46e-06 2020-01-16T00:12:56.723 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.723 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.723 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:56.723 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914515+0000", 2020-01-16T00:12:56.724 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2612999999999999e-05 2020-01-16T00:12:56.724 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.724 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.724 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.724 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914568+0000", 2020-01-16T00:12:56.724 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028312299999999998 2020-01-16T00:12:56.724 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.724 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.725 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:56.725 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914851+0000", 2020-01-16T00:12:56.725 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012282533 2020-01-16T00:12:56.725 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.725 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.725 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.725 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927133+0000", 2020-01-16T00:12:56.725 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4839999999999995e-05 2020-01-16T00:12:56.726 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.726 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.726 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:56.726 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927208+0000", 2020-01-16T00:12:56.726 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0047425610000000002 2020-01-16T00:12:56.726 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.726 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.727 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.727 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.931951+0000", 2020-01-16T00:12:56.727 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3628999999999994e-05 2020-01-16T00:12:56.727 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.727 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.727 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:56.727 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.932045+0000", 2020-01-16T00:12:56.727 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012810180000000001 2020-01-16T00:12:56.728 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.728 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.728 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:56.728 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933326+0000", 2020-01-16T00:12:56.728 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018847004000000001 2020-01-16T00:12:56.728 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.728 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:56.728 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.729 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.729 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.729 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:944 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:56.729 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:12:56.729 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 18.768606281, 2020-01-16T00:12:56.729 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019086901999999999, 2020-01-16T00:12:56.729 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:56.730 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:56.730 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:56.730 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:56.730 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:56.730 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 944 2020-01-16T00:12:56.730 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.730 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:56.730 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.731 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:56.731 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:12:56.731 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:56.731 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.731 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.731 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:56.732 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:12:56.732 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:56.732 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.732 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.733 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:56.733 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914527+0000", 2020-01-16T00:12:56.733 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7509999999999999e-06 2020-01-16T00:12:56.733 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.733 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.733 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:56.734 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914532+0000", 2020-01-16T00:12:56.734 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5149999999999997e-06 2020-01-16T00:12:56.734 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.734 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.734 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:56.734 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914538+0000", 2020-01-16T00:12:56.734 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1890000000000007e-06 2020-01-16T00:12:56.735 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.735 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.735 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:56.735 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914546+0000", 2020-01-16T00:12:56.735 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000354951 2020-01-16T00:12:56.735 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.735 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.735 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.736 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914901+0000", 2020-01-16T00:12:56.736 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1594000000000002e-05 2020-01-16T00:12:56.736 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.736 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.736 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:56.736 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914942+0000", 2020-01-16T00:12:56.736 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012300070999999999 2020-01-16T00:12:56.736 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.737 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.737 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.737 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927242+0000", 2020-01-16T00:12:56.737 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4076999999999997e-05 2020-01-16T00:12:56.737 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.737 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.737 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:56.738 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927296+0000", 2020-01-16T00:12:56.738 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0060609180000000002 2020-01-16T00:12:56.738 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.738 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.738 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.738 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933357+0000", 2020-01-16T00:12:56.738 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4488999999999999e-05 2020-01-16T00:12:56.739 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.739 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.739 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:56.739 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933422+0000", 2020-01-16T00:12:56.739 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019318300000000001 2020-01-16T00:12:56.739 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.739 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.739 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:56.740 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933615+0000", 2020-01-16T00:12:56.740 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019086901999999999 2020-01-16T00:12:56.740 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.740 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:56.740 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.740 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.740 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.740 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:945 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:56.741 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:12:56.741 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 18.768579075000002, 2020-01-16T00:12:56.741 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019342986, 2020-01-16T00:12:56.741 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:56.741 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:56.741 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:56.741 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:56.742 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:56.742 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 945 2020-01-16T00:12:56.742 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.742 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:56.742 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.742 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:56.742 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:12:56.743 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:56.743 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.743 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.743 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:56.743 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:12:56.743 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:56.744 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.744 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.744 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:56.744 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:12:56.744 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4580000000000002e-06 2020-01-16T00:12:56.744 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.745 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.751 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:56.751 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914558+0000", 2020-01-16T00:12:56.751 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7829999999999998e-06 2020-01-16T00:12:56.751 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.751 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.751 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:56.751 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914563+0000", 2020-01-16T00:12:56.752 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0001e-05 2020-01-16T00:12:56.752 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.752 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.752 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:56.752 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914573+0000", 2020-01-16T00:12:56.752 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00040365600000000001 2020-01-16T00:12:56.753 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.753 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.753 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.753 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914976+0000", 2020-01-16T00:12:56.753 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1390000000000001e-05 2020-01-16T00:12:56.754 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.754 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.754 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:56.754 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.915028+0000", 2020-01-16T00:12:56.754 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012288843000000001 2020-01-16T00:12:56.754 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.754 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.754 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.755 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927317+0000", 2020-01-16T00:12:56.755 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7871000000000002e-05 2020-01-16T00:12:56.755 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.755 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.755 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:56.755 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927375+0000", 2020-01-16T00:12:56.755 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0062580589999999998 2020-01-16T00:12:56.755 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.756 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.756 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.756 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933633+0000", 2020-01-16T00:12:56.756 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7086999999999997e-05 2020-01-16T00:12:56.756 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.756 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.756 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:56.757 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933690+0000", 2020-01-16T00:12:56.757 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020855799999999999 2020-01-16T00:12:56.757 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.757 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.757 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:56.757 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933898+0000", 2020-01-16T00:12:56.757 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019342986 2020-01-16T00:12:56.757 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.758 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:56.758 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.758 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.758 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.758 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:963 3.9 3:9557b633:::smithi12313934-43:head [read 0~2705894,checksum] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:56.758 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.046589+0000", 2020-01-16T00:12:56.758 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 18.636545226999999, 2020-01-16T00:12:56.759 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014014016000000001, 2020-01-16T00:12:56.759 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:56.759 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:56.759 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:56.759 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:56.759 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:56.759 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 963 2020-01-16T00:12:56.759 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.760 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:56.760 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.760 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:56.760 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046589+0000", 2020-01-16T00:12:56.760 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:56.760 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.760 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.760 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:56.761 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046589+0000", 2020-01-16T00:12:56.761 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:56.761 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.761 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.761 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:56.761 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046588+0000", 2020-01-16T00:12:56.761 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7469999999999994e-06 2020-01-16T00:12:56.762 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.762 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.762 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:56.762 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046596+0000", 2020-01-16T00:12:56.762 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3445999999999998e-05 2020-01-16T00:12:56.762 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.762 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.763 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:56.763 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046630+0000", 2020-01-16T00:12:56.763 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7779999999999999e-05 2020-01-16T00:12:56.763 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.763 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.763 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:56.764 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046648+0000", 2020-01-16T00:12:56.764 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00075081300000000002 2020-01-16T00:12:56.764 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.764 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.764 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.764 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.047398+0000", 2020-01-16T00:12:56.764 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000116698 2020-01-16T00:12:56.765 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.765 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.765 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:56.765 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.047515+0000", 2020-01-16T00:12:56.765 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0056630689999999997 2020-01-16T00:12:56.765 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.765 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.765 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.766 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.053178+0000", 2020-01-16T00:12:56.766 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012084299999999999 2020-01-16T00:12:56.766 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.766 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.766 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:56.766 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.053299+0000", 2020-01-16T00:12:56.766 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0056921510000000003 2020-01-16T00:12:56.767 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.767 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.767 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.767 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.058991+0000", 2020-01-16T00:12:56.767 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000118606 2020-01-16T00:12:56.767 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.767 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.767 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:56.768 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.059110+0000", 2020-01-16T00:12:56.768 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001493306 2020-01-16T00:12:56.768 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.768 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.768 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:56.768 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.060603+0000", 2020-01-16T00:12:56.768 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014014016000000001 2020-01-16T00:12:56.769 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.769 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:56.769 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.769 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.769 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.769 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:964 3.9 3:9557b633:::smithi12313934-43:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:56.769 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.046668+0000", 2020-01-16T00:12:56.769 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 18.636466623, 2020-01-16T00:12:56.770 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014425670999999999, 2020-01-16T00:12:56.770 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:56.770 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:56.770 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:56.770 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:56.770 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:56.770 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 964 2020-01-16T00:12:56.771 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.771 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:56.771 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.771 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:56.771 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046668+0000", 2020-01-16T00:12:56.771 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:56.771 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.771 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.772 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:56.772 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046668+0000", 2020-01-16T00:12:56.772 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:56.772 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.772 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.772 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:56.772 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046666+0000", 2020-01-16T00:12:56.772 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0879999999999999e-05 2020-01-16T00:12:56.773 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.773 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.773 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:56.773 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046677+0000", 2020-01-16T00:12:56.773 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5767e-05 2020-01-16T00:12:56.773 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.773 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.774 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:56.774 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046693+0000", 2020-01-16T00:12:56.774 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1399999999999999e-05 2020-01-16T00:12:56.774 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.774 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.775 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:56.775 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046704+0000", 2020-01-16T00:12:56.775 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00089331800000000004 2020-01-16T00:12:56.775 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.775 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.775 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.775 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.047597+0000", 2020-01-16T00:12:56.776 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8628000000000005e-05 2020-01-16T00:12:56.776 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.776 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.776 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:56.776 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.047696+0000", 2020-01-16T00:12:56.776 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0056536290000000003 2020-01-16T00:12:56.776 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.776 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.777 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.777 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.053349+0000", 2020-01-16T00:12:56.777 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.6946000000000003e-05 2020-01-16T00:12:56.777 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.777 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.777 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:56.777 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.053446+0000", 2020-01-16T00:12:56.778 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0072058909999999999 2020-01-16T00:12:56.778 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.778 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.778 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.778 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.060652+0000", 2020-01-16T00:12:56.778 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000135597 2020-01-16T00:12:56.778 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.780 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.781 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:56.781 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.060788+0000", 2020-01-16T00:12:56.781 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030548499999999998 2020-01-16T00:12:56.781 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.781 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.781 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:56.781 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.061093+0000", 2020-01-16T00:12:56.782 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014425670999999999 2020-01-16T00:12:56.782 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.782 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:56.782 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.782 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.782 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.782 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:981 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:56.782 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:12:56.783 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 18.367802182999998, 2020-01-16T00:12:56.783 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021222494000000001, 2020-01-16T00:12:56.783 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:56.783 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:56.783 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:56.783 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:56.783 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:56.783 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 981 2020-01-16T00:12:56.784 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.784 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:56.784 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.784 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:56.785 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:12:56.785 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:56.785 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.785 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.785 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:56.785 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:12:56.785 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999948 2020-01-16T00:12:56.785 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.786 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.786 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:56.786 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315327+0000", 2020-01-16T00:12:56.786 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.508e-05 2020-01-16T00:12:56.786 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.786 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.786 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:56.787 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315352+0000", 2020-01-16T00:12:56.787 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1183999999999998e-05 2020-01-16T00:12:56.787 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.787 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.787 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:56.787 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315383+0000", 2020-01-16T00:12:56.787 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7180000000000002e-05 2020-01-16T00:12:56.787 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.788 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.788 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:56.788 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315401+0000", 2020-01-16T00:12:56.788 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017290599999999999 2020-01-16T00:12:56.788 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.788 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.788 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.788 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315574+0000", 2020-01-16T00:12:56.789 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00047908100000000002 2020-01-16T00:12:56.789 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.789 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.789 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:56.789 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316053+0000", 2020-01-16T00:12:56.789 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011808947 2020-01-16T00:12:56.789 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.789 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.790 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.790 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.327862+0000", 2020-01-16T00:12:56.790 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010067499999999999 2020-01-16T00:12:56.790 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.790 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.790 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:56.790 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.327962+0000", 2020-01-16T00:12:56.791 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0074745139999999998 2020-01-16T00:12:56.791 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.791 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.791 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.791 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.335437+0000", 2020-01-16T00:12:56.791 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000103403 2020-01-16T00:12:56.791 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.791 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.792 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:56.792 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.335540+0000", 2020-01-16T00:12:56.792 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001014544 2020-01-16T00:12:56.792 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.792 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.792 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:56.792 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336555+0000", 2020-01-16T00:12:56.792 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021222494000000001 2020-01-16T00:12:56.793 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.793 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:56.793 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.793 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.793 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.793 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:982 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:56.793 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:12:56.793 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 18.367699658999999, 2020-01-16T00:12:56.794 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.022087721000000001, 2020-01-16T00:12:56.794 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:56.794 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:56.794 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:56.794 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:56.794 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:56.795 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 982 2020-01-16T00:12:56.795 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.795 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:56.795 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.795 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:56.795 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:12:56.796 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:56.796 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.796 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.796 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:56.796 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:12:56.796 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:56.796 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.796 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.797 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:56.797 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315433+0000", 2020-01-16T00:12:56.797 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1347e-05 2020-01-16T00:12:56.797 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.797 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.797 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:56.797 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315445+0000", 2020-01-16T00:12:56.798 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4525999999999999e-05 2020-01-16T00:12:56.798 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.798 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.798 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:56.798 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315459+0000", 2020-01-16T00:12:56.798 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.024e-05 2020-01-16T00:12:56.798 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.799 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.799 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:56.799 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315470+0000", 2020-01-16T00:12:56.799 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000736468 2020-01-16T00:12:56.799 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.799 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.799 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.799 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316206+0000", 2020-01-16T00:12:56.800 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000132967 2020-01-16T00:12:56.800 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.800 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.800 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:56.800 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316339+0000", 2020-01-16T00:12:56.800 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011701717 2020-01-16T00:12:56.800 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.801 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.801 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.801 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328041+0000", 2020-01-16T00:12:56.801 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000120339 2020-01-16T00:12:56.801 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.801 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.801 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:56.801 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328161+0000", 2020-01-16T00:12:56.802 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0084341260000000001 2020-01-16T00:12:56.802 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.802 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.802 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.802 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336595+0000", 2020-01-16T00:12:56.802 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5247999999999993e-05 2020-01-16T00:12:56.802 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.803 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.803 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:56.803 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336690+0000", 2020-01-16T00:12:56.803 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00083204000000000004 2020-01-16T00:12:56.803 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.803 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.803 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:56.803 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337522+0000", 2020-01-16T00:12:56.804 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.022087721000000001 2020-01-16T00:12:56.804 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.804 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:56.804 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.804 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.804 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.804 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:983 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:12:56.805 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:12:56.805 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 18.367579949, 2020-01-16T00:12:56.805 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023363387999999999, 2020-01-16T00:12:56.805 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:56.805 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:56.806 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:56.806 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:56.806 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:56.806 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 983 2020-01-16T00:12:56.806 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.806 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:56.806 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.807 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:56.807 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:12:56.807 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:56.807 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.807 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.807 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:56.807 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:12:56.807 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:56.808 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.808 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.808 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:56.808 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315553+0000", 2020-01-16T00:12:56.808 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1760000000000004e-06 2020-01-16T00:12:56.808 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.808 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.808 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:56.809 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315560+0000", 2020-01-16T00:12:56.809 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3587000000000001e-05 2020-01-16T00:12:56.809 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.809 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.809 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:56.809 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315574+0000", 2020-01-16T00:12:56.809 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7800000000000003e-06 2020-01-16T00:12:56.810 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.810 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.810 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:56.810 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315581+0000", 2020-01-16T00:12:56.810 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000823024 2020-01-16T00:12:56.810 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.810 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.810 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.811 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316404+0000", 2020-01-16T00:12:56.811 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2413e-05 2020-01-16T00:12:56.811 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.811 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.811 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:56.811 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316466+0000", 2020-01-16T00:12:56.811 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011738561999999999 2020-01-16T00:12:56.812 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.812 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.812 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.812 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328205+0000", 2020-01-16T00:12:56.812 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6783999999999997e-05 2020-01-16T00:12:56.812 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.812 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.813 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:56.813 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328262+0000", 2020-01-16T00:12:56.813 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0093022560000000001 2020-01-16T00:12:56.813 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.813 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.813 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.813 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337564+0000", 2020-01-16T00:12:56.813 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000183151 2020-01-16T00:12:56.814 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.814 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.814 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:56.814 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337747+0000", 2020-01-16T00:12:56.814 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011707449999999999 2020-01-16T00:12:56.814 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.814 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.815 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:56.815 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.338918+0000", 2020-01-16T00:12:56.815 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023363387999999999 2020-01-16T00:12:56.815 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.815 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:56.815 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.816 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.816 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.816 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:986 3.9 3:93dfe6f1:::smithi12313934-40:head [write 2941380~419987] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:56.816 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:12:56.816 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 18.202280172999998, 2020-01-16T00:12:56.816 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016978400000000001, 2020-01-16T00:12:56.816 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:56.817 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:56.817 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:56.817 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:56.817 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:56.817 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 986 2020-01-16T00:12:56.817 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.817 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:56.817 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.818 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:56.818 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:12:56.818 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:56.818 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.818 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.818 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:56.818 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:12:56.819 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:56.819 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.819 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.819 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:56.819 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480852+0000", 2020-01-16T00:12:56.819 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042503899999999999 2020-01-16T00:12:56.819 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.819 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.820 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:56.820 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481277+0000", 2020-01-16T00:12:56.820 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3097e-05 2020-01-16T00:12:56.820 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.820 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.820 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:56.820 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481290+0000", 2020-01-16T00:12:56.820 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8460000000000002e-06 2020-01-16T00:12:56.821 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.821 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.821 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:56.821 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481297+0000", 2020-01-16T00:12:56.821 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8443999999999999e-05 2020-01-16T00:12:56.821 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.821 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.822 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.822 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481346+0000", 2020-01-16T00:12:56.822 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022115399999999999 2020-01-16T00:12:56.822 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.822 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.822 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:56.822 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481567+0000", 2020-01-16T00:12:56.822 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012531502 2020-01-16T00:12:56.823 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.823 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.823 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.823 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494098+0000", 2020-01-16T00:12:56.823 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.4462999999999993e-05 2020-01-16T00:12:56.823 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.823 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.823 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:56.824 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494193+0000", 2020-01-16T00:12:56.824 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017600999999999999 2020-01-16T00:12:56.824 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.824 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.824 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:12:56.824 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494369+0000", 2020-01-16T00:12:56.824 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0031739619999999998 2020-01-16T00:12:56.825 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.825 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.825 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:56.825 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497543+0000", 2020-01-16T00:12:56.825 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000175107 2020-01-16T00:12:56.825 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.825 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.825 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:56.826 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497718+0000", 2020-01-16T00:12:56.826 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5207000000000003e-05 2020-01-16T00:12:56.826 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.826 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.826 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:56.827 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497763+0000", 2020-01-16T00:12:56.827 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7042999999999998e-05 2020-01-16T00:12:56.827 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.827 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.827 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:56.827 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497800+0000", 2020-01-16T00:12:56.827 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2628e-05 2020-01-16T00:12:56.828 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.828 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.828 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:56.828 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497833+0000", 2020-01-16T00:12:56.828 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016978400000000001 2020-01-16T00:12:56.828 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.828 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:56.829 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.829 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.829 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.829 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:987 3.9 3:93dfe6f1:::smithi12313934-40:head [setxattr _header (56) in=7b,truncate 3361367] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:56.829 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.481307+0000", 2020-01-16T00:12:56.829 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 18.201827806000001, 2020-01-16T00:12:56.829 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016794989, 2020-01-16T00:12:56.829 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:56.830 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:56.830 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:56.830 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:56.830 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:56.830 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 987 2020-01-16T00:12:56.830 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.830 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:56.831 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.831 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:56.831 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481307+0000", 2020-01-16T00:12:56.831 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:56.831 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.831 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.831 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:56.831 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481307+0000", 2020-01-16T00:12:56.832 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:56.832 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.832 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.832 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:56.832 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481306+0000", 2020-01-16T00:12:56.832 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.113e-06 2020-01-16T00:12:56.832 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.832 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.833 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:56.833 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481311+0000", 2020-01-16T00:12:56.833 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1520000000000002e-06 2020-01-16T00:12:56.833 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.833 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.833 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:56.833 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481315+0000", 2020-01-16T00:12:56.834 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.7640000000000001e-06 2020-01-16T00:12:56.834 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.834 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.834 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:56.834 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481318+0000", 2020-01-16T00:12:56.834 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00029012500000000002 2020-01-16T00:12:56.834 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.834 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.835 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.835 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481608+0000", 2020-01-16T00:12:56.835 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6239000000000002e-05 2020-01-16T00:12:56.835 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.835 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.835 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:56.836 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481654+0000", 2020-01-16T00:12:56.836 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014559534000000001 2020-01-16T00:12:56.836 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.836 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.836 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.836 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.496214+0000", 2020-01-16T00:12:56.837 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000225932 2020-01-16T00:12:56.837 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.837 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.837 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:56.837 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.496439+0000", 2020-01-16T00:12:56.837 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00031099600000000002 2020-01-16T00:12:56.837 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.838 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.838 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:12:56.838 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.496750+0000", 2020-01-16T00:12:56.838 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00098608599999999995 2020-01-16T00:12:56.838 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.838 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.838 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:56.838 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497737+0000", 2020-01-16T00:12:56.839 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022081499999999999 2020-01-16T00:12:56.839 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.839 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.839 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:56.839 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497957+0000", 2020-01-16T00:12:56.839 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7071000000000003e-05 2020-01-16T00:12:56.839 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.839 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.840 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:56.840 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498014+0000", 2020-01-16T00:12:56.840 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1111e-05 2020-01-16T00:12:56.840 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.840 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.840 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:56.840 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498056+0000", 2020-01-16T00:12:56.840 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5974999999999999e-05 2020-01-16T00:12:56.841 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.841 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.841 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:56.841 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498102+0000", 2020-01-16T00:12:56.841 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016794989 2020-01-16T00:12:56.841 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.841 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:56.842 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.842 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.842 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.842 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:988 3.9 3:93dfe6f1:::smithi12313934-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:12:56.842 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:12:56.842 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 18.201794709000001, 2020-01-16T00:12:56.842 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017221692, 2020-01-16T00:12:56.842 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:56.843 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:56.843 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:56.843 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:56.843 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:56.843 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 988 2020-01-16T00:12:56.843 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.843 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:56.843 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.844 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:56.844 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:12:56.844 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:56.844 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.844 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.844 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:56.844 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:12:56.845 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:56.845 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.845 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.845 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:56.845 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481339+0000", 2020-01-16T00:12:56.845 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3089999999999997e-06 2020-01-16T00:12:56.845 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.845 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.846 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:56.846 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481343+0000", 2020-01-16T00:12:56.846 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2230000000000004e-06 2020-01-16T00:12:56.846 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.846 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.847 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:56.847 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481350+0000", 2020-01-16T00:12:56.847 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.319e-06 2020-01-16T00:12:56.847 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.847 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.847 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:56.847 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481356+0000", 2020-01-16T00:12:56.847 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000321544 2020-01-16T00:12:56.848 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.848 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.848 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.848 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481678+0000", 2020-01-16T00:12:56.848 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6375999999999999e-05 2020-01-16T00:12:56.848 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.848 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.849 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:56.849 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481714+0000", 2020-01-16T00:12:56.849 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015557666 2020-01-16T00:12:56.849 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.849 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.849 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.849 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497272+0000", 2020-01-16T00:12:56.849 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012791300000000001 2020-01-16T00:12:56.850 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.850 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.850 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:56.850 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497399+0000", 2020-01-16T00:12:56.850 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00076944600000000002 2020-01-16T00:12:56.850 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.850 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.850 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.851 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498169+0000", 2020-01-16T00:12:56.851 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018306799999999999 2020-01-16T00:12:56.851 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.851 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.851 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:56.851 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498352+0000", 2020-01-16T00:12:56.851 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020935999999999999 2020-01-16T00:12:56.852 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.852 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.852 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:56.852 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498561+0000", 2020-01-16T00:12:56.852 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017221692 2020-01-16T00:12:56.852 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.852 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:56.853 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.853 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.853 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.853 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1208 3.3 3:c478bfe8:::smithi12313934-10:head [write 749612~474179,stat] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:12:56.853 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.350952+0000", 2020-01-16T00:12:56.853 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.332181872, 2020-01-16T00:12:56.853 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014967729000000001, 2020-01-16T00:12:56.854 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:56.854 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:56.854 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:56.854 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:56.854 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:56.854 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1208 2020-01-16T00:12:56.854 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.854 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:56.855 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.855 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:56.855 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.350952+0000", 2020-01-16T00:12:56.855 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:56.855 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.855 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.855 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:56.855 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.350952+0000", 2020-01-16T00:12:56.856 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:12:56.856 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.856 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.856 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:56.856 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.350949+0000", 2020-01-16T00:12:56.856 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042894 2020-01-16T00:12:56.856 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.857 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.857 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:56.857 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.351377+0000", 2020-01-16T00:12:56.857 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3921e-05 2020-01-16T00:12:56.858 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.858 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.858 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:56.858 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.351401+0000", 2020-01-16T00:12:56.858 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1087000000000001e-05 2020-01-16T00:12:56.858 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.858 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.858 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:56.859 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.351412+0000", 2020-01-16T00:12:56.859 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8127000000000001e-05 2020-01-16T00:12:56.859 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.859 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.859 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.859 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.351471+0000", 2020-01-16T00:12:56.859 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000230268 2020-01-16T00:12:56.859 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.860 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.860 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:56.860 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.351701+0000", 2020-01-16T00:12:56.860 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0051228369999999999 2020-01-16T00:12:56.860 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.860 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.860 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.861 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.356824+0000", 2020-01-16T00:12:56.861 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015984800000000001 2020-01-16T00:12:56.861 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.862 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.862 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:56.862 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.356984+0000", 2020-01-16T00:12:56.862 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0038163770000000001 2020-01-16T00:12:56.862 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.862 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.863 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.863 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.360800+0000", 2020-01-16T00:12:56.863 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.9878999999999999e-05 2020-01-16T00:12:56.863 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.863 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.863 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:56.864 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.360880+0000", 2020-01-16T00:12:56.864 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000145041 2020-01-16T00:12:56.864 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.864 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.864 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2020-01-16T00:12:56.864 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.361025+0000", 2020-01-16T00:12:56.864 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0020743039999999999 2020-01-16T00:12:56.865 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.865 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.865 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:56.865 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.363099+0000", 2020-01-16T00:12:56.865 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00072876500000000003 2020-01-16T00:12:56.865 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.865 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.865 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:56.866 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.363828+0000", 2020-01-16T00:12:56.866 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00189153 2020-01-16T00:12:56.866 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.866 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.866 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:56.866 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.365719+0000", 2020-01-16T00:12:56.866 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3948000000000005e-05 2020-01-16T00:12:56.867 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.867 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.867 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:56.867 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.365783+0000", 2020-01-16T00:12:56.867 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000136833 2020-01-16T00:12:56.868 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.868 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.868 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:56.868 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.365920+0000", 2020-01-16T00:12:56.868 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014967729000000001 2020-01-16T00:12:56.868 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.868 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:56.869 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.869 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.869 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.869 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1237 3.3 3:c204aa4f:::smithi12313934-45:head [write 400033~699926] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:12:56.869 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.584946+0000", 2020-01-16T00:12:56.869 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.09818799, 2020-01-16T00:12:56.869 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014287855, 2020-01-16T00:12:56.870 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:56.870 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:56.870 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:56.870 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:56.870 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:56.870 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1237 2020-01-16T00:12:56.870 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.870 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:56.871 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.871 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:56.871 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.584946+0000", 2020-01-16T00:12:56.871 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:56.871 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.871 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.871 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:56.871 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.584946+0000", 2020-01-16T00:12:56.872 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999752 2020-01-16T00:12:56.872 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.872 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.872 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:56.872 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.584921+0000", 2020-01-16T00:12:56.872 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00077243400000000001 2020-01-16T00:12:56.872 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.873 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.873 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:56.873 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.585694+0000", 2020-01-16T00:12:56.873 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3083000000000001e-05 2020-01-16T00:12:56.873 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.873 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.873 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:56.873 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.585737+0000", 2020-01-16T00:12:56.874 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1854000000000001e-05 2020-01-16T00:12:56.874 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.874 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.874 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:56.874 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.585759+0000", 2020-01-16T00:12:56.874 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016127000000000001 2020-01-16T00:12:56.874 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.875 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.875 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.875 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.585920+0000", 2020-01-16T00:12:56.875 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024934700000000002 2020-01-16T00:12:56.875 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.875 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.875 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:56.876 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.586169+0000", 2020-01-16T00:12:56.876 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00036508999999999999 2020-01-16T00:12:56.877 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.877 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.877 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2020-01-16T00:12:56.877 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.586534+0000", 2020-01-16T00:12:56.877 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012168910999999999 2020-01-16T00:12:56.878 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.878 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.878 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:56.878 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598703+0000", 2020-01-16T00:12:56.878 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5302000000000004e-05 2020-01-16T00:12:56.878 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.878 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.878 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:56.879 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598789+0000", 2020-01-16T00:12:56.879 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00027301099999999998 2020-01-16T00:12:56.879 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.879 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.879 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:56.879 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599062+0000", 2020-01-16T00:12:56.879 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.4237e-05 2020-01-16T00:12:56.880 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.880 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.880 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:56.880 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599146+0000", 2020-01-16T00:12:56.880 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.8319000000000006e-05 2020-01-16T00:12:56.880 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.880 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.881 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:56.881 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599234+0000", 2020-01-16T00:12:56.881 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014287855 2020-01-16T00:12:56.881 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.881 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:56.881 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.881 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.881 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.882 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1239 3.3 3:c204aa4f:::smithi12313934-45:head [setxattr _header (56) in=7b,truncate 2000033] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:12:56.882 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:12:56.882 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.092720890000001, 2020-01-16T00:12:56.882 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019161661999999999, 2020-01-16T00:12:56.882 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:56.882 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:56.882 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:56.883 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:56.883 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:56.883 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1239 2020-01-16T00:12:56.883 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.883 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:56.883 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.883 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:56.883 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:12:56.884 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:56.884 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.884 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.884 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:56.884 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:12:56.884 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:12:56.884 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.884 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.885 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:56.885 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590411+0000", 2020-01-16T00:12:56.885 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3879999999999999e-05 2020-01-16T00:12:56.885 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.885 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.885 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:56.885 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590425+0000", 2020-01-16T00:12:56.886 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6542999999999998e-05 2020-01-16T00:12:56.886 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.886 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.886 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:56.886 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590471+0000", 2020-01-16T00:12:56.886 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8907999999999999e-05 2020-01-16T00:12:56.886 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.886 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.887 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:56.887 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590490+0000", 2020-01-16T00:12:56.887 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0045691960000000002 2020-01-16T00:12:56.887 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.887 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.887 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.887 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.595059+0000", 2020-01-16T00:12:56.887 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0035638940000000002 2020-01-16T00:12:56.888 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.888 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.888 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:56.888 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598623+0000", 2020-01-16T00:12:56.888 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0077558310000000004 2020-01-16T00:12:56.888 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.888 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.888 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.889 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606379+0000", 2020-01-16T00:12:56.889 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012171999999999999 2020-01-16T00:12:56.889 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.889 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.889 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:56.889 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606501+0000", 2020-01-16T00:12:56.889 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013144200000000001 2020-01-16T00:12:56.890 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.890 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.890 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2020-01-16T00:12:56.890 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606632+0000", 2020-01-16T00:12:56.890 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001288565 2020-01-16T00:12:56.890 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.890 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.890 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:56.891 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607921+0000", 2020-01-16T00:12:56.891 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033399299999999999 2020-01-16T00:12:56.891 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.891 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.891 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:56.891 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.608255+0000", 2020-01-16T00:12:56.891 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012280870000000001 2020-01-16T00:12:56.891 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.892 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.892 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:56.892 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609483+0000", 2020-01-16T00:12:56.892 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2286000000000001e-05 2020-01-16T00:12:56.892 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.892 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.892 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:56.892 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609525+0000", 2020-01-16T00:12:56.893 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0121999999999999e-05 2020-01-16T00:12:56.893 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.893 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.893 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:56.893 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609575+0000", 2020-01-16T00:12:56.893 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019161661999999999 2020-01-16T00:12:56.893 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.894 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:56.894 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.894 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.894 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.894 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1240 3.3 3:c204aa4f:::smithi12313934-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-01-16T00:12:56.894 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:12:56.894 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.092620701, 2020-01-16T00:12:56.895 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019268828000000002, 2020-01-16T00:12:56.895 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:56.895 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:56.895 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:56.895 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:56.895 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:56.895 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1240 2020-01-16T00:12:56.895 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.896 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:56.896 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.896 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:56.896 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:12:56.896 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:56.896 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.896 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.896 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:56.897 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:12:56.897 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:56.897 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.897 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.897 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:56.897 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590512+0000", 2020-01-16T00:12:56.897 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0525e-05 2020-01-16T00:12:56.898 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.898 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.898 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:56.898 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590522+0000", 2020-01-16T00:12:56.898 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6418000000000001e-05 2020-01-16T00:12:56.898 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.898 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.898 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:56.899 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590539+0000", 2020-01-16T00:12:56.899 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0679999999999999e-05 2020-01-16T00:12:56.899 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.899 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.899 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:56.899 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590549+0000", 2020-01-16T00:12:56.899 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0083626789999999996 2020-01-16T00:12:56.899 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.900 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.900 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.900 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598912+0000", 2020-01-16T00:12:56.900 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013344 2020-01-16T00:12:56.900 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.900 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.900 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:56.900 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599045+0000", 2020-01-16T00:12:56.901 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0084002269999999997 2020-01-16T00:12:56.901 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.901 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.901 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.901 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607446+0000", 2020-01-16T00:12:56.901 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000109873 2020-01-16T00:12:56.901 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.902 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.902 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:56.902 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607555+0000", 2020-01-16T00:12:56.902 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0020573309999999999 2020-01-16T00:12:56.902 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.902 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.902 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.902 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609613+0000", 2020-01-16T00:12:56.903 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6351e-05 2020-01-16T00:12:56.903 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.903 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.903 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:56.903 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609699+0000", 2020-01-16T00:12:56.903 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3374999999999994e-05 2020-01-16T00:12:56.903 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.903 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.904 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:56.904 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609782+0000", 2020-01-16T00:12:56.904 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019268828000000002 2020-01-16T00:12:56.904 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.904 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:56.904 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.904 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.905 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.905 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1436 3.3 3:c204aa4f:::smithi12313934-45:head [setxattr _header (56) in=7b,truncate 2386594] snapc 0=[] ondisk+write+known_if_redirected e28)", 2020-01-16T00:12:56.905 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:47.900544+0000", 2020-01-16T00:12:56.905 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 8.7825900200000007, 2020-01-16T00:12:56.905 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016402521, 2020-01-16T00:12:56.905 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:56.905 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:56.905 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:56.906 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:56.906 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:56.906 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1436 2020-01-16T00:12:56.906 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.906 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:56.906 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.906 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:56.906 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900544+0000", 2020-01-16T00:12:56.907 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:56.907 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.907 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.907 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:56.907 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900544+0000", 2020-01-16T00:12:56.907 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:56.907 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.907 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.908 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:56.908 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900542+0000", 2020-01-16T00:12:56.908 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3862999999999999e-05 2020-01-16T00:12:56.908 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.908 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.908 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:56.908 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900556+0000", 2020-01-16T00:12:56.908 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5204e-05 2020-01-16T00:12:56.909 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.909 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.909 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:56.909 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900571+0000", 2020-01-16T00:12:56.909 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.092e-05 2020-01-16T00:12:56.909 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.909 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.910 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:56.910 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900582+0000", 2020-01-16T00:12:56.910 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002786026 2020-01-16T00:12:56.910 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.910 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.910 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.910 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.903368+0000", 2020-01-16T00:12:56.910 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0027123659999999999 2020-01-16T00:12:56.911 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.911 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.911 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:56.911 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.906081+0000", 2020-01-16T00:12:56.911 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0082707790000000007 2020-01-16T00:12:56.911 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.911 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.911 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.912 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.914352+0000", 2020-01-16T00:12:56.912 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020580300000000001 2020-01-16T00:12:56.912 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.912 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.912 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:56.912 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.914557+0000", 2020-01-16T00:12:56.912 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017729900000000001 2020-01-16T00:12:56.913 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.913 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.913 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2020-01-16T00:12:56.913 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.914735+0000", 2020-01-16T00:12:56.913 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00074632099999999999 2020-01-16T00:12:56.913 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.913 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.913 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:56.914 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.915481+0000", 2020-01-16T00:12:56.914 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012279599999999999 2020-01-16T00:12:56.914 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.914 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.914 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:56.914 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.915604+0000", 2020-01-16T00:12:56.914 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012619339999999999 2020-01-16T00:12:56.914 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.915 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.915 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:56.915 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.916866+0000", 2020-01-16T00:12:56.915 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1162000000000003e-05 2020-01-16T00:12:56.915 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.915 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.915 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:56.915 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.916907+0000", 2020-01-16T00:12:56.916 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9987999999999997e-05 2020-01-16T00:12:56.916 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.916 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.916 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:56.916 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.916947+0000", 2020-01-16T00:12:56.916 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016402521 2020-01-16T00:12:56.916 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.917 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:56.917 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.917 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.917 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.917 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1437 3.3 3:c204aa4f:::smithi12313934-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e28)", 2020-01-16T00:12:56.917 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:47.900594+0000", 2020-01-16T00:12:56.917 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 8.7825401569999997, 2020-01-16T00:12:56.917 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016745237999999999, 2020-01-16T00:12:56.918 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:56.918 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:56.918 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:56.918 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:56.918 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:56.918 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1437 2020-01-16T00:12:56.918 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.918 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:56.919 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.919 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:56.919 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900594+0000", 2020-01-16T00:12:56.919 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:56.919 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.919 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.919 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:56.920 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900594+0000", 2020-01-16T00:12:56.920 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:56.920 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.920 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.920 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:56.920 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900593+0000", 2020-01-16T00:12:56.920 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5059999999999995e-06 2020-01-16T00:12:56.920 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.921 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.921 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:56.921 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900601+0000", 2020-01-16T00:12:56.921 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5077e-05 2020-01-16T00:12:56.921 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.921 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.921 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:56.921 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900616+0000", 2020-01-16T00:12:56.922 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0183000000000001e-05 2020-01-16T00:12:56.922 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.922 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.922 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:56.922 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900627+0000", 2020-01-16T00:12:56.922 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0063667660000000003 2020-01-16T00:12:56.922 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.923 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.923 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.923 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.906993+0000", 2020-01-16T00:12:56.923 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8988e-05 2020-01-16T00:12:56.923 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.923 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.923 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:56.923 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.907032+0000", 2020-01-16T00:12:56.924 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0080128850000000008 2020-01-16T00:12:56.924 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.924 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.924 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.924 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.915045+0000", 2020-01-16T00:12:56.924 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013200899999999999 2020-01-16T00:12:56.924 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.924 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.925 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:56.925 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.915177+0000", 2020-01-16T00:12:56.925 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0018373090000000001 2020-01-16T00:12:56.925 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.925 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.925 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:56.925 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.917015+0000", 2020-01-16T00:12:56.925 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016701699999999999 2020-01-16T00:12:56.926 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.926 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.926 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:56.926 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.917182+0000", 2020-01-16T00:12:56.926 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015782400000000001 2020-01-16T00:12:56.926 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:56.926 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:56.927 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:56.927 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.917339+0000", 2020-01-16T00:12:56.927 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016745237999999999 2020-01-16T00:12:56.927 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.927 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:56.927 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.927 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:56.927 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:56.928 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:56.928 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2020-01-16T00:12:56.930 INFO:tasks.rados.rados.0.smithi123.stdout:899: finishing write tid 3 to smithi12313934-11 2020-01-16T00:12:56.930 INFO:tasks.rados.rados.0.smithi123.stdout:899: finishing write tid 5 to smithi12313934-11 2020-01-16T00:12:56.930 INFO:tasks.rados.rados.0.smithi123.stdout:899: finishing write tid 6 to smithi12313934-11 2020-01-16T00:12:56.930 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 11 v 64 (ObjNum 191 snap 0 seq_num 191) dirty exists 2020-01-16T00:12:56.930 INFO:tasks.rados.rados.0.smithi123.stdout:899: left oid 11 (ObjNum 191 snap 0 seq_num 191) 2020-01-16T00:12:56.930 INFO:tasks.rados.rados.0.smithi123.stdout:900: expect (ObjNum 138 snap 0 seq_num 138) 2020-01-16T00:12:56.931 INFO:tasks.rados.rados.0.smithi123.stdout:901: expect (ObjNum 174 snap 0 seq_num 174) 2020-01-16T00:12:56.931 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:56.929+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:12:56.931 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-01-16T00:12:56.950 INFO:tasks.rados.rados.0.smithi123.stdout:905: finishing write tid 1 to smithi12313934-41 2020-01-16T00:12:56.950 INFO:tasks.rados.rados.0.smithi123.stdout:905: finishing write tid 2 to smithi12313934-41 2020-01-16T00:12:56.950 INFO:tasks.rados.rados.0.smithi123.stdout:905: finishing write tid 3 to smithi12313934-41 2020-01-16T00:12:56.950 INFO:tasks.rados.rados.0.smithi123.stdout:905: finishing write tid 5 to smithi12313934-41 2020-01-16T00:12:56.950 INFO:tasks.rados.rados.0.smithi123.stdout:905: finishing write tid 6 to smithi12313934-41 2020-01-16T00:12:56.950 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 41 v 123 (ObjNum 193 snap 0 seq_num 193) dirty exists 2020-01-16T00:12:56.951 INFO:tasks.rados.rados.0.smithi123.stdout:905: left oid 41 (ObjNum 193 snap 0 seq_num 193) 2020-01-16T00:12:56.951 INFO:tasks.rados.rados.0.smithi123.stdout:906: expect (ObjNum 176 snap 0 seq_num 176) 2020-01-16T00:12:57.030 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:12:57.030+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:12:57.087 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:57.087 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:12:57.087 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:12:57.087 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:57.097 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-01-16T00:12:57.143 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:57.145+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:12:57.163 INFO:tasks.rados.rados.0.smithi123.stdout:907: finishing write tid 1 to smithi12313934-39 2020-01-16T00:12:57.163 INFO:tasks.rados.rados.0.smithi123.stdout:907: finishing write tid 2 to smithi12313934-39 2020-01-16T00:12:57.163 INFO:tasks.rados.rados.0.smithi123.stdout:907: finishing write tid 3 to smithi12313934-39 2020-01-16T00:12:57.163 INFO:tasks.rados.rados.0.smithi123.stdout:907: finishing write tid 5 to smithi12313934-39 2020-01-16T00:12:57.163 INFO:tasks.rados.rados.0.smithi123.stdout:907: finishing write tid 6 to smithi12313934-39 2020-01-16T00:12:57.164 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 39 v 111 (ObjNum 194 snap 0 seq_num 194) dirty exists 2020-01-16T00:12:57.164 INFO:tasks.rados.rados.0.smithi123.stdout:907: left oid 39 (ObjNum 194 snap 0 seq_num 194) 2020-01-16T00:12:57.164 INFO:tasks.rados.rados.0.smithi123.stdout:908: finishing write tid 1 to smithi12313934-8 2020-01-16T00:12:57.164 INFO:tasks.rados.rados.0.smithi123.stdout:898: done (15 left) 2020-01-16T00:12:57.164 INFO:tasks.rados.rados.0.smithi123.stdout:899: done (14 left) 2020-01-16T00:12:57.164 INFO:tasks.rados.rados.0.smithi123.stdout:900: done (13 left) 2020-01-16T00:12:57.164 INFO:tasks.rados.rados.0.smithi123.stdout:901: done (12 left) 2020-01-16T00:12:57.164 INFO:tasks.rados.rados.0.smithi123.stdout:902: done (11 left) 2020-01-16T00:12:57.165 INFO:tasks.rados.rados.0.smithi123.stdout:903: done (10 left) 2020-01-16T00:12:57.165 INFO:tasks.rados.rados.0.smithi123.stdout:905: done (9 left) 2020-01-16T00:12:57.165 INFO:tasks.rados.rados.0.smithi123.stdout:906: done (8 left) 2020-01-16T00:12:57.165 INFO:tasks.rados.rados.0.smithi123.stdout:907: done (7 left) 2020-01-16T00:12:57.165 INFO:tasks.rados.rados.0.smithi123.stdout:915: write oid 29 current snap is 0 2020-01-16T00:12:57.165 INFO:tasks.rados.rados.0.smithi123.stdout:915: seq_num 200 ranges {561400=474978,1820281=519723,2859422=457579} 2020-01-16T00:12:57.177 INFO:tasks.rados.rados.0.smithi123.stdout:915: writing smithi12313934-29 from 561400 to 1036378 tid 1 2020-01-16T00:12:57.185 INFO:tasks.rados.rados.0.smithi123.stdout:915: writing smithi12313934-29 from 1820281 to 2340004 tid 2 2020-01-16T00:12:57.191 INFO:tasks.rados.rados.0.smithi123.stdout:915: writing smithi12313934-29 from 2859422 to 3317001 tid 3 2020-01-16T00:12:57.191 INFO:tasks.rados.rados.0.smithi123.stdout:916: write oid 25 current snap is 0 2020-01-16T00:12:57.191 INFO:tasks.rados.rados.0.smithi123.stdout:916: seq_num 201 ranges {578207=550227,1762410=779734,3286845=91362} 2020-01-16T00:12:57.198 INFO:tasks.rados.rados.0.smithi123.stdout:916: writing smithi12313934-25 from 578207 to 1128434 tid 1 2020-01-16T00:12:57.206 INFO:tasks.rados.rados.0.smithi123.stdout:916: writing smithi12313934-25 from 1762410 to 2542144 tid 2 2020-01-16T00:12:57.210 INFO:tasks.rados.rados.0.smithi123.stdout:916: writing smithi12313934-25 from 3286845 to 3378207 tid 3 2020-01-16T00:12:57.211 INFO:tasks.rados.rados.0.smithi123.stdout:917: write oid 33 current snap is 0 2020-01-16T00:12:57.211 INFO:tasks.rados.rados.0.smithi123.stdout:917: seq_num 202 ranges {595014=625476,1820892=523392,2797915=596476} 2020-01-16T00:12:57.211 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:57.211 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:12:57.211 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:12:57.212 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:12:57.212 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:57.217 INFO:tasks.rados.rados.0.smithi123.stdout:917: writing smithi12313934-33 from 595014 to 1220490 tid 1 2020-01-16T00:12:57.226 INFO:tasks.rados.rados.0.smithi123.stdout:917: writing smithi12313934-33 from 1820892 to 2344284 tid 2 2020-01-16T00:12:57.226 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-01-16T00:12:57.230 INFO:tasks.rados.rados.0.smithi123.stdout:917: writing smithi12313934-33 from 2797915 to 3394391 tid 3 2020-01-16T00:12:57.230 INFO:tasks.rados.rados.0.smithi123.stdout:918: write oid 31 current snap is 0 2020-01-16T00:12:57.230 INFO:tasks.rados.rados.0.smithi123.stdout:908: finishing write tid 2 to smithi12313934-8 2020-01-16T00:12:57.230 INFO:tasks.rados.rados.0.smithi123.stdout:918: seq_num 203 ranges {611821=700725,1879374=783403,3341691=70130} 2020-01-16T00:12:57.237 INFO:tasks.rados.rados.0.smithi123.stdout:918: writing smithi12313934-31 from 611821 to 1312546 tid 1 2020-01-16T00:12:57.245 INFO:tasks.rados.rados.0.smithi123.stdout:918: writing smithi12313934-31 from 1879374 to 2662777 tid 2 2020-01-16T00:12:57.250 INFO:tasks.rados.rados.0.smithi123.stdout:918: writing smithi12313934-31 from 3341691 to 3411821 tid 3 2020-01-16T00:12:57.250 INFO:tasks.rados.rados.0.smithi123.stdout:919: read oid 45 snap -1 2020-01-16T00:12:57.250 INFO:tasks.rados.rados.0.smithi123.stdout:919: expect (ObjNum 184 snap 0 seq_num 184) 2020-01-16T00:12:57.251 INFO:tasks.rados.rados.0.smithi123.stdout:920: write oid 42 current snap is 0 2020-01-16T00:12:57.251 INFO:tasks.rados.rados.0.smithi123.stdout:920: seq_num 204 ranges {628628=775974,1937856=527061,3252761=175867} 2020-01-16T00:12:57.258 INFO:tasks.rados.rados.0.smithi123.stdout:920: writing smithi12313934-42 from 628628 to 1404602 tid 1 2020-01-16T00:12:57.264 INFO:tasks.rados.rados.0.smithi123.stdout:920: writing smithi12313934-42 from 1937856 to 2464917 tid 2 2020-01-16T00:12:57.267 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:12:57.269+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:12:57.270 INFO:tasks.rados.rados.0.smithi123.stdout:920: writing smithi12313934-42 from 3252761 to 3428628 tid 3 2020-01-16T00:12:57.271 INFO:tasks.rados.rados.0.smithi123.stdout:921: write (excl) oid 17 current snap is 0 2020-01-16T00:12:57.271 INFO:tasks.rados.rados.0.smithi123.stdout:921: seq_num 205 ranges {645435=451223,1879985=787072,3280184=165251} 2020-01-16T00:12:57.276 INFO:tasks.rados.rados.0.smithi123.stdout:921: writing smithi12313934-17 from 645435 to 1096658 tid 1 2020-01-16T00:12:57.285 INFO:tasks.rados.rados.0.smithi123.stdout:921: writing smithi12313934-17 from 1879985 to 2667057 tid 2 2020-01-16T00:12:57.290 INFO:tasks.rados.rados.0.smithi123.stdout:921: writing smithi12313934-17 from 3280184 to 3445435 tid 3 2020-01-16T00:12:57.290 INFO:tasks.rados.rados.0.smithi123.stdout:922: write (excl) oid 35 current snap is 0 2020-01-16T00:12:57.290 INFO:tasks.rados.rados.0.smithi123.stdout:922: seq_num 206 ranges {662242=642825,2054820=530730,3307607=154635} 2020-01-16T00:12:57.297 INFO:tasks.rados.rados.0.smithi123.stdout:922: writing smithi12313934-35 from 662242 to 1305067 tid 1 2020-01-16T00:12:57.305 INFO:tasks.rados.rados.0.smithi123.stdout:922: writing smithi12313934-35 from 2054820 to 2585550 tid 2 2020-01-16T00:12:57.310 INFO:tasks.rados.rados.0.smithi123.stdout:922: writing smithi12313934-35 from 3307607 to 3462242 tid 3 2020-01-16T00:12:57.310 INFO:tasks.rados.rados.0.smithi123.stdout:923: write oid 2 current snap is 0 2020-01-16T00:12:57.311 INFO:tasks.rados.rados.0.smithi123.stdout:923: seq_num 207 ranges {679049=718074,2113302=674388,3335030=144019} 2020-01-16T00:12:57.319 INFO:tasks.rados.rados.0.smithi123.stdout:923: writing smithi12313934-2 from 679049 to 1397123 tid 1 2020-01-16T00:12:57.327 INFO:tasks.rados.rados.0.smithi123.stdout:923: writing smithi12313934-2 from 2113302 to 2787690 tid 2 2020-01-16T00:12:57.331 INFO:tasks.rados.rados.0.smithi123.stdout:923: writing smithi12313934-2 from 3335030 to 3479049 tid 3 2020-01-16T00:12:57.331 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:57.331 INFO:tasks.rados.rados.0.smithi123.stdout:908: finishing write tid 3 to smithi12313934-8 2020-01-16T00:12:57.332 INFO:tasks.rados.rados.0.smithi123.stdout:908: finishing write tid 5 to smithi12313934-8 2020-01-16T00:12:57.332 INFO:tasks.rados.rados.0.smithi123.stdout:908: finishing write tid 6 to smithi12313934-8 2020-01-16T00:12:57.332 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 8 v 66 (ObjNum 195 snap 0 seq_num 195) dirty exists 2020-01-16T00:12:57.332 INFO:tasks.rados.rados.0.smithi123.stdout:908: left oid 8 (ObjNum 195 snap 0 seq_num 195) 2020-01-16T00:12:57.332 INFO:tasks.rados.rados.0.smithi123.stdout:909: expect (ObjNum 143 snap 0 seq_num 143) 2020-01-16T00:12:57.334 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:57.333+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:57.374 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:57.374 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:12:57.374 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:12:57.374 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:12:57.374 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.374 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:898 3.d 3:bd9fc72f:::smithi12313934-5:head [write 776490~714607] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:57.375 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.131050+0000", 2020-01-16T00:12:57.375 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 20.243847294999998, 2020-01-16T00:12:57.375 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0076965840000000002, 2020-01-16T00:12:57.375 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:57.375 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:57.375 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:57.376 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:57.376 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:57.376 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 898 2020-01-16T00:12:57.376 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.376 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:57.376 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.376 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:57.376 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.131050+0000", 2020-01-16T00:12:57.377 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:57.377 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.377 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.377 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:57.377 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.131050+0000", 2020-01-16T00:12:57.377 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:12:57.377 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.378 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.378 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:57.378 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.131046+0000", 2020-01-16T00:12:57.378 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0013709010000000001 2020-01-16T00:12:57.378 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.378 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.379 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:57.379 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132417+0000", 2020-01-16T00:12:57.379 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1613000000000001e-05 2020-01-16T00:12:57.379 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.379 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.379 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:57.379 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132438+0000", 2020-01-16T00:12:57.379 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0764000000000001e-05 2020-01-16T00:12:57.380 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.380 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.380 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:57.380 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132449+0000", 2020-01-16T00:12:57.380 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5473e-05 2020-01-16T00:12:57.380 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.380 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.380 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:57.381 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132535+0000", 2020-01-16T00:12:57.381 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013572000000000001 2020-01-16T00:12:57.381 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.381 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.381 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:57.382 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132670+0000", 2020-01-16T00:12:57.382 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000124719 2020-01-16T00:12:57.382 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.382 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.382 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:12:57.382 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132795+0000", 2020-01-16T00:12:57.382 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023122680000000001 2020-01-16T00:12:57.383 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.383 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.383 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:57.383 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.135107+0000", 2020-01-16T00:12:57.383 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00094608099999999998 2020-01-16T00:12:57.383 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.384 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.384 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:57.384 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136053+0000", 2020-01-16T00:12:57.384 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0026099780000000002 2020-01-16T00:12:57.384 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.385 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.385 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:57.385 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.138663+0000", 2020-01-16T00:12:57.385 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4024999999999999e-05 2020-01-16T00:12:57.385 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.386 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.386 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:57.386 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.138707+0000", 2020-01-16T00:12:57.386 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9035000000000001e-05 2020-01-16T00:12:57.386 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.387 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.387 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:57.387 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.138746+0000", 2020-01-16T00:12:57.387 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0076965840000000002 2020-01-16T00:12:57.387 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.388 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:57.388 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.388 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.389 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.389 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.5.13:36 2.0 2:20e40cec:::smithi12313934-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 113152~1699840] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:12:57.389 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:12:57.389 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 19.542958222999999, 2020-01-16T00:12:57.389 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011736636, 2020-01-16T00:12:57.390 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:57.390 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:57.390 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:57.390 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.390 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:57.391 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:12:57.391 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:57.391 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.391 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.391 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:57.392 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:12:57.392 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999952 2020-01-16T00:12:57.392 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.392 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.392 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:57.393 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831934+0000", 2020-01-16T00:12:57.393 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001034325 2020-01-16T00:12:57.393 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.393 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.393 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:57.394 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.832968+0000", 2020-01-16T00:12:57.394 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.506e-05 2020-01-16T00:12:57.394 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.394 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.395 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:57.395 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833004+0000", 2020-01-16T00:12:57.395 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3406e-05 2020-01-16T00:12:57.395 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.396 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.396 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:57.396 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833017+0000", 2020-01-16T00:12:57.396 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015477000000000001 2020-01-16T00:12:57.396 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.396 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.396 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:57.397 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833172+0000", 2020-01-16T00:12:57.397 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033750700000000002 2020-01-16T00:12:57.397 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.397 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.397 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:57.398 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833509+0000", 2020-01-16T00:12:57.398 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00055861400000000003 2020-01-16T00:12:57.398 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.398 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.398 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:12:57.399 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.834068+0000", 2020-01-16T00:12:57.399 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.007474545 2020-01-16T00:12:57.399 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.399 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.399 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:57.400 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.841542+0000", 2020-01-16T00:12:57.400 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00056319499999999995 2020-01-16T00:12:57.400 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.400 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.400 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:57.400 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.842106+0000", 2020-01-16T00:12:57.400 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001371017 2020-01-16T00:12:57.400 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.401 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.401 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:57.401 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843477+0000", 2020-01-16T00:12:57.401 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000123938 2020-01-16T00:12:57.401 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.401 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.401 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:57.402 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843601+0000", 2020-01-16T00:12:57.402 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5012000000000006e-05 2020-01-16T00:12:57.402 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.402 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.402 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:57.402 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843676+0000", 2020-01-16T00:12:57.402 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011736636 2020-01-16T00:12:57.402 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.403 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:57.403 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.403 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.403 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.403 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.5.13:40 2.0 2:20e40cec:::smithi12313934-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 113152~1699840] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:12:57.404 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.847376+0000", 2020-01-16T00:12:57.404 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 19.527521494999998, 2020-01-16T00:12:57.404 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010871093, 2020-01-16T00:12:57.404 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:57.404 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:57.404 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:57.405 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.405 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:57.405 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.847376+0000", 2020-01-16T00:12:57.405 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:57.405 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.405 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.405 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:57.406 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.847376+0000", 2020-01-16T00:12:57.406 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:12:57.406 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.406 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.406 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:57.406 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.847373+0000", 2020-01-16T00:12:57.406 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00077394699999999998 2020-01-16T00:12:57.406 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.407 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.407 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:57.407 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.848147+0000", 2020-01-16T00:12:57.407 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.173e-05 2020-01-16T00:12:57.407 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.407 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.407 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:57.408 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.848168+0000", 2020-01-16T00:12:57.408 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2738999999999999e-05 2020-01-16T00:12:57.408 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.408 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.408 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:57.408 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.848181+0000", 2020-01-16T00:12:57.409 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4627999999999999e-05 2020-01-16T00:12:57.409 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.409 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.409 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:57.409 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.848236+0000", 2020-01-16T00:12:57.409 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000141311 2020-01-16T00:12:57.409 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.410 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.410 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:57.410 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.848377+0000", 2020-01-16T00:12:57.410 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015462700000000001 2020-01-16T00:12:57.410 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.410 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.410 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:12:57.410 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.848532+0000", 2020-01-16T00:12:57.411 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0039933540000000002 2020-01-16T00:12:57.411 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.411 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.411 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:57.411 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.852525+0000", 2020-01-16T00:12:57.411 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0051016129999999996 2020-01-16T00:12:57.411 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.411 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.412 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:57.412 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.857627+0000", 2020-01-16T00:12:57.412 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00044689099999999999 2020-01-16T00:12:57.412 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.412 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.412 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:57.412 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.858073+0000", 2020-01-16T00:12:57.413 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8947e-05 2020-01-16T00:12:57.413 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.413 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.413 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:57.413 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.858172+0000", 2020-01-16T00:12:57.413 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4369999999999994e-05 2020-01-16T00:12:57.413 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.413 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.414 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:57.414 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.858247+0000", 2020-01-16T00:12:57.414 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010871093 2020-01-16T00:12:57.414 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.414 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:57.414 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.414 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.415 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.415 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.5.13:42 2.0 2:313458df:::smithi12313934-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 263680~1668096] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:12:57.415 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.939100+0000", 2020-01-16T00:12:57.415 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 19.435797587, 2020-01-16T00:12:57.415 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010934255, 2020-01-16T00:12:57.415 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:57.415 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:57.415 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:57.416 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.416 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:57.416 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939100+0000", 2020-01-16T00:12:57.416 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:57.416 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.416 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.416 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:57.416 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939100+0000", 2020-01-16T00:12:57.417 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:57.417 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.417 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.417 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:57.417 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939097+0000", 2020-01-16T00:12:57.417 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00049374800000000002 2020-01-16T00:12:57.417 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.418 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.418 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:57.418 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939591+0000", 2020-01-16T00:12:57.418 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0840999999999999e-05 2020-01-16T00:12:57.418 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.418 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.418 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:57.418 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939612+0000", 2020-01-16T00:12:57.419 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4749999999999998e-06 2020-01-16T00:12:57.419 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.419 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.419 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:57.419 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939618+0000", 2020-01-16T00:12:57.420 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6371e-05 2020-01-16T00:12:57.420 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.420 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.420 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:57.420 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939674+0000", 2020-01-16T00:12:57.420 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2e-05 2020-01-16T00:12:57.420 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.421 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.421 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:57.421 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939766+0000", 2020-01-16T00:12:57.421 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020133999999999999 2020-01-16T00:12:57.421 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.421 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.421 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:12:57.421 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939968+0000", 2020-01-16T00:12:57.422 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0027942560000000002 2020-01-16T00:12:57.422 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.422 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.422 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:57.422 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.942762+0000", 2020-01-16T00:12:57.422 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.004998466 2020-01-16T00:12:57.422 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.423 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.423 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:57.423 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.947760+0000", 2020-01-16T00:12:57.423 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0021783330000000002 2020-01-16T00:12:57.423 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.423 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.423 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:57.423 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.949939+0000", 2020-01-16T00:12:57.424 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2391999999999997e-05 2020-01-16T00:12:57.424 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.424 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.424 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:57.424 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.949981+0000", 2020-01-16T00:12:57.424 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2629e-05 2020-01-16T00:12:57.424 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.424 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.425 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:57.425 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.950034+0000", 2020-01-16T00:12:57.425 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010934255 2020-01-16T00:12:57.425 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.425 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:57.425 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.425 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.425 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.426 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.5.13:43 2.0 2:313458df:::smithi12313934-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 263680~1668096] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:12:57.426 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.953019+0000", 2020-01-16T00:12:57.426 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 19.421877791, 2020-01-16T00:12:57.426 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0085200620000000001, 2020-01-16T00:12:57.426 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:57.426 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:57.426 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:57.427 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.427 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:57.427 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953019+0000", 2020-01-16T00:12:57.427 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:57.427 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.427 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.427 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:57.428 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953019+0000", 2020-01-16T00:12:57.428 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:12:57.428 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.428 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.428 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:57.428 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953016+0000", 2020-01-16T00:12:57.428 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00058927999999999997 2020-01-16T00:12:57.429 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.429 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.429 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:57.430 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953606+0000", 2020-01-16T00:12:57.430 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0846999999999999e-05 2020-01-16T00:12:57.430 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.430 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.430 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:57.430 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953627+0000", 2020-01-16T00:12:57.431 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7020000000000004e-06 2020-01-16T00:12:57.431 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.431 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.431 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:57.432 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953633+0000", 2020-01-16T00:12:57.432 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.2762000000000005e-05 2020-01-16T00:12:57.432 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.432 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.432 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:57.433 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953716+0000", 2020-01-16T00:12:57.433 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000139238 2020-01-16T00:12:57.433 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.433 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.433 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:57.434 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953855+0000", 2020-01-16T00:12:57.434 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020386300000000001 2020-01-16T00:12:57.434 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.434 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.435 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:12:57.435 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.954059+0000", 2020-01-16T00:12:57.435 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0031229399999999998 2020-01-16T00:12:57.435 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.435 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.436 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:57.436 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.957182+0000", 2020-01-16T00:12:57.436 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0034781980000000001 2020-01-16T00:12:57.436 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.437 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.437 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:57.437 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.960660+0000", 2020-01-16T00:12:57.438 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00076596200000000002 2020-01-16T00:12:57.438 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.439 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.446 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:57.446 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.961426+0000", 2020-01-16T00:12:57.447 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1937e-05 2020-01-16T00:12:57.447 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.447 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.448 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:57.448 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.961488+0000", 2020-01-16T00:12:57.449 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1285000000000001e-05 2020-01-16T00:12:57.449 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.449 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.450 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:57.451 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.961539+0000", 2020-01-16T00:12:57.451 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0085200620000000001 2020-01-16T00:12:57.451 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.452 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:57.452 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.452 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.453 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.453 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1186 3.d 3:b1521a3e:::smithi12313934-25:head [write 682384~456830,stat] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:12:57.454 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.258156+0000", 2020-01-16T00:12:57.454 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 15.116741622999999, 2020-01-16T00:12:57.454 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0092969679999999992, 2020-01-16T00:12:57.455 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:57.455 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:57.455 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:57.456 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:57.456 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:57.456 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1186 2020-01-16T00:12:57.457 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.457 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:57.457 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.458 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:57.459 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.258156+0000", 2020-01-16T00:12:57.459 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:57.459 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.459 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.460 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:57.460 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.258156+0000", 2020-01-16T00:12:57.460 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999943 2020-01-16T00:12:57.460 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.461 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.461 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:57.461 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.258150+0000", 2020-01-16T00:12:57.461 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001866517 2020-01-16T00:12:57.462 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.462 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.462 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:57.462 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.260016+0000", 2020-01-16T00:12:57.462 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4024999999999999e-05 2020-01-16T00:12:57.463 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.463 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.463 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:57.463 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.260070+0000", 2020-01-16T00:12:57.464 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4295e-05 2020-01-16T00:12:57.464 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.464 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.464 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:57.464 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.260085+0000", 2020-01-16T00:12:57.464 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013625599999999999 2020-01-16T00:12:57.465 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.465 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.465 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:57.465 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.260221+0000", 2020-01-16T00:12:57.465 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000230015 2020-01-16T00:12:57.465 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.466 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.466 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:57.466 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.260451+0000", 2020-01-16T00:12:57.466 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00031965999999999999 2020-01-16T00:12:57.466 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.466 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.467 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:12:57.467 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.260770+0000", 2020-01-16T00:12:57.467 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0031788649999999999 2020-01-16T00:12:57.467 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.467 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.468 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:57.468 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.263949+0000", 2020-01-16T00:12:57.468 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0022620090000000002 2020-01-16T00:12:57.468 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.468 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.468 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:57.468 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.266211+0000", 2020-01-16T00:12:57.468 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00097717199999999989 2020-01-16T00:12:57.469 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.469 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.469 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:57.469 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.267188+0000", 2020-01-16T00:12:57.469 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000128594 2020-01-16T00:12:57.469 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.469 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.470 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:57.470 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.267317+0000", 2020-01-16T00:12:57.470 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013541200000000001 2020-01-16T00:12:57.470 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.470 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.471 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:57.471 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.267452+0000", 2020-01-16T00:12:57.471 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0092969679999999992 2020-01-16T00:12:57.471 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.471 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:57.471 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.472 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.472 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.472 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1187 3.d 3:b1521a3e:::smithi12313934-25:head [write 1721042~423699] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:12:57.472 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.263828+0000", 2020-01-16T00:12:57.472 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 15.111069151000001, 2020-01-16T00:12:57.472 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011305709000000001, 2020-01-16T00:12:57.472 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:57.472 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:57.473 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:57.473 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:57.473 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:57.473 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1187 2020-01-16T00:12:57.473 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.473 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:57.473 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.474 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:57.474 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.263828+0000", 2020-01-16T00:12:57.474 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:57.474 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.474 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.474 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:57.474 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.263828+0000", 2020-01-16T00:12:57.474 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:12:57.475 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.475 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.475 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:57.475 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.263824+0000", 2020-01-16T00:12:57.475 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00048485899999999997 2020-01-16T00:12:57.475 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.475 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.476 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:57.476 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264309+0000", 2020-01-16T00:12:57.476 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5622999999999999e-05 2020-01-16T00:12:57.476 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.476 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.476 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:57.476 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264325+0000", 2020-01-16T00:12:57.476 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2530000000000004e-06 2020-01-16T00:12:57.477 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.477 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.477 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:57.477 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264330+0000", 2020-01-16T00:12:57.477 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000102252 2020-01-16T00:12:57.477 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.477 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.478 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:57.478 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264432+0000", 2020-01-16T00:12:57.478 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022361000000000001 2020-01-16T00:12:57.478 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.478 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.478 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:57.478 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264656+0000", 2020-01-16T00:12:57.479 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0028688849999999998 2020-01-16T00:12:57.479 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.479 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.479 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:57.479 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.267525+0000", 2020-01-16T00:12:57.479 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00023640900000000001 2020-01-16T00:12:57.479 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.479 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.480 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:57.480 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.267761+0000", 2020-01-16T00:12:57.480 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00035667000000000001 2020-01-16T00:12:57.480 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.480 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.480 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:12:57.480 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.268118+0000", 2020-01-16T00:12:57.481 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023898040000000001 2020-01-16T00:12:57.481 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.481 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.481 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:57.481 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.270507+0000", 2020-01-16T00:12:57.481 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014919690000000001 2020-01-16T00:12:57.482 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.482 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.482 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:57.482 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.271999+0000", 2020-01-16T00:12:57.482 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0029764510000000002 2020-01-16T00:12:57.482 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.482 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.483 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:57.483 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.274976+0000", 2020-01-16T00:12:57.483 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5409999999999997e-05 2020-01-16T00:12:57.483 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.483 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.483 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:57.483 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.275061+0000", 2020-01-16T00:12:57.483 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2452000000000004e-05 2020-01-16T00:12:57.484 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.484 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.484 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:57.484 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.275134+0000", 2020-01-16T00:12:57.484 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011305709000000001 2020-01-16T00:12:57.484 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.484 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:57.485 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.485 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.485 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.485 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1226 3.d 3:b76659c4:::smithi12313934-47:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:12:57.485 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.386627+0000", 2020-01-16T00:12:57.485 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.988269977, 2020-01-16T00:12:57.485 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0076727160000000004, 2020-01-16T00:12:57.485 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:57.486 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:57.486 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:57.486 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:57.486 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:57.486 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1226 2020-01-16T00:12:57.487 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.487 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:57.487 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.487 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:57.488 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.386627+0000", 2020-01-16T00:12:57.488 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:57.488 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.488 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.488 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:57.489 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.386627+0000", 2020-01-16T00:12:57.489 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:57.489 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.489 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.490 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:57.490 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.386627+0000", 2020-01-16T00:12:57.490 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0120000000000001e-06 2020-01-16T00:12:57.490 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.491 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.491 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:57.491 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.386630+0000", 2020-01-16T00:12:57.491 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3579999999999996e-06 2020-01-16T00:12:57.491 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.491 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.492 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:57.492 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.386634+0000", 2020-01-16T00:12:57.492 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3739999999999998e-06 2020-01-16T00:12:57.492 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.492 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.492 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:57.492 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.386637+0000", 2020-01-16T00:12:57.493 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0067046830000000003 2020-01-16T00:12:57.493 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.493 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.493 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:57.494 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.393342+0000", 2020-01-16T00:12:57.494 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00023329199999999999 2020-01-16T00:12:57.494 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.494 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.494 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:57.495 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.393575+0000", 2020-01-16T00:12:57.495 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00072454099999999999 2020-01-16T00:12:57.495 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.495 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.496 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:57.496 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.394300+0000", 2020-01-16T00:12:57.496 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0076727160000000004 2020-01-16T00:12:57.496 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.497 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:57.497 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.497 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.497 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.497 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1259 3.d 3:b374ba21:::smithi12313934-20:head [read 0~3932805,checksum] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:12:57.498 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.833857+0000", 2020-01-16T00:12:57.498 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.541040407000001, 2020-01-16T00:12:57.498 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0094623380000000007, 2020-01-16T00:12:57.498 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:57.499 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:57.499 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:57.499 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:57.500 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:57.500 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1259 2020-01-16T00:12:57.500 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.501 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:57.501 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.501 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:57.502 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.833857+0000", 2020-01-16T00:12:57.502 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:57.502 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.502 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.502 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:57.502 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.833857+0000", 2020-01-16T00:12:57.503 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:57.503 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.503 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.503 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:57.503 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.833856+0000", 2020-01-16T00:12:57.503 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.9659999999999999e-06 2020-01-16T00:12:57.503 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.504 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.504 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:57.504 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.833863+0000", 2020-01-16T00:12:57.504 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4699999999999996e-06 2020-01-16T00:12:57.504 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.505 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.505 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:57.505 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.833870+0000", 2020-01-16T00:12:57.505 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0039999999999999e-06 2020-01-16T00:12:57.505 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.505 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.505 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:57.506 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.833875+0000", 2020-01-16T00:12:57.506 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0069094480000000003 2020-01-16T00:12:57.506 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.506 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.506 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:57.506 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.840785+0000", 2020-01-16T00:12:57.506 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020800699999999999 2020-01-16T00:12:57.506 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.507 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.507 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:57.507 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.840993+0000", 2020-01-16T00:12:57.507 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023263960000000001 2020-01-16T00:12:57.507 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.507 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.507 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:57.508 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.843319+0000", 2020-01-16T00:12:57.508 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0094623380000000007 2020-01-16T00:12:57.508 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.508 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:57.508 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.508 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.508 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.508 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1272 3.d 3:b3dd5d45:::smithi12313934-16:head [read 0~3008420,checksum] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:12:57.509 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074658+0000", 2020-01-16T00:12:57.509 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.300239567, 2020-01-16T00:12:57.509 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0083000039999999997, 2020-01-16T00:12:57.509 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:57.509 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:57.509 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:57.509 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:57.509 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:57.510 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1272 2020-01-16T00:12:57.510 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.510 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:57.510 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.510 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:57.510 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074658+0000", 2020-01-16T00:12:57.511 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:57.511 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.511 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.511 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:57.512 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074658+0000", 2020-01-16T00:12:57.512 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:57.512 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.512 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.512 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:57.512 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074657+0000", 2020-01-16T00:12:57.513 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7270000000000007e-06 2020-01-16T00:12:57.513 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.513 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.513 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:57.513 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074664+0000", 2020-01-16T00:12:57.513 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1094000000000002e-05 2020-01-16T00:12:57.513 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.513 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.514 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:57.514 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074685+0000", 2020-01-16T00:12:57.514 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0260000000000004e-06 2020-01-16T00:12:57.514 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.514 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.514 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:57.514 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074691+0000", 2020-01-16T00:12:57.515 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0061756880000000004 2020-01-16T00:12:57.515 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.515 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.515 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:57.515 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.080867+0000", 2020-01-16T00:12:57.515 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015871700000000001 2020-01-16T00:12:57.515 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.515 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.516 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:57.516 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.081026+0000", 2020-01-16T00:12:57.516 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0019317519999999999 2020-01-16T00:12:57.516 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.516 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.516 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:57.517 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.082958+0000", 2020-01-16T00:12:57.517 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0083000039999999997 2020-01-16T00:12:57.517 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.517 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:57.517 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.517 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.517 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.518 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1276 3.d 3:b1521a3e:::smithi12313934-25:head [read 0~3882384,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:12:57.518 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:12:57.518 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.300198653000001, 2020-01-16T00:12:57.518 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015839157999999999, 2020-01-16T00:12:57.518 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:57.518 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:57.518 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:57.519 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:57.519 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:57.519 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1276 2020-01-16T00:12:57.519 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.519 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:57.519 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.519 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:57.519 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:12:57.520 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:57.520 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.520 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.520 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:57.520 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:12:57.520 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:57.520 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.520 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.521 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:57.521 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:12:57.521 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5429999999999999e-06 2020-01-16T00:12:57.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.521 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.521 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:57.521 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074704+0000", 2020-01-16T00:12:57.522 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0839999999999992e-06 2020-01-16T00:12:57.522 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.522 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.522 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:57.522 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074712+0000", 2020-01-16T00:12:57.523 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.173e-06 2020-01-16T00:12:57.523 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.523 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.523 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:57.523 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074718+0000", 2020-01-16T00:12:57.523 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0083034700000000003 2020-01-16T00:12:57.523 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.523 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.524 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:57.524 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.083022+0000", 2020-01-16T00:12:57.524 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028456400000000001 2020-01-16T00:12:57.524 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.524 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.524 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:57.524 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.083306+0000", 2020-01-16T00:12:57.525 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0072312640000000003 2020-01-16T00:12:57.525 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.525 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.525 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:57.525 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090538+0000", 2020-01-16T00:12:57.525 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015839157999999999 2020-01-16T00:12:57.525 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.525 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:57.526 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.526 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.526 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.526 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1277 3.d 3:b1521a3e:::smithi12313934-25:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:12:57.526 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:12:57.526 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.300155063, 2020-01-16T00:12:57.527 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016646459999999998, 2020-01-16T00:12:57.527 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:57.527 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:57.527 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:57.527 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:57.528 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:57.528 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1277 2020-01-16T00:12:57.528 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.528 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:57.528 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.528 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:57.529 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:12:57.529 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:57.529 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.529 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.529 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:57.529 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:12:57.529 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:57.530 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.530 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.530 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:57.530 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074741+0000", 2020-01-16T00:12:57.530 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0776e-05 2020-01-16T00:12:57.530 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.530 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.531 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:57.531 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074752+0000", 2020-01-16T00:12:57.531 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7600000000000002e-06 2020-01-16T00:12:57.531 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.531 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.531 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:57.531 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074759+0000", 2020-01-16T00:12:57.531 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6840000000000001e-06 2020-01-16T00:12:57.532 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.532 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.532 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:57.532 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074765+0000", 2020-01-16T00:12:57.532 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015815216 2020-01-16T00:12:57.532 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.533 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.533 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:57.533 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090580+0000", 2020-01-16T00:12:57.533 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000182404 2020-01-16T00:12:57.533 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.533 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.533 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:57.534 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090763+0000", 2020-01-16T00:12:57.534 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00062579300000000003 2020-01-16T00:12:57.534 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.534 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.534 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:57.534 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091389+0000", 2020-01-16T00:12:57.534 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016646459999999998 2020-01-16T00:12:57.534 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.535 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:57.535 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.535 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.535 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.535 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1278 3.d 3:b1521a3e:::smithi12313934-25:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:12:57.535 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:12:57.535 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.30012535, 2020-01-16T00:12:57.535 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017994915, 2020-01-16T00:12:57.536 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:57.536 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:57.536 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:57.536 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:57.536 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:57.536 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1278 2020-01-16T00:12:57.536 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.537 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:57.537 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.537 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:57.537 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:12:57.537 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:57.537 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.537 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.537 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:57.538 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:12:57.538 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:57.538 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.538 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.539 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:57.539 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074771+0000", 2020-01-16T00:12:57.539 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4569999999999998e-06 2020-01-16T00:12:57.539 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.539 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.540 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:57.540 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074775+0000", 2020-01-16T00:12:57.540 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0330000000000003e-06 2020-01-16T00:12:57.540 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.540 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.540 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:57.540 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074782+0000", 2020-01-16T00:12:57.540 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9470000000000002e-06 2020-01-16T00:12:57.541 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.541 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.541 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:57.541 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074787+0000", 2020-01-16T00:12:57.541 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01666134 2020-01-16T00:12:57.541 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.541 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.542 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:57.542 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091449+0000", 2020-01-16T00:12:57.542 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000247731 2020-01-16T00:12:57.542 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.542 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.543 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:57.543 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091696+0000", 2020-01-16T00:12:57.543 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010703170000000001 2020-01-16T00:12:57.543 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.543 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.543 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:57.543 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.092767+0000", 2020-01-16T00:12:57.544 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017994915 2020-01-16T00:12:57.544 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.544 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:57.544 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.544 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.544 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.544 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1419 3.e 3:7cfd7007:::smithi12313934-21:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e28)", 2020-01-16T00:12:57.544 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:47.689893+0000", 2020-01-16T00:12:57.545 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 9.6850044880000006, 2020-01-16T00:12:57.545 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0074137259999999998, 2020-01-16T00:12:57.545 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:57.545 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:57.545 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:57.545 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:57.545 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:57.546 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1419 2020-01-16T00:12:57.546 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.546 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:57.546 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.546 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:57.546 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.689893+0000", 2020-01-16T00:12:57.546 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:57.546 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.547 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.547 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:57.547 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.689893+0000", 2020-01-16T00:12:57.547 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:57.547 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.547 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.547 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:57.548 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.689891+0000", 2020-01-16T00:12:57.548 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.8100000000000004e-06 2020-01-16T00:12:57.548 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.548 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.548 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:57.548 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.689900+0000", 2020-01-16T00:12:57.548 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.509e-05 2020-01-16T00:12:57.548 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.549 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.549 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:57.549 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.689915+0000", 2020-01-16T00:12:57.549 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.73e-06 2020-01-16T00:12:57.549 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.549 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.549 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:57.549 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.689925+0000", 2020-01-16T00:12:57.550 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.006916312 2020-01-16T00:12:57.550 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.550 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.550 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:57.550 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.696841+0000", 2020-01-16T00:12:57.550 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011041599999999999 2020-01-16T00:12:57.550 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.551 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.551 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:57.551 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.696951+0000", 2020-01-16T00:12:57.551 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00035496099999999999 2020-01-16T00:12:57.551 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.551 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.551 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:57.551 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.697306+0000", 2020-01-16T00:12:57.552 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0074137259999999998 2020-01-16T00:12:57.552 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.552 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:57.552 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.552 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.552 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.553 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1420 3.e 3:7cfd7007:::smithi12313934-21:head [read 0~3899191,checksum] snapc 0=[] ondisk+read+known_if_redirected e28)", 2020-01-16T00:12:57.553 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:47.689967+0000", 2020-01-16T00:12:57.553 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 9.6849300649999996, 2020-01-16T00:12:57.553 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010893273, 2020-01-16T00:12:57.553 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:57.553 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:57.553 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:57.554 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:57.554 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:57.554 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1420 2020-01-16T00:12:57.554 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.554 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:57.554 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.554 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:57.554 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.689967+0000", 2020-01-16T00:12:57.555 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:57.555 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.555 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.555 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:57.555 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.689967+0000", 2020-01-16T00:12:57.555 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:57.555 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.555 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.556 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:57.556 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.689965+0000", 2020-01-16T00:12:57.556 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.696e-05 2020-01-16T00:12:57.556 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.556 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.556 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:57.556 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.689982+0000", 2020-01-16T00:12:57.557 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9613e-05 2020-01-16T00:12:57.557 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.557 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.557 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:57.557 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.690002+0000", 2020-01-16T00:12:57.557 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3003000000000001e-05 2020-01-16T00:12:57.558 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.558 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.558 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:57.558 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.690015+0000", 2020-01-16T00:12:57.558 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0073546410000000003 2020-01-16T00:12:57.558 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.558 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.559 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:57.559 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.697370+0000", 2020-01-16T00:12:57.559 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00031806899999999998 2020-01-16T00:12:57.559 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.559 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.559 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:57.559 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.697688+0000", 2020-01-16T00:12:57.559 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003172673 2020-01-16T00:12:57.560 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.560 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.560 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:57.560 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.700860+0000", 2020-01-16T00:12:57.560 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010893273 2020-01-16T00:12:57.560 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.560 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:57.560 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.561 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.561 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.561 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.4.13:40 2.0 2:01b4bead:::smithi12313934-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 640512~403968] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e28)", 2020-01-16T00:12:57.561 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:47.904891+0000", 2020-01-16T00:12:57.561 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 9.4700064170000005, 2020-01-16T00:12:57.561 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0078047129999999996, 2020-01-16T00:12:57.561 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:57.562 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:57.562 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:57.562 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.562 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:57.562 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.904891+0000", 2020-01-16T00:12:57.562 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:57.562 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.562 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.563 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:57.563 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.904891+0000", 2020-01-16T00:12:57.563 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:57.563 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.563 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.563 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:57.564 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.904889+0000", 2020-01-16T00:12:57.564 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017825 2020-01-16T00:12:57.564 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.564 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.564 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:57.564 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.905067+0000", 2020-01-16T00:12:57.564 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4921e-05 2020-01-16T00:12:57.564 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.565 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.565 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:57.565 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.905082+0000", 2020-01-16T00:12:57.565 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5339999999999999e-06 2020-01-16T00:12:57.565 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.565 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.566 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:57.566 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.905087+0000", 2020-01-16T00:12:57.566 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011594 2020-01-16T00:12:57.566 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.566 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.567 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:57.567 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.905203+0000", 2020-01-16T00:12:57.567 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00026064000000000003 2020-01-16T00:12:57.567 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.568 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.568 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:57.568 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.905464+0000", 2020-01-16T00:12:57.568 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00044418700000000002 2020-01-16T00:12:57.568 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.568 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.568 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:12:57.568 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.905908+0000", 2020-01-16T00:12:57.569 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0055870989999999999 2020-01-16T00:12:57.569 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.569 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.569 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:57.569 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.911495+0000", 2020-01-16T00:12:57.569 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3438e-05 2020-01-16T00:12:57.569 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.570 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.570 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:57.570 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.911539+0000", 2020-01-16T00:12:57.570 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001073112 2020-01-16T00:12:57.570 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.570 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.570 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:57.570 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.912612+0000", 2020-01-16T00:12:57.571 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2938e-05 2020-01-16T00:12:57.571 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.571 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.571 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:57.571 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.912665+0000", 2020-01-16T00:12:57.571 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0811e-05 2020-01-16T00:12:57.571 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.571 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.572 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:57.572 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.912695+0000", 2020-01-16T00:12:57.572 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0078047129999999996 2020-01-16T00:12:57.572 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.572 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:57.572 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.572 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.572 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.573 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.4.13:41 2.0 2:01b4bead:::smithi12313934-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 1445888~278528] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e28)", 2020-01-16T00:12:57.573 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:47.905100+0000", 2020-01-16T00:12:57.573 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 9.4697972139999997, 2020-01-16T00:12:57.573 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0079266059999999992, 2020-01-16T00:12:57.573 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:57.574 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:57.574 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:57.574 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.574 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:57.574 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.905100+0000", 2020-01-16T00:12:57.574 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:57.574 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.574 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.575 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:57.575 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.905100+0000", 2020-01-16T00:12:57.575 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:57.575 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.575 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.575 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:57.575 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.905099+0000", 2020-01-16T00:12:57.576 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3053000000000005e-05 2020-01-16T00:12:57.576 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.576 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.576 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:57.576 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.905192+0000", 2020-01-16T00:12:57.576 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2110000000000003e-06 2020-01-16T00:12:57.576 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.576 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.577 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:57.577 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.905198+0000", 2020-01-16T00:12:57.577 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4429999999999999e-06 2020-01-16T00:12:57.577 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.577 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.577 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:57.577 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.905202+0000", 2020-01-16T00:12:57.578 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003197858 2020-01-16T00:12:57.578 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.578 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.578 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:57.578 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.908400+0000", 2020-01-16T00:12:57.578 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012500200000000001 2020-01-16T00:12:57.578 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.578 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.579 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:57.579 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.908525+0000", 2020-01-16T00:12:57.579 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000183332 2020-01-16T00:12:57.579 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.579 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.579 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:12:57.579 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.908708+0000", 2020-01-16T00:12:57.580 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0027965339999999998 2020-01-16T00:12:57.580 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.580 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.580 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:57.580 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.911505+0000", 2020-01-16T00:12:57.580 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012356649999999999 2020-01-16T00:12:57.580 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.580 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.581 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:57.581 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.912740+0000", 2020-01-16T00:12:57.581 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022262900000000001 2020-01-16T00:12:57.581 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.581 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.581 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:57.582 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.912963+0000", 2020-01-16T00:12:57.582 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4484999999999999e-05 2020-01-16T00:12:57.582 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.582 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.582 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:57.583 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.912997+0000", 2020-01-16T00:12:57.583 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9139999999999999e-05 2020-01-16T00:12:57.583 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.583 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.583 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:57.583 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.913027+0000", 2020-01-16T00:12:57.584 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0079266059999999992 2020-01-16T00:12:57.584 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.584 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:57.584 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.584 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.584 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.584 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1870 3.d 3:bd9fc72f:::smithi12313934-5:head [write 2804576~435035] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:12:57.585 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:56.384430+0000", 2020-01-16T00:12:57.585 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.99046729, 2020-01-16T00:12:57.585 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0085577770000000008, 2020-01-16T00:12:57.585 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:57.585 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:57.586 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:57.586 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:57.586 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:57.586 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1870 2020-01-16T00:12:57.586 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.587 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:57.587 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.587 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:57.587 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384430+0000", 2020-01-16T00:12:57.587 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:57.588 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.588 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.588 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:57.588 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384430+0000", 2020-01-16T00:12:57.588 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999933 2020-01-16T00:12:57.589 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.589 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.589 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:57.589 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384423+0000", 2020-01-16T00:12:57.589 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042953000000000002 2020-01-16T00:12:57.590 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.590 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.590 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:57.590 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384853+0000", 2020-01-16T00:12:57.590 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2592000000000001e-05 2020-01-16T00:12:57.591 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.591 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.591 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:57.591 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384865+0000", 2020-01-16T00:12:57.591 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7559999999999999e-06 2020-01-16T00:12:57.592 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.592 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.592 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:57.592 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384870+0000", 2020-01-16T00:12:57.592 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8498000000000003e-05 2020-01-16T00:12:57.593 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.593 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.593 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:57.593 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384909+0000", 2020-01-16T00:12:57.594 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00075234 2020-01-16T00:12:57.594 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.594 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.594 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:57.594 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.385661+0000", 2020-01-16T00:12:57.595 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.004714963 2020-01-16T00:12:57.595 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.596 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.597 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:57.597 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.390376+0000", 2020-01-16T00:12:57.597 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000274324 2020-01-16T00:12:57.597 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.597 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.597 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:57.597 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.390650+0000", 2020-01-16T00:12:57.598 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024631000000000002 2020-01-16T00:12:57.598 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.598 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.598 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:12:57.598 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.390897+0000", 2020-01-16T00:12:57.598 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014908829999999999 2020-01-16T00:12:57.598 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.598 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.599 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:57.599 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.392387+0000", 2020-01-16T00:12:57.599 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00023496300000000001 2020-01-16T00:12:57.599 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.599 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.599 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:57.599 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.392622+0000", 2020-01-16T00:12:57.600 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00027025500000000001 2020-01-16T00:12:57.600 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.600 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.600 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:57.600 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.392893+0000", 2020-01-16T00:12:57.601 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0452000000000003e-05 2020-01-16T00:12:57.601 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.601 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.601 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:57.601 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.392953+0000", 2020-01-16T00:12:57.601 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4564999999999999e-05 2020-01-16T00:12:57.601 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.601 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.602 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:57.602 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.392988+0000", 2020-01-16T00:12:57.602 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0085577770000000008 2020-01-16T00:12:57.602 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.602 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:57.602 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.602 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.603 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.603 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1871 3.d 3:bd9fc72f:::smithi12313934-5:head [setxattr _header (55) in=7b,truncate 3327786] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:12:57.603 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:56.384879+0000", 2020-01-16T00:12:57.603 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.99001826999999998, 2020-01-16T00:12:57.603 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010769713, 2020-01-16T00:12:57.603 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:57.603 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:57.603 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:57.604 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:57.604 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:57.604 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1871 2020-01-16T00:12:57.604 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.604 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:57.604 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.605 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:57.605 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384879+0000", 2020-01-16T00:12:57.605 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:57.605 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.605 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.605 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:57.605 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384879+0000", 2020-01-16T00:12:57.605 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:57.606 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.606 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.606 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:57.606 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384878+0000", 2020-01-16T00:12:57.606 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9370000000000003e-06 2020-01-16T00:12:57.606 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.606 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.607 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:57.607 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384883+0000", 2020-01-16T00:12:57.607 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6589999999999996e-06 2020-01-16T00:12:57.607 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.607 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.607 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:57.607 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384891+0000", 2020-01-16T00:12:57.607 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9330000000000001e-06 2020-01-16T00:12:57.608 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.608 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.608 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:57.608 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384894+0000", 2020-01-16T00:12:57.608 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00080669800000000001 2020-01-16T00:12:57.608 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.608 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.609 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:57.609 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.385701+0000", 2020-01-16T00:12:57.609 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3726000000000002e-05 2020-01-16T00:12:57.609 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.609 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.609 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:57.609 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.385735+0000", 2020-01-16T00:12:57.609 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0059567650000000002 2020-01-16T00:12:57.610 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.610 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.610 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:57.610 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.391692+0000", 2020-01-16T00:12:57.610 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013010300000000001 2020-01-16T00:12:57.610 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.610 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.611 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:57.611 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.391822+0000", 2020-01-16T00:12:57.611 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016664800000000001 2020-01-16T00:12:57.611 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.611 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.611 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:12:57.611 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.391988+0000", 2020-01-16T00:12:57.611 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00052670799999999997 2020-01-16T00:12:57.612 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.612 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.612 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:57.612 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.392515+0000", 2020-01-16T00:12:57.612 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00052796600000000005 2020-01-16T00:12:57.612 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.612 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.612 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:57.613 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.393043+0000", 2020-01-16T00:12:57.613 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002493501 2020-01-16T00:12:57.613 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.613 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.613 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:57.613 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.395537+0000", 2020-01-16T00:12:57.613 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9033999999999998e-05 2020-01-16T00:12:57.613 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.614 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.614 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:57.614 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.395596+0000", 2020-01-16T00:12:57.614 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3052e-05 2020-01-16T00:12:57.614 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.614 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.615 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:57.615 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.395649+0000", 2020-01-16T00:12:57.615 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010769713 2020-01-16T00:12:57.615 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.615 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:57.615 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.615 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.616 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.616 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1872 3.d 3:bd9fc72f:::smithi12313934-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e31)", 2020-01-16T00:12:57.616 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:56.384904+0000", 2020-01-16T00:12:57.616 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.98999267099999999, 2020-01-16T00:12:57.616 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011110386999999999, 2020-01-16T00:12:57.616 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:57.616 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:57.616 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:57.617 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:57.617 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:57.617 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1872 2020-01-16T00:12:57.617 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.617 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:57.617 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.617 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:57.617 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384904+0000", 2020-01-16T00:12:57.618 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:57.618 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.618 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.618 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:57.618 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384904+0000", 2020-01-16T00:12:57.618 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:12:57.618 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.619 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.619 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:57.619 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384904+0000", 2020-01-16T00:12:57.619 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3879999999999998e-06 2020-01-16T00:12:57.619 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.619 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.619 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:57.619 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384906+0000", 2020-01-16T00:12:57.620 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7479999999999999e-06 2020-01-16T00:12:57.620 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.620 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.620 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:57.620 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384910+0000", 2020-01-16T00:12:57.620 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4609999999999998e-06 2020-01-16T00:12:57.620 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.620 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.621 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:57.621 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384913+0000", 2020-01-16T00:12:57.621 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00083872899999999997 2020-01-16T00:12:57.621 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.621 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.621 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:57.621 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.385751+0000", 2020-01-16T00:12:57.622 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3528999999999997e-05 2020-01-16T00:12:57.622 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.622 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.622 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:57.622 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.385785+0000", 2020-01-16T00:12:57.622 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0064799849999999997 2020-01-16T00:12:57.622 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.622 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.623 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:57.623 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.392265+0000", 2020-01-16T00:12:57.623 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010603399999999999 2020-01-16T00:12:57.623 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.623 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.623 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:57.623 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.392371+0000", 2020-01-16T00:12:57.624 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003330576 2020-01-16T00:12:57.624 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.624 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.624 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:57.624 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.395702+0000", 2020-01-16T00:12:57.625 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016566899999999999 2020-01-16T00:12:57.625 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.625 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.625 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:57.625 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.395867+0000", 2020-01-16T00:12:57.625 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014768999999999999 2020-01-16T00:12:57.625 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:57.625 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:57.626 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:57.626 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.396015+0000", 2020-01-16T00:12:57.626 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011110386999999999 2020-01-16T00:12:57.626 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.626 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:57.626 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.626 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:57.627 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:57.627 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:57.627 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4, 7, 6] 2020-01-16T00:12:57.627 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2020-01-16T00:12:57.627 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_0 2020-01-16T00:12:57.627 INFO:tasks.ceph.ceph_manager.ceph:contract_pool unique_pool_0 by 10 min 8 2020-01-16T00:12:57.628 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-01-16T00:12:57.630 INFO:tasks.rados.rados.0.smithi123.stdout:910: finishing write tid 1 to smithi12313934-49 2020-01-16T00:12:57.630 INFO:tasks.rados.rados.0.smithi123.stdout:910: finishing write tid 2 to smithi12313934-49 2020-01-16T00:12:57.630 INFO:tasks.rados.rados.0.smithi123.stdout:910: finishing write tid 3 to smithi12313934-49 2020-01-16T00:12:57.630 INFO:tasks.rados.rados.0.smithi123.stdout:910: finishing write tid 5 to smithi12313934-49 2020-01-16T00:12:57.630 INFO:tasks.rados.rados.0.smithi123.stdout:910: finishing write tid 6 to smithi12313934-49 2020-01-16T00:12:57.631 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 49 v 85 (ObjNum 196 snap 0 seq_num 196) dirty exists 2020-01-16T00:12:57.631 INFO:tasks.rados.rados.0.smithi123.stdout:910: left oid 49 (ObjNum 196 snap 0 seq_num 196) 2020-01-16T00:12:57.631 INFO:tasks.rados.rados.0.smithi123.stdout:911: finishing write tid 1 to smithi12313934-18 2020-01-16T00:12:57.631 INFO:tasks.rados.rados.0.smithi123.stdout:911: finishing write tid 2 to smithi12313934-18 2020-01-16T00:12:57.631 INFO:tasks.rados.rados.0.smithi123.stdout:908: done (15 left) 2020-01-16T00:12:57.631 INFO:tasks.rados.rados.0.smithi123.stdout:909: done (14 left) 2020-01-16T00:12:57.631 INFO:tasks.rados.rados.0.smithi123.stdout:910: done (13 left) 2020-01-16T00:12:57.631 INFO:tasks.rados.rados.0.smithi123.stdout:924: write (excl) oid 10 current snap is 0 2020-01-16T00:12:57.632 INFO:tasks.rados.rados.0.smithi123.stdout:924: seq_num 208 ranges {695856=793323,2171784=534399,3362453=133403} 2020-01-16T00:12:57.632 INFO:tasks.rados.rados.0.smithi123.stdout:924: writing smithi12313934-10 from 695856 to 1489179 tid 1 2020-01-16T00:12:57.632 INFO:tasks.rados.rados.0.smithi123.stdout:924: writing smithi12313934-10 from 2171784 to 2706183 tid 2 2020-01-16T00:12:57.632 INFO:tasks.rados.rados.0.smithi123.stdout:924: writing smithi12313934-10 from 3362453 to 3495856 tid 3 2020-01-16T00:12:57.632 INFO:tasks.rados.rados.0.smithi123.stdout:925: read oid 32 snap -1 2020-01-16T00:12:57.632 INFO:tasks.rados.rados.0.smithi123.stdout:925: expect (ObjNum 143 snap 0 seq_num 143) 2020-01-16T00:12:57.632 INFO:tasks.rados.rados.0.smithi123.stdout:926: read oid 8 snap -1 2020-01-16T00:12:57.632 INFO:tasks.rados.rados.0.smithi123.stdout:926: expect (ObjNum 195 snap 0 seq_num 195) 2020-01-16T00:12:57.633 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:57.633 INFO:tasks.rados.rados.0.smithi123.stdout:911: finishing write tid 3 to smithi12313934-18 2020-01-16T00:12:57.633 INFO:tasks.rados.rados.0.smithi123.stdout:911: finishing write tid 5 to smithi12313934-18 2020-01-16T00:12:57.633 INFO:tasks.rados.rados.0.smithi123.stdout:911: finishing write tid 6 to smithi12313934-18 2020-01-16T00:12:57.633 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 18 v 171 (ObjNum 197 snap 0 seq_num 197) dirty exists 2020-01-16T00:12:57.633 INFO:tasks.rados.rados.0.smithi123.stdout:911: left oid 18 (ObjNum 197 snap 0 seq_num 197) 2020-01-16T00:12:57.633 INFO:tasks.rados.rados.0.smithi123.stdout:912: expect (ObjNum 113 snap 0 seq_num 113) 2020-01-16T00:12:57.634 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:57.629+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:12:57.634 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-01-16T00:12:57.646 INFO:tasks.rados.rados.0.smithi123.stdout:913: finishing write tid 1 to smithi12313934-5 2020-01-16T00:12:57.646 INFO:tasks.rados.rados.0.smithi123.stdout:913: finishing write tid 2 to smithi12313934-5 2020-01-16T00:12:57.647 INFO:tasks.rados.rados.0.smithi123.stdout:913: finishing write tid 3 to smithi12313934-5 2020-01-16T00:12:57.647 INFO:tasks.rados.rados.0.smithi123.stdout:913: finishing write tid 5 to smithi12313934-5 2020-01-16T00:12:57.647 INFO:tasks.rados.rados.0.smithi123.stdout:911: done (15 left) 2020-01-16T00:12:57.647 INFO:tasks.rados.rados.0.smithi123.stdout:912: done (14 left) 2020-01-16T00:12:57.648 INFO:tasks.rados.rados.0.smithi123.stdout:927: read oid 23 snap -1 2020-01-16T00:12:57.648 INFO:tasks.rados.rados.0.smithi123.stdout:927: expect (ObjNum 22 snap 0 seq_num 22) 2020-01-16T00:12:57.648 INFO:tasks.rados.rados.0.smithi123.stdout:928: write oid 43 current snap is 0 2020-01-16T00:12:57.648 INFO:tasks.rados.rados.0.smithi123.stdout:928: seq_num 209 ranges {712663=468572,1713913=678057,2873523=591733} 2020-01-16T00:12:57.652 INFO:tasks.rados.rados.0.smithi123.stdout:928: writing smithi12313934-43 from 712663 to 1181235 tid 1 2020-01-16T00:12:57.659 INFO:tasks.rados.rados.0.smithi123.stdout:928: writing smithi12313934-43 from 1713913 to 2391970 tid 2 2020-01-16T00:12:57.666 INFO:tasks.rados.rados.0.smithi123.stdout:928: writing smithi12313934-43 from 2873523 to 3465256 tid 3 2020-01-16T00:12:57.667 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:57.667 INFO:tasks.rados.rados.0.smithi123.stdout:913: finishing write tid 6 to smithi12313934-5 2020-01-16T00:12:57.667 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 5 v 122 (ObjNum 198 snap 0 seq_num 198) dirty exists 2020-01-16T00:12:57.667 INFO:tasks.rados.rados.0.smithi123.stdout:913: left oid 5 (ObjNum 198 snap 0 seq_num 198) 2020-01-16T00:12:57.667 INFO:tasks.rados.rados.0.smithi123.stdout:913: done (15 left) 2020-01-16T00:12:57.667 INFO:tasks.rados.rados.0.smithi123.stdout:929: read oid 11 snap -1 2020-01-16T00:12:57.667 INFO:tasks.rados.rados.0.smithi123.stdout:929: expect (ObjNum 191 snap 0 seq_num 191) 2020-01-16T00:12:57.668 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:57.668 INFO:tasks.rados.rados.0.smithi123.stdout:914: finishing write tid 1 to smithi12313934-20 2020-01-16T00:12:57.668 INFO:tasks.rados.rados.0.smithi123.stdout:914: finishing write tid 2 to smithi12313934-20 2020-01-16T00:12:57.668 INFO:tasks.rados.rados.0.smithi123.stdout:914: finishing write tid 3 to smithi12313934-20 2020-01-16T00:12:57.668 INFO:tasks.rados.rados.0.smithi123.stdout:914: finishing write tid 5 to smithi12313934-20 2020-01-16T00:12:57.668 INFO:tasks.rados.rados.0.smithi123.stdout:914: finishing write tid 6 to smithi12313934-20 2020-01-16T00:12:57.668 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 20 v 126 (ObjNum 199 snap 0 seq_num 199) dirty exists 2020-01-16T00:12:57.668 INFO:tasks.rados.rados.0.smithi123.stdout:914: left oid 20 (ObjNum 199 snap 0 seq_num 199) 2020-01-16T00:12:57.669 INFO:tasks.rados.rados.0.smithi123.stdout:914: done (15 left) 2020-01-16T00:12:57.669 INFO:tasks.rados.rados.0.smithi123.stdout:930: delete oid 37 current snap is 0 2020-01-16T00:12:57.669 INFO:tasks.rados.rados.0.smithi123.stdout:915: finishing write tid 1 to smithi12313934-29 2020-01-16T00:12:57.669 INFO:tasks.rados.rados.0.smithi123.stdout:915: finishing write tid 2 to smithi12313934-29 2020-01-16T00:12:57.669 INFO:tasks.rados.rados.0.smithi123.stdout:915: finishing write tid 3 to smithi12313934-29 2020-01-16T00:12:57.669 INFO:tasks.rados.rados.0.smithi123.stdout:915: finishing write tid 5 to smithi12313934-29 2020-01-16T00:12:57.669 INFO:tasks.rados.rados.0.smithi123.stdout:915: finishing write tid 6 to smithi12313934-29 2020-01-16T00:12:57.670 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 29 v 82 (ObjNum 200 snap 0 seq_num 200) dirty exists 2020-01-16T00:12:57.670 INFO:tasks.rados.rados.0.smithi123.stdout:915: left oid 29 (ObjNum 200 snap 0 seq_num 200) 2020-01-16T00:12:57.670 INFO:tasks.rados.rados.0.smithi123.stdout:916: finishing write tid 1 to smithi12313934-25 2020-01-16T00:12:57.670 INFO:tasks.rados.rados.0.smithi123.stdout:916: finishing write tid 2 to smithi12313934-25 2020-01-16T00:12:57.670 INFO:tasks.rados.rados.0.smithi123.stdout:916: finishing write tid 3 to smithi12313934-25 2020-01-16T00:12:57.670 INFO:tasks.rados.rados.0.smithi123.stdout:916: finishing write tid 5 to smithi12313934-25 2020-01-16T00:12:57.670 INFO:tasks.rados.rados.0.smithi123.stdout:916: finishing write tid 6 to smithi12313934-25 2020-01-16T00:12:57.670 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 25 v 130 (ObjNum 201 snap 0 seq_num 201) dirty exists 2020-01-16T00:12:57.671 INFO:tasks.rados.rados.0.smithi123.stdout:916: left oid 25 (ObjNum 201 snap 0 seq_num 201) 2020-01-16T00:12:57.671 INFO:tasks.rados.rados.0.smithi123.stdout:917: finishing write tid 1 to smithi12313934-33 2020-01-16T00:12:57.671 INFO:tasks.rados.rados.0.smithi123.stdout:917: finishing write tid 2 to smithi12313934-33 2020-01-16T00:12:57.671 INFO:tasks.rados.rados.0.smithi123.stdout:917: finishing write tid 3 to smithi12313934-33 2020-01-16T00:12:57.671 INFO:tasks.rados.rados.0.smithi123.stdout:918: finishing write tid 1 to smithi12313934-31 2020-01-16T00:12:57.671 INFO:tasks.rados.rados.0.smithi123.stdout:917: finishing write tid 5 to smithi12313934-33 2020-01-16T00:12:57.671 INFO:tasks.rados.rados.0.smithi123.stdout:917: finishing write tid 6 to smithi12313934-33 2020-01-16T00:12:57.671 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 33 v 177 (ObjNum 202 snap 0 seq_num 202) dirty exists 2020-01-16T00:12:57.672 INFO:tasks.rados.rados.0.smithi123.stdout:917: left oid 33 (ObjNum 202 snap 0 seq_num 202) 2020-01-16T00:12:57.672 INFO:tasks.rados.rados.0.smithi123.stdout:918: finishing write tid 2 to smithi12313934-31 2020-01-16T00:12:57.672 INFO:tasks.rados.rados.0.smithi123.stdout:918: finishing write tid 3 to smithi12313934-31 2020-01-16T00:12:57.672 INFO:tasks.rados.rados.0.smithi123.stdout:918: finishing write tid 5 to smithi12313934-31 2020-01-16T00:12:57.672 INFO:tasks.rados.rados.0.smithi123.stdout:918: finishing write tid 6 to smithi12313934-31 2020-01-16T00:12:57.672 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 31 v 95 (ObjNum 203 snap 0 seq_num 203) dirty exists 2020-01-16T00:12:57.672 INFO:tasks.rados.rados.0.smithi123.stdout:918: left oid 31 (ObjNum 203 snap 0 seq_num 203) 2020-01-16T00:12:57.673 INFO:tasks.rados.rados.0.smithi123.stdout:919: expect (ObjNum 184 snap 0 seq_num 184) 2020-01-16T00:12:57.730 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:57.729+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:12:57.779 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:57.779 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:12:57.779 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:12:57.779 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:57.788 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-01-16T00:12:57.831 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:12:57.829+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:12:57.858 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:12:57.859 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2020-01-16T00:12:57.869 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":45,"stamp":"2020-01-16T00:12:55.960521+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":252724978,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1702,"num_read_kb":1209379,"num_write":1090,"num_write_kb":479412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":39,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":987,"ondisk_log_size":987,"up":84,"acting":84,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":84,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":9061600,"kb_used_data":631632,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":745913120,"statfs":{"total":773094113280,"available":763815034880,"internally_reserved":8589934592,"allocated":646791168,"data_stored":643573303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5276441,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":712,"num_read_kb":593072,"num_write":288,"num_write_kb":90096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.937011"},"pg_stats":[{"pgid":"3.f","version":"27'22","reported_seq":"62","reported_epoch":"29","state":"active+clean","last_fresh":"2020-01-16T00:12:49.123889+0000","last_change":"2020-01-16T00:12:32.829755+0000","last_active":"2020-01-16T00:12:49.123889+0000","last_peered":"2020-01-16T00:12:49.123889+0000","last_clean":"2020-01-16T00:12:49.123889+0000","last_became_active":"2020-01-16T00:12:32.829304+0000","last_became_peered":"2020-01-16T00:12:32.829304+0000","last_unstale":"2020-01-16T00:12:49.123889+0000","last_undegraded":"2020-01-16T00:12:49.123889+0000","last_fullsized":"2020-01-16T00:12:49.123889+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2067261,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":56,"num_read_kb":38601,"num_write":19,"num_write_kb":4387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"28'36","reported_seq":"80","reported_epoch":"30","state":"active+clean","last_fresh":"2020-01-16T00:12:54.124734+0000","last_change":"2020-01-16T00:12:32.829508+0000","last_active":"2020-01-16T00:12:54.124734+0000","last_peered":"2020-01-16T00:12:54.124734+0000","last_clean":"2020-01-16T00:12:54.124734+0000","last_became_active":"2020-01-16T00:12:32.829047+0000","last_became_peered":"2020-01-16T00:12:32.829047+0000","last_unstale":"2020-01-16T00:12:54.124734+0000","last_undegraded":"2020-01-16T00:12:54.124734+0000","last_fullsized":"2020-01-16T00:12:54.124734+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6353013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":58,"num_read_kb":44610,"num_write":23,"num_write_kb":7191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"30'117","reported_seq":"208","reported_epoch":"30","state":"active+clean","last_fresh":"2020-01-16T00:12:54.091599+0000","last_change":"2020-01-16T00:12:32.834607+0000","last_active":"2020-01-16T00:12:54.091599+0000","last_peered":"2020-01-16T00:12:54.091599+0000","last_clean":"2020-01-16T00:12:54.091599+0000","last_became_active":"2020-01-16T00:12:32.834155+0000","last_became_peered":"2020-01-16T00:12:32.834155+0000","last_unstale":"2020-01-16T00:12:54.091599+0000","last_undegraded":"2020-01-16T00:12:54.091599+0000","last_fullsized":"2020-01-16T00:12:54.091599+0000","mapping_epoch":20,"log_start":"27'100","ondisk_log_start":"27'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12840515,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":172,"num_read_kb":140071,"num_write":95,"num_write_kb":28105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":5,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"30'91","reported_seq":"166","reported_epoch":"31","state":"active+clean","last_fresh":"2020-01-16T00:12:54.967084+0000","last_change":"2020-01-16T00:12:32.834625+0000","last_active":"2020-01-16T00:12:54.967084+0000","last_peered":"2020-01-16T00:12:54.967084+0000","last_clean":"2020-01-16T00:12:54.967084+0000","last_became_active":"2020-01-16T00:12:32.834210+0000","last_became_peered":"2020-01-16T00:12:32.834210+0000","last_unstale":"2020-01-16T00:12:54.967084+0000","last_undegraded":"2020-01-16T00:12:54.967084+0000","last_fullsized":"2020-01-16T00:12:54.967084+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":91,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10974938,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":130,"num_read_kb":94457,"num_write":67,"num_write_kb":21294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":4,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"30'61","reported_seq":"102","reported_epoch":"30","state":"active+clean","last_fresh":"2020-01-16T00:12:54.551794+0000","last_change":"2020-01-16T00:12:32.833522+0000","last_active":"2020-01-16T00:12:54.551794+0000","last_peered":"2020-01-16T00:12:54.551794+0000","last_clean":"2020-01-16T00:12:54.551794+0000","last_became_active":"2020-01-16T00:12:32.833051+0000","last_became_peered":"2020-01-16T00:12:32.833051+0000","last_unstale":"2020-01-16T00:12:54.551794+0000","last_undegraded":"2020-01-16T00:12:54.551794+0000","last_fullsized":"2020-01-16T00:12:54.551794+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":61,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6134555,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":18669,"num_write":54,"num_write_kb":15447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"30'119","reported_seq":"191","reported_epoch":"30","state":"active+clean","last_fresh":"2020-01-16T00:12:54.178886+0000","last_change":"2020-01-16T00:12:32.834658+0000","last_active":"2020-01-16T00:12:54.178886+0000","last_peered":"2020-01-16T00:12:54.178886+0000","last_clean":"2020-01-16T00:12:54.178886+0000","last_became_active":"2020-01-16T00:12:32.832936+0000","last_became_peered":"2020-01-16T00:12:32.832936+0000","last_unstale":"2020-01-16T00:12:54.178886+0000","last_undegraded":"2020-01-16T00:12:54.178886+0000","last_fullsized":"2020-01-16T00:12:54.178886+0000","mapping_epoch":20,"log_start":"29'100","ondisk_log_start":"29'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11680865,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":108,"num_read_kb":51573,"num_write":94,"num_write_kb":28996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":4,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"29'107","reported_seq":"178","reported_epoch":"30","state":"active+clean","last_fresh":"2020-01-16T00:12:54.141681+0000","last_change":"2020-01-16T00:12:32.832821+0000","last_active":"2020-01-16T00:12:54.141681+0000","last_peered":"2020-01-16T00:12:54.141681+0000","last_clean":"2020-01-16T00:12:54.141681+0000","last_became_active":"2020-01-16T00:12:32.832272+0000","last_became_peered":"2020-01-16T00:12:32.832272+0000","last_unstale":"2020-01-16T00:12:54.141681+0000","last_undegraded":"2020-01-16T00:12:54.141681+0000","last_fullsized":"2020-01-16T00:12:54.141681+0000","mapping_epoch":20,"log_start":"29'100","ondisk_log_start":"29'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13798481,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":117,"num_read_kb":95633,"num_write":82,"num_write_kb":25097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":5,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"30'60","reported_seq":"126","reported_epoch":"30","state":"active+clean","last_fresh":"2020-01-16T00:12:54.143980+0000","last_change":"2020-01-16T00:12:32.830727+0000","last_active":"2020-01-16T00:12:54.143980+0000","last_peered":"2020-01-16T00:12:54.143980+0000","last_clean":"2020-01-16T00:12:54.143980+0000","last_became_active":"2020-01-16T00:12:32.830301+0000","last_became_peered":"2020-01-16T00:12:32.830301+0000","last_unstale":"2020-01-16T00:12:54.143980+0000","last_undegraded":"2020-01-16T00:12:54.143980+0000","last_fullsized":"2020-01-16T00:12:54.143980+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":60,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7832062,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":116,"num_read_kb":83750,"num_write":40,"num_write_kb":11975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"30'77","reported_seq":"138","reported_epoch":"31","state":"active+clean","last_fresh":"2020-01-16T00:12:54.970250+0000","last_change":"2020-01-16T00:12:32.830011+0000","last_active":"2020-01-16T00:12:54.970250+0000","last_peered":"2020-01-16T00:12:54.970250+0000","last_clean":"2020-01-16T00:12:54.970250+0000","last_became_active":"2020-01-16T00:12:32.829571+0000","last_became_peered":"2020-01-16T00:12:32.829571+0000","last_unstale":"2020-01-16T00:12:54.970250+0000","last_undegraded":"2020-01-16T00:12:54.970250+0000","last_fullsized":"2020-01-16T00:12:54.970250+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8588377,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":94,"num_read_kb":77262,"num_write":57,"num_write_kb":18642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"23","reported_epoch":"29","state":"active+clean","last_fresh":"2020-01-16T00:12:49.023660+0000","last_change":"2020-01-16T00:12:20.031741+0000","last_active":"2020-01-16T00:12:49.023660+0000","last_peered":"2020-01-16T00:12:49.023660+0000","last_clean":"2020-01-16T00:12:49.023660+0000","last_became_active":"2020-01-16T00:12:20.030776+0000","last_became_peered":"2020-01-16T00:12:20.030776+0000","last_unstale":"2020-01-16T00:12:49.023660+0000","last_undegraded":"2020-01-16T00:12:49.023660+0000","last_fullsized":"2020-01-16T00:12:49.023660+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"18","reported_epoch":"29","state":"active+clean","last_fresh":"2020-01-16T00:12:49.143168+0000","last_change":"2020-01-16T00:12:32.832234+0000","last_active":"2020-01-16T00:12:49.143168+0000","last_peered":"2020-01-16T00:12:49.143168+0000","last_clean":"2020-01-16T00:12:49.143168+0000","last_became_active":"2020-01-16T00:12:32.831847+0000","last_became_peered":"2020-01-16T00:12:32.831847+0000","last_unstale":"2020-01-16T00:12:49.143168+0000","last_undegraded":"2020-01-16T00:12:49.143168+0000","last_fullsized":"2020-01-16T00:12:49.143168+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"24","reported_epoch":"30","state":"active+clean","last_fresh":"2020-01-16T00:12:51.770352+0000","last_change":"2020-01-16T00:12:20.032733+0000","last_active":"2020-01-16T00:12:51.770352+0000","last_peered":"2020-01-16T00:12:51.770352+0000","last_clean":"2020-01-16T00:12:51.770352+0000","last_became_active":"2020-01-16T00:12:20.032284+0000","last_became_peered":"2020-01-16T00:12:20.032284+0000","last_unstale":"2020-01-16T00:12:51.770352+0000","last_undegraded":"2020-01-16T00:12:51.770352+0000","last_fullsized":"2020-01-16T00:12:51.770352+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"24","reported_epoch":"30","state":"active+clean","last_fresh":"2020-01-16T00:12:51.785574+0000","last_change":"2020-01-16T00:12:20.032010+0000","last_active":"2020-01-16T00:12:51.785574+0000","last_peered":"2020-01-16T00:12:51.785574+0000","last_clean":"2020-01-16T00:12:51.785574+0000","last_became_active":"2020-01-16T00:12:20.031560+0000","last_became_peered":"2020-01-16T00:12:20.031560+0000","last_unstale":"2020-01-16T00:12:51.785574+0000","last_undegraded":"2020-01-16T00:12:51.785574+0000","last_fullsized":"2020-01-16T00:12:51.785574+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"30'42","reported_seq":"104","reported_epoch":"30","state":"active+clean","last_fresh":"2020-01-16T00:12:54.538359+0000","last_change":"2020-01-16T00:12:31.817123+0000","last_active":"2020-01-16T00:12:54.538359+0000","last_peered":"2020-01-16T00:12:54.538359+0000","last_clean":"2020-01-16T00:12:54.538359+0000","last_became_active":"2020-01-16T00:12:31.816785+0000","last_became_peered":"2020-01-16T00:12:31.816785+0000","last_unstale":"2020-01-16T00:12:54.538359+0000","last_undegraded":"2020-01-16T00:12:54.538359+0000","last_fullsized":"2020-01-16T00:12:54.538359+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:30.783643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:30.783643+0000","last_clean_scrub_stamp":"2020-01-16T00:12:30.783643+0000","log_size":42,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33008486,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":43,"num_read_kb":17379,"num_write":42,"num_write_kb":52106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"29'40","reported_seq":"98","reported_epoch":"30","state":"active+clean","last_fresh":"2020-01-16T00:12:52.798666+0000","last_change":"2020-01-16T00:12:32.836948+0000","last_active":"2020-01-16T00:12:52.798666+0000","last_peered":"2020-01-16T00:12:52.798666+0000","last_clean":"2020-01-16T00:12:52.798666+0000","last_became_active":"2020-01-16T00:12:32.836523+0000","last_became_peered":"2020-01-16T00:12:32.836523+0000","last_unstale":"2020-01-16T00:12:52.798666+0000","last_undegraded":"2020-01-16T00:12:52.798666+0000","last_fullsized":"2020-01-16T00:12:52.798666+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4789995,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":100,"num_read_kb":73331,"num_write":27,"num_write_kb":8308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"23","reported_epoch":"29","state":"active+clean","last_fresh":"2020-01-16T00:12:49.120642+0000","last_change":"2020-01-16T00:12:20.032658+0000","last_active":"2020-01-16T00:12:49.120642+0000","last_peered":"2020-01-16T00:12:49.120642+0000","last_clean":"2020-01-16T00:12:49.120642+0000","last_became_active":"2020-01-16T00:12:20.030794+0000","last_became_peered":"2020-01-16T00:12:20.030794+0000","last_unstale":"2020-01-16T00:12:49.120642+0000","last_undegraded":"2020-01-16T00:12:49.120642+0000","last_fullsized":"2020-01-16T00:12:49.120642+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"30'57","reported_seq":"114","reported_epoch":"30","state":"active+clean","last_fresh":"2020-01-16T00:12:52.748149+0000","last_change":"2020-01-16T00:12:31.817506+0000","last_active":"2020-01-16T00:12:52.748149+0000","last_peered":"2020-01-16T00:12:52.748149+0000","last_clean":"2020-01-16T00:12:52.748149+0000","last_became_active":"2020-01-16T00:12:31.817077+0000","last_became_peered":"2020-01-16T00:12:31.817077+0000","last_unstale":"2020-01-16T00:12:52.748149+0000","last_undegraded":"2020-01-16T00:12:52.748149+0000","last_fullsized":"2020-01-16T00:12:52.748149+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:30.783643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:30.783643+0000","last_clean_scrub_stamp":"2020-01-16T00:12:30.783643+0000","log_size":57,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37445501,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":37,"num_read_kb":19216,"num_write":57,"num_write_kb":61986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"30'167","reported_seq":"277","reported_epoch":"30","state":"active+clean","last_fresh":"2020-01-16T00:12:54.544688+0000","last_change":"2020-01-16T00:12:32.830294+0000","last_active":"2020-01-16T00:12:54.544688+0000","last_peered":"2020-01-16T00:12:54.544688+0000","last_clean":"2020-01-16T00:12:54.544688+0000","last_became_active":"2020-01-16T00:12:32.829870+0000","last_became_peered":"2020-01-16T00:12:32.829870+0000","last_unstale":"2020-01-16T00:12:54.544688+0000","last_undegraded":"2020-01-16T00:12:54.544688+0000","last_fullsized":"2020-01-16T00:12:54.544688+0000","mapping_epoch":20,"log_start":"24'100","ondisk_log_start":"24'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":67,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16991844,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":211,"num_read_kb":177729,"num_write":129,"num_write_kb":40432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"23","reported_epoch":"29","state":"active+clean","last_fresh":"2020-01-16T00:12:49.120485+0000","last_change":"2020-01-16T00:12:20.033205+0000","last_active":"2020-01-16T00:12:49.120485+0000","last_peered":"2020-01-16T00:12:49.120485+0000","last_clean":"2020-01-16T00:12:49.120485+0000","last_became_active":"2020-01-16T00:12:20.030792+0000","last_became_peered":"2020-01-16T00:12:20.030792+0000","last_unstale":"2020-01-16T00:12:49.120485+0000","last_undegraded":"2020-01-16T00:12:49.120485+0000","last_fullsized":"2020-01-16T00:12:49.120485+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"30'48","reported_seq":"102","reported_epoch":"30","state":"active+clean","last_fresh":"2020-01-16T00:12:54.170488+0000","last_change":"2020-01-16T00:12:31.815364+0000","last_active":"2020-01-16T00:12:54.170488+0000","last_peered":"2020-01-16T00:12:54.170488+0000","last_clean":"2020-01-16T00:12:54.170488+0000","last_became_active":"2020-01-16T00:12:31.814890+0000","last_became_peered":"2020-01-16T00:12:31.814890+0000","last_unstale":"2020-01-16T00:12:54.170488+0000","last_undegraded":"2020-01-16T00:12:54.170488+0000","last_fullsized":"2020-01-16T00:12:54.170488+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:30.783643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:30.783643+0000","last_clean_scrub_stamp":"2020-01-16T00:12:30.783643+0000","log_size":48,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30319399,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":34,"num_read_kb":16408,"num_write":48,"num_write_kb":49947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"29'77","reported_seq":"140","reported_epoch":"30","state":"active+clean","last_fresh":"2020-01-16T00:12:51.771735+0000","last_change":"2020-01-16T00:12:32.834986+0000","last_active":"2020-01-16T00:12:51.771735+0000","last_peered":"2020-01-16T00:12:51.771735+0000","last_clean":"2020-01-16T00:12:51.771735+0000","last_became_active":"2020-01-16T00:12:32.834537+0000","last_became_peered":"2020-01-16T00:12:32.834537+0000","last_unstale":"2020-01-16T00:12:51.771735+0000","last_undegraded":"2020-01-16T00:12:51.771735+0000","last_fullsized":"2020-01-16T00:12:51.771735+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7613571,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":108,"num_read_kb":95143,"num_write":54,"num_write_kb":19739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"24","reported_epoch":"30","state":"active+clean","last_fresh":"2020-01-16T00:12:51.850928+0000","last_change":"2020-01-16T00:12:20.031201+0000","last_active":"2020-01-16T00:12:51.850928+0000","last_peered":"2020-01-16T00:12:51.850928+0000","last_clean":"2020-01-16T00:12:51.850928+0000","last_became_active":"2020-01-16T00:12:20.030752+0000","last_became_peered":"2020-01-16T00:12:20.030752+0000","last_unstale":"2020-01-16T00:12:51.850928+0000","last_undegraded":"2020-01-16T00:12:51.850928+0000","last_fullsized":"2020-01-16T00:12:51.850928+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"29'24","reported_seq":"65","reported_epoch":"30","state":"active+clean","last_fresh":"2020-01-16T00:12:51.851113+0000","last_change":"2020-01-16T00:12:31.815506+0000","last_active":"2020-01-16T00:12:51.851113+0000","last_peered":"2020-01-16T00:12:51.851113+0000","last_clean":"2020-01-16T00:12:51.851113+0000","last_became_active":"2020-01-16T00:12:31.815111+0000","last_became_peered":"2020-01-16T00:12:31.815111+0000","last_unstale":"2020-01-16T00:12:51.851113+0000","last_undegraded":"2020-01-16T00:12:51.851113+0000","last_fullsized":"2020-01-16T00:12:51.851113+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:30.783643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:30.783643+0000","last_clean_scrub_stamp":"2020-01-16T00:12:30.783643+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19831996,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":21,"num_read_kb":10085,"num_write":24,"num_write_kb":32111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"30'81","reported_seq":"137","reported_epoch":"30","state":"active+clean","last_fresh":"2020-01-16T00:12:54.196010+0000","last_change":"2020-01-16T00:12:32.832002+0000","last_active":"2020-01-16T00:12:54.196010+0000","last_peered":"2020-01-16T00:12:54.196010+0000","last_clean":"2020-01-16T00:12:54.196010+0000","last_became_active":"2020-01-16T00:12:32.831577+0000","last_became_peered":"2020-01-16T00:12:32.831577+0000","last_unstale":"2020-01-16T00:12:54.196010+0000","last_undegraded":"2020-01-16T00:12:54.196010+0000","last_fullsized":"2020-01-16T00:12:54.196010+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":81,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8806868,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":79,"num_read_kb":38059,"num_write":65,"num_write_kb":20268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"31'86","reported_seq":"138","reported_epoch":"31","state":"active+clean","last_fresh":"2020-01-16T00:12:54.968718+0000","last_change":"2020-01-16T00:12:32.834894+0000","last_active":"2020-01-16T00:12:54.968718+0000","last_peered":"2020-01-16T00:12:54.968718+0000","last_clean":"2020-01-16T00:12:54.968718+0000","last_became_active":"2020-01-16T00:12:32.834414+0000","last_became_peered":"2020-01-16T00:12:32.834414+0000","last_unstale":"2020-01-16T00:12:54.968718+0000","last_undegraded":"2020-01-16T00:12:54.968718+0000","last_fullsized":"2020-01-16T00:12:54.968718+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":86,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8470695,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":75,"num_read_kb":54773,"num_write":60,"num_write_kb":17082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":4,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"24","reported_epoch":"30","state":"active+clean","last_fresh":"2020-01-16T00:12:51.770891+0000","last_change":"2020-01-16T00:12:20.033023+0000","last_active":"2020-01-16T00:12:51.770891+0000","last_peered":"2020-01-16T00:12:51.770891+0000","last_clean":"2020-01-16T00:12:51.770891+0000","last_became_active":"2020-01-16T00:12:20.031576+0000","last_became_peered":"2020-01-16T00:12:20.031576+0000","last_unstale":"2020-01-16T00:12:51.770891+0000","last_undegraded":"2020-01-16T00:12:51.770891+0000","last_fullsized":"2020-01-16T00:12:51.770891+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"30'75","reported_seq":"136","reported_epoch":"30","state":"active+clean","last_fresh":"2020-01-16T00:12:54.314972+0000","last_change":"2020-01-16T00:12:32.832313+0000","last_active":"2020-01-16T00:12:54.314972+0000","last_peered":"2020-01-16T00:12:54.314972+0000","last_clean":"2020-01-16T00:12:54.314972+0000","last_became_active":"2020-01-16T00:12:32.831843+0000","last_became_peered":"2020-01-16T00:12:32.831843+0000","last_unstale":"2020-01-16T00:12:54.314972+0000","last_undegraded":"2020-01-16T00:12:54.314972+0000","last_fullsized":"2020-01-16T00:12:54.314972+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":75,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5176556,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":97,"num_read_kb":62630,"num_write":53,"num_write_kb":16299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"24","reported_epoch":"30","state":"active+clean","last_fresh":"2020-01-16T00:12:51.770969+0000","last_change":"2020-01-16T00:12:20.034125+0000","last_active":"2020-01-16T00:12:51.770969+0000","last_peered":"2020-01-16T00:12:51.770969+0000","last_clean":"2020-01-16T00:12:51.770969+0000","last_became_active":"2020-01-16T00:12:20.033645+0000","last_became_peered":"2020-01-16T00:12:20.033645+0000","last_unstale":"2020-01-16T00:12:51.770969+0000","last_undegraded":"2020-01-16T00:12:51.770969+0000","last_fullsized":"2020-01-16T00:12:51.770969+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":132119596,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":1567,"num_read_kb":1146291,"num_write":919,"num_write_kb":283262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":39,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":288178176,"data_stored":286780479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":816,"ondisk_log_size":816,"up":48,"acting":48,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8},{"poolid":2,"num_pg":4,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":135,"num_read_kb":63088,"num_write":171,"num_write_kb":196150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":362127360,"data_stored":361816146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":171,"ondisk_log_size":171,"up":12,"acting":12,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1149388,"kb_used_data":99900,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93222452,"statfs":{"total":96636764160,"available":95459790848,"internally_reserved":1073741824,"allocated":102297600,"data_stored":101856161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574859,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1119916,"kb_used_data":70932,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93251924,"statfs":{"total":96636764160,"available":95489970176,"internally_reserved":1073741824,"allocated":72634368,"data_stored":72172584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1142176,"kb_used_data":85716,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93229664,"statfs":{"total":96636764160,"available":95467175936,"internally_reserved":1073741824,"allocated":87773184,"data_stored":87356729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1144972,"kb_used_data":89860,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93226868,"statfs":{"total":96636764160,"available":95464312832,"internally_reserved":1073741824,"allocated":92016640,"data_stored":91691924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1149404,"kb_used_data":96480,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93222436,"statfs":{"total":96636764160,"available":95459774464,"internally_reserved":1073741824,"allocated":98795520,"data_stored":98388198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1127220,"kb_used_data":74524,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93244620,"statfs":{"total":96636764160,"available":95482490880,"internally_reserved":1073741824,"allocated":76312576,"data_stored":75917351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1090388,"kb_used_data":35432,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93281452,"statfs":{"total":96636764160,"available":95520206848,"internally_reserved":1073741824,"allocated":36282368,"data_stored":35927484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1138136,"kb_used_data":78788,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93233704,"statfs":{"total":96636764160,"available":95471312896,"internally_reserved":1073741824,"allocated":80678912,"data_stored":80262872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}]}} 2020-01-16T00:12:57.873 INFO:tasks.ceph.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2020-01-16T00:12:57.884 INFO:tasks.rados.rados.0.smithi123.stdout:920: finishing write tid 1 to smithi12313934-42 2020-01-16T00:12:57.884 INFO:tasks.rados.rados.0.smithi123.stdout:920: finishing write tid 2 to smithi12313934-42 2020-01-16T00:12:57.884 INFO:tasks.rados.rados.0.smithi123.stdout:920: finishing write tid 3 to smithi12313934-42 2020-01-16T00:12:57.884 INFO:tasks.rados.rados.0.smithi123.stdout:920: finishing write tid 5 to smithi12313934-42 2020-01-16T00:12:57.884 INFO:tasks.rados.rados.0.smithi123.stdout:920: finishing write tid 6 to smithi12313934-42 2020-01-16T00:12:57.884 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 42 v 86 (ObjNum 204 snap 0 seq_num 204) dirty exists 2020-01-16T00:12:57.884 INFO:tasks.rados.rados.0.smithi123.stdout:920: left oid 42 (ObjNum 204 snap 0 seq_num 204) 2020-01-16T00:12:57.885 INFO:tasks.rados.rados.0.smithi123.stdout:921: finishing write tid 1 to smithi12313934-17 2020-01-16T00:12:57.885 INFO:tasks.rados.rados.0.smithi123.stdout:915: done (15 left) 2020-01-16T00:12:57.885 INFO:tasks.rados.rados.0.smithi123.stdout:916: done (14 left) 2020-01-16T00:12:57.885 INFO:tasks.rados.rados.0.smithi123.stdout:917: done (13 left) 2020-01-16T00:12:57.885 INFO:tasks.rados.rados.0.smithi123.stdout:918: done (12 left) 2020-01-16T00:12:57.885 INFO:tasks.rados.rados.0.smithi123.stdout:919: done (11 left) 2020-01-16T00:12:57.885 INFO:tasks.rados.rados.0.smithi123.stdout:920: done (10 left) 2020-01-16T00:12:57.885 INFO:tasks.rados.rados.0.smithi123.stdout:930: done (9 left) 2020-01-16T00:12:57.886 INFO:tasks.rados.rados.0.smithi123.stdout:931: write oid 14 current snap is 0 2020-01-16T00:12:57.886 INFO:tasks.rados.rados.0.smithi123.stdout:931: seq_num 211 ranges {746277=619070,1830877=681726,2928369=617908} 2020-01-16T00:12:57.892 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:57.893 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:12:57.893 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:12:57.893 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:12:57.893 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:57.893 INFO:tasks.rados.rados.0.smithi123.stdout:931: writing smithi12313934-14 from 746277 to 1365347 tid 1 2020-01-16T00:12:57.895 INFO:teuthology.orchestra.run.smithi123.stderr:nodeep-scrub is set 2020-01-16T00:12:57.899 INFO:tasks.rados.rados.0.smithi123.stdout:931: writing smithi12313934-14 from 1830877 to 2512603 tid 2 2020-01-16T00:12:57.904 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-01-16T00:12:57.905 INFO:tasks.rados.rados.0.smithi123.stdout:931: writing smithi12313934-14 from 2928369 to 3546277 tid 3 2020-01-16T00:12:57.905 INFO:tasks.rados.rados.0.smithi123.stdout:932: write (excl) oid 33 current snap is 0 2020-01-16T00:12:57.905 INFO:tasks.rados.rados.0.smithi123.stdout:932: seq_num 212 ranges {763084=694319,1889359=541737,2955792=541902} 2020-01-16T00:12:57.914 INFO:tasks.rados.rados.0.smithi123.stdout:932: writing smithi12313934-33 from 763084 to 1457403 tid 1 2020-01-16T00:12:57.920 INFO:tasks.rados.rados.0.smithi123.stdout:932: writing smithi12313934-33 from 1889359 to 2431096 tid 2 2020-01-16T00:12:57.927 INFO:tasks.rados.rados.0.smithi123.stdout:932: writing smithi12313934-33 from 2955792 to 3497694 tid 3 2020-01-16T00:12:57.927 INFO:tasks.rados.rados.0.smithi123.stdout:933: read oid 29 snap -1 2020-01-16T00:12:57.927 INFO:tasks.rados.rados.0.smithi123.stdout:933: expect (ObjNum 200 snap 0 seq_num 200) 2020-01-16T00:12:57.927 INFO:tasks.rados.rados.0.smithi123.stdout:934: read oid 50 snap -1 2020-01-16T00:12:57.927 INFO:tasks.rados.rados.0.smithi123.stdout:934: expect (ObjNum 177 snap 0 seq_num 177) 2020-01-16T00:12:57.927 INFO:tasks.rados.rados.0.smithi123.stdout:935: write (excl) oid 9 current snap is 0 2020-01-16T00:12:57.927 INFO:tasks.rados.rados.0.smithi123.stdout:935: seq_num 213 ranges {779891=485921,1947841=685395,3383215=196676} 2020-01-16T00:12:57.934 INFO:tasks.rados.rados.0.smithi123.stdout:935: writing smithi12313934-9 from 779891 to 1265812 tid 1 2020-01-16T00:12:57.943 INFO:tasks.rados.rados.0.smithi123.stdout:935: writing smithi12313934-9 from 1947841 to 2633236 tid 2 2020-01-16T00:12:57.950 INFO:tasks.rados.rados.0.smithi123.stdout:935: writing smithi12313934-9 from 3383215 to 3579891 tid 3 2020-01-16T00:12:57.951 INFO:tasks.rados.rados.0.smithi123.stdout:936: read oid 25 snap -1 2020-01-16T00:12:57.951 INFO:tasks.rados.rados.0.smithi123.stdout:936: expect (ObjNum 201 snap 0 seq_num 201) 2020-01-16T00:12:57.951 INFO:tasks.rados.rados.0.smithi123.stdout:937: read oid 6 snap -1 2020-01-16T00:12:57.951 INFO:tasks.rados.rados.0.smithi123.stdout:937: expect (ObjNum 137 snap 0 seq_num 137) 2020-01-16T00:12:57.951 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:57.951 INFO:tasks.rados.rados.0.smithi123.stdout:921: finishing write tid 2 to smithi12313934-17 2020-01-16T00:12:57.951 INFO:tasks.rados.rados.0.smithi123.stdout:921: finishing write tid 3 to smithi12313934-17 2020-01-16T00:12:57.952 INFO:tasks.rados.rados.0.smithi123.stdout:921: finishing write tid 5 to smithi12313934-17 2020-01-16T00:12:57.952 INFO:tasks.rados.rados.0.smithi123.stdout:921: finishing write tid 6 to smithi12313934-17 2020-01-16T00:12:57.952 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 17 v 94 (ObjNum 205 snap 0 seq_num 205) dirty exists 2020-01-16T00:12:57.952 INFO:tasks.rados.rados.0.smithi123.stdout:921: left oid 17 (ObjNum 205 snap 0 seq_num 205) 2020-01-16T00:12:57.952 INFO:tasks.rados.rados.0.smithi123.stdout:922: finishing write tid 1 to smithi12313934-35 2020-01-16T00:12:57.952 INFO:tasks.rados.rados.0.smithi123.stdout:922: finishing write tid 2 to smithi12313934-35 2020-01-16T00:12:57.952 INFO:tasks.rados.rados.0.smithi123.stdout:922: finishing write tid 3 to smithi12313934-35 2020-01-16T00:12:57.952 INFO:tasks.rados.rados.0.smithi123.stdout:921: done (15 left) 2020-01-16T00:12:57.953 INFO:tasks.rados.rados.0.smithi123.stdout:938: read oid 5 snap -1 2020-01-16T00:12:57.953 INFO:tasks.rados.rados.0.smithi123.stdout:938: expect (ObjNum 198 snap 0 seq_num 198) 2020-01-16T00:12:57.953 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:57.953 INFO:tasks.rados.rados.0.smithi123.stdout:922: finishing write tid 5 to smithi12313934-35 2020-01-16T00:12:57.953 INFO:tasks.rados.rados.0.smithi123.stdout:922: finishing write tid 6 to smithi12313934-35 2020-01-16T00:12:57.953 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 35 v 98 (ObjNum 206 snap 0 seq_num 206) dirty exists 2020-01-16T00:12:57.954 INFO:tasks.rados.rados.0.smithi123.stdout:922: left oid 35 (ObjNum 206 snap 0 seq_num 206) 2020-01-16T00:12:57.954 INFO:tasks.rados.rados.0.smithi123.stdout:923: finishing write tid 1 to smithi12313934-2 2020-01-16T00:12:57.954 INFO:tasks.rados.rados.0.smithi123.stdout:923: finishing write tid 2 to smithi12313934-2 2020-01-16T00:12:57.954 INFO:tasks.rados.rados.0.smithi123.stdout:923: finishing write tid 3 to smithi12313934-2 2020-01-16T00:12:57.954 INFO:tasks.rados.rados.0.smithi123.stdout:923: finishing write tid 5 to smithi12313934-2 2020-01-16T00:12:57.954 INFO:tasks.rados.rados.0.smithi123.stdout:923: finishing write tid 6 to smithi12313934-2 2020-01-16T00:12:57.954 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 2 v 128 (ObjNum 207 snap 0 seq_num 207) dirty exists 2020-01-16T00:12:57.954 INFO:tasks.rados.rados.0.smithi123.stdout:923: left oid 2 (ObjNum 207 snap 0 seq_num 207) 2020-01-16T00:12:57.955 INFO:tasks.rados.rados.0.smithi123.stdout:924: finishing write tid 1 to smithi12313934-10 2020-01-16T00:12:57.955 INFO:tasks.rados.rados.0.smithi123.stdout:924: finishing write tid 2 to smithi12313934-10 2020-01-16T00:12:57.955 INFO:tasks.rados.rados.0.smithi123.stdout:922: done (15 left) 2020-01-16T00:12:57.955 INFO:tasks.rados.rados.0.smithi123.stdout:923: done (14 left) 2020-01-16T00:12:57.955 INFO:tasks.rados.rados.0.smithi123.stdout:939: read oid 1 snap -1 2020-01-16T00:12:57.955 INFO:tasks.rados.rados.0.smithi123.stdout:939: expect (ObjNum 182 snap 0 seq_num 182) 2020-01-16T00:12:57.955 INFO:tasks.rados.rados.0.smithi123.stdout:940: write (excl) oid 22 current snap is 0 2020-01-16T00:12:57.956 INFO:tasks.rados.rados.0.smithi123.stdout:940: seq_num 214 ranges {796698=561170,2006323=545406,3126991=469707} 2020-01-16T00:12:57.956 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:12:57.953+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:12:57.958 INFO:tasks.rados.rados.0.smithi123.stdout:940: writing smithi12313934-22 from 796698 to 1357868 tid 1 2020-01-16T00:12:57.968 INFO:tasks.rados.rados.0.smithi123.stdout:940: writing smithi12313934-22 from 2006323 to 2551729 tid 2 2020-01-16T00:12:57.975 INFO:tasks.rados.rados.0.smithi123.stdout:940: writing smithi12313934-22 from 3126991 to 3596698 tid 3 2020-01-16T00:12:57.975 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:57.975 INFO:tasks.rados.rados.0.smithi123.stdout:924: finishing write tid 3 to smithi12313934-10 2020-01-16T00:12:57.975 INFO:tasks.rados.rados.0.smithi123.stdout:924: finishing write tid 5 to smithi12313934-10 2020-01-16T00:12:57.976 INFO:tasks.rados.rados.0.smithi123.stdout:924: finishing write tid 6 to smithi12313934-10 2020-01-16T00:12:57.976 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 10 v 89 (ObjNum 208 snap 0 seq_num 208) dirty exists 2020-01-16T00:12:57.976 INFO:tasks.rados.rados.0.smithi123.stdout:924: left oid 10 (ObjNum 208 snap 0 seq_num 208) 2020-01-16T00:12:57.976 INFO:tasks.rados.rados.0.smithi123.stdout:925: expect (ObjNum 143 snap 0 seq_num 143) 2020-01-16T00:12:58.017 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:12:58.017 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:12:58.017 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:12:58.017 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:12:58.017 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.018 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:890 3.a 3:5c245412:::smithi12313934-19:head [write 759683~639358,stat] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:58.018 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.112181+0000", 2020-01-16T00:12:58.018 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 20.904213550000001, 2020-01-16T00:12:58.018 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0068210429999999997, 2020-01-16T00:12:58.018 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:58.018 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:58.018 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:58.018 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:58.019 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:58.019 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 890 2020-01-16T00:12:58.019 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.019 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:58.019 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.019 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:58.019 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.112181+0000", 2020-01-16T00:12:58.020 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:58.020 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.020 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.020 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:58.020 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.112181+0000", 2020-01-16T00:12:58.020 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999957 2020-01-16T00:12:58.020 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.020 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.021 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:58.021 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.112177+0000", 2020-01-16T00:12:58.021 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00060751999999999996 2020-01-16T00:12:58.021 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.021 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.021 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:58.021 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.112784+0000", 2020-01-16T00:12:58.022 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3646999999999998e-05 2020-01-16T00:12:58.022 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.022 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.022 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:58.022 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.112818+0000", 2020-01-16T00:12:58.022 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.432e-05 2020-01-16T00:12:58.022 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.022 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.023 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:58.023 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.112832+0000", 2020-01-16T00:12:58.023 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000118984 2020-01-16T00:12:58.023 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.023 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.023 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:58.023 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.112951+0000", 2020-01-16T00:12:58.023 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00023189400000000001 2020-01-16T00:12:58.024 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.024 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.024 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:58.024 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.113183+0000", 2020-01-16T00:12:58.024 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00035231999999999999 2020-01-16T00:12:58.024 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.024 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.025 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:12:58.025 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.113535+0000", 2020-01-16T00:12:58.025 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0035763539999999999 2020-01-16T00:12:58.025 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.025 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.025 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:58.026 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.117112+0000", 2020-01-16T00:12:58.026 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4860000000000002e-05 2020-01-16T00:12:58.026 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.026 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.026 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:58.026 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.117147+0000", 2020-01-16T00:12:58.026 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0017898110000000001 2020-01-16T00:12:58.027 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.027 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.027 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:58.027 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.118936+0000", 2020-01-16T00:12:58.027 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6343000000000001e-05 2020-01-16T00:12:58.027 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.027 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.027 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:58.028 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.118973+0000", 2020-01-16T00:12:58.028 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9257000000000001e-05 2020-01-16T00:12:58.028 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.028 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.028 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:58.028 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.119002+0000", 2020-01-16T00:12:58.028 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0068210429999999997 2020-01-16T00:12:58.029 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.029 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:58.029 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.029 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.029 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.029 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(osd.1.0:162 3.0 e24/20 3:00c146d5:::smithi12313934-18:head v 24'99, mlcod=24'97)", 2020-01-16T00:12:58.029 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.038304+0000", 2020-01-16T00:12:58.029 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 19.978090120000001, 2020-01-16T00:12:58.030 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.006772386, 2020-01-16T00:12:58.030 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:58.030 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:58.030 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:58.030 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.030 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:58.030 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.038304+0000", 2020-01-16T00:12:58.031 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:58.031 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.031 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.031 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:58.031 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.038304+0000", 2020-01-16T00:12:58.031 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:12:58.031 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.032 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.032 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:58.032 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.038301+0000", 2020-01-16T00:12:58.032 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012763920000000001 2020-01-16T00:12:58.032 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.032 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.032 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:58.032 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.039577+0000", 2020-01-16T00:12:58.033 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1426e-05 2020-01-16T00:12:58.033 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.033 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.033 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:58.033 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.039608+0000", 2020-01-16T00:12:58.033 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2017e-05 2020-01-16T00:12:58.033 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.033 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.034 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:58.034 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.039620+0000", 2020-01-16T00:12:58.034 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000111969 2020-01-16T00:12:58.034 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.034 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.034 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:58.034 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.039732+0000", 2020-01-16T00:12:58.034 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1180000000000002e-05 2020-01-16T00:12:58.035 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.035 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.035 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:58.035 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.039774+0000", 2020-01-16T00:12:58.035 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.005179286 2020-01-16T00:12:58.035 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.036 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.036 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:58.036 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.044953+0000", 2020-01-16T00:12:58.036 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012396600000000001 2020-01-16T00:12:58.036 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.036 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.036 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:58.036 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.045077+0000", 2020-01-16T00:12:58.037 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.006772386 2020-01-16T00:12:58.037 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.037 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:58.037 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.037 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.037 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.038 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:965 3.a 3:54f2c117:::smithi12313934-41:head [write 527753~708201] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:58.038 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.146756+0000", 2020-01-16T00:12:58.038 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 19.869639017000001, 2020-01-16T00:12:58.038 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0071708459999999998, 2020-01-16T00:12:58.039 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:58.039 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:58.039 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:58.040 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:58.040 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:58.040 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 965 2020-01-16T00:12:58.040 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.040 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:58.041 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.041 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:58.041 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.146756+0000", 2020-01-16T00:12:58.041 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:58.041 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.041 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.041 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:58.041 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.146756+0000", 2020-01-16T00:12:58.042 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999948 2020-01-16T00:12:58.042 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.042 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.042 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:58.042 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.146750+0000", 2020-01-16T00:12:58.042 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00094503399999999998 2020-01-16T00:12:58.042 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.042 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.043 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:58.043 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.147695+0000", 2020-01-16T00:12:58.043 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7846999999999999e-05 2020-01-16T00:12:58.043 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.043 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.043 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:58.043 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.147733+0000", 2020-01-16T00:12:58.044 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4392999999999999e-05 2020-01-16T00:12:58.044 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.044 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.044 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:58.044 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.147747+0000", 2020-01-16T00:12:58.044 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000106737 2020-01-16T00:12:58.044 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.044 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.045 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:58.045 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.147854+0000", 2020-01-16T00:12:58.045 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00026514699999999997 2020-01-16T00:12:58.045 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.045 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.045 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:58.046 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.148119+0000", 2020-01-16T00:12:58.046 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030262600000000001 2020-01-16T00:12:58.046 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.046 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.046 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:12:58.046 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.148422+0000", 2020-01-16T00:12:58.046 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.004368416 2020-01-16T00:12:58.046 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.047 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.047 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:58.047 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.152790+0000", 2020-01-16T00:12:58.047 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000368437 2020-01-16T00:12:58.047 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.047 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.047 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:58.048 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.153159+0000", 2020-01-16T00:12:58.048 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00058380900000000002 2020-01-16T00:12:58.048 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.048 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.048 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:58.048 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.153743+0000", 2020-01-16T00:12:58.048 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8196999999999995e-05 2020-01-16T00:12:58.048 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.049 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.049 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:58.049 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.153841+0000", 2020-01-16T00:12:58.049 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5625999999999996e-05 2020-01-16T00:12:58.049 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.049 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.049 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:58.049 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.153926+0000", 2020-01-16T00:12:58.050 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0071708459999999998 2020-01-16T00:12:58.050 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.050 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:58.050 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.050 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.050 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.050 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:966 3.a 3:54f2c117:::smithi12313934-41:head [write 1856548~737690] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:12:58.050 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.154844+0000", 2020-01-16T00:12:58.051 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 19.861550214000001, 2020-01-16T00:12:58.051 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0077324510000000004, 2020-01-16T00:12:58.051 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:58.051 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:58.051 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:58.051 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:58.051 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:58.051 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 966 2020-01-16T00:12:58.052 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.052 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:58.052 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.052 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:58.052 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.154844+0000", 2020-01-16T00:12:58.052 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:58.052 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.053 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.053 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:58.053 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.154844+0000", 2020-01-16T00:12:58.053 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:58.053 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.053 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.053 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:58.053 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.154842+0000", 2020-01-16T00:12:58.054 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00085401100000000001 2020-01-16T00:12:58.054 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.054 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.054 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:58.054 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.155696+0000", 2020-01-16T00:12:58.054 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9801000000000001e-05 2020-01-16T00:12:58.054 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.054 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.055 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:58.055 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.155726+0000", 2020-01-16T00:12:58.055 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.171e-05 2020-01-16T00:12:58.055 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.055 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.055 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:58.056 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.155738+0000", 2020-01-16T00:12:58.056 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0573000000000003e-05 2020-01-16T00:12:58.056 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.056 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.056 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:58.056 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.155818+0000", 2020-01-16T00:12:58.056 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024165699999999999 2020-01-16T00:12:58.057 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.057 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.057 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:58.057 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.156060+0000", 2020-01-16T00:12:58.057 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033522700000000001 2020-01-16T00:12:58.057 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.057 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.057 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:12:58.058 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.156395+0000", 2020-01-16T00:12:58.058 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0059325039999999999 2020-01-16T00:12:58.058 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.058 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.058 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:58.058 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.162328+0000", 2020-01-16T00:12:58.058 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5182e-05 2020-01-16T00:12:58.058 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.059 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.059 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:58.059 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.162393+0000", 2020-01-16T00:12:58.059 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8604e-05 2020-01-16T00:12:58.059 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.059 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.059 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:58.059 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.162411+0000", 2020-01-16T00:12:58.060 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2386000000000006e-05 2020-01-16T00:12:58.060 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.060 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.060 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:58.060 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.162504+0000", 2020-01-16T00:12:58.060 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2970000000000001e-05 2020-01-16T00:12:58.060 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.060 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.061 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:58.061 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.162577+0000", 2020-01-16T00:12:58.061 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0077324510000000004 2020-01-16T00:12:58.061 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.061 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:58.061 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.061 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.061 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.062 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.4488.0:1237 3.3 e26/20 3:c204aa4f:::smithi12313934-45:head v 26'56, mlcod=26'54)", 2020-01-16T00:12:58.062 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.587132+0000", 2020-01-16T00:12:58.062 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 15.429262994, 2020-01-16T00:12:58.062 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0076804919999999997, 2020-01-16T00:12:58.062 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:58.062 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:58.062 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:58.063 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.063 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:58.063 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.587132+0000", 2020-01-16T00:12:58.063 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:58.063 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.063 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.063 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:58.063 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.587132+0000", 2020-01-16T00:12:58.064 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999957 2020-01-16T00:12:58.064 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.064 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.064 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:58.064 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.587127+0000", 2020-01-16T00:12:58.064 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00047205800000000003 2020-01-16T00:12:58.064 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.064 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.065 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:58.065 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.587599+0000", 2020-01-16T00:12:58.065 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6477000000000003e-05 2020-01-16T00:12:58.065 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.065 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.065 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:58.065 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.587636+0000", 2020-01-16T00:12:58.065 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7294000000000001e-05 2020-01-16T00:12:58.066 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.066 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.066 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:58.066 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.587653+0000", 2020-01-16T00:12:58.066 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000183778 2020-01-16T00:12:58.067 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.067 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.067 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:58.067 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.587837+0000", 2020-01-16T00:12:58.067 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0628000000000001e-05 2020-01-16T00:12:58.067 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.067 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.067 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:58.068 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.587917+0000", 2020-01-16T00:12:58.068 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.006854527 2020-01-16T00:12:58.068 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.068 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.068 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:58.068 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.594772+0000", 2020-01-16T00:12:58.068 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0144000000000001e-05 2020-01-16T00:12:58.068 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.069 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.069 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:58.069 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.594812+0000", 2020-01-16T00:12:58.069 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0076804919999999997 2020-01-16T00:12:58.069 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.069 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:58.069 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.069 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.070 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.070 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1241 3.a 3:5c245412:::smithi12313934-19:head [read 0~3815156,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:12:58.070 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.589903+0000", 2020-01-16T00:12:58.070 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 15.426491596, 2020-01-16T00:12:58.070 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0096065560000000005, 2020-01-16T00:12:58.070 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:58.070 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:58.071 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:58.071 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:58.071 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:58.071 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1241 2020-01-16T00:12:58.071 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.071 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:58.071 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.071 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:58.072 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.589903+0000", 2020-01-16T00:12:58.072 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:58.072 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.072 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.072 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:58.072 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.589903+0000", 2020-01-16T00:12:58.072 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999952 2020-01-16T00:12:58.073 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.073 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.073 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:58.073 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.589898+0000", 2020-01-16T00:12:58.073 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4389e-05 2020-01-16T00:12:58.073 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.073 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.073 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:58.074 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.589922+0000", 2020-01-16T00:12:58.074 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4159999999999998e-05 2020-01-16T00:12:58.074 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.074 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.074 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:58.074 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.589957+0000", 2020-01-16T00:12:58.074 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7368999999999999e-05 2020-01-16T00:12:58.075 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.075 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.075 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:58.075 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.589974+0000", 2020-01-16T00:12:58.075 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000130407 2020-01-16T00:12:58.075 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.075 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.075 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:58.076 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590104+0000", 2020-01-16T00:12:58.076 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00040217699999999998 2020-01-16T00:12:58.076 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.076 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.076 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:58.076 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590506+0000", 2020-01-16T00:12:58.077 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0090030230000000006 2020-01-16T00:12:58.077 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.077 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.077 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:58.077 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599510+0000", 2020-01-16T00:12:58.077 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0096065560000000005 2020-01-16T00:12:58.077 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.078 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:58.078 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.078 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.078 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.078 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1242 3.a 3:5c245412:::smithi12313934-19:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:12:58.078 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.590003+0000", 2020-01-16T00:12:58.078 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 15.426391087000001, 2020-01-16T00:12:58.078 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010247029, 2020-01-16T00:12:58.079 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:58.079 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:58.079 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:58.079 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:58.079 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:58.079 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1242 2020-01-16T00:12:58.079 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.079 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:58.080 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.080 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:58.080 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590003+0000", 2020-01-16T00:12:58.080 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:58.080 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.080 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.080 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:58.080 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590003+0000", 2020-01-16T00:12:58.081 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:58.081 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.081 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.081 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:58.081 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590002+0000", 2020-01-16T00:12:58.081 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0108e-05 2020-01-16T00:12:58.081 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.081 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.082 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:58.082 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590012+0000", 2020-01-16T00:12:58.082 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5318000000000001e-05 2020-01-16T00:12:58.082 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.082 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.082 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:58.082 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590027+0000", 2020-01-16T00:12:58.082 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0720000000000001e-05 2020-01-16T00:12:58.083 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.083 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.083 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:58.083 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590038+0000", 2020-01-16T00:12:58.083 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0095349949999999992 2020-01-16T00:12:58.083 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.083 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.084 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:58.084 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599573+0000", 2020-01-16T00:12:58.084 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017107600000000001 2020-01-16T00:12:58.084 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.084 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.084 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:58.084 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599744+0000", 2020-01-16T00:12:58.084 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00050659499999999998 2020-01-16T00:12:58.085 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.085 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.085 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:58.085 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.600251+0000", 2020-01-16T00:12:58.085 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010247029 2020-01-16T00:12:58.085 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.085 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:58.085 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.086 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.086 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.086 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1243 3.a 3:5c245412:::smithi12313934-19:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:12:58.086 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.590095+0000", 2020-01-16T00:12:58.086 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 15.426299677999999, 2020-01-16T00:12:58.086 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010591455, 2020-01-16T00:12:58.087 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:58.087 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:58.087 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:58.087 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:58.087 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:58.087 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1243 2020-01-16T00:12:58.087 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.088 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:58.088 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.088 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:58.088 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590095+0000", 2020-01-16T00:12:58.088 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:58.088 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.088 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.088 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:58.089 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590095+0000", 2020-01-16T00:12:58.089 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:58.089 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.089 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.089 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:58.089 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590093+0000", 2020-01-16T00:12:58.089 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5619999999999993e-06 2020-01-16T00:12:58.089 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.090 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.090 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:58.090 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590102+0000", 2020-01-16T00:12:58.090 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.259e-05 2020-01-16T00:12:58.090 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.090 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.090 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:58.090 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590115+0000", 2020-01-16T00:12:58.091 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8730000000000001e-06 2020-01-16T00:12:58.091 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.091 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.091 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:58.091 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590122+0000", 2020-01-16T00:12:58.091 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010163165 2020-01-16T00:12:58.091 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.092 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.092 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:58.092 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.600285+0000", 2020-01-16T00:12:58.092 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0128000000000002e-05 2020-01-16T00:12:58.092 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.092 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.092 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:58.092 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.600375+0000", 2020-01-16T00:12:58.093 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000311312 2020-01-16T00:12:58.093 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.093 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.093 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:58.093 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.600686+0000", 2020-01-16T00:12:58.093 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010591455 2020-01-16T00:12:58.093 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.093 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:58.094 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.094 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.094 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.094 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1648 3.a 3:54f2c117:::smithi12313934-41:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:12:58.094 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:51.847266+0000", 2020-01-16T00:12:58.094 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 6.1691290419999998, 2020-01-16T00:12:58.094 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0066949069999999999, 2020-01-16T00:12:58.094 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:58.095 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:58.095 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:58.095 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:58.095 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:58.095 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1648 2020-01-16T00:12:58.095 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.095 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:58.095 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.096 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:58.096 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:51.847266+0000", 2020-01-16T00:12:58.096 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:58.096 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.096 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.096 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:58.097 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:51.847266+0000", 2020-01-16T00:12:58.097 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:58.097 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.097 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.097 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:58.097 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:51.847265+0000", 2020-01-16T00:12:58.097 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6439999999999999e-06 2020-01-16T00:12:58.098 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.098 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.098 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:58.098 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:51.847269+0000", 2020-01-16T00:12:58.098 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.9759999999999998e-06 2020-01-16T00:12:58.098 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.098 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.098 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:58.099 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:51.847276+0000", 2020-01-16T00:12:58.099 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3040000000000004e-06 2020-01-16T00:12:58.099 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.099 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.099 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:58.099 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:51.847282+0000", 2020-01-16T00:12:58.099 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00557239 2020-01-16T00:12:58.099 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.100 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.100 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:58.100 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:51.852854+0000", 2020-01-16T00:12:58.100 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00032399799999999998 2020-01-16T00:12:58.100 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.100 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.100 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:58.100 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:51.853178+0000", 2020-01-16T00:12:58.101 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00078245500000000004 2020-01-16T00:12:58.101 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.101 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.101 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:58.101 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:51.853960+0000", 2020-01-16T00:12:58.101 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0066949069999999999 2020-01-16T00:12:58.101 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.102 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:58.102 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.102 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.102 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.102 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1654 3.a 3:5069665e:::smithi12313934-2:head [write 1614880~581041] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:12:58.102 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.714839+0000", 2020-01-16T00:12:58.102 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.3015556899999998, 2020-01-16T00:12:58.102 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013588093000000001, 2020-01-16T00:12:58.103 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:58.103 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:58.103 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:58.103 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:58.103 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:58.104 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1654 2020-01-16T00:12:58.104 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.104 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:58.104 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.104 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:58.104 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.714839+0000", 2020-01-16T00:12:58.104 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:58.104 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.105 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.105 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:58.105 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.714839+0000", 2020-01-16T00:12:58.105 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:12:58.105 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.105 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.105 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:58.105 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.714836+0000", 2020-01-16T00:12:58.106 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00054657999999999996 2020-01-16T00:12:58.106 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.106 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.106 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:58.106 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.715382+0000", 2020-01-16T00:12:58.106 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7743000000000001e-05 2020-01-16T00:12:58.106 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.107 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.107 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:58.107 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.715420+0000", 2020-01-16T00:12:58.107 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.221e-05 2020-01-16T00:12:58.107 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.107 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.107 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:58.108 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.715432+0000", 2020-01-16T00:12:58.108 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2487000000000001e-05 2020-01-16T00:12:58.108 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.108 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.108 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:58.108 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.715485+0000", 2020-01-16T00:12:58.108 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000726284 2020-01-16T00:12:58.109 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.109 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.109 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:58.109 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.716211+0000", 2020-01-16T00:12:58.109 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0067930899999999999 2020-01-16T00:12:58.109 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.109 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.109 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:58.110 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.723004+0000", 2020-01-16T00:12:58.110 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012579400000000001 2020-01-16T00:12:58.110 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.110 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.110 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:58.110 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.723130+0000", 2020-01-16T00:12:58.110 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013465800000000001 2020-01-16T00:12:58.110 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.111 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.111 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:12:58.111 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.723265+0000", 2020-01-16T00:12:58.111 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023307250000000001 2020-01-16T00:12:58.111 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.111 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.111 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:58.111 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.725595+0000", 2020-01-16T00:12:58.112 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012582139999999999 2020-01-16T00:12:58.112 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.112 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.112 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:58.112 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.726853+0000", 2020-01-16T00:12:58.112 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014479110000000001 2020-01-16T00:12:58.112 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.112 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.113 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:58.113 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.728301+0000", 2020-01-16T00:12:58.113 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6866000000000007e-05 2020-01-16T00:12:58.113 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.113 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.113 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:58.113 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.728368+0000", 2020-01-16T00:12:58.114 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8740999999999999e-05 2020-01-16T00:12:58.114 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.114 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.114 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:58.114 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.728427+0000", 2020-01-16T00:12:58.114 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013588093000000001 2020-01-16T00:12:58.114 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.114 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:58.115 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.115 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.115 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.115 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1655 3.a 3:5069665e:::smithi12313934-2:head [write 2785791~71399] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:12:58.115 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.718297+0000", 2020-01-16T00:12:58.115 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.2980978370000003, 2020-01-16T00:12:58.115 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013817354, 2020-01-16T00:12:58.115 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:58.116 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:58.116 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:58.116 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:58.116 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:58.116 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1655 2020-01-16T00:12:58.116 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.116 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:58.117 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.117 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:58.117 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718297+0000", 2020-01-16T00:12:58.117 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:58.117 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.117 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.117 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:58.118 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718297+0000", 2020-01-16T00:12:58.118 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:12:58.118 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.118 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.118 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:58.118 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718294+0000", 2020-01-16T00:12:58.118 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000103558 2020-01-16T00:12:58.119 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.119 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.119 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:58.119 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718397+0000", 2020-01-16T00:12:58.119 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9287e-05 2020-01-16T00:12:58.119 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.119 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.119 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:58.120 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718417+0000", 2020-01-16T00:12:58.120 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6680000000000001e-06 2020-01-16T00:12:58.120 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.120 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.120 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:58.120 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718424+0000", 2020-01-16T00:12:58.120 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2701999999999994e-05 2020-01-16T00:12:58.120 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.121 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.121 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:58.121 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718517+0000", 2020-01-16T00:12:58.121 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3893000000000006e-05 2020-01-16T00:12:58.121 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.121 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.121 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:58.121 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718581+0000", 2020-01-16T00:12:58.122 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0059630079999999997 2020-01-16T00:12:58.122 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.122 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.122 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:58.122 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.724544+0000", 2020-01-16T00:12:58.122 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000309108 2020-01-16T00:12:58.122 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.122 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.123 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:58.123 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.724853+0000", 2020-01-16T00:12:58.123 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.005469794 2020-01-16T00:12:58.123 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.123 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.123 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:58.123 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.730323+0000", 2020-01-16T00:12:58.124 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018506100000000001 2020-01-16T00:12:58.124 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.124 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.124 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:58.124 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.730508+0000", 2020-01-16T00:12:58.124 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016729099999999999 2020-01-16T00:12:58.124 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.124 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.125 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:12:58.125 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.730675+0000", 2020-01-16T00:12:58.125 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00094193300000000005 2020-01-16T00:12:58.125 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.125 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.125 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:58.125 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731617+0000", 2020-01-16T00:12:58.125 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000214613 2020-01-16T00:12:58.126 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.126 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.126 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:58.126 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731832+0000", 2020-01-16T00:12:58.126 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018326400000000001 2020-01-16T00:12:58.126 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.126 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.126 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:58.127 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732015+0000", 2020-01-16T00:12:58.127 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0392e-05 2020-01-16T00:12:58.127 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.127 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.127 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:58.127 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732075+0000", 2020-01-16T00:12:58.128 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8677999999999998e-05 2020-01-16T00:12:58.128 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.128 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.128 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:58.128 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732114+0000", 2020-01-16T00:12:58.128 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013817354 2020-01-16T00:12:58.128 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.128 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:58.129 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.129 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.129 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.129 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1656 3.a 3:5069665e:::smithi12313934-2:head [setxattr _header (55) in=7b,truncate 2857190] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:12:58.129 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.718453+0000", 2020-01-16T00:12:58.129 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.2979411560000003, 2020-01-16T00:12:58.129 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014083462, 2020-01-16T00:12:58.130 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:58.130 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:58.130 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:58.130 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:58.130 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:58.130 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1656 2020-01-16T00:12:58.130 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.130 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:58.131 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.131 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:58.131 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718453+0000", 2020-01-16T00:12:58.131 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:58.131 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.131 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.131 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:58.131 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718453+0000", 2020-01-16T00:12:58.132 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999828 2020-01-16T00:12:58.132 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.132 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.132 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:58.132 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718436+0000", 2020-01-16T00:12:58.132 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4051999999999999e-05 2020-01-16T00:12:58.132 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.132 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.133 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:58.133 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718460+0000", 2020-01-16T00:12:58.133 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6380000000000004e-06 2020-01-16T00:12:58.133 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.133 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.133 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:58.133 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718468+0000", 2020-01-16T00:12:58.134 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6750000000000001e-06 2020-01-16T00:12:58.134 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.134 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.134 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:58.134 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718472+0000", 2020-01-16T00:12:58.134 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016218499999999999 2020-01-16T00:12:58.134 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.134 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.135 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:58.135 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718635+0000", 2020-01-16T00:12:58.135 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2765e-05 2020-01-16T00:12:58.135 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.135 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.135 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:58.135 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718697+0000", 2020-01-16T00:12:58.135 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0062239249999999999 2020-01-16T00:12:58.136 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.136 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.136 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:58.136 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.724921+0000", 2020-01-16T00:12:58.136 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1527999999999999e-05 2020-01-16T00:12:58.136 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.136 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.136 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:58.137 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.724973+0000", 2020-01-16T00:12:58.137 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0060569719999999999 2020-01-16T00:12:58.137 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.137 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.137 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:58.137 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731030+0000", 2020-01-16T00:12:58.138 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5859999999999996e-05 2020-01-16T00:12:58.138 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.138 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.138 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:58.138 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731096+0000", 2020-01-16T00:12:58.138 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000112018 2020-01-16T00:12:58.138 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.139 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.139 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:12:58.139 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731208+0000", 2020-01-16T00:12:58.139 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000945889 2020-01-16T00:12:58.139 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.139 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.139 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:58.139 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732154+0000", 2020-01-16T00:12:58.140 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0834000000000003e-05 2020-01-16T00:12:58.140 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.140 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.140 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:58.140 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732184+0000", 2020-01-16T00:12:58.140 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028587600000000001 2020-01-16T00:12:58.140 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.141 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.141 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:58.141 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732470+0000", 2020-01-16T00:12:58.141 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1953999999999999e-05 2020-01-16T00:12:58.141 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.141 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.141 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:58.141 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732502+0000", 2020-01-16T00:12:58.142 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4644999999999998e-05 2020-01-16T00:12:58.142 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.142 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.142 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:58.142 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732537+0000", 2020-01-16T00:12:58.142 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014083462 2020-01-16T00:12:58.142 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.143 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:58.143 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.143 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.143 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.143 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1657 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-01-16T00:12:58.143 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.718495+0000", 2020-01-16T00:12:58.143 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.2978994909999999, 2020-01-16T00:12:58.143 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014299630000000001, 2020-01-16T00:12:58.144 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:58.144 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:58.144 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:58.144 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:58.144 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:58.144 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1657 2020-01-16T00:12:58.144 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.145 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:58.145 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.145 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:58.145 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718495+0000", 2020-01-16T00:12:58.145 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:58.145 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.145 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.145 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:58.146 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718495+0000", 2020-01-16T00:12:58.146 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:12:58.146 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.146 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.146 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:58.146 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718494+0000", 2020-01-16T00:12:58.146 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8860000000000003e-06 2020-01-16T00:12:58.147 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.147 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.147 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:58.147 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718499+0000", 2020-01-16T00:12:58.147 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9786000000000002e-05 2020-01-16T00:12:58.147 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.148 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.148 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:58.148 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718539+0000", 2020-01-16T00:12:58.148 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1869999999999999e-05 2020-01-16T00:12:58.148 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.148 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.148 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:58.149 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718561+0000", 2020-01-16T00:12:58.149 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000172206 2020-01-16T00:12:58.149 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.149 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.149 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:58.149 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718733+0000", 2020-01-16T00:12:58.149 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0662999999999998e-05 2020-01-16T00:12:58.150 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.150 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.150 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:58.150 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718793+0000", 2020-01-16T00:12:58.150 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.006217958 2020-01-16T00:12:58.150 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.150 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.150 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:58.151 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.725011+0000", 2020-01-16T00:12:58.151 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1625999999999996e-05 2020-01-16T00:12:58.151 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.151 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.151 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:58.151 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.725073+0000", 2020-01-16T00:12:58.151 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0063923119999999998 2020-01-16T00:12:58.151 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.152 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.152 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:58.152 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731465+0000", 2020-01-16T00:12:58.152 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2534000000000005e-05 2020-01-16T00:12:58.152 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.152 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.152 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:58.152 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731558+0000", 2020-01-16T00:12:58.153 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010238809999999999 2020-01-16T00:12:58.153 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.153 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.153 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:58.153 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732582+0000", 2020-01-16T00:12:58.153 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000100918 2020-01-16T00:12:58.153 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.153 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.154 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:58.154 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732683+0000", 2020-01-16T00:12:58.154 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000112064 2020-01-16T00:12:58.154 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.154 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.154 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:58.154 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732795+0000", 2020-01-16T00:12:58.155 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014299630000000001 2020-01-16T00:12:58.155 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.155 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:58.155 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.155 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.155 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.155 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.4488.0:1708 3.c e30/20 3:378f3722:::smithi12313934-50:head v 30'89, mlcod=29'87)", 2020-01-16T00:12:58.155 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.808249+0000", 2020-01-16T00:12:58.156 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.2081457440000003, 2020-01-16T00:12:58.156 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0067795160000000002, 2020-01-16T00:12:58.156 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:58.156 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:58.156 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:58.156 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.156 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:58.156 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.808249+0000", 2020-01-16T00:12:58.157 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:58.157 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.157 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.157 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:58.157 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.808249+0000", 2020-01-16T00:12:58.157 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:12:58.157 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.157 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.158 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:58.158 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.808246+0000", 2020-01-16T00:12:58.158 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00046200000000000001 2020-01-16T00:12:58.158 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.158 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.158 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:58.159 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.808708+0000", 2020-01-16T00:12:58.159 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.702e-05 2020-01-16T00:12:58.159 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.159 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.159 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:58.159 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.808725+0000", 2020-01-16T00:12:58.159 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4920000000000001e-06 2020-01-16T00:12:58.160 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.160 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.160 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:58.160 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.808731+0000", 2020-01-16T00:12:58.160 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4094000000000001e-05 2020-01-16T00:12:58.160 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.160 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.160 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:58.161 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.808785+0000", 2020-01-16T00:12:58.161 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.7359999999999999e-05 2020-01-16T00:12:58.161 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.161 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.161 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:58.161 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.808812+0000", 2020-01-16T00:12:58.161 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0061494260000000004 2020-01-16T00:12:58.161 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.162 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.162 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:58.162 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.814962+0000", 2020-01-16T00:12:58.162 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6660000000000002e-05 2020-01-16T00:12:58.162 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.162 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.162 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:58.162 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.815028+0000", 2020-01-16T00:12:58.163 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0067795160000000002 2020-01-16T00:12:58.163 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.163 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:58.163 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.163 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.163 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.163 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1834 3.a 3:54f2c117:::smithi12313934-41:head [write 443751~631882] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:12:58.163 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:56.278017+0000", 2020-01-16T00:12:58.164 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 1.7383774999999999, 2020-01-16T00:12:58.164 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0076931760000000004, 2020-01-16T00:12:58.164 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:58.164 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:58.164 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:58.164 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:58.164 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:58.165 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1834 2020-01-16T00:12:58.165 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.165 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:58.165 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.165 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:58.165 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.278017+0000", 2020-01-16T00:12:58.165 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:58.165 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.166 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.166 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:58.166 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.278017+0000", 2020-01-16T00:12:58.166 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:12:58.166 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.166 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.166 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:58.166 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.278013+0000", 2020-01-16T00:12:58.167 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010862669999999999 2020-01-16T00:12:58.167 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.167 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.167 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:58.167 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.279100+0000", 2020-01-16T00:12:58.167 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8034000000000002e-05 2020-01-16T00:12:58.167 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.167 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.168 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:58.168 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.279138+0000", 2020-01-16T00:12:58.168 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7955999999999999e-05 2020-01-16T00:12:58.168 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.168 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.168 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:58.169 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.279156+0000", 2020-01-16T00:12:58.169 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000141691 2020-01-16T00:12:58.169 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.169 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.169 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:58.169 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.279297+0000", 2020-01-16T00:12:58.169 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00025750000000000002 2020-01-16T00:12:58.169 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.170 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.170 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:58.170 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.279555+0000", 2020-01-16T00:12:58.170 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00031473999999999999 2020-01-16T00:12:58.170 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.170 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.170 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:12:58.171 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.279869+0000", 2020-01-16T00:12:58.171 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003741192 2020-01-16T00:12:58.171 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.171 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.171 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:58.171 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.283611+0000", 2020-01-16T00:12:58.171 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00075302099999999999 2020-01-16T00:12:58.171 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.172 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.172 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:58.172 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.284364+0000", 2020-01-16T00:12:58.172 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011594509999999999 2020-01-16T00:12:58.172 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.172 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.172 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:58.172 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.285523+0000", 2020-01-16T00:12:58.173 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000114127 2020-01-16T00:12:58.173 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.173 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.173 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:58.173 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.285637+0000", 2020-01-16T00:12:58.173 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2972999999999994e-05 2020-01-16T00:12:58.173 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.174 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.174 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:58.174 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.285710+0000", 2020-01-16T00:12:58.174 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0076931760000000004 2020-01-16T00:12:58.174 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.174 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:58.174 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.174 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.175 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.175 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1835 3.a 3:54f2c117:::smithi12313934-41:head [write 1810907~765058] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:12:58.175 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:56.286248+0000", 2020-01-16T00:12:58.175 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 1.7301465810000001, 2020-01-16T00:12:58.175 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0067134459999999996, 2020-01-16T00:12:58.175 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:58.175 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:58.175 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:58.176 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:58.176 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:58.176 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1835 2020-01-16T00:12:58.176 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.176 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:58.176 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.176 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:58.176 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.286248+0000", 2020-01-16T00:12:58.177 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:58.177 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.177 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.177 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:58.177 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.286248+0000", 2020-01-16T00:12:58.177 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999886 2020-01-16T00:12:58.177 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.178 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.178 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:58.178 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.286237+0000", 2020-01-16T00:12:58.178 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00073283500000000004 2020-01-16T00:12:58.178 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.178 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.179 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:58.179 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.286969+0000", 2020-01-16T00:12:58.179 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.5443e-05 2020-01-16T00:12:58.179 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.179 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.179 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:58.179 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.286995+0000", 2020-01-16T00:12:58.179 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9250000000000007e-06 2020-01-16T00:12:58.180 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.180 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.180 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:58.180 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.287005+0000", 2020-01-16T00:12:58.180 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7592999999999997e-05 2020-01-16T00:12:58.180 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.180 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.181 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:58.181 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.287042+0000", 2020-01-16T00:12:58.181 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8280000000000001e-05 2020-01-16T00:12:58.181 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.181 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.181 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:58.181 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.287141+0000", 2020-01-16T00:12:58.181 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000110246 2020-01-16T00:12:58.182 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.182 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.182 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:12:58.182 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.287251+0000", 2020-01-16T00:12:58.182 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0026245449999999998 2020-01-16T00:12:58.182 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.182 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.182 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:58.183 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.289875+0000", 2020-01-16T00:12:58.183 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2745e-05 2020-01-16T00:12:58.183 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.183 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.183 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:58.183 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.289898+0000", 2020-01-16T00:12:58.183 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002996154 2020-01-16T00:12:58.183 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.184 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.184 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:58.184 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.292894+0000", 2020-01-16T00:12:58.184 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5630999999999998e-05 2020-01-16T00:12:58.184 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.184 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.184 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:58.184 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.292930+0000", 2020-01-16T00:12:58.185 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1398e-05 2020-01-16T00:12:58.185 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.185 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.185 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:58.185 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.292961+0000", 2020-01-16T00:12:58.185 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0067134459999999996 2020-01-16T00:12:58.185 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.186 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:58.186 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.186 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.186 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.186 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1917 3.a 3:5069665e:::smithi12313934-2:head [write 2113302~674388] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:12:58.186 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:57.330712+0000", 2020-01-16T00:12:58.186 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.68568297600000006, 2020-01-16T00:12:58.186 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012781043000000001, 2020-01-16T00:12:58.187 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:58.187 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:58.187 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:58.187 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:58.187 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:58.187 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1917 2020-01-16T00:12:58.187 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.187 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:58.188 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.188 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:58.188 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.330712+0000", 2020-01-16T00:12:58.188 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:58.188 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.188 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.188 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:58.189 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.330712+0000", 2020-01-16T00:12:58.189 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:12:58.189 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.189 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.189 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:58.189 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.330709+0000", 2020-01-16T00:12:58.190 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00071550499999999996 2020-01-16T00:12:58.190 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.190 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.190 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:58.190 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331424+0000", 2020-01-16T00:12:58.190 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3354e-05 2020-01-16T00:12:58.190 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.190 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.191 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:58.191 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331447+0000", 2020-01-16T00:12:58.191 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8200000000000008e-06 2020-01-16T00:12:58.191 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.191 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.191 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:58.191 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331457+0000", 2020-01-16T00:12:58.191 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5557999999999997e-05 2020-01-16T00:12:58.192 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.192 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.192 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:58.192 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331533+0000", 2020-01-16T00:12:58.192 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030925000000000001 2020-01-16T00:12:58.192 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.192 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.192 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:58.193 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331842+0000", 2020-01-16T00:12:58.193 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0078507860000000002 2020-01-16T00:12:58.193 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.193 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.193 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:58.193 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.339693+0000", 2020-01-16T00:12:58.193 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9321000000000007e-05 2020-01-16T00:12:58.193 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.194 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.194 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:58.194 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.339792+0000", 2020-01-16T00:12:58.194 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015147700000000001 2020-01-16T00:12:58.194 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.194 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.194 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:12:58.194 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.339944+0000", 2020-01-16T00:12:58.195 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002663066 2020-01-16T00:12:58.195 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.195 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.195 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:58.195 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.342607+0000", 2020-01-16T00:12:58.195 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0001552 2020-01-16T00:12:58.195 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.195 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.196 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:58.196 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.342762+0000", 2020-01-16T00:12:58.196 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00063636900000000002 2020-01-16T00:12:58.196 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.196 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.196 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:58.196 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343398+0000", 2020-01-16T00:12:58.196 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9687000000000001e-05 2020-01-16T00:12:58.197 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.197 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.197 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:58.197 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343438+0000", 2020-01-16T00:12:58.197 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4647e-05 2020-01-16T00:12:58.197 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.197 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.198 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:58.198 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343493+0000", 2020-01-16T00:12:58.198 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012781043000000001 2020-01-16T00:12:58.198 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.198 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:58.198 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.198 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.198 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.199 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1918 3.a 3:5069665e:::smithi12313934-2:head [write 3335030~144019] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:12:58.199 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:57.334874+0000", 2020-01-16T00:12:58.199 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.68152032399999996, 2020-01-16T00:12:58.199 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0090004249999999994, 2020-01-16T00:12:58.199 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:58.199 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:58.200 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:58.200 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:58.200 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:58.200 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1918 2020-01-16T00:12:58.200 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.200 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:58.200 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.201 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:58.201 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.334874+0000", 2020-01-16T00:12:58.201 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:58.201 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.201 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.201 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:58.201 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.334874+0000", 2020-01-16T00:12:58.202 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:12:58.202 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.202 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.202 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:58.202 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.334870+0000", 2020-01-16T00:12:58.202 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000113143 2020-01-16T00:12:58.202 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.202 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.203 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:58.203 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.334983+0000", 2020-01-16T00:12:58.203 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4298000000000003e-05 2020-01-16T00:12:58.203 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.203 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.203 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:58.204 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335018+0000", 2020-01-16T00:12:58.204 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6860000000000001e-05 2020-01-16T00:12:58.204 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.204 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.204 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:58.204 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335035+0000", 2020-01-16T00:12:58.204 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000111932 2020-01-16T00:12:58.205 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.205 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.205 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:58.205 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335146+0000", 2020-01-16T00:12:58.205 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012407400000000001 2020-01-16T00:12:58.205 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.205 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.205 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:58.206 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335271+0000", 2020-01-16T00:12:58.206 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0056080180000000002 2020-01-16T00:12:58.206 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.206 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.206 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:58.206 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.340879+0000", 2020-01-16T00:12:58.206 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012836600000000001 2020-01-16T00:12:58.206 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.207 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.207 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:58.208 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.341007+0000", 2020-01-16T00:12:58.208 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019871900000000001 2020-01-16T00:12:58.209 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.209 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.209 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:12:58.209 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.341206+0000", 2020-01-16T00:12:58.209 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001501528 2020-01-16T00:12:58.210 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.210 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.210 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:58.210 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.342707+0000", 2020-01-16T00:12:58.210 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000300422 2020-01-16T00:12:58.210 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.210 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.210 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:58.211 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343008+0000", 2020-01-16T00:12:58.211 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00074073800000000001 2020-01-16T00:12:58.211 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.211 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.211 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:58.211 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343748+0000", 2020-01-16T00:12:58.211 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8270000000000003e-05 2020-01-16T00:12:58.212 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.212 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.212 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:58.212 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343807+0000", 2020-01-16T00:12:58.212 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8032000000000002e-05 2020-01-16T00:12:58.212 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.212 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.212 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:58.213 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343875+0000", 2020-01-16T00:12:58.213 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0090004249999999994 2020-01-16T00:12:58.213 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.213 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:58.213 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.213 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.213 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.213 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1919 3.a 3:5069665e:::smithi12313934-2:head [setxattr _header (55) in=7b,truncate 3479049] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:12:58.214 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:57.335061+0000", 2020-01-16T00:12:58.214 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.68133359599999999, 2020-01-16T00:12:58.214 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0089426209999999996, 2020-01-16T00:12:58.214 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:58.214 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:12:58.214 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:58.214 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:58.215 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:58.215 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1919 2020-01-16T00:12:58.215 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.215 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:58.215 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.215 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:58.215 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335061+0000", 2020-01-16T00:12:58.215 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:58.216 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.216 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.216 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:58.216 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335061+0000", 2020-01-16T00:12:58.216 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:12:58.216 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.216 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.216 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:58.217 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335059+0000", 2020-01-16T00:12:58.217 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5460000000000001e-05 2020-01-16T00:12:58.217 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.217 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.217 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:58.217 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335075+0000", 2020-01-16T00:12:58.217 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6120000000000002e-05 2020-01-16T00:12:58.217 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.218 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.218 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:58.218 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335091+0000", 2020-01-16T00:12:58.218 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0587999999999999e-05 2020-01-16T00:12:58.218 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.218 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.218 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:58.219 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335101+0000", 2020-01-16T00:12:58.219 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00027255500000000001 2020-01-16T00:12:58.219 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.219 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.219 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:58.219 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335374+0000", 2020-01-16T00:12:58.220 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016375999999999999 2020-01-16T00:12:58.220 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.220 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.220 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:58.220 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335538+0000", 2020-01-16T00:12:58.220 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0063584440000000004 2020-01-16T00:12:58.220 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.221 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.221 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:58.221 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.341896+0000", 2020-01-16T00:12:58.221 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012864300000000001 2020-01-16T00:12:58.221 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.221 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.221 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:58.221 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.342025+0000", 2020-01-16T00:12:58.222 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019981299999999999 2020-01-16T00:12:58.222 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.222 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.222 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:12:58.222 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.342225+0000", 2020-01-16T00:12:58.222 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00048735800000000002 2020-01-16T00:12:58.222 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.223 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.223 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:12:58.223 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.342712+0000", 2020-01-16T00:12:58.223 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00047628299999999998 2020-01-16T00:12:58.223 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.223 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.223 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:58.223 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343188+0000", 2020-01-16T00:12:58.224 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000718606 2020-01-16T00:12:58.224 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.224 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.224 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:12:58.224 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343907+0000", 2020-01-16T00:12:58.224 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7583999999999998e-05 2020-01-16T00:12:58.224 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.224 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.225 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:12:58.225 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343944+0000", 2020-01-16T00:12:58.225 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9209000000000002e-05 2020-01-16T00:12:58.225 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.225 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.225 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:58.225 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.344004+0000", 2020-01-16T00:12:58.226 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0089426209999999996 2020-01-16T00:12:58.226 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.226 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:58.226 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.226 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.226 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.226 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1920 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e31)", 2020-01-16T00:12:58.226 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:57.335132+0000", 2020-01-16T00:12:58.227 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.68126284999999998, 2020-01-16T00:12:58.227 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0091378889999999997, 2020-01-16T00:12:58.227 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:12:58.227 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:12:58.227 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:12:58.227 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:12:58.227 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:12:58.227 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1920 2020-01-16T00:12:58.228 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.228 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:12:58.228 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.228 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:12:58.228 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335132+0000", 2020-01-16T00:12:58.228 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:12:58.228 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.228 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.229 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:12:58.229 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335132+0000", 2020-01-16T00:12:58.229 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:12:58.229 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.229 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.229 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:12:58.229 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335130+0000", 2020-01-16T00:12:58.230 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0339999999999997e-06 2020-01-16T00:12:58.230 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.230 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.230 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:12:58.230 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335138+0000", 2020-01-16T00:12:58.230 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5296999999999999e-05 2020-01-16T00:12:58.231 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.231 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.231 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:12:58.231 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335154+0000", 2020-01-16T00:12:58.231 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9110000000000005e-06 2020-01-16T00:12:58.231 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.231 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.231 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:12:58.232 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335164+0000", 2020-01-16T00:12:58.232 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00044850000000000001 2020-01-16T00:12:58.232 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.232 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.232 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:58.232 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335612+0000", 2020-01-16T00:12:58.232 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2644000000000003e-05 2020-01-16T00:12:58.232 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.233 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.233 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:12:58.233 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335705+0000", 2020-01-16T00:12:58.233 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0068223650000000004 2020-01-16T00:12:58.233 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.233 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.233 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:58.233 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.342527+0000", 2020-01-16T00:12:58.234 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1173999999999994e-05 2020-01-16T00:12:58.234 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.234 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.234 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:12:58.234 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.342588+0000", 2020-01-16T00:12:58.234 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014580769999999999 2020-01-16T00:12:58.235 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.235 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.235 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:12:58.235 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.344046+0000", 2020-01-16T00:12:58.236 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000106702 2020-01-16T00:12:58.236 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.236 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.237 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:12:58.237 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.344153+0000", 2020-01-16T00:12:58.237 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011658999999999999 2020-01-16T00:12:58.237 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:12:58.237 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:12:58.238 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:12:58.238 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.344270+0000", 2020-01-16T00:12:58.238 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0091378889999999997 2020-01-16T00:12:58.238 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.238 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:58.238 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.238 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:12:58.238 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:12:58.239 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:12:58.240 INFO:tasks.rados.rados.0.smithi123.stdout:926: expect (ObjNum 195 snap 0 seq_num 195) 2020-01-16T00:12:58.240 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:58.241+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:58.241 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-01-16T00:12:58.289 INFO:tasks.rados.rados.0.smithi123.stdout:927: expect (ObjNum 22 snap 0 seq_num 22) 2020-01-16T00:12:58.299 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-01-16T00:12:58.341 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:12:58.341+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:12:58.371 INFO:tasks.rados.rados.0.smithi123.stdout:928: finishing write tid 1 to smithi12313934-43 2020-01-16T00:12:58.371 INFO:tasks.rados.rados.0.smithi123.stdout:928: finishing write tid 2 to smithi12313934-43 2020-01-16T00:12:58.371 INFO:tasks.rados.rados.0.smithi123.stdout:928: finishing write tid 3 to smithi12313934-43 2020-01-16T00:12:58.372 INFO:tasks.rados.rados.0.smithi123.stdout:928: finishing write tid 5 to smithi12313934-43 2020-01-16T00:12:58.372 INFO:tasks.rados.rados.0.smithi123.stdout:928: finishing write tid 6 to smithi12313934-43 2020-01-16T00:12:58.372 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 43 v 115 (ObjNum 209 snap 0 seq_num 209) dirty exists 2020-01-16T00:12:58.372 INFO:tasks.rados.rados.0.smithi123.stdout:928: left oid 43 (ObjNum 209 snap 0 seq_num 209) 2020-01-16T00:12:58.372 INFO:tasks.rados.rados.0.smithi123.stdout:929: expect (ObjNum 191 snap 0 seq_num 191) 2020-01-16T00:12:58.375 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:12:58.375 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:12:58.375 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:12:58.377 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:12:58.385 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-01-16T00:12:58.442 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:58.441+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:12:58.455 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:12:58.486 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:12:58.520 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:12:58.525 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:58.537 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-01-16T00:12:58.546 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:12:58.547 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:12:58.545+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:12:58.569 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:12:58.574 INFO:tasks.rados.rados.0.smithi123.stdout:931: finishing write tid 1 to smithi12313934-14 2020-01-16T00:12:58.574 INFO:tasks.rados.rados.0.smithi123.stdout:931: finishing write tid 2 to smithi12313934-14 2020-01-16T00:12:58.575 INFO:tasks.rados.rados.0.smithi123.stdout:931: finishing write tid 3 to smithi12313934-14 2020-01-16T00:12:58.575 INFO:tasks.rados.rados.0.smithi123.stdout:924: done (15 left) 2020-01-16T00:12:58.575 INFO:tasks.rados.rados.0.smithi123.stdout:925: done (14 left) 2020-01-16T00:12:58.575 INFO:tasks.rados.rados.0.smithi123.stdout:926: done (13 left) 2020-01-16T00:12:58.575 INFO:tasks.rados.rados.0.smithi123.stdout:927: done (12 left) 2020-01-16T00:12:58.575 INFO:tasks.rados.rados.0.smithi123.stdout:928: done (11 left) 2020-01-16T00:12:58.575 INFO:tasks.rados.rados.0.smithi123.stdout:929: done (10 left) 2020-01-16T00:12:58.576 INFO:tasks.rados.rados.0.smithi123.stdout:941: write oid 42 current snap is 0 2020-01-16T00:12:58.576 INFO:tasks.rados.rados.0.smithi123.stdout:941: seq_num 215 ranges {413505=636419,1664805=689064,3038061=492071} 2020-01-16T00:12:58.586 INFO:tasks.rados.rados.0.smithi123.stdout:941: writing smithi12313934-42 from 413505 to 1049924 tid 1 2020-01-16T00:12:58.596 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5: {} 2020-01-16T00:12:58.596 INFO:tasks.rados.rados.0.smithi123.stdout:941: writing smithi12313934-42 from 1664805 to 2353869 tid 2 2020-01-16T00:12:58.604 INFO:tasks.rados.rados.0.smithi123.stdout:941: writing smithi12313934-42 from 3038061 to 3530132 tid 3 2020-01-16T00:12:58.605 INFO:tasks.rados.rados.0.smithi123.stdout:942: read oid 12 snap -1 2020-01-16T00:12:58.605 INFO:tasks.rados.rados.0.smithi123.stdout:942: expect (ObjNum 148 snap 0 seq_num 148) 2020-01-16T00:12:58.605 INFO:tasks.rados.rados.0.smithi123.stdout:943: read oid 3 snap -1 2020-01-16T00:12:58.605 INFO:tasks.rados.rados.0.smithi123.stdout:943: expect deleted 2020-01-16T00:12:58.605 INFO:tasks.rados.rados.0.smithi123.stdout:944: read oid 37 snap -1 2020-01-16T00:12:58.605 INFO:tasks.rados.rados.0.smithi123.stdout:944: expect deleted 2020-01-16T00:12:58.606 INFO:tasks.rados.rados.0.smithi123.stdout:945: write (excl) oid 45 current snap is 0 2020-01-16T00:12:58.606 INFO:tasks.rados.rados.0.smithi123.stdout:945: seq_num 216 ranges {430312=711668,1723287=549075,2781837=703343} 2020-01-16T00:12:58.611 INFO:tasks.rados.rados.0.smithi123.stdout:945: writing smithi12313934-45 from 430312 to 1141980 tid 1 2020-01-16T00:12:58.618 INFO:tasks.rados.rados.0.smithi123.stdout:945: writing smithi12313934-45 from 1723287 to 2272362 tid 2 2020-01-16T00:12:58.624 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:12:58.634 INFO:tasks.rados.rados.0.smithi123.stdout:945: writing smithi12313934-45 from 2781837 to 3485180 tid 3 2020-01-16T00:12:58.634 INFO:tasks.rados.rados.0.smithi123.stdout:946: read oid 44 snap -1 2020-01-16T00:12:58.634 INFO:tasks.rados.rados.0.smithi123.stdout:946: expect (ObjNum 43 snap 0 seq_num 43) 2020-01-16T00:12:58.634 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:58.634 INFO:tasks.rados.rados.0.smithi123.stdout:931: finishing write tid 5 to smithi12313934-14 2020-01-16T00:12:58.635 INFO:tasks.rados.rados.0.smithi123.stdout:931: finishing write tid 6 to smithi12313934-14 2020-01-16T00:12:58.635 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 14 v 119 (ObjNum 211 snap 0 seq_num 211) dirty exists 2020-01-16T00:12:58.635 INFO:tasks.rados.rados.0.smithi123.stdout:931: left oid 14 (ObjNum 211 snap 0 seq_num 211) 2020-01-16T00:12:58.635 INFO:tasks.rados.rados.0.smithi123.stdout:932: finishing write tid 1 to smithi12313934-33 2020-01-16T00:12:58.635 INFO:tasks.rados.rados.0.smithi123.stdout:931: done (15 left) 2020-01-16T00:12:58.635 INFO:tasks.rados.rados.0.smithi123.stdout:947: write (excl) oid 14 current snap is 0 2020-01-16T00:12:58.635 INFO:tasks.rados.rados.0.smithi123.stdout:947: seq_num 217 ranges {447119=786917,1665416=692733,2976554=514615} 2020-01-16T00:12:58.635 INFO:tasks.rados.rados.0.smithi123.stdout:947: writing smithi12313934-14 from 447119 to 1234036 tid 1 2020-01-16T00:12:58.641 INFO:tasks.rados.rados.0.smithi123.stdout:947: writing smithi12313934-14 from 1665416 to 2358149 tid 2 2020-01-16T00:12:58.644 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:12:58.645+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:12:58.652 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:12:58.652 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-01-16T00:12:58.652 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-01-16T00:12:58.653 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-01-16T00:12:58.653 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-01-16T00:12:58.653 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-01-16T00:12:58.654 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-01-16T00:12:58.655 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-01-16T00:12:58.667 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-01-16T00:12:58.667 INFO:tasks.rados.rados.0.smithi123.stdout:947: writing smithi12313934-14 from 2976554 to 3491169 tid 3 2020-01-16T00:12:58.667 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:12:58.667 INFO:tasks.rados.rados.0.smithi123.stdout:933: expect (ObjNum 200 snap 0 seq_num 200) 2020-01-16T00:12:58.669 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:58.669 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-01-16T00:12:58.754 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:12:58.753+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:12:58.816 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:58.825 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-01-16T00:12:58.866 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:12:58.865+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:12:58.931 INFO:tasks.rados.rados.0.smithi123.stdout:932: finishing write tid 2 to smithi12313934-33 2020-01-16T00:12:58.932 INFO:tasks.rados.rados.0.smithi123.stdout:934: expect (ObjNum 177 snap 0 seq_num 177) 2020-01-16T00:12:58.947 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:12:58.945+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:12:58.967 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:58.977 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-01-16T00:12:59.067 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:12:59.065+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:12:59.140 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:59.151 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:12:59.149+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:12:59.152 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-01-16T00:12:59.153 INFO:tasks.rados.rados.0.smithi123.stdout:935: finishing write tid 1 to smithi12313934-9 2020-01-16T00:12:59.153 INFO:tasks.rados.rados.0.smithi123.stdout:932: finishing write tid 3 to smithi12313934-33 2020-01-16T00:12:59.154 INFO:tasks.rados.rados.0.smithi123.stdout:932: finishing write tid 5 to smithi12313934-33 2020-01-16T00:12:59.154 INFO:tasks.rados.rados.0.smithi123.stdout:932: finishing write tid 6 to smithi12313934-33 2020-01-16T00:12:59.154 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 33 v 184 (ObjNum 212 snap 0 seq_num 212) dirty exists 2020-01-16T00:12:59.154 INFO:tasks.rados.rados.0.smithi123.stdout:932: left oid 33 (ObjNum 212 snap 0 seq_num 212) 2020-01-16T00:12:59.166 INFO:tasks.rados.rados.0.smithi123.stdout:935: finishing write tid 2 to smithi12313934-9 2020-01-16T00:12:59.166 INFO:tasks.rados.rados.0.smithi123.stdout:937: expect (ObjNum 137 snap 0 seq_num 137) 2020-01-16T00:12:59.250 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:59.248+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:12:59.252 INFO:tasks.rados.rados.0.smithi123.stdout:935: finishing write tid 3 to smithi12313934-9 2020-01-16T00:12:59.252 INFO:tasks.rados.rados.0.smithi123.stdout:935: finishing write tid 5 to smithi12313934-9 2020-01-16T00:12:59.252 INFO:tasks.rados.rados.0.smithi123.stdout:935: finishing write tid 6 to smithi12313934-9 2020-01-16T00:12:59.252 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 9 v 79 (ObjNum 213 snap 0 seq_num 213) dirty exists 2020-01-16T00:12:59.252 INFO:tasks.rados.rados.0.smithi123.stdout:935: left oid 9 (ObjNum 213 snap 0 seq_num 213) 2020-01-16T00:12:59.253 INFO:tasks.rados.rados.0.smithi123.stdout:939: expect (ObjNum 182 snap 0 seq_num 182) 2020-01-16T00:12:59.279 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:59.289 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-01-16T00:12:59.351 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:12:59.352+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:12:59.432 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:59.441 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-01-16T00:12:59.461 INFO:tasks.rados.rados.0.smithi123.stdout:936: expect (ObjNum 201 snap 0 seq_num 201) 2020-01-16T00:12:59.462 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:12:59.456+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:12:59.544 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:59.553 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:12:59.552+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:12:59.555 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-01-16T00:12:59.606 INFO:tasks.rados.rados.0.smithi123.stdout:938: expect (ObjNum 198 snap 0 seq_num 198) 2020-01-16T00:12:59.653 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:12:59.652+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:12:59.705 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:59.714 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-01-16T00:12:59.759 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:12:59.756+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:12:59.782 INFO:tasks.rados.rados.0.smithi123.stdout:940: finishing write tid 1 to smithi12313934-22 2020-01-16T00:12:59.782 INFO:tasks.rados.rados.0.smithi123.stdout:940: finishing write tid 2 to smithi12313934-22 2020-01-16T00:12:59.782 INFO:tasks.rados.rados.0.smithi123.stdout:940: finishing write tid 3 to smithi12313934-22 2020-01-16T00:12:59.782 INFO:tasks.rados.rados.0.smithi123.stdout:940: finishing write tid 5 to smithi12313934-22 2020-01-16T00:12:59.782 INFO:tasks.rados.rados.0.smithi123.stdout:940: finishing write tid 6 to smithi12313934-22 2020-01-16T00:12:59.782 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 22 v 26 (ObjNum 214 snap 0 seq_num 214) dirty exists 2020-01-16T00:12:59.783 INFO:tasks.rados.rados.0.smithi123.stdout:940: left oid 22 (ObjNum 214 snap 0 seq_num 214) 2020-01-16T00:12:59.783 INFO:tasks.rados.rados.0.smithi123.stdout:941: finishing write tid 1 to smithi12313934-42 2020-01-16T00:12:59.783 INFO:tasks.rados.rados.0.smithi123.stdout:932: done (15 left) 2020-01-16T00:12:59.783 INFO:tasks.rados.rados.0.smithi123.stdout:933: done (14 left) 2020-01-16T00:12:59.783 INFO:tasks.rados.rados.0.smithi123.stdout:934: done (13 left) 2020-01-16T00:12:59.783 INFO:tasks.rados.rados.0.smithi123.stdout:935: done (12 left) 2020-01-16T00:12:59.783 INFO:tasks.rados.rados.0.smithi123.stdout:936: done (11 left) 2020-01-16T00:12:59.784 INFO:tasks.rados.rados.0.smithi123.stdout:937: done (10 left) 2020-01-16T00:12:59.784 INFO:tasks.rados.rados.0.smithi123.stdout:938: done (9 left) 2020-01-16T00:12:59.784 INFO:tasks.rados.rados.0.smithi123.stdout:939: done (8 left) 2020-01-16T00:12:59.784 INFO:tasks.rados.rados.0.smithi123.stdout:940: done (7 left) 2020-01-16T00:12:59.784 INFO:tasks.rados.rados.0.smithi123.stdout:948: write (excl) oid 20 current snap is 0 2020-01-16T00:12:59.784 INFO:tasks.rados.rados.0.smithi123.stdout:948: seq_num 218 ranges {463926=462166,1723898=436391,2603977=725887} 2020-01-16T00:12:59.795 INFO:tasks.rados.rados.0.smithi123.stdout:948: writing smithi12313934-20 from 463926 to 926092 tid 1 2020-01-16T00:12:59.802 INFO:tasks.rados.rados.0.smithi123.stdout:948: writing smithi12313934-20 from 1723898 to 2160289 tid 2 2020-01-16T00:12:59.809 INFO:tasks.rados.rados.0.smithi123.stdout:948: writing smithi12313934-20 from 2603977 to 3329864 tid 3 2020-01-16T00:12:59.809 INFO:tasks.rados.rados.0.smithi123.stdout:949: write oid 8 current snap is 0 2020-01-16T00:12:59.809 INFO:tasks.rados.rados.0.smithi123.stdout:949: seq_num 219 ranges {480733=537415,1782380=696402,3031400=537159} 2020-01-16T00:12:59.815 INFO:tasks.rados.rados.0.smithi123.stdout:949: writing smithi12313934-8 from 480733 to 1018148 tid 1 2020-01-16T00:12:59.823 INFO:tasks.rados.rados.0.smithi123.stdout:949: writing smithi12313934-8 from 1782380 to 2478782 tid 2 2020-01-16T00:12:59.830 INFO:tasks.rados.rados.0.smithi123.stdout:949: writing smithi12313934-8 from 3031400 to 3568559 tid 3 2020-01-16T00:12:59.830 INFO:tasks.rados.rados.0.smithi123.stdout:950: read oid 11 snap -1 2020-01-16T00:12:59.831 INFO:tasks.rados.rados.0.smithi123.stdout:950: expect (ObjNum 191 snap 0 seq_num 191) 2020-01-16T00:12:59.831 INFO:tasks.rados.rados.0.smithi123.stdout:941: finishing write tid 2 to smithi12313934-42 2020-01-16T00:12:59.831 INFO:tasks.rados.rados.0.smithi123.stdout:942: expect (ObjNum 148 snap 0 seq_num 148) 2020-01-16T00:12:59.855 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:12:59.852+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:12:59.868 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:12:59.877 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-01-16T00:12:59.908 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2020-01-16T00:12:59.956 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:12:59.956+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:00.003 INFO:tasks.rados.rados.0.smithi123.stdout:941: finishing write tid 3 to smithi12313934-42 2020-01-16T00:13:00.004 INFO:tasks.rados.rados.0.smithi123.stdout:941: finishing write tid 5 to smithi12313934-42 2020-01-16T00:13:00.004 INFO:tasks.rados.rados.0.smithi123.stdout:941: finishing write tid 6 to smithi12313934-42 2020-01-16T00:13:00.004 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 42 v 91 (ObjNum 215 snap 0 seq_num 215) dirty exists 2020-01-16T00:13:00.004 INFO:tasks.rados.rados.0.smithi123.stdout:941: left oid 42 (ObjNum 215 snap 0 seq_num 215) 2020-01-16T00:13:00.004 INFO:tasks.rados.rados.0.smithi123.stdout:945: finishing write tid 1 to smithi12313934-45 2020-01-16T00:13:00.004 INFO:tasks.rados.rados.0.smithi123.stdout:945: finishing write tid 2 to smithi12313934-45 2020-01-16T00:13:00.004 INFO:tasks.rados.rados.0.smithi123.stdout:951: read oid 27 snap -1 2020-01-16T00:13:00.004 INFO:tasks.rados.rados.0.smithi123.stdout:951: expect deleted 2020-01-16T00:13:00.005 INFO:tasks.rados.rados.0.smithi123.stdout:941: done (10 left) 2020-01-16T00:13:00.005 INFO:tasks.rados.rados.0.smithi123.stdout:942: done (9 left) 2020-01-16T00:13:00.005 INFO:tasks.rados.rados.0.smithi123.stdout:943: done (8 left) 2020-01-16T00:13:00.005 INFO:tasks.rados.rados.0.smithi123.stdout:944: done (7 left) 2020-01-16T00:13:00.005 INFO:tasks.rados.rados.0.smithi123.stdout:952: read oid 39 snap -1 2020-01-16T00:13:00.005 INFO:tasks.rados.rados.0.smithi123.stdout:952: expect (ObjNum 194 snap 0 seq_num 194) 2020-01-16T00:13:00.005 INFO:tasks.rados.rados.0.smithi123.stdout:953: write oid 18 current snap is 0 2020-01-16T00:13:00.006 INFO:tasks.rados.rados.0.smithi123.stdout:953: seq_num 220 ranges {497540=612664,1840862=440060,3058823=464784} 2020-01-16T00:13:00.013 INFO:tasks.rados.rados.0.smithi123.stdout:953: writing smithi12313934-18 from 497540 to 1110204 tid 1 2020-01-16T00:13:00.019 INFO:tasks.rados.rados.0.smithi123.stdout:953: writing smithi12313934-18 from 1840862 to 2280922 tid 2 2020-01-16T00:13:00.026 INFO:tasks.rados.rados.0.smithi123.stdout:953: writing smithi12313934-18 from 3058823 to 3523607 tid 3 2020-01-16T00:13:00.026 INFO:tasks.rados.rados.0.smithi123.stdout:954: read oid 22 snap -1 2020-01-16T00:13:00.026 INFO:tasks.rados.rados.0.smithi123.stdout:954: expect (ObjNum 214 snap 0 seq_num 214) 2020-01-16T00:13:00.026 INFO:tasks.rados.rados.0.smithi123.stdout:955: write (excl) oid 12 current snap is 0 2020-01-16T00:13:00.026 INFO:tasks.rados.rados.0.smithi123.stdout:955: seq_num 221 ranges {514347=404266,1499344=700071,2686246=676056} 2020-01-16T00:13:00.031 INFO:tasks.rados.rados.0.smithi123.stdout:955: writing smithi12313934-12 from 514347 to 918613 tid 1 2020-01-16T00:13:00.036 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:00.039 INFO:tasks.rados.rados.0.smithi123.stdout:955: writing smithi12313934-12 from 1499344 to 2199415 tid 2 2020-01-16T00:13:00.051 INFO:tasks.rados.rados.0.smithi123.stdout:955: writing smithi12313934-12 from 2686246 to 3362302 tid 3 2020-01-16T00:13:00.051 INFO:tasks.rados.rados.0.smithi123.stdout:956: read oid 2 snap -1 2020-01-16T00:13:00.052 INFO:tasks.rados.rados.0.smithi123.stdout:956: expect (ObjNum 207 snap 0 seq_num 207) 2020-01-16T00:13:00.052 INFO:tasks.rados.rados.0.smithi123.stdout:957: read oid 47 snap -1 2020-01-16T00:13:00.052 INFO:tasks.rados.rados.0.smithi123.stdout:957: expect (ObjNum 133 snap 0 seq_num 133) 2020-01-16T00:13:00.052 INFO:tasks.rados.rados.0.smithi123.stdout:958: write oid 13 current snap is 0 2020-01-16T00:13:00.053 INFO:tasks.rados.rados.0.smithi123.stdout:958: seq_num 222 ranges {531154=479515,1557826=443729,2713669=487328,3679295=51859} 2020-01-16T00:13:00.054 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-01-16T00:13:00.055 INFO:tasks.rados.rados.0.smithi123.stdout:958: writing smithi12313934-13 from 531154 to 1010669 tid 1 2020-01-16T00:13:00.056 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:00.052+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:00.058 INFO:tasks.rados.rados.0.smithi123.stdout:958: writing smithi12313934-13 from 1557826 to 2001555 tid 2 2020-01-16T00:13:00.065 INFO:tasks.rados.rados.0.smithi123.stdout:958: writing smithi12313934-13 from 2713669 to 3200997 tid 3 2020-01-16T00:13:00.069 INFO:tasks.rados.rados.0.smithi123.stdout:958: writing smithi12313934-13 from 3679295 to 3731154 tid 4 2020-01-16T00:13:00.069 INFO:tasks.rados.rados.0.smithi123.stdout:959: read oid 49 snap -1 2020-01-16T00:13:00.070 INFO:tasks.rados.rados.0.smithi123.stdout:959: expect (ObjNum 196 snap 0 seq_num 196) 2020-01-16T00:13:00.070 INFO:tasks.rados.rados.0.smithi123.stdout:960: write (excl) oid 43 current snap is 0 2020-01-16T00:13:00.070 INFO:tasks.rados.rados.0.smithi123.stdout:960: seq_num 223 ranges {547961=554764,1616308=703740,2741092=698600} 2020-01-16T00:13:00.075 INFO:tasks.rados.rados.0.smithi123.stdout:960: writing smithi12313934-43 from 547961 to 1102725 tid 1 2020-01-16T00:13:00.082 INFO:tasks.rados.rados.0.smithi123.stdout:960: writing smithi12313934-43 from 1616308 to 2320048 tid 2 2020-01-16T00:13:00.089 INFO:tasks.rados.rados.0.smithi123.stdout:960: writing smithi12313934-43 from 2741092 to 3439692 tid 3 2020-01-16T00:13:00.089 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:00.089 INFO:tasks.rados.rados.0.smithi123.stdout:946: expect (ObjNum 43 snap 0 seq_num 43) 2020-01-16T00:13:00.159 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:00.156+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:00.169 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:00.179 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-01-16T00:13:00.197 INFO:tasks.rados.rados.0.smithi123.stdout:946: done (15 left) 2020-01-16T00:13:00.197 INFO:tasks.rados.rados.0.smithi123.stdout:961: read oid 34 snap -1 2020-01-16T00:13:00.197 INFO:tasks.rados.rados.0.smithi123.stdout:961: expect (ObjNum 146 snap 0 seq_num 146) 2020-01-16T00:13:00.197 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:00.198 INFO:tasks.rados.rados.0.smithi123.stdout:947: finishing write tid 1 to smithi12313934-14 2020-01-16T00:13:00.198 INFO:tasks.rados.rados.0.smithi123.stdout:945: finishing write tid 3 to smithi12313934-45 2020-01-16T00:13:00.198 INFO:tasks.rados.rados.0.smithi123.stdout:945: finishing write tid 5 to smithi12313934-45 2020-01-16T00:13:00.198 INFO:tasks.rados.rados.0.smithi123.stdout:945: finishing write tid 6 to smithi12313934-45 2020-01-16T00:13:00.198 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 45 v 94 (ObjNum 216 snap 0 seq_num 216) dirty exists 2020-01-16T00:13:00.198 INFO:tasks.rados.rados.0.smithi123.stdout:945: left oid 45 (ObjNum 216 snap 0 seq_num 216) 2020-01-16T00:13:00.199 INFO:tasks.rados.rados.0.smithi123.stdout:947: finishing write tid 2 to smithi12313934-14 2020-01-16T00:13:00.199 INFO:tasks.rados.rados.0.smithi123.stdout:945: done (15 left) 2020-01-16T00:13:00.199 INFO:tasks.rados.rados.0.smithi123.stdout:962: read oid 9 snap -1 2020-01-16T00:13:00.199 INFO:tasks.rados.rados.0.smithi123.stdout:962: expect (ObjNum 213 snap 0 seq_num 213) 2020-01-16T00:13:00.199 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:00.199 INFO:tasks.rados.rados.0.smithi123.stdout:947: finishing write tid 3 to smithi12313934-14 2020-01-16T00:13:00.199 INFO:tasks.rados.rados.0.smithi123.stdout:947: finishing write tid 5 to smithi12313934-14 2020-01-16T00:13:00.200 INFO:tasks.rados.rados.0.smithi123.stdout:947: finishing write tid 6 to smithi12313934-14 2020-01-16T00:13:00.200 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 14 v 123 (ObjNum 217 snap 0 seq_num 217) dirty exists 2020-01-16T00:13:00.200 INFO:tasks.rados.rados.0.smithi123.stdout:947: left oid 14 (ObjNum 217 snap 0 seq_num 217) 2020-01-16T00:13:00.200 INFO:tasks.rados.rados.0.smithi123.stdout:947: done (15 left) 2020-01-16T00:13:00.200 INFO:tasks.rados.rados.0.smithi123.stdout:963: write (excl) oid 40 current snap is 0 2020-01-16T00:13:00.200 INFO:tasks.rados.rados.0.smithi123.stdout:963: seq_num 224 ranges {564768=630013,1674790=447398,2768515=626225} 2020-01-16T00:13:00.202 INFO:tasks.rados.rados.0.smithi123.stdout:963: writing smithi12313934-40 from 564768 to 1194781 tid 1 2020-01-16T00:13:00.209 INFO:tasks.rados.rados.0.smithi123.stdout:963: writing smithi12313934-40 from 1674790 to 2122188 tid 2 2020-01-16T00:13:00.219 INFO:tasks.rados.rados.0.smithi123.stdout:963: writing smithi12313934-40 from 2768515 to 3394740 tid 3 2020-01-16T00:13:00.220 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:00.220 INFO:tasks.rados.rados.0.smithi123.stdout:948: finishing write tid 1 to smithi12313934-20 2020-01-16T00:13:00.220 INFO:tasks.rados.rados.0.smithi123.stdout:948: finishing write tid 2 to smithi12313934-20 2020-01-16T00:13:00.220 INFO:tasks.rados.rados.0.smithi123.stdout:948: finishing write tid 3 to smithi12313934-20 2020-01-16T00:13:00.220 INFO:tasks.rados.rados.0.smithi123.stdout:948: finishing write tid 5 to smithi12313934-20 2020-01-16T00:13:00.220 INFO:tasks.rados.rados.0.smithi123.stdout:948: finishing write tid 6 to smithi12313934-20 2020-01-16T00:13:00.221 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 20 v 134 (ObjNum 218 snap 0 seq_num 218) dirty exists 2020-01-16T00:13:00.221 INFO:tasks.rados.rados.0.smithi123.stdout:948: left oid 20 (ObjNum 218 snap 0 seq_num 218) 2020-01-16T00:13:00.221 INFO:tasks.rados.rados.0.smithi123.stdout:949: finishing write tid 1 to smithi12313934-8 2020-01-16T00:13:00.221 INFO:tasks.rados.rados.0.smithi123.stdout:949: finishing write tid 2 to smithi12313934-8 2020-01-16T00:13:00.221 INFO:tasks.rados.rados.0.smithi123.stdout:949: finishing write tid 3 to smithi12313934-8 2020-01-16T00:13:00.221 INFO:tasks.rados.rados.0.smithi123.stdout:949: finishing write tid 5 to smithi12313934-8 2020-01-16T00:13:00.222 INFO:tasks.rados.rados.0.smithi123.stdout:949: finishing write tid 6 to smithi12313934-8 2020-01-16T00:13:00.222 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 8 v 70 (ObjNum 219 snap 0 seq_num 219) dirty exists 2020-01-16T00:13:00.222 INFO:tasks.rados.rados.0.smithi123.stdout:949: left oid 8 (ObjNum 219 snap 0 seq_num 219) 2020-01-16T00:13:00.222 INFO:tasks.rados.rados.0.smithi123.stdout:948: done (15 left) 2020-01-16T00:13:00.222 INFO:tasks.rados.rados.0.smithi123.stdout:949: done (14 left) 2020-01-16T00:13:00.222 INFO:tasks.rados.rados.0.smithi123.stdout:964: write oid 30 current snap is 0 2020-01-16T00:13:00.222 INFO:tasks.rados.rados.0.smithi123.stdout:964: seq_num 225 ranges {581575=705262,1733272=707409,3195938=437497} 2020-01-16T00:13:00.224 INFO:tasks.rados.rados.0.smithi123.stdout:964: writing smithi12313934-30 from 581575 to 1286837 tid 1 2020-01-16T00:13:00.231 INFO:tasks.rados.rados.0.smithi123.stdout:964: writing smithi12313934-30 from 1733272 to 2440681 tid 2 2020-01-16T00:13:00.238 INFO:tasks.rados.rados.0.smithi123.stdout:964: writing smithi12313934-30 from 3195938 to 3633435 tid 3 2020-01-16T00:13:00.238 INFO:tasks.rados.rados.0.smithi123.stdout:965: write oid 46 current snap is 0 2020-01-16T00:13:00.238 INFO:tasks.rados.rados.0.smithi123.stdout:965: seq_num 226 ranges {598382=780511,2075401=451067,3107008=648769} 2020-01-16T00:13:00.246 INFO:tasks.rados.rados.0.smithi123.stdout:965: writing smithi12313934-46 from 598382 to 1378893 tid 1 2020-01-16T00:13:00.253 INFO:tasks.rados.rados.0.smithi123.stdout:965: writing smithi12313934-46 from 2075401 to 2526468 tid 2 2020-01-16T00:13:00.263 INFO:tasks.rados.rados.0.smithi123.stdout:965: writing smithi12313934-46 from 3107008 to 3755777 tid 3 2020-01-16T00:13:00.263 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:00.263 INFO:tasks.rados.rados.0.smithi123.stdout:950: expect (ObjNum 191 snap 0 seq_num 191) 2020-01-16T00:13:00.264 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:00.260+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:00.297 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:00.306 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-01-16T00:13:00.363 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:00.360+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:00.434 INFO:tasks.rados.rados.0.smithi123.stdout:953: finishing write tid 1 to smithi12313934-18 2020-01-16T00:13:00.434 INFO:tasks.rados.rados.0.smithi123.stdout:953: finishing write tid 2 to smithi12313934-18 2020-01-16T00:13:00.434 INFO:tasks.rados.rados.0.smithi123.stdout:952: expect (ObjNum 194 snap 0 seq_num 194) 2020-01-16T00:13:00.456 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:00.464 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:13:00.464+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:13:00.466 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-01-16T00:13:00.564 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:00.564+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:00.608 INFO:tasks.rados.rados.0.smithi123.stdout:953: finishing write tid 3 to smithi12313934-18 2020-01-16T00:13:00.608 INFO:tasks.rados.rados.0.smithi123.stdout:953: finishing write tid 5 to smithi12313934-18 2020-01-16T00:13:00.608 INFO:tasks.rados.rados.0.smithi123.stdout:953: finishing write tid 6 to smithi12313934-18 2020-01-16T00:13:00.608 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 18 v 188 (ObjNum 220 snap 0 seq_num 220) dirty exists 2020-01-16T00:13:00.608 INFO:tasks.rados.rados.0.smithi123.stdout:953: left oid 18 (ObjNum 220 snap 0 seq_num 220) 2020-01-16T00:13:00.609 INFO:tasks.rados.rados.0.smithi123.stdout:955: finishing write tid 1 to smithi12313934-12 2020-01-16T00:13:00.609 INFO:tasks.rados.rados.0.smithi123.stdout:954: expect (ObjNum 214 snap 0 seq_num 214) 2020-01-16T00:13:00.626 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:00.627 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-01-16T00:13:00.666 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:00.660+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:00.670 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-01-16T00:13:00.766 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:13:00.764+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:13:00.787 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:00.799 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-01-16T00:13:00.817 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:13:00.818 INFO:tasks.rados.rados.0.smithi123.stdout:950: done (15 left) 2020-01-16T00:13:00.818 INFO:tasks.rados.rados.0.smithi123.stdout:951: done (14 left) 2020-01-16T00:13:00.818 INFO:tasks.rados.rados.0.smithi123.stdout:952: done (13 left) 2020-01-16T00:13:00.818 INFO:tasks.rados.rados.0.smithi123.stdout:953: done (12 left) 2020-01-16T00:13:00.818 INFO:tasks.rados.rados.0.smithi123.stdout:954: done (11 left) 2020-01-16T00:13:00.819 INFO:tasks.rados.rados.0.smithi123.stdout:966: write (excl) oid 23 current snap is 0 2020-01-16T00:13:00.819 INFO:tasks.rados.rados.0.smithi123.stdout:966: seq_num 227 ranges {615189=455760,1733883=711078,3134431=460041} 2020-01-16T00:13:00.824 INFO:tasks.rados.rados.0.smithi123.stdout:966: writing smithi12313934-23 from 615189 to 1070949 tid 1 2020-01-16T00:13:00.834 INFO:tasks.rados.rados.0.smithi123.stdout:966: writing smithi12313934-23 from 1733883 to 2444961 tid 2 2020-01-16T00:13:00.835 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:13:00.839 INFO:tasks.rados.rados.0.smithi123.stdout:966: writing smithi12313934-23 from 3134431 to 3594472 tid 3 2020-01-16T00:13:00.840 INFO:tasks.rados.rados.0.smithi123.stdout:967: read oid 38 snap -1 2020-01-16T00:13:00.840 INFO:tasks.rados.rados.0.smithi123.stdout:967: expect (ObjNum 130 snap 0 seq_num 130) 2020-01-16T00:13:00.840 INFO:tasks.rados.rados.0.smithi123.stdout:968: write oid 44 current snap is 0 2020-01-16T00:13:00.840 INFO:tasks.rados.rados.0.smithi123.stdout:968: seq_num 228 ranges {631996=531009,1792365=454736,2761854=671313} 2020-01-16T00:13:00.845 INFO:tasks.rados.rados.0.smithi123.stdout:968: writing smithi12313934-44 from 631996 to 1163005 tid 1 2020-01-16T00:13:00.852 INFO:tasks.rados.rados.0.smithi123.stdout:968: writing smithi12313934-44 from 1792365 to 2247101 tid 2 2020-01-16T00:13:00.855 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:13:00.859 INFO:tasks.rados.rados.0.smithi123.stdout:968: writing smithi12313934-44 from 2761854 to 3433167 tid 3 2020-01-16T00:13:00.859 INFO:tasks.rados.rados.0.smithi123.stdout:969: read oid 26 snap -1 2020-01-16T00:13:00.859 INFO:tasks.rados.rados.0.smithi123.stdout:969: expect (ObjNum 179 snap 0 seq_num 179) 2020-01-16T00:13:00.859 INFO:tasks.rados.rados.0.smithi123.stdout:970: write (excl) oid 5 current snap is 0 2020-01-16T00:13:00.859 INFO:tasks.rados.rados.0.smithi123.stdout:970: seq_num 229 ranges {648803=722611,1967200=714747,3421983=426820} 2020-01-16T00:13:00.866 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:00.860+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:00.867 INFO:tasks.rados.rados.0.smithi123.stdout:970: writing smithi12313934-5 from 648803 to 1371414 tid 1 2020-01-16T00:13:00.875 INFO:tasks.rados.rados.0.smithi123.stdout:970: writing smithi12313934-5 from 1967200 to 2681947 tid 2 2020-01-16T00:13:00.875 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:13:00.882 INFO:tasks.rados.rados.0.smithi123.stdout:970: writing smithi12313934-5 from 3421983 to 3848803 tid 3 2020-01-16T00:13:00.882 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:00.882 INFO:tasks.rados.rados.0.smithi123.stdout:955: finishing write tid 2 to smithi12313934-12 2020-01-16T00:13:00.883 INFO:tasks.rados.rados.0.smithi123.stdout:955: finishing write tid 3 to smithi12313934-12 2020-01-16T00:13:00.883 INFO:tasks.rados.rados.0.smithi123.stdout:955: finishing write tid 5 to smithi12313934-12 2020-01-16T00:13:00.883 INFO:tasks.rados.rados.0.smithi123.stdout:955: finishing write tid 6 to smithi12313934-12 2020-01-16T00:13:00.883 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 12 v 68 (ObjNum 221 snap 0 seq_num 221) dirty exists 2020-01-16T00:13:00.884 INFO:tasks.rados.rados.0.smithi123.stdout:955: left oid 12 (ObjNum 221 snap 0 seq_num 221) 2020-01-16T00:13:00.884 INFO:tasks.rados.rados.0.smithi123.stdout:955: done (15 left) 2020-01-16T00:13:00.884 INFO:tasks.rados.rados.0.smithi123.stdout:971: read oid 28 snap -1 2020-01-16T00:13:00.884 INFO:tasks.rados.rados.0.smithi123.stdout:971: expect (ObjNum 73 snap 0 seq_num 73) 2020-01-16T00:13:00.884 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:00.885 INFO:tasks.rados.rados.0.smithi123.stdout:958: finishing write tid 1 to smithi12313934-13 2020-01-16T00:13:00.885 INFO:tasks.rados.rados.0.smithi123.stdout:958: finishing write tid 2 to smithi12313934-13 2020-01-16T00:13:00.885 INFO:tasks.rados.rados.0.smithi123.stdout:957: expect (ObjNum 133 snap 0 seq_num 133) 2020-01-16T00:13:00.899 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:13:00.919 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5: {} 2020-01-16T00:13:00.937 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:13:00.957 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:13:00.959 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-01-16T00:13:00.959 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-01-16T00:13:00.959 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-01-16T00:13:00.959 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-01-16T00:13:00.959 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-01-16T00:13:00.960 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-01-16T00:13:00.960 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-01-16T00:13:00.960 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-01-16T00:13:00.962 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:00.963 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:13:00.963 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:13:00.963 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:13:00.963 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.963 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:895 3.3 3:cd231094:::smithi12313934-49:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:00.963 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.122390+0000", 2020-01-16T00:13:00.964 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 23.839870015999999, 2020-01-16T00:13:00.964 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014295501, 2020-01-16T00:13:00.964 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:00.964 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:00.964 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:00.965 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:00.965 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:00.965 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 895 2020-01-16T00:13:00.965 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.965 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:00.965 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.965 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:00.966 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122390+0000", 2020-01-16T00:13:00.966 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:00.966 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.966 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.966 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:00.966 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122390+0000", 2020-01-16T00:13:00.966 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999957 2020-01-16T00:13:00.967 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.967 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.967 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:00.967 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122386+0000", 2020-01-16T00:13:00.967 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0656999999999999e-05 2020-01-16T00:13:00.967 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.967 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.967 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:00.968 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122407+0000", 2020-01-16T00:13:00.968 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9102999999999999e-05 2020-01-16T00:13:00.968 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.968 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.968 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:00.968 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122436+0000", 2020-01-16T00:13:00.968 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.774e-05 2020-01-16T00:13:00.968 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.969 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.969 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:00.969 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122483+0000", 2020-01-16T00:13:00.969 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5185000000000004e-05 2020-01-16T00:13:00.969 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.969 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.969 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:00.970 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122579+0000", 2020-01-16T00:13:00.970 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000304085 2020-01-16T00:13:00.970 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.970 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.970 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:00.970 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122883+0000", 2020-01-16T00:13:00.970 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0092077970000000002 2020-01-16T00:13:00.970 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.971 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.971 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:00.971 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132091+0000", 2020-01-16T00:13:00.971 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000170025 2020-01-16T00:13:00.971 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.971 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.971 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:00.971 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132261+0000", 2020-01-16T00:13:00.972 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0040130979999999997 2020-01-16T00:13:00.972 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.972 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.972 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:00.972 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136274+0000", 2020-01-16T00:13:00.972 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014227900000000001 2020-01-16T00:13:00.972 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.973 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.973 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:00.973 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136416+0000", 2020-01-16T00:13:00.973 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00026995200000000001 2020-01-16T00:13:00.973 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.974 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.974 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:00.974 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136686+0000", 2020-01-16T00:13:00.974 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014295501 2020-01-16T00:13:00.974 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:00.975 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:00.975 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:00.975 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.975 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.975 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:896 3.3 3:cd231094:::smithi12313934-49:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:00.975 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.122539+0000", 2020-01-16T00:13:00.975 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 23.839721015999999, 2020-01-16T00:13:00.975 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014440173000000001, 2020-01-16T00:13:00.976 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:00.976 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:00.976 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:00.976 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:00.976 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:00.976 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 896 2020-01-16T00:13:00.976 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.977 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:00.977 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.977 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:00.977 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122539+0000", 2020-01-16T00:13:00.977 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:00.977 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.978 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.978 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:00.978 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122539+0000", 2020-01-16T00:13:00.978 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:00.978 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.978 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.978 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:00.978 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122537+0000", 2020-01-16T00:13:00.979 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2432999999999999e-05 2020-01-16T00:13:00.979 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.979 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.979 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:00.979 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122550+0000", 2020-01-16T00:13:00.979 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9369999999999999e-05 2020-01-16T00:13:00.979 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.979 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.980 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:00.980 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122569+0000", 2020-01-16T00:13:00.980 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1747e-05 2020-01-16T00:13:00.980 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.980 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.980 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:00.980 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122581+0000", 2020-01-16T00:13:00.981 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00041555800000000001 2020-01-16T00:13:00.981 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.981 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.981 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:00.981 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122996+0000", 2020-01-16T00:13:00.981 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016840299999999999 2020-01-16T00:13:00.981 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.981 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.982 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:00.982 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.123165+0000", 2020-01-16T00:13:00.982 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0091641359999999998 2020-01-16T00:13:00.982 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.982 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.982 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:00.982 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132329+0000", 2020-01-16T00:13:00.982 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013433400000000001 2020-01-16T00:13:00.983 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.983 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.983 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:00.983 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132463+0000", 2020-01-16T00:13:00.983 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0042524110000000002 2020-01-16T00:13:00.983 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.984 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.984 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:00.984 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136716+0000", 2020-01-16T00:13:00.984 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4591999999999993e-05 2020-01-16T00:13:00.984 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.985 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.985 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:00.985 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136790+0000", 2020-01-16T00:13:00.986 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018927500000000001 2020-01-16T00:13:00.986 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.986 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.986 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:00.986 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136980+0000", 2020-01-16T00:13:00.987 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014440173000000001 2020-01-16T00:13:00.987 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:00.987 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:00.987 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:00.987 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.987 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.988 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:897 3.3 3:cd231094:::smithi12313934-49:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:00.988 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.122613+0000", 2020-01-16T00:13:00.988 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 23.839647872, 2020-01-16T00:13:00.988 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014606846, 2020-01-16T00:13:00.988 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:00.988 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:00.988 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:00.988 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:00.989 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:00.989 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 897 2020-01-16T00:13:00.989 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.989 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:00.989 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.989 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:00.989 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122613+0000", 2020-01-16T00:13:00.989 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:00.990 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.990 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.990 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:00.990 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122613+0000", 2020-01-16T00:13:00.990 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:00.990 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.990 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.990 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:00.991 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122611+0000", 2020-01-16T00:13:00.991 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3670000000000003e-06 2020-01-16T00:13:00.991 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.991 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.991 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:00.991 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122620+0000", 2020-01-16T00:13:00.991 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5193e-05 2020-01-16T00:13:00.992 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.992 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.992 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:00.992 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122635+0000", 2020-01-16T00:13:00.992 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0696999999999999e-05 2020-01-16T00:13:00.992 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.992 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.992 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:00.993 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122646+0000", 2020-01-16T00:13:00.993 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00059961400000000005 2020-01-16T00:13:00.993 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.993 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.993 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:00.993 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.123246+0000", 2020-01-16T00:13:00.993 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011189200000000001 2020-01-16T00:13:00.994 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.994 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.994 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:00.995 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.123358+0000", 2020-01-16T00:13:00.995 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0091363550000000005 2020-01-16T00:13:00.995 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.995 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.995 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:00.995 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132494+0000", 2020-01-16T00:13:00.995 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2956999999999998e-05 2020-01-16T00:13:00.996 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.996 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.996 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:00.996 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132587+0000", 2020-01-16T00:13:00.996 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0044061769999999998 2020-01-16T00:13:00.996 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.997 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.997 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:00.997 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136993+0000", 2020-01-16T00:13:00.997 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7822999999999998e-05 2020-01-16T00:13:00.997 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.998 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.998 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:00.998 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.137051+0000", 2020-01-16T00:13:00.998 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000168498 2020-01-16T00:13:00.998 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:00.999 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:00.999 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:00.999 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.137219+0000", 2020-01-16T00:13:00.999 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014606846 2020-01-16T00:13:00.999 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.000 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.000 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.000 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.000 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.000 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:943 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:01.000 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:13:01.001 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 23.047781898, 2020-01-16T00:13:01.001 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018847004000000001, 2020-01-16T00:13:01.001 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:01.001 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:01.001 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:01.001 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:01.001 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:01.002 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 943 2020-01-16T00:13:01.002 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.002 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:01.002 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.002 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:01.002 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:13:01.002 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:01.003 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.003 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.003 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:01.003 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:13:01.003 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:01.003 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.004 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.004 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:01.004 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914475+0000", 2020-01-16T00:13:01.004 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.508e-05 2020-01-16T00:13:01.004 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.004 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.004 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:01.005 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914490+0000", 2020-01-16T00:13:01.005 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.857e-05 2020-01-16T00:13:01.005 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.005 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.005 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:01.005 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914509+0000", 2020-01-16T00:13:01.005 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.46e-06 2020-01-16T00:13:01.005 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.006 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.006 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:01.006 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914515+0000", 2020-01-16T00:13:01.006 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2612999999999999e-05 2020-01-16T00:13:01.006 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.006 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.006 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.006 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914568+0000", 2020-01-16T00:13:01.007 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028312299999999998 2020-01-16T00:13:01.007 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.007 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.007 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:01.007 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914851+0000", 2020-01-16T00:13:01.007 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012282533 2020-01-16T00:13:01.007 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.008 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.008 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.008 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927133+0000", 2020-01-16T00:13:01.008 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4839999999999995e-05 2020-01-16T00:13:01.008 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.009 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.009 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:01.009 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927208+0000", 2020-01-16T00:13:01.009 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0047425610000000002 2020-01-16T00:13:01.010 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.010 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.010 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.010 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.931951+0000", 2020-01-16T00:13:01.011 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3628999999999994e-05 2020-01-16T00:13:01.011 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.011 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.011 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:01.011 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.932045+0000", 2020-01-16T00:13:01.012 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012810180000000001 2020-01-16T00:13:01.012 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.012 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.012 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:01.012 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933326+0000", 2020-01-16T00:13:01.013 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018847004000000001 2020-01-16T00:13:01.013 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.013 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.013 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.014 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.014 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.014 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:944 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:01.015 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:13:01.015 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 23.047732342, 2020-01-16T00:13:01.015 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019086901999999999, 2020-01-16T00:13:01.015 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:01.015 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:01.016 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:01.016 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:01.016 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:01.016 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 944 2020-01-16T00:13:01.017 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.017 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:01.017 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.017 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:01.017 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:13:01.018 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:01.018 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.018 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.018 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:01.019 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:13:01.019 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:01.019 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.019 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.020 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:01.020 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914527+0000", 2020-01-16T00:13:01.020 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7509999999999999e-06 2020-01-16T00:13:01.020 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.020 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.021 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:01.021 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914532+0000", 2020-01-16T00:13:01.021 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5149999999999997e-06 2020-01-16T00:13:01.022 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.022 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.022 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:01.022 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914538+0000", 2020-01-16T00:13:01.023 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1890000000000007e-06 2020-01-16T00:13:01.023 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.023 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.023 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:01.024 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914546+0000", 2020-01-16T00:13:01.024 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000354951 2020-01-16T00:13:01.024 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.024 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.024 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.025 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914901+0000", 2020-01-16T00:13:01.025 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1594000000000002e-05 2020-01-16T00:13:01.025 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.025 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.025 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:01.026 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914942+0000", 2020-01-16T00:13:01.026 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012300070999999999 2020-01-16T00:13:01.026 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.026 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.026 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.027 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927242+0000", 2020-01-16T00:13:01.027 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4076999999999997e-05 2020-01-16T00:13:01.027 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.027 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.028 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:01.028 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927296+0000", 2020-01-16T00:13:01.028 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0060609180000000002 2020-01-16T00:13:01.028 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.029 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.029 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.029 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933357+0000", 2020-01-16T00:13:01.029 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4488999999999999e-05 2020-01-16T00:13:01.029 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.030 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.030 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:01.077 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933422+0000", 2020-01-16T00:13:01.078 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019318300000000001 2020-01-16T00:13:01.078 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.078 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.078 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:01.079 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933615+0000", 2020-01-16T00:13:01.080 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019086901999999999 2020-01-16T00:13:01.080 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.081 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.081 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.081 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.081 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.082 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:945 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:01.082 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:13:01.082 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 23.047705136000001, 2020-01-16T00:13:01.082 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019342986, 2020-01-16T00:13:01.082 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:01.083 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:01.083 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:01.083 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:01.083 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:01.084 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 945 2020-01-16T00:13:01.084 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.084 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:01.084 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.085 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:01.085 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:13:01.085 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:01.085 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.085 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.086 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:01.086 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:13:01.086 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:01.086 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.086 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.087 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:01.087 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:13:01.087 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4580000000000002e-06 2020-01-16T00:13:01.087 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.087 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.087 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:01.087 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914558+0000", 2020-01-16T00:13:01.087 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7829999999999998e-06 2020-01-16T00:13:01.088 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.088 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.088 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:01.088 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914563+0000", 2020-01-16T00:13:01.088 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0001e-05 2020-01-16T00:13:01.088 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.088 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.088 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:01.089 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914573+0000", 2020-01-16T00:13:01.089 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00040365600000000001 2020-01-16T00:13:01.089 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.089 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.089 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.089 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914976+0000", 2020-01-16T00:13:01.089 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1390000000000001e-05 2020-01-16T00:13:01.090 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.090 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.090 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:01.090 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.915028+0000", 2020-01-16T00:13:01.090 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012288843000000001 2020-01-16T00:13:01.090 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.090 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.091 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.091 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927317+0000", 2020-01-16T00:13:01.091 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7871000000000002e-05 2020-01-16T00:13:01.091 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.091 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.091 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:01.091 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927375+0000", 2020-01-16T00:13:01.091 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0062580589999999998 2020-01-16T00:13:01.092 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.092 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.092 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.092 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933633+0000", 2020-01-16T00:13:01.092 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7086999999999997e-05 2020-01-16T00:13:01.093 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.093 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.093 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:01.093 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933690+0000", 2020-01-16T00:13:01.093 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020855799999999999 2020-01-16T00:13:01.093 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.094 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.094 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:01.094 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933898+0000", 2020-01-16T00:13:01.094 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019342986 2020-01-16T00:13:01.094 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.094 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.094 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.095 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.095 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.095 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:963 3.9 3:9557b633:::smithi12313934-43:head [read 0~2705894,checksum] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:01.095 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.046589+0000", 2020-01-16T00:13:01.095 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 22.915671287999999, 2020-01-16T00:13:01.095 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014014016000000001, 2020-01-16T00:13:01.096 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:01.096 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:01.096 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:01.096 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:01.096 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:01.096 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 963 2020-01-16T00:13:01.096 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.097 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:01.097 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.097 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:01.097 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046589+0000", 2020-01-16T00:13:01.098 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:01.098 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.098 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.098 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:01.098 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046589+0000", 2020-01-16T00:13:01.099 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:01.099 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.099 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.099 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:01.099 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046588+0000", 2020-01-16T00:13:01.099 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7469999999999994e-06 2020-01-16T00:13:01.099 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.100 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.100 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:01.100 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046596+0000", 2020-01-16T00:13:01.100 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3445999999999998e-05 2020-01-16T00:13:01.100 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.100 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.100 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:01.100 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046630+0000", 2020-01-16T00:13:01.101 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7779999999999999e-05 2020-01-16T00:13:01.101 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.101 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.101 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:01.101 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046648+0000", 2020-01-16T00:13:01.101 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00075081300000000002 2020-01-16T00:13:01.101 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.101 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.102 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.102 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.047398+0000", 2020-01-16T00:13:01.102 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000116698 2020-01-16T00:13:01.102 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.102 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.102 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:01.102 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.047515+0000", 2020-01-16T00:13:01.103 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0056630689999999997 2020-01-16T00:13:01.103 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.103 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.103 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.103 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.053178+0000", 2020-01-16T00:13:01.103 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012084299999999999 2020-01-16T00:13:01.103 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.103 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.104 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:01.104 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.053299+0000", 2020-01-16T00:13:01.104 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0056921510000000003 2020-01-16T00:13:01.104 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.104 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.104 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.104 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.058991+0000", 2020-01-16T00:13:01.104 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000118606 2020-01-16T00:13:01.105 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.105 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.105 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:01.105 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.059110+0000", 2020-01-16T00:13:01.105 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001493306 2020-01-16T00:13:01.105 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.106 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.106 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:01.106 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.060603+0000", 2020-01-16T00:13:01.106 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014014016000000001 2020-01-16T00:13:01.106 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.106 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.106 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.106 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.107 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.107 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:964 3.9 3:9557b633:::smithi12313934-43:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:01.107 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.046668+0000", 2020-01-16T00:13:01.107 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 22.915592684, 2020-01-16T00:13:01.107 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014425670999999999, 2020-01-16T00:13:01.108 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:01.108 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:01.108 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:01.108 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:01.109 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:01.109 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 964 2020-01-16T00:13:01.109 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.110 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:01.110 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.110 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:01.110 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046668+0000", 2020-01-16T00:13:01.110 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:01.111 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.111 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.111 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:01.111 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046668+0000", 2020-01-16T00:13:01.111 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:01.112 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.112 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.112 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:01.112 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046666+0000", 2020-01-16T00:13:01.112 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0879999999999999e-05 2020-01-16T00:13:01.113 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.113 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.113 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:01.113 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046677+0000", 2020-01-16T00:13:01.114 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5767e-05 2020-01-16T00:13:01.114 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.114 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.114 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:01.114 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046693+0000", 2020-01-16T00:13:01.115 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1399999999999999e-05 2020-01-16T00:13:01.115 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.115 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.116 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:01.116 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046704+0000", 2020-01-16T00:13:01.116 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00089331800000000004 2020-01-16T00:13:01.116 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.116 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.117 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.117 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.047597+0000", 2020-01-16T00:13:01.117 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8628000000000005e-05 2020-01-16T00:13:01.117 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.117 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.117 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:01.118 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.047696+0000", 2020-01-16T00:13:01.118 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0056536290000000003 2020-01-16T00:13:01.118 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.118 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.118 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.118 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.053349+0000", 2020-01-16T00:13:01.118 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.6946000000000003e-05 2020-01-16T00:13:01.119 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.119 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.119 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:01.119 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.053446+0000", 2020-01-16T00:13:01.119 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0072058909999999999 2020-01-16T00:13:01.119 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.120 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.120 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.120 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.060652+0000", 2020-01-16T00:13:01.120 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000135597 2020-01-16T00:13:01.120 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.120 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.120 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:01.120 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.060788+0000", 2020-01-16T00:13:01.121 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030548499999999998 2020-01-16T00:13:01.121 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.121 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.121 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:01.121 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.061093+0000", 2020-01-16T00:13:01.121 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014425670999999999 2020-01-16T00:13:01.121 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.121 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.122 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.122 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.122 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.122 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:981 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:01.122 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:13:01.122 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 22.646928244000001, 2020-01-16T00:13:01.122 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021222494000000001, 2020-01-16T00:13:01.123 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:01.123 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:01.123 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:01.123 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:01.123 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:01.123 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 981 2020-01-16T00:13:01.123 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.124 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:01.124 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.124 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:01.124 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:13:01.124 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:01.124 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.124 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.124 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:01.125 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:13:01.125 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999948 2020-01-16T00:13:01.125 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.125 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.125 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:01.125 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315327+0000", 2020-01-16T00:13:01.125 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.508e-05 2020-01-16T00:13:01.126 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.126 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.126 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:01.126 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315352+0000", 2020-01-16T00:13:01.126 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1183999999999998e-05 2020-01-16T00:13:01.127 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.127 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.127 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:01.127 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315383+0000", 2020-01-16T00:13:01.127 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7180000000000002e-05 2020-01-16T00:13:01.127 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.127 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.127 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:01.128 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315401+0000", 2020-01-16T00:13:01.128 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017290599999999999 2020-01-16T00:13:01.128 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.128 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.128 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.128 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315574+0000", 2020-01-16T00:13:01.128 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00047908100000000002 2020-01-16T00:13:01.128 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.129 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.129 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:01.129 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316053+0000", 2020-01-16T00:13:01.129 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011808947 2020-01-16T00:13:01.129 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.129 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.129 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.130 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.327862+0000", 2020-01-16T00:13:01.130 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010067499999999999 2020-01-16T00:13:01.130 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.130 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.130 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:01.130 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.327962+0000", 2020-01-16T00:13:01.130 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0074745139999999998 2020-01-16T00:13:01.130 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.131 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.131 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.131 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.335437+0000", 2020-01-16T00:13:01.131 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000103403 2020-01-16T00:13:01.131 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.131 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.131 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:01.131 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.335540+0000", 2020-01-16T00:13:01.132 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001014544 2020-01-16T00:13:01.132 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.132 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.132 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:01.132 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336555+0000", 2020-01-16T00:13:01.132 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021222494000000001 2020-01-16T00:13:01.132 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.132 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.133 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.133 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.133 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.133 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:982 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:01.133 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:13:01.133 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 22.646825719999999, 2020-01-16T00:13:01.134 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.022087721000000001, 2020-01-16T00:13:01.134 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:01.134 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:01.134 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:01.134 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:01.134 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:01.134 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 982 2020-01-16T00:13:01.134 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.135 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:01.135 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.135 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:01.135 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:13:01.135 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:01.135 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.135 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.135 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:01.136 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:13:01.136 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:01.136 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.136 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.136 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:01.136 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315433+0000", 2020-01-16T00:13:01.137 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1347e-05 2020-01-16T00:13:01.137 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.137 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.137 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:01.137 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315445+0000", 2020-01-16T00:13:01.137 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4525999999999999e-05 2020-01-16T00:13:01.137 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.137 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.138 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:01.138 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315459+0000", 2020-01-16T00:13:01.138 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.024e-05 2020-01-16T00:13:01.138 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.138 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.138 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:01.138 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315470+0000", 2020-01-16T00:13:01.139 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000736468 2020-01-16T00:13:01.139 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.139 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.139 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.139 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316206+0000", 2020-01-16T00:13:01.139 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000132967 2020-01-16T00:13:01.139 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.139 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.140 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:01.140 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316339+0000", 2020-01-16T00:13:01.140 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011701717 2020-01-16T00:13:01.140 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.140 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.140 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.140 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328041+0000", 2020-01-16T00:13:01.140 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000120339 2020-01-16T00:13:01.141 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.141 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.141 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:01.141 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328161+0000", 2020-01-16T00:13:01.141 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0084341260000000001 2020-01-16T00:13:01.141 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.141 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.142 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.142 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336595+0000", 2020-01-16T00:13:01.142 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5247999999999993e-05 2020-01-16T00:13:01.142 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.142 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.142 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:01.142 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336690+0000", 2020-01-16T00:13:01.142 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00083204000000000004 2020-01-16T00:13:01.143 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.143 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.143 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:01.143 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337522+0000", 2020-01-16T00:13:01.143 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.022087721000000001 2020-01-16T00:13:01.143 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.143 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.143 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.144 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.144 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.144 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:983 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:01.144 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:13:01.144 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 22.646706009999999, 2020-01-16T00:13:01.145 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023363387999999999, 2020-01-16T00:13:01.145 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:01.145 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:01.145 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:01.146 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:01.146 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:01.146 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 983 2020-01-16T00:13:01.146 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.147 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:01.147 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.147 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:01.147 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:13:01.147 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:01.147 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.147 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.147 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:01.148 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:13:01.148 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:01.148 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.148 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.148 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:01.148 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315553+0000", 2020-01-16T00:13:01.148 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1760000000000004e-06 2020-01-16T00:13:01.148 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.149 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.149 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:01.149 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315560+0000", 2020-01-16T00:13:01.149 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3587000000000001e-05 2020-01-16T00:13:01.149 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.149 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.149 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:01.150 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315574+0000", 2020-01-16T00:13:01.150 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7800000000000003e-06 2020-01-16T00:13:01.150 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.150 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.150 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:01.150 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315581+0000", 2020-01-16T00:13:01.150 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000823024 2020-01-16T00:13:01.150 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.151 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.151 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.151 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316404+0000", 2020-01-16T00:13:01.151 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2413e-05 2020-01-16T00:13:01.151 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.151 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.152 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:01.152 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316466+0000", 2020-01-16T00:13:01.152 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011738561999999999 2020-01-16T00:13:01.152 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.152 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.152 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.152 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328205+0000", 2020-01-16T00:13:01.153 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6783999999999997e-05 2020-01-16T00:13:01.153 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.153 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.153 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:01.153 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328262+0000", 2020-01-16T00:13:01.153 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0093022560000000001 2020-01-16T00:13:01.153 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.153 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.154 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.154 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337564+0000", 2020-01-16T00:13:01.154 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000183151 2020-01-16T00:13:01.154 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.154 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.154 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:01.154 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337747+0000", 2020-01-16T00:13:01.155 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011707449999999999 2020-01-16T00:13:01.155 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.155 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.155 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:01.155 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.338918+0000", 2020-01-16T00:13:01.155 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023363387999999999 2020-01-16T00:13:01.155 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.155 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.156 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.156 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.156 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.156 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:986 3.9 3:93dfe6f1:::smithi12313934-40:head [write 2941380~419987] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:01.157 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:13:01.157 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 22.481406234000001, 2020-01-16T00:13:01.157 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016978400000000001, 2020-01-16T00:13:01.157 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:01.157 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:01.158 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:01.158 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:01.158 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:01.158 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 986 2020-01-16T00:13:01.158 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.158 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:01.159 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.159 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:01.159 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:13:01.159 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:01.159 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.159 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.159 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:01.160 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:13:01.160 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:01.160 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.160 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.160 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:01.160 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480852+0000", 2020-01-16T00:13:01.160 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042503899999999999 2020-01-16T00:13:01.161 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.161 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.161 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:01.161 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481277+0000", 2020-01-16T00:13:01.161 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3097e-05 2020-01-16T00:13:01.161 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.161 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.161 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:01.162 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481290+0000", 2020-01-16T00:13:01.162 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8460000000000002e-06 2020-01-16T00:13:01.162 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.162 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.162 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:01.162 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481297+0000", 2020-01-16T00:13:01.162 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8443999999999999e-05 2020-01-16T00:13:01.163 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.163 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.163 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.163 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481346+0000", 2020-01-16T00:13:01.163 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022115399999999999 2020-01-16T00:13:01.163 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.163 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.163 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:01.164 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481567+0000", 2020-01-16T00:13:01.164 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012531502 2020-01-16T00:13:01.164 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.164 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.164 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.164 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494098+0000", 2020-01-16T00:13:01.164 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.4462999999999993e-05 2020-01-16T00:13:01.164 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.165 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.165 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:01.165 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494193+0000", 2020-01-16T00:13:01.165 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017600999999999999 2020-01-16T00:13:01.165 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.166 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.166 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:13:01.166 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494369+0000", 2020-01-16T00:13:01.166 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0031739619999999998 2020-01-16T00:13:01.166 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.166 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.167 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:01.167 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497543+0000", 2020-01-16T00:13:01.167 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000175107 2020-01-16T00:13:01.167 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.167 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.167 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:01.168 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497718+0000", 2020-01-16T00:13:01.168 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5207000000000003e-05 2020-01-16T00:13:01.168 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.168 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.168 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:01.168 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497763+0000", 2020-01-16T00:13:01.168 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7042999999999998e-05 2020-01-16T00:13:01.168 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.169 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.169 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:01.169 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497800+0000", 2020-01-16T00:13:01.169 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2628e-05 2020-01-16T00:13:01.169 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.169 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.169 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:01.170 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497833+0000", 2020-01-16T00:13:01.170 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016978400000000001 2020-01-16T00:13:01.170 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.170 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.170 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.170 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.170 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.170 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:987 3.9 3:93dfe6f1:::smithi12313934-40:head [setxattr _header (56) in=7b,truncate 3361367] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:01.171 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.481307+0000", 2020-01-16T00:13:01.171 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 22.480953867, 2020-01-16T00:13:01.171 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016794989, 2020-01-16T00:13:01.171 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:01.171 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:01.171 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:01.171 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:01.171 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:01.172 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 987 2020-01-16T00:13:01.172 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.172 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:01.172 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.172 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:01.172 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481307+0000", 2020-01-16T00:13:01.172 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:01.172 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.173 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.173 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:01.173 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481307+0000", 2020-01-16T00:13:01.173 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:01.173 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.173 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.173 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:01.174 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481306+0000", 2020-01-16T00:13:01.174 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.113e-06 2020-01-16T00:13:01.174 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.174 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.174 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:01.174 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481311+0000", 2020-01-16T00:13:01.175 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1520000000000002e-06 2020-01-16T00:13:01.175 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.175 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.175 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:01.175 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481315+0000", 2020-01-16T00:13:01.175 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.7640000000000001e-06 2020-01-16T00:13:01.175 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.176 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.176 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:01.176 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481318+0000", 2020-01-16T00:13:01.176 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00029012500000000002 2020-01-16T00:13:01.176 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.176 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.176 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.176 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481608+0000", 2020-01-16T00:13:01.177 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6239000000000002e-05 2020-01-16T00:13:01.177 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.177 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.177 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:01.177 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481654+0000", 2020-01-16T00:13:01.178 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014559534000000001 2020-01-16T00:13:01.178 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.178 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.178 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.178 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.496214+0000", 2020-01-16T00:13:01.178 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000225932 2020-01-16T00:13:01.178 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.178 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.179 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:01.179 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.496439+0000", 2020-01-16T00:13:01.179 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00031099600000000002 2020-01-16T00:13:01.179 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.179 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.179 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:13:01.179 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.496750+0000", 2020-01-16T00:13:01.180 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00098608599999999995 2020-01-16T00:13:01.180 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.180 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.180 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:01.180 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497737+0000", 2020-01-16T00:13:01.180 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022081499999999999 2020-01-16T00:13:01.180 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.180 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.181 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:01.181 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497957+0000", 2020-01-16T00:13:01.181 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7071000000000003e-05 2020-01-16T00:13:01.181 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.181 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.181 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:01.181 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498014+0000", 2020-01-16T00:13:01.182 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1111e-05 2020-01-16T00:13:01.182 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.182 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.182 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:01.182 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498056+0000", 2020-01-16T00:13:01.182 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5974999999999999e-05 2020-01-16T00:13:01.182 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.182 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.183 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:01.183 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498102+0000", 2020-01-16T00:13:01.183 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016794989 2020-01-16T00:13:01.183 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.183 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.183 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.183 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.183 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.184 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:988 3.9 3:93dfe6f1:::smithi12313934-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:13:01.184 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:13:01.184 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 22.480920770000001, 2020-01-16T00:13:01.184 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017221692, 2020-01-16T00:13:01.184 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:01.184 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:01.184 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:01.185 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:01.185 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:01.185 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 988 2020-01-16T00:13:01.185 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.185 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:01.185 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.186 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:01.186 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:13:01.186 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:01.186 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.186 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.186 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:01.186 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:13:01.186 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:01.187 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.187 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.187 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:01.187 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481339+0000", 2020-01-16T00:13:01.187 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3089999999999997e-06 2020-01-16T00:13:01.187 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.187 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.188 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:01.188 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481343+0000", 2020-01-16T00:13:01.188 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2230000000000004e-06 2020-01-16T00:13:01.188 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.188 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.188 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:01.188 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481350+0000", 2020-01-16T00:13:01.188 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.319e-06 2020-01-16T00:13:01.189 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.189 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.189 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:01.189 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481356+0000", 2020-01-16T00:13:01.189 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000321544 2020-01-16T00:13:01.189 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.189 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.190 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.190 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481678+0000", 2020-01-16T00:13:01.190 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6375999999999999e-05 2020-01-16T00:13:01.190 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.190 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.190 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:01.190 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481714+0000", 2020-01-16T00:13:01.190 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015557666 2020-01-16T00:13:01.191 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.191 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.191 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.191 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497272+0000", 2020-01-16T00:13:01.191 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012791300000000001 2020-01-16T00:13:01.191 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.191 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.191 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:01.192 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497399+0000", 2020-01-16T00:13:01.192 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00076944600000000002 2020-01-16T00:13:01.192 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.192 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.192 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.192 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498169+0000", 2020-01-16T00:13:01.192 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018306799999999999 2020-01-16T00:13:01.193 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.193 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.193 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:01.193 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498352+0000", 2020-01-16T00:13:01.193 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020935999999999999 2020-01-16T00:13:01.193 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.193 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.193 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:01.194 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498561+0000", 2020-01-16T00:13:01.194 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017221692 2020-01-16T00:13:01.194 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.194 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.194 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.194 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.194 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.195 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1208 3.3 3:c478bfe8:::smithi12313934-10:head [write 749612~474179,stat] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:13:01.195 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.350952+0000", 2020-01-16T00:13:01.195 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 18.611307932999999, 2020-01-16T00:13:01.195 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014967729000000001, 2020-01-16T00:13:01.195 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:01.195 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:01.195 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:01.195 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:01.196 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:01.196 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1208 2020-01-16T00:13:01.196 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.196 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:01.196 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.196 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:01.196 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.350952+0000", 2020-01-16T00:13:01.197 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:01.197 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.197 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.197 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:01.197 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.350952+0000", 2020-01-16T00:13:01.198 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:13:01.198 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.198 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.198 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:01.198 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.350949+0000", 2020-01-16T00:13:01.198 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042894 2020-01-16T00:13:01.198 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.198 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.199 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:01.199 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.351377+0000", 2020-01-16T00:13:01.199 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3921e-05 2020-01-16T00:13:01.199 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.199 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.199 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:01.199 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.351401+0000", 2020-01-16T00:13:01.199 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1087000000000001e-05 2020-01-16T00:13:01.200 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.200 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.200 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:01.200 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.351412+0000", 2020-01-16T00:13:01.200 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8127000000000001e-05 2020-01-16T00:13:01.200 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.200 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.201 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.201 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.351471+0000", 2020-01-16T00:13:01.201 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000230268 2020-01-16T00:13:01.201 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.201 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.201 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:01.201 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.351701+0000", 2020-01-16T00:13:01.201 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0051228369999999999 2020-01-16T00:13:01.202 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.202 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.202 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.202 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.356824+0000", 2020-01-16T00:13:01.202 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015984800000000001 2020-01-16T00:13:01.202 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.202 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.202 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:01.203 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.356984+0000", 2020-01-16T00:13:01.203 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0038163770000000001 2020-01-16T00:13:01.203 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.203 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.203 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.203 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.360800+0000", 2020-01-16T00:13:01.203 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.9878999999999999e-05 2020-01-16T00:13:01.203 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.204 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.204 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:01.204 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.360880+0000", 2020-01-16T00:13:01.204 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000145041 2020-01-16T00:13:01.204 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.204 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.204 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2020-01-16T00:13:01.205 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.361025+0000", 2020-01-16T00:13:01.205 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0020743039999999999 2020-01-16T00:13:01.205 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.205 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.205 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:01.205 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.363099+0000", 2020-01-16T00:13:01.205 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00072876500000000003 2020-01-16T00:13:01.206 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.206 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.206 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:01.206 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.363828+0000", 2020-01-16T00:13:01.207 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00189153 2020-01-16T00:13:01.207 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.207 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.207 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:01.208 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.365719+0000", 2020-01-16T00:13:01.208 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3948000000000005e-05 2020-01-16T00:13:01.208 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.209 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.209 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:01.209 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.365783+0000", 2020-01-16T00:13:01.209 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000136833 2020-01-16T00:13:01.209 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.210 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.210 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:01.210 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.365920+0000", 2020-01-16T00:13:01.210 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014967729000000001 2020-01-16T00:13:01.210 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.211 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.211 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.211 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.211 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.211 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1237 3.3 3:c204aa4f:::smithi12313934-45:head [write 400033~699926] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:13:01.212 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.584946+0000", 2020-01-16T00:13:01.212 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 18.377314050999999, 2020-01-16T00:13:01.212 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014287855, 2020-01-16T00:13:01.212 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:01.212 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:01.213 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:01.213 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:01.213 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:01.213 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1237 2020-01-16T00:13:01.213 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.213 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:01.214 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.214 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:01.214 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.584946+0000", 2020-01-16T00:13:01.214 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:01.215 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.215 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.215 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:01.215 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.584946+0000", 2020-01-16T00:13:01.215 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999752 2020-01-16T00:13:01.216 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.216 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.216 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:01.216 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.584921+0000", 2020-01-16T00:13:01.217 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00077243400000000001 2020-01-16T00:13:01.217 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.217 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.217 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:01.217 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.585694+0000", 2020-01-16T00:13:01.218 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3083000000000001e-05 2020-01-16T00:13:01.218 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.218 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.219 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:01.219 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.585737+0000", 2020-01-16T00:13:01.219 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1854000000000001e-05 2020-01-16T00:13:01.219 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.219 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.220 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:01.220 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.585759+0000", 2020-01-16T00:13:01.220 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016127000000000001 2020-01-16T00:13:01.220 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.220 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.220 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.220 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.585920+0000", 2020-01-16T00:13:01.221 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024934700000000002 2020-01-16T00:13:01.221 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.221 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.221 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:01.221 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.586169+0000", 2020-01-16T00:13:01.221 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00036508999999999999 2020-01-16T00:13:01.221 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.221 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.222 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2020-01-16T00:13:01.222 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.586534+0000", 2020-01-16T00:13:01.222 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012168910999999999 2020-01-16T00:13:01.222 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.222 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.223 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:01.223 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598703+0000", 2020-01-16T00:13:01.223 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5302000000000004e-05 2020-01-16T00:13:01.223 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.223 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.223 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:01.224 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598789+0000", 2020-01-16T00:13:01.224 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00027301099999999998 2020-01-16T00:13:01.224 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.224 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.224 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:01.224 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599062+0000", 2020-01-16T00:13:01.224 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.4237e-05 2020-01-16T00:13:01.225 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.225 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.225 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:01.225 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599146+0000", 2020-01-16T00:13:01.225 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.8319000000000006e-05 2020-01-16T00:13:01.225 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.226 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.226 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:01.226 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599234+0000", 2020-01-16T00:13:01.226 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014287855 2020-01-16T00:13:01.226 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.226 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.226 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.226 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.227 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.227 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1239 3.3 3:c204aa4f:::smithi12313934-45:head [setxattr _header (56) in=7b,truncate 2000033] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:13:01.227 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:13:01.227 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 18.371846950999998, 2020-01-16T00:13:01.227 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019161661999999999, 2020-01-16T00:13:01.227 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:01.227 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:01.228 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:01.228 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:01.228 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:01.228 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1239 2020-01-16T00:13:01.228 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.228 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:01.228 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.229 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:01.229 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:13:01.229 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:01.229 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.229 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.229 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:01.230 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:13:01.230 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:01.230 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.230 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.230 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:01.230 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590411+0000", 2020-01-16T00:13:01.230 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3879999999999999e-05 2020-01-16T00:13:01.230 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.231 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.231 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:01.231 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590425+0000", 2020-01-16T00:13:01.231 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6542999999999998e-05 2020-01-16T00:13:01.231 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.231 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.231 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:01.232 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590471+0000", 2020-01-16T00:13:01.232 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8907999999999999e-05 2020-01-16T00:13:01.232 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.232 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.232 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:01.232 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590490+0000", 2020-01-16T00:13:01.232 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0045691960000000002 2020-01-16T00:13:01.232 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.233 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.233 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.233 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.595059+0000", 2020-01-16T00:13:01.233 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0035638940000000002 2020-01-16T00:13:01.233 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.233 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.233 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:01.234 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598623+0000", 2020-01-16T00:13:01.234 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0077558310000000004 2020-01-16T00:13:01.234 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.234 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.234 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.234 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606379+0000", 2020-01-16T00:13:01.234 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012171999999999999 2020-01-16T00:13:01.234 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.235 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.235 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:01.235 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606501+0000", 2020-01-16T00:13:01.235 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013144200000000001 2020-01-16T00:13:01.235 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.235 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.235 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2020-01-16T00:13:01.235 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606632+0000", 2020-01-16T00:13:01.236 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001288565 2020-01-16T00:13:01.236 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.236 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.236 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:01.236 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607921+0000", 2020-01-16T00:13:01.236 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033399299999999999 2020-01-16T00:13:01.236 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.237 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.237 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:01.237 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.608255+0000", 2020-01-16T00:13:01.237 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012280870000000001 2020-01-16T00:13:01.237 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.237 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.237 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:01.238 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609483+0000", 2020-01-16T00:13:01.238 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2286000000000001e-05 2020-01-16T00:13:01.238 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.238 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.238 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:01.238 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609525+0000", 2020-01-16T00:13:01.238 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0121999999999999e-05 2020-01-16T00:13:01.238 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.239 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.239 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:01.239 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609575+0000", 2020-01-16T00:13:01.239 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019161661999999999 2020-01-16T00:13:01.239 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.240 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.240 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.240 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.240 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.240 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1240 3.3 3:c204aa4f:::smithi12313934-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-01-16T00:13:01.240 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:13:01.240 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 18.371746762000001, 2020-01-16T00:13:01.240 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019268828000000002, 2020-01-16T00:13:01.241 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:01.241 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:01.241 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:01.241 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:01.241 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:01.241 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1240 2020-01-16T00:13:01.242 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.242 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:01.242 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.242 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:01.242 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:13:01.242 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:01.242 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.242 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.243 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:01.243 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:13:01.243 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:01.243 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.243 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.243 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:01.243 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590512+0000", 2020-01-16T00:13:01.243 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0525e-05 2020-01-16T00:13:01.244 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.244 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.244 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:01.244 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590522+0000", 2020-01-16T00:13:01.244 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6418000000000001e-05 2020-01-16T00:13:01.244 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.244 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.244 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:01.245 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590539+0000", 2020-01-16T00:13:01.245 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0679999999999999e-05 2020-01-16T00:13:01.245 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.245 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.245 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:01.245 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590549+0000", 2020-01-16T00:13:01.245 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0083626789999999996 2020-01-16T00:13:01.246 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.246 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.246 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.246 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598912+0000", 2020-01-16T00:13:01.246 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013344 2020-01-16T00:13:01.246 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.246 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.246 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:01.247 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599045+0000", 2020-01-16T00:13:01.247 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0084002269999999997 2020-01-16T00:13:01.247 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.247 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.247 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.247 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607446+0000", 2020-01-16T00:13:01.247 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000109873 2020-01-16T00:13:01.247 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.248 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.248 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:01.248 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607555+0000", 2020-01-16T00:13:01.248 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0020573309999999999 2020-01-16T00:13:01.248 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.248 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.248 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.249 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609613+0000", 2020-01-16T00:13:01.250 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6351e-05 2020-01-16T00:13:01.250 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.250 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.250 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:01.250 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609699+0000", 2020-01-16T00:13:01.250 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3374999999999994e-05 2020-01-16T00:13:01.250 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.250 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.251 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:01.251 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609782+0000", 2020-01-16T00:13:01.251 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019268828000000002 2020-01-16T00:13:01.251 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.251 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.251 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.251 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.252 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.252 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1436 3.3 3:c204aa4f:::smithi12313934-45:head [setxattr _header (56) in=7b,truncate 2386594] snapc 0=[] ondisk+write+known_if_redirected e28)", 2020-01-16T00:13:01.252 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:47.900544+0000", 2020-01-16T00:13:01.252 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 13.061716081, 2020-01-16T00:13:01.252 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016402521, 2020-01-16T00:13:01.252 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:01.252 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:01.252 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:01.253 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:01.253 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:01.253 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1436 2020-01-16T00:13:01.253 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.253 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:01.253 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.253 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:01.253 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900544+0000", 2020-01-16T00:13:01.254 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:01.254 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.254 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.254 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:01.254 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900544+0000", 2020-01-16T00:13:01.254 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:01.254 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.254 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.255 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:01.255 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900542+0000", 2020-01-16T00:13:01.255 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3862999999999999e-05 2020-01-16T00:13:01.255 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.255 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.255 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:01.255 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900556+0000", 2020-01-16T00:13:01.256 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5204e-05 2020-01-16T00:13:01.256 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.256 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.256 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:01.256 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900571+0000", 2020-01-16T00:13:01.256 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.092e-05 2020-01-16T00:13:01.256 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.256 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.257 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:01.257 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900582+0000", 2020-01-16T00:13:01.257 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002786026 2020-01-16T00:13:01.257 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.257 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.257 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.257 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.903368+0000", 2020-01-16T00:13:01.257 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0027123659999999999 2020-01-16T00:13:01.258 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.258 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.258 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:01.258 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.906081+0000", 2020-01-16T00:13:01.258 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0082707790000000007 2020-01-16T00:13:01.258 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.258 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.259 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.259 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.914352+0000", 2020-01-16T00:13:01.259 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020580300000000001 2020-01-16T00:13:01.259 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.259 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.259 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:01.259 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.914557+0000", 2020-01-16T00:13:01.259 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017729900000000001 2020-01-16T00:13:01.260 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.260 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.260 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2020-01-16T00:13:01.260 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.914735+0000", 2020-01-16T00:13:01.260 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00074632099999999999 2020-01-16T00:13:01.260 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.260 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.260 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:01.261 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.915481+0000", 2020-01-16T00:13:01.261 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012279599999999999 2020-01-16T00:13:01.261 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.261 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.261 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:01.261 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.915604+0000", 2020-01-16T00:13:01.262 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012619339999999999 2020-01-16T00:13:01.262 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.262 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.262 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:01.262 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.916866+0000", 2020-01-16T00:13:01.262 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1162000000000003e-05 2020-01-16T00:13:01.262 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.263 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.263 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:01.263 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.916907+0000", 2020-01-16T00:13:01.263 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9987999999999997e-05 2020-01-16T00:13:01.263 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.263 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.264 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:01.264 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.916947+0000", 2020-01-16T00:13:01.264 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016402521 2020-01-16T00:13:01.264 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.264 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.264 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.264 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.264 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.265 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1437 3.3 3:c204aa4f:::smithi12313934-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e28)", 2020-01-16T00:13:01.265 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:47.900594+0000", 2020-01-16T00:13:01.265 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 13.061666217999999, 2020-01-16T00:13:01.265 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016745237999999999, 2020-01-16T00:13:01.265 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:01.265 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:01.265 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:01.265 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:01.266 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:01.266 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1437 2020-01-16T00:13:01.266 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.266 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:01.266 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.266 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:01.266 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900594+0000", 2020-01-16T00:13:01.266 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:01.267 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.267 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.267 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:01.267 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900594+0000", 2020-01-16T00:13:01.270 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:01.270 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.270 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.270 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:01.270 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900593+0000", 2020-01-16T00:13:01.270 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5059999999999995e-06 2020-01-16T00:13:01.271 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.271 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.272 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:01.272 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900601+0000", 2020-01-16T00:13:01.272 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5077e-05 2020-01-16T00:13:01.272 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.272 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.273 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:01.273 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900616+0000", 2020-01-16T00:13:01.273 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0183000000000001e-05 2020-01-16T00:13:01.273 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.273 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.273 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:01.273 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900627+0000", 2020-01-16T00:13:01.273 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0063667660000000003 2020-01-16T00:13:01.274 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.274 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.274 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.274 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.906993+0000", 2020-01-16T00:13:01.274 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8988e-05 2020-01-16T00:13:01.274 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.275 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.275 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:01.275 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.907032+0000", 2020-01-16T00:13:01.275 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0080128850000000008 2020-01-16T00:13:01.275 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.275 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.275 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.276 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.915045+0000", 2020-01-16T00:13:01.276 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013200899999999999 2020-01-16T00:13:01.276 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.276 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.276 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:01.276 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.915177+0000", 2020-01-16T00:13:01.276 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0018373090000000001 2020-01-16T00:13:01.276 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.277 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.277 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.277 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.917015+0000", 2020-01-16T00:13:01.277 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016701699999999999 2020-01-16T00:13:01.277 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.277 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.277 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:01.277 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.917182+0000", 2020-01-16T00:13:01.278 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015782400000000001 2020-01-16T00:13:01.278 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.278 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.278 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:01.278 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.917339+0000", 2020-01-16T00:13:01.278 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016745237999999999 2020-01-16T00:13:01.278 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.278 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.279 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.279 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.279 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.279 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:01.282 INFO:tasks.rados.rados.0.smithi123.stdout:956: expect (ObjNum 207 snap 0 seq_num 207) 2020-01-16T00:13:01.282 INFO:tasks.rados.rados.0.smithi123.stdout:958: finishing write tid 3 to smithi12313934-13 2020-01-16T00:13:01.282 INFO:tasks.rados.rados.0.smithi123.stdout:958: finishing write tid 4 to smithi12313934-13 2020-01-16T00:13:01.282 INFO:tasks.rados.rados.0.smithi123.stdout:958: finishing write tid 6 to smithi12313934-13 2020-01-16T00:13:01.282 INFO:tasks.rados.rados.0.smithi123.stdout:958: finishing write tid 7 to smithi12313934-13 2020-01-16T00:13:01.282 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 46 (ObjNum 222 snap 0 seq_num 222) dirty exists 2020-01-16T00:13:01.283 INFO:tasks.rados.rados.0.smithi123.stdout:958: left oid 13 (ObjNum 222 snap 0 seq_num 222) 2020-01-16T00:13:01.283 INFO:tasks.rados.rados.0.smithi123.stdout:960: finishing write tid 1 to smithi12313934-43 2020-01-16T00:13:01.283 INFO:tasks.rados.rados.0.smithi123.stdout:959: expect (ObjNum 196 snap 0 seq_num 196) 2020-01-16T00:13:01.283 INFO:teuthology.orchestra.run.smithi123.stderr:noscrub is unset 2020-01-16T00:13:01.283 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:01.275+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:01.284 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-01-16T00:13:01.367 INFO:tasks.rados.rados.0.smithi123.stdout:960: finishing write tid 2 to smithi12313934-43 2020-01-16T00:13:01.368 INFO:tasks.rados.rados.0.smithi123.stdout:960: finishing write tid 3 to smithi12313934-43 2020-01-16T00:13:01.368 INFO:tasks.rados.rados.0.smithi123.stdout:960: finishing write tid 5 to smithi12313934-43 2020-01-16T00:13:01.368 INFO:tasks.rados.rados.0.smithi123.stdout:960: finishing write tid 6 to smithi12313934-43 2020-01-16T00:13:01.368 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 43 v 127 (ObjNum 223 snap 0 seq_num 223) dirty exists 2020-01-16T00:13:01.368 INFO:tasks.rados.rados.0.smithi123.stdout:960: left oid 43 (ObjNum 223 snap 0 seq_num 223) 2020-01-16T00:13:01.368 INFO:tasks.rados.rados.0.smithi123.stdout:963: finishing write tid 1 to smithi12313934-40 2020-01-16T00:13:01.368 INFO:tasks.rados.rados.0.smithi123.stdout:962: expect (ObjNum 213 snap 0 seq_num 213) 2020-01-16T00:13:01.381 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:01.379+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:01.402 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:01.402 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:13:01.402 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:13:01.402 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:01.412 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-01-16T00:13:01.482 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:13:01.479+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:13:01.520 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:01.521 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:13:01.521 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:13:01.521 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:13:01.521 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:01.538 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-01-16T00:13:01.543 INFO:tasks.rados.rados.0.smithi123.stdout:963: finishing write tid 2 to smithi12313934-40 2020-01-16T00:13:01.543 INFO:tasks.rados.rados.0.smithi123.stdout:961: expect (ObjNum 146 snap 0 seq_num 146) 2020-01-16T00:13:01.582 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:13:01.579+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:13:01.612 INFO:tasks.rados.rados.0.smithi123.stdout:963: finishing write tid 3 to smithi12313934-40 2020-01-16T00:13:01.612 INFO:tasks.rados.rados.0.smithi123.stdout:963: finishing write tid 5 to smithi12313934-40 2020-01-16T00:13:01.613 INFO:tasks.rados.rados.0.smithi123.stdout:963: finishing write tid 6 to smithi12313934-40 2020-01-16T00:13:01.613 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 40 v 132 (ObjNum 224 snap 0 seq_num 224) dirty exists 2020-01-16T00:13:01.613 INFO:tasks.rados.rados.0.smithi123.stdout:963: left oid 40 (ObjNum 224 snap 0 seq_num 224) 2020-01-16T00:13:01.613 INFO:tasks.rados.rados.0.smithi123.stdout:964: finishing write tid 1 to smithi12313934-30 2020-01-16T00:13:01.613 INFO:tasks.rados.rados.0.smithi123.stdout:964: finishing write tid 2 to smithi12313934-30 2020-01-16T00:13:01.613 INFO:tasks.rados.rados.0.smithi123.stdout:956: done (15 left) 2020-01-16T00:13:01.614 INFO:tasks.rados.rados.0.smithi123.stdout:957: done (14 left) 2020-01-16T00:13:01.614 INFO:tasks.rados.rados.0.smithi123.stdout:958: done (13 left) 2020-01-16T00:13:01.614 INFO:tasks.rados.rados.0.smithi123.stdout:959: done (12 left) 2020-01-16T00:13:01.614 INFO:tasks.rados.rados.0.smithi123.stdout:960: done (11 left) 2020-01-16T00:13:01.614 INFO:tasks.rados.rados.0.smithi123.stdout:961: done (10 left) 2020-01-16T00:13:01.614 INFO:tasks.rados.rados.0.smithi123.stdout:962: done (9 left) 2020-01-16T00:13:01.614 INFO:tasks.rados.rados.0.smithi123.stdout:963: done (8 left) 2020-01-16T00:13:01.614 INFO:tasks.rados.rados.0.smithi123.stdout:972: write (excl) oid 13 current snap is 0 2020-01-16T00:13:01.615 INFO:tasks.rados.rados.0.smithi123.stdout:972: seq_num 230 ranges {665610=797860,1909329=458405,2816700=410210} 2020-01-16T00:13:01.630 INFO:tasks.rados.rados.0.smithi123.stdout:972: writing smithi12313934-13 from 665610 to 1463470 tid 1 2020-01-16T00:13:01.636 INFO:tasks.rados.rados.0.smithi123.stdout:972: writing smithi12313934-13 from 1909329 to 2367734 tid 2 2020-01-16T00:13:01.641 INFO:tasks.rados.rados.0.smithi123.stdout:972: writing smithi12313934-13 from 2816700 to 3226910 tid 3 2020-01-16T00:13:01.641 INFO:tasks.rados.rados.0.smithi123.stdout:973: read oid 29 snap -1 2020-01-16T00:13:01.642 INFO:tasks.rados.rados.0.smithi123.stdout:973: expect (ObjNum 200 snap 0 seq_num 200) 2020-01-16T00:13:01.642 INFO:tasks.rados.rados.0.smithi123.stdout:974: write oid 7 current snap is 0 2020-01-16T00:13:01.642 INFO:tasks.rados.rados.0.smithi123.stdout:974: seq_num 231 ranges {682417=473109,1567811=602063,2844123=621482} 2020-01-16T00:13:01.648 INFO:tasks.rados.rados.0.smithi123.stdout:974: writing smithi12313934-7 from 682417 to 1155526 tid 1 2020-01-16T00:13:01.654 INFO:tasks.rados.rados.0.smithi123.stdout:974: writing smithi12313934-7 from 1567811 to 2169874 tid 2 2020-01-16T00:13:01.662 INFO:tasks.rados.rados.0.smithi123.stdout:974: writing smithi12313934-7 from 2844123 to 3465605 tid 3 2020-01-16T00:13:01.663 INFO:tasks.rados.rados.0.smithi123.stdout:975: write (excl) oid 43 current snap is 0 2020-01-16T00:13:01.663 INFO:tasks.rados.rados.0.smithi123.stdout:975: seq_num 232 ranges {699224=548358,2026293=462074,3271546=432754} 2020-01-16T00:13:01.669 INFO:tasks.rados.rados.0.smithi123.stdout:975: writing smithi12313934-43 from 699224 to 1247582 tid 1 2020-01-16T00:13:01.676 INFO:tasks.rados.rados.0.smithi123.stdout:975: writing smithi12313934-43 from 2026293 to 2488367 tid 2 2020-01-16T00:13:01.683 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:13:01.679+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:13:01.684 INFO:tasks.rados.rados.0.smithi123.stdout:975: writing smithi12313934-43 from 3271546 to 3704300 tid 3 2020-01-16T00:13:01.684 INFO:tasks.rados.rados.0.smithi123.stdout:976: delete oid 1 current snap is 0 2020-01-16T00:13:01.684 INFO:tasks.rados.rados.0.smithi123.stdout:964: finishing write tid 3 to smithi12313934-30 2020-01-16T00:13:01.684 INFO:tasks.rados.rados.0.smithi123.stdout:964: finishing write tid 5 to smithi12313934-30 2020-01-16T00:13:01.684 INFO:tasks.rados.rados.0.smithi123.stdout:964: finishing write tid 6 to smithi12313934-30 2020-01-16T00:13:01.684 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 30 v 87 (ObjNum 225 snap 0 seq_num 225) dirty exists 2020-01-16T00:13:01.684 INFO:tasks.rados.rados.0.smithi123.stdout:964: left oid 30 (ObjNum 225 snap 0 seq_num 225) 2020-01-16T00:13:01.685 INFO:tasks.rados.rados.0.smithi123.stdout:965: finishing write tid 1 to smithi12313934-46 2020-01-16T00:13:01.685 INFO:tasks.rados.rados.0.smithi123.stdout:965: finishing write tid 2 to smithi12313934-46 2020-01-16T00:13:01.685 INFO:tasks.rados.rados.0.smithi123.stdout:965: finishing write tid 3 to smithi12313934-46 2020-01-16T00:13:01.685 INFO:tasks.rados.rados.0.smithi123.stdout:965: finishing write tid 5 to smithi12313934-46 2020-01-16T00:13:01.685 INFO:tasks.rados.rados.0.smithi123.stdout:965: finishing write tid 6 to smithi12313934-46 2020-01-16T00:13:01.685 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 46 v 192 (ObjNum 226 snap 0 seq_num 226) dirty exists 2020-01-16T00:13:01.685 INFO:tasks.rados.rados.0.smithi123.stdout:965: left oid 46 (ObjNum 226 snap 0 seq_num 226) 2020-01-16T00:13:01.686 INFO:tasks.rados.rados.0.smithi123.stdout:966: finishing write tid 1 to smithi12313934-23 2020-01-16T00:13:01.686 INFO:tasks.rados.rados.0.smithi123.stdout:966: finishing write tid 2 to smithi12313934-23 2020-01-16T00:13:01.686 INFO:tasks.rados.rados.0.smithi123.stdout:967: expect (ObjNum 130 snap 0 seq_num 130) 2020-01-16T00:13:01.701 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:01.701 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:13:01.702 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:13:01.702 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:13:01.702 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.702 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.5.13:36 2.0 2:20e40cec:::smithi12313934-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 113152~1699840] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:13:01.702 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:13:01.702 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 23.869297522, 2020-01-16T00:13:01.702 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011736636, 2020-01-16T00:13:01.703 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:01.703 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:01.703 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:01.703 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.703 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:01.703 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:13:01.703 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:01.703 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.704 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.704 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:01.704 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:13:01.704 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999952 2020-01-16T00:13:01.704 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.704 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.704 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:01.705 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831934+0000", 2020-01-16T00:13:01.705 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001034325 2020-01-16T00:13:01.705 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.705 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.705 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:01.705 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.832968+0000", 2020-01-16T00:13:01.710 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.506e-05 2020-01-16T00:13:01.711 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.711 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.711 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:01.711 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833004+0000", 2020-01-16T00:13:01.711 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3406e-05 2020-01-16T00:13:01.712 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.712 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.712 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:01.712 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833017+0000", 2020-01-16T00:13:01.713 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015477000000000001 2020-01-16T00:13:01.713 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.713 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.713 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.713 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833172+0000", 2020-01-16T00:13:01.714 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033750700000000002 2020-01-16T00:13:01.714 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.714 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.714 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:01.714 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833509+0000", 2020-01-16T00:13:01.714 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00055861400000000003 2020-01-16T00:13:01.714 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.715 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.715 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:13:01.715 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.834068+0000", 2020-01-16T00:13:01.715 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.007474545 2020-01-16T00:13:01.715 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.715 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.715 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:01.715 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.841542+0000", 2020-01-16T00:13:01.716 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00056319499999999995 2020-01-16T00:13:01.716 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.718 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.719 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:01.719 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.842106+0000", 2020-01-16T00:13:01.719 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001371017 2020-01-16T00:13:01.719 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.719 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.720 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:01.720 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843477+0000", 2020-01-16T00:13:01.720 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000123938 2020-01-16T00:13:01.720 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.720 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.721 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:01.721 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843601+0000", 2020-01-16T00:13:01.721 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5012000000000006e-05 2020-01-16T00:13:01.721 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.721 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.721 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:01.722 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843676+0000", 2020-01-16T00:13:01.722 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011736636 2020-01-16T00:13:01.722 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.722 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.722 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.722 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.722 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.723 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.5.13:40 2.0 2:20e40cec:::smithi12313934-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 113152~1699840] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:13:01.723 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.847376+0000", 2020-01-16T00:13:01.723 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 23.853860793999999, 2020-01-16T00:13:01.723 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010871093, 2020-01-16T00:13:01.723 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:01.724 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:01.724 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:01.724 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.724 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:01.724 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.847376+0000", 2020-01-16T00:13:01.724 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:01.724 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.725 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.725 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:01.725 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.847376+0000", 2020-01-16T00:13:01.725 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:01.725 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.725 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.725 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:01.726 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.847373+0000", 2020-01-16T00:13:01.726 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00077394699999999998 2020-01-16T00:13:01.726 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.726 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.726 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:01.726 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.848147+0000", 2020-01-16T00:13:01.726 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.173e-05 2020-01-16T00:13:01.727 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.727 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.727 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:01.727 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.848168+0000", 2020-01-16T00:13:01.727 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2738999999999999e-05 2020-01-16T00:13:01.728 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.728 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.728 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:01.728 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.848181+0000", 2020-01-16T00:13:01.729 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4627999999999999e-05 2020-01-16T00:13:01.742 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.742 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.742 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.742 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.848236+0000", 2020-01-16T00:13:01.742 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000141311 2020-01-16T00:13:01.742 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.743 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.743 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:01.743 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.848377+0000", 2020-01-16T00:13:01.743 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015462700000000001 2020-01-16T00:13:01.743 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.743 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.743 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:13:01.743 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.848532+0000", 2020-01-16T00:13:01.744 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0039933540000000002 2020-01-16T00:13:01.744 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.744 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.744 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:01.744 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.852525+0000", 2020-01-16T00:13:01.744 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0051016129999999996 2020-01-16T00:13:01.744 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.744 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.745 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:01.745 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.857627+0000", 2020-01-16T00:13:01.745 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00044689099999999999 2020-01-16T00:13:01.745 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.745 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.745 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:01.745 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.858073+0000", 2020-01-16T00:13:01.746 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8947e-05 2020-01-16T00:13:01.746 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.746 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.746 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:01.746 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.858172+0000", 2020-01-16T00:13:01.746 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4369999999999994e-05 2020-01-16T00:13:01.746 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.746 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.747 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:01.747 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.858247+0000", 2020-01-16T00:13:01.747 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010871093 2020-01-16T00:13:01.747 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.747 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.747 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.747 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.747 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.748 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.5.13:42 2.0 2:313458df:::smithi12313934-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 263680~1668096] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:13:01.748 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.939100+0000", 2020-01-16T00:13:01.748 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 23.762136886, 2020-01-16T00:13:01.748 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010934255, 2020-01-16T00:13:01.748 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:01.748 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:01.748 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:01.749 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.749 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:01.749 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939100+0000", 2020-01-16T00:13:01.749 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:01.749 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.749 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.749 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:01.750 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939100+0000", 2020-01-16T00:13:01.750 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:01.750 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.750 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.750 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:01.750 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939097+0000", 2020-01-16T00:13:01.750 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00049374800000000002 2020-01-16T00:13:01.750 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.751 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.751 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:01.751 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939591+0000", 2020-01-16T00:13:01.751 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0840999999999999e-05 2020-01-16T00:13:01.751 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.751 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.751 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:01.751 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939612+0000", 2020-01-16T00:13:01.752 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4749999999999998e-06 2020-01-16T00:13:01.752 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.752 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.752 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:01.752 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939618+0000", 2020-01-16T00:13:01.752 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6371e-05 2020-01-16T00:13:01.752 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.753 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.753 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.753 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939674+0000", 2020-01-16T00:13:01.753 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2e-05 2020-01-16T00:13:01.753 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.753 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.753 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:01.753 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939766+0000", 2020-01-16T00:13:01.754 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020133999999999999 2020-01-16T00:13:01.754 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.754 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.754 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:13:01.754 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939968+0000", 2020-01-16T00:13:01.754 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0027942560000000002 2020-01-16T00:13:01.754 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.754 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.755 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:01.755 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.942762+0000", 2020-01-16T00:13:01.755 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.004998466 2020-01-16T00:13:01.755 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.755 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.755 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:01.755 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.947760+0000", 2020-01-16T00:13:01.755 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0021783330000000002 2020-01-16T00:13:01.756 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.756 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.756 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:01.756 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.949939+0000", 2020-01-16T00:13:01.756 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2391999999999997e-05 2020-01-16T00:13:01.756 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.756 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.757 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:01.757 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.949981+0000", 2020-01-16T00:13:01.757 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2629e-05 2020-01-16T00:13:01.757 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.757 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.757 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:01.757 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.950034+0000", 2020-01-16T00:13:01.757 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010934255 2020-01-16T00:13:01.758 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.758 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.758 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.758 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.758 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.758 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.5.13:43 2.0 2:313458df:::smithi12313934-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 263680~1668096] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:13:01.758 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.953019+0000", 2020-01-16T00:13:01.759 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 23.748217090000001, 2020-01-16T00:13:01.759 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0085200620000000001, 2020-01-16T00:13:01.759 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:01.759 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:01.759 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:01.759 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.759 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:01.759 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953019+0000", 2020-01-16T00:13:01.760 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:01.760 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.760 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.760 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:01.760 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953019+0000", 2020-01-16T00:13:01.760 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:01.760 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.760 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.761 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:01.761 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953016+0000", 2020-01-16T00:13:01.761 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00058927999999999997 2020-01-16T00:13:01.762 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.762 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.763 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:01.763 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953606+0000", 2020-01-16T00:13:01.763 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0846999999999999e-05 2020-01-16T00:13:01.763 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.763 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.763 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:01.763 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953627+0000", 2020-01-16T00:13:01.763 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7020000000000004e-06 2020-01-16T00:13:01.764 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.764 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.764 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:01.764 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953633+0000", 2020-01-16T00:13:01.764 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.2762000000000005e-05 2020-01-16T00:13:01.764 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.764 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.764 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.765 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953716+0000", 2020-01-16T00:13:01.765 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000139238 2020-01-16T00:13:01.765 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.765 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.765 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:01.765 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.953855+0000", 2020-01-16T00:13:01.765 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020386300000000001 2020-01-16T00:13:01.766 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.766 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.766 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:13:01.766 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.954059+0000", 2020-01-16T00:13:01.766 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0031229399999999998 2020-01-16T00:13:01.766 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.766 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.766 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:01.767 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.957182+0000", 2020-01-16T00:13:01.767 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0034781980000000001 2020-01-16T00:13:01.767 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.767 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.767 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:01.767 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.960660+0000", 2020-01-16T00:13:01.767 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00076596200000000002 2020-01-16T00:13:01.768 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.768 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.768 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:01.768 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.961426+0000", 2020-01-16T00:13:01.768 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1937e-05 2020-01-16T00:13:01.768 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.768 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.768 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:01.769 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.961488+0000", 2020-01-16T00:13:01.769 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1285000000000001e-05 2020-01-16T00:13:01.769 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.769 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.769 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:01.769 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.961539+0000", 2020-01-16T00:13:01.769 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0085200620000000001 2020-01-16T00:13:01.769 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.770 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.770 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.770 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.770 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.770 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1186 3.d 3:b1521a3e:::smithi12313934-25:head [write 682384~456830,stat] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:13:01.772 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.258156+0000", 2020-01-16T00:13:01.772 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 19.443080922, 2020-01-16T00:13:01.772 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0092969679999999992, 2020-01-16T00:13:01.772 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:01.772 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:01.772 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:01.772 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:01.773 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:01.773 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1186 2020-01-16T00:13:01.773 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.773 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:01.773 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.773 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:01.773 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.258156+0000", 2020-01-16T00:13:01.773 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:01.774 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.774 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.774 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:01.774 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.258156+0000", 2020-01-16T00:13:01.774 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999943 2020-01-16T00:13:01.774 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.774 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.776 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:01.776 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.258150+0000", 2020-01-16T00:13:01.776 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001866517 2020-01-16T00:13:01.776 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.776 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.776 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:01.777 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.260016+0000", 2020-01-16T00:13:01.777 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4024999999999999e-05 2020-01-16T00:13:01.777 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.777 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.777 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:01.777 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.260070+0000", 2020-01-16T00:13:01.777 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4295e-05 2020-01-16T00:13:01.778 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.778 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.778 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:01.778 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.260085+0000", 2020-01-16T00:13:01.778 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013625599999999999 2020-01-16T00:13:01.778 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.778 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.778 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.779 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.260221+0000", 2020-01-16T00:13:01.779 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000230015 2020-01-16T00:13:01.779 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.779 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.779 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:01.779 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.260451+0000", 2020-01-16T00:13:01.779 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00031965999999999999 2020-01-16T00:13:01.780 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.780 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.780 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:01.780 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.260770+0000", 2020-01-16T00:13:01.780 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0031788649999999999 2020-01-16T00:13:01.780 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.780 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.780 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:01.781 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.263949+0000", 2020-01-16T00:13:01.781 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0022620090000000002 2020-01-16T00:13:01.781 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.781 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.781 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:01.781 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.266211+0000", 2020-01-16T00:13:01.781 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00097717199999999989 2020-01-16T00:13:01.782 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.782 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.782 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:01.782 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.267188+0000", 2020-01-16T00:13:01.782 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000128594 2020-01-16T00:13:01.782 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.782 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.782 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:01.783 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.267317+0000", 2020-01-16T00:13:01.783 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013541200000000001 2020-01-16T00:13:01.783 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.783 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.783 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:01.783 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.267452+0000", 2020-01-16T00:13:01.783 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0092969679999999992 2020-01-16T00:13:01.783 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.784 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.784 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.784 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.784 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.784 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1187 3.d 3:b1521a3e:::smithi12313934-25:head [write 1721042~423699] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:13:01.784 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.263828+0000", 2020-01-16T00:13:01.784 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 19.43740845, 2020-01-16T00:13:01.785 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011305709000000001, 2020-01-16T00:13:01.785 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:01.785 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:01.785 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:01.785 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:01.785 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:01.785 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1187 2020-01-16T00:13:01.785 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.786 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:01.786 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.786 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:01.786 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.263828+0000", 2020-01-16T00:13:01.786 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:01.786 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.786 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.787 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:01.787 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.263828+0000", 2020-01-16T00:13:01.787 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:13:01.787 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.787 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.787 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:01.787 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.263824+0000", 2020-01-16T00:13:01.787 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00048485899999999997 2020-01-16T00:13:01.788 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.788 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.788 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:01.788 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264309+0000", 2020-01-16T00:13:01.788 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5622999999999999e-05 2020-01-16T00:13:01.789 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.789 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.789 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:01.789 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264325+0000", 2020-01-16T00:13:01.789 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2530000000000004e-06 2020-01-16T00:13:01.789 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.789 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.790 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:01.790 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264330+0000", 2020-01-16T00:13:01.790 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000102252 2020-01-16T00:13:01.790 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.790 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.790 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.790 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264432+0000", 2020-01-16T00:13:01.790 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022361000000000001 2020-01-16T00:13:01.791 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.791 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.791 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:01.791 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264656+0000", 2020-01-16T00:13:01.791 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0028688849999999998 2020-01-16T00:13:01.791 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.791 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.791 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.792 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.267525+0000", 2020-01-16T00:13:01.792 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00023640900000000001 2020-01-16T00:13:01.792 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.792 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.792 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:01.792 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.267761+0000", 2020-01-16T00:13:01.793 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00035667000000000001 2020-01-16T00:13:01.793 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.793 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.793 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:01.793 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.268118+0000", 2020-01-16T00:13:01.793 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023898040000000001 2020-01-16T00:13:01.793 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.793 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.794 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:01.794 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.270507+0000", 2020-01-16T00:13:01.794 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014919690000000001 2020-01-16T00:13:01.794 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.794 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.795 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:01.795 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.271999+0000", 2020-01-16T00:13:01.795 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0029764510000000002 2020-01-16T00:13:01.795 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.796 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.796 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:01.796 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.274976+0000", 2020-01-16T00:13:01.796 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5409999999999997e-05 2020-01-16T00:13:01.796 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.796 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.796 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:01.797 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.275061+0000", 2020-01-16T00:13:01.797 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2452000000000004e-05 2020-01-16T00:13:01.797 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.797 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.797 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:01.797 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.275134+0000", 2020-01-16T00:13:01.798 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011305709000000001 2020-01-16T00:13:01.798 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.798 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.798 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.798 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.798 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.798 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1259 3.d 3:b374ba21:::smithi12313934-20:head [read 0~3932805,checksum] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:13:01.798 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.833857+0000", 2020-01-16T00:13:01.799 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 18.867379706000001, 2020-01-16T00:13:01.799 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0094623380000000007, 2020-01-16T00:13:01.799 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:01.799 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:01.799 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:01.799 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:01.799 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:01.800 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1259 2020-01-16T00:13:01.800 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.800 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:01.800 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.800 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:01.800 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.833857+0000", 2020-01-16T00:13:01.800 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:01.800 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.801 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.801 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:01.801 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.833857+0000", 2020-01-16T00:13:01.801 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:01.801 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.801 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.801 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:01.801 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.833856+0000", 2020-01-16T00:13:01.802 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.9659999999999999e-06 2020-01-16T00:13:01.802 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.802 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.802 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:01.802 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.833863+0000", 2020-01-16T00:13:01.802 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4699999999999996e-06 2020-01-16T00:13:01.802 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.803 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.803 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:01.803 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.833870+0000", 2020-01-16T00:13:01.803 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0039999999999999e-06 2020-01-16T00:13:01.803 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.803 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.803 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:01.803 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.833875+0000", 2020-01-16T00:13:01.804 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0069094480000000003 2020-01-16T00:13:01.804 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.804 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.804 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.804 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.840785+0000", 2020-01-16T00:13:01.804 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020800699999999999 2020-01-16T00:13:01.804 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.804 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.805 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:01.805 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.840993+0000", 2020-01-16T00:13:01.805 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023263960000000001 2020-01-16T00:13:01.805 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.805 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.805 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:01.805 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.843319+0000", 2020-01-16T00:13:01.806 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0094623380000000007 2020-01-16T00:13:01.806 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.806 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.806 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.806 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.806 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.806 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1272 3.d 3:b3dd5d45:::smithi12313934-16:head [read 0~3008420,checksum] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:13:01.806 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074658+0000", 2020-01-16T00:13:01.807 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 18.626578865999999, 2020-01-16T00:13:01.807 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0083000039999999997, 2020-01-16T00:13:01.807 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:01.807 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:01.807 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:01.807 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:01.807 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:01.807 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1272 2020-01-16T00:13:01.808 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.808 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:01.808 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.808 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:01.808 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074658+0000", 2020-01-16T00:13:01.808 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:01.808 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.808 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.809 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:01.809 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074658+0000", 2020-01-16T00:13:01.809 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:01.809 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.809 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.809 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:01.809 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074657+0000", 2020-01-16T00:13:01.810 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7270000000000007e-06 2020-01-16T00:13:01.810 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.810 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.810 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:01.810 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074664+0000", 2020-01-16T00:13:01.810 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1094000000000002e-05 2020-01-16T00:13:01.811 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.811 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.811 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:01.811 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074685+0000", 2020-01-16T00:13:01.811 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0260000000000004e-06 2020-01-16T00:13:01.812 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.812 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.812 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:01.812 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074691+0000", 2020-01-16T00:13:01.812 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0061756880000000004 2020-01-16T00:13:01.812 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.812 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.812 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.813 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.080867+0000", 2020-01-16T00:13:01.813 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015871700000000001 2020-01-16T00:13:01.814 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.814 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.814 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:01.814 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.081026+0000", 2020-01-16T00:13:01.815 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0019317519999999999 2020-01-16T00:13:01.815 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.815 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.815 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:01.815 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.082958+0000", 2020-01-16T00:13:01.816 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0083000039999999997 2020-01-16T00:13:01.816 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.816 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.817 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.817 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.817 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.817 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1276 3.d 3:b1521a3e:::smithi12313934-25:head [read 0~3882384,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:13:01.817 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:13:01.817 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 18.626537952, 2020-01-16T00:13:01.817 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015839157999999999, 2020-01-16T00:13:01.818 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:01.818 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:01.818 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:01.818 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:01.818 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:01.818 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1276 2020-01-16T00:13:01.818 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.819 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:01.819 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.819 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:01.819 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:13:01.819 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:01.819 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.819 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.820 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:01.820 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:13:01.820 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:01.820 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.820 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.820 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:01.821 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:13:01.821 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5429999999999999e-06 2020-01-16T00:13:01.821 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.821 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.821 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:01.821 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074704+0000", 2020-01-16T00:13:01.821 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0839999999999992e-06 2020-01-16T00:13:01.822 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.822 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.822 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:01.822 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074712+0000", 2020-01-16T00:13:01.822 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.173e-06 2020-01-16T00:13:01.822 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.822 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.823 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:01.823 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074718+0000", 2020-01-16T00:13:01.823 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0083034700000000003 2020-01-16T00:13:01.823 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.823 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.823 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.823 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.083022+0000", 2020-01-16T00:13:01.824 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028456400000000001 2020-01-16T00:13:01.824 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.824 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.824 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:01.824 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.083306+0000", 2020-01-16T00:13:01.824 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0072312640000000003 2020-01-16T00:13:01.824 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.825 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.825 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:01.825 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090538+0000", 2020-01-16T00:13:01.825 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015839157999999999 2020-01-16T00:13:01.825 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.825 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.826 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.826 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.826 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.826 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1277 3.d 3:b1521a3e:::smithi12313934-25:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:13:01.827 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:13:01.827 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 18.626494361999999, 2020-01-16T00:13:01.827 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016646459999999998, 2020-01-16T00:13:01.827 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:01.827 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:01.827 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:01.828 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:01.828 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:01.828 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1277 2020-01-16T00:13:01.828 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.828 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:01.828 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.828 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:01.829 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:13:01.829 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:01.829 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.829 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.829 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:01.829 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:13:01.829 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:01.830 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.830 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.830 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:01.830 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074741+0000", 2020-01-16T00:13:01.830 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0776e-05 2020-01-16T00:13:01.830 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.830 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.831 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:01.831 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074752+0000", 2020-01-16T00:13:01.831 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7600000000000002e-06 2020-01-16T00:13:01.831 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.831 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.831 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:01.831 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074759+0000", 2020-01-16T00:13:01.832 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6840000000000001e-06 2020-01-16T00:13:01.832 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.832 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.832 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:01.833 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074765+0000", 2020-01-16T00:13:01.833 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015815216 2020-01-16T00:13:01.833 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.833 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.833 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.834 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090580+0000", 2020-01-16T00:13:01.834 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000182404 2020-01-16T00:13:01.834 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.834 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.834 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:01.834 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090763+0000", 2020-01-16T00:13:01.835 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00062579300000000003 2020-01-16T00:13:01.835 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.835 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.835 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:01.835 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091389+0000", 2020-01-16T00:13:01.835 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016646459999999998 2020-01-16T00:13:01.835 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.836 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.836 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.836 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.836 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.837 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1278 3.d 3:b1521a3e:::smithi12313934-25:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:13:01.837 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:13:01.837 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 18.626464648999999, 2020-01-16T00:13:01.837 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017994915, 2020-01-16T00:13:01.837 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:01.837 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:01.838 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:01.838 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:01.838 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:01.838 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1278 2020-01-16T00:13:01.838 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.838 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:01.839 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.839 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:01.839 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:13:01.839 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:01.839 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.839 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.840 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:01.840 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:13:01.840 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:01.840 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.840 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.840 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:01.841 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074771+0000", 2020-01-16T00:13:01.841 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4569999999999998e-06 2020-01-16T00:13:01.841 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.841 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.841 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:01.841 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074775+0000", 2020-01-16T00:13:01.841 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0330000000000003e-06 2020-01-16T00:13:01.842 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.842 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.842 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:01.842 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074782+0000", 2020-01-16T00:13:01.842 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9470000000000002e-06 2020-01-16T00:13:01.842 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.842 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.843 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:01.843 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074787+0000", 2020-01-16T00:13:01.843 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01666134 2020-01-16T00:13:01.843 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.843 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.843 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.844 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091449+0000", 2020-01-16T00:13:01.844 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000247731 2020-01-16T00:13:01.844 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.844 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.844 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:01.844 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091696+0000", 2020-01-16T00:13:01.845 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010703170000000001 2020-01-16T00:13:01.845 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.845 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.845 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:01.845 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.092767+0000", 2020-01-16T00:13:01.845 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017994915 2020-01-16T00:13:01.846 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.846 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.846 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.846 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.846 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.847 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1420 3.e 3:7cfd7007:::smithi12313934-21:head [read 0~3899191,checksum] snapc 0=[] ondisk+read+known_if_redirected e28)", 2020-01-16T00:13:01.847 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:47.689967+0000", 2020-01-16T00:13:01.847 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.011269364, 2020-01-16T00:13:01.847 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010893273, 2020-01-16T00:13:01.848 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:01.848 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:01.848 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:01.848 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:01.848 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:01.848 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1420 2020-01-16T00:13:01.849 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.849 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:01.849 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.849 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:01.849 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.689967+0000", 2020-01-16T00:13:01.849 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:01.850 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.850 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.850 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:01.850 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.689967+0000", 2020-01-16T00:13:01.850 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:01.850 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.851 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.851 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:01.851 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.689965+0000", 2020-01-16T00:13:01.851 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.696e-05 2020-01-16T00:13:01.851 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.851 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.852 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:01.852 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.689982+0000", 2020-01-16T00:13:01.852 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9613e-05 2020-01-16T00:13:01.852 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.852 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.852 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:01.853 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.690002+0000", 2020-01-16T00:13:01.853 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3003000000000001e-05 2020-01-16T00:13:01.853 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.853 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.853 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:01.853 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.690015+0000", 2020-01-16T00:13:01.854 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0073546410000000003 2020-01-16T00:13:01.854 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.854 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.854 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.854 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.697370+0000", 2020-01-16T00:13:01.854 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00031806899999999998 2020-01-16T00:13:01.854 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.855 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.855 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:01.855 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.697688+0000", 2020-01-16T00:13:01.855 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003172673 2020-01-16T00:13:01.855 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.855 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.855 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:01.856 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.700860+0000", 2020-01-16T00:13:01.856 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010893273 2020-01-16T00:13:01.856 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.856 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.856 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.856 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.857 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.857 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.4.13:40 2.0 2:01b4bead:::smithi12313934-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 640512~403968] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e28)", 2020-01-16T00:13:01.857 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:47.904891+0000", 2020-01-16T00:13:01.857 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 13.796345715999999, 2020-01-16T00:13:01.858 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0078047129999999996, 2020-01-16T00:13:01.858 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:01.858 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:01.858 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:01.858 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.858 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:01.859 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.904891+0000", 2020-01-16T00:13:01.859 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:01.859 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.859 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.859 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:01.860 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.904891+0000", 2020-01-16T00:13:01.860 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:01.860 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.860 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.860 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:01.860 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.904889+0000", 2020-01-16T00:13:01.860 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017825 2020-01-16T00:13:01.861 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.861 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.861 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:01.861 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.905067+0000", 2020-01-16T00:13:01.861 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4921e-05 2020-01-16T00:13:01.861 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.862 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.862 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:01.862 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.905082+0000", 2020-01-16T00:13:01.862 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5339999999999999e-06 2020-01-16T00:13:01.862 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.862 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.862 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:01.863 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.905087+0000", 2020-01-16T00:13:01.863 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011594 2020-01-16T00:13:01.863 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.863 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.863 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.863 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.905203+0000", 2020-01-16T00:13:01.863 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00026064000000000003 2020-01-16T00:13:01.864 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.864 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.864 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:01.864 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.905464+0000", 2020-01-16T00:13:01.864 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00044418700000000002 2020-01-16T00:13:01.864 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.865 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.865 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:13:01.865 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.905908+0000", 2020-01-16T00:13:01.865 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0055870989999999999 2020-01-16T00:13:01.865 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.866 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.866 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:01.866 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.911495+0000", 2020-01-16T00:13:01.866 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3438e-05 2020-01-16T00:13:01.866 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.866 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.867 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:01.867 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.911539+0000", 2020-01-16T00:13:01.867 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001073112 2020-01-16T00:13:01.867 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.867 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.868 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:01.868 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.912612+0000", 2020-01-16T00:13:01.868 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2938e-05 2020-01-16T00:13:01.868 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.868 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.868 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:01.869 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.912665+0000", 2020-01-16T00:13:01.869 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0811e-05 2020-01-16T00:13:01.869 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.869 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.869 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:01.869 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.912695+0000", 2020-01-16T00:13:01.869 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0078047129999999996 2020-01-16T00:13:01.870 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.870 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.870 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.870 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.870 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.870 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.4.13:41 2.0 2:01b4bead:::smithi12313934-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 1445888~278528] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e28)", 2020-01-16T00:13:01.870 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:47.905100+0000", 2020-01-16T00:13:01.871 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 13.796136513, 2020-01-16T00:13:01.871 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0079266059999999992, 2020-01-16T00:13:01.871 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:01.871 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:01.871 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:01.871 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.872 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:01.872 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.905100+0000", 2020-01-16T00:13:01.872 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:01.872 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.872 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.872 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:01.872 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.905100+0000", 2020-01-16T00:13:01.873 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:01.873 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.873 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.873 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:01.873 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.905099+0000", 2020-01-16T00:13:01.873 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3053000000000005e-05 2020-01-16T00:13:01.874 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.874 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.874 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:01.874 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.905192+0000", 2020-01-16T00:13:01.874 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2110000000000003e-06 2020-01-16T00:13:01.874 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.874 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.875 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:01.875 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.905198+0000", 2020-01-16T00:13:01.875 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4429999999999999e-06 2020-01-16T00:13:01.875 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.875 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.875 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:01.876 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.905202+0000", 2020-01-16T00:13:01.876 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003197858 2020-01-16T00:13:01.876 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.876 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.876 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.876 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.908400+0000", 2020-01-16T00:13:01.876 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012500200000000001 2020-01-16T00:13:01.877 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.877 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.877 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:01.877 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.908525+0000", 2020-01-16T00:13:01.877 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000183332 2020-01-16T00:13:01.877 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.878 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.878 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:13:01.878 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.908708+0000", 2020-01-16T00:13:01.879 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0027965339999999998 2020-01-16T00:13:01.879 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.879 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.879 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:01.879 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.911505+0000", 2020-01-16T00:13:01.879 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012356649999999999 2020-01-16T00:13:01.879 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.880 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.880 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:01.880 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.912740+0000", 2020-01-16T00:13:01.880 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022262900000000001 2020-01-16T00:13:01.880 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.880 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.881 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:01.881 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.912963+0000", 2020-01-16T00:13:01.881 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4484999999999999e-05 2020-01-16T00:13:01.881 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.881 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.881 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:01.881 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.912997+0000", 2020-01-16T00:13:01.882 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9139999999999999e-05 2020-01-16T00:13:01.882 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.882 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.882 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:01.882 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.913027+0000", 2020-01-16T00:13:01.882 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0079266059999999992 2020-01-16T00:13:01.883 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.883 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.883 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.883 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.883 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.883 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1870 3.d 3:bd9fc72f:::smithi12313934-5:head [write 2804576~435035] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:13:01.884 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:56.384430+0000", 2020-01-16T00:13:01.884 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.3168065889999996, 2020-01-16T00:13:01.884 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0085577770000000008, 2020-01-16T00:13:01.884 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:01.884 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:01.884 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:01.885 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:01.885 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:01.885 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1870 2020-01-16T00:13:01.885 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.885 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:01.885 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.886 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:01.886 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384430+0000", 2020-01-16T00:13:01.886 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:01.886 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.886 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.886 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:01.886 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384430+0000", 2020-01-16T00:13:01.887 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999933 2020-01-16T00:13:01.887 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.887 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.887 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:01.887 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384423+0000", 2020-01-16T00:13:01.887 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042953000000000002 2020-01-16T00:13:01.888 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.888 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.888 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:01.888 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384853+0000", 2020-01-16T00:13:01.889 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2592000000000001e-05 2020-01-16T00:13:01.889 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.889 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.889 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:01.893 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384865+0000", 2020-01-16T00:13:01.893 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7559999999999999e-06 2020-01-16T00:13:01.893 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.893 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.893 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:01.893 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384870+0000", 2020-01-16T00:13:01.893 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8498000000000003e-05 2020-01-16T00:13:01.894 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.894 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.894 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.894 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384909+0000", 2020-01-16T00:13:01.894 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00075234 2020-01-16T00:13:01.894 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.894 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.894 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:01.895 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.385661+0000", 2020-01-16T00:13:01.895 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.004714963 2020-01-16T00:13:01.895 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.895 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.895 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.895 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.390376+0000", 2020-01-16T00:13:01.895 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000274324 2020-01-16T00:13:01.896 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.896 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.896 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:01.896 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.390650+0000", 2020-01-16T00:13:01.896 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024631000000000002 2020-01-16T00:13:01.896 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.896 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.896 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:01.897 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.390897+0000", 2020-01-16T00:13:01.897 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014908829999999999 2020-01-16T00:13:01.897 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.897 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.897 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:01.897 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.392387+0000", 2020-01-16T00:13:01.898 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00023496300000000001 2020-01-16T00:13:01.898 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.898 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.898 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:01.898 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.392622+0000", 2020-01-16T00:13:01.899 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00027025500000000001 2020-01-16T00:13:01.899 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.899 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.899 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:01.899 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.392893+0000", 2020-01-16T00:13:01.899 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0452000000000003e-05 2020-01-16T00:13:01.900 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.900 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.900 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:01.900 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.392953+0000", 2020-01-16T00:13:01.900 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4564999999999999e-05 2020-01-16T00:13:01.900 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.901 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.901 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:01.901 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.392988+0000", 2020-01-16T00:13:01.901 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0085577770000000008 2020-01-16T00:13:01.901 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.901 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.901 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.901 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.902 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.902 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1871 3.d 3:bd9fc72f:::smithi12313934-5:head [setxattr _header (55) in=7b,truncate 3327786] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:13:01.902 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:56.384879+0000", 2020-01-16T00:13:01.902 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.316357569, 2020-01-16T00:13:01.902 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010769713, 2020-01-16T00:13:01.902 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:01.902 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:01.903 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:01.903 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:01.903 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:01.903 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1871 2020-01-16T00:13:01.903 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.903 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:01.904 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.904 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:01.904 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384879+0000", 2020-01-16T00:13:01.904 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:01.904 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.904 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.904 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:01.905 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384879+0000", 2020-01-16T00:13:01.905 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:01.905 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.905 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.905 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:01.905 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384878+0000", 2020-01-16T00:13:01.905 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9370000000000003e-06 2020-01-16T00:13:01.906 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.906 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.906 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:01.907 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384883+0000", 2020-01-16T00:13:01.907 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6589999999999996e-06 2020-01-16T00:13:01.907 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.907 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.907 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:01.907 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384891+0000", 2020-01-16T00:13:01.907 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9330000000000001e-06 2020-01-16T00:13:01.907 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.908 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.908 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:01.908 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384894+0000", 2020-01-16T00:13:01.908 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00080669800000000001 2020-01-16T00:13:01.908 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.908 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.908 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.909 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.385701+0000", 2020-01-16T00:13:01.909 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3726000000000002e-05 2020-01-16T00:13:01.909 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.909 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.909 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:01.909 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.385735+0000", 2020-01-16T00:13:01.910 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0059567650000000002 2020-01-16T00:13:01.910 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.910 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.910 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.911 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.391692+0000", 2020-01-16T00:13:01.911 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013010300000000001 2020-01-16T00:13:01.911 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.911 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.911 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:01.912 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.391822+0000", 2020-01-16T00:13:01.912 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016664800000000001 2020-01-16T00:13:01.912 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.912 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.913 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:01.913 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.391988+0000", 2020-01-16T00:13:01.913 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00052670799999999997 2020-01-16T00:13:01.913 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.914 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.914 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:01.915 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.392515+0000", 2020-01-16T00:13:01.915 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00052796600000000005 2020-01-16T00:13:01.915 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.915 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.915 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:01.915 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.393043+0000", 2020-01-16T00:13:01.916 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002493501 2020-01-16T00:13:01.916 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.916 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.916 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:01.916 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.395537+0000", 2020-01-16T00:13:01.916 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9033999999999998e-05 2020-01-16T00:13:01.917 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.917 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.917 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:01.917 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.395596+0000", 2020-01-16T00:13:01.917 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3052e-05 2020-01-16T00:13:01.917 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.917 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.917 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:01.918 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.395649+0000", 2020-01-16T00:13:01.918 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010769713 2020-01-16T00:13:01.918 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.918 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.918 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.918 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.918 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.919 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1872 3.d 3:bd9fc72f:::smithi12313934-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e31)", 2020-01-16T00:13:01.919 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:56.384904+0000", 2020-01-16T00:13:01.919 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.3163319700000002, 2020-01-16T00:13:01.919 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011110386999999999, 2020-01-16T00:13:01.919 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:01.919 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:01.919 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:01.919 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:01.920 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:01.920 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1872 2020-01-16T00:13:01.920 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.920 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:01.920 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.920 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:01.920 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384904+0000", 2020-01-16T00:13:01.921 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:01.921 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.921 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.921 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:01.921 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384904+0000", 2020-01-16T00:13:01.921 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:01.921 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.921 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.922 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:01.922 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384904+0000", 2020-01-16T00:13:01.922 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3879999999999998e-06 2020-01-16T00:13:01.922 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.922 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.922 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:01.922 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384906+0000", 2020-01-16T00:13:01.922 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7479999999999999e-06 2020-01-16T00:13:01.923 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.923 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.923 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:01.924 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384910+0000", 2020-01-16T00:13:01.924 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4609999999999998e-06 2020-01-16T00:13:01.924 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.924 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.924 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:01.924 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384913+0000", 2020-01-16T00:13:01.924 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00083872899999999997 2020-01-16T00:13:01.925 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.925 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.925 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.925 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.385751+0000", 2020-01-16T00:13:01.925 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3528999999999997e-05 2020-01-16T00:13:01.925 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.925 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.926 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:01.926 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.385785+0000", 2020-01-16T00:13:01.926 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0064799849999999997 2020-01-16T00:13:01.926 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.926 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.926 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.926 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.392265+0000", 2020-01-16T00:13:01.926 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010603399999999999 2020-01-16T00:13:01.927 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.927 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.927 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:01.927 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.392371+0000", 2020-01-16T00:13:01.927 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003330576 2020-01-16T00:13:01.927 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.927 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.927 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.928 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.395702+0000", 2020-01-16T00:13:01.928 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016566899999999999 2020-01-16T00:13:01.928 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.928 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.928 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:01.928 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.395867+0000", 2020-01-16T00:13:01.928 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014768999999999999 2020-01-16T00:13:01.929 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.929 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.929 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:01.929 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.396015+0000", 2020-01-16T00:13:01.929 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011110386999999999 2020-01-16T00:13:01.929 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.929 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.930 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.930 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.930 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.930 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1972 3.d 3:bd9fc72f:::smithi12313934-5:head [read 0~3327786,checksum] snapc 0=[] ondisk+read+known_if_redirected e32)", 2020-01-16T00:13:01.930 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:57.953277+0000", 2020-01-16T00:13:01.930 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 3.7479597120000001, 2020-01-16T00:13:01.930 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.007738216, 2020-01-16T00:13:01.930 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:01.931 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:01.931 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:01.931 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:01.931 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:01.931 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1972 2020-01-16T00:13:01.931 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.931 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:01.932 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.932 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:01.932 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.953277+0000", 2020-01-16T00:13:01.932 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:01.932 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.932 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.932 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:01.932 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.953277+0000", 2020-01-16T00:13:01.933 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:01.933 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.933 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.933 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:01.933 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.953276+0000", 2020-01-16T00:13:01.933 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.8399999999999999e-06 2020-01-16T00:13:01.933 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.933 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.934 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:01.934 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.953279+0000", 2020-01-16T00:13:01.934 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.8270000000000002e-06 2020-01-16T00:13:01.934 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.934 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.934 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:01.934 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.953282+0000", 2020-01-16T00:13:01.935 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0339999999999999e-06 2020-01-16T00:13:01.935 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.935 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.935 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:01.935 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.953284+0000", 2020-01-16T00:13:01.935 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0068756800000000003 2020-01-16T00:13:01.935 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.935 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.936 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.936 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.960160+0000", 2020-01-16T00:13:01.936 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.9925999999999993e-05 2020-01-16T00:13:01.936 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.936 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.936 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:01.936 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.960240+0000", 2020-01-16T00:13:01.936 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00077526899999999998 2020-01-16T00:13:01.937 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.937 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.937 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:01.937 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.961015+0000", 2020-01-16T00:13:01.937 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.007738216 2020-01-16T00:13:01.937 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.937 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.937 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.938 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.938 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.938 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1973 3.d 3:bd9fc72f:::smithi12313934-5:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e32)", 2020-01-16T00:13:01.938 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:57.953286+0000", 2020-01-16T00:13:01.938 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 3.747950001, 2020-01-16T00:13:01.938 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0080193350000000007, 2020-01-16T00:13:01.938 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:01.939 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:01.939 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:01.939 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:01.939 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:01.939 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1973 2020-01-16T00:13:01.939 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.939 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:01.939 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.940 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:01.940 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.953286+0000", 2020-01-16T00:13:01.940 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:01.940 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.940 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.940 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:01.940 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.953286+0000", 2020-01-16T00:13:01.940 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:01.941 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.941 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.941 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:01.941 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.953286+0000", 2020-01-16T00:13:01.941 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0999999999999998e-06 2020-01-16T00:13:01.941 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.941 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.942 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:01.942 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.953288+0000", 2020-01-16T00:13:01.942 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.5579999999999999e-06 2020-01-16T00:13:01.942 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.942 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.942 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:01.942 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.953291+0000", 2020-01-16T00:13:01.942 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5599999999999999e-06 2020-01-16T00:13:01.943 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.943 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.943 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:01.943 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.953297+0000", 2020-01-16T00:13:01.943 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0077367 2020-01-16T00:13:01.943 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.943 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.943 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.944 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.961034+0000", 2020-01-16T00:13:01.944 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4999000000000005e-05 2020-01-16T00:13:01.944 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.944 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.944 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:01.944 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.961109+0000", 2020-01-16T00:13:01.944 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000196818 2020-01-16T00:13:01.945 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.945 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.945 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:01.945 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.961306+0000", 2020-01-16T00:13:01.945 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0080193350000000007 2020-01-16T00:13:01.946 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.946 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.946 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.946 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.946 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.946 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.4.13:48 2.0 2:01b4bead:::smithi12313934-45 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 1887232~593920] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e32)", 2020-01-16T00:13:01.946 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:58.630331+0000", 2020-01-16T00:13:01.947 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 3.0709051380000001, 2020-01-16T00:13:01.947 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.007946926, 2020-01-16T00:13:01.947 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:01.947 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:01.947 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:01.947 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.947 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:01.947 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:58.630331+0000", 2020-01-16T00:13:01.948 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:01.948 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.948 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.948 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:01.948 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:58.630331+0000", 2020-01-16T00:13:01.948 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:01.948 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.949 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.949 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:01.949 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:58.630331+0000", 2020-01-16T00:13:01.949 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00025828299999999999 2020-01-16T00:13:01.949 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.949 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.950 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:01.950 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:58.630589+0000", 2020-01-16T00:13:01.950 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.774e-05 2020-01-16T00:13:01.950 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.950 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.950 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:01.951 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:58.630607+0000", 2020-01-16T00:13:01.951 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0390000000000001e-06 2020-01-16T00:13:01.951 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.951 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.951 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:01.951 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:58.630613+0000", 2020-01-16T00:13:01.951 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0021615499999999999 2020-01-16T00:13:01.952 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.952 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.952 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:01.952 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:58.632774+0000", 2020-01-16T00:13:01.952 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2879000000000005e-05 2020-01-16T00:13:01.952 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.952 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.952 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:01.953 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:58.632847+0000", 2020-01-16T00:13:01.953 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000147575 2020-01-16T00:13:01.953 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.953 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.953 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:13:01.953 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:58.632995+0000", 2020-01-16T00:13:01.953 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0031611439999999998 2020-01-16T00:13:01.954 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.954 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.954 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:01.954 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:58.636156+0000", 2020-01-16T00:13:01.954 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0013795700000000001 2020-01-16T00:13:01.954 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.954 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.954 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:01.955 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:58.637535+0000", 2020-01-16T00:13:01.955 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00066303499999999997 2020-01-16T00:13:01.955 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.955 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.955 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:01.955 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:58.638198+0000", 2020-01-16T00:13:01.955 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9005000000000003e-05 2020-01-16T00:13:01.956 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.956 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.956 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:01.956 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:58.638247+0000", 2020-01-16T00:13:01.956 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0876000000000001e-05 2020-01-16T00:13:01.956 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:01.956 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:01.956 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:01.957 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:58.638278+0000", 2020-01-16T00:13:01.957 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.007946926 2020-01-16T00:13:01.957 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.957 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.957 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.957 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:01.957 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:01.958 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:01.958 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-01-16T00:13:01.960 INFO:tasks.rados.rados.0.smithi123.stdout:966: finishing write tid 3 to smithi12313934-23 2020-01-16T00:13:01.960 INFO:tasks.rados.rados.0.smithi123.stdout:966: finishing write tid 5 to smithi12313934-23 2020-01-16T00:13:01.960 INFO:tasks.rados.rados.0.smithi123.stdout:966: finishing write tid 6 to smithi12313934-23 2020-01-16T00:13:01.961 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 23 v 102 (ObjNum 227 snap 0 seq_num 227) dirty exists 2020-01-16T00:13:01.961 INFO:tasks.rados.rados.0.smithi123.stdout:966: left oid 23 (ObjNum 227 snap 0 seq_num 227) 2020-01-16T00:13:01.961 INFO:tasks.rados.rados.0.smithi123.stdout:968: finishing write tid 1 to smithi12313934-44 2020-01-16T00:13:01.961 INFO:tasks.rados.rados.0.smithi123.stdout:968: finishing write tid 2 to smithi12313934-44 2020-01-16T00:13:01.961 INFO:tasks.rados.rados.0.smithi123.stdout:969: expect (ObjNum 179 snap 0 seq_num 179) 2020-01-16T00:13:01.962 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:01.959+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:01.982 INFO:tasks.rados.rados.0.smithi123.stdout:970: finishing write tid 1 to smithi12313934-5 2020-01-16T00:13:01.982 INFO:tasks.rados.rados.0.smithi123.stdout:968: finishing write tid 3 to smithi12313934-44 2020-01-16T00:13:01.982 INFO:tasks.rados.rados.0.smithi123.stdout:968: finishing write tid 5 to smithi12313934-44 2020-01-16T00:13:01.982 INFO:tasks.rados.rados.0.smithi123.stdout:968: finishing write tid 6 to smithi12313934-44 2020-01-16T00:13:01.982 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 44 v 137 (ObjNum 228 snap 0 seq_num 228) dirty exists 2020-01-16T00:13:01.982 INFO:tasks.rados.rados.0.smithi123.stdout:968: left oid 44 (ObjNum 228 snap 0 seq_num 228) 2020-01-16T00:13:01.983 INFO:tasks.rados.rados.0.smithi123.stdout:970: finishing write tid 2 to smithi12313934-5 2020-01-16T00:13:01.983 INFO:tasks.rados.rados.0.smithi123.stdout:970: finishing write tid 3 to smithi12313934-5 2020-01-16T00:13:01.983 INFO:tasks.rados.rados.0.smithi123.stdout:970: finishing write tid 5 to smithi12313934-5 2020-01-16T00:13:01.983 INFO:tasks.rados.rados.0.smithi123.stdout:970: finishing write tid 6 to smithi12313934-5 2020-01-16T00:13:01.983 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 5 v 138 (ObjNum 229 snap 0 seq_num 229) dirty exists 2020-01-16T00:13:01.983 INFO:tasks.rados.rados.0.smithi123.stdout:970: left oid 5 (ObjNum 229 snap 0 seq_num 229) 2020-01-16T00:13:01.983 INFO:tasks.rados.rados.0.smithi123.stdout:971: expect (ObjNum 73 snap 0 seq_num 73) 2020-01-16T00:13:02.062 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:02.059+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:02.076 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:02.076 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:13:02.076 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:13:02.077 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:02.086 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-01-16T00:13:02.111 INFO:tasks.rados.rados.0.smithi123.stdout:972: finishing write tid 1 to smithi12313934-13 2020-01-16T00:13:02.111 INFO:tasks.rados.rados.0.smithi123.stdout:972: finishing write tid 2 to smithi12313934-13 2020-01-16T00:13:02.111 INFO:tasks.rados.rados.0.smithi123.stdout:973: expect (ObjNum 200 snap 0 seq_num 200) 2020-01-16T00:13:02.163 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:02.159+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:02.190 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:02.190 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:13:02.190 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:13:02.190 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:13:02.191 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:02.200 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-01-16T00:13:02.264 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:02.263+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:02.345 INFO:tasks.rados.rados.0.smithi123.stdout:974: finishing write tid 1 to smithi12313934-7 2020-01-16T00:13:02.345 INFO:tasks.rados.rados.0.smithi123.stdout:974: finishing write tid 2 to smithi12313934-7 2020-01-16T00:13:02.345 INFO:tasks.rados.rados.0.smithi123.stdout:972: finishing write tid 3 to smithi12313934-13 2020-01-16T00:13:02.345 INFO:tasks.rados.rados.0.smithi123.stdout:972: finishing write tid 5 to smithi12313934-13 2020-01-16T00:13:02.345 INFO:tasks.rados.rados.0.smithi123.stdout:972: finishing write tid 6 to smithi12313934-13 2020-01-16T00:13:02.345 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 51 (ObjNum 230 snap 0 seq_num 230) dirty exists 2020-01-16T00:13:02.346 INFO:tasks.rados.rados.0.smithi123.stdout:972: left oid 13 (ObjNum 230 snap 0 seq_num 230) 2020-01-16T00:13:02.346 INFO:tasks.rados.rados.0.smithi123.stdout:974: finishing write tid 3 to smithi12313934-7 2020-01-16T00:13:02.346 INFO:tasks.rados.rados.0.smithi123.stdout:974: finishing write tid 5 to smithi12313934-7 2020-01-16T00:13:02.346 INFO:tasks.rados.rados.0.smithi123.stdout:974: finishing write tid 6 to smithi12313934-7 2020-01-16T00:13:02.346 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 7 v 132 (ObjNum 231 snap 0 seq_num 231) dirty exists 2020-01-16T00:13:02.346 INFO:tasks.rados.rados.0.smithi123.stdout:974: left oid 7 (ObjNum 231 snap 0 seq_num 231) 2020-01-16T00:13:02.346 INFO:tasks.rados.rados.0.smithi123.stdout:975: finishing write tid 1 to smithi12313934-43 2020-01-16T00:13:02.346 INFO:tasks.rados.rados.0.smithi123.stdout:975: finishing write tid 2 to smithi12313934-43 2020-01-16T00:13:02.347 INFO:tasks.rados.rados.0.smithi123.stdout:975: finishing write tid 3 to smithi12313934-43 2020-01-16T00:13:02.347 INFO:tasks.rados.rados.0.smithi123.stdout:975: finishing write tid 5 to smithi12313934-43 2020-01-16T00:13:02.347 INFO:tasks.rados.rados.0.smithi123.stdout:975: finishing write tid 6 to smithi12313934-43 2020-01-16T00:13:02.347 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 43 v 141 (ObjNum 232 snap 0 seq_num 232) dirty exists 2020-01-16T00:13:02.347 INFO:tasks.rados.rados.0.smithi123.stdout:975: left oid 43 (ObjNum 232 snap 0 seq_num 232) 2020-01-16T00:13:02.347 INFO:tasks.rados.rados.0.smithi123.stdout:964: done (12 left) 2020-01-16T00:13:02.347 INFO:tasks.rados.rados.0.smithi123.stdout:965: done (11 left) 2020-01-16T00:13:02.347 INFO:tasks.rados.rados.0.smithi123.stdout:966: done (10 left) 2020-01-16T00:13:02.348 INFO:tasks.rados.rados.0.smithi123.stdout:967: done (9 left) 2020-01-16T00:13:02.348 INFO:tasks.rados.rados.0.smithi123.stdout:968: done (8 left) 2020-01-16T00:13:02.348 INFO:tasks.rados.rados.0.smithi123.stdout:969: done (7 left) 2020-01-16T00:13:02.348 INFO:tasks.rados.rados.0.smithi123.stdout:970: done (6 left) 2020-01-16T00:13:02.348 INFO:tasks.rados.rados.0.smithi123.stdout:971: done (5 left) 2020-01-16T00:13:02.348 INFO:tasks.rados.rados.0.smithi123.stdout:972: done (4 left) 2020-01-16T00:13:02.348 INFO:tasks.rados.rados.0.smithi123.stdout:973: done (3 left) 2020-01-16T00:13:02.348 INFO:tasks.rados.rados.0.smithi123.stdout:974: done (2 left) 2020-01-16T00:13:02.349 INFO:tasks.rados.rados.0.smithi123.stdout:975: done (1 left) 2020-01-16T00:13:02.349 INFO:tasks.rados.rados.0.smithi123.stdout:976: done (0 left) 2020-01-16T00:13:02.349 INFO:tasks.rados.rados.0.smithi123.stdout:977: read oid 11 snap -1 2020-01-16T00:13:02.349 INFO:tasks.rados.rados.0.smithi123.stdout:977: expect (ObjNum 191 snap 0 seq_num 191) 2020-01-16T00:13:02.349 INFO:tasks.rados.rados.0.smithi123.stdout:978: read oid 33 snap -1 2020-01-16T00:13:02.349 INFO:tasks.rados.rados.0.smithi123.stdout:978: expect (ObjNum 212 snap 0 seq_num 212) 2020-01-16T00:13:02.349 INFO:tasks.rados.rados.0.smithi123.stdout:979: write oid 10 current snap is 0 2020-01-16T00:13:02.350 INFO:tasks.rados.rados.0.smithi123.stdout:979: seq_num 234 ranges {732838=698856,2026904=465743,3210039=571651} 2020-01-16T00:13:02.357 INFO:tasks.rados.rados.0.smithi123.stdout:979: writing smithi12313934-10 from 732838 to 1431694 tid 1 2020-01-16T00:13:02.363 INFO:tasks.rados.rados.0.smithi123.stdout:979: writing smithi12313934-10 from 2026904 to 2492647 tid 2 2020-01-16T00:13:02.365 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:02.363+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:02.369 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:02.369 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:13:02.369 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:13:02.369 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:13:02.369 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.370 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:890 3.a 3:5c245412:::smithi12313934-19:head [write 759683~639358,stat] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:02.370 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.112181+0000", 2020-01-16T00:13:02.370 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 25.251319943999999, 2020-01-16T00:13:02.370 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0068210429999999997, 2020-01-16T00:13:02.370 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:02.370 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:02.370 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:02.370 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:02.371 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:02.371 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 890 2020-01-16T00:13:02.371 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.371 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:02.371 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.371 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:02.371 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.112181+0000", 2020-01-16T00:13:02.371 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:02.372 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.372 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.372 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:02.372 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.112181+0000", 2020-01-16T00:13:02.372 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999957 2020-01-16T00:13:02.372 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.373 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.373 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:02.373 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.112177+0000", 2020-01-16T00:13:02.373 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00060751999999999996 2020-01-16T00:13:02.373 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.373 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.373 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:02.374 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.112784+0000", 2020-01-16T00:13:02.374 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3646999999999998e-05 2020-01-16T00:13:02.374 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.374 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.374 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:02.374 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.112818+0000", 2020-01-16T00:13:02.375 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.432e-05 2020-01-16T00:13:02.375 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.375 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.375 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:02.375 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.112832+0000", 2020-01-16T00:13:02.376 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000118984 2020-01-16T00:13:02.376 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.376 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.376 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:02.377 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.112951+0000", 2020-01-16T00:13:02.377 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00023189400000000001 2020-01-16T00:13:02.377 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.377 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.378 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:02.378 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.113183+0000", 2020-01-16T00:13:02.378 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00035231999999999999 2020-01-16T00:13:02.378 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.379 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.379 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:02.379 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.113535+0000", 2020-01-16T00:13:02.380 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0035763539999999999 2020-01-16T00:13:02.380 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.380 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.380 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:02.381 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.117112+0000", 2020-01-16T00:13:02.381 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4860000000000002e-05 2020-01-16T00:13:02.381 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.382 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.382 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:02.382 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.117147+0000", 2020-01-16T00:13:02.382 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0017898110000000001 2020-01-16T00:13:02.382 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.383 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.383 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:02.383 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.118936+0000", 2020-01-16T00:13:02.383 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6343000000000001e-05 2020-01-16T00:13:02.384 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.384 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.384 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:02.384 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.118973+0000", 2020-01-16T00:13:02.384 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9257000000000001e-05 2020-01-16T00:13:02.385 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.385 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.385 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:02.385 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.119002+0000", 2020-01-16T00:13:02.386 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0068210429999999997 2020-01-16T00:13:02.386 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.386 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:02.386 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.386 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.387 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.387 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(osd.1.0:162 3.0 e24/20 3:00c146d5:::smithi12313934-18:head v 24'99, mlcod=24'97)", 2020-01-16T00:13:02.387 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.038304+0000", 2020-01-16T00:13:02.387 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 24.325196514000002, 2020-01-16T00:13:02.388 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.006772386, 2020-01-16T00:13:02.388 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:02.388 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:02.388 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:02.388 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.389 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:02.389 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.038304+0000", 2020-01-16T00:13:02.389 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:02.389 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.390 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.390 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:02.390 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.038304+0000", 2020-01-16T00:13:02.390 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:13:02.390 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.391 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.391 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:02.391 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.038301+0000", 2020-01-16T00:13:02.391 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012763920000000001 2020-01-16T00:13:02.392 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.392 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.392 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:02.392 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.039577+0000", 2020-01-16T00:13:02.392 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1426e-05 2020-01-16T00:13:02.393 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.393 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.393 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:02.393 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.039608+0000", 2020-01-16T00:13:02.394 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2017e-05 2020-01-16T00:13:02.394 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.394 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.394 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:02.395 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.039620+0000", 2020-01-16T00:13:02.395 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000111969 2020-01-16T00:13:02.395 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.396 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.397 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:02.397 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.039732+0000", 2020-01-16T00:13:02.398 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1180000000000002e-05 2020-01-16T00:13:02.398 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.398 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.399 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:02.399 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.039774+0000", 2020-01-16T00:13:02.399 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.005179286 2020-01-16T00:13:02.399 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.400 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.400 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:02.400 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.044953+0000", 2020-01-16T00:13:02.400 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012396600000000001 2020-01-16T00:13:02.401 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.401 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.401 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:02.401 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.045077+0000", 2020-01-16T00:13:02.402 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.006772386 2020-01-16T00:13:02.402 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.402 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:02.402 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.403 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.403 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.403 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:965 3.a 3:54f2c117:::smithi12313934-41:head [write 527753~708201] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:02.403 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.146756+0000", 2020-01-16T00:13:02.404 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 24.216745411000002, 2020-01-16T00:13:02.404 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0071708459999999998, 2020-01-16T00:13:02.404 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:02.404 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:02.405 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:02.405 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:02.405 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:02.405 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 965 2020-01-16T00:13:02.406 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.406 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:02.406 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.406 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:02.406 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.146756+0000", 2020-01-16T00:13:02.407 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:02.407 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.407 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.407 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:02.408 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.146756+0000", 2020-01-16T00:13:02.408 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999948 2020-01-16T00:13:02.408 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.408 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.408 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:02.409 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.146750+0000", 2020-01-16T00:13:02.409 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00094503399999999998 2020-01-16T00:13:02.409 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.409 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.410 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:02.410 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.147695+0000", 2020-01-16T00:13:02.410 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7846999999999999e-05 2020-01-16T00:13:02.410 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.411 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.411 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:02.411 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.147733+0000", 2020-01-16T00:13:02.413 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4392999999999999e-05 2020-01-16T00:13:02.413 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.413 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.413 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:02.414 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.147747+0000", 2020-01-16T00:13:02.414 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000106737 2020-01-16T00:13:02.414 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.414 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.415 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:02.415 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.147854+0000", 2020-01-16T00:13:02.415 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00026514699999999997 2020-01-16T00:13:02.416 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.416 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.416 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:02.416 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.148119+0000", 2020-01-16T00:13:02.416 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030262600000000001 2020-01-16T00:13:02.417 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.417 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.417 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:02.417 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.148422+0000", 2020-01-16T00:13:02.418 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.004368416 2020-01-16T00:13:02.418 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.418 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.419 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:02.419 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.152790+0000", 2020-01-16T00:13:02.419 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000368437 2020-01-16T00:13:02.419 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.419 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.420 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:02.420 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.153159+0000", 2020-01-16T00:13:02.420 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00058380900000000002 2020-01-16T00:13:02.420 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.421 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.421 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:02.421 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.153743+0000", 2020-01-16T00:13:02.421 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8196999999999995e-05 2020-01-16T00:13:02.422 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.422 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.422 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:02.422 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.153841+0000", 2020-01-16T00:13:02.422 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5625999999999996e-05 2020-01-16T00:13:02.423 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.423 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.423 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:02.423 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.153926+0000", 2020-01-16T00:13:02.424 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0071708459999999998 2020-01-16T00:13:02.424 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.424 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:02.424 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.425 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.425 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.425 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:966 3.a 3:54f2c117:::smithi12313934-41:head [write 1856548~737690] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:02.425 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.154844+0000", 2020-01-16T00:13:02.425 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 24.208656607999998, 2020-01-16T00:13:02.426 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0077324510000000004, 2020-01-16T00:13:02.426 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:02.426 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:02.426 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:02.427 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:02.427 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:02.427 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 966 2020-01-16T00:13:02.427 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.427 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:02.428 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.428 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:02.428 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.154844+0000", 2020-01-16T00:13:02.428 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:02.429 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.429 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.429 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:02.429 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.154844+0000", 2020-01-16T00:13:02.429 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:02.430 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.430 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.430 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:02.430 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.154842+0000", 2020-01-16T00:13:02.431 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00085401100000000001 2020-01-16T00:13:02.431 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.431 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.431 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:02.432 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.155696+0000", 2020-01-16T00:13:02.432 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9801000000000001e-05 2020-01-16T00:13:02.432 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.432 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.433 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:02.433 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.155726+0000", 2020-01-16T00:13:02.433 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.171e-05 2020-01-16T00:13:02.433 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.433 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.434 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:02.434 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.155738+0000", 2020-01-16T00:13:02.434 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0573000000000003e-05 2020-01-16T00:13:02.434 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.435 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.435 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:02.435 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.155818+0000", 2020-01-16T00:13:02.435 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024165699999999999 2020-01-16T00:13:02.435 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.436 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.436 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:02.436 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.156060+0000", 2020-01-16T00:13:02.436 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033522700000000001 2020-01-16T00:13:02.437 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.437 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.437 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:02.437 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.156395+0000", 2020-01-16T00:13:02.437 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0059325039999999999 2020-01-16T00:13:02.438 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.438 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.438 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:02.438 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.162328+0000", 2020-01-16T00:13:02.439 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5182e-05 2020-01-16T00:13:02.439 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.439 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.439 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:02.439 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.162393+0000", 2020-01-16T00:13:02.440 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8604e-05 2020-01-16T00:13:02.440 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.440 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.440 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:02.440 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.162411+0000", 2020-01-16T00:13:02.440 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2386000000000006e-05 2020-01-16T00:13:02.441 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.441 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.441 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:02.441 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.162504+0000", 2020-01-16T00:13:02.441 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2970000000000001e-05 2020-01-16T00:13:02.441 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.441 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.442 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:02.442 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.162577+0000", 2020-01-16T00:13:02.442 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0077324510000000004 2020-01-16T00:13:02.442 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.442 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:02.442 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.442 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.442 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.443 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.4488.0:1237 3.3 e26/20 3:c204aa4f:::smithi12313934-45:head v 26'56, mlcod=26'54)", 2020-01-16T00:13:02.443 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.587132+0000", 2020-01-16T00:13:02.443 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 19.776369387999999, 2020-01-16T00:13:02.443 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0076804919999999997, 2020-01-16T00:13:02.443 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:02.443 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:02.443 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:02.443 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.444 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:02.444 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.587132+0000", 2020-01-16T00:13:02.444 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:02.444 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.444 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.444 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:02.444 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.587132+0000", 2020-01-16T00:13:02.444 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999957 2020-01-16T00:13:02.445 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.445 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.445 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:02.445 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.587127+0000", 2020-01-16T00:13:02.445 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00047205800000000003 2020-01-16T00:13:02.445 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.445 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.446 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:02.446 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.587599+0000", 2020-01-16T00:13:02.446 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6477000000000003e-05 2020-01-16T00:13:02.446 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.446 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.446 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:02.446 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.587636+0000", 2020-01-16T00:13:02.446 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7294000000000001e-05 2020-01-16T00:13:02.447 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.447 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.447 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:02.447 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.587653+0000", 2020-01-16T00:13:02.447 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000183778 2020-01-16T00:13:02.447 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.447 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.447 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:02.448 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.587837+0000", 2020-01-16T00:13:02.448 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0628000000000001e-05 2020-01-16T00:13:02.448 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.448 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.448 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:02.449 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.587917+0000", 2020-01-16T00:13:02.449 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.006854527 2020-01-16T00:13:02.449 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.449 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.449 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:02.450 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.594772+0000", 2020-01-16T00:13:02.450 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0144000000000001e-05 2020-01-16T00:13:02.450 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.450 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.450 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:02.450 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.594812+0000", 2020-01-16T00:13:02.450 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0076804919999999997 2020-01-16T00:13:02.451 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.451 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:02.451 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.451 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.451 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.451 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1241 3.a 3:5c245412:::smithi12313934-19:head [read 0~3815156,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:13:02.451 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.589903+0000", 2020-01-16T00:13:02.451 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 19.773597989999999, 2020-01-16T00:13:02.452 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0096065560000000005, 2020-01-16T00:13:02.452 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:02.452 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:02.452 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:02.452 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:02.452 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:02.452 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1241 2020-01-16T00:13:02.453 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.453 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:02.453 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.453 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:02.453 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.589903+0000", 2020-01-16T00:13:02.453 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:02.453 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.454 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.454 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:02.454 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.589903+0000", 2020-01-16T00:13:02.454 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999952 2020-01-16T00:13:02.454 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.454 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.454 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:02.454 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.589898+0000", 2020-01-16T00:13:02.455 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4389e-05 2020-01-16T00:13:02.455 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.455 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.455 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:02.455 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.589922+0000", 2020-01-16T00:13:02.456 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4159999999999998e-05 2020-01-16T00:13:02.456 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.456 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.456 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:02.456 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.589957+0000", 2020-01-16T00:13:02.456 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7368999999999999e-05 2020-01-16T00:13:02.457 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.457 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.457 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:02.457 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.589974+0000", 2020-01-16T00:13:02.457 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000130407 2020-01-16T00:13:02.457 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.458 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.458 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:02.458 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590104+0000", 2020-01-16T00:13:02.458 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00040217699999999998 2020-01-16T00:13:02.458 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.458 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.458 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:02.459 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590506+0000", 2020-01-16T00:13:02.459 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0090030230000000006 2020-01-16T00:13:02.459 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.459 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.459 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:02.459 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599510+0000", 2020-01-16T00:13:02.460 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0096065560000000005 2020-01-16T00:13:02.460 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.460 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:02.460 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.460 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.460 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.460 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1242 3.a 3:5c245412:::smithi12313934-19:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:13:02.460 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.590003+0000", 2020-01-16T00:13:02.461 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 19.773497481, 2020-01-16T00:13:02.461 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010247029, 2020-01-16T00:13:02.461 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:02.461 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:02.461 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:02.461 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:02.461 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:02.462 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1242 2020-01-16T00:13:02.462 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.462 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:02.462 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.462 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:02.462 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590003+0000", 2020-01-16T00:13:02.462 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:02.462 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.463 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.463 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:02.463 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590003+0000", 2020-01-16T00:13:02.463 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:02.463 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.463 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.463 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:02.463 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590002+0000", 2020-01-16T00:13:02.464 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0108e-05 2020-01-16T00:13:02.464 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.464 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.464 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:02.464 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590012+0000", 2020-01-16T00:13:02.464 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5318000000000001e-05 2020-01-16T00:13:02.464 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.465 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.465 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:02.465 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590027+0000", 2020-01-16T00:13:02.465 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0720000000000001e-05 2020-01-16T00:13:02.465 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.465 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.465 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:02.466 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590038+0000", 2020-01-16T00:13:02.466 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0095349949999999992 2020-01-16T00:13:02.466 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.466 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.466 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:02.466 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599573+0000", 2020-01-16T00:13:02.466 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017107600000000001 2020-01-16T00:13:02.467 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.467 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.467 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:02.467 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599744+0000", 2020-01-16T00:13:02.467 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00050659499999999998 2020-01-16T00:13:02.467 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.467 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.467 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:02.468 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.600251+0000", 2020-01-16T00:13:02.468 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010247029 2020-01-16T00:13:02.468 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.468 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:02.468 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.468 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.468 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.469 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1243 3.a 3:5c245412:::smithi12313934-19:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:13:02.469 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.590095+0000", 2020-01-16T00:13:02.469 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 19.773406072, 2020-01-16T00:13:02.469 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010591455, 2020-01-16T00:13:02.469 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:02.469 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:02.469 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:02.470 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:02.470 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:02.470 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1243 2020-01-16T00:13:02.470 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.470 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:02.470 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.470 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:02.470 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590095+0000", 2020-01-16T00:13:02.471 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:02.471 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.471 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.471 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:02.471 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590095+0000", 2020-01-16T00:13:02.471 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:02.471 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.471 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.472 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:02.472 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590093+0000", 2020-01-16T00:13:02.472 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5619999999999993e-06 2020-01-16T00:13:02.472 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.472 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.472 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:02.472 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590102+0000", 2020-01-16T00:13:02.473 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.259e-05 2020-01-16T00:13:02.473 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.473 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.473 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:02.473 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590115+0000", 2020-01-16T00:13:02.473 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8730000000000001e-06 2020-01-16T00:13:02.474 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.474 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.474 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:02.474 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590122+0000", 2020-01-16T00:13:02.474 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010163165 2020-01-16T00:13:02.474 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.474 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.474 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:02.475 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.600285+0000", 2020-01-16T00:13:02.475 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0128000000000002e-05 2020-01-16T00:13:02.475 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.475 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.475 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:02.475 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.600375+0000", 2020-01-16T00:13:02.475 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000311312 2020-01-16T00:13:02.475 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.476 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.476 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:02.476 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.600686+0000", 2020-01-16T00:13:02.476 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010591455 2020-01-16T00:13:02.476 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.476 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:02.476 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.477 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.477 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.477 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1654 3.a 3:5069665e:::smithi12313934-2:head [write 1614880~581041] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:13:02.477 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.714839+0000", 2020-01-16T00:13:02.477 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 9.6486620839999997, 2020-01-16T00:13:02.477 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013588093000000001, 2020-01-16T00:13:02.477 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:02.477 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:02.478 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:02.478 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:02.478 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:02.478 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1654 2020-01-16T00:13:02.478 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.478 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:02.478 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.479 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:02.479 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.714839+0000", 2020-01-16T00:13:02.479 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:02.479 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.479 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.480 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:02.480 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.714839+0000", 2020-01-16T00:13:02.480 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:02.480 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.480 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.480 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:02.481 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.714836+0000", 2020-01-16T00:13:02.481 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00054657999999999996 2020-01-16T00:13:02.481 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.481 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.481 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:02.481 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.715382+0000", 2020-01-16T00:13:02.482 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7743000000000001e-05 2020-01-16T00:13:02.482 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.482 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.482 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:02.482 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.715420+0000", 2020-01-16T00:13:02.482 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.221e-05 2020-01-16T00:13:02.482 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.483 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.483 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:02.483 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.715432+0000", 2020-01-16T00:13:02.483 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2487000000000001e-05 2020-01-16T00:13:02.483 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.483 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.483 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:02.483 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.715485+0000", 2020-01-16T00:13:02.484 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000726284 2020-01-16T00:13:02.484 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.484 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.484 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:02.484 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.716211+0000", 2020-01-16T00:13:02.484 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0067930899999999999 2020-01-16T00:13:02.484 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.485 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.485 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:02.485 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.723004+0000", 2020-01-16T00:13:02.485 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012579400000000001 2020-01-16T00:13:02.485 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.485 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.485 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:02.486 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.723130+0000", 2020-01-16T00:13:02.486 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013465800000000001 2020-01-16T00:13:02.486 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.486 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.486 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:02.486 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.723265+0000", 2020-01-16T00:13:02.486 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023307250000000001 2020-01-16T00:13:02.486 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.487 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.487 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:02.487 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.725595+0000", 2020-01-16T00:13:02.487 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012582139999999999 2020-01-16T00:13:02.487 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.487 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.487 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:02.488 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.726853+0000", 2020-01-16T00:13:02.488 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014479110000000001 2020-01-16T00:13:02.488 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.488 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.488 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:02.488 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.728301+0000", 2020-01-16T00:13:02.488 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6866000000000007e-05 2020-01-16T00:13:02.488 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.489 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.489 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:02.489 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.728368+0000", 2020-01-16T00:13:02.489 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8740999999999999e-05 2020-01-16T00:13:02.489 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.489 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.489 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:02.489 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.728427+0000", 2020-01-16T00:13:02.490 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013588093000000001 2020-01-16T00:13:02.490 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.490 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:02.490 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.490 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.490 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.490 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1655 3.a 3:5069665e:::smithi12313934-2:head [write 2785791~71399] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:13:02.490 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.718297+0000", 2020-01-16T00:13:02.491 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 9.6452042309999992, 2020-01-16T00:13:02.491 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013817354, 2020-01-16T00:13:02.491 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:02.491 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:02.491 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:02.491 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:02.491 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:02.492 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1655 2020-01-16T00:13:02.492 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.492 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:02.492 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.492 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:02.492 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718297+0000", 2020-01-16T00:13:02.492 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:02.492 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.493 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.493 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:02.493 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718297+0000", 2020-01-16T00:13:02.493 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:02.493 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.493 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.493 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:02.493 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718294+0000", 2020-01-16T00:13:02.494 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000103558 2020-01-16T00:13:02.494 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.494 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.494 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:02.494 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718397+0000", 2020-01-16T00:13:02.494 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9287e-05 2020-01-16T00:13:02.494 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.494 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.495 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:02.495 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718417+0000", 2020-01-16T00:13:02.495 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6680000000000001e-06 2020-01-16T00:13:02.495 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.495 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.495 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:02.495 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718424+0000", 2020-01-16T00:13:02.496 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2701999999999994e-05 2020-01-16T00:13:02.496 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.496 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.496 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:02.496 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718517+0000", 2020-01-16T00:13:02.496 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3893000000000006e-05 2020-01-16T00:13:02.497 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.497 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.497 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:02.497 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718581+0000", 2020-01-16T00:13:02.497 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0059630079999999997 2020-01-16T00:13:02.497 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.498 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.498 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:02.498 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.724544+0000", 2020-01-16T00:13:02.498 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000309108 2020-01-16T00:13:02.498 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.498 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.499 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:02.499 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.724853+0000", 2020-01-16T00:13:02.499 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.005469794 2020-01-16T00:13:02.499 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.499 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.499 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:02.499 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.730323+0000", 2020-01-16T00:13:02.499 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018506100000000001 2020-01-16T00:13:02.500 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.500 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.500 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:02.500 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.730508+0000", 2020-01-16T00:13:02.500 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016729099999999999 2020-01-16T00:13:02.500 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.501 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.501 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:02.501 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.730675+0000", 2020-01-16T00:13:02.501 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00094193300000000005 2020-01-16T00:13:02.501 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.501 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.501 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:02.502 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731617+0000", 2020-01-16T00:13:02.502 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000214613 2020-01-16T00:13:02.502 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.502 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.502 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:02.502 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731832+0000", 2020-01-16T00:13:02.502 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018326400000000001 2020-01-16T00:13:02.503 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.503 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.503 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:02.503 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732015+0000", 2020-01-16T00:13:02.503 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0392e-05 2020-01-16T00:13:02.503 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.503 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.504 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:02.504 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732075+0000", 2020-01-16T00:13:02.504 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8677999999999998e-05 2020-01-16T00:13:02.504 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.504 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.504 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:02.504 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732114+0000", 2020-01-16T00:13:02.504 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013817354 2020-01-16T00:13:02.505 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.505 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:02.505 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.505 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.505 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.505 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1656 3.a 3:5069665e:::smithi12313934-2:head [setxattr _header (55) in=7b,truncate 2857190] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:13:02.506 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.718453+0000", 2020-01-16T00:13:02.506 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 9.6450475499999992, 2020-01-16T00:13:02.506 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014083462, 2020-01-16T00:13:02.506 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:02.506 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:02.506 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:02.506 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:02.507 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:02.507 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1656 2020-01-16T00:13:02.507 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.507 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:02.507 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.507 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:02.507 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718453+0000", 2020-01-16T00:13:02.508 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:02.508 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.508 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.508 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:02.508 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718453+0000", 2020-01-16T00:13:02.508 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999828 2020-01-16T00:13:02.508 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.508 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.509 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:02.509 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718436+0000", 2020-01-16T00:13:02.509 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4051999999999999e-05 2020-01-16T00:13:02.509 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.509 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.509 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:02.509 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718460+0000", 2020-01-16T00:13:02.510 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6380000000000004e-06 2020-01-16T00:13:02.510 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.510 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.510 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:02.510 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718468+0000", 2020-01-16T00:13:02.510 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6750000000000001e-06 2020-01-16T00:13:02.510 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.510 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.511 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:02.511 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718472+0000", 2020-01-16T00:13:02.511 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016218499999999999 2020-01-16T00:13:02.511 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.511 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.511 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:02.511 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718635+0000", 2020-01-16T00:13:02.511 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2765e-05 2020-01-16T00:13:02.512 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.512 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.512 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:02.512 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718697+0000", 2020-01-16T00:13:02.512 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0062239249999999999 2020-01-16T00:13:02.512 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.512 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.513 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:02.513 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.724921+0000", 2020-01-16T00:13:02.513 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1527999999999999e-05 2020-01-16T00:13:02.513 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.513 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.513 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:02.513 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.724973+0000", 2020-01-16T00:13:02.513 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0060569719999999999 2020-01-16T00:13:02.514 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.514 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.514 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:02.514 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731030+0000", 2020-01-16T00:13:02.514 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5859999999999996e-05 2020-01-16T00:13:02.514 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.514 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.514 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:02.515 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731096+0000", 2020-01-16T00:13:02.515 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000112018 2020-01-16T00:13:02.515 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.515 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.515 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:02.515 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731208+0000", 2020-01-16T00:13:02.515 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000945889 2020-01-16T00:13:02.516 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.516 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.516 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:02.516 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732154+0000", 2020-01-16T00:13:02.516 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0834000000000003e-05 2020-01-16T00:13:02.516 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.516 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.517 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:02.517 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732184+0000", 2020-01-16T00:13:02.517 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028587600000000001 2020-01-16T00:13:02.517 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.517 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.517 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:02.517 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732470+0000", 2020-01-16T00:13:02.517 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1953999999999999e-05 2020-01-16T00:13:02.518 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.518 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.518 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:02.518 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732502+0000", 2020-01-16T00:13:02.518 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4644999999999998e-05 2020-01-16T00:13:02.518 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.518 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.519 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:02.519 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732537+0000", 2020-01-16T00:13:02.519 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014083462 2020-01-16T00:13:02.519 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.519 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:02.519 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.519 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.519 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.520 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1657 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-01-16T00:13:02.520 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.718495+0000", 2020-01-16T00:13:02.520 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 9.6450058849999998, 2020-01-16T00:13:02.520 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014299630000000001, 2020-01-16T00:13:02.520 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:02.520 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:02.520 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:02.520 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:02.521 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:02.521 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1657 2020-01-16T00:13:02.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.521 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:02.521 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.521 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:02.521 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718495+0000", 2020-01-16T00:13:02.521 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:02.522 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.522 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.522 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:02.522 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718495+0000", 2020-01-16T00:13:02.522 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:02.522 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.522 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.522 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:02.523 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718494+0000", 2020-01-16T00:13:02.523 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8860000000000003e-06 2020-01-16T00:13:02.523 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.523 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.523 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:02.523 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718499+0000", 2020-01-16T00:13:02.523 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9786000000000002e-05 2020-01-16T00:13:02.524 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.524 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.524 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:02.524 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718539+0000", 2020-01-16T00:13:02.524 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1869999999999999e-05 2020-01-16T00:13:02.524 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.524 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.524 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:02.525 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718561+0000", 2020-01-16T00:13:02.525 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000172206 2020-01-16T00:13:02.525 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.525 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.525 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:02.525 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718733+0000", 2020-01-16T00:13:02.525 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0662999999999998e-05 2020-01-16T00:13:02.526 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.526 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.526 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:02.526 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718793+0000", 2020-01-16T00:13:02.526 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.006217958 2020-01-16T00:13:02.526 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.526 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.526 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:02.527 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.725011+0000", 2020-01-16T00:13:02.527 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1625999999999996e-05 2020-01-16T00:13:02.527 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.527 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.527 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:02.527 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.725073+0000", 2020-01-16T00:13:02.527 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0063923119999999998 2020-01-16T00:13:02.528 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.528 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.528 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:02.528 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731465+0000", 2020-01-16T00:13:02.528 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2534000000000005e-05 2020-01-16T00:13:02.528 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.528 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.528 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:02.529 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731558+0000", 2020-01-16T00:13:02.529 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010238809999999999 2020-01-16T00:13:02.529 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.529 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.529 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:02.529 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732582+0000", 2020-01-16T00:13:02.529 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000100918 2020-01-16T00:13:02.529 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.530 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.530 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:02.530 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732683+0000", 2020-01-16T00:13:02.530 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000112064 2020-01-16T00:13:02.530 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.530 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.530 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:02.531 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732795+0000", 2020-01-16T00:13:02.531 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014299630000000001 2020-01-16T00:13:02.531 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.531 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:02.531 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.531 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.531 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.532 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.4488.0:1708 3.c e30/20 3:378f3722:::smithi12313934-50:head v 30'89, mlcod=29'87)", 2020-01-16T00:13:02.532 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.808249+0000", 2020-01-16T00:13:02.532 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 9.5552521380000002, 2020-01-16T00:13:02.532 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0067795160000000002, 2020-01-16T00:13:02.532 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:02.532 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:02.532 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:02.533 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.533 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:02.533 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.808249+0000", 2020-01-16T00:13:02.533 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:02.533 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.533 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.533 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:02.534 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.808249+0000", 2020-01-16T00:13:02.534 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:02.534 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.534 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.534 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:02.534 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.808246+0000", 2020-01-16T00:13:02.534 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00046200000000000001 2020-01-16T00:13:02.535 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.535 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.535 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:02.535 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.808708+0000", 2020-01-16T00:13:02.535 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.702e-05 2020-01-16T00:13:02.535 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.535 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.535 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:02.536 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.808725+0000", 2020-01-16T00:13:02.536 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4920000000000001e-06 2020-01-16T00:13:02.536 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.536 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.536 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:02.536 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.808731+0000", 2020-01-16T00:13:02.536 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4094000000000001e-05 2020-01-16T00:13:02.536 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.537 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.537 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:02.537 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.808785+0000", 2020-01-16T00:13:02.537 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.7359999999999999e-05 2020-01-16T00:13:02.537 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.537 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.537 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:02.538 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.808812+0000", 2020-01-16T00:13:02.538 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0061494260000000004 2020-01-16T00:13:02.538 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.538 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.538 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:02.538 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.814962+0000", 2020-01-16T00:13:02.538 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6660000000000002e-05 2020-01-16T00:13:02.538 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.539 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.539 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:02.539 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.815028+0000", 2020-01-16T00:13:02.539 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0067795160000000002 2020-01-16T00:13:02.539 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.539 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:02.539 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.539 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.540 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.540 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1834 3.a 3:54f2c117:::smithi12313934-41:head [write 443751~631882] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:13:02.540 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:56.278017+0000", 2020-01-16T00:13:02.540 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 6.0854838940000002, 2020-01-16T00:13:02.540 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0076931760000000004, 2020-01-16T00:13:02.540 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:02.540 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:02.540 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:02.541 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:02.541 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:02.541 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1834 2020-01-16T00:13:02.541 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.541 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:02.541 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.541 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:02.542 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.278017+0000", 2020-01-16T00:13:02.542 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:02.542 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.542 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.542 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:02.542 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.278017+0000", 2020-01-16T00:13:02.542 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:13:02.542 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.543 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.543 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:02.543 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.278013+0000", 2020-01-16T00:13:02.543 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010862669999999999 2020-01-16T00:13:02.543 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.543 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.543 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:02.543 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.279100+0000", 2020-01-16T00:13:02.544 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8034000000000002e-05 2020-01-16T00:13:02.544 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.544 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.544 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:02.544 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.279138+0000", 2020-01-16T00:13:02.544 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7955999999999999e-05 2020-01-16T00:13:02.544 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.545 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.545 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:02.545 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.279156+0000", 2020-01-16T00:13:02.545 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000141691 2020-01-16T00:13:02.545 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.545 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.545 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:02.545 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.279297+0000", 2020-01-16T00:13:02.546 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00025750000000000002 2020-01-16T00:13:02.546 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.546 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.546 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:02.546 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.279555+0000", 2020-01-16T00:13:02.546 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00031473999999999999 2020-01-16T00:13:02.546 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.547 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.547 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:02.547 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.279869+0000", 2020-01-16T00:13:02.547 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003741192 2020-01-16T00:13:02.547 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.547 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.547 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:02.547 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.283611+0000", 2020-01-16T00:13:02.548 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00075302099999999999 2020-01-16T00:13:02.548 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.548 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.548 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:02.548 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.284364+0000", 2020-01-16T00:13:02.548 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011594509999999999 2020-01-16T00:13:02.548 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.548 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.549 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:02.549 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.285523+0000", 2020-01-16T00:13:02.549 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000114127 2020-01-16T00:13:02.549 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.549 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.549 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:02.549 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.285637+0000", 2020-01-16T00:13:02.549 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2972999999999994e-05 2020-01-16T00:13:02.550 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.550 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.550 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:02.550 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.285710+0000", 2020-01-16T00:13:02.550 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0076931760000000004 2020-01-16T00:13:02.550 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.550 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:02.551 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.551 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.551 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.551 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1835 3.a 3:54f2c117:::smithi12313934-41:head [write 1810907~765058] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:13:02.551 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:56.286248+0000", 2020-01-16T00:13:02.551 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 6.0772529750000004, 2020-01-16T00:13:02.551 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0067134459999999996, 2020-01-16T00:13:02.551 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:02.552 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:02.552 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:02.552 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:02.552 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:02.552 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1835 2020-01-16T00:13:02.552 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.552 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:02.552 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.553 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:02.553 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.286248+0000", 2020-01-16T00:13:02.553 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:02.553 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.553 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.553 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:02.553 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.286248+0000", 2020-01-16T00:13:02.554 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999886 2020-01-16T00:13:02.554 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.554 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.554 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:02.554 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.286237+0000", 2020-01-16T00:13:02.554 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00073283500000000004 2020-01-16T00:13:02.554 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.554 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.555 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:02.555 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.286969+0000", 2020-01-16T00:13:02.555 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.5443e-05 2020-01-16T00:13:02.555 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.555 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.555 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:02.556 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.286995+0000", 2020-01-16T00:13:02.556 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9250000000000007e-06 2020-01-16T00:13:02.556 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.556 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.556 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:02.556 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.287005+0000", 2020-01-16T00:13:02.557 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7592999999999997e-05 2020-01-16T00:13:02.557 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.557 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.557 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:02.557 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.287042+0000", 2020-01-16T00:13:02.557 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8280000000000001e-05 2020-01-16T00:13:02.557 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.558 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.558 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:02.558 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.287141+0000", 2020-01-16T00:13:02.558 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000110246 2020-01-16T00:13:02.558 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.558 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.559 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:02.559 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.287251+0000", 2020-01-16T00:13:02.559 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0026245449999999998 2020-01-16T00:13:02.559 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.559 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.559 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:02.559 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.289875+0000", 2020-01-16T00:13:02.560 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2745e-05 2020-01-16T00:13:02.560 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.560 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.560 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:02.560 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.289898+0000", 2020-01-16T00:13:02.560 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002996154 2020-01-16T00:13:02.560 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.560 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.561 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:02.561 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.292894+0000", 2020-01-16T00:13:02.561 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5630999999999998e-05 2020-01-16T00:13:02.561 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.561 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.561 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:02.561 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.292930+0000", 2020-01-16T00:13:02.562 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1398e-05 2020-01-16T00:13:02.562 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.562 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.562 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:02.562 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.292961+0000", 2020-01-16T00:13:02.562 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0067134459999999996 2020-01-16T00:13:02.562 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.562 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:02.563 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.563 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.563 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.563 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1917 3.a 3:5069665e:::smithi12313934-2:head [write 2113302~674388] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:13:02.563 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:57.330712+0000", 2020-01-16T00:13:02.563 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.0327893699999997, 2020-01-16T00:13:02.563 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012781043000000001, 2020-01-16T00:13:02.563 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:02.564 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:02.564 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:02.564 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:02.564 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:02.564 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1917 2020-01-16T00:13:02.564 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.564 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:02.564 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.565 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:02.565 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.330712+0000", 2020-01-16T00:13:02.565 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:02.565 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.565 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.565 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:02.565 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.330712+0000", 2020-01-16T00:13:02.566 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:02.566 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.566 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.566 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:02.566 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.330709+0000", 2020-01-16T00:13:02.566 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00071550499999999996 2020-01-16T00:13:02.566 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.566 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.567 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:02.567 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331424+0000", 2020-01-16T00:13:02.567 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3354e-05 2020-01-16T00:13:02.567 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.567 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.567 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:02.567 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331447+0000", 2020-01-16T00:13:02.567 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8200000000000008e-06 2020-01-16T00:13:02.568 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.568 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.568 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:02.568 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331457+0000", 2020-01-16T00:13:02.568 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5557999999999997e-05 2020-01-16T00:13:02.568 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.568 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.569 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:02.569 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331533+0000", 2020-01-16T00:13:02.569 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030925000000000001 2020-01-16T00:13:02.569 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.569 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.569 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:02.570 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331842+0000", 2020-01-16T00:13:02.570 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0078507860000000002 2020-01-16T00:13:02.570 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.570 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.570 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:02.570 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.339693+0000", 2020-01-16T00:13:02.570 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9321000000000007e-05 2020-01-16T00:13:02.570 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.571 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.571 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:02.571 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.339792+0000", 2020-01-16T00:13:02.571 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015147700000000001 2020-01-16T00:13:02.571 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.571 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.571 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:02.571 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.339944+0000", 2020-01-16T00:13:02.572 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002663066 2020-01-16T00:13:02.572 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.572 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.572 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:02.572 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.342607+0000", 2020-01-16T00:13:02.572 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0001552 2020-01-16T00:13:02.572 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.572 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.573 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:02.573 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.342762+0000", 2020-01-16T00:13:02.573 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00063636900000000002 2020-01-16T00:13:02.573 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.573 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.573 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:02.574 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343398+0000", 2020-01-16T00:13:02.574 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9687000000000001e-05 2020-01-16T00:13:02.574 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.574 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.574 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:02.574 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343438+0000", 2020-01-16T00:13:02.574 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4647e-05 2020-01-16T00:13:02.574 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.575 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.575 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:02.575 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343493+0000", 2020-01-16T00:13:02.575 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012781043000000001 2020-01-16T00:13:02.575 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.575 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:02.575 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.575 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.576 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.576 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1918 3.a 3:5069665e:::smithi12313934-2:head [write 3335030~144019] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:13:02.576 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:57.334874+0000", 2020-01-16T00:13:02.576 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.0286267179999999, 2020-01-16T00:13:02.576 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0090004249999999994, 2020-01-16T00:13:02.576 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:02.576 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:02.577 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:02.577 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:02.577 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:02.577 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1918 2020-01-16T00:13:02.577 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.577 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:02.577 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.577 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:02.578 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.334874+0000", 2020-01-16T00:13:02.578 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:02.578 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.578 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.578 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:02.578 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.334874+0000", 2020-01-16T00:13:02.578 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:13:02.578 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.579 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.579 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:02.579 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.334870+0000", 2020-01-16T00:13:02.579 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000113143 2020-01-16T00:13:02.579 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.579 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.579 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:02.580 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.334983+0000", 2020-01-16T00:13:02.580 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4298000000000003e-05 2020-01-16T00:13:02.580 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.580 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.580 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:02.580 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335018+0000", 2020-01-16T00:13:02.581 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6860000000000001e-05 2020-01-16T00:13:02.581 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.581 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.581 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:02.581 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335035+0000", 2020-01-16T00:13:02.581 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000111932 2020-01-16T00:13:02.581 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.582 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.582 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:02.582 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335146+0000", 2020-01-16T00:13:02.582 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012407400000000001 2020-01-16T00:13:02.582 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.582 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.582 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:02.582 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335271+0000", 2020-01-16T00:13:02.583 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0056080180000000002 2020-01-16T00:13:02.583 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.583 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.583 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:02.583 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.340879+0000", 2020-01-16T00:13:02.583 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012836600000000001 2020-01-16T00:13:02.583 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.584 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.584 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:02.584 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.341007+0000", 2020-01-16T00:13:02.584 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019871900000000001 2020-01-16T00:13:02.584 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.584 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.584 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:02.584 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.341206+0000", 2020-01-16T00:13:02.585 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001501528 2020-01-16T00:13:02.585 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.585 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.585 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:02.585 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.342707+0000", 2020-01-16T00:13:02.585 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000300422 2020-01-16T00:13:02.585 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.586 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.586 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:02.586 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343008+0000", 2020-01-16T00:13:02.586 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00074073800000000001 2020-01-16T00:13:02.586 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.586 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.586 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:02.586 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343748+0000", 2020-01-16T00:13:02.587 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8270000000000003e-05 2020-01-16T00:13:02.587 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.587 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.587 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:02.587 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343807+0000", 2020-01-16T00:13:02.587 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8032000000000002e-05 2020-01-16T00:13:02.587 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.587 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.588 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:02.588 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343875+0000", 2020-01-16T00:13:02.588 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0090004249999999994 2020-01-16T00:13:02.588 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.588 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:02.588 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.588 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.589 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.589 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1919 3.a 3:5069665e:::smithi12313934-2:head [setxattr _header (55) in=7b,truncate 3479049] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:13:02.589 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:57.335061+0000", 2020-01-16T00:13:02.589 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.0284399899999999, 2020-01-16T00:13:02.589 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0089426209999999996, 2020-01-16T00:13:02.589 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:02.589 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:02.589 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:02.590 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:02.590 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:02.590 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1919 2020-01-16T00:13:02.590 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.590 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:02.590 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.590 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:02.591 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335061+0000", 2020-01-16T00:13:02.591 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:02.591 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.591 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.591 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:02.591 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335061+0000", 2020-01-16T00:13:02.591 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:02.591 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.592 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.592 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:02.592 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335059+0000", 2020-01-16T00:13:02.592 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5460000000000001e-05 2020-01-16T00:13:02.592 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.592 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.592 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:02.592 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335075+0000", 2020-01-16T00:13:02.593 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6120000000000002e-05 2020-01-16T00:13:02.593 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.593 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.593 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:02.593 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335091+0000", 2020-01-16T00:13:02.593 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0587999999999999e-05 2020-01-16T00:13:02.593 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.593 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.594 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:02.594 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335101+0000", 2020-01-16T00:13:02.594 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00027255500000000001 2020-01-16T00:13:02.594 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.594 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.594 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:02.594 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335374+0000", 2020-01-16T00:13:02.595 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016375999999999999 2020-01-16T00:13:02.595 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.595 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.595 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:02.595 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335538+0000", 2020-01-16T00:13:02.595 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0063584440000000004 2020-01-16T00:13:02.595 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.595 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.596 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:02.596 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.341896+0000", 2020-01-16T00:13:02.596 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012864300000000001 2020-01-16T00:13:02.596 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.596 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.596 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:02.596 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.342025+0000", 2020-01-16T00:13:02.597 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019981299999999999 2020-01-16T00:13:02.597 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.597 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.597 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:02.597 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.342225+0000", 2020-01-16T00:13:02.597 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00048735800000000002 2020-01-16T00:13:02.597 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.597 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.598 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:02.598 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.342712+0000", 2020-01-16T00:13:02.598 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00047628299999999998 2020-01-16T00:13:02.598 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.598 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.598 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:02.598 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343188+0000", 2020-01-16T00:13:02.598 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000718606 2020-01-16T00:13:02.599 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.599 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.599 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:02.599 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343907+0000", 2020-01-16T00:13:02.599 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7583999999999998e-05 2020-01-16T00:13:02.599 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.600 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.600 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:02.600 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343944+0000", 2020-01-16T00:13:02.600 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9209000000000002e-05 2020-01-16T00:13:02.600 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.600 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.600 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:02.601 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.344004+0000", 2020-01-16T00:13:02.601 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0089426209999999996 2020-01-16T00:13:02.601 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.601 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:02.601 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.601 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.601 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.602 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1920 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e31)", 2020-01-16T00:13:02.602 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:57.335132+0000", 2020-01-16T00:13:02.602 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.0283692440000003, 2020-01-16T00:13:02.602 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0091378889999999997, 2020-01-16T00:13:02.602 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:02.602 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:02.602 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:02.602 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:02.603 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:02.603 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1920 2020-01-16T00:13:02.603 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.603 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:02.603 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.603 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:02.603 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335132+0000", 2020-01-16T00:13:02.603 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:02.604 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.604 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.604 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:02.604 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335132+0000", 2020-01-16T00:13:02.604 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:02.604 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.604 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.604 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:02.605 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335130+0000", 2020-01-16T00:13:02.605 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0339999999999997e-06 2020-01-16T00:13:02.605 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.605 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.605 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:02.605 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335138+0000", 2020-01-16T00:13:02.605 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5296999999999999e-05 2020-01-16T00:13:02.606 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.606 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.606 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:02.606 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335154+0000", 2020-01-16T00:13:02.606 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9110000000000005e-06 2020-01-16T00:13:02.606 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.606 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.606 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:02.607 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335164+0000", 2020-01-16T00:13:02.607 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00044850000000000001 2020-01-16T00:13:02.607 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.607 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.607 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:02.607 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335612+0000", 2020-01-16T00:13:02.607 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2644000000000003e-05 2020-01-16T00:13:02.608 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.608 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.608 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:02.608 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335705+0000", 2020-01-16T00:13:02.608 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0068223650000000004 2020-01-16T00:13:02.608 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.608 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.608 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:02.609 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.342527+0000", 2020-01-16T00:13:02.609 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1173999999999994e-05 2020-01-16T00:13:02.609 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.609 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.609 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:02.609 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.342588+0000", 2020-01-16T00:13:02.609 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014580769999999999 2020-01-16T00:13:02.610 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.610 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.610 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:02.610 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.344046+0000", 2020-01-16T00:13:02.610 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000106702 2020-01-16T00:13:02.610 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.610 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.610 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:02.611 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.344153+0000", 2020-01-16T00:13:02.611 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011658999999999999 2020-01-16T00:13:02.611 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.611 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.611 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:02.611 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.344270+0000", 2020-01-16T00:13:02.611 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0091378889999999997 2020-01-16T00:13:02.611 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.612 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:02.612 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.612 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.612 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.612 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2114 3.a 3:55e7052a:::smithi12313934-7:head [write 682417~473109] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:13:02.612 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:01.650529+0000", 2020-01-16T00:13:02.612 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.71297206099999999, 2020-01-16T00:13:02.612 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0093900500000000005, 2020-01-16T00:13:02.613 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:02.613 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:02.613 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:02.613 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:02.613 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:02.613 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2114 2020-01-16T00:13:02.613 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.614 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:02.614 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.614 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:02.614 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.650529+0000", 2020-01-16T00:13:02.614 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:02.614 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.614 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.614 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:02.615 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.650529+0000", 2020-01-16T00:13:02.615 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:02.615 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.615 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.615 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:02.615 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.650526+0000", 2020-01-16T00:13:02.615 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00074189500000000001 2020-01-16T00:13:02.615 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.616 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.616 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:02.616 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.651267+0000", 2020-01-16T00:13:02.616 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0999e-05 2020-01-16T00:13:02.616 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.616 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.616 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:02.616 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.651288+0000", 2020-01-16T00:13:02.617 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8230000000000006e-06 2020-01-16T00:13:02.617 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.617 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.617 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:02.617 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.651298+0000", 2020-01-16T00:13:02.617 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019944599999999999 2020-01-16T00:13:02.618 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.618 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.618 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:02.618 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.651498+0000", 2020-01-16T00:13:02.618 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0808e-05 2020-01-16T00:13:02.618 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.618 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.618 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op must wait for map", 2020-01-16T00:13:02.619 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.651529+0000", 2020-01-16T00:13:02.619 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0027072860000000002 2020-01-16T00:13:02.619 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.619 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.619 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:02.619 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.654236+0000", 2020-01-16T00:13:02.619 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010767700000000001 2020-01-16T00:13:02.619 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.620 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.620 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:02.620 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.654344+0000", 2020-01-16T00:13:02.620 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000261341 2020-01-16T00:13:02.620 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.620 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.620 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:02.621 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.654605+0000", 2020-01-16T00:13:02.621 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001537737 2020-01-16T00:13:02.621 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.621 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.621 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:02.621 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.656143+0000", 2020-01-16T00:13:02.621 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00302163 2020-01-16T00:13:02.621 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.622 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.622 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:02.622 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.659164+0000", 2020-01-16T00:13:02.622 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00063603599999999998 2020-01-16T00:13:02.622 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.622 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.622 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:02.622 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.659800+0000", 2020-01-16T00:13:02.623 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6470000000000006e-05 2020-01-16T00:13:02.623 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.623 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.623 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:02.623 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.659867+0000", 2020-01-16T00:13:02.623 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2218e-05 2020-01-16T00:13:02.623 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:02.624 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:02.624 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:02.624 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.659919+0000", 2020-01-16T00:13:02.624 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0093900500000000005 2020-01-16T00:13:02.624 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.624 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:02.624 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.624 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:02.625 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:02.625 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:02.630 INFO:tasks.rados.rados.0.smithi123.stdout:979: writing smithi12313934-10 from 3210039 to 3781690 tid 3 2020-01-16T00:13:02.630 INFO:tasks.rados.rados.0.smithi123.stdout:980: write oid 15 current snap is 0 2020-01-16T00:13:02.630 INFO:tasks.rados.rados.0.smithi123.stdout:980: seq_num 235 ranges {749645=774105,2085386=609401,3237462=712183} 2020-01-16T00:13:02.630 INFO:tasks.rados.rados.0.smithi123.stdout:980: writing smithi12313934-15 from 749645 to 1523750 tid 1 2020-01-16T00:13:02.630 INFO:tasks.rados.rados.0.smithi123.stdout:980: writing smithi12313934-15 from 2085386 to 2694787 tid 2 2020-01-16T00:13:02.630 INFO:tasks.rados.rados.0.smithi123.stdout:980: writing smithi12313934-15 from 3237462 to 3949645 tid 3 2020-01-16T00:13:02.631 INFO:tasks.rados.rados.0.smithi123.stdout:981: read oid 4 snap -1 2020-01-16T00:13:02.631 INFO:tasks.rados.rados.0.smithi123.stdout:981: expect (ObjNum 188 snap 0 seq_num 188) 2020-01-16T00:13:02.631 INFO:tasks.rados.rados.0.smithi123.stdout:982: read oid 46 snap -1 2020-01-16T00:13:02.631 INFO:tasks.rados.rados.0.smithi123.stdout:982: expect (ObjNum 226 snap 0 seq_num 226) 2020-01-16T00:13:02.631 INFO:tasks.rados.rados.0.smithi123.stdout:983: read oid 39 snap -1 2020-01-16T00:13:02.631 INFO:tasks.rados.rados.0.smithi123.stdout:983: expect (ObjNum 194 snap 0 seq_num 194) 2020-01-16T00:13:02.631 INFO:tasks.rados.rados.0.smithi123.stdout:984: write oid 9 current snap is 0 2020-01-16T00:13:02.631 INFO:tasks.rados.rados.0.smithi123.stdout:984: seq_num 236 ranges {766452=565707,1860221=469412,2981238=594195} 2020-01-16T00:13:02.632 INFO:tasks.rados.rados.0.smithi123.stdout:984: writing smithi12313934-9 from 766452 to 1332159 tid 1 2020-01-16T00:13:02.632 INFO:tasks.rados.rados.0.smithi123.stdout:984: writing smithi12313934-9 from 1860221 to 2329633 tid 2 2020-01-16T00:13:02.632 INFO:tasks.rados.rados.0.smithi123.stdout:984: writing smithi12313934-9 from 2981238 to 3575433 tid 3 2020-01-16T00:13:02.632 INFO:tasks.rados.rados.0.smithi123.stdout:985: read oid 29 snap -1 2020-01-16T00:13:02.632 INFO:tasks.rados.rados.0.smithi123.stdout:985: expect (ObjNum 200 snap 0 seq_num 200) 2020-01-16T00:13:02.632 INFO:tasks.rados.rados.0.smithi123.stdout:986: read oid 40 snap -1 2020-01-16T00:13:02.632 INFO:tasks.rados.rados.0.smithi123.stdout:986: expect (ObjNum 224 snap 0 seq_num 224) 2020-01-16T00:13:02.632 INFO:tasks.rados.rados.0.smithi123.stdout:987: read oid 13 snap -1 2020-01-16T00:13:02.633 INFO:tasks.rados.rados.0.smithi123.stdout:987: expect (ObjNum 230 snap 0 seq_num 230) 2020-01-16T00:13:02.633 INFO:tasks.rados.rados.0.smithi123.stdout:988: write oid 17 current snap is 0 2020-01-16T00:13:02.633 INFO:tasks.rados.rados.0.smithi123.stdout:988: seq_num 237 ranges {783259=640956,1918703=613070,3008661=405467,3821939=161320} 2020-01-16T00:13:02.633 INFO:tasks.rados.rados.0.smithi123.stdout:988: writing smithi12313934-17 from 783259 to 1424215 tid 1 2020-01-16T00:13:02.633 INFO:tasks.rados.rados.0.smithi123.stdout:988: writing smithi12313934-17 from 1918703 to 2531773 tid 2 2020-01-16T00:13:02.633 INFO:tasks.rados.rados.0.smithi123.stdout:988: writing smithi12313934-17 from 3008661 to 3414128 tid 3 2020-01-16T00:13:02.633 INFO:tasks.rados.rados.0.smithi123.stdout:988: writing smithi12313934-17 from 3821939 to 3983259 tid 4 2020-01-16T00:13:02.633 INFO:tasks.rados.rados.0.smithi123.stdout:989: read oid 35 snap -1 2020-01-16T00:13:02.634 INFO:tasks.rados.rados.0.smithi123.stdout:989: expect (ObjNum 206 snap 0 seq_num 206) 2020-01-16T00:13:02.634 INFO:tasks.rados.rados.0.smithi123.stdout:990: read oid 16 snap -1 2020-01-16T00:13:02.634 INFO:tasks.rados.rados.0.smithi123.stdout:990: expect (ObjNum 60 snap 0 seq_num 60) 2020-01-16T00:13:02.634 INFO:tasks.rados.rados.0.smithi123.stdout:991: read oid 48 snap -1 2020-01-16T00:13:02.634 INFO:tasks.rados.rados.0.smithi123.stdout:991: expect (ObjNum 189 snap 0 seq_num 189) 2020-01-16T00:13:02.634 INFO:tasks.rados.rados.0.smithi123.stdout:992: read oid 14 snap -1 2020-01-16T00:13:02.634 INFO:tasks.rados.rados.0.smithi123.stdout:992: expect (ObjNum 217 snap 0 seq_num 217) 2020-01-16T00:13:02.634 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:02.635 INFO:tasks.rados.rados.0.smithi123.stdout:977: expect (ObjNum 191 snap 0 seq_num 191) 2020-01-16T00:13:02.635 INFO:tasks.rados.rados.0.smithi123.stdout:978: expect (ObjNum 212 snap 0 seq_num 212) 2020-01-16T00:13:02.635 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:02.623+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:02.635 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-01-16T00:13:02.727 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:02.723+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:02.748 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:02.748 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:02.748 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:13:02.749 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:02.757 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-01-16T00:13:02.828 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:02.827+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:02.854 INFO:tasks.rados.rados.0.smithi123.stdout:979: finishing write tid 1 to smithi12313934-10 2020-01-16T00:13:02.855 INFO:tasks.rados.rados.0.smithi123.stdout:979: finishing write tid 2 to smithi12313934-10 2020-01-16T00:13:02.855 INFO:tasks.rados.rados.0.smithi123.stdout:979: finishing write tid 3 to smithi12313934-10 2020-01-16T00:13:02.855 INFO:tasks.rados.rados.0.smithi123.stdout:979: finishing write tid 5 to smithi12313934-10 2020-01-16T00:13:02.855 INFO:tasks.rados.rados.0.smithi123.stdout:979: finishing write tid 6 to smithi12313934-10 2020-01-16T00:13:02.855 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 10 v 98 (ObjNum 234 snap 0 seq_num 234) dirty exists 2020-01-16T00:13:02.855 INFO:tasks.rados.rados.0.smithi123.stdout:979: left oid 10 (ObjNum 234 snap 0 seq_num 234) 2020-01-16T00:13:02.856 INFO:tasks.rados.rados.0.smithi123.stdout:980: finishing write tid 1 to smithi12313934-15 2020-01-16T00:13:02.856 INFO:tasks.rados.rados.0.smithi123.stdout:980: finishing write tid 2 to smithi12313934-15 2020-01-16T00:13:02.856 INFO:tasks.rados.rados.0.smithi123.stdout:980: finishing write tid 3 to smithi12313934-15 2020-01-16T00:13:02.856 INFO:tasks.rados.rados.0.smithi123.stdout:980: finishing write tid 5 to smithi12313934-15 2020-01-16T00:13:02.856 INFO:tasks.rados.rados.0.smithi123.stdout:980: finishing write tid 6 to smithi12313934-15 2020-01-16T00:13:02.856 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 15 v 99 (ObjNum 235 snap 0 seq_num 235) dirty exists 2020-01-16T00:13:02.856 INFO:tasks.rados.rados.0.smithi123.stdout:980: left oid 15 (ObjNum 235 snap 0 seq_num 235) 2020-01-16T00:13:02.856 INFO:tasks.rados.rados.0.smithi123.stdout:982: expect (ObjNum 226 snap 0 seq_num 226) 2020-01-16T00:13:02.859 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:02.859 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:02.859 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:13:02.860 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:13:02.860 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:02.870 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-01-16T00:13:02.873 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4, 7, 6] 2020-01-16T00:13:02.873 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2020-01-16T00:13:02.874 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 1.000000 2020-01-16T00:13:02.874 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2020-01-16T00:13:02.961 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:02.959+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:03.023 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:03.024 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:13:03.024 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:13:03.024 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:13:03.024 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.024 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:885 3.0 3:0b70bb20:::smithi12313934-46:head [write 742876~447756] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:03.024 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:36.940663+0000", 2020-01-16T00:13:03.024 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 26.079172693, 2020-01-16T00:13:03.025 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012667668, 2020-01-16T00:13:03.025 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:03.025 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:03.025 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:03.025 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:03.025 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:03.025 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 885 2020-01-16T00:13:03.025 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.026 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:03.026 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.026 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:03.026 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.940663+0000", 2020-01-16T00:13:03.026 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:03.026 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.026 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.026 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:03.027 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.940663+0000", 2020-01-16T00:13:03.027 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:03.027 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.027 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.027 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:03.027 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.940661+0000", 2020-01-16T00:13:03.027 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000152392 2020-01-16T00:13:03.028 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.028 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.028 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:03.028 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.940813+0000", 2020-01-16T00:13:03.028 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7827000000000001e-05 2020-01-16T00:13:03.028 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.029 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.029 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:03.029 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.940831+0000", 2020-01-16T00:13:03.029 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9649999999999997e-06 2020-01-16T00:13:03.029 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.029 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.029 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:03.030 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.940840+0000", 2020-01-16T00:13:03.030 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4238000000000003e-05 2020-01-16T00:13:03.030 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.030 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.030 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.030 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.940914+0000", 2020-01-16T00:13:03.030 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019430200000000001 2020-01-16T00:13:03.031 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.031 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.031 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:03.031 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.941108+0000", 2020-01-16T00:13:03.031 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.004627511 2020-01-16T00:13:03.031 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.031 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.031 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.032 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.945736+0000", 2020-01-16T00:13:03.032 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000121834 2020-01-16T00:13:03.032 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.032 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.032 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:03.033 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.945858+0000", 2020-01-16T00:13:03.033 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000196589 2020-01-16T00:13:03.033 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.033 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.033 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:03.033 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.946054+0000", 2020-01-16T00:13:03.033 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0038495209999999998 2020-01-16T00:13:03.034 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.034 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.034 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:03.034 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.949904+0000", 2020-01-16T00:13:03.034 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001149148 2020-01-16T00:13:03.034 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.034 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.034 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:03.035 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.951053+0000", 2020-01-16T00:13:03.035 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002184812 2020-01-16T00:13:03.035 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.035 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.035 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:03.035 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.953238+0000", 2020-01-16T00:13:03.035 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8943000000000002e-05 2020-01-16T00:13:03.035 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.036 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.036 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:03.036 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.953287+0000", 2020-01-16T00:13:03.036 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3714000000000002e-05 2020-01-16T00:13:03.036 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.036 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.036 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:03.036 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:36.953330+0000", 2020-01-16T00:13:03.037 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012667668 2020-01-16T00:13:03.037 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.037 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:03.037 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.037 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.037 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.037 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:939 3.4 3:2bdd9e88:::smithi12313934-4:head [write 1681102~474010] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:03.038 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.908392+0000", 2020-01-16T00:13:03.038 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 25.111442970999999, 2020-01-16T00:13:03.038 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019294697, 2020-01-16T00:13:03.038 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:03.038 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:03.038 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:03.038 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:03.038 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:03.039 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 939 2020-01-16T00:13:03.039 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.039 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:03.039 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.039 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:03.039 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908392+0000", 2020-01-16T00:13:03.039 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:03.040 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.040 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.040 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:03.040 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908392+0000", 2020-01-16T00:13:03.040 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:03.040 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.040 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.040 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:03.041 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908390+0000", 2020-01-16T00:13:03.041 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015440399999999999 2020-01-16T00:13:03.041 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.041 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.041 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:03.041 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908545+0000", 2020-01-16T00:13:03.041 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3926e-05 2020-01-16T00:13:03.041 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.042 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.042 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:03.042 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908559+0000", 2020-01-16T00:13:03.042 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2200000000000003e-06 2020-01-16T00:13:03.043 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.043 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.043 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:03.043 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908566+0000", 2020-01-16T00:13:03.044 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020594799999999998 2020-01-16T00:13:03.044 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.044 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.044 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.044 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910625+0000", 2020-01-16T00:13:03.044 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118672 2020-01-16T00:13:03.044 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.045 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.045 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:03.045 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910744+0000", 2020-01-16T00:13:03.045 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031940969999999999 2020-01-16T00:13:03.045 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.045 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.045 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.046 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.913938+0000", 2020-01-16T00:13:03.046 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000309621 2020-01-16T00:13:03.046 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.046 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.046 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:03.046 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.914248+0000", 2020-01-16T00:13:03.046 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.008818579 2020-01-16T00:13:03.046 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.047 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.047 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.047 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.923066+0000", 2020-01-16T00:13:03.047 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000108989 2020-01-16T00:13:03.047 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.047 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.048 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:03.048 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.923175+0000", 2020-01-16T00:13:03.048 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019128299999999999 2020-01-16T00:13:03.048 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.048 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.048 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:13:03.049 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.923367+0000", 2020-01-16T00:13:03.049 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0032083760000000002 2020-01-16T00:13:03.049 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.049 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.049 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:03.049 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926575+0000", 2020-01-16T00:13:03.049 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2475999999999999e-05 2020-01-16T00:13:03.051 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.051 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.051 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:03.051 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926607+0000", 2020-01-16T00:13:03.051 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00099399800000000006 2020-01-16T00:13:03.051 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.052 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.052 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:03.052 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927601+0000", 2020-01-16T00:13:03.052 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0958999999999999e-05 2020-01-16T00:13:03.053 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.053 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.053 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:03.053 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927642+0000", 2020-01-16T00:13:03.053 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4747999999999999e-05 2020-01-16T00:13:03.053 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.053 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.054 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:03.054 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927687+0000", 2020-01-16T00:13:03.054 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019294697 2020-01-16T00:13:03.054 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.054 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:03.054 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.054 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.055 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.055 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:940 3.4 3:2bdd9e88:::smithi12313934-4:head [write 2687912~589420] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:03.055 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.915144+0000", 2020-01-16T00:13:03.055 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 25.104691602999999, 2020-01-16T00:13:03.055 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016092538999999999, 2020-01-16T00:13:03.055 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:03.056 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:03.056 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:03.056 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:03.056 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:03.056 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 940 2020-01-16T00:13:03.056 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.057 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:03.057 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.057 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:03.057 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915144+0000", 2020-01-16T00:13:03.057 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:03.057 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.057 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.058 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:03.058 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915144+0000", 2020-01-16T00:13:03.058 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:03.058 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.058 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.058 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:03.058 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915141+0000", 2020-01-16T00:13:03.059 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016366300000000001 2020-01-16T00:13:03.059 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.059 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.059 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:03.059 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915305+0000", 2020-01-16T00:13:03.059 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4847e-05 2020-01-16T00:13:03.059 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.060 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.060 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:03.060 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915320+0000", 2020-01-16T00:13:03.061 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2599999999999999e-06 2020-01-16T00:13:03.061 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.061 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.061 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:03.061 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915327+0000", 2020-01-16T00:13:03.062 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4282000000000005e-05 2020-01-16T00:13:03.062 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.062 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.062 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.062 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915401+0000", 2020-01-16T00:13:03.063 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4554000000000006e-05 2020-01-16T00:13:03.063 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.063 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.063 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:03.063 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915466+0000", 2020-01-16T00:13:03.064 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0090821900000000004 2020-01-16T00:13:03.064 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.064 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.065 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.065 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.924548+0000", 2020-01-16T00:13:03.065 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000114696 2020-01-16T00:13:03.065 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.065 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.065 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:03.065 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.924663+0000", 2020-01-16T00:13:03.066 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000168651 2020-01-16T00:13:03.066 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.066 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.066 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:13:03.066 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.924832+0000", 2020-01-16T00:13:03.066 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018253430000000001 2020-01-16T00:13:03.066 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.067 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.067 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:03.067 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926657+0000", 2020-01-16T00:13:03.067 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9957999999999999e-05 2020-01-16T00:13:03.067 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.067 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.067 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:03.067 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926697+0000", 2020-01-16T00:13:03.068 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0044685250000000001 2020-01-16T00:13:03.068 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.068 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.069 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:03.069 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931165+0000", 2020-01-16T00:13:03.070 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9211000000000001e-05 2020-01-16T00:13:03.070 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.070 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.070 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:03.070 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931205+0000", 2020-01-16T00:13:03.070 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1699999999999998e-05 2020-01-16T00:13:03.071 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.071 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.072 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:03.072 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931236+0000", 2020-01-16T00:13:03.072 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016092538999999999 2020-01-16T00:13:03.072 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.072 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:03.073 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.073 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.073 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.073 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:941 3.4 3:2bdd9e88:::smithi12313934-4:head [setxattr _header (55) in=7b,truncate 3277332] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:03.073 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.915341+0000", 2020-01-16T00:13:03.073 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 25.104494785, 2020-01-16T00:13:03.074 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016603769000000001, 2020-01-16T00:13:03.075 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:03.075 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:03.075 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:03.075 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:03.075 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:03.075 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 941 2020-01-16T00:13:03.075 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.076 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:03.076 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.076 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:03.076 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915341+0000", 2020-01-16T00:13:03.076 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:03.076 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.076 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.077 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:03.077 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915341+0000", 2020-01-16T00:13:03.077 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:03.077 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.077 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.077 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:03.078 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915340+0000", 2020-01-16T00:13:03.078 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.896e-06 2020-01-16T00:13:03.078 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.078 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.078 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:03.078 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915346+0000", 2020-01-16T00:13:03.079 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2730000000000002e-06 2020-01-16T00:13:03.079 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.079 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.079 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:03.080 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915354+0000", 2020-01-16T00:13:03.080 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0616e-05 2020-01-16T00:13:03.080 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.080 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.080 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:03.081 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915374+0000", 2020-01-16T00:13:03.081 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000133487 2020-01-16T00:13:03.081 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.081 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.082 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.082 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915508+0000", 2020-01-16T00:13:03.082 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8971000000000003e-05 2020-01-16T00:13:03.085 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.085 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.085 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:03.085 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915557+0000", 2020-01-16T00:13:03.085 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010138787999999999 2020-01-16T00:13:03.086 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.086 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.086 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.086 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.925696+0000", 2020-01-16T00:13:03.086 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012825400000000001 2020-01-16T00:13:03.086 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.087 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.087 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:03.087 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.925824+0000", 2020-01-16T00:13:03.087 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017706500000000001 2020-01-16T00:13:03.087 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.087 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.087 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:13:03.088 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926001+0000", 2020-01-16T00:13:03.088 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00079988900000000003 2020-01-16T00:13:03.088 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.088 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.088 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:03.088 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926801+0000", 2020-01-16T00:13:03.088 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119324 2020-01-16T00:13:03.089 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.089 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.089 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:03.089 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926920+0000", 2020-01-16T00:13:03.089 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048879910000000004 2020-01-16T00:13:03.089 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.089 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.090 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:03.090 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931808+0000", 2020-01-16T00:13:03.090 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6999999999999997e-05 2020-01-16T00:13:03.090 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.090 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.090 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:03.090 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931855+0000", 2020-01-16T00:13:03.090 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9092999999999996e-05 2020-01-16T00:13:03.091 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.091 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.091 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:03.092 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931944+0000", 2020-01-16T00:13:03.092 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016603769000000001 2020-01-16T00:13:03.092 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.092 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:03.092 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.093 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.093 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.093 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:942 3.4 3:2bdd9e88:::smithi12313934-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:13:03.094 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.915389+0000", 2020-01-16T00:13:03.094 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 25.104446418999999, 2020-01-16T00:13:03.094 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016820563, 2020-01-16T00:13:03.095 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:03.095 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:03.095 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:03.095 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:03.095 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:03.095 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 942 2020-01-16T00:13:03.095 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.095 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:03.096 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.096 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:03.096 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915389+0000", 2020-01-16T00:13:03.096 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:03.096 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.096 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.096 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:03.097 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915389+0000", 2020-01-16T00:13:03.097 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:03.097 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.097 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.097 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:03.097 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915388+0000", 2020-01-16T00:13:03.097 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.683e-06 2020-01-16T00:13:03.097 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.098 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.098 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:03.098 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915391+0000", 2020-01-16T00:13:03.098 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4839999999999998e-06 2020-01-16T00:13:03.098 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.098 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.098 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:03.098 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915399+0000", 2020-01-16T00:13:03.099 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2259999999999999e-06 2020-01-16T00:13:03.099 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.099 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.099 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:03.099 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915403+0000", 2020-01-16T00:13:03.099 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00031301199999999999 2020-01-16T00:13:03.100 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.100 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.100 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.100 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915716+0000", 2020-01-16T00:13:03.100 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2079999999999998e-05 2020-01-16T00:13:03.100 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.100 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.101 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:03.101 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915748+0000", 2020-01-16T00:13:03.101 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010547076000000001 2020-01-16T00:13:03.101 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.101 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.102 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.102 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926295+0000", 2020-01-16T00:13:03.102 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8040000000000005e-05 2020-01-16T00:13:03.102 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.102 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.102 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:03.103 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926373+0000", 2020-01-16T00:13:03.103 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056061990000000001 2020-01-16T00:13:03.103 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.103 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.104 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.104 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931979+0000", 2020-01-16T00:13:03.104 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011540399999999999 2020-01-16T00:13:03.104 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.105 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.105 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:03.105 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932095+0000", 2020-01-16T00:13:03.105 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011482899999999999 2020-01-16T00:13:03.105 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.105 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.105 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:03.105 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932210+0000", 2020-01-16T00:13:03.106 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016820563 2020-01-16T00:13:03.106 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.106 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:03.106 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.106 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.106 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.106 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:957 3.0 3:00c146d5:::smithi12313934-18:head [write 510946~632952] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:03.107 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.033290+0000", 2020-01-16T00:13:03.107 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 24.986545582000002, 2020-01-16T00:13:03.107 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016764102999999999, 2020-01-16T00:13:03.107 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:03.107 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:03.107 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:03.107 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:03.107 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:03.108 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 957 2020-01-16T00:13:03.108 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.108 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:03.108 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.108 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:03.108 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033290+0000", 2020-01-16T00:13:03.108 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:03.108 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.109 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.109 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:03.109 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033290+0000", 2020-01-16T00:13:03.109 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999952 2020-01-16T00:13:03.109 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.109 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.109 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:03.110 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033285+0000", 2020-01-16T00:13:03.110 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000405917 2020-01-16T00:13:03.110 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.110 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.110 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:03.110 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033691+0000", 2020-01-16T00:13:03.110 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6585999999999997e-05 2020-01-16T00:13:03.110 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.111 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.111 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:03.111 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033738+0000", 2020-01-16T00:13:03.111 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8374000000000001e-05 2020-01-16T00:13:03.111 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.111 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.111 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:03.111 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033756+0000", 2020-01-16T00:13:03.112 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013002900000000001 2020-01-16T00:13:03.112 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.112 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.112 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.112 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033886+0000", 2020-01-16T00:13:03.112 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035882600000000002 2020-01-16T00:13:03.112 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.112 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.113 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:03.113 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.034245+0000", 2020-01-16T00:13:03.113 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0072798489999999997 2020-01-16T00:13:03.113 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.113 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.113 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.113 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.041525+0000", 2020-01-16T00:13:03.114 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000140488 2020-01-16T00:13:03.114 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.114 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.114 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:03.114 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.041665+0000", 2020-01-16T00:13:03.114 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000233766 2020-01-16T00:13:03.114 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.114 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.115 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:03.115 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.041899+0000", 2020-01-16T00:13:03.115 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0035766539999999999 2020-01-16T00:13:03.116 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.117 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.117 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:03.117 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.045475+0000", 2020-01-16T00:13:03.117 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013333150000000001 2020-01-16T00:13:03.117 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.117 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.117 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:03.117 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.046809+0000", 2020-01-16T00:13:03.118 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031447860000000001 2020-01-16T00:13:03.118 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.118 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.118 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:03.118 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.049954+0000", 2020-01-16T00:13:03.118 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2318000000000003e-05 2020-01-16T00:13:03.118 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.118 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.119 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:03.119 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.050006+0000", 2020-01-16T00:13:03.119 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7972000000000001e-05 2020-01-16T00:13:03.119 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.119 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.119 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:03.119 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.050054+0000", 2020-01-16T00:13:03.119 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016764102999999999 2020-01-16T00:13:03.120 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.120 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:03.120 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.120 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.120 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.120 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1168 3.0 3:03b92cf0:::smithi12313934-26:head [write 2905319~482223] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:13:03.120 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:41.672662+0000", 2020-01-16T00:13:03.121 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 21.347173735999998, 2020-01-16T00:13:03.121 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015446582, 2020-01-16T00:13:03.121 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:03.121 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:03.121 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:03.121 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:03.121 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:03.121 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1168 2020-01-16T00:13:03.122 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.122 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:03.122 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.122 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:03.122 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672662+0000", 2020-01-16T00:13:03.122 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:03.122 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.122 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.123 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:03.123 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672662+0000", 2020-01-16T00:13:03.123 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:03.123 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.123 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.123 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:03.123 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672660+0000", 2020-01-16T00:13:03.123 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013924799999999999 2020-01-16T00:13:03.124 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.124 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.124 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:03.124 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672799+0000", 2020-01-16T00:13:03.124 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1304000000000001e-05 2020-01-16T00:13:03.124 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.124 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.124 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:03.125 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672810+0000", 2020-01-16T00:13:03.125 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2379999999999999e-06 2020-01-16T00:13:03.125 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.125 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.125 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:03.125 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672815+0000", 2020-01-16T00:13:03.125 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9329000000000002e-05 2020-01-16T00:13:03.126 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.127 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.127 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.127 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672854+0000", 2020-01-16T00:13:03.127 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010573939999999999 2020-01-16T00:13:03.127 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.127 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.128 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:03.128 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.673911+0000", 2020-01-16T00:13:03.128 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.01042181 2020-01-16T00:13:03.128 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.128 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.128 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.129 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.684333+0000", 2020-01-16T00:13:03.129 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012700499999999999 2020-01-16T00:13:03.129 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.129 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.129 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:03.129 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.684460+0000", 2020-01-16T00:13:03.129 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019923299999999999 2020-01-16T00:13:03.130 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.130 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.130 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:03.130 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.684659+0000", 2020-01-16T00:13:03.130 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020339300000000002 2020-01-16T00:13:03.130 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.130 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.131 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:03.131 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.686693+0000", 2020-01-16T00:13:03.131 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023967899999999999 2020-01-16T00:13:03.131 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.131 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.131 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:03.132 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.686933+0000", 2020-01-16T00:13:03.132 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010489169999999999 2020-01-16T00:13:03.132 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.132 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.132 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:03.133 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.687982+0000", 2020-01-16T00:13:03.133 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3857999999999997e-05 2020-01-16T00:13:03.133 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.133 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.133 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:03.133 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.688056+0000", 2020-01-16T00:13:03.133 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2324999999999998e-05 2020-01-16T00:13:03.134 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.134 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.134 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:03.134 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.688108+0000", 2020-01-16T00:13:03.134 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015446582 2020-01-16T00:13:03.134 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.134 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:03.135 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.135 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.135 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.135 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1169 3.0 3:03b92cf0:::smithi12313934-26:head [setxattr _header (56) in=7b,truncate 3798349] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:13:03.136 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:41.672823+0000", 2020-01-16T00:13:03.136 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 21.347012829000001, 2020-01-16T00:13:03.136 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015457745, 2020-01-16T00:13:03.136 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:03.136 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:03.136 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:03.136 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:03.137 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:03.137 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1169 2020-01-16T00:13:03.137 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.137 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:03.137 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.137 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:03.138 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672823+0000", 2020-01-16T00:13:03.138 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:03.138 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.138 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.138 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:03.139 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672823+0000", 2020-01-16T00:13:03.139 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:03.139 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.139 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.139 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:03.139 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672822+0000", 2020-01-16T00:13:03.140 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.685e-06 2020-01-16T00:13:03.140 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.140 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.140 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:03.140 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672827+0000", 2020-01-16T00:13:03.140 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.485e-06 2020-01-16T00:13:03.140 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.141 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.141 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:03.141 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672832+0000", 2020-01-16T00:13:03.141 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6440000000000002e-06 2020-01-16T00:13:03.141 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.141 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.141 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:03.142 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672836+0000", 2020-01-16T00:13:03.142 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00112415 2020-01-16T00:13:03.142 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.142 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.142 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.142 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.673960+0000", 2020-01-16T00:13:03.142 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5918999999999998e-05 2020-01-16T00:13:03.143 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.143 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.143 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:03.143 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.674016+0000", 2020-01-16T00:13:03.143 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011614579999999999 2020-01-16T00:13:03.143 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.143 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.144 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.144 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.685630+0000", 2020-01-16T00:13:03.144 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011684399999999999 2020-01-16T00:13:03.144 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.144 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.146 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:03.146 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.685747+0000", 2020-01-16T00:13:03.146 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020774199999999999 2020-01-16T00:13:03.146 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.146 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.146 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:03.147 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.685955+0000", 2020-01-16T00:13:03.147 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00095868400000000001 2020-01-16T00:13:03.147 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.147 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.147 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:03.147 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.686914+0000", 2020-01-16T00:13:03.147 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5278000000000001e-05 2020-01-16T00:13:03.147 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.148 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.148 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:03.148 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.686939+0000", 2020-01-16T00:13:03.148 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001227611 2020-01-16T00:13:03.148 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.148 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.148 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:03.149 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.688166+0000", 2020-01-16T00:13:03.149 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1724999999999997e-05 2020-01-16T00:13:03.149 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.149 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.149 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:03.149 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.688218+0000", 2020-01-16T00:13:03.149 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2057000000000006e-05 2020-01-16T00:13:03.149 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.150 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.150 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:03.150 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.688280+0000", 2020-01-16T00:13:03.150 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015457745 2020-01-16T00:13:03.150 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.150 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:03.150 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.150 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.151 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.151 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1170 3.0 3:03b92cf0:::smithi12313934-26:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-01-16T00:13:03.151 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:41.672848+0000", 2020-01-16T00:13:03.151 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 21.346987260999999, 2020-01-16T00:13:03.151 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015733389, 2020-01-16T00:13:03.151 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:03.151 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:03.152 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:03.152 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:03.152 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:03.152 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1170 2020-01-16T00:13:03.152 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.152 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:03.152 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.152 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:03.153 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672848+0000", 2020-01-16T00:13:03.153 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:03.153 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.153 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.153 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:03.153 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672848+0000", 2020-01-16T00:13:03.153 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:03.153 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.154 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.154 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:03.154 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672847+0000", 2020-01-16T00:13:03.154 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.303e-06 2020-01-16T00:13:03.154 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.154 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.154 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:03.155 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672851+0000", 2020-01-16T00:13:03.155 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4970000000000001e-06 2020-01-16T00:13:03.155 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.155 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.155 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:03.155 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672856+0000", 2020-01-16T00:13:03.155 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8870000000000002e-06 2020-01-16T00:13:03.155 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.156 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.156 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:03.156 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672860+0000", 2020-01-16T00:13:03.156 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0011720140000000001 2020-01-16T00:13:03.156 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.156 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.156 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.156 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.674032+0000", 2020-01-16T00:13:03.157 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2577000000000003e-05 2020-01-16T00:13:03.157 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.157 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.157 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:03.157 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.674065+0000", 2020-01-16T00:13:03.157 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012264538 2020-01-16T00:13:03.157 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.157 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.158 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.158 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.686329+0000", 2020-01-16T00:13:03.158 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000106715 2020-01-16T00:13:03.158 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.158 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.158 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:03.158 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.686436+0000", 2020-01-16T00:13:03.158 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018880800000000001 2020-01-16T00:13:03.159 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.159 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.159 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.159 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.688324+0000", 2020-01-16T00:13:03.159 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012818500000000001 2020-01-16T00:13:03.159 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.159 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.160 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:03.160 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.688452+0000", 2020-01-16T00:13:03.160 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000129197 2020-01-16T00:13:03.160 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.160 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.160 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:03.160 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.688581+0000", 2020-01-16T00:13:03.160 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015733389 2020-01-16T00:13:03.161 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.161 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:03.161 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.161 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.161 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.161 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1394 3.4 3:240c7503:::smithi12313934-35:head [setxattr _header (56) in=7b,truncate 2285752] snapc 0=[] ondisk+write+known_if_redirected e28)", 2020-01-16T00:13:03.161 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:46.625161+0000", 2020-01-16T00:13:03.161 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 16.394674464000001, 2020-01-16T00:13:03.162 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015636654, 2020-01-16T00:13:03.162 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:03.162 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:03.162 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:03.162 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:03.162 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:03.162 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1394 2020-01-16T00:13:03.162 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.163 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:03.163 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.163 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:03.163 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625161+0000", 2020-01-16T00:13:03.163 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:03.163 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.163 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.164 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:03.164 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625161+0000", 2020-01-16T00:13:03.164 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:03.164 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.164 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.164 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:03.164 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625160+0000", 2020-01-16T00:13:03.164 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3960000000000002e-06 2020-01-16T00:13:03.165 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.165 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.165 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:03.165 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625164+0000", 2020-01-16T00:13:03.165 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7199999999999997e-06 2020-01-16T00:13:03.165 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.165 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.165 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:03.166 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625168+0000", 2020-01-16T00:13:03.166 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2960000000000001e-06 2020-01-16T00:13:03.166 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.166 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.166 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:03.166 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625171+0000", 2020-01-16T00:13:03.166 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001705972 2020-01-16T00:13:03.166 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.167 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.167 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.167 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.626877+0000", 2020-01-16T00:13:03.167 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039484300000000002 2020-01-16T00:13:03.167 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.167 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.167 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:03.168 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630825+0000", 2020-01-16T00:13:03.168 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0068775310000000001 2020-01-16T00:13:03.168 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.168 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.168 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.168 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.637703+0000", 2020-01-16T00:13:03.168 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013884100000000001 2020-01-16T00:13:03.168 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.169 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.169 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:03.169 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.637842+0000", 2020-01-16T00:13:03.169 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00024336400000000001 2020-01-16T00:13:03.169 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.169 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.169 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:13:03.169 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.638085+0000", 2020-01-16T00:13:03.170 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010183900000000001 2020-01-16T00:13:03.170 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.170 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.170 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:03.170 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.639103+0000", 2020-01-16T00:13:03.170 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012246699999999999 2020-01-16T00:13:03.170 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.170 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.171 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:03.171 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.639226+0000", 2020-01-16T00:13:03.171 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0014329099999999999 2020-01-16T00:13:03.171 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.171 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.171 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:03.171 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.640659+0000", 2020-01-16T00:13:03.172 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8109000000000002e-05 2020-01-16T00:13:03.172 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.172 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.172 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:03.172 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.640717+0000", 2020-01-16T00:13:03.172 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0784000000000004e-05 2020-01-16T00:13:03.172 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.172 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.173 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:03.173 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.640798+0000", 2020-01-16T00:13:03.173 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015636654 2020-01-16T00:13:03.173 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.173 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:03.173 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.173 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.173 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.174 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1395 3.4 3:240c7503:::smithi12313934-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e28)", 2020-01-16T00:13:03.174 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:46.625191+0000", 2020-01-16T00:13:03.174 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 16.394644253999999, 2020-01-16T00:13:03.174 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015878716000000001, 2020-01-16T00:13:03.174 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:03.174 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:03.174 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:03.175 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:03.175 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:03.175 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1395 2020-01-16T00:13:03.175 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.175 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:03.175 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.175 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:03.175 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625191+0000", 2020-01-16T00:13:03.176 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:03.176 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.176 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.176 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:03.176 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625191+0000", 2020-01-16T00:13:03.176 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:03.176 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.176 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.177 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:03.177 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625191+0000", 2020-01-16T00:13:03.177 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.1090000000000001e-06 2020-01-16T00:13:03.177 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.177 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.177 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:03.177 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625193+0000", 2020-01-16T00:13:03.178 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2730000000000008e-06 2020-01-16T00:13:03.178 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.178 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.178 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:03.178 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625202+0000", 2020-01-16T00:13:03.178 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3700000000000002e-06 2020-01-16T00:13:03.178 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.178 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.179 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:03.179 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625204+0000", 2020-01-16T00:13:03.179 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056573049999999996 2020-01-16T00:13:03.179 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.179 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.179 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.179 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630862+0000", 2020-01-16T00:13:03.180 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6047000000000001e-05 2020-01-16T00:13:03.180 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.180 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.180 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:03.180 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630918+0000", 2020-01-16T00:13:03.180 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0077941180000000001 2020-01-16T00:13:03.180 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.180 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.181 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.181 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.638712+0000", 2020-01-16T00:13:03.181 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118096 2020-01-16T00:13:03.181 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.181 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.181 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:03.181 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.638830+0000", 2020-01-16T00:13:03.181 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0019986740000000002 2020-01-16T00:13:03.182 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.182 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.182 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.182 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.640829+0000", 2020-01-16T00:13:03.182 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012562399999999999 2020-01-16T00:13:03.182 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.182 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.183 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:03.183 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.640954+0000", 2020-01-16T00:13:03.183 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000115536 2020-01-16T00:13:03.183 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.183 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.183 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:03.184 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.641070+0000", 2020-01-16T00:13:03.184 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015878716000000001 2020-01-16T00:13:03.184 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.184 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:03.184 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.184 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.184 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.184 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1638 3.0 3:0b70bb20:::smithi12313934-46:head [read 0~3630279,checksum] snapc 0=[] ondisk+read+known_if_redirected e29)", 2020-01-16T00:13:03.185 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.760082+0000", 2020-01-16T00:13:03.185 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 11.259752901000001, 2020-01-16T00:13:03.185 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014248647, 2020-01-16T00:13:03.185 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:03.185 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:03.185 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:03.185 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:03.185 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:03.186 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1638 2020-01-16T00:13:03.186 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.186 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:03.186 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.186 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:03.186 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.760082+0000", 2020-01-16T00:13:03.186 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:03.186 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.187 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.187 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:03.187 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.760082+0000", 2020-01-16T00:13:03.187 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:03.187 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.187 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.187 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:03.188 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.760082+0000", 2020-01-16T00:13:03.188 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2129999999999999e-06 2020-01-16T00:13:03.188 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.188 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.188 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:03.188 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.760087+0000", 2020-01-16T00:13:03.189 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8340000000000001e-06 2020-01-16T00:13:03.189 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.189 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.189 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:03.189 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.760094+0000", 2020-01-16T00:13:03.189 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6599999999999998e-06 2020-01-16T00:13:03.189 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.189 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.190 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:03.190 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.760100+0000", 2020-01-16T00:13:03.190 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012816863 2020-01-16T00:13:03.190 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.190 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.190 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.190 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.772917+0000", 2020-01-16T00:13:03.191 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015354600000000001 2020-01-16T00:13:03.191 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.191 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.191 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:03.191 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.773071+0000", 2020-01-16T00:13:03.191 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012604140000000001 2020-01-16T00:13:03.191 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.191 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.192 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:03.192 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774331+0000", 2020-01-16T00:13:03.192 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014248647 2020-01-16T00:13:03.192 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.192 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:03.192 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.192 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.192 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.193 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1639 3.0 3:00c146d5:::smithi12313934-18:head [write 423576~401598] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:13:03.193 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.774317+0000", 2020-01-16T00:13:03.193 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 11.245518842999999, 2020-01-16T00:13:03.193 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018305219000000001, 2020-01-16T00:13:03.193 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:03.193 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:03.193 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:03.193 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:03.194 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:03.194 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1639 2020-01-16T00:13:03.194 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.194 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:03.194 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.194 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:03.194 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774317+0000", 2020-01-16T00:13:03.195 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:03.195 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.195 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.195 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:03.195 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774317+0000", 2020-01-16T00:13:03.195 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:03.195 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.195 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.196 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:03.196 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774313+0000", 2020-01-16T00:13:03.196 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000111173 2020-01-16T00:13:03.196 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.196 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.196 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:03.196 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774425+0000", 2020-01-16T00:13:03.196 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7598999999999999e-05 2020-01-16T00:13:03.197 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.197 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.197 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:03.197 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774442+0000", 2020-01-16T00:13:03.197 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0830000000000004e-06 2020-01-16T00:13:03.197 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.197 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.197 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:03.198 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774450+0000", 2020-01-16T00:13:03.198 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0017886309999999999 2020-01-16T00:13:03.198 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.198 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.199 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.200 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.776239+0000", 2020-01-16T00:13:03.200 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3789e-05 2020-01-16T00:13:03.200 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.200 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.200 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op must wait for map", 2020-01-16T00:13:03.200 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.776253+0000", 2020-01-16T00:13:03.200 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095730729999999997 2020-01-16T00:13:03.200 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.201 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.201 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.201 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.785826+0000", 2020-01-16T00:13:03.201 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014220300000000001 2020-01-16T00:13:03.202 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.202 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.202 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:03.202 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.785968+0000", 2020-01-16T00:13:03.202 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023618900000000001 2020-01-16T00:13:03.202 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.202 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.203 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:03.203 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.786204+0000", 2020-01-16T00:13:03.203 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0062297389999999998 2020-01-16T00:13:03.203 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.203 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.203 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:03.203 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792434+0000", 2020-01-16T00:13:03.203 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9635999999999998e-05 2020-01-16T00:13:03.204 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.204 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.204 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:03.204 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792473+0000", 2020-01-16T00:13:03.204 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6456000000000002e-05 2020-01-16T00:13:03.204 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.204 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.205 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:03.205 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792540+0000", 2020-01-16T00:13:03.205 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9873999999999998e-05 2020-01-16T00:13:03.205 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.205 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.205 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:03.206 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792580+0000", 2020-01-16T00:13:03.206 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1924e-05 2020-01-16T00:13:03.206 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.206 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.206 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:03.206 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792622+0000", 2020-01-16T00:13:03.207 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018305219000000001 2020-01-16T00:13:03.207 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.207 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:03.207 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.207 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.207 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.207 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1890 3.0 3:0b1f384f:::smithi12313934-33:head [write 2797915~596476] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:13:03.207 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.232534+0000", 2020-01-16T00:13:03.208 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.7873008720000003, 2020-01-16T00:13:03.208 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016106137, 2020-01-16T00:13:03.208 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:03.208 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:03.208 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:03.208 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:03.208 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:03.209 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1890 2020-01-16T00:13:03.209 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.209 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:03.209 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.209 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:03.209 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232534+0000", 2020-01-16T00:13:03.209 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:03.210 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.210 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.210 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:03.210 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232534+0000", 2020-01-16T00:13:03.210 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:03.210 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.210 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.210 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:03.211 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232532+0000", 2020-01-16T00:13:03.211 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00024232100000000001 2020-01-16T00:13:03.211 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.211 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.211 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:03.211 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232774+0000", 2020-01-16T00:13:03.212 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4601e-05 2020-01-16T00:13:03.212 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.212 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.212 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:03.212 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232789+0000", 2020-01-16T00:13:03.212 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8410000000000006e-06 2020-01-16T00:13:03.213 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.213 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.213 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:03.213 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232798+0000", 2020-01-16T00:13:03.213 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9756999999999997e-05 2020-01-16T00:13:03.213 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.213 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.214 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.214 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232857+0000", 2020-01-16T00:13:03.214 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4678000000000007e-05 2020-01-16T00:13:03.214 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.214 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.214 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:03.214 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232922+0000", 2020-01-16T00:13:03.214 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033861020000000002 2020-01-16T00:13:03.215 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.215 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.215 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.215 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.236308+0000", 2020-01-16T00:13:03.215 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000568996 2020-01-16T00:13:03.215 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.215 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.216 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:03.216 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.236877+0000", 2020-01-16T00:13:03.216 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0085444240000000001 2020-01-16T00:13:03.216 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.216 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.216 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.216 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.245422+0000", 2020-01-16T00:13:03.217 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5595999999999998e-05 2020-01-16T00:13:03.217 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.217 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.217 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:03.217 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.245507+0000", 2020-01-16T00:13:03.217 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016984300000000001 2020-01-16T00:13:03.218 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.218 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.218 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:03.218 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.245677+0000", 2020-01-16T00:13:03.218 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001649495 2020-01-16T00:13:03.218 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.218 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.219 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:03.219 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247326+0000", 2020-01-16T00:13:03.219 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023308399999999999 2020-01-16T00:13:03.219 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.219 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.219 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:03.220 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247560+0000", 2020-01-16T00:13:03.220 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00099501800000000007 2020-01-16T00:13:03.220 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.220 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.220 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:03.220 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248555+0000", 2020-01-16T00:13:03.220 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5936999999999998e-05 2020-01-16T00:13:03.220 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.221 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.221 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:03.221 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248591+0000", 2020-01-16T00:13:03.221 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0108999999999997e-05 2020-01-16T00:13:03.221 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.222 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.222 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:03.222 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248641+0000", 2020-01-16T00:13:03.222 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016106137 2020-01-16T00:13:03.222 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.222 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:03.223 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.223 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.223 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.223 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1891 3.0 3:0b1f384f:::smithi12313934-33:head [setxattr _header (56) in=7b,truncate 3395014] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:13:03.223 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.232813+0000", 2020-01-16T00:13:03.223 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.7870220699999999, 2020-01-16T00:13:03.223 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015999295, 2020-01-16T00:13:03.223 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:03.224 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:03.224 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:03.224 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:03.224 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:03.224 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1891 2020-01-16T00:13:03.224 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.224 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:03.225 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.225 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:03.225 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232813+0000", 2020-01-16T00:13:03.225 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:03.226 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.226 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.226 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:03.226 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232813+0000", 2020-01-16T00:13:03.226 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:03.226 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.226 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.227 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:03.227 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232812+0000", 2020-01-16T00:13:03.227 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.576e-06 2020-01-16T00:13:03.227 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.227 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.227 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:03.227 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232820+0000", 2020-01-16T00:13:03.227 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3079999999999997e-06 2020-01-16T00:13:03.228 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.228 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.228 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:03.228 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232829+0000", 2020-01-16T00:13:03.228 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7119999999999997e-06 2020-01-16T00:13:03.228 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.228 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.229 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:03.229 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232835+0000", 2020-01-16T00:13:03.229 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013438400000000001 2020-01-16T00:13:03.229 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.229 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.229 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.229 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232969+0000", 2020-01-16T00:13:03.229 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0562000000000001e-05 2020-01-16T00:13:03.230 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.230 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.230 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:03.230 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.233029+0000", 2020-01-16T00:13:03.230 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039354239999999999 2020-01-16T00:13:03.231 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.231 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.231 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.231 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.236965+0000", 2020-01-16T00:13:03.231 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0482999999999998e-05 2020-01-16T00:13:03.231 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.232 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.232 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:03.232 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.237015+0000", 2020-01-16T00:13:03.233 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0096086209999999995 2020-01-16T00:13:03.233 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.233 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.233 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.233 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.246624+0000", 2020-01-16T00:13:03.234 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6150999999999995e-05 2020-01-16T00:13:03.234 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.234 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.234 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:03.235 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.246720+0000", 2020-01-16T00:13:03.235 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015032899999999999 2020-01-16T00:13:03.235 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.235 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.235 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:03.236 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.246870+0000", 2020-01-16T00:13:03.236 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00064760700000000002 2020-01-16T00:13:03.236 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.236 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.237 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:03.237 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247518+0000", 2020-01-16T00:13:03.237 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000198531 2020-01-16T00:13:03.237 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.237 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.238 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:03.238 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247717+0000", 2020-01-16T00:13:03.238 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010189439999999999 2020-01-16T00:13:03.238 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.239 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.239 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:03.239 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248736+0000", 2020-01-16T00:13:03.239 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4863000000000001e-05 2020-01-16T00:13:03.240 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.240 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.240 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:03.240 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248770+0000", 2020-01-16T00:13:03.241 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2153000000000001e-05 2020-01-16T00:13:03.241 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.241 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.241 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:03.242 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248813+0000", 2020-01-16T00:13:03.242 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015999295 2020-01-16T00:13:03.242 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.243 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:03.243 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.243 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.243 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.244 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1892 3.0 3:0b1f384f:::smithi12313934-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e31)", 2020-01-16T00:13:03.244 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.232856+0000", 2020-01-16T00:13:03.244 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.7869798460000004, 2020-01-16T00:13:03.244 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016171544, 2020-01-16T00:13:03.245 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:03.245 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:03.245 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:03.245 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:03.246 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:03.246 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1892 2020-01-16T00:13:03.246 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.246 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:03.246 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.247 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:03.247 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232856+0000", 2020-01-16T00:13:03.247 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:03.247 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.248 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.248 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:03.248 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232856+0000", 2020-01-16T00:13:03.248 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:03.249 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.249 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.249 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:03.250 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232855+0000", 2020-01-16T00:13:03.250 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8640000000000004e-06 2020-01-16T00:13:03.250 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.251 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.251 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:03.251 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232860+0000", 2020-01-16T00:13:03.251 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0000000000000001e-05 2020-01-16T00:13:03.252 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.252 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.252 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:03.252 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232870+0000", 2020-01-16T00:13:03.252 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7420000000000003e-06 2020-01-16T00:13:03.253 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.253 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.253 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:03.254 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232875+0000", 2020-01-16T00:13:03.254 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017933200000000001 2020-01-16T00:13:03.254 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.254 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.254 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.255 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.233055+0000", 2020-01-16T00:13:03.255 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6269e-05 2020-01-16T00:13:03.255 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.255 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.255 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:03.256 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.233101+0000", 2020-01-16T00:13:03.256 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039780090000000002 2020-01-16T00:13:03.256 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.256 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.256 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.257 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.237079+0000", 2020-01-16T00:13:03.257 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3677000000000001e-05 2020-01-16T00:13:03.257 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.257 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.257 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:03.258 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.237133+0000", 2020-01-16T00:13:03.258 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010036671 2020-01-16T00:13:03.258 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.258 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.258 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.258 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247169+0000", 2020-01-16T00:13:03.259 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010280699999999999 2020-01-16T00:13:03.259 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.259 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.259 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:03.259 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247272+0000", 2020-01-16T00:13:03.259 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001579708 2020-01-16T00:13:03.260 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.260 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.260 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.260 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248852+0000", 2020-01-16T00:13:03.260 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9704999999999996e-05 2020-01-16T00:13:03.260 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.260 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.260 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:03.261 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248942+0000", 2020-01-16T00:13:03.261 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5557999999999996e-05 2020-01-16T00:13:03.261 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.261 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.261 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:03.261 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.249027+0000", 2020-01-16T00:13:03.261 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016171544 2020-01-16T00:13:03.262 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.262 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:03.262 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.262 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.262 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.262 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1950 3.0 3:0b1f384f:::smithi12313934-33:head [write 1889359~541737] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-01-16T00:13:03.262 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.922819+0000", 2020-01-16T00:13:03.263 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.0970164809999998, 2020-01-16T00:13:03.263 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017502846999999998, 2020-01-16T00:13:03.263 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:03.263 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:03.263 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:03.263 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:03.263 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:03.264 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1950 2020-01-16T00:13:03.264 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.264 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:03.264 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.264 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:03.264 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922819+0000", 2020-01-16T00:13:03.264 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:03.265 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.265 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.265 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:03.265 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922819+0000", 2020-01-16T00:13:03.265 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:03.265 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.265 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.265 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:03.266 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922818+0000", 2020-01-16T00:13:03.266 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000146408 2020-01-16T00:13:03.266 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.266 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.266 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:03.266 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922964+0000", 2020-01-16T00:13:03.267 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4720000000000001e-06 2020-01-16T00:13:03.267 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.267 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.267 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:03.268 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922974+0000", 2020-01-16T00:13:03.268 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0720000000000001e-06 2020-01-16T00:13:03.268 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.268 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.268 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:03.268 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922978+0000", 2020-01-16T00:13:03.268 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6152000000000001e-05 2020-01-16T00:13:03.269 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.269 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.269 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.269 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.923024+0000", 2020-01-16T00:13:03.269 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00087368199999999997 2020-01-16T00:13:03.269 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.269 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.270 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:03.270 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.923897+0000", 2020-01-16T00:13:03.270 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011944435999999999 2020-01-16T00:13:03.270 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.270 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.270 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.270 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.935842+0000", 2020-01-16T00:13:03.271 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000144146 2020-01-16T00:13:03.271 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.271 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.271 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:03.271 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.935986+0000", 2020-01-16T00:13:03.271 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019775500000000001 2020-01-16T00:13:03.271 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.272 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.272 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:03.272 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.936184+0000", 2020-01-16T00:13:03.272 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002539342 2020-01-16T00:13:03.272 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.272 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.272 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:03.272 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938723+0000", 2020-01-16T00:13:03.273 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014193699999999999 2020-01-16T00:13:03.273 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.273 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.273 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:03.273 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938865+0000", 2020-01-16T00:13:03.273 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013829020000000001 2020-01-16T00:13:03.274 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.274 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.274 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:03.274 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.940248+0000", 2020-01-16T00:13:03.274 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2558000000000001e-05 2020-01-16T00:13:03.274 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.274 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.275 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:03.275 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.940291+0000", 2020-01-16T00:13:03.275 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1217999999999998e-05 2020-01-16T00:13:03.275 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.275 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.275 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:03.275 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.940322+0000", 2020-01-16T00:13:03.276 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017502846999999998 2020-01-16T00:13:03.276 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.276 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:03.276 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.276 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.276 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.276 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1951 3.0 3:0b1f384f:::smithi12313934-33:head [write 2955792~541902] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-01-16T00:13:03.277 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.929019+0000", 2020-01-16T00:13:03.277 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.0908163310000001, 2020-01-16T00:13:03.277 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021625710999999999, 2020-01-16T00:13:03.277 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:03.277 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:03.278 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:03.278 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:03.279 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:03.279 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1951 2020-01-16T00:13:03.279 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.279 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:03.279 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.279 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:03.279 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929019+0000", 2020-01-16T00:13:03.280 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:03.280 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.280 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.280 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:03.280 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929019+0000", 2020-01-16T00:13:03.280 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999989 2020-01-16T00:13:03.280 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.280 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.281 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:03.281 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929008+0000", 2020-01-16T00:13:03.281 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000212513 2020-01-16T00:13:03.281 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.281 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.281 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:03.281 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929221+0000", 2020-01-16T00:13:03.281 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3229e-05 2020-01-16T00:13:03.282 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.282 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.282 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:03.282 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929234+0000", 2020-01-16T00:13:03.282 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9170000000000003e-06 2020-01-16T00:13:03.282 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.282 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.282 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:03.283 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929240+0000", 2020-01-16T00:13:03.283 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4045000000000002e-05 2020-01-16T00:13:03.283 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.283 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.283 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.283 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929294+0000", 2020-01-16T00:13:03.283 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9797999999999996e-05 2020-01-16T00:13:03.283 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.284 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.284 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:03.284 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929384+0000", 2020-01-16T00:13:03.284 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083450729999999997 2020-01-16T00:13:03.284 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.284 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.284 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.284 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.937729+0000", 2020-01-16T00:13:03.285 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043251699999999999 2020-01-16T00:13:03.285 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.285 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.285 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:03.285 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938161+0000", 2020-01-16T00:13:03.285 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095115489999999994 2020-01-16T00:13:03.285 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.286 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.286 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.286 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.947673+0000", 2020-01-16T00:13:03.286 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012395099999999999 2020-01-16T00:13:03.286 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.286 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.286 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:03.287 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.947797+0000", 2020-01-16T00:13:03.287 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00024849200000000001 2020-01-16T00:13:03.287 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.287 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.287 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:03.287 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.948045+0000", 2020-01-16T00:13:03.287 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0025081050000000001 2020-01-16T00:13:03.287 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.288 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.288 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:03.288 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950553+0000", 2020-01-16T00:13:03.288 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9760000000000003e-06 2020-01-16T00:13:03.288 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.288 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.288 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:03.289 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950558+0000", 2020-01-16T00:13:03.289 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3286999999999999e-05 2020-01-16T00:13:03.289 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.289 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.289 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:03.289 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950581+0000", 2020-01-16T00:13:03.289 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2759000000000003e-05 2020-01-16T00:13:03.289 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.290 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.290 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:03.290 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950614+0000", 2020-01-16T00:13:03.290 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0525999999999999e-05 2020-01-16T00:13:03.290 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.290 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.290 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:03.290 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950645+0000", 2020-01-16T00:13:03.291 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021625710999999999 2020-01-16T00:13:03.291 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.291 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:03.291 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.291 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.291 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.291 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1952 3.0 3:0b1f384f:::smithi12313934-33:head [setxattr _header (56) in=7b,truncate 3563084] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-01-16T00:13:03.291 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:13:03.292 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.0905869939999997, 2020-01-16T00:13:03.292 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021534685000000001, 2020-01-16T00:13:03.292 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:03.292 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:03.292 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:03.292 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:03.292 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:03.292 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1952 2020-01-16T00:13:03.293 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.293 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:03.293 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.293 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:03.293 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:13:03.293 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:03.293 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.293 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.294 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:03.294 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:13:03.294 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:03.294 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.294 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.294 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:03.294 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:13:03.294 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6670000000000004e-06 2020-01-16T00:13:03.295 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.295 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.295 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:03.295 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929252+0000", 2020-01-16T00:13:03.295 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9380000000000002e-06 2020-01-16T00:13:03.295 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.295 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.296 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:03.296 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929256+0000", 2020-01-16T00:13:03.296 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.712e-06 2020-01-16T00:13:03.296 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.296 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.296 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:03.296 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929264+0000", 2020-01-16T00:13:03.296 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018030300000000001 2020-01-16T00:13:03.297 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.297 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.297 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.297 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929444+0000", 2020-01-16T00:13:03.297 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7562000000000002e-05 2020-01-16T00:13:03.297 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.297 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.298 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:03.298 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929502+0000", 2020-01-16T00:13:03.298 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0087297859999999998 2020-01-16T00:13:03.298 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.298 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.298 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.298 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938232+0000", 2020-01-16T00:13:03.298 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0452999999999996e-05 2020-01-16T00:13:03.299 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.299 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.299 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:03.299 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938322+0000", 2020-01-16T00:13:03.299 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011061728 2020-01-16T00:13:03.299 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.299 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.299 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.300 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.949384+0000", 2020-01-16T00:13:03.300 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000117828 2020-01-16T00:13:03.300 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.300 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.300 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:03.300 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.949502+0000", 2020-01-16T00:13:03.300 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022468499999999999 2020-01-16T00:13:03.300 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.301 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.301 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:03.301 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.949726+0000", 2020-01-16T00:13:03.301 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00083532099999999998 2020-01-16T00:13:03.301 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.301 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.301 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:03.301 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950562+0000", 2020-01-16T00:13:03.302 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013017700000000001 2020-01-16T00:13:03.302 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.302 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.302 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:03.302 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950692+0000", 2020-01-16T00:13:03.302 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9813000000000001e-05 2020-01-16T00:13:03.302 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.302 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.303 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:03.303 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950722+0000", 2020-01-16T00:13:03.303 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0592000000000001e-05 2020-01-16T00:13:03.303 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.303 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.303 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:03.303 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950752+0000", 2020-01-16T00:13:03.303 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0920000000000002e-05 2020-01-16T00:13:03.304 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.304 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.304 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:03.304 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950783+0000", 2020-01-16T00:13:03.304 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021534685000000001 2020-01-16T00:13:03.304 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.304 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:03.305 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.305 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.305 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.305 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1953 3.0 3:0b1f384f:::smithi12313934-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e32)", 2020-01-16T00:13:03.305 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:13:03.305 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.0905540030000003, 2020-01-16T00:13:03.305 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021667537000000001, 2020-01-16T00:13:03.306 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:03.306 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:03.306 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:03.306 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:03.306 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:03.306 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1953 2020-01-16T00:13:03.306 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.306 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:03.307 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.307 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:03.307 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:13:03.307 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:03.307 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.307 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.307 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:03.307 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:13:03.308 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:03.308 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.308 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.308 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:03.308 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:13:03.308 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7030000000000002e-06 2020-01-16T00:13:03.308 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.308 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.309 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:03.309 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929284+0000", 2020-01-16T00:13:03.309 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0909999999999993e-06 2020-01-16T00:13:03.309 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.309 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.309 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:03.309 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929292+0000", 2020-01-16T00:13:03.309 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8359999999999999e-06 2020-01-16T00:13:03.310 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.310 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.310 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:03.310 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929297+0000", 2020-01-16T00:13:03.310 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023538199999999999 2020-01-16T00:13:03.310 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.310 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.310 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.311 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929532+0000", 2020-01-16T00:13:03.311 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1173000000000004e-05 2020-01-16T00:13:03.311 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.311 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.311 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:03.311 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929603+0000", 2020-01-16T00:13:03.311 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0087813509999999997 2020-01-16T00:13:03.311 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.312 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.312 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.312 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938384+0000", 2020-01-16T00:13:03.312 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9752000000000006e-05 2020-01-16T00:13:03.312 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.312 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.312 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:03.312 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938454+0000", 2020-01-16T00:13:03.313 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011707851999999999 2020-01-16T00:13:03.313 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.313 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.313 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.313 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950162+0000", 2020-01-16T00:13:03.313 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011060099999999999 2020-01-16T00:13:03.313 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.313 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.314 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:03.314 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950273+0000", 2020-01-16T00:13:03.314 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00053989700000000001 2020-01-16T00:13:03.314 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.314 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.314 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:03.314 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950813+0000", 2020-01-16T00:13:03.314 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8128999999999999e-05 2020-01-16T00:13:03.315 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.315 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.315 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:03.315 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950871+0000", 2020-01-16T00:13:03.315 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8242000000000001e-05 2020-01-16T00:13:03.315 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:03.315 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:03.315 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:03.316 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950949+0000", 2020-01-16T00:13:03.316 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021667537000000001 2020-01-16T00:13:03.316 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.316 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:03.316 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.316 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:03.316 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:03.317 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:03.317 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2020-01-16T00:13:03.318 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-01-16T00:13:03.321 INFO:tasks.rados.rados.0.smithi123.stdout:981: expect (ObjNum 188 snap 0 seq_num 188) 2020-01-16T00:13:03.321 INFO:tasks.rados.rados.0.smithi123.stdout:977: done (15 left) 2020-01-16T00:13:03.322 INFO:tasks.rados.rados.0.smithi123.stdout:978: done (14 left) 2020-01-16T00:13:03.322 INFO:tasks.rados.rados.0.smithi123.stdout:979: done (13 left) 2020-01-16T00:13:03.322 INFO:tasks.rados.rados.0.smithi123.stdout:980: done (12 left) 2020-01-16T00:13:03.322 INFO:tasks.rados.rados.0.smithi123.stdout:981: done (11 left) 2020-01-16T00:13:03.322 INFO:tasks.rados.rados.0.smithi123.stdout:982: done (10 left) 2020-01-16T00:13:03.322 INFO:tasks.rados.rados.0.smithi123.stdout:993: read oid 47 snap -1 2020-01-16T00:13:03.322 INFO:tasks.rados.rados.0.smithi123.stdout:993: expect (ObjNum 133 snap 0 seq_num 133) 2020-01-16T00:13:03.322 INFO:tasks.rados.rados.0.smithi123.stdout:994: write (excl) oid 5 current snap is 0 2020-01-16T00:13:03.323 INFO:tasks.rados.rados.0.smithi123.stdout:994: seq_num 238 ranges {400066=716205,1860832=139234} 2020-01-16T00:13:03.323 INFO:tasks.rados.rados.0.smithi123.stdout:994: writing smithi12313934-5 from 400066 to 1116271 tid 1 2020-01-16T00:13:03.323 INFO:tasks.rados.rados.0.smithi123.stdout:994: writing smithi12313934-5 from 1860832 to 2000066 tid 2 2020-01-16T00:13:03.323 INFO:tasks.rados.rados.0.smithi123.stdout:995: write (excl) oid 46 current snap is 0 2020-01-16T00:13:03.323 INFO:tasks.rados.rados.0.smithi123.stdout:995: seq_num 239 ranges {416873=791454,1919314=97559} 2020-01-16T00:13:03.323 INFO:tasks.rados.rados.0.smithi123.stdout:995: writing smithi12313934-46 from 416873 to 1208327 tid 1 2020-01-16T00:13:03.323 INFO:tasks.rados.rados.0.smithi123.stdout:995: writing smithi12313934-46 from 1919314 to 2016873 tid 2 2020-01-16T00:13:03.324 INFO:tasks.rados.rados.0.smithi123.stdout:996: write (excl) oid 1 current snap is 0 2020-01-16T00:13:03.324 INFO:tasks.rados.rados.0.smithi123.stdout:996: seq_num 240 ranges {433680=466703,1577796=455884} 2020-01-16T00:13:03.324 INFO:tasks.rados.rados.0.smithi123.stdout:996: writing smithi12313934-1 from 433680 to 900383 tid 1 2020-01-16T00:13:03.324 INFO:tasks.rados.rados.0.smithi123.stdout:996: writing smithi12313934-1 from 1577796 to 2033680 tid 2 2020-01-16T00:13:03.324 INFO:tasks.rados.rados.0.smithi123.stdout:984: finishing write tid 1 to smithi12313934-9 2020-01-16T00:13:03.324 INFO:tasks.rados.rados.0.smithi123.stdout:983: expect (ObjNum 194 snap 0 seq_num 194) 2020-01-16T00:13:03.324 INFO:teuthology.orchestra.run.smithi123.stderr:set osd.1 primary-affinity to 1 (8655362) 2020-01-16T00:13:03.325 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:13:03.318+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:13:03.325 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-01-16T00:13:03.419 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:03.414+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:03.464 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:03.464 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:03.464 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:13:03.464 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:03.473 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-01-16T00:13:03.478 INFO:tasks.rados.rados.0.smithi123.stdout:984: finishing write tid 2 to smithi12313934-9 2020-01-16T00:13:03.478 INFO:tasks.rados.rados.0.smithi123.stdout:985: expect (ObjNum 200 snap 0 seq_num 200) 2020-01-16T00:13:03.485 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:13:03.511 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:13:03.520 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:03.514+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:03.540 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:13:03.562 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:13:03.582 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:03.585 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:13:03.591 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-01-16T00:13:03.611 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5: {} 2020-01-16T00:13:03.635 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:13:03.635 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:03.630+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:03.662 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:13:03.663 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-01-16T00:13:03.663 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-01-16T00:13:03.664 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-01-16T00:13:03.664 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-01-16T00:13:03.664 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-01-16T00:13:03.664 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-01-16T00:13:03.664 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-01-16T00:13:03.664 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-01-16T00:13:03.715 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:03.722 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:03.718+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:03.724 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-01-16T00:13:03.773 INFO:tasks.rados.rados.0.smithi123.stdout:984: finishing write tid 3 to smithi12313934-9 2020-01-16T00:13:03.773 INFO:tasks.rados.rados.0.smithi123.stdout:984: finishing write tid 5 to smithi12313934-9 2020-01-16T00:13:03.773 INFO:tasks.rados.rados.0.smithi123.stdout:984: finishing write tid 6 to smithi12313934-9 2020-01-16T00:13:03.774 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 9 v 83 (ObjNum 236 snap 0 seq_num 236) dirty exists 2020-01-16T00:13:03.774 INFO:tasks.rados.rados.0.smithi123.stdout:984: left oid 9 (ObjNum 236 snap 0 seq_num 236) 2020-01-16T00:13:03.774 INFO:tasks.rados.rados.0.smithi123.stdout:997: read oid 20 snap -1 2020-01-16T00:13:03.774 INFO:tasks.rados.rados.0.smithi123.stdout:997: expect (ObjNum 218 snap 0 seq_num 218) 2020-01-16T00:13:03.774 INFO:tasks.rados.rados.0.smithi123.stdout:983: done (14 left) 2020-01-16T00:13:03.774 INFO:tasks.rados.rados.0.smithi123.stdout:984: done (13 left) 2020-01-16T00:13:03.774 INFO:tasks.rados.rados.0.smithi123.stdout:985: done (12 left) 2020-01-16T00:13:03.775 INFO:tasks.rados.rados.0.smithi123.stdout:998: write oid 2 current snap is 0 2020-01-16T00:13:03.775 INFO:tasks.rados.rados.0.smithi123.stdout:998: seq_num 241 ranges {450487=541952,1636278=414209} 2020-01-16T00:13:03.786 INFO:tasks.rados.rados.0.smithi123.stdout:998: writing smithi12313934-2 from 450487 to 992439 tid 1 2020-01-16T00:13:03.798 INFO:tasks.rados.rados.0.smithi123.stdout:998: writing smithi12313934-2 from 1636278 to 2050487 tid 2 2020-01-16T00:13:03.798 INFO:tasks.rados.rados.0.smithi123.stdout:999: read oid 42 snap -1 2020-01-16T00:13:03.799 INFO:tasks.rados.rados.0.smithi123.stdout:999: expect (ObjNum 215 snap 0 seq_num 215) 2020-01-16T00:13:03.799 INFO:tasks.rados.rados.0.smithi123.stdout:1000: read oid 49 snap -1 2020-01-16T00:13:03.799 INFO:tasks.rados.rados.0.smithi123.stdout:1000: expect (ObjNum 196 snap 0 seq_num 196) 2020-01-16T00:13:03.799 INFO:tasks.rados.rados.0.smithi123.stdout:1001: read oid 41 snap -1 2020-01-16T00:13:03.799 INFO:tasks.rados.rados.0.smithi123.stdout:1001: expect (ObjNum 193 snap 0 seq_num 193) 2020-01-16T00:13:03.799 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:03.799 INFO:tasks.rados.rados.0.smithi123.stdout:987: expect (ObjNum 230 snap 0 seq_num 230) 2020-01-16T00:13:03.823 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:03.818+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:03.874 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:03.883 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-01-16T00:13:03.925 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:03.922+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:04.025 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:04.022+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:04.031 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:04.041 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-01-16T00:13:04.064 INFO:tasks.rados.rados.0.smithi123.stdout:988: finishing write tid 1 to smithi12313934-17 2020-01-16T00:13:04.064 INFO:tasks.rados.rados.0.smithi123.stdout:986: expect (ObjNum 224 snap 0 seq_num 224) 2020-01-16T00:13:04.126 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:04.122+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:04.142 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:04.150 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-01-16T00:13:04.227 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:13:04.222+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:13:04.238 INFO:teuthology.orchestra.run.smithi123.stderr:nodeep-scrub is unset 2020-01-16T00:13:04.276 INFO:tasks.rados.rados.0.smithi123.stdout:988: finishing write tid 2 to smithi12313934-17 2020-01-16T00:13:04.276 INFO:tasks.rados.rados.0.smithi123.stdout:988: finishing write tid 3 to smithi12313934-17 2020-01-16T00:13:04.276 INFO:tasks.rados.rados.0.smithi123.stdout:988: finishing write tid 4 to smithi12313934-17 2020-01-16T00:13:04.276 INFO:tasks.rados.rados.0.smithi123.stdout:986: done (15 left) 2020-01-16T00:13:04.276 INFO:tasks.rados.rados.0.smithi123.stdout:987: done (14 left) 2020-01-16T00:13:04.277 INFO:tasks.rados.rados.0.smithi123.stdout:1002: write (excl) oid 28 current snap is 0 2020-01-16T00:13:04.277 INFO:tasks.rados.rados.0.smithi123.stdout:1002: seq_num 242 ranges {467294=617201,1578407=488887} 2020-01-16T00:13:04.289 INFO:tasks.rados.rados.0.smithi123.stdout:1002: writing smithi12313934-28 from 467294 to 1084495 tid 1 2020-01-16T00:13:04.297 INFO:tasks.rados.rados.0.smithi123.stdout:1002: writing smithi12313934-28 from 1578407 to 2067294 tid 2 2020-01-16T00:13:04.297 INFO:tasks.rados.rados.0.smithi123.stdout:1003: write oid 32 current snap is 0 2020-01-16T00:13:04.297 INFO:tasks.rados.rados.0.smithi123.stdout:1003: seq_num 243 ranges {484101=692450,1636889=447212} 2020-01-16T00:13:04.301 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:04.303 INFO:tasks.rados.rados.0.smithi123.stdout:1003: writing smithi12313934-32 from 484101 to 1176551 tid 1 2020-01-16T00:13:04.309 INFO:tasks.rados.rados.0.smithi123.stdout:1003: writing smithi12313934-32 from 1636889 to 2084101 tid 2 2020-01-16T00:13:04.309 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:04.309 INFO:tasks.rados.rados.0.smithi123.stdout:988: finishing write tid 6 to smithi12313934-17 2020-01-16T00:13:04.309 INFO:tasks.rados.rados.0.smithi123.stdout:988: finishing write tid 7 to smithi12313934-17 2020-01-16T00:13:04.309 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 17 v 107 (ObjNum 237 snap 0 seq_num 237) dirty exists 2020-01-16T00:13:04.309 INFO:tasks.rados.rados.0.smithi123.stdout:988: left oid 17 (ObjNum 237 snap 0 seq_num 237) 2020-01-16T00:13:04.309 INFO:tasks.rados.rados.0.smithi123.stdout:989: expect (ObjNum 206 snap 0 seq_num 206) 2020-01-16T00:13:04.312 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-01-16T00:13:04.328 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:04.326+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:04.430 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:04.426+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:04.448 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:04.455 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-01-16T00:13:04.468 INFO:tasks.rados.rados.0.smithi123.stdout:991: expect (ObjNum 189 snap 0 seq_num 189) 2020-01-16T00:13:04.532 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:04.530+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:04.579 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:04.588 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-01-16T00:13:04.634 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:13:04.634+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:13:04.681 INFO:tasks.rados.rados.0.smithi123.stdout:990: expect (ObjNum 60 snap 0 seq_num 60) 2020-01-16T00:13:04.718 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:04.728 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-01-16T00:13:04.733 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:04.730+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:04.804 INFO:tasks.rados.rados.0.smithi123.stdout:992: expect (ObjNum 217 snap 0 seq_num 217) 2020-01-16T00:13:04.834 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:04.830+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:04.881 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:04.891 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-01-16T00:13:04.940 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:04.938+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:05.006 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:05.017 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-01-16T00:13:05.041 INFO:tasks.rados.rados.0.smithi123.stdout:993: expect (ObjNum 133 snap 0 seq_num 133) 2020-01-16T00:13:05.060 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:05.058+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:05.128 INFO:tasks.rados.rados.0.smithi123.stdout:994: finishing write tid 1 to smithi12313934-5 2020-01-16T00:13:05.128 INFO:tasks.rados.rados.0.smithi123.stdout:994: finishing write tid 2 to smithi12313934-5 2020-01-16T00:13:05.129 INFO:tasks.rados.rados.0.smithi123.stdout:994: finishing write tid 4 to smithi12313934-5 2020-01-16T00:13:05.129 INFO:tasks.rados.rados.0.smithi123.stdout:994: finishing write tid 5 to smithi12313934-5 2020-01-16T00:13:05.129 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 5 v 141 (ObjNum 238 snap 0 seq_num 238) dirty exists 2020-01-16T00:13:05.129 INFO:tasks.rados.rados.0.smithi123.stdout:994: left oid 5 (ObjNum 238 snap 0 seq_num 238) 2020-01-16T00:13:05.129 INFO:tasks.rados.rados.0.smithi123.stdout:995: finishing write tid 1 to smithi12313934-46 2020-01-16T00:13:05.129 INFO:tasks.rados.rados.0.smithi123.stdout:995: finishing write tid 2 to smithi12313934-46 2020-01-16T00:13:05.129 INFO:tasks.rados.rados.0.smithi123.stdout:996: finishing write tid 1 to smithi12313934-1 2020-01-16T00:13:05.130 INFO:tasks.rados.rados.0.smithi123.stdout:995: finishing write tid 4 to smithi12313934-46 2020-01-16T00:13:05.130 INFO:tasks.rados.rados.0.smithi123.stdout:995: finishing write tid 5 to smithi12313934-46 2020-01-16T00:13:05.130 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 46 v 198 (ObjNum 239 snap 0 seq_num 239) dirty exists 2020-01-16T00:13:05.130 INFO:tasks.rados.rados.0.smithi123.stdout:995: left oid 46 (ObjNum 239 snap 0 seq_num 239) 2020-01-16T00:13:05.130 INFO:tasks.rados.rados.0.smithi123.stdout:996: finishing write tid 2 to smithi12313934-1 2020-01-16T00:13:05.131 INFO:tasks.rados.rados.0.smithi123.stdout:996: finishing write tid 4 to smithi12313934-1 2020-01-16T00:13:05.131 INFO:tasks.rados.rados.0.smithi123.stdout:996: finishing write tid 5 to smithi12313934-1 2020-01-16T00:13:05.131 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 1 v 200 (ObjNum 240 snap 0 seq_num 240) dirty exists 2020-01-16T00:13:05.131 INFO:tasks.rados.rados.0.smithi123.stdout:996: left oid 1 (ObjNum 240 snap 0 seq_num 240) 2020-01-16T00:13:05.131 INFO:tasks.rados.rados.0.smithi123.stdout:997: expect (ObjNum 218 snap 0 seq_num 218) 2020-01-16T00:13:05.140 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:05.138+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:05.168 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:05.177 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-01-16T00:13:05.242 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:05.237+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:05.287 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:05.298 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-01-16T00:13:05.333 INFO:tasks.rados.rados.0.smithi123.stdout:998: finishing write tid 1 to smithi12313934-2 2020-01-16T00:13:05.333 INFO:tasks.rados.rados.0.smithi123.stdout:999: expect (ObjNum 215 snap 0 seq_num 215) 2020-01-16T00:13:05.342 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:05.338+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:05.443 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:05.438+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:05.454 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:05.463 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-01-16T00:13:05.544 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:13:05.541+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:13:05.561 INFO:tasks.rados.rados.0.smithi123.stdout:1000: expect (ObjNum 196 snap 0 seq_num 196) 2020-01-16T00:13:05.572 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:05.582 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-01-16T00:13:05.673 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-01-16T00:13:05.674 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:05.673+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:05.720 INFO:tasks.rados.rados.0.smithi123.stdout:1001: expect (ObjNum 193 snap 0 seq_num 193) 2020-01-16T00:13:05.738 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:05.748 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:05.745+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:05.748 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-01-16T00:13:05.810 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:13:05.828 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:13:05.845 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:13:05.848 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:13:05.845+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:13:05.864 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:13:05.884 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:13:05.901 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:05.903 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5: {} 2020-01-16T00:13:05.911 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-01-16T00:13:05.922 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:13:05.943 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:13:05.945 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-01-16T00:13:05.945 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-01-16T00:13:05.945 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-01-16T00:13:05.946 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-01-16T00:13:05.946 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-01-16T00:13:05.946 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-01-16T00:13:05.946 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-01-16T00:13:05.946 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-01-16T00:13:05.949 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:05.946+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:05.964 INFO:tasks.rados.rados.0.smithi123.stdout:998: finishing write tid 2 to smithi12313934-2 2020-01-16T00:13:05.964 INFO:tasks.rados.rados.0.smithi123.stdout:998: finishing write tid 4 to smithi12313934-2 2020-01-16T00:13:05.964 INFO:tasks.rados.rados.0.smithi123.stdout:998: finishing write tid 5 to smithi12313934-2 2020-01-16T00:13:05.965 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 2 v 137 (ObjNum 241 snap 0 seq_num 241) dirty exists 2020-01-16T00:13:05.965 INFO:tasks.rados.rados.0.smithi123.stdout:998: left oid 2 (ObjNum 241 snap 0 seq_num 241) 2020-01-16T00:13:05.965 INFO:tasks.rados.rados.0.smithi123.stdout:988: done (15 left) 2020-01-16T00:13:05.965 INFO:tasks.rados.rados.0.smithi123.stdout:989: done (14 left) 2020-01-16T00:13:05.965 INFO:tasks.rados.rados.0.smithi123.stdout:990: done (13 left) 2020-01-16T00:13:05.965 INFO:tasks.rados.rados.0.smithi123.stdout:991: done (12 left) 2020-01-16T00:13:05.965 INFO:tasks.rados.rados.0.smithi123.stdout:992: done (11 left) 2020-01-16T00:13:05.966 INFO:tasks.rados.rados.0.smithi123.stdout:993: done (10 left) 2020-01-16T00:13:05.966 INFO:tasks.rados.rados.0.smithi123.stdout:994: done (9 left) 2020-01-16T00:13:05.966 INFO:tasks.rados.rados.0.smithi123.stdout:995: done (8 left) 2020-01-16T00:13:05.967 INFO:tasks.rados.rados.0.smithi123.stdout:996: done (7 left) 2020-01-16T00:13:05.967 INFO:tasks.rados.rados.0.smithi123.stdout:997: done (6 left) 2020-01-16T00:13:05.967 INFO:tasks.rados.rados.0.smithi123.stdout:998: done (5 left) 2020-01-16T00:13:05.967 INFO:tasks.rados.rados.0.smithi123.stdout:999: done (4 left) 2020-01-16T00:13:05.967 INFO:tasks.rados.rados.0.smithi123.stdout:1000: done (3 left) 2020-01-16T00:13:05.967 INFO:tasks.rados.rados.0.smithi123.stdout:1001: done (2 left) 2020-01-16T00:13:05.967 INFO:tasks.rados.rados.0.smithi123.stdout:1004: read oid 42 snap -1 2020-01-16T00:13:05.967 INFO:tasks.rados.rados.0.smithi123.stdout:1004: expect (ObjNum 215 snap 0 seq_num 215) 2020-01-16T00:13:05.968 INFO:tasks.rados.rados.0.smithi123.stdout:1005: write oid 24 current snap is 0 2020-01-16T00:13:05.968 INFO:tasks.rados.rados.0.smithi123.stdout:1005: seq_num 244 ranges {500908=484052,1411724=689184} 2020-01-16T00:13:05.972 INFO:tasks.rados.rados.0.smithi123.stdout:1005: writing smithi12313934-24 from 500908 to 984960 tid 1 2020-01-16T00:13:05.980 INFO:tasks.rados.rados.0.smithi123.stdout:1005: writing smithi12313934-24 from 1411724 to 2100908 tid 2 2020-01-16T00:13:05.980 INFO:tasks.rados.rados.0.smithi123.stdout:1006: read oid 5 snap -1 2020-01-16T00:13:05.980 INFO:tasks.rados.rados.0.smithi123.stdout:1006: expect (ObjNum 238 snap 0 seq_num 238) 2020-01-16T00:13:05.981 INFO:tasks.rados.rados.0.smithi123.stdout:1007: write oid 48 current snap is 0 2020-01-16T00:13:05.981 INFO:tasks.rados.rados.0.smithi123.stdout:1007: seq_num 245 ranges {517715=559301,1870206=247509} 2020-01-16T00:13:05.989 INFO:tasks.rados.rados.0.smithi123.stdout:1007: writing smithi12313934-48 from 517715 to 1077016 tid 1 2020-01-16T00:13:05.993 INFO:tasks.rados.rados.0.smithi123.stdout:1007: writing smithi12313934-48 from 1870206 to 2117715 tid 2 2020-01-16T00:13:05.993 INFO:tasks.rados.rados.0.smithi123.stdout:1008: write (excl) oid 3 current snap is 0 2020-01-16T00:13:05.993 INFO:tasks.rados.rados.0.smithi123.stdout:1008: seq_num 246 ranges {534522=634550,1812335=322187} 2020-01-16T00:13:06.000 INFO:tasks.rados.rados.0.smithi123.stdout:1008: writing smithi12313934-3 from 534522 to 1169072 tid 1 2020-01-16T00:13:06.006 INFO:tasks.rados.rados.0.smithi123.stdout:1008: writing smithi12313934-3 from 1812335 to 2134522 tid 2 2020-01-16T00:13:06.006 INFO:tasks.rados.rados.0.smithi123.stdout:1009: write oid 36 current snap is 0 2020-01-16T00:13:06.006 INFO:tasks.rados.rados.0.smithi123.stdout:1009: seq_num 247 ranges {551329=709799,1870817=280512} 2020-01-16T00:13:06.013 INFO:tasks.rados.rados.0.smithi123.stdout:1009: writing smithi12313934-36 from 551329 to 1261128 tid 1 2020-01-16T00:13:06.018 INFO:tasks.rados.rados.0.smithi123.stdout:1009: writing smithi12313934-36 from 1870817 to 2151329 tid 2 2020-01-16T00:13:06.018 INFO:tasks.rados.rados.0.smithi123.stdout:1010: write oid 20 current snap is 0 2020-01-16T00:13:06.018 INFO:tasks.rados.rados.0.smithi123.stdout:1010: seq_num 248 ranges {568136=785048,1929299=238837} 2020-01-16T00:13:06.026 INFO:tasks.rados.rados.0.smithi123.stdout:1010: writing smithi12313934-20 from 568136 to 1353184 tid 1 2020-01-16T00:13:06.031 INFO:tasks.rados.rados.0.smithi123.stdout:1010: writing smithi12313934-20 from 1929299 to 2168136 tid 2 2020-01-16T00:13:06.031 INFO:tasks.rados.rados.0.smithi123.stdout:1011: write oid 15 current snap is 0 2020-01-16T00:13:06.031 INFO:tasks.rados.rados.0.smithi123.stdout:1011: seq_num 249 ranges {584943=460297,1587781=597162} 2020-01-16T00:13:06.036 INFO:tasks.rados.rados.0.smithi123.stdout:1011: writing smithi12313934-15 from 584943 to 1045240 tid 1 2020-01-16T00:13:06.043 INFO:tasks.rados.rados.0.smithi123.stdout:1011: writing smithi12313934-15 from 1587781 to 2184943 tid 2 2020-01-16T00:13:06.043 INFO:tasks.rados.rados.0.smithi123.stdout:1012: read oid 39 snap -1 2020-01-16T00:13:06.044 INFO:tasks.rados.rados.0.smithi123.stdout:1012: expect (ObjNum 194 snap 0 seq_num 194) 2020-01-16T00:13:06.044 INFO:tasks.rados.rados.0.smithi123.stdout:1013: read oid 35 snap -1 2020-01-16T00:13:06.044 INFO:tasks.rados.rados.0.smithi123.stdout:1013: expect (ObjNum 206 snap 0 seq_num 206) 2020-01-16T00:13:06.044 INFO:tasks.rados.rados.0.smithi123.stdout:1002: finishing write tid 1 to smithi12313934-28 2020-01-16T00:13:06.045 INFO:tasks.rados.rados.0.smithi123.stdout:1014: read oid 10 snap -1 2020-01-16T00:13:06.045 INFO:tasks.rados.rados.0.smithi123.stdout:1014: expect (ObjNum 234 snap 0 seq_num 234) 2020-01-16T00:13:06.045 INFO:tasks.rados.rados.0.smithi123.stdout:1015: write oid 47 current snap is 0 2020-01-16T00:13:06.045 INFO:tasks.rados.rados.0.smithi123.stdout:1015: seq_num 250 ranges {601750=535546,1929910=271840} 2020-01-16T00:13:06.049 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:06.046+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:06.050 INFO:tasks.rados.rados.0.smithi123.stdout:1015: writing smithi12313934-47 from 601750 to 1137296 tid 1 2020-01-16T00:13:06.058 INFO:tasks.rados.rados.0.smithi123.stdout:1015: writing smithi12313934-47 from 1929910 to 2201750 tid 2 2020-01-16T00:13:06.058 INFO:tasks.rados.rados.0.smithi123.stdout:1016: read oid 40 snap -1 2020-01-16T00:13:06.058 INFO:tasks.rados.rados.0.smithi123.stdout:1016: expect (ObjNum 224 snap 0 seq_num 224) 2020-01-16T00:13:06.058 INFO:tasks.rados.rados.0.smithi123.stdout:1017: read oid 46 snap -1 2020-01-16T00:13:06.058 INFO:tasks.rados.rados.0.smithi123.stdout:1017: expect (ObjNum 239 snap 0 seq_num 239) 2020-01-16T00:13:06.058 INFO:tasks.rados.rados.0.smithi123.stdout:1002: finishing write tid 2 to smithi12313934-28 2020-01-16T00:13:06.059 INFO:tasks.rados.rados.0.smithi123.stdout:1002: finishing write tid 4 to smithi12313934-28 2020-01-16T00:13:06.059 INFO:tasks.rados.rados.0.smithi123.stdout:1002: finishing write tid 5 to smithi12313934-28 2020-01-16T00:13:06.059 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 28 v 102 (ObjNum 242 snap 0 seq_num 242) dirty exists 2020-01-16T00:13:06.059 INFO:tasks.rados.rados.0.smithi123.stdout:1002: left oid 28 (ObjNum 242 snap 0 seq_num 242) 2020-01-16T00:13:06.059 INFO:tasks.rados.rados.0.smithi123.stdout:1003: finishing write tid 1 to smithi12313934-32 2020-01-16T00:13:06.059 INFO:tasks.rados.rados.0.smithi123.stdout:1003: finishing write tid 2 to smithi12313934-32 2020-01-16T00:13:06.059 INFO:tasks.rados.rados.0.smithi123.stdout:1003: finishing write tid 4 to smithi12313934-32 2020-01-16T00:13:06.060 INFO:tasks.rados.rados.0.smithi123.stdout:1002: done (15 left) 2020-01-16T00:13:06.060 INFO:tasks.rados.rados.0.smithi123.stdout:1018: read oid 22 snap -1 2020-01-16T00:13:06.060 INFO:tasks.rados.rados.0.smithi123.stdout:1018: expect (ObjNum 214 snap 0 seq_num 214) 2020-01-16T00:13:06.060 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:06.060 INFO:tasks.rados.rados.0.smithi123.stdout:1003: finishing write tid 5 to smithi12313934-32 2020-01-16T00:13:06.060 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 32 v 86 (ObjNum 243 snap 0 seq_num 243) dirty exists 2020-01-16T00:13:06.060 INFO:tasks.rados.rados.0.smithi123.stdout:1003: left oid 32 (ObjNum 243 snap 0 seq_num 243) 2020-01-16T00:13:06.060 INFO:tasks.rados.rados.0.smithi123.stdout:1003: done (15 left) 2020-01-16T00:13:06.061 INFO:tasks.rados.rados.0.smithi123.stdout:1019: write (excl) oid 13 current snap is 0 2020-01-16T00:13:06.061 INFO:tasks.rados.rados.0.smithi123.stdout:1019: seq_num 251 ranges {618557=727148,2104745=113812} 2020-01-16T00:13:06.064 INFO:tasks.rados.rados.0.smithi123.stdout:1019: writing smithi12313934-13 from 618557 to 1345705 tid 1 2020-01-16T00:13:06.071 INFO:tasks.rados.rados.0.smithi123.stdout:1019: writing smithi12313934-13 from 2104745 to 2218557 tid 2 2020-01-16T00:13:06.071 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:06.071 INFO:tasks.rados.rados.0.smithi123.stdout:1004: expect (ObjNum 215 snap 0 seq_num 215) 2020-01-16T00:13:06.077 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:06.077 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:13:06.078 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:13:06.078 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:06.087 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-01-16T00:13:06.150 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:06.146+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:06.205 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:06.205 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:13:06.206 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:13:06.206 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:13:06.206 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:06.216 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-01-16T00:13:06.252 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2020-01-16T00:13:06.263 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:06.261+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:06.320 INFO:tasks.rados.rados.0.smithi123.stdout:1005: finishing write tid 1 to smithi12313934-24 2020-01-16T00:13:06.320 INFO:tasks.rados.rados.0.smithi123.stdout:1005: finishing write tid 2 to smithi12313934-24 2020-01-16T00:13:06.320 INFO:tasks.rados.rados.0.smithi123.stdout:1006: expect (ObjNum 238 snap 0 seq_num 238) 2020-01-16T00:13:06.334 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:06.335 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:13:06.335 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:13:06.335 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:13:06.335 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.335 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:890 3.a 3:5c245412:::smithi12313934-19:head [write 759683~639358,stat] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:06.335 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.112181+0000", 2020-01-16T00:13:06.335 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 29.217630097000001, 2020-01-16T00:13:06.336 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0068210429999999997, 2020-01-16T00:13:06.336 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:06.336 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:06.336 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:06.336 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:06.336 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:06.337 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 890 2020-01-16T00:13:06.337 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.337 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:06.337 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.337 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:06.337 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.112181+0000", 2020-01-16T00:13:06.338 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:06.338 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.338 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.338 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:06.338 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.112181+0000", 2020-01-16T00:13:06.338 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999957 2020-01-16T00:13:06.339 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.339 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.339 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:06.339 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.112177+0000", 2020-01-16T00:13:06.339 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00060751999999999996 2020-01-16T00:13:06.340 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.340 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.340 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:06.340 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.112784+0000", 2020-01-16T00:13:06.340 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3646999999999998e-05 2020-01-16T00:13:06.340 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.341 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.341 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:06.341 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.112818+0000", 2020-01-16T00:13:06.341 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.432e-05 2020-01-16T00:13:06.341 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.341 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.341 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:06.342 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.112832+0000", 2020-01-16T00:13:06.342 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000118984 2020-01-16T00:13:06.342 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.342 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.342 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:06.342 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.112951+0000", 2020-01-16T00:13:06.342 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00023189400000000001 2020-01-16T00:13:06.343 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.343 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.343 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:06.343 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.113183+0000", 2020-01-16T00:13:06.343 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00035231999999999999 2020-01-16T00:13:06.343 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.343 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.343 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:06.344 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.113535+0000", 2020-01-16T00:13:06.344 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0035763539999999999 2020-01-16T00:13:06.344 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.344 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.344 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:06.344 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.117112+0000", 2020-01-16T00:13:06.344 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4860000000000002e-05 2020-01-16T00:13:06.345 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.345 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.345 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:06.345 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.117147+0000", 2020-01-16T00:13:06.345 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0017898110000000001 2020-01-16T00:13:06.345 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.346 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.346 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:06.346 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.118936+0000", 2020-01-16T00:13:06.346 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6343000000000001e-05 2020-01-16T00:13:06.346 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.346 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.347 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:06.347 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.118973+0000", 2020-01-16T00:13:06.347 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9257000000000001e-05 2020-01-16T00:13:06.347 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.347 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.347 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:06.348 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.119002+0000", 2020-01-16T00:13:06.348 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0068210429999999997 2020-01-16T00:13:06.348 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.348 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:06.348 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.348 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.348 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.349 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(osd.1.0:162 3.0 e24/20 3:00c146d5:::smithi12313934-18:head v 24'99, mlcod=24'97)", 2020-01-16T00:13:06.349 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.038304+0000", 2020-01-16T00:13:06.349 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 28.291506667, 2020-01-16T00:13:06.349 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.006772386, 2020-01-16T00:13:06.349 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:06.349 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:06.350 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:06.350 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.350 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:06.350 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.038304+0000", 2020-01-16T00:13:06.350 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:06.350 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.350 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.351 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:06.351 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.038304+0000", 2020-01-16T00:13:06.351 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:13:06.351 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.351 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.351 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:06.352 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.038301+0000", 2020-01-16T00:13:06.352 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012763920000000001 2020-01-16T00:13:06.352 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.352 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.352 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:06.352 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.039577+0000", 2020-01-16T00:13:06.352 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1426e-05 2020-01-16T00:13:06.352 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.353 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.353 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:06.353 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.039608+0000", 2020-01-16T00:13:06.353 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2017e-05 2020-01-16T00:13:06.353 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.353 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.353 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:06.353 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.039620+0000", 2020-01-16T00:13:06.354 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000111969 2020-01-16T00:13:06.354 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.354 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.354 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:06.354 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.039732+0000", 2020-01-16T00:13:06.354 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1180000000000002e-05 2020-01-16T00:13:06.354 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.355 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.355 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:06.355 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.039774+0000", 2020-01-16T00:13:06.355 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.005179286 2020-01-16T00:13:06.355 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.355 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.355 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:06.355 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.044953+0000", 2020-01-16T00:13:06.356 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012396600000000001 2020-01-16T00:13:06.356 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.356 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.356 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:06.356 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.045077+0000", 2020-01-16T00:13:06.356 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.006772386 2020-01-16T00:13:06.356 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.357 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:06.357 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.357 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.357 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.357 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:965 3.a 3:54f2c117:::smithi12313934-41:head [write 527753~708201] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:06.358 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.146756+0000", 2020-01-16T00:13:06.358 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 28.183055564, 2020-01-16T00:13:06.358 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0071708459999999998, 2020-01-16T00:13:06.358 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:06.358 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:06.358 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:06.358 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:06.358 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:06.359 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 965 2020-01-16T00:13:06.359 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.359 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:06.359 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.359 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:06.359 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.146756+0000", 2020-01-16T00:13:06.359 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:06.360 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.360 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.360 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:06.360 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.146756+0000", 2020-01-16T00:13:06.360 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999948 2020-01-16T00:13:06.360 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.360 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.360 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:06.361 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.146750+0000", 2020-01-16T00:13:06.361 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00094503399999999998 2020-01-16T00:13:06.361 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.361 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.361 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:06.361 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.147695+0000", 2020-01-16T00:13:06.361 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7846999999999999e-05 2020-01-16T00:13:06.362 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.362 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.362 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:06.362 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.147733+0000", 2020-01-16T00:13:06.362 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4392999999999999e-05 2020-01-16T00:13:06.362 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.362 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.362 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:06.363 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.147747+0000", 2020-01-16T00:13:06.363 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000106737 2020-01-16T00:13:06.363 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.363 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.363 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:06.363 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.147854+0000", 2020-01-16T00:13:06.364 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00026514699999999997 2020-01-16T00:13:06.364 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.364 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.364 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:06.364 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.148119+0000", 2020-01-16T00:13:06.364 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030262600000000001 2020-01-16T00:13:06.364 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.364 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.365 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:06.365 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.148422+0000", 2020-01-16T00:13:06.365 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.004368416 2020-01-16T00:13:06.365 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.365 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.365 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:06.365 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.152790+0000", 2020-01-16T00:13:06.366 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000368437 2020-01-16T00:13:06.366 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.366 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.366 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:06.366 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.153159+0000", 2020-01-16T00:13:06.366 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00058380900000000002 2020-01-16T00:13:06.366 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.366 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.367 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:06.367 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.153743+0000", 2020-01-16T00:13:06.367 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8196999999999995e-05 2020-01-16T00:13:06.367 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.367 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.367 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:06.368 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.153841+0000", 2020-01-16T00:13:06.368 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5625999999999996e-05 2020-01-16T00:13:06.368 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.368 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.369 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:06.369 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.153926+0000", 2020-01-16T00:13:06.369 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0071708459999999998 2020-01-16T00:13:06.369 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.369 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:06.369 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.369 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.370 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.370 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:966 3.a 3:54f2c117:::smithi12313934-41:head [write 1856548~737690] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:06.370 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.154844+0000", 2020-01-16T00:13:06.370 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 28.174966761, 2020-01-16T00:13:06.370 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0077324510000000004, 2020-01-16T00:13:06.370 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:06.370 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:06.371 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:06.371 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:06.371 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:06.371 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 966 2020-01-16T00:13:06.371 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.371 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:06.371 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.371 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:06.372 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.154844+0000", 2020-01-16T00:13:06.372 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:06.372 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.372 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.372 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:06.372 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.154844+0000", 2020-01-16T00:13:06.372 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:06.373 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.373 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.373 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:06.373 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.154842+0000", 2020-01-16T00:13:06.373 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00085401100000000001 2020-01-16T00:13:06.373 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.373 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.374 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:06.374 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.155696+0000", 2020-01-16T00:13:06.374 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9801000000000001e-05 2020-01-16T00:13:06.374 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.374 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.374 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:06.374 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.155726+0000", 2020-01-16T00:13:06.375 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.171e-05 2020-01-16T00:13:06.375 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.375 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.375 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:06.375 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.155738+0000", 2020-01-16T00:13:06.375 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0573000000000003e-05 2020-01-16T00:13:06.375 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.375 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.376 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:06.376 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.155818+0000", 2020-01-16T00:13:06.376 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024165699999999999 2020-01-16T00:13:06.376 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.376 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.376 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:06.376 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.156060+0000", 2020-01-16T00:13:06.377 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033522700000000001 2020-01-16T00:13:06.377 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.377 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.377 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:06.377 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.156395+0000", 2020-01-16T00:13:06.378 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0059325039999999999 2020-01-16T00:13:06.378 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.378 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.378 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:06.378 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.162328+0000", 2020-01-16T00:13:06.378 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5182e-05 2020-01-16T00:13:06.378 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.379 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.379 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:06.379 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.162393+0000", 2020-01-16T00:13:06.379 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8604e-05 2020-01-16T00:13:06.379 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.379 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.379 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:06.380 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.162411+0000", 2020-01-16T00:13:06.380 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2386000000000006e-05 2020-01-16T00:13:06.380 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.380 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.380 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:06.380 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.162504+0000", 2020-01-16T00:13:06.380 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2970000000000001e-05 2020-01-16T00:13:06.381 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.381 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.381 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:06.381 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.162577+0000", 2020-01-16T00:13:06.381 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0077324510000000004 2020-01-16T00:13:06.381 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.381 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:06.382 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.382 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.382 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.382 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.4488.0:1237 3.3 e26/20 3:c204aa4f:::smithi12313934-45:head v 26'56, mlcod=26'54)", 2020-01-16T00:13:06.382 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.587132+0000", 2020-01-16T00:13:06.382 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 23.742679541000001, 2020-01-16T00:13:06.382 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0076804919999999997, 2020-01-16T00:13:06.383 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:06.383 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:06.383 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:06.383 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.383 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:06.383 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.587132+0000", 2020-01-16T00:13:06.383 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:06.383 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.384 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.384 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:06.384 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.587132+0000", 2020-01-16T00:13:06.384 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999957 2020-01-16T00:13:06.384 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.384 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.384 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:06.385 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.587127+0000", 2020-01-16T00:13:06.385 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00047205800000000003 2020-01-16T00:13:06.385 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.385 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.385 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:06.385 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.587599+0000", 2020-01-16T00:13:06.385 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6477000000000003e-05 2020-01-16T00:13:06.386 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.386 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.386 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:06.386 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.587636+0000", 2020-01-16T00:13:06.386 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7294000000000001e-05 2020-01-16T00:13:06.386 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.386 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.386 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:06.387 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.587653+0000", 2020-01-16T00:13:06.387 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000183778 2020-01-16T00:13:06.387 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.387 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.387 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:06.387 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.587837+0000", 2020-01-16T00:13:06.387 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0628000000000001e-05 2020-01-16T00:13:06.388 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.388 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.388 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:06.388 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.587917+0000", 2020-01-16T00:13:06.388 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.006854527 2020-01-16T00:13:06.388 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.389 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.389 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:06.389 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.594772+0000", 2020-01-16T00:13:06.389 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0144000000000001e-05 2020-01-16T00:13:06.389 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.389 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.389 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:06.390 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.594812+0000", 2020-01-16T00:13:06.390 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0076804919999999997 2020-01-16T00:13:06.390 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.390 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:06.390 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.390 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.390 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.390 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1241 3.a 3:5c245412:::smithi12313934-19:head [read 0~3815156,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:13:06.391 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.589903+0000", 2020-01-16T00:13:06.391 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 23.739908143000001, 2020-01-16T00:13:06.391 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0096065560000000005, 2020-01-16T00:13:06.391 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:06.391 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:06.391 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:06.391 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:06.392 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:06.392 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1241 2020-01-16T00:13:06.392 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.392 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:06.392 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.392 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:06.392 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.589903+0000", 2020-01-16T00:13:06.393 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:06.393 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.393 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.393 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:06.393 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.589903+0000", 2020-01-16T00:13:06.393 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999952 2020-01-16T00:13:06.393 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.393 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.394 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:06.394 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.589898+0000", 2020-01-16T00:13:06.394 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4389e-05 2020-01-16T00:13:06.394 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.394 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.394 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:06.394 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.589922+0000", 2020-01-16T00:13:06.395 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4159999999999998e-05 2020-01-16T00:13:06.395 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.395 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.395 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:06.395 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.589957+0000", 2020-01-16T00:13:06.395 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7368999999999999e-05 2020-01-16T00:13:06.395 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.395 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.396 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:06.396 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.589974+0000", 2020-01-16T00:13:06.396 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000130407 2020-01-16T00:13:06.396 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.396 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.396 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:06.396 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590104+0000", 2020-01-16T00:13:06.397 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00040217699999999998 2020-01-16T00:13:06.397 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.397 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.397 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:06.397 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590506+0000", 2020-01-16T00:13:06.397 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0090030230000000006 2020-01-16T00:13:06.397 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.398 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.398 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:06.398 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599510+0000", 2020-01-16T00:13:06.398 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0096065560000000005 2020-01-16T00:13:06.398 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.399 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:06.399 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.399 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.399 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.399 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1242 3.a 3:5c245412:::smithi12313934-19:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:13:06.399 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.590003+0000", 2020-01-16T00:13:06.399 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 23.739807634000002, 2020-01-16T00:13:06.399 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010247029, 2020-01-16T00:13:06.400 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:06.400 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:06.400 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:06.400 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:06.400 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:06.400 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1242 2020-01-16T00:13:06.400 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.401 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:06.401 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.401 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:06.401 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590003+0000", 2020-01-16T00:13:06.401 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:06.401 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.401 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.402 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:06.402 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590003+0000", 2020-01-16T00:13:06.402 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:06.402 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.402 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.402 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:06.402 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590002+0000", 2020-01-16T00:13:06.403 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0108e-05 2020-01-16T00:13:06.403 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.403 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.403 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:06.403 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590012+0000", 2020-01-16T00:13:06.403 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5318000000000001e-05 2020-01-16T00:13:06.403 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.404 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.404 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:06.404 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590027+0000", 2020-01-16T00:13:06.404 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0720000000000001e-05 2020-01-16T00:13:06.404 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.404 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.404 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:06.404 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590038+0000", 2020-01-16T00:13:06.405 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0095349949999999992 2020-01-16T00:13:06.405 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.405 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.405 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:06.405 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599573+0000", 2020-01-16T00:13:06.405 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017107600000000001 2020-01-16T00:13:06.405 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.406 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.406 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:06.406 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599744+0000", 2020-01-16T00:13:06.406 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00050659499999999998 2020-01-16T00:13:06.406 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.406 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.406 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:06.407 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.600251+0000", 2020-01-16T00:13:06.407 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010247029 2020-01-16T00:13:06.407 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.407 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:06.407 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.407 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.408 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.408 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1243 3.a 3:5c245412:::smithi12313934-19:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:13:06.408 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.590095+0000", 2020-01-16T00:13:06.408 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 23.739716224999999, 2020-01-16T00:13:06.408 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010591455, 2020-01-16T00:13:06.408 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:06.409 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:06.409 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:06.409 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:06.409 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:06.409 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1243 2020-01-16T00:13:06.409 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.409 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:06.410 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.410 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:06.410 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590095+0000", 2020-01-16T00:13:06.410 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:06.410 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.410 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.410 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:06.411 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590095+0000", 2020-01-16T00:13:06.411 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:06.411 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.411 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.411 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:06.411 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590093+0000", 2020-01-16T00:13:06.411 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5619999999999993e-06 2020-01-16T00:13:06.411 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.412 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.412 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:06.412 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590102+0000", 2020-01-16T00:13:06.412 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.259e-05 2020-01-16T00:13:06.412 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.412 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.412 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:06.413 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590115+0000", 2020-01-16T00:13:06.413 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8730000000000001e-06 2020-01-16T00:13:06.413 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.413 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.413 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:06.414 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590122+0000", 2020-01-16T00:13:06.414 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010163165 2020-01-16T00:13:06.414 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.414 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.414 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:06.414 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.600285+0000", 2020-01-16T00:13:06.415 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0128000000000002e-05 2020-01-16T00:13:06.415 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.415 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.415 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:06.415 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.600375+0000", 2020-01-16T00:13:06.415 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000311312 2020-01-16T00:13:06.415 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.416 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.416 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:06.416 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.600686+0000", 2020-01-16T00:13:06.416 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010591455 2020-01-16T00:13:06.416 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.416 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:06.416 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.416 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.417 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.417 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1654 3.a 3:5069665e:::smithi12313934-2:head [write 1614880~581041] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:13:06.417 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.714839+0000", 2020-01-16T00:13:06.417 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 13.614972237, 2020-01-16T00:13:06.417 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013588093000000001, 2020-01-16T00:13:06.417 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:06.417 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:06.418 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:06.418 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:06.418 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:06.418 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1654 2020-01-16T00:13:06.418 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.418 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:06.419 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.419 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:06.419 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.714839+0000", 2020-01-16T00:13:06.419 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:06.419 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.419 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.419 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:06.420 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.714839+0000", 2020-01-16T00:13:06.420 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:06.420 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.420 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.420 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:06.420 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.714836+0000", 2020-01-16T00:13:06.420 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00054657999999999996 2020-01-16T00:13:06.420 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.421 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.421 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:06.421 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.715382+0000", 2020-01-16T00:13:06.421 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7743000000000001e-05 2020-01-16T00:13:06.421 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.421 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.421 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:06.422 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.715420+0000", 2020-01-16T00:13:06.422 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.221e-05 2020-01-16T00:13:06.422 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.422 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.422 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:06.422 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.715432+0000", 2020-01-16T00:13:06.422 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2487000000000001e-05 2020-01-16T00:13:06.423 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.423 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.423 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:06.423 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.715485+0000", 2020-01-16T00:13:06.423 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000726284 2020-01-16T00:13:06.423 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.423 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.424 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:06.424 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.716211+0000", 2020-01-16T00:13:06.424 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0067930899999999999 2020-01-16T00:13:06.424 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.424 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.424 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:06.424 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.723004+0000", 2020-01-16T00:13:06.424 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012579400000000001 2020-01-16T00:13:06.425 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.425 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.425 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:06.425 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.723130+0000", 2020-01-16T00:13:06.425 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013465800000000001 2020-01-16T00:13:06.425 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.425 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.426 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:06.426 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.723265+0000", 2020-01-16T00:13:06.426 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023307250000000001 2020-01-16T00:13:06.426 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.426 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.426 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:06.427 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.725595+0000", 2020-01-16T00:13:06.427 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012582139999999999 2020-01-16T00:13:06.427 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.427 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.427 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:06.427 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.726853+0000", 2020-01-16T00:13:06.427 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014479110000000001 2020-01-16T00:13:06.428 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.428 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.428 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:06.428 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.728301+0000", 2020-01-16T00:13:06.428 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6866000000000007e-05 2020-01-16T00:13:06.428 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.428 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.429 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:06.429 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.728368+0000", 2020-01-16T00:13:06.429 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8740999999999999e-05 2020-01-16T00:13:06.429 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.430 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.430 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:06.430 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.728427+0000", 2020-01-16T00:13:06.430 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013588093000000001 2020-01-16T00:13:06.430 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.430 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:06.430 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.431 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.431 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.431 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1655 3.a 3:5069665e:::smithi12313934-2:head [write 2785791~71399] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:13:06.431 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.718297+0000", 2020-01-16T00:13:06.431 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 13.611514383999999, 2020-01-16T00:13:06.431 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013817354, 2020-01-16T00:13:06.431 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:06.431 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:06.432 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:06.432 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:06.432 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:06.432 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1655 2020-01-16T00:13:06.432 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.432 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:06.432 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.432 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:06.433 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718297+0000", 2020-01-16T00:13:06.433 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:06.433 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.433 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.433 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:06.433 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718297+0000", 2020-01-16T00:13:06.434 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:06.434 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.434 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.434 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:06.434 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718294+0000", 2020-01-16T00:13:06.434 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000103558 2020-01-16T00:13:06.434 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.434 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.435 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:06.435 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718397+0000", 2020-01-16T00:13:06.435 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9287e-05 2020-01-16T00:13:06.435 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.435 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.435 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:06.435 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718417+0000", 2020-01-16T00:13:06.436 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6680000000000001e-06 2020-01-16T00:13:06.436 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.436 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.436 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:06.436 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718424+0000", 2020-01-16T00:13:06.436 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2701999999999994e-05 2020-01-16T00:13:06.436 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.437 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.437 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:06.437 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718517+0000", 2020-01-16T00:13:06.437 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3893000000000006e-05 2020-01-16T00:13:06.437 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.437 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.437 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:06.437 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718581+0000", 2020-01-16T00:13:06.438 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0059630079999999997 2020-01-16T00:13:06.438 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.438 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.438 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:06.438 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.724544+0000", 2020-01-16T00:13:06.438 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000309108 2020-01-16T00:13:06.438 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.439 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.439 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:06.439 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.724853+0000", 2020-01-16T00:13:06.439 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.005469794 2020-01-16T00:13:06.440 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.440 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.440 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:06.440 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.730323+0000", 2020-01-16T00:13:06.440 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018506100000000001 2020-01-16T00:13:06.440 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.440 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.441 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:06.441 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.730508+0000", 2020-01-16T00:13:06.441 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016729099999999999 2020-01-16T00:13:06.441 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.441 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.441 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:06.442 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.730675+0000", 2020-01-16T00:13:06.442 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00094193300000000005 2020-01-16T00:13:06.442 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.442 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.442 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:06.442 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731617+0000", 2020-01-16T00:13:06.442 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000214613 2020-01-16T00:13:06.443 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.443 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.443 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:06.443 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731832+0000", 2020-01-16T00:13:06.443 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018326400000000001 2020-01-16T00:13:06.443 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.443 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.443 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:06.444 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732015+0000", 2020-01-16T00:13:06.444 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0392e-05 2020-01-16T00:13:06.444 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.444 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.444 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:06.444 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732075+0000", 2020-01-16T00:13:06.444 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8677999999999998e-05 2020-01-16T00:13:06.445 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.445 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.445 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:06.445 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732114+0000", 2020-01-16T00:13:06.445 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013817354 2020-01-16T00:13:06.445 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.445 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:06.446 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.446 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.446 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.446 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1656 3.a 3:5069665e:::smithi12313934-2:head [setxattr _header (55) in=7b,truncate 2857190] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:13:06.447 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.718453+0000", 2020-01-16T00:13:06.447 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 13.611357702999999, 2020-01-16T00:13:06.447 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014083462, 2020-01-16T00:13:06.447 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:06.447 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:06.447 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:06.447 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:06.447 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:06.448 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1656 2020-01-16T00:13:06.448 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.448 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:06.448 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.448 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:06.448 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718453+0000", 2020-01-16T00:13:06.448 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:06.448 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.449 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.449 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:06.449 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718453+0000", 2020-01-16T00:13:06.449 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999828 2020-01-16T00:13:06.449 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.449 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.450 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:06.450 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718436+0000", 2020-01-16T00:13:06.450 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4051999999999999e-05 2020-01-16T00:13:06.450 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.450 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.450 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:06.450 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718460+0000", 2020-01-16T00:13:06.451 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6380000000000004e-06 2020-01-16T00:13:06.451 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.451 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.451 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:06.451 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718468+0000", 2020-01-16T00:13:06.451 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6750000000000001e-06 2020-01-16T00:13:06.452 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.452 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.452 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:06.452 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718472+0000", 2020-01-16T00:13:06.452 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016218499999999999 2020-01-16T00:13:06.452 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.452 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.452 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:06.453 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718635+0000", 2020-01-16T00:13:06.453 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2765e-05 2020-01-16T00:13:06.453 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.453 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.453 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:06.453 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718697+0000", 2020-01-16T00:13:06.453 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0062239249999999999 2020-01-16T00:13:06.454 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.454 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.454 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:06.454 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.724921+0000", 2020-01-16T00:13:06.454 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1527999999999999e-05 2020-01-16T00:13:06.454 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.454 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.455 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:06.455 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.724973+0000", 2020-01-16T00:13:06.455 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0060569719999999999 2020-01-16T00:13:06.455 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.455 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.455 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:06.455 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731030+0000", 2020-01-16T00:13:06.455 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5859999999999996e-05 2020-01-16T00:13:06.456 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.456 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.456 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:06.456 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731096+0000", 2020-01-16T00:13:06.456 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000112018 2020-01-16T00:13:06.456 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.456 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.457 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:06.457 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731208+0000", 2020-01-16T00:13:06.457 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000945889 2020-01-16T00:13:06.457 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.457 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.457 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:06.458 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732154+0000", 2020-01-16T00:13:06.458 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0834000000000003e-05 2020-01-16T00:13:06.458 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.458 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.458 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:06.458 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732184+0000", 2020-01-16T00:13:06.458 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028587600000000001 2020-01-16T00:13:06.458 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.459 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.459 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:06.459 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732470+0000", 2020-01-16T00:13:06.459 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1953999999999999e-05 2020-01-16T00:13:06.459 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.459 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.460 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:06.460 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732502+0000", 2020-01-16T00:13:06.460 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4644999999999998e-05 2020-01-16T00:13:06.460 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.460 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.460 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:06.460 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732537+0000", 2020-01-16T00:13:06.461 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014083462 2020-01-16T00:13:06.461 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.461 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:06.461 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.461 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.461 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.461 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1657 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-01-16T00:13:06.462 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.718495+0000", 2020-01-16T00:13:06.462 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 13.611316038, 2020-01-16T00:13:06.462 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014299630000000001, 2020-01-16T00:13:06.462 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:06.462 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:06.462 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:06.462 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:06.463 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:06.463 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1657 2020-01-16T00:13:06.463 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.463 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:06.463 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.463 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:06.463 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718495+0000", 2020-01-16T00:13:06.463 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:06.464 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.464 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.464 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:06.464 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718495+0000", 2020-01-16T00:13:06.464 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:06.464 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.464 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.465 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:06.465 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718494+0000", 2020-01-16T00:13:06.465 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8860000000000003e-06 2020-01-16T00:13:06.465 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.465 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.465 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:06.465 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718499+0000", 2020-01-16T00:13:06.465 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9786000000000002e-05 2020-01-16T00:13:06.466 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.466 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.466 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:06.466 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718539+0000", 2020-01-16T00:13:06.466 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1869999999999999e-05 2020-01-16T00:13:06.466 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.466 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.467 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:06.467 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718561+0000", 2020-01-16T00:13:06.467 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000172206 2020-01-16T00:13:06.467 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.467 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.467 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:06.467 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718733+0000", 2020-01-16T00:13:06.468 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0662999999999998e-05 2020-01-16T00:13:06.468 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.468 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.468 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:06.468 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718793+0000", 2020-01-16T00:13:06.468 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.006217958 2020-01-16T00:13:06.468 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.469 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.469 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:06.469 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.725011+0000", 2020-01-16T00:13:06.469 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1625999999999996e-05 2020-01-16T00:13:06.470 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.470 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.470 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:06.470 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.725073+0000", 2020-01-16T00:13:06.470 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0063923119999999998 2020-01-16T00:13:06.470 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.470 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.471 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:06.471 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731465+0000", 2020-01-16T00:13:06.471 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2534000000000005e-05 2020-01-16T00:13:06.471 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.471 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.471 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:06.471 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731558+0000", 2020-01-16T00:13:06.471 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010238809999999999 2020-01-16T00:13:06.472 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.472 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.472 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:06.472 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732582+0000", 2020-01-16T00:13:06.472 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000100918 2020-01-16T00:13:06.472 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.472 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.472 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:06.473 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732683+0000", 2020-01-16T00:13:06.473 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000112064 2020-01-16T00:13:06.473 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.473 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.473 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:06.473 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732795+0000", 2020-01-16T00:13:06.473 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014299630000000001 2020-01-16T00:13:06.474 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.474 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:06.474 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.474 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.474 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.474 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.4488.0:1708 3.c e30/20 3:378f3722:::smithi12313934-50:head v 30'89, mlcod=29'87)", 2020-01-16T00:13:06.475 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.808249+0000", 2020-01-16T00:13:06.475 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 13.521562291, 2020-01-16T00:13:06.475 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0067795160000000002, 2020-01-16T00:13:06.475 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:06.475 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:06.475 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:06.475 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.475 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:06.476 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.808249+0000", 2020-01-16T00:13:06.476 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:06.476 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.476 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.476 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:06.476 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.808249+0000", 2020-01-16T00:13:06.476 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:06.477 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.477 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.477 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:06.477 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.808246+0000", 2020-01-16T00:13:06.477 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00046200000000000001 2020-01-16T00:13:06.477 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.477 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.478 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:06.478 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.808708+0000", 2020-01-16T00:13:06.478 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.702e-05 2020-01-16T00:13:06.478 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.478 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.478 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:06.478 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.808725+0000", 2020-01-16T00:13:06.478 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4920000000000001e-06 2020-01-16T00:13:06.479 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.479 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.479 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:06.479 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.808731+0000", 2020-01-16T00:13:06.479 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4094000000000001e-05 2020-01-16T00:13:06.479 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.479 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.479 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:06.480 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.808785+0000", 2020-01-16T00:13:06.480 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.7359999999999999e-05 2020-01-16T00:13:06.480 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.480 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.480 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:06.480 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.808812+0000", 2020-01-16T00:13:06.481 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0061494260000000004 2020-01-16T00:13:06.481 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.481 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.481 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:06.481 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.814962+0000", 2020-01-16T00:13:06.481 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6660000000000002e-05 2020-01-16T00:13:06.481 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.481 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.482 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:06.482 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.815028+0000", 2020-01-16T00:13:06.482 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0067795160000000002 2020-01-16T00:13:06.482 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.482 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:06.482 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.482 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.483 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.483 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1834 3.a 3:54f2c117:::smithi12313934-41:head [write 443751~631882] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:13:06.483 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:56.278017+0000", 2020-01-16T00:13:06.483 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 10.051794047, 2020-01-16T00:13:06.483 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0076931760000000004, 2020-01-16T00:13:06.483 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:06.483 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:06.484 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:06.484 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:06.484 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:06.484 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1834 2020-01-16T00:13:06.484 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.484 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:06.484 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.484 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:06.485 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.278017+0000", 2020-01-16T00:13:06.485 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:06.485 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.485 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.485 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:06.485 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.278017+0000", 2020-01-16T00:13:06.485 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:13:06.486 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.486 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.486 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:06.486 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.278013+0000", 2020-01-16T00:13:06.486 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010862669999999999 2020-01-16T00:13:06.486 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.486 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.487 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:06.487 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.279100+0000", 2020-01-16T00:13:06.487 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8034000000000002e-05 2020-01-16T00:13:06.487 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.487 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.487 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:06.487 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.279138+0000", 2020-01-16T00:13:06.488 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7955999999999999e-05 2020-01-16T00:13:06.488 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.488 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.488 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:06.488 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.279156+0000", 2020-01-16T00:13:06.488 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000141691 2020-01-16T00:13:06.488 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.489 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.489 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:06.489 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.279297+0000", 2020-01-16T00:13:06.489 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00025750000000000002 2020-01-16T00:13:06.489 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.489 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.489 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:06.490 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.279555+0000", 2020-01-16T00:13:06.490 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00031473999999999999 2020-01-16T00:13:06.490 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.490 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.490 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:06.490 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.279869+0000", 2020-01-16T00:13:06.491 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003741192 2020-01-16T00:13:06.491 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.491 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.491 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:06.491 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.283611+0000", 2020-01-16T00:13:06.491 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00075302099999999999 2020-01-16T00:13:06.492 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.492 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.492 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:06.492 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.284364+0000", 2020-01-16T00:13:06.492 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011594509999999999 2020-01-16T00:13:06.492 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.493 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.493 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:06.493 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.285523+0000", 2020-01-16T00:13:06.493 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000114127 2020-01-16T00:13:06.493 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.493 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.493 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:06.494 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.285637+0000", 2020-01-16T00:13:06.494 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2972999999999994e-05 2020-01-16T00:13:06.494 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.494 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.494 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:06.494 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.285710+0000", 2020-01-16T00:13:06.494 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0076931760000000004 2020-01-16T00:13:06.495 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.495 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:06.495 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.495 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.495 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.495 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1917 3.a 3:5069665e:::smithi12313934-2:head [write 2113302~674388] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:13:06.495 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:57.330712+0000", 2020-01-16T00:13:06.496 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 8.9990995229999999, 2020-01-16T00:13:06.496 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012781043000000001, 2020-01-16T00:13:06.496 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:06.496 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:06.496 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:06.496 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:06.496 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:06.496 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1917 2020-01-16T00:13:06.497 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.497 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:06.497 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.497 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:06.497 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.330712+0000", 2020-01-16T00:13:06.497 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:06.497 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.498 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.498 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:06.498 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.330712+0000", 2020-01-16T00:13:06.498 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:06.498 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.498 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.498 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:06.498 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.330709+0000", 2020-01-16T00:13:06.499 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00071550499999999996 2020-01-16T00:13:06.499 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.499 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.499 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:06.499 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331424+0000", 2020-01-16T00:13:06.499 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3354e-05 2020-01-16T00:13:06.499 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.500 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.500 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:06.500 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331447+0000", 2020-01-16T00:13:06.500 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8200000000000008e-06 2020-01-16T00:13:06.500 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.500 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.501 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:06.501 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331457+0000", 2020-01-16T00:13:06.501 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5557999999999997e-05 2020-01-16T00:13:06.501 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.501 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.501 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:06.501 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331533+0000", 2020-01-16T00:13:06.502 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030925000000000001 2020-01-16T00:13:06.502 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.502 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.502 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:06.502 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331842+0000", 2020-01-16T00:13:06.502 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0078507860000000002 2020-01-16T00:13:06.502 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.503 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.503 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:06.503 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.339693+0000", 2020-01-16T00:13:06.503 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9321000000000007e-05 2020-01-16T00:13:06.503 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.503 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.503 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:06.503 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.339792+0000", 2020-01-16T00:13:06.504 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015147700000000001 2020-01-16T00:13:06.504 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.504 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.504 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:06.504 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.339944+0000", 2020-01-16T00:13:06.504 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002663066 2020-01-16T00:13:06.504 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.504 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.505 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:06.505 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.342607+0000", 2020-01-16T00:13:06.505 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0001552 2020-01-16T00:13:06.505 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.505 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.505 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:06.505 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.342762+0000", 2020-01-16T00:13:06.506 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00063636900000000002 2020-01-16T00:13:06.506 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.506 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.506 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:06.506 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343398+0000", 2020-01-16T00:13:06.506 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9687000000000001e-05 2020-01-16T00:13:06.506 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.507 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.507 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:06.507 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343438+0000", 2020-01-16T00:13:06.507 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4647e-05 2020-01-16T00:13:06.507 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.508 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.508 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:06.508 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343493+0000", 2020-01-16T00:13:06.508 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012781043000000001 2020-01-16T00:13:06.508 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.508 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:06.508 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.508 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.509 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.509 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1918 3.a 3:5069665e:::smithi12313934-2:head [write 3335030~144019] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:13:06.509 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:57.334874+0000", 2020-01-16T00:13:06.509 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 8.9949368710000002, 2020-01-16T00:13:06.509 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0090004249999999994, 2020-01-16T00:13:06.509 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:06.509 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:06.510 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:06.510 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:06.510 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:06.510 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1918 2020-01-16T00:13:06.510 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.510 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:06.511 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.511 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:06.511 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.334874+0000", 2020-01-16T00:13:06.511 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:06.511 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.511 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.511 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:06.512 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.334874+0000", 2020-01-16T00:13:06.512 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:13:06.512 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.512 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.512 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:06.512 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.334870+0000", 2020-01-16T00:13:06.512 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000113143 2020-01-16T00:13:06.513 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.513 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.513 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:06.513 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.334983+0000", 2020-01-16T00:13:06.513 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4298000000000003e-05 2020-01-16T00:13:06.513 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.513 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.514 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:06.514 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335018+0000", 2020-01-16T00:13:06.514 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6860000000000001e-05 2020-01-16T00:13:06.514 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.514 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.515 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:06.515 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335035+0000", 2020-01-16T00:13:06.515 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000111932 2020-01-16T00:13:06.515 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.515 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.515 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:06.515 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335146+0000", 2020-01-16T00:13:06.516 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012407400000000001 2020-01-16T00:13:06.516 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.516 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.516 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:06.516 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335271+0000", 2020-01-16T00:13:06.516 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0056080180000000002 2020-01-16T00:13:06.516 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.516 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.517 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:06.517 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.340879+0000", 2020-01-16T00:13:06.517 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012836600000000001 2020-01-16T00:13:06.517 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.517 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.517 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:06.517 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.341007+0000", 2020-01-16T00:13:06.518 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019871900000000001 2020-01-16T00:13:06.518 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.518 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.518 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:06.518 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.341206+0000", 2020-01-16T00:13:06.518 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001501528 2020-01-16T00:13:06.518 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.519 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.519 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:06.519 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.342707+0000", 2020-01-16T00:13:06.519 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000300422 2020-01-16T00:13:06.519 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.519 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.519 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:06.519 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343008+0000", 2020-01-16T00:13:06.520 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00074073800000000001 2020-01-16T00:13:06.520 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.520 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.520 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:06.520 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343748+0000", 2020-01-16T00:13:06.520 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8270000000000003e-05 2020-01-16T00:13:06.520 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.521 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.521 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:06.521 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343807+0000", 2020-01-16T00:13:06.521 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8032000000000002e-05 2020-01-16T00:13:06.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.522 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.522 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:06.522 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343875+0000", 2020-01-16T00:13:06.522 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0090004249999999994 2020-01-16T00:13:06.522 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.522 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:06.522 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.523 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.523 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.523 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1919 3.a 3:5069665e:::smithi12313934-2:head [setxattr _header (55) in=7b,truncate 3479049] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:13:06.523 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:57.335061+0000", 2020-01-16T00:13:06.523 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 8.9947501429999992, 2020-01-16T00:13:06.523 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0089426209999999996, 2020-01-16T00:13:06.523 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:06.524 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:06.524 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:06.524 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:06.524 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:06.524 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1919 2020-01-16T00:13:06.524 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.524 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:06.524 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.525 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:06.525 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335061+0000", 2020-01-16T00:13:06.525 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:06.525 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.525 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.525 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:06.525 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335061+0000", 2020-01-16T00:13:06.525 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:06.526 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.526 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.526 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:06.526 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335059+0000", 2020-01-16T00:13:06.526 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5460000000000001e-05 2020-01-16T00:13:06.526 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.526 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.527 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:06.527 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335075+0000", 2020-01-16T00:13:06.527 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6120000000000002e-05 2020-01-16T00:13:06.527 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.527 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.527 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:06.527 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335091+0000", 2020-01-16T00:13:06.528 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0587999999999999e-05 2020-01-16T00:13:06.528 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.528 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.528 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:06.528 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335101+0000", 2020-01-16T00:13:06.528 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00027255500000000001 2020-01-16T00:13:06.528 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.529 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.529 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:06.529 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335374+0000", 2020-01-16T00:13:06.529 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016375999999999999 2020-01-16T00:13:06.529 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.529 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.530 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:06.530 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335538+0000", 2020-01-16T00:13:06.530 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0063584440000000004 2020-01-16T00:13:06.530 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.530 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.530 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:06.530 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.341896+0000", 2020-01-16T00:13:06.531 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012864300000000001 2020-01-16T00:13:06.531 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.531 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.531 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:06.531 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.342025+0000", 2020-01-16T00:13:06.531 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019981299999999999 2020-01-16T00:13:06.532 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.532 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.532 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:06.532 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.342225+0000", 2020-01-16T00:13:06.533 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00048735800000000002 2020-01-16T00:13:06.533 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.533 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.533 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:06.533 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.342712+0000", 2020-01-16T00:13:06.534 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00047628299999999998 2020-01-16T00:13:06.534 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.534 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.534 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:06.534 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343188+0000", 2020-01-16T00:13:06.534 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000718606 2020-01-16T00:13:06.534 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.535 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.535 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:06.535 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343907+0000", 2020-01-16T00:13:06.535 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7583999999999998e-05 2020-01-16T00:13:06.535 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.535 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.535 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:06.535 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343944+0000", 2020-01-16T00:13:06.536 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9209000000000002e-05 2020-01-16T00:13:06.536 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.536 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.536 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:06.536 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.344004+0000", 2020-01-16T00:13:06.536 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0089426209999999996 2020-01-16T00:13:06.536 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.536 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:06.537 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.537 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.537 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.537 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1920 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e31)", 2020-01-16T00:13:06.537 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:57.335132+0000", 2020-01-16T00:13:06.537 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 8.9946793970000005, 2020-01-16T00:13:06.537 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0091378889999999997, 2020-01-16T00:13:06.537 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:06.538 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:06.538 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:06.538 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:06.538 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:06.538 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1920 2020-01-16T00:13:06.538 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.538 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:06.539 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.539 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:06.539 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335132+0000", 2020-01-16T00:13:06.539 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:06.539 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.539 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.539 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:06.539 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335132+0000", 2020-01-16T00:13:06.540 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:06.540 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.540 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.540 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:06.540 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335130+0000", 2020-01-16T00:13:06.541 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0339999999999997e-06 2020-01-16T00:13:06.541 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.541 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.541 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:06.541 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335138+0000", 2020-01-16T00:13:06.541 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5296999999999999e-05 2020-01-16T00:13:06.541 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.541 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.542 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:06.542 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335154+0000", 2020-01-16T00:13:06.542 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9110000000000005e-06 2020-01-16T00:13:06.542 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.542 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.542 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:06.542 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335164+0000", 2020-01-16T00:13:06.543 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00044850000000000001 2020-01-16T00:13:06.543 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.543 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.543 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:06.543 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335612+0000", 2020-01-16T00:13:06.543 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2644000000000003e-05 2020-01-16T00:13:06.544 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.544 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.544 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:06.544 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.335705+0000", 2020-01-16T00:13:06.544 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0068223650000000004 2020-01-16T00:13:06.544 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.544 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.545 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:06.545 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.342527+0000", 2020-01-16T00:13:06.545 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1173999999999994e-05 2020-01-16T00:13:06.545 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.545 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.545 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:06.545 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.342588+0000", 2020-01-16T00:13:06.545 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014580769999999999 2020-01-16T00:13:06.546 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.546 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.546 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:06.546 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.344046+0000", 2020-01-16T00:13:06.546 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000106702 2020-01-16T00:13:06.546 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.546 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.546 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:06.547 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.344153+0000", 2020-01-16T00:13:06.547 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011658999999999999 2020-01-16T00:13:06.547 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.547 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.547 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:06.547 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.344270+0000", 2020-01-16T00:13:06.547 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0091378889999999997 2020-01-16T00:13:06.547 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.548 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:06.548 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.548 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.548 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.548 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2114 3.a 3:55e7052a:::smithi12313934-7:head [write 682417~473109] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:13:06.548 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:01.650529+0000", 2020-01-16T00:13:06.548 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 4.6792822139999997, 2020-01-16T00:13:06.548 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0093900500000000005, 2020-01-16T00:13:06.549 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:06.549 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:06.549 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:06.549 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:06.549 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:06.549 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2114 2020-01-16T00:13:06.549 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.550 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:06.550 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.550 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:06.550 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.650529+0000", 2020-01-16T00:13:06.550 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:06.550 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.550 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.550 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:06.551 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.650529+0000", 2020-01-16T00:13:06.551 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:06.551 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.551 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.551 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:06.551 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.650526+0000", 2020-01-16T00:13:06.551 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00074189500000000001 2020-01-16T00:13:06.551 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.552 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.552 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:06.552 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.651267+0000", 2020-01-16T00:13:06.552 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0999e-05 2020-01-16T00:13:06.552 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.552 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.552 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:06.552 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.651288+0000", 2020-01-16T00:13:06.553 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8230000000000006e-06 2020-01-16T00:13:06.553 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.553 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.553 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:06.553 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.651298+0000", 2020-01-16T00:13:06.553 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019944599999999999 2020-01-16T00:13:06.554 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.554 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.554 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:06.554 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.651498+0000", 2020-01-16T00:13:06.554 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0808e-05 2020-01-16T00:13:06.554 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.555 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.555 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op must wait for map", 2020-01-16T00:13:06.555 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.651529+0000", 2020-01-16T00:13:06.555 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0027072860000000002 2020-01-16T00:13:06.555 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.555 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.556 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:06.556 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.654236+0000", 2020-01-16T00:13:06.556 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010767700000000001 2020-01-16T00:13:06.556 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.556 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.556 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:06.557 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.654344+0000", 2020-01-16T00:13:06.557 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000261341 2020-01-16T00:13:06.557 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.557 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.557 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:06.557 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.654605+0000", 2020-01-16T00:13:06.557 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001537737 2020-01-16T00:13:06.558 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.558 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.558 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:06.558 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.656143+0000", 2020-01-16T00:13:06.558 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00302163 2020-01-16T00:13:06.558 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.558 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.558 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:06.559 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.659164+0000", 2020-01-16T00:13:06.559 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00063603599999999998 2020-01-16T00:13:06.559 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.559 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.559 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:06.559 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.659800+0000", 2020-01-16T00:13:06.559 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6470000000000006e-05 2020-01-16T00:13:06.559 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.560 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.560 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:06.560 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.659867+0000", 2020-01-16T00:13:06.560 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2218e-05 2020-01-16T00:13:06.560 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.560 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.560 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:06.561 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.659919+0000", 2020-01-16T00:13:06.561 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0093900500000000005 2020-01-16T00:13:06.561 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.561 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:06.561 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.561 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.561 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.561 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.4488.0:2260 3.d e35/20 3:b76659c4:::smithi12313934-47:head v 35'145, mlcod=35'143)", 2020-01-16T00:13:06.562 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:06.054154+0000", 2020-01-16T00:13:06.562 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.27565709999999999, 2020-01-16T00:13:06.562 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0085279180000000007, 2020-01-16T00:13:06.562 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:06.562 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:06.562 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:06.562 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.562 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:06.563 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.054154+0000", 2020-01-16T00:13:06.563 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:06.563 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.563 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.563 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:06.563 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.054154+0000", 2020-01-16T00:13:06.563 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:13:06.564 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.564 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.564 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:06.564 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.054150+0000", 2020-01-16T00:13:06.564 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00034144699999999999 2020-01-16T00:13:06.564 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.564 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.565 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:06.565 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.054491+0000", 2020-01-16T00:13:06.565 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1651e-05 2020-01-16T00:13:06.565 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.565 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.565 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:06.565 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.054513+0000", 2020-01-16T00:13:06.565 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.8859999999999993e-06 2020-01-16T00:13:06.566 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.566 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.566 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:06.566 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.054522+0000", 2020-01-16T00:13:06.566 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016930999999999999 2020-01-16T00:13:06.566 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.566 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.566 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:06.567 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.054691+0000", 2020-01-16T00:13:06.567 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3047999999999999e-05 2020-01-16T00:13:06.567 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.567 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.567 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:06.567 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.054724+0000", 2020-01-16T00:13:06.567 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0079169719999999996 2020-01-16T00:13:06.567 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.568 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.568 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:06.568 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.062641+0000", 2020-01-16T00:13:06.568 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0636999999999997e-05 2020-01-16T00:13:06.568 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:06.568 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:06.568 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:06.568 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.062682+0000", 2020-01-16T00:13:06.569 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0085279180000000007 2020-01-16T00:13:06.569 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.569 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:06.569 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.569 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:06.569 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:06.569 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:06.571 INFO:tasks.rados.rados.0.smithi123.stdout:1007: finishing write tid 1 to smithi12313934-48 2020-01-16T00:13:06.571 INFO:tasks.rados.rados.0.smithi123.stdout:1007: finishing write tid 2 to smithi12313934-48 2020-01-16T00:13:06.571 INFO:tasks.rados.rados.0.smithi123.stdout:1005: finishing write tid 4 to smithi12313934-24 2020-01-16T00:13:06.572 INFO:tasks.rados.rados.0.smithi123.stdout:1005: finishing write tid 5 to smithi12313934-24 2020-01-16T00:13:06.572 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 24 v 91 (ObjNum 244 snap 0 seq_num 244) dirty exists 2020-01-16T00:13:06.572 INFO:tasks.rados.rados.0.smithi123.stdout:1005: left oid 24 (ObjNum 244 snap 0 seq_num 244) 2020-01-16T00:13:06.572 INFO:tasks.rados.rados.0.smithi123.stdout:1008: finishing write tid 1 to smithi12313934-3 2020-01-16T00:13:06.572 INFO:tasks.rados.rados.0.smithi123.stdout:1008: finishing write tid 2 to smithi12313934-3 2020-01-16T00:13:06.572 INFO:tasks.rados.rados.0.smithi123.stdout:1008: finishing write tid 4 to smithi12313934-3 2020-01-16T00:13:06.572 INFO:tasks.rados.rados.0.smithi123.stdout:1008: finishing write tid 5 to smithi12313934-3 2020-01-16T00:13:06.572 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 3 v 73 (ObjNum 246 snap 0 seq_num 246) dirty exists 2020-01-16T00:13:06.573 INFO:tasks.rados.rados.0.smithi123.stdout:1008: left oid 3 (ObjNum 246 snap 0 seq_num 246) 2020-01-16T00:13:06.573 INFO:tasks.rados.rados.0.smithi123.stdout:1007: finishing write tid 4 to smithi12313934-48 2020-01-16T00:13:06.573 INFO:tasks.rados.rados.0.smithi123.stdout:1007: finishing write tid 5 to smithi12313934-48 2020-01-16T00:13:06.573 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 48 v 95 (ObjNum 245 snap 0 seq_num 245) dirty exists 2020-01-16T00:13:06.573 INFO:tasks.rados.rados.0.smithi123.stdout:1007: left oid 48 (ObjNum 245 snap 0 seq_num 245) 2020-01-16T00:13:06.573 INFO:tasks.rados.rados.0.smithi123.stdout:1009: finishing write tid 1 to smithi12313934-36 2020-01-16T00:13:06.574 INFO:tasks.rados.rados.0.smithi123.stdout:1009: finishing write tid 2 to smithi12313934-36 2020-01-16T00:13:06.574 INFO:tasks.rados.rados.0.smithi123.stdout:1009: finishing write tid 4 to smithi12313934-36 2020-01-16T00:13:06.574 INFO:tasks.rados.rados.0.smithi123.stdout:1009: finishing write tid 5 to smithi12313934-36 2020-01-16T00:13:06.574 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 36 v 54 (ObjNum 247 snap 0 seq_num 247) dirty exists 2020-01-16T00:13:06.575 INFO:tasks.rados.rados.0.smithi123.stdout:1009: left oid 36 (ObjNum 247 snap 0 seq_num 247) 2020-01-16T00:13:06.575 INFO:tasks.rados.rados.0.smithi123.stdout:1010: finishing write tid 1 to smithi12313934-20 2020-01-16T00:13:06.575 INFO:tasks.rados.rados.0.smithi123.stdout:1010: finishing write tid 2 to smithi12313934-20 2020-01-16T00:13:06.575 INFO:tasks.rados.rados.0.smithi123.stdout:1010: finishing write tid 4 to smithi12313934-20 2020-01-16T00:13:06.575 INFO:tasks.rados.rados.0.smithi123.stdout:1010: finishing write tid 5 to smithi12313934-20 2020-01-16T00:13:06.575 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 20 v 144 (ObjNum 248 snap 0 seq_num 248) dirty exists 2020-01-16T00:13:06.575 INFO:tasks.rados.rados.0.smithi123.stdout:1010: left oid 20 (ObjNum 248 snap 0 seq_num 248) 2020-01-16T00:13:06.575 INFO:tasks.rados.rados.0.smithi123.stdout:1011: finishing write tid 1 to smithi12313934-15 2020-01-16T00:13:06.576 INFO:tasks.rados.rados.0.smithi123.stdout:1004: done (15 left) 2020-01-16T00:13:06.576 INFO:tasks.rados.rados.0.smithi123.stdout:1005: done (14 left) 2020-01-16T00:13:06.576 INFO:tasks.rados.rados.0.smithi123.stdout:1006: done (13 left) 2020-01-16T00:13:06.576 INFO:tasks.rados.rados.0.smithi123.stdout:1007: done (12 left) 2020-01-16T00:13:06.576 INFO:tasks.rados.rados.0.smithi123.stdout:1008: done (11 left) 2020-01-16T00:13:06.576 INFO:tasks.rados.rados.0.smithi123.stdout:1009: done (10 left) 2020-01-16T00:13:06.576 INFO:tasks.rados.rados.0.smithi123.stdout:1010: done (9 left) 2020-01-16T00:13:06.576 INFO:tasks.rados.rados.0.smithi123.stdout:1020: write oid 7 current snap is 0 2020-01-16T00:13:06.577 INFO:tasks.rados.rados.0.smithi123.stdout:1020: seq_num 252 ranges {635364=402397,1763227=472137} 2020-01-16T00:13:06.577 INFO:tasks.rados.rados.0.smithi123.stdout:1020: writing smithi12313934-7 from 635364 to 1037761 tid 1 2020-01-16T00:13:06.577 INFO:tasks.rados.rados.0.smithi123.stdout:1020: writing smithi12313934-7 from 1763227 to 2235364 tid 2 2020-01-16T00:13:06.577 INFO:tasks.rados.rados.0.smithi123.stdout:1013: expect (ObjNum 206 snap 0 seq_num 206) 2020-01-16T00:13:06.577 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:06.566+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:06.578 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-01-16T00:13:06.633 INFO:tasks.rados.rados.0.smithi123.stdout:1011: finishing write tid 2 to smithi12313934-15 2020-01-16T00:13:06.633 INFO:tasks.rados.rados.0.smithi123.stdout:1011: finishing write tid 4 to smithi12313934-15 2020-01-16T00:13:06.633 INFO:tasks.rados.rados.0.smithi123.stdout:1011: finishing write tid 5 to smithi12313934-15 2020-01-16T00:13:06.633 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 15 v 105 (ObjNum 249 snap 0 seq_num 249) dirty exists 2020-01-16T00:13:06.634 INFO:tasks.rados.rados.0.smithi123.stdout:1011: left oid 15 (ObjNum 249 snap 0 seq_num 249) 2020-01-16T00:13:06.634 INFO:tasks.rados.rados.0.smithi123.stdout:1021: read oid 23 snap -1 2020-01-16T00:13:06.634 INFO:tasks.rados.rados.0.smithi123.stdout:1021: expect (ObjNum 227 snap 0 seq_num 227) 2020-01-16T00:13:06.634 INFO:tasks.rados.rados.0.smithi123.stdout:1011: done (10 left) 2020-01-16T00:13:06.634 INFO:tasks.rados.rados.0.smithi123.stdout:1013: done (9 left) 2020-01-16T00:13:06.634 INFO:tasks.rados.rados.0.smithi123.stdout:1022: write (excl) oid 6 current snap is 0 2020-01-16T00:13:06.634 INFO:tasks.rados.rados.0.smithi123.stdout:1022: seq_num 253 ranges {652171=477646,1821709=430462} 2020-01-16T00:13:06.639 INFO:tasks.rados.rados.0.smithi123.stdout:1022: writing smithi12313934-6 from 652171 to 1129817 tid 1 2020-01-16T00:13:06.646 INFO:tasks.rados.rados.0.smithi123.stdout:1022: writing smithi12313934-6 from 1821709 to 2252171 tid 2 2020-01-16T00:13:06.646 INFO:tasks.rados.rados.0.smithi123.stdout:1023: read oid 1 snap -1 2020-01-16T00:13:06.646 INFO:tasks.rados.rados.0.smithi123.stdout:1023: expect (ObjNum 240 snap 0 seq_num 240) 2020-01-16T00:13:06.647 INFO:tasks.rados.rados.0.smithi123.stdout:1024: read oid 43 snap -1 2020-01-16T00:13:06.647 INFO:tasks.rados.rados.0.smithi123.stdout:1024: expect (ObjNum 232 snap 0 seq_num 232) 2020-01-16T00:13:06.647 INFO:tasks.rados.rados.0.smithi123.stdout:1025: write (excl) oid 25 current snap is 0 2020-01-16T00:13:06.647 INFO:tasks.rados.rados.0.smithi123.stdout:1025: seq_num 254 ranges {668978=552895,1763838=505140} 2020-01-16T00:13:06.658 INFO:tasks.rados.rados.0.smithi123.stdout:1025: writing smithi12313934-25 from 668978 to 1221873 tid 1 2020-01-16T00:13:06.660 INFO:tasks.rados.rados.0.smithi123.stdout:1025: writing smithi12313934-25 from 1763838 to 2268978 tid 2 2020-01-16T00:13:06.660 INFO:tasks.rados.rados.0.smithi123.stdout:1026: read oid 32 snap -1 2020-01-16T00:13:06.660 INFO:tasks.rados.rados.0.smithi123.stdout:1026: expect (ObjNum 243 snap 0 seq_num 243) 2020-01-16T00:13:06.661 INFO:tasks.rados.rados.0.smithi123.stdout:1027: read oid 44 snap -1 2020-01-16T00:13:06.661 INFO:tasks.rados.rados.0.smithi123.stdout:1027: expect (ObjNum 228 snap 0 seq_num 228) 2020-01-16T00:13:06.661 INFO:tasks.rados.rados.0.smithi123.stdout:1028: write (excl) oid 41 current snap is 0 2020-01-16T00:13:06.661 INFO:tasks.rados.rados.0.smithi123.stdout:1028: seq_num 255 ranges {685785=628144,1822320=463465} 2020-01-16T00:13:06.668 INFO:tasks.rados.rados.0.smithi123.stdout:1028: writing smithi12313934-41 from 685785 to 1313929 tid 1 2020-01-16T00:13:06.671 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:13:06.669+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:13:06.673 INFO:tasks.rados.rados.0.smithi123.stdout:1028: writing smithi12313934-41 from 1822320 to 2285785 tid 2 2020-01-16T00:13:06.674 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:06.674 INFO:tasks.rados.rados.0.smithi123.stdout:1012: expect (ObjNum 194 snap 0 seq_num 194) 2020-01-16T00:13:06.695 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:06.695 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:06.695 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:13:06.695 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:06.706 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-01-16T00:13:06.771 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:06.769+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:06.795 INFO:teuthology.orchestra.run.smithi123.stderr:noscrub is set 2020-01-16T00:13:06.813 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:06.813 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:06.814 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:13:06.814 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:13:06.814 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:06.819 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-01-16T00:13:06.869 INFO:tasks.rados.rados.0.smithi123.stdout:1017: expect (ObjNum 239 snap 0 seq_num 239) 2020-01-16T00:13:06.872 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:06.869+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:06.966 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:06.966 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:13:06.966 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:13:06.966 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:13:06.966 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:06.967 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:939 3.4 3:2bdd9e88:::smithi12313934-4:head [write 1681102~474010] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:06.967 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.908392+0000", 2020-01-16T00:13:06.967 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 29.051557378999998, 2020-01-16T00:13:06.967 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019294697, 2020-01-16T00:13:06.967 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:06.967 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:06.967 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:06.968 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:06.968 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:06.968 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 939 2020-01-16T00:13:06.968 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:06.968 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:06.968 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:06.968 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:06.968 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908392+0000", 2020-01-16T00:13:06.969 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:06.969 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:06.969 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:06.969 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:06.969 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908392+0000", 2020-01-16T00:13:06.969 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:06.969 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:06.970 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:06.970 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:06.970 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908390+0000", 2020-01-16T00:13:06.970 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015440399999999999 2020-01-16T00:13:06.970 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:06.970 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:06.970 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:06.971 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908545+0000", 2020-01-16T00:13:06.971 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3926e-05 2020-01-16T00:13:06.971 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:06.971 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:06.971 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:06.971 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908559+0000", 2020-01-16T00:13:06.971 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2200000000000003e-06 2020-01-16T00:13:06.971 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:06.972 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:06.972 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:06.972 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908566+0000", 2020-01-16T00:13:06.972 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020594799999999998 2020-01-16T00:13:06.972 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:06.972 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:06.972 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:06.973 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910625+0000", 2020-01-16T00:13:06.973 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118672 2020-01-16T00:13:06.973 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:06.973 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:06.973 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:06.973 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910744+0000", 2020-01-16T00:13:06.973 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031940969999999999 2020-01-16T00:13:06.973 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:06.974 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:06.974 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:06.974 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.913938+0000", 2020-01-16T00:13:06.974 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000309621 2020-01-16T00:13:06.974 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:06.974 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:06.975 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:06.975 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.914248+0000", 2020-01-16T00:13:06.975 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.008818579 2020-01-16T00:13:06.975 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:06.975 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:06.975 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:06.975 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.923066+0000", 2020-01-16T00:13:06.976 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000108989 2020-01-16T00:13:06.976 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:06.976 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:06.976 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:06.976 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.923175+0000", 2020-01-16T00:13:06.976 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019128299999999999 2020-01-16T00:13:06.976 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:06.978 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:06.978 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:13:06.978 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.923367+0000", 2020-01-16T00:13:06.978 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0032083760000000002 2020-01-16T00:13:06.978 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:06.978 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:06.978 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:06.979 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926575+0000", 2020-01-16T00:13:06.979 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2475999999999999e-05 2020-01-16T00:13:06.979 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:06.979 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:06.979 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:06.979 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926607+0000", 2020-01-16T00:13:06.979 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00099399800000000006 2020-01-16T00:13:06.980 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:06.980 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:06.980 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:06.980 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927601+0000", 2020-01-16T00:13:06.980 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0958999999999999e-05 2020-01-16T00:13:06.980 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:06.981 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:06.981 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:06.981 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927642+0000", 2020-01-16T00:13:06.981 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4747999999999999e-05 2020-01-16T00:13:06.981 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:06.981 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:06.981 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:06.982 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927687+0000", 2020-01-16T00:13:06.982 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019294697 2020-01-16T00:13:06.982 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:06.982 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:06.982 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:06.982 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:06.982 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:06.982 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:940 3.4 3:2bdd9e88:::smithi12313934-4:head [write 2687912~589420] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:06.983 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.915144+0000", 2020-01-16T00:13:06.983 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 29.044806010999999, 2020-01-16T00:13:06.983 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016092538999999999, 2020-01-16T00:13:06.983 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:06.983 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:06.983 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:06.983 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:06.984 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:06.984 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 940 2020-01-16T00:13:06.984 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:06.984 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:06.984 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:06.984 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:06.984 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915144+0000", 2020-01-16T00:13:06.984 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:06.985 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:06.985 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:06.985 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:06.985 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915144+0000", 2020-01-16T00:13:06.985 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:06.986 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:06.986 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:06.986 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:06.986 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915141+0000", 2020-01-16T00:13:06.986 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016366300000000001 2020-01-16T00:13:06.986 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:06.986 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:06.986 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:06.987 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915305+0000", 2020-01-16T00:13:06.987 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4847e-05 2020-01-16T00:13:06.987 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:06.987 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:06.987 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:06.987 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915320+0000", 2020-01-16T00:13:06.987 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2599999999999999e-06 2020-01-16T00:13:06.987 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:06.988 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:06.988 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:06.988 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915327+0000", 2020-01-16T00:13:06.988 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4282000000000005e-05 2020-01-16T00:13:06.988 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:06.988 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:06.988 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:06.989 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915401+0000", 2020-01-16T00:13:06.989 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4554000000000006e-05 2020-01-16T00:13:06.989 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:06.989 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:06.989 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:06.989 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915466+0000", 2020-01-16T00:13:06.990 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0090821900000000004 2020-01-16T00:13:06.991 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:06.991 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:06.991 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:06.991 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.924548+0000", 2020-01-16T00:13:06.992 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000114696 2020-01-16T00:13:06.992 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:06.992 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:06.992 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:06.992 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.924663+0000", 2020-01-16T00:13:06.992 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000168651 2020-01-16T00:13:06.992 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:06.993 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:06.993 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:13:06.993 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.924832+0000", 2020-01-16T00:13:06.993 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018253430000000001 2020-01-16T00:13:06.993 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:06.993 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:06.993 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:06.994 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926657+0000", 2020-01-16T00:13:06.994 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9957999999999999e-05 2020-01-16T00:13:06.994 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:06.994 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:06.994 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:06.994 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926697+0000", 2020-01-16T00:13:06.994 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0044685250000000001 2020-01-16T00:13:06.994 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:06.995 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:06.995 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:06.995 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931165+0000", 2020-01-16T00:13:06.995 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9211000000000001e-05 2020-01-16T00:13:06.995 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:06.995 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:06.996 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:06.996 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931205+0000", 2020-01-16T00:13:06.996 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1699999999999998e-05 2020-01-16T00:13:06.996 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:06.996 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:06.996 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:06.996 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931236+0000", 2020-01-16T00:13:06.997 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016092538999999999 2020-01-16T00:13:06.997 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:06.997 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:06.997 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:06.997 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:06.997 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:06.997 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:941 3.4 3:2bdd9e88:::smithi12313934-4:head [setxattr _header (55) in=7b,truncate 3277332] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:06.997 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.915341+0000", 2020-01-16T00:13:06.998 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 29.044609192999999, 2020-01-16T00:13:06.998 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016603769000000001, 2020-01-16T00:13:06.998 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:06.998 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:06.998 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:06.998 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:06.998 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:06.999 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 941 2020-01-16T00:13:06.999 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:06.999 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:06.999 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:06.999 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:06.999 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915341+0000", 2020-01-16T00:13:06.999 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:07.000 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.000 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.000 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:07.000 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915341+0000", 2020-01-16T00:13:07.000 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:07.000 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.000 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.000 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:07.001 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915340+0000", 2020-01-16T00:13:07.001 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.896e-06 2020-01-16T00:13:07.001 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.001 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.001 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:07.001 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915346+0000", 2020-01-16T00:13:07.001 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2730000000000002e-06 2020-01-16T00:13:07.002 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.002 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.002 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:07.002 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915354+0000", 2020-01-16T00:13:07.002 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0616e-05 2020-01-16T00:13:07.002 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.002 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.002 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:07.003 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915374+0000", 2020-01-16T00:13:07.003 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000133487 2020-01-16T00:13:07.003 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.003 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.003 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.003 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915508+0000", 2020-01-16T00:13:07.003 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8971000000000003e-05 2020-01-16T00:13:07.004 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.004 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.004 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:07.004 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915557+0000", 2020-01-16T00:13:07.004 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010138787999999999 2020-01-16T00:13:07.004 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.004 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.005 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.005 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.925696+0000", 2020-01-16T00:13:07.005 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012825400000000001 2020-01-16T00:13:07.005 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.005 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.005 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:07.006 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.925824+0000", 2020-01-16T00:13:07.006 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017706500000000001 2020-01-16T00:13:07.006 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.006 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.006 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:13:07.006 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926001+0000", 2020-01-16T00:13:07.006 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00079988900000000003 2020-01-16T00:13:07.007 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.007 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.007 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:07.007 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926801+0000", 2020-01-16T00:13:07.007 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119324 2020-01-16T00:13:07.007 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.007 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.007 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:07.008 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926920+0000", 2020-01-16T00:13:07.008 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048879910000000004 2020-01-16T00:13:07.008 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.008 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.008 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:07.008 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931808+0000", 2020-01-16T00:13:07.008 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6999999999999997e-05 2020-01-16T00:13:07.009 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.009 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.009 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:07.009 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931855+0000", 2020-01-16T00:13:07.009 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9092999999999996e-05 2020-01-16T00:13:07.009 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.009 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.010 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:07.010 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931944+0000", 2020-01-16T00:13:07.010 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016603769000000001 2020-01-16T00:13:07.010 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.010 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.010 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.010 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.010 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.011 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:942 3.4 3:2bdd9e88:::smithi12313934-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:13:07.011 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.915389+0000", 2020-01-16T00:13:07.011 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 29.044560827000002, 2020-01-16T00:13:07.011 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016820563, 2020-01-16T00:13:07.011 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:07.011 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:07.011 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:07.012 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:07.012 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:07.012 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 942 2020-01-16T00:13:07.012 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.012 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:07.012 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.012 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:07.012 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915389+0000", 2020-01-16T00:13:07.013 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:07.013 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.013 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.013 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:07.013 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915389+0000", 2020-01-16T00:13:07.013 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:07.013 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.014 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.014 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:07.014 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915388+0000", 2020-01-16T00:13:07.014 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.683e-06 2020-01-16T00:13:07.014 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.014 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.014 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:07.015 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915391+0000", 2020-01-16T00:13:07.015 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4839999999999998e-06 2020-01-16T00:13:07.015 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.015 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.015 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:07.015 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915399+0000", 2020-01-16T00:13:07.016 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2259999999999999e-06 2020-01-16T00:13:07.016 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.016 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.016 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:07.016 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915403+0000", 2020-01-16T00:13:07.016 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00031301199999999999 2020-01-16T00:13:07.016 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.016 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.017 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.017 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915716+0000", 2020-01-16T00:13:07.017 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2079999999999998e-05 2020-01-16T00:13:07.017 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.017 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.017 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:07.017 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915748+0000", 2020-01-16T00:13:07.018 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010547076000000001 2020-01-16T00:13:07.018 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.018 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.018 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.018 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926295+0000", 2020-01-16T00:13:07.018 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8040000000000005e-05 2020-01-16T00:13:07.018 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.018 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.019 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:07.019 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926373+0000", 2020-01-16T00:13:07.019 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056061990000000001 2020-01-16T00:13:07.019 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.019 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.019 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.019 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931979+0000", 2020-01-16T00:13:07.020 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011540399999999999 2020-01-16T00:13:07.020 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.020 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.020 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:07.020 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932095+0000", 2020-01-16T00:13:07.020 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011482899999999999 2020-01-16T00:13:07.020 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.021 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.021 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:07.021 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932210+0000", 2020-01-16T00:13:07.021 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016820563 2020-01-16T00:13:07.021 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.021 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.021 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.022 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.022 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.022 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:957 3.0 3:00c146d5:::smithi12313934-18:head [write 510946~632952] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:07.022 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.033290+0000", 2020-01-16T00:13:07.022 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 28.926659990000001, 2020-01-16T00:13:07.022 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016764102999999999, 2020-01-16T00:13:07.022 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:07.023 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:07.023 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:07.024 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:07.024 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:07.024 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 957 2020-01-16T00:13:07.024 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.024 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:07.024 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.024 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:07.025 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033290+0000", 2020-01-16T00:13:07.025 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:07.025 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.025 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.025 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:07.025 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033290+0000", 2020-01-16T00:13:07.025 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999952 2020-01-16T00:13:07.026 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.026 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.026 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:07.026 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033285+0000", 2020-01-16T00:13:07.027 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000405917 2020-01-16T00:13:07.027 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.027 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.027 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:07.027 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033691+0000", 2020-01-16T00:13:07.027 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6585999999999997e-05 2020-01-16T00:13:07.027 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.027 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.028 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:07.028 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033738+0000", 2020-01-16T00:13:07.028 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8374000000000001e-05 2020-01-16T00:13:07.028 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.028 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.028 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:07.028 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033756+0000", 2020-01-16T00:13:07.028 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013002900000000001 2020-01-16T00:13:07.029 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.029 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.029 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.030 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033886+0000", 2020-01-16T00:13:07.030 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035882600000000002 2020-01-16T00:13:07.030 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.030 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.030 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:07.030 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.034245+0000", 2020-01-16T00:13:07.031 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0072798489999999997 2020-01-16T00:13:07.031 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.031 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.031 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.031 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.041525+0000", 2020-01-16T00:13:07.031 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000140488 2020-01-16T00:13:07.032 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.032 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.032 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:07.032 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.041665+0000", 2020-01-16T00:13:07.033 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000233766 2020-01-16T00:13:07.033 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.033 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.033 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:07.033 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.041899+0000", 2020-01-16T00:13:07.033 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0035766539999999999 2020-01-16T00:13:07.033 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.034 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.034 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:07.034 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.045475+0000", 2020-01-16T00:13:07.034 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013333150000000001 2020-01-16T00:13:07.034 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.034 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.035 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:07.036 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.046809+0000", 2020-01-16T00:13:07.036 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031447860000000001 2020-01-16T00:13:07.036 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.036 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.037 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:07.037 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.049954+0000", 2020-01-16T00:13:07.037 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2318000000000003e-05 2020-01-16T00:13:07.037 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.038 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.038 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:07.038 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.050006+0000", 2020-01-16T00:13:07.038 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7972000000000001e-05 2020-01-16T00:13:07.039 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.039 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.039 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:07.039 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.050054+0000", 2020-01-16T00:13:07.039 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016764102999999999 2020-01-16T00:13:07.040 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.040 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.040 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.040 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.041 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.041 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1168 3.0 3:03b92cf0:::smithi12313934-26:head [write 2905319~482223] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:13:07.041 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:41.672662+0000", 2020-01-16T00:13:07.041 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 25.287288144000001, 2020-01-16T00:13:07.042 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015446582, 2020-01-16T00:13:07.042 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:07.042 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:07.042 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:07.042 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:07.043 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:07.043 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1168 2020-01-16T00:13:07.043 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.044 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:07.044 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.045 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:07.045 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672662+0000", 2020-01-16T00:13:07.045 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:07.045 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.045 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.045 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:07.046 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672662+0000", 2020-01-16T00:13:07.046 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:07.046 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.046 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.046 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:07.046 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672660+0000", 2020-01-16T00:13:07.046 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013924799999999999 2020-01-16T00:13:07.047 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.047 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.047 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:07.047 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672799+0000", 2020-01-16T00:13:07.047 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1304000000000001e-05 2020-01-16T00:13:07.047 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.047 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.047 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:07.048 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672810+0000", 2020-01-16T00:13:07.048 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2379999999999999e-06 2020-01-16T00:13:07.048 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.048 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.048 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:07.048 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672815+0000", 2020-01-16T00:13:07.048 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9329000000000002e-05 2020-01-16T00:13:07.049 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.049 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.049 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.049 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672854+0000", 2020-01-16T00:13:07.049 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010573939999999999 2020-01-16T00:13:07.050 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.050 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.050 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:07.050 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.673911+0000", 2020-01-16T00:13:07.050 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.01042181 2020-01-16T00:13:07.050 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.050 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.051 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.051 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.684333+0000", 2020-01-16T00:13:07.051 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012700499999999999 2020-01-16T00:13:07.051 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.051 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.051 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:07.051 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.684460+0000", 2020-01-16T00:13:07.051 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019923299999999999 2020-01-16T00:13:07.052 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.052 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.052 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:07.052 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.684659+0000", 2020-01-16T00:13:07.052 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020339300000000002 2020-01-16T00:13:07.052 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.052 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.053 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:07.053 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.686693+0000", 2020-01-16T00:13:07.053 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023967899999999999 2020-01-16T00:13:07.053 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.053 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.053 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:07.053 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.686933+0000", 2020-01-16T00:13:07.054 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010489169999999999 2020-01-16T00:13:07.054 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.054 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.054 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:07.054 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.687982+0000", 2020-01-16T00:13:07.054 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3857999999999997e-05 2020-01-16T00:13:07.055 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.055 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.055 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:07.055 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.688056+0000", 2020-01-16T00:13:07.055 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2324999999999998e-05 2020-01-16T00:13:07.055 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.055 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.056 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:07.056 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.688108+0000", 2020-01-16T00:13:07.056 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015446582 2020-01-16T00:13:07.056 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.056 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.056 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.056 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.057 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.057 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1169 3.0 3:03b92cf0:::smithi12313934-26:head [setxattr _header (56) in=7b,truncate 3798349] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:13:07.057 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:41.672823+0000", 2020-01-16T00:13:07.057 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 25.287127237, 2020-01-16T00:13:07.057 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015457745, 2020-01-16T00:13:07.057 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:07.057 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:07.058 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:07.058 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:07.058 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:07.058 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1169 2020-01-16T00:13:07.058 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.058 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:07.058 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.058 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:07.059 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672823+0000", 2020-01-16T00:13:07.059 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:07.059 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.059 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.059 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:07.059 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672823+0000", 2020-01-16T00:13:07.059 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:07.059 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.060 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.060 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:07.060 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672822+0000", 2020-01-16T00:13:07.060 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.685e-06 2020-01-16T00:13:07.060 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.060 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.060 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:07.061 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672827+0000", 2020-01-16T00:13:07.061 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.485e-06 2020-01-16T00:13:07.061 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.061 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.061 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:07.061 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672832+0000", 2020-01-16T00:13:07.061 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6440000000000002e-06 2020-01-16T00:13:07.061 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.062 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.062 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:07.062 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672836+0000", 2020-01-16T00:13:07.062 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00112415 2020-01-16T00:13:07.062 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.062 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.062 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.063 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.673960+0000", 2020-01-16T00:13:07.063 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5918999999999998e-05 2020-01-16T00:13:07.063 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.063 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.063 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:07.063 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.674016+0000", 2020-01-16T00:13:07.063 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011614579999999999 2020-01-16T00:13:07.064 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.064 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.064 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.064 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.685630+0000", 2020-01-16T00:13:07.064 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011684399999999999 2020-01-16T00:13:07.064 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.065 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.065 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:07.065 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.685747+0000", 2020-01-16T00:13:07.065 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020774199999999999 2020-01-16T00:13:07.065 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.065 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.065 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:07.066 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.685955+0000", 2020-01-16T00:13:07.066 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00095868400000000001 2020-01-16T00:13:07.066 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.066 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.066 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:07.066 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.686914+0000", 2020-01-16T00:13:07.066 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5278000000000001e-05 2020-01-16T00:13:07.066 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.067 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.067 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:07.067 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.686939+0000", 2020-01-16T00:13:07.067 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001227611 2020-01-16T00:13:07.067 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.067 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.067 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:07.068 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.688166+0000", 2020-01-16T00:13:07.068 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1724999999999997e-05 2020-01-16T00:13:07.068 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.068 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.068 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:07.068 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.688218+0000", 2020-01-16T00:13:07.068 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2057000000000006e-05 2020-01-16T00:13:07.069 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.069 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.069 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:07.069 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.688280+0000", 2020-01-16T00:13:07.069 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015457745 2020-01-16T00:13:07.069 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.069 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.070 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.070 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.070 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.070 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1170 3.0 3:03b92cf0:::smithi12313934-26:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-01-16T00:13:07.070 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:41.672848+0000", 2020-01-16T00:13:07.070 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 25.287101668999998, 2020-01-16T00:13:07.070 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015733389, 2020-01-16T00:13:07.070 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:07.071 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:07.071 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:07.071 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:07.071 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:07.071 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1170 2020-01-16T00:13:07.071 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.071 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:07.072 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.072 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:07.072 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672848+0000", 2020-01-16T00:13:07.072 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:07.072 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.072 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.072 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:07.072 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672848+0000", 2020-01-16T00:13:07.073 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:07.073 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.073 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.073 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:07.073 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672847+0000", 2020-01-16T00:13:07.073 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.303e-06 2020-01-16T00:13:07.073 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.074 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.074 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:07.074 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672851+0000", 2020-01-16T00:13:07.074 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4970000000000001e-06 2020-01-16T00:13:07.074 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.074 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.075 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:07.075 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672856+0000", 2020-01-16T00:13:07.075 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8870000000000002e-06 2020-01-16T00:13:07.075 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.075 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.075 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:07.075 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672860+0000", 2020-01-16T00:13:07.076 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0011720140000000001 2020-01-16T00:13:07.076 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.076 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.076 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.076 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.674032+0000", 2020-01-16T00:13:07.076 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2577000000000003e-05 2020-01-16T00:13:07.076 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.077 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.077 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:07.077 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.674065+0000", 2020-01-16T00:13:07.077 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012264538 2020-01-16T00:13:07.077 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.077 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.078 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.078 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.686329+0000", 2020-01-16T00:13:07.078 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000106715 2020-01-16T00:13:07.078 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.078 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.078 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:07.078 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.686436+0000", 2020-01-16T00:13:07.079 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018880800000000001 2020-01-16T00:13:07.079 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.079 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.079 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.079 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.688324+0000", 2020-01-16T00:13:07.079 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012818500000000001 2020-01-16T00:13:07.079 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.079 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.080 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:07.080 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.688452+0000", 2020-01-16T00:13:07.080 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000129197 2020-01-16T00:13:07.080 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.080 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.080 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:07.080 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.688581+0000", 2020-01-16T00:13:07.081 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015733389 2020-01-16T00:13:07.081 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.081 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.081 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.081 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.081 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.081 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1394 3.4 3:240c7503:::smithi12313934-35:head [setxattr _header (56) in=7b,truncate 2285752] snapc 0=[] ondisk+write+known_if_redirected e28)", 2020-01-16T00:13:07.081 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:46.625161+0000", 2020-01-16T00:13:07.082 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 20.334788872000001, 2020-01-16T00:13:07.082 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015636654, 2020-01-16T00:13:07.082 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:07.082 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:07.082 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:07.082 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:07.082 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:07.083 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1394 2020-01-16T00:13:07.083 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.083 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:07.083 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.083 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:07.083 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625161+0000", 2020-01-16T00:13:07.083 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:07.084 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.084 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.084 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:07.084 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625161+0000", 2020-01-16T00:13:07.084 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:07.084 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.084 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.085 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:07.085 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625160+0000", 2020-01-16T00:13:07.085 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3960000000000002e-06 2020-01-16T00:13:07.085 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.085 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.086 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:07.086 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625164+0000", 2020-01-16T00:13:07.086 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7199999999999997e-06 2020-01-16T00:13:07.086 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.086 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.086 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:07.086 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625168+0000", 2020-01-16T00:13:07.087 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2960000000000001e-06 2020-01-16T00:13:07.087 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.087 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.087 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:07.087 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625171+0000", 2020-01-16T00:13:07.087 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001705972 2020-01-16T00:13:07.087 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.088 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.088 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.088 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.626877+0000", 2020-01-16T00:13:07.088 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039484300000000002 2020-01-16T00:13:07.088 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.088 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.088 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:07.088 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630825+0000", 2020-01-16T00:13:07.089 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0068775310000000001 2020-01-16T00:13:07.089 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.089 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.089 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.089 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.637703+0000", 2020-01-16T00:13:07.089 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013884100000000001 2020-01-16T00:13:07.089 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.090 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.090 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:07.090 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.637842+0000", 2020-01-16T00:13:07.090 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00024336400000000001 2020-01-16T00:13:07.090 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.090 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.090 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:13:07.091 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.638085+0000", 2020-01-16T00:13:07.091 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010183900000000001 2020-01-16T00:13:07.091 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.091 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.091 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:07.091 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.639103+0000", 2020-01-16T00:13:07.091 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012246699999999999 2020-01-16T00:13:07.091 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.092 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.092 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:07.092 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.639226+0000", 2020-01-16T00:13:07.092 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0014329099999999999 2020-01-16T00:13:07.092 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.092 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.092 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:07.093 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.640659+0000", 2020-01-16T00:13:07.093 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8109000000000002e-05 2020-01-16T00:13:07.093 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.093 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.093 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:07.093 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.640717+0000", 2020-01-16T00:13:07.093 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0784000000000004e-05 2020-01-16T00:13:07.094 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.094 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.094 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:07.094 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.640798+0000", 2020-01-16T00:13:07.094 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015636654 2020-01-16T00:13:07.094 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.094 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.095 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.095 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.095 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.095 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1395 3.4 3:240c7503:::smithi12313934-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e28)", 2020-01-16T00:13:07.095 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:46.625191+0000", 2020-01-16T00:13:07.095 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 20.334758661999999, 2020-01-16T00:13:07.096 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015878716000000001, 2020-01-16T00:13:07.096 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:07.096 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:07.096 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:07.096 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:07.096 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:07.097 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1395 2020-01-16T00:13:07.097 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.097 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:07.097 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.097 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:07.097 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625191+0000", 2020-01-16T00:13:07.098 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:07.098 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.098 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.098 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:07.098 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625191+0000", 2020-01-16T00:13:07.098 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:07.098 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.099 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.099 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:07.099 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625191+0000", 2020-01-16T00:13:07.099 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.1090000000000001e-06 2020-01-16T00:13:07.099 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.099 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.099 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:07.099 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625193+0000", 2020-01-16T00:13:07.100 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2730000000000008e-06 2020-01-16T00:13:07.100 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.100 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.100 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:07.100 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625202+0000", 2020-01-16T00:13:07.100 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3700000000000002e-06 2020-01-16T00:13:07.100 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.100 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.101 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:07.101 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625204+0000", 2020-01-16T00:13:07.101 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056573049999999996 2020-01-16T00:13:07.101 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.101 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.101 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.101 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630862+0000", 2020-01-16T00:13:07.102 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6047000000000001e-05 2020-01-16T00:13:07.102 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.102 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.102 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:07.102 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630918+0000", 2020-01-16T00:13:07.102 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0077941180000000001 2020-01-16T00:13:07.102 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.102 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.103 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.103 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.638712+0000", 2020-01-16T00:13:07.103 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118096 2020-01-16T00:13:07.103 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.103 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.103 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:07.103 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.638830+0000", 2020-01-16T00:13:07.104 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0019986740000000002 2020-01-16T00:13:07.104 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.104 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.104 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.104 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.640829+0000", 2020-01-16T00:13:07.104 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012562399999999999 2020-01-16T00:13:07.104 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.104 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.105 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:07.105 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.640954+0000", 2020-01-16T00:13:07.105 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000115536 2020-01-16T00:13:07.105 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.105 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.105 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:07.106 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.641070+0000", 2020-01-16T00:13:07.106 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015878716000000001 2020-01-16T00:13:07.106 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.106 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.106 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.106 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.106 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.106 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1639 3.0 3:00c146d5:::smithi12313934-18:head [write 423576~401598] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:13:07.107 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.774317+0000", 2020-01-16T00:13:07.107 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 15.185633251, 2020-01-16T00:13:07.107 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018305219000000001, 2020-01-16T00:13:07.107 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:07.107 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:07.107 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:07.107 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:07.108 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:07.108 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1639 2020-01-16T00:13:07.108 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.108 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:07.108 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.108 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:07.108 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774317+0000", 2020-01-16T00:13:07.109 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:07.109 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.109 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.109 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:07.109 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774317+0000", 2020-01-16T00:13:07.109 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:07.109 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.110 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.110 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:07.110 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774313+0000", 2020-01-16T00:13:07.110 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000111173 2020-01-16T00:13:07.110 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.110 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.111 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:07.111 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774425+0000", 2020-01-16T00:13:07.111 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7598999999999999e-05 2020-01-16T00:13:07.111 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.111 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.111 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:07.111 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774442+0000", 2020-01-16T00:13:07.111 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0830000000000004e-06 2020-01-16T00:13:07.112 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.112 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.112 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:07.112 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774450+0000", 2020-01-16T00:13:07.112 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0017886309999999999 2020-01-16T00:13:07.112 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.112 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.112 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.113 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.776239+0000", 2020-01-16T00:13:07.113 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3789e-05 2020-01-16T00:13:07.113 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.113 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.113 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op must wait for map", 2020-01-16T00:13:07.113 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.776253+0000", 2020-01-16T00:13:07.113 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095730729999999997 2020-01-16T00:13:07.114 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.114 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.114 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.114 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.785826+0000", 2020-01-16T00:13:07.114 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014220300000000001 2020-01-16T00:13:07.114 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.114 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.114 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:07.115 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.785968+0000", 2020-01-16T00:13:07.115 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023618900000000001 2020-01-16T00:13:07.115 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.115 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.115 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:07.115 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.786204+0000", 2020-01-16T00:13:07.116 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0062297389999999998 2020-01-16T00:13:07.116 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.116 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.116 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:07.116 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792434+0000", 2020-01-16T00:13:07.116 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9635999999999998e-05 2020-01-16T00:13:07.116 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.117 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.117 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:07.117 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792473+0000", 2020-01-16T00:13:07.117 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6456000000000002e-05 2020-01-16T00:13:07.117 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.117 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.118 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:07.118 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792540+0000", 2020-01-16T00:13:07.118 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9873999999999998e-05 2020-01-16T00:13:07.118 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.118 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.119 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:07.119 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792580+0000", 2020-01-16T00:13:07.119 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1924e-05 2020-01-16T00:13:07.119 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.119 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.119 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:07.119 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792622+0000", 2020-01-16T00:13:07.119 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018305219000000001 2020-01-16T00:13:07.120 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.120 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.120 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.120 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.120 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.120 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1890 3.0 3:0b1f384f:::smithi12313934-33:head [write 2797915~596476] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:13:07.120 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.232534+0000", 2020-01-16T00:13:07.121 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 9.7274152800000007, 2020-01-16T00:13:07.121 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016106137, 2020-01-16T00:13:07.121 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:07.121 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:07.121 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:07.121 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:07.121 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:07.122 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1890 2020-01-16T00:13:07.122 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.122 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:07.122 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.122 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:07.122 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232534+0000", 2020-01-16T00:13:07.122 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:07.123 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.123 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.123 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:07.123 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232534+0000", 2020-01-16T00:13:07.123 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:07.123 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.123 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.123 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:07.124 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232532+0000", 2020-01-16T00:13:07.124 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00024232100000000001 2020-01-16T00:13:07.124 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.124 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.124 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:07.124 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232774+0000", 2020-01-16T00:13:07.124 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4601e-05 2020-01-16T00:13:07.125 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.125 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.125 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:07.125 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232789+0000", 2020-01-16T00:13:07.125 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8410000000000006e-06 2020-01-16T00:13:07.125 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.126 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.126 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:07.126 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232798+0000", 2020-01-16T00:13:07.126 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9756999999999997e-05 2020-01-16T00:13:07.126 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.126 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.126 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.127 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232857+0000", 2020-01-16T00:13:07.127 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4678000000000007e-05 2020-01-16T00:13:07.127 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.127 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.127 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:07.127 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232922+0000", 2020-01-16T00:13:07.127 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033861020000000002 2020-01-16T00:13:07.128 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.128 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.128 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.128 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.236308+0000", 2020-01-16T00:13:07.128 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000568996 2020-01-16T00:13:07.128 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.128 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.129 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:07.129 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.236877+0000", 2020-01-16T00:13:07.129 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0085444240000000001 2020-01-16T00:13:07.129 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.129 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.129 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.129 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.245422+0000", 2020-01-16T00:13:07.129 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5595999999999998e-05 2020-01-16T00:13:07.130 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.130 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.130 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:07.130 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.245507+0000", 2020-01-16T00:13:07.130 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016984300000000001 2020-01-16T00:13:07.130 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.130 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.131 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:07.131 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.245677+0000", 2020-01-16T00:13:07.131 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001649495 2020-01-16T00:13:07.131 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.131 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.131 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:07.131 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247326+0000", 2020-01-16T00:13:07.132 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023308399999999999 2020-01-16T00:13:07.132 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.132 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.132 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:07.132 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247560+0000", 2020-01-16T00:13:07.132 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00099501800000000007 2020-01-16T00:13:07.132 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.133 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.133 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:07.133 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248555+0000", 2020-01-16T00:13:07.133 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5936999999999998e-05 2020-01-16T00:13:07.133 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.133 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.134 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:07.134 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248591+0000", 2020-01-16T00:13:07.134 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0108999999999997e-05 2020-01-16T00:13:07.134 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.134 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.134 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:07.134 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248641+0000", 2020-01-16T00:13:07.135 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016106137 2020-01-16T00:13:07.135 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.135 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.135 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.135 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.135 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.135 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1891 3.0 3:0b1f384f:::smithi12313934-33:head [setxattr _header (56) in=7b,truncate 3395014] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:13:07.135 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.232813+0000", 2020-01-16T00:13:07.136 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 9.7271364780000003, 2020-01-16T00:13:07.136 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015999295, 2020-01-16T00:13:07.136 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:07.136 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:07.136 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:07.137 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:07.137 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:07.137 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1891 2020-01-16T00:13:07.137 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.137 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:07.137 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.137 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:07.138 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232813+0000", 2020-01-16T00:13:07.138 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:07.138 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.138 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.138 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:07.138 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232813+0000", 2020-01-16T00:13:07.138 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:07.139 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.139 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.139 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:07.139 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232812+0000", 2020-01-16T00:13:07.139 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.576e-06 2020-01-16T00:13:07.139 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.139 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.139 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:07.140 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232820+0000", 2020-01-16T00:13:07.140 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3079999999999997e-06 2020-01-16T00:13:07.140 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.140 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.140 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:07.140 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232829+0000", 2020-01-16T00:13:07.140 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7119999999999997e-06 2020-01-16T00:13:07.140 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.141 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.141 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:07.141 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232835+0000", 2020-01-16T00:13:07.141 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013438400000000001 2020-01-16T00:13:07.141 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.141 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.141 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.142 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232969+0000", 2020-01-16T00:13:07.142 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0562000000000001e-05 2020-01-16T00:13:07.142 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.142 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.142 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:07.142 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.233029+0000", 2020-01-16T00:13:07.143 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039354239999999999 2020-01-16T00:13:07.143 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.143 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.143 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.143 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.236965+0000", 2020-01-16T00:13:07.143 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0482999999999998e-05 2020-01-16T00:13:07.143 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.144 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.144 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:07.144 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.237015+0000", 2020-01-16T00:13:07.144 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0096086209999999995 2020-01-16T00:13:07.144 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.144 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.144 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.145 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.246624+0000", 2020-01-16T00:13:07.145 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6150999999999995e-05 2020-01-16T00:13:07.145 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.145 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.145 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:07.145 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.246720+0000", 2020-01-16T00:13:07.146 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015032899999999999 2020-01-16T00:13:07.146 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.146 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.146 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:07.146 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.246870+0000", 2020-01-16T00:13:07.146 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00064760700000000002 2020-01-16T00:13:07.147 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.147 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.147 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:07.147 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247518+0000", 2020-01-16T00:13:07.147 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000198531 2020-01-16T00:13:07.147 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.147 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.147 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:07.148 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247717+0000", 2020-01-16T00:13:07.148 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010189439999999999 2020-01-16T00:13:07.148 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.148 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.148 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:07.148 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248736+0000", 2020-01-16T00:13:07.148 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4863000000000001e-05 2020-01-16T00:13:07.148 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.149 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.149 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:07.149 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248770+0000", 2020-01-16T00:13:07.149 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2153000000000001e-05 2020-01-16T00:13:07.149 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.149 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.149 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:07.150 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248813+0000", 2020-01-16T00:13:07.150 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015999295 2020-01-16T00:13:07.150 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.150 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.151 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.151 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.151 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.151 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1892 3.0 3:0b1f384f:::smithi12313934-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e31)", 2020-01-16T00:13:07.151 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.232856+0000", 2020-01-16T00:13:07.151 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 9.7270942540000007, 2020-01-16T00:13:07.151 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016171544, 2020-01-16T00:13:07.152 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:07.152 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:07.152 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:07.152 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:07.152 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:07.152 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1892 2020-01-16T00:13:07.152 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.152 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:07.153 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.153 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:07.153 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232856+0000", 2020-01-16T00:13:07.153 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:07.153 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.153 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.153 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:07.154 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232856+0000", 2020-01-16T00:13:07.154 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:07.154 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.154 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.154 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:07.154 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232855+0000", 2020-01-16T00:13:07.154 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8640000000000004e-06 2020-01-16T00:13:07.155 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.155 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.155 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:07.155 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232860+0000", 2020-01-16T00:13:07.155 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0000000000000001e-05 2020-01-16T00:13:07.155 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.155 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.155 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:07.156 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232870+0000", 2020-01-16T00:13:07.156 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7420000000000003e-06 2020-01-16T00:13:07.156 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.156 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.156 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:07.157 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232875+0000", 2020-01-16T00:13:07.157 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017933200000000001 2020-01-16T00:13:07.157 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.157 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.157 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.157 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.233055+0000", 2020-01-16T00:13:07.157 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6269e-05 2020-01-16T00:13:07.158 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.158 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.158 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:07.158 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.233101+0000", 2020-01-16T00:13:07.158 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039780090000000002 2020-01-16T00:13:07.158 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.158 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.159 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.159 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.237079+0000", 2020-01-16T00:13:07.159 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3677000000000001e-05 2020-01-16T00:13:07.159 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.159 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.159 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:07.159 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.237133+0000", 2020-01-16T00:13:07.160 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010036671 2020-01-16T00:13:07.160 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.160 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.160 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.160 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247169+0000", 2020-01-16T00:13:07.160 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010280699999999999 2020-01-16T00:13:07.160 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.160 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.161 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:07.161 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247272+0000", 2020-01-16T00:13:07.161 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001579708 2020-01-16T00:13:07.161 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.161 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.161 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.161 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248852+0000", 2020-01-16T00:13:07.161 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9704999999999996e-05 2020-01-16T00:13:07.162 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.162 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.162 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:07.162 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248942+0000", 2020-01-16T00:13:07.162 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5557999999999996e-05 2020-01-16T00:13:07.162 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.162 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.163 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:07.163 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.249027+0000", 2020-01-16T00:13:07.163 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016171544 2020-01-16T00:13:07.163 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.163 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.163 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.163 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.164 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.164 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1950 3.0 3:0b1f384f:::smithi12313934-33:head [write 1889359~541737] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-01-16T00:13:07.164 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.922819+0000", 2020-01-16T00:13:07.164 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 9.0371308890000002, 2020-01-16T00:13:07.164 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017502846999999998, 2020-01-16T00:13:07.164 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:07.164 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:07.165 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:07.165 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:07.165 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:07.165 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1950 2020-01-16T00:13:07.165 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.165 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:07.166 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.166 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:07.166 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922819+0000", 2020-01-16T00:13:07.166 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:07.166 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.166 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.166 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:07.167 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922819+0000", 2020-01-16T00:13:07.167 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:07.167 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.167 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.167 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:07.167 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922818+0000", 2020-01-16T00:13:07.167 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000146408 2020-01-16T00:13:07.167 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.168 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.168 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:07.168 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922964+0000", 2020-01-16T00:13:07.168 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4720000000000001e-06 2020-01-16T00:13:07.168 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.168 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.169 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:07.169 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922974+0000", 2020-01-16T00:13:07.169 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0720000000000001e-06 2020-01-16T00:13:07.169 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.169 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.169 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:07.169 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922978+0000", 2020-01-16T00:13:07.170 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6152000000000001e-05 2020-01-16T00:13:07.170 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.170 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.170 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.170 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.923024+0000", 2020-01-16T00:13:07.170 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00087368199999999997 2020-01-16T00:13:07.170 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.171 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.171 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:07.171 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.923897+0000", 2020-01-16T00:13:07.171 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011944435999999999 2020-01-16T00:13:07.171 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.171 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.171 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.171 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.935842+0000", 2020-01-16T00:13:07.172 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000144146 2020-01-16T00:13:07.172 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.172 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.172 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:07.172 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.935986+0000", 2020-01-16T00:13:07.172 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019775500000000001 2020-01-16T00:13:07.172 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.172 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.173 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:07.173 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.936184+0000", 2020-01-16T00:13:07.173 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002539342 2020-01-16T00:13:07.173 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.173 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.173 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:07.173 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938723+0000", 2020-01-16T00:13:07.173 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014193699999999999 2020-01-16T00:13:07.174 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.174 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.174 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:07.174 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938865+0000", 2020-01-16T00:13:07.174 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013829020000000001 2020-01-16T00:13:07.174 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.174 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.175 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:07.175 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.940248+0000", 2020-01-16T00:13:07.175 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2558000000000001e-05 2020-01-16T00:13:07.175 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.175 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.175 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:07.175 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.940291+0000", 2020-01-16T00:13:07.175 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1217999999999998e-05 2020-01-16T00:13:07.176 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.176 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.176 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:07.176 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.940322+0000", 2020-01-16T00:13:07.176 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017502846999999998 2020-01-16T00:13:07.176 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.176 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.177 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.177 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.177 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.177 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1951 3.0 3:0b1f384f:::smithi12313934-33:head [write 2955792~541902] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-01-16T00:13:07.177 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.929019+0000", 2020-01-16T00:13:07.178 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 9.0309307390000004, 2020-01-16T00:13:07.178 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021625710999999999, 2020-01-16T00:13:07.178 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:07.178 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:07.178 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:07.178 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:07.178 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:07.179 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1951 2020-01-16T00:13:07.179 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.179 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:07.179 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.179 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:07.179 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929019+0000", 2020-01-16T00:13:07.179 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:07.179 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.180 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.180 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:07.180 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929019+0000", 2020-01-16T00:13:07.180 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999989 2020-01-16T00:13:07.180 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.180 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.180 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:07.181 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929008+0000", 2020-01-16T00:13:07.181 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000212513 2020-01-16T00:13:07.181 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.181 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.181 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:07.181 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929221+0000", 2020-01-16T00:13:07.181 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3229e-05 2020-01-16T00:13:07.182 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.182 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.182 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:07.182 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929234+0000", 2020-01-16T00:13:07.182 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9170000000000003e-06 2020-01-16T00:13:07.182 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.182 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.182 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:07.183 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929240+0000", 2020-01-16T00:13:07.183 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4045000000000002e-05 2020-01-16T00:13:07.183 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.183 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.183 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.183 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929294+0000", 2020-01-16T00:13:07.183 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9797999999999996e-05 2020-01-16T00:13:07.184 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.184 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.184 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:07.184 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929384+0000", 2020-01-16T00:13:07.184 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083450729999999997 2020-01-16T00:13:07.184 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.184 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.184 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.185 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.937729+0000", 2020-01-16T00:13:07.185 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043251699999999999 2020-01-16T00:13:07.185 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.185 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.185 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:07.185 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938161+0000", 2020-01-16T00:13:07.185 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095115489999999994 2020-01-16T00:13:07.186 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.186 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.186 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.186 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.947673+0000", 2020-01-16T00:13:07.186 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012395099999999999 2020-01-16T00:13:07.186 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.187 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.187 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:07.187 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.947797+0000", 2020-01-16T00:13:07.187 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00024849200000000001 2020-01-16T00:13:07.187 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.187 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.188 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:07.188 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.948045+0000", 2020-01-16T00:13:07.188 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0025081050000000001 2020-01-16T00:13:07.188 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.188 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.188 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:07.188 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950553+0000", 2020-01-16T00:13:07.189 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9760000000000003e-06 2020-01-16T00:13:07.189 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.189 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.189 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:07.189 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950558+0000", 2020-01-16T00:13:07.189 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3286999999999999e-05 2020-01-16T00:13:07.189 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.190 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.190 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:07.190 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950581+0000", 2020-01-16T00:13:07.190 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2759000000000003e-05 2020-01-16T00:13:07.190 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.190 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.190 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:07.190 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950614+0000", 2020-01-16T00:13:07.191 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0525999999999999e-05 2020-01-16T00:13:07.191 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.191 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.191 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:07.191 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950645+0000", 2020-01-16T00:13:07.191 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021625710999999999 2020-01-16T00:13:07.191 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.191 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.192 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.192 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.192 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.192 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1952 3.0 3:0b1f384f:::smithi12313934-33:head [setxattr _header (56) in=7b,truncate 3563084] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-01-16T00:13:07.192 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:13:07.192 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 9.030701402, 2020-01-16T00:13:07.192 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021534685000000001, 2020-01-16T00:13:07.192 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:07.193 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:07.193 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:07.193 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:07.193 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:07.193 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1952 2020-01-16T00:13:07.193 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.193 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:07.194 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.194 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:07.194 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:13:07.194 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:07.194 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.194 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.194 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:07.195 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:13:07.195 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:07.195 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.195 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.195 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:07.195 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:13:07.195 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6670000000000004e-06 2020-01-16T00:13:07.196 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.196 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.196 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:07.196 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929252+0000", 2020-01-16T00:13:07.196 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9380000000000002e-06 2020-01-16T00:13:07.196 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.197 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.197 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:07.197 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929256+0000", 2020-01-16T00:13:07.197 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.712e-06 2020-01-16T00:13:07.197 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.197 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.198 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:07.198 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929264+0000", 2020-01-16T00:13:07.198 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018030300000000001 2020-01-16T00:13:07.198 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.198 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.198 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.199 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929444+0000", 2020-01-16T00:13:07.199 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7562000000000002e-05 2020-01-16T00:13:07.199 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.199 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.199 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:07.199 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929502+0000", 2020-01-16T00:13:07.199 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0087297859999999998 2020-01-16T00:13:07.200 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.200 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.200 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.200 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938232+0000", 2020-01-16T00:13:07.200 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0452999999999996e-05 2020-01-16T00:13:07.200 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.200 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.200 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:07.201 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938322+0000", 2020-01-16T00:13:07.201 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011061728 2020-01-16T00:13:07.201 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.201 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.201 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.201 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.949384+0000", 2020-01-16T00:13:07.201 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000117828 2020-01-16T00:13:07.201 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.202 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.202 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:07.202 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.949502+0000", 2020-01-16T00:13:07.202 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022468499999999999 2020-01-16T00:13:07.202 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.202 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.202 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:07.203 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.949726+0000", 2020-01-16T00:13:07.203 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00083532099999999998 2020-01-16T00:13:07.203 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.203 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.203 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:07.203 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950562+0000", 2020-01-16T00:13:07.203 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013017700000000001 2020-01-16T00:13:07.204 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.204 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.204 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:07.204 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950692+0000", 2020-01-16T00:13:07.204 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9813000000000001e-05 2020-01-16T00:13:07.204 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.204 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.205 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:07.205 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950722+0000", 2020-01-16T00:13:07.205 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0592000000000001e-05 2020-01-16T00:13:07.205 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.205 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.205 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:07.205 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950752+0000", 2020-01-16T00:13:07.206 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0920000000000002e-05 2020-01-16T00:13:07.206 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.206 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.206 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:07.206 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950783+0000", 2020-01-16T00:13:07.206 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021534685000000001 2020-01-16T00:13:07.206 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.207 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.207 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.207 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.207 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.207 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1953 3.0 3:0b1f384f:::smithi12313934-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e32)", 2020-01-16T00:13:07.207 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:13:07.208 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 9.0306684110000006, 2020-01-16T00:13:07.208 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021667537000000001, 2020-01-16T00:13:07.208 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:07.208 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:07.208 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:07.208 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:07.208 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:07.209 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1953 2020-01-16T00:13:07.209 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.209 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:07.209 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.209 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:07.209 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:13:07.210 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:07.210 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.210 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.210 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:07.210 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:13:07.210 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:07.211 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.211 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.211 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:07.211 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:13:07.211 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7030000000000002e-06 2020-01-16T00:13:07.211 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.211 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.211 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:07.212 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929284+0000", 2020-01-16T00:13:07.212 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0909999999999993e-06 2020-01-16T00:13:07.212 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.212 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.212 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:07.212 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929292+0000", 2020-01-16T00:13:07.212 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8359999999999999e-06 2020-01-16T00:13:07.213 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.213 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.213 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:07.213 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929297+0000", 2020-01-16T00:13:07.213 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023538199999999999 2020-01-16T00:13:07.213 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.213 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.214 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.214 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929532+0000", 2020-01-16T00:13:07.214 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1173000000000004e-05 2020-01-16T00:13:07.214 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.214 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.214 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:07.214 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929603+0000", 2020-01-16T00:13:07.215 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0087813509999999997 2020-01-16T00:13:07.215 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.215 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.215 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.215 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938384+0000", 2020-01-16T00:13:07.215 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9752000000000006e-05 2020-01-16T00:13:07.215 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.216 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.216 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:07.216 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938454+0000", 2020-01-16T00:13:07.216 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011707851999999999 2020-01-16T00:13:07.216 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.216 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.216 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.216 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950162+0000", 2020-01-16T00:13:07.217 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011060099999999999 2020-01-16T00:13:07.217 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.217 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.217 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:07.217 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950273+0000", 2020-01-16T00:13:07.217 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00053989700000000001 2020-01-16T00:13:07.217 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.218 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.218 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.218 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950813+0000", 2020-01-16T00:13:07.218 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8128999999999999e-05 2020-01-16T00:13:07.219 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.219 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.219 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:07.219 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950871+0000", 2020-01-16T00:13:07.219 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8242000000000001e-05 2020-01-16T00:13:07.219 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.219 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.220 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:07.220 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950949+0000", 2020-01-16T00:13:07.220 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021667537000000001 2020-01-16T00:13:07.220 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.220 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.220 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.220 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.221 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.221 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2191 3.0 3:0b70bb20:::smithi12313934-46:head [setxattr _header (56) in=7b,truncate 2016873] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:13:07.221 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:03.256877+0000", 2020-01-16T00:13:07.221 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 3.703072846, 2020-01-16T00:13:07.221 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014406043, 2020-01-16T00:13:07.221 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:07.221 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:07.222 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:07.222 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:07.222 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:07.222 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2191 2020-01-16T00:13:07.222 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.222 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:07.222 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.223 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:07.223 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.256877+0000", 2020-01-16T00:13:07.223 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:07.223 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.223 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.223 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:07.223 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.256877+0000", 2020-01-16T00:13:07.223 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:07.224 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.224 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.224 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:07.224 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.256876+0000", 2020-01-16T00:13:07.224 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7060000000000003e-06 2020-01-16T00:13:07.224 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.224 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.224 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:07.225 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.256881+0000", 2020-01-16T00:13:07.225 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3770000000000004e-06 2020-01-16T00:13:07.225 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.225 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.225 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:07.225 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.256886+0000", 2020-01-16T00:13:07.225 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5980000000000001e-06 2020-01-16T00:13:07.226 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.226 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.226 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:07.226 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.256890+0000", 2020-01-16T00:13:07.226 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000732199 2020-01-16T00:13:07.226 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.226 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.226 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.227 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.257622+0000", 2020-01-16T00:13:07.227 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022652300000000001 2020-01-16T00:13:07.227 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.227 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.227 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:07.227 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.257849+0000", 2020-01-16T00:13:07.227 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010272460000000001 2020-01-16T00:13:07.227 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.228 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.228 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.228 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.268121+0000", 2020-01-16T00:13:07.228 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2525000000000001e-05 2020-01-16T00:13:07.228 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.228 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.229 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:07.229 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.268194+0000", 2020-01-16T00:13:07.229 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010802 2020-01-16T00:13:07.229 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.229 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.229 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:07.229 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.268302+0000", 2020-01-16T00:13:07.229 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0011749659999999999 2020-01-16T00:13:07.230 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.230 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.230 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:07.230 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.269477+0000", 2020-01-16T00:13:07.230 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001587757 2020-01-16T00:13:07.230 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.231 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.231 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:07.231 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.271064+0000", 2020-01-16T00:13:07.231 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1802999999999999e-05 2020-01-16T00:13:07.231 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.231 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.231 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:07.232 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.271116+0000", 2020-01-16T00:13:07.232 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5095e-05 2020-01-16T00:13:07.232 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.232 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.232 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:07.232 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.271171+0000", 2020-01-16T00:13:07.232 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000111723 2020-01-16T00:13:07.233 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.233 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.233 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:07.233 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.271283+0000", 2020-01-16T00:13:07.233 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014406043 2020-01-16T00:13:07.233 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.233 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.234 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.234 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.234 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.234 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2192 3.0 3:0b70bb20:::smithi12313934-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-01-16T00:13:07.234 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:03.256902+0000", 2020-01-16T00:13:07.234 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 3.7030473439999998, 2020-01-16T00:13:07.235 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014663150999999999, 2020-01-16T00:13:07.235 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:07.235 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:07.235 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:07.235 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:07.235 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:07.235 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2192 2020-01-16T00:13:07.236 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.236 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:07.236 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.236 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:07.236 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.256902+0000", 2020-01-16T00:13:07.236 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:07.236 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.237 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.237 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:07.237 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.256902+0000", 2020-01-16T00:13:07.237 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:07.237 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.237 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.237 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:07.238 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.256902+0000", 2020-01-16T00:13:07.238 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3409999999999999e-06 2020-01-16T00:13:07.238 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.238 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.238 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:07.238 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.256905+0000", 2020-01-16T00:13:07.239 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6280000000000002e-06 2020-01-16T00:13:07.239 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.239 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.239 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:07.239 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.256911+0000", 2020-01-16T00:13:07.239 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9009999999999996e-06 2020-01-16T00:13:07.239 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.240 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.240 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:07.240 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.256915+0000", 2020-01-16T00:13:07.240 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001008756 2020-01-16T00:13:07.240 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.240 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.240 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.241 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.257923+0000", 2020-01-16T00:13:07.241 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6691000000000003e-05 2020-01-16T00:13:07.241 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.241 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.241 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:07.241 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.257980+0000", 2020-01-16T00:13:07.241 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010850935000000001 2020-01-16T00:13:07.242 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.242 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.242 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.242 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.268831+0000", 2020-01-16T00:13:07.242 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000109475 2020-01-16T00:13:07.242 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.242 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.243 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:07.243 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.268941+0000", 2020-01-16T00:13:07.243 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0023916050000000002 2020-01-16T00:13:07.243 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.243 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.243 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.243 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.271332+0000", 2020-01-16T00:13:07.244 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000108996 2020-01-16T00:13:07.244 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.244 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.244 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:07.244 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.271441+0000", 2020-01-16T00:13:07.244 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012444600000000001 2020-01-16T00:13:07.244 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.245 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.245 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:07.245 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.271566+0000", 2020-01-16T00:13:07.245 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014663150999999999 2020-01-16T00:13:07.245 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.245 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.245 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.246 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.246 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.246 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:07.248 INFO:tasks.rados.rados.0.smithi123.stdout:1019: finishing write tid 1 to smithi12313934-13 2020-01-16T00:13:07.248 INFO:tasks.rados.rados.0.smithi123.stdout:1014: expect (ObjNum 234 snap 0 seq_num 234) 2020-01-16T00:13:07.248 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:13:07.244+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:13:07.249 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-01-16T00:13:07.280 INFO:tasks.rados.rados.0.smithi123.stdout:1019: finishing write tid 2 to smithi12313934-13 2020-01-16T00:13:07.281 INFO:tasks.rados.rados.0.smithi123.stdout:1019: finishing write tid 4 to smithi12313934-13 2020-01-16T00:13:07.281 INFO:tasks.rados.rados.0.smithi123.stdout:1019: finishing write tid 5 to smithi12313934-13 2020-01-16T00:13:07.281 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 57 (ObjNum 251 snap 0 seq_num 251) dirty exists 2020-01-16T00:13:07.281 INFO:tasks.rados.rados.0.smithi123.stdout:1019: left oid 13 (ObjNum 251 snap 0 seq_num 251) 2020-01-16T00:13:07.281 INFO:tasks.rados.rados.0.smithi123.stdout:1012: done (15 left) 2020-01-16T00:13:07.282 INFO:tasks.rados.rados.0.smithi123.stdout:1014: done (14 left) 2020-01-16T00:13:07.282 INFO:tasks.rados.rados.0.smithi123.stdout:1017: done (13 left) 2020-01-16T00:13:07.282 INFO:tasks.rados.rados.0.smithi123.stdout:1019: done (12 left) 2020-01-16T00:13:07.282 INFO:tasks.rados.rados.0.smithi123.stdout:1029: write oid 19 current snap is 0 2020-01-16T00:13:07.282 INFO:tasks.rados.rados.0.smithi123.stdout:1029: seq_num 256 ranges {702592=703393,1880802=421790} 2020-01-16T00:13:07.298 INFO:tasks.rados.rados.0.smithi123.stdout:1029: writing smithi12313934-19 from 702592 to 1405985 tid 1 2020-01-16T00:13:07.304 INFO:tasks.rados.rados.0.smithi123.stdout:1029: writing smithi12313934-19 from 1880802 to 2302592 tid 2 2020-01-16T00:13:07.304 INFO:tasks.rados.rados.0.smithi123.stdout:1030: write oid 29 current snap is 0 2020-01-16T00:13:07.305 INFO:tasks.rados.rados.0.smithi123.stdout:1030: seq_num 257 ranges {719399=778642,1939284=380115} 2020-01-16T00:13:07.313 INFO:tasks.rados.rados.0.smithi123.stdout:1030: writing smithi12313934-29 from 719399 to 1498041 tid 1 2020-01-16T00:13:07.318 INFO:tasks.rados.rados.0.smithi123.stdout:1030: writing smithi12313934-29 from 1939284 to 2319399 tid 2 2020-01-16T00:13:07.318 INFO:tasks.rados.rados.0.smithi123.stdout:1031: read oid 33 snap -1 2020-01-16T00:13:07.319 INFO:tasks.rados.rados.0.smithi123.stdout:1031: expect (ObjNum 212 snap 0 seq_num 212) 2020-01-16T00:13:07.319 INFO:tasks.rados.rados.0.smithi123.stdout:1032: read oid 4 snap -1 2020-01-16T00:13:07.319 INFO:tasks.rados.rados.0.smithi123.stdout:1032: expect (ObjNum 188 snap 0 seq_num 188) 2020-01-16T00:13:07.319 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:07.319 INFO:tasks.rados.rados.0.smithi123.stdout:1018: expect (ObjNum 214 snap 0 seq_num 214) 2020-01-16T00:13:07.348 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:07.346+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:07.399 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:07.399 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:07.399 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:13:07.399 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:07.407 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-01-16T00:13:07.449 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:07.446+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:07.532 INFO:tasks.rados.rados.0.smithi123.stdout:1015: finishing write tid 1 to smithi12313934-47 2020-01-16T00:13:07.533 INFO:tasks.rados.rados.0.smithi123.stdout:1015: finishing write tid 2 to smithi12313934-47 2020-01-16T00:13:07.533 INFO:tasks.rados.rados.0.smithi123.stdout:1015: finishing write tid 4 to smithi12313934-47 2020-01-16T00:13:07.533 INFO:tasks.rados.rados.0.smithi123.stdout:1015: finishing write tid 5 to smithi12313934-47 2020-01-16T00:13:07.533 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 47 v 147 (ObjNum 250 snap 0 seq_num 250) dirty exists 2020-01-16T00:13:07.533 INFO:tasks.rados.rados.0.smithi123.stdout:1015: left oid 47 (ObjNum 250 snap 0 seq_num 250) 2020-01-16T00:13:07.533 INFO:tasks.rados.rados.0.smithi123.stdout:1015: done (15 left) 2020-01-16T00:13:07.533 INFO:tasks.rados.rados.0.smithi123.stdout:1018: done (14 left) 2020-01-16T00:13:07.534 INFO:tasks.rados.rados.0.smithi123.stdout:1033: write (excl) oid 24 current snap is 0 2020-01-16T00:13:07.534 INFO:tasks.rados.rados.0.smithi123.stdout:1033: seq_num 258 ranges {736206=453891,1881413=454793} 2020-01-16T00:13:07.542 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:07.542 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:07.543 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:13:07.543 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:13:07.543 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:07.544 INFO:tasks.rados.rados.0.smithi123.stdout:1033: writing smithi12313934-24 from 736206 to 1190097 tid 1 2020-01-16T00:13:07.550 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:07.546+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:07.551 INFO:tasks.rados.rados.0.smithi123.stdout:1033: writing smithi12313934-24 from 1881413 to 2336206 tid 2 2020-01-16T00:13:07.551 INFO:tasks.rados.rados.0.smithi123.stdout:1034: write (excl) oid 9 current snap is 0 2020-01-16T00:13:07.551 INFO:tasks.rados.rados.0.smithi123.stdout:1034: seq_num 259 ranges {753013=645493,2056248=296765} 2020-01-16T00:13:07.552 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-01-16T00:13:07.559 INFO:tasks.rados.rados.0.smithi123.stdout:1034: writing smithi12313934-9 from 753013 to 1398506 tid 1 2020-01-16T00:13:07.565 INFO:tasks.rados.rados.0.smithi123.stdout:1034: writing smithi12313934-9 from 2056248 to 2353013 tid 2 2020-01-16T00:13:07.565 INFO:tasks.rados.rados.0.smithi123.stdout:1016: expect (ObjNum 224 snap 0 seq_num 224) 2020-01-16T00:13:07.652 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:07.652 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:13:07.652 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:13:07.653 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:13:07.653 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.653 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.5.13:36 2.0 e24/19 2:20e40cec:::smithi12313934-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 24'16, mlcod=23'14)", 2020-01-16T00:13:07.653 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.834879+0000", 2020-01-16T00:13:07.653 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 29.812183009000002, 2020-01-16T00:13:07.653 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0075184850000000001, 2020-01-16T00:13:07.654 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:07.654 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:07.654 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:07.654 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.654 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:07.654 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.834879+0000", 2020-01-16T00:13:07.654 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:07.654 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.655 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.655 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:07.655 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.834879+0000", 2020-01-16T00:13:07.655 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:07.655 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.655 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.655 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:07.656 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.834877+0000", 2020-01-16T00:13:07.656 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031393089999999998 2020-01-16T00:13:07.656 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.656 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.656 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:07.656 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.838016+0000", 2020-01-16T00:13:07.656 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2144e-05 2020-01-16T00:13:07.657 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.657 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.657 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:07.657 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.838038+0000", 2020-01-16T00:13:07.657 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2058e-05 2020-01-16T00:13:07.657 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.657 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.657 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:07.658 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.838050+0000", 2020-01-16T00:13:07.658 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8533000000000003e-05 2020-01-16T00:13:07.658 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.658 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.658 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.659 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.838119+0000", 2020-01-16T00:13:07.659 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3591000000000003e-05 2020-01-16T00:13:07.659 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.659 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.659 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:07.659 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.838162+0000", 2020-01-16T00:13:07.660 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041910380000000002 2020-01-16T00:13:07.660 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.660 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.660 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:07.660 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.842353+0000", 2020-01-16T00:13:07.660 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4433e-05 2020-01-16T00:13:07.660 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.660 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.661 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:07.661 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.842398+0000", 2020-01-16T00:13:07.661 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0075184850000000001 2020-01-16T00:13:07.661 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.661 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.661 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.661 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.661 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.662 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.5.13:40 2.0 e24/19 2:20e40cec:::smithi12313934-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 24'17, mlcod=23'15)", 2020-01-16T00:13:07.662 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.849607+0000", 2020-01-16T00:13:07.662 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 29.797454884, 2020-01-16T00:13:07.662 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083979039999999994, 2020-01-16T00:13:07.662 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:07.662 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:07.662 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:07.663 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.663 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:07.663 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.849607+0000", 2020-01-16T00:13:07.663 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:07.663 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.663 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.663 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:07.664 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.849607+0000", 2020-01-16T00:13:07.664 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:07.664 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.664 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.664 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:07.664 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.849605+0000", 2020-01-16T00:13:07.664 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0037534560000000001 2020-01-16T00:13:07.665 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.665 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.665 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:07.665 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.853358+0000", 2020-01-16T00:13:07.665 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9181000000000002e-05 2020-01-16T00:13:07.665 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.665 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.666 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:07.666 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.853377+0000", 2020-01-16T00:13:07.666 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0040000000000002e-06 2020-01-16T00:13:07.666 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.666 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.666 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:07.666 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.853384+0000", 2020-01-16T00:13:07.666 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2583000000000001e-05 2020-01-16T00:13:07.667 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.667 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.667 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.667 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.853437+0000", 2020-01-16T00:13:07.667 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8212999999999999e-05 2020-01-16T00:13:07.667 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.667 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.668 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:07.668 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.853465+0000", 2020-01-16T00:13:07.668 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0044847589999999996 2020-01-16T00:13:07.668 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.668 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.668 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:07.669 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.857950+0000", 2020-01-16T00:13:07.669 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5309999999999997e-05 2020-01-16T00:13:07.669 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.669 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.669 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:07.669 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.858005+0000", 2020-01-16T00:13:07.670 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083979039999999994 2020-01-16T00:13:07.670 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.670 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.670 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.670 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.670 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.670 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.5.13:42 2.0 e24/19 2:313458df:::smithi12313934-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 24'18, mlcod=24'16)", 2020-01-16T00:13:07.671 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.940683+0000", 2020-01-16T00:13:07.671 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 29.706379044999998, 2020-01-16T00:13:07.671 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0097026960000000002, 2020-01-16T00:13:07.671 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:07.671 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:07.671 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:07.671 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.671 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:07.672 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.940683+0000", 2020-01-16T00:13:07.672 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:07.672 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.672 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.672 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:07.672 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.940683+0000", 2020-01-16T00:13:07.672 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:07.673 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.673 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.673 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:07.673 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.940680+0000", 2020-01-16T00:13:07.673 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003126775 2020-01-16T00:13:07.673 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.673 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.674 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:07.674 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.943807+0000", 2020-01-16T00:13:07.674 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5426999999999999e-05 2020-01-16T00:13:07.674 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.674 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.674 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:07.674 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.943833+0000", 2020-01-16T00:13:07.674 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7409999999999998e-06 2020-01-16T00:13:07.675 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.675 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.675 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:07.675 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.943841+0000", 2020-01-16T00:13:07.675 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011645399999999999 2020-01-16T00:13:07.675 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.676 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.676 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.676 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.943958+0000", 2020-01-16T00:13:07.676 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7291000000000003e-05 2020-01-16T00:13:07.676 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.676 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.676 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:07.676 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.944025+0000", 2020-01-16T00:13:07.677 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063108449999999998 2020-01-16T00:13:07.677 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.677 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.677 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:07.677 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.950336+0000", 2020-01-16T00:13:07.677 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9874999999999999e-05 2020-01-16T00:13:07.677 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.678 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.678 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:07.678 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.950386+0000", 2020-01-16T00:13:07.678 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0097026960000000002 2020-01-16T00:13:07.678 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.678 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.679 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.679 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.679 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.679 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1387 3.1 3:8c6a83c0:::smithi12313934-36:head [read 0~3579858,checksum] snapc 0=[] ondisk+read+known_if_redirected e28)", 2020-01-16T00:13:07.679 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:46.605159+0000", 2020-01-16T00:13:07.679 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 21.041902748999998, 2020-01-16T00:13:07.679 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0074537709999999997, 2020-01-16T00:13:07.679 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:07.680 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:07.680 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:07.680 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:07.680 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:07.680 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1387 2020-01-16T00:13:07.680 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.680 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:07.680 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.681 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:07.681 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605159+0000", 2020-01-16T00:13:07.681 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:07.681 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.681 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.681 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:07.681 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605159+0000", 2020-01-16T00:13:07.681 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:07.682 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.682 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.682 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:07.682 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605158+0000", 2020-01-16T00:13:07.682 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3860000000000001e-06 2020-01-16T00:13:07.682 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.682 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.682 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:07.683 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605166+0000", 2020-01-16T00:13:07.683 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2139e-05 2020-01-16T00:13:07.683 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.683 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.683 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:07.683 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605178+0000", 2020-01-16T00:13:07.683 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8270000000000003e-06 2020-01-16T00:13:07.683 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.684 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.684 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:07.684 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605184+0000", 2020-01-16T00:13:07.684 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0045873850000000002 2020-01-16T00:13:07.684 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.684 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.684 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.684 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.609771+0000", 2020-01-16T00:13:07.685 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000202387 2020-01-16T00:13:07.685 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.685 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.685 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:07.685 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.609973+0000", 2020-01-16T00:13:07.685 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0026399079999999998 2020-01-16T00:13:07.685 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.685 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.686 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:07.686 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.612613+0000", 2020-01-16T00:13:07.686 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0074537709999999997 2020-01-16T00:13:07.686 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.686 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.686 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.687 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.687 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.687 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1388 3.1 3:8c6a83c0:::smithi12313934-36:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e28)", 2020-01-16T00:13:07.687 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:46.605253+0000", 2020-01-16T00:13:07.687 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 21.041809695000001, 2020-01-16T00:13:07.687 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0080920829999999999, 2020-01-16T00:13:07.687 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:07.687 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:07.688 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:07.688 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:07.688 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:07.688 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1388 2020-01-16T00:13:07.688 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.688 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:07.688 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.688 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:07.689 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605253+0000", 2020-01-16T00:13:07.689 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:07.689 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.689 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.689 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:07.689 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605253+0000", 2020-01-16T00:13:07.690 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:07.690 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.690 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.690 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:07.690 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605252+0000", 2020-01-16T00:13:07.690 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6559999999999997e-06 2020-01-16T00:13:07.691 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.691 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.691 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:07.691 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605257+0000", 2020-01-16T00:13:07.691 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0966e-05 2020-01-16T00:13:07.691 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.691 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.692 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:07.692 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605268+0000", 2020-01-16T00:13:07.692 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5960000000000003e-06 2020-01-16T00:13:07.692 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.692 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.692 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:07.692 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605274+0000", 2020-01-16T00:13:07.693 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.007370176 2020-01-16T00:13:07.693 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.693 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.693 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.693 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.612644+0000", 2020-01-16T00:13:07.693 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000113146 2020-01-16T00:13:07.693 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.693 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.694 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:07.694 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.612757+0000", 2020-01-16T00:13:07.694 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00058744799999999996 2020-01-16T00:13:07.694 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.694 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.694 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:07.694 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.613345+0000", 2020-01-16T00:13:07.694 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0080920829999999999 2020-01-16T00:13:07.695 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.695 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.695 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.695 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.695 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.695 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1644 3.1 3:8c6a83c0:::smithi12313934-36:head [read 0~2705927,checksum] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:13:07.695 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847556+0000", 2020-01-16T00:13:07.696 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 15.799506278000001, 2020-01-16T00:13:07.696 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0081843639999999995, 2020-01-16T00:13:07.696 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:07.696 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:07.696 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:07.696 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:07.696 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:07.696 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1644 2020-01-16T00:13:07.697 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.697 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:07.697 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.697 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:07.697 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847556+0000", 2020-01-16T00:13:07.697 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:07.697 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.697 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.697 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:07.698 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847556+0000", 2020-01-16T00:13:07.698 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:07.698 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.698 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.698 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:07.698 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847555+0000", 2020-01-16T00:13:07.698 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4640000000000002e-06 2020-01-16T00:13:07.699 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.699 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.699 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:07.699 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847559+0000", 2020-01-16T00:13:07.699 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4580000000000003e-06 2020-01-16T00:13:07.699 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.699 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.700 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:07.700 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847563+0000", 2020-01-16T00:13:07.700 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.613e-06 2020-01-16T00:13:07.700 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.700 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.700 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:07.700 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847567+0000", 2020-01-16T00:13:07.701 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000145045 2020-01-16T00:13:07.701 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.701 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.701 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.701 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847712+0000", 2020-01-16T00:13:07.701 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6480000000000003e-06 2020-01-16T00:13:07.702 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.702 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.702 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:13:07.702 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847720+0000", 2020-01-16T00:13:07.702 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0055645269999999997 2020-01-16T00:13:07.702 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.702 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.703 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.703 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.853284+0000", 2020-01-16T00:13:07.703 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00026767400000000003 2020-01-16T00:13:07.703 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.703 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.703 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:07.703 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.853552+0000", 2020-01-16T00:13:07.704 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021884169999999998 2020-01-16T00:13:07.704 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.704 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.704 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:07.704 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.855740+0000", 2020-01-16T00:13:07.704 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0081843639999999995 2020-01-16T00:13:07.704 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.705 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.705 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.705 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.705 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.705 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1645 3.1 3:8c6a83c0:::smithi12313934-36:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:13:07.705 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847602+0000", 2020-01-16T00:13:07.705 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 15.799460479, 2020-01-16T00:13:07.706 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0085169679999999998, 2020-01-16T00:13:07.706 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:07.706 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:07.706 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:07.706 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:07.706 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:07.706 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1645 2020-01-16T00:13:07.707 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.707 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:07.707 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.707 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:07.707 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847602+0000", 2020-01-16T00:13:07.707 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:07.707 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.707 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.708 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:07.708 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847602+0000", 2020-01-16T00:13:07.708 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:07.708 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.708 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.708 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:07.708 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847601+0000", 2020-01-16T00:13:07.708 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7011e-05 2020-01-16T00:13:07.709 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.709 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.709 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:07.709 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847618+0000", 2020-01-16T00:13:07.709 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.187e-06 2020-01-16T00:13:07.709 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.710 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.710 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:07.710 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847625+0000", 2020-01-16T00:13:07.710 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9879999999999998e-06 2020-01-16T00:13:07.710 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.710 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.710 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:07.711 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847629+0000", 2020-01-16T00:13:07.711 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000129694 2020-01-16T00:13:07.711 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.711 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.711 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.711 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847759+0000", 2020-01-16T00:13:07.711 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8379999999999996e-06 2020-01-16T00:13:07.711 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.712 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.712 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:13:07.712 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847766+0000", 2020-01-16T00:13:07.712 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0079977390000000002 2020-01-16T00:13:07.712 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.712 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.712 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.713 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.855764+0000", 2020-01-16T00:13:07.713 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2467999999999996e-05 2020-01-16T00:13:07.713 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.713 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.713 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:07.713 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.855856+0000", 2020-01-16T00:13:07.713 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00026275099999999998 2020-01-16T00:13:07.713 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.714 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.714 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:07.714 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.856119+0000", 2020-01-16T00:13:07.714 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0085169679999999998 2020-01-16T00:13:07.714 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.714 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.714 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.715 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.715 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.715 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1649 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~2084068,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:13:07.715 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847683+0000", 2020-01-16T00:13:07.715 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 15.799379227999999, 2020-01-16T00:13:07.715 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010654691000000001, 2020-01-16T00:13:07.715 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:07.716 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:07.716 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:07.716 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:07.716 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:07.716 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1649 2020-01-16T00:13:07.716 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.716 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:07.716 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.717 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:07.717 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847683+0000", 2020-01-16T00:13:07.717 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:07.717 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.717 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.717 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:07.717 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847683+0000", 2020-01-16T00:13:07.718 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:07.718 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.718 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.718 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:07.718 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847682+0000", 2020-01-16T00:13:07.718 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8569999999999997e-06 2020-01-16T00:13:07.718 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.719 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.719 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:07.719 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847689+0000", 2020-01-16T00:13:07.719 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8090000000000004e-06 2020-01-16T00:13:07.719 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.720 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.720 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:07.720 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847696+0000", 2020-01-16T00:13:07.720 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2690000000000001e-06 2020-01-16T00:13:07.720 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.720 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.721 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:07.721 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847700+0000", 2020-01-16T00:13:07.721 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8189999999999993e-05 2020-01-16T00:13:07.721 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.721 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.721 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.721 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847798+0000", 2020-01-16T00:13:07.722 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7129999999999999e-06 2020-01-16T00:13:07.722 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.722 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.722 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:13:07.722 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847802+0000", 2020-01-16T00:13:07.722 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083913049999999999 2020-01-16T00:13:07.722 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.723 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.723 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.723 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.856193+0000", 2020-01-16T00:13:07.723 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034400300000000001 2020-01-16T00:13:07.723 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.723 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.723 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:07.724 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.856537+0000", 2020-01-16T00:13:07.724 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018006820000000001 2020-01-16T00:13:07.724 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.724 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.724 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:07.724 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858338+0000", 2020-01-16T00:13:07.724 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010654691000000001 2020-01-16T00:13:07.725 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.725 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.725 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.725 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.725 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.725 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1650 3.1 3:8de1d8a3:::smithi12313934-13:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:13:07.726 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847717+0000", 2020-01-16T00:13:07.726 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 15.799345275, 2020-01-16T00:13:07.726 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010913658, 2020-01-16T00:13:07.726 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:07.726 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:07.726 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:07.726 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:07.727 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:07.727 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1650 2020-01-16T00:13:07.727 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.727 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:07.727 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.727 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:07.727 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847717+0000", 2020-01-16T00:13:07.728 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:07.728 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.728 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.728 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:07.728 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847717+0000", 2020-01-16T00:13:07.728 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:07.728 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.729 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.729 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:07.729 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847716+0000", 2020-01-16T00:13:07.729 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.433e-06 2020-01-16T00:13:07.729 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.729 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.729 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:07.729 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847720+0000", 2020-01-16T00:13:07.730 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1800000000000004e-06 2020-01-16T00:13:07.730 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.730 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.730 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:07.730 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847725+0000", 2020-01-16T00:13:07.730 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6689999999999997e-06 2020-01-16T00:13:07.730 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.731 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.731 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:07.731 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847730+0000", 2020-01-16T00:13:07.731 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000102468 2020-01-16T00:13:07.731 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.731 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.731 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.731 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847832+0000", 2020-01-16T00:13:07.732 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3999999999999997e-06 2020-01-16T00:13:07.732 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.732 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.732 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:13:07.732 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847838+0000", 2020-01-16T00:13:07.732 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010519395000000001 2020-01-16T00:13:07.732 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.732 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.733 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.733 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858358+0000", 2020-01-16T00:13:07.733 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7600000000000002e-05 2020-01-16T00:13:07.733 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.733 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.733 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:07.733 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858445+0000", 2020-01-16T00:13:07.733 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018515900000000001 2020-01-16T00:13:07.734 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.734 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.734 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:07.734 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858631+0000", 2020-01-16T00:13:07.734 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010913658 2020-01-16T00:13:07.734 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.734 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.735 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.735 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.735 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.735 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1651 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~2084068,checksum] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:13:07.735 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847752+0000", 2020-01-16T00:13:07.735 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 15.799310659, 2020-01-16T00:13:07.735 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011734249, 2020-01-16T00:13:07.736 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:07.736 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:07.736 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:07.736 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:07.736 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:07.736 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1651 2020-01-16T00:13:07.736 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.737 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:07.737 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.737 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:07.737 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847752+0000", 2020-01-16T00:13:07.737 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:07.737 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.738 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.738 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:07.738 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847752+0000", 2020-01-16T00:13:07.738 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:07.738 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.738 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.738 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:07.739 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847751+0000", 2020-01-16T00:13:07.739 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1980000000000001e-06 2020-01-16T00:13:07.739 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.739 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.739 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:07.739 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847754+0000", 2020-01-16T00:13:07.739 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6679999999999999e-06 2020-01-16T00:13:07.739 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.740 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.740 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:07.740 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847758+0000", 2020-01-16T00:13:07.740 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2500000000000001e-06 2020-01-16T00:13:07.740 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.740 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.741 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:07.741 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847760+0000", 2020-01-16T00:13:07.741 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7614000000000004e-05 2020-01-16T00:13:07.741 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.741 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.741 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.741 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847858+0000", 2020-01-16T00:13:07.742 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7950000000000004e-06 2020-01-16T00:13:07.742 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.742 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.742 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:13:07.742 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847864+0000", 2020-01-16T00:13:07.742 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010790345 2020-01-16T00:13:07.742 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.743 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.743 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.743 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858654+0000", 2020-01-16T00:13:07.743 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000141466 2020-01-16T00:13:07.743 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.743 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.743 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:07.744 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858795+0000", 2020-01-16T00:13:07.744 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00069037899999999997 2020-01-16T00:13:07.744 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.744 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.744 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:07.744 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.859486+0000", 2020-01-16T00:13:07.744 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011734249 2020-01-16T00:13:07.745 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.745 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.745 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.745 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.745 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.745 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:55 2.1 e30/19 2:91f85463:::smithi12313934-24 low_tier:head v 30'56, mlcod=29'54)", 2020-01-16T00:13:07.745 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.738134+0000", 2020-01-16T00:13:07.746 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 14.908928373, 2020-01-16T00:13:07.746 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0086509570000000008, 2020-01-16T00:13:07.746 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:07.746 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:07.746 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:07.746 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.747 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:07.747 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738134+0000", 2020-01-16T00:13:07.747 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:07.747 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.747 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.747 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:07.747 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738134+0000", 2020-01-16T00:13:07.747 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:07.748 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.748 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.748 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:07.748 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738134+0000", 2020-01-16T00:13:07.748 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00053712599999999997 2020-01-16T00:13:07.748 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.748 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.749 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:07.749 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738671+0000", 2020-01-16T00:13:07.749 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4059999999999999e-05 2020-01-16T00:13:07.749 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.749 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.749 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:07.749 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738685+0000", 2020-01-16T00:13:07.750 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7680000000000008e-06 2020-01-16T00:13:07.750 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.750 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.750 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:07.750 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738692+0000", 2020-01-16T00:13:07.750 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1273999999999996e-05 2020-01-16T00:13:07.751 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.751 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.751 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.751 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738764+0000", 2020-01-16T00:13:07.751 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6827999999999999e-05 2020-01-16T00:13:07.751 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.751 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.752 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:07.752 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738781+0000", 2020-01-16T00:13:07.752 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0079370740000000006 2020-01-16T00:13:07.752 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.752 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.752 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:07.752 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746718+0000", 2020-01-16T00:13:07.752 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7181000000000006e-05 2020-01-16T00:13:07.753 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.753 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.753 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:07.753 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746785+0000", 2020-01-16T00:13:07.753 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0086509570000000008 2020-01-16T00:13:07.754 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.754 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.754 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.754 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.754 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.754 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:1668 3.3 e30/20 3:c478bfe8:::smithi12313934-10:head v 30'77, mlcod=29'75)", 2020-01-16T00:13:07.754 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.753756+0000", 2020-01-16T00:13:07.754 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 14.893306322000001, 2020-01-16T00:13:07.755 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0077471110000000001, 2020-01-16T00:13:07.755 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:07.755 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:07.755 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:07.755 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.755 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:07.755 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.753756+0000", 2020-01-16T00:13:07.756 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:07.756 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.756 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.756 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:07.756 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.753756+0000", 2020-01-16T00:13:07.756 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:07.756 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.756 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.757 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:07.757 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.753754+0000", 2020-01-16T00:13:07.757 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00047439299999999998 2020-01-16T00:13:07.757 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.757 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.757 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:07.757 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.754229+0000", 2020-01-16T00:13:07.758 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1018e-05 2020-01-16T00:13:07.758 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.758 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.758 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:07.758 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.754240+0000", 2020-01-16T00:13:07.758 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1700000000000002e-06 2020-01-16T00:13:07.758 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.759 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.759 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:07.759 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.754246+0000", 2020-01-16T00:13:07.759 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0014229519999999999 2020-01-16T00:13:07.759 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.759 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.759 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.760 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.755669+0000", 2020-01-16T00:13:07.760 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8136e-05 2020-01-16T00:13:07.760 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.760 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.760 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:07.760 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.755697+0000", 2020-01-16T00:13:07.761 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0057384799999999998 2020-01-16T00:13:07.761 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.761 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.761 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:07.761 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.761435+0000", 2020-01-16T00:13:07.761 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7742000000000003e-05 2020-01-16T00:13:07.761 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.762 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.762 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:07.762 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.761503+0000", 2020-01-16T00:13:07.762 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0077471110000000001 2020-01-16T00:13:07.762 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.762 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.762 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.763 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.763 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.763 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2106 3.1 3:8de1d8a3:::smithi12313934-13:head [write 665610~797860,stat] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:13:07.763 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.631369+0000", 2020-01-16T00:13:07.763 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 6.0156929469999998, 2020-01-16T00:13:07.763 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0091195330000000008, 2020-01-16T00:13:07.763 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:07.764 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:07.764 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:07.764 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:07.764 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:07.764 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2106 2020-01-16T00:13:07.764 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.764 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:07.764 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.764 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:07.765 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631369+0000", 2020-01-16T00:13:07.765 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:07.765 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.765 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.765 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:07.765 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631369+0000", 2020-01-16T00:13:07.765 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:07.765 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.766 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.766 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:07.766 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631367+0000", 2020-01-16T00:13:07.766 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00025714100000000001 2020-01-16T00:13:07.766 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.766 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.766 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:07.766 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631624+0000", 2020-01-16T00:13:07.767 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4946000000000001e-05 2020-01-16T00:13:07.767 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.767 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.767 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:07.767 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631639+0000", 2020-01-16T00:13:07.767 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7050000000000002e-06 2020-01-16T00:13:07.768 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.768 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.768 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:07.768 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631646+0000", 2020-01-16T00:13:07.768 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010938900000000001 2020-01-16T00:13:07.768 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.768 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.768 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.769 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631755+0000", 2020-01-16T00:13:07.769 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8848e-05 2020-01-16T00:13:07.769 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.769 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.769 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op must wait for map", 2020-01-16T00:13:07.769 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631774+0000", 2020-01-16T00:13:07.769 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0026027730000000001 2020-01-16T00:13:07.770 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.770 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.770 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.770 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.634377+0000", 2020-01-16T00:13:07.770 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9282000000000003e-05 2020-01-16T00:13:07.770 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.771 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.771 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:07.771 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.634466+0000", 2020-01-16T00:13:07.771 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000127868 2020-01-16T00:13:07.771 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.771 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.771 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:13:07.772 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.634594+0000", 2020-01-16T00:13:07.772 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030776549999999999 2020-01-16T00:13:07.772 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.772 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.772 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:07.772 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.637672+0000", 2020-01-16T00:13:07.772 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001634195 2020-01-16T00:13:07.772 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.773 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.773 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:07.773 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.639306+0000", 2020-01-16T00:13:07.773 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010810069999999999 2020-01-16T00:13:07.773 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.773 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.773 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:07.774 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640387+0000", 2020-01-16T00:13:07.774 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0543000000000001e-05 2020-01-16T00:13:07.774 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.774 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.774 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:07.774 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640428+0000", 2020-01-16T00:13:07.774 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1321000000000005e-05 2020-01-16T00:13:07.775 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.775 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.775 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:07.775 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640489+0000", 2020-01-16T00:13:07.775 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0091195330000000008 2020-01-16T00:13:07.775 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.775 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.776 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.776 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.776 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.776 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2107 3.1 3:8de1d8a3:::smithi12313934-13:head [write 1909329~458405] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:13:07.776 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.636786+0000", 2020-01-16T00:13:07.776 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 6.0102758170000001, 2020-01-16T00:13:07.776 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093182809999999994, 2020-01-16T00:13:07.776 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:07.777 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:07.777 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:07.777 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:07.777 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:07.777 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2107 2020-01-16T00:13:07.777 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.777 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:07.777 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.778 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:07.778 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636786+0000", 2020-01-16T00:13:07.778 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:07.778 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.778 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.778 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:07.778 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636786+0000", 2020-01-16T00:13:07.779 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:07.779 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.779 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.779 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:07.779 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636784+0000", 2020-01-16T00:13:07.779 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014276299999999999 2020-01-16T00:13:07.779 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.779 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.780 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:07.780 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636927+0000", 2020-01-16T00:13:07.780 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5017999999999999e-05 2020-01-16T00:13:07.780 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.780 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.780 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:07.780 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636942+0000", 2020-01-16T00:13:07.781 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5589999999999996e-06 2020-01-16T00:13:07.781 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.781 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.781 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:07.781 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636947+0000", 2020-01-16T00:13:07.782 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00040279699999999999 2020-01-16T00:13:07.782 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.782 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.782 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.782 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.637350+0000", 2020-01-16T00:13:07.782 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000233506 2020-01-16T00:13:07.782 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.782 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.783 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:07.783 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.637584+0000", 2020-01-16T00:13:07.783 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029534660000000001 2020-01-16T00:13:07.783 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.783 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.783 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.783 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640537+0000", 2020-01-16T00:13:07.784 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012577700000000001 2020-01-16T00:13:07.784 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.784 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.784 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:07.784 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640663+0000", 2020-01-16T00:13:07.784 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000198593 2020-01-16T00:13:07.784 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.785 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.785 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:13:07.785 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640862+0000", 2020-01-16T00:13:07.785 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.005088197 2020-01-16T00:13:07.785 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.785 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.785 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:07.785 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645950+0000", 2020-01-16T00:13:07.786 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2568999999999999e-05 2020-01-16T00:13:07.786 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.786 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.786 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:07.786 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645962+0000", 2020-01-16T00:13:07.786 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3034999999999995e-05 2020-01-16T00:13:07.786 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.787 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.787 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:07.787 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646025+0000", 2020-01-16T00:13:07.787 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7762999999999997e-05 2020-01-16T00:13:07.787 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.787 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.787 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:07.788 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646063+0000", 2020-01-16T00:13:07.788 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1541000000000001e-05 2020-01-16T00:13:07.788 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.788 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.788 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:07.788 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646105+0000", 2020-01-16T00:13:07.788 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093182809999999994 2020-01-16T00:13:07.788 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.789 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.789 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.789 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.789 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.789 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2108 3.1 3:8de1d8a3:::smithi12313934-13:head [write 2816700~410210] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:13:07.789 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.641861+0000", 2020-01-16T00:13:07.789 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 6.0052014690000002, 2020-01-16T00:13:07.790 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018545604, 2020-01-16T00:13:07.790 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:07.790 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:07.790 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:07.790 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:07.790 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:07.790 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2108 2020-01-16T00:13:07.790 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.791 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:07.791 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.791 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:07.791 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.641861+0000", 2020-01-16T00:13:07.791 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:07.791 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.792 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.792 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:07.792 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.641861+0000", 2020-01-16T00:13:07.792 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:07.792 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.792 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.792 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:07.793 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.641859+0000", 2020-01-16T00:13:07.793 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018466399999999999 2020-01-16T00:13:07.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.793 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.793 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:07.793 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642043+0000", 2020-01-16T00:13:07.793 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2999999999999999e-05 2020-01-16T00:13:07.794 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.794 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.794 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:07.794 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642056+0000", 2020-01-16T00:13:07.794 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0730000000000005e-06 2020-01-16T00:13:07.794 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.794 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.795 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:07.795 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642064+0000", 2020-01-16T00:13:07.795 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9527999999999997e-05 2020-01-16T00:13:07.795 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.795 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.795 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.795 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642114+0000", 2020-01-16T00:13:07.795 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036616050000000001 2020-01-16T00:13:07.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.796 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.796 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:07.796 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645776+0000", 2020-01-16T00:13:07.796 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012222663999999999 2020-01-16T00:13:07.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.796 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.797 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.797 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.657998+0000", 2020-01-16T00:13:07.797 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014280299999999999 2020-01-16T00:13:07.797 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.797 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.797 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:07.797 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.658141+0000", 2020-01-16T00:13:07.797 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0001102 2020-01-16T00:13:07.798 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.798 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.798 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:13:07.798 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.658251+0000", 2020-01-16T00:13:07.798 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020421219999999999 2020-01-16T00:13:07.798 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.798 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.799 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:07.799 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660293+0000", 2020-01-16T00:13:07.799 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7399999999999999e-06 2020-01-16T00:13:07.799 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.799 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.799 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:07.799 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660303+0000", 2020-01-16T00:13:07.799 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9873999999999998e-05 2020-01-16T00:13:07.800 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.800 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.800 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:07.800 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660343+0000", 2020-01-16T00:13:07.800 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4448e-05 2020-01-16T00:13:07.800 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.800 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.801 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:07.801 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660377+0000", 2020-01-16T00:13:07.801 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8915999999999999e-05 2020-01-16T00:13:07.801 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.801 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.801 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:07.802 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660406+0000", 2020-01-16T00:13:07.802 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018545604 2020-01-16T00:13:07.802 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.802 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.802 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.802 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.803 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.803 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2109 3.1 3:8de1d8a3:::smithi12313934-13:head [setxattr _header (56) in=7b,truncate 3865610] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:13:07.803 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.642075+0000", 2020-01-16T00:13:07.803 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 6.0049868960000001, 2020-01-16T00:13:07.803 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018477655999999999, 2020-01-16T00:13:07.803 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:07.804 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:07.804 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:07.804 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:07.804 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:07.804 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2109 2020-01-16T00:13:07.804 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.804 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:07.804 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.805 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:07.805 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642075+0000", 2020-01-16T00:13:07.805 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:07.805 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.805 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.805 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:07.805 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642075+0000", 2020-01-16T00:13:07.806 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:07.806 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.806 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.806 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:07.806 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642074+0000", 2020-01-16T00:13:07.806 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4500000000000003e-06 2020-01-16T00:13:07.806 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.806 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.807 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:07.807 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642080+0000", 2020-01-16T00:13:07.807 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7229999999999998e-06 2020-01-16T00:13:07.807 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.807 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.807 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:07.807 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642087+0000", 2020-01-16T00:13:07.808 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2390000000000004e-06 2020-01-16T00:13:07.808 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.808 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.808 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:07.808 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642091+0000", 2020-01-16T00:13:07.808 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0037396550000000001 2020-01-16T00:13:07.808 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.809 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.809 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.809 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645830+0000", 2020-01-16T00:13:07.809 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2200000000000002e-05 2020-01-16T00:13:07.809 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.809 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.809 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:07.810 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645883+0000", 2020-01-16T00:13:07.810 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013432934000000001 2020-01-16T00:13:07.810 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.810 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.810 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.810 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659316+0000", 2020-01-16T00:13:07.810 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000140006 2020-01-16T00:13:07.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.811 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.811 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:07.811 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659456+0000", 2020-01-16T00:13:07.811 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000184873 2020-01-16T00:13:07.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.812 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.812 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:13:07.812 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659640+0000", 2020-01-16T00:13:07.812 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00066954999999999998 2020-01-16T00:13:07.812 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.812 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.812 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:07.813 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660310+0000", 2020-01-16T00:13:07.813 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014847699999999999 2020-01-16T00:13:07.813 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.813 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.813 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:07.813 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660459+0000", 2020-01-16T00:13:07.813 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6869e-05 2020-01-16T00:13:07.814 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.814 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.814 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:07.814 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660485+0000", 2020-01-16T00:13:07.814 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8478000000000001e-05 2020-01-16T00:13:07.814 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.814 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.815 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:07.815 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660514+0000", 2020-01-16T00:13:07.815 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9153999999999998e-05 2020-01-16T00:13:07.815 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.815 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.815 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:07.815 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660553+0000", 2020-01-16T00:13:07.815 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018477655999999999 2020-01-16T00:13:07.816 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.816 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.816 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.816 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.816 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.816 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2110 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-01-16T00:13:07.816 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.642112+0000", 2020-01-16T00:13:07.817 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 6.004950258, 2020-01-16T00:13:07.817 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018620198000000001, 2020-01-16T00:13:07.817 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:07.817 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:07.817 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:07.817 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:07.817 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:07.817 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2110 2020-01-16T00:13:07.818 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.818 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:07.818 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.818 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:07.818 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642112+0000", 2020-01-16T00:13:07.818 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:07.818 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.818 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.819 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:07.819 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642112+0000", 2020-01-16T00:13:07.819 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:07.819 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.819 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.819 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:07.819 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642111+0000", 2020-01-16T00:13:07.820 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4450000000000001e-06 2020-01-16T00:13:07.820 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.820 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.820 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:07.820 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642117+0000", 2020-01-16T00:13:07.820 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1567e-05 2020-01-16T00:13:07.820 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.821 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.821 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:07.821 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642129+0000", 2020-01-16T00:13:07.821 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.754e-06 2020-01-16T00:13:07.821 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.821 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.821 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:07.822 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642136+0000", 2020-01-16T00:13:07.822 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040239719999999998 2020-01-16T00:13:07.822 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.822 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.822 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.822 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646160+0000", 2020-01-16T00:13:07.823 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5600999999999998e-05 2020-01-16T00:13:07.823 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.823 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.823 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:07.823 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646215+0000", 2020-01-16T00:13:07.823 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013733006000000001 2020-01-16T00:13:07.823 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.824 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.824 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.824 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659948+0000", 2020-01-16T00:13:07.824 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000106277 2020-01-16T00:13:07.824 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.824 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.824 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:07.825 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660054+0000", 2020-01-16T00:13:07.825 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000524327 2020-01-16T00:13:07.825 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.825 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.825 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.825 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660579+0000", 2020-01-16T00:13:07.825 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9841999999999999e-05 2020-01-16T00:13:07.826 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.826 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.826 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:07.826 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660659+0000", 2020-01-16T00:13:07.826 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3572999999999995e-05 2020-01-16T00:13:07.826 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.826 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.827 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:07.827 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660732+0000", 2020-01-16T00:13:07.827 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018620198000000001 2020-01-16T00:13:07.827 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.827 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.827 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.827 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.828 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.828 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:70 2.1 e35/19 2:91f85463:::smithi12313934-24 low_tier:head v 35'66, mlcod=34'64)", 2020-01-16T00:13:07.828 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.986684+0000", 2020-01-16T00:13:07.828 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 1.660378404, 2020-01-16T00:13:07.828 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012287378999999999, 2020-01-16T00:13:07.828 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:07.828 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:07.828 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:07.829 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.829 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:07.829 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986684+0000", 2020-01-16T00:13:07.829 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:07.829 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.829 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.829 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:07.830 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986684+0000", 2020-01-16T00:13:07.830 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:07.830 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.830 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.830 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:07.830 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986683+0000", 2020-01-16T00:13:07.830 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012884179999999999 2020-01-16T00:13:07.830 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.830 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.831 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:07.831 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.987971+0000", 2020-01-16T00:13:07.831 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011558200000000001 2020-01-16T00:13:07.831 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.831 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.831 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:07.831 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988087+0000", 2020-01-16T00:13:07.831 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0472000000000001e-05 2020-01-16T00:13:07.832 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.832 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.832 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:07.832 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988137+0000", 2020-01-16T00:13:07.832 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000107484 2020-01-16T00:13:07.832 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.832 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.833 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.833 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988245+0000", 2020-01-16T00:13:07.833 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4409999999999998e-05 2020-01-16T00:13:07.833 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.833 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.833 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:07.833 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988279+0000", 2020-01-16T00:13:07.833 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010641609999999999 2020-01-16T00:13:07.834 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.834 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.834 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:07.834 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.998921+0000", 2020-01-16T00:13:07.834 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0278000000000002e-05 2020-01-16T00:13:07.834 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.834 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.834 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:07.835 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.998971+0000", 2020-01-16T00:13:07.835 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012287378999999999 2020-01-16T00:13:07.835 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.835 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.835 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.835 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.835 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.835 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:71 2.1 e35/19 2:91f85463:::smithi12313934-24 low_tier:head v 35'67, mlcod=34'64)", 2020-01-16T00:13:07.836 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.991132+0000", 2020-01-16T00:13:07.836 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 1.6559304539999999, 2020-01-16T00:13:07.836 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0082681190000000009, 2020-01-16T00:13:07.836 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:07.836 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:07.836 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:07.836 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.836 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:07.837 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.991132+0000", 2020-01-16T00:13:07.837 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:07.837 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.837 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.837 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:07.837 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.991132+0000", 2020-01-16T00:13:07.837 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:07.838 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.838 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.838 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:07.838 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.991130+0000", 2020-01-16T00:13:07.838 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3583000000000003e-05 2020-01-16T00:13:07.838 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.839 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.839 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:07.839 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.991203+0000", 2020-01-16T00:13:07.839 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.1001000000000001e-05 2020-01-16T00:13:07.839 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.839 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.839 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:07.840 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.991224+0000", 2020-01-16T00:13:07.840 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9329999999999996e-06 2020-01-16T00:13:07.840 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.840 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.840 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:07.840 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.991234+0000", 2020-01-16T00:13:07.840 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0035280070000000001 2020-01-16T00:13:07.840 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.841 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.841 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.841 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.994762+0000", 2020-01-16T00:13:07.841 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.622e-05 2020-01-16T00:13:07.841 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.841 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.841 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:07.842 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.994788+0000", 2020-01-16T00:13:07.842 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0045652219999999999 2020-01-16T00:13:07.842 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.842 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.842 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:07.842 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999354+0000", 2020-01-16T00:13:07.843 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6396000000000001e-05 2020-01-16T00:13:07.843 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.843 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.843 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:07.843 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999400+0000", 2020-01-16T00:13:07.843 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0082681190000000009 2020-01-16T00:13:07.843 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.843 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.844 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.844 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.844 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.844 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:77 2.1 e36/19 2:91f85463:::smithi12313934-24 low_tier:head v 36'69, mlcod=35'67)", 2020-01-16T00:13:07.844 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.556730+0000", 2020-01-16T00:13:07.844 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.090332049999999997, 2020-01-16T00:13:07.844 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093007760000000002, 2020-01-16T00:13:07.845 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:07.845 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:07.845 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:07.845 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.845 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:07.845 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556730+0000", 2020-01-16T00:13:07.845 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:07.846 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.846 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.846 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:07.846 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556730+0000", 2020-01-16T00:13:07.846 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:07.846 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.846 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.847 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:07.847 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556728+0000", 2020-01-16T00:13:07.847 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00056509599999999996 2020-01-16T00:13:07.847 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.847 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.847 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:07.847 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.557293+0000", 2020-01-16T00:13:07.848 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0225e-05 2020-01-16T00:13:07.848 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.848 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.848 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:07.848 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.557314+0000", 2020-01-16T00:13:07.848 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8989999999999996e-06 2020-01-16T00:13:07.848 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.849 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.849 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:07.849 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.557323+0000", 2020-01-16T00:13:07.849 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7923000000000002e-05 2020-01-16T00:13:07.849 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.849 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.849 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:07.850 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.557371+0000", 2020-01-16T00:13:07.850 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9963999999999999e-05 2020-01-16T00:13:07.850 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.850 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.850 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:07.850 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.557411+0000", 2020-01-16T00:13:07.850 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0085873000000000008 2020-01-16T00:13:07.851 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.851 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.851 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:07.851 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.565999+0000", 2020-01-16T00:13:07.851 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2326000000000002e-05 2020-01-16T00:13:07.851 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:07.851 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:07.851 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:07.852 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566031+0000", 2020-01-16T00:13:07.852 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093007760000000002 2020-01-16T00:13:07.852 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.852 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.852 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.852 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:07.853 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:07.853 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:07.853 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:07.646+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:07.855 INFO:tasks.rados.rados.0.smithi123.stdout:1020: finishing write tid 1 to smithi12313934-7 2020-01-16T00:13:07.855 INFO:tasks.rados.rados.0.smithi123.stdout:1020: finishing write tid 2 to smithi12313934-7 2020-01-16T00:13:07.855 INFO:tasks.rados.rados.0.smithi123.stdout:1020: finishing write tid 4 to smithi12313934-7 2020-01-16T00:13:07.855 INFO:tasks.rados.rados.0.smithi123.stdout:1020: finishing write tid 5 to smithi12313934-7 2020-01-16T00:13:07.855 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 7 v 140 (ObjNum 252 snap 0 seq_num 252) dirty exists 2020-01-16T00:13:07.855 INFO:tasks.rados.rados.0.smithi123.stdout:1020: left oid 7 (ObjNum 252 snap 0 seq_num 252) 2020-01-16T00:13:07.855 INFO:tasks.rados.rados.0.smithi123.stdout:1021: expect (ObjNum 227 snap 0 seq_num 227) 2020-01-16T00:13:07.856 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:07.852+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:07.856 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-01-16T00:13:07.945 INFO:tasks.rados.rados.0.smithi123.stdout:1022: finishing write tid 1 to smithi12313934-6 2020-01-16T00:13:07.946 INFO:tasks.rados.rados.0.smithi123.stdout:1023: expect (ObjNum 240 snap 0 seq_num 240) 2020-01-16T00:13:07.955 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-01-16T00:13:07.956 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:13:07.952+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:13:07.998 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:07.998 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:07.998 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:13:07.999 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:08.007 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-01-16T00:13:08.021 INFO:tasks.rados.rados.0.smithi123.stdout:1022: finishing write tid 2 to smithi12313934-6 2020-01-16T00:13:08.021 INFO:tasks.rados.rados.0.smithi123.stdout:1022: finishing write tid 4 to smithi12313934-6 2020-01-16T00:13:08.022 INFO:tasks.rados.rados.0.smithi123.stdout:1022: finishing write tid 5 to smithi12313934-6 2020-01-16T00:13:08.022 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 6 v 72 (ObjNum 253 snap 0 seq_num 253) dirty exists 2020-01-16T00:13:08.022 INFO:tasks.rados.rados.0.smithi123.stdout:1022: left oid 6 (ObjNum 253 snap 0 seq_num 253) 2020-01-16T00:13:08.022 INFO:tasks.rados.rados.0.smithi123.stdout:1025: finishing write tid 1 to smithi12313934-25 2020-01-16T00:13:08.022 INFO:tasks.rados.rados.0.smithi123.stdout:1024: expect (ObjNum 232 snap 0 seq_num 232) 2020-01-16T00:13:08.056 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:08.052+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:08.092 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:13:08.112 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:13:08.119 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:08.119 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:08.119 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:13:08.119 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:13:08.119 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:08.128 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-01-16T00:13:08.138 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:13:08.156 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:08.154+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:08.172 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:13:08.196 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:13:08.219 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5: {} 2020-01-16T00:13:08.229 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:08.237 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-01-16T00:13:08.242 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:13:08.257 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:08.252+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:08.266 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:13:08.268 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-01-16T00:13:08.268 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-01-16T00:13:08.269 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-01-16T00:13:08.269 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-01-16T00:13:08.269 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-01-16T00:13:08.269 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-01-16T00:13:08.269 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-01-16T00:13:08.269 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-01-16T00:13:08.326 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4, 7, 6] 2020-01-16T00:13:08.326 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2020-01-16T00:13:08.326 INFO:tasks.thrashosds.thrasher:Reweighting by: {'overage': '1000', 'type': 'reweight-by-utilization', 'max_change': '0.05'} 2020-01-16T00:13:08.327 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 1000 0.05 2020-01-16T00:13:08.331 INFO:tasks.rados.rados.0.smithi123.stdout:1025: finishing write tid 2 to smithi12313934-25 2020-01-16T00:13:08.331 INFO:tasks.rados.rados.0.smithi123.stdout:1025: finishing write tid 4 to smithi12313934-25 2020-01-16T00:13:08.331 INFO:tasks.rados.rados.0.smithi123.stdout:1025: finishing write tid 5 to smithi12313934-25 2020-01-16T00:13:08.331 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 25 v 150 (ObjNum 254 snap 0 seq_num 254) dirty exists 2020-01-16T00:13:08.331 INFO:tasks.rados.rados.0.smithi123.stdout:1025: left oid 25 (ObjNum 254 snap 0 seq_num 254) 2020-01-16T00:13:08.331 INFO:tasks.rados.rados.0.smithi123.stdout:1028: finishing write tid 1 to smithi12313934-41 2020-01-16T00:13:08.331 INFO:tasks.rados.rados.0.smithi123.stdout:1026: expect (ObjNum 243 snap 0 seq_num 243) 2020-01-16T00:13:08.339 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:08.348 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-01-16T00:13:08.393 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:08.390+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:08.448 INFO:tasks.rados.rados.0.smithi123.stdout:1027: expect (ObjNum 228 snap 0 seq_num 228) 2020-01-16T00:13:08.459 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:08.454+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:08.505 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:08.514 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-01-16T00:13:08.560 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:13:08.556+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:13:08.571 INFO:teuthology.orchestra.run.smithi123.stdout:moved 0 / 84 (0%) 2020-01-16T00:13:08.571 INFO:teuthology.orchestra.run.smithi123.stdout:avg 10.5 2020-01-16T00:13:08.572 INFO:teuthology.orchestra.run.smithi123.stdout:stddev 1.5 -> 1.5 (expected baseline 3.03109) 2020-01-16T00:13:08.572 INFO:teuthology.orchestra.run.smithi123.stdout:min osd.2 with 8 -> 8 pgs (0.761905 -> 0.761905 * mean) 2020-01-16T00:13:08.572 INFO:teuthology.orchestra.run.smithi123.stdout:max osd.0 with 12 -> 12 pgs (1.14286 -> 1.14286 * mean) 2020-01-16T00:13:08.572 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:13:08.573 INFO:teuthology.orchestra.run.smithi123.stdout:oload 1000 2020-01-16T00:13:08.573 INFO:teuthology.orchestra.run.smithi123.stdout:max_change 0.05 2020-01-16T00:13:08.581 INFO:teuthology.orchestra.run.smithi123.stdout:max_change_osds 4 2020-01-16T00:13:08.581 INFO:teuthology.orchestra.run.smithi123.stdout:average_utilization 0.0120 2020-01-16T00:13:08.581 INFO:teuthology.orchestra.run.smithi123.stdout:overload_utilization 0.1204 2020-01-16T00:13:08.582 INFO:teuthology.orchestra.run.smithi123.stderr:no change 2020-01-16T00:13:08.584 INFO:tasks.thrashosds.thrasher:Reweighting by: {'overage': '110', 'type': 'reweight-by-utilization', 'max_change': '1.0'} 2020-01-16T00:13:08.584 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 110 1.0 2020-01-16T00:13:08.611 INFO:tasks.rados.rados.0.smithi123.stdout:1028: finishing write tid 2 to smithi12313934-41 2020-01-16T00:13:08.611 INFO:tasks.rados.rados.0.smithi123.stdout:1028: finishing write tid 4 to smithi12313934-41 2020-01-16T00:13:08.611 INFO:tasks.rados.rados.0.smithi123.stdout:1028: finishing write tid 5 to smithi12313934-41 2020-01-16T00:13:08.612 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 41 v 143 (ObjNum 255 snap 0 seq_num 255) dirty exists 2020-01-16T00:13:08.612 INFO:tasks.rados.rados.0.smithi123.stdout:1028: left oid 41 (ObjNum 255 snap 0 seq_num 255) 2020-01-16T00:13:08.612 INFO:tasks.rados.rados.0.smithi123.stdout:1029: finishing write tid 1 to smithi12313934-19 2020-01-16T00:13:08.612 INFO:tasks.rados.rados.0.smithi123.stdout:1016: done (15 left) 2020-01-16T00:13:08.612 INFO:tasks.rados.rados.0.smithi123.stdout:1020: done (14 left) 2020-01-16T00:13:08.613 INFO:tasks.rados.rados.0.smithi123.stdout:1021: done (13 left) 2020-01-16T00:13:08.613 INFO:tasks.rados.rados.0.smithi123.stdout:1022: done (12 left) 2020-01-16T00:13:08.613 INFO:tasks.rados.rados.0.smithi123.stdout:1023: done (11 left) 2020-01-16T00:13:08.613 INFO:tasks.rados.rados.0.smithi123.stdout:1024: done (10 left) 2020-01-16T00:13:08.614 INFO:tasks.rados.rados.0.smithi123.stdout:1025: done (9 left) 2020-01-16T00:13:08.614 INFO:tasks.rados.rados.0.smithi123.stdout:1026: done (8 left) 2020-01-16T00:13:08.614 INFO:tasks.rados.rados.0.smithi123.stdout:1027: done (7 left) 2020-01-16T00:13:08.614 INFO:tasks.rados.rados.0.smithi123.stdout:1028: done (6 left) 2020-01-16T00:13:08.614 INFO:tasks.rados.rados.0.smithi123.stdout:1035: write oid 3 current snap is 0 2020-01-16T00:13:08.615 INFO:tasks.rados.rados.0.smithi123.stdout:1035: seq_num 260 ranges {769820=720742,2114730=255090} 2020-01-16T00:13:08.625 INFO:tasks.rados.rados.0.smithi123.stdout:1035: writing smithi12313934-3 from 769820 to 1490562 tid 1 2020-01-16T00:13:08.632 INFO:tasks.rados.rados.0.smithi123.stdout:1035: writing smithi12313934-3 from 2114730 to 2369820 tid 2 2020-01-16T00:13:08.632 INFO:tasks.rados.rados.0.smithi123.stdout:1029: finishing write tid 2 to smithi12313934-19 2020-01-16T00:13:08.633 INFO:tasks.rados.rados.0.smithi123.stdout:1029: finishing write tid 4 to smithi12313934-19 2020-01-16T00:13:08.633 INFO:tasks.rados.rados.0.smithi123.stdout:1029: finishing write tid 5 to smithi12313934-19 2020-01-16T00:13:08.633 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 19 v 146 (ObjNum 256 snap 0 seq_num 256) dirty exists 2020-01-16T00:13:08.633 INFO:tasks.rados.rados.0.smithi123.stdout:1029: left oid 19 (ObjNum 256 snap 0 seq_num 256) 2020-01-16T00:13:08.634 INFO:tasks.rados.rados.0.smithi123.stdout:1030: finishing write tid 1 to smithi12313934-29 2020-01-16T00:13:08.634 INFO:tasks.rados.rados.0.smithi123.stdout:1030: finishing write tid 2 to smithi12313934-29 2020-01-16T00:13:08.634 INFO:tasks.rados.rados.0.smithi123.stdout:1036: read oid 12 snap -1 2020-01-16T00:13:08.634 INFO:tasks.rados.rados.0.smithi123.stdout:1036: expect (ObjNum 221 snap 0 seq_num 221) 2020-01-16T00:13:08.634 INFO:tasks.rados.rados.0.smithi123.stdout:1029: done (7 left) 2020-01-16T00:13:08.635 INFO:tasks.rados.rados.0.smithi123.stdout:1037: read oid 27 snap -1 2020-01-16T00:13:08.635 INFO:tasks.rados.rados.0.smithi123.stdout:1037: expect deleted 2020-01-16T00:13:08.635 INFO:tasks.rados.rados.0.smithi123.stdout:1038: write (excl) oid 42 current snap is 0 2020-01-16T00:13:08.635 INFO:tasks.rados.rados.0.smithi123.stdout:1038: seq_num 261 ranges {786627=795991,2173212=213415} 2020-01-16T00:13:08.642 INFO:tasks.rados.rados.0.smithi123.stdout:1038: writing smithi12313934-42 from 786627 to 1582618 tid 1 2020-01-16T00:13:08.646 INFO:tasks.rados.rados.0.smithi123.stdout:1038: writing smithi12313934-42 from 2173212 to 2386627 tid 2 2020-01-16T00:13:08.646 INFO:tasks.rados.rados.0.smithi123.stdout:1039: delete oid 36 current snap is 0 2020-01-16T00:13:08.646 INFO:tasks.rados.rados.0.smithi123.stdout:1032: expect (ObjNum 188 snap 0 seq_num 188) 2020-01-16T00:13:08.661 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:08.656+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:08.680 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:08.692 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-01-16T00:13:08.761 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:08.758+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:08.762 INFO:tasks.rados.rados.0.smithi123.stdout:1031: expect (ObjNum 212 snap 0 seq_num 212) 2020-01-16T00:13:08.806 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2020-01-16T00:13:08.807 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:08.817 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-01-16T00:13:08.854 INFO:teuthology.orchestra.run.smithi123.stdout:moved 0 / 84 (0%) 2020-01-16T00:13:08.854 INFO:teuthology.orchestra.run.smithi123.stdout:avg 10.5 2020-01-16T00:13:08.854 INFO:teuthology.orchestra.run.smithi123.stdout:stddev 1.5 -> 1.5 (expected baseline 3.03109) 2020-01-16T00:13:08.854 INFO:teuthology.orchestra.run.smithi123.stdout:min osd.2 with 8 -> 8 pgs (0.761905 -> 0.761905 * mean) 2020-01-16T00:13:08.854 INFO:teuthology.orchestra.run.smithi123.stdout:max osd.0 with 12 -> 12 pgs (1.14286 -> 1.14286 * mean) 2020-01-16T00:13:08.855 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:13:08.855 INFO:teuthology.orchestra.run.smithi123.stdout:oload 110 2020-01-16T00:13:08.855 INFO:teuthology.orchestra.run.smithi123.stdout:max_change 1 2020-01-16T00:13:08.855 INFO:teuthology.orchestra.run.smithi123.stdout:max_change_osds 4 2020-01-16T00:13:08.855 INFO:teuthology.orchestra.run.smithi123.stdout:average_utilization 0.0120 2020-01-16T00:13:08.855 INFO:teuthology.orchestra.run.smithi123.stdout:overload_utilization 0.0132 2020-01-16T00:13:08.856 INFO:teuthology.orchestra.run.smithi123.stderr:no change 2020-01-16T00:13:08.863 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:08.860+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:08.865 INFO:tasks.thrashosds.thrasher:Reweighting by: {'overage': '110', 'type': 'reweight-by-utilization', 'max_change': '0.05'} 2020-01-16T00:13:08.865 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 110 0.05 2020-01-16T00:13:08.928 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:08.937 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-01-16T00:13:08.963 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:08.958+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:08.989 INFO:tasks.rados.rados.0.smithi123.stdout:1030: finishing write tid 4 to smithi12313934-29 2020-01-16T00:13:08.989 INFO:tasks.rados.rados.0.smithi123.stdout:1030: finishing write tid 5 to smithi12313934-29 2020-01-16T00:13:08.989 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 29 v 95 (ObjNum 257 snap 0 seq_num 257) dirty exists 2020-01-16T00:13:08.989 INFO:tasks.rados.rados.0.smithi123.stdout:1030: left oid 29 (ObjNum 257 snap 0 seq_num 257) 2020-01-16T00:13:08.989 INFO:tasks.rados.rados.0.smithi123.stdout:1033: finishing write tid 1 to smithi12313934-24 2020-01-16T00:13:08.990 INFO:tasks.rados.rados.0.smithi123.stdout:1033: finishing write tid 2 to smithi12313934-24 2020-01-16T00:13:08.990 INFO:tasks.rados.rados.0.smithi123.stdout:1034: finishing write tid 1 to smithi12313934-9 2020-01-16T00:13:08.990 INFO:tasks.rados.rados.0.smithi123.stdout:1033: finishing write tid 4 to smithi12313934-24 2020-01-16T00:13:08.990 INFO:tasks.rados.rados.0.smithi123.stdout:1033: finishing write tid 5 to smithi12313934-24 2020-01-16T00:13:08.990 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 24 v 99 (ObjNum 258 snap 0 seq_num 258) dirty exists 2020-01-16T00:13:08.990 INFO:tasks.rados.rados.0.smithi123.stdout:1033: left oid 24 (ObjNum 258 snap 0 seq_num 258) 2020-01-16T00:13:08.991 INFO:tasks.rados.rados.0.smithi123.stdout:1034: finishing write tid 2 to smithi12313934-9 2020-01-16T00:13:08.991 INFO:tasks.rados.rados.0.smithi123.stdout:1034: finishing write tid 4 to smithi12313934-9 2020-01-16T00:13:08.991 INFO:tasks.rados.rados.0.smithi123.stdout:1034: finishing write tid 5 to smithi12313934-9 2020-01-16T00:13:08.991 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 9 v 89 (ObjNum 259 snap 0 seq_num 259) dirty exists 2020-01-16T00:13:08.991 INFO:tasks.rados.rados.0.smithi123.stdout:1034: left oid 9 (ObjNum 259 snap 0 seq_num 259) 2020-01-16T00:13:08.991 INFO:tasks.rados.rados.0.smithi123.stdout:1030: done (9 left) 2020-01-16T00:13:08.991 INFO:tasks.rados.rados.0.smithi123.stdout:1031: done (8 left) 2020-01-16T00:13:08.991 INFO:tasks.rados.rados.0.smithi123.stdout:1032: done (7 left) 2020-01-16T00:13:08.992 INFO:tasks.rados.rados.0.smithi123.stdout:1033: done (6 left) 2020-01-16T00:13:08.992 INFO:tasks.rados.rados.0.smithi123.stdout:1034: done (5 left) 2020-01-16T00:13:08.992 INFO:tasks.rados.rados.0.smithi123.stdout:1039: done (4 left) 2020-01-16T00:13:08.992 INFO:tasks.rados.rados.0.smithi123.stdout:1040: read oid 28 snap -1 2020-01-16T00:13:08.992 INFO:tasks.rados.rados.0.smithi123.stdout:1040: expect (ObjNum 242 snap 0 seq_num 242) 2020-01-16T00:13:08.992 INFO:tasks.rados.rados.0.smithi123.stdout:1041: read oid 5 snap -1 2020-01-16T00:13:08.992 INFO:tasks.rados.rados.0.smithi123.stdout:1041: expect (ObjNum 238 snap 0 seq_num 238) 2020-01-16T00:13:08.993 INFO:tasks.rados.rados.0.smithi123.stdout:1042: write (excl) oid 26 current snap is 0 2020-01-16T00:13:08.993 INFO:tasks.rados.rados.0.smithi123.stdout:1042: seq_num 263 ranges {420241=546489,1373823=544414} 2020-01-16T00:13:08.994 INFO:tasks.rados.rados.0.smithi123.stdout:1042: writing smithi12313934-26 from 420241 to 966730 tid 1 2020-01-16T00:13:09.000 INFO:tasks.rados.rados.0.smithi123.stdout:1042: writing smithi12313934-26 from 1373823 to 1918237 tid 2 2020-01-16T00:13:09.000 INFO:tasks.rados.rados.0.smithi123.stdout:1043: delete oid 31 current snap is 0 2020-01-16T00:13:09.000 INFO:tasks.rados.rados.0.smithi123.stdout:1035: finishing write tid 1 to smithi12313934-3 2020-01-16T00:13:09.001 INFO:tasks.rados.rados.0.smithi123.stdout:1035: finishing write tid 2 to smithi12313934-3 2020-01-16T00:13:09.001 INFO:tasks.rados.rados.0.smithi123.stdout:1035: finishing write tid 4 to smithi12313934-3 2020-01-16T00:13:09.001 INFO:tasks.rados.rados.0.smithi123.stdout:1035: finishing write tid 5 to smithi12313934-3 2020-01-16T00:13:09.001 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 3 v 76 (ObjNum 260 snap 0 seq_num 260) dirty exists 2020-01-16T00:13:09.001 INFO:tasks.rados.rados.0.smithi123.stdout:1035: left oid 3 (ObjNum 260 snap 0 seq_num 260) 2020-01-16T00:13:09.001 INFO:tasks.rados.rados.0.smithi123.stdout:1036: expect (ObjNum 221 snap 0 seq_num 221) 2020-01-16T00:13:09.064 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:09.058+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:09.088 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:09.098 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-01-16T00:13:09.119 INFO:teuthology.orchestra.run.smithi123.stdout:moved 0 / 84 (0%) 2020-01-16T00:13:09.119 INFO:teuthology.orchestra.run.smithi123.stdout:avg 10.5 2020-01-16T00:13:09.119 INFO:teuthology.orchestra.run.smithi123.stdout:stddev 1.5 -> 1.5 (expected baseline 3.03109) 2020-01-16T00:13:09.119 INFO:teuthology.orchestra.run.smithi123.stdout:min osd.2 with 8 -> 8 pgs (0.761905 -> 0.761905 * mean) 2020-01-16T00:13:09.119 INFO:teuthology.orchestra.run.smithi123.stdout:max osd.0 with 12 -> 12 pgs (1.14286 -> 1.14286 * mean) 2020-01-16T00:13:09.120 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:13:09.120 INFO:teuthology.orchestra.run.smithi123.stdout:oload 110 2020-01-16T00:13:09.120 INFO:teuthology.orchestra.run.smithi123.stdout:max_change 0.05 2020-01-16T00:13:09.120 INFO:teuthology.orchestra.run.smithi123.stdout:max_change_osds 4 2020-01-16T00:13:09.120 INFO:teuthology.orchestra.run.smithi123.stdout:average_utilization 0.0120 2020-01-16T00:13:09.120 INFO:teuthology.orchestra.run.smithi123.stdout:overload_utilization 0.0132 2020-01-16T00:13:09.121 INFO:teuthology.orchestra.run.smithi123.stderr:no change 2020-01-16T00:13:09.130 INFO:tasks.thrashosds.thrasher:Reweighting by: {'overage': '1000', 'type': 'test-reweight-by-utilization', 'max_change': '0.05'} 2020-01-16T00:13:09.131 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 1000 0.05 2020-01-16T00:13:09.165 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:13:09.160+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:13:09.200 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:09.211 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-01-16T00:13:09.236 INFO:tasks.rados.rados.0.smithi123.stdout:1038: finishing write tid 1 to smithi12313934-42 2020-01-16T00:13:09.237 INFO:tasks.rados.rados.0.smithi123.stdout:1038: finishing write tid 2 to smithi12313934-42 2020-01-16T00:13:09.237 INFO:tasks.rados.rados.0.smithi123.stdout:1038: finishing write tid 4 to smithi12313934-42 2020-01-16T00:13:09.237 INFO:tasks.rados.rados.0.smithi123.stdout:1038: finishing write tid 5 to smithi12313934-42 2020-01-16T00:13:09.237 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 42 v 99 (ObjNum 261 snap 0 seq_num 261) dirty exists 2020-01-16T00:13:09.237 INFO:tasks.rados.rados.0.smithi123.stdout:1038: left oid 42 (ObjNum 261 snap 0 seq_num 261) 2020-01-16T00:13:09.237 INFO:tasks.rados.rados.0.smithi123.stdout:1041: expect (ObjNum 238 snap 0 seq_num 238) 2020-01-16T00:13:09.265 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:09.262+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:09.364 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:09.366 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:09.360+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:09.373 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-01-16T00:13:09.389 INFO:tasks.rados.rados.0.smithi123.stdout:1042: finishing write tid 1 to smithi12313934-26 2020-01-16T00:13:09.389 INFO:tasks.rados.rados.0.smithi123.stdout:1040: expect (ObjNum 242 snap 0 seq_num 242) 2020-01-16T00:13:09.411 INFO:teuthology.orchestra.run.smithi123.stdout:moved 0 / 84 (0%) 2020-01-16T00:13:09.412 INFO:teuthology.orchestra.run.smithi123.stdout:avg 10.5 2020-01-16T00:13:09.412 INFO:teuthology.orchestra.run.smithi123.stdout:stddev 1.5 -> 1.5 (expected baseline 3.03109) 2020-01-16T00:13:09.412 INFO:teuthology.orchestra.run.smithi123.stdout:min osd.2 with 8 -> 8 pgs (0.761905 -> 0.761905 * mean) 2020-01-16T00:13:09.412 INFO:teuthology.orchestra.run.smithi123.stdout:max osd.0 with 12 -> 12 pgs (1.14286 -> 1.14286 * mean) 2020-01-16T00:13:09.412 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:13:09.412 INFO:teuthology.orchestra.run.smithi123.stdout:oload 1000 2020-01-16T00:13:09.412 INFO:teuthology.orchestra.run.smithi123.stdout:max_change 0.05 2020-01-16T00:13:09.413 INFO:teuthology.orchestra.run.smithi123.stdout:max_change_osds 4 2020-01-16T00:13:09.413 INFO:teuthology.orchestra.run.smithi123.stdout:average_utilization 0.0120 2020-01-16T00:13:09.413 INFO:teuthology.orchestra.run.smithi123.stdout:overload_utilization 0.1204 2020-01-16T00:13:09.414 INFO:teuthology.orchestra.run.smithi123.stderr:no change 2020-01-16T00:13:09.423 INFO:tasks.thrashosds.thrasher:Reweighting by: {'overage': '110', 'type': 'test-reweight-by-utilization', 'max_change': '1.0'} 2020-01-16T00:13:09.424 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 110 1.0 2020-01-16T00:13:09.471 INFO:tasks.rados.rados.0.smithi123.stdout:1042: finishing write tid 2 to smithi12313934-26 2020-01-16T00:13:09.471 INFO:tasks.rados.rados.0.smithi123.stdout:1042: finishing write tid 4 to smithi12313934-26 2020-01-16T00:13:09.471 INFO:tasks.rados.rados.0.smithi123.stdout:1042: finishing write tid 5 to smithi12313934-26 2020-01-16T00:13:09.471 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 26 v 203 (ObjNum 263 snap 0 seq_num 263) dirty exists 2020-01-16T00:13:09.471 INFO:tasks.rados.rados.0.smithi123.stdout:1042: left oid 26 (ObjNum 263 snap 0 seq_num 263) 2020-01-16T00:13:09.471 INFO:tasks.rados.rados.0.smithi123.stdout:1035: done (7 left) 2020-01-16T00:13:09.471 INFO:tasks.rados.rados.0.smithi123.stdout:1036: done (6 left) 2020-01-16T00:13:09.472 INFO:tasks.rados.rados.0.smithi123.stdout:1037: done (5 left) 2020-01-16T00:13:09.472 INFO:tasks.rados.rados.0.smithi123.stdout:1038: done (4 left) 2020-01-16T00:13:09.472 INFO:tasks.rados.rados.0.smithi123.stdout:1040: done (3 left) 2020-01-16T00:13:09.472 INFO:tasks.rados.rados.0.smithi123.stdout:1041: done (2 left) 2020-01-16T00:13:09.472 INFO:tasks.rados.rados.0.smithi123.stdout:1042: done (1 left) 2020-01-16T00:13:09.472 INFO:tasks.rados.rados.0.smithi123.stdout:1043: done (0 left) 2020-01-16T00:13:09.472 INFO:tasks.rados.rados.0.smithi123.stdout:1044: write oid 27 current snap is 0 2020-01-16T00:13:09.472 INFO:tasks.rados.rados.0.smithi123.stdout:1044: seq_num 265 ranges {453855=696987,1890787=548083} 2020-01-16T00:13:09.473 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:09.466+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:09.486 INFO:tasks.rados.rados.0.smithi123.stdout:1044: writing smithi12313934-27 from 453855 to 1150842 tid 1 2020-01-16T00:13:09.494 INFO:tasks.rados.rados.0.smithi123.stdout:1044: writing smithi12313934-27 from 1890787 to 2438870 tid 2 2020-01-16T00:13:09.494 INFO:tasks.rados.rados.0.smithi123.stdout:1045: write oid 33 current snap is 0 2020-01-16T00:13:09.494 INFO:tasks.rados.rados.0.smithi123.stdout:1045: seq_num 266 ranges {470662=772236,1832916=408094} 2020-01-16T00:13:09.501 INFO:tasks.rados.rados.0.smithi123.stdout:1045: writing smithi12313934-33 from 470662 to 1242898 tid 1 2020-01-16T00:13:09.507 INFO:tasks.rados.rados.0.smithi123.stdout:1045: writing smithi12313934-33 from 1832916 to 2241010 tid 2 2020-01-16T00:13:09.507 INFO:tasks.rados.rados.0.smithi123.stdout:1046: write (excl) oid 6 current snap is 0 2020-01-16T00:13:09.508 INFO:tasks.rados.rados.0.smithi123.stdout:1046: seq_num 267 ranges {487469=563838,1607751=551752} 2020-01-16T00:13:09.513 INFO:tasks.rados.rados.0.smithi123.stdout:1046: writing smithi12313934-6 from 487469 to 1051307 tid 1 2020-01-16T00:13:09.520 INFO:tasks.rados.rados.0.smithi123.stdout:1046: writing smithi12313934-6 from 1607751 to 2159503 tid 2 2020-01-16T00:13:09.520 INFO:tasks.rados.rados.0.smithi123.stdout:1047: read oid 48 snap -1 2020-01-16T00:13:09.521 INFO:tasks.rados.rados.0.smithi123.stdout:1047: expect (ObjNum 245 snap 0 seq_num 245) 2020-01-16T00:13:09.521 INFO:tasks.rados.rados.0.smithi123.stdout:1048: write oid 35 current snap is 0 2020-01-16T00:13:09.521 INFO:tasks.rados.rados.0.smithi123.stdout:1048: seq_num 268 ranges {504276=639087,1666233=695410} 2020-01-16T00:13:09.528 INFO:tasks.rados.rados.0.smithi123.stdout:1048: writing smithi12313934-35 from 504276 to 1143363 tid 1 2020-01-16T00:13:09.528 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:09.535 INFO:tasks.rados.rados.0.smithi123.stdout:1048: writing smithi12313934-35 from 1666233 to 2361643 tid 2 2020-01-16T00:13:09.536 INFO:tasks.rados.rados.0.smithi123.stdout:1049: write (excl) oid 10 current snap is 0 2020-01-16T00:13:09.536 INFO:tasks.rados.rados.0.smithi123.stdout:1049: seq_num 269 ranges {521083=714336,1724715=555421} 2020-01-16T00:13:09.539 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-01-16T00:13:09.542 INFO:tasks.rados.rados.0.smithi123.stdout:1049: writing smithi12313934-10 from 521083 to 1235419 tid 1 2020-01-16T00:13:09.550 INFO:tasks.rados.rados.0.smithi123.stdout:1049: writing smithi12313934-10 from 1724715 to 2280136 tid 2 2020-01-16T00:13:09.551 INFO:tasks.rados.rados.0.smithi123.stdout:1050: read oid 29 snap -1 2020-01-16T00:13:09.551 INFO:tasks.rados.rados.0.smithi123.stdout:1050: expect (ObjNum 257 snap 0 seq_num 257) 2020-01-16T00:13:09.551 INFO:tasks.rados.rados.0.smithi123.stdout:1051: write (excl) oid 11 current snap is 0 2020-01-16T00:13:09.551 INFO:tasks.rados.rados.0.smithi123.stdout:1051: seq_num 270 ranges {537890=789585,1783197=699079} 2020-01-16T00:13:09.557 INFO:tasks.rados.rados.0.smithi123.stdout:1051: writing smithi12313934-11 from 537890 to 1327475 tid 1 2020-01-16T00:13:09.564 INFO:tasks.rados.rados.0.smithi123.stdout:1051: writing smithi12313934-11 from 1783197 to 2482276 tid 2 2020-01-16T00:13:09.565 INFO:tasks.rados.rados.0.smithi123.stdout:1052: read oid 49 snap -1 2020-01-16T00:13:09.565 INFO:tasks.rados.rados.0.smithi123.stdout:1052: expect (ObjNum 196 snap 0 seq_num 196) 2020-01-16T00:13:09.565 INFO:tasks.rados.rados.0.smithi123.stdout:1053: write oid 32 current snap is 0 2020-01-16T00:13:09.565 INFO:tasks.rados.rados.0.smithi123.stdout:1053: seq_num 271 ranges {554697=464834,1725326=559090} 2020-01-16T00:13:09.570 INFO:tasks.rados.rados.0.smithi123.stdout:1053: writing smithi12313934-32 from 554697 to 1019531 tid 1 2020-01-16T00:13:09.579 INFO:tasks.rados.rados.0.smithi123.stdout:1053: writing smithi12313934-32 from 1725326 to 2284416 tid 2 2020-01-16T00:13:09.579 INFO:tasks.rados.rados.0.smithi123.stdout:1054: write (excl) oid 13 current snap is 0 2020-01-16T00:13:09.579 INFO:tasks.rados.rados.0.smithi123.stdout:1054: seq_num 272 ranges {571504=540083,1783808=702748} 2020-01-16T00:13:09.583 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:09.580+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:09.585 INFO:tasks.rados.rados.0.smithi123.stdout:1054: writing smithi12313934-13 from 571504 to 1111587 tid 1 2020-01-16T00:13:09.593 INFO:tasks.rados.rados.0.smithi123.stdout:1054: writing smithi12313934-13 from 1783808 to 2486556 tid 2 2020-01-16T00:13:09.594 INFO:tasks.rados.rados.0.smithi123.stdout:1055: write (excl) oid 16 current snap is 0 2020-01-16T00:13:09.594 INFO:tasks.rados.rados.0.smithi123.stdout:1055: seq_num 273 ranges {588311=615332,1842290=562759} 2020-01-16T00:13:09.602 INFO:tasks.rados.rados.0.smithi123.stdout:1055: writing smithi12313934-16 from 588311 to 1203643 tid 1 2020-01-16T00:13:09.609 INFO:tasks.rados.rados.0.smithi123.stdout:1055: writing smithi12313934-16 from 1842290 to 2405049 tid 2 2020-01-16T00:13:09.609 INFO:tasks.rados.rados.0.smithi123.stdout:1056: write (excl) oid 14 current snap is 0 2020-01-16T00:13:09.609 INFO:tasks.rados.rados.0.smithi123.stdout:1056: seq_num 274 ranges {605118=406934,1617125=706417} 2020-01-16T00:13:09.614 INFO:tasks.rados.rados.0.smithi123.stdout:1056: writing smithi12313934-14 from 605118 to 1012052 tid 1 2020-01-16T00:13:09.622 INFO:tasks.rados.rados.0.smithi123.stdout:1056: writing smithi12313934-14 from 1617125 to 2323542 tid 2 2020-01-16T00:13:09.622 INFO:tasks.rados.rados.0.smithi123.stdout:1057: write (excl) oid 31 current snap is 0 2020-01-16T00:13:09.623 INFO:tasks.rados.rados.0.smithi123.stdout:1057: seq_num 275 ranges {621925=482183,1559254=566428} 2020-01-16T00:13:09.628 INFO:tasks.rados.rados.0.smithi123.stdout:1057: writing smithi12313934-31 from 621925 to 1104108 tid 1 2020-01-16T00:13:09.635 INFO:tasks.rados.rados.0.smithi123.stdout:1057: writing smithi12313934-31 from 1559254 to 2125682 tid 2 2020-01-16T00:13:09.635 INFO:tasks.rados.rados.0.smithi123.stdout:1058: read oid 28 snap -1 2020-01-16T00:13:09.635 INFO:tasks.rados.rados.0.smithi123.stdout:1058: expect (ObjNum 242 snap 0 seq_num 242) 2020-01-16T00:13:09.636 INFO:tasks.rados.rados.0.smithi123.stdout:1059: read oid 45 snap -1 2020-01-16T00:13:09.636 INFO:tasks.rados.rados.0.smithi123.stdout:1059: expect (ObjNum 216 snap 0 seq_num 216) 2020-01-16T00:13:09.636 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:09.636 INFO:tasks.rados.rados.0.smithi123.stdout:1044: finishing write tid 1 to smithi12313934-27 2020-01-16T00:13:09.636 INFO:tasks.rados.rados.0.smithi123.stdout:1044: finishing write tid 2 to smithi12313934-27 2020-01-16T00:13:09.636 INFO:tasks.rados.rados.0.smithi123.stdout:1044: finishing write tid 4 to smithi12313934-27 2020-01-16T00:13:09.636 INFO:tasks.rados.rados.0.smithi123.stdout:1044: finishing write tid 5 to smithi12313934-27 2020-01-16T00:13:09.637 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 27 v 92 (ObjNum 265 snap 0 seq_num 265) dirty exists 2020-01-16T00:13:09.637 INFO:tasks.rados.rados.0.smithi123.stdout:1044: left oid 27 (ObjNum 265 snap 0 seq_num 265) 2020-01-16T00:13:09.637 INFO:tasks.rados.rados.0.smithi123.stdout:1045: finishing write tid 1 to smithi12313934-33 2020-01-16T00:13:09.637 INFO:tasks.rados.rados.0.smithi123.stdout:1045: finishing write tid 2 to smithi12313934-33 2020-01-16T00:13:09.637 INFO:tasks.rados.rados.0.smithi123.stdout:1046: finishing write tid 1 to smithi12313934-6 2020-01-16T00:13:09.637 INFO:tasks.rados.rados.0.smithi123.stdout:1044: done (15 left) 2020-01-16T00:13:09.637 INFO:tasks.rados.rados.0.smithi123.stdout:1060: write (excl) oid 20 current snap is 0 2020-01-16T00:13:09.637 INFO:tasks.rados.rados.0.smithi123.stdout:1060: seq_num 276 ranges {638732=557432,1617736=710086} 2020-01-16T00:13:09.642 INFO:tasks.rados.rados.0.smithi123.stdout:1060: writing smithi12313934-20 from 638732 to 1196164 tid 1 2020-01-16T00:13:09.648 INFO:tasks.rados.rados.0.smithi123.stdout:1060: writing smithi12313934-20 from 1617736 to 2327822 tid 2 2020-01-16T00:13:09.649 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:09.649 INFO:tasks.rados.rados.0.smithi123.stdout:1047: expect (ObjNum 245 snap 0 seq_num 245) 2020-01-16T00:13:09.649 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:09.661 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-01-16T00:13:09.703 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:13:09.700+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:13:09.706 INFO:teuthology.orchestra.run.smithi123.stdout:moved 0 / 84 (0%) 2020-01-16T00:13:09.706 INFO:teuthology.orchestra.run.smithi123.stdout:avg 10.5 2020-01-16T00:13:09.706 INFO:teuthology.orchestra.run.smithi123.stdout:stddev 1.5 -> 1.5 (expected baseline 3.03109) 2020-01-16T00:13:09.706 INFO:teuthology.orchestra.run.smithi123.stdout:min osd.2 with 8 -> 8 pgs (0.761905 -> 0.761905 * mean) 2020-01-16T00:13:09.706 INFO:teuthology.orchestra.run.smithi123.stdout:max osd.0 with 12 -> 12 pgs (1.14286 -> 1.14286 * mean) 2020-01-16T00:13:09.706 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:13:09.706 INFO:teuthology.orchestra.run.smithi123.stdout:oload 110 2020-01-16T00:13:09.707 INFO:teuthology.orchestra.run.smithi123.stdout:max_change 1 2020-01-16T00:13:09.707 INFO:teuthology.orchestra.run.smithi123.stdout:max_change_osds 4 2020-01-16T00:13:09.707 INFO:teuthology.orchestra.run.smithi123.stdout:average_utilization 0.0120 2020-01-16T00:13:09.707 INFO:teuthology.orchestra.run.smithi123.stdout:overload_utilization 0.0132 2020-01-16T00:13:09.708 INFO:teuthology.orchestra.run.smithi123.stderr:no change 2020-01-16T00:13:09.772 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:09.768+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:09.781 INFO:tasks.rados.rados.0.smithi123.stdout:1045: finishing write tid 4 to smithi12313934-33 2020-01-16T00:13:09.782 INFO:tasks.rados.rados.0.smithi123.stdout:1045: finishing write tid 5 to smithi12313934-33 2020-01-16T00:13:09.782 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 33 v 208 (ObjNum 266 snap 0 seq_num 266) dirty exists 2020-01-16T00:13:09.782 INFO:tasks.rados.rados.0.smithi123.stdout:1045: left oid 33 (ObjNum 266 snap 0 seq_num 266) 2020-01-16T00:13:09.782 INFO:tasks.rados.rados.0.smithi123.stdout:1046: finishing write tid 2 to smithi12313934-6 2020-01-16T00:13:09.782 INFO:tasks.rados.rados.0.smithi123.stdout:1046: finishing write tid 4 to smithi12313934-6 2020-01-16T00:13:09.782 INFO:tasks.rados.rados.0.smithi123.stdout:1046: finishing write tid 5 to smithi12313934-6 2020-01-16T00:13:09.782 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 6 v 76 (ObjNum 267 snap 0 seq_num 267) dirty exists 2020-01-16T00:13:09.783 INFO:tasks.rados.rados.0.smithi123.stdout:1046: left oid 6 (ObjNum 267 snap 0 seq_num 267) 2020-01-16T00:13:09.783 INFO:tasks.rados.rados.0.smithi123.stdout:1048: finishing write tid 1 to smithi12313934-35 2020-01-16T00:13:09.783 INFO:tasks.rados.rados.0.smithi123.stdout:1048: finishing write tid 2 to smithi12313934-35 2020-01-16T00:13:09.783 INFO:tasks.rados.rados.0.smithi123.stdout:1048: finishing write tid 4 to smithi12313934-35 2020-01-16T00:13:09.783 INFO:tasks.rados.rados.0.smithi123.stdout:1048: finishing write tid 5 to smithi12313934-35 2020-01-16T00:13:09.783 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 35 v 110 (ObjNum 268 snap 0 seq_num 268) dirty exists 2020-01-16T00:13:09.783 INFO:tasks.rados.rados.0.smithi123.stdout:1048: left oid 35 (ObjNum 268 snap 0 seq_num 268) 2020-01-16T00:13:09.783 INFO:tasks.rados.rados.0.smithi123.stdout:1049: finishing write tid 1 to smithi12313934-10 2020-01-16T00:13:09.784 INFO:tasks.rados.rados.0.smithi123.stdout:1050: expect (ObjNum 257 snap 0 seq_num 257) 2020-01-16T00:13:09.809 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:09.815 INFO:teuthology.orchestra.run.smithi123.stderr:nodeep-scrub is set 2020-01-16T00:13:09.818 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-01-16T00:13:09.903 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:13:09.900+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:13:09.936 INFO:tasks.rados.rados.0.smithi123.stdout:1049: finishing write tid 2 to smithi12313934-10 2020-01-16T00:13:09.936 INFO:tasks.rados.rados.0.smithi123.stdout:1051: finishing write tid 1 to smithi12313934-11 2020-01-16T00:13:09.937 INFO:tasks.rados.rados.0.smithi123.stdout:1051: finishing write tid 2 to smithi12313934-11 2020-01-16T00:13:09.937 INFO:tasks.rados.rados.0.smithi123.stdout:1051: finishing write tid 4 to smithi12313934-11 2020-01-16T00:13:09.937 INFO:tasks.rados.rados.0.smithi123.stdout:1051: finishing write tid 5 to smithi12313934-11 2020-01-16T00:13:09.937 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 11 v 79 (ObjNum 270 snap 0 seq_num 270) dirty exists 2020-01-16T00:13:09.937 INFO:tasks.rados.rados.0.smithi123.stdout:1051: left oid 11 (ObjNum 270 snap 0 seq_num 270) 2020-01-16T00:13:09.937 INFO:tasks.rados.rados.0.smithi123.stdout:1049: finishing write tid 4 to smithi12313934-10 2020-01-16T00:13:09.937 INFO:tasks.rados.rados.0.smithi123.stdout:1049: finishing write tid 5 to smithi12313934-10 2020-01-16T00:13:09.938 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 10 v 102 (ObjNum 269 snap 0 seq_num 269) dirty exists 2020-01-16T00:13:09.938 INFO:tasks.rados.rados.0.smithi123.stdout:1049: left oid 10 (ObjNum 269 snap 0 seq_num 269) 2020-01-16T00:13:09.938 INFO:tasks.rados.rados.0.smithi123.stdout:1053: finishing write tid 1 to smithi12313934-32 2020-01-16T00:13:09.938 INFO:tasks.rados.rados.0.smithi123.stdout:1052: expect (ObjNum 196 snap 0 seq_num 196) 2020-01-16T00:13:09.967 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:09.974 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:09.970+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:09.977 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-01-16T00:13:10.075 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:10.070+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:10.098 INFO:tasks.rados.rados.0.smithi123.stdout:1053: finishing write tid 2 to smithi12313934-32 2020-01-16T00:13:10.099 INFO:tasks.rados.rados.0.smithi123.stdout:1053: finishing write tid 4 to smithi12313934-32 2020-01-16T00:13:10.099 INFO:tasks.rados.rados.0.smithi123.stdout:1053: finishing write tid 5 to smithi12313934-32 2020-01-16T00:13:10.099 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 32 v 95 (ObjNum 271 snap 0 seq_num 271) dirty exists 2020-01-16T00:13:10.099 INFO:tasks.rados.rados.0.smithi123.stdout:1053: left oid 32 (ObjNum 271 snap 0 seq_num 271) 2020-01-16T00:13:10.099 INFO:tasks.rados.rados.0.smithi123.stdout:1054: finishing write tid 1 to smithi12313934-13 2020-01-16T00:13:10.099 INFO:tasks.rados.rados.0.smithi123.stdout:1054: finishing write tid 2 to smithi12313934-13 2020-01-16T00:13:10.100 INFO:tasks.rados.rados.0.smithi123.stdout:1055: finishing write tid 1 to smithi12313934-16 2020-01-16T00:13:10.100 INFO:tasks.rados.rados.0.smithi123.stdout:1054: finishing write tid 4 to smithi12313934-13 2020-01-16T00:13:10.100 INFO:tasks.rados.rados.0.smithi123.stdout:1045: done (15 left) 2020-01-16T00:13:10.100 INFO:tasks.rados.rados.0.smithi123.stdout:1046: done (14 left) 2020-01-16T00:13:10.100 INFO:tasks.rados.rados.0.smithi123.stdout:1047: done (13 left) 2020-01-16T00:13:10.100 INFO:tasks.rados.rados.0.smithi123.stdout:1048: done (12 left) 2020-01-16T00:13:10.100 INFO:tasks.rados.rados.0.smithi123.stdout:1049: done (11 left) 2020-01-16T00:13:10.100 INFO:tasks.rados.rados.0.smithi123.stdout:1050: done (10 left) 2020-01-16T00:13:10.101 INFO:tasks.rados.rados.0.smithi123.stdout:1051: done (9 left) 2020-01-16T00:13:10.101 INFO:tasks.rados.rados.0.smithi123.stdout:1052: done (8 left) 2020-01-16T00:13:10.101 INFO:tasks.rados.rados.0.smithi123.stdout:1053: done (7 left) 2020-01-16T00:13:10.101 INFO:tasks.rados.rados.0.smithi123.stdout:1061: write (excl) oid 15 current snap is 0 2020-01-16T00:13:10.101 INFO:tasks.rados.rados.0.smithi123.stdout:1061: seq_num 277 ranges {655539=632681,2076218=570097} 2020-01-16T00:13:10.111 INFO:tasks.rados.rados.0.smithi123.stdout:1061: writing smithi12313934-15 from 655539 to 1288220 tid 1 2020-01-16T00:13:10.120 INFO:tasks.rados.rados.0.smithi123.stdout:1061: writing smithi12313934-15 from 2076218 to 2646315 tid 2 2020-01-16T00:13:10.120 INFO:tasks.rados.rados.0.smithi123.stdout:1062: write (excl) oid 48 current snap is 0 2020-01-16T00:13:10.120 INFO:tasks.rados.rados.0.smithi123.stdout:1062: seq_num 278 ranges {672346=707930,2134700=537646} 2020-01-16T00:13:10.128 INFO:tasks.rados.rados.0.smithi123.stdout:1062: writing smithi12313934-48 from 672346 to 1380276 tid 1 2020-01-16T00:13:10.136 INFO:tasks.rados.rados.0.smithi123.stdout:1062: writing smithi12313934-48 from 2134700 to 2672346 tid 2 2020-01-16T00:13:10.136 INFO:tasks.rados.rados.0.smithi123.stdout:1063: read oid 37 snap -1 2020-01-16T00:13:10.136 INFO:tasks.rados.rados.0.smithi123.stdout:1063: expect deleted 2020-01-16T00:13:10.136 INFO:tasks.rados.rados.0.smithi123.stdout:1064: read oid 25 snap -1 2020-01-16T00:13:10.136 INFO:tasks.rados.rados.0.smithi123.stdout:1064: expect (ObjNum 254 snap 0 seq_num 254) 2020-01-16T00:13:10.136 INFO:tasks.rados.rados.0.smithi123.stdout:1065: write oid 49 current snap is 0 2020-01-16T00:13:10.137 INFO:tasks.rados.rados.0.smithi123.stdout:1054: finishing write tid 5 to smithi12313934-13 2020-01-16T00:13:10.137 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 62 (ObjNum 272 snap 0 seq_num 272) dirty exists 2020-01-16T00:13:10.137 INFO:tasks.rados.rados.0.smithi123.stdout:1054: left oid 13 (ObjNum 272 snap 0 seq_num 272) 2020-01-16T00:13:10.137 INFO:tasks.rados.rados.0.smithi123.stdout:1065: seq_num 279 ranges {689153=783179,2076829=457413} 2020-01-16T00:13:10.139 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:10.144 INFO:tasks.rados.rados.0.smithi123.stdout:1065: writing smithi12313934-49 from 689153 to 1472332 tid 1 2020-01-16T00:13:10.151 INFO:tasks.rados.rados.0.smithi123.stdout:1065: writing smithi12313934-49 from 2076829 to 2534242 tid 2 2020-01-16T00:13:10.151 INFO:tasks.rados.rados.0.smithi123.stdout:1054: done (11 left) 2020-01-16T00:13:10.151 INFO:tasks.rados.rados.0.smithi123.stdout:1066: write (excl) oid 29 current snap is 0 2020-01-16T00:13:10.151 INFO:tasks.rados.rados.0.smithi123.stdout:1066: seq_num 280 ranges {705960=458428,1735311=717424} 2020-01-16T00:13:10.154 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-01-16T00:13:10.158 INFO:tasks.rados.rados.0.smithi123.stdout:1066: writing smithi12313934-29 from 705960 to 1164388 tid 1 2020-01-16T00:13:10.165 INFO:tasks.rados.rados.0.smithi123.stdout:1066: writing smithi12313934-29 from 1735311 to 2452735 tid 2 2020-01-16T00:13:10.165 INFO:tasks.rados.rados.0.smithi123.stdout:1067: write (excl) oid 2 current snap is 0 2020-01-16T00:13:10.165 INFO:tasks.rados.rados.0.smithi123.stdout:1067: seq_num 281 ranges {722767=533677,1793793=461082} 2020-01-16T00:13:10.173 INFO:tasks.rados.rados.0.smithi123.stdout:1067: writing smithi12313934-2 from 722767 to 1256444 tid 1 2020-01-16T00:13:10.176 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:10.170+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:10.178 INFO:tasks.rados.rados.0.smithi123.stdout:1067: writing smithi12313934-2 from 1793793 to 2254875 tid 2 2020-01-16T00:13:10.179 INFO:tasks.rados.rados.0.smithi123.stdout:1068: write oid 42 current snap is 0 2020-01-16T00:13:10.179 INFO:tasks.rados.rados.0.smithi123.stdout:1068: seq_num 282 ranges {739574=725279,1968628=721093} 2020-01-16T00:13:10.187 INFO:tasks.rados.rados.0.smithi123.stdout:1068: writing smithi12313934-42 from 739574 to 1464853 tid 1 2020-01-16T00:13:10.195 INFO:tasks.rados.rados.0.smithi123.stdout:1068: writing smithi12313934-42 from 1968628 to 2689721 tid 2 2020-01-16T00:13:10.196 INFO:tasks.rados.rados.0.smithi123.stdout:1069: read oid 50 snap -1 2020-01-16T00:13:10.196 INFO:tasks.rados.rados.0.smithi123.stdout:1069: expect (ObjNum 177 snap 0 seq_num 177) 2020-01-16T00:13:10.196 INFO:tasks.rados.rados.0.smithi123.stdout:1070: read oid 26 snap -1 2020-01-16T00:13:10.196 INFO:tasks.rados.rados.0.smithi123.stdout:1070: expect (ObjNum 263 snap 0 seq_num 263) 2020-01-16T00:13:10.196 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:10.196 INFO:tasks.rados.rados.0.smithi123.stdout:1055: finishing write tid 2 to smithi12313934-16 2020-01-16T00:13:10.197 INFO:tasks.rados.rados.0.smithi123.stdout:1055: finishing write tid 4 to smithi12313934-16 2020-01-16T00:13:10.197 INFO:tasks.rados.rados.0.smithi123.stdout:1055: finishing write tid 5 to smithi12313934-16 2020-01-16T00:13:10.197 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 16 v 153 (ObjNum 273 snap 0 seq_num 273) dirty exists 2020-01-16T00:13:10.197 INFO:tasks.rados.rados.0.smithi123.stdout:1055: left oid 16 (ObjNum 273 snap 0 seq_num 273) 2020-01-16T00:13:10.197 INFO:tasks.rados.rados.0.smithi123.stdout:1056: finishing write tid 1 to smithi12313934-14 2020-01-16T00:13:10.197 INFO:tasks.rados.rados.0.smithi123.stdout:1056: finishing write tid 2 to smithi12313934-14 2020-01-16T00:13:10.198 INFO:tasks.rados.rados.0.smithi123.stdout:1056: finishing write tid 4 to smithi12313934-14 2020-01-16T00:13:10.198 INFO:tasks.rados.rados.0.smithi123.stdout:1056: finishing write tid 5 to smithi12313934-14 2020-01-16T00:13:10.199 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 14 v 144 (ObjNum 274 snap 0 seq_num 274) dirty exists 2020-01-16T00:13:10.199 INFO:tasks.rados.rados.0.smithi123.stdout:1056: left oid 14 (ObjNum 274 snap 0 seq_num 274) 2020-01-16T00:13:10.199 INFO:tasks.rados.rados.0.smithi123.stdout:1057: finishing write tid 1 to smithi12313934-31 2020-01-16T00:13:10.199 INFO:tasks.rados.rados.0.smithi123.stdout:1055: done (15 left) 2020-01-16T00:13:10.199 INFO:tasks.rados.rados.0.smithi123.stdout:1056: done (14 left) 2020-01-16T00:13:10.200 INFO:tasks.rados.rados.0.smithi123.stdout:1071: read oid 4 snap -1 2020-01-16T00:13:10.200 INFO:tasks.rados.rados.0.smithi123.stdout:1071: expect (ObjNum 188 snap 0 seq_num 188) 2020-01-16T00:13:10.200 INFO:tasks.rados.rados.0.smithi123.stdout:1072: write (excl) oid 6 current snap is 0 2020-01-16T00:13:10.200 INFO:tasks.rados.rados.0.smithi123.stdout:1072: seq_num 283 ranges {756381=400528,1910757=464751} 2020-01-16T00:13:10.202 INFO:tasks.rados.rados.0.smithi123.stdout:1072: writing smithi12313934-6 from 756381 to 1156909 tid 1 2020-01-16T00:13:10.210 INFO:tasks.rados.rados.0.smithi123.stdout:1072: writing smithi12313934-6 from 1910757 to 2375508 tid 2 2020-01-16T00:13:10.210 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:10.210 INFO:tasks.rados.rados.0.smithi123.stdout:1057: finishing write tid 2 to smithi12313934-31 2020-01-16T00:13:10.210 INFO:tasks.rados.rados.0.smithi123.stdout:1057: finishing write tid 4 to smithi12313934-31 2020-01-16T00:13:10.210 INFO:tasks.rados.rados.0.smithi123.stdout:1057: finishing write tid 5 to smithi12313934-31 2020-01-16T00:13:10.210 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 31 v 109 (ObjNum 275 snap 0 seq_num 275) dirty exists 2020-01-16T00:13:10.211 INFO:tasks.rados.rados.0.smithi123.stdout:1057: left oid 31 (ObjNum 275 snap 0 seq_num 275) 2020-01-16T00:13:10.211 INFO:tasks.rados.rados.0.smithi123.stdout:1058: expect (ObjNum 242 snap 0 seq_num 242) 2020-01-16T00:13:10.253 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:10.262 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-01-16T00:13:10.278 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-01-16T00:13:10.292 INFO:tasks.rados.rados.0.smithi123.stdout:1060: finishing write tid 1 to smithi12313934-20 2020-01-16T00:13:10.292 INFO:tasks.rados.rados.0.smithi123.stdout:1060: finishing write tid 2 to smithi12313934-20 2020-01-16T00:13:10.292 INFO:tasks.rados.rados.0.smithi123.stdout:1060: finishing write tid 4 to smithi12313934-20 2020-01-16T00:13:10.292 INFO:tasks.rados.rados.0.smithi123.stdout:1060: finishing write tid 5 to smithi12313934-20 2020-01-16T00:13:10.292 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 20 v 156 (ObjNum 276 snap 0 seq_num 276) dirty exists 2020-01-16T00:13:10.293 INFO:tasks.rados.rados.0.smithi123.stdout:1060: left oid 20 (ObjNum 276 snap 0 seq_num 276) 2020-01-16T00:13:10.293 INFO:tasks.rados.rados.0.smithi123.stdout:1059: expect (ObjNum 216 snap 0 seq_num 216) 2020-01-16T00:13:10.310 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:10.304+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:10.365 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:10.375 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-01-16T00:13:10.409 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:13:10.415 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:10.412+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:10.426 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:13:10.445 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:13:10.466 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:13:10.477 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:10.478 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:10.478 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:13:10.478 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:10.478 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:10.474+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:10.485 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-01-16T00:13:10.489 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:13:10.512 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5: {} 2020-01-16T00:13:10.534 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:13:10.552 INFO:tasks.rados.rados.0.smithi123.stdout:1061: finishing write tid 1 to smithi12313934-15 2020-01-16T00:13:10.552 INFO:tasks.rados.rados.0.smithi123.stdout:1061: finishing write tid 2 to smithi12313934-15 2020-01-16T00:13:10.552 INFO:tasks.rados.rados.0.smithi123.stdout:1061: finishing write tid 4 to smithi12313934-15 2020-01-16T00:13:10.552 INFO:tasks.rados.rados.0.smithi123.stdout:1061: finishing write tid 5 to smithi12313934-15 2020-01-16T00:13:10.552 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 15 v 112 (ObjNum 277 snap 0 seq_num 277) dirty exists 2020-01-16T00:13:10.552 INFO:tasks.rados.rados.0.smithi123.stdout:1061: left oid 15 (ObjNum 277 snap 0 seq_num 277) 2020-01-16T00:13:10.553 INFO:tasks.rados.rados.0.smithi123.stdout:1062: finishing write tid 1 to smithi12313934-48 2020-01-16T00:13:10.553 INFO:tasks.rados.rados.0.smithi123.stdout:1062: finishing write tid 2 to smithi12313934-48 2020-01-16T00:13:10.553 INFO:tasks.rados.rados.0.smithi123.stdout:1057: done (15 left) 2020-01-16T00:13:10.553 INFO:tasks.rados.rados.0.smithi123.stdout:1058: done (14 left) 2020-01-16T00:13:10.553 INFO:tasks.rados.rados.0.smithi123.stdout:1059: done (13 left) 2020-01-16T00:13:10.553 INFO:tasks.rados.rados.0.smithi123.stdout:1060: done (12 left) 2020-01-16T00:13:10.553 INFO:tasks.rados.rados.0.smithi123.stdout:1061: done (11 left) 2020-01-16T00:13:10.554 INFO:tasks.rados.rados.0.smithi123.stdout:1063: done (10 left) 2020-01-16T00:13:10.554 INFO:tasks.rados.rados.0.smithi123.stdout:1073: write oid 16 current snap is 0 2020-01-16T00:13:10.554 INFO:tasks.rados.rados.0.smithi123.stdout:1064: expect (ObjNum 254 snap 0 seq_num 254) 2020-01-16T00:13:10.556 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:13:10.558 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-01-16T00:13:10.558 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-01-16T00:13:10.558 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-01-16T00:13:10.558 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-01-16T00:13:10.559 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-01-16T00:13:10.559 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: osd_enable_op_tracker = 'true' 2020-01-16T00:13:10.559 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-01-16T00:13:10.559 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-01-16T00:13:10.585 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:10.585 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:10.586 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:13:10.586 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:13:10.586 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:10.586 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:10.582+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:10.594 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-01-16T00:13:10.671 INFO:tasks.rados.rados.0.smithi123.stdout:1065: finishing write tid 1 to smithi12313934-49 2020-01-16T00:13:10.672 INFO:tasks.rados.rados.0.smithi123.stdout:1062: finishing write tid 4 to smithi12313934-48 2020-01-16T00:13:10.672 INFO:tasks.rados.rados.0.smithi123.stdout:1065: finishing write tid 2 to smithi12313934-49 2020-01-16T00:13:10.672 INFO:tasks.rados.rados.0.smithi123.stdout:1062: finishing write tid 5 to smithi12313934-48 2020-01-16T00:13:10.672 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 48 v 103 (ObjNum 278 snap 0 seq_num 278) dirty exists 2020-01-16T00:13:10.672 INFO:tasks.rados.rados.0.smithi123.stdout:1062: left oid 48 (ObjNum 278 snap 0 seq_num 278) 2020-01-16T00:13:10.672 INFO:tasks.rados.rados.0.smithi123.stdout:1065: finishing write tid 4 to smithi12313934-49 2020-01-16T00:13:10.672 INFO:tasks.rados.rados.0.smithi123.stdout:1065: finishing write tid 5 to smithi12313934-49 2020-01-16T00:13:10.672 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 49 v 105 (ObjNum 279 snap 0 seq_num 279) dirty exists 2020-01-16T00:13:10.673 INFO:tasks.rados.rados.0.smithi123.stdout:1065: left oid 49 (ObjNum 279 snap 0 seq_num 279) 2020-01-16T00:13:10.673 INFO:tasks.rados.rados.0.smithi123.stdout:1066: finishing write tid 1 to smithi12313934-29 2020-01-16T00:13:10.673 INFO:tasks.rados.rados.0.smithi123.stdout:1066: finishing write tid 2 to smithi12313934-29 2020-01-16T00:13:10.673 INFO:tasks.rados.rados.0.smithi123.stdout:1067: finishing write tid 1 to smithi12313934-2 2020-01-16T00:13:10.673 INFO:tasks.rados.rados.0.smithi123.stdout:1066: finishing write tid 4 to smithi12313934-29 2020-01-16T00:13:10.673 INFO:tasks.rados.rados.0.smithi123.stdout:1066: finishing write tid 5 to smithi12313934-29 2020-01-16T00:13:10.673 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 29 v 103 (ObjNum 280 snap 0 seq_num 280) dirty exists 2020-01-16T00:13:10.674 INFO:tasks.rados.rados.0.smithi123.stdout:1066: left oid 29 (ObjNum 280 snap 0 seq_num 280) 2020-01-16T00:13:10.674 INFO:tasks.rados.rados.0.smithi123.stdout:1067: finishing write tid 2 to smithi12313934-2 2020-01-16T00:13:10.674 INFO:tasks.rados.rados.0.smithi123.stdout:1067: finishing write tid 4 to smithi12313934-2 2020-01-16T00:13:10.674 INFO:tasks.rados.rados.0.smithi123.stdout:1067: finishing write tid 5 to smithi12313934-2 2020-01-16T00:13:10.674 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 2 v 150 (ObjNum 281 snap 0 seq_num 281) dirty exists 2020-01-16T00:13:10.674 INFO:tasks.rados.rados.0.smithi123.stdout:1067: left oid 2 (ObjNum 281 snap 0 seq_num 281) 2020-01-16T00:13:10.674 INFO:tasks.rados.rados.0.smithi123.stdout:1068: finishing write tid 1 to smithi12313934-42 2020-01-16T00:13:10.675 INFO:tasks.rados.rados.0.smithi123.stdout:1073: seq_num 284 ranges {773188=475777,1969239=724762} 2020-01-16T00:13:10.680 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:10.676+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:10.684 INFO:tasks.rados.rados.0.smithi123.stdout:1073: writing smithi12313934-16 from 773188 to 1248965 tid 1 2020-01-16T00:13:10.693 INFO:tasks.rados.rados.0.smithi123.stdout:1073: writing smithi12313934-16 from 1969239 to 2694001 tid 2 2020-01-16T00:13:10.693 INFO:tasks.rados.rados.0.smithi123.stdout:1062: done (10 left) 2020-01-16T00:13:10.693 INFO:tasks.rados.rados.0.smithi123.stdout:1064: done (9 left) 2020-01-16T00:13:10.693 INFO:tasks.rados.rados.0.smithi123.stdout:1065: done (8 left) 2020-01-16T00:13:10.693 INFO:tasks.rados.rados.0.smithi123.stdout:1066: done (7 left) 2020-01-16T00:13:10.693 INFO:tasks.rados.rados.0.smithi123.stdout:1067: done (6 left) 2020-01-16T00:13:10.693 INFO:tasks.rados.rados.0.smithi123.stdout:1074: write (excl) oid 37 current snap is 0 2020-01-16T00:13:10.694 INFO:tasks.rados.rados.0.smithi123.stdout:1074: seq_num 285 ranges {789995=551026,2027721=468420} 2020-01-16T00:13:10.701 INFO:tasks.rados.rados.0.smithi123.stdout:1074: writing smithi12313934-37 from 789995 to 1341021 tid 1 2020-01-16T00:13:10.707 INFO:tasks.rados.rados.0.smithi123.stdout:1074: writing smithi12313934-37 from 2027721 to 2496141 tid 2 2020-01-16T00:13:10.707 INFO:tasks.rados.rados.0.smithi123.stdout:1075: write (excl) oid 31 current snap is 0 2020-01-16T00:13:10.707 INFO:tasks.rados.rados.0.smithi123.stdout:1075: seq_num 286 ranges {406802=626275,1686203=728431} 2020-01-16T00:13:10.713 INFO:tasks.rados.rados.0.smithi123.stdout:1075: writing smithi12313934-31 from 406802 to 1033077 tid 1 2020-01-16T00:13:10.721 INFO:tasks.rados.rados.0.smithi123.stdout:1075: writing smithi12313934-31 from 1686203 to 2414634 tid 2 2020-01-16T00:13:10.721 INFO:tasks.rados.rados.0.smithi123.stdout:1076: read oid 25 snap -1 2020-01-16T00:13:10.721 INFO:tasks.rados.rados.0.smithi123.stdout:1076: expect (ObjNum 254 snap 0 seq_num 254) 2020-01-16T00:13:10.721 INFO:tasks.rados.rados.0.smithi123.stdout:1077: write oid 43 current snap is 0 2020-01-16T00:13:10.721 INFO:tasks.rados.rados.0.smithi123.stdout:1077: seq_num 287 ranges {423609=701524,1628332=472089,2648987=174622} 2020-01-16T00:13:10.727 INFO:tasks.rados.rados.0.smithi123.stdout:1077: writing smithi12313934-43 from 423609 to 1125133 tid 1 2020-01-16T00:13:10.733 INFO:tasks.rados.rados.0.smithi123.stdout:1077: writing smithi12313934-43 from 1628332 to 2100421 tid 2 2020-01-16T00:13:10.737 INFO:tasks.rados.rados.0.smithi123.stdout:1077: writing smithi12313934-43 from 2648987 to 2823609 tid 3 2020-01-16T00:13:10.737 INFO:tasks.rados.rados.0.smithi123.stdout:1078: write (excl) oid 27 current snap is 0 2020-01-16T00:13:10.737 INFO:tasks.rados.rados.0.smithi123.stdout:1078: seq_num 288 ranges {440416=776773,1686814=732100} 2020-01-16T00:13:10.743 INFO:tasks.rados.rados.0.smithi123.stdout:1078: writing smithi12313934-27 from 440416 to 1217189 tid 1 2020-01-16T00:13:10.747 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:10.747 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:13:10.748 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:13:10.748 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:13:10.748 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.748 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:939 3.4 3:2bdd9e88:::smithi12313934-4:head [write 1681102~474010] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:10.748 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.908392+0000", 2020-01-16T00:13:10.748 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 32.836044864000002, 2020-01-16T00:13:10.748 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019294697, 2020-01-16T00:13:10.749 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:10.749 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:10.749 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:10.749 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:10.749 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:10.749 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 939 2020-01-16T00:13:10.749 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.749 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:10.750 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.750 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:10.750 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908392+0000", 2020-01-16T00:13:10.750 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:10.750 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.750 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.750 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:10.751 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908392+0000", 2020-01-16T00:13:10.751 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:10.751 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.751 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.751 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:10.751 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908390+0000", 2020-01-16T00:13:10.751 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015440399999999999 2020-01-16T00:13:10.751 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.752 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.752 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:10.752 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908545+0000", 2020-01-16T00:13:10.752 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3926e-05 2020-01-16T00:13:10.752 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.752 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.752 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:10.752 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908559+0000", 2020-01-16T00:13:10.753 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2200000000000003e-06 2020-01-16T00:13:10.753 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.753 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.753 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:10.753 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908566+0000", 2020-01-16T00:13:10.753 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020594799999999998 2020-01-16T00:13:10.753 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.754 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.754 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.754 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910625+0000", 2020-01-16T00:13:10.754 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118672 2020-01-16T00:13:10.754 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.754 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.754 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:10.755 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910744+0000", 2020-01-16T00:13:10.755 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031940969999999999 2020-01-16T00:13:10.755 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.755 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.755 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.755 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.913938+0000", 2020-01-16T00:13:10.755 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000309621 2020-01-16T00:13:10.755 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.756 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.756 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:10.756 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.914248+0000", 2020-01-16T00:13:10.756 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.008818579 2020-01-16T00:13:10.756 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.756 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.756 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.756 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.923066+0000", 2020-01-16T00:13:10.757 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000108989 2020-01-16T00:13:10.757 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.757 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.757 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:10.757 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.923175+0000", 2020-01-16T00:13:10.757 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019128299999999999 2020-01-16T00:13:10.758 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.758 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.758 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:13:10.758 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.923367+0000", 2020-01-16T00:13:10.758 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0032083760000000002 2020-01-16T00:13:10.758 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.758 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.758 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:10.759 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926575+0000", 2020-01-16T00:13:10.759 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2475999999999999e-05 2020-01-16T00:13:10.759 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.759 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.759 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:10.759 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926607+0000", 2020-01-16T00:13:10.759 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00099399800000000006 2020-01-16T00:13:10.759 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.760 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.760 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:10.760 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927601+0000", 2020-01-16T00:13:10.760 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0958999999999999e-05 2020-01-16T00:13:10.760 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.760 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.760 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:10.760 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927642+0000", 2020-01-16T00:13:10.761 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4747999999999999e-05 2020-01-16T00:13:10.761 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.761 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.761 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:10.761 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927687+0000", 2020-01-16T00:13:10.761 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019294697 2020-01-16T00:13:10.761 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:10.762 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:10.762 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:10.762 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.762 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.762 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:940 3.4 3:2bdd9e88:::smithi12313934-4:head [write 2687912~589420] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:10.763 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.915144+0000", 2020-01-16T00:13:10.763 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 32.829293495999998, 2020-01-16T00:13:10.763 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016092538999999999, 2020-01-16T00:13:10.763 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:10.763 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:10.763 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:10.763 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:10.764 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:10.764 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 940 2020-01-16T00:13:10.764 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.764 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:10.764 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.764 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:10.764 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915144+0000", 2020-01-16T00:13:10.765 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:10.765 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.765 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.765 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:10.765 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915144+0000", 2020-01-16T00:13:10.765 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:10.765 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.766 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.766 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:10.766 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915141+0000", 2020-01-16T00:13:10.766 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016366300000000001 2020-01-16T00:13:10.766 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.766 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.766 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:10.766 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915305+0000", 2020-01-16T00:13:10.767 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4847e-05 2020-01-16T00:13:10.767 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.767 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.767 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:10.767 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915320+0000", 2020-01-16T00:13:10.767 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2599999999999999e-06 2020-01-16T00:13:10.768 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.768 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.768 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:10.768 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915327+0000", 2020-01-16T00:13:10.768 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4282000000000005e-05 2020-01-16T00:13:10.768 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.768 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.768 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.769 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915401+0000", 2020-01-16T00:13:10.769 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4554000000000006e-05 2020-01-16T00:13:10.769 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.769 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.769 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:10.769 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915466+0000", 2020-01-16T00:13:10.769 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0090821900000000004 2020-01-16T00:13:10.770 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.770 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.770 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.770 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.924548+0000", 2020-01-16T00:13:10.770 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000114696 2020-01-16T00:13:10.770 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.770 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.771 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:10.771 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.924663+0000", 2020-01-16T00:13:10.771 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000168651 2020-01-16T00:13:10.771 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.771 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.771 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:13:10.771 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.924832+0000", 2020-01-16T00:13:10.772 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018253430000000001 2020-01-16T00:13:10.773 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.773 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.773 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:10.773 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926657+0000", 2020-01-16T00:13:10.773 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9957999999999999e-05 2020-01-16T00:13:10.773 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.773 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.774 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:10.774 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926697+0000", 2020-01-16T00:13:10.774 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0044685250000000001 2020-01-16T00:13:10.774 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.774 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.774 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:10.774 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931165+0000", 2020-01-16T00:13:10.774 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9211000000000001e-05 2020-01-16T00:13:10.775 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.775 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.775 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:10.775 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931205+0000", 2020-01-16T00:13:10.775 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1699999999999998e-05 2020-01-16T00:13:10.775 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.775 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.776 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:10.776 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931236+0000", 2020-01-16T00:13:10.776 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016092538999999999 2020-01-16T00:13:10.776 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:10.776 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:10.776 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:10.776 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.777 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.777 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:941 3.4 3:2bdd9e88:::smithi12313934-4:head [setxattr _header (55) in=7b,truncate 3277332] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:10.777 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.915341+0000", 2020-01-16T00:13:10.777 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 32.829096677999999, 2020-01-16T00:13:10.777 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016603769000000001, 2020-01-16T00:13:10.777 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:10.777 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:10.777 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:10.778 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:10.778 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:10.778 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 941 2020-01-16T00:13:10.778 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.778 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:10.778 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.778 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:10.778 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915341+0000", 2020-01-16T00:13:10.779 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:10.779 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.779 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.779 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:10.779 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915341+0000", 2020-01-16T00:13:10.779 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:10.779 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.780 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.780 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:10.780 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915340+0000", 2020-01-16T00:13:10.780 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.896e-06 2020-01-16T00:13:10.780 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.780 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.780 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:10.780 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915346+0000", 2020-01-16T00:13:10.781 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2730000000000002e-06 2020-01-16T00:13:10.781 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.781 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.781 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:10.781 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915354+0000", 2020-01-16T00:13:10.781 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0616e-05 2020-01-16T00:13:10.781 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.782 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.782 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:10.782 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915374+0000", 2020-01-16T00:13:10.782 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000133487 2020-01-16T00:13:10.782 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.782 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.782 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.783 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915508+0000", 2020-01-16T00:13:10.783 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8971000000000003e-05 2020-01-16T00:13:10.783 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.783 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.783 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:10.783 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915557+0000", 2020-01-16T00:13:10.783 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010138787999999999 2020-01-16T00:13:10.784 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.784 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.784 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.784 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.925696+0000", 2020-01-16T00:13:10.784 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012825400000000001 2020-01-16T00:13:10.784 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.784 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.785 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:10.785 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.925824+0000", 2020-01-16T00:13:10.785 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017706500000000001 2020-01-16T00:13:10.785 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.785 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.785 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:13:10.785 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926001+0000", 2020-01-16T00:13:10.786 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00079988900000000003 2020-01-16T00:13:10.786 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.786 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.786 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:10.786 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926801+0000", 2020-01-16T00:13:10.786 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119324 2020-01-16T00:13:10.786 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.787 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.787 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:10.787 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926920+0000", 2020-01-16T00:13:10.787 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048879910000000004 2020-01-16T00:13:10.787 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.787 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.787 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:10.788 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931808+0000", 2020-01-16T00:13:10.788 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6999999999999997e-05 2020-01-16T00:13:10.788 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.788 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.788 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:10.788 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931855+0000", 2020-01-16T00:13:10.788 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9092999999999996e-05 2020-01-16T00:13:10.789 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.789 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.789 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:10.789 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931944+0000", 2020-01-16T00:13:10.789 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016603769000000001 2020-01-16T00:13:10.789 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:10.789 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:10.789 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:10.790 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.790 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.790 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:942 3.4 3:2bdd9e88:::smithi12313934-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:13:10.790 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.915389+0000", 2020-01-16T00:13:10.790 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 32.829048311999998, 2020-01-16T00:13:10.790 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016820563, 2020-01-16T00:13:10.790 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:10.791 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:10.791 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:10.791 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:10.791 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:10.791 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 942 2020-01-16T00:13:10.791 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.791 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:10.791 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.792 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:10.792 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915389+0000", 2020-01-16T00:13:10.792 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:10.792 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.792 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.792 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:10.792 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915389+0000", 2020-01-16T00:13:10.792 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:10.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.793 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.793 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:10.793 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915388+0000", 2020-01-16T00:13:10.793 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.683e-06 2020-01-16T00:13:10.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.793 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.794 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:10.794 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915391+0000", 2020-01-16T00:13:10.794 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4839999999999998e-06 2020-01-16T00:13:10.794 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.794 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.794 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:10.795 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915399+0000", 2020-01-16T00:13:10.795 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2259999999999999e-06 2020-01-16T00:13:10.795 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.795 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.795 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:10.795 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915403+0000", 2020-01-16T00:13:10.795 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00031301199999999999 2020-01-16T00:13:10.795 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.796 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.796 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.796 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915716+0000", 2020-01-16T00:13:10.796 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2079999999999998e-05 2020-01-16T00:13:10.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.796 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.796 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:10.796 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915748+0000", 2020-01-16T00:13:10.797 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010547076000000001 2020-01-16T00:13:10.797 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.797 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.797 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.797 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926295+0000", 2020-01-16T00:13:10.797 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8040000000000005e-05 2020-01-16T00:13:10.797 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.797 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.798 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:10.798 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926373+0000", 2020-01-16T00:13:10.798 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056061990000000001 2020-01-16T00:13:10.798 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.798 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.798 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.798 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931979+0000", 2020-01-16T00:13:10.799 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011540399999999999 2020-01-16T00:13:10.799 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.799 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.799 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:10.799 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932095+0000", 2020-01-16T00:13:10.799 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011482899999999999 2020-01-16T00:13:10.799 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.799 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.800 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:10.800 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932210+0000", 2020-01-16T00:13:10.800 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016820563 2020-01-16T00:13:10.800 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:10.800 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:10.800 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:10.800 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.800 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.801 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:957 3.0 3:00c146d5:::smithi12313934-18:head [write 510946~632952] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:10.801 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.033290+0000", 2020-01-16T00:13:10.801 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 32.711147474999997, 2020-01-16T00:13:10.801 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016764102999999999, 2020-01-16T00:13:10.801 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:10.801 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:10.801 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:10.802 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:10.802 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:10.802 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 957 2020-01-16T00:13:10.802 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.802 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:10.802 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.802 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:10.802 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033290+0000", 2020-01-16T00:13:10.803 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:10.803 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.803 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.803 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:10.803 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033290+0000", 2020-01-16T00:13:10.803 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999952 2020-01-16T00:13:10.803 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.804 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.804 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:10.804 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033285+0000", 2020-01-16T00:13:10.804 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000405917 2020-01-16T00:13:10.804 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.804 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.804 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:10.805 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033691+0000", 2020-01-16T00:13:10.805 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6585999999999997e-05 2020-01-16T00:13:10.805 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.805 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.805 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:10.806 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033738+0000", 2020-01-16T00:13:10.806 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8374000000000001e-05 2020-01-16T00:13:10.806 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.806 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.806 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:10.806 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033756+0000", 2020-01-16T00:13:10.806 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013002900000000001 2020-01-16T00:13:10.806 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.807 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.807 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.807 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033886+0000", 2020-01-16T00:13:10.807 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035882600000000002 2020-01-16T00:13:10.807 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.807 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.807 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:10.808 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.034245+0000", 2020-01-16T00:13:10.808 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0072798489999999997 2020-01-16T00:13:10.808 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.808 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.808 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.808 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.041525+0000", 2020-01-16T00:13:10.808 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000140488 2020-01-16T00:13:10.808 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.809 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.809 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:10.809 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.041665+0000", 2020-01-16T00:13:10.809 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000233766 2020-01-16T00:13:10.809 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.809 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.809 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:10.810 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.041899+0000", 2020-01-16T00:13:10.810 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0035766539999999999 2020-01-16T00:13:10.810 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.810 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.810 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:10.810 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.045475+0000", 2020-01-16T00:13:10.810 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013333150000000001 2020-01-16T00:13:10.810 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.811 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.811 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:10.811 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.046809+0000", 2020-01-16T00:13:10.811 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031447860000000001 2020-01-16T00:13:10.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.811 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.811 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:10.811 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.049954+0000", 2020-01-16T00:13:10.812 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2318000000000003e-05 2020-01-16T00:13:10.812 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.812 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.812 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:10.812 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.050006+0000", 2020-01-16T00:13:10.813 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7972000000000001e-05 2020-01-16T00:13:10.814 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.814 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.814 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:10.814 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.050054+0000", 2020-01-16T00:13:10.814 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016764102999999999 2020-01-16T00:13:10.815 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:10.815 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:10.815 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:10.815 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.815 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.815 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1168 3.0 3:03b92cf0:::smithi12313934-26:head [write 2905319~482223] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:13:10.815 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:41.672662+0000", 2020-01-16T00:13:10.816 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 29.071775629000001, 2020-01-16T00:13:10.816 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015446582, 2020-01-16T00:13:10.816 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:10.816 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:10.816 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:10.816 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:10.816 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:10.816 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1168 2020-01-16T00:13:10.817 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.817 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:10.817 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.817 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:10.817 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672662+0000", 2020-01-16T00:13:10.817 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:10.817 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.818 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.818 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:10.818 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672662+0000", 2020-01-16T00:13:10.818 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:10.818 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.818 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.818 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:10.818 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672660+0000", 2020-01-16T00:13:10.819 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013924799999999999 2020-01-16T00:13:10.819 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.819 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.819 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:10.819 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672799+0000", 2020-01-16T00:13:10.819 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1304000000000001e-05 2020-01-16T00:13:10.819 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.820 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.820 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:10.820 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672810+0000", 2020-01-16T00:13:10.820 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2379999999999999e-06 2020-01-16T00:13:10.820 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.820 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.820 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:10.820 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672815+0000", 2020-01-16T00:13:10.821 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9329000000000002e-05 2020-01-16T00:13:10.821 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.821 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.821 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.821 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672854+0000", 2020-01-16T00:13:10.821 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010573939999999999 2020-01-16T00:13:10.821 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.821 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.822 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:10.822 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.673911+0000", 2020-01-16T00:13:10.822 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.01042181 2020-01-16T00:13:10.822 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.822 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.822 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.822 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.684333+0000", 2020-01-16T00:13:10.822 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012700499999999999 2020-01-16T00:13:10.823 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.823 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.823 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:10.823 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.684460+0000", 2020-01-16T00:13:10.823 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019923299999999999 2020-01-16T00:13:10.823 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.824 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.824 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:10.824 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.684659+0000", 2020-01-16T00:13:10.824 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020339300000000002 2020-01-16T00:13:10.824 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.824 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.824 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:10.825 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.686693+0000", 2020-01-16T00:13:10.825 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023967899999999999 2020-01-16T00:13:10.825 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.825 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.825 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:10.825 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.686933+0000", 2020-01-16T00:13:10.825 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010489169999999999 2020-01-16T00:13:10.825 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.826 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.826 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:10.826 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.687982+0000", 2020-01-16T00:13:10.826 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3857999999999997e-05 2020-01-16T00:13:10.826 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.826 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.826 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:10.826 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.688056+0000", 2020-01-16T00:13:10.827 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2324999999999998e-05 2020-01-16T00:13:10.827 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.827 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.827 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:10.827 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.688108+0000", 2020-01-16T00:13:10.827 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015446582 2020-01-16T00:13:10.827 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:10.827 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:10.828 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:10.828 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.828 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.828 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1169 3.0 3:03b92cf0:::smithi12313934-26:head [setxattr _header (56) in=7b,truncate 3798349] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:13:10.828 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:41.672823+0000", 2020-01-16T00:13:10.828 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 29.071614722, 2020-01-16T00:13:10.828 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015457745, 2020-01-16T00:13:10.829 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:10.829 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:10.829 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:10.829 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:10.829 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:10.829 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1169 2020-01-16T00:13:10.829 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.830 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:10.830 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.830 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:10.830 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672823+0000", 2020-01-16T00:13:10.830 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:10.830 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.830 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.830 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:10.831 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672823+0000", 2020-01-16T00:13:10.831 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:10.831 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.831 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.831 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:10.831 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672822+0000", 2020-01-16T00:13:10.831 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.685e-06 2020-01-16T00:13:10.832 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.832 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.832 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:10.832 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672827+0000", 2020-01-16T00:13:10.832 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.485e-06 2020-01-16T00:13:10.832 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.832 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.832 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:10.833 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672832+0000", 2020-01-16T00:13:10.833 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6440000000000002e-06 2020-01-16T00:13:10.833 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.833 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.833 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:10.833 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672836+0000", 2020-01-16T00:13:10.833 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00112415 2020-01-16T00:13:10.834 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.834 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.834 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.834 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.673960+0000", 2020-01-16T00:13:10.834 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5918999999999998e-05 2020-01-16T00:13:10.834 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.835 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.835 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:10.835 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.674016+0000", 2020-01-16T00:13:10.835 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011614579999999999 2020-01-16T00:13:10.835 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.835 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.835 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.835 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.685630+0000", 2020-01-16T00:13:10.836 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011684399999999999 2020-01-16T00:13:10.836 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.836 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.836 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:10.836 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.685747+0000", 2020-01-16T00:13:10.836 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020774199999999999 2020-01-16T00:13:10.836 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.837 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.837 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:10.837 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.685955+0000", 2020-01-16T00:13:10.837 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00095868400000000001 2020-01-16T00:13:10.837 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.837 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.837 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:10.837 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.686914+0000", 2020-01-16T00:13:10.838 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5278000000000001e-05 2020-01-16T00:13:10.838 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.838 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.838 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:10.838 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.686939+0000", 2020-01-16T00:13:10.838 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001227611 2020-01-16T00:13:10.838 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.838 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.839 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:10.839 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.688166+0000", 2020-01-16T00:13:10.839 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1724999999999997e-05 2020-01-16T00:13:10.839 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.839 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.839 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:10.839 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.688218+0000", 2020-01-16T00:13:10.840 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2057000000000006e-05 2020-01-16T00:13:10.840 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.840 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.840 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:10.840 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.688280+0000", 2020-01-16T00:13:10.840 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015457745 2020-01-16T00:13:10.840 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:10.840 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:10.841 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:10.841 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.841 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.841 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1170 3.0 3:03b92cf0:::smithi12313934-26:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-01-16T00:13:10.841 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:41.672848+0000", 2020-01-16T00:13:10.841 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 29.071589154000002, 2020-01-16T00:13:10.841 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015733389, 2020-01-16T00:13:10.841 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:10.842 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:10.842 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:10.842 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:10.842 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:10.842 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1170 2020-01-16T00:13:10.842 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.842 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:10.842 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.843 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:10.843 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672848+0000", 2020-01-16T00:13:10.843 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:10.843 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.843 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.843 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:10.843 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672848+0000", 2020-01-16T00:13:10.844 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:10.844 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.844 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.844 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:10.844 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672847+0000", 2020-01-16T00:13:10.844 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.303e-06 2020-01-16T00:13:10.845 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.845 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.845 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:10.845 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672851+0000", 2020-01-16T00:13:10.845 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4970000000000001e-06 2020-01-16T00:13:10.845 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.845 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.846 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:10.846 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672856+0000", 2020-01-16T00:13:10.846 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8870000000000002e-06 2020-01-16T00:13:10.846 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.846 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.846 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:10.846 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672860+0000", 2020-01-16T00:13:10.846 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0011720140000000001 2020-01-16T00:13:10.847 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.847 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.847 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.847 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.674032+0000", 2020-01-16T00:13:10.847 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2577000000000003e-05 2020-01-16T00:13:10.848 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.848 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.848 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:10.848 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.674065+0000", 2020-01-16T00:13:10.848 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012264538 2020-01-16T00:13:10.848 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.848 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.848 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.849 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.686329+0000", 2020-01-16T00:13:10.849 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000106715 2020-01-16T00:13:10.849 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.849 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.849 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:10.849 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.686436+0000", 2020-01-16T00:13:10.849 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018880800000000001 2020-01-16T00:13:10.849 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.850 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.850 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.850 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.688324+0000", 2020-01-16T00:13:10.850 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012818500000000001 2020-01-16T00:13:10.850 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.850 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.850 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:10.851 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.688452+0000", 2020-01-16T00:13:10.851 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000129197 2020-01-16T00:13:10.851 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.851 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.851 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:10.851 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.688581+0000", 2020-01-16T00:13:10.852 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015733389 2020-01-16T00:13:10.852 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:10.852 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:10.852 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:10.852 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.852 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.852 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1394 3.4 3:240c7503:::smithi12313934-35:head [setxattr _header (56) in=7b,truncate 2285752] snapc 0=[] ondisk+write+known_if_redirected e28)", 2020-01-16T00:13:10.852 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:46.625161+0000", 2020-01-16T00:13:10.853 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 24.119276357, 2020-01-16T00:13:10.853 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015636654, 2020-01-16T00:13:10.853 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:10.853 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:10.853 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:10.853 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:10.853 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:10.854 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1394 2020-01-16T00:13:10.854 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.854 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:10.854 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.855 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:10.855 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625161+0000", 2020-01-16T00:13:10.855 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:10.855 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.855 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.855 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:10.855 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625161+0000", 2020-01-16T00:13:10.856 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:10.856 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.856 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.856 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:10.856 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625160+0000", 2020-01-16T00:13:10.856 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3960000000000002e-06 2020-01-16T00:13:10.856 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.857 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.857 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:10.857 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625164+0000", 2020-01-16T00:13:10.857 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7199999999999997e-06 2020-01-16T00:13:10.857 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.857 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.857 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:10.857 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625168+0000", 2020-01-16T00:13:10.858 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2960000000000001e-06 2020-01-16T00:13:10.858 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.858 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.858 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:10.858 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625171+0000", 2020-01-16T00:13:10.858 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001705972 2020-01-16T00:13:10.858 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.859 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.859 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.859 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.626877+0000", 2020-01-16T00:13:10.859 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039484300000000002 2020-01-16T00:13:10.859 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.859 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.859 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:10.859 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630825+0000", 2020-01-16T00:13:10.860 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0068775310000000001 2020-01-16T00:13:10.860 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.860 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.860 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.860 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.637703+0000", 2020-01-16T00:13:10.860 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013884100000000001 2020-01-16T00:13:10.860 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.861 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.861 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:10.861 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.637842+0000", 2020-01-16T00:13:10.861 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00024336400000000001 2020-01-16T00:13:10.861 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.861 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.861 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:13:10.862 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.638085+0000", 2020-01-16T00:13:10.862 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010183900000000001 2020-01-16T00:13:10.862 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.862 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.862 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:10.862 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.639103+0000", 2020-01-16T00:13:10.862 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012246699999999999 2020-01-16T00:13:10.862 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.863 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.863 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:10.863 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.639226+0000", 2020-01-16T00:13:10.863 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0014329099999999999 2020-01-16T00:13:10.863 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.863 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.864 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:10.864 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.640659+0000", 2020-01-16T00:13:10.864 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8109000000000002e-05 2020-01-16T00:13:10.864 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.864 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.864 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:10.864 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.640717+0000", 2020-01-16T00:13:10.865 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0784000000000004e-05 2020-01-16T00:13:10.865 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.865 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.865 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:10.865 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.640798+0000", 2020-01-16T00:13:10.865 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015636654 2020-01-16T00:13:10.866 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:10.866 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:10.866 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:10.866 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.866 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.866 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1395 3.4 3:240c7503:::smithi12313934-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e28)", 2020-01-16T00:13:10.866 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:46.625191+0000", 2020-01-16T00:13:10.866 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 24.119246146999998, 2020-01-16T00:13:10.867 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015878716000000001, 2020-01-16T00:13:10.867 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:10.867 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:10.867 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:10.867 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:10.867 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:10.867 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1395 2020-01-16T00:13:10.868 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.868 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:10.868 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.868 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:10.868 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625191+0000", 2020-01-16T00:13:10.868 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:10.868 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.868 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.869 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:10.869 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625191+0000", 2020-01-16T00:13:10.869 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:10.869 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.869 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.869 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:10.869 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625191+0000", 2020-01-16T00:13:10.870 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.1090000000000001e-06 2020-01-16T00:13:10.870 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.870 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.870 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:10.870 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625193+0000", 2020-01-16T00:13:10.870 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2730000000000008e-06 2020-01-16T00:13:10.870 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.871 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.871 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:10.871 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625202+0000", 2020-01-16T00:13:10.871 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3700000000000002e-06 2020-01-16T00:13:10.871 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.871 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.871 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:10.872 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625204+0000", 2020-01-16T00:13:10.872 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056573049999999996 2020-01-16T00:13:10.872 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.872 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.872 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.872 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630862+0000", 2020-01-16T00:13:10.872 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6047000000000001e-05 2020-01-16T00:13:10.872 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.873 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.873 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:10.873 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630918+0000", 2020-01-16T00:13:10.873 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0077941180000000001 2020-01-16T00:13:10.873 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.873 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.873 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.874 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.638712+0000", 2020-01-16T00:13:10.874 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118096 2020-01-16T00:13:10.874 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.874 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.874 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:10.874 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.638830+0000", 2020-01-16T00:13:10.874 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0019986740000000002 2020-01-16T00:13:10.875 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.875 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.875 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.875 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.640829+0000", 2020-01-16T00:13:10.875 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012562399999999999 2020-01-16T00:13:10.876 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.876 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.876 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:10.876 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.640954+0000", 2020-01-16T00:13:10.876 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000115536 2020-01-16T00:13:10.876 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.876 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.876 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:10.877 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.641070+0000", 2020-01-16T00:13:10.877 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015878716000000001 2020-01-16T00:13:10.877 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:10.877 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:10.877 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:10.877 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.877 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.878 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1639 3.0 3:00c146d5:::smithi12313934-18:head [write 423576~401598] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:13:10.878 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.774317+0000", 2020-01-16T00:13:10.878 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 18.970120735999998, 2020-01-16T00:13:10.878 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018305219000000001, 2020-01-16T00:13:10.878 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:10.878 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:10.878 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:10.878 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:10.879 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:10.879 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1639 2020-01-16T00:13:10.879 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.879 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:10.879 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.879 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:10.879 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774317+0000", 2020-01-16T00:13:10.880 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:10.880 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.880 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.880 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:10.880 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774317+0000", 2020-01-16T00:13:10.880 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:10.880 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.880 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.881 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:10.881 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774313+0000", 2020-01-16T00:13:10.881 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000111173 2020-01-16T00:13:10.881 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.881 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.881 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:10.881 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774425+0000", 2020-01-16T00:13:10.882 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7598999999999999e-05 2020-01-16T00:13:10.882 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.882 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.882 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:10.882 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774442+0000", 2020-01-16T00:13:10.882 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0830000000000004e-06 2020-01-16T00:13:10.882 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.882 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.883 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:10.883 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774450+0000", 2020-01-16T00:13:10.883 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0017886309999999999 2020-01-16T00:13:10.883 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.883 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.883 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.883 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.776239+0000", 2020-01-16T00:13:10.883 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3789e-05 2020-01-16T00:13:10.884 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.884 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.884 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op must wait for map", 2020-01-16T00:13:10.884 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.776253+0000", 2020-01-16T00:13:10.884 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095730729999999997 2020-01-16T00:13:10.884 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.884 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.884 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.885 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.785826+0000", 2020-01-16T00:13:10.885 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014220300000000001 2020-01-16T00:13:10.885 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.885 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.885 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:10.885 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.785968+0000", 2020-01-16T00:13:10.885 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023618900000000001 2020-01-16T00:13:10.886 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.886 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.886 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:10.886 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.786204+0000", 2020-01-16T00:13:10.886 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0062297389999999998 2020-01-16T00:13:10.886 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.887 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.887 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:10.887 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792434+0000", 2020-01-16T00:13:10.887 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9635999999999998e-05 2020-01-16T00:13:10.887 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.887 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.888 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:10.888 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792473+0000", 2020-01-16T00:13:10.888 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6456000000000002e-05 2020-01-16T00:13:10.888 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.888 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.888 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:10.888 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792540+0000", 2020-01-16T00:13:10.889 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9873999999999998e-05 2020-01-16T00:13:10.889 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.889 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.889 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:10.890 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792580+0000", 2020-01-16T00:13:10.890 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1924e-05 2020-01-16T00:13:10.890 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.890 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.890 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:10.890 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792622+0000", 2020-01-16T00:13:10.890 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018305219000000001 2020-01-16T00:13:10.890 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:10.891 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:10.891 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:10.891 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.891 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.891 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1890 3.0 3:0b1f384f:::smithi12313934-33:head [write 2797915~596476] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:13:10.891 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.232534+0000", 2020-01-16T00:13:10.891 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 13.511902765, 2020-01-16T00:13:10.892 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016106137, 2020-01-16T00:13:10.892 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:10.892 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:10.892 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:10.893 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:10.893 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:10.893 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1890 2020-01-16T00:13:10.893 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.893 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:10.894 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.894 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:10.894 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232534+0000", 2020-01-16T00:13:10.894 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:10.894 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.894 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.894 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:10.895 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232534+0000", 2020-01-16T00:13:10.895 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:10.895 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.895 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.895 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:10.895 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232532+0000", 2020-01-16T00:13:10.895 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00024232100000000001 2020-01-16T00:13:10.896 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.896 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.896 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:10.896 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232774+0000", 2020-01-16T00:13:10.896 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4601e-05 2020-01-16T00:13:10.896 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.897 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.897 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:10.897 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232789+0000", 2020-01-16T00:13:10.897 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8410000000000006e-06 2020-01-16T00:13:10.897 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.897 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.897 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:10.898 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232798+0000", 2020-01-16T00:13:10.898 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9756999999999997e-05 2020-01-16T00:13:10.898 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.898 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.898 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.898 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232857+0000", 2020-01-16T00:13:10.898 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4678000000000007e-05 2020-01-16T00:13:10.899 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.899 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.900 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:10.901 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232922+0000", 2020-01-16T00:13:10.901 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033861020000000002 2020-01-16T00:13:10.901 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.901 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.901 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.901 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.236308+0000", 2020-01-16T00:13:10.901 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000568996 2020-01-16T00:13:10.902 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.902 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.902 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:10.902 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.236877+0000", 2020-01-16T00:13:10.902 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0085444240000000001 2020-01-16T00:13:10.902 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.902 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.903 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.903 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.245422+0000", 2020-01-16T00:13:10.903 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5595999999999998e-05 2020-01-16T00:13:10.903 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.903 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.903 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:10.903 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.245507+0000", 2020-01-16T00:13:10.904 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016984300000000001 2020-01-16T00:13:10.904 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.904 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.904 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:10.904 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.245677+0000", 2020-01-16T00:13:10.904 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001649495 2020-01-16T00:13:10.904 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.904 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.905 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:10.905 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247326+0000", 2020-01-16T00:13:10.905 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023308399999999999 2020-01-16T00:13:10.905 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.905 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.905 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:10.906 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247560+0000", 2020-01-16T00:13:10.909 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00099501800000000007 2020-01-16T00:13:10.910 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.910 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.910 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:10.910 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248555+0000", 2020-01-16T00:13:10.911 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5936999999999998e-05 2020-01-16T00:13:10.911 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.911 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.911 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:10.911 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248591+0000", 2020-01-16T00:13:10.912 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0108999999999997e-05 2020-01-16T00:13:10.912 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.912 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.912 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:10.913 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248641+0000", 2020-01-16T00:13:10.913 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016106137 2020-01-16T00:13:10.913 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:10.913 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:10.913 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:10.914 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.914 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.914 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1891 3.0 3:0b1f384f:::smithi12313934-33:head [setxattr _header (56) in=7b,truncate 3395014] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:13:10.914 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.232813+0000", 2020-01-16T00:13:10.914 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 13.511623963, 2020-01-16T00:13:10.915 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015999295, 2020-01-16T00:13:10.915 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:10.915 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:10.915 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:10.915 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:10.916 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:10.916 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1891 2020-01-16T00:13:10.916 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.916 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:10.917 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.917 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:10.917 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232813+0000", 2020-01-16T00:13:10.917 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:10.917 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.918 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.918 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:10.918 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232813+0000", 2020-01-16T00:13:10.918 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:10.918 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.919 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.919 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:10.919 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232812+0000", 2020-01-16T00:13:10.919 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.576e-06 2020-01-16T00:13:10.920 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.920 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.920 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:10.920 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232820+0000", 2020-01-16T00:13:10.920 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3079999999999997e-06 2020-01-16T00:13:10.921 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.921 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.921 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:10.921 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232829+0000", 2020-01-16T00:13:10.921 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7119999999999997e-06 2020-01-16T00:13:10.922 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.922 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.922 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:10.922 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232835+0000", 2020-01-16T00:13:10.922 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013438400000000001 2020-01-16T00:13:10.922 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.922 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.923 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.923 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232969+0000", 2020-01-16T00:13:10.923 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0562000000000001e-05 2020-01-16T00:13:10.923 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.923 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.923 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:10.923 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.233029+0000", 2020-01-16T00:13:10.923 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039354239999999999 2020-01-16T00:13:10.924 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.924 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.924 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.924 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.236965+0000", 2020-01-16T00:13:10.924 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0482999999999998e-05 2020-01-16T00:13:10.924 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.924 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.925 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:10.925 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.237015+0000", 2020-01-16T00:13:10.925 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0096086209999999995 2020-01-16T00:13:10.925 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.925 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.925 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.925 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.246624+0000", 2020-01-16T00:13:10.926 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6150999999999995e-05 2020-01-16T00:13:10.926 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.926 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.926 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:10.926 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.246720+0000", 2020-01-16T00:13:10.926 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015032899999999999 2020-01-16T00:13:10.926 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.927 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.927 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:10.927 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.246870+0000", 2020-01-16T00:13:10.927 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00064760700000000002 2020-01-16T00:13:10.927 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.927 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.927 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:10.927 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247518+0000", 2020-01-16T00:13:10.928 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000198531 2020-01-16T00:13:10.928 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.928 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.928 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:10.928 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247717+0000", 2020-01-16T00:13:10.928 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010189439999999999 2020-01-16T00:13:10.928 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.929 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.929 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:10.929 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248736+0000", 2020-01-16T00:13:10.929 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4863000000000001e-05 2020-01-16T00:13:10.929 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.929 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.929 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:10.929 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248770+0000", 2020-01-16T00:13:10.930 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2153000000000001e-05 2020-01-16T00:13:10.930 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.930 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.930 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:10.930 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248813+0000", 2020-01-16T00:13:10.930 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015999295 2020-01-16T00:13:10.930 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:10.931 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:10.931 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:10.931 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.931 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.931 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1892 3.0 3:0b1f384f:::smithi12313934-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e31)", 2020-01-16T00:13:10.931 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.232856+0000", 2020-01-16T00:13:10.931 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 13.511581739, 2020-01-16T00:13:10.932 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016171544, 2020-01-16T00:13:10.932 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:10.932 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:10.932 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:10.932 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:10.932 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:10.932 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1892 2020-01-16T00:13:10.932 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.933 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:10.933 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.933 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:10.933 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232856+0000", 2020-01-16T00:13:10.933 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:10.933 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.933 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.934 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:10.934 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232856+0000", 2020-01-16T00:13:10.934 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:10.934 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.934 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.934 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:10.934 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232855+0000", 2020-01-16T00:13:10.935 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8640000000000004e-06 2020-01-16T00:13:10.935 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.935 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.935 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:10.935 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232860+0000", 2020-01-16T00:13:10.935 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0000000000000001e-05 2020-01-16T00:13:10.935 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.935 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.936 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:10.936 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232870+0000", 2020-01-16T00:13:10.936 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7420000000000003e-06 2020-01-16T00:13:10.936 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.936 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.936 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:10.936 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232875+0000", 2020-01-16T00:13:10.937 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017933200000000001 2020-01-16T00:13:10.937 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.937 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.937 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.937 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.233055+0000", 2020-01-16T00:13:10.937 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6269e-05 2020-01-16T00:13:10.937 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.938 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.938 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:10.938 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.233101+0000", 2020-01-16T00:13:10.938 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039780090000000002 2020-01-16T00:13:10.938 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.938 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.938 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.938 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.237079+0000", 2020-01-16T00:13:10.939 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3677000000000001e-05 2020-01-16T00:13:10.939 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.939 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.939 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:10.939 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.237133+0000", 2020-01-16T00:13:10.939 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010036671 2020-01-16T00:13:10.939 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.939 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.940 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.940 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247169+0000", 2020-01-16T00:13:10.940 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010280699999999999 2020-01-16T00:13:10.940 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.940 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.940 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:10.940 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247272+0000", 2020-01-16T00:13:10.940 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001579708 2020-01-16T00:13:10.941 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.941 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.941 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.941 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248852+0000", 2020-01-16T00:13:10.941 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9704999999999996e-05 2020-01-16T00:13:10.941 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.941 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.942 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:10.942 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248942+0000", 2020-01-16T00:13:10.942 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5557999999999996e-05 2020-01-16T00:13:10.942 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.942 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.942 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:10.942 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.249027+0000", 2020-01-16T00:13:10.942 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016171544 2020-01-16T00:13:10.943 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:10.943 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:10.943 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:10.943 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.943 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.943 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1950 3.0 3:0b1f384f:::smithi12313934-33:head [write 1889359~541737] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-01-16T00:13:10.943 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.922819+0000", 2020-01-16T00:13:10.943 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 12.821618374, 2020-01-16T00:13:10.944 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017502846999999998, 2020-01-16T00:13:10.944 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:10.944 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:10.944 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:10.944 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:10.944 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:10.944 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1950 2020-01-16T00:13:10.945 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.945 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:10.945 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.945 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:10.945 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922819+0000", 2020-01-16T00:13:10.945 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:10.945 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.945 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.946 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:10.946 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922819+0000", 2020-01-16T00:13:10.946 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:10.946 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.946 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.946 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:10.946 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922818+0000", 2020-01-16T00:13:10.947 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000146408 2020-01-16T00:13:10.947 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.947 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.947 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:10.947 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922964+0000", 2020-01-16T00:13:10.947 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4720000000000001e-06 2020-01-16T00:13:10.947 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.948 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.948 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:10.948 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922974+0000", 2020-01-16T00:13:10.948 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0720000000000001e-06 2020-01-16T00:13:10.948 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.948 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.949 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:10.949 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922978+0000", 2020-01-16T00:13:10.949 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6152000000000001e-05 2020-01-16T00:13:10.949 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.949 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.949 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.949 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.923024+0000", 2020-01-16T00:13:10.950 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00087368199999999997 2020-01-16T00:13:10.950 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.950 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.950 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:10.950 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.923897+0000", 2020-01-16T00:13:10.950 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011944435999999999 2020-01-16T00:13:10.950 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.951 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.951 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.951 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.935842+0000", 2020-01-16T00:13:10.951 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000144146 2020-01-16T00:13:10.951 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.951 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.951 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:10.951 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.935986+0000", 2020-01-16T00:13:10.952 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019775500000000001 2020-01-16T00:13:10.952 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.952 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.952 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:10.952 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.936184+0000", 2020-01-16T00:13:10.952 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002539342 2020-01-16T00:13:10.952 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.952 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:10.953 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938723+0000", 2020-01-16T00:13:10.953 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014193699999999999 2020-01-16T00:13:10.953 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.953 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:10.953 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938865+0000", 2020-01-16T00:13:10.954 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013829020000000001 2020-01-16T00:13:10.954 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.954 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.954 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:10.954 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.940248+0000", 2020-01-16T00:13:10.954 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2558000000000001e-05 2020-01-16T00:13:10.954 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.954 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.955 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:10.955 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.940291+0000", 2020-01-16T00:13:10.955 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1217999999999998e-05 2020-01-16T00:13:10.955 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.955 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.955 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:10.955 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.940322+0000", 2020-01-16T00:13:10.955 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017502846999999998 2020-01-16T00:13:10.956 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:10.956 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:10.956 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:10.956 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.956 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.956 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1951 3.0 3:0b1f384f:::smithi12313934-33:head [write 2955792~541902] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-01-16T00:13:10.956 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.929019+0000", 2020-01-16T00:13:10.957 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 12.815418224, 2020-01-16T00:13:10.957 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021625710999999999, 2020-01-16T00:13:10.957 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:10.957 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:10.957 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:10.957 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:10.957 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:10.957 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1951 2020-01-16T00:13:10.958 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.958 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:10.958 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.958 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:10.958 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929019+0000", 2020-01-16T00:13:10.958 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:10.958 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.958 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.959 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:10.959 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929019+0000", 2020-01-16T00:13:10.959 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999989 2020-01-16T00:13:10.959 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.959 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.959 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:10.959 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929008+0000", 2020-01-16T00:13:10.960 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000212513 2020-01-16T00:13:10.960 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.960 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.960 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:10.960 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929221+0000", 2020-01-16T00:13:10.960 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3229e-05 2020-01-16T00:13:10.961 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.961 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.961 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:10.961 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929234+0000", 2020-01-16T00:13:10.961 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9170000000000003e-06 2020-01-16T00:13:10.961 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.961 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.961 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:10.962 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929240+0000", 2020-01-16T00:13:10.962 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4045000000000002e-05 2020-01-16T00:13:10.962 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.962 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.962 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.962 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929294+0000", 2020-01-16T00:13:10.962 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9797999999999996e-05 2020-01-16T00:13:10.962 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.963 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.963 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:10.963 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929384+0000", 2020-01-16T00:13:10.963 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083450729999999997 2020-01-16T00:13:10.963 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.963 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.963 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.963 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.937729+0000", 2020-01-16T00:13:10.964 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043251699999999999 2020-01-16T00:13:10.964 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.964 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.964 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:10.964 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938161+0000", 2020-01-16T00:13:10.964 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095115489999999994 2020-01-16T00:13:10.964 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.965 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.965 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.965 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.947673+0000", 2020-01-16T00:13:10.965 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012395099999999999 2020-01-16T00:13:10.965 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.965 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.965 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:10.965 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.947797+0000", 2020-01-16T00:13:10.966 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00024849200000000001 2020-01-16T00:13:10.966 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.966 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.966 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:10.966 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.948045+0000", 2020-01-16T00:13:10.966 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0025081050000000001 2020-01-16T00:13:10.966 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.967 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.967 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:10.967 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950553+0000", 2020-01-16T00:13:10.967 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9760000000000003e-06 2020-01-16T00:13:10.967 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.967 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.967 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:10.967 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950558+0000", 2020-01-16T00:13:10.968 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3286999999999999e-05 2020-01-16T00:13:10.968 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.968 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.968 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:10.968 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950581+0000", 2020-01-16T00:13:10.968 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2759000000000003e-05 2020-01-16T00:13:10.968 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.968 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.969 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:10.969 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950614+0000", 2020-01-16T00:13:10.969 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0525999999999999e-05 2020-01-16T00:13:10.969 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.969 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.969 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:10.969 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950645+0000", 2020-01-16T00:13:10.970 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021625710999999999 2020-01-16T00:13:10.970 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:10.970 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:10.970 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:10.970 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.970 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.970 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1952 3.0 3:0b1f384f:::smithi12313934-33:head [setxattr _header (56) in=7b,truncate 3563084] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-01-16T00:13:10.971 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:13:10.971 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 12.815188887, 2020-01-16T00:13:10.971 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021534685000000001, 2020-01-16T00:13:10.971 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:10.971 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:10.971 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:10.971 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:10.972 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:10.972 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1952 2020-01-16T00:13:10.972 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.972 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:10.972 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.972 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:10.973 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:13:10.973 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:10.973 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.973 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.973 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:10.973 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:13:10.974 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:10.974 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.974 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.974 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:10.974 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:13:10.974 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6670000000000004e-06 2020-01-16T00:13:10.974 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.974 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.975 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:10.975 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929252+0000", 2020-01-16T00:13:10.975 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9380000000000002e-06 2020-01-16T00:13:10.975 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.975 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.975 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:10.975 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929256+0000", 2020-01-16T00:13:10.975 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.712e-06 2020-01-16T00:13:10.976 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.976 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.976 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:10.976 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929264+0000", 2020-01-16T00:13:10.976 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018030300000000001 2020-01-16T00:13:10.976 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.976 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.976 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.977 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929444+0000", 2020-01-16T00:13:10.977 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7562000000000002e-05 2020-01-16T00:13:10.977 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.977 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.977 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:10.977 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929502+0000", 2020-01-16T00:13:10.977 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0087297859999999998 2020-01-16T00:13:10.978 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.978 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.978 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.978 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938232+0000", 2020-01-16T00:13:10.978 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0452999999999996e-05 2020-01-16T00:13:10.978 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.978 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.978 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:10.979 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938322+0000", 2020-01-16T00:13:10.979 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011061728 2020-01-16T00:13:10.979 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.979 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.979 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.979 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.949384+0000", 2020-01-16T00:13:10.979 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000117828 2020-01-16T00:13:10.979 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.980 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.980 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:10.980 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.949502+0000", 2020-01-16T00:13:10.980 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022468499999999999 2020-01-16T00:13:10.980 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.980 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.980 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:10.980 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.949726+0000", 2020-01-16T00:13:10.981 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00083532099999999998 2020-01-16T00:13:10.981 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.981 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.981 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:10.981 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950562+0000", 2020-01-16T00:13:10.982 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013017700000000001 2020-01-16T00:13:10.982 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.982 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.982 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:10.982 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950692+0000", 2020-01-16T00:13:10.982 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9813000000000001e-05 2020-01-16T00:13:10.982 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.983 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.983 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:10.983 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950722+0000", 2020-01-16T00:13:10.983 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0592000000000001e-05 2020-01-16T00:13:10.983 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.983 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.983 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:10.983 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950752+0000", 2020-01-16T00:13:10.984 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0920000000000002e-05 2020-01-16T00:13:10.984 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.984 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.984 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:10.984 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950783+0000", 2020-01-16T00:13:10.984 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021534685000000001 2020-01-16T00:13:10.984 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:10.985 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:10.985 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:10.985 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.985 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.985 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1953 3.0 3:0b1f384f:::smithi12313934-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e32)", 2020-01-16T00:13:10.986 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:13:10.986 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 12.815155896, 2020-01-16T00:13:10.986 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021667537000000001, 2020-01-16T00:13:10.986 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:10.986 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:10.986 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:10.986 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:10.987 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:10.987 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1953 2020-01-16T00:13:10.987 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.987 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:10.987 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.987 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:10.987 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:13:10.988 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:10.988 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.989 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.989 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:10.989 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:13:10.989 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:10.989 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.989 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.989 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:10.990 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:13:10.990 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7030000000000002e-06 2020-01-16T00:13:10.990 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.990 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.990 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:10.990 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929284+0000", 2020-01-16T00:13:10.990 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0909999999999993e-06 2020-01-16T00:13:10.990 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.991 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.991 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:10.991 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929292+0000", 2020-01-16T00:13:10.991 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8359999999999999e-06 2020-01-16T00:13:10.991 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.991 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.991 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:10.991 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929297+0000", 2020-01-16T00:13:10.992 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023538199999999999 2020-01-16T00:13:10.992 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.992 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.992 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.992 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929532+0000", 2020-01-16T00:13:10.992 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1173000000000004e-05 2020-01-16T00:13:10.993 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.993 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.993 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:10.993 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929603+0000", 2020-01-16T00:13:10.993 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0087813509999999997 2020-01-16T00:13:10.993 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.993 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.993 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.994 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938384+0000", 2020-01-16T00:13:10.994 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9752000000000006e-05 2020-01-16T00:13:10.994 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.994 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.994 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:10.994 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938454+0000", 2020-01-16T00:13:10.994 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011707851999999999 2020-01-16T00:13:10.994 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.995 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.995 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.995 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950162+0000", 2020-01-16T00:13:10.995 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011060099999999999 2020-01-16T00:13:10.995 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.996 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.996 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:10.996 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950273+0000", 2020-01-16T00:13:10.996 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00053989700000000001 2020-01-16T00:13:10.996 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.996 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.997 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:10.997 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950813+0000", 2020-01-16T00:13:10.997 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8128999999999999e-05 2020-01-16T00:13:10.997 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.997 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.997 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:10.998 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950871+0000", 2020-01-16T00:13:10.998 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8242000000000001e-05 2020-01-16T00:13:10.998 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.998 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.998 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:10.999 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950949+0000", 2020-01-16T00:13:10.999 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021667537000000001 2020-01-16T00:13:10.999 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:10.999 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:10.999 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:10.999 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:10.999 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:10.999 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2191 3.0 3:0b70bb20:::smithi12313934-46:head [setxattr _header (56) in=7b,truncate 2016873] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:13:11.000 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:03.256877+0000", 2020-01-16T00:13:11.000 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 7.4875603310000001, 2020-01-16T00:13:11.000 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014406043, 2020-01-16T00:13:11.000 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:11.000 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:11.000 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:11.000 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:11.000 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:11.001 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2191 2020-01-16T00:13:11.001 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.001 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:11.001 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.001 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:11.001 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.256877+0000", 2020-01-16T00:13:11.001 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:11.002 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.002 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.002 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:11.002 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.256877+0000", 2020-01-16T00:13:11.002 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:11.002 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.002 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.002 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:11.003 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.256876+0000", 2020-01-16T00:13:11.003 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7060000000000003e-06 2020-01-16T00:13:11.003 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.003 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.003 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:11.003 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.256881+0000", 2020-01-16T00:13:11.003 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3770000000000004e-06 2020-01-16T00:13:11.003 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.004 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.004 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:11.004 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.256886+0000", 2020-01-16T00:13:11.004 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5980000000000001e-06 2020-01-16T00:13:11.004 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.004 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.004 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:11.004 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.256890+0000", 2020-01-16T00:13:11.005 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000732199 2020-01-16T00:13:11.005 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.005 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.005 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:11.005 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.257622+0000", 2020-01-16T00:13:11.005 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022652300000000001 2020-01-16T00:13:11.006 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.006 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.006 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:11.006 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.257849+0000", 2020-01-16T00:13:11.006 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010272460000000001 2020-01-16T00:13:11.006 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.007 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.007 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:11.007 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.268121+0000", 2020-01-16T00:13:11.008 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2525000000000001e-05 2020-01-16T00:13:11.008 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.008 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.008 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:11.008 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.268194+0000", 2020-01-16T00:13:11.008 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010802 2020-01-16T00:13:11.009 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.009 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.009 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:11.009 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.268302+0000", 2020-01-16T00:13:11.009 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0011749659999999999 2020-01-16T00:13:11.009 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.010 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.010 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:11.010 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.269477+0000", 2020-01-16T00:13:11.010 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001587757 2020-01-16T00:13:11.010 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.010 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.010 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:11.010 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.271064+0000", 2020-01-16T00:13:11.011 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1802999999999999e-05 2020-01-16T00:13:11.011 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.011 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.011 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:11.011 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.271116+0000", 2020-01-16T00:13:11.011 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5095e-05 2020-01-16T00:13:11.011 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.012 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.012 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:11.012 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.271171+0000", 2020-01-16T00:13:11.012 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000111723 2020-01-16T00:13:11.012 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.012 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.012 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:11.012 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.271283+0000", 2020-01-16T00:13:11.013 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014406043 2020-01-16T00:13:11.013 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.013 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:11.013 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.013 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.013 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.013 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2192 3.0 3:0b70bb20:::smithi12313934-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-01-16T00:13:11.014 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:03.256902+0000", 2020-01-16T00:13:11.014 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 7.4875348290000003, 2020-01-16T00:13:11.014 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014663150999999999, 2020-01-16T00:13:11.014 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:11.014 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:11.014 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:11.014 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:11.014 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:11.015 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2192 2020-01-16T00:13:11.015 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.015 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:11.016 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.016 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:11.016 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.256902+0000", 2020-01-16T00:13:11.017 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:11.017 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.017 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.018 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:11.018 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.256902+0000", 2020-01-16T00:13:11.018 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:11.018 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.018 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.018 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:11.018 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.256902+0000", 2020-01-16T00:13:11.019 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3409999999999999e-06 2020-01-16T00:13:11.019 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.019 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.019 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:11.019 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.256905+0000", 2020-01-16T00:13:11.019 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6280000000000002e-06 2020-01-16T00:13:11.020 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.020 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.020 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:11.020 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.256911+0000", 2020-01-16T00:13:11.020 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9009999999999996e-06 2020-01-16T00:13:11.020 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.020 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.021 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:11.021 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.256915+0000", 2020-01-16T00:13:11.021 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001008756 2020-01-16T00:13:11.021 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.021 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.021 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:11.021 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.257923+0000", 2020-01-16T00:13:11.022 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6691000000000003e-05 2020-01-16T00:13:11.022 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.022 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.022 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:11.022 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.257980+0000", 2020-01-16T00:13:11.022 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010850935000000001 2020-01-16T00:13:11.022 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.023 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.023 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:11.023 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.268831+0000", 2020-01-16T00:13:11.023 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000109475 2020-01-16T00:13:11.023 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.023 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.023 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:11.024 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.268941+0000", 2020-01-16T00:13:11.024 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0023916050000000002 2020-01-16T00:13:11.024 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.024 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.024 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:11.024 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.271332+0000", 2020-01-16T00:13:11.024 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000108996 2020-01-16T00:13:11.024 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.025 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.025 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:11.025 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.271441+0000", 2020-01-16T00:13:11.025 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012444600000000001 2020-01-16T00:13:11.025 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.025 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.025 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:11.026 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.271566+0000", 2020-01-16T00:13:11.026 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014663150999999999 2020-01-16T00:13:11.026 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.026 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:11.026 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.026 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.027 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:11.027 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:11.029 INFO:tasks.rados.rados.0.smithi123.stdout:1078: writing smithi12313934-27 from 1686814 to 2418914 tid 2 2020-01-16T00:13:11.029 INFO:tasks.rados.rados.0.smithi123.stdout:1079: read oid 9 snap -1 2020-01-16T00:13:11.030 INFO:tasks.rados.rados.0.smithi123.stdout:1079: expect (ObjNum 259 snap 0 seq_num 259) 2020-01-16T00:13:11.030 INFO:tasks.rados.rados.0.smithi123.stdout:1080: write (excl) oid 49 current snap is 0 2020-01-16T00:13:11.030 INFO:tasks.rados.rados.0.smithi123.stdout:1080: seq_num 289 ranges {457223=568375,1461649=475758,2420186=437037} 2020-01-16T00:13:11.030 INFO:tasks.rados.rados.0.smithi123.stdout:1080: writing smithi12313934-49 from 457223 to 1025598 tid 1 2020-01-16T00:13:11.030 INFO:tasks.rados.rados.0.smithi123.stdout:1080: writing smithi12313934-49 from 1461649 to 1937407 tid 2 2020-01-16T00:13:11.030 INFO:tasks.rados.rados.0.smithi123.stdout:1080: writing smithi12313934-49 from 2420186 to 2857223 tid 3 2020-01-16T00:13:11.030 INFO:tasks.rados.rados.0.smithi123.stdout:1081: read oid 22 snap -1 2020-01-16T00:13:11.030 INFO:tasks.rados.rados.0.smithi123.stdout:1081: expect (ObjNum 214 snap 0 seq_num 214) 2020-01-16T00:13:11.031 INFO:tasks.rados.rados.0.smithi123.stdout:1082: delete oid 35 current snap is 0 2020-01-16T00:13:11.031 INFO:tasks.rados.rados.0.smithi123.stdout:1071: expect (ObjNum 188 snap 0 seq_num 188) 2020-01-16T00:13:11.031 INFO:tasks.rados.rados.0.smithi123.stdout:1070: expect (ObjNum 263 snap 0 seq_num 263) 2020-01-16T00:13:11.031 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-01-16T00:13:11.032 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:11.022+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:11.053 INFO:tasks.rados.rados.0.smithi123.stdout:1069: expect (ObjNum 177 snap 0 seq_num 177) 2020-01-16T00:13:11.129 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:11.124+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:11.159 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:11.159 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:11.159 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:13:11.159 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:11.168 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-01-16T00:13:11.230 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:11.226+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:11.267 INFO:tasks.rados.rados.0.smithi123.stdout:1072: finishing write tid 1 to smithi12313934-6 2020-01-16T00:13:11.267 INFO:tasks.rados.rados.0.smithi123.stdout:1068: finishing write tid 2 to smithi12313934-42 2020-01-16T00:13:11.267 INFO:tasks.rados.rados.0.smithi123.stdout:1068: finishing write tid 4 to smithi12313934-42 2020-01-16T00:13:11.267 INFO:tasks.rados.rados.0.smithi123.stdout:1068: finishing write tid 5 to smithi12313934-42 2020-01-16T00:13:11.267 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 42 v 107 (ObjNum 282 snap 0 seq_num 282) dirty exists 2020-01-16T00:13:11.268 INFO:tasks.rados.rados.0.smithi123.stdout:1068: left oid 42 (ObjNum 282 snap 0 seq_num 282) 2020-01-16T00:13:11.268 INFO:tasks.rados.rados.0.smithi123.stdout:1072: finishing write tid 2 to smithi12313934-6 2020-01-16T00:13:11.268 INFO:tasks.rados.rados.0.smithi123.stdout:1072: finishing write tid 4 to smithi12313934-6 2020-01-16T00:13:11.268 INFO:tasks.rados.rados.0.smithi123.stdout:1072: finishing write tid 5 to smithi12313934-6 2020-01-16T00:13:11.268 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 6 v 83 (ObjNum 283 snap 0 seq_num 283) dirty exists 2020-01-16T00:13:11.268 INFO:tasks.rados.rados.0.smithi123.stdout:1072: left oid 6 (ObjNum 283 snap 0 seq_num 283) 2020-01-16T00:13:11.268 INFO:tasks.rados.rados.0.smithi123.stdout:1073: finishing write tid 1 to smithi12313934-16 2020-01-16T00:13:11.269 INFO:tasks.rados.rados.0.smithi123.stdout:1073: finishing write tid 2 to smithi12313934-16 2020-01-16T00:13:11.269 INFO:tasks.rados.rados.0.smithi123.stdout:1068: done (14 left) 2020-01-16T00:13:11.269 INFO:tasks.rados.rados.0.smithi123.stdout:1069: done (13 left) 2020-01-16T00:13:11.269 INFO:tasks.rados.rados.0.smithi123.stdout:1070: done (12 left) 2020-01-16T00:13:11.269 INFO:tasks.rados.rados.0.smithi123.stdout:1071: done (11 left) 2020-01-16T00:13:11.269 INFO:tasks.rados.rados.0.smithi123.stdout:1072: done (10 left) 2020-01-16T00:13:11.269 INFO:tasks.rados.rados.0.smithi123.stdout:1082: done (9 left) 2020-01-16T00:13:11.269 INFO:tasks.rados.rados.0.smithi123.stdout:1083: read oid 20 snap -1 2020-01-16T00:13:11.270 INFO:tasks.rados.rados.0.smithi123.stdout:1083: expect (ObjNum 276 snap 0 seq_num 276) 2020-01-16T00:13:11.270 INFO:tasks.rados.rados.0.smithi123.stdout:1084: read oid 40 snap -1 2020-01-16T00:13:11.270 INFO:tasks.rados.rados.0.smithi123.stdout:1084: expect (ObjNum 224 snap 0 seq_num 224) 2020-01-16T00:13:11.270 INFO:tasks.rados.rados.0.smithi123.stdout:1085: write oid 2 current snap is 0 2020-01-16T00:13:11.270 INFO:tasks.rados.rados.0.smithi123.stdout:1085: seq_num 291 ranges {490837=718873,1862260=479427,2875032=15805} 2020-01-16T00:13:11.270 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:11.271 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:11.271 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:13:11.271 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:13:11.271 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:11.280 INFO:tasks.rados.rados.0.smithi123.stdout:1085: writing smithi12313934-2 from 490837 to 1209710 tid 1 2020-01-16T00:13:11.281 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-01-16T00:13:11.288 INFO:tasks.rados.rados.0.smithi123.stdout:1085: writing smithi12313934-2 from 1862260 to 2341687 tid 2 2020-01-16T00:13:11.290 INFO:tasks.rados.rados.0.smithi123.stdout:1085: writing smithi12313934-2 from 2875032 to 2890837 tid 3 2020-01-16T00:13:11.290 INFO:tasks.rados.rados.0.smithi123.stdout:1086: write (excl) oid 35 current snap is 0 2020-01-16T00:13:11.290 INFO:tasks.rados.rados.0.smithi123.stdout:1086: seq_num 292 ranges {507644=794122,1920742=623085} 2020-01-16T00:13:11.297 INFO:tasks.rados.rados.0.smithi123.stdout:1086: writing smithi12313934-35 from 507644 to 1301766 tid 1 2020-01-16T00:13:11.305 INFO:tasks.rados.rados.0.smithi123.stdout:1086: writing smithi12313934-35 from 1920742 to 2543827 tid 2 2020-01-16T00:13:11.305 INFO:tasks.rados.rados.0.smithi123.stdout:1073: finishing write tid 4 to smithi12313934-16 2020-01-16T00:13:11.305 INFO:tasks.rados.rados.0.smithi123.stdout:1087: read oid 23 snap -1 2020-01-16T00:13:11.306 INFO:tasks.rados.rados.0.smithi123.stdout:1087: expect (ObjNum 227 snap 0 seq_num 227) 2020-01-16T00:13:11.306 INFO:tasks.rados.rados.0.smithi123.stdout:1088: read oid 13 snap -1 2020-01-16T00:13:11.306 INFO:tasks.rados.rados.0.smithi123.stdout:1088: expect (ObjNum 272 snap 0 seq_num 272) 2020-01-16T00:13:11.306 INFO:tasks.rados.rados.0.smithi123.stdout:1089: write oid 41 current snap is 0 2020-01-16T00:13:11.306 INFO:tasks.rados.rados.0.smithi123.stdout:1089: seq_num 293 ranges {524451=469371,1579224=483096,2529878=394573} 2020-01-16T00:13:11.311 INFO:tasks.rados.rados.0.smithi123.stdout:1089: writing smithi12313934-41 from 524451 to 993822 tid 1 2020-01-16T00:13:11.319 INFO:tasks.rados.rados.0.smithi123.stdout:1089: writing smithi12313934-41 from 1579224 to 2062320 tid 2 2020-01-16T00:13:11.324 INFO:tasks.rados.rados.0.smithi123.stdout:1089: writing smithi12313934-41 from 2529878 to 2924451 tid 3 2020-01-16T00:13:11.324 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:11.324 INFO:tasks.rados.rados.0.smithi123.stdout:1073: finishing write tid 5 to smithi12313934-16 2020-01-16T00:13:11.324 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 16 v 159 (ObjNum 284 snap 0 seq_num 284) dirty exists 2020-01-16T00:13:11.324 INFO:tasks.rados.rados.0.smithi123.stdout:1073: left oid 16 (ObjNum 284 snap 0 seq_num 284) 2020-01-16T00:13:11.324 INFO:tasks.rados.rados.0.smithi123.stdout:1074: finishing write tid 1 to smithi12313934-37 2020-01-16T00:13:11.325 INFO:tasks.rados.rados.0.smithi123.stdout:1074: finishing write tid 2 to smithi12313934-37 2020-01-16T00:13:11.325 INFO:tasks.rados.rados.0.smithi123.stdout:1073: done (15 left) 2020-01-16T00:13:11.325 INFO:tasks.rados.rados.0.smithi123.stdout:1090: read oid 18 snap -1 2020-01-16T00:13:11.325 INFO:tasks.rados.rados.0.smithi123.stdout:1090: expect (ObjNum 220 snap 0 seq_num 220) 2020-01-16T00:13:11.325 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:11.325 INFO:tasks.rados.rados.0.smithi123.stdout:1074: finishing write tid 4 to smithi12313934-37 2020-01-16T00:13:11.325 INFO:tasks.rados.rados.0.smithi123.stdout:1074: finishing write tid 5 to smithi12313934-37 2020-01-16T00:13:11.326 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 37 v 211 (ObjNum 285 snap 0 seq_num 285) dirty exists 2020-01-16T00:13:11.326 INFO:tasks.rados.rados.0.smithi123.stdout:1074: left oid 37 (ObjNum 285 snap 0 seq_num 285) 2020-01-16T00:13:11.326 INFO:tasks.rados.rados.0.smithi123.stdout:1075: finishing write tid 1 to smithi12313934-31 2020-01-16T00:13:11.326 INFO:tasks.rados.rados.0.smithi123.stdout:1074: done (15 left) 2020-01-16T00:13:11.326 INFO:tasks.rados.rados.0.smithi123.stdout:1091: read oid 12 snap -1 2020-01-16T00:13:11.326 INFO:tasks.rados.rados.0.smithi123.stdout:1091: expect (ObjNum 221 snap 0 seq_num 221) 2020-01-16T00:13:11.326 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:11.326 INFO:tasks.rados.rados.0.smithi123.stdout:1075: finishing write tid 2 to smithi12313934-31 2020-01-16T00:13:11.327 INFO:tasks.rados.rados.0.smithi123.stdout:1075: finishing write tid 4 to smithi12313934-31 2020-01-16T00:13:11.327 INFO:tasks.rados.rados.0.smithi123.stdout:1075: finishing write tid 5 to smithi12313934-31 2020-01-16T00:13:11.327 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 31 v 115 (ObjNum 286 snap 0 seq_num 286) dirty exists 2020-01-16T00:13:11.327 INFO:tasks.rados.rados.0.smithi123.stdout:1075: left oid 31 (ObjNum 286 snap 0 seq_num 286) 2020-01-16T00:13:11.327 INFO:tasks.rados.rados.0.smithi123.stdout:1076: expect (ObjNum 254 snap 0 seq_num 254) 2020-01-16T00:13:11.331 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:13:11.324+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:13:11.401 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:11.402 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:13:11.402 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:13:11.402 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:13:11.402 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.402 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.5.13:36 2.0 e24/19 2:20e40cec:::smithi12313934-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 24'16, mlcod=23'14)", 2020-01-16T00:13:11.402 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.834879+0000", 2020-01-16T00:13:11.403 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 33.558155640999999, 2020-01-16T00:13:11.403 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0075184850000000001, 2020-01-16T00:13:11.403 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:11.403 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:11.403 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:11.403 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.403 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:11.404 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.834879+0000", 2020-01-16T00:13:11.404 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:11.404 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.404 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.404 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:11.404 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.834879+0000", 2020-01-16T00:13:11.404 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:11.405 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.405 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.405 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:11.405 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.834877+0000", 2020-01-16T00:13:11.405 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031393089999999998 2020-01-16T00:13:11.405 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.405 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.405 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:11.406 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.838016+0000", 2020-01-16T00:13:11.406 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2144e-05 2020-01-16T00:13:11.406 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.406 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.406 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:11.406 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.838038+0000", 2020-01-16T00:13:11.406 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2058e-05 2020-01-16T00:13:11.407 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.407 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.407 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:11.407 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.838050+0000", 2020-01-16T00:13:11.407 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8533000000000003e-05 2020-01-16T00:13:11.407 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.407 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.407 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:11.408 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.838119+0000", 2020-01-16T00:13:11.408 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3591000000000003e-05 2020-01-16T00:13:11.408 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.408 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.408 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:11.408 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.838162+0000", 2020-01-16T00:13:11.408 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041910380000000002 2020-01-16T00:13:11.410 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.410 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.410 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:11.410 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.842353+0000", 2020-01-16T00:13:11.411 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4433e-05 2020-01-16T00:13:11.411 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.411 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.411 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:11.411 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.842398+0000", 2020-01-16T00:13:11.412 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0075184850000000001 2020-01-16T00:13:11.412 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.412 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:11.412 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.412 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.412 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.412 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.5.13:40 2.0 e24/19 2:20e40cec:::smithi12313934-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 24'17, mlcod=23'15)", 2020-01-16T00:13:11.412 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.849607+0000", 2020-01-16T00:13:11.413 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 33.543427516000001, 2020-01-16T00:13:11.413 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083979039999999994, 2020-01-16T00:13:11.413 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:11.413 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:11.413 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:11.413 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.413 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:11.413 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.849607+0000", 2020-01-16T00:13:11.414 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:11.414 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.414 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.414 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:11.414 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.849607+0000", 2020-01-16T00:13:11.414 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:11.414 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.414 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.415 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:11.415 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.849605+0000", 2020-01-16T00:13:11.415 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0037534560000000001 2020-01-16T00:13:11.415 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.415 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.415 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:11.415 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.853358+0000", 2020-01-16T00:13:11.415 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9181000000000002e-05 2020-01-16T00:13:11.416 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.416 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.416 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:11.416 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.853377+0000", 2020-01-16T00:13:11.416 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0040000000000002e-06 2020-01-16T00:13:11.416 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.416 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.416 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:11.417 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.853384+0000", 2020-01-16T00:13:11.417 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2583000000000001e-05 2020-01-16T00:13:11.417 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.417 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.417 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:11.417 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.853437+0000", 2020-01-16T00:13:11.417 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8212999999999999e-05 2020-01-16T00:13:11.418 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.418 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.418 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:11.418 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.853465+0000", 2020-01-16T00:13:11.418 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0044847589999999996 2020-01-16T00:13:11.418 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.418 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.418 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:11.419 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.857950+0000", 2020-01-16T00:13:11.419 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5309999999999997e-05 2020-01-16T00:13:11.419 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.419 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.419 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:11.419 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.858005+0000", 2020-01-16T00:13:11.420 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083979039999999994 2020-01-16T00:13:11.420 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.420 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:11.420 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.420 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.420 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.420 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.5.13:42 2.0 e24/19 2:313458df:::smithi12313934-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 24'18, mlcod=24'16)", 2020-01-16T00:13:11.421 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.940683+0000", 2020-01-16T00:13:11.421 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 33.452351677000003, 2020-01-16T00:13:11.421 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0097026960000000002, 2020-01-16T00:13:11.421 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:11.421 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:11.421 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:11.421 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.421 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:11.422 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.940683+0000", 2020-01-16T00:13:11.422 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:11.422 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.422 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.422 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:11.423 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.940683+0000", 2020-01-16T00:13:11.423 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:11.423 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.423 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.423 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:11.423 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.940680+0000", 2020-01-16T00:13:11.424 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003126775 2020-01-16T00:13:11.424 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.424 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.424 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:11.424 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.943807+0000", 2020-01-16T00:13:11.424 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5426999999999999e-05 2020-01-16T00:13:11.424 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.425 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.425 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:11.425 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.943833+0000", 2020-01-16T00:13:11.425 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7409999999999998e-06 2020-01-16T00:13:11.425 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.426 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.427 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:11.427 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.943841+0000", 2020-01-16T00:13:11.427 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011645399999999999 2020-01-16T00:13:11.427 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.427 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.427 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:11.427 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.943958+0000", 2020-01-16T00:13:11.427 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7291000000000003e-05 2020-01-16T00:13:11.428 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.428 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.428 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:11.428 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.944025+0000", 2020-01-16T00:13:11.428 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063108449999999998 2020-01-16T00:13:11.428 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.428 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.428 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:11.429 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.950336+0000", 2020-01-16T00:13:11.429 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9874999999999999e-05 2020-01-16T00:13:11.429 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.429 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.429 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:11.429 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.950386+0000", 2020-01-16T00:13:11.429 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0097026960000000002 2020-01-16T00:13:11.429 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.430 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:11.430 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.430 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.430 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.430 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1387 3.1 3:8c6a83c0:::smithi12313934-36:head [read 0~3579858,checksum] snapc 0=[] ondisk+read+known_if_redirected e28)", 2020-01-16T00:13:11.430 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:46.605159+0000", 2020-01-16T00:13:11.431 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 24.787875380999999, 2020-01-16T00:13:11.431 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0074537709999999997, 2020-01-16T00:13:11.431 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:11.431 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:11.431 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:11.431 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:11.431 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:11.431 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1387 2020-01-16T00:13:11.432 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.432 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:11.432 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.432 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:11.432 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605159+0000", 2020-01-16T00:13:11.432 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:11.432 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.432 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.433 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:11.433 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605159+0000", 2020-01-16T00:13:11.433 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:11.433 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.433 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.433 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:11.433 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605158+0000", 2020-01-16T00:13:11.433 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3860000000000001e-06 2020-01-16T00:13:11.434 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.434 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.434 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:11.434 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605166+0000", 2020-01-16T00:13:11.434 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2139e-05 2020-01-16T00:13:11.434 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.434 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.434 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:11.435 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605178+0000", 2020-01-16T00:13:11.435 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8270000000000003e-06 2020-01-16T00:13:11.435 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.435 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.436 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:11.436 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605184+0000", 2020-01-16T00:13:11.436 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0045873850000000002 2020-01-16T00:13:11.436 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.436 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.437 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:11.437 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.609771+0000", 2020-01-16T00:13:11.437 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000202387 2020-01-16T00:13:11.437 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.437 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.437 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:11.437 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.609973+0000", 2020-01-16T00:13:11.437 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0026399079999999998 2020-01-16T00:13:11.438 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.438 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.438 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:11.438 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.612613+0000", 2020-01-16T00:13:11.438 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0074537709999999997 2020-01-16T00:13:11.438 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.438 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:11.439 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.439 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.439 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.439 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1388 3.1 3:8c6a83c0:::smithi12313934-36:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e28)", 2020-01-16T00:13:11.439 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:46.605253+0000", 2020-01-16T00:13:11.439 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 24.787782326999999, 2020-01-16T00:13:11.439 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0080920829999999999, 2020-01-16T00:13:11.440 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:11.440 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:11.440 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:11.440 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:11.440 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:11.440 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1388 2020-01-16T00:13:11.440 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.440 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:11.441 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.441 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:11.441 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605253+0000", 2020-01-16T00:13:11.441 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:11.441 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.441 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.441 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:11.441 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605253+0000", 2020-01-16T00:13:11.442 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:11.442 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.442 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.442 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:11.442 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605252+0000", 2020-01-16T00:13:11.442 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6559999999999997e-06 2020-01-16T00:13:11.442 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.442 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.443 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:11.443 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605257+0000", 2020-01-16T00:13:11.443 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0966e-05 2020-01-16T00:13:11.443 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.443 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.443 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:11.443 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605268+0000", 2020-01-16T00:13:11.443 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5960000000000003e-06 2020-01-16T00:13:11.444 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.444 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.444 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:11.444 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605274+0000", 2020-01-16T00:13:11.444 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.007370176 2020-01-16T00:13:11.444 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.444 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.444 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:11.446 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.612644+0000", 2020-01-16T00:13:11.446 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000113146 2020-01-16T00:13:11.446 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.447 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.447 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:11.447 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.612757+0000", 2020-01-16T00:13:11.447 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00058744799999999996 2020-01-16T00:13:11.447 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.447 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.447 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:11.448 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.613345+0000", 2020-01-16T00:13:11.448 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0080920829999999999 2020-01-16T00:13:11.448 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.448 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:11.448 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.448 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.448 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.448 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1644 3.1 3:8c6a83c0:::smithi12313934-36:head [read 0~2705927,checksum] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:13:11.449 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847556+0000", 2020-01-16T00:13:11.449 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 19.54547891, 2020-01-16T00:13:11.449 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0081843639999999995, 2020-01-16T00:13:11.449 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:11.449 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:11.449 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:11.449 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:11.449 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:11.450 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1644 2020-01-16T00:13:11.450 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.450 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:11.450 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.450 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:11.450 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847556+0000", 2020-01-16T00:13:11.450 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:11.450 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.451 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.451 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:11.451 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847556+0000", 2020-01-16T00:13:11.451 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:11.451 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.451 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.451 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:11.451 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847555+0000", 2020-01-16T00:13:11.452 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4640000000000002e-06 2020-01-16T00:13:11.452 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.452 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.452 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:11.452 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847559+0000", 2020-01-16T00:13:11.452 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4580000000000003e-06 2020-01-16T00:13:11.452 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.452 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.453 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:11.453 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847563+0000", 2020-01-16T00:13:11.453 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.613e-06 2020-01-16T00:13:11.453 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.453 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.453 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:11.453 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847567+0000", 2020-01-16T00:13:11.453 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000145045 2020-01-16T00:13:11.454 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.454 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.454 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:11.454 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847712+0000", 2020-01-16T00:13:11.454 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6480000000000003e-06 2020-01-16T00:13:11.454 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.454 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.455 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:13:11.455 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847720+0000", 2020-01-16T00:13:11.455 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0055645269999999997 2020-01-16T00:13:11.455 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.455 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.455 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:11.455 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.853284+0000", 2020-01-16T00:13:11.456 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00026767400000000003 2020-01-16T00:13:11.456 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.456 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.456 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:11.456 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.853552+0000", 2020-01-16T00:13:11.456 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021884169999999998 2020-01-16T00:13:11.456 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.456 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.457 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:11.457 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.855740+0000", 2020-01-16T00:13:11.457 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0081843639999999995 2020-01-16T00:13:11.457 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.457 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:11.457 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.457 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.457 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.458 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1645 3.1 3:8c6a83c0:::smithi12313934-36:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:13:11.458 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847602+0000", 2020-01-16T00:13:11.458 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 19.545433111000001, 2020-01-16T00:13:11.458 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0085169679999999998, 2020-01-16T00:13:11.458 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:11.458 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:11.458 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:11.459 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:11.459 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:11.459 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1645 2020-01-16T00:13:11.459 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.459 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:11.459 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.459 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:11.459 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847602+0000", 2020-01-16T00:13:11.460 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:11.460 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.460 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.460 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:11.460 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847602+0000", 2020-01-16T00:13:11.460 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:11.460 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.461 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.461 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:11.461 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847601+0000", 2020-01-16T00:13:11.461 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7011e-05 2020-01-16T00:13:11.461 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.461 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.461 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:11.462 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847618+0000", 2020-01-16T00:13:11.462 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.187e-06 2020-01-16T00:13:11.462 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.462 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.462 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:11.462 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847625+0000", 2020-01-16T00:13:11.462 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9879999999999998e-06 2020-01-16T00:13:11.462 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.463 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.463 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:11.463 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847629+0000", 2020-01-16T00:13:11.463 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000129694 2020-01-16T00:13:11.463 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.463 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.463 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:11.463 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847759+0000", 2020-01-16T00:13:11.464 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8379999999999996e-06 2020-01-16T00:13:11.464 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.464 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.464 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:13:11.464 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847766+0000", 2020-01-16T00:13:11.464 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0079977390000000002 2020-01-16T00:13:11.464 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.464 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.465 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:11.465 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.855764+0000", 2020-01-16T00:13:11.465 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2467999999999996e-05 2020-01-16T00:13:11.465 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.465 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.465 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:11.465 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.855856+0000", 2020-01-16T00:13:11.466 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00026275099999999998 2020-01-16T00:13:11.466 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.466 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.466 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:11.466 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.856119+0000", 2020-01-16T00:13:11.466 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0085169679999999998 2020-01-16T00:13:11.467 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.467 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:11.467 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.467 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.467 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.467 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1649 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~2084068,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:13:11.468 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847683+0000", 2020-01-16T00:13:11.468 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 19.54535186, 2020-01-16T00:13:11.468 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010654691000000001, 2020-01-16T00:13:11.468 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:11.468 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:11.468 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:11.469 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:11.469 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:11.469 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1649 2020-01-16T00:13:11.469 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.469 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:11.469 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.470 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:11.470 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847683+0000", 2020-01-16T00:13:11.470 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:11.470 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.470 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.470 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:11.471 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847683+0000", 2020-01-16T00:13:11.471 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:11.471 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.471 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.472 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:11.472 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847682+0000", 2020-01-16T00:13:11.472 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8569999999999997e-06 2020-01-16T00:13:11.472 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.472 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.472 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:11.472 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847689+0000", 2020-01-16T00:13:11.473 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8090000000000004e-06 2020-01-16T00:13:11.473 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.473 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.473 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:11.473 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847696+0000", 2020-01-16T00:13:11.473 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2690000000000001e-06 2020-01-16T00:13:11.473 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.474 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.474 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:11.474 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847700+0000", 2020-01-16T00:13:11.474 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8189999999999993e-05 2020-01-16T00:13:11.474 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.474 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.474 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:11.474 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847798+0000", 2020-01-16T00:13:11.475 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7129999999999999e-06 2020-01-16T00:13:11.475 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.475 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.475 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:13:11.475 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847802+0000", 2020-01-16T00:13:11.475 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083913049999999999 2020-01-16T00:13:11.475 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.475 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.476 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:11.476 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.856193+0000", 2020-01-16T00:13:11.476 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034400300000000001 2020-01-16T00:13:11.476 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.476 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.476 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:11.476 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.856537+0000", 2020-01-16T00:13:11.477 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018006820000000001 2020-01-16T00:13:11.477 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.477 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.477 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:11.477 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858338+0000", 2020-01-16T00:13:11.477 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010654691000000001 2020-01-16T00:13:11.477 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.477 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:11.478 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.478 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.478 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.478 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1650 3.1 3:8de1d8a3:::smithi12313934-13:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:13:11.478 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847717+0000", 2020-01-16T00:13:11.478 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 19.545317907000001, 2020-01-16T00:13:11.478 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010913658, 2020-01-16T00:13:11.478 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:11.479 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:11.479 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:11.479 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:11.479 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:11.479 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1650 2020-01-16T00:13:11.479 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.479 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:11.479 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.480 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:11.480 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847717+0000", 2020-01-16T00:13:11.480 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:11.480 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.480 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.480 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:11.480 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847717+0000", 2020-01-16T00:13:11.480 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:11.481 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.481 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.481 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:11.481 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847716+0000", 2020-01-16T00:13:11.481 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.433e-06 2020-01-16T00:13:11.481 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.481 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.482 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:11.482 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847720+0000", 2020-01-16T00:13:11.482 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1800000000000004e-06 2020-01-16T00:13:11.482 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.482 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.482 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:11.482 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847725+0000", 2020-01-16T00:13:11.482 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6689999999999997e-06 2020-01-16T00:13:11.483 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.483 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.483 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:11.483 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847730+0000", 2020-01-16T00:13:11.483 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000102468 2020-01-16T00:13:11.483 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.483 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.483 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:11.484 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847832+0000", 2020-01-16T00:13:11.484 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3999999999999997e-06 2020-01-16T00:13:11.484 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.484 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.484 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:13:11.484 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847838+0000", 2020-01-16T00:13:11.484 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010519395000000001 2020-01-16T00:13:11.484 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.485 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.485 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:11.485 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858358+0000", 2020-01-16T00:13:11.485 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7600000000000002e-05 2020-01-16T00:13:11.485 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.485 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.485 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:11.486 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858445+0000", 2020-01-16T00:13:11.486 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018515900000000001 2020-01-16T00:13:11.486 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.486 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.486 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:11.486 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858631+0000", 2020-01-16T00:13:11.486 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010913658 2020-01-16T00:13:11.486 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.487 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:11.487 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.487 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.487 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.487 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1651 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~2084068,checksum] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:13:11.487 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847752+0000", 2020-01-16T00:13:11.487 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 19.545283291000001, 2020-01-16T00:13:11.487 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011734249, 2020-01-16T00:13:11.488 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:11.488 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:11.488 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:11.488 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:11.488 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:11.488 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1651 2020-01-16T00:13:11.488 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.488 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:11.489 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.489 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:11.489 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847752+0000", 2020-01-16T00:13:11.489 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:11.489 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.489 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.489 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:11.489 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847752+0000", 2020-01-16T00:13:11.490 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:11.490 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.490 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.490 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:11.490 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847751+0000", 2020-01-16T00:13:11.490 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1980000000000001e-06 2020-01-16T00:13:11.491 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.491 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.491 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:11.491 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847754+0000", 2020-01-16T00:13:11.491 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6679999999999999e-06 2020-01-16T00:13:11.491 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.491 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.491 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:11.492 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847758+0000", 2020-01-16T00:13:11.492 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2500000000000001e-06 2020-01-16T00:13:11.492 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.492 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.492 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:11.493 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847760+0000", 2020-01-16T00:13:11.493 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7614000000000004e-05 2020-01-16T00:13:11.493 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.493 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.493 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:11.493 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847858+0000", 2020-01-16T00:13:11.493 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7950000000000004e-06 2020-01-16T00:13:11.493 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.494 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.494 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:13:11.494 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847864+0000", 2020-01-16T00:13:11.494 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010790345 2020-01-16T00:13:11.494 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.494 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.494 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:11.494 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858654+0000", 2020-01-16T00:13:11.495 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000141466 2020-01-16T00:13:11.495 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.495 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.495 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:11.495 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858795+0000", 2020-01-16T00:13:11.495 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00069037899999999997 2020-01-16T00:13:11.495 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.496 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.496 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:11.496 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.859486+0000", 2020-01-16T00:13:11.496 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011734249 2020-01-16T00:13:11.496 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.497 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:11.497 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.497 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.497 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.497 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:55 2.1 e30/19 2:91f85463:::smithi12313934-24 low_tier:head v 30'56, mlcod=29'54)", 2020-01-16T00:13:11.497 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.738134+0000", 2020-01-16T00:13:11.497 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 18.654901004999999, 2020-01-16T00:13:11.498 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0086509570000000008, 2020-01-16T00:13:11.498 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:11.498 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:11.498 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:11.498 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.498 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:11.498 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738134+0000", 2020-01-16T00:13:11.498 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:11.499 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.499 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.499 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:11.499 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738134+0000", 2020-01-16T00:13:11.499 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:11.499 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.499 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.500 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:11.500 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738134+0000", 2020-01-16T00:13:11.500 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00053712599999999997 2020-01-16T00:13:11.500 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.500 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.500 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:11.500 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738671+0000", 2020-01-16T00:13:11.500 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4059999999999999e-05 2020-01-16T00:13:11.501 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.501 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.501 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:11.501 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738685+0000", 2020-01-16T00:13:11.501 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7680000000000008e-06 2020-01-16T00:13:11.501 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.501 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.501 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:11.502 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738692+0000", 2020-01-16T00:13:11.502 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1273999999999996e-05 2020-01-16T00:13:11.502 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.502 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.502 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:11.502 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738764+0000", 2020-01-16T00:13:11.502 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6827999999999999e-05 2020-01-16T00:13:11.503 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.503 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.503 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:11.503 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738781+0000", 2020-01-16T00:13:11.503 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0079370740000000006 2020-01-16T00:13:11.503 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.503 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.503 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:11.504 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746718+0000", 2020-01-16T00:13:11.504 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7181000000000006e-05 2020-01-16T00:13:11.504 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.504 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.504 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:11.504 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746785+0000", 2020-01-16T00:13:11.504 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0086509570000000008 2020-01-16T00:13:11.505 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.505 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:11.505 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.505 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.505 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.505 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:1668 3.3 e30/20 3:c478bfe8:::smithi12313934-10:head v 30'77, mlcod=29'75)", 2020-01-16T00:13:11.505 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.753756+0000", 2020-01-16T00:13:11.506 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 18.639278954000002, 2020-01-16T00:13:11.506 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0077471110000000001, 2020-01-16T00:13:11.506 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:11.506 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:11.506 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:11.506 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.506 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:11.507 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.753756+0000", 2020-01-16T00:13:11.507 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:11.507 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.507 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.507 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:11.507 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.753756+0000", 2020-01-16T00:13:11.507 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:11.507 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.508 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.508 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:11.508 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.753754+0000", 2020-01-16T00:13:11.508 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00047439299999999998 2020-01-16T00:13:11.508 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.508 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.508 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:11.508 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.754229+0000", 2020-01-16T00:13:11.509 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1018e-05 2020-01-16T00:13:11.509 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.509 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.509 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:11.509 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.754240+0000", 2020-01-16T00:13:11.509 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1700000000000002e-06 2020-01-16T00:13:11.509 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.510 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.510 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:11.510 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.754246+0000", 2020-01-16T00:13:11.510 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0014229519999999999 2020-01-16T00:13:11.510 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.510 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.511 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:11.511 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.755669+0000", 2020-01-16T00:13:11.511 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8136e-05 2020-01-16T00:13:11.511 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.511 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.511 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:11.511 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.755697+0000", 2020-01-16T00:13:11.512 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0057384799999999998 2020-01-16T00:13:11.512 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.512 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.512 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:11.512 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.761435+0000", 2020-01-16T00:13:11.513 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7742000000000003e-05 2020-01-16T00:13:11.513 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.513 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.513 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:11.513 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.761503+0000", 2020-01-16T00:13:11.513 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0077471110000000001 2020-01-16T00:13:11.513 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.513 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:11.514 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.514 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.514 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.514 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2106 3.1 3:8de1d8a3:::smithi12313934-13:head [write 665610~797860,stat] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:13:11.514 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.631369+0000", 2020-01-16T00:13:11.514 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 9.7616655790000006, 2020-01-16T00:13:11.514 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0091195330000000008, 2020-01-16T00:13:11.515 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:11.515 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:11.515 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:11.515 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:11.515 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:11.515 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2106 2020-01-16T00:13:11.515 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.515 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:11.516 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.516 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:11.516 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631369+0000", 2020-01-16T00:13:11.516 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:11.516 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.516 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.516 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:11.516 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631369+0000", 2020-01-16T00:13:11.517 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:11.517 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.517 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.517 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:11.517 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631367+0000", 2020-01-16T00:13:11.517 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00025714100000000001 2020-01-16T00:13:11.517 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.518 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.518 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:11.518 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631624+0000", 2020-01-16T00:13:11.518 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4946000000000001e-05 2020-01-16T00:13:11.518 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.518 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.518 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:11.518 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631639+0000", 2020-01-16T00:13:11.519 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7050000000000002e-06 2020-01-16T00:13:11.519 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.519 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.519 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:11.519 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631646+0000", 2020-01-16T00:13:11.519 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010938900000000001 2020-01-16T00:13:11.519 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.520 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.520 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:11.520 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631755+0000", 2020-01-16T00:13:11.520 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8848e-05 2020-01-16T00:13:11.520 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.521 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.521 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op must wait for map", 2020-01-16T00:13:11.521 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631774+0000", 2020-01-16T00:13:11.521 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0026027730000000001 2020-01-16T00:13:11.521 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.521 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.521 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:11.521 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.634377+0000", 2020-01-16T00:13:11.522 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9282000000000003e-05 2020-01-16T00:13:11.522 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.522 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.522 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:11.522 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.634466+0000", 2020-01-16T00:13:11.522 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000127868 2020-01-16T00:13:11.522 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.522 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.523 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:13:11.523 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.634594+0000", 2020-01-16T00:13:11.523 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030776549999999999 2020-01-16T00:13:11.523 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.523 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.523 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:11.523 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.637672+0000", 2020-01-16T00:13:11.523 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001634195 2020-01-16T00:13:11.524 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.524 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.524 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:11.524 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.639306+0000", 2020-01-16T00:13:11.524 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010810069999999999 2020-01-16T00:13:11.524 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.524 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.525 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:11.525 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640387+0000", 2020-01-16T00:13:11.525 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0543000000000001e-05 2020-01-16T00:13:11.525 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.525 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.526 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:11.526 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640428+0000", 2020-01-16T00:13:11.526 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1321000000000005e-05 2020-01-16T00:13:11.526 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.526 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.526 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:11.527 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640489+0000", 2020-01-16T00:13:11.527 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0091195330000000008 2020-01-16T00:13:11.527 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.527 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:11.527 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.527 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.527 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.527 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2107 3.1 3:8de1d8a3:::smithi12313934-13:head [write 1909329~458405] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:13:11.528 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.636786+0000", 2020-01-16T00:13:11.528 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 9.7562484489999992, 2020-01-16T00:13:11.528 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093182809999999994, 2020-01-16T00:13:11.528 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:11.528 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:11.528 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:11.528 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:11.528 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:11.529 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2107 2020-01-16T00:13:11.529 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.529 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:11.529 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.529 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:11.529 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636786+0000", 2020-01-16T00:13:11.529 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:11.530 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.530 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.530 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:11.530 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636786+0000", 2020-01-16T00:13:11.530 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:11.530 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.530 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.531 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:11.531 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636784+0000", 2020-01-16T00:13:11.531 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014276299999999999 2020-01-16T00:13:11.531 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.531 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.531 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:11.532 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636927+0000", 2020-01-16T00:13:11.532 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5017999999999999e-05 2020-01-16T00:13:11.532 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.532 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.532 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:11.532 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636942+0000", 2020-01-16T00:13:11.532 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5589999999999996e-06 2020-01-16T00:13:11.532 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.533 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.533 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:11.533 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636947+0000", 2020-01-16T00:13:11.533 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00040279699999999999 2020-01-16T00:13:11.533 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.533 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.533 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:11.534 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.637350+0000", 2020-01-16T00:13:11.534 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000233506 2020-01-16T00:13:11.534 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.534 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.534 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:11.534 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.637584+0000", 2020-01-16T00:13:11.534 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029534660000000001 2020-01-16T00:13:11.534 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.535 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.535 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:11.535 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640537+0000", 2020-01-16T00:13:11.535 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012577700000000001 2020-01-16T00:13:11.535 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.535 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.535 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:11.535 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640663+0000", 2020-01-16T00:13:11.536 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000198593 2020-01-16T00:13:11.536 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.536 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.536 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:13:11.536 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640862+0000", 2020-01-16T00:13:11.536 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.005088197 2020-01-16T00:13:11.536 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.537 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.538 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:11.538 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645950+0000", 2020-01-16T00:13:11.538 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2568999999999999e-05 2020-01-16T00:13:11.538 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.538 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.539 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:11.539 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645962+0000", 2020-01-16T00:13:11.539 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3034999999999995e-05 2020-01-16T00:13:11.539 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.539 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.539 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:11.539 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646025+0000", 2020-01-16T00:13:11.540 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7762999999999997e-05 2020-01-16T00:13:11.540 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.541 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.541 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:11.541 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646063+0000", 2020-01-16T00:13:11.541 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1541000000000001e-05 2020-01-16T00:13:11.541 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.541 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.541 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:11.542 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646105+0000", 2020-01-16T00:13:11.542 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093182809999999994 2020-01-16T00:13:11.542 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.542 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:11.542 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.542 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.542 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2108 3.1 3:8de1d8a3:::smithi12313934-13:head [write 2816700~410210] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:13:11.543 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.641861+0000", 2020-01-16T00:13:11.543 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 9.7511741010000001, 2020-01-16T00:13:11.543 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018545604, 2020-01-16T00:13:11.543 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:11.543 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:11.543 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:11.544 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:11.544 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:11.544 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2108 2020-01-16T00:13:11.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.544 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:11.544 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.544 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:11.545 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.641861+0000", 2020-01-16T00:13:11.545 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:11.545 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.545 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.545 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:11.545 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.641861+0000", 2020-01-16T00:13:11.545 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:11.546 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.546 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.546 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:11.546 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.641859+0000", 2020-01-16T00:13:11.546 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018466399999999999 2020-01-16T00:13:11.546 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.546 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.547 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:11.547 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642043+0000", 2020-01-16T00:13:11.547 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2999999999999999e-05 2020-01-16T00:13:11.547 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.547 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.547 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:11.547 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642056+0000", 2020-01-16T00:13:11.548 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0730000000000005e-06 2020-01-16T00:13:11.548 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.548 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.548 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:11.548 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642064+0000", 2020-01-16T00:13:11.548 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9527999999999997e-05 2020-01-16T00:13:11.548 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.549 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.549 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:11.549 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642114+0000", 2020-01-16T00:13:11.549 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036616050000000001 2020-01-16T00:13:11.549 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.549 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.549 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:11.550 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645776+0000", 2020-01-16T00:13:11.550 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012222663999999999 2020-01-16T00:13:11.550 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.550 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.550 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:11.550 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.657998+0000", 2020-01-16T00:13:11.550 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014280299999999999 2020-01-16T00:13:11.551 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.551 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.551 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:11.551 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.658141+0000", 2020-01-16T00:13:11.551 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0001102 2020-01-16T00:13:11.551 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.551 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.552 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:13:11.552 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.658251+0000", 2020-01-16T00:13:11.552 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020421219999999999 2020-01-16T00:13:11.552 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.552 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.552 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:11.552 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660293+0000", 2020-01-16T00:13:11.552 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7399999999999999e-06 2020-01-16T00:13:11.553 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.553 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.553 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:11.553 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660303+0000", 2020-01-16T00:13:11.553 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9873999999999998e-05 2020-01-16T00:13:11.553 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.553 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.554 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:11.554 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660343+0000", 2020-01-16T00:13:11.554 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4448e-05 2020-01-16T00:13:11.554 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.554 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.554 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:11.554 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660377+0000", 2020-01-16T00:13:11.554 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8915999999999999e-05 2020-01-16T00:13:11.555 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.555 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.555 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:11.555 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660406+0000", 2020-01-16T00:13:11.555 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018545604 2020-01-16T00:13:11.555 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.555 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:11.555 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.556 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.556 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.556 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2109 3.1 3:8de1d8a3:::smithi12313934-13:head [setxattr _header (56) in=7b,truncate 3865610] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:13:11.556 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.642075+0000", 2020-01-16T00:13:11.556 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 9.7509595279999992, 2020-01-16T00:13:11.556 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018477655999999999, 2020-01-16T00:13:11.556 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:11.556 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:11.557 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:11.557 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:11.557 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:11.557 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2109 2020-01-16T00:13:11.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.557 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:11.557 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.558 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:11.558 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642075+0000", 2020-01-16T00:13:11.558 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:11.558 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.558 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.558 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:11.558 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642075+0000", 2020-01-16T00:13:11.558 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:11.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.559 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.559 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:11.559 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642074+0000", 2020-01-16T00:13:11.559 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4500000000000003e-06 2020-01-16T00:13:11.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.559 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.559 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:11.560 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642080+0000", 2020-01-16T00:13:11.560 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7229999999999998e-06 2020-01-16T00:13:11.560 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.560 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.560 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:11.560 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642087+0000", 2020-01-16T00:13:11.560 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2390000000000004e-06 2020-01-16T00:13:11.561 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.561 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.561 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:11.561 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642091+0000", 2020-01-16T00:13:11.561 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0037396550000000001 2020-01-16T00:13:11.561 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.561 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.561 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:11.562 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645830+0000", 2020-01-16T00:13:11.562 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2200000000000002e-05 2020-01-16T00:13:11.562 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.562 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.562 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:11.562 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645883+0000", 2020-01-16T00:13:11.562 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013432934000000001 2020-01-16T00:13:11.563 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.563 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.563 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:11.563 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659316+0000", 2020-01-16T00:13:11.563 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000140006 2020-01-16T00:13:11.563 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.563 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.564 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:11.564 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659456+0000", 2020-01-16T00:13:11.564 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000184873 2020-01-16T00:13:11.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.564 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.564 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:13:11.564 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659640+0000", 2020-01-16T00:13:11.564 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00066954999999999998 2020-01-16T00:13:11.565 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.565 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.565 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:11.565 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660310+0000", 2020-01-16T00:13:11.565 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014847699999999999 2020-01-16T00:13:11.565 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.565 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.566 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:11.566 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660459+0000", 2020-01-16T00:13:11.566 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6869e-05 2020-01-16T00:13:11.567 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.567 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.567 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:11.567 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660485+0000", 2020-01-16T00:13:11.567 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8478000000000001e-05 2020-01-16T00:13:11.567 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.567 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.568 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:11.568 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660514+0000", 2020-01-16T00:13:11.568 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9153999999999998e-05 2020-01-16T00:13:11.568 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.568 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.568 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:11.568 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660553+0000", 2020-01-16T00:13:11.568 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018477655999999999 2020-01-16T00:13:11.569 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.569 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:11.569 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.569 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.569 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.569 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2110 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-01-16T00:13:11.569 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.642112+0000", 2020-01-16T00:13:11.570 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 9.75092289, 2020-01-16T00:13:11.570 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018620198000000001, 2020-01-16T00:13:11.570 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:11.570 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:11.571 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:11.571 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:11.571 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:11.571 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2110 2020-01-16T00:13:11.571 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.572 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:11.572 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.572 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:11.572 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642112+0000", 2020-01-16T00:13:11.572 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:11.573 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.573 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.573 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:11.573 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642112+0000", 2020-01-16T00:13:11.573 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:11.573 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.574 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.574 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:11.574 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642111+0000", 2020-01-16T00:13:11.575 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4450000000000001e-06 2020-01-16T00:13:11.575 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.575 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.575 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:11.575 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642117+0000", 2020-01-16T00:13:11.576 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1567e-05 2020-01-16T00:13:11.576 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.577 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.577 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:11.577 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642129+0000", 2020-01-16T00:13:11.577 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.754e-06 2020-01-16T00:13:11.578 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.578 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.578 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:11.578 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642136+0000", 2020-01-16T00:13:11.578 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040239719999999998 2020-01-16T00:13:11.579 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.579 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.579 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:11.579 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646160+0000", 2020-01-16T00:13:11.580 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5600999999999998e-05 2020-01-16T00:13:11.580 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.580 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.580 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:11.581 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646215+0000", 2020-01-16T00:13:11.581 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013733006000000001 2020-01-16T00:13:11.581 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.581 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.582 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:11.582 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659948+0000", 2020-01-16T00:13:11.582 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000106277 2020-01-16T00:13:11.582 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.582 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.583 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:11.583 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660054+0000", 2020-01-16T00:13:11.583 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000524327 2020-01-16T00:13:11.583 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.584 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.584 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:11.584 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660579+0000", 2020-01-16T00:13:11.584 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9841999999999999e-05 2020-01-16T00:13:11.585 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.585 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.585 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:11.585 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660659+0000", 2020-01-16T00:13:11.586 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3572999999999995e-05 2020-01-16T00:13:11.586 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.586 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.587 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:11.587 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660732+0000", 2020-01-16T00:13:11.587 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018620198000000001 2020-01-16T00:13:11.588 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.588 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:11.588 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.588 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.588 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.589 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:70 2.1 e35/19 2:91f85463:::smithi12313934-24 low_tier:head v 35'66, mlcod=34'64)", 2020-01-16T00:13:11.589 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.986684+0000", 2020-01-16T00:13:11.589 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.4063510360000002, 2020-01-16T00:13:11.589 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012287378999999999, 2020-01-16T00:13:11.590 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:11.590 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:11.590 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:11.590 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.590 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:11.591 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986684+0000", 2020-01-16T00:13:11.591 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:11.591 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.591 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.591 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:11.591 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986684+0000", 2020-01-16T00:13:11.591 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:11.592 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.592 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.592 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:11.592 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986683+0000", 2020-01-16T00:13:11.592 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012884179999999999 2020-01-16T00:13:11.593 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.593 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.593 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:11.593 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.987971+0000", 2020-01-16T00:13:11.593 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011558200000000001 2020-01-16T00:13:11.593 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.593 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.594 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:11.594 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988087+0000", 2020-01-16T00:13:11.594 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0472000000000001e-05 2020-01-16T00:13:11.594 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.594 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.594 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:11.594 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988137+0000", 2020-01-16T00:13:11.595 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000107484 2020-01-16T00:13:11.595 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.595 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.595 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:11.595 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988245+0000", 2020-01-16T00:13:11.595 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4409999999999998e-05 2020-01-16T00:13:11.595 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.595 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.596 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:11.596 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988279+0000", 2020-01-16T00:13:11.596 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010641609999999999 2020-01-16T00:13:11.596 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.596 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.596 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:11.596 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.998921+0000", 2020-01-16T00:13:11.596 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0278000000000002e-05 2020-01-16T00:13:11.597 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.597 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.597 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:11.597 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.998971+0000", 2020-01-16T00:13:11.598 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012287378999999999 2020-01-16T00:13:11.598 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.598 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:11.598 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.598 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.598 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.599 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:71 2.1 e35/19 2:91f85463:::smithi12313934-24 low_tier:head v 35'67, mlcod=34'64)", 2020-01-16T00:13:11.599 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.991132+0000", 2020-01-16T00:13:11.599 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.4019030859999999, 2020-01-16T00:13:11.599 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0082681190000000009, 2020-01-16T00:13:11.599 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:11.599 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:11.599 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:11.600 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.600 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:11.600 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.991132+0000", 2020-01-16T00:13:11.600 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:11.600 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.600 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.600 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:11.601 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.991132+0000", 2020-01-16T00:13:11.601 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:11.601 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.601 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.601 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:11.602 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.991130+0000", 2020-01-16T00:13:11.602 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3583000000000003e-05 2020-01-16T00:13:11.602 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.602 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.602 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:11.603 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.991203+0000", 2020-01-16T00:13:11.603 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.1001000000000001e-05 2020-01-16T00:13:11.603 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.603 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.604 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:11.604 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.991224+0000", 2020-01-16T00:13:11.604 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9329999999999996e-06 2020-01-16T00:13:11.604 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.604 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.605 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:11.605 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.991234+0000", 2020-01-16T00:13:11.605 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0035280070000000001 2020-01-16T00:13:11.605 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.606 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.606 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:11.606 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.994762+0000", 2020-01-16T00:13:11.606 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.622e-05 2020-01-16T00:13:11.607 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.607 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.607 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:11.608 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.994788+0000", 2020-01-16T00:13:11.608 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0045652219999999999 2020-01-16T00:13:11.608 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.608 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.609 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:11.609 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999354+0000", 2020-01-16T00:13:11.609 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6396000000000001e-05 2020-01-16T00:13:11.609 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.609 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.610 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:11.610 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999400+0000", 2020-01-16T00:13:11.610 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0082681190000000009 2020-01-16T00:13:11.610 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.611 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:11.611 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.611 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.611 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.611 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:77 2.1 e36/19 2:91f85463:::smithi12313934-24 low_tier:head v 36'69, mlcod=35'67)", 2020-01-16T00:13:11.612 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.556730+0000", 2020-01-16T00:13:11.612 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 3.8363046820000002, 2020-01-16T00:13:11.612 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093007760000000002, 2020-01-16T00:13:11.612 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:11.613 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:11.613 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:11.613 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.613 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:11.614 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556730+0000", 2020-01-16T00:13:11.614 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:11.614 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.614 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.615 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:11.615 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556730+0000", 2020-01-16T00:13:11.615 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:11.615 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.615 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.616 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:11.616 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556728+0000", 2020-01-16T00:13:11.616 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00056509599999999996 2020-01-16T00:13:11.616 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.617 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.617 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:11.617 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.557293+0000", 2020-01-16T00:13:11.617 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0225e-05 2020-01-16T00:13:11.617 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.618 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.618 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:11.618 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.557314+0000", 2020-01-16T00:13:11.619 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8989999999999996e-06 2020-01-16T00:13:11.619 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.619 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.619 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:11.619 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.557323+0000", 2020-01-16T00:13:11.619 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7923000000000002e-05 2020-01-16T00:13:11.620 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.620 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.620 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:11.620 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.557371+0000", 2020-01-16T00:13:11.621 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9963999999999999e-05 2020-01-16T00:13:11.621 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.621 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.621 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:11.621 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.557411+0000", 2020-01-16T00:13:11.621 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0085873000000000008 2020-01-16T00:13:11.621 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.622 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.622 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:11.622 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.565999+0000", 2020-01-16T00:13:11.622 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2326000000000002e-05 2020-01-16T00:13:11.622 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:11.622 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:11.622 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:11.623 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566031+0000", 2020-01-16T00:13:11.623 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093007760000000002 2020-01-16T00:13:11.623 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.623 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:11.623 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.624 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:11.624 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:11.624 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:11.627 INFO:tasks.rados.rados.0.smithi123.stdout:1077: finishing write tid 1 to smithi12313934-43 2020-01-16T00:13:11.627 INFO:tasks.rados.rados.0.smithi123.stdout:1077: finishing write tid 2 to smithi12313934-43 2020-01-16T00:13:11.627 INFO:tasks.rados.rados.0.smithi123.stdout:1077: finishing write tid 3 to smithi12313934-43 2020-01-16T00:13:11.628 INFO:tasks.rados.rados.0.smithi123.stdout:1077: finishing write tid 5 to smithi12313934-43 2020-01-16T00:13:11.628 INFO:tasks.rados.rados.0.smithi123.stdout:1077: finishing write tid 6 to smithi12313934-43 2020-01-16T00:13:11.628 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 43 v 148 (ObjNum 287 snap 0 seq_num 287) dirty exists 2020-01-16T00:13:11.628 INFO:tasks.rados.rados.0.smithi123.stdout:1077: left oid 43 (ObjNum 287 snap 0 seq_num 287) 2020-01-16T00:13:11.628 INFO:tasks.rados.rados.0.smithi123.stdout:1078: finishing write tid 1 to smithi12313934-27 2020-01-16T00:13:11.628 INFO:tasks.rados.rados.0.smithi123.stdout:1078: finishing write tid 2 to smithi12313934-27 2020-01-16T00:13:11.629 INFO:tasks.rados.rados.0.smithi123.stdout:1078: finishing write tid 4 to smithi12313934-27 2020-01-16T00:13:11.629 INFO:tasks.rados.rados.0.smithi123.stdout:1078: finishing write tid 5 to smithi12313934-27 2020-01-16T00:13:11.629 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 27 v 98 (ObjNum 288 snap 0 seq_num 288) dirty exists 2020-01-16T00:13:11.629 INFO:tasks.rados.rados.0.smithi123.stdout:1078: left oid 27 (ObjNum 288 snap 0 seq_num 288) 2020-01-16T00:13:11.629 INFO:tasks.rados.rados.0.smithi123.stdout:1080: finishing write tid 1 to smithi12313934-49 2020-01-16T00:13:11.629 INFO:tasks.rados.rados.0.smithi123.stdout:1079: expect (ObjNum 259 snap 0 seq_num 259) 2020-01-16T00:13:11.629 INFO:tasks.rados.rados.0.smithi123.stdout:1080: finishing write tid 2 to smithi12313934-49 2020-01-16T00:13:11.630 INFO:tasks.rados.rados.0.smithi123.stdout:1080: finishing write tid 3 to smithi12313934-49 2020-01-16T00:13:11.630 INFO:tasks.rados.rados.0.smithi123.stdout:1080: finishing write tid 5 to smithi12313934-49 2020-01-16T00:13:11.630 INFO:tasks.rados.rados.0.smithi123.stdout:1075: done (15 left) 2020-01-16T00:13:11.630 INFO:tasks.rados.rados.0.smithi123.stdout:1076: done (14 left) 2020-01-16T00:13:11.630 INFO:tasks.rados.rados.0.smithi123.stdout:1077: done (13 left) 2020-01-16T00:13:11.630 INFO:tasks.rados.rados.0.smithi123.stdout:1078: done (12 left) 2020-01-16T00:13:11.630 INFO:tasks.rados.rados.0.smithi123.stdout:1079: done (11 left) 2020-01-16T00:13:11.630 INFO:tasks.rados.rados.0.smithi123.stdout:1092: write oid 30 current snap is 0 2020-01-16T00:13:11.631 INFO:tasks.rados.rados.0.smithi123.stdout:1092: seq_num 294 ranges {541258=544620,1637706=626754,2840948=100310} 2020-01-16T00:13:11.631 INFO:tasks.rados.rados.0.smithi123.stdout:1092: writing smithi12313934-30 from 541258 to 1085878 tid 1 2020-01-16T00:13:11.631 INFO:tasks.rados.rados.0.smithi123.stdout:1092: writing smithi12313934-30 from 1637706 to 2264460 tid 2 2020-01-16T00:13:11.631 INFO:tasks.rados.rados.0.smithi123.stdout:1092: writing smithi12313934-30 from 2840948 to 2941258 tid 3 2020-01-16T00:13:11.631 INFO:tasks.rados.rados.0.smithi123.stdout:1093: write oid 43 current snap is 0 2020-01-16T00:13:11.631 INFO:tasks.rados.rados.0.smithi123.stdout:1093: seq_num 295 ranges {558065=619869,1579835=486765,2468371=489694} 2020-01-16T00:13:11.631 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:11.622+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:11.632 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-01-16T00:13:11.632 INFO:tasks.rados.rados.0.smithi123.stdout:1093: writing smithi12313934-43 from 558065 to 1177934 tid 1 2020-01-16T00:13:11.638 INFO:tasks.rados.rados.0.smithi123.stdout:1093: writing smithi12313934-43 from 1579835 to 2066600 tid 2 2020-01-16T00:13:11.644 INFO:tasks.rados.rados.0.smithi123.stdout:1093: writing smithi12313934-43 from 2468371 to 2958065 tid 3 2020-01-16T00:13:11.644 INFO:tasks.rados.rados.0.smithi123.stdout:1080: finishing write tid 6 to smithi12313934-49 2020-01-16T00:13:11.644 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 49 v 109 (ObjNum 289 snap 0 seq_num 289) dirty exists 2020-01-16T00:13:11.644 INFO:tasks.rados.rados.0.smithi123.stdout:1080: left oid 49 (ObjNum 289 snap 0 seq_num 289) 2020-01-16T00:13:11.645 INFO:tasks.rados.rados.0.smithi123.stdout:1094: read oid 45 snap -1 2020-01-16T00:13:11.645 INFO:tasks.rados.rados.0.smithi123.stdout:1094: expect (ObjNum 216 snap 0 seq_num 216) 2020-01-16T00:13:11.645 INFO:tasks.rados.rados.0.smithi123.stdout:1080: done (13 left) 2020-01-16T00:13:11.645 INFO:tasks.rados.rados.0.smithi123.stdout:1095: read oid 33 snap -1 2020-01-16T00:13:11.645 INFO:tasks.rados.rados.0.smithi123.stdout:1095: expect (ObjNum 266 snap 0 seq_num 266) 2020-01-16T00:13:11.645 INFO:tasks.rados.rados.0.smithi123.stdout:1096: write oid 7 current snap is 0 2020-01-16T00:13:11.645 INFO:tasks.rados.rados.0.smithi123.stdout:1096: seq_num 296 ranges {574872=695118,2038317=630423} 2020-01-16T00:13:11.651 INFO:tasks.rados.rados.0.smithi123.stdout:1096: writing smithi12313934-7 from 574872 to 1269990 tid 1 2020-01-16T00:13:11.660 INFO:tasks.rados.rados.0.smithi123.stdout:1096: writing smithi12313934-7 from 2038317 to 2668740 tid 2 2020-01-16T00:13:11.660 INFO:tasks.rados.rados.0.smithi123.stdout:1097: read oid 11 snap -1 2020-01-16T00:13:11.660 INFO:tasks.rados.rados.0.smithi123.stdout:1097: expect (ObjNum 270 snap 0 seq_num 270) 2020-01-16T00:13:11.660 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:11.660 INFO:tasks.rados.rados.0.smithi123.stdout:1081: expect (ObjNum 214 snap 0 seq_num 214) 2020-01-16T00:13:11.727 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:11.722+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:11.731 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:11.731 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:11.731 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:13:11.731 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:11.740 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-01-16T00:13:11.826 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2020-01-16T00:13:11.830 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:11.824+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:11.858 INFO:tasks.rados.rados.0.smithi123.stdout:1083: expect (ObjNum 276 snap 0 seq_num 276) 2020-01-16T00:13:11.884 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:11.884 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:11.884 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:13:11.884 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:13:11.884 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:11.893 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-01-16T00:13:11.928 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:11.924+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:12.025 INFO:tasks.rados.rados.0.smithi123.stdout:1084: expect (ObjNum 224 snap 0 seq_num 224) 2020-01-16T00:13:12.028 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:12.026+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:12.035 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:12.036 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:13:12.036 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:13:12.036 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:13:12.036 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.036 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:922 3.8 3:113947f2:::smithi12313934-12:head [write 443718~615603,stat] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:12.037 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.863202+0000", 2020-01-16T00:13:12.037 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 34.167556118999997, 2020-01-16T00:13:12.037 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017152697000000001, 2020-01-16T00:13:12.037 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.037 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:12.037 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:12.038 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:12.038 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:12.038 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 922 2020-01-16T00:13:12.038 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.038 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.038 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.038 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.038 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863202+0000", 2020-01-16T00:13:12.039 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.039 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.039 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.039 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.039 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863202+0000", 2020-01-16T00:13:12.039 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:12.040 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.040 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.040 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.040 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863200+0000", 2020-01-16T00:13:12.040 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00038569199999999998 2020-01-16T00:13:12.040 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.040 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.041 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.041 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863585+0000", 2020-01-16T00:13:12.041 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5685e-05 2020-01-16T00:13:12.041 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.041 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.041 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.042 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863601+0000", 2020-01-16T00:13:12.042 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.4734000000000001e-05 2020-01-16T00:13:12.042 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.042 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.042 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.042 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863626+0000", 2020-01-16T00:13:12.043 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1037e-05 2020-01-16T00:13:12.043 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.043 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.043 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.043 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863667+0000", 2020-01-16T00:13:12.043 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016041499999999999 2020-01-16T00:13:12.044 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.044 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.044 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:12.044 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863827+0000", 2020-01-16T00:13:12.044 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0094286430000000004 2020-01-16T00:13:12.044 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.044 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.045 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.045 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.873256+0000", 2020-01-16T00:13:12.045 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2867999999999994e-05 2020-01-16T00:13:12.045 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.045 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.045 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:12.045 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.873329+0000", 2020-01-16T00:13:12.046 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021174789999999998 2020-01-16T00:13:12.046 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.046 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.046 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.046 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.875446+0000", 2020-01-16T00:13:12.046 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011244200000000001 2020-01-16T00:13:12.047 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.047 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.047 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.047 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.875559+0000", 2020-01-16T00:13:12.047 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021772600000000001 2020-01-16T00:13:12.047 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.047 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.047 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1,7", 2020-01-16T00:13:12.048 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.875776+0000", 2020-01-16T00:13:12.048 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0028978860000000001 2020-01-16T00:13:12.048 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.048 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.048 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.048 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.878674+0000", 2020-01-16T00:13:12.049 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.324e-05 2020-01-16T00:13:12.049 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.049 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.049 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:12.049 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.878687+0000", 2020-01-16T00:13:12.049 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0014896180000000001 2020-01-16T00:13:12.049 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.050 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.050 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.050 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880177+0000", 2020-01-16T00:13:12.050 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6881000000000001e-05 2020-01-16T00:13:12.050 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.050 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.050 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:12.050 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880224+0000", 2020-01-16T00:13:12.051 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013096700000000001 2020-01-16T00:13:12.051 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.051 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.051 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.051 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880355+0000", 2020-01-16T00:13:12.051 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017152697000000001 2020-01-16T00:13:12.052 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.052 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.052 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.052 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.052 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.052 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:923 3.8 3:113947f2:::smithi12313934-12:head [write 1564138~470341] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:12.052 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.869292+0000", 2020-01-16T00:13:12.052 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 34.161465788999998, 2020-01-16T00:13:12.052 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017661620999999999, 2020-01-16T00:13:12.053 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.053 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:12.053 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:12.053 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:12.053 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:12.053 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 923 2020-01-16T00:13:12.053 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.054 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.054 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.054 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.054 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869292+0000", 2020-01-16T00:13:12.054 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.054 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.054 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.055 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.055 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869292+0000", 2020-01-16T00:13:12.055 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:12.055 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.055 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.055 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.055 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869291+0000", 2020-01-16T00:13:12.055 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00049842899999999999 2020-01-16T00:13:12.056 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.056 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.056 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.056 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869789+0000", 2020-01-16T00:13:12.056 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1014999999999999e-05 2020-01-16T00:13:12.056 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.057 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.057 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.057 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869830+0000", 2020-01-16T00:13:12.057 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7880000000000007e-06 2020-01-16T00:13:12.057 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.057 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.057 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.058 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869838+0000", 2020-01-16T00:13:12.058 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8346999999999997e-05 2020-01-16T00:13:12.058 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.058 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.058 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.058 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869876+0000", 2020-01-16T00:13:12.058 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9726999999999999e-05 2020-01-16T00:13:12.059 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.059 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.059 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:12.059 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869936+0000", 2020-01-16T00:13:12.059 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003434917 2020-01-16T00:13:12.059 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.060 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.060 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.060 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.873371+0000", 2020-01-16T00:13:12.060 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118808 2020-01-16T00:13:12.060 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.060 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.060 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:12.061 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.873490+0000", 2020-01-16T00:13:12.061 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036712910000000001 2020-01-16T00:13:12.061 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.061 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.061 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.061 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877161+0000", 2020-01-16T00:13:12.062 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9909999999999999e-05 2020-01-16T00:13:12.062 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.062 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.062 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:12.062 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877241+0000", 2020-01-16T00:13:12.062 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031619970000000002 2020-01-16T00:13:12.062 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.063 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.063 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.063 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880403+0000", 2020-01-16T00:13:12.063 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013071099999999999 2020-01-16T00:13:12.063 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.063 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.063 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.064 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880534+0000", 2020-01-16T00:13:12.064 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00032823500000000002 2020-01-16T00:13:12.064 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.064 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.064 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1,7", 2020-01-16T00:13:12.064 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880862+0000", 2020-01-16T00:13:12.064 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00513862 2020-01-16T00:13:12.065 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.065 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.065 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:12.065 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886001+0000", 2020-01-16T00:13:12.065 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8869000000000001e-05 2020-01-16T00:13:12.065 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.065 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.066 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.066 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886030+0000", 2020-01-16T00:13:12.066 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00081600200000000003 2020-01-16T00:13:12.066 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.066 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.066 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.067 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886846+0000", 2020-01-16T00:13:12.067 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4095999999999998e-05 2020-01-16T00:13:12.067 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.067 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.067 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:12.067 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886900+0000", 2020-01-16T00:13:12.068 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.448e-05 2020-01-16T00:13:12.068 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.068 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.068 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.068 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886954+0000", 2020-01-16T00:13:12.068 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017661620999999999 2020-01-16T00:13:12.068 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.069 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.069 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.069 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.069 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.069 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:935 3.2 3:4f8fe446:::smithi12313934-48:head [write 3060489~200036] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:12.070 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.895775+0000", 2020-01-16T00:13:12.070 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 34.134983603000002, 2020-01-16T00:13:12.070 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012912735, 2020-01-16T00:13:12.070 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.070 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:12.070 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:12.071 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:12.071 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:12.071 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 935 2020-01-16T00:13:12.072 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.072 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.072 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.072 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.072 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895775+0000", 2020-01-16T00:13:12.072 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.072 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.073 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.073 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.073 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895775+0000", 2020-01-16T00:13:12.073 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:12.073 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.073 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.073 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.074 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895773+0000", 2020-01-16T00:13:12.074 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3350000000000007e-05 2020-01-16T00:13:12.074 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.074 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.074 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.074 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895856+0000", 2020-01-16T00:13:12.074 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.418e-05 2020-01-16T00:13:12.075 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.075 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.075 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.075 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895871+0000", 2020-01-16T00:13:12.075 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2750000000000001e-06 2020-01-16T00:13:12.075 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.076 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.076 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.076 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895877+0000", 2020-01-16T00:13:12.076 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7113000000000002e-05 2020-01-16T00:13:12.076 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.076 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.077 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.078 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895914+0000", 2020-01-16T00:13:12.078 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00064030700000000001 2020-01-16T00:13:12.078 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.078 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.078 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:12.078 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.896554+0000", 2020-01-16T00:13:12.079 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0098022809999999995 2020-01-16T00:13:12.079 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.079 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.079 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.079 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.906357+0000", 2020-01-16T00:13:12.079 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112898 2020-01-16T00:13:12.079 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.080 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.080 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.080 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.906469+0000", 2020-01-16T00:13:12.080 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022189399999999999 2020-01-16T00:13:12.080 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.080 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.080 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:12.081 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.906691+0000", 2020-01-16T00:13:12.081 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018153869999999999 2020-01-16T00:13:12.081 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.081 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.081 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:12.082 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908507+0000", 2020-01-16T00:13:12.082 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6517999999999999e-05 2020-01-16T00:13:12.082 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.082 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.082 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.082 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908533+0000", 2020-01-16T00:13:12.082 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1899e-05 2020-01-16T00:13:12.083 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.083 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.083 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.083 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908575+0000", 2020-01-16T00:13:12.083 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2222999999999998e-05 2020-01-16T00:13:12.083 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.084 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.084 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:12.084 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908627+0000", 2020-01-16T00:13:12.084 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0019999999999998e-05 2020-01-16T00:13:12.084 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.084 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.084 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.084 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908687+0000", 2020-01-16T00:13:12.085 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012912735 2020-01-16T00:13:12.085 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.085 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.085 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.085 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.085 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.086 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:936 3.2 3:4f8fe446:::smithi12313934-48:head [setxattr _header (56) in=7b,truncate 3260525] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:12.086 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.895886+0000", 2020-01-16T00:13:12.086 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 34.134871898999997, 2020-01-16T00:13:12.086 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014382986, 2020-01-16T00:13:12.086 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.086 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:12.087 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:12.087 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:12.087 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:12.087 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 936 2020-01-16T00:13:12.087 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.087 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.087 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.088 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.088 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895886+0000", 2020-01-16T00:13:12.088 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.088 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.088 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.088 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.088 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895886+0000", 2020-01-16T00:13:12.089 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:12.089 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.089 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.089 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.089 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895885+0000", 2020-01-16T00:13:12.089 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0980000000000001e-06 2020-01-16T00:13:12.089 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.090 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.090 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.090 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895891+0000", 2020-01-16T00:13:12.090 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.959e-06 2020-01-16T00:13:12.090 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.090 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.090 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.091 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895897+0000", 2020-01-16T00:13:12.091 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9920000000000001e-06 2020-01-16T00:13:12.091 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.091 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.091 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.091 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895901+0000", 2020-01-16T00:13:12.091 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00068615199999999999 2020-01-16T00:13:12.092 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.092 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.092 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.092 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.896587+0000", 2020-01-16T00:13:12.093 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2934e-05 2020-01-16T00:13:12.093 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.093 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.093 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:12.093 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.896620+0000", 2020-01-16T00:13:12.093 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010751811 2020-01-16T00:13:12.093 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.094 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.094 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.094 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.907371+0000", 2020-01-16T00:13:12.094 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011422899999999999 2020-01-16T00:13:12.094 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.094 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.094 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.095 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.907486+0000", 2020-01-16T00:13:12.095 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000185172 2020-01-16T00:13:12.095 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.095 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.095 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:12.095 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.907671+0000", 2020-01-16T00:13:12.095 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001265164 2020-01-16T00:13:12.096 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.096 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.096 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.096 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908936+0000", 2020-01-16T00:13:12.096 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000297247 2020-01-16T00:13:12.096 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.096 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.097 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.097 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.909233+0000", 2020-01-16T00:13:12.097 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000907528 2020-01-16T00:13:12.097 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.097 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.097 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:12.097 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910141+0000", 2020-01-16T00:13:12.098 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3384000000000002e-05 2020-01-16T00:13:12.098 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.098 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.098 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:12.098 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910204+0000", 2020-01-16T00:13:12.098 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5229999999999997e-05 2020-01-16T00:13:12.099 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.099 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.099 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.099 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910269+0000", 2020-01-16T00:13:12.099 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014382986 2020-01-16T00:13:12.099 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.100 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.100 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.100 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.100 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.100 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:937 3.2 3:4f8fe446:::smithi12313934-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:13:12.100 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.895918+0000", 2020-01-16T00:13:12.101 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 34.134840271000002, 2020-01-16T00:13:12.101 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014610741999999999, 2020-01-16T00:13:12.101 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.101 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:12.101 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:12.101 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:12.101 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:12.102 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 937 2020-01-16T00:13:12.102 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.102 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.102 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.103 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.103 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895918+0000", 2020-01-16T00:13:12.103 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.103 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.103 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.103 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.103 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895918+0000", 2020-01-16T00:13:12.104 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:12.104 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.104 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.104 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.104 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895917+0000", 2020-01-16T00:13:12.104 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8840000000000004e-06 2020-01-16T00:13:12.104 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.105 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.105 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.105 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895921+0000", 2020-01-16T00:13:12.105 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1579999999999999e-06 2020-01-16T00:13:12.105 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.105 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.106 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.106 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895928+0000", 2020-01-16T00:13:12.106 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4139999999999996e-06 2020-01-16T00:13:12.106 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.106 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.106 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.107 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.895933+0000", 2020-01-16T00:13:12.107 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000702218 2020-01-16T00:13:12.107 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.107 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.107 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.107 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.896635+0000", 2020-01-16T00:13:12.107 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6421e-05 2020-01-16T00:13:12.108 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.108 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.108 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:12.108 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.896661+0000", 2020-01-16T00:13:12.108 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011578879 2020-01-16T00:13:12.108 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.108 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.109 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.109 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908240+0000", 2020-01-16T00:13:12.109 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5567000000000003e-05 2020-01-16T00:13:12.109 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.109 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.109 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:12.109 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908316+0000", 2020-01-16T00:13:12.109 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0019937560000000002 2020-01-16T00:13:12.110 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.110 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.110 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.110 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910310+0000", 2020-01-16T00:13:12.110 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000109566 2020-01-16T00:13:12.110 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.110 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.110 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.111 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910419+0000", 2020-01-16T00:13:12.111 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000109629 2020-01-16T00:13:12.111 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.111 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.111 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.111 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910529+0000", 2020-01-16T00:13:12.111 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014610741999999999 2020-01-16T00:13:12.112 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.112 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.112 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.112 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.112 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.112 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1470 3.7 3:e1be6d3b:::smithi12313934-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e28)", 2020-01-16T00:13:12.112 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:47.940452+0000", 2020-01-16T00:13:12.113 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 24.090306094999999, 2020-01-16T00:13:12.113 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011974416999999999, 2020-01-16T00:13:12.113 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.113 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:12.113 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:12.113 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:12.114 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:12.114 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1470 2020-01-16T00:13:12.114 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.114 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.114 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.114 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.114 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.940452+0000", 2020-01-16T00:13:12.114 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.115 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.115 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.115 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.115 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.940452+0000", 2020-01-16T00:13:12.115 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:12.115 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.115 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.115 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.116 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.940452+0000", 2020-01-16T00:13:12.116 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3379999999999999e-06 2020-01-16T00:13:12.116 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.116 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.116 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.116 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.940454+0000", 2020-01-16T00:13:12.116 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2520000000000001e-06 2020-01-16T00:13:12.116 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.117 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.117 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.117 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.940458+0000", 2020-01-16T00:13:12.117 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6220000000000001e-06 2020-01-16T00:13:12.117 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.117 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.117 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.117 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.940461+0000", 2020-01-16T00:13:12.118 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013034050000000001 2020-01-16T00:13:12.118 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.118 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.118 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.118 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.941764+0000", 2020-01-16T00:13:12.118 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6192000000000006e-05 2020-01-16T00:13:12.118 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.119 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.119 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:12.119 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.941831+0000", 2020-01-16T00:13:12.119 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0096340130000000003 2020-01-16T00:13:12.119 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.119 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.119 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.120 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.951465+0000", 2020-01-16T00:13:12.120 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7785999999999999e-05 2020-01-16T00:13:12.120 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.120 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.120 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:12.120 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.951522+0000", 2020-01-16T00:13:12.120 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00066492500000000002 2020-01-16T00:13:12.121 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.121 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.121 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.121 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.952187+0000", 2020-01-16T00:13:12.121 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000111827 2020-01-16T00:13:12.121 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.121 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.122 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.122 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.952299+0000", 2020-01-16T00:13:12.122 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000127529 2020-01-16T00:13:12.122 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.122 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.122 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.123 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.952427+0000", 2020-01-16T00:13:12.123 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011974416999999999 2020-01-16T00:13:12.123 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.123 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.123 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.123 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.123 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.124 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1660 3.7 3:e1be6d3b:::smithi12313934-24:head [write 2529567~344430] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:13:12.124 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.735048+0000", 2020-01-16T00:13:12.124 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 19.295710347, 2020-01-16T00:13:12.124 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015942654000000001, 2020-01-16T00:13:12.124 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.124 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:12.124 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:12.124 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:12.125 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:12.125 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1660 2020-01-16T00:13:12.125 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.125 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.125 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.125 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.125 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735048+0000", 2020-01-16T00:13:12.126 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.126 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.126 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.126 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.126 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735048+0000", 2020-01-16T00:13:12.126 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:12.126 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.126 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.127 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.127 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735047+0000", 2020-01-16T00:13:12.127 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012866 2020-01-16T00:13:12.127 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.127 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.127 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.127 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735175+0000", 2020-01-16T00:13:12.127 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.34e-05 2020-01-16T00:13:12.128 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.128 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.128 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.128 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735189+0000", 2020-01-16T00:13:12.128 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8479999999999997e-06 2020-01-16T00:13:12.128 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.128 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.128 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.129 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735194+0000", 2020-01-16T00:13:12.129 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5005000000000001e-05 2020-01-16T00:13:12.129 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.129 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.129 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.129 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735229+0000", 2020-01-16T00:13:12.129 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001253035 2020-01-16T00:13:12.129 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.130 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.130 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:12.130 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736483+0000", 2020-01-16T00:13:12.130 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012376109 2020-01-16T00:13:12.130 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.130 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.130 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.130 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748859+0000", 2020-01-16T00:13:12.131 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011718999999999999 2020-01-16T00:13:12.131 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.131 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.131 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.131 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748976+0000", 2020-01-16T00:13:12.131 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019464999999999999 2020-01-16T00:13:12.131 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.131 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.132 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,7", 2020-01-16T00:13:12.132 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.749170+0000", 2020-01-16T00:13:12.132 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0015637719999999999 2020-01-16T00:13:12.132 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.132 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.132 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:12.132 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.750734+0000", 2020-01-16T00:13:12.133 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.1329000000000006e-05 2020-01-16T00:13:12.133 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.133 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.133 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.133 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.750826+0000", 2020-01-16T00:13:12.134 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9779999999999998e-05 2020-01-16T00:13:12.134 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.135 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.135 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.137 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.750915+0000", 2020-01-16T00:13:12.138 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0170000000000001e-05 2020-01-16T00:13:12.138 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.138 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.138 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:12.138 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.750946+0000", 2020-01-16T00:13:12.138 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5064000000000001e-05 2020-01-16T00:13:12.139 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.139 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.139 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.139 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.750991+0000", 2020-01-16T00:13:12.139 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015942654000000001 2020-01-16T00:13:12.139 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.139 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.140 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.140 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.140 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.140 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1661 3.7 3:e1be6d3b:::smithi12313934-24:head [setxattr _header (56) in=7b,truncate 2873997] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:13:12.140 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.735205+0000", 2020-01-16T00:13:12.141 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 19.295553547000001, 2020-01-16T00:13:12.141 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016515230999999998, 2020-01-16T00:13:12.141 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.141 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:12.141 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:12.142 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:12.142 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:12.142 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1661 2020-01-16T00:13:12.142 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.142 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.142 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.142 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.143 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735205+0000", 2020-01-16T00:13:12.143 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.143 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.143 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.143 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.144 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735205+0000", 2020-01-16T00:13:12.144 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:12.144 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.144 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.144 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.144 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735204+0000", 2020-01-16T00:13:12.144 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5369999999999997e-06 2020-01-16T00:13:12.144 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.145 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.145 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.145 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735209+0000", 2020-01-16T00:13:12.145 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9839999999999998e-06 2020-01-16T00:13:12.145 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.145 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.145 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.146 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735215+0000", 2020-01-16T00:13:12.146 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0290000000000002e-06 2020-01-16T00:13:12.146 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.146 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.146 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.146 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735221+0000", 2020-01-16T00:13:12.146 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012977889999999999 2020-01-16T00:13:12.147 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.147 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.147 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.147 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736519+0000", 2020-01-16T00:13:12.147 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7575999999999999e-05 2020-01-16T00:13:12.147 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.147 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.148 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:12.148 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736577+0000", 2020-01-16T00:13:12.148 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013478469999999999 2020-01-16T00:13:12.148 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.148 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.148 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.148 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.750055+0000", 2020-01-16T00:13:12.149 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0983999999999998e-05 2020-01-16T00:13:12.149 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.149 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.149 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.149 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.750116+0000", 2020-01-16T00:13:12.149 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7159999999999998e-05 2020-01-16T00:13:12.149 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.150 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.150 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,7", 2020-01-16T00:13:12.150 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.750213+0000", 2020-01-16T00:13:12.150 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00063694100000000003 2020-01-16T00:13:12.150 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.150 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.151 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.151 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.750850+0000", 2020-01-16T00:13:12.151 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000257447 2020-01-16T00:13:12.151 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.151 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.151 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.151 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.751108+0000", 2020-01-16T00:13:12.152 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00054074500000000003 2020-01-16T00:13:12.152 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.152 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.152 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:12.152 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.751648+0000", 2020-01-16T00:13:12.152 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7606999999999998e-05 2020-01-16T00:13:12.152 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.153 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.153 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:12.153 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.751676+0000", 2020-01-16T00:13:12.153 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3871000000000001e-05 2020-01-16T00:13:12.153 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.153 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.154 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.154 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.751720+0000", 2020-01-16T00:13:12.154 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016515230999999998 2020-01-16T00:13:12.154 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.154 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.154 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.154 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.154 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.155 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1662 3.7 3:e1be6d3b:::smithi12313934-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-01-16T00:13:12.155 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.735228+0000", 2020-01-16T00:13:12.155 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 19.295530184, 2020-01-16T00:13:12.155 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016641372000000001, 2020-01-16T00:13:12.155 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.155 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:12.155 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:12.155 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:12.156 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:12.156 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1662 2020-01-16T00:13:12.156 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.156 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.156 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.156 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.157 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735228+0000", 2020-01-16T00:13:12.157 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.157 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.157 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.157 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.157 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735228+0000", 2020-01-16T00:13:12.157 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:12.158 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.158 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.158 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.158 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735228+0000", 2020-01-16T00:13:12.158 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6260000000000001e-06 2020-01-16T00:13:12.158 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.159 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.159 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.159 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735231+0000", 2020-01-16T00:13:12.159 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5429999999999999e-06 2020-01-16T00:13:12.159 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.159 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.159 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.160 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735238+0000", 2020-01-16T00:13:12.160 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1760000000000001e-06 2020-01-16T00:13:12.160 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.160 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.160 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.160 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735243+0000", 2020-01-16T00:13:12.160 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013610829999999999 2020-01-16T00:13:12.160 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.161 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.161 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.161 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736604+0000", 2020-01-16T00:13:12.161 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2194000000000002e-05 2020-01-16T00:13:12.161 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.161 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.161 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:12.162 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736656+0000", 2020-01-16T00:13:12.162 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013872425000000001 2020-01-16T00:13:12.162 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.162 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.162 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.162 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.750529+0000", 2020-01-16T00:13:12.162 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011952599999999999 2020-01-16T00:13:12.163 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.163 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.163 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:12.163 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.750648+0000", 2020-01-16T00:13:12.163 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001095171 2020-01-16T00:13:12.163 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.163 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.164 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.164 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.751743+0000", 2020-01-16T00:13:12.164 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3576000000000003e-05 2020-01-16T00:13:12.164 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.164 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.164 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.164 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.751807+0000", 2020-01-16T00:13:12.165 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2556000000000002e-05 2020-01-16T00:13:12.165 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.165 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.165 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.165 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.751869+0000", 2020-01-16T00:13:12.165 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016641372000000001 2020-01-16T00:13:12.165 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.165 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.166 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.166 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.166 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.166 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1697 3.7 3:e8af1ebf:::smithi12313934-29:head [setxattr _header (56) in=7b,truncate 2958032] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:13:12.166 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.797586+0000", 2020-01-16T00:13:12.166 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 19.233172633999999, 2020-01-16T00:13:12.166 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014536828, 2020-01-16T00:13:12.166 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.167 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:12.167 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:12.167 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:12.167 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:12.167 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1697 2020-01-16T00:13:12.167 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.167 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.168 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.168 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.168 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.797586+0000", 2020-01-16T00:13:12.168 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.168 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.168 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.168 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.169 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.797586+0000", 2020-01-16T00:13:12.169 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:12.169 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.169 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.169 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.169 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.797585+0000", 2020-01-16T00:13:12.169 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.784e-06 2020-01-16T00:13:12.170 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.170 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.170 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.170 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.797591+0000", 2020-01-16T00:13:12.170 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1860000000000002e-06 2020-01-16T00:13:12.170 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.170 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.171 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.171 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.797595+0000", 2020-01-16T00:13:12.171 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5440000000000001e-06 2020-01-16T00:13:12.171 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.171 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.172 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.172 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.797597+0000", 2020-01-16T00:13:12.172 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020385030000000001 2020-01-16T00:13:12.172 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.172 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.172 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.173 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.799636+0000", 2020-01-16T00:13:12.173 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003161189 2020-01-16T00:13:12.173 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.173 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.173 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:12.173 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.802797+0000", 2020-01-16T00:13:12.173 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0078249510000000001 2020-01-16T00:13:12.173 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.174 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.174 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.174 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.810622+0000", 2020-01-16T00:13:12.174 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012995199999999999 2020-01-16T00:13:12.174 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.174 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.175 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.175 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.810752+0000", 2020-01-16T00:13:12.175 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000200411 2020-01-16T00:13:12.175 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.175 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.175 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,7", 2020-01-16T00:13:12.175 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.810952+0000", 2020-01-16T00:13:12.175 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000676657 2020-01-16T00:13:12.176 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.176 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.176 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:12.176 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.811629+0000", 2020-01-16T00:13:12.176 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1894000000000003e-05 2020-01-16T00:13:12.176 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.176 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.177 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.177 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.811661+0000", 2020-01-16T00:13:12.177 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00037927000000000002 2020-01-16T00:13:12.177 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.177 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.177 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.177 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.812040+0000", 2020-01-16T00:13:12.177 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1873999999999999e-05 2020-01-16T00:13:12.178 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.178 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.178 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:12.178 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.812082+0000", 2020-01-16T00:13:12.178 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0417000000000003e-05 2020-01-16T00:13:12.179 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.179 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.179 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.179 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.812122+0000", 2020-01-16T00:13:12.179 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014536828 2020-01-16T00:13:12.179 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.179 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.179 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.180 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.180 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.180 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1698 3.7 3:e8af1ebf:::smithi12313934-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-01-16T00:13:12.180 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.797614+0000", 2020-01-16T00:13:12.180 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 19.233144704000001, 2020-01-16T00:13:12.180 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014685609000000001, 2020-01-16T00:13:12.180 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.181 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:12.181 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:12.181 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:12.181 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:12.181 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1698 2020-01-16T00:13:12.181 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.181 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.181 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.182 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.182 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.797614+0000", 2020-01-16T00:13:12.182 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.182 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.182 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.183 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.183 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.797614+0000", 2020-01-16T00:13:12.183 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:12.183 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.183 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.183 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.183 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.797613+0000", 2020-01-16T00:13:12.184 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8020000000000002e-06 2020-01-16T00:13:12.184 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.184 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.184 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.184 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.797617+0000", 2020-01-16T00:13:12.185 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3980000000000002e-06 2020-01-16T00:13:12.185 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.185 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.185 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.185 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.797624+0000", 2020-01-16T00:13:12.185 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.76e-06 2020-01-16T00:13:12.185 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.185 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.186 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.186 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.797628+0000", 2020-01-16T00:13:12.186 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052759790000000001 2020-01-16T00:13:12.186 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.186 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.186 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.186 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.802904+0000", 2020-01-16T00:13:12.187 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014863100000000001 2020-01-16T00:13:12.187 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.187 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.187 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:12.187 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.803052+0000", 2020-01-16T00:13:12.187 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0082008370000000008 2020-01-16T00:13:12.187 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.187 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.188 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.188 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.811253+0000", 2020-01-16T00:13:12.188 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012582399999999999 2020-01-16T00:13:12.188 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.188 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.188 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:12.188 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.811379+0000", 2020-01-16T00:13:12.189 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00077615099999999999 2020-01-16T00:13:12.189 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.189 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.189 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.189 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.812155+0000", 2020-01-16T00:13:12.189 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2181000000000005e-05 2020-01-16T00:13:12.190 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.190 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.190 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.190 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.812237+0000", 2020-01-16T00:13:12.190 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1796000000000003e-05 2020-01-16T00:13:12.190 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.190 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.190 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.191 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.812299+0000", 2020-01-16T00:13:12.191 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014685609000000001 2020-01-16T00:13:12.191 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.191 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.191 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.191 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.192 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.192 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1902 3.2 3:42b7f20f:::smithi12313934-42:head [write 1937856~527061] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:13:12.192 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.266980+0000", 2020-01-16T00:13:12.192 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 14.763778043, 2020-01-16T00:13:12.192 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.01342162, 2020-01-16T00:13:12.192 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.192 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:12.193 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:12.193 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:12.193 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:12.193 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1902 2020-01-16T00:13:12.193 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.193 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.194 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.194 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.194 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.266980+0000", 2020-01-16T00:13:12.194 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.194 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.194 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.195 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.195 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.266980+0000", 2020-01-16T00:13:12.195 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:12.195 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.195 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.195 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.195 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.266978+0000", 2020-01-16T00:13:12.196 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016624200000000001 2020-01-16T00:13:12.196 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.196 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.196 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.196 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.267145+0000", 2020-01-16T00:13:12.196 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1721e-05 2020-01-16T00:13:12.197 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.197 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.197 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.197 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.267156+0000", 2020-01-16T00:13:12.197 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8590000000000004e-06 2020-01-16T00:13:12.197 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.197 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.198 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.198 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.267161+0000", 2020-01-16T00:13:12.198 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9175999999999999e-05 2020-01-16T00:13:12.198 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.198 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.198 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.198 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.267200+0000", 2020-01-16T00:13:12.198 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001247001 2020-01-16T00:13:12.199 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.199 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.199 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:12.199 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.268447+0000", 2020-01-16T00:13:12.199 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0077335850000000003 2020-01-16T00:13:12.199 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.199 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.199 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.200 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.276181+0000", 2020-01-16T00:13:12.200 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9253999999999999e-05 2020-01-16T00:13:12.200 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.200 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.200 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.200 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.276240+0000", 2020-01-16T00:13:12.200 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000110684 2020-01-16T00:13:12.200 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.201 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.201 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:12.201 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.276351+0000", 2020-01-16T00:13:12.201 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0022978629999999998 2020-01-16T00:13:12.201 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.201 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.201 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:12.201 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.278649+0000", 2020-01-16T00:13:12.202 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00088566800000000005 2020-01-16T00:13:12.202 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.202 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.202 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.202 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.279534+0000", 2020-01-16T00:13:12.202 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00076735799999999999 2020-01-16T00:13:12.203 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.203 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.203 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.203 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.280302+0000", 2020-01-16T00:13:12.204 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3668000000000002e-05 2020-01-16T00:13:12.204 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.204 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.204 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:12.205 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.280355+0000", 2020-01-16T00:13:12.205 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6477999999999998e-05 2020-01-16T00:13:12.205 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.206 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.206 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.206 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.280402+0000", 2020-01-16T00:13:12.206 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.01342162 2020-01-16T00:13:12.206 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.207 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.207 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.207 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.207 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.207 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2226 3.7 3:e1be6d3b:::smithi12313934-24:head [setxattr _header (56) in=7b,truncate 2100908] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:13:12.207 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.980217+0000", 2020-01-16T00:13:12.207 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 6.0505411240000004, 2020-01-16T00:13:12.207 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023113018999999999, 2020-01-16T00:13:12.208 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.208 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:12.208 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:12.208 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:12.208 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:12.208 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2226 2020-01-16T00:13:12.208 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.208 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.209 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.209 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.209 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980217+0000", 2020-01-16T00:13:12.209 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.209 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.209 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.209 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.210 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980217+0000", 2020-01-16T00:13:12.210 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:12.210 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.210 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.210 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.210 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980216+0000", 2020-01-16T00:13:12.210 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5049999999999998e-06 2020-01-16T00:13:12.211 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.211 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.211 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.211 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980222+0000", 2020-01-16T00:13:12.211 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5654999999999998e-05 2020-01-16T00:13:12.211 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.211 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.211 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.211 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980237+0000", 2020-01-16T00:13:12.212 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9740000000000004e-06 2020-01-16T00:13:12.212 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.212 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.212 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.212 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980241+0000", 2020-01-16T00:13:12.212 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002947776 2020-01-16T00:13:12.212 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.212 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.213 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.213 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.983189+0000", 2020-01-16T00:13:12.213 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0027099400000000001 2020-01-16T00:13:12.213 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.213 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.213 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:12.214 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985899+0000", 2020-01-16T00:13:12.214 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014275589999999999 2020-01-16T00:13:12.214 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.214 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.214 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.214 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.000175+0000", 2020-01-16T00:13:12.214 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011211 2020-01-16T00:13:12.215 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.215 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.215 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.215 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.000287+0000", 2020-01-16T00:13:12.215 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000229886 2020-01-16T00:13:12.215 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.215 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.215 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,7", 2020-01-16T00:13:12.216 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.000517+0000", 2020-01-16T00:13:12.216 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00095715399999999999 2020-01-16T00:13:12.216 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.216 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.216 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:12.216 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.001474+0000", 2020-01-16T00:13:12.216 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001341512 2020-01-16T00:13:12.217 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.217 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.217 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.217 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.002815+0000", 2020-01-16T00:13:12.217 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035789099999999999 2020-01-16T00:13:12.217 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.217 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.218 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.218 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003173+0000", 2020-01-16T00:13:12.218 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8571000000000005e-05 2020-01-16T00:13:12.218 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.218 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.218 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:12.218 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003242+0000", 2020-01-16T00:13:12.219 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8293000000000003e-05 2020-01-16T00:13:12.219 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.219 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.219 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.219 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003330+0000", 2020-01-16T00:13:12.219 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023113018999999999 2020-01-16T00:13:12.219 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.219 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.220 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.220 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.220 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.220 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2227 3.7 3:e1be6d3b:::smithi12313934-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e35)", 2020-01-16T00:13:12.220 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:13:12.220 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 6.0504938709999996, 2020-01-16T00:13:12.220 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023303356000000001, 2020-01-16T00:13:12.220 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.221 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:12.221 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:12.221 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:12.221 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:12.221 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2227 2020-01-16T00:13:12.221 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.221 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.222 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.222 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.222 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:13:12.222 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.222 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.222 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.222 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.222 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:13:12.223 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:12.223 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.223 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.223 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.223 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:13:12.223 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3079999999999999e-06 2020-01-16T00:13:12.223 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.224 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.224 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.224 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980267+0000", 2020-01-16T00:13:12.224 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9290000000000001e-06 2020-01-16T00:13:12.224 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.224 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.224 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.224 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980274+0000", 2020-01-16T00:13:12.225 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7400000000000002e-06 2020-01-16T00:13:12.225 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.225 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.225 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.225 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980278+0000", 2020-01-16T00:13:12.225 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056624300000000004 2020-01-16T00:13:12.226 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.226 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.226 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.226 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985940+0000", 2020-01-16T00:13:12.226 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4261000000000002e-05 2020-01-16T00:13:12.226 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.226 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.227 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:12.227 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985985+0000", 2020-01-16T00:13:12.227 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015087676 2020-01-16T00:13:12.227 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.227 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.227 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.228 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.001072+0000", 2020-01-16T00:13:12.228 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011960600000000001 2020-01-16T00:13:12.228 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.228 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.228 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:12.228 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.001192+0000", 2020-01-16T00:13:12.228 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021718309999999999 2020-01-16T00:13:12.229 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.229 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.229 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.229 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003364+0000", 2020-01-16T00:13:12.229 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000102855 2020-01-16T00:13:12.229 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.230 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.230 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.230 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003467+0000", 2020-01-16T00:13:12.230 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010122499999999999 2020-01-16T00:13:12.230 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.230 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.230 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.231 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003568+0000", 2020-01-16T00:13:12.231 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023303356000000001 2020-01-16T00:13:12.231 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.231 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.231 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.231 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.231 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.231 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2233 3.2 3:4f8fe446:::smithi12313934-48:head [setxattr _header (56) in=7b,truncate 2117715] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:13:12.232 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.993103+0000", 2020-01-16T00:13:12.232 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 6.0376556490000004, 2020-01-16T00:13:12.232 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017951763999999999, 2020-01-16T00:13:12.232 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.232 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:12.232 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:12.232 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:12.233 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:12.233 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2233 2020-01-16T00:13:12.233 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.233 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.233 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.233 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.233 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993103+0000", 2020-01-16T00:13:12.233 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.234 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.234 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.234 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.234 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993103+0000", 2020-01-16T00:13:12.234 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:12.234 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.234 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.234 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.235 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993102+0000", 2020-01-16T00:13:12.235 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9369999999999993e-06 2020-01-16T00:13:12.235 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.235 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.235 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.235 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993111+0000", 2020-01-16T00:13:12.235 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3340000000000002e-06 2020-01-16T00:13:12.236 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.236 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.236 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.236 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993119+0000", 2020-01-16T00:13:12.236 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1769999999999997e-06 2020-01-16T00:13:12.236 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.236 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.236 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.237 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993124+0000", 2020-01-16T00:13:12.237 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001601304 2020-01-16T00:13:12.237 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.237 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.237 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.237 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.994725+0000", 2020-01-16T00:13:12.237 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048777250000000003 2020-01-16T00:13:12.238 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.238 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.238 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:12.238 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999603+0000", 2020-01-16T00:13:12.238 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0099793880000000005 2020-01-16T00:13:12.238 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.238 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.238 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.239 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.009583+0000", 2020-01-16T00:13:12.239 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000108207 2020-01-16T00:13:12.239 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.239 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.239 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.239 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.009691+0000", 2020-01-16T00:13:12.239 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012862999999999999 2020-01-16T00:13:12.240 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.240 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.240 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:12.240 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.009819+0000", 2020-01-16T00:13:12.240 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00077947599999999998 2020-01-16T00:13:12.240 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.240 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.240 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:12.241 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010599+0000", 2020-01-16T00:13:12.241 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000221282 2020-01-16T00:13:12.241 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.241 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.241 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.241 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010820+0000", 2020-01-16T00:13:12.241 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7021999999999995e-05 2020-01-16T00:13:12.242 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.242 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.242 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.242 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010907+0000", 2020-01-16T00:13:12.242 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9815999999999997e-05 2020-01-16T00:13:12.242 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.242 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.242 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:12.243 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010977+0000", 2020-01-16T00:13:12.243 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7453999999999994e-05 2020-01-16T00:13:12.243 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.243 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.243 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.243 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.011054+0000", 2020-01-16T00:13:12.243 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017951763999999999 2020-01-16T00:13:12.243 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.244 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.244 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.244 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.244 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.244 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2234 3.2 3:4f8fe446:::smithi12313934-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e35)", 2020-01-16T00:13:12.244 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.993135+0000", 2020-01-16T00:13:12.244 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 6.0376231960000002, 2020-01-16T00:13:12.245 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018067916999999999, 2020-01-16T00:13:12.245 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.245 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:12.245 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:12.245 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:12.245 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:12.245 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2234 2020-01-16T00:13:12.245 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.246 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.246 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.246 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.246 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993135+0000", 2020-01-16T00:13:12.246 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.246 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.246 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.247 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.247 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993135+0000", 2020-01-16T00:13:12.247 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:12.247 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.247 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.247 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.247 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993134+0000", 2020-01-16T00:13:12.247 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5630000000000002e-06 2020-01-16T00:13:12.248 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.248 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.248 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.248 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993139+0000", 2020-01-16T00:13:12.248 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1730000000000003e-06 2020-01-16T00:13:12.248 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.248 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.249 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.249 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993144+0000", 2020-01-16T00:13:12.249 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.937e-06 2020-01-16T00:13:12.249 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.249 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.249 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.249 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993147+0000", 2020-01-16T00:13:12.250 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0065164000000000003 2020-01-16T00:13:12.250 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.250 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.250 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.250 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999663+0000", 2020-01-16T00:13:12.250 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6215000000000002e-05 2020-01-16T00:13:12.250 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.250 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.251 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:12.251 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999729+0000", 2020-01-16T00:13:12.251 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010475887999999999 2020-01-16T00:13:12.251 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.251 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.251 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.251 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010205+0000", 2020-01-16T00:13:12.252 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119471 2020-01-16T00:13:12.252 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.252 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.252 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:12.252 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010325+0000", 2020-01-16T00:13:12.252 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00075498799999999995 2020-01-16T00:13:12.252 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.253 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.253 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.253 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.011080+0000", 2020-01-16T00:13:12.253 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7904e-05 2020-01-16T00:13:12.253 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.253 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.253 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.253 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.011138+0000", 2020-01-16T00:13:12.254 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5273999999999999e-05 2020-01-16T00:13:12.254 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.254 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.254 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.254 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.011203+0000", 2020-01-16T00:13:12.254 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018067916999999999 2020-01-16T00:13:12.254 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.255 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.255 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.255 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.255 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.255 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2314 3.7 3:e8af1ebf:::smithi12313934-29:head [setxattr _header (56) in=7b,truncate 2319399] snapc 0=[] ondisk+write+known_if_redirected e36)", 2020-01-16T00:13:12.255 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.317802+0000", 2020-01-16T00:13:12.255 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.7129560599999998, 2020-01-16T00:13:12.255 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015056285000000001, 2020-01-16T00:13:12.256 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.256 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:12.256 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:12.256 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:12.256 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:12.256 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2314 2020-01-16T00:13:12.256 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.257 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.257 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.257 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.257 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.317802+0000", 2020-01-16T00:13:12.257 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.257 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.258 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.258 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.258 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.317802+0000", 2020-01-16T00:13:12.258 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:12.258 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.258 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.258 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.259 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.317801+0000", 2020-01-16T00:13:12.259 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5580000000000003e-06 2020-01-16T00:13:12.259 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.259 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.259 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.259 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.317809+0000", 2020-01-16T00:13:12.259 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5660000000000001e-06 2020-01-16T00:13:12.259 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.260 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.260 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.260 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.317816+0000", 2020-01-16T00:13:12.260 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4530000000000004e-06 2020-01-16T00:13:12.260 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.260 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.260 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.261 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.317821+0000", 2020-01-16T00:13:12.261 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001622425 2020-01-16T00:13:12.261 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.261 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.261 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.261 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.319443+0000", 2020-01-16T00:13:12.261 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00063579200000000004 2020-01-16T00:13:12.262 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.262 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.262 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:12.262 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.320079+0000", 2020-01-16T00:13:12.262 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010884268000000001 2020-01-16T00:13:12.262 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.262 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.262 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.263 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.330963+0000", 2020-01-16T00:13:12.263 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000114967 2020-01-16T00:13:12.263 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.263 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.263 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.263 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.331078+0000", 2020-01-16T00:13:12.263 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000125964 2020-01-16T00:13:12.264 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.264 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.264 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,7", 2020-01-16T00:13:12.264 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.331204+0000", 2020-01-16T00:13:12.264 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001111938 2020-01-16T00:13:12.264 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.264 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.265 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:12.265 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.332316+0000", 2020-01-16T00:13:12.265 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00032159300000000001 2020-01-16T00:13:12.265 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.265 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.265 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.265 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.332638+0000", 2020-01-16T00:13:12.266 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000149794 2020-01-16T00:13:12.266 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.266 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.266 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.266 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.332787+0000", 2020-01-16T00:13:12.266 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4612e-05 2020-01-16T00:13:12.266 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.266 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.267 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:12.267 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.332822+0000", 2020-01-16T00:13:12.267 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6563999999999998e-05 2020-01-16T00:13:12.267 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.267 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.267 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.267 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.332859+0000", 2020-01-16T00:13:12.268 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015056285000000001 2020-01-16T00:13:12.268 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.268 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.268 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.268 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.268 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.268 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2315 3.7 3:e8af1ebf:::smithi12313934-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e36)", 2020-01-16T00:13:12.268 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.317826+0000", 2020-01-16T00:13:12.269 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.7129325900000003, 2020-01-16T00:13:12.269 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015205279, 2020-01-16T00:13:12.269 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.269 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:12.269 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:12.269 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:12.269 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:12.270 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2315 2020-01-16T00:13:12.270 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.270 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.270 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.270 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.270 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.317826+0000", 2020-01-16T00:13:12.270 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.270 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.271 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.271 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.271 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.317826+0000", 2020-01-16T00:13:12.271 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:12.271 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.271 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.271 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.271 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.317825+0000", 2020-01-16T00:13:12.272 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6370000000000001e-06 2020-01-16T00:13:12.272 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.272 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.272 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.272 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.317829+0000", 2020-01-16T00:13:12.272 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5399999999999998e-06 2020-01-16T00:13:12.272 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.273 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.273 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.273 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.317836+0000", 2020-01-16T00:13:12.273 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4079999999999998e-06 2020-01-16T00:13:12.273 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.273 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.273 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.273 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.317840+0000", 2020-01-16T00:13:12.274 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0025237850000000002 2020-01-16T00:13:12.274 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.274 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.274 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.274 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.320363+0000", 2020-01-16T00:13:12.274 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5877999999999995e-05 2020-01-16T00:13:12.274 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.275 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.275 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:12.275 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.320429+0000", 2020-01-16T00:13:12.275 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011518247000000001 2020-01-16T00:13:12.275 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.275 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.275 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.275 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.331947+0000", 2020-01-16T00:13:12.276 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012454799999999999 2020-01-16T00:13:12.276 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.276 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.276 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:12.276 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.332072+0000", 2020-01-16T00:13:12.276 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00081245299999999998 2020-01-16T00:13:12.276 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.276 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.277 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.277 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.332884+0000", 2020-01-16T00:13:12.277 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4336000000000001e-05 2020-01-16T00:13:12.277 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.277 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.277 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.277 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.332959+0000", 2020-01-16T00:13:12.277 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2204000000000002e-05 2020-01-16T00:13:12.278 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.278 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.278 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.278 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.333031+0000", 2020-01-16T00:13:12.278 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015205279 2020-01-16T00:13:12.278 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.278 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.279 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.279 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.279 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.279 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2324 3.7 3:e1be6d3b:::smithi12313934-24:head [setxattr _header (56) in=7b,truncate 2336206] snapc 0=[] ondisk+write+known_if_redirected e36)", 2020-01-16T00:13:12.279 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.550846+0000", 2020-01-16T00:13:12.279 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.4799126100000004, 2020-01-16T00:13:12.279 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018364742, 2020-01-16T00:13:12.280 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.280 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:12.280 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:12.280 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:12.280 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:12.280 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2324 2020-01-16T00:13:12.280 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.280 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.281 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.281 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.281 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550846+0000", 2020-01-16T00:13:12.281 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.281 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.281 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.281 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.282 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550846+0000", 2020-01-16T00:13:12.282 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:12.282 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.282 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.282 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.282 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550845+0000", 2020-01-16T00:13:12.282 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1829999999999999e-06 2020-01-16T00:13:12.282 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.283 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.283 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.283 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550851+0000", 2020-01-16T00:13:12.283 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1925e-05 2020-01-16T00:13:12.283 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.283 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.283 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.283 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550863+0000", 2020-01-16T00:13:12.284 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1139999999999997e-06 2020-01-16T00:13:12.284 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.284 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.284 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.284 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550868+0000", 2020-01-16T00:13:12.284 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001388423 2020-01-16T00:13:12.284 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.284 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.285 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.285 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.552256+0000", 2020-01-16T00:13:12.285 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0034246540000000001 2020-01-16T00:13:12.285 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.285 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.285 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:12.285 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555681+0000", 2020-01-16T00:13:12.286 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011882762999999999 2020-01-16T00:13:12.286 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.286 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.286 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.286 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.567564+0000", 2020-01-16T00:13:12.286 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015010800000000001 2020-01-16T00:13:12.286 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.287 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.287 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.287 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.567714+0000", 2020-01-16T00:13:12.287 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011527999999999999 2020-01-16T00:13:12.287 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.287 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.287 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,7", 2020-01-16T00:13:12.288 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.567829+0000", 2020-01-16T00:13:12.288 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00039888499999999997 2020-01-16T00:13:12.288 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.288 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.288 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:12.288 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568228+0000", 2020-01-16T00:13:12.288 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00054254599999999998 2020-01-16T00:13:12.288 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.289 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.289 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.289 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568771+0000", 2020-01-16T00:13:12.289 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035206400000000002 2020-01-16T00:13:12.289 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.289 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.289 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.290 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569123+0000", 2020-01-16T00:13:12.290 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1262e-05 2020-01-16T00:13:12.290 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.290 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.290 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:12.290 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569154+0000", 2020-01-16T00:13:12.290 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6499000000000002e-05 2020-01-16T00:13:12.291 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.291 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.291 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.291 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569210+0000", 2020-01-16T00:13:12.291 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018364742 2020-01-16T00:13:12.291 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.291 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.291 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.292 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.292 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.292 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2325 3.7 3:e1be6d3b:::smithi12313934-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e36)", 2020-01-16T00:13:12.292 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.550877+0000", 2020-01-16T00:13:12.292 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.4798815440000004, 2020-01-16T00:13:12.292 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018565154, 2020-01-16T00:13:12.292 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.292 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:12.293 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:12.293 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:12.293 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:12.293 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2325 2020-01-16T00:13:12.293 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.293 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.293 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.294 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.294 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550877+0000", 2020-01-16T00:13:12.294 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.294 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.294 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.294 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.294 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550877+0000", 2020-01-16T00:13:12.294 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:12.295 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.295 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.295 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.295 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550876+0000", 2020-01-16T00:13:12.295 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1080000000000003e-06 2020-01-16T00:13:12.295 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.295 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.295 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.296 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550880+0000", 2020-01-16T00:13:12.296 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3269999999999997e-06 2020-01-16T00:13:12.296 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.296 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.296 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.296 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550886+0000", 2020-01-16T00:13:12.296 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.386e-06 2020-01-16T00:13:12.297 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.297 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.297 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.297 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550891+0000", 2020-01-16T00:13:12.297 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048231410000000004 2020-01-16T00:13:12.297 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.297 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.298 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.298 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555714+0000", 2020-01-16T00:13:12.298 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0071999999999999e-05 2020-01-16T00:13:12.298 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.298 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.299 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:12.299 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555744+0000", 2020-01-16T00:13:12.299 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012329005000000001 2020-01-16T00:13:12.299 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.300 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.300 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.300 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568073+0000", 2020-01-16T00:13:12.300 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9087999999999997e-05 2020-01-16T00:13:12.300 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.301 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.301 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:12.301 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568172+0000", 2020-01-16T00:13:12.301 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010718220000000001 2020-01-16T00:13:12.301 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.301 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.302 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.302 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569244+0000", 2020-01-16T00:13:12.302 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8199e-05 2020-01-16T00:13:12.302 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.302 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.302 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.302 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569342+0000", 2020-01-16T00:13:12.303 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9708000000000001e-05 2020-01-16T00:13:12.303 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.303 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.303 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.303 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569442+0000", 2020-01-16T00:13:12.303 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018565154 2020-01-16T00:13:12.303 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.303 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.304 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.304 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.304 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.304 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:12.305 INFO:tasks.rados.rados.0.smithi123.stdout:1085: finishing write tid 1 to smithi12313934-2 2020-01-16T00:13:12.306 INFO:tasks.rados.rados.0.smithi123.stdout:1085: finishing write tid 2 to smithi12313934-2 2020-01-16T00:13:12.306 INFO:tasks.rados.rados.0.smithi123.stdout:1085: finishing write tid 3 to smithi12313934-2 2020-01-16T00:13:12.306 INFO:tasks.rados.rados.0.smithi123.stdout:1085: finishing write tid 5 to smithi12313934-2 2020-01-16T00:13:12.306 INFO:tasks.rados.rados.0.smithi123.stdout:1085: finishing write tid 6 to smithi12313934-2 2020-01-16T00:13:12.306 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 2 v 155 (ObjNum 291 snap 0 seq_num 291) dirty exists 2020-01-16T00:13:12.306 INFO:tasks.rados.rados.0.smithi123.stdout:1085: left oid 2 (ObjNum 291 snap 0 seq_num 291) 2020-01-16T00:13:12.307 INFO:tasks.rados.rados.0.smithi123.stdout:1086: finishing write tid 1 to smithi12313934-35 2020-01-16T00:13:12.307 INFO:tasks.rados.rados.0.smithi123.stdout:1088: expect (ObjNum 272 snap 0 seq_num 272) 2020-01-16T00:13:12.307 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:12.300+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:12.308 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-01-16T00:13:12.391 INFO:tasks.rados.rados.0.smithi123.stdout:1087: expect (ObjNum 227 snap 0 seq_num 227) 2020-01-16T00:13:12.406 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:12.404+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:12.454 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:12.454 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:12.454 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:13:12.454 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:12.464 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-01-16T00:13:12.506 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:12.502+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:12.552 INFO:tasks.rados.rados.0.smithi123.stdout:1086: finishing write tid 2 to smithi12313934-35 2020-01-16T00:13:12.552 INFO:tasks.rados.rados.0.smithi123.stdout:1086: finishing write tid 4 to smithi12313934-35 2020-01-16T00:13:12.552 INFO:tasks.rados.rados.0.smithi123.stdout:1086: finishing write tid 5 to smithi12313934-35 2020-01-16T00:13:12.552 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 35 v 114 (ObjNum 292 snap 0 seq_num 292) dirty exists 2020-01-16T00:13:12.552 INFO:tasks.rados.rados.0.smithi123.stdout:1086: left oid 35 (ObjNum 292 snap 0 seq_num 292) 2020-01-16T00:13:12.552 INFO:tasks.rados.rados.0.smithi123.stdout:1081: done (15 left) 2020-01-16T00:13:12.553 INFO:tasks.rados.rados.0.smithi123.stdout:1083: done (14 left) 2020-01-16T00:13:12.553 INFO:tasks.rados.rados.0.smithi123.stdout:1084: done (13 left) 2020-01-16T00:13:12.553 INFO:tasks.rados.rados.0.smithi123.stdout:1085: done (12 left) 2020-01-16T00:13:12.553 INFO:tasks.rados.rados.0.smithi123.stdout:1086: done (11 left) 2020-01-16T00:13:12.553 INFO:tasks.rados.rados.0.smithi123.stdout:1087: done (10 left) 2020-01-16T00:13:12.553 INFO:tasks.rados.rados.0.smithi123.stdout:1088: done (9 left) 2020-01-16T00:13:12.553 INFO:tasks.rados.rados.0.smithi123.stdout:1098: write oid 46 current snap is 0 2020-01-16T00:13:12.553 INFO:tasks.rados.rados.0.smithi123.stdout:1098: seq_num 297 ranges {591679=486720,1813152=490434} 2020-01-16T00:13:12.565 INFO:tasks.rados.rados.0.smithi123.stdout:1098: writing smithi12313934-46 from 591679 to 1078399 tid 1 2020-01-16T00:13:12.573 INFO:tasks.rados.rados.0.smithi123.stdout:1098: writing smithi12313934-46 from 1813152 to 2303586 tid 2 2020-01-16T00:13:12.573 INFO:tasks.rados.rados.0.smithi123.stdout:1099: write (excl) oid 38 current snap is 0 2020-01-16T00:13:12.574 INFO:tasks.rados.rados.0.smithi123.stdout:1099: seq_num 298 ranges {608486=561969,1871634=634092,2950640=57846} 2020-01-16T00:13:12.576 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-01-16T00:13:12.580 INFO:tasks.rados.rados.0.smithi123.stdout:1099: writing smithi12313934-38 from 608486 to 1170455 tid 1 2020-01-16T00:13:12.588 INFO:tasks.rados.rados.0.smithi123.stdout:1099: writing smithi12313934-38 from 1871634 to 2505726 tid 2 2020-01-16T00:13:12.592 INFO:tasks.rados.rados.0.smithi123.stdout:1099: writing smithi12313934-38 from 2950640 to 3008486 tid 3 2020-01-16T00:13:12.592 INFO:tasks.rados.rados.0.smithi123.stdout:1100: write oid 1 current snap is 0 2020-01-16T00:13:12.592 INFO:tasks.rados.rados.0.smithi123.stdout:1100: seq_num 299 ranges {625293=637218,1813763=494103,2978063=47230} 2020-01-16T00:13:12.599 INFO:tasks.rados.rados.0.smithi123.stdout:1100: writing smithi12313934-1 from 625293 to 1262511 tid 1 2020-01-16T00:13:12.606 INFO:tasks.rados.rados.0.smithi123.stdout:1100: writing smithi12313934-1 from 1813763 to 2307866 tid 2 2020-01-16T00:13:12.607 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:12.602+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:12.611 INFO:tasks.rados.rados.0.smithi123.stdout:1100: writing smithi12313934-1 from 2978063 to 3025293 tid 3 2020-01-16T00:13:12.611 INFO:tasks.rados.rados.0.smithi123.stdout:1101: write oid 32 current snap is 0 2020-01-16T00:13:12.611 INFO:tasks.rados.rados.0.smithi123.stdout:1101: seq_num 300 ranges {642100=712467,1872245=637761} 2020-01-16T00:13:12.618 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:12.618 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:12.618 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:13:12.618 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:13:12.618 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:12.619 INFO:tasks.rados.rados.0.smithi123.stdout:1101: writing smithi12313934-32 from 642100 to 1354567 tid 1 2020-01-16T00:13:12.625 INFO:tasks.rados.rados.0.smithi123.stdout:1101: writing smithi12313934-32 from 1872245 to 2510006 tid 2 2020-01-16T00:13:12.626 INFO:tasks.rados.rados.0.smithi123.stdout:1102: write (excl) oid 9 current snap is 0 2020-01-16T00:13:12.626 INFO:tasks.rados.rados.0.smithi123.stdout:1102: seq_num 301 ranges {658907=787716,1930727=497772,3032909=25998} 2020-01-16T00:13:12.628 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-01-16T00:13:12.634 INFO:tasks.rados.rados.0.smithi123.stdout:1102: writing smithi12313934-9 from 658907 to 1446623 tid 1 2020-01-16T00:13:12.641 INFO:tasks.rados.rados.0.smithi123.stdout:1102: writing smithi12313934-9 from 1930727 to 2428499 tid 2 2020-01-16T00:13:12.645 INFO:tasks.rados.rados.0.smithi123.stdout:1102: writing smithi12313934-9 from 3032909 to 3058907 tid 3 2020-01-16T00:13:12.645 INFO:tasks.rados.rados.0.smithi123.stdout:1103: write (excl) oid 48 current snap is 0 2020-01-16T00:13:12.645 INFO:tasks.rados.rados.0.smithi123.stdout:1103: seq_num 302 ranges {675714=462965,1589209=641430,2943979=131735} 2020-01-16T00:13:12.652 INFO:tasks.rados.rados.0.smithi123.stdout:1103: writing smithi12313934-48 from 675714 to 1138679 tid 1 2020-01-16T00:13:12.659 INFO:tasks.rados.rados.0.smithi123.stdout:1103: writing smithi12313934-48 from 1589209 to 2230639 tid 2 2020-01-16T00:13:12.672 INFO:tasks.rados.rados.0.smithi123.stdout:1103: writing smithi12313934-48 from 2943979 to 3075714 tid 3 2020-01-16T00:13:12.672 INFO:tasks.rados.rados.0.smithi123.stdout:1104: read oid 28 snap -1 2020-01-16T00:13:12.672 INFO:tasks.rados.rados.0.smithi123.stdout:1104: expect (ObjNum 242 snap 0 seq_num 242) 2020-01-16T00:13:12.672 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:12.672 INFO:tasks.rados.rados.0.smithi123.stdout:1089: finishing write tid 1 to smithi12313934-41 2020-01-16T00:13:12.672 INFO:tasks.rados.rados.0.smithi123.stdout:1089: finishing write tid 2 to smithi12313934-41 2020-01-16T00:13:12.673 INFO:tasks.rados.rados.0.smithi123.stdout:1089: finishing write tid 3 to smithi12313934-41 2020-01-16T00:13:12.673 INFO:tasks.rados.rados.0.smithi123.stdout:1089: finishing write tid 5 to smithi12313934-41 2020-01-16T00:13:12.673 INFO:tasks.rados.rados.0.smithi123.stdout:1089: finishing write tid 6 to smithi12313934-41 2020-01-16T00:13:12.673 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 41 v 159 (ObjNum 293 snap 0 seq_num 293) dirty exists 2020-01-16T00:13:12.673 INFO:tasks.rados.rados.0.smithi123.stdout:1089: left oid 41 (ObjNum 293 snap 0 seq_num 293) 2020-01-16T00:13:12.673 INFO:tasks.rados.rados.0.smithi123.stdout:1091: expect (ObjNum 221 snap 0 seq_num 221) 2020-01-16T00:13:12.707 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:12.704+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:12.729 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:13:12.740 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:12.741 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:13:12.741 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:13:12.741 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:13:12.741 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.741 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.4.0:114 3.9 e24/20 3:9fc687b2:::smithi12313934-44:head v 24'74, mlcod=24'72)", 2020-01-16T00:13:12.741 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:13:12.742 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 34.410766942999999, 2020-01-16T00:13:12.742 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010399429, 2020-01-16T00:13:12.742 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.742 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:12.742 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.742 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.742 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.742 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:13:12.743 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.743 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.743 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.743 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.743 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:13:12.743 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:12.743 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.743 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.744 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.744 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325276+0000", 2020-01-16T00:13:12.744 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0046127069999999997 2020-01-16T00:13:12.744 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.744 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.744 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.744 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329889+0000", 2020-01-16T00:13:12.745 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3519999999999998e-05 2020-01-16T00:13:12.745 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.745 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.745 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.745 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329912+0000", 2020-01-16T00:13:12.745 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4070000000000004e-06 2020-01-16T00:13:12.745 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.749 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.749 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.749 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329920+0000", 2020-01-16T00:13:12.750 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8817000000000003e-05 2020-01-16T00:13:12.750 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.750 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.750 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.750 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329978+0000", 2020-01-16T00:13:12.750 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9721e-05 2020-01-16T00:13:12.751 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.751 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.751 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.751 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329998+0000", 2020-01-16T00:13:12.751 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056328139999999999 2020-01-16T00:13:12.751 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.752 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.752 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:12.752 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.335631+0000", 2020-01-16T00:13:12.752 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7170999999999999e-05 2020-01-16T00:13:12.752 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.752 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.752 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.752 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.335678+0000", 2020-01-16T00:13:12.753 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010399429 2020-01-16T00:13:12.753 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.753 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.753 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.753 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.753 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.753 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:24 2.1 2:8b431f80:::smithi12313934-40 low_tier:head [write 790528~688128] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:13:12.754 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.482303+0000", 2020-01-16T00:13:12.754 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 34.253742846999998, 2020-01-16T00:13:12.754 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010071853, 2020-01-16T00:13:12.754 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.754 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:12.754 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.754 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.754 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.755 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.482303+0000", 2020-01-16T00:13:12.755 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.755 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.755 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.755 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.755 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.482303+0000", 2020-01-16T00:13:12.755 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:12.756 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.756 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.756 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.756 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.482301+0000", 2020-01-16T00:13:12.756 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031173960000000001 2020-01-16T00:13:12.756 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.756 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.756 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.757 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485418+0000", 2020-01-16T00:13:12.757 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9300000000000001e-05 2020-01-16T00:13:12.757 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.757 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.757 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.757 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485448+0000", 2020-01-16T00:13:12.757 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2611e-05 2020-01-16T00:13:12.757 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.758 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.758 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.758 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485460+0000", 2020-01-16T00:13:12.758 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112487 2020-01-16T00:13:12.758 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.758 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.758 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.759 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485573+0000", 2020-01-16T00:13:12.759 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022169299999999999 2020-01-16T00:13:12.759 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.759 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.759 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.759 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485794+0000", 2020-01-16T00:13:12.759 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00036287500000000002 2020-01-16T00:13:12.759 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.760 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.760 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:12.760 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.486157+0000", 2020-01-16T00:13:12.760 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040260310000000002 2020-01-16T00:13:12.760 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.760 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.760 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:12.761 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.490183+0000", 2020-01-16T00:13:12.761 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00081083000000000001 2020-01-16T00:13:12.761 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.761 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.761 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.761 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.490994+0000", 2020-01-16T00:13:12.761 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001287746 2020-01-16T00:13:12.761 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.762 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.762 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.762 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492282+0000", 2020-01-16T00:13:12.762 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8810999999999998e-05 2020-01-16T00:13:12.762 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.762 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.762 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:12.763 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492321+0000", 2020-01-16T00:13:12.763 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4095000000000003e-05 2020-01-16T00:13:12.763 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.763 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.763 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.763 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492375+0000", 2020-01-16T00:13:12.763 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010071853 2020-01-16T00:13:12.764 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.764 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.764 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.764 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.764 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.764 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:25 2.1 2:8b431f80:::smithi12313934-40 low_tier:head [write 1926656~67072] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:13:12.764 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.485481+0000", 2020-01-16T00:13:12.764 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 34.250564382, 2020-01-16T00:13:12.765 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0081599020000000001, 2020-01-16T00:13:12.765 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.765 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:12.765 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.765 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.765 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.765 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485481+0000", 2020-01-16T00:13:12.766 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.766 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.766 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.766 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.766 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485481+0000", 2020-01-16T00:13:12.766 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:12.767 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.767 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.767 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.767 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485479+0000", 2020-01-16T00:13:12.767 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012423800000000001 2020-01-16T00:13:12.767 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.767 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.768 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.768 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485603+0000", 2020-01-16T00:13:12.768 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8366999999999999e-05 2020-01-16T00:13:12.768 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.768 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.768 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.768 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485621+0000", 2020-01-16T00:13:12.768 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0207000000000001e-05 2020-01-16T00:13:12.769 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.769 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.769 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.769 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485632+0000", 2020-01-16T00:13:12.769 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002868532 2020-01-16T00:13:12.769 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.769 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.769 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.770 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.488500+0000", 2020-01-16T00:13:12.770 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011340000000000001 2020-01-16T00:13:12.770 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.770 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.770 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.770 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.488614+0000", 2020-01-16T00:13:12.770 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019056300000000001 2020-01-16T00:13:12.771 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.771 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.771 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:12.771 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.488804+0000", 2020-01-16T00:13:12.771 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00153302 2020-01-16T00:13:12.771 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.771 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.771 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:12.772 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.490337+0000", 2020-01-16T00:13:12.772 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00089946100000000005 2020-01-16T00:13:12.772 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.772 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.772 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.772 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.491237+0000", 2020-01-16T00:13:12.773 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0022862820000000002 2020-01-16T00:13:12.773 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.773 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.773 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.773 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.493523+0000", 2020-01-16T00:13:12.773 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5609999999999998e-05 2020-01-16T00:13:12.774 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.774 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.774 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:12.774 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.493578+0000", 2020-01-16T00:13:12.775 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2770999999999999e-05 2020-01-16T00:13:12.775 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.775 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.775 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.777 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.493641+0000", 2020-01-16T00:13:12.777 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0081599020000000001 2020-01-16T00:13:12.777 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.778 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.778 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.778 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.778 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.778 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:1279 3.0 e26/20 3:05b2bf53:::smithi12313934-37:head v 26'137, mlcod=26'135)", 2020-01-16T00:13:12.778 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:43.082160+0000", 2020-01-16T00:13:12.779 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 29.653885648999999, 2020-01-16T00:13:12.780 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0091277130000000008, 2020-01-16T00:13:12.780 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.780 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:12.780 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.781 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.781 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.781 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.082160+0000", 2020-01-16T00:13:12.781 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.781 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.782 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.782 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.782 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.082160+0000", 2020-01-16T00:13:12.782 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999957 2020-01-16T00:13:12.782 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.782 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.783 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.783 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.082156+0000", 2020-01-16T00:13:12.783 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00046771999999999997 2020-01-16T00:13:12.783 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.783 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.783 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.783 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.082623+0000", 2020-01-16T00:13:12.783 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3099999999999998e-05 2020-01-16T00:13:12.784 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.784 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.784 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.784 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.082656+0000", 2020-01-16T00:13:12.784 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2063e-05 2020-01-16T00:13:12.784 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.784 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.784 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.785 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.082668+0000", 2020-01-16T00:13:12.785 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021980900000000002 2020-01-16T00:13:12.785 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.785 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.785 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.785 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.084867+0000", 2020-01-16T00:13:12.785 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00033252299999999999 2020-01-16T00:13:12.786 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.786 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.786 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.786 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.085199+0000", 2020-01-16T00:13:12.786 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0060364019999999997 2020-01-16T00:13:12.786 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.786 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.786 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:12.787 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.091235+0000", 2020-01-16T00:13:12.787 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2255e-05 2020-01-16T00:13:12.787 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.787 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.787 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.787 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.091288+0000", 2020-01-16T00:13:12.787 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0091277130000000008 2020-01-16T00:13:12.788 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.788 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.788 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.788 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.788 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.788 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:47 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e28)", 2020-01-16T00:13:12.788 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:47.941839+0000", 2020-01-16T00:13:12.789 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 24.794206528, 2020-01-16T00:13:12.789 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.008400154, 2020-01-16T00:13:12.789 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.789 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:12.789 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.789 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.789 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.790 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.941839+0000", 2020-01-16T00:13:12.790 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.790 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.790 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.790 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.790 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.941839+0000", 2020-01-16T00:13:12.790 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:12.791 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.791 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.791 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.791 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.941837+0000", 2020-01-16T00:13:12.791 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00046009399999999998 2020-01-16T00:13:12.791 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.791 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.791 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.792 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942297+0000", 2020-01-16T00:13:12.792 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8628999999999999e-05 2020-01-16T00:13:12.792 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.792 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.792 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.792 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942335+0000", 2020-01-16T00:13:12.793 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1729999999999998e-06 2020-01-16T00:13:12.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.793 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.793 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.793 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942343+0000", 2020-01-16T00:13:12.793 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4845999999999997e-05 2020-01-16T00:13:12.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.794 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.794 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.794 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942387+0000", 2020-01-16T00:13:12.794 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6398999999999999e-05 2020-01-16T00:13:12.794 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.794 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.794 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.794 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942454+0000", 2020-01-16T00:13:12.795 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010786 2020-01-16T00:13:12.795 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.795 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.795 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:12.795 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942562+0000", 2020-01-16T00:13:12.795 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0064317760000000002 2020-01-16T00:13:12.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.796 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.796 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:12.796 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.948993+0000", 2020-01-16T00:13:12.796 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0011048849999999999 2020-01-16T00:13:12.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.796 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.797 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.797 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.950098+0000", 2020-01-16T00:13:12.797 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7064000000000003e-05 2020-01-16T00:13:12.797 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.797 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.797 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.798 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.950135+0000", 2020-01-16T00:13:12.798 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6589000000000003e-05 2020-01-16T00:13:12.798 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.798 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.798 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:12.798 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.950192+0000", 2020-01-16T00:13:12.798 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7286000000000001e-05 2020-01-16T00:13:12.798 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.799 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.799 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.799 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.950239+0000", 2020-01-16T00:13:12.799 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.008400154 2020-01-16T00:13:12.799 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.799 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.799 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.800 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.800 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.800 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:48 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e28)", 2020-01-16T00:13:12.800 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:47.942354+0000", 2020-01-16T00:13:12.800 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 24.793692113999999, 2020-01-16T00:13:12.800 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0080319870000000008, 2020-01-16T00:13:12.800 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.800 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:12.801 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.801 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.801 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.801 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942354+0000", 2020-01-16T00:13:12.801 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.801 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.801 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.802 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.802 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942354+0000", 2020-01-16T00:13:12.802 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:12.802 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.802 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.802 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.802 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942353+0000", 2020-01-16T00:13:12.803 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5825000000000002e-05 2020-01-16T00:13:12.803 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.803 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.803 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.803 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942409+0000", 2020-01-16T00:13:12.803 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5929999999999996e-06 2020-01-16T00:13:12.803 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.804 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.804 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.804 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942414+0000", 2020-01-16T00:13:12.804 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4199999999999999e-06 2020-01-16T00:13:12.804 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.805 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.805 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.805 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942418+0000", 2020-01-16T00:13:12.805 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0027594500000000001 2020-01-16T00:13:12.805 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.805 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.805 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.805 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.945177+0000", 2020-01-16T00:13:12.806 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.4040000000000002e-05 2020-01-16T00:13:12.806 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.806 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.806 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.806 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.945261+0000", 2020-01-16T00:13:12.806 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012906699999999999 2020-01-16T00:13:12.807 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.807 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.807 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:12.807 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.945390+0000", 2020-01-16T00:13:12.808 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0047161060000000003 2020-01-16T00:13:12.808 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.808 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.808 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:12.808 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.950106+0000", 2020-01-16T00:13:12.809 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016987400000000001 2020-01-16T00:13:12.809 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.809 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.809 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.809 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.950276+0000", 2020-01-16T00:13:12.809 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6329000000000002e-05 2020-01-16T00:13:12.810 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.810 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.810 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.810 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.950303+0000", 2020-01-16T00:13:12.810 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1443000000000003e-05 2020-01-16T00:13:12.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.811 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.811 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:12.811 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.950344+0000", 2020-01-16T00:13:12.812 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1486000000000002e-05 2020-01-16T00:13:12.812 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.812 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.812 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.812 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.950386+0000", 2020-01-16T00:13:12.813 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0080319870000000008 2020-01-16T00:13:12.813 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.813 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.813 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.814 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.814 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.814 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:55 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e30)", 2020-01-16T00:13:12.814 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.736036+0000", 2020-01-16T00:13:12.814 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 20.000009724000002, 2020-01-16T00:13:12.815 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010974536, 2020-01-16T00:13:12.815 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.815 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:12.815 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.816 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.816 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.816 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736036+0000", 2020-01-16T00:13:12.816 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.816 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.817 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.817 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.817 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736036+0000", 2020-01-16T00:13:12.817 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:12.818 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.818 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.818 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.818 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736034+0000", 2020-01-16T00:13:12.818 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043589699999999998 2020-01-16T00:13:12.819 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.819 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.819 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.819 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736469+0000", 2020-01-16T00:13:12.820 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9882e-05 2020-01-16T00:13:12.820 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.820 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.820 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.820 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736489+0000", 2020-01-16T00:13:12.821 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8339999999999999e-06 2020-01-16T00:13:12.821 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.821 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.821 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.822 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736497+0000", 2020-01-16T00:13:12.822 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5354999999999999e-05 2020-01-16T00:13:12.822 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.822 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.822 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.823 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736572+0000", 2020-01-16T00:13:12.823 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5228000000000004e-05 2020-01-16T00:13:12.823 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.823 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.824 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.824 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736668+0000", 2020-01-16T00:13:12.824 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017212999999999999 2020-01-16T00:13:12.824 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.824 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.825 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:12.825 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736840+0000", 2020-01-16T00:13:12.825 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0099057020000000006 2020-01-16T00:13:12.825 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.826 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.826 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.826 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746746+0000", 2020-01-16T00:13:12.826 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0586000000000005e-05 2020-01-16T00:13:12.826 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.827 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.827 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:12.827 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746816+0000", 2020-01-16T00:13:12.827 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2445000000000002e-05 2020-01-16T00:13:12.827 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.828 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.828 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.828 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746889+0000", 2020-01-16T00:13:12.828 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6839000000000002e-05 2020-01-16T00:13:12.829 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.829 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.829 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:12.829 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746955+0000", 2020-01-16T00:13:12.829 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5074000000000002e-05 2020-01-16T00:13:12.830 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.830 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.830 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.830 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.747011+0000", 2020-01-16T00:13:12.831 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010974536 2020-01-16T00:13:12.831 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.831 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.831 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.832 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.832 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.832 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:56 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e30)", 2020-01-16T00:13:12.832 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.736514+0000", 2020-01-16T00:13:12.833 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 19.999531889, 2020-01-16T00:13:12.833 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011660584999999999, 2020-01-16T00:13:12.833 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.833 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:12.833 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.834 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.834 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.834 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736514+0000", 2020-01-16T00:13:12.834 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.835 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.835 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.835 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.835 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736514+0000", 2020-01-16T00:13:12.835 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:12.836 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.836 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.836 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.836 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736513+0000", 2020-01-16T00:13:12.837 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9677000000000004e-05 2020-01-16T00:13:12.837 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.837 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.837 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.837 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736583+0000", 2020-01-16T00:13:12.837 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5269999999999998e-06 2020-01-16T00:13:12.838 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.838 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.838 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.838 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736591+0000", 2020-01-16T00:13:12.839 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5769999999999996e-06 2020-01-16T00:13:12.839 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.839 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.839 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.839 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736599+0000", 2020-01-16T00:13:12.840 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0053593310000000002 2020-01-16T00:13:12.840 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.840 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.840 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.840 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.741958+0000", 2020-01-16T00:13:12.840 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014164099999999999 2020-01-16T00:13:12.841 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.841 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.841 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.841 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.742100+0000", 2020-01-16T00:13:12.841 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020126000000000001 2020-01-16T00:13:12.841 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.841 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.842 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:12.842 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.742301+0000", 2020-01-16T00:13:12.842 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0045052859999999998 2020-01-16T00:13:12.842 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.842 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.842 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.842 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746806+0000", 2020-01-16T00:13:12.842 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00070802699999999996 2020-01-16T00:13:12.843 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.843 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.843 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:12.843 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.747514+0000", 2020-01-16T00:13:12.843 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00059247200000000003 2020-01-16T00:13:12.843 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.843 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.843 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.844 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748107+0000", 2020-01-16T00:13:12.844 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1870999999999999e-05 2020-01-16T00:13:12.844 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.844 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.844 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:12.844 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748149+0000", 2020-01-16T00:13:12.844 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5831000000000001e-05 2020-01-16T00:13:12.845 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.845 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.845 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.845 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748174+0000", 2020-01-16T00:13:12.845 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011660584999999999 2020-01-16T00:13:12.845 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.845 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.846 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.846 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.846 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.846 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:68 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 56832~1408000] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e34)", 2020-01-16T00:13:12.846 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:03.257978+0000", 2020-01-16T00:13:12.846 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 9.4780673390000008, 2020-01-16T00:13:12.846 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0084778949999999992, 2020-01-16T00:13:12.846 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.847 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:12.847 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.847 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.847 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.847 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.257978+0000", 2020-01-16T00:13:12.847 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.847 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.847 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.848 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.848 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.257978+0000", 2020-01-16T00:13:12.848 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:12.848 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.848 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.848 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.848 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.257977+0000", 2020-01-16T00:13:12.848 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00042032400000000002 2020-01-16T00:13:12.849 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.849 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.849 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.849 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258397+0000", 2020-01-16T00:13:12.849 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5951999999999999e-05 2020-01-16T00:13:12.849 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.849 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.850 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.850 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258413+0000", 2020-01-16T00:13:12.850 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0729999999999999e-06 2020-01-16T00:13:12.850 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.850 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.850 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.850 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258420+0000", 2020-01-16T00:13:12.850 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013206700000000001 2020-01-16T00:13:12.851 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.851 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.851 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.851 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258552+0000", 2020-01-16T00:13:12.851 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2818999999999996e-05 2020-01-16T00:13:12.851 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.852 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.852 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.852 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258615+0000", 2020-01-16T00:13:12.852 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012016899999999999 2020-01-16T00:13:12.852 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.852 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.853 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:12.853 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258735+0000", 2020-01-16T00:13:12.853 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052078480000000002 2020-01-16T00:13:12.853 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.853 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.853 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:12.853 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.263943+0000", 2020-01-16T00:13:12.854 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0019258649999999999 2020-01-16T00:13:12.854 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.854 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.854 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.854 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.265869+0000", 2020-01-16T00:13:12.854 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00051785399999999999 2020-01-16T00:13:12.855 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.855 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.855 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.855 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.266387+0000", 2020-01-16T00:13:12.855 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0459e-05 2020-01-16T00:13:12.856 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.856 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.856 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:12.856 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.266427+0000", 2020-01-16T00:13:12.856 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.919e-05 2020-01-16T00:13:12.856 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.856 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.857 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.857 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.266456+0000", 2020-01-16T00:13:12.857 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0084778949999999992 2020-01-16T00:13:12.858 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.858 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.858 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.858 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.859 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.859 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:69 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 1922048~118272] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e34)", 2020-01-16T00:13:12.859 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:03.258431+0000", 2020-01-16T00:13:12.860 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 9.4776143079999997, 2020-01-16T00:13:12.860 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0092806959999999997, 2020-01-16T00:13:12.860 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.860 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:12.860 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.860 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.860 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.860 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258431+0000", 2020-01-16T00:13:12.861 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.861 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.861 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.861 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.861 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258431+0000", 2020-01-16T00:13:12.861 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:12.861 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.862 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.862 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.862 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258431+0000", 2020-01-16T00:13:12.862 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3875000000000003e-05 2020-01-16T00:13:12.862 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.862 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.862 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.862 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258474+0000", 2020-01-16T00:13:12.863 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1040000000000007e-06 2020-01-16T00:13:12.863 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.863 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.863 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.863 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258483+0000", 2020-01-16T00:13:12.863 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6720000000000003e-06 2020-01-16T00:13:12.863 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.864 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.864 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.864 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258487+0000", 2020-01-16T00:13:12.864 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020507110000000002 2020-01-16T00:13:12.864 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.864 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.864 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.865 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.260538+0000", 2020-01-16T00:13:12.865 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017511100000000001 2020-01-16T00:13:12.865 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.865 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.865 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.865 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.260713+0000", 2020-01-16T00:13:12.865 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018914100000000001 2020-01-16T00:13:12.865 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.866 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.866 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:12.866 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.260902+0000", 2020-01-16T00:13:12.866 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.004582282 2020-01-16T00:13:12.866 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.866 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.866 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:12.866 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.265485+0000", 2020-01-16T00:13:12.867 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021355329999999998 2020-01-16T00:13:12.867 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.867 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.867 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.867 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.267620+0000", 2020-01-16T00:13:12.867 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1495999999999998e-05 2020-01-16T00:13:12.867 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.867 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.868 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.868 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.267652+0000", 2020-01-16T00:13:12.868 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5474e-05 2020-01-16T00:13:12.868 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.868 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.868 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:12.868 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.267687+0000", 2020-01-16T00:13:12.868 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5055e-05 2020-01-16T00:13:12.869 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.869 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.869 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.869 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.267712+0000", 2020-01-16T00:13:12.869 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0092806959999999997 2020-01-16T00:13:12.869 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.869 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.870 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.870 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.870 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.870 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:70 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e35)", 2020-01-16T00:13:12.870 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.985494+0000", 2020-01-16T00:13:12.870 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 6.7505520929999996, 2020-01-16T00:13:12.870 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013746267, 2020-01-16T00:13:12.870 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.871 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:12.871 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.871 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.871 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.871 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985494+0000", 2020-01-16T00:13:12.871 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.871 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.871 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.872 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.872 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985494+0000", 2020-01-16T00:13:12.872 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:12.872 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.872 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.872 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.872 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985490+0000", 2020-01-16T00:13:12.873 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00077792099999999999 2020-01-16T00:13:12.873 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.873 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.873 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.873 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986268+0000", 2020-01-16T00:13:12.873 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9718000000000001e-05 2020-01-16T00:13:12.874 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.874 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.874 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.874 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986318+0000", 2020-01-16T00:13:12.874 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.4349999999999997e-06 2020-01-16T00:13:12.874 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.874 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.874 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.875 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986326+0000", 2020-01-16T00:13:12.875 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3298999999999999e-05 2020-01-16T00:13:12.875 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.875 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.875 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.875 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986380+0000", 2020-01-16T00:13:12.875 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4969999999999995e-05 2020-01-16T00:13:12.875 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.876 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.876 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.876 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986455+0000", 2020-01-16T00:13:12.876 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000110587 2020-01-16T00:13:12.876 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.876 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.876 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:12.877 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986565+0000", 2020-01-16T00:13:12.877 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063931600000000002 2020-01-16T00:13:12.877 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.877 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.877 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:12.877 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.992958+0000", 2020-01-16T00:13:12.878 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013008480000000001 2020-01-16T00:13:12.878 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.878 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.878 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.878 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.994259+0000", 2020-01-16T00:13:12.878 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048399979999999999 2020-01-16T00:13:12.878 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.878 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.879 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.879 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999099+0000", 2020-01-16T00:13:12.879 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1874999999999998e-05 2020-01-16T00:13:12.879 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.879 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.879 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:12.879 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999181+0000", 2020-01-16T00:13:12.880 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8865e-05 2020-01-16T00:13:12.880 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.880 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.880 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.880 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999240+0000", 2020-01-16T00:13:12.880 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013746267 2020-01-16T00:13:12.880 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.880 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.881 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.881 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.881 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.881 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:71 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e35)", 2020-01-16T00:13:12.881 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.986341+0000", 2020-01-16T00:13:12.881 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 6.7497047319999997, 2020-01-16T00:13:12.881 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013252807, 2020-01-16T00:13:12.882 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.882 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:12.883 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.883 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.883 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.883 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986341+0000", 2020-01-16T00:13:12.883 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.884 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.884 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.884 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.884 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986341+0000", 2020-01-16T00:13:12.884 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:12.885 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.885 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.885 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.885 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986340+0000", 2020-01-16T00:13:12.885 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000103867 2020-01-16T00:13:12.885 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.886 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.886 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.886 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986444+0000", 2020-01-16T00:13:12.886 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0104e-05 2020-01-16T00:13:12.886 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.886 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.886 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.887 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986454+0000", 2020-01-16T00:13:12.887 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4220000000000002e-06 2020-01-16T00:13:12.887 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.887 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.887 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.887 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986458+0000", 2020-01-16T00:13:12.887 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033523070000000001 2020-01-16T00:13:12.888 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.888 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.888 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.888 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.989811+0000", 2020-01-16T00:13:12.888 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3296999999999999e-05 2020-01-16T00:13:12.888 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.889 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.889 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.889 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.989894+0000", 2020-01-16T00:13:12.889 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000117701 2020-01-16T00:13:12.889 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.889 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.890 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:12.890 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.990012+0000", 2020-01-16T00:13:12.890 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0032873490000000002 2020-01-16T00:13:12.890 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.890 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.890 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:12.890 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993299+0000", 2020-01-16T00:13:12.891 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002422619 2020-01-16T00:13:12.891 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.891 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.891 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.891 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.995722+0000", 2020-01-16T00:13:12.891 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0037839589999999999 2020-01-16T00:13:12.891 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.891 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.892 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.892 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999506+0000", 2020-01-16T00:13:12.892 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8559999999999997e-05 2020-01-16T00:13:12.892 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.892 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.892 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:12.892 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999544+0000", 2020-01-16T00:13:12.893 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9599999999999999e-05 2020-01-16T00:13:12.893 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.893 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.893 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.893 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999594+0000", 2020-01-16T00:13:12.893 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013252807 2020-01-16T00:13:12.893 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.894 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.894 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.894 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.894 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.894 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:77 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e36)", 2020-01-16T00:13:12.894 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.555854+0000", 2020-01-16T00:13:12.894 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.1801916429999997, 2020-01-16T00:13:12.895 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010455002999999999, 2020-01-16T00:13:12.895 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.895 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:12.895 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.895 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.895 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.895 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555854+0000", 2020-01-16T00:13:12.896 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.896 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.896 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.896 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.896 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555854+0000", 2020-01-16T00:13:12.896 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:12.896 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.896 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.897 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.897 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555851+0000", 2020-01-16T00:13:12.897 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00048310200000000001 2020-01-16T00:13:12.897 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.897 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.897 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.897 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556334+0000", 2020-01-16T00:13:12.898 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2232e-05 2020-01-16T00:13:12.898 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.898 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.898 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.898 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556356+0000", 2020-01-16T00:13:12.898 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9750000000000007e-06 2020-01-16T00:13:12.898 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.899 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.899 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.899 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556364+0000", 2020-01-16T00:13:12.899 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8872999999999997e-05 2020-01-16T00:13:12.899 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.899 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.900 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.900 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556423+0000", 2020-01-16T00:13:12.900 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8687000000000001e-05 2020-01-16T00:13:12.900 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.900 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.900 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.900 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556502+0000", 2020-01-16T00:13:12.900 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013235300000000001 2020-01-16T00:13:12.901 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.901 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.901 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:12.901 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556634+0000", 2020-01-16T00:13:12.901 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0059770520000000001 2020-01-16T00:13:12.901 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.901 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.901 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:12.902 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.562611+0000", 2020-01-16T00:13:12.902 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029492820000000001 2020-01-16T00:13:12.902 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.902 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.902 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.903 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.565561+0000", 2020-01-16T00:13:12.903 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00060620300000000004 2020-01-16T00:13:12.903 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.903 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.903 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.903 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566167+0000", 2020-01-16T00:13:12.903 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3831000000000004e-05 2020-01-16T00:13:12.904 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.904 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.904 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:12.904 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566251+0000", 2020-01-16T00:13:12.904 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8328000000000001e-05 2020-01-16T00:13:12.904 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.904 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.905 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.905 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566309+0000", 2020-01-16T00:13:12.905 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010455002999999999 2020-01-16T00:13:12.905 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.905 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.905 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.905 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.905 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.906 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:78 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e36)", 2020-01-16T00:13:12.906 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.556378+0000", 2020-01-16T00:13:12.906 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.1796677100000004, 2020-01-16T00:13:12.906 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010396311, 2020-01-16T00:13:12.906 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.906 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:12.906 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.907 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.907 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.907 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556378+0000", 2020-01-16T00:13:12.907 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.907 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.907 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.907 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.907 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556378+0000", 2020-01-16T00:13:12.908 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:12.908 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.908 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.909 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.909 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556377+0000", 2020-01-16T00:13:12.909 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4307000000000001e-05 2020-01-16T00:13:12.909 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.910 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.910 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.910 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556422+0000", 2020-01-16T00:13:12.910 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9620000000000005e-06 2020-01-16T00:13:12.910 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.910 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.910 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.911 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556432+0000", 2020-01-16T00:13:12.911 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5979999999999999e-06 2020-01-16T00:13:12.911 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.911 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.911 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.911 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556436+0000", 2020-01-16T00:13:12.911 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021059469999999999 2020-01-16T00:13:12.911 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.912 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.912 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.912 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.558542+0000", 2020-01-16T00:13:12.912 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000194458 2020-01-16T00:13:12.912 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.912 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.912 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.912 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.558737+0000", 2020-01-16T00:13:12.913 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00032867900000000002 2020-01-16T00:13:12.913 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.913 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.913 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:12.913 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.559065+0000", 2020-01-16T00:13:12.913 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052547499999999999 2020-01-16T00:13:12.913 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.913 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.914 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:12.914 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.564320+0000", 2020-01-16T00:13:12.914 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001565213 2020-01-16T00:13:12.914 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.914 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.914 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.914 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.565885+0000", 2020-01-16T00:13:12.914 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00076994300000000004 2020-01-16T00:13:12.915 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.915 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.915 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.915 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566655+0000", 2020-01-16T00:13:12.915 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7533000000000005e-05 2020-01-16T00:13:12.915 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.915 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.916 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:12.916 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566723+0000", 2020-01-16T00:13:12.916 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1616999999999997e-05 2020-01-16T00:13:12.916 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.916 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.916 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.916 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566774+0000", 2020-01-16T00:13:12.916 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010396311 2020-01-16T00:13:12.917 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.917 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.917 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.917 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.917 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.917 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:72 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 56832~1408000] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e37)", 2020-01-16T00:13:12.917 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.578753+0000", 2020-01-16T00:13:12.917 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.157292407, 2020-01-16T00:13:12.918 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011826918, 2020-01-16T00:13:12.918 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.918 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:12.918 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.918 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.918 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.918 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578753+0000", 2020-01-16T00:13:12.919 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.919 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.919 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.919 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.919 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578753+0000", 2020-01-16T00:13:12.919 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:12.919 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.919 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.919 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.920 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578751+0000", 2020-01-16T00:13:12.920 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00042457699999999998 2020-01-16T00:13:12.920 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.920 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.920 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.920 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579175+0000", 2020-01-16T00:13:12.920 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5404999999999999e-05 2020-01-16T00:13:12.920 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.921 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.921 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.921 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579191+0000", 2020-01-16T00:13:12.921 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6149999999999996e-06 2020-01-16T00:13:12.921 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.921 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.921 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.922 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579196+0000", 2020-01-16T00:13:12.922 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000104794 2020-01-16T00:13:12.922 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.922 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.922 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.922 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579301+0000", 2020-01-16T00:13:12.923 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020947199999999999 2020-01-16T00:13:12.923 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.923 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.923 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.923 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579511+0000", 2020-01-16T00:13:12.923 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034328400000000002 2020-01-16T00:13:12.923 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.924 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.924 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:12.924 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579854+0000", 2020-01-16T00:13:12.924 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.005078117 2020-01-16T00:13:12.924 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.924 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.924 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:12.924 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.584932+0000", 2020-01-16T00:13:12.925 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0014702160000000001 2020-01-16T00:13:12.925 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.925 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.925 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.925 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.586402+0000", 2020-01-16T00:13:12.925 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040623589999999998 2020-01-16T00:13:12.925 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.925 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.926 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.926 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590465+0000", 2020-01-16T00:13:12.926 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4478000000000002e-05 2020-01-16T00:13:12.926 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.926 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.926 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:12.926 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590529+0000", 2020-01-16T00:13:12.927 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1121000000000001e-05 2020-01-16T00:13:12.927 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.927 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.927 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.927 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590580+0000", 2020-01-16T00:13:12.927 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011826918 2020-01-16T00:13:12.927 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.927 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.928 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.928 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.928 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.928 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:73 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 1922048~118272] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e37)", 2020-01-16T00:13:12.928 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:13:12.928 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.15683960999999999, 2020-01-16T00:13:12.928 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011750764, 2020-01-16T00:13:12.929 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.929 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:12.929 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.929 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.929 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.929 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:13:12.929 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.929 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.930 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.930 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.930 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:13:12.930 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:12.930 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.930 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.930 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.930 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:13:12.931 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2780000000000001e-05 2020-01-16T00:13:12.931 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.931 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.931 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.931 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579238+0000", 2020-01-16T00:13:12.931 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4341999999999999e-05 2020-01-16T00:13:12.931 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.932 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.932 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.932 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579253+0000", 2020-01-16T00:13:12.932 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1920000000000001e-06 2020-01-16T00:13:12.932 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.932 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.932 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.932 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579256+0000", 2020-01-16T00:13:12.933 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003224902 2020-01-16T00:13:12.933 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.933 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.933 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.933 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.582481+0000", 2020-01-16T00:13:12.933 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000180729 2020-01-16T00:13:12.933 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.934 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.934 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.934 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.582662+0000", 2020-01-16T00:13:12.934 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00025106100000000001 2020-01-16T00:13:12.934 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.934 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.934 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:12.934 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.582913+0000", 2020-01-16T00:13:12.935 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020814449999999999 2020-01-16T00:13:12.935 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.935 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.935 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:12.935 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.584994+0000", 2020-01-16T00:13:12.935 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001717243 2020-01-16T00:13:12.935 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.935 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.936 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.936 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.586711+0000", 2020-01-16T00:13:12.936 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041349869999999997 2020-01-16T00:13:12.936 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.936 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.936 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.936 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590846+0000", 2020-01-16T00:13:12.936 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3565000000000006e-05 2020-01-16T00:13:12.937 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.937 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.937 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:12.937 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590910+0000", 2020-01-16T00:13:12.937 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7002999999999997e-05 2020-01-16T00:13:12.937 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.937 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.938 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.938 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590957+0000", 2020-01-16T00:13:12.938 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011750764 2020-01-16T00:13:12.938 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.938 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.938 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.938 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.938 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.939 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:89 2.1 2:9b8617ce:::smithi12313934-38 low_tier:head [write 69632~773120] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e37)", 2020-01-16T00:13:12.939 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.593701+0000", 2020-01-16T00:13:12.939 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.14234498400000001, 2020-01-16T00:13:12.939 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010026929, 2020-01-16T00:13:12.939 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.939 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:12.939 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.940 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.940 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.940 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593701+0000", 2020-01-16T00:13:12.940 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.940 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.940 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.941 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.941 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593701+0000", 2020-01-16T00:13:12.941 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:12.941 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.941 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.941 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.941 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593699+0000", 2020-01-16T00:13:12.941 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000237564 2020-01-16T00:13:12.942 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.942 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.942 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.942 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593936+0000", 2020-01-16T00:13:12.942 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2666e-05 2020-01-16T00:13:12.942 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.942 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.943 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.943 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593949+0000", 2020-01-16T00:13:12.943 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0089999999999998e-06 2020-01-16T00:13:12.943 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.943 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.943 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.943 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593954+0000", 2020-01-16T00:13:12.944 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5813000000000003e-05 2020-01-16T00:13:12.944 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.944 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.944 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.944 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.594000+0000", 2020-01-16T00:13:12.944 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010155299999999999 2020-01-16T00:13:12.944 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.944 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.945 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.945 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.594101+0000", 2020-01-16T00:13:12.945 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000203807 2020-01-16T00:13:12.945 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.945 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.945 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:12.945 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.594305+0000", 2020-01-16T00:13:12.945 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021655390000000002 2020-01-16T00:13:12.946 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.946 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.946 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.946 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.596471+0000", 2020-01-16T00:13:12.946 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0070645320000000001 2020-01-16T00:13:12.946 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.946 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.946 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.947 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603535+0000", 2020-01-16T00:13:12.947 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7577999999999998e-05 2020-01-16T00:13:12.947 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.947 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.947 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:12.947 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603573+0000", 2020-01-16T00:13:12.948 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9933999999999997e-05 2020-01-16T00:13:12.948 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.948 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.948 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:12.948 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603653+0000", 2020-01-16T00:13:12.948 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4993000000000005e-05 2020-01-16T00:13:12.949 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.949 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.949 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.949 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603728+0000", 2020-01-16T00:13:12.949 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010026929 2020-01-16T00:13:12.949 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.949 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.950 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.950 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.950 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.950 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:90 2.1 2:9b8617ce:::smithi12313934-38 low_tier:head [write 928256~830464] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e37)", 2020-01-16T00:13:12.950 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.596686+0000", 2020-01-16T00:13:12.950 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.139359911, 2020-01-16T00:13:12.951 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0072411560000000003, 2020-01-16T00:13:12.951 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.951 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:12.951 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.952 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.953 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.596686+0000", 2020-01-16T00:13:12.953 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.953 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.953 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.954 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.954 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.596686+0000", 2020-01-16T00:13:12.954 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:12.954 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.954 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.954 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.954 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.596684+0000", 2020-01-16T00:13:12.954 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00025510799999999998 2020-01-16T00:13:12.955 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.955 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.955 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.955 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.596939+0000", 2020-01-16T00:13:12.955 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6276000000000001e-05 2020-01-16T00:13:12.955 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.955 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.955 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.956 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.596955+0000", 2020-01-16T00:13:12.956 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.754e-06 2020-01-16T00:13:12.956 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.956 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.956 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.956 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.596962+0000", 2020-01-16T00:13:12.956 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00024224500000000001 2020-01-16T00:13:12.957 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.957 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.957 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.957 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.597204+0000", 2020-01-16T00:13:12.957 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000106508 2020-01-16T00:13:12.957 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.957 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.957 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.958 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.597311+0000", 2020-01-16T00:13:12.958 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000217211 2020-01-16T00:13:12.958 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.958 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.958 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:12.958 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.597528+0000", 2020-01-16T00:13:12.958 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0059314939999999998 2020-01-16T00:13:12.958 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.959 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.959 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.959 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603460+0000", 2020-01-16T00:13:12.959 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000298638 2020-01-16T00:13:12.959 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.959 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.959 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:12.959 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603758+0000", 2020-01-16T00:13:12.960 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6199999999999995e-05 2020-01-16T00:13:12.960 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.960 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.960 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.960 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603834+0000", 2020-01-16T00:13:12.960 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2926999999999998e-05 2020-01-16T00:13:12.960 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.960 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.961 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:12.961 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603867+0000", 2020-01-16T00:13:12.961 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9573e-05 2020-01-16T00:13:12.961 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.961 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.961 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.961 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603927+0000", 2020-01-16T00:13:12.961 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0072411560000000003 2020-01-16T00:13:12.962 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.962 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.962 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.962 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.962 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.962 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:91 2.1 2:9b8617ce:::smithi12313934-38 low_tier:head [write 1915392~590334] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e37)", 2020-01-16T00:13:12.962 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.596978+0000", 2020-01-16T00:13:12.962 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.13906728700000001, 2020-01-16T00:13:12.963 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0079576309999999997, 2020-01-16T00:13:12.963 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.963 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:12.963 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.963 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.963 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.963 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.596978+0000", 2020-01-16T00:13:12.964 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.964 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.964 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.964 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.964 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.596978+0000", 2020-01-16T00:13:12.964 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:12.964 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.964 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.965 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.965 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.596978+0000", 2020-01-16T00:13:12.965 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000170557 2020-01-16T00:13:12.965 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.965 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.965 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.965 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.597148+0000", 2020-01-16T00:13:12.965 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5519999999999999e-06 2020-01-16T00:13:12.966 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.966 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.966 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.966 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.597156+0000", 2020-01-16T00:13:12.966 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7520000000000004e-06 2020-01-16T00:13:12.966 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.966 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.966 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.967 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.597161+0000", 2020-01-16T00:13:12.967 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0028888260000000002 2020-01-16T00:13:12.967 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.967 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.967 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.967 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.600049+0000", 2020-01-16T00:13:12.967 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7899999999999994e-05 2020-01-16T00:13:12.967 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.968 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.968 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.968 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.600147+0000", 2020-01-16T00:13:12.968 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000169877 2020-01-16T00:13:12.968 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.968 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.968 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:12.968 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.600317+0000", 2020-01-16T00:13:12.969 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0043677259999999997 2020-01-16T00:13:12.969 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.969 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.969 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.969 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.604685+0000", 2020-01-16T00:13:12.969 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013208999999999999 2020-01-16T00:13:12.969 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.970 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.970 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.970 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.604817+0000", 2020-01-16T00:13:12.970 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4851000000000001e-05 2020-01-16T00:13:12.970 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.970 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.970 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:12.970 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.604832+0000", 2020-01-16T00:13:12.971 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8831e-05 2020-01-16T00:13:12.971 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.971 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.971 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:12.971 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.604891+0000", 2020-01-16T00:13:12.971 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5411000000000003e-05 2020-01-16T00:13:12.971 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.972 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.972 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.972 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.604936+0000", 2020-01-16T00:13:12.972 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0079576309999999997 2020-01-16T00:13:12.972 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.972 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.972 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.973 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.973 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.973 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.5.13:48 2.1 2:b9d70087:::smithi12313934-32 low_tier:head [write 605696~877568] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e37)", 2020-01-16T00:13:12.973 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.629299+0000", 2020-01-16T00:13:12.973 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.106746415, 2020-01-16T00:13:12.973 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.007325748, 2020-01-16T00:13:12.974 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:12.974 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:12.974 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:12.974 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.974 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:12.974 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.629299+0000", 2020-01-16T00:13:12.974 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:12.974 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.975 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.975 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:12.975 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.629299+0000", 2020-01-16T00:13:12.975 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:12.975 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.975 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.975 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:12.975 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.629297+0000", 2020-01-16T00:13:12.976 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001146488 2020-01-16T00:13:12.976 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.976 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.976 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:12.976 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.630444+0000", 2020-01-16T00:13:12.976 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7578e-05 2020-01-16T00:13:12.976 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.976 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.977 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:12.977 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.630462+0000", 2020-01-16T00:13:12.977 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6639999999999998e-06 2020-01-16T00:13:12.977 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.977 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.977 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:12.977 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.630469+0000", 2020-01-16T00:13:12.977 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6004999999999997e-05 2020-01-16T00:13:12.978 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.978 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.978 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:12.978 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.630515+0000", 2020-01-16T00:13:12.978 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8400999999999998e-05 2020-01-16T00:13:12.978 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.978 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.978 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:12.979 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.630584+0000", 2020-01-16T00:13:12.979 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011452500000000001 2020-01-16T00:13:12.979 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.979 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.979 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:12.979 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.630698+0000", 2020-01-16T00:13:12.979 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056979459999999997 2020-01-16T00:13:12.979 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.980 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.980 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.980 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.636396+0000", 2020-01-16T00:13:12.980 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7283e-05 2020-01-16T00:13:12.980 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.980 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.980 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:12.980 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.636453+0000", 2020-01-16T00:13:12.981 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4666999999999997e-05 2020-01-16T00:13:12.981 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.981 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.981 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:12.981 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.636508+0000", 2020-01-16T00:13:12.981 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8066000000000003e-05 2020-01-16T00:13:12.981 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.981 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.982 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:12.982 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.636566+0000", 2020-01-16T00:13:12.982 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8925000000000002e-05 2020-01-16T00:13:12.982 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:12.982 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:12.982 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:12.982 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.636625+0000", 2020-01-16T00:13:12.982 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.007325748 2020-01-16T00:13:12.983 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.983 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.983 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.983 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:12.983 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:12.983 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:12.986 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2020-01-16T00:13:12.987 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:13:12.987 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:13:12.987 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:13:12.987 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:13:12.987 INFO:teuthology.orchestra.run.smithi123.stdout:osd.5: {} 2020-01-16T00:13:12.987 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:13:12.987 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:13:12.988 INFO:teuthology.orchestra.run.smithi123.stderr:noscrub is unset 2020-01-16T00:13:12.988 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-01-16T00:13:12.988 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-01-16T00:13:12.988 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-01-16T00:13:12.988 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-01-16T00:13:12.988 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-01-16T00:13:12.989 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: osd_enable_op_tracker = 'false' 2020-01-16T00:13:12.989 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-01-16T00:13:12.989 INFO:tasks.rados.rados.0.smithi123.stdout:1090: expect (ObjNum 220 snap 0 seq_num 220) 2020-01-16T00:13:12.989 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:12.980+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:13.086 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:13.082+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:13.104 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:13.114 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2020-01-16T00:13:13.187 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:13.182+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:13.230 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:13.233 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2020-01-16T00:13:13.253 INFO:tasks.rados.rados.0.smithi123.stdout:1092: finishing write tid 1 to smithi12313934-30 2020-01-16T00:13:13.253 INFO:tasks.rados.rados.0.smithi123.stdout:1092: finishing write tid 2 to smithi12313934-30 2020-01-16T00:13:13.253 INFO:tasks.rados.rados.0.smithi123.stdout:1092: finishing write tid 3 to smithi12313934-30 2020-01-16T00:13:13.253 INFO:tasks.rados.rados.0.smithi123.stdout:1092: finishing write tid 5 to smithi12313934-30 2020-01-16T00:13:13.253 INFO:tasks.rados.rados.0.smithi123.stdout:1092: finishing write tid 6 to smithi12313934-30 2020-01-16T00:13:13.253 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 30 v 107 (ObjNum 294 snap 0 seq_num 294) dirty exists 2020-01-16T00:13:13.254 INFO:tasks.rados.rados.0.smithi123.stdout:1092: left oid 30 (ObjNum 294 snap 0 seq_num 294) 2020-01-16T00:13:13.254 INFO:tasks.rados.rados.0.smithi123.stdout:1093: finishing write tid 1 to smithi12313934-43 2020-01-16T00:13:13.254 INFO:tasks.rados.rados.0.smithi123.stdout:1089: done (15 left) 2020-01-16T00:13:13.254 INFO:tasks.rados.rados.0.smithi123.stdout:1090: done (14 left) 2020-01-16T00:13:13.254 INFO:tasks.rados.rados.0.smithi123.stdout:1091: done (13 left) 2020-01-16T00:13:13.254 INFO:tasks.rados.rados.0.smithi123.stdout:1092: done (12 left) 2020-01-16T00:13:13.254 INFO:tasks.rados.rados.0.smithi123.stdout:1105: read oid 34 snap -1 2020-01-16T00:13:13.255 INFO:tasks.rados.rados.0.smithi123.stdout:1105: expect (ObjNum 146 snap 0 seq_num 146) 2020-01-16T00:13:13.255 INFO:tasks.rados.rados.0.smithi123.stdout:1106: write oid 27 current snap is 0 2020-01-16T00:13:13.255 INFO:tasks.rados.rados.0.smithi123.stdout:1106: seq_num 303 ranges {692521=538214,1931338=785088} 2020-01-16T00:13:13.270 INFO:tasks.rados.rados.0.smithi123.stdout:1106: writing smithi12313934-27 from 692521 to 1230735 tid 1 2020-01-16T00:13:13.279 INFO:tasks.rados.rados.0.smithi123.stdout:1106: writing smithi12313934-27 from 1931338 to 2716426 tid 2 2020-01-16T00:13:13.279 INFO:tasks.rados.rados.0.smithi123.stdout:1107: write oid 24 current snap is 0 2020-01-16T00:13:13.279 INFO:tasks.rados.rados.0.smithi123.stdout:1107: seq_num 304 ranges {709328=613463,1989820=645099} 2020-01-16T00:13:13.286 INFO:tasks.rados.rados.0.smithi123.stdout:1107: writing smithi12313934-24 from 709328 to 1322791 tid 1 2020-01-16T00:13:13.288 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:13.286+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:13.294 INFO:tasks.rados.rados.0.smithi123.stdout:1107: writing smithi12313934-24 from 1989820 to 2634919 tid 2 2020-01-16T00:13:13.294 INFO:tasks.rados.rados.0.smithi123.stdout:1108: read oid 8 snap -1 2020-01-16T00:13:13.294 INFO:tasks.rados.rados.0.smithi123.stdout:1108: expect (ObjNum 219 snap 0 seq_num 219) 2020-01-16T00:13:13.294 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:13.294 INFO:tasks.rados.rados.0.smithi123.stdout:1093: finishing write tid 2 to smithi12313934-43 2020-01-16T00:13:13.295 INFO:tasks.rados.rados.0.smithi123.stdout:1095: expect (ObjNum 266 snap 0 seq_num 266) 2020-01-16T00:13:13.345 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:13.350 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-01-16T00:13:13.395 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:13.392+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:13.460 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:13.469 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-01-16T00:13:13.505 INFO:tasks.rados.rados.0.smithi123.stdout:1093: finishing write tid 3 to smithi12313934-43 2020-01-16T00:13:13.505 INFO:tasks.rados.rados.0.smithi123.stdout:1094: expect (ObjNum 216 snap 0 seq_num 216) 2020-01-16T00:13:13.511 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:13.508+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:13.591 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:13.586+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:13.620 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:13.630 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-01-16T00:13:13.693 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:13.688+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:13.744 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:13.746 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-01-16T00:13:13.760 INFO:tasks.rados.rados.0.smithi123.stdout:1093: finishing write tid 5 to smithi12313934-43 2020-01-16T00:13:13.760 INFO:tasks.rados.rados.0.smithi123.stdout:1093: finishing write tid 6 to smithi12313934-43 2020-01-16T00:13:13.760 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 43 v 152 (ObjNum 295 snap 0 seq_num 295) dirty exists 2020-01-16T00:13:13.760 INFO:tasks.rados.rados.0.smithi123.stdout:1093: left oid 43 (ObjNum 295 snap 0 seq_num 295) 2020-01-16T00:13:13.761 INFO:tasks.rados.rados.0.smithi123.stdout:1096: finishing write tid 1 to smithi12313934-7 2020-01-16T00:13:13.761 INFO:tasks.rados.rados.0.smithi123.stdout:1096: finishing write tid 2 to smithi12313934-7 2020-01-16T00:13:13.761 INFO:tasks.rados.rados.0.smithi123.stdout:1096: finishing write tid 4 to smithi12313934-7 2020-01-16T00:13:13.761 INFO:tasks.rados.rados.0.smithi123.stdout:1096: finishing write tid 5 to smithi12313934-7 2020-01-16T00:13:13.761 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 7 v 162 (ObjNum 296 snap 0 seq_num 296) dirty exists 2020-01-16T00:13:13.761 INFO:tasks.rados.rados.0.smithi123.stdout:1096: left oid 7 (ObjNum 296 snap 0 seq_num 296) 2020-01-16T00:13:13.761 INFO:tasks.rados.rados.0.smithi123.stdout:1097: expect (ObjNum 270 snap 0 seq_num 270) 2020-01-16T00:13:13.793 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:13.788+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:13.894 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:13.890+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:13.898 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:13.908 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-01-16T00:13:13.929 INFO:tasks.rados.rados.0.smithi123.stdout:1098: finishing write tid 1 to smithi12313934-46 2020-01-16T00:13:13.930 INFO:tasks.rados.rados.0.smithi123.stdout:1098: finishing write tid 2 to smithi12313934-46 2020-01-16T00:13:13.930 INFO:tasks.rados.rados.0.smithi123.stdout:1093: done (15 left) 2020-01-16T00:13:13.930 INFO:tasks.rados.rados.0.smithi123.stdout:1094: done (14 left) 2020-01-16T00:13:13.930 INFO:tasks.rados.rados.0.smithi123.stdout:1095: done (13 left) 2020-01-16T00:13:13.930 INFO:tasks.rados.rados.0.smithi123.stdout:1096: done (12 left) 2020-01-16T00:13:13.930 INFO:tasks.rados.rados.0.smithi123.stdout:1097: done (11 left) 2020-01-16T00:13:13.931 INFO:tasks.rados.rados.0.smithi123.stdout:1109: write (excl) oid 35 current snap is 0 2020-01-16T00:13:13.931 INFO:tasks.rados.rados.0.smithi123.stdout:1109: seq_num 305 ranges {726135=405065,1764655=788757,3026248=99887} 2020-01-16T00:13:13.945 INFO:tasks.rados.rados.0.smithi123.stdout:1109: writing smithi12313934-35 from 726135 to 1131200 tid 1 2020-01-16T00:13:13.953 INFO:tasks.rados.rados.0.smithi123.stdout:1109: writing smithi12313934-35 from 1764655 to 2553412 tid 2 2020-01-16T00:13:13.957 INFO:tasks.rados.rados.0.smithi123.stdout:1109: writing smithi12313934-35 from 3026248 to 3126135 tid 3 2020-01-16T00:13:13.957 INFO:tasks.rados.rados.0.smithi123.stdout:1110: write oid 14 current snap is 0 2020-01-16T00:13:13.957 INFO:tasks.rados.rados.0.smithi123.stdout:1110: seq_num 306 ranges {742942=480314,1823137=648768} 2020-01-16T00:13:13.964 INFO:tasks.rados.rados.0.smithi123.stdout:1110: writing smithi12313934-14 from 742942 to 1223256 tid 1 2020-01-16T00:13:13.971 INFO:tasks.rados.rados.0.smithi123.stdout:1110: writing smithi12313934-14 from 1823137 to 2471905 tid 2 2020-01-16T00:13:13.971 INFO:tasks.rados.rados.0.smithi123.stdout:1111: write oid 49 current snap is 0 2020-01-16T00:13:13.972 INFO:tasks.rados.rados.0.smithi123.stdout:1111: seq_num 307 ranges {759749=555563,1765266=792426,2964741=195008} 2020-01-16T00:13:13.981 INFO:tasks.rados.rados.0.smithi123.stdout:1111: writing smithi12313934-49 from 759749 to 1315312 tid 1 2020-01-16T00:13:13.986 INFO:tasks.rados.rados.0.smithi123.stdout:1111: writing smithi12313934-49 from 1765266 to 2557692 tid 2 2020-01-16T00:13:13.989 INFO:tasks.rados.rados.0.smithi123.stdout:1111: writing smithi12313934-49 from 2964741 to 3159749 tid 3 2020-01-16T00:13:13.989 INFO:tasks.rados.rados.0.smithi123.stdout:1112: read oid 47 snap -1 2020-01-16T00:13:13.989 INFO:tasks.rados.rados.0.smithi123.stdout:1112: expect (ObjNum 250 snap 0 seq_num 250) 2020-01-16T00:13:13.990 INFO:tasks.rados.rados.0.smithi123.stdout:1113: write (excl) oid 16 current snap is 0 2020-01-16T00:13:13.990 INFO:tasks.rados.rados.0.smithi123.stdout:1113: seq_num 308 ranges {776556=630812,1823748=652437,3108517=68039} 2020-01-16T00:13:13.994 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:13.988+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:13.997 INFO:tasks.rados.rados.0.smithi123.stdout:1113: writing smithi12313934-16 from 776556 to 1407368 tid 1 2020-01-16T00:13:14.003 INFO:tasks.rados.rados.0.smithi123.stdout:1113: writing smithi12313934-16 from 1823748 to 2476185 tid 2 2020-01-16T00:13:14.008 INFO:tasks.rados.rados.0.smithi123.stdout:1113: writing smithi12313934-16 from 3108517 to 3176556 tid 3 2020-01-16T00:13:14.008 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:14.008 INFO:tasks.rados.rados.0.smithi123.stdout:1099: finishing write tid 1 to smithi12313934-38 2020-01-16T00:13:14.008 INFO:tasks.rados.rados.0.smithi123.stdout:1098: finishing write tid 4 to smithi12313934-46 2020-01-16T00:13:14.008 INFO:tasks.rados.rados.0.smithi123.stdout:1098: finishing write tid 5 to smithi12313934-46 2020-01-16T00:13:14.009 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 46 v 215 (ObjNum 297 snap 0 seq_num 297) dirty exists 2020-01-16T00:13:14.009 INFO:tasks.rados.rados.0.smithi123.stdout:1098: left oid 46 (ObjNum 297 snap 0 seq_num 297) 2020-01-16T00:13:14.009 INFO:tasks.rados.rados.0.smithi123.stdout:1099: finishing write tid 2 to smithi12313934-38 2020-01-16T00:13:14.009 INFO:tasks.rados.rados.0.smithi123.stdout:1099: finishing write tid 3 to smithi12313934-38 2020-01-16T00:13:14.009 INFO:tasks.rados.rados.0.smithi123.stdout:1099: finishing write tid 5 to smithi12313934-38 2020-01-16T00:13:14.009 INFO:tasks.rados.rados.0.smithi123.stdout:1098: done (15 left) 2020-01-16T00:13:14.010 INFO:tasks.rados.rados.0.smithi123.stdout:1114: read oid 26 snap -1 2020-01-16T00:13:14.010 INFO:tasks.rados.rados.0.smithi123.stdout:1114: expect (ObjNum 263 snap 0 seq_num 263) 2020-01-16T00:13:14.010 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:14.010 INFO:tasks.rados.rados.0.smithi123.stdout:1100: finishing write tid 1 to smithi12313934-1 2020-01-16T00:13:14.010 INFO:tasks.rados.rados.0.smithi123.stdout:1099: finishing write tid 6 to smithi12313934-38 2020-01-16T00:13:14.010 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 38 v 112 (ObjNum 298 snap 0 seq_num 298) dirty exists 2020-01-16T00:13:14.010 INFO:tasks.rados.rados.0.smithi123.stdout:1099: left oid 38 (ObjNum 298 snap 0 seq_num 298) 2020-01-16T00:13:14.010 INFO:tasks.rados.rados.0.smithi123.stdout:1100: finishing write tid 2 to smithi12313934-1 2020-01-16T00:13:14.011 INFO:tasks.rados.rados.0.smithi123.stdout:1099: done (15 left) 2020-01-16T00:13:14.011 INFO:tasks.rados.rados.0.smithi123.stdout:1115: write oid 45 current snap is 0 2020-01-16T00:13:14.011 INFO:tasks.rados.rados.0.smithi123.stdout:1115: seq_num 309 ranges {793363=706061,2282230=796095} 2020-01-16T00:13:14.015 INFO:tasks.rados.rados.0.smithi123.stdout:1115: writing smithi12313934-45 from 793363 to 1499424 tid 1 2020-01-16T00:13:14.024 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:14.026 INFO:tasks.rados.rados.0.smithi123.stdout:1115: writing smithi12313934-45 from 2282230 to 3078325 tid 2 2020-01-16T00:13:14.026 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:14.026 INFO:tasks.rados.rados.0.smithi123.stdout:1100: finishing write tid 3 to smithi12313934-1 2020-01-16T00:13:14.026 INFO:tasks.rados.rados.0.smithi123.stdout:1100: finishing write tid 5 to smithi12313934-1 2020-01-16T00:13:14.026 INFO:tasks.rados.rados.0.smithi123.stdout:1100: finishing write tid 6 to smithi12313934-1 2020-01-16T00:13:14.026 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 1 v 219 (ObjNum 299 snap 0 seq_num 299) dirty exists 2020-01-16T00:13:14.026 INFO:tasks.rados.rados.0.smithi123.stdout:1100: left oid 1 (ObjNum 299 snap 0 seq_num 299) 2020-01-16T00:13:14.027 INFO:tasks.rados.rados.0.smithi123.stdout:1101: finishing write tid 1 to smithi12313934-32 2020-01-16T00:13:14.027 INFO:tasks.rados.rados.0.smithi123.stdout:1101: finishing write tid 2 to smithi12313934-32 2020-01-16T00:13:14.027 INFO:tasks.rados.rados.0.smithi123.stdout:1102: finishing write tid 1 to smithi12313934-9 2020-01-16T00:13:14.027 INFO:tasks.rados.rados.0.smithi123.stdout:1101: finishing write tid 4 to smithi12313934-32 2020-01-16T00:13:14.027 INFO:tasks.rados.rados.0.smithi123.stdout:1101: finishing write tid 5 to smithi12313934-32 2020-01-16T00:13:14.027 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 32 v 103 (ObjNum 300 snap 0 seq_num 300) dirty exists 2020-01-16T00:13:14.027 INFO:tasks.rados.rados.0.smithi123.stdout:1101: left oid 32 (ObjNum 300 snap 0 seq_num 300) 2020-01-16T00:13:14.028 INFO:tasks.rados.rados.0.smithi123.stdout:1102: finishing write tid 2 to smithi12313934-9 2020-01-16T00:13:14.028 INFO:tasks.rados.rados.0.smithi123.stdout:1100: done (15 left) 2020-01-16T00:13:14.028 INFO:tasks.rados.rados.0.smithi123.stdout:1101: done (14 left) 2020-01-16T00:13:14.028 INFO:tasks.rados.rados.0.smithi123.stdout:1116: write oid 41 current snap is 0 2020-01-16T00:13:14.028 INFO:tasks.rados.rados.0.smithi123.stdout:1116: seq_num 310 ranges {410170=781310,1940712=656106,3163363=46807} 2020-01-16T00:13:14.032 INFO:tasks.rados.rados.0.smithi123.stdout:1116: writing smithi12313934-41 from 410170 to 1191480 tid 1 2020-01-16T00:13:14.037 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-01-16T00:13:14.040 INFO:tasks.rados.rados.0.smithi123.stdout:1116: writing smithi12313934-41 from 1940712 to 2596818 tid 2 2020-01-16T00:13:14.044 INFO:tasks.rados.rados.0.smithi123.stdout:1116: writing smithi12313934-41 from 3163363 to 3210170 tid 3 2020-01-16T00:13:14.044 INFO:tasks.rados.rados.0.smithi123.stdout:1117: write (excl) oid 6 current snap is 0 2020-01-16T00:13:14.045 INFO:tasks.rados.rados.0.smithi123.stdout:1117: seq_num 311 ranges {426977=456559,1482841=799764,2958080=268897} 2020-01-16T00:13:14.049 INFO:tasks.rados.rados.0.smithi123.stdout:1117: writing smithi12313934-6 from 426977 to 883536 tid 1 2020-01-16T00:13:14.057 INFO:tasks.rados.rados.0.smithi123.stdout:1117: writing smithi12313934-6 from 1482841 to 2282605 tid 2 2020-01-16T00:13:14.063 INFO:tasks.rados.rados.0.smithi123.stdout:1117: writing smithi12313934-6 from 2958080 to 3226977 tid 3 2020-01-16T00:13:14.063 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:14.063 INFO:tasks.rados.rados.0.smithi123.stdout:1102: finishing write tid 3 to smithi12313934-9 2020-01-16T00:13:14.063 INFO:tasks.rados.rados.0.smithi123.stdout:1102: finishing write tid 5 to smithi12313934-9 2020-01-16T00:13:14.064 INFO:tasks.rados.rados.0.smithi123.stdout:1102: finishing write tid 6 to smithi12313934-9 2020-01-16T00:13:14.064 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 9 v 106 (ObjNum 301 snap 0 seq_num 301) dirty exists 2020-01-16T00:13:14.064 INFO:tasks.rados.rados.0.smithi123.stdout:1102: left oid 9 (ObjNum 301 snap 0 seq_num 301) 2020-01-16T00:13:14.064 INFO:tasks.rados.rados.0.smithi123.stdout:1103: finishing write tid 1 to smithi12313934-48 2020-01-16T00:13:14.064 INFO:tasks.rados.rados.0.smithi123.stdout:1103: finishing write tid 2 to smithi12313934-48 2020-01-16T00:13:14.064 INFO:tasks.rados.rados.0.smithi123.stdout:1102: done (15 left) 2020-01-16T00:13:14.064 INFO:tasks.rados.rados.0.smithi123.stdout:1118: write oid 21 current snap is 0 2020-01-16T00:13:14.065 INFO:tasks.rados.rados.0.smithi123.stdout:1118: seq_num 312 ranges {443784=648161,1657676=659775,2818209=425575} 2020-01-16T00:13:14.070 INFO:tasks.rados.rados.0.smithi123.stdout:1118: writing smithi12313934-21 from 443784 to 1091945 tid 1 2020-01-16T00:13:14.080 INFO:tasks.rados.rados.0.smithi123.stdout:1118: writing smithi12313934-21 from 1657676 to 2317451 tid 2 2020-01-16T00:13:14.082 INFO:tasks.rados.rados.0.smithi123.stdout:1118: writing smithi12313934-21 from 2818209 to 3243784 tid 3 2020-01-16T00:13:14.082 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:14.082 INFO:tasks.rados.rados.0.smithi123.stdout:1104: expect (ObjNum 242 snap 0 seq_num 242) 2020-01-16T00:13:14.131 INFO:tasks.ceph.osd.5.smithi145.stderr:2020-01-16T00:13:14.128+0000 7f61e7ef7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 11842) UID: 0 2020-01-16T00:13:14.168 INFO:tasks.rados.rados.0.smithi123.stdout:1103: finishing write tid 3 to smithi12313934-48 2020-01-16T00:13:14.168 INFO:tasks.rados.rados.0.smithi123.stdout:1103: finishing write tid 5 to smithi12313934-48 2020-01-16T00:13:14.169 INFO:tasks.rados.rados.0.smithi123.stdout:1103: finishing write tid 6 to smithi12313934-48 2020-01-16T00:13:14.169 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 48 v 117 (ObjNum 302 snap 0 seq_num 302) dirty exists 2020-01-16T00:13:14.169 INFO:tasks.rados.rados.0.smithi123.stdout:1103: left oid 48 (ObjNum 302 snap 0 seq_num 302) 2020-01-16T00:13:14.169 INFO:tasks.rados.rados.0.smithi123.stdout:1105: expect (ObjNum 146 snap 0 seq_num 146) 2020-01-16T00:13:14.181 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:14.190 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-01-16T00:13:14.231 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:14.228+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:14.234 INFO:tasks.rados.rados.0.smithi123.stdout:1106: finishing write tid 1 to smithi12313934-27 2020-01-16T00:13:14.234 INFO:tasks.rados.rados.0.smithi123.stdout:1106: finishing write tid 2 to smithi12313934-27 2020-01-16T00:13:14.234 INFO:tasks.rados.rados.0.smithi123.stdout:1106: finishing write tid 4 to smithi12313934-27 2020-01-16T00:13:14.234 INFO:tasks.rados.rados.0.smithi123.stdout:1106: finishing write tid 5 to smithi12313934-27 2020-01-16T00:13:14.234 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 27 v 109 (ObjNum 303 snap 0 seq_num 303) dirty exists 2020-01-16T00:13:14.235 INFO:tasks.rados.rados.0.smithi123.stdout:1106: left oid 27 (ObjNum 303 snap 0 seq_num 303) 2020-01-16T00:13:14.235 INFO:tasks.rados.rados.0.smithi123.stdout:1107: finishing write tid 1 to smithi12313934-24 2020-01-16T00:13:14.235 INFO:tasks.rados.rados.0.smithi123.stdout:1107: finishing write tid 2 to smithi12313934-24 2020-01-16T00:13:14.235 INFO:tasks.rados.rados.0.smithi123.stdout:1108: expect (ObjNum 219 snap 0 seq_num 219) 2020-01-16T00:13:14.303 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:14.298+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:14.333 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:14.343 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-01-16T00:13:14.404 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:14.402+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:14.451 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:14.460 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-01-16T00:13:14.471 INFO:tasks.rados.rados.0.smithi123.stdout:1107: finishing write tid 4 to smithi12313934-24 2020-01-16T00:13:14.471 INFO:tasks.rados.rados.0.smithi123.stdout:1107: finishing write tid 5 to smithi12313934-24 2020-01-16T00:13:14.472 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 24 v 111 (ObjNum 304 snap 0 seq_num 304) dirty exists 2020-01-16T00:13:14.472 INFO:tasks.rados.rados.0.smithi123.stdout:1107: left oid 24 (ObjNum 304 snap 0 seq_num 304) 2020-01-16T00:13:14.472 INFO:tasks.rados.rados.0.smithi123.stdout:1109: finishing write tid 1 to smithi12313934-35 2020-01-16T00:13:14.472 INFO:tasks.rados.rados.0.smithi123.stdout:1109: finishing write tid 2 to smithi12313934-35 2020-01-16T00:13:14.472 INFO:tasks.rados.rados.0.smithi123.stdout:1109: finishing write tid 3 to smithi12313934-35 2020-01-16T00:13:14.472 INFO:tasks.rados.rados.0.smithi123.stdout:1109: finishing write tid 5 to smithi12313934-35 2020-01-16T00:13:14.472 INFO:tasks.rados.rados.0.smithi123.stdout:1109: finishing write tid 6 to smithi12313934-35 2020-01-16T00:13:14.472 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 35 v 118 (ObjNum 305 snap 0 seq_num 305) dirty exists 2020-01-16T00:13:14.473 INFO:tasks.rados.rados.0.smithi123.stdout:1109: left oid 35 (ObjNum 305 snap 0 seq_num 305) 2020-01-16T00:13:14.473 INFO:tasks.rados.rados.0.smithi123.stdout:1110: finishing write tid 1 to smithi12313934-14 2020-01-16T00:13:14.473 INFO:tasks.rados.rados.0.smithi123.stdout:1110: finishing write tid 2 to smithi12313934-14 2020-01-16T00:13:14.473 INFO:tasks.rados.rados.0.smithi123.stdout:1110: finishing write tid 4 to smithi12313934-14 2020-01-16T00:13:14.473 INFO:tasks.rados.rados.0.smithi123.stdout:1110: finishing write tid 5 to smithi12313934-14 2020-01-16T00:13:14.473 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 14 v 155 (ObjNum 306 snap 0 seq_num 306) dirty exists 2020-01-16T00:13:14.473 INFO:tasks.rados.rados.0.smithi123.stdout:1110: left oid 14 (ObjNum 306 snap 0 seq_num 306) 2020-01-16T00:13:14.474 INFO:tasks.rados.rados.0.smithi123.stdout:1111: finishing write tid 1 to smithi12313934-49 2020-01-16T00:13:14.474 INFO:tasks.rados.rados.0.smithi123.stdout:1111: finishing write tid 2 to smithi12313934-49 2020-01-16T00:13:14.474 INFO:tasks.rados.rados.0.smithi123.stdout:1111: finishing write tid 3 to smithi12313934-49 2020-01-16T00:13:14.474 INFO:tasks.rados.rados.0.smithi123.stdout:1103: done (15 left) 2020-01-16T00:13:14.474 INFO:tasks.rados.rados.0.smithi123.stdout:1104: done (14 left) 2020-01-16T00:13:14.474 INFO:tasks.rados.rados.0.smithi123.stdout:1105: done (13 left) 2020-01-16T00:13:14.474 INFO:tasks.rados.rados.0.smithi123.stdout:1106: done (12 left) 2020-01-16T00:13:14.475 INFO:tasks.rados.rados.0.smithi123.stdout:1107: done (11 left) 2020-01-16T00:13:14.475 INFO:tasks.rados.rados.0.smithi123.stdout:1108: done (10 left) 2020-01-16T00:13:14.475 INFO:tasks.rados.rados.0.smithi123.stdout:1109: done (9 left) 2020-01-16T00:13:14.475 INFO:tasks.rados.rados.0.smithi123.stdout:1110: done (8 left) 2020-01-16T00:13:14.475 INFO:tasks.rados.rados.0.smithi123.stdout:1119: read oid 46 snap -1 2020-01-16T00:13:14.475 INFO:tasks.rados.rados.0.smithi123.stdout:1119: expect (ObjNum 297 snap 0 seq_num 297) 2020-01-16T00:13:14.475 INFO:tasks.rados.rados.0.smithi123.stdout:1120: read oid 19 snap -1 2020-01-16T00:13:14.476 INFO:tasks.rados.rados.0.smithi123.stdout:1120: expect (ObjNum 256 snap 0 seq_num 256) 2020-01-16T00:13:14.476 INFO:tasks.rados.rados.0.smithi123.stdout:1121: delete oid 23 current snap is 0 2020-01-16T00:13:14.477 INFO:tasks.rados.rados.0.smithi123.stdout:1111: finishing write tid 5 to smithi12313934-49 2020-01-16T00:13:14.477 INFO:tasks.rados.rados.0.smithi123.stdout:1111: finishing write tid 6 to smithi12313934-49 2020-01-16T00:13:14.477 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 49 v 113 (ObjNum 307 snap 0 seq_num 307) dirty exists 2020-01-16T00:13:14.477 INFO:tasks.rados.rados.0.smithi123.stdout:1111: left oid 49 (ObjNum 307 snap 0 seq_num 307) 2020-01-16T00:13:14.477 INFO:tasks.rados.rados.0.smithi123.stdout:1112: expect (ObjNum 250 snap 0 seq_num 250) 2020-01-16T00:13:14.506 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:14.502+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:14.563 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:14.573 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-01-16T00:13:14.592 INFO:tasks.rados.rados.0.smithi123.stdout:1113: finishing write tid 1 to smithi12313934-16 2020-01-16T00:13:14.592 INFO:tasks.rados.rados.0.smithi123.stdout:1113: finishing write tid 2 to smithi12313934-16 2020-01-16T00:13:14.592 INFO:tasks.rados.rados.0.smithi123.stdout:1111: done (10 left) 2020-01-16T00:13:14.592 INFO:tasks.rados.rados.0.smithi123.stdout:1112: done (9 left) 2020-01-16T00:13:14.592 INFO:tasks.rados.rados.0.smithi123.stdout:1121: done (8 left) 2020-01-16T00:13:14.592 INFO:tasks.rados.rados.0.smithi123.stdout:1122: read oid 13 snap -1 2020-01-16T00:13:14.593 INFO:tasks.rados.rados.0.smithi123.stdout:1122: expect (ObjNum 272 snap 0 seq_num 272) 2020-01-16T00:13:14.593 INFO:tasks.rados.rados.0.smithi123.stdout:1123: write (excl) oid 17 current snap is 0 2020-01-16T00:13:14.593 INFO:tasks.rados.rados.0.smithi123.stdout:1123: seq_num 314 ranges {477398=798659,1774640=663444,2873055=404343} 2020-01-16T00:13:14.600 INFO:tasks.rados.rados.0.smithi123.stdout:1123: writing smithi12313934-17 from 477398 to 1276057 tid 1 2020-01-16T00:13:14.606 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:14.600+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:14.607 INFO:tasks.rados.rados.0.smithi123.stdout:1123: writing smithi12313934-17 from 1774640 to 2438084 tid 2 2020-01-16T00:13:14.612 INFO:tasks.rados.rados.0.smithi123.stdout:1123: writing smithi12313934-17 from 2873055 to 3277398 tid 3 2020-01-16T00:13:14.612 INFO:tasks.rados.rados.0.smithi123.stdout:1124: write (excl) oid 7 current snap is 0 2020-01-16T00:13:14.613 INFO:tasks.rados.rados.0.smithi123.stdout:1124: seq_num 315 ranges {494205=473908,1433122=407102,2384125=462684} 2020-01-16T00:13:14.617 INFO:tasks.rados.rados.0.smithi123.stdout:1124: writing smithi12313934-7 from 494205 to 968113 tid 1 2020-01-16T00:13:14.622 INFO:tasks.rados.rados.0.smithi123.stdout:1124: writing smithi12313934-7 from 1433122 to 1840224 tid 2 2020-01-16T00:13:14.628 INFO:tasks.rados.rados.0.smithi123.stdout:1124: writing smithi12313934-7 from 2384125 to 2846809 tid 3 2020-01-16T00:13:14.628 INFO:tasks.rados.rados.0.smithi123.stdout:1125: write oid 44 current snap is 0 2020-01-16T00:13:14.628 INFO:tasks.rados.rados.0.smithi123.stdout:1125: seq_num 316 ranges {511012=549157,1775251=550760,3095195=215817} 2020-01-16T00:13:14.634 INFO:tasks.rados.rados.0.smithi123.stdout:1125: writing smithi12313934-44 from 511012 to 1060169 tid 1 2020-01-16T00:13:14.641 INFO:tasks.rados.rados.0.smithi123.stdout:1125: writing smithi12313934-44 from 1775251 to 2326011 tid 2 2020-01-16T00:13:14.647 INFO:tasks.rados.rados.0.smithi123.stdout:1125: writing smithi12313934-44 from 3095195 to 3311012 tid 3 2020-01-16T00:13:14.647 INFO:tasks.rados.rados.0.smithi123.stdout:1126: read oid 24 snap -1 2020-01-16T00:13:14.647 INFO:tasks.rados.rados.0.smithi123.stdout:1126: expect (ObjNum 304 snap 0 seq_num 304) 2020-01-16T00:13:14.647 INFO:tasks.rados.rados.0.smithi123.stdout:1127: read oid 27 snap -1 2020-01-16T00:13:14.648 INFO:tasks.rados.rados.0.smithi123.stdout:1127: expect (ObjNum 303 snap 0 seq_num 303) 2020-01-16T00:13:14.648 INFO:tasks.rados.rados.0.smithi123.stdout:1128: write oid 15 current snap is 0 2020-01-16T00:13:14.648 INFO:tasks.rados.rados.0.smithi123.stdout:1128: seq_num 317 ranges {527819=624406,1833733=410771,2722618=601581} 2020-01-16T00:13:14.653 INFO:tasks.rados.rados.0.smithi123.stdout:1128: writing smithi12313934-15 from 527819 to 1152225 tid 1 2020-01-16T00:13:14.661 INFO:tasks.rados.rados.0.smithi123.stdout:1128: writing smithi12313934-15 from 1833733 to 2244504 tid 2 2020-01-16T00:13:14.669 INFO:tasks.rados.rados.0.smithi123.stdout:1128: writing smithi12313934-15 from 2722618 to 3324199 tid 3 2020-01-16T00:13:14.669 INFO:tasks.rados.rados.0.smithi123.stdout:1129: write oid 14 current snap is 0 2020-01-16T00:13:14.669 INFO:tasks.rados.rados.0.smithi123.stdout:1129: seq_num 318 ranges {544626=699655,1892215=554429,3150041=194585} 2020-01-16T00:13:14.675 INFO:tasks.rados.rados.0.smithi123.stdout:1129: writing smithi12313934-14 from 544626 to 1244281 tid 1 2020-01-16T00:13:14.682 INFO:tasks.rados.rados.0.smithi123.stdout:1129: writing smithi12313934-14 from 1892215 to 2446644 tid 2 2020-01-16T00:13:14.688 INFO:tasks.rados.rados.0.smithi123.stdout:1129: writing smithi12313934-14 from 3150041 to 3344626 tid 3 2020-01-16T00:13:14.688 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:14.688 INFO:tasks.rados.rados.0.smithi123.stdout:1113: finishing write tid 3 to smithi12313934-16 2020-01-16T00:13:14.688 INFO:tasks.rados.rados.0.smithi123.stdout:1113: finishing write tid 5 to smithi12313934-16 2020-01-16T00:13:14.688 INFO:tasks.rados.rados.0.smithi123.stdout:1113: finishing write tid 6 to smithi12313934-16 2020-01-16T00:13:14.688 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 16 v 163 (ObjNum 308 snap 0 seq_num 308) dirty exists 2020-01-16T00:13:14.688 INFO:tasks.rados.rados.0.smithi123.stdout:1113: left oid 16 (ObjNum 308 snap 0 seq_num 308) 2020-01-16T00:13:14.689 INFO:tasks.rados.rados.0.smithi123.stdout:1114: expect (ObjNum 263 snap 0 seq_num 263) 2020-01-16T00:13:14.694 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:14.703 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-01-16T00:13:14.707 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:14.704+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:14.717 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [1, 0, 3, 2, 5, 4, 7, 6] 2020-01-16T00:13:14.718 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2020-01-16T00:13:14.718 INFO:tasks.thrashosds.thrasher:Killing osd 5, live_osds are [1, 0, 3, 2, 5, 4, 7, 6] 2020-01-16T00:13:14.718 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2020-01-16T00:13:14.718 INFO:teuthology.orchestra.run:waiting for 300 2020-01-16T00:13:14.781 INFO:tasks.ceph.osd.5:Stopped 2020-01-16T00:13:14.781 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd 2020-01-16T00:13:14.782 INFO:teuthology.orchestra.run.smithi145:> type ceph-objectstore-tool 2020-01-16T00:13:14.787 INFO:teuthology.orchestra.run.smithi145.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2020-01-16T00:13:14.788 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op list-pgs 2020-01-16T00:13:14.808 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:14.802+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:14.861 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:14.870 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-01-16T00:13:14.891 INFO:tasks.rados.rados.0.smithi123.stdout:1115: finishing write tid 1 to smithi12313934-45 2020-01-16T00:13:14.892 INFO:tasks.rados.rados.0.smithi123.stdout:1115: finishing write tid 2 to smithi12313934-45 2020-01-16T00:13:14.892 INFO:tasks.rados.rados.0.smithi123.stdout:1116: finishing write tid 1 to smithi12313934-41 2020-01-16T00:13:14.892 INFO:tasks.rados.rados.0.smithi123.stdout:1115: finishing write tid 4 to smithi12313934-45 2020-01-16T00:13:14.892 INFO:tasks.rados.rados.0.smithi123.stdout:1115: finishing write tid 5 to smithi12313934-45 2020-01-16T00:13:14.892 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 45 v 117 (ObjNum 309 snap 0 seq_num 309) dirty exists 2020-01-16T00:13:14.892 INFO:tasks.rados.rados.0.smithi123.stdout:1115: left oid 45 (ObjNum 309 snap 0 seq_num 309) 2020-01-16T00:13:14.892 INFO:tasks.rados.rados.0.smithi123.stdout:1116: finishing write tid 2 to smithi12313934-41 2020-01-16T00:13:14.893 INFO:tasks.rados.rados.0.smithi123.stdout:1116: finishing write tid 3 to smithi12313934-41 2020-01-16T00:13:14.893 INFO:tasks.rados.rados.0.smithi123.stdout:1116: finishing write tid 5 to smithi12313934-41 2020-01-16T00:13:14.893 INFO:tasks.rados.rados.0.smithi123.stdout:1113: done (15 left) 2020-01-16T00:13:14.893 INFO:tasks.rados.rados.0.smithi123.stdout:1114: done (14 left) 2020-01-16T00:13:14.893 INFO:tasks.rados.rados.0.smithi123.stdout:1115: done (13 left) 2020-01-16T00:13:14.893 INFO:tasks.rados.rados.0.smithi123.stdout:1116: finishing write tid 6 to smithi12313934-41 2020-01-16T00:13:14.893 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 41 v 166 (ObjNum 310 snap 0 seq_num 310) dirty exists 2020-01-16T00:13:14.893 INFO:tasks.rados.rados.0.smithi123.stdout:1116: left oid 41 (ObjNum 310 snap 0 seq_num 310) 2020-01-16T00:13:14.894 INFO:tasks.rados.rados.0.smithi123.stdout:1117: finishing write tid 1 to smithi12313934-6 2020-01-16T00:13:14.894 INFO:tasks.rados.rados.0.smithi123.stdout:1117: finishing write tid 2 to smithi12313934-6 2020-01-16T00:13:14.894 INFO:tasks.rados.rados.0.smithi123.stdout:1117: finishing write tid 3 to smithi12313934-6 2020-01-16T00:13:14.894 INFO:tasks.rados.rados.0.smithi123.stdout:1117: finishing write tid 5 to smithi12313934-6 2020-01-16T00:13:14.894 INFO:tasks.rados.rados.0.smithi123.stdout:1130: read oid 47 snap -1 2020-01-16T00:13:14.894 INFO:tasks.rados.rados.0.smithi123.stdout:1130: expect (ObjNum 250 snap 0 seq_num 250) 2020-01-16T00:13:14.894 INFO:tasks.rados.rados.0.smithi123.stdout:1116: done (13 left) 2020-01-16T00:13:14.894 INFO:tasks.rados.rados.0.smithi123.stdout:1131: read oid 16 snap -1 2020-01-16T00:13:14.895 INFO:tasks.rados.rados.0.smithi123.stdout:1131: expect (ObjNum 308 snap 0 seq_num 308) 2020-01-16T00:13:14.895 INFO:tasks.rados.rados.0.smithi123.stdout:1132: write oid 50 current snap is 0 2020-01-16T00:13:14.895 INFO:tasks.rados.rados.0.smithi123.stdout:1132: seq_num 319 ranges {561433=774904,1950697=414440,2777464=583969} 2020-01-16T00:13:14.903 INFO:tasks.rados.rados.0.smithi123.stdout:1132: writing smithi12313934-50 from 561433 to 1336337 tid 1 2020-01-16T00:13:14.908 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:14.906+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:14.909 INFO:tasks.rados.rados.0.smithi123.stdout:1132: writing smithi12313934-50 from 1950697 to 2365137 tid 2 2020-01-16T00:13:14.914 INFO:tasks.rados.rados.0.smithi123.stdout:1132: writing smithi12313934-50 from 2777464 to 3361433 tid 3 2020-01-16T00:13:14.914 INFO:tasks.rados.rados.0.smithi123.stdout:1133: read oid 30 snap -1 2020-01-16T00:13:14.915 INFO:tasks.rados.rados.0.smithi123.stdout:1133: expect (ObjNum 294 snap 0 seq_num 294) 2020-01-16T00:13:14.915 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:14.915 INFO:tasks.rados.rados.0.smithi123.stdout:1117: finishing write tid 6 to smithi12313934-6 2020-01-16T00:13:14.915 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 6 v 89 (ObjNum 311 snap 0 seq_num 311) dirty exists 2020-01-16T00:13:14.915 INFO:tasks.rados.rados.0.smithi123.stdout:1117: left oid 6 (ObjNum 311 snap 0 seq_num 311) 2020-01-16T00:13:14.915 INFO:tasks.rados.rados.0.smithi123.stdout:1117: done (15 left) 2020-01-16T00:13:14.915 INFO:tasks.rados.rados.0.smithi123.stdout:1134: write oid 12 current snap is 0 2020-01-16T00:13:14.915 INFO:tasks.rados.rados.0.smithi123.stdout:1134: seq_num 320 ranges {578240=566506,1609179=558098,2804887=551750} 2020-01-16T00:13:14.920 INFO:tasks.rados.rados.0.smithi123.stdout:1134: writing smithi12313934-12 from 578240 to 1144746 tid 1 2020-01-16T00:13:14.926 INFO:tasks.rados.rados.0.smithi123.stdout:1134: writing smithi12313934-12 from 1609179 to 2167277 tid 2 2020-01-16T00:13:14.934 INFO:tasks.rados.rados.0.smithi123.stdout:1134: writing smithi12313934-12 from 2804887 to 3356637 tid 3 2020-01-16T00:13:14.934 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:14.934 INFO:tasks.rados.rados.0.smithi123.stdout:1118: finishing write tid 1 to smithi12313934-21 2020-01-16T00:13:14.934 INFO:tasks.rados.rados.0.smithi123.stdout:1118: finishing write tid 2 to smithi12313934-21 2020-01-16T00:13:14.934 INFO:tasks.rados.rados.0.smithi123.stdout:1118: finishing write tid 3 to smithi12313934-21 2020-01-16T00:13:14.934 INFO:tasks.rados.rados.0.smithi123.stdout:1118: finishing write tid 5 to smithi12313934-21 2020-01-16T00:13:14.935 INFO:tasks.rados.rados.0.smithi123.stdout:1118: finishing write tid 6 to smithi12313934-21 2020-01-16T00:13:14.935 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 21 v 41 (ObjNum 312 snap 0 seq_num 312) dirty exists 2020-01-16T00:13:14.935 INFO:tasks.rados.rados.0.smithi123.stdout:1118: left oid 21 (ObjNum 312 snap 0 seq_num 312) 2020-01-16T00:13:14.935 INFO:tasks.rados.rados.0.smithi123.stdout:1118: done (15 left) 2020-01-16T00:13:14.935 INFO:tasks.rados.rados.0.smithi123.stdout:1135: write oid 29 current snap is 0 2020-01-16T00:13:14.935 INFO:tasks.rados.rados.0.smithi123.stdout:1135: seq_num 321 ranges {595047=641755,1667661=418109,2548663=763022} 2020-01-16T00:13:14.940 INFO:tasks.rados.rados.0.smithi123.stdout:1135: writing smithi12313934-29 from 595047 to 1236802 tid 1 2020-01-16T00:13:14.945 INFO:tasks.rados.rados.0.smithi123.stdout:1135: writing smithi12313934-29 from 1667661 to 2085770 tid 2 2020-01-16T00:13:14.952 INFO:tasks.rados.rados.0.smithi123.stdout:1135: writing smithi12313934-29 from 2548663 to 3311685 tid 3 2020-01-16T00:13:14.952 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:14.952 INFO:tasks.rados.rados.0.smithi123.stdout:1119: expect (ObjNum 297 snap 0 seq_num 297) 2020-01-16T00:13:14.989 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2020-01-16T00:13:14.990 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-01-16T00:13:14.999 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:15.010 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:15.006+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:15.011 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-01-16T00:13:15.111 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:15.108+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:15.124 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:13:15.151 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:13:15.164 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:15.165 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:15.165 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:13:15.165 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:15.170 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:13:15.174 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-01-16T00:13:15.181 INFO:tasks.rados.rados.0.smithi123.stdout:1120: expect (ObjNum 256 snap 0 seq_num 256) 2020-01-16T00:13:15.189 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:13:15.210 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:13:15.212 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:15.210+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:15.237 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:13:15.261 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:13:15.261 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-01-16T00:13:15.261 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-01-16T00:13:15.262 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-01-16T00:13:15.262 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-01-16T00:13:15.262 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-01-16T00:13:15.262 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENXIO: problem getting command descriptions from osd.5 2020-01-16T00:13:15.262 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: problem getting command descriptions from osd.5 2020-01-16T00:13:15.262 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-01-16T00:13:15.262 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-01-16T00:13:15.269 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-01-16T00:13:15.269 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-01-16T00:13:15.273 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:15.273 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:15.274 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:13:15.274 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:13:15.274 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:15.282 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-01-16T00:13:15.324 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:15.322+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:15.339 INFO:tasks.rados.rados.0.smithi123.stdout:1122: expect (ObjNum 272 snap 0 seq_num 272) 2020-01-16T00:13:15.416 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:15.408+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:15.430 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:15.431 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:13:15.431 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:13:15.431 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:13:15.431 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.431 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.5.13:36 2.0 e24/19 2:20e40cec:::smithi12313934-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 24'16, mlcod=23'14)", 2020-01-16T00:13:15.431 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.834879+0000", 2020-01-16T00:13:15.431 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 37.590833183999997, 2020-01-16T00:13:15.432 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0075184850000000001, 2020-01-16T00:13:15.432 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:15.432 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:15.432 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:15.432 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.432 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:15.432 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.834879+0000", 2020-01-16T00:13:15.432 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:15.433 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.433 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.433 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:15.433 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.834879+0000", 2020-01-16T00:13:15.433 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:15.433 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.434 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.434 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:15.434 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.834877+0000", 2020-01-16T00:13:15.434 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031393089999999998 2020-01-16T00:13:15.434 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.435 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.435 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:15.436 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.838016+0000", 2020-01-16T00:13:15.436 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2144e-05 2020-01-16T00:13:15.436 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.436 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.437 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:15.437 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.838038+0000", 2020-01-16T00:13:15.437 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2058e-05 2020-01-16T00:13:15.437 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.437 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.437 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:15.438 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.838050+0000", 2020-01-16T00:13:15.438 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8533000000000003e-05 2020-01-16T00:13:15.438 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.438 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.438 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:15.438 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.838119+0000", 2020-01-16T00:13:15.439 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3591000000000003e-05 2020-01-16T00:13:15.439 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.439 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.439 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:15.439 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.838162+0000", 2020-01-16T00:13:15.439 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041910380000000002 2020-01-16T00:13:15.439 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.440 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.440 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:15.440 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.842353+0000", 2020-01-16T00:13:15.441 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4433e-05 2020-01-16T00:13:15.441 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.441 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.441 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:15.442 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.842398+0000", 2020-01-16T00:13:15.442 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0075184850000000001 2020-01-16T00:13:15.442 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.442 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:15.442 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.443 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.443 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.443 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.5.13:40 2.0 e24/19 2:20e40cec:::smithi12313934-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 24'17, mlcod=23'15)", 2020-01-16T00:13:15.443 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.849607+0000", 2020-01-16T00:13:15.443 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 37.576105059, 2020-01-16T00:13:15.443 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083979039999999994, 2020-01-16T00:13:15.443 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:15.444 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:15.444 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:15.444 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.444 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:15.444 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.849607+0000", 2020-01-16T00:13:15.444 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:15.444 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.444 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.445 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:15.445 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.849607+0000", 2020-01-16T00:13:15.445 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:15.445 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.445 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.445 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:15.445 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.849605+0000", 2020-01-16T00:13:15.446 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0037534560000000001 2020-01-16T00:13:15.446 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.446 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.446 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:15.446 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.853358+0000", 2020-01-16T00:13:15.446 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9181000000000002e-05 2020-01-16T00:13:15.446 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.447 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.447 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:15.447 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.853377+0000", 2020-01-16T00:13:15.447 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0040000000000002e-06 2020-01-16T00:13:15.447 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.447 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.447 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:15.447 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.853384+0000", 2020-01-16T00:13:15.448 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2583000000000001e-05 2020-01-16T00:13:15.448 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.448 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.448 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:15.448 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.853437+0000", 2020-01-16T00:13:15.448 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8212999999999999e-05 2020-01-16T00:13:15.448 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.448 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.449 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:15.449 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.853465+0000", 2020-01-16T00:13:15.449 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0044847589999999996 2020-01-16T00:13:15.449 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.449 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.449 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:15.449 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.857950+0000", 2020-01-16T00:13:15.450 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5309999999999997e-05 2020-01-16T00:13:15.450 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.450 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.450 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:15.451 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.858005+0000", 2020-01-16T00:13:15.451 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083979039999999994 2020-01-16T00:13:15.451 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.451 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:15.451 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.451 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.451 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.451 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.5.13:42 2.0 e24/19 2:313458df:::smithi12313934-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 24'18, mlcod=24'16)", 2020-01-16T00:13:15.452 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.940683+0000", 2020-01-16T00:13:15.452 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 37.485029220000001, 2020-01-16T00:13:15.452 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0097026960000000002, 2020-01-16T00:13:15.452 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:15.452 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:15.453 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:15.453 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.453 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:15.453 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.940683+0000", 2020-01-16T00:13:15.454 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:15.454 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.454 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.454 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:15.455 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.940683+0000", 2020-01-16T00:13:15.455 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:15.455 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.455 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.456 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:15.456 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.940680+0000", 2020-01-16T00:13:15.456 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003126775 2020-01-16T00:13:15.456 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.456 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.457 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:15.457 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.943807+0000", 2020-01-16T00:13:15.457 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5426999999999999e-05 2020-01-16T00:13:15.457 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.458 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.458 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:15.458 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.943833+0000", 2020-01-16T00:13:15.458 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7409999999999998e-06 2020-01-16T00:13:15.458 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.459 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.459 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:15.459 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.943841+0000", 2020-01-16T00:13:15.459 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011645399999999999 2020-01-16T00:13:15.460 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.460 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.460 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:15.460 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.943958+0000", 2020-01-16T00:13:15.461 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7291000000000003e-05 2020-01-16T00:13:15.461 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.462 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.462 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:15.462 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.944025+0000", 2020-01-16T00:13:15.462 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063108449999999998 2020-01-16T00:13:15.463 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.463 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.463 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:15.463 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.950336+0000", 2020-01-16T00:13:15.464 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9874999999999999e-05 2020-01-16T00:13:15.464 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.464 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.464 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:15.465 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.950386+0000", 2020-01-16T00:13:15.465 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0097026960000000002 2020-01-16T00:13:15.465 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.465 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:15.465 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.466 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.466 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.466 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1387 3.1 3:8c6a83c0:::smithi12313934-36:head [read 0~3579858,checksum] snapc 0=[] ondisk+read+known_if_redirected e28)", 2020-01-16T00:13:15.466 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:46.605159+0000", 2020-01-16T00:13:15.467 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 28.820552924000001, 2020-01-16T00:13:15.467 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0074537709999999997, 2020-01-16T00:13:15.467 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:15.467 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:15.468 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:15.468 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:15.468 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:15.468 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1387 2020-01-16T00:13:15.469 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.469 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:15.469 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.469 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:15.470 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605159+0000", 2020-01-16T00:13:15.470 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:15.470 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.470 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.471 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:15.471 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605159+0000", 2020-01-16T00:13:15.472 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:15.472 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.472 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.473 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:15.473 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605158+0000", 2020-01-16T00:13:15.474 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3860000000000001e-06 2020-01-16T00:13:15.474 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.474 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.474 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:15.475 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605166+0000", 2020-01-16T00:13:15.475 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2139e-05 2020-01-16T00:13:15.475 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.475 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.475 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:15.476 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605178+0000", 2020-01-16T00:13:15.476 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8270000000000003e-06 2020-01-16T00:13:15.476 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.476 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.477 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:15.477 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605184+0000", 2020-01-16T00:13:15.477 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0045873850000000002 2020-01-16T00:13:15.477 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.478 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.478 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:15.478 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.609771+0000", 2020-01-16T00:13:15.478 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000202387 2020-01-16T00:13:15.479 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.479 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.479 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:15.479 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.609973+0000", 2020-01-16T00:13:15.480 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0026399079999999998 2020-01-16T00:13:15.480 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.480 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.480 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:15.480 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.612613+0000", 2020-01-16T00:13:15.481 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0074537709999999997 2020-01-16T00:13:15.481 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.481 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:15.481 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.482 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.483 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.483 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1388 3.1 3:8c6a83c0:::smithi12313934-36:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e28)", 2020-01-16T00:13:15.483 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:46.605253+0000", 2020-01-16T00:13:15.483 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 28.820459870000001, 2020-01-16T00:13:15.483 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0080920829999999999, 2020-01-16T00:13:15.483 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:15.484 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:15.484 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:15.484 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:15.484 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:15.484 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1388 2020-01-16T00:13:15.485 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.485 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:15.485 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.486 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:15.486 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605253+0000", 2020-01-16T00:13:15.486 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:15.486 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.486 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.487 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:15.487 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605253+0000", 2020-01-16T00:13:15.487 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:15.487 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.487 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.488 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:15.488 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605252+0000", 2020-01-16T00:13:15.488 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6559999999999997e-06 2020-01-16T00:13:15.488 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.488 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.488 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:15.488 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605257+0000", 2020-01-16T00:13:15.488 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0966e-05 2020-01-16T00:13:15.489 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.489 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.489 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:15.489 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605268+0000", 2020-01-16T00:13:15.489 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5960000000000003e-06 2020-01-16T00:13:15.489 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.489 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.490 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:15.490 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605274+0000", 2020-01-16T00:13:15.490 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.007370176 2020-01-16T00:13:15.490 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.490 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.491 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:15.491 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.612644+0000", 2020-01-16T00:13:15.492 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000113146 2020-01-16T00:13:15.492 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.492 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.492 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:15.492 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.612757+0000", 2020-01-16T00:13:15.492 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00058744799999999996 2020-01-16T00:13:15.492 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.494 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.495 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:15.495 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.613345+0000", 2020-01-16T00:13:15.495 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0080920829999999999 2020-01-16T00:13:15.495 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.495 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:15.495 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.495 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.496 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.496 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1644 3.1 3:8c6a83c0:::smithi12313934-36:head [read 0~2705927,checksum] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:13:15.496 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847556+0000", 2020-01-16T00:13:15.496 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 23.578156452999998, 2020-01-16T00:13:15.496 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0081843639999999995, 2020-01-16T00:13:15.496 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:15.497 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:15.497 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:15.497 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:15.497 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:15.497 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1644 2020-01-16T00:13:15.497 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.497 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:15.498 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.498 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:15.498 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847556+0000", 2020-01-16T00:13:15.498 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:15.498 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.498 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.498 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:15.499 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847556+0000", 2020-01-16T00:13:15.499 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:15.499 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.499 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.499 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:15.499 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847555+0000", 2020-01-16T00:13:15.499 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4640000000000002e-06 2020-01-16T00:13:15.499 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.500 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.500 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:15.500 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847559+0000", 2020-01-16T00:13:15.500 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4580000000000003e-06 2020-01-16T00:13:15.500 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.500 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.500 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:15.501 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847563+0000", 2020-01-16T00:13:15.501 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.613e-06 2020-01-16T00:13:15.501 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.501 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.501 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:15.502 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847567+0000", 2020-01-16T00:13:15.502 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000145045 2020-01-16T00:13:15.502 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.502 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.503 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:15.503 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847712+0000", 2020-01-16T00:13:15.503 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6480000000000003e-06 2020-01-16T00:13:15.503 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.503 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.503 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:13:15.503 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847720+0000", 2020-01-16T00:13:15.504 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0055645269999999997 2020-01-16T00:13:15.504 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.504 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.504 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:15.504 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.853284+0000", 2020-01-16T00:13:15.504 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00026767400000000003 2020-01-16T00:13:15.504 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.505 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.505 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:15.505 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.853552+0000", 2020-01-16T00:13:15.505 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021884169999999998 2020-01-16T00:13:15.505 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.505 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.505 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:15.506 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.855740+0000", 2020-01-16T00:13:15.506 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0081843639999999995 2020-01-16T00:13:15.506 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.506 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:15.506 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.506 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.506 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.506 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1645 3.1 3:8c6a83c0:::smithi12313934-36:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:13:15.507 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847602+0000", 2020-01-16T00:13:15.507 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 23.578110654, 2020-01-16T00:13:15.507 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0085169679999999998, 2020-01-16T00:13:15.507 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:15.507 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:15.507 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:15.507 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:15.508 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:15.508 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1645 2020-01-16T00:13:15.508 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.508 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:15.508 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.508 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:15.508 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847602+0000", 2020-01-16T00:13:15.508 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:15.509 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.509 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.509 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:15.509 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847602+0000", 2020-01-16T00:13:15.509 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:15.509 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.510 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.510 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:15.510 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847601+0000", 2020-01-16T00:13:15.510 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7011e-05 2020-01-16T00:13:15.511 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.511 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.511 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:15.511 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847618+0000", 2020-01-16T00:13:15.511 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.187e-06 2020-01-16T00:13:15.511 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.512 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.512 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:15.512 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847625+0000", 2020-01-16T00:13:15.512 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9879999999999998e-06 2020-01-16T00:13:15.512 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.512 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.512 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:15.513 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847629+0000", 2020-01-16T00:13:15.513 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000129694 2020-01-16T00:13:15.513 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.513 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.513 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:15.513 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847759+0000", 2020-01-16T00:13:15.513 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8379999999999996e-06 2020-01-16T00:13:15.514 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.514 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.514 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:13:15.514 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847766+0000", 2020-01-16T00:13:15.514 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0079977390000000002 2020-01-16T00:13:15.515 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.515 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.515 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:15.515 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.855764+0000", 2020-01-16T00:13:15.515 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2467999999999996e-05 2020-01-16T00:13:15.516 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.516 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.516 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:15.516 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.855856+0000", 2020-01-16T00:13:15.516 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00026275099999999998 2020-01-16T00:13:15.516 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.517 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.517 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:15.517 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.856119+0000", 2020-01-16T00:13:15.517 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0085169679999999998 2020-01-16T00:13:15.517 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.517 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:15.517 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.518 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.518 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.518 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1649 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~2084068,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:13:15.518 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847683+0000", 2020-01-16T00:13:15.518 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 23.578029402999999, 2020-01-16T00:13:15.518 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010654691000000001, 2020-01-16T00:13:15.518 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:15.519 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:15.519 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:15.519 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:15.519 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:15.519 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1649 2020-01-16T00:13:15.519 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.519 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:15.520 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.520 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:15.520 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847683+0000", 2020-01-16T00:13:15.520 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:15.520 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.520 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.521 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:15.521 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847683+0000", 2020-01-16T00:13:15.521 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:15.521 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.521 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.522 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:15.522 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847682+0000", 2020-01-16T00:13:15.522 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8569999999999997e-06 2020-01-16T00:13:15.522 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.522 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.523 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:15.523 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847689+0000", 2020-01-16T00:13:15.523 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8090000000000004e-06 2020-01-16T00:13:15.523 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.523 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.523 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:15.523 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847696+0000", 2020-01-16T00:13:15.523 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2690000000000001e-06 2020-01-16T00:13:15.524 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.524 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.524 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:15.524 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847700+0000", 2020-01-16T00:13:15.524 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8189999999999993e-05 2020-01-16T00:13:15.524 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.524 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.525 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:15.525 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847798+0000", 2020-01-16T00:13:15.525 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7129999999999999e-06 2020-01-16T00:13:15.525 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.525 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.525 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:13:15.525 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847802+0000", 2020-01-16T00:13:15.525 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083913049999999999 2020-01-16T00:13:15.526 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.526 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.526 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:15.526 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.856193+0000", 2020-01-16T00:13:15.526 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034400300000000001 2020-01-16T00:13:15.526 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.526 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.527 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:15.527 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.856537+0000", 2020-01-16T00:13:15.527 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018006820000000001 2020-01-16T00:13:15.527 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.527 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.527 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:15.527 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858338+0000", 2020-01-16T00:13:15.528 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010654691000000001 2020-01-16T00:13:15.528 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.529 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:15.529 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.529 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.529 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.529 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1650 3.1 3:8de1d8a3:::smithi12313934-13:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:13:15.530 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847717+0000", 2020-01-16T00:13:15.530 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 23.57799545, 2020-01-16T00:13:15.530 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010913658, 2020-01-16T00:13:15.530 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:15.530 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:15.530 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:15.530 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:15.531 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:15.531 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1650 2020-01-16T00:13:15.531 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.531 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:15.531 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.531 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:15.531 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847717+0000", 2020-01-16T00:13:15.531 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:15.532 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.532 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.532 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:15.532 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847717+0000", 2020-01-16T00:13:15.532 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:15.532 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.533 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.533 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:15.533 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847716+0000", 2020-01-16T00:13:15.533 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.433e-06 2020-01-16T00:13:15.533 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.533 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.533 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:15.533 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847720+0000", 2020-01-16T00:13:15.534 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1800000000000004e-06 2020-01-16T00:13:15.534 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.534 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.534 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:15.534 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847725+0000", 2020-01-16T00:13:15.534 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6689999999999997e-06 2020-01-16T00:13:15.534 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.535 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.535 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:15.535 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847730+0000", 2020-01-16T00:13:15.535 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000102468 2020-01-16T00:13:15.535 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.535 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.535 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:15.535 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847832+0000", 2020-01-16T00:13:15.536 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3999999999999997e-06 2020-01-16T00:13:15.536 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.536 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.536 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:13:15.536 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847838+0000", 2020-01-16T00:13:15.536 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010519395000000001 2020-01-16T00:13:15.536 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.536 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.537 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:15.537 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858358+0000", 2020-01-16T00:13:15.537 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7600000000000002e-05 2020-01-16T00:13:15.537 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.537 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.537 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:15.537 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858445+0000", 2020-01-16T00:13:15.537 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018515900000000001 2020-01-16T00:13:15.538 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.538 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.538 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:15.538 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858631+0000", 2020-01-16T00:13:15.538 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010913658 2020-01-16T00:13:15.538 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.538 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:15.538 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.539 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.539 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.539 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1651 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~2084068,checksum] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:13:15.539 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847752+0000", 2020-01-16T00:13:15.539 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 23.577960833999999, 2020-01-16T00:13:15.539 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011734249, 2020-01-16T00:13:15.540 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:15.540 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:15.540 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:15.540 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:15.540 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:15.540 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1651 2020-01-16T00:13:15.540 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.540 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:15.541 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.541 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:15.541 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847752+0000", 2020-01-16T00:13:15.541 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:15.541 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.541 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.541 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:15.541 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847752+0000", 2020-01-16T00:13:15.542 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:15.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.542 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.542 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:15.542 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847751+0000", 2020-01-16T00:13:15.542 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1980000000000001e-06 2020-01-16T00:13:15.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.543 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.543 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:15.543 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847754+0000", 2020-01-16T00:13:15.543 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6679999999999999e-06 2020-01-16T00:13:15.543 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.543 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.543 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:15.543 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847758+0000", 2020-01-16T00:13:15.544 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2500000000000001e-06 2020-01-16T00:13:15.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.544 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.544 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:15.544 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847760+0000", 2020-01-16T00:13:15.544 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7614000000000004e-05 2020-01-16T00:13:15.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.545 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.545 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:15.545 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847858+0000", 2020-01-16T00:13:15.545 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7950000000000004e-06 2020-01-16T00:13:15.545 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.545 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.545 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:13:15.546 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847864+0000", 2020-01-16T00:13:15.546 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010790345 2020-01-16T00:13:15.546 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.546 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.547 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:15.547 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858654+0000", 2020-01-16T00:13:15.547 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000141466 2020-01-16T00:13:15.547 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.547 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.548 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:15.548 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858795+0000", 2020-01-16T00:13:15.548 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00069037899999999997 2020-01-16T00:13:15.548 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.549 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.549 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:15.549 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.859486+0000", 2020-01-16T00:13:15.549 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011734249 2020-01-16T00:13:15.549 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.549 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:15.550 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.550 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.550 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.550 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:55 2.1 e30/19 2:91f85463:::smithi12313934-24 low_tier:head v 30'56, mlcod=29'54)", 2020-01-16T00:13:15.550 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.738134+0000", 2020-01-16T00:13:15.550 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 22.687578548000001, 2020-01-16T00:13:15.551 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0086509570000000008, 2020-01-16T00:13:15.551 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:15.551 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:15.551 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:15.551 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.551 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:15.551 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738134+0000", 2020-01-16T00:13:15.552 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:15.552 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.552 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.552 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:15.552 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738134+0000", 2020-01-16T00:13:15.552 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:15.552 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.552 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.553 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:15.553 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738134+0000", 2020-01-16T00:13:15.553 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00053712599999999997 2020-01-16T00:13:15.553 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.553 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.553 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:15.553 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738671+0000", 2020-01-16T00:13:15.553 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4059999999999999e-05 2020-01-16T00:13:15.554 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.554 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.554 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:15.554 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738685+0000", 2020-01-16T00:13:15.554 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7680000000000008e-06 2020-01-16T00:13:15.554 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.554 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.554 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:15.555 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738692+0000", 2020-01-16T00:13:15.555 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1273999999999996e-05 2020-01-16T00:13:15.555 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.555 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.555 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:15.555 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738764+0000", 2020-01-16T00:13:15.555 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6827999999999999e-05 2020-01-16T00:13:15.556 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.556 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.556 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:15.556 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738781+0000", 2020-01-16T00:13:15.556 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0079370740000000006 2020-01-16T00:13:15.556 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.557 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.557 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:15.557 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746718+0000", 2020-01-16T00:13:15.557 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7181000000000006e-05 2020-01-16T00:13:15.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.557 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.557 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:15.557 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746785+0000", 2020-01-16T00:13:15.558 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0086509570000000008 2020-01-16T00:13:15.558 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.558 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:15.558 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.558 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.558 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.558 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:1668 3.3 e30/20 3:c478bfe8:::smithi12313934-10:head v 30'77, mlcod=29'75)", 2020-01-16T00:13:15.558 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.753756+0000", 2020-01-16T00:13:15.559 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 22.671956497, 2020-01-16T00:13:15.559 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0077471110000000001, 2020-01-16T00:13:15.559 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:15.559 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:15.559 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:15.559 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.560 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:15.560 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.753756+0000", 2020-01-16T00:13:15.560 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:15.560 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.560 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.560 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:15.560 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.753756+0000", 2020-01-16T00:13:15.561 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:15.561 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.561 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.561 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:15.561 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.753754+0000", 2020-01-16T00:13:15.561 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00047439299999999998 2020-01-16T00:13:15.562 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.562 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.562 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:15.562 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.754229+0000", 2020-01-16T00:13:15.562 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1018e-05 2020-01-16T00:13:15.562 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.562 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.562 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:15.563 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.754240+0000", 2020-01-16T00:13:15.563 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1700000000000002e-06 2020-01-16T00:13:15.563 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.563 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.563 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:15.563 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.754246+0000", 2020-01-16T00:13:15.563 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0014229519999999999 2020-01-16T00:13:15.563 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.564 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.564 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:15.564 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.755669+0000", 2020-01-16T00:13:15.564 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8136e-05 2020-01-16T00:13:15.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.564 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.564 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:15.565 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.755697+0000", 2020-01-16T00:13:15.565 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0057384799999999998 2020-01-16T00:13:15.565 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.565 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.565 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:15.565 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.761435+0000", 2020-01-16T00:13:15.565 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7742000000000003e-05 2020-01-16T00:13:15.565 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.566 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.566 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:15.566 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.761503+0000", 2020-01-16T00:13:15.566 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0077471110000000001 2020-01-16T00:13:15.566 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.566 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:15.566 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.566 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.567 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.567 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2106 3.1 3:8de1d8a3:::smithi12313934-13:head [write 665610~797860,stat] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:13:15.567 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.631369+0000", 2020-01-16T00:13:15.567 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 13.794343122000001, 2020-01-16T00:13:15.567 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0091195330000000008, 2020-01-16T00:13:15.567 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:15.567 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:15.567 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:15.568 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:15.568 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:15.568 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2106 2020-01-16T00:13:15.568 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.568 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:15.568 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.568 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:15.568 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631369+0000", 2020-01-16T00:13:15.569 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:15.569 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.569 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.569 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:15.569 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631369+0000", 2020-01-16T00:13:15.569 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:15.569 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.569 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.570 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:15.570 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631367+0000", 2020-01-16T00:13:15.570 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00025714100000000001 2020-01-16T00:13:15.570 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.570 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.570 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:15.570 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631624+0000", 2020-01-16T00:13:15.571 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4946000000000001e-05 2020-01-16T00:13:15.571 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.571 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.571 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:15.571 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631639+0000", 2020-01-16T00:13:15.571 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7050000000000002e-06 2020-01-16T00:13:15.571 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.571 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.572 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:15.572 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631646+0000", 2020-01-16T00:13:15.572 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010938900000000001 2020-01-16T00:13:15.572 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.572 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.572 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:15.572 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631755+0000", 2020-01-16T00:13:15.572 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8848e-05 2020-01-16T00:13:15.573 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.573 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.573 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op must wait for map", 2020-01-16T00:13:15.573 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631774+0000", 2020-01-16T00:13:15.573 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0026027730000000001 2020-01-16T00:13:15.573 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.573 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.573 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:15.574 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.634377+0000", 2020-01-16T00:13:15.574 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9282000000000003e-05 2020-01-16T00:13:15.574 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.574 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.574 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:15.574 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.634466+0000", 2020-01-16T00:13:15.574 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000127868 2020-01-16T00:13:15.574 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.575 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.575 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:13:15.575 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.634594+0000", 2020-01-16T00:13:15.575 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030776549999999999 2020-01-16T00:13:15.575 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.575 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.575 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:15.576 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.637672+0000", 2020-01-16T00:13:15.576 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001634195 2020-01-16T00:13:15.576 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.576 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.576 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:15.576 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.639306+0000", 2020-01-16T00:13:15.576 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010810069999999999 2020-01-16T00:13:15.577 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.577 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.577 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:15.577 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640387+0000", 2020-01-16T00:13:15.577 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0543000000000001e-05 2020-01-16T00:13:15.577 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.577 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.578 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:15.578 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640428+0000", 2020-01-16T00:13:15.578 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1321000000000005e-05 2020-01-16T00:13:15.578 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.578 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.578 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:15.578 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640489+0000", 2020-01-16T00:13:15.579 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0091195330000000008 2020-01-16T00:13:15.579 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.579 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:15.579 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.579 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.579 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.579 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2107 3.1 3:8de1d8a3:::smithi12313934-13:head [write 1909329~458405] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:13:15.579 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.636786+0000", 2020-01-16T00:13:15.580 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 13.788925991999999, 2020-01-16T00:13:15.580 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093182809999999994, 2020-01-16T00:13:15.581 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:15.581 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:15.581 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:15.581 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:15.581 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:15.581 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2107 2020-01-16T00:13:15.581 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.582 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:15.583 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.583 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:15.583 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636786+0000", 2020-01-16T00:13:15.583 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:15.583 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.583 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.583 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:15.584 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636786+0000", 2020-01-16T00:13:15.584 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:15.584 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.584 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.584 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:15.584 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636784+0000", 2020-01-16T00:13:15.584 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014276299999999999 2020-01-16T00:13:15.585 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.585 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.585 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:15.585 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636927+0000", 2020-01-16T00:13:15.585 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5017999999999999e-05 2020-01-16T00:13:15.585 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.585 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.586 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:15.586 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636942+0000", 2020-01-16T00:13:15.586 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5589999999999996e-06 2020-01-16T00:13:15.586 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.586 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.586 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:15.586 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636947+0000", 2020-01-16T00:13:15.587 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00040279699999999999 2020-01-16T00:13:15.587 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.587 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.587 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:15.587 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.637350+0000", 2020-01-16T00:13:15.587 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000233506 2020-01-16T00:13:15.587 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.588 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.588 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:15.589 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.637584+0000", 2020-01-16T00:13:15.589 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029534660000000001 2020-01-16T00:13:15.589 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.589 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.589 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:15.590 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640537+0000", 2020-01-16T00:13:15.590 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012577700000000001 2020-01-16T00:13:15.590 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.590 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.590 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:15.591 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640663+0000", 2020-01-16T00:13:15.591 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000198593 2020-01-16T00:13:15.591 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.591 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.591 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:13:15.592 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640862+0000", 2020-01-16T00:13:15.592 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.005088197 2020-01-16T00:13:15.592 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.592 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.592 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:15.593 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645950+0000", 2020-01-16T00:13:15.593 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2568999999999999e-05 2020-01-16T00:13:15.593 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.593 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.593 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:15.594 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645962+0000", 2020-01-16T00:13:15.594 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3034999999999995e-05 2020-01-16T00:13:15.594 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.594 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.594 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:15.595 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646025+0000", 2020-01-16T00:13:15.595 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7762999999999997e-05 2020-01-16T00:13:15.595 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.595 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.595 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:15.596 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646063+0000", 2020-01-16T00:13:15.596 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1541000000000001e-05 2020-01-16T00:13:15.598 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.598 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.598 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:15.598 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646105+0000", 2020-01-16T00:13:15.598 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093182809999999994 2020-01-16T00:13:15.598 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.599 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:15.599 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.599 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.599 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.599 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2108 3.1 3:8de1d8a3:::smithi12313934-13:head [write 2816700~410210] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:13:15.599 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.641861+0000", 2020-01-16T00:13:15.599 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 13.783851644, 2020-01-16T00:13:15.599 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018545604, 2020-01-16T00:13:15.600 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:15.600 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:15.600 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:15.600 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:15.600 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:15.600 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2108 2020-01-16T00:13:15.600 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.600 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:15.601 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.601 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:15.601 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.641861+0000", 2020-01-16T00:13:15.601 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:15.601 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.601 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.602 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:15.602 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.641861+0000", 2020-01-16T00:13:15.602 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:15.602 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.602 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.602 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:15.602 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.641859+0000", 2020-01-16T00:13:15.602 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018466399999999999 2020-01-16T00:13:15.603 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.603 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.603 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:15.603 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642043+0000", 2020-01-16T00:13:15.603 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2999999999999999e-05 2020-01-16T00:13:15.603 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.603 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.604 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:15.604 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642056+0000", 2020-01-16T00:13:15.604 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0730000000000005e-06 2020-01-16T00:13:15.604 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.604 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.604 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:15.605 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642064+0000", 2020-01-16T00:13:15.605 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9527999999999997e-05 2020-01-16T00:13:15.605 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.605 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.605 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:15.605 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642114+0000", 2020-01-16T00:13:15.605 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036616050000000001 2020-01-16T00:13:15.605 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.606 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.606 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:15.606 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645776+0000", 2020-01-16T00:13:15.606 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012222663999999999 2020-01-16T00:13:15.606 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.606 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.606 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:15.607 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.657998+0000", 2020-01-16T00:13:15.607 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014280299999999999 2020-01-16T00:13:15.607 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.607 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.607 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:15.607 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.658141+0000", 2020-01-16T00:13:15.607 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0001102 2020-01-16T00:13:15.607 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.608 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.608 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:13:15.608 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.658251+0000", 2020-01-16T00:13:15.608 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020421219999999999 2020-01-16T00:13:15.608 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.608 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.608 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:15.608 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660293+0000", 2020-01-16T00:13:15.609 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7399999999999999e-06 2020-01-16T00:13:15.609 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.609 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.609 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:15.609 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660303+0000", 2020-01-16T00:13:15.609 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9873999999999998e-05 2020-01-16T00:13:15.609 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.610 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.610 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:15.610 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660343+0000", 2020-01-16T00:13:15.610 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4448e-05 2020-01-16T00:13:15.610 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.610 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.610 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:15.611 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660377+0000", 2020-01-16T00:13:15.611 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8915999999999999e-05 2020-01-16T00:13:15.611 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.611 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.611 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:15.611 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660406+0000", 2020-01-16T00:13:15.611 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018545604 2020-01-16T00:13:15.611 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.612 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:15.612 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.612 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.612 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.612 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2109 3.1 3:8de1d8a3:::smithi12313934-13:head [setxattr _header (56) in=7b,truncate 3865610] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:13:15.612 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.642075+0000", 2020-01-16T00:13:15.612 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 13.783637070999999, 2020-01-16T00:13:15.612 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018477655999999999, 2020-01-16T00:13:15.613 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:15.613 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:15.613 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:15.613 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:15.613 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:15.613 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2109 2020-01-16T00:13:15.613 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.614 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:15.614 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.614 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:15.614 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642075+0000", 2020-01-16T00:13:15.614 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:15.614 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.614 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.615 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:15.615 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642075+0000", 2020-01-16T00:13:15.615 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:15.615 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.615 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.615 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:15.615 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642074+0000", 2020-01-16T00:13:15.615 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4500000000000003e-06 2020-01-16T00:13:15.616 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.616 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.616 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:15.616 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642080+0000", 2020-01-16T00:13:15.616 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7229999999999998e-06 2020-01-16T00:13:15.616 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.616 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.616 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:15.617 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642087+0000", 2020-01-16T00:13:15.617 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2390000000000004e-06 2020-01-16T00:13:15.617 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.617 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.617 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:15.617 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642091+0000", 2020-01-16T00:13:15.617 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0037396550000000001 2020-01-16T00:13:15.618 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.618 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.618 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:15.618 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645830+0000", 2020-01-16T00:13:15.618 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2200000000000002e-05 2020-01-16T00:13:15.618 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.618 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.618 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:15.619 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645883+0000", 2020-01-16T00:13:15.619 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013432934000000001 2020-01-16T00:13:15.619 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.619 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.619 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:15.619 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659316+0000", 2020-01-16T00:13:15.619 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000140006 2020-01-16T00:13:15.619 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.620 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.620 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:15.620 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659456+0000", 2020-01-16T00:13:15.620 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000184873 2020-01-16T00:13:15.620 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.620 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.620 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:13:15.621 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659640+0000", 2020-01-16T00:13:15.621 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00066954999999999998 2020-01-16T00:13:15.621 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.621 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.621 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:15.621 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660310+0000", 2020-01-16T00:13:15.621 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014847699999999999 2020-01-16T00:13:15.622 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.622 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.622 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:15.622 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660459+0000", 2020-01-16T00:13:15.622 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6869e-05 2020-01-16T00:13:15.622 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.622 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.623 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:15.623 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660485+0000", 2020-01-16T00:13:15.623 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8478000000000001e-05 2020-01-16T00:13:15.623 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.623 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.623 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:15.623 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660514+0000", 2020-01-16T00:13:15.623 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9153999999999998e-05 2020-01-16T00:13:15.624 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.624 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.624 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:15.624 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660553+0000", 2020-01-16T00:13:15.624 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018477655999999999 2020-01-16T00:13:15.624 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.624 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:15.624 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.625 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.625 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.625 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2110 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-01-16T00:13:15.625 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.642112+0000", 2020-01-16T00:13:15.625 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 13.783600433, 2020-01-16T00:13:15.625 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018620198000000001, 2020-01-16T00:13:15.625 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:15.626 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:15.626 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:15.626 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:15.626 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:15.626 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2110 2020-01-16T00:13:15.626 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.626 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:15.626 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.627 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:15.627 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642112+0000", 2020-01-16T00:13:15.627 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:15.627 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.627 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.627 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:15.627 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642112+0000", 2020-01-16T00:13:15.628 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:15.628 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.628 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.628 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:15.628 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642111+0000", 2020-01-16T00:13:15.628 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4450000000000001e-06 2020-01-16T00:13:15.628 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.628 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.629 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:15.629 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642117+0000", 2020-01-16T00:13:15.629 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1567e-05 2020-01-16T00:13:15.629 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.629 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.629 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:15.629 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642129+0000", 2020-01-16T00:13:15.630 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.754e-06 2020-01-16T00:13:15.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.630 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.630 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:15.630 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642136+0000", 2020-01-16T00:13:15.630 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040239719999999998 2020-01-16T00:13:15.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.630 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.631 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:15.631 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646160+0000", 2020-01-16T00:13:15.631 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5600999999999998e-05 2020-01-16T00:13:15.631 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.631 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.631 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:15.631 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646215+0000", 2020-01-16T00:13:15.632 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013733006000000001 2020-01-16T00:13:15.632 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.632 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.632 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:15.632 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659948+0000", 2020-01-16T00:13:15.632 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000106277 2020-01-16T00:13:15.632 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.632 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.633 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:15.633 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660054+0000", 2020-01-16T00:13:15.633 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000524327 2020-01-16T00:13:15.633 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.633 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.633 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:15.633 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660579+0000", 2020-01-16T00:13:15.634 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9841999999999999e-05 2020-01-16T00:13:15.634 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.634 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.634 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:15.634 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660659+0000", 2020-01-16T00:13:15.634 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3572999999999995e-05 2020-01-16T00:13:15.634 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.634 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.635 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:15.635 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660732+0000", 2020-01-16T00:13:15.635 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018620198000000001 2020-01-16T00:13:15.635 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.635 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:15.635 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.635 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.636 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.636 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:70 2.1 e35/19 2:91f85463:::smithi12313934-24 low_tier:head v 35'66, mlcod=34'64)", 2020-01-16T00:13:15.636 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.986684+0000", 2020-01-16T00:13:15.636 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 9.4390285790000004, 2020-01-16T00:13:15.636 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012287378999999999, 2020-01-16T00:13:15.636 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:15.636 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:15.636 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:15.637 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.637 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:15.637 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986684+0000", 2020-01-16T00:13:15.637 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:15.637 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.637 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.637 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:15.638 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986684+0000", 2020-01-16T00:13:15.638 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:15.638 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.638 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.638 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:15.639 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986683+0000", 2020-01-16T00:13:15.639 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012884179999999999 2020-01-16T00:13:15.639 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.639 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.639 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:15.640 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.987971+0000", 2020-01-16T00:13:15.640 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011558200000000001 2020-01-16T00:13:15.640 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.640 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.640 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:15.640 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988087+0000", 2020-01-16T00:13:15.640 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0472000000000001e-05 2020-01-16T00:13:15.641 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.641 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.641 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:15.641 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988137+0000", 2020-01-16T00:13:15.641 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000107484 2020-01-16T00:13:15.641 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.641 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.641 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:15.642 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988245+0000", 2020-01-16T00:13:15.642 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4409999999999998e-05 2020-01-16T00:13:15.642 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.642 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.642 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:15.642 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988279+0000", 2020-01-16T00:13:15.642 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010641609999999999 2020-01-16T00:13:15.643 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.643 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.643 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:15.644 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.998921+0000", 2020-01-16T00:13:15.644 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0278000000000002e-05 2020-01-16T00:13:15.644 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.644 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.644 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:15.644 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.998971+0000", 2020-01-16T00:13:15.644 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012287378999999999 2020-01-16T00:13:15.644 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.645 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:15.645 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.645 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.645 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.645 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:71 2.1 e35/19 2:91f85463:::smithi12313934-24 low_tier:head v 35'67, mlcod=34'64)", 2020-01-16T00:13:15.645 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.991132+0000", 2020-01-16T00:13:15.645 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 9.4345806289999992, 2020-01-16T00:13:15.646 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0082681190000000009, 2020-01-16T00:13:15.646 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:15.646 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:15.646 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:15.646 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.646 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:15.646 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.991132+0000", 2020-01-16T00:13:15.646 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:15.647 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.647 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.647 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:15.647 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.991132+0000", 2020-01-16T00:13:15.647 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:15.647 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.647 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.648 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:15.648 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.991130+0000", 2020-01-16T00:13:15.648 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3583000000000003e-05 2020-01-16T00:13:15.648 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.648 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.649 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:15.649 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.991203+0000", 2020-01-16T00:13:15.649 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.1001000000000001e-05 2020-01-16T00:13:15.649 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.649 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.649 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:15.649 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.991224+0000", 2020-01-16T00:13:15.649 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9329999999999996e-06 2020-01-16T00:13:15.650 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.650 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.650 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:15.650 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.991234+0000", 2020-01-16T00:13:15.650 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0035280070000000001 2020-01-16T00:13:15.650 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.650 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.651 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:15.651 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.994762+0000", 2020-01-16T00:13:15.651 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.622e-05 2020-01-16T00:13:15.651 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.651 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.651 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:15.651 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.994788+0000", 2020-01-16T00:13:15.652 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0045652219999999999 2020-01-16T00:13:15.652 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.652 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.652 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:15.652 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999354+0000", 2020-01-16T00:13:15.652 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6396000000000001e-05 2020-01-16T00:13:15.652 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.652 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.653 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:15.653 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999400+0000", 2020-01-16T00:13:15.653 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0082681190000000009 2020-01-16T00:13:15.653 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.653 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:15.653 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.653 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.654 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.654 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:77 2.1 e36/19 2:91f85463:::smithi12313934-24 low_tier:head v 36'69, mlcod=35'67)", 2020-01-16T00:13:15.654 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.556730+0000", 2020-01-16T00:13:15.654 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 7.8689822249999999, 2020-01-16T00:13:15.654 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093007760000000002, 2020-01-16T00:13:15.654 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:15.654 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:15.655 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:15.655 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.655 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:15.655 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556730+0000", 2020-01-16T00:13:15.655 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:15.655 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.655 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.655 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:15.656 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556730+0000", 2020-01-16T00:13:15.656 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:15.656 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.656 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.656 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:15.656 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556728+0000", 2020-01-16T00:13:15.656 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00056509599999999996 2020-01-16T00:13:15.656 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.657 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.657 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:15.657 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.557293+0000", 2020-01-16T00:13:15.657 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0225e-05 2020-01-16T00:13:15.657 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.657 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.657 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:15.658 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.557314+0000", 2020-01-16T00:13:15.658 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8989999999999996e-06 2020-01-16T00:13:15.658 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.658 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.658 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:15.658 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.557323+0000", 2020-01-16T00:13:15.658 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7923000000000002e-05 2020-01-16T00:13:15.659 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.659 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.659 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:15.659 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.557371+0000", 2020-01-16T00:13:15.659 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9963999999999999e-05 2020-01-16T00:13:15.659 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.659 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.659 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:15.660 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.557411+0000", 2020-01-16T00:13:15.660 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0085873000000000008 2020-01-16T00:13:15.660 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.660 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.660 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:15.660 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.565999+0000", 2020-01-16T00:13:15.660 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2326000000000002e-05 2020-01-16T00:13:15.660 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:15.661 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:15.661 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:15.661 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566031+0000", 2020-01-16T00:13:15.661 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093007760000000002 2020-01-16T00:13:15.661 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.661 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:15.661 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.662 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:15.662 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:15.662 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:15.663 INFO:tasks.rados.rados.0.smithi123.stdout:1123: finishing write tid 1 to smithi12313934-17 2020-01-16T00:13:15.663 INFO:tasks.rados.rados.0.smithi123.stdout:1123: finishing write tid 2 to smithi12313934-17 2020-01-16T00:13:15.664 INFO:tasks.rados.rados.0.smithi123.stdout:1123: finishing write tid 3 to smithi12313934-17 2020-01-16T00:13:15.664 INFO:tasks.rados.rados.0.smithi123.stdout:1123: finishing write tid 5 to smithi12313934-17 2020-01-16T00:13:15.664 INFO:tasks.rados.rados.0.smithi123.stdout:1123: finishing write tid 6 to smithi12313934-17 2020-01-16T00:13:15.664 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 17 v 123 (ObjNum 314 snap 0 seq_num 314) dirty exists 2020-01-16T00:13:15.664 INFO:tasks.rados.rados.0.smithi123.stdout:1123: left oid 17 (ObjNum 314 snap 0 seq_num 314) 2020-01-16T00:13:15.664 INFO:tasks.rados.rados.0.smithi123.stdout:1124: finishing write tid 1 to smithi12313934-7 2020-01-16T00:13:15.664 INFO:tasks.rados.rados.0.smithi123.stdout:1124: finishing write tid 2 to smithi12313934-7 2020-01-16T00:13:15.665 INFO:tasks.rados.rados.0.smithi123.stdout:1124: finishing write tid 3 to smithi12313934-7 2020-01-16T00:13:15.665 INFO:tasks.rados.rados.0.smithi123.stdout:1124: finishing write tid 5 to smithi12313934-7 2020-01-16T00:13:15.665 INFO:tasks.rados.rados.0.smithi123.stdout:1124: finishing write tid 6 to smithi12313934-7 2020-01-16T00:13:15.665 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 7 v 170 (ObjNum 315 snap 0 seq_num 315) dirty exists 2020-01-16T00:13:15.665 INFO:tasks.rados.rados.0.smithi123.stdout:1124: left oid 7 (ObjNum 315 snap 0 seq_num 315) 2020-01-16T00:13:15.665 INFO:tasks.rados.rados.0.smithi123.stdout:1125: finishing write tid 1 to smithi12313934-44 2020-01-16T00:13:15.665 INFO:tasks.rados.rados.0.smithi123.stdout:1125: finishing write tid 2 to smithi12313934-44 2020-01-16T00:13:15.666 INFO:tasks.rados.rados.0.smithi123.stdout:1125: finishing write tid 3 to smithi12313934-44 2020-01-16T00:13:15.666 INFO:tasks.rados.rados.0.smithi123.stdout:1126: expect (ObjNum 304 snap 0 seq_num 304) 2020-01-16T00:13:15.666 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:15.658+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:15.666 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-01-16T00:13:15.698 INFO:tasks.rados.rados.0.smithi123.stdout:1125: finishing write tid 5 to smithi12313934-44 2020-01-16T00:13:15.698 INFO:tasks.rados.rados.0.smithi123.stdout:1125: finishing write tid 6 to smithi12313934-44 2020-01-16T00:13:15.698 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 44 v 159 (ObjNum 316 snap 0 seq_num 316) dirty exists 2020-01-16T00:13:15.699 INFO:tasks.rados.rados.0.smithi123.stdout:1125: left oid 44 (ObjNum 316 snap 0 seq_num 316) 2020-01-16T00:13:15.699 INFO:tasks.rados.rados.0.smithi123.stdout:1127: expect (ObjNum 303 snap 0 seq_num 303) 2020-01-16T00:13:15.764 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:15.758+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:15.794 INFO:teuthology.orchestra.run.smithi145.stdout:3.d 2020-01-16T00:13:15.794 INFO:teuthology.orchestra.run.smithi145.stdout:3.4 2020-01-16T00:13:15.794 INFO:teuthology.orchestra.run.smithi145.stdout:3.c 2020-01-16T00:13:15.794 INFO:teuthology.orchestra.run.smithi145.stdout:3.2 2020-01-16T00:13:15.794 INFO:teuthology.orchestra.run.smithi145.stdout:3.5 2020-01-16T00:13:15.795 INFO:teuthology.orchestra.run.smithi145.stdout:2.3 2020-01-16T00:13:15.795 INFO:teuthology.orchestra.run.smithi145.stdout:2.2 2020-01-16T00:13:15.795 INFO:teuthology.orchestra.run.smithi145.stdout:1.4 2020-01-16T00:13:15.795 INFO:teuthology.orchestra.run.smithi145.stdout:1.1 2020-01-16T00:13:15.804 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:15.804 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:15.805 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:13:15.805 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:15.814 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-01-16T00:13:15.860 INFO:teuthology.orchestra.run.smithi123.stderr:nodeep-scrub is unset 2020-01-16T00:13:15.865 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:15.860+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:15.877 INFO:tasks.rados.rados.0.smithi123.stdout:1128: finishing write tid 1 to smithi12313934-15 2020-01-16T00:13:15.878 INFO:tasks.rados.rados.0.smithi123.stdout:1128: finishing write tid 2 to smithi12313934-15 2020-01-16T00:13:15.878 INFO:tasks.rados.rados.0.smithi123.stdout:1128: finishing write tid 3 to smithi12313934-15 2020-01-16T00:13:15.878 INFO:tasks.rados.rados.0.smithi123.stdout:1128: finishing write tid 5 to smithi12313934-15 2020-01-16T00:13:15.878 INFO:tasks.rados.rados.0.smithi123.stdout:1128: finishing write tid 6 to smithi12313934-15 2020-01-16T00:13:15.878 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 15 v 120 (ObjNum 317 snap 0 seq_num 317) dirty exists 2020-01-16T00:13:15.878 INFO:tasks.rados.rados.0.smithi123.stdout:1128: left oid 15 (ObjNum 317 snap 0 seq_num 317) 2020-01-16T00:13:15.878 INFO:tasks.rados.rados.0.smithi123.stdout:1129: finishing write tid 1 to smithi12313934-14 2020-01-16T00:13:15.879 INFO:tasks.rados.rados.0.smithi123.stdout:1129: finishing write tid 2 to smithi12313934-14 2020-01-16T00:13:15.879 INFO:tasks.rados.rados.0.smithi123.stdout:1129: finishing write tid 3 to smithi12313934-14 2020-01-16T00:13:15.879 INFO:tasks.rados.rados.0.smithi123.stdout:1129: finishing write tid 5 to smithi12313934-14 2020-01-16T00:13:15.879 INFO:tasks.rados.rados.0.smithi123.stdout:1129: finishing write tid 6 to smithi12313934-14 2020-01-16T00:13:15.879 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 14 v 163 (ObjNum 318 snap 0 seq_num 318) dirty exists 2020-01-16T00:13:15.879 INFO:tasks.rados.rados.0.smithi123.stdout:1129: left oid 14 (ObjNum 318 snap 0 seq_num 318) 2020-01-16T00:13:15.879 INFO:tasks.rados.rados.0.smithi123.stdout:1119: done (15 left) 2020-01-16T00:13:15.879 INFO:tasks.rados.rados.0.smithi123.stdout:1120: done (14 left) 2020-01-16T00:13:15.880 INFO:tasks.rados.rados.0.smithi123.stdout:1122: done (13 left) 2020-01-16T00:13:15.880 INFO:tasks.rados.rados.0.smithi123.stdout:1123: done (12 left) 2020-01-16T00:13:15.880 INFO:tasks.rados.rados.0.smithi123.stdout:1124: done (11 left) 2020-01-16T00:13:15.880 INFO:tasks.rados.rados.0.smithi123.stdout:1125: done (10 left) 2020-01-16T00:13:15.880 INFO:tasks.rados.rados.0.smithi123.stdout:1126: done (9 left) 2020-01-16T00:13:15.880 INFO:tasks.rados.rados.0.smithi123.stdout:1127: done (8 left) 2020-01-16T00:13:15.880 INFO:tasks.rados.rados.0.smithi123.stdout:1128: done (7 left) 2020-01-16T00:13:15.880 INFO:tasks.rados.rados.0.smithi123.stdout:1129: done (6 left) 2020-01-16T00:13:15.881 INFO:tasks.rados.rados.0.smithi123.stdout:1136: write (excl) oid 20 current snap is 0 2020-01-16T00:13:15.881 INFO:tasks.rados.rados.0.smithi123.stdout:1136: seq_num 322 ranges {611854=717004,2126143=561767,3259733=152121} 2020-01-16T00:13:15.884 INFO:tasks.rados.rados.0.smithi123.stdout:1136: writing smithi12313934-20 from 611854 to 1328858 tid 1 2020-01-16T00:13:15.892 INFO:tasks.rados.rados.0.smithi123.stdout:1136: writing smithi12313934-20 from 2126143 to 2687910 tid 2 2020-01-16T00:13:15.896 INFO:tasks.rados.rados.0.smithi123.stdout:1136: writing smithi12313934-20 from 3259733 to 3411854 tid 3 2020-01-16T00:13:15.896 INFO:tasks.rados.rados.0.smithi123.stdout:1137: delete oid 43 current snap is 0 2020-01-16T00:13:15.897 INFO:tasks.rados.rados.0.smithi123.stdout:1133: expect (ObjNum 294 snap 0 seq_num 294) 2020-01-16T00:13:15.961 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:15.962 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:15.962 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:13:15.962 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:13:15.962 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:15.966 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:15.962+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:15.970 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-01-16T00:13:16.067 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:16.062+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:16.126 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:16.126 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:13:16.126 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:13:16.126 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:13:16.127 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.127 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:922 3.8 3:113947f2:::smithi12313934-12:head [write 443718~615603,stat] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:16.127 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.863202+0000", 2020-01-16T00:13:16.127 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 38.256973184000003, 2020-01-16T00:13:16.127 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017152697000000001, 2020-01-16T00:13:16.127 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:16.128 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:16.128 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:16.128 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:16.128 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:16.128 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 922 2020-01-16T00:13:16.128 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.128 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:16.129 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.129 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:16.129 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863202+0000", 2020-01-16T00:13:16.129 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:16.129 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.129 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.129 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:16.129 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863202+0000", 2020-01-16T00:13:16.130 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:16.130 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.130 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.130 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:16.130 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863200+0000", 2020-01-16T00:13:16.130 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00038569199999999998 2020-01-16T00:13:16.130 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.131 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.131 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:16.131 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863585+0000", 2020-01-16T00:13:16.131 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5685e-05 2020-01-16T00:13:16.131 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.131 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.131 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:16.131 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863601+0000", 2020-01-16T00:13:16.132 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.4734000000000001e-05 2020-01-16T00:13:16.132 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.132 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.132 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:16.132 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863626+0000", 2020-01-16T00:13:16.132 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1037e-05 2020-01-16T00:13:16.133 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.133 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.133 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.133 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863667+0000", 2020-01-16T00:13:16.133 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016041499999999999 2020-01-16T00:13:16.134 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.134 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.134 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:16.134 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863827+0000", 2020-01-16T00:13:16.135 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0094286430000000004 2020-01-16T00:13:16.135 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.135 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.135 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.135 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.873256+0000", 2020-01-16T00:13:16.135 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2867999999999994e-05 2020-01-16T00:13:16.135 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.136 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.136 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:16.136 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.873329+0000", 2020-01-16T00:13:16.136 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021174789999999998 2020-01-16T00:13:16.136 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.136 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.136 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.136 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.875446+0000", 2020-01-16T00:13:16.137 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011244200000000001 2020-01-16T00:13:16.137 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.137 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.137 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:16.137 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.875559+0000", 2020-01-16T00:13:16.137 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021772600000000001 2020-01-16T00:13:16.137 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.138 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.138 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1,7", 2020-01-16T00:13:16.138 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.875776+0000", 2020-01-16T00:13:16.138 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0028978860000000001 2020-01-16T00:13:16.138 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.138 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.139 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.139 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.878674+0000", 2020-01-16T00:13:16.139 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.324e-05 2020-01-16T00:13:16.139 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.139 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.139 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:16.139 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.878687+0000", 2020-01-16T00:13:16.140 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0014896180000000001 2020-01-16T00:13:16.140 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.140 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.140 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.140 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880177+0000", 2020-01-16T00:13:16.140 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6881000000000001e-05 2020-01-16T00:13:16.140 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.140 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.141 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:16.141 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880224+0000", 2020-01-16T00:13:16.141 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013096700000000001 2020-01-16T00:13:16.141 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.141 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.141 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:16.141 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880355+0000", 2020-01-16T00:13:16.142 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017152697000000001 2020-01-16T00:13:16.142 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.142 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:16.142 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.142 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.142 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.142 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:923 3.8 3:113947f2:::smithi12313934-12:head [write 1564138~470341] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:16.143 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.869292+0000", 2020-01-16T00:13:16.143 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 38.250882853999997, 2020-01-16T00:13:16.143 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017661620999999999, 2020-01-16T00:13:16.143 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:16.143 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:16.143 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:16.144 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:16.144 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:16.144 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 923 2020-01-16T00:13:16.144 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.144 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:16.144 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.144 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:16.145 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869292+0000", 2020-01-16T00:13:16.145 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:16.145 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.145 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.145 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:16.145 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869292+0000", 2020-01-16T00:13:16.145 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:16.146 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.146 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.146 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:16.146 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869291+0000", 2020-01-16T00:13:16.146 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00049842899999999999 2020-01-16T00:13:16.146 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.146 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.147 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:16.147 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869789+0000", 2020-01-16T00:13:16.147 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1014999999999999e-05 2020-01-16T00:13:16.147 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.147 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.147 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:16.148 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869830+0000", 2020-01-16T00:13:16.148 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7880000000000007e-06 2020-01-16T00:13:16.148 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.148 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.148 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:16.148 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869838+0000", 2020-01-16T00:13:16.148 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8346999999999997e-05 2020-01-16T00:13:16.149 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.149 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.149 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.149 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869876+0000", 2020-01-16T00:13:16.149 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9726999999999999e-05 2020-01-16T00:13:16.149 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.150 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.150 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:16.150 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869936+0000", 2020-01-16T00:13:16.150 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003434917 2020-01-16T00:13:16.150 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.150 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.150 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.151 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.873371+0000", 2020-01-16T00:13:16.151 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118808 2020-01-16T00:13:16.151 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.151 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.151 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:16.151 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.873490+0000", 2020-01-16T00:13:16.151 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036712910000000001 2020-01-16T00:13:16.152 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.152 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.152 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.152 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877161+0000", 2020-01-16T00:13:16.152 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9909999999999999e-05 2020-01-16T00:13:16.152 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.152 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.152 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:16.153 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877241+0000", 2020-01-16T00:13:16.153 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031619970000000002 2020-01-16T00:13:16.153 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.153 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.153 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.154 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880403+0000", 2020-01-16T00:13:16.154 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013071099999999999 2020-01-16T00:13:16.154 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.154 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.154 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:16.154 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880534+0000", 2020-01-16T00:13:16.154 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00032823500000000002 2020-01-16T00:13:16.154 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.155 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.155 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1,7", 2020-01-16T00:13:16.155 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880862+0000", 2020-01-16T00:13:16.155 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00513862 2020-01-16T00:13:16.155 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.155 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.155 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:16.156 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886001+0000", 2020-01-16T00:13:16.156 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8869000000000001e-05 2020-01-16T00:13:16.156 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.156 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.156 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.156 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886030+0000", 2020-01-16T00:13:16.156 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00081600200000000003 2020-01-16T00:13:16.156 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.157 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.157 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.157 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886846+0000", 2020-01-16T00:13:16.157 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4095999999999998e-05 2020-01-16T00:13:16.157 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.157 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.157 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:16.158 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886900+0000", 2020-01-16T00:13:16.158 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.448e-05 2020-01-16T00:13:16.158 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.158 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.158 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:16.158 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886954+0000", 2020-01-16T00:13:16.158 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017661620999999999 2020-01-16T00:13:16.159 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.159 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:16.159 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.159 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.159 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.159 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1660 3.7 3:e1be6d3b:::smithi12313934-24:head [write 2529567~344430] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:13:16.159 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.735048+0000", 2020-01-16T00:13:16.160 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 23.385127411999999, 2020-01-16T00:13:16.160 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015942654000000001, 2020-01-16T00:13:16.160 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:16.160 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:16.160 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:16.161 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:16.161 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:16.161 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1660 2020-01-16T00:13:16.161 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.162 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:16.162 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.162 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:16.162 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735048+0000", 2020-01-16T00:13:16.163 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:16.163 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.163 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.164 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:16.164 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735048+0000", 2020-01-16T00:13:16.164 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:16.164 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.165 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.165 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:16.165 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735047+0000", 2020-01-16T00:13:16.165 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012866 2020-01-16T00:13:16.165 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.166 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.166 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:16.166 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735175+0000", 2020-01-16T00:13:16.166 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.34e-05 2020-01-16T00:13:16.167 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.167 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.167 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:16.167 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735189+0000", 2020-01-16T00:13:16.167 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8479999999999997e-06 2020-01-16T00:13:16.168 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.168 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.168 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:16.168 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735194+0000", 2020-01-16T00:13:16.168 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5005000000000001e-05 2020-01-16T00:13:16.169 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.169 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.169 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.169 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735229+0000", 2020-01-16T00:13:16.170 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001253035 2020-01-16T00:13:16.170 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.170 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.170 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:16.170 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736483+0000", 2020-01-16T00:13:16.170 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012376109 2020-01-16T00:13:16.170 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.171 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.171 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.171 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748859+0000", 2020-01-16T00:13:16.171 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011718999999999999 2020-01-16T00:13:16.171 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.171 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.171 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:16.172 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748976+0000", 2020-01-16T00:13:16.172 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019464999999999999 2020-01-16T00:13:16.172 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.172 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.172 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,7", 2020-01-16T00:13:16.172 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.749170+0000", 2020-01-16T00:13:16.172 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0015637719999999999 2020-01-16T00:13:16.172 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.173 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.173 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:16.173 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.750734+0000", 2020-01-16T00:13:16.173 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.1329000000000006e-05 2020-01-16T00:13:16.173 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.173 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.173 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.174 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.750826+0000", 2020-01-16T00:13:16.174 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9779999999999998e-05 2020-01-16T00:13:16.175 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.175 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.175 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.175 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.750915+0000", 2020-01-16T00:13:16.175 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0170000000000001e-05 2020-01-16T00:13:16.176 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.176 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.176 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:16.176 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.750946+0000", 2020-01-16T00:13:16.176 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5064000000000001e-05 2020-01-16T00:13:16.177 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.177 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.177 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:16.177 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.750991+0000", 2020-01-16T00:13:16.178 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015942654000000001 2020-01-16T00:13:16.178 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.178 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:16.178 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.179 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.179 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.179 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1661 3.7 3:e1be6d3b:::smithi12313934-24:head [setxattr _header (56) in=7b,truncate 2873997] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:13:16.179 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.735205+0000", 2020-01-16T00:13:16.180 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 23.384970612, 2020-01-16T00:13:16.180 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016515230999999998, 2020-01-16T00:13:16.180 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:16.180 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:16.181 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:16.181 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:16.181 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:16.181 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1661 2020-01-16T00:13:16.182 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.182 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:16.182 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.183 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:16.183 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735205+0000", 2020-01-16T00:13:16.183 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:16.183 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.184 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.184 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:16.184 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735205+0000", 2020-01-16T00:13:16.185 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:16.185 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.185 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.185 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:16.186 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735204+0000", 2020-01-16T00:13:16.186 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5369999999999997e-06 2020-01-16T00:13:16.186 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.186 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.187 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:16.187 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735209+0000", 2020-01-16T00:13:16.188 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9839999999999998e-06 2020-01-16T00:13:16.188 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.188 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.188 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:16.188 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735215+0000", 2020-01-16T00:13:16.188 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0290000000000002e-06 2020-01-16T00:13:16.189 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.189 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.189 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:16.189 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735221+0000", 2020-01-16T00:13:16.189 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012977889999999999 2020-01-16T00:13:16.189 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.189 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.190 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.190 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736519+0000", 2020-01-16T00:13:16.190 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7575999999999999e-05 2020-01-16T00:13:16.190 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.190 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.190 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:16.190 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736577+0000", 2020-01-16T00:13:16.191 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013478469999999999 2020-01-16T00:13:16.191 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.191 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.191 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.191 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.750055+0000", 2020-01-16T00:13:16.191 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0983999999999998e-05 2020-01-16T00:13:16.191 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.192 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.192 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:16.192 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.750116+0000", 2020-01-16T00:13:16.192 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7159999999999998e-05 2020-01-16T00:13:16.192 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.192 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.192 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,7", 2020-01-16T00:13:16.193 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.750213+0000", 2020-01-16T00:13:16.193 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00063694100000000003 2020-01-16T00:13:16.193 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.193 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.193 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.193 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.750850+0000", 2020-01-16T00:13:16.193 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000257447 2020-01-16T00:13:16.194 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.194 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.194 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.194 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.751108+0000", 2020-01-16T00:13:16.195 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00054074500000000003 2020-01-16T00:13:16.195 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.195 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.195 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:16.195 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.751648+0000", 2020-01-16T00:13:16.195 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7606999999999998e-05 2020-01-16T00:13:16.196 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.196 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.196 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:16.196 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.751676+0000", 2020-01-16T00:13:16.196 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3871000000000001e-05 2020-01-16T00:13:16.196 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.196 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.197 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:16.197 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.751720+0000", 2020-01-16T00:13:16.197 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016515230999999998 2020-01-16T00:13:16.197 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.197 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:16.197 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.197 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.197 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.198 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1662 3.7 3:e1be6d3b:::smithi12313934-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-01-16T00:13:16.198 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.735228+0000", 2020-01-16T00:13:16.198 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 23.384947249, 2020-01-16T00:13:16.198 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016641372000000001, 2020-01-16T00:13:16.198 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:16.198 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:16.198 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:16.199 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:16.199 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:16.199 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1662 2020-01-16T00:13:16.199 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.199 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:16.199 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.199 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:16.199 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735228+0000", 2020-01-16T00:13:16.200 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:16.200 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.200 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.200 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:16.200 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735228+0000", 2020-01-16T00:13:16.200 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:16.200 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.200 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.201 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:16.201 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735228+0000", 2020-01-16T00:13:16.201 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6260000000000001e-06 2020-01-16T00:13:16.201 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.201 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.201 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:16.201 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735231+0000", 2020-01-16T00:13:16.202 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5429999999999999e-06 2020-01-16T00:13:16.202 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.202 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.202 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:16.202 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735238+0000", 2020-01-16T00:13:16.202 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1760000000000001e-06 2020-01-16T00:13:16.202 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.202 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.203 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:16.203 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735243+0000", 2020-01-16T00:13:16.203 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013610829999999999 2020-01-16T00:13:16.203 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.203 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.203 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.203 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736604+0000", 2020-01-16T00:13:16.203 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2194000000000002e-05 2020-01-16T00:13:16.204 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.204 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.204 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:16.204 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736656+0000", 2020-01-16T00:13:16.204 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013872425000000001 2020-01-16T00:13:16.204 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.204 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.205 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.205 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.750529+0000", 2020-01-16T00:13:16.205 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011952599999999999 2020-01-16T00:13:16.205 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.205 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.205 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:16.206 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.750648+0000", 2020-01-16T00:13:16.206 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001095171 2020-01-16T00:13:16.206 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.206 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.206 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.206 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.751743+0000", 2020-01-16T00:13:16.206 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3576000000000003e-05 2020-01-16T00:13:16.207 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.207 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.207 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:16.207 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.751807+0000", 2020-01-16T00:13:16.207 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2556000000000002e-05 2020-01-16T00:13:16.207 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.207 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.207 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:16.208 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.751869+0000", 2020-01-16T00:13:16.208 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016641372000000001 2020-01-16T00:13:16.208 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.208 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:16.208 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.208 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.208 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.208 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1698 3.7 3:e8af1ebf:::smithi12313934-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-01-16T00:13:16.209 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.797614+0000", 2020-01-16T00:13:16.209 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 23.322561769, 2020-01-16T00:13:16.209 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014685609000000001, 2020-01-16T00:13:16.209 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:16.209 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:16.209 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:16.209 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:16.210 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:16.210 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1698 2020-01-16T00:13:16.210 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.210 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:16.210 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.210 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:16.210 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.797614+0000", 2020-01-16T00:13:16.210 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:16.211 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.211 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.211 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:16.211 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.797614+0000", 2020-01-16T00:13:16.211 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:16.211 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.211 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.211 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:16.212 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.797613+0000", 2020-01-16T00:13:16.212 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8020000000000002e-06 2020-01-16T00:13:16.212 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.212 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.212 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:16.212 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.797617+0000", 2020-01-16T00:13:16.212 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3980000000000002e-06 2020-01-16T00:13:16.212 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.213 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.213 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:16.213 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.797624+0000", 2020-01-16T00:13:16.213 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.76e-06 2020-01-16T00:13:16.213 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.213 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.213 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:16.214 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.797628+0000", 2020-01-16T00:13:16.214 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052759790000000001 2020-01-16T00:13:16.214 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.214 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.214 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.214 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.802904+0000", 2020-01-16T00:13:16.214 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014863100000000001 2020-01-16T00:13:16.214 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.215 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.215 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:16.215 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.803052+0000", 2020-01-16T00:13:16.215 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0082008370000000008 2020-01-16T00:13:16.215 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.216 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.216 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.216 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.811253+0000", 2020-01-16T00:13:16.216 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012582399999999999 2020-01-16T00:13:16.216 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.216 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.216 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:16.216 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.811379+0000", 2020-01-16T00:13:16.217 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00077615099999999999 2020-01-16T00:13:16.217 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.217 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.217 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.217 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.812155+0000", 2020-01-16T00:13:16.217 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2181000000000005e-05 2020-01-16T00:13:16.217 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.217 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.218 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:16.218 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.812237+0000", 2020-01-16T00:13:16.218 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1796000000000003e-05 2020-01-16T00:13:16.218 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.218 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.218 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:16.218 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.812299+0000", 2020-01-16T00:13:16.219 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014685609000000001 2020-01-16T00:13:16.219 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.219 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:16.219 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.219 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.219 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.219 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2226 3.7 3:e1be6d3b:::smithi12313934-24:head [setxattr _header (56) in=7b,truncate 2100908] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:13:16.220 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.980217+0000", 2020-01-16T00:13:16.220 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 10.139958189, 2020-01-16T00:13:16.220 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023113018999999999, 2020-01-16T00:13:16.220 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:16.220 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:16.220 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:16.220 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:16.220 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:16.221 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2226 2020-01-16T00:13:16.221 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.221 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:16.221 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.221 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:16.221 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980217+0000", 2020-01-16T00:13:16.221 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:16.221 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.222 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.222 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:16.222 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980217+0000", 2020-01-16T00:13:16.222 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:16.222 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.222 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.222 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:16.223 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980216+0000", 2020-01-16T00:13:16.223 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5049999999999998e-06 2020-01-16T00:13:16.223 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.223 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.223 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:16.223 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980222+0000", 2020-01-16T00:13:16.223 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5654999999999998e-05 2020-01-16T00:13:16.223 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.224 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.224 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:16.224 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980237+0000", 2020-01-16T00:13:16.224 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9740000000000004e-06 2020-01-16T00:13:16.224 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.224 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.224 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:16.224 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980241+0000", 2020-01-16T00:13:16.225 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002947776 2020-01-16T00:13:16.225 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.225 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.225 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.225 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.983189+0000", 2020-01-16T00:13:16.226 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0027099400000000001 2020-01-16T00:13:16.226 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.226 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.226 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:16.226 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985899+0000", 2020-01-16T00:13:16.226 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014275589999999999 2020-01-16T00:13:16.226 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.226 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.227 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.227 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.000175+0000", 2020-01-16T00:13:16.227 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011211 2020-01-16T00:13:16.227 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.227 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.227 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:16.227 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.000287+0000", 2020-01-16T00:13:16.228 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000229886 2020-01-16T00:13:16.228 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.228 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.228 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,7", 2020-01-16T00:13:16.228 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.000517+0000", 2020-01-16T00:13:16.228 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00095715399999999999 2020-01-16T00:13:16.228 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.229 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.229 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:16.229 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.001474+0000", 2020-01-16T00:13:16.229 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001341512 2020-01-16T00:13:16.229 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.230 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.230 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.230 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.002815+0000", 2020-01-16T00:13:16.230 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035789099999999999 2020-01-16T00:13:16.230 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.230 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.230 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.230 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003173+0000", 2020-01-16T00:13:16.231 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8571000000000005e-05 2020-01-16T00:13:16.231 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.231 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.231 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:16.231 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003242+0000", 2020-01-16T00:13:16.231 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8293000000000003e-05 2020-01-16T00:13:16.231 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.232 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.232 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:16.232 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003330+0000", 2020-01-16T00:13:16.232 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023113018999999999 2020-01-16T00:13:16.232 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.232 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:16.232 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.232 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.233 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.233 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2227 3.7 3:e1be6d3b:::smithi12313934-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e35)", 2020-01-16T00:13:16.233 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:13:16.233 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 10.139910936, 2020-01-16T00:13:16.233 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023303356000000001, 2020-01-16T00:13:16.233 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:16.233 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:16.234 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:16.234 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:16.234 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:16.234 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2227 2020-01-16T00:13:16.234 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.234 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:16.234 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.234 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:16.235 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:13:16.235 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:16.235 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.237 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.237 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:16.237 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:13:16.237 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:16.237 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.237 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.237 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:16.238 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:13:16.238 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3079999999999999e-06 2020-01-16T00:13:16.238 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.238 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.238 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:16.238 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980267+0000", 2020-01-16T00:13:16.238 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9290000000000001e-06 2020-01-16T00:13:16.239 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.239 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.239 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:16.239 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980274+0000", 2020-01-16T00:13:16.239 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7400000000000002e-06 2020-01-16T00:13:16.239 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.239 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.239 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:16.240 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980278+0000", 2020-01-16T00:13:16.240 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056624300000000004 2020-01-16T00:13:16.240 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.240 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.240 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.240 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985940+0000", 2020-01-16T00:13:16.240 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4261000000000002e-05 2020-01-16T00:13:16.241 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.241 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.241 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:16.241 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985985+0000", 2020-01-16T00:13:16.241 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015087676 2020-01-16T00:13:16.241 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.241 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.241 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.242 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.001072+0000", 2020-01-16T00:13:16.242 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011960600000000001 2020-01-16T00:13:16.242 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.242 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.242 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:16.242 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.001192+0000", 2020-01-16T00:13:16.242 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021718309999999999 2020-01-16T00:13:16.243 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.243 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.243 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.243 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003364+0000", 2020-01-16T00:13:16.243 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000102855 2020-01-16T00:13:16.243 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.243 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.243 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:16.244 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003467+0000", 2020-01-16T00:13:16.244 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010122499999999999 2020-01-16T00:13:16.244 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.244 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.244 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:16.244 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003568+0000", 2020-01-16T00:13:16.244 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023303356000000001 2020-01-16T00:13:16.244 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.245 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:16.245 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.245 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.245 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.245 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2233 3.2 3:4f8fe446:::smithi12313934-48:head [setxattr _header (56) in=7b,truncate 2117715] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:13:16.245 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.993103+0000", 2020-01-16T00:13:16.245 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 10.127072714000001, 2020-01-16T00:13:16.246 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017951763999999999, 2020-01-16T00:13:16.246 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:16.246 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:16.246 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:16.246 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:16.246 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:16.246 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2233 2020-01-16T00:13:16.247 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.247 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:16.247 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.247 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:16.247 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993103+0000", 2020-01-16T00:13:16.248 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:16.248 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.248 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.248 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:16.248 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993103+0000", 2020-01-16T00:13:16.248 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:16.248 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.249 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.249 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:16.249 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993102+0000", 2020-01-16T00:13:16.249 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9369999999999993e-06 2020-01-16T00:13:16.249 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.249 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.249 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:16.250 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993111+0000", 2020-01-16T00:13:16.250 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3340000000000002e-06 2020-01-16T00:13:16.250 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.250 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.250 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:16.250 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993119+0000", 2020-01-16T00:13:16.250 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1769999999999997e-06 2020-01-16T00:13:16.250 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.251 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.251 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:16.251 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993124+0000", 2020-01-16T00:13:16.251 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001601304 2020-01-16T00:13:16.251 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.251 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.251 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.252 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.994725+0000", 2020-01-16T00:13:16.253 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048777250000000003 2020-01-16T00:13:16.253 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.253 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.253 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:16.253 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999603+0000", 2020-01-16T00:13:16.253 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0099793880000000005 2020-01-16T00:13:16.253 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.254 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.254 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.254 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.009583+0000", 2020-01-16T00:13:16.254 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000108207 2020-01-16T00:13:16.254 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.254 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.254 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:16.255 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.009691+0000", 2020-01-16T00:13:16.255 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012862999999999999 2020-01-16T00:13:16.255 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.255 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.255 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:16.255 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.009819+0000", 2020-01-16T00:13:16.255 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00077947599999999998 2020-01-16T00:13:16.255 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.256 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.256 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:16.256 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010599+0000", 2020-01-16T00:13:16.256 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000221282 2020-01-16T00:13:16.256 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.256 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.256 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.257 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010820+0000", 2020-01-16T00:13:16.257 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7021999999999995e-05 2020-01-16T00:13:16.257 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.257 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.257 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.257 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010907+0000", 2020-01-16T00:13:16.257 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9815999999999997e-05 2020-01-16T00:13:16.258 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.258 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.258 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:16.258 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010977+0000", 2020-01-16T00:13:16.258 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7453999999999994e-05 2020-01-16T00:13:16.258 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.258 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.258 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:16.259 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.011054+0000", 2020-01-16T00:13:16.259 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017951763999999999 2020-01-16T00:13:16.259 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.259 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:16.259 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.259 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.260 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.260 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2234 3.2 3:4f8fe446:::smithi12313934-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e35)", 2020-01-16T00:13:16.260 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.993135+0000", 2020-01-16T00:13:16.260 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 10.127040260999999, 2020-01-16T00:13:16.260 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018067916999999999, 2020-01-16T00:13:16.260 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:16.261 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:16.261 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:16.261 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:16.261 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:16.261 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2234 2020-01-16T00:13:16.261 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.262 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:16.262 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.262 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:16.262 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993135+0000", 2020-01-16T00:13:16.262 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:16.263 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.263 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.263 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:16.263 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993135+0000", 2020-01-16T00:13:16.263 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:16.263 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.263 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.264 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:16.264 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993134+0000", 2020-01-16T00:13:16.264 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5630000000000002e-06 2020-01-16T00:13:16.264 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.264 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.264 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:16.264 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993139+0000", 2020-01-16T00:13:16.265 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1730000000000003e-06 2020-01-16T00:13:16.265 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.265 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.265 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:16.265 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993144+0000", 2020-01-16T00:13:16.265 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.937e-06 2020-01-16T00:13:16.265 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.265 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.266 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:16.266 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993147+0000", 2020-01-16T00:13:16.266 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0065164000000000003 2020-01-16T00:13:16.266 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.266 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.266 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.267 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999663+0000", 2020-01-16T00:13:16.267 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6215000000000002e-05 2020-01-16T00:13:16.267 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.267 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.267 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:16.267 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999729+0000", 2020-01-16T00:13:16.267 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010475887999999999 2020-01-16T00:13:16.268 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.268 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.268 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.268 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010205+0000", 2020-01-16T00:13:16.268 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119471 2020-01-16T00:13:16.268 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.268 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.268 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:16.269 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010325+0000", 2020-01-16T00:13:16.269 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00075498799999999995 2020-01-16T00:13:16.269 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.269 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.269 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.269 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.011080+0000", 2020-01-16T00:13:16.269 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7904e-05 2020-01-16T00:13:16.269 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.270 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.270 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:16.270 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.011138+0000", 2020-01-16T00:13:16.270 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5273999999999999e-05 2020-01-16T00:13:16.270 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.270 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.270 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:16.271 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.011203+0000", 2020-01-16T00:13:16.271 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018067916999999999 2020-01-16T00:13:16.271 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.271 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:16.271 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.271 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.272 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.272 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2314 3.7 3:e8af1ebf:::smithi12313934-29:head [setxattr _header (56) in=7b,truncate 2319399] snapc 0=[] ondisk+write+known_if_redirected e36)", 2020-01-16T00:13:16.272 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.317802+0000", 2020-01-16T00:13:16.272 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 8.8023731250000008, 2020-01-16T00:13:16.272 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015056285000000001, 2020-01-16T00:13:16.272 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:16.272 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:16.272 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:16.272 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:16.273 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:16.273 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2314 2020-01-16T00:13:16.273 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.273 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:16.273 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.273 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:16.273 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.317802+0000", 2020-01-16T00:13:16.273 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:16.274 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.274 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.274 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:16.274 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.317802+0000", 2020-01-16T00:13:16.274 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:16.274 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.274 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.275 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:16.275 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.317801+0000", 2020-01-16T00:13:16.275 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5580000000000003e-06 2020-01-16T00:13:16.275 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.275 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.275 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:16.275 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.317809+0000", 2020-01-16T00:13:16.275 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5660000000000001e-06 2020-01-16T00:13:16.276 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.276 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.276 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:16.276 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.317816+0000", 2020-01-16T00:13:16.276 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4530000000000004e-06 2020-01-16T00:13:16.276 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.276 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.276 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:16.277 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.317821+0000", 2020-01-16T00:13:16.277 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001622425 2020-01-16T00:13:16.277 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.277 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.277 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.277 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.319443+0000", 2020-01-16T00:13:16.277 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00063579200000000004 2020-01-16T00:13:16.278 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.278 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.278 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:16.278 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.320079+0000", 2020-01-16T00:13:16.278 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010884268000000001 2020-01-16T00:13:16.278 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.278 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.279 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.279 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.330963+0000", 2020-01-16T00:13:16.279 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000114967 2020-01-16T00:13:16.279 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.279 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.279 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:16.279 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.331078+0000", 2020-01-16T00:13:16.279 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000125964 2020-01-16T00:13:16.280 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.280 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.280 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,7", 2020-01-16T00:13:16.280 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.331204+0000", 2020-01-16T00:13:16.280 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001111938 2020-01-16T00:13:16.280 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.280 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.280 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:16.281 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.332316+0000", 2020-01-16T00:13:16.281 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00032159300000000001 2020-01-16T00:13:16.281 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.281 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.281 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.281 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.332638+0000", 2020-01-16T00:13:16.281 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000149794 2020-01-16T00:13:16.282 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.282 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.282 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.282 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.332787+0000", 2020-01-16T00:13:16.282 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4612e-05 2020-01-16T00:13:16.282 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.282 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.282 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:16.283 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.332822+0000", 2020-01-16T00:13:16.283 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6563999999999998e-05 2020-01-16T00:13:16.283 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.283 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.283 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:16.283 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.332859+0000", 2020-01-16T00:13:16.283 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015056285000000001 2020-01-16T00:13:16.283 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.284 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:16.284 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.284 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.284 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.284 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2315 3.7 3:e8af1ebf:::smithi12313934-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e36)", 2020-01-16T00:13:16.284 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.317826+0000", 2020-01-16T00:13:16.284 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 8.8023496550000004, 2020-01-16T00:13:16.285 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015205279, 2020-01-16T00:13:16.285 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:16.285 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:16.285 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:16.285 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:16.285 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:16.285 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2315 2020-01-16T00:13:16.285 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.286 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:16.286 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.286 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:16.286 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.317826+0000", 2020-01-16T00:13:16.286 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:16.286 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.286 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.287 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:16.287 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.317826+0000", 2020-01-16T00:13:16.287 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:16.287 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.287 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.287 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:16.287 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.317825+0000", 2020-01-16T00:13:16.287 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6370000000000001e-06 2020-01-16T00:13:16.288 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.288 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.288 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:16.288 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.317829+0000", 2020-01-16T00:13:16.288 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5399999999999998e-06 2020-01-16T00:13:16.288 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.288 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.288 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:16.289 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.317836+0000", 2020-01-16T00:13:16.289 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4079999999999998e-06 2020-01-16T00:13:16.289 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.289 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.289 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:16.289 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.317840+0000", 2020-01-16T00:13:16.289 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0025237850000000002 2020-01-16T00:13:16.290 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.290 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.290 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.290 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.320363+0000", 2020-01-16T00:13:16.290 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5877999999999995e-05 2020-01-16T00:13:16.290 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.290 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.291 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:16.291 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.320429+0000", 2020-01-16T00:13:16.291 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011518247000000001 2020-01-16T00:13:16.291 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.291 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.291 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.291 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.331947+0000", 2020-01-16T00:13:16.292 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012454799999999999 2020-01-16T00:13:16.292 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.292 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.292 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:16.292 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.332072+0000", 2020-01-16T00:13:16.292 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00081245299999999998 2020-01-16T00:13:16.292 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.292 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.293 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.293 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.332884+0000", 2020-01-16T00:13:16.293 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4336000000000001e-05 2020-01-16T00:13:16.293 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.293 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.293 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:16.293 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.332959+0000", 2020-01-16T00:13:16.293 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2204000000000002e-05 2020-01-16T00:13:16.294 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.294 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.294 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:16.294 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.333031+0000", 2020-01-16T00:13:16.294 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015205279 2020-01-16T00:13:16.294 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.294 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:16.295 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.295 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.295 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.295 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2324 3.7 3:e1be6d3b:::smithi12313934-24:head [setxattr _header (56) in=7b,truncate 2336206] snapc 0=[] ondisk+write+known_if_redirected e36)", 2020-01-16T00:13:16.295 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.550846+0000", 2020-01-16T00:13:16.295 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 8.5693296750000005, 2020-01-16T00:13:16.295 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018364742, 2020-01-16T00:13:16.295 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:16.296 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:16.296 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:16.296 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:16.296 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:16.296 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2324 2020-01-16T00:13:16.296 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.296 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:16.296 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.297 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:16.297 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550846+0000", 2020-01-16T00:13:16.297 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:16.297 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.297 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.297 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:16.297 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550846+0000", 2020-01-16T00:13:16.298 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:16.298 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.298 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.298 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:16.298 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550845+0000", 2020-01-16T00:13:16.298 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1829999999999999e-06 2020-01-16T00:13:16.298 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.298 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.299 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:16.299 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550851+0000", 2020-01-16T00:13:16.299 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1925e-05 2020-01-16T00:13:16.299 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.299 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.299 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:16.299 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550863+0000", 2020-01-16T00:13:16.300 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1139999999999997e-06 2020-01-16T00:13:16.300 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.300 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.300 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:16.300 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550868+0000", 2020-01-16T00:13:16.300 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001388423 2020-01-16T00:13:16.300 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.300 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.301 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.301 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.552256+0000", 2020-01-16T00:13:16.301 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0034246540000000001 2020-01-16T00:13:16.301 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.301 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.301 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:16.301 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555681+0000", 2020-01-16T00:13:16.302 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011882762999999999 2020-01-16T00:13:16.302 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.302 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.302 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.302 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.567564+0000", 2020-01-16T00:13:16.302 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015010800000000001 2020-01-16T00:13:16.302 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.303 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.303 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:16.303 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.567714+0000", 2020-01-16T00:13:16.303 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011527999999999999 2020-01-16T00:13:16.303 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.303 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.303 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,7", 2020-01-16T00:13:16.304 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.567829+0000", 2020-01-16T00:13:16.304 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00039888499999999997 2020-01-16T00:13:16.304 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.304 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.304 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:16.304 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568228+0000", 2020-01-16T00:13:16.304 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00054254599999999998 2020-01-16T00:13:16.304 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.305 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.305 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.305 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568771+0000", 2020-01-16T00:13:16.305 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035206400000000002 2020-01-16T00:13:16.305 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.305 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.305 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.306 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569123+0000", 2020-01-16T00:13:16.306 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1262e-05 2020-01-16T00:13:16.306 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.306 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.306 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:16.306 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569154+0000", 2020-01-16T00:13:16.306 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6499000000000002e-05 2020-01-16T00:13:16.307 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.307 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.307 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:16.307 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569210+0000", 2020-01-16T00:13:16.307 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018364742 2020-01-16T00:13:16.307 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.307 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:16.307 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.308 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.308 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.308 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2325 3.7 3:e1be6d3b:::smithi12313934-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e36)", 2020-01-16T00:13:16.308 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.550877+0000", 2020-01-16T00:13:16.308 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 8.5692986090000005, 2020-01-16T00:13:16.308 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018565154, 2020-01-16T00:13:16.308 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:16.308 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:16.309 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:16.309 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:16.309 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:16.309 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2325 2020-01-16T00:13:16.309 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.309 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:16.309 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.309 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:16.310 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550877+0000", 2020-01-16T00:13:16.310 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:16.310 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.310 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.310 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:16.310 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550877+0000", 2020-01-16T00:13:16.310 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:16.311 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.311 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.311 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:16.311 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550876+0000", 2020-01-16T00:13:16.311 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1080000000000003e-06 2020-01-16T00:13:16.311 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.311 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.311 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:16.312 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550880+0000", 2020-01-16T00:13:16.312 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3269999999999997e-06 2020-01-16T00:13:16.312 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.312 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.312 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:16.312 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550886+0000", 2020-01-16T00:13:16.312 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.386e-06 2020-01-16T00:13:16.313 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.313 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.313 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:16.313 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550891+0000", 2020-01-16T00:13:16.313 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048231410000000004 2020-01-16T00:13:16.313 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.313 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.313 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.314 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555714+0000", 2020-01-16T00:13:16.314 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0071999999999999e-05 2020-01-16T00:13:16.314 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.314 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.314 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:16.314 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555744+0000", 2020-01-16T00:13:16.314 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012329005000000001 2020-01-16T00:13:16.315 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.315 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.315 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.315 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568073+0000", 2020-01-16T00:13:16.315 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9087999999999997e-05 2020-01-16T00:13:16.315 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.315 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.315 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:16.316 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568172+0000", 2020-01-16T00:13:16.316 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010718220000000001 2020-01-16T00:13:16.316 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.316 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.316 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.316 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569244+0000", 2020-01-16T00:13:16.316 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8199e-05 2020-01-16T00:13:16.316 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.317 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.317 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:16.317 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569342+0000", 2020-01-16T00:13:16.317 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9708000000000001e-05 2020-01-16T00:13:16.318 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.318 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.318 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:16.319 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569442+0000", 2020-01-16T00:13:16.319 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018565154 2020-01-16T00:13:16.319 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.319 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:16.319 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.319 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.319 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.320 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2559 3.2 3:4ba19497:::smithi12313934-38:head [write 2950640~57846] snapc 0=[] ondisk+write+known_if_redirected e37)", 2020-01-16T00:13:16.320 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.591485+0000", 2020-01-16T00:13:16.320 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 3.5286904300000002, 2020-01-16T00:13:16.320 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016668697999999999, 2020-01-16T00:13:16.320 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:16.320 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:16.320 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:16.321 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:16.321 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:16.321 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2559 2020-01-16T00:13:16.321 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.321 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:16.321 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.322 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:16.322 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591485+0000", 2020-01-16T00:13:16.322 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:16.322 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.322 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.322 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:16.322 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591485+0000", 2020-01-16T00:13:16.323 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:16.323 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.323 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.323 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:16.323 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591483+0000", 2020-01-16T00:13:16.323 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4780999999999998e-05 2020-01-16T00:13:16.323 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.324 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.324 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:16.324 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591518+0000", 2020-01-16T00:13:16.324 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9706000000000001e-05 2020-01-16T00:13:16.324 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.324 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.325 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:16.325 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591537+0000", 2020-01-16T00:13:16.325 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1940000000000001e-06 2020-01-16T00:13:16.325 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.325 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.326 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:16.326 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591545+0000", 2020-01-16T00:13:16.326 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012233400000000001 2020-01-16T00:13:16.326 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.327 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.327 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.327 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591667+0000", 2020-01-16T00:13:16.327 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0050268719999999999 2020-01-16T00:13:16.327 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.327 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.328 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:16.328 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.596694+0000", 2020-01-16T00:13:16.328 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0090233669999999992 2020-01-16T00:13:16.328 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.328 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.328 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.329 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.605717+0000", 2020-01-16T00:13:16.329 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011849399999999999 2020-01-16T00:13:16.329 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.330 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.330 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:16.330 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.605836+0000", 2020-01-16T00:13:16.330 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022822200000000001 2020-01-16T00:13:16.330 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.330 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.331 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:16.331 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.606064+0000", 2020-01-16T00:13:16.331 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001265555 2020-01-16T00:13:16.331 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.331 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.331 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:16.331 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.607329+0000", 2020-01-16T00:13:16.332 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00052231100000000002 2020-01-16T00:13:16.332 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.332 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.332 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.332 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.607852+0000", 2020-01-16T00:13:16.332 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018900500000000001 2020-01-16T00:13:16.332 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.332 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.333 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.333 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.608041+0000", 2020-01-16T00:13:16.333 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7380000000000003e-05 2020-01-16T00:13:16.333 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.333 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.333 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:16.333 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.608098+0000", 2020-01-16T00:13:16.334 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5467999999999998e-05 2020-01-16T00:13:16.334 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.334 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.334 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:16.334 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.608154+0000", 2020-01-16T00:13:16.334 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016668697999999999 2020-01-16T00:13:16.334 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.334 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:16.335 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.335 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.335 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.335 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2560 3.2 3:4ba19497:::smithi12313934-38:head [setxattr _header (56) in=7b,truncate 3008486] snapc 0=[] ondisk+write+known_if_redirected e37)", 2020-01-16T00:13:16.335 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.591556+0000", 2020-01-16T00:13:16.335 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 3.5286189659999998, 2020-01-16T00:13:16.335 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016869716999999999, 2020-01-16T00:13:16.335 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:16.336 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:16.336 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:16.336 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:16.336 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:16.336 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2560 2020-01-16T00:13:16.336 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.336 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:16.337 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.337 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:16.337 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591556+0000", 2020-01-16T00:13:16.337 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:16.337 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.337 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.338 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:16.338 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591556+0000", 2020-01-16T00:13:16.338 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:16.338 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.338 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.339 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:16.339 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591555+0000", 2020-01-16T00:13:16.339 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.93e-06 2020-01-16T00:13:16.339 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.340 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.340 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:16.340 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591561+0000", 2020-01-16T00:13:16.340 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2089999999999999e-06 2020-01-16T00:13:16.341 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.341 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.341 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:16.341 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591569+0000", 2020-01-16T00:13:16.341 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5530000000000003e-06 2020-01-16T00:13:16.342 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.342 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.342 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:16.342 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591573+0000", 2020-01-16T00:13:16.343 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0053636630000000003 2020-01-16T00:13:16.343 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.343 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.343 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.343 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.596937+0000", 2020-01-16T00:13:16.344 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8693000000000001e-05 2020-01-16T00:13:16.344 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.344 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.344 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:16.344 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.597006+0000", 2020-01-16T00:13:16.344 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095244160000000008 2020-01-16T00:13:16.345 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.345 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.345 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.345 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.606530+0000", 2020-01-16T00:13:16.345 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012407100000000001 2020-01-16T00:13:16.345 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.346 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.346 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:16.346 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.606654+0000", 2020-01-16T00:13:16.346 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000189285 2020-01-16T00:13:16.346 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.346 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.346 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:16.347 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.606843+0000", 2020-01-16T00:13:16.347 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013331490000000001 2020-01-16T00:13:16.347 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.347 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.347 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:16.347 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.608176+0000", 2020-01-16T00:13:16.347 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7118000000000003e-05 2020-01-16T00:13:16.347 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.348 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.348 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.348 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.608254+0000", 2020-01-16T00:13:16.348 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3964e-05 2020-01-16T00:13:16.348 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.348 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.348 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.348 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.608308+0000", 2020-01-16T00:13:16.349 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7191999999999997e-05 2020-01-16T00:13:16.349 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.349 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.349 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:16.349 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.608365+0000", 2020-01-16T00:13:16.349 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1375000000000001e-05 2020-01-16T00:13:16.349 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.350 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.350 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:16.350 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.608426+0000", 2020-01-16T00:13:16.350 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016869716999999999 2020-01-16T00:13:16.350 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.350 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:16.351 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.351 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.351 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.351 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2561 3.2 3:4ba19497:::smithi12313934-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e37)", 2020-01-16T00:13:16.351 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.591578+0000", 2020-01-16T00:13:16.351 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 3.528597215, 2020-01-16T00:13:16.351 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017082362, 2020-01-16T00:13:16.351 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:16.352 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:16.352 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:16.352 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:16.352 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:16.352 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2561 2020-01-16T00:13:16.353 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.353 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:16.353 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.353 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:16.353 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591578+0000", 2020-01-16T00:13:16.354 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:16.354 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.354 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.354 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:16.354 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591578+0000", 2020-01-16T00:13:16.354 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:16.354 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.355 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.355 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:16.355 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591578+0000", 2020-01-16T00:13:16.355 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.428e-06 2020-01-16T00:13:16.355 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.355 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.355 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:16.356 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591581+0000", 2020-01-16T00:13:16.356 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.818e-06 2020-01-16T00:13:16.356 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.356 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.356 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:16.356 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591587+0000", 2020-01-16T00:13:16.356 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9890000000000003e-06 2020-01-16T00:13:16.357 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.357 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.357 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:16.357 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591591+0000", 2020-01-16T00:13:16.357 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.005441909 2020-01-16T00:13:16.357 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.358 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.358 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.359 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.597033+0000", 2020-01-16T00:13:16.359 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1651999999999999e-05 2020-01-16T00:13:16.359 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.359 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.359 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:16.359 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.597084+0000", 2020-01-16T00:13:16.359 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010094819999999999 2020-01-16T00:13:16.359 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.360 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.360 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.360 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.607179+0000", 2020-01-16T00:13:16.360 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012867799999999999 2020-01-16T00:13:16.360 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.360 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.360 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:16.360 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.607308+0000", 2020-01-16T00:13:16.361 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001148975 2020-01-16T00:13:16.361 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.361 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.361 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.361 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.608457+0000", 2020-01-16T00:13:16.361 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000101877 2020-01-16T00:13:16.361 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.362 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.362 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:16.362 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.608559+0000", 2020-01-16T00:13:16.362 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000101845 2020-01-16T00:13:16.362 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.362 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.362 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:16.362 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.608660+0000", 2020-01-16T00:13:16.363 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017082362 2020-01-16T00:13:16.363 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.363 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:16.363 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.363 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.363 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.363 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2578 3.2 3:4f8fe446:::smithi12313934-48:head [write 2943979~131735] snapc 0=[] ondisk+write+known_if_redirected e37)", 2020-01-16T00:13:16.364 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.664740+0000", 2020-01-16T00:13:16.364 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 3.455435687, 2020-01-16T00:13:16.364 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016610650000000001, 2020-01-16T00:13:16.364 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:16.364 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:16.364 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:16.364 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:16.364 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:16.365 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2578 2020-01-16T00:13:16.365 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.365 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:16.365 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.365 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:16.365 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664740+0000", 2020-01-16T00:13:16.365 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:16.365 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.366 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.366 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:16.366 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664740+0000", 2020-01-16T00:13:16.366 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:16.367 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.367 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.367 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:16.368 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664738+0000", 2020-01-16T00:13:16.368 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2001999999999994e-05 2020-01-16T00:13:16.368 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.368 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.368 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:16.368 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664800+0000", 2020-01-16T00:13:16.368 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3665999999999999e-05 2020-01-16T00:13:16.369 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.369 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.369 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:16.369 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664813+0000", 2020-01-16T00:13:16.369 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3029999999999996e-06 2020-01-16T00:13:16.369 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.369 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.369 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:16.370 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664820+0000", 2020-01-16T00:13:16.370 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2528999999999996e-05 2020-01-16T00:13:16.370 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.370 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.370 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.370 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664892+0000", 2020-01-16T00:13:16.370 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0014021140000000001 2020-01-16T00:13:16.371 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.371 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.371 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:16.371 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.666294+0000", 2020-01-16T00:13:16.371 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.01307417 2020-01-16T00:13:16.371 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.371 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.372 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.372 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.679368+0000", 2020-01-16T00:13:16.372 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012269099999999999 2020-01-16T00:13:16.372 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.372 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.372 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:16.373 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.679491+0000", 2020-01-16T00:13:16.373 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021723399999999999 2020-01-16T00:13:16.373 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.373 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.373 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:16.373 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.679708+0000", 2020-01-16T00:13:16.373 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001478507 2020-01-16T00:13:16.373 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.374 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.374 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:16.374 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.681187+0000", 2020-01-16T00:13:16.374 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3408000000000002e-05 2020-01-16T00:13:16.374 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.374 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.374 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.375 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.681230+0000", 2020-01-16T00:13:16.375 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0519999999999999e-05 2020-01-16T00:13:16.375 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.375 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.375 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.375 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.681261+0000", 2020-01-16T00:13:16.375 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1486000000000003e-05 2020-01-16T00:13:16.375 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.376 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.376 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:16.376 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.681292+0000", 2020-01-16T00:13:16.376 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8013999999999997e-05 2020-01-16T00:13:16.376 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.376 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.376 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:16.377 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.681350+0000", 2020-01-16T00:13:16.377 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016610650000000001 2020-01-16T00:13:16.377 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.377 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:16.377 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.377 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.377 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.377 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2579 3.2 3:4f8fe446:::smithi12313934-48:head [setxattr _header (56) in=7b,truncate 3075714] snapc 0=[] ondisk+write+known_if_redirected e37)", 2020-01-16T00:13:16.378 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.664831+0000", 2020-01-16T00:13:16.378 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 3.4553447990000001, 2020-01-16T00:13:16.378 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016678353999999999, 2020-01-16T00:13:16.378 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:16.378 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:16.378 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:16.378 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:16.379 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:16.379 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2579 2020-01-16T00:13:16.380 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.380 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:16.380 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.380 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:16.380 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664831+0000", 2020-01-16T00:13:16.381 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:16.381 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.381 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.382 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:16.382 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664831+0000", 2020-01-16T00:13:16.382 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:16.382 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.383 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.383 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:16.383 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664830+0000", 2020-01-16T00:13:16.383 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7089999999999999e-06 2020-01-16T00:13:16.384 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.384 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.384 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:16.384 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664835+0000", 2020-01-16T00:13:16.384 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6526e-05 2020-01-16T00:13:16.385 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.385 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.385 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:16.385 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664862+0000", 2020-01-16T00:13:16.386 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1869999999999996e-06 2020-01-16T00:13:16.386 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.386 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.386 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:16.387 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664867+0000", 2020-01-16T00:13:16.387 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00148733 2020-01-16T00:13:16.387 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.387 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.388 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.388 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.666354+0000", 2020-01-16T00:13:16.388 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3683999999999995e-05 2020-01-16T00:13:16.388 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.388 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.389 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:16.389 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.666428+0000", 2020-01-16T00:13:16.389 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013904786000000001 2020-01-16T00:13:16.389 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.390 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.390 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.390 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.680333+0000", 2020-01-16T00:13:16.390 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012004 2020-01-16T00:13:16.391 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.391 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.391 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:16.391 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.680453+0000", 2020-01-16T00:13:16.391 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021047700000000001 2020-01-16T00:13:16.392 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.392 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.392 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:16.392 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.680663+0000", 2020-01-16T00:13:16.393 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00070448799999999997 2020-01-16T00:13:16.393 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.393 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.393 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:16.394 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.681368+0000", 2020-01-16T00:13:16.394 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8998999999999997e-05 2020-01-16T00:13:16.394 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.394 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.395 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.395 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.681407+0000", 2020-01-16T00:13:16.395 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9014999999999999e-05 2020-01-16T00:13:16.395 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.395 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.396 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.396 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.681436+0000", 2020-01-16T00:13:16.396 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9249999999999999e-05 2020-01-16T00:13:16.396 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.397 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.397 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:16.397 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.681465+0000", 2020-01-16T00:13:16.397 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3696999999999999e-05 2020-01-16T00:13:16.398 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.398 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.398 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:16.398 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.681509+0000", 2020-01-16T00:13:16.399 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016678353999999999 2020-01-16T00:13:16.399 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.399 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:16.399 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.400 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.400 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.400 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2580 3.2 3:4f8fe446:::smithi12313934-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e37)", 2020-01-16T00:13:16.400 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.664881+0000", 2020-01-16T00:13:16.400 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 3.4552938540000002, 2020-01-16T00:13:16.401 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016925910999999998, 2020-01-16T00:13:16.401 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:16.401 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:16.401 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:16.402 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:16.402 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:16.402 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2580 2020-01-16T00:13:16.402 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.403 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:16.403 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.403 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:16.403 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664881+0000", 2020-01-16T00:13:16.403 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:16.404 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.404 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.404 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:16.404 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664881+0000", 2020-01-16T00:13:16.405 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:16.405 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.405 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.405 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:16.406 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664881+0000", 2020-01-16T00:13:16.406 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1640000000000003e-06 2020-01-16T00:13:16.406 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.406 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.406 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:16.406 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664885+0000", 2020-01-16T00:13:16.407 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6469999999999998e-06 2020-01-16T00:13:16.407 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.407 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.407 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:16.407 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664892+0000", 2020-01-16T00:13:16.407 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8829999999999997e-06 2020-01-16T00:13:16.407 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.407 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.408 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:16.408 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664897+0000", 2020-01-16T00:13:16.408 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0015624059999999999 2020-01-16T00:13:16.408 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.408 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.408 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.408 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.666460+0000", 2020-01-16T00:13:16.409 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9839000000000002e-05 2020-01-16T00:13:16.409 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.409 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.409 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:16.409 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.666510+0000", 2020-01-16T00:13:16.409 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014475200000000001 2020-01-16T00:13:16.409 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.409 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.410 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.410 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.680985+0000", 2020-01-16T00:13:16.410 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8396999999999997e-05 2020-01-16T00:13:16.410 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.410 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.410 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:16.410 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.681043+0000", 2020-01-16T00:13:16.411 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00051206799999999996 2020-01-16T00:13:16.411 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.411 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.411 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.411 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.681555+0000", 2020-01-16T00:13:16.411 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118881 2020-01-16T00:13:16.411 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.411 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.412 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:16.412 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.681674+0000", 2020-01-16T00:13:16.412 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000133242 2020-01-16T00:13:16.412 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.412 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.412 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:16.412 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.681807+0000", 2020-01-16T00:13:16.413 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016925910999999998 2020-01-16T00:13:16.413 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.413 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:16.413 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.413 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.413 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:16.413 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:16.415 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op export --pgid 1.1 --file /var/log/ceph/exp.1.1.5 2020-01-16T00:13:16.416 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-01-16T00:13:16.416 INFO:tasks.rados.rados.0.smithi123.stdout:1134: finishing write tid 1 to smithi12313934-12 2020-01-16T00:13:16.416 INFO:tasks.rados.rados.0.smithi123.stdout:1134: finishing write tid 2 to smithi12313934-12 2020-01-16T00:13:16.417 INFO:tasks.rados.rados.0.smithi123.stdout:1134: finishing write tid 3 to smithi12313934-12 2020-01-16T00:13:16.417 INFO:tasks.rados.rados.0.smithi123.stdout:1134: finishing write tid 5 to smithi12313934-12 2020-01-16T00:13:16.417 INFO:tasks.rados.rados.0.smithi123.stdout:1134: finishing write tid 6 to smithi12313934-12 2020-01-16T00:13:16.417 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 12 v 93 (ObjNum 320 snap 0 seq_num 320) dirty exists 2020-01-16T00:13:16.417 INFO:tasks.rados.rados.0.smithi123.stdout:1134: left oid 12 (ObjNum 320 snap 0 seq_num 320) 2020-01-16T00:13:16.417 INFO:tasks.rados.rados.0.smithi123.stdout:1135: finishing write tid 1 to smithi12313934-29 2020-01-16T00:13:16.417 INFO:tasks.rados.rados.0.smithi123.stdout:1135: finishing write tid 2 to smithi12313934-29 2020-01-16T00:13:16.418 INFO:tasks.rados.rados.0.smithi123.stdout:1135: finishing write tid 3 to smithi12313934-29 2020-01-16T00:13:16.418 INFO:tasks.rados.rados.0.smithi123.stdout:1130: expect (ObjNum 250 snap 0 seq_num 250) 2020-01-16T00:13:16.418 INFO:tasks.rados.rados.0.smithi123.stdout:1132: finishing write tid 1 to smithi12313934-50 2020-01-16T00:13:16.418 INFO:tasks.rados.rados.0.smithi123.stdout:1135: finishing write tid 5 to smithi12313934-29 2020-01-16T00:13:16.418 INFO:tasks.rados.rados.0.smithi123.stdout:1135: finishing write tid 6 to smithi12313934-29 2020-01-16T00:13:16.418 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 29 v 116 (ObjNum 321 snap 0 seq_num 321) dirty exists 2020-01-16T00:13:16.418 INFO:tasks.rados.rados.0.smithi123.stdout:1135: left oid 29 (ObjNum 321 snap 0 seq_num 321) 2020-01-16T00:13:16.419 INFO:tasks.rados.rados.0.smithi123.stdout:1132: finishing write tid 2 to smithi12313934-50 2020-01-16T00:13:16.419 INFO:tasks.rados.rados.0.smithi123.stdout:1132: finishing write tid 3 to smithi12313934-50 2020-01-16T00:13:16.419 INFO:tasks.rados.rados.0.smithi123.stdout:1132: finishing write tid 5 to smithi12313934-50 2020-01-16T00:13:16.419 INFO:tasks.rados.rados.0.smithi123.stdout:1132: finishing write tid 6 to smithi12313934-50 2020-01-16T00:13:16.419 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 50 v 124 (ObjNum 319 snap 0 seq_num 319) dirty exists 2020-01-16T00:13:16.419 INFO:tasks.rados.rados.0.smithi123.stdout:1132: left oid 50 (ObjNum 319 snap 0 seq_num 319) 2020-01-16T00:13:16.419 INFO:tasks.rados.rados.0.smithi123.stdout:1131: expect (ObjNum 308 snap 0 seq_num 308) 2020-01-16T00:13:16.419 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:16.410+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:16.496 INFO:tasks.rados.rados.0.smithi123.stdout:1136: finishing write tid 1 to smithi12313934-20 2020-01-16T00:13:16.496 INFO:tasks.rados.rados.0.smithi123.stdout:1136: finishing write tid 2 to smithi12313934-20 2020-01-16T00:13:16.497 INFO:tasks.rados.rados.0.smithi123.stdout:1136: finishing write tid 3 to smithi12313934-20 2020-01-16T00:13:16.497 INFO:tasks.rados.rados.0.smithi123.stdout:1136: finishing write tid 5 to smithi12313934-20 2020-01-16T00:13:16.497 INFO:tasks.rados.rados.0.smithi123.stdout:1136: finishing write tid 6 to smithi12313934-20 2020-01-16T00:13:16.497 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 20 v 167 (ObjNum 322 snap 0 seq_num 322) dirty exists 2020-01-16T00:13:16.497 INFO:tasks.rados.rados.0.smithi123.stdout:1136: left oid 20 (ObjNum 322 snap 0 seq_num 322) 2020-01-16T00:13:16.497 INFO:tasks.rados.rados.0.smithi123.stdout:1130: done (7 left) 2020-01-16T00:13:16.497 INFO:tasks.rados.rados.0.smithi123.stdout:1131: done (6 left) 2020-01-16T00:13:16.498 INFO:tasks.rados.rados.0.smithi123.stdout:1132: done (5 left) 2020-01-16T00:13:16.498 INFO:tasks.rados.rados.0.smithi123.stdout:1133: done (4 left) 2020-01-16T00:13:16.498 INFO:tasks.rados.rados.0.smithi123.stdout:1134: done (3 left) 2020-01-16T00:13:16.498 INFO:tasks.rados.rados.0.smithi123.stdout:1135: done (2 left) 2020-01-16T00:13:16.498 INFO:tasks.rados.rados.0.smithi123.stdout:1136: done (1 left) 2020-01-16T00:13:16.498 INFO:tasks.rados.rados.0.smithi123.stdout:1137: done (0 left) 2020-01-16T00:13:16.498 INFO:tasks.rados.rados.0.smithi123.stdout:1138: write (excl) oid 17 current snap is 0 2020-01-16T00:13:16.499 INFO:tasks.rados.rados.0.smithi123.stdout:1138: seq_num 324 ranges {645468=467502,1726754=565436,2798226=596838} 2020-01-16T00:13:16.508 INFO:tasks.rados.rados.0.smithi123.stdout:1138: writing smithi12313934-17 from 645468 to 1112970 tid 1 2020-01-16T00:13:16.515 INFO:tasks.rados.rados.0.smithi123.stdout:1138: writing smithi12313934-17 from 1726754 to 2292190 tid 2 2020-01-16T00:13:16.516 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:16.510+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:16.522 INFO:tasks.rados.rados.0.smithi123.stdout:1138: writing smithi12313934-17 from 2798226 to 3395064 tid 3 2020-01-16T00:13:16.522 INFO:tasks.rados.rados.0.smithi123.stdout:1139: write oid 35 current snap is 0 2020-01-16T00:13:16.522 INFO:tasks.rados.rados.0.smithi123.stdout:1139: seq_num 325 ranges {662275=542751,1785236=425447,2942002=520273} 2020-01-16T00:13:16.528 INFO:tasks.rados.rados.0.smithi123.stdout:1139: writing smithi12313934-35 from 662275 to 1205026 tid 1 2020-01-16T00:13:16.534 INFO:tasks.rados.rados.0.smithi123.stdout:1139: writing smithi12313934-35 from 1785236 to 2210683 tid 2 2020-01-16T00:13:16.541 INFO:tasks.rados.rados.0.smithi123.stdout:1139: writing smithi12313934-35 from 2942002 to 3462275 tid 3 2020-01-16T00:13:16.542 INFO:tasks.rados.rados.0.smithi123.stdout:1140: read oid 23 snap -1 2020-01-16T00:13:16.542 INFO:tasks.rados.rados.0.smithi123.stdout:1140: expect deleted 2020-01-16T00:13:16.542 INFO:tasks.rados.rados.0.smithi123.stdout:1141: read oid 26 snap -1 2020-01-16T00:13:16.542 INFO:tasks.rados.rados.0.smithi123.stdout:1141: expect (ObjNum 263 snap 0 seq_num 263) 2020-01-16T00:13:16.542 INFO:tasks.rados.rados.0.smithi123.stdout:1142: read oid 16 snap -1 2020-01-16T00:13:16.542 INFO:tasks.rados.rados.0.smithi123.stdout:1142: expect (ObjNum 308 snap 0 seq_num 308) 2020-01-16T00:13:16.542 INFO:tasks.rados.rados.0.smithi123.stdout:1143: write (excl) oid 1 current snap is 0 2020-01-16T00:13:16.542 INFO:tasks.rados.rados.0.smithi123.stdout:1143: seq_num 326 ranges {679082=618000,1843718=569105,2853072=626010} 2020-01-16T00:13:16.548 INFO:tasks.rados.rados.0.smithi123.stdout:1143: writing smithi12313934-1 from 679082 to 1297082 tid 1 2020-01-16T00:13:16.555 INFO:tasks.rados.rados.0.smithi123.stdout:1143: writing smithi12313934-1 from 1843718 to 2412823 tid 2 2020-01-16T00:13:16.561 INFO:tasks.rados.rados.0.smithi123.stdout:1143: writing smithi12313934-1 from 2853072 to 3479082 tid 3 2020-01-16T00:13:16.561 INFO:tasks.rados.rados.0.smithi123.stdout:1144: write (excl) oid 14 current snap is 0 2020-01-16T00:13:16.561 INFO:tasks.rados.rados.0.smithi123.stdout:1144: seq_num 327 ranges {695889=409602,1618553=712763,2996848=499041} 2020-01-16T00:13:16.567 INFO:tasks.rados.rados.0.smithi123.stdout:1144: writing smithi12313934-14 from 695889 to 1105491 tid 1 2020-01-16T00:13:16.571 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:16.571 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:16.571 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:13:16.571 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:16.574 INFO:tasks.rados.rados.0.smithi123.stdout:1144: writing smithi12313934-14 from 1618553 to 2331316 tid 2 2020-01-16T00:13:16.581 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-01-16T00:13:16.581 INFO:tasks.rados.rados.0.smithi123.stdout:1144: writing smithi12313934-14 from 2996848 to 3495889 tid 3 2020-01-16T00:13:16.581 INFO:tasks.rados.rados.0.smithi123.stdout:1145: read oid 3 snap -1 2020-01-16T00:13:16.581 INFO:tasks.rados.rados.0.smithi123.stdout:1145: expect (ObjNum 260 snap 0 seq_num 260) 2020-01-16T00:13:16.582 INFO:tasks.rados.rados.0.smithi123.stdout:1146: write (excl) oid 46 current snap is 0 2020-01-16T00:13:16.582 INFO:tasks.rados.rados.0.smithi123.stdout:1146: seq_num 328 ranges {712696=484851,1960682=572774,3307918=204778} 2020-01-16T00:13:16.587 INFO:tasks.rados.rados.0.smithi123.stdout:1146: writing smithi12313934-46 from 712696 to 1197547 tid 1 2020-01-16T00:13:16.595 INFO:tasks.rados.rados.0.smithi123.stdout:1146: writing smithi12313934-46 from 1960682 to 2533456 tid 2 2020-01-16T00:13:16.600 INFO:tasks.rados.rados.0.smithi123.stdout:1146: writing smithi12313934-46 from 3307918 to 3512696 tid 3 2020-01-16T00:13:16.601 INFO:tasks.rados.rados.0.smithi123.stdout:1147: write oid 11 current snap is 0 2020-01-16T00:13:16.601 INFO:tasks.rados.rados.0.smithi123.stdout:1147: seq_num 329 ranges {729503=560100,2019164=716432,3335341=194162} 2020-01-16T00:13:16.608 INFO:tasks.rados.rados.0.smithi123.stdout:1147: writing smithi12313934-11 from 729503 to 1289603 tid 1 2020-01-16T00:13:16.619 INFO:tasks.rados.rados.0.smithi123.stdout:1147: writing smithi12313934-11 from 2019164 to 2735596 tid 2 2020-01-16T00:13:16.620 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:16.616+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:16.621 INFO:tasks.rados.rados.0.smithi123.stdout:1147: writing smithi12313934-11 from 3335341 to 3529503 tid 3 2020-01-16T00:13:16.621 INFO:tasks.rados.rados.0.smithi123.stdout:1148: read oid 4 snap -1 2020-01-16T00:13:16.622 INFO:tasks.rados.rados.0.smithi123.stdout:1148: expect (ObjNum 188 snap 0 seq_num 188) 2020-01-16T00:13:16.622 INFO:tasks.rados.rados.0.smithi123.stdout:1149: read oid 15 snap -1 2020-01-16T00:13:16.622 INFO:tasks.rados.rados.0.smithi123.stdout:1149: expect (ObjNum 317 snap 0 seq_num 317) 2020-01-16T00:13:16.622 INFO:tasks.rados.rados.0.smithi123.stdout:1150: write (excl) oid 33 current snap is 0 2020-01-16T00:13:16.622 INFO:tasks.rados.rados.0.smithi123.stdout:1150: seq_num 330 ranges {746310=635349,2077646=576443,3362764=183546} 2020-01-16T00:13:16.629 INFO:tasks.rados.rados.0.smithi123.stdout:1150: writing smithi12313934-33 from 746310 to 1381659 tid 1 2020-01-16T00:13:16.640 INFO:tasks.rados.rados.0.smithi123.stdout:1150: writing smithi12313934-33 from 2077646 to 2654089 tid 2 2020-01-16T00:13:16.641 INFO:tasks.rados.rados.0.smithi123.stdout:1150: writing smithi12313934-33 from 3362764 to 3546310 tid 3 2020-01-16T00:13:16.642 INFO:tasks.rados.rados.0.smithi123.stdout:1151: read oid 6 snap -1 2020-01-16T00:13:16.645 INFO:tasks.rados.rados.0.smithi123.stdout:1151: expect (ObjNum 311 snap 0 seq_num 311) 2020-01-16T00:13:16.646 INFO:tasks.rados.rados.0.smithi123.stdout:1152: write oid 48 current snap is 0 2020-01-16T00:13:16.646 INFO:tasks.rados.rados.0.smithi123.stdout:1152: seq_num 331 ranges {763117=710598,2136128=720101,3390187=172930} 2020-01-16T00:13:16.650 INFO:tasks.rados.rados.0.smithi123.stdout:1152: writing smithi12313934-48 from 763117 to 1473715 tid 1 2020-01-16T00:13:16.658 INFO:tasks.rados.rados.0.smithi123.stdout:1152: writing smithi12313934-48 from 2136128 to 2856229 tid 2 2020-01-16T00:13:16.662 INFO:tasks.rados.rados.0.smithi123.stdout:1152: writing smithi12313934-48 from 3390187 to 3563117 tid 3 2020-01-16T00:13:16.663 INFO:tasks.rados.rados.0.smithi123.stdout:1138: finishing write tid 1 to smithi12313934-17 2020-01-16T00:13:16.663 INFO:tasks.rados.rados.0.smithi123.stdout:1153: read oid 24 snap -1 2020-01-16T00:13:16.663 INFO:tasks.rados.rados.0.smithi123.stdout:1153: expect (ObjNum 304 snap 0 seq_num 304) 2020-01-16T00:13:16.663 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:16.663 INFO:tasks.rados.rados.0.smithi123.stdout:1138: finishing write tid 2 to smithi12313934-17 2020-01-16T00:13:16.663 INFO:tasks.rados.rados.0.smithi123.stdout:1138: finishing write tid 3 to smithi12313934-17 2020-01-16T00:13:16.664 INFO:tasks.rados.rados.0.smithi123.stdout:1138: finishing write tid 5 to smithi12313934-17 2020-01-16T00:13:16.664 INFO:tasks.rados.rados.0.smithi123.stdout:1138: finishing write tid 6 to smithi12313934-17 2020-01-16T00:13:16.664 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 17 v 127 (ObjNum 324 snap 0 seq_num 324) dirty exists 2020-01-16T00:13:16.664 INFO:tasks.rados.rados.0.smithi123.stdout:1138: left oid 17 (ObjNum 324 snap 0 seq_num 324) 2020-01-16T00:13:16.664 INFO:tasks.rados.rados.0.smithi123.stdout:1139: finishing write tid 1 to smithi12313934-35 2020-01-16T00:13:16.664 INFO:tasks.rados.rados.0.smithi123.stdout:1138: done (15 left) 2020-01-16T00:13:16.664 INFO:tasks.rados.rados.0.smithi123.stdout:1154: write oid 25 current snap is 0 2020-01-16T00:13:16.665 INFO:tasks.rados.rados.0.smithi123.stdout:1154: seq_num 332 ranges {779924=785847,2078257=580112,3301257=278667} 2020-01-16T00:13:16.671 INFO:tasks.rados.rados.0.smithi123.stdout:1154: writing smithi12313934-25 from 779924 to 1565771 tid 1 2020-01-16T00:13:16.677 INFO:tasks.rados.rados.0.smithi123.stdout:1154: writing smithi12313934-25 from 2078257 to 2658369 tid 2 2020-01-16T00:13:16.683 INFO:tasks.rados.rados.0.smithi123.stdout:1154: writing smithi12313934-25 from 3301257 to 3579924 tid 3 2020-01-16T00:13:16.684 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:16.684 INFO:tasks.rados.rados.0.smithi123.stdout:1139: finishing write tid 2 to smithi12313934-35 2020-01-16T00:13:16.684 INFO:tasks.rados.rados.0.smithi123.stdout:1139: finishing write tid 3 to smithi12313934-35 2020-01-16T00:13:16.684 INFO:tasks.rados.rados.0.smithi123.stdout:1141: expect (ObjNum 263 snap 0 seq_num 263) 2020-01-16T00:13:16.687 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:16.687 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:16.688 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:13:16.688 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:13:16.688 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:16.696 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-01-16T00:13:16.744 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:16.742+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:16.823 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:16.818+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:16.866 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:16.866 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:13:16.866 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:13:16.867 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:13:16.867 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.867 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.4.0:114 3.9 e24/20 3:9fc687b2:::smithi12313934-44:head v 24'74, mlcod=24'72)", 2020-01-16T00:13:16.867 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:13:16.867 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 38.537882484999997, 2020-01-16T00:13:16.867 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010399429, 2020-01-16T00:13:16.867 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:16.867 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:16.868 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:16.868 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.868 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:16.868 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:13:16.869 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:16.869 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.869 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.869 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:16.869 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:13:16.869 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:16.869 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.869 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.870 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:16.870 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325276+0000", 2020-01-16T00:13:16.870 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0046127069999999997 2020-01-16T00:13:16.870 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.870 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.870 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:16.870 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329889+0000", 2020-01-16T00:13:16.871 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3519999999999998e-05 2020-01-16T00:13:16.871 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.871 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.871 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:16.871 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329912+0000", 2020-01-16T00:13:16.871 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4070000000000004e-06 2020-01-16T00:13:16.871 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.872 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.872 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:16.872 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329920+0000", 2020-01-16T00:13:16.872 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8817000000000003e-05 2020-01-16T00:13:16.872 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.872 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.872 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.873 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329978+0000", 2020-01-16T00:13:16.873 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9721e-05 2020-01-16T00:13:16.873 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.873 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.873 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:16.873 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329998+0000", 2020-01-16T00:13:16.873 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056328139999999999 2020-01-16T00:13:16.873 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.874 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.874 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:16.874 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.335631+0000", 2020-01-16T00:13:16.874 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7170999999999999e-05 2020-01-16T00:13:16.874 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.874 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.874 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:16.875 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.335678+0000", 2020-01-16T00:13:16.875 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010399429 2020-01-16T00:13:16.875 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.875 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:16.875 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.875 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.875 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.876 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:24 2.1 2:8b431f80:::smithi12313934-40 low_tier:head [write 790528~688128] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:13:16.876 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.482303+0000", 2020-01-16T00:13:16.876 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 38.380858388999997, 2020-01-16T00:13:16.876 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010071853, 2020-01-16T00:13:16.876 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:16.876 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:16.876 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:16.877 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.877 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:16.877 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.482303+0000", 2020-01-16T00:13:16.877 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:16.877 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.877 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.877 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:16.878 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.482303+0000", 2020-01-16T00:13:16.878 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:16.878 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.878 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.878 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:16.879 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.482301+0000", 2020-01-16T00:13:16.879 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031173960000000001 2020-01-16T00:13:16.879 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.879 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.879 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:16.879 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485418+0000", 2020-01-16T00:13:16.879 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9300000000000001e-05 2020-01-16T00:13:16.880 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.881 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.881 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:16.881 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485448+0000", 2020-01-16T00:13:16.881 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2611e-05 2020-01-16T00:13:16.881 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.881 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.881 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:16.882 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485460+0000", 2020-01-16T00:13:16.882 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112487 2020-01-16T00:13:16.882 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.882 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.882 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.882 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485573+0000", 2020-01-16T00:13:16.882 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022169299999999999 2020-01-16T00:13:16.883 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.883 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.883 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:16.883 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485794+0000", 2020-01-16T00:13:16.883 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00036287500000000002 2020-01-16T00:13:16.883 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.883 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.883 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:16.884 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.486157+0000", 2020-01-16T00:13:16.884 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040260310000000002 2020-01-16T00:13:16.884 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.884 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.884 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:16.884 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.490183+0000", 2020-01-16T00:13:16.884 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00081083000000000001 2020-01-16T00:13:16.885 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.885 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.885 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.885 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.490994+0000", 2020-01-16T00:13:16.885 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001287746 2020-01-16T00:13:16.885 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.885 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.886 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.886 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492282+0000", 2020-01-16T00:13:16.886 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8810999999999998e-05 2020-01-16T00:13:16.886 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.886 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.886 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:16.886 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492321+0000", 2020-01-16T00:13:16.887 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4095000000000003e-05 2020-01-16T00:13:16.887 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.887 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.887 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:16.887 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492375+0000", 2020-01-16T00:13:16.887 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010071853 2020-01-16T00:13:16.887 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.887 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:16.888 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.888 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.888 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.888 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:25 2.1 2:8b431f80:::smithi12313934-40 low_tier:head [write 1926656~67072] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:13:16.888 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.485481+0000", 2020-01-16T00:13:16.888 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 38.377679923999999, 2020-01-16T00:13:16.888 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0081599020000000001, 2020-01-16T00:13:16.889 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:16.890 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:16.890 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:16.890 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.890 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:16.890 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485481+0000", 2020-01-16T00:13:16.891 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:16.891 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.891 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.891 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:16.891 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485481+0000", 2020-01-16T00:13:16.891 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:16.891 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.892 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.892 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:16.892 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485479+0000", 2020-01-16T00:13:16.892 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012423800000000001 2020-01-16T00:13:16.892 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.892 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.892 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:16.892 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485603+0000", 2020-01-16T00:13:16.893 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8366999999999999e-05 2020-01-16T00:13:16.893 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.893 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.893 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:16.893 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485621+0000", 2020-01-16T00:13:16.893 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0207000000000001e-05 2020-01-16T00:13:16.893 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.893 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.894 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:16.894 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485632+0000", 2020-01-16T00:13:16.895 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002868532 2020-01-16T00:13:16.895 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.895 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.895 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.895 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.488500+0000", 2020-01-16T00:13:16.896 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011340000000000001 2020-01-16T00:13:16.896 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.896 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.896 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:16.896 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.488614+0000", 2020-01-16T00:13:16.896 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019056300000000001 2020-01-16T00:13:16.896 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.897 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.897 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:16.897 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.488804+0000", 2020-01-16T00:13:16.897 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00153302 2020-01-16T00:13:16.897 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.897 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.898 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:16.898 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.490337+0000", 2020-01-16T00:13:16.898 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00089946100000000005 2020-01-16T00:13:16.898 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.898 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.898 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.898 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.491237+0000", 2020-01-16T00:13:16.898 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0022862820000000002 2020-01-16T00:13:16.899 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.899 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.900 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.900 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.493523+0000", 2020-01-16T00:13:16.900 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5609999999999998e-05 2020-01-16T00:13:16.900 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.900 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.900 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:16.900 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.493578+0000", 2020-01-16T00:13:16.901 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2770999999999999e-05 2020-01-16T00:13:16.901 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.901 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.901 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:16.901 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.493641+0000", 2020-01-16T00:13:16.901 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0081599020000000001 2020-01-16T00:13:16.901 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.901 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:16.902 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.902 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.902 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.902 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:1279 3.0 e26/20 3:05b2bf53:::smithi12313934-37:head v 26'137, mlcod=26'135)", 2020-01-16T00:13:16.902 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:43.082160+0000", 2020-01-16T00:13:16.902 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 33.781001191000001, 2020-01-16T00:13:16.902 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0091277130000000008, 2020-01-16T00:13:16.903 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:16.903 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:16.903 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:16.903 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.903 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:16.903 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.082160+0000", 2020-01-16T00:13:16.903 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:16.904 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.904 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.904 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:16.904 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.082160+0000", 2020-01-16T00:13:16.904 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999957 2020-01-16T00:13:16.904 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.904 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.904 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:16.905 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.082156+0000", 2020-01-16T00:13:16.905 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00046771999999999997 2020-01-16T00:13:16.905 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.905 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.905 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:16.905 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.082623+0000", 2020-01-16T00:13:16.905 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3099999999999998e-05 2020-01-16T00:13:16.905 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.906 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.906 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:16.906 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.082656+0000", 2020-01-16T00:13:16.906 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2063e-05 2020-01-16T00:13:16.906 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.906 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.906 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:16.907 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.082668+0000", 2020-01-16T00:13:16.907 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021980900000000002 2020-01-16T00:13:16.907 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.907 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.907 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.907 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.084867+0000", 2020-01-16T00:13:16.907 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00033252299999999999 2020-01-16T00:13:16.908 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.908 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.908 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:16.908 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.085199+0000", 2020-01-16T00:13:16.908 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0060364019999999997 2020-01-16T00:13:16.908 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.908 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.908 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:16.909 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.091235+0000", 2020-01-16T00:13:16.909 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2255e-05 2020-01-16T00:13:16.909 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.909 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.910 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:16.910 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.091288+0000", 2020-01-16T00:13:16.910 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0091277130000000008 2020-01-16T00:13:16.910 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.910 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:16.910 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.910 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.910 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.911 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:47 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e28)", 2020-01-16T00:13:16.911 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:47.941839+0000", 2020-01-16T00:13:16.911 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 28.921322069999999, 2020-01-16T00:13:16.911 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.008400154, 2020-01-16T00:13:16.913 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:16.925 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:16.926 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:16.926 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.926 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:16.926 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.941839+0000", 2020-01-16T00:13:16.926 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:16.926 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.926 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.927 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:16.927 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.941839+0000", 2020-01-16T00:13:16.927 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:16.927 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.927 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.927 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:16.927 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.941837+0000", 2020-01-16T00:13:16.927 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00046009399999999998 2020-01-16T00:13:16.928 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.928 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.928 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:16.928 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942297+0000", 2020-01-16T00:13:16.928 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8628999999999999e-05 2020-01-16T00:13:16.928 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.928 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.928 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:16.929 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942335+0000", 2020-01-16T00:13:16.929 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1729999999999998e-06 2020-01-16T00:13:16.929 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.929 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.929 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:16.929 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942343+0000", 2020-01-16T00:13:16.932 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4845999999999997e-05 2020-01-16T00:13:16.932 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.932 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.932 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.933 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942387+0000", 2020-01-16T00:13:16.933 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6398999999999999e-05 2020-01-16T00:13:16.933 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.933 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.933 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:16.933 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942454+0000", 2020-01-16T00:13:16.933 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010786 2020-01-16T00:13:16.934 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.934 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.934 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:16.934 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942562+0000", 2020-01-16T00:13:16.934 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0064317760000000002 2020-01-16T00:13:16.934 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.934 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.934 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:16.935 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.948993+0000", 2020-01-16T00:13:16.935 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0011048849999999999 2020-01-16T00:13:16.935 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.935 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.935 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.935 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.950098+0000", 2020-01-16T00:13:16.935 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7064000000000003e-05 2020-01-16T00:13:16.935 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.936 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.936 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.936 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.950135+0000", 2020-01-16T00:13:16.936 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6589000000000003e-05 2020-01-16T00:13:16.936 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.936 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.936 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:16.936 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.950192+0000", 2020-01-16T00:13:16.937 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7286000000000001e-05 2020-01-16T00:13:16.937 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.937 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.937 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:16.937 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.950239+0000", 2020-01-16T00:13:16.937 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.008400154 2020-01-16T00:13:16.937 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.937 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:16.938 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.938 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.938 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.938 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:48 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e28)", 2020-01-16T00:13:16.938 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:47.942354+0000", 2020-01-16T00:13:16.939 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 28.920807656000001, 2020-01-16T00:13:16.939 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0080319870000000008, 2020-01-16T00:13:16.939 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:16.940 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:16.940 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:16.940 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.940 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:16.940 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942354+0000", 2020-01-16T00:13:16.941 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:16.941 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.941 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.941 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:16.941 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942354+0000", 2020-01-16T00:13:16.941 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:16.941 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.941 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.942 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:16.942 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942353+0000", 2020-01-16T00:13:16.942 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5825000000000002e-05 2020-01-16T00:13:16.942 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.942 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.942 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:16.942 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942409+0000", 2020-01-16T00:13:16.942 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5929999999999996e-06 2020-01-16T00:13:16.943 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.943 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.943 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:16.943 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942414+0000", 2020-01-16T00:13:16.943 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4199999999999999e-06 2020-01-16T00:13:16.943 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.943 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.944 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:16.944 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942418+0000", 2020-01-16T00:13:16.944 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0027594500000000001 2020-01-16T00:13:16.944 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.944 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.944 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.944 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.945177+0000", 2020-01-16T00:13:16.944 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.4040000000000002e-05 2020-01-16T00:13:16.945 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.945 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.945 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:16.945 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.945261+0000", 2020-01-16T00:13:16.946 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012906699999999999 2020-01-16T00:13:16.946 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.946 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.946 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:16.946 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.945390+0000", 2020-01-16T00:13:16.946 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0047161060000000003 2020-01-16T00:13:16.946 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.946 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.947 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:16.947 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.950106+0000", 2020-01-16T00:13:16.947 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016987400000000001 2020-01-16T00:13:16.947 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.947 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.947 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.947 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.950276+0000", 2020-01-16T00:13:16.947 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6329000000000002e-05 2020-01-16T00:13:16.948 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.948 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.948 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.948 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.950303+0000", 2020-01-16T00:13:16.948 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1443000000000003e-05 2020-01-16T00:13:16.948 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.948 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.948 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:16.949 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.950344+0000", 2020-01-16T00:13:16.949 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1486000000000002e-05 2020-01-16T00:13:16.949 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.949 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.949 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:16.949 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.950386+0000", 2020-01-16T00:13:16.950 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0080319870000000008 2020-01-16T00:13:16.950 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.950 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:16.950 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.950 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.950 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.951 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:55 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e30)", 2020-01-16T00:13:16.951 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.736036+0000", 2020-01-16T00:13:16.951 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 24.127125266, 2020-01-16T00:13:16.951 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010974536, 2020-01-16T00:13:16.951 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:16.951 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:16.951 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:16.952 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.952 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:16.952 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736036+0000", 2020-01-16T00:13:16.952 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:16.952 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.952 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.952 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:16.952 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736036+0000", 2020-01-16T00:13:16.953 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:16.953 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.953 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:16.953 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736034+0000", 2020-01-16T00:13:16.953 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043589699999999998 2020-01-16T00:13:16.953 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.953 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.954 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:16.954 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736469+0000", 2020-01-16T00:13:16.954 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9882e-05 2020-01-16T00:13:16.954 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.954 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.954 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:16.954 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736489+0000", 2020-01-16T00:13:16.954 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8339999999999999e-06 2020-01-16T00:13:16.955 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.955 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.955 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:16.955 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736497+0000", 2020-01-16T00:13:16.955 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5354999999999999e-05 2020-01-16T00:13:16.956 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.956 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.956 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.956 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736572+0000", 2020-01-16T00:13:16.956 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5228000000000004e-05 2020-01-16T00:13:16.956 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.956 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.956 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:16.957 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736668+0000", 2020-01-16T00:13:16.957 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017212999999999999 2020-01-16T00:13:16.957 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.957 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.957 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:16.957 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736840+0000", 2020-01-16T00:13:16.957 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0099057020000000006 2020-01-16T00:13:16.957 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.958 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.958 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.958 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746746+0000", 2020-01-16T00:13:16.958 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0586000000000005e-05 2020-01-16T00:13:16.958 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.958 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.958 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:16.959 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746816+0000", 2020-01-16T00:13:16.959 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2445000000000002e-05 2020-01-16T00:13:16.959 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.959 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.959 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.959 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746889+0000", 2020-01-16T00:13:16.959 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6839000000000002e-05 2020-01-16T00:13:16.959 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.960 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.960 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:16.960 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746955+0000", 2020-01-16T00:13:16.960 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5074000000000002e-05 2020-01-16T00:13:16.960 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.960 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.961 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:16.961 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.747011+0000", 2020-01-16T00:13:16.961 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010974536 2020-01-16T00:13:16.961 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.961 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:16.961 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.961 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.961 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.962 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:56 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e30)", 2020-01-16T00:13:16.962 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.736514+0000", 2020-01-16T00:13:16.962 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 24.126647430999999, 2020-01-16T00:13:16.962 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011660584999999999, 2020-01-16T00:13:16.962 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:16.962 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:16.962 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:16.962 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.963 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:16.963 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736514+0000", 2020-01-16T00:13:16.963 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:16.963 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.963 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.963 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:16.963 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736514+0000", 2020-01-16T00:13:16.963 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:16.964 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.964 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.964 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:16.964 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736513+0000", 2020-01-16T00:13:16.964 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9677000000000004e-05 2020-01-16T00:13:16.964 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.964 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.965 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:16.965 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736583+0000", 2020-01-16T00:13:16.965 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5269999999999998e-06 2020-01-16T00:13:16.965 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.965 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.965 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:16.966 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736591+0000", 2020-01-16T00:13:16.966 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5769999999999996e-06 2020-01-16T00:13:16.966 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.966 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.966 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:16.966 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736599+0000", 2020-01-16T00:13:16.966 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0053593310000000002 2020-01-16T00:13:16.967 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.967 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.967 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.967 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.741958+0000", 2020-01-16T00:13:16.967 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014164099999999999 2020-01-16T00:13:16.967 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.967 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.968 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:16.968 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.742100+0000", 2020-01-16T00:13:16.968 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020126000000000001 2020-01-16T00:13:16.968 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.968 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.968 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:16.968 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.742301+0000", 2020-01-16T00:13:16.969 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0045052859999999998 2020-01-16T00:13:16.969 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.969 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.969 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.969 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746806+0000", 2020-01-16T00:13:16.969 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00070802699999999996 2020-01-16T00:13:16.969 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.969 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.970 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:16.970 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.747514+0000", 2020-01-16T00:13:16.970 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00059247200000000003 2020-01-16T00:13:16.970 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.970 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.970 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.970 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748107+0000", 2020-01-16T00:13:16.970 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1870999999999999e-05 2020-01-16T00:13:16.971 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.971 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.971 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:16.971 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748149+0000", 2020-01-16T00:13:16.971 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5831000000000001e-05 2020-01-16T00:13:16.971 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.972 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.972 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:16.972 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748174+0000", 2020-01-16T00:13:16.972 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011660584999999999 2020-01-16T00:13:16.972 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.972 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:16.972 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.973 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.973 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.973 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:68 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 56832~1408000] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e34)", 2020-01-16T00:13:16.973 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:03.257978+0000", 2020-01-16T00:13:16.973 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 13.605182880999999, 2020-01-16T00:13:16.973 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0084778949999999992, 2020-01-16T00:13:16.973 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:16.973 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:16.974 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:16.974 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.974 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:16.974 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.257978+0000", 2020-01-16T00:13:16.974 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:16.974 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.974 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.975 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:16.975 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.257978+0000", 2020-01-16T00:13:16.975 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:16.975 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.975 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.975 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:16.975 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.257977+0000", 2020-01-16T00:13:16.976 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00042032400000000002 2020-01-16T00:13:16.976 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.976 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.976 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:16.976 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258397+0000", 2020-01-16T00:13:16.977 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5951999999999999e-05 2020-01-16T00:13:16.977 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.977 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.977 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:16.977 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258413+0000", 2020-01-16T00:13:16.977 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0729999999999999e-06 2020-01-16T00:13:16.977 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.977 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.978 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:16.978 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258420+0000", 2020-01-16T00:13:16.978 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013206700000000001 2020-01-16T00:13:16.978 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.978 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.978 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.978 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258552+0000", 2020-01-16T00:13:16.978 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2818999999999996e-05 2020-01-16T00:13:16.979 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.979 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.979 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:16.979 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258615+0000", 2020-01-16T00:13:16.979 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012016899999999999 2020-01-16T00:13:16.979 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.979 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.979 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:16.980 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258735+0000", 2020-01-16T00:13:16.980 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052078480000000002 2020-01-16T00:13:16.980 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.980 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.980 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:16.980 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.263943+0000", 2020-01-16T00:13:16.981 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0019258649999999999 2020-01-16T00:13:16.982 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.982 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.982 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.982 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.265869+0000", 2020-01-16T00:13:16.982 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00051785399999999999 2020-01-16T00:13:16.982 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.983 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.983 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.983 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.266387+0000", 2020-01-16T00:13:16.983 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0459e-05 2020-01-16T00:13:16.983 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.983 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.983 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:16.983 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.266427+0000", 2020-01-16T00:13:16.984 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.919e-05 2020-01-16T00:13:16.984 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.984 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.984 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:16.984 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.266456+0000", 2020-01-16T00:13:16.984 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0084778949999999992 2020-01-16T00:13:16.984 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.985 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:16.985 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.985 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.985 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.985 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:69 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 1922048~118272] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e34)", 2020-01-16T00:13:16.985 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:03.258431+0000", 2020-01-16T00:13:16.985 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 13.60472985, 2020-01-16T00:13:16.985 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0092806959999999997, 2020-01-16T00:13:16.986 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:16.986 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:16.986 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:16.986 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.986 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:16.986 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258431+0000", 2020-01-16T00:13:16.986 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:16.987 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.987 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.987 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:16.987 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258431+0000", 2020-01-16T00:13:16.987 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:16.987 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.987 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.987 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:16.988 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258431+0000", 2020-01-16T00:13:16.988 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3875000000000003e-05 2020-01-16T00:13:16.988 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.988 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.988 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:16.988 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258474+0000", 2020-01-16T00:13:16.988 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1040000000000007e-06 2020-01-16T00:13:16.988 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.989 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.989 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:16.989 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258483+0000", 2020-01-16T00:13:16.989 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6720000000000003e-06 2020-01-16T00:13:16.989 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.989 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.989 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:16.990 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258487+0000", 2020-01-16T00:13:16.990 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020507110000000002 2020-01-16T00:13:16.990 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.990 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.991 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:16.991 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.260538+0000", 2020-01-16T00:13:16.991 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017511100000000001 2020-01-16T00:13:16.991 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.991 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.992 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:16.992 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.260713+0000", 2020-01-16T00:13:16.992 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018914100000000001 2020-01-16T00:13:16.992 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.992 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.992 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:16.992 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.260902+0000", 2020-01-16T00:13:16.992 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.004582282 2020-01-16T00:13:16.993 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.993 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.993 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:16.993 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.265485+0000", 2020-01-16T00:13:16.993 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021355329999999998 2020-01-16T00:13:16.993 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.993 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.994 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.994 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.267620+0000", 2020-01-16T00:13:16.994 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1495999999999998e-05 2020-01-16T00:13:16.994 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.994 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.994 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:16.994 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.267652+0000", 2020-01-16T00:13:16.995 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5474e-05 2020-01-16T00:13:16.995 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.995 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.995 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:16.995 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.267687+0000", 2020-01-16T00:13:16.995 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5055e-05 2020-01-16T00:13:16.995 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.996 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.996 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:16.996 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.267712+0000", 2020-01-16T00:13:16.996 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0092806959999999997 2020-01-16T00:13:16.996 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.996 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:16.996 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:16.996 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.997 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.997 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:70 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e35)", 2020-01-16T00:13:16.997 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.985494+0000", 2020-01-16T00:13:16.997 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 10.877667635, 2020-01-16T00:13:16.997 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013746267, 2020-01-16T00:13:16.997 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:16.997 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:16.997 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:16.998 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.998 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:16.998 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985494+0000", 2020-01-16T00:13:16.998 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:16.998 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.998 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.998 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:16.999 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985494+0000", 2020-01-16T00:13:16.999 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:16.999 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:16.999 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:16.999 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:16.999 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985490+0000", 2020-01-16T00:13:16.999 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00077792099999999999 2020-01-16T00:13:16.999 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.000 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.000 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:17.000 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986268+0000", 2020-01-16T00:13:17.000 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9718000000000001e-05 2020-01-16T00:13:17.000 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.000 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.000 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:17.000 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986318+0000", 2020-01-16T00:13:17.001 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.4349999999999997e-06 2020-01-16T00:13:17.001 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.001 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.001 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:17.001 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986326+0000", 2020-01-16T00:13:17.001 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3298999999999999e-05 2020-01-16T00:13:17.001 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.001 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.002 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:17.002 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986380+0000", 2020-01-16T00:13:17.002 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4969999999999995e-05 2020-01-16T00:13:17.002 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.002 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.002 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:17.002 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986455+0000", 2020-01-16T00:13:17.002 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000110587 2020-01-16T00:13:17.003 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.003 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.003 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:17.003 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986565+0000", 2020-01-16T00:13:17.003 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063931600000000002 2020-01-16T00:13:17.003 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.003 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.003 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:17.004 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.992958+0000", 2020-01-16T00:13:17.004 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013008480000000001 2020-01-16T00:13:17.004 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.004 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.004 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:17.004 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.994259+0000", 2020-01-16T00:13:17.004 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048399979999999999 2020-01-16T00:13:17.004 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.005 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.005 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:17.005 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999099+0000", 2020-01-16T00:13:17.005 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1874999999999998e-05 2020-01-16T00:13:17.005 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.005 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.005 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:17.005 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999181+0000", 2020-01-16T00:13:17.006 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8865e-05 2020-01-16T00:13:17.006 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.006 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.006 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:17.006 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999240+0000", 2020-01-16T00:13:17.006 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013746267 2020-01-16T00:13:17.006 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:17.006 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:17.007 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:17.007 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.007 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.007 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:71 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e35)", 2020-01-16T00:13:17.007 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.986341+0000", 2020-01-16T00:13:17.007 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 10.876820274, 2020-01-16T00:13:17.007 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013252807, 2020-01-16T00:13:17.007 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:17.008 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:17.008 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:17.008 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.008 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:17.008 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986341+0000", 2020-01-16T00:13:17.008 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:17.008 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.008 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.009 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:17.009 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986341+0000", 2020-01-16T00:13:17.009 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:17.009 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.009 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.009 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:17.009 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986340+0000", 2020-01-16T00:13:17.009 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000103867 2020-01-16T00:13:17.010 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.010 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.010 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:17.010 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986444+0000", 2020-01-16T00:13:17.010 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0104e-05 2020-01-16T00:13:17.010 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.010 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.010 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:17.011 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986454+0000", 2020-01-16T00:13:17.011 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4220000000000002e-06 2020-01-16T00:13:17.011 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.011 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.011 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:17.011 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986458+0000", 2020-01-16T00:13:17.011 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033523070000000001 2020-01-16T00:13:17.011 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.012 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.012 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:17.012 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.989811+0000", 2020-01-16T00:13:17.012 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3296999999999999e-05 2020-01-16T00:13:17.012 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.012 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.012 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:17.012 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.989894+0000", 2020-01-16T00:13:17.013 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000117701 2020-01-16T00:13:17.013 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.013 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.013 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:17.013 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.990012+0000", 2020-01-16T00:13:17.013 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0032873490000000002 2020-01-16T00:13:17.013 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.013 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.014 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:17.014 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993299+0000", 2020-01-16T00:13:17.014 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002422619 2020-01-16T00:13:17.014 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.014 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.014 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:17.014 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.995722+0000", 2020-01-16T00:13:17.014 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0037839589999999999 2020-01-16T00:13:17.015 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.015 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.015 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:17.015 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999506+0000", 2020-01-16T00:13:17.015 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8559999999999997e-05 2020-01-16T00:13:17.015 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.015 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.015 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:17.016 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999544+0000", 2020-01-16T00:13:17.016 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9599999999999999e-05 2020-01-16T00:13:17.016 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.016 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.016 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:17.016 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999594+0000", 2020-01-16T00:13:17.016 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013252807 2020-01-16T00:13:17.016 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:17.017 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:17.017 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:17.017 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.017 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.017 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:77 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e36)", 2020-01-16T00:13:17.017 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.555854+0000", 2020-01-16T00:13:17.017 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 9.3073071850000009, 2020-01-16T00:13:17.017 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010455002999999999, 2020-01-16T00:13:17.018 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:17.018 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:17.018 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:17.018 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.018 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:17.018 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555854+0000", 2020-01-16T00:13:17.018 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:17.018 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.019 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.019 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:17.019 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555854+0000", 2020-01-16T00:13:17.019 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:17.019 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.019 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.019 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:17.019 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555851+0000", 2020-01-16T00:13:17.020 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00048310200000000001 2020-01-16T00:13:17.020 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.020 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.020 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:17.020 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556334+0000", 2020-01-16T00:13:17.020 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2232e-05 2020-01-16T00:13:17.020 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.020 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.021 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:17.021 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556356+0000", 2020-01-16T00:13:17.021 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9750000000000007e-06 2020-01-16T00:13:17.021 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.021 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.021 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:17.021 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556364+0000", 2020-01-16T00:13:17.022 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8872999999999997e-05 2020-01-16T00:13:17.022 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.022 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.022 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:17.022 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556423+0000", 2020-01-16T00:13:17.022 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8687000000000001e-05 2020-01-16T00:13:17.022 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.023 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.023 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:17.023 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556502+0000", 2020-01-16T00:13:17.023 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013235300000000001 2020-01-16T00:13:17.023 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.023 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.023 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:17.023 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556634+0000", 2020-01-16T00:13:17.024 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0059770520000000001 2020-01-16T00:13:17.024 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.024 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.024 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:17.024 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.562611+0000", 2020-01-16T00:13:17.024 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029492820000000001 2020-01-16T00:13:17.024 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.024 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.025 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:17.025 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.565561+0000", 2020-01-16T00:13:17.025 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00060620300000000004 2020-01-16T00:13:17.025 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.025 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.025 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:17.025 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566167+0000", 2020-01-16T00:13:17.025 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3831000000000004e-05 2020-01-16T00:13:17.026 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.026 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.026 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:17.026 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566251+0000", 2020-01-16T00:13:17.026 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8328000000000001e-05 2020-01-16T00:13:17.026 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.026 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.026 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:17.027 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566309+0000", 2020-01-16T00:13:17.027 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010455002999999999 2020-01-16T00:13:17.027 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:17.027 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:17.027 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:17.027 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.027 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.027 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:78 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e36)", 2020-01-16T00:13:17.028 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.556378+0000", 2020-01-16T00:13:17.028 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 9.3067832520000007, 2020-01-16T00:13:17.028 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010396311, 2020-01-16T00:13:17.028 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:17.028 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:17.028 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:17.028 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.028 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:17.029 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556378+0000", 2020-01-16T00:13:17.029 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:17.029 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.029 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.029 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:17.029 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556378+0000", 2020-01-16T00:13:17.029 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:17.029 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.030 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.030 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:17.030 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556377+0000", 2020-01-16T00:13:17.030 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4307000000000001e-05 2020-01-16T00:13:17.030 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.030 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.030 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:17.030 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556422+0000", 2020-01-16T00:13:17.031 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9620000000000005e-06 2020-01-16T00:13:17.031 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.031 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.031 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:17.031 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556432+0000", 2020-01-16T00:13:17.031 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5979999999999999e-06 2020-01-16T00:13:17.031 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.031 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.032 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:17.032 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556436+0000", 2020-01-16T00:13:17.032 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021059469999999999 2020-01-16T00:13:17.032 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.032 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.032 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:17.032 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.558542+0000", 2020-01-16T00:13:17.032 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000194458 2020-01-16T00:13:17.033 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.033 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.033 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:17.033 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.558737+0000", 2020-01-16T00:13:17.033 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00032867900000000002 2020-01-16T00:13:17.033 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.033 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.034 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:17.034 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.559065+0000", 2020-01-16T00:13:17.034 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052547499999999999 2020-01-16T00:13:17.034 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.034 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.034 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:17.034 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.564320+0000", 2020-01-16T00:13:17.034 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001565213 2020-01-16T00:13:17.035 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.035 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.035 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:17.035 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.565885+0000", 2020-01-16T00:13:17.035 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00076994300000000004 2020-01-16T00:13:17.035 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.035 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.035 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:17.036 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566655+0000", 2020-01-16T00:13:17.036 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7533000000000005e-05 2020-01-16T00:13:17.036 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.036 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.036 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:17.036 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566723+0000", 2020-01-16T00:13:17.036 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1616999999999997e-05 2020-01-16T00:13:17.036 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.037 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.037 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:17.037 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566774+0000", 2020-01-16T00:13:17.037 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010396311 2020-01-16T00:13:17.037 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:17.037 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:17.037 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:17.037 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.038 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.038 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:72 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 56832~1408000] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e37)", 2020-01-16T00:13:17.038 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.578753+0000", 2020-01-16T00:13:17.038 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.2844079490000002, 2020-01-16T00:13:17.038 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011826918, 2020-01-16T00:13:17.038 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:17.038 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:17.038 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:17.039 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.039 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:17.039 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578753+0000", 2020-01-16T00:13:17.039 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:17.039 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.039 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.039 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:17.039 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578753+0000", 2020-01-16T00:13:17.040 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:17.040 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.040 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.040 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:17.040 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578751+0000", 2020-01-16T00:13:17.040 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00042457699999999998 2020-01-16T00:13:17.040 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.041 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.041 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:17.041 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579175+0000", 2020-01-16T00:13:17.041 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5404999999999999e-05 2020-01-16T00:13:17.041 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.041 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.041 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:17.041 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579191+0000", 2020-01-16T00:13:17.042 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6149999999999996e-06 2020-01-16T00:13:17.042 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.042 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.042 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:17.042 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579196+0000", 2020-01-16T00:13:17.042 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000104794 2020-01-16T00:13:17.042 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.043 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.043 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:17.043 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579301+0000", 2020-01-16T00:13:17.043 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020947199999999999 2020-01-16T00:13:17.043 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.043 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.043 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:17.043 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579511+0000", 2020-01-16T00:13:17.044 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034328400000000002 2020-01-16T00:13:17.044 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.044 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.044 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:17.044 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579854+0000", 2020-01-16T00:13:17.044 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.005078117 2020-01-16T00:13:17.044 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.044 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.045 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:17.045 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.584932+0000", 2020-01-16T00:13:17.045 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0014702160000000001 2020-01-16T00:13:17.045 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.045 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.045 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:17.045 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.586402+0000", 2020-01-16T00:13:17.045 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040623589999999998 2020-01-16T00:13:17.046 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.046 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.046 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:17.046 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590465+0000", 2020-01-16T00:13:17.046 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4478000000000002e-05 2020-01-16T00:13:17.046 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.046 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.046 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:17.047 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590529+0000", 2020-01-16T00:13:17.047 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1121000000000001e-05 2020-01-16T00:13:17.047 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.047 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.047 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:17.047 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590580+0000", 2020-01-16T00:13:17.047 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011826918 2020-01-16T00:13:17.047 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:17.048 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:17.048 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:17.048 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.048 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.048 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:73 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 1922048~118272] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e37)", 2020-01-16T00:13:17.048 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:13:17.048 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.2839551519999999, 2020-01-16T00:13:17.048 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011750764, 2020-01-16T00:13:17.049 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:17.049 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:17.049 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:17.049 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.049 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:17.049 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:13:17.049 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:17.050 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.050 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.050 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:17.050 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:13:17.050 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:17.050 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.051 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.051 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:17.051 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:13:17.051 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2780000000000001e-05 2020-01-16T00:13:17.051 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.051 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.051 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:17.052 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579238+0000", 2020-01-16T00:13:17.052 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4341999999999999e-05 2020-01-16T00:13:17.052 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.052 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.052 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:17.052 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579253+0000", 2020-01-16T00:13:17.052 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1920000000000001e-06 2020-01-16T00:13:17.052 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.053 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.053 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:17.053 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579256+0000", 2020-01-16T00:13:17.053 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003224902 2020-01-16T00:13:17.053 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.053 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.053 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:17.054 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.582481+0000", 2020-01-16T00:13:17.054 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000180729 2020-01-16T00:13:17.054 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.054 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.054 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:17.054 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.582662+0000", 2020-01-16T00:13:17.054 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00025106100000000001 2020-01-16T00:13:17.054 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.055 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.055 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:17.055 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.582913+0000", 2020-01-16T00:13:17.055 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020814449999999999 2020-01-16T00:13:17.055 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.055 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.055 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:17.056 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.584994+0000", 2020-01-16T00:13:17.056 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001717243 2020-01-16T00:13:17.056 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.056 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.056 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:17.056 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.586711+0000", 2020-01-16T00:13:17.056 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041349869999999997 2020-01-16T00:13:17.056 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.057 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.057 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:17.057 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590846+0000", 2020-01-16T00:13:17.057 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3565000000000006e-05 2020-01-16T00:13:17.057 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.057 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.057 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:17.058 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590910+0000", 2020-01-16T00:13:17.058 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7002999999999997e-05 2020-01-16T00:13:17.058 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.058 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.058 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:17.058 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590957+0000", 2020-01-16T00:13:17.058 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011750764 2020-01-16T00:13:17.058 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:17.059 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:17.059 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:17.059 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.059 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.059 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:89 2.1 2:9b8617ce:::smithi12313934-38 low_tier:head [write 69632~773120] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e37)", 2020-01-16T00:13:17.059 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.593701+0000", 2020-01-16T00:13:17.059 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.2694605259999996, 2020-01-16T00:13:17.059 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010026929, 2020-01-16T00:13:17.060 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:17.060 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:17.060 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:17.060 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.060 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:17.060 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593701+0000", 2020-01-16T00:13:17.060 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:17.060 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.061 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.061 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:17.061 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593701+0000", 2020-01-16T00:13:17.061 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:17.061 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.061 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.061 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:17.061 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593699+0000", 2020-01-16T00:13:17.062 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000237564 2020-01-16T00:13:17.062 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.062 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.062 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:17.062 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593936+0000", 2020-01-16T00:13:17.062 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2666e-05 2020-01-16T00:13:17.062 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.063 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.063 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:17.063 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593949+0000", 2020-01-16T00:13:17.063 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0089999999999998e-06 2020-01-16T00:13:17.063 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.063 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.063 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:17.064 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593954+0000", 2020-01-16T00:13:17.064 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5813000000000003e-05 2020-01-16T00:13:17.064 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.064 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.064 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:17.064 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.594000+0000", 2020-01-16T00:13:17.064 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010155299999999999 2020-01-16T00:13:17.064 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.065 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.065 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:17.065 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.594101+0000", 2020-01-16T00:13:17.065 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000203807 2020-01-16T00:13:17.065 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.065 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.065 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:17.065 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.594305+0000", 2020-01-16T00:13:17.066 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021655390000000002 2020-01-16T00:13:17.066 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.066 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.066 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:17.066 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.596471+0000", 2020-01-16T00:13:17.066 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0070645320000000001 2020-01-16T00:13:17.066 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.067 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.067 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:17.067 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603535+0000", 2020-01-16T00:13:17.067 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7577999999999998e-05 2020-01-16T00:13:17.067 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.067 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.067 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:17.067 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603573+0000", 2020-01-16T00:13:17.068 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9933999999999997e-05 2020-01-16T00:13:17.068 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.068 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.068 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:17.068 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603653+0000", 2020-01-16T00:13:17.068 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4993000000000005e-05 2020-01-16T00:13:17.068 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.069 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.069 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:17.069 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603728+0000", 2020-01-16T00:13:17.069 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010026929 2020-01-16T00:13:17.069 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:17.069 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:17.069 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:17.069 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.070 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.070 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:91 2.1 2:9b8617ce:::smithi12313934-38 low_tier:head [write 1915392~590334] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e37)", 2020-01-16T00:13:17.070 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.596978+0000", 2020-01-16T00:13:17.070 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.2661828289999999, 2020-01-16T00:13:17.070 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0079576309999999997, 2020-01-16T00:13:17.070 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:17.071 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:17.071 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:17.071 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.071 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:17.071 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.596978+0000", 2020-01-16T00:13:17.071 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:17.071 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.071 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.072 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:17.072 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.596978+0000", 2020-01-16T00:13:17.072 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:17.072 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.072 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.072 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:17.072 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.596978+0000", 2020-01-16T00:13:17.072 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000170557 2020-01-16T00:13:17.073 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.073 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.073 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:17.073 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.597148+0000", 2020-01-16T00:13:17.073 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5519999999999999e-06 2020-01-16T00:13:17.073 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.073 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.074 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:17.074 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.597156+0000", 2020-01-16T00:13:17.074 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7520000000000004e-06 2020-01-16T00:13:17.074 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.074 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.074 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:17.074 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.597161+0000", 2020-01-16T00:13:17.075 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0028888260000000002 2020-01-16T00:13:17.075 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.075 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.075 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:17.075 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.600049+0000", 2020-01-16T00:13:17.075 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7899999999999994e-05 2020-01-16T00:13:17.075 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.075 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.076 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:17.076 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.600147+0000", 2020-01-16T00:13:17.076 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000169877 2020-01-16T00:13:17.076 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.076 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.076 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:17.076 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.600317+0000", 2020-01-16T00:13:17.076 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0043677259999999997 2020-01-16T00:13:17.077 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.077 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.077 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:17.077 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.604685+0000", 2020-01-16T00:13:17.077 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013208999999999999 2020-01-16T00:13:17.077 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.077 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.077 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:17.078 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.604817+0000", 2020-01-16T00:13:17.078 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4851000000000001e-05 2020-01-16T00:13:17.078 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.078 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.078 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:17.078 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.604832+0000", 2020-01-16T00:13:17.078 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8831e-05 2020-01-16T00:13:17.078 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.079 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.079 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:17.079 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.604891+0000", 2020-01-16T00:13:17.079 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5411000000000003e-05 2020-01-16T00:13:17.079 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.079 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.079 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:17.079 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.604936+0000", 2020-01-16T00:13:17.080 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0079576309999999997 2020-01-16T00:13:17.080 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:17.080 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:17.080 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:17.080 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.080 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.080 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:74 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 56832~1408000] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e41)", 2020-01-16T00:13:17.080 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:16.601158+0000", 2020-01-16T00:13:17.081 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.26200340799999999, 2020-01-16T00:13:17.081 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0076921350000000001, 2020-01-16T00:13:17.081 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:17.081 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:17.081 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:17.081 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.081 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:17.081 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.601158+0000", 2020-01-16T00:13:17.082 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:17.082 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.082 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.082 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:17.082 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.601158+0000", 2020-01-16T00:13:17.082 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:17.082 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.083 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.083 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:17.083 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.601156+0000", 2020-01-16T00:13:17.083 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00042862199999999998 2020-01-16T00:13:17.083 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.083 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.083 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:17.083 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.601585+0000", 2020-01-16T00:13:17.084 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8362000000000001e-05 2020-01-16T00:13:17.084 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.084 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.084 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:17.084 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.601603+0000", 2020-01-16T00:13:17.084 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9269999999999999e-06 2020-01-16T00:13:17.084 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.084 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.085 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:17.085 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.601610+0000", 2020-01-16T00:13:17.085 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7003000000000003e-05 2020-01-16T00:13:17.085 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.085 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.085 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:17.085 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.601667+0000", 2020-01-16T00:13:17.085 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112724 2020-01-16T00:13:17.086 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.086 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.086 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:17.086 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.601780+0000", 2020-01-16T00:13:17.086 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000187176 2020-01-16T00:13:17.086 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.086 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.086 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:17.087 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.601967+0000", 2020-01-16T00:13:17.087 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0065865639999999996 2020-01-16T00:13:17.087 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.087 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.087 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:17.087 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.608553+0000", 2020-01-16T00:13:17.087 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6983000000000006e-05 2020-01-16T00:13:17.087 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.088 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.088 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:17.088 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.608620+0000", 2020-01-16T00:13:17.088 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9728000000000006e-05 2020-01-16T00:13:17.088 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.088 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.088 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:17.088 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.608710+0000", 2020-01-16T00:13:17.089 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0473999999999997e-05 2020-01-16T00:13:17.089 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.089 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.089 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:17.089 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.608771+0000", 2020-01-16T00:13:17.089 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9382000000000007e-05 2020-01-16T00:13:17.089 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.090 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.090 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:17.090 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.608850+0000", 2020-01-16T00:13:17.090 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0076921350000000001 2020-01-16T00:13:17.090 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:17.090 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:17.090 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:17.091 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.091 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.091 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:75 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 1922048~118272] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e41)", 2020-01-16T00:13:17.091 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:16.601624+0000", 2020-01-16T00:13:17.091 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.26153674999999998, 2020-01-16T00:13:17.091 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0079729169999999995, 2020-01-16T00:13:17.091 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:17.091 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:17.092 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:17.092 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.092 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:17.092 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.601624+0000", 2020-01-16T00:13:17.092 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:17.092 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.093 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.093 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:17.093 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.601624+0000", 2020-01-16T00:13:17.093 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:17.093 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.093 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.093 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:17.094 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.601624+0000", 2020-01-16T00:13:17.094 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3544999999999997e-05 2020-01-16T00:13:17.094 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.094 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.094 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:17.094 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.601667+0000", 2020-01-16T00:13:17.094 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2656999999999999e-05 2020-01-16T00:13:17.094 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.095 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.095 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:17.095 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.601680+0000", 2020-01-16T00:13:17.095 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9760000000000003e-06 2020-01-16T00:13:17.095 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.095 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.095 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:17.095 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.601685+0000", 2020-01-16T00:13:17.096 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0026060940000000002 2020-01-16T00:13:17.096 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.096 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.096 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:17.096 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.604291+0000", 2020-01-16T00:13:17.096 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016571000000000001 2020-01-16T00:13:17.096 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.096 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.097 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:17.097 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.604457+0000", 2020-01-16T00:13:17.097 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017136699999999999 2020-01-16T00:13:17.097 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.097 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.097 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:17.097 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.604628+0000", 2020-01-16T00:13:17.098 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040386509999999999 2020-01-16T00:13:17.098 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.098 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.098 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:17.098 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.608667+0000", 2020-01-16T00:13:17.098 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00040341800000000001 2020-01-16T00:13:17.098 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.098 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.099 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:17.099 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.609070+0000", 2020-01-16T00:13:17.099 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00039049100000000003 2020-01-16T00:13:17.099 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.099 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.099 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:17.099 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.609461+0000", 2020-01-16T00:13:17.099 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4151000000000002e-05 2020-01-16T00:13:17.100 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.100 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.100 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:17.100 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.609535+0000", 2020-01-16T00:13:17.100 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2636000000000001e-05 2020-01-16T00:13:17.100 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:17.100 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:17.100 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:17.101 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.609597+0000", 2020-01-16T00:13:17.101 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0079729169999999995 2020-01-16T00:13:17.101 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:17.101 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:17.101 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:17.101 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:17.101 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:17.101 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:17.103 INFO:tasks.rados.rados.0.smithi123.stdout:1139: finishing write tid 5 to smithi12313934-35 2020-01-16T00:13:17.103 INFO:tasks.rados.rados.0.smithi123.stdout:1139: finishing write tid 6 to smithi12313934-35 2020-01-16T00:13:17.103 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 35 v 131 (ObjNum 325 snap 0 seq_num 325) dirty exists 2020-01-16T00:13:17.103 INFO:tasks.rados.rados.0.smithi123.stdout:1139: left oid 35 (ObjNum 325 snap 0 seq_num 325) 2020-01-16T00:13:17.104 INFO:tasks.rados.rados.0.smithi123.stdout:1142: expect (ObjNum 308 snap 0 seq_num 308) 2020-01-16T00:13:17.104 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:17.100+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:17.104 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-01-16T00:13:17.117 INFO:tasks.rados.rados.0.smithi123.stdout:1143: finishing write tid 1 to smithi12313934-1 2020-01-16T00:13:17.117 INFO:tasks.rados.rados.0.smithi123.stdout:1143: finishing write tid 2 to smithi12313934-1 2020-01-16T00:13:17.118 INFO:tasks.rados.rados.0.smithi123.stdout:1143: finishing write tid 3 to smithi12313934-1 2020-01-16T00:13:17.118 INFO:tasks.rados.rados.0.smithi123.stdout:1143: finishing write tid 5 to smithi12313934-1 2020-01-16T00:13:17.118 INFO:tasks.rados.rados.0.smithi123.stdout:1143: finishing write tid 6 to smithi12313934-1 2020-01-16T00:13:17.118 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 1 v 223 (ObjNum 326 snap 0 seq_num 326) dirty exists 2020-01-16T00:13:17.118 INFO:tasks.rados.rados.0.smithi123.stdout:1143: left oid 1 (ObjNum 326 snap 0 seq_num 326) 2020-01-16T00:13:17.118 INFO:tasks.rados.rados.0.smithi123.stdout:1144: finishing write tid 1 to smithi12313934-14 2020-01-16T00:13:17.118 INFO:tasks.rados.rados.0.smithi123.stdout:1144: finishing write tid 2 to smithi12313934-14 2020-01-16T00:13:17.119 INFO:tasks.rados.rados.0.smithi123.stdout:1145: expect (ObjNum 260 snap 0 seq_num 260) 2020-01-16T00:13:17.204 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:17.202+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:17.213 INFO:tasks.rados.rados.0.smithi123.stdout:1144: finishing write tid 3 to smithi12313934-14 2020-01-16T00:13:17.213 INFO:tasks.rados.rados.0.smithi123.stdout:1144: finishing write tid 5 to smithi12313934-14 2020-01-16T00:13:17.213 INFO:tasks.rados.rados.0.smithi123.stdout:1144: finishing write tid 6 to smithi12313934-14 2020-01-16T00:13:17.214 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 14 v 168 (ObjNum 327 snap 0 seq_num 327) dirty exists 2020-01-16T00:13:17.214 INFO:tasks.rados.rados.0.smithi123.stdout:1144: left oid 14 (ObjNum 327 snap 0 seq_num 327) 2020-01-16T00:13:17.214 INFO:tasks.rados.rados.0.smithi123.stdout:1146: finishing write tid 1 to smithi12313934-46 2020-01-16T00:13:17.214 INFO:tasks.rados.rados.0.smithi123.stdout:1146: finishing write tid 2 to smithi12313934-46 2020-01-16T00:13:17.214 INFO:tasks.rados.rados.0.smithi123.stdout:1146: finishing write tid 3 to smithi12313934-46 2020-01-16T00:13:17.214 INFO:tasks.rados.rados.0.smithi123.stdout:1146: finishing write tid 5 to smithi12313934-46 2020-01-16T00:13:17.214 INFO:tasks.rados.rados.0.smithi123.stdout:1146: finishing write tid 6 to smithi12313934-46 2020-01-16T00:13:17.214 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 46 v 228 (ObjNum 328 snap 0 seq_num 328) dirty exists 2020-01-16T00:13:17.215 INFO:tasks.rados.rados.0.smithi123.stdout:1146: left oid 46 (ObjNum 328 snap 0 seq_num 328) 2020-01-16T00:13:17.215 INFO:tasks.rados.rados.0.smithi123.stdout:1147: finishing write tid 1 to smithi12313934-11 2020-01-16T00:13:17.215 INFO:tasks.rados.rados.0.smithi123.stdout:1147: finishing write tid 2 to smithi12313934-11 2020-01-16T00:13:17.215 INFO:tasks.rados.rados.0.smithi123.stdout:1139: done (15 left) 2020-01-16T00:13:17.215 INFO:tasks.rados.rados.0.smithi123.stdout:1140: done (14 left) 2020-01-16T00:13:17.215 INFO:tasks.rados.rados.0.smithi123.stdout:1141: done (13 left) 2020-01-16T00:13:17.215 INFO:tasks.rados.rados.0.smithi123.stdout:1142: done (12 left) 2020-01-16T00:13:17.215 INFO:tasks.rados.rados.0.smithi123.stdout:1143: done (11 left) 2020-01-16T00:13:17.216 INFO:tasks.rados.rados.0.smithi123.stdout:1144: done (10 left) 2020-01-16T00:13:17.217 INFO:tasks.rados.rados.0.smithi123.stdout:1145: done (9 left) 2020-01-16T00:13:17.217 INFO:tasks.rados.rados.0.smithi123.stdout:1146: done (8 left) 2020-01-16T00:13:17.217 INFO:tasks.rados.rados.0.smithi123.stdout:1155: read oid 38 snap -1 2020-01-16T00:13:17.217 INFO:tasks.rados.rados.0.smithi123.stdout:1155: expect (ObjNum 298 snap 0 seq_num 298) 2020-01-16T00:13:17.217 INFO:tasks.rados.rados.0.smithi123.stdout:1156: read oid 19 snap -1 2020-01-16T00:13:17.217 INFO:tasks.rados.rados.0.smithi123.stdout:1156: expect (ObjNum 256 snap 0 seq_num 256) 2020-01-16T00:13:17.217 INFO:tasks.rados.rados.0.smithi123.stdout:1157: read oid 2 snap -1 2020-01-16T00:13:17.218 INFO:tasks.rados.rados.0.smithi123.stdout:1157: expect (ObjNum 291 snap 0 seq_num 291) 2020-01-16T00:13:17.218 INFO:tasks.rados.rados.0.smithi123.stdout:1158: read oid 41 snap -1 2020-01-16T00:13:17.218 INFO:tasks.rados.rados.0.smithi123.stdout:1158: expect (ObjNum 310 snap 0 seq_num 310) 2020-01-16T00:13:17.218 INFO:tasks.rados.rados.0.smithi123.stdout:1159: write (excl) oid 9 current snap is 0 2020-01-16T00:13:17.218 INFO:tasks.rados.rados.0.smithi123.stdout:1159: seq_num 333 ranges {796731=461096,1736739=723770,2928680=668051} 2020-01-16T00:13:17.221 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:17.222 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:13:17.222 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:13:17.222 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:17.231 INFO:tasks.rados.rados.0.smithi123.stdout:1159: writing smithi12313934-9 from 796731 to 1257827 tid 1 2020-01-16T00:13:17.233 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-01-16T00:13:17.238 INFO:tasks.rados.rados.0.smithi123.stdout:1159: writing smithi12313934-9 from 1736739 to 2460509 tid 2 2020-01-16T00:13:17.245 INFO:tasks.rados.rados.0.smithi123.stdout:1159: writing smithi12313934-9 from 2928680 to 3596731 tid 3 2020-01-16T00:13:17.245 INFO:tasks.rados.rados.0.smithi123.stdout:1160: write oid 42 current snap is 0 2020-01-16T00:13:17.245 INFO:tasks.rados.rados.0.smithi123.stdout:1160: seq_num 334 ranges {413538=536345,1395221=583781,2556103=658617} 2020-01-16T00:13:17.250 INFO:tasks.rados.rados.0.smithi123.stdout:1160: writing smithi12313934-42 from 413538 to 949883 tid 1 2020-01-16T00:13:17.256 INFO:tasks.rados.rados.0.smithi123.stdout:1160: writing smithi12313934-42 from 1395221 to 1979002 tid 2 2020-01-16T00:13:17.264 INFO:tasks.rados.rados.0.smithi123.stdout:1160: writing smithi12313934-42 from 2556103 to 3214720 tid 3 2020-01-16T00:13:17.264 INFO:tasks.rados.rados.0.smithi123.stdout:1161: write (excl) oid 45 current snap is 0 2020-01-16T00:13:17.264 INFO:tasks.rados.rados.0.smithi123.stdout:1161: seq_num 335 ranges {430345=727947,1570056=727439,2699879=469889} 2020-01-16T00:13:17.270 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-01-16T00:13:17.271 INFO:tasks.rados.rados.0.smithi123.stdout:1161: writing smithi12313934-45 from 430345 to 1158292 tid 1 2020-01-16T00:13:17.277 INFO:tasks.rados.rados.0.smithi123.stdout:1161: writing smithi12313934-45 from 1570056 to 2297495 tid 2 2020-01-16T00:13:17.282 INFO:tasks.rados.rados.0.smithi123.stdout:1161: writing smithi12313934-45 from 2699879 to 3169768 tid 3 2020-01-16T00:13:17.282 INFO:tasks.rados.rados.0.smithi123.stdout:1162: write oid 47 current snap is 0 2020-01-16T00:13:17.282 INFO:tasks.rados.rados.0.smithi123.stdout:1162: seq_num 336 ranges {447152=403196,1512185=587450,2727302=681161} 2020-01-16T00:13:17.287 INFO:tasks.rados.rados.0.smithi123.stdout:1162: writing smithi12313934-47 from 447152 to 850348 tid 1 2020-01-16T00:13:17.294 INFO:tasks.rados.rados.0.smithi123.stdout:1162: writing smithi12313934-47 from 1512185 to 2099635 tid 2 2020-01-16T00:13:17.302 INFO:tasks.rados.rados.0.smithi123.stdout:1162: writing smithi12313934-47 from 2727302 to 3408463 tid 3 2020-01-16T00:13:17.302 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:17.302 INFO:tasks.rados.rados.0.smithi123.stdout:1148: expect (ObjNum 188 snap 0 seq_num 188) 2020-01-16T00:13:17.304 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:17.302+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:17.338 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:17.338 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:13:17.338 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:13:17.338 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:13:17.338 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:17.339 INFO:teuthology.orchestra.run.smithi145.stderr:Exporting 1.1 info 1.1( empty local-lis/les=15/16 n=0 ec=15/15 lis/c=15/15 les/c/f=16/16/0 sis=15) 2020-01-16T00:13:17.339 INFO:teuthology.orchestra.run.smithi145.stderr:Export successful 2020-01-16T00:13:17.347 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-01-16T00:13:17.386 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:13:17.400 INFO:tasks.rados.rados.0.smithi123.stdout:1147: finishing write tid 3 to smithi12313934-11 2020-01-16T00:13:17.400 INFO:tasks.rados.rados.0.smithi123.stdout:1147: finishing write tid 5 to smithi12313934-11 2020-01-16T00:13:17.400 INFO:tasks.rados.rados.0.smithi123.stdout:1147: finishing write tid 6 to smithi12313934-11 2020-01-16T00:13:17.400 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 11 v 97 (ObjNum 329 snap 0 seq_num 329) dirty exists 2020-01-16T00:13:17.400 INFO:tasks.rados.rados.0.smithi123.stdout:1147: left oid 11 (ObjNum 329 snap 0 seq_num 329) 2020-01-16T00:13:17.401 INFO:tasks.rados.rados.0.smithi123.stdout:1149: expect (ObjNum 317 snap 0 seq_num 317) 2020-01-16T00:13:17.403 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:13:17.405 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:17.400+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:17.422 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:13:17.442 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:13:17.449 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:17.449 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:13:17.450 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:13:17.450 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:13:17.450 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.450 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:895 3.3 3:cd231094:::smithi12313934-49:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:17.450 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.122390+0000", 2020-01-16T00:13:17.450 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 40.322033066000003, 2020-01-16T00:13:17.451 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014295501, 2020-01-16T00:13:17.451 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:17.451 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:17.451 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:17.451 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:17.451 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:17.451 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 895 2020-01-16T00:13:17.451 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.452 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:17.452 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.452 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:17.452 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122390+0000", 2020-01-16T00:13:17.452 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:17.452 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.452 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.453 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:17.453 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122390+0000", 2020-01-16T00:13:17.453 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999957 2020-01-16T00:13:17.453 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.453 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.453 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:17.454 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122386+0000", 2020-01-16T00:13:17.454 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0656999999999999e-05 2020-01-16T00:13:17.454 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.454 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.454 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:17.454 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122407+0000", 2020-01-16T00:13:17.454 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9102999999999999e-05 2020-01-16T00:13:17.454 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.455 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.455 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:17.455 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122436+0000", 2020-01-16T00:13:17.455 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.774e-05 2020-01-16T00:13:17.455 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.455 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.456 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:17.456 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122483+0000", 2020-01-16T00:13:17.456 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5185000000000004e-05 2020-01-16T00:13:17.456 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.456 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.456 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.456 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122579+0000", 2020-01-16T00:13:17.457 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000304085 2020-01-16T00:13:17.457 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.457 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.457 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:17.457 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122883+0000", 2020-01-16T00:13:17.457 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0092077970000000002 2020-01-16T00:13:17.457 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.457 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.458 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.458 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132091+0000", 2020-01-16T00:13:17.458 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000170025 2020-01-16T00:13:17.458 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.458 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.458 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:17.458 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132261+0000", 2020-01-16T00:13:17.459 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0040130979999999997 2020-01-16T00:13:17.459 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.459 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.459 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.459 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136274+0000", 2020-01-16T00:13:17.459 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014227900000000001 2020-01-16T00:13:17.459 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.459 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.460 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:17.460 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136416+0000", 2020-01-16T00:13:17.460 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00026995200000000001 2020-01-16T00:13:17.460 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.460 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.460 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:17.460 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136686+0000", 2020-01-16T00:13:17.460 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014295501 2020-01-16T00:13:17.461 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.461 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:17.461 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.461 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.461 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.461 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:896 3.3 3:cd231094:::smithi12313934-49:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:17.461 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.122539+0000", 2020-01-16T00:13:17.462 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 40.321884066000003, 2020-01-16T00:13:17.462 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014440173000000001, 2020-01-16T00:13:17.462 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:17.462 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:17.462 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:17.462 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:17.462 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:17.462 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 896 2020-01-16T00:13:17.463 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.463 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:17.463 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.465 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:17.465 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122539+0000", 2020-01-16T00:13:17.465 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:17.465 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.465 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.466 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:17.466 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122539+0000", 2020-01-16T00:13:17.466 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:17.466 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.466 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.466 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:17.466 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122537+0000", 2020-01-16T00:13:17.466 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2432999999999999e-05 2020-01-16T00:13:17.467 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.467 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.467 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:17.467 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122550+0000", 2020-01-16T00:13:17.467 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9369999999999999e-05 2020-01-16T00:13:17.467 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.467 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.467 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:17.468 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122569+0000", 2020-01-16T00:13:17.468 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1747e-05 2020-01-16T00:13:17.468 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.468 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.468 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:17.468 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122581+0000", 2020-01-16T00:13:17.468 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00041555800000000001 2020-01-16T00:13:17.468 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.469 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.469 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.469 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122996+0000", 2020-01-16T00:13:17.469 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016840299999999999 2020-01-16T00:13:17.469 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.469 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.469 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:17.470 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.123165+0000", 2020-01-16T00:13:17.470 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0091641359999999998 2020-01-16T00:13:17.470 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.470 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.470 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.471 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132329+0000", 2020-01-16T00:13:17.471 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013433400000000001 2020-01-16T00:13:17.471 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.471 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.471 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:17.471 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132463+0000", 2020-01-16T00:13:17.471 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0042524110000000002 2020-01-16T00:13:17.471 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.472 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.472 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.472 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136716+0000", 2020-01-16T00:13:17.472 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4591999999999993e-05 2020-01-16T00:13:17.472 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.472 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.472 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:17.472 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136790+0000", 2020-01-16T00:13:17.473 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018927500000000001 2020-01-16T00:13:17.473 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.473 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.473 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:17.473 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136980+0000", 2020-01-16T00:13:17.473 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014440173000000001 2020-01-16T00:13:17.473 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.473 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:17.474 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.474 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.474 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.474 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:897 3.3 3:cd231094:::smithi12313934-49:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:17.474 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.122613+0000", 2020-01-16T00:13:17.474 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 40.321810921999997, 2020-01-16T00:13:17.474 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014606846, 2020-01-16T00:13:17.475 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:17.475 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:17.475 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:17.475 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:17.475 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:17.476 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 897 2020-01-16T00:13:17.476 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.476 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:17.476 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.476 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:17.476 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122613+0000", 2020-01-16T00:13:17.476 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:17.477 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.477 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.477 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:17.477 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122613+0000", 2020-01-16T00:13:17.477 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:17.477 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.477 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.477 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:17.478 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122611+0000", 2020-01-16T00:13:17.478 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3670000000000003e-06 2020-01-16T00:13:17.478 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.478 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.478 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:17.478 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122620+0000", 2020-01-16T00:13:17.478 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5193e-05 2020-01-16T00:13:17.479 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.479 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.479 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:17.479 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122635+0000", 2020-01-16T00:13:17.479 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0696999999999999e-05 2020-01-16T00:13:17.479 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.479 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.479 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:17.480 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122646+0000", 2020-01-16T00:13:17.480 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00059961400000000005 2020-01-16T00:13:17.480 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.480 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.480 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.480 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.123246+0000", 2020-01-16T00:13:17.480 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011189200000000001 2020-01-16T00:13:17.480 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.481 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.481 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:17.481 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.123358+0000", 2020-01-16T00:13:17.481 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0091363550000000005 2020-01-16T00:13:17.481 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.481 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.481 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.482 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132494+0000", 2020-01-16T00:13:17.482 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2956999999999998e-05 2020-01-16T00:13:17.482 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.482 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.482 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:17.482 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132587+0000", 2020-01-16T00:13:17.482 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0044061769999999998 2020-01-16T00:13:17.482 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.483 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.483 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.483 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136993+0000", 2020-01-16T00:13:17.483 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7822999999999998e-05 2020-01-16T00:13:17.483 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.483 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.483 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:17.483 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.137051+0000", 2020-01-16T00:13:17.484 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000168498 2020-01-16T00:13:17.484 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.484 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.484 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:17.484 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.137219+0000", 2020-01-16T00:13:17.484 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014606846 2020-01-16T00:13:17.484 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.484 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:17.485 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.485 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.485 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.485 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:943 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:17.488 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:13:17.488 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 39.529944948000001, 2020-01-16T00:13:17.488 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018847004000000001, 2020-01-16T00:13:17.488 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:17.488 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:17.489 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:17.489 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:17.489 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:17.489 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 943 2020-01-16T00:13:17.489 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.490 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:17.490 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.490 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:17.490 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:13:17.490 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:17.490 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.490 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.490 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:17.491 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:13:17.491 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:17.491 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.491 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.491 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:17.491 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914475+0000", 2020-01-16T00:13:17.491 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.508e-05 2020-01-16T00:13:17.492 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.492 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.492 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:17.492 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914490+0000", 2020-01-16T00:13:17.492 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.857e-05 2020-01-16T00:13:17.492 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.492 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.492 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:17.493 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914509+0000", 2020-01-16T00:13:17.493 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.46e-06 2020-01-16T00:13:17.493 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.493 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.493 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:17.493 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914515+0000", 2020-01-16T00:13:17.493 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2612999999999999e-05 2020-01-16T00:13:17.494 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.494 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.494 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.494 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914568+0000", 2020-01-16T00:13:17.494 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028312299999999998 2020-01-16T00:13:17.494 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.494 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.494 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:17.495 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914851+0000", 2020-01-16T00:13:17.495 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012282533 2020-01-16T00:13:17.495 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.495 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.495 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.495 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927133+0000", 2020-01-16T00:13:17.496 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4839999999999995e-05 2020-01-16T00:13:17.496 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.496 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.496 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:17.496 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927208+0000", 2020-01-16T00:13:17.496 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0047425610000000002 2020-01-16T00:13:17.496 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.496 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.497 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.497 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.931951+0000", 2020-01-16T00:13:17.497 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3628999999999994e-05 2020-01-16T00:13:17.497 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.497 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.497 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:17.498 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.932045+0000", 2020-01-16T00:13:17.498 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012810180000000001 2020-01-16T00:13:17.498 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.498 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.498 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:17.498 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933326+0000", 2020-01-16T00:13:17.498 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018847004000000001 2020-01-16T00:13:17.499 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.499 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:17.499 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.499 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.499 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.499 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:944 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:17.499 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:13:17.500 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 39.529895392, 2020-01-16T00:13:17.500 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019086901999999999, 2020-01-16T00:13:17.500 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:17.500 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:17.500 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:17.500 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:17.500 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:17.501 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 944 2020-01-16T00:13:17.501 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.501 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:17.501 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.502 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:17.502 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:13:17.502 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:17.502 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.502 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.502 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:17.502 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:13:17.503 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:17.503 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.503 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.503 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:17.503 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914527+0000", 2020-01-16T00:13:17.503 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7509999999999999e-06 2020-01-16T00:13:17.504 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.504 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.504 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:17.504 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914532+0000", 2020-01-16T00:13:17.504 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5149999999999997e-06 2020-01-16T00:13:17.504 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.504 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.505 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:17.505 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914538+0000", 2020-01-16T00:13:17.505 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1890000000000007e-06 2020-01-16T00:13:17.505 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.505 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.505 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:17.506 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914546+0000", 2020-01-16T00:13:17.506 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000354951 2020-01-16T00:13:17.506 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.506 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.506 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.506 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914901+0000", 2020-01-16T00:13:17.506 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1594000000000002e-05 2020-01-16T00:13:17.507 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.507 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.507 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:17.507 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914942+0000", 2020-01-16T00:13:17.507 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012300070999999999 2020-01-16T00:13:17.507 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.507 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.507 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.508 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927242+0000", 2020-01-16T00:13:17.508 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4076999999999997e-05 2020-01-16T00:13:17.508 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.508 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.508 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:17.508 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927296+0000", 2020-01-16T00:13:17.508 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0060609180000000002 2020-01-16T00:13:17.509 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.509 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.509 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.509 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933357+0000", 2020-01-16T00:13:17.509 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4488999999999999e-05 2020-01-16T00:13:17.509 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.509 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.510 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:17.510 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933422+0000", 2020-01-16T00:13:17.510 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019318300000000001 2020-01-16T00:13:17.510 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.510 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.510 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:17.511 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933615+0000", 2020-01-16T00:13:17.511 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019086901999999999 2020-01-16T00:13:17.511 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.511 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:17.511 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.511 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.511 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.512 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:945 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:17.512 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:13:17.512 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 39.529868186000002, 2020-01-16T00:13:17.512 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019342986, 2020-01-16T00:13:17.512 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:17.512 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:17.512 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:17.512 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:17.513 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:17.513 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 945 2020-01-16T00:13:17.513 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.513 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:17.513 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.513 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:17.513 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:13:17.513 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:17.514 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.514 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.514 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:17.514 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:13:17.514 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:17.514 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.514 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.515 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:17.515 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:13:17.515 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4580000000000002e-06 2020-01-16T00:13:17.515 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.515 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.515 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:17.515 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914558+0000", 2020-01-16T00:13:17.515 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7829999999999998e-06 2020-01-16T00:13:17.516 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.516 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.516 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:17.516 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914563+0000", 2020-01-16T00:13:17.516 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0001e-05 2020-01-16T00:13:17.516 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.517 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.517 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:17.517 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914573+0000", 2020-01-16T00:13:17.517 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00040365600000000001 2020-01-16T00:13:17.517 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.517 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.517 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.518 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914976+0000", 2020-01-16T00:13:17.518 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1390000000000001e-05 2020-01-16T00:13:17.518 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.518 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.518 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:17.518 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.915028+0000", 2020-01-16T00:13:17.518 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012288843000000001 2020-01-16T00:13:17.518 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.519 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.519 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.519 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927317+0000", 2020-01-16T00:13:17.519 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7871000000000002e-05 2020-01-16T00:13:17.519 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.519 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.519 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:17.520 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927375+0000", 2020-01-16T00:13:17.520 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0062580589999999998 2020-01-16T00:13:17.520 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.520 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.520 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.520 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933633+0000", 2020-01-16T00:13:17.521 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7086999999999997e-05 2020-01-16T00:13:17.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.521 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.521 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:17.521 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933690+0000", 2020-01-16T00:13:17.521 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020855799999999999 2020-01-16T00:13:17.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.521 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.522 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:17.522 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933898+0000", 2020-01-16T00:13:17.522 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019342986 2020-01-16T00:13:17.522 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.522 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:17.522 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.522 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.523 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.523 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:963 3.9 3:9557b633:::smithi12313934-43:head [read 0~2705894,checksum] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:17.523 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.046589+0000", 2020-01-16T00:13:17.523 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 39.397834338000003, 2020-01-16T00:13:17.523 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014014016000000001, 2020-01-16T00:13:17.523 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:17.523 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:17.523 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:17.524 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:17.524 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:17.524 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 963 2020-01-16T00:13:17.524 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.524 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:17.524 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.524 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:17.524 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046589+0000", 2020-01-16T00:13:17.525 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:17.525 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.525 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.525 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:17.525 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046589+0000", 2020-01-16T00:13:17.525 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:17.525 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.526 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.526 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:17.526 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046588+0000", 2020-01-16T00:13:17.526 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7469999999999994e-06 2020-01-16T00:13:17.526 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.526 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.527 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:17.527 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046596+0000", 2020-01-16T00:13:17.527 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3445999999999998e-05 2020-01-16T00:13:17.527 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.527 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.527 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:17.527 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046630+0000", 2020-01-16T00:13:17.527 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7779999999999999e-05 2020-01-16T00:13:17.528 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.528 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.528 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:17.528 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046648+0000", 2020-01-16T00:13:17.528 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00075081300000000002 2020-01-16T00:13:17.528 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.528 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.528 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.529 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.047398+0000", 2020-01-16T00:13:17.529 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000116698 2020-01-16T00:13:17.529 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.529 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.529 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:17.529 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.047515+0000", 2020-01-16T00:13:17.529 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0056630689999999997 2020-01-16T00:13:17.529 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.530 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.530 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.530 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.053178+0000", 2020-01-16T00:13:17.530 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012084299999999999 2020-01-16T00:13:17.530 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.530 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.531 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:17.531 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.053299+0000", 2020-01-16T00:13:17.531 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0056921510000000003 2020-01-16T00:13:17.531 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.531 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.531 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.531 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.058991+0000", 2020-01-16T00:13:17.531 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000118606 2020-01-16T00:13:17.532 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.532 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.532 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:17.532 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.059110+0000", 2020-01-16T00:13:17.532 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001493306 2020-01-16T00:13:17.532 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.532 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.533 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:17.533 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.060603+0000", 2020-01-16T00:13:17.533 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014014016000000001 2020-01-16T00:13:17.533 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.533 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:17.533 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.533 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.534 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.534 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:964 3.9 3:9557b633:::smithi12313934-43:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:17.534 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.046668+0000", 2020-01-16T00:13:17.534 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 39.397755734, 2020-01-16T00:13:17.534 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014425670999999999, 2020-01-16T00:13:17.534 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:17.534 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:17.535 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:17.535 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:17.535 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:17.535 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 964 2020-01-16T00:13:17.535 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.535 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:17.535 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.535 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:17.536 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046668+0000", 2020-01-16T00:13:17.536 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:17.536 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.536 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.536 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:17.536 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046668+0000", 2020-01-16T00:13:17.537 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:17.537 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.537 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.537 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:17.537 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046666+0000", 2020-01-16T00:13:17.537 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0879999999999999e-05 2020-01-16T00:13:17.537 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.538 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.538 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:17.538 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046677+0000", 2020-01-16T00:13:17.538 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5767e-05 2020-01-16T00:13:17.538 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.538 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.538 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:17.538 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046693+0000", 2020-01-16T00:13:17.539 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1399999999999999e-05 2020-01-16T00:13:17.539 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.539 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.539 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:17.539 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046704+0000", 2020-01-16T00:13:17.539 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00089331800000000004 2020-01-16T00:13:17.539 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.539 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.540 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.540 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.047597+0000", 2020-01-16T00:13:17.540 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8628000000000005e-05 2020-01-16T00:13:17.540 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.540 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.540 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:17.541 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.047696+0000", 2020-01-16T00:13:17.541 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0056536290000000003 2020-01-16T00:13:17.541 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.541 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.541 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.541 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.053349+0000", 2020-01-16T00:13:17.541 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.6946000000000003e-05 2020-01-16T00:13:17.541 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.542 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.542 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:17.542 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.053446+0000", 2020-01-16T00:13:17.542 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0072058909999999999 2020-01-16T00:13:17.542 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.542 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.542 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.543 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.060652+0000", 2020-01-16T00:13:17.543 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000135597 2020-01-16T00:13:17.543 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.543 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.543 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:17.543 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.060788+0000", 2020-01-16T00:13:17.543 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030548499999999998 2020-01-16T00:13:17.543 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.544 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.544 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:17.544 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.061093+0000", 2020-01-16T00:13:17.544 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014425670999999999 2020-01-16T00:13:17.544 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.544 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:17.544 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.544 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.545 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.545 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:981 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:17.545 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:13:17.545 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 39.129091293999998, 2020-01-16T00:13:17.545 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021222494000000001, 2020-01-16T00:13:17.545 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:17.545 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:17.546 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:17.546 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:17.546 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:17.546 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 981 2020-01-16T00:13:17.546 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.546 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:17.547 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.547 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:17.547 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:13:17.547 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:17.547 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.547 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.547 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:17.547 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:13:17.548 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999948 2020-01-16T00:13:17.548 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.548 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.548 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:17.548 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315327+0000", 2020-01-16T00:13:17.548 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.508e-05 2020-01-16T00:13:17.548 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.548 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.549 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:17.549 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315352+0000", 2020-01-16T00:13:17.549 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1183999999999998e-05 2020-01-16T00:13:17.549 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.549 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.549 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:17.549 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315383+0000", 2020-01-16T00:13:17.550 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7180000000000002e-05 2020-01-16T00:13:17.550 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.550 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.550 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:17.550 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315401+0000", 2020-01-16T00:13:17.550 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017290599999999999 2020-01-16T00:13:17.550 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.551 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.551 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.551 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315574+0000", 2020-01-16T00:13:17.551 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00047908100000000002 2020-01-16T00:13:17.551 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.551 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.551 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:17.552 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316053+0000", 2020-01-16T00:13:17.552 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011808947 2020-01-16T00:13:17.552 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.552 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.552 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.552 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.327862+0000", 2020-01-16T00:13:17.552 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010067499999999999 2020-01-16T00:13:17.552 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.553 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.553 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:17.553 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.327962+0000", 2020-01-16T00:13:17.553 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0074745139999999998 2020-01-16T00:13:17.553 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.553 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.553 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.554 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.335437+0000", 2020-01-16T00:13:17.554 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000103403 2020-01-16T00:13:17.554 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.554 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.554 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:17.554 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.335540+0000", 2020-01-16T00:13:17.554 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001014544 2020-01-16T00:13:17.554 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.555 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.555 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:17.555 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336555+0000", 2020-01-16T00:13:17.555 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021222494000000001 2020-01-16T00:13:17.555 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.555 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:17.555 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.555 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.556 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.556 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:982 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:17.556 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:13:17.556 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 39.128988769999999, 2020-01-16T00:13:17.556 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.022087721000000001, 2020-01-16T00:13:17.556 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:17.556 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:17.556 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:17.557 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:17.557 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:17.557 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 982 2020-01-16T00:13:17.557 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.557 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:17.557 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.558 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:17.558 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:13:17.558 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:17.558 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.558 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.558 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:17.558 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:13:17.558 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:17.559 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.559 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.559 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:17.559 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315433+0000", 2020-01-16T00:13:17.560 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1347e-05 2020-01-16T00:13:17.560 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.560 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.560 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:17.560 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315445+0000", 2020-01-16T00:13:17.560 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4525999999999999e-05 2020-01-16T00:13:17.560 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.560 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.561 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:17.561 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315459+0000", 2020-01-16T00:13:17.561 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.024e-05 2020-01-16T00:13:17.561 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.561 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.562 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:17.562 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315470+0000", 2020-01-16T00:13:17.562 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000736468 2020-01-16T00:13:17.562 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.562 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.562 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.562 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316206+0000", 2020-01-16T00:13:17.562 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000132967 2020-01-16T00:13:17.563 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.563 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.563 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:17.563 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316339+0000", 2020-01-16T00:13:17.563 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011701717 2020-01-16T00:13:17.563 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.563 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.563 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.564 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328041+0000", 2020-01-16T00:13:17.564 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000120339 2020-01-16T00:13:17.564 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.564 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.564 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:17.564 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328161+0000", 2020-01-16T00:13:17.564 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0084341260000000001 2020-01-16T00:13:17.565 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.565 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.565 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.565 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336595+0000", 2020-01-16T00:13:17.565 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5247999999999993e-05 2020-01-16T00:13:17.565 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.565 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.565 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:17.566 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336690+0000", 2020-01-16T00:13:17.566 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00083204000000000004 2020-01-16T00:13:17.566 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.566 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.566 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:17.566 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337522+0000", 2020-01-16T00:13:17.566 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.022087721000000001 2020-01-16T00:13:17.566 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.567 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:17.567 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.567 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.567 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.567 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:983 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:17.567 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:13:17.568 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 39.12886906, 2020-01-16T00:13:17.568 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023363387999999999, 2020-01-16T00:13:17.568 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:17.568 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:17.568 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:17.568 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:17.568 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:17.568 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 983 2020-01-16T00:13:17.569 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.569 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:17.569 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.569 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:17.569 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:13:17.569 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:17.569 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.570 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.570 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:17.570 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:13:17.570 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:17.570 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.570 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.570 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:17.570 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315553+0000", 2020-01-16T00:13:17.571 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1760000000000004e-06 2020-01-16T00:13:17.571 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.571 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.571 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:17.572 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315560+0000", 2020-01-16T00:13:17.572 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3587000000000001e-05 2020-01-16T00:13:17.572 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.572 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.572 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:17.572 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315574+0000", 2020-01-16T00:13:17.572 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7800000000000003e-06 2020-01-16T00:13:17.572 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.573 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.573 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:17.573 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315581+0000", 2020-01-16T00:13:17.573 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000823024 2020-01-16T00:13:17.573 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.573 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.573 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.573 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316404+0000", 2020-01-16T00:13:17.574 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2413e-05 2020-01-16T00:13:17.574 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.574 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.574 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:17.574 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316466+0000", 2020-01-16T00:13:17.574 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011738561999999999 2020-01-16T00:13:17.574 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.574 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.575 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.575 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328205+0000", 2020-01-16T00:13:17.575 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6783999999999997e-05 2020-01-16T00:13:17.575 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.575 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.575 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:17.575 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328262+0000", 2020-01-16T00:13:17.576 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0093022560000000001 2020-01-16T00:13:17.576 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.576 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.576 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.576 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337564+0000", 2020-01-16T00:13:17.576 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000183151 2020-01-16T00:13:17.576 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.576 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.577 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:17.577 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337747+0000", 2020-01-16T00:13:17.577 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011707449999999999 2020-01-16T00:13:17.577 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.577 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.577 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:17.578 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.338918+0000", 2020-01-16T00:13:17.578 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023363387999999999 2020-01-16T00:13:17.578 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.578 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:17.578 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.578 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.578 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.578 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:986 3.9 3:93dfe6f1:::smithi12313934-40:head [write 2941380~419987] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:17.579 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:13:17.579 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 38.963569284000002, 2020-01-16T00:13:17.579 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016978400000000001, 2020-01-16T00:13:17.579 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:17.579 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:17.579 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:17.579 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:17.580 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:17.580 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 986 2020-01-16T00:13:17.580 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.580 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:17.580 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.580 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:17.580 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:13:17.580 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:17.581 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.581 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.581 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:17.581 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:13:17.581 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:17.581 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.582 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.582 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:17.582 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480852+0000", 2020-01-16T00:13:17.582 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042503899999999999 2020-01-16T00:13:17.582 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.582 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.582 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:17.582 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481277+0000", 2020-01-16T00:13:17.583 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3097e-05 2020-01-16T00:13:17.583 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.583 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.583 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:17.583 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481290+0000", 2020-01-16T00:13:17.583 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8460000000000002e-06 2020-01-16T00:13:17.583 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.583 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.584 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:17.584 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481297+0000", 2020-01-16T00:13:17.584 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8443999999999999e-05 2020-01-16T00:13:17.584 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.584 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.584 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.584 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481346+0000", 2020-01-16T00:13:17.585 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022115399999999999 2020-01-16T00:13:17.585 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.585 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.585 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:17.585 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481567+0000", 2020-01-16T00:13:17.585 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012531502 2020-01-16T00:13:17.585 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.585 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.586 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.586 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494098+0000", 2020-01-16T00:13:17.586 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.4462999999999993e-05 2020-01-16T00:13:17.586 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.586 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.586 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:17.586 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494193+0000", 2020-01-16T00:13:17.586 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017600999999999999 2020-01-16T00:13:17.587 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.587 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.587 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:13:17.587 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494369+0000", 2020-01-16T00:13:17.587 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0031739619999999998 2020-01-16T00:13:17.587 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.588 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.588 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:17.588 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497543+0000", 2020-01-16T00:13:17.588 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000175107 2020-01-16T00:13:17.588 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.588 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.588 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:17.589 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497718+0000", 2020-01-16T00:13:17.589 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5207000000000003e-05 2020-01-16T00:13:17.589 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.589 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.589 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:17.589 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497763+0000", 2020-01-16T00:13:17.589 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7042999999999998e-05 2020-01-16T00:13:17.589 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.590 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.590 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:17.590 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497800+0000", 2020-01-16T00:13:17.590 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2628e-05 2020-01-16T00:13:17.590 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.590 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.590 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:17.590 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497833+0000", 2020-01-16T00:13:17.591 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016978400000000001 2020-01-16T00:13:17.591 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.591 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:17.591 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.591 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.591 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.592 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:987 3.9 3:93dfe6f1:::smithi12313934-40:head [setxattr _header (56) in=7b,truncate 3361367] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:17.592 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.481307+0000", 2020-01-16T00:13:17.592 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 38.963116917000001, 2020-01-16T00:13:17.592 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016794989, 2020-01-16T00:13:17.592 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:17.592 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:17.592 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:17.593 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:17.593 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:17.593 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 987 2020-01-16T00:13:17.593 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.593 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:17.593 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.593 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:17.593 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481307+0000", 2020-01-16T00:13:17.594 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:17.594 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.594 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.594 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:17.594 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481307+0000", 2020-01-16T00:13:17.594 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:17.594 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.594 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.595 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:17.595 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481306+0000", 2020-01-16T00:13:17.595 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.113e-06 2020-01-16T00:13:17.595 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.595 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.595 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:17.595 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481311+0000", 2020-01-16T00:13:17.595 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1520000000000002e-06 2020-01-16T00:13:17.596 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.596 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.596 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:17.596 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481315+0000", 2020-01-16T00:13:17.596 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.7640000000000001e-06 2020-01-16T00:13:17.596 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.596 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.596 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:17.597 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481318+0000", 2020-01-16T00:13:17.597 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00029012500000000002 2020-01-16T00:13:17.597 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.597 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.597 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.597 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481608+0000", 2020-01-16T00:13:17.597 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6239000000000002e-05 2020-01-16T00:13:17.598 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.598 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.598 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:17.598 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481654+0000", 2020-01-16T00:13:17.598 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014559534000000001 2020-01-16T00:13:17.598 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.598 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.599 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.599 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.496214+0000", 2020-01-16T00:13:17.599 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000225932 2020-01-16T00:13:17.599 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.599 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.599 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:17.599 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.496439+0000", 2020-01-16T00:13:17.600 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00031099600000000002 2020-01-16T00:13:17.600 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.600 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.600 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:13:17.600 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.496750+0000", 2020-01-16T00:13:17.600 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00098608599999999995 2020-01-16T00:13:17.601 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.601 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.601 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:17.601 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497737+0000", 2020-01-16T00:13:17.602 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022081499999999999 2020-01-16T00:13:17.602 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.602 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.602 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:17.602 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497957+0000", 2020-01-16T00:13:17.602 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7071000000000003e-05 2020-01-16T00:13:17.602 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.603 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.603 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:17.603 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498014+0000", 2020-01-16T00:13:17.603 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1111e-05 2020-01-16T00:13:17.603 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.603 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.603 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:17.604 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498056+0000", 2020-01-16T00:13:17.604 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5974999999999999e-05 2020-01-16T00:13:17.604 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.604 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.604 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:17.604 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498102+0000", 2020-01-16T00:13:17.604 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016794989 2020-01-16T00:13:17.605 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.605 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:17.605 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.605 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.605 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.605 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:988 3.9 3:93dfe6f1:::smithi12313934-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:13:17.605 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:13:17.606 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 38.963083820000001, 2020-01-16T00:13:17.606 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017221692, 2020-01-16T00:13:17.606 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:17.606 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:17.606 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:17.606 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:17.606 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:17.606 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 988 2020-01-16T00:13:17.607 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.607 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:17.607 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.607 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:17.607 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:13:17.607 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:17.607 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.607 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.608 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:17.608 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:13:17.608 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:17.608 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.608 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.608 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:17.609 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481339+0000", 2020-01-16T00:13:17.609 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3089999999999997e-06 2020-01-16T00:13:17.609 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.609 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.609 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:17.609 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481343+0000", 2020-01-16T00:13:17.609 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2230000000000004e-06 2020-01-16T00:13:17.610 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.610 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.610 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:17.610 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481350+0000", 2020-01-16T00:13:17.610 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.319e-06 2020-01-16T00:13:17.610 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.610 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.610 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:17.611 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481356+0000", 2020-01-16T00:13:17.611 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000321544 2020-01-16T00:13:17.611 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.611 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.611 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.611 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481678+0000", 2020-01-16T00:13:17.611 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6375999999999999e-05 2020-01-16T00:13:17.611 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.612 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.612 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:17.612 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481714+0000", 2020-01-16T00:13:17.612 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015557666 2020-01-16T00:13:17.612 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.612 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.612 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.612 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497272+0000", 2020-01-16T00:13:17.613 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012791300000000001 2020-01-16T00:13:17.613 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.613 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.613 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:17.613 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497399+0000", 2020-01-16T00:13:17.613 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00076944600000000002 2020-01-16T00:13:17.614 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.614 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.614 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.614 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498169+0000", 2020-01-16T00:13:17.614 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018306799999999999 2020-01-16T00:13:17.614 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.614 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.615 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:17.615 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498352+0000", 2020-01-16T00:13:17.615 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020935999999999999 2020-01-16T00:13:17.615 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.615 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.615 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:17.615 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498561+0000", 2020-01-16T00:13:17.615 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017221692 2020-01-16T00:13:17.616 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.616 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:17.616 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.616 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.616 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.616 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1208 3.3 3:c478bfe8:::smithi12313934-10:head [write 749612~474179,stat] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:13:17.616 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.350952+0000", 2020-01-16T00:13:17.616 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 35.093470983000003, 2020-01-16T00:13:17.617 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014967729000000001, 2020-01-16T00:13:17.617 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:17.617 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:17.617 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:17.617 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:17.617 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:17.617 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1208 2020-01-16T00:13:17.618 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.618 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:17.618 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.618 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:17.618 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.350952+0000", 2020-01-16T00:13:17.618 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:17.619 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.619 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.620 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:17.620 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.350952+0000", 2020-01-16T00:13:17.620 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:13:17.621 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.621 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.621 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:17.621 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.350949+0000", 2020-01-16T00:13:17.621 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042894 2020-01-16T00:13:17.621 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.622 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.622 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:17.622 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.351377+0000", 2020-01-16T00:13:17.622 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3921e-05 2020-01-16T00:13:17.622 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.622 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.622 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:17.622 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.351401+0000", 2020-01-16T00:13:17.623 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1087000000000001e-05 2020-01-16T00:13:17.623 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.623 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.623 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:17.623 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.351412+0000", 2020-01-16T00:13:17.623 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8127000000000001e-05 2020-01-16T00:13:17.623 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.624 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.624 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.624 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.351471+0000", 2020-01-16T00:13:17.624 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000230268 2020-01-16T00:13:17.624 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.624 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.624 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:17.625 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.351701+0000", 2020-01-16T00:13:17.625 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0051228369999999999 2020-01-16T00:13:17.625 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.625 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.625 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.625 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.356824+0000", 2020-01-16T00:13:17.625 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015984800000000001 2020-01-16T00:13:17.626 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.626 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.626 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:17.626 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.356984+0000", 2020-01-16T00:13:17.626 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0038163770000000001 2020-01-16T00:13:17.626 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.627 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.627 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.627 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.360800+0000", 2020-01-16T00:13:17.627 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.9878999999999999e-05 2020-01-16T00:13:17.627 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.627 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.628 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:17.628 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.360880+0000", 2020-01-16T00:13:17.628 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000145041 2020-01-16T00:13:17.628 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.628 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.628 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2020-01-16T00:13:17.628 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.361025+0000", 2020-01-16T00:13:17.628 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0020743039999999999 2020-01-16T00:13:17.629 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.629 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.629 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:17.629 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.363099+0000", 2020-01-16T00:13:17.629 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00072876500000000003 2020-01-16T00:13:17.629 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.629 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.629 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:17.630 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.363828+0000", 2020-01-16T00:13:17.630 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00189153 2020-01-16T00:13:17.630 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.630 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.630 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:17.630 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.365719+0000", 2020-01-16T00:13:17.630 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3948000000000005e-05 2020-01-16T00:13:17.631 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.631 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.631 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:17.631 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.365783+0000", 2020-01-16T00:13:17.631 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000136833 2020-01-16T00:13:17.631 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.631 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.631 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:17.632 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.365920+0000", 2020-01-16T00:13:17.632 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014967729000000001 2020-01-16T00:13:17.632 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.632 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:17.632 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.632 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.632 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.632 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1237 3.3 3:c204aa4f:::smithi12313934-45:head [write 400033~699926] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:13:17.633 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.584946+0000", 2020-01-16T00:13:17.633 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 34.859477101000003, 2020-01-16T00:13:17.633 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014287855, 2020-01-16T00:13:17.633 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:17.633 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:17.634 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:17.634 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:17.634 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:17.634 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1237 2020-01-16T00:13:17.634 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.635 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:17.635 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.635 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:17.635 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.584946+0000", 2020-01-16T00:13:17.635 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:17.635 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.636 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.636 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:17.636 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.584946+0000", 2020-01-16T00:13:17.636 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999752 2020-01-16T00:13:17.636 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.636 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.636 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:17.636 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.584921+0000", 2020-01-16T00:13:17.637 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00077243400000000001 2020-01-16T00:13:17.637 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.637 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.637 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:17.637 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.585694+0000", 2020-01-16T00:13:17.637 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3083000000000001e-05 2020-01-16T00:13:17.637 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.637 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.638 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:17.638 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.585737+0000", 2020-01-16T00:13:17.638 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1854000000000001e-05 2020-01-16T00:13:17.638 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.638 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.638 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:17.638 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.585759+0000", 2020-01-16T00:13:17.638 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016127000000000001 2020-01-16T00:13:17.639 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.639 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.639 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.639 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.585920+0000", 2020-01-16T00:13:17.639 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024934700000000002 2020-01-16T00:13:17.639 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.640 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.640 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:17.640 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.586169+0000", 2020-01-16T00:13:17.640 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00036508999999999999 2020-01-16T00:13:17.640 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.641 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.641 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2020-01-16T00:13:17.641 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.586534+0000", 2020-01-16T00:13:17.641 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012168910999999999 2020-01-16T00:13:17.642 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.642 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.642 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:17.642 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598703+0000", 2020-01-16T00:13:17.642 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5302000000000004e-05 2020-01-16T00:13:17.642 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.642 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.642 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:17.643 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598789+0000", 2020-01-16T00:13:17.643 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00027301099999999998 2020-01-16T00:13:17.643 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.643 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.643 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:17.643 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599062+0000", 2020-01-16T00:13:17.643 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.4237e-05 2020-01-16T00:13:17.644 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.644 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.644 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:17.644 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599146+0000", 2020-01-16T00:13:17.644 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.8319000000000006e-05 2020-01-16T00:13:17.644 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.644 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.645 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:17.645 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599234+0000", 2020-01-16T00:13:17.645 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014287855 2020-01-16T00:13:17.645 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.645 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:17.645 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.645 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.646 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.646 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1239 3.3 3:c204aa4f:::smithi12313934-45:head [setxattr _header (56) in=7b,truncate 2000033] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:13:17.646 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:13:17.646 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 34.854010000999999, 2020-01-16T00:13:17.646 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019161661999999999, 2020-01-16T00:13:17.646 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:17.646 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:17.647 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:17.647 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:17.648 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:17.648 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1239 2020-01-16T00:13:17.648 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.648 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:17.649 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.649 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:17.649 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:13:17.649 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:17.649 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.649 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.649 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:17.650 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:13:17.650 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:17.650 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.650 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.650 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:17.650 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590411+0000", 2020-01-16T00:13:17.651 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3879999999999999e-05 2020-01-16T00:13:17.651 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.651 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.651 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:17.651 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590425+0000", 2020-01-16T00:13:17.651 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6542999999999998e-05 2020-01-16T00:13:17.651 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.652 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.652 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:17.652 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590471+0000", 2020-01-16T00:13:17.652 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8907999999999999e-05 2020-01-16T00:13:17.652 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.652 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.652 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:17.652 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590490+0000", 2020-01-16T00:13:17.653 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0045691960000000002 2020-01-16T00:13:17.653 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.653 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.653 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.653 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.595059+0000", 2020-01-16T00:13:17.653 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0035638940000000002 2020-01-16T00:13:17.654 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.654 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.654 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:17.654 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598623+0000", 2020-01-16T00:13:17.654 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0077558310000000004 2020-01-16T00:13:17.654 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.654 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.656 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.656 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606379+0000", 2020-01-16T00:13:17.656 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012171999999999999 2020-01-16T00:13:17.656 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.656 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.656 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:17.657 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606501+0000", 2020-01-16T00:13:17.657 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013144200000000001 2020-01-16T00:13:17.657 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.657 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.657 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2020-01-16T00:13:17.657 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606632+0000", 2020-01-16T00:13:17.657 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001288565 2020-01-16T00:13:17.658 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.658 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.658 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:17.658 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607921+0000", 2020-01-16T00:13:17.658 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033399299999999999 2020-01-16T00:13:17.658 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.658 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.659 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:17.659 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.608255+0000", 2020-01-16T00:13:17.659 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012280870000000001 2020-01-16T00:13:17.659 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.659 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.659 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:17.659 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609483+0000", 2020-01-16T00:13:17.659 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2286000000000001e-05 2020-01-16T00:13:17.660 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.660 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.660 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:17.660 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609525+0000", 2020-01-16T00:13:17.660 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0121999999999999e-05 2020-01-16T00:13:17.660 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.661 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.661 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:17.661 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609575+0000", 2020-01-16T00:13:17.661 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019161661999999999 2020-01-16T00:13:17.661 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.661 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:17.661 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.662 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.662 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.662 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1240 3.3 3:c204aa4f:::smithi12313934-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-01-16T00:13:17.662 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:13:17.662 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 34.853909811999998, 2020-01-16T00:13:17.662 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019268828000000002, 2020-01-16T00:13:17.662 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:17.662 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:17.663 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:17.663 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:17.663 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:17.663 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1240 2020-01-16T00:13:17.663 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.663 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:17.663 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.663 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:17.664 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:13:17.664 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:17.664 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.664 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.664 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:17.664 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:13:17.664 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:17.664 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.665 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.665 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:17.665 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590512+0000", 2020-01-16T00:13:17.665 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0525e-05 2020-01-16T00:13:17.665 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.665 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.665 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:17.666 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590522+0000", 2020-01-16T00:13:17.667 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6418000000000001e-05 2020-01-16T00:13:17.667 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.667 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.668 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:17.668 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590539+0000", 2020-01-16T00:13:17.669 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0679999999999999e-05 2020-01-16T00:13:17.669 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.669 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.669 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:17.669 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590549+0000", 2020-01-16T00:13:17.669 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0083626789999999996 2020-01-16T00:13:17.669 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.670 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.670 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.670 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598912+0000", 2020-01-16T00:13:17.670 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013344 2020-01-16T00:13:17.670 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.670 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.670 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:17.670 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599045+0000", 2020-01-16T00:13:17.671 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0084002269999999997 2020-01-16T00:13:17.671 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.671 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.671 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.671 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607446+0000", 2020-01-16T00:13:17.671 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000109873 2020-01-16T00:13:17.671 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.671 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.672 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:17.672 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607555+0000", 2020-01-16T00:13:17.672 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0020573309999999999 2020-01-16T00:13:17.672 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.672 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.672 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.672 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609613+0000", 2020-01-16T00:13:17.672 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6351e-05 2020-01-16T00:13:17.673 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.673 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.673 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:17.673 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609699+0000", 2020-01-16T00:13:17.673 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3374999999999994e-05 2020-01-16T00:13:17.673 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.673 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.673 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:17.674 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609782+0000", 2020-01-16T00:13:17.674 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019268828000000002 2020-01-16T00:13:17.674 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.674 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:17.674 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.674 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.674 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.675 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1436 3.3 3:c204aa4f:::smithi12313934-45:head [setxattr _header (56) in=7b,truncate 2386594] snapc 0=[] ondisk+write+known_if_redirected e28)", 2020-01-16T00:13:17.675 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:47.900544+0000", 2020-01-16T00:13:17.675 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 29.543879131000001, 2020-01-16T00:13:17.675 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016402521, 2020-01-16T00:13:17.675 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:17.675 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:17.676 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:17.676 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:17.676 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:17.676 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1436 2020-01-16T00:13:17.676 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.676 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:17.676 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.677 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:17.677 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900544+0000", 2020-01-16T00:13:17.677 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:17.677 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.677 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.677 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:17.677 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900544+0000", 2020-01-16T00:13:17.677 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:17.678 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.678 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.678 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:17.678 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900542+0000", 2020-01-16T00:13:17.678 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3862999999999999e-05 2020-01-16T00:13:17.678 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.678 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.679 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:17.679 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900556+0000", 2020-01-16T00:13:17.679 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5204e-05 2020-01-16T00:13:17.679 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.679 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.679 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:17.679 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900571+0000", 2020-01-16T00:13:17.680 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.092e-05 2020-01-16T00:13:17.680 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.680 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.680 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:17.680 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900582+0000", 2020-01-16T00:13:17.680 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002786026 2020-01-16T00:13:17.680 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.680 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.681 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.681 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.903368+0000", 2020-01-16T00:13:17.681 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0027123659999999999 2020-01-16T00:13:17.681 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.681 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.681 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:17.681 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.906081+0000", 2020-01-16T00:13:17.681 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0082707790000000007 2020-01-16T00:13:17.682 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.682 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.682 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.682 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.914352+0000", 2020-01-16T00:13:17.682 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020580300000000001 2020-01-16T00:13:17.682 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.682 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.682 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:17.683 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.914557+0000", 2020-01-16T00:13:17.683 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017729900000000001 2020-01-16T00:13:17.683 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.683 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.683 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2020-01-16T00:13:17.683 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.914735+0000", 2020-01-16T00:13:17.683 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00074632099999999999 2020-01-16T00:13:17.683 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.684 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.684 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:17.684 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.915481+0000", 2020-01-16T00:13:17.684 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012279599999999999 2020-01-16T00:13:17.684 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.684 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.684 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:17.684 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.915604+0000", 2020-01-16T00:13:17.685 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012619339999999999 2020-01-16T00:13:17.685 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.685 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.685 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:17.685 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.916866+0000", 2020-01-16T00:13:17.685 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1162000000000003e-05 2020-01-16T00:13:17.685 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.686 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.686 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:17.686 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.916907+0000", 2020-01-16T00:13:17.686 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9987999999999997e-05 2020-01-16T00:13:17.686 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.686 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.686 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:17.687 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.916947+0000", 2020-01-16T00:13:17.687 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016402521 2020-01-16T00:13:17.687 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.687 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:17.687 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.687 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.687 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.687 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1437 3.3 3:c204aa4f:::smithi12313934-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e28)", 2020-01-16T00:13:17.688 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:47.900594+0000", 2020-01-16T00:13:17.688 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 29.543829268, 2020-01-16T00:13:17.688 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016745237999999999, 2020-01-16T00:13:17.688 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:17.688 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:17.688 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:17.688 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:17.688 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:17.689 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1437 2020-01-16T00:13:17.689 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.689 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:17.689 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.689 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:17.689 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900594+0000", 2020-01-16T00:13:17.689 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:17.689 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.690 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.690 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:17.690 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900594+0000", 2020-01-16T00:13:17.690 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:17.690 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.690 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.690 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:17.690 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900593+0000", 2020-01-16T00:13:17.691 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5059999999999995e-06 2020-01-16T00:13:17.691 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.691 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.691 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:17.691 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900601+0000", 2020-01-16T00:13:17.691 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5077e-05 2020-01-16T00:13:17.691 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.691 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.692 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:17.692 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900616+0000", 2020-01-16T00:13:17.692 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0183000000000001e-05 2020-01-16T00:13:17.692 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.692 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.692 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:17.692 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900627+0000", 2020-01-16T00:13:17.692 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0063667660000000003 2020-01-16T00:13:17.693 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.693 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.693 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.693 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.906993+0000", 2020-01-16T00:13:17.693 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8988e-05 2020-01-16T00:13:17.693 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.693 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.694 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:17.694 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.907032+0000", 2020-01-16T00:13:17.694 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0080128850000000008 2020-01-16T00:13:17.694 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.694 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.694 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.694 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.915045+0000", 2020-01-16T00:13:17.694 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013200899999999999 2020-01-16T00:13:17.695 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.695 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.695 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:17.695 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.915177+0000", 2020-01-16T00:13:17.695 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0018373090000000001 2020-01-16T00:13:17.695 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.695 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.695 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:17.696 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.917015+0000", 2020-01-16T00:13:17.696 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016701699999999999 2020-01-16T00:13:17.696 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.696 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.696 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:17.697 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.917182+0000", 2020-01-16T00:13:17.697 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015782400000000001 2020-01-16T00:13:17.697 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:17.697 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:17.697 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:17.697 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.917339+0000", 2020-01-16T00:13:17.697 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016745237999999999 2020-01-16T00:13:17.698 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.698 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:17.698 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.698 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:17.698 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:17.698 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:17.700 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:13:17.700 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:13:17.701 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:13:17.701 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-01-16T00:13:17.701 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-01-16T00:13:17.701 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-01-16T00:13:17.701 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-01-16T00:13:17.701 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-01-16T00:13:17.701 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENXIO: problem getting command descriptions from osd.5 2020-01-16T00:13:17.702 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: problem getting command descriptions from osd.5 2020-01-16T00:13:17.702 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-01-16T00:13:17.702 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-01-16T00:13:17.702 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-01-16T00:13:17.702 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-01-16T00:13:17.703 INFO:tasks.rados.rados.0.smithi123.stdout:1150: finishing write tid 1 to smithi12313934-33 2020-01-16T00:13:17.703 INFO:tasks.rados.rados.0.smithi123.stdout:1151: expect (ObjNum 311 snap 0 seq_num 311) 2020-01-16T00:13:17.703 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:17.696+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:17.703 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-01-16T00:13:17.801 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:17.798+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:17.848 INFO:tasks.rados.rados.0.smithi123.stdout:1150: finishing write tid 2 to smithi12313934-33 2020-01-16T00:13:17.848 INFO:tasks.rados.rados.0.smithi123.stdout:1150: finishing write tid 3 to smithi12313934-33 2020-01-16T00:13:17.848 INFO:tasks.rados.rados.0.smithi123.stdout:1150: finishing write tid 5 to smithi12313934-33 2020-01-16T00:13:17.848 INFO:tasks.rados.rados.0.smithi123.stdout:1150: finishing write tid 6 to smithi12313934-33 2020-01-16T00:13:17.848 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 33 v 233 (ObjNum 330 snap 0 seq_num 330) dirty exists 2020-01-16T00:13:17.848 INFO:tasks.rados.rados.0.smithi123.stdout:1150: left oid 33 (ObjNum 330 snap 0 seq_num 330) 2020-01-16T00:13:17.848 INFO:tasks.rados.rados.0.smithi123.stdout:1152: finishing write tid 1 to smithi12313934-48 2020-01-16T00:13:17.849 INFO:tasks.rados.rados.0.smithi123.stdout:1152: finishing write tid 2 to smithi12313934-48 2020-01-16T00:13:17.849 INFO:tasks.rados.rados.0.smithi123.stdout:1153: expect (ObjNum 304 snap 0 seq_num 304) 2020-01-16T00:13:17.860 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:17.870 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-01-16T00:13:17.876 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2020-01-16T00:13:17.881 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --force --op remove --pgid 1.1 2020-01-16T00:13:17.923 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:17.920+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:17.978 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:17.988 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-01-16T00:13:18.002 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:17.998+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:18.038 INFO:tasks.rados.rados.0.smithi123.stdout:1152: finishing write tid 3 to smithi12313934-48 2020-01-16T00:13:18.038 INFO:tasks.rados.rados.0.smithi123.stdout:1152: finishing write tid 5 to smithi12313934-48 2020-01-16T00:13:18.038 INFO:tasks.rados.rados.0.smithi123.stdout:1152: finishing write tid 6 to smithi12313934-48 2020-01-16T00:13:18.039 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 48 v 122 (ObjNum 331 snap 0 seq_num 331) dirty exists 2020-01-16T00:13:18.039 INFO:tasks.rados.rados.0.smithi123.stdout:1152: left oid 48 (ObjNum 331 snap 0 seq_num 331) 2020-01-16T00:13:18.039 INFO:tasks.rados.rados.0.smithi123.stdout:1154: finishing write tid 1 to smithi12313934-25 2020-01-16T00:13:18.039 INFO:tasks.rados.rados.0.smithi123.stdout:1154: finishing write tid 2 to smithi12313934-25 2020-01-16T00:13:18.039 INFO:tasks.rados.rados.0.smithi123.stdout:1154: finishing write tid 3 to smithi12313934-25 2020-01-16T00:13:18.039 INFO:tasks.rados.rados.0.smithi123.stdout:1154: finishing write tid 5 to smithi12313934-25 2020-01-16T00:13:18.039 INFO:tasks.rados.rados.0.smithi123.stdout:1154: finishing write tid 6 to smithi12313934-25 2020-01-16T00:13:18.040 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 25 v 171 (ObjNum 332 snap 0 seq_num 332) dirty exists 2020-01-16T00:13:18.040 INFO:tasks.rados.rados.0.smithi123.stdout:1154: left oid 25 (ObjNum 332 snap 0 seq_num 332) 2020-01-16T00:13:18.040 INFO:tasks.rados.rados.0.smithi123.stdout:1155: expect (ObjNum 298 snap 0 seq_num 298) 2020-01-16T00:13:18.086 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:18.095 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-01-16T00:13:18.139 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:18.136+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:18.189 INFO:tasks.rados.rados.0.smithi123.stdout:1156: expect (ObjNum 256 snap 0 seq_num 256) 2020-01-16T00:13:18.200 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:18.204 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:18.200+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:18.210 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-01-16T00:13:18.306 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:18.300+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:18.333 INFO:tasks.rados.rados.0.smithi123.stdout:1157: expect (ObjNum 291 snap 0 seq_num 291) 2020-01-16T00:13:18.361 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:18.370 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-01-16T00:13:18.424 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:18.418+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:18.522 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:18.518+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:18.526 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:18.536 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-01-16T00:13:18.585 INFO:tasks.rados.rados.0.smithi123.stdout:1159: finishing write tid 1 to smithi12313934-9 2020-01-16T00:13:18.585 INFO:tasks.rados.rados.0.smithi123.stdout:1158: expect (ObjNum 310 snap 0 seq_num 310) 2020-01-16T00:13:18.623 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:18.618+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:18.641 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:18.650 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-01-16T00:13:18.736 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:18.734+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:18.779 INFO:tasks.rados.rados.0.smithi123.stdout:1159: finishing write tid 2 to smithi12313934-9 2020-01-16T00:13:18.779 INFO:tasks.rados.rados.0.smithi123.stdout:1159: finishing write tid 3 to smithi12313934-9 2020-01-16T00:13:18.779 INFO:tasks.rados.rados.0.smithi123.stdout:1159: finishing write tid 5 to smithi12313934-9 2020-01-16T00:13:18.779 INFO:tasks.rados.rados.0.smithi123.stdout:1159: finishing write tid 6 to smithi12313934-9 2020-01-16T00:13:18.779 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 9 v 113 (ObjNum 333 snap 0 seq_num 333) dirty exists 2020-01-16T00:13:18.779 INFO:tasks.rados.rados.0.smithi123.stdout:1159: left oid 9 (ObjNum 333 snap 0 seq_num 333) 2020-01-16T00:13:18.780 INFO:tasks.rados.rados.0.smithi123.stdout:1160: finishing write tid 1 to smithi12313934-42 2020-01-16T00:13:18.780 INFO:tasks.rados.rados.0.smithi123.stdout:1160: finishing write tid 2 to smithi12313934-42 2020-01-16T00:13:18.780 INFO:tasks.rados.rados.0.smithi123.stdout:1160: finishing write tid 3 to smithi12313934-42 2020-01-16T00:13:18.780 INFO:tasks.rados.rados.0.smithi123.stdout:1147: done (15 left) 2020-01-16T00:13:18.780 INFO:tasks.rados.rados.0.smithi123.stdout:1148: done (14 left) 2020-01-16T00:13:18.780 INFO:tasks.rados.rados.0.smithi123.stdout:1149: done (13 left) 2020-01-16T00:13:18.780 INFO:tasks.rados.rados.0.smithi123.stdout:1150: done (12 left) 2020-01-16T00:13:18.781 INFO:tasks.rados.rados.0.smithi123.stdout:1151: done (11 left) 2020-01-16T00:13:18.781 INFO:tasks.rados.rados.0.smithi123.stdout:1152: done (10 left) 2020-01-16T00:13:18.781 INFO:tasks.rados.rados.0.smithi123.stdout:1153: done (9 left) 2020-01-16T00:13:18.781 INFO:tasks.rados.rados.0.smithi123.stdout:1154: done (8 left) 2020-01-16T00:13:18.781 INFO:tasks.rados.rados.0.smithi123.stdout:1155: done (7 left) 2020-01-16T00:13:18.781 INFO:tasks.rados.rados.0.smithi123.stdout:1156: done (6 left) 2020-01-16T00:13:18.781 INFO:tasks.rados.rados.0.smithi123.stdout:1157: done (5 left) 2020-01-16T00:13:18.781 INFO:tasks.rados.rados.0.smithi123.stdout:1158: done (4 left) 2020-01-16T00:13:18.782 INFO:tasks.rados.rados.0.smithi123.stdout:1159: done (3 left) 2020-01-16T00:13:18.783 INFO:tasks.rados.rados.0.smithi123.stdout:1163: read oid 29 snap -1 2020-01-16T00:13:18.783 INFO:tasks.rados.rados.0.smithi123.stdout:1163: expect (ObjNum 321 snap 0 seq_num 321) 2020-01-16T00:13:18.783 INFO:tasks.rados.rados.0.smithi123.stdout:1164: write (excl) oid 16 current snap is 0 2020-01-16T00:13:18.783 INFO:tasks.rados.rados.0.smithi123.stdout:1164: seq_num 337 ranges {463959=478445,1570667=731108,3038372=492433} 2020-01-16T00:13:18.790 INFO:tasks.rados.rados.0.smithi123.stdout:1164: writing smithi12313934-16 from 463959 to 942404 tid 1 2020-01-16T00:13:18.799 INFO:tasks.rados.rados.0.smithi123.stdout:1164: writing smithi12313934-16 from 1570667 to 2301775 tid 2 2020-01-16T00:13:18.799 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:18.806 INFO:tasks.rados.rados.0.smithi123.stdout:1164: writing smithi12313934-16 from 3038372 to 3530805 tid 3 2020-01-16T00:13:18.806 INFO:tasks.rados.rados.0.smithi123.stdout:1165: write (excl) oid 36 current snap is 0 2020-01-16T00:13:18.806 INFO:tasks.rados.rados.0.smithi123.stdout:1165: seq_num 338 ranges {480766=553694,1629149=591119,2782148=420058} 2020-01-16T00:13:18.808 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-01-16T00:13:18.812 INFO:tasks.rados.rados.0.smithi123.stdout:1165: writing smithi12313934-36 from 480766 to 1034460 tid 1 2020-01-16T00:13:18.818 INFO:teuthology.orchestra.run.smithi145.stdout: marking collection for removal 2020-01-16T00:13:18.818 INFO:teuthology.orchestra.run.smithi145.stdout:setting '_remove' omap key 2020-01-16T00:13:18.819 INFO:tasks.rados.rados.0.smithi123.stdout:1165: writing smithi12313934-36 from 1629149 to 2220268 tid 2 2020-01-16T00:13:18.819 INFO:teuthology.orchestra.run.smithi145.stdout:finish_remove_pgs 1.1_head removing 1.1 2020-01-16T00:13:18.819 INFO:teuthology.orchestra.run.smithi145.stdout:Remove successful 2020-01-16T00:13:18.825 INFO:tasks.rados.rados.0.smithi123.stdout:1165: writing smithi12313934-36 from 2782148 to 3202206 tid 3 2020-01-16T00:13:18.825 INFO:tasks.rados.rados.0.smithi123.stdout:1166: read oid 30 snap -1 2020-01-16T00:13:18.825 INFO:tasks.rados.rados.0.smithi123.stdout:1166: expect (ObjNum 294 snap 0 seq_num 294) 2020-01-16T00:13:18.826 INFO:tasks.rados.rados.0.smithi123.stdout:1167: write oid 13 current snap is 0 2020-01-16T00:13:18.826 INFO:tasks.rados.rados.0.smithi123.stdout:1167: seq_num 339 ranges {497573=628943,1687631=734777,3093218=604355} 2020-01-16T00:13:18.826 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:18.820+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:18.831 INFO:tasks.rados.rados.0.smithi123.stdout:1167: writing smithi12313934-13 from 497573 to 1126516 tid 1 2020-01-16T00:13:18.838 INFO:tasks.rados.rados.0.smithi123.stdout:1167: writing smithi12313934-13 from 1687631 to 2422408 tid 2 2020-01-16T00:13:18.846 INFO:tasks.rados.rados.0.smithi123.stdout:1167: writing smithi12313934-13 from 3093218 to 3697573 tid 3 2020-01-16T00:13:18.846 INFO:tasks.rados.rados.0.smithi123.stdout:1168: write oid 6 current snap is 0 2020-01-16T00:13:18.846 INFO:tasks.rados.rados.0.smithi123.stdout:1168: seq_num 340 ranges {514380=704192,1629760=478435,2604288=442602} 2020-01-16T00:13:18.853 INFO:tasks.rados.rados.0.smithi123.stdout:1168: writing smithi12313934-6 from 514380 to 1218572 tid 1 2020-01-16T00:13:18.858 INFO:tasks.rados.rados.0.smithi123.stdout:1168: writing smithi12313934-6 from 1629760 to 2108195 tid 2 2020-01-16T00:13:18.864 INFO:tasks.rados.rados.0.smithi123.stdout:1168: writing smithi12313934-6 from 2604288 to 3046890 tid 3 2020-01-16T00:13:18.864 INFO:tasks.rados.rados.0.smithi123.stdout:1169: write oid 31 current snap is 0 2020-01-16T00:13:18.864 INFO:tasks.rados.rados.0.smithi123.stdout:1169: seq_num 341 ranges {531187=779441,2088242=738446,3431711=299476} 2020-01-16T00:13:18.872 INFO:tasks.rados.rados.0.smithi123.stdout:1169: writing smithi12313934-31 from 531187 to 1310628 tid 1 2020-01-16T00:13:18.880 INFO:teuthology.orchestra.run.smithi123.stderr:noscrub is set 2020-01-16T00:13:18.881 INFO:tasks.rados.rados.0.smithi123.stdout:1169: writing smithi12313934-31 from 2088242 to 2826688 tid 2 2020-01-16T00:13:18.887 INFO:tasks.rados.rados.0.smithi123.stdout:1169: writing smithi12313934-31 from 3431711 to 3731187 tid 3 2020-01-16T00:13:18.887 INFO:tasks.rados.rados.0.smithi123.stdout:1170: write (excl) oid 19 current snap is 0 2020-01-16T00:13:18.887 INFO:tasks.rados.rados.0.smithi123.stdout:1170: seq_num 342 ranges {547994=454690,1746724=482104,2659134=465146,3605976=142018} 2020-01-16T00:13:18.892 INFO:tasks.rados.rados.0.smithi123.stdout:1170: writing smithi12313934-19 from 547994 to 1002684 tid 1 2020-01-16T00:13:18.901 INFO:tasks.rados.rados.0.smithi123.stdout:1170: writing smithi12313934-19 from 1746724 to 2228828 tid 2 2020-01-16T00:13:18.904 INFO:tasks.rados.rados.0.smithi123.stdout:1170: writing smithi12313934-19 from 2659134 to 3124280 tid 3 2020-01-16T00:13:18.909 INFO:tasks.rados.rados.0.smithi123.stdout:1170: writing smithi12313934-19 from 3605976 to 3747994 tid 4 2020-01-16T00:13:18.909 INFO:tasks.rados.rados.0.smithi123.stdout:1171: read oid 14 snap -1 2020-01-16T00:13:18.909 INFO:tasks.rados.rados.0.smithi123.stdout:1171: expect (ObjNum 327 snap 0 seq_num 327) 2020-01-16T00:13:18.909 INFO:tasks.rados.rados.0.smithi123.stdout:1172: read oid 37 snap -1 2020-01-16T00:13:18.909 INFO:tasks.rados.rados.0.smithi123.stdout:1172: expect (ObjNum 285 snap 0 seq_num 285) 2020-01-16T00:13:18.909 INFO:tasks.rados.rados.0.smithi123.stdout:1173: write oid 5 current snap is 0 2020-01-16T00:13:18.910 INFO:tasks.rados.rados.0.smithi123.stdout:1173: seq_num 343 ranges {564801=646292,1921559=742115,3202910=561891} 2020-01-16T00:13:18.915 INFO:tasks.rados.rados.0.smithi123.stdout:1173: writing smithi12313934-5 from 564801 to 1211093 tid 1 2020-01-16T00:13:18.923 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:18.923 INFO:tasks.rados.rados.0.smithi123.stdout:1173: writing smithi12313934-5 from 1921559 to 2663674 tid 2 2020-01-16T00:13:18.925 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:18.920+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:18.930 INFO:tasks.rados.rados.0.smithi123.stdout:1173: writing smithi12313934-5 from 3202910 to 3764801 tid 3 2020-01-16T00:13:18.930 INFO:tasks.rados.rados.0.smithi123.stdout:1174: write oid 32 current snap is 0 2020-01-16T00:13:18.930 INFO:tasks.rados.rados.0.smithi123.stdout:1174: seq_num 344 ranges {581608=721541,1863688=485773,3113980=604043} 2020-01-16T00:13:18.933 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-01-16T00:13:18.937 INFO:tasks.rados.rados.0.smithi123.stdout:1174: writing smithi12313934-32 from 581608 to 1303149 tid 1 2020-01-16T00:13:18.943 INFO:tasks.rados.rados.0.smithi123.stdout:1174: writing smithi12313934-32 from 1863688 to 2349461 tid 2 2020-01-16T00:13:18.951 INFO:tasks.rados.rados.0.smithi123.stdout:1174: writing smithi12313934-32 from 3113980 to 3718023 tid 3 2020-01-16T00:13:18.951 INFO:tasks.rados.rados.0.smithi123.stdout:1175: write oid 21 current snap is 0 2020-01-16T00:13:18.951 INFO:tasks.rados.rados.0.smithi123.stdout:1175: seq_num 345 ranges {598415=796790,1922170=745784,3141403=415315} 2020-01-16T00:13:18.959 INFO:tasks.rados.rados.0.smithi123.stdout:1175: writing smithi12313934-21 from 598415 to 1395205 tid 1 2020-01-16T00:13:18.966 INFO:tasks.rados.rados.0.smithi123.stdout:1175: writing smithi12313934-21 from 1922170 to 2667954 tid 2 2020-01-16T00:13:18.972 INFO:tasks.rados.rados.0.smithi123.stdout:1175: writing smithi12313934-21 from 3141403 to 3556718 tid 3 2020-01-16T00:13:18.972 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:18.972 INFO:tasks.rados.rados.0.smithi123.stdout:1161: finishing write tid 1 to smithi12313934-45 2020-01-16T00:13:18.972 INFO:tasks.rados.rados.0.smithi123.stdout:1160: finishing write tid 5 to smithi12313934-42 2020-01-16T00:13:18.973 INFO:tasks.rados.rados.0.smithi123.stdout:1160: finishing write tid 6 to smithi12313934-42 2020-01-16T00:13:18.973 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 42 v 127 (ObjNum 334 snap 0 seq_num 334) dirty exists 2020-01-16T00:13:18.973 INFO:tasks.rados.rados.0.smithi123.stdout:1160: left oid 42 (ObjNum 334 snap 0 seq_num 334) 2020-01-16T00:13:18.973 INFO:tasks.rados.rados.0.smithi123.stdout:1161: finishing write tid 2 to smithi12313934-45 2020-01-16T00:13:18.973 INFO:tasks.rados.rados.0.smithi123.stdout:1160: done (15 left) 2020-01-16T00:13:18.973 INFO:tasks.rados.rados.0.smithi123.stdout:1176: read oid 11 snap -1 2020-01-16T00:13:18.973 INFO:tasks.rados.rados.0.smithi123.stdout:1176: expect (ObjNum 329 snap 0 seq_num 329) 2020-01-16T00:13:18.974 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:18.974 INFO:tasks.rados.rados.0.smithi123.stdout:1162: finishing write tid 1 to smithi12313934-47 2020-01-16T00:13:18.974 INFO:tasks.rados.rados.0.smithi123.stdout:1161: finishing write tid 3 to smithi12313934-45 2020-01-16T00:13:18.974 INFO:tasks.rados.rados.0.smithi123.stdout:1161: finishing write tid 5 to smithi12313934-45 2020-01-16T00:13:18.974 INFO:tasks.rados.rados.0.smithi123.stdout:1161: finishing write tid 6 to smithi12313934-45 2020-01-16T00:13:18.974 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 45 v 122 (ObjNum 335 snap 0 seq_num 335) dirty exists 2020-01-16T00:13:18.975 INFO:tasks.rados.rados.0.smithi123.stdout:1161: left oid 45 (ObjNum 335 snap 0 seq_num 335) 2020-01-16T00:13:18.975 INFO:tasks.rados.rados.0.smithi123.stdout:1162: finishing write tid 2 to smithi12313934-47 2020-01-16T00:13:18.975 INFO:tasks.rados.rados.0.smithi123.stdout:1161: done (15 left) 2020-01-16T00:13:18.975 INFO:tasks.rados.rados.0.smithi123.stdout:1177: read oid 3 snap -1 2020-01-16T00:13:18.975 INFO:tasks.rados.rados.0.smithi123.stdout:1177: expect (ObjNum 260 snap 0 seq_num 260) 2020-01-16T00:13:18.975 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:18.975 INFO:tasks.rados.rados.0.smithi123.stdout:1162: finishing write tid 3 to smithi12313934-47 2020-01-16T00:13:18.976 INFO:tasks.rados.rados.0.smithi123.stdout:1162: finishing write tid 5 to smithi12313934-47 2020-01-16T00:13:18.976 INFO:tasks.rados.rados.0.smithi123.stdout:1162: finishing write tid 6 to smithi12313934-47 2020-01-16T00:13:18.976 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 47 v 175 (ObjNum 336 snap 0 seq_num 336) dirty exists 2020-01-16T00:13:18.976 INFO:tasks.rados.rados.0.smithi123.stdout:1162: left oid 47 (ObjNum 336 snap 0 seq_num 336) 2020-01-16T00:13:18.976 INFO:tasks.rados.rados.0.smithi123.stdout:1162: done (15 left) 2020-01-16T00:13:18.976 INFO:tasks.rados.rados.0.smithi123.stdout:1178: read oid 23 snap -1 2020-01-16T00:13:18.977 INFO:tasks.rados.rados.0.smithi123.stdout:1178: expect deleted 2020-01-16T00:13:18.977 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:18.977 INFO:tasks.rados.rados.0.smithi123.stdout:1163: expect (ObjNum 321 snap 0 seq_num 321) 2020-01-16T00:13:19.027 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:19.022+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:19.034 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:19.049 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-01-16T00:13:19.136 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:19.134+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:19.195 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:19.205 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-01-16T00:13:19.216 INFO:tasks.rados.rados.0.smithi123.stdout:1164: finishing write tid 1 to smithi12313934-16 2020-01-16T00:13:19.216 INFO:tasks.rados.rados.0.smithi123.stdout:1164: finishing write tid 2 to smithi12313934-16 2020-01-16T00:13:19.216 INFO:tasks.rados.rados.0.smithi123.stdout:1164: finishing write tid 3 to smithi12313934-16 2020-01-16T00:13:19.216 INFO:tasks.rados.rados.0.smithi123.stdout:1164: finishing write tid 5 to smithi12313934-16 2020-01-16T00:13:19.217 INFO:tasks.rados.rados.0.smithi123.stdout:1164: finishing write tid 6 to smithi12313934-16 2020-01-16T00:13:19.217 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 16 v 179 (ObjNum 337 snap 0 seq_num 337) dirty exists 2020-01-16T00:13:19.217 INFO:tasks.rados.rados.0.smithi123.stdout:1164: left oid 16 (ObjNum 337 snap 0 seq_num 337) 2020-01-16T00:13:19.217 INFO:tasks.rados.rados.0.smithi123.stdout:1165: finishing write tid 1 to smithi12313934-36 2020-01-16T00:13:19.217 INFO:tasks.rados.rados.0.smithi123.stdout:1165: finishing write tid 2 to smithi12313934-36 2020-01-16T00:13:19.217 INFO:tasks.rados.rados.0.smithi123.stdout:1165: finishing write tid 3 to smithi12313934-36 2020-01-16T00:13:19.218 INFO:tasks.rados.rados.0.smithi123.stdout:1165: finishing write tid 5 to smithi12313934-36 2020-01-16T00:13:19.218 INFO:tasks.rados.rados.0.smithi123.stdout:1165: finishing write tid 6 to smithi12313934-36 2020-01-16T00:13:19.218 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 36 v 66 (ObjNum 338 snap 0 seq_num 338) dirty exists 2020-01-16T00:13:19.218 INFO:tasks.rados.rados.0.smithi123.stdout:1165: left oid 36 (ObjNum 338 snap 0 seq_num 338) 2020-01-16T00:13:19.218 INFO:tasks.rados.rados.0.smithi123.stdout:1163: done (15 left) 2020-01-16T00:13:19.218 INFO:tasks.rados.rados.0.smithi123.stdout:1164: done (14 left) 2020-01-16T00:13:19.218 INFO:tasks.rados.rados.0.smithi123.stdout:1165: done (13 left) 2020-01-16T00:13:19.219 INFO:tasks.rados.rados.0.smithi123.stdout:1179: read oid 8 snap -1 2020-01-16T00:13:19.219 INFO:tasks.rados.rados.0.smithi123.stdout:1179: expect (ObjNum 219 snap 0 seq_num 219) 2020-01-16T00:13:19.219 INFO:tasks.rados.rados.0.smithi123.stdout:1180: read oid 38 snap -1 2020-01-16T00:13:19.219 INFO:tasks.rados.rados.0.smithi123.stdout:1180: expect (ObjNum 298 snap 0 seq_num 298) 2020-01-16T00:13:19.219 INFO:tasks.rados.rados.0.smithi123.stdout:1181: delete oid 34 current snap is 0 2020-01-16T00:13:19.219 INFO:tasks.rados.rados.0.smithi123.stdout:1166: expect (ObjNum 294 snap 0 seq_num 294) 2020-01-16T00:13:19.229 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:19.224+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:19.330 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:19.331 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:19.324+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:19.333 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-01-16T00:13:19.356 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --log-file '/var/log/ceph/objectstore_tool.\$pid.log' --op import --file /var/log/ceph/exp.1.1.5 2020-01-16T00:13:19.430 INFO:teuthology.orchestra.run.smithi145.stderr:Can't specify both --op and object command syntax 2020-01-16T00:13:19.430 INFO:teuthology.orchestra.run.smithi145.stderr: 2020-01-16T00:13:19.430 INFO:teuthology.orchestra.run.smithi145.stderr:Allowed options: 2020-01-16T00:13:19.433 INFO:teuthology.orchestra.run.smithi145.stderr: --help produce help message 2020-01-16T00:13:19.433 INFO:teuthology.orchestra.run.smithi145.stderr: --type arg Arg is one of [bluestore (default), filestore, 2020-01-16T00:13:19.433 INFO:teuthology.orchestra.run.smithi145.stderr: memstore] 2020-01-16T00:13:19.433 INFO:teuthology.orchestra.run.smithi145.stderr: --data-path arg path to object store, mandatory 2020-01-16T00:13:19.434 INFO:teuthology.orchestra.run.smithi145.stderr: --journal-path arg path to journal, use if tool can't find it 2020-01-16T00:13:19.463 INFO:teuthology.orchestra.run.smithi145.stderr: --pgid arg PG id, mandatory for info, log, remove, export, 2020-01-16T00:13:19.463 INFO:teuthology.orchestra.run.smithi145.stderr: export-remove, mark-complete, trim-pg-log, and 2020-01-16T00:13:19.463 INFO:teuthology.orchestra.run.smithi145.stderr: mandatory for apply-layout-settings if --pool is 2020-01-16T00:13:19.463 INFO:teuthology.orchestra.run.smithi145.stderr: not specified 2020-01-16T00:13:19.463 INFO:teuthology.orchestra.run.smithi145.stderr: --pool arg Pool name, mandatory for apply-layout-settings if 2020-01-16T00:13:19.463 INFO:teuthology.orchestra.run.smithi145.stderr: --pgid is not specified 2020-01-16T00:13:19.463 INFO:teuthology.orchestra.run.smithi145.stderr: --op arg Arg is one of [info, log, remove, mkfs, fsck, 2020-01-16T00:13:19.474 INFO:teuthology.orchestra.run.smithi145.stderr: repair, fuse, dup, export, export-remove, import, 2020-01-16T00:13:19.474 INFO:teuthology.orchestra.run.smithi145.stderr: list, list-slow-omap, fix-lost, list-pgs, 2020-01-16T00:13:19.474 INFO:teuthology.orchestra.run.smithi145.stderr: dump-journal, dump-super, meta-list, get-osdmap, 2020-01-16T00:13:19.474 INFO:teuthology.orchestra.run.smithi145.stderr: set-osdmap, get-inc-osdmap, set-inc-osdmap, 2020-01-16T00:13:19.474 INFO:teuthology.orchestra.run.smithi145.stderr: mark-complete, reset-last-complete, 2020-01-16T00:13:19.474 INFO:teuthology.orchestra.run.smithi145.stderr: apply-layout-settings, update-mon-db, 2020-01-16T00:13:19.474 INFO:teuthology.orchestra.run.smithi145.stderr: dump-export, trim-pg-log] 2020-01-16T00:13:19.475 INFO:teuthology.orchestra.run.smithi145.stderr: --epoch arg epoch# for get-osdmap and get-inc-osdmap, the 2020-01-16T00:13:19.475 INFO:teuthology.orchestra.run.smithi145.stderr: current epoch in use if not specified 2020-01-16T00:13:19.475 INFO:teuthology.orchestra.run.smithi145.stderr: --file arg path of file to export, export-remove, import, 2020-01-16T00:13:19.475 INFO:teuthology.orchestra.run.smithi145.stderr: get-osdmap, set-osdmap, get-inc-osdmap or 2020-01-16T00:13:19.475 INFO:teuthology.orchestra.run.smithi145.stderr: set-inc-osdmap 2020-01-16T00:13:19.475 INFO:teuthology.orchestra.run.smithi145.stderr: --mon-store-path arg path of monstore to update-mon-db 2020-01-16T00:13:19.475 INFO:teuthology.orchestra.run.smithi145.stderr: --fsid arg fsid for new store created by mkfs 2020-01-16T00:13:19.475 INFO:teuthology.orchestra.run.smithi145.stderr: --target-data-path arg path of target object store (for --op dup) 2020-01-16T00:13:19.476 INFO:teuthology.orchestra.run.smithi145.stderr: --mountpoint arg fuse mountpoint 2020-01-16T00:13:19.476 INFO:teuthology.orchestra.run.smithi145.stderr: --format arg (=json-pretty) Output format which may be json, json-pretty, 2020-01-16T00:13:19.476 INFO:teuthology.orchestra.run.smithi145.stderr: xml, xml-pretty 2020-01-16T00:13:19.476 INFO:teuthology.orchestra.run.smithi145.stderr: --debug Enable diagnostic output to stderr 2020-01-16T00:13:19.476 INFO:teuthology.orchestra.run.smithi145.stderr: --no-mon-config Do not contact mons for config 2020-01-16T00:13:19.476 INFO:teuthology.orchestra.run.smithi145.stderr: --no-superblock Do not read superblock 2020-01-16T00:13:19.476 INFO:teuthology.orchestra.run.smithi145.stderr: --force Ignore some types of errors and proceed with 2020-01-16T00:13:19.477 INFO:teuthology.orchestra.run.smithi145.stderr: operation - USE WITH CAUTION: CORRUPTION POSSIBLE 2020-01-16T00:13:19.477 INFO:teuthology.orchestra.run.smithi145.stderr: NOW OR IN THE FUTURE 2020-01-16T00:13:19.477 INFO:teuthology.orchestra.run.smithi145.stderr: --skip-journal-replay Disable journal replay 2020-01-16T00:13:19.477 INFO:teuthology.orchestra.run.smithi145.stderr: --skip-mount-omap Disable mounting of omap 2020-01-16T00:13:19.477 INFO:teuthology.orchestra.run.smithi145.stderr: --head Find head/snapdir when searching for objects by 2020-01-16T00:13:19.477 INFO:teuthology.orchestra.run.smithi145.stderr: name 2020-01-16T00:13:19.477 INFO:teuthology.orchestra.run.smithi145.stderr: --dry-run Don't modify the objectstore 2020-01-16T00:13:19.477 INFO:teuthology.orchestra.run.smithi145.stderr: --namespace arg Specify namespace when searching for objects 2020-01-16T00:13:19.478 INFO:teuthology.orchestra.run.smithi145.stderr: --rmtype arg Specify corrupting object removal 'snapmap' or 2020-01-16T00:13:19.478 INFO:teuthology.orchestra.run.smithi145.stderr: 'nosnapmap' - TESTING USE ONLY 2020-01-16T00:13:19.478 INFO:teuthology.orchestra.run.smithi145.stderr: --slow-omap-threshold arg Threshold (in seconds) to consider omap listing 2020-01-16T00:13:19.478 INFO:teuthology.orchestra.run.smithi145.stderr: slow (for op=list-slow-omap) 2020-01-16T00:13:19.478 INFO:teuthology.orchestra.run.smithi145.stderr: 2020-01-16T00:13:19.478 INFO:teuthology.orchestra.run.smithi145.stderr: 2020-01-16T00:13:19.478 INFO:teuthology.orchestra.run.smithi145.stderr:Positional syntax: 2020-01-16T00:13:19.478 INFO:teuthology.orchestra.run.smithi145.stderr: 2020-01-16T00:13:19.479 INFO:teuthology.orchestra.run.smithi145.stderr:ceph-objectstore-tool ... (get|set)-bytes [file] 2020-01-16T00:13:19.479 INFO:teuthology.orchestra.run.smithi145.stderr:ceph-objectstore-tool ... set-(attr|omap) [file] 2020-01-16T00:13:19.479 INFO:teuthology.orchestra.run.smithi145.stderr:ceph-objectstore-tool ... (get|rm)-(attr|omap) 2020-01-16T00:13:19.479 INFO:teuthology.orchestra.run.smithi145.stderr:ceph-objectstore-tool ... get-omaphdr 2020-01-16T00:13:19.479 INFO:teuthology.orchestra.run.smithi145.stderr:ceph-objectstore-tool ... set-omaphdr [file] 2020-01-16T00:13:19.479 INFO:teuthology.orchestra.run.smithi145.stderr:ceph-objectstore-tool ... list-attrs 2020-01-16T00:13:19.479 INFO:teuthology.orchestra.run.smithi145.stderr:ceph-objectstore-tool ... list-omap 2020-01-16T00:13:19.479 INFO:teuthology.orchestra.run.smithi145.stderr:ceph-objectstore-tool ... remove|removeall 2020-01-16T00:13:19.480 INFO:teuthology.orchestra.run.smithi145.stderr:ceph-objectstore-tool ... dump 2020-01-16T00:13:19.480 INFO:teuthology.orchestra.run.smithi145.stderr:ceph-objectstore-tool ... set-size 2020-01-16T00:13:19.480 INFO:teuthology.orchestra.run.smithi145.stderr:ceph-objectstore-tool ... clear-data-digest 2020-01-16T00:13:19.480 INFO:teuthology.orchestra.run.smithi145.stderr:ceph-objectstore-tool ... remove-clone-metadata 2020-01-16T00:13:19.480 INFO:teuthology.orchestra.run.smithi145.stderr: 2020-01-16T00:13:19.480 INFO:teuthology.orchestra.run.smithi145.stderr: can be a JSON object description as displayed 2020-01-16T00:13:19.480 INFO:teuthology.orchestra.run.smithi145.stderr:by --op list. 2020-01-16T00:13:19.480 INFO:teuthology.orchestra.run.smithi145.stderr: can be an object name which will be looked up in all 2020-01-16T00:13:19.481 INFO:teuthology.orchestra.run.smithi145.stderr:the OSD's PGs. 2020-01-16T00:13:19.481 INFO:teuthology.orchestra.run.smithi145.stderr: can be the empty string ('') which with a provided pgid 2020-01-16T00:13:19.481 INFO:teuthology.orchestra.run.smithi145.stderr:specifies the pgmeta object 2020-01-16T00:13:19.481 INFO:teuthology.orchestra.run.smithi145.stderr: 2020-01-16T00:13:19.481 INFO:teuthology.orchestra.run.smithi145.stderr:The optional [file] argument will read stdin or write stdout 2020-01-16T00:13:19.481 INFO:teuthology.orchestra.run.smithi145.stderr:if not specified or if '-' specified. 2020-01-16T00:13:19.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-01-16T00:13:19.482 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:19.432+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:19.482 INFO:teuthology.orchestra.run.smithi145:> sudo rm -f /var/log/ceph/exp.1.1.5 2020-01-16T00:13:19.485 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:19.485 INFO:tasks.rados.rados.0.smithi123.stdout:1167: finishing write tid 1 to smithi12313934-13 2020-01-16T00:13:19.485 INFO:tasks.rados.rados.0.smithi123.stdout:1167: finishing write tid 2 to smithi12313934-13 2020-01-16T00:13:19.485 INFO:tasks.rados.rados.0.smithi123.stdout:1168: finishing write tid 1 to smithi12313934-6 2020-01-16T00:13:19.485 INFO:tasks.rados.rados.0.smithi123.stdout:1167: finishing write tid 3 to smithi12313934-13 2020-01-16T00:13:19.486 INFO:tasks.rados.rados.0.smithi123.stdout:1166: done (15 left) 2020-01-16T00:13:19.486 INFO:tasks.rados.rados.0.smithi123.stdout:1181: done (14 left) 2020-01-16T00:13:19.486 INFO:tasks.rados.rados.0.smithi123.stdout:1182: read oid 30 snap -1 2020-01-16T00:13:19.486 INFO:tasks.rados.rados.0.smithi123.stdout:1182: expect (ObjNum 294 snap 0 seq_num 294) 2020-01-16T00:13:19.486 INFO:tasks.rados.rados.0.smithi123.stdout:1183: write (excl) oid 40 current snap is 0 2020-01-16T00:13:19.486 INFO:tasks.rados.rados.0.smithi123.stdout:1183: seq_num 347 ranges {632029=547288,1639134=749453,2796249=437859,3653524=178505} 2020-01-16T00:13:19.487 INFO:tasks.rados.rados.0.smithi123.stdout:1183: writing smithi12313934-40 from 632029 to 1179317 tid 1 2020-01-16T00:13:19.487 INFO:tasks.rados.rados.0.smithi123.stdout:1183: writing smithi12313934-40 from 1639134 to 2388587 tid 2 2020-01-16T00:13:19.487 INFO:tasks.rados.rados.0.smithi123.stdout:1183: writing smithi12313934-40 from 2796249 to 3234108 tid 3 2020-01-16T00:13:19.487 INFO:tasks.rados.rados.0.smithi123.stdout:1183: writing smithi12313934-40 from 3653524 to 3832029 tid 4 2020-01-16T00:13:19.487 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:19.487 INFO:tasks.rados.rados.0.smithi123.stdout:1167: finishing write tid 5 to smithi12313934-13 2020-01-16T00:13:19.487 INFO:tasks.rados.rados.0.smithi123.stdout:1167: finishing write tid 6 to smithi12313934-13 2020-01-16T00:13:19.487 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 71 (ObjNum 339 snap 0 seq_num 339) dirty exists 2020-01-16T00:13:19.488 INFO:tasks.rados.rados.0.smithi123.stdout:1167: left oid 13 (ObjNum 339 snap 0 seq_num 339) 2020-01-16T00:13:19.488 INFO:tasks.rados.rados.0.smithi123.stdout:1169: finishing write tid 1 to smithi12313934-31 2020-01-16T00:13:19.488 INFO:tasks.rados.rados.0.smithi123.stdout:1167: done (15 left) 2020-01-16T00:13:19.488 INFO:tasks.rados.rados.0.smithi123.stdout:1184: read oid 9 snap -1 2020-01-16T00:13:19.488 INFO:tasks.rados.rados.0.smithi123.stdout:1184: expect (ObjNum 333 snap 0 seq_num 333) 2020-01-16T00:13:19.488 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:19.488 INFO:tasks.rados.rados.0.smithi123.stdout:1168: finishing write tid 2 to smithi12313934-6 2020-01-16T00:13:19.488 INFO:tasks.rados.rados.0.smithi123.stdout:1168: finishing write tid 3 to smithi12313934-6 2020-01-16T00:13:19.489 INFO:tasks.rados.rados.0.smithi123.stdout:1168: finishing write tid 5 to smithi12313934-6 2020-01-16T00:13:19.489 INFO:tasks.rados.rados.0.smithi123.stdout:1168: finishing write tid 6 to smithi12313934-6 2020-01-16T00:13:19.489 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 6 v 102 (ObjNum 340 snap 0 seq_num 340) dirty exists 2020-01-16T00:13:19.489 INFO:tasks.rados.rados.0.smithi123.stdout:1168: left oid 6 (ObjNum 340 snap 0 seq_num 340) 2020-01-16T00:13:19.489 INFO:tasks.rados.rados.0.smithi123.stdout:1169: finishing write tid 2 to smithi12313934-31 2020-01-16T00:13:19.489 INFO:tasks.rados.rados.0.smithi123.stdout:1169: finishing write tid 3 to smithi12313934-31 2020-01-16T00:13:19.489 INFO:tasks.rados.rados.0.smithi123.stdout:1169: finishing write tid 5 to smithi12313934-31 2020-01-16T00:13:19.489 INFO:tasks.rados.rados.0.smithi123.stdout:1169: finishing write tid 6 to smithi12313934-31 2020-01-16T00:13:19.490 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 31 v 128 (ObjNum 341 snap 0 seq_num 341) dirty exists 2020-01-16T00:13:19.490 INFO:tasks.rados.rados.0.smithi123.stdout:1169: left oid 31 (ObjNum 341 snap 0 seq_num 341) 2020-01-16T00:13:19.490 INFO:tasks.rados.rados.0.smithi123.stdout:1170: finishing write tid 1 to smithi12313934-19 2020-01-16T00:13:19.490 INFO:tasks.rados.rados.0.smithi123.stdout:1170: finishing write tid 2 to smithi12313934-19 2020-01-16T00:13:19.490 INFO:tasks.rados.rados.0.smithi123.stdout:1170: finishing write tid 3 to smithi12313934-19 2020-01-16T00:13:19.490 INFO:tasks.rados.rados.0.smithi123.stdout:1168: done (15 left) 2020-01-16T00:13:19.490 INFO:tasks.rados.rados.0.smithi123.stdout:1169: done (14 left) 2020-01-16T00:13:19.490 INFO:tasks.rados.rados.0.smithi123.stdout:1185: write (excl) oid 17 current snap is 0 2020-01-16T00:13:19.491 INFO:tasks.rados.rados.0.smithi123.stdout:1185: seq_num 348 ranges {648836=622537,1981263=493111,3107319=741517} 2020-01-16T00:13:19.491 INFO:tasks.rados.rados.0.smithi123.stdout:1185: writing smithi12313934-17 from 648836 to 1271373 tid 1 2020-01-16T00:13:19.491 INFO:tasks.rados.rados.0.smithi123.stdout:1185: writing smithi12313934-17 from 1981263 to 2474374 tid 2 2020-01-16T00:13:19.491 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-01-16T00:13:19.492 INFO:tasks.rados.rados.0.smithi123.stdout:1185: writing smithi12313934-17 from 3107319 to 3848836 tid 3 2020-01-16T00:13:19.492 INFO:tasks.rados.rados.0.smithi123.stdout:1186: write oid 36 current snap is 0 2020-01-16T00:13:19.492 INFO:tasks.rados.rados.0.smithi123.stdout:1186: seq_num 349 ranges {665643=697786,2039745=753122,3534742=330901} 2020-01-16T00:13:19.499 INFO:tasks.rados.rados.0.smithi123.stdout:1186: writing smithi12313934-36 from 665643 to 1363429 tid 1 2020-01-16T00:13:19.508 INFO:tasks.rados.rados.0.smithi123.stdout:1186: writing smithi12313934-36 from 2039745 to 2792867 tid 2 2020-01-16T00:13:19.514 INFO:tasks.rados.rados.0.smithi123.stdout:1186: writing smithi12313934-36 from 3534742 to 3865643 tid 3 2020-01-16T00:13:19.514 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:19.514 INFO:tasks.rados.rados.0.smithi123.stdout:1170: finishing write tid 4 to smithi12313934-19 2020-01-16T00:13:19.514 INFO:tasks.rados.rados.0.smithi123.stdout:1172: expect (ObjNum 285 snap 0 seq_num 285) 2020-01-16T00:13:19.525 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:13:19.568 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:19.566+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:19.584 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:19.594 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-01-16T00:13:19.607 INFO:tasks.rados.rados.0.smithi123.stdout:1170: finishing write tid 6 to smithi12313934-19 2020-01-16T00:13:19.607 INFO:tasks.rados.rados.0.smithi123.stdout:1170: finishing write tid 7 to smithi12313934-19 2020-01-16T00:13:19.607 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 19 v 175 (ObjNum 342 snap 0 seq_num 342) dirty exists 2020-01-16T00:13:19.607 INFO:tasks.rados.rados.0.smithi123.stdout:1170: left oid 19 (ObjNum 342 snap 0 seq_num 342) 2020-01-16T00:13:19.608 INFO:tasks.rados.rados.0.smithi123.stdout:1173: finishing write tid 1 to smithi12313934-5 2020-01-16T00:13:19.608 INFO:tasks.rados.rados.0.smithi123.stdout:1170: done (15 left) 2020-01-16T00:13:19.608 INFO:tasks.rados.rados.0.smithi123.stdout:1172: done (14 left) 2020-01-16T00:13:19.608 INFO:tasks.rados.rados.0.smithi123.stdout:1187: write oid 48 current snap is 0 2020-01-16T00:13:19.608 INFO:tasks.rados.rados.0.smithi123.stdout:1187: seq_num 350 ranges {682450=489388,1814580=496780,2878518=788028} 2020-01-16T00:13:19.616 INFO:tasks.rados.rados.0.smithi123.stdout:1187: writing smithi12313934-48 from 682450 to 1171838 tid 1 2020-01-16T00:13:19.630 INFO:tasks.rados.rados.0.smithi123.stdout:1187: writing smithi12313934-48 from 1814580 to 2311360 tid 2 2020-01-16T00:13:19.632 INFO:tasks.rados.rados.0.smithi123.stdout:1187: writing smithi12313934-48 from 2878518 to 3666546 tid 3 2020-01-16T00:13:19.632 INFO:tasks.rados.rados.0.smithi123.stdout:1188: read oid 18 snap -1 2020-01-16T00:13:19.633 INFO:tasks.rados.rados.0.smithi123.stdout:1188: expect (ObjNum 220 snap 0 seq_num 220) 2020-01-16T00:13:19.633 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:19.633 INFO:tasks.rados.rados.0.smithi123.stdout:1171: expect (ObjNum 327 snap 0 seq_num 327) 2020-01-16T00:13:19.638 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:19.634+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:19.703 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-01-16T00:13:19.730 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:19.739 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:19.734+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:19.739 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-01-16T00:13:19.773 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:13:19.774 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":42,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:13:18.855447+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2020-01-16T00:12:31.098841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.514177+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.443103+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:12.903469+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.557899+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.225222+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.687721+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.421064+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:13:19.786 INFO:tasks.ceph.ceph_manager.ceph:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2020-01-16T00:12:18.200931+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 3, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 2, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'17', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 4, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2020-01-16T00:12:30.019477+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 3, u'pg_num_pending': 4, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 4, u'pg_num': 4, u'type': 1, u'grade_table': [], u'pool_name': u'low_tier', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 4, u'cache_mode': u'none', u'min_size': 2, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 2, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'21', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 16, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2020-01-16T00:12:31.098841+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 3, u'pg_num_pending': 16, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 16, u'pg_num': 16, u'type': 1, u'grade_table': [], u'pool_name': u'unique_pool_0', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 16, u'cache_mode': u'none', u'min_size': 2, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 3, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rados': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'26', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2020-01-16T00:13:19.787 INFO:teuthology.orchestra.run.smithi145:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --log-file=/var/log/ceph/objectstore_tool.$pid.log--op apply-layout-settings --pool rbd 2020-01-16T00:13:19.800 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: /var/lib/ceph/osd/ceph-5: command not found 2020-01-16T00:13:19.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-01-16T00:13:19.802 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage-testing-2020-01-15-1209/qa/tasks/ceph_manager.py", line 1023, in wrapper return func(self) File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage-testing-2020-01-15-1209/qa/tasks/ceph_manager.py", line 1153, in _do_thrash self.choose_action()() File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage-testing-2020-01-15-1209/qa/tasks/ceph_manager.py", line 408, in kill_osd " failed with {status}".format(status=proc.exitstatus)) Exception: ceph-objectstore-tool apply-layout-settings failed with 1 2020-01-16T00:13:19.802 ERROR:tasks.thrashosds.thrasher:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage-testing-2020-01-15-1209/qa/tasks/ceph_manager.py", line 1011, in do_thrash self._do_thrash() File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage-testing-2020-01-15-1209/qa/tasks/ceph_manager.py", line 1023, in wrapper return func(self) File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage-testing-2020-01-15-1209/qa/tasks/ceph_manager.py", line 1153, in _do_thrash self.choose_action()() File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage-testing-2020-01-15-1209/qa/tasks/ceph_manager.py", line 408, in kill_osd " failed with {status}".format(status=proc.exitstatus)) Exception: ceph-objectstore-tool apply-layout-settings failed with 1 2020-01-16T00:13:19.847 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:19.844+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:19.853 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:13:19.884 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:13:19.898 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:19.915 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-01-16T00:13:19.916 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:13:19.934 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:13:19.940 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:19.936+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:19.955 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:13:19.979 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:13:20.002 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:13:20.004 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-01-16T00:13:20.004 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-01-16T00:13:20.004 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-01-16T00:13:20.004 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-01-16T00:13:20.005 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-01-16T00:13:20.005 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENXIO: problem getting command descriptions from osd.5 2020-01-16T00:13:20.005 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: problem getting command descriptions from osd.5 2020-01-16T00:13:20.005 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-01-16T00:13:20.005 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-01-16T00:13:20.009 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:20.010 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:20.010 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:13:20.010 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:13:20.010 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:20.014 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-01-16T00:13:20.014 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-01-16T00:13:20.019 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-01-16T00:13:20.021 INFO:tasks.rados.rados.0.smithi123.stdout:1173: finishing write tid 2 to smithi12313934-5 2020-01-16T00:13:20.021 INFO:tasks.rados.rados.0.smithi123.stdout:1173: finishing write tid 3 to smithi12313934-5 2020-01-16T00:13:20.021 INFO:tasks.rados.rados.0.smithi123.stdout:1171: done (15 left) 2020-01-16T00:13:20.021 INFO:tasks.rados.rados.0.smithi123.stdout:1189: read oid 33 snap -1 2020-01-16T00:13:20.021 INFO:tasks.rados.rados.0.smithi123.stdout:1189: expect (ObjNum 330 snap 0 seq_num 330) 2020-01-16T00:13:20.021 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:20.022 INFO:tasks.rados.rados.0.smithi123.stdout:1173: finishing write tid 5 to smithi12313934-5 2020-01-16T00:13:20.022 INFO:tasks.rados.rados.0.smithi123.stdout:1173: finishing write tid 6 to smithi12313934-5 2020-01-16T00:13:20.022 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 5 v 183 (ObjNum 343 snap 0 seq_num 343) dirty exists 2020-01-16T00:13:20.022 INFO:tasks.rados.rados.0.smithi123.stdout:1173: left oid 5 (ObjNum 343 snap 0 seq_num 343) 2020-01-16T00:13:20.022 INFO:tasks.rados.rados.0.smithi123.stdout:1174: finishing write tid 1 to smithi12313934-32 2020-01-16T00:13:20.022 INFO:tasks.rados.rados.0.smithi123.stdout:1173: done (15 left) 2020-01-16T00:13:20.022 INFO:tasks.rados.rados.0.smithi123.stdout:1190: write oid 39 current snap is 0 2020-01-16T00:13:20.023 INFO:tasks.rados.rados.0.smithi123.stdout:1190: seq_num 351 ranges {699257=564637,1873062=640438,3305941=593316} 2020-01-16T00:13:20.028 INFO:tasks.rados.rados.0.smithi123.stdout:1190: writing smithi12313934-39 from 699257 to 1263894 tid 1 2020-01-16T00:13:20.036 INFO:tasks.rados.rados.0.smithi123.stdout:1190: writing smithi12313934-39 from 1873062 to 2513500 tid 2 2020-01-16T00:13:20.041 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:20.036+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:20.044 INFO:tasks.rados.rados.0.smithi123.stdout:1190: writing smithi12313934-39 from 3305941 to 3899257 tid 3 2020-01-16T00:13:20.045 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:20.045 INFO:tasks.rados.rados.0.smithi123.stdout:1174: finishing write tid 2 to smithi12313934-32 2020-01-16T00:13:20.045 INFO:tasks.rados.rados.0.smithi123.stdout:1174: finishing write tid 3 to smithi12313934-32 2020-01-16T00:13:20.045 INFO:tasks.rados.rados.0.smithi123.stdout:1174: finishing write tid 5 to smithi12313934-32 2020-01-16T00:13:20.045 INFO:tasks.rados.rados.0.smithi123.stdout:1174: finishing write tid 6 to smithi12313934-32 2020-01-16T00:13:20.045 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 32 v 117 (ObjNum 344 snap 0 seq_num 344) dirty exists 2020-01-16T00:13:20.045 INFO:tasks.rados.rados.0.smithi123.stdout:1174: left oid 32 (ObjNum 344 snap 0 seq_num 344) 2020-01-16T00:13:20.046 INFO:tasks.rados.rados.0.smithi123.stdout:1175: finishing write tid 1 to smithi12313934-21 2020-01-16T00:13:20.046 INFO:tasks.rados.rados.0.smithi123.stdout:1174: done (15 left) 2020-01-16T00:13:20.046 INFO:tasks.rados.rados.0.smithi123.stdout:1191: read oid 1 snap -1 2020-01-16T00:13:20.046 INFO:tasks.rados.rados.0.smithi123.stdout:1191: expect (ObjNum 326 snap 0 seq_num 326) 2020-01-16T00:13:20.046 INFO:tasks.rados.rados.0.smithi123.stdout:1175: finishing write tid 2 to smithi12313934-21 2020-01-16T00:13:20.046 INFO:tasks.rados.rados.0.smithi123.stdout:1178: done (15 left) 2020-01-16T00:13:20.046 INFO:tasks.rados.rados.0.smithi123.stdout:1192: read oid 19 snap -1 2020-01-16T00:13:20.047 INFO:tasks.rados.rados.0.smithi123.stdout:1192: expect (ObjNum 342 snap 0 seq_num 342) 2020-01-16T00:13:20.047 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:20.047 INFO:tasks.rados.rados.0.smithi123.stdout:1177: expect (ObjNum 260 snap 0 seq_num 260) 2020-01-16T00:13:20.138 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:20.138 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:13:20.139 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:13:20.139 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:13:20.139 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.139 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.4.0:114 3.9 e24/20 3:9fc687b2:::smithi12313934-44:head v 24'74, mlcod=24'72)", 2020-01-16T00:13:20.139 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:13:20.139 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 41.809747807999997, 2020-01-16T00:13:20.139 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010399429, 2020-01-16T00:13:20.140 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:20.140 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:20.140 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:20.140 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.140 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:20.140 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:13:20.140 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:20.140 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.141 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.141 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:20.141 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:13:20.141 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:20.141 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.141 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.141 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:20.142 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325276+0000", 2020-01-16T00:13:20.142 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0046127069999999997 2020-01-16T00:13:20.142 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.142 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.142 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:20.142 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329889+0000", 2020-01-16T00:13:20.142 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3519999999999998e-05 2020-01-16T00:13:20.143 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.143 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.143 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:20.143 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329912+0000", 2020-01-16T00:13:20.143 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4070000000000004e-06 2020-01-16T00:13:20.143 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.143 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.143 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.144 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329920+0000", 2020-01-16T00:13:20.144 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8817000000000003e-05 2020-01-16T00:13:20.144 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.144 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.144 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:20.144 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329978+0000", 2020-01-16T00:13:20.144 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9721e-05 2020-01-16T00:13:20.144 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.145 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.145 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:20.145 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329998+0000", 2020-01-16T00:13:20.145 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056328139999999999 2020-01-16T00:13:20.145 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.145 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.146 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:20.146 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.335631+0000", 2020-01-16T00:13:20.146 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7170999999999999e-05 2020-01-16T00:13:20.146 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.146 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.146 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:20.147 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.335678+0000", 2020-01-16T00:13:20.147 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010399429 2020-01-16T00:13:20.147 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.147 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:20.148 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.148 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.148 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.148 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:24 2.1 2:8b431f80:::smithi12313934-40 low_tier:head [write 790528~688128] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:13:20.148 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.482303+0000", 2020-01-16T00:13:20.149 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 41.652723711999997, 2020-01-16T00:13:20.149 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010071853, 2020-01-16T00:13:20.149 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:20.149 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:20.149 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:20.149 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.150 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:20.150 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.482303+0000", 2020-01-16T00:13:20.150 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:20.150 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.150 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.150 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:20.150 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.482303+0000", 2020-01-16T00:13:20.151 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:20.151 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.151 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.151 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:20.151 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.482301+0000", 2020-01-16T00:13:20.151 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031173960000000001 2020-01-16T00:13:20.152 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.152 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.152 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:20.152 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485418+0000", 2020-01-16T00:13:20.152 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9300000000000001e-05 2020-01-16T00:13:20.152 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.152 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.152 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:20.153 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485448+0000", 2020-01-16T00:13:20.153 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2611e-05 2020-01-16T00:13:20.153 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.153 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.153 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.153 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485460+0000", 2020-01-16T00:13:20.153 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112487 2020-01-16T00:13:20.154 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.154 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.154 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:20.154 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485573+0000", 2020-01-16T00:13:20.154 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022169299999999999 2020-01-16T00:13:20.154 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.154 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.155 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:20.155 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485794+0000", 2020-01-16T00:13:20.155 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00036287500000000002 2020-01-16T00:13:20.155 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.156 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.156 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:20.156 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.486157+0000", 2020-01-16T00:13:20.156 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040260310000000002 2020-01-16T00:13:20.156 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.157 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.157 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:20.157 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.490183+0000", 2020-01-16T00:13:20.157 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00081083000000000001 2020-01-16T00:13:20.157 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.158 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.158 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.158 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.490994+0000", 2020-01-16T00:13:20.158 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001287746 2020-01-16T00:13:20.159 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.159 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.159 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.159 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492282+0000", 2020-01-16T00:13:20.159 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8810999999999998e-05 2020-01-16T00:13:20.160 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.160 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.160 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:20.160 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492321+0000", 2020-01-16T00:13:20.161 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4095000000000003e-05 2020-01-16T00:13:20.161 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.161 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.162 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:20.162 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492375+0000", 2020-01-16T00:13:20.162 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010071853 2020-01-16T00:13:20.162 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.162 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:20.162 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.163 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.163 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.163 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:25 2.1 2:8b431f80:::smithi12313934-40 low_tier:head [write 1926656~67072] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:13:20.163 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.485481+0000", 2020-01-16T00:13:20.163 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 41.649545246999999, 2020-01-16T00:13:20.163 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0081599020000000001, 2020-01-16T00:13:20.163 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:20.163 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:20.164 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:20.164 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.164 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:20.164 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485481+0000", 2020-01-16T00:13:20.164 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:20.164 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.164 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.164 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:20.165 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485481+0000", 2020-01-16T00:13:20.165 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:20.165 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.165 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.165 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:20.165 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485479+0000", 2020-01-16T00:13:20.165 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012423800000000001 2020-01-16T00:13:20.165 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.166 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.166 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:20.166 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485603+0000", 2020-01-16T00:13:20.166 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8366999999999999e-05 2020-01-16T00:13:20.166 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.166 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.166 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:20.167 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485621+0000", 2020-01-16T00:13:20.167 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0207000000000001e-05 2020-01-16T00:13:20.167 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.167 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.167 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.167 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485632+0000", 2020-01-16T00:13:20.167 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002868532 2020-01-16T00:13:20.167 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.168 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.168 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:20.168 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.488500+0000", 2020-01-16T00:13:20.168 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011340000000000001 2020-01-16T00:13:20.168 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.168 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.168 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:20.168 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.488614+0000", 2020-01-16T00:13:20.169 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019056300000000001 2020-01-16T00:13:20.169 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.169 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.169 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:20.169 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.488804+0000", 2020-01-16T00:13:20.169 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00153302 2020-01-16T00:13:20.169 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.169 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.170 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:20.170 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.490337+0000", 2020-01-16T00:13:20.170 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00089946100000000005 2020-01-16T00:13:20.170 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.170 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.170 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.170 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.491237+0000", 2020-01-16T00:13:20.171 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0022862820000000002 2020-01-16T00:13:20.171 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.171 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.171 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.171 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.493523+0000", 2020-01-16T00:13:20.171 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5609999999999998e-05 2020-01-16T00:13:20.171 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.171 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.172 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:20.172 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.493578+0000", 2020-01-16T00:13:20.172 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2770999999999999e-05 2020-01-16T00:13:20.172 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.172 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.172 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:20.172 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.493641+0000", 2020-01-16T00:13:20.172 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0081599020000000001 2020-01-16T00:13:20.173 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.173 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:20.173 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.173 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.173 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.173 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:1279 3.0 e26/20 3:05b2bf53:::smithi12313934-37:head v 26'137, mlcod=26'135)", 2020-01-16T00:13:20.173 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:43.082160+0000", 2020-01-16T00:13:20.173 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 37.052866514000002, 2020-01-16T00:13:20.174 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0091277130000000008, 2020-01-16T00:13:20.174 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:20.174 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:20.174 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:20.174 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.174 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:20.174 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.082160+0000", 2020-01-16T00:13:20.174 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:20.175 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.175 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.175 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:20.175 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.082160+0000", 2020-01-16T00:13:20.175 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999957 2020-01-16T00:13:20.175 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.175 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.175 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:20.176 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.082156+0000", 2020-01-16T00:13:20.176 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00046771999999999997 2020-01-16T00:13:20.176 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.176 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.176 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:20.176 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.082623+0000", 2020-01-16T00:13:20.176 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3099999999999998e-05 2020-01-16T00:13:20.177 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.177 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.177 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:20.177 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.082656+0000", 2020-01-16T00:13:20.177 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2063e-05 2020-01-16T00:13:20.177 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.177 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.177 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.178 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.082668+0000", 2020-01-16T00:13:20.178 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021980900000000002 2020-01-16T00:13:20.178 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.178 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.178 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:20.178 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.084867+0000", 2020-01-16T00:13:20.178 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00033252299999999999 2020-01-16T00:13:20.178 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.179 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.179 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:20.179 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.085199+0000", 2020-01-16T00:13:20.179 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0060364019999999997 2020-01-16T00:13:20.179 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.179 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.179 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:20.179 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.091235+0000", 2020-01-16T00:13:20.180 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2255e-05 2020-01-16T00:13:20.180 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.180 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.180 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:20.180 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:43.091288+0000", 2020-01-16T00:13:20.180 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0091277130000000008 2020-01-16T00:13:20.180 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.180 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:20.181 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.181 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.181 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.181 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:47 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e28)", 2020-01-16T00:13:20.181 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:47.941839+0000", 2020-01-16T00:13:20.181 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 32.193187393000002, 2020-01-16T00:13:20.181 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.008400154, 2020-01-16T00:13:20.182 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:20.182 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:20.182 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:20.182 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.182 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:20.182 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.941839+0000", 2020-01-16T00:13:20.182 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:20.182 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.183 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.183 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:20.183 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.941839+0000", 2020-01-16T00:13:20.183 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:20.183 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.184 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.184 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:20.184 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.941837+0000", 2020-01-16T00:13:20.184 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00046009399999999998 2020-01-16T00:13:20.184 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.185 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.185 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:20.185 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942297+0000", 2020-01-16T00:13:20.185 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8628999999999999e-05 2020-01-16T00:13:20.185 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.185 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.185 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:20.186 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942335+0000", 2020-01-16T00:13:20.186 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1729999999999998e-06 2020-01-16T00:13:20.186 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.186 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.186 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.186 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942343+0000", 2020-01-16T00:13:20.186 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4845999999999997e-05 2020-01-16T00:13:20.186 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.187 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.187 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:20.187 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942387+0000", 2020-01-16T00:13:20.187 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6398999999999999e-05 2020-01-16T00:13:20.187 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.187 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.187 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:20.187 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942454+0000", 2020-01-16T00:13:20.188 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010786 2020-01-16T00:13:20.188 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.188 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.188 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:20.188 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942562+0000", 2020-01-16T00:13:20.188 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0064317760000000002 2020-01-16T00:13:20.188 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.188 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.189 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:20.189 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.948993+0000", 2020-01-16T00:13:20.189 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0011048849999999999 2020-01-16T00:13:20.189 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.189 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.189 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.189 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.950098+0000", 2020-01-16T00:13:20.189 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7064000000000003e-05 2020-01-16T00:13:20.190 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.190 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.190 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.190 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.950135+0000", 2020-01-16T00:13:20.190 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6589000000000003e-05 2020-01-16T00:13:20.190 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.190 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.190 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:20.191 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.950192+0000", 2020-01-16T00:13:20.191 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7286000000000001e-05 2020-01-16T00:13:20.191 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.191 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.191 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:20.191 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.950239+0000", 2020-01-16T00:13:20.191 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.008400154 2020-01-16T00:13:20.192 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.192 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:20.192 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.192 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.192 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.192 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:48 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e28)", 2020-01-16T00:13:20.192 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:47.942354+0000", 2020-01-16T00:13:20.192 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 32.192672979000001, 2020-01-16T00:13:20.193 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0080319870000000008, 2020-01-16T00:13:20.193 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:20.193 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:20.193 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:20.193 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.193 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:20.193 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942354+0000", 2020-01-16T00:13:20.194 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:20.194 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.194 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.194 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:20.194 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942354+0000", 2020-01-16T00:13:20.194 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:20.194 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.194 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.195 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:20.195 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942353+0000", 2020-01-16T00:13:20.195 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5825000000000002e-05 2020-01-16T00:13:20.195 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.195 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.195 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:20.195 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942409+0000", 2020-01-16T00:13:20.195 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5929999999999996e-06 2020-01-16T00:13:20.196 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.196 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.196 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:20.196 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942414+0000", 2020-01-16T00:13:20.196 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4199999999999999e-06 2020-01-16T00:13:20.196 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.196 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.196 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.197 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.942418+0000", 2020-01-16T00:13:20.197 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0027594500000000001 2020-01-16T00:13:20.197 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.197 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.197 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:20.197 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.945177+0000", 2020-01-16T00:13:20.197 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.4040000000000002e-05 2020-01-16T00:13:20.198 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.198 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.198 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:20.198 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.945261+0000", 2020-01-16T00:13:20.198 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012906699999999999 2020-01-16T00:13:20.198 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.198 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.198 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:20.199 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.945390+0000", 2020-01-16T00:13:20.199 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0047161060000000003 2020-01-16T00:13:20.199 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.199 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.199 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:20.199 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.950106+0000", 2020-01-16T00:13:20.199 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016987400000000001 2020-01-16T00:13:20.199 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.200 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.200 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.200 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.950276+0000", 2020-01-16T00:13:20.200 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6329000000000002e-05 2020-01-16T00:13:20.200 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.200 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.200 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.200 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.950303+0000", 2020-01-16T00:13:20.201 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1443000000000003e-05 2020-01-16T00:13:20.201 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.201 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.201 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:20.201 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.950344+0000", 2020-01-16T00:13:20.201 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1486000000000002e-05 2020-01-16T00:13:20.201 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.202 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.202 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:20.202 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:47.950386+0000", 2020-01-16T00:13:20.202 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0080319870000000008 2020-01-16T00:13:20.202 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.202 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:20.202 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.202 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.203 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.203 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:55 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e30)", 2020-01-16T00:13:20.203 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.736036+0000", 2020-01-16T00:13:20.203 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 27.398990589, 2020-01-16T00:13:20.203 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010974536, 2020-01-16T00:13:20.203 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:20.203 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:20.203 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:20.204 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.204 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:20.204 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736036+0000", 2020-01-16T00:13:20.204 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:20.204 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.204 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.204 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:20.204 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736036+0000", 2020-01-16T00:13:20.205 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:20.205 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.205 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.205 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:20.205 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736034+0000", 2020-01-16T00:13:20.205 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043589699999999998 2020-01-16T00:13:20.205 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.205 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.206 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:20.206 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736469+0000", 2020-01-16T00:13:20.206 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9882e-05 2020-01-16T00:13:20.206 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.206 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.206 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:20.206 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736489+0000", 2020-01-16T00:13:20.207 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8339999999999999e-06 2020-01-16T00:13:20.207 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.207 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.207 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.207 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736497+0000", 2020-01-16T00:13:20.207 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5354999999999999e-05 2020-01-16T00:13:20.207 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.207 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.208 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:20.208 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736572+0000", 2020-01-16T00:13:20.208 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5228000000000004e-05 2020-01-16T00:13:20.208 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.208 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.208 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:20.208 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736668+0000", 2020-01-16T00:13:20.208 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017212999999999999 2020-01-16T00:13:20.209 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.209 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.209 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:20.209 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736840+0000", 2020-01-16T00:13:20.209 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0099057020000000006 2020-01-16T00:13:20.209 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.209 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.209 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.210 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746746+0000", 2020-01-16T00:13:20.210 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0586000000000005e-05 2020-01-16T00:13:20.210 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.210 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.210 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:20.210 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746816+0000", 2020-01-16T00:13:20.210 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2445000000000002e-05 2020-01-16T00:13:20.210 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.211 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.211 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.211 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746889+0000", 2020-01-16T00:13:20.211 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6839000000000002e-05 2020-01-16T00:13:20.211 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.211 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.211 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:20.212 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746955+0000", 2020-01-16T00:13:20.212 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5074000000000002e-05 2020-01-16T00:13:20.212 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.212 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.212 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:20.212 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.747011+0000", 2020-01-16T00:13:20.212 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010974536 2020-01-16T00:13:20.212 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.213 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:20.213 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.213 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.213 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.213 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:56 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e30)", 2020-01-16T00:13:20.213 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.736514+0000", 2020-01-16T00:13:20.213 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 27.398512753999999, 2020-01-16T00:13:20.213 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011660584999999999, 2020-01-16T00:13:20.214 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:20.214 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:20.214 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:20.214 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.214 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:20.214 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736514+0000", 2020-01-16T00:13:20.214 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:20.214 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.215 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.215 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:20.215 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736514+0000", 2020-01-16T00:13:20.215 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:20.215 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.215 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.215 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:20.215 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736513+0000", 2020-01-16T00:13:20.216 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9677000000000004e-05 2020-01-16T00:13:20.216 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.216 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.216 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:20.216 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736583+0000", 2020-01-16T00:13:20.216 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5269999999999998e-06 2020-01-16T00:13:20.216 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.216 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.217 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:20.217 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736591+0000", 2020-01-16T00:13:20.217 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5769999999999996e-06 2020-01-16T00:13:20.217 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.217 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.217 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.217 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736599+0000", 2020-01-16T00:13:20.218 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0053593310000000002 2020-01-16T00:13:20.218 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.218 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.218 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:20.218 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.741958+0000", 2020-01-16T00:13:20.218 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014164099999999999 2020-01-16T00:13:20.218 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.218 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.219 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:20.219 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.742100+0000", 2020-01-16T00:13:20.219 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020126000000000001 2020-01-16T00:13:20.219 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.219 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.219 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:20.219 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.742301+0000", 2020-01-16T00:13:20.219 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0045052859999999998 2020-01-16T00:13:20.220 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.220 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.220 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.220 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746806+0000", 2020-01-16T00:13:20.220 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00070802699999999996 2020-01-16T00:13:20.220 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.220 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.220 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:20.221 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.747514+0000", 2020-01-16T00:13:20.221 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00059247200000000003 2020-01-16T00:13:20.221 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.221 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.221 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.221 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748107+0000", 2020-01-16T00:13:20.221 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1870999999999999e-05 2020-01-16T00:13:20.222 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.222 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.222 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:20.222 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748149+0000", 2020-01-16T00:13:20.222 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5831000000000001e-05 2020-01-16T00:13:20.222 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.222 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.222 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:20.223 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748174+0000", 2020-01-16T00:13:20.223 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011660584999999999 2020-01-16T00:13:20.223 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.223 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:20.223 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.223 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.223 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.223 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:68 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 56832~1408000] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e34)", 2020-01-16T00:13:20.224 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:03.257978+0000", 2020-01-16T00:13:20.224 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 16.877048204000001, 2020-01-16T00:13:20.224 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0084778949999999992, 2020-01-16T00:13:20.224 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:20.224 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:20.224 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:20.224 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.224 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:20.225 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.257978+0000", 2020-01-16T00:13:20.225 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:20.225 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.225 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.225 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:20.225 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.257978+0000", 2020-01-16T00:13:20.225 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:20.226 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.226 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.226 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:20.226 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.257977+0000", 2020-01-16T00:13:20.226 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00042032400000000002 2020-01-16T00:13:20.226 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.226 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.226 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:20.227 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258397+0000", 2020-01-16T00:13:20.227 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5951999999999999e-05 2020-01-16T00:13:20.227 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.227 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.227 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:20.227 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258413+0000", 2020-01-16T00:13:20.227 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0729999999999999e-06 2020-01-16T00:13:20.227 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.228 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.228 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.228 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258420+0000", 2020-01-16T00:13:20.228 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013206700000000001 2020-01-16T00:13:20.228 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.228 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.228 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:20.228 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258552+0000", 2020-01-16T00:13:20.229 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2818999999999996e-05 2020-01-16T00:13:20.229 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.229 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.229 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:20.229 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258615+0000", 2020-01-16T00:13:20.229 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012016899999999999 2020-01-16T00:13:20.229 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.229 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.230 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:20.230 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258735+0000", 2020-01-16T00:13:20.230 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052078480000000002 2020-01-16T00:13:20.230 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.230 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.230 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:20.230 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.263943+0000", 2020-01-16T00:13:20.230 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0019258649999999999 2020-01-16T00:13:20.231 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.231 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.231 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.231 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.265869+0000", 2020-01-16T00:13:20.231 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00051785399999999999 2020-01-16T00:13:20.231 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.231 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.232 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.232 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.266387+0000", 2020-01-16T00:13:20.232 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0459e-05 2020-01-16T00:13:20.232 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.232 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.232 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:20.232 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.266427+0000", 2020-01-16T00:13:20.232 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.919e-05 2020-01-16T00:13:20.233 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.233 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.233 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:20.233 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.266456+0000", 2020-01-16T00:13:20.233 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0084778949999999992 2020-01-16T00:13:20.233 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.233 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:20.233 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.234 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.234 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.234 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:69 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 1922048~118272] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e34)", 2020-01-16T00:13:20.234 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:03.258431+0000", 2020-01-16T00:13:20.234 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 16.876595172999998, 2020-01-16T00:13:20.234 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0092806959999999997, 2020-01-16T00:13:20.234 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:20.234 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:20.235 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:20.235 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.235 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:20.235 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258431+0000", 2020-01-16T00:13:20.235 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:20.235 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.235 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.235 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:20.236 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258431+0000", 2020-01-16T00:13:20.236 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:20.236 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.236 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.236 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:20.236 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258431+0000", 2020-01-16T00:13:20.237 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3875000000000003e-05 2020-01-16T00:13:20.237 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.237 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.237 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:20.237 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258474+0000", 2020-01-16T00:13:20.237 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1040000000000007e-06 2020-01-16T00:13:20.237 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.237 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.238 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:20.238 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258483+0000", 2020-01-16T00:13:20.238 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6720000000000003e-06 2020-01-16T00:13:20.238 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.238 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.238 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.238 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258487+0000", 2020-01-16T00:13:20.238 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020507110000000002 2020-01-16T00:13:20.239 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.239 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.239 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:20.239 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.260538+0000", 2020-01-16T00:13:20.239 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017511100000000001 2020-01-16T00:13:20.239 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.239 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.239 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:20.240 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.260713+0000", 2020-01-16T00:13:20.240 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018914100000000001 2020-01-16T00:13:20.240 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.240 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.240 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:20.240 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.260902+0000", 2020-01-16T00:13:20.240 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.004582282 2020-01-16T00:13:20.240 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.241 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.241 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:20.241 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.265485+0000", 2020-01-16T00:13:20.241 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021355329999999998 2020-01-16T00:13:20.241 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.241 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.241 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.242 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.267620+0000", 2020-01-16T00:13:20.242 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1495999999999998e-05 2020-01-16T00:13:20.242 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.242 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.242 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.242 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.267652+0000", 2020-01-16T00:13:20.242 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5474e-05 2020-01-16T00:13:20.242 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.243 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.243 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:20.243 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.267687+0000", 2020-01-16T00:13:20.243 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5055e-05 2020-01-16T00:13:20.243 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.243 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.243 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:20.243 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.267712+0000", 2020-01-16T00:13:20.244 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0092806959999999997 2020-01-16T00:13:20.244 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.244 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:20.244 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.244 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.244 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.244 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:70 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e35)", 2020-01-16T00:13:20.244 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.985494+0000", 2020-01-16T00:13:20.245 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 14.149532958, 2020-01-16T00:13:20.245 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013746267, 2020-01-16T00:13:20.245 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:20.245 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:20.245 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:20.245 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.245 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:20.246 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985494+0000", 2020-01-16T00:13:20.246 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:20.246 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.246 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.246 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:20.246 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985494+0000", 2020-01-16T00:13:20.246 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:20.246 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.247 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.247 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:20.247 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985490+0000", 2020-01-16T00:13:20.247 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00077792099999999999 2020-01-16T00:13:20.247 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.247 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.247 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:20.247 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986268+0000", 2020-01-16T00:13:20.248 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9718000000000001e-05 2020-01-16T00:13:20.248 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.248 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.248 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:20.248 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986318+0000", 2020-01-16T00:13:20.248 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.4349999999999997e-06 2020-01-16T00:13:20.248 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.248 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.249 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.249 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986326+0000", 2020-01-16T00:13:20.249 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3298999999999999e-05 2020-01-16T00:13:20.249 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.249 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.249 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:20.249 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986380+0000", 2020-01-16T00:13:20.249 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4969999999999995e-05 2020-01-16T00:13:20.250 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.250 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.250 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:20.250 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986455+0000", 2020-01-16T00:13:20.250 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000110587 2020-01-16T00:13:20.250 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.250 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.251 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:20.251 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986565+0000", 2020-01-16T00:13:20.251 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063931600000000002 2020-01-16T00:13:20.251 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.251 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.251 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:20.252 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.992958+0000", 2020-01-16T00:13:20.252 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013008480000000001 2020-01-16T00:13:20.253 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.253 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.253 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.253 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.994259+0000", 2020-01-16T00:13:20.253 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048399979999999999 2020-01-16T00:13:20.253 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.253 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.254 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.254 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999099+0000", 2020-01-16T00:13:20.254 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1874999999999998e-05 2020-01-16T00:13:20.254 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.254 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.254 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:20.254 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999181+0000", 2020-01-16T00:13:20.254 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8865e-05 2020-01-16T00:13:20.255 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.255 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.255 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:20.255 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999240+0000", 2020-01-16T00:13:20.255 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013746267 2020-01-16T00:13:20.255 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.255 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:20.256 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.256 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.256 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.256 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:71 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e35)", 2020-01-16T00:13:20.256 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.986341+0000", 2020-01-16T00:13:20.256 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 14.148685597, 2020-01-16T00:13:20.256 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013252807, 2020-01-16T00:13:20.256 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:20.257 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:20.257 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:20.257 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.257 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:20.257 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986341+0000", 2020-01-16T00:13:20.257 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:20.257 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.258 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.258 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:20.258 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986341+0000", 2020-01-16T00:13:20.258 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:20.258 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.258 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.258 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:20.258 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986340+0000", 2020-01-16T00:13:20.259 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000103867 2020-01-16T00:13:20.259 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.259 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.259 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:20.259 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986444+0000", 2020-01-16T00:13:20.259 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0104e-05 2020-01-16T00:13:20.259 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.259 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.260 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:20.260 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986454+0000", 2020-01-16T00:13:20.260 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4220000000000002e-06 2020-01-16T00:13:20.260 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.260 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.260 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.260 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986458+0000", 2020-01-16T00:13:20.261 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033523070000000001 2020-01-16T00:13:20.261 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.261 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.261 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:20.261 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.989811+0000", 2020-01-16T00:13:20.261 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3296999999999999e-05 2020-01-16T00:13:20.261 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.261 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.262 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:20.262 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.989894+0000", 2020-01-16T00:13:20.262 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000117701 2020-01-16T00:13:20.262 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.262 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.262 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:20.262 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.990012+0000", 2020-01-16T00:13:20.262 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0032873490000000002 2020-01-16T00:13:20.263 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.263 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.263 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:20.263 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993299+0000", 2020-01-16T00:13:20.263 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002422619 2020-01-16T00:13:20.263 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.263 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.264 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.264 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.995722+0000", 2020-01-16T00:13:20.264 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0037839589999999999 2020-01-16T00:13:20.264 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.264 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.264 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.264 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999506+0000", 2020-01-16T00:13:20.264 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8559999999999997e-05 2020-01-16T00:13:20.265 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.265 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.265 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:20.265 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999544+0000", 2020-01-16T00:13:20.265 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9599999999999999e-05 2020-01-16T00:13:20.265 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.265 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.266 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:20.266 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999594+0000", 2020-01-16T00:13:20.266 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013252807 2020-01-16T00:13:20.266 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.266 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:20.266 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.267 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.267 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.267 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:77 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e36)", 2020-01-16T00:13:20.267 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.555854+0000", 2020-01-16T00:13:20.267 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 12.579172507999999, 2020-01-16T00:13:20.267 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010455002999999999, 2020-01-16T00:13:20.267 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:20.268 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:20.268 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:20.268 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.268 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:20.268 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555854+0000", 2020-01-16T00:13:20.269 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:20.269 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.269 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.269 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:20.269 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555854+0000", 2020-01-16T00:13:20.269 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:20.269 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.270 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.270 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:20.270 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555851+0000", 2020-01-16T00:13:20.270 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00048310200000000001 2020-01-16T00:13:20.270 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.270 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.270 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:20.270 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556334+0000", 2020-01-16T00:13:20.271 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2232e-05 2020-01-16T00:13:20.271 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.271 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.271 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:20.271 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556356+0000", 2020-01-16T00:13:20.271 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9750000000000007e-06 2020-01-16T00:13:20.271 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.271 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.272 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.272 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556364+0000", 2020-01-16T00:13:20.272 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8872999999999997e-05 2020-01-16T00:13:20.272 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.272 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.272 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:20.272 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556423+0000", 2020-01-16T00:13:20.273 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8687000000000001e-05 2020-01-16T00:13:20.273 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.273 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.273 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:20.273 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556502+0000", 2020-01-16T00:13:20.273 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013235300000000001 2020-01-16T00:13:20.273 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.273 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.274 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:20.274 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556634+0000", 2020-01-16T00:13:20.274 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0059770520000000001 2020-01-16T00:13:20.274 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.274 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.274 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:20.274 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.562611+0000", 2020-01-16T00:13:20.275 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029492820000000001 2020-01-16T00:13:20.275 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.275 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.275 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.276 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.565561+0000", 2020-01-16T00:13:20.276 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00060620300000000004 2020-01-16T00:13:20.276 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.276 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.276 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.276 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566167+0000", 2020-01-16T00:13:20.276 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3831000000000004e-05 2020-01-16T00:13:20.276 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.277 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.277 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:20.277 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566251+0000", 2020-01-16T00:13:20.277 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8328000000000001e-05 2020-01-16T00:13:20.277 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.277 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.277 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:20.278 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566309+0000", 2020-01-16T00:13:20.278 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010455002999999999 2020-01-16T00:13:20.278 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.278 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:20.278 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.278 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.278 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.278 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:78 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e36)", 2020-01-16T00:13:20.279 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.556378+0000", 2020-01-16T00:13:20.279 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 12.578648575000001, 2020-01-16T00:13:20.279 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010396311, 2020-01-16T00:13:20.279 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:20.279 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:20.279 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:20.279 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.280 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:20.280 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556378+0000", 2020-01-16T00:13:20.280 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:20.280 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.280 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.280 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:20.280 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556378+0000", 2020-01-16T00:13:20.280 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:20.281 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.281 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.281 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:20.281 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556377+0000", 2020-01-16T00:13:20.281 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4307000000000001e-05 2020-01-16T00:13:20.281 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.281 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.281 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:20.282 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556422+0000", 2020-01-16T00:13:20.282 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9620000000000005e-06 2020-01-16T00:13:20.282 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.282 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.282 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:20.282 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556432+0000", 2020-01-16T00:13:20.282 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5979999999999999e-06 2020-01-16T00:13:20.283 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.283 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.283 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.283 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556436+0000", 2020-01-16T00:13:20.283 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021059469999999999 2020-01-16T00:13:20.283 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.283 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.283 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:20.284 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.558542+0000", 2020-01-16T00:13:20.284 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000194458 2020-01-16T00:13:20.284 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.284 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.284 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:20.284 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.558737+0000", 2020-01-16T00:13:20.284 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00032867900000000002 2020-01-16T00:13:20.284 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.285 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.285 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:20.285 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.559065+0000", 2020-01-16T00:13:20.285 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052547499999999999 2020-01-16T00:13:20.285 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.285 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.285 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:20.286 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.564320+0000", 2020-01-16T00:13:20.286 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001565213 2020-01-16T00:13:20.286 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.286 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.286 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.286 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.565885+0000", 2020-01-16T00:13:20.286 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00076994300000000004 2020-01-16T00:13:20.286 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.287 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.287 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.287 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566655+0000", 2020-01-16T00:13:20.287 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7533000000000005e-05 2020-01-16T00:13:20.287 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.287 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.287 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:20.287 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566723+0000", 2020-01-16T00:13:20.288 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1616999999999997e-05 2020-01-16T00:13:20.288 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.288 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.288 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:20.288 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566774+0000", 2020-01-16T00:13:20.288 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010396311 2020-01-16T00:13:20.288 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.289 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:20.289 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.289 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.289 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.289 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:72 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 56832~1408000] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e37)", 2020-01-16T00:13:20.289 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.578753+0000", 2020-01-16T00:13:20.289 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 7.5562732720000003, 2020-01-16T00:13:20.289 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011826918, 2020-01-16T00:13:20.290 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:20.290 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:20.290 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:20.290 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.290 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:20.290 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578753+0000", 2020-01-16T00:13:20.290 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:20.291 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.291 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.291 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:20.291 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578753+0000", 2020-01-16T00:13:20.291 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:20.291 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.291 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.291 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:20.292 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578751+0000", 2020-01-16T00:13:20.292 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00042457699999999998 2020-01-16T00:13:20.292 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.292 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.292 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:20.292 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579175+0000", 2020-01-16T00:13:20.292 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5404999999999999e-05 2020-01-16T00:13:20.292 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.293 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.293 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:20.293 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579191+0000", 2020-01-16T00:13:20.293 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6149999999999996e-06 2020-01-16T00:13:20.293 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.293 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.293 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.294 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579196+0000", 2020-01-16T00:13:20.294 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000104794 2020-01-16T00:13:20.294 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.294 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.294 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:20.294 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579301+0000", 2020-01-16T00:13:20.294 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020947199999999999 2020-01-16T00:13:20.294 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.295 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.295 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:20.295 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579511+0000", 2020-01-16T00:13:20.295 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034328400000000002 2020-01-16T00:13:20.295 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.295 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.296 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:20.296 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579854+0000", 2020-01-16T00:13:20.296 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.005078117 2020-01-16T00:13:20.296 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.296 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.296 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:20.296 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.584932+0000", 2020-01-16T00:13:20.297 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0014702160000000001 2020-01-16T00:13:20.297 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.297 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.297 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.297 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.586402+0000", 2020-01-16T00:13:20.297 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040623589999999998 2020-01-16T00:13:20.297 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.297 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.298 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.298 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590465+0000", 2020-01-16T00:13:20.298 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4478000000000002e-05 2020-01-16T00:13:20.298 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.298 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.298 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:20.298 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590529+0000", 2020-01-16T00:13:20.298 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1121000000000001e-05 2020-01-16T00:13:20.299 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.299 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.299 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:20.299 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590580+0000", 2020-01-16T00:13:20.299 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011826918 2020-01-16T00:13:20.299 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.299 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:20.300 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.300 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.300 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.300 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:73 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 1922048~118272] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e37)", 2020-01-16T00:13:20.300 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:13:20.300 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 7.555820475, 2020-01-16T00:13:20.300 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011750764, 2020-01-16T00:13:20.300 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:20.301 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:20.301 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:20.301 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.301 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:20.301 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:13:20.301 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:20.301 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.302 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.302 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:20.302 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:13:20.302 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:20.302 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.302 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.302 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:20.302 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:13:20.303 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2780000000000001e-05 2020-01-16T00:13:20.303 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.303 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.303 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:20.303 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579238+0000", 2020-01-16T00:13:20.303 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4341999999999999e-05 2020-01-16T00:13:20.303 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.303 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.304 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:20.304 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579253+0000", 2020-01-16T00:13:20.304 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1920000000000001e-06 2020-01-16T00:13:20.304 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.304 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.304 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.304 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579256+0000", 2020-01-16T00:13:20.305 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003224902 2020-01-16T00:13:20.305 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.305 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.305 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:20.305 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.582481+0000", 2020-01-16T00:13:20.305 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000180729 2020-01-16T00:13:20.305 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.305 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.306 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:20.306 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.582662+0000", 2020-01-16T00:13:20.306 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00025106100000000001 2020-01-16T00:13:20.306 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.306 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.306 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:20.306 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.582913+0000", 2020-01-16T00:13:20.306 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020814449999999999 2020-01-16T00:13:20.307 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.307 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.307 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:20.307 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.584994+0000", 2020-01-16T00:13:20.307 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001717243 2020-01-16T00:13:20.307 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.307 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.308 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.308 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.586711+0000", 2020-01-16T00:13:20.308 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041349869999999997 2020-01-16T00:13:20.308 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.308 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.308 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.308 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590846+0000", 2020-01-16T00:13:20.308 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3565000000000006e-05 2020-01-16T00:13:20.309 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.309 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.309 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:20.309 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590910+0000", 2020-01-16T00:13:20.309 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7002999999999997e-05 2020-01-16T00:13:20.309 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.309 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.309 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:20.310 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590957+0000", 2020-01-16T00:13:20.310 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011750764 2020-01-16T00:13:20.310 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.310 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:20.310 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.310 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.310 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.311 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:89 2.1 2:9b8617ce:::smithi12313934-38 low_tier:head [write 69632~773120] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e37)", 2020-01-16T00:13:20.311 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.593701+0000", 2020-01-16T00:13:20.311 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 7.5413258489999997, 2020-01-16T00:13:20.311 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010026929, 2020-01-16T00:13:20.311 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:20.311 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:20.311 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:20.311 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.312 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:20.312 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593701+0000", 2020-01-16T00:13:20.312 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:20.312 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.312 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.312 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:20.312 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593701+0000", 2020-01-16T00:13:20.312 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:20.313 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.313 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.313 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:20.313 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593699+0000", 2020-01-16T00:13:20.313 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000237564 2020-01-16T00:13:20.313 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.313 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.314 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:20.314 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593936+0000", 2020-01-16T00:13:20.314 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2666e-05 2020-01-16T00:13:20.314 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.314 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.314 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:20.314 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593949+0000", 2020-01-16T00:13:20.314 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0089999999999998e-06 2020-01-16T00:13:20.315 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.315 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.315 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.315 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593954+0000", 2020-01-16T00:13:20.315 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5813000000000003e-05 2020-01-16T00:13:20.315 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.315 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.315 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:20.316 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.594000+0000", 2020-01-16T00:13:20.316 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010155299999999999 2020-01-16T00:13:20.316 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.316 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.316 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:20.316 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.594101+0000", 2020-01-16T00:13:20.316 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000203807 2020-01-16T00:13:20.316 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.317 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.317 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:20.317 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.594305+0000", 2020-01-16T00:13:20.317 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021655390000000002 2020-01-16T00:13:20.317 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.317 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.317 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.318 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.596471+0000", 2020-01-16T00:13:20.318 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0070645320000000001 2020-01-16T00:13:20.318 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.318 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.318 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.318 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603535+0000", 2020-01-16T00:13:20.318 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7577999999999998e-05 2020-01-16T00:13:20.319 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.319 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.319 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:20.319 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603573+0000", 2020-01-16T00:13:20.319 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9933999999999997e-05 2020-01-16T00:13:20.319 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.319 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.319 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:20.320 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603653+0000", 2020-01-16T00:13:20.320 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4993000000000005e-05 2020-01-16T00:13:20.320 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.320 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.320 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:20.320 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603728+0000", 2020-01-16T00:13:20.320 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010026929 2020-01-16T00:13:20.320 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.321 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:20.321 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.321 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.321 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.321 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:91 2.1 2:9b8617ce:::smithi12313934-38 low_tier:head [write 1915392~590334] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e37)", 2020-01-16T00:13:20.321 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.596978+0000", 2020-01-16T00:13:20.321 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 7.538048152, 2020-01-16T00:13:20.322 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0079576309999999997, 2020-01-16T00:13:20.322 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:20.322 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:20.322 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:20.322 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.322 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:20.322 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.596978+0000", 2020-01-16T00:13:20.322 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:20.323 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.323 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.323 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:20.323 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.596978+0000", 2020-01-16T00:13:20.323 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:20.323 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.323 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.323 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:20.324 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.596978+0000", 2020-01-16T00:13:20.324 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000170557 2020-01-16T00:13:20.324 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.324 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.324 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:20.324 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.597148+0000", 2020-01-16T00:13:20.324 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5519999999999999e-06 2020-01-16T00:13:20.325 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.325 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.325 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:20.325 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.597156+0000", 2020-01-16T00:13:20.325 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7520000000000004e-06 2020-01-16T00:13:20.325 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.325 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.325 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.326 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.597161+0000", 2020-01-16T00:13:20.326 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0028888260000000002 2020-01-16T00:13:20.326 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.326 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.326 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:20.326 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.600049+0000", 2020-01-16T00:13:20.326 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7899999999999994e-05 2020-01-16T00:13:20.327 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.327 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.327 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:20.327 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.600147+0000", 2020-01-16T00:13:20.327 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000169877 2020-01-16T00:13:20.327 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.327 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.327 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:20.328 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.600317+0000", 2020-01-16T00:13:20.328 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0043677259999999997 2020-01-16T00:13:20.328 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.328 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.328 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.328 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.604685+0000", 2020-01-16T00:13:20.328 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013208999999999999 2020-01-16T00:13:20.328 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.329 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.329 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.329 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.604817+0000", 2020-01-16T00:13:20.329 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4851000000000001e-05 2020-01-16T00:13:20.329 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.329 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.329 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:20.330 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.604832+0000", 2020-01-16T00:13:20.330 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8831e-05 2020-01-16T00:13:20.330 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.330 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.330 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:20.330 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.604891+0000", 2020-01-16T00:13:20.330 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5411000000000003e-05 2020-01-16T00:13:20.330 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.331 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.331 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:20.331 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.604936+0000", 2020-01-16T00:13:20.331 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0079576309999999997 2020-01-16T00:13:20.331 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.331 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:20.331 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.332 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.332 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.332 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:74 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 56832~1408000] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e41)", 2020-01-16T00:13:20.332 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:16.601158+0000", 2020-01-16T00:13:20.332 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 3.5338687310000001, 2020-01-16T00:13:20.332 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0076921350000000001, 2020-01-16T00:13:20.332 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:20.332 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:20.333 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:20.333 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.333 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:20.333 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.601158+0000", 2020-01-16T00:13:20.333 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:20.333 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.333 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.333 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:20.334 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.601158+0000", 2020-01-16T00:13:20.334 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:20.334 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.334 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.334 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:20.334 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.601156+0000", 2020-01-16T00:13:20.334 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00042862199999999998 2020-01-16T00:13:20.335 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.335 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.335 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:20.335 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.601585+0000", 2020-01-16T00:13:20.335 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8362000000000001e-05 2020-01-16T00:13:20.335 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.335 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.335 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:20.336 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.601603+0000", 2020-01-16T00:13:20.336 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9269999999999999e-06 2020-01-16T00:13:20.336 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.336 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.336 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.336 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.601610+0000", 2020-01-16T00:13:20.336 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7003000000000003e-05 2020-01-16T00:13:20.336 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.337 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.337 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:20.337 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.601667+0000", 2020-01-16T00:13:20.337 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112724 2020-01-16T00:13:20.337 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.337 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.337 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:20.338 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.601780+0000", 2020-01-16T00:13:20.338 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000187176 2020-01-16T00:13:20.338 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.338 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.338 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:20.338 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.601967+0000", 2020-01-16T00:13:20.338 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0065865639999999996 2020-01-16T00:13:20.338 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.339 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.339 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:20.339 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.608553+0000", 2020-01-16T00:13:20.339 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6983000000000006e-05 2020-01-16T00:13:20.339 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.339 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.339 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.339 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.608620+0000", 2020-01-16T00:13:20.340 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9728000000000006e-05 2020-01-16T00:13:20.340 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.340 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.340 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.340 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.608710+0000", 2020-01-16T00:13:20.340 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0473999999999997e-05 2020-01-16T00:13:20.340 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.341 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.341 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:20.341 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.608771+0000", 2020-01-16T00:13:20.341 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9382000000000007e-05 2020-01-16T00:13:20.341 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.341 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.341 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:20.341 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.608850+0000", 2020-01-16T00:13:20.342 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0076921350000000001 2020-01-16T00:13:20.342 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.342 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:20.342 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.342 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.342 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.342 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:75 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 1922048~118272] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e41)", 2020-01-16T00:13:20.343 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:16.601624+0000", 2020-01-16T00:13:20.343 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 3.533402073, 2020-01-16T00:13:20.343 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0079729169999999995, 2020-01-16T00:13:20.343 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:20.343 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:20.343 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:20.343 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.343 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:20.344 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.601624+0000", 2020-01-16T00:13:20.344 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:20.344 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.344 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.344 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:20.344 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.601624+0000", 2020-01-16T00:13:20.344 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:20.344 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.345 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.345 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:20.345 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.601624+0000", 2020-01-16T00:13:20.345 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3544999999999997e-05 2020-01-16T00:13:20.345 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.345 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.345 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:20.346 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.601667+0000", 2020-01-16T00:13:20.346 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2656999999999999e-05 2020-01-16T00:13:20.346 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.346 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.346 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:20.346 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.601680+0000", 2020-01-16T00:13:20.346 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9760000000000003e-06 2020-01-16T00:13:20.346 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.347 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.347 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.347 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.601685+0000", 2020-01-16T00:13:20.347 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0026060940000000002 2020-01-16T00:13:20.347 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.347 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.347 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:20.347 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.604291+0000", 2020-01-16T00:13:20.348 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016571000000000001 2020-01-16T00:13:20.348 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.348 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.348 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:20.348 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.604457+0000", 2020-01-16T00:13:20.348 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017136699999999999 2020-01-16T00:13:20.348 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.348 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.349 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:20.349 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.604628+0000", 2020-01-16T00:13:20.349 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040386509999999999 2020-01-16T00:13:20.349 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.349 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.349 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.349 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.608667+0000", 2020-01-16T00:13:20.350 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00040341800000000001 2020-01-16T00:13:20.350 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.350 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.350 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:20.350 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.609070+0000", 2020-01-16T00:13:20.350 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00039049100000000003 2020-01-16T00:13:20.350 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.350 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.351 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.351 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.609461+0000", 2020-01-16T00:13:20.351 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4151000000000002e-05 2020-01-16T00:13:20.351 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.351 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.351 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:20.351 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.609535+0000", 2020-01-16T00:13:20.351 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2636000000000001e-05 2020-01-16T00:13:20.352 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:20.352 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:20.352 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:20.352 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:16.609597+0000", 2020-01-16T00:13:20.352 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0079729169999999995 2020-01-16T00:13:20.352 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.352 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:20.353 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.353 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:20.353 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:20.353 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:20.355 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-01-16T00:13:20.355 INFO:tasks.rados.rados.0.smithi123.stdout:1176: expect (ObjNum 329 snap 0 seq_num 329) 2020-01-16T00:13:20.355 INFO:tasks.rados.rados.0.smithi123.stdout:1175: finishing write tid 3 to smithi12313934-21 2020-01-16T00:13:20.355 INFO:tasks.rados.rados.0.smithi123.stdout:1175: finishing write tid 5 to smithi12313934-21 2020-01-16T00:13:20.356 INFO:tasks.rados.rados.0.smithi123.stdout:1175: finishing write tid 6 to smithi12313934-21 2020-01-16T00:13:20.356 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 21 v 45 (ObjNum 345 snap 0 seq_num 345) dirty exists 2020-01-16T00:13:20.356 INFO:tasks.rados.rados.0.smithi123.stdout:1175: left oid 21 (ObjNum 345 snap 0 seq_num 345) 2020-01-16T00:13:20.356 INFO:tasks.rados.rados.0.smithi123.stdout:1180: expect (ObjNum 298 snap 0 seq_num 298) 2020-01-16T00:13:20.356 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:20.350+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:20.455 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:20.450+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:20.473 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:20.473 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:13:20.474 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:13:20.474 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:20.484 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-01-16T00:13:20.490 INFO:tasks.rados.rados.0.smithi123.stdout:1179: expect (ObjNum 219 snap 0 seq_num 219) 2020-01-16T00:13:20.556 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:20.554+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:20.592 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:20.593 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:13:20.593 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:13:20.593 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:13:20.593 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:20.602 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-01-16T00:13:20.657 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:20.652+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:20.713 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:20.713 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:13:20.713 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:13:20.713 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:13:20.713 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.714 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:895 3.3 3:cd231094:::smithi12313934-49:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:20.714 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.122390+0000", 2020-01-16T00:13:20.714 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 43.585373406999999, 2020-01-16T00:13:20.714 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014295501, 2020-01-16T00:13:20.714 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:20.714 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:20.714 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:20.714 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:20.715 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:20.715 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 895 2020-01-16T00:13:20.715 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.715 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:20.715 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.715 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:20.715 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122390+0000", 2020-01-16T00:13:20.716 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:20.716 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.716 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.716 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:20.716 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122390+0000", 2020-01-16T00:13:20.716 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999957 2020-01-16T00:13:20.716 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.716 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.717 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:20.717 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122386+0000", 2020-01-16T00:13:20.717 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0656999999999999e-05 2020-01-16T00:13:20.717 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.717 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.717 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:20.717 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122407+0000", 2020-01-16T00:13:20.717 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9102999999999999e-05 2020-01-16T00:13:20.718 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.718 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.718 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:20.718 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122436+0000", 2020-01-16T00:13:20.718 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.774e-05 2020-01-16T00:13:20.718 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.718 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.718 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.719 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122483+0000", 2020-01-16T00:13:20.719 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5185000000000004e-05 2020-01-16T00:13:20.719 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.719 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.719 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.719 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122579+0000", 2020-01-16T00:13:20.719 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000304085 2020-01-16T00:13:20.719 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.720 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.720 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:20.720 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122883+0000", 2020-01-16T00:13:20.720 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0092077970000000002 2020-01-16T00:13:20.720 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.720 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.720 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.721 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132091+0000", 2020-01-16T00:13:20.721 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000170025 2020-01-16T00:13:20.721 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.721 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.721 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:20.721 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132261+0000", 2020-01-16T00:13:20.721 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0040130979999999997 2020-01-16T00:13:20.721 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.722 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.722 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.722 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136274+0000", 2020-01-16T00:13:20.722 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014227900000000001 2020-01-16T00:13:20.722 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.722 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.722 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:20.722 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136416+0000", 2020-01-16T00:13:20.723 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00026995200000000001 2020-01-16T00:13:20.723 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.723 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.723 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:20.723 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136686+0000", 2020-01-16T00:13:20.723 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014295501 2020-01-16T00:13:20.723 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.724 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:20.724 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.724 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.724 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.724 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:896 3.3 3:cd231094:::smithi12313934-49:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:20.724 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.122539+0000", 2020-01-16T00:13:20.724 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 43.585224406999998, 2020-01-16T00:13:20.724 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014440173000000001, 2020-01-16T00:13:20.725 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:20.725 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:20.725 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:20.725 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:20.725 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:20.725 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 896 2020-01-16T00:13:20.725 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.725 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:20.726 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.726 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:20.726 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122539+0000", 2020-01-16T00:13:20.726 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:20.726 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.726 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.726 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:20.727 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122539+0000", 2020-01-16T00:13:20.727 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:20.727 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.727 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.727 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:20.727 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122537+0000", 2020-01-16T00:13:20.727 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2432999999999999e-05 2020-01-16T00:13:20.727 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.728 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.728 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:20.728 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122550+0000", 2020-01-16T00:13:20.728 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9369999999999999e-05 2020-01-16T00:13:20.728 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.728 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.728 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:20.728 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122569+0000", 2020-01-16T00:13:20.729 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1747e-05 2020-01-16T00:13:20.729 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.729 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.729 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.729 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122581+0000", 2020-01-16T00:13:20.729 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00041555800000000001 2020-01-16T00:13:20.729 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.729 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.730 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.730 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122996+0000", 2020-01-16T00:13:20.730 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016840299999999999 2020-01-16T00:13:20.730 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.730 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.730 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:20.730 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.123165+0000", 2020-01-16T00:13:20.731 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0091641359999999998 2020-01-16T00:13:20.731 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.731 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.731 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.731 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132329+0000", 2020-01-16T00:13:20.731 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013433400000000001 2020-01-16T00:13:20.731 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.731 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.732 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:20.732 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132463+0000", 2020-01-16T00:13:20.732 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0042524110000000002 2020-01-16T00:13:20.732 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.732 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.732 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.732 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136716+0000", 2020-01-16T00:13:20.732 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4591999999999993e-05 2020-01-16T00:13:20.733 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.733 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.733 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:20.733 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136790+0000", 2020-01-16T00:13:20.733 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018927500000000001 2020-01-16T00:13:20.733 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.733 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.733 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:20.734 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136980+0000", 2020-01-16T00:13:20.734 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014440173000000001 2020-01-16T00:13:20.734 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.734 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:20.734 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.734 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.734 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.734 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:897 3.3 3:cd231094:::smithi12313934-49:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:20.735 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.122613+0000", 2020-01-16T00:13:20.735 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 43.585151263, 2020-01-16T00:13:20.735 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014606846, 2020-01-16T00:13:20.735 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:20.735 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:20.735 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:20.735 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:20.735 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:20.736 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 897 2020-01-16T00:13:20.736 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.736 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:20.736 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.736 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:20.736 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122613+0000", 2020-01-16T00:13:20.736 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:20.736 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.737 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.737 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:20.737 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122613+0000", 2020-01-16T00:13:20.737 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:20.737 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.737 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.737 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:20.737 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122611+0000", 2020-01-16T00:13:20.738 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3670000000000003e-06 2020-01-16T00:13:20.738 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.738 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.738 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:20.738 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122620+0000", 2020-01-16T00:13:20.738 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5193e-05 2020-01-16T00:13:20.738 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.739 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.739 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:20.739 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122635+0000", 2020-01-16T00:13:20.739 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0696999999999999e-05 2020-01-16T00:13:20.739 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.739 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.739 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.739 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.122646+0000", 2020-01-16T00:13:20.740 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00059961400000000005 2020-01-16T00:13:20.740 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.740 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.740 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.740 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.123246+0000", 2020-01-16T00:13:20.740 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011189200000000001 2020-01-16T00:13:20.740 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.740 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.741 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:20.741 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.123358+0000", 2020-01-16T00:13:20.741 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0091363550000000005 2020-01-16T00:13:20.741 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.741 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.741 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.741 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132494+0000", 2020-01-16T00:13:20.742 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2956999999999998e-05 2020-01-16T00:13:20.742 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.742 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.742 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:20.742 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.132587+0000", 2020-01-16T00:13:20.742 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0044061769999999998 2020-01-16T00:13:20.742 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.742 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.743 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.743 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.136993+0000", 2020-01-16T00:13:20.743 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7822999999999998e-05 2020-01-16T00:13:20.743 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.743 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.743 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:20.743 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.137051+0000", 2020-01-16T00:13:20.743 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000168498 2020-01-16T00:13:20.744 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.744 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.744 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:20.744 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.137219+0000", 2020-01-16T00:13:20.744 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014606846 2020-01-16T00:13:20.744 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.744 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:20.744 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.745 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.745 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.745 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:943 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:20.745 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:13:20.745 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 42.793285289000004, 2020-01-16T00:13:20.745 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018847004000000001, 2020-01-16T00:13:20.745 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:20.745 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:20.746 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:20.746 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:20.746 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:20.746 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 943 2020-01-16T00:13:20.746 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.746 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:20.746 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.746 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:20.747 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:13:20.747 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:20.747 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.747 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.747 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:20.747 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:13:20.747 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:20.747 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.748 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.748 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:20.748 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914475+0000", 2020-01-16T00:13:20.748 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.508e-05 2020-01-16T00:13:20.748 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.748 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.748 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:20.749 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914490+0000", 2020-01-16T00:13:20.749 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.857e-05 2020-01-16T00:13:20.749 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.749 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.749 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:20.749 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914509+0000", 2020-01-16T00:13:20.749 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.46e-06 2020-01-16T00:13:20.749 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.750 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.750 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.750 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914515+0000", 2020-01-16T00:13:20.750 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2612999999999999e-05 2020-01-16T00:13:20.750 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.750 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.750 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.750 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914568+0000", 2020-01-16T00:13:20.751 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028312299999999998 2020-01-16T00:13:20.751 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.751 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.751 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:20.751 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914851+0000", 2020-01-16T00:13:20.751 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012282533 2020-01-16T00:13:20.751 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.751 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.752 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.752 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927133+0000", 2020-01-16T00:13:20.752 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4839999999999995e-05 2020-01-16T00:13:20.752 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.752 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.752 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:20.752 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927208+0000", 2020-01-16T00:13:20.752 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0047425610000000002 2020-01-16T00:13:20.753 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.753 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.753 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.753 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.931951+0000", 2020-01-16T00:13:20.753 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3628999999999994e-05 2020-01-16T00:13:20.753 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.753 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.753 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:20.754 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.932045+0000", 2020-01-16T00:13:20.754 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012810180000000001 2020-01-16T00:13:20.754 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.754 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.754 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:20.754 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933326+0000", 2020-01-16T00:13:20.754 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018847004000000001 2020-01-16T00:13:20.754 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.755 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:20.755 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.755 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.755 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.755 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:944 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:20.755 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:13:20.755 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 42.793235733000003, 2020-01-16T00:13:20.756 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019086901999999999, 2020-01-16T00:13:20.756 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:20.756 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:20.756 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:20.756 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:20.756 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:20.756 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 944 2020-01-16T00:13:20.756 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.757 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:20.757 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.757 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:20.757 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:13:20.757 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:20.757 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.758 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.758 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:20.758 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:13:20.758 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:20.758 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.758 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.758 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:20.758 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914527+0000", 2020-01-16T00:13:20.759 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7509999999999999e-06 2020-01-16T00:13:20.759 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.759 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.759 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:20.759 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914532+0000", 2020-01-16T00:13:20.759 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5149999999999997e-06 2020-01-16T00:13:20.759 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.759 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.760 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:20.760 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914538+0000", 2020-01-16T00:13:20.760 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1890000000000007e-06 2020-01-16T00:13:20.760 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.760 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.760 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.760 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914546+0000", 2020-01-16T00:13:20.760 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000354951 2020-01-16T00:13:20.761 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.761 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.761 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.761 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914901+0000", 2020-01-16T00:13:20.761 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1594000000000002e-05 2020-01-16T00:13:20.761 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.761 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.761 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:20.762 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914942+0000", 2020-01-16T00:13:20.762 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012300070999999999 2020-01-16T00:13:20.762 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.762 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.762 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.762 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927242+0000", 2020-01-16T00:13:20.762 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4076999999999997e-05 2020-01-16T00:13:20.762 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.763 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.763 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:20.763 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927296+0000", 2020-01-16T00:13:20.763 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0060609180000000002 2020-01-16T00:13:20.763 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.763 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.763 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.764 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933357+0000", 2020-01-16T00:13:20.764 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4488999999999999e-05 2020-01-16T00:13:20.764 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.764 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.764 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:20.764 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933422+0000", 2020-01-16T00:13:20.764 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019318300000000001 2020-01-16T00:13:20.764 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.765 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.765 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:20.765 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933615+0000", 2020-01-16T00:13:20.765 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019086901999999999 2020-01-16T00:13:20.765 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.765 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:20.765 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.765 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.766 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.766 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:945 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:20.766 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:13:20.766 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 42.793208526999997, 2020-01-16T00:13:20.766 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019342986, 2020-01-16T00:13:20.766 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:20.766 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:20.767 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:20.767 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:20.767 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:20.767 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 945 2020-01-16T00:13:20.767 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.767 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:20.768 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.768 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:20.768 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:13:20.768 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:20.768 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.768 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.768 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:20.768 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:13:20.769 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:20.769 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.769 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.769 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:20.769 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:13:20.769 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4580000000000002e-06 2020-01-16T00:13:20.769 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.770 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.770 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:20.770 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914558+0000", 2020-01-16T00:13:20.770 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7829999999999998e-06 2020-01-16T00:13:20.770 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.771 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.771 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:20.771 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914563+0000", 2020-01-16T00:13:20.771 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0001e-05 2020-01-16T00:13:20.771 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.771 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.771 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.772 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914573+0000", 2020-01-16T00:13:20.772 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00040365600000000001 2020-01-16T00:13:20.772 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.772 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.772 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.772 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914976+0000", 2020-01-16T00:13:20.772 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1390000000000001e-05 2020-01-16T00:13:20.772 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.773 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.773 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:20.773 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.915028+0000", 2020-01-16T00:13:20.773 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012288843000000001 2020-01-16T00:13:20.773 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.773 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.774 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.774 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927317+0000", 2020-01-16T00:13:20.774 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7871000000000002e-05 2020-01-16T00:13:20.774 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.774 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.774 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:20.774 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927375+0000", 2020-01-16T00:13:20.774 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0062580589999999998 2020-01-16T00:13:20.775 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.775 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.775 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.775 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933633+0000", 2020-01-16T00:13:20.775 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7086999999999997e-05 2020-01-16T00:13:20.775 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.775 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.776 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:20.776 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933690+0000", 2020-01-16T00:13:20.776 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020855799999999999 2020-01-16T00:13:20.776 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.776 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.776 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:20.776 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933898+0000", 2020-01-16T00:13:20.776 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019342986 2020-01-16T00:13:20.777 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.777 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:20.777 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.777 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.777 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.777 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:963 3.9 3:9557b633:::smithi12313934-43:head [read 0~2705894,checksum] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:20.777 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.046589+0000", 2020-01-16T00:13:20.778 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 42.661174678999998, 2020-01-16T00:13:20.778 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014014016000000001, 2020-01-16T00:13:20.778 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:20.778 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:20.778 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:20.778 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:20.778 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:20.778 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 963 2020-01-16T00:13:20.779 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.779 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:20.779 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.779 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:20.779 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046589+0000", 2020-01-16T00:13:20.779 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:20.779 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.779 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.780 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:20.780 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046589+0000", 2020-01-16T00:13:20.780 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:20.780 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.780 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.780 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:20.780 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046588+0000", 2020-01-16T00:13:20.781 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7469999999999994e-06 2020-01-16T00:13:20.781 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.781 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.781 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:20.781 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046596+0000", 2020-01-16T00:13:20.781 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3445999999999998e-05 2020-01-16T00:13:20.781 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.781 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.782 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:20.782 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046630+0000", 2020-01-16T00:13:20.782 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7779999999999999e-05 2020-01-16T00:13:20.782 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.782 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.782 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.782 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046648+0000", 2020-01-16T00:13:20.782 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00075081300000000002 2020-01-16T00:13:20.783 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.783 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.783 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.783 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.047398+0000", 2020-01-16T00:13:20.783 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000116698 2020-01-16T00:13:20.783 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.783 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.783 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:20.784 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.047515+0000", 2020-01-16T00:13:20.784 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0056630689999999997 2020-01-16T00:13:20.784 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.784 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.784 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.784 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.053178+0000", 2020-01-16T00:13:20.784 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012084299999999999 2020-01-16T00:13:20.784 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.785 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.785 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:20.785 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.053299+0000", 2020-01-16T00:13:20.785 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0056921510000000003 2020-01-16T00:13:20.785 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.785 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.785 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.786 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.058991+0000", 2020-01-16T00:13:20.786 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000118606 2020-01-16T00:13:20.786 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.786 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.786 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:20.786 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.059110+0000", 2020-01-16T00:13:20.786 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001493306 2020-01-16T00:13:20.786 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.787 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.787 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:20.787 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.060603+0000", 2020-01-16T00:13:20.787 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014014016000000001 2020-01-16T00:13:20.787 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.787 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:20.787 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.788 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.788 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.788 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:964 3.9 3:9557b633:::smithi12313934-43:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:20.788 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.046668+0000", 2020-01-16T00:13:20.788 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 42.661096075000003, 2020-01-16T00:13:20.788 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014425670999999999, 2020-01-16T00:13:20.788 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:20.789 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:20.789 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:20.789 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:20.789 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:20.789 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 964 2020-01-16T00:13:20.789 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.789 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:20.789 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.790 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:20.790 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046668+0000", 2020-01-16T00:13:20.790 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:20.790 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.790 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.790 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:20.790 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046668+0000", 2020-01-16T00:13:20.791 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:20.791 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.791 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.791 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:20.791 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046666+0000", 2020-01-16T00:13:20.791 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0879999999999999e-05 2020-01-16T00:13:20.791 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.791 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.792 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:20.792 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046677+0000", 2020-01-16T00:13:20.792 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5767e-05 2020-01-16T00:13:20.792 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.792 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.792 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:20.792 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046693+0000", 2020-01-16T00:13:20.793 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1399999999999999e-05 2020-01-16T00:13:20.793 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.793 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.793 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.793 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.046704+0000", 2020-01-16T00:13:20.793 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00089331800000000004 2020-01-16T00:13:20.793 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.793 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.794 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.794 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.047597+0000", 2020-01-16T00:13:20.794 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8628000000000005e-05 2020-01-16T00:13:20.794 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.794 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.794 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:20.794 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.047696+0000", 2020-01-16T00:13:20.795 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0056536290000000003 2020-01-16T00:13:20.795 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.795 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.795 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.795 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.053349+0000", 2020-01-16T00:13:20.795 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.6946000000000003e-05 2020-01-16T00:13:20.795 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.795 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.796 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:20.796 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.053446+0000", 2020-01-16T00:13:20.796 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0072058909999999999 2020-01-16T00:13:20.796 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.796 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.796 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.796 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.060652+0000", 2020-01-16T00:13:20.796 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000135597 2020-01-16T00:13:20.797 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.797 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.797 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:20.797 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.060788+0000", 2020-01-16T00:13:20.797 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030548499999999998 2020-01-16T00:13:20.797 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.797 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.798 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:20.798 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.061093+0000", 2020-01-16T00:13:20.798 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014425670999999999 2020-01-16T00:13:20.798 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.798 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:20.798 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.798 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.798 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.799 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:981 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:20.799 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:13:20.799 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 42.392431635000001, 2020-01-16T00:13:20.799 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021222494000000001, 2020-01-16T00:13:20.799 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:20.799 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:20.799 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:20.800 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:20.800 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:20.800 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 981 2020-01-16T00:13:20.800 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.800 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:20.800 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.800 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:20.801 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:13:20.801 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:20.801 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.801 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.801 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:20.801 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:13:20.801 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999948 2020-01-16T00:13:20.802 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.802 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.802 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:20.802 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315327+0000", 2020-01-16T00:13:20.802 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.508e-05 2020-01-16T00:13:20.802 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.802 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.803 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:20.803 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315352+0000", 2020-01-16T00:13:20.803 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1183999999999998e-05 2020-01-16T00:13:20.803 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.803 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.803 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:20.803 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315383+0000", 2020-01-16T00:13:20.803 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7180000000000002e-05 2020-01-16T00:13:20.803 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.804 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.804 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.804 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315401+0000", 2020-01-16T00:13:20.804 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017290599999999999 2020-01-16T00:13:20.804 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.804 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.804 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.804 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315574+0000", 2020-01-16T00:13:20.805 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00047908100000000002 2020-01-16T00:13:20.805 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.805 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.805 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:20.805 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316053+0000", 2020-01-16T00:13:20.805 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011808947 2020-01-16T00:13:20.805 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.806 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.806 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.806 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.327862+0000", 2020-01-16T00:13:20.806 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010067499999999999 2020-01-16T00:13:20.806 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.806 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.806 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:20.806 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.327962+0000", 2020-01-16T00:13:20.807 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0074745139999999998 2020-01-16T00:13:20.807 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.807 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.807 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.807 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.335437+0000", 2020-01-16T00:13:20.807 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000103403 2020-01-16T00:13:20.807 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.807 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.808 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:20.808 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.335540+0000", 2020-01-16T00:13:20.808 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001014544 2020-01-16T00:13:20.808 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.808 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.808 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:20.808 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336555+0000", 2020-01-16T00:13:20.808 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021222494000000001 2020-01-16T00:13:20.809 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.809 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:20.809 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.809 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.809 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.809 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:982 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:20.809 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:13:20.810 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 42.392329111000002, 2020-01-16T00:13:20.810 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.022087721000000001, 2020-01-16T00:13:20.810 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:20.810 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:20.810 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:20.810 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:20.810 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:20.810 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 982 2020-01-16T00:13:20.811 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.811 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:20.811 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.811 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:20.811 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:13:20.811 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:20.811 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.811 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.812 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:20.812 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:13:20.812 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:20.812 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.812 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.812 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:20.812 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315433+0000", 2020-01-16T00:13:20.812 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1347e-05 2020-01-16T00:13:20.813 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.813 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.813 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:20.813 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315445+0000", 2020-01-16T00:13:20.813 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4525999999999999e-05 2020-01-16T00:13:20.813 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.813 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.814 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:20.814 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315459+0000", 2020-01-16T00:13:20.814 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.024e-05 2020-01-16T00:13:20.814 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.814 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.814 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.814 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315470+0000", 2020-01-16T00:13:20.815 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000736468 2020-01-16T00:13:20.815 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.815 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.815 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.815 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316206+0000", 2020-01-16T00:13:20.815 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000132967 2020-01-16T00:13:20.815 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.815 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.816 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:20.816 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316339+0000", 2020-01-16T00:13:20.816 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011701717 2020-01-16T00:13:20.816 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.816 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.816 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.816 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328041+0000", 2020-01-16T00:13:20.816 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000120339 2020-01-16T00:13:20.817 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.817 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.817 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:20.817 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328161+0000", 2020-01-16T00:13:20.817 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0084341260000000001 2020-01-16T00:13:20.817 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.817 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.818 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.818 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336595+0000", 2020-01-16T00:13:20.818 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5247999999999993e-05 2020-01-16T00:13:20.818 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.818 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.818 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:20.818 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336690+0000", 2020-01-16T00:13:20.818 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00083204000000000004 2020-01-16T00:13:20.819 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.819 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.819 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:20.819 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337522+0000", 2020-01-16T00:13:20.819 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.022087721000000001 2020-01-16T00:13:20.819 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.819 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:20.819 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.820 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.820 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.820 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:983 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:20.820 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:13:20.820 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 42.392209401000002, 2020-01-16T00:13:20.820 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023363387999999999, 2020-01-16T00:13:20.820 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:20.820 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:20.821 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:20.821 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:20.821 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:20.821 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 983 2020-01-16T00:13:20.821 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.821 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:20.821 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.821 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:20.822 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:13:20.822 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:20.822 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.822 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.822 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:20.822 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:13:20.822 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:20.823 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.823 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.823 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:20.823 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315553+0000", 2020-01-16T00:13:20.823 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1760000000000004e-06 2020-01-16T00:13:20.823 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.823 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.823 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:20.824 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315560+0000", 2020-01-16T00:13:20.824 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3587000000000001e-05 2020-01-16T00:13:20.824 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.824 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.824 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:20.824 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315574+0000", 2020-01-16T00:13:20.824 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7800000000000003e-06 2020-01-16T00:13:20.824 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.825 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.825 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.825 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315581+0000", 2020-01-16T00:13:20.825 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000823024 2020-01-16T00:13:20.825 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.825 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.825 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.825 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316404+0000", 2020-01-16T00:13:20.826 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2413e-05 2020-01-16T00:13:20.826 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.826 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.826 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:20.826 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316466+0000", 2020-01-16T00:13:20.826 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011738561999999999 2020-01-16T00:13:20.826 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.826 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.827 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.827 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328205+0000", 2020-01-16T00:13:20.827 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6783999999999997e-05 2020-01-16T00:13:20.827 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.827 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.827 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:20.827 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328262+0000", 2020-01-16T00:13:20.827 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0093022560000000001 2020-01-16T00:13:20.828 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.828 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.828 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.828 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337564+0000", 2020-01-16T00:13:20.828 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000183151 2020-01-16T00:13:20.828 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.828 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.828 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:20.829 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337747+0000", 2020-01-16T00:13:20.829 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011707449999999999 2020-01-16T00:13:20.829 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.829 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.829 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:20.829 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.338918+0000", 2020-01-16T00:13:20.829 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023363387999999999 2020-01-16T00:13:20.830 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.830 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:20.830 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.830 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.830 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.830 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:986 3.9 3:93dfe6f1:::smithi12313934-40:head [write 2941380~419987] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:20.830 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:13:20.830 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 42.226909624999998, 2020-01-16T00:13:20.831 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016978400000000001, 2020-01-16T00:13:20.831 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:20.831 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:20.831 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:20.831 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:20.831 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:20.831 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 986 2020-01-16T00:13:20.831 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.832 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:20.832 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.832 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:20.832 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:13:20.832 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:20.832 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.832 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.832 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:20.833 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:13:20.833 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:20.833 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.833 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.833 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:20.833 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480852+0000", 2020-01-16T00:13:20.833 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042503899999999999 2020-01-16T00:13:20.833 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.834 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.834 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:20.834 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481277+0000", 2020-01-16T00:13:20.834 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3097e-05 2020-01-16T00:13:20.834 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.834 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.834 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:20.834 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481290+0000", 2020-01-16T00:13:20.835 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8460000000000002e-06 2020-01-16T00:13:20.835 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.835 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.835 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.835 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481297+0000", 2020-01-16T00:13:20.835 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8443999999999999e-05 2020-01-16T00:13:20.835 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.835 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.836 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.836 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481346+0000", 2020-01-16T00:13:20.836 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022115399999999999 2020-01-16T00:13:20.836 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.836 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.836 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:20.836 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481567+0000", 2020-01-16T00:13:20.836 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012531502 2020-01-16T00:13:20.837 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.837 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.837 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.837 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494098+0000", 2020-01-16T00:13:20.837 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.4462999999999993e-05 2020-01-16T00:13:20.837 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.837 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.837 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:20.838 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494193+0000", 2020-01-16T00:13:20.838 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017600999999999999 2020-01-16T00:13:20.838 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.838 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.838 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:13:20.838 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494369+0000", 2020-01-16T00:13:20.838 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0031739619999999998 2020-01-16T00:13:20.839 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.839 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.839 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:20.839 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497543+0000", 2020-01-16T00:13:20.839 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000175107 2020-01-16T00:13:20.839 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.839 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.839 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.840 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497718+0000", 2020-01-16T00:13:20.840 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5207000000000003e-05 2020-01-16T00:13:20.840 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.840 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.840 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.840 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497763+0000", 2020-01-16T00:13:20.840 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7042999999999998e-05 2020-01-16T00:13:20.840 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.841 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.841 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:20.841 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497800+0000", 2020-01-16T00:13:20.841 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2628e-05 2020-01-16T00:13:20.841 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.841 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.841 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:20.841 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497833+0000", 2020-01-16T00:13:20.842 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016978400000000001 2020-01-16T00:13:20.842 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.842 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:20.842 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.842 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.842 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.842 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:987 3.9 3:93dfe6f1:::smithi12313934-40:head [setxattr _header (56) in=7b,truncate 3361367] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:20.842 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.481307+0000", 2020-01-16T00:13:20.843 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 42.226457258000003, 2020-01-16T00:13:20.843 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016794989, 2020-01-16T00:13:20.843 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:20.843 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:20.843 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:20.843 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:20.843 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:20.843 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 987 2020-01-16T00:13:20.844 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.844 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:20.844 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.844 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:20.844 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481307+0000", 2020-01-16T00:13:20.844 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:20.844 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.844 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.845 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:20.845 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481307+0000", 2020-01-16T00:13:20.845 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:20.845 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.845 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.845 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:20.845 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481306+0000", 2020-01-16T00:13:20.846 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.113e-06 2020-01-16T00:13:20.846 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.846 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.846 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:20.846 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481311+0000", 2020-01-16T00:13:20.846 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1520000000000002e-06 2020-01-16T00:13:20.846 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.846 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.847 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:20.847 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481315+0000", 2020-01-16T00:13:20.847 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.7640000000000001e-06 2020-01-16T00:13:20.847 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.847 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.847 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.847 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481318+0000", 2020-01-16T00:13:20.847 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00029012500000000002 2020-01-16T00:13:20.848 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.848 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.848 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.848 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481608+0000", 2020-01-16T00:13:20.848 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6239000000000002e-05 2020-01-16T00:13:20.848 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.848 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.848 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:20.849 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481654+0000", 2020-01-16T00:13:20.849 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014559534000000001 2020-01-16T00:13:20.849 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.849 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.849 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.849 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.496214+0000", 2020-01-16T00:13:20.849 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000225932 2020-01-16T00:13:20.849 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.850 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.850 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:20.850 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.496439+0000", 2020-01-16T00:13:20.850 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00031099600000000002 2020-01-16T00:13:20.850 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.850 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.850 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:13:20.850 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.496750+0000", 2020-01-16T00:13:20.851 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00098608599999999995 2020-01-16T00:13:20.851 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.851 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.851 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:20.851 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497737+0000", 2020-01-16T00:13:20.851 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022081499999999999 2020-01-16T00:13:20.851 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.851 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.852 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.852 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497957+0000", 2020-01-16T00:13:20.852 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7071000000000003e-05 2020-01-16T00:13:20.852 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.852 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.852 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.852 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498014+0000", 2020-01-16T00:13:20.852 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1111e-05 2020-01-16T00:13:20.853 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.853 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.853 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:20.853 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498056+0000", 2020-01-16T00:13:20.853 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5974999999999999e-05 2020-01-16T00:13:20.853 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.853 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.854 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:20.854 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498102+0000", 2020-01-16T00:13:20.854 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016794989 2020-01-16T00:13:20.854 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.854 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:20.854 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.854 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.854 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.855 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:988 3.9 3:93dfe6f1:::smithi12313934-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:13:20.855 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:13:20.855 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 42.226424160999997, 2020-01-16T00:13:20.855 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017221692, 2020-01-16T00:13:20.855 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:20.855 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:20.855 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:20.855 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:20.856 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:20.856 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 988 2020-01-16T00:13:20.856 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.856 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:20.856 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.856 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:20.856 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:13:20.856 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:20.857 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.857 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.857 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:20.857 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:13:20.857 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:20.857 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.857 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.857 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:20.858 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481339+0000", 2020-01-16T00:13:20.858 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3089999999999997e-06 2020-01-16T00:13:20.858 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.858 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.858 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:20.858 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481343+0000", 2020-01-16T00:13:20.859 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2230000000000004e-06 2020-01-16T00:13:20.859 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.859 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.859 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:20.859 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481350+0000", 2020-01-16T00:13:20.859 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.319e-06 2020-01-16T00:13:20.859 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.859 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.860 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.860 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481356+0000", 2020-01-16T00:13:20.860 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000321544 2020-01-16T00:13:20.860 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.860 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.860 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.860 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481678+0000", 2020-01-16T00:13:20.860 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6375999999999999e-05 2020-01-16T00:13:20.861 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.861 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.861 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:20.861 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481714+0000", 2020-01-16T00:13:20.861 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015557666 2020-01-16T00:13:20.861 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.861 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.861 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.862 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497272+0000", 2020-01-16T00:13:20.862 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012791300000000001 2020-01-16T00:13:20.862 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.862 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.862 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:20.862 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497399+0000", 2020-01-16T00:13:20.862 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00076944600000000002 2020-01-16T00:13:20.862 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.863 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.863 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.863 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498169+0000", 2020-01-16T00:13:20.863 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018306799999999999 2020-01-16T00:13:20.863 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.863 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.863 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:20.864 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498352+0000", 2020-01-16T00:13:20.864 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020935999999999999 2020-01-16T00:13:20.864 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.864 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.864 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:20.864 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498561+0000", 2020-01-16T00:13:20.864 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017221692 2020-01-16T00:13:20.864 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.865 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:20.865 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.865 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.865 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.865 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1208 3.3 3:c478bfe8:::smithi12313934-10:head [write 749612~474179,stat] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:13:20.865 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.350952+0000", 2020-01-16T00:13:20.865 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 38.356811323999999, 2020-01-16T00:13:20.865 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014967729000000001, 2020-01-16T00:13:20.866 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:20.866 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:20.866 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:20.866 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:20.866 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:20.866 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1208 2020-01-16T00:13:20.866 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.866 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:20.867 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.867 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:20.867 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.350952+0000", 2020-01-16T00:13:20.867 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:20.867 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.867 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.867 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:20.868 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.350952+0000", 2020-01-16T00:13:20.868 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:13:20.868 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.868 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.868 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:20.868 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.350949+0000", 2020-01-16T00:13:20.869 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042894 2020-01-16T00:13:20.869 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.869 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.869 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:20.869 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.351377+0000", 2020-01-16T00:13:20.869 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3921e-05 2020-01-16T00:13:20.869 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.869 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.870 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:20.870 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.351401+0000", 2020-01-16T00:13:20.870 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1087000000000001e-05 2020-01-16T00:13:20.870 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.870 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.870 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.870 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.351412+0000", 2020-01-16T00:13:20.871 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8127000000000001e-05 2020-01-16T00:13:20.871 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.871 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.871 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.871 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.351471+0000", 2020-01-16T00:13:20.871 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000230268 2020-01-16T00:13:20.872 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.872 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.872 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:20.872 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.351701+0000", 2020-01-16T00:13:20.872 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0051228369999999999 2020-01-16T00:13:20.872 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.872 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.872 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.873 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.356824+0000", 2020-01-16T00:13:20.873 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015984800000000001 2020-01-16T00:13:20.873 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.873 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.873 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:20.873 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.356984+0000", 2020-01-16T00:13:20.873 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0038163770000000001 2020-01-16T00:13:20.873 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.874 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.874 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.874 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.360800+0000", 2020-01-16T00:13:20.874 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.9878999999999999e-05 2020-01-16T00:13:20.874 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.874 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.874 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:20.874 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.360880+0000", 2020-01-16T00:13:20.875 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000145041 2020-01-16T00:13:20.875 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.875 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.875 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2020-01-16T00:13:20.875 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.361025+0000", 2020-01-16T00:13:20.875 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0020743039999999999 2020-01-16T00:13:20.875 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.876 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.876 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:20.876 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.363099+0000", 2020-01-16T00:13:20.876 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00072876500000000003 2020-01-16T00:13:20.876 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.876 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.876 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.876 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.363828+0000", 2020-01-16T00:13:20.877 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00189153 2020-01-16T00:13:20.877 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.877 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.877 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.877 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.365719+0000", 2020-01-16T00:13:20.877 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3948000000000005e-05 2020-01-16T00:13:20.877 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.877 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.878 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:20.878 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.365783+0000", 2020-01-16T00:13:20.878 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000136833 2020-01-16T00:13:20.878 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.878 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.878 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:20.878 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.365920+0000", 2020-01-16T00:13:20.878 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014967729000000001 2020-01-16T00:13:20.879 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.879 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:20.879 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.879 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.879 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.879 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1237 3.3 3:c204aa4f:::smithi12313934-45:head [write 400033~699926] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:13:20.879 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.584946+0000", 2020-01-16T00:13:20.879 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 38.122817441999999, 2020-01-16T00:13:20.880 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014287855, 2020-01-16T00:13:20.880 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:20.880 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:20.880 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:20.880 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:20.880 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:20.880 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1237 2020-01-16T00:13:20.881 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.881 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:20.881 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.881 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:20.881 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.584946+0000", 2020-01-16T00:13:20.881 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:20.881 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.881 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.882 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:20.882 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.584946+0000", 2020-01-16T00:13:20.882 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999752 2020-01-16T00:13:20.882 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.882 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.882 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:20.882 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.584921+0000", 2020-01-16T00:13:20.882 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00077243400000000001 2020-01-16T00:13:20.883 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.883 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.883 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:20.883 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.585694+0000", 2020-01-16T00:13:20.883 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3083000000000001e-05 2020-01-16T00:13:20.883 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.883 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.883 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:20.884 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.585737+0000", 2020-01-16T00:13:20.884 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1854000000000001e-05 2020-01-16T00:13:20.884 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.884 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.884 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.884 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.585759+0000", 2020-01-16T00:13:20.884 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016127000000000001 2020-01-16T00:13:20.885 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.885 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.885 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.885 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.585920+0000", 2020-01-16T00:13:20.885 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024934700000000002 2020-01-16T00:13:20.885 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.885 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.885 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:20.886 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.586169+0000", 2020-01-16T00:13:20.886 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00036508999999999999 2020-01-16T00:13:20.886 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.886 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.886 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2020-01-16T00:13:20.886 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.586534+0000", 2020-01-16T00:13:20.886 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012168910999999999 2020-01-16T00:13:20.886 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.887 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.887 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.887 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598703+0000", 2020-01-16T00:13:20.887 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5302000000000004e-05 2020-01-16T00:13:20.887 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.887 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.887 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.888 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598789+0000", 2020-01-16T00:13:20.888 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00027301099999999998 2020-01-16T00:13:20.888 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.888 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.888 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:20.888 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599062+0000", 2020-01-16T00:13:20.889 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.4237e-05 2020-01-16T00:13:20.889 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.889 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.889 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:20.889 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599146+0000", 2020-01-16T00:13:20.889 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.8319000000000006e-05 2020-01-16T00:13:20.889 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.889 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.890 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:20.890 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599234+0000", 2020-01-16T00:13:20.890 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014287855 2020-01-16T00:13:20.890 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.890 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:20.890 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.890 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.890 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.891 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1239 3.3 3:c204aa4f:::smithi12313934-45:head [setxattr _header (56) in=7b,truncate 2000033] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:13:20.891 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:13:20.891 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 38.117350342000002, 2020-01-16T00:13:20.891 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019161661999999999, 2020-01-16T00:13:20.891 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:20.891 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:20.891 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:20.891 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:20.892 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:20.892 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1239 2020-01-16T00:13:20.892 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.892 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:20.892 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.892 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:20.892 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:13:20.892 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:20.893 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.893 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.893 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:20.893 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:13:20.893 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:20.893 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.893 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.894 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:20.894 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590411+0000", 2020-01-16T00:13:20.894 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3879999999999999e-05 2020-01-16T00:13:20.894 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.894 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.894 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:20.894 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590425+0000", 2020-01-16T00:13:20.894 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6542999999999998e-05 2020-01-16T00:13:20.895 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.895 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.895 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:20.895 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590471+0000", 2020-01-16T00:13:20.895 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8907999999999999e-05 2020-01-16T00:13:20.895 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.895 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.895 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.896 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590490+0000", 2020-01-16T00:13:20.896 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0045691960000000002 2020-01-16T00:13:20.896 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.896 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.896 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.896 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.595059+0000", 2020-01-16T00:13:20.896 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0035638940000000002 2020-01-16T00:13:20.896 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.897 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.897 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:20.897 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598623+0000", 2020-01-16T00:13:20.897 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0077558310000000004 2020-01-16T00:13:20.897 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.897 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.897 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.897 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606379+0000", 2020-01-16T00:13:20.898 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012171999999999999 2020-01-16T00:13:20.898 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.898 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.898 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:20.898 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606501+0000", 2020-01-16T00:13:20.898 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013144200000000001 2020-01-16T00:13:20.898 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.899 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.899 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2020-01-16T00:13:20.899 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606632+0000", 2020-01-16T00:13:20.899 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001288565 2020-01-16T00:13:20.899 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.899 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.899 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:20.899 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607921+0000", 2020-01-16T00:13:20.900 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033399299999999999 2020-01-16T00:13:20.900 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.900 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.900 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.900 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.608255+0000", 2020-01-16T00:13:20.900 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012280870000000001 2020-01-16T00:13:20.900 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.901 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.901 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.901 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609483+0000", 2020-01-16T00:13:20.901 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2286000000000001e-05 2020-01-16T00:13:20.901 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.901 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.901 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:20.902 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609525+0000", 2020-01-16T00:13:20.902 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0121999999999999e-05 2020-01-16T00:13:20.902 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.902 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.902 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:20.902 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609575+0000", 2020-01-16T00:13:20.902 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019161661999999999 2020-01-16T00:13:20.903 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.903 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:20.903 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.903 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.903 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.903 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1240 3.3 3:c204aa4f:::smithi12313934-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-01-16T00:13:20.903 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:13:20.903 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 38.117250153000001, 2020-01-16T00:13:20.904 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019268828000000002, 2020-01-16T00:13:20.904 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:20.904 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:20.904 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:20.904 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:20.904 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:20.904 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1240 2020-01-16T00:13:20.904 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.905 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:20.905 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.905 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:20.905 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:13:20.905 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:20.905 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.905 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.905 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:20.906 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:13:20.906 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:20.906 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.906 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.906 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:20.906 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590512+0000", 2020-01-16T00:13:20.906 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0525e-05 2020-01-16T00:13:20.906 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.907 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.907 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:20.907 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590522+0000", 2020-01-16T00:13:20.907 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6418000000000001e-05 2020-01-16T00:13:20.907 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.907 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.907 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:20.907 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590539+0000", 2020-01-16T00:13:20.908 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0679999999999999e-05 2020-01-16T00:13:20.908 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.908 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.908 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.908 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590549+0000", 2020-01-16T00:13:20.908 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0083626789999999996 2020-01-16T00:13:20.908 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.909 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.909 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.909 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598912+0000", 2020-01-16T00:13:20.909 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013344 2020-01-16T00:13:20.909 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.909 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.909 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:20.909 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599045+0000", 2020-01-16T00:13:20.910 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0084002269999999997 2020-01-16T00:13:20.910 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.910 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.910 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.910 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607446+0000", 2020-01-16T00:13:20.910 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000109873 2020-01-16T00:13:20.910 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.910 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.911 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:20.911 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607555+0000", 2020-01-16T00:13:20.911 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0020573309999999999 2020-01-16T00:13:20.911 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.911 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.911 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.911 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609613+0000", 2020-01-16T00:13:20.911 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6351e-05 2020-01-16T00:13:20.912 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.912 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.912 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:20.912 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609699+0000", 2020-01-16T00:13:20.912 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3374999999999994e-05 2020-01-16T00:13:20.912 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.912 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.912 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:20.913 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609782+0000", 2020-01-16T00:13:20.913 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019268828000000002 2020-01-16T00:13:20.913 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.913 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:20.913 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.913 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.913 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.914 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1436 3.3 3:c204aa4f:::smithi12313934-45:head [setxattr _header (56) in=7b,truncate 2386594] snapc 0=[] ondisk+write+known_if_redirected e28)", 2020-01-16T00:13:20.914 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:47.900544+0000", 2020-01-16T00:13:20.914 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 32.807219472, 2020-01-16T00:13:20.914 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016402521, 2020-01-16T00:13:20.914 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:20.914 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:20.914 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:20.914 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:20.915 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:20.915 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1436 2020-01-16T00:13:20.915 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.915 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:20.915 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.915 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:20.915 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900544+0000", 2020-01-16T00:13:20.916 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:20.916 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.916 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.916 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:20.916 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900544+0000", 2020-01-16T00:13:20.916 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:20.916 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.916 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.917 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:20.917 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900542+0000", 2020-01-16T00:13:20.917 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3862999999999999e-05 2020-01-16T00:13:20.917 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.917 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.917 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:20.917 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900556+0000", 2020-01-16T00:13:20.917 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5204e-05 2020-01-16T00:13:20.918 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.918 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.918 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:20.918 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900571+0000", 2020-01-16T00:13:20.918 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.092e-05 2020-01-16T00:13:20.918 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.918 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.918 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.919 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900582+0000", 2020-01-16T00:13:20.919 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002786026 2020-01-16T00:13:20.919 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.919 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.919 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.919 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.903368+0000", 2020-01-16T00:13:20.919 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0027123659999999999 2020-01-16T00:13:20.919 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.920 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.920 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:20.920 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.906081+0000", 2020-01-16T00:13:20.920 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0082707790000000007 2020-01-16T00:13:20.920 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.920 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.920 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.920 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.914352+0000", 2020-01-16T00:13:20.921 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020580300000000001 2020-01-16T00:13:20.921 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.921 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.921 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:20.921 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.914557+0000", 2020-01-16T00:13:20.921 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017729900000000001 2020-01-16T00:13:20.921 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.921 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.922 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2020-01-16T00:13:20.922 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.914735+0000", 2020-01-16T00:13:20.922 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00074632099999999999 2020-01-16T00:13:20.922 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.922 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.922 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:20.922 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.915481+0000", 2020-01-16T00:13:20.922 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012279599999999999 2020-01-16T00:13:20.923 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.923 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.923 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.923 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.915604+0000", 2020-01-16T00:13:20.923 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012619339999999999 2020-01-16T00:13:20.923 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.923 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.923 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:20.924 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.916866+0000", 2020-01-16T00:13:20.924 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1162000000000003e-05 2020-01-16T00:13:20.924 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.924 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.924 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:20.924 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.916907+0000", 2020-01-16T00:13:20.924 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9987999999999997e-05 2020-01-16T00:13:20.924 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.925 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.925 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:20.925 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.916947+0000", 2020-01-16T00:13:20.925 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016402521 2020-01-16T00:13:20.925 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.925 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:20.925 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.926 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.926 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.926 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1437 3.3 3:c204aa4f:::smithi12313934-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e28)", 2020-01-16T00:13:20.926 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:47.900594+0000", 2020-01-16T00:13:20.926 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 32.807169608999999, 2020-01-16T00:13:20.926 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016745237999999999, 2020-01-16T00:13:20.926 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:20.926 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:20.927 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:20.927 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:20.927 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:20.927 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1437 2020-01-16T00:13:20.927 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.927 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:20.927 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.927 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:20.928 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900594+0000", 2020-01-16T00:13:20.928 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:20.928 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.928 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.928 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:20.928 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900594+0000", 2020-01-16T00:13:20.928 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:20.928 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.929 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.929 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:20.929 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900593+0000", 2020-01-16T00:13:20.929 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5059999999999995e-06 2020-01-16T00:13:20.929 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.929 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.929 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:20.929 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900601+0000", 2020-01-16T00:13:20.930 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5077e-05 2020-01-16T00:13:20.930 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.930 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.930 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:20.930 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900616+0000", 2020-01-16T00:13:20.930 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0183000000000001e-05 2020-01-16T00:13:20.930 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.930 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.931 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:20.931 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900627+0000", 2020-01-16T00:13:20.931 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0063667660000000003 2020-01-16T00:13:20.931 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.931 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.931 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.931 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.906993+0000", 2020-01-16T00:13:20.931 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8988e-05 2020-01-16T00:13:20.932 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.932 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.932 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:20.932 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.907032+0000", 2020-01-16T00:13:20.932 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0080128850000000008 2020-01-16T00:13:20.932 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.932 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.932 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.933 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.915045+0000", 2020-01-16T00:13:20.933 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013200899999999999 2020-01-16T00:13:20.933 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.933 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.933 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:20.933 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.915177+0000", 2020-01-16T00:13:20.933 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0018373090000000001 2020-01-16T00:13:20.934 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.934 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.934 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:20.934 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.917015+0000", 2020-01-16T00:13:20.934 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016701699999999999 2020-01-16T00:13:20.934 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.934 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.934 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:20.935 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.917182+0000", 2020-01-16T00:13:20.935 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015782400000000001 2020-01-16T00:13:20.935 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:20.935 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:20.935 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:20.935 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.917339+0000", 2020-01-16T00:13:20.935 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016745237999999999 2020-01-16T00:13:20.935 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.936 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:20.936 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.936 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:20.936 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:20.936 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:20.937 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2020-01-16T00:13:20.938 INFO:tasks.rados.rados.0.smithi123.stdout:1182: expect (ObjNum 294 snap 0 seq_num 294) 2020-01-16T00:13:20.938 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:20.934+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:20.939 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-01-16T00:13:20.950 INFO:tasks.rados.rados.0.smithi123.stdout:1183: finishing write tid 1 to smithi12313934-40 2020-01-16T00:13:20.951 INFO:tasks.rados.rados.0.smithi123.stdout:1183: finishing write tid 2 to smithi12313934-40 2020-01-16T00:13:20.951 INFO:tasks.rados.rados.0.smithi123.stdout:1184: expect (ObjNum 333 snap 0 seq_num 333) 2020-01-16T00:13:21.039 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:21.034+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:21.098 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:21.098 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:13:21.098 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:13:21.098 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:21.108 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-01-16T00:13:21.140 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:21.134+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:21.148 INFO:tasks.rados.rados.0.smithi123.stdout:1183: finishing write tid 3 to smithi12313934-40 2020-01-16T00:13:21.148 INFO:tasks.rados.rados.0.smithi123.stdout:1183: finishing write tid 4 to smithi12313934-40 2020-01-16T00:13:21.148 INFO:tasks.rados.rados.0.smithi123.stdout:1175: done (15 left) 2020-01-16T00:13:21.148 INFO:tasks.rados.rados.0.smithi123.stdout:1176: done (14 left) 2020-01-16T00:13:21.148 INFO:tasks.rados.rados.0.smithi123.stdout:1177: done (13 left) 2020-01-16T00:13:21.149 INFO:tasks.rados.rados.0.smithi123.stdout:1179: done (12 left) 2020-01-16T00:13:21.149 INFO:tasks.rados.rados.0.smithi123.stdout:1180: done (11 left) 2020-01-16T00:13:21.149 INFO:tasks.rados.rados.0.smithi123.stdout:1182: done (10 left) 2020-01-16T00:13:21.149 INFO:tasks.rados.rados.0.smithi123.stdout:1184: done (9 left) 2020-01-16T00:13:21.149 INFO:tasks.rados.rados.0.smithi123.stdout:1193: write (excl) oid 7 current snap is 0 2020-01-16T00:13:21.149 INFO:tasks.rados.rados.0.smithi123.stdout:1193: seq_num 352 ranges {716064=639886,1815191=500449,2817011=526925} 2020-01-16T00:13:21.156 INFO:tasks.rados.rados.0.smithi123.stdout:1193: writing smithi12313934-7 from 716064 to 1355950 tid 1 2020-01-16T00:13:21.163 INFO:tasks.rados.rados.0.smithi123.stdout:1193: writing smithi12313934-7 from 1815191 to 2315640 tid 2 2020-01-16T00:13:21.169 INFO:tasks.rados.rados.0.smithi123.stdout:1193: writing smithi12313934-7 from 2817011 to 3343936 tid 3 2020-01-16T00:13:21.170 INFO:tasks.rados.rados.0.smithi123.stdout:1194: write (excl) oid 4 current snap is 0 2020-01-16T00:13:21.170 INFO:tasks.rados.rados.0.smithi123.stdout:1194: seq_num 353 ranges {732871=715135,1873673=644107,3244434=688437} 2020-01-16T00:13:21.181 INFO:tasks.rados.rados.0.smithi123.stdout:1194: writing smithi12313934-4 from 732871 to 1448006 tid 1 2020-01-16T00:13:21.190 INFO:tasks.rados.rados.0.smithi123.stdout:1194: writing smithi12313934-4 from 1873673 to 2517780 tid 2 2020-01-16T00:13:21.198 INFO:tasks.rados.rados.0.smithi123.stdout:1194: writing smithi12313934-4 from 3244434 to 3932871 tid 3 2020-01-16T00:13:21.198 INFO:tasks.rados.rados.0.smithi123.stdout:1195: write oid 31 current snap is 0 2020-01-16T00:13:21.198 INFO:tasks.rados.rados.0.smithi123.stdout:1195: seq_num 354 ranges {749678=790384,2332155=504118,3271857=549469} 2020-01-16T00:13:21.206 INFO:tasks.rados.rados.0.smithi123.stdout:1195: writing smithi12313934-31 from 749678 to 1540062 tid 1 2020-01-16T00:13:21.214 INFO:tasks.rados.rados.0.smithi123.stdout:1195: writing smithi12313934-31 from 2332155 to 2836273 tid 2 2020-01-16T00:13:21.220 INFO:tasks.rados.rados.0.smithi123.stdout:1195: writing smithi12313934-31 from 3271857 to 3821326 tid 3 2020-01-16T00:13:21.220 INFO:tasks.rados.rados.0.smithi123.stdout:1196: delete oid 9 current snap is 0 2020-01-16T00:13:21.220 INFO:tasks.rados.rados.0.smithi123.stdout:1183: finishing write tid 6 to smithi12313934-40 2020-01-16T00:13:21.220 INFO:tasks.rados.rados.0.smithi123.stdout:1183: finishing write tid 7 to smithi12313934-40 2020-01-16T00:13:21.220 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 40 v 174 (ObjNum 347 snap 0 seq_num 347) dirty exists 2020-01-16T00:13:21.220 INFO:tasks.rados.rados.0.smithi123.stdout:1183: left oid 40 (ObjNum 347 snap 0 seq_num 347) 2020-01-16T00:13:21.220 INFO:tasks.rados.rados.0.smithi123.stdout:1185: finishing write tid 1 to smithi12313934-17 2020-01-16T00:13:21.221 INFO:tasks.rados.rados.0.smithi123.stdout:1185: finishing write tid 2 to smithi12313934-17 2020-01-16T00:13:21.221 INFO:tasks.rados.rados.0.smithi123.stdout:1185: finishing write tid 3 to smithi12313934-17 2020-01-16T00:13:21.221 INFO:tasks.rados.rados.0.smithi123.stdout:1185: finishing write tid 5 to smithi12313934-17 2020-01-16T00:13:21.221 INFO:tasks.rados.rados.0.smithi123.stdout:1185: finishing write tid 6 to smithi12313934-17 2020-01-16T00:13:21.221 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 17 v 135 (ObjNum 348 snap 0 seq_num 348) dirty exists 2020-01-16T00:13:21.221 INFO:tasks.rados.rados.0.smithi123.stdout:1185: left oid 17 (ObjNum 348 snap 0 seq_num 348) 2020-01-16T00:13:21.221 INFO:tasks.rados.rados.0.smithi123.stdout:1186: finishing write tid 1 to smithi12313934-36 2020-01-16T00:13:21.221 INFO:tasks.rados.rados.0.smithi123.stdout:1186: finishing write tid 2 to smithi12313934-36 2020-01-16T00:13:21.222 INFO:tasks.rados.rados.0.smithi123.stdout:1186: finishing write tid 3 to smithi12313934-36 2020-01-16T00:13:21.222 INFO:tasks.rados.rados.0.smithi123.stdout:1186: finishing write tid 5 to smithi12313934-36 2020-01-16T00:13:21.222 INFO:tasks.rados.rados.0.smithi123.stdout:1186: finishing write tid 6 to smithi12313934-36 2020-01-16T00:13:21.222 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 36 v 75 (ObjNum 349 snap 0 seq_num 349) dirty exists 2020-01-16T00:13:21.222 INFO:tasks.rados.rados.0.smithi123.stdout:1186: left oid 36 (ObjNum 349 snap 0 seq_num 349) 2020-01-16T00:13:21.222 INFO:tasks.rados.rados.0.smithi123.stdout:1187: finishing write tid 1 to smithi12313934-48 2020-01-16T00:13:21.222 INFO:tasks.rados.rados.0.smithi123.stdout:1187: finishing write tid 2 to smithi12313934-48 2020-01-16T00:13:21.223 INFO:tasks.rados.rados.0.smithi123.stdout:1188: expect (ObjNum 220 snap 0 seq_num 220) 2020-01-16T00:13:21.223 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:21.223 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:13:21.223 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:13:21.223 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:13:21.224 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:21.231 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-01-16T00:13:21.241 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:21.238+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:21.338 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:21.338 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:13:21.338 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:13:21.339 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:13:21.339 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.339 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.5.13:36 2.0 2:20e40cec:::smithi12313934-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 113152~1699840] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:13:21.339 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:13:21.339 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 43.501018672999997, 2020-01-16T00:13:21.339 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011736636, 2020-01-16T00:13:21.339 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:21.340 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:21.340 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:21.340 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.340 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:21.340 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:13:21.340 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:21.340 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.340 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.341 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:21.341 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:13:21.341 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999952 2020-01-16T00:13:21.341 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.341 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.341 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:21.341 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831934+0000", 2020-01-16T00:13:21.342 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001034325 2020-01-16T00:13:21.342 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.342 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.342 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:21.342 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.832968+0000", 2020-01-16T00:13:21.342 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.506e-05 2020-01-16T00:13:21.342 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.342 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.343 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:21.343 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833004+0000", 2020-01-16T00:13:21.343 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3406e-05 2020-01-16T00:13:21.343 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.343 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.343 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:21.343 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833017+0000", 2020-01-16T00:13:21.344 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015477000000000001 2020-01-16T00:13:21.344 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.344 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.344 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:21.344 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833172+0000", 2020-01-16T00:13:21.344 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033750700000000002 2020-01-16T00:13:21.344 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.344 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.345 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:21.345 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833509+0000", 2020-01-16T00:13:21.345 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00055861400000000003 2020-01-16T00:13:21.345 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.345 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.345 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:13:21.345 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.834068+0000", 2020-01-16T00:13:21.346 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.007474545 2020-01-16T00:13:21.346 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.346 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.346 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:21.346 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.841542+0000", 2020-01-16T00:13:21.346 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00056319499999999995 2020-01-16T00:13:21.346 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.346 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.347 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:21.347 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.842106+0000", 2020-01-16T00:13:21.347 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001371017 2020-01-16T00:13:21.347 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.347 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.347 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:21.347 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843477+0000", 2020-01-16T00:13:21.347 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000123938 2020-01-16T00:13:21.348 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.348 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.348 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:21.348 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843601+0000", 2020-01-16T00:13:21.348 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5012000000000006e-05 2020-01-16T00:13:21.348 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.348 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.348 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:21.349 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843676+0000", 2020-01-16T00:13:21.349 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011736636 2020-01-16T00:13:21.349 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.349 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:21.349 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.349 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.349 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.350 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.5.13:40 2.0 2:20e40cec:::smithi12313934-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 113152~1699840] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:13:21.350 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.847376+0000", 2020-01-16T00:13:21.350 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 43.485581945, 2020-01-16T00:13:21.350 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010871093, 2020-01-16T00:13:21.350 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:21.350 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:21.350 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:21.350 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.351 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:21.351 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.847376+0000", 2020-01-16T00:13:21.351 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:21.351 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.351 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.351 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:21.351 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.847376+0000", 2020-01-16T00:13:21.351 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:21.352 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.352 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.352 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:21.352 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.847373+0000", 2020-01-16T00:13:21.352 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00077394699999999998 2020-01-16T00:13:21.352 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.352 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.352 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:21.353 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.848147+0000", 2020-01-16T00:13:21.353 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.173e-05 2020-01-16T00:13:21.353 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.353 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.353 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:21.353 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.848168+0000", 2020-01-16T00:13:21.353 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2738999999999999e-05 2020-01-16T00:13:21.353 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.354 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.354 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:21.354 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.848181+0000", 2020-01-16T00:13:21.354 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4627999999999999e-05 2020-01-16T00:13:21.354 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.354 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.354 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:21.355 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.848236+0000", 2020-01-16T00:13:21.355 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000141311 2020-01-16T00:13:21.355 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.355 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.355 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:21.355 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.848377+0000", 2020-01-16T00:13:21.355 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015462700000000001 2020-01-16T00:13:21.356 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.356 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.356 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:13:21.356 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.848532+0000", 2020-01-16T00:13:21.356 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0039933540000000002 2020-01-16T00:13:21.356 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.356 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.356 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:21.357 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.852525+0000", 2020-01-16T00:13:21.357 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0051016129999999996 2020-01-16T00:13:21.357 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.357 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.357 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:21.357 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.857627+0000", 2020-01-16T00:13:21.357 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00044689099999999999 2020-01-16T00:13:21.358 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.358 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.358 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:21.358 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.858073+0000", 2020-01-16T00:13:21.358 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8947e-05 2020-01-16T00:13:21.358 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.358 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.358 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:21.359 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.858172+0000", 2020-01-16T00:13:21.359 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4369999999999994e-05 2020-01-16T00:13:21.359 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.359 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.359 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:21.359 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.858247+0000", 2020-01-16T00:13:21.359 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010871093 2020-01-16T00:13:21.359 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.360 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:21.360 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.360 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.360 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.360 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.5.13:42 2.0 2:313458df:::smithi12313934-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 263680~1668096] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:13:21.360 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.939100+0000", 2020-01-16T00:13:21.360 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 43.393858037000001, 2020-01-16T00:13:21.361 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010934255, 2020-01-16T00:13:21.361 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:21.361 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:21.361 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:21.361 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.361 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:21.361 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939100+0000", 2020-01-16T00:13:21.362 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:21.362 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.362 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.362 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:21.362 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939100+0000", 2020-01-16T00:13:21.362 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:21.363 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.363 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.363 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:21.363 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939097+0000", 2020-01-16T00:13:21.363 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00049374800000000002 2020-01-16T00:13:21.363 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.363 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.364 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:21.364 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939591+0000", 2020-01-16T00:13:21.364 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0840999999999999e-05 2020-01-16T00:13:21.364 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.364 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.364 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:21.364 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939612+0000", 2020-01-16T00:13:21.364 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4749999999999998e-06 2020-01-16T00:13:21.365 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.365 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.365 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:21.365 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939618+0000", 2020-01-16T00:13:21.365 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6371e-05 2020-01-16T00:13:21.365 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.365 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.366 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:21.366 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939674+0000", 2020-01-16T00:13:21.366 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2e-05 2020-01-16T00:13:21.366 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.366 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.366 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:21.366 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939766+0000", 2020-01-16T00:13:21.367 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020133999999999999 2020-01-16T00:13:21.367 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.367 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.367 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:13:21.367 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.939968+0000", 2020-01-16T00:13:21.367 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0027942560000000002 2020-01-16T00:13:21.367 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.367 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.368 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:21.368 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.942762+0000", 2020-01-16T00:13:21.368 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.004998466 2020-01-16T00:13:21.368 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.368 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.368 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:21.368 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.947760+0000", 2020-01-16T00:13:21.369 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0021783330000000002 2020-01-16T00:13:21.369 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.369 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.369 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:21.369 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.949939+0000", 2020-01-16T00:13:21.369 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2391999999999997e-05 2020-01-16T00:13:21.369 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.369 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.370 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:21.370 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.949981+0000", 2020-01-16T00:13:21.370 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2629e-05 2020-01-16T00:13:21.370 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.370 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.370 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:21.371 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.950034+0000", 2020-01-16T00:13:21.371 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010934255 2020-01-16T00:13:21.371 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.371 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:21.371 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.371 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.371 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.371 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1186 3.d 3:b1521a3e:::smithi12313934-25:head [write 682384~456830,stat] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:13:21.372 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.258156+0000", 2020-01-16T00:13:21.372 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 39.074802073000001, 2020-01-16T00:13:21.372 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0092969679999999992, 2020-01-16T00:13:21.372 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:21.372 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:21.372 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:21.372 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:21.372 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:21.373 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1186 2020-01-16T00:13:21.373 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.373 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:21.373 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.373 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:21.373 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.258156+0000", 2020-01-16T00:13:21.374 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:21.374 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.374 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.374 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:21.375 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.258156+0000", 2020-01-16T00:13:21.375 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999943 2020-01-16T00:13:21.375 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.375 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.376 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:21.376 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.258150+0000", 2020-01-16T00:13:21.376 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001866517 2020-01-16T00:13:21.376 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.377 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.377 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:21.377 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.260016+0000", 2020-01-16T00:13:21.377 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4024999999999999e-05 2020-01-16T00:13:21.377 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.377 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.377 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:21.377 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.260070+0000", 2020-01-16T00:13:21.378 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4295e-05 2020-01-16T00:13:21.378 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.378 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.378 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:21.378 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.260085+0000", 2020-01-16T00:13:21.378 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013625599999999999 2020-01-16T00:13:21.378 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.379 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.379 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:21.379 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.260221+0000", 2020-01-16T00:13:21.379 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000230015 2020-01-16T00:13:21.379 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.379 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.379 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:21.380 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.260451+0000", 2020-01-16T00:13:21.380 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00031965999999999999 2020-01-16T00:13:21.380 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.380 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.380 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:21.380 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.260770+0000", 2020-01-16T00:13:21.380 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0031788649999999999 2020-01-16T00:13:21.381 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.381 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.381 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:21.381 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.263949+0000", 2020-01-16T00:13:21.381 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0022620090000000002 2020-01-16T00:13:21.381 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.382 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.382 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:21.382 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.266211+0000", 2020-01-16T00:13:21.382 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00097717199999999989 2020-01-16T00:13:21.382 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.382 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.382 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:21.382 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.267188+0000", 2020-01-16T00:13:21.383 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000128594 2020-01-16T00:13:21.383 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.383 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.383 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:21.383 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.267317+0000", 2020-01-16T00:13:21.383 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013541200000000001 2020-01-16T00:13:21.383 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.384 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.384 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:21.384 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.267452+0000", 2020-01-16T00:13:21.384 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0092969679999999992 2020-01-16T00:13:21.384 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.384 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:21.384 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.384 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.385 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.385 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1187 3.d 3:b1521a3e:::smithi12313934-25:head [write 1721042~423699] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:13:21.385 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.263828+0000", 2020-01-16T00:13:21.385 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 39.069129601, 2020-01-16T00:13:21.385 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011305709000000001, 2020-01-16T00:13:21.385 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:21.385 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:21.385 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:21.386 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:21.386 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:21.386 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1187 2020-01-16T00:13:21.386 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.386 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:21.386 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.386 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:21.386 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.263828+0000", 2020-01-16T00:13:21.387 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:21.387 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.387 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.387 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:21.387 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.263828+0000", 2020-01-16T00:13:21.387 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:13:21.387 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.387 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.388 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:21.388 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.263824+0000", 2020-01-16T00:13:21.388 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00048485899999999997 2020-01-16T00:13:21.388 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.388 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.388 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:21.388 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264309+0000", 2020-01-16T00:13:21.388 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5622999999999999e-05 2020-01-16T00:13:21.389 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.389 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.389 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:21.389 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264325+0000", 2020-01-16T00:13:21.389 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2530000000000004e-06 2020-01-16T00:13:21.389 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.389 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.390 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:21.390 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264330+0000", 2020-01-16T00:13:21.390 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000102252 2020-01-16T00:13:21.390 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.390 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.390 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:21.390 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264432+0000", 2020-01-16T00:13:21.391 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022361000000000001 2020-01-16T00:13:21.391 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.391 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.391 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:21.391 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264656+0000", 2020-01-16T00:13:21.391 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0028688849999999998 2020-01-16T00:13:21.391 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.391 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.392 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:21.392 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.267525+0000", 2020-01-16T00:13:21.392 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00023640900000000001 2020-01-16T00:13:21.392 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.392 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.392 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:21.393 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.267761+0000", 2020-01-16T00:13:21.393 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00035667000000000001 2020-01-16T00:13:21.393 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.393 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.393 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:21.393 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.268118+0000", 2020-01-16T00:13:21.393 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023898040000000001 2020-01-16T00:13:21.393 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.394 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.394 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:21.394 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.270507+0000", 2020-01-16T00:13:21.394 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014919690000000001 2020-01-16T00:13:21.394 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.394 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.394 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:21.395 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.271999+0000", 2020-01-16T00:13:21.395 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0029764510000000002 2020-01-16T00:13:21.395 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.395 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.395 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:21.395 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.274976+0000", 2020-01-16T00:13:21.395 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5409999999999997e-05 2020-01-16T00:13:21.395 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.396 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.396 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:21.396 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.275061+0000", 2020-01-16T00:13:21.396 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2452000000000004e-05 2020-01-16T00:13:21.396 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.396 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.396 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:21.396 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.275134+0000", 2020-01-16T00:13:21.397 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011305709000000001 2020-01-16T00:13:21.397 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.397 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:21.397 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.397 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.397 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.397 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1259 3.d 3:b374ba21:::smithi12313934-20:head [read 0~3932805,checksum] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:13:21.398 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.833857+0000", 2020-01-16T00:13:21.398 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 38.499100857000002, 2020-01-16T00:13:21.398 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0094623380000000007, 2020-01-16T00:13:21.398 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:21.398 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:21.398 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:21.398 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:21.399 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:21.399 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1259 2020-01-16T00:13:21.399 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.399 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:21.399 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.399 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:21.399 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.833857+0000", 2020-01-16T00:13:21.399 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:21.400 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.400 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.400 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:21.400 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.833857+0000", 2020-01-16T00:13:21.400 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:21.400 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.400 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.400 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:21.401 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.833856+0000", 2020-01-16T00:13:21.401 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.9659999999999999e-06 2020-01-16T00:13:21.401 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.401 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.401 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:21.401 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.833863+0000", 2020-01-16T00:13:21.401 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4699999999999996e-06 2020-01-16T00:13:21.401 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.402 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.402 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:21.402 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.833870+0000", 2020-01-16T00:13:21.402 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0039999999999999e-06 2020-01-16T00:13:21.402 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.402 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.402 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:21.403 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.833875+0000", 2020-01-16T00:13:21.403 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0069094480000000003 2020-01-16T00:13:21.403 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.403 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.403 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:21.403 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.840785+0000", 2020-01-16T00:13:21.403 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020800699999999999 2020-01-16T00:13:21.403 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.403 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.404 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:21.404 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.840993+0000", 2020-01-16T00:13:21.404 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023263960000000001 2020-01-16T00:13:21.404 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.404 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.404 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:21.404 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.843319+0000", 2020-01-16T00:13:21.405 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0094623380000000007 2020-01-16T00:13:21.405 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.405 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:21.405 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.405 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.405 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.405 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1276 3.d 3:b1521a3e:::smithi12313934-25:head [read 0~3882384,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:13:21.406 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:13:21.406 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 38.258259103, 2020-01-16T00:13:21.406 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015839157999999999, 2020-01-16T00:13:21.406 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:21.406 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:21.406 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:21.407 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:21.407 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:21.407 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1276 2020-01-16T00:13:21.407 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.407 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:21.407 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.407 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:21.407 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:13:21.408 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:21.408 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.408 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.408 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:21.408 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:13:21.408 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:21.408 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.408 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.409 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:21.409 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:13:21.409 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5429999999999999e-06 2020-01-16T00:13:21.409 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.409 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.409 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:21.409 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074704+0000", 2020-01-16T00:13:21.409 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0839999999999992e-06 2020-01-16T00:13:21.410 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.410 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.410 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:21.410 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074712+0000", 2020-01-16T00:13:21.410 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.173e-06 2020-01-16T00:13:21.410 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.410 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.410 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:21.411 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074718+0000", 2020-01-16T00:13:21.411 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0083034700000000003 2020-01-16T00:13:21.411 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.411 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.411 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:21.411 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.083022+0000", 2020-01-16T00:13:21.411 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028456400000000001 2020-01-16T00:13:21.411 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.412 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.412 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:21.412 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.083306+0000", 2020-01-16T00:13:21.412 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0072312640000000003 2020-01-16T00:13:21.412 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.412 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.412 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:21.412 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090538+0000", 2020-01-16T00:13:21.413 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015839157999999999 2020-01-16T00:13:21.413 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.413 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:21.413 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.413 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.413 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.413 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1277 3.d 3:b1521a3e:::smithi12313934-25:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:13:21.413 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:13:21.414 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 38.258215513000003, 2020-01-16T00:13:21.414 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016646459999999998, 2020-01-16T00:13:21.414 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:21.414 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:21.414 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:21.414 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:21.414 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:21.415 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1277 2020-01-16T00:13:21.415 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.415 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:21.415 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.415 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:21.415 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:13:21.415 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:21.415 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.416 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.416 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:21.416 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:13:21.416 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:21.416 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.416 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.416 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:21.417 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074741+0000", 2020-01-16T00:13:21.417 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0776e-05 2020-01-16T00:13:21.417 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.417 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.417 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:21.417 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074752+0000", 2020-01-16T00:13:21.417 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7600000000000002e-06 2020-01-16T00:13:21.417 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.418 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.418 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:21.418 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074759+0000", 2020-01-16T00:13:21.418 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6840000000000001e-06 2020-01-16T00:13:21.418 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.418 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.418 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:21.418 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074765+0000", 2020-01-16T00:13:21.419 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015815216 2020-01-16T00:13:21.419 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.419 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.419 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:21.419 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090580+0000", 2020-01-16T00:13:21.419 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000182404 2020-01-16T00:13:21.419 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.419 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.420 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:21.420 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090763+0000", 2020-01-16T00:13:21.420 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00062579300000000003 2020-01-16T00:13:21.420 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.420 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.420 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:21.420 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091389+0000", 2020-01-16T00:13:21.420 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016646459999999998 2020-01-16T00:13:21.421 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.421 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:21.421 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.421 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.421 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.421 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1278 3.d 3:b1521a3e:::smithi12313934-25:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:13:21.421 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:13:21.422 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 38.2581858, 2020-01-16T00:13:21.422 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017994915, 2020-01-16T00:13:21.422 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:21.422 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:21.422 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:21.422 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:21.422 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:21.422 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1278 2020-01-16T00:13:21.423 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.423 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:21.423 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.423 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:21.423 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:13:21.423 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:21.423 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.423 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.424 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:21.424 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:13:21.424 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:21.424 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.424 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.424 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:21.424 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074771+0000", 2020-01-16T00:13:21.424 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4569999999999998e-06 2020-01-16T00:13:21.425 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.425 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.425 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:21.425 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074775+0000", 2020-01-16T00:13:21.425 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0330000000000003e-06 2020-01-16T00:13:21.425 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.425 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.425 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:21.426 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074782+0000", 2020-01-16T00:13:21.426 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9470000000000002e-06 2020-01-16T00:13:21.426 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.426 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.426 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:21.426 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074787+0000", 2020-01-16T00:13:21.426 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01666134 2020-01-16T00:13:21.426 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.427 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.427 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:21.427 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091449+0000", 2020-01-16T00:13:21.427 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000247731 2020-01-16T00:13:21.427 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.427 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.427 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:21.427 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091696+0000", 2020-01-16T00:13:21.428 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010703170000000001 2020-01-16T00:13:21.428 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.428 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.428 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:21.428 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.092767+0000", 2020-01-16T00:13:21.428 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017994915 2020-01-16T00:13:21.428 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.428 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:21.429 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.429 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.429 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.429 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1420 3.e 3:7cfd7007:::smithi12313934-21:head [read 0~3899191,checksum] snapc 0=[] ondisk+read+known_if_redirected e28)", 2020-01-16T00:13:21.429 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:47.689967+0000", 2020-01-16T00:13:21.429 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 33.642990515000001, 2020-01-16T00:13:21.429 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010893273, 2020-01-16T00:13:21.429 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:21.430 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:21.430 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:21.430 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:21.430 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:21.430 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1420 2020-01-16T00:13:21.430 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.430 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:21.431 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.431 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:21.431 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.689967+0000", 2020-01-16T00:13:21.431 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:21.431 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.431 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.431 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:21.432 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.689967+0000", 2020-01-16T00:13:21.432 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:21.432 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.432 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.432 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:21.432 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.689965+0000", 2020-01-16T00:13:21.432 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.696e-05 2020-01-16T00:13:21.432 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.433 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.433 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:21.433 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.689982+0000", 2020-01-16T00:13:21.433 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9613e-05 2020-01-16T00:13:21.433 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.433 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.433 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:21.434 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.690002+0000", 2020-01-16T00:13:21.434 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3003000000000001e-05 2020-01-16T00:13:21.434 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.434 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.434 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:21.434 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.690015+0000", 2020-01-16T00:13:21.434 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0073546410000000003 2020-01-16T00:13:21.434 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.435 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.435 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:21.435 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.697370+0000", 2020-01-16T00:13:21.435 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00031806899999999998 2020-01-16T00:13:21.435 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.435 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.435 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:21.435 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.697688+0000", 2020-01-16T00:13:21.436 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003172673 2020-01-16T00:13:21.436 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.436 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.436 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:21.436 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.700860+0000", 2020-01-16T00:13:21.436 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010893273 2020-01-16T00:13:21.436 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.436 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:21.436 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.437 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.437 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.437 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1870 3.d 3:bd9fc72f:::smithi12313934-5:head [write 2804576~435035] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:13:21.437 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:56.384430+0000", 2020-01-16T00:13:21.437 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 24.948527739999999, 2020-01-16T00:13:21.437 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0085577770000000008, 2020-01-16T00:13:21.437 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:21.438 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:21.438 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:21.438 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:21.438 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:21.438 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1870 2020-01-16T00:13:21.439 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.439 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:21.439 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.439 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:21.439 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384430+0000", 2020-01-16T00:13:21.439 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:21.439 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.439 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.440 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:21.440 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384430+0000", 2020-01-16T00:13:21.440 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999933 2020-01-16T00:13:21.440 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.440 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.440 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:21.440 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384423+0000", 2020-01-16T00:13:21.440 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042953000000000002 2020-01-16T00:13:21.441 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.441 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.441 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:21.441 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384853+0000", 2020-01-16T00:13:21.441 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2592000000000001e-05 2020-01-16T00:13:21.441 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.441 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.442 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:21.442 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384865+0000", 2020-01-16T00:13:21.442 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7559999999999999e-06 2020-01-16T00:13:21.442 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.442 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.442 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:21.442 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384870+0000", 2020-01-16T00:13:21.443 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8498000000000003e-05 2020-01-16T00:13:21.443 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.443 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.443 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:21.443 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384909+0000", 2020-01-16T00:13:21.443 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00075234 2020-01-16T00:13:21.443 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.443 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.444 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:21.444 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.385661+0000", 2020-01-16T00:13:21.444 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.004714963 2020-01-16T00:13:21.444 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.444 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.444 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:21.444 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.390376+0000", 2020-01-16T00:13:21.444 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000274324 2020-01-16T00:13:21.445 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.445 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.445 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:21.445 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.390650+0000", 2020-01-16T00:13:21.445 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024631000000000002 2020-01-16T00:13:21.445 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.445 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.445 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:21.446 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.390897+0000", 2020-01-16T00:13:21.446 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014908829999999999 2020-01-16T00:13:21.446 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.446 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.446 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:21.446 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.392387+0000", 2020-01-16T00:13:21.446 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00023496300000000001 2020-01-16T00:13:21.447 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.447 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.447 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:21.447 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.392622+0000", 2020-01-16T00:13:21.447 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00027025500000000001 2020-01-16T00:13:21.447 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.447 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.447 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:21.447 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.392893+0000", 2020-01-16T00:13:21.448 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0452000000000003e-05 2020-01-16T00:13:21.448 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.448 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.448 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:21.448 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.392953+0000", 2020-01-16T00:13:21.448 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4564999999999999e-05 2020-01-16T00:13:21.448 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.448 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.449 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:21.449 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.392988+0000", 2020-01-16T00:13:21.449 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0085577770000000008 2020-01-16T00:13:21.449 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.449 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:21.449 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.449 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.449 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.450 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1871 3.d 3:bd9fc72f:::smithi12313934-5:head [setxattr _header (55) in=7b,truncate 3327786] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:13:21.450 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:56.384879+0000", 2020-01-16T00:13:21.450 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 24.948078720000002, 2020-01-16T00:13:21.450 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010769713, 2020-01-16T00:13:21.450 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:21.450 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:21.450 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:21.451 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:21.451 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:21.451 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1871 2020-01-16T00:13:21.451 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.451 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:21.451 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.451 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:21.451 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384879+0000", 2020-01-16T00:13:21.452 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:21.452 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.452 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.452 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:21.452 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384879+0000", 2020-01-16T00:13:21.452 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:21.452 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.452 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.452 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:21.453 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384878+0000", 2020-01-16T00:13:21.453 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9370000000000003e-06 2020-01-16T00:13:21.453 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.453 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.453 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:21.453 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384883+0000", 2020-01-16T00:13:21.453 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6589999999999996e-06 2020-01-16T00:13:21.453 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.454 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.454 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:21.454 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384891+0000", 2020-01-16T00:13:21.454 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9330000000000001e-06 2020-01-16T00:13:21.454 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.454 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.454 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:21.454 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384894+0000", 2020-01-16T00:13:21.455 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00080669800000000001 2020-01-16T00:13:21.455 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.455 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.455 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:21.455 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.385701+0000", 2020-01-16T00:13:21.455 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3726000000000002e-05 2020-01-16T00:13:21.455 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.456 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.456 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:21.456 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.385735+0000", 2020-01-16T00:13:21.456 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0059567650000000002 2020-01-16T00:13:21.456 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.456 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.456 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:21.456 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.391692+0000", 2020-01-16T00:13:21.456 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013010300000000001 2020-01-16T00:13:21.457 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.457 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.457 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:21.457 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.391822+0000", 2020-01-16T00:13:21.457 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016664800000000001 2020-01-16T00:13:21.457 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.457 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.458 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:21.458 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.391988+0000", 2020-01-16T00:13:21.458 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00052670799999999997 2020-01-16T00:13:21.458 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.458 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.458 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:21.458 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.392515+0000", 2020-01-16T00:13:21.458 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00052796600000000005 2020-01-16T00:13:21.459 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.459 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.459 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:21.459 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.393043+0000", 2020-01-16T00:13:21.459 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002493501 2020-01-16T00:13:21.459 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.459 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.459 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:21.460 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.395537+0000", 2020-01-16T00:13:21.460 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9033999999999998e-05 2020-01-16T00:13:21.460 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.460 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.460 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:21.460 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.395596+0000", 2020-01-16T00:13:21.460 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3052e-05 2020-01-16T00:13:21.460 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.461 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.461 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:21.461 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.395649+0000", 2020-01-16T00:13:21.461 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010769713 2020-01-16T00:13:21.461 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.461 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:21.461 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.461 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.462 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.462 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1872 3.d 3:bd9fc72f:::smithi12313934-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e31)", 2020-01-16T00:13:21.462 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:56.384904+0000", 2020-01-16T00:13:21.462 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 24.948053121000001, 2020-01-16T00:13:21.462 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011110386999999999, 2020-01-16T00:13:21.462 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:21.463 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:21.463 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:21.463 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:21.463 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:21.463 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1872 2020-01-16T00:13:21.464 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.464 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:21.464 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.464 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:21.464 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384904+0000", 2020-01-16T00:13:21.464 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:21.464 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.464 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.465 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:21.465 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384904+0000", 2020-01-16T00:13:21.465 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:21.465 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.465 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.465 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:21.465 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384904+0000", 2020-01-16T00:13:21.465 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3879999999999998e-06 2020-01-16T00:13:21.466 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.466 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.466 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:21.466 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384906+0000", 2020-01-16T00:13:21.466 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7479999999999999e-06 2020-01-16T00:13:21.466 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.466 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.467 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:21.467 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384910+0000", 2020-01-16T00:13:21.467 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4609999999999998e-06 2020-01-16T00:13:21.467 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.467 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.467 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:21.467 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384913+0000", 2020-01-16T00:13:21.467 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00083872899999999997 2020-01-16T00:13:21.468 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.468 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.468 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:21.468 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.385751+0000", 2020-01-16T00:13:21.468 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3528999999999997e-05 2020-01-16T00:13:21.468 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.468 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.468 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:21.469 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.385785+0000", 2020-01-16T00:13:21.469 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0064799849999999997 2020-01-16T00:13:21.469 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.469 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.469 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:21.469 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.392265+0000", 2020-01-16T00:13:21.469 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010603399999999999 2020-01-16T00:13:21.470 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.470 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.470 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:21.470 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.392371+0000", 2020-01-16T00:13:21.470 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003330576 2020-01-16T00:13:21.470 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.471 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.471 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:21.471 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.395702+0000", 2020-01-16T00:13:21.471 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016566899999999999 2020-01-16T00:13:21.471 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.471 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.471 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:21.471 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.395867+0000", 2020-01-16T00:13:21.472 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014768999999999999 2020-01-16T00:13:21.472 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.472 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.472 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:21.472 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.396015+0000", 2020-01-16T00:13:21.472 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011110386999999999 2020-01-16T00:13:21.472 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.472 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:21.473 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.473 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.473 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.473 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.0.13:16 2.0 2:1afbc8ac:::smithi12313934-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 4096~737792] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e33)", 2020-01-16T00:13:21.473 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:01.646860+0000", 2020-01-16T00:13:21.473 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 19.686097806999999, 2020-01-16T00:13:21.473 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010798039000000001, 2020-01-16T00:13:21.473 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:21.474 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:21.474 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:21.474 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.474 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:21.474 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.646860+0000", 2020-01-16T00:13:21.474 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:21.474 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.475 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.475 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:21.475 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.646860+0000", 2020-01-16T00:13:21.475 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999957 2020-01-16T00:13:21.475 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.475 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.475 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:21.475 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.646856+0000", 2020-01-16T00:13:21.476 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001014547 2020-01-16T00:13:21.476 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.476 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.476 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:21.476 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.647870+0000", 2020-01-16T00:13:21.476 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.8031999999999999e-05 2020-01-16T00:13:21.476 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.476 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.477 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:21.477 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.647898+0000", 2020-01-16T00:13:21.477 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1080000000000004e-06 2020-01-16T00:13:21.477 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.477 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.477 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:21.477 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.647905+0000", 2020-01-16T00:13:21.477 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0057999999999997e-05 2020-01-16T00:13:21.478 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.478 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.478 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:21.478 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.647995+0000", 2020-01-16T00:13:21.478 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.364e-05 2020-01-16T00:13:21.478 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.478 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.479 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op must wait for map", 2020-01-16T00:13:21.479 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.648009+0000", 2020-01-16T00:13:21.479 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0028913789999999999 2020-01-16T00:13:21.479 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.479 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.479 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:21.479 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.650900+0000", 2020-01-16T00:13:21.479 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013687700000000001 2020-01-16T00:13:21.480 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.480 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.480 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:21.480 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.651037+0000", 2020-01-16T00:13:21.480 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018024999999999999 2020-01-16T00:13:21.480 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.480 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.480 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:13:21.481 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.651218+0000", 2020-01-16T00:13:21.481 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0062268159999999996 2020-01-16T00:13:21.481 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.481 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.481 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:21.482 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.657444+0000", 2020-01-16T00:13:21.482 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2280999999999995e-05 2020-01-16T00:13:21.482 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.482 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.482 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:21.482 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.657507+0000", 2020-01-16T00:13:21.482 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3654e-05 2020-01-16T00:13:21.482 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.483 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.483 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:21.483 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.657520+0000", 2020-01-16T00:13:21.483 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.9783999999999995e-05 2020-01-16T00:13:21.483 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.483 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.483 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:21.484 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.657590+0000", 2020-01-16T00:13:21.484 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7750999999999996e-05 2020-01-16T00:13:21.484 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.484 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.484 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:21.484 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.657658+0000", 2020-01-16T00:13:21.484 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010798039000000001 2020-01-16T00:13:21.484 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.485 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:21.485 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.485 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.485 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.485 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.0.13:17 2.0 2:1afbc8ac:::smithi12313934-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 916992~596480] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e33)", 2020-01-16T00:13:21.485 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:01.647924+0000", 2020-01-16T00:13:21.485 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 19.685034029000001, 2020-01-16T00:13:21.486 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01120495, 2020-01-16T00:13:21.486 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:21.486 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:21.486 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:21.486 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.486 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:21.486 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.647924+0000", 2020-01-16T00:13:21.486 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:21.487 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.487 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.487 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:21.487 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.647924+0000", 2020-01-16T00:13:21.487 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:21.487 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.487 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.488 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:21.488 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.647923+0000", 2020-01-16T00:13:21.488 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00054779600000000003 2020-01-16T00:13:21.488 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.488 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.488 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:21.488 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.648470+0000", 2020-01-16T00:13:21.488 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2118e-05 2020-01-16T00:13:21.489 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.489 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.489 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:21.489 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.648483+0000", 2020-01-16T00:13:21.489 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5599999999999999e-06 2020-01-16T00:13:21.489 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.489 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.489 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:21.490 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.648489+0000", 2020-01-16T00:13:21.490 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3761000000000002e-05 2020-01-16T00:13:21.490 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.490 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.490 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:21.490 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.648543+0000", 2020-01-16T00:13:21.490 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0154e-05 2020-01-16T00:13:21.491 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.491 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.491 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:13:21.491 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.648553+0000", 2020-01-16T00:13:21.491 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0039950130000000004 2020-01-16T00:13:21.491 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.491 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.491 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:21.492 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.652548+0000", 2020-01-16T00:13:21.492 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014782599999999999 2020-01-16T00:13:21.492 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.492 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.492 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:21.492 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.652696+0000", 2020-01-16T00:13:21.493 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000212034 2020-01-16T00:13:21.493 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.493 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.493 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:13:21.493 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.652908+0000", 2020-01-16T00:13:21.493 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0047602879999999997 2020-01-16T00:13:21.494 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.494 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.494 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:21.494 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.657668+0000", 2020-01-16T00:13:21.494 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0006999999999997e-05 2020-01-16T00:13:21.494 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.494 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.494 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:21.495 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.657718+0000", 2020-01-16T00:13:21.495 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001303991 2020-01-16T00:13:21.495 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.495 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.495 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:21.495 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.659022+0000", 2020-01-16T00:13:21.495 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2401000000000001e-05 2020-01-16T00:13:21.495 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.496 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.496 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:21.496 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.659085+0000", 2020-01-16T00:13:21.496 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3877e-05 2020-01-16T00:13:21.496 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.496 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.496 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:21.496 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.659129+0000", 2020-01-16T00:13:21.497 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01120495 2020-01-16T00:13:21.497 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.497 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:21.497 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.497 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.497 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.497 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.0.13:18 2.0 2:1afbc8ac:::smithi12313934-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 1541120~378368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e33)", 2020-01-16T00:13:21.498 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:01.648496+0000", 2020-01-16T00:13:21.498 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 19.684461305999999, 2020-01-16T00:13:21.498 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010882143, 2020-01-16T00:13:21.498 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:21.498 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:21.498 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:21.498 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.498 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:21.499 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.648496+0000", 2020-01-16T00:13:21.499 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:21.499 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.499 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.499 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:21.499 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.648496+0000", 2020-01-16T00:13:21.499 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:21.500 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.500 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.500 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:21.500 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.648496+0000", 2020-01-16T00:13:21.500 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00029179500000000002 2020-01-16T00:13:21.500 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.500 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.500 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:21.501 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.648788+0000", 2020-01-16T00:13:21.501 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3240000000000003e-06 2020-01-16T00:13:21.501 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.501 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.501 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:21.501 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.648796+0000", 2020-01-16T00:13:21.501 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0180000000000002e-06 2020-01-16T00:13:21.501 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.502 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.502 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:21.502 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.648799+0000", 2020-01-16T00:13:21.502 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6814000000000003e-05 2020-01-16T00:13:21.502 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.502 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.502 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:21.503 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.648846+0000", 2020-01-16T00:13:21.503 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9180000000000006e-06 2020-01-16T00:13:21.503 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.503 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.503 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:13:21.504 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.648856+0000", 2020-01-16T00:13:21.504 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0068411820000000003 2020-01-16T00:13:21.504 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.505 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.505 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:21.505 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.655697+0000", 2020-01-16T00:13:21.506 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015195599999999999 2020-01-16T00:13:21.506 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.506 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.506 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:21.506 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.655849+0000", 2020-01-16T00:13:21.506 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024666500000000002 2020-01-16T00:13:21.506 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.507 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.507 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:13:21.507 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.656095+0000", 2020-01-16T00:13:21.507 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0016602030000000001 2020-01-16T00:13:21.507 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.507 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.508 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:21.508 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.657756+0000", 2020-01-16T00:13:21.508 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00072331900000000005 2020-01-16T00:13:21.508 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.508 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.508 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:21.508 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.658479+0000", 2020-01-16T00:13:21.508 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00083484900000000003 2020-01-16T00:13:21.509 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.509 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.509 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:21.509 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.659314+0000", 2020-01-16T00:13:21.509 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8080999999999997e-05 2020-01-16T00:13:21.509 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.509 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.509 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:21.510 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.659352+0000", 2020-01-16T00:13:21.510 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.6512999999999998e-05 2020-01-16T00:13:21.510 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.510 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.510 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:21.510 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.659378+0000", 2020-01-16T00:13:21.510 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010882143 2020-01-16T00:13:21.510 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.511 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:21.511 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.511 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.511 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.511 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2260 3.d 3:b76659c4:::smithi12313934-47:head [write 601750~535546] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:13:21.511 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:06.050577+0000", 2020-01-16T00:13:21.511 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 15.282380484000001, 2020-01-16T00:13:21.512 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012415381, 2020-01-16T00:13:21.512 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:21.512 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:21.512 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:21.512 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:21.512 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:21.512 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2260 2020-01-16T00:13:21.512 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.513 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:21.513 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.513 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:21.513 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.050577+0000", 2020-01-16T00:13:21.513 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:21.513 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.513 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.513 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:21.514 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.050577+0000", 2020-01-16T00:13:21.514 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:21.514 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.514 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.514 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:21.514 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.050574+0000", 2020-01-16T00:13:21.514 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001523487 2020-01-16T00:13:21.514 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.515 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.515 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:21.515 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052097+0000", 2020-01-16T00:13:21.515 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3697e-05 2020-01-16T00:13:21.515 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.515 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.515 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:21.516 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052121+0000", 2020-01-16T00:13:21.516 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0070000000000003e-06 2020-01-16T00:13:21.516 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.516 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.516 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:21.516 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052130+0000", 2020-01-16T00:13:21.516 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4559000000000003e-05 2020-01-16T00:13:21.516 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.517 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.517 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:21.517 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052194+0000", 2020-01-16T00:13:21.517 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000165878 2020-01-16T00:13:21.517 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.517 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.517 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:21.517 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052360+0000", 2020-01-16T00:13:21.518 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018825499999999999 2020-01-16T00:13:21.518 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.518 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.518 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:21.518 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052549+0000", 2020-01-16T00:13:21.518 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002761782 2020-01-16T00:13:21.518 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.518 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.519 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:21.519 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.055310+0000", 2020-01-16T00:13:21.519 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00060385500000000004 2020-01-16T00:13:21.519 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.519 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.519 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:21.519 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.055914+0000", 2020-01-16T00:13:21.519 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0069733809999999999 2020-01-16T00:13:21.520 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.520 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.520 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:21.520 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.062888+0000", 2020-01-16T00:13:21.520 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2884999999999999e-05 2020-01-16T00:13:21.520 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.520 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.520 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:21.521 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.062940+0000", 2020-01-16T00:13:21.521 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1996999999999997e-05 2020-01-16T00:13:21.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.521 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.521 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:21.521 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.062992+0000", 2020-01-16T00:13:21.521 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012415381 2020-01-16T00:13:21.521 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.522 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:21.522 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.522 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.522 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.523 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2295 3.d 3:b1521a3e:::smithi12313934-25:head [write 1763838~505140] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:13:21.523 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:06.660258+0000", 2020-01-16T00:13:21.523 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.672699257, 2020-01-16T00:13:21.523 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011681408000000001, 2020-01-16T00:13:21.523 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:21.523 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:21.523 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:21.523 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:21.524 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:21.524 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2295 2020-01-16T00:13:21.524 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.524 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:21.524 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.525 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:21.525 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660258+0000", 2020-01-16T00:13:21.525 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:21.525 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.525 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.525 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:21.526 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660258+0000", 2020-01-16T00:13:21.526 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:21.526 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.526 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.526 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:21.526 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660256+0000", 2020-01-16T00:13:21.526 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00066501699999999995 2020-01-16T00:13:21.526 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.527 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.527 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:21.527 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660921+0000", 2020-01-16T00:13:21.527 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7339000000000001e-05 2020-01-16T00:13:21.527 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.527 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.527 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:21.527 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660938+0000", 2020-01-16T00:13:21.528 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0219999999999993e-06 2020-01-16T00:13:21.528 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.528 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.528 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:21.528 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660947+0000", 2020-01-16T00:13:21.528 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00094237200000000002 2020-01-16T00:13:21.528 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.529 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.529 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:21.529 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.661890+0000", 2020-01-16T00:13:21.529 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000117649 2020-01-16T00:13:21.529 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.529 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.529 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:21.529 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662007+0000", 2020-01-16T00:13:21.530 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00060662599999999998 2020-01-16T00:13:21.530 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.530 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.530 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:21.530 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662614+0000", 2020-01-16T00:13:21.530 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028424899999999998 2020-01-16T00:13:21.530 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.530 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.531 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:21.531 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662898+0000", 2020-01-16T00:13:21.531 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00054904599999999997 2020-01-16T00:13:21.531 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.531 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.531 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:21.531 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.663447+0000", 2020-01-16T00:13:21.531 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0051479990000000003 2020-01-16T00:13:21.532 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.532 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.532 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:21.532 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.668595+0000", 2020-01-16T00:13:21.532 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00080881799999999999 2020-01-16T00:13:21.532 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.533 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.533 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:21.533 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.669404+0000", 2020-01-16T00:13:21.533 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0024495290000000002 2020-01-16T00:13:21.533 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.533 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.533 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:21.533 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671854+0000", 2020-01-16T00:13:21.534 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6635000000000003e-05 2020-01-16T00:13:21.534 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.534 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.534 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:21.534 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671900+0000", 2020-01-16T00:13:21.534 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9567999999999998e-05 2020-01-16T00:13:21.534 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.535 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.535 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:21.535 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671940+0000", 2020-01-16T00:13:21.535 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011681408000000001 2020-01-16T00:13:21.535 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.535 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:21.535 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.535 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.536 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.536 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2296 3.d 3:b1521a3e:::smithi12313934-25:head [setxattr _header (56) in=7b,truncate 2268978] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:13:21.536 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:06.660958+0000", 2020-01-16T00:13:21.536 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.671999745999999, 2020-01-16T00:13:21.536 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011054015, 2020-01-16T00:13:21.536 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:21.536 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:21.536 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:21.537 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:21.537 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:21.537 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2296 2020-01-16T00:13:21.537 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.537 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:21.537 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.537 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:21.537 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660958+0000", 2020-01-16T00:13:21.538 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:21.538 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.538 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.538 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:21.538 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660958+0000", 2020-01-16T00:13:21.538 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:21.538 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.539 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.539 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:21.539 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660957+0000", 2020-01-16T00:13:21.539 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3699999999999995e-06 2020-01-16T00:13:21.539 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.539 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.539 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:21.539 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660965+0000", 2020-01-16T00:13:21.540 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3590000000000008e-06 2020-01-16T00:13:21.540 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.540 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.540 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:21.540 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660973+0000", 2020-01-16T00:13:21.540 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9880000000000001e-06 2020-01-16T00:13:21.540 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.540 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.541 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:21.541 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660979+0000", 2020-01-16T00:13:21.541 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001100136 2020-01-16T00:13:21.541 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.541 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.541 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:21.541 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662079+0000", 2020-01-16T00:13:21.541 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013484700000000001 2020-01-16T00:13:21.542 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.542 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.542 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:21.542 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662214+0000", 2020-01-16T00:13:21.542 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0048090349999999997 2020-01-16T00:13:21.542 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.542 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.542 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:21.543 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.667023+0000", 2020-01-16T00:13:21.543 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012664600000000001 2020-01-16T00:13:21.543 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.543 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.543 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:21.543 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.667150+0000", 2020-01-16T00:13:21.543 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000235911 2020-01-16T00:13:21.543 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.544 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.544 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:21.544 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.667386+0000", 2020-01-16T00:13:21.544 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0017131500000000001 2020-01-16T00:13:21.544 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.544 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.544 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:21.545 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.669099+0000", 2020-01-16T00:13:21.545 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001018866 2020-01-16T00:13:21.545 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.545 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.545 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:21.545 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.670118+0000", 2020-01-16T00:13:21.545 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001828585 2020-01-16T00:13:21.545 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.546 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.546 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:21.546 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671947+0000", 2020-01-16T00:13:21.546 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2214e-05 2020-01-16T00:13:21.546 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.546 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.546 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:21.546 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671979+0000", 2020-01-16T00:13:21.547 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3135e-05 2020-01-16T00:13:21.547 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.547 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.547 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:21.547 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.672012+0000", 2020-01-16T00:13:21.547 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011054015 2020-01-16T00:13:21.547 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.547 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:21.548 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.548 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.548 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.548 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2297 3.d 3:b1521a3e:::smithi12313934-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e35)", 2020-01-16T00:13:21.548 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:06.660986+0000", 2020-01-16T00:13:21.548 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.671971399, 2020-01-16T00:13:21.548 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011220677, 2020-01-16T00:13:21.548 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:21.549 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:21.549 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:21.549 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:21.549 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:21.549 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2297 2020-01-16T00:13:21.549 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.549 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:21.549 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.550 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:21.550 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660986+0000", 2020-01-16T00:13:21.550 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:21.550 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.550 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.550 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:21.550 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660986+0000", 2020-01-16T00:13:21.550 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:21.551 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.551 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.551 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:21.551 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660985+0000", 2020-01-16T00:13:21.551 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8760000000000004e-06 2020-01-16T00:13:21.551 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.551 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.552 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:21.552 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660990+0000", 2020-01-16T00:13:21.552 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.4190000000000002e-06 2020-01-16T00:13:21.552 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.552 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.552 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:21.552 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660999+0000", 2020-01-16T00:13:21.552 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7740000000000001e-06 2020-01-16T00:13:21.553 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.553 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.553 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:21.553 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.661004+0000", 2020-01-16T00:13:21.553 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0073783590000000001 2020-01-16T00:13:21.553 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.553 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.553 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:21.554 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.668383+0000", 2020-01-16T00:13:21.554 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014443 2020-01-16T00:13:21.554 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.554 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.554 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:21.554 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.668527+0000", 2020-01-16T00:13:21.554 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0035336539999999998 2020-01-16T00:13:21.554 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.555 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.555 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:21.555 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.672061+0000", 2020-01-16T00:13:21.555 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6445000000000005e-05 2020-01-16T00:13:21.555 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.555 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.555 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:21.555 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.672127+0000", 2020-01-16T00:13:21.556 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.9497000000000001e-05 2020-01-16T00:13:21.556 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.556 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.556 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:21.556 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.672207+0000", 2020-01-16T00:13:21.556 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011220677 2020-01-16T00:13:21.556 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.556 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:21.557 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.557 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.557 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:21.557 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:21.558 INFO:tasks.rados.rados.0.smithi123.stdout:1187: finishing write tid 3 to smithi12313934-48 2020-01-16T00:13:21.558 INFO:tasks.rados.rados.0.smithi123.stdout:1187: finishing write tid 5 to smithi12313934-48 2020-01-16T00:13:21.558 INFO:tasks.rados.rados.0.smithi123.stdout:1187: finishing write tid 6 to smithi12313934-48 2020-01-16T00:13:21.558 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 48 v 132 (ObjNum 350 snap 0 seq_num 350) dirty exists 2020-01-16T00:13:21.559 INFO:tasks.rados.rados.0.smithi123.stdout:1187: left oid 48 (ObjNum 350 snap 0 seq_num 350) 2020-01-16T00:13:21.559 INFO:tasks.rados.rados.0.smithi123.stdout:1189: expect (ObjNum 330 snap 0 seq_num 330) 2020-01-16T00:13:21.559 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-01-16T00:13:21.563 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:21.558+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:21.661 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:21.656+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:21.695 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:21.695 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:13:21.695 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:13:21.695 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:21.705 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-01-16T00:13:21.760 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:21.758+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:21.780 INFO:tasks.rados.rados.0.smithi123.stdout:1190: finishing write tid 1 to smithi12313934-39 2020-01-16T00:13:21.780 INFO:tasks.rados.rados.0.smithi123.stdout:1190: finishing write tid 2 to smithi12313934-39 2020-01-16T00:13:21.780 INFO:tasks.rados.rados.0.smithi123.stdout:1190: finishing write tid 3 to smithi12313934-39 2020-01-16T00:13:21.780 INFO:tasks.rados.rados.0.smithi123.stdout:1183: done (12 left) 2020-01-16T00:13:21.780 INFO:tasks.rados.rados.0.smithi123.stdout:1185: done (11 left) 2020-01-16T00:13:21.781 INFO:tasks.rados.rados.0.smithi123.stdout:1186: done (10 left) 2020-01-16T00:13:21.781 INFO:tasks.rados.rados.0.smithi123.stdout:1187: done (9 left) 2020-01-16T00:13:21.781 INFO:tasks.rados.rados.0.smithi123.stdout:1188: done (8 left) 2020-01-16T00:13:21.781 INFO:tasks.rados.rados.0.smithi123.stdout:1189: done (7 left) 2020-01-16T00:13:21.781 INFO:tasks.rados.rados.0.smithi123.stdout:1196: done (6 left) 2020-01-16T00:13:21.781 INFO:tasks.rados.rados.0.smithi123.stdout:1197: read oid 18 snap -1 2020-01-16T00:13:21.781 INFO:tasks.rados.rados.0.smithi123.stdout:1197: expect (ObjNum 220 snap 0 seq_num 220) 2020-01-16T00:13:21.782 INFO:tasks.rados.rados.0.smithi123.stdout:1198: write (excl) oid 48 current snap is 0 2020-01-16T00:13:21.782 INFO:tasks.rados.rados.0.smithi123.stdout:1198: seq_num 356 ranges {783292=540882,1932766=507787,3210350=572013} 2020-01-16T00:13:21.791 INFO:tasks.rados.rados.0.smithi123.stdout:1198: writing smithi12313934-48 from 783292 to 1324174 tid 1 2020-01-16T00:13:21.798 INFO:tasks.rados.rados.0.smithi123.stdout:1198: writing smithi12313934-48 from 1932766 to 2440553 tid 2 2020-01-16T00:13:21.807 INFO:tasks.rados.rados.0.smithi123.stdout:1198: writing smithi12313934-48 from 3210350 to 3782363 tid 3 2020-01-16T00:13:21.807 INFO:tasks.rados.rados.0.smithi123.stdout:1199: read oid 43 snap -1 2020-01-16T00:13:21.807 INFO:tasks.rados.rados.0.smithi123.stdout:1199: expect deleted 2020-01-16T00:13:21.807 INFO:tasks.rados.rados.0.smithi123.stdout:1200: write (excl) oid 28 current snap is 0 2020-01-16T00:13:21.807 INFO:tasks.rados.rados.0.smithi123.stdout:1200: seq_num 357 ranges {400099=616131,1591248=408851} 2020-01-16T00:13:21.811 INFO:tasks.rados.rados.0.smithi123.stdout:1200: writing smithi12313934-28 from 400099 to 1016230 tid 1 2020-01-16T00:13:21.817 INFO:tasks.rados.rados.0.smithi123.stdout:1200: writing smithi12313934-28 from 1591248 to 2000099 tid 2 2020-01-16T00:13:21.818 INFO:tasks.rados.rados.0.smithi123.stdout:1201: write (excl) oid 25 current snap is 0 2020-01-16T00:13:21.818 INFO:tasks.rados.rados.0.smithi123.stdout:1201: seq_num 358 ranges {416906=407733,1366083=511456} 2020-01-16T00:13:21.822 INFO:tasks.rados.rados.0.smithi123.stdout:1201: writing smithi12313934-25 from 416906 to 824639 tid 1 2020-01-16T00:13:21.828 INFO:tasks.rados.rados.0.smithi123.stdout:1201: writing smithi12313934-25 from 1366083 to 1877539 tid 2 2020-01-16T00:13:21.829 INFO:tasks.rados.rados.0.smithi123.stdout:1202: read oid 9 snap -1 2020-01-16T00:13:21.829 INFO:tasks.rados.rados.0.smithi123.stdout:1202: expect deleted 2020-01-16T00:13:21.829 INFO:tasks.rados.rados.0.smithi123.stdout:1203: write (excl) oid 15 current snap is 0 2020-01-16T00:13:21.829 INFO:tasks.rados.rados.0.smithi123.stdout:1190: finishing write tid 5 to smithi12313934-39 2020-01-16T00:13:21.830 INFO:tasks.rados.rados.0.smithi123.stdout:1203: seq_num 359 ranges {433713=482982,1424565=609148} 2020-01-16T00:13:21.833 INFO:tasks.rados.rados.0.smithi123.stdout:1203: writing smithi12313934-15 from 433713 to 916695 tid 1 2020-01-16T00:13:21.840 INFO:tasks.rados.rados.0.smithi123.stdout:1203: writing smithi12313934-15 from 1424565 to 2033713 tid 2 2020-01-16T00:13:21.840 INFO:tasks.rados.rados.0.smithi123.stdout:1204: read oid 34 snap -1 2020-01-16T00:13:21.840 INFO:tasks.rados.rados.0.smithi123.stdout:1204: expect deleted 2020-01-16T00:13:21.841 INFO:tasks.rados.rados.0.smithi123.stdout:1205: read oid 3 snap -1 2020-01-16T00:13:21.841 INFO:tasks.rados.rados.0.smithi123.stdout:1205: expect (ObjNum 260 snap 0 seq_num 260) 2020-01-16T00:13:21.841 INFO:tasks.rados.rados.0.smithi123.stdout:1206: write oid 6 current snap is 0 2020-01-16T00:13:21.841 INFO:tasks.rados.rados.0.smithi123.stdout:1206: seq_num 360 ranges {450520=558231,1483047=515125} 2020-01-16T00:13:21.846 INFO:tasks.rados.rados.0.smithi123.stdout:1206: writing smithi12313934-6 from 450520 to 1008751 tid 1 2020-01-16T00:13:21.852 INFO:tasks.rados.rados.0.smithi123.stdout:1206: writing smithi12313934-6 from 1483047 to 1998172 tid 2 2020-01-16T00:13:21.852 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:21.852 INFO:tasks.rados.rados.0.smithi123.stdout:1190: finishing write tid 6 to smithi12313934-39 2020-01-16T00:13:21.852 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 39 v 178 (ObjNum 351 snap 0 seq_num 351) dirty exists 2020-01-16T00:13:21.853 INFO:tasks.rados.rados.0.smithi123.stdout:1190: left oid 39 (ObjNum 351 snap 0 seq_num 351) 2020-01-16T00:13:21.853 INFO:tasks.rados.rados.0.smithi123.stdout:1191: expect (ObjNum 326 snap 0 seq_num 326) 2020-01-16T00:13:21.861 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:21.858+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:21.864 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:21.865 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:13:21.865 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:13:21.865 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:13:21.865 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:21.875 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-01-16T00:13:21.876 INFO:teuthology.orchestra.run.smithi123.stderr:nodeep-scrub is set 2020-01-16T00:13:21.962 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:21.958+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:21.979 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:21.979 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:13:21.979 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:13:21.980 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:13:21.980 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.980 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1241 3.a 3:5c245412:::smithi12313934-19:head [read 0~3815156,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:13:21.980 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.589903+0000", 2020-01-16T00:13:21.980 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 39.384439678, 2020-01-16T00:13:21.980 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0096065560000000005, 2020-01-16T00:13:21.980 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:21.981 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:21.981 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:21.981 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:21.981 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:21.981 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1241 2020-01-16T00:13:21.981 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.981 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:21.981 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.982 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:21.982 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.589903+0000", 2020-01-16T00:13:21.982 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:21.982 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.982 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.982 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:21.982 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.589903+0000", 2020-01-16T00:13:21.983 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999952 2020-01-16T00:13:21.983 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.983 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.983 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:21.983 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.589898+0000", 2020-01-16T00:13:21.983 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4389e-05 2020-01-16T00:13:21.983 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.983 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.984 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:21.984 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.589922+0000", 2020-01-16T00:13:21.984 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4159999999999998e-05 2020-01-16T00:13:21.984 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.984 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.984 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:21.984 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.589957+0000", 2020-01-16T00:13:21.985 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7368999999999999e-05 2020-01-16T00:13:21.985 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.985 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.986 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:21.986 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.589974+0000", 2020-01-16T00:13:21.986 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000130407 2020-01-16T00:13:21.986 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.986 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.987 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:21.987 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590104+0000", 2020-01-16T00:13:21.987 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00040217699999999998 2020-01-16T00:13:21.987 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.987 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.987 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:21.987 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590506+0000", 2020-01-16T00:13:21.988 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0090030230000000006 2020-01-16T00:13:21.988 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.988 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.988 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:21.988 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599510+0000", 2020-01-16T00:13:21.988 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0096065560000000005 2020-01-16T00:13:21.988 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.988 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:21.989 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.989 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.989 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.989 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1242 3.a 3:5c245412:::smithi12313934-19:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:13:21.989 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.590003+0000", 2020-01-16T00:13:21.989 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 39.384339169, 2020-01-16T00:13:21.989 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010247029, 2020-01-16T00:13:21.990 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:21.990 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:21.990 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:21.990 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:21.990 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:21.990 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1242 2020-01-16T00:13:21.990 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.990 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:21.991 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.991 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:21.991 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590003+0000", 2020-01-16T00:13:21.991 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:21.991 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.991 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.991 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:21.991 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590003+0000", 2020-01-16T00:13:21.992 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:21.992 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.992 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.992 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:21.992 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590002+0000", 2020-01-16T00:13:21.992 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0108e-05 2020-01-16T00:13:21.992 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.992 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.993 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:21.993 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590012+0000", 2020-01-16T00:13:21.993 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5318000000000001e-05 2020-01-16T00:13:21.993 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.993 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.993 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:21.993 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590027+0000", 2020-01-16T00:13:21.994 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0720000000000001e-05 2020-01-16T00:13:21.994 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.994 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.994 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:21.994 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590038+0000", 2020-01-16T00:13:21.994 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0095349949999999992 2020-01-16T00:13:21.994 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.994 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.995 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:21.995 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599573+0000", 2020-01-16T00:13:21.995 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017107600000000001 2020-01-16T00:13:21.995 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.995 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.995 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:21.996 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599744+0000", 2020-01-16T00:13:21.996 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00050659499999999998 2020-01-16T00:13:21.996 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.996 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.996 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:21.996 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.600251+0000", 2020-01-16T00:13:21.996 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010247029 2020-01-16T00:13:21.996 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.997 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:21.997 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:21.997 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.997 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.997 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1243 3.a 3:5c245412:::smithi12313934-19:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:13:21.997 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.590095+0000", 2020-01-16T00:13:21.997 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 39.384247760000001, 2020-01-16T00:13:21.998 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010591455, 2020-01-16T00:13:21.998 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:21.998 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:21.998 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:21.998 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:21.998 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:21.998 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1243 2020-01-16T00:13:21.998 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.999 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:21.999 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.999 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:21.999 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590095+0000", 2020-01-16T00:13:21.999 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:21.999 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:21.999 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:21.999 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:22.000 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590095+0000", 2020-01-16T00:13:22.000 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:22.000 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.000 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.000 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:22.000 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590093+0000", 2020-01-16T00:13:22.000 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5619999999999993e-06 2020-01-16T00:13:22.001 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.001 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.001 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:22.001 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590102+0000", 2020-01-16T00:13:22.001 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.259e-05 2020-01-16T00:13:22.001 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.001 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.002 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:22.002 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590115+0000", 2020-01-16T00:13:22.002 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8730000000000001e-06 2020-01-16T00:13:22.002 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.002 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.002 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:22.002 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590122+0000", 2020-01-16T00:13:22.003 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010163165 2020-01-16T00:13:22.003 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.003 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.003 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:22.003 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.600285+0000", 2020-01-16T00:13:22.003 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0128000000000002e-05 2020-01-16T00:13:22.003 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.004 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.004 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:22.004 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.600375+0000", 2020-01-16T00:13:22.004 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000311312 2020-01-16T00:13:22.004 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.004 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.004 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:22.005 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.600686+0000", 2020-01-16T00:13:22.005 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010591455 2020-01-16T00:13:22.005 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:22.005 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:22.005 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:22.005 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.005 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.005 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1654 3.a 3:5069665e:::smithi12313934-2:head [write 1614880~581041] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:13:22.006 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.714839+0000", 2020-01-16T00:13:22.006 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 29.259503771999999, 2020-01-16T00:13:22.006 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013588093000000001, 2020-01-16T00:13:22.006 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:22.006 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:22.006 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:22.006 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:22.007 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:22.007 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1654 2020-01-16T00:13:22.007 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.007 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:22.007 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.007 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:22.007 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.714839+0000", 2020-01-16T00:13:22.008 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:22.008 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.008 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.008 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:22.008 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.714839+0000", 2020-01-16T00:13:22.008 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:22.008 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.008 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.009 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:22.009 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.714836+0000", 2020-01-16T00:13:22.009 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00054657999999999996 2020-01-16T00:13:22.009 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.009 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.009 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:22.009 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.715382+0000", 2020-01-16T00:13:22.010 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7743000000000001e-05 2020-01-16T00:13:22.010 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.010 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.010 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:22.010 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.715420+0000", 2020-01-16T00:13:22.010 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.221e-05 2020-01-16T00:13:22.010 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.010 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.011 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:22.011 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.715432+0000", 2020-01-16T00:13:22.011 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2487000000000001e-05 2020-01-16T00:13:22.011 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.011 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.011 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:22.011 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.715485+0000", 2020-01-16T00:13:22.011 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000726284 2020-01-16T00:13:22.012 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.012 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.012 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:22.012 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.716211+0000", 2020-01-16T00:13:22.012 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0067930899999999999 2020-01-16T00:13:22.012 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.012 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.013 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:22.013 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.723004+0000", 2020-01-16T00:13:22.013 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012579400000000001 2020-01-16T00:13:22.013 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.013 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.013 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:22.013 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.723130+0000", 2020-01-16T00:13:22.013 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013465800000000001 2020-01-16T00:13:22.014 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.014 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.014 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:22.014 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.723265+0000", 2020-01-16T00:13:22.014 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023307250000000001 2020-01-16T00:13:22.014 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.014 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.015 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:22.015 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.725595+0000", 2020-01-16T00:13:22.015 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012582139999999999 2020-01-16T00:13:22.015 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.015 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.015 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:22.015 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.726853+0000", 2020-01-16T00:13:22.015 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014479110000000001 2020-01-16T00:13:22.016 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.016 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.016 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:22.016 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.728301+0000", 2020-01-16T00:13:22.016 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6866000000000007e-05 2020-01-16T00:13:22.016 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.016 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.016 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:22.017 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.728368+0000", 2020-01-16T00:13:22.017 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8740999999999999e-05 2020-01-16T00:13:22.017 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.017 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.017 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:22.017 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.728427+0000", 2020-01-16T00:13:22.017 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013588093000000001 2020-01-16T00:13:22.017 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:22.018 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:22.018 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:22.018 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.018 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.018 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1655 3.a 3:5069665e:::smithi12313934-2:head [write 2785791~71399] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:13:22.018 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.718297+0000", 2020-01-16T00:13:22.018 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 29.256045919000002, 2020-01-16T00:13:22.018 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013817354, 2020-01-16T00:13:22.019 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:22.019 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:22.019 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:22.019 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:22.019 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:22.019 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1655 2020-01-16T00:13:22.019 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.020 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:22.020 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.020 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:22.020 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718297+0000", 2020-01-16T00:13:22.020 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:22.020 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.020 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.020 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:22.021 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718297+0000", 2020-01-16T00:13:22.021 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:22.021 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.021 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.021 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:22.021 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718294+0000", 2020-01-16T00:13:22.021 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000103558 2020-01-16T00:13:22.021 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.022 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.022 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:22.022 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718397+0000", 2020-01-16T00:13:22.022 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9287e-05 2020-01-16T00:13:22.022 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.022 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.022 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:22.022 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718417+0000", 2020-01-16T00:13:22.023 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6680000000000001e-06 2020-01-16T00:13:22.023 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.023 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.023 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:22.023 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718424+0000", 2020-01-16T00:13:22.023 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2701999999999994e-05 2020-01-16T00:13:22.023 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.023 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.024 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:22.024 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718517+0000", 2020-01-16T00:13:22.024 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3893000000000006e-05 2020-01-16T00:13:22.024 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.024 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.024 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:22.024 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718581+0000", 2020-01-16T00:13:22.024 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0059630079999999997 2020-01-16T00:13:22.025 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.025 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.025 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:22.025 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.724544+0000", 2020-01-16T00:13:22.025 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000309108 2020-01-16T00:13:22.025 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.025 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.026 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:22.026 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.724853+0000", 2020-01-16T00:13:22.026 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.005469794 2020-01-16T00:13:22.026 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.026 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.026 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:22.026 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.730323+0000", 2020-01-16T00:13:22.026 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018506100000000001 2020-01-16T00:13:22.027 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.027 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.027 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:22.027 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.730508+0000", 2020-01-16T00:13:22.027 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016729099999999999 2020-01-16T00:13:22.027 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.027 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.027 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:22.028 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.730675+0000", 2020-01-16T00:13:22.028 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00094193300000000005 2020-01-16T00:13:22.028 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.028 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.028 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:22.028 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731617+0000", 2020-01-16T00:13:22.028 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000214613 2020-01-16T00:13:22.028 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.029 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.029 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:22.029 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731832+0000", 2020-01-16T00:13:22.029 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018326400000000001 2020-01-16T00:13:22.029 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.029 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.029 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:22.029 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732015+0000", 2020-01-16T00:13:22.030 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0392e-05 2020-01-16T00:13:22.030 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.030 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.030 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:22.030 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732075+0000", 2020-01-16T00:13:22.030 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8677999999999998e-05 2020-01-16T00:13:22.030 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.030 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.031 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:22.031 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732114+0000", 2020-01-16T00:13:22.031 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013817354 2020-01-16T00:13:22.031 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:22.031 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:22.031 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:22.031 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.031 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.032 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1656 3.a 3:5069665e:::smithi12313934-2:head [setxattr _header (55) in=7b,truncate 2857190] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:13:22.032 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.718453+0000", 2020-01-16T00:13:22.032 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 29.255889238000002, 2020-01-16T00:13:22.032 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014083462, 2020-01-16T00:13:22.032 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:22.032 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:22.032 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:22.033 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:22.033 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:22.033 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1656 2020-01-16T00:13:22.033 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.033 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:22.033 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.033 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:22.033 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718453+0000", 2020-01-16T00:13:22.034 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:22.034 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.034 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.034 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:22.034 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718453+0000", 2020-01-16T00:13:22.034 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999828 2020-01-16T00:13:22.035 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.035 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.035 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:22.035 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718436+0000", 2020-01-16T00:13:22.035 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4051999999999999e-05 2020-01-16T00:13:22.035 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.035 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.035 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:22.036 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718460+0000", 2020-01-16T00:13:22.036 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6380000000000004e-06 2020-01-16T00:13:22.036 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.036 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.036 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:22.036 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718468+0000", 2020-01-16T00:13:22.036 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6750000000000001e-06 2020-01-16T00:13:22.037 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.037 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.037 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:22.037 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718472+0000", 2020-01-16T00:13:22.037 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016218499999999999 2020-01-16T00:13:22.037 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.037 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.037 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:22.038 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718635+0000", 2020-01-16T00:13:22.038 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2765e-05 2020-01-16T00:13:22.038 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.038 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.038 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:22.038 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718697+0000", 2020-01-16T00:13:22.038 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0062239249999999999 2020-01-16T00:13:22.039 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.039 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.039 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:22.039 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.724921+0000", 2020-01-16T00:13:22.039 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1527999999999999e-05 2020-01-16T00:13:22.039 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.039 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.039 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:22.040 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.724973+0000", 2020-01-16T00:13:22.040 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0060569719999999999 2020-01-16T00:13:22.040 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.040 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.040 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:22.040 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731030+0000", 2020-01-16T00:13:22.040 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5859999999999996e-05 2020-01-16T00:13:22.040 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.041 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.041 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:22.041 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731096+0000", 2020-01-16T00:13:22.041 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000112018 2020-01-16T00:13:22.041 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.041 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.041 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:22.042 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731208+0000", 2020-01-16T00:13:22.042 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000945889 2020-01-16T00:13:22.042 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.042 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.042 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:22.042 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732154+0000", 2020-01-16T00:13:22.042 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0834000000000003e-05 2020-01-16T00:13:22.042 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.043 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.043 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:22.043 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732184+0000", 2020-01-16T00:13:22.043 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028587600000000001 2020-01-16T00:13:22.043 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.043 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.043 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:22.044 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732470+0000", 2020-01-16T00:13:22.044 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1953999999999999e-05 2020-01-16T00:13:22.044 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.044 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.044 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:22.044 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732502+0000", 2020-01-16T00:13:22.044 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4644999999999998e-05 2020-01-16T00:13:22.044 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.045 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.045 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:22.045 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732537+0000", 2020-01-16T00:13:22.045 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014083462 2020-01-16T00:13:22.045 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:22.045 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:22.045 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:22.046 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.046 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.046 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1657 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-01-16T00:13:22.046 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.718495+0000", 2020-01-16T00:13:22.046 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 29.255847573, 2020-01-16T00:13:22.046 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014299630000000001, 2020-01-16T00:13:22.046 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:22.047 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:22.047 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:22.047 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:22.047 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:22.047 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1657 2020-01-16T00:13:22.047 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.047 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:22.048 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.048 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:22.048 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718495+0000", 2020-01-16T00:13:22.048 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:22.048 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.048 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.048 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:22.048 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718495+0000", 2020-01-16T00:13:22.049 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:22.049 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.049 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.049 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:22.049 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718494+0000", 2020-01-16T00:13:22.049 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8860000000000003e-06 2020-01-16T00:13:22.049 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.050 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.050 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:22.050 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718499+0000", 2020-01-16T00:13:22.050 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9786000000000002e-05 2020-01-16T00:13:22.050 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.050 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.050 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:22.051 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718539+0000", 2020-01-16T00:13:22.051 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1869999999999999e-05 2020-01-16T00:13:22.051 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.051 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.051 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:22.051 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718561+0000", 2020-01-16T00:13:22.051 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000172206 2020-01-16T00:13:22.051 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.052 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.052 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:22.052 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718733+0000", 2020-01-16T00:13:22.052 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0662999999999998e-05 2020-01-16T00:13:22.052 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.052 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.052 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:22.052 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718793+0000", 2020-01-16T00:13:22.053 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.006217958 2020-01-16T00:13:22.053 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.053 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.053 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:22.053 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.725011+0000", 2020-01-16T00:13:22.053 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1625999999999996e-05 2020-01-16T00:13:22.053 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.054 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.054 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:22.054 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.725073+0000", 2020-01-16T00:13:22.054 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0063923119999999998 2020-01-16T00:13:22.054 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.054 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.054 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:22.054 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731465+0000", 2020-01-16T00:13:22.055 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2534000000000005e-05 2020-01-16T00:13:22.055 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.055 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.055 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:22.055 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731558+0000", 2020-01-16T00:13:22.055 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010238809999999999 2020-01-16T00:13:22.055 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.056 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.056 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:22.056 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732582+0000", 2020-01-16T00:13:22.056 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000100918 2020-01-16T00:13:22.056 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.056 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.056 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:22.056 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732683+0000", 2020-01-16T00:13:22.057 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000112064 2020-01-16T00:13:22.057 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.057 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.057 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:22.057 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732795+0000", 2020-01-16T00:13:22.057 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014299630000000001 2020-01-16T00:13:22.057 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:22.057 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:22.058 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:22.058 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.058 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.058 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1917 3.a 3:5069665e:::smithi12313934-2:head [write 2113302~674388] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:13:22.058 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:57.330712+0000", 2020-01-16T00:13:22.058 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 24.643631058, 2020-01-16T00:13:22.058 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012781043000000001, 2020-01-16T00:13:22.059 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:22.059 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:22.059 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:22.059 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:22.059 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:22.059 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1917 2020-01-16T00:13:22.059 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.060 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:22.060 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.060 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:22.060 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.330712+0000", 2020-01-16T00:13:22.060 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:22.060 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.060 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.060 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:22.061 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.330712+0000", 2020-01-16T00:13:22.061 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:22.061 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.061 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.061 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:22.061 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.330709+0000", 2020-01-16T00:13:22.062 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00071550499999999996 2020-01-16T00:13:22.062 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.062 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.062 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:22.062 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331424+0000", 2020-01-16T00:13:22.062 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3354e-05 2020-01-16T00:13:22.062 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.062 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.063 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:22.063 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331447+0000", 2020-01-16T00:13:22.063 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8200000000000008e-06 2020-01-16T00:13:22.063 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.063 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.063 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:22.063 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331457+0000", 2020-01-16T00:13:22.064 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5557999999999997e-05 2020-01-16T00:13:22.064 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.064 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.064 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:22.064 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331533+0000", 2020-01-16T00:13:22.064 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030925000000000001 2020-01-16T00:13:22.064 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.064 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.065 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:22.065 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331842+0000", 2020-01-16T00:13:22.065 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0078507860000000002 2020-01-16T00:13:22.065 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.065 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.065 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:22.066 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.339693+0000", 2020-01-16T00:13:22.066 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9321000000000007e-05 2020-01-16T00:13:22.066 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.066 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.066 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:22.066 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.339792+0000", 2020-01-16T00:13:22.066 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015147700000000001 2020-01-16T00:13:22.066 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.067 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.067 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:22.067 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.339944+0000", 2020-01-16T00:13:22.067 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002663066 2020-01-16T00:13:22.067 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.067 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.068 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:22.068 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.342607+0000", 2020-01-16T00:13:22.068 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0001552 2020-01-16T00:13:22.068 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.068 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.068 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:22.069 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.342762+0000", 2020-01-16T00:13:22.069 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00063636900000000002 2020-01-16T00:13:22.069 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.070 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.070 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:22.070 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343398+0000", 2020-01-16T00:13:22.070 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9687000000000001e-05 2020-01-16T00:13:22.070 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.070 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.071 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:22.071 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343438+0000", 2020-01-16T00:13:22.071 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4647e-05 2020-01-16T00:13:22.071 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.071 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.071 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:22.071 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343493+0000", 2020-01-16T00:13:22.071 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012781043000000001 2020-01-16T00:13:22.072 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:22.072 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:22.072 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:22.072 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.072 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.072 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2114 3.a 3:55e7052a:::smithi12313934-7:head [write 682417~473109] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:13:22.072 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:01.650529+0000", 2020-01-16T00:13:22.073 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 20.323813748999999, 2020-01-16T00:13:22.073 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0093900500000000005, 2020-01-16T00:13:22.073 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:22.073 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:22.073 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:22.073 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:22.073 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:22.074 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2114 2020-01-16T00:13:22.074 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.074 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:22.074 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.074 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:22.074 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.650529+0000", 2020-01-16T00:13:22.074 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:22.074 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.075 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.075 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:22.075 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.650529+0000", 2020-01-16T00:13:22.075 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:22.075 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.075 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.075 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:22.075 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.650526+0000", 2020-01-16T00:13:22.076 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00074189500000000001 2020-01-16T00:13:22.076 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.076 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.076 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:22.076 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.651267+0000", 2020-01-16T00:13:22.076 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0999e-05 2020-01-16T00:13:22.076 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.077 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.077 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:22.077 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.651288+0000", 2020-01-16T00:13:22.077 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8230000000000006e-06 2020-01-16T00:13:22.077 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.077 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.077 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:22.077 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.651298+0000", 2020-01-16T00:13:22.078 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019944599999999999 2020-01-16T00:13:22.078 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.078 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.078 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:22.078 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.651498+0000", 2020-01-16T00:13:22.078 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0808e-05 2020-01-16T00:13:22.078 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.079 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.079 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op must wait for map", 2020-01-16T00:13:22.079 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.651529+0000", 2020-01-16T00:13:22.079 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0027072860000000002 2020-01-16T00:13:22.079 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.079 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.079 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:22.079 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.654236+0000", 2020-01-16T00:13:22.080 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010767700000000001 2020-01-16T00:13:22.080 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.080 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.080 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:22.080 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.654344+0000", 2020-01-16T00:13:22.080 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000261341 2020-01-16T00:13:22.080 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.080 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.081 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:22.081 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.654605+0000", 2020-01-16T00:13:22.081 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001537737 2020-01-16T00:13:22.081 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.081 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.081 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:22.081 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.656143+0000", 2020-01-16T00:13:22.082 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00302163 2020-01-16T00:13:22.082 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.082 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.082 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:22.082 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.659164+0000", 2020-01-16T00:13:22.082 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00063603599999999998 2020-01-16T00:13:22.082 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.082 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.083 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:22.083 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.659800+0000", 2020-01-16T00:13:22.083 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6470000000000006e-05 2020-01-16T00:13:22.083 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.083 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.083 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:22.083 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.659867+0000", 2020-01-16T00:13:22.083 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2218e-05 2020-01-16T00:13:22.084 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.084 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.084 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:22.084 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.659919+0000", 2020-01-16T00:13:22.084 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0093900500000000005 2020-01-16T00:13:22.084 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:22.084 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:22.084 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:22.085 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.085 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.085 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2308 3.a 3:5c245412:::smithi12313934-19:head [write 702592~703393] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:13:22.085 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:07.298509+0000", 2020-01-16T00:13:22.085 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.675833402, 2020-01-16T00:13:22.085 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012139225999999999, 2020-01-16T00:13:22.085 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:22.086 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:22.086 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:22.086 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:22.086 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:22.086 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2308 2020-01-16T00:13:22.086 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.086 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:22.087 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.087 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:22.087 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.298509+0000", 2020-01-16T00:13:22.087 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:22.087 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.087 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.088 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:22.088 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.298509+0000", 2020-01-16T00:13:22.088 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999957 2020-01-16T00:13:22.088 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.088 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.088 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:22.088 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.298505+0000", 2020-01-16T00:13:22.088 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00085313800000000003 2020-01-16T00:13:22.089 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.089 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.089 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:22.089 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.299358+0000", 2020-01-16T00:13:22.089 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5397e-05 2020-01-16T00:13:22.089 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.089 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.090 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:22.090 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.299394+0000", 2020-01-16T00:13:22.090 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3699e-05 2020-01-16T00:13:22.090 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.090 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.090 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:22.090 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.299407+0000", 2020-01-16T00:13:22.090 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014281999999999999 2020-01-16T00:13:22.091 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.091 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.091 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:22.091 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.299550+0000", 2020-01-16T00:13:22.091 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024247099999999999 2020-01-16T00:13:22.091 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.091 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.091 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:22.092 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.299793+0000", 2020-01-16T00:13:22.092 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00031642200000000001 2020-01-16T00:13:22.092 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.092 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.092 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:22.092 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.300109+0000", 2020-01-16T00:13:22.092 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0078127560000000006 2020-01-16T00:13:22.093 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.093 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.093 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:22.093 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.307922+0000", 2020-01-16T00:13:22.093 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0024170189999999999 2020-01-16T00:13:22.093 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.093 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.093 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:22.094 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.310339+0000", 2020-01-16T00:13:22.094 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.4764000000000004e-05 2020-01-16T00:13:22.094 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.094 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.094 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:22.094 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.310424+0000", 2020-01-16T00:13:22.094 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000128319 2020-01-16T00:13:22.095 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.095 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.095 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:22.095 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.310552+0000", 2020-01-16T00:13:22.095 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.6587000000000002e-05 2020-01-16T00:13:22.095 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.095 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.095 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:22.096 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.310648+0000", 2020-01-16T00:13:22.096 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012139225999999999 2020-01-16T00:13:22.096 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:22.096 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:22.096 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:22.096 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.096 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.097 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2504 3.a 3:5069665e:::smithi12313934-2:head [write 490837~718873] snapc 0=[] ondisk+write+known_if_redirected e37)", 2020-01-16T00:13:22.097 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:11.279881+0000", 2020-01-16T00:13:22.097 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 10.694461458999999, 2020-01-16T00:13:22.097 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010249138, 2020-01-16T00:13:22.097 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:22.097 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:22.097 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:22.098 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:22.098 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:22.098 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2504 2020-01-16T00:13:22.098 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.098 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:22.098 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.098 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:22.098 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.279881+0000", 2020-01-16T00:13:22.099 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:22.099 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.099 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.099 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:22.099 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.279881+0000", 2020-01-16T00:13:22.099 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999948 2020-01-16T00:13:22.099 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.100 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.100 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:22.100 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.279876+0000", 2020-01-16T00:13:22.100 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0025362599999999998 2020-01-16T00:13:22.100 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.100 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.100 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:22.100 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.282412+0000", 2020-01-16T00:13:22.101 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5186000000000003e-05 2020-01-16T00:13:22.101 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.101 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.101 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:22.101 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.282467+0000", 2020-01-16T00:13:22.101 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8800000000000003e-06 2020-01-16T00:13:22.101 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.101 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.102 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:22.102 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.282477+0000", 2020-01-16T00:13:22.102 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015124100000000001 2020-01-16T00:13:22.102 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.102 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.102 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:22.102 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.282628+0000", 2020-01-16T00:13:22.102 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00026402000000000001 2020-01-16T00:13:22.103 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.103 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.103 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:22.103 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.282892+0000", 2020-01-16T00:13:22.103 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00044943700000000001 2020-01-16T00:13:22.103 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.103 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.104 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:22.104 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.283342+0000", 2020-01-16T00:13:22.104 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0041154019999999998 2020-01-16T00:13:22.104 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.104 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.104 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:22.104 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.287457+0000", 2020-01-16T00:13:22.105 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0865999999999995e-05 2020-01-16T00:13:22.105 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.105 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.105 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:22.105 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.287538+0000", 2020-01-16T00:13:22.105 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0022868670000000002 2020-01-16T00:13:22.105 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.105 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.106 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:22.106 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289825+0000", 2020-01-16T00:13:22.106 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016695699999999999 2020-01-16T00:13:22.106 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.106 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.106 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:22.106 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289992+0000", 2020-01-16T00:13:22.106 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013842600000000001 2020-01-16T00:13:22.107 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.107 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.107 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:22.107 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.290130+0000", 2020-01-16T00:13:22.107 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010249138 2020-01-16T00:13:22.107 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:22.107 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:22.108 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:22.108 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.108 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.108 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2505 3.a 3:5069665e:::smithi12313934-2:head [write 1862260~479427] snapc 0=[] ondisk+write+known_if_redirected e37)", 2020-01-16T00:13:22.108 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:11.286548+0000", 2020-01-16T00:13:22.108 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 10.687794890999999, 2020-01-16T00:13:22.108 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013314061, 2020-01-16T00:13:22.108 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:22.109 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:22.109 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:22.109 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:22.109 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:22.109 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2505 2020-01-16T00:13:22.109 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.109 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:22.109 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.110 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:22.110 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.286548+0000", 2020-01-16T00:13:22.110 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:22.110 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.110 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.111 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:22.111 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.286548+0000", 2020-01-16T00:13:22.111 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999957 2020-01-16T00:13:22.111 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.111 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.112 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:22.112 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.286544+0000", 2020-01-16T00:13:22.112 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042043700000000001 2020-01-16T00:13:22.112 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.112 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.112 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:22.112 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.286964+0000", 2020-01-16T00:13:22.112 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5756999999999997e-05 2020-01-16T00:13:22.113 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.113 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.113 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:22.113 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.287000+0000", 2020-01-16T00:13:22.113 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8960000000000001e-05 2020-01-16T00:13:22.113 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.113 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.113 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:22.114 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.287019+0000", 2020-01-16T00:13:22.114 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.7219000000000005e-05 2020-01-16T00:13:22.114 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.114 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.114 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:22.114 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.287116+0000", 2020-01-16T00:13:22.114 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033208899999999998 2020-01-16T00:13:22.115 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.115 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.115 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:22.115 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.287448+0000", 2020-01-16T00:13:22.115 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0082732420000000001 2020-01-16T00:13:22.115 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.115 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.115 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:22.116 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.295721+0000", 2020-01-16T00:13:22.116 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017259999999999999 2020-01-16T00:13:22.116 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.116 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.116 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:22.116 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.295894+0000", 2020-01-16T00:13:22.116 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00029238000000000002 2020-01-16T00:13:22.116 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.117 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.117 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:22.117 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.296186+0000", 2020-01-16T00:13:22.117 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0034626409999999998 2020-01-16T00:13:22.117 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.117 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.117 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:22.117 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299649+0000", 2020-01-16T00:13:22.118 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2300000000000002e-06 2020-01-16T00:13:22.118 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.118 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.118 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:22.118 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299656+0000", 2020-01-16T00:13:22.118 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1561000000000001e-05 2020-01-16T00:13:22.118 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.119 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.119 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:22.119 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299738+0000", 2020-01-16T00:13:22.119 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8491000000000005e-05 2020-01-16T00:13:22.119 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.119 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.119 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:22.119 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299806+0000", 2020-01-16T00:13:22.120 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5541000000000002e-05 2020-01-16T00:13:22.120 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.120 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.120 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:22.120 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299862+0000", 2020-01-16T00:13:22.120 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013314061 2020-01-16T00:13:22.120 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:22.120 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:22.121 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:22.121 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.121 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.121 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2506 3.a 3:5069665e:::smithi12313934-2:head [write 2875032~15805] snapc 0=[] ondisk+write+known_if_redirected e37)", 2020-01-16T00:13:22.121 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:11.289486+0000", 2020-01-16T00:13:22.121 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 10.684856484000001, 2020-01-16T00:13:22.121 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01052632, 2020-01-16T00:13:22.122 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:22.122 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:22.122 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:22.122 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:22.122 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:22.122 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2506 2020-01-16T00:13:22.122 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.122 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:22.123 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.123 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:22.123 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289486+0000", 2020-01-16T00:13:22.123 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:22.123 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.123 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.123 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:22.123 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289486+0000", 2020-01-16T00:13:22.124 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:22.124 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.124 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.124 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:22.124 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289484+0000", 2020-01-16T00:13:22.124 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2185999999999998e-05 2020-01-16T00:13:22.124 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.124 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.125 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:22.125 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289536+0000", 2020-01-16T00:13:22.125 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.469e-05 2020-01-16T00:13:22.125 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.125 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.125 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:22.125 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289551+0000", 2020-01-16T00:13:22.125 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2469999999999999e-06 2020-01-16T00:13:22.126 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.126 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.126 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:22.126 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289556+0000", 2020-01-16T00:13:22.126 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6541e-05 2020-01-16T00:13:22.126 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.126 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.127 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:22.127 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289613+0000", 2020-01-16T00:13:22.127 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2774000000000006e-05 2020-01-16T00:13:22.127 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.127 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.127 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:22.127 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289676+0000", 2020-01-16T00:13:22.127 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0081875709999999994 2020-01-16T00:13:22.128 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.128 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.128 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:22.128 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.297863+0000", 2020-01-16T00:13:22.128 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012167699999999999 2020-01-16T00:13:22.128 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.128 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.128 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:22.129 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.297985+0000", 2020-01-16T00:13:22.129 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000118027 2020-01-16T00:13:22.129 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.129 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.129 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:22.129 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.298103+0000", 2020-01-16T00:13:22.129 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001559717 2020-01-16T00:13:22.129 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.130 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.130 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:22.130 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299663+0000", 2020-01-16T00:13:22.130 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00025753800000000003 2020-01-16T00:13:22.130 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.130 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.130 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:22.130 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299920+0000", 2020-01-16T00:13:22.131 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2994000000000003e-05 2020-01-16T00:13:22.131 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.131 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.131 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:22.131 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299953+0000", 2020-01-16T00:13:22.131 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9802e-05 2020-01-16T00:13:22.131 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.131 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.132 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:22.132 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299983+0000", 2020-01-16T00:13:22.132 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9578999999999999e-05 2020-01-16T00:13:22.132 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.132 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.132 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:22.132 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.300012+0000", 2020-01-16T00:13:22.133 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01052632 2020-01-16T00:13:22.133 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:22.133 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:22.133 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:22.133 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.133 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.133 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2507 3.a 3:5069665e:::smithi12313934-2:head [setxattr _header (55) in=7b,truncate 2890837] snapc 0=[] ondisk+write+known_if_redirected e37)", 2020-01-16T00:13:22.133 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:11.289625+0000", 2020-01-16T00:13:22.134 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 10.684717157, 2020-01-16T00:13:22.134 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010520467, 2020-01-16T00:13:22.134 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:22.134 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:22.134 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:22.134 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:22.135 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:22.135 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2507 2020-01-16T00:13:22.135 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.135 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:22.135 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.135 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:22.135 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289625+0000", 2020-01-16T00:13:22.135 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:22.136 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.136 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.136 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:22.136 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289625+0000", 2020-01-16T00:13:22.136 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:22.136 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.136 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.137 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:22.137 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289625+0000", 2020-01-16T00:13:22.137 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7239999999999997e-06 2020-01-16T00:13:22.137 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.137 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.137 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:22.137 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289630+0000", 2020-01-16T00:13:22.137 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4060000000000002e-06 2020-01-16T00:13:22.138 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.138 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.138 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:22.138 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289637+0000", 2020-01-16T00:13:22.138 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2569999999999999e-06 2020-01-16T00:13:22.138 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.138 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.138 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:22.139 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289640+0000", 2020-01-16T00:13:22.139 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00060159099999999995 2020-01-16T00:13:22.139 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.139 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.139 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:22.139 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.290242+0000", 2020-01-16T00:13:22.139 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016406 2020-01-16T00:13:22.139 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.140 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.140 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:22.140 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.290406+0000", 2020-01-16T00:13:22.140 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0082527190000000004 2020-01-16T00:13:22.140 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.140 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.140 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:22.140 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.298658+0000", 2020-01-16T00:13:22.141 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013482300000000001 2020-01-16T00:13:22.141 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.141 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.141 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:22.141 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.298793+0000", 2020-01-16T00:13:22.141 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00023945699999999999 2020-01-16T00:13:22.141 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.142 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.142 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:22.142 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299033+0000", 2020-01-16T00:13:22.142 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00089435600000000001 2020-01-16T00:13:22.142 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.142 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.142 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:22.142 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299927+0000", 2020-01-16T00:13:22.143 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012217599999999999 2020-01-16T00:13:22.143 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.143 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.143 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:22.143 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.300049+0000", 2020-01-16T00:13:22.143 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.5884000000000002e-05 2020-01-16T00:13:22.143 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.143 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.144 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:22.144 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.300075+0000", 2020-01-16T00:13:22.144 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.8819e-05 2020-01-16T00:13:22.144 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.144 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.144 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:22.144 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.300104+0000", 2020-01-16T00:13:22.144 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2049999999999999e-05 2020-01-16T00:13:22.145 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.145 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.145 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:22.145 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.300146+0000", 2020-01-16T00:13:22.145 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010520467 2020-01-16T00:13:22.145 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:22.145 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:22.146 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:22.146 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.146 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.146 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2508 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e37)", 2020-01-16T00:13:22.146 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:11.289655+0000", 2020-01-16T00:13:22.146 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 10.684687948000001, 2020-01-16T00:13:22.147 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010659678000000001, 2020-01-16T00:13:22.147 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:22.147 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:22.147 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:22.147 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:22.147 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:22.148 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2508 2020-01-16T00:13:22.148 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.148 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:22.148 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.148 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:22.149 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289655+0000", 2020-01-16T00:13:22.149 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:22.149 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.149 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.149 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:22.149 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289655+0000", 2020-01-16T00:13:22.150 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:22.150 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.150 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.150 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:22.150 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289654+0000", 2020-01-16T00:13:22.150 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1619999999999999e-06 2020-01-16T00:13:22.150 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.151 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.151 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:22.151 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289659+0000", 2020-01-16T00:13:22.151 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6009999999999997e-06 2020-01-16T00:13:22.151 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.151 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.151 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:22.151 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289664+0000", 2020-01-16T00:13:22.152 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.5409999999999999e-06 2020-01-16T00:13:22.152 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.152 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.152 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:22.152 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289667+0000", 2020-01-16T00:13:22.152 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00088239600000000003 2020-01-16T00:13:22.152 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.153 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.153 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:22.153 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.290549+0000", 2020-01-16T00:13:22.153 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013208 2020-01-16T00:13:22.153 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.153 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.153 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:22.153 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.290681+0000", 2020-01-16T00:13:22.154 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0086573699999999993 2020-01-16T00:13:22.154 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.154 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.154 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:22.154 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299338+0000", 2020-01-16T00:13:22.154 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8002000000000001e-05 2020-01-16T00:13:22.154 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.155 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.155 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:22.155 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299436+0000", 2020-01-16T00:13:22.155 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00073602499999999996 2020-01-16T00:13:22.155 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.155 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.155 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:22.155 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.300172+0000", 2020-01-16T00:13:22.156 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0544000000000002e-05 2020-01-16T00:13:22.156 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.156 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.156 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:22.156 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.300233+0000", 2020-01-16T00:13:22.156 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1453999999999996e-05 2020-01-16T00:13:22.156 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.156 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.157 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:22.157 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.300314+0000", 2020-01-16T00:13:22.157 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010659678000000001 2020-01-16T00:13:22.157 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:22.157 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:22.157 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:22.157 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.157 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.158 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2789 3.a 3:5069665e:::smithi12313934-2:head [read 0~2890837,checksum] snapc 0=[] ondisk+read+known_if_redirected e41)", 2020-01-16T00:13:22.158 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:17.213121+0000", 2020-01-16T00:13:22.158 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 4.7612221300000002, 2020-01-16T00:13:22.158 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011386347, 2020-01-16T00:13:22.158 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:22.158 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:22.158 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:22.159 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:22.159 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:22.159 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2789 2020-01-16T00:13:22.159 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.159 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:22.159 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.159 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:22.159 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213121+0000", 2020-01-16T00:13:22.160 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:22.160 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.160 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.160 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:22.160 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213121+0000", 2020-01-16T00:13:22.160 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:22.160 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.160 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.161 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:22.161 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213119+0000", 2020-01-16T00:13:22.161 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3570000000000003e-06 2020-01-16T00:13:22.161 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.161 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.161 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:22.161 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213126+0000", 2020-01-16T00:13:22.162 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2060000000000001e-06 2020-01-16T00:13:22.162 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.162 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.162 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:22.162 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213133+0000", 2020-01-16T00:13:22.162 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7700000000000004e-06 2020-01-16T00:13:22.162 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.162 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.163 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:22.163 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213140+0000", 2020-01-16T00:13:22.163 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0089949739999999993 2020-01-16T00:13:22.163 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.163 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.163 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:22.163 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.222135+0000", 2020-01-16T00:13:22.163 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000191697 2020-01-16T00:13:22.164 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.164 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.164 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:22.164 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.222326+0000", 2020-01-16T00:13:22.164 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002180436 2020-01-16T00:13:22.164 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.164 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.165 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:22.165 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.224507+0000", 2020-01-16T00:13:22.165 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011386347 2020-01-16T00:13:22.165 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:22.165 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:22.165 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:22.165 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.166 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.166 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2790 3.a 3:5069665e:::smithi12313934-2:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e41)", 2020-01-16T00:13:22.166 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:17.213146+0000", 2020-01-16T00:13:22.166 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 4.7611968930000002, 2020-01-16T00:13:22.166 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012098164999999999, 2020-01-16T00:13:22.166 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:22.166 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:22.167 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:22.167 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:22.167 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:22.167 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2790 2020-01-16T00:13:22.167 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.167 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:22.167 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.167 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:22.168 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213146+0000", 2020-01-16T00:13:22.168 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:22.168 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.168 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.168 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:22.168 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213146+0000", 2020-01-16T00:13:22.168 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:22.169 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.169 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.169 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:22.169 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213145+0000", 2020-01-16T00:13:22.169 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4399999999999998e-06 2020-01-16T00:13:22.169 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.169 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.170 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:22.170 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213149+0000", 2020-01-16T00:13:22.170 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1029999999999996e-06 2020-01-16T00:13:22.170 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.170 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.170 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:22.170 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213156+0000", 2020-01-16T00:13:22.170 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0699999999999997e-06 2020-01-16T00:13:22.171 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.171 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.171 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:22.171 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213161+0000", 2020-01-16T00:13:22.171 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011386617 2020-01-16T00:13:22.171 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.171 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.171 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:22.172 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.224548+0000", 2020-01-16T00:13:22.172 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017114499999999999 2020-01-16T00:13:22.172 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.172 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.172 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:22.172 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.224719+0000", 2020-01-16T00:13:22.172 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00052473000000000005 2020-01-16T00:13:22.173 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.173 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.173 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:22.173 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.225244+0000", 2020-01-16T00:13:22.173 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012098164999999999 2020-01-16T00:13:22.173 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:22.173 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:22.173 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:22.174 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.174 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.174 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2791 3.a 3:54f2c117:::smithi12313934-41:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e41)", 2020-01-16T00:13:22.174 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:17.213167+0000", 2020-01-16T00:13:22.174 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 4.7611758799999997, 2020-01-16T00:13:22.174 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016024037000000001, 2020-01-16T00:13:22.174 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:22.174 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:22.175 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:22.175 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:22.175 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:22.175 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2791 2020-01-16T00:13:22.175 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.175 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:22.175 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.175 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:22.176 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213167+0000", 2020-01-16T00:13:22.176 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:22.176 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.176 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.176 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:22.176 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213167+0000", 2020-01-16T00:13:22.176 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:22.176 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.177 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.177 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:22.177 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213166+0000", 2020-01-16T00:13:22.177 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6030000000000004e-06 2020-01-16T00:13:22.177 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.177 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.177 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:22.177 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213172+0000", 2020-01-16T00:13:22.178 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0929999999999997e-06 2020-01-16T00:13:22.178 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.178 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.178 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:22.178 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213179+0000", 2020-01-16T00:13:22.178 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4299999999999997e-06 2020-01-16T00:13:22.178 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.179 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.179 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:22.179 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213184+0000", 2020-01-16T00:13:22.179 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012106695000000001 2020-01-16T00:13:22.179 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.180 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.180 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:22.181 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.225291+0000", 2020-01-16T00:13:22.181 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020863699999999999 2020-01-16T00:13:22.181 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.181 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.181 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:22.181 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.225499+0000", 2020-01-16T00:13:22.181 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0036913050000000002 2020-01-16T00:13:22.181 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.182 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.182 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:22.183 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.229191+0000", 2020-01-16T00:13:22.183 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016024037000000001 2020-01-16T00:13:22.183 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:22.183 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:22.183 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:22.183 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.183 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.183 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2792 3.a 3:54f2c117:::smithi12313934-41:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e41)", 2020-01-16T00:13:22.184 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:17.213190+0000", 2020-01-16T00:13:22.184 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 4.7611524059999999, 2020-01-16T00:13:22.184 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017165949, 2020-01-16T00:13:22.184 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:22.184 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:22.184 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:22.184 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:22.184 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:22.185 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2792 2020-01-16T00:13:22.185 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.185 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:22.185 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.185 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:22.185 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213190+0000", 2020-01-16T00:13:22.185 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:22.186 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.186 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.186 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:22.186 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213190+0000", 2020-01-16T00:13:22.186 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:22.186 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.186 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.186 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:22.187 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213189+0000", 2020-01-16T00:13:22.187 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2699999999999998e-06 2020-01-16T00:13:22.187 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.187 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.187 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:22.187 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213194+0000", 2020-01-16T00:13:22.187 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8600000000000004e-06 2020-01-16T00:13:22.187 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.188 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.188 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:22.188 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213201+0000", 2020-01-16T00:13:22.188 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8500000000000002e-06 2020-01-16T00:13:22.188 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.189 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.189 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:22.189 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213205+0000", 2020-01-16T00:13:22.189 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016028924999999999 2020-01-16T00:13:22.189 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.189 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.189 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:22.189 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.229234+0000", 2020-01-16T00:13:22.190 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019773999999999999 2020-01-16T00:13:22.190 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.190 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.190 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:22.190 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.229432+0000", 2020-01-16T00:13:22.190 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00092414400000000001 2020-01-16T00:13:22.190 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.190 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.191 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:22.191 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.230356+0000", 2020-01-16T00:13:22.191 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017165949 2020-01-16T00:13:22.191 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:22.191 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:22.191 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:22.191 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.191 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.192 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2793 3.a 3:54f2c117:::smithi12313934-41:head [read 0~3210170,checksum] snapc 0=[] ondisk+read+known_if_redirected e41)", 2020-01-16T00:13:22.192 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:17.213211+0000", 2020-01-16T00:13:22.192 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 4.7611318599999999, 2020-01-16T00:13:22.192 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019069413, 2020-01-16T00:13:22.192 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:22.192 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:22.192 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:22.193 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:22.193 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:22.193 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2793 2020-01-16T00:13:22.193 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.193 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:22.193 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.193 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:22.193 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213211+0000", 2020-01-16T00:13:22.194 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:22.194 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.194 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.194 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:22.194 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213211+0000", 2020-01-16T00:13:22.194 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:22.194 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.194 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.195 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:22.195 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213210+0000", 2020-01-16T00:13:22.195 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.66e-06 2020-01-16T00:13:22.195 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.195 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.195 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:22.195 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213216+0000", 2020-01-16T00:13:22.195 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.9469999999999997e-06 2020-01-16T00:13:22.196 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.196 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.196 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:22.196 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213223+0000", 2020-01-16T00:13:22.196 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9259999999999999e-06 2020-01-16T00:13:22.196 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.196 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.196 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:22.197 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213228+0000", 2020-01-16T00:13:22.197 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017171046999999998 2020-01-16T00:13:22.197 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.197 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.197 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:22.197 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.230399+0000", 2020-01-16T00:13:22.197 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000189932 2020-01-16T00:13:22.198 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.198 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.198 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:22.198 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.230589+0000", 2020-01-16T00:13:22.198 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0016916539999999999 2020-01-16T00:13:22.198 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:22.198 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:22.198 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:22.199 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.232280+0000", 2020-01-16T00:13:22.199 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019069413 2020-01-16T00:13:22.199 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:22.199 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:22.199 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:22.199 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:22.200 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:22.200 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:22.200 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-01-16T00:13:22.202 INFO:tasks.rados.rados.0.smithi123.stdout:1192: expect (ObjNum 342 snap 0 seq_num 342) 2020-01-16T00:13:22.202 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:22.196+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:22.203 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-01-16T00:13:22.247 INFO:tasks.rados.rados.0.smithi123.stdout:1193: finishing write tid 1 to smithi12313934-7 2020-01-16T00:13:22.247 INFO:tasks.rados.rados.0.smithi123.stdout:1193: finishing write tid 2 to smithi12313934-7 2020-01-16T00:13:22.247 INFO:tasks.rados.rados.0.smithi123.stdout:1193: finishing write tid 3 to smithi12313934-7 2020-01-16T00:13:22.247 INFO:tasks.rados.rados.0.smithi123.stdout:1193: finishing write tid 5 to smithi12313934-7 2020-01-16T00:13:22.247 INFO:tasks.rados.rados.0.smithi123.stdout:1193: finishing write tid 6 to smithi12313934-7 2020-01-16T00:13:22.247 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 7 v 179 (ObjNum 352 snap 0 seq_num 352) dirty exists 2020-01-16T00:13:22.248 INFO:tasks.rados.rados.0.smithi123.stdout:1193: left oid 7 (ObjNum 352 snap 0 seq_num 352) 2020-01-16T00:13:22.248 INFO:tasks.rados.rados.0.smithi123.stdout:1194: finishing write tid 1 to smithi12313934-4 2020-01-16T00:13:22.248 INFO:tasks.rados.rados.0.smithi123.stdout:1190: done (15 left) 2020-01-16T00:13:22.248 INFO:tasks.rados.rados.0.smithi123.stdout:1191: done (14 left) 2020-01-16T00:13:22.248 INFO:tasks.rados.rados.0.smithi123.stdout:1192: done (13 left) 2020-01-16T00:13:22.248 INFO:tasks.rados.rados.0.smithi123.stdout:1193: done (12 left) 2020-01-16T00:13:22.248 INFO:tasks.rados.rados.0.smithi123.stdout:1194: finishing write tid 2 to smithi12313934-4 2020-01-16T00:13:22.248 INFO:tasks.rados.rados.0.smithi123.stdout:1194: finishing write tid 3 to smithi12313934-4 2020-01-16T00:13:22.249 INFO:tasks.rados.rados.0.smithi123.stdout:1194: finishing write tid 5 to smithi12313934-4 2020-01-16T00:13:22.249 INFO:tasks.rados.rados.0.smithi123.stdout:1194: finishing write tid 6 to smithi12313934-4 2020-01-16T00:13:22.249 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 4 v 139 (ObjNum 353 snap 0 seq_num 353) dirty exists 2020-01-16T00:13:22.249 INFO:tasks.rados.rados.0.smithi123.stdout:1194: left oid 4 (ObjNum 353 snap 0 seq_num 353) 2020-01-16T00:13:22.249 INFO:tasks.rados.rados.0.smithi123.stdout:1195: finishing write tid 1 to smithi12313934-31 2020-01-16T00:13:22.249 INFO:tasks.rados.rados.0.smithi123.stdout:1207: read oid 37 snap -1 2020-01-16T00:13:22.250 INFO:tasks.rados.rados.0.smithi123.stdout:1207: expect (ObjNum 285 snap 0 seq_num 285) 2020-01-16T00:13:22.250 INFO:tasks.rados.rados.0.smithi123.stdout:1194: done (12 left) 2020-01-16T00:13:22.250 INFO:tasks.rados.rados.0.smithi123.stdout:1208: write oid 16 current snap is 0 2020-01-16T00:13:22.250 INFO:tasks.rados.rados.0.smithi123.stdout:1208: seq_num 361 ranges {467327=633480,1825176=242151} 2020-01-16T00:13:22.258 INFO:tasks.rados.rados.0.smithi123.stdout:1208: writing smithi12313934-16 from 467327 to 1100807 tid 1 2020-01-16T00:13:22.263 INFO:tasks.rados.rados.0.smithi123.stdout:1208: writing smithi12313934-16 from 1825176 to 2067327 tid 2 2020-01-16T00:13:22.264 INFO:tasks.rados.rados.0.smithi123.stdout:1209: read oid 39 snap -1 2020-01-16T00:13:22.264 INFO:tasks.rados.rados.0.smithi123.stdout:1209: expect (ObjNum 351 snap 0 seq_num 351) 2020-01-16T00:13:22.264 INFO:tasks.rados.rados.0.smithi123.stdout:1210: write oid 14 current snap is 0 2020-01-16T00:13:22.264 INFO:tasks.rados.rados.0.smithi123.stdout:1195: finishing write tid 2 to smithi12313934-31 2020-01-16T00:13:22.264 INFO:tasks.rados.rados.0.smithi123.stdout:1210: seq_num 362 ranges {484134=708729,1883658=200476} 2020-01-16T00:13:22.270 INFO:tasks.rados.rados.0.smithi123.stdout:1210: writing smithi12313934-14 from 484134 to 1192863 tid 1 2020-01-16T00:13:22.275 INFO:tasks.rados.rados.0.smithi123.stdout:1210: writing smithi12313934-14 from 1883658 to 2084134 tid 2 2020-01-16T00:13:22.276 INFO:tasks.rados.rados.0.smithi123.stdout:1211: write (excl) oid 30 current snap is 0 2020-01-16T00:13:22.276 INFO:tasks.rados.rados.0.smithi123.stdout:1211: seq_num 363 ranges {500941=783978,1942140=158801} 2020-01-16T00:13:22.283 INFO:tasks.rados.rados.0.smithi123.stdout:1211: writing smithi12313934-30 from 500941 to 1284919 tid 1 2020-01-16T00:13:22.290 INFO:tasks.rados.rados.0.smithi123.stdout:1211: writing smithi12313934-30 from 1942140 to 2100941 tid 2 2020-01-16T00:13:22.290 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:22.291 INFO:tasks.rados.rados.0.smithi123.stdout:1195: finishing write tid 3 to smithi12313934-31 2020-01-16T00:13:22.291 INFO:tasks.rados.rados.0.smithi123.stdout:1195: finishing write tid 5 to smithi12313934-31 2020-01-16T00:13:22.291 INFO:tasks.rados.rados.0.smithi123.stdout:1195: finishing write tid 6 to smithi12313934-31 2020-01-16T00:13:22.291 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 31 v 132 (ObjNum 354 snap 0 seq_num 354) dirty exists 2020-01-16T00:13:22.291 INFO:tasks.rados.rados.0.smithi123.stdout:1195: left oid 31 (ObjNum 354 snap 0 seq_num 354) 2020-01-16T00:13:22.291 INFO:tasks.rados.rados.0.smithi123.stdout:1195: done (15 left) 2020-01-16T00:13:22.292 INFO:tasks.rados.rados.0.smithi123.stdout:1212: read oid 20 snap -1 2020-01-16T00:13:22.292 INFO:tasks.rados.rados.0.smithi123.stdout:1212: expect (ObjNum 322 snap 0 seq_num 322) 2020-01-16T00:13:22.292 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:22.292 INFO:tasks.rados.rados.0.smithi123.stdout:1197: expect (ObjNum 220 snap 0 seq_num 220) 2020-01-16T00:13:22.302 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:22.298+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:22.319 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:22.319 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:22.319 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:13:22.319 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:22.328 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-01-16T00:13:22.330 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:13:22.350 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:13:22.371 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:13:22.422 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:13:22.422 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:13:22.424 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:22.416+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:22.426 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:22.434 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:13:22.435 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-01-16T00:13:22.459 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:13:22.461 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-01-16T00:13:22.462 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-01-16T00:13:22.462 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-01-16T00:13:22.462 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-01-16T00:13:22.462 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-01-16T00:13:22.462 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENXIO: problem getting command descriptions from osd.5 2020-01-16T00:13:22.462 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: problem getting command descriptions from osd.5 2020-01-16T00:13:22.463 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-01-16T00:13:22.463 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-01-16T00:13:22.470 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-01-16T00:13:22.471 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-01-16T00:13:22.521 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:22.518+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:22.560 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:22.570 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-01-16T00:13:22.630 INFO:tasks.rados.rados.0.smithi123.stdout:1198: finishing write tid 1 to smithi12313934-48 2020-01-16T00:13:22.631 INFO:tasks.rados.rados.0.smithi123.stdout:1198: finishing write tid 2 to smithi12313934-48 2020-01-16T00:13:22.631 INFO:tasks.rados.rados.0.smithi123.stdout:1200: finishing write tid 1 to smithi12313934-28 2020-01-16T00:13:22.631 INFO:tasks.rados.rados.0.smithi123.stdout:1198: finishing write tid 3 to smithi12313934-48 2020-01-16T00:13:22.631 INFO:tasks.rados.rados.0.smithi123.stdout:1198: finishing write tid 5 to smithi12313934-48 2020-01-16T00:13:22.631 INFO:tasks.rados.rados.0.smithi123.stdout:1197: done (15 left) 2020-01-16T00:13:22.631 INFO:tasks.rados.rados.0.smithi123.stdout:1199: done (14 left) 2020-01-16T00:13:22.631 INFO:tasks.rados.rados.0.smithi123.stdout:1213: write (excl) oid 29 current snap is 0 2020-01-16T00:13:22.632 INFO:tasks.rados.rados.0.smithi123.stdout:1213: seq_num 364 ranges {517748=459227,1600622=406110} 2020-01-16T00:13:22.632 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:22.626+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:22.636 INFO:tasks.rados.rados.0.smithi123.stdout:1213: writing smithi12313934-29 from 517748 to 976975 tid 1 2020-01-16T00:13:22.645 INFO:tasks.rados.rados.0.smithi123.stdout:1213: writing smithi12313934-29 from 1600622 to 2006732 tid 2 2020-01-16T00:13:22.645 INFO:tasks.rados.rados.0.smithi123.stdout:1214: read oid 50 snap -1 2020-01-16T00:13:22.646 INFO:tasks.rados.rados.0.smithi123.stdout:1214: expect (ObjNum 319 snap 0 seq_num 319) 2020-01-16T00:13:22.646 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:22.646 INFO:tasks.rados.rados.0.smithi123.stdout:1198: finishing write tid 6 to smithi12313934-48 2020-01-16T00:13:22.646 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 48 v 137 (ObjNum 356 snap 0 seq_num 356) dirty exists 2020-01-16T00:13:22.646 INFO:tasks.rados.rados.0.smithi123.stdout:1198: left oid 48 (ObjNum 356 snap 0 seq_num 356) 2020-01-16T00:13:22.646 INFO:tasks.rados.rados.0.smithi123.stdout:1198: done (15 left) 2020-01-16T00:13:22.646 INFO:tasks.rados.rados.0.smithi123.stdout:1215: write (excl) oid 36 current snap is 0 2020-01-16T00:13:22.647 INFO:tasks.rados.rados.0.smithi123.stdout:1215: seq_num 365 ranges {534555=650829,1659104=475451} 2020-01-16T00:13:22.653 INFO:tasks.rados.rados.0.smithi123.stdout:1215: writing smithi12313934-36 from 534555 to 1185384 tid 1 2020-01-16T00:13:22.660 INFO:tasks.rados.rados.0.smithi123.stdout:1215: writing smithi12313934-36 from 1659104 to 2134555 tid 2 2020-01-16T00:13:22.660 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:22.660 INFO:tasks.rados.rados.0.smithi123.stdout:1200: finishing write tid 2 to smithi12313934-28 2020-01-16T00:13:22.660 INFO:tasks.rados.rados.0.smithi123.stdout:1200: finishing write tid 4 to smithi12313934-28 2020-01-16T00:13:22.660 INFO:tasks.rados.rados.0.smithi123.stdout:1200: finishing write tid 5 to smithi12313934-28 2020-01-16T00:13:22.660 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 28 v 135 (ObjNum 357 snap 0 seq_num 357) dirty exists 2020-01-16T00:13:22.660 INFO:tasks.rados.rados.0.smithi123.stdout:1200: left oid 28 (ObjNum 357 snap 0 seq_num 357) 2020-01-16T00:13:22.661 INFO:tasks.rados.rados.0.smithi123.stdout:1201: finishing write tid 1 to smithi12313934-25 2020-01-16T00:13:22.661 INFO:tasks.rados.rados.0.smithi123.stdout:1200: done (15 left) 2020-01-16T00:13:22.661 INFO:tasks.rados.rados.0.smithi123.stdout:1216: write (excl) oid 8 current snap is 0 2020-01-16T00:13:22.661 INFO:tasks.rados.rados.0.smithi123.stdout:1216: seq_num 366 ranges {551362=726078,1717586=409779} 2020-01-16T00:13:22.668 INFO:tasks.rados.rados.0.smithi123.stdout:1216: writing smithi12313934-8 from 551362 to 1277440 tid 1 2020-01-16T00:13:22.673 INFO:tasks.rados.rados.0.smithi123.stdout:1216: writing smithi12313934-8 from 1717586 to 2127365 tid 2 2020-01-16T00:13:22.673 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:22.674 INFO:tasks.rados.rados.0.smithi123.stdout:1202: done (15 left) 2020-01-16T00:13:22.674 INFO:tasks.rados.rados.0.smithi123.stdout:1217: write oid 32 current snap is 0 2020-01-16T00:13:22.674 INFO:tasks.rados.rados.0.smithi123.stdout:1217: seq_num 367 ranges {568169=401327,1376068=669790} 2020-01-16T00:13:22.679 INFO:tasks.rados.rados.0.smithi123.stdout:1217: writing smithi12313934-32 from 568169 to 969496 tid 1 2020-01-16T00:13:22.686 INFO:tasks.rados.rados.0.smithi123.stdout:1217: writing smithi12313934-32 from 1376068 to 2045858 tid 2 2020-01-16T00:13:22.687 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:22.687 INFO:tasks.rados.rados.0.smithi123.stdout:1201: finishing write tid 2 to smithi12313934-25 2020-01-16T00:13:22.687 INFO:tasks.rados.rados.0.smithi123.stdout:1201: finishing write tid 4 to smithi12313934-25 2020-01-16T00:13:22.687 INFO:tasks.rados.rados.0.smithi123.stdout:1201: finishing write tid 5 to smithi12313934-25 2020-01-16T00:13:22.687 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 25 v 186 (ObjNum 358 snap 0 seq_num 358) dirty exists 2020-01-16T00:13:22.687 INFO:tasks.rados.rados.0.smithi123.stdout:1201: left oid 25 (ObjNum 358 snap 0 seq_num 358) 2020-01-16T00:13:22.687 INFO:tasks.rados.rados.0.smithi123.stdout:1203: finishing write tid 1 to smithi12313934-15 2020-01-16T00:13:22.688 INFO:tasks.rados.rados.0.smithi123.stdout:1205: expect (ObjNum 260 snap 0 seq_num 260) 2020-01-16T00:13:22.722 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:22.718+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:22.728 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:22.737 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-01-16T00:13:22.790 INFO:tasks.rados.rados.0.smithi123.stdout:1206: finishing write tid 1 to smithi12313934-6 2020-01-16T00:13:22.791 INFO:tasks.rados.rados.0.smithi123.stdout:1203: finishing write tid 2 to smithi12313934-15 2020-01-16T00:13:22.791 INFO:tasks.rados.rados.0.smithi123.stdout:1203: finishing write tid 4 to smithi12313934-15 2020-01-16T00:13:22.791 INFO:tasks.rados.rados.0.smithi123.stdout:1203: finishing write tid 5 to smithi12313934-15 2020-01-16T00:13:22.793 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 15 v 139 (ObjNum 359 snap 0 seq_num 359) dirty exists 2020-01-16T00:13:22.793 INFO:tasks.rados.rados.0.smithi123.stdout:1203: left oid 15 (ObjNum 359 snap 0 seq_num 359) 2020-01-16T00:13:22.793 INFO:tasks.rados.rados.0.smithi123.stdout:1206: finishing write tid 2 to smithi12313934-6 2020-01-16T00:13:22.793 INFO:tasks.rados.rados.0.smithi123.stdout:1206: finishing write tid 4 to smithi12313934-6 2020-01-16T00:13:22.793 INFO:tasks.rados.rados.0.smithi123.stdout:1206: finishing write tid 5 to smithi12313934-6 2020-01-16T00:13:22.793 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 6 v 107 (ObjNum 360 snap 0 seq_num 360) dirty exists 2020-01-16T00:13:22.793 INFO:tasks.rados.rados.0.smithi123.stdout:1206: left oid 6 (ObjNum 360 snap 0 seq_num 360) 2020-01-16T00:13:22.794 INFO:tasks.rados.rados.0.smithi123.stdout:1207: expect (ObjNum 285 snap 0 seq_num 285) 2020-01-16T00:13:22.823 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:22.820+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:22.892 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:22.895 INFO:tasks.rados.rados.0.smithi123.stdout:1208: finishing write tid 1 to smithi12313934-16 2020-01-16T00:13:22.896 INFO:tasks.rados.rados.0.smithi123.stdout:1208: finishing write tid 2 to smithi12313934-16 2020-01-16T00:13:22.896 INFO:tasks.rados.rados.0.smithi123.stdout:1208: finishing write tid 4 to smithi12313934-16 2020-01-16T00:13:22.896 INFO:tasks.rados.rados.0.smithi123.stdout:1208: finishing write tid 5 to smithi12313934-16 2020-01-16T00:13:22.896 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 16 v 189 (ObjNum 361 snap 0 seq_num 361) dirty exists 2020-01-16T00:13:22.896 INFO:tasks.rados.rados.0.smithi123.stdout:1208: left oid 16 (ObjNum 361 snap 0 seq_num 361) 2020-01-16T00:13:22.896 INFO:tasks.rados.rados.0.smithi123.stdout:1209: expect (ObjNum 351 snap 0 seq_num 351) 2020-01-16T00:13:22.901 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-01-16T00:13:22.924 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:22.920+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:23.025 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:23.022+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:23.050 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:23.060 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-01-16T00:13:23.127 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:23.124+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:23.149 INFO:tasks.rados.rados.0.smithi123.stdout:1210: finishing write tid 1 to smithi12313934-14 2020-01-16T00:13:23.149 INFO:tasks.rados.rados.0.smithi123.stdout:1210: finishing write tid 2 to smithi12313934-14 2020-01-16T00:13:23.150 INFO:tasks.rados.rados.0.smithi123.stdout:1210: finishing write tid 4 to smithi12313934-14 2020-01-16T00:13:23.150 INFO:tasks.rados.rados.0.smithi123.stdout:1210: finishing write tid 5 to smithi12313934-14 2020-01-16T00:13:23.150 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 14 v 181 (ObjNum 362 snap 0 seq_num 362) dirty exists 2020-01-16T00:13:23.150 INFO:tasks.rados.rados.0.smithi123.stdout:1210: left oid 14 (ObjNum 362 snap 0 seq_num 362) 2020-01-16T00:13:23.150 INFO:tasks.rados.rados.0.smithi123.stdout:1211: finishing write tid 1 to smithi12313934-30 2020-01-16T00:13:23.150 INFO:tasks.rados.rados.0.smithi123.stdout:1211: finishing write tid 2 to smithi12313934-30 2020-01-16T00:13:23.150 INFO:tasks.rados.rados.0.smithi123.stdout:1212: expect (ObjNum 322 snap 0 seq_num 322) 2020-01-16T00:13:23.206 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:23.215 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-01-16T00:13:23.262 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:23.258+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:23.317 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:23.323 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-01-16T00:13:23.328 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:23.324+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:23.363 INFO:tasks.rados.rados.0.smithi123.stdout:1211: finishing write tid 4 to smithi12313934-30 2020-01-16T00:13:23.364 INFO:tasks.rados.rados.0.smithi123.stdout:1211: finishing write tid 5 to smithi12313934-30 2020-01-16T00:13:23.364 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 30 v 120 (ObjNum 363 snap 0 seq_num 363) dirty exists 2020-01-16T00:13:23.364 INFO:tasks.rados.rados.0.smithi123.stdout:1211: left oid 30 (ObjNum 363 snap 0 seq_num 363) 2020-01-16T00:13:23.364 INFO:tasks.rados.rados.0.smithi123.stdout:1213: finishing write tid 1 to smithi12313934-29 2020-01-16T00:13:23.364 INFO:tasks.rados.rados.0.smithi123.stdout:1213: finishing write tid 2 to smithi12313934-29 2020-01-16T00:13:23.364 INFO:tasks.rados.rados.0.smithi123.stdout:1214: expect (ObjNum 319 snap 0 seq_num 319) 2020-01-16T00:13:23.430 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:23.424+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:23.463 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:23.473 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-01-16T00:13:23.531 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:23.528+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:23.559 INFO:tasks.rados.rados.0.smithi123.stdout:1215: finishing write tid 1 to smithi12313934-36 2020-01-16T00:13:23.559 INFO:tasks.rados.rados.0.smithi123.stdout:1213: finishing write tid 4 to smithi12313934-29 2020-01-16T00:13:23.559 INFO:tasks.rados.rados.0.smithi123.stdout:1213: finishing write tid 5 to smithi12313934-29 2020-01-16T00:13:23.559 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 29 v 124 (ObjNum 364 snap 0 seq_num 364) dirty exists 2020-01-16T00:13:23.560 INFO:tasks.rados.rados.0.smithi123.stdout:1213: left oid 29 (ObjNum 364 snap 0 seq_num 364) 2020-01-16T00:13:23.560 INFO:tasks.rados.rados.0.smithi123.stdout:1215: finishing write tid 2 to smithi12313934-36 2020-01-16T00:13:23.560 INFO:tasks.rados.rados.0.smithi123.stdout:1215: finishing write tid 4 to smithi12313934-36 2020-01-16T00:13:23.560 INFO:tasks.rados.rados.0.smithi123.stdout:1215: finishing write tid 5 to smithi12313934-36 2020-01-16T00:13:23.560 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 36 v 78 (ObjNum 365 snap 0 seq_num 365) dirty exists 2020-01-16T00:13:23.560 INFO:tasks.rados.rados.0.smithi123.stdout:1215: left oid 36 (ObjNum 365 snap 0 seq_num 365) 2020-01-16T00:13:23.560 INFO:tasks.rados.rados.0.smithi123.stdout:1216: finishing write tid 1 to smithi12313934-8 2020-01-16T00:13:23.560 INFO:tasks.rados.rados.0.smithi123.stdout:1216: finishing write tid 2 to smithi12313934-8 2020-01-16T00:13:23.561 INFO:tasks.rados.rados.0.smithi123.stdout:1216: finishing write tid 4 to smithi12313934-8 2020-01-16T00:13:23.561 INFO:tasks.rados.rados.0.smithi123.stdout:1216: finishing write tid 5 to smithi12313934-8 2020-01-16T00:13:23.561 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 8 v 79 (ObjNum 366 snap 0 seq_num 366) dirty exists 2020-01-16T00:13:23.561 INFO:tasks.rados.rados.0.smithi123.stdout:1216: left oid 8 (ObjNum 366 snap 0 seq_num 366) 2020-01-16T00:13:23.561 INFO:tasks.rados.rados.0.smithi123.stdout:1217: finishing write tid 1 to smithi12313934-32 2020-01-16T00:13:23.561 INFO:tasks.rados.rados.0.smithi123.stdout:1217: finishing write tid 2 to smithi12313934-32 2020-01-16T00:13:23.565 INFO:tasks.rados.rados.0.smithi123.stdout:1201: done (15 left) 2020-01-16T00:13:23.576 INFO:tasks.rados.rados.0.smithi123.stdout:1203: done (14 left) 2020-01-16T00:13:23.576 INFO:tasks.rados.rados.0.smithi123.stdout:1204: done (13 left) 2020-01-16T00:13:23.576 INFO:tasks.rados.rados.0.smithi123.stdout:1205: done (12 left) 2020-01-16T00:13:23.576 INFO:tasks.rados.rados.0.smithi123.stdout:1206: done (11 left) 2020-01-16T00:13:23.576 INFO:tasks.rados.rados.0.smithi123.stdout:1207: done (10 left) 2020-01-16T00:13:23.577 INFO:tasks.rados.rados.0.smithi123.stdout:1208: done (9 left) 2020-01-16T00:13:23.577 INFO:tasks.rados.rados.0.smithi123.stdout:1209: done (8 left) 2020-01-16T00:13:23.577 INFO:tasks.rados.rados.0.smithi123.stdout:1210: done (7 left) 2020-01-16T00:13:23.577 INFO:tasks.rados.rados.0.smithi123.stdout:1211: done (6 left) 2020-01-16T00:13:23.577 INFO:tasks.rados.rados.0.smithi123.stdout:1212: done (5 left) 2020-01-16T00:13:23.577 INFO:tasks.rados.rados.0.smithi123.stdout:1213: done (4 left) 2020-01-16T00:13:23.578 INFO:tasks.rados.rados.0.smithi123.stdout:1214: done (3 left) 2020-01-16T00:13:23.578 INFO:tasks.rados.rados.0.smithi123.stdout:1215: done (2 left) 2020-01-16T00:13:23.578 INFO:tasks.rados.rados.0.smithi123.stdout:1216: done (1 left) 2020-01-16T00:13:23.578 INFO:tasks.rados.rados.0.smithi123.stdout:1218: read oid 38 snap -1 2020-01-16T00:13:23.578 INFO:tasks.rados.rados.0.smithi123.stdout:1218: expect (ObjNum 298 snap 0 seq_num 298) 2020-01-16T00:13:23.578 INFO:tasks.rados.rados.0.smithi123.stdout:1219: read oid 3 snap -1 2020-01-16T00:13:23.578 INFO:tasks.rados.rados.0.smithi123.stdout:1219: expect (ObjNum 260 snap 0 seq_num 260) 2020-01-16T00:13:23.578 INFO:tasks.rados.rados.0.smithi123.stdout:1220: write oid 22 current snap is 0 2020-01-16T00:13:23.579 INFO:tasks.rados.rados.0.smithi123.stdout:1220: seq_num 368 ranges {584976=476576,1834550=350426} 2020-01-16T00:13:23.580 INFO:tasks.rados.rados.0.smithi123.stdout:1220: writing smithi12313934-22 from 584976 to 1061552 tid 1 2020-01-16T00:13:23.580 INFO:tasks.rados.rados.0.smithi123.stdout:1220: writing smithi12313934-22 from 1834550 to 2184976 tid 2 2020-01-16T00:13:23.580 INFO:tasks.rados.rados.0.smithi123.stdout:1221: write (excl) oid 31 current snap is 0 2020-01-16T00:13:23.580 INFO:tasks.rados.rados.0.smithi123.stdout:1221: seq_num 369 ranges {601783=551825,1776679=425104} 2020-01-16T00:13:23.580 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:23.585 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-01-16T00:13:23.586 INFO:tasks.rados.rados.0.smithi123.stdout:1221: writing smithi12313934-31 from 601783 to 1153608 tid 1 2020-01-16T00:13:23.592 INFO:tasks.rados.rados.0.smithi123.stdout:1221: writing smithi12313934-31 from 1776679 to 2201783 tid 2 2020-01-16T00:13:23.592 INFO:tasks.rados.rados.0.smithi123.stdout:1222: read oid 4 snap -1 2020-01-16T00:13:23.592 INFO:tasks.rados.rados.0.smithi123.stdout:1222: expect (ObjNum 353 snap 0 seq_num 353) 2020-01-16T00:13:23.593 INFO:tasks.rados.rados.0.smithi123.stdout:1223: write oid 30 current snap is 0 2020-01-16T00:13:23.593 INFO:tasks.rados.rados.0.smithi123.stdout:1223: seq_num 370 ranges {618590=627074,1835161=383429} 2020-01-16T00:13:23.600 INFO:tasks.rados.rados.0.smithi123.stdout:1223: writing smithi12313934-30 from 618590 to 1245664 tid 1 2020-01-16T00:13:23.609 INFO:tasks.rados.rados.0.smithi123.stdout:1223: writing smithi12313934-30 from 1835161 to 2218590 tid 2 2020-01-16T00:13:23.609 INFO:tasks.rados.rados.0.smithi123.stdout:1224: read oid 42 snap -1 2020-01-16T00:13:23.609 INFO:tasks.rados.rados.0.smithi123.stdout:1224: expect (ObjNum 334 snap 0 seq_num 334) 2020-01-16T00:13:23.609 INFO:tasks.rados.rados.0.smithi123.stdout:1225: read oid 26 snap -1 2020-01-16T00:13:23.609 INFO:tasks.rados.rados.0.smithi123.stdout:1225: expect (ObjNum 263 snap 0 seq_num 263) 2020-01-16T00:13:23.609 INFO:tasks.rados.rados.0.smithi123.stdout:1217: finishing write tid 4 to smithi12313934-32 2020-01-16T00:13:23.610 INFO:tasks.rados.rados.0.smithi123.stdout:1217: finishing write tid 5 to smithi12313934-32 2020-01-16T00:13:23.610 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 32 v 121 (ObjNum 367 snap 0 seq_num 367) dirty exists 2020-01-16T00:13:23.610 INFO:tasks.rados.rados.0.smithi123.stdout:1217: left oid 32 (ObjNum 367 snap 0 seq_num 367) 2020-01-16T00:13:23.610 INFO:tasks.rados.rados.0.smithi123.stdout:1226: read oid 45 snap -1 2020-01-16T00:13:23.610 INFO:tasks.rados.rados.0.smithi123.stdout:1226: expect (ObjNum 335 snap 0 seq_num 335) 2020-01-16T00:13:23.610 INFO:tasks.rados.rados.0.smithi123.stdout:1217: done (9 left) 2020-01-16T00:13:23.610 INFO:tasks.rados.rados.0.smithi123.stdout:1227: write oid 49 current snap is 0 2020-01-16T00:13:23.611 INFO:tasks.rados.rados.0.smithi123.stdout:1227: seq_num 371 ranges {635397=702323,1893643=341754} 2020-01-16T00:13:23.615 INFO:tasks.rados.rados.0.smithi123.stdout:1227: writing smithi12313934-49 from 635397 to 1337720 tid 1 2020-01-16T00:13:23.623 INFO:tasks.rados.rados.0.smithi123.stdout:1227: writing smithi12313934-49 from 1893643 to 2235397 tid 2 2020-01-16T00:13:23.623 INFO:tasks.rados.rados.0.smithi123.stdout:1228: read oid 44 snap -1 2020-01-16T00:13:23.623 INFO:tasks.rados.rados.0.smithi123.stdout:1228: expect (ObjNum 316 snap 0 seq_num 316) 2020-01-16T00:13:23.623 INFO:tasks.rados.rados.0.smithi123.stdout:1229: read oid 13 snap -1 2020-01-16T00:13:23.623 INFO:tasks.rados.rados.0.smithi123.stdout:1229: expect (ObjNum 339 snap 0 seq_num 339) 2020-01-16T00:13:23.624 INFO:tasks.rados.rados.0.smithi123.stdout:1230: write (excl) oid 9 current snap is 0 2020-01-16T00:13:23.624 INFO:tasks.rados.rados.0.smithi123.stdout:1230: seq_num 372 ranges {652204=777572,1952125=300079} 2020-01-16T00:13:23.631 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:23.626+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:23.632 INFO:tasks.rados.rados.0.smithi123.stdout:1230: writing smithi12313934-9 from 652204 to 1429776 tid 1 2020-01-16T00:13:23.638 INFO:tasks.rados.rados.0.smithi123.stdout:1230: writing smithi12313934-9 from 1952125 to 2252204 tid 2 2020-01-16T00:13:23.638 INFO:tasks.rados.rados.0.smithi123.stdout:1231: read oid 36 snap -1 2020-01-16T00:13:23.638 INFO:tasks.rados.rados.0.smithi123.stdout:1231: expect (ObjNum 365 snap 0 seq_num 365) 2020-01-16T00:13:23.638 INFO:tasks.rados.rados.0.smithi123.stdout:1232: write oid 21 current snap is 0 2020-01-16T00:13:23.638 INFO:tasks.rados.rados.0.smithi123.stdout:1232: seq_num 373 ranges {669011=569174,2010607=258404} 2020-01-16T00:13:23.645 INFO:tasks.rados.rados.0.smithi123.stdout:1232: writing smithi12313934-21 from 669011 to 1238185 tid 1 2020-01-16T00:13:23.652 INFO:tasks.rados.rados.0.smithi123.stdout:1232: writing smithi12313934-21 from 2010607 to 2269011 tid 2 2020-01-16T00:13:23.652 INFO:tasks.rados.rados.0.smithi123.stdout:1233: write (excl) oid 15 current snap is 0 2020-01-16T00:13:23.652 INFO:tasks.rados.rados.0.smithi123.stdout:1233: seq_num 374 ranges {685818=644423,2069089=216729} 2020-01-16T00:13:23.661 INFO:tasks.rados.rados.0.smithi123.stdout:1233: writing smithi12313934-15 from 685818 to 1330241 tid 1 2020-01-16T00:13:23.667 INFO:tasks.rados.rados.0.smithi123.stdout:1233: writing smithi12313934-15 from 2069089 to 2285818 tid 2 2020-01-16T00:13:23.667 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:23.667 INFO:tasks.rados.rados.0.smithi123.stdout:1219: expect (ObjNum 260 snap 0 seq_num 260) 2020-01-16T00:13:23.703 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:23.716 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-01-16T00:13:23.760 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:23.758+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:23.766 INFO:tasks.rados.rados.0.smithi123.stdout:1218: expect (ObjNum 298 snap 0 seq_num 298) 2020-01-16T00:13:23.833 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:23.828+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:23.862 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:23.870 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-01-16T00:13:23.887 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2020-01-16T00:13:23.919 INFO:tasks.rados.rados.0.smithi123.stdout:1220: finishing write tid 1 to smithi12313934-22 2020-01-16T00:13:23.920 INFO:tasks.rados.rados.0.smithi123.stdout:1220: finishing write tid 2 to smithi12313934-22 2020-01-16T00:13:23.920 INFO:tasks.rados.rados.0.smithi123.stdout:1220: finishing write tid 4 to smithi12313934-22 2020-01-16T00:13:23.920 INFO:tasks.rados.rados.0.smithi123.stdout:1220: finishing write tid 5 to smithi12313934-22 2020-01-16T00:13:23.920 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 22 v 29 (ObjNum 368 snap 0 seq_num 368) dirty exists 2020-01-16T00:13:23.920 INFO:tasks.rados.rados.0.smithi123.stdout:1220: left oid 22 (ObjNum 368 snap 0 seq_num 368) 2020-01-16T00:13:23.920 INFO:tasks.rados.rados.0.smithi123.stdout:1218: done (15 left) 2020-01-16T00:13:23.920 INFO:tasks.rados.rados.0.smithi123.stdout:1219: done (14 left) 2020-01-16T00:13:23.920 INFO:tasks.rados.rados.0.smithi123.stdout:1220: done (13 left) 2020-01-16T00:13:23.921 INFO:tasks.rados.rados.0.smithi123.stdout:1234: write oid 32 current snap is 0 2020-01-16T00:13:23.921 INFO:tasks.rados.rados.0.smithi123.stdout:1234: seq_num 375 ranges {702625=719672,2127571=175054} 2020-01-16T00:13:23.943 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:23.930+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:23.943 INFO:tasks.rados.rados.0.smithi123.stdout:1234: writing smithi12313934-32 from 702625 to 1422297 tid 1 2020-01-16T00:13:23.943 INFO:tasks.rados.rados.0.smithi123.stdout:1234: writing smithi12313934-32 from 2127571 to 2302625 tid 2 2020-01-16T00:13:23.944 INFO:tasks.rados.rados.0.smithi123.stdout:1235: read oid 40 snap -1 2020-01-16T00:13:23.944 INFO:tasks.rados.rados.0.smithi123.stdout:1235: expect (ObjNum 347 snap 0 seq_num 347) 2020-01-16T00:13:23.944 INFO:tasks.rados.rados.0.smithi123.stdout:1236: write (excl) oid 6 current snap is 0 2020-01-16T00:13:23.944 INFO:tasks.rados.rados.0.smithi123.stdout:1236: seq_num 376 ranges {719432=794921,2186053=133379} 2020-01-16T00:13:23.950 INFO:tasks.rados.rados.0.smithi123.stdout:1236: writing smithi12313934-6 from 719432 to 1514353 tid 1 2020-01-16T00:13:23.958 INFO:tasks.rados.rados.0.smithi123.stdout:1236: writing smithi12313934-6 from 2186053 to 2319432 tid 2 2020-01-16T00:13:23.958 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:23.958 INFO:tasks.rados.rados.0.smithi123.stdout:1221: finishing write tid 1 to smithi12313934-31 2020-01-16T00:13:23.959 INFO:tasks.rados.rados.0.smithi123.stdout:1221: finishing write tid 2 to smithi12313934-31 2020-01-16T00:13:23.959 INFO:tasks.rados.rados.0.smithi123.stdout:1221: finishing write tid 4 to smithi12313934-31 2020-01-16T00:13:23.959 INFO:tasks.rados.rados.0.smithi123.stdout:1221: finishing write tid 5 to smithi12313934-31 2020-01-16T00:13:23.959 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 31 v 142 (ObjNum 369 snap 0 seq_num 369) dirty exists 2020-01-16T00:13:23.959 INFO:tasks.rados.rados.0.smithi123.stdout:1221: left oid 31 (ObjNum 369 snap 0 seq_num 369) 2020-01-16T00:13:23.960 INFO:tasks.rados.rados.0.smithi123.stdout:1222: expect (ObjNum 353 snap 0 seq_num 353) 2020-01-16T00:13:23.985 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:23.996 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-01-16T00:13:24.043 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:24.040+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:24.099 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:24.109 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-01-16T00:13:24.129 INFO:tasks.rados.rados.0.smithi123.stdout:1223: finishing write tid 1 to smithi12313934-30 2020-01-16T00:13:24.130 INFO:tasks.rados.rados.0.smithi123.stdout:1225: expect (ObjNum 263 snap 0 seq_num 263) 2020-01-16T00:13:24.136 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:24.134+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:24.237 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:24.234+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:24.259 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:24.269 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-01-16T00:13:24.309 INFO:tasks.rados.rados.0.smithi123.stdout:1224: expect (ObjNum 334 snap 0 seq_num 334) 2020-01-16T00:13:24.337 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:24.334+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:24.371 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:24.380 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-01-16T00:13:24.438 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:24.432+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:24.452 INFO:tasks.rados.rados.0.smithi123.stdout:1223: finishing write tid 2 to smithi12313934-30 2020-01-16T00:13:24.453 INFO:tasks.rados.rados.0.smithi123.stdout:1223: finishing write tid 4 to smithi12313934-30 2020-01-16T00:13:24.453 INFO:tasks.rados.rados.0.smithi123.stdout:1223: finishing write tid 5 to smithi12313934-30 2020-01-16T00:13:24.453 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 30 v 128 (ObjNum 370 snap 0 seq_num 370) dirty exists 2020-01-16T00:13:24.453 INFO:tasks.rados.rados.0.smithi123.stdout:1223: left oid 30 (ObjNum 370 snap 0 seq_num 370) 2020-01-16T00:13:24.453 INFO:tasks.rados.rados.0.smithi123.stdout:1226: expect (ObjNum 335 snap 0 seq_num 335) 2020-01-16T00:13:24.472 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-01-16T00:13:24.488 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:24.497 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-01-16T00:13:24.538 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:24.534+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:24.605 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:13:24.631 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:13:24.639 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:24.634+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:24.646 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:24.656 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-01-16T00:13:24.656 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:13:24.676 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:13:24.678 INFO:tasks.rados.rados.0.smithi123.stdout:1227: finishing write tid 1 to smithi12313934-49 2020-01-16T00:13:24.678 INFO:tasks.rados.rados.0.smithi123.stdout:1227: finishing write tid 2 to smithi12313934-49 2020-01-16T00:13:24.678 INFO:tasks.rados.rados.0.smithi123.stdout:1227: finishing write tid 4 to smithi12313934-49 2020-01-16T00:13:24.679 INFO:tasks.rados.rados.0.smithi123.stdout:1221: done (15 left) 2020-01-16T00:13:24.679 INFO:tasks.rados.rados.0.smithi123.stdout:1222: done (14 left) 2020-01-16T00:13:24.679 INFO:tasks.rados.rados.0.smithi123.stdout:1223: done (13 left) 2020-01-16T00:13:24.679 INFO:tasks.rados.rados.0.smithi123.stdout:1224: done (12 left) 2020-01-16T00:13:24.679 INFO:tasks.rados.rados.0.smithi123.stdout:1225: done (11 left) 2020-01-16T00:13:24.680 INFO:tasks.rados.rados.0.smithi123.stdout:1226: done (10 left) 2020-01-16T00:13:24.680 INFO:tasks.rados.rados.0.smithi123.stdout:1237: write oid 3 current snap is 0 2020-01-16T00:13:24.680 INFO:tasks.rados.rados.0.smithi123.stdout:1237: seq_num 377 ranges {736239=470170,1728182=571782} 2020-01-16T00:13:24.690 INFO:tasks.rados.rados.0.smithi123.stdout:1237: writing smithi12313934-3 from 736239 to 1206409 tid 1 2020-01-16T00:13:24.698 INFO:tasks.rados.rados.0.smithi123.stdout:1237: writing smithi12313934-3 from 1728182 to 2299964 tid 2 2020-01-16T00:13:24.698 INFO:tasks.rados.rados.0.smithi123.stdout:1238: read oid 8 snap -1 2020-01-16T00:13:24.698 INFO:tasks.rados.rados.0.smithi123.stdout:1238: expect (ObjNum 366 snap 0 seq_num 366) 2020-01-16T00:13:24.699 INFO:tasks.rados.rados.0.smithi123.stdout:1239: read oid 31 snap -1 2020-01-16T00:13:24.699 INFO:tasks.rados.rados.0.smithi123.stdout:1239: expect (ObjNum 369 snap 0 seq_num 369) 2020-01-16T00:13:24.699 INFO:tasks.rados.rados.0.smithi123.stdout:1240: read oid 33 snap -1 2020-01-16T00:13:24.699 INFO:tasks.rados.rados.0.smithi123.stdout:1240: expect (ObjNum 330 snap 0 seq_num 330) 2020-01-16T00:13:24.700 INFO:tasks.rados.rados.0.smithi123.stdout:1241: read oid 42 snap -1 2020-01-16T00:13:24.700 INFO:tasks.rados.rados.0.smithi123.stdout:1241: expect (ObjNum 334 snap 0 seq_num 334) 2020-01-16T00:13:24.700 INFO:tasks.rados.rados.0.smithi123.stdout:1242: read oid 14 snap -1 2020-01-16T00:13:24.700 INFO:tasks.rados.rados.0.smithi123.stdout:1242: expect (ObjNum 362 snap 0 seq_num 362) 2020-01-16T00:13:24.700 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:24.701 INFO:tasks.rados.rados.0.smithi123.stdout:1229: expect (ObjNum 339 snap 0 seq_num 339) 2020-01-16T00:13:24.701 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:13:24.729 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:13:24.740 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:24.738+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:24.749 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:13:24.751 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-01-16T00:13:24.751 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-01-16T00:13:24.751 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-01-16T00:13:24.751 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-01-16T00:13:24.751 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-01-16T00:13:24.752 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENXIO: problem getting command descriptions from osd.5 2020-01-16T00:13:24.752 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: problem getting command descriptions from osd.5 2020-01-16T00:13:24.752 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-01-16T00:13:24.752 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-01-16T00:13:24.760 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-01-16T00:13:24.760 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-01-16T00:13:24.761 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:24.762 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:13:24.762 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:13:24.762 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:24.771 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-01-16T00:13:24.841 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:24.838+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:24.874 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:24.874 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:13:24.874 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:13:24.874 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:13:24.874 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:24.883 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-01-16T00:13:24.897 INFO:teuthology.orchestra.run.smithi123.stderr:noscrub is unset 2020-01-16T00:13:24.942 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:24.936+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:24.946 INFO:tasks.rados.rados.0.smithi123.stdout:1227: finishing write tid 5 to smithi12313934-49 2020-01-16T00:13:24.946 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 49 v 125 (ObjNum 371 snap 0 seq_num 371) dirty exists 2020-01-16T00:13:24.946 INFO:tasks.rados.rados.0.smithi123.stdout:1227: left oid 49 (ObjNum 371 snap 0 seq_num 371) 2020-01-16T00:13:24.947 INFO:tasks.rados.rados.0.smithi123.stdout:1230: finishing write tid 1 to smithi12313934-9 2020-01-16T00:13:24.947 INFO:tasks.rados.rados.0.smithi123.stdout:1228: expect (ObjNum 316 snap 0 seq_num 316) 2020-01-16T00:13:24.993 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:24.993 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:13:24.993 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:13:24.994 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:13:24.994 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:24.994 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1241 3.a 3:5c245412:::smithi12313934-19:head [read 0~3815156,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:13:24.994 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.589903+0000", 2020-01-16T00:13:24.994 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 42.397663436000002, 2020-01-16T00:13:24.995 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0096065560000000005, 2020-01-16T00:13:24.995 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:24.995 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:24.995 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:24.995 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:24.996 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:24.996 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1241 2020-01-16T00:13:24.996 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:24.996 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:24.996 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:24.997 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:24.997 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.589903+0000", 2020-01-16T00:13:24.997 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:24.997 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:24.997 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:24.998 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:24.998 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.589903+0000", 2020-01-16T00:13:24.998 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999952 2020-01-16T00:13:24.998 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:24.998 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:24.998 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:24.998 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.589898+0000", 2020-01-16T00:13:24.999 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4389e-05 2020-01-16T00:13:24.999 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:24.999 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:24.999 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:24.999 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.589922+0000", 2020-01-16T00:13:24.999 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4159999999999998e-05 2020-01-16T00:13:24.999 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.000 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.000 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:25.000 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.589957+0000", 2020-01-16T00:13:25.000 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7368999999999999e-05 2020-01-16T00:13:25.000 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.000 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.001 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.001 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.589974+0000", 2020-01-16T00:13:25.001 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000130407 2020-01-16T00:13:25.001 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.001 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.001 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:25.001 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590104+0000", 2020-01-16T00:13:25.002 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00040217699999999998 2020-01-16T00:13:25.002 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.002 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.002 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:25.002 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590506+0000", 2020-01-16T00:13:25.002 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0090030230000000006 2020-01-16T00:13:25.003 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.003 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.003 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:25.003 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599510+0000", 2020-01-16T00:13:25.003 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0096065560000000005 2020-01-16T00:13:25.003 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.004 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:25.004 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.004 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.004 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.004 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1242 3.a 3:5c245412:::smithi12313934-19:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:13:25.004 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.590003+0000", 2020-01-16T00:13:25.004 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 42.397562927000003, 2020-01-16T00:13:25.004 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010247029, 2020-01-16T00:13:25.005 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:25.005 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:25.005 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:25.005 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:25.005 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:25.005 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1242 2020-01-16T00:13:25.005 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.006 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:25.006 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.006 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:25.006 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590003+0000", 2020-01-16T00:13:25.006 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:25.006 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.006 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.007 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:25.007 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590003+0000", 2020-01-16T00:13:25.007 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:25.008 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.008 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.008 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:25.008 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590002+0000", 2020-01-16T00:13:25.008 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0108e-05 2020-01-16T00:13:25.008 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.008 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.009 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:25.009 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590012+0000", 2020-01-16T00:13:25.009 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5318000000000001e-05 2020-01-16T00:13:25.009 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.009 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.009 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:25.009 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590027+0000", 2020-01-16T00:13:25.009 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0720000000000001e-05 2020-01-16T00:13:25.010 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.017 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.017 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.018 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590038+0000", 2020-01-16T00:13:25.028 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0095349949999999992 2020-01-16T00:13:25.028 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.028 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.028 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:25.028 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599573+0000", 2020-01-16T00:13:25.028 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017107600000000001 2020-01-16T00:13:25.028 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.029 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.029 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:25.029 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599744+0000", 2020-01-16T00:13:25.029 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00050659499999999998 2020-01-16T00:13:25.029 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.029 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.029 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:25.030 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.600251+0000", 2020-01-16T00:13:25.030 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010247029 2020-01-16T00:13:25.030 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.030 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:25.030 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.030 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.031 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.031 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1243 3.a 3:5c245412:::smithi12313934-19:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:13:25.031 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.590095+0000", 2020-01-16T00:13:25.031 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 42.397471518000003, 2020-01-16T00:13:25.031 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010591455, 2020-01-16T00:13:25.031 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:25.031 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:25.032 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:25.032 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:25.032 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:25.032 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1243 2020-01-16T00:13:25.032 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.032 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:25.033 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.033 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:25.033 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590095+0000", 2020-01-16T00:13:25.033 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:25.034 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.034 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.034 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:25.034 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590095+0000", 2020-01-16T00:13:25.034 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:25.035 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.035 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.035 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:25.035 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590093+0000", 2020-01-16T00:13:25.035 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5619999999999993e-06 2020-01-16T00:13:25.035 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.036 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.036 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:25.036 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590102+0000", 2020-01-16T00:13:25.036 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.259e-05 2020-01-16T00:13:25.036 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.036 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.036 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:25.037 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590115+0000", 2020-01-16T00:13:25.037 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8730000000000001e-06 2020-01-16T00:13:25.037 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.037 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.037 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.037 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590122+0000", 2020-01-16T00:13:25.038 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010163165 2020-01-16T00:13:25.038 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.038 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.038 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:25.038 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.600285+0000", 2020-01-16T00:13:25.039 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0128000000000002e-05 2020-01-16T00:13:25.039 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.039 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.039 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:25.039 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.600375+0000", 2020-01-16T00:13:25.039 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000311312 2020-01-16T00:13:25.039 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.039 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.040 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:25.040 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.600686+0000", 2020-01-16T00:13:25.040 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010591455 2020-01-16T00:13:25.040 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.040 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:25.040 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.041 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.041 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.041 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1654 3.a 3:5069665e:::smithi12313934-2:head [write 1614880~581041] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:13:25.041 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.714839+0000", 2020-01-16T00:13:25.041 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 32.272727529999997, 2020-01-16T00:13:25.041 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013588093000000001, 2020-01-16T00:13:25.041 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:25.042 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:25.042 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:25.042 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:25.042 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:25.042 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1654 2020-01-16T00:13:25.042 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.042 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:25.043 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.043 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:25.043 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.714839+0000", 2020-01-16T00:13:25.043 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:25.043 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.043 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.043 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:25.044 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.714839+0000", 2020-01-16T00:13:25.044 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:25.044 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.044 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.044 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:25.045 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.714836+0000", 2020-01-16T00:13:25.045 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00054657999999999996 2020-01-16T00:13:25.045 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.045 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.045 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:25.046 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.715382+0000", 2020-01-16T00:13:25.046 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7743000000000001e-05 2020-01-16T00:13:25.046 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.046 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.046 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:25.047 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.715420+0000", 2020-01-16T00:13:25.047 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.221e-05 2020-01-16T00:13:25.047 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.047 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.048 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.048 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.715432+0000", 2020-01-16T00:13:25.049 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2487000000000001e-05 2020-01-16T00:13:25.049 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.049 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.049 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:25.050 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.715485+0000", 2020-01-16T00:13:25.050 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000726284 2020-01-16T00:13:25.050 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.050 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.050 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:25.051 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.716211+0000", 2020-01-16T00:13:25.051 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0067930899999999999 2020-01-16T00:13:25.051 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.051 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.052 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:25.052 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.723004+0000", 2020-01-16T00:13:25.052 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012579400000000001 2020-01-16T00:13:25.052 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.053 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.053 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:25.053 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.723130+0000", 2020-01-16T00:13:25.053 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013465800000000001 2020-01-16T00:13:25.054 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.054 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.054 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:25.054 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.723265+0000", 2020-01-16T00:13:25.054 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023307250000000001 2020-01-16T00:13:25.055 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.055 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.055 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:25.055 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.725595+0000", 2020-01-16T00:13:25.055 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012582139999999999 2020-01-16T00:13:25.056 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.056 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.056 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.056 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.726853+0000", 2020-01-16T00:13:25.056 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014479110000000001 2020-01-16T00:13:25.057 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.057 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.057 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.057 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.728301+0000", 2020-01-16T00:13:25.057 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6866000000000007e-05 2020-01-16T00:13:25.058 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.059 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.059 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:25.059 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.728368+0000", 2020-01-16T00:13:25.059 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8740999999999999e-05 2020-01-16T00:13:25.059 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.060 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.060 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:25.060 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.728427+0000", 2020-01-16T00:13:25.060 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013588093000000001 2020-01-16T00:13:25.060 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.061 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:25.061 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.061 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.061 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.062 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1655 3.a 3:5069665e:::smithi12313934-2:head [write 2785791~71399] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:13:25.062 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.718297+0000", 2020-01-16T00:13:25.062 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 32.269269676999997, 2020-01-16T00:13:25.062 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013817354, 2020-01-16T00:13:25.062 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:25.063 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:25.063 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:25.063 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:25.063 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:25.063 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1655 2020-01-16T00:13:25.064 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.064 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:25.064 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.064 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:25.064 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718297+0000", 2020-01-16T00:13:25.065 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:25.065 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.065 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.065 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:25.066 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718297+0000", 2020-01-16T00:13:25.066 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:25.066 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.066 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.066 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:25.067 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718294+0000", 2020-01-16T00:13:25.067 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000103558 2020-01-16T00:13:25.067 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.067 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.067 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:25.068 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718397+0000", 2020-01-16T00:13:25.068 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9287e-05 2020-01-16T00:13:25.068 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.069 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.069 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:25.069 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718417+0000", 2020-01-16T00:13:25.069 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6680000000000001e-06 2020-01-16T00:13:25.070 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.070 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.070 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.070 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718424+0000", 2020-01-16T00:13:25.070 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2701999999999994e-05 2020-01-16T00:13:25.071 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.071 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.071 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:25.071 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718517+0000", 2020-01-16T00:13:25.071 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3893000000000006e-05 2020-01-16T00:13:25.072 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.072 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.072 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:25.072 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718581+0000", 2020-01-16T00:13:25.072 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0059630079999999997 2020-01-16T00:13:25.073 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.073 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.073 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:25.073 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.724544+0000", 2020-01-16T00:13:25.075 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000309108 2020-01-16T00:13:25.075 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.075 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.075 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:25.075 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.724853+0000", 2020-01-16T00:13:25.076 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.005469794 2020-01-16T00:13:25.076 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.076 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.077 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:25.077 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.730323+0000", 2020-01-16T00:13:25.077 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018506100000000001 2020-01-16T00:13:25.077 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.078 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.078 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:25.078 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.730508+0000", 2020-01-16T00:13:25.078 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016729099999999999 2020-01-16T00:13:25.078 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.079 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.079 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:25.079 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.730675+0000", 2020-01-16T00:13:25.080 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00094193300000000005 2020-01-16T00:13:25.080 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.080 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.080 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:25.080 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731617+0000", 2020-01-16T00:13:25.081 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000214613 2020-01-16T00:13:25.081 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.081 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.081 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.081 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731832+0000", 2020-01-16T00:13:25.082 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018326400000000001 2020-01-16T00:13:25.082 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.082 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.082 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.082 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732015+0000", 2020-01-16T00:13:25.083 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0392e-05 2020-01-16T00:13:25.083 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.083 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.083 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:25.084 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732075+0000", 2020-01-16T00:13:25.084 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8677999999999998e-05 2020-01-16T00:13:25.084 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.084 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.084 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:25.085 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732114+0000", 2020-01-16T00:13:25.085 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013817354 2020-01-16T00:13:25.085 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.085 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:25.085 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.086 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.086 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.086 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1656 3.a 3:5069665e:::smithi12313934-2:head [setxattr _header (55) in=7b,truncate 2857190] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:13:25.086 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.718453+0000", 2020-01-16T00:13:25.086 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 32.269112995999997, 2020-01-16T00:13:25.087 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014083462, 2020-01-16T00:13:25.087 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:25.087 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:25.087 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:25.088 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:25.088 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:25.088 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1656 2020-01-16T00:13:25.088 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.088 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:25.089 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.089 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:25.089 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718453+0000", 2020-01-16T00:13:25.090 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:25.090 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.090 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.090 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:25.090 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718453+0000", 2020-01-16T00:13:25.091 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999828 2020-01-16T00:13:25.091 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.091 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.091 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:25.092 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718436+0000", 2020-01-16T00:13:25.092 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4051999999999999e-05 2020-01-16T00:13:25.092 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.092 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.092 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:25.093 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718460+0000", 2020-01-16T00:13:25.093 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6380000000000004e-06 2020-01-16T00:13:25.093 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.093 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.094 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:25.094 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718468+0000", 2020-01-16T00:13:25.094 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6750000000000001e-06 2020-01-16T00:13:25.094 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.094 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.094 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.095 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718472+0000", 2020-01-16T00:13:25.095 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016218499999999999 2020-01-16T00:13:25.095 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.095 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.095 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:25.095 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718635+0000", 2020-01-16T00:13:25.096 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2765e-05 2020-01-16T00:13:25.096 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.096 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.096 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:25.096 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718697+0000", 2020-01-16T00:13:25.096 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0062239249999999999 2020-01-16T00:13:25.097 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.097 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.097 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:25.097 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.724921+0000", 2020-01-16T00:13:25.097 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1527999999999999e-05 2020-01-16T00:13:25.098 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.098 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.098 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:25.098 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.724973+0000", 2020-01-16T00:13:25.099 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0060569719999999999 2020-01-16T00:13:25.099 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.099 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.100 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:25.100 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731030+0000", 2020-01-16T00:13:25.100 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5859999999999996e-05 2020-01-16T00:13:25.100 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.100 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.100 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:25.100 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731096+0000", 2020-01-16T00:13:25.101 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000112018 2020-01-16T00:13:25.101 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.101 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.101 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:25.101 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731208+0000", 2020-01-16T00:13:25.101 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000945889 2020-01-16T00:13:25.101 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.101 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.102 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.102 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732154+0000", 2020-01-16T00:13:25.102 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0834000000000003e-05 2020-01-16T00:13:25.102 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.102 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.102 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.102 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732184+0000", 2020-01-16T00:13:25.103 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028587600000000001 2020-01-16T00:13:25.103 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.103 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.103 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:25.103 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732470+0000", 2020-01-16T00:13:25.103 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1953999999999999e-05 2020-01-16T00:13:25.104 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.104 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.104 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:25.104 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732502+0000", 2020-01-16T00:13:25.104 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4644999999999998e-05 2020-01-16T00:13:25.104 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.104 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.104 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:25.105 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732537+0000", 2020-01-16T00:13:25.105 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014083462 2020-01-16T00:13:25.105 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.105 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:25.105 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.105 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.105 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.106 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1657 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-01-16T00:13:25.106 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.718495+0000", 2020-01-16T00:13:25.106 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 32.269071330999999, 2020-01-16T00:13:25.106 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014299630000000001, 2020-01-16T00:13:25.106 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:25.106 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:25.106 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:25.106 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:25.107 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:25.107 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1657 2020-01-16T00:13:25.107 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.107 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:25.107 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.107 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:25.107 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718495+0000", 2020-01-16T00:13:25.108 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:25.108 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.108 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.108 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:25.108 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718495+0000", 2020-01-16T00:13:25.108 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:25.108 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.109 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.109 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:25.109 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718494+0000", 2020-01-16T00:13:25.109 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8860000000000003e-06 2020-01-16T00:13:25.110 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.110 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.110 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:25.110 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718499+0000", 2020-01-16T00:13:25.110 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9786000000000002e-05 2020-01-16T00:13:25.111 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.111 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.111 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:25.111 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718539+0000", 2020-01-16T00:13:25.111 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1869999999999999e-05 2020-01-16T00:13:25.111 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.111 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.112 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.112 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718561+0000", 2020-01-16T00:13:25.112 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000172206 2020-01-16T00:13:25.112 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.112 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.112 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:25.112 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718733+0000", 2020-01-16T00:13:25.113 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0662999999999998e-05 2020-01-16T00:13:25.113 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.113 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.113 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:25.113 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718793+0000", 2020-01-16T00:13:25.113 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.006217958 2020-01-16T00:13:25.113 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.113 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.114 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:25.114 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.725011+0000", 2020-01-16T00:13:25.114 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1625999999999996e-05 2020-01-16T00:13:25.114 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.114 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.114 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:25.114 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.725073+0000", 2020-01-16T00:13:25.115 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0063923119999999998 2020-01-16T00:13:25.115 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.115 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.115 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:25.115 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731465+0000", 2020-01-16T00:13:25.115 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2534000000000005e-05 2020-01-16T00:13:25.115 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.115 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.116 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:25.116 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731558+0000", 2020-01-16T00:13:25.116 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010238809999999999 2020-01-16T00:13:25.116 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.116 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.116 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:25.116 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732582+0000", 2020-01-16T00:13:25.116 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000100918 2020-01-16T00:13:25.117 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.117 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.117 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:25.117 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732683+0000", 2020-01-16T00:13:25.117 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000112064 2020-01-16T00:13:25.117 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.117 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.118 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:25.118 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732795+0000", 2020-01-16T00:13:25.118 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014299630000000001 2020-01-16T00:13:25.118 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.118 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:25.118 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.118 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.119 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.119 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1917 3.a 3:5069665e:::smithi12313934-2:head [write 2113302~674388] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:13:25.119 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:57.330712+0000", 2020-01-16T00:13:25.119 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 27.656854815999999, 2020-01-16T00:13:25.119 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012781043000000001, 2020-01-16T00:13:25.119 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:25.119 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:25.120 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:25.121 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:25.121 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:25.121 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1917 2020-01-16T00:13:25.121 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.121 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:25.122 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.122 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:25.122 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.330712+0000", 2020-01-16T00:13:25.122 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:25.123 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.123 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.123 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:25.123 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.330712+0000", 2020-01-16T00:13:25.123 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:25.123 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.123 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.123 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:25.124 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.330709+0000", 2020-01-16T00:13:25.124 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00071550499999999996 2020-01-16T00:13:25.124 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.124 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.124 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:25.124 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331424+0000", 2020-01-16T00:13:25.124 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3354e-05 2020-01-16T00:13:25.125 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.125 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.125 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:25.125 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331447+0000", 2020-01-16T00:13:25.125 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8200000000000008e-06 2020-01-16T00:13:25.125 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.125 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.125 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.126 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331457+0000", 2020-01-16T00:13:25.126 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5557999999999997e-05 2020-01-16T00:13:25.126 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.126 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.126 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:25.126 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331533+0000", 2020-01-16T00:13:25.126 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030925000000000001 2020-01-16T00:13:25.127 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.127 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.127 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:25.127 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331842+0000", 2020-01-16T00:13:25.127 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0078507860000000002 2020-01-16T00:13:25.127 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.127 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.127 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:25.128 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.339693+0000", 2020-01-16T00:13:25.128 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9321000000000007e-05 2020-01-16T00:13:25.128 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.128 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.128 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:25.128 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.339792+0000", 2020-01-16T00:13:25.128 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015147700000000001 2020-01-16T00:13:25.129 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.129 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.129 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:25.129 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.339944+0000", 2020-01-16T00:13:25.129 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002663066 2020-01-16T00:13:25.129 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.129 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.129 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:25.130 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.342607+0000", 2020-01-16T00:13:25.130 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0001552 2020-01-16T00:13:25.130 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.130 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.130 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.131 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.342762+0000", 2020-01-16T00:13:25.131 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00063636900000000002 2020-01-16T00:13:25.131 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.131 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.131 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.131 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343398+0000", 2020-01-16T00:13:25.131 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9687000000000001e-05 2020-01-16T00:13:25.132 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.132 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.132 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:25.132 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343438+0000", 2020-01-16T00:13:25.132 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4647e-05 2020-01-16T00:13:25.132 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.133 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.133 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:25.133 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343493+0000", 2020-01-16T00:13:25.133 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012781043000000001 2020-01-16T00:13:25.134 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.134 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:25.134 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.134 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.135 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.135 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2308 3.a 3:5c245412:::smithi12313934-19:head [write 702592~703393] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:13:25.135 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:07.298509+0000", 2020-01-16T00:13:25.135 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 17.689057160000001, 2020-01-16T00:13:25.135 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012139225999999999, 2020-01-16T00:13:25.136 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:25.136 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:25.136 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:25.136 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:25.137 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:25.137 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2308 2020-01-16T00:13:25.137 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.137 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:25.138 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.138 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:25.138 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.298509+0000", 2020-01-16T00:13:25.138 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:25.139 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.139 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.139 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:25.140 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.298509+0000", 2020-01-16T00:13:25.140 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999957 2020-01-16T00:13:25.140 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.140 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.141 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:25.141 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.298505+0000", 2020-01-16T00:13:25.141 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00085313800000000003 2020-01-16T00:13:25.141 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.142 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.142 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:25.142 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.299358+0000", 2020-01-16T00:13:25.142 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5397e-05 2020-01-16T00:13:25.143 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.143 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.143 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:25.143 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.299394+0000", 2020-01-16T00:13:25.143 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3699e-05 2020-01-16T00:13:25.144 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.144 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.144 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.144 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.299407+0000", 2020-01-16T00:13:25.145 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014281999999999999 2020-01-16T00:13:25.145 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.145 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.145 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:25.146 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.299550+0000", 2020-01-16T00:13:25.146 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024247099999999999 2020-01-16T00:13:25.146 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.146 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.147 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:25.147 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.299793+0000", 2020-01-16T00:13:25.147 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00031642200000000001 2020-01-16T00:13:25.147 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.147 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.148 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:25.148 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.300109+0000", 2020-01-16T00:13:25.148 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0078127560000000006 2020-01-16T00:13:25.148 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.148 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.149 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:25.149 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.307922+0000", 2020-01-16T00:13:25.149 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0024170189999999999 2020-01-16T00:13:25.149 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.150 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.150 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.150 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.310339+0000", 2020-01-16T00:13:25.150 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.4764000000000004e-05 2020-01-16T00:13:25.150 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.151 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.151 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.151 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.310424+0000", 2020-01-16T00:13:25.151 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000128319 2020-01-16T00:13:25.151 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.152 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.152 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:25.152 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.310552+0000", 2020-01-16T00:13:25.152 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.6587000000000002e-05 2020-01-16T00:13:25.153 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.153 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.153 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:25.153 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.310648+0000", 2020-01-16T00:13:25.153 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012139225999999999 2020-01-16T00:13:25.154 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.154 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:25.154 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.154 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.155 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.155 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2504 3.a 3:5069665e:::smithi12313934-2:head [write 490837~718873] snapc 0=[] ondisk+write+known_if_redirected e37)", 2020-01-16T00:13:25.155 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:11.279881+0000", 2020-01-16T00:13:25.155 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 13.707685217, 2020-01-16T00:13:25.155 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010249138, 2020-01-16T00:13:25.156 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:25.156 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:25.156 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:25.156 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:25.156 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:25.157 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2504 2020-01-16T00:13:25.157 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.157 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:25.157 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.158 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:25.158 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.279881+0000", 2020-01-16T00:13:25.158 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:25.158 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.158 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.159 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:25.159 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.279881+0000", 2020-01-16T00:13:25.159 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999948 2020-01-16T00:13:25.159 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.160 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.160 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:25.160 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.279876+0000", 2020-01-16T00:13:25.160 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0025362599999999998 2020-01-16T00:13:25.161 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.161 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.161 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:25.161 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.282412+0000", 2020-01-16T00:13:25.161 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5186000000000003e-05 2020-01-16T00:13:25.162 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.162 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.162 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:25.162 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.282467+0000", 2020-01-16T00:13:25.162 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8800000000000003e-06 2020-01-16T00:13:25.163 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.163 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.163 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.163 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.282477+0000", 2020-01-16T00:13:25.163 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015124100000000001 2020-01-16T00:13:25.164 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.164 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.164 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:25.164 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.282628+0000", 2020-01-16T00:13:25.164 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00026402000000000001 2020-01-16T00:13:25.165 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.165 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.165 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:25.165 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.282892+0000", 2020-01-16T00:13:25.165 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00044943700000000001 2020-01-16T00:13:25.166 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.166 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.166 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:25.166 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.283342+0000", 2020-01-16T00:13:25.166 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0041154019999999998 2020-01-16T00:13:25.167 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.167 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.167 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:25.167 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.287457+0000", 2020-01-16T00:13:25.168 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0865999999999995e-05 2020-01-16T00:13:25.168 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.168 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.168 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.168 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.287538+0000", 2020-01-16T00:13:25.169 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0022868670000000002 2020-01-16T00:13:25.169 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.169 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.169 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.169 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289825+0000", 2020-01-16T00:13:25.170 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016695699999999999 2020-01-16T00:13:25.170 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.170 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.170 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:25.170 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289992+0000", 2020-01-16T00:13:25.171 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013842600000000001 2020-01-16T00:13:25.171 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.171 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.171 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:25.171 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.290130+0000", 2020-01-16T00:13:25.172 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010249138 2020-01-16T00:13:25.172 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.172 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:25.172 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.173 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.173 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.173 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2505 3.a 3:5069665e:::smithi12313934-2:head [write 1862260~479427] snapc 0=[] ondisk+write+known_if_redirected e37)", 2020-01-16T00:13:25.173 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:11.286548+0000", 2020-01-16T00:13:25.173 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 13.701018649, 2020-01-16T00:13:25.174 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013314061, 2020-01-16T00:13:25.174 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:25.174 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:25.174 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:25.174 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:25.175 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:25.175 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2505 2020-01-16T00:13:25.175 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.175 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:25.175 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.176 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:25.176 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.286548+0000", 2020-01-16T00:13:25.176 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:25.176 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.176 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.177 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:25.177 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.286548+0000", 2020-01-16T00:13:25.177 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999957 2020-01-16T00:13:25.177 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.178 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.178 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:25.178 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.286544+0000", 2020-01-16T00:13:25.178 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042043700000000001 2020-01-16T00:13:25.179 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.179 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.179 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:25.179 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.286964+0000", 2020-01-16T00:13:25.179 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5756999999999997e-05 2020-01-16T00:13:25.180 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.180 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.180 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:25.180 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.287000+0000", 2020-01-16T00:13:25.180 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8960000000000001e-05 2020-01-16T00:13:25.181 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.181 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.181 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.181 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.287019+0000", 2020-01-16T00:13:25.182 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.7219000000000005e-05 2020-01-16T00:13:25.182 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.182 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.182 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:25.182 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.287116+0000", 2020-01-16T00:13:25.182 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033208899999999998 2020-01-16T00:13:25.183 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.183 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.183 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:25.183 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.287448+0000", 2020-01-16T00:13:25.183 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0082732420000000001 2020-01-16T00:13:25.184 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.184 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.184 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:25.184 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.295721+0000", 2020-01-16T00:13:25.184 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017259999999999999 2020-01-16T00:13:25.184 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.185 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.185 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:25.185 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.295894+0000", 2020-01-16T00:13:25.185 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00029238000000000002 2020-01-16T00:13:25.185 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.186 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.186 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:25.186 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.296186+0000", 2020-01-16T00:13:25.186 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0034626409999999998 2020-01-16T00:13:25.186 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.187 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.187 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.187 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299649+0000", 2020-01-16T00:13:25.187 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2300000000000002e-06 2020-01-16T00:13:25.187 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.188 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.188 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:25.188 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299656+0000", 2020-01-16T00:13:25.188 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1561000000000001e-05 2020-01-16T00:13:25.188 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.188 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.188 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.188 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299738+0000", 2020-01-16T00:13:25.189 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8491000000000005e-05 2020-01-16T00:13:25.189 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.189 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.189 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:25.189 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299806+0000", 2020-01-16T00:13:25.189 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5541000000000002e-05 2020-01-16T00:13:25.189 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.190 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.190 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:25.190 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299862+0000", 2020-01-16T00:13:25.190 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013314061 2020-01-16T00:13:25.190 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.190 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:25.190 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.190 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.191 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.191 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2506 3.a 3:5069665e:::smithi12313934-2:head [write 2875032~15805] snapc 0=[] ondisk+write+known_if_redirected e37)", 2020-01-16T00:13:25.191 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:11.289486+0000", 2020-01-16T00:13:25.191 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 13.698080242, 2020-01-16T00:13:25.191 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01052632, 2020-01-16T00:13:25.191 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:25.191 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:25.192 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:25.192 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:25.192 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:25.192 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2506 2020-01-16T00:13:25.192 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.192 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:25.192 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.192 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:25.193 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289486+0000", 2020-01-16T00:13:25.193 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:25.193 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.193 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.193 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:25.193 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289486+0000", 2020-01-16T00:13:25.193 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:25.194 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.194 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.194 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:25.194 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289484+0000", 2020-01-16T00:13:25.194 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2185999999999998e-05 2020-01-16T00:13:25.194 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.194 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.194 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:25.195 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289536+0000", 2020-01-16T00:13:25.195 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.469e-05 2020-01-16T00:13:25.195 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.195 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.195 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:25.195 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289551+0000", 2020-01-16T00:13:25.195 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2469999999999999e-06 2020-01-16T00:13:25.196 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.196 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.196 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.196 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289556+0000", 2020-01-16T00:13:25.196 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6541e-05 2020-01-16T00:13:25.196 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.196 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.196 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:25.197 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289613+0000", 2020-01-16T00:13:25.197 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2774000000000006e-05 2020-01-16T00:13:25.197 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.197 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.197 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:25.197 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289676+0000", 2020-01-16T00:13:25.197 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0081875709999999994 2020-01-16T00:13:25.198 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.198 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.198 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:25.198 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.297863+0000", 2020-01-16T00:13:25.198 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012167699999999999 2020-01-16T00:13:25.199 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.199 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.199 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:25.199 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.297985+0000", 2020-01-16T00:13:25.199 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000118027 2020-01-16T00:13:25.199 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.199 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.199 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:25.200 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.298103+0000", 2020-01-16T00:13:25.200 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001559717 2020-01-16T00:13:25.200 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.200 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.200 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:25.200 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299663+0000", 2020-01-16T00:13:25.200 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00025753800000000003 2020-01-16T00:13:25.201 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.201 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.201 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.201 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299920+0000", 2020-01-16T00:13:25.201 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2994000000000003e-05 2020-01-16T00:13:25.201 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.201 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.201 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.202 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299953+0000", 2020-01-16T00:13:25.202 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9802e-05 2020-01-16T00:13:25.202 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.202 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.202 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:25.202 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299983+0000", 2020-01-16T00:13:25.202 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9578999999999999e-05 2020-01-16T00:13:25.202 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.203 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.203 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:25.203 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.300012+0000", 2020-01-16T00:13:25.203 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01052632 2020-01-16T00:13:25.203 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.203 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:25.204 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.204 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.204 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.204 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2507 3.a 3:5069665e:::smithi12313934-2:head [setxattr _header (55) in=7b,truncate 2890837] snapc 0=[] ondisk+write+known_if_redirected e37)", 2020-01-16T00:13:25.204 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:11.289625+0000", 2020-01-16T00:13:25.204 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 13.697940915, 2020-01-16T00:13:25.204 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010520467, 2020-01-16T00:13:25.204 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:25.205 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:25.205 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:25.205 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:25.205 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:25.205 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2507 2020-01-16T00:13:25.205 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.205 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:25.206 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.206 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:25.206 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289625+0000", 2020-01-16T00:13:25.206 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:25.206 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.206 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.206 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:25.207 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289625+0000", 2020-01-16T00:13:25.207 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:25.207 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.207 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.207 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:25.207 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289625+0000", 2020-01-16T00:13:25.207 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7239999999999997e-06 2020-01-16T00:13:25.208 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.208 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.208 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:25.208 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289630+0000", 2020-01-16T00:13:25.208 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4060000000000002e-06 2020-01-16T00:13:25.208 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.208 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.209 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:25.209 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289637+0000", 2020-01-16T00:13:25.209 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2569999999999999e-06 2020-01-16T00:13:25.209 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.210 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.210 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.210 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289640+0000", 2020-01-16T00:13:25.210 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00060159099999999995 2020-01-16T00:13:25.210 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.211 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.211 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:25.211 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.290242+0000", 2020-01-16T00:13:25.211 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016406 2020-01-16T00:13:25.211 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.212 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.212 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:25.212 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.290406+0000", 2020-01-16T00:13:25.212 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0082527190000000004 2020-01-16T00:13:25.212 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.212 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.212 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:25.212 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.298658+0000", 2020-01-16T00:13:25.213 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013482300000000001 2020-01-16T00:13:25.213 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.213 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.213 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:25.213 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.298793+0000", 2020-01-16T00:13:25.213 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00023945699999999999 2020-01-16T00:13:25.213 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.214 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.214 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:25.214 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299033+0000", 2020-01-16T00:13:25.214 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00089435600000000001 2020-01-16T00:13:25.214 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.214 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.214 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:25.215 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299927+0000", 2020-01-16T00:13:25.215 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012217599999999999 2020-01-16T00:13:25.215 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.215 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.215 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.215 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.300049+0000", 2020-01-16T00:13:25.215 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.5884000000000002e-05 2020-01-16T00:13:25.215 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.216 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.216 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.216 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.300075+0000", 2020-01-16T00:13:25.216 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.8819e-05 2020-01-16T00:13:25.216 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.216 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.216 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:25.217 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.300104+0000", 2020-01-16T00:13:25.217 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2049999999999999e-05 2020-01-16T00:13:25.217 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.217 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.217 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:25.217 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.300146+0000", 2020-01-16T00:13:25.217 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010520467 2020-01-16T00:13:25.217 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.218 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:25.218 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.218 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.218 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.218 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2508 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e37)", 2020-01-16T00:13:25.218 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:11.289655+0000", 2020-01-16T00:13:25.218 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 13.697911705999999, 2020-01-16T00:13:25.218 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010659678000000001, 2020-01-16T00:13:25.219 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:25.219 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:25.219 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:25.219 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:25.219 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:25.219 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2508 2020-01-16T00:13:25.219 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.220 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:25.220 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.220 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:25.220 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289655+0000", 2020-01-16T00:13:25.220 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:25.220 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.220 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.221 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:25.221 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289655+0000", 2020-01-16T00:13:25.221 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:25.221 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.221 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.222 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:25.222 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289654+0000", 2020-01-16T00:13:25.222 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1619999999999999e-06 2020-01-16T00:13:25.222 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.223 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.223 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:25.223 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289659+0000", 2020-01-16T00:13:25.223 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6009999999999997e-06 2020-01-16T00:13:25.223 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.224 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.224 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:25.224 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289664+0000", 2020-01-16T00:13:25.224 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.5409999999999999e-06 2020-01-16T00:13:25.225 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.225 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.225 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.225 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.289667+0000", 2020-01-16T00:13:25.226 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00088239600000000003 2020-01-16T00:13:25.226 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.226 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.226 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:25.227 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.290549+0000", 2020-01-16T00:13:25.227 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013208 2020-01-16T00:13:25.227 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.227 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.228 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:25.228 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.290681+0000", 2020-01-16T00:13:25.228 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0086573699999999993 2020-01-16T00:13:25.228 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.229 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.229 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:25.229 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299338+0000", 2020-01-16T00:13:25.229 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8002000000000001e-05 2020-01-16T00:13:25.229 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.230 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.230 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:25.230 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299436+0000", 2020-01-16T00:13:25.230 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00073602499999999996 2020-01-16T00:13:25.231 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.231 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.231 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:25.231 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.300172+0000", 2020-01-16T00:13:25.232 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0544000000000002e-05 2020-01-16T00:13:25.232 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.232 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.232 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:25.232 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.300233+0000", 2020-01-16T00:13:25.233 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1453999999999996e-05 2020-01-16T00:13:25.233 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.233 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.233 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:25.234 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.300314+0000", 2020-01-16T00:13:25.234 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010659678000000001 2020-01-16T00:13:25.234 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.234 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:25.234 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.235 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.235 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.235 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2789 3.a 3:5069665e:::smithi12313934-2:head [read 0~2890837,checksum] snapc 0=[] ondisk+read+known_if_redirected e41)", 2020-01-16T00:13:25.235 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:17.213121+0000", 2020-01-16T00:13:25.235 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 7.7744458879999998, 2020-01-16T00:13:25.236 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011386347, 2020-01-16T00:13:25.236 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:25.236 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:25.236 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:25.236 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:25.237 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:25.237 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2789 2020-01-16T00:13:25.237 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.237 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:25.238 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.238 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:25.238 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213121+0000", 2020-01-16T00:13:25.238 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:25.238 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.239 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.239 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:25.239 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213121+0000", 2020-01-16T00:13:25.240 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:25.240 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.240 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.240 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:25.240 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213119+0000", 2020-01-16T00:13:25.241 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3570000000000003e-06 2020-01-16T00:13:25.241 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.241 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.241 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:25.242 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213126+0000", 2020-01-16T00:13:25.242 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2060000000000001e-06 2020-01-16T00:13:25.242 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.242 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.242 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:25.243 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213133+0000", 2020-01-16T00:13:25.243 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7700000000000004e-06 2020-01-16T00:13:25.244 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.245 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.245 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.245 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213140+0000", 2020-01-16T00:13:25.245 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0089949739999999993 2020-01-16T00:13:25.246 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.246 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.246 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:25.246 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.222135+0000", 2020-01-16T00:13:25.247 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000191697 2020-01-16T00:13:25.247 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.247 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.247 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:25.248 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.222326+0000", 2020-01-16T00:13:25.248 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002180436 2020-01-16T00:13:25.248 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.248 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.248 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:25.249 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.224507+0000", 2020-01-16T00:13:25.249 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011386347 2020-01-16T00:13:25.249 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.249 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:25.250 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.250 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.250 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.250 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2790 3.a 3:5069665e:::smithi12313934-2:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e41)", 2020-01-16T00:13:25.251 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:17.213146+0000", 2020-01-16T00:13:25.251 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 7.7744206509999998, 2020-01-16T00:13:25.251 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012098164999999999, 2020-01-16T00:13:25.251 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:25.251 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:25.252 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:25.252 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:25.252 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:25.252 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2790 2020-01-16T00:13:25.253 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.253 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:25.253 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.253 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:25.254 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213146+0000", 2020-01-16T00:13:25.254 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:25.254 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.254 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.254 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:25.255 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213146+0000", 2020-01-16T00:13:25.255 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:25.255 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.255 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.255 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:25.256 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213145+0000", 2020-01-16T00:13:25.256 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4399999999999998e-06 2020-01-16T00:13:25.256 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.256 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.257 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:25.257 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213149+0000", 2020-01-16T00:13:25.257 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1029999999999996e-06 2020-01-16T00:13:25.257 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.257 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.258 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:25.258 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213156+0000", 2020-01-16T00:13:25.258 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0699999999999997e-06 2020-01-16T00:13:25.258 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.259 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.259 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.259 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213161+0000", 2020-01-16T00:13:25.259 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011386617 2020-01-16T00:13:25.259 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.259 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.260 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:25.260 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.224548+0000", 2020-01-16T00:13:25.260 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017114499999999999 2020-01-16T00:13:25.260 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.260 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.260 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:25.260 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.224719+0000", 2020-01-16T00:13:25.261 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00052473000000000005 2020-01-16T00:13:25.261 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.261 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.261 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:25.261 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.225244+0000", 2020-01-16T00:13:25.261 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012098164999999999 2020-01-16T00:13:25.261 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.262 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:25.262 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.262 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.262 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.262 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2791 3.a 3:54f2c117:::smithi12313934-41:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e41)", 2020-01-16T00:13:25.262 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:17.213167+0000", 2020-01-16T00:13:25.262 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 7.7743996380000002, 2020-01-16T00:13:25.262 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016024037000000001, 2020-01-16T00:13:25.263 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:25.263 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:25.263 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:25.263 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:25.263 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:25.263 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2791 2020-01-16T00:13:25.263 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.264 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:25.264 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.264 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:25.264 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213167+0000", 2020-01-16T00:13:25.264 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:25.264 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.264 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.264 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:25.265 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213167+0000", 2020-01-16T00:13:25.265 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:25.265 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.265 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.265 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:25.265 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213166+0000", 2020-01-16T00:13:25.265 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6030000000000004e-06 2020-01-16T00:13:25.265 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.266 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.266 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:25.266 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213172+0000", 2020-01-16T00:13:25.266 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0929999999999997e-06 2020-01-16T00:13:25.266 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.266 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.266 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:25.267 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213179+0000", 2020-01-16T00:13:25.267 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4299999999999997e-06 2020-01-16T00:13:25.267 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.267 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.267 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.267 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213184+0000", 2020-01-16T00:13:25.267 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012106695000000001 2020-01-16T00:13:25.267 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.268 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.268 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:25.268 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.225291+0000", 2020-01-16T00:13:25.268 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020863699999999999 2020-01-16T00:13:25.268 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.268 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.268 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:25.268 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.225499+0000", 2020-01-16T00:13:25.269 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0036913050000000002 2020-01-16T00:13:25.269 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.269 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.269 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:25.269 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.229191+0000", 2020-01-16T00:13:25.269 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016024037000000001 2020-01-16T00:13:25.269 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.270 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:25.270 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.270 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.270 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.270 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2792 3.a 3:54f2c117:::smithi12313934-41:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e41)", 2020-01-16T00:13:25.270 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:17.213190+0000", 2020-01-16T00:13:25.270 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 7.7743761640000004, 2020-01-16T00:13:25.270 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017165949, 2020-01-16T00:13:25.271 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:25.271 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:25.271 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:25.271 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:25.271 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:25.271 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2792 2020-01-16T00:13:25.271 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.271 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:25.272 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.272 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:25.272 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213190+0000", 2020-01-16T00:13:25.272 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:25.272 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.272 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.272 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:25.272 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213190+0000", 2020-01-16T00:13:25.273 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:25.273 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.273 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.273 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:25.273 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213189+0000", 2020-01-16T00:13:25.273 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2699999999999998e-06 2020-01-16T00:13:25.273 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.274 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.274 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:25.274 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213194+0000", 2020-01-16T00:13:25.274 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8600000000000004e-06 2020-01-16T00:13:25.274 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.274 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.274 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:25.274 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213201+0000", 2020-01-16T00:13:25.275 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8500000000000002e-06 2020-01-16T00:13:25.275 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.275 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.275 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.275 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213205+0000", 2020-01-16T00:13:25.275 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016028924999999999 2020-01-16T00:13:25.275 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.276 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.276 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:25.276 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.229234+0000", 2020-01-16T00:13:25.276 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019773999999999999 2020-01-16T00:13:25.276 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.276 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.276 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:25.276 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.229432+0000", 2020-01-16T00:13:25.277 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00092414400000000001 2020-01-16T00:13:25.277 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.277 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.277 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:25.277 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.230356+0000", 2020-01-16T00:13:25.277 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017165949 2020-01-16T00:13:25.277 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.278 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:25.278 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.278 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.278 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.278 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2793 3.a 3:54f2c117:::smithi12313934-41:head [read 0~3210170,checksum] snapc 0=[] ondisk+read+known_if_redirected e41)", 2020-01-16T00:13:25.278 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:17.213211+0000", 2020-01-16T00:13:25.279 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 7.7743556180000004, 2020-01-16T00:13:25.279 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019069413, 2020-01-16T00:13:25.279 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:25.279 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:25.279 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:25.279 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:25.279 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:25.279 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2793 2020-01-16T00:13:25.280 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.280 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:25.280 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.280 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:25.280 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213211+0000", 2020-01-16T00:13:25.280 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:25.280 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.280 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.281 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:25.281 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213211+0000", 2020-01-16T00:13:25.281 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:25.281 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.281 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.281 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:25.281 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213210+0000", 2020-01-16T00:13:25.282 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.66e-06 2020-01-16T00:13:25.282 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.282 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.282 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:25.282 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213216+0000", 2020-01-16T00:13:25.282 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.9469999999999997e-06 2020-01-16T00:13:25.282 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.282 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.283 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:25.283 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213223+0000", 2020-01-16T00:13:25.283 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9259999999999999e-06 2020-01-16T00:13:25.283 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.283 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.283 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.283 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213228+0000", 2020-01-16T00:13:25.283 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017171046999999998 2020-01-16T00:13:25.284 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.284 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.284 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:25.284 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.230399+0000", 2020-01-16T00:13:25.284 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000189932 2020-01-16T00:13:25.284 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.284 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.285 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:25.285 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.230589+0000", 2020-01-16T00:13:25.285 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0016916539999999999 2020-01-16T00:13:25.285 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.285 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.285 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:25.285 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.232280+0000", 2020-01-16T00:13:25.285 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019069413 2020-01-16T00:13:25.286 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.286 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:25.286 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.286 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.286 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.286 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.4488.0:2987 3.d e43/40 3:b3dd5d45:::smithi12313934-16:head v 43'187, mlcod=42'185)", 2020-01-16T00:13:25.286 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:22.260074+0000", 2020-01-16T00:13:25.286 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 2.7274921220000001, 2020-01-16T00:13:25.287 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0096331850000000007, 2020-01-16T00:13:25.287 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:25.287 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:25.287 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:25.287 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.287 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:25.287 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.260074+0000", 2020-01-16T00:13:25.288 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:25.288 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.288 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.288 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:25.288 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.260074+0000", 2020-01-16T00:13:25.288 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:25.288 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.288 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.289 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:25.289 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.260073+0000", 2020-01-16T00:13:25.289 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00090244200000000004 2020-01-16T00:13:25.289 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.289 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.289 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:25.289 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.260975+0000", 2020-01-16T00:13:25.289 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0963000000000002e-05 2020-01-16T00:13:25.290 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.290 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.290 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:25.290 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.261006+0000", 2020-01-16T00:13:25.290 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0463e-05 2020-01-16T00:13:25.290 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.290 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.291 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.291 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.261027+0000", 2020-01-16T00:13:25.291 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014083100000000001 2020-01-16T00:13:25.291 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.291 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.291 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:25.291 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.261168+0000", 2020-01-16T00:13:25.291 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.8326999999999996e-05 2020-01-16T00:13:25.292 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.292 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.292 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:25.292 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.261256+0000", 2020-01-16T00:13:25.292 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0084050059999999996 2020-01-16T00:13:25.292 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.292 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.292 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:25.293 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.269661+0000", 2020-01-16T00:13:25.293 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6492999999999997e-05 2020-01-16T00:13:25.293 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:25.293 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:25.293 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:25.293 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.269707+0000", 2020-01-16T00:13:25.293 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0096331850000000007 2020-01-16T00:13:25.294 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.294 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:25.294 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.294 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:25.294 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:25.294 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:25.297 INFO:tasks.rados.rados.0.smithi123.stdout:1230: finishing write tid 2 to smithi12313934-9 2020-01-16T00:13:25.298 INFO:tasks.rados.rados.0.smithi123.stdout:1230: finishing write tid 4 to smithi12313934-9 2020-01-16T00:13:25.298 INFO:tasks.rados.rados.0.smithi123.stdout:1230: finishing write tid 5 to smithi12313934-9 2020-01-16T00:13:25.298 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 9 v 124 (ObjNum 372 snap 0 seq_num 372) dirty exists 2020-01-16T00:13:25.298 INFO:tasks.rados.rados.0.smithi123.stdout:1230: left oid 9 (ObjNum 372 snap 0 seq_num 372) 2020-01-16T00:13:25.298 INFO:tasks.rados.rados.0.smithi123.stdout:1231: expect (ObjNum 365 snap 0 seq_num 365) 2020-01-16T00:13:25.298 INFO:tasks.rados.rados.0.smithi123.stdout:1232: finishing write tid 1 to smithi12313934-21 2020-01-16T00:13:25.298 INFO:tasks.rados.rados.0.smithi123.stdout:1232: finishing write tid 2 to smithi12313934-21 2020-01-16T00:13:25.299 INFO:tasks.rados.rados.0.smithi123.stdout:1232: finishing write tid 4 to smithi12313934-21 2020-01-16T00:13:25.299 INFO:tasks.rados.rados.0.smithi123.stdout:1232: finishing write tid 5 to smithi12313934-21 2020-01-16T00:13:25.299 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 21 v 49 (ObjNum 373 snap 0 seq_num 373) dirty exists 2020-01-16T00:13:25.299 INFO:tasks.rados.rados.0.smithi123.stdout:1232: left oid 21 (ObjNum 373 snap 0 seq_num 373) 2020-01-16T00:13:25.299 INFO:tasks.rados.rados.0.smithi123.stdout:1233: finishing write tid 1 to smithi12313934-15 2020-01-16T00:13:25.299 INFO:tasks.rados.rados.0.smithi123.stdout:1233: finishing write tid 2 to smithi12313934-15 2020-01-16T00:13:25.299 INFO:tasks.rados.rados.0.smithi123.stdout:1233: finishing write tid 4 to smithi12313934-15 2020-01-16T00:13:25.300 INFO:tasks.rados.rados.0.smithi123.stdout:1233: finishing write tid 5 to smithi12313934-15 2020-01-16T00:13:25.300 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 15 v 145 (ObjNum 374 snap 0 seq_num 374) dirty exists 2020-01-16T00:13:25.300 INFO:tasks.rados.rados.0.smithi123.stdout:1233: left oid 15 (ObjNum 374 snap 0 seq_num 374) 2020-01-16T00:13:25.300 INFO:tasks.rados.rados.0.smithi123.stdout:1234: finishing write tid 1 to smithi12313934-32 2020-01-16T00:13:25.300 INFO:tasks.rados.rados.0.smithi123.stdout:1227: done (15 left) 2020-01-16T00:13:25.300 INFO:tasks.rados.rados.0.smithi123.stdout:1228: done (14 left) 2020-01-16T00:13:25.300 INFO:tasks.rados.rados.0.smithi123.stdout:1229: done (13 left) 2020-01-16T00:13:25.301 INFO:tasks.rados.rados.0.smithi123.stdout:1230: done (12 left) 2020-01-16T00:13:25.301 INFO:tasks.rados.rados.0.smithi123.stdout:1231: done (11 left) 2020-01-16T00:13:25.301 INFO:tasks.rados.rados.0.smithi123.stdout:1232: done (10 left) 2020-01-16T00:13:25.301 INFO:tasks.rados.rados.0.smithi123.stdout:1233: done (9 left) 2020-01-16T00:13:25.301 INFO:tasks.rados.rados.0.smithi123.stdout:1243: read oid 16 snap -1 2020-01-16T00:13:25.301 INFO:tasks.rados.rados.0.smithi123.stdout:1243: expect (ObjNum 361 snap 0 seq_num 361) 2020-01-16T00:13:25.301 INFO:tasks.rados.rados.0.smithi123.stdout:1244: read oid 38 snap -1 2020-01-16T00:13:25.301 INFO:tasks.rados.rados.0.smithi123.stdout:1244: expect (ObjNum 298 snap 0 seq_num 298) 2020-01-16T00:13:25.302 INFO:tasks.rados.rados.0.smithi123.stdout:1245: write oid 37 current snap is 0 2020-01-16T00:13:25.302 INFO:tasks.rados.rados.0.smithi123.stdout:1245: seq_num 378 ranges {753046=545419,1786664=431793} 2020-01-16T00:13:25.302 INFO:tasks.rados.rados.0.smithi123.stdout:1245: writing smithi12313934-37 from 753046 to 1298465 tid 1 2020-01-16T00:13:25.302 INFO:tasks.rados.rados.0.smithi123.stdout:1245: writing smithi12313934-37 from 1786664 to 2218457 tid 2 2020-01-16T00:13:25.302 INFO:tasks.rados.rados.0.smithi123.stdout:1246: write (excl) oid 2 current snap is 0 2020-01-16T00:13:25.302 INFO:tasks.rados.rados.0.smithi123.stdout:1246: seq_num 379 ranges {769853=620668,1845146=524707} 2020-01-16T00:13:25.302 INFO:tasks.rados.rados.0.smithi123.stdout:1246: writing smithi12313934-2 from 769853 to 1390521 tid 1 2020-01-16T00:13:25.303 INFO:tasks.rados.rados.0.smithi123.stdout:1246: writing smithi12313934-2 from 1845146 to 2369853 tid 2 2020-01-16T00:13:25.303 INFO:tasks.rados.rados.0.smithi123.stdout:1247: write oid 44 current snap is 0 2020-01-16T00:13:25.303 INFO:tasks.rados.rados.0.smithi123.stdout:1247: seq_num 380 ranges {786660=695917,1903628=435462} 2020-01-16T00:13:25.303 INFO:tasks.rados.rados.0.smithi123.stdout:1247: writing smithi12313934-44 from 786660 to 1482577 tid 1 2020-01-16T00:13:25.303 INFO:tasks.rados.rados.0.smithi123.stdout:1247: writing smithi12313934-44 from 1903628 to 2339090 tid 2 2020-01-16T00:13:25.303 INFO:tasks.rados.rados.0.smithi123.stdout:1248: read oid 11 snap -1 2020-01-16T00:13:25.303 INFO:tasks.rados.rados.0.smithi123.stdout:1248: expect (ObjNum 329 snap 0 seq_num 329) 2020-01-16T00:13:25.303 INFO:tasks.rados.rados.0.smithi123.stdout:1249: read oid 46 snap -1 2020-01-16T00:13:25.304 INFO:tasks.rados.rados.0.smithi123.stdout:1249: expect (ObjNum 328 snap 0 seq_num 328) 2020-01-16T00:13:25.304 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:25.304 INFO:tasks.rados.rados.0.smithi123.stdout:1234: finishing write tid 2 to smithi12313934-32 2020-01-16T00:13:25.304 INFO:tasks.rados.rados.0.smithi123.stdout:1234: finishing write tid 4 to smithi12313934-32 2020-01-16T00:13:25.304 INFO:tasks.rados.rados.0.smithi123.stdout:1234: finishing write tid 5 to smithi12313934-32 2020-01-16T00:13:25.304 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 32 v 127 (ObjNum 375 snap 0 seq_num 375) dirty exists 2020-01-16T00:13:25.304 INFO:tasks.rados.rados.0.smithi123.stdout:1234: left oid 32 (ObjNum 375 snap 0 seq_num 375) 2020-01-16T00:13:25.305 INFO:tasks.rados.rados.0.smithi123.stdout:1235: expect (ObjNum 347 snap 0 seq_num 347) 2020-01-16T00:13:25.305 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:25.290+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:25.305 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-01-16T00:13:25.397 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:25.394+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:25.402 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:25.403 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:25.403 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:13:25.403 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:25.412 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-01-16T00:13:25.500 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:25.498+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:25.537 INFO:tasks.rados.rados.0.smithi123.stdout:1236: finishing write tid 1 to smithi12313934-6 2020-01-16T00:13:25.538 INFO:tasks.rados.rados.0.smithi123.stdout:1236: finishing write tid 2 to smithi12313934-6 2020-01-16T00:13:25.538 INFO:tasks.rados.rados.0.smithi123.stdout:1236: finishing write tid 4 to smithi12313934-6 2020-01-16T00:13:25.538 INFO:tasks.rados.rados.0.smithi123.stdout:1236: finishing write tid 5 to smithi12313934-6 2020-01-16T00:13:25.538 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 6 v 111 (ObjNum 376 snap 0 seq_num 376) dirty exists 2020-01-16T00:13:25.539 INFO:tasks.rados.rados.0.smithi123.stdout:1236: left oid 6 (ObjNum 376 snap 0 seq_num 376) 2020-01-16T00:13:25.539 INFO:tasks.rados.rados.0.smithi123.stdout:1234: done (15 left) 2020-01-16T00:13:25.539 INFO:tasks.rados.rados.0.smithi123.stdout:1235: done (14 left) 2020-01-16T00:13:25.539 INFO:tasks.rados.rados.0.smithi123.stdout:1236: done (13 left) 2020-01-16T00:13:25.539 INFO:tasks.rados.rados.0.smithi123.stdout:1250: read oid 48 snap -1 2020-01-16T00:13:25.540 INFO:tasks.rados.rados.0.smithi123.stdout:1250: expect (ObjNum 356 snap 0 seq_num 356) 2020-01-16T00:13:25.540 INFO:tasks.rados.rados.0.smithi123.stdout:1251: read oid 49 snap -1 2020-01-16T00:13:25.540 INFO:tasks.rados.rados.0.smithi123.stdout:1251: expect (ObjNum 371 snap 0 seq_num 371) 2020-01-16T00:13:25.540 INFO:tasks.rados.rados.0.smithi123.stdout:1252: read oid 41 snap -1 2020-01-16T00:13:25.541 INFO:tasks.rados.rados.0.smithi123.stdout:1252: expect (ObjNum 310 snap 0 seq_num 310) 2020-01-16T00:13:25.541 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:25.541 INFO:tasks.rados.rados.0.smithi123.stdout:1237: finishing write tid 1 to smithi12313934-3 2020-01-16T00:13:25.541 INFO:tasks.rados.rados.0.smithi123.stdout:1240: expect (ObjNum 330 snap 0 seq_num 330) 2020-01-16T00:13:25.557 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:25.557 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:25.558 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:13:25.558 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:13:25.558 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:25.566 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-01-16T00:13:25.600 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:25.596+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:25.700 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:25.698+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:25.719 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:25.719 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:13:25.720 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:13:25.720 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:13:25.720 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.720 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:939 3.4 3:2bdd9e88:::smithi12313934-4:head [write 1681102~474010] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:25.720 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.908392+0000", 2020-01-16T00:13:25.721 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 47.804825321999999, 2020-01-16T00:13:25.721 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019294697, 2020-01-16T00:13:25.721 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:25.721 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:25.722 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:25.722 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:25.722 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:25.722 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 939 2020-01-16T00:13:25.723 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.723 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:25.723 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.723 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:25.724 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908392+0000", 2020-01-16T00:13:25.724 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:25.724 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.724 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.725 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:25.725 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908392+0000", 2020-01-16T00:13:25.725 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:25.726 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.726 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.726 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:25.726 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908390+0000", 2020-01-16T00:13:25.727 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015440399999999999 2020-01-16T00:13:25.727 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.727 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.727 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:25.728 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908545+0000", 2020-01-16T00:13:25.728 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3926e-05 2020-01-16T00:13:25.728 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.728 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.728 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:25.729 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908559+0000", 2020-01-16T00:13:25.729 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2200000000000003e-06 2020-01-16T00:13:25.729 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.729 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.730 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.730 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908566+0000", 2020-01-16T00:13:25.730 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020594799999999998 2020-01-16T00:13:25.730 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.731 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.731 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.731 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910625+0000", 2020-01-16T00:13:25.731 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118672 2020-01-16T00:13:25.731 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.732 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.732 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:25.732 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910744+0000", 2020-01-16T00:13:25.732 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031940969999999999 2020-01-16T00:13:25.733 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.733 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.733 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.733 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.913938+0000", 2020-01-16T00:13:25.734 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000309621 2020-01-16T00:13:25.734 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.734 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.734 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:25.734 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.914248+0000", 2020-01-16T00:13:25.735 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.008818579 2020-01-16T00:13:25.735 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.735 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.735 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.736 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.923066+0000", 2020-01-16T00:13:25.736 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000108989 2020-01-16T00:13:25.736 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.736 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.737 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:25.737 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.923175+0000", 2020-01-16T00:13:25.737 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019128299999999999 2020-01-16T00:13:25.737 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.738 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.738 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:13:25.738 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.923367+0000", 2020-01-16T00:13:25.738 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0032083760000000002 2020-01-16T00:13:25.738 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.739 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.739 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:25.739 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926575+0000", 2020-01-16T00:13:25.739 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2475999999999999e-05 2020-01-16T00:13:25.740 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.740 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.740 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.740 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926607+0000", 2020-01-16T00:13:25.740 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00099399800000000006 2020-01-16T00:13:25.741 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.741 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.741 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.741 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927601+0000", 2020-01-16T00:13:25.742 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0958999999999999e-05 2020-01-16T00:13:25.742 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.742 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.742 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:25.743 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927642+0000", 2020-01-16T00:13:25.743 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4747999999999999e-05 2020-01-16T00:13:25.743 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.743 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.744 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:25.744 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927687+0000", 2020-01-16T00:13:25.744 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019294697 2020-01-16T00:13:25.744 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:25.745 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:25.745 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:25.745 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.745 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.745 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:940 3.4 3:2bdd9e88:::smithi12313934-4:head [write 2687912~589420] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:25.746 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.915144+0000", 2020-01-16T00:13:25.746 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 47.798073954000003, 2020-01-16T00:13:25.746 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016092538999999999, 2020-01-16T00:13:25.746 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:25.747 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:25.747 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:25.747 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:25.747 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:25.747 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 940 2020-01-16T00:13:25.748 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.748 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:25.748 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.748 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:25.749 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915144+0000", 2020-01-16T00:13:25.749 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:25.749 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.749 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.750 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:25.750 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915144+0000", 2020-01-16T00:13:25.750 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:25.750 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.751 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.751 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:25.751 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915141+0000", 2020-01-16T00:13:25.751 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016366300000000001 2020-01-16T00:13:25.752 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.752 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.752 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:25.752 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915305+0000", 2020-01-16T00:13:25.752 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4847e-05 2020-01-16T00:13:25.753 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.753 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.753 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:25.753 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915320+0000", 2020-01-16T00:13:25.754 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2599999999999999e-06 2020-01-16T00:13:25.754 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.754 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.754 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.755 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915327+0000", 2020-01-16T00:13:25.755 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4282000000000005e-05 2020-01-16T00:13:25.755 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.755 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.755 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.756 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915401+0000", 2020-01-16T00:13:25.756 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4554000000000006e-05 2020-01-16T00:13:25.756 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.756 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.757 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:25.757 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915466+0000", 2020-01-16T00:13:25.757 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0090821900000000004 2020-01-16T00:13:25.757 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.757 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.758 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.758 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.924548+0000", 2020-01-16T00:13:25.758 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000114696 2020-01-16T00:13:25.758 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.759 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.759 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:25.759 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.924663+0000", 2020-01-16T00:13:25.759 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000168651 2020-01-16T00:13:25.760 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.760 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.760 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:13:25.760 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.924832+0000", 2020-01-16T00:13:25.760 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018253430000000001 2020-01-16T00:13:25.761 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.761 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.761 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:25.761 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926657+0000", 2020-01-16T00:13:25.761 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9957999999999999e-05 2020-01-16T00:13:25.761 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.761 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.762 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.762 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926697+0000", 2020-01-16T00:13:25.762 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0044685250000000001 2020-01-16T00:13:25.762 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.762 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.762 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.762 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931165+0000", 2020-01-16T00:13:25.762 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9211000000000001e-05 2020-01-16T00:13:25.763 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.763 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.763 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:25.763 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931205+0000", 2020-01-16T00:13:25.763 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1699999999999998e-05 2020-01-16T00:13:25.763 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.763 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.764 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:25.764 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931236+0000", 2020-01-16T00:13:25.764 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016092538999999999 2020-01-16T00:13:25.764 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:25.764 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:25.764 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:25.764 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.765 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.765 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:941 3.4 3:2bdd9e88:::smithi12313934-4:head [setxattr _header (55) in=7b,truncate 3277332] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:25.765 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.915341+0000", 2020-01-16T00:13:25.765 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 47.797877135999997, 2020-01-16T00:13:25.765 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016603769000000001, 2020-01-16T00:13:25.765 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:25.765 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:25.766 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:25.766 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:25.766 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:25.766 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 941 2020-01-16T00:13:25.766 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.766 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:25.767 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.767 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:25.767 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915341+0000", 2020-01-16T00:13:25.767 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:25.767 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.767 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.767 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:25.767 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915341+0000", 2020-01-16T00:13:25.768 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:25.768 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.768 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.768 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:25.768 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915340+0000", 2020-01-16T00:13:25.768 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.896e-06 2020-01-16T00:13:25.768 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.769 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.769 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:25.769 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915346+0000", 2020-01-16T00:13:25.769 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2730000000000002e-06 2020-01-16T00:13:25.769 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.769 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.769 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:25.769 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915354+0000", 2020-01-16T00:13:25.770 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0616e-05 2020-01-16T00:13:25.770 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.770 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.770 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.770 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915374+0000", 2020-01-16T00:13:25.770 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000133487 2020-01-16T00:13:25.770 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.770 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.771 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.771 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915508+0000", 2020-01-16T00:13:25.771 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8971000000000003e-05 2020-01-16T00:13:25.771 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.771 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.771 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:25.771 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915557+0000", 2020-01-16T00:13:25.772 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010138787999999999 2020-01-16T00:13:25.772 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.772 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.772 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.772 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.925696+0000", 2020-01-16T00:13:25.772 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012825400000000001 2020-01-16T00:13:25.772 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.772 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.773 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:25.773 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.925824+0000", 2020-01-16T00:13:25.773 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017706500000000001 2020-01-16T00:13:25.773 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.773 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.773 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:13:25.773 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926001+0000", 2020-01-16T00:13:25.773 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00079988900000000003 2020-01-16T00:13:25.774 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.774 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.774 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.774 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926801+0000", 2020-01-16T00:13:25.774 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119324 2020-01-16T00:13:25.774 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.774 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.775 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:25.775 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926920+0000", 2020-01-16T00:13:25.775 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048879910000000004 2020-01-16T00:13:25.775 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.775 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.775 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.775 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931808+0000", 2020-01-16T00:13:25.776 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6999999999999997e-05 2020-01-16T00:13:25.776 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.776 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.776 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:25.776 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931855+0000", 2020-01-16T00:13:25.776 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9092999999999996e-05 2020-01-16T00:13:25.776 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.776 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.777 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:25.777 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931944+0000", 2020-01-16T00:13:25.777 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016603769000000001 2020-01-16T00:13:25.777 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:25.777 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:25.778 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:25.778 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.778 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.778 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:942 3.4 3:2bdd9e88:::smithi12313934-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:13:25.778 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.915389+0000", 2020-01-16T00:13:25.778 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 47.797828770000002, 2020-01-16T00:13:25.778 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016820563, 2020-01-16T00:13:25.779 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:25.779 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:25.779 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:25.779 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:25.779 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:25.779 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 942 2020-01-16T00:13:25.779 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.779 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:25.780 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.780 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:25.780 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915389+0000", 2020-01-16T00:13:25.780 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:25.780 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.780 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.781 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:25.781 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915389+0000", 2020-01-16T00:13:25.781 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:25.781 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.781 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.781 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:25.781 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915388+0000", 2020-01-16T00:13:25.781 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.683e-06 2020-01-16T00:13:25.782 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.782 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.782 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:25.782 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915391+0000", 2020-01-16T00:13:25.782 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4839999999999998e-06 2020-01-16T00:13:25.782 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.783 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.783 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:25.783 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915399+0000", 2020-01-16T00:13:25.783 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2259999999999999e-06 2020-01-16T00:13:25.783 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.783 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.783 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.784 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915403+0000", 2020-01-16T00:13:25.784 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00031301199999999999 2020-01-16T00:13:25.784 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.784 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.784 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.784 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915716+0000", 2020-01-16T00:13:25.784 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2079999999999998e-05 2020-01-16T00:13:25.784 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.785 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.785 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:25.785 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915748+0000", 2020-01-16T00:13:25.785 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010547076000000001 2020-01-16T00:13:25.785 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.785 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.785 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.786 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926295+0000", 2020-01-16T00:13:25.786 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8040000000000005e-05 2020-01-16T00:13:25.786 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.786 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.786 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:25.786 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926373+0000", 2020-01-16T00:13:25.786 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056061990000000001 2020-01-16T00:13:25.786 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.787 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.787 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.787 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931979+0000", 2020-01-16T00:13:25.787 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011540399999999999 2020-01-16T00:13:25.787 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.787 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.787 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:25.787 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932095+0000", 2020-01-16T00:13:25.788 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011482899999999999 2020-01-16T00:13:25.788 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.788 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.788 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:25.788 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932210+0000", 2020-01-16T00:13:25.788 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016820563 2020-01-16T00:13:25.788 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:25.788 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:25.789 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:25.789 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.789 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.789 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:957 3.0 3:00c146d5:::smithi12313934-18:head [write 510946~632952] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:25.789 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.033290+0000", 2020-01-16T00:13:25.789 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 47.679927933000002, 2020-01-16T00:13:25.789 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016764102999999999, 2020-01-16T00:13:25.790 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:25.790 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:25.790 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:25.790 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:25.790 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:25.790 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 957 2020-01-16T00:13:25.790 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.791 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:25.791 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.791 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:25.791 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033290+0000", 2020-01-16T00:13:25.791 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:25.791 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.791 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.791 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:25.792 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033290+0000", 2020-01-16T00:13:25.792 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999952 2020-01-16T00:13:25.792 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.792 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.792 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:25.792 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033285+0000", 2020-01-16T00:13:25.792 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000405917 2020-01-16T00:13:25.792 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.793 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.793 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:25.793 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033691+0000", 2020-01-16T00:13:25.793 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6585999999999997e-05 2020-01-16T00:13:25.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.793 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.793 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:25.794 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033738+0000", 2020-01-16T00:13:25.794 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8374000000000001e-05 2020-01-16T00:13:25.794 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.794 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.794 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.794 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033756+0000", 2020-01-16T00:13:25.794 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013002900000000001 2020-01-16T00:13:25.794 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.795 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.795 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.795 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033886+0000", 2020-01-16T00:13:25.795 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035882600000000002 2020-01-16T00:13:25.795 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.795 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.795 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:25.795 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.034245+0000", 2020-01-16T00:13:25.796 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0072798489999999997 2020-01-16T00:13:25.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.796 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.796 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.796 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.041525+0000", 2020-01-16T00:13:25.796 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000140488 2020-01-16T00:13:25.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.797 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.797 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:25.797 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.041665+0000", 2020-01-16T00:13:25.797 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000233766 2020-01-16T00:13:25.797 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.797 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.797 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:25.797 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.041899+0000", 2020-01-16T00:13:25.798 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0035766539999999999 2020-01-16T00:13:25.798 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.798 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.798 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:25.798 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.045475+0000", 2020-01-16T00:13:25.798 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013333150000000001 2020-01-16T00:13:25.798 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.799 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.799 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.799 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.046809+0000", 2020-01-16T00:13:25.799 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031447860000000001 2020-01-16T00:13:25.799 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.799 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.799 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.799 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.049954+0000", 2020-01-16T00:13:25.800 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2318000000000003e-05 2020-01-16T00:13:25.800 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.800 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.800 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:25.800 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.050006+0000", 2020-01-16T00:13:25.800 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7972000000000001e-05 2020-01-16T00:13:25.800 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.800 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.801 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:25.801 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.050054+0000", 2020-01-16T00:13:25.801 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016764102999999999 2020-01-16T00:13:25.801 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:25.801 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:25.801 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:25.801 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.802 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.802 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1168 3.0 3:03b92cf0:::smithi12313934-26:head [write 2905319~482223] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:13:25.802 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:41.672662+0000", 2020-01-16T00:13:25.802 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 44.040556086999999, 2020-01-16T00:13:25.802 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015446582, 2020-01-16T00:13:25.802 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:25.802 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:25.802 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:25.803 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:25.803 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:25.803 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1168 2020-01-16T00:13:25.803 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.803 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:25.803 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.803 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:25.803 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672662+0000", 2020-01-16T00:13:25.804 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:25.804 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.804 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.804 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:25.804 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672662+0000", 2020-01-16T00:13:25.804 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:25.804 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.805 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.805 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:25.805 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672660+0000", 2020-01-16T00:13:25.805 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013924799999999999 2020-01-16T00:13:25.805 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.805 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.805 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:25.805 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672799+0000", 2020-01-16T00:13:25.806 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1304000000000001e-05 2020-01-16T00:13:25.806 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.806 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.806 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:25.806 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672810+0000", 2020-01-16T00:13:25.806 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2379999999999999e-06 2020-01-16T00:13:25.806 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.806 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.807 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.807 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672815+0000", 2020-01-16T00:13:25.807 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9329000000000002e-05 2020-01-16T00:13:25.807 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.807 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.807 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.807 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672854+0000", 2020-01-16T00:13:25.808 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010573939999999999 2020-01-16T00:13:25.808 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.808 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.808 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:25.808 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.673911+0000", 2020-01-16T00:13:25.808 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.01042181 2020-01-16T00:13:25.808 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.808 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.809 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.809 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.684333+0000", 2020-01-16T00:13:25.809 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012700499999999999 2020-01-16T00:13:25.809 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.809 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.809 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:25.809 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.684460+0000", 2020-01-16T00:13:25.809 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019923299999999999 2020-01-16T00:13:25.810 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.810 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.810 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:25.810 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.684659+0000", 2020-01-16T00:13:25.810 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020339300000000002 2020-01-16T00:13:25.810 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.810 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.811 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.811 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.686693+0000", 2020-01-16T00:13:25.811 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023967899999999999 2020-01-16T00:13:25.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.811 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.811 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:25.811 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.686933+0000", 2020-01-16T00:13:25.811 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010489169999999999 2020-01-16T00:13:25.812 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.812 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.812 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.812 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.687982+0000", 2020-01-16T00:13:25.812 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3857999999999997e-05 2020-01-16T00:13:25.812 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.812 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.812 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:25.813 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.688056+0000", 2020-01-16T00:13:25.813 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2324999999999998e-05 2020-01-16T00:13:25.813 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.813 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.813 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:25.813 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.688108+0000", 2020-01-16T00:13:25.813 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015446582 2020-01-16T00:13:25.814 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:25.814 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:25.814 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:25.814 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.814 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.814 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1169 3.0 3:03b92cf0:::smithi12313934-26:head [setxattr _header (56) in=7b,truncate 3798349] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:13:25.814 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:41.672823+0000", 2020-01-16T00:13:25.815 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 44.040395179999997, 2020-01-16T00:13:25.815 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015457745, 2020-01-16T00:13:25.815 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:25.815 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:25.815 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:25.815 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:25.815 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:25.815 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1169 2020-01-16T00:13:25.816 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.816 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:25.816 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.816 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:25.816 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672823+0000", 2020-01-16T00:13:25.816 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:25.816 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.816 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.817 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:25.817 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672823+0000", 2020-01-16T00:13:25.817 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:25.817 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.817 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.817 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:25.817 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672822+0000", 2020-01-16T00:13:25.818 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.685e-06 2020-01-16T00:13:25.818 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.818 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.818 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:25.818 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672827+0000", 2020-01-16T00:13:25.818 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.485e-06 2020-01-16T00:13:25.818 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.818 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.819 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:25.819 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672832+0000", 2020-01-16T00:13:25.819 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6440000000000002e-06 2020-01-16T00:13:25.819 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.819 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.819 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.819 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672836+0000", 2020-01-16T00:13:25.819 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00112415 2020-01-16T00:13:25.820 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.820 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.820 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.820 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.673960+0000", 2020-01-16T00:13:25.820 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5918999999999998e-05 2020-01-16T00:13:25.820 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.821 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.821 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:25.821 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.674016+0000", 2020-01-16T00:13:25.821 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011614579999999999 2020-01-16T00:13:25.821 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.822 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.822 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.822 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.685630+0000", 2020-01-16T00:13:25.822 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011684399999999999 2020-01-16T00:13:25.822 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.822 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.823 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:25.823 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.685747+0000", 2020-01-16T00:13:25.823 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020774199999999999 2020-01-16T00:13:25.823 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.823 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.823 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:25.823 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.685955+0000", 2020-01-16T00:13:25.824 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00095868400000000001 2020-01-16T00:13:25.824 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.824 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.824 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.824 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.686914+0000", 2020-01-16T00:13:25.824 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5278000000000001e-05 2020-01-16T00:13:25.824 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.824 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.825 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:25.825 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.686939+0000", 2020-01-16T00:13:25.825 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001227611 2020-01-16T00:13:25.825 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.825 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.825 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.826 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.688166+0000", 2020-01-16T00:13:25.826 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1724999999999997e-05 2020-01-16T00:13:25.826 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.826 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.826 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:25.826 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.688218+0000", 2020-01-16T00:13:25.827 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2057000000000006e-05 2020-01-16T00:13:25.827 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.827 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.827 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:25.827 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.688280+0000", 2020-01-16T00:13:25.827 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015457745 2020-01-16T00:13:25.827 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:25.828 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:25.828 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:25.828 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.828 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.828 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1170 3.0 3:03b92cf0:::smithi12313934-26:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-01-16T00:13:25.828 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:41.672848+0000", 2020-01-16T00:13:25.828 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 44.040369611999999, 2020-01-16T00:13:25.828 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015733389, 2020-01-16T00:13:25.829 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:25.829 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:25.829 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:25.829 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:25.829 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:25.829 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1170 2020-01-16T00:13:25.829 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.830 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:25.830 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.830 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:25.830 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672848+0000", 2020-01-16T00:13:25.830 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:25.830 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.830 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.830 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:25.831 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672848+0000", 2020-01-16T00:13:25.831 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:25.831 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.831 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.831 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:25.831 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672847+0000", 2020-01-16T00:13:25.831 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.303e-06 2020-01-16T00:13:25.832 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.832 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.832 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:25.832 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672851+0000", 2020-01-16T00:13:25.832 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4970000000000001e-06 2020-01-16T00:13:25.832 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.832 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.832 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:25.833 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672856+0000", 2020-01-16T00:13:25.833 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8870000000000002e-06 2020-01-16T00:13:25.833 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.833 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.833 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.833 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.672860+0000", 2020-01-16T00:13:25.833 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0011720140000000001 2020-01-16T00:13:25.834 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.834 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.834 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.834 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.674032+0000", 2020-01-16T00:13:25.834 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2577000000000003e-05 2020-01-16T00:13:25.834 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.834 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.835 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:25.835 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.674065+0000", 2020-01-16T00:13:25.835 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012264538 2020-01-16T00:13:25.835 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.835 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.835 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.835 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.686329+0000", 2020-01-16T00:13:25.835 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000106715 2020-01-16T00:13:25.836 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.836 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.836 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:25.836 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.686436+0000", 2020-01-16T00:13:25.836 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018880800000000001 2020-01-16T00:13:25.836 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.836 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.837 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.837 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.688324+0000", 2020-01-16T00:13:25.837 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012818500000000001 2020-01-16T00:13:25.837 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.837 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.837 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:25.837 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.688452+0000", 2020-01-16T00:13:25.838 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000129197 2020-01-16T00:13:25.838 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.838 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.838 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:25.838 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:41.688581+0000", 2020-01-16T00:13:25.838 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015733389 2020-01-16T00:13:25.838 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:25.838 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:25.839 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:25.839 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.839 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.839 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1394 3.4 3:240c7503:::smithi12313934-35:head [setxattr _header (56) in=7b,truncate 2285752] snapc 0=[] ondisk+write+known_if_redirected e28)", 2020-01-16T00:13:25.839 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:46.625161+0000", 2020-01-16T00:13:25.839 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 39.088056815000002, 2020-01-16T00:13:25.839 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015636654, 2020-01-16T00:13:25.839 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:25.840 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:25.840 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:25.840 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:25.840 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:25.840 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1394 2020-01-16T00:13:25.840 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.840 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:25.841 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.841 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:25.841 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625161+0000", 2020-01-16T00:13:25.841 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:25.841 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.841 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.841 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:25.841 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625161+0000", 2020-01-16T00:13:25.842 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:25.842 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.842 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.842 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:25.842 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625160+0000", 2020-01-16T00:13:25.842 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3960000000000002e-06 2020-01-16T00:13:25.842 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.843 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.843 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:25.843 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625164+0000", 2020-01-16T00:13:25.843 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7199999999999997e-06 2020-01-16T00:13:25.843 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.843 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.843 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:25.844 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625168+0000", 2020-01-16T00:13:25.844 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2960000000000001e-06 2020-01-16T00:13:25.844 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.844 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.844 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.844 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625171+0000", 2020-01-16T00:13:25.844 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001705972 2020-01-16T00:13:25.845 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.845 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.845 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.845 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.626877+0000", 2020-01-16T00:13:25.845 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039484300000000002 2020-01-16T00:13:25.845 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.845 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.846 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:25.846 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630825+0000", 2020-01-16T00:13:25.846 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0068775310000000001 2020-01-16T00:13:25.846 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.846 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.846 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.846 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.637703+0000", 2020-01-16T00:13:25.846 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013884100000000001 2020-01-16T00:13:25.847 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.847 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.847 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:25.847 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.637842+0000", 2020-01-16T00:13:25.847 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00024336400000000001 2020-01-16T00:13:25.847 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.847 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.847 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:13:25.848 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.638085+0000", 2020-01-16T00:13:25.848 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010183900000000001 2020-01-16T00:13:25.848 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.848 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.848 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:25.848 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.639103+0000", 2020-01-16T00:13:25.848 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012246699999999999 2020-01-16T00:13:25.849 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.849 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.849 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.849 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.639226+0000", 2020-01-16T00:13:25.849 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0014329099999999999 2020-01-16T00:13:25.849 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.849 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.849 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.850 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.640659+0000", 2020-01-16T00:13:25.850 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8109000000000002e-05 2020-01-16T00:13:25.850 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.850 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.850 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:25.850 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.640717+0000", 2020-01-16T00:13:25.850 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0784000000000004e-05 2020-01-16T00:13:25.851 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.851 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.851 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:25.851 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.640798+0000", 2020-01-16T00:13:25.851 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015636654 2020-01-16T00:13:25.851 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:25.851 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:25.852 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:25.852 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.852 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.852 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1395 3.4 3:240c7503:::smithi12313934-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e28)", 2020-01-16T00:13:25.852 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:46.625191+0000", 2020-01-16T00:13:25.852 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 39.088026605000003, 2020-01-16T00:13:25.852 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015878716000000001, 2020-01-16T00:13:25.853 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:25.853 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:25.853 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:25.853 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:25.853 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:25.853 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1395 2020-01-16T00:13:25.853 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.853 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:25.854 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.854 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:25.854 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625191+0000", 2020-01-16T00:13:25.854 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:25.854 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.854 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.854 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:25.855 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625191+0000", 2020-01-16T00:13:25.855 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:25.855 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.855 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.855 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:25.855 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625191+0000", 2020-01-16T00:13:25.855 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.1090000000000001e-06 2020-01-16T00:13:25.855 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.856 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.856 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:25.856 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625193+0000", 2020-01-16T00:13:25.856 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2730000000000008e-06 2020-01-16T00:13:25.856 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.856 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.856 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:25.856 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625202+0000", 2020-01-16T00:13:25.857 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3700000000000002e-06 2020-01-16T00:13:25.857 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.857 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.857 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.857 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625204+0000", 2020-01-16T00:13:25.857 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056573049999999996 2020-01-16T00:13:25.857 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.857 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.858 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.858 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630862+0000", 2020-01-16T00:13:25.858 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6047000000000001e-05 2020-01-16T00:13:25.858 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.858 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.858 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:25.858 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630918+0000", 2020-01-16T00:13:25.859 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0077941180000000001 2020-01-16T00:13:25.859 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.859 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.859 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.859 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.638712+0000", 2020-01-16T00:13:25.859 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118096 2020-01-16T00:13:25.859 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.859 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.860 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:25.860 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.638830+0000", 2020-01-16T00:13:25.860 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0019986740000000002 2020-01-16T00:13:25.860 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.860 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.860 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.860 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.640829+0000", 2020-01-16T00:13:25.860 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012562399999999999 2020-01-16T00:13:25.861 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.861 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.861 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:25.861 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.640954+0000", 2020-01-16T00:13:25.861 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000115536 2020-01-16T00:13:25.861 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.861 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.862 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:25.862 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.641070+0000", 2020-01-16T00:13:25.862 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015878716000000001 2020-01-16T00:13:25.862 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:25.862 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:25.862 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:25.862 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.862 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.863 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1639 3.0 3:00c146d5:::smithi12313934-18:head [write 423576~401598] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:13:25.863 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.774317+0000", 2020-01-16T00:13:25.863 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 33.938901194000003, 2020-01-16T00:13:25.863 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018305219000000001, 2020-01-16T00:13:25.863 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:25.863 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:25.863 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:25.863 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:25.864 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:25.864 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1639 2020-01-16T00:13:25.864 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.864 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:25.864 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.864 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:25.864 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774317+0000", 2020-01-16T00:13:25.864 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:25.865 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.865 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.865 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:25.865 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774317+0000", 2020-01-16T00:13:25.865 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:25.865 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.865 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.866 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:25.866 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774313+0000", 2020-01-16T00:13:25.866 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000111173 2020-01-16T00:13:25.866 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.866 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.866 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:25.867 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774425+0000", 2020-01-16T00:13:25.867 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7598999999999999e-05 2020-01-16T00:13:25.867 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.868 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.868 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:25.868 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774442+0000", 2020-01-16T00:13:25.868 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0830000000000004e-06 2020-01-16T00:13:25.868 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.868 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.869 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.869 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774450+0000", 2020-01-16T00:13:25.869 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0017886309999999999 2020-01-16T00:13:25.869 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.870 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.870 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.870 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.776239+0000", 2020-01-16T00:13:25.870 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3789e-05 2020-01-16T00:13:25.870 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.871 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.871 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op must wait for map", 2020-01-16T00:13:25.871 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.776253+0000", 2020-01-16T00:13:25.871 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095730729999999997 2020-01-16T00:13:25.872 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.872 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.872 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.872 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.785826+0000", 2020-01-16T00:13:25.872 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014220300000000001 2020-01-16T00:13:25.873 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.873 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.873 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:25.873 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.785968+0000", 2020-01-16T00:13:25.874 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023618900000000001 2020-01-16T00:13:25.874 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.874 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.874 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:25.875 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.786204+0000", 2020-01-16T00:13:25.875 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0062297389999999998 2020-01-16T00:13:25.875 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.875 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.875 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:25.876 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792434+0000", 2020-01-16T00:13:25.876 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9635999999999998e-05 2020-01-16T00:13:25.876 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.876 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.877 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.877 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792473+0000", 2020-01-16T00:13:25.877 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6456000000000002e-05 2020-01-16T00:13:25.877 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.878 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.878 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.878 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792540+0000", 2020-01-16T00:13:25.878 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9873999999999998e-05 2020-01-16T00:13:25.878 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.879 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.879 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:25.879 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792580+0000", 2020-01-16T00:13:25.879 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1924e-05 2020-01-16T00:13:25.880 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.880 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.880 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:25.880 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792622+0000", 2020-01-16T00:13:25.881 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018305219000000001 2020-01-16T00:13:25.881 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:25.881 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:25.881 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:25.882 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.882 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.882 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1890 3.0 3:0b1f384f:::smithi12313934-33:head [write 2797915~596476] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:13:25.882 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.232534+0000", 2020-01-16T00:13:25.882 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 28.480683223, 2020-01-16T00:13:25.883 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016106137, 2020-01-16T00:13:25.883 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:25.883 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:25.884 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:25.884 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:25.884 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:25.884 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1890 2020-01-16T00:13:25.885 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.885 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:25.885 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.885 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:25.885 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232534+0000", 2020-01-16T00:13:25.886 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:25.886 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.886 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.886 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:25.887 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232534+0000", 2020-01-16T00:13:25.887 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:25.887 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.887 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.888 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:25.888 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232532+0000", 2020-01-16T00:13:25.888 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00024232100000000001 2020-01-16T00:13:25.888 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.888 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.889 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:25.889 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232774+0000", 2020-01-16T00:13:25.889 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4601e-05 2020-01-16T00:13:25.889 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.890 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.890 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:25.890 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232789+0000", 2020-01-16T00:13:25.890 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8410000000000006e-06 2020-01-16T00:13:25.891 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.891 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.891 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.891 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232798+0000", 2020-01-16T00:13:25.891 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9756999999999997e-05 2020-01-16T00:13:25.892 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.892 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.892 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.892 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232857+0000", 2020-01-16T00:13:25.893 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4678000000000007e-05 2020-01-16T00:13:25.893 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.893 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.893 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:25.893 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232922+0000", 2020-01-16T00:13:25.894 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033861020000000002 2020-01-16T00:13:25.894 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.894 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.894 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.894 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.236308+0000", 2020-01-16T00:13:25.894 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000568996 2020-01-16T00:13:25.895 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.895 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.895 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:25.895 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.236877+0000", 2020-01-16T00:13:25.895 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0085444240000000001 2020-01-16T00:13:25.895 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.895 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.895 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.896 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.245422+0000", 2020-01-16T00:13:25.896 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5595999999999998e-05 2020-01-16T00:13:25.896 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.896 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.896 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:25.896 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.245507+0000", 2020-01-16T00:13:25.896 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016984300000000001 2020-01-16T00:13:25.896 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.897 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.897 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:25.897 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.245677+0000", 2020-01-16T00:13:25.897 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001649495 2020-01-16T00:13:25.897 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.897 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.897 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:25.898 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247326+0000", 2020-01-16T00:13:25.898 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023308399999999999 2020-01-16T00:13:25.898 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.898 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.898 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.898 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247560+0000", 2020-01-16T00:13:25.898 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00099501800000000007 2020-01-16T00:13:25.898 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.899 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.899 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.899 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248555+0000", 2020-01-16T00:13:25.899 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5936999999999998e-05 2020-01-16T00:13:25.899 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.899 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.899 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:25.899 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248591+0000", 2020-01-16T00:13:25.900 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0108999999999997e-05 2020-01-16T00:13:25.900 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.900 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.900 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:25.900 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248641+0000", 2020-01-16T00:13:25.900 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016106137 2020-01-16T00:13:25.900 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:25.901 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:25.901 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:25.901 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.901 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.901 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1891 3.0 3:0b1f384f:::smithi12313934-33:head [setxattr _header (56) in=7b,truncate 3395014] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:13:25.901 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.232813+0000", 2020-01-16T00:13:25.901 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 28.480404420999999, 2020-01-16T00:13:25.901 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015999295, 2020-01-16T00:13:25.902 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:25.902 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:25.902 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:25.902 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:25.902 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:25.902 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1891 2020-01-16T00:13:25.902 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.903 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:25.903 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.903 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:25.903 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232813+0000", 2020-01-16T00:13:25.903 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:25.903 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.903 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.903 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:25.904 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232813+0000", 2020-01-16T00:13:25.904 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:25.904 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.904 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.904 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:25.904 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232812+0000", 2020-01-16T00:13:25.904 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.576e-06 2020-01-16T00:13:25.905 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.905 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.905 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:25.905 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232820+0000", 2020-01-16T00:13:25.905 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3079999999999997e-06 2020-01-16T00:13:25.905 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.906 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.906 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:25.906 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232829+0000", 2020-01-16T00:13:25.906 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7119999999999997e-06 2020-01-16T00:13:25.906 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.906 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.906 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.907 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232835+0000", 2020-01-16T00:13:25.907 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013438400000000001 2020-01-16T00:13:25.907 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.907 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.907 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.907 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232969+0000", 2020-01-16T00:13:25.907 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0562000000000001e-05 2020-01-16T00:13:25.907 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.908 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.908 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:25.908 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.233029+0000", 2020-01-16T00:13:25.908 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039354239999999999 2020-01-16T00:13:25.908 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.908 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.908 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.909 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.236965+0000", 2020-01-16T00:13:25.909 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0482999999999998e-05 2020-01-16T00:13:25.909 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.909 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.909 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:25.909 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.237015+0000", 2020-01-16T00:13:25.909 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0096086209999999995 2020-01-16T00:13:25.909 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.910 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.910 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.910 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.246624+0000", 2020-01-16T00:13:25.910 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6150999999999995e-05 2020-01-16T00:13:25.910 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.910 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.910 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:25.910 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.246720+0000", 2020-01-16T00:13:25.911 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015032899999999999 2020-01-16T00:13:25.911 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.911 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.911 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:25.911 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.246870+0000", 2020-01-16T00:13:25.911 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00064760700000000002 2020-01-16T00:13:25.911 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.912 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.912 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:25.912 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247518+0000", 2020-01-16T00:13:25.912 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000198531 2020-01-16T00:13:25.912 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.912 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.912 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.912 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247717+0000", 2020-01-16T00:13:25.913 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010189439999999999 2020-01-16T00:13:25.913 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.913 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.913 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.913 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248736+0000", 2020-01-16T00:13:25.913 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4863000000000001e-05 2020-01-16T00:13:25.913 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.914 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.914 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:25.914 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248770+0000", 2020-01-16T00:13:25.914 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2153000000000001e-05 2020-01-16T00:13:25.914 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.914 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.914 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:25.915 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248813+0000", 2020-01-16T00:13:25.915 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015999295 2020-01-16T00:13:25.915 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:25.915 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:25.915 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:25.915 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.915 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.915 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1892 3.0 3:0b1f384f:::smithi12313934-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e31)", 2020-01-16T00:13:25.916 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.232856+0000", 2020-01-16T00:13:25.916 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 28.480362197000002, 2020-01-16T00:13:25.916 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016171544, 2020-01-16T00:13:25.916 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:25.916 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:25.916 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:25.916 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:25.917 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:25.917 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1892 2020-01-16T00:13:25.917 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.917 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:25.917 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.917 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:25.917 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232856+0000", 2020-01-16T00:13:25.917 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:25.918 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.918 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.918 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:25.918 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232856+0000", 2020-01-16T00:13:25.918 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:25.918 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.919 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.919 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:25.919 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232855+0000", 2020-01-16T00:13:25.919 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8640000000000004e-06 2020-01-16T00:13:25.920 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.920 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.920 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:25.920 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232860+0000", 2020-01-16T00:13:25.921 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0000000000000001e-05 2020-01-16T00:13:25.921 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.921 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.921 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:25.921 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232870+0000", 2020-01-16T00:13:25.922 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7420000000000003e-06 2020-01-16T00:13:25.922 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.922 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.922 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.922 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232875+0000", 2020-01-16T00:13:25.923 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017933200000000001 2020-01-16T00:13:25.923 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.923 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.923 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.924 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.233055+0000", 2020-01-16T00:13:25.924 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6269e-05 2020-01-16T00:13:25.924 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.924 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.924 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:25.925 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.233101+0000", 2020-01-16T00:13:25.925 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039780090000000002 2020-01-16T00:13:25.925 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.925 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.926 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.926 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.237079+0000", 2020-01-16T00:13:25.926 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3677000000000001e-05 2020-01-16T00:13:25.926 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.927 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.927 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:25.927 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.237133+0000", 2020-01-16T00:13:25.927 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010036671 2020-01-16T00:13:25.927 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.928 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.928 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.928 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247169+0000", 2020-01-16T00:13:25.928 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010280699999999999 2020-01-16T00:13:25.928 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.928 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.929 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:25.929 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247272+0000", 2020-01-16T00:13:25.929 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001579708 2020-01-16T00:13:25.929 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.929 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.929 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.929 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248852+0000", 2020-01-16T00:13:25.930 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9704999999999996e-05 2020-01-16T00:13:25.930 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.930 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.930 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:25.930 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248942+0000", 2020-01-16T00:13:25.930 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5557999999999996e-05 2020-01-16T00:13:25.930 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.930 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.931 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:25.931 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.249027+0000", 2020-01-16T00:13:25.931 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016171544 2020-01-16T00:13:25.931 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:25.931 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:25.931 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:25.931 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.931 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.932 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1950 3.0 3:0b1f384f:::smithi12313934-33:head [write 1889359~541737] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-01-16T00:13:25.932 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.922819+0000", 2020-01-16T00:13:25.932 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 27.790398832000001, 2020-01-16T00:13:25.932 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017502846999999998, 2020-01-16T00:13:25.932 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:25.932 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:25.932 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:25.933 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:25.933 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:25.933 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1950 2020-01-16T00:13:25.933 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.933 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:25.933 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.933 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:25.933 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922819+0000", 2020-01-16T00:13:25.934 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:25.934 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.934 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.934 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:25.934 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922819+0000", 2020-01-16T00:13:25.934 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:25.934 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.935 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.935 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:25.935 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922818+0000", 2020-01-16T00:13:25.935 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000146408 2020-01-16T00:13:25.935 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.935 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.935 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:25.935 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922964+0000", 2020-01-16T00:13:25.936 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4720000000000001e-06 2020-01-16T00:13:25.936 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.936 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.936 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:25.936 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922974+0000", 2020-01-16T00:13:25.936 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0720000000000001e-06 2020-01-16T00:13:25.936 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.937 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.937 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.937 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922978+0000", 2020-01-16T00:13:25.937 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6152000000000001e-05 2020-01-16T00:13:25.937 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.937 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.937 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.937 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.923024+0000", 2020-01-16T00:13:25.938 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00087368199999999997 2020-01-16T00:13:25.938 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.938 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.938 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:25.938 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.923897+0000", 2020-01-16T00:13:25.938 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011944435999999999 2020-01-16T00:13:25.938 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.939 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.939 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.939 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.935842+0000", 2020-01-16T00:13:25.939 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000144146 2020-01-16T00:13:25.939 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.939 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.939 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:25.939 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.935986+0000", 2020-01-16T00:13:25.940 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019775500000000001 2020-01-16T00:13:25.940 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.940 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.940 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:25.940 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.936184+0000", 2020-01-16T00:13:25.940 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002539342 2020-01-16T00:13:25.940 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.940 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.941 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.941 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938723+0000", 2020-01-16T00:13:25.941 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014193699999999999 2020-01-16T00:13:25.941 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.941 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.941 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.941 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938865+0000", 2020-01-16T00:13:25.942 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013829020000000001 2020-01-16T00:13:25.942 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.942 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.942 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:25.942 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.940248+0000", 2020-01-16T00:13:25.942 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2558000000000001e-05 2020-01-16T00:13:25.942 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.942 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.943 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:25.943 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.940291+0000", 2020-01-16T00:13:25.943 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1217999999999998e-05 2020-01-16T00:13:25.943 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.943 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.943 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:25.943 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.940322+0000", 2020-01-16T00:13:25.943 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017502846999999998 2020-01-16T00:13:25.944 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:25.944 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:25.944 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:25.944 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.944 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.944 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1951 3.0 3:0b1f384f:::smithi12313934-33:head [write 2955792~541902] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-01-16T00:13:25.944 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.929019+0000", 2020-01-16T00:13:25.945 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 27.784198682, 2020-01-16T00:13:25.945 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021625710999999999, 2020-01-16T00:13:25.945 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:25.945 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:25.945 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:25.945 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:25.946 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:25.946 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1951 2020-01-16T00:13:25.946 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.946 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:25.946 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.946 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:25.946 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929019+0000", 2020-01-16T00:13:25.947 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:25.947 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.947 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.947 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:25.947 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929019+0000", 2020-01-16T00:13:25.947 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999989 2020-01-16T00:13:25.947 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.947 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.948 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:25.948 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929008+0000", 2020-01-16T00:13:25.948 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000212513 2020-01-16T00:13:25.948 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.948 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.948 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:25.948 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929221+0000", 2020-01-16T00:13:25.949 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3229e-05 2020-01-16T00:13:25.949 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.949 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.949 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:25.949 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929234+0000", 2020-01-16T00:13:25.949 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9170000000000003e-06 2020-01-16T00:13:25.949 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.950 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.950 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.950 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929240+0000", 2020-01-16T00:13:25.950 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4045000000000002e-05 2020-01-16T00:13:25.950 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.950 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.950 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.950 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929294+0000", 2020-01-16T00:13:25.951 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9797999999999996e-05 2020-01-16T00:13:25.951 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.951 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.951 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:25.951 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929384+0000", 2020-01-16T00:13:25.951 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083450729999999997 2020-01-16T00:13:25.951 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.952 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.952 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.952 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.937729+0000", 2020-01-16T00:13:25.952 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043251699999999999 2020-01-16T00:13:25.952 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.952 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:25.953 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938161+0000", 2020-01-16T00:13:25.953 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095115489999999994 2020-01-16T00:13:25.953 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.953 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.953 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.947673+0000", 2020-01-16T00:13:25.954 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012395099999999999 2020-01-16T00:13:25.954 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.954 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.954 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:25.954 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.947797+0000", 2020-01-16T00:13:25.954 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00024849200000000001 2020-01-16T00:13:25.954 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.954 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.955 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:25.955 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.948045+0000", 2020-01-16T00:13:25.955 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0025081050000000001 2020-01-16T00:13:25.955 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.955 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.955 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.956 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950553+0000", 2020-01-16T00:13:25.956 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9760000000000003e-06 2020-01-16T00:13:25.956 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.956 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.956 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:25.956 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950558+0000", 2020-01-16T00:13:25.956 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3286999999999999e-05 2020-01-16T00:13:25.957 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.957 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.957 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.957 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950581+0000", 2020-01-16T00:13:25.957 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2759000000000003e-05 2020-01-16T00:13:25.957 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.957 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.957 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:25.958 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950614+0000", 2020-01-16T00:13:25.958 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0525999999999999e-05 2020-01-16T00:13:25.958 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.958 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.958 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:25.958 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950645+0000", 2020-01-16T00:13:25.958 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021625710999999999 2020-01-16T00:13:25.959 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:25.959 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:25.959 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:25.959 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.959 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.959 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1952 3.0 3:0b1f384f:::smithi12313934-33:head [setxattr _header (56) in=7b,truncate 3563084] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-01-16T00:13:25.959 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:13:25.959 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 27.783969344999999, 2020-01-16T00:13:25.960 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021534685000000001, 2020-01-16T00:13:25.960 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:25.960 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:25.960 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:25.960 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:25.960 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:25.960 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1952 2020-01-16T00:13:25.960 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.961 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:25.961 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.961 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:25.961 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:13:25.961 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:25.961 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.961 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.962 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:25.962 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:13:25.962 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:25.962 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.962 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.962 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:25.962 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:13:25.962 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6670000000000004e-06 2020-01-16T00:13:25.963 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.963 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.963 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:25.963 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929252+0000", 2020-01-16T00:13:25.963 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9380000000000002e-06 2020-01-16T00:13:25.963 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.963 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.963 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:25.964 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929256+0000", 2020-01-16T00:13:25.964 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.712e-06 2020-01-16T00:13:25.964 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.964 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.964 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.964 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929264+0000", 2020-01-16T00:13:25.964 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018030300000000001 2020-01-16T00:13:25.965 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.965 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.965 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.965 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929444+0000", 2020-01-16T00:13:25.965 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7562000000000002e-05 2020-01-16T00:13:25.965 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.965 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.966 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:25.966 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929502+0000", 2020-01-16T00:13:25.966 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0087297859999999998 2020-01-16T00:13:25.966 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.966 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.966 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.966 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938232+0000", 2020-01-16T00:13:25.967 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0452999999999996e-05 2020-01-16T00:13:25.967 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.967 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.967 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:25.967 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938322+0000", 2020-01-16T00:13:25.967 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011061728 2020-01-16T00:13:25.967 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.967 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.968 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.968 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.949384+0000", 2020-01-16T00:13:25.968 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000117828 2020-01-16T00:13:25.968 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.968 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.968 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:25.969 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.949502+0000", 2020-01-16T00:13:25.969 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022468499999999999 2020-01-16T00:13:25.969 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.969 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.969 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:25.969 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.949726+0000", 2020-01-16T00:13:25.969 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00083532099999999998 2020-01-16T00:13:25.969 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.970 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.970 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:25.970 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950562+0000", 2020-01-16T00:13:25.970 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013017700000000001 2020-01-16T00:13:25.970 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.970 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.970 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.971 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950692+0000", 2020-01-16T00:13:25.971 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9813000000000001e-05 2020-01-16T00:13:25.971 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.971 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.971 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.971 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950722+0000", 2020-01-16T00:13:25.971 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0592000000000001e-05 2020-01-16T00:13:25.972 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.972 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.972 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:25.972 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950752+0000", 2020-01-16T00:13:25.972 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0920000000000002e-05 2020-01-16T00:13:25.972 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.972 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.972 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:25.973 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950783+0000", 2020-01-16T00:13:25.973 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021534685000000001 2020-01-16T00:13:25.973 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:25.973 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:25.973 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:25.973 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.973 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.974 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1953 3.0 3:0b1f384f:::smithi12313934-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e32)", 2020-01-16T00:13:25.974 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:13:25.974 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 27.783936354000002, 2020-01-16T00:13:25.974 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021667537000000001, 2020-01-16T00:13:25.974 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:25.974 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:25.974 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:25.975 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:25.975 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:25.975 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1953 2020-01-16T00:13:25.975 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.975 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:25.975 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.975 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:25.975 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:13:25.976 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:25.976 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.976 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.976 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:25.976 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:13:25.976 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:25.976 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.977 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.977 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:25.977 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:13:25.977 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7030000000000002e-06 2020-01-16T00:13:25.977 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.977 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.977 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:25.977 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929284+0000", 2020-01-16T00:13:25.978 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0909999999999993e-06 2020-01-16T00:13:25.978 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.978 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.978 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:25.978 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929292+0000", 2020-01-16T00:13:25.978 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8359999999999999e-06 2020-01-16T00:13:25.978 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.979 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.979 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.979 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929297+0000", 2020-01-16T00:13:25.979 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023538199999999999 2020-01-16T00:13:25.979 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.979 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.979 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.979 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929532+0000", 2020-01-16T00:13:25.980 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1173000000000004e-05 2020-01-16T00:13:25.980 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.980 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.980 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:25.980 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929603+0000", 2020-01-16T00:13:25.980 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0087813509999999997 2020-01-16T00:13:25.980 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.980 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.981 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.981 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938384+0000", 2020-01-16T00:13:25.981 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9752000000000006e-05 2020-01-16T00:13:25.981 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.981 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.981 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:25.981 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938454+0000", 2020-01-16T00:13:25.982 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011707851999999999 2020-01-16T00:13:25.982 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.982 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.982 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.982 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950162+0000", 2020-01-16T00:13:25.982 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011060099999999999 2020-01-16T00:13:25.982 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.982 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.983 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:25.983 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950273+0000", 2020-01-16T00:13:25.983 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00053989700000000001 2020-01-16T00:13:25.983 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.983 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.983 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.983 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950813+0000", 2020-01-16T00:13:25.984 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8128999999999999e-05 2020-01-16T00:13:25.984 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.984 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.984 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:25.984 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950871+0000", 2020-01-16T00:13:25.984 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8242000000000001e-05 2020-01-16T00:13:25.985 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.985 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.985 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:25.985 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950949+0000", 2020-01-16T00:13:25.985 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021667537000000001 2020-01-16T00:13:25.985 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:25.985 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:25.985 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:25.986 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.986 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.986 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2555 3.0 3:0b70bb20:::smithi12313934-46:head [setxattr _header (56) in=7b,truncate 2991679] snapc 0=[] ondisk+write+known_if_redirected e37)", 2020-01-16T00:13:25.986 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.573432+0000", 2020-01-16T00:13:25.986 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 13.139785702999999, 2020-01-16T00:13:25.986 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019893926999999999, 2020-01-16T00:13:25.986 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:25.987 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:25.987 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:25.987 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:25.987 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:25.987 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2555 2020-01-16T00:13:25.987 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.987 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:25.987 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.988 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:25.988 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573432+0000", 2020-01-16T00:13:25.988 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:25.988 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.988 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.988 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:25.988 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573432+0000", 2020-01-16T00:13:25.989 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:25.989 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.989 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.989 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:25.989 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573432+0000", 2020-01-16T00:13:25.989 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4290000000000001e-06 2020-01-16T00:13:25.989 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.989 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.990 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:25.990 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573435+0000", 2020-01-16T00:13:25.990 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1300000000000001e-06 2020-01-16T00:13:25.990 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.990 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.990 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:25.990 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573438+0000", 2020-01-16T00:13:25.990 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9399999999999996e-06 2020-01-16T00:13:25.991 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.991 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.991 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:25.991 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573445+0000", 2020-01-16T00:13:25.991 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013947269999999999 2020-01-16T00:13:25.991 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.991 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.992 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.992 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.574840+0000", 2020-01-16T00:13:25.992 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003850058 2020-01-16T00:13:25.992 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.992 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.992 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:25.992 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578690+0000", 2020-01-16T00:13:25.992 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012896489000000001 2020-01-16T00:13:25.993 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.993 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.993 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:25.993 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591586+0000", 2020-01-16T00:13:25.993 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4757000000000001e-05 2020-01-16T00:13:25.993 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.993 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.993 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:25.994 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591681+0000", 2020-01-16T00:13:25.994 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000191535 2020-01-16T00:13:25.994 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.994 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.994 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:25.994 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591873+0000", 2020-01-16T00:13:25.994 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00069888899999999996 2020-01-16T00:13:25.995 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.995 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.995 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:25.995 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.592571+0000", 2020-01-16T00:13:25.995 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00029525900000000003 2020-01-16T00:13:25.995 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.995 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.995 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.996 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.592867+0000", 2020-01-16T00:13:25.996 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035243500000000001 2020-01-16T00:13:25.996 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.996 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.996 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:25.996 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593219+0000", 2020-01-16T00:13:25.996 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0954e-05 2020-01-16T00:13:25.996 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.997 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.997 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:25.997 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593270+0000", 2020-01-16T00:13:25.997 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5813000000000002e-05 2020-01-16T00:13:25.997 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.997 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.997 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:25.998 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593326+0000", 2020-01-16T00:13:25.998 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019893926999999999 2020-01-16T00:13:25.998 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:25.998 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:25.998 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:25.998 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:25.998 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:25.998 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2556 3.0 3:0b70bb20:::smithi12313934-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e37)", 2020-01-16T00:13:25.999 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.573472+0000", 2020-01-16T00:13:25.999 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 13.139746132999999, 2020-01-16T00:13:25.999 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020070068999999999, 2020-01-16T00:13:25.999 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:25.999 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:25.999 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:25.999 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:26.000 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:26.000 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2556 2020-01-16T00:13:26.000 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.000 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:26.000 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.000 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:26.000 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573472+0000", 2020-01-16T00:13:26.000 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:26.001 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.001 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.001 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:26.001 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573472+0000", 2020-01-16T00:13:26.001 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:26.001 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.001 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.002 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:26.002 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573471+0000", 2020-01-16T00:13:26.002 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4449999999999999e-06 2020-01-16T00:13:26.002 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.002 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.002 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:26.002 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573479+0000", 2020-01-16T00:13:26.002 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9789999999999998e-06 2020-01-16T00:13:26.003 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.003 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.003 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:26.003 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573485+0000", 2020-01-16T00:13:26.003 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3270000000000001e-06 2020-01-16T00:13:26.003 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.003 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.004 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:26.004 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573488+0000", 2020-01-16T00:13:26.004 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052586849999999999 2020-01-16T00:13:26.004 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.004 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.004 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:26.004 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578747+0000", 2020-01-16T00:13:26.004 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5667000000000007e-05 2020-01-16T00:13:26.005 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.005 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.005 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:26.005 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578812+0000", 2020-01-16T00:13:26.005 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013599539000000001 2020-01-16T00:13:26.005 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.005 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.006 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:26.006 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.592412+0000", 2020-01-16T00:13:26.006 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6018000000000001e-05 2020-01-16T00:13:26.006 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.006 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.006 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:26.006 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.592508+0000", 2020-01-16T00:13:26.006 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00084582199999999998 2020-01-16T00:13:26.007 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.007 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.007 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:26.007 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593354+0000", 2020-01-16T00:13:26.007 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4984999999999999e-05 2020-01-16T00:13:26.007 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.007 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.007 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:26.008 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593449+0000", 2020-01-16T00:13:26.008 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3104999999999997e-05 2020-01-16T00:13:26.008 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.008 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.008 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:26.008 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593542+0000", 2020-01-16T00:13:26.008 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020070068999999999 2020-01-16T00:13:26.009 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.009 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:26.009 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.009 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.009 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:26.009 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:26.011 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-01-16T00:13:26.011 INFO:tasks.rados.rados.0.smithi123.stdout:1239: expect (ObjNum 369 snap 0 seq_num 369) 2020-01-16T00:13:26.011 INFO:tasks.rados.rados.0.smithi123.stdout:1237: finishing write tid 2 to smithi12313934-3 2020-01-16T00:13:26.012 INFO:tasks.rados.rados.0.smithi123.stdout:1237: finishing write tid 4 to smithi12313934-3 2020-01-16T00:13:26.012 INFO:tasks.rados.rados.0.smithi123.stdout:1237: finishing write tid 5 to smithi12313934-3 2020-01-16T00:13:26.012 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 3 v 82 (ObjNum 377 snap 0 seq_num 377) dirty exists 2020-01-16T00:13:26.012 INFO:tasks.rados.rados.0.smithi123.stdout:1237: left oid 3 (ObjNum 377 snap 0 seq_num 377) 2020-01-16T00:13:26.012 INFO:tasks.rados.rados.0.smithi123.stdout:1241: expect (ObjNum 334 snap 0 seq_num 334) 2020-01-16T00:13:26.012 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:26.006+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:26.073 INFO:tasks.rados.rados.0.smithi123.stdout:1238: expect (ObjNum 366 snap 0 seq_num 366) 2020-01-16T00:13:26.112 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:26.106+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:26.155 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:26.156 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:26.156 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:13:26.156 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:26.164 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-01-16T00:13:26.198 INFO:tasks.rados.rados.0.smithi123.stdout:1242: expect (ObjNum 362 snap 0 seq_num 362) 2020-01-16T00:13:26.214 INFO:tasks.daemonwatchdog.daemon_watchdog:OSDThrasher failed 2020-01-16T00:13:26.214 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2020-01-16T00:13:26.214 ERROR:tasks.daemonwatchdog.daemon_watchdog:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage-testing-2020-01-15-1209/qa/tasks/daemonwatchdog.py", line 37, in _run self.watch() File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage-testing-2020-01-15-1209/qa/tasks/daemonwatchdog.py", line 113, in watch self.bark() File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage-testing-2020-01-15-1209/qa/tasks/daemonwatchdog.py", line 53, in bark for mount in self.ctx.mounts.values(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/config.py", line 253, in __getattr__ raise AttributeError(name) AttributeError: mounts 2020-01-16T00:13:26.240 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:26.238+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:26.300 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:26.301 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:26.301 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:13:26.301 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:13:26.301 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:26.309 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-01-16T00:13:26.352 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:26.350+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:26.363 INFO:tasks.rados.rados.0.smithi123.stdout:1244: expect (ObjNum 298 snap 0 seq_num 298) 2020-01-16T00:13:26.417 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:26.412+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:26.460 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:26.460 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:13:26.460 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:13:26.460 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:13:26.460 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.460 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.5.13:40 2.0 e24/19 2:20e40cec:::smithi12313934-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 24'17, mlcod=23'15)", 2020-01-16T00:13:26.461 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.849607+0000", 2020-01-16T00:13:26.461 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 48.605478234000003, 2020-01-16T00:13:26.461 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083979039999999994, 2020-01-16T00:13:26.461 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:26.461 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:26.461 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:26.461 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.462 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:26.462 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.849607+0000", 2020-01-16T00:13:26.462 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:26.462 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.462 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.462 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:26.462 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.849607+0000", 2020-01-16T00:13:26.462 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:26.463 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.463 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.463 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:26.463 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.849605+0000", 2020-01-16T00:13:26.463 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0037534560000000001 2020-01-16T00:13:26.463 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.463 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.464 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:26.464 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.853358+0000", 2020-01-16T00:13:26.464 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9181000000000002e-05 2020-01-16T00:13:26.464 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.464 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.464 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:26.464 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.853377+0000", 2020-01-16T00:13:26.464 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0040000000000002e-06 2020-01-16T00:13:26.465 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.465 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.465 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:26.465 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.853384+0000", 2020-01-16T00:13:26.465 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2583000000000001e-05 2020-01-16T00:13:26.465 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.466 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.466 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:26.466 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.853437+0000", 2020-01-16T00:13:26.466 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8212999999999999e-05 2020-01-16T00:13:26.466 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.466 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.466 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:26.467 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.853465+0000", 2020-01-16T00:13:26.467 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0044847589999999996 2020-01-16T00:13:26.467 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.467 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.467 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:26.467 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.857950+0000", 2020-01-16T00:13:26.467 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5309999999999997e-05 2020-01-16T00:13:26.467 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.468 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.468 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:26.468 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.858005+0000", 2020-01-16T00:13:26.468 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083979039999999994 2020-01-16T00:13:26.468 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.468 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:26.468 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.469 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.471 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.471 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.5.13:42 2.0 e24/19 2:313458df:::smithi12313934-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 24'18, mlcod=24'16)", 2020-01-16T00:13:26.471 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.940683+0000", 2020-01-16T00:13:26.472 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 48.514402394999998, 2020-01-16T00:13:26.472 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0097026960000000002, 2020-01-16T00:13:26.472 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:26.473 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:26.473 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:26.473 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.473 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:26.473 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.940683+0000", 2020-01-16T00:13:26.474 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:26.474 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.474 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.474 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:26.474 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.940683+0000", 2020-01-16T00:13:26.474 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:26.474 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.475 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.475 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:26.475 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.940680+0000", 2020-01-16T00:13:26.475 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003126775 2020-01-16T00:13:26.475 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.475 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.475 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:26.475 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.943807+0000", 2020-01-16T00:13:26.476 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5426999999999999e-05 2020-01-16T00:13:26.476 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.476 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.476 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:26.476 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.943833+0000", 2020-01-16T00:13:26.476 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7409999999999998e-06 2020-01-16T00:13:26.476 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.477 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.477 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:26.477 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.943841+0000", 2020-01-16T00:13:26.477 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011645399999999999 2020-01-16T00:13:26.477 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.477 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.477 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:26.477 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.943958+0000", 2020-01-16T00:13:26.478 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7291000000000003e-05 2020-01-16T00:13:26.478 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.478 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.478 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:26.478 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.944025+0000", 2020-01-16T00:13:26.478 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063108449999999998 2020-01-16T00:13:26.478 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.479 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.479 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:26.479 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.950336+0000", 2020-01-16T00:13:26.483 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9874999999999999e-05 2020-01-16T00:13:26.483 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.483 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.483 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:26.483 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.950386+0000", 2020-01-16T00:13:26.484 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0097026960000000002 2020-01-16T00:13:26.484 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.484 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:26.484 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.484 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.484 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.484 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1388 3.1 3:8c6a83c0:::smithi12313934-36:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e28)", 2020-01-16T00:13:26.485 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:46.605253+0000", 2020-01-16T00:13:26.485 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 39.849833044999997, 2020-01-16T00:13:26.485 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0080920829999999999, 2020-01-16T00:13:26.485 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:26.486 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:26.486 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:26.486 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:26.486 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:26.486 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1388 2020-01-16T00:13:26.486 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.487 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:26.487 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.487 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:26.487 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605253+0000", 2020-01-16T00:13:26.487 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:26.487 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.487 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.487 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:26.488 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605253+0000", 2020-01-16T00:13:26.488 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:26.488 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.488 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.488 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:26.488 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605252+0000", 2020-01-16T00:13:26.488 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6559999999999997e-06 2020-01-16T00:13:26.489 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.489 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.489 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:26.489 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605257+0000", 2020-01-16T00:13:26.489 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0966e-05 2020-01-16T00:13:26.489 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.489 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.490 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:26.490 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605268+0000", 2020-01-16T00:13:26.490 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5960000000000003e-06 2020-01-16T00:13:26.490 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.490 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.490 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:26.490 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605274+0000", 2020-01-16T00:13:26.491 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.007370176 2020-01-16T00:13:26.491 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.491 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.491 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:26.491 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.612644+0000", 2020-01-16T00:13:26.491 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000113146 2020-01-16T00:13:26.491 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.491 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.492 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:26.492 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.612757+0000", 2020-01-16T00:13:26.492 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00058744799999999996 2020-01-16T00:13:26.492 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.492 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.492 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:26.492 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.613345+0000", 2020-01-16T00:13:26.493 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0080920829999999999 2020-01-16T00:13:26.493 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.493 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:26.493 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.493 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.493 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.493 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1644 3.1 3:8c6a83c0:::smithi12313934-36:head [read 0~2705927,checksum] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:13:26.493 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847556+0000", 2020-01-16T00:13:26.494 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 34.607529628000002, 2020-01-16T00:13:26.494 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0081843639999999995, 2020-01-16T00:13:26.494 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:26.494 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:26.494 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:26.494 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:26.494 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:26.495 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1644 2020-01-16T00:13:26.495 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.495 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:26.495 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.495 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:26.495 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847556+0000", 2020-01-16T00:13:26.495 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:26.495 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.496 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.496 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:26.496 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847556+0000", 2020-01-16T00:13:26.496 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:26.497 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.497 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.497 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:26.497 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847555+0000", 2020-01-16T00:13:26.497 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4640000000000002e-06 2020-01-16T00:13:26.498 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.498 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.498 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:26.498 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847559+0000", 2020-01-16T00:13:26.498 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4580000000000003e-06 2020-01-16T00:13:26.498 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.498 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.498 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:26.499 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847563+0000", 2020-01-16T00:13:26.499 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.613e-06 2020-01-16T00:13:26.499 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.499 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.499 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:26.501 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847567+0000", 2020-01-16T00:13:26.501 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000145045 2020-01-16T00:13:26.501 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.501 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.501 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:26.501 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847712+0000", 2020-01-16T00:13:26.502 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6480000000000003e-06 2020-01-16T00:13:26.502 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.502 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.502 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:13:26.502 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847720+0000", 2020-01-16T00:13:26.502 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0055645269999999997 2020-01-16T00:13:26.503 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.503 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.503 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:26.503 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.853284+0000", 2020-01-16T00:13:26.503 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00026767400000000003 2020-01-16T00:13:26.503 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.503 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.503 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:26.504 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.853552+0000", 2020-01-16T00:13:26.504 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021884169999999998 2020-01-16T00:13:26.504 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.504 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.504 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:26.504 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.855740+0000", 2020-01-16T00:13:26.504 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0081843639999999995 2020-01-16T00:13:26.504 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.505 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:26.505 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.505 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.505 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.505 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1645 3.1 3:8c6a83c0:::smithi12313934-36:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:13:26.505 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847602+0000", 2020-01-16T00:13:26.505 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 34.607483829000003, 2020-01-16T00:13:26.506 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0085169679999999998, 2020-01-16T00:13:26.506 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:26.506 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:26.506 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:26.506 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:26.506 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:26.506 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1645 2020-01-16T00:13:26.507 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.507 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:26.507 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.507 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:26.507 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847602+0000", 2020-01-16T00:13:26.507 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:26.507 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.507 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.508 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:26.508 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847602+0000", 2020-01-16T00:13:26.508 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:26.508 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.508 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.508 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:26.508 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847601+0000", 2020-01-16T00:13:26.509 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7011e-05 2020-01-16T00:13:26.509 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.509 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.509 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:26.509 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847618+0000", 2020-01-16T00:13:26.509 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.187e-06 2020-01-16T00:13:26.510 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.510 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.510 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:26.510 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847625+0000", 2020-01-16T00:13:26.510 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9879999999999998e-06 2020-01-16T00:13:26.511 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.511 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.511 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:26.511 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847629+0000", 2020-01-16T00:13:26.511 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000129694 2020-01-16T00:13:26.511 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.511 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.511 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:26.512 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847759+0000", 2020-01-16T00:13:26.512 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8379999999999996e-06 2020-01-16T00:13:26.512 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.512 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.512 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:13:26.512 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847766+0000", 2020-01-16T00:13:26.512 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0079977390000000002 2020-01-16T00:13:26.513 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.513 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.513 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:26.513 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.855764+0000", 2020-01-16T00:13:26.513 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2467999999999996e-05 2020-01-16T00:13:26.513 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.514 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.514 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:26.514 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.855856+0000", 2020-01-16T00:13:26.514 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00026275099999999998 2020-01-16T00:13:26.514 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.514 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.515 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:26.515 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.856119+0000", 2020-01-16T00:13:26.515 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0085169679999999998 2020-01-16T00:13:26.515 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.515 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:26.515 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.515 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.515 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.516 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1649 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~2084068,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:13:26.516 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847683+0000", 2020-01-16T00:13:26.516 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 34.607402577999999, 2020-01-16T00:13:26.516 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010654691000000001, 2020-01-16T00:13:26.516 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:26.516 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:26.516 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:26.517 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:26.517 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:26.517 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1649 2020-01-16T00:13:26.517 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.517 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:26.517 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.517 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:26.518 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847683+0000", 2020-01-16T00:13:26.518 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:26.518 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.518 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.518 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:26.518 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847683+0000", 2020-01-16T00:13:26.518 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:26.518 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.519 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.519 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:26.519 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847682+0000", 2020-01-16T00:13:26.519 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8569999999999997e-06 2020-01-16T00:13:26.519 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.519 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.519 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:26.519 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847689+0000", 2020-01-16T00:13:26.520 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8090000000000004e-06 2020-01-16T00:13:26.520 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.520 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.520 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:26.520 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847696+0000", 2020-01-16T00:13:26.520 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2690000000000001e-06 2020-01-16T00:13:26.520 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.521 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.521 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:26.521 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847700+0000", 2020-01-16T00:13:26.521 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8189999999999993e-05 2020-01-16T00:13:26.521 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.521 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.521 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:26.522 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847798+0000", 2020-01-16T00:13:26.522 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7129999999999999e-06 2020-01-16T00:13:26.522 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.522 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.522 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:13:26.522 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847802+0000", 2020-01-16T00:13:26.522 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083913049999999999 2020-01-16T00:13:26.523 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.523 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.523 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:26.523 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.856193+0000", 2020-01-16T00:13:26.523 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034400300000000001 2020-01-16T00:13:26.523 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.523 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.523 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:26.524 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.856537+0000", 2020-01-16T00:13:26.524 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018006820000000001 2020-01-16T00:13:26.524 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.524 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.524 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:26.524 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858338+0000", 2020-01-16T00:13:26.524 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010654691000000001 2020-01-16T00:13:26.525 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.525 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:26.525 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.525 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.525 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.525 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1650 3.1 3:8de1d8a3:::smithi12313934-13:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:13:26.525 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847717+0000", 2020-01-16T00:13:26.526 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 34.607368624999999, 2020-01-16T00:13:26.526 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010913658, 2020-01-16T00:13:26.526 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:26.526 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:26.527 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:26.527 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:26.527 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:26.527 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1650 2020-01-16T00:13:26.527 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.527 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:26.527 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.528 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:26.528 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847717+0000", 2020-01-16T00:13:26.528 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:26.528 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.528 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.528 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:26.528 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847717+0000", 2020-01-16T00:13:26.528 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:26.529 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.529 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.529 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:26.529 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847716+0000", 2020-01-16T00:13:26.529 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.433e-06 2020-01-16T00:13:26.529 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.529 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.529 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:26.530 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847720+0000", 2020-01-16T00:13:26.530 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1800000000000004e-06 2020-01-16T00:13:26.530 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.530 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.530 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:26.530 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847725+0000", 2020-01-16T00:13:26.530 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6689999999999997e-06 2020-01-16T00:13:26.531 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.531 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.531 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:26.531 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847730+0000", 2020-01-16T00:13:26.531 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000102468 2020-01-16T00:13:26.531 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.531 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.531 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:26.532 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847832+0000", 2020-01-16T00:13:26.532 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3999999999999997e-06 2020-01-16T00:13:26.532 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.532 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.532 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:13:26.532 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847838+0000", 2020-01-16T00:13:26.532 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010519395000000001 2020-01-16T00:13:26.533 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.533 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.533 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:26.533 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858358+0000", 2020-01-16T00:13:26.533 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7600000000000002e-05 2020-01-16T00:13:26.533 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.533 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.533 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:26.534 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858445+0000", 2020-01-16T00:13:26.534 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018515900000000001 2020-01-16T00:13:26.534 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.534 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.534 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:26.534 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858631+0000", 2020-01-16T00:13:26.534 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010913658 2020-01-16T00:13:26.535 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.535 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:26.535 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.535 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.535 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.535 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1651 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~2084068,checksum] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:13:26.535 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847752+0000", 2020-01-16T00:13:26.535 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 34.607334008999999, 2020-01-16T00:13:26.536 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011734249, 2020-01-16T00:13:26.536 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:26.536 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:26.536 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:26.536 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:26.536 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:26.536 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1651 2020-01-16T00:13:26.536 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.537 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:26.537 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.537 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:26.537 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847752+0000", 2020-01-16T00:13:26.537 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:26.537 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.537 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.538 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:26.538 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847752+0000", 2020-01-16T00:13:26.538 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:26.538 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.538 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.538 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:26.538 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847751+0000", 2020-01-16T00:13:26.539 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1980000000000001e-06 2020-01-16T00:13:26.539 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.539 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.539 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:26.539 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847754+0000", 2020-01-16T00:13:26.539 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6679999999999999e-06 2020-01-16T00:13:26.539 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.539 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.540 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:26.540 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847758+0000", 2020-01-16T00:13:26.540 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2500000000000001e-06 2020-01-16T00:13:26.540 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.540 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.540 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:26.540 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847760+0000", 2020-01-16T00:13:26.541 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7614000000000004e-05 2020-01-16T00:13:26.541 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.541 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.541 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:26.541 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847858+0000", 2020-01-16T00:13:26.541 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7950000000000004e-06 2020-01-16T00:13:26.541 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.541 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.542 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:13:26.542 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847864+0000", 2020-01-16T00:13:26.542 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010790345 2020-01-16T00:13:26.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.542 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.542 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:26.543 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858654+0000", 2020-01-16T00:13:26.543 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000141466 2020-01-16T00:13:26.543 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.543 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.543 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:26.543 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858795+0000", 2020-01-16T00:13:26.543 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00069037899999999997 2020-01-16T00:13:26.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.544 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.544 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:26.544 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.859486+0000", 2020-01-16T00:13:26.544 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011734249 2020-01-16T00:13:26.544 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.545 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:26.545 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.545 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.545 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.545 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:55 2.1 e30/19 2:91f85463:::smithi12313934-24 low_tier:head v 30'56, mlcod=29'54)", 2020-01-16T00:13:26.545 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.738134+0000", 2020-01-16T00:13:26.545 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 33.716951723000001, 2020-01-16T00:13:26.546 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0086509570000000008, 2020-01-16T00:13:26.546 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:26.546 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:26.546 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:26.546 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.546 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:26.546 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738134+0000", 2020-01-16T00:13:26.546 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:26.547 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.547 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.547 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:26.547 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738134+0000", 2020-01-16T00:13:26.547 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:26.547 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.547 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.548 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:26.548 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738134+0000", 2020-01-16T00:13:26.548 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00053712599999999997 2020-01-16T00:13:26.548 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.548 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.548 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:26.548 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738671+0000", 2020-01-16T00:13:26.549 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4059999999999999e-05 2020-01-16T00:13:26.549 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.549 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.549 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:26.549 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738685+0000", 2020-01-16T00:13:26.549 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7680000000000008e-06 2020-01-16T00:13:26.550 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.550 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.550 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:26.550 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738692+0000", 2020-01-16T00:13:26.550 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1273999999999996e-05 2020-01-16T00:13:26.550 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.550 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.550 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:26.551 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738764+0000", 2020-01-16T00:13:26.551 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6827999999999999e-05 2020-01-16T00:13:26.551 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.551 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.551 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:26.551 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738781+0000", 2020-01-16T00:13:26.551 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0079370740000000006 2020-01-16T00:13:26.551 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.552 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.552 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:26.552 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746718+0000", 2020-01-16T00:13:26.552 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7181000000000006e-05 2020-01-16T00:13:26.552 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.552 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.552 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:26.553 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746785+0000", 2020-01-16T00:13:26.553 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0086509570000000008 2020-01-16T00:13:26.553 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.553 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:26.553 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.553 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.553 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.554 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2106 3.1 3:8de1d8a3:::smithi12313934-13:head [write 665610~797860,stat] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:13:26.554 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.631369+0000", 2020-01-16T00:13:26.554 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 24.823716297000001, 2020-01-16T00:13:26.554 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0091195330000000008, 2020-01-16T00:13:26.554 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:26.555 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:26.555 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:26.555 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:26.555 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:26.555 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2106 2020-01-16T00:13:26.556 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.556 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:26.557 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.557 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:26.557 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631369+0000", 2020-01-16T00:13:26.557 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:26.558 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.558 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.558 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:26.558 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631369+0000", 2020-01-16T00:13:26.559 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:26.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.559 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.559 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:26.559 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631367+0000", 2020-01-16T00:13:26.559 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00025714100000000001 2020-01-16T00:13:26.560 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.560 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.560 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:26.560 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631624+0000", 2020-01-16T00:13:26.560 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4946000000000001e-05 2020-01-16T00:13:26.560 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.560 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.561 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:26.561 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631639+0000", 2020-01-16T00:13:26.561 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7050000000000002e-06 2020-01-16T00:13:26.561 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.561 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.561 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:26.561 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631646+0000", 2020-01-16T00:13:26.562 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010938900000000001 2020-01-16T00:13:26.562 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.562 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.562 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:26.562 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631755+0000", 2020-01-16T00:13:26.563 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8848e-05 2020-01-16T00:13:26.563 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.563 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.564 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op must wait for map", 2020-01-16T00:13:26.564 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631774+0000", 2020-01-16T00:13:26.564 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0026027730000000001 2020-01-16T00:13:26.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.565 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.565 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:26.565 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.634377+0000", 2020-01-16T00:13:26.565 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9282000000000003e-05 2020-01-16T00:13:26.566 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.567 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.567 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:26.567 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.634466+0000", 2020-01-16T00:13:26.567 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000127868 2020-01-16T00:13:26.568 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.568 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.568 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:13:26.568 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.634594+0000", 2020-01-16T00:13:26.568 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030776549999999999 2020-01-16T00:13:26.569 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.569 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.570 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:26.570 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.637672+0000", 2020-01-16T00:13:26.570 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001634195 2020-01-16T00:13:26.570 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.570 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.570 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:26.570 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.639306+0000", 2020-01-16T00:13:26.571 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010810069999999999 2020-01-16T00:13:26.571 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.571 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.571 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:26.571 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640387+0000", 2020-01-16T00:13:26.571 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0543000000000001e-05 2020-01-16T00:13:26.571 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.571 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.572 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:26.572 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640428+0000", 2020-01-16T00:13:26.572 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1321000000000005e-05 2020-01-16T00:13:26.572 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.572 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.572 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:26.572 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640489+0000", 2020-01-16T00:13:26.572 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0091195330000000008 2020-01-16T00:13:26.573 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.573 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:26.573 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.573 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.573 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.573 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2107 3.1 3:8de1d8a3:::smithi12313934-13:head [write 1909329~458405] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:13:26.573 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.636786+0000", 2020-01-16T00:13:26.574 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 24.818299166999999, 2020-01-16T00:13:26.574 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093182809999999994, 2020-01-16T00:13:26.574 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:26.574 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:26.574 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:26.574 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:26.574 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:26.574 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2107 2020-01-16T00:13:26.575 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.575 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:26.575 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.575 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:26.575 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636786+0000", 2020-01-16T00:13:26.575 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:26.575 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.575 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.576 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:26.576 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636786+0000", 2020-01-16T00:13:26.576 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:26.576 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.576 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.576 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:26.576 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636784+0000", 2020-01-16T00:13:26.576 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014276299999999999 2020-01-16T00:13:26.577 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.577 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.577 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:26.577 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636927+0000", 2020-01-16T00:13:26.577 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5017999999999999e-05 2020-01-16T00:13:26.577 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.577 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.578 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:26.578 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636942+0000", 2020-01-16T00:13:26.578 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5589999999999996e-06 2020-01-16T00:13:26.578 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.578 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.578 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:26.578 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636947+0000", 2020-01-16T00:13:26.578 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00040279699999999999 2020-01-16T00:13:26.579 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.579 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.579 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:26.579 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.637350+0000", 2020-01-16T00:13:26.579 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000233506 2020-01-16T00:13:26.579 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.580 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.580 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:26.580 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.637584+0000", 2020-01-16T00:13:26.580 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029534660000000001 2020-01-16T00:13:26.580 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.580 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.580 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:26.580 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640537+0000", 2020-01-16T00:13:26.581 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012577700000000001 2020-01-16T00:13:26.581 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.581 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.581 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:26.581 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640663+0000", 2020-01-16T00:13:26.581 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000198593 2020-01-16T00:13:26.581 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.581 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.582 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:13:26.582 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640862+0000", 2020-01-16T00:13:26.582 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.005088197 2020-01-16T00:13:26.582 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.582 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.582 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:26.582 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645950+0000", 2020-01-16T00:13:26.583 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2568999999999999e-05 2020-01-16T00:13:26.583 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.583 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.583 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:26.583 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645962+0000", 2020-01-16T00:13:26.583 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3034999999999995e-05 2020-01-16T00:13:26.583 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.583 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.584 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:26.584 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646025+0000", 2020-01-16T00:13:26.584 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7762999999999997e-05 2020-01-16T00:13:26.584 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.584 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.584 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:26.584 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646063+0000", 2020-01-16T00:13:26.584 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1541000000000001e-05 2020-01-16T00:13:26.585 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.585 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.585 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:26.585 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646105+0000", 2020-01-16T00:13:26.585 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093182809999999994 2020-01-16T00:13:26.585 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.585 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:26.586 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.586 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.586 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.586 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2108 3.1 3:8de1d8a3:::smithi12313934-13:head [write 2816700~410210] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:13:26.586 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.641861+0000", 2020-01-16T00:13:26.586 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 24.813224818999998, 2020-01-16T00:13:26.586 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018545604, 2020-01-16T00:13:26.586 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:26.587 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:26.587 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:26.587 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:26.587 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:26.587 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2108 2020-01-16T00:13:26.587 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.587 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:26.588 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.588 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:26.588 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.641861+0000", 2020-01-16T00:13:26.588 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:26.588 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.588 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.588 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:26.588 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.641861+0000", 2020-01-16T00:13:26.589 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:26.589 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.589 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.589 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:26.589 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.641859+0000", 2020-01-16T00:13:26.589 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018466399999999999 2020-01-16T00:13:26.589 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.590 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.590 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:26.590 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642043+0000", 2020-01-16T00:13:26.590 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2999999999999999e-05 2020-01-16T00:13:26.590 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.590 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.591 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:26.591 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642056+0000", 2020-01-16T00:13:26.591 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0730000000000005e-06 2020-01-16T00:13:26.591 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.591 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.591 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:26.591 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642064+0000", 2020-01-16T00:13:26.591 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9527999999999997e-05 2020-01-16T00:13:26.592 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.592 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.592 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:26.592 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642114+0000", 2020-01-16T00:13:26.592 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036616050000000001 2020-01-16T00:13:26.592 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.592 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.592 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:26.593 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645776+0000", 2020-01-16T00:13:26.593 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012222663999999999 2020-01-16T00:13:26.593 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.593 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.593 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:26.593 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.657998+0000", 2020-01-16T00:13:26.593 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014280299999999999 2020-01-16T00:13:26.594 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.594 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.594 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:26.594 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.658141+0000", 2020-01-16T00:13:26.594 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0001102 2020-01-16T00:13:26.594 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.594 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.594 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:13:26.595 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.658251+0000", 2020-01-16T00:13:26.595 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020421219999999999 2020-01-16T00:13:26.595 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.595 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.595 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:26.595 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660293+0000", 2020-01-16T00:13:26.595 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7399999999999999e-06 2020-01-16T00:13:26.595 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.596 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.596 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:26.596 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660303+0000", 2020-01-16T00:13:26.596 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9873999999999998e-05 2020-01-16T00:13:26.596 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.596 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.596 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:26.597 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660343+0000", 2020-01-16T00:13:26.597 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4448e-05 2020-01-16T00:13:26.597 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.597 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.597 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:26.598 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660377+0000", 2020-01-16T00:13:26.598 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8915999999999999e-05 2020-01-16T00:13:26.598 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.598 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.599 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:26.599 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660406+0000", 2020-01-16T00:13:26.599 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018545604 2020-01-16T00:13:26.599 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.601 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:26.601 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.602 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.602 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.602 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2109 3.1 3:8de1d8a3:::smithi12313934-13:head [setxattr _header (56) in=7b,truncate 3865610] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:13:26.602 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.642075+0000", 2020-01-16T00:13:26.602 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 24.813010246000001, 2020-01-16T00:13:26.602 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018477655999999999, 2020-01-16T00:13:26.603 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:26.603 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:26.603 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:26.603 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:26.603 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:26.603 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2109 2020-01-16T00:13:26.604 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.604 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:26.604 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.604 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:26.604 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642075+0000", 2020-01-16T00:13:26.604 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:26.604 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.604 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.605 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:26.605 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642075+0000", 2020-01-16T00:13:26.605 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:26.605 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.605 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.605 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:26.606 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642074+0000", 2020-01-16T00:13:26.606 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4500000000000003e-06 2020-01-16T00:13:26.606 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.606 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.606 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:26.606 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642080+0000", 2020-01-16T00:13:26.606 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7229999999999998e-06 2020-01-16T00:13:26.607 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.607 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.607 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:26.607 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642087+0000", 2020-01-16T00:13:26.607 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2390000000000004e-06 2020-01-16T00:13:26.607 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.607 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.608 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:26.608 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642091+0000", 2020-01-16T00:13:26.608 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0037396550000000001 2020-01-16T00:13:26.608 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.608 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.608 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:26.608 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645830+0000", 2020-01-16T00:13:26.608 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2200000000000002e-05 2020-01-16T00:13:26.609 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.609 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.609 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:26.609 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645883+0000", 2020-01-16T00:13:26.609 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013432934000000001 2020-01-16T00:13:26.609 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.609 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.610 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:26.610 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659316+0000", 2020-01-16T00:13:26.610 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000140006 2020-01-16T00:13:26.610 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.610 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.610 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:26.610 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659456+0000", 2020-01-16T00:13:26.611 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000184873 2020-01-16T00:13:26.611 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.611 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.611 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:13:26.611 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659640+0000", 2020-01-16T00:13:26.611 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00066954999999999998 2020-01-16T00:13:26.611 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.611 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.612 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:26.612 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660310+0000", 2020-01-16T00:13:26.612 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014847699999999999 2020-01-16T00:13:26.612 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.612 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.612 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:26.612 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660459+0000", 2020-01-16T00:13:26.612 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6869e-05 2020-01-16T00:13:26.613 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.613 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.613 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:26.613 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660485+0000", 2020-01-16T00:13:26.613 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8478000000000001e-05 2020-01-16T00:13:26.613 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.613 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.613 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:26.613 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660514+0000", 2020-01-16T00:13:26.614 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9153999999999998e-05 2020-01-16T00:13:26.614 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.614 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.614 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:26.614 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660553+0000", 2020-01-16T00:13:26.614 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018477655999999999 2020-01-16T00:13:26.614 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.615 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:26.615 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.615 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.616 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.616 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2110 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-01-16T00:13:26.616 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.642112+0000", 2020-01-16T00:13:26.616 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 24.812973608, 2020-01-16T00:13:26.616 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018620198000000001, 2020-01-16T00:13:26.616 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:26.617 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:26.617 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:26.617 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:26.617 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:26.617 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2110 2020-01-16T00:13:26.617 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.618 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:26.618 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.618 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:26.618 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642112+0000", 2020-01-16T00:13:26.618 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:26.618 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.619 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.619 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:26.619 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642112+0000", 2020-01-16T00:13:26.619 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:26.619 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.619 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.619 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:26.620 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642111+0000", 2020-01-16T00:13:26.620 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4450000000000001e-06 2020-01-16T00:13:26.620 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.620 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.620 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:26.620 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642117+0000", 2020-01-16T00:13:26.620 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1567e-05 2020-01-16T00:13:26.620 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.621 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.621 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:26.621 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642129+0000", 2020-01-16T00:13:26.621 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.754e-06 2020-01-16T00:13:26.622 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.622 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.622 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:26.623 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642136+0000", 2020-01-16T00:13:26.623 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040239719999999998 2020-01-16T00:13:26.623 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.623 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.623 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:26.623 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646160+0000", 2020-01-16T00:13:26.623 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5600999999999998e-05 2020-01-16T00:13:26.623 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.624 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.624 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:26.624 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646215+0000", 2020-01-16T00:13:26.624 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013733006000000001 2020-01-16T00:13:26.624 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.624 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.624 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:26.625 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659948+0000", 2020-01-16T00:13:26.625 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000106277 2020-01-16T00:13:26.625 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.625 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.625 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:26.625 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660054+0000", 2020-01-16T00:13:26.625 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000524327 2020-01-16T00:13:26.626 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.626 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.626 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:26.626 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660579+0000", 2020-01-16T00:13:26.626 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9841999999999999e-05 2020-01-16T00:13:26.626 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.626 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.626 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:26.627 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660659+0000", 2020-01-16T00:13:26.627 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3572999999999995e-05 2020-01-16T00:13:26.627 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.627 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.627 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:26.627 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660732+0000", 2020-01-16T00:13:26.627 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018620198000000001 2020-01-16T00:13:26.627 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.628 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:26.628 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.628 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.628 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.628 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:70 2.1 e35/19 2:91f85463:::smithi12313934-24 low_tier:head v 35'66, mlcod=34'64)", 2020-01-16T00:13:26.628 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.986684+0000", 2020-01-16T00:13:26.628 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 20.468401753999999, 2020-01-16T00:13:26.629 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012287378999999999, 2020-01-16T00:13:26.629 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:26.629 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:26.629 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:26.629 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.629 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:26.629 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986684+0000", 2020-01-16T00:13:26.629 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:26.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.630 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.630 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:26.630 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986684+0000", 2020-01-16T00:13:26.630 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:26.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.630 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.631 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:26.631 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986683+0000", 2020-01-16T00:13:26.631 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012884179999999999 2020-01-16T00:13:26.631 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.631 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.631 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:26.631 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.987971+0000", 2020-01-16T00:13:26.632 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011558200000000001 2020-01-16T00:13:26.632 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.632 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.632 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:26.632 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988087+0000", 2020-01-16T00:13:26.632 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0472000000000001e-05 2020-01-16T00:13:26.632 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.632 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.633 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:26.633 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988137+0000", 2020-01-16T00:13:26.633 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000107484 2020-01-16T00:13:26.633 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.633 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.633 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:26.633 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988245+0000", 2020-01-16T00:13:26.633 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4409999999999998e-05 2020-01-16T00:13:26.634 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.634 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.634 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:26.634 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988279+0000", 2020-01-16T00:13:26.634 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010641609999999999 2020-01-16T00:13:26.634 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.634 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.635 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:26.635 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.998921+0000", 2020-01-16T00:13:26.635 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0278000000000002e-05 2020-01-16T00:13:26.635 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.635 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.635 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:26.635 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.998971+0000", 2020-01-16T00:13:26.635 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012287378999999999 2020-01-16T00:13:26.636 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.636 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:26.636 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.636 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.636 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.636 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:71 2.1 e35/19 2:91f85463:::smithi12313934-24 low_tier:head v 35'67, mlcod=34'64)", 2020-01-16T00:13:26.636 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.991132+0000", 2020-01-16T00:13:26.636 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 20.463953803999999, 2020-01-16T00:13:26.637 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0082681190000000009, 2020-01-16T00:13:26.637 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:26.637 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:26.637 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:26.637 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.637 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:26.637 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.991132+0000", 2020-01-16T00:13:26.638 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:26.638 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.638 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.638 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:26.638 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.991132+0000", 2020-01-16T00:13:26.638 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:26.638 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.638 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.639 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:26.639 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.991130+0000", 2020-01-16T00:13:26.639 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3583000000000003e-05 2020-01-16T00:13:26.639 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.639 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.639 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:26.639 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.991203+0000", 2020-01-16T00:13:26.639 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.1001000000000001e-05 2020-01-16T00:13:26.640 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.640 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.640 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:26.640 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.991224+0000", 2020-01-16T00:13:26.640 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9329999999999996e-06 2020-01-16T00:13:26.640 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.640 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.640 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:26.641 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.991234+0000", 2020-01-16T00:13:26.641 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0035280070000000001 2020-01-16T00:13:26.641 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.641 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.641 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:26.641 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.994762+0000", 2020-01-16T00:13:26.641 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.622e-05 2020-01-16T00:13:26.642 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.642 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.642 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:26.642 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.994788+0000", 2020-01-16T00:13:26.642 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0045652219999999999 2020-01-16T00:13:26.642 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.642 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.643 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:26.643 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999354+0000", 2020-01-16T00:13:26.643 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6396000000000001e-05 2020-01-16T00:13:26.643 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.643 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.643 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:26.643 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999400+0000", 2020-01-16T00:13:26.643 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0082681190000000009 2020-01-16T00:13:26.644 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.644 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:26.644 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.644 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.644 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.644 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:77 2.1 e36/19 2:91f85463:::smithi12313934-24 low_tier:head v 36'69, mlcod=35'67)", 2020-01-16T00:13:26.644 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.556730+0000", 2020-01-16T00:13:26.645 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 18.8983554, 2020-01-16T00:13:26.645 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093007760000000002, 2020-01-16T00:13:26.645 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:26.645 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:26.645 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:26.645 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.645 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:26.645 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556730+0000", 2020-01-16T00:13:26.646 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:26.646 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.646 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.646 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:26.646 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556730+0000", 2020-01-16T00:13:26.646 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:26.646 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.647 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.647 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:26.647 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556728+0000", 2020-01-16T00:13:26.647 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00056509599999999996 2020-01-16T00:13:26.647 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.647 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.648 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:26.648 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.557293+0000", 2020-01-16T00:13:26.648 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0225e-05 2020-01-16T00:13:26.648 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.648 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.648 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:26.648 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.557314+0000", 2020-01-16T00:13:26.648 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8989999999999996e-06 2020-01-16T00:13:26.649 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.649 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.649 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:26.649 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.557323+0000", 2020-01-16T00:13:26.649 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7923000000000002e-05 2020-01-16T00:13:26.649 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.649 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.649 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:26.650 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.557371+0000", 2020-01-16T00:13:26.650 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9963999999999999e-05 2020-01-16T00:13:26.650 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.650 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.650 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:26.650 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.557411+0000", 2020-01-16T00:13:26.650 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0085873000000000008 2020-01-16T00:13:26.651 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.651 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.651 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:26.651 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.565999+0000", 2020-01-16T00:13:26.651 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2326000000000002e-05 2020-01-16T00:13:26.651 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.651 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.651 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:26.652 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566031+0000", 2020-01-16T00:13:26.652 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093007760000000002 2020-01-16T00:13:26.652 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.652 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:26.653 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.653 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.653 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.653 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:109 2.0 e42/19 2:11fa79bc:::smithi12313934-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 42'67, mlcod=41'65)", 2020-01-16T00:13:26.653 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:21.841140+0000", 2020-01-16T00:13:26.653 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.6139456860000001, 2020-01-16T00:13:26.654 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010439159999999999, 2020-01-16T00:13:26.654 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:26.654 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:26.654 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:26.654 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.654 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:26.654 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841140+0000", 2020-01-16T00:13:26.654 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:26.655 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.655 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.655 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:26.655 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841140+0000", 2020-01-16T00:13:26.655 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:26.655 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.655 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.655 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:26.656 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841137+0000", 2020-01-16T00:13:26.656 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00061049800000000001 2020-01-16T00:13:26.656 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.656 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.656 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:26.656 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841748+0000", 2020-01-16T00:13:26.656 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8703e-05 2020-01-16T00:13:26.656 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.657 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.657 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:26.657 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841767+0000", 2020-01-16T00:13:26.657 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5410000000000003e-06 2020-01-16T00:13:26.657 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.657 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.657 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:26.658 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841774+0000", 2020-01-16T00:13:26.658 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000100144 2020-01-16T00:13:26.658 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.658 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.658 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:26.658 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841874+0000", 2020-01-16T00:13:26.658 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9402999999999999e-05 2020-01-16T00:13:26.659 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.659 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.659 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:26.659 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841944+0000", 2020-01-16T00:13:26.659 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095901649999999995 2020-01-16T00:13:26.659 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.659 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.659 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:26.660 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.851534+0000", 2020-01-16T00:13:26.660 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5296000000000001e-05 2020-01-16T00:13:26.660 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.660 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.660 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:26.660 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.851579+0000", 2020-01-16T00:13:26.660 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010439159999999999 2020-01-16T00:13:26.660 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.661 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:26.661 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.661 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.661 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.661 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:110 2.0 e42/19 2:11fa79bc:::smithi12313934-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 42'68, mlcod=41'65)", 2020-01-16T00:13:26.661 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:21.842301+0000", 2020-01-16T00:13:26.661 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.6127842550000002, 2020-01-16T00:13:26.662 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093040050000000006, 2020-01-16T00:13:26.662 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:26.662 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:26.662 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:26.662 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.662 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:26.662 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.842301+0000", 2020-01-16T00:13:26.662 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:26.663 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.663 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.663 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:26.663 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.842301+0000", 2020-01-16T00:13:26.663 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:26.663 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.663 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.663 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:26.664 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.842300+0000", 2020-01-16T00:13:26.664 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001154504 2020-01-16T00:13:26.664 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.664 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.664 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:26.664 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.843455+0000", 2020-01-16T00:13:26.664 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3263e-05 2020-01-16T00:13:26.664 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.665 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.665 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:26.665 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.843468+0000", 2020-01-16T00:13:26.665 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0786e-05 2020-01-16T00:13:26.665 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.665 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.665 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:26.666 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.843479+0000", 2020-01-16T00:13:26.666 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021168810000000001 2020-01-16T00:13:26.666 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.666 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.666 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:26.666 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.845596+0000", 2020-01-16T00:13:26.666 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0740000000000001e-05 2020-01-16T00:13:26.666 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.667 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.667 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:26.667 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.845616+0000", 2020-01-16T00:13:26.667 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0059668830000000001 2020-01-16T00:13:26.667 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.667 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.667 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:26.667 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.851583+0000", 2020-01-16T00:13:26.668 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2133999999999998e-05 2020-01-16T00:13:26.668 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.668 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.668 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:26.668 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.851605+0000", 2020-01-16T00:13:26.668 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093040050000000006 2020-01-16T00:13:26.668 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.669 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:26.669 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.669 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.669 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.669 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:111 2.0 e42/19 2:11fa79bc:::smithi12313934-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 42'69, mlcod=41'65)", 2020-01-16T00:13:26.669 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:21.843698+0000", 2020-01-16T00:13:26.669 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.6113875970000002, 2020-01-16T00:13:26.670 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0082463989999999997, 2020-01-16T00:13:26.670 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:26.670 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:26.670 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:26.670 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.670 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:26.670 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.843698+0000", 2020-01-16T00:13:26.670 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:26.671 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.671 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.671 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:26.671 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.843698+0000", 2020-01-16T00:13:26.671 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:26.671 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.672 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.672 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:26.672 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.843697+0000", 2020-01-16T00:13:26.672 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0011376730000000001 2020-01-16T00:13:26.672 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.672 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.672 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:26.673 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.844835+0000", 2020-01-16T00:13:26.673 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7294000000000001e-05 2020-01-16T00:13:26.673 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.673 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.673 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:26.673 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.844852+0000", 2020-01-16T00:13:26.674 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0639999999999994e-06 2020-01-16T00:13:26.674 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.674 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.674 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:26.674 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.844860+0000", 2020-01-16T00:13:26.674 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036989520000000001 2020-01-16T00:13:26.674 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.674 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.675 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:26.675 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.848559+0000", 2020-01-16T00:13:26.675 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.4658e-05 2020-01-16T00:13:26.675 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.675 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.675 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:26.675 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.848584+0000", 2020-01-16T00:13:26.675 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003318128 2020-01-16T00:13:26.676 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.676 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.676 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:26.676 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.851902+0000", 2020-01-16T00:13:26.676 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2719000000000002e-05 2020-01-16T00:13:26.676 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:26.676 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:26.677 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:26.677 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.851944+0000", 2020-01-16T00:13:26.677 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0082463989999999997 2020-01-16T00:13:26.677 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.677 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:26.677 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.677 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:26.677 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:26.678 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:26.679 INFO:tasks.rados.rados.0.smithi123.stdout:1243: expect (ObjNum 361 snap 0 seq_num 361) 2020-01-16T00:13:26.679 INFO:tasks.rados.rados.0.smithi123.stdout:1245: finishing write tid 1 to smithi12313934-37 2020-01-16T00:13:26.679 INFO:tasks.rados.rados.0.smithi123.stdout:1245: finishing write tid 2 to smithi12313934-37 2020-01-16T00:13:26.679 INFO:tasks.rados.rados.0.smithi123.stdout:1245: finishing write tid 4 to smithi12313934-37 2020-01-16T00:13:26.680 INFO:tasks.rados.rados.0.smithi123.stdout:1245: finishing write tid 5 to smithi12313934-37 2020-01-16T00:13:26.680 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 37 v 236 (ObjNum 378 snap 0 seq_num 378) dirty exists 2020-01-16T00:13:26.680 INFO:tasks.rados.rados.0.smithi123.stdout:1245: left oid 37 (ObjNum 378 snap 0 seq_num 378) 2020-01-16T00:13:26.680 INFO:tasks.rados.rados.0.smithi123.stdout:1246: finishing write tid 1 to smithi12313934-2 2020-01-16T00:13:26.680 INFO:tasks.rados.rados.0.smithi123.stdout:1247: finishing write tid 1 to smithi12313934-44 2020-01-16T00:13:26.680 INFO:tasks.rados.rados.0.smithi123.stdout:1246: finishing write tid 2 to smithi12313934-2 2020-01-16T00:13:26.680 INFO:tasks.rados.rados.0.smithi123.stdout:1246: finishing write tid 4 to smithi12313934-2 2020-01-16T00:13:26.680 INFO:tasks.rados.rados.0.smithi123.stdout:1246: finishing write tid 5 to smithi12313934-2 2020-01-16T00:13:26.681 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 2 v 183 (ObjNum 379 snap 0 seq_num 379) dirty exists 2020-01-16T00:13:26.681 INFO:tasks.rados.rados.0.smithi123.stdout:1246: left oid 2 (ObjNum 379 snap 0 seq_num 379) 2020-01-16T00:13:26.681 INFO:tasks.rados.rados.0.smithi123.stdout:1248: expect (ObjNum 329 snap 0 seq_num 329) 2020-01-16T00:13:26.681 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-01-16T00:13:26.684 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:26.674+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:26.761 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-01-16T00:13:26.782 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:26.776+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:26.795 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:26.795 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:26.796 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:13:26.796 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:26.804 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-01-16T00:13:26.807 INFO:tasks.rados.rados.0.smithi123.stdout:1249: expect (ObjNum 328 snap 0 seq_num 328) 2020-01-16T00:13:26.881 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:26.876+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:26.908 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2020-01-16T00:13:26.909 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:13:26.910 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:26.910 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:26.910 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:13:26.910 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:13:26.910 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:26.919 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-01-16T00:13:26.931 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:13:26.956 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:13:26.975 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:13:26.981 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:26.978+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:26.998 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:13:27.023 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:13:27.035 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:27.044 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-01-16T00:13:27.058 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:13:27.060 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-01-16T00:13:27.060 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-01-16T00:13:27.060 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-01-16T00:13:27.061 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-01-16T00:13:27.061 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-01-16T00:13:27.061 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENXIO: problem getting command descriptions from osd.5 2020-01-16T00:13:27.061 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: problem getting command descriptions from osd.5 2020-01-16T00:13:27.061 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-01-16T00:13:27.061 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-01-16T00:13:27.077 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-01-16T00:13:27.078 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-01-16T00:13:27.097 INFO:tasks.rados.rados.0.smithi123.stdout:1247: finishing write tid 2 to smithi12313934-44 2020-01-16T00:13:27.097 INFO:tasks.rados.rados.0.smithi123.stdout:1247: finishing write tid 4 to smithi12313934-44 2020-01-16T00:13:27.097 INFO:tasks.rados.rados.0.smithi123.stdout:1247: finishing write tid 5 to smithi12313934-44 2020-01-16T00:13:27.097 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 44 v 185 (ObjNum 380 snap 0 seq_num 380) dirty exists 2020-01-16T00:13:27.098 INFO:tasks.rados.rados.0.smithi123.stdout:1247: left oid 44 (ObjNum 380 snap 0 seq_num 380) 2020-01-16T00:13:27.098 INFO:tasks.rados.rados.0.smithi123.stdout:1237: done (15 left) 2020-01-16T00:13:27.098 INFO:tasks.rados.rados.0.smithi123.stdout:1238: done (14 left) 2020-01-16T00:13:27.098 INFO:tasks.rados.rados.0.smithi123.stdout:1239: done (13 left) 2020-01-16T00:13:27.098 INFO:tasks.rados.rados.0.smithi123.stdout:1240: done (12 left) 2020-01-16T00:13:27.098 INFO:tasks.rados.rados.0.smithi123.stdout:1241: done (11 left) 2020-01-16T00:13:27.098 INFO:tasks.rados.rados.0.smithi123.stdout:1242: done (10 left) 2020-01-16T00:13:27.098 INFO:tasks.rados.rados.0.smithi123.stdout:1243: done (9 left) 2020-01-16T00:13:27.099 INFO:tasks.rados.rados.0.smithi123.stdout:1244: done (8 left) 2020-01-16T00:13:27.099 INFO:tasks.rados.rados.0.smithi123.stdout:1245: done (7 left) 2020-01-16T00:13:27.099 INFO:tasks.rados.rados.0.smithi123.stdout:1246: done (6 left) 2020-01-16T00:13:27.099 INFO:tasks.rados.rados.0.smithi123.stdout:1247: done (5 left) 2020-01-16T00:13:27.099 INFO:tasks.rados.rados.0.smithi123.stdout:1248: done (4 left) 2020-01-16T00:13:27.099 INFO:tasks.rados.rados.0.smithi123.stdout:1249: done (3 left) 2020-01-16T00:13:27.099 INFO:tasks.rados.rados.0.smithi123.stdout:1253: read oid 18 snap -1 2020-01-16T00:13:27.100 INFO:tasks.rados.rados.0.smithi123.stdout:1253: expect (ObjNum 220 snap 0 seq_num 220) 2020-01-16T00:13:27.100 INFO:tasks.rados.rados.0.smithi123.stdout:1254: read oid 13 snap -1 2020-01-16T00:13:27.100 INFO:tasks.rados.rados.0.smithi123.stdout:1254: expect (ObjNum 339 snap 0 seq_num 339) 2020-01-16T00:13:27.100 INFO:tasks.rados.rados.0.smithi123.stdout:1255: write (excl) oid 10 current snap is 0 2020-01-16T00:13:27.100 INFO:tasks.rados.rados.0.smithi123.stdout:1255: seq_num 381 ranges {403467=487519,1562110=579120} 2020-01-16T00:13:27.100 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:27.094+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:27.108 INFO:tasks.rados.rados.0.smithi123.stdout:1255: writing smithi12313934-10 from 403467 to 890986 tid 1 2020-01-16T00:13:27.118 INFO:tasks.rados.rados.0.smithi123.stdout:1255: writing smithi12313934-10 from 1562110 to 2141230 tid 2 2020-01-16T00:13:27.118 INFO:tasks.rados.rados.0.smithi123.stdout:1256: read oid 12 snap -1 2020-01-16T00:13:27.118 INFO:tasks.rados.rados.0.smithi123.stdout:1256: expect (ObjNum 320 snap 0 seq_num 320) 2020-01-16T00:13:27.118 INFO:tasks.rados.rados.0.smithi123.stdout:1257: read oid 8 snap -1 2020-01-16T00:13:27.118 INFO:tasks.rados.rados.0.smithi123.stdout:1257: expect (ObjNum 366 snap 0 seq_num 366) 2020-01-16T00:13:27.118 INFO:tasks.rados.rados.0.smithi123.stdout:1258: write oid 31 current snap is 0 2020-01-16T00:13:27.119 INFO:tasks.rados.rados.0.smithi123.stdout:1258: seq_num 382 ranges {420274=562768,1620592=439131} 2020-01-16T00:13:27.123 INFO:tasks.rados.rados.0.smithi123.stdout:1258: writing smithi12313934-31 from 420274 to 983042 tid 1 2020-01-16T00:13:27.130 INFO:tasks.rados.rados.0.smithi123.stdout:1258: writing smithi12313934-31 from 1620592 to 2059723 tid 2 2020-01-16T00:13:27.130 INFO:tasks.rados.rados.0.smithi123.stdout:1259: write (excl) oid 35 current snap is 0 2020-01-16T00:13:27.130 INFO:tasks.rados.rados.0.smithi123.stdout:1259: seq_num 383 ranges {437081=638017,1679074=582789} 2020-01-16T00:13:27.136 INFO:tasks.rados.rados.0.smithi123.stdout:1259: writing smithi12313934-35 from 437081 to 1075098 tid 1 2020-01-16T00:13:27.143 INFO:tasks.rados.rados.0.smithi123.stdout:1259: writing smithi12313934-35 from 1679074 to 2261863 tid 2 2020-01-16T00:13:27.144 INFO:tasks.rados.rados.0.smithi123.stdout:1260: read oid 7 snap -1 2020-01-16T00:13:27.144 INFO:tasks.rados.rados.0.smithi123.stdout:1260: expect (ObjNum 352 snap 0 seq_num 352) 2020-01-16T00:13:27.144 INFO:tasks.rados.rados.0.smithi123.stdout:1261: read oid 42 snap -1 2020-01-16T00:13:27.144 INFO:tasks.rados.rados.0.smithi123.stdout:1261: expect (ObjNum 334 snap 0 seq_num 334) 2020-01-16T00:13:27.144 INFO:tasks.rados.rados.0.smithi123.stdout:1251: expect (ObjNum 371 snap 0 seq_num 371) 2020-01-16T00:13:27.183 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:27.180+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:27.217 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:27.224 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-01-16T00:13:27.303 INFO:tasks.rados.rados.0.smithi123.stdout:1262: read oid 34 snap -1 2020-01-16T00:13:27.303 INFO:tasks.rados.rados.0.smithi123.stdout:1262: expect deleted 2020-01-16T00:13:27.303 INFO:tasks.rados.rados.0.smithi123.stdout:1251: done (12 left) 2020-01-16T00:13:27.304 INFO:tasks.rados.rados.0.smithi123.stdout:1263: write oid 15 current snap is 0 2020-01-16T00:13:27.304 INFO:tasks.rados.rados.0.smithi123.stdout:1263: seq_num 384 ranges {453888=713266,1737556=442800} 2020-01-16T00:13:27.304 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:27.298+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:27.310 INFO:tasks.rados.rados.0.smithi123.stdout:1263: writing smithi12313934-15 from 453888 to 1167154 tid 1 2020-01-16T00:13:27.316 INFO:tasks.rados.rados.0.smithi123.stdout:1263: writing smithi12313934-15 from 1737556 to 2180356 tid 2 2020-01-16T00:13:27.317 INFO:tasks.rados.rados.0.smithi123.stdout:1264: read oid 26 snap -1 2020-01-16T00:13:27.317 INFO:tasks.rados.rados.0.smithi123.stdout:1264: expect (ObjNum 263 snap 0 seq_num 263) 2020-01-16T00:13:27.317 INFO:tasks.rados.rados.0.smithi123.stdout:1265: read oid 29 snap -1 2020-01-16T00:13:27.317 INFO:tasks.rados.rados.0.smithi123.stdout:1265: expect (ObjNum 364 snap 0 seq_num 364) 2020-01-16T00:13:27.317 INFO:tasks.rados.rados.0.smithi123.stdout:1266: write oid 50 current snap is 0 2020-01-16T00:13:27.318 INFO:tasks.rados.rados.0.smithi123.stdout:1266: seq_num 385 ranges {470695=788515,1679685=586458} 2020-01-16T00:13:27.324 INFO:tasks.rados.rados.0.smithi123.stdout:1266: writing smithi12313934-50 from 470695 to 1259210 tid 1 2020-01-16T00:13:27.330 INFO:tasks.rados.rados.0.smithi123.stdout:1266: writing smithi12313934-50 from 1679685 to 2266143 tid 2 2020-01-16T00:13:27.330 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:27.331 INFO:tasks.rados.rados.0.smithi123.stdout:1250: expect (ObjNum 356 snap 0 seq_num 356) 2020-01-16T00:13:27.381 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:27.384 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:27.382+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:27.389 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-01-16T00:13:27.486 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:27.482+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:27.538 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:27.546 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-01-16T00:13:27.591 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:27.588+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:27.659 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:27.660 INFO:tasks.rados.rados.0.smithi123.stdout:1252: expect (ObjNum 310 snap 0 seq_num 310) 2020-01-16T00:13:27.669 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-01-16T00:13:27.687 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:27.682+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:27.789 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:27.784+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:27.824 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:27.834 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-01-16T00:13:27.847 INFO:tasks.rados.rados.0.smithi123.stdout:1253: expect (ObjNum 220 snap 0 seq_num 220) 2020-01-16T00:13:27.893 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:27.890+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:27.915 INFO:teuthology.orchestra.run.smithi123.stderr:nodeep-scrub is unset 2020-01-16T00:13:27.988 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:27.994 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-01-16T00:13:27.995 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:27.990+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:28.093 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:28.095 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:28.090+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:28.102 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-01-16T00:13:28.171 INFO:tasks.rados.rados.0.smithi123.stdout:1254: expect (ObjNum 339 snap 0 seq_num 339) 2020-01-16T00:13:28.195 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:28.192+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:28.214 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:28.224 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-01-16T00:13:28.297 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:28.294+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:28.375 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:28.385 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-01-16T00:13:28.399 INFO:tasks.rados.rados.0.smithi123.stdout:1255: finishing write tid 1 to smithi12313934-10 2020-01-16T00:13:28.399 INFO:tasks.rados.rados.0.smithi123.stdout:1257: expect (ObjNum 366 snap 0 seq_num 366) 2020-01-16T00:13:28.431 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:28.428+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:28.496 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:28.499 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:28.494+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:28.505 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-01-16T00:13:28.522 INFO:tasks.rados.rados.0.smithi123.stdout:1256: expect (ObjNum 320 snap 0 seq_num 320) 2020-01-16T00:13:28.600 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:28.596+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:28.656 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:28.665 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-01-16T00:13:28.707 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:28.704+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:28.751 INFO:tasks.rados.rados.0.smithi123.stdout:1255: finishing write tid 2 to smithi12313934-10 2020-01-16T00:13:28.751 INFO:tasks.rados.rados.0.smithi123.stdout:1258: finishing write tid 1 to smithi12313934-31 2020-01-16T00:13:28.751 INFO:tasks.rados.rados.0.smithi123.stdout:1255: finishing write tid 4 to smithi12313934-10 2020-01-16T00:13:28.751 INFO:tasks.rados.rados.0.smithi123.stdout:1255: finishing write tid 5 to smithi12313934-10 2020-01-16T00:13:28.751 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 10 v 129 (ObjNum 381 snap 0 seq_num 381) dirty exists 2020-01-16T00:13:28.752 INFO:tasks.rados.rados.0.smithi123.stdout:1255: left oid 10 (ObjNum 381 snap 0 seq_num 381) 2020-01-16T00:13:28.752 INFO:tasks.rados.rados.0.smithi123.stdout:1258: finishing write tid 2 to smithi12313934-31 2020-01-16T00:13:28.752 INFO:tasks.rados.rados.0.smithi123.stdout:1258: finishing write tid 4 to smithi12313934-31 2020-01-16T00:13:28.753 INFO:tasks.rados.rados.0.smithi123.stdout:1250: done (15 left) 2020-01-16T00:13:28.753 INFO:tasks.rados.rados.0.smithi123.stdout:1252: done (14 left) 2020-01-16T00:13:28.753 INFO:tasks.rados.rados.0.smithi123.stdout:1253: done (13 left) 2020-01-16T00:13:28.753 INFO:tasks.rados.rados.0.smithi123.stdout:1254: done (12 left) 2020-01-16T00:13:28.753 INFO:tasks.rados.rados.0.smithi123.stdout:1255: done (11 left) 2020-01-16T00:13:28.754 INFO:tasks.rados.rados.0.smithi123.stdout:1256: done (10 left) 2020-01-16T00:13:28.754 INFO:tasks.rados.rados.0.smithi123.stdout:1257: done (9 left) 2020-01-16T00:13:28.754 INFO:tasks.rados.rados.0.smithi123.stdout:1267: write (excl) oid 13 current snap is 0 2020-01-16T00:13:28.754 INFO:tasks.rados.rados.0.smithi123.stdout:1267: seq_num 386 ranges {487502=463764,1738167=446469} 2020-01-16T00:13:28.764 INFO:tasks.rados.rados.0.smithi123.stdout:1267: writing smithi12313934-13 from 487502 to 951266 tid 1 2020-01-16T00:13:28.771 INFO:tasks.rados.rados.0.smithi123.stdout:1267: writing smithi12313934-13 from 1738167 to 2184636 tid 2 2020-01-16T00:13:28.771 INFO:tasks.rados.rados.0.smithi123.stdout:1268: write oid 4 current snap is 0 2020-01-16T00:13:28.772 INFO:tasks.rados.rados.0.smithi123.stdout:1258: finishing write tid 5 to smithi12313934-31 2020-01-16T00:13:28.772 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 31 v 148 (ObjNum 382 snap 0 seq_num 382) dirty exists 2020-01-16T00:13:28.772 INFO:tasks.rados.rados.0.smithi123.stdout:1258: left oid 31 (ObjNum 382 snap 0 seq_num 382) 2020-01-16T00:13:28.772 INFO:tasks.rados.rados.0.smithi123.stdout:1268: seq_num 387 ranges {504309=539013,1796649=590127} 2020-01-16T00:13:28.777 INFO:tasks.rados.rados.0.smithi123.stdout:1268: writing smithi12313934-4 from 504309 to 1043322 tid 1 2020-01-16T00:13:28.785 INFO:tasks.rados.rados.0.smithi123.stdout:1268: writing smithi12313934-4 from 1796649 to 2386776 tid 2 2020-01-16T00:13:28.785 INFO:tasks.rados.rados.0.smithi123.stdout:1258: done (10 left) 2020-01-16T00:13:28.785 INFO:tasks.rados.rados.0.smithi123.stdout:1269: write (excl) oid 3 current snap is 0 2020-01-16T00:13:28.786 INFO:tasks.rados.rados.0.smithi123.stdout:1269: seq_num 388 ranges {521116=730615,1971484=549632} 2020-01-16T00:13:28.792 INFO:tasks.rados.rados.0.smithi123.stdout:1269: writing smithi12313934-3 from 521116 to 1251731 tid 1 2020-01-16T00:13:28.800 INFO:tasks.rados.rados.0.smithi123.stdout:1269: writing smithi12313934-3 from 1971484 to 2521116 tid 2 2020-01-16T00:13:28.800 INFO:tasks.rados.rados.0.smithi123.stdout:1270: delete oid 22 current snap is 0 2020-01-16T00:13:28.800 INFO:tasks.rados.rados.0.smithi123.stdout:1259: finishing write tid 1 to smithi12313934-35 2020-01-16T00:13:28.800 INFO:tasks.rados.rados.0.smithi123.stdout:1259: finishing write tid 2 to smithi12313934-35 2020-01-16T00:13:28.800 INFO:tasks.rados.rados.0.smithi123.stdout:1259: finishing write tid 4 to smithi12313934-35 2020-01-16T00:13:28.801 INFO:tasks.rados.rados.0.smithi123.stdout:1259: finishing write tid 5 to smithi12313934-35 2020-01-16T00:13:28.801 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 35 v 142 (ObjNum 383 snap 0 seq_num 383) dirty exists 2020-01-16T00:13:28.801 INFO:tasks.rados.rados.0.smithi123.stdout:1259: left oid 35 (ObjNum 383 snap 0 seq_num 383) 2020-01-16T00:13:28.801 INFO:tasks.rados.rados.0.smithi123.stdout:1261: expect (ObjNum 334 snap 0 seq_num 334) 2020-01-16T00:13:28.803 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:28.798+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:28.820 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:28.830 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-01-16T00:13:28.904 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:28.900+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:28.933 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:28.942 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-01-16T00:13:28.953 INFO:tasks.rados.rados.0.smithi123.stdout:1260: expect (ObjNum 352 snap 0 seq_num 352) 2020-01-16T00:13:29.006 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:29.004+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:29.042 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:29.051 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-01-16T00:13:29.079 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-01-16T00:13:29.106 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:29.100+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:29.190 INFO:tasks.rados.rados.0.smithi123.stdout:1263: finishing write tid 1 to smithi12313934-15 2020-01-16T00:13:29.190 INFO:tasks.rados.rados.0.smithi123.stdout:1265: expect (ObjNum 364 snap 0 seq_num 364) 2020-01-16T00:13:29.196 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:29.206 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-01-16T00:13:29.207 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:29.202+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:29.218 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:13:29.235 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:13:29.253 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:13:29.284 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:13:29.301 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:13:29.301 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:29.302 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:29.302 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:13:29.302 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:29.308 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:29.304+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:29.310 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-01-16T00:13:29.325 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:13:29.347 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:13:29.348 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-01-16T00:13:29.348 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-01-16T00:13:29.348 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-01-16T00:13:29.349 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-01-16T00:13:29.349 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-01-16T00:13:29.349 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENXIO: problem getting command descriptions from osd.5 2020-01-16T00:13:29.349 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: problem getting command descriptions from osd.5 2020-01-16T00:13:29.349 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-01-16T00:13:29.349 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-01-16T00:13:29.357 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-01-16T00:13:29.357 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-01-16T00:13:29.372 INFO:tasks.rados.rados.0.smithi123.stdout:1264: expect (ObjNum 263 snap 0 seq_num 263) 2020-01-16T00:13:29.409 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:29.406+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:29.427 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:29.427 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:29.427 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:13:29.427 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:13:29.428 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:29.436 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-01-16T00:13:29.524 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:29.522+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:29.545 INFO:tasks.rados.rados.0.smithi123.stdout:1266: finishing write tid 1 to smithi12313934-50 2020-01-16T00:13:29.545 INFO:tasks.rados.rados.0.smithi123.stdout:1266: finishing write tid 2 to smithi12313934-50 2020-01-16T00:13:29.545 INFO:tasks.rados.rados.0.smithi123.stdout:1266: finishing write tid 4 to smithi12313934-50 2020-01-16T00:13:29.546 INFO:tasks.rados.rados.0.smithi123.stdout:1266: finishing write tid 5 to smithi12313934-50 2020-01-16T00:13:29.546 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 50 v 152 (ObjNum 385 snap 0 seq_num 385) dirty exists 2020-01-16T00:13:29.546 INFO:tasks.rados.rados.0.smithi123.stdout:1266: left oid 50 (ObjNum 385 snap 0 seq_num 385) 2020-01-16T00:13:29.546 INFO:tasks.rados.rados.0.smithi123.stdout:1263: finishing write tid 2 to smithi12313934-15 2020-01-16T00:13:29.546 INFO:tasks.rados.rados.0.smithi123.stdout:1263: finishing write tid 4 to smithi12313934-15 2020-01-16T00:13:29.546 INFO:tasks.rados.rados.0.smithi123.stdout:1263: finishing write tid 5 to smithi12313934-15 2020-01-16T00:13:29.546 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 15 v 155 (ObjNum 384 snap 0 seq_num 384) dirty exists 2020-01-16T00:13:29.547 INFO:tasks.rados.rados.0.smithi123.stdout:1263: left oid 15 (ObjNum 384 snap 0 seq_num 384) 2020-01-16T00:13:29.547 INFO:tasks.rados.rados.0.smithi123.stdout:1267: finishing write tid 1 to smithi12313934-13 2020-01-16T00:13:29.547 INFO:tasks.rados.rados.0.smithi123.stdout:1267: finishing write tid 2 to smithi12313934-13 2020-01-16T00:13:29.547 INFO:tasks.rados.rados.0.smithi123.stdout:1268: finishing write tid 1 to smithi12313934-4 2020-01-16T00:13:29.547 INFO:tasks.rados.rados.0.smithi123.stdout:1268: finishing write tid 2 to smithi12313934-4 2020-01-16T00:13:29.547 INFO:tasks.rados.rados.0.smithi123.stdout:1268: finishing write tid 4 to smithi12313934-4 2020-01-16T00:13:29.547 INFO:tasks.rados.rados.0.smithi123.stdout:1268: finishing write tid 5 to smithi12313934-4 2020-01-16T00:13:29.548 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 4 v 145 (ObjNum 387 snap 0 seq_num 387) dirty exists 2020-01-16T00:13:29.548 INFO:tasks.rados.rados.0.smithi123.stdout:1268: left oid 4 (ObjNum 387 snap 0 seq_num 387) 2020-01-16T00:13:29.548 INFO:tasks.rados.rados.0.smithi123.stdout:1267: finishing write tid 4 to smithi12313934-13 2020-01-16T00:13:29.548 INFO:tasks.rados.rados.0.smithi123.stdout:1267: finishing write tid 5 to smithi12313934-13 2020-01-16T00:13:29.548 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 82 (ObjNum 386 snap 0 seq_num 386) dirty exists 2020-01-16T00:13:29.548 INFO:tasks.rados.rados.0.smithi123.stdout:1267: left oid 13 (ObjNum 386 snap 0 seq_num 386) 2020-01-16T00:13:29.548 INFO:tasks.rados.rados.0.smithi123.stdout:1269: finishing write tid 1 to smithi12313934-3 2020-01-16T00:13:29.548 INFO:tasks.rados.rados.0.smithi123.stdout:1269: finishing write tid 2 to smithi12313934-3 2020-01-16T00:13:29.549 INFO:tasks.rados.rados.0.smithi123.stdout:1269: finishing write tid 4 to smithi12313934-3 2020-01-16T00:13:29.549 INFO:tasks.rados.rados.0.smithi123.stdout:1269: finishing write tid 5 to smithi12313934-3 2020-01-16T00:13:29.549 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 3 v 85 (ObjNum 388 snap 0 seq_num 388) dirty exists 2020-01-16T00:13:29.549 INFO:tasks.rados.rados.0.smithi123.stdout:1269: left oid 3 (ObjNum 388 snap 0 seq_num 388) 2020-01-16T00:13:29.549 INFO:tasks.rados.rados.0.smithi123.stdout:1259: done (11 left) 2020-01-16T00:13:29.549 INFO:tasks.rados.rados.0.smithi123.stdout:1260: done (10 left) 2020-01-16T00:13:29.549 INFO:tasks.rados.rados.0.smithi123.stdout:1261: done (9 left) 2020-01-16T00:13:29.550 INFO:tasks.rados.rados.0.smithi123.stdout:1262: done (8 left) 2020-01-16T00:13:29.550 INFO:tasks.rados.rados.0.smithi123.stdout:1263: done (7 left) 2020-01-16T00:13:29.550 INFO:tasks.rados.rados.0.smithi123.stdout:1264: done (6 left) 2020-01-16T00:13:29.550 INFO:tasks.rados.rados.0.smithi123.stdout:1265: done (5 left) 2020-01-16T00:13:29.550 INFO:tasks.rados.rados.0.smithi123.stdout:1266: done (4 left) 2020-01-16T00:13:29.550 INFO:tasks.rados.rados.0.smithi123.stdout:1267: done (3 left) 2020-01-16T00:13:29.550 INFO:tasks.rados.rados.0.smithi123.stdout:1268: done (2 left) 2020-01-16T00:13:29.551 INFO:tasks.rados.rados.0.smithi123.stdout:1269: done (1 left) 2020-01-16T00:13:29.551 INFO:tasks.rados.rados.0.smithi123.stdout:1270: done (0 left) 2020-01-16T00:13:29.551 INFO:tasks.rados.rados.0.smithi123.stdout:1271: write oid 40 current snap is 0 2020-01-16T00:13:29.551 INFO:tasks.rados.rados.0.smithi123.stdout:1271: seq_num 390 ranges {554730=481113,1572095=737454} 2020-01-16T00:13:29.560 INFO:tasks.rados.rados.0.smithi123.stdout:1271: writing smithi12313934-40 from 554730 to 1035843 tid 1 2020-01-16T00:13:29.568 INFO:tasks.rados.rados.0.smithi123.stdout:1271: writing smithi12313934-40 from 1572095 to 2309549 tid 2 2020-01-16T00:13:29.568 INFO:tasks.rados.rados.0.smithi123.stdout:1272: read oid 23 snap -1 2020-01-16T00:13:29.568 INFO:tasks.rados.rados.0.smithi123.stdout:1272: expect deleted 2020-01-16T00:13:29.568 INFO:tasks.rados.rados.0.smithi123.stdout:1273: write oid 46 current snap is 0 2020-01-16T00:13:29.568 INFO:tasks.rados.rados.0.smithi123.stdout:1273: seq_num 391 ranges {571537=556362,1630577=597465} 2020-01-16T00:13:29.574 INFO:tasks.rados.rados.0.smithi123.stdout:1273: writing smithi12313934-46 from 571537 to 1127899 tid 1 2020-01-16T00:13:29.581 INFO:tasks.rados.rados.0.smithi123.stdout:1273: writing smithi12313934-46 from 1630577 to 2228042 tid 2 2020-01-16T00:13:29.581 INFO:tasks.rados.rados.0.smithi123.stdout:1274: read oid 45 snap -1 2020-01-16T00:13:29.581 INFO:tasks.rados.rados.0.smithi123.stdout:1274: expect (ObjNum 335 snap 0 seq_num 335) 2020-01-16T00:13:29.582 INFO:tasks.rados.rados.0.smithi123.stdout:1275: read oid 15 snap -1 2020-01-16T00:13:29.582 INFO:tasks.rados.rados.0.smithi123.stdout:1275: expect (ObjNum 384 snap 0 seq_num 384) 2020-01-16T00:13:29.582 INFO:tasks.rados.rados.0.smithi123.stdout:1276: read oid 10 snap -1 2020-01-16T00:13:29.582 INFO:tasks.rados.rados.0.smithi123.stdout:1276: expect (ObjNum 381 snap 0 seq_num 381) 2020-01-16T00:13:29.582 INFO:tasks.rados.rados.0.smithi123.stdout:1277: read oid 41 snap -1 2020-01-16T00:13:29.582 INFO:tasks.rados.rados.0.smithi123.stdout:1277: expect (ObjNum 310 snap 0 seq_num 310) 2020-01-16T00:13:29.582 INFO:tasks.rados.rados.0.smithi123.stdout:1278: read oid 9 snap -1 2020-01-16T00:13:29.583 INFO:tasks.rados.rados.0.smithi123.stdout:1278: expect (ObjNum 372 snap 0 seq_num 372) 2020-01-16T00:13:29.583 INFO:tasks.rados.rados.0.smithi123.stdout:1279: write (excl) oid 13 current snap is 0 2020-01-16T00:13:29.583 INFO:tasks.rados.rados.0.smithi123.stdout:1279: seq_num 392 ranges {588344=631611,1689059=741123} 2020-01-16T00:13:29.589 INFO:tasks.rados.rados.0.smithi123.stdout:1279: writing smithi12313934-13 from 588344 to 1219955 tid 1 2020-01-16T00:13:29.593 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:29.593 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:13:29.593 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:13:29.593 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:13:29.594 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.594 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.5.13:40 2.0 e24/19 2:20e40cec:::smithi12313934-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 24'17, mlcod=23'15)", 2020-01-16T00:13:29.594 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.849607+0000", 2020-01-16T00:13:29.594 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 51.740196603999998, 2020-01-16T00:13:29.594 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083979039999999994, 2020-01-16T00:13:29.594 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:29.594 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:29.595 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:29.595 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.595 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:29.595 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.849607+0000", 2020-01-16T00:13:29.595 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:29.595 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.595 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.596 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:29.596 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.849607+0000", 2020-01-16T00:13:29.596 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:29.596 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.596 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.596 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:29.596 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.849605+0000", 2020-01-16T00:13:29.596 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0037534560000000001 2020-01-16T00:13:29.597 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.597 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.597 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:29.597 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.853358+0000", 2020-01-16T00:13:29.597 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9181000000000002e-05 2020-01-16T00:13:29.597 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.597 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.598 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:29.598 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.853377+0000", 2020-01-16T00:13:29.598 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0040000000000002e-06 2020-01-16T00:13:29.598 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.598 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.598 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:29.598 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.853384+0000", 2020-01-16T00:13:29.598 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2583000000000001e-05 2020-01-16T00:13:29.599 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.599 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.599 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:29.599 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.853437+0000", 2020-01-16T00:13:29.599 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8212999999999999e-05 2020-01-16T00:13:29.599 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.599 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.600 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:29.600 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.853465+0000", 2020-01-16T00:13:29.600 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0044847589999999996 2020-01-16T00:13:29.600 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.600 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.600 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:29.600 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.857950+0000", 2020-01-16T00:13:29.600 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5309999999999997e-05 2020-01-16T00:13:29.601 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.601 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.601 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:29.601 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.858005+0000", 2020-01-16T00:13:29.601 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083979039999999994 2020-01-16T00:13:29.601 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.601 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:29.601 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.602 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.602 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.602 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.5.13:42 2.0 e24/19 2:313458df:::smithi12313934-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 24'18, mlcod=24'16)", 2020-01-16T00:13:29.602 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.940683+0000", 2020-01-16T00:13:29.602 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 51.649120764999999, 2020-01-16T00:13:29.602 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0097026960000000002, 2020-01-16T00:13:29.602 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:29.603 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:29.603 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:29.603 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.603 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:29.603 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.940683+0000", 2020-01-16T00:13:29.603 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:29.603 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.603 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.604 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:29.604 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.940683+0000", 2020-01-16T00:13:29.604 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:29.604 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.604 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.604 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:29.604 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.940680+0000", 2020-01-16T00:13:29.604 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003126775 2020-01-16T00:13:29.605 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.605 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.605 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:29.605 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.943807+0000", 2020-01-16T00:13:29.605 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5426999999999999e-05 2020-01-16T00:13:29.605 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.605 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.605 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:29.606 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.943833+0000", 2020-01-16T00:13:29.606 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7409999999999998e-06 2020-01-16T00:13:29.606 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.606 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.606 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:29.606 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.943841+0000", 2020-01-16T00:13:29.606 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011645399999999999 2020-01-16T00:13:29.607 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.607 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.607 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:29.607 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.943958+0000", 2020-01-16T00:13:29.607 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7291000000000003e-05 2020-01-16T00:13:29.607 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.607 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.607 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:29.608 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.944025+0000", 2020-01-16T00:13:29.608 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063108449999999998 2020-01-16T00:13:29.608 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.608 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.608 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:29.608 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.950336+0000", 2020-01-16T00:13:29.608 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9874999999999999e-05 2020-01-16T00:13:29.608 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.609 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.609 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:29.609 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.950386+0000", 2020-01-16T00:13:29.609 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0097026960000000002 2020-01-16T00:13:29.609 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.609 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:29.609 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.609 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.610 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.610 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1388 3.1 3:8c6a83c0:::smithi12313934-36:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e28)", 2020-01-16T00:13:29.610 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:46.605253+0000", 2020-01-16T00:13:29.610 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 42.984551414999999, 2020-01-16T00:13:29.610 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0080920829999999999, 2020-01-16T00:13:29.610 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:29.610 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:29.611 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:29.611 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:29.611 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:29.611 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1388 2020-01-16T00:13:29.611 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.611 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:29.611 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.611 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:29.612 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605253+0000", 2020-01-16T00:13:29.612 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:29.612 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.612 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.612 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:29.612 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605253+0000", 2020-01-16T00:13:29.612 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:29.612 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.613 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.613 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:29.613 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605252+0000", 2020-01-16T00:13:29.613 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6559999999999997e-06 2020-01-16T00:13:29.613 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.613 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.613 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:29.614 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605257+0000", 2020-01-16T00:13:29.614 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0966e-05 2020-01-16T00:13:29.614 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.614 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.614 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:29.614 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605268+0000", 2020-01-16T00:13:29.615 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5960000000000003e-06 2020-01-16T00:13:29.615 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.615 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.615 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:29.615 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.605274+0000", 2020-01-16T00:13:29.615 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.007370176 2020-01-16T00:13:29.615 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.615 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.616 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:29.616 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.612644+0000", 2020-01-16T00:13:29.616 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000113146 2020-01-16T00:13:29.616 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.616 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.616 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:29.616 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.612757+0000", 2020-01-16T00:13:29.617 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00058744799999999996 2020-01-16T00:13:29.617 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.617 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.617 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:29.617 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.613345+0000", 2020-01-16T00:13:29.617 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0080920829999999999 2020-01-16T00:13:29.617 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.618 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:29.618 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.618 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.618 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.618 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1644 3.1 3:8c6a83c0:::smithi12313934-36:head [read 0~2705927,checksum] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:13:29.618 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847556+0000", 2020-01-16T00:13:29.618 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 37.742247998000003, 2020-01-16T00:13:29.619 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0081843639999999995, 2020-01-16T00:13:29.619 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:29.619 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:29.619 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:29.619 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:29.619 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:29.619 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1644 2020-01-16T00:13:29.619 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.620 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:29.620 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.620 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:29.620 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847556+0000", 2020-01-16T00:13:29.620 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:29.620 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.620 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.620 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:29.621 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847556+0000", 2020-01-16T00:13:29.621 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:29.621 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.621 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.621 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:29.621 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847555+0000", 2020-01-16T00:13:29.621 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4640000000000002e-06 2020-01-16T00:13:29.622 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.622 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.622 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:29.622 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847559+0000", 2020-01-16T00:13:29.622 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4580000000000003e-06 2020-01-16T00:13:29.622 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.622 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.622 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:29.623 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847563+0000", 2020-01-16T00:13:29.623 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.613e-06 2020-01-16T00:13:29.623 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.623 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.623 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:29.623 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847567+0000", 2020-01-16T00:13:29.623 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000145045 2020-01-16T00:13:29.623 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.624 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.624 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:29.624 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847712+0000", 2020-01-16T00:13:29.624 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6480000000000003e-06 2020-01-16T00:13:29.624 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.624 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.624 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:13:29.624 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847720+0000", 2020-01-16T00:13:29.625 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0055645269999999997 2020-01-16T00:13:29.625 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.625 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.625 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:29.625 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.853284+0000", 2020-01-16T00:13:29.625 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00026767400000000003 2020-01-16T00:13:29.625 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.626 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.626 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:29.626 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.853552+0000", 2020-01-16T00:13:29.626 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021884169999999998 2020-01-16T00:13:29.626 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.626 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.626 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:29.626 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.855740+0000", 2020-01-16T00:13:29.627 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0081843639999999995 2020-01-16T00:13:29.627 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.627 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:29.627 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.627 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.628 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.628 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1645 3.1 3:8c6a83c0:::smithi12313934-36:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:13:29.628 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847602+0000", 2020-01-16T00:13:29.629 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 37.742202198999998, 2020-01-16T00:13:29.629 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0085169679999999998, 2020-01-16T00:13:29.629 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:29.629 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:29.629 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:29.629 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:29.629 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:29.630 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1645 2020-01-16T00:13:29.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.630 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:29.630 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.630 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:29.630 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847602+0000", 2020-01-16T00:13:29.630 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:29.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.631 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.631 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:29.631 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847602+0000", 2020-01-16T00:13:29.631 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:29.631 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.631 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.631 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:29.632 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847601+0000", 2020-01-16T00:13:29.632 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7011e-05 2020-01-16T00:13:29.632 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.632 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.632 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:29.632 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847618+0000", 2020-01-16T00:13:29.632 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.187e-06 2020-01-16T00:13:29.632 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.633 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.633 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:29.633 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847625+0000", 2020-01-16T00:13:29.633 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9879999999999998e-06 2020-01-16T00:13:29.633 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.633 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.633 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:29.634 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847629+0000", 2020-01-16T00:13:29.634 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000129694 2020-01-16T00:13:29.634 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.634 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.634 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:29.634 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847759+0000", 2020-01-16T00:13:29.634 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8379999999999996e-06 2020-01-16T00:13:29.634 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.635 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.635 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:13:29.635 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847766+0000", 2020-01-16T00:13:29.635 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0079977390000000002 2020-01-16T00:13:29.635 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.635 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.635 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:29.636 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.855764+0000", 2020-01-16T00:13:29.636 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2467999999999996e-05 2020-01-16T00:13:29.636 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.636 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.636 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:29.636 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.855856+0000", 2020-01-16T00:13:29.636 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00026275099999999998 2020-01-16T00:13:29.636 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.637 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.637 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:29.637 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.856119+0000", 2020-01-16T00:13:29.637 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0085169679999999998 2020-01-16T00:13:29.637 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.637 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:29.637 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.638 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.638 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.638 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1649 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~2084068,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:13:29.638 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847683+0000", 2020-01-16T00:13:29.638 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 37.742120948, 2020-01-16T00:13:29.638 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010654691000000001, 2020-01-16T00:13:29.638 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:29.638 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:29.639 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:29.639 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:29.639 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:29.639 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1649 2020-01-16T00:13:29.639 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.639 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:29.639 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.640 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:29.640 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847683+0000", 2020-01-16T00:13:29.640 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:29.640 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.640 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.640 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:29.640 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847683+0000", 2020-01-16T00:13:29.641 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:29.641 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.641 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.641 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:29.641 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847682+0000", 2020-01-16T00:13:29.641 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8569999999999997e-06 2020-01-16T00:13:29.641 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.642 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.642 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:29.642 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847689+0000", 2020-01-16T00:13:29.642 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8090000000000004e-06 2020-01-16T00:13:29.642 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.642 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.642 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:29.642 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847696+0000", 2020-01-16T00:13:29.643 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2690000000000001e-06 2020-01-16T00:13:29.643 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.643 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.643 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:29.643 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847700+0000", 2020-01-16T00:13:29.643 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8189999999999993e-05 2020-01-16T00:13:29.643 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.644 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.644 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:29.644 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847798+0000", 2020-01-16T00:13:29.644 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7129999999999999e-06 2020-01-16T00:13:29.644 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.644 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.644 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:13:29.645 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847802+0000", 2020-01-16T00:13:29.645 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083913049999999999 2020-01-16T00:13:29.645 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.645 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.645 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:29.645 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.856193+0000", 2020-01-16T00:13:29.645 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034400300000000001 2020-01-16T00:13:29.645 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.646 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.646 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:29.646 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.856537+0000", 2020-01-16T00:13:29.646 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018006820000000001 2020-01-16T00:13:29.646 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.646 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.646 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:29.646 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858338+0000", 2020-01-16T00:13:29.647 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010654691000000001 2020-01-16T00:13:29.647 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.647 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:29.647 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.647 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.647 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.647 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1650 3.1 3:8de1d8a3:::smithi12313934-13:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:13:29.648 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847717+0000", 2020-01-16T00:13:29.648 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 37.742086995000001, 2020-01-16T00:13:29.648 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010913658, 2020-01-16T00:13:29.648 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:29.648 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:29.648 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:29.648 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:29.648 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:29.649 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1650 2020-01-16T00:13:29.649 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.649 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:29.649 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.649 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:29.649 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847717+0000", 2020-01-16T00:13:29.649 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:29.650 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.650 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.650 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:29.650 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847717+0000", 2020-01-16T00:13:29.650 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:29.650 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.650 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.651 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:29.651 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847716+0000", 2020-01-16T00:13:29.651 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.433e-06 2020-01-16T00:13:29.651 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.651 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.651 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:29.651 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847720+0000", 2020-01-16T00:13:29.651 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1800000000000004e-06 2020-01-16T00:13:29.652 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.652 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.652 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:29.652 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847725+0000", 2020-01-16T00:13:29.652 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6689999999999997e-06 2020-01-16T00:13:29.652 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.652 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.653 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:29.653 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847730+0000", 2020-01-16T00:13:29.653 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000102468 2020-01-16T00:13:29.653 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.653 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.653 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:29.653 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847832+0000", 2020-01-16T00:13:29.654 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3999999999999997e-06 2020-01-16T00:13:29.654 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.654 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.654 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:13:29.654 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847838+0000", 2020-01-16T00:13:29.654 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010519395000000001 2020-01-16T00:13:29.654 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.654 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.655 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:29.655 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858358+0000", 2020-01-16T00:13:29.655 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7600000000000002e-05 2020-01-16T00:13:29.655 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.655 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.655 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:29.655 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858445+0000", 2020-01-16T00:13:29.655 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018515900000000001 2020-01-16T00:13:29.656 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.656 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.656 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:29.656 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858631+0000", 2020-01-16T00:13:29.656 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010913658 2020-01-16T00:13:29.656 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.656 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:29.656 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.657 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.657 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.657 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1651 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~2084068,checksum] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:13:29.657 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847752+0000", 2020-01-16T00:13:29.657 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 37.742052379, 2020-01-16T00:13:29.657 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011734249, 2020-01-16T00:13:29.657 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:29.658 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:29.658 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:29.658 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:29.658 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:29.658 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1651 2020-01-16T00:13:29.658 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.658 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:29.658 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.659 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:29.659 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847752+0000", 2020-01-16T00:13:29.659 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:29.659 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.659 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.659 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:29.659 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847752+0000", 2020-01-16T00:13:29.659 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:29.660 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.660 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.660 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:29.660 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847751+0000", 2020-01-16T00:13:29.660 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1980000000000001e-06 2020-01-16T00:13:29.660 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.660 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.661 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:29.661 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847754+0000", 2020-01-16T00:13:29.661 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6679999999999999e-06 2020-01-16T00:13:29.661 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.661 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.661 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:29.661 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847758+0000", 2020-01-16T00:13:29.661 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2500000000000001e-06 2020-01-16T00:13:29.662 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.662 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.662 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:29.662 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847760+0000", 2020-01-16T00:13:29.662 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7614000000000004e-05 2020-01-16T00:13:29.662 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.662 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.662 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:29.663 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847858+0000", 2020-01-16T00:13:29.663 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7950000000000004e-06 2020-01-16T00:13:29.663 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.663 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.663 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:13:29.663 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847864+0000", 2020-01-16T00:13:29.663 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010790345 2020-01-16T00:13:29.663 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.664 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.664 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:29.664 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858654+0000", 2020-01-16T00:13:29.664 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000141466 2020-01-16T00:13:29.664 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.664 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.664 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:29.665 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858795+0000", 2020-01-16T00:13:29.665 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00069037899999999997 2020-01-16T00:13:29.665 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.665 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.665 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:29.665 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.859486+0000", 2020-01-16T00:13:29.665 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011734249 2020-01-16T00:13:29.665 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.666 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:29.666 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.666 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.666 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.666 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:55 2.1 e30/19 2:91f85463:::smithi12313934-24 low_tier:head v 30'56, mlcod=29'54)", 2020-01-16T00:13:29.666 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.738134+0000", 2020-01-16T00:13:29.666 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 36.851670093000003, 2020-01-16T00:13:29.666 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0086509570000000008, 2020-01-16T00:13:29.667 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:29.667 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:29.667 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:29.667 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.667 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:29.667 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738134+0000", 2020-01-16T00:13:29.667 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:29.668 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.668 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.668 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:29.668 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738134+0000", 2020-01-16T00:13:29.668 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:29.668 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.668 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.668 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:29.669 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738134+0000", 2020-01-16T00:13:29.669 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00053712599999999997 2020-01-16T00:13:29.669 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.669 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.669 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:29.669 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738671+0000", 2020-01-16T00:13:29.669 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4059999999999999e-05 2020-01-16T00:13:29.670 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.670 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.670 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:29.670 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738685+0000", 2020-01-16T00:13:29.670 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7680000000000008e-06 2020-01-16T00:13:29.670 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.670 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.670 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:29.671 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738692+0000", 2020-01-16T00:13:29.671 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1273999999999996e-05 2020-01-16T00:13:29.671 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.671 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.671 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:29.671 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738764+0000", 2020-01-16T00:13:29.671 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6827999999999999e-05 2020-01-16T00:13:29.671 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.672 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.672 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:29.672 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738781+0000", 2020-01-16T00:13:29.672 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0079370740000000006 2020-01-16T00:13:29.672 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.672 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.672 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:29.673 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746718+0000", 2020-01-16T00:13:29.673 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7181000000000006e-05 2020-01-16T00:13:29.673 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.673 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.673 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:29.673 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746785+0000", 2020-01-16T00:13:29.673 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0086509570000000008 2020-01-16T00:13:29.673 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.674 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:29.674 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.674 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.674 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.674 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2106 3.1 3:8de1d8a3:::smithi12313934-13:head [write 665610~797860,stat] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:13:29.674 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.631369+0000", 2020-01-16T00:13:29.674 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 27.958434666999999, 2020-01-16T00:13:29.674 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0091195330000000008, 2020-01-16T00:13:29.675 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:29.675 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:29.675 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:29.675 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:29.675 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:29.675 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2106 2020-01-16T00:13:29.675 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.675 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:29.676 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.676 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:29.676 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631369+0000", 2020-01-16T00:13:29.676 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:29.676 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.676 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.676 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:29.677 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631369+0000", 2020-01-16T00:13:29.677 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:29.677 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.677 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.677 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:29.677 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631367+0000", 2020-01-16T00:13:29.677 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00025714100000000001 2020-01-16T00:13:29.678 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.678 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.678 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:29.678 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631624+0000", 2020-01-16T00:13:29.678 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4946000000000001e-05 2020-01-16T00:13:29.678 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.678 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.678 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:29.679 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631639+0000", 2020-01-16T00:13:29.679 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7050000000000002e-06 2020-01-16T00:13:29.679 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.679 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.679 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:29.679 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631646+0000", 2020-01-16T00:13:29.679 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010938900000000001 2020-01-16T00:13:29.679 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.680 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.680 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:29.680 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631755+0000", 2020-01-16T00:13:29.680 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8848e-05 2020-01-16T00:13:29.680 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.680 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.680 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op must wait for map", 2020-01-16T00:13:29.680 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631774+0000", 2020-01-16T00:13:29.681 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0026027730000000001 2020-01-16T00:13:29.681 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.681 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.681 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:29.681 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.634377+0000", 2020-01-16T00:13:29.681 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9282000000000003e-05 2020-01-16T00:13:29.681 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.682 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.682 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:29.682 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.634466+0000", 2020-01-16T00:13:29.682 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000127868 2020-01-16T00:13:29.682 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.682 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.682 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:13:29.682 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.634594+0000", 2020-01-16T00:13:29.683 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030776549999999999 2020-01-16T00:13:29.683 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.683 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.683 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:29.683 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.637672+0000", 2020-01-16T00:13:29.683 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001634195 2020-01-16T00:13:29.683 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.683 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.684 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:29.684 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.639306+0000", 2020-01-16T00:13:29.684 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010810069999999999 2020-01-16T00:13:29.684 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.684 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.684 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:29.685 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640387+0000", 2020-01-16T00:13:29.685 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0543000000000001e-05 2020-01-16T00:13:29.685 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.685 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.685 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:29.685 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640428+0000", 2020-01-16T00:13:29.685 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1321000000000005e-05 2020-01-16T00:13:29.686 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.686 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.686 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:29.686 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640489+0000", 2020-01-16T00:13:29.686 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0091195330000000008 2020-01-16T00:13:29.686 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.686 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:29.686 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.687 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.687 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.687 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2107 3.1 3:8de1d8a3:::smithi12313934-13:head [write 1909329~458405] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:13:29.687 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.636786+0000", 2020-01-16T00:13:29.687 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 27.953017537000001, 2020-01-16T00:13:29.687 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093182809999999994, 2020-01-16T00:13:29.687 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:29.688 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:29.688 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:29.688 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:29.688 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:29.688 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2107 2020-01-16T00:13:29.688 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.689 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:29.689 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.689 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:29.689 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636786+0000", 2020-01-16T00:13:29.689 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:29.689 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.690 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.690 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:29.690 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636786+0000", 2020-01-16T00:13:29.690 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:29.690 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.690 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.690 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:29.690 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636784+0000", 2020-01-16T00:13:29.691 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014276299999999999 2020-01-16T00:13:29.691 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.691 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.691 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:29.691 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636927+0000", 2020-01-16T00:13:29.691 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5017999999999999e-05 2020-01-16T00:13:29.691 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.691 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.692 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:29.692 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636942+0000", 2020-01-16T00:13:29.692 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5589999999999996e-06 2020-01-16T00:13:29.692 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.692 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.692 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:29.692 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636947+0000", 2020-01-16T00:13:29.693 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00040279699999999999 2020-01-16T00:13:29.693 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.693 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.693 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:29.693 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.637350+0000", 2020-01-16T00:13:29.693 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000233506 2020-01-16T00:13:29.693 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.693 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.694 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:29.694 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.637584+0000", 2020-01-16T00:13:29.694 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029534660000000001 2020-01-16T00:13:29.694 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.694 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.694 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:29.694 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640537+0000", 2020-01-16T00:13:29.695 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012577700000000001 2020-01-16T00:13:29.695 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.695 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.695 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:29.695 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640663+0000", 2020-01-16T00:13:29.695 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000198593 2020-01-16T00:13:29.695 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.696 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.696 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:13:29.696 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640862+0000", 2020-01-16T00:13:29.696 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.005088197 2020-01-16T00:13:29.696 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.696 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.696 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:29.696 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645950+0000", 2020-01-16T00:13:29.697 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2568999999999999e-05 2020-01-16T00:13:29.697 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.697 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.697 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:29.697 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645962+0000", 2020-01-16T00:13:29.697 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3034999999999995e-05 2020-01-16T00:13:29.697 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.698 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.698 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:29.698 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646025+0000", 2020-01-16T00:13:29.698 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7762999999999997e-05 2020-01-16T00:13:29.698 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.698 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.698 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:29.698 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646063+0000", 2020-01-16T00:13:29.699 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1541000000000001e-05 2020-01-16T00:13:29.699 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.699 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.699 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:29.699 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646105+0000", 2020-01-16T00:13:29.699 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093182809999999994 2020-01-16T00:13:29.699 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.700 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:29.700 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.700 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.700 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.700 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2108 3.1 3:8de1d8a3:::smithi12313934-13:head [write 2816700~410210] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:13:29.700 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.641861+0000", 2020-01-16T00:13:29.700 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 27.947943189, 2020-01-16T00:13:29.700 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018545604, 2020-01-16T00:13:29.701 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:29.701 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:29.701 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:29.701 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:29.701 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:29.701 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2108 2020-01-16T00:13:29.701 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.702 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:29.702 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.702 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:29.702 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.641861+0000", 2020-01-16T00:13:29.702 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:29.702 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.702 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.702 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:29.703 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.641861+0000", 2020-01-16T00:13:29.703 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:29.703 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.703 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.703 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:29.703 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.641859+0000", 2020-01-16T00:13:29.703 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018466399999999999 2020-01-16T00:13:29.704 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.704 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.704 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:29.704 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642043+0000", 2020-01-16T00:13:29.704 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2999999999999999e-05 2020-01-16T00:13:29.704 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.704 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.704 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:29.705 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642056+0000", 2020-01-16T00:13:29.705 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0730000000000005e-06 2020-01-16T00:13:29.705 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.705 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.705 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:29.705 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642064+0000", 2020-01-16T00:13:29.705 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9527999999999997e-05 2020-01-16T00:13:29.705 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.706 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.706 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:29.706 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642114+0000", 2020-01-16T00:13:29.706 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036616050000000001 2020-01-16T00:13:29.706 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.706 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.706 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:29.707 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645776+0000", 2020-01-16T00:13:29.707 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012222663999999999 2020-01-16T00:13:29.707 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.707 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.707 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:29.707 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.657998+0000", 2020-01-16T00:13:29.707 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014280299999999999 2020-01-16T00:13:29.708 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.708 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.708 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:29.708 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.658141+0000", 2020-01-16T00:13:29.708 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0001102 2020-01-16T00:13:29.708 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.708 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.708 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:13:29.709 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.658251+0000", 2020-01-16T00:13:29.709 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020421219999999999 2020-01-16T00:13:29.709 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.709 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.709 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:29.709 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660293+0000", 2020-01-16T00:13:29.709 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7399999999999999e-06 2020-01-16T00:13:29.710 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.710 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.710 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:29.710 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660303+0000", 2020-01-16T00:13:29.710 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9873999999999998e-05 2020-01-16T00:13:29.710 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.710 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.710 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:29.711 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660343+0000", 2020-01-16T00:13:29.711 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4448e-05 2020-01-16T00:13:29.711 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.711 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.711 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:29.711 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660377+0000", 2020-01-16T00:13:29.711 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8915999999999999e-05 2020-01-16T00:13:29.712 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.712 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.712 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:29.712 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660406+0000", 2020-01-16T00:13:29.712 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018545604 2020-01-16T00:13:29.712 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.712 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:29.712 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.713 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.713 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.713 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2109 3.1 3:8de1d8a3:::smithi12313934-13:head [setxattr _header (56) in=7b,truncate 3865610] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:13:29.713 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.642075+0000", 2020-01-16T00:13:29.713 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 27.947728615999999, 2020-01-16T00:13:29.713 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018477655999999999, 2020-01-16T00:13:29.713 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:29.714 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:29.714 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:29.714 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:29.714 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:29.714 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2109 2020-01-16T00:13:29.714 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.715 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:29.715 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.715 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:29.715 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642075+0000", 2020-01-16T00:13:29.715 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:29.715 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.715 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.716 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:29.716 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642075+0000", 2020-01-16T00:13:29.716 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:29.716 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.716 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.716 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:29.716 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642074+0000", 2020-01-16T00:13:29.717 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4500000000000003e-06 2020-01-16T00:13:29.717 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.717 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.717 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:29.717 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642080+0000", 2020-01-16T00:13:29.717 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7229999999999998e-06 2020-01-16T00:13:29.717 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.718 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.718 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:29.718 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642087+0000", 2020-01-16T00:13:29.718 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2390000000000004e-06 2020-01-16T00:13:29.718 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.718 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.718 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:29.719 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642091+0000", 2020-01-16T00:13:29.719 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0037396550000000001 2020-01-16T00:13:29.719 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.719 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.719 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:29.719 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645830+0000", 2020-01-16T00:13:29.719 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2200000000000002e-05 2020-01-16T00:13:29.719 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.720 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.720 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:29.720 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645883+0000", 2020-01-16T00:13:29.720 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013432934000000001 2020-01-16T00:13:29.720 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.720 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.720 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:29.721 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659316+0000", 2020-01-16T00:13:29.721 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000140006 2020-01-16T00:13:29.721 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.721 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.721 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:29.721 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659456+0000", 2020-01-16T00:13:29.721 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000184873 2020-01-16T00:13:29.722 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.722 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.722 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:13:29.722 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659640+0000", 2020-01-16T00:13:29.722 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00066954999999999998 2020-01-16T00:13:29.722 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.722 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.723 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:29.723 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660310+0000", 2020-01-16T00:13:29.723 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014847699999999999 2020-01-16T00:13:29.723 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.723 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.723 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:29.723 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660459+0000", 2020-01-16T00:13:29.723 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6869e-05 2020-01-16T00:13:29.724 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.724 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.724 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:29.724 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660485+0000", 2020-01-16T00:13:29.724 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8478000000000001e-05 2020-01-16T00:13:29.724 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.724 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.725 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:29.725 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660514+0000", 2020-01-16T00:13:29.725 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9153999999999998e-05 2020-01-16T00:13:29.725 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.725 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.725 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:29.725 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660553+0000", 2020-01-16T00:13:29.725 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018477655999999999 2020-01-16T00:13:29.726 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.726 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:29.726 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.726 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.726 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.726 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2110 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-01-16T00:13:29.726 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.642112+0000", 2020-01-16T00:13:29.727 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 27.947691978000002, 2020-01-16T00:13:29.727 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018620198000000001, 2020-01-16T00:13:29.727 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:29.727 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:29.727 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:29.727 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:29.727 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:29.728 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2110 2020-01-16T00:13:29.728 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.728 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:29.728 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.728 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:29.728 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642112+0000", 2020-01-16T00:13:29.728 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:29.728 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.729 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.729 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:29.729 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642112+0000", 2020-01-16T00:13:29.729 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:29.729 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.730 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.730 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:29.730 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642111+0000", 2020-01-16T00:13:29.730 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4450000000000001e-06 2020-01-16T00:13:29.730 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.730 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.730 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:29.731 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642117+0000", 2020-01-16T00:13:29.731 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1567e-05 2020-01-16T00:13:29.731 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.731 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.731 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:29.731 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642129+0000", 2020-01-16T00:13:29.731 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.754e-06 2020-01-16T00:13:29.731 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.732 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.732 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:29.732 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642136+0000", 2020-01-16T00:13:29.732 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040239719999999998 2020-01-16T00:13:29.732 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.732 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.732 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:29.732 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646160+0000", 2020-01-16T00:13:29.733 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5600999999999998e-05 2020-01-16T00:13:29.733 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.733 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.733 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:29.733 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646215+0000", 2020-01-16T00:13:29.733 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013733006000000001 2020-01-16T00:13:29.733 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.734 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.734 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:29.734 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659948+0000", 2020-01-16T00:13:29.734 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000106277 2020-01-16T00:13:29.734 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.734 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.734 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:29.734 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660054+0000", 2020-01-16T00:13:29.735 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000524327 2020-01-16T00:13:29.735 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.735 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.735 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:29.735 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660579+0000", 2020-01-16T00:13:29.735 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9841999999999999e-05 2020-01-16T00:13:29.735 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.736 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.736 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:29.736 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660659+0000", 2020-01-16T00:13:29.736 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3572999999999995e-05 2020-01-16T00:13:29.736 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.736 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.736 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:29.736 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660732+0000", 2020-01-16T00:13:29.737 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018620198000000001 2020-01-16T00:13:29.737 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.737 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:29.737 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.737 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.737 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.737 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:70 2.1 e35/19 2:91f85463:::smithi12313934-24 low_tier:head v 35'66, mlcod=34'64)", 2020-01-16T00:13:29.738 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.986684+0000", 2020-01-16T00:13:29.738 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 23.603120124, 2020-01-16T00:13:29.738 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012287378999999999, 2020-01-16T00:13:29.738 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:29.738 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:29.738 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:29.738 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.738 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:29.739 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986684+0000", 2020-01-16T00:13:29.739 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:29.739 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.739 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.739 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:29.739 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986684+0000", 2020-01-16T00:13:29.739 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:29.740 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.740 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.740 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:29.740 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986683+0000", 2020-01-16T00:13:29.740 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012884179999999999 2020-01-16T00:13:29.740 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.740 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.740 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:29.741 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.987971+0000", 2020-01-16T00:13:29.741 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011558200000000001 2020-01-16T00:13:29.741 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.741 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.741 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:29.741 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988087+0000", 2020-01-16T00:13:29.741 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0472000000000001e-05 2020-01-16T00:13:29.742 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.742 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.742 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:29.742 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988137+0000", 2020-01-16T00:13:29.742 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000107484 2020-01-16T00:13:29.742 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.742 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.742 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:29.743 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988245+0000", 2020-01-16T00:13:29.743 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4409999999999998e-05 2020-01-16T00:13:29.743 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.743 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.743 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:29.743 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988279+0000", 2020-01-16T00:13:29.743 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010641609999999999 2020-01-16T00:13:29.743 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.744 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.744 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:29.744 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.998921+0000", 2020-01-16T00:13:29.744 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0278000000000002e-05 2020-01-16T00:13:29.744 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.744 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.744 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:29.745 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.998971+0000", 2020-01-16T00:13:29.745 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012287378999999999 2020-01-16T00:13:29.745 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.745 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:29.745 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.745 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.745 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.745 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:71 2.1 e35/19 2:91f85463:::smithi12313934-24 low_tier:head v 35'67, mlcod=34'64)", 2020-01-16T00:13:29.746 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.991132+0000", 2020-01-16T00:13:29.746 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 23.598672174000001, 2020-01-16T00:13:29.746 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0082681190000000009, 2020-01-16T00:13:29.746 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:29.746 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:29.746 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:29.746 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.747 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:29.747 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.991132+0000", 2020-01-16T00:13:29.747 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:29.747 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.747 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.747 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:29.747 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.991132+0000", 2020-01-16T00:13:29.747 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:29.748 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.748 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.748 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:29.748 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.991130+0000", 2020-01-16T00:13:29.748 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3583000000000003e-05 2020-01-16T00:13:29.748 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.748 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.749 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:29.749 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.991203+0000", 2020-01-16T00:13:29.749 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.1001000000000001e-05 2020-01-16T00:13:29.749 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.749 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.749 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:29.749 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.991224+0000", 2020-01-16T00:13:29.749 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9329999999999996e-06 2020-01-16T00:13:29.750 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.750 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.750 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:29.750 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.991234+0000", 2020-01-16T00:13:29.750 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0035280070000000001 2020-01-16T00:13:29.750 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.750 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.751 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:29.751 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.994762+0000", 2020-01-16T00:13:29.751 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.622e-05 2020-01-16T00:13:29.751 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.751 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.751 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:29.751 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.994788+0000", 2020-01-16T00:13:29.751 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0045652219999999999 2020-01-16T00:13:29.752 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.752 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.752 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:29.752 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999354+0000", 2020-01-16T00:13:29.752 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6396000000000001e-05 2020-01-16T00:13:29.752 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.752 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.752 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:29.753 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999400+0000", 2020-01-16T00:13:29.753 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0082681190000000009 2020-01-16T00:13:29.753 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.753 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:29.753 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.753 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.753 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.754 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:77 2.1 e36/19 2:91f85463:::smithi12313934-24 low_tier:head v 36'69, mlcod=35'67)", 2020-01-16T00:13:29.754 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.556730+0000", 2020-01-16T00:13:29.754 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 22.033073770000001, 2020-01-16T00:13:29.754 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093007760000000002, 2020-01-16T00:13:29.754 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:29.754 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:29.754 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:29.754 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.755 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:29.755 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556730+0000", 2020-01-16T00:13:29.755 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:29.755 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.755 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.755 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:29.755 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556730+0000", 2020-01-16T00:13:29.756 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:29.756 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.756 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.756 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:29.756 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556728+0000", 2020-01-16T00:13:29.756 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00056509599999999996 2020-01-16T00:13:29.756 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.756 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.757 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:29.757 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.557293+0000", 2020-01-16T00:13:29.757 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0225e-05 2020-01-16T00:13:29.757 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.757 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.757 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:29.757 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.557314+0000", 2020-01-16T00:13:29.758 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8989999999999996e-06 2020-01-16T00:13:29.758 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.758 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.758 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:29.758 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.557323+0000", 2020-01-16T00:13:29.758 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7923000000000002e-05 2020-01-16T00:13:29.758 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.758 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.759 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:29.759 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.557371+0000", 2020-01-16T00:13:29.759 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9963999999999999e-05 2020-01-16T00:13:29.759 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.759 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.759 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:29.759 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.557411+0000", 2020-01-16T00:13:29.759 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0085873000000000008 2020-01-16T00:13:29.760 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.760 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.760 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:29.760 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.565999+0000", 2020-01-16T00:13:29.760 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2326000000000002e-05 2020-01-16T00:13:29.760 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.760 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.760 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:29.761 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566031+0000", 2020-01-16T00:13:29.761 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093007760000000002 2020-01-16T00:13:29.761 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.761 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:29.761 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.761 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.761 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.762 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:109 2.0 e42/19 2:11fa79bc:::smithi12313934-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 42'67, mlcod=41'65)", 2020-01-16T00:13:29.762 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:21.841140+0000", 2020-01-16T00:13:29.762 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 7.748664056, 2020-01-16T00:13:29.762 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010439159999999999, 2020-01-16T00:13:29.762 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:29.762 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:29.762 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:29.763 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.763 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:29.763 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841140+0000", 2020-01-16T00:13:29.763 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:29.763 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.763 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.763 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:29.763 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841140+0000", 2020-01-16T00:13:29.764 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:29.764 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.764 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.764 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:29.764 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841137+0000", 2020-01-16T00:13:29.764 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00061049800000000001 2020-01-16T00:13:29.764 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.764 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.765 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:29.765 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841748+0000", 2020-01-16T00:13:29.765 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8703e-05 2020-01-16T00:13:29.765 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.765 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.765 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:29.765 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841767+0000", 2020-01-16T00:13:29.766 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5410000000000003e-06 2020-01-16T00:13:29.766 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.766 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.766 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:29.766 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841774+0000", 2020-01-16T00:13:29.766 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000100144 2020-01-16T00:13:29.766 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.767 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.767 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:29.767 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841874+0000", 2020-01-16T00:13:29.767 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9402999999999999e-05 2020-01-16T00:13:29.767 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.767 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.767 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:29.767 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841944+0000", 2020-01-16T00:13:29.768 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095901649999999995 2020-01-16T00:13:29.768 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.768 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.768 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:29.768 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.851534+0000", 2020-01-16T00:13:29.768 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5296000000000001e-05 2020-01-16T00:13:29.768 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.768 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.769 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:29.769 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.851579+0000", 2020-01-16T00:13:29.769 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010439159999999999 2020-01-16T00:13:29.769 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.769 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:29.769 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.769 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.770 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.770 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:110 2.0 e42/19 2:11fa79bc:::smithi12313934-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 42'68, mlcod=41'65)", 2020-01-16T00:13:29.770 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:21.842301+0000", 2020-01-16T00:13:29.770 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 7.7475026250000001, 2020-01-16T00:13:29.770 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093040050000000006, 2020-01-16T00:13:29.770 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:29.770 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:29.771 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:29.771 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.771 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:29.771 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.842301+0000", 2020-01-16T00:13:29.771 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:29.771 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.771 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.771 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:29.772 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.842301+0000", 2020-01-16T00:13:29.772 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:29.772 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.772 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.772 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:29.772 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.842300+0000", 2020-01-16T00:13:29.772 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001154504 2020-01-16T00:13:29.772 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.773 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.773 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:29.773 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.843455+0000", 2020-01-16T00:13:29.773 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3263e-05 2020-01-16T00:13:29.773 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.773 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.773 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:29.774 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.843468+0000", 2020-01-16T00:13:29.774 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0786e-05 2020-01-16T00:13:29.774 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.774 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.774 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:29.774 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.843479+0000", 2020-01-16T00:13:29.774 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021168810000000001 2020-01-16T00:13:29.774 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.775 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.775 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:29.775 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.845596+0000", 2020-01-16T00:13:29.775 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0740000000000001e-05 2020-01-16T00:13:29.775 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.775 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.775 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:29.775 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.845616+0000", 2020-01-16T00:13:29.776 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0059668830000000001 2020-01-16T00:13:29.776 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.776 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.776 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:29.776 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.851583+0000", 2020-01-16T00:13:29.776 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2133999999999998e-05 2020-01-16T00:13:29.776 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.777 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.777 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:29.777 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.851605+0000", 2020-01-16T00:13:29.777 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093040050000000006 2020-01-16T00:13:29.777 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.777 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:29.777 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.777 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.778 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.778 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:111 2.0 e42/19 2:11fa79bc:::smithi12313934-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 42'69, mlcod=41'65)", 2020-01-16T00:13:29.778 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:21.843698+0000", 2020-01-16T00:13:29.778 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 7.7461059670000001, 2020-01-16T00:13:29.778 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0082463989999999997, 2020-01-16T00:13:29.778 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:29.778 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:29.779 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:29.779 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.779 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:29.779 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.843698+0000", 2020-01-16T00:13:29.779 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:29.779 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.779 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.779 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:29.780 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.843698+0000", 2020-01-16T00:13:29.780 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:29.780 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.780 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.780 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:29.780 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.843697+0000", 2020-01-16T00:13:29.780 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0011376730000000001 2020-01-16T00:13:29.780 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.781 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.781 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:29.781 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.844835+0000", 2020-01-16T00:13:29.781 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7294000000000001e-05 2020-01-16T00:13:29.781 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.781 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.781 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:29.782 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.844852+0000", 2020-01-16T00:13:29.782 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0639999999999994e-06 2020-01-16T00:13:29.782 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.782 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.782 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:29.782 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.844860+0000", 2020-01-16T00:13:29.782 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036989520000000001 2020-01-16T00:13:29.783 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.783 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.783 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:29.783 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.848559+0000", 2020-01-16T00:13:29.783 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.4658e-05 2020-01-16T00:13:29.783 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.783 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.783 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:29.784 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.848584+0000", 2020-01-16T00:13:29.784 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003318128 2020-01-16T00:13:29.784 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.784 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.784 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:29.784 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.851902+0000", 2020-01-16T00:13:29.784 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2719000000000002e-05 2020-01-16T00:13:29.784 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:29.785 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:29.785 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:29.785 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.851944+0000", 2020-01-16T00:13:29.785 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0082463989999999997 2020-01-16T00:13:29.785 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.785 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:29.785 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.786 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:29.786 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:29.786 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:29.788 INFO:tasks.rados.rados.0.smithi123.stdout:1279: writing smithi12313934-13 from 1689059 to 2430182 tid 2 2020-01-16T00:13:29.789 INFO:tasks.rados.rados.0.smithi123.stdout:1280: write oid 22 current snap is 0 2020-01-16T00:13:29.789 INFO:tasks.rados.rados.0.smithi123.stdout:1280: seq_num 393 ranges {605151=706860,2031188=573963} 2020-01-16T00:13:29.789 INFO:tasks.rados.rados.0.smithi123.stdout:1280: writing smithi12313934-22 from 605151 to 1312011 tid 1 2020-01-16T00:13:29.789 INFO:tasks.rados.rados.0.smithi123.stdout:1280: writing smithi12313934-22 from 2031188 to 2605151 tid 2 2020-01-16T00:13:29.789 INFO:tasks.rados.rados.0.smithi123.stdout:1281: read oid 49 snap -1 2020-01-16T00:13:29.790 INFO:tasks.rados.rados.0.smithi123.stdout:1281: expect (ObjNum 371 snap 0 seq_num 371) 2020-01-16T00:13:29.790 INFO:tasks.rados.rados.0.smithi123.stdout:1282: write oid 42 current snap is 0 2020-01-16T00:13:29.790 INFO:tasks.rados.rados.0.smithi123.stdout:1282: seq_num 394 ranges {621958=782109,2089670=532288} 2020-01-16T00:13:29.790 INFO:tasks.rados.rados.0.smithi123.stdout:1282: writing smithi12313934-42 from 621958 to 1404067 tid 1 2020-01-16T00:13:29.790 INFO:tasks.rados.rados.0.smithi123.stdout:1282: writing smithi12313934-42 from 2089670 to 2621958 tid 2 2020-01-16T00:13:29.790 INFO:tasks.rados.rados.0.smithi123.stdout:1283: write oid 12 current snap is 0 2020-01-16T00:13:29.791 INFO:tasks.rados.rados.0.smithi123.stdout:1283: seq_num 395 ranges {638765=457358,1748152=604803} 2020-01-16T00:13:29.791 INFO:tasks.rados.rados.0.smithi123.stdout:1283: writing smithi12313934-12 from 638765 to 1096123 tid 1 2020-01-16T00:13:29.791 INFO:tasks.rados.rados.0.smithi123.stdout:1283: writing smithi12313934-12 from 1748152 to 2352955 tid 2 2020-01-16T00:13:29.791 INFO:tasks.rados.rados.0.smithi123.stdout:1284: delete oid 43 current snap is 0 2020-01-16T00:13:29.791 INFO:tasks.rados.rados.0.smithi123.stdout:1271: finishing write tid 1 to smithi12313934-40 2020-01-16T00:13:29.791 INFO:tasks.rados.rados.0.smithi123.stdout:1271: finishing write tid 2 to smithi12313934-40 2020-01-16T00:13:29.791 INFO:tasks.rados.rados.0.smithi123.stdout:1273: finishing write tid 1 to smithi12313934-46 2020-01-16T00:13:29.791 INFO:tasks.rados.rados.0.smithi123.stdout:1278: expect (ObjNum 372 snap 0 seq_num 372) 2020-01-16T00:13:29.792 INFO:tasks.rados.rados.0.smithi123.stdout:1273: finishing write tid 2 to smithi12313934-46 2020-01-16T00:13:29.792 INFO:tasks.rados.rados.0.smithi123.stdout:1275: expect (ObjNum 384 snap 0 seq_num 384) 2020-01-16T00:13:29.792 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-01-16T00:13:29.792 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:29.782+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:29.866 INFO:tasks.rados.rados.0.smithi123.stdout:1279: finishing write tid 1 to smithi12313934-13 2020-01-16T00:13:29.867 INFO:tasks.rados.rados.0.smithi123.stdout:1277: expect (ObjNum 310 snap 0 seq_num 310) 2020-01-16T00:13:29.889 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:29.884+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:29.924 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2020-01-16T00:13:29.938 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:29.938 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:29.938 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:13:29.939 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:29.948 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-01-16T00:13:29.992 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:29.990+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:30.067 INFO:tasks.rados.rados.0.smithi123.stdout:1271: finishing write tid 4 to smithi12313934-40 2020-01-16T00:13:30.067 INFO:tasks.rados.rados.0.smithi123.stdout:1271: finishing write tid 5 to smithi12313934-40 2020-01-16T00:13:30.067 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 40 v 189 (ObjNum 390 snap 0 seq_num 390) dirty exists 2020-01-16T00:13:30.067 INFO:tasks.rados.rados.0.smithi123.stdout:1271: left oid 40 (ObjNum 390 snap 0 seq_num 390) 2020-01-16T00:13:30.067 INFO:tasks.rados.rados.0.smithi123.stdout:1273: finishing write tid 4 to smithi12313934-46 2020-01-16T00:13:30.067 INFO:tasks.rados.rados.0.smithi123.stdout:1273: finishing write tid 5 to smithi12313934-46 2020-01-16T00:13:30.068 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 46 v 240 (ObjNum 391 snap 0 seq_num 391) dirty exists 2020-01-16T00:13:30.068 INFO:tasks.rados.rados.0.smithi123.stdout:1273: left oid 46 (ObjNum 391 snap 0 seq_num 391) 2020-01-16T00:13:30.068 INFO:tasks.rados.rados.0.smithi123.stdout:1279: finishing write tid 2 to smithi12313934-13 2020-01-16T00:13:30.068 INFO:tasks.rados.rados.0.smithi123.stdout:1271: done (13 left) 2020-01-16T00:13:30.068 INFO:tasks.rados.rados.0.smithi123.stdout:1272: done (12 left) 2020-01-16T00:13:30.068 INFO:tasks.rados.rados.0.smithi123.stdout:1273: done (11 left) 2020-01-16T00:13:30.068 INFO:tasks.rados.rados.0.smithi123.stdout:1275: done (10 left) 2020-01-16T00:13:30.068 INFO:tasks.rados.rados.0.smithi123.stdout:1277: done (9 left) 2020-01-16T00:13:30.069 INFO:tasks.rados.rados.0.smithi123.stdout:1278: done (8 left) 2020-01-16T00:13:30.069 INFO:tasks.rados.rados.0.smithi123.stdout:1284: done (7 left) 2020-01-16T00:13:30.069 INFO:tasks.rados.rados.0.smithi123.stdout:1285: delete oid 21 current snap is 0 2020-01-16T00:13:30.069 INFO:tasks.rados.rados.0.smithi123.stdout:1274: expect (ObjNum 335 snap 0 seq_num 335) 2020-01-16T00:13:30.090 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:30.088+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:30.108 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:30.109 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:30.109 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:13:30.109 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:13:30.109 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:30.117 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-01-16T00:13:30.192 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:30.188+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:30.222 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:30.223 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:13:30.223 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:13:30.223 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:13:30.223 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.223 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:922 3.8 3:113947f2:::smithi12313934-12:head [write 443718~615603,stat] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:30.224 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.863202+0000", 2020-01-16T00:13:30.224 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 52.353724577999998, 2020-01-16T00:13:30.224 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017152697000000001, 2020-01-16T00:13:30.224 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:30.224 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:30.225 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:30.225 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:30.225 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:30.225 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 922 2020-01-16T00:13:30.225 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.226 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:30.226 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.226 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:30.227 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863202+0000", 2020-01-16T00:13:30.227 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:30.227 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.227 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.227 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:30.228 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863202+0000", 2020-01-16T00:13:30.228 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:30.228 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.228 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.229 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:30.229 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863200+0000", 2020-01-16T00:13:30.229 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00038569199999999998 2020-01-16T00:13:30.229 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.229 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.230 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:30.230 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863585+0000", 2020-01-16T00:13:30.230 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5685e-05 2020-01-16T00:13:30.230 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.230 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.230 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:30.230 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863601+0000", 2020-01-16T00:13:30.231 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.4734000000000001e-05 2020-01-16T00:13:30.231 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.231 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.231 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:30.231 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863626+0000", 2020-01-16T00:13:30.231 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1037e-05 2020-01-16T00:13:30.231 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.232 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.232 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.232 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863667+0000", 2020-01-16T00:13:30.232 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016041499999999999 2020-01-16T00:13:30.232 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.232 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.232 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:30.232 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863827+0000", 2020-01-16T00:13:30.233 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0094286430000000004 2020-01-16T00:13:30.233 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.233 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.233 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.233 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.873256+0000", 2020-01-16T00:13:30.233 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2867999999999994e-05 2020-01-16T00:13:30.233 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.233 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.234 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:30.234 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.873329+0000", 2020-01-16T00:13:30.234 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021174789999999998 2020-01-16T00:13:30.234 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.234 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.234 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.234 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.875446+0000", 2020-01-16T00:13:30.235 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011244200000000001 2020-01-16T00:13:30.235 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.235 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.235 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:30.235 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.875559+0000", 2020-01-16T00:13:30.235 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021772600000000001 2020-01-16T00:13:30.235 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.236 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.236 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1,7", 2020-01-16T00:13:30.236 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.875776+0000", 2020-01-16T00:13:30.236 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0028978860000000001 2020-01-16T00:13:30.236 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.236 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.236 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.237 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.878674+0000", 2020-01-16T00:13:30.237 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.324e-05 2020-01-16T00:13:30.237 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.237 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.237 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:30.238 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.878687+0000", 2020-01-16T00:13:30.238 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0014896180000000001 2020-01-16T00:13:30.238 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.238 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.238 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.239 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880177+0000", 2020-01-16T00:13:30.239 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6881000000000001e-05 2020-01-16T00:13:30.239 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.239 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.240 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:30.240 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880224+0000", 2020-01-16T00:13:30.240 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013096700000000001 2020-01-16T00:13:30.240 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.240 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.241 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:30.241 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880355+0000", 2020-01-16T00:13:30.241 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017152697000000001 2020-01-16T00:13:30.241 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.241 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:30.242 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.242 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.242 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.242 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:923 3.8 3:113947f2:::smithi12313934-12:head [write 1564138~470341] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:30.242 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.869292+0000", 2020-01-16T00:13:30.243 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 52.347634247999999, 2020-01-16T00:13:30.243 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017661620999999999, 2020-01-16T00:13:30.243 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:30.243 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:30.243 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:30.244 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:30.244 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:30.244 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 923 2020-01-16T00:13:30.244 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.244 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:30.245 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.245 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:30.245 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869292+0000", 2020-01-16T00:13:30.245 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:30.246 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.246 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.246 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:30.246 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869292+0000", 2020-01-16T00:13:30.246 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:30.247 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.247 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.247 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:30.247 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869291+0000", 2020-01-16T00:13:30.247 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00049842899999999999 2020-01-16T00:13:30.247 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.248 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.248 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:30.248 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869789+0000", 2020-01-16T00:13:30.248 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1014999999999999e-05 2020-01-16T00:13:30.248 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.248 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.248 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:30.249 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869830+0000", 2020-01-16T00:13:30.249 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7880000000000007e-06 2020-01-16T00:13:30.249 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.249 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.249 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:30.249 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869838+0000", 2020-01-16T00:13:30.249 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8346999999999997e-05 2020-01-16T00:13:30.250 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.250 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.250 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.250 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869876+0000", 2020-01-16T00:13:30.250 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9726999999999999e-05 2020-01-16T00:13:30.250 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.250 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.250 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:30.251 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869936+0000", 2020-01-16T00:13:30.251 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003434917 2020-01-16T00:13:30.251 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.251 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.251 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.251 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.873371+0000", 2020-01-16T00:13:30.251 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118808 2020-01-16T00:13:30.251 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.252 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.252 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:30.252 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.873490+0000", 2020-01-16T00:13:30.252 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036712910000000001 2020-01-16T00:13:30.252 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.252 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.252 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.253 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877161+0000", 2020-01-16T00:13:30.253 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9909999999999999e-05 2020-01-16T00:13:30.253 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.253 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.253 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:30.253 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877241+0000", 2020-01-16T00:13:30.253 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031619970000000002 2020-01-16T00:13:30.253 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.254 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.254 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.254 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880403+0000", 2020-01-16T00:13:30.254 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013071099999999999 2020-01-16T00:13:30.254 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.254 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.254 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:30.255 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880534+0000", 2020-01-16T00:13:30.255 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00032823500000000002 2020-01-16T00:13:30.255 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.255 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.255 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1,7", 2020-01-16T00:13:30.256 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880862+0000", 2020-01-16T00:13:30.256 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00513862 2020-01-16T00:13:30.256 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.256 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.256 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:30.256 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886001+0000", 2020-01-16T00:13:30.256 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8869000000000001e-05 2020-01-16T00:13:30.257 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.257 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.257 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.257 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886030+0000", 2020-01-16T00:13:30.257 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00081600200000000003 2020-01-16T00:13:30.257 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.257 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.258 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.258 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886846+0000", 2020-01-16T00:13:30.258 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4095999999999998e-05 2020-01-16T00:13:30.258 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.258 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.258 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:30.258 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886900+0000", 2020-01-16T00:13:30.259 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.448e-05 2020-01-16T00:13:30.259 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.259 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.259 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:30.259 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886954+0000", 2020-01-16T00:13:30.259 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017661620999999999 2020-01-16T00:13:30.259 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.260 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:30.260 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.260 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.260 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.260 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1662 3.7 3:e1be6d3b:::smithi12313934-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-01-16T00:13:30.260 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.735228+0000", 2020-01-16T00:13:30.260 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 37.481698643000001, 2020-01-16T00:13:30.260 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016641372000000001, 2020-01-16T00:13:30.261 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:30.261 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:30.261 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:30.261 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:30.261 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:30.261 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1662 2020-01-16T00:13:30.261 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.262 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:30.262 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.262 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:30.262 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735228+0000", 2020-01-16T00:13:30.262 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:30.262 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.262 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.263 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:30.263 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735228+0000", 2020-01-16T00:13:30.263 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:30.263 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.263 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.263 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:30.264 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735228+0000", 2020-01-16T00:13:30.264 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6260000000000001e-06 2020-01-16T00:13:30.264 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.264 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.265 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:30.265 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735231+0000", 2020-01-16T00:13:30.265 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5429999999999999e-06 2020-01-16T00:13:30.265 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.265 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.265 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:30.265 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735238+0000", 2020-01-16T00:13:30.266 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1760000000000001e-06 2020-01-16T00:13:30.266 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.266 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.266 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:30.266 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.735243+0000", 2020-01-16T00:13:30.266 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013610829999999999 2020-01-16T00:13:30.266 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.266 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.267 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.267 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736604+0000", 2020-01-16T00:13:30.267 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2194000000000002e-05 2020-01-16T00:13:30.267 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.267 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.267 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:30.267 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736656+0000", 2020-01-16T00:13:30.267 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013872425000000001 2020-01-16T00:13:30.268 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.268 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.268 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.268 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.750529+0000", 2020-01-16T00:13:30.268 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011952599999999999 2020-01-16T00:13:30.268 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.268 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.269 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:30.269 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.750648+0000", 2020-01-16T00:13:30.269 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001095171 2020-01-16T00:13:30.269 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.269 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.269 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.269 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.751743+0000", 2020-01-16T00:13:30.269 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3576000000000003e-05 2020-01-16T00:13:30.270 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.270 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.270 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:30.270 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.751807+0000", 2020-01-16T00:13:30.270 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2556000000000002e-05 2020-01-16T00:13:30.270 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.270 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.271 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:30.271 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.751869+0000", 2020-01-16T00:13:30.271 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016641372000000001 2020-01-16T00:13:30.271 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.271 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:30.271 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.271 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.271 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.272 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2226 3.7 3:e1be6d3b:::smithi12313934-24:head [setxattr _header (56) in=7b,truncate 2100908] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:13:30.272 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.980217+0000", 2020-01-16T00:13:30.272 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 24.236709583, 2020-01-16T00:13:30.272 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023113018999999999, 2020-01-16T00:13:30.272 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:30.272 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:30.272 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:30.273 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:30.273 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:30.273 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2226 2020-01-16T00:13:30.273 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.273 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:30.273 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.273 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:30.273 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980217+0000", 2020-01-16T00:13:30.274 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:30.274 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.274 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.274 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:30.274 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980217+0000", 2020-01-16T00:13:30.274 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:30.274 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.275 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.275 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:30.275 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980216+0000", 2020-01-16T00:13:30.275 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5049999999999998e-06 2020-01-16T00:13:30.275 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.275 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.275 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:30.275 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980222+0000", 2020-01-16T00:13:30.276 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5654999999999998e-05 2020-01-16T00:13:30.276 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.276 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.276 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:30.276 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980237+0000", 2020-01-16T00:13:30.276 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9740000000000004e-06 2020-01-16T00:13:30.276 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.277 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.277 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:30.277 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980241+0000", 2020-01-16T00:13:30.277 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002947776 2020-01-16T00:13:30.277 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.277 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.277 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.278 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.983189+0000", 2020-01-16T00:13:30.278 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0027099400000000001 2020-01-16T00:13:30.278 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.278 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.278 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:30.278 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985899+0000", 2020-01-16T00:13:30.278 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014275589999999999 2020-01-16T00:13:30.279 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.279 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.279 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.279 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.000175+0000", 2020-01-16T00:13:30.279 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011211 2020-01-16T00:13:30.279 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.279 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.279 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:30.279 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.000287+0000", 2020-01-16T00:13:30.280 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000229886 2020-01-16T00:13:30.280 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.280 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.280 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,7", 2020-01-16T00:13:30.280 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.000517+0000", 2020-01-16T00:13:30.280 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00095715399999999999 2020-01-16T00:13:30.280 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.280 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.281 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:30.281 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.001474+0000", 2020-01-16T00:13:30.281 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001341512 2020-01-16T00:13:30.281 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.281 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.281 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.281 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.002815+0000", 2020-01-16T00:13:30.281 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035789099999999999 2020-01-16T00:13:30.282 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.282 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.282 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.282 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003173+0000", 2020-01-16T00:13:30.282 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8571000000000005e-05 2020-01-16T00:13:30.282 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.283 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.283 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:30.283 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003242+0000", 2020-01-16T00:13:30.284 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8293000000000003e-05 2020-01-16T00:13:30.284 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.284 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.284 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:30.284 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003330+0000", 2020-01-16T00:13:30.284 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023113018999999999 2020-01-16T00:13:30.284 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.285 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:30.285 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.285 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.285 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.285 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2227 3.7 3:e1be6d3b:::smithi12313934-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e35)", 2020-01-16T00:13:30.285 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:13:30.285 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 24.236662330000001, 2020-01-16T00:13:30.286 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023303356000000001, 2020-01-16T00:13:30.286 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:30.286 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:30.286 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:30.286 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:30.286 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:30.286 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2227 2020-01-16T00:13:30.287 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.287 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:30.287 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.287 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:30.287 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:13:30.287 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:30.287 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.287 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.288 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:30.288 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:13:30.288 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:30.288 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.288 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.288 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:30.288 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:13:30.289 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3079999999999999e-06 2020-01-16T00:13:30.289 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.289 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.289 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:30.289 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980267+0000", 2020-01-16T00:13:30.289 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9290000000000001e-06 2020-01-16T00:13:30.289 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.289 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.290 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:30.290 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980274+0000", 2020-01-16T00:13:30.290 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7400000000000002e-06 2020-01-16T00:13:30.290 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.290 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.290 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:30.290 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980278+0000", 2020-01-16T00:13:30.291 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056624300000000004 2020-01-16T00:13:30.291 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.291 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.291 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.291 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985940+0000", 2020-01-16T00:13:30.291 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4261000000000002e-05 2020-01-16T00:13:30.291 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.291 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.292 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:30.292 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985985+0000", 2020-01-16T00:13:30.292 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015087676 2020-01-16T00:13:30.292 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.292 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.292 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.292 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.001072+0000", 2020-01-16T00:13:30.292 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011960600000000001 2020-01-16T00:13:30.293 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.293 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.293 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:30.293 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.001192+0000", 2020-01-16T00:13:30.293 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021718309999999999 2020-01-16T00:13:30.293 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.293 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.294 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.294 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003364+0000", 2020-01-16T00:13:30.294 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000102855 2020-01-16T00:13:30.294 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.294 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.294 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:30.294 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003467+0000", 2020-01-16T00:13:30.295 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010122499999999999 2020-01-16T00:13:30.295 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.295 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.295 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:30.295 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003568+0000", 2020-01-16T00:13:30.295 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023303356000000001 2020-01-16T00:13:30.295 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.296 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:30.296 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.296 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.296 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.296 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2233 3.2 3:4f8fe446:::smithi12313934-48:head [setxattr _header (56) in=7b,truncate 2117715] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:13:30.296 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.993103+0000", 2020-01-16T00:13:30.296 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 24.223824107999999, 2020-01-16T00:13:30.296 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017951763999999999, 2020-01-16T00:13:30.297 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:30.297 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:30.297 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:30.297 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:30.297 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:30.297 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2233 2020-01-16T00:13:30.298 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.298 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:30.298 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.298 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:30.298 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993103+0000", 2020-01-16T00:13:30.298 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:30.298 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.298 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.299 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:30.299 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993103+0000", 2020-01-16T00:13:30.299 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:30.299 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.299 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.299 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:30.299 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993102+0000", 2020-01-16T00:13:30.300 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9369999999999993e-06 2020-01-16T00:13:30.300 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.300 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.300 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:30.300 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993111+0000", 2020-01-16T00:13:30.300 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3340000000000002e-06 2020-01-16T00:13:30.300 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.301 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.301 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:30.301 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993119+0000", 2020-01-16T00:13:30.301 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1769999999999997e-06 2020-01-16T00:13:30.301 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.301 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.301 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:30.301 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993124+0000", 2020-01-16T00:13:30.302 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001601304 2020-01-16T00:13:30.302 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.302 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.302 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.302 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.994725+0000", 2020-01-16T00:13:30.302 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048777250000000003 2020-01-16T00:13:30.302 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.303 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.303 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:30.303 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999603+0000", 2020-01-16T00:13:30.303 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0099793880000000005 2020-01-16T00:13:30.303 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.303 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.304 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.304 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.009583+0000", 2020-01-16T00:13:30.304 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000108207 2020-01-16T00:13:30.304 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.304 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.304 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:30.304 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.009691+0000", 2020-01-16T00:13:30.305 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012862999999999999 2020-01-16T00:13:30.305 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.305 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.305 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:30.305 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.009819+0000", 2020-01-16T00:13:30.305 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00077947599999999998 2020-01-16T00:13:30.305 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.306 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.306 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:30.306 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010599+0000", 2020-01-16T00:13:30.306 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000221282 2020-01-16T00:13:30.306 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.306 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.306 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.307 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010820+0000", 2020-01-16T00:13:30.307 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7021999999999995e-05 2020-01-16T00:13:30.307 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.307 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.307 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.307 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010907+0000", 2020-01-16T00:13:30.307 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9815999999999997e-05 2020-01-16T00:13:30.307 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.307 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.308 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:30.308 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010977+0000", 2020-01-16T00:13:30.308 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7453999999999994e-05 2020-01-16T00:13:30.308 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.308 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.308 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:30.308 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.011054+0000", 2020-01-16T00:13:30.309 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017951763999999999 2020-01-16T00:13:30.309 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.309 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:30.309 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.309 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.309 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.309 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2234 3.2 3:4f8fe446:::smithi12313934-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e35)", 2020-01-16T00:13:30.309 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.993135+0000", 2020-01-16T00:13:30.310 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 24.223791654999999, 2020-01-16T00:13:30.310 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018067916999999999, 2020-01-16T00:13:30.310 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:30.310 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:30.310 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:30.310 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:30.311 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:30.311 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2234 2020-01-16T00:13:30.311 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.311 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:30.311 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.311 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:30.311 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993135+0000", 2020-01-16T00:13:30.311 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:30.312 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.312 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.312 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:30.312 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993135+0000", 2020-01-16T00:13:30.312 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:30.312 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.312 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.312 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:30.313 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993134+0000", 2020-01-16T00:13:30.313 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5630000000000002e-06 2020-01-16T00:13:30.313 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.313 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.313 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:30.313 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993139+0000", 2020-01-16T00:13:30.313 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1730000000000003e-06 2020-01-16T00:13:30.314 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.314 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.314 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:30.314 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993144+0000", 2020-01-16T00:13:30.314 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.937e-06 2020-01-16T00:13:30.314 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.314 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.315 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:30.315 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993147+0000", 2020-01-16T00:13:30.315 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0065164000000000003 2020-01-16T00:13:30.315 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.315 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.315 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.315 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999663+0000", 2020-01-16T00:13:30.316 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6215000000000002e-05 2020-01-16T00:13:30.316 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.316 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.316 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:30.316 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999729+0000", 2020-01-16T00:13:30.316 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010475887999999999 2020-01-16T00:13:30.316 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.316 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.317 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.317 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010205+0000", 2020-01-16T00:13:30.317 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119471 2020-01-16T00:13:30.317 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.317 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.317 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:30.317 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010325+0000", 2020-01-16T00:13:30.318 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00075498799999999995 2020-01-16T00:13:30.318 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.318 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.318 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.318 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.011080+0000", 2020-01-16T00:13:30.318 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7904e-05 2020-01-16T00:13:30.318 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.318 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.319 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:30.319 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.011138+0000", 2020-01-16T00:13:30.319 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5273999999999999e-05 2020-01-16T00:13:30.319 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.319 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.319 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:30.319 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.011203+0000", 2020-01-16T00:13:30.319 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018067916999999999 2020-01-16T00:13:30.320 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.320 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:30.320 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.320 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.320 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.320 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2324 3.7 3:e1be6d3b:::smithi12313934-24:head [setxattr _header (56) in=7b,truncate 2336206] snapc 0=[] ondisk+write+known_if_redirected e36)", 2020-01-16T00:13:30.320 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.550846+0000", 2020-01-16T00:13:30.320 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 22.666081069000001, 2020-01-16T00:13:30.321 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018364742, 2020-01-16T00:13:30.321 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:30.321 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:30.321 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:30.321 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:30.321 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:30.322 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2324 2020-01-16T00:13:30.322 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.322 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:30.322 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.322 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:30.322 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550846+0000", 2020-01-16T00:13:30.322 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:30.323 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.323 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.323 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:30.323 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550846+0000", 2020-01-16T00:13:30.323 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:30.323 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.323 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.323 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:30.324 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550845+0000", 2020-01-16T00:13:30.324 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1829999999999999e-06 2020-01-16T00:13:30.324 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.324 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.324 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:30.324 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550851+0000", 2020-01-16T00:13:30.324 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1925e-05 2020-01-16T00:13:30.325 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.325 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.325 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:30.325 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550863+0000", 2020-01-16T00:13:30.325 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1139999999999997e-06 2020-01-16T00:13:30.325 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.325 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.326 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:30.326 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550868+0000", 2020-01-16T00:13:30.326 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001388423 2020-01-16T00:13:30.326 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.326 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.326 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.326 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.552256+0000", 2020-01-16T00:13:30.327 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0034246540000000001 2020-01-16T00:13:30.327 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.327 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.327 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:30.327 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555681+0000", 2020-01-16T00:13:30.327 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011882762999999999 2020-01-16T00:13:30.327 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.328 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.328 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.328 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.567564+0000", 2020-01-16T00:13:30.328 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015010800000000001 2020-01-16T00:13:30.328 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.328 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.328 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:30.328 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.567714+0000", 2020-01-16T00:13:30.329 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011527999999999999 2020-01-16T00:13:30.329 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.329 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.329 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,7", 2020-01-16T00:13:30.329 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.567829+0000", 2020-01-16T00:13:30.329 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00039888499999999997 2020-01-16T00:13:30.329 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.330 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.330 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:30.330 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568228+0000", 2020-01-16T00:13:30.330 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00054254599999999998 2020-01-16T00:13:30.330 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.330 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.331 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.331 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568771+0000", 2020-01-16T00:13:30.331 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035206400000000002 2020-01-16T00:13:30.331 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.331 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.331 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.331 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569123+0000", 2020-01-16T00:13:30.331 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1262e-05 2020-01-16T00:13:30.332 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.332 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.332 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:30.332 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569154+0000", 2020-01-16T00:13:30.332 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6499000000000002e-05 2020-01-16T00:13:30.332 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.332 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.332 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:30.333 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569210+0000", 2020-01-16T00:13:30.333 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018364742 2020-01-16T00:13:30.333 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.333 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:30.333 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.333 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.333 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.333 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2325 3.7 3:e1be6d3b:::smithi12313934-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e36)", 2020-01-16T00:13:30.334 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.550877+0000", 2020-01-16T00:13:30.334 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 22.666050002999999, 2020-01-16T00:13:30.334 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018565154, 2020-01-16T00:13:30.334 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:30.334 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:30.335 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:30.335 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:30.335 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:30.335 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2325 2020-01-16T00:13:30.335 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.335 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:30.335 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.335 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:30.336 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550877+0000", 2020-01-16T00:13:30.336 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:30.336 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.336 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.336 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:30.336 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550877+0000", 2020-01-16T00:13:30.336 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:30.336 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.337 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.337 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:30.337 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550876+0000", 2020-01-16T00:13:30.337 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1080000000000003e-06 2020-01-16T00:13:30.337 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.337 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.337 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:30.337 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550880+0000", 2020-01-16T00:13:30.338 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3269999999999997e-06 2020-01-16T00:13:30.338 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.338 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.338 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:30.338 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550886+0000", 2020-01-16T00:13:30.338 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.386e-06 2020-01-16T00:13:30.338 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.338 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.339 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:30.339 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550891+0000", 2020-01-16T00:13:30.339 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048231410000000004 2020-01-16T00:13:30.339 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.339 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.339 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.339 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555714+0000", 2020-01-16T00:13:30.339 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0071999999999999e-05 2020-01-16T00:13:30.340 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.340 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.340 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:30.340 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555744+0000", 2020-01-16T00:13:30.340 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012329005000000001 2020-01-16T00:13:30.340 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.340 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.341 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.341 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568073+0000", 2020-01-16T00:13:30.341 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9087999999999997e-05 2020-01-16T00:13:30.341 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.341 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.341 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:30.341 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568172+0000", 2020-01-16T00:13:30.342 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010718220000000001 2020-01-16T00:13:30.342 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.342 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.342 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.342 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569244+0000", 2020-01-16T00:13:30.342 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8199e-05 2020-01-16T00:13:30.342 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.343 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.343 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:30.343 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569342+0000", 2020-01-16T00:13:30.343 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9708000000000001e-05 2020-01-16T00:13:30.343 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.343 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.343 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:30.343 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569442+0000", 2020-01-16T00:13:30.344 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018565154 2020-01-16T00:13:30.344 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.344 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:30.344 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.344 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.344 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.345 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2559 3.2 3:4ba19497:::smithi12313934-38:head [write 2950640~57846] snapc 0=[] ondisk+write+known_if_redirected e37)", 2020-01-16T00:13:30.345 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.591485+0000", 2020-01-16T00:13:30.345 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 17.625441823999999, 2020-01-16T00:13:30.345 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016668697999999999, 2020-01-16T00:13:30.345 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:30.345 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:30.345 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:30.345 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:30.346 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:30.346 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2559 2020-01-16T00:13:30.346 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.346 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:30.346 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.346 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:30.346 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591485+0000", 2020-01-16T00:13:30.347 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:30.347 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.347 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.347 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:30.347 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591485+0000", 2020-01-16T00:13:30.347 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:30.347 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.348 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.348 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:30.348 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591483+0000", 2020-01-16T00:13:30.348 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4780999999999998e-05 2020-01-16T00:13:30.348 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.348 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.348 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:30.348 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591518+0000", 2020-01-16T00:13:30.349 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9706000000000001e-05 2020-01-16T00:13:30.349 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.349 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.349 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:30.349 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591537+0000", 2020-01-16T00:13:30.349 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1940000000000001e-06 2020-01-16T00:13:30.349 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.350 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.350 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:30.350 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591545+0000", 2020-01-16T00:13:30.350 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012233400000000001 2020-01-16T00:13:30.350 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.350 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.350 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.351 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591667+0000", 2020-01-16T00:13:30.351 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0050268719999999999 2020-01-16T00:13:30.351 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.351 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.351 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:30.351 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.596694+0000", 2020-01-16T00:13:30.351 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0090233669999999992 2020-01-16T00:13:30.351 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.351 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.352 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.352 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.605717+0000", 2020-01-16T00:13:30.352 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011849399999999999 2020-01-16T00:13:30.352 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.352 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.352 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:30.352 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.605836+0000", 2020-01-16T00:13:30.352 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022822200000000001 2020-01-16T00:13:30.353 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.353 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.353 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:30.353 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.606064+0000", 2020-01-16T00:13:30.353 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001265555 2020-01-16T00:13:30.353 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.353 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.354 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:30.354 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.607329+0000", 2020-01-16T00:13:30.354 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00052231100000000002 2020-01-16T00:13:30.354 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.354 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.354 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.355 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.607852+0000", 2020-01-16T00:13:30.355 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018900500000000001 2020-01-16T00:13:30.355 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.355 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.355 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.355 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.608041+0000", 2020-01-16T00:13:30.355 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7380000000000003e-05 2020-01-16T00:13:30.355 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.356 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.356 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:30.356 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.608098+0000", 2020-01-16T00:13:30.356 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5467999999999998e-05 2020-01-16T00:13:30.356 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.356 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.356 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:30.356 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.608154+0000", 2020-01-16T00:13:30.357 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016668697999999999 2020-01-16T00:13:30.357 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.357 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:30.357 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.357 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.357 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.357 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2560 3.2 3:4ba19497:::smithi12313934-38:head [setxattr _header (56) in=7b,truncate 3008486] snapc 0=[] ondisk+write+known_if_redirected e37)", 2020-01-16T00:13:30.357 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.591556+0000", 2020-01-16T00:13:30.358 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 17.625370360000002, 2020-01-16T00:13:30.358 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016869716999999999, 2020-01-16T00:13:30.358 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:30.358 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:30.358 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:30.358 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:30.358 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:30.359 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2560 2020-01-16T00:13:30.359 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.359 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:30.359 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.359 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:30.359 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591556+0000", 2020-01-16T00:13:30.359 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:30.360 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.360 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.360 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:30.360 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591556+0000", 2020-01-16T00:13:30.360 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:30.361 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.361 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.361 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:30.361 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591555+0000", 2020-01-16T00:13:30.361 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.93e-06 2020-01-16T00:13:30.361 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.362 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.362 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:30.362 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591561+0000", 2020-01-16T00:13:30.362 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2089999999999999e-06 2020-01-16T00:13:30.362 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.362 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.362 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:30.362 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591569+0000", 2020-01-16T00:13:30.363 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5530000000000003e-06 2020-01-16T00:13:30.363 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.363 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.363 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:30.363 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591573+0000", 2020-01-16T00:13:30.363 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0053636630000000003 2020-01-16T00:13:30.364 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.364 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.364 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.364 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.596937+0000", 2020-01-16T00:13:30.364 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8693000000000001e-05 2020-01-16T00:13:30.364 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.364 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.364 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:30.365 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.597006+0000", 2020-01-16T00:13:30.365 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095244160000000008 2020-01-16T00:13:30.365 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.365 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.365 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.366 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.606530+0000", 2020-01-16T00:13:30.366 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012407100000000001 2020-01-16T00:13:30.366 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.366 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.366 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:30.366 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.606654+0000", 2020-01-16T00:13:30.366 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000189285 2020-01-16T00:13:30.367 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.367 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.367 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:30.367 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.606843+0000", 2020-01-16T00:13:30.367 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013331490000000001 2020-01-16T00:13:30.367 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.367 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.367 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:30.368 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.608176+0000", 2020-01-16T00:13:30.368 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7118000000000003e-05 2020-01-16T00:13:30.368 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.368 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.368 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.368 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.608254+0000", 2020-01-16T00:13:30.368 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3964e-05 2020-01-16T00:13:30.368 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.369 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.369 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.369 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.608308+0000", 2020-01-16T00:13:30.369 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7191999999999997e-05 2020-01-16T00:13:30.369 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.369 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.369 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:30.369 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.608365+0000", 2020-01-16T00:13:30.370 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1375000000000001e-05 2020-01-16T00:13:30.370 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.370 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.370 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:30.370 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.608426+0000", 2020-01-16T00:13:30.370 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016869716999999999 2020-01-16T00:13:30.370 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.371 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:30.371 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.371 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.371 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.371 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2561 3.2 3:4ba19497:::smithi12313934-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e37)", 2020-01-16T00:13:30.371 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.591578+0000", 2020-01-16T00:13:30.371 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 17.625348609, 2020-01-16T00:13:30.371 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017082362, 2020-01-16T00:13:30.372 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:30.372 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:30.372 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:30.372 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:30.372 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:30.372 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2561 2020-01-16T00:13:30.372 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.373 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:30.373 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.373 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:30.373 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591578+0000", 2020-01-16T00:13:30.373 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:30.373 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.373 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.373 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:30.374 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591578+0000", 2020-01-16T00:13:30.374 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:30.374 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.374 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.374 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:30.374 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591578+0000", 2020-01-16T00:13:30.374 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.428e-06 2020-01-16T00:13:30.374 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.375 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.375 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:30.375 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591581+0000", 2020-01-16T00:13:30.375 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.818e-06 2020-01-16T00:13:30.375 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.375 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.376 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:30.376 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591587+0000", 2020-01-16T00:13:30.376 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9890000000000003e-06 2020-01-16T00:13:30.376 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.376 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.376 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:30.376 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591591+0000", 2020-01-16T00:13:30.376 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.005441909 2020-01-16T00:13:30.377 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.377 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.377 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.377 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.597033+0000", 2020-01-16T00:13:30.377 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1651999999999999e-05 2020-01-16T00:13:30.377 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.377 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.377 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:30.378 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.597084+0000", 2020-01-16T00:13:30.378 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010094819999999999 2020-01-16T00:13:30.378 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.378 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.378 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.378 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.607179+0000", 2020-01-16T00:13:30.378 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012867799999999999 2020-01-16T00:13:30.379 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.379 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.379 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:30.379 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.607308+0000", 2020-01-16T00:13:30.379 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001148975 2020-01-16T00:13:30.379 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.379 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.380 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.380 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.608457+0000", 2020-01-16T00:13:30.380 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000101877 2020-01-16T00:13:30.380 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.380 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.380 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:30.380 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.608559+0000", 2020-01-16T00:13:30.380 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000101845 2020-01-16T00:13:30.381 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.381 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.381 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:30.381 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.608660+0000", 2020-01-16T00:13:30.381 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017082362 2020-01-16T00:13:30.381 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.381 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:30.381 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.381 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.382 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.382 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2578 3.2 3:4f8fe446:::smithi12313934-48:head [write 2943979~131735] snapc 0=[] ondisk+write+known_if_redirected e37)", 2020-01-16T00:13:30.382 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.664740+0000", 2020-01-16T00:13:30.382 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 17.552187081, 2020-01-16T00:13:30.382 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016610650000000001, 2020-01-16T00:13:30.382 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:30.382 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:30.382 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:30.383 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:30.383 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:30.383 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2578 2020-01-16T00:13:30.383 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.383 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:30.383 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.383 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:30.383 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664740+0000", 2020-01-16T00:13:30.384 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:30.384 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.384 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.384 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:30.384 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664740+0000", 2020-01-16T00:13:30.384 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:30.384 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.384 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.385 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:30.385 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664738+0000", 2020-01-16T00:13:30.385 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2001999999999994e-05 2020-01-16T00:13:30.385 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.385 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.385 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:30.385 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664800+0000", 2020-01-16T00:13:30.386 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3665999999999999e-05 2020-01-16T00:13:30.386 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.386 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.386 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:30.386 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664813+0000", 2020-01-16T00:13:30.386 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3029999999999996e-06 2020-01-16T00:13:30.386 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.387 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.387 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:30.387 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664820+0000", 2020-01-16T00:13:30.387 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2528999999999996e-05 2020-01-16T00:13:30.387 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.387 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.387 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.387 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664892+0000", 2020-01-16T00:13:30.388 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0014021140000000001 2020-01-16T00:13:30.388 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.388 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.388 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:30.388 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.666294+0000", 2020-01-16T00:13:30.388 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.01307417 2020-01-16T00:13:30.388 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.388 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.389 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.389 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.679368+0000", 2020-01-16T00:13:30.389 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012269099999999999 2020-01-16T00:13:30.389 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.389 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.389 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:30.389 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.679491+0000", 2020-01-16T00:13:30.389 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021723399999999999 2020-01-16T00:13:30.390 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.390 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.390 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:30.390 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.679708+0000", 2020-01-16T00:13:30.390 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001478507 2020-01-16T00:13:30.390 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.390 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.390 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:30.391 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.681187+0000", 2020-01-16T00:13:30.391 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3408000000000002e-05 2020-01-16T00:13:30.391 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.391 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.391 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.391 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.681230+0000", 2020-01-16T00:13:30.391 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0519999999999999e-05 2020-01-16T00:13:30.391 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.392 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.392 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.392 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.681261+0000", 2020-01-16T00:13:30.392 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1486000000000003e-05 2020-01-16T00:13:30.392 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.392 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.392 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:30.392 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.681292+0000", 2020-01-16T00:13:30.393 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8013999999999997e-05 2020-01-16T00:13:30.393 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.393 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.393 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:30.393 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.681350+0000", 2020-01-16T00:13:30.393 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016610650000000001 2020-01-16T00:13:30.393 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.393 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:30.394 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.394 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.394 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.394 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2579 3.2 3:4f8fe446:::smithi12313934-48:head [setxattr _header (56) in=7b,truncate 3075714] snapc 0=[] ondisk+write+known_if_redirected e37)", 2020-01-16T00:13:30.394 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.664831+0000", 2020-01-16T00:13:30.394 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 17.552096193000001, 2020-01-16T00:13:30.394 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016678353999999999, 2020-01-16T00:13:30.394 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:30.395 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:30.395 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:30.395 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:30.395 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:30.395 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2579 2020-01-16T00:13:30.395 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.396 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:30.396 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.396 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:30.396 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664831+0000", 2020-01-16T00:13:30.396 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:30.396 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.396 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.396 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:30.397 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664831+0000", 2020-01-16T00:13:30.397 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:30.397 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.397 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.397 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:30.397 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664830+0000", 2020-01-16T00:13:30.397 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7089999999999999e-06 2020-01-16T00:13:30.397 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.398 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.398 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:30.398 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664835+0000", 2020-01-16T00:13:30.398 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6526e-05 2020-01-16T00:13:30.398 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.398 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.398 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:30.399 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664862+0000", 2020-01-16T00:13:30.399 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1869999999999996e-06 2020-01-16T00:13:30.399 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.399 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.399 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:30.399 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664867+0000", 2020-01-16T00:13:30.399 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00148733 2020-01-16T00:13:30.399 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.400 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.400 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.400 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.666354+0000", 2020-01-16T00:13:30.400 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3683999999999995e-05 2020-01-16T00:13:30.400 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.400 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.400 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:30.400 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.666428+0000", 2020-01-16T00:13:30.401 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013904786000000001 2020-01-16T00:13:30.401 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.401 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.401 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.401 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.680333+0000", 2020-01-16T00:13:30.401 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012004 2020-01-16T00:13:30.401 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.401 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.402 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:30.402 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.680453+0000", 2020-01-16T00:13:30.402 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021047700000000001 2020-01-16T00:13:30.402 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.402 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.402 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:30.402 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.680663+0000", 2020-01-16T00:13:30.403 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00070448799999999997 2020-01-16T00:13:30.403 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.403 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.403 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:30.403 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.681368+0000", 2020-01-16T00:13:30.403 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8998999999999997e-05 2020-01-16T00:13:30.403 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.403 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.404 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.404 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.681407+0000", 2020-01-16T00:13:30.404 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9014999999999999e-05 2020-01-16T00:13:30.404 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.404 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.404 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.404 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.681436+0000", 2020-01-16T00:13:30.405 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9249999999999999e-05 2020-01-16T00:13:30.405 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.405 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.405 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:30.405 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.681465+0000", 2020-01-16T00:13:30.405 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3696999999999999e-05 2020-01-16T00:13:30.405 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.406 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.406 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:30.406 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.681509+0000", 2020-01-16T00:13:30.406 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016678353999999999 2020-01-16T00:13:30.406 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.406 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:30.407 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.407 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.407 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.407 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2580 3.2 3:4f8fe446:::smithi12313934-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e37)", 2020-01-16T00:13:30.407 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.664881+0000", 2020-01-16T00:13:30.407 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 17.552045247999999, 2020-01-16T00:13:30.407 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016925910999999998, 2020-01-16T00:13:30.407 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:30.408 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:30.408 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:30.408 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:30.408 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:30.408 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2580 2020-01-16T00:13:30.408 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.408 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:30.408 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.409 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:30.409 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664881+0000", 2020-01-16T00:13:30.409 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:30.409 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.409 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.409 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:30.409 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664881+0000", 2020-01-16T00:13:30.410 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:30.410 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.410 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.410 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:30.410 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664881+0000", 2020-01-16T00:13:30.410 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1640000000000003e-06 2020-01-16T00:13:30.410 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.411 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.411 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:30.411 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664885+0000", 2020-01-16T00:13:30.411 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6469999999999998e-06 2020-01-16T00:13:30.411 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.411 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.411 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:30.412 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664892+0000", 2020-01-16T00:13:30.412 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8829999999999997e-06 2020-01-16T00:13:30.412 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.412 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.412 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:30.412 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664897+0000", 2020-01-16T00:13:30.412 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0015624059999999999 2020-01-16T00:13:30.412 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.413 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.413 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.413 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.666460+0000", 2020-01-16T00:13:30.413 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9839000000000002e-05 2020-01-16T00:13:30.413 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.413 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.413 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:30.413 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.666510+0000", 2020-01-16T00:13:30.414 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014475200000000001 2020-01-16T00:13:30.414 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.414 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.414 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.414 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.680985+0000", 2020-01-16T00:13:30.414 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8396999999999997e-05 2020-01-16T00:13:30.414 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.415 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.415 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:30.415 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.681043+0000", 2020-01-16T00:13:30.415 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00051206799999999996 2020-01-16T00:13:30.415 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.415 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.415 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.416 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.681555+0000", 2020-01-16T00:13:30.416 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118881 2020-01-16T00:13:30.416 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.416 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.416 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:30.416 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.681674+0000", 2020-01-16T00:13:30.416 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000133242 2020-01-16T00:13:30.416 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.417 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.417 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:30.417 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.681807+0000", 2020-01-16T00:13:30.417 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016925910999999998 2020-01-16T00:13:30.417 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.418 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:30.418 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.418 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.418 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.418 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2800 3.2 3:42b7f20f:::smithi12313934-42:head [write 1395221~583781] snapc 0=[] ondisk+write+known_if_redirected e41)", 2020-01-16T00:13:30.418 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:17.256182+0000", 2020-01-16T00:13:30.418 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 12.960744462999999, 2020-01-16T00:13:30.419 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018509650999999998, 2020-01-16T00:13:30.419 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:30.419 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:30.419 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:30.419 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:30.419 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:30.419 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2800 2020-01-16T00:13:30.419 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.420 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:30.420 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.420 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:30.420 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256182+0000", 2020-01-16T00:13:30.420 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:30.420 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.420 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.421 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:30.421 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256182+0000", 2020-01-16T00:13:30.421 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:30.421 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.421 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.421 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:30.421 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256180+0000", 2020-01-16T00:13:30.422 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020514799999999999 2020-01-16T00:13:30.422 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.422 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.422 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:30.422 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256385+0000", 2020-01-16T00:13:30.422 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0288e-05 2020-01-16T00:13:30.422 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.423 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.423 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:30.423 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256406+0000", 2020-01-16T00:13:30.423 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7160000000000003e-06 2020-01-16T00:13:30.423 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.423 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.423 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:30.423 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256413+0000", 2020-01-16T00:13:30.424 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5293999999999998e-05 2020-01-16T00:13:30.424 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.424 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.424 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.424 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256449+0000", 2020-01-16T00:13:30.424 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001297737 2020-01-16T00:13:30.424 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.425 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.425 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:30.425 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.257746+0000", 2020-01-16T00:13:30.425 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0079866529999999998 2020-01-16T00:13:30.425 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.425 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.425 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.426 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.265733+0000", 2020-01-16T00:13:30.426 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4028000000000004e-05 2020-01-16T00:13:30.426 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.426 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.426 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:30.426 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.265797+0000", 2020-01-16T00:13:30.426 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011338899999999999 2020-01-16T00:13:30.426 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.427 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.427 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:30.427 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.265910+0000", 2020-01-16T00:13:30.427 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003372496 2020-01-16T00:13:30.427 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.427 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.427 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:30.428 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.269283+0000", 2020-01-16T00:13:30.428 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0053107959999999996 2020-01-16T00:13:30.428 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.428 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.428 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.428 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.274594+0000", 2020-01-16T00:13:30.428 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8000999999999998e-05 2020-01-16T00:13:30.428 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.429 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.429 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:30.429 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.274632+0000", 2020-01-16T00:13:30.429 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0265e-05 2020-01-16T00:13:30.429 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.429 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.430 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:30.430 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.274692+0000", 2020-01-16T00:13:30.430 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018509650999999998 2020-01-16T00:13:30.430 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.430 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:30.430 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.430 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.431 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.431 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2971 3.c 3:3025546b:::smithi12313934-15:head [write 1424565~609148] snapc 0=[] ondisk+write+known_if_redirected e42)", 2020-01-16T00:13:30.431 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:21.839711+0000", 2020-01-16T00:13:30.431 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 8.3772156199999994, 2020-01-16T00:13:30.431 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017534376000000001, 2020-01-16T00:13:30.431 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:30.432 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:30.432 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:30.432 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:30.432 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:30.432 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2971 2020-01-16T00:13:30.432 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.432 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:30.433 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.433 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:30.433 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839711+0000", 2020-01-16T00:13:30.433 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:30.433 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.433 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.433 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:30.433 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839711+0000", 2020-01-16T00:13:30.434 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:30.434 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.434 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.434 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:30.434 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839710+0000", 2020-01-16T00:13:30.434 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017658600000000001 2020-01-16T00:13:30.434 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.435 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.435 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:30.435 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839886+0000", 2020-01-16T00:13:30.435 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0492000000000001e-05 2020-01-16T00:13:30.436 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.436 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.436 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:30.436 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839897+0000", 2020-01-16T00:13:30.437 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.019e-06 2020-01-16T00:13:30.437 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.437 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.437 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:30.437 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839901+0000", 2020-01-16T00:13:30.437 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5572000000000001e-05 2020-01-16T00:13:30.438 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.438 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.438 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.438 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839976+0000", 2020-01-16T00:13:30.438 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034616099999999999 2020-01-16T00:13:30.438 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.438 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.439 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:30.439 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.840323+0000", 2020-01-16T00:13:30.439 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.01246249 2020-01-16T00:13:30.439 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.439 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.439 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.439 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.852785+0000", 2020-01-16T00:13:30.439 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014474799999999999 2020-01-16T00:13:30.440 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.440 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.440 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:30.440 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.852930+0000", 2020-01-16T00:13:30.440 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000213907 2020-01-16T00:13:30.440 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.440 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.441 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:30.441 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.853144+0000", 2020-01-16T00:13:30.441 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0028426979999999998 2020-01-16T00:13:30.441 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.441 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.442 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.442 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.855986+0000", 2020-01-16T00:13:30.442 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010694280000000001 2020-01-16T00:13:30.442 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.442 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.442 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:30.442 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.857056+0000", 2020-01-16T00:13:30.443 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8935000000000003e-05 2020-01-16T00:13:30.443 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.443 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.443 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:30.443 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.857125+0000", 2020-01-16T00:13:30.443 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000120721 2020-01-16T00:13:30.444 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.444 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.444 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:30.444 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.857245+0000", 2020-01-16T00:13:30.444 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017534376000000001 2020-01-16T00:13:30.444 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.444 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:30.445 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.445 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.445 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.445 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2972 3.c 3:3025546b:::smithi12313934-15:head [setxattr _header (56) in=7b,truncate 2033713] snapc 0=[] ondisk+write+known_if_redirected e42)", 2020-01-16T00:13:30.445 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:21.839923+0000", 2020-01-16T00:13:30.445 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 8.3770033559999995, 2020-01-16T00:13:30.445 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018367108, 2020-01-16T00:13:30.446 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:30.446 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:30.446 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:30.446 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:30.446 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:30.446 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2972 2020-01-16T00:13:30.446 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.446 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:30.447 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.447 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:30.447 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839923+0000", 2020-01-16T00:13:30.447 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:30.447 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.447 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.447 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:30.448 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839923+0000", 2020-01-16T00:13:30.448 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:30.448 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.448 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.448 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:30.448 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839923+0000", 2020-01-16T00:13:30.448 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0629999999999999e-06 2020-01-16T00:13:30.448 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.449 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.449 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:30.449 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839927+0000", 2020-01-16T00:13:30.449 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1980000000000003e-06 2020-01-16T00:13:30.449 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.449 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.449 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:30.450 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839931+0000", 2020-01-16T00:13:30.450 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5979999999999999e-06 2020-01-16T00:13:30.450 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.450 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.450 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:30.450 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839934+0000", 2020-01-16T00:13:30.450 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00042258500000000001 2020-01-16T00:13:30.450 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.451 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.451 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.451 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.840356+0000", 2020-01-16T00:13:30.451 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4919000000000002e-05 2020-01-16T00:13:30.451 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.451 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.452 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:30.452 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.840391+0000", 2020-01-16T00:13:30.452 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013783103 2020-01-16T00:13:30.452 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.452 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.452 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.453 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.854174+0000", 2020-01-16T00:13:30.453 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3455999999999995e-05 2020-01-16T00:13:30.453 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.453 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.453 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:30.453 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.854258+0000", 2020-01-16T00:13:30.453 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011060700000000001 2020-01-16T00:13:30.454 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.454 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.454 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:30.454 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.854368+0000", 2020-01-16T00:13:30.454 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020642500000000001 2020-01-16T00:13:30.454 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.454 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.455 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.455 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.856433+0000", 2020-01-16T00:13:30.455 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0016284820000000001 2020-01-16T00:13:30.455 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.455 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.455 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:30.455 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858061+0000", 2020-01-16T00:13:30.456 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0663999999999998e-05 2020-01-16T00:13:30.456 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.456 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.456 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:30.456 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858152+0000", 2020-01-16T00:13:30.456 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013882900000000001 2020-01-16T00:13:30.456 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.457 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.457 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:30.457 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858290+0000", 2020-01-16T00:13:30.457 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018367108 2020-01-16T00:13:30.457 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.457 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:30.457 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.458 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.458 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.458 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2973 3.c 3:3025546b:::smithi12313934-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e42)", 2020-01-16T00:13:30.458 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:21.839961+0000", 2020-01-16T00:13:30.458 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 8.3769658620000005, 2020-01-16T00:13:30.458 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018691670000000001, 2020-01-16T00:13:30.458 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:30.458 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:30.459 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:30.459 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:30.459 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:30.459 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2973 2020-01-16T00:13:30.459 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.459 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:30.459 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.460 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:30.460 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839961+0000", 2020-01-16T00:13:30.460 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:30.460 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.460 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.460 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:30.460 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839961+0000", 2020-01-16T00:13:30.460 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:30.461 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.461 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.461 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:30.461 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839960+0000", 2020-01-16T00:13:30.461 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4060000000000001e-06 2020-01-16T00:13:30.462 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.462 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.462 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:30.462 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839964+0000", 2020-01-16T00:13:30.462 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5412000000000001e-05 2020-01-16T00:13:30.462 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.462 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.462 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:30.463 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839979+0000", 2020-01-16T00:13:30.463 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.444e-06 2020-01-16T00:13:30.463 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.463 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.463 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:30.463 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839983+0000", 2020-01-16T00:13:30.463 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000430028 2020-01-16T00:13:30.464 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.464 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.464 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.464 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.840413+0000", 2020-01-16T00:13:30.464 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9270999999999997e-05 2020-01-16T00:13:30.464 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.464 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.464 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:30.465 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.840452+0000", 2020-01-16T00:13:30.465 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014853975 2020-01-16T00:13:30.465 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.465 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.465 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.465 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.855306+0000", 2020-01-16T00:13:30.465 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011098400000000001 2020-01-16T00:13:30.465 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.466 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.466 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:30.466 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.855417+0000", 2020-01-16T00:13:30.466 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029108060000000002 2020-01-16T00:13:30.466 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.466 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.466 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.467 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858328+0000", 2020-01-16T00:13:30.467 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000144697 2020-01-16T00:13:30.467 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.467 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.467 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:30.467 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858472+0000", 2020-01-16T00:13:30.467 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018011899999999999 2020-01-16T00:13:30.467 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.468 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.468 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:30.468 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858653+0000", 2020-01-16T00:13:30.468 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018691670000000001 2020-01-16T00:13:30.468 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.468 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:30.468 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.468 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.469 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.469 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3242 3.2 3:42b7f20f:::smithi12313934-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e45)", 2020-01-16T00:13:30.469 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.626793+0000", 2020-01-16T00:13:30.469 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.59013378100000002, 2020-01-16T00:13:30.469 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016576252999999999, 2020-01-16T00:13:30.469 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:30.469 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:30.469 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:30.470 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:30.470 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:30.470 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3242 2020-01-16T00:13:30.470 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.470 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:30.470 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.470 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:30.471 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.626793+0000", 2020-01-16T00:13:30.471 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:30.471 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.471 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.471 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:30.471 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.626793+0000", 2020-01-16T00:13:30.472 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:30.472 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.472 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.472 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:30.472 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.626792+0000", 2020-01-16T00:13:30.472 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5080000000000001e-06 2020-01-16T00:13:30.472 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.473 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.473 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:30.473 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.626796+0000", 2020-01-16T00:13:30.473 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3750000000000002e-06 2020-01-16T00:13:30.473 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.473 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.473 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:30.474 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.626801+0000", 2020-01-16T00:13:30.474 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1880000000000004e-06 2020-01-16T00:13:30.474 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.474 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.474 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:30.474 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.626805+0000", 2020-01-16T00:13:30.474 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00040277899999999997 2020-01-16T00:13:30.475 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.475 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.475 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.475 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627208+0000", 2020-01-16T00:13:30.475 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5717000000000001e-05 2020-01-16T00:13:30.475 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.475 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.476 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:30.476 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627234+0000", 2020-01-16T00:13:30.476 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013494844000000001 2020-01-16T00:13:30.476 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.476 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.476 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.476 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.640729+0000", 2020-01-16T00:13:30.477 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011387099999999999 2020-01-16T00:13:30.477 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.477 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.477 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:30.477 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.640843+0000", 2020-01-16T00:13:30.477 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002228968 2020-01-16T00:13:30.478 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.478 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.478 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.478 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.643072+0000", 2020-01-16T00:13:30.478 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000113645 2020-01-16T00:13:30.478 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.478 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.479 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:30.479 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.643185+0000", 2020-01-16T00:13:30.479 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018395999999999999 2020-01-16T00:13:30.479 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.479 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.479 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:30.479 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.643369+0000", 2020-01-16T00:13:30.479 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016576252999999999 2020-01-16T00:13:30.479 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.480 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:30.480 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.480 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.480 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:30.480 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:30.483 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-01-16T00:13:30.483 INFO:tasks.rados.rados.0.smithi123.stdout:1280: finishing write tid 1 to smithi12313934-22 2020-01-16T00:13:30.483 INFO:tasks.rados.rados.0.smithi123.stdout:1280: finishing write tid 2 to smithi12313934-22 2020-01-16T00:13:30.484 INFO:tasks.rados.rados.0.smithi123.stdout:1280: finishing write tid 4 to smithi12313934-22 2020-01-16T00:13:30.484 INFO:tasks.rados.rados.0.smithi123.stdout:1276: expect (ObjNum 381 snap 0 seq_num 381) 2020-01-16T00:13:30.484 INFO:tasks.rados.rados.0.smithi123.stdout:1280: finishing write tid 5 to smithi12313934-22 2020-01-16T00:13:30.484 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 22 v 33 (ObjNum 393 snap 0 seq_num 393) dirty exists 2020-01-16T00:13:30.484 INFO:tasks.rados.rados.0.smithi123.stdout:1280: left oid 22 (ObjNum 393 snap 0 seq_num 393) 2020-01-16T00:13:30.485 INFO:tasks.rados.rados.0.smithi123.stdout:1279: finishing write tid 4 to smithi12313934-13 2020-01-16T00:13:30.485 INFO:tasks.rados.rados.0.smithi123.stdout:1279: finishing write tid 5 to smithi12313934-13 2020-01-16T00:13:30.485 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 86 (ObjNum 392 snap 0 seq_num 392) dirty exists 2020-01-16T00:13:30.485 INFO:tasks.rados.rados.0.smithi123.stdout:1279: left oid 13 (ObjNum 392 snap 0 seq_num 392) 2020-01-16T00:13:30.485 INFO:tasks.rados.rados.0.smithi123.stdout:1281: expect (ObjNum 371 snap 0 seq_num 371) 2020-01-16T00:13:30.485 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:30.476+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:30.582 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:30.580+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:30.589 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:30.590 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:30.590 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:13:30.590 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:30.595 INFO:tasks.rados.rados.0.smithi123.stdout:1282: finishing write tid 1 to smithi12313934-42 2020-01-16T00:13:30.596 INFO:tasks.rados.rados.0.smithi123.stdout:1283: finishing write tid 1 to smithi12313934-12 2020-01-16T00:13:30.596 INFO:tasks.rados.rados.0.smithi123.stdout:1282: finishing write tid 2 to smithi12313934-42 2020-01-16T00:13:30.596 INFO:tasks.rados.rados.0.smithi123.stdout:1282: finishing write tid 4 to smithi12313934-42 2020-01-16T00:13:30.596 INFO:tasks.rados.rados.0.smithi123.stdout:1282: finishing write tid 5 to smithi12313934-42 2020-01-16T00:13:30.596 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 42 v 141 (ObjNum 394 snap 0 seq_num 394) dirty exists 2020-01-16T00:13:30.596 INFO:tasks.rados.rados.0.smithi123.stdout:1282: left oid 42 (ObjNum 394 snap 0 seq_num 394) 2020-01-16T00:13:30.597 INFO:tasks.rados.rados.0.smithi123.stdout:1283: finishing write tid 2 to smithi12313934-12 2020-01-16T00:13:30.597 INFO:tasks.rados.rados.0.smithi123.stdout:1283: finishing write tid 4 to smithi12313934-12 2020-01-16T00:13:30.597 INFO:tasks.rados.rados.0.smithi123.stdout:1274: done (7 left) 2020-01-16T00:13:30.597 INFO:tasks.rados.rados.0.smithi123.stdout:1276: done (6 left) 2020-01-16T00:13:30.597 INFO:tasks.rados.rados.0.smithi123.stdout:1279: done (5 left) 2020-01-16T00:13:30.597 INFO:tasks.rados.rados.0.smithi123.stdout:1280: done (4 left) 2020-01-16T00:13:30.597 INFO:tasks.rados.rados.0.smithi123.stdout:1281: done (3 left) 2020-01-16T00:13:30.597 INFO:tasks.rados.rados.0.smithi123.stdout:1282: done (2 left) 2020-01-16T00:13:30.598 INFO:tasks.rados.rados.0.smithi123.stdout:1285: done (1 left) 2020-01-16T00:13:30.598 INFO:tasks.rados.rados.0.smithi123.stdout:1286: read oid 29 snap -1 2020-01-16T00:13:30.598 INFO:tasks.rados.rados.0.smithi123.stdout:1286: expect (ObjNum 364 snap 0 seq_num 364) 2020-01-16T00:13:30.598 INFO:tasks.rados.rados.0.smithi123.stdout:1287: read oid 28 snap -1 2020-01-16T00:13:30.598 INFO:tasks.rados.rados.0.smithi123.stdout:1287: expect (ObjNum 357 snap 0 seq_num 357) 2020-01-16T00:13:30.598 INFO:tasks.rados.rados.0.smithi123.stdout:1288: read oid 24 snap -1 2020-01-16T00:13:30.598 INFO:tasks.rados.rados.0.smithi123.stdout:1288: expect (ObjNum 304 snap 0 seq_num 304) 2020-01-16T00:13:30.599 INFO:tasks.rados.rados.0.smithi123.stdout:1289: read oid 10 snap -1 2020-01-16T00:13:30.599 INFO:tasks.rados.rados.0.smithi123.stdout:1289: expect (ObjNum 381 snap 0 seq_num 381) 2020-01-16T00:13:30.599 INFO:tasks.rados.rados.0.smithi123.stdout:1290: read oid 35 snap -1 2020-01-16T00:13:30.599 INFO:tasks.rados.rados.0.smithi123.stdout:1290: expect (ObjNum 383 snap 0 seq_num 383) 2020-01-16T00:13:30.599 INFO:tasks.rados.rados.0.smithi123.stdout:1291: delete oid 17 current snap is 0 2020-01-16T00:13:30.599 INFO:tasks.rados.rados.0.smithi123.stdout:1283: finishing write tid 5 to smithi12313934-12 2020-01-16T00:13:30.600 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 12 v 114 (ObjNum 395 snap 0 seq_num 395) dirty exists 2020-01-16T00:13:30.600 INFO:tasks.rados.rados.0.smithi123.stdout:1283: left oid 12 (ObjNum 395 snap 0 seq_num 395) 2020-01-16T00:13:30.601 INFO:tasks.rados.rados.0.smithi123.stdout:1287: expect (ObjNum 357 snap 0 seq_num 357) 2020-01-16T00:13:30.601 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-01-16T00:13:30.683 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:30.678+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:30.693 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:30.694 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:30.694 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:13:30.694 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:13:30.695 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:30.701 INFO:tasks.rados.rados.0.smithi123.stdout:1283: done (6 left) 2020-01-16T00:13:30.701 INFO:tasks.rados.rados.0.smithi123.stdout:1287: done (5 left) 2020-01-16T00:13:30.701 INFO:tasks.rados.rados.0.smithi123.stdout:1291: done (4 left) 2020-01-16T00:13:30.701 INFO:tasks.rados.rados.0.smithi123.stdout:1292: write (excl) oid 7 current snap is 0 2020-01-16T00:13:30.701 INFO:tasks.rados.rados.0.smithi123.stdout:1292: seq_num 399 ranges {705993=474707,1582080=612141} 2020-01-16T00:13:30.702 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-01-16T00:13:30.707 INFO:tasks.rados.rados.0.smithi123.stdout:1292: writing smithi12313934-7 from 705993 to 1180700 tid 1 2020-01-16T00:13:30.713 INFO:tasks.rados.rados.0.smithi123.stdout:1292: writing smithi12313934-7 from 1582080 to 2194221 tid 2 2020-01-16T00:13:30.713 INFO:tasks.rados.rados.0.smithi123.stdout:1293: write oid 40 current snap is 0 2020-01-16T00:13:30.714 INFO:tasks.rados.rados.0.smithi123.stdout:1293: seq_num 400 ranges {722800=549956,2040562=682238} 2020-01-16T00:13:30.720 INFO:tasks.rados.rados.0.smithi123.stdout:1293: writing smithi12313934-40 from 722800 to 1272756 tid 1 2020-01-16T00:13:30.729 INFO:tasks.rados.rados.0.smithi123.stdout:1293: writing smithi12313934-40 from 2040562 to 2722800 tid 2 2020-01-16T00:13:30.730 INFO:tasks.rados.rados.0.smithi123.stdout:1294: read oid 47 snap -1 2020-01-16T00:13:30.730 INFO:tasks.rados.rados.0.smithi123.stdout:1294: expect (ObjNum 336 snap 0 seq_num 336) 2020-01-16T00:13:30.730 INFO:tasks.rados.rados.0.smithi123.stdout:1295: read oid 21 snap -1 2020-01-16T00:13:30.730 INFO:tasks.rados.rados.0.smithi123.stdout:1295: expect deleted 2020-01-16T00:13:30.730 INFO:tasks.rados.rados.0.smithi123.stdout:1296: read oid 31 snap -1 2020-01-16T00:13:30.731 INFO:tasks.rados.rados.0.smithi123.stdout:1296: expect (ObjNum 382 snap 0 seq_num 382) 2020-01-16T00:13:30.731 INFO:tasks.rados.rados.0.smithi123.stdout:1297: read oid 41 snap -1 2020-01-16T00:13:30.731 INFO:tasks.rados.rados.0.smithi123.stdout:1297: expect (ObjNum 310 snap 0 seq_num 310) 2020-01-16T00:13:30.731 INFO:tasks.rados.rados.0.smithi123.stdout:1298: read oid 5 snap -1 2020-01-16T00:13:30.731 INFO:tasks.rados.rados.0.smithi123.stdout:1298: expect (ObjNum 343 snap 0 seq_num 343) 2020-01-16T00:13:30.731 INFO:tasks.rados.rados.0.smithi123.stdout:1299: read oid 45 snap -1 2020-01-16T00:13:30.731 INFO:tasks.rados.rados.0.smithi123.stdout:1299: expect (ObjNum 335 snap 0 seq_num 335) 2020-01-16T00:13:30.732 INFO:tasks.rados.rados.0.smithi123.stdout:1300: read oid 17 snap -1 2020-01-16T00:13:30.732 INFO:tasks.rados.rados.0.smithi123.stdout:1300: expect deleted 2020-01-16T00:13:30.732 INFO:tasks.rados.rados.0.smithi123.stdout:1301: write oid 42 current snap is 0 2020-01-16T00:13:30.732 INFO:tasks.rados.rados.0.smithi123.stdout:1301: seq_num 401 ranges {739607=625205,1982691=499457} 2020-01-16T00:13:30.737 INFO:tasks.rados.rados.0.smithi123.stdout:1301: writing smithi12313934-42 from 739607 to 1364812 tid 1 2020-01-16T00:13:30.744 INFO:tasks.rados.rados.0.smithi123.stdout:1301: writing smithi12313934-42 from 1982691 to 2482148 tid 2 2020-01-16T00:13:30.744 INFO:tasks.rados.rados.0.smithi123.stdout:1302: read oid 13 snap -1 2020-01-16T00:13:30.744 INFO:tasks.rados.rados.0.smithi123.stdout:1302: expect (ObjNum 392 snap 0 seq_num 392) 2020-01-16T00:13:30.745 INFO:tasks.rados.rados.0.smithi123.stdout:1303: read oid 37 snap -1 2020-01-16T00:13:30.745 INFO:tasks.rados.rados.0.smithi123.stdout:1303: expect (ObjNum 378 snap 0 seq_num 378) 2020-01-16T00:13:30.745 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:30.745 INFO:tasks.rados.rados.0.smithi123.stdout:1286: expect (ObjNum 364 snap 0 seq_num 364) 2020-01-16T00:13:30.788 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:30.786+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:30.834 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:30.835 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:13:30.835 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:13:30.835 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:13:30.835 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.835 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.4.0:114 3.9 e24/20 3:9fc687b2:::smithi12313934-44:head v 24'74, mlcod=24'72)", 2020-01-16T00:13:30.835 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:13:30.835 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 52.504525893, 2020-01-16T00:13:30.835 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010399429, 2020-01-16T00:13:30.836 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:30.836 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:30.836 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:30.836 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.836 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:30.836 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:13:30.836 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:30.837 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.837 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.837 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:30.837 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:13:30.837 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:30.837 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.837 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.837 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:30.838 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325276+0000", 2020-01-16T00:13:30.838 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0046127069999999997 2020-01-16T00:13:30.838 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.838 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.838 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:30.838 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329889+0000", 2020-01-16T00:13:30.838 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3519999999999998e-05 2020-01-16T00:13:30.839 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.839 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.839 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:30.840 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329912+0000", 2020-01-16T00:13:30.840 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4070000000000004e-06 2020-01-16T00:13:30.840 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.840 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.840 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:30.840 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329920+0000", 2020-01-16T00:13:30.841 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8817000000000003e-05 2020-01-16T00:13:30.841 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.841 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.841 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.841 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329978+0000", 2020-01-16T00:13:30.841 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9721e-05 2020-01-16T00:13:30.841 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.842 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.842 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:30.842 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329998+0000", 2020-01-16T00:13:30.842 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056328139999999999 2020-01-16T00:13:30.842 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.842 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.842 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:30.842 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.335631+0000", 2020-01-16T00:13:30.843 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7170999999999999e-05 2020-01-16T00:13:30.843 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.843 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.843 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:30.843 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.335678+0000", 2020-01-16T00:13:30.843 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010399429 2020-01-16T00:13:30.843 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.843 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:30.844 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.844 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.844 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.844 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:24 2.1 2:8b431f80:::smithi12313934-40 low_tier:head [write 790528~688128] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:13:30.844 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.482303+0000", 2020-01-16T00:13:30.844 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 52.347501797, 2020-01-16T00:13:30.844 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010071853, 2020-01-16T00:13:30.845 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:30.845 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:30.845 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:30.845 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.845 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:30.845 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.482303+0000", 2020-01-16T00:13:30.845 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:30.845 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.846 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.846 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:30.846 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.482303+0000", 2020-01-16T00:13:30.846 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:30.846 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.846 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.846 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:30.847 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.482301+0000", 2020-01-16T00:13:30.847 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031173960000000001 2020-01-16T00:13:30.847 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.847 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.847 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:30.847 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485418+0000", 2020-01-16T00:13:30.847 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9300000000000001e-05 2020-01-16T00:13:30.847 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.848 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.848 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:30.848 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485448+0000", 2020-01-16T00:13:30.848 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2611e-05 2020-01-16T00:13:30.848 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.848 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.848 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:30.848 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485460+0000", 2020-01-16T00:13:30.849 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112487 2020-01-16T00:13:30.849 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.850 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.851 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.851 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485573+0000", 2020-01-16T00:13:30.851 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022169299999999999 2020-01-16T00:13:30.851 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.851 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.851 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:30.851 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485794+0000", 2020-01-16T00:13:30.851 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00036287500000000002 2020-01-16T00:13:30.852 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.852 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.852 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:30.852 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.486157+0000", 2020-01-16T00:13:30.852 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040260310000000002 2020-01-16T00:13:30.852 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.852 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.852 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:30.853 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.490183+0000", 2020-01-16T00:13:30.853 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00081083000000000001 2020-01-16T00:13:30.853 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.853 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.853 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.853 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.490994+0000", 2020-01-16T00:13:30.853 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001287746 2020-01-16T00:13:30.854 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.854 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.854 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.854 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492282+0000", 2020-01-16T00:13:30.854 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8810999999999998e-05 2020-01-16T00:13:30.854 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.854 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.855 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:30.855 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492321+0000", 2020-01-16T00:13:30.855 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4095000000000003e-05 2020-01-16T00:13:30.855 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.855 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.855 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:30.855 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492375+0000", 2020-01-16T00:13:30.855 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010071853 2020-01-16T00:13:30.856 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.856 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:30.856 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.856 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.856 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.856 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:55 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e30)", 2020-01-16T00:13:30.856 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.736036+0000", 2020-01-16T00:13:30.857 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 38.093768674000003, 2020-01-16T00:13:30.857 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010974536, 2020-01-16T00:13:30.857 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:30.857 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:30.857 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:30.857 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.857 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:30.858 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736036+0000", 2020-01-16T00:13:30.858 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:30.858 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.858 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.858 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:30.858 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736036+0000", 2020-01-16T00:13:30.858 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:30.858 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.859 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.859 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:30.859 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736034+0000", 2020-01-16T00:13:30.859 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043589699999999998 2020-01-16T00:13:30.859 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.860 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.860 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:30.860 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736469+0000", 2020-01-16T00:13:30.860 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9882e-05 2020-01-16T00:13:30.860 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.861 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.861 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:30.861 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736489+0000", 2020-01-16T00:13:30.861 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8339999999999999e-06 2020-01-16T00:13:30.861 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.861 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.861 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:30.862 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736497+0000", 2020-01-16T00:13:30.862 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5354999999999999e-05 2020-01-16T00:13:30.862 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.862 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.862 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.862 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736572+0000", 2020-01-16T00:13:30.862 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5228000000000004e-05 2020-01-16T00:13:30.863 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.863 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.863 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:30.863 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736668+0000", 2020-01-16T00:13:30.863 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017212999999999999 2020-01-16T00:13:30.863 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.863 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.864 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:30.864 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736840+0000", 2020-01-16T00:13:30.864 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0099057020000000006 2020-01-16T00:13:30.864 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.864 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.864 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.864 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746746+0000", 2020-01-16T00:13:30.864 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0586000000000005e-05 2020-01-16T00:13:30.865 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.865 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.865 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:30.865 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746816+0000", 2020-01-16T00:13:30.865 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2445000000000002e-05 2020-01-16T00:13:30.865 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.865 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.865 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.866 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746889+0000", 2020-01-16T00:13:30.866 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6839000000000002e-05 2020-01-16T00:13:30.866 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.866 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.866 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:30.866 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746955+0000", 2020-01-16T00:13:30.866 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5074000000000002e-05 2020-01-16T00:13:30.867 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.867 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.867 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:30.867 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.747011+0000", 2020-01-16T00:13:30.867 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010974536 2020-01-16T00:13:30.867 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.868 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:30.868 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.868 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.868 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.868 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:56 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e30)", 2020-01-16T00:13:30.868 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.736514+0000", 2020-01-16T00:13:30.869 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 38.093290838999998, 2020-01-16T00:13:30.869 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011660584999999999, 2020-01-16T00:13:30.869 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:30.869 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:30.869 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:30.869 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.870 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:30.870 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736514+0000", 2020-01-16T00:13:30.870 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:30.871 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.871 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.871 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:30.871 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736514+0000", 2020-01-16T00:13:30.871 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:30.871 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.871 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.872 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:30.872 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736513+0000", 2020-01-16T00:13:30.872 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9677000000000004e-05 2020-01-16T00:13:30.872 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.872 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.872 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:30.873 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736583+0000", 2020-01-16T00:13:30.873 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5269999999999998e-06 2020-01-16T00:13:30.873 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.873 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.873 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:30.873 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736591+0000", 2020-01-16T00:13:30.873 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5769999999999996e-06 2020-01-16T00:13:30.874 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.874 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.874 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:30.874 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736599+0000", 2020-01-16T00:13:30.874 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0053593310000000002 2020-01-16T00:13:30.874 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.874 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.874 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.875 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.741958+0000", 2020-01-16T00:13:30.875 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014164099999999999 2020-01-16T00:13:30.875 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.875 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.875 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:30.875 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.742100+0000", 2020-01-16T00:13:30.875 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020126000000000001 2020-01-16T00:13:30.876 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.876 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.876 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:30.876 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.742301+0000", 2020-01-16T00:13:30.876 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0045052859999999998 2020-01-16T00:13:30.876 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.876 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.877 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.877 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746806+0000", 2020-01-16T00:13:30.877 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00070802699999999996 2020-01-16T00:13:30.877 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.877 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.877 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:30.877 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.747514+0000", 2020-01-16T00:13:30.878 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00059247200000000003 2020-01-16T00:13:30.878 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.878 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.878 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.878 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748107+0000", 2020-01-16T00:13:30.878 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1870999999999999e-05 2020-01-16T00:13:30.878 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.879 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.879 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:30.879 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748149+0000", 2020-01-16T00:13:30.879 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5831000000000001e-05 2020-01-16T00:13:30.879 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.879 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.879 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:30.879 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748174+0000", 2020-01-16T00:13:30.880 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011660584999999999 2020-01-16T00:13:30.880 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.880 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:30.880 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.880 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.881 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.881 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:69 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 1922048~118272] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e34)", 2020-01-16T00:13:30.881 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:03.258431+0000", 2020-01-16T00:13:30.881 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 27.571373258000001, 2020-01-16T00:13:30.882 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0092806959999999997, 2020-01-16T00:13:30.882 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:30.882 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:30.882 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:30.882 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.882 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:30.882 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258431+0000", 2020-01-16T00:13:30.882 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:30.883 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.883 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.883 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:30.883 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258431+0000", 2020-01-16T00:13:30.883 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:30.883 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.883 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.884 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:30.884 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258431+0000", 2020-01-16T00:13:30.884 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3875000000000003e-05 2020-01-16T00:13:30.884 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.884 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.884 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:30.884 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258474+0000", 2020-01-16T00:13:30.885 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1040000000000007e-06 2020-01-16T00:13:30.885 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.885 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.885 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:30.885 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258483+0000", 2020-01-16T00:13:30.885 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6720000000000003e-06 2020-01-16T00:13:30.885 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.885 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.886 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:30.886 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258487+0000", 2020-01-16T00:13:30.886 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020507110000000002 2020-01-16T00:13:30.886 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.886 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.886 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.886 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.260538+0000", 2020-01-16T00:13:30.887 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017511100000000001 2020-01-16T00:13:30.887 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.887 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.887 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:30.887 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.260713+0000", 2020-01-16T00:13:30.887 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018914100000000001 2020-01-16T00:13:30.887 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.887 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.888 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:30.888 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.260902+0000", 2020-01-16T00:13:30.888 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.004582282 2020-01-16T00:13:30.888 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.888 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.888 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:30.888 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.265485+0000", 2020-01-16T00:13:30.888 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021355329999999998 2020-01-16T00:13:30.889 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.889 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.889 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.889 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.267620+0000", 2020-01-16T00:13:30.889 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1495999999999998e-05 2020-01-16T00:13:30.889 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.889 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.890 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.890 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.267652+0000", 2020-01-16T00:13:30.890 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5474e-05 2020-01-16T00:13:30.890 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.890 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.891 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:30.891 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.267687+0000", 2020-01-16T00:13:30.891 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5055e-05 2020-01-16T00:13:30.891 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.892 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.892 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:30.892 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.267712+0000", 2020-01-16T00:13:30.892 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0092806959999999997 2020-01-16T00:13:30.892 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.892 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:30.892 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.893 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.893 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.893 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:70 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e35)", 2020-01-16T00:13:30.893 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.985494+0000", 2020-01-16T00:13:30.893 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 24.844311043000001, 2020-01-16T00:13:30.893 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013746267, 2020-01-16T00:13:30.893 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:30.894 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:30.894 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:30.894 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.894 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:30.894 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985494+0000", 2020-01-16T00:13:30.894 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:30.894 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.895 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.895 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:30.895 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985494+0000", 2020-01-16T00:13:30.895 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:30.895 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.895 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.896 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:30.896 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985490+0000", 2020-01-16T00:13:30.896 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00077792099999999999 2020-01-16T00:13:30.896 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.896 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.896 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:30.896 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986268+0000", 2020-01-16T00:13:30.897 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9718000000000001e-05 2020-01-16T00:13:30.897 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.897 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.897 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:30.897 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986318+0000", 2020-01-16T00:13:30.897 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.4349999999999997e-06 2020-01-16T00:13:30.897 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.897 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.898 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:30.898 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986326+0000", 2020-01-16T00:13:30.898 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3298999999999999e-05 2020-01-16T00:13:30.898 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.898 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.898 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.898 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986380+0000", 2020-01-16T00:13:30.899 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4969999999999995e-05 2020-01-16T00:13:30.899 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.899 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.899 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:30.899 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986455+0000", 2020-01-16T00:13:30.899 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000110587 2020-01-16T00:13:30.899 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.899 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.900 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:30.900 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986565+0000", 2020-01-16T00:13:30.900 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063931600000000002 2020-01-16T00:13:30.900 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.900 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.900 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:30.900 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.992958+0000", 2020-01-16T00:13:30.901 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013008480000000001 2020-01-16T00:13:30.901 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.901 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.902 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.902 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.994259+0000", 2020-01-16T00:13:30.902 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048399979999999999 2020-01-16T00:13:30.902 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.902 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.902 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.902 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999099+0000", 2020-01-16T00:13:30.902 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1874999999999998e-05 2020-01-16T00:13:30.903 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.903 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.903 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:30.903 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999181+0000", 2020-01-16T00:13:30.903 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8865e-05 2020-01-16T00:13:30.903 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.903 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.904 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:30.904 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999240+0000", 2020-01-16T00:13:30.904 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013746267 2020-01-16T00:13:30.904 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.904 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:30.904 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.904 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.904 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.905 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:71 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e35)", 2020-01-16T00:13:30.905 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.986341+0000", 2020-01-16T00:13:30.905 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 24.843463681999999, 2020-01-16T00:13:30.905 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013252807, 2020-01-16T00:13:30.905 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:30.905 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:30.905 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:30.905 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.906 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:30.906 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986341+0000", 2020-01-16T00:13:30.906 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:30.906 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.906 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.906 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:30.906 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986341+0000", 2020-01-16T00:13:30.906 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:30.907 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.907 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.907 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:30.907 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986340+0000", 2020-01-16T00:13:30.907 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000103867 2020-01-16T00:13:30.907 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.907 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.907 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:30.908 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986444+0000", 2020-01-16T00:13:30.908 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0104e-05 2020-01-16T00:13:30.908 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.908 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.908 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:30.908 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986454+0000", 2020-01-16T00:13:30.908 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4220000000000002e-06 2020-01-16T00:13:30.909 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.909 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.909 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:30.909 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986458+0000", 2020-01-16T00:13:30.909 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033523070000000001 2020-01-16T00:13:30.909 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.909 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.909 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.910 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.989811+0000", 2020-01-16T00:13:30.910 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3296999999999999e-05 2020-01-16T00:13:30.910 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.910 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.910 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:30.910 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.989894+0000", 2020-01-16T00:13:30.910 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000117701 2020-01-16T00:13:30.910 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.911 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.911 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:30.911 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.990012+0000", 2020-01-16T00:13:30.911 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0032873490000000002 2020-01-16T00:13:30.912 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.912 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.912 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:30.912 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993299+0000", 2020-01-16T00:13:30.913 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002422619 2020-01-16T00:13:30.913 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.913 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.913 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.913 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.995722+0000", 2020-01-16T00:13:30.913 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0037839589999999999 2020-01-16T00:13:30.913 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.913 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.914 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.914 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999506+0000", 2020-01-16T00:13:30.914 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8559999999999997e-05 2020-01-16T00:13:30.914 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.914 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.914 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:30.914 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999544+0000", 2020-01-16T00:13:30.914 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9599999999999999e-05 2020-01-16T00:13:30.915 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.915 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.915 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:30.915 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999594+0000", 2020-01-16T00:13:30.915 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013252807 2020-01-16T00:13:30.915 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.915 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:30.915 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.916 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.916 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.916 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:77 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e36)", 2020-01-16T00:13:30.916 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.555854+0000", 2020-01-16T00:13:30.916 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 23.273950592999999, 2020-01-16T00:13:30.916 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010455002999999999, 2020-01-16T00:13:30.916 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:30.916 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:30.917 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:30.917 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.917 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:30.917 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555854+0000", 2020-01-16T00:13:30.917 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:30.917 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.917 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.918 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:30.918 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555854+0000", 2020-01-16T00:13:30.918 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:30.918 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.918 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.918 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:30.919 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555851+0000", 2020-01-16T00:13:30.919 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00048310200000000001 2020-01-16T00:13:30.919 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.919 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.919 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:30.919 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556334+0000", 2020-01-16T00:13:30.919 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2232e-05 2020-01-16T00:13:30.919 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.920 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.920 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:30.920 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556356+0000", 2020-01-16T00:13:30.920 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9750000000000007e-06 2020-01-16T00:13:30.920 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.920 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.920 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:30.920 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556364+0000", 2020-01-16T00:13:30.921 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8872999999999997e-05 2020-01-16T00:13:30.921 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.921 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.921 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.921 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556423+0000", 2020-01-16T00:13:30.922 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8687000000000001e-05 2020-01-16T00:13:30.922 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.922 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.923 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:30.923 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556502+0000", 2020-01-16T00:13:30.923 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013235300000000001 2020-01-16T00:13:30.923 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.923 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.923 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:30.924 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556634+0000", 2020-01-16T00:13:30.924 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0059770520000000001 2020-01-16T00:13:30.924 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.924 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.924 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:30.924 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.562611+0000", 2020-01-16T00:13:30.924 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029492820000000001 2020-01-16T00:13:30.924 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.925 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.925 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.925 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.565561+0000", 2020-01-16T00:13:30.925 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00060620300000000004 2020-01-16T00:13:30.925 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.925 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.925 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.926 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566167+0000", 2020-01-16T00:13:30.926 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3831000000000004e-05 2020-01-16T00:13:30.926 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.926 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.926 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:30.926 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566251+0000", 2020-01-16T00:13:30.926 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8328000000000001e-05 2020-01-16T00:13:30.927 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.927 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.927 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:30.927 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566309+0000", 2020-01-16T00:13:30.927 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010455002999999999 2020-01-16T00:13:30.927 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.927 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:30.927 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.928 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.928 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.928 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:78 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e36)", 2020-01-16T00:13:30.928 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.556378+0000", 2020-01-16T00:13:30.928 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 23.273426659999998, 2020-01-16T00:13:30.928 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010396311, 2020-01-16T00:13:30.928 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:30.929 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:30.929 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:30.929 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.929 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:30.929 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556378+0000", 2020-01-16T00:13:30.929 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:30.929 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.929 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.930 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:30.930 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556378+0000", 2020-01-16T00:13:30.930 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:30.930 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.930 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.930 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:30.930 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556377+0000", 2020-01-16T00:13:30.931 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4307000000000001e-05 2020-01-16T00:13:30.931 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.931 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.931 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:30.931 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556422+0000", 2020-01-16T00:13:30.931 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9620000000000005e-06 2020-01-16T00:13:30.931 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.931 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.932 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:30.932 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556432+0000", 2020-01-16T00:13:30.932 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5979999999999999e-06 2020-01-16T00:13:30.932 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.932 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.932 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:30.932 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556436+0000", 2020-01-16T00:13:30.933 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021059469999999999 2020-01-16T00:13:30.933 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.933 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.933 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.933 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.558542+0000", 2020-01-16T00:13:30.933 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000194458 2020-01-16T00:13:30.933 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.933 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.934 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:30.934 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.558737+0000", 2020-01-16T00:13:30.934 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00032867900000000002 2020-01-16T00:13:30.934 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.934 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.934 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:30.934 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.559065+0000", 2020-01-16T00:13:30.935 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052547499999999999 2020-01-16T00:13:30.935 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.935 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.935 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:30.935 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.564320+0000", 2020-01-16T00:13:30.935 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001565213 2020-01-16T00:13:30.935 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.935 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.936 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.936 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.565885+0000", 2020-01-16T00:13:30.936 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00076994300000000004 2020-01-16T00:13:30.936 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.936 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.936 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.936 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566655+0000", 2020-01-16T00:13:30.937 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7533000000000005e-05 2020-01-16T00:13:30.937 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.937 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.937 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:30.937 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566723+0000", 2020-01-16T00:13:30.937 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1616999999999997e-05 2020-01-16T00:13:30.937 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.938 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.938 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:30.939 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566774+0000", 2020-01-16T00:13:30.939 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010396311 2020-01-16T00:13:30.939 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.939 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:30.939 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.939 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.939 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.939 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:72 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 56832~1408000] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e37)", 2020-01-16T00:13:30.940 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.578753+0000", 2020-01-16T00:13:30.940 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 18.251051357000001, 2020-01-16T00:13:30.940 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011826918, 2020-01-16T00:13:30.940 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:30.940 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:30.940 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:30.940 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.941 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:30.941 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578753+0000", 2020-01-16T00:13:30.941 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:30.941 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.941 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.941 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:30.941 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578753+0000", 2020-01-16T00:13:30.942 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:30.942 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.942 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.942 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:30.942 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578751+0000", 2020-01-16T00:13:30.942 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00042457699999999998 2020-01-16T00:13:30.942 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.942 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.943 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:30.943 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579175+0000", 2020-01-16T00:13:30.943 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5404999999999999e-05 2020-01-16T00:13:30.943 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.943 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.943 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:30.943 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579191+0000", 2020-01-16T00:13:30.944 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6149999999999996e-06 2020-01-16T00:13:30.944 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.944 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.944 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:30.944 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579196+0000", 2020-01-16T00:13:30.944 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000104794 2020-01-16T00:13:30.944 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.944 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.945 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.945 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579301+0000", 2020-01-16T00:13:30.945 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020947199999999999 2020-01-16T00:13:30.945 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.945 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.945 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:30.945 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579511+0000", 2020-01-16T00:13:30.946 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034328400000000002 2020-01-16T00:13:30.946 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.946 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.946 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:30.946 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579854+0000", 2020-01-16T00:13:30.946 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.005078117 2020-01-16T00:13:30.946 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.947 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.947 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:30.947 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.584932+0000", 2020-01-16T00:13:30.947 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0014702160000000001 2020-01-16T00:13:30.947 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.948 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.948 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.948 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.586402+0000", 2020-01-16T00:13:30.948 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040623589999999998 2020-01-16T00:13:30.948 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.948 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.948 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.949 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590465+0000", 2020-01-16T00:13:30.949 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4478000000000002e-05 2020-01-16T00:13:30.949 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.949 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.949 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:30.949 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590529+0000", 2020-01-16T00:13:30.949 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1121000000000001e-05 2020-01-16T00:13:30.950 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.950 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.950 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:30.950 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590580+0000", 2020-01-16T00:13:30.950 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011826918 2020-01-16T00:13:30.950 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.950 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:30.951 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.951 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.951 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.951 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:73 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 1922048~118272] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e37)", 2020-01-16T00:13:30.951 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:13:30.951 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 18.25059856, 2020-01-16T00:13:30.951 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011750764, 2020-01-16T00:13:30.952 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:30.952 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:30.952 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:30.952 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.952 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:30.952 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:13:30.952 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:30.953 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.953 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:30.953 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:13:30.953 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:30.953 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.953 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:30.954 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:13:30.954 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2780000000000001e-05 2020-01-16T00:13:30.954 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.954 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.954 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:30.954 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579238+0000", 2020-01-16T00:13:30.954 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4341999999999999e-05 2020-01-16T00:13:30.955 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.955 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.955 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:30.955 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579253+0000", 2020-01-16T00:13:30.955 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1920000000000001e-06 2020-01-16T00:13:30.955 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.955 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.956 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:30.956 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579256+0000", 2020-01-16T00:13:30.956 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003224902 2020-01-16T00:13:30.956 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.956 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.956 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.956 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.582481+0000", 2020-01-16T00:13:30.956 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000180729 2020-01-16T00:13:30.957 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.957 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.957 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:30.957 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.582662+0000", 2020-01-16T00:13:30.957 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00025106100000000001 2020-01-16T00:13:30.957 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.957 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.958 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:30.958 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.582913+0000", 2020-01-16T00:13:30.958 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020814449999999999 2020-01-16T00:13:30.958 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.958 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.958 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:30.958 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.584994+0000", 2020-01-16T00:13:30.959 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001717243 2020-01-16T00:13:30.959 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.959 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.959 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.959 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.586711+0000", 2020-01-16T00:13:30.959 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041349869999999997 2020-01-16T00:13:30.959 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.959 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.960 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.960 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590846+0000", 2020-01-16T00:13:30.960 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3565000000000006e-05 2020-01-16T00:13:30.960 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.960 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.960 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:30.960 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590910+0000", 2020-01-16T00:13:30.961 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7002999999999997e-05 2020-01-16T00:13:30.961 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.961 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.961 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:30.961 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590957+0000", 2020-01-16T00:13:30.961 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011750764 2020-01-16T00:13:30.961 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.961 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:30.962 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.962 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.962 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.962 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:89 2.1 2:9b8617ce:::smithi12313934-38 low_tier:head [write 69632~773120] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e37)", 2020-01-16T00:13:30.962 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.593701+0000", 2020-01-16T00:13:30.962 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 18.236103933999999, 2020-01-16T00:13:30.962 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010026929, 2020-01-16T00:13:30.963 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:30.963 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:30.963 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:30.963 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.963 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:30.963 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593701+0000", 2020-01-16T00:13:30.963 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:30.964 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.964 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.964 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:30.964 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593701+0000", 2020-01-16T00:13:30.964 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:30.964 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.964 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.964 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:30.965 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593699+0000", 2020-01-16T00:13:30.965 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000237564 2020-01-16T00:13:30.965 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.965 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.965 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:30.965 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593936+0000", 2020-01-16T00:13:30.965 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2666e-05 2020-01-16T00:13:30.966 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.966 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.966 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:30.966 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593949+0000", 2020-01-16T00:13:30.966 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0089999999999998e-06 2020-01-16T00:13:30.966 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.966 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.966 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:30.967 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593954+0000", 2020-01-16T00:13:30.967 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5813000000000003e-05 2020-01-16T00:13:30.967 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.967 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.967 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.967 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.594000+0000", 2020-01-16T00:13:30.967 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010155299999999999 2020-01-16T00:13:30.968 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.968 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.968 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:30.968 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.594101+0000", 2020-01-16T00:13:30.968 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000203807 2020-01-16T00:13:30.968 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.968 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.968 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:30.969 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.594305+0000", 2020-01-16T00:13:30.969 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021655390000000002 2020-01-16T00:13:30.969 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.969 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.969 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.969 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.596471+0000", 2020-01-16T00:13:30.969 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0070645320000000001 2020-01-16T00:13:30.970 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.970 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.970 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.970 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603535+0000", 2020-01-16T00:13:30.970 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7577999999999998e-05 2020-01-16T00:13:30.971 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.971 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.971 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:30.971 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603573+0000", 2020-01-16T00:13:30.971 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9933999999999997e-05 2020-01-16T00:13:30.971 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.971 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.971 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:30.972 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603653+0000", 2020-01-16T00:13:30.972 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4993000000000005e-05 2020-01-16T00:13:30.972 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.972 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.972 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:30.972 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603728+0000", 2020-01-16T00:13:30.972 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010026929 2020-01-16T00:13:30.973 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.973 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:30.973 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.973 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.973 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.973 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:66 2.3 2:c3b82d87:::smithi12313934-44 low_tier:head [write 169472~818176] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e44)", 2020-01-16T00:13:30.973 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:25.278583+0000", 2020-01-16T00:13:30.973 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.551221537, 2020-01-16T00:13:30.974 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010294862, 2020-01-16T00:13:30.974 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:30.974 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:30.974 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:30.975 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.975 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:30.975 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.278583+0000", 2020-01-16T00:13:30.975 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:30.975 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.975 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.975 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:30.976 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.278583+0000", 2020-01-16T00:13:30.976 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:30.976 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.976 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.976 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:30.976 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.278581+0000", 2020-01-16T00:13:30.976 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00136721 2020-01-16T00:13:30.977 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.977 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.977 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:30.977 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.279948+0000", 2020-01-16T00:13:30.977 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6109999999999999e-05 2020-01-16T00:13:30.977 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.978 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.978 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:30.978 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.279974+0000", 2020-01-16T00:13:30.978 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.102e-05 2020-01-16T00:13:30.978 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.978 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.978 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:30.978 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.279985+0000", 2020-01-16T00:13:30.979 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6035e-05 2020-01-16T00:13:30.979 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.979 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.979 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.979 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.280061+0000", 2020-01-16T00:13:30.979 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000176243 2020-01-16T00:13:30.979 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.980 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.980 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:30.980 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.280238+0000", 2020-01-16T00:13:30.980 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019866099999999999 2020-01-16T00:13:30.980 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.980 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.980 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 7", 2020-01-16T00:13:30.980 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.280436+0000", 2020-01-16T00:13:30.981 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0049761830000000003 2020-01-16T00:13:30.981 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.981 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.981 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.981 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.285412+0000", 2020-01-16T00:13:30.981 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033347339999999998 2020-01-16T00:13:30.981 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.982 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.982 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:30.982 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.288747+0000", 2020-01-16T00:13:30.982 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9156000000000001e-05 2020-01-16T00:13:30.982 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.982 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.982 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:30.982 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.288806+0000", 2020-01-16T00:13:30.983 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1643000000000005e-05 2020-01-16T00:13:30.983 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.983 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.983 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:30.983 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.288878+0000", 2020-01-16T00:13:30.983 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010294862 2020-01-16T00:13:30.983 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.984 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:30.984 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.984 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.984 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.984 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:73 2.1 2:8b431f80:::smithi12313934-40 low_tier:head [write 790528~688128] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e45)", 2020-01-16T00:13:30.984 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.574742+0000", 2020-01-16T00:13:30.984 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 1.255062785, 2020-01-16T00:13:30.984 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0091537509999999999, 2020-01-16T00:13:30.985 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:30.985 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:30.985 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:30.985 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.985 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:30.985 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.574742+0000", 2020-01-16T00:13:30.985 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:30.985 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.986 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.986 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:30.986 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.574742+0000", 2020-01-16T00:13:30.986 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:30.986 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.986 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.986 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:30.987 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.574740+0000", 2020-01-16T00:13:30.987 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001119203 2020-01-16T00:13:30.987 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.987 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.987 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:30.987 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.575859+0000", 2020-01-16T00:13:30.987 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4990999999999999e-05 2020-01-16T00:13:30.987 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.988 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.988 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:30.988 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.575874+0000", 2020-01-16T00:13:30.988 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9240000000000001e-06 2020-01-16T00:13:30.988 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.988 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.988 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:30.988 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.575881+0000", 2020-01-16T00:13:30.989 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2113000000000005e-05 2020-01-16T00:13:30.989 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.989 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.989 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:30.989 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.575963+0000", 2020-01-16T00:13:30.989 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000104929 2020-01-16T00:13:30.989 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.990 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.990 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:30.990 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.576068+0000", 2020-01-16T00:13:30.990 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016916000000000001 2020-01-16T00:13:30.990 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.990 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.990 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:30.990 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.576237+0000", 2020-01-16T00:13:30.991 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0025967379999999999 2020-01-16T00:13:30.991 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.991 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.991 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:30.991 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.578834+0000", 2020-01-16T00:13:30.991 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001768862 2020-01-16T00:13:30.991 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.992 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.992 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.992 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.580603+0000", 2020-01-16T00:13:30.992 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003177221 2020-01-16T00:13:30.992 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.992 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.992 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:30.992 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.583780+0000", 2020-01-16T00:13:30.993 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1255999999999998e-05 2020-01-16T00:13:30.993 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.993 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.993 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:30.993 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.583841+0000", 2020-01-16T00:13:30.993 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4144999999999997e-05 2020-01-16T00:13:30.993 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.993 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.994 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:30.994 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.583896+0000", 2020-01-16T00:13:30.994 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0091537509999999999 2020-01-16T00:13:30.994 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.994 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:30.994 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:30.994 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.995 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.995 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.4.0:316 3.9 e45/20 3:93dfe6f1:::smithi12313934-40:head v 45'188, mlcod=45'186)", 2020-01-16T00:13:30.995 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.584377+0000", 2020-01-16T00:13:30.995 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 1.245427611, 2020-01-16T00:13:30.995 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095979069999999993, 2020-01-16T00:13:30.995 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:30.995 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:30.995 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:30.996 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.996 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:30.996 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584377+0000", 2020-01-16T00:13:30.996 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:30.996 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.996 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.996 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:30.997 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584377+0000", 2020-01-16T00:13:30.997 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:30.997 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.997 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.997 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:30.997 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584376+0000", 2020-01-16T00:13:30.997 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9059999999999997e-06 2020-01-16T00:13:30.998 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.998 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.998 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:30.998 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584385+0000", 2020-01-16T00:13:30.998 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3200000000000001e-05 2020-01-16T00:13:30.998 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.998 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.998 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:30.999 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584399+0000", 2020-01-16T00:13:30.999 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9660000000000001e-06 2020-01-16T00:13:30.999 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:30.999 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:30.999 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:30.999 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584405+0000", 2020-01-16T00:13:30.999 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0035787800000000002 2020-01-16T00:13:31.000 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.000 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.000 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:31.000 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.587983+0000", 2020-01-16T00:13:31.000 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0368000000000001e-05 2020-01-16T00:13:31.000 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.000 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.000 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:31.001 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588014+0000", 2020-01-16T00:13:31.001 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0059140290000000003 2020-01-16T00:13:31.001 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.001 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.001 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:31.001 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.593928+0000", 2020-01-16T00:13:31.001 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7135999999999997e-05 2020-01-16T00:13:31.002 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.002 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.002 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:31.002 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.593975+0000", 2020-01-16T00:13:31.002 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095979069999999993 2020-01-16T00:13:31.002 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:31.002 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:31.002 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:31.003 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.003 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.003 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:77 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 56832~1408000] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e45)", 2020-01-16T00:13:31.003 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.584566+0000", 2020-01-16T00:13:31.003 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 1.2452388860000001, 2020-01-16T00:13:31.003 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017037775000000002, 2020-01-16T00:13:31.003 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:31.003 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:31.004 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:31.004 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.004 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:31.004 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584566+0000", 2020-01-16T00:13:31.004 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:31.004 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.004 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.005 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:31.005 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584566+0000", 2020-01-16T00:13:31.005 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:31.005 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.005 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.005 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:31.005 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584564+0000", 2020-01-16T00:13:31.005 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043024300000000002 2020-01-16T00:13:31.006 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.006 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.006 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:31.006 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584994+0000", 2020-01-16T00:13:31.006 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9485999999999999e-05 2020-01-16T00:13:31.006 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.006 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.007 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:31.007 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585014+0000", 2020-01-16T00:13:31.007 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8909999999999995e-06 2020-01-16T00:13:31.007 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.007 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.007 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:31.007 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585022+0000", 2020-01-16T00:13:31.007 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039114459999999998 2020-01-16T00:13:31.008 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.008 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.008 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:31.008 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588933+0000", 2020-01-16T00:13:31.008 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011178700000000001 2020-01-16T00:13:31.008 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.009 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.009 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:31.009 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589045+0000", 2020-01-16T00:13:31.009 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016967300000000001 2020-01-16T00:13:31.009 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.009 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.010 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:31.010 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589214+0000", 2020-01-16T00:13:31.010 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012142356 2020-01-16T00:13:31.010 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.010 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.010 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:31.010 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601357+0000", 2020-01-16T00:13:31.010 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.1466999999999997e-05 2020-01-16T00:13:31.011 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.011 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.011 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:31.011 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601448+0000", 2020-01-16T00:13:31.011 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4817000000000001e-05 2020-01-16T00:13:31.011 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.011 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.012 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:31.012 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601493+0000", 2020-01-16T00:13:31.012 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9734000000000001e-05 2020-01-16T00:13:31.012 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.012 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.012 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:31.012 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601553+0000", 2020-01-16T00:13:31.013 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0677999999999998e-05 2020-01-16T00:13:31.013 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.013 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.013 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:31.013 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601604+0000", 2020-01-16T00:13:31.013 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017037775000000002 2020-01-16T00:13:31.013 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:31.014 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:31.014 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:31.014 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.014 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.014 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:78 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 1922048~118272] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e45)", 2020-01-16T00:13:31.014 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.585030+0000", 2020-01-16T00:13:31.014 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 1.2447741640000001, 2020-01-16T00:13:31.014 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016699964000000001, 2020-01-16T00:13:31.015 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:31.015 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:31.015 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:31.015 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.015 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:31.015 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585030+0000", 2020-01-16T00:13:31.015 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:31.015 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.016 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.016 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:31.016 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585030+0000", 2020-01-16T00:13:31.016 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:31.016 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.016 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.016 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:31.017 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585030+0000", 2020-01-16T00:13:31.017 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7085000000000001e-05 2020-01-16T00:13:31.017 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.017 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.017 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:31.017 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585077+0000", 2020-01-16T00:13:31.017 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5380000000000008e-06 2020-01-16T00:13:31.017 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.018 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.018 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:31.018 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585086+0000", 2020-01-16T00:13:31.018 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5410000000000002e-06 2020-01-16T00:13:31.018 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.018 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.018 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:31.019 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585091+0000", 2020-01-16T00:13:31.019 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0088486620000000002 2020-01-16T00:13:31.019 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.019 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.019 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:31.019 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.593940+0000", 2020-01-16T00:13:31.019 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112594 2020-01-16T00:13:31.019 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.020 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.020 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:31.020 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.594052+0000", 2020-01-16T00:13:31.020 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019893999999999999 2020-01-16T00:13:31.020 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.020 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.020 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:31.020 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.594251+0000", 2020-01-16T00:13:31.021 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0071632730000000004 2020-01-16T00:13:31.021 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.021 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.021 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:31.021 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601414+0000", 2020-01-16T00:13:31.021 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4672999999999996e-05 2020-01-16T00:13:31.021 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.022 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.022 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:31.022 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601479+0000", 2020-01-16T00:13:31.022 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000158453 2020-01-16T00:13:31.022 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.022 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.022 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:31.022 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601638+0000", 2020-01-16T00:13:31.023 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8605e-05 2020-01-16T00:13:31.023 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.023 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.023 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:31.023 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601686+0000", 2020-01-16T00:13:31.023 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4283999999999998e-05 2020-01-16T00:13:31.024 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.024 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.024 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:31.024 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601730+0000", 2020-01-16T00:13:31.024 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016699964000000001 2020-01-16T00:13:31.024 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:31.024 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:31.024 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:31.025 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.025 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.025 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:126 2.1 2:a4df40e2:::smithi12313934-42 low_tier:head [write 70656~727552] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e45)", 2020-01-16T00:13:31.025 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.627078+0000", 2020-01-16T00:13:31.025 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 1.202726508, 2020-01-16T00:13:31.025 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010098995, 2020-01-16T00:13:31.025 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:31.026 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:31.026 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:31.026 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.026 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:31.026 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627078+0000", 2020-01-16T00:13:31.026 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:31.026 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.026 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.027 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:31.027 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627078+0000", 2020-01-16T00:13:31.027 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:31.027 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.027 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.027 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:31.027 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627076+0000", 2020-01-16T00:13:31.028 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019321599999999999 2020-01-16T00:13:31.028 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.028 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.028 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:31.028 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627269+0000", 2020-01-16T00:13:31.028 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8471999999999999e-05 2020-01-16T00:13:31.028 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.028 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.029 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:31.029 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627288+0000", 2020-01-16T00:13:31.029 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3199999999999996e-06 2020-01-16T00:13:31.029 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.029 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.029 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:31.029 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627294+0000", 2020-01-16T00:13:31.030 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3341000000000002e-05 2020-01-16T00:13:31.030 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.030 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.030 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:31.030 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627357+0000", 2020-01-16T00:13:31.030 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010985 2020-01-16T00:13:31.030 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.030 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.031 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:31.031 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627467+0000", 2020-01-16T00:13:31.031 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022914499999999999 2020-01-16T00:13:31.031 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.031 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.031 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:31.031 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627696+0000", 2020-01-16T00:13:31.031 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0055120000000000004 2020-01-16T00:13:31.032 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.032 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.032 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:31.032 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.633208+0000", 2020-01-16T00:13:31.032 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0019968569999999999 2020-01-16T00:13:31.032 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.032 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.032 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:31.033 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.635205+0000", 2020-01-16T00:13:31.033 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018113160000000001 2020-01-16T00:13:31.033 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.033 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.033 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:31.033 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.637016+0000", 2020-01-16T00:13:31.033 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9515e-05 2020-01-16T00:13:31.034 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.034 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.034 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:31.034 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.637096+0000", 2020-01-16T00:13:31.034 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1213999999999999e-05 2020-01-16T00:13:31.034 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.034 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.034 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:31.035 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.637177+0000", 2020-01-16T00:13:31.035 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010098995 2020-01-16T00:13:31.035 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:31.035 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:31.035 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:31.035 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.035 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.036 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:127 2.1 2:a4df40e2:::smithi12313934-42 low_tier:head [write 1364992~1145344] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e45)", 2020-01-16T00:13:31.036 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.627308+0000", 2020-01-16T00:13:31.036 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 1.2024967120000001, 2020-01-16T00:13:31.036 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011068883999999999, 2020-01-16T00:13:31.036 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:31.036 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:31.036 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:31.036 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.037 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:31.037 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627308+0000", 2020-01-16T00:13:31.037 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:31.037 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.037 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.037 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:31.037 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627308+0000", 2020-01-16T00:13:31.037 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:31.038 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.038 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.038 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:31.038 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627307+0000", 2020-01-16T00:13:31.038 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000278597 2020-01-16T00:13:31.038 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.038 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.039 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:31.039 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627586+0000", 2020-01-16T00:13:31.039 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9169999999999997e-06 2020-01-16T00:13:31.039 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.039 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.039 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:31.040 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627591+0000", 2020-01-16T00:13:31.040 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0479999999999999e-06 2020-01-16T00:13:31.040 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.040 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.040 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:31.040 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627594+0000", 2020-01-16T00:13:31.040 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030987269999999999 2020-01-16T00:13:31.040 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.041 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.041 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:31.041 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.630693+0000", 2020-01-16T00:13:31.041 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000103688 2020-01-16T00:13:31.041 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.041 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.041 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:31.041 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.630796+0000", 2020-01-16T00:13:31.042 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000144606 2020-01-16T00:13:31.042 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.042 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.042 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:31.042 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.630941+0000", 2020-01-16T00:13:31.042 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039264269999999997 2020-01-16T00:13:31.042 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.043 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.043 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:31.043 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.634867+0000", 2020-01-16T00:13:31.043 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00038581200000000003 2020-01-16T00:13:31.043 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.043 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.043 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:31.043 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.635253+0000", 2020-01-16T00:13:31.044 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030568879999999998 2020-01-16T00:13:31.044 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.044 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.044 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:31.044 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.638310+0000", 2020-01-16T00:13:31.044 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8390999999999999e-05 2020-01-16T00:13:31.044 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.044 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.045 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:31.045 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.638348+0000", 2020-01-16T00:13:31.045 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8402999999999999e-05 2020-01-16T00:13:31.045 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.045 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.045 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:31.045 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.638377+0000", 2020-01-16T00:13:31.046 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011068883999999999 2020-01-16T00:13:31.046 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:31.046 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:31.046 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:31.046 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.046 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.046 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:129 2.1 2:a4df40e2:::smithi12313934-42 low_tier:head [write 1364992~1145344] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e45)", 2020-01-16T00:13:31.046 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:30.748851+0000", 2020-01-16T00:13:31.047 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.080953345999999995, 2020-01-16T00:13:31.047 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0091922689999999994, 2020-01-16T00:13:31.047 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:31.047 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:31.047 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:31.047 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.047 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:31.048 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748851+0000", 2020-01-16T00:13:31.048 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:31.048 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.048 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.048 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:31.049 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748851+0000", 2020-01-16T00:13:31.049 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:31.049 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.049 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.049 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:31.049 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748848+0000", 2020-01-16T00:13:31.049 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00037828800000000001 2020-01-16T00:13:31.050 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.050 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.050 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:31.050 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.749226+0000", 2020-01-16T00:13:31.050 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3696000000000001e-05 2020-01-16T00:13:31.050 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.051 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.051 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:31.051 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.749250+0000", 2020-01-16T00:13:31.051 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0899e-05 2020-01-16T00:13:31.051 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.051 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.051 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:31.051 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.749261+0000", 2020-01-16T00:13:31.052 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0016265450000000001 2020-01-16T00:13:31.052 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.052 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.052 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:31.052 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.750887+0000", 2020-01-16T00:13:31.052 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015506800000000001 2020-01-16T00:13:31.052 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.053 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.053 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:31.053 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.751043+0000", 2020-01-16T00:13:31.053 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000180206 2020-01-16T00:13:31.053 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.053 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.053 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:31.054 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.751223+0000", 2020-01-16T00:13:31.054 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0024072910000000002 2020-01-16T00:13:31.054 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.054 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.054 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:31.054 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.753630+0000", 2020-01-16T00:13:31.054 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012397969999999999 2020-01-16T00:13:31.054 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.055 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.055 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:31.055 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.754870+0000", 2020-01-16T00:13:31.055 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003021648 2020-01-16T00:13:31.055 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.055 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.055 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:31.056 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.757892+0000", 2020-01-16T00:13:31.056 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1149999999999996e-05 2020-01-16T00:13:31.056 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.056 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.056 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:31.056 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.757953+0000", 2020-01-16T00:13:31.056 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0906e-05 2020-01-16T00:13:31.056 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:31.057 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:31.057 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:31.057 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.758044+0000", 2020-01-16T00:13:31.057 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0091922689999999994 2020-01-16T00:13:31.057 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:31.057 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:31.057 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:31.058 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:31.058 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:31.058 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:31.060 INFO:tasks.rados.rados.0.smithi123.stdout:1286: done (15 left) 2020-01-16T00:13:31.060 INFO:tasks.rados.rados.0.smithi123.stdout:1304: write (excl) oid 3 current snap is 0 2020-01-16T00:13:31.060 INFO:tasks.rados.rados.0.smithi123.stdout:1304: seq_num 402 ranges {756414=700454,2041173=715241} 2020-01-16T00:13:31.060 INFO:tasks.rados.rados.0.smithi123.stdout:1304: writing smithi12313934-3 from 756414 to 1456868 tid 1 2020-01-16T00:13:31.061 INFO:tasks.rados.rados.0.smithi123.stdout:1304: writing smithi12313934-3 from 2041173 to 2756414 tid 2 2020-01-16T00:13:31.061 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:31.061 INFO:tasks.rados.rados.0.smithi123.stdout:1289: expect (ObjNum 381 snap 0 seq_num 381) 2020-01-16T00:13:31.061 INFO:teuthology.orchestra.run.smithi123.stderr:noscrub is set 2020-01-16T00:13:31.061 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:31.056+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:31.062 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-01-16T00:13:31.104 INFO:tasks.rados.rados.0.smithi123.stdout:1290: expect (ObjNum 383 snap 0 seq_num 383) 2020-01-16T00:13:31.160 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:31.158+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:31.177 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:31.177 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:13:31.178 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:13:31.178 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:31.190 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-01-16T00:13:31.242 INFO:tasks.rados.rados.0.smithi123.stdout:1288: expect (ObjNum 304 snap 0 seq_num 304) 2020-01-16T00:13:31.260 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:31.258+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:31.299 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:31.299 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:13:31.299 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:13:31.299 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:13:31.299 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:31.309 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-01-16T00:13:31.358 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-01-16T00:13:31.361 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:31.358+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:31.413 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:31.413 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:13:31.413 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:13:31.413 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:13:31.413 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.413 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:943 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:31.414 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:13:31.414 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 53.493630754999998, 2020-01-16T00:13:31.414 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018847004000000001, 2020-01-16T00:13:31.414 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:31.414 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:31.414 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:31.414 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:31.415 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:31.415 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 943 2020-01-16T00:13:31.415 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.415 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:31.415 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.415 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:31.415 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:13:31.415 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:31.416 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.416 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.416 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:31.416 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:13:31.416 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:31.416 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.416 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.417 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:31.417 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914475+0000", 2020-01-16T00:13:31.417 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.508e-05 2020-01-16T00:13:31.417 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.417 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.417 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:31.417 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914490+0000", 2020-01-16T00:13:31.417 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.857e-05 2020-01-16T00:13:31.418 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.418 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.418 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:31.418 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914509+0000", 2020-01-16T00:13:31.418 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.46e-06 2020-01-16T00:13:31.418 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.418 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.419 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:31.419 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914515+0000", 2020-01-16T00:13:31.419 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2612999999999999e-05 2020-01-16T00:13:31.419 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.419 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.419 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.419 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914568+0000", 2020-01-16T00:13:31.420 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028312299999999998 2020-01-16T00:13:31.420 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.420 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.420 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:31.420 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914851+0000", 2020-01-16T00:13:31.420 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012282533 2020-01-16T00:13:31.421 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.421 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.421 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.421 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927133+0000", 2020-01-16T00:13:31.421 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4839999999999995e-05 2020-01-16T00:13:31.421 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.421 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.422 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:31.422 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927208+0000", 2020-01-16T00:13:31.422 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0047425610000000002 2020-01-16T00:13:31.422 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.422 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.422 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.422 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.931951+0000", 2020-01-16T00:13:31.423 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3628999999999994e-05 2020-01-16T00:13:31.423 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.423 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.423 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:31.423 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.932045+0000", 2020-01-16T00:13:31.423 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012810180000000001 2020-01-16T00:13:31.423 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.423 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.424 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:31.424 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933326+0000", 2020-01-16T00:13:31.424 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018847004000000001 2020-01-16T00:13:31.424 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.424 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:31.424 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.425 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.425 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.425 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:944 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:31.425 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:13:31.425 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 53.493581198999998, 2020-01-16T00:13:31.425 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019086901999999999, 2020-01-16T00:13:31.425 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:31.426 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:31.426 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:31.426 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:31.426 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:31.426 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 944 2020-01-16T00:13:31.426 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.426 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:31.426 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.427 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:31.427 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:13:31.427 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:31.427 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.427 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.427 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:31.428 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:13:31.428 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:31.428 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.428 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.428 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:31.428 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914527+0000", 2020-01-16T00:13:31.428 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7509999999999999e-06 2020-01-16T00:13:31.428 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.429 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.429 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:31.429 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914532+0000", 2020-01-16T00:13:31.429 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5149999999999997e-06 2020-01-16T00:13:31.429 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.429 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.429 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:31.436 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914538+0000", 2020-01-16T00:13:31.436 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1890000000000007e-06 2020-01-16T00:13:31.437 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.437 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.437 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:31.437 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914546+0000", 2020-01-16T00:13:31.437 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000354951 2020-01-16T00:13:31.437 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.437 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.438 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.438 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914901+0000", 2020-01-16T00:13:31.438 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1594000000000002e-05 2020-01-16T00:13:31.438 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.438 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.438 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:31.438 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914942+0000", 2020-01-16T00:13:31.438 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012300070999999999 2020-01-16T00:13:31.439 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.439 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.439 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.439 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927242+0000", 2020-01-16T00:13:31.439 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4076999999999997e-05 2020-01-16T00:13:31.439 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.439 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.440 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:31.440 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927296+0000", 2020-01-16T00:13:31.440 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0060609180000000002 2020-01-16T00:13:31.440 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.440 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.440 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.440 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933357+0000", 2020-01-16T00:13:31.440 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4488999999999999e-05 2020-01-16T00:13:31.441 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.441 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.441 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:31.441 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933422+0000", 2020-01-16T00:13:31.441 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019318300000000001 2020-01-16T00:13:31.442 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.442 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.442 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:31.442 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933615+0000", 2020-01-16T00:13:31.442 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019086901999999999 2020-01-16T00:13:31.442 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.442 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:31.443 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.443 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.443 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.443 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:945 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:31.443 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:13:31.443 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 53.493553992999999, 2020-01-16T00:13:31.443 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019342986, 2020-01-16T00:13:31.443 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:31.444 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:31.444 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:31.444 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:31.444 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:31.444 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 945 2020-01-16T00:13:31.444 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.444 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:31.444 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.445 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:31.445 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:13:31.445 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:31.445 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.445 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.445 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:31.445 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:13:31.446 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:31.446 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.446 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.446 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:31.446 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:13:31.446 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4580000000000002e-06 2020-01-16T00:13:31.446 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.446 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.447 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:31.447 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914558+0000", 2020-01-16T00:13:31.447 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7829999999999998e-06 2020-01-16T00:13:31.447 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.447 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.447 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:31.447 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914563+0000", 2020-01-16T00:13:31.447 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0001e-05 2020-01-16T00:13:31.448 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.448 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.448 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:31.448 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914573+0000", 2020-01-16T00:13:31.448 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00040365600000000001 2020-01-16T00:13:31.448 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.449 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.449 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.449 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914976+0000", 2020-01-16T00:13:31.449 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1390000000000001e-05 2020-01-16T00:13:31.449 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.449 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.449 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:31.450 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.915028+0000", 2020-01-16T00:13:31.450 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012288843000000001 2020-01-16T00:13:31.450 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.450 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.450 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.450 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927317+0000", 2020-01-16T00:13:31.450 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7871000000000002e-05 2020-01-16T00:13:31.451 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.451 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.451 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:31.451 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927375+0000", 2020-01-16T00:13:31.451 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0062580589999999998 2020-01-16T00:13:31.451 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.451 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.452 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.452 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933633+0000", 2020-01-16T00:13:31.452 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7086999999999997e-05 2020-01-16T00:13:31.452 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.452 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.452 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:31.453 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933690+0000", 2020-01-16T00:13:31.453 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020855799999999999 2020-01-16T00:13:31.453 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.453 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.453 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:31.453 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933898+0000", 2020-01-16T00:13:31.453 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019342986 2020-01-16T00:13:31.454 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.454 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:31.454 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.454 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.454 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.454 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:981 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:31.454 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:13:31.454 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 53.092777101000003, 2020-01-16T00:13:31.455 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021222494000000001, 2020-01-16T00:13:31.455 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:31.455 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:31.455 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:31.455 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:31.455 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:31.455 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 981 2020-01-16T00:13:31.455 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.456 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:31.456 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.456 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:31.456 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:13:31.456 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:31.456 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.456 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.456 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:31.457 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:13:31.457 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999948 2020-01-16T00:13:31.457 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.457 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.457 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:31.457 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315327+0000", 2020-01-16T00:13:31.457 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.508e-05 2020-01-16T00:13:31.458 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.458 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.458 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:31.458 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315352+0000", 2020-01-16T00:13:31.458 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1183999999999998e-05 2020-01-16T00:13:31.458 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.458 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.458 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:31.459 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315383+0000", 2020-01-16T00:13:31.459 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7180000000000002e-05 2020-01-16T00:13:31.459 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.459 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.459 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:31.459 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315401+0000", 2020-01-16T00:13:31.459 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017290599999999999 2020-01-16T00:13:31.459 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.460 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.460 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.460 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315574+0000", 2020-01-16T00:13:31.460 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00047908100000000002 2020-01-16T00:13:31.460 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.460 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.460 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:31.461 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316053+0000", 2020-01-16T00:13:31.461 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011808947 2020-01-16T00:13:31.461 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.461 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.461 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.461 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.327862+0000", 2020-01-16T00:13:31.461 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010067499999999999 2020-01-16T00:13:31.462 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.462 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.462 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:31.462 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.327962+0000", 2020-01-16T00:13:31.462 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0074745139999999998 2020-01-16T00:13:31.463 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.463 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.463 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.463 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.335437+0000", 2020-01-16T00:13:31.463 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000103403 2020-01-16T00:13:31.464 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.464 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.464 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:31.464 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.335540+0000", 2020-01-16T00:13:31.465 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001014544 2020-01-16T00:13:31.465 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.465 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.465 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:31.465 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336555+0000", 2020-01-16T00:13:31.466 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021222494000000001 2020-01-16T00:13:31.466 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.466 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:31.466 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.466 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.467 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.467 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:982 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:31.467 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:13:31.467 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 53.092674576999997, 2020-01-16T00:13:31.468 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.022087721000000001, 2020-01-16T00:13:31.468 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:31.468 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:31.468 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:31.468 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:31.469 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:31.469 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 982 2020-01-16T00:13:31.469 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.469 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:31.470 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.470 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:31.470 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:13:31.470 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:31.471 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.471 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.472 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:31.472 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:13:31.473 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:31.473 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.474 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.474 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:31.474 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315433+0000", 2020-01-16T00:13:31.475 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1347e-05 2020-01-16T00:13:31.475 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.475 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.480 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:31.481 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315445+0000", 2020-01-16T00:13:31.481 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4525999999999999e-05 2020-01-16T00:13:31.481 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.481 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.481 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:31.482 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315459+0000", 2020-01-16T00:13:31.482 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.024e-05 2020-01-16T00:13:31.482 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.482 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.482 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:31.483 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315470+0000", 2020-01-16T00:13:31.483 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000736468 2020-01-16T00:13:31.483 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.483 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.483 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.483 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316206+0000", 2020-01-16T00:13:31.484 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000132967 2020-01-16T00:13:31.484 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.484 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.484 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:31.484 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316339+0000", 2020-01-16T00:13:31.485 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011701717 2020-01-16T00:13:31.485 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.485 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.485 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.485 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328041+0000", 2020-01-16T00:13:31.486 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000120339 2020-01-16T00:13:31.486 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.486 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.486 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:31.486 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328161+0000", 2020-01-16T00:13:31.487 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0084341260000000001 2020-01-16T00:13:31.487 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.487 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.487 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.487 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336595+0000", 2020-01-16T00:13:31.487 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5247999999999993e-05 2020-01-16T00:13:31.487 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.487 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.488 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:31.488 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336690+0000", 2020-01-16T00:13:31.488 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00083204000000000004 2020-01-16T00:13:31.488 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.488 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.488 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:31.488 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337522+0000", 2020-01-16T00:13:31.489 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.022087721000000001 2020-01-16T00:13:31.489 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.489 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:31.489 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.489 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.489 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.489 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:983 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:31.489 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:13:31.490 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 53.092554866999997, 2020-01-16T00:13:31.490 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023363387999999999, 2020-01-16T00:13:31.490 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:31.490 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:31.490 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:31.490 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:31.490 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:31.491 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 983 2020-01-16T00:13:31.491 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.491 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:31.491 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.491 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:31.491 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:13:31.491 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:31.491 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.492 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.492 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:31.492 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:13:31.492 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:31.492 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.492 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.492 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:31.492 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315553+0000", 2020-01-16T00:13:31.493 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1760000000000004e-06 2020-01-16T00:13:31.493 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.493 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.493 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:31.493 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315560+0000", 2020-01-16T00:13:31.493 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3587000000000001e-05 2020-01-16T00:13:31.493 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.494 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.494 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:31.494 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315574+0000", 2020-01-16T00:13:31.494 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7800000000000003e-06 2020-01-16T00:13:31.494 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.494 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.494 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:31.494 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315581+0000", 2020-01-16T00:13:31.495 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000823024 2020-01-16T00:13:31.495 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.495 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.495 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.495 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316404+0000", 2020-01-16T00:13:31.496 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2413e-05 2020-01-16T00:13:31.496 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.496 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.496 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:31.496 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316466+0000", 2020-01-16T00:13:31.497 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011738561999999999 2020-01-16T00:13:31.497 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.497 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.497 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.498 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328205+0000", 2020-01-16T00:13:31.498 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6783999999999997e-05 2020-01-16T00:13:31.498 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.498 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.498 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:31.498 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328262+0000", 2020-01-16T00:13:31.499 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0093022560000000001 2020-01-16T00:13:31.499 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.499 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.499 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.499 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337564+0000", 2020-01-16T00:13:31.499 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000183151 2020-01-16T00:13:31.499 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.500 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.500 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:31.500 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337747+0000", 2020-01-16T00:13:31.500 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011707449999999999 2020-01-16T00:13:31.500 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.500 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.500 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:31.500 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.338918+0000", 2020-01-16T00:13:31.501 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023363387999999999 2020-01-16T00:13:31.501 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.501 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:31.501 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.501 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.501 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.501 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:986 3.9 3:93dfe6f1:::smithi12313934-40:head [write 2941380~419987] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:31.502 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:13:31.502 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 52.927255090999999, 2020-01-16T00:13:31.502 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016978400000000001, 2020-01-16T00:13:31.502 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:31.502 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:31.502 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:31.502 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:31.503 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:31.503 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 986 2020-01-16T00:13:31.503 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.503 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:31.503 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.503 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:31.503 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:13:31.503 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:31.504 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.504 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.504 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:31.504 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:13:31.504 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:31.504 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.504 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.504 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:31.505 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480852+0000", 2020-01-16T00:13:31.505 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042503899999999999 2020-01-16T00:13:31.505 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.505 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.505 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:31.505 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481277+0000", 2020-01-16T00:13:31.505 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3097e-05 2020-01-16T00:13:31.506 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.506 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.506 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:31.506 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481290+0000", 2020-01-16T00:13:31.506 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8460000000000002e-06 2020-01-16T00:13:31.506 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.506 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.506 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:31.507 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481297+0000", 2020-01-16T00:13:31.507 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8443999999999999e-05 2020-01-16T00:13:31.507 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.507 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.507 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.507 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481346+0000", 2020-01-16T00:13:31.507 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022115399999999999 2020-01-16T00:13:31.507 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.508 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.508 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:31.508 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481567+0000", 2020-01-16T00:13:31.508 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012531502 2020-01-16T00:13:31.508 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.508 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.508 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.509 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494098+0000", 2020-01-16T00:13:31.509 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.4462999999999993e-05 2020-01-16T00:13:31.509 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.509 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.509 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:31.509 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494193+0000", 2020-01-16T00:13:31.509 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017600999999999999 2020-01-16T00:13:31.509 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.510 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.510 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:13:31.510 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494369+0000", 2020-01-16T00:13:31.510 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0031739619999999998 2020-01-16T00:13:31.510 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.510 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.510 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:31.510 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497543+0000", 2020-01-16T00:13:31.511 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000175107 2020-01-16T00:13:31.511 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.511 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.511 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:31.511 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497718+0000", 2020-01-16T00:13:31.511 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5207000000000003e-05 2020-01-16T00:13:31.511 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.511 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.512 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:31.512 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497763+0000", 2020-01-16T00:13:31.512 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7042999999999998e-05 2020-01-16T00:13:31.512 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.512 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.512 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:31.512 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497800+0000", 2020-01-16T00:13:31.513 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2628e-05 2020-01-16T00:13:31.513 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.513 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.513 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:31.513 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497833+0000", 2020-01-16T00:13:31.513 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016978400000000001 2020-01-16T00:13:31.513 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.513 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:31.514 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.514 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.514 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.514 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:987 3.9 3:93dfe6f1:::smithi12313934-40:head [setxattr _header (56) in=7b,truncate 3361367] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:31.514 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.481307+0000", 2020-01-16T00:13:31.514 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 52.926802723999998, 2020-01-16T00:13:31.514 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016794989, 2020-01-16T00:13:31.514 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:31.515 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:31.515 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:31.515 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:31.515 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:31.515 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 987 2020-01-16T00:13:31.515 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.515 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:31.516 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.516 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:31.516 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481307+0000", 2020-01-16T00:13:31.516 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:31.516 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.516 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.516 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:31.516 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481307+0000", 2020-01-16T00:13:31.517 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:31.517 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.517 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.517 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:31.517 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481306+0000", 2020-01-16T00:13:31.517 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.113e-06 2020-01-16T00:13:31.517 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.517 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.518 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:31.518 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481311+0000", 2020-01-16T00:13:31.518 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1520000000000002e-06 2020-01-16T00:13:31.518 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.518 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.518 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:31.518 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481315+0000", 2020-01-16T00:13:31.518 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.7640000000000001e-06 2020-01-16T00:13:31.519 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.519 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.519 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:31.519 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481318+0000", 2020-01-16T00:13:31.519 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00029012500000000002 2020-01-16T00:13:31.519 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.519 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.520 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.520 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481608+0000", 2020-01-16T00:13:31.520 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6239000000000002e-05 2020-01-16T00:13:31.520 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.520 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.520 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:31.520 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481654+0000", 2020-01-16T00:13:31.520 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014559534000000001 2020-01-16T00:13:31.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.521 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.521 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.521 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.496214+0000", 2020-01-16T00:13:31.521 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000225932 2020-01-16T00:13:31.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.521 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.521 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:31.522 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.496439+0000", 2020-01-16T00:13:31.522 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00031099600000000002 2020-01-16T00:13:31.522 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.522 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.522 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:13:31.522 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.496750+0000", 2020-01-16T00:13:31.522 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00098608599999999995 2020-01-16T00:13:31.522 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.523 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.523 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:31.523 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497737+0000", 2020-01-16T00:13:31.523 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022081499999999999 2020-01-16T00:13:31.523 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.523 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.523 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:31.524 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497957+0000", 2020-01-16T00:13:31.524 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7071000000000003e-05 2020-01-16T00:13:31.524 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.524 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.524 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:31.524 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498014+0000", 2020-01-16T00:13:31.524 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1111e-05 2020-01-16T00:13:31.524 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.525 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.525 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:31.525 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498056+0000", 2020-01-16T00:13:31.525 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5974999999999999e-05 2020-01-16T00:13:31.525 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.525 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.526 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:31.526 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498102+0000", 2020-01-16T00:13:31.526 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016794989 2020-01-16T00:13:31.526 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.526 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:31.526 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.526 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.526 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.527 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:988 3.9 3:93dfe6f1:::smithi12313934-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:13:31.527 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:13:31.527 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 52.926769626999999, 2020-01-16T00:13:31.527 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017221692, 2020-01-16T00:13:31.527 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:31.527 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:31.527 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:31.528 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:31.528 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:31.528 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 988 2020-01-16T00:13:31.528 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.528 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:31.528 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.528 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:31.528 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:13:31.529 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:31.529 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.529 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.529 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:31.529 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:13:31.529 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:31.529 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.529 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.530 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:31.530 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481339+0000", 2020-01-16T00:13:31.530 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3089999999999997e-06 2020-01-16T00:13:31.530 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.530 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.530 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:31.530 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481343+0000", 2020-01-16T00:13:31.531 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2230000000000004e-06 2020-01-16T00:13:31.531 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.531 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.531 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:31.531 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481350+0000", 2020-01-16T00:13:31.531 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.319e-06 2020-01-16T00:13:31.531 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.531 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.532 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:31.532 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481356+0000", 2020-01-16T00:13:31.532 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000321544 2020-01-16T00:13:31.532 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.532 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.532 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.532 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481678+0000", 2020-01-16T00:13:31.532 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6375999999999999e-05 2020-01-16T00:13:31.533 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.533 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.533 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:31.533 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481714+0000", 2020-01-16T00:13:31.533 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015557666 2020-01-16T00:13:31.533 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.533 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.533 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.534 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497272+0000", 2020-01-16T00:13:31.534 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012791300000000001 2020-01-16T00:13:31.534 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.534 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.534 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:31.534 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497399+0000", 2020-01-16T00:13:31.534 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00076944600000000002 2020-01-16T00:13:31.534 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.535 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.535 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.535 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498169+0000", 2020-01-16T00:13:31.535 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018306799999999999 2020-01-16T00:13:31.535 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.535 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.535 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:31.536 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498352+0000", 2020-01-16T00:13:31.536 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020935999999999999 2020-01-16T00:13:31.536 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.536 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.536 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:31.536 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498561+0000", 2020-01-16T00:13:31.536 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017221692 2020-01-16T00:13:31.536 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.537 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:31.537 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.537 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.537 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.537 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1239 3.3 3:c204aa4f:::smithi12313934-45:head [setxattr _header (56) in=7b,truncate 2000033] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:13:31.537 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:13:31.537 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 48.817695808000003, 2020-01-16T00:13:31.537 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019161661999999999, 2020-01-16T00:13:31.538 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:31.538 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:31.538 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:31.538 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:31.538 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:31.538 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1239 2020-01-16T00:13:31.538 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.539 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:31.539 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.539 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:31.539 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:13:31.539 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:31.539 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.539 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.539 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:31.540 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:13:31.540 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:31.540 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.540 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.540 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:31.540 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590411+0000", 2020-01-16T00:13:31.540 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3879999999999999e-05 2020-01-16T00:13:31.540 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.541 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.541 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:31.541 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590425+0000", 2020-01-16T00:13:31.541 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6542999999999998e-05 2020-01-16T00:13:31.541 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.541 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.541 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:31.542 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590471+0000", 2020-01-16T00:13:31.542 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8907999999999999e-05 2020-01-16T00:13:31.542 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.542 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.542 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:31.542 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590490+0000", 2020-01-16T00:13:31.542 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0045691960000000002 2020-01-16T00:13:31.542 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.543 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.543 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.543 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.595059+0000", 2020-01-16T00:13:31.543 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0035638940000000002 2020-01-16T00:13:31.543 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.543 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.543 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:31.543 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598623+0000", 2020-01-16T00:13:31.544 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0077558310000000004 2020-01-16T00:13:31.544 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.544 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.544 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.544 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606379+0000", 2020-01-16T00:13:31.544 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012171999999999999 2020-01-16T00:13:31.544 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.544 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.545 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:31.545 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606501+0000", 2020-01-16T00:13:31.545 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013144200000000001 2020-01-16T00:13:31.545 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.545 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.545 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2020-01-16T00:13:31.545 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606632+0000", 2020-01-16T00:13:31.546 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001288565 2020-01-16T00:13:31.546 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.546 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.546 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:31.546 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607921+0000", 2020-01-16T00:13:31.546 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033399299999999999 2020-01-16T00:13:31.546 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.546 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.547 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:31.547 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.608255+0000", 2020-01-16T00:13:31.547 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012280870000000001 2020-01-16T00:13:31.547 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.547 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.547 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:31.547 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609483+0000", 2020-01-16T00:13:31.547 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2286000000000001e-05 2020-01-16T00:13:31.548 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.548 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.548 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:31.548 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609525+0000", 2020-01-16T00:13:31.548 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0121999999999999e-05 2020-01-16T00:13:31.548 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.548 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.548 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:31.549 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609575+0000", 2020-01-16T00:13:31.549 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019161661999999999 2020-01-16T00:13:31.549 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.549 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:31.549 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.549 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.549 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.550 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1240 3.3 3:c204aa4f:::smithi12313934-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-01-16T00:13:31.550 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:13:31.550 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 48.817595619000002, 2020-01-16T00:13:31.550 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019268828000000002, 2020-01-16T00:13:31.550 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:31.550 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:31.550 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:31.550 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:31.551 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:31.551 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1240 2020-01-16T00:13:31.551 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.551 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:31.551 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.551 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:31.551 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:13:31.551 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:31.552 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.552 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.552 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:31.552 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:13:31.552 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:31.552 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.552 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.553 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:31.553 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590512+0000", 2020-01-16T00:13:31.553 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0525e-05 2020-01-16T00:13:31.553 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.553 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.553 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:31.553 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590522+0000", 2020-01-16T00:13:31.553 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6418000000000001e-05 2020-01-16T00:13:31.554 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.554 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.554 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:31.554 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590539+0000", 2020-01-16T00:13:31.554 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0679999999999999e-05 2020-01-16T00:13:31.554 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.554 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.554 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:31.555 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590549+0000", 2020-01-16T00:13:31.555 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0083626789999999996 2020-01-16T00:13:31.555 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.555 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.555 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.555 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598912+0000", 2020-01-16T00:13:31.555 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013344 2020-01-16T00:13:31.555 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.556 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.556 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:31.556 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599045+0000", 2020-01-16T00:13:31.556 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0084002269999999997 2020-01-16T00:13:31.556 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.556 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.556 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.556 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607446+0000", 2020-01-16T00:13:31.557 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000109873 2020-01-16T00:13:31.557 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.557 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.557 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:31.557 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607555+0000", 2020-01-16T00:13:31.557 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0020573309999999999 2020-01-16T00:13:31.557 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.558 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.558 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.558 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609613+0000", 2020-01-16T00:13:31.558 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6351e-05 2020-01-16T00:13:31.558 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.558 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.558 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:31.558 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609699+0000", 2020-01-16T00:13:31.559 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3374999999999994e-05 2020-01-16T00:13:31.559 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.559 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.559 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:31.559 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609782+0000", 2020-01-16T00:13:31.559 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019268828000000002 2020-01-16T00:13:31.559 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.559 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:31.560 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.560 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.560 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.560 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1437 3.3 3:c204aa4f:::smithi12313934-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e28)", 2020-01-16T00:13:31.560 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:47.900594+0000", 2020-01-16T00:13:31.560 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 43.507515075000001, 2020-01-16T00:13:31.560 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016745237999999999, 2020-01-16T00:13:31.560 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:31.561 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:31.561 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:31.561 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:31.561 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:31.561 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1437 2020-01-16T00:13:31.561 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.561 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:31.562 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.562 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:31.562 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900594+0000", 2020-01-16T00:13:31.562 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:31.562 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.562 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.562 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:31.562 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900594+0000", 2020-01-16T00:13:31.563 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:31.563 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.563 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.563 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:31.563 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900593+0000", 2020-01-16T00:13:31.563 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5059999999999995e-06 2020-01-16T00:13:31.563 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.563 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.564 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:31.564 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900601+0000", 2020-01-16T00:13:31.564 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5077e-05 2020-01-16T00:13:31.564 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.564 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.564 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:31.564 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900616+0000", 2020-01-16T00:13:31.565 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0183000000000001e-05 2020-01-16T00:13:31.565 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.565 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.565 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:31.565 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900627+0000", 2020-01-16T00:13:31.565 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0063667660000000003 2020-01-16T00:13:31.565 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.565 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.566 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.566 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.906993+0000", 2020-01-16T00:13:31.566 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8988e-05 2020-01-16T00:13:31.566 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.566 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.566 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:31.566 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.907032+0000", 2020-01-16T00:13:31.566 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0080128850000000008 2020-01-16T00:13:31.567 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.567 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.567 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.567 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.915045+0000", 2020-01-16T00:13:31.567 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013200899999999999 2020-01-16T00:13:31.567 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.567 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.567 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:31.568 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.915177+0000", 2020-01-16T00:13:31.568 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0018373090000000001 2020-01-16T00:13:31.568 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.568 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.568 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.568 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.917015+0000", 2020-01-16T00:13:31.568 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016701699999999999 2020-01-16T00:13:31.569 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.569 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.569 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:31.569 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.917182+0000", 2020-01-16T00:13:31.569 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015782400000000001 2020-01-16T00:13:31.569 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.569 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.569 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:31.570 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.917339+0000", 2020-01-16T00:13:31.570 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016745237999999999 2020-01-16T00:13:31.570 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.570 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:31.570 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.570 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.570 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.570 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3124 3.9 3:9fc687b2:::smithi12313934-44:head [write 1903628~435462] snapc 0=[] ondisk+write+known_if_redirected e44)", 2020-01-16T00:13:31.571 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.276145+0000", 2020-01-16T00:13:31.571 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 6.1319638980000004, 2020-01-16T00:13:31.571 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017361898000000001, 2020-01-16T00:13:31.571 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:31.571 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:31.571 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:31.571 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:31.571 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:31.572 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3124 2020-01-16T00:13:31.572 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.572 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:31.572 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.572 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:31.572 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276145+0000", 2020-01-16T00:13:31.572 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:31.573 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.573 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.573 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:31.573 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276145+0000", 2020-01-16T00:13:31.573 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:31.573 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.573 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.573 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:31.574 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276143+0000", 2020-01-16T00:13:31.574 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042136099999999998 2020-01-16T00:13:31.574 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.574 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.574 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:31.574 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276564+0000", 2020-01-16T00:13:31.574 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.349e-05 2020-01-16T00:13:31.574 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.575 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.575 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:31.575 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276578+0000", 2020-01-16T00:13:31.575 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2469999999999999e-06 2020-01-16T00:13:31.575 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.575 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.575 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:31.575 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276583+0000", 2020-01-16T00:13:31.576 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4400000000000001e-05 2020-01-16T00:13:31.576 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.576 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.576 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.576 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276637+0000", 2020-01-16T00:13:31.576 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0046361789999999998 2020-01-16T00:13:31.576 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.577 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.577 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:31.577 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281274+0000", 2020-01-16T00:13:31.577 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0090025520000000005 2020-01-16T00:13:31.577 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.577 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.577 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.577 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.290276+0000", 2020-01-16T00:13:31.578 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011584200000000001 2020-01-16T00:13:31.578 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.578 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.578 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:31.578 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.290392+0000", 2020-01-16T00:13:31.578 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00025563099999999998 2020-01-16T00:13:31.578 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.578 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.579 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:13:31.579 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.290648+0000", 2020-01-16T00:13:31.579 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0026741360000000001 2020-01-16T00:13:31.579 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.579 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.579 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:31.579 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293322+0000", 2020-01-16T00:13:31.579 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9764000000000002e-05 2020-01-16T00:13:31.580 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.580 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.580 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:31.580 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293421+0000", 2020-01-16T00:13:31.580 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4779000000000001e-05 2020-01-16T00:13:31.580 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.580 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.581 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:31.581 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293446+0000", 2020-01-16T00:13:31.581 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4628000000000001e-05 2020-01-16T00:13:31.581 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.581 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.581 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:31.581 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293481+0000", 2020-01-16T00:13:31.581 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.6444999999999998e-05 2020-01-16T00:13:31.582 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.582 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.582 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:31.582 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293507+0000", 2020-01-16T00:13:31.582 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017361898000000001 2020-01-16T00:13:31.582 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.582 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:31.582 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.583 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.583 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.583 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3125 3.9 3:9fc687b2:::smithi12313934-44:head [setxattr _header (56) in=7b,truncate 2386660] snapc 0=[] ondisk+write+known_if_redirected e44)", 2020-01-16T00:13:31.583 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.276594+0000", 2020-01-16T00:13:31.583 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 6.131515126, 2020-01-16T00:13:31.583 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017760560000000002, 2020-01-16T00:13:31.583 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:31.583 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:31.584 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:31.584 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:31.584 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:31.584 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3125 2020-01-16T00:13:31.584 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.584 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:31.584 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.584 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:31.585 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276594+0000", 2020-01-16T00:13:31.585 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:31.585 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.585 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.585 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:31.585 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276594+0000", 2020-01-16T00:13:31.585 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:31.586 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.586 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.586 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:31.586 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276593+0000", 2020-01-16T00:13:31.586 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7320000000000003e-06 2020-01-16T00:13:31.586 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.586 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.586 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:31.587 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276599+0000", 2020-01-16T00:13:31.587 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3302000000000001e-05 2020-01-16T00:13:31.587 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.587 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.587 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:31.587 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276612+0000", 2020-01-16T00:13:31.587 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1659999999999998e-06 2020-01-16T00:13:31.588 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.588 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.588 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:31.588 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276615+0000", 2020-01-16T00:13:31.588 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0047328450000000003 2020-01-16T00:13:31.588 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.588 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.588 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.589 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281348+0000", 2020-01-16T00:13:31.589 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3966000000000003e-05 2020-01-16T00:13:31.589 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.589 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.589 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:31.589 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281422+0000", 2020-01-16T00:13:31.589 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01082613 2020-01-16T00:13:31.589 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.590 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.590 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.590 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.292248+0000", 2020-01-16T00:13:31.590 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000148617 2020-01-16T00:13:31.590 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.590 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.590 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:31.590 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.292397+0000", 2020-01-16T00:13:31.591 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019936300000000001 2020-01-16T00:13:31.591 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.591 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.591 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:13:31.591 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.292596+0000", 2020-01-16T00:13:31.591 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011208870000000001 2020-01-16T00:13:31.591 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.592 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.592 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:31.592 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293717+0000", 2020-01-16T00:13:31.592 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00039491599999999998 2020-01-16T00:13:31.592 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.592 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.592 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:31.592 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294112+0000", 2020-01-16T00:13:31.593 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000129326 2020-01-16T00:13:31.593 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.593 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.593 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:31.593 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294241+0000", 2020-01-16T00:13:31.593 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5586000000000001e-05 2020-01-16T00:13:31.593 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.593 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.594 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:31.594 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294277+0000", 2020-01-16T00:13:31.594 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7893999999999997e-05 2020-01-16T00:13:31.594 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.594 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.594 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:31.594 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294355+0000", 2020-01-16T00:13:31.595 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017760560000000002 2020-01-16T00:13:31.595 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.595 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:31.595 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.595 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.595 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.595 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3126 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e44)", 2020-01-16T00:13:31.595 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.276634+0000", 2020-01-16T00:13:31.596 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 6.1314754110000003, 2020-01-16T00:13:31.596 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017942583000000002, 2020-01-16T00:13:31.596 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:31.596 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:31.596 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:31.596 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:31.596 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:31.596 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3126 2020-01-16T00:13:31.597 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.597 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:31.597 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.597 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:31.597 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276634+0000", 2020-01-16T00:13:31.597 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:31.597 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.598 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.598 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:31.598 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276634+0000", 2020-01-16T00:13:31.598 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:31.598 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.598 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.598 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:31.598 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276633+0000", 2020-01-16T00:13:31.599 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0769999999999999e-06 2020-01-16T00:13:31.599 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.600 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.601 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:31.601 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276636+0000", 2020-01-16T00:13:31.601 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5390000000000001e-06 2020-01-16T00:13:31.601 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.601 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.601 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:31.602 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276641+0000", 2020-01-16T00:13:31.602 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9969999999999999e-06 2020-01-16T00:13:31.602 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.602 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.602 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:31.602 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276644+0000", 2020-01-16T00:13:31.602 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0048246970000000002 2020-01-16T00:13:31.603 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.603 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.603 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.603 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281469+0000", 2020-01-16T00:13:31.603 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8723999999999999e-05 2020-01-16T00:13:31.603 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.603 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.603 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:31.604 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281567+0000", 2020-01-16T00:13:31.604 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011633447 2020-01-16T00:13:31.604 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.604 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.604 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.604 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293201+0000", 2020-01-16T00:13:31.604 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6796000000000002e-05 2020-01-16T00:13:31.605 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.605 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.605 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:31.605 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293268+0000", 2020-01-16T00:13:31.605 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001125247 2020-01-16T00:13:31.605 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.605 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.605 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.606 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294393+0000", 2020-01-16T00:13:31.606 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.6082000000000006e-05 2020-01-16T00:13:31.606 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.606 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.606 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:31.606 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294489+0000", 2020-01-16T00:13:31.606 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7515000000000005e-05 2020-01-16T00:13:31.607 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.607 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.607 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:31.607 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294576+0000", 2020-01-16T00:13:31.607 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017942583000000002 2020-01-16T00:13:31.607 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.607 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:31.607 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.608 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.608 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.608 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3204 3.9 3:93dfe6f1:::smithi12313934-40:head [setxattr _header (56) in=7b,truncate 2554730] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:13:31.608 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.568544+0000", 2020-01-16T00:13:31.608 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 1.8395651909999999, 2020-01-16T00:13:31.608 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.026260867, 2020-01-16T00:13:31.608 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:31.608 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:31.609 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:31.609 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:31.609 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:31.609 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3204 2020-01-16T00:13:31.609 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.609 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:31.609 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.610 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:31.610 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568544+0000", 2020-01-16T00:13:31.610 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:31.610 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.610 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.610 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:31.610 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568544+0000", 2020-01-16T00:13:31.610 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:31.611 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.611 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.611 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:31.611 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568543+0000", 2020-01-16T00:13:31.611 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0810000000000001e-06 2020-01-16T00:13:31.611 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.611 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.611 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:31.612 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568548+0000", 2020-01-16T00:13:31.612 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6950000000000002e-06 2020-01-16T00:13:31.612 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.612 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.612 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:31.612 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568552+0000", 2020-01-16T00:13:31.612 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.875e-06 2020-01-16T00:13:31.613 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.613 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.613 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:31.613 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568555+0000", 2020-01-16T00:13:31.613 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0033130849999999999 2020-01-16T00:13:31.613 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.613 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.613 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.614 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.571868+0000", 2020-01-16T00:13:31.614 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0026014609999999998 2020-01-16T00:13:31.614 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.614 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.614 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:31.614 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.574469+0000", 2020-01-16T00:13:31.614 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010261381999999999 2020-01-16T00:13:31.614 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.615 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.615 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.615 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.584731+0000", 2020-01-16T00:13:31.615 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019704 2020-01-16T00:13:31.615 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.615 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.615 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:31.616 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.584928+0000", 2020-01-16T00:13:31.616 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000280869 2020-01-16T00:13:31.616 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.616 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.616 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:13:31.616 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.585209+0000", 2020-01-16T00:13:31.616 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002273859 2020-01-16T00:13:31.616 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.617 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.617 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:31.617 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.587483+0000", 2020-01-16T00:13:31.617 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3356000000000001e-05 2020-01-16T00:13:31.617 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.617 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.617 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:31.617 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.587506+0000", 2020-01-16T00:13:31.618 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0071864939999999999 2020-01-16T00:13:31.618 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.618 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.618 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:31.618 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594692+0000", 2020-01-16T00:13:31.618 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5475000000000001e-05 2020-01-16T00:13:31.618 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.619 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.619 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:31.619 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594738+0000", 2020-01-16T00:13:31.619 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7070000000000007e-05 2020-01-16T00:13:31.619 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.619 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.619 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:31.619 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594805+0000", 2020-01-16T00:13:31.620 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.026260867 2020-01-16T00:13:31.620 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.620 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:31.620 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.620 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.620 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.620 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3205 3.9 3:93dfe6f1:::smithi12313934-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e45)", 2020-01-16T00:13:31.620 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.568599+0000", 2020-01-16T00:13:31.621 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 1.8395102560000001, 2020-01-16T00:13:31.621 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.026406882, 2020-01-16T00:13:31.621 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:31.621 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:31.621 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:31.621 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:31.621 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:31.622 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3205 2020-01-16T00:13:31.622 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.622 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:31.622 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.622 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:31.622 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568599+0000", 2020-01-16T00:13:31.622 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:31.622 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.623 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.623 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:31.623 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568599+0000", 2020-01-16T00:13:31.623 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:31.623 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.623 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.623 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:31.623 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568598+0000", 2020-01-16T00:13:31.624 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3939999999999998e-06 2020-01-16T00:13:31.624 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.624 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.624 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:31.624 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568603+0000", 2020-01-16T00:13:31.624 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9391000000000001e-05 2020-01-16T00:13:31.624 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.624 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.625 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:31.625 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568622+0000", 2020-01-16T00:13:31.625 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9599999999999997e-06 2020-01-16T00:13:31.625 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.625 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.626 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:31.626 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568628+0000", 2020-01-16T00:13:31.626 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0058679700000000001 2020-01-16T00:13:31.626 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.626 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.626 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.626 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.574496+0000", 2020-01-16T00:13:31.626 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5528999999999998e-05 2020-01-16T00:13:31.627 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.627 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.627 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:31.627 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.574532+0000", 2020-01-16T00:13:31.627 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011986749 2020-01-16T00:13:31.627 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.627 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.628 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.628 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.586518+0000", 2020-01-16T00:13:31.628 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016522600000000001 2020-01-16T00:13:31.628 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.628 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.628 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:31.628 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.586684+0000", 2020-01-16T00:13:31.628 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0081693890000000009 2020-01-16T00:13:31.629 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.629 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.629 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.629 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594853+0000", 2020-01-16T00:13:31.629 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7082000000000004e-05 2020-01-16T00:13:31.629 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.629 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.630 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:31.630 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594940+0000", 2020-01-16T00:13:31.630 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5789000000000003e-05 2020-01-16T00:13:31.630 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.630 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.630 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:31.630 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.595006+0000", 2020-01-16T00:13:31.630 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.026406882 2020-01-16T00:13:31.631 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.631 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:31.631 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.631 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.631 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.631 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3219 3.3 3:c478bfe8:::smithi12313934-10:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:13:31.631 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.580957+0000", 2020-01-16T00:13:31.632 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 1.8271527400000001, 2020-01-16T00:13:31.632 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016807971000000001, 2020-01-16T00:13:31.632 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:31.632 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:31.632 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:31.632 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:31.632 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:31.632 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3219 2020-01-16T00:13:31.633 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.633 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:31.633 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.633 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:31.633 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580957+0000", 2020-01-16T00:13:31.633 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:31.633 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.633 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.634 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:31.634 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580957+0000", 2020-01-16T00:13:31.634 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:31.634 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.634 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.634 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:31.634 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580955+0000", 2020-01-16T00:13:31.635 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5120000000000002e-06 2020-01-16T00:13:31.635 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.635 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.635 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:31.635 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580963+0000", 2020-01-16T00:13:31.635 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6799999999999993e-06 2020-01-16T00:13:31.635 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.635 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.636 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:31.636 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580971+0000", 2020-01-16T00:13:31.636 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3109999999999996e-06 2020-01-16T00:13:31.636 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.636 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.636 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:31.636 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580976+0000", 2020-01-16T00:13:31.636 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014306146 2020-01-16T00:13:31.637 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.637 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.637 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.637 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.595282+0000", 2020-01-16T00:13:31.637 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.1883000000000001e-05 2020-01-16T00:13:31.637 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.637 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.638 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:31.638 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.595374+0000", 2020-01-16T00:13:31.638 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023904989999999999 2020-01-16T00:13:31.638 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.638 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.638 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:31.638 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.597765+0000", 2020-01-16T00:13:31.638 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016807971000000001 2020-01-16T00:13:31.639 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.639 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:31.639 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.639 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.639 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.639 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3220 3.3 3:c478bfe8:::smithi12313934-10:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:13:31.639 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.580982+0000", 2020-01-16T00:13:31.639 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 1.8271276919999999, 2020-01-16T00:13:31.640 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017920330000000002, 2020-01-16T00:13:31.640 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:31.640 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:31.640 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:31.640 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:31.640 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:31.640 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3220 2020-01-16T00:13:31.640 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.641 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:31.641 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.641 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:31.641 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580982+0000", 2020-01-16T00:13:31.641 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:31.641 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.641 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.642 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:31.642 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580982+0000", 2020-01-16T00:13:31.642 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:31.642 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.642 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.642 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:31.642 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580981+0000", 2020-01-16T00:13:31.642 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0380000000000001e-06 2020-01-16T00:13:31.643 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.643 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.643 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:31.643 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580985+0000", 2020-01-16T00:13:31.643 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8929000000000001e-05 2020-01-16T00:13:31.643 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.643 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.643 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:31.644 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581004+0000", 2020-01-16T00:13:31.644 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7339999999999999e-06 2020-01-16T00:13:31.644 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.644 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.644 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:31.644 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581009+0000", 2020-01-16T00:13:31.644 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01679812 2020-01-16T00:13:31.644 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.645 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.645 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.645 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.597807+0000", 2020-01-16T00:13:31.645 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013333400000000001 2020-01-16T00:13:31.645 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.645 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.646 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:31.646 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.597940+0000", 2020-01-16T00:13:31.646 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000961899 2020-01-16T00:13:31.646 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.646 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.646 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:31.646 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.598902+0000", 2020-01-16T00:13:31.646 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017920330000000002 2020-01-16T00:13:31.647 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.647 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:31.647 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.647 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.647 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.647 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3221 3.3 3:c478bfe8:::smithi12313934-10:head [read 0~2403467,checksum] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:13:31.647 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.581014+0000", 2020-01-16T00:13:31.647 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 1.8270953430000001, 2020-01-16T00:13:31.648 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018784551999999999, 2020-01-16T00:13:31.648 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:31.648 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:31.648 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:31.648 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:31.648 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:31.648 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3221 2020-01-16T00:13:31.648 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.649 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:31.649 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.649 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:31.649 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581014+0000", 2020-01-16T00:13:31.649 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:31.649 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.649 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.649 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:31.650 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581014+0000", 2020-01-16T00:13:31.650 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:31.650 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.650 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.650 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:31.650 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581013+0000", 2020-01-16T00:13:31.650 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9390000000000002e-06 2020-01-16T00:13:31.651 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.651 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.651 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:31.651 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581019+0000", 2020-01-16T00:13:31.651 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1131000000000001e-05 2020-01-16T00:13:31.651 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.651 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.651 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:31.652 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581030+0000", 2020-01-16T00:13:31.652 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.55e-06 2020-01-16T00:13:31.652 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.652 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.652 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:31.652 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581037+0000", 2020-01-16T00:13:31.652 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017890321000000001 2020-01-16T00:13:31.652 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.653 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.653 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:31.653 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.598927+0000", 2020-01-16T00:13:31.653 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000105999 2020-01-16T00:13:31.653 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.653 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.653 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:31.653 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.599033+0000", 2020-01-16T00:13:31.654 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00076542400000000001 2020-01-16T00:13:31.654 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:31.654 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:31.654 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:31.654 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.599798+0000", 2020-01-16T00:13:31.654 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018784551999999999 2020-01-16T00:13:31.654 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.654 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:31.655 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.655 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:31.655 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:31.655 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:31.659 INFO:tasks.rados.rados.0.smithi123.stdout:1292: finishing write tid 1 to smithi12313934-7 2020-01-16T00:13:31.660 INFO:tasks.rados.rados.0.smithi123.stdout:1292: finishing write tid 2 to smithi12313934-7 2020-01-16T00:13:31.660 INFO:tasks.rados.rados.0.smithi123.stdout:1292: finishing write tid 4 to smithi12313934-7 2020-01-16T00:13:31.660 INFO:tasks.rados.rados.0.smithi123.stdout:1292: finishing write tid 5 to smithi12313934-7 2020-01-16T00:13:31.660 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 7 v 186 (ObjNum 399 snap 0 seq_num 399) dirty exists 2020-01-16T00:13:31.660 INFO:tasks.rados.rados.0.smithi123.stdout:1292: left oid 7 (ObjNum 399 snap 0 seq_num 399) 2020-01-16T00:13:31.660 INFO:tasks.rados.rados.0.smithi123.stdout:1293: finishing write tid 1 to smithi12313934-40 2020-01-16T00:13:31.660 INFO:tasks.rados.rados.0.smithi123.stdout:1288: done (15 left) 2020-01-16T00:13:31.661 INFO:tasks.rados.rados.0.smithi123.stdout:1289: done (14 left) 2020-01-16T00:13:31.661 INFO:tasks.rados.rados.0.smithi123.stdout:1290: done (13 left) 2020-01-16T00:13:31.661 INFO:tasks.rados.rados.0.smithi123.stdout:1292: done (12 left) 2020-01-16T00:13:31.661 INFO:tasks.rados.rados.0.smithi123.stdout:1295: done (11 left) 2020-01-16T00:13:31.661 INFO:tasks.rados.rados.0.smithi123.stdout:1300: done (10 left) 2020-01-16T00:13:31.661 INFO:tasks.rados.rados.0.smithi123.stdout:1305: read oid 6 snap -1 2020-01-16T00:13:31.661 INFO:tasks.rados.rados.0.smithi123.stdout:1305: expect (ObjNum 376 snap 0 seq_num 376) 2020-01-16T00:13:31.661 INFO:tasks.rados.rados.0.smithi123.stdout:1306: read oid 14 snap -1 2020-01-16T00:13:31.662 INFO:tasks.rados.rados.0.smithi123.stdout:1306: expect (ObjNum 362 snap 0 seq_num 362) 2020-01-16T00:13:31.662 INFO:tasks.rados.rados.0.smithi123.stdout:1307: write oid 39 current snap is 0 2020-01-16T00:13:31.662 INFO:tasks.rados.rados.0.smithi123.stdout:1307: seq_num 403 ranges {773221=492056,1816008=503126} 2020-01-16T00:13:31.662 INFO:tasks.rados.rados.0.smithi123.stdout:1307: writing smithi12313934-39 from 773221 to 1265277 tid 1 2020-01-16T00:13:31.662 INFO:tasks.rados.rados.0.smithi123.stdout:1307: writing smithi12313934-39 from 1816008 to 2319134 tid 2 2020-01-16T00:13:31.662 INFO:tasks.rados.rados.0.smithi123.stdout:1308: write (excl) oid 49 current snap is 0 2020-01-16T00:13:31.662 INFO:tasks.rados.rados.0.smithi123.stdout:1308: seq_num 404 ranges {790028=567305,1874490=763137} 2020-01-16T00:13:31.662 INFO:tasks.rados.rados.0.smithi123.stdout:1308: writing smithi12313934-49 from 790028 to 1357333 tid 1 2020-01-16T00:13:31.663 INFO:tasks.rados.rados.0.smithi123.stdout:1308: writing smithi12313934-49 from 1874490 to 2637627 tid 2 2020-01-16T00:13:31.663 INFO:tasks.rados.rados.0.smithi123.stdout:1309: write oid 10 current snap is 0 2020-01-16T00:13:31.663 INFO:tasks.rados.rados.0.smithi123.stdout:1309: seq_num 405 ranges {406835=642554,1532972=506795,2772312=34523} 2020-01-16T00:13:31.663 INFO:tasks.rados.rados.0.smithi123.stdout:1309: writing smithi12313934-10 from 406835 to 1049389 tid 1 2020-01-16T00:13:31.663 INFO:tasks.rados.rados.0.smithi123.stdout:1309: writing smithi12313934-10 from 1532972 to 2039767 tid 2 2020-01-16T00:13:31.663 INFO:tasks.rados.rados.0.smithi123.stdout:1309: writing smithi12313934-10 from 2772312 to 2806835 tid 3 2020-01-16T00:13:31.663 INFO:tasks.rados.rados.0.smithi123.stdout:1310: read oid 23 snap -1 2020-01-16T00:13:31.664 INFO:tasks.rados.rados.0.smithi123.stdout:1310: expect deleted 2020-01-16T00:13:31.664 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:31.664 INFO:tasks.rados.rados.0.smithi123.stdout:1293: finishing write tid 2 to smithi12313934-40 2020-01-16T00:13:31.664 INFO:tasks.rados.rados.0.smithi123.stdout:1296: expect (ObjNum 382 snap 0 seq_num 382) 2020-01-16T00:13:31.664 INFO:tasks.rados.rados.0.smithi123.stdout:1293: finishing write tid 4 to smithi12313934-40 2020-01-16T00:13:31.664 INFO:tasks.rados.rados.0.smithi123.stdout:1293: finishing write tid 5 to smithi12313934-40 2020-01-16T00:13:31.664 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 40 v 193 (ObjNum 400 snap 0 seq_num 400) dirty exists 2020-01-16T00:13:31.664 INFO:tasks.rados.rados.0.smithi123.stdout:1293: left oid 40 (ObjNum 400 snap 0 seq_num 400) 2020-01-16T00:13:31.665 INFO:tasks.rados.rados.0.smithi123.stdout:1301: finishing write tid 1 to smithi12313934-42 2020-01-16T00:13:31.665 INFO:tasks.rados.rados.0.smithi123.stdout:1297: expect (ObjNum 310 snap 0 seq_num 310) 2020-01-16T00:13:31.665 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:13:31.665 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:13:31.665 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:13:31.665 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:13:31.665 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:13:31.666 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:13:31.666 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:13:31.666 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-01-16T00:13:31.666 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-01-16T00:13:31.666 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-01-16T00:13:31.666 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-01-16T00:13:31.666 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-01-16T00:13:31.667 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENXIO: problem getting command descriptions from osd.5 2020-01-16T00:13:31.667 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: problem getting command descriptions from osd.5 2020-01-16T00:13:31.667 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-01-16T00:13:31.667 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-01-16T00:13:31.667 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-01-16T00:13:31.667 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-01-16T00:13:31.667 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:31.654+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:31.668 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-01-16T00:13:31.757 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:31.754+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:31.785 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:32.051 INFO:tasks.rados.rados.0.smithi123.stdout:1302: expect (ObjNum 392 snap 0 seq_num 392) 2020-01-16T00:13:32.051 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:32.048+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:32.052 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-01-16T00:13:32.109 INFO:tasks.rados.rados.0.smithi123.stdout:1303: expect (ObjNum 378 snap 0 seq_num 378) 2020-01-16T00:13:32.152 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:32.150+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:32.215 INFO:tasks.rados.rados.0.smithi123.stdout:1299: expect (ObjNum 335 snap 0 seq_num 335) 2020-01-16T00:13:32.216 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:32.219 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-01-16T00:13:32.254 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:32.250+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:32.334 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:32.339 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-01-16T00:13:32.354 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:32.350+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:32.429 INFO:tasks.rados.rados.0.smithi123.stdout:1301: finishing write tid 2 to smithi12313934-42 2020-01-16T00:13:32.429 INFO:tasks.rados.rados.0.smithi123.stdout:1301: finishing write tid 4 to smithi12313934-42 2020-01-16T00:13:32.429 INFO:tasks.rados.rados.0.smithi123.stdout:1301: finishing write tid 5 to smithi12313934-42 2020-01-16T00:13:32.429 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 42 v 145 (ObjNum 401 snap 0 seq_num 401) dirty exists 2020-01-16T00:13:32.430 INFO:tasks.rados.rados.0.smithi123.stdout:1301: left oid 42 (ObjNum 401 snap 0 seq_num 401) 2020-01-16T00:13:32.430 INFO:tasks.rados.rados.0.smithi123.stdout:1294: expect (ObjNum 336 snap 0 seq_num 336) 2020-01-16T00:13:32.447 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:32.455 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:32.450+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:32.457 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-01-16T00:13:32.579 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:32.560+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:32.579 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:32.579 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-01-16T00:13:32.597 INFO:tasks.rados.rados.0.smithi123.stdout:1298: expect (ObjNum 343 snap 0 seq_num 343) 2020-01-16T00:13:32.664 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:32.662+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:32.731 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:32.741 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-01-16T00:13:32.788 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:32.786+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:32.834 INFO:tasks.rados.rados.0.smithi123.stdout:1304: finishing write tid 1 to smithi12313934-3 2020-01-16T00:13:32.835 INFO:tasks.rados.rados.0.smithi123.stdout:1304: finishing write tid 2 to smithi12313934-3 2020-01-16T00:13:32.835 INFO:tasks.rados.rados.0.smithi123.stdout:1304: finishing write tid 4 to smithi12313934-3 2020-01-16T00:13:32.835 INFO:tasks.rados.rados.0.smithi123.stdout:1304: finishing write tid 5 to smithi12313934-3 2020-01-16T00:13:32.835 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 3 v 88 (ObjNum 402 snap 0 seq_num 402) dirty exists 2020-01-16T00:13:32.835 INFO:tasks.rados.rados.0.smithi123.stdout:1304: left oid 3 (ObjNum 402 snap 0 seq_num 402) 2020-01-16T00:13:32.835 INFO:tasks.rados.rados.0.smithi123.stdout:1293: done (15 left) 2020-01-16T00:13:32.836 INFO:tasks.rados.rados.0.smithi123.stdout:1294: done (14 left) 2020-01-16T00:13:32.836 INFO:tasks.rados.rados.0.smithi123.stdout:1296: done (13 left) 2020-01-16T00:13:32.836 INFO:tasks.rados.rados.0.smithi123.stdout:1297: done (12 left) 2020-01-16T00:13:32.836 INFO:tasks.rados.rados.0.smithi123.stdout:1298: done (11 left) 2020-01-16T00:13:32.836 INFO:tasks.rados.rados.0.smithi123.stdout:1299: done (10 left) 2020-01-16T00:13:32.837 INFO:tasks.rados.rados.0.smithi123.stdout:1301: done (9 left) 2020-01-16T00:13:32.837 INFO:tasks.rados.rados.0.smithi123.stdout:1302: done (8 left) 2020-01-16T00:13:32.837 INFO:tasks.rados.rados.0.smithi123.stdout:1303: done (7 left) 2020-01-16T00:13:32.837 INFO:tasks.rados.rados.0.smithi123.stdout:1304: done (6 left) 2020-01-16T00:13:32.837 INFO:tasks.rados.rados.0.smithi123.stdout:1311: read oid 7 snap -1 2020-01-16T00:13:32.837 INFO:tasks.rados.rados.0.smithi123.stdout:1311: expect (ObjNum 399 snap 0 seq_num 399) 2020-01-16T00:13:32.838 INFO:tasks.rados.rados.0.smithi123.stdout:1312: read oid 12 snap -1 2020-01-16T00:13:32.842 INFO:tasks.rados.rados.0.smithi123.stdout:1312: expect (ObjNum 395 snap 0 seq_num 395) 2020-01-16T00:13:32.843 INFO:tasks.rados.rados.0.smithi123.stdout:1313: write oid 21 current snap is 0 2020-01-16T00:13:32.843 INFO:tasks.rados.rados.0.smithi123.stdout:1313: seq_num 406 ranges {423642=717803,1875101=766806} 2020-01-16T00:13:32.843 INFO:tasks.rados.rados.0.smithi123.stdout:1313: writing smithi12313934-21 from 423642 to 1141445 tid 1 2020-01-16T00:13:32.854 INFO:tasks.rados.rados.0.smithi123.stdout:1313: writing smithi12313934-21 from 1875101 to 2641907 tid 2 2020-01-16T00:13:32.854 INFO:tasks.rados.rados.0.smithi123.stdout:1314: write (excl) oid 32 current snap is 0 2020-01-16T00:13:32.854 INFO:tasks.rados.rados.0.smithi123.stdout:1314: seq_num 407 ranges {440449=793052,1933583=510464} 2020-01-16T00:13:32.858 INFO:tasks.rados.rados.0.smithi123.stdout:1314: writing smithi12313934-32 from 440449 to 1233501 tid 1 2020-01-16T00:13:32.864 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:32.860+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:32.866 INFO:tasks.rados.rados.0.smithi123.stdout:1314: writing smithi12313934-32 from 1933583 to 2444047 tid 2 2020-01-16T00:13:32.867 INFO:tasks.rados.rados.0.smithi123.stdout:1315: read oid 4 snap -1 2020-01-16T00:13:32.867 INFO:tasks.rados.rados.0.smithi123.stdout:1315: expect (ObjNum 387 snap 0 seq_num 387) 2020-01-16T00:13:32.867 INFO:tasks.rados.rados.0.smithi123.stdout:1316: read oid 27 snap -1 2020-01-16T00:13:32.867 INFO:tasks.rados.rados.0.smithi123.stdout:1316: expect (ObjNum 303 snap 0 seq_num 303) 2020-01-16T00:13:32.867 INFO:tasks.rados.rados.0.smithi123.stdout:1317: write (excl) oid 50 current snap is 0 2020-01-16T00:13:32.867 INFO:tasks.rados.rados.0.smithi123.stdout:1317: seq_num 408 ranges {457256=468301,1592065=770475,2854581=2675} 2020-01-16T00:13:32.868 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:32.872 INFO:tasks.rados.rados.0.smithi123.stdout:1317: writing smithi12313934-50 from 457256 to 925557 tid 1 2020-01-16T00:13:32.882 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-01-16T00:13:32.883 INFO:tasks.rados.rados.0.smithi123.stdout:1317: writing smithi12313934-50 from 1592065 to 2362540 tid 2 2020-01-16T00:13:32.884 INFO:tasks.rados.rados.0.smithi123.stdout:1317: writing smithi12313934-50 from 2854581 to 2857256 tid 3 2020-01-16T00:13:32.885 INFO:tasks.rados.rados.0.smithi123.stdout:1318: write (excl) oid 8 current snap is 0 2020-01-16T00:13:32.885 INFO:tasks.rados.rados.0.smithi123.stdout:1318: seq_num 409 ranges {474063=543550,1650547=514133,2765651=108412} 2020-01-16T00:13:32.890 INFO:tasks.rados.rados.0.smithi123.stdout:1318: writing smithi12313934-8 from 474063 to 1017613 tid 1 2020-01-16T00:13:32.897 INFO:tasks.rados.rados.0.smithi123.stdout:1318: writing smithi12313934-8 from 1650547 to 2164680 tid 2 2020-01-16T00:13:32.903 INFO:tasks.rados.rados.0.smithi123.stdout:1318: writing smithi12313934-8 from 2765651 to 2874063 tid 3 2020-01-16T00:13:32.903 INFO:tasks.rados.rados.0.smithi123.stdout:1319: read oid 41 snap -1 2020-01-16T00:13:32.904 INFO:tasks.rados.rados.0.smithi123.stdout:1319: expect (ObjNum 310 snap 0 seq_num 310) 2020-01-16T00:13:32.904 INFO:tasks.rados.rados.0.smithi123.stdout:1320: write oid 20 current snap is 0 2020-01-16T00:13:32.904 INFO:tasks.rados.rados.0.smithi123.stdout:1320: seq_num 410 ranges {490870=618799,1592676=774144,2793074=97796} 2020-01-16T00:13:32.908 INFO:tasks.rados.rados.0.smithi123.stdout:1320: writing smithi12313934-20 from 490870 to 1109669 tid 1 2020-01-16T00:13:32.915 INFO:tasks.rados.rados.0.smithi123.stdout:1320: writing smithi12313934-20 from 1592676 to 2366820 tid 2 2020-01-16T00:13:32.918 INFO:tasks.rados.rados.0.smithi123.stdout:1320: writing smithi12313934-20 from 2793074 to 2890870 tid 3 2020-01-16T00:13:32.918 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:32.919 INFO:tasks.rados.rados.0.smithi123.stdout:1305: expect (ObjNum 376 snap 0 seq_num 376) 2020-01-16T00:13:32.965 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:32.960+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:32.981 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:32.989 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-01-16T00:13:33.058 INFO:tasks.rados.rados.0.smithi123.stdout:1306: expect (ObjNum 362 snap 0 seq_num 362) 2020-01-16T00:13:33.063 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2020-01-16T00:13:33.066 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:33.062+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:33.129 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:33.139 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-01-16T00:13:33.167 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:33.164+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:33.240 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:33.249 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-01-16T00:13:33.254 INFO:tasks.rados.rados.0.smithi123.stdout:1307: finishing write tid 1 to smithi12313934-39 2020-01-16T00:13:33.254 INFO:tasks.rados.rados.0.smithi123.stdout:1307: finishing write tid 2 to smithi12313934-39 2020-01-16T00:13:33.254 INFO:tasks.rados.rados.0.smithi123.stdout:1305: done (15 left) 2020-01-16T00:13:33.254 INFO:tasks.rados.rados.0.smithi123.stdout:1306: done (14 left) 2020-01-16T00:13:33.255 INFO:tasks.rados.rados.0.smithi123.stdout:1321: write (excl) oid 5 current snap is 0 2020-01-16T00:13:33.255 INFO:tasks.rados.rados.0.smithi123.stdout:1321: seq_num 411 ranges {507677=410401,1367511=517802,2420497=487180} 2020-01-16T00:13:33.259 INFO:tasks.rados.rados.0.smithi123.stdout:1321: writing smithi12313934-5 from 507677 to 918078 tid 1 2020-01-16T00:13:33.267 INFO:tasks.rados.rados.0.smithi123.stdout:1321: writing smithi12313934-5 from 1367511 to 1885313 tid 2 2020-01-16T00:13:33.267 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:33.262+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:33.273 INFO:tasks.rados.rados.0.smithi123.stdout:1321: writing smithi12313934-5 from 2420497 to 2907677 tid 3 2020-01-16T00:13:33.273 INFO:tasks.rados.rados.0.smithi123.stdout:1322: write (excl) oid 48 current snap is 0 2020-01-16T00:13:33.273 INFO:tasks.rados.rados.0.smithi123.stdout:1322: seq_num 412 ranges {524484=485650,1425993=661460,2847920=76564} 2020-01-16T00:13:33.279 INFO:tasks.rados.rados.0.smithi123.stdout:1322: writing smithi12313934-48 from 524484 to 1010134 tid 1 2020-01-16T00:13:33.285 INFO:tasks.rados.rados.0.smithi123.stdout:1322: writing smithi12313934-48 from 1425993 to 2087453 tid 2 2020-01-16T00:13:33.290 INFO:tasks.rados.rados.0.smithi123.stdout:1322: writing smithi12313934-48 from 2847920 to 2924484 tid 3 2020-01-16T00:13:33.291 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:33.291 INFO:tasks.rados.rados.0.smithi123.stdout:1307: finishing write tid 4 to smithi12313934-39 2020-01-16T00:13:33.291 INFO:tasks.rados.rados.0.smithi123.stdout:1307: finishing write tid 5 to smithi12313934-39 2020-01-16T00:13:33.291 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 39 v 196 (ObjNum 403 snap 0 seq_num 403) dirty exists 2020-01-16T00:13:33.291 INFO:tasks.rados.rados.0.smithi123.stdout:1307: left oid 39 (ObjNum 403 snap 0 seq_num 403) 2020-01-16T00:13:33.291 INFO:tasks.rados.rados.0.smithi123.stdout:1308: finishing write tid 1 to smithi12313934-49 2020-01-16T00:13:33.291 INFO:tasks.rados.rados.0.smithi123.stdout:1308: finishing write tid 2 to smithi12313934-49 2020-01-16T00:13:33.292 INFO:tasks.rados.rados.0.smithi123.stdout:1308: finishing write tid 4 to smithi12313934-49 2020-01-16T00:13:33.292 INFO:tasks.rados.rados.0.smithi123.stdout:1308: finishing write tid 5 to smithi12313934-49 2020-01-16T00:13:33.292 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 49 v 132 (ObjNum 404 snap 0 seq_num 404) dirty exists 2020-01-16T00:13:33.292 INFO:tasks.rados.rados.0.smithi123.stdout:1308: left oid 49 (ObjNum 404 snap 0 seq_num 404) 2020-01-16T00:13:33.292 INFO:tasks.rados.rados.0.smithi123.stdout:1309: finishing write tid 1 to smithi12313934-10 2020-01-16T00:13:33.292 INFO:tasks.rados.rados.0.smithi123.stdout:1309: finishing write tid 2 to smithi12313934-10 2020-01-16T00:13:33.292 INFO:tasks.rados.rados.0.smithi123.stdout:1309: finishing write tid 3 to smithi12313934-10 2020-01-16T00:13:33.292 INFO:tasks.rados.rados.0.smithi123.stdout:1309: finishing write tid 5 to smithi12313934-10 2020-01-16T00:13:33.293 INFO:tasks.rados.rados.0.smithi123.stdout:1309: finishing write tid 6 to smithi12313934-10 2020-01-16T00:13:33.293 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 10 v 137 (ObjNum 405 snap 0 seq_num 405) dirty exists 2020-01-16T00:13:33.293 INFO:tasks.rados.rados.0.smithi123.stdout:1309: left oid 10 (ObjNum 405 snap 0 seq_num 405) 2020-01-16T00:13:33.293 INFO:tasks.rados.rados.0.smithi123.stdout:1307: done (15 left) 2020-01-16T00:13:33.293 INFO:tasks.rados.rados.0.smithi123.stdout:1308: done (14 left) 2020-01-16T00:13:33.293 INFO:tasks.rados.rados.0.smithi123.stdout:1309: done (13 left) 2020-01-16T00:13:33.293 INFO:tasks.rados.rados.0.smithi123.stdout:1310: done (12 left) 2020-01-16T00:13:33.293 INFO:tasks.rados.rados.0.smithi123.stdout:1323: write (excl) oid 9 current snap is 0 2020-01-16T00:13:33.294 INFO:tasks.rados.rados.0.smithi123.stdout:1323: seq_num 413 ranges {541291=560899,1884475=521471} 2020-01-16T00:13:33.297 INFO:tasks.rados.rados.0.smithi123.stdout:1323: writing smithi12313934-9 from 541291 to 1102190 tid 1 2020-01-16T00:13:33.305 INFO:tasks.rados.rados.0.smithi123.stdout:1323: writing smithi12313934-9 from 1884475 to 2405946 tid 2 2020-01-16T00:13:33.305 INFO:tasks.rados.rados.0.smithi123.stdout:1324: read oid 1 snap -1 2020-01-16T00:13:33.305 INFO:tasks.rados.rados.0.smithi123.stdout:1324: expect (ObjNum 326 snap 0 seq_num 326) 2020-01-16T00:13:33.305 INFO:tasks.rados.rados.0.smithi123.stdout:1325: read oid 18 snap -1 2020-01-16T00:13:33.305 INFO:tasks.rados.rados.0.smithi123.stdout:1325: expect (ObjNum 220 snap 0 seq_num 220) 2020-01-16T00:13:33.306 INFO:tasks.rados.rados.0.smithi123.stdout:1326: write (excl) oid 29 current snap is 0 2020-01-16T00:13:33.306 INFO:tasks.rados.rados.0.smithi123.stdout:1326: seq_num 414 ranges {558098=636148,1826604=665129} 2020-01-16T00:13:33.312 INFO:tasks.rados.rados.0.smithi123.stdout:1326: writing smithi12313934-29 from 558098 to 1194246 tid 1 2020-01-16T00:13:33.320 INFO:tasks.rados.rados.0.smithi123.stdout:1326: writing smithi12313934-29 from 1826604 to 2491733 tid 2 2020-01-16T00:13:33.320 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:33.320 INFO:tasks.rados.rados.0.smithi123.stdout:1311: expect (ObjNum 399 snap 0 seq_num 399) 2020-01-16T00:13:33.363 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:33.368 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:33.366+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:33.371 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-01-16T00:13:33.469 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:33.466+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:33.517 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:33.526 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-01-16T00:13:33.570 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:33.566+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:33.575 INFO:tasks.rados.rados.0.smithi123.stdout:1312: expect (ObjNum 395 snap 0 seq_num 395) 2020-01-16T00:13:33.669 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-01-16T00:13:33.677 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:33.678 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:33.668+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:33.684 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-01-16T00:13:33.772 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:33.768+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:33.779 INFO:tasks.rados.rados.0.smithi123.stdout:1313: finishing write tid 1 to smithi12313934-21 2020-01-16T00:13:33.779 INFO:tasks.rados.rados.0.smithi123.stdout:1313: finishing write tid 2 to smithi12313934-21 2020-01-16T00:13:33.779 INFO:tasks.rados.rados.0.smithi123.stdout:1313: finishing write tid 4 to smithi12313934-21 2020-01-16T00:13:33.779 INFO:tasks.rados.rados.0.smithi123.stdout:1313: finishing write tid 5 to smithi12313934-21 2020-01-16T00:13:33.779 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 21 v 53 (ObjNum 406 snap 0 seq_num 406) dirty exists 2020-01-16T00:13:33.779 INFO:tasks.rados.rados.0.smithi123.stdout:1313: left oid 21 (ObjNum 406 snap 0 seq_num 406) 2020-01-16T00:13:33.780 INFO:tasks.rados.rados.0.smithi123.stdout:1314: finishing write tid 1 to smithi12313934-32 2020-01-16T00:13:33.780 INFO:tasks.rados.rados.0.smithi123.stdout:1311: done (15 left) 2020-01-16T00:13:33.780 INFO:tasks.rados.rados.0.smithi123.stdout:1312: done (14 left) 2020-01-16T00:13:33.780 INFO:tasks.rados.rados.0.smithi123.stdout:1313: done (13 left) 2020-01-16T00:13:33.780 INFO:tasks.rados.rados.0.smithi123.stdout:1327: read oid 19 snap -1 2020-01-16T00:13:33.780 INFO:tasks.rados.rados.0.smithi123.stdout:1327: expect (ObjNum 342 snap 0 seq_num 342) 2020-01-16T00:13:33.780 INFO:tasks.rados.rados.0.smithi123.stdout:1328: write (excl) oid 45 current snap is 0 2020-01-16T00:13:33.780 INFO:tasks.rados.rados.0.smithi123.stdout:1328: seq_num 415 ranges {574905=711397,1885086=525140,2930189=44716} 2020-01-16T00:13:33.786 INFO:tasks.rados.rados.0.smithi123.stdout:1328: writing smithi12313934-45 from 574905 to 1286302 tid 1 2020-01-16T00:13:33.792 INFO:tasks.rados.rados.0.smithi123.stdout:1328: writing smithi12313934-45 from 1885086 to 2410226 tid 2 2020-01-16T00:13:33.796 INFO:tasks.rados.rados.0.smithi123.stdout:1328: writing smithi12313934-45 from 2930189 to 2974905 tid 3 2020-01-16T00:13:33.797 INFO:tasks.rados.rados.0.smithi123.stdout:1329: read oid 10 snap -1 2020-01-16T00:13:33.797 INFO:tasks.rados.rados.0.smithi123.stdout:1329: expect (ObjNum 405 snap 0 seq_num 405) 2020-01-16T00:13:33.797 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:33.797 INFO:tasks.rados.rados.0.smithi123.stdout:1315: expect (ObjNum 387 snap 0 seq_num 387) 2020-01-16T00:13:33.814 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:13:33.848 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:13:33.862 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:33.871 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-01-16T00:13:33.872 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:33.868+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:33.877 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:13:33.905 INFO:tasks.rados.rados.0.smithi123.stdout:1314: finishing write tid 2 to smithi12313934-32 2020-01-16T00:13:33.906 INFO:tasks.rados.rados.0.smithi123.stdout:1314: finishing write tid 4 to smithi12313934-32 2020-01-16T00:13:33.906 INFO:tasks.rados.rados.0.smithi123.stdout:1314: finishing write tid 5 to smithi12313934-32 2020-01-16T00:13:33.906 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 32 v 130 (ObjNum 407 snap 0 seq_num 407) dirty exists 2020-01-16T00:13:33.906 INFO:tasks.rados.rados.0.smithi123.stdout:1314: left oid 32 (ObjNum 407 snap 0 seq_num 407) 2020-01-16T00:13:33.906 INFO:tasks.rados.rados.0.smithi123.stdout:1316: expect (ObjNum 303 snap 0 seq_num 303) 2020-01-16T00:13:33.909 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:13:33.935 INFO:teuthology.orchestra.run.smithi123.stderr:nodeep-scrub is set 2020-01-16T00:13:33.951 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:13:33.980 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:13:33.981 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:33.978+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:34.005 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:34.005 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:13:34.005 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:13:34.006 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:13:34.006 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.006 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:922 3.8 3:113947f2:::smithi12313934-12:head [write 443718~615603,stat] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:34.006 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.863202+0000", 2020-01-16T00:13:34.006 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 56.137586124000002, 2020-01-16T00:13:34.006 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017152697000000001, 2020-01-16T00:13:34.006 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.007 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:34.007 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:34.007 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:34.007 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:34.007 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 922 2020-01-16T00:13:34.007 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.007 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.008 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.008 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.008 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863202+0000", 2020-01-16T00:13:34.008 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.008 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.008 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.008 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.008 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863202+0000", 2020-01-16T00:13:34.009 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:34.009 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.009 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.009 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.009 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863200+0000", 2020-01-16T00:13:34.009 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00038569199999999998 2020-01-16T00:13:34.009 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.010 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.010 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.010 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863585+0000", 2020-01-16T00:13:34.010 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5685e-05 2020-01-16T00:13:34.010 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.010 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.010 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.010 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863601+0000", 2020-01-16T00:13:34.011 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.4734000000000001e-05 2020-01-16T00:13:34.011 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.011 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.011 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.011 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863626+0000", 2020-01-16T00:13:34.011 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1037e-05 2020-01-16T00:13:34.011 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.011 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.012 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.012 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863667+0000", 2020-01-16T00:13:34.012 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016041499999999999 2020-01-16T00:13:34.012 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.012 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.012 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:34.012 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.863827+0000", 2020-01-16T00:13:34.012 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0094286430000000004 2020-01-16T00:13:34.013 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.013 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.013 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.013 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.873256+0000", 2020-01-16T00:13:34.013 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2867999999999994e-05 2020-01-16T00:13:34.013 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.013 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.014 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:34.014 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.873329+0000", 2020-01-16T00:13:34.014 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021174789999999998 2020-01-16T00:13:34.015 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.015 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.015 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.015 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.875446+0000", 2020-01-16T00:13:34.015 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011244200000000001 2020-01-16T00:13:34.015 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.015 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.016 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.016 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.875559+0000", 2020-01-16T00:13:34.016 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021772600000000001 2020-01-16T00:13:34.016 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.016 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.016 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1,7", 2020-01-16T00:13:34.016 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.875776+0000", 2020-01-16T00:13:34.016 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0028978860000000001 2020-01-16T00:13:34.017 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.017 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.017 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.017 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.878674+0000", 2020-01-16T00:13:34.017 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.324e-05 2020-01-16T00:13:34.017 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.017 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.018 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:34.018 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.878687+0000", 2020-01-16T00:13:34.018 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0014896180000000001 2020-01-16T00:13:34.018 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.018 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.018 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.018 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880177+0000", 2020-01-16T00:13:34.018 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6881000000000001e-05 2020-01-16T00:13:34.019 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.019 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.019 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:34.019 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880224+0000", 2020-01-16T00:13:34.019 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013096700000000001 2020-01-16T00:13:34.019 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.019 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.019 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.020 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880355+0000", 2020-01-16T00:13:34.020 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017152697000000001 2020-01-16T00:13:34.020 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.020 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.020 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.020 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.020 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.020 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:923 3.8 3:113947f2:::smithi12313934-12:head [write 1564138~470341] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:34.021 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.869292+0000", 2020-01-16T00:13:34.021 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 56.131495794000003, 2020-01-16T00:13:34.021 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017661620999999999, 2020-01-16T00:13:34.021 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.021 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:34.021 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:34.021 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:34.022 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:34.022 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 923 2020-01-16T00:13:34.022 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.022 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.022 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.022 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.022 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869292+0000", 2020-01-16T00:13:34.022 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.023 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.023 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.023 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.023 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869292+0000", 2020-01-16T00:13:34.023 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:34.023 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.023 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.023 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.024 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869291+0000", 2020-01-16T00:13:34.024 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00049842899999999999 2020-01-16T00:13:34.024 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.024 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.024 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.024 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869789+0000", 2020-01-16T00:13:34.024 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1014999999999999e-05 2020-01-16T00:13:34.024 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.025 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.025 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.025 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869830+0000", 2020-01-16T00:13:34.025 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7880000000000007e-06 2020-01-16T00:13:34.025 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.025 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.025 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.026 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869838+0000", 2020-01-16T00:13:34.026 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8346999999999997e-05 2020-01-16T00:13:34.026 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.026 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.026 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.026 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869876+0000", 2020-01-16T00:13:34.026 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9726999999999999e-05 2020-01-16T00:13:34.026 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.027 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.027 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:34.027 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869936+0000", 2020-01-16T00:13:34.027 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003434917 2020-01-16T00:13:34.027 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.027 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.027 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.027 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.873371+0000", 2020-01-16T00:13:34.028 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118808 2020-01-16T00:13:34.028 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.028 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.028 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:34.028 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.873490+0000", 2020-01-16T00:13:34.028 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036712910000000001 2020-01-16T00:13:34.028 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.028 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.029 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.029 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877161+0000", 2020-01-16T00:13:34.029 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9909999999999999e-05 2020-01-16T00:13:34.029 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.029 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.029 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:34.029 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877241+0000", 2020-01-16T00:13:34.029 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031619970000000002 2020-01-16T00:13:34.030 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.030 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.030 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.030 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880403+0000", 2020-01-16T00:13:34.030 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013071099999999999 2020-01-16T00:13:34.030 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.030 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.030 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.031 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880534+0000", 2020-01-16T00:13:34.031 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00032823500000000002 2020-01-16T00:13:34.031 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.031 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.031 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1,7", 2020-01-16T00:13:34.031 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880862+0000", 2020-01-16T00:13:34.031 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00513862 2020-01-16T00:13:34.032 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.032 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.032 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:34.032 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886001+0000", 2020-01-16T00:13:34.032 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8869000000000001e-05 2020-01-16T00:13:34.032 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.032 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.032 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.033 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886030+0000", 2020-01-16T00:13:34.033 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00081600200000000003 2020-01-16T00:13:34.033 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.033 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.033 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.033 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886846+0000", 2020-01-16T00:13:34.033 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4095999999999998e-05 2020-01-16T00:13:34.033 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.034 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.034 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:34.034 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886900+0000", 2020-01-16T00:13:34.034 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.448e-05 2020-01-16T00:13:34.034 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.034 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.034 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.035 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886954+0000", 2020-01-16T00:13:34.035 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017661620999999999 2020-01-16T00:13:34.035 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.035 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.035 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.035 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.035 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.035 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2226 3.7 3:e1be6d3b:::smithi12313934-24:head [setxattr _header (56) in=7b,truncate 2100908] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:13:34.036 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.980217+0000", 2020-01-16T00:13:34.036 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 28.020571129, 2020-01-16T00:13:34.036 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023113018999999999, 2020-01-16T00:13:34.036 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.036 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:34.036 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:34.036 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:34.036 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:34.037 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2226 2020-01-16T00:13:34.037 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.037 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.037 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.037 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.037 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980217+0000", 2020-01-16T00:13:34.037 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.038 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.038 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.038 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.038 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980217+0000", 2020-01-16T00:13:34.038 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:34.038 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.038 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.038 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.039 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980216+0000", 2020-01-16T00:13:34.039 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5049999999999998e-06 2020-01-16T00:13:34.039 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.039 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.039 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.039 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980222+0000", 2020-01-16T00:13:34.039 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5654999999999998e-05 2020-01-16T00:13:34.039 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.040 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.040 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.040 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980237+0000", 2020-01-16T00:13:34.040 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9740000000000004e-06 2020-01-16T00:13:34.040 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.040 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.040 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.041 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980241+0000", 2020-01-16T00:13:34.041 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002947776 2020-01-16T00:13:34.041 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.048 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.048 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.048 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.983189+0000", 2020-01-16T00:13:34.048 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0027099400000000001 2020-01-16T00:13:34.048 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.048 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.048 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:34.049 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985899+0000", 2020-01-16T00:13:34.049 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014275589999999999 2020-01-16T00:13:34.049 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.049 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.049 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.049 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.000175+0000", 2020-01-16T00:13:34.049 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011211 2020-01-16T00:13:34.050 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.050 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.050 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.050 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.000287+0000", 2020-01-16T00:13:34.050 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000229886 2020-01-16T00:13:34.050 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.050 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.050 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,7", 2020-01-16T00:13:34.051 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.000517+0000", 2020-01-16T00:13:34.051 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00095715399999999999 2020-01-16T00:13:34.051 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.051 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.051 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:34.051 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.001474+0000", 2020-01-16T00:13:34.052 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001341512 2020-01-16T00:13:34.052 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.052 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.052 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.052 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.002815+0000", 2020-01-16T00:13:34.052 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035789099999999999 2020-01-16T00:13:34.052 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.052 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.053 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.053 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003173+0000", 2020-01-16T00:13:34.053 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8571000000000005e-05 2020-01-16T00:13:34.053 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.053 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.053 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:34.053 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003242+0000", 2020-01-16T00:13:34.054 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8293000000000003e-05 2020-01-16T00:13:34.054 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.054 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.054 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.054 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003330+0000", 2020-01-16T00:13:34.054 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023113018999999999 2020-01-16T00:13:34.054 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.054 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.055 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.055 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.055 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.055 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2227 3.7 3:e1be6d3b:::smithi12313934-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e35)", 2020-01-16T00:13:34.055 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:13:34.055 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 28.020523875999999, 2020-01-16T00:13:34.055 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023303356000000001, 2020-01-16T00:13:34.055 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.056 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:34.056 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:34.056 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:34.056 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:34.056 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2227 2020-01-16T00:13:34.056 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.056 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.056 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.056 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.057 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:13:34.057 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.057 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.057 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.057 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.057 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:13:34.057 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:34.058 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.058 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.058 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.058 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:13:34.058 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3079999999999999e-06 2020-01-16T00:13:34.058 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.058 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.058 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.059 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980267+0000", 2020-01-16T00:13:34.059 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9290000000000001e-06 2020-01-16T00:13:34.059 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.059 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.059 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.059 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980274+0000", 2020-01-16T00:13:34.059 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7400000000000002e-06 2020-01-16T00:13:34.059 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.060 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.060 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.060 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980278+0000", 2020-01-16T00:13:34.060 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056624300000000004 2020-01-16T00:13:34.060 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.060 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.060 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.060 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985940+0000", 2020-01-16T00:13:34.061 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4261000000000002e-05 2020-01-16T00:13:34.061 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.061 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.061 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:34.061 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985985+0000", 2020-01-16T00:13:34.061 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015087676 2020-01-16T00:13:34.061 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.061 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.062 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.062 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.001072+0000", 2020-01-16T00:13:34.062 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011960600000000001 2020-01-16T00:13:34.062 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.062 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.062 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:34.062 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.001192+0000", 2020-01-16T00:13:34.063 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021718309999999999 2020-01-16T00:13:34.063 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.063 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.063 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.063 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003364+0000", 2020-01-16T00:13:34.063 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000102855 2020-01-16T00:13:34.063 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.063 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.063 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.064 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003467+0000", 2020-01-16T00:13:34.064 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010122499999999999 2020-01-16T00:13:34.064 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.064 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.064 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.064 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003568+0000", 2020-01-16T00:13:34.064 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023303356000000001 2020-01-16T00:13:34.064 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.065 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.065 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.065 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.065 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.065 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2233 3.2 3:4f8fe446:::smithi12313934-48:head [setxattr _header (56) in=7b,truncate 2117715] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:13:34.065 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.993103+0000", 2020-01-16T00:13:34.065 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 28.007685653999999, 2020-01-16T00:13:34.066 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017951763999999999, 2020-01-16T00:13:34.066 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.066 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:34.066 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:34.066 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:34.066 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:34.066 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2233 2020-01-16T00:13:34.067 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.067 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.067 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.067 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.067 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993103+0000", 2020-01-16T00:13:34.067 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.067 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.068 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.068 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.068 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993103+0000", 2020-01-16T00:13:34.068 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:34.068 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.068 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.068 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.068 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993102+0000", 2020-01-16T00:13:34.069 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9369999999999993e-06 2020-01-16T00:13:34.069 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.069 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.069 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.069 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993111+0000", 2020-01-16T00:13:34.069 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3340000000000002e-06 2020-01-16T00:13:34.069 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.070 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.070 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.070 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993119+0000", 2020-01-16T00:13:34.070 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1769999999999997e-06 2020-01-16T00:13:34.071 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.071 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.071 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.071 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993124+0000", 2020-01-16T00:13:34.071 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001601304 2020-01-16T00:13:34.071 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.071 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.071 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.072 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.994725+0000", 2020-01-16T00:13:34.072 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048777250000000003 2020-01-16T00:13:34.072 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.072 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.072 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:34.072 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999603+0000", 2020-01-16T00:13:34.072 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0099793880000000005 2020-01-16T00:13:34.073 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.073 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.073 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.073 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.009583+0000", 2020-01-16T00:13:34.073 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000108207 2020-01-16T00:13:34.073 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.073 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.073 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.074 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.009691+0000", 2020-01-16T00:13:34.074 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012862999999999999 2020-01-16T00:13:34.074 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.074 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.074 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:34.074 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.009819+0000", 2020-01-16T00:13:34.074 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00077947599999999998 2020-01-16T00:13:34.074 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.075 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.075 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:34.075 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010599+0000", 2020-01-16T00:13:34.075 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000221282 2020-01-16T00:13:34.075 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.075 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.075 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.075 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010820+0000", 2020-01-16T00:13:34.076 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7021999999999995e-05 2020-01-16T00:13:34.076 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.076 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.076 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.076 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010907+0000", 2020-01-16T00:13:34.076 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9815999999999997e-05 2020-01-16T00:13:34.076 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.076 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.077 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:34.077 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010977+0000", 2020-01-16T00:13:34.077 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7453999999999994e-05 2020-01-16T00:13:34.077 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.077 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.077 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.077 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.011054+0000", 2020-01-16T00:13:34.077 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017951763999999999 2020-01-16T00:13:34.078 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.078 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.078 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.078 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.078 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.078 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2234 3.2 3:4f8fe446:::smithi12313934-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e35)", 2020-01-16T00:13:34.078 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.993135+0000", 2020-01-16T00:13:34.078 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 28.007653201, 2020-01-16T00:13:34.079 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018067916999999999, 2020-01-16T00:13:34.079 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.079 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:34.079 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:34.079 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:34.079 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:34.079 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2234 2020-01-16T00:13:34.080 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.080 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.080 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.080 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.080 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993135+0000", 2020-01-16T00:13:34.080 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.080 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.080 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.081 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.081 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993135+0000", 2020-01-16T00:13:34.081 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:34.081 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.081 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.081 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.081 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993134+0000", 2020-01-16T00:13:34.081 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5630000000000002e-06 2020-01-16T00:13:34.082 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.082 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.082 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.082 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993139+0000", 2020-01-16T00:13:34.082 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1730000000000003e-06 2020-01-16T00:13:34.082 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.082 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.082 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.082 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993144+0000", 2020-01-16T00:13:34.083 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.937e-06 2020-01-16T00:13:34.083 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.083 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.083 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.083 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993147+0000", 2020-01-16T00:13:34.083 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0065164000000000003 2020-01-16T00:13:34.083 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.083 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.084 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.084 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999663+0000", 2020-01-16T00:13:34.084 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6215000000000002e-05 2020-01-16T00:13:34.084 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.084 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.084 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:34.084 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999729+0000", 2020-01-16T00:13:34.084 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010475887999999999 2020-01-16T00:13:34.085 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.085 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.085 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.085 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010205+0000", 2020-01-16T00:13:34.085 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119471 2020-01-16T00:13:34.085 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.085 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.085 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:34.086 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010325+0000", 2020-01-16T00:13:34.086 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00075498799999999995 2020-01-16T00:13:34.086 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.086 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.086 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.086 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.011080+0000", 2020-01-16T00:13:34.086 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7904e-05 2020-01-16T00:13:34.086 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.087 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.087 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.087 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.011138+0000", 2020-01-16T00:13:34.087 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5273999999999999e-05 2020-01-16T00:13:34.087 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.087 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.087 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.088 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.011203+0000", 2020-01-16T00:13:34.088 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018067916999999999 2020-01-16T00:13:34.088 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.088 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.088 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.088 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.088 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.088 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2324 3.7 3:e1be6d3b:::smithi12313934-24:head [setxattr _header (56) in=7b,truncate 2336206] snapc 0=[] ondisk+write+known_if_redirected e36)", 2020-01-16T00:13:34.089 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.550846+0000", 2020-01-16T00:13:34.089 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 26.449942615000001, 2020-01-16T00:13:34.089 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018364742, 2020-01-16T00:13:34.089 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.089 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:34.089 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:34.089 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:34.090 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:34.090 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2324 2020-01-16T00:13:34.090 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.090 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.090 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.090 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.090 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550846+0000", 2020-01-16T00:13:34.090 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.090 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.091 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.091 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.091 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550846+0000", 2020-01-16T00:13:34.091 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:34.091 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.091 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.091 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.092 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550845+0000", 2020-01-16T00:13:34.092 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1829999999999999e-06 2020-01-16T00:13:34.092 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.092 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.092 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.092 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550851+0000", 2020-01-16T00:13:34.092 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1925e-05 2020-01-16T00:13:34.092 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.093 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.093 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.093 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550863+0000", 2020-01-16T00:13:34.093 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1139999999999997e-06 2020-01-16T00:13:34.093 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.093 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.093 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.093 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550868+0000", 2020-01-16T00:13:34.094 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001388423 2020-01-16T00:13:34.094 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.094 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.094 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.094 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.552256+0000", 2020-01-16T00:13:34.094 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0034246540000000001 2020-01-16T00:13:34.094 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.094 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.095 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:34.095 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555681+0000", 2020-01-16T00:13:34.095 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011882762999999999 2020-01-16T00:13:34.095 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.095 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.095 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.095 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.567564+0000", 2020-01-16T00:13:34.095 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015010800000000001 2020-01-16T00:13:34.096 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.096 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.096 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.096 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.567714+0000", 2020-01-16T00:13:34.096 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011527999999999999 2020-01-16T00:13:34.096 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.096 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.096 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,7", 2020-01-16T00:13:34.097 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.567829+0000", 2020-01-16T00:13:34.097 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00039888499999999997 2020-01-16T00:13:34.097 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.097 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.097 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:34.097 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568228+0000", 2020-01-16T00:13:34.097 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00054254599999999998 2020-01-16T00:13:34.098 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.098 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.098 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.098 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568771+0000", 2020-01-16T00:13:34.098 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035206400000000002 2020-01-16T00:13:34.098 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.099 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.099 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.099 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569123+0000", 2020-01-16T00:13:34.099 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1262e-05 2020-01-16T00:13:34.099 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.099 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.099 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:34.099 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569154+0000", 2020-01-16T00:13:34.100 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6499000000000002e-05 2020-01-16T00:13:34.100 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.100 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.100 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.100 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569210+0000", 2020-01-16T00:13:34.101 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018364742 2020-01-16T00:13:34.101 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.101 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.101 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.101 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.101 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.101 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2325 3.7 3:e1be6d3b:::smithi12313934-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e36)", 2020-01-16T00:13:34.101 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.550877+0000", 2020-01-16T00:13:34.102 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 26.449911548999999, 2020-01-16T00:13:34.102 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018565154, 2020-01-16T00:13:34.102 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.102 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:34.102 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:34.102 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:34.102 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:34.102 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2325 2020-01-16T00:13:34.103 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.103 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.103 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.103 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.103 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550877+0000", 2020-01-16T00:13:34.103 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.103 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.103 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.104 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.104 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550877+0000", 2020-01-16T00:13:34.104 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:34.104 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.104 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.104 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.104 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550876+0000", 2020-01-16T00:13:34.104 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1080000000000003e-06 2020-01-16T00:13:34.105 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.105 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.105 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.105 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550880+0000", 2020-01-16T00:13:34.105 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3269999999999997e-06 2020-01-16T00:13:34.105 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.105 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.106 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.106 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550886+0000", 2020-01-16T00:13:34.106 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.386e-06 2020-01-16T00:13:34.106 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.106 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.106 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.106 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550891+0000", 2020-01-16T00:13:34.106 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048231410000000004 2020-01-16T00:13:34.107 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.107 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.107 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.107 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555714+0000", 2020-01-16T00:13:34.107 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0071999999999999e-05 2020-01-16T00:13:34.107 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.107 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.107 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:34.108 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555744+0000", 2020-01-16T00:13:34.108 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012329005000000001 2020-01-16T00:13:34.108 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.108 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.108 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.108 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568073+0000", 2020-01-16T00:13:34.108 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9087999999999997e-05 2020-01-16T00:13:34.108 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.108 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.109 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:34.109 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568172+0000", 2020-01-16T00:13:34.109 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010718220000000001 2020-01-16T00:13:34.109 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.109 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.109 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.109 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569244+0000", 2020-01-16T00:13:34.109 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8199e-05 2020-01-16T00:13:34.110 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.110 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.110 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.110 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569342+0000", 2020-01-16T00:13:34.110 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9708000000000001e-05 2020-01-16T00:13:34.110 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.110 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.110 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.111 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569442+0000", 2020-01-16T00:13:34.111 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018565154 2020-01-16T00:13:34.111 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.111 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.111 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.111 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.111 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.111 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2559 3.2 3:4ba19497:::smithi12313934-38:head [write 2950640~57846] snapc 0=[] ondisk+write+known_if_redirected e37)", 2020-01-16T00:13:34.112 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.591485+0000", 2020-01-16T00:13:34.112 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 21.40930337, 2020-01-16T00:13:34.112 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016668697999999999, 2020-01-16T00:13:34.112 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.112 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:34.112 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:34.112 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:34.112 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:34.113 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2559 2020-01-16T00:13:34.113 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.113 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.113 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.113 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.113 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591485+0000", 2020-01-16T00:13:34.113 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.114 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.114 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.114 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.114 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591485+0000", 2020-01-16T00:13:34.114 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:34.114 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.114 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.114 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.115 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591483+0000", 2020-01-16T00:13:34.115 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4780999999999998e-05 2020-01-16T00:13:34.115 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.115 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.115 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.115 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591518+0000", 2020-01-16T00:13:34.115 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9706000000000001e-05 2020-01-16T00:13:34.115 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.116 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.116 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.116 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591537+0000", 2020-01-16T00:13:34.116 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1940000000000001e-06 2020-01-16T00:13:34.116 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.116 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.116 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.116 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591545+0000", 2020-01-16T00:13:34.117 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012233400000000001 2020-01-16T00:13:34.117 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.117 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.117 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.117 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591667+0000", 2020-01-16T00:13:34.117 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0050268719999999999 2020-01-16T00:13:34.117 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.118 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.118 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:34.118 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.596694+0000", 2020-01-16T00:13:34.118 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0090233669999999992 2020-01-16T00:13:34.118 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.118 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.118 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.118 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.605717+0000", 2020-01-16T00:13:34.119 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011849399999999999 2020-01-16T00:13:34.119 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.119 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.119 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.119 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.605836+0000", 2020-01-16T00:13:34.119 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022822200000000001 2020-01-16T00:13:34.119 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.119 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.119 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:34.120 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.606064+0000", 2020-01-16T00:13:34.120 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001265555 2020-01-16T00:13:34.120 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.120 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.120 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:34.120 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.607329+0000", 2020-01-16T00:13:34.120 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00052231100000000002 2020-01-16T00:13:34.120 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.121 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.121 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.121 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.607852+0000", 2020-01-16T00:13:34.121 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018900500000000001 2020-01-16T00:13:34.121 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.121 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.121 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.122 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.608041+0000", 2020-01-16T00:13:34.122 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7380000000000003e-05 2020-01-16T00:13:34.122 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.122 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.122 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:34.122 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.608098+0000", 2020-01-16T00:13:34.122 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5467999999999998e-05 2020-01-16T00:13:34.122 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.123 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.123 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.123 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.608154+0000", 2020-01-16T00:13:34.123 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016668697999999999 2020-01-16T00:13:34.123 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.123 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.123 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.123 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.124 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.124 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2560 3.2 3:4ba19497:::smithi12313934-38:head [setxattr _header (56) in=7b,truncate 3008486] snapc 0=[] ondisk+write+known_if_redirected e37)", 2020-01-16T00:13:34.124 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.591556+0000", 2020-01-16T00:13:34.124 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 21.409231905999999, 2020-01-16T00:13:34.124 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016869716999999999, 2020-01-16T00:13:34.124 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.124 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:34.124 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:34.125 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:34.125 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:34.125 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2560 2020-01-16T00:13:34.125 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.125 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.125 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.125 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.125 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591556+0000", 2020-01-16T00:13:34.126 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.126 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.126 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.126 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.126 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591556+0000", 2020-01-16T00:13:34.126 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:34.126 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.127 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.127 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.127 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591555+0000", 2020-01-16T00:13:34.127 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.93e-06 2020-01-16T00:13:34.127 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.127 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.127 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.127 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591561+0000", 2020-01-16T00:13:34.128 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2089999999999999e-06 2020-01-16T00:13:34.128 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.128 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.128 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.128 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591569+0000", 2020-01-16T00:13:34.128 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5530000000000003e-06 2020-01-16T00:13:34.128 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.128 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.129 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.129 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591573+0000", 2020-01-16T00:13:34.129 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0053636630000000003 2020-01-16T00:13:34.129 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.129 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.129 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.129 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.596937+0000", 2020-01-16T00:13:34.129 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8693000000000001e-05 2020-01-16T00:13:34.130 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.130 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.130 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:34.130 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.597006+0000", 2020-01-16T00:13:34.130 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095244160000000008 2020-01-16T00:13:34.130 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.130 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.130 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.131 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.606530+0000", 2020-01-16T00:13:34.131 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012407100000000001 2020-01-16T00:13:34.131 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.131 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.131 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.131 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.606654+0000", 2020-01-16T00:13:34.131 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000189285 2020-01-16T00:13:34.131 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.132 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.132 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:34.132 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.606843+0000", 2020-01-16T00:13:34.132 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013331490000000001 2020-01-16T00:13:34.132 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.132 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.132 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:34.132 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.608176+0000", 2020-01-16T00:13:34.133 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7118000000000003e-05 2020-01-16T00:13:34.133 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.133 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.133 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.133 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.608254+0000", 2020-01-16T00:13:34.133 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3964e-05 2020-01-16T00:13:34.133 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.133 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.134 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.134 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.608308+0000", 2020-01-16T00:13:34.134 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7191999999999997e-05 2020-01-16T00:13:34.134 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.134 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.134 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:34.134 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.608365+0000", 2020-01-16T00:13:34.134 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1375000000000001e-05 2020-01-16T00:13:34.135 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.135 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.135 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.135 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.608426+0000", 2020-01-16T00:13:34.135 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016869716999999999 2020-01-16T00:13:34.135 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.135 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.135 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.136 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.136 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.136 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2561 3.2 3:4ba19497:::smithi12313934-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e37)", 2020-01-16T00:13:34.136 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.591578+0000", 2020-01-16T00:13:34.136 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 21.409210155, 2020-01-16T00:13:34.136 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017082362, 2020-01-16T00:13:34.136 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.136 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:34.136 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:34.137 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:34.137 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:34.137 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2561 2020-01-16T00:13:34.137 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.137 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.137 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.137 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.137 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591578+0000", 2020-01-16T00:13:34.138 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.138 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.138 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.138 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.138 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591578+0000", 2020-01-16T00:13:34.138 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:34.138 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.139 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.139 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.139 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591578+0000", 2020-01-16T00:13:34.139 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.428e-06 2020-01-16T00:13:34.139 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.139 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.139 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.140 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591581+0000", 2020-01-16T00:13:34.140 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.818e-06 2020-01-16T00:13:34.140 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.140 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.140 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.140 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591587+0000", 2020-01-16T00:13:34.140 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9890000000000003e-06 2020-01-16T00:13:34.140 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.141 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.141 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.141 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591591+0000", 2020-01-16T00:13:34.141 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.005441909 2020-01-16T00:13:34.141 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.141 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.141 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.141 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.597033+0000", 2020-01-16T00:13:34.142 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1651999999999999e-05 2020-01-16T00:13:34.142 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.142 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.142 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:34.142 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.597084+0000", 2020-01-16T00:13:34.142 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010094819999999999 2020-01-16T00:13:34.142 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.142 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.143 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.143 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.607179+0000", 2020-01-16T00:13:34.143 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012867799999999999 2020-01-16T00:13:34.143 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.143 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.143 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:34.143 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.607308+0000", 2020-01-16T00:13:34.143 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001148975 2020-01-16T00:13:34.144 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.144 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.144 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.144 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.608457+0000", 2020-01-16T00:13:34.144 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000101877 2020-01-16T00:13:34.144 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.144 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.144 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.145 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.608559+0000", 2020-01-16T00:13:34.145 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000101845 2020-01-16T00:13:34.145 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.145 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.145 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.145 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.608660+0000", 2020-01-16T00:13:34.145 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017082362 2020-01-16T00:13:34.145 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.146 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.146 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.146 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.146 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.146 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2579 3.2 3:4f8fe446:::smithi12313934-48:head [setxattr _header (56) in=7b,truncate 3075714] snapc 0=[] ondisk+write+known_if_redirected e37)", 2020-01-16T00:13:34.146 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.664831+0000", 2020-01-16T00:13:34.146 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 21.335957739000001, 2020-01-16T00:13:34.146 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016678353999999999, 2020-01-16T00:13:34.147 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.147 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:34.147 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:34.147 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:34.147 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:34.147 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2579 2020-01-16T00:13:34.147 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.147 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.148 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.148 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.148 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664831+0000", 2020-01-16T00:13:34.148 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.148 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.148 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.148 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.148 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664831+0000", 2020-01-16T00:13:34.149 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:34.149 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.149 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.149 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.149 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664830+0000", 2020-01-16T00:13:34.149 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7089999999999999e-06 2020-01-16T00:13:34.149 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.149 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.150 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.150 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664835+0000", 2020-01-16T00:13:34.150 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6526e-05 2020-01-16T00:13:34.150 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.150 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.150 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.150 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664862+0000", 2020-01-16T00:13:34.151 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1869999999999996e-06 2020-01-16T00:13:34.151 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.151 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.151 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.151 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664867+0000", 2020-01-16T00:13:34.151 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00148733 2020-01-16T00:13:34.151 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.151 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.152 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.152 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.666354+0000", 2020-01-16T00:13:34.152 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3683999999999995e-05 2020-01-16T00:13:34.152 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.152 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.152 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:34.153 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.666428+0000", 2020-01-16T00:13:34.153 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013904786000000001 2020-01-16T00:13:34.153 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.153 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.153 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.154 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.680333+0000", 2020-01-16T00:13:34.154 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012004 2020-01-16T00:13:34.154 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.155 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.155 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.155 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.680453+0000", 2020-01-16T00:13:34.155 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021047700000000001 2020-01-16T00:13:34.155 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.155 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.155 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:34.156 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.680663+0000", 2020-01-16T00:13:34.156 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00070448799999999997 2020-01-16T00:13:34.156 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.156 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.156 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:34.156 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.681368+0000", 2020-01-16T00:13:34.156 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8998999999999997e-05 2020-01-16T00:13:34.156 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.157 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.157 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.157 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.681407+0000", 2020-01-16T00:13:34.157 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9014999999999999e-05 2020-01-16T00:13:34.157 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.157 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.157 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.157 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.681436+0000", 2020-01-16T00:13:34.158 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9249999999999999e-05 2020-01-16T00:13:34.158 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.158 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.158 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:34.158 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.681465+0000", 2020-01-16T00:13:34.158 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3696999999999999e-05 2020-01-16T00:13:34.158 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.158 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.159 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.159 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.681509+0000", 2020-01-16T00:13:34.159 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016678353999999999 2020-01-16T00:13:34.159 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.159 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.159 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.159 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.159 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.160 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2580 3.2 3:4f8fe446:::smithi12313934-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e37)", 2020-01-16T00:13:34.160 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.664881+0000", 2020-01-16T00:13:34.160 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 21.335906794, 2020-01-16T00:13:34.160 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016925910999999998, 2020-01-16T00:13:34.160 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.160 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:34.160 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:34.160 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:34.161 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:34.161 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2580 2020-01-16T00:13:34.161 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.161 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.161 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.161 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.161 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664881+0000", 2020-01-16T00:13:34.161 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.162 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.162 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.162 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.162 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664881+0000", 2020-01-16T00:13:34.162 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:34.162 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.162 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.163 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.163 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664881+0000", 2020-01-16T00:13:34.163 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1640000000000003e-06 2020-01-16T00:13:34.163 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.163 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.163 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.163 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664885+0000", 2020-01-16T00:13:34.163 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6469999999999998e-06 2020-01-16T00:13:34.164 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.164 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.164 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.164 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664892+0000", 2020-01-16T00:13:34.164 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8829999999999997e-06 2020-01-16T00:13:34.164 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.164 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.164 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.165 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.664897+0000", 2020-01-16T00:13:34.165 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0015624059999999999 2020-01-16T00:13:34.165 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.165 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.165 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.165 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.666460+0000", 2020-01-16T00:13:34.165 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9839000000000002e-05 2020-01-16T00:13:34.165 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.166 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.166 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:34.166 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.666510+0000", 2020-01-16T00:13:34.166 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014475200000000001 2020-01-16T00:13:34.166 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.166 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.166 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.167 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.680985+0000", 2020-01-16T00:13:34.167 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8396999999999997e-05 2020-01-16T00:13:34.167 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.167 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.167 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:34.167 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.681043+0000", 2020-01-16T00:13:34.167 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00051206799999999996 2020-01-16T00:13:34.168 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.168 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.168 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.168 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.681555+0000", 2020-01-16T00:13:34.168 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118881 2020-01-16T00:13:34.168 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.168 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.168 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.168 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.681674+0000", 2020-01-16T00:13:34.169 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000133242 2020-01-16T00:13:34.169 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.169 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.169 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.169 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.681807+0000", 2020-01-16T00:13:34.169 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016925910999999998 2020-01-16T00:13:34.169 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.170 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.170 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.170 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.170 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.170 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2800 3.2 3:42b7f20f:::smithi12313934-42:head [write 1395221~583781] snapc 0=[] ondisk+write+known_if_redirected e41)", 2020-01-16T00:13:34.170 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:17.256182+0000", 2020-01-16T00:13:34.170 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 16.744606009000002, 2020-01-16T00:13:34.170 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018509650999999998, 2020-01-16T00:13:34.171 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.171 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:34.171 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:34.171 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:34.171 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:34.171 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2800 2020-01-16T00:13:34.171 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.171 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.171 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.172 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.172 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256182+0000", 2020-01-16T00:13:34.172 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.172 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.172 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.172 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.172 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256182+0000", 2020-01-16T00:13:34.172 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:34.173 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.173 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.173 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.173 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256180+0000", 2020-01-16T00:13:34.173 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020514799999999999 2020-01-16T00:13:34.173 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.173 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.173 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.174 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256385+0000", 2020-01-16T00:13:34.174 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0288e-05 2020-01-16T00:13:34.174 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.174 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.174 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.174 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256406+0000", 2020-01-16T00:13:34.174 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7160000000000003e-06 2020-01-16T00:13:34.174 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.175 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.175 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.175 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256413+0000", 2020-01-16T00:13:34.175 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5293999999999998e-05 2020-01-16T00:13:34.175 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.175 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.175 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.175 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256449+0000", 2020-01-16T00:13:34.176 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001297737 2020-01-16T00:13:34.176 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.176 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.176 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:34.176 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.257746+0000", 2020-01-16T00:13:34.176 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0079866529999999998 2020-01-16T00:13:34.176 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.176 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.177 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.177 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.265733+0000", 2020-01-16T00:13:34.177 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4028000000000004e-05 2020-01-16T00:13:34.177 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.177 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.177 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.177 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.265797+0000", 2020-01-16T00:13:34.177 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011338899999999999 2020-01-16T00:13:34.178 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.178 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.178 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:34.178 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.265910+0000", 2020-01-16T00:13:34.178 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003372496 2020-01-16T00:13:34.178 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.178 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.178 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:34.179 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.269283+0000", 2020-01-16T00:13:34.179 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0053107959999999996 2020-01-16T00:13:34.179 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.179 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.179 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.179 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.274594+0000", 2020-01-16T00:13:34.179 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8000999999999998e-05 2020-01-16T00:13:34.179 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.180 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.180 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:34.180 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.274632+0000", 2020-01-16T00:13:34.180 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0265e-05 2020-01-16T00:13:34.180 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.180 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.180 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.180 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.274692+0000", 2020-01-16T00:13:34.181 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018509650999999998 2020-01-16T00:13:34.181 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.181 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.181 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.181 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.181 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.181 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2971 3.c 3:3025546b:::smithi12313934-15:head [write 1424565~609148] snapc 0=[] ondisk+write+known_if_redirected e42)", 2020-01-16T00:13:34.181 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:21.839711+0000", 2020-01-16T00:13:34.182 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 12.161077166, 2020-01-16T00:13:34.182 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017534376000000001, 2020-01-16T00:13:34.182 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.182 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:34.182 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:34.182 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:34.182 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:34.183 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2971 2020-01-16T00:13:34.183 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.183 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.183 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.183 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.183 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839711+0000", 2020-01-16T00:13:34.183 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.183 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.183 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.184 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.184 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839711+0000", 2020-01-16T00:13:34.184 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:34.184 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.184 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.184 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.184 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839710+0000", 2020-01-16T00:13:34.185 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017658600000000001 2020-01-16T00:13:34.185 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.185 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.185 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.185 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839886+0000", 2020-01-16T00:13:34.185 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0492000000000001e-05 2020-01-16T00:13:34.185 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.185 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.186 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.186 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839897+0000", 2020-01-16T00:13:34.186 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.019e-06 2020-01-16T00:13:34.186 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.186 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.186 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.186 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839901+0000", 2020-01-16T00:13:34.186 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5572000000000001e-05 2020-01-16T00:13:34.187 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.187 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.187 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.187 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839976+0000", 2020-01-16T00:13:34.187 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034616099999999999 2020-01-16T00:13:34.187 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.187 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.187 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:34.187 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.840323+0000", 2020-01-16T00:13:34.188 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.01246249 2020-01-16T00:13:34.188 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.188 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.188 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.188 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.852785+0000", 2020-01-16T00:13:34.188 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014474799999999999 2020-01-16T00:13:34.188 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.188 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.189 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.189 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.852930+0000", 2020-01-16T00:13:34.189 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000213907 2020-01-16T00:13:34.189 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.189 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.189 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:34.189 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.853144+0000", 2020-01-16T00:13:34.189 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0028426979999999998 2020-01-16T00:13:34.190 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.190 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.190 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.190 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.855986+0000", 2020-01-16T00:13:34.190 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010694280000000001 2020-01-16T00:13:34.190 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.190 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.190 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:34.191 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.857056+0000", 2020-01-16T00:13:34.191 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8935000000000003e-05 2020-01-16T00:13:34.191 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.191 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.191 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:34.191 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.857125+0000", 2020-01-16T00:13:34.191 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000120721 2020-01-16T00:13:34.191 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.192 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.192 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.192 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.857245+0000", 2020-01-16T00:13:34.192 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017534376000000001 2020-01-16T00:13:34.192 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.192 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.192 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.192 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.193 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.193 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2972 3.c 3:3025546b:::smithi12313934-15:head [setxattr _header (56) in=7b,truncate 2033713] snapc 0=[] ondisk+write+known_if_redirected e42)", 2020-01-16T00:13:34.193 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:21.839923+0000", 2020-01-16T00:13:34.193 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 12.160864902, 2020-01-16T00:13:34.193 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018367108, 2020-01-16T00:13:34.193 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.194 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:34.194 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:34.194 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:34.194 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:34.194 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2972 2020-01-16T00:13:34.194 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.194 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.194 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.195 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.195 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839923+0000", 2020-01-16T00:13:34.195 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.195 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.195 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.195 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.195 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839923+0000", 2020-01-16T00:13:34.196 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:34.196 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.196 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.196 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.196 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839923+0000", 2020-01-16T00:13:34.196 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0629999999999999e-06 2020-01-16T00:13:34.196 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.196 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.197 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.197 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839927+0000", 2020-01-16T00:13:34.197 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1980000000000003e-06 2020-01-16T00:13:34.197 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.197 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.197 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.197 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839931+0000", 2020-01-16T00:13:34.198 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5979999999999999e-06 2020-01-16T00:13:34.198 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.198 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.198 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.198 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839934+0000", 2020-01-16T00:13:34.198 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00042258500000000001 2020-01-16T00:13:34.198 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.199 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.199 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.199 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.840356+0000", 2020-01-16T00:13:34.199 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4919000000000002e-05 2020-01-16T00:13:34.199 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.199 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.200 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:34.200 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.840391+0000", 2020-01-16T00:13:34.200 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013783103 2020-01-16T00:13:34.200 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.200 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.200 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.200 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.854174+0000", 2020-01-16T00:13:34.200 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3455999999999995e-05 2020-01-16T00:13:34.201 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.201 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.201 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.201 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.854258+0000", 2020-01-16T00:13:34.201 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011060700000000001 2020-01-16T00:13:34.201 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.202 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.202 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:34.202 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.854368+0000", 2020-01-16T00:13:34.202 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020642500000000001 2020-01-16T00:13:34.202 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.202 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.202 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.203 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.856433+0000", 2020-01-16T00:13:34.203 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0016284820000000001 2020-01-16T00:13:34.203 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.203 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.203 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:34.203 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858061+0000", 2020-01-16T00:13:34.203 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0663999999999998e-05 2020-01-16T00:13:34.203 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.204 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.204 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:34.204 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858152+0000", 2020-01-16T00:13:34.204 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013882900000000001 2020-01-16T00:13:34.204 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.204 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.204 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.204 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858290+0000", 2020-01-16T00:13:34.205 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018367108 2020-01-16T00:13:34.205 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.205 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.205 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.205 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.205 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.206 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2973 3.c 3:3025546b:::smithi12313934-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e42)", 2020-01-16T00:13:34.206 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:21.839961+0000", 2020-01-16T00:13:34.206 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 12.160827407999999, 2020-01-16T00:13:34.207 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018691670000000001, 2020-01-16T00:13:34.207 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.207 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:34.207 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:34.207 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:34.207 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:34.208 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2973 2020-01-16T00:13:34.208 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.208 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.208 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.208 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.208 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839961+0000", 2020-01-16T00:13:34.208 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.209 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.209 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.209 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.209 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839961+0000", 2020-01-16T00:13:34.209 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:34.209 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.209 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.209 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.210 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839960+0000", 2020-01-16T00:13:34.210 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4060000000000001e-06 2020-01-16T00:13:34.210 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.210 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.210 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.210 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839964+0000", 2020-01-16T00:13:34.210 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5412000000000001e-05 2020-01-16T00:13:34.210 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.211 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.211 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.211 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839979+0000", 2020-01-16T00:13:34.211 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.444e-06 2020-01-16T00:13:34.211 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.211 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.212 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.212 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839983+0000", 2020-01-16T00:13:34.212 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000430028 2020-01-16T00:13:34.212 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.212 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.212 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.212 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.840413+0000", 2020-01-16T00:13:34.213 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9270999999999997e-05 2020-01-16T00:13:34.213 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.213 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.213 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:34.213 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.840452+0000", 2020-01-16T00:13:34.213 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014853975 2020-01-16T00:13:34.213 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.213 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.214 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.214 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.855306+0000", 2020-01-16T00:13:34.214 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011098400000000001 2020-01-16T00:13:34.214 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.214 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.214 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:34.214 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.855417+0000", 2020-01-16T00:13:34.215 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029108060000000002 2020-01-16T00:13:34.215 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.215 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.215 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.215 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858328+0000", 2020-01-16T00:13:34.215 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000144697 2020-01-16T00:13:34.216 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.216 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.216 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.216 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858472+0000", 2020-01-16T00:13:34.216 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018011899999999999 2020-01-16T00:13:34.216 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.216 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.216 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.217 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858653+0000", 2020-01-16T00:13:34.217 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018691670000000001 2020-01-16T00:13:34.217 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.217 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.217 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.217 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.217 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.217 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3295 3.2 3:42b7f20f:::smithi12313934-42:head [write 1982691~499457] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:13:34.218 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:30.744201+0000", 2020-01-16T00:13:34.218 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 3.2565871319999999, 2020-01-16T00:13:34.218 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018346738000000001, 2020-01-16T00:13:34.218 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.218 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:34.218 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:34.218 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:34.218 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:34.219 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3295 2020-01-16T00:13:34.219 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.219 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.219 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.219 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.219 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744201+0000", 2020-01-16T00:13:34.219 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.220 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.220 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.220 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.220 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744201+0000", 2020-01-16T00:13:34.220 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:34.220 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.220 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.220 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.221 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744198+0000", 2020-01-16T00:13:34.221 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00030375699999999999 2020-01-16T00:13:34.221 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.221 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.221 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.221 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744502+0000", 2020-01-16T00:13:34.221 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0121e-05 2020-01-16T00:13:34.221 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.222 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.222 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.222 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744512+0000", 2020-01-16T00:13:34.222 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2293000000000001e-05 2020-01-16T00:13:34.222 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.222 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.222 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.223 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744524+0000", 2020-01-16T00:13:34.223 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000274837 2020-01-16T00:13:34.223 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.223 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.223 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.223 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744799+0000", 2020-01-16T00:13:34.223 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036951879999999999 2020-01-16T00:13:34.223 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.224 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.224 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:34.224 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748494+0000", 2020-01-16T00:13:34.224 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010091493 2020-01-16T00:13:34.224 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.224 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.224 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.224 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.758586+0000", 2020-01-16T00:13:34.225 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011674400000000001 2020-01-16T00:13:34.225 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.225 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.225 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.225 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.758703+0000", 2020-01-16T00:13:34.225 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021412699999999999 2020-01-16T00:13:34.225 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.225 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.226 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:34.226 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.758917+0000", 2020-01-16T00:13:34.226 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0022519340000000001 2020-01-16T00:13:34.226 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.226 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.226 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:34.226 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.761169+0000", 2020-01-16T00:13:34.227 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001240891 2020-01-16T00:13:34.227 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.227 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.227 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.227 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.762410+0000", 2020-01-16T00:13:34.227 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6942e-05 2020-01-16T00:13:34.227 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.227 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.228 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:34.228 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.762447+0000", 2020-01-16T00:13:34.228 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000101358 2020-01-16T00:13:34.228 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.228 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.228 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.228 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.762548+0000", 2020-01-16T00:13:34.228 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018346738000000001 2020-01-16T00:13:34.229 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.229 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.229 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.229 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.229 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.229 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3296 3.2 3:42b7f20f:::smithi12313934-42:head [setxattr _header (56) in=7b,truncate 2739607] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:13:34.229 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:30.744532+0000", 2020-01-16T00:13:34.229 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 3.2562563990000002, 2020-01-16T00:13:34.230 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018472374, 2020-01-16T00:13:34.230 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.230 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:34.230 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:34.230 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:34.230 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:34.230 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3296 2020-01-16T00:13:34.231 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.231 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.231 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.231 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.231 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744532+0000", 2020-01-16T00:13:34.231 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.231 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.231 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.232 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.232 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744532+0000", 2020-01-16T00:13:34.232 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:34.232 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.232 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.232 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.232 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744531+0000", 2020-01-16T00:13:34.232 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7059999999999993e-06 2020-01-16T00:13:34.233 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.233 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.233 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.233 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744540+0000", 2020-01-16T00:13:34.233 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8589999999999998e-06 2020-01-16T00:13:34.233 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.233 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.233 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.234 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744544+0000", 2020-01-16T00:13:34.234 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7609999999999999e-06 2020-01-16T00:13:34.234 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.234 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.234 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.234 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744546+0000", 2020-01-16T00:13:34.234 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039933670000000003 2020-01-16T00:13:34.234 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.235 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.235 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.235 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748540+0000", 2020-01-16T00:13:34.235 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000110281 2020-01-16T00:13:34.235 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.235 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.235 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:34.235 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748650+0000", 2020-01-16T00:13:34.236 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.01160457 2020-01-16T00:13:34.236 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.236 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.236 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.236 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.760255+0000", 2020-01-16T00:13:34.236 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000124714 2020-01-16T00:13:34.236 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.236 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.237 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.237 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.760379+0000", 2020-01-16T00:13:34.237 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022099700000000001 2020-01-16T00:13:34.237 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.237 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.237 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:34.237 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.760600+0000", 2020-01-16T00:13:34.238 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00057843300000000005 2020-01-16T00:13:34.238 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.238 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.238 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:34.238 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.761179+0000", 2020-01-16T00:13:34.238 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00161477 2020-01-16T00:13:34.238 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.238 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.239 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.239 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.762794+0000", 2020-01-16T00:13:34.239 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4408999999999999e-05 2020-01-16T00:13:34.239 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.239 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.239 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:34.239 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.762858+0000", 2020-01-16T00:13:34.240 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014634199999999999 2020-01-16T00:13:34.240 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.240 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.240 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.240 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.763004+0000", 2020-01-16T00:13:34.240 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018472374 2020-01-16T00:13:34.240 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.240 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.241 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.241 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.241 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.241 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3297 3.2 3:42b7f20f:::smithi12313934-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e45)", 2020-01-16T00:13:34.241 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:30.744551+0000", 2020-01-16T00:13:34.241 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 3.256237429, 2020-01-16T00:13:34.241 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018660209000000001, 2020-01-16T00:13:34.241 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.242 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:34.242 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:34.242 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:34.242 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:34.242 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3297 2020-01-16T00:13:34.242 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.242 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.243 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.243 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.243 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744551+0000", 2020-01-16T00:13:34.243 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.243 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.243 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.243 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.243 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744551+0000", 2020-01-16T00:13:34.244 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:34.244 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.244 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.244 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.244 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744550+0000", 2020-01-16T00:13:34.244 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5279999999999999e-06 2020-01-16T00:13:34.244 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.244 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.245 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.245 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744554+0000", 2020-01-16T00:13:34.245 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8420000000000001e-06 2020-01-16T00:13:34.245 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.245 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.245 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.245 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744559+0000", 2020-01-16T00:13:34.245 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.965e-06 2020-01-16T00:13:34.246 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.246 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.246 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.246 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744562+0000", 2020-01-16T00:13:34.246 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041215380000000001 2020-01-16T00:13:34.246 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.246 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.246 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.247 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748684+0000", 2020-01-16T00:13:34.247 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8165999999999997e-05 2020-01-16T00:13:34.247 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.247 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.247 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:34.247 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748762+0000", 2020-01-16T00:13:34.247 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012136152000000001 2020-01-16T00:13:34.247 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.248 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.248 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.248 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.760898+0000", 2020-01-16T00:13:34.248 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000114833 2020-01-16T00:13:34.248 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.248 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.248 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:34.248 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.761013+0000", 2020-01-16T00:13:34.248 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002017015 2020-01-16T00:13:34.249 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.249 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.249 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.249 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.763030+0000", 2020-01-16T00:13:34.249 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5741000000000006e-05 2020-01-16T00:13:34.249 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.249 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.250 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.250 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.763096+0000", 2020-01-16T00:13:34.250 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011519900000000001 2020-01-16T00:13:34.250 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.250 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.250 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.250 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.763211+0000", 2020-01-16T00:13:34.250 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018660209000000001 2020-01-16T00:13:34.251 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.251 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.251 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.251 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.251 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.251 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:34.254 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:13:34.255 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-01-16T00:13:34.255 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-01-16T00:13:34.255 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-01-16T00:13:34.255 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-01-16T00:13:34.255 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-01-16T00:13:34.255 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENXIO: problem getting command descriptions from osd.5 2020-01-16T00:13:34.255 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: problem getting command descriptions from osd.5 2020-01-16T00:13:34.256 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-01-16T00:13:34.256 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-01-16T00:13:34.256 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-01-16T00:13:34.256 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-01-16T00:13:34.257 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-01-16T00:13:34.257 INFO:tasks.rados.rados.0.smithi123.stdout:1317: finishing write tid 1 to smithi12313934-50 2020-01-16T00:13:34.257 INFO:tasks.rados.rados.0.smithi123.stdout:1317: finishing write tid 2 to smithi12313934-50 2020-01-16T00:13:34.257 INFO:tasks.rados.rados.0.smithi123.stdout:1317: finishing write tid 3 to smithi12313934-50 2020-01-16T00:13:34.258 INFO:tasks.rados.rados.0.smithi123.stdout:1317: finishing write tid 5 to smithi12313934-50 2020-01-16T00:13:34.258 INFO:tasks.rados.rados.0.smithi123.stdout:1317: finishing write tid 6 to smithi12313934-50 2020-01-16T00:13:34.258 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 50 v 159 (ObjNum 408 snap 0 seq_num 408) dirty exists 2020-01-16T00:13:34.258 INFO:tasks.rados.rados.0.smithi123.stdout:1317: left oid 50 (ObjNum 408 snap 0 seq_num 408) 2020-01-16T00:13:34.258 INFO:tasks.rados.rados.0.smithi123.stdout:1318: finishing write tid 1 to smithi12313934-8 2020-01-16T00:13:34.258 INFO:tasks.rados.rados.0.smithi123.stdout:1318: finishing write tid 2 to smithi12313934-8 2020-01-16T00:13:34.258 INFO:tasks.rados.rados.0.smithi123.stdout:1318: finishing write tid 3 to smithi12313934-8 2020-01-16T00:13:34.258 INFO:tasks.rados.rados.0.smithi123.stdout:1318: finishing write tid 5 to smithi12313934-8 2020-01-16T00:13:34.259 INFO:tasks.rados.rados.0.smithi123.stdout:1314: done (15 left) 2020-01-16T00:13:34.259 INFO:tasks.rados.rados.0.smithi123.stdout:1315: done (14 left) 2020-01-16T00:13:34.259 INFO:tasks.rados.rados.0.smithi123.stdout:1316: done (13 left) 2020-01-16T00:13:34.259 INFO:tasks.rados.rados.0.smithi123.stdout:1317: done (12 left) 2020-01-16T00:13:34.259 INFO:tasks.rados.rados.0.smithi123.stdout:1330: read oid 23 snap -1 2020-01-16T00:13:34.259 INFO:tasks.rados.rados.0.smithi123.stdout:1330: expect deleted 2020-01-16T00:13:34.259 INFO:tasks.rados.rados.0.smithi123.stdout:1331: read oid 38 snap -1 2020-01-16T00:13:34.260 INFO:tasks.rados.rados.0.smithi123.stdout:1331: expect (ObjNum 298 snap 0 seq_num 298) 2020-01-16T00:13:34.260 INFO:tasks.rados.rados.0.smithi123.stdout:1332: write (excl) oid 16 current snap is 0 2020-01-16T00:13:34.260 INFO:tasks.rados.rados.0.smithi123.stdout:1332: seq_num 416 ranges {591712=786646,1943568=668798} 2020-01-16T00:13:34.260 INFO:tasks.rados.rados.0.smithi123.stdout:1332: writing smithi12313934-16 from 591712 to 1378358 tid 1 2020-01-16T00:13:34.260 INFO:tasks.rados.rados.0.smithi123.stdout:1332: writing smithi12313934-16 from 1943568 to 2612366 tid 2 2020-01-16T00:13:34.260 INFO:tasks.rados.rados.0.smithi123.stdout:1318: finishing write tid 6 to smithi12313934-8 2020-01-16T00:13:34.260 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 8 v 92 (ObjNum 409 snap 0 seq_num 409) dirty exists 2020-01-16T00:13:34.260 INFO:tasks.rados.rados.0.smithi123.stdout:1318: left oid 8 (ObjNum 409 snap 0 seq_num 409) 2020-01-16T00:13:34.261 INFO:tasks.rados.rados.0.smithi123.stdout:1333: read oid 35 snap -1 2020-01-16T00:13:34.261 INFO:tasks.rados.rados.0.smithi123.stdout:1333: expect (ObjNum 383 snap 0 seq_num 383) 2020-01-16T00:13:34.261 INFO:tasks.rados.rados.0.smithi123.stdout:1318: done (15 left) 2020-01-16T00:13:34.261 INFO:tasks.rados.rados.0.smithi123.stdout:1334: delete oid 24 current snap is 0 2020-01-16T00:13:34.261 INFO:tasks.rados.rados.0.smithi123.stdout:1319: expect (ObjNum 310 snap 0 seq_num 310) 2020-01-16T00:13:34.261 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:34.250+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:34.304 INFO:tasks.rados.rados.0.smithi123.stdout:1320: finishing write tid 1 to smithi12313934-20 2020-01-16T00:13:34.304 INFO:tasks.rados.rados.0.smithi123.stdout:1320: finishing write tid 2 to smithi12313934-20 2020-01-16T00:13:34.304 INFO:tasks.rados.rados.0.smithi123.stdout:1320: finishing write tid 3 to smithi12313934-20 2020-01-16T00:13:34.304 INFO:tasks.rados.rados.0.smithi123.stdout:1320: finishing write tid 5 to smithi12313934-20 2020-01-16T00:13:34.305 INFO:tasks.rados.rados.0.smithi123.stdout:1320: finishing write tid 6 to smithi12313934-20 2020-01-16T00:13:34.305 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 20 v 193 (ObjNum 410 snap 0 seq_num 410) dirty exists 2020-01-16T00:13:34.305 INFO:tasks.rados.rados.0.smithi123.stdout:1320: left oid 20 (ObjNum 410 snap 0 seq_num 410) 2020-01-16T00:13:34.305 INFO:tasks.rados.rados.0.smithi123.stdout:1321: finishing write tid 1 to smithi12313934-5 2020-01-16T00:13:34.305 INFO:tasks.rados.rados.0.smithi123.stdout:1321: finishing write tid 2 to smithi12313934-5 2020-01-16T00:13:34.305 INFO:tasks.rados.rados.0.smithi123.stdout:1322: finishing write tid 1 to smithi12313934-48 2020-01-16T00:13:34.305 INFO:tasks.rados.rados.0.smithi123.stdout:1319: done (15 left) 2020-01-16T00:13:34.306 INFO:tasks.rados.rados.0.smithi123.stdout:1320: done (14 left) 2020-01-16T00:13:34.306 INFO:tasks.rados.rados.0.smithi123.stdout:1334: done (13 left) 2020-01-16T00:13:34.306 INFO:tasks.rados.rados.0.smithi123.stdout:1335: write oid 8 current snap is 0 2020-01-16T00:13:34.306 INFO:tasks.rados.rados.0.smithi123.stdout:1335: seq_num 418 ranges {625326=537144,1944179=672467} 2020-01-16T00:13:34.321 INFO:tasks.rados.rados.0.smithi123.stdout:1335: writing smithi12313934-8 from 625326 to 1162470 tid 1 2020-01-16T00:13:34.331 INFO:tasks.rados.rados.0.smithi123.stdout:1335: writing smithi12313934-8 from 1944179 to 2616646 tid 2 2020-01-16T00:13:34.331 INFO:tasks.rados.rados.0.smithi123.stdout:1336: write (excl) oid 11 current snap is 0 2020-01-16T00:13:34.331 INFO:tasks.rados.rados.0.smithi123.stdout:1336: seq_num 419 ranges {642133=728746,2119014=532478} 2020-01-16T00:13:34.340 INFO:tasks.rados.rados.0.smithi123.stdout:1336: writing smithi12313934-11 from 642133 to 1370879 tid 1 2020-01-16T00:13:34.349 INFO:tasks.rados.rados.0.smithi123.stdout:1336: writing smithi12313934-11 from 2119014 to 2651492 tid 2 2020-01-16T00:13:34.349 INFO:tasks.rados.rados.0.smithi123.stdout:1337: write oid 26 current snap is 0 2020-01-16T00:13:34.349 INFO:tasks.rados.rados.0.smithi123.stdout:1337: seq_num 420 ranges {658940=403995,1777496=676136,2950951=107989} 2020-01-16T00:13:34.353 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:34.350+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:34.355 INFO:tasks.rados.rados.0.smithi123.stdout:1337: writing smithi12313934-26 from 658940 to 1062935 tid 1 2020-01-16T00:13:34.370 INFO:tasks.rados.rados.0.smithi123.stdout:1337: writing smithi12313934-26 from 1777496 to 2453632 tid 2 2020-01-16T00:13:34.371 INFO:tasks.rados.rados.0.smithi123.stdout:1337: writing smithi12313934-26 from 2950951 to 3058940 tid 3 2020-01-16T00:13:34.372 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:34.372 INFO:tasks.rados.rados.0.smithi123.stdout:1321: finishing write tid 3 to smithi12313934-5 2020-01-16T00:13:34.372 INFO:tasks.rados.rados.0.smithi123.stdout:1321: finishing write tid 5 to smithi12313934-5 2020-01-16T00:13:34.372 INFO:tasks.rados.rados.0.smithi123.stdout:1321: finishing write tid 6 to smithi12313934-5 2020-01-16T00:13:34.372 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 5 v 198 (ObjNum 411 snap 0 seq_num 411) dirty exists 2020-01-16T00:13:34.372 INFO:tasks.rados.rados.0.smithi123.stdout:1321: left oid 5 (ObjNum 411 snap 0 seq_num 411) 2020-01-16T00:13:34.372 INFO:tasks.rados.rados.0.smithi123.stdout:1322: finishing write tid 2 to smithi12313934-48 2020-01-16T00:13:34.373 INFO:tasks.rados.rados.0.smithi123.stdout:1323: finishing write tid 1 to smithi12313934-9 2020-01-16T00:13:34.373 INFO:tasks.rados.rados.0.smithi123.stdout:1321: done (15 left) 2020-01-16T00:13:34.373 INFO:tasks.rados.rados.0.smithi123.stdout:1338: read oid 44 snap -1 2020-01-16T00:13:34.373 INFO:tasks.rados.rados.0.smithi123.stdout:1338: expect (ObjNum 380 snap 0 seq_num 380) 2020-01-16T00:13:34.373 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:34.373 INFO:tasks.rados.rados.0.smithi123.stdout:1322: finishing write tid 3 to smithi12313934-48 2020-01-16T00:13:34.373 INFO:tasks.rados.rados.0.smithi123.stdout:1322: finishing write tid 5 to smithi12313934-48 2020-01-16T00:13:34.373 INFO:tasks.rados.rados.0.smithi123.stdout:1322: finishing write tid 6 to smithi12313934-48 2020-01-16T00:13:34.374 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 48 v 150 (ObjNum 412 snap 0 seq_num 412) dirty exists 2020-01-16T00:13:34.374 INFO:tasks.rados.rados.0.smithi123.stdout:1322: left oid 48 (ObjNum 412 snap 0 seq_num 412) 2020-01-16T00:13:34.374 INFO:tasks.rados.rados.0.smithi123.stdout:1323: finishing write tid 2 to smithi12313934-9 2020-01-16T00:13:34.374 INFO:tasks.rados.rados.0.smithi123.stdout:1323: finishing write tid 4 to smithi12313934-9 2020-01-16T00:13:34.374 INFO:tasks.rados.rados.0.smithi123.stdout:1323: finishing write tid 5 to smithi12313934-9 2020-01-16T00:13:34.374 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 9 v 133 (ObjNum 413 snap 0 seq_num 413) dirty exists 2020-01-16T00:13:34.374 INFO:tasks.rados.rados.0.smithi123.stdout:1323: left oid 9 (ObjNum 413 snap 0 seq_num 413) 2020-01-16T00:13:34.375 INFO:tasks.rados.rados.0.smithi123.stdout:1322: done (15 left) 2020-01-16T00:13:34.375 INFO:tasks.rados.rados.0.smithi123.stdout:1323: done (14 left) 2020-01-16T00:13:34.375 INFO:tasks.rados.rados.0.smithi123.stdout:1339: write (excl) oid 46 current snap is 0 2020-01-16T00:13:34.375 INFO:tasks.rados.rados.0.smithi123.stdout:1339: seq_num 421 ranges {675747=479244,1835978=536147} 2020-01-16T00:13:34.381 INFO:tasks.rados.rados.0.smithi123.stdout:1339: writing smithi12313934-46 from 675747 to 1154991 tid 1 2020-01-16T00:13:34.388 INFO:tasks.rados.rados.0.smithi123.stdout:1339: writing smithi12313934-46 from 1835978 to 2372125 tid 2 2020-01-16T00:13:34.388 INFO:tasks.rados.rados.0.smithi123.stdout:1324: expect (ObjNum 326 snap 0 seq_num 326) 2020-01-16T00:13:34.428 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:34.428 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:34.428 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:13:34.428 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:34.437 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-01-16T00:13:34.484 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:34.482+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:34.566 INFO:tasks.rados.rados.0.smithi123.stdout:1325: expect (ObjNum 220 snap 0 seq_num 220) 2020-01-16T00:13:34.567 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:34.562+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:34.583 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:34.583 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:34.583 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:13:34.583 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:13:34.583 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:34.591 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-01-16T00:13:34.668 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:34.664+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:34.742 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:34.742 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:13:34.742 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:13:34.742 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:13:34.742 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.743 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.4.0:114 3.9 e24/20 3:9fc687b2:::smithi12313934-44:head v 24'74, mlcod=24'72)", 2020-01-16T00:13:34.743 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:13:34.743 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 56.412274787000001, 2020-01-16T00:13:34.743 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010399429, 2020-01-16T00:13:34.743 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.743 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:34.743 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.744 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.744 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.744 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:13:34.744 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.744 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.744 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.744 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.744 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:13:34.745 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:34.745 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.745 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.745 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.745 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325276+0000", 2020-01-16T00:13:34.745 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0046127069999999997 2020-01-16T00:13:34.745 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.746 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.746 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.746 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329889+0000", 2020-01-16T00:13:34.746 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3519999999999998e-05 2020-01-16T00:13:34.746 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.746 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.746 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.746 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329912+0000", 2020-01-16T00:13:34.747 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4070000000000004e-06 2020-01-16T00:13:34.747 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.747 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.747 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.747 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329920+0000", 2020-01-16T00:13:34.747 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8817000000000003e-05 2020-01-16T00:13:34.747 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.747 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.748 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.748 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329978+0000", 2020-01-16T00:13:34.748 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9721e-05 2020-01-16T00:13:34.748 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.748 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.748 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.748 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329998+0000", 2020-01-16T00:13:34.748 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056328139999999999 2020-01-16T00:13:34.749 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.749 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.749 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:34.749 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.335631+0000", 2020-01-16T00:13:34.749 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7170999999999999e-05 2020-01-16T00:13:34.749 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.749 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.750 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.750 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.335678+0000", 2020-01-16T00:13:34.750 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010399429 2020-01-16T00:13:34.750 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.750 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.750 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.750 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.750 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.751 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:24 2.1 2:8b431f80:::smithi12313934-40 low_tier:head [write 790528~688128] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:13:34.751 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.482303+0000", 2020-01-16T00:13:34.751 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 56.255250691000001, 2020-01-16T00:13:34.751 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010071853, 2020-01-16T00:13:34.751 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.751 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:34.751 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.751 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.752 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.752 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.482303+0000", 2020-01-16T00:13:34.752 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.752 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.752 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.752 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.752 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.482303+0000", 2020-01-16T00:13:34.752 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:34.753 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.753 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.753 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.753 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.482301+0000", 2020-01-16T00:13:34.753 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031173960000000001 2020-01-16T00:13:34.753 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.753 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.753 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.754 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485418+0000", 2020-01-16T00:13:34.754 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9300000000000001e-05 2020-01-16T00:13:34.754 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.754 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.754 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.754 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485448+0000", 2020-01-16T00:13:34.754 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2611e-05 2020-01-16T00:13:34.754 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.755 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.755 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.755 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485460+0000", 2020-01-16T00:13:34.755 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112487 2020-01-16T00:13:34.755 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.755 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.755 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.755 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485573+0000", 2020-01-16T00:13:34.756 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022169299999999999 2020-01-16T00:13:34.756 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.756 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.756 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.756 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485794+0000", 2020-01-16T00:13:34.756 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00036287500000000002 2020-01-16T00:13:34.756 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.757 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.757 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:34.757 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.486157+0000", 2020-01-16T00:13:34.757 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040260310000000002 2020-01-16T00:13:34.757 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.757 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.757 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:34.757 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.490183+0000", 2020-01-16T00:13:34.758 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00081083000000000001 2020-01-16T00:13:34.758 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.758 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.758 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.758 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.490994+0000", 2020-01-16T00:13:34.758 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001287746 2020-01-16T00:13:34.758 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.758 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.759 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.759 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492282+0000", 2020-01-16T00:13:34.759 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8810999999999998e-05 2020-01-16T00:13:34.759 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.759 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.759 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:34.759 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492321+0000", 2020-01-16T00:13:34.759 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4095000000000003e-05 2020-01-16T00:13:34.760 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.760 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.760 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.760 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492375+0000", 2020-01-16T00:13:34.760 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010071853 2020-01-16T00:13:34.760 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.760 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.761 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.761 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.761 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.761 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:55 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e30)", 2020-01-16T00:13:34.761 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.736036+0000", 2020-01-16T00:13:34.761 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 42.001517567999997, 2020-01-16T00:13:34.761 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010974536, 2020-01-16T00:13:34.761 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.762 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:34.762 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.762 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.762 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.762 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736036+0000", 2020-01-16T00:13:34.762 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.762 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.762 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.763 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.763 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736036+0000", 2020-01-16T00:13:34.763 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:34.763 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.763 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.763 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.763 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736034+0000", 2020-01-16T00:13:34.763 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043589699999999998 2020-01-16T00:13:34.764 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.764 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.764 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.764 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736469+0000", 2020-01-16T00:13:34.764 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9882e-05 2020-01-16T00:13:34.764 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.764 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.764 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.765 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736489+0000", 2020-01-16T00:13:34.765 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8339999999999999e-06 2020-01-16T00:13:34.765 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.765 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.765 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.765 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736497+0000", 2020-01-16T00:13:34.765 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5354999999999999e-05 2020-01-16T00:13:34.765 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.766 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.766 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.766 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736572+0000", 2020-01-16T00:13:34.766 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5228000000000004e-05 2020-01-16T00:13:34.766 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.766 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.766 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.767 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736668+0000", 2020-01-16T00:13:34.767 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017212999999999999 2020-01-16T00:13:34.767 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.767 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.767 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:34.767 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736840+0000", 2020-01-16T00:13:34.767 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0099057020000000006 2020-01-16T00:13:34.768 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.768 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.768 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.768 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746746+0000", 2020-01-16T00:13:34.768 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0586000000000005e-05 2020-01-16T00:13:34.768 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.768 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.769 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:34.769 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746816+0000", 2020-01-16T00:13:34.769 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2445000000000002e-05 2020-01-16T00:13:34.769 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.769 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.769 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.770 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746889+0000", 2020-01-16T00:13:34.770 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6839000000000002e-05 2020-01-16T00:13:34.770 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.770 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.770 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:34.770 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746955+0000", 2020-01-16T00:13:34.770 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5074000000000002e-05 2020-01-16T00:13:34.770 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.771 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.771 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.771 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.747011+0000", 2020-01-16T00:13:34.771 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010974536 2020-01-16T00:13:34.771 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.771 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.771 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.771 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.772 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.772 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:56 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e30)", 2020-01-16T00:13:34.772 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.736514+0000", 2020-01-16T00:13:34.772 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 42.001039732999999, 2020-01-16T00:13:34.772 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011660584999999999, 2020-01-16T00:13:34.772 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.772 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:34.772 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.773 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.773 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.773 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736514+0000", 2020-01-16T00:13:34.773 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.773 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.773 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.773 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.773 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736514+0000", 2020-01-16T00:13:34.774 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:34.774 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.774 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.774 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.774 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736513+0000", 2020-01-16T00:13:34.774 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9677000000000004e-05 2020-01-16T00:13:34.774 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.774 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.775 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.775 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736583+0000", 2020-01-16T00:13:34.775 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5269999999999998e-06 2020-01-16T00:13:34.775 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.775 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.775 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.776 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736591+0000", 2020-01-16T00:13:34.776 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5769999999999996e-06 2020-01-16T00:13:34.776 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.776 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.776 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.776 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736599+0000", 2020-01-16T00:13:34.776 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0053593310000000002 2020-01-16T00:13:34.776 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.777 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.777 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.777 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.741958+0000", 2020-01-16T00:13:34.777 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014164099999999999 2020-01-16T00:13:34.777 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.777 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.777 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.777 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.742100+0000", 2020-01-16T00:13:34.778 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020126000000000001 2020-01-16T00:13:34.778 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.778 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.778 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:34.778 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.742301+0000", 2020-01-16T00:13:34.778 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0045052859999999998 2020-01-16T00:13:34.778 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.778 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.779 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.779 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746806+0000", 2020-01-16T00:13:34.779 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00070802699999999996 2020-01-16T00:13:34.779 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.779 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.779 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:34.779 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.747514+0000", 2020-01-16T00:13:34.779 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00059247200000000003 2020-01-16T00:13:34.780 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.780 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.780 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.780 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748107+0000", 2020-01-16T00:13:34.780 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1870999999999999e-05 2020-01-16T00:13:34.780 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.780 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.780 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:34.781 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748149+0000", 2020-01-16T00:13:34.781 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5831000000000001e-05 2020-01-16T00:13:34.781 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.781 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.781 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.781 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748174+0000", 2020-01-16T00:13:34.781 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011660584999999999 2020-01-16T00:13:34.781 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.782 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.782 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.782 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.782 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.782 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:69 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 1922048~118272] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e34)", 2020-01-16T00:13:34.782 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:03.258431+0000", 2020-01-16T00:13:34.782 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 31.479122151999999, 2020-01-16T00:13:34.783 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0092806959999999997, 2020-01-16T00:13:34.783 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.783 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:34.783 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.783 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.783 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.783 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258431+0000", 2020-01-16T00:13:34.783 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.783 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.784 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.784 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.784 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258431+0000", 2020-01-16T00:13:34.784 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:34.784 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.784 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.784 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.785 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258431+0000", 2020-01-16T00:13:34.785 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3875000000000003e-05 2020-01-16T00:13:34.785 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.785 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.785 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.785 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258474+0000", 2020-01-16T00:13:34.785 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1040000000000007e-06 2020-01-16T00:13:34.785 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.786 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.786 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.786 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258483+0000", 2020-01-16T00:13:34.786 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6720000000000003e-06 2020-01-16T00:13:34.786 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.786 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.786 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.786 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.258487+0000", 2020-01-16T00:13:34.787 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020507110000000002 2020-01-16T00:13:34.787 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.787 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.787 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.787 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.260538+0000", 2020-01-16T00:13:34.787 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017511100000000001 2020-01-16T00:13:34.787 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.787 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.788 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.788 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.260713+0000", 2020-01-16T00:13:34.788 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018914100000000001 2020-01-16T00:13:34.788 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.788 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.788 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:34.788 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.260902+0000", 2020-01-16T00:13:34.789 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.004582282 2020-01-16T00:13:34.789 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.789 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.789 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:34.789 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.265485+0000", 2020-01-16T00:13:34.789 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021355329999999998 2020-01-16T00:13:34.790 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.790 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.790 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.790 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.267620+0000", 2020-01-16T00:13:34.790 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1495999999999998e-05 2020-01-16T00:13:34.790 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.790 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.790 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.791 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.267652+0000", 2020-01-16T00:13:34.791 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5474e-05 2020-01-16T00:13:34.791 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.791 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.791 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:34.791 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.267687+0000", 2020-01-16T00:13:34.791 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5055e-05 2020-01-16T00:13:34.791 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.792 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.792 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.792 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:03.267712+0000", 2020-01-16T00:13:34.792 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0092806959999999997 2020-01-16T00:13:34.792 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.792 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.792 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.792 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.793 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.793 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:70 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e35)", 2020-01-16T00:13:34.793 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.985494+0000", 2020-01-16T00:13:34.793 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 28.752059936999999, 2020-01-16T00:13:34.793 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013746267, 2020-01-16T00:13:34.793 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.793 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:34.793 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.794 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.794 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.794 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985494+0000", 2020-01-16T00:13:34.794 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.794 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.794 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.794 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.794 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985494+0000", 2020-01-16T00:13:34.795 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:34.795 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.795 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.795 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.795 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985490+0000", 2020-01-16T00:13:34.795 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00077792099999999999 2020-01-16T00:13:34.795 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.795 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.796 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.796 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986268+0000", 2020-01-16T00:13:34.796 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9718000000000001e-05 2020-01-16T00:13:34.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.796 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.796 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.796 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986318+0000", 2020-01-16T00:13:34.796 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.4349999999999997e-06 2020-01-16T00:13:34.797 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.797 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.797 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.797 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986326+0000", 2020-01-16T00:13:34.797 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3298999999999999e-05 2020-01-16T00:13:34.797 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.797 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.797 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.798 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986380+0000", 2020-01-16T00:13:34.798 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4969999999999995e-05 2020-01-16T00:13:34.798 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.798 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.798 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.798 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986455+0000", 2020-01-16T00:13:34.798 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000110587 2020-01-16T00:13:34.798 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.799 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.799 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:34.799 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986565+0000", 2020-01-16T00:13:34.799 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063931600000000002 2020-01-16T00:13:34.799 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.799 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.799 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:34.800 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.992958+0000", 2020-01-16T00:13:34.800 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013008480000000001 2020-01-16T00:13:34.800 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.800 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.800 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.800 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.994259+0000", 2020-01-16T00:13:34.800 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048399979999999999 2020-01-16T00:13:34.800 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.801 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.801 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.801 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999099+0000", 2020-01-16T00:13:34.801 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1874999999999998e-05 2020-01-16T00:13:34.801 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.801 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.801 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:34.802 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999181+0000", 2020-01-16T00:13:34.802 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8865e-05 2020-01-16T00:13:34.802 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.802 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.802 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.802 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999240+0000", 2020-01-16T00:13:34.803 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013746267 2020-01-16T00:13:34.803 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.803 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.803 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.803 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.803 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.803 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:71 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e35)", 2020-01-16T00:13:34.803 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.986341+0000", 2020-01-16T00:13:34.804 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 28.751212576, 2020-01-16T00:13:34.804 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013252807, 2020-01-16T00:13:34.804 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.804 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:34.804 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.804 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.804 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.804 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986341+0000", 2020-01-16T00:13:34.805 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.805 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.805 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.805 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.805 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986341+0000", 2020-01-16T00:13:34.805 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:34.805 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.805 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.806 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.806 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986340+0000", 2020-01-16T00:13:34.806 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000103867 2020-01-16T00:13:34.806 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.806 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.806 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.806 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986444+0000", 2020-01-16T00:13:34.806 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0104e-05 2020-01-16T00:13:34.807 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.807 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.807 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.807 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986454+0000", 2020-01-16T00:13:34.807 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4220000000000002e-06 2020-01-16T00:13:34.807 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.807 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.807 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.808 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986458+0000", 2020-01-16T00:13:34.808 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033523070000000001 2020-01-16T00:13:34.808 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.808 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.808 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.808 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.989811+0000", 2020-01-16T00:13:34.808 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3296999999999999e-05 2020-01-16T00:13:34.808 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.809 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.809 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.809 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.989894+0000", 2020-01-16T00:13:34.809 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000117701 2020-01-16T00:13:34.809 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.809 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.810 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:34.810 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.990012+0000", 2020-01-16T00:13:34.810 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0032873490000000002 2020-01-16T00:13:34.810 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.810 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.810 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:34.810 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993299+0000", 2020-01-16T00:13:34.811 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002422619 2020-01-16T00:13:34.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.811 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.811 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.811 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.995722+0000", 2020-01-16T00:13:34.811 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0037839589999999999 2020-01-16T00:13:34.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.811 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.812 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.812 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999506+0000", 2020-01-16T00:13:34.812 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8559999999999997e-05 2020-01-16T00:13:34.812 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.812 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.812 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:34.812 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999544+0000", 2020-01-16T00:13:34.812 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9599999999999999e-05 2020-01-16T00:13:34.813 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.813 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.813 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.813 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999594+0000", 2020-01-16T00:13:34.813 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013252807 2020-01-16T00:13:34.813 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.813 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.813 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.814 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.814 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.814 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:77 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e36)", 2020-01-16T00:13:34.814 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.555854+0000", 2020-01-16T00:13:34.814 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 27.181699486999999, 2020-01-16T00:13:34.814 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010455002999999999, 2020-01-16T00:13:34.814 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.814 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:34.815 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.815 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.815 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.815 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555854+0000", 2020-01-16T00:13:34.815 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.815 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.815 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.815 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.816 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555854+0000", 2020-01-16T00:13:34.816 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:34.816 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.816 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.816 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.816 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555851+0000", 2020-01-16T00:13:34.816 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00048310200000000001 2020-01-16T00:13:34.817 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.817 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.817 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.817 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556334+0000", 2020-01-16T00:13:34.817 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2232e-05 2020-01-16T00:13:34.817 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.817 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.818 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.818 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556356+0000", 2020-01-16T00:13:34.818 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9750000000000007e-06 2020-01-16T00:13:34.818 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.818 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.818 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.818 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556364+0000", 2020-01-16T00:13:34.818 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8872999999999997e-05 2020-01-16T00:13:34.819 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.819 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.819 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.819 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556423+0000", 2020-01-16T00:13:34.819 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8687000000000001e-05 2020-01-16T00:13:34.819 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.820 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.820 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.820 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556502+0000", 2020-01-16T00:13:34.820 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013235300000000001 2020-01-16T00:13:34.820 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.820 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.820 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:34.820 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556634+0000", 2020-01-16T00:13:34.821 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0059770520000000001 2020-01-16T00:13:34.821 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.821 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.821 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:34.821 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.562611+0000", 2020-01-16T00:13:34.821 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029492820000000001 2020-01-16T00:13:34.821 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.821 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.822 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.822 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.565561+0000", 2020-01-16T00:13:34.822 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00060620300000000004 2020-01-16T00:13:34.822 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.822 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.822 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.822 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566167+0000", 2020-01-16T00:13:34.823 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3831000000000004e-05 2020-01-16T00:13:34.823 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.823 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.823 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:34.823 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566251+0000", 2020-01-16T00:13:34.823 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8328000000000001e-05 2020-01-16T00:13:34.823 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.823 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.824 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.824 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566309+0000", 2020-01-16T00:13:34.824 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010455002999999999 2020-01-16T00:13:34.824 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.824 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.824 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.824 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.824 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.825 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:78 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e36)", 2020-01-16T00:13:34.825 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.556378+0000", 2020-01-16T00:13:34.825 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 27.181175553999999, 2020-01-16T00:13:34.825 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010396311, 2020-01-16T00:13:34.825 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.825 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:34.825 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.825 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.826 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.826 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556378+0000", 2020-01-16T00:13:34.826 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.826 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.826 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.826 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.826 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556378+0000", 2020-01-16T00:13:34.827 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:34.827 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.827 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.827 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.827 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556377+0000", 2020-01-16T00:13:34.827 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4307000000000001e-05 2020-01-16T00:13:34.827 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.827 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.828 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.828 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556422+0000", 2020-01-16T00:13:34.828 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9620000000000005e-06 2020-01-16T00:13:34.828 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.828 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.828 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.828 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556432+0000", 2020-01-16T00:13:34.829 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5979999999999999e-06 2020-01-16T00:13:34.829 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.829 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.829 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.829 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556436+0000", 2020-01-16T00:13:34.829 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021059469999999999 2020-01-16T00:13:34.829 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.829 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.830 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.830 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.558542+0000", 2020-01-16T00:13:34.830 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000194458 2020-01-16T00:13:34.830 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.830 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.830 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.830 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.558737+0000", 2020-01-16T00:13:34.830 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00032867900000000002 2020-01-16T00:13:34.831 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.831 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.831 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:34.831 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.559065+0000", 2020-01-16T00:13:34.831 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052547499999999999 2020-01-16T00:13:34.831 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.831 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.832 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:34.832 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.564320+0000", 2020-01-16T00:13:34.832 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001565213 2020-01-16T00:13:34.832 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.832 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.832 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.832 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.565885+0000", 2020-01-16T00:13:34.833 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00076994300000000004 2020-01-16T00:13:34.833 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.833 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.833 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.833 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566655+0000", 2020-01-16T00:13:34.833 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7533000000000005e-05 2020-01-16T00:13:34.833 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.833 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.834 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:34.834 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566723+0000", 2020-01-16T00:13:34.834 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1616999999999997e-05 2020-01-16T00:13:34.834 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.834 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.834 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.834 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566774+0000", 2020-01-16T00:13:34.834 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010396311 2020-01-16T00:13:34.835 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.835 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.835 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.835 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.835 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.835 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:72 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 56832~1408000] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e37)", 2020-01-16T00:13:34.835 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.578753+0000", 2020-01-16T00:13:34.835 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 22.158800250999999, 2020-01-16T00:13:34.836 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011826918, 2020-01-16T00:13:34.836 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.836 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:34.836 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.836 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.836 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.836 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578753+0000", 2020-01-16T00:13:34.836 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.837 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.837 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.837 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.837 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578753+0000", 2020-01-16T00:13:34.837 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:34.837 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.837 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.837 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.838 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578751+0000", 2020-01-16T00:13:34.838 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00042457699999999998 2020-01-16T00:13:34.838 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.838 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.838 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.838 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579175+0000", 2020-01-16T00:13:34.838 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5404999999999999e-05 2020-01-16T00:13:34.838 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.839 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.839 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.839 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579191+0000", 2020-01-16T00:13:34.839 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6149999999999996e-06 2020-01-16T00:13:34.839 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.839 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.839 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.840 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579196+0000", 2020-01-16T00:13:34.840 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000104794 2020-01-16T00:13:34.840 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.840 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.840 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.840 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579301+0000", 2020-01-16T00:13:34.840 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020947199999999999 2020-01-16T00:13:34.841 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.841 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.841 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.841 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579511+0000", 2020-01-16T00:13:34.841 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034328400000000002 2020-01-16T00:13:34.841 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.841 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.841 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:34.842 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579854+0000", 2020-01-16T00:13:34.842 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.005078117 2020-01-16T00:13:34.842 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.842 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.842 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:34.842 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.584932+0000", 2020-01-16T00:13:34.842 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0014702160000000001 2020-01-16T00:13:34.842 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.843 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.843 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.843 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.586402+0000", 2020-01-16T00:13:34.843 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040623589999999998 2020-01-16T00:13:34.843 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.843 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.843 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.843 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590465+0000", 2020-01-16T00:13:34.844 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4478000000000002e-05 2020-01-16T00:13:34.844 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.844 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.844 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:34.844 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590529+0000", 2020-01-16T00:13:34.844 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1121000000000001e-05 2020-01-16T00:13:34.844 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.844 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.845 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.845 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590580+0000", 2020-01-16T00:13:34.845 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011826918 2020-01-16T00:13:34.845 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.845 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.845 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.845 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.846 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.846 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:73 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 1922048~118272] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e37)", 2020-01-16T00:13:34.846 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:13:34.846 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 22.158347454000001, 2020-01-16T00:13:34.846 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011750764, 2020-01-16T00:13:34.846 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.846 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:34.846 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.847 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.847 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.847 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:13:34.847 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.847 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.847 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.847 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.847 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:13:34.848 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:34.848 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.848 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.848 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.848 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:13:34.848 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2780000000000001e-05 2020-01-16T00:13:34.848 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.848 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.849 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.849 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579238+0000", 2020-01-16T00:13:34.849 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4341999999999999e-05 2020-01-16T00:13:34.849 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.849 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.849 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.849 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579253+0000", 2020-01-16T00:13:34.849 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1920000000000001e-06 2020-01-16T00:13:34.850 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.850 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.850 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.850 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579256+0000", 2020-01-16T00:13:34.850 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003224902 2020-01-16T00:13:34.850 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.850 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.850 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.851 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.582481+0000", 2020-01-16T00:13:34.851 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000180729 2020-01-16T00:13:34.851 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.851 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.851 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.851 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.582662+0000", 2020-01-16T00:13:34.851 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00025106100000000001 2020-01-16T00:13:34.851 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.852 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.852 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:34.852 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.582913+0000", 2020-01-16T00:13:34.852 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020814449999999999 2020-01-16T00:13:34.852 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.852 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.852 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:34.852 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.584994+0000", 2020-01-16T00:13:34.853 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001717243 2020-01-16T00:13:34.853 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.853 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.853 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.853 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.586711+0000", 2020-01-16T00:13:34.853 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041349869999999997 2020-01-16T00:13:34.853 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.854 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.854 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.854 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590846+0000", 2020-01-16T00:13:34.854 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3565000000000006e-05 2020-01-16T00:13:34.854 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.854 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.855 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:34.855 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590910+0000", 2020-01-16T00:13:34.855 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7002999999999997e-05 2020-01-16T00:13:34.855 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.855 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.855 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.855 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590957+0000", 2020-01-16T00:13:34.856 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011750764 2020-01-16T00:13:34.856 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.856 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.856 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.856 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.856 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.856 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:89 2.1 2:9b8617ce:::smithi12313934-38 low_tier:head [write 69632~773120] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e37)", 2020-01-16T00:13:34.856 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.593701+0000", 2020-01-16T00:13:34.857 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 22.143852828, 2020-01-16T00:13:34.857 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010026929, 2020-01-16T00:13:34.857 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.857 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:34.857 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.857 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.857 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.857 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593701+0000", 2020-01-16T00:13:34.858 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.858 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.858 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.858 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.858 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593701+0000", 2020-01-16T00:13:34.858 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:34.858 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.858 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.859 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.859 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593699+0000", 2020-01-16T00:13:34.859 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000237564 2020-01-16T00:13:34.859 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.859 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.859 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.859 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593936+0000", 2020-01-16T00:13:34.859 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2666e-05 2020-01-16T00:13:34.860 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.860 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.860 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.860 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593949+0000", 2020-01-16T00:13:34.860 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0089999999999998e-06 2020-01-16T00:13:34.860 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.860 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.860 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.861 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593954+0000", 2020-01-16T00:13:34.861 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5813000000000003e-05 2020-01-16T00:13:34.861 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.861 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.861 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.861 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.594000+0000", 2020-01-16T00:13:34.861 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010155299999999999 2020-01-16T00:13:34.861 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.862 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.862 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.862 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.594101+0000", 2020-01-16T00:13:34.862 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000203807 2020-01-16T00:13:34.862 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.862 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.862 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:34.862 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.594305+0000", 2020-01-16T00:13:34.863 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021655390000000002 2020-01-16T00:13:34.863 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.863 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.863 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.863 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.596471+0000", 2020-01-16T00:13:34.863 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0070645320000000001 2020-01-16T00:13:34.863 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.863 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.864 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.864 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603535+0000", 2020-01-16T00:13:34.864 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7577999999999998e-05 2020-01-16T00:13:34.864 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.864 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.864 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:34.864 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603573+0000", 2020-01-16T00:13:34.864 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9933999999999997e-05 2020-01-16T00:13:34.865 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.865 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.865 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:34.865 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603653+0000", 2020-01-16T00:13:34.865 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4993000000000005e-05 2020-01-16T00:13:34.865 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.865 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.865 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.866 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603728+0000", 2020-01-16T00:13:34.866 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010026929 2020-01-16T00:13:34.866 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.866 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.866 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.866 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.866 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.866 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:66 2.3 2:c3b82d87:::smithi12313934-44 low_tier:head [write 169472~818176] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e44)", 2020-01-16T00:13:34.867 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:25.278583+0000", 2020-01-16T00:13:34.867 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 9.4589704309999991, 2020-01-16T00:13:34.867 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010294862, 2020-01-16T00:13:34.867 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.867 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:34.867 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.867 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.868 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.868 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.278583+0000", 2020-01-16T00:13:34.868 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.868 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.868 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.868 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.869 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.278583+0000", 2020-01-16T00:13:34.869 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:34.869 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.869 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.869 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.869 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.278581+0000", 2020-01-16T00:13:34.869 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00136721 2020-01-16T00:13:34.869 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.870 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.870 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.870 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.279948+0000", 2020-01-16T00:13:34.870 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6109999999999999e-05 2020-01-16T00:13:34.870 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.870 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.871 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.871 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.279974+0000", 2020-01-16T00:13:34.871 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.102e-05 2020-01-16T00:13:34.871 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.871 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.871 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.871 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.279985+0000", 2020-01-16T00:13:34.871 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6035e-05 2020-01-16T00:13:34.872 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.872 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.872 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.872 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.280061+0000", 2020-01-16T00:13:34.872 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000176243 2020-01-16T00:13:34.872 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.872 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.872 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.873 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.280238+0000", 2020-01-16T00:13:34.873 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019866099999999999 2020-01-16T00:13:34.873 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.873 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.873 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 7", 2020-01-16T00:13:34.873 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.280436+0000", 2020-01-16T00:13:34.873 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0049761830000000003 2020-01-16T00:13:34.873 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.874 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.874 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.874 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.285412+0000", 2020-01-16T00:13:34.874 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033347339999999998 2020-01-16T00:13:34.874 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.874 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.874 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:34.874 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.288747+0000", 2020-01-16T00:13:34.875 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9156000000000001e-05 2020-01-16T00:13:34.875 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.875 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.875 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:34.875 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.288806+0000", 2020-01-16T00:13:34.875 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1643000000000005e-05 2020-01-16T00:13:34.876 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.876 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.876 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.876 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.288878+0000", 2020-01-16T00:13:34.876 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010294862 2020-01-16T00:13:34.876 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.876 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.876 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.877 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.877 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.877 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:73 2.1 2:8b431f80:::smithi12313934-40 low_tier:head [write 790528~688128] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e45)", 2020-01-16T00:13:34.877 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.574742+0000", 2020-01-16T00:13:34.877 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.1628116789999998, 2020-01-16T00:13:34.877 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0091537509999999999, 2020-01-16T00:13:34.877 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.877 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:34.878 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.881 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.881 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.881 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.574742+0000", 2020-01-16T00:13:34.881 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.881 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.881 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.883 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.883 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.574742+0000", 2020-01-16T00:13:34.883 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:34.883 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.884 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.884 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.884 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.574740+0000", 2020-01-16T00:13:34.884 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001119203 2020-01-16T00:13:34.884 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.884 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.884 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.885 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.575859+0000", 2020-01-16T00:13:34.885 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4990999999999999e-05 2020-01-16T00:13:34.885 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.885 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.885 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.885 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.575874+0000", 2020-01-16T00:13:34.886 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9240000000000001e-06 2020-01-16T00:13:34.886 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.886 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.886 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.886 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.575881+0000", 2020-01-16T00:13:34.886 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2113000000000005e-05 2020-01-16T00:13:34.886 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.887 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.887 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.887 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.575963+0000", 2020-01-16T00:13:34.887 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000104929 2020-01-16T00:13:34.887 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.888 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.888 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.888 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.576068+0000", 2020-01-16T00:13:34.888 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016916000000000001 2020-01-16T00:13:34.888 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.888 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.889 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:34.889 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.576237+0000", 2020-01-16T00:13:34.889 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0025967379999999999 2020-01-16T00:13:34.889 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.889 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.889 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:34.889 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.578834+0000", 2020-01-16T00:13:34.890 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001768862 2020-01-16T00:13:34.890 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.890 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.890 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.890 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.580603+0000", 2020-01-16T00:13:34.890 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003177221 2020-01-16T00:13:34.890 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.891 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.891 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.891 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.583780+0000", 2020-01-16T00:13:34.891 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1255999999999998e-05 2020-01-16T00:13:34.891 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.891 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.891 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:34.891 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.583841+0000", 2020-01-16T00:13:34.892 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4144999999999997e-05 2020-01-16T00:13:34.892 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.892 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.892 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.892 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.583896+0000", 2020-01-16T00:13:34.892 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0091537509999999999 2020-01-16T00:13:34.892 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.892 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.893 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.893 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.893 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.893 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.4.0:316 3.9 e45/20 3:93dfe6f1:::smithi12313934-40:head v 45'188, mlcod=45'186)", 2020-01-16T00:13:34.893 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.584377+0000", 2020-01-16T00:13:34.893 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.1531765050000002, 2020-01-16T00:13:34.893 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095979069999999993, 2020-01-16T00:13:34.893 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.894 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:34.894 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.894 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.894 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.894 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584377+0000", 2020-01-16T00:13:34.894 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.894 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.895 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.895 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.895 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584377+0000", 2020-01-16T00:13:34.895 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:34.895 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.895 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.895 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.895 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584376+0000", 2020-01-16T00:13:34.896 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9059999999999997e-06 2020-01-16T00:13:34.896 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.896 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.896 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.896 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584385+0000", 2020-01-16T00:13:34.896 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3200000000000001e-05 2020-01-16T00:13:34.896 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.896 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.897 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.897 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584399+0000", 2020-01-16T00:13:34.897 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9660000000000001e-06 2020-01-16T00:13:34.897 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.897 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.897 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.898 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584405+0000", 2020-01-16T00:13:34.898 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0035787800000000002 2020-01-16T00:13:34.898 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.898 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.898 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.898 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.587983+0000", 2020-01-16T00:13:34.898 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0368000000000001e-05 2020-01-16T00:13:34.898 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.899 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.899 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.899 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588014+0000", 2020-01-16T00:13:34.899 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0059140290000000003 2020-01-16T00:13:34.899 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.899 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.900 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:34.900 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.593928+0000", 2020-01-16T00:13:34.900 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7135999999999997e-05 2020-01-16T00:13:34.900 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.900 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.900 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.900 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.593975+0000", 2020-01-16T00:13:34.901 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095979069999999993 2020-01-16T00:13:34.901 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.901 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.901 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.901 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.901 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.901 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:77 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 56832~1408000] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e45)", 2020-01-16T00:13:34.901 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.584566+0000", 2020-01-16T00:13:34.902 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.1529877800000001, 2020-01-16T00:13:34.902 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017037775000000002, 2020-01-16T00:13:34.902 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.902 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:34.902 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.902 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.903 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.903 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584566+0000", 2020-01-16T00:13:34.903 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.903 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.903 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.903 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.903 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584566+0000", 2020-01-16T00:13:34.903 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:34.904 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.904 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.904 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.904 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584564+0000", 2020-01-16T00:13:34.904 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043024300000000002 2020-01-16T00:13:34.905 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.905 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.905 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.905 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584994+0000", 2020-01-16T00:13:34.906 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9485999999999999e-05 2020-01-16T00:13:34.906 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.906 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.906 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.906 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585014+0000", 2020-01-16T00:13:34.906 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8909999999999995e-06 2020-01-16T00:13:34.906 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.907 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.907 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.907 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585022+0000", 2020-01-16T00:13:34.907 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039114459999999998 2020-01-16T00:13:34.907 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.907 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.908 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.908 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588933+0000", 2020-01-16T00:13:34.908 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011178700000000001 2020-01-16T00:13:34.908 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.908 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.908 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.908 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589045+0000", 2020-01-16T00:13:34.908 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016967300000000001 2020-01-16T00:13:34.909 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.909 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.909 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:34.909 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589214+0000", 2020-01-16T00:13:34.909 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012142356 2020-01-16T00:13:34.909 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.910 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.910 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.910 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601357+0000", 2020-01-16T00:13:34.910 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.1466999999999997e-05 2020-01-16T00:13:34.910 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.910 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.910 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.910 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601448+0000", 2020-01-16T00:13:34.911 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4817000000000001e-05 2020-01-16T00:13:34.911 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.911 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.911 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:34.911 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601493+0000", 2020-01-16T00:13:34.911 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9734000000000001e-05 2020-01-16T00:13:34.911 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.911 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.912 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:34.912 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601553+0000", 2020-01-16T00:13:34.912 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0677999999999998e-05 2020-01-16T00:13:34.912 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.912 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.912 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.912 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601604+0000", 2020-01-16T00:13:34.913 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017037775000000002 2020-01-16T00:13:34.913 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.913 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.913 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.913 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.913 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.913 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:78 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 1922048~118272] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e45)", 2020-01-16T00:13:34.914 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.585030+0000", 2020-01-16T00:13:34.914 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.1525230579999999, 2020-01-16T00:13:34.914 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016699964000000001, 2020-01-16T00:13:34.914 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.914 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:34.914 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.914 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.915 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.915 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585030+0000", 2020-01-16T00:13:34.915 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.915 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.915 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.915 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.915 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585030+0000", 2020-01-16T00:13:34.915 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:34.916 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.916 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.916 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.916 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585030+0000", 2020-01-16T00:13:34.916 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7085000000000001e-05 2020-01-16T00:13:34.916 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.916 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.916 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.917 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585077+0000", 2020-01-16T00:13:34.917 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5380000000000008e-06 2020-01-16T00:13:34.917 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.917 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.917 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.917 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585086+0000", 2020-01-16T00:13:34.917 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5410000000000002e-06 2020-01-16T00:13:34.917 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.918 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.918 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.918 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585091+0000", 2020-01-16T00:13:34.918 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0088486620000000002 2020-01-16T00:13:34.918 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.918 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.918 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.918 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.593940+0000", 2020-01-16T00:13:34.919 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112594 2020-01-16T00:13:34.919 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.919 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.919 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.919 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.594052+0000", 2020-01-16T00:13:34.919 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019893999999999999 2020-01-16T00:13:34.919 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.920 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.920 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:34.920 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.594251+0000", 2020-01-16T00:13:34.920 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0071632730000000004 2020-01-16T00:13:34.920 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.921 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.921 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.921 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601414+0000", 2020-01-16T00:13:34.921 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4672999999999996e-05 2020-01-16T00:13:34.921 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.921 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.922 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.922 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601479+0000", 2020-01-16T00:13:34.922 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000158453 2020-01-16T00:13:34.922 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.922 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.922 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:34.923 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601638+0000", 2020-01-16T00:13:34.923 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8605e-05 2020-01-16T00:13:34.923 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.923 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.923 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:34.924 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601686+0000", 2020-01-16T00:13:34.924 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4283999999999998e-05 2020-01-16T00:13:34.924 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.924 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.924 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.925 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601730+0000", 2020-01-16T00:13:34.925 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016699964000000001 2020-01-16T00:13:34.925 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.925 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.925 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.925 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.925 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.926 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:126 2.1 2:a4df40e2:::smithi12313934-42 low_tier:head [write 70656~727552] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e45)", 2020-01-16T00:13:34.926 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.627078+0000", 2020-01-16T00:13:34.926 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.1104754019999996, 2020-01-16T00:13:34.926 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010098995, 2020-01-16T00:13:34.926 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.926 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:34.926 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.926 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.927 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.927 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627078+0000", 2020-01-16T00:13:34.927 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.927 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.927 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.927 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.927 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627078+0000", 2020-01-16T00:13:34.927 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:34.928 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.928 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.928 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.928 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627076+0000", 2020-01-16T00:13:34.928 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019321599999999999 2020-01-16T00:13:34.928 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.928 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.929 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.929 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627269+0000", 2020-01-16T00:13:34.929 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8471999999999999e-05 2020-01-16T00:13:34.929 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.929 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.929 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.929 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627288+0000", 2020-01-16T00:13:34.929 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3199999999999996e-06 2020-01-16T00:13:34.930 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.930 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.930 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.930 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627294+0000", 2020-01-16T00:13:34.930 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3341000000000002e-05 2020-01-16T00:13:34.930 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.930 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.930 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.931 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627357+0000", 2020-01-16T00:13:34.931 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010985 2020-01-16T00:13:34.931 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.931 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.931 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.931 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627467+0000", 2020-01-16T00:13:34.931 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022914499999999999 2020-01-16T00:13:34.932 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.932 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.932 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:34.932 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627696+0000", 2020-01-16T00:13:34.932 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0055120000000000004 2020-01-16T00:13:34.932 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.932 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.932 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:34.933 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.633208+0000", 2020-01-16T00:13:34.933 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0019968569999999999 2020-01-16T00:13:34.933 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.933 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.933 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.933 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.635205+0000", 2020-01-16T00:13:34.933 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018113160000000001 2020-01-16T00:13:34.933 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.934 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.934 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.934 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.637016+0000", 2020-01-16T00:13:34.934 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9515e-05 2020-01-16T00:13:34.934 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.934 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.934 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:34.935 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.637096+0000", 2020-01-16T00:13:34.935 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1213999999999999e-05 2020-01-16T00:13:34.935 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.935 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.935 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.936 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.637177+0000", 2020-01-16T00:13:34.936 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010098995 2020-01-16T00:13:34.936 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.936 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.936 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.936 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.936 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.936 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:127 2.1 2:a4df40e2:::smithi12313934-42 low_tier:head [write 1364992~1145344] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e45)", 2020-01-16T00:13:34.937 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.627308+0000", 2020-01-16T00:13:34.937 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.1102456060000003, 2020-01-16T00:13:34.937 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011068883999999999, 2020-01-16T00:13:34.937 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.937 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:34.937 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.937 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.938 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.938 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627308+0000", 2020-01-16T00:13:34.938 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.938 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.938 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.938 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.938 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627308+0000", 2020-01-16T00:13:34.938 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:34.939 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.939 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.939 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.939 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627307+0000", 2020-01-16T00:13:34.939 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000278597 2020-01-16T00:13:34.939 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.939 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.939 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.940 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627586+0000", 2020-01-16T00:13:34.940 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9169999999999997e-06 2020-01-16T00:13:34.940 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.940 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.940 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.940 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627591+0000", 2020-01-16T00:13:34.940 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0479999999999999e-06 2020-01-16T00:13:34.940 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.941 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.941 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.941 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627594+0000", 2020-01-16T00:13:34.941 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030987269999999999 2020-01-16T00:13:34.941 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.941 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.941 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.942 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.630693+0000", 2020-01-16T00:13:34.942 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000103688 2020-01-16T00:13:34.942 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.942 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.942 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.942 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.630796+0000", 2020-01-16T00:13:34.943 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000144606 2020-01-16T00:13:34.943 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.943 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.943 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:34.943 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.630941+0000", 2020-01-16T00:13:34.943 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039264269999999997 2020-01-16T00:13:34.943 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.944 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.944 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:34.944 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.634867+0000", 2020-01-16T00:13:34.944 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00038581200000000003 2020-01-16T00:13:34.944 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.945 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.945 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.945 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.635253+0000", 2020-01-16T00:13:34.945 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030568879999999998 2020-01-16T00:13:34.945 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.945 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.945 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.946 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.638310+0000", 2020-01-16T00:13:34.946 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8390999999999999e-05 2020-01-16T00:13:34.946 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.946 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.946 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:34.946 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.638348+0000", 2020-01-16T00:13:34.946 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8402999999999999e-05 2020-01-16T00:13:34.947 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.947 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.947 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.947 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.638377+0000", 2020-01-16T00:13:34.947 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011068883999999999 2020-01-16T00:13:34.947 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.947 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.948 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.948 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.948 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.948 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:129 2.1 2:a4df40e2:::smithi12313934-42 low_tier:head [write 1364992~1145344] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e45)", 2020-01-16T00:13:34.948 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:30.748851+0000", 2020-01-16T00:13:34.948 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 3.9887022399999998, 2020-01-16T00:13:34.948 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0091922689999999994, 2020-01-16T00:13:34.948 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:34.949 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:34.949 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:34.949 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.949 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:34.949 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748851+0000", 2020-01-16T00:13:34.949 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:34.949 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.949 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.950 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:34.950 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748851+0000", 2020-01-16T00:13:34.950 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:34.950 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.950 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.950 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:34.950 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748848+0000", 2020-01-16T00:13:34.951 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00037828800000000001 2020-01-16T00:13:34.951 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.951 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.951 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:34.951 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.749226+0000", 2020-01-16T00:13:34.951 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3696000000000001e-05 2020-01-16T00:13:34.951 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.952 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.952 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:34.952 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.749250+0000", 2020-01-16T00:13:34.952 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0899e-05 2020-01-16T00:13:34.952 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.952 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.952 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:34.953 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.749261+0000", 2020-01-16T00:13:34.953 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0016265450000000001 2020-01-16T00:13:34.953 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.953 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:34.953 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.750887+0000", 2020-01-16T00:13:34.953 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015506800000000001 2020-01-16T00:13:34.953 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.954 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.954 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:34.954 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.751043+0000", 2020-01-16T00:13:34.954 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000180206 2020-01-16T00:13:34.954 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.954 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.954 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:34.954 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.751223+0000", 2020-01-16T00:13:34.955 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0024072910000000002 2020-01-16T00:13:34.955 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.955 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.955 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:34.955 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.753630+0000", 2020-01-16T00:13:34.955 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012397969999999999 2020-01-16T00:13:34.955 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.955 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.956 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.956 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.754870+0000", 2020-01-16T00:13:34.956 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003021648 2020-01-16T00:13:34.956 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.956 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.956 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:34.957 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.757892+0000", 2020-01-16T00:13:34.957 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1149999999999996e-05 2020-01-16T00:13:34.957 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.957 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.957 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:34.958 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.757953+0000", 2020-01-16T00:13:34.958 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0906e-05 2020-01-16T00:13:34.958 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:34.958 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:34.958 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:34.959 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.758044+0000", 2020-01-16T00:13:34.959 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0091922689999999994 2020-01-16T00:13:34.959 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.959 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.959 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.959 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:34.959 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:34.959 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:34.962 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-01-16T00:13:34.963 INFO:tasks.rados.rados.0.smithi123.stdout:1326: finishing write tid 1 to smithi12313934-29 2020-01-16T00:13:34.963 INFO:tasks.rados.rados.0.smithi123.stdout:1326: finishing write tid 2 to smithi12313934-29 2020-01-16T00:13:34.963 INFO:tasks.rados.rados.0.smithi123.stdout:1326: finishing write tid 4 to smithi12313934-29 2020-01-16T00:13:34.963 INFO:tasks.rados.rados.0.smithi123.stdout:1326: finishing write tid 5 to smithi12313934-29 2020-01-16T00:13:34.963 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 29 v 132 (ObjNum 414 snap 0 seq_num 414) dirty exists 2020-01-16T00:13:34.964 INFO:tasks.rados.rados.0.smithi123.stdout:1326: left oid 29 (ObjNum 414 snap 0 seq_num 414) 2020-01-16T00:13:34.964 INFO:tasks.rados.rados.0.smithi123.stdout:1340: read oid 39 snap -1 2020-01-16T00:13:34.964 INFO:tasks.rados.rados.0.smithi123.stdout:1340: expect (ObjNum 403 snap 0 seq_num 403) 2020-01-16T00:13:34.964 INFO:tasks.rados.rados.0.smithi123.stdout:1324: done (15 left) 2020-01-16T00:13:34.964 INFO:tasks.rados.rados.0.smithi123.stdout:1325: done (14 left) 2020-01-16T00:13:34.964 INFO:tasks.rados.rados.0.smithi123.stdout:1326: done (13 left) 2020-01-16T00:13:34.964 INFO:tasks.rados.rados.0.smithi123.stdout:1341: read oid 28 snap -1 2020-01-16T00:13:34.964 INFO:tasks.rados.rados.0.smithi123.stdout:1341: expect (ObjNum 357 snap 0 seq_num 357) 2020-01-16T00:13:34.965 INFO:tasks.rados.rados.0.smithi123.stdout:1342: read oid 43 snap -1 2020-01-16T00:13:34.965 INFO:tasks.rados.rados.0.smithi123.stdout:1342: expect deleted 2020-01-16T00:13:34.965 INFO:tasks.rados.rados.0.smithi123.stdout:1343: write (excl) oid 36 current snap is 0 2020-01-16T00:13:34.965 INFO:tasks.rados.rados.0.smithi123.stdout:1343: seq_num 422 ranges {692554=554493,1778107=679805,2889444=203110} 2020-01-16T00:13:34.965 INFO:tasks.rados.rados.0.smithi123.stdout:1343: writing smithi12313934-36 from 692554 to 1247047 tid 1 2020-01-16T00:13:34.965 INFO:tasks.rados.rados.0.smithi123.stdout:1343: writing smithi12313934-36 from 1778107 to 2457912 tid 2 2020-01-16T00:13:34.965 INFO:tasks.rados.rados.0.smithi123.stdout:1343: writing smithi12313934-36 from 2889444 to 3092554 tid 3 2020-01-16T00:13:34.966 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:34.966 INFO:tasks.rados.rados.0.smithi123.stdout:1328: finishing write tid 1 to smithi12313934-45 2020-01-16T00:13:34.966 INFO:tasks.rados.rados.0.smithi123.stdout:1327: expect (ObjNum 342 snap 0 seq_num 342) 2020-01-16T00:13:34.966 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:34.956+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:35.065 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:35.062+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:35.080 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:35.081 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:13:35.081 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:13:35.081 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:35.091 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-01-16T00:13:35.159 INFO:tasks.rados.rados.0.smithi123.stdout:1328: finishing write tid 2 to smithi12313934-45 2020-01-16T00:13:35.159 INFO:tasks.rados.rados.0.smithi123.stdout:1328: finishing write tid 3 to smithi12313934-45 2020-01-16T00:13:35.159 INFO:tasks.rados.rados.0.smithi123.stdout:1328: finishing write tid 5 to smithi12313934-45 2020-01-16T00:13:35.159 INFO:tasks.rados.rados.0.smithi123.stdout:1328: finishing write tid 6 to smithi12313934-45 2020-01-16T00:13:35.159 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 45 v 141 (ObjNum 415 snap 0 seq_num 415) dirty exists 2020-01-16T00:13:35.159 INFO:tasks.rados.rados.0.smithi123.stdout:1328: left oid 45 (ObjNum 415 snap 0 seq_num 415) 2020-01-16T00:13:35.159 INFO:tasks.rados.rados.0.smithi123.stdout:1329: expect (ObjNum 405 snap 0 seq_num 405) 2020-01-16T00:13:35.166 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:35.160+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:35.202 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:35.202 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:13:35.203 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:13:35.203 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:13:35.203 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:35.212 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-01-16T00:13:35.267 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:35.262+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:35.362 INFO:tasks.rados.rados.0.smithi123.stdout:1331: expect (ObjNum 298 snap 0 seq_num 298) 2020-01-16T00:13:35.367 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:35.366+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:35.369 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:35.370 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:13:35.370 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:13:35.370 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:13:35.370 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.371 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:943 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:35.371 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:13:35.371 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 57.450082291000001, 2020-01-16T00:13:35.371 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018847004000000001, 2020-01-16T00:13:35.371 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:35.372 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:35.372 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:35.372 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:35.372 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:35.372 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 943 2020-01-16T00:13:35.373 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.373 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:35.373 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.373 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:35.374 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:13:35.374 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:35.374 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.374 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.375 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:35.375 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:13:35.375 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:35.375 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.375 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.376 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:35.376 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914475+0000", 2020-01-16T00:13:35.376 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.508e-05 2020-01-16T00:13:35.376 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.377 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.377 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:35.377 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914490+0000", 2020-01-16T00:13:35.377 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.857e-05 2020-01-16T00:13:35.377 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.378 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.378 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:35.378 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914509+0000", 2020-01-16T00:13:35.378 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.46e-06 2020-01-16T00:13:35.379 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.379 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.379 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:35.379 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914515+0000", 2020-01-16T00:13:35.380 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2612999999999999e-05 2020-01-16T00:13:35.380 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.380 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.380 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.380 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914568+0000", 2020-01-16T00:13:35.381 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028312299999999998 2020-01-16T00:13:35.381 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.381 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.381 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:35.382 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914851+0000", 2020-01-16T00:13:35.382 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012282533 2020-01-16T00:13:35.382 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.382 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.382 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.383 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927133+0000", 2020-01-16T00:13:35.383 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4839999999999995e-05 2020-01-16T00:13:35.383 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.383 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.384 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:35.384 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927208+0000", 2020-01-16T00:13:35.384 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0047425610000000002 2020-01-16T00:13:35.384 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.384 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.385 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.385 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.931951+0000", 2020-01-16T00:13:35.385 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3628999999999994e-05 2020-01-16T00:13:35.385 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.386 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.386 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:35.386 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.932045+0000", 2020-01-16T00:13:35.386 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012810180000000001 2020-01-16T00:13:35.386 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.387 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.387 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:35.387 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933326+0000", 2020-01-16T00:13:35.387 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018847004000000001 2020-01-16T00:13:35.388 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.388 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:35.388 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.388 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.388 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.389 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:944 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:35.389 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:13:35.389 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 57.450032735000001, 2020-01-16T00:13:35.389 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019086901999999999, 2020-01-16T00:13:35.390 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:35.390 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:35.390 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:35.390 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:35.391 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:35.391 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 944 2020-01-16T00:13:35.391 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.391 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:35.391 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.392 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:35.392 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:13:35.392 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:35.392 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.393 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.393 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:35.393 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:13:35.393 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:35.394 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.394 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.394 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:35.394 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914527+0000", 2020-01-16T00:13:35.394 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7509999999999999e-06 2020-01-16T00:13:35.395 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.395 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.395 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:35.395 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914532+0000", 2020-01-16T00:13:35.396 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5149999999999997e-06 2020-01-16T00:13:35.396 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.396 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.396 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:35.396 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914538+0000", 2020-01-16T00:13:35.397 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1890000000000007e-06 2020-01-16T00:13:35.397 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.397 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.397 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:35.398 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914546+0000", 2020-01-16T00:13:35.398 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000354951 2020-01-16T00:13:35.398 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.398 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.398 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.399 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914901+0000", 2020-01-16T00:13:35.399 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1594000000000002e-05 2020-01-16T00:13:35.399 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.399 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.400 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:35.400 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914942+0000", 2020-01-16T00:13:35.400 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012300070999999999 2020-01-16T00:13:35.400 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.400 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.401 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.401 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927242+0000", 2020-01-16T00:13:35.401 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4076999999999997e-05 2020-01-16T00:13:35.401 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.402 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.402 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:35.402 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927296+0000", 2020-01-16T00:13:35.402 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0060609180000000002 2020-01-16T00:13:35.402 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.403 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.403 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.403 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933357+0000", 2020-01-16T00:13:35.403 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4488999999999999e-05 2020-01-16T00:13:35.404 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.404 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.404 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:35.404 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933422+0000", 2020-01-16T00:13:35.404 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019318300000000001 2020-01-16T00:13:35.405 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.405 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.405 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:35.405 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933615+0000", 2020-01-16T00:13:35.406 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019086901999999999 2020-01-16T00:13:35.406 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.406 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:35.406 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.406 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.407 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.407 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:945 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:35.407 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:13:35.407 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 57.450005529000002, 2020-01-16T00:13:35.408 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019342986, 2020-01-16T00:13:35.408 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:35.408 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:35.408 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:35.408 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:35.409 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:35.409 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 945 2020-01-16T00:13:35.409 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.409 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:35.410 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.410 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:35.410 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:13:35.410 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:35.411 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.411 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.411 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:35.411 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:13:35.411 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:35.412 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.412 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.412 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:35.412 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:13:35.413 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4580000000000002e-06 2020-01-16T00:13:35.413 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.413 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.413 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:35.414 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914558+0000", 2020-01-16T00:13:35.414 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7829999999999998e-06 2020-01-16T00:13:35.414 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.414 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.414 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:35.414 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914563+0000", 2020-01-16T00:13:35.415 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0001e-05 2020-01-16T00:13:35.415 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.415 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.415 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:35.416 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914573+0000", 2020-01-16T00:13:35.416 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00040365600000000001 2020-01-16T00:13:35.416 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.416 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.416 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.417 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914976+0000", 2020-01-16T00:13:35.417 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1390000000000001e-05 2020-01-16T00:13:35.417 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.417 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.418 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:35.418 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.915028+0000", 2020-01-16T00:13:35.418 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012288843000000001 2020-01-16T00:13:35.418 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.418 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.418 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.419 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927317+0000", 2020-01-16T00:13:35.419 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7871000000000002e-05 2020-01-16T00:13:35.419 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.419 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.419 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:35.419 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927375+0000", 2020-01-16T00:13:35.420 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0062580589999999998 2020-01-16T00:13:35.420 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.420 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.420 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.420 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933633+0000", 2020-01-16T00:13:35.420 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7086999999999997e-05 2020-01-16T00:13:35.421 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.421 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.421 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:35.421 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933690+0000", 2020-01-16T00:13:35.421 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020855799999999999 2020-01-16T00:13:35.422 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.422 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.422 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:35.422 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933898+0000", 2020-01-16T00:13:35.422 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019342986 2020-01-16T00:13:35.422 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.422 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:35.423 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.423 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.423 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.423 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:981 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:35.423 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:13:35.423 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 57.049228636999999, 2020-01-16T00:13:35.423 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021222494000000001, 2020-01-16T00:13:35.423 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:35.424 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:35.424 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:35.424 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:35.424 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:35.424 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 981 2020-01-16T00:13:35.424 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.424 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:35.424 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.425 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:35.425 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:13:35.425 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:35.425 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.425 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.425 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:35.426 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:13:35.426 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999948 2020-01-16T00:13:35.426 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.426 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.427 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:35.427 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315327+0000", 2020-01-16T00:13:35.427 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.508e-05 2020-01-16T00:13:35.427 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.427 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.427 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:35.428 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315352+0000", 2020-01-16T00:13:35.428 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1183999999999998e-05 2020-01-16T00:13:35.428 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.428 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.428 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:35.428 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315383+0000", 2020-01-16T00:13:35.428 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7180000000000002e-05 2020-01-16T00:13:35.429 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.429 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.429 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:35.429 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315401+0000", 2020-01-16T00:13:35.429 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017290599999999999 2020-01-16T00:13:35.429 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.429 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.429 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.430 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315574+0000", 2020-01-16T00:13:35.430 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00047908100000000002 2020-01-16T00:13:35.430 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.430 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.430 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:35.430 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316053+0000", 2020-01-16T00:13:35.430 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011808947 2020-01-16T00:13:35.431 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.431 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.431 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.431 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.327862+0000", 2020-01-16T00:13:35.431 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010067499999999999 2020-01-16T00:13:35.431 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.431 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.431 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:35.432 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.327962+0000", 2020-01-16T00:13:35.432 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0074745139999999998 2020-01-16T00:13:35.432 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.432 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.432 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.432 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.335437+0000", 2020-01-16T00:13:35.432 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000103403 2020-01-16T00:13:35.432 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.433 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.433 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:35.433 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.335540+0000", 2020-01-16T00:13:35.433 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001014544 2020-01-16T00:13:35.433 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.433 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.433 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:35.434 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336555+0000", 2020-01-16T00:13:35.434 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021222494000000001 2020-01-16T00:13:35.434 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.434 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:35.434 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.434 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.434 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.434 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:982 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:35.435 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:13:35.435 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 57.049126113, 2020-01-16T00:13:35.435 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.022087721000000001, 2020-01-16T00:13:35.435 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:35.435 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:35.435 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:35.436 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:35.436 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:35.436 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 982 2020-01-16T00:13:35.436 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.436 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:35.436 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.436 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:35.437 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:13:35.437 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:35.437 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.437 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.437 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:35.437 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:13:35.438 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:35.438 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.438 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.438 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:35.438 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315433+0000", 2020-01-16T00:13:35.438 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1347e-05 2020-01-16T00:13:35.438 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.438 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.439 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:35.439 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315445+0000", 2020-01-16T00:13:35.439 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4525999999999999e-05 2020-01-16T00:13:35.439 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.439 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.439 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:35.439 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315459+0000", 2020-01-16T00:13:35.439 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.024e-05 2020-01-16T00:13:35.440 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.440 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.440 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:35.440 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315470+0000", 2020-01-16T00:13:35.440 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000736468 2020-01-16T00:13:35.440 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.440 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.440 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.441 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316206+0000", 2020-01-16T00:13:35.441 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000132967 2020-01-16T00:13:35.441 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.441 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.441 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:35.441 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316339+0000", 2020-01-16T00:13:35.441 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011701717 2020-01-16T00:13:35.442 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.442 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.442 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.442 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328041+0000", 2020-01-16T00:13:35.442 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000120339 2020-01-16T00:13:35.443 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.443 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.443 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:35.443 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328161+0000", 2020-01-16T00:13:35.443 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0084341260000000001 2020-01-16T00:13:35.444 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.444 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.444 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.444 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336595+0000", 2020-01-16T00:13:35.444 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5247999999999993e-05 2020-01-16T00:13:35.445 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.445 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.445 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:35.445 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336690+0000", 2020-01-16T00:13:35.445 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00083204000000000004 2020-01-16T00:13:35.445 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.446 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.446 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:35.446 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337522+0000", 2020-01-16T00:13:35.446 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.022087721000000001 2020-01-16T00:13:35.446 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.446 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:35.446 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.446 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.447 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.447 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:983 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:35.447 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:13:35.447 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 57.049006403, 2020-01-16T00:13:35.447 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023363387999999999, 2020-01-16T00:13:35.448 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:35.448 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:35.448 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:35.448 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:35.448 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:35.448 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 983 2020-01-16T00:13:35.449 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.449 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:35.449 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.449 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:35.449 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:13:35.449 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:35.449 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.449 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.450 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:35.450 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:13:35.450 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:35.450 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.450 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.451 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:35.451 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315553+0000", 2020-01-16T00:13:35.451 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1760000000000004e-06 2020-01-16T00:13:35.451 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.451 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.451 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:35.451 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315560+0000", 2020-01-16T00:13:35.452 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3587000000000001e-05 2020-01-16T00:13:35.452 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.452 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.452 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:35.452 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315574+0000", 2020-01-16T00:13:35.452 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7800000000000003e-06 2020-01-16T00:13:35.452 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.452 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.453 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:35.453 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315581+0000", 2020-01-16T00:13:35.453 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000823024 2020-01-16T00:13:35.453 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.453 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.453 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.453 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316404+0000", 2020-01-16T00:13:35.453 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2413e-05 2020-01-16T00:13:35.454 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.454 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.454 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:35.454 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316466+0000", 2020-01-16T00:13:35.454 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011738561999999999 2020-01-16T00:13:35.454 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.454 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.455 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.455 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328205+0000", 2020-01-16T00:13:35.455 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6783999999999997e-05 2020-01-16T00:13:35.455 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.455 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.455 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:35.455 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328262+0000", 2020-01-16T00:13:35.455 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0093022560000000001 2020-01-16T00:13:35.456 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.456 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.456 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.456 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337564+0000", 2020-01-16T00:13:35.456 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000183151 2020-01-16T00:13:35.456 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.456 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.457 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:35.457 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337747+0000", 2020-01-16T00:13:35.457 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011707449999999999 2020-01-16T00:13:35.457 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.457 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.457 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:35.457 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.338918+0000", 2020-01-16T00:13:35.458 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023363387999999999 2020-01-16T00:13:35.458 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.458 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:35.458 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.458 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.459 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.459 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:986 3.9 3:93dfe6f1:::smithi12313934-40:head [write 2941380~419987] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:35.459 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:13:35.459 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 56.883706627000002, 2020-01-16T00:13:35.459 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016978400000000001, 2020-01-16T00:13:35.459 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:35.459 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:35.459 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:35.460 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:35.460 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:35.460 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 986 2020-01-16T00:13:35.460 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.460 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:35.460 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.460 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:35.461 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:13:35.461 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:35.461 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.461 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.461 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:35.462 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:13:35.462 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:35.462 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.462 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.462 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:35.462 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480852+0000", 2020-01-16T00:13:35.463 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042503899999999999 2020-01-16T00:13:35.463 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.463 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.463 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:35.463 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481277+0000", 2020-01-16T00:13:35.463 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3097e-05 2020-01-16T00:13:35.463 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.463 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.464 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:35.464 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481290+0000", 2020-01-16T00:13:35.464 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8460000000000002e-06 2020-01-16T00:13:35.464 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.464 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.464 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:35.464 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481297+0000", 2020-01-16T00:13:35.464 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8443999999999999e-05 2020-01-16T00:13:35.465 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.465 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.465 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.465 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481346+0000", 2020-01-16T00:13:35.465 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022115399999999999 2020-01-16T00:13:35.465 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.465 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.465 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:35.466 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481567+0000", 2020-01-16T00:13:35.466 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012531502 2020-01-16T00:13:35.466 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.466 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.466 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.466 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494098+0000", 2020-01-16T00:13:35.466 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.4462999999999993e-05 2020-01-16T00:13:35.467 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.467 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.467 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:35.467 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494193+0000", 2020-01-16T00:13:35.467 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017600999999999999 2020-01-16T00:13:35.467 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.467 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.467 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:13:35.468 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494369+0000", 2020-01-16T00:13:35.468 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0031739619999999998 2020-01-16T00:13:35.468 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.468 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.468 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:35.468 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497543+0000", 2020-01-16T00:13:35.468 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000175107 2020-01-16T00:13:35.469 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.469 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.469 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:35.469 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497718+0000", 2020-01-16T00:13:35.469 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5207000000000003e-05 2020-01-16T00:13:35.469 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.470 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.470 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:35.470 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497763+0000", 2020-01-16T00:13:35.470 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7042999999999998e-05 2020-01-16T00:13:35.471 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.471 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.471 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:35.471 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497800+0000", 2020-01-16T00:13:35.471 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2628e-05 2020-01-16T00:13:35.472 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.472 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.472 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:35.472 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497833+0000", 2020-01-16T00:13:35.472 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016978400000000001 2020-01-16T00:13:35.472 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.472 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:35.473 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.473 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.473 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.473 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:987 3.9 3:93dfe6f1:::smithi12313934-40:head [setxattr _header (56) in=7b,truncate 3361367] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:35.473 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.481307+0000", 2020-01-16T00:13:35.473 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 56.883254260000001, 2020-01-16T00:13:35.473 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016794989, 2020-01-16T00:13:35.474 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:35.474 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:35.474 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:35.474 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:35.474 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:35.474 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 987 2020-01-16T00:13:35.474 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.474 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:35.475 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.475 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:35.475 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481307+0000", 2020-01-16T00:13:35.475 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:35.475 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.475 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.475 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:35.475 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481307+0000", 2020-01-16T00:13:35.476 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:35.476 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.476 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.476 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:35.476 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481306+0000", 2020-01-16T00:13:35.476 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.113e-06 2020-01-16T00:13:35.477 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.477 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.477 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:35.477 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481311+0000", 2020-01-16T00:13:35.477 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1520000000000002e-06 2020-01-16T00:13:35.477 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.478 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.478 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:35.478 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481315+0000", 2020-01-16T00:13:35.478 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.7640000000000001e-06 2020-01-16T00:13:35.478 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.478 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.478 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:35.479 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481318+0000", 2020-01-16T00:13:35.479 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00029012500000000002 2020-01-16T00:13:35.479 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.479 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.479 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.480 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481608+0000", 2020-01-16T00:13:35.480 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6239000000000002e-05 2020-01-16T00:13:35.480 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.480 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.480 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:35.481 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481654+0000", 2020-01-16T00:13:35.481 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014559534000000001 2020-01-16T00:13:35.481 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.481 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.481 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.481 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.496214+0000", 2020-01-16T00:13:35.482 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000225932 2020-01-16T00:13:35.482 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.482 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.482 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:35.482 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.496439+0000", 2020-01-16T00:13:35.482 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00031099600000000002 2020-01-16T00:13:35.482 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.483 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.483 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:13:35.483 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.496750+0000", 2020-01-16T00:13:35.483 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00098608599999999995 2020-01-16T00:13:35.483 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.483 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.483 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:35.484 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497737+0000", 2020-01-16T00:13:35.484 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022081499999999999 2020-01-16T00:13:35.484 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.484 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.484 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:35.484 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497957+0000", 2020-01-16T00:13:35.484 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7071000000000003e-05 2020-01-16T00:13:35.484 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.485 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.485 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:35.485 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498014+0000", 2020-01-16T00:13:35.485 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1111e-05 2020-01-16T00:13:35.485 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.485 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.485 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:35.485 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498056+0000", 2020-01-16T00:13:35.486 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5974999999999999e-05 2020-01-16T00:13:35.486 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.486 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.486 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:35.486 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498102+0000", 2020-01-16T00:13:35.486 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016794989 2020-01-16T00:13:35.486 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.487 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:35.487 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.487 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.487 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.487 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:988 3.9 3:93dfe6f1:::smithi12313934-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:13:35.487 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:13:35.487 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 56.883221163000002, 2020-01-16T00:13:35.487 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017221692, 2020-01-16T00:13:35.488 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:35.488 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:35.488 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:35.488 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:35.488 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:35.488 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 988 2020-01-16T00:13:35.488 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.489 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:35.489 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.489 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:35.489 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:13:35.489 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:35.490 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.490 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.490 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:35.490 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:13:35.490 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:35.491 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.491 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.491 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:35.491 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481339+0000", 2020-01-16T00:13:35.491 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3089999999999997e-06 2020-01-16T00:13:35.492 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.492 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.492 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:35.492 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481343+0000", 2020-01-16T00:13:35.492 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2230000000000004e-06 2020-01-16T00:13:35.492 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.493 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.493 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:35.493 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481350+0000", 2020-01-16T00:13:35.493 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.319e-06 2020-01-16T00:13:35.493 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.493 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.493 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:35.494 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481356+0000", 2020-01-16T00:13:35.494 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000321544 2020-01-16T00:13:35.494 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.494 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.494 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.494 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481678+0000", 2020-01-16T00:13:35.494 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6375999999999999e-05 2020-01-16T00:13:35.494 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.495 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.495 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:35.495 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481714+0000", 2020-01-16T00:13:35.495 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015557666 2020-01-16T00:13:35.495 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.495 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.495 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.495 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497272+0000", 2020-01-16T00:13:35.496 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012791300000000001 2020-01-16T00:13:35.496 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.496 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.496 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:35.496 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497399+0000", 2020-01-16T00:13:35.496 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00076944600000000002 2020-01-16T00:13:35.496 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.496 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.497 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.497 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498169+0000", 2020-01-16T00:13:35.497 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018306799999999999 2020-01-16T00:13:35.497 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.497 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.497 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:35.497 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498352+0000", 2020-01-16T00:13:35.498 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020935999999999999 2020-01-16T00:13:35.498 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.498 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.498 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:35.498 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498561+0000", 2020-01-16T00:13:35.498 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017221692 2020-01-16T00:13:35.498 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.498 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:35.499 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.499 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.499 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.499 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1239 3.3 3:c204aa4f:::smithi12313934-45:head [setxattr _header (56) in=7b,truncate 2000033] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:13:35.499 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:13:35.499 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 52.774147343999999, 2020-01-16T00:13:35.499 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019161661999999999, 2020-01-16T00:13:35.499 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:35.500 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:35.500 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:35.500 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:35.500 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:35.500 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1239 2020-01-16T00:13:35.501 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.501 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:35.501 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.501 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:35.501 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:13:35.502 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:35.502 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.502 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.502 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:35.503 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:13:35.503 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:35.503 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.503 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.503 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:35.504 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590411+0000", 2020-01-16T00:13:35.504 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3879999999999999e-05 2020-01-16T00:13:35.504 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.504 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.505 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:35.505 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590425+0000", 2020-01-16T00:13:35.505 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6542999999999998e-05 2020-01-16T00:13:35.505 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.506 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.506 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:35.506 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590471+0000", 2020-01-16T00:13:35.506 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8907999999999999e-05 2020-01-16T00:13:35.506 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.507 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.507 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:35.507 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590490+0000", 2020-01-16T00:13:35.507 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0045691960000000002 2020-01-16T00:13:35.507 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.507 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.507 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.507 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.595059+0000", 2020-01-16T00:13:35.508 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0035638940000000002 2020-01-16T00:13:35.508 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.508 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.508 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:35.508 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598623+0000", 2020-01-16T00:13:35.508 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0077558310000000004 2020-01-16T00:13:35.509 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.509 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.509 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.509 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606379+0000", 2020-01-16T00:13:35.509 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012171999999999999 2020-01-16T00:13:35.509 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.509 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.509 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:35.510 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606501+0000", 2020-01-16T00:13:35.510 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013144200000000001 2020-01-16T00:13:35.510 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.510 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.510 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2020-01-16T00:13:35.510 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606632+0000", 2020-01-16T00:13:35.510 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001288565 2020-01-16T00:13:35.511 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.511 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.511 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:35.511 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607921+0000", 2020-01-16T00:13:35.511 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033399299999999999 2020-01-16T00:13:35.512 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.512 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.512 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:35.512 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.608255+0000", 2020-01-16T00:13:35.512 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012280870000000001 2020-01-16T00:13:35.512 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.513 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.513 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:35.513 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609483+0000", 2020-01-16T00:13:35.513 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2286000000000001e-05 2020-01-16T00:13:35.514 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.514 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.514 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:35.514 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609525+0000", 2020-01-16T00:13:35.514 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0121999999999999e-05 2020-01-16T00:13:35.514 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.514 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.514 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:35.515 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609575+0000", 2020-01-16T00:13:35.515 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019161661999999999 2020-01-16T00:13:35.515 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.515 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:35.515 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.515 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.515 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.516 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1240 3.3 3:c204aa4f:::smithi12313934-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-01-16T00:13:35.516 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:13:35.516 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 52.774047154999998, 2020-01-16T00:13:35.516 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019268828000000002, 2020-01-16T00:13:35.516 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:35.516 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:35.516 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:35.516 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:35.517 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:35.517 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1240 2020-01-16T00:13:35.517 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.517 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:35.517 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.517 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:35.517 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:13:35.517 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:35.518 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.518 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.518 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:35.518 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:13:35.518 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:35.518 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.519 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.519 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:35.519 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590512+0000", 2020-01-16T00:13:35.519 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0525e-05 2020-01-16T00:13:35.519 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.519 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.519 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:35.520 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590522+0000", 2020-01-16T00:13:35.520 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6418000000000001e-05 2020-01-16T00:13:35.520 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.520 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.520 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:35.520 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590539+0000", 2020-01-16T00:13:35.520 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0679999999999999e-05 2020-01-16T00:13:35.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.521 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.521 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:35.521 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590549+0000", 2020-01-16T00:13:35.521 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0083626789999999996 2020-01-16T00:13:35.522 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.522 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.522 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.522 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598912+0000", 2020-01-16T00:13:35.522 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013344 2020-01-16T00:13:35.522 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.522 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.523 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:35.523 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599045+0000", 2020-01-16T00:13:35.523 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0084002269999999997 2020-01-16T00:13:35.523 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.523 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.524 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.524 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607446+0000", 2020-01-16T00:13:35.524 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000109873 2020-01-16T00:13:35.524 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.525 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.525 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:35.525 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607555+0000", 2020-01-16T00:13:35.525 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0020573309999999999 2020-01-16T00:13:35.525 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.525 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.525 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.526 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609613+0000", 2020-01-16T00:13:35.526 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6351e-05 2020-01-16T00:13:35.526 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.526 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.526 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:35.526 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609699+0000", 2020-01-16T00:13:35.526 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3374999999999994e-05 2020-01-16T00:13:35.526 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.527 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.527 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:35.527 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609782+0000", 2020-01-16T00:13:35.527 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019268828000000002 2020-01-16T00:13:35.527 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.527 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:35.527 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.528 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.528 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.528 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1437 3.3 3:c204aa4f:::smithi12313934-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e28)", 2020-01-16T00:13:35.528 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:47.900594+0000", 2020-01-16T00:13:35.528 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 47.463966610999996, 2020-01-16T00:13:35.529 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016745237999999999, 2020-01-16T00:13:35.529 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:35.529 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:35.529 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:35.529 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:35.529 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:35.529 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1437 2020-01-16T00:13:35.529 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.530 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:35.530 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.530 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:35.530 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900594+0000", 2020-01-16T00:13:35.530 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:35.530 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.530 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.531 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:35.531 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900594+0000", 2020-01-16T00:13:35.531 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:35.531 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.531 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.531 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:35.531 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900593+0000", 2020-01-16T00:13:35.532 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5059999999999995e-06 2020-01-16T00:13:35.532 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.532 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.532 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:35.532 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900601+0000", 2020-01-16T00:13:35.532 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5077e-05 2020-01-16T00:13:35.532 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.532 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.533 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:35.533 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900616+0000", 2020-01-16T00:13:35.533 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0183000000000001e-05 2020-01-16T00:13:35.533 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.533 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.533 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:35.533 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900627+0000", 2020-01-16T00:13:35.534 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0063667660000000003 2020-01-16T00:13:35.534 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.534 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.534 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.534 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.906993+0000", 2020-01-16T00:13:35.534 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8988e-05 2020-01-16T00:13:35.535 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.535 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.535 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:35.535 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.907032+0000", 2020-01-16T00:13:35.535 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0080128850000000008 2020-01-16T00:13:35.535 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.535 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.535 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.536 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.915045+0000", 2020-01-16T00:13:35.536 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013200899999999999 2020-01-16T00:13:35.536 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.536 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.536 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:35.536 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.915177+0000", 2020-01-16T00:13:35.536 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0018373090000000001 2020-01-16T00:13:35.536 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.537 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.537 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.537 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.917015+0000", 2020-01-16T00:13:35.537 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016701699999999999 2020-01-16T00:13:35.537 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.537 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.537 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:35.538 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.917182+0000", 2020-01-16T00:13:35.538 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015782400000000001 2020-01-16T00:13:35.538 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.538 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.538 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:35.539 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.917339+0000", 2020-01-16T00:13:35.539 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016745237999999999 2020-01-16T00:13:35.539 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.539 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:35.539 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.540 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.540 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.540 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3124 3.9 3:9fc687b2:::smithi12313934-44:head [write 1903628~435462] snapc 0=[] ondisk+write+known_if_redirected e44)", 2020-01-16T00:13:35.540 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.276145+0000", 2020-01-16T00:13:35.540 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 10.088415434, 2020-01-16T00:13:35.540 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017361898000000001, 2020-01-16T00:13:35.540 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:35.540 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:35.541 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:35.541 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:35.541 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:35.541 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3124 2020-01-16T00:13:35.542 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.542 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:35.542 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.542 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:35.542 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276145+0000", 2020-01-16T00:13:35.542 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:35.543 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.543 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.543 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:35.543 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276145+0000", 2020-01-16T00:13:35.543 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:35.543 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.543 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.543 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:35.544 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276143+0000", 2020-01-16T00:13:35.544 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042136099999999998 2020-01-16T00:13:35.544 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.544 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.545 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:35.545 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276564+0000", 2020-01-16T00:13:35.545 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.349e-05 2020-01-16T00:13:35.545 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.545 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.545 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:35.545 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276578+0000", 2020-01-16T00:13:35.546 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2469999999999999e-06 2020-01-16T00:13:35.546 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.546 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.546 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:35.546 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276583+0000", 2020-01-16T00:13:35.546 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4400000000000001e-05 2020-01-16T00:13:35.546 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.547 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.547 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.547 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276637+0000", 2020-01-16T00:13:35.547 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0046361789999999998 2020-01-16T00:13:35.547 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.547 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.547 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:35.547 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281274+0000", 2020-01-16T00:13:35.548 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0090025520000000005 2020-01-16T00:13:35.548 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.548 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.548 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.548 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.290276+0000", 2020-01-16T00:13:35.548 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011584200000000001 2020-01-16T00:13:35.548 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.548 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.549 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:35.549 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.290392+0000", 2020-01-16T00:13:35.549 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00025563099999999998 2020-01-16T00:13:35.549 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.549 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.549 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:13:35.549 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.290648+0000", 2020-01-16T00:13:35.550 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0026741360000000001 2020-01-16T00:13:35.550 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.550 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.550 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:35.550 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293322+0000", 2020-01-16T00:13:35.550 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9764000000000002e-05 2020-01-16T00:13:35.550 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.551 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.551 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:35.551 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293421+0000", 2020-01-16T00:13:35.551 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4779000000000001e-05 2020-01-16T00:13:35.551 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.551 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.551 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:35.551 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293446+0000", 2020-01-16T00:13:35.552 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4628000000000001e-05 2020-01-16T00:13:35.552 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.552 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.552 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:35.552 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293481+0000", 2020-01-16T00:13:35.552 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.6444999999999998e-05 2020-01-16T00:13:35.553 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.553 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.553 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:35.553 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293507+0000", 2020-01-16T00:13:35.553 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017361898000000001 2020-01-16T00:13:35.553 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.553 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:35.554 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.554 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.554 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.554 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3125 3.9 3:9fc687b2:::smithi12313934-44:head [setxattr _header (56) in=7b,truncate 2386660] snapc 0=[] ondisk+write+known_if_redirected e44)", 2020-01-16T00:13:35.554 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.276594+0000", 2020-01-16T00:13:35.554 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 10.087966661999999, 2020-01-16T00:13:35.554 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017760560000000002, 2020-01-16T00:13:35.555 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:35.555 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:35.555 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:35.555 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:35.555 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:35.556 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3125 2020-01-16T00:13:35.556 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.556 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:35.556 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.556 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:35.557 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276594+0000", 2020-01-16T00:13:35.557 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:35.557 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.557 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.558 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:35.558 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276594+0000", 2020-01-16T00:13:35.558 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:35.558 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.559 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.559 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:35.559 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276593+0000", 2020-01-16T00:13:35.559 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7320000000000003e-06 2020-01-16T00:13:35.559 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.559 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.560 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:35.560 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276599+0000", 2020-01-16T00:13:35.560 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3302000000000001e-05 2020-01-16T00:13:35.560 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.560 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.560 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:35.561 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276612+0000", 2020-01-16T00:13:35.561 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1659999999999998e-06 2020-01-16T00:13:35.561 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.561 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.561 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:35.561 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276615+0000", 2020-01-16T00:13:35.561 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0047328450000000003 2020-01-16T00:13:35.561 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.562 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.562 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.562 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281348+0000", 2020-01-16T00:13:35.562 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3966000000000003e-05 2020-01-16T00:13:35.563 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.563 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.563 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:35.563 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281422+0000", 2020-01-16T00:13:35.563 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01082613 2020-01-16T00:13:35.563 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.563 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.564 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.564 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.292248+0000", 2020-01-16T00:13:35.564 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000148617 2020-01-16T00:13:35.564 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.564 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.564 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:35.565 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.292397+0000", 2020-01-16T00:13:35.565 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019936300000000001 2020-01-16T00:13:35.565 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.565 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.565 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:13:35.565 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.292596+0000", 2020-01-16T00:13:35.565 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011208870000000001 2020-01-16T00:13:35.566 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.566 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.566 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:35.566 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293717+0000", 2020-01-16T00:13:35.566 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00039491599999999998 2020-01-16T00:13:35.566 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.567 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.567 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:35.567 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294112+0000", 2020-01-16T00:13:35.567 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000129326 2020-01-16T00:13:35.567 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.567 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.567 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:35.567 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294241+0000", 2020-01-16T00:13:35.568 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5586000000000001e-05 2020-01-16T00:13:35.568 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.568 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.568 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:35.568 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294277+0000", 2020-01-16T00:13:35.568 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7893999999999997e-05 2020-01-16T00:13:35.568 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.568 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.569 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:35.569 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294355+0000", 2020-01-16T00:13:35.569 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017760560000000002 2020-01-16T00:13:35.569 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.569 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:35.569 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.569 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.570 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.570 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3126 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e44)", 2020-01-16T00:13:35.570 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.276634+0000", 2020-01-16T00:13:35.570 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 10.087926947, 2020-01-16T00:13:35.570 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017942583000000002, 2020-01-16T00:13:35.570 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:35.571 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:35.571 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:35.571 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:35.571 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:35.571 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3126 2020-01-16T00:13:35.571 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.571 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:35.572 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.572 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:35.572 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276634+0000", 2020-01-16T00:13:35.572 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:35.572 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.572 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.572 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:35.573 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276634+0000", 2020-01-16T00:13:35.573 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:35.573 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.573 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.573 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:35.573 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276633+0000", 2020-01-16T00:13:35.573 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0769999999999999e-06 2020-01-16T00:13:35.574 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.574 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.574 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:35.574 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276636+0000", 2020-01-16T00:13:35.574 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5390000000000001e-06 2020-01-16T00:13:35.574 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.574 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.574 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:35.575 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276641+0000", 2020-01-16T00:13:35.575 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9969999999999999e-06 2020-01-16T00:13:35.575 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.575 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.576 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:35.576 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276644+0000", 2020-01-16T00:13:35.576 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0048246970000000002 2020-01-16T00:13:35.576 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.576 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.577 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.577 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281469+0000", 2020-01-16T00:13:35.577 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8723999999999999e-05 2020-01-16T00:13:35.577 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.577 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.578 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:35.578 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281567+0000", 2020-01-16T00:13:35.578 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011633447 2020-01-16T00:13:35.578 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.579 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.579 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.579 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293201+0000", 2020-01-16T00:13:35.579 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6796000000000002e-05 2020-01-16T00:13:35.579 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.580 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.580 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:35.580 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293268+0000", 2020-01-16T00:13:35.580 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001125247 2020-01-16T00:13:35.580 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.581 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.581 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.581 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294393+0000", 2020-01-16T00:13:35.581 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.6082000000000006e-05 2020-01-16T00:13:35.581 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.581 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.582 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:35.582 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294489+0000", 2020-01-16T00:13:35.582 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7515000000000005e-05 2020-01-16T00:13:35.582 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.582 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.583 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:35.583 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294576+0000", 2020-01-16T00:13:35.583 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017942583000000002 2020-01-16T00:13:35.583 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.583 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:35.583 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.584 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.584 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.584 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3204 3.9 3:93dfe6f1:::smithi12313934-40:head [setxattr _header (56) in=7b,truncate 2554730] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:13:35.584 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.568544+0000", 2020-01-16T00:13:35.585 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.7960167269999996, 2020-01-16T00:13:35.585 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.026260867, 2020-01-16T00:13:35.585 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:35.585 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:35.585 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:35.586 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:35.586 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:35.586 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3204 2020-01-16T00:13:35.586 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.586 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:35.586 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.587 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:35.587 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568544+0000", 2020-01-16T00:13:35.587 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:35.587 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.587 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.587 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:35.588 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568544+0000", 2020-01-16T00:13:35.588 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:35.588 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.588 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.588 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:35.588 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568543+0000", 2020-01-16T00:13:35.588 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0810000000000001e-06 2020-01-16T00:13:35.589 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.589 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.589 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:35.589 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568548+0000", 2020-01-16T00:13:35.589 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6950000000000002e-06 2020-01-16T00:13:35.589 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.589 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.590 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:35.590 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568552+0000", 2020-01-16T00:13:35.590 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.875e-06 2020-01-16T00:13:35.590 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.590 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.590 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:35.590 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568555+0000", 2020-01-16T00:13:35.591 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0033130849999999999 2020-01-16T00:13:35.591 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.591 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.591 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.591 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.571868+0000", 2020-01-16T00:13:35.591 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0026014609999999998 2020-01-16T00:13:35.592 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.592 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.592 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:35.592 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.574469+0000", 2020-01-16T00:13:35.592 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010261381999999999 2020-01-16T00:13:35.592 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.592 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.592 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.593 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.584731+0000", 2020-01-16T00:13:35.593 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019704 2020-01-16T00:13:35.593 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.593 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.593 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:35.593 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.584928+0000", 2020-01-16T00:13:35.593 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000280869 2020-01-16T00:13:35.594 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.594 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.594 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:13:35.594 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.585209+0000", 2020-01-16T00:13:35.594 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002273859 2020-01-16T00:13:35.594 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.594 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.594 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:35.595 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.587483+0000", 2020-01-16T00:13:35.595 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3356000000000001e-05 2020-01-16T00:13:35.595 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.595 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.595 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:35.595 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.587506+0000", 2020-01-16T00:13:35.596 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0071864939999999999 2020-01-16T00:13:35.596 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.596 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.596 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:35.597 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594692+0000", 2020-01-16T00:13:35.597 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5475000000000001e-05 2020-01-16T00:13:35.597 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.597 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.597 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:35.598 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594738+0000", 2020-01-16T00:13:35.598 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7070000000000007e-05 2020-01-16T00:13:35.598 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.598 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.598 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:35.598 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594805+0000", 2020-01-16T00:13:35.599 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.026260867 2020-01-16T00:13:35.599 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.599 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:35.599 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.599 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.599 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.599 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3205 3.9 3:93dfe6f1:::smithi12313934-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e45)", 2020-01-16T00:13:35.599 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.568599+0000", 2020-01-16T00:13:35.600 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.7959617919999999, 2020-01-16T00:13:35.600 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.026406882, 2020-01-16T00:13:35.600 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:35.600 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:35.600 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:35.600 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:35.600 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:35.601 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3205 2020-01-16T00:13:35.601 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.601 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:35.601 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.601 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:35.602 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568599+0000", 2020-01-16T00:13:35.602 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:35.602 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.602 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.602 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:35.603 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568599+0000", 2020-01-16T00:13:35.603 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:35.603 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.604 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.604 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:35.604 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568598+0000", 2020-01-16T00:13:35.604 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3939999999999998e-06 2020-01-16T00:13:35.604 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.604 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.604 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:35.605 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568603+0000", 2020-01-16T00:13:35.605 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9391000000000001e-05 2020-01-16T00:13:35.605 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.605 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.605 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:35.605 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568622+0000", 2020-01-16T00:13:35.605 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9599999999999997e-06 2020-01-16T00:13:35.606 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.606 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.606 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:35.606 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568628+0000", 2020-01-16T00:13:35.606 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0058679700000000001 2020-01-16T00:13:35.606 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.607 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.607 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.607 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.574496+0000", 2020-01-16T00:13:35.607 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5528999999999998e-05 2020-01-16T00:13:35.607 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.607 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.607 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:35.608 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.574532+0000", 2020-01-16T00:13:35.608 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011986749 2020-01-16T00:13:35.608 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.608 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.608 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.609 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.586518+0000", 2020-01-16T00:13:35.609 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016522600000000001 2020-01-16T00:13:35.609 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.609 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.610 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:35.610 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.586684+0000", 2020-01-16T00:13:35.610 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0081693890000000009 2020-01-16T00:13:35.610 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.610 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.611 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.611 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594853+0000", 2020-01-16T00:13:35.611 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7082000000000004e-05 2020-01-16T00:13:35.611 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.612 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.612 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:35.612 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594940+0000", 2020-01-16T00:13:35.612 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5789000000000003e-05 2020-01-16T00:13:35.613 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.613 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.613 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:35.613 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.595006+0000", 2020-01-16T00:13:35.613 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.026406882 2020-01-16T00:13:35.614 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.614 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:35.614 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.614 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.614 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.614 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3219 3.3 3:c478bfe8:::smithi12313934-10:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:13:35.614 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.580957+0000", 2020-01-16T00:13:35.614 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.7836042760000002, 2020-01-16T00:13:35.615 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016807971000000001, 2020-01-16T00:13:35.615 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:35.615 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:35.615 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:35.615 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:35.615 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:35.616 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3219 2020-01-16T00:13:35.616 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.616 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:35.616 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.616 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:35.616 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580957+0000", 2020-01-16T00:13:35.616 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:35.617 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.617 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.617 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:35.617 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580957+0000", 2020-01-16T00:13:35.617 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:35.617 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.618 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.618 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:35.618 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580955+0000", 2020-01-16T00:13:35.618 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5120000000000002e-06 2020-01-16T00:13:35.618 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.618 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.618 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:35.619 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580963+0000", 2020-01-16T00:13:35.619 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6799999999999993e-06 2020-01-16T00:13:35.619 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.619 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.619 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:35.620 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580971+0000", 2020-01-16T00:13:35.620 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3109999999999996e-06 2020-01-16T00:13:35.620 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.620 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.620 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:35.620 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580976+0000", 2020-01-16T00:13:35.620 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014306146 2020-01-16T00:13:35.620 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.621 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.621 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.621 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.595282+0000", 2020-01-16T00:13:35.621 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.1883000000000001e-05 2020-01-16T00:13:35.621 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.621 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.622 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:35.622 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.595374+0000", 2020-01-16T00:13:35.622 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023904989999999999 2020-01-16T00:13:35.622 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.622 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.623 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:35.623 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.597765+0000", 2020-01-16T00:13:35.623 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016807971000000001 2020-01-16T00:13:35.623 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.623 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:35.623 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.623 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.623 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.624 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3220 3.3 3:c478bfe8:::smithi12313934-10:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:13:35.624 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.580982+0000", 2020-01-16T00:13:35.624 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.7835792279999998, 2020-01-16T00:13:35.624 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017920330000000002, 2020-01-16T00:13:35.624 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:35.624 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:35.624 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:35.625 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:35.625 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:35.625 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3220 2020-01-16T00:13:35.625 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.625 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:35.625 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.625 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:35.626 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580982+0000", 2020-01-16T00:13:35.626 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:35.626 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.626 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.626 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:35.626 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580982+0000", 2020-01-16T00:13:35.627 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:35.627 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.627 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.627 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:35.627 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580981+0000", 2020-01-16T00:13:35.627 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0380000000000001e-06 2020-01-16T00:13:35.627 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.627 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.628 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:35.628 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580985+0000", 2020-01-16T00:13:35.628 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8929000000000001e-05 2020-01-16T00:13:35.628 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.628 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.628 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:35.628 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581004+0000", 2020-01-16T00:13:35.628 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7339999999999999e-06 2020-01-16T00:13:35.629 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.629 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.629 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:35.629 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581009+0000", 2020-01-16T00:13:35.629 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01679812 2020-01-16T00:13:35.629 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.630 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.630 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.630 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.597807+0000", 2020-01-16T00:13:35.630 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013333400000000001 2020-01-16T00:13:35.630 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.630 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.630 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:35.631 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.597940+0000", 2020-01-16T00:13:35.631 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000961899 2020-01-16T00:13:35.631 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.631 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.631 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:35.631 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.598902+0000", 2020-01-16T00:13:35.631 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017920330000000002 2020-01-16T00:13:35.631 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.632 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:35.632 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.632 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.632 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.632 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3221 3.3 3:c478bfe8:::smithi12313934-10:head [read 0~2403467,checksum] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:13:35.632 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.581014+0000", 2020-01-16T00:13:35.632 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.7835468790000002, 2020-01-16T00:13:35.632 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018784551999999999, 2020-01-16T00:13:35.633 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:35.633 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:35.633 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:35.633 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:35.633 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:35.633 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3221 2020-01-16T00:13:35.633 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.634 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:35.634 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.634 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:35.634 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581014+0000", 2020-01-16T00:13:35.634 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:35.634 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.634 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.634 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:35.635 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581014+0000", 2020-01-16T00:13:35.635 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:35.635 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.635 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.635 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:35.635 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581013+0000", 2020-01-16T00:13:35.635 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9390000000000002e-06 2020-01-16T00:13:35.635 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.636 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.636 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:35.636 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581019+0000", 2020-01-16T00:13:35.636 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1131000000000001e-05 2020-01-16T00:13:35.636 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.636 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.636 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:35.636 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581030+0000", 2020-01-16T00:13:35.637 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.55e-06 2020-01-16T00:13:35.637 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.637 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.637 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:35.637 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581037+0000", 2020-01-16T00:13:35.637 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017890321000000001 2020-01-16T00:13:35.637 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.638 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.638 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:35.638 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.598927+0000", 2020-01-16T00:13:35.638 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000105999 2020-01-16T00:13:35.638 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.638 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.638 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:35.639 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.599033+0000", 2020-01-16T00:13:35.639 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00076542400000000001 2020-01-16T00:13:35.639 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:35.639 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:35.639 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:35.640 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.599798+0000", 2020-01-16T00:13:35.640 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018784551999999999 2020-01-16T00:13:35.640 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.640 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:35.640 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.640 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:35.640 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:35.641 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:35.642 INFO:tasks.rados.rados.0.smithi123.stdout:1333: expect (ObjNum 383 snap 0 seq_num 383) 2020-01-16T00:13:35.642 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:35.636+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:35.643 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-01-16T00:13:35.653 INFO:tasks.rados.rados.0.smithi123.stdout:1332: finishing write tid 1 to smithi12313934-16 2020-01-16T00:13:35.654 INFO:tasks.rados.rados.0.smithi123.stdout:1332: finishing write tid 2 to smithi12313934-16 2020-01-16T00:13:35.654 INFO:tasks.rados.rados.0.smithi123.stdout:1332: finishing write tid 4 to smithi12313934-16 2020-01-16T00:13:35.654 INFO:tasks.rados.rados.0.smithi123.stdout:1332: finishing write tid 5 to smithi12313934-16 2020-01-16T00:13:35.654 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 16 v 201 (ObjNum 416 snap 0 seq_num 416) dirty exists 2020-01-16T00:13:35.654 INFO:tasks.rados.rados.0.smithi123.stdout:1332: left oid 16 (ObjNum 416 snap 0 seq_num 416) 2020-01-16T00:13:35.654 INFO:tasks.rados.rados.0.smithi123.stdout:1335: finishing write tid 1 to smithi12313934-8 2020-01-16T00:13:35.654 INFO:tasks.rados.rados.0.smithi123.stdout:1335: finishing write tid 2 to smithi12313934-8 2020-01-16T00:13:35.655 INFO:tasks.rados.rados.0.smithi123.stdout:1335: finishing write tid 4 to smithi12313934-8 2020-01-16T00:13:35.655 INFO:tasks.rados.rados.0.smithi123.stdout:1335: finishing write tid 5 to smithi12313934-8 2020-01-16T00:13:35.655 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 8 v 95 (ObjNum 418 snap 0 seq_num 418) dirty exists 2020-01-16T00:13:35.655 INFO:tasks.rados.rados.0.smithi123.stdout:1335: left oid 8 (ObjNum 418 snap 0 seq_num 418) 2020-01-16T00:13:35.655 INFO:tasks.rados.rados.0.smithi123.stdout:1336: finishing write tid 1 to smithi12313934-11 2020-01-16T00:13:35.655 INFO:tasks.rados.rados.0.smithi123.stdout:1336: finishing write tid 2 to smithi12313934-11 2020-01-16T00:13:35.655 INFO:tasks.rados.rados.0.smithi123.stdout:1327: done (15 left) 2020-01-16T00:13:35.655 INFO:tasks.rados.rados.0.smithi123.stdout:1328: done (14 left) 2020-01-16T00:13:35.656 INFO:tasks.rados.rados.0.smithi123.stdout:1329: done (13 left) 2020-01-16T00:13:35.656 INFO:tasks.rados.rados.0.smithi123.stdout:1330: done (12 left) 2020-01-16T00:13:35.656 INFO:tasks.rados.rados.0.smithi123.stdout:1331: done (11 left) 2020-01-16T00:13:35.656 INFO:tasks.rados.rados.0.smithi123.stdout:1332: done (10 left) 2020-01-16T00:13:35.656 INFO:tasks.rados.rados.0.smithi123.stdout:1333: done (9 left) 2020-01-16T00:13:35.656 INFO:tasks.rados.rados.0.smithi123.stdout:1335: done (8 left) 2020-01-16T00:13:35.657 INFO:tasks.rados.rados.0.smithi123.stdout:1344: read oid 38 snap -1 2020-01-16T00:13:35.657 INFO:tasks.rados.rados.0.smithi123.stdout:1344: expect (ObjNum 298 snap 0 seq_num 298) 2020-01-16T00:13:35.657 INFO:tasks.rados.rados.0.smithi123.stdout:1345: delete oid 21 current snap is 0 2020-01-16T00:13:35.657 INFO:tasks.rados.rados.0.smithi123.stdout:1336: finishing write tid 4 to smithi12313934-11 2020-01-16T00:13:35.657 INFO:tasks.rados.rados.0.smithi123.stdout:1336: finishing write tid 5 to smithi12313934-11 2020-01-16T00:13:35.657 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 11 v 117 (ObjNum 419 snap 0 seq_num 419) dirty exists 2020-01-16T00:13:35.657 INFO:tasks.rados.rados.0.smithi123.stdout:1336: left oid 11 (ObjNum 419 snap 0 seq_num 419) 2020-01-16T00:13:35.658 INFO:tasks.rados.rados.0.smithi123.stdout:1337: finishing write tid 1 to smithi12313934-26 2020-01-16T00:13:35.658 INFO:tasks.rados.rados.0.smithi123.stdout:1337: finishing write tid 2 to smithi12313934-26 2020-01-16T00:13:35.658 INFO:tasks.rados.rados.0.smithi123.stdout:1338: expect (ObjNum 380 snap 0 seq_num 380) 2020-01-16T00:13:35.743 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:35.738+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:35.800 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:35.801 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:13:35.801 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:13:35.801 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:35.801 INFO:tasks.rados.rados.0.smithi123.stdout:1337: finishing write tid 3 to smithi12313934-26 2020-01-16T00:13:35.801 INFO:tasks.rados.rados.0.smithi123.stdout:1337: finishing write tid 5 to smithi12313934-26 2020-01-16T00:13:35.802 INFO:tasks.rados.rados.0.smithi123.stdout:1337: finishing write tid 6 to smithi12313934-26 2020-01-16T00:13:35.802 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 26 v 245 (ObjNum 420 snap 0 seq_num 420) dirty exists 2020-01-16T00:13:35.802 INFO:tasks.rados.rados.0.smithi123.stdout:1337: left oid 26 (ObjNum 420 snap 0 seq_num 420) 2020-01-16T00:13:35.802 INFO:tasks.rados.rados.0.smithi123.stdout:1339: finishing write tid 1 to smithi12313934-46 2020-01-16T00:13:35.802 INFO:tasks.rados.rados.0.smithi123.stdout:1339: finishing write tid 2 to smithi12313934-46 2020-01-16T00:13:35.802 INFO:tasks.rados.rados.0.smithi123.stdout:1339: finishing write tid 4 to smithi12313934-46 2020-01-16T00:13:35.802 INFO:tasks.rados.rados.0.smithi123.stdout:1339: finishing write tid 5 to smithi12313934-46 2020-01-16T00:13:35.803 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 46 v 249 (ObjNum 421 snap 0 seq_num 421) dirty exists 2020-01-16T00:13:35.803 INFO:tasks.rados.rados.0.smithi123.stdout:1339: left oid 46 (ObjNum 421 snap 0 seq_num 421) 2020-01-16T00:13:35.803 INFO:tasks.rados.rados.0.smithi123.stdout:1341: expect (ObjNum 357 snap 0 seq_num 357) 2020-01-16T00:13:35.810 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-01-16T00:13:35.844 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:35.842+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:35.901 INFO:tasks.rados.rados.0.smithi123.stdout:1340: expect (ObjNum 403 snap 0 seq_num 403) 2020-01-16T00:13:35.921 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:35.921 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:13:35.921 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:13:35.921 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:13:35.922 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:35.930 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-01-16T00:13:35.944 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:35.942+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:35.952 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2020-01-16T00:13:36.043 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:36.043 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:13:36.043 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:13:36.044 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:13:36.044 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.044 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.5.13:36 2.0 2:20e40cec:::smithi12313934-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 113152~1699840] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:13:36.044 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:13:36.044 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 58.206579713000004, 2020-01-16T00:13:36.044 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011736636, 2020-01-16T00:13:36.044 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:36.045 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:36.045 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:36.045 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.045 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:36.045 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:13:36.045 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:36.045 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.046 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.046 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:36.046 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:13:36.046 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999952 2020-01-16T00:13:36.046 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.046 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.046 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:36.046 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831934+0000", 2020-01-16T00:13:36.047 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001034325 2020-01-16T00:13:36.047 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.047 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.047 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:36.047 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.832968+0000", 2020-01-16T00:13:36.047 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.506e-05 2020-01-16T00:13:36.047 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.048 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.048 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:36.048 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833004+0000", 2020-01-16T00:13:36.048 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3406e-05 2020-01-16T00:13:36.048 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.048 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.048 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:36.048 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833017+0000", 2020-01-16T00:13:36.049 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015477000000000001 2020-01-16T00:13:36.049 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.049 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.049 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:36.049 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833172+0000", 2020-01-16T00:13:36.049 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033750700000000002 2020-01-16T00:13:36.050 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.050 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.050 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:36.050 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833509+0000", 2020-01-16T00:13:36.050 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00055861400000000003 2020-01-16T00:13:36.050 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.050 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.050 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:13:36.051 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.834068+0000", 2020-01-16T00:13:36.051 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.007474545 2020-01-16T00:13:36.051 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.051 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.051 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:36.051 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.841542+0000", 2020-01-16T00:13:36.051 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00056319499999999995 2020-01-16T00:13:36.051 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.052 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.052 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:36.052 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.842106+0000", 2020-01-16T00:13:36.052 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001371017 2020-01-16T00:13:36.052 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.052 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.052 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:36.053 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843477+0000", 2020-01-16T00:13:36.053 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000123938 2020-01-16T00:13:36.053 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.053 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.053 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:36.053 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843601+0000", 2020-01-16T00:13:36.054 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5012000000000006e-05 2020-01-16T00:13:36.054 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.054 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.054 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:36.054 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843676+0000", 2020-01-16T00:13:36.054 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011736636 2020-01-16T00:13:36.054 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.054 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:36.055 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.055 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.055 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.055 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1187 3.d 3:b1521a3e:::smithi12313934-25:head [write 1721042~423699] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:13:36.055 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.263828+0000", 2020-01-16T00:13:36.055 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 53.774690640999999, 2020-01-16T00:13:36.055 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011305709000000001, 2020-01-16T00:13:36.056 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:36.056 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:36.056 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:36.056 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:36.056 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:36.056 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1187 2020-01-16T00:13:36.056 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.056 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:36.057 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.057 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:36.057 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.263828+0000", 2020-01-16T00:13:36.057 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:36.057 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.057 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.057 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:36.058 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.263828+0000", 2020-01-16T00:13:36.058 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:13:36.058 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.058 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.058 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:36.058 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.263824+0000", 2020-01-16T00:13:36.058 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00048485899999999997 2020-01-16T00:13:36.058 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.059 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.059 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:36.059 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264309+0000", 2020-01-16T00:13:36.059 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5622999999999999e-05 2020-01-16T00:13:36.059 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.059 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.059 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:36.059 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264325+0000", 2020-01-16T00:13:36.060 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2530000000000004e-06 2020-01-16T00:13:36.060 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.060 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.060 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:36.060 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264330+0000", 2020-01-16T00:13:36.060 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000102252 2020-01-16T00:13:36.060 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.061 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.061 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:36.061 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264432+0000", 2020-01-16T00:13:36.061 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022361000000000001 2020-01-16T00:13:36.061 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.061 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.061 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:36.061 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264656+0000", 2020-01-16T00:13:36.062 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0028688849999999998 2020-01-16T00:13:36.062 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.062 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.062 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:36.062 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.267525+0000", 2020-01-16T00:13:36.062 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00023640900000000001 2020-01-16T00:13:36.062 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.062 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.063 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:36.063 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.267761+0000", 2020-01-16T00:13:36.063 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00035667000000000001 2020-01-16T00:13:36.063 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.063 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.063 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:36.063 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.268118+0000", 2020-01-16T00:13:36.064 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023898040000000001 2020-01-16T00:13:36.064 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.064 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.064 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:36.064 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.270507+0000", 2020-01-16T00:13:36.064 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014919690000000001 2020-01-16T00:13:36.064 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.064 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.065 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:36.065 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.271999+0000", 2020-01-16T00:13:36.065 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0029764510000000002 2020-01-16T00:13:36.065 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.065 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.065 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:36.066 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.274976+0000", 2020-01-16T00:13:36.066 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5409999999999997e-05 2020-01-16T00:13:36.066 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.066 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.066 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:36.066 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.275061+0000", 2020-01-16T00:13:36.066 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2452000000000004e-05 2020-01-16T00:13:36.067 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.067 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.067 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:36.067 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.275134+0000", 2020-01-16T00:13:36.067 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011305709000000001 2020-01-16T00:13:36.067 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.067 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:36.067 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.068 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.068 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.068 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1276 3.d 3:b1521a3e:::smithi12313934-25:head [read 0~3882384,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:13:36.068 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:13:36.068 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 52.963820143, 2020-01-16T00:13:36.068 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015839157999999999, 2020-01-16T00:13:36.068 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:36.069 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:36.069 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:36.069 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:36.069 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:36.069 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1276 2020-01-16T00:13:36.069 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.069 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:36.069 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.070 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:36.070 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:13:36.070 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:36.070 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.070 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.070 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:36.070 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:13:36.071 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:36.071 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.071 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.071 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:36.071 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:13:36.071 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5429999999999999e-06 2020-01-16T00:13:36.071 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.072 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.072 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:36.072 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074704+0000", 2020-01-16T00:13:36.072 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0839999999999992e-06 2020-01-16T00:13:36.072 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.072 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.072 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:36.072 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074712+0000", 2020-01-16T00:13:36.073 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.173e-06 2020-01-16T00:13:36.073 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.073 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.073 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:36.073 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074718+0000", 2020-01-16T00:13:36.073 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0083034700000000003 2020-01-16T00:13:36.073 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.074 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.074 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:36.074 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.083022+0000", 2020-01-16T00:13:36.074 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028456400000000001 2020-01-16T00:13:36.074 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.074 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.074 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:36.075 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.083306+0000", 2020-01-16T00:13:36.075 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0072312640000000003 2020-01-16T00:13:36.075 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.075 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.075 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:36.075 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090538+0000", 2020-01-16T00:13:36.075 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015839157999999999 2020-01-16T00:13:36.076 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.076 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:36.076 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.076 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.076 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.076 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1277 3.d 3:b1521a3e:::smithi12313934-25:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:13:36.076 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:13:36.077 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 52.963776553000002, 2020-01-16T00:13:36.077 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016646459999999998, 2020-01-16T00:13:36.077 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:36.077 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:36.077 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:36.077 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:36.077 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:36.077 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1277 2020-01-16T00:13:36.078 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.078 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:36.078 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.078 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:36.078 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:13:36.078 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:36.078 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.079 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.079 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:36.079 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:13:36.079 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:36.079 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.079 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.079 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:36.079 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074741+0000", 2020-01-16T00:13:36.080 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0776e-05 2020-01-16T00:13:36.080 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.080 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.080 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:36.080 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074752+0000", 2020-01-16T00:13:36.080 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7600000000000002e-06 2020-01-16T00:13:36.080 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.080 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.081 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:36.081 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074759+0000", 2020-01-16T00:13:36.081 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6840000000000001e-06 2020-01-16T00:13:36.081 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.081 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.081 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:36.081 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074765+0000", 2020-01-16T00:13:36.082 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015815216 2020-01-16T00:13:36.082 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.082 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.082 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:36.082 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090580+0000", 2020-01-16T00:13:36.082 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000182404 2020-01-16T00:13:36.082 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.082 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.083 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:36.083 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090763+0000", 2020-01-16T00:13:36.083 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00062579300000000003 2020-01-16T00:13:36.083 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.083 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.083 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:36.084 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091389+0000", 2020-01-16T00:13:36.084 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016646459999999998 2020-01-16T00:13:36.084 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.084 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:36.085 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.085 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.085 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.085 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1278 3.d 3:b1521a3e:::smithi12313934-25:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:13:36.085 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:13:36.086 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 52.963746839999999, 2020-01-16T00:13:36.086 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017994915, 2020-01-16T00:13:36.086 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:36.086 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:36.087 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:36.087 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:36.087 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:36.087 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1278 2020-01-16T00:13:36.087 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.088 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:36.088 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.088 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:36.088 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:13:36.088 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:36.089 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.089 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.089 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:36.089 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:13:36.089 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:36.089 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.090 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.090 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:36.090 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074771+0000", 2020-01-16T00:13:36.090 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4569999999999998e-06 2020-01-16T00:13:36.090 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.090 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.090 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:36.090 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074775+0000", 2020-01-16T00:13:36.091 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0330000000000003e-06 2020-01-16T00:13:36.091 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.091 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.091 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:36.091 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074782+0000", 2020-01-16T00:13:36.091 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9470000000000002e-06 2020-01-16T00:13:36.091 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.092 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.092 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:36.092 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074787+0000", 2020-01-16T00:13:36.092 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01666134 2020-01-16T00:13:36.092 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.092 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.092 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:36.092 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091449+0000", 2020-01-16T00:13:36.093 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000247731 2020-01-16T00:13:36.093 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.093 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.093 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:36.093 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091696+0000", 2020-01-16T00:13:36.093 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010703170000000001 2020-01-16T00:13:36.093 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.093 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.094 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:36.094 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.092767+0000", 2020-01-16T00:13:36.094 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017994915 2020-01-16T00:13:36.094 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.094 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:36.094 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.094 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.095 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.095 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1872 3.d 3:bd9fc72f:::smithi12313934-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e31)", 2020-01-16T00:13:36.095 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:56.384904+0000", 2020-01-16T00:13:36.095 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 39.653614161, 2020-01-16T00:13:36.095 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011110386999999999, 2020-01-16T00:13:36.095 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:36.095 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:36.095 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:36.096 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:36.096 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:36.096 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1872 2020-01-16T00:13:36.096 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.096 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:36.096 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.096 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:36.097 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384904+0000", 2020-01-16T00:13:36.097 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:36.097 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.097 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.097 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:36.097 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384904+0000", 2020-01-16T00:13:36.097 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:36.097 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.098 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.098 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:36.098 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384904+0000", 2020-01-16T00:13:36.098 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3879999999999998e-06 2020-01-16T00:13:36.098 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.098 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.098 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:36.098 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384906+0000", 2020-01-16T00:13:36.099 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7479999999999999e-06 2020-01-16T00:13:36.099 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.099 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.099 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:36.099 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384910+0000", 2020-01-16T00:13:36.099 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4609999999999998e-06 2020-01-16T00:13:36.099 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.100 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.100 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:36.100 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384913+0000", 2020-01-16T00:13:36.100 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00083872899999999997 2020-01-16T00:13:36.100 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.100 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.100 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:36.100 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.385751+0000", 2020-01-16T00:13:36.101 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3528999999999997e-05 2020-01-16T00:13:36.101 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.101 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.101 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:36.101 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.385785+0000", 2020-01-16T00:13:36.101 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0064799849999999997 2020-01-16T00:13:36.101 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.101 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.102 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:36.102 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.392265+0000", 2020-01-16T00:13:36.102 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010603399999999999 2020-01-16T00:13:36.102 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.102 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.102 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:36.102 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.392371+0000", 2020-01-16T00:13:36.103 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003330576 2020-01-16T00:13:36.103 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.103 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.103 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:36.105 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.395702+0000", 2020-01-16T00:13:36.105 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016566899999999999 2020-01-16T00:13:36.105 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.105 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.106 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:36.106 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.395867+0000", 2020-01-16T00:13:36.106 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014768999999999999 2020-01-16T00:13:36.106 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.106 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.106 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:36.106 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.396015+0000", 2020-01-16T00:13:36.107 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011110386999999999 2020-01-16T00:13:36.107 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.107 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:36.107 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.107 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.107 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.107 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.0.13:17 2.0 2:1afbc8ac:::smithi12313934-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 916992~596480] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e33)", 2020-01-16T00:13:36.107 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:01.647924+0000", 2020-01-16T00:13:36.108 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 34.390595069, 2020-01-16T00:13:36.108 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01120495, 2020-01-16T00:13:36.108 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:36.108 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:36.108 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:36.108 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.108 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:36.108 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.647924+0000", 2020-01-16T00:13:36.109 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:36.109 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.109 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.109 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:36.109 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.647924+0000", 2020-01-16T00:13:36.109 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:36.109 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.110 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.110 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:36.110 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.647923+0000", 2020-01-16T00:13:36.110 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00054779600000000003 2020-01-16T00:13:36.110 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.110 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.111 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:36.111 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.648470+0000", 2020-01-16T00:13:36.111 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2118e-05 2020-01-16T00:13:36.111 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.111 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.111 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:36.111 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.648483+0000", 2020-01-16T00:13:36.111 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5599999999999999e-06 2020-01-16T00:13:36.112 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.112 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.112 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:36.112 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.648489+0000", 2020-01-16T00:13:36.112 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3761000000000002e-05 2020-01-16T00:13:36.112 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.112 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.112 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:36.113 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.648543+0000", 2020-01-16T00:13:36.113 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0154e-05 2020-01-16T00:13:36.113 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.113 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.113 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:13:36.113 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.648553+0000", 2020-01-16T00:13:36.113 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0039950130000000004 2020-01-16T00:13:36.114 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.114 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.114 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:36.114 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.652548+0000", 2020-01-16T00:13:36.114 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014782599999999999 2020-01-16T00:13:36.114 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.114 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.114 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:36.115 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.652696+0000", 2020-01-16T00:13:36.115 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000212034 2020-01-16T00:13:36.115 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.115 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.115 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:13:36.115 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.652908+0000", 2020-01-16T00:13:36.115 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0047602879999999997 2020-01-16T00:13:36.115 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.116 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.116 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:36.116 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.657668+0000", 2020-01-16T00:13:36.116 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0006999999999997e-05 2020-01-16T00:13:36.116 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.116 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.116 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:36.117 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.657718+0000", 2020-01-16T00:13:36.117 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001303991 2020-01-16T00:13:36.117 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.117 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.117 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:36.117 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.659022+0000", 2020-01-16T00:13:36.117 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2401000000000001e-05 2020-01-16T00:13:36.117 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.118 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.118 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:36.118 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.659085+0000", 2020-01-16T00:13:36.118 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3877e-05 2020-01-16T00:13:36.118 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.118 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.118 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:36.118 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.659129+0000", 2020-01-16T00:13:36.119 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01120495 2020-01-16T00:13:36.119 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.119 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:36.119 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.119 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.119 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.119 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2260 3.d 3:b76659c4:::smithi12313934-47:head [write 601750~535546] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:13:36.120 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:06.050577+0000", 2020-01-16T00:13:36.120 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 29.987941524, 2020-01-16T00:13:36.120 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012415381, 2020-01-16T00:13:36.120 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:36.120 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:36.120 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:36.120 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:36.120 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:36.121 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2260 2020-01-16T00:13:36.121 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.121 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:36.121 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.121 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:36.121 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.050577+0000", 2020-01-16T00:13:36.121 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:36.121 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.122 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.122 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:36.122 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.050577+0000", 2020-01-16T00:13:36.122 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:36.122 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.122 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.122 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:36.123 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.050574+0000", 2020-01-16T00:13:36.123 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001523487 2020-01-16T00:13:36.123 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.123 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.123 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:36.123 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052097+0000", 2020-01-16T00:13:36.123 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3697e-05 2020-01-16T00:13:36.124 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.124 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.124 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:36.124 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052121+0000", 2020-01-16T00:13:36.124 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0070000000000003e-06 2020-01-16T00:13:36.124 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.124 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.124 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:36.125 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052130+0000", 2020-01-16T00:13:36.125 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4559000000000003e-05 2020-01-16T00:13:36.125 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.125 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.125 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:36.125 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052194+0000", 2020-01-16T00:13:36.125 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000165878 2020-01-16T00:13:36.126 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.126 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.126 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:36.126 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052360+0000", 2020-01-16T00:13:36.126 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018825499999999999 2020-01-16T00:13:36.126 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.126 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.126 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:36.127 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052549+0000", 2020-01-16T00:13:36.127 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002761782 2020-01-16T00:13:36.127 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.127 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.127 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:36.127 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.055310+0000", 2020-01-16T00:13:36.127 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00060385500000000004 2020-01-16T00:13:36.128 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.128 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.128 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:36.128 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.055914+0000", 2020-01-16T00:13:36.128 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0069733809999999999 2020-01-16T00:13:36.128 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.128 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.128 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:36.129 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.062888+0000", 2020-01-16T00:13:36.129 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2884999999999999e-05 2020-01-16T00:13:36.129 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.129 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.129 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:36.129 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.062940+0000", 2020-01-16T00:13:36.129 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1996999999999997e-05 2020-01-16T00:13:36.130 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.130 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.130 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:36.130 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.062992+0000", 2020-01-16T00:13:36.130 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012415381 2020-01-16T00:13:36.130 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.130 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:36.130 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.131 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.131 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.131 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2295 3.d 3:b1521a3e:::smithi12313934-25:head [write 1763838~505140] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:13:36.131 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:06.660258+0000", 2020-01-16T00:13:36.131 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 29.378260297000001, 2020-01-16T00:13:36.131 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011681408000000001, 2020-01-16T00:13:36.131 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:36.131 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:36.132 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:36.132 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:36.132 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:36.132 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2295 2020-01-16T00:13:36.132 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.132 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:36.132 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.132 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:36.133 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660258+0000", 2020-01-16T00:13:36.133 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:36.133 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.133 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.133 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:36.133 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660258+0000", 2020-01-16T00:13:36.133 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:36.134 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.134 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.134 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:36.134 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660256+0000", 2020-01-16T00:13:36.134 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00066501699999999995 2020-01-16T00:13:36.134 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.134 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.134 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:36.135 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660921+0000", 2020-01-16T00:13:36.135 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7339000000000001e-05 2020-01-16T00:13:36.135 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.135 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.135 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:36.135 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660938+0000", 2020-01-16T00:13:36.135 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0219999999999993e-06 2020-01-16T00:13:36.136 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.136 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.136 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:36.136 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660947+0000", 2020-01-16T00:13:36.136 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00094237200000000002 2020-01-16T00:13:36.136 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.136 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.136 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:36.137 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.661890+0000", 2020-01-16T00:13:36.137 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000117649 2020-01-16T00:13:36.137 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.137 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.137 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:36.137 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662007+0000", 2020-01-16T00:13:36.137 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00060662599999999998 2020-01-16T00:13:36.137 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.138 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.138 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:36.138 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662614+0000", 2020-01-16T00:13:36.138 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028424899999999998 2020-01-16T00:13:36.138 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.138 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.138 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:36.138 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662898+0000", 2020-01-16T00:13:36.139 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00054904599999999997 2020-01-16T00:13:36.139 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.139 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.139 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:36.139 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.663447+0000", 2020-01-16T00:13:36.139 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0051479990000000003 2020-01-16T00:13:36.139 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.140 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.140 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:36.140 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.668595+0000", 2020-01-16T00:13:36.140 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00080881799999999999 2020-01-16T00:13:36.140 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.140 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.140 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:36.140 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.669404+0000", 2020-01-16T00:13:36.141 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0024495290000000002 2020-01-16T00:13:36.141 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.141 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.141 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:36.141 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671854+0000", 2020-01-16T00:13:36.141 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6635000000000003e-05 2020-01-16T00:13:36.142 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.142 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.142 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:36.142 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671900+0000", 2020-01-16T00:13:36.142 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9567999999999998e-05 2020-01-16T00:13:36.142 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.142 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.142 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:36.143 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671940+0000", 2020-01-16T00:13:36.143 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011681408000000001 2020-01-16T00:13:36.143 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.143 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:36.143 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.143 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.143 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.144 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2296 3.d 3:b1521a3e:::smithi12313934-25:head [setxattr _header (56) in=7b,truncate 2268978] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:13:36.144 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:06.660958+0000", 2020-01-16T00:13:36.144 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 29.377560786, 2020-01-16T00:13:36.144 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011054015, 2020-01-16T00:13:36.144 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:36.144 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:36.145 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:36.145 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:36.145 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:36.145 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2296 2020-01-16T00:13:36.145 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.145 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:36.146 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.146 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:36.146 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660958+0000", 2020-01-16T00:13:36.146 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:36.146 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.146 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.146 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:36.146 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660958+0000", 2020-01-16T00:13:36.147 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:36.147 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.147 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.147 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:36.147 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660957+0000", 2020-01-16T00:13:36.147 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3699999999999995e-06 2020-01-16T00:13:36.147 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.147 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.148 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:36.148 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660965+0000", 2020-01-16T00:13:36.148 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3590000000000008e-06 2020-01-16T00:13:36.148 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.148 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.148 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:36.148 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660973+0000", 2020-01-16T00:13:36.149 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9880000000000001e-06 2020-01-16T00:13:36.149 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.149 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.149 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:36.149 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660979+0000", 2020-01-16T00:13:36.149 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001100136 2020-01-16T00:13:36.149 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.149 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.150 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:36.150 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662079+0000", 2020-01-16T00:13:36.150 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013484700000000001 2020-01-16T00:13:36.150 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.150 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.150 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:36.150 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662214+0000", 2020-01-16T00:13:36.151 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0048090349999999997 2020-01-16T00:13:36.151 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.151 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.151 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:36.151 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.667023+0000", 2020-01-16T00:13:36.151 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012664600000000001 2020-01-16T00:13:36.152 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.152 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.152 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:36.152 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.667150+0000", 2020-01-16T00:13:36.152 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000235911 2020-01-16T00:13:36.152 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.152 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.153 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:36.153 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.667386+0000", 2020-01-16T00:13:36.153 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0017131500000000001 2020-01-16T00:13:36.153 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.153 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.153 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:36.153 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.669099+0000", 2020-01-16T00:13:36.153 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001018866 2020-01-16T00:13:36.154 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.154 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.154 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:36.154 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.670118+0000", 2020-01-16T00:13:36.154 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001828585 2020-01-16T00:13:36.154 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.154 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.154 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:36.155 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671947+0000", 2020-01-16T00:13:36.155 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2214e-05 2020-01-16T00:13:36.155 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.155 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.155 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:36.155 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671979+0000", 2020-01-16T00:13:36.155 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3135e-05 2020-01-16T00:13:36.156 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.156 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.156 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:36.156 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.672012+0000", 2020-01-16T00:13:36.156 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011054015 2020-01-16T00:13:36.156 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.156 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:36.156 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.157 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.157 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.157 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2297 3.d 3:b1521a3e:::smithi12313934-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e35)", 2020-01-16T00:13:36.157 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:06.660986+0000", 2020-01-16T00:13:36.157 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 29.377532438999999, 2020-01-16T00:13:36.157 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011220677, 2020-01-16T00:13:36.157 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:36.157 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:36.158 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:36.158 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:36.158 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:36.158 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2297 2020-01-16T00:13:36.158 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.158 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:36.158 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.159 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:36.159 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660986+0000", 2020-01-16T00:13:36.159 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:36.159 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.159 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.159 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:36.159 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660986+0000", 2020-01-16T00:13:36.159 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:36.160 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.160 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.160 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:36.160 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660985+0000", 2020-01-16T00:13:36.160 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8760000000000004e-06 2020-01-16T00:13:36.160 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.160 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.160 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:36.161 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660990+0000", 2020-01-16T00:13:36.161 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.4190000000000002e-06 2020-01-16T00:13:36.161 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.161 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.161 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:36.161 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660999+0000", 2020-01-16T00:13:36.161 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7740000000000001e-06 2020-01-16T00:13:36.162 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.162 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.162 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:36.162 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.661004+0000", 2020-01-16T00:13:36.162 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0073783590000000001 2020-01-16T00:13:36.162 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.162 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.162 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:36.163 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.668383+0000", 2020-01-16T00:13:36.163 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014443 2020-01-16T00:13:36.163 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.163 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.163 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:36.163 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.668527+0000", 2020-01-16T00:13:36.163 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0035336539999999998 2020-01-16T00:13:36.163 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.164 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.164 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:36.164 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.672061+0000", 2020-01-16T00:13:36.164 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6445000000000005e-05 2020-01-16T00:13:36.164 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.164 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.164 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:36.165 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.672127+0000", 2020-01-16T00:13:36.165 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.9497000000000001e-05 2020-01-16T00:13:36.165 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.165 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.165 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:36.165 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.672207+0000", 2020-01-16T00:13:36.165 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011220677 2020-01-16T00:13:36.166 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.166 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:36.166 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.166 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.166 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.166 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.3.13:109 2.0 2:11fa79bc:::smithi12313934-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 331264~198656] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e42)", 2020-01-16T00:13:36.166 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:21.840075+0000", 2020-01-16T00:13:36.167 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.198444, 2020-01-16T00:13:36.167 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011558677999999999, 2020-01-16T00:13:36.167 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:36.167 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:36.167 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:36.167 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.167 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:36.167 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840075+0000", 2020-01-16T00:13:36.168 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:36.168 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.168 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.168 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:36.168 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840075+0000", 2020-01-16T00:13:36.168 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:36.168 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.168 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.169 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:36.169 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840073+0000", 2020-01-16T00:13:36.170 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033576299999999999 2020-01-16T00:13:36.170 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.170 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.170 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:36.171 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840409+0000", 2020-01-16T00:13:36.171 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2082999999999998e-05 2020-01-16T00:13:36.171 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.171 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.171 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:36.171 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840431+0000", 2020-01-16T00:13:36.171 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4410000000000003e-06 2020-01-16T00:13:36.172 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.172 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.172 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:36.172 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840436+0000", 2020-01-16T00:13:36.172 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000101016 2020-01-16T00:13:36.172 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.172 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.172 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:36.173 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840537+0000", 2020-01-16T00:13:36.173 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000117639 2020-01-16T00:13:36.173 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.173 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.173 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:36.174 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840654+0000", 2020-01-16T00:13:36.174 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000196162 2020-01-16T00:13:36.174 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.174 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.175 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:13:36.175 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840850+0000", 2020-01-16T00:13:36.175 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003319092 2020-01-16T00:13:36.175 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.176 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.176 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:36.176 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.844169+0000", 2020-01-16T00:13:36.176 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011349229999999999 2020-01-16T00:13:36.176 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.176 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.177 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:36.177 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.845304+0000", 2020-01-16T00:13:36.177 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0062517670000000001 2020-01-16T00:13:36.177 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.177 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.177 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:36.177 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851556+0000", 2020-01-16T00:13:36.178 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4138000000000003e-05 2020-01-16T00:13:36.178 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.178 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.178 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:36.178 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851600+0000", 2020-01-16T00:13:36.178 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2993000000000001e-05 2020-01-16T00:13:36.178 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.179 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.179 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:36.179 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851633+0000", 2020-01-16T00:13:36.179 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011558677999999999 2020-01-16T00:13:36.179 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.179 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:36.179 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.179 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.180 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.180 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.3.13:110 2.0 2:11fa79bc:::smithi12313934-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 752640~650240] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e42)", 2020-01-16T00:13:36.180 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:21.840457+0000", 2020-01-16T00:13:36.180 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.198061707999999, 2020-01-16T00:13:36.180 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011329271, 2020-01-16T00:13:36.180 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:36.180 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:36.181 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:36.181 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.181 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:36.181 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840457+0000", 2020-01-16T00:13:36.181 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:36.182 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.182 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.182 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:36.182 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840457+0000", 2020-01-16T00:13:36.182 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:36.182 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.182 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.182 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:36.183 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840456+0000", 2020-01-16T00:13:36.183 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00062607899999999998 2020-01-16T00:13:36.183 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.183 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.183 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:36.183 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841082+0000", 2020-01-16T00:13:36.183 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1239999999999999e-06 2020-01-16T00:13:36.183 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.184 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.184 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:36.184 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841090+0000", 2020-01-16T00:13:36.184 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7129999999999999e-06 2020-01-16T00:13:36.184 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.184 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.184 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:36.185 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841093+0000", 2020-01-16T00:13:36.185 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00075997800000000002 2020-01-16T00:13:36.185 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.185 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.185 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:36.185 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841853+0000", 2020-01-16T00:13:36.185 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0690000000000003e-05 2020-01-16T00:13:36.186 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.186 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.186 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:36.186 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841924+0000", 2020-01-16T00:13:36.186 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000115753 2020-01-16T00:13:36.187 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.187 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.187 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:13:36.187 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.842040+0000", 2020-01-16T00:13:36.187 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0021358890000000002 2020-01-16T00:13:36.187 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.187 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.187 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:36.188 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.844176+0000", 2020-01-16T00:13:36.188 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0042541870000000004 2020-01-16T00:13:36.188 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.188 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.188 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:36.188 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.848430+0000", 2020-01-16T00:13:36.188 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0032476929999999998 2020-01-16T00:13:36.189 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.189 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.189 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:36.189 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851677+0000", 2020-01-16T00:13:36.189 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0370000000000001e-05 2020-01-16T00:13:36.189 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.189 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.190 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:36.190 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851728+0000", 2020-01-16T00:13:36.190 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.834e-05 2020-01-16T00:13:36.190 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.190 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.190 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:36.190 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851786+0000", 2020-01-16T00:13:36.190 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011329271 2020-01-16T00:13:36.191 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.191 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:36.191 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.191 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.191 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.191 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2987 3.d 3:b3dd5d45:::smithi12313934-16:head [write 467327~633480] snapc 0=[] ondisk+write+known_if_redirected e42)", 2020-01-16T00:13:36.192 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:22.257904+0000", 2020-01-16T00:13:36.192 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 13.780614414, 2020-01-16T00:13:36.192 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012187450000000001, 2020-01-16T00:13:36.192 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:36.192 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:36.192 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:36.192 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:36.193 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:36.193 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2987 2020-01-16T00:13:36.193 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.193 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:36.193 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.193 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:36.194 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.257904+0000", 2020-01-16T00:13:36.194 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:36.194 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.194 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.194 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:36.194 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.257904+0000", 2020-01-16T00:13:36.194 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999933 2020-01-16T00:13:36.195 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.195 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.195 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:36.195 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.257898+0000", 2020-01-16T00:13:36.195 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00097099800000000004 2020-01-16T00:13:36.195 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.195 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.195 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:36.196 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.258869+0000", 2020-01-16T00:13:36.196 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7970000000000003e-05 2020-01-16T00:13:36.196 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.196 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.196 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:36.197 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.258917+0000", 2020-01-16T00:13:36.197 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5982999999999999e-05 2020-01-16T00:13:36.197 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.197 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.197 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:36.198 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.258933+0000", 2020-01-16T00:13:36.198 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00026239999999999998 2020-01-16T00:13:36.198 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.198 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.198 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:36.198 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.259195+0000", 2020-01-16T00:13:36.198 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024907400000000001 2020-01-16T00:13:36.199 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.199 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.199 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:36.199 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.259444+0000", 2020-01-16T00:13:36.199 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00034824999999999998 2020-01-16T00:13:36.199 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.199 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.199 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:36.200 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.259792+0000", 2020-01-16T00:13:36.200 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003251538 2020-01-16T00:13:36.200 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.200 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.200 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:36.200 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.263044+0000", 2020-01-16T00:13:36.200 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0068502880000000004 2020-01-16T00:13:36.201 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.201 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.201 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:36.201 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.269894+0000", 2020-01-16T00:13:36.201 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0945000000000006e-05 2020-01-16T00:13:36.201 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.202 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.202 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:36.202 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.269965+0000", 2020-01-16T00:13:36.202 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012667399999999999 2020-01-16T00:13:36.202 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.202 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.203 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:36.203 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.270092+0000", 2020-01-16T00:13:36.203 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012187450000000001 2020-01-16T00:13:36.203 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.203 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:36.203 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.204 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.204 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.204 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3273 3.d 3:b76659c4:::smithi12313934-47:head [read 0~3647152,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:13:36.204 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.728775+0000", 2020-01-16T00:13:36.204 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.3097438600000002, 2020-01-16T00:13:36.204 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012922950000000001, 2020-01-16T00:13:36.204 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:36.204 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:36.205 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:36.205 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:36.205 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:36.205 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3273 2020-01-16T00:13:36.205 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.205 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:36.206 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.206 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:36.206 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728775+0000", 2020-01-16T00:13:36.206 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:36.206 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.206 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.206 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:36.206 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728775+0000", 2020-01-16T00:13:36.207 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:36.207 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.207 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.207 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:36.207 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728771+0000", 2020-01-16T00:13:36.207 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.5009000000000001e-05 2020-01-16T00:13:36.208 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.208 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.208 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:36.208 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728796+0000", 2020-01-16T00:13:36.208 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2357999999999998e-05 2020-01-16T00:13:36.209 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.209 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.209 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:36.209 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728819+0000", 2020-01-16T00:13:36.209 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.4800000000000007e-06 2020-01-16T00:13:36.209 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.210 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.210 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:36.210 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728828+0000", 2020-01-16T00:13:36.210 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000124824 2020-01-16T00:13:36.210 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.210 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.211 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:36.211 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728953+0000", 2020-01-16T00:13:36.211 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00044904899999999998 2020-01-16T00:13:36.211 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.211 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.211 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:36.211 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729402+0000", 2020-01-16T00:13:36.211 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012295706999999999 2020-01-16T00:13:36.212 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.212 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.212 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:36.212 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.741698+0000", 2020-01-16T00:13:36.212 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012922950000000001 2020-01-16T00:13:36.213 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.213 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:36.213 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.213 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.213 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.213 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3275 3.d 3:b76659c4:::smithi12313934-47:head [read 0~3647152,checksum] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:13:36.213 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.728933+0000", 2020-01-16T00:13:36.214 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.309585588, 2020-01-16T00:13:36.214 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01204874, 2020-01-16T00:13:36.214 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:36.214 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:36.214 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:36.215 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:36.215 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:36.215 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3275 2020-01-16T00:13:36.215 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.215 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:36.216 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.216 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:36.216 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728933+0000", 2020-01-16T00:13:36.216 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:36.217 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.217 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.217 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:36.217 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728933+0000", 2020-01-16T00:13:36.218 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:36.218 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.218 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.218 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:36.219 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728932+0000", 2020-01-16T00:13:36.219 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3569999999999998e-06 2020-01-16T00:13:36.219 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.219 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.220 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:36.220 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728940+0000", 2020-01-16T00:13:36.220 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0457e-05 2020-01-16T00:13:36.220 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.220 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.221 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:36.221 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728951+0000", 2020-01-16T00:13:36.221 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6799999999999993e-06 2020-01-16T00:13:36.221 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.222 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.222 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:36.222 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728959+0000", 2020-01-16T00:13:36.222 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010076307 2020-01-16T00:13:36.222 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.223 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.223 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:36.223 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.739035+0000", 2020-01-16T00:13:36.223 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024455400000000002 2020-01-16T00:13:36.224 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.224 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.224 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:36.224 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.739279+0000", 2020-01-16T00:13:36.225 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0017023660000000001 2020-01-16T00:13:36.225 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.225 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.225 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:36.225 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.740982+0000", 2020-01-16T00:13:36.226 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01204874 2020-01-16T00:13:36.226 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.226 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:36.227 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.227 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.227 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.227 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3285 3.d 3:bd9fc72f:::smithi12313934-5:head [read 0~3764801,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:13:36.227 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.729068+0000", 2020-01-16T00:13:36.227 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.3094507960000001, 2020-01-16T00:13:36.228 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018537010999999999, 2020-01-16T00:13:36.228 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:36.228 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:36.228 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:36.228 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:36.228 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:36.228 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3285 2020-01-16T00:13:36.229 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.229 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:36.229 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.229 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:36.229 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729068+0000", 2020-01-16T00:13:36.229 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:36.230 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.230 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.230 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:36.230 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729068+0000", 2020-01-16T00:13:36.231 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:36.231 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.231 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.232 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:36.232 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729067+0000", 2020-01-16T00:13:36.232 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3339999999999996e-06 2020-01-16T00:13:36.233 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.233 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.233 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:36.233 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729075+0000", 2020-01-16T00:13:36.234 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4187e-05 2020-01-16T00:13:36.234 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.234 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.234 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:36.235 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729089+0000", 2020-01-16T00:13:36.235 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5600000000000004e-06 2020-01-16T00:13:36.235 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.235 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.235 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:36.235 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729093+0000", 2020-01-16T00:13:36.235 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011944309 2020-01-16T00:13:36.236 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.236 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.236 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:36.236 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.741038+0000", 2020-01-16T00:13:36.236 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016625799999999999 2020-01-16T00:13:36.236 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.237 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.237 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:36.237 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.741204+0000", 2020-01-16T00:13:36.237 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0064010120000000002 2020-01-16T00:13:36.237 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.237 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.237 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:36.238 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.747605+0000", 2020-01-16T00:13:36.238 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018537010999999999 2020-01-16T00:13:36.238 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.238 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:36.238 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.238 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.238 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.239 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3286 3.d 3:bd9fc72f:::smithi12313934-5:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:13:36.239 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.729099+0000", 2020-01-16T00:13:36.239 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.309420126, 2020-01-16T00:13:36.239 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018983897999999999, 2020-01-16T00:13:36.239 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:36.239 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:36.239 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:36.240 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:36.240 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:36.240 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3286 2020-01-16T00:13:36.240 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.240 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:36.240 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.240 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:36.241 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729099+0000", 2020-01-16T00:13:36.241 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:36.241 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.241 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.241 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:36.241 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729099+0000", 2020-01-16T00:13:36.241 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:36.242 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.242 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.242 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:36.242 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729098+0000", 2020-01-16T00:13:36.242 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8500000000000004e-06 2020-01-16T00:13:36.242 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.242 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.242 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:36.243 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729102+0000", 2020-01-16T00:13:36.243 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7329999999999997e-06 2020-01-16T00:13:36.243 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.243 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.243 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:36.243 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729108+0000", 2020-01-16T00:13:36.243 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.764e-06 2020-01-16T00:13:36.244 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.244 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.244 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:36.244 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729118+0000", 2020-01-16T00:13:36.244 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018524286000000001 2020-01-16T00:13:36.244 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.244 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.245 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:36.245 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.747642+0000", 2020-01-16T00:13:36.245 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9487999999999993e-05 2020-01-16T00:13:36.245 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.245 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.245 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:36.245 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.747742+0000", 2020-01-16T00:13:36.246 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000340429 2020-01-16T00:13:36.246 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.246 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.246 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:36.246 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.748082+0000", 2020-01-16T00:13:36.246 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018983897999999999 2020-01-16T00:13:36.246 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.246 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:36.247 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.247 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.247 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.247 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3287 3.d 3:bd9fc72f:::smithi12313934-5:head [read 0~3764801,checksum] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:13:36.247 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.729124+0000", 2020-01-16T00:13:36.247 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.3093949870000001, 2020-01-16T00:13:36.247 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020552323000000001, 2020-01-16T00:13:36.248 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:36.248 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:36.248 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:36.248 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:36.248 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:36.248 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3287 2020-01-16T00:13:36.248 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.248 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:36.249 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.249 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:36.249 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729124+0000", 2020-01-16T00:13:36.249 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:36.249 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.249 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.250 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:36.250 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729124+0000", 2020-01-16T00:13:36.250 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:36.250 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.250 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.250 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:36.250 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729123+0000", 2020-01-16T00:13:36.250 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8239999999999996e-06 2020-01-16T00:13:36.251 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.251 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.251 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:36.251 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729129+0000", 2020-01-16T00:13:36.251 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6909999999999997e-06 2020-01-16T00:13:36.251 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.251 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.252 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:36.252 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729136+0000", 2020-01-16T00:13:36.252 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3329999999999997e-06 2020-01-16T00:13:36.252 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.252 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.252 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:36.252 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729145+0000", 2020-01-16T00:13:36.253 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018965065999999999 2020-01-16T00:13:36.253 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.253 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.253 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:36.253 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.748110+0000", 2020-01-16T00:13:36.253 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015781700000000001 2020-01-16T00:13:36.253 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.254 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.254 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:36.254 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.748268+0000", 2020-01-16T00:13:36.254 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014083800000000001 2020-01-16T00:13:36.255 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.255 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.255 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:36.255 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.749676+0000", 2020-01-16T00:13:36.255 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020552323000000001 2020-01-16T00:13:36.255 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.255 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:36.256 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.256 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.256 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.256 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3409 3.d 3:b3dd5d45:::smithi12313934-16:head [write 591712~786646,stat] snapc 0=[] ondisk+write+known_if_redirected e47)", 2020-01-16T00:13:36.256 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:34.100681+0000", 2020-01-16T00:13:36.256 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 1.9378378469999999, 2020-01-16T00:13:36.256 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012248446, 2020-01-16T00:13:36.257 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:36.257 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:36.257 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:36.257 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:36.257 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:36.257 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3409 2020-01-16T00:13:36.258 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.258 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:36.258 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.258 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:36.258 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.100681+0000", 2020-01-16T00:13:36.258 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:36.258 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.259 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.259 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:36.259 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.100681+0000", 2020-01-16T00:13:36.259 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:13:36.259 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.259 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.259 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:36.260 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.100677+0000", 2020-01-16T00:13:36.260 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000998828 2020-01-16T00:13:36.260 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.260 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.260 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:36.260 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101676+0000", 2020-01-16T00:13:36.260 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5846999999999997e-05 2020-01-16T00:13:36.261 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.262 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.262 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:36.262 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101712+0000", 2020-01-16T00:13:36.262 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.364e-05 2020-01-16T00:13:36.262 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.262 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.263 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:36.263 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101725+0000", 2020-01-16T00:13:36.263 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015902700000000001 2020-01-16T00:13:36.263 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.263 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.264 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:36.264 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101884+0000", 2020-01-16T00:13:36.264 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0486e-05 2020-01-16T00:13:36.264 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.264 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.264 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op must wait for map", 2020-01-16T00:13:36.264 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101905+0000", 2020-01-16T00:13:36.264 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0038869009999999999 2020-01-16T00:13:36.265 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.265 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.265 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:36.265 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.105792+0000", 2020-01-16T00:13:36.265 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030748800000000002 2020-01-16T00:13:36.265 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.265 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.266 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:36.266 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.106099+0000", 2020-01-16T00:13:36.266 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042069299999999998 2020-01-16T00:13:36.266 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.266 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.266 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:36.266 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.106520+0000", 2020-01-16T00:13:36.266 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0052864560000000001 2020-01-16T00:13:36.267 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.267 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.267 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:36.267 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.111806+0000", 2020-01-16T00:13:36.267 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00080835800000000001 2020-01-16T00:13:36.267 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.267 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.268 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:36.268 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.112615+0000", 2020-01-16T00:13:36.268 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1490999999999997e-05 2020-01-16T00:13:36.268 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.268 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.268 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:36.268 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.112686+0000", 2020-01-16T00:13:36.268 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000243174 2020-01-16T00:13:36.269 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.269 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.269 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:36.269 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.112929+0000", 2020-01-16T00:13:36.269 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012248446 2020-01-16T00:13:36.269 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.269 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:36.270 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.270 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.270 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:36.270 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:36.270 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-01-16T00:13:36.272 INFO:tasks.rados.rados.0.smithi123.stdout:1343: finishing write tid 1 to smithi12313934-36 2020-01-16T00:13:36.272 INFO:tasks.rados.rados.0.smithi123.stdout:1343: finishing write tid 2 to smithi12313934-36 2020-01-16T00:13:36.272 INFO:tasks.rados.rados.0.smithi123.stdout:1343: finishing write tid 3 to smithi12313934-36 2020-01-16T00:13:36.272 INFO:tasks.rados.rados.0.smithi123.stdout:1343: finishing write tid 5 to smithi12313934-36 2020-01-16T00:13:36.273 INFO:tasks.rados.rados.0.smithi123.stdout:1343: finishing write tid 6 to smithi12313934-36 2020-01-16T00:13:36.273 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 36 v 90 (ObjNum 422 snap 0 seq_num 422) dirty exists 2020-01-16T00:13:36.273 INFO:tasks.rados.rados.0.smithi123.stdout:1343: left oid 36 (ObjNum 422 snap 0 seq_num 422) 2020-01-16T00:13:36.273 INFO:tasks.rados.rados.0.smithi123.stdout:1344: expect (ObjNum 298 snap 0 seq_num 298) 2020-01-16T00:13:36.273 INFO:tasks.rados.rados.0.smithi123.stdout:1336: done (9 left) 2020-01-16T00:13:36.273 INFO:tasks.rados.rados.0.smithi123.stdout:1337: done (8 left) 2020-01-16T00:13:36.273 INFO:tasks.rados.rados.0.smithi123.stdout:1338: done (7 left) 2020-01-16T00:13:36.273 INFO:tasks.rados.rados.0.smithi123.stdout:1339: done (6 left) 2020-01-16T00:13:36.274 INFO:tasks.rados.rados.0.smithi123.stdout:1340: done (5 left) 2020-01-16T00:13:36.274 INFO:tasks.rados.rados.0.smithi123.stdout:1341: done (4 left) 2020-01-16T00:13:36.274 INFO:tasks.rados.rados.0.smithi123.stdout:1342: done (3 left) 2020-01-16T00:13:36.274 INFO:tasks.rados.rados.0.smithi123.stdout:1343: done (2 left) 2020-01-16T00:13:36.274 INFO:tasks.rados.rados.0.smithi123.stdout:1344: done (1 left) 2020-01-16T00:13:36.274 INFO:tasks.rados.rados.0.smithi123.stdout:1345: done (0 left) 2020-01-16T00:13:36.274 INFO:tasks.rados.rados.0.smithi123.stdout:1346: write oid 30 current snap is 0 2020-01-16T00:13:36.274 INFO:tasks.rados.rados.0.smithi123.stdout:1346: seq_num 424 ranges {726168=704991,1895071=683474} 2020-01-16T00:13:36.275 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:36.266+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:36.275 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-01-16T00:13:36.276 INFO:tasks.rados.rados.0.smithi123.stdout:1346: writing smithi12313934-30 from 726168 to 1431159 tid 1 2020-01-16T00:13:36.281 INFO:tasks.rados.rados.0.smithi123.stdout:1346: writing smithi12313934-30 from 1895071 to 2578545 tid 2 2020-01-16T00:13:36.281 INFO:tasks.rados.rados.0.smithi123.stdout:1347: read oid 33 snap -1 2020-01-16T00:13:36.281 INFO:tasks.rados.rados.0.smithi123.stdout:1347: expect (ObjNum 330 snap 0 seq_num 330) 2020-01-16T00:13:36.282 INFO:tasks.rados.rados.0.smithi123.stdout:1348: write (excl) oid 31 current snap is 0 2020-01-16T00:13:36.282 INFO:tasks.rados.rados.0.smithi123.stdout:1348: seq_num 425 ranges {742975=780240,1953553=427132,2971713=171262} 2020-01-16T00:13:36.291 INFO:tasks.rados.rados.0.smithi123.stdout:1348: writing smithi12313934-31 from 742975 to 1523215 tid 1 2020-01-16T00:13:36.295 INFO:tasks.rados.rados.0.smithi123.stdout:1348: writing smithi12313934-31 from 1953553 to 2380685 tid 2 2020-01-16T00:13:36.299 INFO:tasks.rados.rados.0.smithi123.stdout:1348: writing smithi12313934-31 from 2971713 to 3142975 tid 3 2020-01-16T00:13:36.300 INFO:tasks.rados.rados.0.smithi123.stdout:1349: write oid 27 current snap is 0 2020-01-16T00:13:36.300 INFO:tasks.rados.rados.0.smithi123.stdout:1349: seq_num 426 ranges {759782=571842,2012035=687143} 2020-01-16T00:13:36.309 INFO:tasks.rados.rados.0.smithi123.stdout:1349: writing smithi12313934-27 from 759782 to 1331624 tid 1 2020-01-16T00:13:36.315 INFO:tasks.rados.rados.0.smithi123.stdout:1349: writing smithi12313934-27 from 2012035 to 2699178 tid 2 2020-01-16T00:13:36.315 INFO:tasks.rados.rados.0.smithi123.stdout:1350: read oid 24 snap -1 2020-01-16T00:13:36.315 INFO:tasks.rados.rados.0.smithi123.stdout:1350: expect deleted 2020-01-16T00:13:36.315 INFO:tasks.rados.rados.0.smithi123.stdout:1351: write oid 7 current snap is 0 2020-01-16T00:13:36.316 INFO:tasks.rados.rados.0.smithi123.stdout:1351: seq_num 427 ranges {776589=647091,2070517=430801,3026559=150030} 2020-01-16T00:13:36.323 INFO:tasks.rados.rados.0.smithi123.stdout:1351: writing smithi12313934-7 from 776589 to 1423680 tid 1 2020-01-16T00:13:36.329 INFO:tasks.rados.rados.0.smithi123.stdout:1351: writing smithi12313934-7 from 2070517 to 2501318 tid 2 2020-01-16T00:13:36.333 INFO:tasks.rados.rados.0.smithi123.stdout:1351: writing smithi12313934-7 from 3026559 to 3176589 tid 3 2020-01-16T00:13:36.333 INFO:tasks.rados.rados.0.smithi123.stdout:1352: write (excl) oid 48 current snap is 0 2020-01-16T00:13:36.333 INFO:tasks.rados.rados.0.smithi123.stdout:1352: seq_num 428 ranges {793396=722340,2128999=690812} 2020-01-16T00:13:36.341 INFO:tasks.rados.rados.0.smithi123.stdout:1352: writing smithi12313934-48 from 793396 to 1515736 tid 1 2020-01-16T00:13:36.349 INFO:tasks.rados.rados.0.smithi123.stdout:1352: writing smithi12313934-48 from 2128999 to 2819811 tid 2 2020-01-16T00:13:36.349 INFO:tasks.rados.rados.0.smithi123.stdout:1353: write oid 35 current snap is 0 2020-01-16T00:13:36.349 INFO:tasks.rados.rados.0.smithi123.stdout:1353: seq_num 429 ranges {410203=797589,1787481=434470,2681405=528798} 2020-01-16T00:13:36.356 INFO:tasks.rados.rados.0.smithi123.stdout:1353: writing smithi12313934-35 from 410203 to 1207792 tid 1 2020-01-16T00:13:36.362 INFO:tasks.rados.rados.0.smithi123.stdout:1353: writing smithi12313934-35 from 1787481 to 2221951 tid 2 2020-01-16T00:13:36.368 INFO:tasks.rados.rados.0.smithi123.stdout:1353: writing smithi12313934-35 from 2681405 to 3210203 tid 3 2020-01-16T00:13:36.368 INFO:tasks.rados.rados.0.smithi123.stdout:1354: read oid 19 snap -1 2020-01-16T00:13:36.369 INFO:tasks.rados.rados.0.smithi123.stdout:1354: expect (ObjNum 342 snap 0 seq_num 342) 2020-01-16T00:13:36.369 INFO:tasks.rados.rados.0.smithi123.stdout:1355: write (excl) oid 12 current snap is 0 2020-01-16T00:13:36.369 INFO:tasks.rados.rados.0.smithi123.stdout:1355: seq_num 430 ranges {427010=472838,1329610=694481,2708828=518182} 2020-01-16T00:13:36.372 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:36.370+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:36.373 INFO:tasks.rados.rados.0.smithi123.stdout:1355: writing smithi12313934-12 from 427010 to 899848 tid 1 2020-01-16T00:13:36.381 INFO:tasks.rados.rados.0.smithi123.stdout:1355: writing smithi12313934-12 from 1329610 to 2024091 tid 2 2020-01-16T00:13:36.388 INFO:tasks.rados.rados.0.smithi123.stdout:1355: writing smithi12313934-12 from 2708828 to 3227010 tid 3 2020-01-16T00:13:36.388 INFO:tasks.rados.rados.0.smithi123.stdout:1356: write oid 47 current snap is 0 2020-01-16T00:13:36.388 INFO:tasks.rados.rados.0.smithi123.stdout:1356: seq_num 431 ranges {443817=548087,1788092=438139,3019898=223919} 2020-01-16T00:13:36.394 INFO:tasks.rados.rados.0.smithi123.stdout:1356: writing smithi12313934-47 from 443817 to 991904 tid 1 2020-01-16T00:13:36.400 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:13:36.401 INFO:tasks.rados.rados.0.smithi123.stdout:1356: writing smithi12313934-47 from 1788092 to 2226231 tid 2 2020-01-16T00:13:36.403 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:36.403 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:13:36.403 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.403 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.4488.0:3497 3.d e47/40 3:b76659c4:::smithi12313934-47:head v 47'203, mlcod=47'201)", 2020-01-16T00:13:36.403 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:36.401982+0000", 2020-01-16T00:13:36.404 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.00085877900000000005, 2020-01-16T00:13:36.404 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00088374899999999997, 2020-01-16T00:13:36.404 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:36.404 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:36.404 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:36.404 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.404 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:36.405 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:36.401982+0000", 2020-01-16T00:13:36.405 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:36.405 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.405 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.405 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:36.405 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:36.401982+0000", 2020-01-16T00:13:36.406 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:36.406 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.406 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.406 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:36.407 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:36.401981+0000", 2020-01-16T00:13:36.407 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017295599999999999 2020-01-16T00:13:36.407 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.407 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.407 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:36.407 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:36.402153+0000", 2020-01-16T00:13:36.407 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6262000000000001e-05 2020-01-16T00:13:36.407 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.408 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.408 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:36.408 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:36.402170+0000", 2020-01-16T00:13:36.408 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.866e-06 2020-01-16T00:13:36.408 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.408 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.408 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:36.409 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:36.402177+0000", 2020-01-16T00:13:36.409 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011862399999999999 2020-01-16T00:13:36.409 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.409 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.409 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:36.410 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:36.402295+0000", 2020-01-16T00:13:36.410 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2471999999999993e-05 2020-01-16T00:13:36.410 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:36.410 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:36.410 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:36.410 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:36.402368+0000", 2020-01-16T00:13:36.410 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00038546300000000001 2020-01-16T00:13:36.410 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.411 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:36.411 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.411 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:36.411 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2020-01-16T00:13:36.411 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 1 2020-01-16T00:13:36.411 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:36.412 INFO:tasks.rados.rados.0.smithi123.stdout:1356: writing smithi12313934-47 from 3019898 to 3243817 tid 3 2020-01-16T00:13:36.412 INFO:tasks.rados.rados.0.smithi123.stdout:1357: read oid 4 snap -1 2020-01-16T00:13:36.412 INFO:tasks.rados.rados.0.smithi123.stdout:1357: expect (ObjNum 387 snap 0 seq_num 387) 2020-01-16T00:13:36.413 INFO:tasks.rados.rados.0.smithi123.stdout:1358: write oid 42 current snap is 0 2020-01-16T00:13:36.413 INFO:tasks.rados.rados.0.smithi123.stdout:1358: seq_num 432 ranges {460624=623336,1846574=698150,3163674=96950} 2020-01-16T00:13:36.413 INFO:tasks.rados.rados.0.smithi123.stdout:1358: writing smithi12313934-42 from 460624 to 1083960 tid 1 2020-01-16T00:13:36.414 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-01-16T00:13:36.418 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:13:36.421 INFO:tasks.rados.rados.0.smithi123.stdout:1358: writing smithi12313934-42 from 1846574 to 2544724 tid 2 2020-01-16T00:13:36.425 INFO:tasks.rados.rados.0.smithi123.stdout:1358: writing smithi12313934-42 from 3163674 to 3260624 tid 3 2020-01-16T00:13:36.426 INFO:tasks.rados.rados.0.smithi123.stdout:1359: delete oid 5 current snap is 0 2020-01-16T00:13:36.431 INFO:tasks.rados.rados.0.smithi123.stdout:1346: finishing write tid 1 to smithi12313934-30 2020-01-16T00:13:36.431 INFO:tasks.rados.rados.0.smithi123.stdout:1346: finishing write tid 2 to smithi12313934-30 2020-01-16T00:13:36.432 INFO:tasks.rados.rados.0.smithi123.stdout:1346: finishing write tid 4 to smithi12313934-30 2020-01-16T00:13:36.432 INFO:tasks.rados.rados.0.smithi123.stdout:1346: finishing write tid 5 to smithi12313934-30 2020-01-16T00:13:36.432 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 30 v 136 (ObjNum 424 snap 0 seq_num 424) dirty exists 2020-01-16T00:13:36.432 INFO:tasks.rados.rados.0.smithi123.stdout:1346: left oid 30 (ObjNum 424 snap 0 seq_num 424) 2020-01-16T00:13:36.432 INFO:tasks.rados.rados.0.smithi123.stdout:1347: expect (ObjNum 330 snap 0 seq_num 330) 2020-01-16T00:13:36.441 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:13:36.459 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:13:36.473 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:36.470+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:36.478 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:13:36.499 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:13:36.511 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:36.522 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-01-16T00:13:36.523 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:13:36.523 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-01-16T00:13:36.524 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-01-16T00:13:36.524 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-01-16T00:13:36.524 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-01-16T00:13:36.524 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-01-16T00:13:36.525 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENXIO: problem getting command descriptions from osd.5 2020-01-16T00:13:36.525 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: problem getting command descriptions from osd.5 2020-01-16T00:13:36.525 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-01-16T00:13:36.525 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-01-16T00:13:36.533 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-01-16T00:13:36.534 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-01-16T00:13:36.576 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:36.574+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:36.633 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:36.642 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-01-16T00:13:36.674 INFO:tasks.rados.rados.0.smithi123.stdout:1348: finishing write tid 1 to smithi12313934-31 2020-01-16T00:13:36.674 INFO:tasks.rados.rados.0.smithi123.stdout:1348: finishing write tid 2 to smithi12313934-31 2020-01-16T00:13:36.674 INFO:tasks.rados.rados.0.smithi123.stdout:1348: finishing write tid 3 to smithi12313934-31 2020-01-16T00:13:36.674 INFO:tasks.rados.rados.0.smithi123.stdout:1348: finishing write tid 5 to smithi12313934-31 2020-01-16T00:13:36.675 INFO:tasks.rados.rados.0.smithi123.stdout:1348: finishing write tid 6 to smithi12313934-31 2020-01-16T00:13:36.675 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 31 v 163 (ObjNum 425 snap 0 seq_num 425) dirty exists 2020-01-16T00:13:36.675 INFO:tasks.rados.rados.0.smithi123.stdout:1348: left oid 31 (ObjNum 425 snap 0 seq_num 425) 2020-01-16T00:13:36.675 INFO:tasks.rados.rados.0.smithi123.stdout:1349: finishing write tid 1 to smithi12313934-27 2020-01-16T00:13:36.675 INFO:tasks.rados.rados.0.smithi123.stdout:1349: finishing write tid 2 to smithi12313934-27 2020-01-16T00:13:36.675 INFO:tasks.rados.rados.0.smithi123.stdout:1349: finishing write tid 4 to smithi12313934-27 2020-01-16T00:13:36.675 INFO:tasks.rados.rados.0.smithi123.stdout:1349: finishing write tid 5 to smithi12313934-27 2020-01-16T00:13:36.675 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 27 v 136 (ObjNum 426 snap 0 seq_num 426) dirty exists 2020-01-16T00:13:36.676 INFO:tasks.rados.rados.0.smithi123.stdout:1349: left oid 27 (ObjNum 426 snap 0 seq_num 426) 2020-01-16T00:13:36.676 INFO:tasks.rados.rados.0.smithi123.stdout:1351: finishing write tid 1 to smithi12313934-7 2020-01-16T00:13:36.676 INFO:tasks.rados.rados.0.smithi123.stdout:1351: finishing write tid 2 to smithi12313934-7 2020-01-16T00:13:36.676 INFO:tasks.rados.rados.0.smithi123.stdout:1351: finishing write tid 3 to smithi12313934-7 2020-01-16T00:13:36.676 INFO:tasks.rados.rados.0.smithi123.stdout:1346: done (13 left) 2020-01-16T00:13:36.676 INFO:tasks.rados.rados.0.smithi123.stdout:1347: done (12 left) 2020-01-16T00:13:36.676 INFO:tasks.rados.rados.0.smithi123.stdout:1348: done (11 left) 2020-01-16T00:13:36.677 INFO:tasks.rados.rados.0.smithi123.stdout:1349: done (10 left) 2020-01-16T00:13:36.677 INFO:tasks.rados.rados.0.smithi123.stdout:1350: done (9 left) 2020-01-16T00:13:36.677 INFO:tasks.rados.rados.0.smithi123.stdout:1359: done (8 left) 2020-01-16T00:13:36.683 INFO:tasks.rados.rados.0.smithi123.stdout:1360: delete oid 25 current snap is 0 2020-01-16T00:13:36.683 INFO:tasks.rados.rados.0.smithi123.stdout:1351: finishing write tid 5 to smithi12313934-7 2020-01-16T00:13:36.684 INFO:tasks.rados.rados.0.smithi123.stdout:1351: finishing write tid 6 to smithi12313934-7 2020-01-16T00:13:36.684 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 7 v 190 (ObjNum 427 snap 0 seq_num 427) dirty exists 2020-01-16T00:13:36.684 INFO:tasks.rados.rados.0.smithi123.stdout:1351: left oid 7 (ObjNum 427 snap 0 seq_num 427) 2020-01-16T00:13:36.684 INFO:tasks.rados.rados.0.smithi123.stdout:1352: finishing write tid 1 to smithi12313934-48 2020-01-16T00:13:36.684 INFO:tasks.rados.rados.0.smithi123.stdout:1352: finishing write tid 2 to smithi12313934-48 2020-01-16T00:13:36.684 INFO:tasks.rados.rados.0.smithi123.stdout:1353: finishing write tid 1 to smithi12313934-35 2020-01-16T00:13:36.684 INFO:tasks.rados.rados.0.smithi123.stdout:1353: finishing write tid 2 to smithi12313934-35 2020-01-16T00:13:36.685 INFO:tasks.rados.rados.0.smithi123.stdout:1352: finishing write tid 4 to smithi12313934-48 2020-01-16T00:13:36.685 INFO:tasks.rados.rados.0.smithi123.stdout:1352: finishing write tid 5 to smithi12313934-48 2020-01-16T00:13:36.685 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 48 v 154 (ObjNum 428 snap 0 seq_num 428) dirty exists 2020-01-16T00:13:36.685 INFO:tasks.rados.rados.0.smithi123.stdout:1352: left oid 48 (ObjNum 428 snap 0 seq_num 428) 2020-01-16T00:13:36.685 INFO:tasks.rados.rados.0.smithi123.stdout:1353: finishing write tid 3 to smithi12313934-35 2020-01-16T00:13:36.685 INFO:tasks.rados.rados.0.smithi123.stdout:1353: finishing write tid 5 to smithi12313934-35 2020-01-16T00:13:36.685 INFO:tasks.rados.rados.0.smithi123.stdout:1353: finishing write tid 6 to smithi12313934-35 2020-01-16T00:13:36.685 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 35 v 150 (ObjNum 429 snap 0 seq_num 429) dirty exists 2020-01-16T00:13:36.686 INFO:tasks.rados.rados.0.smithi123.stdout:1353: left oid 35 (ObjNum 429 snap 0 seq_num 429) 2020-01-16T00:13:36.686 INFO:tasks.rados.rados.0.smithi123.stdout:1354: expect (ObjNum 342 snap 0 seq_num 342) 2020-01-16T00:13:36.695 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:36.692+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:36.747 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:36.754 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-01-16T00:13:36.787 INFO:teuthology.orchestra.run.smithi123.stderr:noscrub is unset 2020-01-16T00:13:36.796 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:36.792+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:36.876 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:36.885 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-01-16T00:13:36.896 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:36.892+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:36.913 INFO:tasks.rados.rados.0.smithi123.stdout:1355: finishing write tid 1 to smithi12313934-12 2020-01-16T00:13:36.914 INFO:tasks.rados.rados.0.smithi123.stdout:1355: finishing write tid 2 to smithi12313934-12 2020-01-16T00:13:36.914 INFO:tasks.rados.rados.0.smithi123.stdout:1355: finishing write tid 3 to smithi12313934-12 2020-01-16T00:13:36.914 INFO:tasks.rados.rados.0.smithi123.stdout:1351: done (8 left) 2020-01-16T00:13:36.914 INFO:tasks.rados.rados.0.smithi123.stdout:1352: done (7 left) 2020-01-16T00:13:36.914 INFO:tasks.rados.rados.0.smithi123.stdout:1353: done (6 left) 2020-01-16T00:13:36.914 INFO:tasks.rados.rados.0.smithi123.stdout:1354: done (5 left) 2020-01-16T00:13:36.915 INFO:tasks.rados.rados.0.smithi123.stdout:1360: done (4 left) 2020-01-16T00:13:36.915 INFO:tasks.rados.rados.0.smithi123.stdout:1361: read oid 11 snap -1 2020-01-16T00:13:36.915 INFO:tasks.rados.rados.0.smithi123.stdout:1361: expect (ObjNum 419 snap 0 seq_num 419) 2020-01-16T00:13:36.915 INFO:tasks.rados.rados.0.smithi123.stdout:1362: read oid 16 snap -1 2020-01-16T00:13:36.915 INFO:tasks.rados.rados.0.smithi123.stdout:1362: expect (ObjNum 416 snap 0 seq_num 416) 2020-01-16T00:13:36.915 INFO:tasks.rados.rados.0.smithi123.stdout:1363: read oid 24 snap -1 2020-01-16T00:13:36.915 INFO:tasks.rados.rados.0.smithi123.stdout:1363: expect deleted 2020-01-16T00:13:36.916 INFO:tasks.rados.rados.0.smithi123.stdout:1364: write oid 9 current snap is 0 2020-01-16T00:13:36.916 INFO:tasks.rados.rados.0.smithi123.stdout:1364: seq_num 435 ranges {511045=565436,1622020=445477,2729590=556855} 2020-01-16T00:13:36.921 INFO:tasks.rados.rados.0.smithi123.stdout:1364: writing smithi12313934-9 from 511045 to 1076481 tid 1 2020-01-16T00:13:36.927 INFO:tasks.rados.rados.0.smithi123.stdout:1364: writing smithi12313934-9 from 1622020 to 2067497 tid 2 2020-01-16T00:13:36.936 INFO:tasks.rados.rados.0.smithi123.stdout:1364: writing smithi12313934-9 from 2729590 to 3286445 tid 3 2020-01-16T00:13:36.936 INFO:tasks.rados.rados.0.smithi123.stdout:1365: read oid 39 snap -1 2020-01-16T00:13:36.936 INFO:tasks.rados.rados.0.smithi123.stdout:1365: expect (ObjNum 403 snap 0 seq_num 403) 2020-01-16T00:13:36.936 INFO:tasks.rados.rados.0.smithi123.stdout:1366: read oid 8 snap -1 2020-01-16T00:13:36.936 INFO:tasks.rados.rados.0.smithi123.stdout:1366: expect (ObjNum 418 snap 0 seq_num 418) 2020-01-16T00:13:36.936 INFO:tasks.rados.rados.0.smithi123.stdout:1367: write (excl) oid 30 current snap is 0 2020-01-16T00:13:36.936 INFO:tasks.rados.rados.0.smithi123.stdout:1367: seq_num 436 ranges {527852=640685,1680502=589135,2757013=570839} 2020-01-16T00:13:36.943 INFO:tasks.rados.rados.0.smithi123.stdout:1367: writing smithi12313934-30 from 527852 to 1168537 tid 1 2020-01-16T00:13:36.951 INFO:tasks.rados.rados.0.smithi123.stdout:1367: writing smithi12313934-30 from 1680502 to 2269637 tid 2 2020-01-16T00:13:36.958 INFO:tasks.rados.rados.0.smithi123.stdout:1367: writing smithi12313934-30 from 2757013 to 3327852 tid 3 2020-01-16T00:13:36.958 INFO:tasks.rados.rados.0.smithi123.stdout:1368: read oid 21 snap -1 2020-01-16T00:13:36.959 INFO:tasks.rados.rados.0.smithi123.stdout:1368: expect deleted 2020-01-16T00:13:36.959 INFO:tasks.rados.rados.0.smithi123.stdout:1369: read oid 44 snap -1 2020-01-16T00:13:36.959 INFO:tasks.rados.rados.0.smithi123.stdout:1369: expect (ObjNum 380 snap 0 seq_num 380) 2020-01-16T00:13:36.959 INFO:tasks.rados.rados.0.smithi123.stdout:1370: write (excl) oid 31 current snap is 0 2020-01-16T00:13:36.959 INFO:tasks.rados.rados.0.smithi123.stdout:1370: seq_num 437 ranges {544659=715934,1738984=449146,2784436=560223} 2020-01-16T00:13:36.966 INFO:tasks.rados.rados.0.smithi123.stdout:1370: writing smithi12313934-31 from 544659 to 1260593 tid 1 2020-01-16T00:13:36.973 INFO:tasks.rados.rados.0.smithi123.stdout:1370: writing smithi12313934-31 from 1738984 to 2188130 tid 2 2020-01-16T00:13:36.980 INFO:tasks.rados.rados.0.smithi123.stdout:1370: writing smithi12313934-31 from 2784436 to 3344659 tid 3 2020-01-16T00:13:36.980 INFO:tasks.rados.rados.0.smithi123.stdout:1371: write (excl) oid 25 current snap is 0 2020-01-16T00:13:36.981 INFO:tasks.rados.rados.0.smithi123.stdout:1371: seq_num 438 ranges {561466=791183,2081113=592804,3095506=265960} 2020-01-16T00:13:36.989 INFO:tasks.rados.rados.0.smithi123.stdout:1371: writing smithi12313934-25 from 561466 to 1352649 tid 1 2020-01-16T00:13:36.997 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:36.994+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:36.998 INFO:tasks.rados.rados.0.smithi123.stdout:1371: writing smithi12313934-25 from 2081113 to 2673917 tid 2 2020-01-16T00:13:37.006 INFO:tasks.rados.rados.0.smithi123.stdout:1371: writing smithi12313934-25 from 3095506 to 3361466 tid 3 2020-01-16T00:13:37.006 INFO:tasks.rados.rados.0.smithi123.stdout:1372: delete oid 38 current snap is 0 2020-01-16T00:13:37.006 INFO:tasks.rados.rados.0.smithi123.stdout:1355: finishing write tid 5 to smithi12313934-12 2020-01-16T00:13:37.006 INFO:tasks.rados.rados.0.smithi123.stdout:1355: finishing write tid 6 to smithi12313934-12 2020-01-16T00:13:37.006 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 12 v 121 (ObjNum 430 snap 0 seq_num 430) dirty exists 2020-01-16T00:13:37.007 INFO:tasks.rados.rados.0.smithi123.stdout:1355: left oid 12 (ObjNum 430 snap 0 seq_num 430) 2020-01-16T00:13:37.007 INFO:tasks.rados.rados.0.smithi123.stdout:1356: finishing write tid 1 to smithi12313934-47 2020-01-16T00:13:37.007 INFO:tasks.rados.rados.0.smithi123.stdout:1357: expect (ObjNum 387 snap 0 seq_num 387) 2020-01-16T00:13:37.009 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:37.018 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-01-16T00:13:37.108 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:37.106+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:37.119 INFO:tasks.rados.rados.0.smithi123.stdout:1356: finishing write tid 2 to smithi12313934-47 2020-01-16T00:13:37.119 INFO:tasks.rados.rados.0.smithi123.stdout:1356: finishing write tid 3 to smithi12313934-47 2020-01-16T00:13:37.121 INFO:tasks.rados.rados.0.smithi123.stdout:1355: done (15 left) 2020-01-16T00:13:37.121 INFO:tasks.rados.rados.0.smithi123.stdout:1357: done (14 left) 2020-01-16T00:13:37.121 INFO:tasks.rados.rados.0.smithi123.stdout:1372: done (13 left) 2020-01-16T00:13:37.121 INFO:tasks.rados.rados.0.smithi123.stdout:1373: read oid 17 snap -1 2020-01-16T00:13:37.121 INFO:tasks.rados.rados.0.smithi123.stdout:1373: expect deleted 2020-01-16T00:13:37.122 INFO:tasks.rados.rados.0.smithi123.stdout:1374: read oid 19 snap -1 2020-01-16T00:13:37.122 INFO:tasks.rados.rados.0.smithi123.stdout:1374: expect (ObjNum 342 snap 0 seq_num 342) 2020-01-16T00:13:37.122 INFO:tasks.rados.rados.0.smithi123.stdout:1375: write (excl) oid 26 current snap is 0 2020-01-16T00:13:37.122 INFO:tasks.rados.rados.0.smithi123.stdout:1375: seq_num 440 ranges {595080=541681,1798077=596473,3150352=244728} 2020-01-16T00:13:37.125 INFO:tasks.rados.rados.0.smithi123.stdout:1375: writing smithi12313934-26 from 595080 to 1136761 tid 1 2020-01-16T00:13:37.134 INFO:tasks.rados.rados.0.smithi123.stdout:1375: writing smithi12313934-26 from 1798077 to 2394550 tid 2 2020-01-16T00:13:37.141 INFO:tasks.rados.rados.0.smithi123.stdout:1375: writing smithi12313934-26 from 3150352 to 3395080 tid 3 2020-01-16T00:13:37.141 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:37.141 INFO:tasks.rados.rados.0.smithi123.stdout:1356: finishing write tid 5 to smithi12313934-47 2020-01-16T00:13:37.141 INFO:tasks.rados.rados.0.smithi123.stdout:1356: finishing write tid 6 to smithi12313934-47 2020-01-16T00:13:37.141 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 47 v 205 (ObjNum 431 snap 0 seq_num 431) dirty exists 2020-01-16T00:13:37.141 INFO:tasks.rados.rados.0.smithi123.stdout:1356: left oid 47 (ObjNum 431 snap 0 seq_num 431) 2020-01-16T00:13:37.142 INFO:tasks.rados.rados.0.smithi123.stdout:1358: finishing write tid 1 to smithi12313934-42 2020-01-16T00:13:37.142 INFO:tasks.rados.rados.0.smithi123.stdout:1356: done (15 left) 2020-01-16T00:13:37.142 INFO:tasks.rados.rados.0.smithi123.stdout:1376: read oid 35 snap -1 2020-01-16T00:13:37.142 INFO:tasks.rados.rados.0.smithi123.stdout:1376: expect (ObjNum 429 snap 0 seq_num 429) 2020-01-16T00:13:37.142 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:37.142 INFO:tasks.rados.rados.0.smithi123.stdout:1358: finishing write tid 2 to smithi12313934-42 2020-01-16T00:13:37.142 INFO:tasks.rados.rados.0.smithi123.stdout:1358: finishing write tid 3 to smithi12313934-42 2020-01-16T00:13:37.142 INFO:tasks.rados.rados.0.smithi123.stdout:1358: finishing write tid 5 to smithi12313934-42 2020-01-16T00:13:37.143 INFO:tasks.rados.rados.0.smithi123.stdout:1358: finishing write tid 6 to smithi12313934-42 2020-01-16T00:13:37.143 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 42 v 159 (ObjNum 432 snap 0 seq_num 432) dirty exists 2020-01-16T00:13:37.143 INFO:tasks.rados.rados.0.smithi123.stdout:1358: left oid 42 (ObjNum 432 snap 0 seq_num 432) 2020-01-16T00:13:37.143 INFO:tasks.rados.rados.0.smithi123.stdout:1361: expect (ObjNum 419 snap 0 seq_num 419) 2020-01-16T00:13:37.174 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:37.183 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-01-16T00:13:37.224 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:37.222+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:37.303 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:37.300+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:37.321 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:37.330 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-01-16T00:13:37.346 INFO:tasks.rados.rados.0.smithi123.stdout:1364: finishing write tid 1 to smithi12313934-9 2020-01-16T00:13:37.347 INFO:tasks.rados.rados.0.smithi123.stdout:1362: expect (ObjNum 416 snap 0 seq_num 416) 2020-01-16T00:13:37.403 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:37.400+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:37.430 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:37.445 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-01-16T00:13:37.525 INFO:tasks.rados.rados.0.smithi123.stdout:1364: finishing write tid 2 to smithi12313934-9 2020-01-16T00:13:37.525 INFO:tasks.rados.rados.0.smithi123.stdout:1364: finishing write tid 3 to smithi12313934-9 2020-01-16T00:13:37.525 INFO:tasks.rados.rados.0.smithi123.stdout:1364: finishing write tid 5 to smithi12313934-9 2020-01-16T00:13:37.525 INFO:tasks.rados.rados.0.smithi123.stdout:1364: finishing write tid 6 to smithi12313934-9 2020-01-16T00:13:37.525 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 9 v 140 (ObjNum 435 snap 0 seq_num 435) dirty exists 2020-01-16T00:13:37.525 INFO:tasks.rados.rados.0.smithi123.stdout:1364: left oid 9 (ObjNum 435 snap 0 seq_num 435) 2020-01-16T00:13:37.526 INFO:tasks.rados.rados.0.smithi123.stdout:1358: done (15 left) 2020-01-16T00:13:37.526 INFO:tasks.rados.rados.0.smithi123.stdout:1361: done (14 left) 2020-01-16T00:13:37.526 INFO:tasks.rados.rados.0.smithi123.stdout:1362: done (13 left) 2020-01-16T00:13:37.526 INFO:tasks.rados.rados.0.smithi123.stdout:1363: done (12 left) 2020-01-16T00:13:37.526 INFO:tasks.rados.rados.0.smithi123.stdout:1364: done (11 left) 2020-01-16T00:13:37.526 INFO:tasks.rados.rados.0.smithi123.stdout:1377: read oid 36 snap -1 2020-01-16T00:13:37.526 INFO:tasks.rados.rados.0.smithi123.stdout:1377: expect (ObjNum 422 snap 0 seq_num 422) 2020-01-16T00:13:37.526 INFO:tasks.rados.rados.0.smithi123.stdout:1378: read oid 16 snap -1 2020-01-16T00:13:37.527 INFO:tasks.rados.rados.0.smithi123.stdout:1378: expect (ObjNum 416 snap 0 seq_num 416) 2020-01-16T00:13:37.527 INFO:tasks.rados.rados.0.smithi123.stdout:1366: expect (ObjNum 418 snap 0 seq_num 418) 2020-01-16T00:13:37.527 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:37.522+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:37.586 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:37.596 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-01-16T00:13:37.606 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:37.600+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:37.706 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:37.702+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:37.737 INFO:tasks.rados.rados.0.smithi123.stdout:1365: expect (ObjNum 403 snap 0 seq_num 403) 2020-01-16T00:13:37.737 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:37.746 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-01-16T00:13:37.808 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:37.804+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:37.889 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:37.897 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-01-16T00:13:37.946 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:37.942+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:37.949 INFO:tasks.rados.rados.0.smithi123.stdout:1367: finishing write tid 1 to smithi12313934-30 2020-01-16T00:13:37.949 INFO:tasks.rados.rados.0.smithi123.stdout:1379: read oid 13 snap -1 2020-01-16T00:13:37.949 INFO:tasks.rados.rados.0.smithi123.stdout:1379: expect (ObjNum 392 snap 0 seq_num 392) 2020-01-16T00:13:37.949 INFO:tasks.rados.rados.0.smithi123.stdout:1365: done (13 left) 2020-01-16T00:13:37.950 INFO:tasks.rados.rados.0.smithi123.stdout:1366: done (12 left) 2020-01-16T00:13:37.950 INFO:tasks.rados.rados.0.smithi123.stdout:1380: write (excl) oid 41 current snap is 0 2020-01-16T00:13:37.950 INFO:tasks.rados.rados.0.smithi123.stdout:1380: seq_num 441 ranges {611887=733283,1972912=456484,2894128=517759} 2020-01-16T00:13:37.955 INFO:tasks.rados.rados.0.smithi123.stdout:1380: writing smithi12313934-41 from 611887 to 1345170 tid 1 2020-01-16T00:13:37.961 INFO:tasks.rados.rados.0.smithi123.stdout:1380: writing smithi12313934-41 from 1972912 to 2429396 tid 2 2020-01-16T00:13:37.968 INFO:tasks.rados.rados.0.smithi123.stdout:1380: writing smithi12313934-41 from 2894128 to 3411887 tid 3 2020-01-16T00:13:37.968 INFO:tasks.rados.rados.0.smithi123.stdout:1381: read oid 15 snap -1 2020-01-16T00:13:37.968 INFO:tasks.rados.rados.0.smithi123.stdout:1381: expect (ObjNum 384 snap 0 seq_num 384) 2020-01-16T00:13:37.968 INFO:tasks.rados.rados.0.smithi123.stdout:1382: write oid 29 current snap is 0 2020-01-16T00:13:37.969 INFO:tasks.rados.rados.0.smithi123.stdout:1382: seq_num 442 ranges {628694=408532,1515041=600142,2805198=552112} 2020-01-16T00:13:37.973 INFO:tasks.rados.rados.0.smithi123.stdout:1382: writing smithi12313934-29 from 628694 to 1037226 tid 1 2020-01-16T00:13:37.980 INFO:tasks.rados.rados.0.smithi123.stdout:1382: writing smithi12313934-29 from 1515041 to 2115183 tid 2 2020-01-16T00:13:37.990 INFO:tasks.rados.rados.0.smithi123.stdout:1382: writing smithi12313934-29 from 2805198 to 3357310 tid 3 2020-01-16T00:13:37.990 INFO:tasks.rados.rados.0.smithi123.stdout:1383: write oid 48 current snap is 0 2020-01-16T00:13:37.991 INFO:tasks.rados.rados.0.smithi123.stdout:1383: seq_num 443 ranges {645501=483781,1573523=460153,2548974=763384} 2020-01-16T00:13:37.993 INFO:tasks.rados.rados.0.smithi123.stdout:1383: writing smithi12313934-48 from 645501 to 1129282 tid 1 2020-01-16T00:13:38.004 INFO:tasks.rados.rados.0.smithi123.stdout:1383: writing smithi12313934-48 from 1573523 to 2033676 tid 2 2020-01-16T00:13:38.006 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:38.006 INFO:tasks.rados.rados.0.smithi123.stdout:1383: writing smithi12313934-48 from 2548974 to 3312358 tid 3 2020-01-16T00:13:38.006 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:38.006 INFO:tasks.rados.rados.0.smithi123.stdout:1367: finishing write tid 2 to smithi12313934-30 2020-01-16T00:13:38.007 INFO:tasks.rados.rados.0.smithi123.stdout:1367: finishing write tid 3 to smithi12313934-30 2020-01-16T00:13:38.007 INFO:tasks.rados.rados.0.smithi123.stdout:1369: expect (ObjNum 380 snap 0 seq_num 380) 2020-01-16T00:13:38.009 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:38.006+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:38.016 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-01-16T00:13:38.111 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:38.106+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:38.153 INFO:tasks.rados.rados.0.smithi123.stdout:1370: finishing write tid 1 to smithi12313934-31 2020-01-16T00:13:38.154 INFO:tasks.rados.rados.0.smithi123.stdout:1367: finishing write tid 5 to smithi12313934-30 2020-01-16T00:13:38.154 INFO:tasks.rados.rados.0.smithi123.stdout:1367: finishing write tid 6 to smithi12313934-30 2020-01-16T00:13:38.154 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 30 v 141 (ObjNum 436 snap 0 seq_num 436) dirty exists 2020-01-16T00:13:38.154 INFO:tasks.rados.rados.0.smithi123.stdout:1367: left oid 30 (ObjNum 436 snap 0 seq_num 436) 2020-01-16T00:13:38.154 INFO:tasks.rados.rados.0.smithi123.stdout:1370: finishing write tid 2 to smithi12313934-31 2020-01-16T00:13:38.154 INFO:tasks.rados.rados.0.smithi123.stdout:1370: finishing write tid 3 to smithi12313934-31 2020-01-16T00:13:38.155 INFO:tasks.rados.rados.0.smithi123.stdout:1370: finishing write tid 5 to smithi12313934-31 2020-01-16T00:13:38.155 INFO:tasks.rados.rados.0.smithi123.stdout:1370: finishing write tid 6 to smithi12313934-31 2020-01-16T00:13:38.155 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 31 v 167 (ObjNum 437 snap 0 seq_num 437) dirty exists 2020-01-16T00:13:38.155 INFO:tasks.rados.rados.0.smithi123.stdout:1370: left oid 31 (ObjNum 437 snap 0 seq_num 437) 2020-01-16T00:13:38.155 INFO:tasks.rados.rados.0.smithi123.stdout:1371: finishing write tid 1 to smithi12313934-25 2020-01-16T00:13:38.155 INFO:tasks.rados.rados.0.smithi123.stdout:1371: finishing write tid 2 to smithi12313934-25 2020-01-16T00:13:38.155 INFO:tasks.rados.rados.0.smithi123.stdout:1371: finishing write tid 3 to smithi12313934-25 2020-01-16T00:13:38.156 INFO:tasks.rados.rados.0.smithi123.stdout:1371: finishing write tid 5 to smithi12313934-25 2020-01-16T00:13:38.156 INFO:tasks.rados.rados.0.smithi123.stdout:1367: done (15 left) 2020-01-16T00:13:38.156 INFO:tasks.rados.rados.0.smithi123.stdout:1368: done (14 left) 2020-01-16T00:13:38.156 INFO:tasks.rados.rados.0.smithi123.stdout:1369: done (13 left) 2020-01-16T00:13:38.156 INFO:tasks.rados.rados.0.smithi123.stdout:1370: done (12 left) 2020-01-16T00:13:38.156 INFO:tasks.rados.rados.0.smithi123.stdout:1384: read oid 49 snap -1 2020-01-16T00:13:38.156 INFO:tasks.rados.rados.0.smithi123.stdout:1384: expect (ObjNum 404 snap 0 seq_num 404) 2020-01-16T00:13:38.156 INFO:tasks.rados.rados.0.smithi123.stdout:1385: write oid 44 current snap is 0 2020-01-16T00:13:38.157 INFO:tasks.rados.rados.0.smithi123.stdout:1385: seq_num 444 ranges {662308=559030,1632005=603811,2860044=602264} 2020-01-16T00:13:38.165 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:38.170 INFO:tasks.rados.rados.0.smithi123.stdout:1385: writing smithi12313934-44 from 662308 to 1221338 tid 1 2020-01-16T00:13:38.176 INFO:tasks.rados.rados.0.smithi123.stdout:1385: writing smithi12313934-44 from 1632005 to 2235816 tid 2 2020-01-16T00:13:38.177 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-01-16T00:13:38.184 INFO:tasks.rados.rados.0.smithi123.stdout:1385: writing smithi12313934-44 from 2860044 to 3462308 tid 3 2020-01-16T00:13:38.185 INFO:tasks.rados.rados.0.smithi123.stdout:1386: write oid 18 current snap is 0 2020-01-16T00:13:38.185 INFO:tasks.rados.rados.0.smithi123.stdout:1371: finishing write tid 6 to smithi12313934-25 2020-01-16T00:13:38.185 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 25 v 211 (ObjNum 438 snap 0 seq_num 438) dirty exists 2020-01-16T00:13:38.185 INFO:tasks.rados.rados.0.smithi123.stdout:1371: left oid 25 (ObjNum 438 snap 0 seq_num 438) 2020-01-16T00:13:38.186 INFO:tasks.rados.rados.0.smithi123.stdout:1386: seq_num 445 ranges {679115=634279,2090487=463822,3003820=475295} 2020-01-16T00:13:38.189 INFO:tasks.rados.rados.0.smithi123.stdout:1386: writing smithi12313934-18 from 679115 to 1313394 tid 1 2020-01-16T00:13:38.196 INFO:tasks.rados.rados.0.smithi123.stdout:1386: writing smithi12313934-18 from 2090487 to 2554309 tid 2 2020-01-16T00:13:38.203 INFO:tasks.rados.rados.0.smithi123.stdout:1386: writing smithi12313934-18 from 3003820 to 3479115 tid 3 2020-01-16T00:13:38.203 INFO:tasks.rados.rados.0.smithi123.stdout:1371: done (14 left) 2020-01-16T00:13:38.203 INFO:tasks.rados.rados.0.smithi123.stdout:1387: read oid 34 snap -1 2020-01-16T00:13:38.203 INFO:tasks.rados.rados.0.smithi123.stdout:1387: expect deleted 2020-01-16T00:13:38.203 INFO:tasks.rados.rados.0.smithi123.stdout:1388: write (excl) oid 27 current snap is 0 2020-01-16T00:13:38.203 INFO:tasks.rados.rados.0.smithi123.stdout:1388: seq_num 446 ranges {695922=709528,2032616=607480,3198537=297385} 2020-01-16T00:13:38.211 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:38.206+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:38.211 INFO:tasks.rados.rados.0.smithi123.stdout:1388: writing smithi12313934-27 from 695922 to 1405450 tid 1 2020-01-16T00:13:38.219 INFO:tasks.rados.rados.0.smithi123.stdout:1388: writing smithi12313934-27 from 2032616 to 2640096 tid 2 2020-01-16T00:13:38.225 INFO:tasks.rados.rados.0.smithi123.stdout:1388: writing smithi12313934-27 from 3198537 to 3495922 tid 3 2020-01-16T00:13:38.225 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:38.225 INFO:tasks.rados.rados.0.smithi123.stdout:1374: expect (ObjNum 342 snap 0 seq_num 342) 2020-01-16T00:13:38.284 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:38.292 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-01-16T00:13:38.339 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:38.336+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:38.402 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:38.411 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-01-16T00:13:38.414 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:38.410+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:38.482 INFO:tasks.rados.rados.0.smithi123.stdout:1375: finishing write tid 1 to smithi12313934-26 2020-01-16T00:13:38.483 INFO:tasks.rados.rados.0.smithi123.stdout:1375: finishing write tid 2 to smithi12313934-26 2020-01-16T00:13:38.483 INFO:tasks.rados.rados.0.smithi123.stdout:1376: expect (ObjNum 429 snap 0 seq_num 429) 2020-01-16T00:13:38.515 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:38.510+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:38.535 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-01-16T00:13:38.565 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:38.574 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-01-16T00:13:38.623 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:38.614+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:38.662 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:13:38.670 INFO:tasks.rados.rados.0.smithi123.stdout:1375: finishing write tid 3 to smithi12313934-26 2020-01-16T00:13:38.670 INFO:tasks.rados.rados.0.smithi123.stdout:1375: finishing write tid 5 to smithi12313934-26 2020-01-16T00:13:38.670 INFO:tasks.rados.rados.0.smithi123.stdout:1375: finishing write tid 6 to smithi12313934-26 2020-01-16T00:13:38.670 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 26 v 254 (ObjNum 440 snap 0 seq_num 440) dirty exists 2020-01-16T00:13:38.670 INFO:tasks.rados.rados.0.smithi123.stdout:1375: left oid 26 (ObjNum 440 snap 0 seq_num 440) 2020-01-16T00:13:38.670 INFO:tasks.rados.rados.0.smithi123.stdout:1377: expect (ObjNum 422 snap 0 seq_num 422) 2020-01-16T00:13:38.679 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:13:38.683 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:38.693 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-01-16T00:13:38.698 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:13:38.715 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:13:38.717 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:38.714+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:38.740 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:13:38.756 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:13:38.777 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:13:38.779 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-01-16T00:13:38.779 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-01-16T00:13:38.779 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-01-16T00:13:38.780 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-01-16T00:13:38.780 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-01-16T00:13:38.780 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENXIO: problem getting command descriptions from osd.5 2020-01-16T00:13:38.780 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: problem getting command descriptions from osd.5 2020-01-16T00:13:38.780 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-01-16T00:13:38.780 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-01-16T00:13:38.789 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-01-16T00:13:38.789 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-01-16T00:13:38.798 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2020-01-16T00:13:38.801 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:38.801 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:13:38.801 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:13:38.801 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:38.811 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-01-16T00:13:38.832 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:38.830+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:38.841 INFO:tasks.rados.rados.0.smithi123.stdout:1378: expect (ObjNum 416 snap 0 seq_num 416) 2020-01-16T00:13:38.914 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:38.914 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:13:38.915 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:13:38.915 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:13:38.915 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:38.918 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:38.914+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:38.924 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-01-16T00:13:39.013 INFO:tasks.rados.rados.0.smithi123.stdout:1379: expect (ObjNum 392 snap 0 seq_num 392) 2020-01-16T00:13:39.019 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:39.014+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:39.042 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:39.042 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:13:39.042 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:13:39.042 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:13:39.043 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.043 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.5.13:36 2.0 2:20e40cec:::smithi12313934-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 113152~1699840] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:13:39.043 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:13:39.043 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 61.205791972, 2020-01-16T00:13:39.043 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011736636, 2020-01-16T00:13:39.043 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.043 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:39.044 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.044 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.044 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.044 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:13:39.044 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.044 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.044 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.045 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.045 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:13:39.045 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999952 2020-01-16T00:13:39.045 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.045 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.045 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.046 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831934+0000", 2020-01-16T00:13:39.046 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001034325 2020-01-16T00:13:39.046 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.046 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.046 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.046 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.832968+0000", 2020-01-16T00:13:39.047 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.506e-05 2020-01-16T00:13:39.047 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.047 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.047 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.047 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833004+0000", 2020-01-16T00:13:39.047 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3406e-05 2020-01-16T00:13:39.048 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.048 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.048 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.048 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833017+0000", 2020-01-16T00:13:39.048 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015477000000000001 2020-01-16T00:13:39.048 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.048 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.048 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.049 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833172+0000", 2020-01-16T00:13:39.049 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033750700000000002 2020-01-16T00:13:39.049 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.049 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.049 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.049 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833509+0000", 2020-01-16T00:13:39.049 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00055861400000000003 2020-01-16T00:13:39.050 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.050 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.050 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:13:39.050 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.834068+0000", 2020-01-16T00:13:39.050 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.007474545 2020-01-16T00:13:39.050 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.050 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.050 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:39.051 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.841542+0000", 2020-01-16T00:13:39.051 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00056319499999999995 2020-01-16T00:13:39.051 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.051 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.051 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.051 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.842106+0000", 2020-01-16T00:13:39.051 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001371017 2020-01-16T00:13:39.052 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.052 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.052 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.052 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843477+0000", 2020-01-16T00:13:39.052 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000123938 2020-01-16T00:13:39.052 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.053 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.054 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:39.054 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843601+0000", 2020-01-16T00:13:39.054 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5012000000000006e-05 2020-01-16T00:13:39.054 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.054 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.054 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.054 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843676+0000", 2020-01-16T00:13:39.055 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011736636 2020-01-16T00:13:39.055 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.055 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.055 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.055 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.055 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.055 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1187 3.d 3:b1521a3e:::smithi12313934-25:head [write 1721042~423699] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:13:39.058 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.263828+0000", 2020-01-16T00:13:39.058 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 56.773902900000003, 2020-01-16T00:13:39.059 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011305709000000001, 2020-01-16T00:13:39.059 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.059 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:39.059 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:39.059 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:39.059 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:39.059 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1187 2020-01-16T00:13:39.060 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.060 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.060 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.060 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.060 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.263828+0000", 2020-01-16T00:13:39.060 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.060 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.061 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.061 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.061 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.263828+0000", 2020-01-16T00:13:39.061 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:13:39.061 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.061 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.061 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.062 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.263824+0000", 2020-01-16T00:13:39.062 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00048485899999999997 2020-01-16T00:13:39.062 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.062 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.062 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.062 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264309+0000", 2020-01-16T00:13:39.062 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5622999999999999e-05 2020-01-16T00:13:39.063 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.063 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.063 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.063 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264325+0000", 2020-01-16T00:13:39.063 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2530000000000004e-06 2020-01-16T00:13:39.063 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.063 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.063 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.064 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264330+0000", 2020-01-16T00:13:39.064 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000102252 2020-01-16T00:13:39.064 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.064 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.064 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.064 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264432+0000", 2020-01-16T00:13:39.064 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022361000000000001 2020-01-16T00:13:39.064 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.065 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.065 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:39.065 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264656+0000", 2020-01-16T00:13:39.065 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0028688849999999998 2020-01-16T00:13:39.065 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.065 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.065 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.066 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.267525+0000", 2020-01-16T00:13:39.066 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00023640900000000001 2020-01-16T00:13:39.066 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.066 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.066 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.066 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.267761+0000", 2020-01-16T00:13:39.066 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00035667000000000001 2020-01-16T00:13:39.066 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.067 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.067 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:39.067 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.268118+0000", 2020-01-16T00:13:39.067 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023898040000000001 2020-01-16T00:13:39.067 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.067 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.067 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:39.068 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.270507+0000", 2020-01-16T00:13:39.068 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014919690000000001 2020-01-16T00:13:39.068 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.068 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.068 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.068 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.271999+0000", 2020-01-16T00:13:39.069 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0029764510000000002 2020-01-16T00:13:39.069 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.069 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.069 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.069 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.274976+0000", 2020-01-16T00:13:39.069 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5409999999999997e-05 2020-01-16T00:13:39.069 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.070 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.070 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:39.070 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.275061+0000", 2020-01-16T00:13:39.070 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2452000000000004e-05 2020-01-16T00:13:39.070 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.070 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.070 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.070 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.275134+0000", 2020-01-16T00:13:39.071 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011305709000000001 2020-01-16T00:13:39.071 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.071 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.071 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.074 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.076 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.076 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1276 3.d 3:b1521a3e:::smithi12313934-25:head [read 0~3882384,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:13:39.076 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:13:39.077 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 55.963032402000003, 2020-01-16T00:13:39.077 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015839157999999999, 2020-01-16T00:13:39.077 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.077 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:39.078 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:39.078 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:39.078 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:39.079 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1276 2020-01-16T00:13:39.079 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.079 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.080 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.080 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.080 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:13:39.080 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.081 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.081 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.081 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.081 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:13:39.081 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:39.082 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.082 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.082 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.082 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:13:39.083 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5429999999999999e-06 2020-01-16T00:13:39.083 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.083 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.083 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.083 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074704+0000", 2020-01-16T00:13:39.084 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0839999999999992e-06 2020-01-16T00:13:39.084 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.084 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.084 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.085 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074712+0000", 2020-01-16T00:13:39.085 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.173e-06 2020-01-16T00:13:39.085 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.086 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.086 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.086 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074718+0000", 2020-01-16T00:13:39.086 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0083034700000000003 2020-01-16T00:13:39.087 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.087 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.087 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.087 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.083022+0000", 2020-01-16T00:13:39.088 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028456400000000001 2020-01-16T00:13:39.088 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.088 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.088 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.088 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.083306+0000", 2020-01-16T00:13:39.089 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0072312640000000003 2020-01-16T00:13:39.089 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.089 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.090 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.090 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090538+0000", 2020-01-16T00:13:39.090 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015839157999999999 2020-01-16T00:13:39.090 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.091 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.091 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.091 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.091 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.091 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1277 3.d 3:b1521a3e:::smithi12313934-25:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:13:39.092 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:13:39.092 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 55.962988811999999, 2020-01-16T00:13:39.092 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016646459999999998, 2020-01-16T00:13:39.092 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.092 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:39.092 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:39.092 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:39.092 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:39.093 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1277 2020-01-16T00:13:39.093 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.093 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.093 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.093 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.093 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:13:39.093 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.093 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.094 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.094 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.094 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:13:39.094 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:39.094 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.094 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.094 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.094 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074741+0000", 2020-01-16T00:13:39.095 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0776e-05 2020-01-16T00:13:39.095 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.095 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.095 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.095 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074752+0000", 2020-01-16T00:13:39.095 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7600000000000002e-06 2020-01-16T00:13:39.095 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.095 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.096 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.096 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074759+0000", 2020-01-16T00:13:39.096 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6840000000000001e-06 2020-01-16T00:13:39.096 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.096 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.096 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.096 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074765+0000", 2020-01-16T00:13:39.096 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015815216 2020-01-16T00:13:39.097 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.097 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.097 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.097 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090580+0000", 2020-01-16T00:13:39.097 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000182404 2020-01-16T00:13:39.097 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.097 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.098 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.098 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090763+0000", 2020-01-16T00:13:39.098 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00062579300000000003 2020-01-16T00:13:39.098 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.098 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.098 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.098 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091389+0000", 2020-01-16T00:13:39.098 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016646459999999998 2020-01-16T00:13:39.099 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.099 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.099 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.099 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.099 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.100 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1278 3.d 3:b1521a3e:::smithi12313934-25:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:13:39.100 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:13:39.100 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 55.962959099000003, 2020-01-16T00:13:39.100 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017994915, 2020-01-16T00:13:39.100 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.100 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:39.100 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:39.101 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:39.101 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:39.101 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1278 2020-01-16T00:13:39.101 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.101 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.101 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.101 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.101 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:13:39.102 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.102 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.102 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.102 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.102 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:13:39.102 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:39.102 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.102 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.103 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.103 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074771+0000", 2020-01-16T00:13:39.103 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4569999999999998e-06 2020-01-16T00:13:39.103 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.103 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.103 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.103 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074775+0000", 2020-01-16T00:13:39.103 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0330000000000003e-06 2020-01-16T00:13:39.104 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.104 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.104 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.104 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074782+0000", 2020-01-16T00:13:39.104 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9470000000000002e-06 2020-01-16T00:13:39.104 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.104 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.104 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.105 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074787+0000", 2020-01-16T00:13:39.105 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01666134 2020-01-16T00:13:39.105 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.105 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.105 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.105 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091449+0000", 2020-01-16T00:13:39.105 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000247731 2020-01-16T00:13:39.106 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.106 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.106 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.106 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091696+0000", 2020-01-16T00:13:39.106 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010703170000000001 2020-01-16T00:13:39.106 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.106 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.106 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.107 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.092767+0000", 2020-01-16T00:13:39.107 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017994915 2020-01-16T00:13:39.107 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.107 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.107 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.107 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.107 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.107 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1872 3.d 3:bd9fc72f:::smithi12313934-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e31)", 2020-01-16T00:13:39.108 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:56.384904+0000", 2020-01-16T00:13:39.108 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 42.652826419999997, 2020-01-16T00:13:39.108 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011110386999999999, 2020-01-16T00:13:39.108 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.108 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:39.108 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:39.108 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:39.108 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:39.109 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1872 2020-01-16T00:13:39.109 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.109 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.109 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.109 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.110 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384904+0000", 2020-01-16T00:13:39.110 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.110 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.110 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.110 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.110 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384904+0000", 2020-01-16T00:13:39.111 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:39.111 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.111 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.111 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.111 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384904+0000", 2020-01-16T00:13:39.111 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3879999999999998e-06 2020-01-16T00:13:39.111 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.111 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.112 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.112 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384906+0000", 2020-01-16T00:13:39.112 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7479999999999999e-06 2020-01-16T00:13:39.112 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.112 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.112 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.112 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384910+0000", 2020-01-16T00:13:39.113 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4609999999999998e-06 2020-01-16T00:13:39.113 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.113 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.113 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.113 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.384913+0000", 2020-01-16T00:13:39.113 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00083872899999999997 2020-01-16T00:13:39.113 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.113 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.114 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.114 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.385751+0000", 2020-01-16T00:13:39.114 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3528999999999997e-05 2020-01-16T00:13:39.114 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.114 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.114 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:39.114 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.385785+0000", 2020-01-16T00:13:39.114 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0064799849999999997 2020-01-16T00:13:39.115 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.115 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.115 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.115 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.392265+0000", 2020-01-16T00:13:39.115 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010603399999999999 2020-01-16T00:13:39.115 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.116 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.116 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:39.116 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.392371+0000", 2020-01-16T00:13:39.116 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003330576 2020-01-16T00:13:39.116 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.116 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.116 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.116 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.395702+0000", 2020-01-16T00:13:39.117 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016566899999999999 2020-01-16T00:13:39.117 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.117 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.117 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.117 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.395867+0000", 2020-01-16T00:13:39.117 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014768999999999999 2020-01-16T00:13:39.117 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.117 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.118 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.118 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:56.396015+0000", 2020-01-16T00:13:39.118 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011110386999999999 2020-01-16T00:13:39.118 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.118 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.118 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.118 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.118 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.119 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.0.13:17 2.0 2:1afbc8ac:::smithi12313934-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 916992~596480] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e33)", 2020-01-16T00:13:39.119 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:01.647924+0000", 2020-01-16T00:13:39.119 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 37.389807328000003, 2020-01-16T00:13:39.119 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01120495, 2020-01-16T00:13:39.119 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.119 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:39.119 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.120 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.120 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.120 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.647924+0000", 2020-01-16T00:13:39.120 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.120 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.120 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.121 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.121 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.647924+0000", 2020-01-16T00:13:39.121 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:39.121 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.121 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.121 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.121 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.647923+0000", 2020-01-16T00:13:39.121 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00054779600000000003 2020-01-16T00:13:39.122 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.122 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.122 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.122 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.648470+0000", 2020-01-16T00:13:39.122 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2118e-05 2020-01-16T00:13:39.122 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.122 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.122 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.123 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.648483+0000", 2020-01-16T00:13:39.123 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5599999999999999e-06 2020-01-16T00:13:39.123 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.123 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.123 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.123 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.648489+0000", 2020-01-16T00:13:39.123 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3761000000000002e-05 2020-01-16T00:13:39.124 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.124 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.124 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.124 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.648543+0000", 2020-01-16T00:13:39.124 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0154e-05 2020-01-16T00:13:39.124 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.124 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.124 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:13:39.125 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.648553+0000", 2020-01-16T00:13:39.125 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0039950130000000004 2020-01-16T00:13:39.125 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.125 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.125 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.125 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.652548+0000", 2020-01-16T00:13:39.125 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014782599999999999 2020-01-16T00:13:39.125 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.126 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.126 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.126 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.652696+0000", 2020-01-16T00:13:39.126 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000212034 2020-01-16T00:13:39.126 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.126 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.126 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:13:39.127 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.652908+0000", 2020-01-16T00:13:39.127 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0047602879999999997 2020-01-16T00:13:39.127 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.127 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.127 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:39.127 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.657668+0000", 2020-01-16T00:13:39.127 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0006999999999997e-05 2020-01-16T00:13:39.127 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.128 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.128 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.128 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.657718+0000", 2020-01-16T00:13:39.128 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001303991 2020-01-16T00:13:39.128 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.128 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.129 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.129 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.659022+0000", 2020-01-16T00:13:39.129 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2401000000000001e-05 2020-01-16T00:13:39.130 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.130 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.130 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:39.130 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.659085+0000", 2020-01-16T00:13:39.130 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3877e-05 2020-01-16T00:13:39.130 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.131 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.131 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.131 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:01.659129+0000", 2020-01-16T00:13:39.131 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01120495 2020-01-16T00:13:39.131 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.131 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.131 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.131 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.132 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.132 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2260 3.d 3:b76659c4:::smithi12313934-47:head [write 601750~535546] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:13:39.132 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:06.050577+0000", 2020-01-16T00:13:39.132 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 32.987153782999997, 2020-01-16T00:13:39.132 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012415381, 2020-01-16T00:13:39.132 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.132 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:39.133 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:39.133 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:39.133 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:39.133 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2260 2020-01-16T00:13:39.133 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.133 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.133 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.133 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.134 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.050577+0000", 2020-01-16T00:13:39.134 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.134 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.134 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.134 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.134 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.050577+0000", 2020-01-16T00:13:39.134 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:39.134 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.135 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.135 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.135 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.050574+0000", 2020-01-16T00:13:39.135 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001523487 2020-01-16T00:13:39.135 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.135 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.135 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.135 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052097+0000", 2020-01-16T00:13:39.136 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3697e-05 2020-01-16T00:13:39.136 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.136 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.136 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.136 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052121+0000", 2020-01-16T00:13:39.136 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0070000000000003e-06 2020-01-16T00:13:39.136 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.136 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.137 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.137 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052130+0000", 2020-01-16T00:13:39.137 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4559000000000003e-05 2020-01-16T00:13:39.137 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.137 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.137 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.137 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052194+0000", 2020-01-16T00:13:39.138 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000165878 2020-01-16T00:13:39.138 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.138 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.138 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.138 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052360+0000", 2020-01-16T00:13:39.138 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018825499999999999 2020-01-16T00:13:39.138 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.138 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.139 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:39.139 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052549+0000", 2020-01-16T00:13:39.139 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002761782 2020-01-16T00:13:39.139 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.139 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.139 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.139 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.055310+0000", 2020-01-16T00:13:39.139 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00060385500000000004 2020-01-16T00:13:39.140 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.140 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.140 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:39.140 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.055914+0000", 2020-01-16T00:13:39.140 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0069733809999999999 2020-01-16T00:13:39.140 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.141 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.141 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.141 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.062888+0000", 2020-01-16T00:13:39.141 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2884999999999999e-05 2020-01-16T00:13:39.141 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.141 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.141 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:39.141 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.062940+0000", 2020-01-16T00:13:39.142 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1996999999999997e-05 2020-01-16T00:13:39.142 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.142 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.142 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.142 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.062992+0000", 2020-01-16T00:13:39.142 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012415381 2020-01-16T00:13:39.142 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.142 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.143 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.143 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.143 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.143 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2295 3.d 3:b1521a3e:::smithi12313934-25:head [write 1763838~505140] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:13:39.143 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:06.660258+0000", 2020-01-16T00:13:39.143 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 32.377472556000001, 2020-01-16T00:13:39.143 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011681408000000001, 2020-01-16T00:13:39.143 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.144 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:39.144 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:39.144 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:39.144 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:39.144 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2295 2020-01-16T00:13:39.144 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.144 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.145 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.145 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.145 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660258+0000", 2020-01-16T00:13:39.145 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.145 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.145 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.145 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.146 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660258+0000", 2020-01-16T00:13:39.146 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:39.146 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.146 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.146 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.146 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660256+0000", 2020-01-16T00:13:39.146 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00066501699999999995 2020-01-16T00:13:39.147 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.147 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.147 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.147 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660921+0000", 2020-01-16T00:13:39.147 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7339000000000001e-05 2020-01-16T00:13:39.147 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.147 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.147 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.148 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660938+0000", 2020-01-16T00:13:39.148 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0219999999999993e-06 2020-01-16T00:13:39.148 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.148 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.148 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.148 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660947+0000", 2020-01-16T00:13:39.148 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00094237200000000002 2020-01-16T00:13:39.148 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.149 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.149 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.149 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.661890+0000", 2020-01-16T00:13:39.149 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000117649 2020-01-16T00:13:39.149 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.149 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.149 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:39.149 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662007+0000", 2020-01-16T00:13:39.150 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00060662599999999998 2020-01-16T00:13:39.150 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.150 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.150 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.150 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662614+0000", 2020-01-16T00:13:39.150 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028424899999999998 2020-01-16T00:13:39.150 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.150 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.151 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.151 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662898+0000", 2020-01-16T00:13:39.152 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00054904599999999997 2020-01-16T00:13:39.152 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.152 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.152 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:39.152 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.663447+0000", 2020-01-16T00:13:39.152 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0051479990000000003 2020-01-16T00:13:39.152 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.153 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.153 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.153 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.668595+0000", 2020-01-16T00:13:39.153 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00080881799999999999 2020-01-16T00:13:39.153 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.153 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.153 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.154 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.669404+0000", 2020-01-16T00:13:39.154 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0024495290000000002 2020-01-16T00:13:39.154 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.154 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.154 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:39.154 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671854+0000", 2020-01-16T00:13:39.154 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6635000000000003e-05 2020-01-16T00:13:39.154 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.155 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.155 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:39.155 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671900+0000", 2020-01-16T00:13:39.155 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9567999999999998e-05 2020-01-16T00:13:39.155 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.155 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.155 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.155 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671940+0000", 2020-01-16T00:13:39.156 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011681408000000001 2020-01-16T00:13:39.156 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.156 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.156 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.156 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.156 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.156 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2296 3.d 3:b1521a3e:::smithi12313934-25:head [setxattr _header (56) in=7b,truncate 2268978] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:13:39.157 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:06.660958+0000", 2020-01-16T00:13:39.157 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 32.376773045, 2020-01-16T00:13:39.157 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011054015, 2020-01-16T00:13:39.157 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.157 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:39.157 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:39.157 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:39.157 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:39.158 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2296 2020-01-16T00:13:39.158 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.158 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.158 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.158 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.159 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660958+0000", 2020-01-16T00:13:39.159 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.159 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.159 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.159 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.159 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660958+0000", 2020-01-16T00:13:39.160 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:39.160 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.160 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.160 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.160 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660957+0000", 2020-01-16T00:13:39.160 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3699999999999995e-06 2020-01-16T00:13:39.160 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.161 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.161 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.161 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660965+0000", 2020-01-16T00:13:39.161 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3590000000000008e-06 2020-01-16T00:13:39.161 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.161 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.161 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.161 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660973+0000", 2020-01-16T00:13:39.162 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9880000000000001e-06 2020-01-16T00:13:39.162 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.162 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.162 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.162 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660979+0000", 2020-01-16T00:13:39.162 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001100136 2020-01-16T00:13:39.162 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.163 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.163 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.163 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662079+0000", 2020-01-16T00:13:39.163 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013484700000000001 2020-01-16T00:13:39.163 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.163 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.163 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:39.163 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662214+0000", 2020-01-16T00:13:39.164 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0048090349999999997 2020-01-16T00:13:39.164 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.164 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.164 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.164 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.667023+0000", 2020-01-16T00:13:39.164 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012664600000000001 2020-01-16T00:13:39.164 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.164 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.165 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.165 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.667150+0000", 2020-01-16T00:13:39.165 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000235911 2020-01-16T00:13:39.165 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.165 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.165 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:39.165 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.667386+0000", 2020-01-16T00:13:39.165 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0017131500000000001 2020-01-16T00:13:39.166 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.166 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.166 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.166 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.669099+0000", 2020-01-16T00:13:39.166 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001018866 2020-01-16T00:13:39.166 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.166 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.167 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.167 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.670118+0000", 2020-01-16T00:13:39.167 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001828585 2020-01-16T00:13:39.167 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.167 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.167 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:39.167 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671947+0000", 2020-01-16T00:13:39.167 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2214e-05 2020-01-16T00:13:39.168 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.168 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.168 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:39.168 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671979+0000", 2020-01-16T00:13:39.168 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3135e-05 2020-01-16T00:13:39.168 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.168 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.168 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.169 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.672012+0000", 2020-01-16T00:13:39.169 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011054015 2020-01-16T00:13:39.169 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.169 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.169 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.169 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.169 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.169 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2297 3.d 3:b1521a3e:::smithi12313934-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e35)", 2020-01-16T00:13:39.170 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:06.660986+0000", 2020-01-16T00:13:39.170 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 32.376744698000003, 2020-01-16T00:13:39.170 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011220677, 2020-01-16T00:13:39.170 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.170 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:39.170 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:39.170 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:39.170 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:39.171 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2297 2020-01-16T00:13:39.171 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.171 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.171 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.171 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.171 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660986+0000", 2020-01-16T00:13:39.171 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.172 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.172 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.172 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.172 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660986+0000", 2020-01-16T00:13:39.172 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:39.172 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.172 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.172 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.173 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660985+0000", 2020-01-16T00:13:39.173 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8760000000000004e-06 2020-01-16T00:13:39.173 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.173 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.173 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.173 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660990+0000", 2020-01-16T00:13:39.173 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.4190000000000002e-06 2020-01-16T00:13:39.173 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.174 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.174 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.174 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660999+0000", 2020-01-16T00:13:39.174 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7740000000000001e-06 2020-01-16T00:13:39.174 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.174 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.174 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.174 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.661004+0000", 2020-01-16T00:13:39.175 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0073783590000000001 2020-01-16T00:13:39.175 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.175 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.175 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.175 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.668383+0000", 2020-01-16T00:13:39.175 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014443 2020-01-16T00:13:39.175 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.176 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.176 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:39.176 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.668527+0000", 2020-01-16T00:13:39.176 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0035336539999999998 2020-01-16T00:13:39.176 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.176 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.176 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.176 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.672061+0000", 2020-01-16T00:13:39.177 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6445000000000005e-05 2020-01-16T00:13:39.177 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.177 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.177 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.177 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.672127+0000", 2020-01-16T00:13:39.177 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.9497000000000001e-05 2020-01-16T00:13:39.177 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.177 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.178 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.178 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.672207+0000", 2020-01-16T00:13:39.178 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011220677 2020-01-16T00:13:39.178 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.178 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.178 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.178 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.179 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.179 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.3.13:109 2.0 2:11fa79bc:::smithi12313934-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 331264~198656] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e42)", 2020-01-16T00:13:39.179 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:21.840075+0000", 2020-01-16T00:13:39.179 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 17.197656258999999, 2020-01-16T00:13:39.179 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011558677999999999, 2020-01-16T00:13:39.179 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.179 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:39.180 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.180 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.180 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.180 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840075+0000", 2020-01-16T00:13:39.180 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.180 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.180 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.181 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.181 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840075+0000", 2020-01-16T00:13:39.181 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:39.181 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.181 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.181 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.181 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840073+0000", 2020-01-16T00:13:39.181 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033576299999999999 2020-01-16T00:13:39.182 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.182 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.182 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.182 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840409+0000", 2020-01-16T00:13:39.182 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2082999999999998e-05 2020-01-16T00:13:39.182 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.182 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.182 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.183 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840431+0000", 2020-01-16T00:13:39.183 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4410000000000003e-06 2020-01-16T00:13:39.183 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.183 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.183 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.183 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840436+0000", 2020-01-16T00:13:39.183 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000101016 2020-01-16T00:13:39.183 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.184 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.184 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.184 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840537+0000", 2020-01-16T00:13:39.184 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000117639 2020-01-16T00:13:39.184 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.184 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.184 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.184 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840654+0000", 2020-01-16T00:13:39.185 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000196162 2020-01-16T00:13:39.185 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.185 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.185 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:13:39.185 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840850+0000", 2020-01-16T00:13:39.185 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003319092 2020-01-16T00:13:39.185 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.186 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.186 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:39.186 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.844169+0000", 2020-01-16T00:13:39.186 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011349229999999999 2020-01-16T00:13:39.186 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.186 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.186 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.186 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.845304+0000", 2020-01-16T00:13:39.187 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0062517670000000001 2020-01-16T00:13:39.187 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.187 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.187 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.187 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851556+0000", 2020-01-16T00:13:39.187 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4138000000000003e-05 2020-01-16T00:13:39.187 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.188 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.188 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:39.188 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851600+0000", 2020-01-16T00:13:39.188 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2993000000000001e-05 2020-01-16T00:13:39.188 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.188 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.188 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.188 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851633+0000", 2020-01-16T00:13:39.189 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011558677999999999 2020-01-16T00:13:39.189 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.189 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.189 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.189 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.189 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.189 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.3.13:110 2.0 2:11fa79bc:::smithi12313934-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 752640~650240] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e42)", 2020-01-16T00:13:39.189 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:21.840457+0000", 2020-01-16T00:13:39.190 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 17.197273967000001, 2020-01-16T00:13:39.190 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011329271, 2020-01-16T00:13:39.190 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.190 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:39.190 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.190 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.190 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.190 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840457+0000", 2020-01-16T00:13:39.191 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.191 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.191 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.191 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.191 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840457+0000", 2020-01-16T00:13:39.191 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:39.191 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.191 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.192 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.192 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840456+0000", 2020-01-16T00:13:39.192 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00062607899999999998 2020-01-16T00:13:39.192 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.192 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.192 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.192 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841082+0000", 2020-01-16T00:13:39.193 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1239999999999999e-06 2020-01-16T00:13:39.193 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.193 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.193 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.193 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841090+0000", 2020-01-16T00:13:39.193 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7129999999999999e-06 2020-01-16T00:13:39.193 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.193 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.194 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.194 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841093+0000", 2020-01-16T00:13:39.194 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00075997800000000002 2020-01-16T00:13:39.194 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.194 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.194 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.194 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841853+0000", 2020-01-16T00:13:39.194 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0690000000000003e-05 2020-01-16T00:13:39.195 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.195 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.195 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.195 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841924+0000", 2020-01-16T00:13:39.195 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000115753 2020-01-16T00:13:39.195 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.195 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.195 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:13:39.196 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.842040+0000", 2020-01-16T00:13:39.196 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0021358890000000002 2020-01-16T00:13:39.196 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.196 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.196 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:39.196 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.844176+0000", 2020-01-16T00:13:39.196 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0042541870000000004 2020-01-16T00:13:39.196 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.197 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.197 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.197 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.848430+0000", 2020-01-16T00:13:39.197 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0032476929999999998 2020-01-16T00:13:39.197 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.197 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.197 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.198 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851677+0000", 2020-01-16T00:13:39.198 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0370000000000001e-05 2020-01-16T00:13:39.198 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.198 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.198 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:39.198 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851728+0000", 2020-01-16T00:13:39.198 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.834e-05 2020-01-16T00:13:39.198 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.199 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.199 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.199 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851786+0000", 2020-01-16T00:13:39.199 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011329271 2020-01-16T00:13:39.199 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.199 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.199 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.199 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.200 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.200 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2987 3.d 3:b3dd5d45:::smithi12313934-16:head [write 467327~633480] snapc 0=[] ondisk+write+known_if_redirected e42)", 2020-01-16T00:13:39.200 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:22.257904+0000", 2020-01-16T00:13:39.200 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 16.779826672999999, 2020-01-16T00:13:39.200 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012187450000000001, 2020-01-16T00:13:39.200 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.200 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:39.200 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:39.201 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:39.201 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:39.201 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2987 2020-01-16T00:13:39.201 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.201 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.201 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.201 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.202 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.257904+0000", 2020-01-16T00:13:39.202 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.202 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.202 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.202 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.202 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.257904+0000", 2020-01-16T00:13:39.202 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999933 2020-01-16T00:13:39.202 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.203 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.203 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.203 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.257898+0000", 2020-01-16T00:13:39.203 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00097099800000000004 2020-01-16T00:13:39.203 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.203 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.203 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.203 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.258869+0000", 2020-01-16T00:13:39.204 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7970000000000003e-05 2020-01-16T00:13:39.204 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.204 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.204 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.204 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.258917+0000", 2020-01-16T00:13:39.204 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5982999999999999e-05 2020-01-16T00:13:39.204 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.204 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.205 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.205 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.258933+0000", 2020-01-16T00:13:39.205 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00026239999999999998 2020-01-16T00:13:39.205 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.205 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.205 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.205 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.259195+0000", 2020-01-16T00:13:39.205 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024907400000000001 2020-01-16T00:13:39.206 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.206 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.206 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.206 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.259444+0000", 2020-01-16T00:13:39.206 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00034824999999999998 2020-01-16T00:13:39.206 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.206 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.206 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:39.207 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.259792+0000", 2020-01-16T00:13:39.207 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003251538 2020-01-16T00:13:39.207 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.207 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.207 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:39.207 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.263044+0000", 2020-01-16T00:13:39.207 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0068502880000000004 2020-01-16T00:13:39.207 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.208 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.208 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.208 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.269894+0000", 2020-01-16T00:13:39.208 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0945000000000006e-05 2020-01-16T00:13:39.208 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.208 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.208 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:39.209 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.269965+0000", 2020-01-16T00:13:39.209 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012667399999999999 2020-01-16T00:13:39.209 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.209 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.209 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.209 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.270092+0000", 2020-01-16T00:13:39.209 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012187450000000001 2020-01-16T00:13:39.209 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.210 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.210 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.210 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.210 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.210 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3273 3.d 3:b76659c4:::smithi12313934-47:head [read 0~3647152,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:13:39.210 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.728775+0000", 2020-01-16T00:13:39.210 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 8.3089561189999994, 2020-01-16T00:13:39.210 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012922950000000001, 2020-01-16T00:13:39.211 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.211 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:39.211 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:39.211 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:39.211 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:39.211 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3273 2020-01-16T00:13:39.211 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.212 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.212 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.212 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.212 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728775+0000", 2020-01-16T00:13:39.212 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.212 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.212 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.212 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.213 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728775+0000", 2020-01-16T00:13:39.213 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:39.213 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.213 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.213 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.213 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728771+0000", 2020-01-16T00:13:39.213 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.5009000000000001e-05 2020-01-16T00:13:39.213 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.214 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.214 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.214 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728796+0000", 2020-01-16T00:13:39.214 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2357999999999998e-05 2020-01-16T00:13:39.214 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.214 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.214 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.214 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728819+0000", 2020-01-16T00:13:39.215 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.4800000000000007e-06 2020-01-16T00:13:39.215 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.215 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.215 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.215 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728828+0000", 2020-01-16T00:13:39.215 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000124824 2020-01-16T00:13:39.215 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.216 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.216 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.216 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728953+0000", 2020-01-16T00:13:39.216 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00044904899999999998 2020-01-16T00:13:39.216 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.216 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.216 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.216 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729402+0000", 2020-01-16T00:13:39.217 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012295706999999999 2020-01-16T00:13:39.217 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.217 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.217 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.217 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.741698+0000", 2020-01-16T00:13:39.217 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012922950000000001 2020-01-16T00:13:39.217 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.217 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.218 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.218 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.218 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.218 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3275 3.d 3:b76659c4:::smithi12313934-47:head [read 0~3647152,checksum] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:13:39.218 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.728933+0000", 2020-01-16T00:13:39.218 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 8.3087978469999992, 2020-01-16T00:13:39.218 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01204874, 2020-01-16T00:13:39.219 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.219 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:39.219 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:39.219 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:39.219 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:39.219 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3275 2020-01-16T00:13:39.219 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.219 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.220 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.220 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.220 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728933+0000", 2020-01-16T00:13:39.220 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.220 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.220 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.220 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.220 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728933+0000", 2020-01-16T00:13:39.221 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:39.221 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.221 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.221 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.221 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728932+0000", 2020-01-16T00:13:39.221 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3569999999999998e-06 2020-01-16T00:13:39.221 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.222 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.222 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.222 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728940+0000", 2020-01-16T00:13:39.222 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0457e-05 2020-01-16T00:13:39.222 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.222 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.222 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.222 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728951+0000", 2020-01-16T00:13:39.223 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6799999999999993e-06 2020-01-16T00:13:39.223 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.223 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.223 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.223 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728959+0000", 2020-01-16T00:13:39.223 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010076307 2020-01-16T00:13:39.223 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.223 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.224 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.224 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.739035+0000", 2020-01-16T00:13:39.224 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024455400000000002 2020-01-16T00:13:39.224 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.224 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.224 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.224 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.739279+0000", 2020-01-16T00:13:39.224 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0017023660000000001 2020-01-16T00:13:39.225 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.225 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.225 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.225 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.740982+0000", 2020-01-16T00:13:39.225 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01204874 2020-01-16T00:13:39.225 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.225 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.226 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.226 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.226 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.226 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3285 3.d 3:bd9fc72f:::smithi12313934-5:head [read 0~3764801,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:13:39.226 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.729068+0000", 2020-01-16T00:13:39.226 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 8.3086630550000002, 2020-01-16T00:13:39.226 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018537010999999999, 2020-01-16T00:13:39.226 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.227 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:39.227 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:39.227 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:39.227 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:39.227 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3285 2020-01-16T00:13:39.227 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.227 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.227 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.228 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.228 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729068+0000", 2020-01-16T00:13:39.228 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.228 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.228 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.228 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.228 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729068+0000", 2020-01-16T00:13:39.228 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:39.229 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.229 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.229 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.229 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729067+0000", 2020-01-16T00:13:39.229 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3339999999999996e-06 2020-01-16T00:13:39.229 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.229 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.229 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.230 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729075+0000", 2020-01-16T00:13:39.230 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4187e-05 2020-01-16T00:13:39.230 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.230 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.230 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.230 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729089+0000", 2020-01-16T00:13:39.230 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5600000000000004e-06 2020-01-16T00:13:39.231 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.231 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.231 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.231 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729093+0000", 2020-01-16T00:13:39.231 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011944309 2020-01-16T00:13:39.231 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.231 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.231 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.232 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.741038+0000", 2020-01-16T00:13:39.232 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016625799999999999 2020-01-16T00:13:39.232 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.232 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.232 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.232 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.741204+0000", 2020-01-16T00:13:39.232 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0064010120000000002 2020-01-16T00:13:39.232 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.233 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.233 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.233 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.747605+0000", 2020-01-16T00:13:39.233 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018537010999999999 2020-01-16T00:13:39.233 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.233 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.233 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.233 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.234 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.234 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3286 3.d 3:bd9fc72f:::smithi12313934-5:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:13:39.234 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.729099+0000", 2020-01-16T00:13:39.234 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 8.3086323849999992, 2020-01-16T00:13:39.234 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018983897999999999, 2020-01-16T00:13:39.234 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.234 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:39.235 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:39.235 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:39.235 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:39.235 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3286 2020-01-16T00:13:39.235 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.235 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.235 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.235 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.236 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729099+0000", 2020-01-16T00:13:39.236 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.236 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.236 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.236 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.236 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729099+0000", 2020-01-16T00:13:39.236 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:39.236 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.237 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.237 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.237 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729098+0000", 2020-01-16T00:13:39.237 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8500000000000004e-06 2020-01-16T00:13:39.237 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.237 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.237 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.237 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729102+0000", 2020-01-16T00:13:39.238 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7329999999999997e-06 2020-01-16T00:13:39.238 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.238 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.238 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.238 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729108+0000", 2020-01-16T00:13:39.238 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.764e-06 2020-01-16T00:13:39.238 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.239 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.239 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.239 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729118+0000", 2020-01-16T00:13:39.239 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018524286000000001 2020-01-16T00:13:39.239 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.239 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.239 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.239 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.747642+0000", 2020-01-16T00:13:39.240 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9487999999999993e-05 2020-01-16T00:13:39.240 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.240 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.240 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.240 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.747742+0000", 2020-01-16T00:13:39.240 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000340429 2020-01-16T00:13:39.240 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.240 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.241 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.241 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.748082+0000", 2020-01-16T00:13:39.241 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018983897999999999 2020-01-16T00:13:39.241 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.241 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.241 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.241 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.241 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.242 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3287 3.d 3:bd9fc72f:::smithi12313934-5:head [read 0~3764801,checksum] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:13:39.242 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.729124+0000", 2020-01-16T00:13:39.242 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 8.3086072459999993, 2020-01-16T00:13:39.242 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020552323000000001, 2020-01-16T00:13:39.242 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.242 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:39.242 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:39.242 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:39.243 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:39.243 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3287 2020-01-16T00:13:39.243 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.243 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.243 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.243 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.243 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729124+0000", 2020-01-16T00:13:39.243 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.244 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.244 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.244 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.244 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729124+0000", 2020-01-16T00:13:39.244 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:39.244 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.244 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.245 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.245 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729123+0000", 2020-01-16T00:13:39.245 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8239999999999996e-06 2020-01-16T00:13:39.245 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.245 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.245 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.245 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729129+0000", 2020-01-16T00:13:39.245 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6909999999999997e-06 2020-01-16T00:13:39.246 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.246 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.246 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.246 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729136+0000", 2020-01-16T00:13:39.246 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3329999999999997e-06 2020-01-16T00:13:39.246 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.246 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.246 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.247 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729145+0000", 2020-01-16T00:13:39.247 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018965065999999999 2020-01-16T00:13:39.247 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.247 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.247 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.247 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.748110+0000", 2020-01-16T00:13:39.247 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015781700000000001 2020-01-16T00:13:39.247 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.248 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.248 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.248 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.748268+0000", 2020-01-16T00:13:39.248 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014083800000000001 2020-01-16T00:13:39.248 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.248 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.248 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.248 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.749676+0000", 2020-01-16T00:13:39.249 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020552323000000001 2020-01-16T00:13:39.249 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.249 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.249 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.249 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.249 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.249 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3409 3.d 3:b3dd5d45:::smithi12313934-16:head [write 591712~786646,stat] snapc 0=[] ondisk+write+known_if_redirected e47)", 2020-01-16T00:13:39.250 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:34.100681+0000", 2020-01-16T00:13:39.250 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 4.937050106, 2020-01-16T00:13:39.250 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012248446, 2020-01-16T00:13:39.250 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.250 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:39.250 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:39.250 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:39.250 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:39.251 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3409 2020-01-16T00:13:39.251 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.251 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.251 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.251 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.251 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.100681+0000", 2020-01-16T00:13:39.251 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.251 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.252 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.252 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.252 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.100681+0000", 2020-01-16T00:13:39.252 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:13:39.252 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.252 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.252 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.252 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.100677+0000", 2020-01-16T00:13:39.253 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000998828 2020-01-16T00:13:39.253 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.253 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.253 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.253 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101676+0000", 2020-01-16T00:13:39.253 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5846999999999997e-05 2020-01-16T00:13:39.253 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.253 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.254 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.254 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101712+0000", 2020-01-16T00:13:39.254 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.364e-05 2020-01-16T00:13:39.254 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.254 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.254 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.254 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101725+0000", 2020-01-16T00:13:39.254 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015902700000000001 2020-01-16T00:13:39.255 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.255 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.255 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.255 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101884+0000", 2020-01-16T00:13:39.255 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0486e-05 2020-01-16T00:13:39.255 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.255 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.256 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op must wait for map", 2020-01-16T00:13:39.256 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101905+0000", 2020-01-16T00:13:39.256 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0038869009999999999 2020-01-16T00:13:39.256 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.256 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.256 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.256 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.105792+0000", 2020-01-16T00:13:39.256 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030748800000000002 2020-01-16T00:13:39.257 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.257 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.257 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.257 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.106099+0000", 2020-01-16T00:13:39.257 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042069299999999998 2020-01-16T00:13:39.257 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.257 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.257 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:39.258 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.106520+0000", 2020-01-16T00:13:39.258 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0052864560000000001 2020-01-16T00:13:39.258 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.258 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.258 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:39.258 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.111806+0000", 2020-01-16T00:13:39.258 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00080835800000000001 2020-01-16T00:13:39.259 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.259 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.259 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.259 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.112615+0000", 2020-01-16T00:13:39.259 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1490999999999997e-05 2020-01-16T00:13:39.259 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.259 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.259 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:39.260 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.112686+0000", 2020-01-16T00:13:39.260 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000243174 2020-01-16T00:13:39.260 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.260 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.260 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.260 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.112929+0000", 2020-01-16T00:13:39.260 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012248446 2020-01-16T00:13:39.260 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.261 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.261 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.261 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.261 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.261 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:39.262 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:39.258+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:39.263 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-01-16T00:13:39.277 INFO:tasks.rados.rados.0.smithi123.stdout:1380: finishing write tid 1 to smithi12313934-41 2020-01-16T00:13:39.277 INFO:tasks.rados.rados.0.smithi123.stdout:1381: expect (ObjNum 384 snap 0 seq_num 384) 2020-01-16T00:13:39.365 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:39.358+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:39.374 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:39.375 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:13:39.375 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:13:39.375 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:39.386 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-01-16T00:13:39.424 INFO:tasks.rados.rados.0.smithi123.stdout:1382: finishing write tid 1 to smithi12313934-29 2020-01-16T00:13:39.424 INFO:tasks.rados.rados.0.smithi123.stdout:1380: finishing write tid 2 to smithi12313934-41 2020-01-16T00:13:39.425 INFO:tasks.rados.rados.0.smithi123.stdout:1382: finishing write tid 2 to smithi12313934-29 2020-01-16T00:13:39.425 INFO:tasks.rados.rados.0.smithi123.stdout:1380: finishing write tid 3 to smithi12313934-41 2020-01-16T00:13:39.425 INFO:tasks.rados.rados.0.smithi123.stdout:1380: finishing write tid 5 to smithi12313934-41 2020-01-16T00:13:39.425 INFO:tasks.rados.rados.0.smithi123.stdout:1380: finishing write tid 6 to smithi12313934-41 2020-01-16T00:13:39.425 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 41 v 195 (ObjNum 441 snap 0 seq_num 441) dirty exists 2020-01-16T00:13:39.425 INFO:tasks.rados.rados.0.smithi123.stdout:1380: left oid 41 (ObjNum 441 snap 0 seq_num 441) 2020-01-16T00:13:39.425 INFO:tasks.rados.rados.0.smithi123.stdout:1383: finishing write tid 1 to smithi12313934-48 2020-01-16T00:13:39.426 INFO:tasks.rados.rados.0.smithi123.stdout:1382: finishing write tid 3 to smithi12313934-29 2020-01-16T00:13:39.426 INFO:tasks.rados.rados.0.smithi123.stdout:1382: finishing write tid 5 to smithi12313934-29 2020-01-16T00:13:39.426 INFO:tasks.rados.rados.0.smithi123.stdout:1382: finishing write tid 6 to smithi12313934-29 2020-01-16T00:13:39.426 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 29 v 146 (ObjNum 442 snap 0 seq_num 442) dirty exists 2020-01-16T00:13:39.426 INFO:tasks.rados.rados.0.smithi123.stdout:1382: left oid 29 (ObjNum 442 snap 0 seq_num 442) 2020-01-16T00:13:39.426 INFO:tasks.rados.rados.0.smithi123.stdout:1383: finishing write tid 2 to smithi12313934-48 2020-01-16T00:13:39.427 INFO:tasks.rados.rados.0.smithi123.stdout:1373: done (15 left) 2020-01-16T00:13:39.427 INFO:tasks.rados.rados.0.smithi123.stdout:1374: done (14 left) 2020-01-16T00:13:39.427 INFO:tasks.rados.rados.0.smithi123.stdout:1375: done (13 left) 2020-01-16T00:13:39.427 INFO:tasks.rados.rados.0.smithi123.stdout:1376: done (12 left) 2020-01-16T00:13:39.427 INFO:tasks.rados.rados.0.smithi123.stdout:1377: done (11 left) 2020-01-16T00:13:39.427 INFO:tasks.rados.rados.0.smithi123.stdout:1378: done (10 left) 2020-01-16T00:13:39.427 INFO:tasks.rados.rados.0.smithi123.stdout:1379: done (9 left) 2020-01-16T00:13:39.428 INFO:tasks.rados.rados.0.smithi123.stdout:1380: done (8 left) 2020-01-16T00:13:39.428 INFO:tasks.rados.rados.0.smithi123.stdout:1381: done (7 left) 2020-01-16T00:13:39.428 INFO:tasks.rados.rados.0.smithi123.stdout:1382: done (6 left) 2020-01-16T00:13:39.428 INFO:tasks.rados.rados.0.smithi123.stdout:1389: read oid 46 snap -1 2020-01-16T00:13:39.428 INFO:tasks.rados.rados.0.smithi123.stdout:1389: expect (ObjNum 421 snap 0 seq_num 421) 2020-01-16T00:13:39.428 INFO:tasks.rados.rados.0.smithi123.stdout:1390: read oid 15 snap -1 2020-01-16T00:13:39.429 INFO:tasks.rados.rados.0.smithi123.stdout:1390: expect (ObjNum 384 snap 0 seq_num 384) 2020-01-16T00:13:39.429 INFO:tasks.rados.rados.0.smithi123.stdout:1391: write oid 3 current snap is 0 2020-01-16T00:13:39.429 INFO:tasks.rados.rados.0.smithi123.stdout:1391: seq_num 447 ranges {712729=784777,2091098=467491,3342313=170416} 2020-01-16T00:13:39.438 INFO:tasks.rados.rados.0.smithi123.stdout:1391: writing smithi12313934-3 from 712729 to 1497506 tid 1 2020-01-16T00:13:39.444 INFO:tasks.rados.rados.0.smithi123.stdout:1391: writing smithi12313934-3 from 2091098 to 2558589 tid 2 2020-01-16T00:13:39.450 INFO:tasks.rados.rados.0.smithi123.stdout:1391: writing smithi12313934-3 from 3342313 to 3512729 tid 3 2020-01-16T00:13:39.450 INFO:tasks.rados.rados.0.smithi123.stdout:1392: write oid 2 current snap is 0 2020-01-16T00:13:39.450 INFO:tasks.rados.rados.0.smithi123.stdout:1392: seq_num 448 ranges {729536=460026,1749580=611149,2853383=452450} 2020-01-16T00:13:39.456 INFO:tasks.rados.rados.0.smithi123.stdout:1392: writing smithi12313934-2 from 729536 to 1189562 tid 1 2020-01-16T00:13:39.462 INFO:tasks.rados.rados.0.smithi123.stdout:1392: writing smithi12313934-2 from 1749580 to 2360729 tid 2 2020-01-16T00:13:39.463 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:39.460+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:39.468 INFO:tasks.rados.rados.0.smithi123.stdout:1392: writing smithi12313934-2 from 2853383 to 3305833 tid 3 2020-01-16T00:13:39.468 INFO:tasks.rados.rados.0.smithi123.stdout:1393: write (excl) oid 1 current snap is 0 2020-01-16T00:13:39.469 INFO:tasks.rados.rados.0.smithi123.stdout:1393: seq_num 449 ranges {746343=651628,1924415=754807,3397159=149184} 2020-01-16T00:13:39.476 INFO:tasks.rados.rados.0.smithi123.stdout:1393: writing smithi12313934-1 from 746343 to 1397971 tid 1 2020-01-16T00:13:39.483 INFO:tasks.rados.rados.0.smithi123.stdout:1393: writing smithi12313934-1 from 1924415 to 2679222 tid 2 2020-01-16T00:13:39.489 INFO:tasks.rados.rados.0.smithi123.stdout:1393: writing smithi12313934-1 from 3397159 to 3546343 tid 3 2020-01-16T00:13:39.489 INFO:tasks.rados.rados.0.smithi123.stdout:1394: read oid 50 snap -1 2020-01-16T00:13:39.489 INFO:tasks.rados.rados.0.smithi123.stdout:1394: expect (ObjNum 408 snap 0 seq_num 408) 2020-01-16T00:13:39.489 INFO:tasks.rados.rados.0.smithi123.stdout:1395: read oid 19 snap -1 2020-01-16T00:13:39.490 INFO:tasks.rados.rados.0.smithi123.stdout:1395: expect (ObjNum 342 snap 0 seq_num 342) 2020-01-16T00:13:39.490 INFO:tasks.rados.rados.0.smithi123.stdout:1396: write (excl) oid 6 current snap is 0 2020-01-16T00:13:39.490 INFO:tasks.rados.rados.0.smithi123.stdout:1396: seq_num 450 ranges {763150=726877,1982897=614818,3024582=474994} 2020-01-16T00:13:39.497 INFO:tasks.rados.rados.0.smithi123.stdout:1396: writing smithi12313934-6 from 763150 to 1490027 tid 1 2020-01-16T00:13:39.502 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:39.502 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:13:39.502 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:13:39.502 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:13:39.502 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:39.503 INFO:tasks.rados.rados.0.smithi123.stdout:1396: writing smithi12313934-6 from 1982897 to 2597715 tid 2 2020-01-16T00:13:39.509 INFO:tasks.rados.rados.0.smithi123.stdout:1396: writing smithi12313934-6 from 3024582 to 3499576 tid 3 2020-01-16T00:13:39.510 INFO:tasks.rados.rados.0.smithi123.stdout:1397: read oid 45 snap -1 2020-01-16T00:13:39.510 INFO:tasks.rados.rados.0.smithi123.stdout:1397: expect (ObjNum 415 snap 0 seq_num 415) 2020-01-16T00:13:39.510 INFO:tasks.rados.rados.0.smithi123.stdout:1398: read oid 42 snap -1 2020-01-16T00:13:39.510 INFO:tasks.rados.rados.0.smithi123.stdout:1398: expect (ObjNum 432 snap 0 seq_num 432) 2020-01-16T00:13:39.510 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:39.510 INFO:tasks.rados.rados.0.smithi123.stdout:1383: finishing write tid 3 to smithi12313934-48 2020-01-16T00:13:39.510 INFO:tasks.rados.rados.0.smithi123.stdout:1383: finishing write tid 5 to smithi12313934-48 2020-01-16T00:13:39.511 INFO:tasks.rados.rados.0.smithi123.stdout:1383: finishing write tid 6 to smithi12313934-48 2020-01-16T00:13:39.511 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 48 v 165 (ObjNum 443 snap 0 seq_num 443) dirty exists 2020-01-16T00:13:39.511 INFO:tasks.rados.rados.0.smithi123.stdout:1383: left oid 48 (ObjNum 443 snap 0 seq_num 443) 2020-01-16T00:13:39.511 INFO:tasks.rados.rados.0.smithi123.stdout:1384: expect (ObjNum 404 snap 0 seq_num 404) 2020-01-16T00:13:39.512 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-01-16T00:13:39.598 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:39.596+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:39.665 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:39.660+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:39.667 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:39.667 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:13:39.668 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:13:39.668 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:13:39.668 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.668 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1654 3.a 3:5069665e:::smithi12313934-2:head [write 1614880~581041] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:13:39.669 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.714839+0000", 2020-01-16T00:13:39.669 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 46.949486045, 2020-01-16T00:13:39.669 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013588093000000001, 2020-01-16T00:13:39.669 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.670 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:39.670 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:39.670 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:39.670 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:39.670 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1654 2020-01-16T00:13:39.671 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.671 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.671 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.671 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.671 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.714839+0000", 2020-01-16T00:13:39.671 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.672 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.672 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.672 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.672 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.714839+0000", 2020-01-16T00:13:39.672 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:39.672 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.672 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.673 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.673 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.714836+0000", 2020-01-16T00:13:39.673 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00054657999999999996 2020-01-16T00:13:39.673 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.673 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.673 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.675 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.715382+0000", 2020-01-16T00:13:39.675 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7743000000000001e-05 2020-01-16T00:13:39.675 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.676 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.676 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.676 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.715420+0000", 2020-01-16T00:13:39.676 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.221e-05 2020-01-16T00:13:39.676 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.676 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.676 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.677 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.715432+0000", 2020-01-16T00:13:39.677 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2487000000000001e-05 2020-01-16T00:13:39.677 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.677 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.677 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.677 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.715485+0000", 2020-01-16T00:13:39.677 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000726284 2020-01-16T00:13:39.677 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.678 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.678 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:39.678 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.716211+0000", 2020-01-16T00:13:39.678 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0067930899999999999 2020-01-16T00:13:39.678 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.678 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.678 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.678 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.723004+0000", 2020-01-16T00:13:39.679 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012579400000000001 2020-01-16T00:13:39.679 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.679 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.679 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.679 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.723130+0000", 2020-01-16T00:13:39.679 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013465800000000001 2020-01-16T00:13:39.680 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.680 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.680 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:39.680 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.723265+0000", 2020-01-16T00:13:39.680 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023307250000000001 2020-01-16T00:13:39.680 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.681 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.681 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:39.681 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.725595+0000", 2020-01-16T00:13:39.681 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012582139999999999 2020-01-16T00:13:39.681 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.681 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.682 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.682 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.726853+0000", 2020-01-16T00:13:39.682 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014479110000000001 2020-01-16T00:13:39.682 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.682 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.682 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.682 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.728301+0000", 2020-01-16T00:13:39.682 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6866000000000007e-05 2020-01-16T00:13:39.683 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.683 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.683 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:39.683 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.728368+0000", 2020-01-16T00:13:39.683 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8740999999999999e-05 2020-01-16T00:13:39.683 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.683 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.684 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.684 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.728427+0000", 2020-01-16T00:13:39.684 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013588093000000001 2020-01-16T00:13:39.684 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.685 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.685 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.685 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.685 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.686 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1655 3.a 3:5069665e:::smithi12313934-2:head [write 2785791~71399] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:13:39.686 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.718297+0000", 2020-01-16T00:13:39.686 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 46.946028192, 2020-01-16T00:13:39.686 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013817354, 2020-01-16T00:13:39.687 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.687 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:39.687 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:39.687 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:39.687 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:39.687 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1655 2020-01-16T00:13:39.687 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.688 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.688 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.688 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.688 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718297+0000", 2020-01-16T00:13:39.688 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.688 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.688 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.688 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.689 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718297+0000", 2020-01-16T00:13:39.689 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:39.689 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.689 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.689 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.689 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718294+0000", 2020-01-16T00:13:39.689 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000103558 2020-01-16T00:13:39.690 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.690 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.690 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.690 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718397+0000", 2020-01-16T00:13:39.690 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9287e-05 2020-01-16T00:13:39.690 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.690 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.690 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.691 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718417+0000", 2020-01-16T00:13:39.691 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6680000000000001e-06 2020-01-16T00:13:39.691 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.691 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.691 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.691 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718424+0000", 2020-01-16T00:13:39.691 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2701999999999994e-05 2020-01-16T00:13:39.692 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.692 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.692 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.692 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718517+0000", 2020-01-16T00:13:39.692 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3893000000000006e-05 2020-01-16T00:13:39.692 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.692 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.692 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:39.693 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718581+0000", 2020-01-16T00:13:39.693 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0059630079999999997 2020-01-16T00:13:39.693 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.693 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.693 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.693 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.724544+0000", 2020-01-16T00:13:39.693 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000309108 2020-01-16T00:13:39.694 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.694 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.694 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:39.694 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.724853+0000", 2020-01-16T00:13:39.694 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.005469794 2020-01-16T00:13:39.694 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.694 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.694 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.695 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.730323+0000", 2020-01-16T00:13:39.695 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018506100000000001 2020-01-16T00:13:39.695 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.695 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.695 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.695 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.730508+0000", 2020-01-16T00:13:39.695 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016729099999999999 2020-01-16T00:13:39.695 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.696 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.696 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:39.696 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.730675+0000", 2020-01-16T00:13:39.696 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00094193300000000005 2020-01-16T00:13:39.696 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.696 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.696 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:39.696 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731617+0000", 2020-01-16T00:13:39.697 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000214613 2020-01-16T00:13:39.697 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.697 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.697 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.697 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731832+0000", 2020-01-16T00:13:39.697 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018326400000000001 2020-01-16T00:13:39.697 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.697 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.698 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.698 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732015+0000", 2020-01-16T00:13:39.698 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0392e-05 2020-01-16T00:13:39.698 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.699 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.699 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:39.699 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732075+0000", 2020-01-16T00:13:39.699 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8677999999999998e-05 2020-01-16T00:13:39.699 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.699 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.699 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.699 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732114+0000", 2020-01-16T00:13:39.700 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013817354 2020-01-16T00:13:39.700 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.700 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.700 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.700 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.700 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.700 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1656 3.a 3:5069665e:::smithi12313934-2:head [setxattr _header (55) in=7b,truncate 2857190] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:13:39.701 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.718453+0000", 2020-01-16T00:13:39.701 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 46.945871511, 2020-01-16T00:13:39.701 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014083462, 2020-01-16T00:13:39.701 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.701 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:39.701 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:39.701 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:39.701 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:39.702 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1656 2020-01-16T00:13:39.702 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.702 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.702 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.702 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.702 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718453+0000", 2020-01-16T00:13:39.702 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.703 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.703 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.703 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.703 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718453+0000", 2020-01-16T00:13:39.703 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999828 2020-01-16T00:13:39.703 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.703 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.703 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.704 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718436+0000", 2020-01-16T00:13:39.704 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4051999999999999e-05 2020-01-16T00:13:39.704 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.704 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.704 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.704 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718460+0000", 2020-01-16T00:13:39.704 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6380000000000004e-06 2020-01-16T00:13:39.704 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.705 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.705 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.705 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718468+0000", 2020-01-16T00:13:39.705 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6750000000000001e-06 2020-01-16T00:13:39.705 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.705 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.706 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.706 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718472+0000", 2020-01-16T00:13:39.706 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016218499999999999 2020-01-16T00:13:39.706 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.706 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.706 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.706 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718635+0000", 2020-01-16T00:13:39.706 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2765e-05 2020-01-16T00:13:39.707 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.707 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.707 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:39.707 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718697+0000", 2020-01-16T00:13:39.707 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0062239249999999999 2020-01-16T00:13:39.707 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.707 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.708 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.708 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.724921+0000", 2020-01-16T00:13:39.708 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1527999999999999e-05 2020-01-16T00:13:39.708 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.708 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.708 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:39.708 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.724973+0000", 2020-01-16T00:13:39.708 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0060569719999999999 2020-01-16T00:13:39.709 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.709 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.709 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.709 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731030+0000", 2020-01-16T00:13:39.709 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5859999999999996e-05 2020-01-16T00:13:39.709 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.709 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.709 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.710 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731096+0000", 2020-01-16T00:13:39.710 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000112018 2020-01-16T00:13:39.710 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.710 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.710 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:39.710 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731208+0000", 2020-01-16T00:13:39.710 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000945889 2020-01-16T00:13:39.710 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.711 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.711 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.711 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732154+0000", 2020-01-16T00:13:39.711 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0834000000000003e-05 2020-01-16T00:13:39.711 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.711 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.711 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.711 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732184+0000", 2020-01-16T00:13:39.712 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028587600000000001 2020-01-16T00:13:39.712 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.712 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.712 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:39.712 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732470+0000", 2020-01-16T00:13:39.712 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1953999999999999e-05 2020-01-16T00:13:39.712 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.713 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.713 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:39.713 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732502+0000", 2020-01-16T00:13:39.713 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4644999999999998e-05 2020-01-16T00:13:39.713 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.713 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.713 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.713 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732537+0000", 2020-01-16T00:13:39.714 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014083462 2020-01-16T00:13:39.714 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.714 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.714 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.714 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.714 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.714 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1657 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-01-16T00:13:39.715 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.718495+0000", 2020-01-16T00:13:39.715 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 46.945829846000002, 2020-01-16T00:13:39.715 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014299630000000001, 2020-01-16T00:13:39.715 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.715 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:39.715 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:39.715 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:39.716 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:39.716 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1657 2020-01-16T00:13:39.716 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.716 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.716 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.716 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.716 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718495+0000", 2020-01-16T00:13:39.716 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.717 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.717 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.717 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.717 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718495+0000", 2020-01-16T00:13:39.717 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:39.717 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.717 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.718 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.718 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718494+0000", 2020-01-16T00:13:39.718 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8860000000000003e-06 2020-01-16T00:13:39.718 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.718 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.718 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.719 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718499+0000", 2020-01-16T00:13:39.719 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9786000000000002e-05 2020-01-16T00:13:39.719 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.719 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.719 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.719 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718539+0000", 2020-01-16T00:13:39.719 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1869999999999999e-05 2020-01-16T00:13:39.720 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.720 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.720 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.720 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718561+0000", 2020-01-16T00:13:39.720 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000172206 2020-01-16T00:13:39.720 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.720 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.721 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.721 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718733+0000", 2020-01-16T00:13:39.721 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0662999999999998e-05 2020-01-16T00:13:39.721 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.721 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.721 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:39.721 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718793+0000", 2020-01-16T00:13:39.722 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.006217958 2020-01-16T00:13:39.722 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.722 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.722 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.722 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.725011+0000", 2020-01-16T00:13:39.722 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1625999999999996e-05 2020-01-16T00:13:39.723 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.723 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.723 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:39.723 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.725073+0000", 2020-01-16T00:13:39.723 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0063923119999999998 2020-01-16T00:13:39.723 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.723 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.724 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.724 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731465+0000", 2020-01-16T00:13:39.724 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2534000000000005e-05 2020-01-16T00:13:39.724 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.724 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.724 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:39.724 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731558+0000", 2020-01-16T00:13:39.725 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010238809999999999 2020-01-16T00:13:39.725 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.725 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.725 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.725 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732582+0000", 2020-01-16T00:13:39.725 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000100918 2020-01-16T00:13:39.726 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.726 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.726 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.726 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732683+0000", 2020-01-16T00:13:39.726 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000112064 2020-01-16T00:13:39.726 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.726 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.727 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.727 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732795+0000", 2020-01-16T00:13:39.727 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014299630000000001 2020-01-16T00:13:39.727 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.727 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.727 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.727 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.728 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.728 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1917 3.a 3:5069665e:::smithi12313934-2:head [write 2113302~674388] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:13:39.728 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:57.330712+0000", 2020-01-16T00:13:39.728 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 42.333613331000002, 2020-01-16T00:13:39.728 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012781043000000001, 2020-01-16T00:13:39.728 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.728 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:39.729 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:39.729 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:39.729 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:39.729 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1917 2020-01-16T00:13:39.729 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.729 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.730 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.730 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.730 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.330712+0000", 2020-01-16T00:13:39.730 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.731 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.731 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.731 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.731 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.330712+0000", 2020-01-16T00:13:39.731 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:39.731 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.732 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.732 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.732 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.330709+0000", 2020-01-16T00:13:39.732 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00071550499999999996 2020-01-16T00:13:39.732 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.732 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.732 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.733 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331424+0000", 2020-01-16T00:13:39.733 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3354e-05 2020-01-16T00:13:39.733 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.733 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.733 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.733 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331447+0000", 2020-01-16T00:13:39.733 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8200000000000008e-06 2020-01-16T00:13:39.734 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.734 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.734 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.734 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331457+0000", 2020-01-16T00:13:39.734 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5557999999999997e-05 2020-01-16T00:13:39.734 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.734 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.735 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.735 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331533+0000", 2020-01-16T00:13:39.735 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030925000000000001 2020-01-16T00:13:39.735 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.735 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.735 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:39.735 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331842+0000", 2020-01-16T00:13:39.735 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0078507860000000002 2020-01-16T00:13:39.736 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.736 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.736 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.736 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.339693+0000", 2020-01-16T00:13:39.736 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9321000000000007e-05 2020-01-16T00:13:39.736 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.736 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.737 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.737 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.339792+0000", 2020-01-16T00:13:39.737 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015147700000000001 2020-01-16T00:13:39.737 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.737 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.737 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:39.737 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.339944+0000", 2020-01-16T00:13:39.738 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002663066 2020-01-16T00:13:39.738 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.738 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.738 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:39.738 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.342607+0000", 2020-01-16T00:13:39.738 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0001552 2020-01-16T00:13:39.739 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.739 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.739 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.739 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.342762+0000", 2020-01-16T00:13:39.739 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00063636900000000002 2020-01-16T00:13:39.739 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.739 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.740 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.740 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343398+0000", 2020-01-16T00:13:39.740 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9687000000000001e-05 2020-01-16T00:13:39.740 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.740 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.740 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:39.740 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343438+0000", 2020-01-16T00:13:39.741 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4647e-05 2020-01-16T00:13:39.741 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.741 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.741 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.741 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343493+0000", 2020-01-16T00:13:39.741 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012781043000000001 2020-01-16T00:13:39.742 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.742 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.742 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.742 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.742 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.742 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2308 3.a 3:5c245412:::smithi12313934-19:head [write 702592~703393] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:13:39.742 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:07.298509+0000", 2020-01-16T00:13:39.743 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 32.365815675, 2020-01-16T00:13:39.743 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012139225999999999, 2020-01-16T00:13:39.743 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.743 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:39.743 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:39.743 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:39.743 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:39.744 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2308 2020-01-16T00:13:39.744 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.744 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.744 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.744 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.744 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.298509+0000", 2020-01-16T00:13:39.745 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.745 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.745 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.745 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.745 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.298509+0000", 2020-01-16T00:13:39.745 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999957 2020-01-16T00:13:39.745 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.746 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.746 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.746 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.298505+0000", 2020-01-16T00:13:39.746 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00085313800000000003 2020-01-16T00:13:39.746 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.746 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.747 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.747 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.299358+0000", 2020-01-16T00:13:39.747 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5397e-05 2020-01-16T00:13:39.747 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.747 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.747 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.747 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.299394+0000", 2020-01-16T00:13:39.747 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3699e-05 2020-01-16T00:13:39.748 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.748 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.748 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.748 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.299407+0000", 2020-01-16T00:13:39.748 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014281999999999999 2020-01-16T00:13:39.748 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.748 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.749 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.749 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.299550+0000", 2020-01-16T00:13:39.749 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024247099999999999 2020-01-16T00:13:39.749 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.749 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.749 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.749 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.299793+0000", 2020-01-16T00:13:39.750 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00031642200000000001 2020-01-16T00:13:39.750 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.750 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.750 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:39.750 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.300109+0000", 2020-01-16T00:13:39.751 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0078127560000000006 2020-01-16T00:13:39.751 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.751 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.751 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:39.751 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.307922+0000", 2020-01-16T00:13:39.751 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0024170189999999999 2020-01-16T00:13:39.752 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.752 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.752 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.752 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.310339+0000", 2020-01-16T00:13:39.752 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.4764000000000004e-05 2020-01-16T00:13:39.752 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.753 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.753 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.753 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.310424+0000", 2020-01-16T00:13:39.753 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000128319 2020-01-16T00:13:39.753 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.753 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.754 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:39.754 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.310552+0000", 2020-01-16T00:13:39.754 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.6587000000000002e-05 2020-01-16T00:13:39.754 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.754 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.754 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.754 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.310648+0000", 2020-01-16T00:13:39.755 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012139225999999999 2020-01-16T00:13:39.755 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.755 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.755 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.755 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.755 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.756 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2505 3.a 3:5069665e:::smithi12313934-2:head [write 1862260~479427] snapc 0=[] ondisk+write+known_if_redirected e37)", 2020-01-16T00:13:39.756 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:11.286548+0000", 2020-01-16T00:13:39.756 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 28.377777164000001, 2020-01-16T00:13:39.756 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013314061, 2020-01-16T00:13:39.756 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.756 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:39.756 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:39.757 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:39.757 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:39.757 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2505 2020-01-16T00:13:39.757 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.757 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.757 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.757 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.758 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.286548+0000", 2020-01-16T00:13:39.758 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.758 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.758 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.758 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.758 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.286548+0000", 2020-01-16T00:13:39.758 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999957 2020-01-16T00:13:39.759 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.759 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.759 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.759 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.286544+0000", 2020-01-16T00:13:39.759 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042043700000000001 2020-01-16T00:13:39.759 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.759 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.760 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.760 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.286964+0000", 2020-01-16T00:13:39.760 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5756999999999997e-05 2020-01-16T00:13:39.760 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.760 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.760 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.760 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.287000+0000", 2020-01-16T00:13:39.760 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8960000000000001e-05 2020-01-16T00:13:39.761 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.761 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.761 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.761 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.287019+0000", 2020-01-16T00:13:39.761 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.7219000000000005e-05 2020-01-16T00:13:39.761 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.761 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.762 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.762 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.287116+0000", 2020-01-16T00:13:39.762 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033208899999999998 2020-01-16T00:13:39.762 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.762 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.762 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:39.762 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.287448+0000", 2020-01-16T00:13:39.763 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0082732420000000001 2020-01-16T00:13:39.763 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.763 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.763 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.763 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.295721+0000", 2020-01-16T00:13:39.763 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017259999999999999 2020-01-16T00:13:39.763 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.764 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.764 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.764 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.295894+0000", 2020-01-16T00:13:39.764 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00029238000000000002 2020-01-16T00:13:39.764 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.764 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.764 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:39.765 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.296186+0000", 2020-01-16T00:13:39.765 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0034626409999999998 2020-01-16T00:13:39.765 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.765 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.765 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.765 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299649+0000", 2020-01-16T00:13:39.765 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2300000000000002e-06 2020-01-16T00:13:39.766 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.766 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.766 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:39.766 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299656+0000", 2020-01-16T00:13:39.766 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1561000000000001e-05 2020-01-16T00:13:39.766 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.766 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.767 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.767 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299738+0000", 2020-01-16T00:13:39.767 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8491000000000005e-05 2020-01-16T00:13:39.767 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.767 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.767 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:39.767 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299806+0000", 2020-01-16T00:13:39.768 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5541000000000002e-05 2020-01-16T00:13:39.768 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.768 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.768 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.768 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299862+0000", 2020-01-16T00:13:39.768 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013314061 2020-01-16T00:13:39.768 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.769 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.769 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.769 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.769 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.769 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2789 3.a 3:5069665e:::smithi12313934-2:head [read 0~2890837,checksum] snapc 0=[] ondisk+read+known_if_redirected e41)", 2020-01-16T00:13:39.769 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:17.213121+0000", 2020-01-16T00:13:39.769 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 22.451204402999998, 2020-01-16T00:13:39.770 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011386347, 2020-01-16T00:13:39.770 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.770 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:39.770 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:39.770 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:39.770 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:39.770 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2789 2020-01-16T00:13:39.771 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.771 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.771 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.771 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.771 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213121+0000", 2020-01-16T00:13:39.771 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.771 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.772 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.772 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.772 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213121+0000", 2020-01-16T00:13:39.772 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:39.772 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.772 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.772 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.773 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213119+0000", 2020-01-16T00:13:39.773 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3570000000000003e-06 2020-01-16T00:13:39.773 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.773 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.773 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.773 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213126+0000", 2020-01-16T00:13:39.773 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2060000000000001e-06 2020-01-16T00:13:39.774 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.774 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.774 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.774 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213133+0000", 2020-01-16T00:13:39.774 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7700000000000004e-06 2020-01-16T00:13:39.774 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.775 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.775 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.775 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213140+0000", 2020-01-16T00:13:39.775 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0089949739999999993 2020-01-16T00:13:39.775 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.776 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.776 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.776 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.222135+0000", 2020-01-16T00:13:39.776 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000191697 2020-01-16T00:13:39.776 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.776 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.776 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.777 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.222326+0000", 2020-01-16T00:13:39.777 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002180436 2020-01-16T00:13:39.777 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.777 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.777 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.777 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.224507+0000", 2020-01-16T00:13:39.778 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011386347 2020-01-16T00:13:39.778 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.778 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.778 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.778 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.778 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.778 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2790 3.a 3:5069665e:::smithi12313934-2:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e41)", 2020-01-16T00:13:39.779 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:17.213146+0000", 2020-01-16T00:13:39.779 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 22.451179165999999, 2020-01-16T00:13:39.779 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012098164999999999, 2020-01-16T00:13:39.779 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.779 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:39.779 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:39.779 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:39.780 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:39.780 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2790 2020-01-16T00:13:39.780 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.780 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.780 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.780 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.780 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213146+0000", 2020-01-16T00:13:39.781 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.781 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.781 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.781 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.781 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213146+0000", 2020-01-16T00:13:39.781 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:39.782 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.782 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.782 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.782 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213145+0000", 2020-01-16T00:13:39.782 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4399999999999998e-06 2020-01-16T00:13:39.782 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.782 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.783 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.783 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213149+0000", 2020-01-16T00:13:39.783 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1029999999999996e-06 2020-01-16T00:13:39.783 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.783 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.783 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.783 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213156+0000", 2020-01-16T00:13:39.784 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0699999999999997e-06 2020-01-16T00:13:39.784 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.784 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.784 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.784 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213161+0000", 2020-01-16T00:13:39.784 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011386617 2020-01-16T00:13:39.785 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.785 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.785 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.785 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.224548+0000", 2020-01-16T00:13:39.785 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017114499999999999 2020-01-16T00:13:39.785 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.786 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.786 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.786 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.224719+0000", 2020-01-16T00:13:39.786 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00052473000000000005 2020-01-16T00:13:39.786 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.786 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.786 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.787 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.225244+0000", 2020-01-16T00:13:39.787 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012098164999999999 2020-01-16T00:13:39.787 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.787 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.787 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.787 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.787 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.788 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2791 3.a 3:54f2c117:::smithi12313934-41:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e41)", 2020-01-16T00:13:39.788 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:17.213167+0000", 2020-01-16T00:13:39.788 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 22.451158153000002, 2020-01-16T00:13:39.788 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016024037000000001, 2020-01-16T00:13:39.788 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.788 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:39.788 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:39.788 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:39.789 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:39.789 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2791 2020-01-16T00:13:39.789 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.789 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.789 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.789 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.790 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213167+0000", 2020-01-16T00:13:39.790 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.790 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.790 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.790 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.790 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213167+0000", 2020-01-16T00:13:39.790 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:39.791 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.791 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.791 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.791 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213166+0000", 2020-01-16T00:13:39.791 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6030000000000004e-06 2020-01-16T00:13:39.791 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.791 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.792 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.792 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213172+0000", 2020-01-16T00:13:39.792 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0929999999999997e-06 2020-01-16T00:13:39.792 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.792 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.792 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.792 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213179+0000", 2020-01-16T00:13:39.793 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4299999999999997e-06 2020-01-16T00:13:39.793 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.793 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.793 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.793 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213184+0000", 2020-01-16T00:13:39.793 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012106695000000001 2020-01-16T00:13:39.794 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.794 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.794 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.794 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.225291+0000", 2020-01-16T00:13:39.794 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020863699999999999 2020-01-16T00:13:39.794 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.794 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.795 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.795 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.225499+0000", 2020-01-16T00:13:39.795 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0036913050000000002 2020-01-16T00:13:39.795 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.795 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.795 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.795 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.229191+0000", 2020-01-16T00:13:39.796 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016024037000000001 2020-01-16T00:13:39.796 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.796 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.796 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.796 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.796 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.796 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2792 3.a 3:54f2c117:::smithi12313934-41:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e41)", 2020-01-16T00:13:39.797 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:17.213190+0000", 2020-01-16T00:13:39.797 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 22.451134678999999, 2020-01-16T00:13:39.797 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017165949, 2020-01-16T00:13:39.797 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.797 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:39.797 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:39.797 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:39.798 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:39.798 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2792 2020-01-16T00:13:39.798 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.798 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.798 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.798 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.798 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213190+0000", 2020-01-16T00:13:39.799 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.799 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.799 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.799 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.799 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213190+0000", 2020-01-16T00:13:39.799 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:39.799 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.799 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.800 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.800 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213189+0000", 2020-01-16T00:13:39.800 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2699999999999998e-06 2020-01-16T00:13:39.800 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.800 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.800 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.800 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213194+0000", 2020-01-16T00:13:39.800 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8600000000000004e-06 2020-01-16T00:13:39.801 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.801 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.801 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.801 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213201+0000", 2020-01-16T00:13:39.801 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8500000000000002e-06 2020-01-16T00:13:39.801 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.801 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.802 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.802 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213205+0000", 2020-01-16T00:13:39.802 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016028924999999999 2020-01-16T00:13:39.802 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.802 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.802 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.802 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.229234+0000", 2020-01-16T00:13:39.803 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019773999999999999 2020-01-16T00:13:39.803 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.803 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.803 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.803 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.229432+0000", 2020-01-16T00:13:39.803 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00092414400000000001 2020-01-16T00:13:39.803 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.804 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.804 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.804 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.230356+0000", 2020-01-16T00:13:39.804 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017165949 2020-01-16T00:13:39.804 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.804 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.804 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.805 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.805 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.805 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2793 3.a 3:54f2c117:::smithi12313934-41:head [read 0~3210170,checksum] snapc 0=[] ondisk+read+known_if_redirected e41)", 2020-01-16T00:13:39.805 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:17.213211+0000", 2020-01-16T00:13:39.805 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 22.451114133000001, 2020-01-16T00:13:39.805 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019069413, 2020-01-16T00:13:39.805 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.806 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:39.806 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:39.806 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:39.806 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:39.806 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2793 2020-01-16T00:13:39.806 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.807 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.807 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.807 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.807 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213211+0000", 2020-01-16T00:13:39.807 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.807 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.808 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.808 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.808 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213211+0000", 2020-01-16T00:13:39.808 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:39.808 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.808 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.808 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.809 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213210+0000", 2020-01-16T00:13:39.809 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.66e-06 2020-01-16T00:13:39.809 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.809 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.809 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.809 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213216+0000", 2020-01-16T00:13:39.810 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.9469999999999997e-06 2020-01-16T00:13:39.810 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.810 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.810 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.810 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213223+0000", 2020-01-16T00:13:39.810 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9259999999999999e-06 2020-01-16T00:13:39.811 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.811 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.811 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.811 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213228+0000", 2020-01-16T00:13:39.811 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017171046999999998 2020-01-16T00:13:39.811 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.811 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.812 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.812 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.230399+0000", 2020-01-16T00:13:39.812 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000189932 2020-01-16T00:13:39.812 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.812 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.812 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.813 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.230589+0000", 2020-01-16T00:13:39.813 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0016916539999999999 2020-01-16T00:13:39.813 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.813 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.813 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.813 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.232280+0000", 2020-01-16T00:13:39.814 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019069413 2020-01-16T00:13:39.814 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.814 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.814 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.814 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.814 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.815 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3120 3.a 3:5069665e:::smithi12313934-2:head [write 1845146~524707] snapc 0=[] ondisk+write+known_if_redirected e44)", 2020-01-16T00:13:39.815 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.262852+0000", 2020-01-16T00:13:39.815 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.40147295, 2020-01-16T00:13:39.815 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014991175000000001, 2020-01-16T00:13:39.815 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.815 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:39.815 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:39.816 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:39.816 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:39.816 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3120 2020-01-16T00:13:39.816 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.816 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.817 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.817 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.817 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.262852+0000", 2020-01-16T00:13:39.817 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.817 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.817 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.817 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.818 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.262852+0000", 2020-01-16T00:13:39.818 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:39.818 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.818 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.818 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.818 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.262849+0000", 2020-01-16T00:13:39.819 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00058469900000000005 2020-01-16T00:13:39.819 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.819 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.819 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.819 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263434+0000", 2020-01-16T00:13:39.819 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.6650000000000001e-05 2020-01-16T00:13:39.819 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.820 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.820 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.820 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263461+0000", 2020-01-16T00:13:39.820 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3478e-05 2020-01-16T00:13:39.820 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.820 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.821 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.821 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263474+0000", 2020-01-16T00:13:39.821 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7824000000000002e-05 2020-01-16T00:13:39.821 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.821 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.821 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.821 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263522+0000", 2020-01-16T00:13:39.822 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001365464 2020-01-16T00:13:39.822 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.822 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.822 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:39.822 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.264888+0000", 2020-01-16T00:13:39.822 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0081368250000000003 2020-01-16T00:13:39.822 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.823 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.823 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.823 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.273024+0000", 2020-01-16T00:13:39.823 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6130000000000006e-05 2020-01-16T00:13:39.823 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.823 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.824 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.824 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.273090+0000", 2020-01-16T00:13:39.824 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000207507 2020-01-16T00:13:39.824 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.824 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.824 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:39.824 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.273298+0000", 2020-01-16T00:13:39.824 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0035649980000000002 2020-01-16T00:13:39.825 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.825 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.825 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.825 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276863+0000", 2020-01-16T00:13:39.825 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.1459999999999995e-05 2020-01-16T00:13:39.825 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.826 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.826 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.826 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276954+0000", 2020-01-16T00:13:39.826 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00076363899999999998 2020-01-16T00:13:39.826 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.826 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.826 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:39.827 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277718+0000", 2020-01-16T00:13:39.827 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0154999999999995e-05 2020-01-16T00:13:39.827 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.827 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.827 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:39.827 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277788+0000", 2020-01-16T00:13:39.827 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4914000000000003e-05 2020-01-16T00:13:39.828 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.828 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.828 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.828 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277843+0000", 2020-01-16T00:13:39.828 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014991175000000001 2020-01-16T00:13:39.828 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.828 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.829 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.829 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.829 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.829 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3121 3.a 3:5069665e:::smithi12313934-2:head [setxattr _header (55) in=7b,truncate 2369853] snapc 0=[] ondisk+write+known_if_redirected e44)", 2020-01-16T00:13:39.829 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.263494+0000", 2020-01-16T00:13:39.829 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.400830662000001, 2020-01-16T00:13:39.829 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014559274000000001, 2020-01-16T00:13:39.830 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.830 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:39.830 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:39.830 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:39.830 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:39.830 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3121 2020-01-16T00:13:39.830 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.831 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.831 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.831 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.831 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263494+0000", 2020-01-16T00:13:39.831 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.831 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.831 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.832 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.832 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263494+0000", 2020-01-16T00:13:39.832 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:39.832 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.832 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.832 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.832 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263493+0000", 2020-01-16T00:13:39.833 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3359999999999995e-06 2020-01-16T00:13:39.833 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.833 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.833 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.833 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263501+0000", 2020-01-16T00:13:39.833 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4100000000000002e-06 2020-01-16T00:13:39.833 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.834 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.834 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.834 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263509+0000", 2020-01-16T00:13:39.834 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3839999999999996e-06 2020-01-16T00:13:39.834 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.834 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.834 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.835 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263514+0000", 2020-01-16T00:13:39.835 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001430295 2020-01-16T00:13:39.835 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.835 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.835 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.835 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.264944+0000", 2020-01-16T00:13:39.836 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9710999999999999e-05 2020-01-16T00:13:39.836 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.836 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.836 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:39.836 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.264994+0000", 2020-01-16T00:13:39.836 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010036188999999999 2020-01-16T00:13:39.837 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.837 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.837 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.837 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.275030+0000", 2020-01-16T00:13:39.837 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1743000000000007e-05 2020-01-16T00:13:39.837 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.837 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.838 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.838 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.275112+0000", 2020-01-16T00:13:39.838 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013036100000000001 2020-01-16T00:13:39.838 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.838 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.838 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:39.839 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.275242+0000", 2020-01-16T00:13:39.839 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0016795549999999999 2020-01-16T00:13:39.839 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.839 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.839 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.839 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276922+0000", 2020-01-16T00:13:39.839 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018479400000000001 2020-01-16T00:13:39.840 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.840 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.840 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.840 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277107+0000", 2020-01-16T00:13:39.840 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00083585300000000001 2020-01-16T00:13:39.840 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.840 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.841 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:39.841 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277943+0000", 2020-01-16T00:13:39.841 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1719999999999999e-05 2020-01-16T00:13:39.841 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.841 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.841 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:39.841 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277994+0000", 2020-01-16T00:13:39.842 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9212999999999997e-05 2020-01-16T00:13:39.842 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.842 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.842 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.842 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.278054+0000", 2020-01-16T00:13:39.842 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014559274000000001 2020-01-16T00:13:39.842 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.843 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.843 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.843 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.843 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.843 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3122 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e44)", 2020-01-16T00:13:39.843 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.263525+0000", 2020-01-16T00:13:39.844 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.400799473999999, 2020-01-16T00:13:39.844 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014731833999999999, 2020-01-16T00:13:39.844 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.844 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:39.844 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:39.844 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:39.844 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:39.845 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3122 2020-01-16T00:13:39.845 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.845 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.845 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.845 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.845 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263525+0000", 2020-01-16T00:13:39.845 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.846 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.846 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.846 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.846 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263525+0000", 2020-01-16T00:13:39.846 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:39.846 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.847 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.847 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.847 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263525+0000", 2020-01-16T00:13:39.847 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5000000000000001e-06 2020-01-16T00:13:39.847 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.847 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.847 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.848 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263529+0000", 2020-01-16T00:13:39.848 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4669999999999998e-06 2020-01-16T00:13:39.848 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.848 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.848 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.848 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263537+0000", 2020-01-16T00:13:39.848 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0819999999999998e-06 2020-01-16T00:13:39.849 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.849 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.849 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.849 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263542+0000", 2020-01-16T00:13:39.849 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0015024960000000001 2020-01-16T00:13:39.849 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.850 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.850 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.850 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.265044+0000", 2020-01-16T00:13:39.850 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6650999999999994e-05 2020-01-16T00:13:39.850 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.850 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.850 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:39.851 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.265131+0000", 2020-01-16T00:13:39.851 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010495310000000001 2020-01-16T00:13:39.851 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.851 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.851 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.852 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.275626+0000", 2020-01-16T00:13:39.852 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9536000000000004e-05 2020-01-16T00:13:39.852 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.852 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.852 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:39.853 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.275726+0000", 2020-01-16T00:13:39.853 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023719489999999999 2020-01-16T00:13:39.853 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.853 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.853 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.853 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.278098+0000", 2020-01-16T00:13:39.854 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3718000000000001e-05 2020-01-16T00:13:39.854 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.854 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.854 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.854 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.278171+0000", 2020-01-16T00:13:39.854 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5859999999999994e-05 2020-01-16T00:13:39.854 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.855 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.855 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.855 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.278257+0000", 2020-01-16T00:13:39.855 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014731833999999999 2020-01-16T00:13:39.855 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.855 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.855 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.856 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.856 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.856 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3141 3.a 3:54f2c117:::smithi12313934-41:head [read 0~3210170,checksum] snapc 0=[] ondisk+read+known_if_redirected e44)", 2020-01-16T00:13:39.856 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.536650+0000", 2020-01-16T00:13:39.856 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.127674882999999, 2020-01-16T00:13:39.856 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010701029000000001, 2020-01-16T00:13:39.857 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.857 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:39.857 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:39.857 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:39.857 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:39.857 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3141 2020-01-16T00:13:39.857 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.858 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.858 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.858 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.858 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.536650+0000", 2020-01-16T00:13:39.858 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.858 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.858 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.859 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.859 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.536650+0000", 2020-01-16T00:13:39.859 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:39.859 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.859 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.859 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.859 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.536649+0000", 2020-01-16T00:13:39.860 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0310000000000004e-06 2020-01-16T00:13:39.860 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.860 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.860 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.860 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.536655+0000", 2020-01-16T00:13:39.860 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2523e-05 2020-01-16T00:13:39.860 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.861 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.861 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.861 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.536668+0000", 2020-01-16T00:13:39.861 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.5720000000000001e-05 2020-01-16T00:13:39.861 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.861 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.861 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.862 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.536693+0000", 2020-01-16T00:13:39.862 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0085664409999999993 2020-01-16T00:13:39.862 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.862 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.862 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.862 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.545260+0000", 2020-01-16T00:13:39.862 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000238077 2020-01-16T00:13:39.863 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.863 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.863 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.863 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.545498+0000", 2020-01-16T00:13:39.863 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0018533060000000001 2020-01-16T00:13:39.863 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.863 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.864 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.864 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.547351+0000", 2020-01-16T00:13:39.864 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010701029000000001 2020-01-16T00:13:39.864 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.864 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.864 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.865 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.865 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.865 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3627 3.a 3:5069665e:::smithi12313934-2:head [write 1749580~611149] snapc 0=[] ondisk+write+known_if_redirected e48)", 2020-01-16T00:13:39.865 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:39.462779+0000", 2020-01-16T00:13:39.865 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.20154566199999999, 2020-01-16T00:13:39.865 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020946985000000001, 2020-01-16T00:13:39.865 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.866 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:39.866 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:39.866 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:39.866 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:39.866 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3627 2020-01-16T00:13:39.866 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.866 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.867 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.867 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.867 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.462779+0000", 2020-01-16T00:13:39.867 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.867 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.867 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.867 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.868 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.462779+0000", 2020-01-16T00:13:39.868 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:39.868 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.868 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.868 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.868 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.462777+0000", 2020-01-16T00:13:39.869 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00064136 2020-01-16T00:13:39.869 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.869 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.869 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.869 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.463418+0000", 2020-01-16T00:13:39.869 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4607e-05 2020-01-16T00:13:39.869 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.870 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.870 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.870 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.463443+0000", 2020-01-16T00:13:39.870 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0346999999999999e-05 2020-01-16T00:13:39.870 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.870 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.871 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.871 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.463453+0000", 2020-01-16T00:13:39.871 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000100988 2020-01-16T00:13:39.871 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.871 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.871 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.871 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.463554+0000", 2020-01-16T00:13:39.872 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023059280000000001 2020-01-16T00:13:39.872 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.872 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.872 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:39.872 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.465860+0000", 2020-01-16T00:13:39.872 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011316961 2020-01-16T00:13:39.872 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.873 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.873 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.873 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.477177+0000", 2020-01-16T00:13:39.873 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000132402 2020-01-16T00:13:39.873 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.873 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.874 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.874 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.477309+0000", 2020-01-16T00:13:39.874 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000223606 2020-01-16T00:13:39.874 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.874 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.874 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:39.875 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.477533+0000", 2020-01-16T00:13:39.875 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0057105209999999997 2020-01-16T00:13:39.875 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.875 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.876 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:39.876 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483244+0000", 2020-01-16T00:13:39.876 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000193031 2020-01-16T00:13:39.876 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.876 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.876 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.876 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483437+0000", 2020-01-16T00:13:39.877 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0902999999999995e-05 2020-01-16T00:13:39.877 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.877 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.877 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.877 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483507+0000", 2020-01-16T00:13:39.877 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000129412 2020-01-16T00:13:39.878 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.878 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.878 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:39.878 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483637+0000", 2020-01-16T00:13:39.878 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.9345999999999995e-05 2020-01-16T00:13:39.878 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.878 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.879 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.879 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483726+0000", 2020-01-16T00:13:39.879 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020946985000000001 2020-01-16T00:13:39.879 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.879 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.879 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.879 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.880 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.880 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3628 3.a 3:5069665e:::smithi12313934-2:head [write 2853383~452450] snapc 0=[] ondisk+write+known_if_redirected e48)", 2020-01-16T00:13:39.880 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:39.468178+0000", 2020-01-16T00:13:39.880 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.19614695300000001, 2020-01-16T00:13:39.880 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015705872999999999, 2020-01-16T00:13:39.880 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.880 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:39.881 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:39.881 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:39.881 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:39.881 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3628 2020-01-16T00:13:39.881 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.881 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.881 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.881 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.882 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468178+0000", 2020-01-16T00:13:39.882 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.882 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.882 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.882 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.882 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468178+0000", 2020-01-16T00:13:39.883 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:39.883 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.883 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.883 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.883 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468175+0000", 2020-01-16T00:13:39.883 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00065948000000000005 2020-01-16T00:13:39.883 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.884 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.884 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.884 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468834+0000", 2020-01-16T00:13:39.884 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3557000000000002e-05 2020-01-16T00:13:39.884 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.884 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.884 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.885 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468858+0000", 2020-01-16T00:13:39.885 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2931e-05 2020-01-16T00:13:39.885 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.885 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.885 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.885 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468871+0000", 2020-01-16T00:13:39.886 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000123113 2020-01-16T00:13:39.886 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.886 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.886 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.886 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468994+0000", 2020-01-16T00:13:39.886 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017615999999999999 2020-01-16T00:13:39.887 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.887 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.887 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:39.887 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.469170+0000", 2020-01-16T00:13:39.887 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010336908000000001 2020-01-16T00:13:39.887 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.888 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.888 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.888 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.479507+0000", 2020-01-16T00:13:39.888 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015856400000000001 2020-01-16T00:13:39.888 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.888 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.888 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.888 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.479665+0000", 2020-01-16T00:13:39.889 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030344600000000002 2020-01-16T00:13:39.889 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.889 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.889 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:39.889 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.479969+0000", 2020-01-16T00:13:39.889 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0037935389999999999 2020-01-16T00:13:39.889 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.890 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.890 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.890 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483762+0000", 2020-01-16T00:13:39.890 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3170999999999997e-05 2020-01-16T00:13:39.890 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.890 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.890 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:39.891 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483806+0000", 2020-01-16T00:13:39.891 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3821000000000001e-05 2020-01-16T00:13:39.891 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.891 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.891 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.891 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483829+0000", 2020-01-16T00:13:39.891 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0076000000000001e-05 2020-01-16T00:13:39.892 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.892 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.892 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:39.892 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483859+0000", 2020-01-16T00:13:39.892 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.438e-05 2020-01-16T00:13:39.892 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.892 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.893 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.893 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483884+0000", 2020-01-16T00:13:39.893 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015705872999999999 2020-01-16T00:13:39.893 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.893 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.893 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.893 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.894 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.894 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3629 3.a 3:5069665e:::smithi12313934-2:head [setxattr _header (55) in=7b,truncate 3529536] snapc 0=[] ondisk+write+known_if_redirected e48)", 2020-01-16T00:13:39.894 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:39.468888+0000", 2020-01-16T00:13:39.894 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.19543733199999999, 2020-01-16T00:13:39.894 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015108511, 2020-01-16T00:13:39.894 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.895 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:39.895 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:39.895 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:39.895 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:39.895 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3629 2020-01-16T00:13:39.895 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.895 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.896 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.896 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.896 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468888+0000", 2020-01-16T00:13:39.896 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.896 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.896 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.896 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.896 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468888+0000", 2020-01-16T00:13:39.897 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:39.897 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.897 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.897 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.897 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468886+0000", 2020-01-16T00:13:39.897 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5599999999999996e-06 2020-01-16T00:13:39.897 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.898 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.898 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.898 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468894+0000", 2020-01-16T00:13:39.898 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5279999999999998e-06 2020-01-16T00:13:39.898 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.898 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.898 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.899 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468901+0000", 2020-01-16T00:13:39.899 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2290000000000002e-06 2020-01-16T00:13:39.899 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.899 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.899 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.899 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468907+0000", 2020-01-16T00:13:39.900 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030197500000000001 2020-01-16T00:13:39.900 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.900 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.900 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.900 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.469209+0000", 2020-01-16T00:13:39.900 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6659999999999997e-05 2020-01-16T00:13:39.900 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.901 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.901 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:39.901 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.469255+0000", 2020-01-16T00:13:39.901 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013117904999999999 2020-01-16T00:13:39.901 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.901 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.901 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.902 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.482373+0000", 2020-01-16T00:13:39.902 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1260000000000006e-05 2020-01-16T00:13:39.902 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.902 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.902 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.902 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.482434+0000", 2020-01-16T00:13:39.902 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014433 2020-01-16T00:13:39.903 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.903 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.903 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:39.903 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.482579+0000", 2020-01-16T00:13:39.903 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012198160000000001 2020-01-16T00:13:39.903 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.903 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.904 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.904 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483799+0000", 2020-01-16T00:13:39.904 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.188e-05 2020-01-16T00:13:39.904 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.904 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.904 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:39.904 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483810+0000", 2020-01-16T00:13:39.905 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000114863 2020-01-16T00:13:39.905 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.905 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.905 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:39.905 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483925+0000", 2020-01-16T00:13:39.905 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0861000000000001e-05 2020-01-16T00:13:39.905 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.906 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.906 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:39.906 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483956+0000", 2020-01-16T00:13:39.906 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9894999999999997e-05 2020-01-16T00:13:39.906 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.906 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.906 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.907 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483996+0000", 2020-01-16T00:13:39.907 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015108511 2020-01-16T00:13:39.907 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.907 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.907 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.908 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.908 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.908 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3630 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e48)", 2020-01-16T00:13:39.908 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:39.468967+0000", 2020-01-16T00:13:39.908 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.19535819800000001, 2020-01-16T00:13:39.908 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015160579, 2020-01-16T00:13:39.908 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:39.909 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:39.909 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:39.909 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:39.909 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:39.909 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3630 2020-01-16T00:13:39.909 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.910 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:39.910 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.910 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:39.910 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468967+0000", 2020-01-16T00:13:39.910 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:39.910 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.910 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.911 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:39.911 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468967+0000", 2020-01-16T00:13:39.911 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:39.911 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.911 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.911 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:39.911 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468966+0000", 2020-01-16T00:13:39.912 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3430000000000003e-06 2020-01-16T00:13:39.912 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.912 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.912 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:39.912 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468971+0000", 2020-01-16T00:13:39.912 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6577000000000001e-05 2020-01-16T00:13:39.913 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.913 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.913 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:39.913 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468988+0000", 2020-01-16T00:13:39.913 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.378e-06 2020-01-16T00:13:39.913 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.914 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.914 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:39.914 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468993+0000", 2020-01-16T00:13:39.914 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000320601 2020-01-16T00:13:39.914 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.914 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.914 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.915 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.469314+0000", 2020-01-16T00:13:39.915 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013833699999999999 2020-01-16T00:13:39.915 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.915 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.915 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:39.915 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.469452+0000", 2020-01-16T00:13:39.915 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013447838 2020-01-16T00:13:39.916 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.916 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.916 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.916 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.482900+0000", 2020-01-16T00:13:39.916 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028006899999999998 2020-01-16T00:13:39.916 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.916 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.916 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:39.917 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483180+0000", 2020-01-16T00:13:39.917 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000833594 2020-01-16T00:13:39.917 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.917 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.917 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:39.917 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.484013+0000", 2020-01-16T00:13:39.917 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2601e-05 2020-01-16T00:13:39.918 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.918 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.918 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:39.918 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.484066+0000", 2020-01-16T00:13:39.918 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1303999999999994e-05 2020-01-16T00:13:39.918 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:39.919 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:39.919 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:39.919 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.484127+0000", 2020-01-16T00:13:39.919 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015160579 2020-01-16T00:13:39.919 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.919 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.919 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.919 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:39.920 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:39.920 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:39.923 INFO:tasks.rados.rados.0.smithi123.stdout:1385: finishing write tid 1 to smithi12313934-44 2020-01-16T00:13:39.924 INFO:tasks.rados.rados.0.smithi123.stdout:1386: finishing write tid 1 to smithi12313934-18 2020-01-16T00:13:39.924 INFO:tasks.rados.rados.0.smithi123.stdout:1385: finishing write tid 2 to smithi12313934-44 2020-01-16T00:13:39.924 INFO:tasks.rados.rados.0.smithi123.stdout:1385: finishing write tid 3 to smithi12313934-44 2020-01-16T00:13:39.924 INFO:tasks.rados.rados.0.smithi123.stdout:1386: finishing write tid 2 to smithi12313934-18 2020-01-16T00:13:39.924 INFO:tasks.rados.rados.0.smithi123.stdout:1385: finishing write tid 5 to smithi12313934-44 2020-01-16T00:13:39.924 INFO:tasks.rados.rados.0.smithi123.stdout:1385: finishing write tid 6 to smithi12313934-44 2020-01-16T00:13:39.925 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 44 v 201 (ObjNum 444 snap 0 seq_num 444) dirty exists 2020-01-16T00:13:39.925 INFO:tasks.rados.rados.0.smithi123.stdout:1385: left oid 44 (ObjNum 444 snap 0 seq_num 444) 2020-01-16T00:13:39.925 INFO:tasks.rados.rados.0.smithi123.stdout:1383: done (15 left) 2020-01-16T00:13:39.925 INFO:tasks.rados.rados.0.smithi123.stdout:1384: done (14 left) 2020-01-16T00:13:39.925 INFO:tasks.rados.rados.0.smithi123.stdout:1385: done (13 left) 2020-01-16T00:13:39.925 INFO:tasks.rados.rados.0.smithi123.stdout:1399: write oid 43 current snap is 0 2020-01-16T00:13:39.925 INFO:tasks.rados.rados.0.smithi123.stdout:1399: seq_num 451 ranges {779957=402126,1925026=758476,3335652=244305} 2020-01-16T00:13:39.926 INFO:tasks.rados.rados.0.smithi123.stdout:1399: writing smithi12313934-43 from 779957 to 1182083 tid 1 2020-01-16T00:13:39.926 INFO:tasks.rados.rados.0.smithi123.stdout:1399: writing smithi12313934-43 from 1925026 to 2683502 tid 2 2020-01-16T00:13:39.926 INFO:tasks.rados.rados.0.smithi123.stdout:1399: writing smithi12313934-43 from 3335652 to 3579957 tid 3 2020-01-16T00:13:39.926 INFO:tasks.rados.rados.0.smithi123.stdout:1400: write oid 23 current snap is 0 2020-01-16T00:13:39.926 INFO:tasks.rados.rados.0.smithi123.stdout:1400: seq_num 452 ranges {796764=477375,1983508=618487,3363075=233689} 2020-01-16T00:13:39.926 INFO:tasks.rados.rados.0.smithi123.stdout:1400: writing smithi12313934-23 from 796764 to 1274139 tid 1 2020-01-16T00:13:39.926 INFO:tasks.rados.rados.0.smithi123.stdout:1400: writing smithi12313934-23 from 1983508 to 2601995 tid 2 2020-01-16T00:13:39.927 INFO:tasks.rados.rados.0.smithi123.stdout:1400: writing smithi12313934-23 from 3363075 to 3596764 tid 3 2020-01-16T00:13:39.927 INFO:tasks.rados.rados.0.smithi123.stdout:1401: write oid 30 current snap is 0 2020-01-16T00:13:39.927 INFO:tasks.rados.rados.0.smithi123.stdout:1401: seq_num 453 ranges {413571=552624,1641990=762145,2990498=425163} 2020-01-16T00:13:39.927 INFO:tasks.rados.rados.0.smithi123.stdout:1401: writing smithi12313934-30 from 413571 to 966195 tid 1 2020-01-16T00:13:39.927 INFO:tasks.rados.rados.0.smithi123.stdout:1401: writing smithi12313934-30 from 1641990 to 2404135 tid 2 2020-01-16T00:13:39.927 INFO:tasks.rados.rados.0.smithi123.stdout:1401: writing smithi12313934-30 from 2990498 to 3415661 tid 3 2020-01-16T00:13:39.927 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:39.928 INFO:tasks.rados.rados.0.smithi123.stdout:1386: finishing write tid 3 to smithi12313934-18 2020-01-16T00:13:39.928 INFO:tasks.rados.rados.0.smithi123.stdout:1386: finishing write tid 5 to smithi12313934-18 2020-01-16T00:13:39.928 INFO:tasks.rados.rados.0.smithi123.stdout:1386: finishing write tid 6 to smithi12313934-18 2020-01-16T00:13:39.928 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 18 v 258 (ObjNum 445 snap 0 seq_num 445) dirty exists 2020-01-16T00:13:39.928 INFO:tasks.rados.rados.0.smithi123.stdout:1386: left oid 18 (ObjNum 445 snap 0 seq_num 445) 2020-01-16T00:13:39.928 INFO:tasks.rados.rados.0.smithi123.stdout:1388: finishing write tid 1 to smithi12313934-27 2020-01-16T00:13:39.928 INFO:tasks.rados.rados.0.smithi123.stdout:1388: finishing write tid 2 to smithi12313934-27 2020-01-16T00:13:39.929 INFO:tasks.rados.rados.0.smithi123.stdout:1388: finishing write tid 3 to smithi12313934-27 2020-01-16T00:13:39.929 INFO:tasks.rados.rados.0.smithi123.stdout:1388: finishing write tid 5 to smithi12313934-27 2020-01-16T00:13:39.929 INFO:tasks.rados.rados.0.smithi123.stdout:1388: finishing write tid 6 to smithi12313934-27 2020-01-16T00:13:39.929 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 27 v 144 (ObjNum 446 snap 0 seq_num 446) dirty exists 2020-01-16T00:13:39.929 INFO:tasks.rados.rados.0.smithi123.stdout:1388: left oid 27 (ObjNum 446 snap 0 seq_num 446) 2020-01-16T00:13:39.929 INFO:tasks.rados.rados.0.smithi123.stdout:1389: expect (ObjNum 421 snap 0 seq_num 421) 2020-01-16T00:13:39.929 INFO:teuthology.orchestra.run.smithi123.stderr:nodeep-scrub is unset 2020-01-16T00:13:39.930 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:39.918+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:39.930 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-01-16T00:13:40.021 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:40.018+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:40.062 INFO:tasks.rados.rados.0.smithi123.stdout:1390: expect (ObjNum 384 snap 0 seq_num 384) 2020-01-16T00:13:40.072 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:40.072 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:40.072 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:13:40.072 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:40.081 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-01-16T00:13:40.124 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:40.122+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:40.210 INFO:tasks.rados.rados.0.smithi123.stdout:1391: finishing write tid 1 to smithi12313934-3 2020-01-16T00:13:40.210 INFO:tasks.rados.rados.0.smithi123.stdout:1391: finishing write tid 2 to smithi12313934-3 2020-01-16T00:13:40.210 INFO:tasks.rados.rados.0.smithi123.stdout:1391: finishing write tid 3 to smithi12313934-3 2020-01-16T00:13:40.210 INFO:tasks.rados.rados.0.smithi123.stdout:1391: finishing write tid 5 to smithi12313934-3 2020-01-16T00:13:40.210 INFO:tasks.rados.rados.0.smithi123.stdout:1391: finishing write tid 6 to smithi12313934-3 2020-01-16T00:13:40.210 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 3 v 99 (ObjNum 447 snap 0 seq_num 447) dirty exists 2020-01-16T00:13:40.210 INFO:tasks.rados.rados.0.smithi123.stdout:1391: left oid 3 (ObjNum 447 snap 0 seq_num 447) 2020-01-16T00:13:40.211 INFO:tasks.rados.rados.0.smithi123.stdout:1392: finishing write tid 1 to smithi12313934-2 2020-01-16T00:13:40.211 INFO:tasks.rados.rados.0.smithi123.stdout:1393: finishing write tid 1 to smithi12313934-1 2020-01-16T00:13:40.211 INFO:tasks.rados.rados.0.smithi123.stdout:1386: done (15 left) 2020-01-16T00:13:40.211 INFO:tasks.rados.rados.0.smithi123.stdout:1387: done (14 left) 2020-01-16T00:13:40.211 INFO:tasks.rados.rados.0.smithi123.stdout:1388: done (13 left) 2020-01-16T00:13:40.211 INFO:tasks.rados.rados.0.smithi123.stdout:1389: done (12 left) 2020-01-16T00:13:40.211 INFO:tasks.rados.rados.0.smithi123.stdout:1390: done (11 left) 2020-01-16T00:13:40.211 INFO:tasks.rados.rados.0.smithi123.stdout:1391: done (10 left) 2020-01-16T00:13:40.212 INFO:tasks.rados.rados.0.smithi123.stdout:1402: read oid 36 snap -1 2020-01-16T00:13:40.212 INFO:tasks.rados.rados.0.smithi123.stdout:1402: expect (ObjNum 422 snap 0 seq_num 422) 2020-01-16T00:13:40.212 INFO:tasks.rados.rados.0.smithi123.stdout:1403: write oid 8 current snap is 0 2020-01-16T00:13:40.212 INFO:tasks.rados.rados.0.smithi123.stdout:1403: seq_num 454 ranges {430378=627873,1700472=622156,3017921=612457} 2020-01-16T00:13:40.223 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:40.222+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:40.224 INFO:tasks.rados.rados.0.smithi123.stdout:1403: writing smithi12313934-8 from 430378 to 1058251 tid 1 2020-01-16T00:13:40.226 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:40.226 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:40.227 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:13:40.227 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:13:40.227 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:40.232 INFO:tasks.rados.rados.0.smithi123.stdout:1403: writing smithi12313934-8 from 1700472 to 2322628 tid 2 2020-01-16T00:13:40.236 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-01-16T00:13:40.240 INFO:tasks.rados.rados.0.smithi123.stdout:1403: writing smithi12313934-8 from 3017921 to 3630378 tid 3 2020-01-16T00:13:40.240 INFO:tasks.rados.rados.0.smithi123.stdout:1404: write oid 40 current snap is 0 2020-01-16T00:13:40.240 INFO:tasks.rados.rados.0.smithi123.stdout:1404: seq_num 455 ranges {447185=703122,1642601=765814,2928991=447707} 2020-01-16T00:13:40.247 INFO:tasks.rados.rados.0.smithi123.stdout:1404: writing smithi12313934-40 from 447185 to 1150307 tid 1 2020-01-16T00:13:40.254 INFO:tasks.rados.rados.0.smithi123.stdout:1404: writing smithi12313934-40 from 1642601 to 2408415 tid 2 2020-01-16T00:13:40.260 INFO:tasks.rados.rados.0.smithi123.stdout:1404: writing smithi12313934-40 from 2928991 to 3376698 tid 3 2020-01-16T00:13:40.260 INFO:tasks.rados.rados.0.smithi123.stdout:1405: delete oid 16 current snap is 0 2020-01-16T00:13:40.261 INFO:tasks.rados.rados.0.smithi123.stdout:1392: finishing write tid 2 to smithi12313934-2 2020-01-16T00:13:40.261 INFO:tasks.rados.rados.0.smithi123.stdout:1392: finishing write tid 3 to smithi12313934-2 2020-01-16T00:13:40.261 INFO:tasks.rados.rados.0.smithi123.stdout:1392: finishing write tid 5 to smithi12313934-2 2020-01-16T00:13:40.262 INFO:tasks.rados.rados.0.smithi123.stdout:1392: finishing write tid 6 to smithi12313934-2 2020-01-16T00:13:40.263 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 2 v 200 (ObjNum 448 snap 0 seq_num 448) dirty exists 2020-01-16T00:13:40.263 INFO:tasks.rados.rados.0.smithi123.stdout:1392: left oid 2 (ObjNum 448 snap 0 seq_num 448) 2020-01-16T00:13:40.263 INFO:tasks.rados.rados.0.smithi123.stdout:1393: finishing write tid 2 to smithi12313934-1 2020-01-16T00:13:40.263 INFO:tasks.rados.rados.0.smithi123.stdout:1393: finishing write tid 3 to smithi12313934-1 2020-01-16T00:13:40.263 INFO:tasks.rados.rados.0.smithi123.stdout:1393: finishing write tid 5 to smithi12313934-1 2020-01-16T00:13:40.263 INFO:tasks.rados.rados.0.smithi123.stdout:1393: finishing write tid 6 to smithi12313934-1 2020-01-16T00:13:40.263 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 1 v 262 (ObjNum 449 snap 0 seq_num 449) dirty exists 2020-01-16T00:13:40.263 INFO:tasks.rados.rados.0.smithi123.stdout:1393: left oid 1 (ObjNum 449 snap 0 seq_num 449) 2020-01-16T00:13:40.264 INFO:tasks.rados.rados.0.smithi123.stdout:1394: expect (ObjNum 408 snap 0 seq_num 408) 2020-01-16T00:13:40.324 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:40.320+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:40.336 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:40.336 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:13:40.336 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:13:40.336 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:13:40.336 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.336 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:939 3.4 3:2bdd9e88:::smithi12313934-4:head [write 1681102~474010] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:40.336 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.908392+0000", 2020-01-16T00:13:40.337 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 62.425091412999997, 2020-01-16T00:13:40.337 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019294697, 2020-01-16T00:13:40.337 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:40.337 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:40.337 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:40.337 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:40.337 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:40.338 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 939 2020-01-16T00:13:40.338 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.338 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:40.338 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.338 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:40.338 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908392+0000", 2020-01-16T00:13:40.338 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:40.338 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.339 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.339 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:40.339 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908392+0000", 2020-01-16T00:13:40.339 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:40.339 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.339 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.339 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:40.339 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908390+0000", 2020-01-16T00:13:40.340 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015440399999999999 2020-01-16T00:13:40.340 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.340 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.340 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:40.340 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908545+0000", 2020-01-16T00:13:40.340 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3926e-05 2020-01-16T00:13:40.340 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.341 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.341 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:40.341 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908559+0000", 2020-01-16T00:13:40.341 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2200000000000003e-06 2020-01-16T00:13:40.341 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.341 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.341 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:40.341 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908566+0000", 2020-01-16T00:13:40.342 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020594799999999998 2020-01-16T00:13:40.342 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.342 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.342 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.342 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910625+0000", 2020-01-16T00:13:40.342 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118672 2020-01-16T00:13:40.342 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.342 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.343 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:40.343 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910744+0000", 2020-01-16T00:13:40.343 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031940969999999999 2020-01-16T00:13:40.343 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.343 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.343 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.343 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.913938+0000", 2020-01-16T00:13:40.343 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000309621 2020-01-16T00:13:40.344 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.344 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.344 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:40.344 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.914248+0000", 2020-01-16T00:13:40.344 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.008818579 2020-01-16T00:13:40.344 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.344 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.344 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.345 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.923066+0000", 2020-01-16T00:13:40.345 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000108989 2020-01-16T00:13:40.345 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.345 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.345 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:40.345 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.923175+0000", 2020-01-16T00:13:40.345 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019128299999999999 2020-01-16T00:13:40.346 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.346 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.346 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:13:40.346 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.923367+0000", 2020-01-16T00:13:40.346 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0032083760000000002 2020-01-16T00:13:40.346 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.346 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.346 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:40.347 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926575+0000", 2020-01-16T00:13:40.347 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2475999999999999e-05 2020-01-16T00:13:40.347 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.347 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.347 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:40.347 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926607+0000", 2020-01-16T00:13:40.347 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00099399800000000006 2020-01-16T00:13:40.348 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.348 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.348 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:40.349 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927601+0000", 2020-01-16T00:13:40.349 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0958999999999999e-05 2020-01-16T00:13:40.349 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.349 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.349 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:40.349 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927642+0000", 2020-01-16T00:13:40.349 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4747999999999999e-05 2020-01-16T00:13:40.349 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.350 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.350 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:40.350 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927687+0000", 2020-01-16T00:13:40.350 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019294697 2020-01-16T00:13:40.350 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.350 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:40.350 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.350 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.351 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.351 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:940 3.4 3:2bdd9e88:::smithi12313934-4:head [write 2687912~589420] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:40.351 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.915144+0000", 2020-01-16T00:13:40.351 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 62.418340045000001, 2020-01-16T00:13:40.351 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016092538999999999, 2020-01-16T00:13:40.351 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:40.351 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:40.352 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:40.352 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:40.352 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:40.352 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 940 2020-01-16T00:13:40.352 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.352 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:40.352 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.352 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:40.353 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915144+0000", 2020-01-16T00:13:40.353 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:40.353 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.353 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.353 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:40.353 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915144+0000", 2020-01-16T00:13:40.353 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:40.353 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.354 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.354 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:40.354 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915141+0000", 2020-01-16T00:13:40.354 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016366300000000001 2020-01-16T00:13:40.354 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.354 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.354 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:40.354 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915305+0000", 2020-01-16T00:13:40.355 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4847e-05 2020-01-16T00:13:40.355 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.355 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.355 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:40.355 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915320+0000", 2020-01-16T00:13:40.355 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2599999999999999e-06 2020-01-16T00:13:40.355 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.355 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.356 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:40.356 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915327+0000", 2020-01-16T00:13:40.356 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4282000000000005e-05 2020-01-16T00:13:40.356 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.356 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.357 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.357 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915401+0000", 2020-01-16T00:13:40.357 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4554000000000006e-05 2020-01-16T00:13:40.357 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.357 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.357 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:40.357 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915466+0000", 2020-01-16T00:13:40.358 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0090821900000000004 2020-01-16T00:13:40.358 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.358 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.358 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.358 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.924548+0000", 2020-01-16T00:13:40.358 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000114696 2020-01-16T00:13:40.358 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.358 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.359 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:40.359 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.924663+0000", 2020-01-16T00:13:40.359 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000168651 2020-01-16T00:13:40.359 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.359 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.359 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:13:40.359 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.924832+0000", 2020-01-16T00:13:40.359 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018253430000000001 2020-01-16T00:13:40.360 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.360 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.360 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:40.360 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926657+0000", 2020-01-16T00:13:40.360 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9957999999999999e-05 2020-01-16T00:13:40.360 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.360 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.361 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:40.361 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926697+0000", 2020-01-16T00:13:40.361 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0044685250000000001 2020-01-16T00:13:40.361 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.361 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.361 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:40.361 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931165+0000", 2020-01-16T00:13:40.361 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9211000000000001e-05 2020-01-16T00:13:40.362 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.362 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.362 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:40.362 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931205+0000", 2020-01-16T00:13:40.362 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1699999999999998e-05 2020-01-16T00:13:40.362 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.362 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.362 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:40.363 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931236+0000", 2020-01-16T00:13:40.363 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016092538999999999 2020-01-16T00:13:40.363 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.363 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:40.363 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.363 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.363 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.364 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:941 3.4 3:2bdd9e88:::smithi12313934-4:head [setxattr _header (55) in=7b,truncate 3277332] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:40.364 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.915341+0000", 2020-01-16T00:13:40.364 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 62.418143227000002, 2020-01-16T00:13:40.364 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016603769000000001, 2020-01-16T00:13:40.364 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:40.365 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:40.365 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:40.365 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:40.365 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:40.365 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 941 2020-01-16T00:13:40.365 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.365 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:40.366 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.366 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:40.366 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915341+0000", 2020-01-16T00:13:40.366 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:40.366 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.366 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.366 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:40.366 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915341+0000", 2020-01-16T00:13:40.367 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:40.367 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.367 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.367 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:40.367 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915340+0000", 2020-01-16T00:13:40.367 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.896e-06 2020-01-16T00:13:40.367 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.367 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.368 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:40.368 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915346+0000", 2020-01-16T00:13:40.368 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2730000000000002e-06 2020-01-16T00:13:40.369 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.369 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.369 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:40.369 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915354+0000", 2020-01-16T00:13:40.369 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0616e-05 2020-01-16T00:13:40.369 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.369 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.370 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:40.370 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915374+0000", 2020-01-16T00:13:40.370 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000133487 2020-01-16T00:13:40.370 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.370 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.370 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.371 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915508+0000", 2020-01-16T00:13:40.371 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8971000000000003e-05 2020-01-16T00:13:40.371 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.371 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.371 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:40.371 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915557+0000", 2020-01-16T00:13:40.371 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010138787999999999 2020-01-16T00:13:40.371 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.372 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.372 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.372 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.925696+0000", 2020-01-16T00:13:40.372 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012825400000000001 2020-01-16T00:13:40.372 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.374 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.374 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:40.374 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.925824+0000", 2020-01-16T00:13:40.374 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017706500000000001 2020-01-16T00:13:40.374 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.375 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.375 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:13:40.375 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926001+0000", 2020-01-16T00:13:40.375 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00079988900000000003 2020-01-16T00:13:40.375 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.375 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.375 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:40.375 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926801+0000", 2020-01-16T00:13:40.376 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119324 2020-01-16T00:13:40.376 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.376 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.376 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:40.376 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926920+0000", 2020-01-16T00:13:40.376 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048879910000000004 2020-01-16T00:13:40.376 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.377 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.377 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:40.379 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931808+0000", 2020-01-16T00:13:40.379 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6999999999999997e-05 2020-01-16T00:13:40.379 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.379 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.379 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:40.379 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931855+0000", 2020-01-16T00:13:40.379 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9092999999999996e-05 2020-01-16T00:13:40.380 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.380 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.380 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:40.380 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931944+0000", 2020-01-16T00:13:40.380 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016603769000000001 2020-01-16T00:13:40.380 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.380 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:40.381 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.381 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.381 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.381 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:942 3.4 3:2bdd9e88:::smithi12313934-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:13:40.381 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.915389+0000", 2020-01-16T00:13:40.381 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 62.418094861, 2020-01-16T00:13:40.381 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016820563, 2020-01-16T00:13:40.381 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:40.382 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:40.382 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:40.382 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:40.382 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:40.382 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 942 2020-01-16T00:13:40.382 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.382 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:40.382 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.383 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:40.383 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915389+0000", 2020-01-16T00:13:40.383 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:40.383 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.383 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.383 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:40.383 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915389+0000", 2020-01-16T00:13:40.383 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:40.384 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.384 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.384 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:40.384 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915388+0000", 2020-01-16T00:13:40.384 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.683e-06 2020-01-16T00:13:40.384 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.384 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.384 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:40.385 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915391+0000", 2020-01-16T00:13:40.385 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4839999999999998e-06 2020-01-16T00:13:40.385 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.385 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.385 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:40.385 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915399+0000", 2020-01-16T00:13:40.385 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2259999999999999e-06 2020-01-16T00:13:40.386 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.386 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.386 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:40.386 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915403+0000", 2020-01-16T00:13:40.386 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00031301199999999999 2020-01-16T00:13:40.386 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.386 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.386 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.387 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915716+0000", 2020-01-16T00:13:40.387 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2079999999999998e-05 2020-01-16T00:13:40.387 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.387 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.387 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:40.387 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915748+0000", 2020-01-16T00:13:40.387 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010547076000000001 2020-01-16T00:13:40.388 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.388 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.388 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.388 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926295+0000", 2020-01-16T00:13:40.388 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8040000000000005e-05 2020-01-16T00:13:40.388 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.388 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.388 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:40.389 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926373+0000", 2020-01-16T00:13:40.389 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056061990000000001 2020-01-16T00:13:40.389 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.389 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.389 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.389 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931979+0000", 2020-01-16T00:13:40.389 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011540399999999999 2020-01-16T00:13:40.389 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.390 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.390 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:40.390 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932095+0000", 2020-01-16T00:13:40.390 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011482899999999999 2020-01-16T00:13:40.390 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.390 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.390 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:40.390 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932210+0000", 2020-01-16T00:13:40.391 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016820563 2020-01-16T00:13:40.391 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.391 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:40.391 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.391 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.391 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.391 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:957 3.0 3:00c146d5:::smithi12313934-18:head [write 510946~632952] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:40.391 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.033290+0000", 2020-01-16T00:13:40.392 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 62.300194024, 2020-01-16T00:13:40.392 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016764102999999999, 2020-01-16T00:13:40.392 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:40.392 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:40.392 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:40.392 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:40.392 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:40.393 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 957 2020-01-16T00:13:40.393 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.393 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:40.393 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.393 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:40.393 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033290+0000", 2020-01-16T00:13:40.393 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:40.393 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.394 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.394 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:40.394 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033290+0000", 2020-01-16T00:13:40.394 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999952 2020-01-16T00:13:40.394 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.394 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.394 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:40.394 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033285+0000", 2020-01-16T00:13:40.395 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000405917 2020-01-16T00:13:40.395 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.395 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.395 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:40.395 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033691+0000", 2020-01-16T00:13:40.395 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6585999999999997e-05 2020-01-16T00:13:40.395 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.395 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.396 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:40.396 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033738+0000", 2020-01-16T00:13:40.396 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8374000000000001e-05 2020-01-16T00:13:40.396 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.396 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.396 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:40.396 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033756+0000", 2020-01-16T00:13:40.396 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013002900000000001 2020-01-16T00:13:40.397 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.397 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.397 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.397 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033886+0000", 2020-01-16T00:13:40.397 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035882600000000002 2020-01-16T00:13:40.397 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.397 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.398 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:40.398 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.034245+0000", 2020-01-16T00:13:40.398 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0072798489999999997 2020-01-16T00:13:40.398 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.398 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.398 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.398 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.041525+0000", 2020-01-16T00:13:40.398 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000140488 2020-01-16T00:13:40.399 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.399 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.399 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:40.399 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.041665+0000", 2020-01-16T00:13:40.399 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000233766 2020-01-16T00:13:40.399 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.399 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.399 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:40.400 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.041899+0000", 2020-01-16T00:13:40.400 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0035766539999999999 2020-01-16T00:13:40.400 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.400 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.400 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:40.400 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.045475+0000", 2020-01-16T00:13:40.400 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013333150000000001 2020-01-16T00:13:40.400 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.401 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.401 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:40.401 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.046809+0000", 2020-01-16T00:13:40.401 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031447860000000001 2020-01-16T00:13:40.401 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.401 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.401 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:40.402 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.049954+0000", 2020-01-16T00:13:40.402 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2318000000000003e-05 2020-01-16T00:13:40.402 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.402 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.402 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:40.402 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.050006+0000", 2020-01-16T00:13:40.402 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7972000000000001e-05 2020-01-16T00:13:40.402 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.403 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.403 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:40.403 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.050054+0000", 2020-01-16T00:13:40.403 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016764102999999999 2020-01-16T00:13:40.404 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.405 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:40.405 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.405 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.405 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.405 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1395 3.4 3:240c7503:::smithi12313934-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e28)", 2020-01-16T00:13:40.405 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:46.625191+0000", 2020-01-16T00:13:40.405 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 53.708292696000001, 2020-01-16T00:13:40.406 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015878716000000001, 2020-01-16T00:13:40.406 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:40.406 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:40.406 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:40.406 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:40.406 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:40.406 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1395 2020-01-16T00:13:40.406 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.407 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:40.407 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.407 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:40.407 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625191+0000", 2020-01-16T00:13:40.407 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:40.407 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.407 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.407 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:40.408 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625191+0000", 2020-01-16T00:13:40.408 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:40.408 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.408 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.408 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:40.408 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625191+0000", 2020-01-16T00:13:40.408 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.1090000000000001e-06 2020-01-16T00:13:40.408 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.409 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.409 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:40.409 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625193+0000", 2020-01-16T00:13:40.409 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2730000000000008e-06 2020-01-16T00:13:40.409 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.409 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.409 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:40.410 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625202+0000", 2020-01-16T00:13:40.410 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3700000000000002e-06 2020-01-16T00:13:40.410 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.410 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.410 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:40.410 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625204+0000", 2020-01-16T00:13:40.410 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056573049999999996 2020-01-16T00:13:40.411 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.411 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.411 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.411 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630862+0000", 2020-01-16T00:13:40.411 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6047000000000001e-05 2020-01-16T00:13:40.411 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.412 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.412 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:40.412 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630918+0000", 2020-01-16T00:13:40.412 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0077941180000000001 2020-01-16T00:13:40.412 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.412 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.412 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.413 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.638712+0000", 2020-01-16T00:13:40.413 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118096 2020-01-16T00:13:40.413 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.413 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.413 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:40.414 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.638830+0000", 2020-01-16T00:13:40.414 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0019986740000000002 2020-01-16T00:13:40.414 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.414 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.414 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.414 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.640829+0000", 2020-01-16T00:13:40.414 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012562399999999999 2020-01-16T00:13:40.415 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.415 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.415 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:40.415 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.640954+0000", 2020-01-16T00:13:40.415 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000115536 2020-01-16T00:13:40.415 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.415 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.415 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:40.416 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.641070+0000", 2020-01-16T00:13:40.416 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015878716000000001 2020-01-16T00:13:40.416 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.416 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:40.416 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.416 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.416 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.417 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1639 3.0 3:00c146d5:::smithi12313934-18:head [write 423576~401598] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:13:40.417 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.774317+0000", 2020-01-16T00:13:40.417 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 48.559167285000001, 2020-01-16T00:13:40.417 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018305219000000001, 2020-01-16T00:13:40.417 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:40.417 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:40.417 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:40.417 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:40.418 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:40.418 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1639 2020-01-16T00:13:40.418 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.418 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:40.418 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.418 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:40.418 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774317+0000", 2020-01-16T00:13:40.418 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:40.419 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.419 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.419 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:40.419 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774317+0000", 2020-01-16T00:13:40.419 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:40.419 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.419 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.420 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:40.420 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774313+0000", 2020-01-16T00:13:40.420 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000111173 2020-01-16T00:13:40.420 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.420 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.420 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:40.420 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774425+0000", 2020-01-16T00:13:40.420 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7598999999999999e-05 2020-01-16T00:13:40.421 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.421 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.421 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:40.421 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774442+0000", 2020-01-16T00:13:40.421 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0830000000000004e-06 2020-01-16T00:13:40.421 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.421 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.421 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:40.422 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774450+0000", 2020-01-16T00:13:40.422 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0017886309999999999 2020-01-16T00:13:40.422 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.422 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.422 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.422 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.776239+0000", 2020-01-16T00:13:40.422 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3789e-05 2020-01-16T00:13:40.423 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.423 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.423 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op must wait for map", 2020-01-16T00:13:40.423 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.776253+0000", 2020-01-16T00:13:40.423 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095730729999999997 2020-01-16T00:13:40.423 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.423 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.424 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.424 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.785826+0000", 2020-01-16T00:13:40.424 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014220300000000001 2020-01-16T00:13:40.424 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.424 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.424 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:40.424 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.785968+0000", 2020-01-16T00:13:40.424 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023618900000000001 2020-01-16T00:13:40.425 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.425 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.425 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:40.425 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.786204+0000", 2020-01-16T00:13:40.425 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0062297389999999998 2020-01-16T00:13:40.425 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.425 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.425 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:40.426 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792434+0000", 2020-01-16T00:13:40.426 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9635999999999998e-05 2020-01-16T00:13:40.426 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.426 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.426 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:40.426 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792473+0000", 2020-01-16T00:13:40.426 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6456000000000002e-05 2020-01-16T00:13:40.427 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.427 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.427 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:40.427 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792540+0000", 2020-01-16T00:13:40.427 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9873999999999998e-05 2020-01-16T00:13:40.427 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.427 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.427 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:40.428 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792580+0000", 2020-01-16T00:13:40.428 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1924e-05 2020-01-16T00:13:40.428 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.428 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.428 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:40.428 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792622+0000", 2020-01-16T00:13:40.428 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018305219000000001 2020-01-16T00:13:40.429 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.429 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:40.429 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.429 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.429 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.429 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1890 3.0 3:0b1f384f:::smithi12313934-33:head [write 2797915~596476] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:13:40.429 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.232534+0000", 2020-01-16T00:13:40.430 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 43.100949313999998, 2020-01-16T00:13:40.430 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016106137, 2020-01-16T00:13:40.430 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:40.430 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:40.430 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:40.430 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:40.430 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:40.431 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1890 2020-01-16T00:13:40.431 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.431 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:40.431 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.431 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:40.431 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232534+0000", 2020-01-16T00:13:40.431 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:40.431 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.432 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.432 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:40.432 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232534+0000", 2020-01-16T00:13:40.432 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:40.432 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.432 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.432 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:40.433 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232532+0000", 2020-01-16T00:13:40.433 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00024232100000000001 2020-01-16T00:13:40.433 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.433 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.433 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:40.433 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232774+0000", 2020-01-16T00:13:40.433 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4601e-05 2020-01-16T00:13:40.433 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.434 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.434 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:40.434 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232789+0000", 2020-01-16T00:13:40.434 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8410000000000006e-06 2020-01-16T00:13:40.434 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.434 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.434 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:40.434 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232798+0000", 2020-01-16T00:13:40.435 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9756999999999997e-05 2020-01-16T00:13:40.435 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.435 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.435 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.435 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232857+0000", 2020-01-16T00:13:40.435 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4678000000000007e-05 2020-01-16T00:13:40.435 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.436 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.436 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:40.436 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232922+0000", 2020-01-16T00:13:40.436 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033861020000000002 2020-01-16T00:13:40.436 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.436 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.436 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.436 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.236308+0000", 2020-01-16T00:13:40.437 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000568996 2020-01-16T00:13:40.437 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.437 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.437 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:40.437 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.236877+0000", 2020-01-16T00:13:40.437 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0085444240000000001 2020-01-16T00:13:40.437 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.437 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.438 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.438 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.245422+0000", 2020-01-16T00:13:40.438 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5595999999999998e-05 2020-01-16T00:13:40.438 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.438 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.438 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:40.438 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.245507+0000", 2020-01-16T00:13:40.439 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016984300000000001 2020-01-16T00:13:40.439 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.439 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.439 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:40.439 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.245677+0000", 2020-01-16T00:13:40.439 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001649495 2020-01-16T00:13:40.439 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.439 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.440 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:40.440 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247326+0000", 2020-01-16T00:13:40.440 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023308399999999999 2020-01-16T00:13:40.440 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.440 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.440 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:40.440 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247560+0000", 2020-01-16T00:13:40.441 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00099501800000000007 2020-01-16T00:13:40.441 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.441 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.441 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:40.441 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248555+0000", 2020-01-16T00:13:40.441 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5936999999999998e-05 2020-01-16T00:13:40.441 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.441 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.442 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:40.442 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248591+0000", 2020-01-16T00:13:40.442 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0108999999999997e-05 2020-01-16T00:13:40.442 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.442 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.442 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:40.442 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248641+0000", 2020-01-16T00:13:40.443 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016106137 2020-01-16T00:13:40.443 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.443 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:40.443 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.443 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.443 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.443 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1891 3.0 3:0b1f384f:::smithi12313934-33:head [setxattr _header (56) in=7b,truncate 3395014] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:13:40.443 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.232813+0000", 2020-01-16T00:13:40.444 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 43.100670512000001, 2020-01-16T00:13:40.444 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015999295, 2020-01-16T00:13:40.444 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:40.444 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:40.444 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:40.444 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:40.444 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:40.444 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1891 2020-01-16T00:13:40.444 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.445 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:40.445 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.445 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:40.445 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232813+0000", 2020-01-16T00:13:40.445 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:40.445 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.445 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.445 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:40.446 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232813+0000", 2020-01-16T00:13:40.446 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:40.446 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.446 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.446 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:40.446 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232812+0000", 2020-01-16T00:13:40.446 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.576e-06 2020-01-16T00:13:40.447 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.447 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.447 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:40.447 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232820+0000", 2020-01-16T00:13:40.447 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3079999999999997e-06 2020-01-16T00:13:40.447 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.447 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.447 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:40.447 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232829+0000", 2020-01-16T00:13:40.448 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7119999999999997e-06 2020-01-16T00:13:40.448 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.448 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.448 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:40.448 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232835+0000", 2020-01-16T00:13:40.448 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013438400000000001 2020-01-16T00:13:40.448 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.448 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.449 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.449 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232969+0000", 2020-01-16T00:13:40.449 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0562000000000001e-05 2020-01-16T00:13:40.449 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.449 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.449 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:40.449 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.233029+0000", 2020-01-16T00:13:40.450 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039354239999999999 2020-01-16T00:13:40.450 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.450 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.450 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.450 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.236965+0000", 2020-01-16T00:13:40.450 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0482999999999998e-05 2020-01-16T00:13:40.450 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.450 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.451 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:40.451 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.237015+0000", 2020-01-16T00:13:40.451 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0096086209999999995 2020-01-16T00:13:40.451 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.451 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.451 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.452 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.246624+0000", 2020-01-16T00:13:40.452 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6150999999999995e-05 2020-01-16T00:13:40.452 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.452 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.452 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:40.452 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.246720+0000", 2020-01-16T00:13:40.452 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015032899999999999 2020-01-16T00:13:40.452 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.453 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.453 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:40.453 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.246870+0000", 2020-01-16T00:13:40.453 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00064760700000000002 2020-01-16T00:13:40.453 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.453 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.453 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:40.454 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247518+0000", 2020-01-16T00:13:40.454 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000198531 2020-01-16T00:13:40.454 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.454 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.454 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:40.454 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247717+0000", 2020-01-16T00:13:40.454 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010189439999999999 2020-01-16T00:13:40.454 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.455 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.455 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:40.455 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248736+0000", 2020-01-16T00:13:40.455 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4863000000000001e-05 2020-01-16T00:13:40.455 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.455 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.455 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:40.456 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248770+0000", 2020-01-16T00:13:40.456 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2153000000000001e-05 2020-01-16T00:13:40.456 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.456 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.456 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:40.456 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248813+0000", 2020-01-16T00:13:40.456 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015999295 2020-01-16T00:13:40.456 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.457 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:40.457 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.457 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.457 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.457 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1892 3.0 3:0b1f384f:::smithi12313934-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e31)", 2020-01-16T00:13:40.457 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.232856+0000", 2020-01-16T00:13:40.457 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 43.100628288000003, 2020-01-16T00:13:40.458 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016171544, 2020-01-16T00:13:40.458 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:40.458 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:40.458 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:40.458 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:40.458 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:40.458 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1892 2020-01-16T00:13:40.459 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.459 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:40.459 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.459 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:40.459 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232856+0000", 2020-01-16T00:13:40.459 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:40.459 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.459 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.460 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:40.460 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232856+0000", 2020-01-16T00:13:40.460 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:40.460 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.460 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.460 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:40.460 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232855+0000", 2020-01-16T00:13:40.460 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8640000000000004e-06 2020-01-16T00:13:40.461 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.461 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.461 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:40.461 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232860+0000", 2020-01-16T00:13:40.461 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0000000000000001e-05 2020-01-16T00:13:40.461 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.461 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.461 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:40.462 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232870+0000", 2020-01-16T00:13:40.462 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7420000000000003e-06 2020-01-16T00:13:40.462 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.462 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.462 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:40.462 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232875+0000", 2020-01-16T00:13:40.462 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017933200000000001 2020-01-16T00:13:40.462 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.463 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.463 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.463 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.233055+0000", 2020-01-16T00:13:40.463 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6269e-05 2020-01-16T00:13:40.463 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.463 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.463 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:40.463 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.233101+0000", 2020-01-16T00:13:40.464 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039780090000000002 2020-01-16T00:13:40.464 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.464 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.464 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.464 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.237079+0000", 2020-01-16T00:13:40.464 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3677000000000001e-05 2020-01-16T00:13:40.464 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.464 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.465 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:40.465 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.237133+0000", 2020-01-16T00:13:40.465 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010036671 2020-01-16T00:13:40.465 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.465 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.465 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.465 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247169+0000", 2020-01-16T00:13:40.466 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010280699999999999 2020-01-16T00:13:40.466 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.466 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.466 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:40.466 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247272+0000", 2020-01-16T00:13:40.466 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001579708 2020-01-16T00:13:40.466 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.466 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.467 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.467 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248852+0000", 2020-01-16T00:13:40.467 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9704999999999996e-05 2020-01-16T00:13:40.467 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.467 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.467 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:40.467 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248942+0000", 2020-01-16T00:13:40.467 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5557999999999996e-05 2020-01-16T00:13:40.468 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.468 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.468 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:40.468 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.249027+0000", 2020-01-16T00:13:40.468 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016171544 2020-01-16T00:13:40.468 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.468 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:40.469 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.469 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.469 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.469 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1950 3.0 3:0b1f384f:::smithi12313934-33:head [write 1889359~541737] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-01-16T00:13:40.469 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.922819+0000", 2020-01-16T00:13:40.469 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 42.410664922999999, 2020-01-16T00:13:40.469 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017502846999999998, 2020-01-16T00:13:40.469 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:40.470 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:40.470 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:40.470 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:40.470 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:40.470 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1950 2020-01-16T00:13:40.470 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.470 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:40.471 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.471 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:40.471 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922819+0000", 2020-01-16T00:13:40.471 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:40.471 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.471 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.471 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:40.471 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922819+0000", 2020-01-16T00:13:40.472 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:40.472 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.472 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.472 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:40.472 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922818+0000", 2020-01-16T00:13:40.472 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000146408 2020-01-16T00:13:40.472 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.472 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.473 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:40.473 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922964+0000", 2020-01-16T00:13:40.473 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4720000000000001e-06 2020-01-16T00:13:40.473 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.473 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.473 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:40.473 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922974+0000", 2020-01-16T00:13:40.474 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0720000000000001e-06 2020-01-16T00:13:40.474 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.474 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.474 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:40.474 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922978+0000", 2020-01-16T00:13:40.474 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6152000000000001e-05 2020-01-16T00:13:40.474 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.475 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.475 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.475 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.923024+0000", 2020-01-16T00:13:40.475 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00087368199999999997 2020-01-16T00:13:40.475 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.475 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.476 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:40.476 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.923897+0000", 2020-01-16T00:13:40.476 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011944435999999999 2020-01-16T00:13:40.476 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.476 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.476 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.476 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.935842+0000", 2020-01-16T00:13:40.476 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000144146 2020-01-16T00:13:40.477 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.477 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.477 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:40.477 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.935986+0000", 2020-01-16T00:13:40.477 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019775500000000001 2020-01-16T00:13:40.477 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.478 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.478 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:40.478 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.936184+0000", 2020-01-16T00:13:40.478 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002539342 2020-01-16T00:13:40.479 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.479 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.479 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:40.479 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938723+0000", 2020-01-16T00:13:40.479 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014193699999999999 2020-01-16T00:13:40.479 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.479 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.479 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:40.480 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938865+0000", 2020-01-16T00:13:40.480 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013829020000000001 2020-01-16T00:13:40.480 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.480 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.480 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:40.480 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.940248+0000", 2020-01-16T00:13:40.480 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2558000000000001e-05 2020-01-16T00:13:40.480 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.481 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.481 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:40.481 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.940291+0000", 2020-01-16T00:13:40.481 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1217999999999998e-05 2020-01-16T00:13:40.481 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.481 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.481 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:40.481 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.940322+0000", 2020-01-16T00:13:40.482 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017502846999999998 2020-01-16T00:13:40.482 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.482 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:40.482 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.482 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.483 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.483 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1951 3.0 3:0b1f384f:::smithi12313934-33:head [write 2955792~541902] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-01-16T00:13:40.483 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.929019+0000", 2020-01-16T00:13:40.483 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 42.404464773000001, 2020-01-16T00:13:40.483 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021625710999999999, 2020-01-16T00:13:40.484 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:40.484 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:40.484 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:40.484 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:40.484 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:40.484 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1951 2020-01-16T00:13:40.485 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.485 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:40.485 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.485 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:40.485 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929019+0000", 2020-01-16T00:13:40.485 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:40.485 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.485 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.486 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:40.486 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929019+0000", 2020-01-16T00:13:40.486 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999989 2020-01-16T00:13:40.486 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.486 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.486 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:40.486 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929008+0000", 2020-01-16T00:13:40.486 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000212513 2020-01-16T00:13:40.487 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.487 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.487 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:40.487 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929221+0000", 2020-01-16T00:13:40.487 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3229e-05 2020-01-16T00:13:40.487 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.487 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.487 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:40.488 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929234+0000", 2020-01-16T00:13:40.488 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9170000000000003e-06 2020-01-16T00:13:40.488 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.488 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.488 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:40.488 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929240+0000", 2020-01-16T00:13:40.488 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4045000000000002e-05 2020-01-16T00:13:40.489 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.489 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.489 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.489 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929294+0000", 2020-01-16T00:13:40.489 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9797999999999996e-05 2020-01-16T00:13:40.489 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.489 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.489 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:40.490 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929384+0000", 2020-01-16T00:13:40.490 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083450729999999997 2020-01-16T00:13:40.490 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.490 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.490 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.490 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.937729+0000", 2020-01-16T00:13:40.490 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043251699999999999 2020-01-16T00:13:40.490 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.491 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.491 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:40.491 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938161+0000", 2020-01-16T00:13:40.491 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095115489999999994 2020-01-16T00:13:40.491 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.491 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.491 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.491 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.947673+0000", 2020-01-16T00:13:40.492 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012395099999999999 2020-01-16T00:13:40.492 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.492 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.492 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:40.492 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.947797+0000", 2020-01-16T00:13:40.492 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00024849200000000001 2020-01-16T00:13:40.492 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.492 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.493 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:40.493 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.948045+0000", 2020-01-16T00:13:40.493 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0025081050000000001 2020-01-16T00:13:40.493 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.493 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.493 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:40.493 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950553+0000", 2020-01-16T00:13:40.494 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9760000000000003e-06 2020-01-16T00:13:40.494 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.494 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.494 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:40.494 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950558+0000", 2020-01-16T00:13:40.494 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3286999999999999e-05 2020-01-16T00:13:40.494 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.494 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.495 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:40.495 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950581+0000", 2020-01-16T00:13:40.495 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2759000000000003e-05 2020-01-16T00:13:40.495 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.495 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.495 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:40.495 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950614+0000", 2020-01-16T00:13:40.496 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0525999999999999e-05 2020-01-16T00:13:40.496 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.496 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.496 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:40.496 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950645+0000", 2020-01-16T00:13:40.496 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021625710999999999 2020-01-16T00:13:40.496 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.496 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:40.497 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.497 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.497 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.497 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1952 3.0 3:0b1f384f:::smithi12313934-33:head [setxattr _header (56) in=7b,truncate 3563084] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-01-16T00:13:40.497 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:13:40.497 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 42.404235436, 2020-01-16T00:13:40.497 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021534685000000001, 2020-01-16T00:13:40.497 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:40.498 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:40.498 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:40.498 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:40.498 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:40.498 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1952 2020-01-16T00:13:40.498 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.498 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:40.498 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.499 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:40.499 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:13:40.499 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:40.499 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.499 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.499 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:40.499 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:13:40.500 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:40.500 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.500 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.500 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:40.500 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:13:40.501 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6670000000000004e-06 2020-01-16T00:13:40.501 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.501 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.501 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:40.501 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929252+0000", 2020-01-16T00:13:40.502 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9380000000000002e-06 2020-01-16T00:13:40.502 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.502 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.502 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:40.502 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929256+0000", 2020-01-16T00:13:40.502 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.712e-06 2020-01-16T00:13:40.502 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.503 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.503 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:40.503 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929264+0000", 2020-01-16T00:13:40.503 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018030300000000001 2020-01-16T00:13:40.503 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.503 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.503 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.504 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929444+0000", 2020-01-16T00:13:40.504 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7562000000000002e-05 2020-01-16T00:13:40.504 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.504 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.504 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:40.504 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929502+0000", 2020-01-16T00:13:40.504 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0087297859999999998 2020-01-16T00:13:40.504 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.505 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.505 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.505 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938232+0000", 2020-01-16T00:13:40.505 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0452999999999996e-05 2020-01-16T00:13:40.505 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.505 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.505 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:40.506 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938322+0000", 2020-01-16T00:13:40.506 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011061728 2020-01-16T00:13:40.506 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.506 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.506 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.506 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.949384+0000", 2020-01-16T00:13:40.506 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000117828 2020-01-16T00:13:40.506 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.507 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.507 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:40.507 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.949502+0000", 2020-01-16T00:13:40.507 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022468499999999999 2020-01-16T00:13:40.507 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.507 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.507 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:40.508 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.949726+0000", 2020-01-16T00:13:40.508 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00083532099999999998 2020-01-16T00:13:40.508 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.508 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.508 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:40.508 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950562+0000", 2020-01-16T00:13:40.508 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013017700000000001 2020-01-16T00:13:40.508 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.509 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.509 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:40.509 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950692+0000", 2020-01-16T00:13:40.509 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9813000000000001e-05 2020-01-16T00:13:40.509 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.509 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.509 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:40.510 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950722+0000", 2020-01-16T00:13:40.510 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0592000000000001e-05 2020-01-16T00:13:40.510 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.510 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.510 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:40.510 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950752+0000", 2020-01-16T00:13:40.510 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0920000000000002e-05 2020-01-16T00:13:40.510 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.511 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.511 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:40.511 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950783+0000", 2020-01-16T00:13:40.511 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021534685000000001 2020-01-16T00:13:40.511 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.511 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:40.512 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.512 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.512 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.512 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1953 3.0 3:0b1f384f:::smithi12313934-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e32)", 2020-01-16T00:13:40.512 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:13:40.512 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 42.404202445000003, 2020-01-16T00:13:40.513 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021667537000000001, 2020-01-16T00:13:40.513 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:40.513 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:40.513 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:40.513 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:40.513 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:40.513 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1953 2020-01-16T00:13:40.513 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.514 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:40.514 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.514 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:40.514 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:13:40.514 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:40.515 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.515 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.515 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:40.515 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:13:40.515 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:40.515 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.515 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.515 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:40.516 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:13:40.516 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7030000000000002e-06 2020-01-16T00:13:40.516 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.516 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.516 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:40.516 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929284+0000", 2020-01-16T00:13:40.516 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0909999999999993e-06 2020-01-16T00:13:40.517 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.517 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.517 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:40.517 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929292+0000", 2020-01-16T00:13:40.517 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8359999999999999e-06 2020-01-16T00:13:40.517 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.517 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.517 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:40.518 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929297+0000", 2020-01-16T00:13:40.518 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023538199999999999 2020-01-16T00:13:40.518 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.518 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.518 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.518 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929532+0000", 2020-01-16T00:13:40.518 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1173000000000004e-05 2020-01-16T00:13:40.518 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.519 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.519 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:40.519 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929603+0000", 2020-01-16T00:13:40.519 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0087813509999999997 2020-01-16T00:13:40.519 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.519 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.519 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.519 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938384+0000", 2020-01-16T00:13:40.520 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9752000000000006e-05 2020-01-16T00:13:40.520 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.520 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.520 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:40.520 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938454+0000", 2020-01-16T00:13:40.520 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011707851999999999 2020-01-16T00:13:40.520 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.520 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.521 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.521 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950162+0000", 2020-01-16T00:13:40.521 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011060099999999999 2020-01-16T00:13:40.521 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.521 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.521 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:40.521 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950273+0000", 2020-01-16T00:13:40.522 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00053989700000000001 2020-01-16T00:13:40.522 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.522 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.522 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.522 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950813+0000", 2020-01-16T00:13:40.522 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8128999999999999e-05 2020-01-16T00:13:40.522 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.522 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.523 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:40.523 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950871+0000", 2020-01-16T00:13:40.523 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8242000000000001e-05 2020-01-16T00:13:40.523 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.523 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.523 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:40.523 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950949+0000", 2020-01-16T00:13:40.523 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021667537000000001 2020-01-16T00:13:40.524 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.524 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:40.524 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.524 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.524 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.524 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2555 3.0 3:0b70bb20:::smithi12313934-46:head [setxattr _header (56) in=7b,truncate 2991679] snapc 0=[] ondisk+write+known_if_redirected e37)", 2020-01-16T00:13:40.524 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.573432+0000", 2020-01-16T00:13:40.524 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 27.760051793999999, 2020-01-16T00:13:40.525 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019893926999999999, 2020-01-16T00:13:40.525 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:40.525 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:40.525 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:40.525 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:40.525 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:40.525 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2555 2020-01-16T00:13:40.526 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.526 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:40.526 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.526 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:40.526 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573432+0000", 2020-01-16T00:13:40.526 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:40.526 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.527 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.527 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:40.527 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573432+0000", 2020-01-16T00:13:40.527 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:40.527 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.527 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.527 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:40.527 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573432+0000", 2020-01-16T00:13:40.528 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4290000000000001e-06 2020-01-16T00:13:40.528 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.528 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.528 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:40.528 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573435+0000", 2020-01-16T00:13:40.528 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1300000000000001e-06 2020-01-16T00:13:40.528 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.528 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.529 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:40.529 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573438+0000", 2020-01-16T00:13:40.529 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9399999999999996e-06 2020-01-16T00:13:40.529 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.529 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.529 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:40.530 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573445+0000", 2020-01-16T00:13:40.530 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013947269999999999 2020-01-16T00:13:40.530 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.530 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.530 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.530 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.574840+0000", 2020-01-16T00:13:40.530 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003850058 2020-01-16T00:13:40.531 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.531 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.531 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:40.531 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578690+0000", 2020-01-16T00:13:40.531 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012896489000000001 2020-01-16T00:13:40.531 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.531 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.531 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.532 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591586+0000", 2020-01-16T00:13:40.532 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4757000000000001e-05 2020-01-16T00:13:40.532 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.532 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.532 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:40.532 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591681+0000", 2020-01-16T00:13:40.532 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000191535 2020-01-16T00:13:40.532 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.533 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.533 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:40.533 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591873+0000", 2020-01-16T00:13:40.533 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00069888899999999996 2020-01-16T00:13:40.533 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.533 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.533 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:40.533 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.592571+0000", 2020-01-16T00:13:40.534 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00029525900000000003 2020-01-16T00:13:40.534 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.534 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.534 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:40.534 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.592867+0000", 2020-01-16T00:13:40.534 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035243500000000001 2020-01-16T00:13:40.534 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.535 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.535 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:40.535 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593219+0000", 2020-01-16T00:13:40.535 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0954e-05 2020-01-16T00:13:40.535 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.535 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.535 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:40.535 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593270+0000", 2020-01-16T00:13:40.536 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5813000000000002e-05 2020-01-16T00:13:40.536 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.536 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.536 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:40.536 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593326+0000", 2020-01-16T00:13:40.536 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019893926999999999 2020-01-16T00:13:40.536 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.536 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:40.537 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.537 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.537 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.537 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2556 3.0 3:0b70bb20:::smithi12313934-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e37)", 2020-01-16T00:13:40.537 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.573472+0000", 2020-01-16T00:13:40.537 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 27.760012224, 2020-01-16T00:13:40.537 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020070068999999999, 2020-01-16T00:13:40.538 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:40.538 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:40.538 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:40.538 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:40.538 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:40.538 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2556 2020-01-16T00:13:40.538 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.538 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:40.539 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.539 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:40.539 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573472+0000", 2020-01-16T00:13:40.539 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:40.539 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.539 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.539 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:40.539 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573472+0000", 2020-01-16T00:13:40.540 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:40.540 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.540 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.540 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:40.540 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573471+0000", 2020-01-16T00:13:40.540 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4449999999999999e-06 2020-01-16T00:13:40.540 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.541 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.541 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:40.541 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573479+0000", 2020-01-16T00:13:40.541 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9789999999999998e-06 2020-01-16T00:13:40.541 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.541 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.541 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:40.541 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573485+0000", 2020-01-16T00:13:40.542 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3270000000000001e-06 2020-01-16T00:13:40.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.542 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.542 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:40.542 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573488+0000", 2020-01-16T00:13:40.542 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052586849999999999 2020-01-16T00:13:40.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.542 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.543 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.543 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578747+0000", 2020-01-16T00:13:40.543 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5667000000000007e-05 2020-01-16T00:13:40.543 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.543 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.543 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:40.543 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578812+0000", 2020-01-16T00:13:40.544 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013599539000000001 2020-01-16T00:13:40.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.544 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.544 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.544 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.592412+0000", 2020-01-16T00:13:40.544 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6018000000000001e-05 2020-01-16T00:13:40.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.544 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.545 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:40.545 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.592508+0000", 2020-01-16T00:13:40.545 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00084582199999999998 2020-01-16T00:13:40.545 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.545 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.545 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.545 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593354+0000", 2020-01-16T00:13:40.545 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4984999999999999e-05 2020-01-16T00:13:40.546 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.546 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.546 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:40.546 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593449+0000", 2020-01-16T00:13:40.546 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3104999999999997e-05 2020-01-16T00:13:40.546 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.546 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.547 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:40.547 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593542+0000", 2020-01-16T00:13:40.547 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020070068999999999 2020-01-16T00:13:40.547 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.547 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:40.547 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.547 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.547 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.548 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3211 3.0 3:0b70bb20:::smithi12313934-46:head [setxattr _header (56) in=7b,truncate 2571537] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:13:40.548 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.581146+0000", 2020-01-16T00:13:40.548 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 10.752337654, 2020-01-16T00:13:40.548 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024318514999999999, 2020-01-16T00:13:40.548 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:40.548 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:40.548 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:40.548 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:40.549 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:40.549 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3211 2020-01-16T00:13:40.549 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.549 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:40.549 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.549 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:40.549 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581146+0000", 2020-01-16T00:13:40.550 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:40.550 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.550 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.550 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:40.550 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581146+0000", 2020-01-16T00:13:40.550 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:40.550 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.550 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.551 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:40.551 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581145+0000", 2020-01-16T00:13:40.551 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2990000000000002e-06 2020-01-16T00:13:40.551 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.551 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.551 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:40.551 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581151+0000", 2020-01-16T00:13:40.552 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.041e-06 2020-01-16T00:13:40.552 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.552 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.552 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:40.552 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581159+0000", 2020-01-16T00:13:40.552 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5859999999999998e-06 2020-01-16T00:13:40.553 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.553 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.553 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:40.553 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581163+0000", 2020-01-16T00:13:40.553 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020769790000000001 2020-01-16T00:13:40.553 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.553 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.553 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.554 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.583240+0000", 2020-01-16T00:13:40.554 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012019579999999999 2020-01-16T00:13:40.554 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.554 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.554 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:40.554 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584442+0000", 2020-01-16T00:13:40.554 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018359812 2020-01-16T00:13:40.554 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.555 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.555 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.555 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.602802+0000", 2020-01-16T00:13:40.555 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000133072 2020-01-16T00:13:40.555 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.555 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.555 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:40.556 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.602935+0000", 2020-01-16T00:13:40.556 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019007200000000001 2020-01-16T00:13:40.556 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.556 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.556 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:40.556 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.603125+0000", 2020-01-16T00:13:40.556 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001261751 2020-01-16T00:13:40.556 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.557 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.557 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:40.557 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.604387+0000", 2020-01-16T00:13:40.557 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000304987 2020-01-16T00:13:40.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.557 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.557 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:40.558 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.604692+0000", 2020-01-16T00:13:40.558 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00063369299999999995 2020-01-16T00:13:40.558 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.558 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.558 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:40.558 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605325+0000", 2020-01-16T00:13:40.558 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1470000000000005e-05 2020-01-16T00:13:40.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.559 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.559 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:40.559 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605397+0000", 2020-01-16T00:13:40.559 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7784e-05 2020-01-16T00:13:40.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.559 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.559 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:40.560 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605465+0000", 2020-01-16T00:13:40.560 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024318514999999999 2020-01-16T00:13:40.560 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.560 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:40.560 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.560 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.560 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.560 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3212 3.0 3:0b70bb20:::smithi12313934-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e45)", 2020-01-16T00:13:40.561 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.581171+0000", 2020-01-16T00:13:40.561 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 10.752312955000001, 2020-01-16T00:13:40.561 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024540131, 2020-01-16T00:13:40.561 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:40.561 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:40.561 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:40.561 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:40.561 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:40.562 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3212 2020-01-16T00:13:40.562 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.562 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:40.562 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.562 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:40.562 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581171+0000", 2020-01-16T00:13:40.562 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:40.563 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.563 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.563 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:40.563 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581171+0000", 2020-01-16T00:13:40.563 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:40.563 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.563 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.563 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:40.564 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581170+0000", 2020-01-16T00:13:40.564 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5259999999999996e-06 2020-01-16T00:13:40.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.564 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.564 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:40.564 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581178+0000", 2020-01-16T00:13:40.564 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.469e-06 2020-01-16T00:13:40.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.565 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.565 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:40.565 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581185+0000", 2020-01-16T00:13:40.565 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4460000000000003e-06 2020-01-16T00:13:40.565 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.565 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.565 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:40.566 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581190+0000", 2020-01-16T00:13:40.566 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003286696 2020-01-16T00:13:40.566 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.566 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.566 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.566 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584476+0000", 2020-01-16T00:13:40.566 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9681e-05 2020-01-16T00:13:40.567 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.567 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.567 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:40.567 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584536+0000", 2020-01-16T00:13:40.567 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019075267 2020-01-16T00:13:40.567 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.567 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.567 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.568 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.603611+0000", 2020-01-16T00:13:40.568 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5981000000000001e-05 2020-01-16T00:13:40.568 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.568 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.568 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:40.568 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.603707+0000", 2020-01-16T00:13:40.568 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0017922649999999999 2020-01-16T00:13:40.568 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.569 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.569 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.569 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605500+0000", 2020-01-16T00:13:40.569 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5396999999999995e-05 2020-01-16T00:13:40.569 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.569 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.569 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:40.570 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605595+0000", 2020-01-16T00:13:40.570 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000116038 2020-01-16T00:13:40.570 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.570 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.570 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:40.570 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605711+0000", 2020-01-16T00:13:40.570 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024540131 2020-01-16T00:13:40.570 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.571 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:40.571 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.571 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.571 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.571 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3435 3.0 3:0b70bb20:::smithi12313934-46:head [setxattr _header (56) in=7b,truncate 3075747] snapc 0=[] ondisk+write+known_if_redirected e47)", 2020-01-16T00:13:40.571 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:34.388891+0000", 2020-01-16T00:13:40.571 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.9445929389999996, 2020-01-16T00:13:40.571 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016440475999999999, 2020-01-16T00:13:40.572 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:40.572 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:40.572 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:40.572 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:40.572 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:40.572 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3435 2020-01-16T00:13:40.572 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.573 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:40.573 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.573 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:40.573 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388891+0000", 2020-01-16T00:13:40.573 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:40.573 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.573 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.574 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:40.574 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388891+0000", 2020-01-16T00:13:40.574 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:40.574 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.574 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.574 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:40.574 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388890+0000", 2020-01-16T00:13:40.574 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.647e-06 2020-01-16T00:13:40.575 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.575 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.575 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:40.575 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388894+0000", 2020-01-16T00:13:40.575 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2839999999999997e-06 2020-01-16T00:13:40.576 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.576 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.576 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:40.576 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388899+0000", 2020-01-16T00:13:40.576 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3149999999999999e-06 2020-01-16T00:13:40.576 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.576 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.576 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:40.577 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388901+0000", 2020-01-16T00:13:40.577 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001768543 2020-01-16T00:13:40.577 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.577 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.577 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.577 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.390670+0000", 2020-01-16T00:13:40.577 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0034533680000000001 2020-01-16T00:13:40.578 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.578 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.578 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:40.578 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.394123+0000", 2020-01-16T00:13:40.578 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010189962 2020-01-16T00:13:40.579 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.579 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.579 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.579 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404313+0000", 2020-01-16T00:13:40.579 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000108646 2020-01-16T00:13:40.580 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.580 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.580 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:40.580 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404422+0000", 2020-01-16T00:13:40.580 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119133 2020-01-16T00:13:40.580 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.580 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.580 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:40.581 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404541+0000", 2020-01-16T00:13:40.581 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000433764 2020-01-16T00:13:40.581 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.581 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.581 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:40.581 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404975+0000", 2020-01-16T00:13:40.581 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000266013 2020-01-16T00:13:40.582 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.582 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.582 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:40.582 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405241+0000", 2020-01-16T00:13:40.582 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5187000000000001e-05 2020-01-16T00:13:40.582 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.582 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.583 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:40.583 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405266+0000", 2020-01-16T00:13:40.583 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0889000000000002e-05 2020-01-16T00:13:40.583 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.583 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.583 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:40.583 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405297+0000", 2020-01-16T00:13:40.583 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4354999999999999e-05 2020-01-16T00:13:40.584 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.584 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.584 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:40.584 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405331+0000", 2020-01-16T00:13:40.584 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016440475999999999 2020-01-16T00:13:40.584 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.584 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:40.584 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.585 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.585 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.585 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3436 3.0 3:0b70bb20:::smithi12313934-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e47)", 2020-01-16T00:13:40.585 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:34.388920+0000", 2020-01-16T00:13:40.585 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.9445641499999997, 2020-01-16T00:13:40.585 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016552641, 2020-01-16T00:13:40.585 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:40.586 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:40.586 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:40.586 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:40.586 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:40.586 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3436 2020-01-16T00:13:40.586 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.586 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:40.587 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.587 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:40.587 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388920+0000", 2020-01-16T00:13:40.587 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:40.587 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.587 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.587 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:40.587 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388920+0000", 2020-01-16T00:13:40.588 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:40.588 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.588 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.588 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:40.588 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388919+0000", 2020-01-16T00:13:40.588 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.181e-06 2020-01-16T00:13:40.588 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.588 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.589 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:40.589 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388921+0000", 2020-01-16T00:13:40.589 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9940000000000003e-06 2020-01-16T00:13:40.589 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.589 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.589 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:40.589 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388928+0000", 2020-01-16T00:13:40.589 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3549999999999999e-06 2020-01-16T00:13:40.590 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.590 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.590 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:40.590 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388931+0000", 2020-01-16T00:13:40.590 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.005242483 2020-01-16T00:13:40.590 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.590 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.590 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.591 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.394173+0000", 2020-01-16T00:13:40.591 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7435999999999995e-05 2020-01-16T00:13:40.591 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.591 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.591 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:40.591 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.394251+0000", 2020-01-16T00:13:40.591 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010593286 2020-01-16T00:13:40.592 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.592 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.592 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.592 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404844+0000", 2020-01-16T00:13:40.592 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0620999999999999e-05 2020-01-16T00:13:40.592 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.592 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.592 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:40.593 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404925+0000", 2020-01-16T00:13:40.593 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00042800899999999997 2020-01-16T00:13:40.593 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.593 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.593 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:40.593 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405353+0000", 2020-01-16T00:13:40.593 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2250000000000003e-05 2020-01-16T00:13:40.593 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.594 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.594 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:40.594 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405405+0000", 2020-01-16T00:13:40.594 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7411999999999998e-05 2020-01-16T00:13:40.594 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:40.594 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:40.594 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:40.594 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405472+0000", 2020-01-16T00:13:40.595 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016552641 2020-01-16T00:13:40.595 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.595 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:40.595 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.595 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:40.595 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:40.595 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:40.597 INFO:tasks.rados.rados.0.smithi123.stdout:1395: expect (ObjNum 342 snap 0 seq_num 342) 2020-01-16T00:13:40.597 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:40.594+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:40.597 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-01-16T00:13:40.698 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:40.692+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:40.712 INFO:tasks.rados.rados.0.smithi123.stdout:1396: finishing write tid 1 to smithi12313934-6 2020-01-16T00:13:40.712 INFO:tasks.rados.rados.0.smithi123.stdout:1396: finishing write tid 2 to smithi12313934-6 2020-01-16T00:13:40.712 INFO:tasks.rados.rados.0.smithi123.stdout:1396: finishing write tid 3 to smithi12313934-6 2020-01-16T00:13:40.712 INFO:tasks.rados.rados.0.smithi123.stdout:1396: finishing write tid 5 to smithi12313934-6 2020-01-16T00:13:40.712 INFO:tasks.rados.rados.0.smithi123.stdout:1396: finishing write tid 6 to smithi12313934-6 2020-01-16T00:13:40.713 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 6 v 126 (ObjNum 450 snap 0 seq_num 450) dirty exists 2020-01-16T00:13:40.713 INFO:tasks.rados.rados.0.smithi123.stdout:1396: left oid 6 (ObjNum 450 snap 0 seq_num 450) 2020-01-16T00:13:40.713 INFO:tasks.rados.rados.0.smithi123.stdout:1398: expect (ObjNum 432 snap 0 seq_num 432) 2020-01-16T00:13:40.742 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:40.743 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:40.743 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:13:40.743 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:40.751 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-01-16T00:13:40.790 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-01-16T00:13:40.799 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:40.796+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:40.852 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:40.852 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:40.852 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:13:40.852 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:13:40.852 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:40.861 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-01-16T00:13:40.899 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:40.896+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:40.922 INFO:tasks.rados.rados.0.smithi123.stdout:1397: expect (ObjNum 415 snap 0 seq_num 415) 2020-01-16T00:13:40.926 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:13:40.945 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:13:40.960 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:40.966 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:13:40.969 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-01-16T00:13:40.990 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:13:41.000 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:40.998+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:41.020 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:13:41.050 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:13:41.065 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:41.077 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-01-16T00:13:41.080 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:13:41.082 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-01-16T00:13:41.082 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-01-16T00:13:41.082 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-01-16T00:13:41.082 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-01-16T00:13:41.083 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-01-16T00:13:41.083 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENXIO: problem getting command descriptions from osd.5 2020-01-16T00:13:41.083 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: problem getting command descriptions from osd.5 2020-01-16T00:13:41.083 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-01-16T00:13:41.083 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-01-16T00:13:41.098 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-01-16T00:13:41.099 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-01-16T00:13:41.139 INFO:tasks.rados.rados.0.smithi123.stdout:1399: finishing write tid 1 to smithi12313934-43 2020-01-16T00:13:41.139 INFO:tasks.rados.rados.0.smithi123.stdout:1399: finishing write tid 2 to smithi12313934-43 2020-01-16T00:13:41.139 INFO:tasks.rados.rados.0.smithi123.stdout:1399: finishing write tid 3 to smithi12313934-43 2020-01-16T00:13:41.139 INFO:tasks.rados.rados.0.smithi123.stdout:1399: finishing write tid 5 to smithi12313934-43 2020-01-16T00:13:41.139 INFO:tasks.rados.rados.0.smithi123.stdout:1399: finishing write tid 6 to smithi12313934-43 2020-01-16T00:13:41.139 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 43 v 205 (ObjNum 451 snap 0 seq_num 451) dirty exists 2020-01-16T00:13:41.140 INFO:tasks.rados.rados.0.smithi123.stdout:1399: left oid 43 (ObjNum 451 snap 0 seq_num 451) 2020-01-16T00:13:41.140 INFO:tasks.rados.rados.0.smithi123.stdout:1400: finishing write tid 1 to smithi12313934-23 2020-01-16T00:13:41.140 INFO:tasks.rados.rados.0.smithi123.stdout:1392: done (13 left) 2020-01-16T00:13:41.140 INFO:tasks.rados.rados.0.smithi123.stdout:1393: done (12 left) 2020-01-16T00:13:41.140 INFO:tasks.rados.rados.0.smithi123.stdout:1394: done (11 left) 2020-01-16T00:13:41.140 INFO:tasks.rados.rados.0.smithi123.stdout:1395: done (10 left) 2020-01-16T00:13:41.140 INFO:tasks.rados.rados.0.smithi123.stdout:1396: done (9 left) 2020-01-16T00:13:41.141 INFO:tasks.rados.rados.0.smithi123.stdout:1397: done (8 left) 2020-01-16T00:13:41.141 INFO:tasks.rados.rados.0.smithi123.stdout:1398: done (7 left) 2020-01-16T00:13:41.141 INFO:tasks.rados.rados.0.smithi123.stdout:1399: done (6 left) 2020-01-16T00:13:41.141 INFO:tasks.rados.rados.0.smithi123.stdout:1405: done (5 left) 2020-01-16T00:13:41.141 INFO:tasks.rados.rados.0.smithi123.stdout:1406: write oid 48 current snap is 0 2020-01-16T00:13:41.141 INFO:tasks.rados.rados.0.smithi123.stdout:1406: seq_num 457 ranges {480799=569973,1475918=769483,2700190=586604} 2020-01-16T00:13:41.141 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:41.134+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:41.144 INFO:tasks.rados.rados.0.smithi123.stdout:1406: writing smithi12313934-48 from 480799 to 1050772 tid 1 2020-01-16T00:13:41.151 INFO:tasks.rados.rados.0.smithi123.stdout:1406: writing smithi12313934-48 from 1475918 to 2245401 tid 2 2020-01-16T00:13:41.158 INFO:tasks.rados.rados.0.smithi123.stdout:1406: writing smithi12313934-48 from 2700190 to 3286794 tid 3 2020-01-16T00:13:41.158 INFO:tasks.rados.rados.0.smithi123.stdout:1407: read oid 5 snap -1 2020-01-16T00:13:41.158 INFO:tasks.rados.rados.0.smithi123.stdout:1407: expect deleted 2020-01-16T00:13:41.158 INFO:tasks.rados.rados.0.smithi123.stdout:1408: read oid 31 snap -1 2020-01-16T00:13:41.158 INFO:tasks.rados.rados.0.smithi123.stdout:1408: expect (ObjNum 437 snap 0 seq_num 437) 2020-01-16T00:13:41.159 INFO:tasks.rados.rados.0.smithi123.stdout:1409: write (excl) oid 19 current snap is 0 2020-01-16T00:13:41.159 INFO:tasks.rados.rados.0.smithi123.stdout:1409: seq_num 458 ranges {497606=645222,1934400=629494,3243966=453640} 2020-01-16T00:13:41.164 INFO:tasks.rados.rados.0.smithi123.stdout:1409: writing smithi12313934-19 from 497606 to 1142828 tid 1 2020-01-16T00:13:41.173 INFO:tasks.rados.rados.0.smithi123.stdout:1409: writing smithi12313934-19 from 1934400 to 2563894 tid 2 2020-01-16T00:13:41.179 INFO:tasks.rados.rados.0.smithi123.stdout:1409: writing smithi12313934-19 from 3243966 to 3697606 tid 3 2020-01-16T00:13:41.180 INFO:tasks.rados.rados.0.smithi123.stdout:1410: read oid 29 snap -1 2020-01-16T00:13:41.180 INFO:tasks.rados.rados.0.smithi123.stdout:1410: expect (ObjNum 442 snap 0 seq_num 442) 2020-01-16T00:13:41.180 INFO:tasks.rados.rados.0.smithi123.stdout:1411: write oid 50 current snap is 0 2020-01-16T00:13:41.180 INFO:tasks.rados.rados.0.smithi123.stdout:1411: seq_num 459 ranges {514413=720471,1876529=773152,3438683=275730} 2020-01-16T00:13:41.186 INFO:tasks.rados.rados.0.smithi123.stdout:1411: writing smithi12313934-50 from 514413 to 1234884 tid 1 2020-01-16T00:13:41.189 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:41.194 INFO:tasks.rados.rados.0.smithi123.stdout:1411: writing smithi12313934-50 from 1876529 to 2649681 tid 2 2020-01-16T00:13:41.198 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-01-16T00:13:41.201 INFO:tasks.rados.rados.0.smithi123.stdout:1411: writing smithi12313934-50 from 3438683 to 3714413 tid 3 2020-01-16T00:13:41.201 INFO:tasks.rados.rados.0.smithi123.stdout:1412: read oid 4 snap -1 2020-01-16T00:13:41.201 INFO:tasks.rados.rados.0.smithi123.stdout:1412: expect (ObjNum 387 snap 0 seq_num 387) 2020-01-16T00:13:41.202 INFO:tasks.rados.rados.0.smithi123.stdout:1413: write (excl) oid 32 current snap is 0 2020-01-16T00:13:41.202 INFO:tasks.rados.rados.0.smithi123.stdout:1413: seq_num 460 ranges {531220=795720,1935011=516810,3066106=420420} 2020-01-16T00:13:41.204 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:41.202+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:41.209 INFO:tasks.rados.rados.0.smithi123.stdout:1413: writing smithi12313934-32 from 531220 to 1326940 tid 1 2020-01-16T00:13:41.216 INFO:tasks.rados.rados.0.smithi123.stdout:1413: writing smithi12313934-32 from 1935011 to 2451821 tid 2 2020-01-16T00:13:41.222 INFO:tasks.rados.rados.0.smithi123.stdout:1413: writing smithi12313934-32 from 3066106 to 3486526 tid 3 2020-01-16T00:13:41.222 INFO:tasks.rados.rados.0.smithi123.stdout:1414: write oid 44 current snap is 0 2020-01-16T00:13:41.222 INFO:tasks.rados.rados.0.smithi123.stdout:1414: seq_num 461 ranges {548027=470969,1593493=776821,3093529=631692} 2020-01-16T00:13:41.228 INFO:tasks.rados.rados.0.smithi123.stdout:1414: writing smithi12313934-44 from 548027 to 1018996 tid 1 2020-01-16T00:13:41.235 INFO:tasks.rados.rados.0.smithi123.stdout:1414: writing smithi12313934-44 from 1593493 to 2370314 tid 2 2020-01-16T00:13:41.243 INFO:tasks.rados.rados.0.smithi123.stdout:1414: writing smithi12313934-44 from 3093529 to 3725221 tid 3 2020-01-16T00:13:41.244 INFO:tasks.rados.rados.0.smithi123.stdout:1415: read oid 15 snap -1 2020-01-16T00:13:41.244 INFO:tasks.rados.rados.0.smithi123.stdout:1415: expect (ObjNum 384 snap 0 seq_num 384) 2020-01-16T00:13:41.244 INFO:tasks.rados.rados.0.smithi123.stdout:1416: read oid 25 snap -1 2020-01-16T00:13:41.244 INFO:tasks.rados.rados.0.smithi123.stdout:1416: expect (ObjNum 438 snap 0 seq_num 438) 2020-01-16T00:13:41.244 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:41.244 INFO:tasks.rados.rados.0.smithi123.stdout:1400: finishing write tid 2 to smithi12313934-23 2020-01-16T00:13:41.244 INFO:tasks.rados.rados.0.smithi123.stdout:1400: finishing write tid 3 to smithi12313934-23 2020-01-16T00:13:41.245 INFO:tasks.rados.rados.0.smithi123.stdout:1400: finishing write tid 5 to smithi12313934-23 2020-01-16T00:13:41.245 INFO:tasks.rados.rados.0.smithi123.stdout:1400: finishing write tid 6 to smithi12313934-23 2020-01-16T00:13:41.245 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 23 v 154 (ObjNum 452 snap 0 seq_num 452) dirty exists 2020-01-16T00:13:41.245 INFO:tasks.rados.rados.0.smithi123.stdout:1400: left oid 23 (ObjNum 452 snap 0 seq_num 452) 2020-01-16T00:13:41.245 INFO:tasks.rados.rados.0.smithi123.stdout:1401: finishing write tid 1 to smithi12313934-30 2020-01-16T00:13:41.245 INFO:tasks.rados.rados.0.smithi123.stdout:1400: done (15 left) 2020-01-16T00:13:41.245 INFO:tasks.rados.rados.0.smithi123.stdout:1417: read oid 9 snap -1 2020-01-16T00:13:41.246 INFO:tasks.rados.rados.0.smithi123.stdout:1417: expect (ObjNum 435 snap 0 seq_num 435) 2020-01-16T00:13:41.247 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:41.247 INFO:tasks.rados.rados.0.smithi123.stdout:1401: finishing write tid 2 to smithi12313934-30 2020-01-16T00:13:41.247 INFO:tasks.rados.rados.0.smithi123.stdout:1401: finishing write tid 3 to smithi12313934-30 2020-01-16T00:13:41.247 INFO:tasks.rados.rados.0.smithi123.stdout:1401: finishing write tid 5 to smithi12313934-30 2020-01-16T00:13:41.247 INFO:tasks.rados.rados.0.smithi123.stdout:1401: finishing write tid 6 to smithi12313934-30 2020-01-16T00:13:41.248 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 30 v 150 (ObjNum 453 snap 0 seq_num 453) dirty exists 2020-01-16T00:13:41.248 INFO:tasks.rados.rados.0.smithi123.stdout:1401: left oid 30 (ObjNum 453 snap 0 seq_num 453) 2020-01-16T00:13:41.248 INFO:tasks.rados.rados.0.smithi123.stdout:1402: expect (ObjNum 422 snap 0 seq_num 422) 2020-01-16T00:13:41.303 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:41.307 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:41.302+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:41.312 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-01-16T00:13:41.407 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:41.402+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:41.430 INFO:tasks.rados.rados.0.smithi123.stdout:1403: finishing write tid 1 to smithi12313934-8 2020-01-16T00:13:41.430 INFO:tasks.rados.rados.0.smithi123.stdout:1403: finishing write tid 2 to smithi12313934-8 2020-01-16T00:13:41.430 INFO:tasks.rados.rados.0.smithi123.stdout:1403: finishing write tid 3 to smithi12313934-8 2020-01-16T00:13:41.431 INFO:tasks.rados.rados.0.smithi123.stdout:1403: finishing write tid 5 to smithi12313934-8 2020-01-16T00:13:41.431 INFO:tasks.rados.rados.0.smithi123.stdout:1403: finishing write tid 6 to smithi12313934-8 2020-01-16T00:13:41.431 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 8 v 103 (ObjNum 454 snap 0 seq_num 454) dirty exists 2020-01-16T00:13:41.432 INFO:tasks.rados.rados.0.smithi123.stdout:1403: left oid 8 (ObjNum 454 snap 0 seq_num 454) 2020-01-16T00:13:41.433 INFO:tasks.rados.rados.0.smithi123.stdout:1404: finishing write tid 1 to smithi12313934-40 2020-01-16T00:13:41.433 INFO:tasks.rados.rados.0.smithi123.stdout:1404: finishing write tid 2 to smithi12313934-40 2020-01-16T00:13:41.433 INFO:tasks.rados.rados.0.smithi123.stdout:1404: finishing write tid 3 to smithi12313934-40 2020-01-16T00:13:41.433 INFO:tasks.rados.rados.0.smithi123.stdout:1404: finishing write tid 5 to smithi12313934-40 2020-01-16T00:13:41.433 INFO:tasks.rados.rados.0.smithi123.stdout:1404: finishing write tid 6 to smithi12313934-40 2020-01-16T00:13:41.433 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 40 v 210 (ObjNum 455 snap 0 seq_num 455) dirty exists 2020-01-16T00:13:41.433 INFO:tasks.rados.rados.0.smithi123.stdout:1404: left oid 40 (ObjNum 455 snap 0 seq_num 455) 2020-01-16T00:13:41.434 INFO:tasks.rados.rados.0.smithi123.stdout:1406: finishing write tid 1 to smithi12313934-48 2020-01-16T00:13:41.434 INFO:tasks.rados.rados.0.smithi123.stdout:1406: finishing write tid 2 to smithi12313934-48 2020-01-16T00:13:41.434 INFO:tasks.rados.rados.0.smithi123.stdout:1408: expect (ObjNum 437 snap 0 seq_num 437) 2020-01-16T00:13:41.468 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:41.469 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-01-16T00:13:41.507 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:41.504+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:41.608 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:41.604+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:41.613 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:41.623 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-01-16T00:13:41.694 INFO:tasks.rados.rados.0.smithi123.stdout:1409: finishing write tid 1 to smithi12313934-19 2020-01-16T00:13:41.694 INFO:tasks.rados.rados.0.smithi123.stdout:1406: finishing write tid 3 to smithi12313934-48 2020-01-16T00:13:41.694 INFO:tasks.rados.rados.0.smithi123.stdout:1406: finishing write tid 5 to smithi12313934-48 2020-01-16T00:13:41.695 INFO:tasks.rados.rados.0.smithi123.stdout:1406: finishing write tid 6 to smithi12313934-48 2020-01-16T00:13:41.695 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 48 v 170 (ObjNum 457 snap 0 seq_num 457) dirty exists 2020-01-16T00:13:41.695 INFO:tasks.rados.rados.0.smithi123.stdout:1406: left oid 48 (ObjNum 457 snap 0 seq_num 457) 2020-01-16T00:13:41.695 INFO:tasks.rados.rados.0.smithi123.stdout:1409: finishing write tid 2 to smithi12313934-19 2020-01-16T00:13:41.695 INFO:tasks.rados.rados.0.smithi123.stdout:1409: finishing write tid 3 to smithi12313934-19 2020-01-16T00:13:41.695 INFO:tasks.rados.rados.0.smithi123.stdout:1409: finishing write tid 5 to smithi12313934-19 2020-01-16T00:13:41.695 INFO:tasks.rados.rados.0.smithi123.stdout:1409: finishing write tid 6 to smithi12313934-19 2020-01-16T00:13:41.695 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 19 v 204 (ObjNum 458 snap 0 seq_num 458) dirty exists 2020-01-16T00:13:41.696 INFO:tasks.rados.rados.0.smithi123.stdout:1409: left oid 19 (ObjNum 458 snap 0 seq_num 458) 2020-01-16T00:13:41.696 INFO:tasks.rados.rados.0.smithi123.stdout:1410: expect (ObjNum 442 snap 0 seq_num 442) 2020-01-16T00:13:41.709 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:41.704+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:41.727 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:41.731 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-01-16T00:13:41.810 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:41.808+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:41.845 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:41.855 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-01-16T00:13:41.911 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:41.906+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:41.931 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2020-01-16T00:13:41.931 INFO:tasks.rados.rados.0.smithi123.stdout:1411: finishing write tid 1 to smithi12313934-50 2020-01-16T00:13:41.932 INFO:tasks.rados.rados.0.smithi123.stdout:1412: expect (ObjNum 387 snap 0 seq_num 387) 2020-01-16T00:13:42.009 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:42.011 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:42.006+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:42.019 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-01-16T00:13:42.038 INFO:tasks.rados.rados.0.smithi123.stdout:1411: finishing write tid 2 to smithi12313934-50 2020-01-16T00:13:42.038 INFO:tasks.rados.rados.0.smithi123.stdout:1413: finishing write tid 1 to smithi12313934-32 2020-01-16T00:13:42.038 INFO:tasks.rados.rados.0.smithi123.stdout:1401: done (15 left) 2020-01-16T00:13:42.039 INFO:tasks.rados.rados.0.smithi123.stdout:1402: done (14 left) 2020-01-16T00:13:42.039 INFO:tasks.rados.rados.0.smithi123.stdout:1403: done (13 left) 2020-01-16T00:13:42.039 INFO:tasks.rados.rados.0.smithi123.stdout:1404: done (12 left) 2020-01-16T00:13:42.039 INFO:tasks.rados.rados.0.smithi123.stdout:1406: done (11 left) 2020-01-16T00:13:42.039 INFO:tasks.rados.rados.0.smithi123.stdout:1407: done (10 left) 2020-01-16T00:13:42.039 INFO:tasks.rados.rados.0.smithi123.stdout:1408: done (9 left) 2020-01-16T00:13:42.039 INFO:tasks.rados.rados.0.smithi123.stdout:1409: done (8 left) 2020-01-16T00:13:42.040 INFO:tasks.rados.rados.0.smithi123.stdout:1410: done (7 left) 2020-01-16T00:13:42.040 INFO:tasks.rados.rados.0.smithi123.stdout:1412: done (6 left) 2020-01-16T00:13:42.040 INFO:tasks.rados.rados.0.smithi123.stdout:1418: read oid 30 snap -1 2020-01-16T00:13:42.040 INFO:tasks.rados.rados.0.smithi123.stdout:1418: expect (ObjNum 453 snap 0 seq_num 453) 2020-01-16T00:13:42.040 INFO:tasks.rados.rados.0.smithi123.stdout:1419: write oid 39 current snap is 0 2020-01-16T00:13:42.040 INFO:tasks.rados.rados.0.smithi123.stdout:1419: seq_num 462 ranges {564834=546218,1651975=520479,2720952=559317} 2020-01-16T00:13:42.044 INFO:tasks.rados.rados.0.smithi123.stdout:1419: writing smithi12313934-39 from 564834 to 1111052 tid 1 2020-01-16T00:13:42.050 INFO:tasks.rados.rados.0.smithi123.stdout:1419: writing smithi12313934-39 from 1651975 to 2172454 tid 2 2020-01-16T00:13:42.057 INFO:tasks.rados.rados.0.smithi123.stdout:1419: writing smithi12313934-39 from 2720952 to 3280269 tid 3 2020-01-16T00:13:42.058 INFO:tasks.rados.rados.0.smithi123.stdout:1420: write oid 7 current snap is 0 2020-01-16T00:13:42.058 INFO:tasks.rados.rados.0.smithi123.stdout:1411: finishing write tid 3 to smithi12313934-50 2020-01-16T00:13:42.058 INFO:tasks.rados.rados.0.smithi123.stdout:1420: seq_num 463 ranges {581641=621467,1994104=780490,3432022=349619} 2020-01-16T00:13:42.064 INFO:tasks.rados.rados.0.smithi123.stdout:1420: writing smithi12313934-7 from 581641 to 1203108 tid 1 2020-01-16T00:13:42.073 INFO:tasks.rados.rados.0.smithi123.stdout:1420: writing smithi12313934-7 from 1994104 to 2774594 tid 2 2020-01-16T00:13:42.081 INFO:tasks.rados.rados.0.smithi123.stdout:1420: writing smithi12313934-7 from 3432022 to 3781641 tid 3 2020-01-16T00:13:42.081 INFO:tasks.rados.rados.0.smithi123.stdout:1421: write oid 11 current snap is 0 2020-01-16T00:13:42.081 INFO:tasks.rados.rados.0.smithi123.stdout:1421: seq_num 464 ranges {598448=413069,1768939=524148,2775798=581861,3781488=16960} 2020-01-16T00:13:42.084 INFO:tasks.rados.rados.0.smithi123.stdout:1421: writing smithi12313934-11 from 598448 to 1011517 tid 1 2020-01-16T00:13:42.098 INFO:tasks.rados.rados.0.smithi123.stdout:1421: writing smithi12313934-11 from 1768939 to 2293087 tid 2 2020-01-16T00:13:42.098 INFO:tasks.rados.rados.0.smithi123.stdout:1421: writing smithi12313934-11 from 2775798 to 3357659 tid 3 2020-01-16T00:13:42.108 INFO:tasks.rados.rados.0.smithi123.stdout:1421: writing smithi12313934-11 from 3781488 to 3798448 tid 4 2020-01-16T00:13:42.109 INFO:tasks.rados.rados.0.smithi123.stdout:1422: write oid 35 current snap is 0 2020-01-16T00:13:42.109 INFO:tasks.rados.rados.0.smithi123.stdout:1422: seq_num 465 ranges {615255=488318,1827421=784159,3203221=612034} 2020-01-16T00:13:42.109 INFO:tasks.rados.rados.0.smithi123.stdout:1422: writing smithi12313934-35 from 615255 to 1103573 tid 1 2020-01-16T00:13:42.112 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:42.110+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:42.114 INFO:tasks.rados.rados.0.smithi123.stdout:1422: writing smithi12313934-35 from 1827421 to 2611580 tid 2 2020-01-16T00:13:42.121 INFO:tasks.rados.rados.0.smithi123.stdout:1422: writing smithi12313934-35 from 3203221 to 3815255 tid 3 2020-01-16T00:13:42.121 INFO:tasks.rados.rados.0.smithi123.stdout:1423: read oid 2 snap -1 2020-01-16T00:13:42.121 INFO:tasks.rados.rados.0.smithi123.stdout:1423: expect (ObjNum 448 snap 0 seq_num 448) 2020-01-16T00:13:42.121 INFO:tasks.rados.rados.0.smithi123.stdout:1424: write (excl) oid 45 current snap is 0 2020-01-16T00:13:42.121 INFO:tasks.rados.rados.0.smithi123.stdout:1424: seq_num 466 ranges {632062=563567,1885903=527817,2830644=720758} 2020-01-16T00:13:42.127 INFO:tasks.rados.rados.0.smithi123.stdout:1424: writing smithi12313934-45 from 632062 to 1195629 tid 1 2020-01-16T00:13:42.134 INFO:tasks.rados.rados.0.smithi123.stdout:1424: writing smithi12313934-45 from 1885903 to 2413720 tid 2 2020-01-16T00:13:42.141 INFO:tasks.rados.rados.0.smithi123.stdout:1424: writing smithi12313934-45 from 2830644 to 3551402 tid 3 2020-01-16T00:13:42.141 INFO:tasks.rados.rados.0.smithi123.stdout:1425: read oid 14 snap -1 2020-01-16T00:13:42.141 INFO:tasks.rados.rados.0.smithi123.stdout:1425: expect (ObjNum 362 snap 0 seq_num 362) 2020-01-16T00:13:42.141 INFO:tasks.rados.rados.0.smithi123.stdout:1426: write (excl) oid 40 current snap is 0 2020-01-16T00:13:42.141 INFO:tasks.rados.rados.0.smithi123.stdout:1426: seq_num 467 ranges {648869=638816,1828032=787828,3141714=532030} 2020-01-16T00:13:42.147 INFO:tasks.rados.rados.0.smithi123.stdout:1426: writing smithi12313934-40 from 648869 to 1287685 tid 1 2020-01-16T00:13:42.148 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:42.155 INFO:tasks.rados.rados.0.smithi123.stdout:1426: writing smithi12313934-40 from 1828032 to 2615860 tid 2 2020-01-16T00:13:42.159 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-01-16T00:13:42.162 INFO:tasks.rados.rados.0.smithi123.stdout:1426: writing smithi12313934-40 from 3141714 to 3673744 tid 3 2020-01-16T00:13:42.162 INFO:tasks.rados.rados.0.smithi123.stdout:1427: read oid 38 snap -1 2020-01-16T00:13:42.162 INFO:tasks.rados.rados.0.smithi123.stdout:1427: expect deleted 2020-01-16T00:13:42.162 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:42.163 INFO:tasks.rados.rados.0.smithi123.stdout:1411: finishing write tid 5 to smithi12313934-50 2020-01-16T00:13:42.163 INFO:tasks.rados.rados.0.smithi123.stdout:1411: finishing write tid 6 to smithi12313934-50 2020-01-16T00:13:42.163 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 50 v 172 (ObjNum 459 snap 0 seq_num 459) dirty exists 2020-01-16T00:13:42.163 INFO:tasks.rados.rados.0.smithi123.stdout:1411: left oid 50 (ObjNum 459 snap 0 seq_num 459) 2020-01-16T00:13:42.163 INFO:tasks.rados.rados.0.smithi123.stdout:1413: finishing write tid 2 to smithi12313934-32 2020-01-16T00:13:42.163 INFO:tasks.rados.rados.0.smithi123.stdout:1411: done (15 left) 2020-01-16T00:13:42.163 INFO:tasks.rados.rados.0.smithi123.stdout:1428: read oid 43 snap -1 2020-01-16T00:13:42.164 INFO:tasks.rados.rados.0.smithi123.stdout:1428: expect (ObjNum 451 snap 0 seq_num 451) 2020-01-16T00:13:42.164 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:42.164 INFO:tasks.rados.rados.0.smithi123.stdout:1413: finishing write tid 3 to smithi12313934-32 2020-01-16T00:13:42.164 INFO:tasks.rados.rados.0.smithi123.stdout:1413: finishing write tid 5 to smithi12313934-32 2020-01-16T00:13:42.164 INFO:tasks.rados.rados.0.smithi123.stdout:1413: finishing write tid 6 to smithi12313934-32 2020-01-16T00:13:42.164 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 32 v 149 (ObjNum 460 snap 0 seq_num 460) dirty exists 2020-01-16T00:13:42.164 INFO:tasks.rados.rados.0.smithi123.stdout:1413: left oid 32 (ObjNum 460 snap 0 seq_num 460) 2020-01-16T00:13:42.164 INFO:tasks.rados.rados.0.smithi123.stdout:1414: finishing write tid 1 to smithi12313934-44 2020-01-16T00:13:42.165 INFO:tasks.rados.rados.0.smithi123.stdout:1414: finishing write tid 2 to smithi12313934-44 2020-01-16T00:13:42.165 INFO:tasks.rados.rados.0.smithi123.stdout:1413: done (15 left) 2020-01-16T00:13:42.165 INFO:tasks.rados.rados.0.smithi123.stdout:1429: write oid 12 current snap is 0 2020-01-16T00:13:42.165 INFO:tasks.rados.rados.0.smithi123.stdout:1429: seq_num 468 ranges {665676=714065,1886514=531486,3169137=696539} 2020-01-16T00:13:42.171 INFO:tasks.rados.rados.0.smithi123.stdout:1429: writing smithi12313934-12 from 665676 to 1379741 tid 1 2020-01-16T00:13:42.176 INFO:tasks.rados.rados.0.smithi123.stdout:1429: writing smithi12313934-12 from 1886514 to 2418000 tid 2 2020-01-16T00:13:42.185 INFO:tasks.rados.rados.0.smithi123.stdout:1429: writing smithi12313934-12 from 3169137 to 3865676 tid 3 2020-01-16T00:13:42.185 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:42.185 INFO:tasks.rados.rados.0.smithi123.stdout:1414: finishing write tid 3 to smithi12313934-44 2020-01-16T00:13:42.185 INFO:tasks.rados.rados.0.smithi123.stdout:1414: finishing write tid 5 to smithi12313934-44 2020-01-16T00:13:42.185 INFO:tasks.rados.rados.0.smithi123.stdout:1414: finishing write tid 6 to smithi12313934-44 2020-01-16T00:13:42.185 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 44 v 214 (ObjNum 461 snap 0 seq_num 461) dirty exists 2020-01-16T00:13:42.186 INFO:tasks.rados.rados.0.smithi123.stdout:1414: left oid 44 (ObjNum 461 snap 0 seq_num 461) 2020-01-16T00:13:42.186 INFO:tasks.rados.rados.0.smithi123.stdout:1416: expect (ObjNum 438 snap 0 seq_num 438) 2020-01-16T00:13:42.213 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:42.208+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:42.255 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:42.264 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-01-16T00:13:42.284 INFO:tasks.rados.rados.0.smithi123.stdout:1417: expect (ObjNum 435 snap 0 seq_num 435) 2020-01-16T00:13:42.314 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:42.310+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:42.415 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:42.412+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:42.418 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:42.428 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-01-16T00:13:42.457 INFO:tasks.rados.rados.0.smithi123.stdout:1415: expect (ObjNum 384 snap 0 seq_num 384) 2020-01-16T00:13:42.515 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:42.512+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:42.578 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:42.588 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-01-16T00:13:42.602 INFO:tasks.rados.rados.0.smithi123.stdout:1418: expect (ObjNum 453 snap 0 seq_num 453) 2020-01-16T00:13:42.620 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:42.618+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:42.721 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:42.718+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:42.743 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:42.752 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-01-16T00:13:42.822 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:42.818+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:42.829 INFO:teuthology.orchestra.run.smithi123.stderr:noscrub is set 2020-01-16T00:13:42.861 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:42.870 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-01-16T00:13:42.902 INFO:tasks.rados.rados.0.smithi123.stdout:1419: finishing write tid 1 to smithi12313934-39 2020-01-16T00:13:42.902 INFO:tasks.rados.rados.0.smithi123.stdout:1419: finishing write tid 2 to smithi12313934-39 2020-01-16T00:13:42.902 INFO:tasks.rados.rados.0.smithi123.stdout:1419: finishing write tid 3 to smithi12313934-39 2020-01-16T00:13:42.902 INFO:tasks.rados.rados.0.smithi123.stdout:1419: finishing write tid 5 to smithi12313934-39 2020-01-16T00:13:42.902 INFO:tasks.rados.rados.0.smithi123.stdout:1419: finishing write tid 6 to smithi12313934-39 2020-01-16T00:13:42.903 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 39 v 218 (ObjNum 462 snap 0 seq_num 462) dirty exists 2020-01-16T00:13:42.903 INFO:tasks.rados.rados.0.smithi123.stdout:1419: left oid 39 (ObjNum 462 snap 0 seq_num 462) 2020-01-16T00:13:42.903 INFO:tasks.rados.rados.0.smithi123.stdout:1420: finishing write tid 1 to smithi12313934-7 2020-01-16T00:13:42.903 INFO:tasks.rados.rados.0.smithi123.stdout:1420: finishing write tid 2 to smithi12313934-7 2020-01-16T00:13:42.903 INFO:tasks.rados.rados.0.smithi123.stdout:1420: finishing write tid 3 to smithi12313934-7 2020-01-16T00:13:42.903 INFO:tasks.rados.rados.0.smithi123.stdout:1420: finishing write tid 5 to smithi12313934-7 2020-01-16T00:13:42.903 INFO:tasks.rados.rados.0.smithi123.stdout:1420: finishing write tid 6 to smithi12313934-7 2020-01-16T00:13:42.904 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 7 v 208 (ObjNum 463 snap 0 seq_num 463) dirty exists 2020-01-16T00:13:42.904 INFO:tasks.rados.rados.0.smithi123.stdout:1420: left oid 7 (ObjNum 463 snap 0 seq_num 463) 2020-01-16T00:13:42.904 INFO:tasks.rados.rados.0.smithi123.stdout:1421: finishing write tid 1 to smithi12313934-11 2020-01-16T00:13:42.904 INFO:tasks.rados.rados.0.smithi123.stdout:1421: finishing write tid 2 to smithi12313934-11 2020-01-16T00:13:42.904 INFO:tasks.rados.rados.0.smithi123.stdout:1421: finishing write tid 3 to smithi12313934-11 2020-01-16T00:13:42.904 INFO:tasks.rados.rados.0.smithi123.stdout:1421: finishing write tid 4 to smithi12313934-11 2020-01-16T00:13:42.904 INFO:tasks.rados.rados.0.smithi123.stdout:1421: finishing write tid 6 to smithi12313934-11 2020-01-16T00:13:42.905 INFO:tasks.rados.rados.0.smithi123.stdout:1414: done (15 left) 2020-01-16T00:13:42.905 INFO:tasks.rados.rados.0.smithi123.stdout:1415: done (14 left) 2020-01-16T00:13:42.905 INFO:tasks.rados.rados.0.smithi123.stdout:1416: done (13 left) 2020-01-16T00:13:42.905 INFO:tasks.rados.rados.0.smithi123.stdout:1417: done (12 left) 2020-01-16T00:13:42.905 INFO:tasks.rados.rados.0.smithi123.stdout:1418: done (11 left) 2020-01-16T00:13:42.905 INFO:tasks.rados.rados.0.smithi123.stdout:1419: done (10 left) 2020-01-16T00:13:42.906 INFO:tasks.rados.rados.0.smithi123.stdout:1420: done (9 left) 2020-01-16T00:13:42.906 INFO:tasks.rados.rados.0.smithi123.stdout:1430: read oid 9 snap -1 2020-01-16T00:13:42.906 INFO:tasks.rados.rados.0.smithi123.stdout:1430: expect (ObjNum 435 snap 0 seq_num 435) 2020-01-16T00:13:42.906 INFO:tasks.rados.rados.0.smithi123.stdout:1431: write oid 49 current snap is 0 2020-01-16T00:13:42.906 INFO:tasks.rados.rados.0.smithi123.stdout:1421: finishing write tid 7 to smithi12313934-11 2020-01-16T00:13:42.906 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 11 v 131 (ObjNum 464 snap 0 seq_num 464) dirty exists 2020-01-16T00:13:42.906 INFO:tasks.rados.rados.0.smithi123.stdout:1421: left oid 11 (ObjNum 464 snap 0 seq_num 464) 2020-01-16T00:13:42.907 INFO:tasks.rados.rados.0.smithi123.stdout:1422: finishing write tid 1 to smithi12313934-35 2020-01-16T00:13:42.907 INFO:tasks.rados.rados.0.smithi123.stdout:1422: finishing write tid 2 to smithi12313934-35 2020-01-16T00:13:42.907 INFO:tasks.rados.rados.0.smithi123.stdout:1422: finishing write tid 3 to smithi12313934-35 2020-01-16T00:13:42.907 INFO:tasks.rados.rados.0.smithi123.stdout:1422: finishing write tid 5 to smithi12313934-35 2020-01-16T00:13:42.907 INFO:tasks.rados.rados.0.smithi123.stdout:1422: finishing write tid 6 to smithi12313934-35 2020-01-16T00:13:42.907 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 35 v 158 (ObjNum 465 snap 0 seq_num 465) dirty exists 2020-01-16T00:13:42.907 INFO:tasks.rados.rados.0.smithi123.stdout:1422: left oid 35 (ObjNum 465 snap 0 seq_num 465) 2020-01-16T00:13:42.908 INFO:tasks.rados.rados.0.smithi123.stdout:1423: expect (ObjNum 448 snap 0 seq_num 448) 2020-01-16T00:13:42.923 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:42.918+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:42.974 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:42.984 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-01-16T00:13:43.028 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:43.026+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:43.089 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:43.096 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-01-16T00:13:43.100 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-01-16T00:13:43.140 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:43.138+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:43.185 INFO:tasks.rados.rados.0.smithi123.stdout:1424: finishing write tid 1 to smithi12313934-45 2020-01-16T00:13:43.186 INFO:tasks.rados.rados.0.smithi123.stdout:1424: finishing write tid 2 to smithi12313934-45 2020-01-16T00:13:43.186 INFO:tasks.rados.rados.0.smithi123.stdout:1424: finishing write tid 3 to smithi12313934-45 2020-01-16T00:13:43.186 INFO:tasks.rados.rados.0.smithi123.stdout:1424: finishing write tid 5 to smithi12313934-45 2020-01-16T00:13:43.186 INFO:tasks.rados.rados.0.smithi123.stdout:1424: finishing write tid 6 to smithi12313934-45 2020-01-16T00:13:43.186 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 45 v 145 (ObjNum 466 snap 0 seq_num 466) dirty exists 2020-01-16T00:13:43.186 INFO:tasks.rados.rados.0.smithi123.stdout:1424: left oid 45 (ObjNum 466 snap 0 seq_num 466) 2020-01-16T00:13:43.186 INFO:tasks.rados.rados.0.smithi123.stdout:1425: expect (ObjNum 362 snap 0 seq_num 362) 2020-01-16T00:13:43.225 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:43.224+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:43.241 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:43.251 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-01-16T00:13:43.274 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:13:43.291 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:13:43.309 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:13:43.326 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:43.324+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:43.327 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:13:43.347 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:13:43.350 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:43.350 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:13:43.351 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:13:43.351 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:13:43.351 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.351 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:939 3.4 3:2bdd9e88:::smithi12313934-4:head [write 1681102~474010] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:43.351 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.908392+0000", 2020-01-16T00:13:43.351 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 65.437242733999994, 2020-01-16T00:13:43.351 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019294697, 2020-01-16T00:13:43.352 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:43.352 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:43.352 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:43.352 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:43.352 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:43.352 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 939 2020-01-16T00:13:43.352 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.353 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:43.353 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.353 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:43.353 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908392+0000", 2020-01-16T00:13:43.353 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:43.353 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.353 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.353 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:43.354 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908392+0000", 2020-01-16T00:13:43.354 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:43.354 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.354 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.354 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:43.354 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908390+0000", 2020-01-16T00:13:43.354 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015440399999999999 2020-01-16T00:13:43.354 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.355 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.355 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:43.355 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908545+0000", 2020-01-16T00:13:43.355 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3926e-05 2020-01-16T00:13:43.355 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.355 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.355 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:43.356 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908559+0000", 2020-01-16T00:13:43.356 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2200000000000003e-06 2020-01-16T00:13:43.356 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.356 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.356 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:43.356 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908566+0000", 2020-01-16T00:13:43.356 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020594799999999998 2020-01-16T00:13:43.356 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.357 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.357 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.357 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910625+0000", 2020-01-16T00:13:43.357 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118672 2020-01-16T00:13:43.357 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.357 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.357 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:43.358 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910744+0000", 2020-01-16T00:13:43.358 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031940969999999999 2020-01-16T00:13:43.358 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.358 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.358 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.358 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.913938+0000", 2020-01-16T00:13:43.358 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000309621 2020-01-16T00:13:43.358 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.359 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.359 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:43.359 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.914248+0000", 2020-01-16T00:13:43.359 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.008818579 2020-01-16T00:13:43.359 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.359 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.359 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.359 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.923066+0000", 2020-01-16T00:13:43.360 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000108989 2020-01-16T00:13:43.360 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.360 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.360 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:43.360 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.923175+0000", 2020-01-16T00:13:43.360 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019128299999999999 2020-01-16T00:13:43.360 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.360 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.361 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:13:43.361 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.923367+0000", 2020-01-16T00:13:43.361 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0032083760000000002 2020-01-16T00:13:43.361 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.361 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.361 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:43.361 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926575+0000", 2020-01-16T00:13:43.361 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2475999999999999e-05 2020-01-16T00:13:43.362 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.362 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.362 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:43.362 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926607+0000", 2020-01-16T00:13:43.362 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00099399800000000006 2020-01-16T00:13:43.362 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.362 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.362 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:43.363 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927601+0000", 2020-01-16T00:13:43.363 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0958999999999999e-05 2020-01-16T00:13:43.363 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.363 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.363 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:43.363 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927642+0000", 2020-01-16T00:13:43.363 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4747999999999999e-05 2020-01-16T00:13:43.364 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.364 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.364 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:43.364 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927687+0000", 2020-01-16T00:13:43.364 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019294697 2020-01-16T00:13:43.364 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.364 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:43.364 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.365 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.365 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.365 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:940 3.4 3:2bdd9e88:::smithi12313934-4:head [write 2687912~589420] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:43.365 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.915144+0000", 2020-01-16T00:13:43.365 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 65.430491365999998, 2020-01-16T00:13:43.365 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016092538999999999, 2020-01-16T00:13:43.365 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:43.365 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:43.366 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:43.366 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:43.366 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:43.366 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 940 2020-01-16T00:13:43.366 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.366 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:43.366 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.367 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:43.367 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915144+0000", 2020-01-16T00:13:43.367 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:43.367 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.367 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.367 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:43.367 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915144+0000", 2020-01-16T00:13:43.367 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:43.368 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.368 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.368 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:43.368 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915141+0000", 2020-01-16T00:13:43.368 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016366300000000001 2020-01-16T00:13:43.368 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.368 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.368 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:43.369 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915305+0000", 2020-01-16T00:13:43.369 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4847e-05 2020-01-16T00:13:43.369 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.369 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.369 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:43.369 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915320+0000", 2020-01-16T00:13:43.369 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2599999999999999e-06 2020-01-16T00:13:43.369 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.370 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.370 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:43.370 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915327+0000", 2020-01-16T00:13:43.370 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4282000000000005e-05 2020-01-16T00:13:43.370 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.370 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.370 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.371 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915401+0000", 2020-01-16T00:13:43.371 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4554000000000006e-05 2020-01-16T00:13:43.371 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.371 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.371 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:43.371 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915466+0000", 2020-01-16T00:13:43.372 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0090821900000000004 2020-01-16T00:13:43.372 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.372 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.372 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.372 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.924548+0000", 2020-01-16T00:13:43.372 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000114696 2020-01-16T00:13:43.373 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.373 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.373 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:43.373 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.924663+0000", 2020-01-16T00:13:43.373 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000168651 2020-01-16T00:13:43.373 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.373 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.373 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:13:43.374 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.924832+0000", 2020-01-16T00:13:43.374 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018253430000000001 2020-01-16T00:13:43.374 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.374 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.374 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:43.374 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926657+0000", 2020-01-16T00:13:43.374 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9957999999999999e-05 2020-01-16T00:13:43.374 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.375 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.375 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:43.375 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926697+0000", 2020-01-16T00:13:43.375 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0044685250000000001 2020-01-16T00:13:43.375 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.375 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.375 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:43.375 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931165+0000", 2020-01-16T00:13:43.376 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9211000000000001e-05 2020-01-16T00:13:43.376 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.376 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.376 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:43.376 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931205+0000", 2020-01-16T00:13:43.376 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1699999999999998e-05 2020-01-16T00:13:43.376 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.377 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.377 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:43.377 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931236+0000", 2020-01-16T00:13:43.377 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016092538999999999 2020-01-16T00:13:43.377 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.377 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:43.377 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.377 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.378 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.378 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:941 3.4 3:2bdd9e88:::smithi12313934-4:head [setxattr _header (55) in=7b,truncate 3277332] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:43.378 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.915341+0000", 2020-01-16T00:13:43.378 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 65.430294548000006, 2020-01-16T00:13:43.378 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016603769000000001, 2020-01-16T00:13:43.378 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:43.378 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:43.378 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:43.379 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:43.379 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:43.379 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 941 2020-01-16T00:13:43.379 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.379 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:43.379 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.379 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:43.379 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915341+0000", 2020-01-16T00:13:43.380 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:43.380 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.380 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.380 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:43.380 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915341+0000", 2020-01-16T00:13:43.380 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:43.380 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.380 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.381 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:43.381 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915340+0000", 2020-01-16T00:13:43.381 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.896e-06 2020-01-16T00:13:43.381 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.381 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.381 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:43.381 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915346+0000", 2020-01-16T00:13:43.382 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2730000000000002e-06 2020-01-16T00:13:43.382 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.382 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.382 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:43.382 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915354+0000", 2020-01-16T00:13:43.382 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0616e-05 2020-01-16T00:13:43.382 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.382 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.383 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:43.383 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915374+0000", 2020-01-16T00:13:43.383 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000133487 2020-01-16T00:13:43.383 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.383 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.383 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.383 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915508+0000", 2020-01-16T00:13:43.383 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8971000000000003e-05 2020-01-16T00:13:43.384 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.384 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.384 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:43.384 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915557+0000", 2020-01-16T00:13:43.384 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010138787999999999 2020-01-16T00:13:43.384 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.384 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.384 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.385 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.925696+0000", 2020-01-16T00:13:43.385 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012825400000000001 2020-01-16T00:13:43.385 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.385 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.385 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:43.385 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.925824+0000", 2020-01-16T00:13:43.385 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017706500000000001 2020-01-16T00:13:43.386 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.386 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.386 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:13:43.386 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926001+0000", 2020-01-16T00:13:43.386 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00079988900000000003 2020-01-16T00:13:43.386 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.387 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.387 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:43.387 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926801+0000", 2020-01-16T00:13:43.387 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119324 2020-01-16T00:13:43.388 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.388 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.388 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:43.388 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926920+0000", 2020-01-16T00:13:43.388 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048879910000000004 2020-01-16T00:13:43.388 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.388 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.388 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:43.389 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931808+0000", 2020-01-16T00:13:43.389 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6999999999999997e-05 2020-01-16T00:13:43.389 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.389 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.389 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:43.389 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931855+0000", 2020-01-16T00:13:43.389 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9092999999999996e-05 2020-01-16T00:13:43.390 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.390 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.390 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:43.390 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931944+0000", 2020-01-16T00:13:43.390 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016603769000000001 2020-01-16T00:13:43.390 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.390 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:43.390 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.391 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.391 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.391 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:942 3.4 3:2bdd9e88:::smithi12313934-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:13:43.391 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.915389+0000", 2020-01-16T00:13:43.391 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 65.430246182000005, 2020-01-16T00:13:43.391 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016820563, 2020-01-16T00:13:43.391 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:43.391 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:43.392 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:43.392 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:43.392 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:43.392 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 942 2020-01-16T00:13:43.392 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.392 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:43.392 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.393 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:43.393 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915389+0000", 2020-01-16T00:13:43.393 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:43.393 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.393 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.393 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:43.393 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915389+0000", 2020-01-16T00:13:43.393 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:43.394 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.394 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.394 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:43.394 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915388+0000", 2020-01-16T00:13:43.394 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.683e-06 2020-01-16T00:13:43.394 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.394 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.395 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:43.395 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915391+0000", 2020-01-16T00:13:43.395 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4839999999999998e-06 2020-01-16T00:13:43.395 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.395 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.395 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:43.395 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915399+0000", 2020-01-16T00:13:43.395 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2259999999999999e-06 2020-01-16T00:13:43.396 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.396 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.396 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:43.396 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915403+0000", 2020-01-16T00:13:43.396 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00031301199999999999 2020-01-16T00:13:43.396 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.396 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.396 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.397 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915716+0000", 2020-01-16T00:13:43.397 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2079999999999998e-05 2020-01-16T00:13:43.397 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.397 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.397 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:43.397 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915748+0000", 2020-01-16T00:13:43.397 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010547076000000001 2020-01-16T00:13:43.397 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.398 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.398 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.398 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926295+0000", 2020-01-16T00:13:43.398 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8040000000000005e-05 2020-01-16T00:13:43.398 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.398 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.398 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:43.398 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926373+0000", 2020-01-16T00:13:43.399 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056061990000000001 2020-01-16T00:13:43.399 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.399 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.399 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.399 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931979+0000", 2020-01-16T00:13:43.399 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011540399999999999 2020-01-16T00:13:43.399 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.399 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.400 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:43.400 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932095+0000", 2020-01-16T00:13:43.400 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011482899999999999 2020-01-16T00:13:43.400 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.400 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.400 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:43.400 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932210+0000", 2020-01-16T00:13:43.401 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016820563 2020-01-16T00:13:43.401 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.401 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:43.401 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.401 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.401 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.401 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:957 3.0 3:00c146d5:::smithi12313934-18:head [write 510946~632952] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:43.401 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.033290+0000", 2020-01-16T00:13:43.402 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 65.312345344999997, 2020-01-16T00:13:43.402 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016764102999999999, 2020-01-16T00:13:43.402 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:43.402 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:43.402 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:43.402 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:43.402 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:43.403 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 957 2020-01-16T00:13:43.403 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.403 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:43.403 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.403 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:43.403 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033290+0000", 2020-01-16T00:13:43.403 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:43.403 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.404 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.404 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:43.404 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033290+0000", 2020-01-16T00:13:43.404 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999952 2020-01-16T00:13:43.404 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.404 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.404 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:43.404 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033285+0000", 2020-01-16T00:13:43.405 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000405917 2020-01-16T00:13:43.405 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.405 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.405 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:43.405 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033691+0000", 2020-01-16T00:13:43.405 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6585999999999997e-05 2020-01-16T00:13:43.405 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.406 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.406 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:43.406 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033738+0000", 2020-01-16T00:13:43.406 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8374000000000001e-05 2020-01-16T00:13:43.406 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.406 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.406 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:43.406 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033756+0000", 2020-01-16T00:13:43.407 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013002900000000001 2020-01-16T00:13:43.407 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.407 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.407 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.407 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033886+0000", 2020-01-16T00:13:43.407 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035882600000000002 2020-01-16T00:13:43.407 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.407 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.408 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:43.408 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.034245+0000", 2020-01-16T00:13:43.408 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0072798489999999997 2020-01-16T00:13:43.408 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.408 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.408 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.408 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.041525+0000", 2020-01-16T00:13:43.408 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000140488 2020-01-16T00:13:43.409 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.409 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.409 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:43.409 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.041665+0000", 2020-01-16T00:13:43.409 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000233766 2020-01-16T00:13:43.409 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.409 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.409 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:43.410 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.041899+0000", 2020-01-16T00:13:43.410 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0035766539999999999 2020-01-16T00:13:43.410 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.410 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.410 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:43.410 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.045475+0000", 2020-01-16T00:13:43.410 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013333150000000001 2020-01-16T00:13:43.410 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.411 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.411 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:43.411 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.046809+0000", 2020-01-16T00:13:43.411 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031447860000000001 2020-01-16T00:13:43.411 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.411 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.411 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:43.411 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.049954+0000", 2020-01-16T00:13:43.412 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2318000000000003e-05 2020-01-16T00:13:43.412 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.412 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.412 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:43.412 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.050006+0000", 2020-01-16T00:13:43.412 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7972000000000001e-05 2020-01-16T00:13:43.412 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.412 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.412 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:43.413 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.050054+0000", 2020-01-16T00:13:43.413 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016764102999999999 2020-01-16T00:13:43.413 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.413 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:43.413 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.413 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.413 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.413 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1395 3.4 3:240c7503:::smithi12313934-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e28)", 2020-01-16T00:13:43.414 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:46.625191+0000", 2020-01-16T00:13:43.414 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 56.720444016999998, 2020-01-16T00:13:43.414 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015878716000000001, 2020-01-16T00:13:43.414 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:43.414 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:43.414 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:43.414 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:43.414 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:43.415 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1395 2020-01-16T00:13:43.415 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.415 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:43.415 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.415 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:43.415 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625191+0000", 2020-01-16T00:13:43.415 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:43.415 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.416 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.416 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:43.416 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625191+0000", 2020-01-16T00:13:43.416 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:43.416 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.416 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.416 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:43.416 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625191+0000", 2020-01-16T00:13:43.417 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.1090000000000001e-06 2020-01-16T00:13:43.417 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.417 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.417 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:43.417 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625193+0000", 2020-01-16T00:13:43.417 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2730000000000008e-06 2020-01-16T00:13:43.417 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.417 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.418 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:43.418 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625202+0000", 2020-01-16T00:13:43.418 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3700000000000002e-06 2020-01-16T00:13:43.418 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.418 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.418 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:43.418 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.625204+0000", 2020-01-16T00:13:43.418 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056573049999999996 2020-01-16T00:13:43.419 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.419 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.419 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.419 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630862+0000", 2020-01-16T00:13:43.419 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6047000000000001e-05 2020-01-16T00:13:43.419 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.419 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.420 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:43.420 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.630918+0000", 2020-01-16T00:13:43.420 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0077941180000000001 2020-01-16T00:13:43.420 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.420 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.420 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.420 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.638712+0000", 2020-01-16T00:13:43.420 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118096 2020-01-16T00:13:43.421 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.421 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.421 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:43.421 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.638830+0000", 2020-01-16T00:13:43.421 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0019986740000000002 2020-01-16T00:13:43.421 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.421 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.421 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.422 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.640829+0000", 2020-01-16T00:13:43.422 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012562399999999999 2020-01-16T00:13:43.422 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.422 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.422 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:43.422 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.640954+0000", 2020-01-16T00:13:43.422 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000115536 2020-01-16T00:13:43.422 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.423 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.423 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:43.423 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:46.641070+0000", 2020-01-16T00:13:43.423 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015878716000000001 2020-01-16T00:13:43.423 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.423 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:43.423 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.423 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.424 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.424 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1639 3.0 3:00c146d5:::smithi12313934-18:head [write 423576~401598] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:13:43.424 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.774317+0000", 2020-01-16T00:13:43.424 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 51.571318605999998, 2020-01-16T00:13:43.424 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018305219000000001, 2020-01-16T00:13:43.424 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:43.424 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:43.424 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:43.425 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:43.425 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:43.425 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1639 2020-01-16T00:13:43.425 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.426 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:43.426 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.427 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:43.427 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774317+0000", 2020-01-16T00:13:43.427 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:43.427 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.427 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.427 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:43.427 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774317+0000", 2020-01-16T00:13:43.427 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:43.428 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.428 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.428 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:43.428 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774313+0000", 2020-01-16T00:13:43.428 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000111173 2020-01-16T00:13:43.428 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.428 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.428 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:43.429 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774425+0000", 2020-01-16T00:13:43.429 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7598999999999999e-05 2020-01-16T00:13:43.429 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.429 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.429 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:43.429 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774442+0000", 2020-01-16T00:13:43.429 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0830000000000004e-06 2020-01-16T00:13:43.430 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.430 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.430 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:43.430 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774450+0000", 2020-01-16T00:13:43.430 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0017886309999999999 2020-01-16T00:13:43.430 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.430 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.430 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.431 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.776239+0000", 2020-01-16T00:13:43.431 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3789e-05 2020-01-16T00:13:43.431 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.431 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.431 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op must wait for map", 2020-01-16T00:13:43.431 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.776253+0000", 2020-01-16T00:13:43.431 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095730729999999997 2020-01-16T00:13:43.431 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.432 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.432 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.432 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.785826+0000", 2020-01-16T00:13:43.432 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014220300000000001 2020-01-16T00:13:43.432 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.432 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.432 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:43.433 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.785968+0000", 2020-01-16T00:13:43.433 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023618900000000001 2020-01-16T00:13:43.433 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.433 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.433 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:43.433 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.786204+0000", 2020-01-16T00:13:43.433 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0062297389999999998 2020-01-16T00:13:43.433 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.434 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.434 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:43.434 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792434+0000", 2020-01-16T00:13:43.434 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9635999999999998e-05 2020-01-16T00:13:43.434 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.434 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.434 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:43.434 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792473+0000", 2020-01-16T00:13:43.435 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6456000000000002e-05 2020-01-16T00:13:43.435 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.435 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.435 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:43.435 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792540+0000", 2020-01-16T00:13:43.435 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9873999999999998e-05 2020-01-16T00:13:43.435 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.436 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.436 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:43.436 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792580+0000", 2020-01-16T00:13:43.436 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1924e-05 2020-01-16T00:13:43.436 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.436 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.436 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:43.436 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792622+0000", 2020-01-16T00:13:43.437 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018305219000000001 2020-01-16T00:13:43.437 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.437 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:43.437 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.437 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.437 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.437 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1890 3.0 3:0b1f384f:::smithi12313934-33:head [write 2797915~596476] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:13:43.437 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.232534+0000", 2020-01-16T00:13:43.438 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 46.113100635000002, 2020-01-16T00:13:43.438 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016106137, 2020-01-16T00:13:43.438 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:43.438 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:43.438 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:43.438 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:43.438 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:43.439 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1890 2020-01-16T00:13:43.439 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.439 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:43.439 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.439 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:43.439 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232534+0000", 2020-01-16T00:13:43.439 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:43.439 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.440 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.440 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:43.440 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232534+0000", 2020-01-16T00:13:43.440 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:43.440 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.440 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.440 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:43.440 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232532+0000", 2020-01-16T00:13:43.441 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00024232100000000001 2020-01-16T00:13:43.441 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.441 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.441 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:43.441 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232774+0000", 2020-01-16T00:13:43.441 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4601e-05 2020-01-16T00:13:43.441 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.441 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.442 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:43.442 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232789+0000", 2020-01-16T00:13:43.442 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8410000000000006e-06 2020-01-16T00:13:43.442 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.442 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.442 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:43.442 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232798+0000", 2020-01-16T00:13:43.442 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9756999999999997e-05 2020-01-16T00:13:43.443 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.443 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.443 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.443 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232857+0000", 2020-01-16T00:13:43.443 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4678000000000007e-05 2020-01-16T00:13:43.443 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.443 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.443 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:43.444 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232922+0000", 2020-01-16T00:13:43.444 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033861020000000002 2020-01-16T00:13:43.444 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.444 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.444 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.444 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.236308+0000", 2020-01-16T00:13:43.444 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000568996 2020-01-16T00:13:43.445 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.445 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.445 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:43.445 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.236877+0000", 2020-01-16T00:13:43.445 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0085444240000000001 2020-01-16T00:13:43.445 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.445 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.446 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.446 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.245422+0000", 2020-01-16T00:13:43.446 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5595999999999998e-05 2020-01-16T00:13:43.446 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.446 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.446 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:43.446 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.245507+0000", 2020-01-16T00:13:43.446 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016984300000000001 2020-01-16T00:13:43.447 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.447 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.447 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:43.447 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.245677+0000", 2020-01-16T00:13:43.447 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001649495 2020-01-16T00:13:43.447 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.447 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.447 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:43.448 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247326+0000", 2020-01-16T00:13:43.448 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023308399999999999 2020-01-16T00:13:43.448 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.448 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.448 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:43.448 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247560+0000", 2020-01-16T00:13:43.448 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00099501800000000007 2020-01-16T00:13:43.448 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.449 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.449 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:43.449 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248555+0000", 2020-01-16T00:13:43.449 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5936999999999998e-05 2020-01-16T00:13:43.449 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.449 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.449 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:43.450 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248591+0000", 2020-01-16T00:13:43.450 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0108999999999997e-05 2020-01-16T00:13:43.450 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.450 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.450 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:43.450 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248641+0000", 2020-01-16T00:13:43.450 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016106137 2020-01-16T00:13:43.450 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.451 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:43.451 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.451 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.451 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.451 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1891 3.0 3:0b1f384f:::smithi12313934-33:head [setxattr _header (56) in=7b,truncate 3395014] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:13:43.451 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.232813+0000", 2020-01-16T00:13:43.451 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 46.112821832999998, 2020-01-16T00:13:43.451 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015999295, 2020-01-16T00:13:43.452 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:43.452 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:43.452 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:43.452 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:43.452 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:43.452 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1891 2020-01-16T00:13:43.452 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.453 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:43.453 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.453 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:43.453 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232813+0000", 2020-01-16T00:13:43.453 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:43.453 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.453 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.454 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:43.454 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232813+0000", 2020-01-16T00:13:43.454 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:43.454 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.454 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.454 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:43.454 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232812+0000", 2020-01-16T00:13:43.454 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.576e-06 2020-01-16T00:13:43.455 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.455 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.455 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:43.455 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232820+0000", 2020-01-16T00:13:43.455 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3079999999999997e-06 2020-01-16T00:13:43.455 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.455 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.456 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:43.456 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232829+0000", 2020-01-16T00:13:43.456 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7119999999999997e-06 2020-01-16T00:13:43.456 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.456 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.456 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:43.456 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232835+0000", 2020-01-16T00:13:43.457 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013438400000000001 2020-01-16T00:13:43.457 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.457 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.457 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.457 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232969+0000", 2020-01-16T00:13:43.457 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0562000000000001e-05 2020-01-16T00:13:43.457 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.457 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.458 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:43.458 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.233029+0000", 2020-01-16T00:13:43.458 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039354239999999999 2020-01-16T00:13:43.458 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.458 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.458 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.458 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.236965+0000", 2020-01-16T00:13:43.459 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0482999999999998e-05 2020-01-16T00:13:43.459 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.459 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.459 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:43.459 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.237015+0000", 2020-01-16T00:13:43.459 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0096086209999999995 2020-01-16T00:13:43.460 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.460 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.460 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.460 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.246624+0000", 2020-01-16T00:13:43.460 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6150999999999995e-05 2020-01-16T00:13:43.460 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.460 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.460 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:43.461 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.246720+0000", 2020-01-16T00:13:43.461 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015032899999999999 2020-01-16T00:13:43.461 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.461 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.461 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:43.461 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.246870+0000", 2020-01-16T00:13:43.461 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00064760700000000002 2020-01-16T00:13:43.462 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.462 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.462 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:43.462 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247518+0000", 2020-01-16T00:13:43.462 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000198531 2020-01-16T00:13:43.462 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.462 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.463 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:43.463 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247717+0000", 2020-01-16T00:13:43.463 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010189439999999999 2020-01-16T00:13:43.463 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.463 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.463 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:43.463 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248736+0000", 2020-01-16T00:13:43.464 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4863000000000001e-05 2020-01-16T00:13:43.464 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.464 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.464 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:43.464 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248770+0000", 2020-01-16T00:13:43.464 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2153000000000001e-05 2020-01-16T00:13:43.464 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.464 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.465 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:43.465 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248813+0000", 2020-01-16T00:13:43.465 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015999295 2020-01-16T00:13:43.465 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.465 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:43.465 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.465 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.465 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.466 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1892 3.0 3:0b1f384f:::smithi12313934-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e31)", 2020-01-16T00:13:43.466 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.232856+0000", 2020-01-16T00:13:43.466 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 46.112779609, 2020-01-16T00:13:43.466 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016171544, 2020-01-16T00:13:43.466 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:43.466 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:43.466 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:43.467 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:43.467 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:43.467 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1892 2020-01-16T00:13:43.467 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.467 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:43.467 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.467 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:43.467 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232856+0000", 2020-01-16T00:13:43.468 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:43.468 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.468 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.468 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:43.468 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232856+0000", 2020-01-16T00:13:43.468 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:43.468 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.468 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.469 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:43.469 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232855+0000", 2020-01-16T00:13:43.469 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8640000000000004e-06 2020-01-16T00:13:43.469 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.469 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.469 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:43.469 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232860+0000", 2020-01-16T00:13:43.469 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0000000000000001e-05 2020-01-16T00:13:43.470 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.470 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.470 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:43.470 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232870+0000", 2020-01-16T00:13:43.470 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7420000000000003e-06 2020-01-16T00:13:43.470 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.470 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.470 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:43.471 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232875+0000", 2020-01-16T00:13:43.471 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017933200000000001 2020-01-16T00:13:43.471 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.471 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.471 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.471 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.233055+0000", 2020-01-16T00:13:43.471 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6269e-05 2020-01-16T00:13:43.472 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.472 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.472 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:43.472 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.233101+0000", 2020-01-16T00:13:43.472 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039780090000000002 2020-01-16T00:13:43.472 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.472 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.472 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.473 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.237079+0000", 2020-01-16T00:13:43.473 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3677000000000001e-05 2020-01-16T00:13:43.473 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.473 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.473 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:43.473 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.237133+0000", 2020-01-16T00:13:43.474 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010036671 2020-01-16T00:13:43.474 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.474 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.474 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.474 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247169+0000", 2020-01-16T00:13:43.474 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010280699999999999 2020-01-16T00:13:43.474 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.474 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.475 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:43.475 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247272+0000", 2020-01-16T00:13:43.475 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001579708 2020-01-16T00:13:43.476 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.476 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.476 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.477 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248852+0000", 2020-01-16T00:13:43.477 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9704999999999996e-05 2020-01-16T00:13:43.477 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.477 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.477 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:43.477 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248942+0000", 2020-01-16T00:13:43.477 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5557999999999996e-05 2020-01-16T00:13:43.478 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.478 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.478 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:43.478 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.249027+0000", 2020-01-16T00:13:43.478 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016171544 2020-01-16T00:13:43.478 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.478 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:43.479 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.479 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.479 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.479 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1950 3.0 3:0b1f384f:::smithi12313934-33:head [write 1889359~541737] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-01-16T00:13:43.479 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.922819+0000", 2020-01-16T00:13:43.479 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 45.422816244000003, 2020-01-16T00:13:43.479 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017502846999999998, 2020-01-16T00:13:43.479 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:43.480 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:43.480 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:43.480 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:43.480 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:43.480 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1950 2020-01-16T00:13:43.480 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.480 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:43.480 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.481 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:43.481 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922819+0000", 2020-01-16T00:13:43.481 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:43.481 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.481 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.481 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:43.481 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922819+0000", 2020-01-16T00:13:43.482 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:43.482 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.482 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.482 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:43.482 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922818+0000", 2020-01-16T00:13:43.482 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000146408 2020-01-16T00:13:43.482 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.482 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.483 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:43.483 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922964+0000", 2020-01-16T00:13:43.483 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4720000000000001e-06 2020-01-16T00:13:43.483 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.483 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.483 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:43.483 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922974+0000", 2020-01-16T00:13:43.483 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0720000000000001e-06 2020-01-16T00:13:43.484 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.484 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.484 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:43.484 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922978+0000", 2020-01-16T00:13:43.484 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6152000000000001e-05 2020-01-16T00:13:43.484 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.484 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.484 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.485 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.923024+0000", 2020-01-16T00:13:43.485 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00087368199999999997 2020-01-16T00:13:43.485 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.485 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.485 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:43.485 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.923897+0000", 2020-01-16T00:13:43.485 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011944435999999999 2020-01-16T00:13:43.486 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.486 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.486 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.486 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.935842+0000", 2020-01-16T00:13:43.486 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000144146 2020-01-16T00:13:43.486 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.486 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.486 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:43.487 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.935986+0000", 2020-01-16T00:13:43.487 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019775500000000001 2020-01-16T00:13:43.487 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.487 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.487 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:43.487 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.936184+0000", 2020-01-16T00:13:43.487 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002539342 2020-01-16T00:13:43.487 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.488 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.488 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:43.488 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938723+0000", 2020-01-16T00:13:43.488 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014193699999999999 2020-01-16T00:13:43.488 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.488 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.488 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:43.488 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938865+0000", 2020-01-16T00:13:43.489 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013829020000000001 2020-01-16T00:13:43.489 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.489 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.489 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:43.489 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.940248+0000", 2020-01-16T00:13:43.489 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2558000000000001e-05 2020-01-16T00:13:43.489 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.490 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.490 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:43.490 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.940291+0000", 2020-01-16T00:13:43.490 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1217999999999998e-05 2020-01-16T00:13:43.490 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.490 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.490 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:43.490 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.940322+0000", 2020-01-16T00:13:43.491 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017502846999999998 2020-01-16T00:13:43.491 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.491 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:43.491 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.491 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.491 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.491 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1951 3.0 3:0b1f384f:::smithi12313934-33:head [write 2955792~541902] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-01-16T00:13:43.491 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.929019+0000", 2020-01-16T00:13:43.492 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 45.416616093999998, 2020-01-16T00:13:43.492 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021625710999999999, 2020-01-16T00:13:43.492 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:43.492 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:43.492 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:43.492 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:43.492 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:43.493 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1951 2020-01-16T00:13:43.493 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.493 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:43.493 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.493 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:43.493 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929019+0000", 2020-01-16T00:13:43.493 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:43.493 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.494 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.494 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:43.494 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929019+0000", 2020-01-16T00:13:43.494 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999989 2020-01-16T00:13:43.494 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.494 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.494 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:43.494 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929008+0000", 2020-01-16T00:13:43.495 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000212513 2020-01-16T00:13:43.495 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.495 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.495 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:43.495 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929221+0000", 2020-01-16T00:13:43.495 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3229e-05 2020-01-16T00:13:43.495 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.496 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.496 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:43.496 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929234+0000", 2020-01-16T00:13:43.496 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9170000000000003e-06 2020-01-16T00:13:43.496 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.496 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.496 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:43.496 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929240+0000", 2020-01-16T00:13:43.497 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4045000000000002e-05 2020-01-16T00:13:43.497 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.497 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.497 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.497 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929294+0000", 2020-01-16T00:13:43.497 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9797999999999996e-05 2020-01-16T00:13:43.497 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.497 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.498 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:43.498 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929384+0000", 2020-01-16T00:13:43.498 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083450729999999997 2020-01-16T00:13:43.498 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.498 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.498 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.498 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.937729+0000", 2020-01-16T00:13:43.499 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043251699999999999 2020-01-16T00:13:43.499 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.499 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.499 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:43.499 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938161+0000", 2020-01-16T00:13:43.499 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095115489999999994 2020-01-16T00:13:43.499 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.499 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.500 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.500 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.947673+0000", 2020-01-16T00:13:43.500 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012395099999999999 2020-01-16T00:13:43.500 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.500 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.500 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:43.500 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.947797+0000", 2020-01-16T00:13:43.501 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00024849200000000001 2020-01-16T00:13:43.501 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.501 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.501 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:43.501 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.948045+0000", 2020-01-16T00:13:43.501 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0025081050000000001 2020-01-16T00:13:43.501 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.501 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.502 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:43.502 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950553+0000", 2020-01-16T00:13:43.502 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9760000000000003e-06 2020-01-16T00:13:43.502 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.502 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.502 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:43.502 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950558+0000", 2020-01-16T00:13:43.503 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3286999999999999e-05 2020-01-16T00:13:43.503 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.503 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.503 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:43.503 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950581+0000", 2020-01-16T00:13:43.503 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2759000000000003e-05 2020-01-16T00:13:43.503 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.503 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.504 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:43.504 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950614+0000", 2020-01-16T00:13:43.504 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0525999999999999e-05 2020-01-16T00:13:43.504 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.504 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.504 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:43.504 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950645+0000", 2020-01-16T00:13:43.504 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021625710999999999 2020-01-16T00:13:43.505 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.505 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:43.505 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.505 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.505 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.505 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1952 3.0 3:0b1f384f:::smithi12313934-33:head [setxattr _header (56) in=7b,truncate 3563084] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-01-16T00:13:43.505 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:13:43.506 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 45.416386756999998, 2020-01-16T00:13:43.506 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021534685000000001, 2020-01-16T00:13:43.506 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:43.506 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:43.506 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:43.506 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:43.506 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:43.506 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1952 2020-01-16T00:13:43.507 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.507 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:43.507 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.507 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:43.507 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:13:43.507 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:43.507 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.507 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.508 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:43.508 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:13:43.508 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:43.508 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.508 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.508 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:43.508 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:13:43.509 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6670000000000004e-06 2020-01-16T00:13:43.509 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.509 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.509 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:43.509 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929252+0000", 2020-01-16T00:13:43.509 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9380000000000002e-06 2020-01-16T00:13:43.509 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.509 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.510 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:43.510 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929256+0000", 2020-01-16T00:13:43.510 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.712e-06 2020-01-16T00:13:43.510 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.510 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.510 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:43.510 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929264+0000", 2020-01-16T00:13:43.511 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018030300000000001 2020-01-16T00:13:43.511 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.511 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.511 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.511 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929444+0000", 2020-01-16T00:13:43.511 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7562000000000002e-05 2020-01-16T00:13:43.511 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.511 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.512 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:43.512 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929502+0000", 2020-01-16T00:13:43.512 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0087297859999999998 2020-01-16T00:13:43.512 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.512 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.512 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.512 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938232+0000", 2020-01-16T00:13:43.512 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0452999999999996e-05 2020-01-16T00:13:43.513 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.513 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.513 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:43.513 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938322+0000", 2020-01-16T00:13:43.513 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011061728 2020-01-16T00:13:43.513 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.513 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.514 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.514 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.949384+0000", 2020-01-16T00:13:43.514 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000117828 2020-01-16T00:13:43.514 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.514 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.514 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:43.514 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.949502+0000", 2020-01-16T00:13:43.514 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022468499999999999 2020-01-16T00:13:43.515 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.515 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.515 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:43.515 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.949726+0000", 2020-01-16T00:13:43.515 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00083532099999999998 2020-01-16T00:13:43.515 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.515 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.516 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:43.516 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950562+0000", 2020-01-16T00:13:43.516 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013017700000000001 2020-01-16T00:13:43.516 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.516 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.516 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:43.516 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950692+0000", 2020-01-16T00:13:43.516 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9813000000000001e-05 2020-01-16T00:13:43.517 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.517 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.517 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:43.517 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950722+0000", 2020-01-16T00:13:43.517 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0592000000000001e-05 2020-01-16T00:13:43.517 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.517 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.517 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:43.518 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950752+0000", 2020-01-16T00:13:43.518 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0920000000000002e-05 2020-01-16T00:13:43.518 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.518 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.518 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:43.518 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950783+0000", 2020-01-16T00:13:43.518 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021534685000000001 2020-01-16T00:13:43.518 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.519 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:43.519 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.519 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.519 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.519 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1953 3.0 3:0b1f384f:::smithi12313934-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e32)", 2020-01-16T00:13:43.519 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:13:43.519 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 45.416353766, 2020-01-16T00:13:43.519 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021667537000000001, 2020-01-16T00:13:43.520 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:43.520 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:43.520 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:43.520 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:43.520 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:43.520 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1953 2020-01-16T00:13:43.520 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.521 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:43.521 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.521 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:43.521 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:13:43.521 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:43.521 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.521 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.521 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:43.522 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:13:43.522 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:43.522 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.522 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.522 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:43.522 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:13:43.522 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7030000000000002e-06 2020-01-16T00:13:43.522 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.523 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.523 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:43.523 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929284+0000", 2020-01-16T00:13:43.523 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0909999999999993e-06 2020-01-16T00:13:43.523 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.523 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.523 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:43.523 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929292+0000", 2020-01-16T00:13:43.524 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8359999999999999e-06 2020-01-16T00:13:43.524 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.524 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.524 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:43.524 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929297+0000", 2020-01-16T00:13:43.524 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023538199999999999 2020-01-16T00:13:43.524 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.524 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.525 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.525 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929532+0000", 2020-01-16T00:13:43.525 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1173000000000004e-05 2020-01-16T00:13:43.525 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.525 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.525 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:43.525 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929603+0000", 2020-01-16T00:13:43.526 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0087813509999999997 2020-01-16T00:13:43.526 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.526 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.526 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.526 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938384+0000", 2020-01-16T00:13:43.526 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9752000000000006e-05 2020-01-16T00:13:43.527 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.527 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.527 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:43.527 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938454+0000", 2020-01-16T00:13:43.527 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011707851999999999 2020-01-16T00:13:43.527 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.527 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.527 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.528 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950162+0000", 2020-01-16T00:13:43.528 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011060099999999999 2020-01-16T00:13:43.528 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.528 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.528 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:43.528 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950273+0000", 2020-01-16T00:13:43.528 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00053989700000000001 2020-01-16T00:13:43.529 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.529 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.529 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.529 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950813+0000", 2020-01-16T00:13:43.529 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8128999999999999e-05 2020-01-16T00:13:43.529 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.529 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.529 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:43.530 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950871+0000", 2020-01-16T00:13:43.530 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8242000000000001e-05 2020-01-16T00:13:43.530 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.530 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.530 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:43.530 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950949+0000", 2020-01-16T00:13:43.530 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021667537000000001 2020-01-16T00:13:43.530 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.531 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:43.531 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.531 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.531 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.531 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2555 3.0 3:0b70bb20:::smithi12313934-46:head [setxattr _header (56) in=7b,truncate 2991679] snapc 0=[] ondisk+write+known_if_redirected e37)", 2020-01-16T00:13:43.531 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.573432+0000", 2020-01-16T00:13:43.531 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 30.772203115, 2020-01-16T00:13:43.531 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019893926999999999, 2020-01-16T00:13:43.532 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:43.532 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:43.532 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:43.532 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:43.532 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:43.532 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2555 2020-01-16T00:13:43.532 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.532 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:43.532 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.533 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:43.533 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573432+0000", 2020-01-16T00:13:43.533 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:43.533 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.533 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.533 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:43.533 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573432+0000", 2020-01-16T00:13:43.534 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:43.534 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.534 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.534 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:43.534 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573432+0000", 2020-01-16T00:13:43.534 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4290000000000001e-06 2020-01-16T00:13:43.534 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.534 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.535 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:43.535 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573435+0000", 2020-01-16T00:13:43.535 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1300000000000001e-06 2020-01-16T00:13:43.535 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.535 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.535 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:43.535 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573438+0000", 2020-01-16T00:13:43.535 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9399999999999996e-06 2020-01-16T00:13:43.536 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.536 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.536 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:43.536 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573445+0000", 2020-01-16T00:13:43.536 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013947269999999999 2020-01-16T00:13:43.536 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.536 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.536 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.537 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.574840+0000", 2020-01-16T00:13:43.537 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003850058 2020-01-16T00:13:43.537 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.537 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.537 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:43.537 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578690+0000", 2020-01-16T00:13:43.537 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012896489000000001 2020-01-16T00:13:43.538 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.538 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.538 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.538 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591586+0000", 2020-01-16T00:13:43.538 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4757000000000001e-05 2020-01-16T00:13:43.538 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.538 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.538 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:43.539 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591681+0000", 2020-01-16T00:13:43.539 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000191535 2020-01-16T00:13:43.539 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.539 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.539 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:43.539 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591873+0000", 2020-01-16T00:13:43.539 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00069888899999999996 2020-01-16T00:13:43.539 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.540 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.540 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:43.540 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.592571+0000", 2020-01-16T00:13:43.540 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00029525900000000003 2020-01-16T00:13:43.540 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.540 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.540 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:43.540 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.592867+0000", 2020-01-16T00:13:43.541 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035243500000000001 2020-01-16T00:13:43.541 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.541 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.541 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:43.541 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593219+0000", 2020-01-16T00:13:43.541 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0954e-05 2020-01-16T00:13:43.541 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.542 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.542 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:43.542 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593270+0000", 2020-01-16T00:13:43.542 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5813000000000002e-05 2020-01-16T00:13:43.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.542 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.542 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:43.542 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593326+0000", 2020-01-16T00:13:43.543 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019893926999999999 2020-01-16T00:13:43.543 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.543 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:43.543 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.543 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.543 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.543 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2556 3.0 3:0b70bb20:::smithi12313934-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e37)", 2020-01-16T00:13:43.543 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.573472+0000", 2020-01-16T00:13:43.544 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 30.772163545000002, 2020-01-16T00:13:43.544 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020070068999999999, 2020-01-16T00:13:43.544 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:43.544 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:43.544 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:43.544 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:43.544 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:43.544 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2556 2020-01-16T00:13:43.545 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.545 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:43.545 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.545 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:43.545 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573472+0000", 2020-01-16T00:13:43.545 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:43.546 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.546 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.546 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:43.546 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573472+0000", 2020-01-16T00:13:43.546 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:43.546 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.546 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.547 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:43.547 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573471+0000", 2020-01-16T00:13:43.547 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4449999999999999e-06 2020-01-16T00:13:43.547 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.547 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.547 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:43.547 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573479+0000", 2020-01-16T00:13:43.547 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9789999999999998e-06 2020-01-16T00:13:43.548 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.548 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.548 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:43.548 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573485+0000", 2020-01-16T00:13:43.548 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3270000000000001e-06 2020-01-16T00:13:43.548 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.548 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.548 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:43.549 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573488+0000", 2020-01-16T00:13:43.549 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052586849999999999 2020-01-16T00:13:43.549 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.549 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.549 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.549 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578747+0000", 2020-01-16T00:13:43.549 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5667000000000007e-05 2020-01-16T00:13:43.550 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.550 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.550 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:43.550 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578812+0000", 2020-01-16T00:13:43.550 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013599539000000001 2020-01-16T00:13:43.550 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.550 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.550 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.551 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.592412+0000", 2020-01-16T00:13:43.551 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6018000000000001e-05 2020-01-16T00:13:43.551 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.551 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.551 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:43.551 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.592508+0000", 2020-01-16T00:13:43.551 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00084582199999999998 2020-01-16T00:13:43.551 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.552 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.552 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.552 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593354+0000", 2020-01-16T00:13:43.552 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4984999999999999e-05 2020-01-16T00:13:43.552 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.552 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.552 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:43.553 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593449+0000", 2020-01-16T00:13:43.553 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3104999999999997e-05 2020-01-16T00:13:43.553 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.553 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.553 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:43.553 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593542+0000", 2020-01-16T00:13:43.553 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020070068999999999 2020-01-16T00:13:43.553 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.554 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:43.554 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.554 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.554 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.554 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3211 3.0 3:0b70bb20:::smithi12313934-46:head [setxattr _header (56) in=7b,truncate 2571537] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:13:43.554 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.581146+0000", 2020-01-16T00:13:43.554 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 13.764488975000001, 2020-01-16T00:13:43.554 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024318514999999999, 2020-01-16T00:13:43.555 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:43.555 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:43.555 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:43.555 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:43.555 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:43.555 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3211 2020-01-16T00:13:43.555 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.555 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:43.556 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.556 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:43.556 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581146+0000", 2020-01-16T00:13:43.556 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:43.556 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.556 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.556 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:43.556 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581146+0000", 2020-01-16T00:13:43.557 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:43.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.557 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.557 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:43.557 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581145+0000", 2020-01-16T00:13:43.557 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2990000000000002e-06 2020-01-16T00:13:43.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.557 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.558 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:43.558 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581151+0000", 2020-01-16T00:13:43.558 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.041e-06 2020-01-16T00:13:43.558 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.558 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.558 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:43.558 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581159+0000", 2020-01-16T00:13:43.558 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5859999999999998e-06 2020-01-16T00:13:43.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.559 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.559 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:43.559 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581163+0000", 2020-01-16T00:13:43.559 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020769790000000001 2020-01-16T00:13:43.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.559 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.559 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.560 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.583240+0000", 2020-01-16T00:13:43.560 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012019579999999999 2020-01-16T00:13:43.560 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.560 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.560 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:43.560 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584442+0000", 2020-01-16T00:13:43.560 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018359812 2020-01-16T00:13:43.561 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.561 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.561 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.561 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.602802+0000", 2020-01-16T00:13:43.561 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000133072 2020-01-16T00:13:43.561 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.561 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.562 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:43.562 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.602935+0000", 2020-01-16T00:13:43.562 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019007200000000001 2020-01-16T00:13:43.562 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.562 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.562 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:43.562 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.603125+0000", 2020-01-16T00:13:43.562 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001261751 2020-01-16T00:13:43.563 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.563 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.563 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:43.563 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.604387+0000", 2020-01-16T00:13:43.563 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000304987 2020-01-16T00:13:43.563 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.563 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.563 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:43.564 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.604692+0000", 2020-01-16T00:13:43.564 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00063369299999999995 2020-01-16T00:13:43.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.564 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.564 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:43.564 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605325+0000", 2020-01-16T00:13:43.564 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1470000000000005e-05 2020-01-16T00:13:43.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.565 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.565 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:43.565 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605397+0000", 2020-01-16T00:13:43.565 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7784e-05 2020-01-16T00:13:43.565 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.565 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.565 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:43.566 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605465+0000", 2020-01-16T00:13:43.566 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024318514999999999 2020-01-16T00:13:43.566 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.566 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:43.566 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.566 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.566 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.566 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3212 3.0 3:0b70bb20:::smithi12313934-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e45)", 2020-01-16T00:13:43.567 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.581171+0000", 2020-01-16T00:13:43.567 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 13.764464276, 2020-01-16T00:13:43.567 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024540131, 2020-01-16T00:13:43.567 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:43.567 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:43.567 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:43.567 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:43.567 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:43.568 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3212 2020-01-16T00:13:43.568 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.568 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:43.568 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.568 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:43.568 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581171+0000", 2020-01-16T00:13:43.568 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:43.568 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.569 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.569 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:43.569 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581171+0000", 2020-01-16T00:13:43.569 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:43.569 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.569 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.569 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:43.570 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581170+0000", 2020-01-16T00:13:43.570 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5259999999999996e-06 2020-01-16T00:13:43.570 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.570 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.570 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:43.570 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581178+0000", 2020-01-16T00:13:43.570 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.469e-06 2020-01-16T00:13:43.570 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.571 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.571 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:43.571 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581185+0000", 2020-01-16T00:13:43.571 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4460000000000003e-06 2020-01-16T00:13:43.571 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.571 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.571 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:43.572 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581190+0000", 2020-01-16T00:13:43.572 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003286696 2020-01-16T00:13:43.572 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.572 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.572 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.572 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584476+0000", 2020-01-16T00:13:43.572 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9681e-05 2020-01-16T00:13:43.572 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.573 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.573 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:43.573 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584536+0000", 2020-01-16T00:13:43.573 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019075267 2020-01-16T00:13:43.573 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.573 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.573 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.573 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.603611+0000", 2020-01-16T00:13:43.574 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5981000000000001e-05 2020-01-16T00:13:43.574 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.574 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.574 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:43.574 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.603707+0000", 2020-01-16T00:13:43.574 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0017922649999999999 2020-01-16T00:13:43.574 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.575 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.575 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.575 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605500+0000", 2020-01-16T00:13:43.575 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5396999999999995e-05 2020-01-16T00:13:43.575 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.575 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.575 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:43.575 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605595+0000", 2020-01-16T00:13:43.576 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000116038 2020-01-16T00:13:43.576 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.576 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.576 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:43.576 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605711+0000", 2020-01-16T00:13:43.576 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024540131 2020-01-16T00:13:43.576 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.576 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:43.577 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.577 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.577 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.577 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3435 3.0 3:0b70bb20:::smithi12313934-46:head [setxattr _header (56) in=7b,truncate 3075747] snapc 0=[] ondisk+write+known_if_redirected e47)", 2020-01-16T00:13:43.577 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:34.388891+0000", 2020-01-16T00:13:43.577 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 8.9567442600000007, 2020-01-16T00:13:43.577 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016440475999999999, 2020-01-16T00:13:43.578 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:43.578 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:43.578 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:43.578 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:43.578 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:43.578 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3435 2020-01-16T00:13:43.578 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.578 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:43.579 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.579 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:43.579 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388891+0000", 2020-01-16T00:13:43.579 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:43.579 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.579 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.579 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:43.580 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388891+0000", 2020-01-16T00:13:43.580 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:43.580 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.580 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.580 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:43.580 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388890+0000", 2020-01-16T00:13:43.580 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.647e-06 2020-01-16T00:13:43.580 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.581 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.581 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:43.581 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388894+0000", 2020-01-16T00:13:43.581 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2839999999999997e-06 2020-01-16T00:13:43.581 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.581 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.581 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:43.582 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388899+0000", 2020-01-16T00:13:43.582 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3149999999999999e-06 2020-01-16T00:13:43.582 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.582 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.582 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:43.582 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388901+0000", 2020-01-16T00:13:43.582 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001768543 2020-01-16T00:13:43.582 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.583 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.583 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.583 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.390670+0000", 2020-01-16T00:13:43.583 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0034533680000000001 2020-01-16T00:13:43.583 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.583 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.583 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:43.583 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.394123+0000", 2020-01-16T00:13:43.584 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010189962 2020-01-16T00:13:43.584 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.584 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.584 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.584 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404313+0000", 2020-01-16T00:13:43.584 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000108646 2020-01-16T00:13:43.584 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.584 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.585 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:43.585 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404422+0000", 2020-01-16T00:13:43.585 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119133 2020-01-16T00:13:43.585 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.585 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.585 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:43.585 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404541+0000", 2020-01-16T00:13:43.585 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000433764 2020-01-16T00:13:43.586 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.586 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.586 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:43.586 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404975+0000", 2020-01-16T00:13:43.586 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000266013 2020-01-16T00:13:43.586 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.586 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.586 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:43.587 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405241+0000", 2020-01-16T00:13:43.587 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5187000000000001e-05 2020-01-16T00:13:43.587 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.587 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.587 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:43.587 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405266+0000", 2020-01-16T00:13:43.587 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0889000000000002e-05 2020-01-16T00:13:43.588 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.588 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.588 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:43.588 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405297+0000", 2020-01-16T00:13:43.588 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4354999999999999e-05 2020-01-16T00:13:43.588 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.588 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.588 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:43.589 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405331+0000", 2020-01-16T00:13:43.589 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016440475999999999 2020-01-16T00:13:43.589 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.589 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:43.589 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.589 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.589 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.589 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3436 3.0 3:0b70bb20:::smithi12313934-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e47)", 2020-01-16T00:13:43.590 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:34.388920+0000", 2020-01-16T00:13:43.590 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 8.9567154710000008, 2020-01-16T00:13:43.590 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016552641, 2020-01-16T00:13:43.590 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:43.590 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:43.590 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:43.590 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:43.590 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:43.591 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3436 2020-01-16T00:13:43.591 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.591 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:43.591 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.591 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:43.591 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388920+0000", 2020-01-16T00:13:43.591 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:43.591 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.592 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.592 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:43.592 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388920+0000", 2020-01-16T00:13:43.592 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:43.592 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.592 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.592 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:43.592 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388919+0000", 2020-01-16T00:13:43.593 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.181e-06 2020-01-16T00:13:43.593 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.593 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.593 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:43.593 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388921+0000", 2020-01-16T00:13:43.593 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9940000000000003e-06 2020-01-16T00:13:43.593 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.594 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.594 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:43.594 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388928+0000", 2020-01-16T00:13:43.594 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3549999999999999e-06 2020-01-16T00:13:43.594 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.594 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.594 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:43.594 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388931+0000", 2020-01-16T00:13:43.595 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.005242483 2020-01-16T00:13:43.595 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.595 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.595 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.595 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.394173+0000", 2020-01-16T00:13:43.595 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7435999999999995e-05 2020-01-16T00:13:43.595 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.596 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.596 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:43.596 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.394251+0000", 2020-01-16T00:13:43.596 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010593286 2020-01-16T00:13:43.596 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.596 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.596 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.596 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404844+0000", 2020-01-16T00:13:43.597 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0620999999999999e-05 2020-01-16T00:13:43.597 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.597 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.597 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:43.597 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404925+0000", 2020-01-16T00:13:43.597 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00042800899999999997 2020-01-16T00:13:43.597 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.597 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.598 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:43.598 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405353+0000", 2020-01-16T00:13:43.598 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2250000000000003e-05 2020-01-16T00:13:43.598 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.598 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.598 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:43.598 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405405+0000", 2020-01-16T00:13:43.599 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7411999999999998e-05 2020-01-16T00:13:43.599 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:43.599 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:43.599 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:43.599 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405472+0000", 2020-01-16T00:13:43.599 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016552641 2020-01-16T00:13:43.599 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.599 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:43.600 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.600 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:43.600 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:43.600 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:43.602 INFO:tasks.rados.rados.0.smithi123.stdout:1426: finishing write tid 1 to smithi12313934-40 2020-01-16T00:13:43.602 INFO:tasks.rados.rados.0.smithi123.stdout:1426: finishing write tid 2 to smithi12313934-40 2020-01-16T00:13:43.602 INFO:tasks.rados.rados.0.smithi123.stdout:1428: expect (ObjNum 451 snap 0 seq_num 451) 2020-01-16T00:13:43.603 INFO:tasks.rados.rados.0.smithi123.stdout:1426: finishing write tid 3 to smithi12313934-40 2020-01-16T00:13:43.603 INFO:tasks.rados.rados.0.smithi123.stdout:1426: finishing write tid 5 to smithi12313934-40 2020-01-16T00:13:43.603 INFO:tasks.rados.rados.0.smithi123.stdout:1426: finishing write tid 6 to smithi12313934-40 2020-01-16T00:13:43.603 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 40 v 223 (ObjNum 467 snap 0 seq_num 467) dirty exists 2020-01-16T00:13:43.603 INFO:tasks.rados.rados.0.smithi123.stdout:1426: left oid 40 (ObjNum 467 snap 0 seq_num 467) 2020-01-16T00:13:43.603 INFO:tasks.rados.rados.0.smithi123.stdout:1429: finishing write tid 1 to smithi12313934-12 2020-01-16T00:13:43.603 INFO:tasks.rados.rados.0.smithi123.stdout:1429: finishing write tid 2 to smithi12313934-12 2020-01-16T00:13:43.603 INFO:tasks.rados.rados.0.smithi123.stdout:1429: finishing write tid 3 to smithi12313934-12 2020-01-16T00:13:43.604 INFO:tasks.rados.rados.0.smithi123.stdout:1429: finishing write tid 5 to smithi12313934-12 2020-01-16T00:13:43.604 INFO:tasks.rados.rados.0.smithi123.stdout:1429: finishing write tid 6 to smithi12313934-12 2020-01-16T00:13:43.604 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 12 v 135 (ObjNum 468 snap 0 seq_num 468) dirty exists 2020-01-16T00:13:43.604 INFO:tasks.rados.rados.0.smithi123.stdout:1429: left oid 12 (ObjNum 468 snap 0 seq_num 468) 2020-01-16T00:13:43.604 INFO:tasks.rados.rados.0.smithi123.stdout:1430: expect (ObjNum 435 snap 0 seq_num 435) 2020-01-16T00:13:43.604 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-01-16T00:13:43.605 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:13:43.605 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:13:43.605 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-01-16T00:13:43.605 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-01-16T00:13:43.606 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-01-16T00:13:43.606 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-01-16T00:13:43.606 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-01-16T00:13:43.606 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENXIO: problem getting command descriptions from osd.5 2020-01-16T00:13:43.606 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: problem getting command descriptions from osd.5 2020-01-16T00:13:43.606 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-01-16T00:13:43.606 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-01-16T00:13:43.606 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-01-16T00:13:43.607 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-01-16T00:13:43.607 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:43.596+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:43.637 INFO:tasks.rados.rados.0.smithi123.stdout:1431: seq_num 469 ranges {682483=789314,1944996=791497,3196560=554574} 2020-01-16T00:13:43.653 INFO:tasks.rados.rados.0.smithi123.stdout:1431: writing smithi12313934-49 from 682483 to 1471797 tid 1 2020-01-16T00:13:43.661 INFO:tasks.rados.rados.0.smithi123.stdout:1431: writing smithi12313934-49 from 1944996 to 2736493 tid 2 2020-01-16T00:13:43.668 INFO:tasks.rados.rados.0.smithi123.stdout:1431: writing smithi12313934-49 from 3196560 to 3751134 tid 3 2020-01-16T00:13:43.668 INFO:tasks.rados.rados.0.smithi123.stdout:1421: done (10 left) 2020-01-16T00:13:43.668 INFO:tasks.rados.rados.0.smithi123.stdout:1422: done (9 left) 2020-01-16T00:13:43.669 INFO:tasks.rados.rados.0.smithi123.stdout:1423: done (8 left) 2020-01-16T00:13:43.669 INFO:tasks.rados.rados.0.smithi123.stdout:1424: done (7 left) 2020-01-16T00:13:43.669 INFO:tasks.rados.rados.0.smithi123.stdout:1425: done (6 left) 2020-01-16T00:13:43.669 INFO:tasks.rados.rados.0.smithi123.stdout:1426: done (5 left) 2020-01-16T00:13:43.669 INFO:tasks.rados.rados.0.smithi123.stdout:1427: done (4 left) 2020-01-16T00:13:43.669 INFO:tasks.rados.rados.0.smithi123.stdout:1428: done (3 left) 2020-01-16T00:13:43.669 INFO:tasks.rados.rados.0.smithi123.stdout:1429: done (2 left) 2020-01-16T00:13:43.669 INFO:tasks.rados.rados.0.smithi123.stdout:1430: done (1 left) 2020-01-16T00:13:43.670 INFO:tasks.rados.rados.0.smithi123.stdout:1432: write oid 12 current snap is 0 2020-01-16T00:13:43.670 INFO:tasks.rados.rados.0.smithi123.stdout:1432: seq_num 470 ranges {699290=464563,1603478=535155,2823983=765846} 2020-01-16T00:13:43.677 INFO:tasks.rados.rados.0.smithi123.stdout:1432: writing smithi12313934-12 from 699290 to 1163853 tid 1 2020-01-16T00:13:43.682 INFO:tasks.rados.rados.0.smithi123.stdout:1432: writing smithi12313934-12 from 1603478 to 2138633 tid 2 2020-01-16T00:13:43.692 INFO:tasks.rados.rados.0.smithi123.stdout:1432: writing smithi12313934-12 from 2823983 to 3589829 tid 3 2020-01-16T00:13:43.692 INFO:tasks.rados.rados.0.smithi123.stdout:1433: write oid 34 current snap is 0 2020-01-16T00:13:43.692 INFO:tasks.rados.rados.0.smithi123.stdout:1433: seq_num 471 ranges {716097=539812,1945607=795166,3535053=381044} 2020-01-16T00:13:43.699 INFO:tasks.rados.rados.0.smithi123.stdout:1433: writing smithi12313934-34 from 716097 to 1255909 tid 1 2020-01-16T00:13:43.702 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:43.698+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:43.709 INFO:tasks.rados.rados.0.smithi123.stdout:1433: writing smithi12313934-34 from 1945607 to 2740773 tid 2 2020-01-16T00:13:43.720 INFO:tasks.rados.rados.0.smithi123.stdout:1433: writing smithi12313934-34 from 3535053 to 3916097 tid 3 2020-01-16T00:13:43.720 INFO:tasks.rados.rados.0.smithi123.stdout:1434: read oid 36 snap -1 2020-01-16T00:13:43.720 INFO:tasks.rados.rados.0.smithi123.stdout:1434: expect (ObjNum 422 snap 0 seq_num 422) 2020-01-16T00:13:43.720 INFO:tasks.rados.rados.0.smithi123.stdout:1435: read oid 22 snap -1 2020-01-16T00:13:43.720 INFO:tasks.rados.rados.0.smithi123.stdout:1435: expect (ObjNum 393 snap 0 seq_num 393) 2020-01-16T00:13:43.720 INFO:tasks.rados.rados.0.smithi123.stdout:1436: delete oid 10 current snap is 0 2020-01-16T00:13:43.721 INFO:tasks.rados.rados.0.smithi123.stdout:1431: finishing write tid 1 to smithi12313934-49 2020-01-16T00:13:43.721 INFO:tasks.rados.rados.0.smithi123.stdout:1431: finishing write tid 2 to smithi12313934-49 2020-01-16T00:13:43.721 INFO:tasks.rados.rados.0.smithi123.stdout:1431: finishing write tid 3 to smithi12313934-49 2020-01-16T00:13:43.721 INFO:tasks.rados.rados.0.smithi123.stdout:1431: finishing write tid 5 to smithi12313934-49 2020-01-16T00:13:43.721 INFO:tasks.rados.rados.0.smithi123.stdout:1431: finishing write tid 6 to smithi12313934-49 2020-01-16T00:13:43.721 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 49 v 149 (ObjNum 469 snap 0 seq_num 469) dirty exists 2020-01-16T00:13:43.721 INFO:tasks.rados.rados.0.smithi123.stdout:1431: left oid 49 (ObjNum 469 snap 0 seq_num 469) 2020-01-16T00:13:43.721 INFO:tasks.rados.rados.0.smithi123.stdout:1432: finishing write tid 1 to smithi12313934-12 2020-01-16T00:13:43.722 INFO:tasks.rados.rados.0.smithi123.stdout:1432: finishing write tid 2 to smithi12313934-12 2020-01-16T00:13:43.722 INFO:tasks.rados.rados.0.smithi123.stdout:1432: finishing write tid 3 to smithi12313934-12 2020-01-16T00:13:43.722 INFO:tasks.rados.rados.0.smithi123.stdout:1432: finishing write tid 5 to smithi12313934-12 2020-01-16T00:13:43.722 INFO:tasks.rados.rados.0.smithi123.stdout:1432: finishing write tid 6 to smithi12313934-12 2020-01-16T00:13:43.722 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 12 v 139 (ObjNum 470 snap 0 seq_num 470) dirty exists 2020-01-16T00:13:43.722 INFO:tasks.rados.rados.0.smithi123.stdout:1432: left oid 12 (ObjNum 470 snap 0 seq_num 470) 2020-01-16T00:13:43.722 INFO:tasks.rados.rados.0.smithi123.stdout:1433: finishing write tid 1 to smithi12313934-34 2020-01-16T00:13:43.723 INFO:tasks.rados.rados.0.smithi123.stdout:1433: finishing write tid 2 to smithi12313934-34 2020-01-16T00:13:43.724 INFO:tasks.rados.rados.0.smithi123.stdout:1431: done (5 left) 2020-01-16T00:13:43.724 INFO:tasks.rados.rados.0.smithi123.stdout:1432: done (4 left) 2020-01-16T00:13:43.724 INFO:tasks.rados.rados.0.smithi123.stdout:1436: done (3 left) 2020-01-16T00:13:43.724 INFO:tasks.rados.rados.0.smithi123.stdout:1437: write (excl) oid 31 current snap is 0 2020-01-16T00:13:43.724 INFO:tasks.rados.rados.0.smithi123.stdout:1437: seq_num 473 ranges {749711=406663,1778924=682482,3189899=716015} 2020-01-16T00:13:43.739 INFO:tasks.rados.rados.0.smithi123.stdout:1437: writing smithi12313934-31 from 749711 to 1156374 tid 1 2020-01-16T00:13:43.746 INFO:tasks.rados.rados.0.smithi123.stdout:1437: writing smithi12313934-31 from 1778924 to 2461406 tid 2 2020-01-16T00:13:43.755 INFO:tasks.rados.rados.0.smithi123.stdout:1437: writing smithi12313934-31 from 3189899 to 3905914 tid 3 2020-01-16T00:13:43.755 INFO:tasks.rados.rados.0.smithi123.stdout:1438: write oid 30 current snap is 0 2020-01-16T00:13:43.755 INFO:tasks.rados.rados.0.smithi123.stdout:1438: seq_num 474 ranges {766518=481912,1837406=542493,2933675=527287,3876584=89934} 2020-01-16T00:13:43.761 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:43.761 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:43.761 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:13:43.761 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:43.762 INFO:tasks.rados.rados.0.smithi123.stdout:1438: writing smithi12313934-30 from 766518 to 1248430 tid 1 2020-01-16T00:13:43.768 INFO:tasks.rados.rados.0.smithi123.stdout:1438: writing smithi12313934-30 from 1837406 to 2379899 tid 2 2020-01-16T00:13:43.771 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-01-16T00:13:43.774 INFO:tasks.rados.rados.0.smithi123.stdout:1438: writing smithi12313934-30 from 2933675 to 3460962 tid 3 2020-01-16T00:13:43.777 INFO:tasks.rados.rados.0.smithi123.stdout:1438: writing smithi12313934-30 from 3876584 to 3966518 tid 4 2020-01-16T00:13:43.777 INFO:tasks.rados.rados.0.smithi123.stdout:1439: delete oid 20 current snap is 0 2020-01-16T00:13:43.777 INFO:tasks.rados.rados.0.smithi123.stdout:1433: finishing write tid 3 to smithi12313934-34 2020-01-16T00:13:43.777 INFO:tasks.rados.rados.0.smithi123.stdout:1433: finishing write tid 5 to smithi12313934-34 2020-01-16T00:13:43.777 INFO:tasks.rados.rados.0.smithi123.stdout:1433: finishing write tid 6 to smithi12313934-34 2020-01-16T00:13:43.778 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 34 v 58 (ObjNum 471 snap 0 seq_num 471) dirty exists 2020-01-16T00:13:43.778 INFO:tasks.rados.rados.0.smithi123.stdout:1433: left oid 34 (ObjNum 471 snap 0 seq_num 471) 2020-01-16T00:13:43.778 INFO:tasks.rados.rados.0.smithi123.stdout:1434: expect (ObjNum 422 snap 0 seq_num 422) 2020-01-16T00:13:43.803 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:43.798+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:43.868 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:43.868 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:43.869 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:13:43.869 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:13:43.869 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:43.877 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-01-16T00:13:43.904 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:43.900+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:43.962 INFO:tasks.rados.rados.0.smithi123.stdout:1435: expect (ObjNum 393 snap 0 seq_num 393) 2020-01-16T00:13:44.005 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:44.002+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:44.032 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:44.032 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:13:44.032 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:13:44.033 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:13:44.033 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.033 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.5.13:42 2.0 e24/19 2:313458df:::smithi12313934-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 24'18, mlcod=24'16)", 2020-01-16T00:13:44.033 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.940683+0000", 2020-01-16T00:13:44.034 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 66.087617864999999, 2020-01-16T00:13:44.034 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0097026960000000002, 2020-01-16T00:13:44.034 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:44.034 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:44.034 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:44.035 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.035 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:44.035 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.940683+0000", 2020-01-16T00:13:44.035 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:44.036 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.036 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.036 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:44.036 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.940683+0000", 2020-01-16T00:13:44.036 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:44.037 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.037 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.037 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:44.037 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.940680+0000", 2020-01-16T00:13:44.038 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003126775 2020-01-16T00:13:44.038 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.038 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.038 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:44.039 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.943807+0000", 2020-01-16T00:13:44.039 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5426999999999999e-05 2020-01-16T00:13:44.039 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.039 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.039 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:44.040 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.943833+0000", 2020-01-16T00:13:44.040 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7409999999999998e-06 2020-01-16T00:13:44.040 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.040 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.040 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:44.041 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.943841+0000", 2020-01-16T00:13:44.041 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011645399999999999 2020-01-16T00:13:44.041 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.041 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.042 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.042 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.943958+0000", 2020-01-16T00:13:44.042 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7291000000000003e-05 2020-01-16T00:13:44.042 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.043 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.043 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:44.043 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.944025+0000", 2020-01-16T00:13:44.043 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063108449999999998 2020-01-16T00:13:44.043 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.044 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.044 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:44.044 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.950336+0000", 2020-01-16T00:13:44.044 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9874999999999999e-05 2020-01-16T00:13:44.045 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.045 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.045 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:44.045 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.950386+0000", 2020-01-16T00:13:44.046 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0097026960000000002 2020-01-16T00:13:44.046 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.046 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:44.046 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.047 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.047 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.047 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1649 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~2084068,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:13:44.047 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847683+0000", 2020-01-16T00:13:44.047 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 52.180618047999999, 2020-01-16T00:13:44.048 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010654691000000001, 2020-01-16T00:13:44.048 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:44.048 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:44.048 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:44.049 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:44.049 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:44.049 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1649 2020-01-16T00:13:44.049 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.050 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:44.050 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.050 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:44.050 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847683+0000", 2020-01-16T00:13:44.051 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:44.051 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.051 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.051 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:44.052 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847683+0000", 2020-01-16T00:13:44.052 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:44.052 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.052 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.053 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:44.053 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847682+0000", 2020-01-16T00:13:44.053 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8569999999999997e-06 2020-01-16T00:13:44.053 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.054 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.054 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:44.055 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847689+0000", 2020-01-16T00:13:44.055 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8090000000000004e-06 2020-01-16T00:13:44.055 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.057 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.057 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:44.057 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847696+0000", 2020-01-16T00:13:44.057 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2690000000000001e-06 2020-01-16T00:13:44.058 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.058 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.058 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:44.058 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847700+0000", 2020-01-16T00:13:44.059 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8189999999999993e-05 2020-01-16T00:13:44.059 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.059 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.059 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.059 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847798+0000", 2020-01-16T00:13:44.060 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7129999999999999e-06 2020-01-16T00:13:44.060 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.060 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.060 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:13:44.061 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847802+0000", 2020-01-16T00:13:44.061 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083913049999999999 2020-01-16T00:13:44.061 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.061 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.062 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.062 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.856193+0000", 2020-01-16T00:13:44.062 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034400300000000001 2020-01-16T00:13:44.062 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.062 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.063 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:44.063 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.856537+0000", 2020-01-16T00:13:44.063 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018006820000000001 2020-01-16T00:13:44.063 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.064 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.064 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:44.064 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858338+0000", 2020-01-16T00:13:44.064 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010654691000000001 2020-01-16T00:13:44.064 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.065 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:44.065 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.065 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.065 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.066 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1650 3.1 3:8de1d8a3:::smithi12313934-13:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:13:44.066 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847717+0000", 2020-01-16T00:13:44.066 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 52.180584095, 2020-01-16T00:13:44.066 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010913658, 2020-01-16T00:13:44.066 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:44.068 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:44.069 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:44.069 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:44.069 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:44.069 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1650 2020-01-16T00:13:44.070 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.070 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:44.070 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.070 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:44.070 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847717+0000", 2020-01-16T00:13:44.071 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:44.071 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.071 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.071 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:44.072 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847717+0000", 2020-01-16T00:13:44.072 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:44.072 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.072 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.072 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:44.073 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847716+0000", 2020-01-16T00:13:44.073 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.433e-06 2020-01-16T00:13:44.073 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.073 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.074 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:44.074 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847720+0000", 2020-01-16T00:13:44.074 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1800000000000004e-06 2020-01-16T00:13:44.074 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.075 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.075 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:44.075 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847725+0000", 2020-01-16T00:13:44.075 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6689999999999997e-06 2020-01-16T00:13:44.076 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.076 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.076 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:44.076 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847730+0000", 2020-01-16T00:13:44.076 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000102468 2020-01-16T00:13:44.077 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.077 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.077 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.077 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847832+0000", 2020-01-16T00:13:44.078 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3999999999999997e-06 2020-01-16T00:13:44.078 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.078 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.078 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:13:44.079 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847838+0000", 2020-01-16T00:13:44.079 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010519395000000001 2020-01-16T00:13:44.079 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.079 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.080 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.080 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858358+0000", 2020-01-16T00:13:44.080 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7600000000000002e-05 2020-01-16T00:13:44.080 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.080 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.081 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:44.081 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858445+0000", 2020-01-16T00:13:44.081 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018515900000000001 2020-01-16T00:13:44.081 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.082 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.082 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:44.082 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858631+0000", 2020-01-16T00:13:44.082 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010913658 2020-01-16T00:13:44.082 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.082 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:44.083 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.083 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.083 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.083 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1651 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~2084068,checksum] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:13:44.083 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847752+0000", 2020-01-16T00:13:44.084 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 52.180549479, 2020-01-16T00:13:44.084 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011734249, 2020-01-16T00:13:44.084 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:44.086 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:44.086 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:44.086 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:44.086 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:44.087 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1651 2020-01-16T00:13:44.087 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.087 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:44.087 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.087 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:44.087 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847752+0000", 2020-01-16T00:13:44.088 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:44.088 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.088 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.088 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:44.088 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847752+0000", 2020-01-16T00:13:44.089 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:44.089 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.089 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.089 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:44.090 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847751+0000", 2020-01-16T00:13:44.090 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1980000000000001e-06 2020-01-16T00:13:44.090 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.090 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.090 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:44.090 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847754+0000", 2020-01-16T00:13:44.091 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6679999999999999e-06 2020-01-16T00:13:44.091 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.091 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.091 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:44.091 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847758+0000", 2020-01-16T00:13:44.091 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2500000000000001e-06 2020-01-16T00:13:44.091 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.091 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.092 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:44.092 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847760+0000", 2020-01-16T00:13:44.092 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7614000000000004e-05 2020-01-16T00:13:44.092 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.092 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.092 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.092 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847858+0000", 2020-01-16T00:13:44.092 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7950000000000004e-06 2020-01-16T00:13:44.093 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.093 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.093 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:13:44.093 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847864+0000", 2020-01-16T00:13:44.093 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010790345 2020-01-16T00:13:44.093 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.093 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.094 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.094 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858654+0000", 2020-01-16T00:13:44.094 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000141466 2020-01-16T00:13:44.094 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.094 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.094 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:44.094 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858795+0000", 2020-01-16T00:13:44.094 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00069037899999999997 2020-01-16T00:13:44.095 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.095 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.095 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:44.095 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.859486+0000", 2020-01-16T00:13:44.095 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011734249 2020-01-16T00:13:44.095 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.095 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:44.095 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.096 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.096 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.096 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:55 2.1 e30/19 2:91f85463:::smithi12313934-24 low_tier:head v 30'56, mlcod=29'54)", 2020-01-16T00:13:44.096 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.738134+0000", 2020-01-16T00:13:44.096 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 51.290167193000002, 2020-01-16T00:13:44.096 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0086509570000000008, 2020-01-16T00:13:44.096 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:44.096 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:44.097 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:44.097 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.097 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:44.097 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738134+0000", 2020-01-16T00:13:44.097 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:44.097 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.097 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.098 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:44.098 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738134+0000", 2020-01-16T00:13:44.098 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:44.098 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.098 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.098 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:44.098 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738134+0000", 2020-01-16T00:13:44.098 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00053712599999999997 2020-01-16T00:13:44.099 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.099 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.099 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:44.099 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738671+0000", 2020-01-16T00:13:44.099 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4059999999999999e-05 2020-01-16T00:13:44.099 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.099 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.099 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:44.100 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738685+0000", 2020-01-16T00:13:44.100 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7680000000000008e-06 2020-01-16T00:13:44.100 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.100 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.100 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:44.100 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738692+0000", 2020-01-16T00:13:44.100 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1273999999999996e-05 2020-01-16T00:13:44.101 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.101 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.101 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.101 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738764+0000", 2020-01-16T00:13:44.101 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.6827999999999999e-05 2020-01-16T00:13:44.101 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.101 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.101 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:44.102 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.738781+0000", 2020-01-16T00:13:44.102 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0079370740000000006 2020-01-16T00:13:44.102 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.102 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.102 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:44.102 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746718+0000", 2020-01-16T00:13:44.102 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7181000000000006e-05 2020-01-16T00:13:44.102 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.103 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.103 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:44.103 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746785+0000", 2020-01-16T00:13:44.103 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0086509570000000008 2020-01-16T00:13:44.103 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.103 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:44.103 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.103 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.104 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.104 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2106 3.1 3:8de1d8a3:::smithi12313934-13:head [write 665610~797860,stat] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:13:44.104 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.631369+0000", 2020-01-16T00:13:44.104 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 42.396931766999998, 2020-01-16T00:13:44.104 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0091195330000000008, 2020-01-16T00:13:44.104 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:44.104 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:44.105 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:44.105 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:44.105 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:44.105 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2106 2020-01-16T00:13:44.105 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.105 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:44.105 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.105 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:44.106 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631369+0000", 2020-01-16T00:13:44.106 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:44.106 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.106 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.106 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:44.106 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631369+0000", 2020-01-16T00:13:44.106 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:44.106 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.107 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.107 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:44.107 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631367+0000", 2020-01-16T00:13:44.107 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00025714100000000001 2020-01-16T00:13:44.107 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.107 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.107 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:44.107 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631624+0000", 2020-01-16T00:13:44.108 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4946000000000001e-05 2020-01-16T00:13:44.108 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.108 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.109 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:44.109 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631639+0000", 2020-01-16T00:13:44.109 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7050000000000002e-06 2020-01-16T00:13:44.109 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.109 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.109 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:44.110 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631646+0000", 2020-01-16T00:13:44.110 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010938900000000001 2020-01-16T00:13:44.110 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.111 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.111 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.111 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631755+0000", 2020-01-16T00:13:44.111 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8848e-05 2020-01-16T00:13:44.112 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.112 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.112 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op must wait for map", 2020-01-16T00:13:44.112 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.631774+0000", 2020-01-16T00:13:44.113 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0026027730000000001 2020-01-16T00:13:44.113 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.113 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.113 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.113 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.634377+0000", 2020-01-16T00:13:44.114 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9282000000000003e-05 2020-01-16T00:13:44.114 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.114 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.114 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:44.115 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.634466+0000", 2020-01-16T00:13:44.115 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000127868 2020-01-16T00:13:44.115 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.115 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.116 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:13:44.116 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.634594+0000", 2020-01-16T00:13:44.116 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030776549999999999 2020-01-16T00:13:44.116 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.116 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.117 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:44.117 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.637672+0000", 2020-01-16T00:13:44.117 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001634195 2020-01-16T00:13:44.117 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.117 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.118 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:44.118 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.639306+0000", 2020-01-16T00:13:44.118 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010810069999999999 2020-01-16T00:13:44.118 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.119 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.119 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:44.119 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640387+0000", 2020-01-16T00:13:44.120 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0543000000000001e-05 2020-01-16T00:13:44.120 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.120 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.120 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:44.120 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640428+0000", 2020-01-16T00:13:44.121 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1321000000000005e-05 2020-01-16T00:13:44.121 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.121 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.121 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:44.122 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640489+0000", 2020-01-16T00:13:44.122 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0091195330000000008 2020-01-16T00:13:44.122 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.122 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:44.122 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.123 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.123 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.123 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2107 3.1 3:8de1d8a3:::smithi12313934-13:head [write 1909329~458405] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:13:44.123 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.636786+0000", 2020-01-16T00:13:44.124 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 42.391514637, 2020-01-16T00:13:44.124 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093182809999999994, 2020-01-16T00:13:44.124 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:44.124 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:44.124 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:44.125 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:44.125 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:44.125 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2107 2020-01-16T00:13:44.125 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.126 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:44.126 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.126 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:44.126 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636786+0000", 2020-01-16T00:13:44.127 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:44.127 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.127 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.127 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:44.127 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636786+0000", 2020-01-16T00:13:44.128 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:44.128 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.128 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.128 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:44.129 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636784+0000", 2020-01-16T00:13:44.129 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014276299999999999 2020-01-16T00:13:44.129 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.129 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.130 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:44.130 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636927+0000", 2020-01-16T00:13:44.130 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5017999999999999e-05 2020-01-16T00:13:44.130 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.130 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.131 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:44.131 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636942+0000", 2020-01-16T00:13:44.131 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5589999999999996e-06 2020-01-16T00:13:44.131 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.132 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.132 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:44.132 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636947+0000", 2020-01-16T00:13:44.132 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00040279699999999999 2020-01-16T00:13:44.132 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.133 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.133 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.133 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.637350+0000", 2020-01-16T00:13:44.133 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000233506 2020-01-16T00:13:44.134 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.134 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.134 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:44.134 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.637584+0000", 2020-01-16T00:13:44.135 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029534660000000001 2020-01-16T00:13:44.135 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.135 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.135 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.135 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640537+0000", 2020-01-16T00:13:44.136 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012577700000000001 2020-01-16T00:13:44.136 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.136 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.136 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:44.136 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640663+0000", 2020-01-16T00:13:44.137 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000198593 2020-01-16T00:13:44.137 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.137 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.137 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:13:44.138 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640862+0000", 2020-01-16T00:13:44.138 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.005088197 2020-01-16T00:13:44.138 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.138 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.139 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:44.139 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645950+0000", 2020-01-16T00:13:44.139 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2568999999999999e-05 2020-01-16T00:13:44.139 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.139 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.140 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:44.140 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645962+0000", 2020-01-16T00:13:44.140 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3034999999999995e-05 2020-01-16T00:13:44.140 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.141 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.141 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:44.141 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646025+0000", 2020-01-16T00:13:44.141 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7762999999999997e-05 2020-01-16T00:13:44.141 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.142 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.142 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:44.142 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646063+0000", 2020-01-16T00:13:44.142 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1541000000000001e-05 2020-01-16T00:13:44.143 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.143 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.143 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:44.143 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646105+0000", 2020-01-16T00:13:44.143 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093182809999999994 2020-01-16T00:13:44.144 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.144 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:44.144 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.144 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.144 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.145 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2108 3.1 3:8de1d8a3:::smithi12313934-13:head [write 2816700~410210] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:13:44.145 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.641861+0000", 2020-01-16T00:13:44.145 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 42.386440288999999, 2020-01-16T00:13:44.145 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018545604, 2020-01-16T00:13:44.146 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:44.146 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:44.146 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:44.146 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:44.146 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:44.147 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2108 2020-01-16T00:13:44.147 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.147 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:44.147 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.147 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:44.148 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.641861+0000", 2020-01-16T00:13:44.148 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:44.148 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.148 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.148 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:44.148 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.641861+0000", 2020-01-16T00:13:44.148 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:44.148 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.149 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.149 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:44.149 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.641859+0000", 2020-01-16T00:13:44.149 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018466399999999999 2020-01-16T00:13:44.149 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.149 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.149 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:44.150 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642043+0000", 2020-01-16T00:13:44.150 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2999999999999999e-05 2020-01-16T00:13:44.150 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.150 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.150 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:44.150 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642056+0000", 2020-01-16T00:13:44.150 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0730000000000005e-06 2020-01-16T00:13:44.151 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.151 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.151 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:44.151 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642064+0000", 2020-01-16T00:13:44.151 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9527999999999997e-05 2020-01-16T00:13:44.151 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.151 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.152 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.152 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642114+0000", 2020-01-16T00:13:44.152 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036616050000000001 2020-01-16T00:13:44.152 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.152 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.152 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:44.152 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645776+0000", 2020-01-16T00:13:44.153 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012222663999999999 2020-01-16T00:13:44.153 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.153 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.154 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.154 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.657998+0000", 2020-01-16T00:13:44.155 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014280299999999999 2020-01-16T00:13:44.155 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.155 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.155 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:44.156 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.658141+0000", 2020-01-16T00:13:44.156 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0001102 2020-01-16T00:13:44.156 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.156 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.157 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:13:44.157 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.658251+0000", 2020-01-16T00:13:44.157 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020421219999999999 2020-01-16T00:13:44.158 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.158 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.158 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:44.158 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660293+0000", 2020-01-16T00:13:44.158 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7399999999999999e-06 2020-01-16T00:13:44.159 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.159 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.159 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:44.159 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660303+0000", 2020-01-16T00:13:44.160 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9873999999999998e-05 2020-01-16T00:13:44.160 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.160 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.160 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:44.160 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660343+0000", 2020-01-16T00:13:44.161 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4448e-05 2020-01-16T00:13:44.161 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.161 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.161 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:44.162 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660377+0000", 2020-01-16T00:13:44.162 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8915999999999999e-05 2020-01-16T00:13:44.162 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.162 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.162 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:44.163 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660406+0000", 2020-01-16T00:13:44.163 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018545604 2020-01-16T00:13:44.163 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.163 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:44.164 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.164 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.164 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.164 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2109 3.1 3:8de1d8a3:::smithi12313934-13:head [setxattr _header (56) in=7b,truncate 3865610] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:13:44.164 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.642075+0000", 2020-01-16T00:13:44.165 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 42.386225715999998, 2020-01-16T00:13:44.165 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018477655999999999, 2020-01-16T00:13:44.165 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:44.165 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:44.166 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:44.166 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:44.166 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:44.166 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2109 2020-01-16T00:13:44.166 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.167 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:44.167 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.167 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:44.167 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642075+0000", 2020-01-16T00:13:44.167 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:44.167 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.168 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.168 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:44.168 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642075+0000", 2020-01-16T00:13:44.168 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:44.168 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.169 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.169 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:44.169 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642074+0000", 2020-01-16T00:13:44.169 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4500000000000003e-06 2020-01-16T00:13:44.169 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.169 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.169 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:44.169 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642080+0000", 2020-01-16T00:13:44.170 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7229999999999998e-06 2020-01-16T00:13:44.170 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.170 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.170 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:44.170 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642087+0000", 2020-01-16T00:13:44.170 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2390000000000004e-06 2020-01-16T00:13:44.170 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.170 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.171 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:44.171 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642091+0000", 2020-01-16T00:13:44.171 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0037396550000000001 2020-01-16T00:13:44.171 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.171 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.171 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.171 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645830+0000", 2020-01-16T00:13:44.171 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2200000000000002e-05 2020-01-16T00:13:44.172 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.172 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.172 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:44.172 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645883+0000", 2020-01-16T00:13:44.172 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013432934000000001 2020-01-16T00:13:44.172 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.172 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.173 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.173 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659316+0000", 2020-01-16T00:13:44.173 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000140006 2020-01-16T00:13:44.173 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.173 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.173 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:44.173 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659456+0000", 2020-01-16T00:13:44.173 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000184873 2020-01-16T00:13:44.174 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.174 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.174 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:13:44.174 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659640+0000", 2020-01-16T00:13:44.174 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00066954999999999998 2020-01-16T00:13:44.174 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.174 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.175 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:44.175 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660310+0000", 2020-01-16T00:13:44.175 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014847699999999999 2020-01-16T00:13:44.175 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.175 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.175 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:44.175 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660459+0000", 2020-01-16T00:13:44.175 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6869e-05 2020-01-16T00:13:44.176 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.176 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.176 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:44.176 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660485+0000", 2020-01-16T00:13:44.176 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8478000000000001e-05 2020-01-16T00:13:44.176 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.176 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.176 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:44.177 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660514+0000", 2020-01-16T00:13:44.177 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9153999999999998e-05 2020-01-16T00:13:44.177 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.177 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.177 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:44.177 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660553+0000", 2020-01-16T00:13:44.177 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018477655999999999 2020-01-16T00:13:44.177 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.178 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:44.178 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.178 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.178 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.178 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2110 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-01-16T00:13:44.178 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.642112+0000", 2020-01-16T00:13:44.178 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 42.386189078000001, 2020-01-16T00:13:44.178 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018620198000000001, 2020-01-16T00:13:44.179 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:44.179 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:44.179 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:44.179 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:44.179 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:44.179 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2110 2020-01-16T00:13:44.179 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.179 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:44.180 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.180 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:44.180 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642112+0000", 2020-01-16T00:13:44.180 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:44.180 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.180 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.180 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:44.180 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642112+0000", 2020-01-16T00:13:44.181 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:44.181 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.181 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.181 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:44.181 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642111+0000", 2020-01-16T00:13:44.181 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4450000000000001e-06 2020-01-16T00:13:44.181 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.181 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.182 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:44.182 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642117+0000", 2020-01-16T00:13:44.182 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1567e-05 2020-01-16T00:13:44.182 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.182 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.182 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:44.182 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642129+0000", 2020-01-16T00:13:44.183 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.754e-06 2020-01-16T00:13:44.183 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.183 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.183 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:44.183 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642136+0000", 2020-01-16T00:13:44.183 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040239719999999998 2020-01-16T00:13:44.183 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.184 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.184 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.184 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646160+0000", 2020-01-16T00:13:44.184 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5600999999999998e-05 2020-01-16T00:13:44.184 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.184 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.184 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:44.184 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646215+0000", 2020-01-16T00:13:44.185 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013733006000000001 2020-01-16T00:13:44.185 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.185 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.185 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.185 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659948+0000", 2020-01-16T00:13:44.185 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000106277 2020-01-16T00:13:44.185 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.186 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.186 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:44.186 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660054+0000", 2020-01-16T00:13:44.186 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000524327 2020-01-16T00:13:44.186 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.186 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.186 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.186 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660579+0000", 2020-01-16T00:13:44.187 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9841999999999999e-05 2020-01-16T00:13:44.187 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.187 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.187 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:44.187 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660659+0000", 2020-01-16T00:13:44.187 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3572999999999995e-05 2020-01-16T00:13:44.188 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.188 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.188 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:44.188 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660732+0000", 2020-01-16T00:13:44.188 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018620198000000001 2020-01-16T00:13:44.188 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.188 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:44.188 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.189 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.189 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.189 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:70 2.1 e35/19 2:91f85463:::smithi12313934-24 low_tier:head v 35'66, mlcod=34'64)", 2020-01-16T00:13:44.189 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.986684+0000", 2020-01-16T00:13:44.189 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 38.041617223999999, 2020-01-16T00:13:44.189 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012287378999999999, 2020-01-16T00:13:44.189 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:44.190 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:44.190 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:44.190 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.190 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:44.190 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986684+0000", 2020-01-16T00:13:44.190 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:44.190 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.190 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.191 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:44.191 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986684+0000", 2020-01-16T00:13:44.191 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:44.191 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.191 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.191 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:44.192 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986683+0000", 2020-01-16T00:13:44.192 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012884179999999999 2020-01-16T00:13:44.192 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.192 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.192 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:44.192 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.987971+0000", 2020-01-16T00:13:44.192 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011558200000000001 2020-01-16T00:13:44.193 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.193 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.194 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:44.194 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988087+0000", 2020-01-16T00:13:44.194 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0472000000000001e-05 2020-01-16T00:13:44.194 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.194 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.194 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:44.194 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988137+0000", 2020-01-16T00:13:44.194 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000107484 2020-01-16T00:13:44.195 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.195 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.195 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.195 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988245+0000", 2020-01-16T00:13:44.195 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4409999999999998e-05 2020-01-16T00:13:44.195 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.195 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.196 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:44.196 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988279+0000", 2020-01-16T00:13:44.196 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010641609999999999 2020-01-16T00:13:44.196 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.196 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.196 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:44.196 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.998921+0000", 2020-01-16T00:13:44.196 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0278000000000002e-05 2020-01-16T00:13:44.197 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.197 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.197 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:44.197 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.998971+0000", 2020-01-16T00:13:44.197 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012287378999999999 2020-01-16T00:13:44.198 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.198 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:44.198 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.198 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.198 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.199 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:77 2.1 e36/19 2:91f85463:::smithi12313934-24 low_tier:head v 36'69, mlcod=35'67)", 2020-01-16T00:13:44.199 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.556730+0000", 2020-01-16T00:13:44.199 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 36.471570870000001, 2020-01-16T00:13:44.199 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093007760000000002, 2020-01-16T00:13:44.200 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:44.200 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:44.200 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:44.200 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.200 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:44.201 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556730+0000", 2020-01-16T00:13:44.201 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:44.201 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.201 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.202 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:44.202 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556730+0000", 2020-01-16T00:13:44.202 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:44.202 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.202 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.203 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:44.203 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556728+0000", 2020-01-16T00:13:44.203 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00056509599999999996 2020-01-16T00:13:44.204 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.204 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.204 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:44.204 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.557293+0000", 2020-01-16T00:13:44.204 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0225e-05 2020-01-16T00:13:44.204 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.204 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.205 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:44.205 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.557314+0000", 2020-01-16T00:13:44.205 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8989999999999996e-06 2020-01-16T00:13:44.205 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.205 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.205 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:44.206 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.557323+0000", 2020-01-16T00:13:44.206 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7923000000000002e-05 2020-01-16T00:13:44.206 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.206 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.206 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.207 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.557371+0000", 2020-01-16T00:13:44.207 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9963999999999999e-05 2020-01-16T00:13:44.207 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.207 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.207 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:44.207 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.557411+0000", 2020-01-16T00:13:44.208 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0085873000000000008 2020-01-16T00:13:44.208 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.208 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.208 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:44.208 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.565999+0000", 2020-01-16T00:13:44.208 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2326000000000002e-05 2020-01-16T00:13:44.209 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.210 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.210 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:44.210 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566031+0000", 2020-01-16T00:13:44.210 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093007760000000002 2020-01-16T00:13:44.210 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.210 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:44.211 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.211 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.211 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.211 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:109 2.0 e42/19 2:11fa79bc:::smithi12313934-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 42'67, mlcod=41'65)", 2020-01-16T00:13:44.211 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:21.841140+0000", 2020-01-16T00:13:44.211 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 22.187161155999998, 2020-01-16T00:13:44.211 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010439159999999999, 2020-01-16T00:13:44.212 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:44.212 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:44.212 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:44.212 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.212 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:44.212 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841140+0000", 2020-01-16T00:13:44.213 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:44.213 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.213 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.213 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:44.213 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841140+0000", 2020-01-16T00:13:44.214 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:44.214 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.214 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.214 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:44.215 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841137+0000", 2020-01-16T00:13:44.215 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00061049800000000001 2020-01-16T00:13:44.216 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.216 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.216 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:44.217 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841748+0000", 2020-01-16T00:13:44.217 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8703e-05 2020-01-16T00:13:44.217 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.217 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.217 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:44.218 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841767+0000", 2020-01-16T00:13:44.218 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5410000000000003e-06 2020-01-16T00:13:44.219 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.220 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.220 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:44.220 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841774+0000", 2020-01-16T00:13:44.220 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000100144 2020-01-16T00:13:44.220 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.220 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.220 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.221 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841874+0000", 2020-01-16T00:13:44.221 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9402999999999999e-05 2020-01-16T00:13:44.221 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.221 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.221 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:44.221 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841944+0000", 2020-01-16T00:13:44.221 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095901649999999995 2020-01-16T00:13:44.222 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.222 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.222 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:44.222 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.851534+0000", 2020-01-16T00:13:44.222 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5296000000000001e-05 2020-01-16T00:13:44.223 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.223 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.223 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:44.223 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.851579+0000", 2020-01-16T00:13:44.223 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010439159999999999 2020-01-16T00:13:44.224 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.224 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:44.224 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.224 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.224 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.224 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:110 2.0 e42/19 2:11fa79bc:::smithi12313934-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 42'68, mlcod=41'65)", 2020-01-16T00:13:44.224 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:21.842301+0000", 2020-01-16T00:13:44.224 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 22.185999724999999, 2020-01-16T00:13:44.225 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093040050000000006, 2020-01-16T00:13:44.225 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:44.225 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:44.225 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:44.225 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.225 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:44.225 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.842301+0000", 2020-01-16T00:13:44.226 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:44.226 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.226 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.226 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:44.226 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.842301+0000", 2020-01-16T00:13:44.226 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:44.226 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.226 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.227 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:44.227 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.842300+0000", 2020-01-16T00:13:44.227 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001154504 2020-01-16T00:13:44.227 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.227 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.227 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:44.227 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.843455+0000", 2020-01-16T00:13:44.228 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3263e-05 2020-01-16T00:13:44.228 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.228 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.228 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:44.228 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.843468+0000", 2020-01-16T00:13:44.228 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0786e-05 2020-01-16T00:13:44.228 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.229 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.229 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:44.229 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.843479+0000", 2020-01-16T00:13:44.229 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021168810000000001 2020-01-16T00:13:44.229 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.230 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.230 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.230 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.845596+0000", 2020-01-16T00:13:44.230 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0740000000000001e-05 2020-01-16T00:13:44.230 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.230 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.231 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:44.231 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.845616+0000", 2020-01-16T00:13:44.231 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0059668830000000001 2020-01-16T00:13:44.231 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.231 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.231 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:44.232 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.851583+0000", 2020-01-16T00:13:44.232 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2133999999999998e-05 2020-01-16T00:13:44.232 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.232 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.232 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:44.232 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.851605+0000", 2020-01-16T00:13:44.232 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093040050000000006 2020-01-16T00:13:44.232 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.233 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:44.233 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.233 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.233 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.233 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3228 3.1 3:8de1d8a3:::smithi12313934-13:head [write 588344~631611,stat] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:13:44.233 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.588539+0000", 2020-01-16T00:13:44.233 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 14.439761987000001, 2020-01-16T00:13:44.234 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012776884000000001, 2020-01-16T00:13:44.234 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:44.234 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:44.234 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:44.234 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:44.234 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:44.234 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3228 2020-01-16T00:13:44.235 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.235 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:44.235 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.235 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:44.235 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588539+0000", 2020-01-16T00:13:44.235 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:44.235 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.235 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.236 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:44.236 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588539+0000", 2020-01-16T00:13:44.236 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:44.236 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.236 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.236 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:44.237 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588536+0000", 2020-01-16T00:13:44.237 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021452 2020-01-16T00:13:44.237 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.237 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.237 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:44.237 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588750+0000", 2020-01-16T00:13:44.238 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.1432000000000001e-05 2020-01-16T00:13:44.238 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.238 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.238 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:44.238 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588772+0000", 2020-01-16T00:13:44.238 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1799999999999999e-06 2020-01-16T00:13:44.238 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.239 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.239 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:44.239 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588779+0000", 2020-01-16T00:13:44.239 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00095777700000000004 2020-01-16T00:13:44.239 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.239 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.239 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.240 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589737+0000", 2020-01-16T00:13:44.240 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013306599999999999 2020-01-16T00:13:44.240 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.240 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.241 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:44.241 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589870+0000", 2020-01-16T00:13:44.241 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020556 2020-01-16T00:13:44.241 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.241 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.241 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:13:44.241 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.590075+0000", 2020-01-16T00:13:44.242 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040226080000000004 2020-01-16T00:13:44.242 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.242 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.242 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:44.242 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.594098+0000", 2020-01-16T00:13:44.242 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.006043288 2020-01-16T00:13:44.243 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.243 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.243 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:44.243 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.600141+0000", 2020-01-16T00:13:44.243 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00089242900000000003 2020-01-16T00:13:44.244 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.244 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.244 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:44.244 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601034+0000", 2020-01-16T00:13:44.244 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6097e-05 2020-01-16T00:13:44.244 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.244 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.245 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:44.245 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601110+0000", 2020-01-16T00:13:44.245 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000206088 2020-01-16T00:13:44.245 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.246 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.246 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:44.246 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601316+0000", 2020-01-16T00:13:44.246 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012776884000000001 2020-01-16T00:13:44.246 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.246 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:44.247 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.247 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.247 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.247 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.1.13:77 2.1 e45/19 2:bafac7f5:::smithi12313934-46 low_tier:head v 45'100, mlcod=45'98)", 2020-01-16T00:13:44.247 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.589633+0000", 2020-01-16T00:13:44.247 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 14.438667764, 2020-01-16T00:13:44.247 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010560460000000001, 2020-01-16T00:13:44.248 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:44.248 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:44.248 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:44.248 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.248 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:44.248 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589633+0000", 2020-01-16T00:13:44.248 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:44.248 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.249 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.249 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:44.249 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589633+0000", 2020-01-16T00:13:44.249 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:44.249 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.249 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.249 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:44.250 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589630+0000", 2020-01-16T00:13:44.250 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00050499799999999999 2020-01-16T00:13:44.250 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.250 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.251 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:44.251 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.590135+0000", 2020-01-16T00:13:44.251 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0473e-05 2020-01-16T00:13:44.251 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.252 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.252 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:44.252 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.590146+0000", 2020-01-16T00:13:44.252 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9710000000000001e-06 2020-01-16T00:13:44.252 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.252 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.252 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:44.252 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.590152+0000", 2020-01-16T00:13:44.253 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00040412999999999999 2020-01-16T00:13:44.253 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.253 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.253 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.253 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.590556+0000", 2020-01-16T00:13:44.253 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6491000000000001e-05 2020-01-16T00:13:44.253 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.253 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.254 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:44.254 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.590583+0000", 2020-01-16T00:13:44.254 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095621360000000006 2020-01-16T00:13:44.254 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.254 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.254 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:44.255 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.600145+0000", 2020-01-16T00:13:44.255 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9072e-05 2020-01-16T00:13:44.255 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.256 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.256 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:44.256 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.600194+0000", 2020-01-16T00:13:44.256 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010560460000000001 2020-01-16T00:13:44.256 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.256 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:44.256 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.257 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.257 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.257 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3229 3.1 3:8de1d8a3:::smithi12313934-13:head [write 1689059~741123] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:13:44.257 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.596087+0000", 2020-01-16T00:13:44.257 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 14.432213925999999, 2020-01-16T00:13:44.257 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010699248, 2020-01-16T00:13:44.257 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:44.257 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:44.258 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:44.258 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:44.258 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:44.258 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3229 2020-01-16T00:13:44.258 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.258 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:44.258 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.259 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:44.259 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596087+0000", 2020-01-16T00:13:44.259 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:44.259 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.259 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.259 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:44.259 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596087+0000", 2020-01-16T00:13:44.259 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:44.260 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.260 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.260 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:44.260 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596084+0000", 2020-01-16T00:13:44.260 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00026243600000000001 2020-01-16T00:13:44.261 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.261 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.261 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:44.261 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596347+0000", 2020-01-16T00:13:44.262 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3594000000000001e-05 2020-01-16T00:13:44.262 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.262 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.262 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:44.263 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596370+0000", 2020-01-16T00:13:44.263 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5259999999999999e-06 2020-01-16T00:13:44.263 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.263 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.263 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:44.263 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596380+0000", 2020-01-16T00:13:44.264 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2298000000000005e-05 2020-01-16T00:13:44.264 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.264 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.264 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.264 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596452+0000", 2020-01-16T00:13:44.265 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000163702 2020-01-16T00:13:44.265 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.265 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.266 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:44.266 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596616+0000", 2020-01-16T00:13:44.266 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0047611729999999996 2020-01-16T00:13:44.266 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.266 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.266 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.266 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601377+0000", 2020-01-16T00:13:44.267 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014076900000000001 2020-01-16T00:13:44.267 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.267 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.267 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:44.267 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601518+0000", 2020-01-16T00:13:44.267 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021367599999999999 2020-01-16T00:13:44.267 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.268 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.268 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:13:44.268 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601731+0000", 2020-01-16T00:13:44.268 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0028479149999999999 2020-01-16T00:13:44.268 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.268 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.268 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:44.269 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.604579+0000", 2020-01-16T00:13:44.269 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001408133 2020-01-16T00:13:44.269 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.269 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.269 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:44.269 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605987+0000", 2020-01-16T00:13:44.270 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00072335599999999998 2020-01-16T00:13:44.270 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.270 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.270 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:44.270 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.606711+0000", 2020-01-16T00:13:44.270 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0289e-05 2020-01-16T00:13:44.271 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.271 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.271 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:44.271 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.606751+0000", 2020-01-16T00:13:44.271 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5298e-05 2020-01-16T00:13:44.271 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.271 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.271 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:44.272 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.606786+0000", 2020-01-16T00:13:44.272 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010699248 2020-01-16T00:13:44.272 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.272 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:44.272 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.272 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.272 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.272 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3230 3.1 3:8de1d8a3:::smithi12313934-13:head [setxattr _header (56) in=7b,truncate 2588344] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:13:44.273 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.596394+0000", 2020-01-16T00:13:44.273 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 14.431906967, 2020-01-16T00:13:44.273 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021205134, 2020-01-16T00:13:44.273 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:44.273 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:44.273 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:44.273 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:44.274 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:44.274 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3230 2020-01-16T00:13:44.274 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.274 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:44.274 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.274 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:44.274 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596394+0000", 2020-01-16T00:13:44.275 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:44.276 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.276 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.276 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:44.276 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596394+0000", 2020-01-16T00:13:44.277 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:44.277 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.277 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.278 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:44.278 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596393+0000", 2020-01-16T00:13:44.278 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7159999999999997e-06 2020-01-16T00:13:44.278 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.278 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.278 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:44.278 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596400+0000", 2020-01-16T00:13:44.279 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0074e-05 2020-01-16T00:13:44.279 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.279 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.279 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:44.279 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596410+0000", 2020-01-16T00:13:44.279 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1679999999999997e-06 2020-01-16T00:13:44.279 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.280 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.280 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:44.280 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596414+0000", 2020-01-16T00:13:44.280 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023609 2020-01-16T00:13:44.280 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.280 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.280 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.280 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596650+0000", 2020-01-16T00:13:44.281 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012316399999999999 2020-01-16T00:13:44.281 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.281 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.282 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:44.282 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596774+0000", 2020-01-16T00:13:44.282 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0067849670000000003 2020-01-16T00:13:44.282 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.282 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.283 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.283 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.603558+0000", 2020-01-16T00:13:44.283 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00092385999999999998 2020-01-16T00:13:44.283 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.283 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.283 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:44.283 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.604482+0000", 2020-01-16T00:13:44.284 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011729201 2020-01-16T00:13:44.284 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.284 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.284 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.284 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.616212+0000", 2020-01-16T00:13:44.284 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019515099999999999 2020-01-16T00:13:44.284 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.284 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.285 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:44.285 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.616407+0000", 2020-01-16T00:13:44.285 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000182843 2020-01-16T00:13:44.285 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.285 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.285 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:13:44.285 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.616590+0000", 2020-01-16T00:13:44.286 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000660572 2020-01-16T00:13:44.286 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.289 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.290 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:44.290 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617250+0000", 2020-01-16T00:13:44.290 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5333999999999997e-05 2020-01-16T00:13:44.290 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.290 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.290 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:44.290 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617285+0000", 2020-01-16T00:13:44.290 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023020899999999999 2020-01-16T00:13:44.291 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.291 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.291 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:44.291 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617516+0000", 2020-01-16T00:13:44.291 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4214000000000001e-05 2020-01-16T00:13:44.291 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.291 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.292 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:44.292 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617560+0000", 2020-01-16T00:13:44.292 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9366999999999998e-05 2020-01-16T00:13:44.292 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.292 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.292 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:44.293 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617599+0000", 2020-01-16T00:13:44.293 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021205134 2020-01-16T00:13:44.293 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.293 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:44.293 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.294 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.294 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.294 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3231 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e45)", 2020-01-16T00:13:44.294 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.596438+0000", 2020-01-16T00:13:44.294 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 14.431862957, 2020-01-16T00:13:44.295 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021455667000000001, 2020-01-16T00:13:44.295 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:44.295 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:44.295 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:44.295 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:44.295 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:44.295 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3231 2020-01-16T00:13:44.296 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.296 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:44.296 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.296 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:44.296 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596438+0000", 2020-01-16T00:13:44.296 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:44.297 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.297 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.297 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:44.297 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596438+0000", 2020-01-16T00:13:44.297 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:44.297 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.298 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.298 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:44.298 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596437+0000", 2020-01-16T00:13:44.298 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8500000000000004e-06 2020-01-16T00:13:44.298 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.298 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.298 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:44.299 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596441+0000", 2020-01-16T00:13:44.299 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9439999999999994e-06 2020-01-16T00:13:44.299 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.299 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.299 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:44.299 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596450+0000", 2020-01-16T00:13:44.299 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.267e-06 2020-01-16T00:13:44.299 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.300 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.300 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:44.300 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596455+0000", 2020-01-16T00:13:44.300 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034332399999999999 2020-01-16T00:13:44.300 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.300 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.300 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.300 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596798+0000", 2020-01-16T00:13:44.301 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011687400000000001 2020-01-16T00:13:44.301 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.301 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.301 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:44.301 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596915+0000", 2020-01-16T00:13:44.301 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0076057049999999999 2020-01-16T00:13:44.301 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.302 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.302 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.302 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.604520+0000", 2020-01-16T00:13:44.302 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8143999999999998e-05 2020-01-16T00:13:44.302 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.302 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.302 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:44.302 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.604569+0000", 2020-01-16T00:13:44.303 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012379960000000001 2020-01-16T00:13:44.303 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.303 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.304 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.304 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.616949+0000", 2020-01-16T00:13:44.304 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000120617 2020-01-16T00:13:44.304 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.305 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.306 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:44.306 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617069+0000", 2020-01-16T00:13:44.306 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00056715400000000005 2020-01-16T00:13:44.306 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.307 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.307 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.307 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617636+0000", 2020-01-16T00:13:44.307 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012844700000000001 2020-01-16T00:13:44.308 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.308 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.308 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:44.308 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617765+0000", 2020-01-16T00:13:44.309 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000129037 2020-01-16T00:13:44.309 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.309 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.309 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:44.310 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617894+0000", 2020-01-16T00:13:44.310 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021455667000000001 2020-01-16T00:13:44.310 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.310 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:44.310 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.311 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.311 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.311 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:126 2.1 e45/19 2:a4df40e2:::smithi12313934-42 low_tier:head v 45'102, mlcod=45'100)", 2020-01-16T00:13:44.311 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.627785+0000", 2020-01-16T00:13:44.312 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 14.400515615, 2020-01-16T00:13:44.312 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.009094207, 2020-01-16T00:13:44.312 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:44.312 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:44.312 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:44.313 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.313 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:44.313 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627785+0000", 2020-01-16T00:13:44.313 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:44.314 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.314 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.314 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:44.314 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627785+0000", 2020-01-16T00:13:44.315 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:44.315 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.315 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.315 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:44.316 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627783+0000", 2020-01-16T00:13:44.316 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00025870400000000001 2020-01-16T00:13:44.316 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.316 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.316 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:44.317 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.628042+0000", 2020-01-16T00:13:44.317 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7609999999999999e-05 2020-01-16T00:13:44.317 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.317 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.318 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:44.318 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.628059+0000", 2020-01-16T00:13:44.318 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7719999999999994e-06 2020-01-16T00:13:44.318 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.318 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.319 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:44.319 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.628067+0000", 2020-01-16T00:13:44.319 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9407999999999998e-05 2020-01-16T00:13:44.319 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.319 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.320 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.320 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.628127+0000", 2020-01-16T00:13:44.320 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9740000000000001e-05 2020-01-16T00:13:44.320 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.321 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.321 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:44.321 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.628146+0000", 2020-01-16T00:13:44.321 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0086960460000000007 2020-01-16T00:13:44.321 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.322 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.322 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:44.322 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.636842+0000", 2020-01-16T00:13:44.322 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7321999999999999e-05 2020-01-16T00:13:44.322 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.323 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.323 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:44.323 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.636880+0000", 2020-01-16T00:13:44.323 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.009094207 2020-01-16T00:13:44.324 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.324 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:44.324 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.324 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.324 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:44.325 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:44.330 INFO:tasks.rados.rados.0.smithi123.stdout:1437: finishing write tid 1 to smithi12313934-31 2020-01-16T00:13:44.330 INFO:tasks.rados.rados.0.smithi123.stdout:1437: finishing write tid 2 to smithi12313934-31 2020-01-16T00:13:44.330 INFO:tasks.rados.rados.0.smithi123.stdout:1437: finishing write tid 3 to smithi12313934-31 2020-01-16T00:13:44.330 INFO:tasks.rados.rados.0.smithi123.stdout:1437: finishing write tid 5 to smithi12313934-31 2020-01-16T00:13:44.330 INFO:tasks.rados.rados.0.smithi123.stdout:1437: finishing write tid 6 to smithi12313934-31 2020-01-16T00:13:44.331 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 31 v 176 (ObjNum 473 snap 0 seq_num 473) dirty exists 2020-01-16T00:13:44.331 INFO:tasks.rados.rados.0.smithi123.stdout:1437: left oid 31 (ObjNum 473 snap 0 seq_num 473) 2020-01-16T00:13:44.331 INFO:tasks.rados.rados.0.smithi123.stdout:1438: finishing write tid 1 to smithi12313934-30 2020-01-16T00:13:44.331 INFO:tasks.rados.rados.0.smithi123.stdout:1433: done (5 left) 2020-01-16T00:13:44.331 INFO:tasks.rados.rados.0.smithi123.stdout:1434: done (4 left) 2020-01-16T00:13:44.331 INFO:tasks.rados.rados.0.smithi123.stdout:1435: done (3 left) 2020-01-16T00:13:44.331 INFO:tasks.rados.rados.0.smithi123.stdout:1437: done (2 left) 2020-01-16T00:13:44.331 INFO:tasks.rados.rados.0.smithi123.stdout:1439: done (1 left) 2020-01-16T00:13:44.332 INFO:tasks.rados.rados.0.smithi123.stdout:1440: delete oid 43 current snap is 0 2020-01-16T00:13:44.332 INFO:tasks.rados.rados.0.smithi123.stdout:1438: finishing write tid 2 to smithi12313934-30 2020-01-16T00:13:44.332 INFO:tasks.rados.rados.0.smithi123.stdout:1438: finishing write tid 3 to smithi12313934-30 2020-01-16T00:13:44.332 INFO:tasks.rados.rados.0.smithi123.stdout:1438: finishing write tid 4 to smithi12313934-30 2020-01-16T00:13:44.332 INFO:tasks.rados.rados.0.smithi123.stdout:1438: finishing write tid 6 to smithi12313934-30 2020-01-16T00:13:44.332 INFO:tasks.rados.rados.0.smithi123.stdout:1438: finishing write tid 7 to smithi12313934-30 2020-01-16T00:13:44.332 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 30 v 156 (ObjNum 474 snap 0 seq_num 474) dirty exists 2020-01-16T00:13:44.333 INFO:tasks.rados.rados.0.smithi123.stdout:1438: left oid 30 (ObjNum 474 snap 0 seq_num 474) 2020-01-16T00:13:44.333 INFO:tasks.rados.rados.0.smithi123.stdout:1438: done (1 left) 2020-01-16T00:13:44.333 INFO:tasks.rados.rados.0.smithi123.stdout:1440: done (0 left) 2020-01-16T00:13:44.333 INFO:tasks.rados.rados.0.smithi123.stdout:1441: read oid 34 snap -1 2020-01-16T00:13:44.333 INFO:tasks.rados.rados.0.smithi123.stdout:1441: expect (ObjNum 471 snap 0 seq_num 471) 2020-01-16T00:13:44.333 INFO:tasks.rados.rados.0.smithi123.stdout:1442: write oid 13 current snap is 0 2020-01-16T00:13:44.333 INFO:tasks.rados.rados.0.smithi123.stdout:1442: seq_num 477 ranges {416939=707659,1896499=120440} 2020-01-16T00:13:44.333 INFO:tasks.rados.rados.0.smithi123.stdout:1442: writing smithi12313934-13 from 416939 to 1124598 tid 1 2020-01-16T00:13:44.334 INFO:tasks.rados.rados.0.smithi123.stdout:1442: writing smithi12313934-13 from 1896499 to 2016939 tid 2 2020-01-16T00:13:44.334 INFO:tasks.rados.rados.0.smithi123.stdout:1443: write (excl) oid 49 current snap is 0 2020-01-16T00:13:44.334 INFO:tasks.rados.rados.0.smithi123.stdout:1443: seq_num 478 ranges {433746=782908,1954981=78765} 2020-01-16T00:13:44.334 INFO:tasks.rados.rados.0.smithi123.stdout:1443: writing smithi12313934-49 from 433746 to 1216654 tid 1 2020-01-16T00:13:44.334 INFO:tasks.rados.rados.0.smithi123.stdout:1443: writing smithi12313934-49 from 1954981 to 2033746 tid 2 2020-01-16T00:13:44.334 INFO:tasks.rados.rados.0.smithi123.stdout:1444: write oid 28 current snap is 0 2020-01-16T00:13:44.334 INFO:tasks.rados.rados.0.smithi123.stdout:1444: seq_num 479 ranges {450553=574510,1613463=437090} 2020-01-16T00:13:44.334 INFO:tasks.rados.rados.0.smithi123.stdout:1444: writing smithi12313934-28 from 450553 to 1025063 tid 1 2020-01-16T00:13:44.335 INFO:tasks.rados.rados.0.smithi123.stdout:1444: writing smithi12313934-28 from 1613463 to 2050553 tid 2 2020-01-16T00:13:44.335 INFO:tasks.rados.rados.0.smithi123.stdout:1445: write oid 6 current snap is 0 2020-01-16T00:13:44.335 INFO:tasks.rados.rados.0.smithi123.stdout:1445: seq_num 480 ranges {467360=649759,1671945=395415} 2020-01-16T00:13:44.335 INFO:tasks.rados.rados.0.smithi123.stdout:1445: writing smithi12313934-6 from 467360 to 1117119 tid 1 2020-01-16T00:13:44.335 INFO:tasks.rados.rados.0.smithi123.stdout:1445: writing smithi12313934-6 from 1671945 to 2067360 tid 2 2020-01-16T00:13:44.335 INFO:tasks.rados.rados.0.smithi123.stdout:1446: read oid 48 snap -1 2020-01-16T00:13:44.335 INFO:tasks.rados.rados.0.smithi123.stdout:1446: expect (ObjNum 457 snap 0 seq_num 457) 2020-01-16T00:13:44.336 INFO:tasks.rados.rados.0.smithi123.stdout:1447: write oid 18 current snap is 0 2020-01-16T00:13:44.336 INFO:tasks.rados.rados.0.smithi123.stdout:1447: seq_num 481 ranges {484167=725008,1730427=353740} 2020-01-16T00:13:44.336 INFO:tasks.rados.rados.0.smithi123.stdout:1447: writing smithi12313934-18 from 484167 to 1209175 tid 1 2020-01-16T00:13:44.336 INFO:tasks.rados.rados.0.smithi123.stdout:1447: writing smithi12313934-18 from 1730427 to 2084167 tid 2 2020-01-16T00:13:44.336 INFO:tasks.rados.rados.0.smithi123.stdout:1448: delete oid 43 current snap is 0 2020-01-16T00:13:44.336 INFO:tasks.rados.rados.0.smithi123.stdout:1441: expect (ObjNum 471 snap 0 seq_num 471) 2020-01-16T00:13:44.336 INFO:tasks.rados.rados.0.smithi123.stdout:1442: finishing write tid 1 to smithi12313934-13 2020-01-16T00:13:44.336 INFO:tasks.rados.rados.0.smithi123.stdout:1442: finishing write tid 2 to smithi12313934-13 2020-01-16T00:13:44.337 INFO:tasks.rados.rados.0.smithi123.stdout:1443: finishing write tid 1 to smithi12313934-49 2020-01-16T00:13:44.337 INFO:tasks.rados.rados.0.smithi123.stdout:1443: finishing write tid 2 to smithi12313934-49 2020-01-16T00:13:44.337 INFO:tasks.rados.rados.0.smithi123.stdout:1443: finishing write tid 4 to smithi12313934-49 2020-01-16T00:13:44.337 INFO:tasks.rados.rados.0.smithi123.stdout:1443: finishing write tid 5 to smithi12313934-49 2020-01-16T00:13:44.337 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 49 v 153 (ObjNum 478 snap 0 seq_num 478) dirty exists 2020-01-16T00:13:44.337 INFO:tasks.rados.rados.0.smithi123.stdout:1443: left oid 49 (ObjNum 478 snap 0 seq_num 478) 2020-01-16T00:13:44.337 INFO:tasks.rados.rados.0.smithi123.stdout:1444: finishing write tid 1 to smithi12313934-28 2020-01-16T00:13:44.337 INFO:tasks.rados.rados.0.smithi123.stdout:1442: finishing write tid 4 to smithi12313934-13 2020-01-16T00:13:44.338 INFO:tasks.rados.rados.0.smithi123.stdout:1442: finishing write tid 5 to smithi12313934-13 2020-01-16T00:13:44.338 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 94 (ObjNum 477 snap 0 seq_num 477) dirty exists 2020-01-16T00:13:44.338 INFO:tasks.rados.rados.0.smithi123.stdout:1442: left oid 13 (ObjNum 477 snap 0 seq_num 477) 2020-01-16T00:13:44.338 INFO:tasks.rados.rados.0.smithi123.stdout:1444: finishing write tid 2 to smithi12313934-28 2020-01-16T00:13:44.338 INFO:tasks.rados.rados.0.smithi123.stdout:1444: finishing write tid 4 to smithi12313934-28 2020-01-16T00:13:44.338 INFO:tasks.rados.rados.0.smithi123.stdout:1444: finishing write tid 5 to smithi12313934-28 2020-01-16T00:13:44.338 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 28 v 179 (ObjNum 479 snap 0 seq_num 479) dirty exists 2020-01-16T00:13:44.339 INFO:tasks.rados.rados.0.smithi123.stdout:1444: left oid 28 (ObjNum 479 snap 0 seq_num 479) 2020-01-16T00:13:44.339 INFO:tasks.rados.rados.0.smithi123.stdout:1445: finishing write tid 1 to smithi12313934-6 2020-01-16T00:13:44.339 INFO:tasks.rados.rados.0.smithi123.stdout:1441: done (7 left) 2020-01-16T00:13:44.339 INFO:tasks.rados.rados.0.smithi123.stdout:1442: done (6 left) 2020-01-16T00:13:44.339 INFO:tasks.rados.rados.0.smithi123.stdout:1443: done (5 left) 2020-01-16T00:13:44.339 INFO:tasks.rados.rados.0.smithi123.stdout:1444: done (4 left) 2020-01-16T00:13:44.339 INFO:tasks.rados.rados.0.smithi123.stdout:1448: done (3 left) 2020-01-16T00:13:44.339 INFO:tasks.rados.rados.0.smithi123.stdout:1449: read oid 1 snap -1 2020-01-16T00:13:44.340 INFO:tasks.rados.rados.0.smithi123.stdout:1449: expect (ObjNum 449 snap 0 seq_num 449) 2020-01-16T00:13:44.340 INFO:tasks.rados.rados.0.smithi123.stdout:1450: write oid 25 current snap is 0 2020-01-16T00:13:44.340 INFO:tasks.rados.rados.0.smithi123.stdout:1450: seq_num 483 ranges {517781=475506,1731038=386743} 2020-01-16T00:13:44.340 INFO:tasks.rados.rados.0.smithi123.stdout:1450: writing smithi12313934-25 from 517781 to 993287 tid 1 2020-01-16T00:13:44.340 INFO:tasks.rados.rados.0.smithi123.stdout:1450: writing smithi12313934-25 from 1731038 to 2117781 tid 2 2020-01-16T00:13:44.340 INFO:tasks.rados.rados.0.smithi123.stdout:1451: write oid 11 current snap is 0 2020-01-16T00:13:44.340 INFO:tasks.rados.rados.0.smithi123.stdout:1451: seq_num 484 ranges {534588=550755,1789520=345068} 2020-01-16T00:13:44.340 INFO:tasks.rados.rados.0.smithi123.stdout:1451: writing smithi12313934-11 from 534588 to 1085343 tid 1 2020-01-16T00:13:44.341 INFO:tasks.rados.rados.0.smithi123.stdout:1451: writing smithi12313934-11 from 1789520 to 2134588 tid 2 2020-01-16T00:13:44.341 INFO:tasks.rados.rados.0.smithi123.stdout:1446: expect (ObjNum 457 snap 0 seq_num 457) 2020-01-16T00:13:44.341 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:44.322+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:44.341 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-01-16T00:13:44.426 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:44.422+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:44.475 INFO:tasks.rados.rados.0.smithi123.stdout:1445: finishing write tid 2 to smithi12313934-6 2020-01-16T00:13:44.476 INFO:tasks.rados.rados.0.smithi123.stdout:1445: finishing write tid 4 to smithi12313934-6 2020-01-16T00:13:44.476 INFO:tasks.rados.rados.0.smithi123.stdout:1452: read oid 13 snap -1 2020-01-16T00:13:44.476 INFO:tasks.rados.rados.0.smithi123.stdout:1452: expect (ObjNum 477 snap 0 seq_num 477) 2020-01-16T00:13:44.476 INFO:tasks.rados.rados.0.smithi123.stdout:1446: done (6 left) 2020-01-16T00:13:44.476 INFO:tasks.rados.rados.0.smithi123.stdout:1453: read oid 39 snap -1 2020-01-16T00:13:44.476 INFO:tasks.rados.rados.0.smithi123.stdout:1453: expect (ObjNum 462 snap 0 seq_num 462) 2020-01-16T00:13:44.477 INFO:tasks.rados.rados.0.smithi123.stdout:1454: write oid 48 current snap is 0 2020-01-16T00:13:44.477 INFO:tasks.rados.rados.0.smithi123.stdout:1454: seq_num 485 ranges {551395=626004,1848002=303393} 2020-01-16T00:13:44.477 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:44.477 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:44.477 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:13:44.477 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:44.482 INFO:tasks.rados.rados.0.smithi123.stdout:1454: writing smithi12313934-48 from 551395 to 1177399 tid 1 2020-01-16T00:13:44.487 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-01-16T00:13:44.489 INFO:tasks.rados.rados.0.smithi123.stdout:1454: writing smithi12313934-48 from 1848002 to 2151395 tid 2 2020-01-16T00:13:44.489 INFO:tasks.rados.rados.0.smithi123.stdout:1455: read oid 4 snap -1 2020-01-16T00:13:44.489 INFO:tasks.rados.rados.0.smithi123.stdout:1455: expect (ObjNum 387 snap 0 seq_num 387) 2020-01-16T00:13:44.490 INFO:tasks.rados.rados.0.smithi123.stdout:1445: finishing write tid 5 to smithi12313934-6 2020-01-16T00:13:44.490 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 6 v 143 (ObjNum 480 snap 0 seq_num 480) dirty exists 2020-01-16T00:13:44.490 INFO:tasks.rados.rados.0.smithi123.stdout:1445: left oid 6 (ObjNum 480 snap 0 seq_num 480) 2020-01-16T00:13:44.490 INFO:tasks.rados.rados.0.smithi123.stdout:1447: finishing write tid 1 to smithi12313934-18 2020-01-16T00:13:44.490 INFO:tasks.rados.rados.0.smithi123.stdout:1447: finishing write tid 2 to smithi12313934-18 2020-01-16T00:13:44.490 INFO:tasks.rados.rados.0.smithi123.stdout:1456: read oid 31 snap -1 2020-01-16T00:13:44.491 INFO:tasks.rados.rados.0.smithi123.stdout:1456: expect (ObjNum 473 snap 0 seq_num 473) 2020-01-16T00:13:44.491 INFO:tasks.rados.rados.0.smithi123.stdout:1445: done (9 left) 2020-01-16T00:13:44.491 INFO:tasks.rados.rados.0.smithi123.stdout:1457: read oid 33 snap -1 2020-01-16T00:13:44.491 INFO:tasks.rados.rados.0.smithi123.stdout:1457: expect (ObjNum 330 snap 0 seq_num 330) 2020-01-16T00:13:44.492 INFO:tasks.rados.rados.0.smithi123.stdout:1458: write (excl) oid 50 current snap is 0 2020-01-16T00:13:44.492 INFO:tasks.rados.rados.0.smithi123.stdout:1458: seq_num 486 ranges {568202=701253,1906484=261718} 2020-01-16T00:13:44.497 INFO:tasks.rados.rados.0.smithi123.stdout:1458: writing smithi12313934-50 from 568202 to 1269455 tid 1 2020-01-16T00:13:44.502 INFO:tasks.rados.rados.0.smithi123.stdout:1458: writing smithi12313934-50 from 1906484 to 2168202 tid 2 2020-01-16T00:13:44.502 INFO:tasks.rados.rados.0.smithi123.stdout:1459: read oid 42 snap -1 2020-01-16T00:13:44.502 INFO:tasks.rados.rados.0.smithi123.stdout:1459: expect (ObjNum 432 snap 0 seq_num 432) 2020-01-16T00:13:44.503 INFO:tasks.rados.rados.0.smithi123.stdout:1460: read oid 20 snap -1 2020-01-16T00:13:44.503 INFO:tasks.rados.rados.0.smithi123.stdout:1460: expect deleted 2020-01-16T00:13:44.503 INFO:tasks.rados.rados.0.smithi123.stdout:1461: read oid 30 snap -1 2020-01-16T00:13:44.503 INFO:tasks.rados.rados.0.smithi123.stdout:1461: expect (ObjNum 474 snap 0 seq_num 474) 2020-01-16T00:13:44.503 INFO:tasks.rados.rados.0.smithi123.stdout:1447: finishing write tid 4 to smithi12313934-18 2020-01-16T00:13:44.503 INFO:tasks.rados.rados.0.smithi123.stdout:1447: finishing write tid 5 to smithi12313934-18 2020-01-16T00:13:44.503 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 18 v 266 (ObjNum 481 snap 0 seq_num 481) dirty exists 2020-01-16T00:13:44.503 INFO:tasks.rados.rados.0.smithi123.stdout:1447: left oid 18 (ObjNum 481 snap 0 seq_num 481) 2020-01-16T00:13:44.504 INFO:tasks.rados.rados.0.smithi123.stdout:1449: expect (ObjNum 449 snap 0 seq_num 449) 2020-01-16T00:13:44.527 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:44.522+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:44.591 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:44.591 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:44.591 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:13:44.591 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:13:44.591 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:44.600 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-01-16T00:13:44.628 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:44.624+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:44.728 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:44.726+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:44.730 INFO:tasks.rados.rados.0.smithi123.stdout:1450: finishing write tid 1 to smithi12313934-25 2020-01-16T00:13:44.730 INFO:tasks.rados.rados.0.smithi123.stdout:1450: finishing write tid 2 to smithi12313934-25 2020-01-16T00:13:44.730 INFO:tasks.rados.rados.0.smithi123.stdout:1450: finishing write tid 4 to smithi12313934-25 2020-01-16T00:13:44.730 INFO:tasks.rados.rados.0.smithi123.stdout:1450: finishing write tid 5 to smithi12313934-25 2020-01-16T00:13:44.730 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 25 v 216 (ObjNum 483 snap 0 seq_num 483) dirty exists 2020-01-16T00:13:44.731 INFO:tasks.rados.rados.0.smithi123.stdout:1450: left oid 25 (ObjNum 483 snap 0 seq_num 483) 2020-01-16T00:13:44.731 INFO:tasks.rados.rados.0.smithi123.stdout:1451: finishing write tid 1 to smithi12313934-11 2020-01-16T00:13:44.731 INFO:tasks.rados.rados.0.smithi123.stdout:1451: finishing write tid 2 to smithi12313934-11 2020-01-16T00:13:44.731 INFO:tasks.rados.rados.0.smithi123.stdout:1451: finishing write tid 4 to smithi12313934-11 2020-01-16T00:13:44.731 INFO:tasks.rados.rados.0.smithi123.stdout:1447: done (13 left) 2020-01-16T00:13:44.731 INFO:tasks.rados.rados.0.smithi123.stdout:1449: done (12 left) 2020-01-16T00:13:44.731 INFO:tasks.rados.rados.0.smithi123.stdout:1450: done (11 left) 2020-01-16T00:13:44.731 INFO:tasks.rados.rados.0.smithi123.stdout:1462: read oid 49 snap -1 2020-01-16T00:13:44.732 INFO:tasks.rados.rados.0.smithi123.stdout:1462: expect (ObjNum 478 snap 0 seq_num 478) 2020-01-16T00:13:44.732 INFO:tasks.rados.rados.0.smithi123.stdout:1463: write oid 36 current snap is 0 2020-01-16T00:13:44.732 INFO:tasks.rados.rados.0.smithi123.stdout:1463: seq_num 487 ranges {585009=492855,1564966=620043} 2020-01-16T00:13:44.737 INFO:tasks.rados.rados.0.smithi123.stdout:1463: writing smithi12313934-36 from 585009 to 1077864 tid 1 2020-01-16T00:13:44.746 INFO:tasks.rados.rados.0.smithi123.stdout:1463: writing smithi12313934-36 from 1564966 to 2185009 tid 2 2020-01-16T00:13:44.746 INFO:tasks.rados.rados.0.smithi123.stdout:1464: write (excl) oid 7 current snap is 0 2020-01-16T00:13:44.746 INFO:tasks.rados.rados.0.smithi123.stdout:1464: seq_num 488 ranges {601816=568104,1623448=451823} 2020-01-16T00:13:44.752 INFO:tasks.rados.rados.0.smithi123.stdout:1464: writing smithi12313934-7 from 601816 to 1169920 tid 1 2020-01-16T00:13:44.767 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:44.767 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:13:44.767 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:13:44.768 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:13:44.768 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.768 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:923 3.8 3:113947f2:::smithi12313934-12:head [write 1564138~470341] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:44.769 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.869292+0000", 2020-01-16T00:13:44.769 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 66.886533514999996, 2020-01-16T00:13:44.769 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017661620999999999, 2020-01-16T00:13:44.769 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:44.795 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:44.796 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:44.796 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:44.796 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:44.796 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 923 2020-01-16T00:13:44.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.797 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:44.797 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.797 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:44.797 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869292+0000", 2020-01-16T00:13:44.797 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:44.798 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.798 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.798 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:44.798 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869292+0000", 2020-01-16T00:13:44.798 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:44.799 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.799 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.799 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:44.799 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869291+0000", 2020-01-16T00:13:44.799 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00049842899999999999 2020-01-16T00:13:44.800 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.800 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.800 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:44.800 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869789+0000", 2020-01-16T00:13:44.800 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1014999999999999e-05 2020-01-16T00:13:44.801 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.801 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.801 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:44.801 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869830+0000", 2020-01-16T00:13:44.801 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7880000000000007e-06 2020-01-16T00:13:44.802 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.802 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.802 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:44.802 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869838+0000", 2020-01-16T00:13:44.802 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8346999999999997e-05 2020-01-16T00:13:44.803 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.803 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.803 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.803 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869876+0000", 2020-01-16T00:13:44.803 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9726999999999999e-05 2020-01-16T00:13:44.804 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.804 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.804 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:44.804 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869936+0000", 2020-01-16T00:13:44.804 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003434917 2020-01-16T00:13:44.804 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.805 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.805 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.805 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.873371+0000", 2020-01-16T00:13:44.805 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118808 2020-01-16T00:13:44.814 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.815 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.815 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:44.815 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.873490+0000", 2020-01-16T00:13:44.815 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036712910000000001 2020-01-16T00:13:44.816 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.816 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.816 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.816 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877161+0000", 2020-01-16T00:13:44.817 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9909999999999999e-05 2020-01-16T00:13:44.821 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.822 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.822 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:44.822 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877241+0000", 2020-01-16T00:13:44.822 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031619970000000002 2020-01-16T00:13:44.823 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.823 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.823 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.823 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880403+0000", 2020-01-16T00:13:44.824 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013071099999999999 2020-01-16T00:13:44.824 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.824 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.824 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:44.824 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880534+0000", 2020-01-16T00:13:44.825 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00032823500000000002 2020-01-16T00:13:44.825 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.825 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.825 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1,7", 2020-01-16T00:13:44.826 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880862+0000", 2020-01-16T00:13:44.826 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00513862 2020-01-16T00:13:44.827 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.827 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.827 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:44.827 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886001+0000", 2020-01-16T00:13:44.827 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8869000000000001e-05 2020-01-16T00:13:44.828 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.828 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.828 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:44.828 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886030+0000", 2020-01-16T00:13:44.829 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00081600200000000003 2020-01-16T00:13:44.829 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.829 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.830 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:44.830 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886846+0000", 2020-01-16T00:13:44.830 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4095999999999998e-05 2020-01-16T00:13:44.830 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.830 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.831 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:44.831 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886900+0000", 2020-01-16T00:13:44.831 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.448e-05 2020-01-16T00:13:44.831 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.835 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.835 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:44.836 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886954+0000", 2020-01-16T00:13:44.836 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017661620999999999 2020-01-16T00:13:44.846 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.846 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:44.846 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.846 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.846 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.846 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2226 3.7 3:e1be6d3b:::smithi12313934-24:head [setxattr _header (56) in=7b,truncate 2100908] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:13:44.846 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.980217+0000", 2020-01-16T00:13:44.847 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 38.775608849999998, 2020-01-16T00:13:44.847 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023113018999999999, 2020-01-16T00:13:44.847 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:44.847 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:44.847 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:44.847 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:44.847 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:44.848 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2226 2020-01-16T00:13:44.848 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.848 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:44.848 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.848 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:44.848 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980217+0000", 2020-01-16T00:13:44.848 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:44.848 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.849 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.849 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:44.849 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980217+0000", 2020-01-16T00:13:44.849 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:44.849 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.849 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.849 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:44.849 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980216+0000", 2020-01-16T00:13:44.850 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5049999999999998e-06 2020-01-16T00:13:44.850 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.850 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.850 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:44.850 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980222+0000", 2020-01-16T00:13:44.850 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5654999999999998e-05 2020-01-16T00:13:44.851 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.851 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.851 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:44.851 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980237+0000", 2020-01-16T00:13:44.851 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9740000000000004e-06 2020-01-16T00:13:44.851 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.851 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.852 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:44.852 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980241+0000", 2020-01-16T00:13:44.852 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002947776 2020-01-16T00:13:44.852 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.852 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.852 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.852 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.983189+0000", 2020-01-16T00:13:44.852 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0027099400000000001 2020-01-16T00:13:44.853 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.853 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.853 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:44.853 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985899+0000", 2020-01-16T00:13:44.853 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014275589999999999 2020-01-16T00:13:44.853 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.853 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.853 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.854 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.000175+0000", 2020-01-16T00:13:44.854 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011211 2020-01-16T00:13:44.854 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.854 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.854 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:44.854 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.000287+0000", 2020-01-16T00:13:44.854 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000229886 2020-01-16T00:13:44.855 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.855 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.855 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,7", 2020-01-16T00:13:44.855 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.000517+0000", 2020-01-16T00:13:44.855 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00095715399999999999 2020-01-16T00:13:44.855 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.855 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.855 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:44.856 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.001474+0000", 2020-01-16T00:13:44.856 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001341512 2020-01-16T00:13:44.856 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.856 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.856 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:44.856 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.002815+0000", 2020-01-16T00:13:44.856 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035789099999999999 2020-01-16T00:13:44.857 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.857 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.857 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:44.857 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003173+0000", 2020-01-16T00:13:44.857 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8571000000000005e-05 2020-01-16T00:13:44.857 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.857 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.857 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:44.858 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003242+0000", 2020-01-16T00:13:44.858 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8293000000000003e-05 2020-01-16T00:13:44.858 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.858 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.858 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:44.858 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003330+0000", 2020-01-16T00:13:44.858 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023113018999999999 2020-01-16T00:13:44.859 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.859 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:44.859 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.859 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.859 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.859 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2227 3.7 3:e1be6d3b:::smithi12313934-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e35)", 2020-01-16T00:13:44.859 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:13:44.859 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 38.775561596999999, 2020-01-16T00:13:44.860 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023303356000000001, 2020-01-16T00:13:44.860 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:44.860 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:44.860 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:44.860 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:44.860 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:44.860 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2227 2020-01-16T00:13:44.860 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.861 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:44.862 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.862 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:44.862 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:13:44.862 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:44.862 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.862 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.862 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:44.863 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:13:44.863 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:44.863 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.863 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.863 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:44.863 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:13:44.863 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3079999999999999e-06 2020-01-16T00:13:44.864 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.864 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.864 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:44.864 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980267+0000", 2020-01-16T00:13:44.864 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9290000000000001e-06 2020-01-16T00:13:44.864 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.864 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.864 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:44.865 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980274+0000", 2020-01-16T00:13:44.865 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7400000000000002e-06 2020-01-16T00:13:44.865 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.865 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.865 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:44.865 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980278+0000", 2020-01-16T00:13:44.865 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056624300000000004 2020-01-16T00:13:44.866 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.866 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.866 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.866 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985940+0000", 2020-01-16T00:13:44.866 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4261000000000002e-05 2020-01-16T00:13:44.866 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.866 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.866 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:44.867 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985985+0000", 2020-01-16T00:13:44.867 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015087676 2020-01-16T00:13:44.867 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.867 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.867 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.867 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.001072+0000", 2020-01-16T00:13:44.867 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011960600000000001 2020-01-16T00:13:44.867 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.868 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.868 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:44.868 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.001192+0000", 2020-01-16T00:13:44.868 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021718309999999999 2020-01-16T00:13:44.868 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.868 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.868 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.868 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003364+0000", 2020-01-16T00:13:44.869 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000102855 2020-01-16T00:13:44.869 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.869 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.869 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:44.869 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003467+0000", 2020-01-16T00:13:44.869 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010122499999999999 2020-01-16T00:13:44.869 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.870 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.870 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:44.870 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003568+0000", 2020-01-16T00:13:44.870 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023303356000000001 2020-01-16T00:13:44.870 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.870 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:44.870 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.870 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.871 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.871 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2233 3.2 3:4f8fe446:::smithi12313934-48:head [setxattr _header (56) in=7b,truncate 2117715] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:13:44.871 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.993103+0000", 2020-01-16T00:13:44.871 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 38.762723375, 2020-01-16T00:13:44.871 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017951763999999999, 2020-01-16T00:13:44.871 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:44.871 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:44.872 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:44.872 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:44.872 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:44.872 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2233 2020-01-16T00:13:44.872 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.872 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:44.872 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.872 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:44.873 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993103+0000", 2020-01-16T00:13:44.873 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:44.873 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.873 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.873 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:44.873 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993103+0000", 2020-01-16T00:13:44.873 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:44.873 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.874 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.874 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:44.874 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993102+0000", 2020-01-16T00:13:44.874 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9369999999999993e-06 2020-01-16T00:13:44.874 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.874 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.874 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:44.875 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993111+0000", 2020-01-16T00:13:44.875 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3340000000000002e-06 2020-01-16T00:13:44.875 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.875 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.875 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:44.875 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993119+0000", 2020-01-16T00:13:44.875 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1769999999999997e-06 2020-01-16T00:13:44.875 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.876 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.876 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:44.876 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993124+0000", 2020-01-16T00:13:44.876 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001601304 2020-01-16T00:13:44.876 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.876 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.876 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.876 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.994725+0000", 2020-01-16T00:13:44.877 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048777250000000003 2020-01-16T00:13:44.877 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.877 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.877 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:44.877 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999603+0000", 2020-01-16T00:13:44.877 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0099793880000000005 2020-01-16T00:13:44.877 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.877 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.878 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.878 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.009583+0000", 2020-01-16T00:13:44.878 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000108207 2020-01-16T00:13:44.878 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.878 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.878 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:44.878 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.009691+0000", 2020-01-16T00:13:44.879 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012862999999999999 2020-01-16T00:13:44.879 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.879 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.879 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:44.879 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.009819+0000", 2020-01-16T00:13:44.879 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00077947599999999998 2020-01-16T00:13:44.879 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.879 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.880 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:44.880 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010599+0000", 2020-01-16T00:13:44.880 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000221282 2020-01-16T00:13:44.880 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.880 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.880 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:44.880 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010820+0000", 2020-01-16T00:13:44.880 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7021999999999995e-05 2020-01-16T00:13:44.881 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.881 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.881 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:44.881 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010907+0000", 2020-01-16T00:13:44.881 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9815999999999997e-05 2020-01-16T00:13:44.881 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.881 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.882 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:44.882 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010977+0000", 2020-01-16T00:13:44.882 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7453999999999994e-05 2020-01-16T00:13:44.882 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.882 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.882 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:44.882 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.011054+0000", 2020-01-16T00:13:44.882 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017951763999999999 2020-01-16T00:13:44.883 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.883 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:44.883 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.883 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.883 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.883 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2234 3.2 3:4f8fe446:::smithi12313934-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e35)", 2020-01-16T00:13:44.883 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.993135+0000", 2020-01-16T00:13:44.884 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 38.762690921999997, 2020-01-16T00:13:44.884 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018067916999999999, 2020-01-16T00:13:44.884 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:44.884 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:44.884 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:44.884 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:44.884 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:44.884 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2234 2020-01-16T00:13:44.885 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.885 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:44.885 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.885 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:44.885 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993135+0000", 2020-01-16T00:13:44.885 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:44.885 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.885 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.886 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:44.886 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993135+0000", 2020-01-16T00:13:44.886 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:44.886 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.886 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.886 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:44.886 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993134+0000", 2020-01-16T00:13:44.886 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5630000000000002e-06 2020-01-16T00:13:44.887 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.887 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.887 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:44.887 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993139+0000", 2020-01-16T00:13:44.887 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1730000000000003e-06 2020-01-16T00:13:44.887 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.887 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.888 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:44.888 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993144+0000", 2020-01-16T00:13:44.888 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.937e-06 2020-01-16T00:13:44.888 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.888 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.888 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:44.888 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993147+0000", 2020-01-16T00:13:44.888 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0065164000000000003 2020-01-16T00:13:44.889 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.889 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.889 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.889 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999663+0000", 2020-01-16T00:13:44.889 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6215000000000002e-05 2020-01-16T00:13:44.889 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.889 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.889 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:44.890 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999729+0000", 2020-01-16T00:13:44.890 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010475887999999999 2020-01-16T00:13:44.890 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.890 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.890 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.890 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010205+0000", 2020-01-16T00:13:44.891 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119471 2020-01-16T00:13:44.891 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.891 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.891 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:44.891 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010325+0000", 2020-01-16T00:13:44.891 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00075498799999999995 2020-01-16T00:13:44.891 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.891 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.892 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.892 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.011080+0000", 2020-01-16T00:13:44.892 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7904e-05 2020-01-16T00:13:44.892 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.892 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.892 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:44.892 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.011138+0000", 2020-01-16T00:13:44.893 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5273999999999999e-05 2020-01-16T00:13:44.893 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.893 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.893 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:44.893 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.011203+0000", 2020-01-16T00:13:44.893 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018067916999999999 2020-01-16T00:13:44.893 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.894 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:44.894 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.894 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.894 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.894 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2324 3.7 3:e1be6d3b:::smithi12313934-24:head [setxattr _header (56) in=7b,truncate 2336206] snapc 0=[] ondisk+write+known_if_redirected e36)", 2020-01-16T00:13:44.894 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.550846+0000", 2020-01-16T00:13:44.894 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 37.204980335999998, 2020-01-16T00:13:44.895 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018364742, 2020-01-16T00:13:44.895 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:44.895 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:44.895 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:44.895 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:44.895 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:44.895 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2324 2020-01-16T00:13:44.895 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.896 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:44.896 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.896 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:44.896 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550846+0000", 2020-01-16T00:13:44.896 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:44.896 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.896 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.896 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:44.897 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550846+0000", 2020-01-16T00:13:44.897 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:44.897 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.897 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.897 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:44.897 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550845+0000", 2020-01-16T00:13:44.897 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1829999999999999e-06 2020-01-16T00:13:44.898 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.898 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.898 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:44.898 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550851+0000", 2020-01-16T00:13:44.898 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1925e-05 2020-01-16T00:13:44.898 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.898 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.898 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:44.899 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550863+0000", 2020-01-16T00:13:44.899 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1139999999999997e-06 2020-01-16T00:13:44.899 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.899 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.900 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:44.900 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550868+0000", 2020-01-16T00:13:44.900 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001388423 2020-01-16T00:13:44.901 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.901 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.901 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.901 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.552256+0000", 2020-01-16T00:13:44.901 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0034246540000000001 2020-01-16T00:13:44.901 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.902 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.902 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:44.902 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555681+0000", 2020-01-16T00:13:44.902 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011882762999999999 2020-01-16T00:13:44.902 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.902 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.903 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.903 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.567564+0000", 2020-01-16T00:13:44.903 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015010800000000001 2020-01-16T00:13:44.904 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.905 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.905 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:44.905 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.567714+0000", 2020-01-16T00:13:44.905 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011527999999999999 2020-01-16T00:13:44.905 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.905 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.905 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,7", 2020-01-16T00:13:44.905 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.567829+0000", 2020-01-16T00:13:44.906 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00039888499999999997 2020-01-16T00:13:44.906 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.906 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.906 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:44.907 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568228+0000", 2020-01-16T00:13:44.907 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00054254599999999998 2020-01-16T00:13:44.907 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.907 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.907 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:44.907 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568771+0000", 2020-01-16T00:13:44.907 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035206400000000002 2020-01-16T00:13:44.908 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.908 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.908 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:44.908 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569123+0000", 2020-01-16T00:13:44.908 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1262e-05 2020-01-16T00:13:44.908 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.908 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.908 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:44.909 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569154+0000", 2020-01-16T00:13:44.909 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6499000000000002e-05 2020-01-16T00:13:44.909 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.909 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.909 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:44.909 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569210+0000", 2020-01-16T00:13:44.909 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018364742 2020-01-16T00:13:44.910 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.910 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:44.910 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.910 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.910 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.910 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2325 3.7 3:e1be6d3b:::smithi12313934-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e36)", 2020-01-16T00:13:44.910 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.550877+0000", 2020-01-16T00:13:44.910 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 37.20494927, 2020-01-16T00:13:44.911 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018565154, 2020-01-16T00:13:44.911 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:44.911 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:44.911 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:44.911 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:44.911 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:44.912 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2325 2020-01-16T00:13:44.912 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.912 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:44.912 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.912 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:44.912 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550877+0000", 2020-01-16T00:13:44.912 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:44.913 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.913 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.913 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:44.913 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550877+0000", 2020-01-16T00:13:44.913 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:44.913 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.914 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.914 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:44.914 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550876+0000", 2020-01-16T00:13:44.914 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1080000000000003e-06 2020-01-16T00:13:44.914 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.914 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.914 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:44.914 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550880+0000", 2020-01-16T00:13:44.915 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3269999999999997e-06 2020-01-16T00:13:44.915 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.915 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.915 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:44.915 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550886+0000", 2020-01-16T00:13:44.915 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.386e-06 2020-01-16T00:13:44.916 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.916 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.916 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:44.916 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550891+0000", 2020-01-16T00:13:44.916 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048231410000000004 2020-01-16T00:13:44.916 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.916 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.917 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.917 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555714+0000", 2020-01-16T00:13:44.917 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0071999999999999e-05 2020-01-16T00:13:44.917 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.917 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.917 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:44.918 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555744+0000", 2020-01-16T00:13:44.918 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012329005000000001 2020-01-16T00:13:44.918 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.918 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.918 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.918 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568073+0000", 2020-01-16T00:13:44.919 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9087999999999997e-05 2020-01-16T00:13:44.919 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.919 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.919 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:44.919 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568172+0000", 2020-01-16T00:13:44.919 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010718220000000001 2020-01-16T00:13:44.919 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.920 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.920 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.920 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569244+0000", 2020-01-16T00:13:44.920 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8199e-05 2020-01-16T00:13:44.920 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.920 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.920 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:44.920 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569342+0000", 2020-01-16T00:13:44.921 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9708000000000001e-05 2020-01-16T00:13:44.921 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.921 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.921 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:44.921 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569442+0000", 2020-01-16T00:13:44.921 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018565154 2020-01-16T00:13:44.921 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.922 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:44.922 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.922 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.922 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.922 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2800 3.2 3:42b7f20f:::smithi12313934-42:head [write 1395221~583781] snapc 0=[] ondisk+write+known_if_redirected e41)", 2020-01-16T00:13:44.922 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:17.256182+0000", 2020-01-16T00:13:44.922 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 27.499643729999999, 2020-01-16T00:13:44.922 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018509650999999998, 2020-01-16T00:13:44.923 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:44.923 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:44.923 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:44.923 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:44.923 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:44.923 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2800 2020-01-16T00:13:44.923 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.923 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:44.924 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.924 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:44.924 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256182+0000", 2020-01-16T00:13:44.924 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:44.924 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.924 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.924 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:44.924 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256182+0000", 2020-01-16T00:13:44.925 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:44.925 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.925 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.925 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:44.925 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256180+0000", 2020-01-16T00:13:44.925 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020514799999999999 2020-01-16T00:13:44.925 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.926 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.926 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:44.926 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256385+0000", 2020-01-16T00:13:44.926 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0288e-05 2020-01-16T00:13:44.926 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.926 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.927 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:44.927 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256406+0000", 2020-01-16T00:13:44.927 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7160000000000003e-06 2020-01-16T00:13:44.927 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.927 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.927 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:44.927 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256413+0000", 2020-01-16T00:13:44.927 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5293999999999998e-05 2020-01-16T00:13:44.928 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.928 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.928 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.928 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256449+0000", 2020-01-16T00:13:44.928 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001297737 2020-01-16T00:13:44.928 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.928 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.928 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:44.929 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.257746+0000", 2020-01-16T00:13:44.929 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0079866529999999998 2020-01-16T00:13:44.929 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.929 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.929 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.929 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.265733+0000", 2020-01-16T00:13:44.929 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4028000000000004e-05 2020-01-16T00:13:44.930 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.930 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.930 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:44.930 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.265797+0000", 2020-01-16T00:13:44.930 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011338899999999999 2020-01-16T00:13:44.930 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.930 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.930 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:44.931 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.265910+0000", 2020-01-16T00:13:44.931 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003372496 2020-01-16T00:13:44.931 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.931 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.931 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:44.931 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.269283+0000", 2020-01-16T00:13:44.931 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0053107959999999996 2020-01-16T00:13:44.931 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.932 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.932 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:44.932 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.274594+0000", 2020-01-16T00:13:44.932 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8000999999999998e-05 2020-01-16T00:13:44.932 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.932 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.932 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:44.932 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.274632+0000", 2020-01-16T00:13:44.933 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0265e-05 2020-01-16T00:13:44.933 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.933 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.933 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:44.933 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.274692+0000", 2020-01-16T00:13:44.933 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018509650999999998 2020-01-16T00:13:44.933 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.934 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:44.934 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.934 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.934 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.934 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2971 3.c 3:3025546b:::smithi12313934-15:head [write 1424565~609148] snapc 0=[] ondisk+write+known_if_redirected e42)", 2020-01-16T00:13:44.934 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:21.839711+0000", 2020-01-16T00:13:44.934 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 22.916114886999999, 2020-01-16T00:13:44.934 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017534376000000001, 2020-01-16T00:13:44.935 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:44.935 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:44.935 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:44.935 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:44.935 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:44.935 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2971 2020-01-16T00:13:44.935 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.935 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:44.936 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.936 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:44.936 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839711+0000", 2020-01-16T00:13:44.936 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:44.936 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.936 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.936 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:44.936 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839711+0000", 2020-01-16T00:13:44.937 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:44.937 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.937 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.937 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:44.937 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839710+0000", 2020-01-16T00:13:44.937 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017658600000000001 2020-01-16T00:13:44.937 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.938 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.938 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:44.938 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839886+0000", 2020-01-16T00:13:44.938 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0492000000000001e-05 2020-01-16T00:13:44.938 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.938 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.939 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:44.939 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839897+0000", 2020-01-16T00:13:44.939 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.019e-06 2020-01-16T00:13:44.939 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.939 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.939 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:44.939 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839901+0000", 2020-01-16T00:13:44.939 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5572000000000001e-05 2020-01-16T00:13:44.940 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.940 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.940 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.940 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839976+0000", 2020-01-16T00:13:44.940 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034616099999999999 2020-01-16T00:13:44.940 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.940 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.940 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:44.941 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.840323+0000", 2020-01-16T00:13:44.941 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.01246249 2020-01-16T00:13:44.941 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.941 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.941 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.941 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.852785+0000", 2020-01-16T00:13:44.941 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014474799999999999 2020-01-16T00:13:44.942 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.942 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.942 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:44.942 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.852930+0000", 2020-01-16T00:13:44.942 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000213907 2020-01-16T00:13:44.942 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.942 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.942 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:44.943 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.853144+0000", 2020-01-16T00:13:44.943 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0028426979999999998 2020-01-16T00:13:44.943 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.943 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.943 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:44.943 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.855986+0000", 2020-01-16T00:13:44.943 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010694280000000001 2020-01-16T00:13:44.943 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.944 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.944 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:44.944 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.857056+0000", 2020-01-16T00:13:44.944 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8935000000000003e-05 2020-01-16T00:13:44.944 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.944 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.944 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:44.945 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.857125+0000", 2020-01-16T00:13:44.945 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000120721 2020-01-16T00:13:44.945 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.945 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.945 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:44.945 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.857245+0000", 2020-01-16T00:13:44.946 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017534376000000001 2020-01-16T00:13:44.946 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.946 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:44.947 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.947 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.947 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.947 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2972 3.c 3:3025546b:::smithi12313934-15:head [setxattr _header (56) in=7b,truncate 2033713] snapc 0=[] ondisk+write+known_if_redirected e42)", 2020-01-16T00:13:44.947 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:21.839923+0000", 2020-01-16T00:13:44.947 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 22.915902623000001, 2020-01-16T00:13:44.947 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018367108, 2020-01-16T00:13:44.948 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:44.948 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:44.948 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:44.948 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:44.948 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:44.948 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2972 2020-01-16T00:13:44.948 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.948 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:44.949 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.949 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:44.949 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839923+0000", 2020-01-16T00:13:44.949 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:44.949 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.949 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.949 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:44.950 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839923+0000", 2020-01-16T00:13:44.950 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:44.950 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.950 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.950 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:44.950 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839923+0000", 2020-01-16T00:13:44.950 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0629999999999999e-06 2020-01-16T00:13:44.950 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.951 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.951 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:44.951 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839927+0000", 2020-01-16T00:13:44.951 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1980000000000003e-06 2020-01-16T00:13:44.951 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.951 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.951 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:44.952 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839931+0000", 2020-01-16T00:13:44.952 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5979999999999999e-06 2020-01-16T00:13:44.952 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.952 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.952 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:44.952 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839934+0000", 2020-01-16T00:13:44.952 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00042258500000000001 2020-01-16T00:13:44.952 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.953 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.953 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.840356+0000", 2020-01-16T00:13:44.953 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4919000000000002e-05 2020-01-16T00:13:44.953 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.953 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:44.953 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.840391+0000", 2020-01-16T00:13:44.954 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013783103 2020-01-16T00:13:44.954 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.954 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.954 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.954 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.854174+0000", 2020-01-16T00:13:44.954 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3455999999999995e-05 2020-01-16T00:13:44.954 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.954 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.955 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:44.955 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.854258+0000", 2020-01-16T00:13:44.955 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011060700000000001 2020-01-16T00:13:44.955 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.955 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.955 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:44.955 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.854368+0000", 2020-01-16T00:13:44.956 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020642500000000001 2020-01-16T00:13:44.956 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.956 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.956 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:44.956 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.856433+0000", 2020-01-16T00:13:44.956 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0016284820000000001 2020-01-16T00:13:44.956 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.957 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.957 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:44.957 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858061+0000", 2020-01-16T00:13:44.957 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0663999999999998e-05 2020-01-16T00:13:44.957 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.957 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.957 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:44.958 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858152+0000", 2020-01-16T00:13:44.958 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013882900000000001 2020-01-16T00:13:44.958 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.958 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.958 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:44.958 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858290+0000", 2020-01-16T00:13:44.958 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018367108 2020-01-16T00:13:44.959 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.959 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:44.959 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.959 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.959 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.959 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2973 3.c 3:3025546b:::smithi12313934-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e42)", 2020-01-16T00:13:44.959 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:21.839961+0000", 2020-01-16T00:13:44.959 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 22.915865129, 2020-01-16T00:13:44.960 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018691670000000001, 2020-01-16T00:13:44.960 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:44.960 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:44.960 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:44.960 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:44.960 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:44.960 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2973 2020-01-16T00:13:44.960 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.961 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:44.961 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.961 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:44.961 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839961+0000", 2020-01-16T00:13:44.961 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:44.961 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.961 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.961 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:44.962 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839961+0000", 2020-01-16T00:13:44.962 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:44.962 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.962 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.962 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:44.962 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839960+0000", 2020-01-16T00:13:44.962 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4060000000000001e-06 2020-01-16T00:13:44.962 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.963 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.963 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:44.963 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839964+0000", 2020-01-16T00:13:44.963 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5412000000000001e-05 2020-01-16T00:13:44.963 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.963 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.963 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:44.963 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839979+0000", 2020-01-16T00:13:44.964 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.444e-06 2020-01-16T00:13:44.964 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.964 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.964 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:44.964 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839983+0000", 2020-01-16T00:13:44.964 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000430028 2020-01-16T00:13:44.964 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.964 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.965 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.965 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.840413+0000", 2020-01-16T00:13:44.965 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9270999999999997e-05 2020-01-16T00:13:44.965 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.965 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.965 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:44.965 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.840452+0000", 2020-01-16T00:13:44.966 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014853975 2020-01-16T00:13:44.966 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.966 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.966 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.966 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.855306+0000", 2020-01-16T00:13:44.966 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011098400000000001 2020-01-16T00:13:44.966 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.966 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.967 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:44.967 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.855417+0000", 2020-01-16T00:13:44.967 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029108060000000002 2020-01-16T00:13:44.967 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.967 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.967 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.967 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858328+0000", 2020-01-16T00:13:44.967 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000144697 2020-01-16T00:13:44.968 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.968 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.968 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:44.968 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858472+0000", 2020-01-16T00:13:44.968 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018011899999999999 2020-01-16T00:13:44.968 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.968 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.968 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:44.969 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858653+0000", 2020-01-16T00:13:44.969 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018691670000000001 2020-01-16T00:13:44.969 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.969 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:44.969 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.969 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.969 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.970 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3295 3.2 3:42b7f20f:::smithi12313934-42:head [write 1982691~499457] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:13:44.970 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:30.744201+0000", 2020-01-16T00:13:44.970 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 14.011624853000001, 2020-01-16T00:13:44.970 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018346738000000001, 2020-01-16T00:13:44.970 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:44.970 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:44.971 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:44.971 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:44.971 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:44.971 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3295 2020-01-16T00:13:44.971 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.971 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:44.971 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.971 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:44.972 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744201+0000", 2020-01-16T00:13:44.972 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:44.972 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.972 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.972 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:44.972 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744201+0000", 2020-01-16T00:13:44.972 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:44.973 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.973 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.973 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:44.973 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744198+0000", 2020-01-16T00:13:44.973 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00030375699999999999 2020-01-16T00:13:44.973 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.973 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.973 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:44.974 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744502+0000", 2020-01-16T00:13:44.974 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0121e-05 2020-01-16T00:13:44.974 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.974 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.974 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:44.974 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744512+0000", 2020-01-16T00:13:44.974 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2293000000000001e-05 2020-01-16T00:13:44.974 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.975 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.975 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:44.975 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744524+0000", 2020-01-16T00:13:44.975 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000274837 2020-01-16T00:13:44.975 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.975 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.975 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.976 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744799+0000", 2020-01-16T00:13:44.976 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036951879999999999 2020-01-16T00:13:44.976 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.976 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.976 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:44.976 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748494+0000", 2020-01-16T00:13:44.976 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010091493 2020-01-16T00:13:44.977 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.977 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.977 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.977 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.758586+0000", 2020-01-16T00:13:44.977 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011674400000000001 2020-01-16T00:13:44.977 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.977 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.978 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:44.978 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.758703+0000", 2020-01-16T00:13:44.978 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021412699999999999 2020-01-16T00:13:44.978 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.978 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.978 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:44.978 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.758917+0000", 2020-01-16T00:13:44.979 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0022519340000000001 2020-01-16T00:13:44.979 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.979 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.979 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:44.979 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.761169+0000", 2020-01-16T00:13:44.980 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001240891 2020-01-16T00:13:44.980 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.980 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.980 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:44.980 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.762410+0000", 2020-01-16T00:13:44.980 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6942e-05 2020-01-16T00:13:44.980 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.981 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.981 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:44.981 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.762447+0000", 2020-01-16T00:13:44.981 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000101358 2020-01-16T00:13:44.981 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.981 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.981 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:44.981 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.762548+0000", 2020-01-16T00:13:44.982 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018346738000000001 2020-01-16T00:13:44.982 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.982 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:44.982 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.982 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.982 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.982 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3296 3.2 3:42b7f20f:::smithi12313934-42:head [setxattr _header (56) in=7b,truncate 2739607] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:13:44.982 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:30.744532+0000", 2020-01-16T00:13:44.983 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 14.011294120000001, 2020-01-16T00:13:44.983 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018472374, 2020-01-16T00:13:44.983 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:44.983 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:44.983 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:44.983 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:44.983 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:44.983 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3296 2020-01-16T00:13:44.984 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.984 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:44.984 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.984 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:44.984 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744532+0000", 2020-01-16T00:13:44.984 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:44.984 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.985 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.985 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:44.985 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744532+0000", 2020-01-16T00:13:44.985 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:44.985 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.985 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.985 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:44.985 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744531+0000", 2020-01-16T00:13:44.986 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7059999999999993e-06 2020-01-16T00:13:44.986 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.986 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.986 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:44.986 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744540+0000", 2020-01-16T00:13:44.986 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8589999999999998e-06 2020-01-16T00:13:44.986 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.986 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.987 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:44.987 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744544+0000", 2020-01-16T00:13:44.987 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7609999999999999e-06 2020-01-16T00:13:44.987 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.987 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.987 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:44.988 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744546+0000", 2020-01-16T00:13:44.988 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039933670000000003 2020-01-16T00:13:44.988 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.988 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.988 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.988 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748540+0000", 2020-01-16T00:13:44.988 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000110281 2020-01-16T00:13:44.988 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.989 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.989 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:44.989 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748650+0000", 2020-01-16T00:13:44.989 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.01160457 2020-01-16T00:13:44.989 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.989 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.989 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:44.990 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.760255+0000", 2020-01-16T00:13:44.990 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000124714 2020-01-16T00:13:44.990 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.990 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.990 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:44.990 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.760379+0000", 2020-01-16T00:13:44.990 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022099700000000001 2020-01-16T00:13:44.990 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.991 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.991 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:44.991 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.760600+0000", 2020-01-16T00:13:44.991 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00057843300000000005 2020-01-16T00:13:44.991 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.991 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.991 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:44.991 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.761179+0000", 2020-01-16T00:13:44.992 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00161477 2020-01-16T00:13:44.992 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.992 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.992 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:44.992 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.762794+0000", 2020-01-16T00:13:44.992 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4408999999999999e-05 2020-01-16T00:13:44.992 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.993 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.993 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:44.993 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.762858+0000", 2020-01-16T00:13:44.993 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014634199999999999 2020-01-16T00:13:44.993 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.993 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.993 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:44.993 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.763004+0000", 2020-01-16T00:13:44.994 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018472374 2020-01-16T00:13:44.994 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.994 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:44.994 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:44.994 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.994 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.994 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3297 3.2 3:42b7f20f:::smithi12313934-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e45)", 2020-01-16T00:13:44.994 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:30.744551+0000", 2020-01-16T00:13:44.995 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 14.011275149999999, 2020-01-16T00:13:44.995 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018660209000000001, 2020-01-16T00:13:44.995 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:44.995 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:44.995 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:44.995 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:44.995 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:44.996 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3297 2020-01-16T00:13:44.996 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.996 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:44.996 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.996 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:44.996 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744551+0000", 2020-01-16T00:13:44.996 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:44.996 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.997 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.997 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:44.997 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744551+0000", 2020-01-16T00:13:44.997 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:44.997 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.997 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.998 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:44.998 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744550+0000", 2020-01-16T00:13:44.998 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5279999999999999e-06 2020-01-16T00:13:44.998 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.998 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.998 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:44.998 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744554+0000", 2020-01-16T00:13:44.998 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8420000000000001e-06 2020-01-16T00:13:44.999 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.999 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.999 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:44.999 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744559+0000", 2020-01-16T00:13:44.999 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.965e-06 2020-01-16T00:13:44.999 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:44.999 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:44.999 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:45.000 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744562+0000", 2020-01-16T00:13:45.000 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041215380000000001 2020-01-16T00:13:45.000 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.000 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.000 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:45.000 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748684+0000", 2020-01-16T00:13:45.000 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8165999999999997e-05 2020-01-16T00:13:45.000 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.001 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.001 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:45.001 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748762+0000", 2020-01-16T00:13:45.001 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012136152000000001 2020-01-16T00:13:45.001 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.001 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.001 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:45.002 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.760898+0000", 2020-01-16T00:13:45.002 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000114833 2020-01-16T00:13:45.002 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.002 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.002 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:45.002 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.761013+0000", 2020-01-16T00:13:45.002 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002017015 2020-01-16T00:13:45.002 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.003 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.003 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:45.003 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.763030+0000", 2020-01-16T00:13:45.003 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5741000000000006e-05 2020-01-16T00:13:45.003 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.003 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.003 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:45.003 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.763096+0000", 2020-01-16T00:13:45.004 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011519900000000001 2020-01-16T00:13:45.004 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.004 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.004 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:45.004 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.763211+0000", 2020-01-16T00:13:45.004 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018660209000000001 2020-01-16T00:13:45.004 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.005 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:45.005 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.005 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.005 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.005 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3481 3.2 3:4f8fe446:::smithi12313934-48:head [setxattr _header (56) in=7b,truncate 3193396] snapc 0=[] ondisk+write+known_if_redirected e47)", 2020-01-16T00:13:45.005 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:36.349654+0000", 2020-01-16T00:13:45.005 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 8.4061716969999996, 2020-01-16T00:13:45.005 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017529643000000001, 2020-01-16T00:13:45.006 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:45.006 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:45.006 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:45.006 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:45.006 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:45.006 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3481 2020-01-16T00:13:45.006 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.006 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:45.007 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.007 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:45.007 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349654+0000", 2020-01-16T00:13:45.007 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:45.007 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.007 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.007 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:45.008 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349654+0000", 2020-01-16T00:13:45.008 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:45.008 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.008 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.008 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:45.008 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349654+0000", 2020-01-16T00:13:45.009 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2519999999999996e-06 2020-01-16T00:13:45.009 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.009 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.009 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:45.009 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349660+0000", 2020-01-16T00:13:45.009 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2789999999999999e-06 2020-01-16T00:13:45.009 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.010 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.010 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:45.010 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349663+0000", 2020-01-16T00:13:45.010 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2479999999999999e-06 2020-01-16T00:13:45.010 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.010 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.010 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:45.010 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349666+0000", 2020-01-16T00:13:45.011 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0026359180000000001 2020-01-16T00:13:45.011 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.011 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.011 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:45.011 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.352301+0000", 2020-01-16T00:13:45.011 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0035532799999999998 2020-01-16T00:13:45.011 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.011 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.012 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:45.012 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.355855+0000", 2020-01-16T00:13:45.012 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0099904050000000008 2020-01-16T00:13:45.012 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.012 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.012 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:45.012 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.365845+0000", 2020-01-16T00:13:45.012 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012439300000000001 2020-01-16T00:13:45.013 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.013 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.013 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:45.013 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.365970+0000", 2020-01-16T00:13:45.013 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00026105699999999998 2020-01-16T00:13:45.013 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.013 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.013 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:45.014 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.366231+0000", 2020-01-16T00:13:45.014 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00056610199999999997 2020-01-16T00:13:45.014 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.014 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.014 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:45.014 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.366797+0000", 2020-01-16T00:13:45.014 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000171996 2020-01-16T00:13:45.014 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.015 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.015 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:45.015 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.366969+0000", 2020-01-16T00:13:45.015 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5282000000000003e-05 2020-01-16T00:13:45.015 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.015 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.015 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:45.016 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.367034+0000", 2020-01-16T00:13:45.016 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014998100000000001 2020-01-16T00:13:45.016 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.016 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.016 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:45.016 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.367184+0000", 2020-01-16T00:13:45.016 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017529643000000001 2020-01-16T00:13:45.016 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.017 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:45.017 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.017 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.017 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.017 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3482 3.2 3:4f8fe446:::smithi12313934-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e47)", 2020-01-16T00:13:45.017 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:36.349724+0000", 2020-01-16T00:13:45.017 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 8.4061020600000003, 2020-01-16T00:13:45.018 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017836446999999998, 2020-01-16T00:13:45.018 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:45.018 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:45.018 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:45.018 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:45.018 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:45.018 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3482 2020-01-16T00:13:45.018 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.019 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:45.019 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.019 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:45.019 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349724+0000", 2020-01-16T00:13:45.019 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:45.019 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.019 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.019 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:45.020 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349724+0000", 2020-01-16T00:13:45.020 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:45.020 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.020 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.020 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:45.020 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349723+0000", 2020-01-16T00:13:45.020 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9709999999999998e-06 2020-01-16T00:13:45.020 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.021 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.021 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:45.021 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349727+0000", 2020-01-16T00:13:45.022 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5419999999999999e-06 2020-01-16T00:13:45.022 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.022 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.022 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:45.022 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349732+0000", 2020-01-16T00:13:45.022 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.481e-06 2020-01-16T00:13:45.022 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.023 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.023 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:45.023 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349734+0000", 2020-01-16T00:13:45.023 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.006195028 2020-01-16T00:13:45.023 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.023 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.023 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:45.024 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.355929+0000", 2020-01-16T00:13:45.024 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1483999999999996e-05 2020-01-16T00:13:45.024 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.024 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.024 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:45.024 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.355991+0000", 2020-01-16T00:13:45.024 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010536415 2020-01-16T00:13:45.024 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.025 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.025 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:45.025 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.366527+0000", 2020-01-16T00:13:45.025 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8099e-05 2020-01-16T00:13:45.025 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.025 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.025 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:45.026 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.366595+0000", 2020-01-16T00:13:45.026 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00063624500000000002 2020-01-16T00:13:45.026 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.026 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.026 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:45.026 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.367231+0000", 2020-01-16T00:13:45.026 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000121196 2020-01-16T00:13:45.026 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.027 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.027 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:45.027 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.367353+0000", 2020-01-16T00:13:45.027 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020780500000000001 2020-01-16T00:13:45.027 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.027 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.027 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:45.028 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.367560+0000", 2020-01-16T00:13:45.028 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017836446999999998 2020-01-16T00:13:45.028 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.028 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:45.028 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.028 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.028 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.028 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3810 3.7 3:efab040c:::smithi12313934-30:head [write 1837406~542493] snapc 0=[] ondisk+write+known_if_redirected e50)", 2020-01-16T00:13:45.029 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:43.768525+0000", 2020-01-16T00:13:45.029 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.98730097699999997, 2020-01-16T00:13:45.029 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018385839000000001, 2020-01-16T00:13:45.029 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:45.029 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:45.029 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:45.030 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:45.030 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:45.030 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3810 2020-01-16T00:13:45.030 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.030 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:45.030 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.030 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:45.031 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768525+0000", 2020-01-16T00:13:45.031 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:45.031 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.031 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.032 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:45.032 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768525+0000", 2020-01-16T00:13:45.032 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:45.032 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.032 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.032 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:45.032 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768523+0000", 2020-01-16T00:13:45.032 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000149512 2020-01-16T00:13:45.033 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.033 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.033 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:45.033 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768673+0000", 2020-01-16T00:13:45.033 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0732e-05 2020-01-16T00:13:45.033 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.033 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.034 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:45.034 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768684+0000", 2020-01-16T00:13:45.034 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3719999999999998e-06 2020-01-16T00:13:45.034 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.034 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.034 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:45.034 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768688+0000", 2020-01-16T00:13:45.034 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6513999999999994e-05 2020-01-16T00:13:45.035 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.035 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.035 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:45.035 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768754+0000", 2020-01-16T00:13:45.035 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00075867800000000004 2020-01-16T00:13:45.035 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.035 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.035 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:45.036 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.769513+0000", 2020-01-16T00:13:45.036 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012395692 2020-01-16T00:13:45.036 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.036 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.036 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:45.036 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.781909+0000", 2020-01-16T00:13:45.036 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118372 2020-01-16T00:13:45.037 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.037 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.037 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:45.037 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.782027+0000", 2020-01-16T00:13:45.037 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019909699999999999 2020-01-16T00:13:45.037 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.037 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.037 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,7", 2020-01-16T00:13:45.038 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.782226+0000", 2020-01-16T00:13:45.038 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0038778369999999999 2020-01-16T00:13:45.038 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.038 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.038 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:45.038 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786104+0000", 2020-01-16T00:13:45.038 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022391199999999999 2020-01-16T00:13:45.038 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.040 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.040 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:45.040 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786328+0000", 2020-01-16T00:13:45.040 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00048851299999999999 2020-01-16T00:13:45.040 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.040 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.041 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:45.041 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786817+0000", 2020-01-16T00:13:45.041 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3597999999999998e-05 2020-01-16T00:13:45.041 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.041 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.041 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:45.041 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786860+0000", 2020-01-16T00:13:45.041 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0741000000000001e-05 2020-01-16T00:13:45.042 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.042 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.042 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:45.042 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786911+0000", 2020-01-16T00:13:45.042 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018385839000000001 2020-01-16T00:13:45.042 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.042 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:45.043 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.043 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.043 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.043 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3863 3.2 3:4f8fe446:::smithi12313934-48:head [setxattr _header (56) in=7b,truncate 2151395] snapc 0=[] ondisk+write+known_if_redirected e50)", 2020-01-16T00:13:45.043 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:44.489284+0000", 2020-01-16T00:13:45.043 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.26654250499999999, 2020-01-16T00:13:45.043 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.026323742000000001, 2020-01-16T00:13:45.044 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:45.044 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:45.044 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:45.044 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:45.044 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:45.044 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3863 2020-01-16T00:13:45.044 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.045 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:45.045 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.045 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:45.045 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489284+0000", 2020-01-16T00:13:45.045 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:45.045 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.045 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.046 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:45.046 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489284+0000", 2020-01-16T00:13:45.046 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:45.046 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.046 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.046 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:45.046 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489282+0000", 2020-01-16T00:13:45.047 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6660000000000002e-06 2020-01-16T00:13:45.047 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.047 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.047 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:45.047 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489289+0000", 2020-01-16T00:13:45.047 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3069999999999999e-06 2020-01-16T00:13:45.047 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.047 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.048 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:45.048 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489295+0000", 2020-01-16T00:13:45.048 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6090000000000003e-06 2020-01-16T00:13:45.048 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.048 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.048 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:45.048 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489300+0000", 2020-01-16T00:13:45.049 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002242464 2020-01-16T00:13:45.049 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.049 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.049 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:45.049 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491542+0000", 2020-01-16T00:13:45.050 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00059546200000000005 2020-01-16T00:13:45.050 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.050 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.050 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:45.051 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.492138+0000", 2020-01-16T00:13:45.051 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018547999999999999 2020-01-16T00:13:45.051 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.051 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.051 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:45.052 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.510686+0000", 2020-01-16T00:13:45.052 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018590100000000001 2020-01-16T00:13:45.052 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.052 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.052 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:45.052 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.510872+0000", 2020-01-16T00:13:45.053 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035115299999999999 2020-01-16T00:13:45.053 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.053 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.053 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:45.053 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.511223+0000", 2020-01-16T00:13:45.053 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003765493 2020-01-16T00:13:45.053 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.054 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.054 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:45.054 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.514988+0000", 2020-01-16T00:13:45.054 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000432022 2020-01-16T00:13:45.054 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.054 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.054 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:45.055 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515421+0000", 2020-01-16T00:13:45.055 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2874000000000002e-05 2020-01-16T00:13:45.055 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.055 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.055 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:45.055 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515473+0000", 2020-01-16T00:13:45.055 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013385 2020-01-16T00:13:45.055 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.056 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.056 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:45.056 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515607+0000", 2020-01-16T00:13:45.056 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.026323742000000001 2020-01-16T00:13:45.056 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.056 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:45.056 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.057 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.057 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.057 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3864 3.2 3:4f8fe446:::smithi12313934-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e50)", 2020-01-16T00:13:45.057 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:44.491131+0000", 2020-01-16T00:13:45.057 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.26469510699999999, 2020-01-16T00:13:45.057 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024813806000000001, 2020-01-16T00:13:45.057 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:45.057 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:45.058 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:45.058 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:45.058 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:45.058 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3864 2020-01-16T00:13:45.058 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.058 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:45.058 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.059 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:45.059 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491131+0000", 2020-01-16T00:13:45.059 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:45.059 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.059 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.059 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:45.059 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491131+0000", 2020-01-16T00:13:45.060 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:45.060 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.060 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.060 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:45.060 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491128+0000", 2020-01-16T00:13:45.061 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2053e-05 2020-01-16T00:13:45.061 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.061 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.061 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:45.061 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491140+0000", 2020-01-16T00:13:45.061 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5869999999999999e-05 2020-01-16T00:13:45.061 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.062 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.062 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:45.062 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491156+0000", 2020-01-16T00:13:45.062 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2571e-05 2020-01-16T00:13:45.062 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.062 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.062 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:45.062 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491168+0000", 2020-01-16T00:13:45.063 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001031045 2020-01-16T00:13:45.063 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.063 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.063 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:45.063 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.492199+0000", 2020-01-16T00:13:45.063 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000170865 2020-01-16T00:13:45.063 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.064 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.064 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:45.064 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.492370+0000", 2020-01-16T00:13:45.064 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020180638000000001 2020-01-16T00:13:45.064 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.064 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.064 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:45.064 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.512551+0000", 2020-01-16T00:13:45.065 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012478599999999999 2020-01-16T00:13:45.065 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.065 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.065 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:45.065 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.512676+0000", 2020-01-16T00:13:45.065 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029628660000000002 2020-01-16T00:13:45.065 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.066 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.066 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:45.066 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515639+0000", 2020-01-16T00:13:45.066 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012250799999999999 2020-01-16T00:13:45.066 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.066 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.066 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:45.067 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515761+0000", 2020-01-16T00:13:45.067 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018356399999999999 2020-01-16T00:13:45.067 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.067 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.067 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:45.067 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515945+0000", 2020-01-16T00:13:45.067 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024813806000000001 2020-01-16T00:13:45.067 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.068 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:45.068 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.068 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.068 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.068 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3874 3.c 3:378f3722:::smithi12313934-50:head [write 568202~701253,stat] snapc 0=[] ondisk+write+known_if_redirected e50)", 2020-01-16T00:13:45.068 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:44.497041+0000", 2020-01-16T00:13:45.068 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.25878547600000001, 2020-01-16T00:13:45.069 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019602525999999999, 2020-01-16T00:13:45.069 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:45.069 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:45.069 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:45.069 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:45.069 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:45.069 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3874 2020-01-16T00:13:45.070 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.070 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:45.070 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.070 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:45.071 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497041+0000", 2020-01-16T00:13:45.071 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:45.071 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.071 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.071 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:45.071 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497041+0000", 2020-01-16T00:13:45.071 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999962 2020-01-16T00:13:45.072 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.072 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.072 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:45.072 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497037+0000", 2020-01-16T00:13:45.072 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00027168899999999998 2020-01-16T00:13:45.072 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.072 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.072 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:45.073 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497308+0000", 2020-01-16T00:13:45.073 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4202999999999999e-05 2020-01-16T00:13:45.073 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.073 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.073 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:45.073 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497323+0000", 2020-01-16T00:13:45.073 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0280000000000003e-06 2020-01-16T00:13:45.074 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.074 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.074 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:45.074 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497330+0000", 2020-01-16T00:13:45.074 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010681352 2020-01-16T00:13:45.074 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.074 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.074 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:45.075 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.508011+0000", 2020-01-16T00:13:45.075 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5886999999999999e-05 2020-01-16T00:13:45.075 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.075 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.075 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:45.075 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.508097+0000", 2020-01-16T00:13:45.075 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023353600000000001 2020-01-16T00:13:45.075 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.076 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.076 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:45.076 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.508331+0000", 2020-01-16T00:13:45.076 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0075218339999999998 2020-01-16T00:13:45.076 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.076 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.076 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:45.076 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515852+0000", 2020-01-16T00:13:45.077 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00064178599999999998 2020-01-16T00:13:45.077 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.077 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.077 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:45.077 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.516494+0000", 2020-01-16T00:13:45.077 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6712999999999999e-05 2020-01-16T00:13:45.077 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.078 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.078 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:45.078 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.516531+0000", 2020-01-16T00:13:45.078 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112225 2020-01-16T00:13:45.078 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.078 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.079 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:45.079 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.516643+0000", 2020-01-16T00:13:45.079 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019602525999999999 2020-01-16T00:13:45.079 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.079 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:45.079 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.079 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.079 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:45.080 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:45.080 INFO:tasks.rados.rados.0.smithi123.stdout:1464: writing smithi12313934-7 from 1623448 to 2075271 tid 2 2020-01-16T00:13:45.080 INFO:tasks.rados.rados.0.smithi123.stdout:1465: read oid 26 snap -1 2020-01-16T00:13:45.080 INFO:tasks.rados.rados.0.smithi123.stdout:1465: expect (ObjNum 440 snap 0 seq_num 440) 2020-01-16T00:13:45.081 INFO:tasks.rados.rados.0.smithi123.stdout:1466: read oid 45 snap -1 2020-01-16T00:13:45.081 INFO:tasks.rados.rados.0.smithi123.stdout:1466: expect (ObjNum 466 snap 0 seq_num 466) 2020-01-16T00:13:45.081 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:45.081 INFO:tasks.rados.rados.0.smithi123.stdout:1451: finishing write tid 5 to smithi12313934-11 2020-01-16T00:13:45.081 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 11 v 146 (ObjNum 484 snap 0 seq_num 484) dirty exists 2020-01-16T00:13:45.081 INFO:tasks.rados.rados.0.smithi123.stdout:1451: left oid 11 (ObjNum 484 snap 0 seq_num 484) 2020-01-16T00:13:45.082 INFO:tasks.rados.rados.0.smithi123.stdout:1451: done (15 left) 2020-01-16T00:13:45.082 INFO:tasks.rados.rados.0.smithi123.stdout:1467: read oid 21 snap -1 2020-01-16T00:13:45.082 INFO:tasks.rados.rados.0.smithi123.stdout:1467: expect deleted 2020-01-16T00:13:45.082 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:45.082 INFO:tasks.rados.rados.0.smithi123.stdout:1452: expect (ObjNum 477 snap 0 seq_num 477) 2020-01-16T00:13:45.083 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2020-01-16T00:13:45.085 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-01-16T00:13:45.085 INFO:tasks.rados.rados.0.smithi123.stdout:1454: finishing write tid 1 to smithi12313934-48 2020-01-16T00:13:45.086 INFO:tasks.rados.rados.0.smithi123.stdout:1455: expect (ObjNum 387 snap 0 seq_num 387) 2020-01-16T00:13:45.086 INFO:tasks.rados.rados.0.smithi123.stdout:1453: expect (ObjNum 462 snap 0 seq_num 462) 2020-01-16T00:13:45.086 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:45.082+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:45.184 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:45.182+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:45.231 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:45.231 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:45.231 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:13:45.232 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:45.241 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-01-16T00:13:45.285 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:45.280+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:45.326 INFO:tasks.rados.rados.0.smithi123.stdout:1454: finishing write tid 2 to smithi12313934-48 2020-01-16T00:13:45.326 INFO:tasks.rados.rados.0.smithi123.stdout:1452: done (15 left) 2020-01-16T00:13:45.326 INFO:tasks.rados.rados.0.smithi123.stdout:1453: done (14 left) 2020-01-16T00:13:45.327 INFO:tasks.rados.rados.0.smithi123.stdout:1455: done (13 left) 2020-01-16T00:13:45.327 INFO:tasks.rados.rados.0.smithi123.stdout:1468: read oid 6 snap -1 2020-01-16T00:13:45.327 INFO:tasks.rados.rados.0.smithi123.stdout:1468: expect (ObjNum 480 snap 0 seq_num 480) 2020-01-16T00:13:45.327 INFO:tasks.rados.rados.0.smithi123.stdout:1469: write oid 16 current snap is 0 2020-01-16T00:13:45.327 INFO:tasks.rados.rados.0.smithi123.stdout:1469: seq_num 489 ranges {618623=643353,1681930=536693} 2020-01-16T00:13:45.333 INFO:tasks.rados.rados.0.smithi123.stdout:1469: writing smithi12313934-16 from 618623 to 1261976 tid 1 2020-01-16T00:13:45.338 INFO:tasks.rados.rados.0.smithi123.stdout:1469: writing smithi12313934-16 from 1681930 to 2218623 tid 2 2020-01-16T00:13:45.338 INFO:tasks.rados.rados.0.smithi123.stdout:1470: write oid 43 current snap is 0 2020-01-16T00:13:45.339 INFO:tasks.rados.rados.0.smithi123.stdout:1470: seq_num 490 ranges {635430=718602,2140412=95018} 2020-01-16T00:13:45.346 INFO:tasks.rados.rados.0.smithi123.stdout:1470: writing smithi12313934-43 from 635430 to 1354032 tid 1 2020-01-16T00:13:45.351 INFO:tasks.rados.rados.0.smithi123.stdout:1470: writing smithi12313934-43 from 2140412 to 2235430 tid 2 2020-01-16T00:13:45.351 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:45.351 INFO:tasks.rados.rados.0.smithi123.stdout:1457: expect (ObjNum 330 snap 0 seq_num 330) 2020-01-16T00:13:45.386 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:45.382+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:45.394 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:45.394 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:45.394 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:13:45.395 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:13:45.395 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:45.401 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-01-16T00:13:45.487 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:45.484+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:45.548 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:45.549 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:13:45.549 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:13:45.549 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:13:45.549 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.549 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.4.0:114 3.9 e24/20 3:9fc687b2:::smithi12313934-44:head v 24'74, mlcod=24'72)", 2020-01-16T00:13:45.549 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:13:45.550 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 67.219442373000007, 2020-01-16T00:13:45.550 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010399429, 2020-01-16T00:13:45.550 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:45.550 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:45.550 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:45.550 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.550 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:45.551 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:13:45.551 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:45.551 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.551 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.551 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:45.551 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:13:45.551 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:45.551 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.552 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.552 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:45.552 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325276+0000", 2020-01-16T00:13:45.552 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0046127069999999997 2020-01-16T00:13:45.552 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.552 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.553 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:45.553 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329889+0000", 2020-01-16T00:13:45.553 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3519999999999998e-05 2020-01-16T00:13:45.553 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.553 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.554 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:45.554 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329912+0000", 2020-01-16T00:13:45.554 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4070000000000004e-06 2020-01-16T00:13:45.554 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.554 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.554 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:45.555 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329920+0000", 2020-01-16T00:13:45.555 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8817000000000003e-05 2020-01-16T00:13:45.555 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.555 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.555 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:45.555 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329978+0000", 2020-01-16T00:13:45.555 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9721e-05 2020-01-16T00:13:45.556 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.556 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.556 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:45.556 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329998+0000", 2020-01-16T00:13:45.556 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056328139999999999 2020-01-16T00:13:45.556 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.556 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.557 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:45.557 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.335631+0000", 2020-01-16T00:13:45.557 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7170999999999999e-05 2020-01-16T00:13:45.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.557 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.558 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:45.558 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.335678+0000", 2020-01-16T00:13:45.558 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010399429 2020-01-16T00:13:45.558 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.558 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:45.558 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.558 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.558 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.559 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:24 2.1 2:8b431f80:::smithi12313934-40 low_tier:head [write 790528~688128] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:13:45.559 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.482303+0000", 2020-01-16T00:13:45.559 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 67.062418277000006, 2020-01-16T00:13:45.559 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010071853, 2020-01-16T00:13:45.559 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:45.559 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:45.559 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:45.559 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.560 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:45.560 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.482303+0000", 2020-01-16T00:13:45.560 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:45.560 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.560 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.560 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:45.560 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.482303+0000", 2020-01-16T00:13:45.560 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:45.561 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.561 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.561 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:45.561 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.482301+0000", 2020-01-16T00:13:45.561 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031173960000000001 2020-01-16T00:13:45.561 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.561 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.562 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:45.562 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485418+0000", 2020-01-16T00:13:45.562 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9300000000000001e-05 2020-01-16T00:13:45.562 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.562 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.562 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:45.562 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485448+0000", 2020-01-16T00:13:45.563 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2611e-05 2020-01-16T00:13:45.563 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.563 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.563 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:45.563 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485460+0000", 2020-01-16T00:13:45.563 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112487 2020-01-16T00:13:45.563 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.563 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.564 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:45.564 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485573+0000", 2020-01-16T00:13:45.564 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022169299999999999 2020-01-16T00:13:45.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.564 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.564 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:45.564 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485794+0000", 2020-01-16T00:13:45.565 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00036287500000000002 2020-01-16T00:13:45.565 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.565 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.565 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:45.565 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.486157+0000", 2020-01-16T00:13:45.565 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040260310000000002 2020-01-16T00:13:45.565 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.566 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.566 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:45.566 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.490183+0000", 2020-01-16T00:13:45.566 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00081083000000000001 2020-01-16T00:13:45.566 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.566 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.566 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:45.567 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.490994+0000", 2020-01-16T00:13:45.567 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001287746 2020-01-16T00:13:45.567 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.567 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.567 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:45.567 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492282+0000", 2020-01-16T00:13:45.567 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8810999999999998e-05 2020-01-16T00:13:45.567 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.568 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.568 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:45.568 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492321+0000", 2020-01-16T00:13:45.568 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4095000000000003e-05 2020-01-16T00:13:45.568 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.568 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.573 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:45.573 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492375+0000", 2020-01-16T00:13:45.573 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010071853 2020-01-16T00:13:45.573 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.573 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:45.574 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.574 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.574 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.584 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:55 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e30)", 2020-01-16T00:13:45.585 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.736036+0000", 2020-01-16T00:13:45.585 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 52.808685154000003, 2020-01-16T00:13:45.585 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010974536, 2020-01-16T00:13:45.585 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:45.585 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:45.585 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:45.585 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.586 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:45.586 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736036+0000", 2020-01-16T00:13:45.586 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:45.586 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.587 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.587 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:45.587 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736036+0000", 2020-01-16T00:13:45.587 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:45.587 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.587 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.587 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:45.587 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736034+0000", 2020-01-16T00:13:45.588 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043589699999999998 2020-01-16T00:13:45.588 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.588 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.588 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:45.588 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736469+0000", 2020-01-16T00:13:45.588 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9882e-05 2020-01-16T00:13:45.588 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.589 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.589 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:45.589 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736489+0000", 2020-01-16T00:13:45.589 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8339999999999999e-06 2020-01-16T00:13:45.589 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.589 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.589 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:45.589 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736497+0000", 2020-01-16T00:13:45.590 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5354999999999999e-05 2020-01-16T00:13:45.590 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.624 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.624 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:45.625 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736572+0000", 2020-01-16T00:13:45.625 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5228000000000004e-05 2020-01-16T00:13:45.625 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.625 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.625 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:45.625 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736668+0000", 2020-01-16T00:13:45.625 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017212999999999999 2020-01-16T00:13:45.626 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.626 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.626 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:45.626 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736840+0000", 2020-01-16T00:13:45.626 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0099057020000000006 2020-01-16T00:13:45.626 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.626 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.626 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:45.627 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746746+0000", 2020-01-16T00:13:45.627 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0586000000000005e-05 2020-01-16T00:13:45.627 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.627 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.627 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:45.627 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746816+0000", 2020-01-16T00:13:45.627 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2445000000000002e-05 2020-01-16T00:13:45.628 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.628 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.628 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:45.628 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746889+0000", 2020-01-16T00:13:45.628 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6839000000000002e-05 2020-01-16T00:13:45.628 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.628 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.628 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:45.629 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746955+0000", 2020-01-16T00:13:45.629 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5074000000000002e-05 2020-01-16T00:13:45.629 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.629 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.629 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:45.629 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.747011+0000", 2020-01-16T00:13:45.629 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010974536 2020-01-16T00:13:45.629 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.630 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:45.630 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.630 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.630 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:56 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e30)", 2020-01-16T00:13:45.630 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.736514+0000", 2020-01-16T00:13:45.630 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 52.808207318999997, 2020-01-16T00:13:45.631 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011660584999999999, 2020-01-16T00:13:45.631 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:45.631 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:45.631 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:45.631 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.631 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:45.631 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736514+0000", 2020-01-16T00:13:45.631 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:45.632 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.632 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.632 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:45.632 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736514+0000", 2020-01-16T00:13:45.632 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:45.632 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.632 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.632 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:45.633 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736513+0000", 2020-01-16T00:13:45.633 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9677000000000004e-05 2020-01-16T00:13:45.633 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.633 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.633 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:45.633 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736583+0000", 2020-01-16T00:13:45.633 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5269999999999998e-06 2020-01-16T00:13:45.642 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.642 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.642 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:45.642 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736591+0000", 2020-01-16T00:13:45.642 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5769999999999996e-06 2020-01-16T00:13:45.642 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.643 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.643 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:45.643 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736599+0000", 2020-01-16T00:13:45.643 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0053593310000000002 2020-01-16T00:13:45.643 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.643 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.643 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:45.644 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.741958+0000", 2020-01-16T00:13:45.644 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014164099999999999 2020-01-16T00:13:45.644 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.644 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.644 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:45.644 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.742100+0000", 2020-01-16T00:13:45.645 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020126000000000001 2020-01-16T00:13:45.645 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.645 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.645 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:45.645 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.742301+0000", 2020-01-16T00:13:45.645 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0045052859999999998 2020-01-16T00:13:45.645 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.645 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.646 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:45.646 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746806+0000", 2020-01-16T00:13:45.646 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00070802699999999996 2020-01-16T00:13:45.646 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.646 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.646 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:45.646 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.747514+0000", 2020-01-16T00:13:45.647 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00059247200000000003 2020-01-16T00:13:45.647 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.647 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.647 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:45.647 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748107+0000", 2020-01-16T00:13:45.647 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1870999999999999e-05 2020-01-16T00:13:45.647 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.647 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.648 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:45.648 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748149+0000", 2020-01-16T00:13:45.648 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5831000000000001e-05 2020-01-16T00:13:45.648 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.648 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.648 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:45.648 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748174+0000", 2020-01-16T00:13:45.648 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011660584999999999 2020-01-16T00:13:45.649 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.649 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:45.649 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.649 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.649 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.649 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:70 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e35)", 2020-01-16T00:13:45.649 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.985494+0000", 2020-01-16T00:13:45.650 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 39.559227522999997, 2020-01-16T00:13:45.650 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013746267, 2020-01-16T00:13:45.650 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:45.650 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:45.650 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:45.650 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.650 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:45.651 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985494+0000", 2020-01-16T00:13:45.651 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:45.651 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.651 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.651 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:45.651 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985494+0000", 2020-01-16T00:13:45.651 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:45.652 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.652 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.652 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:45.652 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985490+0000", 2020-01-16T00:13:45.652 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00077792099999999999 2020-01-16T00:13:45.652 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.652 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.652 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:45.653 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986268+0000", 2020-01-16T00:13:45.653 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9718000000000001e-05 2020-01-16T00:13:45.653 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.653 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.653 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:45.653 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986318+0000", 2020-01-16T00:13:45.653 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.4349999999999997e-06 2020-01-16T00:13:45.654 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.654 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.654 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:45.654 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986326+0000", 2020-01-16T00:13:45.654 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3298999999999999e-05 2020-01-16T00:13:45.654 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.654 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.654 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:45.655 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986380+0000", 2020-01-16T00:13:45.655 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4969999999999995e-05 2020-01-16T00:13:45.655 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.655 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.655 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:45.655 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986455+0000", 2020-01-16T00:13:45.655 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000110587 2020-01-16T00:13:45.655 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.656 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.656 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:45.656 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986565+0000", 2020-01-16T00:13:45.656 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063931600000000002 2020-01-16T00:13:45.656 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.656 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.656 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:45.656 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.992958+0000", 2020-01-16T00:13:45.657 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013008480000000001 2020-01-16T00:13:45.657 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.657 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.657 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:45.657 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.994259+0000", 2020-01-16T00:13:45.657 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048399979999999999 2020-01-16T00:13:45.657 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.658 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.658 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:45.658 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999099+0000", 2020-01-16T00:13:45.658 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1874999999999998e-05 2020-01-16T00:13:45.658 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.658 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.658 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:45.658 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999181+0000", 2020-01-16T00:13:45.659 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8865e-05 2020-01-16T00:13:45.659 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.659 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.659 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:45.659 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999240+0000", 2020-01-16T00:13:45.659 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013746267 2020-01-16T00:13:45.659 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.660 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:45.660 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.660 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.660 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.660 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:71 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e35)", 2020-01-16T00:13:45.660 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.986341+0000", 2020-01-16T00:13:45.660 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 39.558380161999999, 2020-01-16T00:13:45.660 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013252807, 2020-01-16T00:13:45.661 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:45.661 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:45.661 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:45.661 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.661 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:45.661 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986341+0000", 2020-01-16T00:13:45.661 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:45.661 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.662 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.662 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:45.662 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986341+0000", 2020-01-16T00:13:45.662 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:45.662 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.662 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.662 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:45.663 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986340+0000", 2020-01-16T00:13:45.663 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000103867 2020-01-16T00:13:45.663 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.663 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.663 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:45.663 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986444+0000", 2020-01-16T00:13:45.663 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0104e-05 2020-01-16T00:13:45.663 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.664 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.664 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:45.664 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986454+0000", 2020-01-16T00:13:45.664 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4220000000000002e-06 2020-01-16T00:13:45.664 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.664 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.664 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:45.664 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986458+0000", 2020-01-16T00:13:45.665 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033523070000000001 2020-01-16T00:13:45.665 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.665 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.666 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:45.666 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.989811+0000", 2020-01-16T00:13:45.666 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3296999999999999e-05 2020-01-16T00:13:45.666 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.667 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.667 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:45.667 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.989894+0000", 2020-01-16T00:13:45.667 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000117701 2020-01-16T00:13:45.668 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.668 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.669 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:45.669 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.990012+0000", 2020-01-16T00:13:45.669 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0032873490000000002 2020-01-16T00:13:45.669 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.669 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.669 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:45.669 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993299+0000", 2020-01-16T00:13:45.669 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002422619 2020-01-16T00:13:45.670 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.670 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.670 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:45.670 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.995722+0000", 2020-01-16T00:13:45.670 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0037839589999999999 2020-01-16T00:13:45.670 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.670 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.670 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:45.671 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999506+0000", 2020-01-16T00:13:45.671 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8559999999999997e-05 2020-01-16T00:13:45.671 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.671 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.671 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:45.671 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999544+0000", 2020-01-16T00:13:45.671 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9599999999999999e-05 2020-01-16T00:13:45.672 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.672 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.672 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:45.672 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999594+0000", 2020-01-16T00:13:45.672 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013252807 2020-01-16T00:13:45.672 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.672 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:45.672 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.673 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.673 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.673 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:77 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e36)", 2020-01-16T00:13:45.673 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.555854+0000", 2020-01-16T00:13:45.673 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 37.988867073000002, 2020-01-16T00:13:45.673 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010455002999999999, 2020-01-16T00:13:45.673 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:45.673 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:45.674 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:45.674 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.674 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:45.674 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555854+0000", 2020-01-16T00:13:45.674 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:45.674 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.674 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.674 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:45.675 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555854+0000", 2020-01-16T00:13:45.675 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:45.675 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.675 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.675 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:45.675 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555851+0000", 2020-01-16T00:13:45.675 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00048310200000000001 2020-01-16T00:13:45.675 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.676 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.676 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:45.676 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556334+0000", 2020-01-16T00:13:45.676 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2232e-05 2020-01-16T00:13:45.676 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.676 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.676 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:45.676 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556356+0000", 2020-01-16T00:13:45.677 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9750000000000007e-06 2020-01-16T00:13:45.677 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.677 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.677 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:45.677 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556364+0000", 2020-01-16T00:13:45.677 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8872999999999997e-05 2020-01-16T00:13:45.677 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.677 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.678 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:45.678 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556423+0000", 2020-01-16T00:13:45.678 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8687000000000001e-05 2020-01-16T00:13:45.678 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.678 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.678 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:45.678 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556502+0000", 2020-01-16T00:13:45.678 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013235300000000001 2020-01-16T00:13:45.679 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.679 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.679 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:45.679 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556634+0000", 2020-01-16T00:13:45.679 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0059770520000000001 2020-01-16T00:13:45.679 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.679 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.679 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:45.680 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.562611+0000", 2020-01-16T00:13:45.680 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029492820000000001 2020-01-16T00:13:45.680 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.680 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.680 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:45.680 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.565561+0000", 2020-01-16T00:13:45.680 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00060620300000000004 2020-01-16T00:13:45.680 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.681 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.681 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:45.681 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566167+0000", 2020-01-16T00:13:45.681 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3831000000000004e-05 2020-01-16T00:13:45.681 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.681 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.681 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:45.682 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566251+0000", 2020-01-16T00:13:45.682 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8328000000000001e-05 2020-01-16T00:13:45.682 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.682 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.682 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:45.682 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566309+0000", 2020-01-16T00:13:45.682 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010455002999999999 2020-01-16T00:13:45.682 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.683 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:45.683 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.683 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.683 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.683 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:78 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e36)", 2020-01-16T00:13:45.683 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.556378+0000", 2020-01-16T00:13:45.683 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 37.988343139999998, 2020-01-16T00:13:45.683 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010396311, 2020-01-16T00:13:45.684 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:45.684 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:45.684 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:45.684 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.684 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:45.684 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556378+0000", 2020-01-16T00:13:45.684 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:45.684 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.685 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.685 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:45.685 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556378+0000", 2020-01-16T00:13:45.685 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:45.685 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.685 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.685 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:45.686 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556377+0000", 2020-01-16T00:13:45.686 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4307000000000001e-05 2020-01-16T00:13:45.686 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.686 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.686 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:45.686 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556422+0000", 2020-01-16T00:13:45.686 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9620000000000005e-06 2020-01-16T00:13:45.686 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.687 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.687 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:45.687 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556432+0000", 2020-01-16T00:13:45.687 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5979999999999999e-06 2020-01-16T00:13:45.687 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.687 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.687 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:45.687 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556436+0000", 2020-01-16T00:13:45.688 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021059469999999999 2020-01-16T00:13:45.688 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.688 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.688 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:45.688 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.558542+0000", 2020-01-16T00:13:45.688 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000194458 2020-01-16T00:13:45.688 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.688 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.689 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:45.689 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.558737+0000", 2020-01-16T00:13:45.689 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00032867900000000002 2020-01-16T00:13:45.689 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.689 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.689 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:45.689 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.559065+0000", 2020-01-16T00:13:45.689 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052547499999999999 2020-01-16T00:13:45.690 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.690 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.690 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:45.690 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.564320+0000", 2020-01-16T00:13:45.690 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001565213 2020-01-16T00:13:45.690 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.690 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.690 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:45.691 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.565885+0000", 2020-01-16T00:13:45.691 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00076994300000000004 2020-01-16T00:13:45.691 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.691 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.691 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:45.691 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566655+0000", 2020-01-16T00:13:45.691 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7533000000000005e-05 2020-01-16T00:13:45.691 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.692 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.692 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:45.692 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566723+0000", 2020-01-16T00:13:45.692 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1616999999999997e-05 2020-01-16T00:13:45.692 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.692 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.692 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:45.692 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566774+0000", 2020-01-16T00:13:45.693 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010396311 2020-01-16T00:13:45.693 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.693 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:45.693 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.693 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.693 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.693 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:72 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 56832~1408000] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e37)", 2020-01-16T00:13:45.693 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.578753+0000", 2020-01-16T00:13:45.694 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 32.965967837000001, 2020-01-16T00:13:45.694 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011826918, 2020-01-16T00:13:45.694 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:45.694 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:45.694 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:45.694 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.694 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:45.694 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578753+0000", 2020-01-16T00:13:45.695 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:45.695 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.695 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.695 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:45.695 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578753+0000", 2020-01-16T00:13:45.695 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:45.695 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.695 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.696 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:45.696 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578751+0000", 2020-01-16T00:13:45.696 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00042457699999999998 2020-01-16T00:13:45.696 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.696 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.696 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:45.696 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579175+0000", 2020-01-16T00:13:45.696 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5404999999999999e-05 2020-01-16T00:13:45.697 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.697 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.697 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:45.697 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579191+0000", 2020-01-16T00:13:45.697 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6149999999999996e-06 2020-01-16T00:13:45.698 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.699 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.699 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:45.699 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579196+0000", 2020-01-16T00:13:45.700 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000104794 2020-01-16T00:13:45.700 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.700 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.700 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:45.700 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579301+0000", 2020-01-16T00:13:45.700 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020947199999999999 2020-01-16T00:13:45.700 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.700 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.701 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:45.701 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579511+0000", 2020-01-16T00:13:45.701 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034328400000000002 2020-01-16T00:13:45.701 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.701 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.701 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:45.701 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579854+0000", 2020-01-16T00:13:45.701 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.005078117 2020-01-16T00:13:45.702 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.702 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.702 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:45.702 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.584932+0000", 2020-01-16T00:13:45.702 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0014702160000000001 2020-01-16T00:13:45.702 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.702 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.702 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:45.703 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.586402+0000", 2020-01-16T00:13:45.703 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040623589999999998 2020-01-16T00:13:45.703 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.703 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.703 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:45.703 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590465+0000", 2020-01-16T00:13:45.703 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4478000000000002e-05 2020-01-16T00:13:45.703 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.704 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.704 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:45.704 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590529+0000", 2020-01-16T00:13:45.704 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1121000000000001e-05 2020-01-16T00:13:45.704 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.704 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.704 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:45.704 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590580+0000", 2020-01-16T00:13:45.705 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011826918 2020-01-16T00:13:45.705 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.705 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:45.705 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.705 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.705 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.705 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:73 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 1922048~118272] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e37)", 2020-01-16T00:13:45.705 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:13:45.706 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 32.96551504, 2020-01-16T00:13:45.706 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011750764, 2020-01-16T00:13:45.706 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:45.706 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:45.706 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:45.706 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.706 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:45.706 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:13:45.707 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:45.707 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.707 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.707 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:45.707 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:13:45.707 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:45.707 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.707 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.708 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:45.708 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:13:45.708 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2780000000000001e-05 2020-01-16T00:13:45.708 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.708 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.708 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:45.708 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579238+0000", 2020-01-16T00:13:45.708 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4341999999999999e-05 2020-01-16T00:13:45.709 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.709 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.709 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:45.709 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579253+0000", 2020-01-16T00:13:45.709 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1920000000000001e-06 2020-01-16T00:13:45.709 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.709 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.709 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:45.710 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579256+0000", 2020-01-16T00:13:45.710 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003224902 2020-01-16T00:13:45.710 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.710 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.710 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:45.710 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.582481+0000", 2020-01-16T00:13:45.710 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000180729 2020-01-16T00:13:45.711 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.711 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.711 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:45.711 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.582662+0000", 2020-01-16T00:13:45.711 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00025106100000000001 2020-01-16T00:13:45.711 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.711 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.711 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:45.712 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.582913+0000", 2020-01-16T00:13:45.712 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020814449999999999 2020-01-16T00:13:45.712 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.712 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.712 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:45.712 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.584994+0000", 2020-01-16T00:13:45.712 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001717243 2020-01-16T00:13:45.712 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.713 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.713 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:45.713 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.586711+0000", 2020-01-16T00:13:45.713 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041349869999999997 2020-01-16T00:13:45.713 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.713 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.713 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:45.713 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590846+0000", 2020-01-16T00:13:45.714 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3565000000000006e-05 2020-01-16T00:13:45.714 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.714 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.714 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:45.714 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590910+0000", 2020-01-16T00:13:45.714 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7002999999999997e-05 2020-01-16T00:13:45.714 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.714 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.715 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:45.715 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590957+0000", 2020-01-16T00:13:45.715 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011750764 2020-01-16T00:13:45.715 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.715 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:45.715 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.715 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.715 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.716 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:89 2.1 2:9b8617ce:::smithi12313934-38 low_tier:head [write 69632~773120] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e37)", 2020-01-16T00:13:45.716 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.593701+0000", 2020-01-16T00:13:45.716 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 32.951020413999998, 2020-01-16T00:13:45.716 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010026929, 2020-01-16T00:13:45.716 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:45.716 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:45.716 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:45.716 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.717 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:45.717 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593701+0000", 2020-01-16T00:13:45.717 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:45.717 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.717 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.717 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:45.717 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593701+0000", 2020-01-16T00:13:45.717 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:45.718 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.718 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.718 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:45.718 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593699+0000", 2020-01-16T00:13:45.718 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000237564 2020-01-16T00:13:45.718 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.718 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.718 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:45.719 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593936+0000", 2020-01-16T00:13:45.719 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2666e-05 2020-01-16T00:13:45.719 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.719 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.719 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:45.719 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593949+0000", 2020-01-16T00:13:45.719 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0089999999999998e-06 2020-01-16T00:13:45.719 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.720 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.720 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:45.720 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593954+0000", 2020-01-16T00:13:45.720 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5813000000000003e-05 2020-01-16T00:13:45.720 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.720 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.720 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:45.720 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.594000+0000", 2020-01-16T00:13:45.721 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010155299999999999 2020-01-16T00:13:45.721 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.721 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.721 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:45.721 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.594101+0000", 2020-01-16T00:13:45.721 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000203807 2020-01-16T00:13:45.721 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.721 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.722 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:45.722 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.594305+0000", 2020-01-16T00:13:45.722 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021655390000000002 2020-01-16T00:13:45.722 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.722 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.722 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:45.722 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.596471+0000", 2020-01-16T00:13:45.722 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0070645320000000001 2020-01-16T00:13:45.723 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.723 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.723 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:45.723 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603535+0000", 2020-01-16T00:13:45.723 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7577999999999998e-05 2020-01-16T00:13:45.723 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.723 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.723 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:45.724 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603573+0000", 2020-01-16T00:13:45.724 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9933999999999997e-05 2020-01-16T00:13:45.724 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.724 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.724 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:45.724 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603653+0000", 2020-01-16T00:13:45.724 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4993000000000005e-05 2020-01-16T00:13:45.724 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.725 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.725 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:45.725 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603728+0000", 2020-01-16T00:13:45.725 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010026929 2020-01-16T00:13:45.725 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.725 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:45.725 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.725 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.726 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.726 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:66 2.3 2:c3b82d87:::smithi12313934-44 low_tier:head [write 169472~818176] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e44)", 2020-01-16T00:13:45.726 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:25.278583+0000", 2020-01-16T00:13:45.726 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 20.266138016999999, 2020-01-16T00:13:45.726 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010294862, 2020-01-16T00:13:45.726 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:45.726 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:45.726 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:45.727 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.727 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:45.728 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.278583+0000", 2020-01-16T00:13:45.728 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:45.729 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.729 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.729 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:45.729 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.278583+0000", 2020-01-16T00:13:45.729 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:45.729 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.729 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.729 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:45.730 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.278581+0000", 2020-01-16T00:13:45.730 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00136721 2020-01-16T00:13:45.730 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.730 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.730 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:45.730 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.279948+0000", 2020-01-16T00:13:45.730 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6109999999999999e-05 2020-01-16T00:13:45.730 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.731 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.731 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:45.731 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.279974+0000", 2020-01-16T00:13:45.731 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.102e-05 2020-01-16T00:13:45.731 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.731 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.731 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:45.731 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.279985+0000", 2020-01-16T00:13:45.732 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6035e-05 2020-01-16T00:13:45.732 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.732 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.732 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:45.732 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.280061+0000", 2020-01-16T00:13:45.732 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000176243 2020-01-16T00:13:45.732 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.732 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.733 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:45.733 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.280238+0000", 2020-01-16T00:13:45.733 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019866099999999999 2020-01-16T00:13:45.733 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.733 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.733 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 7", 2020-01-16T00:13:45.733 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.280436+0000", 2020-01-16T00:13:45.733 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0049761830000000003 2020-01-16T00:13:45.734 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.734 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.734 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:45.734 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.285412+0000", 2020-01-16T00:13:45.734 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033347339999999998 2020-01-16T00:13:45.734 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.734 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.735 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:45.735 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.288747+0000", 2020-01-16T00:13:45.735 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9156000000000001e-05 2020-01-16T00:13:45.735 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.735 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.735 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:45.735 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.288806+0000", 2020-01-16T00:13:45.736 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1643000000000005e-05 2020-01-16T00:13:45.736 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.736 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.736 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:45.736 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.288878+0000", 2020-01-16T00:13:45.736 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010294862 2020-01-16T00:13:45.736 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.737 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:45.737 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.737 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.737 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.737 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.4.0:316 3.9 e45/20 3:93dfe6f1:::smithi12313934-40:head v 45'188, mlcod=45'186)", 2020-01-16T00:13:45.737 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.584377+0000", 2020-01-16T00:13:45.737 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 15.960344091, 2020-01-16T00:13:45.737 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095979069999999993, 2020-01-16T00:13:45.738 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:45.738 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:45.738 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:45.738 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.738 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:45.738 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584377+0000", 2020-01-16T00:13:45.738 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:45.738 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.739 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.739 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:45.739 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584377+0000", 2020-01-16T00:13:45.739 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:45.739 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.739 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.739 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:45.739 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584376+0000", 2020-01-16T00:13:45.740 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9059999999999997e-06 2020-01-16T00:13:45.740 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.740 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.740 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:45.740 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584385+0000", 2020-01-16T00:13:45.740 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3200000000000001e-05 2020-01-16T00:13:45.740 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.741 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.741 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:45.741 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584399+0000", 2020-01-16T00:13:45.741 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9660000000000001e-06 2020-01-16T00:13:45.741 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.741 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.741 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:45.741 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584405+0000", 2020-01-16T00:13:45.742 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0035787800000000002 2020-01-16T00:13:45.742 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.742 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.742 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:45.742 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.587983+0000", 2020-01-16T00:13:45.742 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0368000000000001e-05 2020-01-16T00:13:45.742 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.742 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.743 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:45.743 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588014+0000", 2020-01-16T00:13:45.743 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0059140290000000003 2020-01-16T00:13:45.743 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.743 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.743 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:45.743 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.593928+0000", 2020-01-16T00:13:45.743 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7135999999999997e-05 2020-01-16T00:13:45.744 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.744 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.744 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:45.744 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.593975+0000", 2020-01-16T00:13:45.744 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095979069999999993 2020-01-16T00:13:45.744 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.744 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:45.744 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.745 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.745 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.745 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:77 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 56832~1408000] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e45)", 2020-01-16T00:13:45.745 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.584566+0000", 2020-01-16T00:13:45.745 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 15.960155366, 2020-01-16T00:13:45.745 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017037775000000002, 2020-01-16T00:13:45.745 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:45.746 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:45.746 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:45.746 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.746 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:45.746 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584566+0000", 2020-01-16T00:13:45.746 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:45.746 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.746 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.746 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:45.747 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584566+0000", 2020-01-16T00:13:45.747 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:45.747 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.747 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.747 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:45.747 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584564+0000", 2020-01-16T00:13:45.747 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043024300000000002 2020-01-16T00:13:45.748 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.748 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.748 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:45.748 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584994+0000", 2020-01-16T00:13:45.748 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9485999999999999e-05 2020-01-16T00:13:45.748 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.748 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.748 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:45.749 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585014+0000", 2020-01-16T00:13:45.749 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8909999999999995e-06 2020-01-16T00:13:45.749 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.749 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.749 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:45.749 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585022+0000", 2020-01-16T00:13:45.749 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039114459999999998 2020-01-16T00:13:45.750 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.750 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.750 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:45.750 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588933+0000", 2020-01-16T00:13:45.750 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011178700000000001 2020-01-16T00:13:45.750 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.750 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.750 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:45.751 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589045+0000", 2020-01-16T00:13:45.751 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016967300000000001 2020-01-16T00:13:45.751 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.751 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.751 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:45.751 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589214+0000", 2020-01-16T00:13:45.751 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012142356 2020-01-16T00:13:45.751 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.752 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.752 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:45.752 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601357+0000", 2020-01-16T00:13:45.752 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.1466999999999997e-05 2020-01-16T00:13:45.752 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.752 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.752 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:45.752 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601448+0000", 2020-01-16T00:13:45.753 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4817000000000001e-05 2020-01-16T00:13:45.753 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.753 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.753 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:45.753 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601493+0000", 2020-01-16T00:13:45.753 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9734000000000001e-05 2020-01-16T00:13:45.753 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.753 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.754 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:45.754 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601553+0000", 2020-01-16T00:13:45.754 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0677999999999998e-05 2020-01-16T00:13:45.754 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.754 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.754 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:45.754 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601604+0000", 2020-01-16T00:13:45.754 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017037775000000002 2020-01-16T00:13:45.755 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.755 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:45.755 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.755 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.755 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.755 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:78 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 1922048~118272] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e45)", 2020-01-16T00:13:45.755 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.585030+0000", 2020-01-16T00:13:45.756 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 15.959690644, 2020-01-16T00:13:45.756 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016699964000000001, 2020-01-16T00:13:45.756 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:45.756 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:45.756 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:45.756 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.756 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:45.756 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585030+0000", 2020-01-16T00:13:45.757 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:45.757 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.757 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.757 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:45.757 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585030+0000", 2020-01-16T00:13:45.757 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:45.757 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.757 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.758 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:45.758 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585030+0000", 2020-01-16T00:13:45.758 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7085000000000001e-05 2020-01-16T00:13:45.758 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.758 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.758 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:45.758 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585077+0000", 2020-01-16T00:13:45.758 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5380000000000008e-06 2020-01-16T00:13:45.759 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.759 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.759 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:45.759 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585086+0000", 2020-01-16T00:13:45.759 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5410000000000002e-06 2020-01-16T00:13:45.759 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.759 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.760 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:45.760 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585091+0000", 2020-01-16T00:13:45.760 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0088486620000000002 2020-01-16T00:13:45.760 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.760 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.760 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:45.760 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.593940+0000", 2020-01-16T00:13:45.761 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112594 2020-01-16T00:13:45.761 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.761 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.761 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:45.761 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.594052+0000", 2020-01-16T00:13:45.762 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019893999999999999 2020-01-16T00:13:45.762 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.762 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.762 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:45.762 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.594251+0000", 2020-01-16T00:13:45.762 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0071632730000000004 2020-01-16T00:13:45.762 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.762 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.763 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:45.763 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601414+0000", 2020-01-16T00:13:45.763 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4672999999999996e-05 2020-01-16T00:13:45.763 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.763 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.763 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:45.763 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601479+0000", 2020-01-16T00:13:45.763 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000158453 2020-01-16T00:13:45.764 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.764 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.764 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:45.764 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601638+0000", 2020-01-16T00:13:45.764 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8605e-05 2020-01-16T00:13:45.764 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.764 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.764 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:45.765 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601686+0000", 2020-01-16T00:13:45.765 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4283999999999998e-05 2020-01-16T00:13:45.765 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.765 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.765 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:45.765 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601730+0000", 2020-01-16T00:13:45.765 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016699964000000001 2020-01-16T00:13:45.766 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.766 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:45.767 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.767 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.767 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.767 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:126 2.1 2:a4df40e2:::smithi12313934-42 low_tier:head [write 70656~727552] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e45)", 2020-01-16T00:13:45.767 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.627078+0000", 2020-01-16T00:13:45.767 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 15.917642988000001, 2020-01-16T00:13:45.767 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010098995, 2020-01-16T00:13:45.768 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:45.768 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:45.768 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:45.768 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.768 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:45.768 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627078+0000", 2020-01-16T00:13:45.768 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:45.768 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.769 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.769 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:45.769 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627078+0000", 2020-01-16T00:13:45.769 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:45.769 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.769 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.769 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:45.769 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627076+0000", 2020-01-16T00:13:45.770 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019321599999999999 2020-01-16T00:13:45.770 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.770 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.770 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:45.770 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627269+0000", 2020-01-16T00:13:45.770 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8471999999999999e-05 2020-01-16T00:13:45.770 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.770 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.771 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:45.771 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627288+0000", 2020-01-16T00:13:45.771 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3199999999999996e-06 2020-01-16T00:13:45.771 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.771 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.771 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:45.771 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627294+0000", 2020-01-16T00:13:45.771 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3341000000000002e-05 2020-01-16T00:13:45.772 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.772 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.772 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:45.772 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627357+0000", 2020-01-16T00:13:45.772 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010985 2020-01-16T00:13:45.772 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.772 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.772 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:45.773 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627467+0000", 2020-01-16T00:13:45.773 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022914499999999999 2020-01-16T00:13:45.773 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.773 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.773 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:45.773 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627696+0000", 2020-01-16T00:13:45.773 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0055120000000000004 2020-01-16T00:13:45.774 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.774 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.774 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:45.774 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.633208+0000", 2020-01-16T00:13:45.774 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0019968569999999999 2020-01-16T00:13:45.774 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.774 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.774 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:45.775 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.635205+0000", 2020-01-16T00:13:45.775 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018113160000000001 2020-01-16T00:13:45.775 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.775 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.775 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:45.775 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.637016+0000", 2020-01-16T00:13:45.775 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9515e-05 2020-01-16T00:13:45.775 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.776 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.776 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:45.776 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.637096+0000", 2020-01-16T00:13:45.776 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1213999999999999e-05 2020-01-16T00:13:45.776 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.776 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.776 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:45.776 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.637177+0000", 2020-01-16T00:13:45.777 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010098995 2020-01-16T00:13:45.777 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.777 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:45.777 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.777 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.777 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.777 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:127 2.1 2:a4df40e2:::smithi12313934-42 low_tier:head [write 1364992~1145344] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e45)", 2020-01-16T00:13:45.777 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.627308+0000", 2020-01-16T00:13:45.778 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 15.917413192, 2020-01-16T00:13:45.778 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011068883999999999, 2020-01-16T00:13:45.778 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:45.778 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:45.778 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:45.778 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.778 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:45.778 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627308+0000", 2020-01-16T00:13:45.779 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:45.779 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.779 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.779 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:45.779 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627308+0000", 2020-01-16T00:13:45.779 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:45.779 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.779 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.780 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:45.780 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627307+0000", 2020-01-16T00:13:45.780 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000278597 2020-01-16T00:13:45.780 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.780 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.780 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:45.780 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627586+0000", 2020-01-16T00:13:45.780 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9169999999999997e-06 2020-01-16T00:13:45.781 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.781 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.781 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:45.781 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627591+0000", 2020-01-16T00:13:45.781 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0479999999999999e-06 2020-01-16T00:13:45.781 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.781 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.781 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:45.782 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627594+0000", 2020-01-16T00:13:45.782 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030987269999999999 2020-01-16T00:13:45.782 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.782 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.782 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:45.782 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.630693+0000", 2020-01-16T00:13:45.782 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000103688 2020-01-16T00:13:45.782 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.783 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.783 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:45.783 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.630796+0000", 2020-01-16T00:13:45.783 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000144606 2020-01-16T00:13:45.783 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.783 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.783 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:45.783 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.630941+0000", 2020-01-16T00:13:45.784 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039264269999999997 2020-01-16T00:13:45.784 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.784 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.784 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:45.784 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.634867+0000", 2020-01-16T00:13:45.784 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00038581200000000003 2020-01-16T00:13:45.784 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.784 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.785 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:45.785 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.635253+0000", 2020-01-16T00:13:45.785 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030568879999999998 2020-01-16T00:13:45.785 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.785 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.785 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:45.785 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.638310+0000", 2020-01-16T00:13:45.786 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8390999999999999e-05 2020-01-16T00:13:45.786 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.786 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.786 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:45.786 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.638348+0000", 2020-01-16T00:13:45.786 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8402999999999999e-05 2020-01-16T00:13:45.786 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.786 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.787 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:45.787 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.638377+0000", 2020-01-16T00:13:45.787 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011068883999999999 2020-01-16T00:13:45.787 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.787 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:45.787 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.787 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.787 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.788 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.6.13:30 2.3 2:fc791323:::smithi12313934-2 low_tier:head [write 476160~1240064] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e48)", 2020-01-16T00:13:45.788 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:39.466268+0000", 2020-01-16T00:13:45.788 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 6.0784528880000002, 2020-01-16T00:13:45.788 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010593112, 2020-01-16T00:13:45.788 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:45.788 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:45.788 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:45.788 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.789 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:45.789 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.466268+0000", 2020-01-16T00:13:45.789 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:45.789 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.789 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.790 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:45.790 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.466268+0000", 2020-01-16T00:13:45.790 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:45.790 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.790 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.790 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:45.790 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.466266+0000", 2020-01-16T00:13:45.791 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012489210000000001 2020-01-16T00:13:45.791 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.791 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.791 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:45.791 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467515+0000", 2020-01-16T00:13:45.791 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8411999999999999e-05 2020-01-16T00:13:45.791 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.791 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.792 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:45.792 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467533+0000", 2020-01-16T00:13:45.792 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9319999999999996e-06 2020-01-16T00:13:45.792 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.792 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.792 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:45.792 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467541+0000", 2020-01-16T00:13:45.792 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5289999999999993e-05 2020-01-16T00:13:45.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.793 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.793 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:45.793 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467607+0000", 2020-01-16T00:13:45.793 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010962 2020-01-16T00:13:45.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.793 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.793 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:45.794 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467716+0000", 2020-01-16T00:13:45.794 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020883699999999999 2020-01-16T00:13:45.794 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.794 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.794 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 7", 2020-01-16T00:13:45.794 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467925+0000", 2020-01-16T00:13:45.794 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063872829999999997 2020-01-16T00:13:45.794 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.795 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.795 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:45.795 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.474312+0000", 2020-01-16T00:13:45.795 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0023165769999999998 2020-01-16T00:13:45.795 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.795 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.795 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:45.795 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.476629+0000", 2020-01-16T00:13:45.796 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8543999999999999e-05 2020-01-16T00:13:45.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.796 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.796 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:45.796 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.476707+0000", 2020-01-16T00:13:45.796 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000153914 2020-01-16T00:13:45.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.796 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.797 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:45.797 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.476861+0000", 2020-01-16T00:13:45.797 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010593112 2020-01-16T00:13:45.797 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.797 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:45.797 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.797 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.797 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.798 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:3653 3.9 e48/20 3:9557b633:::smithi12313934-43:head v 48'202, mlcod=48'200)", 2020-01-16T00:13:45.798 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:39.734299+0000", 2020-01-16T00:13:45.798 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.8104225119999997, 2020-01-16T00:13:45.798 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095124579999999997, 2020-01-16T00:13:45.798 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:45.798 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:45.798 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:45.799 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.799 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:45.799 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.734299+0000", 2020-01-16T00:13:45.799 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:45.799 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.799 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.799 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:45.799 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.734299+0000", 2020-01-16T00:13:45.800 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:45.800 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.800 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.800 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:45.800 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.734295+0000", 2020-01-16T00:13:45.800 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00064037399999999998 2020-01-16T00:13:45.800 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.800 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.801 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:45.801 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.734935+0000", 2020-01-16T00:13:45.801 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0710000000000002e-05 2020-01-16T00:13:45.801 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.801 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.801 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:45.801 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.734966+0000", 2020-01-16T00:13:45.801 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5869999999999999e-05 2020-01-16T00:13:45.802 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.802 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.802 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:45.802 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.734982+0000", 2020-01-16T00:13:45.802 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119574 2020-01-16T00:13:45.802 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.802 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.802 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:45.803 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.735102+0000", 2020-01-16T00:13:45.803 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2585999999999999e-05 2020-01-16T00:13:45.803 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.803 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.803 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:45.803 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.735174+0000", 2020-01-16T00:13:45.803 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0085451599999999996 2020-01-16T00:13:45.803 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.804 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.804 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:45.804 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.743719+0000", 2020-01-16T00:13:45.804 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.1713999999999996e-05 2020-01-16T00:13:45.804 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.804 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.804 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:45.804 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.743811+0000", 2020-01-16T00:13:45.805 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095124579999999997 2020-01-16T00:13:45.805 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.805 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:45.805 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.805 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.805 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.805 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:155 2.3 2:c9873fc7:::smithi12313934-30 low_tier:head [write 1542656~711168] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e50)", 2020-01-16T00:13:45.805 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:43.769943+0000", 2020-01-16T00:13:45.806 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 1.7747779619999999, 2020-01-16T00:13:45.806 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010710127999999999, 2020-01-16T00:13:45.806 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:45.806 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:45.806 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:45.806 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.806 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:45.806 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.769943+0000", 2020-01-16T00:13:45.807 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:45.807 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.807 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.807 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:45.807 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.769943+0000", 2020-01-16T00:13:45.807 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:45.807 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.807 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.808 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:45.808 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.769942+0000", 2020-01-16T00:13:45.808 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017878100000000001 2020-01-16T00:13:45.808 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.808 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.808 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:45.808 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.770121+0000", 2020-01-16T00:13:45.808 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2980000000000003e-06 2020-01-16T00:13:45.809 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.809 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.809 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:45.809 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.770129+0000", 2020-01-16T00:13:45.809 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9129999999999999e-06 2020-01-16T00:13:45.809 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.809 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.809 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:45.810 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.770131+0000", 2020-01-16T00:13:45.810 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0053233519999999999 2020-01-16T00:13:45.810 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.810 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.810 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:45.810 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.775455+0000", 2020-01-16T00:13:45.810 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012385099999999999 2020-01-16T00:13:45.810 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.811 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.811 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:45.811 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.775579+0000", 2020-01-16T00:13:45.811 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000211554 2020-01-16T00:13:45.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.811 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.811 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 7", 2020-01-16T00:13:45.811 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.775790+0000", 2020-01-16T00:13:45.812 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029614210000000001 2020-01-16T00:13:45.812 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.812 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.812 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:45.812 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.778752+0000", 2020-01-16T00:13:45.812 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001798928 2020-01-16T00:13:45.812 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.812 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.813 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:45.813 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.780551+0000", 2020-01-16T00:13:45.813 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9413999999999999e-05 2020-01-16T00:13:45.813 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.813 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.813 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:45.813 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.780590+0000", 2020-01-16T00:13:45.813 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3275999999999995e-05 2020-01-16T00:13:45.814 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:45.814 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:45.814 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:45.814 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.780653+0000", 2020-01-16T00:13:45.814 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010710127999999999 2020-01-16T00:13:45.814 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.814 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:45.814 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.815 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:45.815 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:45.815 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:45.815 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-01-16T00:13:45.818 INFO:tasks.rados.rados.0.smithi123.stdout:1456: expect (ObjNum 473 snap 0 seq_num 473) 2020-01-16T00:13:45.818 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:45.812+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:45.819 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-01-16T00:13:45.831 INFO:teuthology.orchestra.run.smithi123.stderr:nodeep-scrub is set 2020-01-16T00:13:45.888 INFO:tasks.rados.rados.0.smithi123.stdout:1461: expect (ObjNum 474 snap 0 seq_num 474) 2020-01-16T00:13:45.916 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:45.912+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:45.928 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:45.929 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:13:45.929 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:13:45.929 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:45.938 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-01-16T00:13:45.951 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:13:45.970 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:13:45.990 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:13:46.010 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:13:46.030 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:46.028+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:46.031 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:13:46.056 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:13:46.076 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:13:46.078 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-01-16T00:13:46.078 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-01-16T00:13:46.078 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-01-16T00:13:46.078 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-01-16T00:13:46.079 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-01-16T00:13:46.079 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENXIO: problem getting command descriptions from osd.5 2020-01-16T00:13:46.079 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: problem getting command descriptions from osd.5 2020-01-16T00:13:46.079 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-01-16T00:13:46.079 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-01-16T00:13:46.087 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-01-16T00:13:46.087 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-01-16T00:13:46.093 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:46.103 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-01-16T00:13:46.126 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:46.118+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:46.221 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:46.216+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:46.255 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:46.258 INFO:tasks.rados.rados.0.smithi123.stdout:1459: expect (ObjNum 432 snap 0 seq_num 432) 2020-01-16T00:13:46.265 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-01-16T00:13:46.358 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:46.356+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:46.416 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:46.423 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:46.418+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:46.426 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-01-16T00:13:46.461 INFO:tasks.rados.rados.0.smithi123.stdout:1454: finishing write tid 4 to smithi12313934-48 2020-01-16T00:13:46.461 INFO:tasks.rados.rados.0.smithi123.stdout:1454: finishing write tid 5 to smithi12313934-48 2020-01-16T00:13:46.461 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 48 v 174 (ObjNum 485 snap 0 seq_num 485) dirty exists 2020-01-16T00:13:46.461 INFO:tasks.rados.rados.0.smithi123.stdout:1454: left oid 48 (ObjNum 485 snap 0 seq_num 485) 2020-01-16T00:13:46.461 INFO:tasks.rados.rados.0.smithi123.stdout:1458: finishing write tid 1 to smithi12313934-50 2020-01-16T00:13:46.461 INFO:tasks.rados.rados.0.smithi123.stdout:1458: finishing write tid 2 to smithi12313934-50 2020-01-16T00:13:46.462 INFO:tasks.rados.rados.0.smithi123.stdout:1458: finishing write tid 4 to smithi12313934-50 2020-01-16T00:13:46.462 INFO:tasks.rados.rados.0.smithi123.stdout:1458: finishing write tid 5 to smithi12313934-50 2020-01-16T00:13:46.462 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 50 v 182 (ObjNum 486 snap 0 seq_num 486) dirty exists 2020-01-16T00:13:46.462 INFO:tasks.rados.rados.0.smithi123.stdout:1458: left oid 50 (ObjNum 486 snap 0 seq_num 486) 2020-01-16T00:13:46.462 INFO:tasks.rados.rados.0.smithi123.stdout:1462: expect (ObjNum 478 snap 0 seq_num 478) 2020-01-16T00:13:46.524 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:46.522+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:46.575 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:46.585 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-01-16T00:13:46.625 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:46.622+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:46.640 INFO:tasks.rados.rados.0.smithi123.stdout:1463: finishing write tid 1 to smithi12313934-36 2020-01-16T00:13:46.640 INFO:tasks.rados.rados.0.smithi123.stdout:1463: finishing write tid 2 to smithi12313934-36 2020-01-16T00:13:46.641 INFO:tasks.rados.rados.0.smithi123.stdout:1463: finishing write tid 4 to smithi12313934-36 2020-01-16T00:13:46.641 INFO:tasks.rados.rados.0.smithi123.stdout:1463: finishing write tid 5 to smithi12313934-36 2020-01-16T00:13:46.641 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 36 v 97 (ObjNum 487 snap 0 seq_num 487) dirty exists 2020-01-16T00:13:46.641 INFO:tasks.rados.rados.0.smithi123.stdout:1463: left oid 36 (ObjNum 487 snap 0 seq_num 487) 2020-01-16T00:13:46.641 INFO:tasks.rados.rados.0.smithi123.stdout:1464: finishing write tid 1 to smithi12313934-7 2020-01-16T00:13:46.641 INFO:tasks.rados.rados.0.smithi123.stdout:1464: finishing write tid 2 to smithi12313934-7 2020-01-16T00:13:46.641 INFO:tasks.rados.rados.0.smithi123.stdout:1464: finishing write tid 4 to smithi12313934-7 2020-01-16T00:13:46.642 INFO:tasks.rados.rados.0.smithi123.stdout:1464: finishing write tid 5 to smithi12313934-7 2020-01-16T00:13:46.642 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 7 v 211 (ObjNum 488 snap 0 seq_num 488) dirty exists 2020-01-16T00:13:46.642 INFO:tasks.rados.rados.0.smithi123.stdout:1464: left oid 7 (ObjNum 488 snap 0 seq_num 488) 2020-01-16T00:13:46.642 INFO:tasks.rados.rados.0.smithi123.stdout:1465: expect (ObjNum 440 snap 0 seq_num 440) 2020-01-16T00:13:46.704 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:46.707 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-01-16T00:13:46.725 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:46.722+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:46.823 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:46.826 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:46.822+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:46.833 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-01-16T00:13:46.858 INFO:tasks.rados.rados.0.smithi123.stdout:1466: expect (ObjNum 466 snap 0 seq_num 466) 2020-01-16T00:13:46.927 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:46.922+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:46.939 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:46.947 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-01-16T00:13:47.028 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:47.024+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:47.053 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:47.063 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-01-16T00:13:47.129 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:47.126+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:47.130 INFO:tasks.rados.rados.0.smithi123.stdout:1468: expect (ObjNum 480 snap 0 seq_num 480) 2020-01-16T00:13:47.170 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:47.180 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-01-16T00:13:47.262 INFO:tasks.rados.rados.0.smithi123.stdout:1469: finishing write tid 1 to smithi12313934-16 2020-01-16T00:13:47.262 INFO:tasks.rados.rados.0.smithi123.stdout:1469: finishing write tid 2 to smithi12313934-16 2020-01-16T00:13:47.263 INFO:tasks.rados.rados.0.smithi123.stdout:1469: finishing write tid 4 to smithi12313934-16 2020-01-16T00:13:47.263 INFO:tasks.rados.rados.0.smithi123.stdout:1469: finishing write tid 5 to smithi12313934-16 2020-01-16T00:13:47.263 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 16 v 219 (ObjNum 489 snap 0 seq_num 489) dirty exists 2020-01-16T00:13:47.263 INFO:tasks.rados.rados.0.smithi123.stdout:1469: left oid 16 (ObjNum 489 snap 0 seq_num 489) 2020-01-16T00:13:47.263 INFO:tasks.rados.rados.0.smithi123.stdout:1470: finishing write tid 1 to smithi12313934-43 2020-01-16T00:13:47.263 INFO:tasks.rados.rados.0.smithi123.stdout:1470: finishing write tid 2 to smithi12313934-43 2020-01-16T00:13:47.263 INFO:tasks.rados.rados.0.smithi123.stdout:1470: finishing write tid 4 to smithi12313934-43 2020-01-16T00:13:47.263 INFO:tasks.rados.rados.0.smithi123.stdout:1454: done (15 left) 2020-01-16T00:13:47.264 INFO:tasks.rados.rados.0.smithi123.stdout:1456: done (14 left) 2020-01-16T00:13:47.264 INFO:tasks.rados.rados.0.smithi123.stdout:1457: done (13 left) 2020-01-16T00:13:47.264 INFO:tasks.rados.rados.0.smithi123.stdout:1458: done (12 left) 2020-01-16T00:13:47.264 INFO:tasks.rados.rados.0.smithi123.stdout:1459: done (11 left) 2020-01-16T00:13:47.264 INFO:tasks.rados.rados.0.smithi123.stdout:1460: done (10 left) 2020-01-16T00:13:47.264 INFO:tasks.rados.rados.0.smithi123.stdout:1461: done (9 left) 2020-01-16T00:13:47.264 INFO:tasks.rados.rados.0.smithi123.stdout:1462: done (8 left) 2020-01-16T00:13:47.264 INFO:tasks.rados.rados.0.smithi123.stdout:1463: done (7 left) 2020-01-16T00:13:47.265 INFO:tasks.rados.rados.0.smithi123.stdout:1464: done (6 left) 2020-01-16T00:13:47.265 INFO:tasks.rados.rados.0.smithi123.stdout:1465: done (5 left) 2020-01-16T00:13:47.265 INFO:tasks.rados.rados.0.smithi123.stdout:1466: done (4 left) 2020-01-16T00:13:47.265 INFO:tasks.rados.rados.0.smithi123.stdout:1467: done (3 left) 2020-01-16T00:13:47.265 INFO:tasks.rados.rados.0.smithi123.stdout:1468: done (2 left) 2020-01-16T00:13:47.265 INFO:tasks.rados.rados.0.smithi123.stdout:1469: done (1 left) 2020-01-16T00:13:47.265 INFO:tasks.rados.rados.0.smithi123.stdout:1471: read oid 1 snap -1 2020-01-16T00:13:47.266 INFO:tasks.rados.rados.0.smithi123.stdout:1471: expect (ObjNum 449 snap 0 seq_num 449) 2020-01-16T00:13:47.266 INFO:tasks.rados.rados.0.smithi123.stdout:1472: read oid 27 snap -1 2020-01-16T00:13:47.266 INFO:tasks.rados.rados.0.smithi123.stdout:1472: expect (ObjNum 446 snap 0 seq_num 446) 2020-01-16T00:13:47.266 INFO:tasks.rados.rados.0.smithi123.stdout:1473: read oid 50 snap -1 2020-01-16T00:13:47.266 INFO:tasks.rados.rados.0.smithi123.stdout:1473: expect (ObjNum 486 snap 0 seq_num 486) 2020-01-16T00:13:47.266 INFO:tasks.rados.rados.0.smithi123.stdout:1474: write (excl) oid 16 current snap is 0 2020-01-16T00:13:47.266 INFO:tasks.rados.rados.0.smithi123.stdout:1474: seq_num 491 ranges {652237=793851,2198894=53343} 2020-01-16T00:13:47.267 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:47.258+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:47.277 INFO:tasks.rados.rados.0.smithi123.stdout:1474: writing smithi12313934-16 from 652237 to 1446088 tid 1 2020-01-16T00:13:47.283 INFO:tasks.rados.rados.0.smithi123.stdout:1474: writing smithi12313934-16 from 2198894 to 2252237 tid 2 2020-01-16T00:13:47.283 INFO:tasks.rados.rados.0.smithi123.stdout:1475: read oid 23 snap -1 2020-01-16T00:13:47.283 INFO:tasks.rados.rados.0.smithi123.stdout:1475: expect (ObjNum 452 snap 0 seq_num 452) 2020-01-16T00:13:47.283 INFO:tasks.rados.rados.0.smithi123.stdout:1470: finishing write tid 5 to smithi12313934-43 2020-01-16T00:13:47.283 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 43 v 228 (ObjNum 490 snap 0 seq_num 490) dirty exists 2020-01-16T00:13:47.283 INFO:tasks.rados.rados.0.smithi123.stdout:1470: left oid 43 (ObjNum 490 snap 0 seq_num 490) 2020-01-16T00:13:47.283 INFO:tasks.rados.rados.0.smithi123.stdout:1476: read oid 12 snap -1 2020-01-16T00:13:47.283 INFO:tasks.rados.rados.0.smithi123.stdout:1476: expect (ObjNum 470 snap 0 seq_num 470) 2020-01-16T00:13:47.284 INFO:tasks.rados.rados.0.smithi123.stdout:1470: done (6 left) 2020-01-16T00:13:47.284 INFO:tasks.rados.rados.0.smithi123.stdout:1477: write (excl) oid 32 current snap is 0 2020-01-16T00:13:47.284 INFO:tasks.rados.rados.0.smithi123.stdout:1477: seq_num 492 ranges {669044=469100,1741023=459161} 2020-01-16T00:13:47.289 INFO:tasks.rados.rados.0.smithi123.stdout:1477: writing smithi12313934-32 from 669044 to 1138144 tid 1 2020-01-16T00:13:47.295 INFO:tasks.rados.rados.0.smithi123.stdout:1477: writing smithi12313934-32 from 1741023 to 2200184 tid 2 2020-01-16T00:13:47.296 INFO:tasks.rados.rados.0.smithi123.stdout:1478: write (excl) oid 30 current snap is 0 2020-01-16T00:13:47.296 INFO:tasks.rados.rados.0.smithi123.stdout:1478: seq_num 493 ranges {685851=544349,1799505=486346} 2020-01-16T00:13:47.303 INFO:tasks.rados.rados.0.smithi123.stdout:1478: writing smithi12313934-30 from 685851 to 1230200 tid 1 2020-01-16T00:13:47.309 INFO:tasks.rados.rados.0.smithi123.stdout:1478: writing smithi12313934-30 from 1799505 to 2285851 tid 2 2020-01-16T00:13:47.309 INFO:tasks.rados.rados.0.smithi123.stdout:1479: read oid 22 snap -1 2020-01-16T00:13:47.310 INFO:tasks.rados.rados.0.smithi123.stdout:1479: expect (ObjNum 393 snap 0 seq_num 393) 2020-01-16T00:13:47.310 INFO:tasks.rados.rados.0.smithi123.stdout:1480: read oid 6 snap -1 2020-01-16T00:13:47.310 INFO:tasks.rados.rados.0.smithi123.stdout:1480: expect (ObjNum 480 snap 0 seq_num 480) 2020-01-16T00:13:47.310 INFO:tasks.rados.rados.0.smithi123.stdout:1481: read oid 20 snap -1 2020-01-16T00:13:47.310 INFO:tasks.rados.rados.0.smithi123.stdout:1481: expect deleted 2020-01-16T00:13:47.310 INFO:tasks.rados.rados.0.smithi123.stdout:1482: write oid 7 current snap is 0 2020-01-16T00:13:47.310 INFO:tasks.rados.rados.0.smithi123.stdout:1482: seq_num 494 ranges {702658=619598,1857987=444671} 2020-01-16T00:13:47.317 INFO:tasks.rados.rados.0.smithi123.stdout:1482: writing smithi12313934-7 from 702658 to 1322256 tid 1 2020-01-16T00:13:47.323 INFO:tasks.rados.rados.0.smithi123.stdout:1482: writing smithi12313934-7 from 1857987 to 2302658 tid 2 2020-01-16T00:13:47.324 INFO:tasks.rados.rados.0.smithi123.stdout:1483: read oid 45 snap -1 2020-01-16T00:13:47.324 INFO:tasks.rados.rados.0.smithi123.stdout:1483: expect (ObjNum 466 snap 0 seq_num 466) 2020-01-16T00:13:47.324 INFO:tasks.rados.rados.0.smithi123.stdout:1484: read oid 35 snap -1 2020-01-16T00:13:47.324 INFO:tasks.rados.rados.0.smithi123.stdout:1484: expect (ObjNum 465 snap 0 seq_num 465) 2020-01-16T00:13:47.324 INFO:tasks.rados.rados.0.smithi123.stdout:1485: write (excl) oid 39 current snap is 0 2020-01-16T00:13:47.324 INFO:tasks.rados.rados.0.smithi123.stdout:1485: seq_num 495 ranges {719465=411200,1632822=686643} 2020-01-16T00:13:47.330 INFO:tasks.rados.rados.0.smithi123.stdout:1485: writing smithi12313934-39 from 719465 to 1130665 tid 1 2020-01-16T00:13:47.332 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:47.328+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:47.337 INFO:tasks.rados.rados.0.smithi123.stdout:1485: writing smithi12313934-39 from 1632822 to 2319465 tid 2 2020-01-16T00:13:47.337 INFO:tasks.rados.rados.0.smithi123.stdout:1486: write (excl) oid 13 current snap is 0 2020-01-16T00:13:47.338 INFO:tasks.rados.rados.0.smithi123.stdout:1486: seq_num 496 ranges {736272=486449,1974951=361321} 2020-01-16T00:13:47.344 INFO:tasks.rados.rados.0.smithi123.stdout:1486: writing smithi12313934-13 from 736272 to 1222721 tid 1 2020-01-16T00:13:47.346 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:47.350 INFO:tasks.rados.rados.0.smithi123.stdout:1486: writing smithi12313934-13 from 1974951 to 2336272 tid 2 2020-01-16T00:13:47.351 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:47.351 INFO:tasks.rados.rados.0.smithi123.stdout:1473: expect (ObjNum 486 snap 0 seq_num 486) 2020-01-16T00:13:47.357 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-01-16T00:13:47.434 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:47.428+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:47.502 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:47.511 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-01-16T00:13:47.534 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:47.528+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:47.540 INFO:tasks.rados.rados.0.smithi123.stdout:1472: expect (ObjNum 446 snap 0 seq_num 446) 2020-01-16T00:13:47.611 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:47.626 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-01-16T00:13:47.635 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:47.632+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:47.736 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:47.732+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:47.773 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:47.781 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-01-16T00:13:47.819 INFO:tasks.rados.rados.0.smithi123.stdout:1471: expect (ObjNum 449 snap 0 seq_num 449) 2020-01-16T00:13:47.837 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:47.832+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:47.842 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2020-01-16T00:13:47.884 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:47.893 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-01-16T00:13:47.937 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:47.932+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:48.038 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:48.036+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:48.044 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:48.053 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-01-16T00:13:48.055 INFO:tasks.rados.rados.0.smithi123.stdout:1471: done (15 left) 2020-01-16T00:13:48.055 INFO:tasks.rados.rados.0.smithi123.stdout:1472: done (14 left) 2020-01-16T00:13:48.056 INFO:tasks.rados.rados.0.smithi123.stdout:1473: done (13 left) 2020-01-16T00:13:48.056 INFO:tasks.rados.rados.0.smithi123.stdout:1487: write (excl) oid 11 current snap is 0 2020-01-16T00:13:48.056 INFO:tasks.rados.rados.0.smithi123.stdout:1487: seq_num 497 ranges {753079=561698,2033433=319646} 2020-01-16T00:13:48.063 INFO:tasks.rados.rados.0.smithi123.stdout:1487: writing smithi12313934-11 from 753079 to 1314777 tid 1 2020-01-16T00:13:48.070 INFO:tasks.rados.rados.0.smithi123.stdout:1487: writing smithi12313934-11 from 2033433 to 2353079 tid 2 2020-01-16T00:13:48.071 INFO:tasks.rados.rados.0.smithi123.stdout:1488: read oid 8 snap -1 2020-01-16T00:13:48.071 INFO:tasks.rados.rados.0.smithi123.stdout:1488: expect (ObjNum 454 snap 0 seq_num 454) 2020-01-16T00:13:48.071 INFO:tasks.rados.rados.0.smithi123.stdout:1489: read oid 18 snap -1 2020-01-16T00:13:48.071 INFO:tasks.rados.rados.0.smithi123.stdout:1489: expect (ObjNum 481 snap 0 seq_num 481) 2020-01-16T00:13:48.071 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:48.071 INFO:tasks.rados.rados.0.smithi123.stdout:1474: finishing write tid 1 to smithi12313934-16 2020-01-16T00:13:48.071 INFO:tasks.rados.rados.0.smithi123.stdout:1475: expect (ObjNum 452 snap 0 seq_num 452) 2020-01-16T00:13:48.088 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-01-16T00:13:48.139 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:48.138+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:48.152 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:48.161 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-01-16T00:13:48.186 INFO:tasks.rados.rados.0.smithi123.stdout:1474: finishing write tid 2 to smithi12313934-16 2020-01-16T00:13:48.186 INFO:tasks.rados.rados.0.smithi123.stdout:1474: finishing write tid 4 to smithi12313934-16 2020-01-16T00:13:48.186 INFO:tasks.rados.rados.0.smithi123.stdout:1474: finishing write tid 5 to smithi12313934-16 2020-01-16T00:13:48.187 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 16 v 222 (ObjNum 491 snap 0 seq_num 491) dirty exists 2020-01-16T00:13:48.187 INFO:tasks.rados.rados.0.smithi123.stdout:1474: left oid 16 (ObjNum 491 snap 0 seq_num 491) 2020-01-16T00:13:48.187 INFO:tasks.rados.rados.0.smithi123.stdout:1476: expect (ObjNum 470 snap 0 seq_num 470) 2020-01-16T00:13:48.203 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:13:48.222 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:13:48.240 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:48.238+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:48.241 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:13:48.261 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:13:48.283 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:13:48.286 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:48.286 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:13:48.286 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:13:48.286 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:13:48.286 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.286 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:923 3.8 3:113947f2:::smithi12313934-12:head [write 1564138~470341] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:48.287 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.869292+0000", 2020-01-16T00:13:48.287 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 70.414745289999999, 2020-01-16T00:13:48.287 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017661620999999999, 2020-01-16T00:13:48.287 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:48.287 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:48.287 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:48.287 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:48.287 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:48.288 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 923 2020-01-16T00:13:48.288 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.288 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:48.288 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.288 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:48.288 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869292+0000", 2020-01-16T00:13:48.288 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:48.288 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.289 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.289 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:48.289 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869292+0000", 2020-01-16T00:13:48.289 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:48.289 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.289 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.289 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:48.289 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869291+0000", 2020-01-16T00:13:48.290 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00049842899999999999 2020-01-16T00:13:48.290 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.290 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.290 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:48.290 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869789+0000", 2020-01-16T00:13:48.290 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1014999999999999e-05 2020-01-16T00:13:48.290 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.291 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.291 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:48.291 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869830+0000", 2020-01-16T00:13:48.291 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7880000000000007e-06 2020-01-16T00:13:48.291 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.291 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.291 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:48.291 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869838+0000", 2020-01-16T00:13:48.292 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8346999999999997e-05 2020-01-16T00:13:48.292 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.292 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.292 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.292 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869876+0000", 2020-01-16T00:13:48.292 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9726999999999999e-05 2020-01-16T00:13:48.292 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.292 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.293 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:48.293 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869936+0000", 2020-01-16T00:13:48.293 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003434917 2020-01-16T00:13:48.293 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.293 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.293 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.293 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.873371+0000", 2020-01-16T00:13:48.293 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118808 2020-01-16T00:13:48.294 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.294 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.294 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:48.294 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.873490+0000", 2020-01-16T00:13:48.294 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036712910000000001 2020-01-16T00:13:48.294 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.294 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.294 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.295 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877161+0000", 2020-01-16T00:13:48.295 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9909999999999999e-05 2020-01-16T00:13:48.295 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.295 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.295 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:48.295 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877241+0000", 2020-01-16T00:13:48.295 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031619970000000002 2020-01-16T00:13:48.295 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.296 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.296 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.296 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880403+0000", 2020-01-16T00:13:48.296 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013071099999999999 2020-01-16T00:13:48.296 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.296 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.296 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:48.296 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880534+0000", 2020-01-16T00:13:48.297 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00032823500000000002 2020-01-16T00:13:48.297 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.297 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.297 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1,7", 2020-01-16T00:13:48.297 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880862+0000", 2020-01-16T00:13:48.297 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00513862 2020-01-16T00:13:48.297 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.309 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.309 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:48.309 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886001+0000", 2020-01-16T00:13:48.310 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8869000000000001e-05 2020-01-16T00:13:48.310 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.310 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.310 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:48.310 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886030+0000", 2020-01-16T00:13:48.310 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00081600200000000003 2020-01-16T00:13:48.310 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.311 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.311 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:48.311 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886846+0000", 2020-01-16T00:13:48.311 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4095999999999998e-05 2020-01-16T00:13:48.311 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.311 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.312 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:48.312 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886900+0000", 2020-01-16T00:13:48.312 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.448e-05 2020-01-16T00:13:48.312 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.312 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.312 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:48.312 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886954+0000", 2020-01-16T00:13:48.312 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017661620999999999 2020-01-16T00:13:48.313 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.313 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:48.313 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.313 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.313 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.313 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2226 3.7 3:e1be6d3b:::smithi12313934-24:head [setxattr _header (56) in=7b,truncate 2100908] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:13:48.313 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.980217+0000", 2020-01-16T00:13:48.313 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 42.303820625, 2020-01-16T00:13:48.314 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023113018999999999, 2020-01-16T00:13:48.314 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:48.314 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:48.314 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:48.314 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:48.314 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:48.314 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2226 2020-01-16T00:13:48.314 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.315 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:48.315 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.315 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:48.315 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980217+0000", 2020-01-16T00:13:48.315 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:48.315 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.315 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.315 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:48.316 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980217+0000", 2020-01-16T00:13:48.316 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:48.316 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.316 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.316 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:48.316 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980216+0000", 2020-01-16T00:13:48.316 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5049999999999998e-06 2020-01-16T00:13:48.316 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.317 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.317 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:48.317 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980222+0000", 2020-01-16T00:13:48.317 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5654999999999998e-05 2020-01-16T00:13:48.317 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.317 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.317 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:48.318 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980237+0000", 2020-01-16T00:13:48.318 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9740000000000004e-06 2020-01-16T00:13:48.318 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.318 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.318 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:48.318 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980241+0000", 2020-01-16T00:13:48.318 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002947776 2020-01-16T00:13:48.318 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.319 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.319 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.319 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.983189+0000", 2020-01-16T00:13:48.320 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0027099400000000001 2020-01-16T00:13:48.320 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.320 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.320 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:48.320 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985899+0000", 2020-01-16T00:13:48.320 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014275589999999999 2020-01-16T00:13:48.320 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.320 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.321 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.321 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.000175+0000", 2020-01-16T00:13:48.321 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011211 2020-01-16T00:13:48.321 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.321 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.321 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:48.321 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.000287+0000", 2020-01-16T00:13:48.322 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000229886 2020-01-16T00:13:48.322 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.322 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.322 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,7", 2020-01-16T00:13:48.322 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.000517+0000", 2020-01-16T00:13:48.322 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00095715399999999999 2020-01-16T00:13:48.322 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.322 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.323 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:48.323 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.001474+0000", 2020-01-16T00:13:48.323 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001341512 2020-01-16T00:13:48.323 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.323 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.323 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:48.323 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.002815+0000", 2020-01-16T00:13:48.323 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035789099999999999 2020-01-16T00:13:48.324 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.324 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.324 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:48.324 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003173+0000", 2020-01-16T00:13:48.324 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8571000000000005e-05 2020-01-16T00:13:48.324 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.324 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.324 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:48.325 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003242+0000", 2020-01-16T00:13:48.325 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8293000000000003e-05 2020-01-16T00:13:48.325 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.325 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.325 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:48.325 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003330+0000", 2020-01-16T00:13:48.325 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023113018999999999 2020-01-16T00:13:48.326 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.326 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:48.326 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.326 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.326 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.326 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2227 3.7 3:e1be6d3b:::smithi12313934-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e35)", 2020-01-16T00:13:48.326 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:13:48.326 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 42.303773372000002, 2020-01-16T00:13:48.327 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023303356000000001, 2020-01-16T00:13:48.327 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:48.327 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:48.327 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:48.327 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:48.327 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:48.327 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2227 2020-01-16T00:13:48.327 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.328 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:48.328 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.328 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:48.328 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:13:48.328 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:48.328 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.328 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.328 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:48.329 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:13:48.329 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:48.329 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.329 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.329 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:48.329 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:13:48.329 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3079999999999999e-06 2020-01-16T00:13:48.329 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.330 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.330 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:48.331 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980267+0000", 2020-01-16T00:13:48.331 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9290000000000001e-06 2020-01-16T00:13:48.331 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.331 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.331 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:48.331 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980274+0000", 2020-01-16T00:13:48.331 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7400000000000002e-06 2020-01-16T00:13:48.332 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.332 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.332 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:48.332 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980278+0000", 2020-01-16T00:13:48.332 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056624300000000004 2020-01-16T00:13:48.332 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.332 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.332 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.333 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985940+0000", 2020-01-16T00:13:48.333 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4261000000000002e-05 2020-01-16T00:13:48.333 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.333 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.333 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:48.333 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985985+0000", 2020-01-16T00:13:48.333 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015087676 2020-01-16T00:13:48.334 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.334 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.334 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.334 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.001072+0000", 2020-01-16T00:13:48.334 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011960600000000001 2020-01-16T00:13:48.334 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.334 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.334 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:48.335 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.001192+0000", 2020-01-16T00:13:48.335 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021718309999999999 2020-01-16T00:13:48.335 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.335 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.335 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.335 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003364+0000", 2020-01-16T00:13:48.335 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000102855 2020-01-16T00:13:48.336 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.336 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.336 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:48.336 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003467+0000", 2020-01-16T00:13:48.336 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010122499999999999 2020-01-16T00:13:48.336 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.336 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.336 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:48.337 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003568+0000", 2020-01-16T00:13:48.337 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023303356000000001 2020-01-16T00:13:48.337 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.337 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:48.337 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.337 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.337 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.338 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2233 3.2 3:4f8fe446:::smithi12313934-48:head [setxattr _header (56) in=7b,truncate 2117715] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:13:48.338 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.993103+0000", 2020-01-16T00:13:48.338 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 42.290935150000003, 2020-01-16T00:13:48.338 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017951763999999999, 2020-01-16T00:13:48.338 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:48.338 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:48.338 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:48.338 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:48.339 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:48.339 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2233 2020-01-16T00:13:48.339 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.339 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:48.339 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.339 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:48.339 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993103+0000", 2020-01-16T00:13:48.340 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:48.340 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.340 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.340 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:48.340 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993103+0000", 2020-01-16T00:13:48.340 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:48.340 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.341 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.341 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:48.341 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993102+0000", 2020-01-16T00:13:48.341 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9369999999999993e-06 2020-01-16T00:13:48.341 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.341 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.341 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:48.342 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993111+0000", 2020-01-16T00:13:48.342 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3340000000000002e-06 2020-01-16T00:13:48.342 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.342 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.342 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:48.342 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993119+0000", 2020-01-16T00:13:48.342 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1769999999999997e-06 2020-01-16T00:13:48.342 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.343 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.343 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:48.343 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993124+0000", 2020-01-16T00:13:48.343 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001601304 2020-01-16T00:13:48.343 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.343 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.343 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.344 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.994725+0000", 2020-01-16T00:13:48.344 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048777250000000003 2020-01-16T00:13:48.344 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.344 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.344 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:48.344 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999603+0000", 2020-01-16T00:13:48.345 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0099793880000000005 2020-01-16T00:13:48.345 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.345 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.345 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.345 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.009583+0000", 2020-01-16T00:13:48.345 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000108207 2020-01-16T00:13:48.345 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.346 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.346 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:48.346 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.009691+0000", 2020-01-16T00:13:48.346 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012862999999999999 2020-01-16T00:13:48.346 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.346 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.346 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:48.347 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.009819+0000", 2020-01-16T00:13:48.347 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00077947599999999998 2020-01-16T00:13:48.347 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.347 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.347 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:48.347 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010599+0000", 2020-01-16T00:13:48.347 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000221282 2020-01-16T00:13:48.347 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.348 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.348 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:48.348 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010820+0000", 2020-01-16T00:13:48.348 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7021999999999995e-05 2020-01-16T00:13:48.348 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.348 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.348 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:48.348 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010907+0000", 2020-01-16T00:13:48.349 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9815999999999997e-05 2020-01-16T00:13:48.349 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.349 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.349 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:48.349 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010977+0000", 2020-01-16T00:13:48.349 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7453999999999994e-05 2020-01-16T00:13:48.349 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.350 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.350 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:48.350 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.011054+0000", 2020-01-16T00:13:48.350 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017951763999999999 2020-01-16T00:13:48.350 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.350 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:48.350 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.351 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.351 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.351 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2234 3.2 3:4f8fe446:::smithi12313934-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e35)", 2020-01-16T00:13:48.351 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.993135+0000", 2020-01-16T00:13:48.351 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 42.290902697, 2020-01-16T00:13:48.351 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018067916999999999, 2020-01-16T00:13:48.351 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:48.352 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:48.352 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:48.352 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:48.352 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:48.352 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2234 2020-01-16T00:13:48.352 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.352 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:48.352 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.353 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:48.353 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993135+0000", 2020-01-16T00:13:48.353 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:48.353 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.353 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.353 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:48.353 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993135+0000", 2020-01-16T00:13:48.354 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:48.354 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.354 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.354 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:48.354 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993134+0000", 2020-01-16T00:13:48.354 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5630000000000002e-06 2020-01-16T00:13:48.354 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.355 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.355 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:48.355 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993139+0000", 2020-01-16T00:13:48.355 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1730000000000003e-06 2020-01-16T00:13:48.355 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.355 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.355 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:48.356 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993144+0000", 2020-01-16T00:13:48.356 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.937e-06 2020-01-16T00:13:48.356 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.356 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.356 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:48.356 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993147+0000", 2020-01-16T00:13:48.356 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0065164000000000003 2020-01-16T00:13:48.356 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.357 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.357 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.357 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999663+0000", 2020-01-16T00:13:48.357 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6215000000000002e-05 2020-01-16T00:13:48.357 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.357 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.357 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:48.358 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999729+0000", 2020-01-16T00:13:48.358 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010475887999999999 2020-01-16T00:13:48.358 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.358 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.358 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.358 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010205+0000", 2020-01-16T00:13:48.358 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119471 2020-01-16T00:13:48.358 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.359 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.359 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:48.359 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010325+0000", 2020-01-16T00:13:48.359 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00075498799999999995 2020-01-16T00:13:48.359 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.359 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.359 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.360 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.011080+0000", 2020-01-16T00:13:48.360 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7904e-05 2020-01-16T00:13:48.360 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.360 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.360 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:48.360 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.011138+0000", 2020-01-16T00:13:48.360 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5273999999999999e-05 2020-01-16T00:13:48.361 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.361 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.361 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:48.361 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.011203+0000", 2020-01-16T00:13:48.361 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018067916999999999 2020-01-16T00:13:48.361 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.361 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:48.361 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.362 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.362 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.362 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2324 3.7 3:e1be6d3b:::smithi12313934-24:head [setxattr _header (56) in=7b,truncate 2336206] snapc 0=[] ondisk+write+known_if_redirected e36)", 2020-01-16T00:13:48.362 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.550846+0000", 2020-01-16T00:13:48.362 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 40.733192111000001, 2020-01-16T00:13:48.362 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018364742, 2020-01-16T00:13:48.362 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:48.363 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:48.363 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:48.363 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:48.363 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:48.363 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2324 2020-01-16T00:13:48.363 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.363 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:48.363 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.364 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:48.364 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550846+0000", 2020-01-16T00:13:48.364 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:48.364 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.364 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.364 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:48.364 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550846+0000", 2020-01-16T00:13:48.365 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:48.365 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.365 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.365 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:48.365 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550845+0000", 2020-01-16T00:13:48.365 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1829999999999999e-06 2020-01-16T00:13:48.365 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.365 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.366 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:48.366 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550851+0000", 2020-01-16T00:13:48.366 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1925e-05 2020-01-16T00:13:48.366 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.366 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.366 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:48.366 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550863+0000", 2020-01-16T00:13:48.367 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1139999999999997e-06 2020-01-16T00:13:48.367 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.367 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.367 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:48.367 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550868+0000", 2020-01-16T00:13:48.367 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001388423 2020-01-16T00:13:48.367 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.368 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.368 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.368 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.552256+0000", 2020-01-16T00:13:48.368 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0034246540000000001 2020-01-16T00:13:48.368 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.368 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.368 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:48.368 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555681+0000", 2020-01-16T00:13:48.369 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011882762999999999 2020-01-16T00:13:48.369 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.369 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.369 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.369 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.567564+0000", 2020-01-16T00:13:48.369 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015010800000000001 2020-01-16T00:13:48.369 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.370 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.370 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:48.370 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.567714+0000", 2020-01-16T00:13:48.370 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011527999999999999 2020-01-16T00:13:48.370 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.370 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.370 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,7", 2020-01-16T00:13:48.370 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.567829+0000", 2020-01-16T00:13:48.371 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00039888499999999997 2020-01-16T00:13:48.371 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.371 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.371 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:48.371 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568228+0000", 2020-01-16T00:13:48.371 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00054254599999999998 2020-01-16T00:13:48.371 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.371 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.372 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:48.372 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568771+0000", 2020-01-16T00:13:48.372 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035206400000000002 2020-01-16T00:13:48.372 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.372 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.372 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:48.372 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569123+0000", 2020-01-16T00:13:48.372 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1262e-05 2020-01-16T00:13:48.373 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.373 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.373 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:48.373 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569154+0000", 2020-01-16T00:13:48.373 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6499000000000002e-05 2020-01-16T00:13:48.373 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.373 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.374 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:48.374 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569210+0000", 2020-01-16T00:13:48.374 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018364742 2020-01-16T00:13:48.374 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.374 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:48.374 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.374 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.375 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.375 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2325 3.7 3:e1be6d3b:::smithi12313934-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e36)", 2020-01-16T00:13:48.375 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.550877+0000", 2020-01-16T00:13:48.375 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 40.733161045000003, 2020-01-16T00:13:48.375 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018565154, 2020-01-16T00:13:48.375 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:48.375 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:48.375 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:48.376 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:48.376 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:48.376 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2325 2020-01-16T00:13:48.376 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.376 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:48.376 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.376 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:48.377 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550877+0000", 2020-01-16T00:13:48.377 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:48.377 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.377 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.377 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:48.377 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550877+0000", 2020-01-16T00:13:48.377 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:48.378 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.378 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.378 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:48.378 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550876+0000", 2020-01-16T00:13:48.378 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1080000000000003e-06 2020-01-16T00:13:48.378 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.378 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.378 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:48.379 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550880+0000", 2020-01-16T00:13:48.379 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3269999999999997e-06 2020-01-16T00:13:48.379 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.379 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.379 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:48.379 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550886+0000", 2020-01-16T00:13:48.379 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.386e-06 2020-01-16T00:13:48.380 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.380 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.380 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:48.380 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550891+0000", 2020-01-16T00:13:48.380 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048231410000000004 2020-01-16T00:13:48.380 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.380 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.380 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.381 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555714+0000", 2020-01-16T00:13:48.381 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0071999999999999e-05 2020-01-16T00:13:48.381 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.381 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.381 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:48.381 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555744+0000", 2020-01-16T00:13:48.381 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012329005000000001 2020-01-16T00:13:48.382 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.382 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.382 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.382 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568073+0000", 2020-01-16T00:13:48.382 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9087999999999997e-05 2020-01-16T00:13:48.382 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.382 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.382 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:48.383 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568172+0000", 2020-01-16T00:13:48.383 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010718220000000001 2020-01-16T00:13:48.383 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.383 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.383 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.383 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569244+0000", 2020-01-16T00:13:48.383 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8199e-05 2020-01-16T00:13:48.384 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.384 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.384 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:48.384 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569342+0000", 2020-01-16T00:13:48.384 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9708000000000001e-05 2020-01-16T00:13:48.384 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.384 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.385 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:48.385 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569442+0000", 2020-01-16T00:13:48.385 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018565154 2020-01-16T00:13:48.385 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.385 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:48.385 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.385 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.385 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.386 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2800 3.2 3:42b7f20f:::smithi12313934-42:head [write 1395221~583781] snapc 0=[] ondisk+write+known_if_redirected e41)", 2020-01-16T00:13:48.386 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:17.256182+0000", 2020-01-16T00:13:48.386 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 31.027855505000002, 2020-01-16T00:13:48.386 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018509650999999998, 2020-01-16T00:13:48.386 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:48.386 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:48.386 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:48.387 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:48.387 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:48.387 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2800 2020-01-16T00:13:48.387 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.387 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:48.387 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.388 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:48.388 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256182+0000", 2020-01-16T00:13:48.388 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:48.388 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.388 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.388 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:48.388 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256182+0000", 2020-01-16T00:13:48.389 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:48.389 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.389 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.389 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:48.389 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256180+0000", 2020-01-16T00:13:48.389 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020514799999999999 2020-01-16T00:13:48.389 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.390 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.390 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:48.390 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256385+0000", 2020-01-16T00:13:48.390 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0288e-05 2020-01-16T00:13:48.390 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.390 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.390 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:48.390 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256406+0000", 2020-01-16T00:13:48.391 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7160000000000003e-06 2020-01-16T00:13:48.391 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.391 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.391 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:48.391 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256413+0000", 2020-01-16T00:13:48.391 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5293999999999998e-05 2020-01-16T00:13:48.391 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.392 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.392 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.392 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256449+0000", 2020-01-16T00:13:48.392 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001297737 2020-01-16T00:13:48.392 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.392 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.392 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:48.393 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.257746+0000", 2020-01-16T00:13:48.393 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0079866529999999998 2020-01-16T00:13:48.393 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.393 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.393 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.393 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.265733+0000", 2020-01-16T00:13:48.394 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4028000000000004e-05 2020-01-16T00:13:48.394 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.394 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.394 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:48.394 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.265797+0000", 2020-01-16T00:13:48.394 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011338899999999999 2020-01-16T00:13:48.394 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.395 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.395 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:48.395 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.265910+0000", 2020-01-16T00:13:48.395 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003372496 2020-01-16T00:13:48.395 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.396 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.396 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:48.396 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.269283+0000", 2020-01-16T00:13:48.396 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0053107959999999996 2020-01-16T00:13:48.396 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.396 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.397 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:48.397 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.274594+0000", 2020-01-16T00:13:48.397 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8000999999999998e-05 2020-01-16T00:13:48.397 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.397 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.397 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:48.397 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.274632+0000", 2020-01-16T00:13:48.398 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0265e-05 2020-01-16T00:13:48.398 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.398 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.398 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:48.398 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.274692+0000", 2020-01-16T00:13:48.398 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018509650999999998 2020-01-16T00:13:48.399 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.399 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:48.399 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.399 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.399 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.399 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2971 3.c 3:3025546b:::smithi12313934-15:head [write 1424565~609148] snapc 0=[] ondisk+write+known_if_redirected e42)", 2020-01-16T00:13:48.399 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:21.839711+0000", 2020-01-16T00:13:48.400 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 26.444326662000002, 2020-01-16T00:13:48.400 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017534376000000001, 2020-01-16T00:13:48.400 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:48.400 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:48.400 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:48.400 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:48.400 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:48.401 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2971 2020-01-16T00:13:48.401 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.401 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:48.401 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.401 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:48.401 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839711+0000", 2020-01-16T00:13:48.401 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:48.401 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.402 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.402 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:48.402 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839711+0000", 2020-01-16T00:13:48.402 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:48.402 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.402 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.402 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:48.403 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839710+0000", 2020-01-16T00:13:48.403 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017658600000000001 2020-01-16T00:13:48.403 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.403 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.403 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:48.403 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839886+0000", 2020-01-16T00:13:48.403 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0492000000000001e-05 2020-01-16T00:13:48.404 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.404 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.404 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:48.404 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839897+0000", 2020-01-16T00:13:48.404 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.019e-06 2020-01-16T00:13:48.404 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.404 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.404 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:48.405 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839901+0000", 2020-01-16T00:13:48.405 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5572000000000001e-05 2020-01-16T00:13:48.405 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.405 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.405 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.405 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839976+0000", 2020-01-16T00:13:48.405 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034616099999999999 2020-01-16T00:13:48.406 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.406 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.406 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:48.406 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.840323+0000", 2020-01-16T00:13:48.406 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.01246249 2020-01-16T00:13:48.406 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.406 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.406 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.407 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.852785+0000", 2020-01-16T00:13:48.407 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014474799999999999 2020-01-16T00:13:48.407 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.407 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.407 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:48.407 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.852930+0000", 2020-01-16T00:13:48.408 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000213907 2020-01-16T00:13:48.408 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.408 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.408 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:48.408 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.853144+0000", 2020-01-16T00:13:48.408 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0028426979999999998 2020-01-16T00:13:48.408 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.409 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.409 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:48.409 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.855986+0000", 2020-01-16T00:13:48.409 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010694280000000001 2020-01-16T00:13:48.409 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.409 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.409 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:48.410 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.857056+0000", 2020-01-16T00:13:48.410 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8935000000000003e-05 2020-01-16T00:13:48.410 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.410 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.410 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:48.410 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.857125+0000", 2020-01-16T00:13:48.410 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000120721 2020-01-16T00:13:48.411 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.411 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.411 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:48.411 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.857245+0000", 2020-01-16T00:13:48.411 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017534376000000001 2020-01-16T00:13:48.411 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.411 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:48.411 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.412 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.412 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.412 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2972 3.c 3:3025546b:::smithi12313934-15:head [setxattr _header (56) in=7b,truncate 2033713] snapc 0=[] ondisk+write+known_if_redirected e42)", 2020-01-16T00:13:48.412 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:21.839923+0000", 2020-01-16T00:13:48.412 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 26.444114398, 2020-01-16T00:13:48.412 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018367108, 2020-01-16T00:13:48.412 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:48.413 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:48.413 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:48.413 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:48.413 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:48.413 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2972 2020-01-16T00:13:48.413 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.413 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:48.413 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.414 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:48.414 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839923+0000", 2020-01-16T00:13:48.414 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:48.414 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.414 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.414 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:48.414 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839923+0000", 2020-01-16T00:13:48.415 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:48.415 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.415 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.415 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:48.415 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839923+0000", 2020-01-16T00:13:48.415 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0629999999999999e-06 2020-01-16T00:13:48.415 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.416 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.416 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:48.416 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839927+0000", 2020-01-16T00:13:48.416 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1980000000000003e-06 2020-01-16T00:13:48.416 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.416 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.416 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:48.417 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839931+0000", 2020-01-16T00:13:48.417 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5979999999999999e-06 2020-01-16T00:13:48.417 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.417 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.417 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:48.417 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839934+0000", 2020-01-16T00:13:48.417 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00042258500000000001 2020-01-16T00:13:48.417 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.418 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.418 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.418 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.840356+0000", 2020-01-16T00:13:48.418 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4919000000000002e-05 2020-01-16T00:13:48.418 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.418 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.419 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:48.419 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.840391+0000", 2020-01-16T00:13:48.419 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013783103 2020-01-16T00:13:48.419 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.419 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.419 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.419 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.854174+0000", 2020-01-16T00:13:48.420 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3455999999999995e-05 2020-01-16T00:13:48.420 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.420 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.420 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:48.420 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.854258+0000", 2020-01-16T00:13:48.420 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011060700000000001 2020-01-16T00:13:48.420 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.421 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.421 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:48.421 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.854368+0000", 2020-01-16T00:13:48.421 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020642500000000001 2020-01-16T00:13:48.421 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.421 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.421 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:48.422 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.856433+0000", 2020-01-16T00:13:48.422 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0016284820000000001 2020-01-16T00:13:48.422 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.422 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.422 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:48.422 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858061+0000", 2020-01-16T00:13:48.422 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0663999999999998e-05 2020-01-16T00:13:48.422 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.423 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.423 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:48.423 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858152+0000", 2020-01-16T00:13:48.423 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013882900000000001 2020-01-16T00:13:48.423 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.423 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.423 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:48.423 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858290+0000", 2020-01-16T00:13:48.424 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018367108 2020-01-16T00:13:48.424 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.424 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:48.424 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.424 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.424 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.424 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2973 3.c 3:3025546b:::smithi12313934-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e42)", 2020-01-16T00:13:48.425 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:21.839961+0000", 2020-01-16T00:13:48.425 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 26.444076903999999, 2020-01-16T00:13:48.425 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018691670000000001, 2020-01-16T00:13:48.425 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:48.425 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:48.425 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:48.426 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:48.426 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:48.426 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2973 2020-01-16T00:13:48.426 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.426 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:48.426 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.426 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:48.427 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839961+0000", 2020-01-16T00:13:48.427 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:48.427 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.427 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.427 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:48.427 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839961+0000", 2020-01-16T00:13:48.427 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:48.427 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.428 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.428 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:48.428 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839960+0000", 2020-01-16T00:13:48.428 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4060000000000001e-06 2020-01-16T00:13:48.428 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.428 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.428 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:48.429 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839964+0000", 2020-01-16T00:13:48.429 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5412000000000001e-05 2020-01-16T00:13:48.429 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.429 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.429 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:48.429 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839979+0000", 2020-01-16T00:13:48.430 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.444e-06 2020-01-16T00:13:48.430 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.430 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.430 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:48.430 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839983+0000", 2020-01-16T00:13:48.430 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000430028 2020-01-16T00:13:48.430 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.431 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.431 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.431 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.840413+0000", 2020-01-16T00:13:48.431 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9270999999999997e-05 2020-01-16T00:13:48.431 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.431 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.431 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:48.431 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.840452+0000", 2020-01-16T00:13:48.432 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014853975 2020-01-16T00:13:48.432 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.432 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.432 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.432 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.855306+0000", 2020-01-16T00:13:48.432 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011098400000000001 2020-01-16T00:13:48.432 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.432 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.433 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:48.433 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.855417+0000", 2020-01-16T00:13:48.433 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029108060000000002 2020-01-16T00:13:48.433 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.433 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.433 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.433 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858328+0000", 2020-01-16T00:13:48.434 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000144697 2020-01-16T00:13:48.434 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.434 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.434 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:48.434 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858472+0000", 2020-01-16T00:13:48.434 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018011899999999999 2020-01-16T00:13:48.434 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.435 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.435 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:48.435 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858653+0000", 2020-01-16T00:13:48.435 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018691670000000001 2020-01-16T00:13:48.435 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.435 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:48.435 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.435 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.436 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.436 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3295 3.2 3:42b7f20f:::smithi12313934-42:head [write 1982691~499457] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:13:48.436 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:30.744201+0000", 2020-01-16T00:13:48.436 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 17.539836628, 2020-01-16T00:13:48.436 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018346738000000001, 2020-01-16T00:13:48.436 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:48.436 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:48.437 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:48.437 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:48.437 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:48.437 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3295 2020-01-16T00:13:48.437 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.437 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:48.437 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.438 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:48.438 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744201+0000", 2020-01-16T00:13:48.438 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:48.438 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.438 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.438 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:48.438 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744201+0000", 2020-01-16T00:13:48.439 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:48.439 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.439 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.439 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:48.439 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744198+0000", 2020-01-16T00:13:48.439 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00030375699999999999 2020-01-16T00:13:48.439 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.439 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.440 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:48.440 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744502+0000", 2020-01-16T00:13:48.440 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0121e-05 2020-01-16T00:13:48.440 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.440 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.440 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:48.440 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744512+0000", 2020-01-16T00:13:48.441 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2293000000000001e-05 2020-01-16T00:13:48.441 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.441 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.441 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:48.441 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744524+0000", 2020-01-16T00:13:48.441 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000274837 2020-01-16T00:13:48.441 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.441 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.442 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.442 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744799+0000", 2020-01-16T00:13:48.442 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036951879999999999 2020-01-16T00:13:48.442 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.442 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.442 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:48.442 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748494+0000", 2020-01-16T00:13:48.443 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010091493 2020-01-16T00:13:48.443 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.443 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.443 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.443 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.758586+0000", 2020-01-16T00:13:48.443 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011674400000000001 2020-01-16T00:13:48.443 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.444 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.444 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:48.444 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.758703+0000", 2020-01-16T00:13:48.444 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021412699999999999 2020-01-16T00:13:48.444 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.444 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.445 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:48.445 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.758917+0000", 2020-01-16T00:13:48.445 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0022519340000000001 2020-01-16T00:13:48.445 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.445 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.445 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:48.445 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.761169+0000", 2020-01-16T00:13:48.446 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001240891 2020-01-16T00:13:48.446 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.446 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.446 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:48.446 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.762410+0000", 2020-01-16T00:13:48.446 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6942e-05 2020-01-16T00:13:48.446 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.447 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.447 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:48.447 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.762447+0000", 2020-01-16T00:13:48.447 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000101358 2020-01-16T00:13:48.447 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.447 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.447 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:48.447 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.762548+0000", 2020-01-16T00:13:48.448 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018346738000000001 2020-01-16T00:13:48.448 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.448 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:48.448 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.448 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.448 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.448 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3296 3.2 3:42b7f20f:::smithi12313934-42:head [setxattr _header (56) in=7b,truncate 2739607] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:13:48.449 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:30.744532+0000", 2020-01-16T00:13:48.449 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 17.539505895000001, 2020-01-16T00:13:48.449 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018472374, 2020-01-16T00:13:48.449 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:48.449 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:48.449 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:48.449 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:48.449 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:48.450 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3296 2020-01-16T00:13:48.450 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.450 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:48.450 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.450 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:48.450 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744532+0000", 2020-01-16T00:13:48.450 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:48.451 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.451 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.451 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:48.451 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744532+0000", 2020-01-16T00:13:48.451 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:48.451 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.451 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.452 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:48.452 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744531+0000", 2020-01-16T00:13:48.452 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7059999999999993e-06 2020-01-16T00:13:48.452 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.452 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.452 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:48.452 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744540+0000", 2020-01-16T00:13:48.452 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8589999999999998e-06 2020-01-16T00:13:48.453 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.453 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.453 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:48.453 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744544+0000", 2020-01-16T00:13:48.453 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7609999999999999e-06 2020-01-16T00:13:48.453 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.453 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.453 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:48.454 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744546+0000", 2020-01-16T00:13:48.454 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039933670000000003 2020-01-16T00:13:48.454 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.454 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.454 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.454 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748540+0000", 2020-01-16T00:13:48.454 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000110281 2020-01-16T00:13:48.454 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.455 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.455 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:48.455 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748650+0000", 2020-01-16T00:13:48.455 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.01160457 2020-01-16T00:13:48.455 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.455 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.455 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.456 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.760255+0000", 2020-01-16T00:13:48.456 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000124714 2020-01-16T00:13:48.456 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.456 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.456 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:48.456 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.760379+0000", 2020-01-16T00:13:48.456 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022099700000000001 2020-01-16T00:13:48.457 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.457 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.457 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:48.457 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.760600+0000", 2020-01-16T00:13:48.457 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00057843300000000005 2020-01-16T00:13:48.457 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.457 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.458 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:48.458 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.761179+0000", 2020-01-16T00:13:48.458 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00161477 2020-01-16T00:13:48.458 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.458 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.458 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:48.458 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.762794+0000", 2020-01-16T00:13:48.459 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4408999999999999e-05 2020-01-16T00:13:48.459 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.459 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.459 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:48.459 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.762858+0000", 2020-01-16T00:13:48.459 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014634199999999999 2020-01-16T00:13:48.459 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.459 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.460 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:48.460 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.763004+0000", 2020-01-16T00:13:48.460 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018472374 2020-01-16T00:13:48.460 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.460 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:48.460 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.460 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.461 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.461 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3297 3.2 3:42b7f20f:::smithi12313934-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e45)", 2020-01-16T00:13:48.461 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:30.744551+0000", 2020-01-16T00:13:48.461 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 17.539486924999999, 2020-01-16T00:13:48.461 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018660209000000001, 2020-01-16T00:13:48.461 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:48.461 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:48.461 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:48.462 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:48.462 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:48.462 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3297 2020-01-16T00:13:48.462 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.462 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:48.462 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.462 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:48.462 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744551+0000", 2020-01-16T00:13:48.463 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:48.463 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.463 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.463 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:48.463 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744551+0000", 2020-01-16T00:13:48.463 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:48.463 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.464 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.464 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:48.464 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744550+0000", 2020-01-16T00:13:48.464 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5279999999999999e-06 2020-01-16T00:13:48.464 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.464 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.464 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:48.464 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744554+0000", 2020-01-16T00:13:48.465 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8420000000000001e-06 2020-01-16T00:13:48.465 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.465 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.465 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:48.465 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744559+0000", 2020-01-16T00:13:48.465 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.965e-06 2020-01-16T00:13:48.465 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.466 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.466 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:48.466 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744562+0000", 2020-01-16T00:13:48.466 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041215380000000001 2020-01-16T00:13:48.466 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.466 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.466 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.467 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748684+0000", 2020-01-16T00:13:48.467 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8165999999999997e-05 2020-01-16T00:13:48.467 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.467 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.467 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:48.467 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748762+0000", 2020-01-16T00:13:48.467 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012136152000000001 2020-01-16T00:13:48.467 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.468 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.468 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.468 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.760898+0000", 2020-01-16T00:13:48.468 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000114833 2020-01-16T00:13:48.468 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.468 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.468 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:48.469 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.761013+0000", 2020-01-16T00:13:48.469 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002017015 2020-01-16T00:13:48.469 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.469 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.469 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.469 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.763030+0000", 2020-01-16T00:13:48.469 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5741000000000006e-05 2020-01-16T00:13:48.470 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.470 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.470 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:48.470 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.763096+0000", 2020-01-16T00:13:48.470 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011519900000000001 2020-01-16T00:13:48.470 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.470 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.470 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:48.471 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.763211+0000", 2020-01-16T00:13:48.471 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018660209000000001 2020-01-16T00:13:48.471 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.471 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:48.471 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.471 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.471 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.471 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3481 3.2 3:4f8fe446:::smithi12313934-48:head [setxattr _header (56) in=7b,truncate 3193396] snapc 0=[] ondisk+write+known_if_redirected e47)", 2020-01-16T00:13:48.472 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:36.349654+0000", 2020-01-16T00:13:48.472 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 11.934383472, 2020-01-16T00:13:48.472 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017529643000000001, 2020-01-16T00:13:48.472 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:48.472 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:48.472 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:48.472 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:48.473 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:48.473 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3481 2020-01-16T00:13:48.473 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.473 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:48.473 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.473 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:48.473 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349654+0000", 2020-01-16T00:13:48.473 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:48.474 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.474 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.474 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:48.474 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349654+0000", 2020-01-16T00:13:48.474 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:48.474 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.474 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.475 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:48.475 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349654+0000", 2020-01-16T00:13:48.475 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2519999999999996e-06 2020-01-16T00:13:48.475 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.475 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.475 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:48.475 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349660+0000", 2020-01-16T00:13:48.475 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2789999999999999e-06 2020-01-16T00:13:48.476 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.476 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.476 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:48.476 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349663+0000", 2020-01-16T00:13:48.476 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2479999999999999e-06 2020-01-16T00:13:48.476 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.476 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.477 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:48.477 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349666+0000", 2020-01-16T00:13:48.477 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0026359180000000001 2020-01-16T00:13:48.477 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.477 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.477 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.477 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.352301+0000", 2020-01-16T00:13:48.477 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0035532799999999998 2020-01-16T00:13:48.478 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.478 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.478 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:48.478 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.355855+0000", 2020-01-16T00:13:48.478 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0099904050000000008 2020-01-16T00:13:48.478 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.478 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.479 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.479 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.365845+0000", 2020-01-16T00:13:48.479 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012439300000000001 2020-01-16T00:13:48.479 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.479 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.479 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:48.479 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.365970+0000", 2020-01-16T00:13:48.479 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00026105699999999998 2020-01-16T00:13:48.480 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.480 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.480 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:48.480 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.366231+0000", 2020-01-16T00:13:48.480 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00056610199999999997 2020-01-16T00:13:48.480 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.480 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.481 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:48.481 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.366797+0000", 2020-01-16T00:13:48.481 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000171996 2020-01-16T00:13:48.481 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.481 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.481 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:48.481 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.366969+0000", 2020-01-16T00:13:48.482 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5282000000000003e-05 2020-01-16T00:13:48.482 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.482 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.482 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:48.482 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.367034+0000", 2020-01-16T00:13:48.482 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014998100000000001 2020-01-16T00:13:48.482 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.482 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.483 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:48.483 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.367184+0000", 2020-01-16T00:13:48.483 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017529643000000001 2020-01-16T00:13:48.483 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.483 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:48.483 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.483 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.484 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.484 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3482 3.2 3:4f8fe446:::smithi12313934-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e47)", 2020-01-16T00:13:48.484 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:36.349724+0000", 2020-01-16T00:13:48.484 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 11.934313834999999, 2020-01-16T00:13:48.484 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017836446999999998, 2020-01-16T00:13:48.484 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:48.484 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:48.485 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:48.485 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:48.485 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:48.485 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3482 2020-01-16T00:13:48.485 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.485 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:48.485 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.486 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:48.486 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349724+0000", 2020-01-16T00:13:48.486 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:48.486 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.486 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.486 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:48.486 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349724+0000", 2020-01-16T00:13:48.487 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:48.487 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.487 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.487 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:48.487 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349723+0000", 2020-01-16T00:13:48.487 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9709999999999998e-06 2020-01-16T00:13:48.487 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.487 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.488 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:48.488 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349727+0000", 2020-01-16T00:13:48.488 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5419999999999999e-06 2020-01-16T00:13:48.488 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.488 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.488 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:48.489 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349732+0000", 2020-01-16T00:13:48.489 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.481e-06 2020-01-16T00:13:48.489 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.489 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.489 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:48.489 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349734+0000", 2020-01-16T00:13:48.489 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.006195028 2020-01-16T00:13:48.490 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.490 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.490 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.490 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.355929+0000", 2020-01-16T00:13:48.490 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1483999999999996e-05 2020-01-16T00:13:48.490 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.490 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.491 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:48.491 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.355991+0000", 2020-01-16T00:13:48.491 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010536415 2020-01-16T00:13:48.491 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.491 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.491 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.491 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.366527+0000", 2020-01-16T00:13:48.492 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8099e-05 2020-01-16T00:13:48.492 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.492 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.492 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:48.492 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.366595+0000", 2020-01-16T00:13:48.492 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00063624500000000002 2020-01-16T00:13:48.492 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.493 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.493 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.493 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.367231+0000", 2020-01-16T00:13:48.493 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000121196 2020-01-16T00:13:48.493 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.493 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.493 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:48.493 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.367353+0000", 2020-01-16T00:13:48.494 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020780500000000001 2020-01-16T00:13:48.494 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.494 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.494 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:48.494 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.367560+0000", 2020-01-16T00:13:48.495 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017836446999999998 2020-01-16T00:13:48.495 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.495 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:48.495 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.495 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.495 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.495 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3810 3.7 3:efab040c:::smithi12313934-30:head [write 1837406~542493] snapc 0=[] ondisk+write+known_if_redirected e50)", 2020-01-16T00:13:48.496 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:43.768525+0000", 2020-01-16T00:13:48.496 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 4.5155127520000002, 2020-01-16T00:13:48.496 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018385839000000001, 2020-01-16T00:13:48.496 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:48.496 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:48.497 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:48.497 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:48.497 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:48.497 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3810 2020-01-16T00:13:48.497 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.498 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:48.498 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.498 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:48.498 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768525+0000", 2020-01-16T00:13:48.498 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:48.498 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.498 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.498 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:48.499 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768525+0000", 2020-01-16T00:13:48.499 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:48.499 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.499 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.499 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:48.499 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768523+0000", 2020-01-16T00:13:48.499 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000149512 2020-01-16T00:13:48.500 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.500 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.500 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:48.500 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768673+0000", 2020-01-16T00:13:48.500 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0732e-05 2020-01-16T00:13:48.500 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.500 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.501 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:48.501 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768684+0000", 2020-01-16T00:13:48.501 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3719999999999998e-06 2020-01-16T00:13:48.501 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.501 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.501 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:48.501 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768688+0000", 2020-01-16T00:13:48.501 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6513999999999994e-05 2020-01-16T00:13:48.502 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.502 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.502 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.502 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768754+0000", 2020-01-16T00:13:48.502 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00075867800000000004 2020-01-16T00:13:48.502 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.502 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.503 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:48.503 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.769513+0000", 2020-01-16T00:13:48.503 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012395692 2020-01-16T00:13:48.503 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.503 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.503 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.503 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.781909+0000", 2020-01-16T00:13:48.503 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118372 2020-01-16T00:13:48.504 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.504 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.504 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:48.504 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.782027+0000", 2020-01-16T00:13:48.504 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019909699999999999 2020-01-16T00:13:48.504 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.504 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.504 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,7", 2020-01-16T00:13:48.505 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.782226+0000", 2020-01-16T00:13:48.505 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0038778369999999999 2020-01-16T00:13:48.505 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.505 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.505 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:48.505 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786104+0000", 2020-01-16T00:13:48.505 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022391199999999999 2020-01-16T00:13:48.506 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.506 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.506 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:48.506 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786328+0000", 2020-01-16T00:13:48.506 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00048851299999999999 2020-01-16T00:13:48.506 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.506 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.507 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:48.507 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786817+0000", 2020-01-16T00:13:48.507 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3597999999999998e-05 2020-01-16T00:13:48.507 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.507 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.507 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:48.507 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786860+0000", 2020-01-16T00:13:48.508 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0741000000000001e-05 2020-01-16T00:13:48.508 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.508 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.508 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:48.508 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786911+0000", 2020-01-16T00:13:48.508 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018385839000000001 2020-01-16T00:13:48.509 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.509 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:48.509 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.509 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.509 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.509 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3863 3.2 3:4f8fe446:::smithi12313934-48:head [setxattr _header (56) in=7b,truncate 2151395] snapc 0=[] ondisk+write+known_if_redirected e50)", 2020-01-16T00:13:48.509 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:44.489284+0000", 2020-01-16T00:13:48.510 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 3.7947542799999998, 2020-01-16T00:13:48.510 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.026323742000000001, 2020-01-16T00:13:48.510 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:48.510 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:48.510 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:48.510 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:48.510 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:48.511 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3863 2020-01-16T00:13:48.511 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.511 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:48.511 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.511 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:48.511 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489284+0000", 2020-01-16T00:13:48.511 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:48.511 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.512 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.512 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:48.512 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489284+0000", 2020-01-16T00:13:48.512 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:48.512 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.512 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.512 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:48.513 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489282+0000", 2020-01-16T00:13:48.513 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6660000000000002e-06 2020-01-16T00:13:48.513 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.513 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.513 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:48.513 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489289+0000", 2020-01-16T00:13:48.513 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3069999999999999e-06 2020-01-16T00:13:48.513 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.514 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.514 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:48.514 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489295+0000", 2020-01-16T00:13:48.514 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6090000000000003e-06 2020-01-16T00:13:48.514 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.514 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.514 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:48.515 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489300+0000", 2020-01-16T00:13:48.515 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002242464 2020-01-16T00:13:48.515 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.515 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.515 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.515 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491542+0000", 2020-01-16T00:13:48.515 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00059546200000000005 2020-01-16T00:13:48.515 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.516 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.516 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:48.516 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.492138+0000", 2020-01-16T00:13:48.516 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018547999999999999 2020-01-16T00:13:48.516 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.516 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.516 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.517 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.510686+0000", 2020-01-16T00:13:48.517 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018590100000000001 2020-01-16T00:13:48.517 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.517 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.517 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:48.517 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.510872+0000", 2020-01-16T00:13:48.517 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035115299999999999 2020-01-16T00:13:48.517 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.518 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.518 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:48.518 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.511223+0000", 2020-01-16T00:13:48.518 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003765493 2020-01-16T00:13:48.518 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.518 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.519 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:48.519 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.514988+0000", 2020-01-16T00:13:48.519 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000432022 2020-01-16T00:13:48.519 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.519 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.519 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:48.520 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515421+0000", 2020-01-16T00:13:48.520 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2874000000000002e-05 2020-01-16T00:13:48.520 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.520 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.520 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:48.520 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515473+0000", 2020-01-16T00:13:48.520 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013385 2020-01-16T00:13:48.520 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.521 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.521 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:48.521 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515607+0000", 2020-01-16T00:13:48.521 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.026323742000000001 2020-01-16T00:13:48.521 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.521 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:48.521 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.522 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.522 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.522 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3864 3.2 3:4f8fe446:::smithi12313934-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e50)", 2020-01-16T00:13:48.522 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:44.491131+0000", 2020-01-16T00:13:48.522 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 3.792906882, 2020-01-16T00:13:48.522 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024813806000000001, 2020-01-16T00:13:48.522 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:48.523 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:48.523 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:48.523 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:48.523 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:48.523 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3864 2020-01-16T00:13:48.523 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.523 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:48.523 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.524 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:48.524 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491131+0000", 2020-01-16T00:13:48.524 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:48.524 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.524 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.524 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:48.524 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491131+0000", 2020-01-16T00:13:48.525 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:48.525 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.525 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.525 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:48.525 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491128+0000", 2020-01-16T00:13:48.525 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2053e-05 2020-01-16T00:13:48.526 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.526 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.526 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:48.526 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491140+0000", 2020-01-16T00:13:48.526 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5869999999999999e-05 2020-01-16T00:13:48.526 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.526 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.526 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:48.527 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491156+0000", 2020-01-16T00:13:48.527 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2571e-05 2020-01-16T00:13:48.527 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.527 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.527 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:48.527 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491168+0000", 2020-01-16T00:13:48.527 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001031045 2020-01-16T00:13:48.528 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.528 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.528 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.528 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.492199+0000", 2020-01-16T00:13:48.528 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000170865 2020-01-16T00:13:48.528 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.528 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.529 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:48.529 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.492370+0000", 2020-01-16T00:13:48.529 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020180638000000001 2020-01-16T00:13:48.529 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.529 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.529 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.529 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.512551+0000", 2020-01-16T00:13:48.530 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012478599999999999 2020-01-16T00:13:48.530 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.530 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.530 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:48.530 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.512676+0000", 2020-01-16T00:13:48.530 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029628660000000002 2020-01-16T00:13:48.531 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.531 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.531 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.531 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515639+0000", 2020-01-16T00:13:48.531 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012250799999999999 2020-01-16T00:13:48.531 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.531 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.532 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:48.532 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515761+0000", 2020-01-16T00:13:48.532 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018356399999999999 2020-01-16T00:13:48.532 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.532 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.532 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:48.532 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515945+0000", 2020-01-16T00:13:48.532 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024813806000000001 2020-01-16T00:13:48.533 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.533 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:48.533 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.533 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.533 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.533 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3874 3.c 3:378f3722:::smithi12313934-50:head [write 568202~701253,stat] snapc 0=[] ondisk+write+known_if_redirected e50)", 2020-01-16T00:13:48.533 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:44.497041+0000", 2020-01-16T00:13:48.534 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 3.7869972509999998, 2020-01-16T00:13:48.534 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019602525999999999, 2020-01-16T00:13:48.534 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:48.534 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:48.534 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:48.534 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:48.534 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:48.535 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3874 2020-01-16T00:13:48.535 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.535 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:48.535 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.535 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:48.535 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497041+0000", 2020-01-16T00:13:48.535 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:48.536 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.536 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.536 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:48.536 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497041+0000", 2020-01-16T00:13:48.536 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999962 2020-01-16T00:13:48.536 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.536 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.536 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:48.537 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497037+0000", 2020-01-16T00:13:48.537 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00027168899999999998 2020-01-16T00:13:48.537 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.537 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.537 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:48.537 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497308+0000", 2020-01-16T00:13:48.537 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4202999999999999e-05 2020-01-16T00:13:48.538 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.538 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.538 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:48.538 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497323+0000", 2020-01-16T00:13:48.538 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0280000000000003e-06 2020-01-16T00:13:48.538 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.538 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.538 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:48.539 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497330+0000", 2020-01-16T00:13:48.539 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010681352 2020-01-16T00:13:48.539 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.539 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.539 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.539 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.508011+0000", 2020-01-16T00:13:48.539 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5886999999999999e-05 2020-01-16T00:13:48.540 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.540 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.540 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:48.540 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.508097+0000", 2020-01-16T00:13:48.540 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023353600000000001 2020-01-16T00:13:48.540 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.540 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.540 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:48.541 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.508331+0000", 2020-01-16T00:13:48.541 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0075218339999999998 2020-01-16T00:13:48.541 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.541 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.541 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:48.541 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515852+0000", 2020-01-16T00:13:48.541 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00064178599999999998 2020-01-16T00:13:48.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.542 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.542 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:48.542 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.516494+0000", 2020-01-16T00:13:48.542 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6712999999999999e-05 2020-01-16T00:13:48.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.542 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.542 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:48.543 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.516531+0000", 2020-01-16T00:13:48.543 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112225 2020-01-16T00:13:48.543 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.543 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.543 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:48.543 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.516643+0000", 2020-01-16T00:13:48.543 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019602525999999999 2020-01-16T00:13:48.543 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.544 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:48.544 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.544 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.544 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:48.544 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:48.547 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:13:48.547 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:13:48.547 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-01-16T00:13:48.547 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-01-16T00:13:48.547 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-01-16T00:13:48.547 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-01-16T00:13:48.548 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-01-16T00:13:48.548 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENXIO: problem getting command descriptions from osd.5 2020-01-16T00:13:48.548 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: problem getting command descriptions from osd.5 2020-01-16T00:13:48.548 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-01-16T00:13:48.548 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-01-16T00:13:48.548 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-01-16T00:13:48.548 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-01-16T00:13:48.549 INFO:tasks.rados.rados.0.smithi123.stdout:1477: finishing write tid 1 to smithi12313934-32 2020-01-16T00:13:48.549 INFO:tasks.rados.rados.0.smithi123.stdout:1477: finishing write tid 2 to smithi12313934-32 2020-01-16T00:13:48.549 INFO:tasks.rados.rados.0.smithi123.stdout:1477: finishing write tid 4 to smithi12313934-32 2020-01-16T00:13:48.549 INFO:tasks.rados.rados.0.smithi123.stdout:1477: finishing write tid 5 to smithi12313934-32 2020-01-16T00:13:48.549 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 32 v 152 (ObjNum 492 snap 0 seq_num 492) dirty exists 2020-01-16T00:13:48.549 INFO:tasks.rados.rados.0.smithi123.stdout:1477: left oid 32 (ObjNum 492 snap 0 seq_num 492) 2020-01-16T00:13:48.550 INFO:tasks.rados.rados.0.smithi123.stdout:1478: finishing write tid 1 to smithi12313934-30 2020-01-16T00:13:48.550 INFO:tasks.rados.rados.0.smithi123.stdout:1479: expect (ObjNum 393 snap 0 seq_num 393) 2020-01-16T00:13:48.550 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:48.540+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:48.550 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-01-16T00:13:48.579 INFO:tasks.rados.rados.0.smithi123.stdout:1478: finishing write tid 2 to smithi12313934-30 2020-01-16T00:13:48.579 INFO:tasks.rados.rados.0.smithi123.stdout:1478: finishing write tid 4 to smithi12313934-30 2020-01-16T00:13:48.579 INFO:tasks.rados.rados.0.smithi123.stdout:1478: finishing write tid 5 to smithi12313934-30 2020-01-16T00:13:48.580 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 30 v 159 (ObjNum 493 snap 0 seq_num 493) dirty exists 2020-01-16T00:13:48.580 INFO:tasks.rados.rados.0.smithi123.stdout:1478: left oid 30 (ObjNum 493 snap 0 seq_num 493) 2020-01-16T00:13:48.580 INFO:tasks.rados.rados.0.smithi123.stdout:1480: expect (ObjNum 480 snap 0 seq_num 480) 2020-01-16T00:13:48.646 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:48.642+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:48.683 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:48.684 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:48.684 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:13:48.684 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:48.692 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-01-16T00:13:48.711 INFO:tasks.rados.rados.0.smithi123.stdout:1482: finishing write tid 1 to smithi12313934-7 2020-01-16T00:13:48.712 INFO:tasks.rados.rados.0.smithi123.stdout:1482: finishing write tid 2 to smithi12313934-7 2020-01-16T00:13:48.712 INFO:tasks.rados.rados.0.smithi123.stdout:1482: finishing write tid 4 to smithi12313934-7 2020-01-16T00:13:48.712 INFO:tasks.rados.rados.0.smithi123.stdout:1482: finishing write tid 5 to smithi12313934-7 2020-01-16T00:13:48.712 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 7 v 214 (ObjNum 494 snap 0 seq_num 494) dirty exists 2020-01-16T00:13:48.712 INFO:tasks.rados.rados.0.smithi123.stdout:1482: left oid 7 (ObjNum 494 snap 0 seq_num 494) 2020-01-16T00:13:48.712 INFO:tasks.rados.rados.0.smithi123.stdout:1484: expect (ObjNum 465 snap 0 seq_num 465) 2020-01-16T00:13:48.756 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:48.754+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:48.816 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:48.816 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:48.816 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:13:48.817 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:13:48.817 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:48.825 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-01-16T00:13:48.868 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:48.866+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:48.916 INFO:tasks.rados.rados.0.smithi123.stdout:1483: expect (ObjNum 466 snap 0 seq_num 466) 2020-01-16T00:13:48.950 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:48.946+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:48.976 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:48.977 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:13:48.977 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:13:48.977 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:13:48.977 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.977 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.4.0:114 3.9 e24/20 3:9fc687b2:::smithi12313934-44:head v 24'74, mlcod=24'72)", 2020-01-16T00:13:48.977 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:13:48.978 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 70.647402361000005, 2020-01-16T00:13:48.978 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010399429, 2020-01-16T00:13:48.978 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:48.978 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:48.978 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:48.978 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.978 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:48.978 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:13:48.979 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:48.979 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.979 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.979 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:48.979 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:13:48.979 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:48.979 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.980 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.980 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:48.980 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325276+0000", 2020-01-16T00:13:48.980 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0046127069999999997 2020-01-16T00:13:48.980 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.980 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.980 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:48.980 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329889+0000", 2020-01-16T00:13:48.981 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3519999999999998e-05 2020-01-16T00:13:48.981 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.981 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.981 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:48.981 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329912+0000", 2020-01-16T00:13:48.981 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4070000000000004e-06 2020-01-16T00:13:48.981 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.982 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.982 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:48.982 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329920+0000", 2020-01-16T00:13:48.982 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8817000000000003e-05 2020-01-16T00:13:48.982 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.982 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.982 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.982 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329978+0000", 2020-01-16T00:13:48.983 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9721e-05 2020-01-16T00:13:48.983 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.983 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.983 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:48.983 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329998+0000", 2020-01-16T00:13:48.983 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056328139999999999 2020-01-16T00:13:48.983 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.984 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.984 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:48.984 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.335631+0000", 2020-01-16T00:13:48.984 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7170999999999999e-05 2020-01-16T00:13:48.984 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.984 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.984 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:48.984 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.335678+0000", 2020-01-16T00:13:48.985 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010399429 2020-01-16T00:13:48.985 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.985 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:48.985 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.985 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.985 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.985 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:24 2.1 2:8b431f80:::smithi12313934-40 low_tier:head [write 790528~688128] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:13:48.986 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.482303+0000", 2020-01-16T00:13:48.986 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 70.490378265000004, 2020-01-16T00:13:48.986 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010071853, 2020-01-16T00:13:48.986 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:48.986 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:48.986 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:48.986 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.986 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:48.987 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.482303+0000", 2020-01-16T00:13:48.987 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:48.987 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.987 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.987 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:48.987 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.482303+0000", 2020-01-16T00:13:48.987 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:48.988 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.988 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.988 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:48.988 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.482301+0000", 2020-01-16T00:13:48.988 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031173960000000001 2020-01-16T00:13:48.988 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.988 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.989 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:48.989 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485418+0000", 2020-01-16T00:13:48.989 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9300000000000001e-05 2020-01-16T00:13:48.989 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.989 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.989 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:48.989 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485448+0000", 2020-01-16T00:13:48.989 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2611e-05 2020-01-16T00:13:48.990 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.990 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.990 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:48.990 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485460+0000", 2020-01-16T00:13:48.990 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112487 2020-01-16T00:13:48.990 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.990 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.990 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:48.991 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485573+0000", 2020-01-16T00:13:48.991 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022169299999999999 2020-01-16T00:13:48.991 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.991 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.991 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:48.991 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485794+0000", 2020-01-16T00:13:48.991 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00036287500000000002 2020-01-16T00:13:48.992 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.992 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.992 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:48.992 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.486157+0000", 2020-01-16T00:13:48.992 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040260310000000002 2020-01-16T00:13:48.992 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.992 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.992 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:48.993 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.490183+0000", 2020-01-16T00:13:48.993 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00081083000000000001 2020-01-16T00:13:48.993 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.993 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.993 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:48.993 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.490994+0000", 2020-01-16T00:13:48.994 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001287746 2020-01-16T00:13:48.994 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.994 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.994 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:48.994 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492282+0000", 2020-01-16T00:13:48.994 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8810999999999998e-05 2020-01-16T00:13:48.994 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.995 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.995 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:48.995 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492321+0000", 2020-01-16T00:13:48.995 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4095000000000003e-05 2020-01-16T00:13:48.995 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.995 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.995 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:48.995 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492375+0000", 2020-01-16T00:13:48.996 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010071853 2020-01-16T00:13:48.996 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.996 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:48.996 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:48.996 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.996 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.996 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:55 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e30)", 2020-01-16T00:13:48.996 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.736036+0000", 2020-01-16T00:13:48.997 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 56.236645142, 2020-01-16T00:13:48.997 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010974536, 2020-01-16T00:13:48.997 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:48.997 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:48.997 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:48.997 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.997 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:48.997 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736036+0000", 2020-01-16T00:13:48.998 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:48.998 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.998 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.998 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:48.998 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736036+0000", 2020-01-16T00:13:48.998 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:48.998 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.999 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.999 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:48.999 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736034+0000", 2020-01-16T00:13:48.999 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043589699999999998 2020-01-16T00:13:48.999 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:48.999 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:48.999 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:48.999 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736469+0000", 2020-01-16T00:13:49.000 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9882e-05 2020-01-16T00:13:49.000 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.000 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.000 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:49.000 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736489+0000", 2020-01-16T00:13:49.000 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8339999999999999e-06 2020-01-16T00:13:49.000 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.000 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.001 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:49.001 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736497+0000", 2020-01-16T00:13:49.001 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5354999999999999e-05 2020-01-16T00:13:49.001 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.001 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.001 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:49.001 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736572+0000", 2020-01-16T00:13:49.002 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5228000000000004e-05 2020-01-16T00:13:49.002 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.002 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.002 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:49.002 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736668+0000", 2020-01-16T00:13:49.002 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017212999999999999 2020-01-16T00:13:49.002 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.003 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.003 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:49.003 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736840+0000", 2020-01-16T00:13:49.003 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0099057020000000006 2020-01-16T00:13:49.003 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.003 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.003 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.003 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746746+0000", 2020-01-16T00:13:49.004 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0586000000000005e-05 2020-01-16T00:13:49.004 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.004 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.004 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:49.004 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746816+0000", 2020-01-16T00:13:49.004 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2445000000000002e-05 2020-01-16T00:13:49.004 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.004 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.005 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.005 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746889+0000", 2020-01-16T00:13:49.005 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6839000000000002e-05 2020-01-16T00:13:49.005 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.005 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.005 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:49.005 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746955+0000", 2020-01-16T00:13:49.006 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5074000000000002e-05 2020-01-16T00:13:49.006 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.006 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.006 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:49.006 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.747011+0000", 2020-01-16T00:13:49.006 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010974536 2020-01-16T00:13:49.006 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:49.006 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:49.007 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:49.007 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.007 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.007 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:56 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e30)", 2020-01-16T00:13:49.007 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.736514+0000", 2020-01-16T00:13:49.007 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 56.236167307000002, 2020-01-16T00:13:49.007 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011660584999999999, 2020-01-16T00:13:49.008 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:49.008 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:49.008 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:49.008 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.008 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:49.008 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736514+0000", 2020-01-16T00:13:49.009 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:49.009 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.009 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.010 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:49.010 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736514+0000", 2020-01-16T00:13:49.010 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:49.010 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.010 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.010 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:49.010 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736513+0000", 2020-01-16T00:13:49.010 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9677000000000004e-05 2020-01-16T00:13:49.011 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.011 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.011 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:49.011 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736583+0000", 2020-01-16T00:13:49.011 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5269999999999998e-06 2020-01-16T00:13:49.011 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.012 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.012 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:49.012 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736591+0000", 2020-01-16T00:13:49.012 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5769999999999996e-06 2020-01-16T00:13:49.012 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.012 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.012 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:49.012 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736599+0000", 2020-01-16T00:13:49.013 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0053593310000000002 2020-01-16T00:13:49.013 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.013 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.013 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:49.013 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.741958+0000", 2020-01-16T00:13:49.013 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014164099999999999 2020-01-16T00:13:49.013 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.013 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.014 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:49.014 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.742100+0000", 2020-01-16T00:13:49.014 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020126000000000001 2020-01-16T00:13:49.014 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.014 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.014 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:49.014 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.742301+0000", 2020-01-16T00:13:49.015 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0045052859999999998 2020-01-16T00:13:49.015 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.015 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.015 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.015 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746806+0000", 2020-01-16T00:13:49.015 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00070802699999999996 2020-01-16T00:13:49.016 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.016 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.016 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:49.016 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.747514+0000", 2020-01-16T00:13:49.017 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00059247200000000003 2020-01-16T00:13:49.017 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.017 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.017 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.017 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748107+0000", 2020-01-16T00:13:49.017 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1870999999999999e-05 2020-01-16T00:13:49.018 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.018 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.018 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:49.018 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748149+0000", 2020-01-16T00:13:49.018 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5831000000000001e-05 2020-01-16T00:13:49.018 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.018 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.019 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:49.019 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748174+0000", 2020-01-16T00:13:49.019 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011660584999999999 2020-01-16T00:13:49.019 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:49.019 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:49.019 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:49.019 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.019 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.020 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:70 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e35)", 2020-01-16T00:13:49.020 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.985494+0000", 2020-01-16T00:13:49.020 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 42.987187511000002, 2020-01-16T00:13:49.020 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013746267, 2020-01-16T00:13:49.020 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:49.020 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:49.020 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:49.020 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.021 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:49.021 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985494+0000", 2020-01-16T00:13:49.021 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:49.021 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.021 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.021 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:49.021 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985494+0000", 2020-01-16T00:13:49.022 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:49.022 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.022 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.022 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:49.022 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985490+0000", 2020-01-16T00:13:49.022 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00077792099999999999 2020-01-16T00:13:49.022 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.023 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.023 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:49.023 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986268+0000", 2020-01-16T00:13:49.023 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9718000000000001e-05 2020-01-16T00:13:49.023 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.023 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.023 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:49.024 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986318+0000", 2020-01-16T00:13:49.024 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.4349999999999997e-06 2020-01-16T00:13:49.024 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.024 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.024 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:49.024 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986326+0000", 2020-01-16T00:13:49.024 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3298999999999999e-05 2020-01-16T00:13:49.025 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.025 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.025 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:49.025 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986380+0000", 2020-01-16T00:13:49.025 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4969999999999995e-05 2020-01-16T00:13:49.025 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.025 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.025 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:49.026 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986455+0000", 2020-01-16T00:13:49.026 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000110587 2020-01-16T00:13:49.026 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.026 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.026 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:49.026 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986565+0000", 2020-01-16T00:13:49.026 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063931600000000002 2020-01-16T00:13:49.027 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.027 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.027 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:49.027 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.992958+0000", 2020-01-16T00:13:49.027 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013008480000000001 2020-01-16T00:13:49.027 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.028 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.028 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.028 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.994259+0000", 2020-01-16T00:13:49.028 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048399979999999999 2020-01-16T00:13:49.028 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.028 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.028 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.028 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999099+0000", 2020-01-16T00:13:49.029 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1874999999999998e-05 2020-01-16T00:13:49.029 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.029 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.029 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:49.029 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999181+0000", 2020-01-16T00:13:49.029 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8865e-05 2020-01-16T00:13:49.029 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.030 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.030 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:49.030 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999240+0000", 2020-01-16T00:13:49.030 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013746267 2020-01-16T00:13:49.030 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:49.030 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:49.030 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:49.031 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.031 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.031 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:71 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e35)", 2020-01-16T00:13:49.031 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.986341+0000", 2020-01-16T00:13:49.031 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 42.986340149999997, 2020-01-16T00:13:49.031 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013252807, 2020-01-16T00:13:49.031 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:49.031 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:49.032 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:49.032 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.032 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:49.032 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986341+0000", 2020-01-16T00:13:49.032 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:49.032 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.032 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.032 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:49.033 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986341+0000", 2020-01-16T00:13:49.033 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:49.033 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.033 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.033 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:49.033 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986340+0000", 2020-01-16T00:13:49.033 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000103867 2020-01-16T00:13:49.034 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.034 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.034 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:49.034 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986444+0000", 2020-01-16T00:13:49.034 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0104e-05 2020-01-16T00:13:49.034 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.034 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.034 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:49.035 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986454+0000", 2020-01-16T00:13:49.035 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4220000000000002e-06 2020-01-16T00:13:49.035 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.035 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.035 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:49.035 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986458+0000", 2020-01-16T00:13:49.035 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033523070000000001 2020-01-16T00:13:49.035 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.036 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.036 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:49.036 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.989811+0000", 2020-01-16T00:13:49.036 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3296999999999999e-05 2020-01-16T00:13:49.036 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.036 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.036 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:49.036 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.989894+0000", 2020-01-16T00:13:49.037 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000117701 2020-01-16T00:13:49.037 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.037 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.037 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:49.037 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.990012+0000", 2020-01-16T00:13:49.037 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0032873490000000002 2020-01-16T00:13:49.037 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.038 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.038 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:49.038 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993299+0000", 2020-01-16T00:13:49.038 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002422619 2020-01-16T00:13:49.038 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.038 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.038 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.038 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.995722+0000", 2020-01-16T00:13:49.039 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0037839589999999999 2020-01-16T00:13:49.039 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.039 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.039 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.039 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999506+0000", 2020-01-16T00:13:49.039 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8559999999999997e-05 2020-01-16T00:13:49.039 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.039 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.040 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:49.040 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999544+0000", 2020-01-16T00:13:49.040 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9599999999999999e-05 2020-01-16T00:13:49.040 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.040 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.040 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:49.040 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999594+0000", 2020-01-16T00:13:49.040 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013252807 2020-01-16T00:13:49.041 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:49.041 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:49.041 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:49.041 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.041 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.041 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:77 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e36)", 2020-01-16T00:13:49.041 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.555854+0000", 2020-01-16T00:13:49.042 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 41.416827060999999, 2020-01-16T00:13:49.042 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010455002999999999, 2020-01-16T00:13:49.042 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:49.042 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:49.042 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:49.042 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.042 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:49.043 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555854+0000", 2020-01-16T00:13:49.043 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:49.043 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.043 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.043 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:49.043 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555854+0000", 2020-01-16T00:13:49.043 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:49.044 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.044 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.044 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:49.044 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555851+0000", 2020-01-16T00:13:49.044 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00048310200000000001 2020-01-16T00:13:49.044 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.044 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.044 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:49.045 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556334+0000", 2020-01-16T00:13:49.045 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2232e-05 2020-01-16T00:13:49.045 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.045 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.045 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:49.045 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556356+0000", 2020-01-16T00:13:49.045 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9750000000000007e-06 2020-01-16T00:13:49.045 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.046 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.046 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:49.046 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556364+0000", 2020-01-16T00:13:49.046 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8872999999999997e-05 2020-01-16T00:13:49.046 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.046 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.046 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:49.047 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556423+0000", 2020-01-16T00:13:49.047 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8687000000000001e-05 2020-01-16T00:13:49.047 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.047 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.047 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:49.048 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556502+0000", 2020-01-16T00:13:49.048 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013235300000000001 2020-01-16T00:13:49.048 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.048 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.048 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:49.048 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556634+0000", 2020-01-16T00:13:49.049 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0059770520000000001 2020-01-16T00:13:49.049 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.049 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.049 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:49.049 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.562611+0000", 2020-01-16T00:13:49.049 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029492820000000001 2020-01-16T00:13:49.049 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.050 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.050 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.050 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.565561+0000", 2020-01-16T00:13:49.050 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00060620300000000004 2020-01-16T00:13:49.050 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.050 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.050 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.050 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566167+0000", 2020-01-16T00:13:49.051 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3831000000000004e-05 2020-01-16T00:13:49.051 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.051 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.051 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:49.051 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566251+0000", 2020-01-16T00:13:49.051 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8328000000000001e-05 2020-01-16T00:13:49.051 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.051 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.052 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:49.052 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566309+0000", 2020-01-16T00:13:49.052 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010455002999999999 2020-01-16T00:13:49.052 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:49.052 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:49.052 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:49.052 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.052 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.053 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:78 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e36)", 2020-01-16T00:13:49.053 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.556378+0000", 2020-01-16T00:13:49.053 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 41.416303128000003, 2020-01-16T00:13:49.053 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010396311, 2020-01-16T00:13:49.053 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:49.053 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:49.053 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:49.054 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.054 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:49.054 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556378+0000", 2020-01-16T00:13:49.054 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:49.054 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.054 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.054 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:49.054 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556378+0000", 2020-01-16T00:13:49.055 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:49.055 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.055 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.055 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:49.055 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556377+0000", 2020-01-16T00:13:49.055 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4307000000000001e-05 2020-01-16T00:13:49.055 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.056 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.056 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:49.056 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556422+0000", 2020-01-16T00:13:49.056 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9620000000000005e-06 2020-01-16T00:13:49.056 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.056 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.056 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:49.056 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556432+0000", 2020-01-16T00:13:49.057 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5979999999999999e-06 2020-01-16T00:13:49.057 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.057 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.057 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:49.057 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556436+0000", 2020-01-16T00:13:49.057 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021059469999999999 2020-01-16T00:13:49.057 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.057 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.058 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:49.058 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.558542+0000", 2020-01-16T00:13:49.058 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000194458 2020-01-16T00:13:49.058 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.058 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.058 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:49.058 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.558737+0000", 2020-01-16T00:13:49.059 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00032867900000000002 2020-01-16T00:13:49.059 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.059 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.059 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:49.059 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.559065+0000", 2020-01-16T00:13:49.059 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052547499999999999 2020-01-16T00:13:49.059 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.059 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.060 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:49.060 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.564320+0000", 2020-01-16T00:13:49.060 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001565213 2020-01-16T00:13:49.060 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.060 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.060 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.060 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.565885+0000", 2020-01-16T00:13:49.061 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00076994300000000004 2020-01-16T00:13:49.061 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.061 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.061 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.061 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566655+0000", 2020-01-16T00:13:49.061 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7533000000000005e-05 2020-01-16T00:13:49.061 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.061 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.062 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:49.062 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566723+0000", 2020-01-16T00:13:49.062 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1616999999999997e-05 2020-01-16T00:13:49.062 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.062 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.062 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:49.062 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566774+0000", 2020-01-16T00:13:49.062 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010396311 2020-01-16T00:13:49.063 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:49.063 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:49.063 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:49.063 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.063 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.063 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:72 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 56832~1408000] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e37)", 2020-01-16T00:13:49.063 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.578753+0000", 2020-01-16T00:13:49.063 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 36.393927824999999, 2020-01-16T00:13:49.064 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011826918, 2020-01-16T00:13:49.064 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:49.064 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:49.064 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:49.064 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.064 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:49.064 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578753+0000", 2020-01-16T00:13:49.065 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:49.065 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.065 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.065 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:49.065 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578753+0000", 2020-01-16T00:13:49.065 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:49.065 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.066 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.066 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:49.066 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578751+0000", 2020-01-16T00:13:49.066 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00042457699999999998 2020-01-16T00:13:49.066 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.066 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.067 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:49.067 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579175+0000", 2020-01-16T00:13:49.067 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5404999999999999e-05 2020-01-16T00:13:49.067 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.067 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.067 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:49.067 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579191+0000", 2020-01-16T00:13:49.067 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6149999999999996e-06 2020-01-16T00:13:49.068 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.068 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.068 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:49.068 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579196+0000", 2020-01-16T00:13:49.068 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000104794 2020-01-16T00:13:49.068 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.068 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.068 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:49.069 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579301+0000", 2020-01-16T00:13:49.069 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020947199999999999 2020-01-16T00:13:49.069 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.069 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.069 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:49.069 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579511+0000", 2020-01-16T00:13:49.069 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034328400000000002 2020-01-16T00:13:49.070 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.070 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.070 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:49.070 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579854+0000", 2020-01-16T00:13:49.070 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.005078117 2020-01-16T00:13:49.070 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.070 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.070 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:49.071 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.584932+0000", 2020-01-16T00:13:49.071 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0014702160000000001 2020-01-16T00:13:49.071 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.071 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.071 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.071 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.586402+0000", 2020-01-16T00:13:49.071 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040623589999999998 2020-01-16T00:13:49.072 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.072 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.072 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.072 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590465+0000", 2020-01-16T00:13:49.072 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4478000000000002e-05 2020-01-16T00:13:49.072 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.072 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.072 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:49.073 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590529+0000", 2020-01-16T00:13:49.073 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1121000000000001e-05 2020-01-16T00:13:49.073 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.073 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.073 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:49.073 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590580+0000", 2020-01-16T00:13:49.073 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011826918 2020-01-16T00:13:49.073 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:49.074 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:49.074 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:49.074 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.074 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.074 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:73 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 1922048~118272] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e37)", 2020-01-16T00:13:49.074 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:13:49.074 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 36.393475027999997, 2020-01-16T00:13:49.075 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011750764, 2020-01-16T00:13:49.075 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:49.075 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:49.075 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:49.075 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.075 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:49.075 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:13:49.075 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:49.076 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.076 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.076 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:49.076 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:13:49.076 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:49.076 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.076 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.076 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:49.077 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:13:49.077 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2780000000000001e-05 2020-01-16T00:13:49.077 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.077 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.077 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:49.077 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579238+0000", 2020-01-16T00:13:49.077 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4341999999999999e-05 2020-01-16T00:13:49.078 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.078 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.078 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:49.078 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579253+0000", 2020-01-16T00:13:49.078 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1920000000000001e-06 2020-01-16T00:13:49.078 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.078 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.078 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:49.079 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579256+0000", 2020-01-16T00:13:49.079 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003224902 2020-01-16T00:13:49.079 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.079 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.079 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:49.079 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.582481+0000", 2020-01-16T00:13:49.079 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000180729 2020-01-16T00:13:49.080 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.080 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.080 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:49.080 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.582662+0000", 2020-01-16T00:13:49.080 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00025106100000000001 2020-01-16T00:13:49.080 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.080 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.080 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:49.081 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.582913+0000", 2020-01-16T00:13:49.081 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020814449999999999 2020-01-16T00:13:49.081 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.081 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.081 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:49.081 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.584994+0000", 2020-01-16T00:13:49.081 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001717243 2020-01-16T00:13:49.082 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.082 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.082 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.082 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.586711+0000", 2020-01-16T00:13:49.082 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041349869999999997 2020-01-16T00:13:49.082 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.082 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.083 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.083 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590846+0000", 2020-01-16T00:13:49.083 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3565000000000006e-05 2020-01-16T00:13:49.083 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.083 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.083 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:49.083 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590910+0000", 2020-01-16T00:13:49.083 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7002999999999997e-05 2020-01-16T00:13:49.084 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.084 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.084 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:49.084 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590957+0000", 2020-01-16T00:13:49.084 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011750764 2020-01-16T00:13:49.084 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:49.084 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:49.084 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:49.085 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.085 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.085 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:89 2.1 2:9b8617ce:::smithi12313934-38 low_tier:head [write 69632~773120] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e37)", 2020-01-16T00:13:49.085 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.593701+0000", 2020-01-16T00:13:49.085 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 36.378980402000003, 2020-01-16T00:13:49.085 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010026929, 2020-01-16T00:13:49.085 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:49.086 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:49.086 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:49.086 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.086 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:49.086 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593701+0000", 2020-01-16T00:13:49.086 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:49.086 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.087 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.087 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:49.087 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593701+0000", 2020-01-16T00:13:49.087 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:49.087 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.087 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.087 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:49.087 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593699+0000", 2020-01-16T00:13:49.088 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000237564 2020-01-16T00:13:49.088 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.088 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.088 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:49.088 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593936+0000", 2020-01-16T00:13:49.088 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2666e-05 2020-01-16T00:13:49.088 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.089 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.089 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:49.089 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593949+0000", 2020-01-16T00:13:49.089 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0089999999999998e-06 2020-01-16T00:13:49.089 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.089 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.089 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:49.089 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593954+0000", 2020-01-16T00:13:49.090 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5813000000000003e-05 2020-01-16T00:13:49.090 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.090 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.090 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:49.090 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.594000+0000", 2020-01-16T00:13:49.090 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010155299999999999 2020-01-16T00:13:49.090 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.091 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.091 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:49.091 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.594101+0000", 2020-01-16T00:13:49.091 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000203807 2020-01-16T00:13:49.091 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.091 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.091 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:49.092 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.594305+0000", 2020-01-16T00:13:49.092 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021655390000000002 2020-01-16T00:13:49.092 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.092 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.092 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.092 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.596471+0000", 2020-01-16T00:13:49.092 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0070645320000000001 2020-01-16T00:13:49.092 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.093 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.093 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.093 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603535+0000", 2020-01-16T00:13:49.093 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7577999999999998e-05 2020-01-16T00:13:49.093 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.093 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.093 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:49.094 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603573+0000", 2020-01-16T00:13:49.094 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9933999999999997e-05 2020-01-16T00:13:49.094 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.094 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.094 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:49.094 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603653+0000", 2020-01-16T00:13:49.094 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4993000000000005e-05 2020-01-16T00:13:49.095 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.095 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.095 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:49.095 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603728+0000", 2020-01-16T00:13:49.095 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010026929 2020-01-16T00:13:49.095 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:49.096 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:49.096 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:49.096 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.096 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.096 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:66 2.3 2:c3b82d87:::smithi12313934-44 low_tier:head [write 169472~818176] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e44)", 2020-01-16T00:13:49.096 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:25.278583+0000", 2020-01-16T00:13:49.096 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 23.694098005000001, 2020-01-16T00:13:49.096 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010294862, 2020-01-16T00:13:49.097 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:49.097 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:49.097 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:49.097 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.097 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:49.097 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.278583+0000", 2020-01-16T00:13:49.097 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:49.098 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.098 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.098 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:49.098 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.278583+0000", 2020-01-16T00:13:49.098 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:49.098 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.098 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.098 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:49.099 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.278581+0000", 2020-01-16T00:13:49.099 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00136721 2020-01-16T00:13:49.099 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.099 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.099 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:49.099 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.279948+0000", 2020-01-16T00:13:49.099 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6109999999999999e-05 2020-01-16T00:13:49.099 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.100 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.100 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:49.100 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.279974+0000", 2020-01-16T00:13:49.100 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.102e-05 2020-01-16T00:13:49.100 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.100 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.100 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:49.100 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.279985+0000", 2020-01-16T00:13:49.101 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6035e-05 2020-01-16T00:13:49.101 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.101 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.101 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:49.101 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.280061+0000", 2020-01-16T00:13:49.101 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000176243 2020-01-16T00:13:49.101 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.102 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.102 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:49.102 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.280238+0000", 2020-01-16T00:13:49.102 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019866099999999999 2020-01-16T00:13:49.102 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.102 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.102 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 7", 2020-01-16T00:13:49.102 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.280436+0000", 2020-01-16T00:13:49.103 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0049761830000000003 2020-01-16T00:13:49.103 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.103 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.103 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.103 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.285412+0000", 2020-01-16T00:13:49.103 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033347339999999998 2020-01-16T00:13:49.103 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.104 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.104 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:49.104 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.288747+0000", 2020-01-16T00:13:49.104 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9156000000000001e-05 2020-01-16T00:13:49.104 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.104 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.105 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:49.105 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.288806+0000", 2020-01-16T00:13:49.105 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1643000000000005e-05 2020-01-16T00:13:49.105 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.105 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.106 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:49.106 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.288878+0000", 2020-01-16T00:13:49.106 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010294862 2020-01-16T00:13:49.106 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:49.106 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:49.106 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:49.106 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.107 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.107 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.4.0:316 3.9 e45/20 3:93dfe6f1:::smithi12313934-40:head v 45'188, mlcod=45'186)", 2020-01-16T00:13:49.107 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.584377+0000", 2020-01-16T00:13:49.107 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 19.388304079000001, 2020-01-16T00:13:49.107 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095979069999999993, 2020-01-16T00:13:49.107 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:49.107 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:49.108 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:49.108 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.108 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:49.108 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584377+0000", 2020-01-16T00:13:49.108 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:49.108 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.108 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.109 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:49.109 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584377+0000", 2020-01-16T00:13:49.109 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:49.109 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.109 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.109 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:49.110 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584376+0000", 2020-01-16T00:13:49.110 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9059999999999997e-06 2020-01-16T00:13:49.110 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.110 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.110 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:49.110 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584385+0000", 2020-01-16T00:13:49.110 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3200000000000001e-05 2020-01-16T00:13:49.111 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.111 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.111 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:49.111 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584399+0000", 2020-01-16T00:13:49.111 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9660000000000001e-06 2020-01-16T00:13:49.111 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.111 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.112 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:49.112 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584405+0000", 2020-01-16T00:13:49.112 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0035787800000000002 2020-01-16T00:13:49.112 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.112 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.112 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:49.112 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.587983+0000", 2020-01-16T00:13:49.113 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0368000000000001e-05 2020-01-16T00:13:49.113 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.113 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.113 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:49.113 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588014+0000", 2020-01-16T00:13:49.113 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0059140290000000003 2020-01-16T00:13:49.113 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.114 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.114 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:49.114 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.593928+0000", 2020-01-16T00:13:49.114 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7135999999999997e-05 2020-01-16T00:13:49.114 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.114 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.114 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:49.115 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.593975+0000", 2020-01-16T00:13:49.115 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095979069999999993 2020-01-16T00:13:49.115 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:49.115 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:49.115 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:49.115 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.116 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.116 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:77 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 56832~1408000] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e45)", 2020-01-16T00:13:49.116 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.584566+0000", 2020-01-16T00:13:49.116 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 19.388115354, 2020-01-16T00:13:49.116 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017037775000000002, 2020-01-16T00:13:49.116 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:49.117 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:49.117 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:49.117 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.117 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:49.117 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584566+0000", 2020-01-16T00:13:49.117 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:49.118 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.118 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.118 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:49.118 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584566+0000", 2020-01-16T00:13:49.118 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:49.118 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.118 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.118 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:49.119 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584564+0000", 2020-01-16T00:13:49.119 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043024300000000002 2020-01-16T00:13:49.119 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.119 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.119 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:49.119 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584994+0000", 2020-01-16T00:13:49.119 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9485999999999999e-05 2020-01-16T00:13:49.120 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.120 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.120 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:49.120 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585014+0000", 2020-01-16T00:13:49.120 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8909999999999995e-06 2020-01-16T00:13:49.120 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.120 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.120 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:49.121 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585022+0000", 2020-01-16T00:13:49.121 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039114459999999998 2020-01-16T00:13:49.121 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.121 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.121 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:49.121 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588933+0000", 2020-01-16T00:13:49.121 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011178700000000001 2020-01-16T00:13:49.121 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.122 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.122 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:49.122 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589045+0000", 2020-01-16T00:13:49.122 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016967300000000001 2020-01-16T00:13:49.122 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.122 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.122 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:49.123 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589214+0000", 2020-01-16T00:13:49.123 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012142356 2020-01-16T00:13:49.123 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.123 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.123 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.123 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601357+0000", 2020-01-16T00:13:49.124 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.1466999999999997e-05 2020-01-16T00:13:49.124 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.124 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.124 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.124 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601448+0000", 2020-01-16T00:13:49.124 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4817000000000001e-05 2020-01-16T00:13:49.124 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.124 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.125 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:49.125 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601493+0000", 2020-01-16T00:13:49.125 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9734000000000001e-05 2020-01-16T00:13:49.125 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.125 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.125 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:49.125 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601553+0000", 2020-01-16T00:13:49.126 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0677999999999998e-05 2020-01-16T00:13:49.126 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.126 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.126 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:49.126 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601604+0000", 2020-01-16T00:13:49.126 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017037775000000002 2020-01-16T00:13:49.126 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:49.127 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:49.127 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:49.127 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.127 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.127 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:78 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 1922048~118272] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e45)", 2020-01-16T00:13:49.127 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.585030+0000", 2020-01-16T00:13:49.127 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 19.387650632, 2020-01-16T00:13:49.128 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016699964000000001, 2020-01-16T00:13:49.128 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:49.128 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:49.128 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:49.128 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.128 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:49.128 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585030+0000", 2020-01-16T00:13:49.128 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:49.129 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.129 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.129 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:49.129 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585030+0000", 2020-01-16T00:13:49.129 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:49.129 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.129 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.130 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:49.130 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585030+0000", 2020-01-16T00:13:49.130 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7085000000000001e-05 2020-01-16T00:13:49.130 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.130 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.130 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:49.130 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585077+0000", 2020-01-16T00:13:49.130 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5380000000000008e-06 2020-01-16T00:13:49.131 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.131 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.131 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:49.131 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585086+0000", 2020-01-16T00:13:49.131 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5410000000000002e-06 2020-01-16T00:13:49.131 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.131 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.131 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:49.132 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585091+0000", 2020-01-16T00:13:49.132 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0088486620000000002 2020-01-16T00:13:49.132 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.132 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.132 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:49.132 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.593940+0000", 2020-01-16T00:13:49.132 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112594 2020-01-16T00:13:49.133 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.133 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.133 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:49.133 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.594052+0000", 2020-01-16T00:13:49.133 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019893999999999999 2020-01-16T00:13:49.133 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.133 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.134 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:49.134 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.594251+0000", 2020-01-16T00:13:49.134 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0071632730000000004 2020-01-16T00:13:49.134 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.134 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.134 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.134 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601414+0000", 2020-01-16T00:13:49.135 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4672999999999996e-05 2020-01-16T00:13:49.135 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.135 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.135 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.135 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601479+0000", 2020-01-16T00:13:49.135 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000158453 2020-01-16T00:13:49.135 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.135 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.136 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:49.136 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601638+0000", 2020-01-16T00:13:49.136 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8605e-05 2020-01-16T00:13:49.136 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.136 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.136 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:49.136 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601686+0000", 2020-01-16T00:13:49.136 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4283999999999998e-05 2020-01-16T00:13:49.137 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.137 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.137 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:49.138 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601730+0000", 2020-01-16T00:13:49.138 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016699964000000001 2020-01-16T00:13:49.138 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:49.138 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:49.138 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:49.138 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.139 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.139 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:126 2.1 2:a4df40e2:::smithi12313934-42 low_tier:head [write 70656~727552] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e45)", 2020-01-16T00:13:49.139 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.627078+0000", 2020-01-16T00:13:49.139 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 19.345602975999999, 2020-01-16T00:13:49.139 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010098995, 2020-01-16T00:13:49.139 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:49.139 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:49.140 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:49.140 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.140 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:49.140 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627078+0000", 2020-01-16T00:13:49.140 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:49.140 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.140 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.140 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:49.141 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627078+0000", 2020-01-16T00:13:49.141 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:49.141 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.141 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.141 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:49.141 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627076+0000", 2020-01-16T00:13:49.141 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019321599999999999 2020-01-16T00:13:49.141 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.142 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.142 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:49.142 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627269+0000", 2020-01-16T00:13:49.143 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8471999999999999e-05 2020-01-16T00:13:49.143 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.143 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.143 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:49.144 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627288+0000", 2020-01-16T00:13:49.144 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3199999999999996e-06 2020-01-16T00:13:49.144 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.144 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.144 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:49.145 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627294+0000", 2020-01-16T00:13:49.145 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3341000000000002e-05 2020-01-16T00:13:49.145 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.145 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.146 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:49.147 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627357+0000", 2020-01-16T00:13:49.147 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010985 2020-01-16T00:13:49.147 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.147 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.147 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:49.147 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627467+0000", 2020-01-16T00:13:49.148 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022914499999999999 2020-01-16T00:13:49.148 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.148 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.148 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:49.148 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627696+0000", 2020-01-16T00:13:49.148 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0055120000000000004 2020-01-16T00:13:49.148 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.148 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.149 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:49.149 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.633208+0000", 2020-01-16T00:13:49.149 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0019968569999999999 2020-01-16T00:13:49.149 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.149 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.149 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.149 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.635205+0000", 2020-01-16T00:13:49.150 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018113160000000001 2020-01-16T00:13:49.150 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.150 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.150 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.150 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.637016+0000", 2020-01-16T00:13:49.150 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9515e-05 2020-01-16T00:13:49.150 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.150 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.151 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:49.151 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.637096+0000", 2020-01-16T00:13:49.151 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1213999999999999e-05 2020-01-16T00:13:49.151 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.151 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.151 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:49.151 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.637177+0000", 2020-01-16T00:13:49.151 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010098995 2020-01-16T00:13:49.152 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:49.152 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:49.152 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:49.152 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.152 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.152 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:127 2.1 2:a4df40e2:::smithi12313934-42 low_tier:head [write 1364992~1145344] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e45)", 2020-01-16T00:13:49.152 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.627308+0000", 2020-01-16T00:13:49.153 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 19.345373179999999, 2020-01-16T00:13:49.153 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011068883999999999, 2020-01-16T00:13:49.153 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:49.153 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:49.153 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:49.153 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.153 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:49.153 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627308+0000", 2020-01-16T00:13:49.154 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:49.154 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.154 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.154 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:49.154 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627308+0000", 2020-01-16T00:13:49.154 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:49.154 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.154 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.155 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:49.155 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627307+0000", 2020-01-16T00:13:49.155 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000278597 2020-01-16T00:13:49.155 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.155 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.155 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:49.155 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627586+0000", 2020-01-16T00:13:49.155 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9169999999999997e-06 2020-01-16T00:13:49.156 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.156 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.156 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:49.156 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627591+0000", 2020-01-16T00:13:49.156 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0479999999999999e-06 2020-01-16T00:13:49.156 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.156 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.156 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:49.158 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627594+0000", 2020-01-16T00:13:49.158 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030987269999999999 2020-01-16T00:13:49.158 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.158 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.158 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:49.158 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.630693+0000", 2020-01-16T00:13:49.158 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000103688 2020-01-16T00:13:49.159 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.159 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.159 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:49.159 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.630796+0000", 2020-01-16T00:13:49.159 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000144606 2020-01-16T00:13:49.159 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.159 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.159 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:49.160 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.630941+0000", 2020-01-16T00:13:49.160 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039264269999999997 2020-01-16T00:13:49.160 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.160 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.160 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:49.160 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.634867+0000", 2020-01-16T00:13:49.160 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00038581200000000003 2020-01-16T00:13:49.161 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.161 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.161 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.161 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.635253+0000", 2020-01-16T00:13:49.161 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030568879999999998 2020-01-16T00:13:49.161 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.161 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.161 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.162 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.638310+0000", 2020-01-16T00:13:49.162 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8390999999999999e-05 2020-01-16T00:13:49.162 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.162 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.162 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:49.162 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.638348+0000", 2020-01-16T00:13:49.162 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8402999999999999e-05 2020-01-16T00:13:49.162 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.163 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.163 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:49.163 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.638377+0000", 2020-01-16T00:13:49.163 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011068883999999999 2020-01-16T00:13:49.163 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:49.163 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:49.163 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:49.164 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.164 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.164 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.6.13:30 2.3 2:fc791323:::smithi12313934-2 low_tier:head [write 476160~1240064] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e48)", 2020-01-16T00:13:49.164 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:39.466268+0000", 2020-01-16T00:13:49.164 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 9.5064128760000006, 2020-01-16T00:13:49.164 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010593112, 2020-01-16T00:13:49.164 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:49.165 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:49.165 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:49.165 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.165 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:49.165 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.466268+0000", 2020-01-16T00:13:49.165 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:49.165 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.165 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.166 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:49.166 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.466268+0000", 2020-01-16T00:13:49.166 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:49.166 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.166 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.166 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:49.166 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.466266+0000", 2020-01-16T00:13:49.166 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012489210000000001 2020-01-16T00:13:49.167 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.167 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.167 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:49.167 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467515+0000", 2020-01-16T00:13:49.167 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8411999999999999e-05 2020-01-16T00:13:49.167 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.167 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.168 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:49.168 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467533+0000", 2020-01-16T00:13:49.168 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9319999999999996e-06 2020-01-16T00:13:49.168 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.168 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.168 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:49.168 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467541+0000", 2020-01-16T00:13:49.168 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5289999999999993e-05 2020-01-16T00:13:49.169 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.169 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.169 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:49.169 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467607+0000", 2020-01-16T00:13:49.169 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010962 2020-01-16T00:13:49.170 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.170 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.170 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:49.170 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467716+0000", 2020-01-16T00:13:49.170 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020883699999999999 2020-01-16T00:13:49.170 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.170 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.171 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 7", 2020-01-16T00:13:49.171 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467925+0000", 2020-01-16T00:13:49.171 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063872829999999997 2020-01-16T00:13:49.171 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.171 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.171 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:49.171 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.474312+0000", 2020-01-16T00:13:49.171 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0023165769999999998 2020-01-16T00:13:49.172 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.172 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.172 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.172 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.476629+0000", 2020-01-16T00:13:49.172 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8543999999999999e-05 2020-01-16T00:13:49.172 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.172 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.172 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:49.173 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.476707+0000", 2020-01-16T00:13:49.173 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000153914 2020-01-16T00:13:49.173 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.173 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.173 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:49.173 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.476861+0000", 2020-01-16T00:13:49.173 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010593112 2020-01-16T00:13:49.173 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:49.174 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:49.174 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:49.174 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.174 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.174 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(client.4488.0:3653 3.9 e48/20 3:9557b633:::smithi12313934-43:head v 48'202, mlcod=48'200)", 2020-01-16T00:13:49.174 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:39.734299+0000", 2020-01-16T00:13:49.174 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 9.2383825000000002, 2020-01-16T00:13:49.175 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095124579999999997, 2020-01-16T00:13:49.175 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:49.175 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:49.175 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:49.175 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.175 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:49.175 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.734299+0000", 2020-01-16T00:13:49.175 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:49.176 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.176 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.176 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:49.176 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.734299+0000", 2020-01-16T00:13:49.176 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:49.176 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.176 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.176 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:49.177 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.734295+0000", 2020-01-16T00:13:49.177 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00064037399999999998 2020-01-16T00:13:49.177 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.177 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.177 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:49.177 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.734935+0000", 2020-01-16T00:13:49.177 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0710000000000002e-05 2020-01-16T00:13:49.177 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.178 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.178 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:49.178 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.734966+0000", 2020-01-16T00:13:49.178 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5869999999999999e-05 2020-01-16T00:13:49.178 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.178 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.178 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:49.178 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.734982+0000", 2020-01-16T00:13:49.179 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119574 2020-01-16T00:13:49.179 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.179 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.179 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:49.179 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.735102+0000", 2020-01-16T00:13:49.179 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2585999999999999e-05 2020-01-16T00:13:49.179 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.179 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.180 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:49.180 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.735174+0000", 2020-01-16T00:13:49.180 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0085451599999999996 2020-01-16T00:13:49.180 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.180 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.180 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:49.180 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.743719+0000", 2020-01-16T00:13:49.181 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.1713999999999996e-05 2020-01-16T00:13:49.181 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.181 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.181 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:49.181 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.743811+0000", 2020-01-16T00:13:49.181 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095124579999999997 2020-01-16T00:13:49.181 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:49.181 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:49.182 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:49.182 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.183 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.183 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:155 2.3 2:c9873fc7:::smithi12313934-30 low_tier:head [write 1542656~711168] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e50)", 2020-01-16T00:13:49.183 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:43.769943+0000", 2020-01-16T00:13:49.183 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.2027379500000004, 2020-01-16T00:13:49.184 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010710127999999999, 2020-01-16T00:13:49.184 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:49.184 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:49.184 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:49.184 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.184 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:49.184 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.769943+0000", 2020-01-16T00:13:49.185 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:49.185 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.185 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.185 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:49.185 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.769943+0000", 2020-01-16T00:13:49.185 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:49.185 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.186 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.186 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:49.186 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.769942+0000", 2020-01-16T00:13:49.186 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017878100000000001 2020-01-16T00:13:49.186 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.186 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.186 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:49.187 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.770121+0000", 2020-01-16T00:13:49.187 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2980000000000003e-06 2020-01-16T00:13:49.187 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.187 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.187 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:49.187 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.770129+0000", 2020-01-16T00:13:49.187 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9129999999999999e-06 2020-01-16T00:13:49.187 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.188 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.188 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:49.188 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.770131+0000", 2020-01-16T00:13:49.188 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0053233519999999999 2020-01-16T00:13:49.188 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.188 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.188 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:49.189 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.775455+0000", 2020-01-16T00:13:49.189 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012385099999999999 2020-01-16T00:13:49.189 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.189 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.189 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:49.189 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.775579+0000", 2020-01-16T00:13:49.189 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000211554 2020-01-16T00:13:49.190 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.190 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.190 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 7", 2020-01-16T00:13:49.190 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.775790+0000", 2020-01-16T00:13:49.190 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029614210000000001 2020-01-16T00:13:49.190 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.190 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.191 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:49.191 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.778752+0000", 2020-01-16T00:13:49.191 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001798928 2020-01-16T00:13:49.191 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.191 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.191 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.191 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.780551+0000", 2020-01-16T00:13:49.192 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9413999999999999e-05 2020-01-16T00:13:49.192 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.192 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.192 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:49.192 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.780590+0000", 2020-01-16T00:13:49.192 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3275999999999995e-05 2020-01-16T00:13:49.192 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:49.192 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:49.193 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:49.193 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.780653+0000", 2020-01-16T00:13:49.193 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010710127999999999 2020-01-16T00:13:49.193 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:49.193 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:49.193 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:49.194 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:49.194 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:49.194 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:49.197 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-01-16T00:13:49.197 INFO:teuthology.orchestra.run.smithi123.stderr:noscrub is unset 2020-01-16T00:13:49.198 INFO:tasks.rados.rados.0.smithi123.stdout:1485: finishing write tid 1 to smithi12313934-39 2020-01-16T00:13:49.198 INFO:tasks.rados.rados.0.smithi123.stdout:1485: finishing write tid 2 to smithi12313934-39 2020-01-16T00:13:49.198 INFO:tasks.rados.rados.0.smithi123.stdout:1485: finishing write tid 4 to smithi12313934-39 2020-01-16T00:13:49.198 INFO:tasks.rados.rados.0.smithi123.stdout:1485: finishing write tid 5 to smithi12313934-39 2020-01-16T00:13:49.198 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 39 v 231 (ObjNum 495 snap 0 seq_num 495) dirty exists 2020-01-16T00:13:49.198 INFO:tasks.rados.rados.0.smithi123.stdout:1485: left oid 39 (ObjNum 495 snap 0 seq_num 495) 2020-01-16T00:13:49.199 INFO:tasks.rados.rados.0.smithi123.stdout:1486: finishing write tid 1 to smithi12313934-13 2020-01-16T00:13:49.199 INFO:tasks.rados.rados.0.smithi123.stdout:1486: finishing write tid 2 to smithi12313934-13 2020-01-16T00:13:49.199 INFO:tasks.rados.rados.0.smithi123.stdout:1486: finishing write tid 4 to smithi12313934-13 2020-01-16T00:13:49.199 INFO:tasks.rados.rados.0.smithi123.stdout:1486: finishing write tid 5 to smithi12313934-13 2020-01-16T00:13:49.199 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 100 (ObjNum 496 snap 0 seq_num 496) dirty exists 2020-01-16T00:13:49.199 INFO:tasks.rados.rados.0.smithi123.stdout:1486: left oid 13 (ObjNum 496 snap 0 seq_num 496) 2020-01-16T00:13:49.199 INFO:tasks.rados.rados.0.smithi123.stdout:1487: finishing write tid 1 to smithi12313934-11 2020-01-16T00:13:49.200 INFO:tasks.rados.rados.0.smithi123.stdout:1474: done (15 left) 2020-01-16T00:13:49.200 INFO:tasks.rados.rados.0.smithi123.stdout:1475: done (14 left) 2020-01-16T00:13:49.200 INFO:tasks.rados.rados.0.smithi123.stdout:1476: done (13 left) 2020-01-16T00:13:49.200 INFO:tasks.rados.rados.0.smithi123.stdout:1477: done (12 left) 2020-01-16T00:13:49.200 INFO:tasks.rados.rados.0.smithi123.stdout:1478: done (11 left) 2020-01-16T00:13:49.200 INFO:tasks.rados.rados.0.smithi123.stdout:1479: done (10 left) 2020-01-16T00:13:49.200 INFO:tasks.rados.rados.0.smithi123.stdout:1480: done (9 left) 2020-01-16T00:13:49.201 INFO:tasks.rados.rados.0.smithi123.stdout:1481: done (8 left) 2020-01-16T00:13:49.201 INFO:tasks.rados.rados.0.smithi123.stdout:1482: done (7 left) 2020-01-16T00:13:49.201 INFO:tasks.rados.rados.0.smithi123.stdout:1483: done (6 left) 2020-01-16T00:13:49.201 INFO:tasks.rados.rados.0.smithi123.stdout:1484: done (5 left) 2020-01-16T00:13:49.201 INFO:tasks.rados.rados.0.smithi123.stdout:1485: done (4 left) 2020-01-16T00:13:49.201 INFO:tasks.rados.rados.0.smithi123.stdout:1486: done (3 left) 2020-01-16T00:13:49.201 INFO:tasks.rados.rados.0.smithi123.stdout:1490: write oid 46 current snap is 0 2020-01-16T00:13:49.202 INFO:tasks.rados.rados.0.smithi123.stdout:1490: seq_num 498 ranges {769886=636947,2091915=277971} 2020-01-16T00:13:49.202 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:49.192+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:49.207 INFO:tasks.rados.rados.0.smithi123.stdout:1490: writing smithi12313934-46 from 769886 to 1406833 tid 1 2020-01-16T00:13:49.212 INFO:tasks.rados.rados.0.smithi123.stdout:1490: writing smithi12313934-46 from 2091915 to 2369886 tid 2 2020-01-16T00:13:49.213 INFO:tasks.rados.rados.0.smithi123.stdout:1491: write (excl) oid 38 current snap is 0 2020-01-16T00:13:49.213 INFO:tasks.rados.rados.0.smithi123.stdout:1491: seq_num 499 ranges {786693=712196,2150397=236296} 2020-01-16T00:13:49.221 INFO:tasks.rados.rados.0.smithi123.stdout:1491: writing smithi12313934-38 from 786693 to 1498889 tid 1 2020-01-16T00:13:49.226 INFO:tasks.rados.rados.0.smithi123.stdout:1491: writing smithi12313934-38 from 2150397 to 2386693 tid 2 2020-01-16T00:13:49.226 INFO:tasks.rados.rados.0.smithi123.stdout:1492: read oid 27 snap -1 2020-01-16T00:13:49.227 INFO:tasks.rados.rados.0.smithi123.stdout:1492: expect (ObjNum 446 snap 0 seq_num 446) 2020-01-16T00:13:49.227 INFO:tasks.rados.rados.0.smithi123.stdout:1493: read oid 42 snap -1 2020-01-16T00:13:49.227 INFO:tasks.rados.rados.0.smithi123.stdout:1493: expect (ObjNum 432 snap 0 seq_num 432) 2020-01-16T00:13:49.227 INFO:tasks.rados.rados.0.smithi123.stdout:1494: write (excl) oid 25 current snap is 0 2020-01-16T00:13:49.227 INFO:tasks.rados.rados.0.smithi123.stdout:1494: seq_num 500 ranges {403500=787445,1692526=473837} 2020-01-16T00:13:49.233 INFO:tasks.rados.rados.0.smithi123.stdout:1494: writing smithi12313934-25 from 403500 to 1190945 tid 1 2020-01-16T00:13:49.239 INFO:tasks.rados.rados.0.smithi123.stdout:1494: writing smithi12313934-25 from 1692526 to 2166363 tid 2 2020-01-16T00:13:49.239 INFO:tasks.rados.rados.0.smithi123.stdout:1495: write oid 3 current snap is 0 2020-01-16T00:13:49.239 INFO:tasks.rados.rados.0.smithi123.stdout:1495: seq_num 501 ranges {420307=462694,1351008=617495} 2020-01-16T00:13:49.244 INFO:tasks.rados.rados.0.smithi123.stdout:1495: writing smithi12313934-3 from 420307 to 883001 tid 1 2020-01-16T00:13:49.250 INFO:tasks.rados.rados.0.smithi123.stdout:1495: writing smithi12313934-3 from 1351008 to 1968503 tid 2 2020-01-16T00:13:49.250 INFO:tasks.rados.rados.0.smithi123.stdout:1496: write (excl) oid 37 current snap is 0 2020-01-16T00:13:49.251 INFO:tasks.rados.rados.0.smithi123.stdout:1496: seq_num 502 ranges {437114=654296,1525843=477506} 2020-01-16T00:13:49.256 INFO:tasks.rados.rados.0.smithi123.stdout:1496: writing smithi12313934-37 from 437114 to 1091410 tid 1 2020-01-16T00:13:49.262 INFO:tasks.rados.rados.0.smithi123.stdout:1496: writing smithi12313934-37 from 1525843 to 2003349 tid 2 2020-01-16T00:13:49.263 INFO:tasks.rados.rados.0.smithi123.stdout:1497: write oid 34 current snap is 0 2020-01-16T00:13:49.263 INFO:tasks.rados.rados.0.smithi123.stdout:1497: seq_num 503 ranges {453921=729545,1584325=621164} 2020-01-16T00:13:49.268 INFO:tasks.rados.rados.0.smithi123.stdout:1497: writing smithi12313934-34 from 453921 to 1183466 tid 1 2020-01-16T00:13:49.274 INFO:tasks.rados.rados.0.smithi123.stdout:1497: writing smithi12313934-34 from 1584325 to 2205489 tid 2 2020-01-16T00:13:49.275 INFO:tasks.rados.rados.0.smithi123.stdout:1498: read oid 45 snap -1 2020-01-16T00:13:49.275 INFO:tasks.rados.rados.0.smithi123.stdout:1498: expect (ObjNum 466 snap 0 seq_num 466) 2020-01-16T00:13:49.275 INFO:tasks.rados.rados.0.smithi123.stdout:1499: write (excl) oid 48 current snap is 0 2020-01-16T00:13:49.275 INFO:tasks.rados.rados.0.smithi123.stdout:1499: seq_num 504 ranges {470728=404794,1526454=481175} 2020-01-16T00:13:49.279 INFO:tasks.rados.rados.0.smithi123.stdout:1499: writing smithi12313934-48 from 470728 to 875522 tid 1 2020-01-16T00:13:49.286 INFO:tasks.rados.rados.0.smithi123.stdout:1499: writing smithi12313934-48 from 1526454 to 2007629 tid 2 2020-01-16T00:13:49.286 INFO:tasks.rados.rados.0.smithi123.stdout:1500: write (excl) oid 23 current snap is 0 2020-01-16T00:13:49.286 INFO:tasks.rados.rados.0.smithi123.stdout:1500: seq_num 505 ranges {487535=480043,1584936=624833} 2020-01-16T00:13:49.292 INFO:tasks.rados.rados.0.smithi123.stdout:1500: writing smithi12313934-23 from 487535 to 967578 tid 1 2020-01-16T00:13:49.295 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:49.294+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:49.299 INFO:tasks.rados.rados.0.smithi123.stdout:1500: writing smithi12313934-23 from 1584936 to 2209769 tid 2 2020-01-16T00:13:49.299 INFO:tasks.rados.rados.0.smithi123.stdout:1501: write (excl) oid 28 current snap is 0 2020-01-16T00:13:49.300 INFO:tasks.rados.rados.0.smithi123.stdout:1501: seq_num 506 ranges {504342=555292,1643418=484844} 2020-01-16T00:13:49.305 INFO:tasks.rados.rados.0.smithi123.stdout:1501: writing smithi12313934-28 from 504342 to 1059634 tid 1 2020-01-16T00:13:49.310 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:49.310 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:13:49.310 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:13:49.310 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:49.311 INFO:tasks.rados.rados.0.smithi123.stdout:1501: writing smithi12313934-28 from 1643418 to 2128262 tid 2 2020-01-16T00:13:49.312 INFO:tasks.rados.rados.0.smithi123.stdout:1502: write oid 6 current snap is 0 2020-01-16T00:13:49.312 INFO:tasks.rados.rados.0.smithi123.stdout:1502: seq_num 507 ranges {521149=630541,1701900=628502} 2020-01-16T00:13:49.318 INFO:tasks.rados.rados.0.smithi123.stdout:1502: writing smithi12313934-6 from 521149 to 1151690 tid 1 2020-01-16T00:13:49.320 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-01-16T00:13:49.324 INFO:tasks.rados.rados.0.smithi123.stdout:1502: writing smithi12313934-6 from 1701900 to 2330402 tid 2 2020-01-16T00:13:49.325 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:49.325 INFO:tasks.rados.rados.0.smithi123.stdout:1489: expect (ObjNum 481 snap 0 seq_num 481) 2020-01-16T00:13:49.396 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:49.392+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:49.417 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:49.418 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:13:49.418 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:13:49.418 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:13:49.418 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:49.426 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-01-16T00:13:49.492 INFO:tasks.rados.rados.0.smithi123.stdout:1488: expect (ObjNum 454 snap 0 seq_num 454) 2020-01-16T00:13:49.514 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:49.512+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:49.585 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:49.585 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:13:49.585 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:13:49.586 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:13:49.586 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.586 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:943 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:49.586 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:13:49.586 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 71.668529282999998, 2020-01-16T00:13:49.586 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018847004000000001, 2020-01-16T00:13:49.586 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:49.587 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:49.587 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:49.587 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:49.587 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:49.587 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 943 2020-01-16T00:13:49.587 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.587 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:49.587 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.588 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:49.588 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:13:49.588 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:49.588 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.588 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.588 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:49.588 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:13:49.589 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:49.589 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.589 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.589 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:49.589 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914475+0000", 2020-01-16T00:13:49.589 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.508e-05 2020-01-16T00:13:49.589 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.590 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.590 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:49.590 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914490+0000", 2020-01-16T00:13:49.590 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.857e-05 2020-01-16T00:13:49.590 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.590 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.590 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:49.591 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914509+0000", 2020-01-16T00:13:49.591 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.46e-06 2020-01-16T00:13:49.591 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.591 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.591 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:49.591 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914515+0000", 2020-01-16T00:13:49.591 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2612999999999999e-05 2020-01-16T00:13:49.591 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.592 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.592 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.592 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914568+0000", 2020-01-16T00:13:49.592 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028312299999999998 2020-01-16T00:13:49.592 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.592 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.592 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:49.593 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914851+0000", 2020-01-16T00:13:49.593 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012282533 2020-01-16T00:13:49.593 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.593 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.593 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.593 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927133+0000", 2020-01-16T00:13:49.593 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4839999999999995e-05 2020-01-16T00:13:49.593 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.600 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.600 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:49.600 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927208+0000", 2020-01-16T00:13:49.600 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0047425610000000002 2020-01-16T00:13:49.600 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.600 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.601 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.601 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.931951+0000", 2020-01-16T00:13:49.601 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3628999999999994e-05 2020-01-16T00:13:49.601 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.601 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.601 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:49.601 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.932045+0000", 2020-01-16T00:13:49.602 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012810180000000001 2020-01-16T00:13:49.602 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.602 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.602 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:49.602 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933326+0000", 2020-01-16T00:13:49.602 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018847004000000001 2020-01-16T00:13:49.602 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.602 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:49.603 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.603 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.603 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.603 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:944 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:49.603 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:13:49.603 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 71.668479727000005, 2020-01-16T00:13:49.603 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019086901999999999, 2020-01-16T00:13:49.604 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:49.604 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:49.604 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:49.604 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:49.604 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:49.604 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 944 2020-01-16T00:13:49.604 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.604 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:49.605 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.605 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:49.605 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:13:49.605 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:49.605 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.605 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.605 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:49.606 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:13:49.606 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:49.606 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.606 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.606 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:49.606 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914527+0000", 2020-01-16T00:13:49.606 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7509999999999999e-06 2020-01-16T00:13:49.606 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.607 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.607 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:49.607 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914532+0000", 2020-01-16T00:13:49.607 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5149999999999997e-06 2020-01-16T00:13:49.607 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.607 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.607 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:49.607 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914538+0000", 2020-01-16T00:13:49.608 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1890000000000007e-06 2020-01-16T00:13:49.608 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.608 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.608 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:49.608 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914546+0000", 2020-01-16T00:13:49.608 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000354951 2020-01-16T00:13:49.608 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.609 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.609 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.609 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914901+0000", 2020-01-16T00:13:49.615 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1594000000000002e-05 2020-01-16T00:13:49.615 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.616 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.616 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:49.616 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914942+0000", 2020-01-16T00:13:49.616 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012300070999999999 2020-01-16T00:13:49.616 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.616 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.616 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.617 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927242+0000", 2020-01-16T00:13:49.617 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4076999999999997e-05 2020-01-16T00:13:49.617 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.617 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.617 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:49.617 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927296+0000", 2020-01-16T00:13:49.617 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0060609180000000002 2020-01-16T00:13:49.618 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.618 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.618 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.618 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933357+0000", 2020-01-16T00:13:49.618 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4488999999999999e-05 2020-01-16T00:13:49.618 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.618 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.618 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:49.619 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933422+0000", 2020-01-16T00:13:49.619 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019318300000000001 2020-01-16T00:13:49.619 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.619 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.619 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:49.619 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933615+0000", 2020-01-16T00:13:49.619 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019086901999999999 2020-01-16T00:13:49.620 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.620 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:49.620 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.620 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.620 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.621 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:945 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:49.621 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:13:49.621 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 71.668452521000006, 2020-01-16T00:13:49.621 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019342986, 2020-01-16T00:13:49.621 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:49.621 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:49.621 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:49.622 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:49.622 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:49.622 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 945 2020-01-16T00:13:49.622 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.622 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:49.622 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.622 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:49.622 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:13:49.623 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:49.623 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.623 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.623 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:49.623 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:13:49.623 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:49.623 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.623 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.624 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:49.624 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:13:49.624 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4580000000000002e-06 2020-01-16T00:13:49.624 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.624 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.624 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:49.624 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914558+0000", 2020-01-16T00:13:49.625 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7829999999999998e-06 2020-01-16T00:13:49.625 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.625 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.625 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:49.625 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914563+0000", 2020-01-16T00:13:49.625 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0001e-05 2020-01-16T00:13:49.625 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.626 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.626 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:49.626 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914573+0000", 2020-01-16T00:13:49.626 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00040365600000000001 2020-01-16T00:13:49.626 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.626 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.626 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.626 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914976+0000", 2020-01-16T00:13:49.627 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1390000000000001e-05 2020-01-16T00:13:49.627 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.627 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.627 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:49.627 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.915028+0000", 2020-01-16T00:13:49.627 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012288843000000001 2020-01-16T00:13:49.627 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.627 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.628 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.628 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927317+0000", 2020-01-16T00:13:49.628 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7871000000000002e-05 2020-01-16T00:13:49.628 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.628 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.628 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:49.628 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927375+0000", 2020-01-16T00:13:49.629 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0062580589999999998 2020-01-16T00:13:49.629 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.629 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.629 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.629 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933633+0000", 2020-01-16T00:13:49.629 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7086999999999997e-05 2020-01-16T00:13:49.629 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.629 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.630 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:49.630 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933690+0000", 2020-01-16T00:13:49.630 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020855799999999999 2020-01-16T00:13:49.630 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.631 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.631 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:49.631 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933898+0000", 2020-01-16T00:13:49.631 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019342986 2020-01-16T00:13:49.631 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.631 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:49.631 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.631 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.632 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.632 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:981 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:49.632 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:13:49.632 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 71.267675628999996, 2020-01-16T00:13:49.632 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021222494000000001, 2020-01-16T00:13:49.632 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:49.632 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:49.633 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:49.633 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:49.633 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:49.633 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 981 2020-01-16T00:13:49.633 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.633 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:49.633 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.634 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:49.634 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:13:49.634 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:49.634 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.634 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.634 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:49.635 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:13:49.635 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999948 2020-01-16T00:13:49.635 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.635 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.635 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:49.635 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315327+0000", 2020-01-16T00:13:49.635 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.508e-05 2020-01-16T00:13:49.636 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.636 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.636 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:49.636 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315352+0000", 2020-01-16T00:13:49.636 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1183999999999998e-05 2020-01-16T00:13:49.636 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.636 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.636 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:49.637 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315383+0000", 2020-01-16T00:13:49.637 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7180000000000002e-05 2020-01-16T00:13:49.637 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.637 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.637 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:49.637 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315401+0000", 2020-01-16T00:13:49.637 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017290599999999999 2020-01-16T00:13:49.638 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.638 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.638 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.638 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315574+0000", 2020-01-16T00:13:49.638 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00047908100000000002 2020-01-16T00:13:49.638 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.639 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.639 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:49.639 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316053+0000", 2020-01-16T00:13:49.639 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011808947 2020-01-16T00:13:49.639 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.639 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.639 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.639 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.327862+0000", 2020-01-16T00:13:49.640 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010067499999999999 2020-01-16T00:13:49.640 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.640 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.640 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:49.640 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.327962+0000", 2020-01-16T00:13:49.641 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0074745139999999998 2020-01-16T00:13:49.641 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.641 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.641 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.641 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.335437+0000", 2020-01-16T00:13:49.641 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000103403 2020-01-16T00:13:49.641 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.642 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.642 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:49.642 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.335540+0000", 2020-01-16T00:13:49.642 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001014544 2020-01-16T00:13:49.642 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.642 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.642 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:49.643 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336555+0000", 2020-01-16T00:13:49.643 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021222494000000001 2020-01-16T00:13:49.643 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.643 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:49.643 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.643 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.643 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.643 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:982 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:49.644 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:13:49.644 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 71.267573104999997, 2020-01-16T00:13:49.644 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.022087721000000001, 2020-01-16T00:13:49.644 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:49.644 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:49.644 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:49.644 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:49.645 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:49.645 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 982 2020-01-16T00:13:49.645 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.645 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:49.645 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.645 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:49.645 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:13:49.645 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:49.646 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.646 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.646 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:49.646 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:13:49.646 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:49.646 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.646 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.646 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:49.647 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315433+0000", 2020-01-16T00:13:49.647 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1347e-05 2020-01-16T00:13:49.647 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.647 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.647 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:49.647 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315445+0000", 2020-01-16T00:13:49.647 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4525999999999999e-05 2020-01-16T00:13:49.648 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.648 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.648 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:49.648 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315459+0000", 2020-01-16T00:13:49.648 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.024e-05 2020-01-16T00:13:49.648 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.648 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.648 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:49.649 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315470+0000", 2020-01-16T00:13:49.649 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000736468 2020-01-16T00:13:49.649 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.649 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.649 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.649 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316206+0000", 2020-01-16T00:13:49.649 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000132967 2020-01-16T00:13:49.649 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.650 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.650 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:49.650 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316339+0000", 2020-01-16T00:13:49.650 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011701717 2020-01-16T00:13:49.650 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.650 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.650 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.651 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328041+0000", 2020-01-16T00:13:49.651 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000120339 2020-01-16T00:13:49.651 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.651 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.652 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:49.652 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328161+0000", 2020-01-16T00:13:49.652 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0084341260000000001 2020-01-16T00:13:49.652 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.652 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.652 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.652 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336595+0000", 2020-01-16T00:13:49.652 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5247999999999993e-05 2020-01-16T00:13:49.653 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.653 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.653 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:49.653 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336690+0000", 2020-01-16T00:13:49.653 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00083204000000000004 2020-01-16T00:13:49.653 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.653 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.654 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:49.654 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337522+0000", 2020-01-16T00:13:49.654 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.022087721000000001 2020-01-16T00:13:49.654 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.654 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:49.654 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.654 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.654 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.655 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:983 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:13:49.655 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:13:49.655 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 71.267453395000004, 2020-01-16T00:13:49.655 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023363387999999999, 2020-01-16T00:13:49.655 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:49.655 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:49.655 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:49.656 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:49.656 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:49.656 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 983 2020-01-16T00:13:49.656 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.656 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:49.656 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.656 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:49.656 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:13:49.657 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:49.657 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.657 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.657 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:49.657 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:13:49.657 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:49.657 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.658 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.658 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:49.658 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315553+0000", 2020-01-16T00:13:49.658 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1760000000000004e-06 2020-01-16T00:13:49.658 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.658 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.658 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:49.659 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315560+0000", 2020-01-16T00:13:49.659 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3587000000000001e-05 2020-01-16T00:13:49.659 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.659 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.659 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:49.659 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315574+0000", 2020-01-16T00:13:49.659 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7800000000000003e-06 2020-01-16T00:13:49.659 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.660 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.660 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:49.660 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315581+0000", 2020-01-16T00:13:49.660 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000823024 2020-01-16T00:13:49.660 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.660 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.660 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.661 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316404+0000", 2020-01-16T00:13:49.661 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2413e-05 2020-01-16T00:13:49.661 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.661 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.661 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:49.662 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316466+0000", 2020-01-16T00:13:49.662 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011738561999999999 2020-01-16T00:13:49.662 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.662 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.662 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.662 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328205+0000", 2020-01-16T00:13:49.662 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6783999999999997e-05 2020-01-16T00:13:49.663 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.663 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.663 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:49.663 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328262+0000", 2020-01-16T00:13:49.663 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0093022560000000001 2020-01-16T00:13:49.663 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.663 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.663 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.664 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337564+0000", 2020-01-16T00:13:49.664 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000183151 2020-01-16T00:13:49.664 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.664 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.664 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:49.664 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337747+0000", 2020-01-16T00:13:49.664 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011707449999999999 2020-01-16T00:13:49.665 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.665 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.665 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:49.665 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.338918+0000", 2020-01-16T00:13:49.665 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023363387999999999 2020-01-16T00:13:49.665 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.665 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:49.665 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.666 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.666 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.666 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:986 3.9 3:93dfe6f1:::smithi12313934-40:head [write 2941380~419987] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:49.666 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:13:49.666 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 71.102153619000006, 2020-01-16T00:13:49.666 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016978400000000001, 2020-01-16T00:13:49.666 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:49.667 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:49.667 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:49.667 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:49.667 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:49.667 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 986 2020-01-16T00:13:49.667 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.667 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:49.667 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.668 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:49.668 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:13:49.668 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:49.668 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.668 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.668 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:49.668 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:13:49.668 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:49.669 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.669 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.669 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:49.669 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480852+0000", 2020-01-16T00:13:49.669 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042503899999999999 2020-01-16T00:13:49.669 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.669 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.670 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:49.670 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481277+0000", 2020-01-16T00:13:49.670 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3097e-05 2020-01-16T00:13:49.670 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.670 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.670 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:49.670 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481290+0000", 2020-01-16T00:13:49.670 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8460000000000002e-06 2020-01-16T00:13:49.671 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.671 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.671 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:49.671 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481297+0000", 2020-01-16T00:13:49.671 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8443999999999999e-05 2020-01-16T00:13:49.671 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.671 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.672 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.672 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481346+0000", 2020-01-16T00:13:49.672 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022115399999999999 2020-01-16T00:13:49.672 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.672 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.672 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:49.673 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481567+0000", 2020-01-16T00:13:49.673 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012531502 2020-01-16T00:13:49.673 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.673 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.673 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.673 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494098+0000", 2020-01-16T00:13:49.673 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.4462999999999993e-05 2020-01-16T00:13:49.674 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.674 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.674 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:49.674 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494193+0000", 2020-01-16T00:13:49.674 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017600999999999999 2020-01-16T00:13:49.674 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.674 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.674 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:13:49.675 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494369+0000", 2020-01-16T00:13:49.675 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0031739619999999998 2020-01-16T00:13:49.675 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.675 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.675 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:49.675 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497543+0000", 2020-01-16T00:13:49.675 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000175107 2020-01-16T00:13:49.675 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.676 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.676 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.676 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497718+0000", 2020-01-16T00:13:49.676 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5207000000000003e-05 2020-01-16T00:13:49.676 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.676 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.676 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.677 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497763+0000", 2020-01-16T00:13:49.677 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7042999999999998e-05 2020-01-16T00:13:49.677 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.677 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.677 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:49.677 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497800+0000", 2020-01-16T00:13:49.677 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2628e-05 2020-01-16T00:13:49.677 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.678 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.678 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:49.678 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497833+0000", 2020-01-16T00:13:49.678 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016978400000000001 2020-01-16T00:13:49.678 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.678 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:49.678 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.679 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.679 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.679 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:987 3.9 3:93dfe6f1:::smithi12313934-40:head [setxattr _header (56) in=7b,truncate 3361367] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:49.679 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.481307+0000", 2020-01-16T00:13:49.679 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 71.101701251999998, 2020-01-16T00:13:49.679 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016794989, 2020-01-16T00:13:49.679 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:49.679 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:49.680 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:49.680 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:49.680 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:49.680 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 987 2020-01-16T00:13:49.680 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.680 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:49.680 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.680 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:49.681 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481307+0000", 2020-01-16T00:13:49.681 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:49.681 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.681 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.681 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:49.681 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481307+0000", 2020-01-16T00:13:49.681 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:49.682 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.682 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.682 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:49.683 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481306+0000", 2020-01-16T00:13:49.683 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.113e-06 2020-01-16T00:13:49.683 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.683 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.683 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:49.683 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481311+0000", 2020-01-16T00:13:49.683 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1520000000000002e-06 2020-01-16T00:13:49.683 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.684 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.684 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:49.684 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481315+0000", 2020-01-16T00:13:49.684 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.7640000000000001e-06 2020-01-16T00:13:49.684 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.684 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.684 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:49.685 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481318+0000", 2020-01-16T00:13:49.685 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00029012500000000002 2020-01-16T00:13:49.685 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.685 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.685 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.685 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481608+0000", 2020-01-16T00:13:49.685 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6239000000000002e-05 2020-01-16T00:13:49.685 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.686 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.686 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:49.686 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481654+0000", 2020-01-16T00:13:49.686 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014559534000000001 2020-01-16T00:13:49.686 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.686 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.686 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.686 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.496214+0000", 2020-01-16T00:13:49.687 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000225932 2020-01-16T00:13:49.687 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.687 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.687 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:49.687 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.496439+0000", 2020-01-16T00:13:49.687 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00031099600000000002 2020-01-16T00:13:49.687 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.688 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.688 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:13:49.688 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.496750+0000", 2020-01-16T00:13:49.688 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00098608599999999995 2020-01-16T00:13:49.688 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.688 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.688 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:49.688 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497737+0000", 2020-01-16T00:13:49.689 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022081499999999999 2020-01-16T00:13:49.689 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.689 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.689 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.689 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497957+0000", 2020-01-16T00:13:49.689 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7071000000000003e-05 2020-01-16T00:13:49.689 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.690 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.690 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.690 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498014+0000", 2020-01-16T00:13:49.690 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1111e-05 2020-01-16T00:13:49.690 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.690 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.690 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:49.690 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498056+0000", 2020-01-16T00:13:49.691 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5974999999999999e-05 2020-01-16T00:13:49.691 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.691 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.691 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:49.691 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498102+0000", 2020-01-16T00:13:49.691 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016794989 2020-01-16T00:13:49.691 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.691 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:49.692 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.692 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.692 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.692 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:988 3.9 3:93dfe6f1:::smithi12313934-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:13:49.692 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:13:49.693 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 71.101668154999999, 2020-01-16T00:13:49.693 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017221692, 2020-01-16T00:13:49.693 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:49.693 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:49.693 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:49.693 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:49.693 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:49.694 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 988 2020-01-16T00:13:49.694 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.694 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:49.694 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.694 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:49.694 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:13:49.694 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:49.694 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.695 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.695 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:49.695 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:13:49.695 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:49.695 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.695 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.695 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:49.696 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481339+0000", 2020-01-16T00:13:49.696 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3089999999999997e-06 2020-01-16T00:13:49.696 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.696 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.696 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:49.696 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481343+0000", 2020-01-16T00:13:49.696 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2230000000000004e-06 2020-01-16T00:13:49.696 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.697 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.697 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:49.697 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481350+0000", 2020-01-16T00:13:49.697 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.319e-06 2020-01-16T00:13:49.697 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.697 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.697 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:49.697 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481356+0000", 2020-01-16T00:13:49.698 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000321544 2020-01-16T00:13:49.698 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.698 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.698 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.698 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481678+0000", 2020-01-16T00:13:49.698 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6375999999999999e-05 2020-01-16T00:13:49.698 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.699 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.699 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:49.699 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481714+0000", 2020-01-16T00:13:49.699 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015557666 2020-01-16T00:13:49.699 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.699 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.699 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.699 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497272+0000", 2020-01-16T00:13:49.700 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012791300000000001 2020-01-16T00:13:49.700 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.700 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.700 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:49.700 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497399+0000", 2020-01-16T00:13:49.700 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00076944600000000002 2020-01-16T00:13:49.700 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.700 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.701 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.701 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498169+0000", 2020-01-16T00:13:49.701 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018306799999999999 2020-01-16T00:13:49.701 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.701 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.701 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:49.701 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498352+0000", 2020-01-16T00:13:49.702 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020935999999999999 2020-01-16T00:13:49.702 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.702 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.702 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:49.702 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498561+0000", 2020-01-16T00:13:49.702 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017221692 2020-01-16T00:13:49.702 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.703 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:49.703 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.703 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.703 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.703 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1239 3.3 3:c204aa4f:::smithi12313934-45:head [setxattr _header (56) in=7b,truncate 2000033] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:13:49.704 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:13:49.704 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 66.992594335999996, 2020-01-16T00:13:49.704 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019161661999999999, 2020-01-16T00:13:49.704 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:49.704 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:49.704 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:49.704 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:49.705 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:49.705 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1239 2020-01-16T00:13:49.705 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.705 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:49.705 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.705 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:49.705 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:13:49.705 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:49.706 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.706 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.706 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:49.706 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:13:49.706 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:49.706 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.706 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.707 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:49.707 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590411+0000", 2020-01-16T00:13:49.707 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3879999999999999e-05 2020-01-16T00:13:49.707 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.707 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.707 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:49.707 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590425+0000", 2020-01-16T00:13:49.707 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6542999999999998e-05 2020-01-16T00:13:49.708 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.708 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.708 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:49.708 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590471+0000", 2020-01-16T00:13:49.708 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8907999999999999e-05 2020-01-16T00:13:49.708 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.708 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.709 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:49.709 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590490+0000", 2020-01-16T00:13:49.709 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0045691960000000002 2020-01-16T00:13:49.709 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.709 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.709 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.709 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.595059+0000", 2020-01-16T00:13:49.709 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0035638940000000002 2020-01-16T00:13:49.710 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.710 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.710 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:49.710 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598623+0000", 2020-01-16T00:13:49.710 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0077558310000000004 2020-01-16T00:13:49.710 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.710 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.711 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.711 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606379+0000", 2020-01-16T00:13:49.711 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012171999999999999 2020-01-16T00:13:49.711 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.711 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.711 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:49.711 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606501+0000", 2020-01-16T00:13:49.711 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013144200000000001 2020-01-16T00:13:49.712 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.712 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.712 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2020-01-16T00:13:49.712 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606632+0000", 2020-01-16T00:13:49.712 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001288565 2020-01-16T00:13:49.712 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.712 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.712 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:49.713 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607921+0000", 2020-01-16T00:13:49.713 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033399299999999999 2020-01-16T00:13:49.713 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.713 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.713 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.714 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.608255+0000", 2020-01-16T00:13:49.714 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012280870000000001 2020-01-16T00:13:49.714 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.714 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.714 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.714 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609483+0000", 2020-01-16T00:13:49.714 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2286000000000001e-05 2020-01-16T00:13:49.715 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.715 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.715 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:49.715 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609525+0000", 2020-01-16T00:13:49.715 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0121999999999999e-05 2020-01-16T00:13:49.715 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.715 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.715 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:49.716 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609575+0000", 2020-01-16T00:13:49.716 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019161661999999999 2020-01-16T00:13:49.716 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.716 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:49.716 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.716 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.716 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.717 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1240 3.3 3:c204aa4f:::smithi12313934-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-01-16T00:13:49.717 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:13:49.717 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 66.992494147000002, 2020-01-16T00:13:49.717 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019268828000000002, 2020-01-16T00:13:49.717 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:49.717 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:49.717 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:49.718 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:49.718 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:49.718 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1240 2020-01-16T00:13:49.718 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.718 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:49.718 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.718 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:49.718 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:13:49.719 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:49.719 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.719 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.719 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:49.719 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:13:49.719 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:49.719 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.720 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.720 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:49.720 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590512+0000", 2020-01-16T00:13:49.720 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0525e-05 2020-01-16T00:13:49.720 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.720 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.720 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:49.720 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590522+0000", 2020-01-16T00:13:49.721 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6418000000000001e-05 2020-01-16T00:13:49.721 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.721 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.721 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:49.721 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590539+0000", 2020-01-16T00:13:49.721 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0679999999999999e-05 2020-01-16T00:13:49.721 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.721 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.722 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:49.722 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590549+0000", 2020-01-16T00:13:49.722 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0083626789999999996 2020-01-16T00:13:49.722 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.722 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.722 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.722 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598912+0000", 2020-01-16T00:13:49.723 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013344 2020-01-16T00:13:49.723 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.723 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.723 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:49.723 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599045+0000", 2020-01-16T00:13:49.723 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0084002269999999997 2020-01-16T00:13:49.724 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.724 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.724 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.724 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607446+0000", 2020-01-16T00:13:49.724 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000109873 2020-01-16T00:13:49.724 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.724 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.725 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:49.725 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607555+0000", 2020-01-16T00:13:49.725 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0020573309999999999 2020-01-16T00:13:49.725 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.725 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.725 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.725 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609613+0000", 2020-01-16T00:13:49.726 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6351e-05 2020-01-16T00:13:49.726 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.726 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.726 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:49.726 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609699+0000", 2020-01-16T00:13:49.726 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3374999999999994e-05 2020-01-16T00:13:49.726 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.726 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.727 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:49.727 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609782+0000", 2020-01-16T00:13:49.727 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019268828000000002 2020-01-16T00:13:49.727 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.727 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:49.727 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.728 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.728 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.728 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1437 3.3 3:c204aa4f:::smithi12313934-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e28)", 2020-01-16T00:13:49.728 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:47.900594+0000", 2020-01-16T00:13:49.728 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 61.682413603000001, 2020-01-16T00:13:49.728 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016745237999999999, 2020-01-16T00:13:49.728 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:49.728 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:49.729 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:49.729 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:49.729 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:49.729 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1437 2020-01-16T00:13:49.729 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.729 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:49.729 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.730 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:49.730 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900594+0000", 2020-01-16T00:13:49.730 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:49.730 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.730 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.730 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:49.730 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900594+0000", 2020-01-16T00:13:49.730 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:49.731 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.731 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.731 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:49.731 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900593+0000", 2020-01-16T00:13:49.731 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5059999999999995e-06 2020-01-16T00:13:49.731 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.731 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.732 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:49.732 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900601+0000", 2020-01-16T00:13:49.732 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5077e-05 2020-01-16T00:13:49.732 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.732 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.732 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:49.732 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900616+0000", 2020-01-16T00:13:49.732 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0183000000000001e-05 2020-01-16T00:13:49.733 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.733 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.733 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:49.733 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.900627+0000", 2020-01-16T00:13:49.733 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0063667660000000003 2020-01-16T00:13:49.733 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.733 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.734 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.734 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.906993+0000", 2020-01-16T00:13:49.734 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8988e-05 2020-01-16T00:13:49.735 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.735 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.735 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:49.735 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.907032+0000", 2020-01-16T00:13:49.735 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0080128850000000008 2020-01-16T00:13:49.735 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.735 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.736 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.736 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.915045+0000", 2020-01-16T00:13:49.736 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013200899999999999 2020-01-16T00:13:49.736 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.736 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.736 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:49.736 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.915177+0000", 2020-01-16T00:13:49.737 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0018373090000000001 2020-01-16T00:13:49.737 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.737 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.737 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.737 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.917015+0000", 2020-01-16T00:13:49.737 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016701699999999999 2020-01-16T00:13:49.737 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.738 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.738 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:49.738 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.917182+0000", 2020-01-16T00:13:49.738 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015782400000000001 2020-01-16T00:13:49.738 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.738 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.738 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:49.739 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:47.917339+0000", 2020-01-16T00:13:49.739 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016745237999999999 2020-01-16T00:13:49.739 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.739 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:49.739 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.739 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.739 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.740 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3124 3.9 3:9fc687b2:::smithi12313934-44:head [write 1903628~435462] snapc 0=[] ondisk+write+known_if_redirected e44)", 2020-01-16T00:13:49.740 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.276145+0000", 2020-01-16T00:13:49.740 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 24.306862425999999, 2020-01-16T00:13:49.740 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017361898000000001, 2020-01-16T00:13:49.740 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:49.740 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:49.740 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:49.740 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:49.741 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:49.741 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3124 2020-01-16T00:13:49.741 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.741 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:49.741 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.741 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:49.741 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276145+0000", 2020-01-16T00:13:49.742 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:49.742 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.742 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.742 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:49.742 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276145+0000", 2020-01-16T00:13:49.742 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:49.742 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.742 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.743 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:49.743 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276143+0000", 2020-01-16T00:13:49.743 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042136099999999998 2020-01-16T00:13:49.743 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.743 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.743 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:49.743 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276564+0000", 2020-01-16T00:13:49.744 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.349e-05 2020-01-16T00:13:49.744 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.744 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.744 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:49.744 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276578+0000", 2020-01-16T00:13:49.745 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2469999999999999e-06 2020-01-16T00:13:49.745 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.745 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.745 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:49.745 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276583+0000", 2020-01-16T00:13:49.745 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4400000000000001e-05 2020-01-16T00:13:49.745 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.746 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.746 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.746 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276637+0000", 2020-01-16T00:13:49.746 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0046361789999999998 2020-01-16T00:13:49.746 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.746 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.747 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:49.747 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281274+0000", 2020-01-16T00:13:49.747 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0090025520000000005 2020-01-16T00:13:49.747 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.747 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.747 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.747 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.290276+0000", 2020-01-16T00:13:49.747 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011584200000000001 2020-01-16T00:13:49.748 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.748 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.748 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:49.748 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.290392+0000", 2020-01-16T00:13:49.748 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00025563099999999998 2020-01-16T00:13:49.748 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.748 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.748 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:13:49.749 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.290648+0000", 2020-01-16T00:13:49.749 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0026741360000000001 2020-01-16T00:13:49.749 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.749 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.749 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:49.749 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293322+0000", 2020-01-16T00:13:49.749 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9764000000000002e-05 2020-01-16T00:13:49.750 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.750 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.750 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.750 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293421+0000", 2020-01-16T00:13:49.750 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4779000000000001e-05 2020-01-16T00:13:49.750 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.750 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.750 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.751 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293446+0000", 2020-01-16T00:13:49.751 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4628000000000001e-05 2020-01-16T00:13:49.751 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.751 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.751 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:49.751 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293481+0000", 2020-01-16T00:13:49.751 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.6444999999999998e-05 2020-01-16T00:13:49.752 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.752 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.752 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:49.752 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293507+0000", 2020-01-16T00:13:49.752 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017361898000000001 2020-01-16T00:13:49.752 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.752 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:49.752 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.753 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.753 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.753 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3125 3.9 3:9fc687b2:::smithi12313934-44:head [setxattr _header (56) in=7b,truncate 2386660] snapc 0=[] ondisk+write+known_if_redirected e44)", 2020-01-16T00:13:49.753 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.276594+0000", 2020-01-16T00:13:49.753 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 24.306413654, 2020-01-16T00:13:49.753 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017760560000000002, 2020-01-16T00:13:49.753 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:49.753 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:49.754 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:49.754 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:49.754 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:49.754 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3125 2020-01-16T00:13:49.754 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.755 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:49.755 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.755 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:49.755 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276594+0000", 2020-01-16T00:13:49.755 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:49.755 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.755 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.756 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:49.756 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276594+0000", 2020-01-16T00:13:49.756 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:49.756 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.756 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.756 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:49.756 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276593+0000", 2020-01-16T00:13:49.757 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7320000000000003e-06 2020-01-16T00:13:49.757 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.757 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.757 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:49.757 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276599+0000", 2020-01-16T00:13:49.757 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3302000000000001e-05 2020-01-16T00:13:49.757 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.757 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.758 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:49.758 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276612+0000", 2020-01-16T00:13:49.758 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1659999999999998e-06 2020-01-16T00:13:49.758 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.758 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.758 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:49.758 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276615+0000", 2020-01-16T00:13:49.759 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0047328450000000003 2020-01-16T00:13:49.759 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.759 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.759 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.759 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281348+0000", 2020-01-16T00:13:49.759 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3966000000000003e-05 2020-01-16T00:13:49.759 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.759 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.760 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:49.760 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281422+0000", 2020-01-16T00:13:49.760 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01082613 2020-01-16T00:13:49.760 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.760 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.761 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.761 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.292248+0000", 2020-01-16T00:13:49.761 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000148617 2020-01-16T00:13:49.761 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.761 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.761 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:49.761 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.292397+0000", 2020-01-16T00:13:49.762 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019936300000000001 2020-01-16T00:13:49.762 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.762 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.762 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:13:49.762 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.292596+0000", 2020-01-16T00:13:49.762 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011208870000000001 2020-01-16T00:13:49.762 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.763 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.763 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.763 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293717+0000", 2020-01-16T00:13:49.763 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00039491599999999998 2020-01-16T00:13:49.763 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.763 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.763 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:49.764 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294112+0000", 2020-01-16T00:13:49.764 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000129326 2020-01-16T00:13:49.764 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.764 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.764 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.764 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294241+0000", 2020-01-16T00:13:49.764 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5586000000000001e-05 2020-01-16T00:13:49.764 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.765 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.765 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:49.765 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294277+0000", 2020-01-16T00:13:49.765 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7893999999999997e-05 2020-01-16T00:13:49.765 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.766 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.766 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:49.766 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294355+0000", 2020-01-16T00:13:49.766 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017760560000000002 2020-01-16T00:13:49.766 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.766 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:49.766 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.767 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.767 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.767 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3126 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e44)", 2020-01-16T00:13:49.767 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.276634+0000", 2020-01-16T00:13:49.767 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 24.306373939, 2020-01-16T00:13:49.767 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017942583000000002, 2020-01-16T00:13:49.767 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:49.767 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:49.768 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:49.768 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:49.768 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:49.768 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3126 2020-01-16T00:13:49.768 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.768 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:49.768 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.769 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:49.769 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276634+0000", 2020-01-16T00:13:49.769 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:49.769 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.769 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.769 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:49.769 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276634+0000", 2020-01-16T00:13:49.769 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:49.770 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.770 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.770 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:49.770 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276633+0000", 2020-01-16T00:13:49.770 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0769999999999999e-06 2020-01-16T00:13:49.770 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.770 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.771 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:49.771 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276636+0000", 2020-01-16T00:13:49.771 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5390000000000001e-06 2020-01-16T00:13:49.771 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.771 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.771 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:49.771 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276641+0000", 2020-01-16T00:13:49.771 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9969999999999999e-06 2020-01-16T00:13:49.772 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.772 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.772 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:49.772 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276644+0000", 2020-01-16T00:13:49.772 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0048246970000000002 2020-01-16T00:13:49.772 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.772 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.772 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.773 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281469+0000", 2020-01-16T00:13:49.773 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8723999999999999e-05 2020-01-16T00:13:49.773 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.773 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.773 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:49.773 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281567+0000", 2020-01-16T00:13:49.773 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011633447 2020-01-16T00:13:49.774 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.774 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.774 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.774 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293201+0000", 2020-01-16T00:13:49.774 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6796000000000002e-05 2020-01-16T00:13:49.774 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.774 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.774 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:49.775 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293268+0000", 2020-01-16T00:13:49.775 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001125247 2020-01-16T00:13:49.775 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.775 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.775 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.776 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294393+0000", 2020-01-16T00:13:49.776 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.6082000000000006e-05 2020-01-16T00:13:49.776 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.776 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.776 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:49.776 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294489+0000", 2020-01-16T00:13:49.776 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7515000000000005e-05 2020-01-16T00:13:49.777 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.777 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.777 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:49.777 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294576+0000", 2020-01-16T00:13:49.777 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017942583000000002 2020-01-16T00:13:49.777 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.777 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:49.778 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.778 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.778 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.778 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3204 3.9 3:93dfe6f1:::smithi12313934-40:head [setxattr _header (56) in=7b,truncate 2554730] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:13:49.778 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.568544+0000", 2020-01-16T00:13:49.778 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 20.014463718999998, 2020-01-16T00:13:49.778 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.026260867, 2020-01-16T00:13:49.778 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:49.779 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:49.779 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:49.779 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:49.779 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:49.779 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3204 2020-01-16T00:13:49.779 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.779 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:49.779 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.780 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:49.780 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568544+0000", 2020-01-16T00:13:49.780 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:49.780 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.780 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.780 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:49.780 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568544+0000", 2020-01-16T00:13:49.781 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:49.781 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.781 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.781 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:49.781 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568543+0000", 2020-01-16T00:13:49.781 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0810000000000001e-06 2020-01-16T00:13:49.781 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.781 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.782 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:49.782 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568548+0000", 2020-01-16T00:13:49.782 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6950000000000002e-06 2020-01-16T00:13:49.782 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.782 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.782 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:49.782 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568552+0000", 2020-01-16T00:13:49.783 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.875e-06 2020-01-16T00:13:49.783 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.783 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.783 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:49.783 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568555+0000", 2020-01-16T00:13:49.783 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0033130849999999999 2020-01-16T00:13:49.784 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.784 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.784 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.784 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.571868+0000", 2020-01-16T00:13:49.784 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0026014609999999998 2020-01-16T00:13:49.784 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.784 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.784 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:49.785 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.574469+0000", 2020-01-16T00:13:49.785 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010261381999999999 2020-01-16T00:13:49.785 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.785 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.785 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.785 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.584731+0000", 2020-01-16T00:13:49.786 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019704 2020-01-16T00:13:49.786 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.786 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.786 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:49.786 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.584928+0000", 2020-01-16T00:13:49.786 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000280869 2020-01-16T00:13:49.787 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.787 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.787 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:13:49.787 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.585209+0000", 2020-01-16T00:13:49.787 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002273859 2020-01-16T00:13:49.787 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.787 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.787 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.788 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.587483+0000", 2020-01-16T00:13:49.788 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3356000000000001e-05 2020-01-16T00:13:49.788 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.789 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.789 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:49.789 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.587506+0000", 2020-01-16T00:13:49.789 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0071864939999999999 2020-01-16T00:13:49.789 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.790 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.790 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:49.790 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594692+0000", 2020-01-16T00:13:49.790 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5475000000000001e-05 2020-01-16T00:13:49.790 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.790 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.790 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:49.791 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594738+0000", 2020-01-16T00:13:49.791 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7070000000000007e-05 2020-01-16T00:13:49.791 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.791 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.791 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:49.791 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594805+0000", 2020-01-16T00:13:49.792 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.026260867 2020-01-16T00:13:49.792 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.792 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:49.792 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.792 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.792 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.793 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3205 3.9 3:93dfe6f1:::smithi12313934-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e45)", 2020-01-16T00:13:49.793 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.568599+0000", 2020-01-16T00:13:49.793 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 20.014408784, 2020-01-16T00:13:49.793 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.026406882, 2020-01-16T00:13:49.793 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:49.793 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:49.793 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:49.794 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:49.794 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:49.794 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3205 2020-01-16T00:13:49.794 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.794 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:49.794 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.794 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:49.794 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568599+0000", 2020-01-16T00:13:49.795 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:49.795 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.795 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.795 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:49.795 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568599+0000", 2020-01-16T00:13:49.795 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:49.795 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.796 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.796 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:49.796 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568598+0000", 2020-01-16T00:13:49.796 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3939999999999998e-06 2020-01-16T00:13:49.796 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.796 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.796 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:49.797 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568603+0000", 2020-01-16T00:13:49.797 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9391000000000001e-05 2020-01-16T00:13:49.797 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.797 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.797 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:49.797 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568622+0000", 2020-01-16T00:13:49.798 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9599999999999997e-06 2020-01-16T00:13:49.798 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.798 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.798 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:49.798 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568628+0000", 2020-01-16T00:13:49.798 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0058679700000000001 2020-01-16T00:13:49.799 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.799 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.799 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.799 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.574496+0000", 2020-01-16T00:13:49.799 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5528999999999998e-05 2020-01-16T00:13:49.799 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.799 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.799 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:49.800 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.574532+0000", 2020-01-16T00:13:49.800 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011986749 2020-01-16T00:13:49.800 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.800 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.800 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.800 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.586518+0000", 2020-01-16T00:13:49.800 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016522600000000001 2020-01-16T00:13:49.801 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.801 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.801 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:49.801 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.586684+0000", 2020-01-16T00:13:49.801 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0081693890000000009 2020-01-16T00:13:49.801 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.802 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.802 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.802 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594853+0000", 2020-01-16T00:13:49.802 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7082000000000004e-05 2020-01-16T00:13:49.802 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.802 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.803 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:49.803 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594940+0000", 2020-01-16T00:13:49.803 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5789000000000003e-05 2020-01-16T00:13:49.803 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.803 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.803 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:49.803 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.595006+0000", 2020-01-16T00:13:49.803 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.026406882 2020-01-16T00:13:49.804 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.804 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:49.804 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.804 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.804 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.804 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3219 3.3 3:c478bfe8:::smithi12313934-10:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:13:49.804 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.580957+0000", 2020-01-16T00:13:49.805 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 20.002051267999999, 2020-01-16T00:13:49.805 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016807971000000001, 2020-01-16T00:13:49.805 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:49.805 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:49.805 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:49.805 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:49.805 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:49.806 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3219 2020-01-16T00:13:49.806 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.806 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:49.806 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.806 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:49.806 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580957+0000", 2020-01-16T00:13:49.806 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:49.806 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.807 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.807 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:49.807 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580957+0000", 2020-01-16T00:13:49.807 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:49.807 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.807 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.808 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:49.808 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580955+0000", 2020-01-16T00:13:49.808 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5120000000000002e-06 2020-01-16T00:13:49.808 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.808 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.809 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:49.809 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580963+0000", 2020-01-16T00:13:49.809 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6799999999999993e-06 2020-01-16T00:13:49.809 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.809 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.809 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:49.810 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580971+0000", 2020-01-16T00:13:49.810 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3109999999999996e-06 2020-01-16T00:13:49.810 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.810 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.810 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:49.810 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580976+0000", 2020-01-16T00:13:49.811 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014306146 2020-01-16T00:13:49.811 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.811 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.811 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.811 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.595282+0000", 2020-01-16T00:13:49.811 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.1883000000000001e-05 2020-01-16T00:13:49.811 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.812 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.812 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:49.812 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.595374+0000", 2020-01-16T00:13:49.812 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023904989999999999 2020-01-16T00:13:49.812 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.812 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.812 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:49.813 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.597765+0000", 2020-01-16T00:13:49.813 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016807971000000001 2020-01-16T00:13:49.813 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.813 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:49.813 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.813 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.813 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.814 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3220 3.3 3:c478bfe8:::smithi12313934-10:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:13:49.814 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.580982+0000", 2020-01-16T00:13:49.814 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 20.002026220000001, 2020-01-16T00:13:49.814 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017920330000000002, 2020-01-16T00:13:49.814 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:49.814 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:49.814 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:49.814 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:49.815 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:49.815 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3220 2020-01-16T00:13:49.815 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.815 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:49.815 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.815 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:49.815 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580982+0000", 2020-01-16T00:13:49.816 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:49.816 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.816 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.816 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:49.816 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580982+0000", 2020-01-16T00:13:49.816 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:49.816 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.816 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.817 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:49.817 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580981+0000", 2020-01-16T00:13:49.817 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0380000000000001e-06 2020-01-16T00:13:49.817 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.817 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.817 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:49.817 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580985+0000", 2020-01-16T00:13:49.817 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8929000000000001e-05 2020-01-16T00:13:49.818 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.818 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.818 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:49.818 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581004+0000", 2020-01-16T00:13:49.818 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7339999999999999e-06 2020-01-16T00:13:49.818 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.818 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.818 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:49.819 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581009+0000", 2020-01-16T00:13:49.819 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01679812 2020-01-16T00:13:49.819 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.819 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.819 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.819 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.597807+0000", 2020-01-16T00:13:49.819 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013333400000000001 2020-01-16T00:13:49.819 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.820 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.820 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:49.820 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.597940+0000", 2020-01-16T00:13:49.821 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000961899 2020-01-16T00:13:49.821 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.821 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.821 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:49.821 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.598902+0000", 2020-01-16T00:13:49.821 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017920330000000002 2020-01-16T00:13:49.821 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.822 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:49.822 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.822 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.822 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.822 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3221 3.3 3:c478bfe8:::smithi12313934-10:head [read 0~2403467,checksum] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:13:49.822 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.581014+0000", 2020-01-16T00:13:49.822 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 20.001993871, 2020-01-16T00:13:49.822 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018784551999999999, 2020-01-16T00:13:49.823 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:49.823 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:49.823 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:49.823 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:49.823 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:49.823 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3221 2020-01-16T00:13:49.823 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.823 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:49.824 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.824 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:49.824 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581014+0000", 2020-01-16T00:13:49.824 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:49.824 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.824 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.824 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:49.825 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581014+0000", 2020-01-16T00:13:49.825 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:49.825 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.825 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.825 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:49.825 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581013+0000", 2020-01-16T00:13:49.825 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9390000000000002e-06 2020-01-16T00:13:49.825 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.826 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.826 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:49.826 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581019+0000", 2020-01-16T00:13:49.826 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1131000000000001e-05 2020-01-16T00:13:49.826 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.826 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.826 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:49.827 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581030+0000", 2020-01-16T00:13:49.827 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.55e-06 2020-01-16T00:13:49.827 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.827 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.827 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:49.827 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581037+0000", 2020-01-16T00:13:49.827 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017890321000000001 2020-01-16T00:13:49.827 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.828 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.828 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:49.828 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.598927+0000", 2020-01-16T00:13:49.828 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000105999 2020-01-16T00:13:49.828 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.828 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.828 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:49.828 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.599033+0000", 2020-01-16T00:13:49.829 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00076542400000000001 2020-01-16T00:13:49.829 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:49.829 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:49.829 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:49.829 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.599798+0000", 2020-01-16T00:13:49.829 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018784551999999999 2020-01-16T00:13:49.829 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.829 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:49.830 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.830 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:49.830 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:49.830 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:49.833 INFO:tasks.rados.rados.0.smithi123.stdout:1487: finishing write tid 2 to smithi12313934-11 2020-01-16T00:13:49.833 INFO:tasks.rados.rados.0.smithi123.stdout:1487: finishing write tid 4 to smithi12313934-11 2020-01-16T00:13:49.833 INFO:tasks.rados.rados.0.smithi123.stdout:1487: finishing write tid 5 to smithi12313934-11 2020-01-16T00:13:49.833 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 11 v 149 (ObjNum 497 snap 0 seq_num 497) dirty exists 2020-01-16T00:13:49.833 INFO:tasks.rados.rados.0.smithi123.stdout:1487: left oid 11 (ObjNum 497 snap 0 seq_num 497) 2020-01-16T00:13:49.834 INFO:tasks.rados.rados.0.smithi123.stdout:1490: finishing write tid 1 to smithi12313934-46 2020-01-16T00:13:49.834 INFO:tasks.rados.rados.0.smithi123.stdout:1490: finishing write tid 2 to smithi12313934-46 2020-01-16T00:13:49.834 INFO:tasks.rados.rados.0.smithi123.stdout:1490: finishing write tid 4 to smithi12313934-46 2020-01-16T00:13:49.834 INFO:tasks.rados.rados.0.smithi123.stdout:1490: finishing write tid 5 to smithi12313934-46 2020-01-16T00:13:49.834 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 46 v 269 (ObjNum 498 snap 0 seq_num 498) dirty exists 2020-01-16T00:13:49.834 INFO:tasks.rados.rados.0.smithi123.stdout:1490: left oid 46 (ObjNum 498 snap 0 seq_num 498) 2020-01-16T00:13:49.834 INFO:tasks.rados.rados.0.smithi123.stdout:1491: finishing write tid 1 to smithi12313934-38 2020-01-16T00:13:49.834 INFO:tasks.rados.rados.0.smithi123.stdout:1487: done (15 left) 2020-01-16T00:13:49.835 INFO:tasks.rados.rados.0.smithi123.stdout:1488: done (14 left) 2020-01-16T00:13:49.835 INFO:tasks.rados.rados.0.smithi123.stdout:1489: done (13 left) 2020-01-16T00:13:49.835 INFO:tasks.rados.rados.0.smithi123.stdout:1490: done (12 left) 2020-01-16T00:13:49.835 INFO:tasks.rados.rados.0.smithi123.stdout:1503: write oid 35 current snap is 0 2020-01-16T00:13:49.835 INFO:tasks.rados.rados.0.smithi123.stdout:1503: seq_num 508 ranges {537956=705790,1644029=772160} 2020-01-16T00:13:49.835 INFO:tasks.rados.rados.0.smithi123.stdout:1503: writing smithi12313934-35 from 537956 to 1243746 tid 1 2020-01-16T00:13:49.835 INFO:tasks.rados.rados.0.smithi123.stdout:1503: writing smithi12313934-35 from 1644029 to 2416189 tid 2 2020-01-16T00:13:49.835 INFO:tasks.rados.rados.0.smithi123.stdout:1504: write oid 8 current snap is 0 2020-01-16T00:13:49.836 INFO:tasks.rados.rados.0.smithi123.stdout:1504: seq_num 509 ranges {554763=781039,2102511=452252} 2020-01-16T00:13:49.836 INFO:tasks.rados.rados.0.smithi123.stdout:1504: writing smithi12313934-8 from 554763 to 1335802 tid 1 2020-01-16T00:13:49.836 INFO:tasks.rados.rados.0.smithi123.stdout:1504: writing smithi12313934-8 from 2102511 to 2554763 tid 2 2020-01-16T00:13:49.836 INFO:tasks.rados.rados.0.smithi123.stdout:1505: read oid 14 snap -1 2020-01-16T00:13:49.836 INFO:tasks.rados.rados.0.smithi123.stdout:1505: expect (ObjNum 362 snap 0 seq_num 362) 2020-01-16T00:13:49.836 INFO:tasks.rados.rados.0.smithi123.stdout:1506: write (excl) oid 24 current snap is 0 2020-01-16T00:13:49.836 INFO:tasks.rados.rados.0.smithi123.stdout:1506: seq_num 510 ranges {571570=572641,1877346=694224} 2020-01-16T00:13:49.836 INFO:tasks.rados.rados.0.smithi123.stdout:1506: writing smithi12313934-24 from 571570 to 1144211 tid 1 2020-01-16T00:13:49.837 INFO:tasks.rados.rados.0.smithi123.stdout:1506: writing smithi12313934-24 from 1877346 to 2571570 tid 2 2020-01-16T00:13:49.837 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:49.837 INFO:tasks.rados.rados.0.smithi123.stdout:1491: finishing write tid 2 to smithi12313934-38 2020-01-16T00:13:49.837 INFO:tasks.rados.rados.0.smithi123.stdout:1492: expect (ObjNum 446 snap 0 seq_num 446) 2020-01-16T00:13:49.837 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:49.830+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:49.838 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-01-16T00:13:49.932 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:49.930+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:49.986 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:49.987 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:13:49.987 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:13:49.987 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:49.996 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-01-16T00:13:50.050 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:50.048+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:50.078 INFO:tasks.rados.rados.0.smithi123.stdout:1491: finishing write tid 4 to smithi12313934-38 2020-01-16T00:13:50.078 INFO:tasks.rados.rados.0.smithi123.stdout:1491: finishing write tid 5 to smithi12313934-38 2020-01-16T00:13:50.078 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 38 v 177 (ObjNum 499 snap 0 seq_num 499) dirty exists 2020-01-16T00:13:50.079 INFO:tasks.rados.rados.0.smithi123.stdout:1491: left oid 38 (ObjNum 499 snap 0 seq_num 499) 2020-01-16T00:13:50.079 INFO:tasks.rados.rados.0.smithi123.stdout:1493: expect (ObjNum 432 snap 0 seq_num 432) 2020-01-16T00:13:50.109 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:50.110 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:13:50.110 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:13:50.110 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:13:50.110 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:50.118 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-01-16T00:13:50.135 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:50.130+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:50.241 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:50.236+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:50.272 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:50.272 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:13:50.272 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:13:50.273 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:13:50.273 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.273 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.5.13:36 2.0 2:20e40cec:::smithi12313934-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 113152~1699840] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:13:50.273 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:13:50.273 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 72.436152786999997, 2020-01-16T00:13:50.273 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011736636, 2020-01-16T00:13:50.273 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:50.274 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:50.274 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:50.274 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.274 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:50.274 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:13:50.274 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:50.274 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.275 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.275 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:50.275 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:13:50.275 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999952 2020-01-16T00:13:50.275 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.275 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.275 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:50.275 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831934+0000", 2020-01-16T00:13:50.276 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001034325 2020-01-16T00:13:50.276 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.276 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.276 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:50.276 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.832968+0000", 2020-01-16T00:13:50.276 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.506e-05 2020-01-16T00:13:50.276 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.277 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.277 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:50.277 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833004+0000", 2020-01-16T00:13:50.277 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3406e-05 2020-01-16T00:13:50.277 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.277 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.277 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:50.277 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833017+0000", 2020-01-16T00:13:50.278 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015477000000000001 2020-01-16T00:13:50.278 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.278 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.278 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:50.278 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833172+0000", 2020-01-16T00:13:50.278 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033750700000000002 2020-01-16T00:13:50.278 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.279 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.279 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:50.279 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833509+0000", 2020-01-16T00:13:50.279 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00055861400000000003 2020-01-16T00:13:50.279 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.279 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.279 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:13:50.280 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.834068+0000", 2020-01-16T00:13:50.280 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.007474545 2020-01-16T00:13:50.280 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.280 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.280 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:50.280 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.841542+0000", 2020-01-16T00:13:50.280 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00056319499999999995 2020-01-16T00:13:50.280 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.281 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.281 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:50.281 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.842106+0000", 2020-01-16T00:13:50.281 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001371017 2020-01-16T00:13:50.281 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.281 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.281 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:50.282 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843477+0000", 2020-01-16T00:13:50.282 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000123938 2020-01-16T00:13:50.282 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.282 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.282 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:50.282 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843601+0000", 2020-01-16T00:13:50.283 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5012000000000006e-05 2020-01-16T00:13:50.283 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.283 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.283 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:50.283 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843676+0000", 2020-01-16T00:13:50.283 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011736636 2020-01-16T00:13:50.283 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.284 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:50.284 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.284 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.284 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.284 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1187 3.d 3:b1521a3e:::smithi12313934-25:head [write 1721042~423699] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:13:50.284 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.263828+0000", 2020-01-16T00:13:50.284 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 68.004263714999993, 2020-01-16T00:13:50.285 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011305709000000001, 2020-01-16T00:13:50.285 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:50.285 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:50.285 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:50.285 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:50.285 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:50.285 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1187 2020-01-16T00:13:50.285 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.286 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:50.286 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.286 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:50.286 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.263828+0000", 2020-01-16T00:13:50.286 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:50.286 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.286 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.287 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:50.287 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.263828+0000", 2020-01-16T00:13:50.287 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:13:50.287 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.287 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.287 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:50.287 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.263824+0000", 2020-01-16T00:13:50.287 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00048485899999999997 2020-01-16T00:13:50.288 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.288 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.288 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:50.288 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264309+0000", 2020-01-16T00:13:50.288 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5622999999999999e-05 2020-01-16T00:13:50.288 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.288 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.288 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:50.289 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264325+0000", 2020-01-16T00:13:50.289 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2530000000000004e-06 2020-01-16T00:13:50.289 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.289 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.289 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:50.289 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264330+0000", 2020-01-16T00:13:50.289 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000102252 2020-01-16T00:13:50.290 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.290 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.290 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:50.290 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264432+0000", 2020-01-16T00:13:50.290 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022361000000000001 2020-01-16T00:13:50.290 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.290 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.290 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:50.291 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264656+0000", 2020-01-16T00:13:50.291 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0028688849999999998 2020-01-16T00:13:50.291 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.291 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.291 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:50.291 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.267525+0000", 2020-01-16T00:13:50.291 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00023640900000000001 2020-01-16T00:13:50.291 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.292 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.292 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:50.292 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.267761+0000", 2020-01-16T00:13:50.292 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00035667000000000001 2020-01-16T00:13:50.292 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.292 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.292 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:50.292 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.268118+0000", 2020-01-16T00:13:50.293 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023898040000000001 2020-01-16T00:13:50.293 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.293 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.293 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:50.293 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.270507+0000", 2020-01-16T00:13:50.293 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014919690000000001 2020-01-16T00:13:50.293 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.294 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.294 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:50.294 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.271999+0000", 2020-01-16T00:13:50.294 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0029764510000000002 2020-01-16T00:13:50.294 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.294 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.294 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:50.294 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.274976+0000", 2020-01-16T00:13:50.295 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5409999999999997e-05 2020-01-16T00:13:50.295 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.295 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.295 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:50.295 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.275061+0000", 2020-01-16T00:13:50.295 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2452000000000004e-05 2020-01-16T00:13:50.295 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.295 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.296 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:50.296 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.275134+0000", 2020-01-16T00:13:50.296 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011305709000000001 2020-01-16T00:13:50.296 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.296 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:50.296 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.296 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.296 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.297 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1276 3.d 3:b1521a3e:::smithi12313934-25:head [read 0~3882384,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:13:50.297 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:13:50.297 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 67.193393216999993, 2020-01-16T00:13:50.297 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015839157999999999, 2020-01-16T00:13:50.297 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:50.297 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:50.298 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:50.298 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:50.298 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:50.298 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1276 2020-01-16T00:13:50.298 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.298 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:50.298 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.298 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:50.299 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:13:50.299 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:50.299 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.299 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.299 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:50.299 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:13:50.299 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:50.299 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.300 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.300 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:50.300 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:13:50.300 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5429999999999999e-06 2020-01-16T00:13:50.300 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.300 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.300 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:50.301 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074704+0000", 2020-01-16T00:13:50.301 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0839999999999992e-06 2020-01-16T00:13:50.301 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.301 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.301 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:50.304 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074712+0000", 2020-01-16T00:13:50.304 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.173e-06 2020-01-16T00:13:50.304 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.304 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.304 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:50.305 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074718+0000", 2020-01-16T00:13:50.305 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0083034700000000003 2020-01-16T00:13:50.305 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.305 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.305 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:50.305 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.083022+0000", 2020-01-16T00:13:50.305 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028456400000000001 2020-01-16T00:13:50.305 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.306 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.306 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:50.306 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.083306+0000", 2020-01-16T00:13:50.306 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0072312640000000003 2020-01-16T00:13:50.306 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.306 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.306 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:50.307 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090538+0000", 2020-01-16T00:13:50.307 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015839157999999999 2020-01-16T00:13:50.307 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.307 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:50.307 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.307 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.307 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.307 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1277 3.d 3:b1521a3e:::smithi12313934-25:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:13:50.308 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:13:50.308 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 67.193349627000003, 2020-01-16T00:13:50.308 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016646459999999998, 2020-01-16T00:13:50.308 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:50.308 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:50.308 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:50.308 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:50.309 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:50.309 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1277 2020-01-16T00:13:50.309 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.309 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:50.309 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.309 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:50.309 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:13:50.309 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:50.310 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.310 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.310 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:50.310 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:13:50.310 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:50.310 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.310 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.311 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:50.311 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074741+0000", 2020-01-16T00:13:50.311 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0776e-05 2020-01-16T00:13:50.311 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.311 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.311 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:50.311 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074752+0000", 2020-01-16T00:13:50.311 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7600000000000002e-06 2020-01-16T00:13:50.312 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.312 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.312 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:50.312 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074759+0000", 2020-01-16T00:13:50.312 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6840000000000001e-06 2020-01-16T00:13:50.312 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.312 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.312 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:50.313 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074765+0000", 2020-01-16T00:13:50.313 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015815216 2020-01-16T00:13:50.313 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.313 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.313 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:50.313 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090580+0000", 2020-01-16T00:13:50.313 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000182404 2020-01-16T00:13:50.314 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.314 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.314 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:50.314 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090763+0000", 2020-01-16T00:13:50.314 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00062579300000000003 2020-01-16T00:13:50.314 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.314 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.315 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:50.315 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091389+0000", 2020-01-16T00:13:50.315 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016646459999999998 2020-01-16T00:13:50.315 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.315 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:50.315 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.315 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.316 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.316 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1278 3.d 3:b1521a3e:::smithi12313934-25:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:13:50.316 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:13:50.316 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 67.193319914, 2020-01-16T00:13:50.316 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017994915, 2020-01-16T00:13:50.316 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:50.316 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:50.317 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:50.317 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:50.317 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:50.317 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1278 2020-01-16T00:13:50.317 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.317 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:50.317 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.317 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:50.318 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:13:50.318 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:50.318 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.319 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.319 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:50.319 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:13:50.319 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:50.319 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.319 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.319 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:50.320 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074771+0000", 2020-01-16T00:13:50.320 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4569999999999998e-06 2020-01-16T00:13:50.320 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.320 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.320 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:50.321 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074775+0000", 2020-01-16T00:13:50.321 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0330000000000003e-06 2020-01-16T00:13:50.321 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.321 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.321 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:50.321 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074782+0000", 2020-01-16T00:13:50.321 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9470000000000002e-06 2020-01-16T00:13:50.322 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.322 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.322 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:50.322 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074787+0000", 2020-01-16T00:13:50.322 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01666134 2020-01-16T00:13:50.322 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.323 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.323 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:50.323 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091449+0000", 2020-01-16T00:13:50.323 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000247731 2020-01-16T00:13:50.323 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.323 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.324 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:50.324 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091696+0000", 2020-01-16T00:13:50.324 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010703170000000001 2020-01-16T00:13:50.324 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.324 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.324 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:50.324 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.092767+0000", 2020-01-16T00:13:50.325 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017994915 2020-01-16T00:13:50.325 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.325 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:50.325 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.326 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.326 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.326 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2260 3.d 3:b76659c4:::smithi12313934-47:head [write 601750~535546] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:13:50.326 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:06.050577+0000", 2020-01-16T00:13:50.326 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 44.217514598000001, 2020-01-16T00:13:50.326 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012415381, 2020-01-16T00:13:50.326 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:50.327 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:50.327 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:50.327 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:50.327 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:50.327 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2260 2020-01-16T00:13:50.327 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.327 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:50.328 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.328 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:50.328 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.050577+0000", 2020-01-16T00:13:50.328 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:50.328 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.328 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.328 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:50.328 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.050577+0000", 2020-01-16T00:13:50.329 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:50.329 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.329 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.329 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:50.329 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.050574+0000", 2020-01-16T00:13:50.329 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001523487 2020-01-16T00:13:50.329 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.330 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.330 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:50.330 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052097+0000", 2020-01-16T00:13:50.330 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3697e-05 2020-01-16T00:13:50.330 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.330 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.330 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:50.330 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052121+0000", 2020-01-16T00:13:50.331 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0070000000000003e-06 2020-01-16T00:13:50.331 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.331 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.331 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:50.331 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052130+0000", 2020-01-16T00:13:50.331 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4559000000000003e-05 2020-01-16T00:13:50.331 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.332 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.332 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:50.332 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052194+0000", 2020-01-16T00:13:50.332 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000165878 2020-01-16T00:13:50.332 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.332 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.332 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:50.333 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052360+0000", 2020-01-16T00:13:50.333 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018825499999999999 2020-01-16T00:13:50.333 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.333 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.333 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:50.333 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052549+0000", 2020-01-16T00:13:50.333 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002761782 2020-01-16T00:13:50.334 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.334 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.334 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:50.334 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.055310+0000", 2020-01-16T00:13:50.334 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00060385500000000004 2020-01-16T00:13:50.334 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.334 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.334 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:50.335 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.055914+0000", 2020-01-16T00:13:50.335 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0069733809999999999 2020-01-16T00:13:50.335 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.335 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.335 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:50.335 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.062888+0000", 2020-01-16T00:13:50.335 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2884999999999999e-05 2020-01-16T00:13:50.336 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.336 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.336 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:50.336 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.062940+0000", 2020-01-16T00:13:50.336 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1996999999999997e-05 2020-01-16T00:13:50.336 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.337 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.337 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:50.337 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.062992+0000", 2020-01-16T00:13:50.337 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012415381 2020-01-16T00:13:50.337 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.337 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:50.337 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.338 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.338 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.338 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2295 3.d 3:b1521a3e:::smithi12313934-25:head [write 1763838~505140] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:13:50.338 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:06.660258+0000", 2020-01-16T00:13:50.338 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 43.607833370999998, 2020-01-16T00:13:50.338 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011681408000000001, 2020-01-16T00:13:50.338 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:50.339 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:50.339 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:50.339 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:50.339 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:50.339 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2295 2020-01-16T00:13:50.339 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.339 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:50.339 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.340 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:50.340 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660258+0000", 2020-01-16T00:13:50.340 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:50.340 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.340 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.340 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:50.341 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660258+0000", 2020-01-16T00:13:50.341 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:50.341 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.341 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.341 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:50.341 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660256+0000", 2020-01-16T00:13:50.342 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00066501699999999995 2020-01-16T00:13:50.342 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.342 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.342 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:50.342 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660921+0000", 2020-01-16T00:13:50.343 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7339000000000001e-05 2020-01-16T00:13:50.343 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.343 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.343 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:50.343 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660938+0000", 2020-01-16T00:13:50.343 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0219999999999993e-06 2020-01-16T00:13:50.343 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.344 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.344 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:50.344 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660947+0000", 2020-01-16T00:13:50.344 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00094237200000000002 2020-01-16T00:13:50.344 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.344 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.344 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:50.345 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.661890+0000", 2020-01-16T00:13:50.345 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000117649 2020-01-16T00:13:50.345 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.345 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.345 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:50.345 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662007+0000", 2020-01-16T00:13:50.345 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00060662599999999998 2020-01-16T00:13:50.346 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.346 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.346 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:50.346 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662614+0000", 2020-01-16T00:13:50.346 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028424899999999998 2020-01-16T00:13:50.346 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.346 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.346 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:50.347 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662898+0000", 2020-01-16T00:13:50.347 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00054904599999999997 2020-01-16T00:13:50.347 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.347 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.347 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:50.347 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.663447+0000", 2020-01-16T00:13:50.347 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0051479990000000003 2020-01-16T00:13:50.348 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.348 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.348 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:50.348 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.668595+0000", 2020-01-16T00:13:50.348 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00080881799999999999 2020-01-16T00:13:50.348 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.348 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.349 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:50.349 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.669404+0000", 2020-01-16T00:13:50.349 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0024495290000000002 2020-01-16T00:13:50.349 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.349 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.349 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:50.349 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671854+0000", 2020-01-16T00:13:50.350 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6635000000000003e-05 2020-01-16T00:13:50.350 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.350 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.350 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:50.350 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671900+0000", 2020-01-16T00:13:50.350 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9567999999999998e-05 2020-01-16T00:13:50.350 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.350 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.351 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:50.351 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671940+0000", 2020-01-16T00:13:50.351 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011681408000000001 2020-01-16T00:13:50.351 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.351 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:50.351 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.351 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.351 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.352 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2297 3.d 3:b1521a3e:::smithi12313934-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e35)", 2020-01-16T00:13:50.352 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:06.660986+0000", 2020-01-16T00:13:50.352 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 43.607105513, 2020-01-16T00:13:50.352 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011220677, 2020-01-16T00:13:50.352 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:50.352 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:50.352 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:50.353 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:50.353 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:50.353 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2297 2020-01-16T00:13:50.353 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.353 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:50.353 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.353 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:50.353 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660986+0000", 2020-01-16T00:13:50.354 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:50.354 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.354 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.354 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:50.354 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660986+0000", 2020-01-16T00:13:50.354 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:50.354 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.355 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.355 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:50.355 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660985+0000", 2020-01-16T00:13:50.355 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8760000000000004e-06 2020-01-16T00:13:50.355 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.355 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.355 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:50.355 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660990+0000", 2020-01-16T00:13:50.356 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.4190000000000002e-06 2020-01-16T00:13:50.356 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.356 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.356 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:50.356 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660999+0000", 2020-01-16T00:13:50.356 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7740000000000001e-06 2020-01-16T00:13:50.356 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.356 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.357 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:50.357 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.661004+0000", 2020-01-16T00:13:50.357 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0073783590000000001 2020-01-16T00:13:50.358 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.358 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.358 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:50.358 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.668383+0000", 2020-01-16T00:13:50.358 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014443 2020-01-16T00:13:50.358 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.358 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.358 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:50.359 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.668527+0000", 2020-01-16T00:13:50.359 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0035336539999999998 2020-01-16T00:13:50.359 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.359 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.359 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:50.359 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.672061+0000", 2020-01-16T00:13:50.359 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6445000000000005e-05 2020-01-16T00:13:50.360 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.360 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.360 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:50.360 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.672127+0000", 2020-01-16T00:13:50.360 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.9497000000000001e-05 2020-01-16T00:13:50.360 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.361 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.361 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:50.361 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.672207+0000", 2020-01-16T00:13:50.361 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011220677 2020-01-16T00:13:50.361 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.361 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:50.361 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.362 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.362 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.362 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.3.13:109 2.0 2:11fa79bc:::smithi12313934-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 331264~198656] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e42)", 2020-01-16T00:13:50.362 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:21.840075+0000", 2020-01-16T00:13:50.362 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 28.428017074, 2020-01-16T00:13:50.362 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011558677999999999, 2020-01-16T00:13:50.362 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:50.363 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:50.363 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:50.363 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.363 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:50.363 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840075+0000", 2020-01-16T00:13:50.363 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:50.363 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.363 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.364 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:50.364 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840075+0000", 2020-01-16T00:13:50.364 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:50.364 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.364 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.364 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:50.364 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840073+0000", 2020-01-16T00:13:50.364 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033576299999999999 2020-01-16T00:13:50.365 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.365 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.365 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:50.365 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840409+0000", 2020-01-16T00:13:50.365 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2082999999999998e-05 2020-01-16T00:13:50.365 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.365 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.366 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:50.366 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840431+0000", 2020-01-16T00:13:50.366 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4410000000000003e-06 2020-01-16T00:13:50.366 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.366 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.366 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:50.366 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840436+0000", 2020-01-16T00:13:50.366 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000101016 2020-01-16T00:13:50.367 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.367 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.367 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:50.367 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840537+0000", 2020-01-16T00:13:50.367 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000117639 2020-01-16T00:13:50.367 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.367 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.368 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:50.368 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840654+0000", 2020-01-16T00:13:50.368 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000196162 2020-01-16T00:13:50.368 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.368 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.368 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:13:50.368 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840850+0000", 2020-01-16T00:13:50.368 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003319092 2020-01-16T00:13:50.369 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.369 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.369 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:50.369 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.844169+0000", 2020-01-16T00:13:50.369 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011349229999999999 2020-01-16T00:13:50.369 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.369 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.369 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:50.370 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.845304+0000", 2020-01-16T00:13:50.370 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0062517670000000001 2020-01-16T00:13:50.370 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.370 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.370 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:50.370 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851556+0000", 2020-01-16T00:13:50.370 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4138000000000003e-05 2020-01-16T00:13:50.371 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.371 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.371 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:50.371 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851600+0000", 2020-01-16T00:13:50.371 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2993000000000001e-05 2020-01-16T00:13:50.371 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.371 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.371 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:50.372 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851633+0000", 2020-01-16T00:13:50.372 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011558677999999999 2020-01-16T00:13:50.372 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.372 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:50.372 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.372 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.372 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.373 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.3.13:110 2.0 2:11fa79bc:::smithi12313934-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 752640~650240] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e42)", 2020-01-16T00:13:50.373 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:21.840457+0000", 2020-01-16T00:13:50.373 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 28.427634781999998, 2020-01-16T00:13:50.373 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011329271, 2020-01-16T00:13:50.373 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:50.373 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:50.373 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:50.373 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.374 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:50.374 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840457+0000", 2020-01-16T00:13:50.374 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:50.374 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.374 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.374 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:50.374 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840457+0000", 2020-01-16T00:13:50.375 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:50.375 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.375 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.375 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:50.375 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840456+0000", 2020-01-16T00:13:50.375 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00062607899999999998 2020-01-16T00:13:50.376 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.376 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.376 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:50.376 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841082+0000", 2020-01-16T00:13:50.376 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1239999999999999e-06 2020-01-16T00:13:50.376 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.376 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.377 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:50.377 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841090+0000", 2020-01-16T00:13:50.377 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7129999999999999e-06 2020-01-16T00:13:50.377 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.377 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.377 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:50.378 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841093+0000", 2020-01-16T00:13:50.378 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00075997800000000002 2020-01-16T00:13:50.378 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.378 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.378 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:50.379 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841853+0000", 2020-01-16T00:13:50.379 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0690000000000003e-05 2020-01-16T00:13:50.379 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.379 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.379 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:50.380 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841924+0000", 2020-01-16T00:13:50.380 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000115753 2020-01-16T00:13:50.380 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.380 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.380 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:13:50.380 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.842040+0000", 2020-01-16T00:13:50.380 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0021358890000000002 2020-01-16T00:13:50.381 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.381 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.381 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:50.381 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.844176+0000", 2020-01-16T00:13:50.381 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0042541870000000004 2020-01-16T00:13:50.381 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.382 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.382 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:50.382 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.848430+0000", 2020-01-16T00:13:50.382 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0032476929999999998 2020-01-16T00:13:50.382 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.382 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.383 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:50.383 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851677+0000", 2020-01-16T00:13:50.383 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0370000000000001e-05 2020-01-16T00:13:50.383 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.383 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.383 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:50.383 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851728+0000", 2020-01-16T00:13:50.383 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.834e-05 2020-01-16T00:13:50.384 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.384 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.384 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:50.384 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851786+0000", 2020-01-16T00:13:50.384 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011329271 2020-01-16T00:13:50.384 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.384 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:50.385 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.385 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.385 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.385 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2987 3.d 3:b3dd5d45:::smithi12313934-16:head [write 467327~633480] snapc 0=[] ondisk+write+known_if_redirected e42)", 2020-01-16T00:13:50.385 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:22.257904+0000", 2020-01-16T00:13:50.385 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 28.010187488, 2020-01-16T00:13:50.385 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012187450000000001, 2020-01-16T00:13:50.385 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:50.386 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:50.386 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:50.386 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:50.386 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:50.386 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2987 2020-01-16T00:13:50.386 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.386 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:50.386 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.387 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:50.387 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.257904+0000", 2020-01-16T00:13:50.387 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:50.387 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.387 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.387 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:50.387 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.257904+0000", 2020-01-16T00:13:50.388 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999933 2020-01-16T00:13:50.388 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.388 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.388 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:50.388 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.257898+0000", 2020-01-16T00:13:50.388 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00097099800000000004 2020-01-16T00:13:50.388 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.388 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.389 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:50.389 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.258869+0000", 2020-01-16T00:13:50.389 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7970000000000003e-05 2020-01-16T00:13:50.389 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.389 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.389 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:50.389 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.258917+0000", 2020-01-16T00:13:50.389 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5982999999999999e-05 2020-01-16T00:13:50.390 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.390 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.390 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:50.390 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.258933+0000", 2020-01-16T00:13:50.390 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00026239999999999998 2020-01-16T00:13:50.390 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.390 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.391 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:50.391 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.259195+0000", 2020-01-16T00:13:50.391 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024907400000000001 2020-01-16T00:13:50.391 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.391 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.391 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:50.391 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.259444+0000", 2020-01-16T00:13:50.391 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00034824999999999998 2020-01-16T00:13:50.392 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.392 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.392 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:50.392 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.259792+0000", 2020-01-16T00:13:50.392 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003251538 2020-01-16T00:13:50.392 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.392 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.392 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:50.393 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.263044+0000", 2020-01-16T00:13:50.393 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0068502880000000004 2020-01-16T00:13:50.393 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.393 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.393 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:50.393 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.269894+0000", 2020-01-16T00:13:50.393 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0945000000000006e-05 2020-01-16T00:13:50.394 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.394 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.394 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:50.394 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.269965+0000", 2020-01-16T00:13:50.394 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012667399999999999 2020-01-16T00:13:50.394 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.394 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.395 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:50.395 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.270092+0000", 2020-01-16T00:13:50.395 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012187450000000001 2020-01-16T00:13:50.395 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.395 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:50.395 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.395 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.395 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.396 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3273 3.d 3:b76659c4:::smithi12313934-47:head [read 0~3647152,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:13:50.396 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.728775+0000", 2020-01-16T00:13:50.396 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 19.539316933999999, 2020-01-16T00:13:50.396 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012922950000000001, 2020-01-16T00:13:50.396 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:50.396 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:50.396 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:50.397 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:50.397 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:50.397 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3273 2020-01-16T00:13:50.397 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.397 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:50.397 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.397 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:50.397 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728775+0000", 2020-01-16T00:13:50.398 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:50.398 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.398 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.398 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:50.398 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728775+0000", 2020-01-16T00:13:50.398 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:50.398 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.399 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.399 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:50.399 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728771+0000", 2020-01-16T00:13:50.399 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.5009000000000001e-05 2020-01-16T00:13:50.399 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.399 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.399 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:50.399 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728796+0000", 2020-01-16T00:13:50.400 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2357999999999998e-05 2020-01-16T00:13:50.400 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.400 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.400 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:50.400 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728819+0000", 2020-01-16T00:13:50.400 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.4800000000000007e-06 2020-01-16T00:13:50.400 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.401 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.401 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:50.401 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728828+0000", 2020-01-16T00:13:50.401 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000124824 2020-01-16T00:13:50.401 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.401 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.401 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:50.401 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728953+0000", 2020-01-16T00:13:50.402 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00044904899999999998 2020-01-16T00:13:50.402 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.402 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.402 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:50.402 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729402+0000", 2020-01-16T00:13:50.402 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012295706999999999 2020-01-16T00:13:50.402 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.403 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.403 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:50.403 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.741698+0000", 2020-01-16T00:13:50.403 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012922950000000001 2020-01-16T00:13:50.403 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.403 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:50.404 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.404 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.404 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.404 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3275 3.d 3:b76659c4:::smithi12313934-47:head [read 0~3647152,checksum] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:13:50.404 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.728933+0000", 2020-01-16T00:13:50.404 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 19.539158661999998, 2020-01-16T00:13:50.404 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01204874, 2020-01-16T00:13:50.405 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:50.405 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:50.405 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:50.405 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:50.405 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:50.405 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3275 2020-01-16T00:13:50.405 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.406 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:50.406 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.406 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:50.406 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728933+0000", 2020-01-16T00:13:50.406 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:50.406 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.406 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.407 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:50.407 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728933+0000", 2020-01-16T00:13:50.407 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:50.407 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.407 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.407 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:50.407 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728932+0000", 2020-01-16T00:13:50.408 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3569999999999998e-06 2020-01-16T00:13:50.408 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.408 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.408 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:50.408 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728940+0000", 2020-01-16T00:13:50.408 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0457e-05 2020-01-16T00:13:50.408 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.408 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.409 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:50.409 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728951+0000", 2020-01-16T00:13:50.409 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6799999999999993e-06 2020-01-16T00:13:50.409 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.409 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.409 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:50.409 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728959+0000", 2020-01-16T00:13:50.409 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010076307 2020-01-16T00:13:50.410 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.410 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.410 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:50.410 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.739035+0000", 2020-01-16T00:13:50.410 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024455400000000002 2020-01-16T00:13:50.410 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.410 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.411 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:50.411 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.739279+0000", 2020-01-16T00:13:50.411 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0017023660000000001 2020-01-16T00:13:50.411 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.411 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.411 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:50.411 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.740982+0000", 2020-01-16T00:13:50.411 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01204874 2020-01-16T00:13:50.412 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.412 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:50.412 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.412 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.412 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.412 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3285 3.d 3:bd9fc72f:::smithi12313934-5:head [read 0~3764801,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:13:50.412 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.729068+0000", 2020-01-16T00:13:50.412 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 19.539023870000001, 2020-01-16T00:13:50.413 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018537010999999999, 2020-01-16T00:13:50.413 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:50.413 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:50.413 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:50.413 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:50.413 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:50.413 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3285 2020-01-16T00:13:50.414 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.414 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:50.414 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.414 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:50.414 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729068+0000", 2020-01-16T00:13:50.414 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:50.414 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.414 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.415 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:50.415 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729068+0000", 2020-01-16T00:13:50.415 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:50.415 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.415 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.415 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:50.415 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729067+0000", 2020-01-16T00:13:50.415 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3339999999999996e-06 2020-01-16T00:13:50.416 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.416 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.416 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:50.416 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729075+0000", 2020-01-16T00:13:50.416 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4187e-05 2020-01-16T00:13:50.416 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.416 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.416 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:50.417 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729089+0000", 2020-01-16T00:13:50.417 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5600000000000004e-06 2020-01-16T00:13:50.417 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.417 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.417 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:50.418 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729093+0000", 2020-01-16T00:13:50.418 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011944309 2020-01-16T00:13:50.418 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.419 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.419 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:50.419 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.741038+0000", 2020-01-16T00:13:50.419 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016625799999999999 2020-01-16T00:13:50.419 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.419 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.419 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:50.420 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.741204+0000", 2020-01-16T00:13:50.420 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0064010120000000002 2020-01-16T00:13:50.420 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.420 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.420 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:50.420 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.747605+0000", 2020-01-16T00:13:50.420 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018537010999999999 2020-01-16T00:13:50.420 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.421 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:50.421 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.421 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.421 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.421 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3286 3.d 3:bd9fc72f:::smithi12313934-5:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:13:50.421 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.729099+0000", 2020-01-16T00:13:50.421 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 19.5389932, 2020-01-16T00:13:50.422 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018983897999999999, 2020-01-16T00:13:50.422 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:50.422 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:50.422 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:50.422 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:50.422 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:50.422 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3286 2020-01-16T00:13:50.422 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.423 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:50.423 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.423 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:50.423 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729099+0000", 2020-01-16T00:13:50.423 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:50.423 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.423 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.423 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:50.424 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729099+0000", 2020-01-16T00:13:50.424 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:50.424 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.424 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.424 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:50.424 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729098+0000", 2020-01-16T00:13:50.424 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8500000000000004e-06 2020-01-16T00:13:50.425 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.425 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.425 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:50.425 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729102+0000", 2020-01-16T00:13:50.425 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7329999999999997e-06 2020-01-16T00:13:50.425 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.425 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.425 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:50.426 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729108+0000", 2020-01-16T00:13:50.426 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.764e-06 2020-01-16T00:13:50.426 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.426 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.426 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:50.426 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729118+0000", 2020-01-16T00:13:50.426 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018524286000000001 2020-01-16T00:13:50.427 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.427 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.427 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:50.427 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.747642+0000", 2020-01-16T00:13:50.427 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9487999999999993e-05 2020-01-16T00:13:50.427 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.427 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.427 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:50.428 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.747742+0000", 2020-01-16T00:13:50.428 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000340429 2020-01-16T00:13:50.428 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.428 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.428 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:50.428 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.748082+0000", 2020-01-16T00:13:50.428 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018983897999999999 2020-01-16T00:13:50.429 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.429 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:50.429 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.429 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.429 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.429 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3287 3.d 3:bd9fc72f:::smithi12313934-5:head [read 0~3764801,checksum] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:13:50.429 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.729124+0000", 2020-01-16T00:13:50.429 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 19.538968060999998, 2020-01-16T00:13:50.430 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020552323000000001, 2020-01-16T00:13:50.430 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:50.430 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:50.430 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:50.430 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:50.430 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:50.430 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3287 2020-01-16T00:13:50.431 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.431 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:50.431 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.431 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:50.431 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729124+0000", 2020-01-16T00:13:50.431 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:50.431 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.431 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.432 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:50.432 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729124+0000", 2020-01-16T00:13:50.432 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:50.432 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.432 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.432 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:50.432 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729123+0000", 2020-01-16T00:13:50.433 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8239999999999996e-06 2020-01-16T00:13:50.433 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.433 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.433 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:50.433 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729129+0000", 2020-01-16T00:13:50.433 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6909999999999997e-06 2020-01-16T00:13:50.433 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.433 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.434 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:50.434 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729136+0000", 2020-01-16T00:13:50.434 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3329999999999997e-06 2020-01-16T00:13:50.434 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.434 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.434 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:50.434 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729145+0000", 2020-01-16T00:13:50.435 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018965065999999999 2020-01-16T00:13:50.435 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.435 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.435 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:50.435 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.748110+0000", 2020-01-16T00:13:50.435 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015781700000000001 2020-01-16T00:13:50.435 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.435 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.436 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:50.436 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.748268+0000", 2020-01-16T00:13:50.436 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014083800000000001 2020-01-16T00:13:50.436 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.436 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.436 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:50.436 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.749676+0000", 2020-01-16T00:13:50.436 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020552323000000001 2020-01-16T00:13:50.437 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.437 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:50.437 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.437 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.437 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.437 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3409 3.d 3:b3dd5d45:::smithi12313934-16:head [write 591712~786646,stat] snapc 0=[] ondisk+write+known_if_redirected e47)", 2020-01-16T00:13:50.437 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:34.100681+0000", 2020-01-16T00:13:50.438 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 16.167410920999998, 2020-01-16T00:13:50.438 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012248446, 2020-01-16T00:13:50.438 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:50.438 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:50.438 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:50.438 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:50.438 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:50.438 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3409 2020-01-16T00:13:50.439 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.439 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:50.439 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.439 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:50.439 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.100681+0000", 2020-01-16T00:13:50.439 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:50.439 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.439 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.440 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:50.440 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.100681+0000", 2020-01-16T00:13:50.440 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:13:50.440 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.440 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.440 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:50.440 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.100677+0000", 2020-01-16T00:13:50.441 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000998828 2020-01-16T00:13:50.441 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.441 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.441 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:50.441 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101676+0000", 2020-01-16T00:13:50.441 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5846999999999997e-05 2020-01-16T00:13:50.441 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.442 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.442 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:50.442 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101712+0000", 2020-01-16T00:13:50.442 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.364e-05 2020-01-16T00:13:50.442 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.442 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.443 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:50.443 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101725+0000", 2020-01-16T00:13:50.443 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015902700000000001 2020-01-16T00:13:50.443 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.443 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.443 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:50.443 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101884+0000", 2020-01-16T00:13:50.443 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0486e-05 2020-01-16T00:13:50.444 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.444 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.444 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op must wait for map", 2020-01-16T00:13:50.444 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101905+0000", 2020-01-16T00:13:50.444 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0038869009999999999 2020-01-16T00:13:50.444 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.444 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.445 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:50.445 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.105792+0000", 2020-01-16T00:13:50.445 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030748800000000002 2020-01-16T00:13:50.445 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.445 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.445 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:50.445 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.106099+0000", 2020-01-16T00:13:50.445 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042069299999999998 2020-01-16T00:13:50.446 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.446 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.446 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:50.446 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.106520+0000", 2020-01-16T00:13:50.446 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0052864560000000001 2020-01-16T00:13:50.446 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.446 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.446 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:50.447 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.111806+0000", 2020-01-16T00:13:50.447 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00080835800000000001 2020-01-16T00:13:50.447 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.447 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.447 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:50.447 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.112615+0000", 2020-01-16T00:13:50.447 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1490999999999997e-05 2020-01-16T00:13:50.448 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.448 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.448 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:50.448 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.112686+0000", 2020-01-16T00:13:50.448 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000243174 2020-01-16T00:13:50.448 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.448 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.448 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:50.449 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.112929+0000", 2020-01-16T00:13:50.449 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012248446 2020-01-16T00:13:50.449 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.449 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:50.449 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.449 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.449 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.450 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.0.13:31 2.0 2:1afbc8ac:::smithi12313934-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 4096~737792] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e50)", 2020-01-16T00:13:50.450 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:44.061109+0000", 2020-01-16T00:13:50.450 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 6.2069826969999999, 2020-01-16T00:13:50.450 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012186723, 2020-01-16T00:13:50.450 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:50.450 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:50.450 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:50.450 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.451 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:50.451 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.061109+0000", 2020-01-16T00:13:50.451 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:50.451 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.451 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.451 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:50.451 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.061109+0000", 2020-01-16T00:13:50.451 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:50.452 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.452 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.452 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:50.452 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.061106+0000", 2020-01-16T00:13:50.452 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012349870000000001 2020-01-16T00:13:50.452 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.452 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.453 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:50.453 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062341+0000", 2020-01-16T00:13:50.453 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0235999999999996e-05 2020-01-16T00:13:50.453 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.453 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.453 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:50.453 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062411+0000", 2020-01-16T00:13:50.453 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7078000000000001e-05 2020-01-16T00:13:50.454 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.454 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.454 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:50.454 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062428+0000", 2020-01-16T00:13:50.454 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2158999999999996e-05 2020-01-16T00:13:50.454 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.454 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.455 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:50.455 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062520+0000", 2020-01-16T00:13:50.455 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011203499999999999 2020-01-16T00:13:50.455 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.455 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.455 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:50.455 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062632+0000", 2020-01-16T00:13:50.455 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017238400000000001 2020-01-16T00:13:50.456 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.456 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.456 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:13:50.456 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062805+0000", 2020-01-16T00:13:50.456 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010264989 2020-01-16T00:13:50.456 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.456 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.456 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:50.457 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073070+0000", 2020-01-16T00:13:50.457 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0533000000000001e-05 2020-01-16T00:13:50.457 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.457 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.457 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:50.457 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073090+0000", 2020-01-16T00:13:50.457 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2309000000000002e-05 2020-01-16T00:13:50.458 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.458 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.458 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:50.458 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073163+0000", 2020-01-16T00:13:50.458 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3035999999999997e-05 2020-01-16T00:13:50.458 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.458 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.458 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:50.459 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073236+0000", 2020-01-16T00:13:50.459 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0056999999999998e-05 2020-01-16T00:13:50.459 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.459 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.459 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:50.459 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073296+0000", 2020-01-16T00:13:50.459 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012186723 2020-01-16T00:13:50.460 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.460 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:50.460 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.460 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.460 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.460 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.0.13:32 2.0 2:1afbc8ac:::smithi12313934-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 916992~596480] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e50)", 2020-01-16T00:13:50.460 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:44.062464+0000", 2020-01-16T00:13:50.461 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 6.2056279119999997, 2020-01-16T00:13:50.461 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011316405999999999, 2020-01-16T00:13:50.461 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:50.461 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:50.461 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:50.461 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.462 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:50.462 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062464+0000", 2020-01-16T00:13:50.462 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:50.462 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.462 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.462 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:50.462 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062464+0000", 2020-01-16T00:13:50.463 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:50.463 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.463 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.463 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:50.463 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062462+0000", 2020-01-16T00:13:50.463 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00073330900000000002 2020-01-16T00:13:50.463 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.463 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.464 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:50.464 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063196+0000", 2020-01-16T00:13:50.464 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5330000000000001e-05 2020-01-16T00:13:50.464 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.464 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.464 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:50.464 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063211+0000", 2020-01-16T00:13:50.464 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5840000000000001e-06 2020-01-16T00:13:50.465 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.465 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.465 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:50.465 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063221+0000", 2020-01-16T00:13:50.465 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001786915 2020-01-16T00:13:50.465 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.465 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.466 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:50.466 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.065007+0000", 2020-01-16T00:13:50.466 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00021770000000000001 2020-01-16T00:13:50.466 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.466 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.466 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:50.466 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.065225+0000", 2020-01-16T00:13:50.466 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00032623 2020-01-16T00:13:50.467 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.467 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.467 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:13:50.467 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.065551+0000", 2020-01-16T00:13:50.467 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0078227499999999998 2020-01-16T00:13:50.467 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.467 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.468 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:50.468 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073374+0000", 2020-01-16T00:13:50.468 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.717e-05 2020-01-16T00:13:50.468 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.468 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.468 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:50.468 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073391+0000", 2020-01-16T00:13:50.469 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011493999999999999 2020-01-16T00:13:50.469 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.469 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.469 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:50.469 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073506+0000", 2020-01-16T00:13:50.469 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000159281 2020-01-16T00:13:50.469 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.469 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.470 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:50.470 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073666+0000", 2020-01-16T00:13:50.470 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000114675 2020-01-16T00:13:50.470 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.470 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.470 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:50.470 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073780+0000", 2020-01-16T00:13:50.471 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011316405999999999 2020-01-16T00:13:50.471 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.471 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:50.471 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.471 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.471 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.471 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.0.13:33 2.0 2:1afbc8ac:::smithi12313934-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 1541120~378368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e50)", 2020-01-16T00:13:50.471 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:44.063236+0000", 2020-01-16T00:13:50.472 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 6.2048560330000004, 2020-01-16T00:13:50.472 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015290962999999999, 2020-01-16T00:13:50.472 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:50.472 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:50.472 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:50.472 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.472 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:50.472 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063236+0000", 2020-01-16T00:13:50.473 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:50.473 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.473 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.473 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:50.473 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063236+0000", 2020-01-16T00:13:50.473 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:50.473 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.474 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.474 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:50.474 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063234+0000", 2020-01-16T00:13:50.474 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000500702 2020-01-16T00:13:50.474 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.474 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.474 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:50.475 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063735+0000", 2020-01-16T00:13:50.475 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4734e-05 2020-01-16T00:13:50.475 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.475 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.475 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:50.475 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063750+0000", 2020-01-16T00:13:50.475 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.382e-05 2020-01-16T00:13:50.475 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.476 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.476 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:50.476 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063764+0000", 2020-01-16T00:13:50.476 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0059030799999999998 2020-01-16T00:13:50.476 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.476 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.476 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:50.477 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.069667+0000", 2020-01-16T00:13:50.477 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3961999999999994e-05 2020-01-16T00:13:50.477 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.477 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.477 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:50.477 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.069741+0000", 2020-01-16T00:13:50.477 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000114931 2020-01-16T00:13:50.477 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.478 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.478 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:13:50.478 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.069856+0000", 2020-01-16T00:13:50.478 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0039948270000000003 2020-01-16T00:13:50.478 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.478 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.478 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:50.479 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073850+0000", 2020-01-16T00:13:50.479 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0016459140000000001 2020-01-16T00:13:50.479 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.479 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.479 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:50.479 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.075496+0000", 2020-01-16T00:13:50.479 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002957997 2020-01-16T00:13:50.479 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.480 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.480 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:50.480 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.078454+0000", 2020-01-16T00:13:50.480 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9190999999999998e-05 2020-01-16T00:13:50.480 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.480 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.480 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:50.481 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.078493+0000", 2020-01-16T00:13:50.481 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3176999999999998e-05 2020-01-16T00:13:50.481 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:50.481 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:50.481 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:50.481 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.078527+0000", 2020-01-16T00:13:50.481 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015290962999999999 2020-01-16T00:13:50.481 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.482 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:50.482 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.482 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:50.482 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:50.482 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:50.486 INFO:tasks.rados.rados.0.smithi123.stdout:1494: finishing write tid 1 to smithi12313934-25 2020-01-16T00:13:50.486 INFO:tasks.rados.rados.0.smithi123.stdout:1494: finishing write tid 2 to smithi12313934-25 2020-01-16T00:13:50.486 INFO:tasks.rados.rados.0.smithi123.stdout:1494: finishing write tid 4 to smithi12313934-25 2020-01-16T00:13:50.486 INFO:tasks.rados.rados.0.smithi123.stdout:1494: finishing write tid 5 to smithi12313934-25 2020-01-16T00:13:50.486 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 25 v 225 (ObjNum 500 snap 0 seq_num 500) dirty exists 2020-01-16T00:13:50.487 INFO:tasks.rados.rados.0.smithi123.stdout:1494: left oid 25 (ObjNum 500 snap 0 seq_num 500) 2020-01-16T00:13:50.487 INFO:tasks.rados.rados.0.smithi123.stdout:1495: finishing write tid 1 to smithi12313934-3 2020-01-16T00:13:50.487 INFO:tasks.rados.rados.0.smithi123.stdout:1495: finishing write tid 2 to smithi12313934-3 2020-01-16T00:13:50.487 INFO:tasks.rados.rados.0.smithi123.stdout:1495: finishing write tid 4 to smithi12313934-3 2020-01-16T00:13:50.487 INFO:tasks.rados.rados.0.smithi123.stdout:1495: finishing write tid 5 to smithi12313934-3 2020-01-16T00:13:50.487 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 3 v 106 (ObjNum 501 snap 0 seq_num 501) dirty exists 2020-01-16T00:13:50.487 INFO:tasks.rados.rados.0.smithi123.stdout:1495: left oid 3 (ObjNum 501 snap 0 seq_num 501) 2020-01-16T00:13:50.487 INFO:tasks.rados.rados.0.smithi123.stdout:1496: finishing write tid 1 to smithi12313934-37 2020-01-16T00:13:50.488 INFO:tasks.rados.rados.0.smithi123.stdout:1496: finishing write tid 2 to smithi12313934-37 2020-01-16T00:13:50.488 INFO:tasks.rados.rados.0.smithi123.stdout:1491: done (15 left) 2020-01-16T00:13:50.488 INFO:tasks.rados.rados.0.smithi123.stdout:1492: done (14 left) 2020-01-16T00:13:50.488 INFO:tasks.rados.rados.0.smithi123.stdout:1493: done (13 left) 2020-01-16T00:13:50.488 INFO:tasks.rados.rados.0.smithi123.stdout:1494: done (12 left) 2020-01-16T00:13:50.488 INFO:tasks.rados.rados.0.smithi123.stdout:1495: done (11 left) 2020-01-16T00:13:50.488 INFO:tasks.rados.rados.0.smithi123.stdout:1507: write oid 2 current snap is 0 2020-01-16T00:13:50.489 INFO:tasks.rados.rados.0.smithi123.stdout:1507: seq_num 511 ranges {588377=647890,1935828=635840} 2020-01-16T00:13:50.489 INFO:tasks.rados.rados.0.smithi123.stdout:1507: writing smithi12313934-2 from 588377 to 1236267 tid 1 2020-01-16T00:13:50.489 INFO:tasks.rados.rados.0.smithi123.stdout:1507: writing smithi12313934-2 from 1935828 to 2571668 tid 2 2020-01-16T00:13:50.489 INFO:tasks.rados.rados.0.smithi123.stdout:1508: read oid 44 snap -1 2020-01-16T00:13:50.489 INFO:tasks.rados.rados.0.smithi123.stdout:1508: expect (ObjNum 461 snap 0 seq_num 461) 2020-01-16T00:13:50.489 INFO:tasks.rados.rados.0.smithi123.stdout:1509: read oid 33 snap -1 2020-01-16T00:13:50.489 INFO:tasks.rados.rados.0.smithi123.stdout:1509: expect (ObjNum 330 snap 0 seq_num 330) 2020-01-16T00:13:50.490 INFO:tasks.rados.rados.0.smithi123.stdout:1510: write (excl) oid 30 current snap is 0 2020-01-16T00:13:50.490 INFO:tasks.rados.rados.0.smithi123.stdout:1510: seq_num 512 ranges {605184=723139,1877957=727227} 2020-01-16T00:13:50.490 INFO:tasks.rados.rados.0.smithi123.stdout:1510: writing smithi12313934-30 from 605184 to 1328323 tid 1 2020-01-16T00:13:50.490 INFO:tasks.rados.rados.0.smithi123.stdout:1510: writing smithi12313934-30 from 1877957 to 2605184 tid 2 2020-01-16T00:13:50.490 INFO:tasks.rados.rados.0.smithi123.stdout:1511: write oid 32 current snap is 0 2020-01-16T00:13:50.490 INFO:tasks.rados.rados.0.smithi123.stdout:1511: seq_num 513 ranges {621991=798388,1936439=639509} 2020-01-16T00:13:50.490 INFO:tasks.rados.rados.0.smithi123.stdout:1511: writing smithi12313934-32 from 621991 to 1420379 tid 1 2020-01-16T00:13:50.491 INFO:tasks.rados.rados.0.smithi123.stdout:1511: writing smithi12313934-32 from 1936439 to 2575948 tid 2 2020-01-16T00:13:50.491 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:50.491 INFO:tasks.rados.rados.0.smithi123.stdout:1496: finishing write tid 4 to smithi12313934-37 2020-01-16T00:13:50.491 INFO:tasks.rados.rados.0.smithi123.stdout:1496: finishing write tid 5 to smithi12313934-37 2020-01-16T00:13:50.491 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 37 v 272 (ObjNum 502 snap 0 seq_num 502) dirty exists 2020-01-16T00:13:50.491 INFO:tasks.rados.rados.0.smithi123.stdout:1496: left oid 37 (ObjNum 502 snap 0 seq_num 502) 2020-01-16T00:13:50.491 INFO:tasks.rados.rados.0.smithi123.stdout:1497: finishing write tid 1 to smithi12313934-34 2020-01-16T00:13:50.491 INFO:tasks.rados.rados.0.smithi123.stdout:1496: done (15 left) 2020-01-16T00:13:50.492 INFO:tasks.rados.rados.0.smithi123.stdout:1512: write (excl) oid 15 current snap is 0 2020-01-16T00:13:50.492 INFO:tasks.rados.rados.0.smithi123.stdout:1512: seq_num 514 ranges {638798=473637,1594921=783167} 2020-01-16T00:13:50.492 INFO:tasks.rados.rados.0.smithi123.stdout:1512: writing smithi12313934-15 from 638798 to 1112435 tid 1 2020-01-16T00:13:50.492 INFO:tasks.rados.rados.0.smithi123.stdout:1512: writing smithi12313934-15 from 1594921 to 2378088 tid 2 2020-01-16T00:13:50.492 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:50.492 INFO:tasks.rados.rados.0.smithi123.stdout:1497: finishing write tid 2 to smithi12313934-34 2020-01-16T00:13:50.492 INFO:tasks.rados.rados.0.smithi123.stdout:1497: finishing write tid 4 to smithi12313934-34 2020-01-16T00:13:50.493 INFO:tasks.rados.rados.0.smithi123.stdout:1497: finishing write tid 5 to smithi12313934-34 2020-01-16T00:13:50.493 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 34 v 61 (ObjNum 503 snap 0 seq_num 503) dirty exists 2020-01-16T00:13:50.493 INFO:tasks.rados.rados.0.smithi123.stdout:1497: left oid 34 (ObjNum 503 snap 0 seq_num 503) 2020-01-16T00:13:50.493 INFO:tasks.rados.rados.0.smithi123.stdout:1499: finishing write tid 1 to smithi12313934-48 2020-01-16T00:13:50.493 INFO:tasks.rados.rados.0.smithi123.stdout:1498: expect (ObjNum 466 snap 0 seq_num 466) 2020-01-16T00:13:50.493 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:50.482+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:50.494 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-01-16T00:13:50.549 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-01-16T00:13:50.586 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:50.580+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:50.621 INFO:tasks.rados.rados.0.smithi123.stdout:1499: finishing write tid 2 to smithi12313934-48 2020-01-16T00:13:50.621 INFO:tasks.rados.rados.0.smithi123.stdout:1500: finishing write tid 1 to smithi12313934-23 2020-01-16T00:13:50.621 INFO:tasks.rados.rados.0.smithi123.stdout:1499: finishing write tid 4 to smithi12313934-48 2020-01-16T00:13:50.621 INFO:tasks.rados.rados.0.smithi123.stdout:1499: finishing write tid 5 to smithi12313934-48 2020-01-16T00:13:50.621 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 48 v 181 (ObjNum 504 snap 0 seq_num 504) dirty exists 2020-01-16T00:13:50.622 INFO:tasks.rados.rados.0.smithi123.stdout:1499: left oid 48 (ObjNum 504 snap 0 seq_num 504) 2020-01-16T00:13:50.622 INFO:tasks.rados.rados.0.smithi123.stdout:1500: finishing write tid 2 to smithi12313934-23 2020-01-16T00:13:50.622 INFO:tasks.rados.rados.0.smithi123.stdout:1500: finishing write tid 4 to smithi12313934-23 2020-01-16T00:13:50.622 INFO:tasks.rados.rados.0.smithi123.stdout:1500: finishing write tid 5 to smithi12313934-23 2020-01-16T00:13:50.622 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 23 v 161 (ObjNum 505 snap 0 seq_num 505) dirty exists 2020-01-16T00:13:50.622 INFO:tasks.rados.rados.0.smithi123.stdout:1500: left oid 23 (ObjNum 505 snap 0 seq_num 505) 2020-01-16T00:13:50.622 INFO:tasks.rados.rados.0.smithi123.stdout:1501: finishing write tid 1 to smithi12313934-28 2020-01-16T00:13:50.623 INFO:tasks.rados.rados.0.smithi123.stdout:1501: finishing write tid 2 to smithi12313934-28 2020-01-16T00:13:50.623 INFO:tasks.rados.rados.0.smithi123.stdout:1501: finishing write tid 4 to smithi12313934-28 2020-01-16T00:13:50.623 INFO:tasks.rados.rados.0.smithi123.stdout:1501: finishing write tid 5 to smithi12313934-28 2020-01-16T00:13:50.623 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 28 v 185 (ObjNum 506 snap 0 seq_num 506) dirty exists 2020-01-16T00:13:50.623 INFO:tasks.rados.rados.0.smithi123.stdout:1501: left oid 28 (ObjNum 506 snap 0 seq_num 506) 2020-01-16T00:13:50.623 INFO:tasks.rados.rados.0.smithi123.stdout:1502: finishing write tid 1 to smithi12313934-6 2020-01-16T00:13:50.623 INFO:tasks.rados.rados.0.smithi123.stdout:1502: finishing write tid 2 to smithi12313934-6 2020-01-16T00:13:50.624 INFO:tasks.rados.rados.0.smithi123.stdout:1497: done (15 left) 2020-01-16T00:13:50.624 INFO:tasks.rados.rados.0.smithi123.stdout:1498: done (14 left) 2020-01-16T00:13:50.624 INFO:tasks.rados.rados.0.smithi123.stdout:1499: done (13 left) 2020-01-16T00:13:50.624 INFO:tasks.rados.rados.0.smithi123.stdout:1500: done (12 left) 2020-01-16T00:13:50.624 INFO:tasks.rados.rados.0.smithi123.stdout:1501: done (11 left) 2020-01-16T00:13:50.624 INFO:tasks.rados.rados.0.smithi123.stdout:1513: write (excl) oid 25 current snap is 0 2020-01-16T00:13:50.624 INFO:tasks.rados.rados.0.smithi123.stdout:1513: seq_num 515 ranges {655605=548886,1653403=643178} 2020-01-16T00:13:50.639 INFO:tasks.rados.rados.0.smithi123.stdout:1513: writing smithi12313934-25 from 655605 to 1204491 tid 1 2020-01-16T00:13:50.644 INFO:tasks.rados.rados.0.smithi123.stdout:1513: writing smithi12313934-25 from 1653403 to 2296581 tid 2 2020-01-16T00:13:50.644 INFO:tasks.rados.rados.0.smithi123.stdout:1514: read oid 40 snap -1 2020-01-16T00:13:50.644 INFO:tasks.rados.rados.0.smithi123.stdout:1514: expect (ObjNum 467 snap 0 seq_num 467) 2020-01-16T00:13:50.644 INFO:tasks.rados.rados.0.smithi123.stdout:1515: write (excl) oid 21 current snap is 0 2020-01-16T00:13:50.645 INFO:tasks.rados.rados.0.smithi123.stdout:1515: seq_num 516 ranges {672412=624135,1995532=676880} 2020-01-16T00:13:50.650 INFO:tasks.rados.rados.0.smithi123.stdout:1515: writing smithi12313934-21 from 672412 to 1296547 tid 1 2020-01-16T00:13:50.651 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:50.651 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:13:50.651 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:13:50.652 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:50.659 INFO:tasks.rados.rados.0.smithi123.stdout:1515: writing smithi12313934-21 from 1995532 to 2672412 tid 2 2020-01-16T00:13:50.660 INFO:tasks.rados.rados.0.smithi123.stdout:1516: read oid 28 snap -1 2020-01-16T00:13:50.660 INFO:tasks.rados.rados.0.smithi123.stdout:1516: expect (ObjNum 506 snap 0 seq_num 506) 2020-01-16T00:13:50.660 INFO:tasks.rados.rados.0.smithi123.stdout:1517: read oid 3 snap -1 2020-01-16T00:13:50.660 INFO:tasks.rados.rados.0.smithi123.stdout:1517: expect (ObjNum 501 snap 0 seq_num 501) 2020-01-16T00:13:50.660 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:50.660 INFO:tasks.rados.rados.0.smithi123.stdout:1502: finishing write tid 4 to smithi12313934-6 2020-01-16T00:13:50.660 INFO:tasks.rados.rados.0.smithi123.stdout:1502: finishing write tid 5 to smithi12313934-6 2020-01-16T00:13:50.661 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 6 v 153 (ObjNum 507 snap 0 seq_num 507) dirty exists 2020-01-16T00:13:50.661 INFO:tasks.rados.rados.0.smithi123.stdout:1502: left oid 6 (ObjNum 507 snap 0 seq_num 507) 2020-01-16T00:13:50.661 INFO:tasks.rados.rados.0.smithi123.stdout:1503: finishing write tid 1 to smithi12313934-35 2020-01-16T00:13:50.661 INFO:tasks.rados.rados.0.smithi123.stdout:1503: finishing write tid 2 to smithi12313934-35 2020-01-16T00:13:50.661 INFO:tasks.rados.rados.0.smithi123.stdout:1502: done (15 left) 2020-01-16T00:13:50.661 INFO:tasks.rados.rados.0.smithi123.stdout:1518: read oid 1 snap -1 2020-01-16T00:13:50.662 INFO:tasks.rados.rados.0.smithi123.stdout:1518: expect (ObjNum 449 snap 0 seq_num 449) 2020-01-16T00:13:50.662 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:50.662 INFO:tasks.rados.rados.0.smithi123.stdout:1503: finishing write tid 4 to smithi12313934-35 2020-01-16T00:13:50.662 INFO:tasks.rados.rados.0.smithi123.stdout:1503: finishing write tid 5 to smithi12313934-35 2020-01-16T00:13:50.662 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 35 v 164 (ObjNum 508 snap 0 seq_num 508) dirty exists 2020-01-16T00:13:50.662 INFO:tasks.rados.rados.0.smithi123.stdout:1503: left oid 35 (ObjNum 508 snap 0 seq_num 508) 2020-01-16T00:13:50.662 INFO:tasks.rados.rados.0.smithi123.stdout:1504: finishing write tid 1 to smithi12313934-8 2020-01-16T00:13:50.662 INFO:tasks.rados.rados.0.smithi123.stdout:1504: finishing write tid 2 to smithi12313934-8 2020-01-16T00:13:50.663 INFO:tasks.rados.rados.0.smithi123.stdout:1504: finishing write tid 4 to smithi12313934-8 2020-01-16T00:13:50.663 INFO:tasks.rados.rados.0.smithi123.stdout:1504: finishing write tid 5 to smithi12313934-8 2020-01-16T00:13:50.663 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 8 v 109 (ObjNum 509 snap 0 seq_num 509) dirty exists 2020-01-16T00:13:50.663 INFO:tasks.rados.rados.0.smithi123.stdout:1504: left oid 8 (ObjNum 509 snap 0 seq_num 509) 2020-01-16T00:13:50.663 INFO:tasks.rados.rados.0.smithi123.stdout:1505: expect (ObjNum 362 snap 0 seq_num 362) 2020-01-16T00:13:50.667 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-01-16T00:13:50.682 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:13:50.686 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:50.682+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:50.699 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:13:50.717 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:13:50.736 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:13:50.758 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:13:50.780 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:13:50.787 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:50.782+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:50.798 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:13:50.800 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-01-16T00:13:50.800 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-01-16T00:13:50.800 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-01-16T00:13:50.801 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-01-16T00:13:50.801 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-01-16T00:13:50.801 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENXIO: problem getting command descriptions from osd.5 2020-01-16T00:13:50.801 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: problem getting command descriptions from osd.5 2020-01-16T00:13:50.801 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-01-16T00:13:50.802 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-01-16T00:13:50.809 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-01-16T00:13:50.809 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-01-16T00:13:50.822 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:50.838 INFO:tasks.rados.rados.0.smithi123.stdout:1506: finishing write tid 1 to smithi12313934-24 2020-01-16T00:13:50.839 INFO:tasks.rados.rados.0.smithi123.stdout:1506: finishing write tid 2 to smithi12313934-24 2020-01-16T00:13:50.839 INFO:tasks.rados.rados.0.smithi123.stdout:1503: done (15 left) 2020-01-16T00:13:50.839 INFO:tasks.rados.rados.0.smithi123.stdout:1504: done (14 left) 2020-01-16T00:13:50.839 INFO:tasks.rados.rados.0.smithi123.stdout:1505: done (13 left) 2020-01-16T00:13:50.839 INFO:tasks.rados.rados.0.smithi123.stdout:1519: read oid 37 snap -1 2020-01-16T00:13:50.839 INFO:tasks.rados.rados.0.smithi123.stdout:1519: expect (ObjNum 502 snap 0 seq_num 502) 2020-01-16T00:13:50.839 INFO:tasks.rados.rados.0.smithi123.stdout:1520: read oid 35 snap -1 2020-01-16T00:13:50.840 INFO:tasks.rados.rados.0.smithi123.stdout:1520: expect (ObjNum 508 snap 0 seq_num 508) 2020-01-16T00:13:50.840 INFO:tasks.rados.rados.0.smithi123.stdout:1521: write (excl) oid 12 current snap is 0 2020-01-16T00:13:50.840 INFO:tasks.rados.rados.0.smithi123.stdout:1521: seq_num 517 ranges {689219=415737,1770367=646847} 2020-01-16T00:13:50.841 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-01-16T00:13:50.845 INFO:tasks.rados.rados.0.smithi123.stdout:1521: writing smithi12313934-12 from 689219 to 1104956 tid 1 2020-01-16T00:13:50.856 INFO:tasks.rados.rados.0.smithi123.stdout:1521: writing smithi12313934-12 from 1770367 to 2417214 tid 2 2020-01-16T00:13:50.856 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:50.856 INFO:tasks.rados.rados.0.smithi123.stdout:1506: finishing write tid 4 to smithi12313934-24 2020-01-16T00:13:50.858 INFO:tasks.rados.rados.0.smithi123.stdout:1506: finishing write tid 5 to smithi12313934-24 2020-01-16T00:13:50.858 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 24 v 162 (ObjNum 510 snap 0 seq_num 510) dirty exists 2020-01-16T00:13:50.859 INFO:tasks.rados.rados.0.smithi123.stdout:1506: left oid 24 (ObjNum 510 snap 0 seq_num 510) 2020-01-16T00:13:50.859 INFO:tasks.rados.rados.0.smithi123.stdout:1507: finishing write tid 1 to smithi12313934-2 2020-01-16T00:13:50.859 INFO:tasks.rados.rados.0.smithi123.stdout:1506: done (15 left) 2020-01-16T00:13:50.859 INFO:tasks.rados.rados.0.smithi123.stdout:1522: write (excl) oid 8 current snap is 0 2020-01-16T00:13:50.859 INFO:tasks.rados.rados.0.smithi123.stdout:1522: seq_num 518 ranges {706026=490986,1828849=790505} 2020-01-16T00:13:50.862 INFO:tasks.rados.rados.0.smithi123.stdout:1522: writing smithi12313934-8 from 706026 to 1197012 tid 1 2020-01-16T00:13:50.870 INFO:tasks.rados.rados.0.smithi123.stdout:1522: writing smithi12313934-8 from 1828849 to 2619354 tid 2 2020-01-16T00:13:50.870 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:50.870 INFO:tasks.rados.rados.0.smithi123.stdout:1509: expect (ObjNum 330 snap 0 seq_num 330) 2020-01-16T00:13:50.890 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:50.886+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:50.952 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:50.952 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-01-16T00:13:50.999 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:50.998+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:51.062 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:51.071 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-01-16T00:13:51.092 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:51.088+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:51.117 INFO:tasks.rados.rados.0.smithi123.stdout:1507: finishing write tid 2 to smithi12313934-2 2020-01-16T00:13:51.117 INFO:tasks.rados.rados.0.smithi123.stdout:1507: finishing write tid 4 to smithi12313934-2 2020-01-16T00:13:51.117 INFO:tasks.rados.rados.0.smithi123.stdout:1510: finishing write tid 1 to smithi12313934-30 2020-01-16T00:13:51.118 INFO:tasks.rados.rados.0.smithi123.stdout:1507: finishing write tid 5 to smithi12313934-2 2020-01-16T00:13:51.118 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 2 v 218 (ObjNum 511 snap 0 seq_num 511) dirty exists 2020-01-16T00:13:51.118 INFO:tasks.rados.rados.0.smithi123.stdout:1507: left oid 2 (ObjNum 511 snap 0 seq_num 511) 2020-01-16T00:13:51.118 INFO:tasks.rados.rados.0.smithi123.stdout:1508: expect (ObjNum 461 snap 0 seq_num 461) 2020-01-16T00:13:51.193 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:51.188+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:51.203 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2020-01-16T00:13:51.217 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:51.226 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-01-16T00:13:51.294 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:51.292+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:51.363 INFO:tasks.rados.rados.0.smithi123.stdout:1510: finishing write tid 2 to smithi12313934-30 2020-01-16T00:13:51.363 INFO:tasks.rados.rados.0.smithi123.stdout:1510: finishing write tid 4 to smithi12313934-30 2020-01-16T00:13:51.364 INFO:tasks.rados.rados.0.smithi123.stdout:1510: finishing write tid 5 to smithi12313934-30 2020-01-16T00:13:51.364 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 30 v 166 (ObjNum 512 snap 0 seq_num 512) dirty exists 2020-01-16T00:13:51.364 INFO:tasks.rados.rados.0.smithi123.stdout:1510: left oid 30 (ObjNum 512 snap 0 seq_num 512) 2020-01-16T00:13:51.364 INFO:tasks.rados.rados.0.smithi123.stdout:1511: finishing write tid 1 to smithi12313934-32 2020-01-16T00:13:51.364 INFO:tasks.rados.rados.0.smithi123.stdout:1511: finishing write tid 2 to smithi12313934-32 2020-01-16T00:13:51.364 INFO:tasks.rados.rados.0.smithi123.stdout:1512: finishing write tid 1 to smithi12313934-15 2020-01-16T00:13:51.364 INFO:tasks.rados.rados.0.smithi123.stdout:1511: finishing write tid 4 to smithi12313934-32 2020-01-16T00:13:51.364 INFO:tasks.rados.rados.0.smithi123.stdout:1511: finishing write tid 5 to smithi12313934-32 2020-01-16T00:13:51.365 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 32 v 156 (ObjNum 513 snap 0 seq_num 513) dirty exists 2020-01-16T00:13:51.365 INFO:tasks.rados.rados.0.smithi123.stdout:1511: left oid 32 (ObjNum 513 snap 0 seq_num 513) 2020-01-16T00:13:51.365 INFO:tasks.rados.rados.0.smithi123.stdout:1512: finishing write tid 2 to smithi12313934-15 2020-01-16T00:13:51.365 INFO:tasks.rados.rados.0.smithi123.stdout:1512: finishing write tid 4 to smithi12313934-15 2020-01-16T00:13:51.365 INFO:tasks.rados.rados.0.smithi123.stdout:1507: done (15 left) 2020-01-16T00:13:51.365 INFO:tasks.rados.rados.0.smithi123.stdout:1508: done (14 left) 2020-01-16T00:13:51.365 INFO:tasks.rados.rados.0.smithi123.stdout:1509: done (13 left) 2020-01-16T00:13:51.366 INFO:tasks.rados.rados.0.smithi123.stdout:1510: done (12 left) 2020-01-16T00:13:51.366 INFO:tasks.rados.rados.0.smithi123.stdout:1511: done (11 left) 2020-01-16T00:13:51.366 INFO:tasks.rados.rados.0.smithi123.stdout:1523: read oid 18 snap -1 2020-01-16T00:13:51.366 INFO:tasks.rados.rados.0.smithi123.stdout:1523: expect (ObjNum 481 snap 0 seq_num 481) 2020-01-16T00:13:51.366 INFO:tasks.rados.rados.0.smithi123.stdout:1524: read oid 42 snap -1 2020-01-16T00:13:51.366 INFO:tasks.rados.rados.0.smithi123.stdout:1524: expect (ObjNum 432 snap 0 seq_num 432) 2020-01-16T00:13:51.366 INFO:tasks.rados.rados.0.smithi123.stdout:1525: read oid 45 snap -1 2020-01-16T00:13:51.367 INFO:tasks.rados.rados.0.smithi123.stdout:1525: expect (ObjNum 466 snap 0 seq_num 466) 2020-01-16T00:13:51.367 INFO:tasks.rados.rados.0.smithi123.stdout:1526: read oid 44 snap -1 2020-01-16T00:13:51.367 INFO:tasks.rados.rados.0.smithi123.stdout:1526: expect (ObjNum 461 snap 0 seq_num 461) 2020-01-16T00:13:51.367 INFO:tasks.rados.rados.0.smithi123.stdout:1527: write oid 39 current snap is 0 2020-01-16T00:13:51.367 INFO:tasks.rados.rados.0.smithi123.stdout:1527: seq_num 519 ranges {722833=566235,1887331=650516} 2020-01-16T00:13:51.372 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:51.380 INFO:tasks.rados.rados.0.smithi123.stdout:1527: writing smithi12313934-39 from 722833 to 1289068 tid 1 2020-01-16T00:13:51.382 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-01-16T00:13:51.392 INFO:tasks.rados.rados.0.smithi123.stdout:1527: writing smithi12313934-39 from 1887331 to 2537847 tid 2 2020-01-16T00:13:51.392 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:51.392 INFO:tasks.rados.rados.0.smithi123.stdout:1512: finishing write tid 5 to smithi12313934-15 2020-01-16T00:13:51.392 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 15 v 188 (ObjNum 514 snap 0 seq_num 514) dirty exists 2020-01-16T00:13:51.392 INFO:tasks.rados.rados.0.smithi123.stdout:1512: left oid 15 (ObjNum 514 snap 0 seq_num 514) 2020-01-16T00:13:51.392 INFO:tasks.rados.rados.0.smithi123.stdout:1512: done (15 left) 2020-01-16T00:13:51.393 INFO:tasks.rados.rados.0.smithi123.stdout:1528: read oid 4 snap -1 2020-01-16T00:13:51.393 INFO:tasks.rados.rados.0.smithi123.stdout:1528: expect (ObjNum 387 snap 0 seq_num 387) 2020-01-16T00:13:51.393 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:51.393 INFO:tasks.rados.rados.0.smithi123.stdout:1513: finishing write tid 1 to smithi12313934-25 2020-01-16T00:13:51.393 INFO:tasks.rados.rados.0.smithi123.stdout:1513: finishing write tid 2 to smithi12313934-25 2020-01-16T00:13:51.393 INFO:tasks.rados.rados.0.smithi123.stdout:1513: finishing write tid 4 to smithi12313934-25 2020-01-16T00:13:51.393 INFO:tasks.rados.rados.0.smithi123.stdout:1513: finishing write tid 5 to smithi12313934-25 2020-01-16T00:13:51.393 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 25 v 228 (ObjNum 515 snap 0 seq_num 515) dirty exists 2020-01-16T00:13:51.394 INFO:tasks.rados.rados.0.smithi123.stdout:1513: left oid 25 (ObjNum 515 snap 0 seq_num 515) 2020-01-16T00:13:51.394 INFO:tasks.rados.rados.0.smithi123.stdout:1515: finishing write tid 1 to smithi12313934-21 2020-01-16T00:13:51.394 INFO:tasks.rados.rados.0.smithi123.stdout:1513: done (15 left) 2020-01-16T00:13:51.394 INFO:tasks.rados.rados.0.smithi123.stdout:1529: read oid 13 snap -1 2020-01-16T00:13:51.394 INFO:tasks.rados.rados.0.smithi123.stdout:1529: expect (ObjNum 496 snap 0 seq_num 496) 2020-01-16T00:13:51.394 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:51.394 INFO:tasks.rados.rados.0.smithi123.stdout:1515: finishing write tid 2 to smithi12313934-21 2020-01-16T00:13:51.395 INFO:tasks.rados.rados.0.smithi123.stdout:1515: finishing write tid 4 to smithi12313934-21 2020-01-16T00:13:51.395 INFO:tasks.rados.rados.0.smithi123.stdout:1515: finishing write tid 5 to smithi12313934-21 2020-01-16T00:13:51.395 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 21 v 64 (ObjNum 516 snap 0 seq_num 516) dirty exists 2020-01-16T00:13:51.395 INFO:tasks.rados.rados.0.smithi123.stdout:1515: left oid 21 (ObjNum 516 snap 0 seq_num 516) 2020-01-16T00:13:51.395 INFO:tasks.rados.rados.0.smithi123.stdout:1514: expect (ObjNum 467 snap 0 seq_num 467) 2020-01-16T00:13:51.399 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:51.394+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:51.500 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:51.498+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:51.501 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:51.509 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-01-16T00:13:51.601 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:51.598+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:51.652 INFO:tasks.rados.rados.0.smithi123.stdout:1516: expect (ObjNum 506 snap 0 seq_num 506) 2020-01-16T00:13:51.653 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:51.666 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-01-16T00:13:51.705 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:51.700+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:51.804 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:51.802+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:51.808 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:51.817 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-01-16T00:13:51.830 INFO:tasks.rados.rados.0.smithi123.stdout:1517: expect (ObjNum 501 snap 0 seq_num 501) 2020-01-16T00:13:51.906 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:51.902+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:51.931 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:51.940 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-01-16T00:13:52.024 INFO:tasks.rados.rados.0.smithi123.stdout:1518: expect (ObjNum 449 snap 0 seq_num 449) 2020-01-16T00:13:52.024 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:52.022+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:52.060 INFO:teuthology.orchestra.run.smithi123.stderr:nodeep-scrub is unset 2020-01-16T00:13:52.103 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:52.107 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:52.104+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:52.109 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-01-16T00:13:52.208 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:52.206+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:52.250 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:52.259 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-01-16T00:13:52.260 INFO:tasks.rados.rados.0.smithi123.stdout:1519: expect (ObjNum 502 snap 0 seq_num 502) 2020-01-16T00:13:52.310 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:52.308+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:52.354 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:52.363 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-01-16T00:13:52.405 INFO:tasks.rados.rados.0.smithi123.stdout:1520: expect (ObjNum 508 snap 0 seq_num 508) 2020-01-16T00:13:52.411 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:52.408+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:52.464 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:52.473 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-01-16T00:13:52.515 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:52.514+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:52.540 INFO:tasks.rados.rados.0.smithi123.stdout:1521: finishing write tid 1 to smithi12313934-12 2020-01-16T00:13:52.541 INFO:tasks.rados.rados.0.smithi123.stdout:1521: finishing write tid 2 to smithi12313934-12 2020-01-16T00:13:52.541 INFO:tasks.rados.rados.0.smithi123.stdout:1521: finishing write tid 4 to smithi12313934-12 2020-01-16T00:13:52.541 INFO:tasks.rados.rados.0.smithi123.stdout:1521: finishing write tid 5 to smithi12313934-12 2020-01-16T00:13:52.541 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 12 v 156 (ObjNum 517 snap 0 seq_num 517) dirty exists 2020-01-16T00:13:52.541 INFO:tasks.rados.rados.0.smithi123.stdout:1521: left oid 12 (ObjNum 517 snap 0 seq_num 517) 2020-01-16T00:13:52.541 INFO:tasks.rados.rados.0.smithi123.stdout:1522: finishing write tid 1 to smithi12313934-8 2020-01-16T00:13:52.541 INFO:tasks.rados.rados.0.smithi123.stdout:1522: finishing write tid 2 to smithi12313934-8 2020-01-16T00:13:52.542 INFO:tasks.rados.rados.0.smithi123.stdout:1522: finishing write tid 4 to smithi12313934-8 2020-01-16T00:13:52.542 INFO:tasks.rados.rados.0.smithi123.stdout:1522: finishing write tid 5 to smithi12313934-8 2020-01-16T00:13:52.542 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 8 v 112 (ObjNum 518 snap 0 seq_num 518) dirty exists 2020-01-16T00:13:52.542 INFO:tasks.rados.rados.0.smithi123.stdout:1522: left oid 8 (ObjNum 518 snap 0 seq_num 518) 2020-01-16T00:13:52.542 INFO:tasks.rados.rados.0.smithi123.stdout:1524: expect (ObjNum 432 snap 0 seq_num 432) 2020-01-16T00:13:52.615 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:52.612+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:52.617 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:52.626 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-01-16T00:13:52.716 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:52.714+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:52.742 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:52.743 INFO:tasks.rados.rados.0.smithi123.stdout:1523: expect (ObjNum 481 snap 0 seq_num 481) 2020-01-16T00:13:52.751 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-01-16T00:13:52.810 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-01-16T00:13:52.817 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:52.812+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:52.853 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:52.863 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-01-16T00:13:52.898 INFO:tasks.rados.rados.0.smithi123.stdout:1525: expect (ObjNum 466 snap 0 seq_num 466) 2020-01-16T00:13:52.917 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:52.914+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:52.944 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:13:52.961 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:13:52.979 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:13:52.998 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:13:53.016 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:53.018 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:53.014+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:53.019 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:13:53.026 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-01-16T00:13:53.041 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:13:53.062 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:13:53.064 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-01-16T00:13:53.064 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-01-16T00:13:53.064 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-01-16T00:13:53.065 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-01-16T00:13:53.065 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-01-16T00:13:53.065 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENXIO: problem getting command descriptions from osd.5 2020-01-16T00:13:53.065 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: problem getting command descriptions from osd.5 2020-01-16T00:13:53.065 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-01-16T00:13:53.065 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-01-16T00:13:53.073 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-01-16T00:13:53.074 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-01-16T00:13:53.119 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:53.116+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:53.130 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:53.130 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:13:53.130 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:13:53.130 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:53.139 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-01-16T00:13:53.174 INFO:tasks.rados.rados.0.smithi123.stdout:1526: expect (ObjNum 461 snap 0 seq_num 461) 2020-01-16T00:13:53.220 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:53.218+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:53.291 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:53.291 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:13:53.291 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:13:53.291 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:13:53.292 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:53.300 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-01-16T00:13:53.321 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:53.318+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:53.408 INFO:tasks.rados.rados.0.smithi123.stdout:1527: finishing write tid 1 to smithi12313934-39 2020-01-16T00:13:53.408 INFO:tasks.rados.rados.0.smithi123.stdout:1527: finishing write tid 2 to smithi12313934-39 2020-01-16T00:13:53.408 INFO:tasks.rados.rados.0.smithi123.stdout:1529: expect (ObjNum 496 snap 0 seq_num 496) 2020-01-16T00:13:53.413 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:53.413 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:13:53.414 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:13:53.414 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:13:53.414 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.414 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.5.13:36 2.0 2:20e40cec:::smithi12313934-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 113152~1699840] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:13:53.414 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:13:53.415 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 75.577236146999994, 2020-01-16T00:13:53.415 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011736636, 2020-01-16T00:13:53.415 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:53.415 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:53.415 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:53.415 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.415 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:53.415 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:13:53.416 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:53.416 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.416 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.416 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:53.416 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:13:53.416 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999952 2020-01-16T00:13:53.416 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.417 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.417 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:53.417 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831934+0000", 2020-01-16T00:13:53.417 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001034325 2020-01-16T00:13:53.417 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.417 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.417 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:53.418 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.832968+0000", 2020-01-16T00:13:53.418 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.506e-05 2020-01-16T00:13:53.418 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.418 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.418 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:53.418 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833004+0000", 2020-01-16T00:13:53.418 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3406e-05 2020-01-16T00:13:53.419 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.419 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.419 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:53.419 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833017+0000", 2020-01-16T00:13:53.419 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015477000000000001 2020-01-16T00:13:53.419 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.419 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.420 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:53.420 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833172+0000", 2020-01-16T00:13:53.420 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033750700000000002 2020-01-16T00:13:53.420 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.420 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.420 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:53.420 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833509+0000", 2020-01-16T00:13:53.420 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00055861400000000003 2020-01-16T00:13:53.421 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.421 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.421 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:13:53.421 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.834068+0000", 2020-01-16T00:13:53.421 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.007474545 2020-01-16T00:13:53.421 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.421 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.422 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:53.422 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.841542+0000", 2020-01-16T00:13:53.422 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00056319499999999995 2020-01-16T00:13:53.422 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.422 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.422 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:53.422 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.842106+0000", 2020-01-16T00:13:53.422 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001371017 2020-01-16T00:13:53.423 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.423 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.423 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:53.423 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843477+0000", 2020-01-16T00:13:53.423 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000123938 2020-01-16T00:13:53.423 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.423 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.424 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:53.424 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843601+0000", 2020-01-16T00:13:53.424 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5012000000000006e-05 2020-01-16T00:13:53.424 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.424 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.424 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:53.424 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843676+0000", 2020-01-16T00:13:53.424 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011736636 2020-01-16T00:13:53.425 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.425 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:53.425 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.425 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.425 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.425 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1187 3.d 3:b1521a3e:::smithi12313934-25:head [write 1721042~423699] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:13:53.425 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.263828+0000", 2020-01-16T00:13:53.425 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 71.145347075000004, 2020-01-16T00:13:53.426 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011305709000000001, 2020-01-16T00:13:53.426 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:53.426 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:53.426 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:53.426 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:53.426 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:53.426 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1187 2020-01-16T00:13:53.426 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.427 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:53.427 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.427 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:53.427 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.263828+0000", 2020-01-16T00:13:53.427 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:53.427 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.427 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.428 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:53.428 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.263828+0000", 2020-01-16T00:13:53.428 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:13:53.428 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.428 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.428 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:53.428 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.263824+0000", 2020-01-16T00:13:53.429 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00048485899999999997 2020-01-16T00:13:53.429 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.429 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.429 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:53.429 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264309+0000", 2020-01-16T00:13:53.429 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5622999999999999e-05 2020-01-16T00:13:53.430 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.430 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.430 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:53.430 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264325+0000", 2020-01-16T00:13:53.430 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2530000000000004e-06 2020-01-16T00:13:53.430 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.430 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.430 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:53.431 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264330+0000", 2020-01-16T00:13:53.431 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000102252 2020-01-16T00:13:53.431 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.431 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.431 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:53.431 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264432+0000", 2020-01-16T00:13:53.431 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022361000000000001 2020-01-16T00:13:53.432 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.432 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.432 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:53.432 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264656+0000", 2020-01-16T00:13:53.432 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0028688849999999998 2020-01-16T00:13:53.432 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.432 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.432 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:53.433 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.267525+0000", 2020-01-16T00:13:53.433 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00023640900000000001 2020-01-16T00:13:53.433 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.433 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.433 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:53.433 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.267761+0000", 2020-01-16T00:13:53.433 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00035667000000000001 2020-01-16T00:13:53.434 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.434 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.434 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:53.434 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.268118+0000", 2020-01-16T00:13:53.434 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023898040000000001 2020-01-16T00:13:53.434 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.434 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.434 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:53.435 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.270507+0000", 2020-01-16T00:13:53.435 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014919690000000001 2020-01-16T00:13:53.435 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.435 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.435 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:53.435 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.271999+0000", 2020-01-16T00:13:53.435 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0029764510000000002 2020-01-16T00:13:53.435 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.436 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.436 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:53.436 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.274976+0000", 2020-01-16T00:13:53.436 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5409999999999997e-05 2020-01-16T00:13:53.436 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.436 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.436 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:53.437 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.275061+0000", 2020-01-16T00:13:53.437 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2452000000000004e-05 2020-01-16T00:13:53.437 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.437 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.437 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:53.437 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.275134+0000", 2020-01-16T00:13:53.437 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011305709000000001 2020-01-16T00:13:53.438 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.438 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:53.438 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.438 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.438 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.438 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1276 3.d 3:b1521a3e:::smithi12313934-25:head [read 0~3882384,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:13:53.438 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:13:53.439 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 70.334476577000004, 2020-01-16T00:13:53.439 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015839157999999999, 2020-01-16T00:13:53.439 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:53.439 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:53.439 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:53.439 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:53.439 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:53.439 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1276 2020-01-16T00:13:53.440 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.440 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:53.440 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.440 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:53.440 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:13:53.440 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:53.440 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.441 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.441 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:53.441 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:13:53.441 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:53.441 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.441 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.441 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:53.441 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:13:53.442 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5429999999999999e-06 2020-01-16T00:13:53.442 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.442 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.442 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:53.442 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074704+0000", 2020-01-16T00:13:53.442 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0839999999999992e-06 2020-01-16T00:13:53.442 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.443 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.443 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:53.443 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074712+0000", 2020-01-16T00:13:53.443 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.173e-06 2020-01-16T00:13:53.443 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.443 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.443 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:53.443 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074718+0000", 2020-01-16T00:13:53.444 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0083034700000000003 2020-01-16T00:13:53.444 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.444 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.444 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:53.444 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.083022+0000", 2020-01-16T00:13:53.444 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028456400000000001 2020-01-16T00:13:53.444 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.445 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.445 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:53.445 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.083306+0000", 2020-01-16T00:13:53.445 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0072312640000000003 2020-01-16T00:13:53.445 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.445 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.445 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:53.446 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090538+0000", 2020-01-16T00:13:53.446 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015839157999999999 2020-01-16T00:13:53.446 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.446 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:53.446 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.446 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.446 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.447 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1277 3.d 3:b1521a3e:::smithi12313934-25:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:13:53.447 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:13:53.447 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 70.334432987, 2020-01-16T00:13:53.447 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016646459999999998, 2020-01-16T00:13:53.447 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:53.447 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:53.447 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:53.447 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:53.448 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:53.448 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1277 2020-01-16T00:13:53.448 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.448 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:53.448 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.448 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:53.448 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:13:53.449 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:53.449 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.449 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.449 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:53.449 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:13:53.449 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:53.449 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.450 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.450 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:53.450 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074741+0000", 2020-01-16T00:13:53.450 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0776e-05 2020-01-16T00:13:53.450 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.450 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.450 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:53.450 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074752+0000", 2020-01-16T00:13:53.451 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7600000000000002e-06 2020-01-16T00:13:53.451 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.451 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.451 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:53.451 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074759+0000", 2020-01-16T00:13:53.451 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6840000000000001e-06 2020-01-16T00:13:53.451 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.451 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.452 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:53.452 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074765+0000", 2020-01-16T00:13:53.452 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015815216 2020-01-16T00:13:53.452 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.452 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.452 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:53.452 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090580+0000", 2020-01-16T00:13:53.453 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000182404 2020-01-16T00:13:53.453 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.453 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.453 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:53.453 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090763+0000", 2020-01-16T00:13:53.453 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00062579300000000003 2020-01-16T00:13:53.453 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.453 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.454 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:53.454 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091389+0000", 2020-01-16T00:13:53.454 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016646459999999998 2020-01-16T00:13:53.454 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.454 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:53.454 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.454 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.455 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.455 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1278 3.d 3:b1521a3e:::smithi12313934-25:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:13:53.455 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:13:53.455 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 70.334403273999996, 2020-01-16T00:13:53.455 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017994915, 2020-01-16T00:13:53.455 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:53.455 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:53.456 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:53.456 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:53.456 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:53.456 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1278 2020-01-16T00:13:53.456 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.456 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:53.456 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.456 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:53.457 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:13:53.457 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:53.457 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.457 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.457 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:53.457 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:13:53.457 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:53.458 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.458 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.458 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:53.458 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074771+0000", 2020-01-16T00:13:53.458 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4569999999999998e-06 2020-01-16T00:13:53.458 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.458 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.458 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:53.459 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074775+0000", 2020-01-16T00:13:53.459 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0330000000000003e-06 2020-01-16T00:13:53.459 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.459 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.459 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:53.459 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074782+0000", 2020-01-16T00:13:53.459 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9470000000000002e-06 2020-01-16T00:13:53.460 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.460 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.460 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:53.460 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074787+0000", 2020-01-16T00:13:53.460 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01666134 2020-01-16T00:13:53.460 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.460 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.460 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:53.461 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091449+0000", 2020-01-16T00:13:53.461 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000247731 2020-01-16T00:13:53.461 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.461 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.461 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:53.461 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091696+0000", 2020-01-16T00:13:53.461 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010703170000000001 2020-01-16T00:13:53.462 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.462 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.462 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:53.462 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.092767+0000", 2020-01-16T00:13:53.462 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017994915 2020-01-16T00:13:53.462 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.462 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:53.463 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.463 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.463 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.463 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2260 3.d 3:b76659c4:::smithi12313934-47:head [write 601750~535546] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:13:53.463 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:06.050577+0000", 2020-01-16T00:13:53.463 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 47.358597957999997, 2020-01-16T00:13:53.464 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012415381, 2020-01-16T00:13:53.464 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:53.464 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:53.464 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:53.464 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:53.464 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:53.464 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2260 2020-01-16T00:13:53.465 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.465 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:53.465 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.465 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:53.465 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.050577+0000", 2020-01-16T00:13:53.465 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:53.465 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.466 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.466 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:53.466 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.050577+0000", 2020-01-16T00:13:53.466 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:53.466 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.466 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.466 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:53.466 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.050574+0000", 2020-01-16T00:13:53.467 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001523487 2020-01-16T00:13:53.467 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.467 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.467 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:53.467 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052097+0000", 2020-01-16T00:13:53.467 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3697e-05 2020-01-16T00:13:53.467 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.467 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.468 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:53.468 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052121+0000", 2020-01-16T00:13:53.468 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0070000000000003e-06 2020-01-16T00:13:53.468 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.468 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.468 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:53.468 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052130+0000", 2020-01-16T00:13:53.468 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4559000000000003e-05 2020-01-16T00:13:53.469 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.469 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.469 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:53.469 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052194+0000", 2020-01-16T00:13:53.469 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000165878 2020-01-16T00:13:53.469 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.469 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.470 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:53.470 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052360+0000", 2020-01-16T00:13:53.470 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018825499999999999 2020-01-16T00:13:53.470 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.470 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.470 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:53.470 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052549+0000", 2020-01-16T00:13:53.471 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002761782 2020-01-16T00:13:53.471 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.471 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.471 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:53.471 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.055310+0000", 2020-01-16T00:13:53.471 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00060385500000000004 2020-01-16T00:13:53.471 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.471 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.471 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:53.472 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.055914+0000", 2020-01-16T00:13:53.472 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0069733809999999999 2020-01-16T00:13:53.472 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.472 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.472 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:53.472 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.062888+0000", 2020-01-16T00:13:53.472 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2884999999999999e-05 2020-01-16T00:13:53.473 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.473 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.473 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:53.473 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.062940+0000", 2020-01-16T00:13:53.473 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1996999999999997e-05 2020-01-16T00:13:53.473 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.473 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.473 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:53.474 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.062992+0000", 2020-01-16T00:13:53.474 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012415381 2020-01-16T00:13:53.474 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.474 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:53.474 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.474 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.474 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.475 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2295 3.d 3:b1521a3e:::smithi12313934-25:head [write 1763838~505140] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:13:53.475 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:06.660258+0000", 2020-01-16T00:13:53.475 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 46.748916731000001, 2020-01-16T00:13:53.475 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011681408000000001, 2020-01-16T00:13:53.475 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:53.475 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:53.475 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:53.476 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:53.476 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:53.476 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2295 2020-01-16T00:13:53.476 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.476 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:53.476 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.476 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:53.477 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660258+0000", 2020-01-16T00:13:53.477 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:53.477 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.477 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.477 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:53.477 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660258+0000", 2020-01-16T00:13:53.477 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:53.478 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.478 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.478 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:53.478 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660256+0000", 2020-01-16T00:13:53.478 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00066501699999999995 2020-01-16T00:13:53.478 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.478 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.479 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:53.479 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660921+0000", 2020-01-16T00:13:53.479 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7339000000000001e-05 2020-01-16T00:13:53.479 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.479 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.479 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:53.479 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660938+0000", 2020-01-16T00:13:53.479 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0219999999999993e-06 2020-01-16T00:13:53.480 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.480 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.480 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:53.480 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660947+0000", 2020-01-16T00:13:53.480 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00094237200000000002 2020-01-16T00:13:53.480 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.480 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.480 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:53.481 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.661890+0000", 2020-01-16T00:13:53.481 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000117649 2020-01-16T00:13:53.481 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.481 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.481 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:53.481 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662007+0000", 2020-01-16T00:13:53.481 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00060662599999999998 2020-01-16T00:13:53.482 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.482 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.482 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:53.482 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662614+0000", 2020-01-16T00:13:53.482 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028424899999999998 2020-01-16T00:13:53.482 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.482 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.483 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:53.483 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662898+0000", 2020-01-16T00:13:53.483 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00054904599999999997 2020-01-16T00:13:53.483 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.483 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.483 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:53.483 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.663447+0000", 2020-01-16T00:13:53.484 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0051479990000000003 2020-01-16T00:13:53.484 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.484 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.484 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:53.484 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.668595+0000", 2020-01-16T00:13:53.484 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00080881799999999999 2020-01-16T00:13:53.484 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.484 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.485 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:53.485 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.669404+0000", 2020-01-16T00:13:53.485 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0024495290000000002 2020-01-16T00:13:53.485 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.485 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.485 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:53.485 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671854+0000", 2020-01-16T00:13:53.486 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6635000000000003e-05 2020-01-16T00:13:53.486 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.486 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.486 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:53.486 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671900+0000", 2020-01-16T00:13:53.486 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9567999999999998e-05 2020-01-16T00:13:53.486 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.486 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.487 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:53.487 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671940+0000", 2020-01-16T00:13:53.487 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011681408000000001 2020-01-16T00:13:53.487 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.487 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:53.487 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.487 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.488 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.488 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2297 3.d 3:b1521a3e:::smithi12313934-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e35)", 2020-01-16T00:13:53.488 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:06.660986+0000", 2020-01-16T00:13:53.488 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 46.748188872999997, 2020-01-16T00:13:53.488 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011220677, 2020-01-16T00:13:53.488 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:53.488 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:53.489 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:53.489 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:53.489 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:53.489 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2297 2020-01-16T00:13:53.489 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.489 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:53.489 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.489 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:53.490 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660986+0000", 2020-01-16T00:13:53.490 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:53.490 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.490 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.490 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:53.490 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660986+0000", 2020-01-16T00:13:53.490 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:53.490 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.491 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.491 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:53.491 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660985+0000", 2020-01-16T00:13:53.491 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8760000000000004e-06 2020-01-16T00:13:53.491 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.491 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.491 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:53.492 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660990+0000", 2020-01-16T00:13:53.492 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.4190000000000002e-06 2020-01-16T00:13:53.492 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.492 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.492 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:53.492 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660999+0000", 2020-01-16T00:13:53.492 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7740000000000001e-06 2020-01-16T00:13:53.493 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.493 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.493 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:53.493 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.661004+0000", 2020-01-16T00:13:53.493 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0073783590000000001 2020-01-16T00:13:53.493 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.493 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.494 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:53.494 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.668383+0000", 2020-01-16T00:13:53.494 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014443 2020-01-16T00:13:53.494 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.494 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.494 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:53.494 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.668527+0000", 2020-01-16T00:13:53.494 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0035336539999999998 2020-01-16T00:13:53.495 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.495 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.495 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:53.495 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.672061+0000", 2020-01-16T00:13:53.495 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6445000000000005e-05 2020-01-16T00:13:53.495 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.495 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.496 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:53.496 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.672127+0000", 2020-01-16T00:13:53.496 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.9497000000000001e-05 2020-01-16T00:13:53.496 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.496 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.496 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:53.496 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.672207+0000", 2020-01-16T00:13:53.496 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011220677 2020-01-16T00:13:53.497 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.497 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:53.497 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.497 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.497 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.497 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.3.13:109 2.0 2:11fa79bc:::smithi12313934-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 331264~198656] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e42)", 2020-01-16T00:13:53.497 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:21.840075+0000", 2020-01-16T00:13:53.498 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 31.569100433999999, 2020-01-16T00:13:53.498 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011558677999999999, 2020-01-16T00:13:53.498 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:53.498 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:53.498 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:53.498 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.498 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:53.499 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840075+0000", 2020-01-16T00:13:53.499 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:53.499 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.499 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.499 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:53.499 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840075+0000", 2020-01-16T00:13:53.499 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:53.500 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.500 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.500 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:53.500 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840073+0000", 2020-01-16T00:13:53.500 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033576299999999999 2020-01-16T00:13:53.500 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.500 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.500 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:53.501 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840409+0000", 2020-01-16T00:13:53.501 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2082999999999998e-05 2020-01-16T00:13:53.501 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.501 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.501 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:53.501 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840431+0000", 2020-01-16T00:13:53.501 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4410000000000003e-06 2020-01-16T00:13:53.502 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.502 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.502 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:53.502 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840436+0000", 2020-01-16T00:13:53.502 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000101016 2020-01-16T00:13:53.502 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.502 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.502 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:53.503 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840537+0000", 2020-01-16T00:13:53.503 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000117639 2020-01-16T00:13:53.503 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.503 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.503 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:53.503 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840654+0000", 2020-01-16T00:13:53.503 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000196162 2020-01-16T00:13:53.504 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.504 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.504 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:13:53.504 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840850+0000", 2020-01-16T00:13:53.504 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003319092 2020-01-16T00:13:53.504 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.504 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.504 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:53.505 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.844169+0000", 2020-01-16T00:13:53.505 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011349229999999999 2020-01-16T00:13:53.505 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.505 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.505 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:53.505 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.845304+0000", 2020-01-16T00:13:53.505 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0062517670000000001 2020-01-16T00:13:53.505 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.506 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.506 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:53.506 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851556+0000", 2020-01-16T00:13:53.506 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4138000000000003e-05 2020-01-16T00:13:53.506 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.506 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.506 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:53.507 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851600+0000", 2020-01-16T00:13:53.507 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2993000000000001e-05 2020-01-16T00:13:53.507 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.507 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.507 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:53.507 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851633+0000", 2020-01-16T00:13:53.507 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011558677999999999 2020-01-16T00:13:53.507 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.508 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:53.508 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.508 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.508 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.508 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.3.13:110 2.0 2:11fa79bc:::smithi12313934-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 752640~650240] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e42)", 2020-01-16T00:13:53.508 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:21.840457+0000", 2020-01-16T00:13:53.508 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 31.568718142000002, 2020-01-16T00:13:53.509 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011329271, 2020-01-16T00:13:53.509 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:53.509 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:53.509 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:53.509 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.509 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:53.509 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840457+0000", 2020-01-16T00:13:53.510 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:53.510 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.510 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.510 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:53.510 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840457+0000", 2020-01-16T00:13:53.510 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:53.510 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.511 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.511 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:53.511 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840456+0000", 2020-01-16T00:13:53.511 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00062607899999999998 2020-01-16T00:13:53.511 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.511 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.512 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:53.512 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841082+0000", 2020-01-16T00:13:53.512 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1239999999999999e-06 2020-01-16T00:13:53.512 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.513 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.513 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:53.513 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841090+0000", 2020-01-16T00:13:53.513 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7129999999999999e-06 2020-01-16T00:13:53.513 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.513 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.513 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:53.513 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841093+0000", 2020-01-16T00:13:53.514 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00075997800000000002 2020-01-16T00:13:53.514 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.514 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.514 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:53.514 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841853+0000", 2020-01-16T00:13:53.514 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0690000000000003e-05 2020-01-16T00:13:53.514 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.515 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.515 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:53.515 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841924+0000", 2020-01-16T00:13:53.515 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000115753 2020-01-16T00:13:53.515 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.515 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.515 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:13:53.516 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.842040+0000", 2020-01-16T00:13:53.516 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0021358890000000002 2020-01-16T00:13:53.516 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.516 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.516 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:53.516 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.844176+0000", 2020-01-16T00:13:53.516 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0042541870000000004 2020-01-16T00:13:53.517 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.517 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.517 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:53.517 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.848430+0000", 2020-01-16T00:13:53.517 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0032476929999999998 2020-01-16T00:13:53.517 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.517 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.517 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:53.518 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851677+0000", 2020-01-16T00:13:53.518 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0370000000000001e-05 2020-01-16T00:13:53.518 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.518 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.518 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:53.518 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851728+0000", 2020-01-16T00:13:53.518 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.834e-05 2020-01-16T00:13:53.519 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.519 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.519 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:53.519 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851786+0000", 2020-01-16T00:13:53.519 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011329271 2020-01-16T00:13:53.519 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.519 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:53.520 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.520 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.520 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.520 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2987 3.d 3:b3dd5d45:::smithi12313934-16:head [write 467327~633480] snapc 0=[] ondisk+write+known_if_redirected e42)", 2020-01-16T00:13:53.520 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:22.257904+0000", 2020-01-16T00:13:53.520 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 31.151270847999999, 2020-01-16T00:13:53.520 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012187450000000001, 2020-01-16T00:13:53.521 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:53.521 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:53.521 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:53.521 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:53.521 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:53.521 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2987 2020-01-16T00:13:53.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.522 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:53.522 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.522 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:53.522 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.257904+0000", 2020-01-16T00:13:53.522 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:53.522 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.522 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.522 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:53.523 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.257904+0000", 2020-01-16T00:13:53.523 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999933 2020-01-16T00:13:53.523 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.523 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.523 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:53.523 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.257898+0000", 2020-01-16T00:13:53.523 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00097099800000000004 2020-01-16T00:13:53.524 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.524 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.524 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:53.524 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.258869+0000", 2020-01-16T00:13:53.524 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7970000000000003e-05 2020-01-16T00:13:53.524 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.524 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.524 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:53.525 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.258917+0000", 2020-01-16T00:13:53.525 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5982999999999999e-05 2020-01-16T00:13:53.525 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.525 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.525 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:53.525 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.258933+0000", 2020-01-16T00:13:53.525 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00026239999999999998 2020-01-16T00:13:53.526 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.526 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.526 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:53.526 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.259195+0000", 2020-01-16T00:13:53.526 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024907400000000001 2020-01-16T00:13:53.526 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.526 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.526 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:53.527 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.259444+0000", 2020-01-16T00:13:53.527 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00034824999999999998 2020-01-16T00:13:53.527 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.527 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.527 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:53.527 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.259792+0000", 2020-01-16T00:13:53.527 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003251538 2020-01-16T00:13:53.527 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.528 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.528 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:53.528 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.263044+0000", 2020-01-16T00:13:53.528 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0068502880000000004 2020-01-16T00:13:53.528 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.528 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.528 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:53.529 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.269894+0000", 2020-01-16T00:13:53.529 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0945000000000006e-05 2020-01-16T00:13:53.529 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.529 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.529 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:53.529 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.269965+0000", 2020-01-16T00:13:53.529 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012667399999999999 2020-01-16T00:13:53.530 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.530 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.530 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:53.530 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.270092+0000", 2020-01-16T00:13:53.530 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012187450000000001 2020-01-16T00:13:53.530 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.530 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:53.531 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.531 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.531 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.531 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3273 3.d 3:b76659c4:::smithi12313934-47:head [read 0~3647152,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:13:53.531 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.728775+0000", 2020-01-16T00:13:53.531 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 22.680400293999998, 2020-01-16T00:13:53.531 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012922950000000001, 2020-01-16T00:13:53.531 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:53.532 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:53.532 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:53.532 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:53.532 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:53.532 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3273 2020-01-16T00:13:53.532 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.532 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:53.533 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.533 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:53.533 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728775+0000", 2020-01-16T00:13:53.533 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:53.533 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.533 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.533 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:53.533 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728775+0000", 2020-01-16T00:13:53.534 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:53.534 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.534 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.534 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:53.534 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728771+0000", 2020-01-16T00:13:53.534 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.5009000000000001e-05 2020-01-16T00:13:53.534 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.535 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.535 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:53.535 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728796+0000", 2020-01-16T00:13:53.535 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2357999999999998e-05 2020-01-16T00:13:53.535 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.535 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.535 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:53.535 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728819+0000", 2020-01-16T00:13:53.536 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.4800000000000007e-06 2020-01-16T00:13:53.536 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.536 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.536 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:53.536 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728828+0000", 2020-01-16T00:13:53.536 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000124824 2020-01-16T00:13:53.536 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.537 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.537 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:53.537 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728953+0000", 2020-01-16T00:13:53.537 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00044904899999999998 2020-01-16T00:13:53.537 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.537 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.537 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:53.538 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729402+0000", 2020-01-16T00:13:53.538 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012295706999999999 2020-01-16T00:13:53.538 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.538 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.538 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:53.538 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.741698+0000", 2020-01-16T00:13:53.538 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012922950000000001 2020-01-16T00:13:53.538 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.539 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:53.539 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.539 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.539 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.539 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3275 3.d 3:b76659c4:::smithi12313934-47:head [read 0~3647152,checksum] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:13:53.539 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.728933+0000", 2020-01-16T00:13:53.539 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 22.680242022000002, 2020-01-16T00:13:53.540 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01204874, 2020-01-16T00:13:53.540 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:53.540 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:53.540 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:53.540 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:53.540 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:53.540 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3275 2020-01-16T00:13:53.540 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.541 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:53.541 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.541 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:53.541 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728933+0000", 2020-01-16T00:13:53.541 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:53.541 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.541 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.542 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:53.542 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728933+0000", 2020-01-16T00:13:53.542 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:53.542 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.542 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.542 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:53.542 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728932+0000", 2020-01-16T00:13:53.543 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3569999999999998e-06 2020-01-16T00:13:53.543 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.543 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.543 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:53.543 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728940+0000", 2020-01-16T00:13:53.543 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0457e-05 2020-01-16T00:13:53.543 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.543 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.544 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:53.544 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728951+0000", 2020-01-16T00:13:53.544 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6799999999999993e-06 2020-01-16T00:13:53.544 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.544 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.544 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:53.544 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728959+0000", 2020-01-16T00:13:53.545 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010076307 2020-01-16T00:13:53.545 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.545 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.545 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:53.545 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.739035+0000", 2020-01-16T00:13:53.545 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024455400000000002 2020-01-16T00:13:53.545 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.545 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.546 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:53.546 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.739279+0000", 2020-01-16T00:13:53.546 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0017023660000000001 2020-01-16T00:13:53.546 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.546 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.546 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:53.546 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.740982+0000", 2020-01-16T00:13:53.547 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01204874 2020-01-16T00:13:53.547 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.547 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:53.547 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.547 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.547 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.547 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3285 3.d 3:bd9fc72f:::smithi12313934-5:head [read 0~3764801,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:13:53.547 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.729068+0000", 2020-01-16T00:13:53.548 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 22.680107230000001, 2020-01-16T00:13:53.548 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018537010999999999, 2020-01-16T00:13:53.548 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:53.548 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:53.548 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:53.548 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:53.548 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:53.548 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3285 2020-01-16T00:13:53.549 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.549 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:53.549 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.549 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:53.549 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729068+0000", 2020-01-16T00:13:53.549 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:53.549 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.549 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.550 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:53.550 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729068+0000", 2020-01-16T00:13:53.550 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:53.550 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.550 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.550 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:53.550 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729067+0000", 2020-01-16T00:13:53.551 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3339999999999996e-06 2020-01-16T00:13:53.551 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.551 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.551 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:53.551 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729075+0000", 2020-01-16T00:13:53.551 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4187e-05 2020-01-16T00:13:53.551 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.551 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.552 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:53.552 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729089+0000", 2020-01-16T00:13:53.552 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5600000000000004e-06 2020-01-16T00:13:53.552 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.552 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.552 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:53.552 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729093+0000", 2020-01-16T00:13:53.553 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011944309 2020-01-16T00:13:53.553 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.553 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.553 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:53.553 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.741038+0000", 2020-01-16T00:13:53.553 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016625799999999999 2020-01-16T00:13:53.553 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.553 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.554 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:53.554 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.741204+0000", 2020-01-16T00:13:53.554 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0064010120000000002 2020-01-16T00:13:53.554 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.554 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.554 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:53.554 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.747605+0000", 2020-01-16T00:13:53.555 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018537010999999999 2020-01-16T00:13:53.555 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.555 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:53.555 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.555 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.555 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.555 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3286 3.d 3:bd9fc72f:::smithi12313934-5:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:13:53.556 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.729099+0000", 2020-01-16T00:13:53.556 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 22.68007656, 2020-01-16T00:13:53.556 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018983897999999999, 2020-01-16T00:13:53.556 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:53.556 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:53.556 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:53.556 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:53.556 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:53.557 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3286 2020-01-16T00:13:53.557 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.557 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:53.557 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.557 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:53.557 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729099+0000", 2020-01-16T00:13:53.557 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:53.558 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.558 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.558 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:53.558 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729099+0000", 2020-01-16T00:13:53.558 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:53.558 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.558 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.558 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:53.559 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729098+0000", 2020-01-16T00:13:53.559 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8500000000000004e-06 2020-01-16T00:13:53.559 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.559 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.559 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:53.559 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729102+0000", 2020-01-16T00:13:53.559 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7329999999999997e-06 2020-01-16T00:13:53.560 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.560 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.560 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:53.560 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729108+0000", 2020-01-16T00:13:53.560 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.764e-06 2020-01-16T00:13:53.560 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.560 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.560 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:53.561 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729118+0000", 2020-01-16T00:13:53.561 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018524286000000001 2020-01-16T00:13:53.561 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.561 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.561 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:53.561 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.747642+0000", 2020-01-16T00:13:53.561 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9487999999999993e-05 2020-01-16T00:13:53.562 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.562 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.562 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:53.562 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.747742+0000", 2020-01-16T00:13:53.562 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000340429 2020-01-16T00:13:53.562 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.562 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.562 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:53.563 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.748082+0000", 2020-01-16T00:13:53.563 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018983897999999999 2020-01-16T00:13:53.563 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.563 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:53.563 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.563 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.563 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.564 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3287 3.d 3:bd9fc72f:::smithi12313934-5:head [read 0~3764801,checksum] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:13:53.564 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.729124+0000", 2020-01-16T00:13:53.564 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 22.680051421000002, 2020-01-16T00:13:53.564 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020552323000000001, 2020-01-16T00:13:53.564 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:53.564 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:53.564 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:53.564 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:53.565 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:53.565 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3287 2020-01-16T00:13:53.565 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.565 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:53.565 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.565 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:53.565 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729124+0000", 2020-01-16T00:13:53.566 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:53.566 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.566 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.566 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:53.566 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729124+0000", 2020-01-16T00:13:53.566 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:53.566 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.567 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.567 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:53.567 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729123+0000", 2020-01-16T00:13:53.567 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8239999999999996e-06 2020-01-16T00:13:53.567 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.567 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.567 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:53.568 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729129+0000", 2020-01-16T00:13:53.568 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6909999999999997e-06 2020-01-16T00:13:53.568 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.568 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.568 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:53.568 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729136+0000", 2020-01-16T00:13:53.568 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3329999999999997e-06 2020-01-16T00:13:53.568 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.569 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.569 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:53.569 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729145+0000", 2020-01-16T00:13:53.569 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018965065999999999 2020-01-16T00:13:53.569 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.569 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.569 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:53.570 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.748110+0000", 2020-01-16T00:13:53.570 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015781700000000001 2020-01-16T00:13:53.570 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.570 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.570 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:53.570 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.748268+0000", 2020-01-16T00:13:53.570 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014083800000000001 2020-01-16T00:13:53.570 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.571 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.571 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:53.571 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.749676+0000", 2020-01-16T00:13:53.571 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020552323000000001 2020-01-16T00:13:53.571 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.571 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:53.571 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.571 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.572 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.572 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3409 3.d 3:b3dd5d45:::smithi12313934-16:head [write 591712~786646,stat] snapc 0=[] ondisk+write+known_if_redirected e47)", 2020-01-16T00:13:53.572 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:34.100681+0000", 2020-01-16T00:13:53.572 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 19.308494281000002, 2020-01-16T00:13:53.572 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012248446, 2020-01-16T00:13:53.572 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:53.572 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:53.572 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:53.573 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:53.573 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:53.573 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3409 2020-01-16T00:13:53.573 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.573 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:53.573 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.573 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:53.574 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.100681+0000", 2020-01-16T00:13:53.574 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:53.574 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.574 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.574 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:53.574 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.100681+0000", 2020-01-16T00:13:53.574 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:13:53.574 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.575 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.575 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:53.575 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.100677+0000", 2020-01-16T00:13:53.575 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000998828 2020-01-16T00:13:53.575 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.575 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.575 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:53.576 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101676+0000", 2020-01-16T00:13:53.576 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5846999999999997e-05 2020-01-16T00:13:53.576 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.576 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.576 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:53.576 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101712+0000", 2020-01-16T00:13:53.576 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.364e-05 2020-01-16T00:13:53.576 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.577 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.577 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:53.577 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101725+0000", 2020-01-16T00:13:53.577 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015902700000000001 2020-01-16T00:13:53.577 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.577 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.577 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:53.578 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101884+0000", 2020-01-16T00:13:53.578 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0486e-05 2020-01-16T00:13:53.578 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.578 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.578 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op must wait for map", 2020-01-16T00:13:53.578 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101905+0000", 2020-01-16T00:13:53.578 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0038869009999999999 2020-01-16T00:13:53.578 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.579 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.579 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:53.579 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.105792+0000", 2020-01-16T00:13:53.579 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030748800000000002 2020-01-16T00:13:53.579 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.579 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.579 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:53.579 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.106099+0000", 2020-01-16T00:13:53.580 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042069299999999998 2020-01-16T00:13:53.580 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.580 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.580 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:53.580 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.106520+0000", 2020-01-16T00:13:53.580 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0052864560000000001 2020-01-16T00:13:53.580 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.580 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.581 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:53.581 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.111806+0000", 2020-01-16T00:13:53.581 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00080835800000000001 2020-01-16T00:13:53.581 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.581 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.581 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:53.581 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.112615+0000", 2020-01-16T00:13:53.582 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1490999999999997e-05 2020-01-16T00:13:53.582 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.582 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.582 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:53.582 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.112686+0000", 2020-01-16T00:13:53.582 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000243174 2020-01-16T00:13:53.582 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.582 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.583 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:53.583 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.112929+0000", 2020-01-16T00:13:53.583 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012248446 2020-01-16T00:13:53.583 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.583 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:53.583 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.583 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.583 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.584 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.0.13:31 2.0 2:1afbc8ac:::smithi12313934-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 4096~737792] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e50)", 2020-01-16T00:13:53.584 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:44.061109+0000", 2020-01-16T00:13:53.584 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 9.3480660570000005, 2020-01-16T00:13:53.584 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012186723, 2020-01-16T00:13:53.584 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:53.584 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:53.584 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:53.585 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.585 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:53.585 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.061109+0000", 2020-01-16T00:13:53.585 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:53.585 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.585 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.585 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:53.585 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.061109+0000", 2020-01-16T00:13:53.586 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:53.586 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.586 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.586 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:53.586 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.061106+0000", 2020-01-16T00:13:53.586 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012349870000000001 2020-01-16T00:13:53.586 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.586 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.587 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:53.587 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062341+0000", 2020-01-16T00:13:53.587 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0235999999999996e-05 2020-01-16T00:13:53.587 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.587 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.587 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:53.587 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062411+0000", 2020-01-16T00:13:53.588 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7078000000000001e-05 2020-01-16T00:13:53.588 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.588 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.588 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:53.588 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062428+0000", 2020-01-16T00:13:53.588 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2158999999999996e-05 2020-01-16T00:13:53.588 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.588 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.589 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:53.589 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062520+0000", 2020-01-16T00:13:53.589 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011203499999999999 2020-01-16T00:13:53.589 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.589 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.589 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:53.589 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062632+0000", 2020-01-16T00:13:53.589 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017238400000000001 2020-01-16T00:13:53.590 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.590 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.590 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:13:53.590 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062805+0000", 2020-01-16T00:13:53.590 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010264989 2020-01-16T00:13:53.590 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.590 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.591 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:53.591 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073070+0000", 2020-01-16T00:13:53.591 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0533000000000001e-05 2020-01-16T00:13:53.591 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.591 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.591 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:53.591 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073090+0000", 2020-01-16T00:13:53.591 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2309000000000002e-05 2020-01-16T00:13:53.592 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.592 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.592 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:53.592 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073163+0000", 2020-01-16T00:13:53.592 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3035999999999997e-05 2020-01-16T00:13:53.592 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.592 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.592 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:53.593 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073236+0000", 2020-01-16T00:13:53.593 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0056999999999998e-05 2020-01-16T00:13:53.593 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.593 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.593 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:53.593 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073296+0000", 2020-01-16T00:13:53.593 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012186723 2020-01-16T00:13:53.594 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.594 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:53.594 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.594 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.594 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.594 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.0.13:32 2.0 2:1afbc8ac:::smithi12313934-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 916992~596480] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e50)", 2020-01-16T00:13:53.594 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:44.062464+0000", 2020-01-16T00:13:53.595 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 9.3467112720000003, 2020-01-16T00:13:53.595 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011316405999999999, 2020-01-16T00:13:53.595 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:53.595 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:53.595 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:53.595 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.595 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:53.595 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062464+0000", 2020-01-16T00:13:53.596 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:53.596 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.596 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.596 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:53.596 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062464+0000", 2020-01-16T00:13:53.596 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:53.596 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.596 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.597 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:53.597 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062462+0000", 2020-01-16T00:13:53.597 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00073330900000000002 2020-01-16T00:13:53.597 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.597 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.597 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:53.597 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063196+0000", 2020-01-16T00:13:53.598 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5330000000000001e-05 2020-01-16T00:13:53.598 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.598 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.598 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:53.598 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063211+0000", 2020-01-16T00:13:53.598 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5840000000000001e-06 2020-01-16T00:13:53.598 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.599 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.599 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:53.599 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063221+0000", 2020-01-16T00:13:53.599 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001786915 2020-01-16T00:13:53.599 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.599 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.599 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:53.599 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.065007+0000", 2020-01-16T00:13:53.600 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00021770000000000001 2020-01-16T00:13:53.600 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.600 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.600 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:53.600 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.065225+0000", 2020-01-16T00:13:53.600 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00032623 2020-01-16T00:13:53.600 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.601 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.601 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:13:53.601 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.065551+0000", 2020-01-16T00:13:53.601 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0078227499999999998 2020-01-16T00:13:53.601 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.601 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.601 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:53.602 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073374+0000", 2020-01-16T00:13:53.602 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.717e-05 2020-01-16T00:13:53.602 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.602 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.602 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:53.602 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073391+0000", 2020-01-16T00:13:53.602 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011493999999999999 2020-01-16T00:13:53.603 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.603 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.603 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:53.603 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073506+0000", 2020-01-16T00:13:53.603 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000159281 2020-01-16T00:13:53.603 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.603 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.603 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:53.604 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073666+0000", 2020-01-16T00:13:53.604 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000114675 2020-01-16T00:13:53.604 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.604 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.604 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:53.604 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073780+0000", 2020-01-16T00:13:53.604 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011316405999999999 2020-01-16T00:13:53.605 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.605 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:53.605 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.605 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.605 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.605 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.0.13:33 2.0 2:1afbc8ac:::smithi12313934-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 1541120~378368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e50)", 2020-01-16T00:13:53.605 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:44.063236+0000", 2020-01-16T00:13:53.606 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 9.3459393930000001, 2020-01-16T00:13:53.606 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015290962999999999, 2020-01-16T00:13:53.606 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:53.606 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:53.606 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:53.606 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.606 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:53.607 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063236+0000", 2020-01-16T00:13:53.607 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:53.607 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.607 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.607 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:53.607 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063236+0000", 2020-01-16T00:13:53.607 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:53.607 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.608 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.608 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:53.608 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063234+0000", 2020-01-16T00:13:53.608 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000500702 2020-01-16T00:13:53.608 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.608 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.608 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:53.608 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063735+0000", 2020-01-16T00:13:53.609 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4734e-05 2020-01-16T00:13:53.609 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.609 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.609 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:53.609 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063750+0000", 2020-01-16T00:13:53.609 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.382e-05 2020-01-16T00:13:53.609 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.609 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.610 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:53.610 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063764+0000", 2020-01-16T00:13:53.610 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0059030799999999998 2020-01-16T00:13:53.610 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.611 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.611 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:53.611 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.069667+0000", 2020-01-16T00:13:53.611 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3961999999999994e-05 2020-01-16T00:13:53.611 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.611 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.611 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:53.612 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.069741+0000", 2020-01-16T00:13:53.612 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000114931 2020-01-16T00:13:53.612 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.612 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.612 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:13:53.612 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.069856+0000", 2020-01-16T00:13:53.612 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0039948270000000003 2020-01-16T00:13:53.612 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.613 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.613 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:53.613 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073850+0000", 2020-01-16T00:13:53.613 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0016459140000000001 2020-01-16T00:13:53.613 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.613 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.613 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:53.614 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.075496+0000", 2020-01-16T00:13:53.614 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002957997 2020-01-16T00:13:53.614 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.614 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.614 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:53.614 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.078454+0000", 2020-01-16T00:13:53.614 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9190999999999998e-05 2020-01-16T00:13:53.615 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.615 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.615 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:53.615 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.078493+0000", 2020-01-16T00:13:53.615 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3176999999999998e-05 2020-01-16T00:13:53.615 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:53.616 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:53.616 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:53.616 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.078527+0000", 2020-01-16T00:13:53.616 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015290962999999999 2020-01-16T00:13:53.616 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.616 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:53.616 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.617 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:53.617 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:53.617 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:53.618 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:53.614+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:53.619 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-01-16T00:13:53.620 INFO:tasks.rados.rados.0.smithi123.stdout:1527: finishing write tid 4 to smithi12313934-39 2020-01-16T00:13:53.620 INFO:tasks.rados.rados.0.smithi123.stdout:1527: finishing write tid 5 to smithi12313934-39 2020-01-16T00:13:53.621 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 39 v 234 (ObjNum 519 snap 0 seq_num 519) dirty exists 2020-01-16T00:13:53.621 INFO:tasks.rados.rados.0.smithi123.stdout:1527: left oid 39 (ObjNum 519 snap 0 seq_num 519) 2020-01-16T00:13:53.621 INFO:tasks.rados.rados.0.smithi123.stdout:1528: expect (ObjNum 387 snap 0 seq_num 387) 2020-01-16T00:13:53.719 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:53.716+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:53.725 INFO:tasks.rados.rados.0.smithi123.stdout:1514: done (15 left) 2020-01-16T00:13:53.725 INFO:tasks.rados.rados.0.smithi123.stdout:1515: done (14 left) 2020-01-16T00:13:53.725 INFO:tasks.rados.rados.0.smithi123.stdout:1516: done (13 left) 2020-01-16T00:13:53.725 INFO:tasks.rados.rados.0.smithi123.stdout:1517: done (12 left) 2020-01-16T00:13:53.725 INFO:tasks.rados.rados.0.smithi123.stdout:1518: done (11 left) 2020-01-16T00:13:53.726 INFO:tasks.rados.rados.0.smithi123.stdout:1519: done (10 left) 2020-01-16T00:13:53.726 INFO:tasks.rados.rados.0.smithi123.stdout:1520: done (9 left) 2020-01-16T00:13:53.726 INFO:tasks.rados.rados.0.smithi123.stdout:1521: done (8 left) 2020-01-16T00:13:53.726 INFO:tasks.rados.rados.0.smithi123.stdout:1522: done (7 left) 2020-01-16T00:13:53.726 INFO:tasks.rados.rados.0.smithi123.stdout:1523: done (6 left) 2020-01-16T00:13:53.726 INFO:tasks.rados.rados.0.smithi123.stdout:1524: done (5 left) 2020-01-16T00:13:53.726 INFO:tasks.rados.rados.0.smithi123.stdout:1525: done (4 left) 2020-01-16T00:13:53.726 INFO:tasks.rados.rados.0.smithi123.stdout:1526: done (3 left) 2020-01-16T00:13:53.727 INFO:tasks.rados.rados.0.smithi123.stdout:1527: done (2 left) 2020-01-16T00:13:53.727 INFO:tasks.rados.rados.0.smithi123.stdout:1528: done (1 left) 2020-01-16T00:13:53.727 INFO:tasks.rados.rados.0.smithi123.stdout:1529: done (0 left) 2020-01-16T00:13:53.727 INFO:tasks.rados.rados.0.smithi123.stdout:1530: write (excl) oid 19 current snap is 0 2020-01-16T00:13:53.727 INFO:tasks.rados.rados.0.smithi123.stdout:1530: seq_num 520 ranges {739640=641484,1829460=794174} 2020-01-16T00:13:53.739 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:53.739 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:13:53.739 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:13:53.739 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:53.742 INFO:tasks.rados.rados.0.smithi123.stdout:1530: writing smithi12313934-19 from 739640 to 1381124 tid 1 2020-01-16T00:13:53.746 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-01-16T00:13:53.749 INFO:tasks.rados.rados.0.smithi123.stdout:1530: writing smithi12313934-19 from 1829460 to 2623634 tid 2 2020-01-16T00:13:53.749 INFO:tasks.rados.rados.0.smithi123.stdout:1531: write oid 1 current snap is 0 2020-01-16T00:13:53.749 INFO:tasks.rados.rados.0.smithi123.stdout:1531: seq_num 521 ranges {756447=716733,1887942=537832} 2020-01-16T00:13:53.757 INFO:tasks.rados.rados.0.smithi123.stdout:1531: writing smithi12313934-1 from 756447 to 1473180 tid 1 2020-01-16T00:13:53.763 INFO:tasks.rados.rados.0.smithi123.stdout:1531: writing smithi12313934-1 from 1887942 to 2425774 tid 2 2020-01-16T00:13:53.763 INFO:tasks.rados.rados.0.smithi123.stdout:1532: write oid 31 current snap is 0 2020-01-16T00:13:53.763 INFO:tasks.rados.rados.0.smithi123.stdout:1532: seq_num 522 ranges {773254=791982,2346424=426830} 2020-01-16T00:13:53.772 INFO:tasks.rados.rados.0.smithi123.stdout:1532: writing smithi12313934-31 from 773254 to 1565236 tid 1 2020-01-16T00:13:53.779 INFO:tasks.rados.rados.0.smithi123.stdout:1532: writing smithi12313934-31 from 2346424 to 2773254 tid 2 2020-01-16T00:13:53.779 INFO:tasks.rados.rados.0.smithi123.stdout:1533: read oid 16 snap -1 2020-01-16T00:13:53.780 INFO:tasks.rados.rados.0.smithi123.stdout:1533: expect (ObjNum 491 snap 0 seq_num 491) 2020-01-16T00:13:53.780 INFO:tasks.rados.rados.0.smithi123.stdout:1534: read oid 8 snap -1 2020-01-16T00:13:53.780 INFO:tasks.rados.rados.0.smithi123.stdout:1534: expect (ObjNum 518 snap 0 seq_num 518) 2020-01-16T00:13:53.780 INFO:tasks.rados.rados.0.smithi123.stdout:1535: read oid 4 snap -1 2020-01-16T00:13:53.780 INFO:tasks.rados.rados.0.smithi123.stdout:1535: expect (ObjNum 387 snap 0 seq_num 387) 2020-01-16T00:13:53.780 INFO:tasks.rados.rados.0.smithi123.stdout:1536: write oid 5 current snap is 0 2020-01-16T00:13:53.780 INFO:tasks.rados.rados.0.smithi123.stdout:1536: seq_num 523 ranges {790061=467231,2004906=541501} 2020-01-16T00:13:53.787 INFO:tasks.rados.rados.0.smithi123.stdout:1536: writing smithi12313934-5 from 790061 to 1257292 tid 1 2020-01-16T00:13:53.794 INFO:tasks.rados.rados.0.smithi123.stdout:1536: writing smithi12313934-5 from 2004906 to 2546407 tid 2 2020-01-16T00:13:53.794 INFO:tasks.rados.rados.0.smithi123.stdout:1537: write oid 40 current snap is 0 2020-01-16T00:13:53.794 INFO:tasks.rados.rados.0.smithi123.stdout:1537: seq_num 524 ranges {406868=542480,1547035=401512,2690354=116514} 2020-01-16T00:13:53.799 INFO:tasks.rados.rados.0.smithi123.stdout:1537: writing smithi12313934-40 from 406868 to 949348 tid 1 2020-01-16T00:13:53.805 INFO:tasks.rados.rados.0.smithi123.stdout:1537: writing smithi12313934-40 from 1547035 to 1948547 tid 2 2020-01-16T00:13:53.811 INFO:tasks.rados.rados.0.smithi123.stdout:1537: writing smithi12313934-40 from 2690354 to 2806868 tid 3 2020-01-16T00:13:53.811 INFO:tasks.rados.rados.0.smithi123.stdout:1538: read oid 22 snap -1 2020-01-16T00:13:53.811 INFO:tasks.rados.rados.0.smithi123.stdout:1538: expect (ObjNum 393 snap 0 seq_num 393) 2020-01-16T00:13:53.811 INFO:tasks.rados.rados.0.smithi123.stdout:1539: delete oid 21 current snap is 0 2020-01-16T00:13:53.812 INFO:tasks.rados.rados.0.smithi123.stdout:1530: finishing write tid 1 to smithi12313934-19 2020-01-16T00:13:53.812 INFO:tasks.rados.rados.0.smithi123.stdout:1530: finishing write tid 2 to smithi12313934-19 2020-01-16T00:13:53.812 INFO:tasks.rados.rados.0.smithi123.stdout:1530: finishing write tid 4 to smithi12313934-19 2020-01-16T00:13:53.812 INFO:tasks.rados.rados.0.smithi123.stdout:1530: finishing write tid 5 to smithi12313934-19 2020-01-16T00:13:53.812 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 19 v 221 (ObjNum 520 snap 0 seq_num 520) dirty exists 2020-01-16T00:13:53.812 INFO:tasks.rados.rados.0.smithi123.stdout:1530: left oid 19 (ObjNum 520 snap 0 seq_num 520) 2020-01-16T00:13:53.812 INFO:tasks.rados.rados.0.smithi123.stdout:1531: finishing write tid 1 to smithi12313934-1 2020-01-16T00:13:53.812 INFO:tasks.rados.rados.0.smithi123.stdout:1531: finishing write tid 2 to smithi12313934-1 2020-01-16T00:13:53.813 INFO:tasks.rados.rados.0.smithi123.stdout:1531: finishing write tid 4 to smithi12313934-1 2020-01-16T00:13:53.813 INFO:tasks.rados.rados.0.smithi123.stdout:1531: finishing write tid 5 to smithi12313934-1 2020-01-16T00:13:53.813 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 1 v 275 (ObjNum 521 snap 0 seq_num 521) dirty exists 2020-01-16T00:13:53.813 INFO:tasks.rados.rados.0.smithi123.stdout:1531: left oid 1 (ObjNum 521 snap 0 seq_num 521) 2020-01-16T00:13:53.813 INFO:tasks.rados.rados.0.smithi123.stdout:1532: finishing write tid 1 to smithi12313934-31 2020-01-16T00:13:53.813 INFO:tasks.rados.rados.0.smithi123.stdout:1532: finishing write tid 2 to smithi12313934-31 2020-01-16T00:13:53.813 INFO:tasks.rados.rados.0.smithi123.stdout:1532: finishing write tid 4 to smithi12313934-31 2020-01-16T00:13:53.814 INFO:tasks.rados.rados.0.smithi123.stdout:1534: expect (ObjNum 518 snap 0 seq_num 518) 2020-01-16T00:13:53.820 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:53.818+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:53.888 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:53.889 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:13:53.889 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:13:53.889 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:13:53.889 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:53.898 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-01-16T00:13:53.942 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:53.940+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:54.004 INFO:tasks.rados.rados.0.smithi123.stdout:1532: finishing write tid 5 to smithi12313934-31 2020-01-16T00:13:54.004 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 31 v 191 (ObjNum 522 snap 0 seq_num 522) dirty exists 2020-01-16T00:13:54.004 INFO:tasks.rados.rados.0.smithi123.stdout:1532: left oid 31 (ObjNum 522 snap 0 seq_num 522) 2020-01-16T00:13:54.004 INFO:tasks.rados.rados.0.smithi123.stdout:1530: done (9 left) 2020-01-16T00:13:54.005 INFO:tasks.rados.rados.0.smithi123.stdout:1531: done (8 left) 2020-01-16T00:13:54.005 INFO:tasks.rados.rados.0.smithi123.stdout:1532: done (7 left) 2020-01-16T00:13:54.005 INFO:tasks.rados.rados.0.smithi123.stdout:1534: done (6 left) 2020-01-16T00:13:54.005 INFO:tasks.rados.rados.0.smithi123.stdout:1539: done (5 left) 2020-01-16T00:13:54.005 INFO:tasks.rados.rados.0.smithi123.stdout:1540: read oid 47 snap -1 2020-01-16T00:13:54.005 INFO:tasks.rados.rados.0.smithi123.stdout:1540: expect (ObjNum 431 snap 0 seq_num 431) 2020-01-16T00:13:54.005 INFO:tasks.rados.rados.0.smithi123.stdout:1541: write oid 49 current snap is 0 2020-01-16T00:13:54.005 INFO:tasks.rados.rados.0.smithi123.stdout:1541: seq_num 526 ranges {440482=409331,1380352=405181,2461553=378929} 2020-01-16T00:13:54.007 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:13:54.007 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:13:54.008 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:13:54.008 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:13:54.008 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.008 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1654 3.a 3:5069665e:::smithi12313934-2:head [write 1614880~581041] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:13:54.008 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.714839+0000", 2020-01-16T00:13:54.008 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 61.287845476999998, 2020-01-16T00:13:54.009 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013588093000000001, 2020-01-16T00:13:54.009 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:54.009 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:54.009 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:54.009 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:54.009 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.009 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1654 2020-01-16T00:13:54.010 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.010 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:54.010 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.010 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:54.010 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.714839+0000", 2020-01-16T00:13:54.010 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:54.010 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.010 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.011 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:54.011 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.714839+0000", 2020-01-16T00:13:54.011 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:54.011 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.011 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.011 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:54.011 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.714836+0000", 2020-01-16T00:13:54.012 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00054657999999999996 2020-01-16T00:13:54.012 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.012 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.012 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:54.012 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.715382+0000", 2020-01-16T00:13:54.012 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7743000000000001e-05 2020-01-16T00:13:54.012 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.012 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.013 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:54.013 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.715420+0000", 2020-01-16T00:13:54.013 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.221e-05 2020-01-16T00:13:54.013 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.013 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.013 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.013 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.715432+0000", 2020-01-16T00:13:54.014 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2487000000000001e-05 2020-01-16T00:13:54.014 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.014 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.014 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.014 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.715485+0000", 2020-01-16T00:13:54.014 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000726284 2020-01-16T00:13:54.014 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.015 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.015 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.015 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.716211+0000", 2020-01-16T00:13:54.015 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0067930899999999999 2020-01-16T00:13:54.015 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.015 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.015 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.015 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.723004+0000", 2020-01-16T00:13:54.016 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012579400000000001 2020-01-16T00:13:54.016 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.016 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.016 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:54.016 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.723130+0000", 2020-01-16T00:13:54.016 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013465800000000001 2020-01-16T00:13:54.016 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.016 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.017 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:54.017 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.723265+0000", 2020-01-16T00:13:54.017 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023307250000000001 2020-01-16T00:13:54.017 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.017 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.017 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:54.018 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.725595+0000", 2020-01-16T00:13:54.018 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012582139999999999 2020-01-16T00:13:54.018 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.018 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.018 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.018 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.726853+0000", 2020-01-16T00:13:54.018 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014479110000000001 2020-01-16T00:13:54.019 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.019 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.019 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.019 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.728301+0000", 2020-01-16T00:13:54.019 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6866000000000007e-05 2020-01-16T00:13:54.019 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.019 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.019 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:54.020 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.728368+0000", 2020-01-16T00:13:54.020 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8740999999999999e-05 2020-01-16T00:13:54.020 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.020 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.020 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:54.020 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.728427+0000", 2020-01-16T00:13:54.020 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013588093000000001 2020-01-16T00:13:54.020 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.020 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:54.021 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.021 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.021 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.021 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1655 3.a 3:5069665e:::smithi12313934-2:head [write 2785791~71399] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:13:54.021 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.718297+0000", 2020-01-16T00:13:54.021 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 61.284387623999997, 2020-01-16T00:13:54.022 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013817354, 2020-01-16T00:13:54.022 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:54.022 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:54.022 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:54.022 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:54.022 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.023 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1655 2020-01-16T00:13:54.023 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.023 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:54.023 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.023 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:54.024 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718297+0000", 2020-01-16T00:13:54.024 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:54.024 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.024 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.024 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:54.024 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718297+0000", 2020-01-16T00:13:54.024 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:54.024 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.025 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.025 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:54.025 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718294+0000", 2020-01-16T00:13:54.025 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000103558 2020-01-16T00:13:54.026 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.026 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.026 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:54.026 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718397+0000", 2020-01-16T00:13:54.026 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9287e-05 2020-01-16T00:13:54.026 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.027 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.027 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:54.027 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718417+0000", 2020-01-16T00:13:54.027 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6680000000000001e-06 2020-01-16T00:13:54.027 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.027 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.027 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.027 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718424+0000", 2020-01-16T00:13:54.028 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2701999999999994e-05 2020-01-16T00:13:54.028 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.028 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.028 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.028 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718517+0000", 2020-01-16T00:13:54.028 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3893000000000006e-05 2020-01-16T00:13:54.028 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.029 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.029 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.029 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718581+0000", 2020-01-16T00:13:54.029 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0059630079999999997 2020-01-16T00:13:54.029 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.029 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.029 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.029 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.724544+0000", 2020-01-16T00:13:54.030 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000309108 2020-01-16T00:13:54.030 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.030 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.030 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.030 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.724853+0000", 2020-01-16T00:13:54.030 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.005469794 2020-01-16T00:13:54.031 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.031 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.031 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.031 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.730323+0000", 2020-01-16T00:13:54.031 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018506100000000001 2020-01-16T00:13:54.031 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.031 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.032 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:54.032 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.730508+0000", 2020-01-16T00:13:54.032 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016729099999999999 2020-01-16T00:13:54.032 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.032 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.032 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:54.033 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.730675+0000", 2020-01-16T00:13:54.033 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00094193300000000005 2020-01-16T00:13:54.033 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.033 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.033 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:54.033 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731617+0000", 2020-01-16T00:13:54.033 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000214613 2020-01-16T00:13:54.033 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.034 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.034 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.034 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731832+0000", 2020-01-16T00:13:54.034 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018326400000000001 2020-01-16T00:13:54.034 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.034 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.034 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.035 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732015+0000", 2020-01-16T00:13:54.035 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0392e-05 2020-01-16T00:13:54.035 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.035 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.035 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:54.035 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732075+0000", 2020-01-16T00:13:54.035 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8677999999999998e-05 2020-01-16T00:13:54.035 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.036 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.036 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:54.036 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732114+0000", 2020-01-16T00:13:54.036 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013817354 2020-01-16T00:13:54.036 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.036 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:54.036 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.037 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.037 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.037 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1656 3.a 3:5069665e:::smithi12313934-2:head [setxattr _header (55) in=7b,truncate 2857190] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:13:54.037 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.718453+0000", 2020-01-16T00:13:54.037 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 61.284230942999997, 2020-01-16T00:13:54.037 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014083462, 2020-01-16T00:13:54.037 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:54.037 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:54.038 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:54.038 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:54.038 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.038 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1656 2020-01-16T00:13:54.038 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.038 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:54.039 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.039 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:54.039 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718453+0000", 2020-01-16T00:13:54.039 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:54.039 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.039 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.039 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:54.040 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718453+0000", 2020-01-16T00:13:54.040 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999828 2020-01-16T00:13:54.040 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.040 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.040 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:54.040 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718436+0000", 2020-01-16T00:13:54.041 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4051999999999999e-05 2020-01-16T00:13:54.041 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.041 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.041 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:54.041 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718460+0000", 2020-01-16T00:13:54.041 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6380000000000004e-06 2020-01-16T00:13:54.041 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.041 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.042 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:54.042 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718468+0000", 2020-01-16T00:13:54.042 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6750000000000001e-06 2020-01-16T00:13:54.042 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.042 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.042 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.042 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718472+0000", 2020-01-16T00:13:54.042 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016218499999999999 2020-01-16T00:13:54.043 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.043 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.043 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.043 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718635+0000", 2020-01-16T00:13:54.043 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2765e-05 2020-01-16T00:13:54.043 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.043 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.043 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.044 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718697+0000", 2020-01-16T00:13:54.044 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0062239249999999999 2020-01-16T00:13:54.044 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.044 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.044 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.044 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.724921+0000", 2020-01-16T00:13:54.044 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1527999999999999e-05 2020-01-16T00:13:54.044 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.045 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.045 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.045 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.724973+0000", 2020-01-16T00:13:54.045 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0060569719999999999 2020-01-16T00:13:54.045 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.045 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.045 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.045 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731030+0000", 2020-01-16T00:13:54.046 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5859999999999996e-05 2020-01-16T00:13:54.046 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.046 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.046 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:54.046 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731096+0000", 2020-01-16T00:13:54.046 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000112018 2020-01-16T00:13:54.046 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.047 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.047 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:54.047 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731208+0000", 2020-01-16T00:13:54.047 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000945889 2020-01-16T00:13:54.047 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.047 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.047 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.047 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732154+0000", 2020-01-16T00:13:54.048 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0834000000000003e-05 2020-01-16T00:13:54.048 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.048 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.048 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.048 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732184+0000", 2020-01-16T00:13:54.048 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028587600000000001 2020-01-16T00:13:54.048 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.049 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.049 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:54.049 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732470+0000", 2020-01-16T00:13:54.049 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1953999999999999e-05 2020-01-16T00:13:54.049 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.049 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.049 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:54.050 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732502+0000", 2020-01-16T00:13:54.050 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4644999999999998e-05 2020-01-16T00:13:54.050 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.050 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.050 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:54.051 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732537+0000", 2020-01-16T00:13:54.051 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014083462 2020-01-16T00:13:54.051 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.051 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:54.051 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.051 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.051 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.051 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1657 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-01-16T00:13:54.052 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.718495+0000", 2020-01-16T00:13:54.052 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 61.284189277999999, 2020-01-16T00:13:54.052 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014299630000000001, 2020-01-16T00:13:54.052 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:54.052 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:54.052 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:54.052 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:54.052 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.053 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1657 2020-01-16T00:13:54.053 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.053 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:54.053 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.053 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:54.053 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718495+0000", 2020-01-16T00:13:54.053 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:54.054 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.054 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.054 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:54.054 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718495+0000", 2020-01-16T00:13:54.054 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:54.054 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.054 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.054 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:54.055 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718494+0000", 2020-01-16T00:13:54.055 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8860000000000003e-06 2020-01-16T00:13:54.055 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.055 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.055 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:54.055 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718499+0000", 2020-01-16T00:13:54.055 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9786000000000002e-05 2020-01-16T00:13:54.055 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.056 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.056 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:54.056 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718539+0000", 2020-01-16T00:13:54.056 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1869999999999999e-05 2020-01-16T00:13:54.056 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.056 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.056 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.056 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718561+0000", 2020-01-16T00:13:54.057 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000172206 2020-01-16T00:13:54.057 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.057 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.057 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.057 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718733+0000", 2020-01-16T00:13:54.057 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0662999999999998e-05 2020-01-16T00:13:54.057 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.058 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.058 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.058 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718793+0000", 2020-01-16T00:13:54.058 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.006217958 2020-01-16T00:13:54.058 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.058 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.058 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.059 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.725011+0000", 2020-01-16T00:13:54.059 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1625999999999996e-05 2020-01-16T00:13:54.059 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.059 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.059 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.059 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.725073+0000", 2020-01-16T00:13:54.059 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0063923119999999998 2020-01-16T00:13:54.059 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.060 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.060 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.060 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731465+0000", 2020-01-16T00:13:54.060 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2534000000000005e-05 2020-01-16T00:13:54.060 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.060 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.060 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:54.060 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731558+0000", 2020-01-16T00:13:54.061 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010238809999999999 2020-01-16T00:13:54.061 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.061 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.062 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.062 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732582+0000", 2020-01-16T00:13:54.062 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000100918 2020-01-16T00:13:54.062 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.062 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.063 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:54.063 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732683+0000", 2020-01-16T00:13:54.063 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000112064 2020-01-16T00:13:54.063 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.064 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.064 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:54.064 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732795+0000", 2020-01-16T00:13:54.064 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014299630000000001 2020-01-16T00:13:54.064 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.064 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:54.064 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.065 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.065 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.065 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1917 3.a 3:5069665e:::smithi12313934-2:head [write 2113302~674388] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:13:54.066 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:57.330712+0000", 2020-01-16T00:13:54.066 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 56.671972762999999, 2020-01-16T00:13:54.067 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012781043000000001, 2020-01-16T00:13:54.067 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:54.067 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:54.067 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:54.067 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:54.067 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.067 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1917 2020-01-16T00:13:54.067 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.067 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:54.068 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.068 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:54.068 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.330712+0000", 2020-01-16T00:13:54.068 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:54.068 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.068 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.068 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:54.069 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.330712+0000", 2020-01-16T00:13:54.069 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:54.069 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.069 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.069 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:54.069 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.330709+0000", 2020-01-16T00:13:54.069 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00071550499999999996 2020-01-16T00:13:54.069 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.070 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.070 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:54.070 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331424+0000", 2020-01-16T00:13:54.070 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3354e-05 2020-01-16T00:13:54.070 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.070 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.070 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:54.070 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331447+0000", 2020-01-16T00:13:54.071 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8200000000000008e-06 2020-01-16T00:13:54.071 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.071 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.071 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.071 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331457+0000", 2020-01-16T00:13:54.071 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5557999999999997e-05 2020-01-16T00:13:54.071 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.072 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.072 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.072 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331533+0000", 2020-01-16T00:13:54.072 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030925000000000001 2020-01-16T00:13:54.072 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.072 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.073 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.073 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331842+0000", 2020-01-16T00:13:54.073 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0078507860000000002 2020-01-16T00:13:54.074 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.074 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.074 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.074 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.339693+0000", 2020-01-16T00:13:54.074 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9321000000000007e-05 2020-01-16T00:13:54.075 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.075 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.075 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:54.075 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.339792+0000", 2020-01-16T00:13:54.075 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015147700000000001 2020-01-16T00:13:54.075 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.075 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.076 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:54.076 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.339944+0000", 2020-01-16T00:13:54.076 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002663066 2020-01-16T00:13:54.076 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.076 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.076 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:54.076 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.342607+0000", 2020-01-16T00:13:54.077 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0001552 2020-01-16T00:13:54.077 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.077 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.077 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.077 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.342762+0000", 2020-01-16T00:13:54.077 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00063636900000000002 2020-01-16T00:13:54.077 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.077 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.078 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.078 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343398+0000", 2020-01-16T00:13:54.078 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9687000000000001e-05 2020-01-16T00:13:54.078 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.078 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.079 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:54.079 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343438+0000", 2020-01-16T00:13:54.079 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4647e-05 2020-01-16T00:13:54.079 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.080 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.080 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:54.080 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343493+0000", 2020-01-16T00:13:54.080 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012781043000000001 2020-01-16T00:13:54.080 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.081 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:54.081 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.081 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.081 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.081 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2308 3.a 3:5c245412:::smithi12313934-19:head [write 702592~703393] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:13:54.081 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:07.298509+0000", 2020-01-16T00:13:54.082 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 46.704175106999998, 2020-01-16T00:13:54.082 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012139225999999999, 2020-01-16T00:13:54.082 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:54.082 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:54.082 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:54.082 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:54.083 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.083 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2308 2020-01-16T00:13:54.083 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.083 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:54.083 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.083 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:54.083 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.298509+0000", 2020-01-16T00:13:54.083 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:54.084 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.084 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.084 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:54.084 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.298509+0000", 2020-01-16T00:13:54.084 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999957 2020-01-16T00:13:54.084 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.084 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.085 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:54.085 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.298505+0000", 2020-01-16T00:13:54.085 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00085313800000000003 2020-01-16T00:13:54.085 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.085 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.085 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:54.086 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.299358+0000", 2020-01-16T00:13:54.086 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5397e-05 2020-01-16T00:13:54.086 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.086 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.086 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:54.086 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.299394+0000", 2020-01-16T00:13:54.087 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3699e-05 2020-01-16T00:13:54.087 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.087 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.087 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.087 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.299407+0000", 2020-01-16T00:13:54.087 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014281999999999999 2020-01-16T00:13:54.088 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.088 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.088 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.088 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.299550+0000", 2020-01-16T00:13:54.088 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024247099999999999 2020-01-16T00:13:54.088 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.088 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.089 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:54.089 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.299793+0000", 2020-01-16T00:13:54.089 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00031642200000000001 2020-01-16T00:13:54.089 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.089 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.089 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:54.089 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.300109+0000", 2020-01-16T00:13:54.089 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0078127560000000006 2020-01-16T00:13:54.090 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.090 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.090 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:54.090 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.307922+0000", 2020-01-16T00:13:54.090 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0024170189999999999 2020-01-16T00:13:54.090 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.091 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.091 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.091 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.310339+0000", 2020-01-16T00:13:54.091 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.4764000000000004e-05 2020-01-16T00:13:54.091 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.092 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.092 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.092 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.310424+0000", 2020-01-16T00:13:54.092 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000128319 2020-01-16T00:13:54.092 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.092 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.093 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:54.093 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.310552+0000", 2020-01-16T00:13:54.093 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.6587000000000002e-05 2020-01-16T00:13:54.093 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.093 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.093 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:54.093 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:07.310648+0000", 2020-01-16T00:13:54.094 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012139225999999999 2020-01-16T00:13:54.094 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.094 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:54.094 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.094 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.094 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.094 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2505 3.a 3:5069665e:::smithi12313934-2:head [write 1862260~479427] snapc 0=[] ondisk+write+known_if_redirected e37)", 2020-01-16T00:13:54.095 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:11.286548+0000", 2020-01-16T00:13:54.095 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 42.716136595999998, 2020-01-16T00:13:54.095 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013314061, 2020-01-16T00:13:54.095 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:54.095 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:54.095 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:54.095 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:54.095 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.096 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2505 2020-01-16T00:13:54.096 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.096 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:54.096 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.096 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:54.096 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.286548+0000", 2020-01-16T00:13:54.096 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:54.096 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.097 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.097 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:54.097 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.286548+0000", 2020-01-16T00:13:54.097 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999957 2020-01-16T00:13:54.097 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.097 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.097 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:54.098 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.286544+0000", 2020-01-16T00:13:54.098 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042043700000000001 2020-01-16T00:13:54.098 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.098 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.098 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:54.098 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.286964+0000", 2020-01-16T00:13:54.098 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5756999999999997e-05 2020-01-16T00:13:54.099 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.099 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.099 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:54.099 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.287000+0000", 2020-01-16T00:13:54.099 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8960000000000001e-05 2020-01-16T00:13:54.099 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.099 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.100 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.100 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.287019+0000", 2020-01-16T00:13:54.100 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.7219000000000005e-05 2020-01-16T00:13:54.100 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.100 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.100 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.100 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.287116+0000", 2020-01-16T00:13:54.100 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033208899999999998 2020-01-16T00:13:54.101 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.101 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.101 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.101 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.287448+0000", 2020-01-16T00:13:54.101 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0082732420000000001 2020-01-16T00:13:54.101 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.101 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.102 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.102 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.295721+0000", 2020-01-16T00:13:54.102 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017259999999999999 2020-01-16T00:13:54.102 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.102 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.102 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:54.102 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.295894+0000", 2020-01-16T00:13:54.102 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00029238000000000002 2020-01-16T00:13:54.103 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.103 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.103 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:54.103 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.296186+0000", 2020-01-16T00:13:54.103 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0034626409999999998 2020-01-16T00:13:54.103 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.103 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.104 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.104 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299649+0000", 2020-01-16T00:13:54.104 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2300000000000002e-06 2020-01-16T00:13:54.104 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.104 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.104 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:54.104 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299656+0000", 2020-01-16T00:13:54.104 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1561000000000001e-05 2020-01-16T00:13:54.105 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.105 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.105 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.105 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299738+0000", 2020-01-16T00:13:54.105 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8491000000000005e-05 2020-01-16T00:13:54.105 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.105 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.105 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:54.106 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299806+0000", 2020-01-16T00:13:54.106 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5541000000000002e-05 2020-01-16T00:13:54.106 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.106 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.106 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:54.106 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299862+0000", 2020-01-16T00:13:54.106 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013314061 2020-01-16T00:13:54.107 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.107 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:54.107 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.107 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.107 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.107 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2791 3.a 3:54f2c117:::smithi12313934-41:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e41)", 2020-01-16T00:13:54.107 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:17.213167+0000", 2020-01-16T00:13:54.107 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 36.789517584999999, 2020-01-16T00:13:54.108 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016024037000000001, 2020-01-16T00:13:54.108 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:54.108 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:54.108 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:54.108 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:54.108 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.108 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2791 2020-01-16T00:13:54.108 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.109 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:54.109 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.109 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:54.109 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213167+0000", 2020-01-16T00:13:54.109 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:54.109 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.109 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.109 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:54.110 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213167+0000", 2020-01-16T00:13:54.110 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:54.110 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.110 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.110 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:54.110 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213166+0000", 2020-01-16T00:13:54.110 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6030000000000004e-06 2020-01-16T00:13:54.111 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.111 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.111 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:54.111 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213172+0000", 2020-01-16T00:13:54.111 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0929999999999997e-06 2020-01-16T00:13:54.111 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.111 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.111 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:54.112 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213179+0000", 2020-01-16T00:13:54.112 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4299999999999997e-06 2020-01-16T00:13:54.112 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.112 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.112 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.112 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213184+0000", 2020-01-16T00:13:54.112 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012106695000000001 2020-01-16T00:13:54.113 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.113 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.113 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.113 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.225291+0000", 2020-01-16T00:13:54.113 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020863699999999999 2020-01-16T00:13:54.113 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.113 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.113 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:54.114 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.225499+0000", 2020-01-16T00:13:54.114 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0036913050000000002 2020-01-16T00:13:54.114 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.114 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.114 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:54.114 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.229191+0000", 2020-01-16T00:13:54.114 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016024037000000001 2020-01-16T00:13:54.114 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.115 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:54.115 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.115 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.115 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.115 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2792 3.a 3:54f2c117:::smithi12313934-41:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e41)", 2020-01-16T00:13:54.115 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:17.213190+0000", 2020-01-16T00:13:54.115 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 36.789494111000003, 2020-01-16T00:13:54.115 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017165949, 2020-01-16T00:13:54.116 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:54.116 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:54.116 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:54.116 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:54.116 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.116 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2792 2020-01-16T00:13:54.116 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.116 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:54.117 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.117 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:54.117 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213190+0000", 2020-01-16T00:13:54.117 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:54.117 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.117 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.117 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:54.117 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213190+0000", 2020-01-16T00:13:54.118 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:54.118 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.118 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.118 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:54.118 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213189+0000", 2020-01-16T00:13:54.118 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2699999999999998e-06 2020-01-16T00:13:54.119 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.119 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.119 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:54.119 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213194+0000", 2020-01-16T00:13:54.119 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8600000000000004e-06 2020-01-16T00:13:54.119 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.120 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.120 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:54.120 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213201+0000", 2020-01-16T00:13:54.120 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8500000000000002e-06 2020-01-16T00:13:54.120 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.121 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.121 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.121 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213205+0000", 2020-01-16T00:13:54.121 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016028924999999999 2020-01-16T00:13:54.121 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.121 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.121 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.122 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.229234+0000", 2020-01-16T00:13:54.122 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019773999999999999 2020-01-16T00:13:54.122 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.122 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.122 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:54.122 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.229432+0000", 2020-01-16T00:13:54.122 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00092414400000000001 2020-01-16T00:13:54.123 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.123 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.123 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:54.123 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.230356+0000", 2020-01-16T00:13:54.123 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017165949 2020-01-16T00:13:54.123 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.124 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:54.124 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.124 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.124 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.124 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2793 3.a 3:54f2c117:::smithi12313934-41:head [read 0~3210170,checksum] snapc 0=[] ondisk+read+known_if_redirected e41)", 2020-01-16T00:13:54.124 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:17.213211+0000", 2020-01-16T00:13:54.124 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 36.789473565000002, 2020-01-16T00:13:54.124 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019069413, 2020-01-16T00:13:54.125 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:54.125 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:54.125 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:54.125 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:54.125 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.125 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2793 2020-01-16T00:13:54.125 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.125 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:54.126 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.126 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:54.126 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213211+0000", 2020-01-16T00:13:54.126 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:54.126 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.126 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.126 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:54.127 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213211+0000", 2020-01-16T00:13:54.127 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:54.127 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.127 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.127 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:54.127 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213210+0000", 2020-01-16T00:13:54.128 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.66e-06 2020-01-16T00:13:54.128 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.128 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.128 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:54.128 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213216+0000", 2020-01-16T00:13:54.128 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.9469999999999997e-06 2020-01-16T00:13:54.128 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.128 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.129 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:54.129 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213223+0000", 2020-01-16T00:13:54.129 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9259999999999999e-06 2020-01-16T00:13:54.129 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.129 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.129 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.129 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213228+0000", 2020-01-16T00:13:54.130 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017171046999999998 2020-01-16T00:13:54.130 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.130 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.130 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.130 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.230399+0000", 2020-01-16T00:13:54.130 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000189932 2020-01-16T00:13:54.130 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.130 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.131 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:54.131 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.230589+0000", 2020-01-16T00:13:54.131 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0016916539999999999 2020-01-16T00:13:54.131 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.131 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.131 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:54.131 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.232280+0000", 2020-01-16T00:13:54.131 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019069413 2020-01-16T00:13:54.132 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.132 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:54.132 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.132 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.132 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.132 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3120 3.a 3:5069665e:::smithi12313934-2:head [write 1845146~524707] snapc 0=[] ondisk+write+known_if_redirected e44)", 2020-01-16T00:13:54.132 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.262852+0000", 2020-01-16T00:13:54.132 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 28.739832381999999, 2020-01-16T00:13:54.133 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014991175000000001, 2020-01-16T00:13:54.133 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:54.133 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:54.133 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:54.133 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:54.133 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.133 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3120 2020-01-16T00:13:54.133 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.133 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:54.134 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.134 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:54.134 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.262852+0000", 2020-01-16T00:13:54.134 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:54.134 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.134 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.134 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:54.135 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.262852+0000", 2020-01-16T00:13:54.135 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:54.135 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.135 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.135 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:54.135 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.262849+0000", 2020-01-16T00:13:54.135 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00058469900000000005 2020-01-16T00:13:54.135 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.136 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.136 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:54.136 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263434+0000", 2020-01-16T00:13:54.136 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.6650000000000001e-05 2020-01-16T00:13:54.136 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.136 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.136 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:54.136 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263461+0000", 2020-01-16T00:13:54.137 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3478e-05 2020-01-16T00:13:54.137 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.137 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.137 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.137 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263474+0000", 2020-01-16T00:13:54.137 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7824000000000002e-05 2020-01-16T00:13:54.137 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.138 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.138 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.138 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263522+0000", 2020-01-16T00:13:54.138 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001365464 2020-01-16T00:13:54.138 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.138 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.138 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.139 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.264888+0000", 2020-01-16T00:13:54.139 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0081368250000000003 2020-01-16T00:13:54.139 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.139 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.139 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.139 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.273024+0000", 2020-01-16T00:13:54.140 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6130000000000006e-05 2020-01-16T00:13:54.140 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.140 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.140 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:54.140 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.273090+0000", 2020-01-16T00:13:54.140 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000207507 2020-01-16T00:13:54.140 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.141 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.141 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:54.141 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.273298+0000", 2020-01-16T00:13:54.141 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0035649980000000002 2020-01-16T00:13:54.141 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.141 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.142 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.142 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276863+0000", 2020-01-16T00:13:54.142 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.1459999999999995e-05 2020-01-16T00:13:54.142 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.142 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.142 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.142 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276954+0000", 2020-01-16T00:13:54.143 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00076363899999999998 2020-01-16T00:13:54.143 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.143 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.143 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:54.143 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277718+0000", 2020-01-16T00:13:54.143 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0154999999999995e-05 2020-01-16T00:13:54.143 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.144 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.144 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:54.144 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277788+0000", 2020-01-16T00:13:54.144 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4914000000000003e-05 2020-01-16T00:13:54.144 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.144 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.144 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:54.144 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277843+0000", 2020-01-16T00:13:54.145 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014991175000000001 2020-01-16T00:13:54.145 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.145 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:54.145 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.145 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.145 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.145 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3121 3.a 3:5069665e:::smithi12313934-2:head [setxattr _header (55) in=7b,truncate 2369853] snapc 0=[] ondisk+write+known_if_redirected e44)", 2020-01-16T00:13:54.146 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.263494+0000", 2020-01-16T00:13:54.146 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 28.739190094000001, 2020-01-16T00:13:54.146 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014559274000000001, 2020-01-16T00:13:54.146 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:54.146 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:54.146 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:54.146 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:54.146 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.147 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3121 2020-01-16T00:13:54.147 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.147 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:54.147 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.147 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:54.147 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263494+0000", 2020-01-16T00:13:54.147 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:54.148 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.148 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.148 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:54.148 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263494+0000", 2020-01-16T00:13:54.148 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:54.148 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.148 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.148 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:54.149 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263493+0000", 2020-01-16T00:13:54.149 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3359999999999995e-06 2020-01-16T00:13:54.149 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.149 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.149 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:54.149 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263501+0000", 2020-01-16T00:13:54.149 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4100000000000002e-06 2020-01-16T00:13:54.150 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.150 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.150 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:54.150 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263509+0000", 2020-01-16T00:13:54.150 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3839999999999996e-06 2020-01-16T00:13:54.150 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.150 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.151 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.151 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263514+0000", 2020-01-16T00:13:54.151 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001430295 2020-01-16T00:13:54.151 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.151 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.151 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.151 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.264944+0000", 2020-01-16T00:13:54.152 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9710999999999999e-05 2020-01-16T00:13:54.152 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.152 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.152 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.152 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.264994+0000", 2020-01-16T00:13:54.152 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010036188999999999 2020-01-16T00:13:54.152 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.152 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.153 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.153 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.275030+0000", 2020-01-16T00:13:54.153 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1743000000000007e-05 2020-01-16T00:13:54.153 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.153 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.153 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:54.153 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.275112+0000", 2020-01-16T00:13:54.154 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013036100000000001 2020-01-16T00:13:54.154 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.154 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.154 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:54.154 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.275242+0000", 2020-01-16T00:13:54.154 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0016795549999999999 2020-01-16T00:13:54.154 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.154 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.155 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.155 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276922+0000", 2020-01-16T00:13:54.155 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018479400000000001 2020-01-16T00:13:54.155 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.155 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.155 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.155 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277107+0000", 2020-01-16T00:13:54.156 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00083585300000000001 2020-01-16T00:13:54.156 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.156 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.156 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:54.157 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277943+0000", 2020-01-16T00:13:54.157 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1719999999999999e-05 2020-01-16T00:13:54.157 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.158 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.158 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:54.158 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277994+0000", 2020-01-16T00:13:54.158 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9212999999999997e-05 2020-01-16T00:13:54.158 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.159 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.159 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:54.159 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.278054+0000", 2020-01-16T00:13:54.159 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014559274000000001 2020-01-16T00:13:54.159 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.159 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:54.159 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.159 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.160 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.160 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3122 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e44)", 2020-01-16T00:13:54.160 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.263525+0000", 2020-01-16T00:13:54.160 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 28.739158906, 2020-01-16T00:13:54.160 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014731833999999999, 2020-01-16T00:13:54.161 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:54.161 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:54.161 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:54.161 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:54.161 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.161 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3122 2020-01-16T00:13:54.161 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.161 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:54.162 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.162 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:54.162 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263525+0000", 2020-01-16T00:13:54.162 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:54.162 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.162 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.162 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:54.162 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263525+0000", 2020-01-16T00:13:54.163 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:54.163 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.163 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.163 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:54.163 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263525+0000", 2020-01-16T00:13:54.163 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5000000000000001e-06 2020-01-16T00:13:54.163 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.163 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.164 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:54.164 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263529+0000", 2020-01-16T00:13:54.164 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4669999999999998e-06 2020-01-16T00:13:54.164 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.164 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.164 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:54.164 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263537+0000", 2020-01-16T00:13:54.164 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0819999999999998e-06 2020-01-16T00:13:54.165 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.165 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.165 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.165 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263542+0000", 2020-01-16T00:13:54.165 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0015024960000000001 2020-01-16T00:13:54.165 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.165 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.166 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.166 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.265044+0000", 2020-01-16T00:13:54.166 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6650999999999994e-05 2020-01-16T00:13:54.166 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.166 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.166 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.167 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.265131+0000", 2020-01-16T00:13:54.167 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010495310000000001 2020-01-16T00:13:54.167 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.167 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.167 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.167 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.275626+0000", 2020-01-16T00:13:54.167 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9536000000000004e-05 2020-01-16T00:13:54.167 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.168 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.168 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:54.168 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.275726+0000", 2020-01-16T00:13:54.168 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023719489999999999 2020-01-16T00:13:54.168 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.168 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.168 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.169 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.278098+0000", 2020-01-16T00:13:54.169 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3718000000000001e-05 2020-01-16T00:13:54.169 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.169 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.169 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:54.170 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.278171+0000", 2020-01-16T00:13:54.170 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5859999999999994e-05 2020-01-16T00:13:54.170 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.170 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.171 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:54.171 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.278257+0000", 2020-01-16T00:13:54.171 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014731833999999999 2020-01-16T00:13:54.171 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.171 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:54.171 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.171 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.171 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.172 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3627 3.a 3:5069665e:::smithi12313934-2:head [write 1749580~611149] snapc 0=[] ondisk+write+known_if_redirected e48)", 2020-01-16T00:13:54.172 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:39.462779+0000", 2020-01-16T00:13:54.172 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.539905094, 2020-01-16T00:13:54.172 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020946985000000001, 2020-01-16T00:13:54.172 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:54.172 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:54.173 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:54.173 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:54.173 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.173 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3627 2020-01-16T00:13:54.173 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.173 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:54.174 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.174 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:54.174 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.462779+0000", 2020-01-16T00:13:54.174 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:54.174 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.175 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.175 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:54.175 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.462779+0000", 2020-01-16T00:13:54.175 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:54.175 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.175 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.176 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:54.176 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.462777+0000", 2020-01-16T00:13:54.176 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00064136 2020-01-16T00:13:54.176 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.176 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.176 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:54.176 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.463418+0000", 2020-01-16T00:13:54.177 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4607e-05 2020-01-16T00:13:54.177 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.177 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.177 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:54.177 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.463443+0000", 2020-01-16T00:13:54.177 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0346999999999999e-05 2020-01-16T00:13:54.177 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.177 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.178 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.178 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.463453+0000", 2020-01-16T00:13:54.178 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000100988 2020-01-16T00:13:54.178 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.178 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.178 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.178 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.463554+0000", 2020-01-16T00:13:54.178 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023059280000000001 2020-01-16T00:13:54.179 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.179 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.179 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.179 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.465860+0000", 2020-01-16T00:13:54.179 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011316961 2020-01-16T00:13:54.179 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.179 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.180 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.180 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.477177+0000", 2020-01-16T00:13:54.180 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000132402 2020-01-16T00:13:54.180 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.180 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.180 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:54.180 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.477309+0000", 2020-01-16T00:13:54.180 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000223606 2020-01-16T00:13:54.181 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.181 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.181 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:54.181 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.477533+0000", 2020-01-16T00:13:54.181 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0057105209999999997 2020-01-16T00:13:54.181 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.181 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.182 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:54.182 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483244+0000", 2020-01-16T00:13:54.182 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000193031 2020-01-16T00:13:54.182 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.182 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.182 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.183 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483437+0000", 2020-01-16T00:13:54.183 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0902999999999995e-05 2020-01-16T00:13:54.183 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.183 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.183 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.183 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483507+0000", 2020-01-16T00:13:54.184 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000129412 2020-01-16T00:13:54.184 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.184 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.184 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:54.184 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483637+0000", 2020-01-16T00:13:54.184 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.9345999999999995e-05 2020-01-16T00:13:54.184 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.185 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.185 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:54.185 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483726+0000", 2020-01-16T00:13:54.185 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020946985000000001 2020-01-16T00:13:54.185 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.185 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:54.185 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.186 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.186 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.186 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3628 3.a 3:5069665e:::smithi12313934-2:head [write 2853383~452450] snapc 0=[] ondisk+write+known_if_redirected e48)", 2020-01-16T00:13:54.186 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:39.468178+0000", 2020-01-16T00:13:54.187 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.534506385, 2020-01-16T00:13:54.187 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015705872999999999, 2020-01-16T00:13:54.187 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:54.187 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:54.187 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:54.187 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:54.187 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.187 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3628 2020-01-16T00:13:54.188 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.188 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:54.188 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.188 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:54.188 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468178+0000", 2020-01-16T00:13:54.188 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:54.189 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.189 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.189 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:54.189 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468178+0000", 2020-01-16T00:13:54.189 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:54.189 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.189 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.190 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:54.190 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468175+0000", 2020-01-16T00:13:54.190 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00065948000000000005 2020-01-16T00:13:54.190 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.190 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.190 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:54.190 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468834+0000", 2020-01-16T00:13:54.191 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3557000000000002e-05 2020-01-16T00:13:54.191 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.191 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.191 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:54.191 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468858+0000", 2020-01-16T00:13:54.191 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2931e-05 2020-01-16T00:13:54.191 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.192 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.192 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.192 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468871+0000", 2020-01-16T00:13:54.192 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000123113 2020-01-16T00:13:54.192 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.193 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.193 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.193 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468994+0000", 2020-01-16T00:13:54.193 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017615999999999999 2020-01-16T00:13:54.193 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.194 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.194 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.194 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.469170+0000", 2020-01-16T00:13:54.194 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010336908000000001 2020-01-16T00:13:54.194 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.194 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.194 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.194 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.479507+0000", 2020-01-16T00:13:54.195 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015856400000000001 2020-01-16T00:13:54.195 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.195 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.195 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:54.195 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.479665+0000", 2020-01-16T00:13:54.195 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030344600000000002 2020-01-16T00:13:54.195 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.196 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.196 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:54.196 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.479969+0000", 2020-01-16T00:13:54.196 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0037935389999999999 2020-01-16T00:13:54.196 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.196 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.196 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.196 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483762+0000", 2020-01-16T00:13:54.197 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3170999999999997e-05 2020-01-16T00:13:54.197 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.197 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.197 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:54.197 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483806+0000", 2020-01-16T00:13:54.197 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3821000000000001e-05 2020-01-16T00:13:54.197 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.198 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.198 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.198 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483829+0000", 2020-01-16T00:13:54.198 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0076000000000001e-05 2020-01-16T00:13:54.198 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.198 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.198 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:54.199 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483859+0000", 2020-01-16T00:13:54.199 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.438e-05 2020-01-16T00:13:54.199 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.199 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.199 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:54.199 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483884+0000", 2020-01-16T00:13:54.199 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015705872999999999 2020-01-16T00:13:54.200 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.200 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:54.200 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.200 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.200 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.200 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3629 3.a 3:5069665e:::smithi12313934-2:head [setxattr _header (55) in=7b,truncate 3529536] snapc 0=[] ondisk+write+known_if_redirected e48)", 2020-01-16T00:13:54.200 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:39.468888+0000", 2020-01-16T00:13:54.201 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.533796764, 2020-01-16T00:13:54.201 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015108511, 2020-01-16T00:13:54.201 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:54.201 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:54.201 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:54.201 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:54.201 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.202 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3629 2020-01-16T00:13:54.202 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.202 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:54.202 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.202 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:54.202 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468888+0000", 2020-01-16T00:13:54.202 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:54.203 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.203 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.203 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:54.203 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468888+0000", 2020-01-16T00:13:54.203 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:54.203 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.203 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.203 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:54.204 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468886+0000", 2020-01-16T00:13:54.204 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5599999999999996e-06 2020-01-16T00:13:54.204 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.204 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.204 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:54.204 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468894+0000", 2020-01-16T00:13:54.204 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5279999999999998e-06 2020-01-16T00:13:54.204 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.205 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.205 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:54.205 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468901+0000", 2020-01-16T00:13:54.205 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2290000000000002e-06 2020-01-16T00:13:54.205 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.205 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.205 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.205 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468907+0000", 2020-01-16T00:13:54.206 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030197500000000001 2020-01-16T00:13:54.206 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.206 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.206 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.206 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.469209+0000", 2020-01-16T00:13:54.206 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6659999999999997e-05 2020-01-16T00:13:54.206 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.207 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.207 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.207 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.469255+0000", 2020-01-16T00:13:54.207 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013117904999999999 2020-01-16T00:13:54.207 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.207 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.207 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.207 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.482373+0000", 2020-01-16T00:13:54.208 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1260000000000006e-05 2020-01-16T00:13:54.208 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.208 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.208 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:54.208 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.482434+0000", 2020-01-16T00:13:54.208 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014433 2020-01-16T00:13:54.208 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.208 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.209 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:54.209 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.482579+0000", 2020-01-16T00:13:54.209 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012198160000000001 2020-01-16T00:13:54.209 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.209 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.209 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.209 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483799+0000", 2020-01-16T00:13:54.209 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.188e-05 2020-01-16T00:13:54.210 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.210 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.210 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:54.210 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483810+0000", 2020-01-16T00:13:54.210 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000114863 2020-01-16T00:13:54.210 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.210 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.211 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.211 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483925+0000", 2020-01-16T00:13:54.211 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0861000000000001e-05 2020-01-16T00:13:54.211 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.211 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.211 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:54.211 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483956+0000", 2020-01-16T00:13:54.212 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9894999999999997e-05 2020-01-16T00:13:54.212 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.212 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.212 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:54.212 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483996+0000", 2020-01-16T00:13:54.212 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015108511 2020-01-16T00:13:54.212 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.212 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:54.213 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.213 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.213 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.213 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3630 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e48)", 2020-01-16T00:13:54.213 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:39.468967+0000", 2020-01-16T00:13:54.213 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.53371763, 2020-01-16T00:13:54.213 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015160579, 2020-01-16T00:13:54.214 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:54.214 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:54.214 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:54.214 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:54.214 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.214 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3630 2020-01-16T00:13:54.214 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.214 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:54.215 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.215 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:54.215 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468967+0000", 2020-01-16T00:13:54.215 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:54.215 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.215 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.215 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:54.216 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468967+0000", 2020-01-16T00:13:54.216 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:13:54.216 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.216 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.216 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:54.216 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468966+0000", 2020-01-16T00:13:54.216 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3430000000000003e-06 2020-01-16T00:13:54.216 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.217 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.217 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:54.217 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468971+0000", 2020-01-16T00:13:54.217 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6577000000000001e-05 2020-01-16T00:13:54.217 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.217 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.217 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:54.218 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468988+0000", 2020-01-16T00:13:54.218 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.378e-06 2020-01-16T00:13:54.218 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.218 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.218 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.218 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468993+0000", 2020-01-16T00:13:54.218 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000320601 2020-01-16T00:13:54.219 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.219 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.219 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.219 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.469314+0000", 2020-01-16T00:13:54.219 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013833699999999999 2020-01-16T00:13:54.219 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.219 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.220 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.220 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.469452+0000", 2020-01-16T00:13:54.220 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013447838 2020-01-16T00:13:54.220 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.220 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.220 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.220 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.482900+0000", 2020-01-16T00:13:54.221 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028006899999999998 2020-01-16T00:13:54.221 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.221 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.221 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:54.221 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483180+0000", 2020-01-16T00:13:54.221 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000833594 2020-01-16T00:13:54.221 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.221 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.222 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.222 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.484013+0000", 2020-01-16T00:13:54.222 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2601e-05 2020-01-16T00:13:54.222 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.222 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.222 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:54.222 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.484066+0000", 2020-01-16T00:13:54.223 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1303999999999994e-05 2020-01-16T00:13:54.223 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.223 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.223 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:54.223 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.484127+0000", 2020-01-16T00:13:54.223 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015160579 2020-01-16T00:13:54.223 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.225 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:54.225 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.225 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.225 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.226 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4047 3.a 3:5069665e:::smithi12313934-2:head [write 1935828~635840] snapc 0=[] ondisk+write+known_if_redirected e52)", 2020-01-16T00:13:54.226 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:50.302087+0000", 2020-01-16T00:13:54.226 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 3.7005973590000001, 2020-01-16T00:13:54.226 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014518982, 2020-01-16T00:13:54.226 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:54.226 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:54.226 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:54.227 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:54.227 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.227 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4047 2020-01-16T00:13:54.227 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.227 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:54.227 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.227 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:54.227 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302087+0000", 2020-01-16T00:13:54.228 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:54.228 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.228 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.228 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:54.228 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302087+0000", 2020-01-16T00:13:54.228 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:54.228 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.228 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.229 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:54.229 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302085+0000", 2020-01-16T00:13:54.229 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00057147600000000002 2020-01-16T00:13:54.229 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.229 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.229 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:54.229 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302657+0000", 2020-01-16T00:13:54.230 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6427999999999999e-05 2020-01-16T00:13:54.230 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.230 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.230 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:54.230 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302673+0000", 2020-01-16T00:13:54.230 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.127e-06 2020-01-16T00:13:54.231 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.231 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.231 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.231 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302680+0000", 2020-01-16T00:13:54.231 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6378000000000006e-05 2020-01-16T00:13:54.231 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.231 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.231 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.232 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302756+0000", 2020-01-16T00:13:54.232 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00039898000000000001 2020-01-16T00:13:54.232 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.232 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.232 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.232 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.303155+0000", 2020-01-16T00:13:54.232 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0079666870000000001 2020-01-16T00:13:54.232 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.233 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.233 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.233 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.311122+0000", 2020-01-16T00:13:54.233 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019101400000000001 2020-01-16T00:13:54.233 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.233 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.233 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:54.233 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.311313+0000", 2020-01-16T00:13:54.234 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033332499999999998 2020-01-16T00:13:54.234 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.234 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.234 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:54.234 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.311646+0000", 2020-01-16T00:13:54.234 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0029841440000000002 2020-01-16T00:13:54.234 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.235 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.235 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.235 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.314630+0000", 2020-01-16T00:13:54.235 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00027996800000000001 2020-01-16T00:13:54.235 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.235 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.235 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.235 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.314910+0000", 2020-01-16T00:13:54.236 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0015637159999999999 2020-01-16T00:13:54.236 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.236 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.236 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:54.236 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.316474+0000", 2020-01-16T00:13:54.236 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5986000000000001e-05 2020-01-16T00:13:54.236 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.236 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.237 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:54.237 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.316550+0000", 2020-01-16T00:13:54.237 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6258000000000002e-05 2020-01-16T00:13:54.237 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.237 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.237 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:54.237 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.316606+0000", 2020-01-16T00:13:54.237 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014518982 2020-01-16T00:13:54.238 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.238 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:54.238 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.238 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.238 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.238 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4048 3.a 3:5069665e:::smithi12313934-2:head [setxattr _header (55) in=7b,truncate 2588377] snapc 0=[] ondisk+write+known_if_redirected e52)", 2020-01-16T00:13:54.238 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:50.302688+0000", 2020-01-16T00:13:54.238 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 3.6999962470000001, 2020-01-16T00:13:54.239 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014049927, 2020-01-16T00:13:54.239 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:54.239 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:54.239 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:54.239 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:54.239 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.239 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4048 2020-01-16T00:13:54.239 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.240 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:54.240 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.240 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:54.240 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302688+0000", 2020-01-16T00:13:54.240 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:54.240 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.241 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.241 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:54.241 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302688+0000", 2020-01-16T00:13:54.241 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:54.241 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.241 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.241 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:54.242 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302687+0000", 2020-01-16T00:13:54.242 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6e-06 2020-01-16T00:13:54.242 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.242 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.242 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:54.242 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302692+0000", 2020-01-16T00:13:54.242 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1999999999999996e-06 2020-01-16T00:13:54.242 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.243 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.243 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:54.243 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302696+0000", 2020-01-16T00:13:54.243 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9330000000000001e-06 2020-01-16T00:13:54.243 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.243 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.243 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.243 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302699+0000", 2020-01-16T00:13:54.244 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00049592099999999997 2020-01-16T00:13:54.244 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.244 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.244 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.244 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.303195+0000", 2020-01-16T00:13:54.244 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0647000000000003e-05 2020-01-16T00:13:54.244 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.245 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.245 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.245 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.303266+0000", 2020-01-16T00:13:54.245 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010088420000000001 2020-01-16T00:13:54.245 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.245 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.245 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.245 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.313354+0000", 2020-01-16T00:13:54.246 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000113136 2020-01-16T00:13:54.246 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.246 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.246 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:54.246 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.313467+0000", 2020-01-16T00:13:54.246 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012389000000000001 2020-01-16T00:13:54.246 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.246 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.247 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:13:54.247 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.313591+0000", 2020-01-16T00:13:54.247 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014897409999999999 2020-01-16T00:13:54.247 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.247 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.247 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.247 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.315081+0000", 2020-01-16T00:13:54.247 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00023842700000000001 2020-01-16T00:13:54.248 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.248 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.248 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.248 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.315319+0000", 2020-01-16T00:13:54.248 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012958449999999999 2020-01-16T00:13:54.248 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.248 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.248 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:13:54.249 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.316615+0000", 2020-01-16T00:13:54.249 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4045000000000002e-05 2020-01-16T00:13:54.249 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.249 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.249 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:13:54.249 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.316669+0000", 2020-01-16T00:13:54.249 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8827999999999999e-05 2020-01-16T00:13:54.250 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.250 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.250 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:54.250 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.316738+0000", 2020-01-16T00:13:54.250 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014049927 2020-01-16T00:13:54.250 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.250 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:54.251 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.251 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.251 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.251 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4049 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e52)", 2020-01-16T00:13:54.251 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:50.302702+0000", 2020-01-16T00:13:54.251 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 3.699982726, 2020-01-16T00:13:54.251 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014215574999999999, 2020-01-16T00:13:54.252 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:13:54.252 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:13:54.252 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:13:54.252 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:13:54.252 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.252 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4049 2020-01-16T00:13:54.252 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.252 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:13:54.253 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.253 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:13:54.253 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302702+0000", 2020-01-16T00:13:54.253 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:13:54.253 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.253 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.253 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:13:54.254 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302702+0000", 2020-01-16T00:13:54.254 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:54.254 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.254 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.254 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:13:54.254 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302701+0000", 2020-01-16T00:13:54.254 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0020000000000001e-06 2020-01-16T00:13:54.254 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.255 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.255 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:13:54.255 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302703+0000", 2020-01-16T00:13:54.255 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7740000000000002e-06 2020-01-16T00:13:54.255 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.255 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.255 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:13:54.255 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302707+0000", 2020-01-16T00:13:54.256 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.512e-06 2020-01-16T00:13:54.256 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.256 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.256 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.256 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302711+0000", 2020-01-16T00:13:54.256 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00058642799999999995 2020-01-16T00:13:54.256 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.256 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.257 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.257 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.303297+0000", 2020-01-16T00:13:54.257 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5887000000000001e-05 2020-01-16T00:13:54.257 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.257 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.257 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.257 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.303363+0000", 2020-01-16T00:13:54.257 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010875398999999999 2020-01-16T00:13:54.258 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.258 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.258 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.258 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.314238+0000", 2020-01-16T00:13:54.258 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000187014 2020-01-16T00:13:54.258 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.258 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.258 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:54.259 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.314425+0000", 2020-01-16T00:13:54.259 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023405000000000001 2020-01-16T00:13:54.259 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.259 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.259 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:13:54.259 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.316766+0000", 2020-01-16T00:13:54.259 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0636000000000006e-05 2020-01-16T00:13:54.260 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.260 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.260 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:13:54.260 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.316836+0000", 2020-01-16T00:13:54.260 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0747999999999993e-05 2020-01-16T00:13:54.260 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:13:54.260 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:13:54.260 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:13:54.261 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.316917+0000", 2020-01-16T00:13:54.261 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014215574999999999 2020-01-16T00:13:54.261 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.261 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:54.261 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.261 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:13:54.262 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:13:54.262 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:13:54.262 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2020-01-16T00:13:54.266 INFO:tasks.rados.rados.0.smithi123.stdout:1541: writing smithi12313934-49 from 440482 to 849813 tid 1 2020-01-16T00:13:54.266 INFO:tasks.rados.rados.0.smithi123.stdout:1541: writing smithi12313934-49 from 1380352 to 1785533 tid 2 2020-01-16T00:13:54.266 INFO:tasks.rados.rados.0.smithi123.stdout:1541: writing smithi12313934-49 from 2461553 to 2840482 tid 3 2020-01-16T00:13:54.266 INFO:tasks.rados.rados.0.smithi123.stdout:1542: read oid 20 snap -1 2020-01-16T00:13:54.266 INFO:tasks.rados.rados.0.smithi123.stdout:1542: expect deleted 2020-01-16T00:13:54.266 INFO:tasks.rados.rados.0.smithi123.stdout:1543: read oid 43 snap -1 2020-01-16T00:13:54.266 INFO:tasks.rados.rados.0.smithi123.stdout:1543: expect (ObjNum 490 snap 0 seq_num 490) 2020-01-16T00:13:54.267 INFO:tasks.rados.rados.0.smithi123.stdout:1544: write oid 26 current snap is 0 2020-01-16T00:13:54.267 INFO:tasks.rados.rados.0.smithi123.stdout:1544: seq_num 527 ranges {457289=484580,1438834=548839,2772623=84666} 2020-01-16T00:13:54.267 INFO:tasks.rados.rados.0.smithi123.stdout:1544: writing smithi12313934-26 from 457289 to 941869 tid 1 2020-01-16T00:13:54.267 INFO:tasks.rados.rados.0.smithi123.stdout:1544: writing smithi12313934-26 from 1438834 to 1987673 tid 2 2020-01-16T00:13:54.267 INFO:tasks.rados.rados.0.smithi123.stdout:1544: writing smithi12313934-26 from 2772623 to 2857289 tid 3 2020-01-16T00:13:54.267 INFO:tasks.rados.rados.0.smithi123.stdout:1545: read oid 12 snap -1 2020-01-16T00:13:54.267 INFO:tasks.rados.rados.0.smithi123.stdout:1545: expect (ObjNum 517 snap 0 seq_num 517) 2020-01-16T00:13:54.267 INFO:tasks.rados.rados.0.smithi123.stdout:1546: read oid 31 snap -1 2020-01-16T00:13:54.268 INFO:tasks.rados.rados.0.smithi123.stdout:1546: expect (ObjNum 522 snap 0 seq_num 522) 2020-01-16T00:13:54.268 INFO:tasks.rados.rados.0.smithi123.stdout:1547: write oid 48 current snap is 0 2020-01-16T00:13:54.268 INFO:tasks.rados.rados.0.smithi123.stdout:1547: seq_num 528 ranges {474096=559829,1780963=408850,2800046=74050} 2020-01-16T00:13:54.268 INFO:tasks.rados.rados.0.smithi123.stdout:1547: writing smithi12313934-48 from 474096 to 1033925 tid 1 2020-01-16T00:13:54.268 INFO:tasks.rados.rados.0.smithi123.stdout:1547: writing smithi12313934-48 from 1780963 to 2189813 tid 2 2020-01-16T00:13:54.268 INFO:tasks.rados.rados.0.smithi123.stdout:1547: writing smithi12313934-48 from 2800046 to 2874096 tid 3 2020-01-16T00:13:54.268 INFO:tasks.rados.rados.0.smithi123.stdout:1548: read oid 15 snap -1 2020-01-16T00:13:54.269 INFO:tasks.rados.rados.0.smithi123.stdout:1548: expect (ObjNum 514 snap 0 seq_num 514) 2020-01-16T00:13:54.269 INFO:tasks.rados.rados.0.smithi123.stdout:1549: write oid 45 current snap is 0 2020-01-16T00:13:54.269 INFO:tasks.rados.rados.0.smithi123.stdout:1549: seq_num 529 ranges {490903=635078,1839445=552508} 2020-01-16T00:13:54.269 INFO:tasks.rados.rados.0.smithi123.stdout:1549: writing smithi12313934-45 from 490903 to 1125981 tid 1 2020-01-16T00:13:54.269 INFO:tasks.rados.rados.0.smithi123.stdout:1549: writing smithi12313934-45 from 1839445 to 2391953 tid 2 2020-01-16T00:13:54.269 INFO:tasks.rados.rados.0.smithi123.stdout:1550: delete oid 38 current snap is 0 2020-01-16T00:13:54.269 INFO:tasks.rados.rados.0.smithi123.stdout:1535: expect (ObjNum 387 snap 0 seq_num 387) 2020-01-16T00:13:54.269 INFO:tasks.rados.rados.0.smithi123.stdout:1533: expect (ObjNum 491 snap 0 seq_num 491) 2020-01-16T00:13:54.270 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:54.260+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:54.270 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-01-16T00:13:54.292 INFO:tasks.rados.rados.0.smithi123.stdout:1536: finishing write tid 1 to smithi12313934-5 2020-01-16T00:13:54.292 INFO:tasks.rados.rados.0.smithi123.stdout:1536: finishing write tid 2 to smithi12313934-5 2020-01-16T00:13:54.293 INFO:tasks.rados.rados.0.smithi123.stdout:1536: finishing write tid 4 to smithi12313934-5 2020-01-16T00:13:54.293 INFO:tasks.rados.rados.0.smithi123.stdout:1536: finishing write tid 5 to smithi12313934-5 2020-01-16T00:13:54.293 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 5 v 231 (ObjNum 523 snap 0 seq_num 523) dirty exists 2020-01-16T00:13:54.293 INFO:tasks.rados.rados.0.smithi123.stdout:1536: left oid 5 (ObjNum 523 snap 0 seq_num 523) 2020-01-16T00:13:54.293 INFO:tasks.rados.rados.0.smithi123.stdout:1537: finishing write tid 1 to smithi12313934-40 2020-01-16T00:13:54.293 INFO:tasks.rados.rados.0.smithi123.stdout:1533: done (15 left) 2020-01-16T00:13:54.293 INFO:tasks.rados.rados.0.smithi123.stdout:1535: done (14 left) 2020-01-16T00:13:54.294 INFO:tasks.rados.rados.0.smithi123.stdout:1536: done (13 left) 2020-01-16T00:13:54.294 INFO:tasks.rados.rados.0.smithi123.stdout:1550: done (12 left) 2020-01-16T00:13:54.294 INFO:tasks.rados.rados.0.smithi123.stdout:1551: read oid 5 snap -1 2020-01-16T00:13:54.294 INFO:tasks.rados.rados.0.smithi123.stdout:1551: expect (ObjNum 523 snap 0 seq_num 523) 2020-01-16T00:13:54.294 INFO:tasks.rados.rados.0.smithi123.stdout:1552: read oid 19 snap -1 2020-01-16T00:13:54.294 INFO:tasks.rados.rados.0.smithi123.stdout:1552: expect (ObjNum 520 snap 0 seq_num 520) 2020-01-16T00:13:54.294 INFO:tasks.rados.rados.0.smithi123.stdout:1553: write oid 13 current snap is 0 2020-01-16T00:13:54.294 INFO:tasks.rados.rados.0.smithi123.stdout:1553: seq_num 531 ranges {524517=785576,1956409=556177} 2020-01-16T00:13:54.303 INFO:tasks.rados.rados.0.smithi123.stdout:1553: writing smithi12313934-13 from 524517 to 1310093 tid 1 2020-01-16T00:13:54.312 INFO:tasks.rados.rados.0.smithi123.stdout:1553: writing smithi12313934-13 from 1956409 to 2512586 tid 2 2020-01-16T00:13:54.312 INFO:tasks.rados.rados.0.smithi123.stdout:1537: finishing write tid 2 to smithi12313934-40 2020-01-16T00:13:54.312 INFO:tasks.rados.rados.0.smithi123.stdout:1538: expect (ObjNum 393 snap 0 seq_num 393) 2020-01-16T00:13:54.364 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:54.362+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:54.379 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:54.379 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:54.380 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:13:54.380 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:54.390 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-01-16T00:13:54.393 INFO:tasks.rados.rados.0.smithi123.stdout:1537: finishing write tid 3 to smithi12313934-40 2020-01-16T00:13:54.393 INFO:tasks.rados.rados.0.smithi123.stdout:1537: finishing write tid 5 to smithi12313934-40 2020-01-16T00:13:54.393 INFO:tasks.rados.rados.0.smithi123.stdout:1537: finishing write tid 6 to smithi12313934-40 2020-01-16T00:13:54.393 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 40 v 239 (ObjNum 524 snap 0 seq_num 524) dirty exists 2020-01-16T00:13:54.393 INFO:tasks.rados.rados.0.smithi123.stdout:1537: left oid 40 (ObjNum 524 snap 0 seq_num 524) 2020-01-16T00:13:54.393 INFO:tasks.rados.rados.0.smithi123.stdout:1554: read oid 11 snap -1 2020-01-16T00:13:54.393 INFO:tasks.rados.rados.0.smithi123.stdout:1554: expect (ObjNum 497 snap 0 seq_num 497) 2020-01-16T00:13:54.394 INFO:tasks.rados.rados.0.smithi123.stdout:1537: done (15 left) 2020-01-16T00:13:54.394 INFO:tasks.rados.rados.0.smithi123.stdout:1538: done (14 left) 2020-01-16T00:13:54.394 INFO:tasks.rados.rados.0.smithi123.stdout:1555: write (excl) oid 34 current snap is 0 2020-01-16T00:13:54.394 INFO:tasks.rados.rados.0.smithi123.stdout:1555: seq_num 532 ranges {541324=460825,1498538=416188,2393385=547939} 2020-01-16T00:13:54.398 INFO:tasks.rados.rados.0.smithi123.stdout:1555: writing smithi12313934-34 from 541324 to 1002149 tid 1 2020-01-16T00:13:54.404 INFO:tasks.rados.rados.0.smithi123.stdout:1555: writing smithi12313934-34 from 1498538 to 1914726 tid 2 2020-01-16T00:13:54.411 INFO:tasks.rados.rados.0.smithi123.stdout:1555: writing smithi12313934-34 from 2393385 to 2941324 tid 3 2020-01-16T00:13:54.411 INFO:tasks.rados.rados.0.smithi123.stdout:1556: read oid 42 snap -1 2020-01-16T00:13:54.412 INFO:tasks.rados.rados.0.smithi123.stdout:1556: expect (ObjNum 432 snap 0 seq_num 432) 2020-01-16T00:13:54.412 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:54.412 INFO:tasks.rados.rados.0.smithi123.stdout:1540: expect (ObjNum 431 snap 0 seq_num 431) 2020-01-16T00:13:54.464 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:54.462+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:54.501 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:54.501 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:54.501 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:13:54.501 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:13:54.502 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:54.502 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-01-16T00:13:54.587 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:54.586+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:54.610 INFO:tasks.rados.rados.0.smithi123.stdout:1541: finishing write tid 1 to smithi12313934-49 2020-01-16T00:13:54.610 INFO:tasks.rados.rados.0.smithi123.stdout:1541: finishing write tid 2 to smithi12313934-49 2020-01-16T00:13:54.610 INFO:tasks.rados.rados.0.smithi123.stdout:1541: finishing write tid 3 to smithi12313934-49 2020-01-16T00:13:54.610 INFO:tasks.rados.rados.0.smithi123.stdout:1543: expect (ObjNum 490 snap 0 seq_num 490) 2020-01-16T00:13:54.647 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:54.647 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:13:54.647 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:13:54.648 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:13:54.648 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.648 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:939 3.4 3:2bdd9e88:::smithi12313934-4:head [write 1681102~474010] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:54.648 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.908392+0000", 2020-01-16T00:13:54.648 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 76.734558978999999, 2020-01-16T00:13:54.648 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019294697, 2020-01-16T00:13:54.649 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:54.649 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:54.649 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:54.649 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:54.649 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.649 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 939 2020-01-16T00:13:54.649 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.649 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:54.650 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.650 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:54.650 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908392+0000", 2020-01-16T00:13:54.650 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:54.650 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.651 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.651 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:54.651 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908392+0000", 2020-01-16T00:13:54.651 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:54.651 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.651 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.651 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:54.652 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908390+0000", 2020-01-16T00:13:54.652 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015440399999999999 2020-01-16T00:13:54.652 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.652 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.652 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:54.652 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908545+0000", 2020-01-16T00:13:54.652 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3926e-05 2020-01-16T00:13:54.652 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.653 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.653 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:54.653 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908559+0000", 2020-01-16T00:13:54.653 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2200000000000003e-06 2020-01-16T00:13:54.653 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.653 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.653 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.654 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908566+0000", 2020-01-16T00:13:54.654 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020594799999999998 2020-01-16T00:13:54.654 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.654 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.654 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.654 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910625+0000", 2020-01-16T00:13:54.655 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118672 2020-01-16T00:13:54.655 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.655 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.655 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:54.655 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910744+0000", 2020-01-16T00:13:54.655 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031940969999999999 2020-01-16T00:13:54.655 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.655 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.656 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.656 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.913938+0000", 2020-01-16T00:13:54.656 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000309621 2020-01-16T00:13:54.656 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.656 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.656 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.656 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.914248+0000", 2020-01-16T00:13:54.656 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.008818579 2020-01-16T00:13:54.657 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.657 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.657 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.657 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.923066+0000", 2020-01-16T00:13:54.657 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000108989 2020-01-16T00:13:54.657 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.657 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.658 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:54.658 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.923175+0000", 2020-01-16T00:13:54.658 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019128299999999999 2020-01-16T00:13:54.658 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.658 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.658 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:13:54.658 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.923367+0000", 2020-01-16T00:13:54.658 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0032083760000000002 2020-01-16T00:13:54.659 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.659 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.659 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:54.659 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926575+0000", 2020-01-16T00:13:54.659 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2475999999999999e-05 2020-01-16T00:13:54.659 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.659 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.659 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.660 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926607+0000", 2020-01-16T00:13:54.660 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00099399800000000006 2020-01-16T00:13:54.660 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.660 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.660 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.660 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927601+0000", 2020-01-16T00:13:54.660 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0958999999999999e-05 2020-01-16T00:13:54.660 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.661 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.661 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:54.661 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927642+0000", 2020-01-16T00:13:54.661 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4747999999999999e-05 2020-01-16T00:13:54.661 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.661 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.661 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:54.662 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927687+0000", 2020-01-16T00:13:54.662 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019294697 2020-01-16T00:13:54.662 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.662 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:54.662 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.662 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.662 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.662 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:940 3.4 3:2bdd9e88:::smithi12313934-4:head [write 2687912~589420] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:54.663 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.915144+0000", 2020-01-16T00:13:54.663 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 76.727807611000003, 2020-01-16T00:13:54.663 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016092538999999999, 2020-01-16T00:13:54.663 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:54.663 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:54.663 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:54.663 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:54.664 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.664 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 940 2020-01-16T00:13:54.664 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.664 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:54.664 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.664 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:54.665 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915144+0000", 2020-01-16T00:13:54.665 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:54.665 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.665 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.665 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:54.665 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915144+0000", 2020-01-16T00:13:54.666 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:54.666 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.666 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.666 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:54.666 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915141+0000", 2020-01-16T00:13:54.666 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016366300000000001 2020-01-16T00:13:54.666 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.666 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.667 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:54.667 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915305+0000", 2020-01-16T00:13:54.667 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4847e-05 2020-01-16T00:13:54.667 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.667 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.667 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:54.667 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915320+0000", 2020-01-16T00:13:54.667 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2599999999999999e-06 2020-01-16T00:13:54.668 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.668 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.668 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.668 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915327+0000", 2020-01-16T00:13:54.668 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4282000000000005e-05 2020-01-16T00:13:54.668 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.668 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.668 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.669 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915401+0000", 2020-01-16T00:13:54.669 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4554000000000006e-05 2020-01-16T00:13:54.669 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.669 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.669 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.669 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915466+0000", 2020-01-16T00:13:54.669 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0090821900000000004 2020-01-16T00:13:54.669 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.670 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.670 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.670 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.924548+0000", 2020-01-16T00:13:54.670 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000114696 2020-01-16T00:13:54.670 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.670 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.670 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:54.670 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.924663+0000", 2020-01-16T00:13:54.671 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000168651 2020-01-16T00:13:54.671 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.671 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.671 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:13:54.671 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.924832+0000", 2020-01-16T00:13:54.671 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018253430000000001 2020-01-16T00:13:54.671 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.671 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.672 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:54.672 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926657+0000", 2020-01-16T00:13:54.672 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9957999999999999e-05 2020-01-16T00:13:54.672 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.672 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.672 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.672 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926697+0000", 2020-01-16T00:13:54.672 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0044685250000000001 2020-01-16T00:13:54.673 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.673 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.673 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.673 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931165+0000", 2020-01-16T00:13:54.673 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9211000000000001e-05 2020-01-16T00:13:54.673 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.673 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.674 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:54.674 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931205+0000", 2020-01-16T00:13:54.674 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1699999999999998e-05 2020-01-16T00:13:54.674 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.674 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.674 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:54.674 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931236+0000", 2020-01-16T00:13:54.675 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016092538999999999 2020-01-16T00:13:54.675 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.675 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:54.675 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.675 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.676 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.676 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:941 3.4 3:2bdd9e88:::smithi12313934-4:head [setxattr _header (55) in=7b,truncate 3277332] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:54.676 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.915341+0000", 2020-01-16T00:13:54.676 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 76.727610792999997, 2020-01-16T00:13:54.676 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016603769000000001, 2020-01-16T00:13:54.676 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:54.676 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:54.677 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:54.677 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:54.677 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.677 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 941 2020-01-16T00:13:54.677 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.677 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:54.677 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.677 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:54.678 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915341+0000", 2020-01-16T00:13:54.678 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:54.678 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.678 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.678 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:54.678 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915341+0000", 2020-01-16T00:13:54.678 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:54.678 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.679 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.679 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:54.679 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915340+0000", 2020-01-16T00:13:54.679 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.896e-06 2020-01-16T00:13:54.679 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.679 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.679 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:54.679 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915346+0000", 2020-01-16T00:13:54.680 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2730000000000002e-06 2020-01-16T00:13:54.680 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.680 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.680 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:54.680 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915354+0000", 2020-01-16T00:13:54.680 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0616e-05 2020-01-16T00:13:54.680 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.680 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.681 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.681 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915374+0000", 2020-01-16T00:13:54.681 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000133487 2020-01-16T00:13:54.681 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.681 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.681 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.681 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915508+0000", 2020-01-16T00:13:54.682 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8971000000000003e-05 2020-01-16T00:13:54.682 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.682 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.682 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.682 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915557+0000", 2020-01-16T00:13:54.682 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010138787999999999 2020-01-16T00:13:54.682 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.682 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.683 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.683 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.925696+0000", 2020-01-16T00:13:54.683 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012825400000000001 2020-01-16T00:13:54.683 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.683 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.683 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:54.683 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.925824+0000", 2020-01-16T00:13:54.683 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017706500000000001 2020-01-16T00:13:54.684 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.684 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.684 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:13:54.684 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926001+0000", 2020-01-16T00:13:54.684 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00079988900000000003 2020-01-16T00:13:54.684 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.684 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.684 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.685 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926801+0000", 2020-01-16T00:13:54.685 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119324 2020-01-16T00:13:54.685 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.685 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.685 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:54.685 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926920+0000", 2020-01-16T00:13:54.686 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048879910000000004 2020-01-16T00:13:54.686 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.686 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.686 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.686 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931808+0000", 2020-01-16T00:13:54.686 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6999999999999997e-05 2020-01-16T00:13:54.686 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.686 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.687 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:54.687 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931855+0000", 2020-01-16T00:13:54.687 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9092999999999996e-05 2020-01-16T00:13:54.687 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.687 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.687 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:54.687 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931944+0000", 2020-01-16T00:13:54.687 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016603769000000001 2020-01-16T00:13:54.688 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.688 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:54.688 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.688 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.688 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.688 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:942 3.4 3:2bdd9e88:::smithi12313934-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:13:54.688 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.915389+0000", 2020-01-16T00:13:54.689 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 76.727562426999995, 2020-01-16T00:13:54.689 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016820563, 2020-01-16T00:13:54.689 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:54.689 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:54.689 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:54.689 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:54.689 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.689 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 942 2020-01-16T00:13:54.690 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.690 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:54.690 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.690 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:54.690 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915389+0000", 2020-01-16T00:13:54.690 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:54.690 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.690 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.691 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:54.691 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915389+0000", 2020-01-16T00:13:54.691 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:54.691 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.691 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.691 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:54.691 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915388+0000", 2020-01-16T00:13:54.691 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.683e-06 2020-01-16T00:13:54.692 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.692 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.692 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:54.692 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915391+0000", 2020-01-16T00:13:54.692 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4839999999999998e-06 2020-01-16T00:13:54.692 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.692 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.692 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:54.693 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915399+0000", 2020-01-16T00:13:54.693 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2259999999999999e-06 2020-01-16T00:13:54.693 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.693 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.693 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.693 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915403+0000", 2020-01-16T00:13:54.693 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00031301199999999999 2020-01-16T00:13:54.693 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.694 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.694 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.694 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915716+0000", 2020-01-16T00:13:54.694 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2079999999999998e-05 2020-01-16T00:13:54.694 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.694 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.694 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.694 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915748+0000", 2020-01-16T00:13:54.695 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010547076000000001 2020-01-16T00:13:54.695 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.695 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.695 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.695 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926295+0000", 2020-01-16T00:13:54.695 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8040000000000005e-05 2020-01-16T00:13:54.696 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.696 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.696 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:54.696 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926373+0000", 2020-01-16T00:13:54.696 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056061990000000001 2020-01-16T00:13:54.696 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.696 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.697 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.697 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931979+0000", 2020-01-16T00:13:54.697 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011540399999999999 2020-01-16T00:13:54.697 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.697 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.697 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:54.697 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932095+0000", 2020-01-16T00:13:54.697 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011482899999999999 2020-01-16T00:13:54.698 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.698 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.698 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:54.698 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932210+0000", 2020-01-16T00:13:54.698 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016820563 2020-01-16T00:13:54.698 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.698 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:54.698 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.699 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.699 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.699 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:957 3.0 3:00c146d5:::smithi12313934-18:head [write 510946~632952] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:54.699 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.033290+0000", 2020-01-16T00:13:54.699 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 76.609661590000002, 2020-01-16T00:13:54.699 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016764102999999999, 2020-01-16T00:13:54.699 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:54.699 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:54.700 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:54.700 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:54.700 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.700 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 957 2020-01-16T00:13:54.700 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.700 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:54.700 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.701 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:54.701 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033290+0000", 2020-01-16T00:13:54.701 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:54.701 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.701 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.701 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:54.701 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033290+0000", 2020-01-16T00:13:54.701 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999952 2020-01-16T00:13:54.702 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.702 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.702 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:54.702 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033285+0000", 2020-01-16T00:13:54.702 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000405917 2020-01-16T00:13:54.702 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.702 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.702 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:54.703 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033691+0000", 2020-01-16T00:13:54.703 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6585999999999997e-05 2020-01-16T00:13:54.703 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.703 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.703 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:54.703 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033738+0000", 2020-01-16T00:13:54.703 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8374000000000001e-05 2020-01-16T00:13:54.703 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.704 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.704 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.704 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033756+0000", 2020-01-16T00:13:54.704 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013002900000000001 2020-01-16T00:13:54.704 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.704 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.704 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.704 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033886+0000", 2020-01-16T00:13:54.705 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035882600000000002 2020-01-16T00:13:54.705 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.705 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.705 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.705 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.034245+0000", 2020-01-16T00:13:54.705 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0072798489999999997 2020-01-16T00:13:54.706 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.706 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.706 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.706 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.041525+0000", 2020-01-16T00:13:54.706 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000140488 2020-01-16T00:13:54.706 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.706 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.706 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:54.707 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.041665+0000", 2020-01-16T00:13:54.707 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000233766 2020-01-16T00:13:54.707 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.707 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.707 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:54.707 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.041899+0000", 2020-01-16T00:13:54.707 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0035766539999999999 2020-01-16T00:13:54.707 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.708 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.708 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:54.708 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.045475+0000", 2020-01-16T00:13:54.708 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013333150000000001 2020-01-16T00:13:54.708 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.708 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.708 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.708 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.046809+0000", 2020-01-16T00:13:54.709 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031447860000000001 2020-01-16T00:13:54.709 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.709 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.709 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.709 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.049954+0000", 2020-01-16T00:13:54.709 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2318000000000003e-05 2020-01-16T00:13:54.709 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.710 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.710 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:54.710 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.050006+0000", 2020-01-16T00:13:54.710 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7972000000000001e-05 2020-01-16T00:13:54.710 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.710 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.710 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:54.710 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.050054+0000", 2020-01-16T00:13:54.711 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016764102999999999 2020-01-16T00:13:54.711 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.711 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:54.711 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.711 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.711 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.711 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1639 3.0 3:00c146d5:::smithi12313934-18:head [write 423576~401598] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:13:54.711 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.774317+0000", 2020-01-16T00:13:54.712 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 62.868634851000003, 2020-01-16T00:13:54.712 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018305219000000001, 2020-01-16T00:13:54.712 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:54.712 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:54.712 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:54.712 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:54.712 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.712 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1639 2020-01-16T00:13:54.713 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.713 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:54.713 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.713 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:54.713 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774317+0000", 2020-01-16T00:13:54.713 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:54.713 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.713 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.714 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:54.714 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774317+0000", 2020-01-16T00:13:54.714 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:54.714 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.714 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.714 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:54.714 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774313+0000", 2020-01-16T00:13:54.714 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000111173 2020-01-16T00:13:54.715 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.715 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.715 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:54.715 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774425+0000", 2020-01-16T00:13:54.715 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7598999999999999e-05 2020-01-16T00:13:54.715 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.715 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.715 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:54.716 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774442+0000", 2020-01-16T00:13:54.716 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0830000000000004e-06 2020-01-16T00:13:54.716 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.716 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.716 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.716 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774450+0000", 2020-01-16T00:13:54.717 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0017886309999999999 2020-01-16T00:13:54.717 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.717 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.717 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.717 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.776239+0000", 2020-01-16T00:13:54.717 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3789e-05 2020-01-16T00:13:54.717 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.718 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.718 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op must wait for map", 2020-01-16T00:13:54.718 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.776253+0000", 2020-01-16T00:13:54.718 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095730729999999997 2020-01-16T00:13:54.718 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.718 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.718 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.718 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.785826+0000", 2020-01-16T00:13:54.719 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014220300000000001 2020-01-16T00:13:54.719 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.719 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.719 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:54.719 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.785968+0000", 2020-01-16T00:13:54.719 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023618900000000001 2020-01-16T00:13:54.719 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.719 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.720 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:54.720 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.786204+0000", 2020-01-16T00:13:54.720 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0062297389999999998 2020-01-16T00:13:54.720 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.720 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.720 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:54.720 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792434+0000", 2020-01-16T00:13:54.720 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9635999999999998e-05 2020-01-16T00:13:54.721 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.721 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.721 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.721 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792473+0000", 2020-01-16T00:13:54.721 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6456000000000002e-05 2020-01-16T00:13:54.721 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.721 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.721 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.722 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792540+0000", 2020-01-16T00:13:54.722 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9873999999999998e-05 2020-01-16T00:13:54.722 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.722 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.722 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:54.722 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792580+0000", 2020-01-16T00:13:54.722 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1924e-05 2020-01-16T00:13:54.722 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.723 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.723 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:54.723 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792622+0000", 2020-01-16T00:13:54.723 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018305219000000001 2020-01-16T00:13:54.723 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.723 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:54.723 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.724 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.724 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.724 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1890 3.0 3:0b1f384f:::smithi12313934-33:head [write 2797915~596476] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:13:54.724 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.232534+0000", 2020-01-16T00:13:54.724 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 57.41041688, 2020-01-16T00:13:54.724 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016106137, 2020-01-16T00:13:54.724 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:54.725 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:54.725 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:54.725 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:54.725 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.725 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1890 2020-01-16T00:13:54.725 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.725 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:54.726 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.726 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:54.726 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232534+0000", 2020-01-16T00:13:54.726 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:54.726 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.726 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.727 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:54.727 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232534+0000", 2020-01-16T00:13:54.727 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:54.727 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.727 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.727 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:54.727 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232532+0000", 2020-01-16T00:13:54.727 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00024232100000000001 2020-01-16T00:13:54.728 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.728 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.728 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:54.728 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232774+0000", 2020-01-16T00:13:54.728 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4601e-05 2020-01-16T00:13:54.728 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.728 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.728 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:54.729 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232789+0000", 2020-01-16T00:13:54.729 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8410000000000006e-06 2020-01-16T00:13:54.729 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.729 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.729 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.729 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232798+0000", 2020-01-16T00:13:54.729 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9756999999999997e-05 2020-01-16T00:13:54.730 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.730 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.730 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.730 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232857+0000", 2020-01-16T00:13:54.730 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4678000000000007e-05 2020-01-16T00:13:54.730 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.730 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.730 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.731 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232922+0000", 2020-01-16T00:13:54.731 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033861020000000002 2020-01-16T00:13:54.731 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.731 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.731 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.731 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.236308+0000", 2020-01-16T00:13:54.731 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000568996 2020-01-16T00:13:54.732 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.732 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.732 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.732 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.236877+0000", 2020-01-16T00:13:54.732 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0085444240000000001 2020-01-16T00:13:54.732 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.732 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.732 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.733 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.245422+0000", 2020-01-16T00:13:54.733 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5595999999999998e-05 2020-01-16T00:13:54.733 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.733 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.733 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:54.733 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.245507+0000", 2020-01-16T00:13:54.733 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016984300000000001 2020-01-16T00:13:54.733 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.734 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.734 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:54.734 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.245677+0000", 2020-01-16T00:13:54.734 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001649495 2020-01-16T00:13:54.734 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.734 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.734 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:54.735 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247326+0000", 2020-01-16T00:13:54.735 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023308399999999999 2020-01-16T00:13:54.735 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.735 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.735 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.735 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247560+0000", 2020-01-16T00:13:54.735 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00099501800000000007 2020-01-16T00:13:54.735 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.736 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.736 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.736 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248555+0000", 2020-01-16T00:13:54.736 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5936999999999998e-05 2020-01-16T00:13:54.736 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.736 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.737 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:54.737 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248591+0000", 2020-01-16T00:13:54.737 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0108999999999997e-05 2020-01-16T00:13:54.737 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.737 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.737 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:54.737 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248641+0000", 2020-01-16T00:13:54.737 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016106137 2020-01-16T00:13:54.738 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.738 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:54.738 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.738 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.738 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.738 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1891 3.0 3:0b1f384f:::smithi12313934-33:head [setxattr _header (56) in=7b,truncate 3395014] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:13:54.738 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.232813+0000", 2020-01-16T00:13:54.738 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 57.410138078000003, 2020-01-16T00:13:54.739 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015999295, 2020-01-16T00:13:54.739 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:54.739 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:54.739 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:54.739 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:54.739 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.739 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1891 2020-01-16T00:13:54.739 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.740 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:54.740 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.740 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:54.740 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232813+0000", 2020-01-16T00:13:54.740 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:54.740 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.740 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.740 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:54.741 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232813+0000", 2020-01-16T00:13:54.741 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:54.741 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.741 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.741 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:54.741 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232812+0000", 2020-01-16T00:13:54.741 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.576e-06 2020-01-16T00:13:54.742 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.742 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.742 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:54.742 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232820+0000", 2020-01-16T00:13:54.742 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3079999999999997e-06 2020-01-16T00:13:54.742 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.742 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.742 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:54.743 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232829+0000", 2020-01-16T00:13:54.743 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7119999999999997e-06 2020-01-16T00:13:54.743 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.743 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.743 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.743 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232835+0000", 2020-01-16T00:13:54.743 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013438400000000001 2020-01-16T00:13:54.743 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.744 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.744 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.744 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232969+0000", 2020-01-16T00:13:54.744 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0562000000000001e-05 2020-01-16T00:13:54.744 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.744 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.744 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.744 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.233029+0000", 2020-01-16T00:13:54.745 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039354239999999999 2020-01-16T00:13:54.745 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.745 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.745 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.745 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.236965+0000", 2020-01-16T00:13:54.745 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0482999999999998e-05 2020-01-16T00:13:54.745 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.745 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.746 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.746 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.237015+0000", 2020-01-16T00:13:54.746 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0096086209999999995 2020-01-16T00:13:54.746 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.746 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.746 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.747 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.246624+0000", 2020-01-16T00:13:54.747 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6150999999999995e-05 2020-01-16T00:13:54.747 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.747 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.747 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:54.747 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.246720+0000", 2020-01-16T00:13:54.747 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015032899999999999 2020-01-16T00:13:54.747 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.748 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.748 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:54.748 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.246870+0000", 2020-01-16T00:13:54.748 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00064760700000000002 2020-01-16T00:13:54.748 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.748 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.748 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:54.748 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247518+0000", 2020-01-16T00:13:54.749 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000198531 2020-01-16T00:13:54.749 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.749 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.749 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.749 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247717+0000", 2020-01-16T00:13:54.749 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010189439999999999 2020-01-16T00:13:54.749 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.750 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.750 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.750 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248736+0000", 2020-01-16T00:13:54.750 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4863000000000001e-05 2020-01-16T00:13:54.750 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.750 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.750 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:54.750 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248770+0000", 2020-01-16T00:13:54.751 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2153000000000001e-05 2020-01-16T00:13:54.751 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.751 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.751 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:54.751 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248813+0000", 2020-01-16T00:13:54.751 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015999295 2020-01-16T00:13:54.751 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.751 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:54.752 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.752 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.752 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.752 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1892 3.0 3:0b1f384f:::smithi12313934-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e31)", 2020-01-16T00:13:54.752 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.232856+0000", 2020-01-16T00:13:54.752 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 57.410095853999998, 2020-01-16T00:13:54.752 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016171544, 2020-01-16T00:13:54.752 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:54.753 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:54.753 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:54.753 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:54.753 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.753 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1892 2020-01-16T00:13:54.753 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.753 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:54.753 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.754 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:54.754 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232856+0000", 2020-01-16T00:13:54.754 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:54.754 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.754 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.754 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:54.754 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232856+0000", 2020-01-16T00:13:54.754 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:54.755 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.755 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.755 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:54.755 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232855+0000", 2020-01-16T00:13:54.755 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8640000000000004e-06 2020-01-16T00:13:54.755 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.755 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.755 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:54.756 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232860+0000", 2020-01-16T00:13:54.756 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0000000000000001e-05 2020-01-16T00:13:54.756 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.756 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.756 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:54.756 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232870+0000", 2020-01-16T00:13:54.756 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7420000000000003e-06 2020-01-16T00:13:54.756 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.757 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.757 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.757 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232875+0000", 2020-01-16T00:13:54.757 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017933200000000001 2020-01-16T00:13:54.757 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.757 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.758 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.758 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.233055+0000", 2020-01-16T00:13:54.758 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6269e-05 2020-01-16T00:13:54.758 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.758 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.758 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.758 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.233101+0000", 2020-01-16T00:13:54.759 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039780090000000002 2020-01-16T00:13:54.759 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.759 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.759 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.759 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.237079+0000", 2020-01-16T00:13:54.759 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3677000000000001e-05 2020-01-16T00:13:54.759 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.759 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.760 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.760 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.237133+0000", 2020-01-16T00:13:54.760 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010036671 2020-01-16T00:13:54.760 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.760 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.760 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.760 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247169+0000", 2020-01-16T00:13:54.760 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010280699999999999 2020-01-16T00:13:54.761 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.761 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.761 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:54.761 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247272+0000", 2020-01-16T00:13:54.761 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001579708 2020-01-16T00:13:54.761 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.761 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.761 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.762 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248852+0000", 2020-01-16T00:13:54.762 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9704999999999996e-05 2020-01-16T00:13:54.762 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.762 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.762 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:54.762 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248942+0000", 2020-01-16T00:13:54.762 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5557999999999996e-05 2020-01-16T00:13:54.762 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.763 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.763 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:54.763 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.249027+0000", 2020-01-16T00:13:54.763 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016171544 2020-01-16T00:13:54.763 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.763 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:54.763 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.763 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.764 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.764 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1950 3.0 3:0b1f384f:::smithi12313934-33:head [write 1889359~541737] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-01-16T00:13:54.764 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.922819+0000", 2020-01-16T00:13:54.764 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 56.720132489000001, 2020-01-16T00:13:54.764 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017502846999999998, 2020-01-16T00:13:54.764 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:54.764 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:54.765 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:54.765 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:54.765 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.765 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1950 2020-01-16T00:13:54.765 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.765 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:54.765 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.765 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:54.766 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922819+0000", 2020-01-16T00:13:54.766 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:54.766 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.766 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.766 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:54.766 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922819+0000", 2020-01-16T00:13:54.766 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:54.766 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.767 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.767 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:54.767 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922818+0000", 2020-01-16T00:13:54.767 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000146408 2020-01-16T00:13:54.767 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.767 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.768 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:54.768 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922964+0000", 2020-01-16T00:13:54.768 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4720000000000001e-06 2020-01-16T00:13:54.768 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.768 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.768 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:54.768 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922974+0000", 2020-01-16T00:13:54.768 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0720000000000001e-06 2020-01-16T00:13:54.769 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.769 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.769 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.769 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922978+0000", 2020-01-16T00:13:54.769 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6152000000000001e-05 2020-01-16T00:13:54.769 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.769 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.769 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.770 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.923024+0000", 2020-01-16T00:13:54.770 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00087368199999999997 2020-01-16T00:13:54.770 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.770 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.770 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.770 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.923897+0000", 2020-01-16T00:13:54.770 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011944435999999999 2020-01-16T00:13:54.771 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.771 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.771 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.771 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.935842+0000", 2020-01-16T00:13:54.771 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000144146 2020-01-16T00:13:54.771 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.771 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.771 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:54.772 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.935986+0000", 2020-01-16T00:13:54.772 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019775500000000001 2020-01-16T00:13:54.772 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.772 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.772 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:54.772 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.936184+0000", 2020-01-16T00:13:54.772 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002539342 2020-01-16T00:13:54.772 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.773 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.773 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.773 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938723+0000", 2020-01-16T00:13:54.773 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014193699999999999 2020-01-16T00:13:54.773 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.773 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.773 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.774 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938865+0000", 2020-01-16T00:13:54.774 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013829020000000001 2020-01-16T00:13:54.774 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.774 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.774 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:54.774 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.940248+0000", 2020-01-16T00:13:54.774 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2558000000000001e-05 2020-01-16T00:13:54.774 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.775 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.775 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:54.775 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.940291+0000", 2020-01-16T00:13:54.775 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1217999999999998e-05 2020-01-16T00:13:54.775 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.775 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.775 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:54.776 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.940322+0000", 2020-01-16T00:13:54.776 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017502846999999998 2020-01-16T00:13:54.776 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.776 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:54.776 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.776 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.776 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.777 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1951 3.0 3:0b1f384f:::smithi12313934-33:head [write 2955792~541902] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-01-16T00:13:54.777 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.929019+0000", 2020-01-16T00:13:54.777 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 56.713932339000003, 2020-01-16T00:13:54.777 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021625710999999999, 2020-01-16T00:13:54.777 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:54.777 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:54.778 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:54.778 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:54.778 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.778 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1951 2020-01-16T00:13:54.778 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.778 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:54.778 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.779 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:54.779 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929019+0000", 2020-01-16T00:13:54.779 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:54.779 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.779 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.779 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:54.779 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929019+0000", 2020-01-16T00:13:54.779 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999989 2020-01-16T00:13:54.780 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.780 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.780 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:54.780 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929008+0000", 2020-01-16T00:13:54.780 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000212513 2020-01-16T00:13:54.780 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.780 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.780 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:54.781 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929221+0000", 2020-01-16T00:13:54.781 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3229e-05 2020-01-16T00:13:54.781 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.781 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.781 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:54.781 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929234+0000", 2020-01-16T00:13:54.781 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9170000000000003e-06 2020-01-16T00:13:54.781 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.782 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.782 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.782 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929240+0000", 2020-01-16T00:13:54.782 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4045000000000002e-05 2020-01-16T00:13:54.782 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.782 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.782 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.782 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929294+0000", 2020-01-16T00:13:54.783 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9797999999999996e-05 2020-01-16T00:13:54.783 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.783 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.783 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.783 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929384+0000", 2020-01-16T00:13:54.783 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083450729999999997 2020-01-16T00:13:54.783 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.783 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.784 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.784 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.937729+0000", 2020-01-16T00:13:54.784 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043251699999999999 2020-01-16T00:13:54.784 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.784 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.784 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.784 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938161+0000", 2020-01-16T00:13:54.784 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095115489999999994 2020-01-16T00:13:54.785 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.785 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.785 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.785 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.947673+0000", 2020-01-16T00:13:54.785 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012395099999999999 2020-01-16T00:13:54.785 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.785 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.786 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:54.786 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.947797+0000", 2020-01-16T00:13:54.786 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00024849200000000001 2020-01-16T00:13:54.786 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.786 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.786 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:54.786 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.948045+0000", 2020-01-16T00:13:54.786 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0025081050000000001 2020-01-16T00:13:54.787 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.787 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.787 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.787 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950553+0000", 2020-01-16T00:13:54.787 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9760000000000003e-06 2020-01-16T00:13:54.787 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.787 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.788 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:54.788 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950558+0000", 2020-01-16T00:13:54.788 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3286999999999999e-05 2020-01-16T00:13:54.788 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.788 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.788 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.788 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950581+0000", 2020-01-16T00:13:54.789 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2759000000000003e-05 2020-01-16T00:13:54.789 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.789 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.789 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:54.789 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950614+0000", 2020-01-16T00:13:54.789 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0525999999999999e-05 2020-01-16T00:13:54.789 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.789 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.790 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:54.790 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950645+0000", 2020-01-16T00:13:54.790 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021625710999999999 2020-01-16T00:13:54.790 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.790 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:54.790 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.790 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.790 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.791 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1952 3.0 3:0b1f384f:::smithi12313934-33:head [setxattr _header (56) in=7b,truncate 3563084] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-01-16T00:13:54.791 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:13:54.791 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 56.713703002000003, 2020-01-16T00:13:54.791 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021534685000000001, 2020-01-16T00:13:54.791 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:54.791 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:54.791 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:54.791 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:54.792 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.792 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1952 2020-01-16T00:13:54.792 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.792 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:54.792 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.792 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:54.792 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:13:54.792 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:54.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.793 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.793 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:54.793 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:13:54.793 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:54.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.793 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.794 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:54.794 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:13:54.794 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6670000000000004e-06 2020-01-16T00:13:54.794 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.794 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.794 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:54.794 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929252+0000", 2020-01-16T00:13:54.794 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9380000000000002e-06 2020-01-16T00:13:54.795 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.795 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.795 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:54.795 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929256+0000", 2020-01-16T00:13:54.795 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.712e-06 2020-01-16T00:13:54.795 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.795 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.795 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.796 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929264+0000", 2020-01-16T00:13:54.796 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018030300000000001 2020-01-16T00:13:54.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.796 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.796 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.796 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929444+0000", 2020-01-16T00:13:54.796 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7562000000000002e-05 2020-01-16T00:13:54.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.797 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.797 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.797 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929502+0000", 2020-01-16T00:13:54.797 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0087297859999999998 2020-01-16T00:13:54.797 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.797 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.797 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.798 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938232+0000", 2020-01-16T00:13:54.798 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0452999999999996e-05 2020-01-16T00:13:54.798 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.798 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.798 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.798 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938322+0000", 2020-01-16T00:13:54.798 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011061728 2020-01-16T00:13:54.799 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.799 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.799 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.799 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.949384+0000", 2020-01-16T00:13:54.799 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000117828 2020-01-16T00:13:54.799 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.799 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.800 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:54.800 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.949502+0000", 2020-01-16T00:13:54.800 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022468499999999999 2020-01-16T00:13:54.800 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.800 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.800 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:54.800 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.949726+0000", 2020-01-16T00:13:54.800 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00083532099999999998 2020-01-16T00:13:54.801 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.801 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.801 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:54.801 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950562+0000", 2020-01-16T00:13:54.801 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013017700000000001 2020-01-16T00:13:54.801 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.801 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.801 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.802 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950692+0000", 2020-01-16T00:13:54.802 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9813000000000001e-05 2020-01-16T00:13:54.802 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.802 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.802 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.802 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950722+0000", 2020-01-16T00:13:54.802 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0592000000000001e-05 2020-01-16T00:13:54.802 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.803 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.803 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:54.803 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950752+0000", 2020-01-16T00:13:54.803 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0920000000000002e-05 2020-01-16T00:13:54.803 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.803 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.803 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:54.804 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950783+0000", 2020-01-16T00:13:54.804 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021534685000000001 2020-01-16T00:13:54.804 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.804 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:54.804 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.804 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.804 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.804 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1953 3.0 3:0b1f384f:::smithi12313934-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e32)", 2020-01-16T00:13:54.805 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:13:54.805 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 56.713670010999998, 2020-01-16T00:13:54.805 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021667537000000001, 2020-01-16T00:13:54.805 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:54.805 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:54.805 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:54.805 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:54.805 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.806 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1953 2020-01-16T00:13:54.806 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.806 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:54.806 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.806 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:54.806 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:13:54.806 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:54.806 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.807 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.807 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:54.807 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:13:54.807 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:54.807 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.807 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.807 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:54.808 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:13:54.808 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7030000000000002e-06 2020-01-16T00:13:54.808 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.808 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.808 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:54.808 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929284+0000", 2020-01-16T00:13:54.809 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0909999999999993e-06 2020-01-16T00:13:54.809 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.809 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.809 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:54.809 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929292+0000", 2020-01-16T00:13:54.809 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8359999999999999e-06 2020-01-16T00:13:54.809 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.809 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.810 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.810 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929297+0000", 2020-01-16T00:13:54.810 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023538199999999999 2020-01-16T00:13:54.810 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.810 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.810 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.810 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929532+0000", 2020-01-16T00:13:54.811 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1173000000000004e-05 2020-01-16T00:13:54.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.811 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.811 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.811 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929603+0000", 2020-01-16T00:13:54.811 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0087813509999999997 2020-01-16T00:13:54.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.811 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.812 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.812 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938384+0000", 2020-01-16T00:13:54.812 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9752000000000006e-05 2020-01-16T00:13:54.812 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.812 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.812 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.812 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938454+0000", 2020-01-16T00:13:54.812 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011707851999999999 2020-01-16T00:13:54.813 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.813 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.813 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.813 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950162+0000", 2020-01-16T00:13:54.813 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011060099999999999 2020-01-16T00:13:54.813 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.813 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.813 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:54.814 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950273+0000", 2020-01-16T00:13:54.814 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00053989700000000001 2020-01-16T00:13:54.814 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.814 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.814 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.814 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950813+0000", 2020-01-16T00:13:54.814 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8128999999999999e-05 2020-01-16T00:13:54.815 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.815 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.815 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:54.815 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950871+0000", 2020-01-16T00:13:54.815 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8242000000000001e-05 2020-01-16T00:13:54.815 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.815 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.815 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:54.816 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950949+0000", 2020-01-16T00:13:54.816 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021667537000000001 2020-01-16T00:13:54.816 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.816 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:54.816 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.816 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.816 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.816 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2555 3.0 3:0b70bb20:::smithi12313934-46:head [setxattr _header (56) in=7b,truncate 2991679] snapc 0=[] ondisk+write+known_if_redirected e37)", 2020-01-16T00:13:54.817 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.573432+0000", 2020-01-16T00:13:54.817 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 42.069519360000001, 2020-01-16T00:13:54.817 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019893926999999999, 2020-01-16T00:13:54.817 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:54.817 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:54.817 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:54.817 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:54.817 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.818 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2555 2020-01-16T00:13:54.818 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.818 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:54.818 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.818 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:54.819 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573432+0000", 2020-01-16T00:13:54.819 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:54.819 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.819 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.819 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:54.819 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573432+0000", 2020-01-16T00:13:54.819 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:54.819 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.820 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.820 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:54.820 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573432+0000", 2020-01-16T00:13:54.820 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4290000000000001e-06 2020-01-16T00:13:54.820 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.820 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.820 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:54.820 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573435+0000", 2020-01-16T00:13:54.821 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1300000000000001e-06 2020-01-16T00:13:54.821 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.821 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.821 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:54.821 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573438+0000", 2020-01-16T00:13:54.821 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9399999999999996e-06 2020-01-16T00:13:54.821 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.821 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.822 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.822 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573445+0000", 2020-01-16T00:13:54.822 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013947269999999999 2020-01-16T00:13:54.822 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.822 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.822 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.822 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.574840+0000", 2020-01-16T00:13:54.822 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003850058 2020-01-16T00:13:54.823 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.823 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.823 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.824 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578690+0000", 2020-01-16T00:13:54.825 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012896489000000001 2020-01-16T00:13:54.825 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.825 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.825 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.825 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591586+0000", 2020-01-16T00:13:54.825 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4757000000000001e-05 2020-01-16T00:13:54.826 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.826 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.826 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:54.826 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591681+0000", 2020-01-16T00:13:54.826 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000191535 2020-01-16T00:13:54.826 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.826 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.827 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:54.827 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591873+0000", 2020-01-16T00:13:54.827 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00069888899999999996 2020-01-16T00:13:54.827 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.827 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.827 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:54.827 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.592571+0000", 2020-01-16T00:13:54.827 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00029525900000000003 2020-01-16T00:13:54.828 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.828 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.828 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.828 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.592867+0000", 2020-01-16T00:13:54.828 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035243500000000001 2020-01-16T00:13:54.828 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.829 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.829 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.829 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593219+0000", 2020-01-16T00:13:54.829 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0954e-05 2020-01-16T00:13:54.829 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.829 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.829 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:54.830 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593270+0000", 2020-01-16T00:13:54.830 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5813000000000002e-05 2020-01-16T00:13:54.830 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.830 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.830 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:54.830 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593326+0000", 2020-01-16T00:13:54.830 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019893926999999999 2020-01-16T00:13:54.830 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.831 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:54.831 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.831 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.831 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.831 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2556 3.0 3:0b70bb20:::smithi12313934-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e37)", 2020-01-16T00:13:54.831 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.573472+0000", 2020-01-16T00:13:54.832 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 42.069479790000003, 2020-01-16T00:13:54.832 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020070068999999999, 2020-01-16T00:13:54.832 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:54.832 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:54.832 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:54.832 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:54.832 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.832 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2556 2020-01-16T00:13:54.833 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.833 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:54.833 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.833 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:54.833 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573472+0000", 2020-01-16T00:13:54.833 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:54.833 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.833 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.834 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:54.834 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573472+0000", 2020-01-16T00:13:54.834 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:54.834 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.834 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.834 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:54.834 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573471+0000", 2020-01-16T00:13:54.834 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4449999999999999e-06 2020-01-16T00:13:54.835 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.835 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.835 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:54.835 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573479+0000", 2020-01-16T00:13:54.835 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9789999999999998e-06 2020-01-16T00:13:54.835 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.835 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.835 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:54.836 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573485+0000", 2020-01-16T00:13:54.836 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3270000000000001e-06 2020-01-16T00:13:54.836 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.836 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.836 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.836 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573488+0000", 2020-01-16T00:13:54.836 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052586849999999999 2020-01-16T00:13:54.837 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.837 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.837 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.837 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578747+0000", 2020-01-16T00:13:54.837 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5667000000000007e-05 2020-01-16T00:13:54.837 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.837 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.837 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.838 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578812+0000", 2020-01-16T00:13:54.838 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013599539000000001 2020-01-16T00:13:54.838 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.838 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.838 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.838 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.592412+0000", 2020-01-16T00:13:54.838 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6018000000000001e-05 2020-01-16T00:13:54.838 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.839 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.839 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:54.839 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.592508+0000", 2020-01-16T00:13:54.839 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00084582199999999998 2020-01-16T00:13:54.840 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.840 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.840 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.840 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593354+0000", 2020-01-16T00:13:54.840 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4984999999999999e-05 2020-01-16T00:13:54.840 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.840 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.840 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:54.841 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593449+0000", 2020-01-16T00:13:54.841 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3104999999999997e-05 2020-01-16T00:13:54.841 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.841 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.841 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:54.841 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593542+0000", 2020-01-16T00:13:54.841 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020070068999999999 2020-01-16T00:13:54.841 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.842 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:54.842 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.842 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.842 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.842 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3211 3.0 3:0b70bb20:::smithi12313934-46:head [setxattr _header (56) in=7b,truncate 2571537] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:13:54.842 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.581146+0000", 2020-01-16T00:13:54.842 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 25.06180522, 2020-01-16T00:13:54.842 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024318514999999999, 2020-01-16T00:13:54.843 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:54.843 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:54.843 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:54.843 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:54.843 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.843 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3211 2020-01-16T00:13:54.843 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.844 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:54.844 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.844 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:54.844 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581146+0000", 2020-01-16T00:13:54.844 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:54.844 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.844 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.844 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:54.845 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581146+0000", 2020-01-16T00:13:54.845 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:54.845 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.845 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.845 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:54.845 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581145+0000", 2020-01-16T00:13:54.845 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2990000000000002e-06 2020-01-16T00:13:54.845 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.846 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.846 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:54.846 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581151+0000", 2020-01-16T00:13:54.846 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.041e-06 2020-01-16T00:13:54.846 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.846 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.846 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:54.846 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581159+0000", 2020-01-16T00:13:54.847 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5859999999999998e-06 2020-01-16T00:13:54.847 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.847 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.847 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.847 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581163+0000", 2020-01-16T00:13:54.847 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020769790000000001 2020-01-16T00:13:54.847 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.847 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.848 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.848 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.583240+0000", 2020-01-16T00:13:54.848 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012019579999999999 2020-01-16T00:13:54.848 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.848 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.848 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.848 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584442+0000", 2020-01-16T00:13:54.848 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018359812 2020-01-16T00:13:54.849 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.849 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.849 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.849 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.602802+0000", 2020-01-16T00:13:54.849 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000133072 2020-01-16T00:13:54.850 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.850 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.850 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:54.850 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.602935+0000", 2020-01-16T00:13:54.850 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019007200000000001 2020-01-16T00:13:54.850 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.850 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.851 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:54.851 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.603125+0000", 2020-01-16T00:13:54.851 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001261751 2020-01-16T00:13:54.851 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.851 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.851 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.852 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.604387+0000", 2020-01-16T00:13:54.852 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000304987 2020-01-16T00:13:54.852 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.852 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.852 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:54.852 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.604692+0000", 2020-01-16T00:13:54.853 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00063369299999999995 2020-01-16T00:13:54.853 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.853 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.853 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.853 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605325+0000", 2020-01-16T00:13:54.853 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1470000000000005e-05 2020-01-16T00:13:54.854 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.854 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.854 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:54.854 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605397+0000", 2020-01-16T00:13:54.854 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7784e-05 2020-01-16T00:13:54.854 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.854 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.855 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:54.855 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605465+0000", 2020-01-16T00:13:54.855 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024318514999999999 2020-01-16T00:13:54.855 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.855 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:54.855 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.855 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.855 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.856 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3212 3.0 3:0b70bb20:::smithi12313934-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e45)", 2020-01-16T00:13:54.856 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.581171+0000", 2020-01-16T00:13:54.856 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 25.061780520999999, 2020-01-16T00:13:54.856 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024540131, 2020-01-16T00:13:54.856 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:54.856 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:54.856 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:54.856 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:54.857 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.857 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3212 2020-01-16T00:13:54.857 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.857 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:54.857 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.857 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:54.857 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581171+0000", 2020-01-16T00:13:54.858 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:54.858 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.858 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.858 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:54.858 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581171+0000", 2020-01-16T00:13:54.858 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:54.858 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.859 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.859 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:54.859 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581170+0000", 2020-01-16T00:13:54.859 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5259999999999996e-06 2020-01-16T00:13:54.859 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.859 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.859 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:54.860 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581178+0000", 2020-01-16T00:13:54.860 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.469e-06 2020-01-16T00:13:54.860 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.860 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.860 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:54.860 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581185+0000", 2020-01-16T00:13:54.860 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4460000000000003e-06 2020-01-16T00:13:54.861 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.861 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.861 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.861 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581190+0000", 2020-01-16T00:13:54.861 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003286696 2020-01-16T00:13:54.861 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.862 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.862 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.862 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584476+0000", 2020-01-16T00:13:54.862 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9681e-05 2020-01-16T00:13:54.862 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.862 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.862 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.863 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584536+0000", 2020-01-16T00:13:54.863 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019075267 2020-01-16T00:13:54.863 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.863 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.863 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.863 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.603611+0000", 2020-01-16T00:13:54.863 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5981000000000001e-05 2020-01-16T00:13:54.864 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.864 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.864 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:54.864 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.603707+0000", 2020-01-16T00:13:54.864 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0017922649999999999 2020-01-16T00:13:54.864 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.864 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.865 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.865 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605500+0000", 2020-01-16T00:13:54.865 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5396999999999995e-05 2020-01-16T00:13:54.865 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.865 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.865 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:54.865 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605595+0000", 2020-01-16T00:13:54.865 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000116038 2020-01-16T00:13:54.866 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.866 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.866 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:54.866 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605711+0000", 2020-01-16T00:13:54.866 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024540131 2020-01-16T00:13:54.866 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.867 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:54.867 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.867 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.867 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.867 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3435 3.0 3:0b70bb20:::smithi12313934-46:head [setxattr _header (56) in=7b,truncate 3075747] snapc 0=[] ondisk+write+known_if_redirected e47)", 2020-01-16T00:13:54.867 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:34.388891+0000", 2020-01-16T00:13:54.867 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 20.254060505000002, 2020-01-16T00:13:54.868 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016440475999999999, 2020-01-16T00:13:54.868 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:54.869 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:54.869 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:54.869 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:54.869 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.869 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3435 2020-01-16T00:13:54.869 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.869 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:54.870 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.870 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:54.870 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388891+0000", 2020-01-16T00:13:54.870 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:54.870 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.870 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.870 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:54.870 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388891+0000", 2020-01-16T00:13:54.871 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:54.871 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.871 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.871 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:54.871 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388890+0000", 2020-01-16T00:13:54.871 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.647e-06 2020-01-16T00:13:54.871 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.872 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.872 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:54.872 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388894+0000", 2020-01-16T00:13:54.872 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2839999999999997e-06 2020-01-16T00:13:54.872 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.872 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.873 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:54.873 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388899+0000", 2020-01-16T00:13:54.873 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3149999999999999e-06 2020-01-16T00:13:54.873 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.873 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.873 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.873 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388901+0000", 2020-01-16T00:13:54.874 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001768543 2020-01-16T00:13:54.874 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.874 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.874 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.874 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.390670+0000", 2020-01-16T00:13:54.874 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0034533680000000001 2020-01-16T00:13:54.874 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.874 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.875 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.875 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.394123+0000", 2020-01-16T00:13:54.875 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010189962 2020-01-16T00:13:54.875 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.875 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.875 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.875 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404313+0000", 2020-01-16T00:13:54.875 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000108646 2020-01-16T00:13:54.876 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.876 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.876 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:54.876 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404422+0000", 2020-01-16T00:13:54.876 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119133 2020-01-16T00:13:54.876 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.876 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.877 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:54.877 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404541+0000", 2020-01-16T00:13:54.877 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000433764 2020-01-16T00:13:54.877 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.877 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.877 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:54.877 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404975+0000", 2020-01-16T00:13:54.878 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000266013 2020-01-16T00:13:54.878 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.878 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.878 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.878 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405241+0000", 2020-01-16T00:13:54.878 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5187000000000001e-05 2020-01-16T00:13:54.878 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.878 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.879 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:54.879 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405266+0000", 2020-01-16T00:13:54.879 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0889000000000002e-05 2020-01-16T00:13:54.879 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.879 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.879 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:54.880 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405297+0000", 2020-01-16T00:13:54.880 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4354999999999999e-05 2020-01-16T00:13:54.880 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.880 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.880 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:54.880 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405331+0000", 2020-01-16T00:13:54.880 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016440475999999999 2020-01-16T00:13:54.881 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.881 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:54.881 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.881 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.881 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.881 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3436 3.0 3:0b70bb20:::smithi12313934-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e47)", 2020-01-16T00:13:54.881 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:34.388920+0000", 2020-01-16T00:13:54.882 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 20.254031716, 2020-01-16T00:13:54.882 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016552641, 2020-01-16T00:13:54.882 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:54.882 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:54.882 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:54.882 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:54.882 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.883 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3436 2020-01-16T00:13:54.883 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.883 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:54.883 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.883 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:54.883 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388920+0000", 2020-01-16T00:13:54.883 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:54.884 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.884 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.884 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:54.884 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388920+0000", 2020-01-16T00:13:54.884 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:54.884 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.885 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.885 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:54.885 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388919+0000", 2020-01-16T00:13:54.885 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.181e-06 2020-01-16T00:13:54.885 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.885 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.885 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:54.886 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388921+0000", 2020-01-16T00:13:54.886 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9940000000000003e-06 2020-01-16T00:13:54.886 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.886 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.886 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:54.886 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388928+0000", 2020-01-16T00:13:54.886 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3549999999999999e-06 2020-01-16T00:13:54.887 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.887 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.887 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.887 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388931+0000", 2020-01-16T00:13:54.887 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.005242483 2020-01-16T00:13:54.887 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.887 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.887 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.888 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.394173+0000", 2020-01-16T00:13:54.888 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7435999999999995e-05 2020-01-16T00:13:54.888 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.888 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.888 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:54.888 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.394251+0000", 2020-01-16T00:13:54.888 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010593286 2020-01-16T00:13:54.888 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.889 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.889 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.889 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404844+0000", 2020-01-16T00:13:54.889 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0620999999999999e-05 2020-01-16T00:13:54.889 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.889 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.889 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:54.890 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404925+0000", 2020-01-16T00:13:54.890 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00042800899999999997 2020-01-16T00:13:54.890 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.890 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.890 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.890 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405353+0000", 2020-01-16T00:13:54.890 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2250000000000003e-05 2020-01-16T00:13:54.891 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.891 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.891 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:54.891 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405405+0000", 2020-01-16T00:13:54.891 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7411999999999998e-05 2020-01-16T00:13:54.891 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.891 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.891 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:54.892 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405472+0000", 2020-01-16T00:13:54.892 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016552641 2020-01-16T00:13:54.892 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.892 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:54.892 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.892 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.892 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.893 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3873 3.0 3:0b1f384f:::smithi12313934-33:head [read 0~3546310,checksum] snapc 0=[] ondisk+read+known_if_redirected e50)", 2020-01-16T00:13:54.893 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:44.489248+0000", 2020-01-16T00:13:54.893 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 10.153703424, 2020-01-16T00:13:54.893 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016088641000000001, 2020-01-16T00:13:54.893 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:54.893 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:54.894 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:54.894 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:54.894 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:54.894 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3873 2020-01-16T00:13:54.894 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.894 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:54.894 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.895 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:54.895 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489248+0000", 2020-01-16T00:13:54.895 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:54.895 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.895 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.895 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:54.895 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489248+0000", 2020-01-16T00:13:54.895 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:54.896 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.896 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.896 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:54.896 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489247+0000", 2020-01-16T00:13:54.896 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8080000000000004e-06 2020-01-16T00:13:54.896 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.896 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.896 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:54.897 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489252+0000", 2020-01-16T00:13:54.897 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1639999999999998e-06 2020-01-16T00:13:54.897 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.897 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.897 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:54.897 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489258+0000", 2020-01-16T00:13:54.897 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5169999999999996e-06 2020-01-16T00:13:54.898 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.898 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.898 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:54.898 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489265+0000", 2020-01-16T00:13:54.898 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.01457085 2020-01-16T00:13:54.898 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.898 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.898 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:54.899 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.503836+0000", 2020-01-16T00:13:54.899 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4594999999999998e-05 2020-01-16T00:13:54.899 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.899 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.899 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:54.899 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.503930+0000", 2020-01-16T00:13:54.899 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001406355 2020-01-16T00:13:54.900 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:54.900 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:54.900 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:54.900 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.505337+0000", 2020-01-16T00:13:54.900 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016088641000000001 2020-01-16T00:13:54.900 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.900 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:54.900 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.901 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:54.901 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:54.901 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:54.905 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-01-16T00:13:54.905 INFO:tasks.rados.rados.0.smithi123.stdout:1541: finishing write tid 5 to smithi12313934-49 2020-01-16T00:13:54.905 INFO:tasks.rados.rados.0.smithi123.stdout:1541: finishing write tid 6 to smithi12313934-49 2020-01-16T00:13:54.905 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 49 v 157 (ObjNum 526 snap 0 seq_num 526) dirty exists 2020-01-16T00:13:54.906 INFO:tasks.rados.rados.0.smithi123.stdout:1541: left oid 49 (ObjNum 526 snap 0 seq_num 526) 2020-01-16T00:13:54.906 INFO:tasks.rados.rados.0.smithi123.stdout:1544: finishing write tid 1 to smithi12313934-26 2020-01-16T00:13:54.906 INFO:tasks.rados.rados.0.smithi123.stdout:1544: finishing write tid 2 to smithi12313934-26 2020-01-16T00:13:54.906 INFO:tasks.rados.rados.0.smithi123.stdout:1544: finishing write tid 3 to smithi12313934-26 2020-01-16T00:13:54.906 INFO:tasks.rados.rados.0.smithi123.stdout:1540: done (15 left) 2020-01-16T00:13:54.906 INFO:tasks.rados.rados.0.smithi123.stdout:1541: done (14 left) 2020-01-16T00:13:54.906 INFO:tasks.rados.rados.0.smithi123.stdout:1542: done (13 left) 2020-01-16T00:13:54.907 INFO:tasks.rados.rados.0.smithi123.stdout:1543: done (12 left) 2020-01-16T00:13:54.907 INFO:tasks.rados.rados.0.smithi123.stdout:1557: read oid 41 snap -1 2020-01-16T00:13:54.907 INFO:tasks.rados.rados.0.smithi123.stdout:1557: expect (ObjNum 441 snap 0 seq_num 441) 2020-01-16T00:13:54.907 INFO:tasks.rados.rados.0.smithi123.stdout:1558: read oid 28 snap -1 2020-01-16T00:13:54.907 INFO:tasks.rados.rados.0.smithi123.stdout:1558: expect (ObjNum 506 snap 0 seq_num 506) 2020-01-16T00:13:54.907 INFO:tasks.rados.rados.0.smithi123.stdout:1559: write oid 18 current snap is 0 2020-01-16T00:13:54.907 INFO:tasks.rados.rados.0.smithi123.stdout:1559: seq_num 533 ranges {558131=652427,1673373=559846,2820808=137323} 2020-01-16T00:13:54.907 INFO:tasks.rados.rados.0.smithi123.stdout:1559: writing smithi12313934-18 from 558131 to 1210558 tid 1 2020-01-16T00:13:54.908 INFO:tasks.rados.rados.0.smithi123.stdout:1559: writing smithi12313934-18 from 1673373 to 2233219 tid 2 2020-01-16T00:13:54.908 INFO:tasks.rados.rados.0.smithi123.stdout:1559: writing smithi12313934-18 from 2820808 to 2958131 tid 3 2020-01-16T00:13:54.908 INFO:tasks.rados.rados.0.smithi123.stdout:1560: write oid 3 current snap is 0 2020-01-16T00:13:54.908 INFO:tasks.rados.rados.0.smithi123.stdout:1560: seq_num 534 ranges {574938=727676,1731855=703504,2848231=126707} 2020-01-16T00:13:54.908 INFO:tasks.rados.rados.0.smithi123.stdout:1560: writing smithi12313934-3 from 574938 to 1302614 tid 1 2020-01-16T00:13:54.908 INFO:tasks.rados.rados.0.smithi123.stdout:1560: writing smithi12313934-3 from 1731855 to 2435359 tid 2 2020-01-16T00:13:54.908 INFO:tasks.rados.rados.0.smithi123.stdout:1560: writing smithi12313934-3 from 2848231 to 2974938 tid 3 2020-01-16T00:13:54.909 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:54.909 INFO:tasks.rados.rados.0.smithi123.stdout:1545: expect (ObjNum 517 snap 0 seq_num 517) 2020-01-16T00:13:54.909 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:54.898+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:54.998 INFO:tasks.rados.rados.0.smithi123.stdout:1544: finishing write tid 5 to smithi12313934-26 2020-01-16T00:13:54.999 INFO:tasks.rados.rados.0.smithi123.stdout:1544: finishing write tid 6 to smithi12313934-26 2020-01-16T00:13:54.999 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 26 v 279 (ObjNum 527 snap 0 seq_num 527) dirty exists 2020-01-16T00:13:54.999 INFO:tasks.rados.rados.0.smithi123.stdout:1544: left oid 26 (ObjNum 527 snap 0 seq_num 527) 2020-01-16T00:13:54.999 INFO:tasks.rados.rados.0.smithi123.stdout:1546: expect (ObjNum 522 snap 0 seq_num 522) 2020-01-16T00:13:55.004 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:55.000+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:55.047 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:55.048 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:55.048 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:13:55.048 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:55.056 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-01-16T00:13:55.073 INFO:teuthology.orchestra.run.smithi123.stderr:noscrub is set 2020-01-16T00:13:55.075 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-01-16T00:13:55.104 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:55.100+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:55.149 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:55.150 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:55.150 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:13:55.150 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:13:55.150 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:55.158 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-01-16T00:13:55.203 INFO:tasks.rados.rados.0.smithi123.stdout:1547: finishing write tid 1 to smithi12313934-48 2020-01-16T00:13:55.203 INFO:tasks.rados.rados.0.smithi123.stdout:1547: finishing write tid 2 to smithi12313934-48 2020-01-16T00:13:55.203 INFO:tasks.rados.rados.0.smithi123.stdout:1544: done (15 left) 2020-01-16T00:13:55.203 INFO:tasks.rados.rados.0.smithi123.stdout:1545: done (14 left) 2020-01-16T00:13:55.203 INFO:tasks.rados.rados.0.smithi123.stdout:1546: done (13 left) 2020-01-16T00:13:55.203 INFO:tasks.rados.rados.0.smithi123.stdout:1561: write oid 39 current snap is 0 2020-01-16T00:13:55.204 INFO:tasks.rados.rados.0.smithi123.stdout:1561: seq_num 535 ranges {591745=402925,1790337=563515} 2020-01-16T00:13:55.204 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:13:55.205 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:55.202+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:55.210 INFO:tasks.rados.rados.0.smithi123.stdout:1561: writing smithi12313934-39 from 591745 to 994670 tid 1 2020-01-16T00:13:55.223 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:13:55.225 INFO:tasks.rados.rados.0.smithi123.stdout:1561: writing smithi12313934-39 from 1790337 to 2353852 tid 2 2020-01-16T00:13:55.225 INFO:tasks.rados.rados.0.smithi123.stdout:1562: delete oid 20 current snap is 0 2020-01-16T00:13:55.225 INFO:tasks.rados.rados.0.smithi123.stdout:1548: expect (ObjNum 514 snap 0 seq_num 514) 2020-01-16T00:13:55.252 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:13:55.257 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:55.266 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-01-16T00:13:55.281 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:13:55.312 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:13:55.312 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:55.310+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:55.345 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:13:55.366 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:13:55.368 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-01-16T00:13:55.368 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-01-16T00:13:55.368 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-01-16T00:13:55.368 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-01-16T00:13:55.369 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-01-16T00:13:55.369 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENXIO: problem getting command descriptions from osd.5 2020-01-16T00:13:55.369 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: problem getting command descriptions from osd.5 2020-01-16T00:13:55.369 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-01-16T00:13:55.369 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-01-16T00:13:55.378 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:55.381 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-01-16T00:13:55.382 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-01-16T00:13:55.387 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-01-16T00:13:55.431 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:55.430+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:55.447 INFO:tasks.rados.rados.0.smithi123.stdout:1547: finishing write tid 3 to smithi12313934-48 2020-01-16T00:13:55.447 INFO:tasks.rados.rados.0.smithi123.stdout:1547: finishing write tid 5 to smithi12313934-48 2020-01-16T00:13:55.448 INFO:tasks.rados.rados.0.smithi123.stdout:1549: finishing write tid 1 to smithi12313934-45 2020-01-16T00:13:55.448 INFO:tasks.rados.rados.0.smithi123.stdout:1547: finishing write tid 6 to smithi12313934-48 2020-01-16T00:13:55.448 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 48 v 186 (ObjNum 528 snap 0 seq_num 528) dirty exists 2020-01-16T00:13:55.448 INFO:tasks.rados.rados.0.smithi123.stdout:1547: left oid 48 (ObjNum 528 snap 0 seq_num 528) 2020-01-16T00:13:55.448 INFO:tasks.rados.rados.0.smithi123.stdout:1549: finishing write tid 2 to smithi12313934-45 2020-01-16T00:13:55.448 INFO:tasks.rados.rados.0.smithi123.stdout:1549: finishing write tid 4 to smithi12313934-45 2020-01-16T00:13:55.448 INFO:tasks.rados.rados.0.smithi123.stdout:1547: done (14 left) 2020-01-16T00:13:55.448 INFO:tasks.rados.rados.0.smithi123.stdout:1548: done (13 left) 2020-01-16T00:13:55.449 INFO:tasks.rados.rados.0.smithi123.stdout:1562: done (12 left) 2020-01-16T00:13:55.449 INFO:tasks.rados.rados.0.smithi123.stdout:1563: write oid 14 current snap is 0 2020-01-16T00:13:55.449 INFO:tasks.rados.rados.0.smithi123.stdout:1563: seq_num 537 ranges {625359=553423,1790948=567184,2930500=94859} 2020-01-16T00:13:55.454 INFO:tasks.rados.rados.0.smithi123.stdout:1563: writing smithi12313934-14 from 625359 to 1178782 tid 1 2020-01-16T00:13:55.462 INFO:tasks.rados.rados.0.smithi123.stdout:1563: writing smithi12313934-14 from 1790948 to 2358132 tid 2 2020-01-16T00:13:55.466 INFO:tasks.rados.rados.0.smithi123.stdout:1563: writing smithi12313934-14 from 2930500 to 3025359 tid 3 2020-01-16T00:13:55.467 INFO:tasks.rados.rados.0.smithi123.stdout:1564: write (excl) oid 35 current snap is 0 2020-01-16T00:13:55.467 INFO:tasks.rados.rados.0.smithi123.stdout:1564: seq_num 538 ranges {642166=628672,1849430=710842} 2020-01-16T00:13:55.473 INFO:tasks.rados.rados.0.smithi123.stdout:1564: writing smithi12313934-35 from 642166 to 1270838 tid 1 2020-01-16T00:13:55.481 INFO:tasks.rados.rados.0.smithi123.stdout:1564: writing smithi12313934-35 from 1849430 to 2560272 tid 2 2020-01-16T00:13:55.482 INFO:tasks.rados.rados.0.smithi123.stdout:1565: read oid 10 snap -1 2020-01-16T00:13:55.482 INFO:tasks.rados.rados.0.smithi123.stdout:1565: expect deleted 2020-01-16T00:13:55.482 INFO:tasks.rados.rados.0.smithi123.stdout:1566: read oid 50 snap -1 2020-01-16T00:13:55.482 INFO:tasks.rados.rados.0.smithi123.stdout:1566: expect (ObjNum 486 snap 0 seq_num 486) 2020-01-16T00:13:55.482 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:55.482 INFO:tasks.rados.rados.0.smithi123.stdout:1549: finishing write tid 5 to smithi12313934-45 2020-01-16T00:13:55.482 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 45 v 160 (ObjNum 529 snap 0 seq_num 529) dirty exists 2020-01-16T00:13:55.483 INFO:tasks.rados.rados.0.smithi123.stdout:1549: left oid 45 (ObjNum 529 snap 0 seq_num 529) 2020-01-16T00:13:55.483 INFO:tasks.rados.rados.0.smithi123.stdout:1549: done (15 left) 2020-01-16T00:13:55.483 INFO:tasks.rados.rados.0.smithi123.stdout:1567: write (excl) oid 43 current snap is 0 2020-01-16T00:13:55.483 INFO:tasks.rados.rados.0.smithi123.stdout:1567: seq_num 539 ranges {658973=703921,1907912=570853,2985346=73627} 2020-01-16T00:13:55.492 INFO:tasks.rados.rados.0.smithi123.stdout:1567: writing smithi12313934-43 from 658973 to 1362894 tid 1 2020-01-16T00:13:55.496 INFO:tasks.rados.rados.0.smithi123.stdout:1567: writing smithi12313934-43 from 1907912 to 2478765 tid 2 2020-01-16T00:13:55.500 INFO:tasks.rados.rados.0.smithi123.stdout:1567: writing smithi12313934-43 from 2985346 to 3058973 tid 3 2020-01-16T00:13:55.500 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:55.500 INFO:tasks.rados.rados.0.smithi123.stdout:1551: expect (ObjNum 523 snap 0 seq_num 523) 2020-01-16T00:13:55.507 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:55.506+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:55.541 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:55.548 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-01-16T00:13:55.592 INFO:tasks.rados.rados.0.smithi123.stdout:1552: expect (ObjNum 520 snap 0 seq_num 520) 2020-01-16T00:13:55.608 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:55.604+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:55.688 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:55.697 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-01-16T00:13:55.743 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:55.742+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:55.804 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:55.810 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:55.806+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:55.813 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-01-16T00:13:55.853 INFO:tasks.rados.rados.0.smithi123.stdout:1553: finishing write tid 1 to smithi12313934-13 2020-01-16T00:13:55.853 INFO:tasks.rados.rados.0.smithi123.stdout:1553: finishing write tid 2 to smithi12313934-13 2020-01-16T00:13:55.853 INFO:tasks.rados.rados.0.smithi123.stdout:1553: finishing write tid 4 to smithi12313934-13 2020-01-16T00:13:55.854 INFO:tasks.rados.rados.0.smithi123.stdout:1553: finishing write tid 5 to smithi12313934-13 2020-01-16T00:13:55.854 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 103 (ObjNum 531 snap 0 seq_num 531) dirty exists 2020-01-16T00:13:55.854 INFO:tasks.rados.rados.0.smithi123.stdout:1553: left oid 13 (ObjNum 531 snap 0 seq_num 531) 2020-01-16T00:13:55.854 INFO:tasks.rados.rados.0.smithi123.stdout:1554: expect (ObjNum 497 snap 0 seq_num 497) 2020-01-16T00:13:55.910 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:55.908+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:55.958 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:55.967 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-01-16T00:13:56.011 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:56.008+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:56.021 INFO:tasks.rados.rados.0.smithi123.stdout:1555: finishing write tid 1 to smithi12313934-34 2020-01-16T00:13:56.021 INFO:tasks.rados.rados.0.smithi123.stdout:1555: finishing write tid 2 to smithi12313934-34 2020-01-16T00:13:56.021 INFO:tasks.rados.rados.0.smithi123.stdout:1555: finishing write tid 3 to smithi12313934-34 2020-01-16T00:13:56.021 INFO:tasks.rados.rados.0.smithi123.stdout:1555: finishing write tid 5 to smithi12313934-34 2020-01-16T00:13:56.021 INFO:tasks.rados.rados.0.smithi123.stdout:1555: finishing write tid 6 to smithi12313934-34 2020-01-16T00:13:56.022 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 34 v 69 (ObjNum 532 snap 0 seq_num 532) dirty exists 2020-01-16T00:13:56.022 INFO:tasks.rados.rados.0.smithi123.stdout:1555: left oid 34 (ObjNum 532 snap 0 seq_num 532) 2020-01-16T00:13:56.022 INFO:tasks.rados.rados.0.smithi123.stdout:1556: expect (ObjNum 432 snap 0 seq_num 432) 2020-01-16T00:13:56.075 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:56.084 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-01-16T00:13:56.112 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:56.110+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:56.197 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:56.206 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-01-16T00:13:56.212 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:56.210+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:56.226 INFO:tasks.rados.rados.0.smithi123.stdout:1558: expect (ObjNum 506 snap 0 seq_num 506) 2020-01-16T00:13:56.313 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:56.310+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:56.315 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:56.325 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-01-16T00:13:56.396 INFO:tasks.rados.rados.0.smithi123.stdout:1557: expect (ObjNum 441 snap 0 seq_num 441) 2020-01-16T00:13:56.415 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:56.412+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:56.434 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:56.443 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-01-16T00:13:56.530 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:56.528+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:56.595 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:56.610 INFO:tasks.rados.rados.0.smithi123.stdout:1559: finishing write tid 1 to smithi12313934-18 2020-01-16T00:13:56.610 INFO:tasks.rados.rados.0.smithi123.stdout:1559: finishing write tid 2 to smithi12313934-18 2020-01-16T00:13:56.610 INFO:tasks.rados.rados.0.smithi123.stdout:1559: finishing write tid 3 to smithi12313934-18 2020-01-16T00:13:56.611 INFO:tasks.rados.rados.0.smithi123.stdout:1559: finishing write tid 5 to smithi12313934-18 2020-01-16T00:13:56.611 INFO:tasks.rados.rados.0.smithi123.stdout:1559: finishing write tid 6 to smithi12313934-18 2020-01-16T00:13:56.611 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 18 v 283 (ObjNum 533 snap 0 seq_num 533) dirty exists 2020-01-16T00:13:56.611 INFO:tasks.rados.rados.0.smithi123.stdout:1559: left oid 18 (ObjNum 533 snap 0 seq_num 533) 2020-01-16T00:13:56.611 INFO:tasks.rados.rados.0.smithi123.stdout:1560: finishing write tid 1 to smithi12313934-3 2020-01-16T00:13:56.611 INFO:tasks.rados.rados.0.smithi123.stdout:1560: finishing write tid 2 to smithi12313934-3 2020-01-16T00:13:56.611 INFO:tasks.rados.rados.0.smithi123.stdout:1560: finishing write tid 3 to smithi12313934-3 2020-01-16T00:13:56.612 INFO:tasks.rados.rados.0.smithi123.stdout:1560: finishing write tid 5 to smithi12313934-3 2020-01-16T00:13:56.612 INFO:tasks.rados.rados.0.smithi123.stdout:1560: finishing write tid 6 to smithi12313934-3 2020-01-16T00:13:56.612 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 3 v 116 (ObjNum 534 snap 0 seq_num 534) dirty exists 2020-01-16T00:13:56.612 INFO:tasks.rados.rados.0.smithi123.stdout:1560: left oid 3 (ObjNum 534 snap 0 seq_num 534) 2020-01-16T00:13:56.612 INFO:tasks.rados.rados.0.smithi123.stdout:1561: finishing write tid 1 to smithi12313934-39 2020-01-16T00:13:56.612 INFO:tasks.rados.rados.0.smithi123.stdout:1561: finishing write tid 2 to smithi12313934-39 2020-01-16T00:13:56.612 INFO:tasks.rados.rados.0.smithi123.stdout:1561: finishing write tid 4 to smithi12313934-39 2020-01-16T00:13:56.612 INFO:tasks.rados.rados.0.smithi123.stdout:1551: done (15 left) 2020-01-16T00:13:56.613 INFO:tasks.rados.rados.0.smithi123.stdout:1552: done (14 left) 2020-01-16T00:13:56.613 INFO:tasks.rados.rados.0.smithi123.stdout:1553: done (13 left) 2020-01-16T00:13:56.613 INFO:tasks.rados.rados.0.smithi123.stdout:1554: done (12 left) 2020-01-16T00:13:56.613 INFO:tasks.rados.rados.0.smithi123.stdout:1555: done (11 left) 2020-01-16T00:13:56.613 INFO:tasks.rados.rados.0.smithi123.stdout:1556: done (10 left) 2020-01-16T00:13:56.613 INFO:tasks.rados.rados.0.smithi123.stdout:1557: done (9 left) 2020-01-16T00:13:56.614 INFO:tasks.rados.rados.0.smithi123.stdout:1558: done (8 left) 2020-01-16T00:13:56.614 INFO:tasks.rados.rados.0.smithi123.stdout:1559: done (7 left) 2020-01-16T00:13:56.614 INFO:tasks.rados.rados.0.smithi123.stdout:1560: done (6 left) 2020-01-16T00:13:56.614 INFO:tasks.rados.rados.0.smithi123.stdout:1568: write (excl) oid 30 current snap is 0 2020-01-16T00:13:56.614 INFO:tasks.rados.rados.0.smithi123.stdout:1568: seq_num 540 ranges {675780=495523,1682747=714511,3012769=63011} 2020-01-16T00:13:56.615 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-01-16T00:13:56.621 INFO:tasks.rados.rados.0.smithi123.stdout:1568: writing smithi12313934-30 from 675780 to 1171303 tid 1 2020-01-16T00:13:56.628 INFO:tasks.rados.rados.0.smithi123.stdout:1568: writing smithi12313934-30 from 1682747 to 2397258 tid 2 2020-01-16T00:13:56.632 INFO:tasks.rados.rados.0.smithi123.stdout:1568: writing smithi12313934-30 from 3012769 to 3075780 tid 3 2020-01-16T00:13:56.632 INFO:tasks.rados.rados.0.smithi123.stdout:1569: write (excl) oid 16 current snap is 0 2020-01-16T00:13:56.632 INFO:tasks.rados.rados.0.smithi123.stdout:1569: seq_num 541 ranges {692587=570772,2024876=574522,3040192=52395} 2020-01-16T00:13:56.639 INFO:tasks.rados.rados.0.smithi123.stdout:1569: writing smithi12313934-16 from 692587 to 1263359 tid 1 2020-01-16T00:13:56.646 INFO:tasks.rados.rados.0.smithi123.stdout:1569: writing smithi12313934-16 from 2024876 to 2599398 tid 2 2020-01-16T00:13:56.649 INFO:tasks.rados.rados.0.smithi123.stdout:1569: writing smithi12313934-16 from 3040192 to 3092587 tid 3 2020-01-16T00:13:56.649 INFO:tasks.rados.rados.0.smithi123.stdout:1570: read oid 12 snap -1 2020-01-16T00:13:56.649 INFO:tasks.rados.rados.0.smithi123.stdout:1570: expect (ObjNum 517 snap 0 seq_num 517) 2020-01-16T00:13:56.649 INFO:tasks.rados.rados.0.smithi123.stdout:1571: write (excl) oid 21 current snap is 0 2020-01-16T00:13:56.649 INFO:tasks.rados.rados.0.smithi123.stdout:1571: seq_num 542 ranges {709394=646021,2083358=718180} 2020-01-16T00:13:56.654 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:56.652+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:56.656 INFO:tasks.rados.rados.0.smithi123.stdout:1571: writing smithi12313934-21 from 709394 to 1355415 tid 1 2020-01-16T00:13:56.664 INFO:tasks.rados.rados.0.smithi123.stdout:1571: writing smithi12313934-21 from 2083358 to 2801538 tid 2 2020-01-16T00:13:56.665 INFO:tasks.rados.rados.0.smithi123.stdout:1572: read oid 38 snap -1 2020-01-16T00:13:56.665 INFO:tasks.rados.rados.0.smithi123.stdout:1572: expect deleted 2020-01-16T00:13:56.665 INFO:tasks.rados.rados.0.smithi123.stdout:1573: write oid 15 current snap is 0 2020-01-16T00:13:56.665 INFO:tasks.rados.rados.0.smithi123.stdout:1561: finishing write tid 5 to smithi12313934-39 2020-01-16T00:13:56.665 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 39 v 242 (ObjNum 535 snap 0 seq_num 535) dirty exists 2020-01-16T00:13:56.665 INFO:tasks.rados.rados.0.smithi123.stdout:1561: left oid 39 (ObjNum 535 snap 0 seq_num 535) 2020-01-16T00:13:56.666 INFO:tasks.rados.rados.0.smithi123.stdout:1573: seq_num 543 ranges {726201=721270,2141840=578191} 2020-01-16T00:13:56.672 INFO:tasks.rados.rados.0.smithi123.stdout:1573: writing smithi12313934-15 from 726201 to 1447471 tid 1 2020-01-16T00:13:56.680 INFO:tasks.rados.rados.0.smithi123.stdout:1573: writing smithi12313934-15 from 2141840 to 2720031 tid 2 2020-01-16T00:13:56.680 INFO:tasks.rados.rados.0.smithi123.stdout:1561: done (11 left) 2020-01-16T00:13:56.680 INFO:tasks.rados.rados.0.smithi123.stdout:1574: read oid 34 snap -1 2020-01-16T00:13:56.680 INFO:tasks.rados.rados.0.smithi123.stdout:1574: expect (ObjNum 532 snap 0 seq_num 532) 2020-01-16T00:13:56.680 INFO:tasks.rados.rados.0.smithi123.stdout:1575: write (excl) oid 23 current snap is 0 2020-01-16T00:13:56.680 INFO:tasks.rados.rados.0.smithi123.stdout:1575: seq_num 544 ranges {743008=796519,2200322=721849} 2020-01-16T00:13:56.689 INFO:tasks.rados.rados.0.smithi123.stdout:1575: writing smithi12313934-23 from 743008 to 1539527 tid 1 2020-01-16T00:13:56.697 INFO:tasks.rados.rados.0.smithi123.stdout:1575: writing smithi12313934-23 from 2200322 to 2922171 tid 2 2020-01-16T00:13:56.697 INFO:tasks.rados.rados.0.smithi123.stdout:1576: read oid 37 snap -1 2020-01-16T00:13:56.697 INFO:tasks.rados.rados.0.smithi123.stdout:1576: expect (ObjNum 502 snap 0 seq_num 502) 2020-01-16T00:13:56.697 INFO:tasks.rados.rados.0.smithi123.stdout:1577: write (excl) oid 1 current snap is 0 2020-01-16T00:13:56.697 INFO:tasks.rados.rados.0.smithi123.stdout:1577: seq_num 545 ranges {759815=471768,1742451=465507,2917178=242637} 2020-01-16T00:13:56.703 INFO:tasks.rados.rados.0.smithi123.stdout:1577: writing smithi12313934-1 from 759815 to 1231583 tid 1 2020-01-16T00:13:56.709 INFO:tasks.rados.rados.0.smithi123.stdout:1577: writing smithi12313934-1 from 1742451 to 2207958 tid 2 2020-01-16T00:13:56.715 INFO:tasks.rados.rados.0.smithi123.stdout:1577: writing smithi12313934-1 from 2917178 to 3159815 tid 3 2020-01-16T00:13:56.716 INFO:tasks.rados.rados.0.smithi123.stdout:1578: read oid 20 snap -1 2020-01-16T00:13:56.716 INFO:tasks.rados.rados.0.smithi123.stdout:1578: expect deleted 2020-01-16T00:13:56.716 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:56.716 INFO:tasks.rados.rados.0.smithi123.stdout:1563: finishing write tid 1 to smithi12313934-14 2020-01-16T00:13:56.716 INFO:tasks.rados.rados.0.smithi123.stdout:1563: finishing write tid 2 to smithi12313934-14 2020-01-16T00:13:56.716 INFO:tasks.rados.rados.0.smithi123.stdout:1563: finishing write tid 3 to smithi12313934-14 2020-01-16T00:13:56.716 INFO:tasks.rados.rados.0.smithi123.stdout:1563: finishing write tid 5 to smithi12313934-14 2020-01-16T00:13:56.717 INFO:tasks.rados.rados.0.smithi123.stdout:1563: finishing write tid 6 to smithi12313934-14 2020-01-16T00:13:56.717 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 14 v 246 (ObjNum 537 snap 0 seq_num 537) dirty exists 2020-01-16T00:13:56.717 INFO:tasks.rados.rados.0.smithi123.stdout:1563: left oid 14 (ObjNum 537 snap 0 seq_num 537) 2020-01-16T00:13:56.717 INFO:tasks.rados.rados.0.smithi123.stdout:1564: finishing write tid 1 to smithi12313934-35 2020-01-16T00:13:56.717 INFO:tasks.rados.rados.0.smithi123.stdout:1563: done (15 left) 2020-01-16T00:13:56.717 INFO:tasks.rados.rados.0.smithi123.stdout:1579: write (excl) oid 26 current snap is 0 2020-01-16T00:13:56.717 INFO:tasks.rados.rados.0.smithi123.stdout:1579: seq_num 546 ranges {776622=547017,1800933=725518,2944601=232021} 2020-01-16T00:13:56.720 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:56.716+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:56.722 INFO:tasks.rados.rados.0.smithi123.stdout:1579: writing smithi12313934-26 from 776622 to 1323639 tid 1 2020-01-16T00:13:56.734 INFO:tasks.rados.rados.0.smithi123.stdout:1579: writing smithi12313934-26 from 1800933 to 2526451 tid 2 2020-01-16T00:13:56.735 INFO:tasks.rados.rados.0.smithi123.stdout:1579: writing smithi12313934-26 from 2944601 to 3176622 tid 3 2020-01-16T00:13:56.735 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:56.735 INFO:tasks.rados.rados.0.smithi123.stdout:1564: finishing write tid 2 to smithi12313934-35 2020-01-16T00:13:56.735 INFO:tasks.rados.rados.0.smithi123.stdout:1564: finishing write tid 4 to smithi12313934-35 2020-01-16T00:13:56.735 INFO:tasks.rados.rados.0.smithi123.stdout:1565: done (15 left) 2020-01-16T00:13:56.735 INFO:tasks.rados.rados.0.smithi123.stdout:1580: read oid 46 snap -1 2020-01-16T00:13:56.736 INFO:tasks.rados.rados.0.smithi123.stdout:1580: expect (ObjNum 498 snap 0 seq_num 498) 2020-01-16T00:13:56.736 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:56.736 INFO:tasks.rados.rados.0.smithi123.stdout:1564: finishing write tid 5 to smithi12313934-35 2020-01-16T00:13:56.736 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 35 v 167 (ObjNum 538 snap 0 seq_num 538) dirty exists 2020-01-16T00:13:56.736 INFO:tasks.rados.rados.0.smithi123.stdout:1564: left oid 35 (ObjNum 538 snap 0 seq_num 538) 2020-01-16T00:13:56.736 INFO:tasks.rados.rados.0.smithi123.stdout:1564: done (15 left) 2020-01-16T00:13:56.736 INFO:tasks.rados.rados.0.smithi123.stdout:1581: write (excl) oid 49 current snap is 0 2020-01-16T00:13:56.737 INFO:tasks.rados.rados.0.smithi123.stdout:1581: seq_num 547 ranges {793429=622266,1859415=469176,2972024=221405} 2020-01-16T00:13:56.741 INFO:tasks.rados.rados.0.smithi123.stdout:1581: writing smithi12313934-49 from 793429 to 1415695 tid 1 2020-01-16T00:13:56.746 INFO:tasks.rados.rados.0.smithi123.stdout:1581: writing smithi12313934-49 from 1859415 to 2328591 tid 2 2020-01-16T00:13:56.751 INFO:tasks.rados.rados.0.smithi123.stdout:1581: writing smithi12313934-49 from 2972024 to 3193429 tid 3 2020-01-16T00:13:56.752 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:56.752 INFO:tasks.rados.rados.0.smithi123.stdout:1566: expect (ObjNum 486 snap 0 seq_num 486) 2020-01-16T00:13:56.757 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:56.767 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-01-16T00:13:56.821 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:56.818+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:56.922 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:56.918+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:56.924 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:56.933 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-01-16T00:13:56.944 INFO:tasks.rados.rados.0.smithi123.stdout:1567: finishing write tid 1 to smithi12313934-43 2020-01-16T00:13:56.944 INFO:tasks.rados.rados.0.smithi123.stdout:1567: finishing write tid 2 to smithi12313934-43 2020-01-16T00:13:56.944 INFO:tasks.rados.rados.0.smithi123.stdout:1567: finishing write tid 3 to smithi12313934-43 2020-01-16T00:13:56.944 INFO:tasks.rados.rados.0.smithi123.stdout:1567: finishing write tid 5 to smithi12313934-43 2020-01-16T00:13:56.944 INFO:tasks.rados.rados.0.smithi123.stdout:1567: finishing write tid 6 to smithi12313934-43 2020-01-16T00:13:56.944 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 43 v 250 (ObjNum 539 snap 0 seq_num 539) dirty exists 2020-01-16T00:13:56.944 INFO:tasks.rados.rados.0.smithi123.stdout:1567: left oid 43 (ObjNum 539 snap 0 seq_num 539) 2020-01-16T00:13:56.945 INFO:tasks.rados.rados.0.smithi123.stdout:1568: finishing write tid 1 to smithi12313934-30 2020-01-16T00:13:56.945 INFO:tasks.rados.rados.0.smithi123.stdout:1568: finishing write tid 2 to smithi12313934-30 2020-01-16T00:13:56.945 INFO:tasks.rados.rados.0.smithi123.stdout:1568: finishing write tid 3 to smithi12313934-30 2020-01-16T00:13:56.945 INFO:tasks.rados.rados.0.smithi123.stdout:1568: finishing write tid 5 to smithi12313934-30 2020-01-16T00:13:56.945 INFO:tasks.rados.rados.0.smithi123.stdout:1566: done (15 left) 2020-01-16T00:13:56.945 INFO:tasks.rados.rados.0.smithi123.stdout:1567: done (14 left) 2020-01-16T00:13:56.945 INFO:tasks.rados.rados.0.smithi123.stdout:1582: write (excl) oid 13 current snap is 0 2020-01-16T00:13:56.945 INFO:tasks.rados.rados.0.smithi123.stdout:1582: seq_num 548 ranges {410236=413868,1234250=729187,2715800=423063} 2020-01-16T00:13:56.956 INFO:tasks.rados.rados.0.smithi123.stdout:1582: writing smithi12313934-13 from 410236 to 824104 tid 1 2020-01-16T00:13:56.964 INFO:tasks.rados.rados.0.smithi123.stdout:1582: writing smithi12313934-13 from 1234250 to 1963437 tid 2 2020-01-16T00:13:56.971 INFO:tasks.rados.rados.0.smithi123.stdout:1582: writing smithi12313934-13 from 2715800 to 3138863 tid 3 2020-01-16T00:13:56.971 INFO:tasks.rados.rados.0.smithi123.stdout:1583: read oid 8 snap -1 2020-01-16T00:13:56.972 INFO:tasks.rados.rados.0.smithi123.stdout:1583: expect (ObjNum 518 snap 0 seq_num 518) 2020-01-16T00:13:56.972 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:56.972 INFO:tasks.rados.rados.0.smithi123.stdout:1568: finishing write tid 6 to smithi12313934-30 2020-01-16T00:13:56.972 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 30 v 170 (ObjNum 540 snap 0 seq_num 540) dirty exists 2020-01-16T00:13:56.972 INFO:tasks.rados.rados.0.smithi123.stdout:1568: left oid 30 (ObjNum 540 snap 0 seq_num 540) 2020-01-16T00:13:56.972 INFO:tasks.rados.rados.0.smithi123.stdout:1568: done (15 left) 2020-01-16T00:13:56.972 INFO:tasks.rados.rados.0.smithi123.stdout:1584: write oid 44 current snap is 0 2020-01-16T00:13:56.972 INFO:tasks.rados.rados.0.smithi123.stdout:1584: seq_num 549 ranges {427043=489117,1576379=472845,2626870=600173} 2020-01-16T00:13:56.977 INFO:tasks.rados.rados.0.smithi123.stdout:1584: writing smithi12313934-44 from 427043 to 916160 tid 1 2020-01-16T00:13:56.984 INFO:tasks.rados.rados.0.smithi123.stdout:1584: writing smithi12313934-44 from 1576379 to 2049224 tid 2 2020-01-16T00:13:56.991 INFO:tasks.rados.rados.0.smithi123.stdout:1584: writing smithi12313934-44 from 2626870 to 3227043 tid 3 2020-01-16T00:13:56.991 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:56.992 INFO:tasks.rados.rados.0.smithi123.stdout:1569: finishing write tid 1 to smithi12313934-16 2020-01-16T00:13:56.992 INFO:tasks.rados.rados.0.smithi123.stdout:1569: finishing write tid 2 to smithi12313934-16 2020-01-16T00:13:56.992 INFO:tasks.rados.rados.0.smithi123.stdout:1569: finishing write tid 3 to smithi12313934-16 2020-01-16T00:13:56.992 INFO:tasks.rados.rados.0.smithi123.stdout:1569: finishing write tid 5 to smithi12313934-16 2020-01-16T00:13:56.992 INFO:tasks.rados.rados.0.smithi123.stdout:1569: finishing write tid 6 to smithi12313934-16 2020-01-16T00:13:56.992 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 16 v 236 (ObjNum 541 snap 0 seq_num 541) dirty exists 2020-01-16T00:13:56.992 INFO:tasks.rados.rados.0.smithi123.stdout:1569: left oid 16 (ObjNum 541 snap 0 seq_num 541) 2020-01-16T00:13:56.993 INFO:tasks.rados.rados.0.smithi123.stdout:1570: expect (ObjNum 517 snap 0 seq_num 517) 2020-01-16T00:13:57.023 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:57.022+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:57.049 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:57.058 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-01-16T00:13:57.085 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2020-01-16T00:13:57.124 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:57.120+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:57.162 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:57.172 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-01-16T00:13:57.224 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:57.222+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:57.285 INFO:tasks.rados.rados.0.smithi123.stdout:1571: finishing write tid 1 to smithi12313934-21 2020-01-16T00:13:57.286 INFO:tasks.rados.rados.0.smithi123.stdout:1571: finishing write tid 2 to smithi12313934-21 2020-01-16T00:13:57.286 INFO:tasks.rados.rados.0.smithi123.stdout:1571: finishing write tid 4 to smithi12313934-21 2020-01-16T00:13:57.286 INFO:tasks.rados.rados.0.smithi123.stdout:1571: finishing write tid 5 to smithi12313934-21 2020-01-16T00:13:57.286 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 21 v 72 (ObjNum 542 snap 0 seq_num 542) dirty exists 2020-01-16T00:13:57.286 INFO:tasks.rados.rados.0.smithi123.stdout:1571: left oid 21 (ObjNum 542 snap 0 seq_num 542) 2020-01-16T00:13:57.286 INFO:tasks.rados.rados.0.smithi123.stdout:1573: finishing write tid 1 to smithi12313934-15 2020-01-16T00:13:57.286 INFO:tasks.rados.rados.0.smithi123.stdout:1573: finishing write tid 2 to smithi12313934-15 2020-01-16T00:13:57.287 INFO:tasks.rados.rados.0.smithi123.stdout:1573: finishing write tid 4 to smithi12313934-15 2020-01-16T00:13:57.287 INFO:tasks.rados.rados.0.smithi123.stdout:1573: finishing write tid 5 to smithi12313934-15 2020-01-16T00:13:57.287 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 15 v 194 (ObjNum 543 snap 0 seq_num 543) dirty exists 2020-01-16T00:13:57.287 INFO:tasks.rados.rados.0.smithi123.stdout:1573: left oid 15 (ObjNum 543 snap 0 seq_num 543) 2020-01-16T00:13:57.287 INFO:tasks.rados.rados.0.smithi123.stdout:1574: expect (ObjNum 532 snap 0 seq_num 532) 2020-01-16T00:13:57.324 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:57.325 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:57.322+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:57.334 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-01-16T00:13:57.382 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-01-16T00:13:57.427 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:57.426+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:57.435 INFO:tasks.rados.rados.0.smithi123.stdout:1575: finishing write tid 1 to smithi12313934-23 2020-01-16T00:13:57.435 INFO:tasks.rados.rados.0.smithi123.stdout:1576: expect (ObjNum 502 snap 0 seq_num 502) 2020-01-16T00:13:57.439 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:13:57.448 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-01-16T00:13:57.509 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:13:57.538 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:57.522+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:57.539 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:13:57.553 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:13:57.576 INFO:tasks.rados.rados.0.smithi123.stdout:1575: finishing write tid 2 to smithi12313934-23 2020-01-16T00:13:57.576 INFO:tasks.rados.rados.0.smithi123.stdout:1575: finishing write tid 4 to smithi12313934-23 2020-01-16T00:13:57.577 INFO:tasks.rados.rados.0.smithi123.stdout:1575: finishing write tid 5 to smithi12313934-23 2020-01-16T00:13:57.577 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 23 v 170 (ObjNum 544 snap 0 seq_num 544) dirty exists 2020-01-16T00:13:57.577 INFO:tasks.rados.rados.0.smithi123.stdout:1575: left oid 23 (ObjNum 544 snap 0 seq_num 544) 2020-01-16T00:13:57.577 INFO:tasks.rados.rados.0.smithi123.stdout:1577: finishing write tid 1 to smithi12313934-1 2020-01-16T00:13:57.577 INFO:tasks.rados.rados.0.smithi123.stdout:1577: finishing write tid 2 to smithi12313934-1 2020-01-16T00:13:57.577 INFO:tasks.rados.rados.0.smithi123.stdout:1569: done (15 left) 2020-01-16T00:13:57.577 INFO:tasks.rados.rados.0.smithi123.stdout:1570: done (14 left) 2020-01-16T00:13:57.578 INFO:tasks.rados.rados.0.smithi123.stdout:1571: done (13 left) 2020-01-16T00:13:57.578 INFO:tasks.rados.rados.0.smithi123.stdout:1572: done (12 left) 2020-01-16T00:13:57.578 INFO:tasks.rados.rados.0.smithi123.stdout:1573: done (11 left) 2020-01-16T00:13:57.578 INFO:tasks.rados.rados.0.smithi123.stdout:1574: done (10 left) 2020-01-16T00:13:57.578 INFO:tasks.rados.rados.0.smithi123.stdout:1575: done (9 left) 2020-01-16T00:13:57.578 INFO:tasks.rados.rados.0.smithi123.stdout:1576: done (8 left) 2020-01-16T00:13:57.578 INFO:tasks.rados.rados.0.smithi123.stdout:1585: read oid 25 snap -1 2020-01-16T00:13:57.578 INFO:tasks.rados.rados.0.smithi123.stdout:1585: expect (ObjNum 515 snap 0 seq_num 515) 2020-01-16T00:13:57.579 INFO:tasks.rados.rados.0.smithi123.stdout:1586: write oid 47 current snap is 0 2020-01-16T00:13:57.579 INFO:tasks.rados.rados.0.smithi123.stdout:1586: seq_num 550 ranges {443850=564366,1634861=732856,2770646=473204} 2020-01-16T00:13:57.579 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:13:57.588 INFO:tasks.rados.rados.0.smithi123.stdout:1586: writing smithi12313934-47 from 443850 to 1008216 tid 1 2020-01-16T00:13:57.595 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:57.595 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:57.595 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:13:57.595 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:13:57.595 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:57.597 INFO:tasks.rados.rados.0.smithi123.stdout:1586: writing smithi12313934-47 from 1634861 to 2367717 tid 2 2020-01-16T00:13:57.600 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:13:57.604 INFO:tasks.rados.rados.0.smithi123.stdout:1586: writing smithi12313934-47 from 2770646 to 3243850 tid 3 2020-01-16T00:13:57.605 INFO:tasks.rados.rados.0.smithi123.stdout:1587: write (excl) oid 14 current snap is 0 2020-01-16T00:13:57.605 INFO:tasks.rados.rados.0.smithi123.stdout:1587: seq_num 551 ranges {460657=639615,1693343=476514,2681716=578941} 2020-01-16T00:13:57.605 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-01-16T00:13:57.609 INFO:tasks.rados.rados.0.smithi123.stdout:1587: writing smithi12313934-14 from 460657 to 1100272 tid 1 2020-01-16T00:13:57.615 INFO:tasks.rados.rados.0.smithi123.stdout:1587: writing smithi12313934-14 from 1693343 to 2169857 tid 2 2020-01-16T00:13:57.622 INFO:tasks.rados.rados.0.smithi123.stdout:1587: writing smithi12313934-14 from 2681716 to 3260657 tid 3 2020-01-16T00:13:57.622 INFO:tasks.rados.rados.0.smithi123.stdout:1588: delete oid 16 current snap is 0 2020-01-16T00:13:57.622 INFO:tasks.rados.rados.0.smithi123.stdout:1577: finishing write tid 3 to smithi12313934-1 2020-01-16T00:13:57.622 INFO:tasks.rados.rados.0.smithi123.stdout:1577: finishing write tid 5 to smithi12313934-1 2020-01-16T00:13:57.622 INFO:tasks.rados.rados.0.smithi123.stdout:1577: finishing write tid 6 to smithi12313934-1 2020-01-16T00:13:57.622 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 1 v 287 (ObjNum 545 snap 0 seq_num 545) dirty exists 2020-01-16T00:13:57.622 INFO:tasks.rados.rados.0.smithi123.stdout:1577: left oid 1 (ObjNum 545 snap 0 seq_num 545) 2020-01-16T00:13:57.623 INFO:tasks.rados.rados.0.smithi123.stdout:1579: finishing write tid 1 to smithi12313934-26 2020-01-16T00:13:57.623 INFO:tasks.rados.rados.0.smithi123.stdout:1580: expect (ObjNum 498 snap 0 seq_num 498) 2020-01-16T00:13:57.623 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:13:57.627 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:57.624+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:57.648 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:13:57.650 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-01-16T00:13:57.650 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-01-16T00:13:57.650 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-01-16T00:13:57.650 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-01-16T00:13:57.650 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-01-16T00:13:57.651 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENXIO: problem getting command descriptions from osd.5 2020-01-16T00:13:57.651 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: problem getting command descriptions from osd.5 2020-01-16T00:13:57.651 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-01-16T00:13:57.651 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-01-16T00:13:57.660 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-01-16T00:13:57.660 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-01-16T00:13:57.704 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:57.704 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:13:57.704 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:13:57.704 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:13:57.704 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.705 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:939 3.4 3:2bdd9e88:::smithi12313934-4:head [write 1681102~474010] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:57.705 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.908392+0000", 2020-01-16T00:13:57.705 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 79.793916397000004, 2020-01-16T00:13:57.705 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019294697, 2020-01-16T00:13:57.705 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:57.705 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:57.705 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:57.705 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:57.706 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:57.706 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 939 2020-01-16T00:13:57.706 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.706 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:57.706 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.706 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:57.706 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908392+0000", 2020-01-16T00:13:57.707 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:57.707 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.707 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.707 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:57.707 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908392+0000", 2020-01-16T00:13:57.707 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:57.707 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.707 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.708 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:57.708 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908390+0000", 2020-01-16T00:13:57.708 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015440399999999999 2020-01-16T00:13:57.708 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.708 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.708 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:57.708 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908545+0000", 2020-01-16T00:13:57.708 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3926e-05 2020-01-16T00:13:57.709 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.709 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.709 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:57.709 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908559+0000", 2020-01-16T00:13:57.709 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2200000000000003e-06 2020-01-16T00:13:57.709 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.709 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.710 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:57.710 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908566+0000", 2020-01-16T00:13:57.712 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020594799999999998 2020-01-16T00:13:57.712 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.713 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.713 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.713 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910625+0000", 2020-01-16T00:13:57.713 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118672 2020-01-16T00:13:57.713 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.714 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.714 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:57.714 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910744+0000", 2020-01-16T00:13:57.714 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031940969999999999 2020-01-16T00:13:57.714 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.715 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.715 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.715 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.913938+0000", 2020-01-16T00:13:57.715 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000309621 2020-01-16T00:13:57.715 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.715 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.715 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:57.716 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.914248+0000", 2020-01-16T00:13:57.716 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.008818579 2020-01-16T00:13:57.716 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.716 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.716 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.716 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.923066+0000", 2020-01-16T00:13:57.716 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000108989 2020-01-16T00:13:57.716 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.717 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.719 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:57.719 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.923175+0000", 2020-01-16T00:13:57.719 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019128299999999999 2020-01-16T00:13:57.719 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.719 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.719 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:13:57.719 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.923367+0000", 2020-01-16T00:13:57.720 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0032083760000000002 2020-01-16T00:13:57.720 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.720 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.720 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:57.720 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926575+0000", 2020-01-16T00:13:57.720 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2475999999999999e-05 2020-01-16T00:13:57.721 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.721 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.721 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:57.721 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926607+0000", 2020-01-16T00:13:57.721 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00099399800000000006 2020-01-16T00:13:57.721 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.721 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.721 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:57.722 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927601+0000", 2020-01-16T00:13:57.722 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0958999999999999e-05 2020-01-16T00:13:57.722 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.722 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.722 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:57.722 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927642+0000", 2020-01-16T00:13:57.722 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4747999999999999e-05 2020-01-16T00:13:57.723 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.723 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.723 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:57.723 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927687+0000", 2020-01-16T00:13:57.723 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019294697 2020-01-16T00:13:57.723 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:57.723 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:57.723 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:57.724 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.724 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.724 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:940 3.4 3:2bdd9e88:::smithi12313934-4:head [write 2687912~589420] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:57.724 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.915144+0000", 2020-01-16T00:13:57.724 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 79.787165028999993, 2020-01-16T00:13:57.724 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016092538999999999, 2020-01-16T00:13:57.724 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:57.724 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:57.725 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:57.725 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:57.725 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:57.725 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 940 2020-01-16T00:13:57.725 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.725 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:57.725 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.726 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:57.726 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915144+0000", 2020-01-16T00:13:57.726 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:57.726 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.726 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.726 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:57.726 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915144+0000", 2020-01-16T00:13:57.726 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:57.727 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.727 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.727 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:57.727 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915141+0000", 2020-01-16T00:13:57.727 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016366300000000001 2020-01-16T00:13:57.727 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.727 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.727 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:57.728 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915305+0000", 2020-01-16T00:13:57.728 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4847e-05 2020-01-16T00:13:57.728 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.729 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.729 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:57.729 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915320+0000", 2020-01-16T00:13:57.729 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2599999999999999e-06 2020-01-16T00:13:57.729 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.729 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.730 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:57.730 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915327+0000", 2020-01-16T00:13:57.730 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4282000000000005e-05 2020-01-16T00:13:57.730 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.730 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.730 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.730 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915401+0000", 2020-01-16T00:13:57.731 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4554000000000006e-05 2020-01-16T00:13:57.731 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.731 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.731 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:57.731 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915466+0000", 2020-01-16T00:13:57.731 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0090821900000000004 2020-01-16T00:13:57.731 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.731 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.732 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.732 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.924548+0000", 2020-01-16T00:13:57.732 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000114696 2020-01-16T00:13:57.732 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.732 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.732 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:57.732 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.924663+0000", 2020-01-16T00:13:57.732 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000168651 2020-01-16T00:13:57.733 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.733 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.733 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:13:57.733 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.924832+0000", 2020-01-16T00:13:57.733 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018253430000000001 2020-01-16T00:13:57.735 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.735 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.735 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:57.735 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926657+0000", 2020-01-16T00:13:57.736 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9957999999999999e-05 2020-01-16T00:13:57.736 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.736 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.736 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:57.736 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926697+0000", 2020-01-16T00:13:57.736 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0044685250000000001 2020-01-16T00:13:57.736 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.737 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.737 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:57.737 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931165+0000", 2020-01-16T00:13:57.737 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9211000000000001e-05 2020-01-16T00:13:57.737 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.737 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.737 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:57.738 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931205+0000", 2020-01-16T00:13:57.738 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1699999999999998e-05 2020-01-16T00:13:57.739 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.739 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.739 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:57.739 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931236+0000", 2020-01-16T00:13:57.739 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016092538999999999 2020-01-16T00:13:57.739 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:57.739 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:57.740 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:57.740 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.740 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.740 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:941 3.4 3:2bdd9e88:::smithi12313934-4:head [setxattr _header (55) in=7b,truncate 3277332] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:57.740 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.915341+0000", 2020-01-16T00:13:57.740 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 79.786968211000001, 2020-01-16T00:13:57.740 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016603769000000001, 2020-01-16T00:13:57.740 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:57.741 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:57.741 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:57.741 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:57.741 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:57.741 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 941 2020-01-16T00:13:57.741 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.741 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:57.742 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.742 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:57.742 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915341+0000", 2020-01-16T00:13:57.742 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:57.742 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.742 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.742 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:57.742 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915341+0000", 2020-01-16T00:13:57.743 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:57.743 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.743 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.743 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:57.743 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915340+0000", 2020-01-16T00:13:57.743 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.896e-06 2020-01-16T00:13:57.743 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.743 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.744 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:57.744 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915346+0000", 2020-01-16T00:13:57.744 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2730000000000002e-06 2020-01-16T00:13:57.744 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.744 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.744 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:57.744 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915354+0000", 2020-01-16T00:13:57.745 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0616e-05 2020-01-16T00:13:57.745 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.745 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.745 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:57.745 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915374+0000", 2020-01-16T00:13:57.745 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000133487 2020-01-16T00:13:57.745 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.745 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.746 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.746 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915508+0000", 2020-01-16T00:13:57.746 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8971000000000003e-05 2020-01-16T00:13:57.746 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.746 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.746 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:57.746 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915557+0000", 2020-01-16T00:13:57.747 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010138787999999999 2020-01-16T00:13:57.749 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.749 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.749 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.749 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.925696+0000", 2020-01-16T00:13:57.749 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012825400000000001 2020-01-16T00:13:57.749 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.752 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.752 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:57.752 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.925824+0000", 2020-01-16T00:13:57.752 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017706500000000001 2020-01-16T00:13:57.753 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.753 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.754 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:13:57.754 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926001+0000", 2020-01-16T00:13:57.754 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00079988900000000003 2020-01-16T00:13:57.755 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.755 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.755 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:57.755 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926801+0000", 2020-01-16T00:13:57.755 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119324 2020-01-16T00:13:57.755 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.755 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.755 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:57.756 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926920+0000", 2020-01-16T00:13:57.756 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048879910000000004 2020-01-16T00:13:57.756 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.756 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.756 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:57.756 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931808+0000", 2020-01-16T00:13:57.756 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6999999999999997e-05 2020-01-16T00:13:57.756 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.757 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.757 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:57.757 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931855+0000", 2020-01-16T00:13:57.757 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9092999999999996e-05 2020-01-16T00:13:57.757 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.757 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.757 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:57.757 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931944+0000", 2020-01-16T00:13:57.758 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016603769000000001 2020-01-16T00:13:57.758 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:57.758 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:57.758 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:57.758 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.758 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.758 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:942 3.4 3:2bdd9e88:::smithi12313934-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:13:57.758 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.915389+0000", 2020-01-16T00:13:57.759 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 79.786919845, 2020-01-16T00:13:57.759 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016820563, 2020-01-16T00:13:57.759 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:57.759 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:57.759 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:57.759 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:57.759 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:57.759 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 942 2020-01-16T00:13:57.760 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.760 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:57.760 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.760 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:57.760 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915389+0000", 2020-01-16T00:13:57.760 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:57.761 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.761 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.761 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:57.761 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915389+0000", 2020-01-16T00:13:57.761 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:57.761 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.761 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.762 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:57.762 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915388+0000", 2020-01-16T00:13:57.762 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.683e-06 2020-01-16T00:13:57.762 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.762 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.762 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:57.763 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915391+0000", 2020-01-16T00:13:57.763 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4839999999999998e-06 2020-01-16T00:13:57.763 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.763 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.763 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:57.763 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915399+0000", 2020-01-16T00:13:57.763 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2259999999999999e-06 2020-01-16T00:13:57.764 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.764 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.764 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:57.764 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915403+0000", 2020-01-16T00:13:57.764 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00031301199999999999 2020-01-16T00:13:57.764 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.764 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.764 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.765 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915716+0000", 2020-01-16T00:13:57.765 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2079999999999998e-05 2020-01-16T00:13:57.765 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.765 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.765 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:57.765 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915748+0000", 2020-01-16T00:13:57.765 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010547076000000001 2020-01-16T00:13:57.765 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.766 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.766 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.766 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926295+0000", 2020-01-16T00:13:57.766 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8040000000000005e-05 2020-01-16T00:13:57.766 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.766 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.766 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:57.766 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926373+0000", 2020-01-16T00:13:57.767 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056061990000000001 2020-01-16T00:13:57.767 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.767 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.767 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.767 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931979+0000", 2020-01-16T00:13:57.767 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011540399999999999 2020-01-16T00:13:57.767 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.768 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.768 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:57.768 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932095+0000", 2020-01-16T00:13:57.768 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011482899999999999 2020-01-16T00:13:57.768 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.768 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.768 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:57.768 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932210+0000", 2020-01-16T00:13:57.768 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016820563 2020-01-16T00:13:57.769 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:57.769 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:57.769 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:57.769 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.769 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.769 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:957 3.0 3:00c146d5:::smithi12313934-18:head [write 510946~632952] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:57.769 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.033290+0000", 2020-01-16T00:13:57.769 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 79.669019008000006, 2020-01-16T00:13:57.770 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016764102999999999, 2020-01-16T00:13:57.770 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:57.770 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:57.770 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:57.770 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:57.770 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:57.770 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 957 2020-01-16T00:13:57.771 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.771 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:57.771 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.771 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:57.771 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033290+0000", 2020-01-16T00:13:57.771 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:57.771 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.772 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.772 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:57.772 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033290+0000", 2020-01-16T00:13:57.772 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999952 2020-01-16T00:13:57.772 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.772 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.772 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:57.772 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033285+0000", 2020-01-16T00:13:57.773 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000405917 2020-01-16T00:13:57.773 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.773 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.773 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:57.773 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033691+0000", 2020-01-16T00:13:57.773 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6585999999999997e-05 2020-01-16T00:13:57.773 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.773 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.774 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:57.774 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033738+0000", 2020-01-16T00:13:57.774 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8374000000000001e-05 2020-01-16T00:13:57.774 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.774 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.774 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:57.774 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033756+0000", 2020-01-16T00:13:57.774 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013002900000000001 2020-01-16T00:13:57.774 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.775 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.775 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.775 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033886+0000", 2020-01-16T00:13:57.775 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035882600000000002 2020-01-16T00:13:57.775 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.775 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.775 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:57.776 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.034245+0000", 2020-01-16T00:13:57.776 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0072798489999999997 2020-01-16T00:13:57.776 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.776 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.776 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.776 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.041525+0000", 2020-01-16T00:13:57.776 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000140488 2020-01-16T00:13:57.776 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.777 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.777 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:57.777 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.041665+0000", 2020-01-16T00:13:57.777 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000233766 2020-01-16T00:13:57.777 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.777 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.777 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:57.778 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.041899+0000", 2020-01-16T00:13:57.778 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0035766539999999999 2020-01-16T00:13:57.778 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.778 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.778 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:57.778 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.045475+0000", 2020-01-16T00:13:57.778 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013333150000000001 2020-01-16T00:13:57.778 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.779 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.779 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:57.779 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.046809+0000", 2020-01-16T00:13:57.779 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031447860000000001 2020-01-16T00:13:57.779 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.779 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.779 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:57.780 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.049954+0000", 2020-01-16T00:13:57.780 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2318000000000003e-05 2020-01-16T00:13:57.780 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.780 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.780 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:57.780 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.050006+0000", 2020-01-16T00:13:57.780 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7972000000000001e-05 2020-01-16T00:13:57.781 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.781 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.781 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:57.781 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.050054+0000", 2020-01-16T00:13:57.781 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016764102999999999 2020-01-16T00:13:57.781 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:57.781 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:57.781 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:57.782 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.782 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.782 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1639 3.0 3:00c146d5:::smithi12313934-18:head [write 423576~401598] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:13:57.782 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.774317+0000", 2020-01-16T00:13:57.782 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 65.927992269000001, 2020-01-16T00:13:57.782 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018305219000000001, 2020-01-16T00:13:57.782 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:57.782 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:57.783 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:57.783 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:57.783 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:57.783 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1639 2020-01-16T00:13:57.783 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.783 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:57.783 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.783 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:57.784 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774317+0000", 2020-01-16T00:13:57.784 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:57.784 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.784 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.784 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:57.784 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774317+0000", 2020-01-16T00:13:57.784 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:57.784 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.785 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.785 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:57.785 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774313+0000", 2020-01-16T00:13:57.785 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000111173 2020-01-16T00:13:57.785 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.785 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.785 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:57.785 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774425+0000", 2020-01-16T00:13:57.786 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7598999999999999e-05 2020-01-16T00:13:57.786 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.786 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.786 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:57.786 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774442+0000", 2020-01-16T00:13:57.786 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0830000000000004e-06 2020-01-16T00:13:57.786 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.787 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.787 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:57.787 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774450+0000", 2020-01-16T00:13:57.787 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0017886309999999999 2020-01-16T00:13:57.787 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.787 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.787 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.788 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.776239+0000", 2020-01-16T00:13:57.788 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3789e-05 2020-01-16T00:13:57.788 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.788 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.788 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op must wait for map", 2020-01-16T00:13:57.788 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.776253+0000", 2020-01-16T00:13:57.788 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095730729999999997 2020-01-16T00:13:57.788 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.789 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.789 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.789 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.785826+0000", 2020-01-16T00:13:57.789 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014220300000000001 2020-01-16T00:13:57.789 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.789 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.789 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:57.789 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.785968+0000", 2020-01-16T00:13:57.790 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023618900000000001 2020-01-16T00:13:57.790 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.790 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.790 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:57.790 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.786204+0000", 2020-01-16T00:13:57.790 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0062297389999999998 2020-01-16T00:13:57.790 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.790 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.791 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:57.791 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792434+0000", 2020-01-16T00:13:57.791 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9635999999999998e-05 2020-01-16T00:13:57.791 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.791 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.791 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:57.792 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792473+0000", 2020-01-16T00:13:57.792 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6456000000000002e-05 2020-01-16T00:13:57.792 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.792 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.792 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:57.792 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792540+0000", 2020-01-16T00:13:57.792 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9873999999999998e-05 2020-01-16T00:13:57.792 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.792 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.793 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:57.793 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792580+0000", 2020-01-16T00:13:57.793 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1924e-05 2020-01-16T00:13:57.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.793 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.793 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:57.793 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792622+0000", 2020-01-16T00:13:57.794 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018305219000000001 2020-01-16T00:13:57.794 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:57.794 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:57.794 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:57.794 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.794 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.794 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1890 3.0 3:0b1f384f:::smithi12313934-33:head [write 2797915~596476] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:13:57.794 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.232534+0000", 2020-01-16T00:13:57.795 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 60.469774297999997, 2020-01-16T00:13:57.795 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016106137, 2020-01-16T00:13:57.795 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:57.795 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:57.795 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:57.795 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:57.795 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:57.795 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1890 2020-01-16T00:13:57.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.796 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:57.796 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.796 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:57.796 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232534+0000", 2020-01-16T00:13:57.796 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:57.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.796 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.797 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:57.797 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232534+0000", 2020-01-16T00:13:57.797 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:57.797 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.797 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.797 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:57.797 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232532+0000", 2020-01-16T00:13:57.798 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00024232100000000001 2020-01-16T00:13:57.798 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.798 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.798 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:57.798 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232774+0000", 2020-01-16T00:13:57.798 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4601e-05 2020-01-16T00:13:57.798 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.799 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.799 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:57.799 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232789+0000", 2020-01-16T00:13:57.799 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8410000000000006e-06 2020-01-16T00:13:57.799 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.799 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.799 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:57.799 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232798+0000", 2020-01-16T00:13:57.800 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9756999999999997e-05 2020-01-16T00:13:57.800 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.800 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.800 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.800 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232857+0000", 2020-01-16T00:13:57.800 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4678000000000007e-05 2020-01-16T00:13:57.800 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.800 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.801 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:57.801 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232922+0000", 2020-01-16T00:13:57.801 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033861020000000002 2020-01-16T00:13:57.801 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.801 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.801 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.802 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.236308+0000", 2020-01-16T00:13:57.802 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000568996 2020-01-16T00:13:57.802 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.802 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.802 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:57.802 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.236877+0000", 2020-01-16T00:13:57.802 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0085444240000000001 2020-01-16T00:13:57.802 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.802 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.803 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.803 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.245422+0000", 2020-01-16T00:13:57.803 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5595999999999998e-05 2020-01-16T00:13:57.803 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.803 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.803 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:57.803 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.245507+0000", 2020-01-16T00:13:57.803 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016984300000000001 2020-01-16T00:13:57.803 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.804 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.804 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:57.804 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.245677+0000", 2020-01-16T00:13:57.804 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001649495 2020-01-16T00:13:57.804 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.805 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.805 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:57.805 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247326+0000", 2020-01-16T00:13:57.805 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023308399999999999 2020-01-16T00:13:57.805 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.805 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.805 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:57.806 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247560+0000", 2020-01-16T00:13:57.806 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00099501800000000007 2020-01-16T00:13:57.806 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.806 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.806 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:57.806 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248555+0000", 2020-01-16T00:13:57.807 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5936999999999998e-05 2020-01-16T00:13:57.807 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.807 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.807 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:57.807 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248591+0000", 2020-01-16T00:13:57.807 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0108999999999997e-05 2020-01-16T00:13:57.807 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.807 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.807 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:57.808 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248641+0000", 2020-01-16T00:13:57.808 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016106137 2020-01-16T00:13:57.808 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:57.808 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:57.808 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:57.808 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.808 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.809 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1891 3.0 3:0b1f384f:::smithi12313934-33:head [setxattr _header (56) in=7b,truncate 3395014] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:13:57.809 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.232813+0000", 2020-01-16T00:13:57.809 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 60.469495496, 2020-01-16T00:13:57.809 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015999295, 2020-01-16T00:13:57.809 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:57.809 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:57.809 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:57.809 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:57.810 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:57.810 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1891 2020-01-16T00:13:57.810 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.810 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:57.810 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.810 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:57.810 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232813+0000", 2020-01-16T00:13:57.811 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:57.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.811 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.811 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:57.811 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232813+0000", 2020-01-16T00:13:57.811 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:57.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.812 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.812 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:57.812 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232812+0000", 2020-01-16T00:13:57.812 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.576e-06 2020-01-16T00:13:57.812 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.812 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.812 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:57.812 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232820+0000", 2020-01-16T00:13:57.813 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3079999999999997e-06 2020-01-16T00:13:57.813 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.813 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.813 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:57.813 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232829+0000", 2020-01-16T00:13:57.813 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7119999999999997e-06 2020-01-16T00:13:57.814 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.814 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.814 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:57.814 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232835+0000", 2020-01-16T00:13:57.814 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013438400000000001 2020-01-16T00:13:57.814 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.814 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.814 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.815 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232969+0000", 2020-01-16T00:13:57.815 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0562000000000001e-05 2020-01-16T00:13:57.815 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.815 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.815 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:57.815 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.233029+0000", 2020-01-16T00:13:57.815 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039354239999999999 2020-01-16T00:13:57.815 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.816 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.816 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.816 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.236965+0000", 2020-01-16T00:13:57.816 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0482999999999998e-05 2020-01-16T00:13:57.816 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.816 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.816 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:57.816 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.237015+0000", 2020-01-16T00:13:57.817 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0096086209999999995 2020-01-16T00:13:57.817 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.817 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.817 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.817 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.246624+0000", 2020-01-16T00:13:57.817 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6150999999999995e-05 2020-01-16T00:13:57.817 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.818 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.818 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:57.818 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.246720+0000", 2020-01-16T00:13:57.818 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015032899999999999 2020-01-16T00:13:57.818 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.818 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.818 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:57.819 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.246870+0000", 2020-01-16T00:13:57.819 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00064760700000000002 2020-01-16T00:13:57.819 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.819 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.819 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:57.819 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247518+0000", 2020-01-16T00:13:57.819 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000198531 2020-01-16T00:13:57.820 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.820 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.820 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:57.820 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247717+0000", 2020-01-16T00:13:57.820 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010189439999999999 2020-01-16T00:13:57.820 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.820 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.820 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:57.821 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248736+0000", 2020-01-16T00:13:57.821 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4863000000000001e-05 2020-01-16T00:13:57.821 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.821 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.821 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:57.821 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248770+0000", 2020-01-16T00:13:57.822 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2153000000000001e-05 2020-01-16T00:13:57.822 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.822 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.822 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:57.822 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248813+0000", 2020-01-16T00:13:57.822 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015999295 2020-01-16T00:13:57.822 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:57.822 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:57.823 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:57.823 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.823 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.823 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1892 3.0 3:0b1f384f:::smithi12313934-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e31)", 2020-01-16T00:13:57.823 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.232856+0000", 2020-01-16T00:13:57.823 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 60.469453272000003, 2020-01-16T00:13:57.823 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016171544, 2020-01-16T00:13:57.823 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:57.823 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:57.824 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:57.824 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:57.824 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:57.824 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1892 2020-01-16T00:13:57.824 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.824 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:57.824 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.824 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:57.825 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232856+0000", 2020-01-16T00:13:57.825 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:57.825 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.825 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.825 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:57.825 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232856+0000", 2020-01-16T00:13:57.825 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:57.825 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.825 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.826 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:57.826 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232855+0000", 2020-01-16T00:13:57.826 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8640000000000004e-06 2020-01-16T00:13:57.826 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.826 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.826 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:57.826 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232860+0000", 2020-01-16T00:13:57.826 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0000000000000001e-05 2020-01-16T00:13:57.827 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.827 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.827 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:57.827 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232870+0000", 2020-01-16T00:13:57.827 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7420000000000003e-06 2020-01-16T00:13:57.828 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.828 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.828 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:57.829 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232875+0000", 2020-01-16T00:13:57.829 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017933200000000001 2020-01-16T00:13:57.829 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.829 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.829 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.830 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.233055+0000", 2020-01-16T00:13:57.830 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6269e-05 2020-01-16T00:13:57.830 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.830 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.831 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:57.831 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.233101+0000", 2020-01-16T00:13:57.831 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039780090000000002 2020-01-16T00:13:57.831 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.832 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.832 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.832 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.237079+0000", 2020-01-16T00:13:57.832 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3677000000000001e-05 2020-01-16T00:13:57.832 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.833 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.833 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:57.833 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.237133+0000", 2020-01-16T00:13:57.834 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010036671 2020-01-16T00:13:57.834 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.834 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.834 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.834 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247169+0000", 2020-01-16T00:13:57.835 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010280699999999999 2020-01-16T00:13:57.835 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.837 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.837 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:57.837 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247272+0000", 2020-01-16T00:13:57.837 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001579708 2020-01-16T00:13:57.837 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.838 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.838 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.838 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248852+0000", 2020-01-16T00:13:57.838 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9704999999999996e-05 2020-01-16T00:13:57.839 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.839 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.839 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:57.839 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248942+0000", 2020-01-16T00:13:57.839 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5557999999999996e-05 2020-01-16T00:13:57.839 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.840 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.840 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:57.840 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.249027+0000", 2020-01-16T00:13:57.840 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016171544 2020-01-16T00:13:57.840 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:57.841 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:57.841 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:57.841 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.841 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.842 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1950 3.0 3:0b1f384f:::smithi12313934-33:head [write 1889359~541737] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-01-16T00:13:57.842 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.922819+0000", 2020-01-16T00:13:57.842 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 59.779489906999999, 2020-01-16T00:13:57.842 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017502846999999998, 2020-01-16T00:13:57.843 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:57.843 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:57.843 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:57.843 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:57.844 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:57.844 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1950 2020-01-16T00:13:57.844 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.844 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:57.844 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.845 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:57.845 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922819+0000", 2020-01-16T00:13:57.845 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:57.845 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.846 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.846 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:57.846 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922819+0000", 2020-01-16T00:13:57.846 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:57.847 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.847 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.847 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:57.847 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922818+0000", 2020-01-16T00:13:57.848 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000146408 2020-01-16T00:13:57.848 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.848 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.848 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:57.848 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922964+0000", 2020-01-16T00:13:57.849 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4720000000000001e-06 2020-01-16T00:13:57.849 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.849 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.849 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:57.850 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922974+0000", 2020-01-16T00:13:57.850 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0720000000000001e-06 2020-01-16T00:13:57.850 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.850 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.851 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:57.851 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922978+0000", 2020-01-16T00:13:57.851 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6152000000000001e-05 2020-01-16T00:13:57.851 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.851 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.852 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.852 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.923024+0000", 2020-01-16T00:13:57.852 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00087368199999999997 2020-01-16T00:13:57.852 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.853 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.853 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:57.853 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.923897+0000", 2020-01-16T00:13:57.853 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011944435999999999 2020-01-16T00:13:57.853 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.854 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.854 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.854 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.935842+0000", 2020-01-16T00:13:57.854 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000144146 2020-01-16T00:13:57.854 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.855 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.855 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:57.855 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.935986+0000", 2020-01-16T00:13:57.855 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019775500000000001 2020-01-16T00:13:57.855 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.856 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.856 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:57.856 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.936184+0000", 2020-01-16T00:13:57.856 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002539342 2020-01-16T00:13:57.856 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.857 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.857 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:57.857 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938723+0000", 2020-01-16T00:13:57.857 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014193699999999999 2020-01-16T00:13:57.858 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.858 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.858 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:57.858 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938865+0000", 2020-01-16T00:13:57.859 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013829020000000001 2020-01-16T00:13:57.859 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.859 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.859 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:57.859 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.940248+0000", 2020-01-16T00:13:57.860 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2558000000000001e-05 2020-01-16T00:13:57.860 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.860 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.860 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:57.861 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.940291+0000", 2020-01-16T00:13:57.861 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1217999999999998e-05 2020-01-16T00:13:57.861 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.861 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.861 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:57.861 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.940322+0000", 2020-01-16T00:13:57.862 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017502846999999998 2020-01-16T00:13:57.862 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:57.862 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:57.862 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:57.862 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.863 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.863 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1951 3.0 3:0b1f384f:::smithi12313934-33:head [write 2955792~541902] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-01-16T00:13:57.863 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.929019+0000", 2020-01-16T00:13:57.863 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 59.773289757000001, 2020-01-16T00:13:57.863 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021625710999999999, 2020-01-16T00:13:57.864 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:57.864 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:57.864 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:57.864 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:57.864 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:57.864 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1951 2020-01-16T00:13:57.865 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.865 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:57.865 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.865 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:57.865 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929019+0000", 2020-01-16T00:13:57.866 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:57.866 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.866 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.866 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:57.866 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929019+0000", 2020-01-16T00:13:57.867 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999989 2020-01-16T00:13:57.867 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.867 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.867 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:57.867 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929008+0000", 2020-01-16T00:13:57.867 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000212513 2020-01-16T00:13:57.867 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.868 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.868 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:57.868 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929221+0000", 2020-01-16T00:13:57.868 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3229e-05 2020-01-16T00:13:57.868 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.868 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.868 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:57.868 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929234+0000", 2020-01-16T00:13:57.869 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9170000000000003e-06 2020-01-16T00:13:57.869 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.869 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.869 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:57.869 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929240+0000", 2020-01-16T00:13:57.869 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4045000000000002e-05 2020-01-16T00:13:57.869 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.870 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.870 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.870 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929294+0000", 2020-01-16T00:13:57.870 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9797999999999996e-05 2020-01-16T00:13:57.870 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.870 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.870 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:57.870 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929384+0000", 2020-01-16T00:13:57.871 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083450729999999997 2020-01-16T00:13:57.871 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.871 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.871 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.871 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.937729+0000", 2020-01-16T00:13:57.871 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043251699999999999 2020-01-16T00:13:57.871 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.871 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.872 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:57.872 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938161+0000", 2020-01-16T00:13:57.872 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095115489999999994 2020-01-16T00:13:57.872 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.872 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.872 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.872 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.947673+0000", 2020-01-16T00:13:57.873 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012395099999999999 2020-01-16T00:13:57.873 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.873 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.873 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:57.873 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.947797+0000", 2020-01-16T00:13:57.873 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00024849200000000001 2020-01-16T00:13:57.873 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.873 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.874 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:57.874 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.948045+0000", 2020-01-16T00:13:57.874 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0025081050000000001 2020-01-16T00:13:57.874 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.874 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.874 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:57.874 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950553+0000", 2020-01-16T00:13:57.875 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9760000000000003e-06 2020-01-16T00:13:57.875 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.875 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.875 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:57.875 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950558+0000", 2020-01-16T00:13:57.875 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3286999999999999e-05 2020-01-16T00:13:57.875 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.875 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.876 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:57.876 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950581+0000", 2020-01-16T00:13:57.876 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2759000000000003e-05 2020-01-16T00:13:57.876 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.876 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.876 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:57.876 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950614+0000", 2020-01-16T00:13:57.876 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0525999999999999e-05 2020-01-16T00:13:57.877 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.877 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.877 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:57.877 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950645+0000", 2020-01-16T00:13:57.877 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021625710999999999 2020-01-16T00:13:57.877 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:57.877 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:57.878 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:57.878 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.878 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.878 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1952 3.0 3:0b1f384f:::smithi12313934-33:head [setxattr _header (56) in=7b,truncate 3563084] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-01-16T00:13:57.878 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:13:57.878 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 59.77306042, 2020-01-16T00:13:57.878 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021534685000000001, 2020-01-16T00:13:57.878 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:57.879 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:57.879 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:57.879 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:57.879 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:57.879 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1952 2020-01-16T00:13:57.879 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.879 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:57.880 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.880 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:57.880 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:13:57.880 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:57.880 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.880 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.880 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:57.880 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:13:57.881 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:57.881 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.881 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.881 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:57.881 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:13:57.881 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6670000000000004e-06 2020-01-16T00:13:57.881 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.882 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.882 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:57.882 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929252+0000", 2020-01-16T00:13:57.882 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9380000000000002e-06 2020-01-16T00:13:57.882 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.882 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.883 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:57.883 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929256+0000", 2020-01-16T00:13:57.883 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.712e-06 2020-01-16T00:13:57.883 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.884 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.884 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:57.884 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929264+0000", 2020-01-16T00:13:57.884 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018030300000000001 2020-01-16T00:13:57.884 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.885 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.885 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.885 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929444+0000", 2020-01-16T00:13:57.885 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7562000000000002e-05 2020-01-16T00:13:57.885 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.886 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.886 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:57.886 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929502+0000", 2020-01-16T00:13:57.886 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0087297859999999998 2020-01-16T00:13:57.887 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.887 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.887 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.887 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938232+0000", 2020-01-16T00:13:57.887 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0452999999999996e-05 2020-01-16T00:13:57.888 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.888 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.888 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:57.888 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938322+0000", 2020-01-16T00:13:57.889 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011061728 2020-01-16T00:13:57.889 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.889 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.889 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.890 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.949384+0000", 2020-01-16T00:13:57.890 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000117828 2020-01-16T00:13:57.890 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.890 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.890 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:57.891 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.949502+0000", 2020-01-16T00:13:57.891 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022468499999999999 2020-01-16T00:13:57.891 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.891 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.892 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:57.892 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.949726+0000", 2020-01-16T00:13:57.892 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00083532099999999998 2020-01-16T00:13:57.892 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.892 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.893 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:57.893 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950562+0000", 2020-01-16T00:13:57.893 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013017700000000001 2020-01-16T00:13:57.893 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.894 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.894 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:57.894 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950692+0000", 2020-01-16T00:13:57.894 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9813000000000001e-05 2020-01-16T00:13:57.895 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.895 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.895 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:57.895 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950722+0000", 2020-01-16T00:13:57.895 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0592000000000001e-05 2020-01-16T00:13:57.896 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.896 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.896 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:57.896 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950752+0000", 2020-01-16T00:13:57.896 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0920000000000002e-05 2020-01-16T00:13:57.897 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.897 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.897 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:57.897 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950783+0000", 2020-01-16T00:13:57.898 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021534685000000001 2020-01-16T00:13:57.898 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:57.898 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:57.898 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:57.899 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.899 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.899 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1953 3.0 3:0b1f384f:::smithi12313934-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e32)", 2020-01-16T00:13:57.899 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:13:57.899 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 59.773027429000003, 2020-01-16T00:13:57.900 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021667537000000001, 2020-01-16T00:13:57.900 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:57.900 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:57.900 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:57.901 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:57.901 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:57.901 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1953 2020-01-16T00:13:57.901 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.901 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:57.902 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.902 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:57.902 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:13:57.902 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:57.903 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.903 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.903 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:57.903 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:13:57.904 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:57.904 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.904 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.904 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:57.904 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:13:57.905 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7030000000000002e-06 2020-01-16T00:13:57.905 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.905 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.905 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:57.906 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929284+0000", 2020-01-16T00:13:57.906 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0909999999999993e-06 2020-01-16T00:13:57.906 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.906 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.907 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:57.907 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929292+0000", 2020-01-16T00:13:57.907 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8359999999999999e-06 2020-01-16T00:13:57.907 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.907 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.908 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:57.908 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929297+0000", 2020-01-16T00:13:57.908 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023538199999999999 2020-01-16T00:13:57.908 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.909 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.909 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.909 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929532+0000", 2020-01-16T00:13:57.910 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1173000000000004e-05 2020-01-16T00:13:57.910 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.910 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.910 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:57.911 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929603+0000", 2020-01-16T00:13:57.911 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0087813509999999997 2020-01-16T00:13:57.911 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.911 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.911 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.911 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938384+0000", 2020-01-16T00:13:57.911 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9752000000000006e-05 2020-01-16T00:13:57.911 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.912 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.912 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:57.912 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938454+0000", 2020-01-16T00:13:57.912 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011707851999999999 2020-01-16T00:13:57.912 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.912 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.912 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.913 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950162+0000", 2020-01-16T00:13:57.913 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011060099999999999 2020-01-16T00:13:57.913 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.913 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.913 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:57.913 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950273+0000", 2020-01-16T00:13:57.913 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00053989700000000001 2020-01-16T00:13:57.913 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.914 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.914 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.914 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950813+0000", 2020-01-16T00:13:57.914 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8128999999999999e-05 2020-01-16T00:13:57.914 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.914 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.914 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:57.914 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950871+0000", 2020-01-16T00:13:57.915 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8242000000000001e-05 2020-01-16T00:13:57.915 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.915 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.915 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:57.915 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950949+0000", 2020-01-16T00:13:57.915 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021667537000000001 2020-01-16T00:13:57.915 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:57.915 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:57.916 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:57.916 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.916 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.917 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2555 3.0 3:0b70bb20:::smithi12313934-46:head [setxattr _header (56) in=7b,truncate 2991679] snapc 0=[] ondisk+write+known_if_redirected e37)", 2020-01-16T00:13:57.917 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.573432+0000", 2020-01-16T00:13:57.917 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 45.128876777999999, 2020-01-16T00:13:57.917 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019893926999999999, 2020-01-16T00:13:57.917 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:57.918 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:57.918 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:57.918 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:57.918 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:57.918 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2555 2020-01-16T00:13:57.918 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.918 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:57.918 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.919 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:57.919 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573432+0000", 2020-01-16T00:13:57.919 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:57.919 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.919 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.919 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:57.919 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573432+0000", 2020-01-16T00:13:57.920 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:57.920 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.920 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.920 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:57.920 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573432+0000", 2020-01-16T00:13:57.920 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4290000000000001e-06 2020-01-16T00:13:57.920 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.920 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.921 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:57.921 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573435+0000", 2020-01-16T00:13:57.921 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1300000000000001e-06 2020-01-16T00:13:57.921 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.921 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.921 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:57.921 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573438+0000", 2020-01-16T00:13:57.921 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9399999999999996e-06 2020-01-16T00:13:57.922 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.922 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.922 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:57.922 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573445+0000", 2020-01-16T00:13:57.922 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013947269999999999 2020-01-16T00:13:57.922 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.922 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.923 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.923 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.574840+0000", 2020-01-16T00:13:57.923 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003850058 2020-01-16T00:13:57.923 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.923 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.923 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:57.923 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578690+0000", 2020-01-16T00:13:57.923 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012896489000000001 2020-01-16T00:13:57.924 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.924 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.924 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.924 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591586+0000", 2020-01-16T00:13:57.924 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4757000000000001e-05 2020-01-16T00:13:57.924 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.924 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.924 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:57.925 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591681+0000", 2020-01-16T00:13:57.925 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000191535 2020-01-16T00:13:57.925 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.925 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.925 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:57.925 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591873+0000", 2020-01-16T00:13:57.925 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00069888899999999996 2020-01-16T00:13:57.925 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.926 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.926 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:57.926 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.592571+0000", 2020-01-16T00:13:57.926 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00029525900000000003 2020-01-16T00:13:57.926 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.926 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.926 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:57.927 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.592867+0000", 2020-01-16T00:13:57.927 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035243500000000001 2020-01-16T00:13:57.927 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.927 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.927 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:57.927 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593219+0000", 2020-01-16T00:13:57.928 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0954e-05 2020-01-16T00:13:57.928 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.928 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.928 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:57.928 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593270+0000", 2020-01-16T00:13:57.928 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5813000000000002e-05 2020-01-16T00:13:57.928 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.929 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.929 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:57.929 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593326+0000", 2020-01-16T00:13:57.929 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019893926999999999 2020-01-16T00:13:57.929 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:57.930 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:57.930 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:57.930 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.930 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.930 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2556 3.0 3:0b70bb20:::smithi12313934-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e37)", 2020-01-16T00:13:57.930 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.573472+0000", 2020-01-16T00:13:57.930 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 45.128837208, 2020-01-16T00:13:57.930 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020070068999999999, 2020-01-16T00:13:57.931 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:57.931 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:57.931 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:57.931 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:57.931 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:57.931 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2556 2020-01-16T00:13:57.931 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.932 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:57.932 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.932 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:57.932 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573472+0000", 2020-01-16T00:13:57.932 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:57.932 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.932 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.932 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:57.933 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573472+0000", 2020-01-16T00:13:57.933 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:57.933 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.933 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.933 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:57.933 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573471+0000", 2020-01-16T00:13:57.933 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4449999999999999e-06 2020-01-16T00:13:57.933 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.934 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.934 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:57.934 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573479+0000", 2020-01-16T00:13:57.934 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9789999999999998e-06 2020-01-16T00:13:57.934 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.934 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.934 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:57.934 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573485+0000", 2020-01-16T00:13:57.935 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3270000000000001e-06 2020-01-16T00:13:57.935 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.935 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.935 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:57.935 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573488+0000", 2020-01-16T00:13:57.935 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052586849999999999 2020-01-16T00:13:57.935 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.936 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.936 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.936 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578747+0000", 2020-01-16T00:13:57.936 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5667000000000007e-05 2020-01-16T00:13:57.936 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.936 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.936 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:57.936 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578812+0000", 2020-01-16T00:13:57.937 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013599539000000001 2020-01-16T00:13:57.937 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.937 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.937 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.937 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.592412+0000", 2020-01-16T00:13:57.937 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6018000000000001e-05 2020-01-16T00:13:57.938 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.938 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.938 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:57.938 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.592508+0000", 2020-01-16T00:13:57.938 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00084582199999999998 2020-01-16T00:13:57.938 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.938 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.938 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.939 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593354+0000", 2020-01-16T00:13:57.939 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4984999999999999e-05 2020-01-16T00:13:57.939 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.939 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.939 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:57.939 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593449+0000", 2020-01-16T00:13:57.939 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3104999999999997e-05 2020-01-16T00:13:57.940 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.940 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.940 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:57.940 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593542+0000", 2020-01-16T00:13:57.940 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020070068999999999 2020-01-16T00:13:57.940 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:57.940 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:57.940 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:57.941 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.941 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.941 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3211 3.0 3:0b70bb20:::smithi12313934-46:head [setxattr _header (56) in=7b,truncate 2571537] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:13:57.941 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.581146+0000", 2020-01-16T00:13:57.941 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 28.121162638000001, 2020-01-16T00:13:57.941 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024318514999999999, 2020-01-16T00:13:57.941 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:57.941 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:57.942 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:57.942 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:57.942 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:57.942 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3211 2020-01-16T00:13:57.942 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.942 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:57.942 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.943 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:57.943 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581146+0000", 2020-01-16T00:13:57.943 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:57.943 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.943 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.944 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:57.944 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581146+0000", 2020-01-16T00:13:57.944 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:57.944 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.944 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.944 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:57.944 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581145+0000", 2020-01-16T00:13:57.944 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2990000000000002e-06 2020-01-16T00:13:57.945 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.945 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.945 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:57.945 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581151+0000", 2020-01-16T00:13:57.945 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.041e-06 2020-01-16T00:13:57.945 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.945 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.946 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:57.946 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581159+0000", 2020-01-16T00:13:57.946 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5859999999999998e-06 2020-01-16T00:13:57.946 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.946 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.946 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:57.946 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581163+0000", 2020-01-16T00:13:57.946 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020769790000000001 2020-01-16T00:13:57.947 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.947 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.947 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.947 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.583240+0000", 2020-01-16T00:13:57.947 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012019579999999999 2020-01-16T00:13:57.947 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.948 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.948 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:57.948 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584442+0000", 2020-01-16T00:13:57.948 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018359812 2020-01-16T00:13:57.948 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.948 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.948 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.948 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.602802+0000", 2020-01-16T00:13:57.949 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000133072 2020-01-16T00:13:57.949 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.949 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.949 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:57.949 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.602935+0000", 2020-01-16T00:13:57.949 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019007200000000001 2020-01-16T00:13:57.949 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.949 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.950 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:57.950 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.603125+0000", 2020-01-16T00:13:57.950 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001261751 2020-01-16T00:13:57.950 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.950 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.950 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:57.950 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.604387+0000", 2020-01-16T00:13:57.951 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000304987 2020-01-16T00:13:57.951 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.951 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.951 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:57.951 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.604692+0000", 2020-01-16T00:13:57.952 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00063369299999999995 2020-01-16T00:13:57.952 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.952 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.952 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:57.952 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605325+0000", 2020-01-16T00:13:57.952 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1470000000000005e-05 2020-01-16T00:13:57.952 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.952 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:57.953 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605397+0000", 2020-01-16T00:13:57.953 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7784e-05 2020-01-16T00:13:57.953 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.953 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:57.953 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605465+0000", 2020-01-16T00:13:57.954 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024318514999999999 2020-01-16T00:13:57.954 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:57.954 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:57.954 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:57.954 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.954 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.954 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3212 3.0 3:0b70bb20:::smithi12313934-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e45)", 2020-01-16T00:13:57.954 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.581171+0000", 2020-01-16T00:13:57.955 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 28.121137939, 2020-01-16T00:13:57.955 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024540131, 2020-01-16T00:13:57.955 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:57.955 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:57.955 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:57.955 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:57.955 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:57.955 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3212 2020-01-16T00:13:57.956 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.956 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:57.956 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.956 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:57.957 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581171+0000", 2020-01-16T00:13:57.957 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:57.957 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.957 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.958 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:57.958 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581171+0000", 2020-01-16T00:13:57.958 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:57.958 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.958 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.958 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:57.958 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581170+0000", 2020-01-16T00:13:57.959 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5259999999999996e-06 2020-01-16T00:13:57.959 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.959 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.959 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:57.959 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581178+0000", 2020-01-16T00:13:57.959 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.469e-06 2020-01-16T00:13:57.959 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.960 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.960 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:57.960 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581185+0000", 2020-01-16T00:13:57.960 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4460000000000003e-06 2020-01-16T00:13:57.960 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.961 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.961 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:57.961 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581190+0000", 2020-01-16T00:13:57.961 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003286696 2020-01-16T00:13:57.961 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.962 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.962 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.962 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584476+0000", 2020-01-16T00:13:57.962 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9681e-05 2020-01-16T00:13:57.962 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.962 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.962 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:57.963 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584536+0000", 2020-01-16T00:13:57.963 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019075267 2020-01-16T00:13:57.963 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.963 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.963 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.963 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.603611+0000", 2020-01-16T00:13:57.963 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5981000000000001e-05 2020-01-16T00:13:57.964 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.964 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.964 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:57.964 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.603707+0000", 2020-01-16T00:13:57.964 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0017922649999999999 2020-01-16T00:13:57.964 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.964 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.964 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.965 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605500+0000", 2020-01-16T00:13:57.965 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5396999999999995e-05 2020-01-16T00:13:57.965 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.965 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.965 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:57.965 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605595+0000", 2020-01-16T00:13:57.966 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000116038 2020-01-16T00:13:57.966 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.966 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.966 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:57.966 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605711+0000", 2020-01-16T00:13:57.966 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024540131 2020-01-16T00:13:57.967 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:57.967 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:57.967 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:57.967 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.967 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.967 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3435 3.0 3:0b70bb20:::smithi12313934-46:head [setxattr _header (56) in=7b,truncate 3075747] snapc 0=[] ondisk+write+known_if_redirected e47)", 2020-01-16T00:13:57.968 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:34.388891+0000", 2020-01-16T00:13:57.968 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 23.313417922999999, 2020-01-16T00:13:57.968 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016440475999999999, 2020-01-16T00:13:57.968 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:57.968 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:57.968 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:57.968 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:57.968 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:57.969 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3435 2020-01-16T00:13:57.969 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.969 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:57.969 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.969 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:57.969 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388891+0000", 2020-01-16T00:13:57.969 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:57.970 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.970 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.970 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:57.970 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388891+0000", 2020-01-16T00:13:57.970 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:57.970 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.970 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.971 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:57.971 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388890+0000", 2020-01-16T00:13:57.971 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.647e-06 2020-01-16T00:13:57.971 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.972 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.972 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:57.972 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388894+0000", 2020-01-16T00:13:57.972 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2839999999999997e-06 2020-01-16T00:13:57.972 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.972 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.972 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:57.973 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388899+0000", 2020-01-16T00:13:57.973 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3149999999999999e-06 2020-01-16T00:13:57.973 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.973 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.973 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:57.973 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388901+0000", 2020-01-16T00:13:57.973 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001768543 2020-01-16T00:13:57.974 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.974 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.974 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.974 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.390670+0000", 2020-01-16T00:13:57.974 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0034533680000000001 2020-01-16T00:13:57.974 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.974 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.974 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:57.975 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.394123+0000", 2020-01-16T00:13:57.975 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010189962 2020-01-16T00:13:57.975 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.975 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.975 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.975 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404313+0000", 2020-01-16T00:13:57.975 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000108646 2020-01-16T00:13:57.976 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.976 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.976 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:57.976 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404422+0000", 2020-01-16T00:13:57.976 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119133 2020-01-16T00:13:57.976 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.976 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.977 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:13:57.977 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404541+0000", 2020-01-16T00:13:57.977 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000433764 2020-01-16T00:13:57.977 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.977 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.977 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:57.977 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404975+0000", 2020-01-16T00:13:57.978 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000266013 2020-01-16T00:13:57.978 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.978 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.978 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:57.979 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405241+0000", 2020-01-16T00:13:57.979 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5187000000000001e-05 2020-01-16T00:13:57.979 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.979 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.979 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:57.979 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405266+0000", 2020-01-16T00:13:57.980 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0889000000000002e-05 2020-01-16T00:13:57.980 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.980 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.980 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:57.980 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405297+0000", 2020-01-16T00:13:57.980 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4354999999999999e-05 2020-01-16T00:13:57.981 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.981 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.981 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:57.981 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405331+0000", 2020-01-16T00:13:57.981 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016440475999999999 2020-01-16T00:13:57.982 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:57.982 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:57.982 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:57.982 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.982 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.982 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3436 3.0 3:0b70bb20:::smithi12313934-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e47)", 2020-01-16T00:13:57.982 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:34.388920+0000", 2020-01-16T00:13:57.983 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 23.313389134000001, 2020-01-16T00:13:57.983 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016552641, 2020-01-16T00:13:57.983 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:57.983 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:57.983 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:57.983 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:57.983 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:57.984 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3436 2020-01-16T00:13:57.984 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.984 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:57.984 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.984 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:57.984 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388920+0000", 2020-01-16T00:13:57.984 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:57.985 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.985 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.985 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:57.985 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388920+0000", 2020-01-16T00:13:57.986 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:57.986 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.986 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.986 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:57.986 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388919+0000", 2020-01-16T00:13:57.986 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.181e-06 2020-01-16T00:13:57.986 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.987 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.987 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:57.987 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388921+0000", 2020-01-16T00:13:57.987 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9940000000000003e-06 2020-01-16T00:13:57.987 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.987 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.987 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:57.988 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388928+0000", 2020-01-16T00:13:57.988 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3549999999999999e-06 2020-01-16T00:13:57.988 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.988 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.988 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:57.988 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388931+0000", 2020-01-16T00:13:57.988 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.005242483 2020-01-16T00:13:57.988 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.989 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.989 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.989 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.394173+0000", 2020-01-16T00:13:57.989 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7435999999999995e-05 2020-01-16T00:13:57.989 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.989 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.990 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:57.990 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.394251+0000", 2020-01-16T00:13:57.990 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010593286 2020-01-16T00:13:57.990 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.990 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.990 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.990 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404844+0000", 2020-01-16T00:13:57.991 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0620999999999999e-05 2020-01-16T00:13:57.991 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.991 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.992 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:57.992 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404925+0000", 2020-01-16T00:13:57.992 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00042800899999999997 2020-01-16T00:13:57.992 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.992 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.992 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:57.992 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405353+0000", 2020-01-16T00:13:57.993 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2250000000000003e-05 2020-01-16T00:13:57.993 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.993 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.993 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:57.993 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405405+0000", 2020-01-16T00:13:57.993 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7411999999999998e-05 2020-01-16T00:13:57.993 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.994 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.994 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:57.994 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405472+0000", 2020-01-16T00:13:57.994 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016552641 2020-01-16T00:13:57.994 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:57.994 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:57.994 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:57.994 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.995 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.995 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3873 3.0 3:0b1f384f:::smithi12313934-33:head [read 0~3546310,checksum] snapc 0=[] ondisk+read+known_if_redirected e50)", 2020-01-16T00:13:57.995 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:44.489248+0000", 2020-01-16T00:13:57.995 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 13.213060842000001, 2020-01-16T00:13:57.995 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016088641000000001, 2020-01-16T00:13:57.995 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:57.995 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:57.996 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:57.996 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:57.996 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:57.996 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3873 2020-01-16T00:13:57.996 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.996 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:57.996 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.997 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:57.997 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489248+0000", 2020-01-16T00:13:57.997 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:57.997 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.997 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.997 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:57.998 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489248+0000", 2020-01-16T00:13:57.998 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:57.998 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.998 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.998 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:57.998 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489247+0000", 2020-01-16T00:13:57.998 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8080000000000004e-06 2020-01-16T00:13:57.999 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.999 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.999 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:57.999 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489252+0000", 2020-01-16T00:13:57.999 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1639999999999998e-06 2020-01-16T00:13:57.999 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:57.999 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:57.999 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:58.000 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489258+0000", 2020-01-16T00:13:58.000 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5169999999999996e-06 2020-01-16T00:13:58.000 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.000 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.000 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:58.000 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489265+0000", 2020-01-16T00:13:58.001 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.01457085 2020-01-16T00:13:58.001 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.001 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.001 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.001 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.503836+0000", 2020-01-16T00:13:58.001 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4594999999999998e-05 2020-01-16T00:13:58.002 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.002 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.002 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:58.002 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.503930+0000", 2020-01-16T00:13:58.002 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001406355 2020-01-16T00:13:58.002 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.002 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.003 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:58.003 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.505337+0000", 2020-01-16T00:13:58.003 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016088641000000001 2020-01-16T00:13:58.003 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.003 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:58.003 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.003 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.003 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:58.004 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:58.007 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-01-16T00:13:58.008 INFO:tasks.rados.rados.0.smithi123.stdout:1579: finishing write tid 2 to smithi12313934-26 2020-01-16T00:13:58.008 INFO:tasks.rados.rados.0.smithi123.stdout:1579: finishing write tid 3 to smithi12313934-26 2020-01-16T00:13:58.008 INFO:tasks.rados.rados.0.smithi123.stdout:1579: finishing write tid 5 to smithi12313934-26 2020-01-16T00:13:58.008 INFO:tasks.rados.rados.0.smithi123.stdout:1579: finishing write tid 6 to smithi12313934-26 2020-01-16T00:13:58.008 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 26 v 292 (ObjNum 546 snap 0 seq_num 546) dirty exists 2020-01-16T00:13:58.008 INFO:tasks.rados.rados.0.smithi123.stdout:1579: left oid 26 (ObjNum 546 snap 0 seq_num 546) 2020-01-16T00:13:58.008 INFO:tasks.rados.rados.0.smithi123.stdout:1581: finishing write tid 1 to smithi12313934-49 2020-01-16T00:13:58.009 INFO:tasks.rados.rados.0.smithi123.stdout:1581: finishing write tid 2 to smithi12313934-49 2020-01-16T00:13:58.009 INFO:tasks.rados.rados.0.smithi123.stdout:1581: finishing write tid 3 to smithi12313934-49 2020-01-16T00:13:58.009 INFO:tasks.rados.rados.0.smithi123.stdout:1581: finishing write tid 5 to smithi12313934-49 2020-01-16T00:13:58.009 INFO:tasks.rados.rados.0.smithi123.stdout:1581: finishing write tid 6 to smithi12313934-49 2020-01-16T00:13:58.009 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 49 v 164 (ObjNum 547 snap 0 seq_num 547) dirty exists 2020-01-16T00:13:58.009 INFO:tasks.rados.rados.0.smithi123.stdout:1581: left oid 49 (ObjNum 547 snap 0 seq_num 547) 2020-01-16T00:13:58.009 INFO:tasks.rados.rados.0.smithi123.stdout:1582: finishing write tid 1 to smithi12313934-13 2020-01-16T00:13:58.010 INFO:tasks.rados.rados.0.smithi123.stdout:1582: finishing write tid 2 to smithi12313934-13 2020-01-16T00:13:58.010 INFO:tasks.rados.rados.0.smithi123.stdout:1583: expect (ObjNum 518 snap 0 seq_num 518) 2020-01-16T00:13:58.010 INFO:tasks.rados.rados.0.smithi123.stdout:1584: finishing write tid 1 to smithi12313934-44 2020-01-16T00:13:58.010 INFO:tasks.rados.rados.0.smithi123.stdout:1584: finishing write tid 2 to smithi12313934-44 2020-01-16T00:13:58.010 INFO:tasks.rados.rados.0.smithi123.stdout:1582: finishing write tid 3 to smithi12313934-13 2020-01-16T00:13:58.010 INFO:tasks.rados.rados.0.smithi123.stdout:1582: finishing write tid 5 to smithi12313934-13 2020-01-16T00:13:58.010 INFO:tasks.rados.rados.0.smithi123.stdout:1582: finishing write tid 6 to smithi12313934-13 2020-01-16T00:13:58.011 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 108 (ObjNum 548 snap 0 seq_num 548) dirty exists 2020-01-16T00:13:58.011 INFO:tasks.rados.rados.0.smithi123.stdout:1582: left oid 13 (ObjNum 548 snap 0 seq_num 548) 2020-01-16T00:13:58.011 INFO:tasks.rados.rados.0.smithi123.stdout:1584: finishing write tid 3 to smithi12313934-44 2020-01-16T00:13:58.011 INFO:tasks.rados.rados.0.smithi123.stdout:1584: finishing write tid 5 to smithi12313934-44 2020-01-16T00:13:58.011 INFO:tasks.rados.rados.0.smithi123.stdout:1584: finishing write tid 6 to smithi12313934-44 2020-01-16T00:13:58.011 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 44 v 254 (ObjNum 549 snap 0 seq_num 549) dirty exists 2020-01-16T00:13:58.011 INFO:tasks.rados.rados.0.smithi123.stdout:1584: left oid 44 (ObjNum 549 snap 0 seq_num 549) 2020-01-16T00:13:58.012 INFO:tasks.rados.rados.0.smithi123.stdout:1585: expect (ObjNum 515 snap 0 seq_num 515) 2020-01-16T00:13:58.012 INFO:teuthology.orchestra.run.smithi123.stderr:nodeep-scrub is set 2020-01-16T00:13:58.012 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:58.002+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:58.106 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:13:58.100+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:13:58.153 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:58.153 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:58.153 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:13:58.154 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:58.162 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-01-16T00:13:58.171 INFO:tasks.rados.rados.0.smithi123.stdout:1586: finishing write tid 1 to smithi12313934-47 2020-01-16T00:13:58.171 INFO:tasks.rados.rados.0.smithi123.stdout:1586: finishing write tid 2 to smithi12313934-47 2020-01-16T00:13:58.171 INFO:tasks.rados.rados.0.smithi123.stdout:1586: finishing write tid 3 to smithi12313934-47 2020-01-16T00:13:58.171 INFO:tasks.rados.rados.0.smithi123.stdout:1586: finishing write tid 5 to smithi12313934-47 2020-01-16T00:13:58.171 INFO:tasks.rados.rados.0.smithi123.stdout:1586: finishing write tid 6 to smithi12313934-47 2020-01-16T00:13:58.172 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 47 v 240 (ObjNum 550 snap 0 seq_num 550) dirty exists 2020-01-16T00:13:58.172 INFO:tasks.rados.rados.0.smithi123.stdout:1586: left oid 47 (ObjNum 550 snap 0 seq_num 550) 2020-01-16T00:13:58.172 INFO:tasks.rados.rados.0.smithi123.stdout:1577: done (11 left) 2020-01-16T00:13:58.172 INFO:tasks.rados.rados.0.smithi123.stdout:1578: done (10 left) 2020-01-16T00:13:58.172 INFO:tasks.rados.rados.0.smithi123.stdout:1579: done (9 left) 2020-01-16T00:13:58.172 INFO:tasks.rados.rados.0.smithi123.stdout:1580: done (8 left) 2020-01-16T00:13:58.172 INFO:tasks.rados.rados.0.smithi123.stdout:1581: done (7 left) 2020-01-16T00:13:58.172 INFO:tasks.rados.rados.0.smithi123.stdout:1582: done (6 left) 2020-01-16T00:13:58.173 INFO:tasks.rados.rados.0.smithi123.stdout:1583: done (5 left) 2020-01-16T00:13:58.173 INFO:tasks.rados.rados.0.smithi123.stdout:1584: done (4 left) 2020-01-16T00:13:58.173 INFO:tasks.rados.rados.0.smithi123.stdout:1585: done (3 left) 2020-01-16T00:13:58.173 INFO:tasks.rados.rados.0.smithi123.stdout:1586: done (2 left) 2020-01-16T00:13:58.173 INFO:tasks.rados.rados.0.smithi123.stdout:1588: done (1 left) 2020-01-16T00:13:58.173 INFO:tasks.rados.rados.0.smithi123.stdout:1589: read oid 48 snap -1 2020-01-16T00:13:58.173 INFO:tasks.rados.rados.0.smithi123.stdout:1589: expect (ObjNum 528 snap 0 seq_num 528) 2020-01-16T00:13:58.174 INFO:tasks.rados.rados.0.smithi123.stdout:1590: write oid 50 current snap is 0 2020-01-16T00:13:58.174 INFO:tasks.rados.rados.0.smithi123.stdout:1590: seq_num 553 ranges {494271=790113,1693954=480183,2620209=512129} 2020-01-16T00:13:58.180 INFO:tasks.rados.rados.0.smithi123.stdout:1590: writing smithi12313934-50 from 494271 to 1284384 tid 1 2020-01-16T00:13:58.187 INFO:tasks.rados.rados.0.smithi123.stdout:1590: writing smithi12313934-50 from 1693954 to 2174137 tid 2 2020-01-16T00:13:58.194 INFO:tasks.rados.rados.0.smithi123.stdout:1590: writing smithi12313934-50 from 2620209 to 3132338 tid 3 2020-01-16T00:13:58.194 INFO:tasks.rados.rados.0.smithi123.stdout:1591: read oid 1 snap -1 2020-01-16T00:13:58.194 INFO:tasks.rados.rados.0.smithi123.stdout:1591: expect (ObjNum 545 snap 0 seq_num 545) 2020-01-16T00:13:58.194 INFO:tasks.rados.rados.0.smithi123.stdout:1592: read oid 41 snap -1 2020-01-16T00:13:58.194 INFO:tasks.rados.rados.0.smithi123.stdout:1592: expect (ObjNum 441 snap 0 seq_num 441) 2020-01-16T00:13:58.195 INFO:tasks.rados.rados.0.smithi123.stdout:1593: write oid 19 current snap is 0 2020-01-16T00:13:58.195 INFO:tasks.rados.rados.0.smithi123.stdout:1593: seq_num 554 ranges {511078=465362,1752436=740194,3163985=147093} 2020-01-16T00:13:58.200 INFO:tasks.rados.rados.0.smithi123.stdout:1593: writing smithi12313934-19 from 511078 to 976440 tid 1 2020-01-16T00:13:58.207 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:58.204+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:58.216 INFO:tasks.rados.rados.0.smithi123.stdout:1593: writing smithi12313934-19 from 1752436 to 2492630 tid 2 2020-01-16T00:13:58.218 INFO:tasks.rados.rados.0.smithi123.stdout:1593: writing smithi12313934-19 from 3163985 to 3311078 tid 3 2020-01-16T00:13:58.218 INFO:tasks.rados.rados.0.smithi123.stdout:1594: read oid 3 snap -1 2020-01-16T00:13:58.218 INFO:tasks.rados.rados.0.smithi123.stdout:1594: expect (ObjNum 534 snap 0 seq_num 534) 2020-01-16T00:13:58.218 INFO:tasks.rados.rados.0.smithi123.stdout:1595: read oid 35 snap -1 2020-01-16T00:13:58.218 INFO:tasks.rados.rados.0.smithi123.stdout:1595: expect (ObjNum 538 snap 0 seq_num 538) 2020-01-16T00:13:58.218 INFO:tasks.rados.rados.0.smithi123.stdout:1596: read oid 32 snap -1 2020-01-16T00:13:58.219 INFO:tasks.rados.rados.0.smithi123.stdout:1596: expect (ObjNum 513 snap 0 seq_num 513) 2020-01-16T00:13:58.219 INFO:tasks.rados.rados.0.smithi123.stdout:1597: write (excl) oid 11 current snap is 0 2020-01-16T00:13:58.219 INFO:tasks.rados.rados.0.smithi123.stdout:1597: seq_num 555 ranges {527885=656964,1927271=483852,3191408=136477} 2020-01-16T00:13:58.223 INFO:tasks.rados.rados.0.smithi123.stdout:1597: writing smithi12313934-11 from 527885 to 1184849 tid 1 2020-01-16T00:13:58.236 INFO:tasks.rados.rados.0.smithi123.stdout:1597: writing smithi12313934-11 from 1927271 to 2411123 tid 2 2020-01-16T00:13:58.237 INFO:tasks.rados.rados.0.smithi123.stdout:1597: writing smithi12313934-11 from 3191408 to 3327885 tid 3 2020-01-16T00:13:58.237 INFO:tasks.rados.rados.0.smithi123.stdout:1598: write oid 26 current snap is 0 2020-01-16T00:13:58.238 INFO:tasks.rados.rados.0.smithi123.stdout:1598: seq_num 556 ranges {544692=732213,1985753=743863,3335184=9508} 2020-01-16T00:13:58.246 INFO:tasks.rados.rados.0.smithi123.stdout:1598: writing smithi12313934-26 from 544692 to 1276905 tid 1 2020-01-16T00:13:58.256 INFO:tasks.rados.rados.0.smithi123.stdout:1598: writing smithi12313934-26 from 1985753 to 2729616 tid 2 2020-01-16T00:13:58.260 INFO:tasks.rados.rados.0.smithi123.stdout:1598: writing smithi12313934-26 from 3335184 to 3344692 tid 3 2020-01-16T00:13:58.260 INFO:tasks.rados.rados.0.smithi123.stdout:1599: write oid 4 current snap is 0 2020-01-16T00:13:58.260 INFO:tasks.rados.rados.0.smithi123.stdout:1599: seq_num 557 ranges {561499=407462,1527882=487521,2729901=557217} 2020-01-16T00:13:58.266 INFO:tasks.rados.rados.0.smithi123.stdout:1599: writing smithi12313934-4 from 561499 to 968961 tid 1 2020-01-16T00:13:58.271 INFO:tasks.rados.rados.0.smithi123.stdout:1599: writing smithi12313934-4 from 1527882 to 2015403 tid 2 2020-01-16T00:13:58.279 INFO:tasks.rados.rados.0.smithi123.stdout:1599: writing smithi12313934-4 from 2729901 to 3287118 tid 3 2020-01-16T00:13:58.279 INFO:tasks.rados.rados.0.smithi123.stdout:1600: write (excl) oid 25 current snap is 0 2020-01-16T00:13:58.279 INFO:tasks.rados.rados.0.smithi123.stdout:1600: seq_num 558 ranges {578306=482711,1586364=631179,2757324=484842} 2020-01-16T00:13:58.284 INFO:tasks.rados.rados.0.smithi123.stdout:1600: writing smithi12313934-25 from 578306 to 1061017 tid 1 2020-01-16T00:13:58.290 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:58.290 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:58.290 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:13:58.291 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:13:58.291 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:58.291 INFO:tasks.rados.rados.0.smithi123.stdout:1600: writing smithi12313934-25 from 1586364 to 2217543 tid 2 2020-01-16T00:13:58.297 INFO:tasks.rados.rados.0.smithi123.stdout:1600: writing smithi12313934-25 from 2757324 to 3242166 tid 3 2020-01-16T00:13:58.297 INFO:tasks.rados.rados.0.smithi123.stdout:1601: write oid 45 current snap is 0 2020-01-16T00:13:58.297 INFO:tasks.rados.rados.0.smithi123.stdout:1601: seq_num 559 ranges {595113=557960,1644846=491190,2784747=610366} 2020-01-16T00:13:58.299 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-01-16T00:13:58.303 INFO:tasks.rados.rados.0.smithi123.stdout:1601: writing smithi12313934-45 from 595113 to 1153073 tid 1 2020-01-16T00:13:58.307 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:58.304+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:58.308 INFO:tasks.rados.rados.0.smithi123.stdout:1601: writing smithi12313934-45 from 1644846 to 2136036 tid 2 2020-01-16T00:13:58.316 INFO:tasks.rados.rados.0.smithi123.stdout:1601: writing smithi12313934-45 from 2784747 to 3395113 tid 3 2020-01-16T00:13:58.316 INFO:tasks.rados.rados.0.smithi123.stdout:1602: read oid 6 snap -1 2020-01-16T00:13:58.316 INFO:tasks.rados.rados.0.smithi123.stdout:1602: expect (ObjNum 507 snap 0 seq_num 507) 2020-01-16T00:13:58.316 INFO:tasks.rados.rados.0.smithi123.stdout:1603: write (excl) oid 13 current snap is 0 2020-01-16T00:13:58.316 INFO:tasks.rados.rados.0.smithi123.stdout:1603: seq_num 560 ranges {611920=633209,1703328=634848,2812170=507386} 2020-01-16T00:13:58.323 INFO:tasks.rados.rados.0.smithi123.stdout:1603: writing smithi12313934-13 from 611920 to 1245129 tid 1 2020-01-16T00:13:58.330 INFO:tasks.rados.rados.0.smithi123.stdout:1603: writing smithi12313934-13 from 1703328 to 2338176 tid 2 2020-01-16T00:13:58.336 INFO:tasks.rados.rados.0.smithi123.stdout:1603: writing smithi12313934-13 from 2812170 to 3319556 tid 3 2020-01-16T00:13:58.336 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:58.336 INFO:tasks.rados.rados.0.smithi123.stdout:1587: finishing write tid 1 to smithi12313934-14 2020-01-16T00:13:58.336 INFO:tasks.rados.rados.0.smithi123.stdout:1587: finishing write tid 2 to smithi12313934-14 2020-01-16T00:13:58.336 INFO:tasks.rados.rados.0.smithi123.stdout:1587: finishing write tid 3 to smithi12313934-14 2020-01-16T00:13:58.336 INFO:tasks.rados.rados.0.smithi123.stdout:1587: finishing write tid 5 to smithi12313934-14 2020-01-16T00:13:58.337 INFO:tasks.rados.rados.0.smithi123.stdout:1587: finishing write tid 6 to smithi12313934-14 2020-01-16T00:13:58.337 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 14 v 258 (ObjNum 551 snap 0 seq_num 551) dirty exists 2020-01-16T00:13:58.337 INFO:tasks.rados.rados.0.smithi123.stdout:1587: left oid 14 (ObjNum 551 snap 0 seq_num 551) 2020-01-16T00:13:58.337 INFO:tasks.rados.rados.0.smithi123.stdout:1589: expect (ObjNum 528 snap 0 seq_num 528) 2020-01-16T00:13:58.407 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:58.407 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:13:58.407 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:13:58.407 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:13:58.407 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.408 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.5.13:42 2.0 e24/19 2:313458df:::smithi12313934-31 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 24'18, mlcod=24'16)", 2020-01-16T00:13:58.408 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.940683+0000", 2020-01-16T00:13:58.408 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 80.462432012999997, 2020-01-16T00:13:58.408 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0097026960000000002, 2020-01-16T00:13:58.408 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:58.408 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:58.409 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:58.409 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.409 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:58.409 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.940683+0000", 2020-01-16T00:13:58.409 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:58.409 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.409 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.410 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:58.410 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.940683+0000", 2020-01-16T00:13:58.410 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:58.410 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.410 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.410 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:58.410 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.940680+0000", 2020-01-16T00:13:58.411 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003126775 2020-01-16T00:13:58.411 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.411 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.411 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:58.411 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.943807+0000", 2020-01-16T00:13:58.411 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5426999999999999e-05 2020-01-16T00:13:58.411 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.411 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.412 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:58.412 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.943833+0000", 2020-01-16T00:13:58.412 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7409999999999998e-06 2020-01-16T00:13:58.412 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.412 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.412 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:58.413 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.943841+0000", 2020-01-16T00:13:58.413 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011645399999999999 2020-01-16T00:13:58.413 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.413 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.413 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.413 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.943958+0000", 2020-01-16T00:13:58.413 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7291000000000003e-05 2020-01-16T00:13:58.414 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.414 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.414 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:58.414 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.944025+0000", 2020-01-16T00:13:58.414 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063108449999999998 2020-01-16T00:13:58.414 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.414 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.415 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:58.415 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.950336+0000", 2020-01-16T00:13:58.415 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9874999999999999e-05 2020-01-16T00:13:58.415 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.415 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.415 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:58.415 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.950386+0000", 2020-01-16T00:13:58.415 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0097026960000000002 2020-01-16T00:13:58.416 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.416 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:58.416 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.416 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.416 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.416 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1649 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~2084068,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:13:58.416 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847683+0000", 2020-01-16T00:13:58.416 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 66.555432195999998, 2020-01-16T00:13:58.417 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010654691000000001, 2020-01-16T00:13:58.417 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:58.417 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:58.417 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:58.417 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:58.417 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:58.417 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1649 2020-01-16T00:13:58.418 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.418 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:58.418 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.418 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:58.418 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847683+0000", 2020-01-16T00:13:58.418 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:58.418 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.419 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.419 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:58.419 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847683+0000", 2020-01-16T00:13:58.419 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:58.420 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.420 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.420 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:58.420 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847682+0000", 2020-01-16T00:13:58.420 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8569999999999997e-06 2020-01-16T00:13:58.421 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.421 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.421 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:58.421 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847689+0000", 2020-01-16T00:13:58.422 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8090000000000004e-06 2020-01-16T00:13:58.422 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.422 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.423 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:58.423 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847696+0000", 2020-01-16T00:13:58.423 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2690000000000001e-06 2020-01-16T00:13:58.423 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.424 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.424 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:58.424 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847700+0000", 2020-01-16T00:13:58.424 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8189999999999993e-05 2020-01-16T00:13:58.424 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.424 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.425 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.425 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847798+0000", 2020-01-16T00:13:58.425 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7129999999999999e-06 2020-01-16T00:13:58.425 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.425 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.425 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:13:58.425 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847802+0000", 2020-01-16T00:13:58.426 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083913049999999999 2020-01-16T00:13:58.426 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.426 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.426 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.426 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.856193+0000", 2020-01-16T00:13:58.426 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034400300000000001 2020-01-16T00:13:58.426 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.426 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.427 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:58.427 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.856537+0000", 2020-01-16T00:13:58.427 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018006820000000001 2020-01-16T00:13:58.427 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.427 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.427 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:58.427 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858338+0000", 2020-01-16T00:13:58.427 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010654691000000001 2020-01-16T00:13:58.428 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.428 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:58.428 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.428 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.428 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.428 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1650 3.1 3:8de1d8a3:::smithi12313934-13:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:13:58.428 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847717+0000", 2020-01-16T00:13:58.429 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 66.555398242999999, 2020-01-16T00:13:58.429 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010913658, 2020-01-16T00:13:58.429 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:58.429 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:58.429 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:58.429 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:58.429 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:58.429 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1650 2020-01-16T00:13:58.430 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.430 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:58.430 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.430 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:58.430 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847717+0000", 2020-01-16T00:13:58.430 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:58.430 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.431 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.431 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:58.431 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847717+0000", 2020-01-16T00:13:58.431 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:58.431 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.431 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.431 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:58.431 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847716+0000", 2020-01-16T00:13:58.432 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.433e-06 2020-01-16T00:13:58.432 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.432 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.432 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:58.432 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847720+0000", 2020-01-16T00:13:58.432 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1800000000000004e-06 2020-01-16T00:13:58.433 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.433 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.433 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:58.433 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847725+0000", 2020-01-16T00:13:58.433 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6689999999999997e-06 2020-01-16T00:13:58.433 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.433 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.434 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:58.434 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847730+0000", 2020-01-16T00:13:58.434 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000102468 2020-01-16T00:13:58.434 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.434 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.434 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.434 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847832+0000", 2020-01-16T00:13:58.435 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3999999999999997e-06 2020-01-16T00:13:58.435 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.435 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.435 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:13:58.435 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847838+0000", 2020-01-16T00:13:58.435 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010519395000000001 2020-01-16T00:13:58.435 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.435 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.436 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.436 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858358+0000", 2020-01-16T00:13:58.436 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7600000000000002e-05 2020-01-16T00:13:58.436 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.436 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.436 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:58.436 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858445+0000", 2020-01-16T00:13:58.437 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018515900000000001 2020-01-16T00:13:58.437 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.437 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.437 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:58.437 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858631+0000", 2020-01-16T00:13:58.437 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010913658 2020-01-16T00:13:58.437 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.437 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:58.438 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.438 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.438 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.438 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1651 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~2084068,checksum] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:13:58.438 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847752+0000", 2020-01-16T00:13:58.438 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 66.555363627000006, 2020-01-16T00:13:58.438 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011734249, 2020-01-16T00:13:58.439 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:58.439 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:58.439 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:58.439 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:58.439 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:58.439 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1651 2020-01-16T00:13:58.439 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.440 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:58.440 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.440 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:58.440 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847752+0000", 2020-01-16T00:13:58.440 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:58.440 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.440 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.440 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:58.441 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847752+0000", 2020-01-16T00:13:58.441 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:58.441 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.441 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.441 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:58.441 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847751+0000", 2020-01-16T00:13:58.441 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1980000000000001e-06 2020-01-16T00:13:58.442 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.442 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.442 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:58.442 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847754+0000", 2020-01-16T00:13:58.442 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6679999999999999e-06 2020-01-16T00:13:58.442 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.442 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.443 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:58.443 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847758+0000", 2020-01-16T00:13:58.443 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2500000000000001e-06 2020-01-16T00:13:58.443 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.443 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.443 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:58.444 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847760+0000", 2020-01-16T00:13:58.444 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7614000000000004e-05 2020-01-16T00:13:58.444 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.444 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.444 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.444 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847858+0000", 2020-01-16T00:13:58.444 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7950000000000004e-06 2020-01-16T00:13:58.445 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.445 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.445 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:13:58.445 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847864+0000", 2020-01-16T00:13:58.445 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010790345 2020-01-16T00:13:58.445 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.445 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.446 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.446 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858654+0000", 2020-01-16T00:13:58.446 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000141466 2020-01-16T00:13:58.446 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.446 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.446 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:58.446 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858795+0000", 2020-01-16T00:13:58.446 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00069037899999999997 2020-01-16T00:13:58.447 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.447 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.447 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:58.447 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.859486+0000", 2020-01-16T00:13:58.447 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011734249 2020-01-16T00:13:58.447 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.447 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:58.448 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.448 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.448 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.448 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2107 3.1 3:8de1d8a3:::smithi12313934-13:head [write 1909329~458405] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:13:58.448 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.636786+0000", 2020-01-16T00:13:58.448 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 56.766328784999999, 2020-01-16T00:13:58.448 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093182809999999994, 2020-01-16T00:13:58.449 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:58.449 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:58.449 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:58.449 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:58.449 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:58.449 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2107 2020-01-16T00:13:58.449 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.449 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:58.450 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.450 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:58.450 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636786+0000", 2020-01-16T00:13:58.450 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:58.450 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.450 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.450 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:58.451 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636786+0000", 2020-01-16T00:13:58.451 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:58.451 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.451 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.451 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:58.451 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636784+0000", 2020-01-16T00:13:58.451 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014276299999999999 2020-01-16T00:13:58.452 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.452 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.452 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:58.452 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636927+0000", 2020-01-16T00:13:58.452 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5017999999999999e-05 2020-01-16T00:13:58.452 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.452 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.453 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:58.453 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636942+0000", 2020-01-16T00:13:58.453 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5589999999999996e-06 2020-01-16T00:13:58.453 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.453 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.454 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:58.454 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.636947+0000", 2020-01-16T00:13:58.454 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00040279699999999999 2020-01-16T00:13:58.454 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.454 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.454 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.454 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.637350+0000", 2020-01-16T00:13:58.455 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000233506 2020-01-16T00:13:58.455 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.456 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.457 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:58.457 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.637584+0000", 2020-01-16T00:13:58.457 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029534660000000001 2020-01-16T00:13:58.457 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.457 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.457 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.457 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640537+0000", 2020-01-16T00:13:58.458 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012577700000000001 2020-01-16T00:13:58.458 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.458 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.458 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:58.458 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640663+0000", 2020-01-16T00:13:58.458 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000198593 2020-01-16T00:13:58.458 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.458 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.459 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:13:58.459 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.640862+0000", 2020-01-16T00:13:58.459 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.005088197 2020-01-16T00:13:58.459 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.459 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.459 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:58.460 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645950+0000", 2020-01-16T00:13:58.460 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2568999999999999e-05 2020-01-16T00:13:58.460 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.460 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.460 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:58.460 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645962+0000", 2020-01-16T00:13:58.460 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3034999999999995e-05 2020-01-16T00:13:58.460 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.461 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.461 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:58.461 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646025+0000", 2020-01-16T00:13:58.461 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7762999999999997e-05 2020-01-16T00:13:58.461 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.461 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.461 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:58.462 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646063+0000", 2020-01-16T00:13:58.462 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1541000000000001e-05 2020-01-16T00:13:58.462 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.462 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.462 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:58.462 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646105+0000", 2020-01-16T00:13:58.462 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0093182809999999994 2020-01-16T00:13:58.462 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.463 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:58.463 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.463 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.463 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.464 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2108 3.1 3:8de1d8a3:::smithi12313934-13:head [write 2816700~410210] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:13:58.464 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.641861+0000", 2020-01-16T00:13:58.464 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 56.761254436999998, 2020-01-16T00:13:58.464 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018545604, 2020-01-16T00:13:58.464 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:58.464 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:58.464 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:58.465 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:58.465 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:58.465 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2108 2020-01-16T00:13:58.465 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.465 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:58.465 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.465 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:58.466 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.641861+0000", 2020-01-16T00:13:58.466 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:58.466 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.466 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.466 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:58.466 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.641861+0000", 2020-01-16T00:13:58.466 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:58.466 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.467 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.467 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:58.467 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.641859+0000", 2020-01-16T00:13:58.467 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018466399999999999 2020-01-16T00:13:58.467 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.467 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.467 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:58.468 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642043+0000", 2020-01-16T00:13:58.468 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2999999999999999e-05 2020-01-16T00:13:58.468 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.468 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.468 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:58.468 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642056+0000", 2020-01-16T00:13:58.468 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0730000000000005e-06 2020-01-16T00:13:58.468 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.469 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.469 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:58.469 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642064+0000", 2020-01-16T00:13:58.469 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9527999999999997e-05 2020-01-16T00:13:58.469 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.469 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.469 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.469 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642114+0000", 2020-01-16T00:13:58.470 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036616050000000001 2020-01-16T00:13:58.470 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.470 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.470 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:58.470 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645776+0000", 2020-01-16T00:13:58.470 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012222663999999999 2020-01-16T00:13:58.470 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.471 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.471 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.471 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.657998+0000", 2020-01-16T00:13:58.471 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014280299999999999 2020-01-16T00:13:58.471 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.471 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.471 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:58.471 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.658141+0000", 2020-01-16T00:13:58.472 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0001102 2020-01-16T00:13:58.472 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.472 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.472 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:13:58.472 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.658251+0000", 2020-01-16T00:13:58.472 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020421219999999999 2020-01-16T00:13:58.473 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.473 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.473 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:58.473 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660293+0000", 2020-01-16T00:13:58.473 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7399999999999999e-06 2020-01-16T00:13:58.473 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.473 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.473 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:58.474 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660303+0000", 2020-01-16T00:13:58.474 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9873999999999998e-05 2020-01-16T00:13:58.474 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.474 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.474 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:58.475 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660343+0000", 2020-01-16T00:13:58.475 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4448e-05 2020-01-16T00:13:58.475 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.475 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.475 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:58.475 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660377+0000", 2020-01-16T00:13:58.475 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8915999999999999e-05 2020-01-16T00:13:58.475 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.476 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.476 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:58.476 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660406+0000", 2020-01-16T00:13:58.476 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018545604 2020-01-16T00:13:58.476 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.476 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:58.476 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.476 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.477 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.477 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2109 3.1 3:8de1d8a3:::smithi12313934-13:head [setxattr _header (56) in=7b,truncate 3865610] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:13:58.477 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.642075+0000", 2020-01-16T00:13:58.477 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 56.761039863999997, 2020-01-16T00:13:58.477 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018477655999999999, 2020-01-16T00:13:58.477 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:58.477 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:58.478 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:58.478 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:58.478 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:58.478 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2109 2020-01-16T00:13:58.478 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.478 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:58.478 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.478 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:58.479 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642075+0000", 2020-01-16T00:13:58.479 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:58.479 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.479 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.479 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:58.479 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642075+0000", 2020-01-16T00:13:58.479 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:58.480 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.480 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.480 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:58.480 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642074+0000", 2020-01-16T00:13:58.480 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4500000000000003e-06 2020-01-16T00:13:58.480 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.480 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.480 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:58.481 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642080+0000", 2020-01-16T00:13:58.481 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7229999999999998e-06 2020-01-16T00:13:58.481 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.481 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.481 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:58.481 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642087+0000", 2020-01-16T00:13:58.481 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2390000000000004e-06 2020-01-16T00:13:58.482 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.482 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.482 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:58.482 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642091+0000", 2020-01-16T00:13:58.482 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0037396550000000001 2020-01-16T00:13:58.482 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.482 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.483 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.483 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645830+0000", 2020-01-16T00:13:58.483 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2200000000000002e-05 2020-01-16T00:13:58.483 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.484 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.484 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:58.484 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645883+0000", 2020-01-16T00:13:58.484 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013432934000000001 2020-01-16T00:13:58.484 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.485 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.485 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.485 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659316+0000", 2020-01-16T00:13:58.485 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000140006 2020-01-16T00:13:58.485 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.485 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.485 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:58.485 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659456+0000", 2020-01-16T00:13:58.486 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000184873 2020-01-16T00:13:58.486 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.486 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.486 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:13:58.486 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659640+0000", 2020-01-16T00:13:58.487 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00066954999999999998 2020-01-16T00:13:58.487 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.487 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.487 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:58.487 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660310+0000", 2020-01-16T00:13:58.487 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014847699999999999 2020-01-16T00:13:58.487 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.487 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.488 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:58.488 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660459+0000", 2020-01-16T00:13:58.488 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6869e-05 2020-01-16T00:13:58.488 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.488 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.488 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:58.488 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660485+0000", 2020-01-16T00:13:58.489 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8478000000000001e-05 2020-01-16T00:13:58.489 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.489 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.489 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:58.489 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660514+0000", 2020-01-16T00:13:58.489 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9153999999999998e-05 2020-01-16T00:13:58.489 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.489 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.490 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:58.490 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660553+0000", 2020-01-16T00:13:58.490 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018477655999999999 2020-01-16T00:13:58.490 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.490 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:58.490 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.490 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.490 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.491 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2110 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-01-16T00:13:58.491 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.642112+0000", 2020-01-16T00:13:58.492 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 56.761003226, 2020-01-16T00:13:58.492 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018620198000000001, 2020-01-16T00:13:58.492 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:58.492 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:58.493 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:58.493 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:58.493 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:58.493 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2110 2020-01-16T00:13:58.493 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.493 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:58.494 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.494 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:58.494 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642112+0000", 2020-01-16T00:13:58.494 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:58.494 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.494 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.494 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:58.494 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642112+0000", 2020-01-16T00:13:58.495 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:58.495 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.495 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.495 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:58.495 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642111+0000", 2020-01-16T00:13:58.495 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4450000000000001e-06 2020-01-16T00:13:58.495 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.496 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.496 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:58.496 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642117+0000", 2020-01-16T00:13:58.496 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1567e-05 2020-01-16T00:13:58.496 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.496 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.496 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:58.496 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642129+0000", 2020-01-16T00:13:58.497 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.754e-06 2020-01-16T00:13:58.497 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.497 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.497 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:58.498 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642136+0000", 2020-01-16T00:13:58.498 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040239719999999998 2020-01-16T00:13:58.498 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.498 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.498 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.498 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646160+0000", 2020-01-16T00:13:58.498 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5600999999999998e-05 2020-01-16T00:13:58.499 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.499 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.499 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:58.499 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646215+0000", 2020-01-16T00:13:58.499 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013733006000000001 2020-01-16T00:13:58.500 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.500 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.500 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.500 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659948+0000", 2020-01-16T00:13:58.500 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000106277 2020-01-16T00:13:58.500 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.500 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.501 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:58.501 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660054+0000", 2020-01-16T00:13:58.501 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000524327 2020-01-16T00:13:58.501 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.501 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.501 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.501 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660579+0000", 2020-01-16T00:13:58.502 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9841999999999999e-05 2020-01-16T00:13:58.502 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.502 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.503 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:58.503 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660659+0000", 2020-01-16T00:13:58.503 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3572999999999995e-05 2020-01-16T00:13:58.503 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.504 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.504 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:58.504 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660732+0000", 2020-01-16T00:13:58.504 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018620198000000001 2020-01-16T00:13:58.504 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.504 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:58.505 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.506 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.507 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.507 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:70 2.1 e35/19 2:91f85463:::smithi12313934-24 low_tier:head v 35'66, mlcod=34'64)", 2020-01-16T00:13:58.507 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.986684+0000", 2020-01-16T00:13:58.507 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 52.416431371999998, 2020-01-16T00:13:58.507 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012287378999999999, 2020-01-16T00:13:58.508 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:58.508 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:58.508 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:58.508 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.509 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:58.509 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986684+0000", 2020-01-16T00:13:58.509 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:58.509 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.509 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.510 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:58.510 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986684+0000", 2020-01-16T00:13:58.510 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:58.510 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.511 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.511 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:58.511 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986683+0000", 2020-01-16T00:13:58.511 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012884179999999999 2020-01-16T00:13:58.512 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.512 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.512 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:58.513 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.987971+0000", 2020-01-16T00:13:58.513 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011558200000000001 2020-01-16T00:13:58.513 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.513 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.514 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:58.514 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988087+0000", 2020-01-16T00:13:58.514 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0472000000000001e-05 2020-01-16T00:13:58.514 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.514 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.514 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:58.515 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988137+0000", 2020-01-16T00:13:58.515 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000107484 2020-01-16T00:13:58.515 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.515 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.515 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.515 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988245+0000", 2020-01-16T00:13:58.516 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4409999999999998e-05 2020-01-16T00:13:58.516 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.516 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.516 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:58.516 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988279+0000", 2020-01-16T00:13:58.516 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010641609999999999 2020-01-16T00:13:58.516 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.517 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.517 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:58.517 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.998921+0000", 2020-01-16T00:13:58.518 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0278000000000002e-05 2020-01-16T00:13:58.518 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.518 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.518 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:58.518 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.998971+0000", 2020-01-16T00:13:58.518 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012287378999999999 2020-01-16T00:13:58.519 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.519 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:58.519 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.519 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.519 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.519 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:109 2.0 e42/19 2:11fa79bc:::smithi12313934-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 42'67, mlcod=41'65)", 2020-01-16T00:13:58.519 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:21.841140+0000", 2020-01-16T00:13:58.519 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 36.561975304000001, 2020-01-16T00:13:58.520 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010439159999999999, 2020-01-16T00:13:58.520 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:58.520 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:58.520 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:58.520 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.520 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:58.520 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841140+0000", 2020-01-16T00:13:58.520 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:58.521 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.521 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.521 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:58.521 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841140+0000", 2020-01-16T00:13:58.521 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:58.521 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.521 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.522 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:58.522 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841137+0000", 2020-01-16T00:13:58.522 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00061049800000000001 2020-01-16T00:13:58.522 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.522 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.522 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:58.522 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841748+0000", 2020-01-16T00:13:58.523 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8703e-05 2020-01-16T00:13:58.523 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.523 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.523 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:58.523 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841767+0000", 2020-01-16T00:13:58.523 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5410000000000003e-06 2020-01-16T00:13:58.524 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.524 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.524 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:58.525 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841774+0000", 2020-01-16T00:13:58.525 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000100144 2020-01-16T00:13:58.525 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.525 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.525 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.525 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841874+0000", 2020-01-16T00:13:58.525 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9402999999999999e-05 2020-01-16T00:13:58.526 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.526 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.526 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:58.526 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841944+0000", 2020-01-16T00:13:58.526 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095901649999999995 2020-01-16T00:13:58.526 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.526 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.527 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:58.527 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.851534+0000", 2020-01-16T00:13:58.527 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5296000000000001e-05 2020-01-16T00:13:58.527 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.527 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.527 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:58.527 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.851579+0000", 2020-01-16T00:13:58.527 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010439159999999999 2020-01-16T00:13:58.528 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.528 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:58.528 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.528 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.528 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.528 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3228 3.1 3:8de1d8a3:::smithi12313934-13:head [write 588344~631611,stat] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:13:58.528 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.588539+0000", 2020-01-16T00:13:58.529 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 28.814576134999999, 2020-01-16T00:13:58.529 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012776884000000001, 2020-01-16T00:13:58.529 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:58.529 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:58.529 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:58.529 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:58.529 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:58.529 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3228 2020-01-16T00:13:58.530 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.530 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:58.530 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.530 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:58.530 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588539+0000", 2020-01-16T00:13:58.530 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:58.530 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.530 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.531 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:58.531 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588539+0000", 2020-01-16T00:13:58.531 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:58.531 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.531 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.531 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:58.531 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588536+0000", 2020-01-16T00:13:58.532 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021452 2020-01-16T00:13:58.532 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.532 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.532 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:58.532 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588750+0000", 2020-01-16T00:13:58.532 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.1432000000000001e-05 2020-01-16T00:13:58.532 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.533 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.533 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:58.533 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588772+0000", 2020-01-16T00:13:58.533 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1799999999999999e-06 2020-01-16T00:13:58.533 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.533 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.533 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:58.534 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588779+0000", 2020-01-16T00:13:58.534 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00095777700000000004 2020-01-16T00:13:58.534 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.534 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.534 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.534 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589737+0000", 2020-01-16T00:13:58.534 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013306599999999999 2020-01-16T00:13:58.534 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.535 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.535 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:58.535 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589870+0000", 2020-01-16T00:13:58.535 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020556 2020-01-16T00:13:58.535 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.536 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.536 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:13:58.536 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.590075+0000", 2020-01-16T00:13:58.536 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040226080000000004 2020-01-16T00:13:58.536 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.536 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.536 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:58.536 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.594098+0000", 2020-01-16T00:13:58.537 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.006043288 2020-01-16T00:13:58.537 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.537 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.537 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:58.537 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.600141+0000", 2020-01-16T00:13:58.537 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00089242900000000003 2020-01-16T00:13:58.537 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.537 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.538 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:58.538 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601034+0000", 2020-01-16T00:13:58.538 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6097e-05 2020-01-16T00:13:58.538 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.538 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.538 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:58.538 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601110+0000", 2020-01-16T00:13:58.539 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000206088 2020-01-16T00:13:58.539 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.539 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.539 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:58.539 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601316+0000", 2020-01-16T00:13:58.539 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012776884000000001 2020-01-16T00:13:58.539 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.540 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:58.540 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.540 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.540 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.540 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.1.13:77 2.1 e45/19 2:bafac7f5:::smithi12313934-46 low_tier:head v 45'100, mlcod=45'98)", 2020-01-16T00:13:58.541 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.589633+0000", 2020-01-16T00:13:58.541 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 28.813481912, 2020-01-16T00:13:58.541 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010560460000000001, 2020-01-16T00:13:58.541 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:58.541 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:58.541 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:58.541 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.541 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:58.542 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589633+0000", 2020-01-16T00:13:58.542 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:58.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.542 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.542 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:58.542 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589633+0000", 2020-01-16T00:13:58.542 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:58.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.543 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.543 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:58.543 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589630+0000", 2020-01-16T00:13:58.543 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00050499799999999999 2020-01-16T00:13:58.543 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.543 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.543 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:58.543 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.590135+0000", 2020-01-16T00:13:58.544 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0473e-05 2020-01-16T00:13:58.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.544 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.544 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:58.544 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.590146+0000", 2020-01-16T00:13:58.544 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9710000000000001e-06 2020-01-16T00:13:58.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.545 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.545 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:58.545 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.590152+0000", 2020-01-16T00:13:58.545 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00040412999999999999 2020-01-16T00:13:58.545 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.545 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.546 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.546 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.590556+0000", 2020-01-16T00:13:58.546 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6491000000000001e-05 2020-01-16T00:13:58.546 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.546 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.546 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:58.547 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.590583+0000", 2020-01-16T00:13:58.547 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095621360000000006 2020-01-16T00:13:58.547 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.547 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.547 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:58.547 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.600145+0000", 2020-01-16T00:13:58.548 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9072e-05 2020-01-16T00:13:58.548 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.548 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.548 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:58.548 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.600194+0000", 2020-01-16T00:13:58.548 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010560460000000001 2020-01-16T00:13:58.548 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.549 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:58.549 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.549 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.550 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.550 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3229 3.1 3:8de1d8a3:::smithi12313934-13:head [write 1689059~741123] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:13:58.550 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.596087+0000", 2020-01-16T00:13:58.550 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 28.807028074000002, 2020-01-16T00:13:58.550 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010699248, 2020-01-16T00:13:58.551 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:58.551 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:58.551 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:58.551 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:58.552 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:58.552 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3229 2020-01-16T00:13:58.552 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.552 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:58.552 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.552 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:58.553 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596087+0000", 2020-01-16T00:13:58.553 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:58.553 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.553 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.553 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:58.554 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596087+0000", 2020-01-16T00:13:58.554 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:58.554 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.554 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.555 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:58.555 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596084+0000", 2020-01-16T00:13:58.555 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00026243600000000001 2020-01-16T00:13:58.555 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.555 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.556 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:58.556 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596347+0000", 2020-01-16T00:13:58.556 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3594000000000001e-05 2020-01-16T00:13:58.556 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.557 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.557 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:58.557 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596370+0000", 2020-01-16T00:13:58.557 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5259999999999999e-06 2020-01-16T00:13:58.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.558 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.558 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:58.558 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596380+0000", 2020-01-16T00:13:58.558 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2298000000000005e-05 2020-01-16T00:13:58.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.559 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.559 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.559 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596452+0000", 2020-01-16T00:13:58.560 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000163702 2020-01-16T00:13:58.560 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.560 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.560 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:58.560 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596616+0000", 2020-01-16T00:13:58.561 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0047611729999999996 2020-01-16T00:13:58.561 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.561 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.561 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.562 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601377+0000", 2020-01-16T00:13:58.562 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014076900000000001 2020-01-16T00:13:58.562 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.562 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.563 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:58.563 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601518+0000", 2020-01-16T00:13:58.563 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021367599999999999 2020-01-16T00:13:58.563 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.564 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.564 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:13:58.564 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601731+0000", 2020-01-16T00:13:58.564 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0028479149999999999 2020-01-16T00:13:58.565 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.565 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.565 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:58.565 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.604579+0000", 2020-01-16T00:13:58.565 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001408133 2020-01-16T00:13:58.565 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.566 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.566 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:58.566 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605987+0000", 2020-01-16T00:13:58.566 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00072335599999999998 2020-01-16T00:13:58.566 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.566 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.567 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:58.567 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.606711+0000", 2020-01-16T00:13:58.567 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0289e-05 2020-01-16T00:13:58.567 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.567 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.567 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:58.567 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.606751+0000", 2020-01-16T00:13:58.568 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5298e-05 2020-01-16T00:13:58.568 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.568 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.568 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:58.568 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.606786+0000", 2020-01-16T00:13:58.568 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010699248 2020-01-16T00:13:58.568 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.569 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:58.569 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.569 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.569 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.569 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3230 3.1 3:8de1d8a3:::smithi12313934-13:head [setxattr _header (56) in=7b,truncate 2588344] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:13:58.569 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.596394+0000", 2020-01-16T00:13:58.569 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 28.806721114999998, 2020-01-16T00:13:58.570 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021205134, 2020-01-16T00:13:58.570 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:58.570 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:58.570 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:58.570 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:58.570 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:58.570 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3230 2020-01-16T00:13:58.570 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.571 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:58.571 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.571 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:58.571 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596394+0000", 2020-01-16T00:13:58.571 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:58.571 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.571 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.571 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:58.572 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596394+0000", 2020-01-16T00:13:58.572 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:58.572 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.572 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.572 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:58.572 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596393+0000", 2020-01-16T00:13:58.572 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7159999999999997e-06 2020-01-16T00:13:58.572 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.573 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.573 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:58.573 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596400+0000", 2020-01-16T00:13:58.573 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0074e-05 2020-01-16T00:13:58.573 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.573 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.573 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:58.573 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596410+0000", 2020-01-16T00:13:58.574 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1679999999999997e-06 2020-01-16T00:13:58.574 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.574 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.574 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:58.574 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596414+0000", 2020-01-16T00:13:58.574 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023609 2020-01-16T00:13:58.574 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.574 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.575 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.575 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596650+0000", 2020-01-16T00:13:58.575 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012316399999999999 2020-01-16T00:13:58.575 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.575 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.575 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:58.575 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596774+0000", 2020-01-16T00:13:58.575 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0067849670000000003 2020-01-16T00:13:58.576 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.576 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.576 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.576 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.603558+0000", 2020-01-16T00:13:58.576 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00092385999999999998 2020-01-16T00:13:58.576 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.576 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.577 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:58.577 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.604482+0000", 2020-01-16T00:13:58.577 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011729201 2020-01-16T00:13:58.577 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.577 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.577 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.577 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.616212+0000", 2020-01-16T00:13:58.578 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019515099999999999 2020-01-16T00:13:58.578 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.578 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.578 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:58.578 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.616407+0000", 2020-01-16T00:13:58.578 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000182843 2020-01-16T00:13:58.578 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.579 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.579 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:13:58.579 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.616590+0000", 2020-01-16T00:13:58.579 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000660572 2020-01-16T00:13:58.579 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.579 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.580 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:58.580 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617250+0000", 2020-01-16T00:13:58.580 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5333999999999997e-05 2020-01-16T00:13:58.580 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.580 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.580 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:58.580 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617285+0000", 2020-01-16T00:13:58.581 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023020899999999999 2020-01-16T00:13:58.581 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.581 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.581 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:58.582 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617516+0000", 2020-01-16T00:13:58.582 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4214000000000001e-05 2020-01-16T00:13:58.582 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.582 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.582 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:58.582 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617560+0000", 2020-01-16T00:13:58.582 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9366999999999998e-05 2020-01-16T00:13:58.582 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.583 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.583 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:58.583 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617599+0000", 2020-01-16T00:13:58.583 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021205134 2020-01-16T00:13:58.583 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.583 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:58.583 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.583 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.584 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.584 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3231 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e45)", 2020-01-16T00:13:58.584 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.596438+0000", 2020-01-16T00:13:58.584 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 28.806677104999999, 2020-01-16T00:13:58.584 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021455667000000001, 2020-01-16T00:13:58.584 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:58.584 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:58.584 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:58.585 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:58.585 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:58.585 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3231 2020-01-16T00:13:58.585 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.585 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:58.585 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.585 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:58.585 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596438+0000", 2020-01-16T00:13:58.586 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:58.586 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.586 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.586 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:58.586 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596438+0000", 2020-01-16T00:13:58.586 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:58.586 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.587 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.587 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:58.587 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596437+0000", 2020-01-16T00:13:58.587 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8500000000000004e-06 2020-01-16T00:13:58.587 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.587 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.587 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:58.587 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596441+0000", 2020-01-16T00:13:58.588 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9439999999999994e-06 2020-01-16T00:13:58.588 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.588 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.588 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:58.588 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596450+0000", 2020-01-16T00:13:58.588 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.267e-06 2020-01-16T00:13:58.588 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.588 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.589 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:58.589 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596455+0000", 2020-01-16T00:13:58.589 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034332399999999999 2020-01-16T00:13:58.589 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.589 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.589 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.589 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596798+0000", 2020-01-16T00:13:58.589 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011687400000000001 2020-01-16T00:13:58.590 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.590 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.590 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:58.590 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596915+0000", 2020-01-16T00:13:58.590 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0076057049999999999 2020-01-16T00:13:58.590 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.590 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.590 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.591 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.604520+0000", 2020-01-16T00:13:58.591 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8143999999999998e-05 2020-01-16T00:13:58.591 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.591 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.591 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:58.591 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.604569+0000", 2020-01-16T00:13:58.591 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012379960000000001 2020-01-16T00:13:58.591 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.592 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.592 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.592 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.616949+0000", 2020-01-16T00:13:58.592 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000120617 2020-01-16T00:13:58.592 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.592 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.592 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:58.592 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617069+0000", 2020-01-16T00:13:58.593 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00056715400000000005 2020-01-16T00:13:58.593 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.593 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.593 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.593 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617636+0000", 2020-01-16T00:13:58.593 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012844700000000001 2020-01-16T00:13:58.593 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.594 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.594 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:58.594 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617765+0000", 2020-01-16T00:13:58.594 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000129037 2020-01-16T00:13:58.594 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.594 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.594 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:58.594 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617894+0000", 2020-01-16T00:13:58.595 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021455667000000001 2020-01-16T00:13:58.595 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.595 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:58.595 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.595 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.595 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.595 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3822 3.1 3:8de1d8a3:::smithi12313934-13:head [setxattr _header (56) in=7b,truncate 2016939] snapc 0=[] ondisk+write+known_if_redirected e50)", 2020-01-16T00:13:58.596 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:44.059584+0000", 2020-01-16T00:13:58.596 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 14.343530789000001, 2020-01-16T00:13:58.596 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021977219999999999, 2020-01-16T00:13:58.596 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:58.596 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:58.596 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:58.596 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:58.596 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:58.597 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3822 2020-01-16T00:13:58.597 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.597 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:58.597 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.597 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:58.597 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059584+0000", 2020-01-16T00:13:58.597 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:58.597 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.598 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.598 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:58.598 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059584+0000", 2020-01-16T00:13:58.598 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:58.598 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.598 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.598 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:58.599 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059584+0000", 2020-01-16T00:13:58.599 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1329999999999998e-06 2020-01-16T00:13:58.599 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.599 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.599 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:58.599 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059589+0000", 2020-01-16T00:13:58.599 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8136000000000002e-05 2020-01-16T00:13:58.599 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.600 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.600 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:58.600 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059627+0000", 2020-01-16T00:13:58.600 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1409999999999998e-06 2020-01-16T00:13:58.600 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.600 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.601 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:58.601 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059631+0000", 2020-01-16T00:13:58.601 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0011314020000000001 2020-01-16T00:13:58.601 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.601 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.601 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.601 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.060763+0000", 2020-01-16T00:13:58.602 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00085078200000000001 2020-01-16T00:13:58.602 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.602 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.602 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:58.604 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.061613+0000", 2020-01-16T00:13:58.604 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017766311999999999 2020-01-16T00:13:58.605 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.605 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.605 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.605 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.079380+0000", 2020-01-16T00:13:58.605 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000133144 2020-01-16T00:13:58.605 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.606 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.606 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:58.606 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.079513+0000", 2020-01-16T00:13:58.606 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021271500000000001 2020-01-16T00:13:58.606 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.606 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.606 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:13:58.606 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.079725+0000", 2020-01-16T00:13:58.607 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001302582 2020-01-16T00:13:58.607 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.607 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.607 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:58.607 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081028+0000", 2020-01-16T00:13:58.607 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017197 2020-01-16T00:13:58.607 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.608 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.608 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:58.608 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081200+0000", 2020-01-16T00:13:58.608 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000202757 2020-01-16T00:13:58.608 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.608 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.608 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:58.608 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081403+0000", 2020-01-16T00:13:58.609 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1068999999999993e-05 2020-01-16T00:13:58.609 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.609 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.609 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:58.609 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081474+0000", 2020-01-16T00:13:58.609 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7800000000000006e-05 2020-01-16T00:13:58.609 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.609 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.610 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:58.610 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081562+0000", 2020-01-16T00:13:58.610 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021977219999999999 2020-01-16T00:13:58.610 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.610 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:58.610 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.610 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.610 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.611 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3823 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e50)", 2020-01-16T00:13:58.611 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:44.059639+0000", 2020-01-16T00:13:58.611 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 14.343476634, 2020-01-16T00:13:58.611 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022245396000000001, 2020-01-16T00:13:58.611 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:58.611 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:58.611 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:58.611 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:58.612 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:58.612 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3823 2020-01-16T00:13:58.612 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.612 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:58.612 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.612 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:58.612 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059639+0000", 2020-01-16T00:13:58.612 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:58.613 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.613 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.613 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:58.613 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059639+0000", 2020-01-16T00:13:58.613 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:58.613 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.613 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.614 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:58.614 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059638+0000", 2020-01-16T00:13:58.614 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.405e-06 2020-01-16T00:13:58.614 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.614 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.614 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:58.614 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059641+0000", 2020-01-16T00:13:58.614 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9770000000000003e-06 2020-01-16T00:13:58.615 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.615 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.615 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:58.615 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059648+0000", 2020-01-16T00:13:58.615 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1140000000000002e-06 2020-01-16T00:13:58.615 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.615 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.616 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:58.616 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059652+0000", 2020-01-16T00:13:58.616 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020063659999999999 2020-01-16T00:13:58.616 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.616 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.616 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.616 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.061658+0000", 2020-01-16T00:13:58.616 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3183999999999999e-05 2020-01-16T00:13:58.617 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.617 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.617 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:58.617 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.061691+0000", 2020-01-16T00:13:58.617 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018817654999999999 2020-01-16T00:13:58.617 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.617 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.617 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.618 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.080509+0000", 2020-01-16T00:13:58.618 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000104409 2020-01-16T00:13:58.618 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.618 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.618 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:58.618 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.080613+0000", 2020-01-16T00:13:58.618 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00098755999999999996 2020-01-16T00:13:58.618 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.619 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.619 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.619 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081601+0000", 2020-01-16T00:13:58.619 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000145766 2020-01-16T00:13:58.619 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.619 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.619 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:58.619 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081746+0000", 2020-01-16T00:13:58.620 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013751799999999999 2020-01-16T00:13:58.620 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.620 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.620 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:58.620 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081884+0000", 2020-01-16T00:13:58.620 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022245396000000001 2020-01-16T00:13:58.620 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.620 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:58.621 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.621 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.621 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.621 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4398 3.1 3:8de1d8a3:::smithi12313934-13:head [write 2812170~507386] snapc 0=[] ondisk+write+known_if_redirected e55)", 2020-01-16T00:13:58.621 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:58.336543+0000", 2020-01-16T00:13:58.621 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.066572200999999998, 2020-01-16T00:13:58.621 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015697994, 2020-01-16T00:13:58.622 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:58.622 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:58.622 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:58.622 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:58.622 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:58.622 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4398 2020-01-16T00:13:58.622 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.622 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:58.623 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.623 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:58.623 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336543+0000", 2020-01-16T00:13:58.623 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:58.623 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.623 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.623 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:58.624 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336543+0000", 2020-01-16T00:13:58.624 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:58.624 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.624 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.624 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:58.624 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336542+0000", 2020-01-16T00:13:58.624 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014254 2020-01-16T00:13:58.624 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.625 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.625 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:58.625 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336684+0000", 2020-01-16T00:13:58.625 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2019e-05 2020-01-16T00:13:58.625 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.625 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.625 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:58.625 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336696+0000", 2020-01-16T00:13:58.626 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4269999999999996e-06 2020-01-16T00:13:58.626 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.626 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.626 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:58.626 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336702+0000", 2020-01-16T00:13:58.626 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8040000000000002e-05 2020-01-16T00:13:58.626 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.626 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.627 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.627 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336741+0000", 2020-01-16T00:13:58.627 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00030631400000000002 2020-01-16T00:13:58.627 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.627 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.627 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:58.627 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.337047+0000", 2020-01-16T00:13:58.627 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012249832 2020-01-16T00:13:58.628 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.628 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.628 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.628 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.349297+0000", 2020-01-16T00:13:58.628 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013870899999999999 2020-01-16T00:13:58.628 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.628 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.628 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:58.629 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.349435+0000", 2020-01-16T00:13:58.629 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021913600000000001 2020-01-16T00:13:58.629 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.629 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.629 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:13:58.629 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.349655+0000", 2020-01-16T00:13:58.629 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021982490000000002 2020-01-16T00:13:58.629 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.630 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.630 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:58.630 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351853+0000", 2020-01-16T00:13:58.630 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8295999999999996e-05 2020-01-16T00:13:58.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.630 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.630 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:58.631 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351941+0000", 2020-01-16T00:13:58.631 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021662 2020-01-16T00:13:58.631 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.631 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.631 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:58.631 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352158+0000", 2020-01-16T00:13:58.631 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6031000000000001e-05 2020-01-16T00:13:58.631 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.632 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.632 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:58.632 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352194+0000", 2020-01-16T00:13:58.632 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7290000000000003e-05 2020-01-16T00:13:58.632 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.632 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.632 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:58.632 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352241+0000", 2020-01-16T00:13:58.633 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015697994 2020-01-16T00:13:58.633 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.633 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:58.633 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.633 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.633 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.633 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4399 3.1 3:8de1d8a3:::smithi12313934-13:head [setxattr _header (56) in=7b,truncate 3411920] snapc 0=[] ondisk+write+known_if_redirected e55)", 2020-01-16T00:13:58.633 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:58.336714+0000", 2020-01-16T00:13:58.634 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.066400844000000001, 2020-01-16T00:13:58.634 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016051421, 2020-01-16T00:13:58.634 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:58.634 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:58.634 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:58.634 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:58.634 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:58.634 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4399 2020-01-16T00:13:58.635 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.635 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:58.635 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.635 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:58.635 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336714+0000", 2020-01-16T00:13:58.635 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:58.635 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.635 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.636 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:58.636 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336714+0000", 2020-01-16T00:13:58.636 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:58.636 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.636 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.636 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:58.636 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336714+0000", 2020-01-16T00:13:58.636 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7540000000000003e-06 2020-01-16T00:13:58.637 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.637 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.637 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:58.637 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336719+0000", 2020-01-16T00:13:58.637 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2879999999999998e-06 2020-01-16T00:13:58.637 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.637 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.637 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:58.638 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336726+0000", 2020-01-16T00:13:58.638 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6399999999999996e-06 2020-01-16T00:13:58.638 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.638 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.638 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:58.638 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336730+0000", 2020-01-16T00:13:58.638 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00036299699999999999 2020-01-16T00:13:58.639 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.639 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.639 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.639 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.337093+0000", 2020-01-16T00:13:58.639 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1297999999999994e-05 2020-01-16T00:13:58.639 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.639 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.639 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:58.640 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.337165+0000", 2020-01-16T00:13:58.640 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013848911 2020-01-16T00:13:58.640 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.640 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.640 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.640 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351013+0000", 2020-01-16T00:13:58.640 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7732999999999994e-05 2020-01-16T00:13:58.640 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.641 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.641 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:58.641 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351111+0000", 2020-01-16T00:13:58.641 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000109089 2020-01-16T00:13:58.641 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.641 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.641 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:13:58.641 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351220+0000", 2020-01-16T00:13:58.642 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00063960199999999997 2020-01-16T00:13:58.642 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.642 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.642 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:58.642 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351860+0000", 2020-01-16T00:13:58.643 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00050205699999999998 2020-01-16T00:13:58.643 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.643 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.643 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:58.643 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352362+0000", 2020-01-16T00:13:58.643 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000282195 2020-01-16T00:13:58.643 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.644 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.644 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:58.644 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352644+0000", 2020-01-16T00:13:58.644 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3927000000000001e-05 2020-01-16T00:13:58.644 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.645 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.645 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:58.645 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352698+0000", 2020-01-16T00:13:58.646 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7770999999999999e-05 2020-01-16T00:13:58.646 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.646 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.646 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:58.647 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352766+0000", 2020-01-16T00:13:58.647 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016051421 2020-01-16T00:13:58.647 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.647 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:58.648 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.648 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.648 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.648 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4400 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e55)", 2020-01-16T00:13:58.649 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:58.336738+0000", 2020-01-16T00:13:58.649 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.066377115, 2020-01-16T00:13:58.649 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016207452000000001, 2020-01-16T00:13:58.649 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:58.649 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:58.650 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:58.650 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:58.650 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:58.650 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4400 2020-01-16T00:13:58.650 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.651 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:58.651 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.651 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:58.651 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336738+0000", 2020-01-16T00:13:58.651 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:58.651 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.651 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.652 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:58.652 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336738+0000", 2020-01-16T00:13:58.652 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:58.652 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.652 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.652 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:58.652 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336738+0000", 2020-01-16T00:13:58.652 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5200000000000002e-06 2020-01-16T00:13:58.653 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.653 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.653 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:58.653 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336741+0000", 2020-01-16T00:13:58.653 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1500000000000004e-06 2020-01-16T00:13:58.653 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.653 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.654 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:58.654 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336747+0000", 2020-01-16T00:13:58.654 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7500000000000001e-06 2020-01-16T00:13:58.654 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.654 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.654 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:58.654 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336751+0000", 2020-01-16T00:13:58.654 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043512299999999999 2020-01-16T00:13:58.655 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.655 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.655 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.655 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.337186+0000", 2020-01-16T00:13:58.655 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7048000000000002e-05 2020-01-16T00:13:58.655 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.655 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.655 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:58.656 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.337223+0000", 2020-01-16T00:13:58.656 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014258785 2020-01-16T00:13:58.656 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.656 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.656 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.656 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351482+0000", 2020-01-16T00:13:58.656 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012002000000000001 2020-01-16T00:13:58.657 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.657 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.657 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:58.657 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351602+0000", 2020-01-16T00:13:58.657 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001192404 2020-01-16T00:13:58.657 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.657 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.657 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:58.658 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352794+0000", 2020-01-16T00:13:58.658 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8901999999999998e-05 2020-01-16T00:13:58.658 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.658 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.658 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:58.658 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352873+0000", 2020-01-16T00:13:58.658 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2317999999999994e-05 2020-01-16T00:13:58.658 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:58.659 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:58.659 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:58.659 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352946+0000", 2020-01-16T00:13:58.659 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016207452000000001 2020-01-16T00:13:58.659 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.659 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:58.659 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.659 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:58.660 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:58.660 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:58.661 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-01-16T00:13:58.662 INFO:tasks.rados.rados.0.smithi123.stdout:1590: finishing write tid 1 to smithi12313934-50 2020-01-16T00:13:58.662 INFO:tasks.rados.rados.0.smithi123.stdout:1590: finishing write tid 2 to smithi12313934-50 2020-01-16T00:13:58.662 INFO:tasks.rados.rados.0.smithi123.stdout:1590: finishing write tid 3 to smithi12313934-50 2020-01-16T00:13:58.662 INFO:tasks.rados.rados.0.smithi123.stdout:1591: expect (ObjNum 545 snap 0 seq_num 545) 2020-01-16T00:13:58.662 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:13:58.656+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:13:58.762 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:58.760+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:58.788 INFO:tasks.rados.rados.0.smithi123.stdout:1592: expect (ObjNum 441 snap 0 seq_num 441) 2020-01-16T00:13:58.810 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:58.811 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:58.811 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:13:58.811 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:58.825 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-01-16T00:13:58.863 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:58.860+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:58.920 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:58.921 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:58.921 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:13:58.921 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:13:58.921 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:58.929 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-01-16T00:13:58.975 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:13:58.974+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:13:59.004 INFO:tasks.rados.rados.0.smithi123.stdout:1590: finishing write tid 5 to smithi12313934-50 2020-01-16T00:13:59.004 INFO:tasks.rados.rados.0.smithi123.stdout:1590: finishing write tid 6 to smithi12313934-50 2020-01-16T00:13:59.004 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 50 v 199 (ObjNum 553 snap 0 seq_num 553) dirty exists 2020-01-16T00:13:59.005 INFO:tasks.rados.rados.0.smithi123.stdout:1590: left oid 50 (ObjNum 553 snap 0 seq_num 553) 2020-01-16T00:13:59.005 INFO:tasks.rados.rados.0.smithi123.stdout:1593: finishing write tid 1 to smithi12313934-19 2020-01-16T00:13:59.005 INFO:tasks.rados.rados.0.smithi123.stdout:1593: finishing write tid 2 to smithi12313934-19 2020-01-16T00:13:59.005 INFO:tasks.rados.rados.0.smithi123.stdout:1593: finishing write tid 3 to smithi12313934-19 2020-01-16T00:13:59.006 INFO:tasks.rados.rados.0.smithi123.stdout:1587: done (15 left) 2020-01-16T00:13:59.008 INFO:tasks.rados.rados.0.smithi123.stdout:1589: done (14 left) 2020-01-16T00:13:59.008 INFO:tasks.rados.rados.0.smithi123.stdout:1590: done (13 left) 2020-01-16T00:13:59.009 INFO:tasks.rados.rados.0.smithi123.stdout:1591: done (12 left) 2020-01-16T00:13:59.009 INFO:tasks.rados.rados.0.smithi123.stdout:1592: done (11 left) 2020-01-16T00:13:59.009 INFO:tasks.rados.rados.0.smithi123.stdout:1604: read oid 43 snap -1 2020-01-16T00:13:59.010 INFO:tasks.rados.rados.0.smithi123.stdout:1604: expect (ObjNum 539 snap 0 seq_num 539) 2020-01-16T00:13:59.010 INFO:tasks.rados.rados.0.smithi123.stdout:1605: read oid 20 snap -1 2020-01-16T00:13:59.014 INFO:tasks.rados.rados.0.smithi123.stdout:1605: expect deleted 2020-01-16T00:13:59.015 INFO:tasks.rados.rados.0.smithi123.stdout:1606: write oid 38 current snap is 0 2020-01-16T00:13:59.015 INFO:tasks.rados.rados.0.smithi123.stdout:1606: seq_num 561 ranges {628727=708458,2045457=494859,3123240=305487} 2020-01-16T00:13:59.018 INFO:tasks.rados.rados.0.smithi123.stdout:1606: writing smithi12313934-38 from 628727 to 1337185 tid 1 2020-01-16T00:13:59.026 INFO:tasks.rados.rados.0.smithi123.stdout:1606: writing smithi12313934-38 from 2045457 to 2540316 tid 2 2020-01-16T00:13:59.031 INFO:tasks.rados.rados.0.smithi123.stdout:1606: writing smithi12313934-38 from 3123240 to 3428727 tid 3 2020-01-16T00:13:59.032 INFO:tasks.rados.rados.0.smithi123.stdout:1607: write (excl) oid 27 current snap is 0 2020-01-16T00:13:59.032 INFO:tasks.rados.rados.0.smithi123.stdout:1607: seq_num 562 ranges {645534=783707,2103939=638517,3150663=294871} 2020-01-16T00:13:59.039 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:59.039 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:13:59.039 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:13:59.040 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:13:59.040 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.040 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:923 3.8 3:113947f2:::smithi12313934-12:head [write 1564138~470341] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:13:59.040 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.869292+0000", 2020-01-16T00:13:59.040 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 81.166872982000001, 2020-01-16T00:13:59.041 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017661620999999999, 2020-01-16T00:13:59.041 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.041 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:59.041 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:59.041 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:59.041 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:59.042 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 923 2020-01-16T00:13:59.042 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.042 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.042 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.043 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.043 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869292+0000", 2020-01-16T00:13:59.043 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.043 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.045 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.045 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.046 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869292+0000", 2020-01-16T00:13:59.046 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:59.046 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.046 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.046 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.047 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869291+0000", 2020-01-16T00:13:59.047 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00049842899999999999 2020-01-16T00:13:59.047 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.047 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.047 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.047 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869789+0000", 2020-01-16T00:13:59.047 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1014999999999999e-05 2020-01-16T00:13:59.048 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.048 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.048 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.048 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869830+0000", 2020-01-16T00:13:59.048 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7880000000000007e-06 2020-01-16T00:13:59.048 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.048 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.049 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.049 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869838+0000", 2020-01-16T00:13:59.049 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8346999999999997e-05 2020-01-16T00:13:59.049 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.049 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.049 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.049 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869876+0000", 2020-01-16T00:13:59.049 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9726999999999999e-05 2020-01-16T00:13:59.050 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.050 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.050 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:59.050 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869936+0000", 2020-01-16T00:13:59.050 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003434917 2020-01-16T00:13:59.050 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.050 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.051 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.051 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.873371+0000", 2020-01-16T00:13:59.051 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118808 2020-01-16T00:13:59.051 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.051 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.051 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:59.051 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.873490+0000", 2020-01-16T00:13:59.051 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036712910000000001 2020-01-16T00:13:59.052 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.052 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.052 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.052 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877161+0000", 2020-01-16T00:13:59.052 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9909999999999999e-05 2020-01-16T00:13:59.052 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.052 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.052 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:59.053 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877241+0000", 2020-01-16T00:13:59.053 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031619970000000002 2020-01-16T00:13:59.053 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.053 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.053 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.053 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880403+0000", 2020-01-16T00:13:59.053 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013071099999999999 2020-01-16T00:13:59.062 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.062 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.063 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.063 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880534+0000", 2020-01-16T00:13:59.063 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00032823500000000002 2020-01-16T00:13:59.063 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.064 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.064 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1,7", 2020-01-16T00:13:59.064 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880862+0000", 2020-01-16T00:13:59.064 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00513862 2020-01-16T00:13:59.064 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.065 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.065 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:59.065 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886001+0000", 2020-01-16T00:13:59.065 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8869000000000001e-05 2020-01-16T00:13:59.065 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.066 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.066 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.066 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886030+0000", 2020-01-16T00:13:59.066 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00081600200000000003 2020-01-16T00:13:59.066 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.067 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.067 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.067 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886846+0000", 2020-01-16T00:13:59.067 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4095999999999998e-05 2020-01-16T00:13:59.067 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.068 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.068 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:59.068 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886900+0000", 2020-01-16T00:13:59.068 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.448e-05 2020-01-16T00:13:59.068 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.069 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.069 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:59.069 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886954+0000", 2020-01-16T00:13:59.069 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017661620999999999 2020-01-16T00:13:59.070 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.070 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.070 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.070 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.071 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.071 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2226 3.7 3:e1be6d3b:::smithi12313934-24:head [setxattr _header (56) in=7b,truncate 2100908] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:13:59.071 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.980217+0000", 2020-01-16T00:13:59.071 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 53.055948317000002, 2020-01-16T00:13:59.072 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023113018999999999, 2020-01-16T00:13:59.072 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.072 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:59.072 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:59.072 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:59.073 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:59.073 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2226 2020-01-16T00:13:59.073 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.073 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.073 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.074 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.074 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980217+0000", 2020-01-16T00:13:59.074 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.074 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.074 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.075 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.075 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980217+0000", 2020-01-16T00:13:59.078 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:59.078 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.078 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.079 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.079 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980216+0000", 2020-01-16T00:13:59.079 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5049999999999998e-06 2020-01-16T00:13:59.079 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.079 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.079 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.080 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980222+0000", 2020-01-16T00:13:59.080 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5654999999999998e-05 2020-01-16T00:13:59.080 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.080 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.080 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.081 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980237+0000", 2020-01-16T00:13:59.081 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9740000000000004e-06 2020-01-16T00:13:59.081 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.081 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.081 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.081 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980241+0000", 2020-01-16T00:13:59.082 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002947776 2020-01-16T00:13:59.082 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.082 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.082 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.083 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.983189+0000", 2020-01-16T00:13:59.083 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0027099400000000001 2020-01-16T00:13:59.083 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.083 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.083 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:59.084 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985899+0000", 2020-01-16T00:13:59.084 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014275589999999999 2020-01-16T00:13:59.084 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.084 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.085 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.085 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.000175+0000", 2020-01-16T00:13:59.085 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011211 2020-01-16T00:13:59.086 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.086 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.086 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.086 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.000287+0000", 2020-01-16T00:13:59.087 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000229886 2020-01-16T00:13:59.087 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.087 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.087 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,7", 2020-01-16T00:13:59.087 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.000517+0000", 2020-01-16T00:13:59.088 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00095715399999999999 2020-01-16T00:13:59.088 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.088 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.088 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:59.089 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.001474+0000", 2020-01-16T00:13:59.089 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001341512 2020-01-16T00:13:59.089 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.089 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.090 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.090 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.002815+0000", 2020-01-16T00:13:59.090 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035789099999999999 2020-01-16T00:13:59.090 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.090 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.091 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.091 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003173+0000", 2020-01-16T00:13:59.091 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8571000000000005e-05 2020-01-16T00:13:59.091 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.092 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.092 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:59.092 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003242+0000", 2020-01-16T00:13:59.092 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8293000000000003e-05 2020-01-16T00:13:59.092 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.093 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.093 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:59.093 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003330+0000", 2020-01-16T00:13:59.093 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023113018999999999 2020-01-16T00:13:59.094 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.094 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.094 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.094 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.094 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.095 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2227 3.7 3:e1be6d3b:::smithi12313934-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e35)", 2020-01-16T00:13:59.095 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:13:59.095 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 53.055901063999997, 2020-01-16T00:13:59.095 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023303356000000001, 2020-01-16T00:13:59.096 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.096 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:59.096 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:59.096 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:59.096 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:59.097 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2227 2020-01-16T00:13:59.097 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.097 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.097 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.106 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.106 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:13:59.106 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.106 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.106 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.107 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.107 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:13:59.107 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:59.107 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.108 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.108 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.108 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:13:59.108 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3079999999999999e-06 2020-01-16T00:13:59.108 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.108 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.108 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.108 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980267+0000", 2020-01-16T00:13:59.109 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9290000000000001e-06 2020-01-16T00:13:59.109 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.109 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.109 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.109 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980274+0000", 2020-01-16T00:13:59.109 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7400000000000002e-06 2020-01-16T00:13:59.110 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.110 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.110 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.110 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980278+0000", 2020-01-16T00:13:59.110 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056624300000000004 2020-01-16T00:13:59.110 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.110 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.110 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.111 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985940+0000", 2020-01-16T00:13:59.111 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4261000000000002e-05 2020-01-16T00:13:59.111 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.111 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.111 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:59.111 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985985+0000", 2020-01-16T00:13:59.111 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015087676 2020-01-16T00:13:59.111 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.112 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.112 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.112 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.001072+0000", 2020-01-16T00:13:59.112 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011960600000000001 2020-01-16T00:13:59.112 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.112 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.112 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:59.113 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.001192+0000", 2020-01-16T00:13:59.113 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021718309999999999 2020-01-16T00:13:59.113 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.113 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.113 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.113 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003364+0000", 2020-01-16T00:13:59.113 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000102855 2020-01-16T00:13:59.114 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.114 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.114 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.114 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003467+0000", 2020-01-16T00:13:59.114 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010122499999999999 2020-01-16T00:13:59.114 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.115 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.115 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:59.115 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003568+0000", 2020-01-16T00:13:59.115 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023303356000000001 2020-01-16T00:13:59.115 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.115 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.115 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.115 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.116 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.116 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2233 3.2 3:4f8fe446:::smithi12313934-48:head [setxattr _header (56) in=7b,truncate 2117715] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:13:59.116 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.993103+0000", 2020-01-16T00:13:59.116 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 53.043062841999998, 2020-01-16T00:13:59.116 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017951763999999999, 2020-01-16T00:13:59.116 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.116 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:59.116 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:59.117 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:59.117 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:59.117 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2233 2020-01-16T00:13:59.117 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.117 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.117 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.117 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.117 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993103+0000", 2020-01-16T00:13:59.118 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.118 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.118 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.118 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.118 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993103+0000", 2020-01-16T00:13:59.118 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:59.118 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.119 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.119 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.119 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993102+0000", 2020-01-16T00:13:59.119 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9369999999999993e-06 2020-01-16T00:13:59.119 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.119 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.119 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.119 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993111+0000", 2020-01-16T00:13:59.120 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3340000000000002e-06 2020-01-16T00:13:59.120 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.120 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.120 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.120 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993119+0000", 2020-01-16T00:13:59.120 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1769999999999997e-06 2020-01-16T00:13:59.120 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.120 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.121 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.121 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993124+0000", 2020-01-16T00:13:59.121 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001601304 2020-01-16T00:13:59.121 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.121 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.121 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.121 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.994725+0000", 2020-01-16T00:13:59.122 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048777250000000003 2020-01-16T00:13:59.122 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.122 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.122 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:59.122 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999603+0000", 2020-01-16T00:13:59.122 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0099793880000000005 2020-01-16T00:13:59.122 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.122 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.123 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.123 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.009583+0000", 2020-01-16T00:13:59.123 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000108207 2020-01-16T00:13:59.123 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.123 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.123 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.123 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.009691+0000", 2020-01-16T00:13:59.123 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012862999999999999 2020-01-16T00:13:59.124 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.124 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.124 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:13:59.124 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.009819+0000", 2020-01-16T00:13:59.124 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00077947599999999998 2020-01-16T00:13:59.124 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.124 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.124 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:59.125 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010599+0000", 2020-01-16T00:13:59.125 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000221282 2020-01-16T00:13:59.125 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.125 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.125 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.125 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010820+0000", 2020-01-16T00:13:59.125 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7021999999999995e-05 2020-01-16T00:13:59.126 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.126 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.126 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.126 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010907+0000", 2020-01-16T00:13:59.126 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9815999999999997e-05 2020-01-16T00:13:59.126 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.126 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.126 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:59.127 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010977+0000", 2020-01-16T00:13:59.127 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7453999999999994e-05 2020-01-16T00:13:59.127 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.127 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.127 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:59.127 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.011054+0000", 2020-01-16T00:13:59.127 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017951763999999999 2020-01-16T00:13:59.128 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.128 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.128 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.128 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.128 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.128 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2234 3.2 3:4f8fe446:::smithi12313934-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e35)", 2020-01-16T00:13:59.128 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.993135+0000", 2020-01-16T00:13:59.128 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 53.043030389000002, 2020-01-16T00:13:59.129 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018067916999999999, 2020-01-16T00:13:59.129 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.129 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:59.129 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:59.129 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:59.129 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:59.129 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2234 2020-01-16T00:13:59.130 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.130 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.130 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.130 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.130 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993135+0000", 2020-01-16T00:13:59.130 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.130 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.130 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.131 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.131 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993135+0000", 2020-01-16T00:13:59.131 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:59.131 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.131 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.131 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.131 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993134+0000", 2020-01-16T00:13:59.131 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5630000000000002e-06 2020-01-16T00:13:59.132 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.132 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.132 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.132 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993139+0000", 2020-01-16T00:13:59.132 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1730000000000003e-06 2020-01-16T00:13:59.132 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.132 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.132 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.133 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993144+0000", 2020-01-16T00:13:59.133 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.937e-06 2020-01-16T00:13:59.133 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.133 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.133 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.133 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993147+0000", 2020-01-16T00:13:59.133 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0065164000000000003 2020-01-16T00:13:59.134 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.134 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.134 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.134 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999663+0000", 2020-01-16T00:13:59.134 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6215000000000002e-05 2020-01-16T00:13:59.134 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.134 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.134 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:59.135 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999729+0000", 2020-01-16T00:13:59.135 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010475887999999999 2020-01-16T00:13:59.135 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.135 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.135 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.135 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010205+0000", 2020-01-16T00:13:59.135 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119471 2020-01-16T00:13:59.135 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.136 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.136 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:59.136 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010325+0000", 2020-01-16T00:13:59.136 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00075498799999999995 2020-01-16T00:13:59.136 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.136 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.136 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.136 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.011080+0000", 2020-01-16T00:13:59.137 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7904e-05 2020-01-16T00:13:59.137 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.137 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.137 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.137 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.011138+0000", 2020-01-16T00:13:59.137 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5273999999999999e-05 2020-01-16T00:13:59.137 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.138 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.138 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:59.138 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.011203+0000", 2020-01-16T00:13:59.138 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018067916999999999 2020-01-16T00:13:59.138 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.138 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.138 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.138 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.139 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.139 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2324 3.7 3:e1be6d3b:::smithi12313934-24:head [setxattr _header (56) in=7b,truncate 2336206] snapc 0=[] ondisk+write+known_if_redirected e36)", 2020-01-16T00:13:59.139 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.550846+0000", 2020-01-16T00:13:59.139 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 51.485319803000003, 2020-01-16T00:13:59.139 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018364742, 2020-01-16T00:13:59.139 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.139 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:59.140 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:59.140 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:59.140 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:59.140 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2324 2020-01-16T00:13:59.140 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.140 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.140 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.140 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.141 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550846+0000", 2020-01-16T00:13:59.141 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.141 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.141 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.141 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.141 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550846+0000", 2020-01-16T00:13:59.141 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:59.142 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.142 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.142 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.142 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550845+0000", 2020-01-16T00:13:59.142 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1829999999999999e-06 2020-01-16T00:13:59.142 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.142 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.142 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.143 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550851+0000", 2020-01-16T00:13:59.143 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1925e-05 2020-01-16T00:13:59.143 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.143 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.143 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.143 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550863+0000", 2020-01-16T00:13:59.143 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1139999999999997e-06 2020-01-16T00:13:59.144 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.144 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.144 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.144 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550868+0000", 2020-01-16T00:13:59.144 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001388423 2020-01-16T00:13:59.144 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.144 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.144 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.145 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.552256+0000", 2020-01-16T00:13:59.145 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0034246540000000001 2020-01-16T00:13:59.145 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.145 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.145 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:59.145 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555681+0000", 2020-01-16T00:13:59.145 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011882762999999999 2020-01-16T00:13:59.145 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.146 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.146 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.146 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.567564+0000", 2020-01-16T00:13:59.146 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015010800000000001 2020-01-16T00:13:59.146 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.146 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.146 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.146 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.567714+0000", 2020-01-16T00:13:59.147 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011527999999999999 2020-01-16T00:13:59.147 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.147 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.147 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,7", 2020-01-16T00:13:59.147 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.567829+0000", 2020-01-16T00:13:59.147 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00039888499999999997 2020-01-16T00:13:59.147 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.147 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.148 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:59.148 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568228+0000", 2020-01-16T00:13:59.148 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00054254599999999998 2020-01-16T00:13:59.148 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.148 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.148 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.149 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568771+0000", 2020-01-16T00:13:59.149 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035206400000000002 2020-01-16T00:13:59.149 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.149 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.149 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.149 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569123+0000", 2020-01-16T00:13:59.149 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1262e-05 2020-01-16T00:13:59.149 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.150 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.150 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:59.150 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569154+0000", 2020-01-16T00:13:59.150 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6499000000000002e-05 2020-01-16T00:13:59.150 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.150 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.150 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:59.150 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569210+0000", 2020-01-16T00:13:59.151 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018364742 2020-01-16T00:13:59.151 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.151 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.151 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.151 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.151 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.151 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2325 3.7 3:e1be6d3b:::smithi12313934-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e36)", 2020-01-16T00:13:59.152 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.550877+0000", 2020-01-16T00:13:59.152 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 51.485288736999998, 2020-01-16T00:13:59.152 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018565154, 2020-01-16T00:13:59.152 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.152 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:59.152 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:59.152 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:59.152 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:59.153 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2325 2020-01-16T00:13:59.153 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.153 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.153 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.153 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.153 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550877+0000", 2020-01-16T00:13:59.153 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.154 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.154 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.154 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.154 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550877+0000", 2020-01-16T00:13:59.154 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:59.154 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.154 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.154 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.155 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550876+0000", 2020-01-16T00:13:59.155 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1080000000000003e-06 2020-01-16T00:13:59.155 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.155 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.155 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.155 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550880+0000", 2020-01-16T00:13:59.155 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3269999999999997e-06 2020-01-16T00:13:59.155 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.156 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.156 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.156 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550886+0000", 2020-01-16T00:13:59.156 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.386e-06 2020-01-16T00:13:59.156 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.156 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.156 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.156 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550891+0000", 2020-01-16T00:13:59.157 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048231410000000004 2020-01-16T00:13:59.157 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.157 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.157 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.157 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555714+0000", 2020-01-16T00:13:59.157 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0071999999999999e-05 2020-01-16T00:13:59.157 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.158 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.158 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:59.158 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555744+0000", 2020-01-16T00:13:59.158 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012329005000000001 2020-01-16T00:13:59.158 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.158 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.158 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.158 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568073+0000", 2020-01-16T00:13:59.159 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9087999999999997e-05 2020-01-16T00:13:59.159 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.159 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.159 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:59.159 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568172+0000", 2020-01-16T00:13:59.159 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010718220000000001 2020-01-16T00:13:59.159 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.159 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.160 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.160 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569244+0000", 2020-01-16T00:13:59.160 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8199e-05 2020-01-16T00:13:59.160 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.160 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.160 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.160 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569342+0000", 2020-01-16T00:13:59.161 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9708000000000001e-05 2020-01-16T00:13:59.161 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.161 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.161 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:59.161 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569442+0000", 2020-01-16T00:13:59.161 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018565154 2020-01-16T00:13:59.161 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.161 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.162 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.162 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.162 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.162 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2800 3.2 3:42b7f20f:::smithi12313934-42:head [write 1395221~583781] snapc 0=[] ondisk+write+known_if_redirected e41)", 2020-01-16T00:13:59.162 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:17.256182+0000", 2020-01-16T00:13:59.162 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 41.779983197, 2020-01-16T00:13:59.162 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018509650999999998, 2020-01-16T00:13:59.162 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.163 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:59.163 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:59.163 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:59.163 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:59.163 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2800 2020-01-16T00:13:59.163 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.163 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.164 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.164 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.164 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256182+0000", 2020-01-16T00:13:59.164 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.164 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.164 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.164 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.164 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256182+0000", 2020-01-16T00:13:59.165 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:59.165 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.165 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.165 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.165 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256180+0000", 2020-01-16T00:13:59.165 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020514799999999999 2020-01-16T00:13:59.165 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.165 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.166 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.166 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256385+0000", 2020-01-16T00:13:59.166 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0288e-05 2020-01-16T00:13:59.166 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.166 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.166 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.166 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256406+0000", 2020-01-16T00:13:59.167 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7160000000000003e-06 2020-01-16T00:13:59.167 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.167 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.167 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.167 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256413+0000", 2020-01-16T00:13:59.167 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5293999999999998e-05 2020-01-16T00:13:59.167 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.167 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.168 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.168 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256449+0000", 2020-01-16T00:13:59.168 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001297737 2020-01-16T00:13:59.168 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.168 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.168 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:59.168 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.257746+0000", 2020-01-16T00:13:59.168 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0079866529999999998 2020-01-16T00:13:59.169 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.169 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.169 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.169 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.265733+0000", 2020-01-16T00:13:59.169 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4028000000000004e-05 2020-01-16T00:13:59.169 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.169 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.170 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.170 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.265797+0000", 2020-01-16T00:13:59.170 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011338899999999999 2020-01-16T00:13:59.170 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.170 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.170 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:59.171 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.265910+0000", 2020-01-16T00:13:59.171 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003372496 2020-01-16T00:13:59.171 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.171 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.171 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:59.171 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.269283+0000", 2020-01-16T00:13:59.171 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0053107959999999996 2020-01-16T00:13:59.171 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.172 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.172 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.172 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.274594+0000", 2020-01-16T00:13:59.172 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8000999999999998e-05 2020-01-16T00:13:59.172 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.172 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.172 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:59.172 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.274632+0000", 2020-01-16T00:13:59.173 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0265e-05 2020-01-16T00:13:59.173 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.173 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.173 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:59.173 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.274692+0000", 2020-01-16T00:13:59.173 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018509650999999998 2020-01-16T00:13:59.173 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.173 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.174 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.174 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.174 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.174 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2971 3.c 3:3025546b:::smithi12313934-15:head [write 1424565~609148] snapc 0=[] ondisk+write+known_if_redirected e42)", 2020-01-16T00:13:59.174 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:21.839711+0000", 2020-01-16T00:13:59.174 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 37.196454353999997, 2020-01-16T00:13:59.174 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017534376000000001, 2020-01-16T00:13:59.175 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.175 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:59.175 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:59.175 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:59.175 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:59.175 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2971 2020-01-16T00:13:59.175 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.175 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.176 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.176 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.176 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839711+0000", 2020-01-16T00:13:59.176 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.176 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.176 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.176 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.176 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839711+0000", 2020-01-16T00:13:59.177 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:13:59.177 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.177 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.177 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.177 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839710+0000", 2020-01-16T00:13:59.177 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017658600000000001 2020-01-16T00:13:59.177 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.178 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.178 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.178 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839886+0000", 2020-01-16T00:13:59.178 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0492000000000001e-05 2020-01-16T00:13:59.178 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.178 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.178 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.178 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839897+0000", 2020-01-16T00:13:59.179 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.019e-06 2020-01-16T00:13:59.179 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.179 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.179 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.179 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839901+0000", 2020-01-16T00:13:59.179 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5572000000000001e-05 2020-01-16T00:13:59.179 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.179 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.180 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.180 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839976+0000", 2020-01-16T00:13:59.180 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034616099999999999 2020-01-16T00:13:59.180 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.180 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.180 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:59.180 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.840323+0000", 2020-01-16T00:13:59.181 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.01246249 2020-01-16T00:13:59.181 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.181 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.181 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.181 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.852785+0000", 2020-01-16T00:13:59.181 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014474799999999999 2020-01-16T00:13:59.181 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.181 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.182 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.182 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.852930+0000", 2020-01-16T00:13:59.182 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000213907 2020-01-16T00:13:59.182 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.182 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.183 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:59.183 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.853144+0000", 2020-01-16T00:13:59.183 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0028426979999999998 2020-01-16T00:13:59.183 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.183 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.183 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.183 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.855986+0000", 2020-01-16T00:13:59.184 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010694280000000001 2020-01-16T00:13:59.184 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.184 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.184 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:59.184 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.857056+0000", 2020-01-16T00:13:59.184 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8935000000000003e-05 2020-01-16T00:13:59.184 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.184 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.185 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:59.185 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.857125+0000", 2020-01-16T00:13:59.185 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000120721 2020-01-16T00:13:59.185 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.185 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.185 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:59.185 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.857245+0000", 2020-01-16T00:13:59.186 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017534376000000001 2020-01-16T00:13:59.186 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.186 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.186 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.186 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.186 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.186 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2972 3.c 3:3025546b:::smithi12313934-15:head [setxattr _header (56) in=7b,truncate 2033713] snapc 0=[] ondisk+write+known_if_redirected e42)", 2020-01-16T00:13:59.186 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:21.839923+0000", 2020-01-16T00:13:59.187 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 37.196242089999998, 2020-01-16T00:13:59.187 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018367108, 2020-01-16T00:13:59.187 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.187 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:59.187 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:59.187 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:59.187 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:59.187 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2972 2020-01-16T00:13:59.188 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.188 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.188 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.188 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.188 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839923+0000", 2020-01-16T00:13:59.188 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.188 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.189 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.189 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.189 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839923+0000", 2020-01-16T00:13:59.189 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:59.189 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.189 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.189 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.189 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839923+0000", 2020-01-16T00:13:59.190 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0629999999999999e-06 2020-01-16T00:13:59.190 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.190 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.190 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.190 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839927+0000", 2020-01-16T00:13:59.190 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1980000000000003e-06 2020-01-16T00:13:59.190 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.190 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.191 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.191 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839931+0000", 2020-01-16T00:13:59.191 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5979999999999999e-06 2020-01-16T00:13:59.191 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.191 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.191 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.191 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839934+0000", 2020-01-16T00:13:59.191 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00042258500000000001 2020-01-16T00:13:59.192 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.192 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.192 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.192 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.840356+0000", 2020-01-16T00:13:59.192 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4919000000000002e-05 2020-01-16T00:13:59.192 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.192 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.193 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:59.193 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.840391+0000", 2020-01-16T00:13:59.193 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013783103 2020-01-16T00:13:59.193 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.193 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.193 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.193 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.854174+0000", 2020-01-16T00:13:59.193 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3455999999999995e-05 2020-01-16T00:13:59.194 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.194 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.194 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.194 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.854258+0000", 2020-01-16T00:13:59.194 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011060700000000001 2020-01-16T00:13:59.194 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.194 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.194 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:59.195 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.854368+0000", 2020-01-16T00:13:59.195 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020642500000000001 2020-01-16T00:13:59.195 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.195 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.196 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.196 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.856433+0000", 2020-01-16T00:13:59.196 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0016284820000000001 2020-01-16T00:13:59.196 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.197 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.197 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:59.197 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858061+0000", 2020-01-16T00:13:59.197 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0663999999999998e-05 2020-01-16T00:13:59.197 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.197 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.197 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:59.198 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858152+0000", 2020-01-16T00:13:59.198 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013882900000000001 2020-01-16T00:13:59.198 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.198 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.198 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:59.198 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858290+0000", 2020-01-16T00:13:59.198 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018367108 2020-01-16T00:13:59.198 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.199 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.199 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.199 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.199 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.199 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2973 3.c 3:3025546b:::smithi12313934-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e42)", 2020-01-16T00:13:59.199 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:21.839961+0000", 2020-01-16T00:13:59.199 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 37.196204596000001, 2020-01-16T00:13:59.199 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018691670000000001, 2020-01-16T00:13:59.200 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.200 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:59.200 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:59.200 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:59.200 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:59.200 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2973 2020-01-16T00:13:59.200 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.200 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.201 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.201 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.201 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839961+0000", 2020-01-16T00:13:59.201 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.201 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.201 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.201 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.202 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839961+0000", 2020-01-16T00:13:59.202 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:59.202 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.202 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.202 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.202 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839960+0000", 2020-01-16T00:13:59.202 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4060000000000001e-06 2020-01-16T00:13:59.202 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.203 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.203 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.203 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839964+0000", 2020-01-16T00:13:59.203 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5412000000000001e-05 2020-01-16T00:13:59.203 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.203 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.203 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.203 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839979+0000", 2020-01-16T00:13:59.204 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.444e-06 2020-01-16T00:13:59.204 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.204 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.204 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.204 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839983+0000", 2020-01-16T00:13:59.204 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000430028 2020-01-16T00:13:59.204 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.204 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.205 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.205 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.840413+0000", 2020-01-16T00:13:59.205 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9270999999999997e-05 2020-01-16T00:13:59.205 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.205 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.205 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:59.205 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.840452+0000", 2020-01-16T00:13:59.206 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014853975 2020-01-16T00:13:59.206 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.206 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.206 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.206 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.855306+0000", 2020-01-16T00:13:59.206 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011098400000000001 2020-01-16T00:13:59.206 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.206 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.207 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:59.207 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.855417+0000", 2020-01-16T00:13:59.207 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029108060000000002 2020-01-16T00:13:59.207 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.207 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.207 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.207 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858328+0000", 2020-01-16T00:13:59.207 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000144697 2020-01-16T00:13:59.208 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.208 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.208 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.208 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858472+0000", 2020-01-16T00:13:59.208 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018011899999999999 2020-01-16T00:13:59.208 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.208 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.209 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:59.209 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858653+0000", 2020-01-16T00:13:59.209 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018691670000000001 2020-01-16T00:13:59.209 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.209 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.209 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.209 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.209 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.210 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3295 3.2 3:42b7f20f:::smithi12313934-42:head [write 1982691~499457] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:13:59.210 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:30.744201+0000", 2020-01-16T00:13:59.210 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 28.291964320000002, 2020-01-16T00:13:59.210 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018346738000000001, 2020-01-16T00:13:59.210 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.210 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:59.210 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:59.210 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:59.211 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:59.211 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3295 2020-01-16T00:13:59.211 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.211 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.211 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.211 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.211 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744201+0000", 2020-01-16T00:13:59.211 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.212 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.212 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.212 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.212 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744201+0000", 2020-01-16T00:13:59.212 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:59.212 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.212 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.213 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.213 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744198+0000", 2020-01-16T00:13:59.213 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00030375699999999999 2020-01-16T00:13:59.213 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.213 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.213 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.213 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744502+0000", 2020-01-16T00:13:59.213 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0121e-05 2020-01-16T00:13:59.214 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.214 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.214 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.214 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744512+0000", 2020-01-16T00:13:59.214 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2293000000000001e-05 2020-01-16T00:13:59.214 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.214 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.214 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.215 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744524+0000", 2020-01-16T00:13:59.215 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000274837 2020-01-16T00:13:59.215 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.215 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.215 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.215 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744799+0000", 2020-01-16T00:13:59.216 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036951879999999999 2020-01-16T00:13:59.216 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.216 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.216 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:59.216 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748494+0000", 2020-01-16T00:13:59.216 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010091493 2020-01-16T00:13:59.216 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.217 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.217 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.217 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.758586+0000", 2020-01-16T00:13:59.217 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011674400000000001 2020-01-16T00:13:59.217 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.217 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.217 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.217 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.758703+0000", 2020-01-16T00:13:59.218 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021412699999999999 2020-01-16T00:13:59.218 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.218 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.218 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:59.218 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.758917+0000", 2020-01-16T00:13:59.218 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0022519340000000001 2020-01-16T00:13:59.218 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.218 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.219 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:59.219 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.761169+0000", 2020-01-16T00:13:59.219 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001240891 2020-01-16T00:13:59.219 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.219 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.219 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.219 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.762410+0000", 2020-01-16T00:13:59.219 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6942e-05 2020-01-16T00:13:59.220 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.220 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.220 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:59.220 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.762447+0000", 2020-01-16T00:13:59.220 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000101358 2020-01-16T00:13:59.220 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.220 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.220 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:59.221 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.762548+0000", 2020-01-16T00:13:59.221 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018346738000000001 2020-01-16T00:13:59.221 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.221 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.221 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.221 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.221 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.222 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3296 3.2 3:42b7f20f:::smithi12313934-42:head [setxattr _header (56) in=7b,truncate 2739607] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:13:59.222 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:30.744532+0000", 2020-01-16T00:13:59.222 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 28.291633587, 2020-01-16T00:13:59.222 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018472374, 2020-01-16T00:13:59.222 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.222 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:59.222 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:59.222 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:59.223 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:59.223 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3296 2020-01-16T00:13:59.223 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.223 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.223 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.223 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.223 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744532+0000", 2020-01-16T00:13:59.223 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.224 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.224 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.224 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.224 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744532+0000", 2020-01-16T00:13:59.224 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:59.224 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.224 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.224 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.225 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744531+0000", 2020-01-16T00:13:59.225 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7059999999999993e-06 2020-01-16T00:13:59.225 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.225 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.225 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.225 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744540+0000", 2020-01-16T00:13:59.225 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8589999999999998e-06 2020-01-16T00:13:59.226 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.226 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.226 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.226 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744544+0000", 2020-01-16T00:13:59.226 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7609999999999999e-06 2020-01-16T00:13:59.226 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.226 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.226 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.227 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744546+0000", 2020-01-16T00:13:59.227 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039933670000000003 2020-01-16T00:13:59.227 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.227 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.227 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.227 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748540+0000", 2020-01-16T00:13:59.227 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000110281 2020-01-16T00:13:59.227 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.228 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.228 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:59.228 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748650+0000", 2020-01-16T00:13:59.228 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.01160457 2020-01-16T00:13:59.228 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.228 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.228 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.228 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.760255+0000", 2020-01-16T00:13:59.229 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000124714 2020-01-16T00:13:59.229 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.229 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.229 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.229 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.760379+0000", 2020-01-16T00:13:59.229 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022099700000000001 2020-01-16T00:13:59.229 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.230 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.230 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:59.230 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.760600+0000", 2020-01-16T00:13:59.230 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00057843300000000005 2020-01-16T00:13:59.230 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.230 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.230 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:59.230 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.761179+0000", 2020-01-16T00:13:59.231 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00161477 2020-01-16T00:13:59.231 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.231 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.231 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.231 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.762794+0000", 2020-01-16T00:13:59.231 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4408999999999999e-05 2020-01-16T00:13:59.231 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.232 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.232 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:59.232 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.762858+0000", 2020-01-16T00:13:59.232 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014634199999999999 2020-01-16T00:13:59.232 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.232 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.232 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:59.232 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.763004+0000", 2020-01-16T00:13:59.233 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018472374 2020-01-16T00:13:59.233 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.233 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.233 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.233 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.233 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.233 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3297 3.2 3:42b7f20f:::smithi12313934-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e45)", 2020-01-16T00:13:59.233 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:30.744551+0000", 2020-01-16T00:13:59.234 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 28.291614617, 2020-01-16T00:13:59.234 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018660209000000001, 2020-01-16T00:13:59.234 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.234 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:59.234 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:59.234 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:59.234 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:59.234 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3297 2020-01-16T00:13:59.235 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.235 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.235 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.235 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.235 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744551+0000", 2020-01-16T00:13:59.235 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.235 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.236 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.236 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.236 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744551+0000", 2020-01-16T00:13:59.236 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:59.236 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.236 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.236 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.236 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744550+0000", 2020-01-16T00:13:59.237 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5279999999999999e-06 2020-01-16T00:13:59.237 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.237 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.237 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.237 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744554+0000", 2020-01-16T00:13:59.237 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8420000000000001e-06 2020-01-16T00:13:59.237 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.237 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.238 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.238 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744559+0000", 2020-01-16T00:13:59.238 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.965e-06 2020-01-16T00:13:59.238 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.238 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.238 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.238 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744562+0000", 2020-01-16T00:13:59.238 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041215380000000001 2020-01-16T00:13:59.239 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.239 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.239 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.239 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748684+0000", 2020-01-16T00:13:59.239 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8165999999999997e-05 2020-01-16T00:13:59.239 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.239 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.240 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:59.240 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748762+0000", 2020-01-16T00:13:59.240 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012136152000000001 2020-01-16T00:13:59.240 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.240 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.240 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.240 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.760898+0000", 2020-01-16T00:13:59.240 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000114833 2020-01-16T00:13:59.241 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.241 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.241 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:59.241 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.761013+0000", 2020-01-16T00:13:59.241 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002017015 2020-01-16T00:13:59.241 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.241 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.241 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.242 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.763030+0000", 2020-01-16T00:13:59.242 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5741000000000006e-05 2020-01-16T00:13:59.242 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.242 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.242 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.242 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.763096+0000", 2020-01-16T00:13:59.242 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011519900000000001 2020-01-16T00:13:59.242 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.243 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.243 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:59.243 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.763211+0000", 2020-01-16T00:13:59.243 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018660209000000001 2020-01-16T00:13:59.243 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.243 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.243 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.243 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.244 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.244 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3481 3.2 3:4f8fe446:::smithi12313934-48:head [setxattr _header (56) in=7b,truncate 3193396] snapc 0=[] ondisk+write+known_if_redirected e47)", 2020-01-16T00:13:59.244 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:36.349654+0000", 2020-01-16T00:13:59.244 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 22.686511163999999, 2020-01-16T00:13:59.244 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017529643000000001, 2020-01-16T00:13:59.244 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.244 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:59.245 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:59.245 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:59.245 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:59.245 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3481 2020-01-16T00:13:59.245 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.245 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.245 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.245 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.246 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349654+0000", 2020-01-16T00:13:59.246 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.246 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.246 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.246 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.246 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349654+0000", 2020-01-16T00:13:59.246 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:59.246 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.247 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.247 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.247 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349654+0000", 2020-01-16T00:13:59.247 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2519999999999996e-06 2020-01-16T00:13:59.247 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.247 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.247 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.247 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349660+0000", 2020-01-16T00:13:59.248 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2789999999999999e-06 2020-01-16T00:13:59.248 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.248 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.248 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.248 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349663+0000", 2020-01-16T00:13:59.248 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2479999999999999e-06 2020-01-16T00:13:59.248 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.248 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.249 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.249 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349666+0000", 2020-01-16T00:13:59.249 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0026359180000000001 2020-01-16T00:13:59.249 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.249 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.249 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.249 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.352301+0000", 2020-01-16T00:13:59.249 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0035532799999999998 2020-01-16T00:13:59.250 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.250 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.250 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:59.250 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.355855+0000", 2020-01-16T00:13:59.250 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0099904050000000008 2020-01-16T00:13:59.250 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.250 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.251 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.251 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.365845+0000", 2020-01-16T00:13:59.251 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012439300000000001 2020-01-16T00:13:59.251 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.251 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.251 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.251 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.365970+0000", 2020-01-16T00:13:59.251 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00026105699999999998 2020-01-16T00:13:59.252 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.252 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.252 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:59.252 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.366231+0000", 2020-01-16T00:13:59.252 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00056610199999999997 2020-01-16T00:13:59.252 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.252 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.252 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:59.253 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.366797+0000", 2020-01-16T00:13:59.253 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000171996 2020-01-16T00:13:59.253 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.253 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.253 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.253 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.366969+0000", 2020-01-16T00:13:59.253 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5282000000000003e-05 2020-01-16T00:13:59.254 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.254 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.254 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:59.254 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.367034+0000", 2020-01-16T00:13:59.254 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014998100000000001 2020-01-16T00:13:59.254 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.254 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.254 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:59.255 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.367184+0000", 2020-01-16T00:13:59.255 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017529643000000001 2020-01-16T00:13:59.255 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.255 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.255 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.255 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.255 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.255 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3482 3.2 3:4f8fe446:::smithi12313934-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e47)", 2020-01-16T00:13:59.256 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:36.349724+0000", 2020-01-16T00:13:59.256 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 22.686441526999999, 2020-01-16T00:13:59.256 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017836446999999998, 2020-01-16T00:13:59.256 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.256 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:59.256 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:59.256 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:59.256 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:59.257 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3482 2020-01-16T00:13:59.257 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.257 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.257 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.257 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.257 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349724+0000", 2020-01-16T00:13:59.257 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.258 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.258 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.258 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.258 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349724+0000", 2020-01-16T00:13:59.258 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:59.258 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.258 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.258 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.259 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349723+0000", 2020-01-16T00:13:59.259 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9709999999999998e-06 2020-01-16T00:13:59.259 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.259 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.259 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.259 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349727+0000", 2020-01-16T00:13:59.259 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5419999999999999e-06 2020-01-16T00:13:59.259 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.260 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.260 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.260 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349732+0000", 2020-01-16T00:13:59.260 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.481e-06 2020-01-16T00:13:59.260 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.260 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.260 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.260 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349734+0000", 2020-01-16T00:13:59.261 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.006195028 2020-01-16T00:13:59.261 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.261 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.261 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.261 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.355929+0000", 2020-01-16T00:13:59.261 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1483999999999996e-05 2020-01-16T00:13:59.261 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.262 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.262 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:59.262 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.355991+0000", 2020-01-16T00:13:59.262 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010536415 2020-01-16T00:13:59.262 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.262 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.262 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.262 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.366527+0000", 2020-01-16T00:13:59.263 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8099e-05 2020-01-16T00:13:59.263 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.263 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.263 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:59.263 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.366595+0000", 2020-01-16T00:13:59.263 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00063624500000000002 2020-01-16T00:13:59.263 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.263 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.264 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.264 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.367231+0000", 2020-01-16T00:13:59.264 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000121196 2020-01-16T00:13:59.264 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.264 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.264 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.264 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.367353+0000", 2020-01-16T00:13:59.265 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020780500000000001 2020-01-16T00:13:59.265 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.265 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.265 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:59.265 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.367560+0000", 2020-01-16T00:13:59.265 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017836446999999998 2020-01-16T00:13:59.265 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.265 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.266 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.266 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.266 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.266 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3810 3.7 3:efab040c:::smithi12313934-30:head [write 1837406~542493] snapc 0=[] ondisk+write+known_if_redirected e50)", 2020-01-16T00:13:59.266 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:43.768525+0000", 2020-01-16T00:13:59.266 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 15.267640444, 2020-01-16T00:13:59.266 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018385839000000001, 2020-01-16T00:13:59.266 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.267 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:59.267 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:59.267 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:59.267 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:59.267 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3810 2020-01-16T00:13:59.267 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.267 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.267 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.268 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.268 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768525+0000", 2020-01-16T00:13:59.268 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.268 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.268 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.268 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.268 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768525+0000", 2020-01-16T00:13:59.268 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:59.269 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.269 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.269 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.269 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768523+0000", 2020-01-16T00:13:59.269 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000149512 2020-01-16T00:13:59.269 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.269 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.270 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.270 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768673+0000", 2020-01-16T00:13:59.270 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0732e-05 2020-01-16T00:13:59.270 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.270 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.270 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.270 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768684+0000", 2020-01-16T00:13:59.270 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3719999999999998e-06 2020-01-16T00:13:59.271 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.271 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.271 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.271 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768688+0000", 2020-01-16T00:13:59.271 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6513999999999994e-05 2020-01-16T00:13:59.271 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.271 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.271 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.272 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768754+0000", 2020-01-16T00:13:59.272 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00075867800000000004 2020-01-16T00:13:59.272 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.272 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.272 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:59.272 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.769513+0000", 2020-01-16T00:13:59.273 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012395692 2020-01-16T00:13:59.273 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.273 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.273 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.273 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.781909+0000", 2020-01-16T00:13:59.273 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118372 2020-01-16T00:13:59.273 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.273 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.274 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.274 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.782027+0000", 2020-01-16T00:13:59.274 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019909699999999999 2020-01-16T00:13:59.274 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.274 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.274 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,7", 2020-01-16T00:13:59.274 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.782226+0000", 2020-01-16T00:13:59.274 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0038778369999999999 2020-01-16T00:13:59.275 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.275 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.275 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:59.275 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786104+0000", 2020-01-16T00:13:59.275 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022391199999999999 2020-01-16T00:13:59.275 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.275 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.275 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.276 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786328+0000", 2020-01-16T00:13:59.276 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00048851299999999999 2020-01-16T00:13:59.276 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.276 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.276 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.276 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786817+0000", 2020-01-16T00:13:59.276 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3597999999999998e-05 2020-01-16T00:13:59.277 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.277 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.277 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:59.277 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786860+0000", 2020-01-16T00:13:59.277 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0741000000000001e-05 2020-01-16T00:13:59.277 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.277 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.278 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:59.278 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786911+0000", 2020-01-16T00:13:59.278 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018385839000000001 2020-01-16T00:13:59.278 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.278 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.278 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.278 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.278 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.279 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3863 3.2 3:4f8fe446:::smithi12313934-48:head [setxattr _header (56) in=7b,truncate 2151395] snapc 0=[] ondisk+write+known_if_redirected e50)", 2020-01-16T00:13:59.279 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:44.489284+0000", 2020-01-16T00:13:59.279 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 14.546881972, 2020-01-16T00:13:59.279 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.026323742000000001, 2020-01-16T00:13:59.279 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.279 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:59.282 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:59.282 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:59.282 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:59.282 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3863 2020-01-16T00:13:59.282 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.282 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.282 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.283 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.283 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489284+0000", 2020-01-16T00:13:59.283 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.283 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.283 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.283 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.283 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489284+0000", 2020-01-16T00:13:59.284 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:59.284 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.284 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.284 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.284 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489282+0000", 2020-01-16T00:13:59.284 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6660000000000002e-06 2020-01-16T00:13:59.284 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.284 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.285 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.285 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489289+0000", 2020-01-16T00:13:59.285 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3069999999999999e-06 2020-01-16T00:13:59.285 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.285 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.285 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.285 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489295+0000", 2020-01-16T00:13:59.285 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6090000000000003e-06 2020-01-16T00:13:59.286 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.286 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.286 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.286 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489300+0000", 2020-01-16T00:13:59.286 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002242464 2020-01-16T00:13:59.286 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.286 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.286 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.287 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491542+0000", 2020-01-16T00:13:59.287 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00059546200000000005 2020-01-16T00:13:59.287 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.287 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.287 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:59.287 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.492138+0000", 2020-01-16T00:13:59.287 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018547999999999999 2020-01-16T00:13:59.288 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.288 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.288 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.288 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.510686+0000", 2020-01-16T00:13:59.288 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018590100000000001 2020-01-16T00:13:59.288 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.288 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.288 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.289 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.510872+0000", 2020-01-16T00:13:59.289 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035115299999999999 2020-01-16T00:13:59.289 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.289 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.289 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:59.289 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.511223+0000", 2020-01-16T00:13:59.289 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003765493 2020-01-16T00:13:59.289 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.290 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.290 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.290 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.514988+0000", 2020-01-16T00:13:59.290 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000432022 2020-01-16T00:13:59.290 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.290 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.290 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:59.290 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515421+0000", 2020-01-16T00:13:59.291 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2874000000000002e-05 2020-01-16T00:13:59.291 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.291 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.291 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:59.291 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515473+0000", 2020-01-16T00:13:59.291 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013385 2020-01-16T00:13:59.291 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.291 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.292 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:59.292 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515607+0000", 2020-01-16T00:13:59.292 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.026323742000000001 2020-01-16T00:13:59.292 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.292 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.292 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.292 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.293 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.293 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3864 3.2 3:4f8fe446:::smithi12313934-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e50)", 2020-01-16T00:13:59.293 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:44.491131+0000", 2020-01-16T00:13:59.293 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 14.545034574000001, 2020-01-16T00:13:59.293 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024813806000000001, 2020-01-16T00:13:59.293 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.293 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:13:59.293 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:59.294 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:59.294 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:59.294 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3864 2020-01-16T00:13:59.294 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.294 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.294 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.294 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.294 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491131+0000", 2020-01-16T00:13:59.295 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.295 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.295 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.295 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.295 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491131+0000", 2020-01-16T00:13:59.295 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:59.295 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.295 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.296 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.296 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491128+0000", 2020-01-16T00:13:59.296 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2053e-05 2020-01-16T00:13:59.296 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.296 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.296 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.296 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491140+0000", 2020-01-16T00:13:59.296 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5869999999999999e-05 2020-01-16T00:13:59.297 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.297 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.297 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.297 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491156+0000", 2020-01-16T00:13:59.297 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2571e-05 2020-01-16T00:13:59.297 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.297 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.298 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.298 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491168+0000", 2020-01-16T00:13:59.298 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001031045 2020-01-16T00:13:59.298 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.298 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.298 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.298 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.492199+0000", 2020-01-16T00:13:59.298 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000170865 2020-01-16T00:13:59.299 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.299 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.299 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:13:59.299 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.492370+0000", 2020-01-16T00:13:59.299 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020180638000000001 2020-01-16T00:13:59.299 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.299 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.299 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.300 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.512551+0000", 2020-01-16T00:13:59.300 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012478599999999999 2020-01-16T00:13:59.300 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.300 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.300 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:13:59.300 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.512676+0000", 2020-01-16T00:13:59.300 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029628660000000002 2020-01-16T00:13:59.300 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.301 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.301 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.301 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515639+0000", 2020-01-16T00:13:59.301 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012250799999999999 2020-01-16T00:13:59.301 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.301 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.301 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.301 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515761+0000", 2020-01-16T00:13:59.302 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018356399999999999 2020-01-16T00:13:59.302 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.302 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.302 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:59.302 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515945+0000", 2020-01-16T00:13:59.302 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024813806000000001 2020-01-16T00:13:59.302 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.302 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.303 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.303 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.303 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.303 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3874 3.c 3:378f3722:::smithi12313934-50:head [write 568202~701253,stat] snapc 0=[] ondisk+write+known_if_redirected e50)", 2020-01-16T00:13:59.303 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:44.497041+0000", 2020-01-16T00:13:59.303 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 14.539124942999999, 2020-01-16T00:13:59.303 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019602525999999999, 2020-01-16T00:13:59.304 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.304 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:59.304 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:13:59.304 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:13:59.304 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:13:59.304 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3874 2020-01-16T00:13:59.304 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.304 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.305 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.305 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.305 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497041+0000", 2020-01-16T00:13:59.305 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.305 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.305 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.305 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.305 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497041+0000", 2020-01-16T00:13:59.306 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999962 2020-01-16T00:13:59.306 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.306 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.306 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.306 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497037+0000", 2020-01-16T00:13:59.306 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00027168899999999998 2020-01-16T00:13:59.306 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.306 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.307 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.307 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497308+0000", 2020-01-16T00:13:59.307 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4202999999999999e-05 2020-01-16T00:13:59.307 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.307 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.307 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.307 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497323+0000", 2020-01-16T00:13:59.307 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0280000000000003e-06 2020-01-16T00:13:59.308 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.308 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.308 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.308 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497330+0000", 2020-01-16T00:13:59.308 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010681352 2020-01-16T00:13:59.308 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.308 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.309 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.309 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.508011+0000", 2020-01-16T00:13:59.309 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5886999999999999e-05 2020-01-16T00:13:59.309 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.309 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.309 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.309 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.508097+0000", 2020-01-16T00:13:59.309 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023353600000000001 2020-01-16T00:13:59.310 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.310 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.310 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:13:59.310 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.508331+0000", 2020-01-16T00:13:59.310 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0075218339999999998 2020-01-16T00:13:59.310 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.310 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.310 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:59.311 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515852+0000", 2020-01-16T00:13:59.311 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00064178599999999998 2020-01-16T00:13:59.311 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.311 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.311 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.311 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.516494+0000", 2020-01-16T00:13:59.311 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6712999999999999e-05 2020-01-16T00:13:59.311 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.312 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.312 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:59.312 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.516531+0000", 2020-01-16T00:13:59.312 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112225 2020-01-16T00:13:59.312 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.312 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.312 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:59.312 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.516643+0000", 2020-01-16T00:13:59.313 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019602525999999999 2020-01-16T00:13:59.313 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.313 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.313 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.313 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.313 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.313 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:59.314 INFO:tasks.rados.rados.0.smithi123.stdout:1607: writing smithi12313934-27 from 645534 to 1429241 tid 1 2020-01-16T00:13:59.316 INFO:tasks.rados.rados.0.smithi123.stdout:1607: writing smithi12313934-27 from 2103939 to 2742456 tid 2 2020-01-16T00:13:59.316 INFO:tasks.rados.rados.0.smithi123.stdout:1607: writing smithi12313934-27 from 3150663 to 3445534 tid 3 2020-01-16T00:13:59.316 INFO:tasks.rados.rados.0.smithi123.stdout:1608: read oid 39 snap -1 2020-01-16T00:13:59.316 INFO:tasks.rados.rados.0.smithi123.stdout:1608: expect (ObjNum 535 snap 0 seq_num 535) 2020-01-16T00:13:59.316 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:59.317 INFO:tasks.rados.rados.0.smithi123.stdout:1593: finishing write tid 5 to smithi12313934-19 2020-01-16T00:13:59.317 INFO:tasks.rados.rados.0.smithi123.stdout:1593: finishing write tid 6 to smithi12313934-19 2020-01-16T00:13:59.317 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 19 v 225 (ObjNum 554 snap 0 seq_num 554) dirty exists 2020-01-16T00:13:59.317 INFO:tasks.rados.rados.0.smithi123.stdout:1593: left oid 19 (ObjNum 554 snap 0 seq_num 554) 2020-01-16T00:13:59.317 INFO:tasks.rados.rados.0.smithi123.stdout:1594: expect (ObjNum 534 snap 0 seq_num 534) 2020-01-16T00:13:59.317 INFO:tasks.rados.rados.0.smithi123.stdout:1595: expect (ObjNum 538 snap 0 seq_num 538) 2020-01-16T00:13:59.318 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-01-16T00:13:59.318 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:59.310+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:59.336 INFO:tasks.rados.rados.0.smithi123.stdout:1596: expect (ObjNum 513 snap 0 seq_num 513) 2020-01-16T00:13:59.416 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:13:59.414+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:13:59.442 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:59.442 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:59.442 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:13:59.443 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:59.451 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-01-16T00:13:59.518 INFO:tasks.rados.rados.0.smithi123.stdout:1597: finishing write tid 1 to smithi12313934-11 2020-01-16T00:13:59.518 INFO:tasks.rados.rados.0.smithi123.stdout:1597: finishing write tid 2 to smithi12313934-11 2020-01-16T00:13:59.518 INFO:tasks.rados.rados.0.smithi123.stdout:1597: finishing write tid 3 to smithi12313934-11 2020-01-16T00:13:59.518 INFO:tasks.rados.rados.0.smithi123.stdout:1597: finishing write tid 5 to smithi12313934-11 2020-01-16T00:13:59.518 INFO:tasks.rados.rados.0.smithi123.stdout:1597: finishing write tid 6 to smithi12313934-11 2020-01-16T00:13:59.519 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 11 v 160 (ObjNum 555 snap 0 seq_num 555) dirty exists 2020-01-16T00:13:59.519 INFO:tasks.rados.rados.0.smithi123.stdout:1597: left oid 11 (ObjNum 555 snap 0 seq_num 555) 2020-01-16T00:13:59.519 INFO:tasks.rados.rados.0.smithi123.stdout:1598: finishing write tid 1 to smithi12313934-26 2020-01-16T00:13:59.519 INFO:tasks.rados.rados.0.smithi123.stdout:1598: finishing write tid 2 to smithi12313934-26 2020-01-16T00:13:59.519 INFO:tasks.rados.rados.0.smithi123.stdout:1599: finishing write tid 1 to smithi12313934-4 2020-01-16T00:13:59.519 INFO:tasks.rados.rados.0.smithi123.stdout:1593: done (15 left) 2020-01-16T00:13:59.519 INFO:tasks.rados.rados.0.smithi123.stdout:1594: done (14 left) 2020-01-16T00:13:59.519 INFO:tasks.rados.rados.0.smithi123.stdout:1595: done (13 left) 2020-01-16T00:13:59.520 INFO:tasks.rados.rados.0.smithi123.stdout:1596: done (12 left) 2020-01-16T00:13:59.520 INFO:tasks.rados.rados.0.smithi123.stdout:1597: done (11 left) 2020-01-16T00:13:59.520 INFO:tasks.rados.rados.0.smithi123.stdout:1609: read oid 34 snap -1 2020-01-16T00:13:59.520 INFO:tasks.rados.rados.0.smithi123.stdout:1609: expect (ObjNum 532 snap 0 seq_num 532) 2020-01-16T00:13:59.520 INFO:tasks.rados.rados.0.smithi123.stdout:1610: write (excl) oid 23 current snap is 0 2020-01-16T00:13:59.520 INFO:tasks.rados.rados.0.smithi123.stdout:1610: seq_num 563 ranges {662341=575309,1878774=498528,2894439=457555} 2020-01-16T00:13:59.520 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:13:59.514+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:13:59.529 INFO:tasks.rados.rados.0.smithi123.stdout:1610: writing smithi12313934-23 from 662341 to 1237650 tid 1 2020-01-16T00:13:59.536 INFO:tasks.rados.rados.0.smithi123.stdout:1610: writing smithi12313934-23 from 1878774 to 2377302 tid 2 2020-01-16T00:13:59.542 INFO:tasks.rados.rados.0.smithi123.stdout:1610: writing smithi12313934-23 from 2894439 to 3351994 tid 3 2020-01-16T00:13:59.542 INFO:tasks.rados.rados.0.smithi123.stdout:1611: write (excl) oid 49 current snap is 0 2020-01-16T00:13:59.542 INFO:tasks.rados.rados.0.smithi123.stdout:1611: seq_num 564 ranges {679148=650558,1937256=642186,3321862=157286} 2020-01-16T00:13:59.550 INFO:tasks.rados.rados.0.smithi123.stdout:1611: writing smithi12313934-49 from 679148 to 1329706 tid 1 2020-01-16T00:13:59.557 INFO:tasks.rados.rados.0.smithi123.stdout:1611: writing smithi12313934-49 from 1937256 to 2579442 tid 2 2020-01-16T00:13:59.563 INFO:tasks.rados.rados.0.smithi123.stdout:1611: writing smithi12313934-49 from 3321862 to 3479148 tid 3 2020-01-16T00:13:59.563 INFO:tasks.rados.rados.0.smithi123.stdout:1612: write (excl) oid 8 current snap is 0 2020-01-16T00:13:59.563 INFO:tasks.rados.rados.0.smithi123.stdout:1612: seq_num 565 ranges {695955=725807,1879385=502197,2949285=480099} 2020-01-16T00:13:59.570 INFO:tasks.rados.rados.0.smithi123.stdout:1612: writing smithi12313934-8 from 695955 to 1421762 tid 1 2020-01-16T00:13:59.580 INFO:tasks.rados.rados.0.smithi123.stdout:1612: writing smithi12313934-8 from 1879385 to 2381582 tid 2 2020-01-16T00:13:59.587 INFO:tasks.rados.rados.0.smithi123.stdout:1612: writing smithi12313934-8 from 2949285 to 3429384 tid 3 2020-01-16T00:13:59.587 INFO:tasks.rados.rados.0.smithi123.stdout:1613: write oid 18 current snap is 0 2020-01-16T00:13:59.587 INFO:tasks.rados.rados.0.smithi123.stdout:1613: seq_num 566 ranges {712762=401056,1537867=645855,2860355=652407} 2020-01-16T00:13:59.588 INFO:tasks.rados.rados.0.smithi123.stdout:1613: writing smithi12313934-18 from 712762 to 1113818 tid 1 2020-01-16T00:13:59.595 INFO:tasks.rados.rados.0.smithi123.stdout:1613: writing smithi12313934-18 from 1537867 to 2183722 tid 2 2020-01-16T00:13:59.603 INFO:tasks.rados.rados.0.smithi123.stdout:1613: writing smithi12313934-18 from 2860355 to 3512762 tid 3 2020-01-16T00:13:59.603 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:59.604 INFO:tasks.rados.rados.0.smithi123.stdout:1598: finishing write tid 3 to smithi12313934-26 2020-01-16T00:13:59.604 INFO:tasks.rados.rados.0.smithi123.stdout:1598: finishing write tid 5 to smithi12313934-26 2020-01-16T00:13:59.604 INFO:tasks.rados.rados.0.smithi123.stdout:1598: finishing write tid 6 to smithi12313934-26 2020-01-16T00:13:59.604 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 26 v 297 (ObjNum 556 snap 0 seq_num 556) dirty exists 2020-01-16T00:13:59.604 INFO:tasks.rados.rados.0.smithi123.stdout:1598: left oid 26 (ObjNum 556 snap 0 seq_num 556) 2020-01-16T00:13:59.604 INFO:tasks.rados.rados.0.smithi123.stdout:1599: finishing write tid 2 to smithi12313934-4 2020-01-16T00:13:59.604 INFO:tasks.rados.rados.0.smithi123.stdout:1599: finishing write tid 3 to smithi12313934-4 2020-01-16T00:13:59.605 INFO:tasks.rados.rados.0.smithi123.stdout:1598: done (15 left) 2020-01-16T00:13:59.605 INFO:tasks.rados.rados.0.smithi123.stdout:1614: read oid 15 snap -1 2020-01-16T00:13:59.605 INFO:tasks.rados.rados.0.smithi123.stdout:1614: expect (ObjNum 543 snap 0 seq_num 543) 2020-01-16T00:13:59.605 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:13:59.606 INFO:tasks.rados.rados.0.smithi123.stdout:1599: finishing write tid 5 to smithi12313934-4 2020-01-16T00:13:59.606 INFO:tasks.rados.rados.0.smithi123.stdout:1599: finishing write tid 6 to smithi12313934-4 2020-01-16T00:13:59.606 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 4 v 174 (ObjNum 557 snap 0 seq_num 557) dirty exists 2020-01-16T00:13:59.606 INFO:tasks.rados.rados.0.smithi123.stdout:1599: left oid 4 (ObjNum 557 snap 0 seq_num 557) 2020-01-16T00:13:59.606 INFO:tasks.rados.rados.0.smithi123.stdout:1600: finishing write tid 1 to smithi12313934-25 2020-01-16T00:13:59.607 INFO:tasks.rados.rados.0.smithi123.stdout:1599: done (15 left) 2020-01-16T00:13:59.607 INFO:tasks.rados.rados.0.smithi123.stdout:1615: delete oid 40 current snap is 0 2020-01-16T00:13:59.607 INFO:tasks.rados.rados.0.smithi123.stdout:1600: finishing write tid 2 to smithi12313934-25 2020-01-16T00:13:59.608 INFO:tasks.rados.rados.0.smithi123.stdout:1600: finishing write tid 3 to smithi12313934-25 2020-01-16T00:13:59.608 INFO:tasks.rados.rados.0.smithi123.stdout:1600: finishing write tid 5 to smithi12313934-25 2020-01-16T00:13:59.608 INFO:tasks.rados.rados.0.smithi123.stdout:1600: finishing write tid 6 to smithi12313934-25 2020-01-16T00:13:59.608 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 25 v 245 (ObjNum 558 snap 0 seq_num 558) dirty exists 2020-01-16T00:13:59.608 INFO:tasks.rados.rados.0.smithi123.stdout:1600: left oid 25 (ObjNum 558 snap 0 seq_num 558) 2020-01-16T00:13:59.608 INFO:tasks.rados.rados.0.smithi123.stdout:1601: finishing write tid 1 to smithi12313934-45 2020-01-16T00:13:59.608 INFO:tasks.rados.rados.0.smithi123.stdout:1601: finishing write tid 2 to smithi12313934-45 2020-01-16T00:13:59.609 INFO:tasks.rados.rados.0.smithi123.stdout:1602: expect (ObjNum 507 snap 0 seq_num 507) 2020-01-16T00:13:59.609 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:59.609 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:13:59.609 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:13:59.609 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:13:59.610 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:13:59.618 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:13:59.614+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:13:59.619 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-01-16T00:13:59.661 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-01-16T00:13:59.719 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:13:59.716+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:13:59.781 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:13:59.781 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:13:59.782 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:13:59.782 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:13:59.782 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.782 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.4.0:114 3.9 e24/20 3:9fc687b2:::smithi12313934-44:head v 24'74, mlcod=24'72)", 2020-01-16T00:13:59.783 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:13:59.783 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 81.453777118000005, 2020-01-16T00:13:59.783 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010399429, 2020-01-16T00:13:59.783 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.783 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:59.784 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.784 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.784 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.784 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:13:59.785 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.785 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.785 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.786 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.786 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:13:59.786 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:59.786 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.787 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.787 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.787 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325276+0000", 2020-01-16T00:13:59.787 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0046127069999999997 2020-01-16T00:13:59.787 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.788 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.788 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.788 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329889+0000", 2020-01-16T00:13:59.788 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3519999999999998e-05 2020-01-16T00:13:59.788 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.788 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.788 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.789 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329912+0000", 2020-01-16T00:13:59.789 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4070000000000004e-06 2020-01-16T00:13:59.789 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.789 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.789 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.789 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329920+0000", 2020-01-16T00:13:59.789 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8817000000000003e-05 2020-01-16T00:13:59.790 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.790 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.790 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.790 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329978+0000", 2020-01-16T00:13:59.790 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9721e-05 2020-01-16T00:13:59.790 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.790 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.791 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.791 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329998+0000", 2020-01-16T00:13:59.791 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056328139999999999 2020-01-16T00:13:59.791 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.791 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.791 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:59.791 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.335631+0000", 2020-01-16T00:13:59.791 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7170999999999999e-05 2020-01-16T00:13:59.792 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.792 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.792 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:59.792 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.335678+0000", 2020-01-16T00:13:59.792 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010399429 2020-01-16T00:13:59.792 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.792 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.793 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.793 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.793 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:24 2.1 2:8b431f80:::smithi12313934-40 low_tier:head [write 790528~688128] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:13:59.794 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.482303+0000", 2020-01-16T00:13:59.794 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 81.296753022000004, 2020-01-16T00:13:59.794 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010071853, 2020-01-16T00:13:59.794 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.794 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:59.794 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.794 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.794 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.795 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.482303+0000", 2020-01-16T00:13:59.795 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.795 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.795 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.795 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.795 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.482303+0000", 2020-01-16T00:13:59.795 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:59.795 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.796 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.796 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.796 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.482301+0000", 2020-01-16T00:13:59.796 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031173960000000001 2020-01-16T00:13:59.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.796 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.796 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.797 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485418+0000", 2020-01-16T00:13:59.797 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9300000000000001e-05 2020-01-16T00:13:59.797 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.797 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.797 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.797 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485448+0000", 2020-01-16T00:13:59.797 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2611e-05 2020-01-16T00:13:59.797 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.798 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.798 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.798 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485460+0000", 2020-01-16T00:13:59.798 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112487 2020-01-16T00:13:59.798 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.798 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.798 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.798 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485573+0000", 2020-01-16T00:13:59.799 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022169299999999999 2020-01-16T00:13:59.799 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.799 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.799 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.799 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485794+0000", 2020-01-16T00:13:59.799 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00036287500000000002 2020-01-16T00:13:59.799 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.800 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.800 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:59.800 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.486157+0000", 2020-01-16T00:13:59.800 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040260310000000002 2020-01-16T00:13:59.800 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.800 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.801 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:59.801 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.490183+0000", 2020-01-16T00:13:59.801 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00081083000000000001 2020-01-16T00:13:59.801 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.801 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.801 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.801 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.490994+0000", 2020-01-16T00:13:59.802 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001287746 2020-01-16T00:13:59.802 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.802 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.802 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.802 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492282+0000", 2020-01-16T00:13:59.802 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8810999999999998e-05 2020-01-16T00:13:59.802 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.803 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.803 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:59.803 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492321+0000", 2020-01-16T00:13:59.803 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4095000000000003e-05 2020-01-16T00:13:59.803 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.803 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.803 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:59.804 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492375+0000", 2020-01-16T00:13:59.804 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010071853 2020-01-16T00:13:59.804 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.804 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.804 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.804 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.804 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.805 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:55 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e30)", 2020-01-16T00:13:59.805 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.736036+0000", 2020-01-16T00:13:59.805 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 67.043019899000001, 2020-01-16T00:13:59.805 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010974536, 2020-01-16T00:13:59.805 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.805 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:59.805 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.805 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.806 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.806 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736036+0000", 2020-01-16T00:13:59.806 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.806 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.806 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.806 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.806 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736036+0000", 2020-01-16T00:13:59.806 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:59.807 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.807 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.807 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.807 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736034+0000", 2020-01-16T00:13:59.807 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043589699999999998 2020-01-16T00:13:59.807 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.807 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.808 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.808 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736469+0000", 2020-01-16T00:13:59.808 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9882e-05 2020-01-16T00:13:59.808 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.808 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.808 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.808 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736489+0000", 2020-01-16T00:13:59.808 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8339999999999999e-06 2020-01-16T00:13:59.809 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.809 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.809 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.809 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736497+0000", 2020-01-16T00:13:59.809 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5354999999999999e-05 2020-01-16T00:13:59.809 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.809 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.809 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.810 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736572+0000", 2020-01-16T00:13:59.810 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5228000000000004e-05 2020-01-16T00:13:59.810 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.810 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.810 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.810 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736668+0000", 2020-01-16T00:13:59.810 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017212999999999999 2020-01-16T00:13:59.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.811 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.811 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:59.811 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736840+0000", 2020-01-16T00:13:59.811 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0099057020000000006 2020-01-16T00:13:59.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.811 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.811 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.812 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746746+0000", 2020-01-16T00:13:59.812 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0586000000000005e-05 2020-01-16T00:13:59.812 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.812 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.812 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:59.812 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746816+0000", 2020-01-16T00:13:59.812 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2445000000000002e-05 2020-01-16T00:13:59.812 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.813 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.813 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.813 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746889+0000", 2020-01-16T00:13:59.813 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6839000000000002e-05 2020-01-16T00:13:59.813 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.813 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.814 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:59.814 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746955+0000", 2020-01-16T00:13:59.814 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5074000000000002e-05 2020-01-16T00:13:59.814 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.815 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.815 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:59.815 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.747011+0000", 2020-01-16T00:13:59.815 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010974536 2020-01-16T00:13:59.815 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.815 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.815 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.816 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.816 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.816 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:56 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e30)", 2020-01-16T00:13:59.816 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.736514+0000", 2020-01-16T00:13:59.816 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 67.042542064000003, 2020-01-16T00:13:59.816 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011660584999999999, 2020-01-16T00:13:59.816 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.817 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:59.817 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.817 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.817 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.817 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736514+0000", 2020-01-16T00:13:59.817 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.818 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.818 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.818 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.818 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736514+0000", 2020-01-16T00:13:59.818 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:59.819 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.819 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.819 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.819 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736513+0000", 2020-01-16T00:13:59.819 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9677000000000004e-05 2020-01-16T00:13:59.819 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.819 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.820 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.820 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736583+0000", 2020-01-16T00:13:59.820 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5269999999999998e-06 2020-01-16T00:13:59.820 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.820 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.820 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.820 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736591+0000", 2020-01-16T00:13:59.821 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5769999999999996e-06 2020-01-16T00:13:59.821 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.821 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.821 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.821 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736599+0000", 2020-01-16T00:13:59.821 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0053593310000000002 2020-01-16T00:13:59.821 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.822 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.822 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.822 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.741958+0000", 2020-01-16T00:13:59.822 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014164099999999999 2020-01-16T00:13:59.822 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.822 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.822 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.822 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.742100+0000", 2020-01-16T00:13:59.823 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020126000000000001 2020-01-16T00:13:59.823 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.823 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.823 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:59.823 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.742301+0000", 2020-01-16T00:13:59.823 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0045052859999999998 2020-01-16T00:13:59.823 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.823 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.824 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.824 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746806+0000", 2020-01-16T00:13:59.824 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00070802699999999996 2020-01-16T00:13:59.824 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.824 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.824 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:59.825 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.747514+0000", 2020-01-16T00:13:59.825 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00059247200000000003 2020-01-16T00:13:59.825 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.825 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.825 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.825 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748107+0000", 2020-01-16T00:13:59.825 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1870999999999999e-05 2020-01-16T00:13:59.826 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.826 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.826 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:59.826 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748149+0000", 2020-01-16T00:13:59.826 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5831000000000001e-05 2020-01-16T00:13:59.826 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.826 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.827 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:59.827 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748174+0000", 2020-01-16T00:13:59.827 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011660584999999999 2020-01-16T00:13:59.827 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.827 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.827 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.828 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.828 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.828 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:70 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e35)", 2020-01-16T00:13:59.828 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.985494+0000", 2020-01-16T00:13:59.828 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 53.793562268000002, 2020-01-16T00:13:59.828 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013746267, 2020-01-16T00:13:59.828 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.829 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:59.829 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.829 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.829 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.829 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985494+0000", 2020-01-16T00:13:59.829 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.829 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.830 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.830 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.830 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985494+0000", 2020-01-16T00:13:59.830 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2020-01-16T00:13:59.831 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.831 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.831 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.831 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985490+0000", 2020-01-16T00:13:59.831 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00077792099999999999 2020-01-16T00:13:59.831 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.831 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.832 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.832 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986268+0000", 2020-01-16T00:13:59.832 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9718000000000001e-05 2020-01-16T00:13:59.832 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.832 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.833 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.833 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986318+0000", 2020-01-16T00:13:59.833 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.4349999999999997e-06 2020-01-16T00:13:59.833 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.833 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.833 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.833 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986326+0000", 2020-01-16T00:13:59.833 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3298999999999999e-05 2020-01-16T00:13:59.834 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.834 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.834 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.834 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986380+0000", 2020-01-16T00:13:59.834 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4969999999999995e-05 2020-01-16T00:13:59.834 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.834 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.835 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.835 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986455+0000", 2020-01-16T00:13:59.835 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000110587 2020-01-16T00:13:59.835 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.835 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.835 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:59.836 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986565+0000", 2020-01-16T00:13:59.836 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063931600000000002 2020-01-16T00:13:59.836 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.836 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.836 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:59.836 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.992958+0000", 2020-01-16T00:13:59.836 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013008480000000001 2020-01-16T00:13:59.837 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.837 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.837 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.837 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.994259+0000", 2020-01-16T00:13:59.837 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048399979999999999 2020-01-16T00:13:59.837 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.837 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.838 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.838 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999099+0000", 2020-01-16T00:13:59.838 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1874999999999998e-05 2020-01-16T00:13:59.838 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.838 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.838 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:59.838 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999181+0000", 2020-01-16T00:13:59.838 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8865e-05 2020-01-16T00:13:59.839 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.839 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.839 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:59.839 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999240+0000", 2020-01-16T00:13:59.839 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013746267 2020-01-16T00:13:59.839 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.839 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.840 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.840 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.840 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.841 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:71 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e35)", 2020-01-16T00:13:59.841 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.986341+0000", 2020-01-16T00:13:59.842 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 53.792714906999997, 2020-01-16T00:13:59.842 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013252807, 2020-01-16T00:13:59.842 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.842 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:59.842 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.842 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.842 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.843 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986341+0000", 2020-01-16T00:13:59.843 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.843 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.843 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.843 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.843 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986341+0000", 2020-01-16T00:13:59.843 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:59.843 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.844 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.844 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.844 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986340+0000", 2020-01-16T00:13:59.844 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000103867 2020-01-16T00:13:59.844 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.845 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.845 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.845 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986444+0000", 2020-01-16T00:13:59.845 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0104e-05 2020-01-16T00:13:59.846 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.846 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.846 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.846 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986454+0000", 2020-01-16T00:13:59.846 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4220000000000002e-06 2020-01-16T00:13:59.846 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.846 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.847 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.847 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986458+0000", 2020-01-16T00:13:59.847 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033523070000000001 2020-01-16T00:13:59.847 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.847 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.848 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.848 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.989811+0000", 2020-01-16T00:13:59.848 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3296999999999999e-05 2020-01-16T00:13:59.848 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.848 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.848 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.848 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.989894+0000", 2020-01-16T00:13:59.848 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000117701 2020-01-16T00:13:59.849 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.849 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.849 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:59.849 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.990012+0000", 2020-01-16T00:13:59.849 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0032873490000000002 2020-01-16T00:13:59.849 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.849 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.850 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:59.850 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993299+0000", 2020-01-16T00:13:59.850 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002422619 2020-01-16T00:13:59.850 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.850 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.850 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.851 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.995722+0000", 2020-01-16T00:13:59.851 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0037839589999999999 2020-01-16T00:13:59.851 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.851 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.851 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.851 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999506+0000", 2020-01-16T00:13:59.851 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8559999999999997e-05 2020-01-16T00:13:59.851 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.852 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.852 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:59.852 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999544+0000", 2020-01-16T00:13:59.852 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9599999999999999e-05 2020-01-16T00:13:59.853 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.853 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.854 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:59.854 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999594+0000", 2020-01-16T00:13:59.854 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013252807 2020-01-16T00:13:59.854 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.855 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.855 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.855 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.855 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.855 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:77 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e36)", 2020-01-16T00:13:59.856 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.555854+0000", 2020-01-16T00:13:59.856 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 52.223201818, 2020-01-16T00:13:59.856 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010455002999999999, 2020-01-16T00:13:59.856 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.856 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:59.856 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.857 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.857 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.857 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555854+0000", 2020-01-16T00:13:59.857 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.857 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.857 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.857 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.858 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555854+0000", 2020-01-16T00:13:59.858 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:13:59.858 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.858 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.858 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.858 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555851+0000", 2020-01-16T00:13:59.859 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00048310200000000001 2020-01-16T00:13:59.859 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.859 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.859 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.859 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556334+0000", 2020-01-16T00:13:59.859 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2232e-05 2020-01-16T00:13:59.859 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.859 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.860 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.860 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556356+0000", 2020-01-16T00:13:59.860 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9750000000000007e-06 2020-01-16T00:13:59.860 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.860 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.860 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.860 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556364+0000", 2020-01-16T00:13:59.860 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8872999999999997e-05 2020-01-16T00:13:59.861 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.861 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.861 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.862 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556423+0000", 2020-01-16T00:13:59.862 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8687000000000001e-05 2020-01-16T00:13:59.862 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.862 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.862 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.862 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556502+0000", 2020-01-16T00:13:59.863 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013235300000000001 2020-01-16T00:13:59.863 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.863 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.863 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:59.863 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556634+0000", 2020-01-16T00:13:59.863 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0059770520000000001 2020-01-16T00:13:59.863 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.863 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.864 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:59.864 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.562611+0000", 2020-01-16T00:13:59.864 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029492820000000001 2020-01-16T00:13:59.864 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.864 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.864 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.864 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.565561+0000", 2020-01-16T00:13:59.864 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00060620300000000004 2020-01-16T00:13:59.865 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.865 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.865 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.865 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566167+0000", 2020-01-16T00:13:59.865 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3831000000000004e-05 2020-01-16T00:13:59.865 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.865 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.866 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:59.866 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566251+0000", 2020-01-16T00:13:59.866 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8328000000000001e-05 2020-01-16T00:13:59.866 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.866 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.866 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:59.866 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566309+0000", 2020-01-16T00:13:59.866 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010455002999999999 2020-01-16T00:13:59.867 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.867 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.867 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.867 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.867 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.867 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:78 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e36)", 2020-01-16T00:13:59.867 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.556378+0000", 2020-01-16T00:13:59.867 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 52.222677885000003, 2020-01-16T00:13:59.868 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010396311, 2020-01-16T00:13:59.868 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.868 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:59.868 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.868 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.868 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.868 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556378+0000", 2020-01-16T00:13:59.868 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.869 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.869 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.869 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.869 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556378+0000", 2020-01-16T00:13:59.869 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:59.869 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.869 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.870 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.870 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556377+0000", 2020-01-16T00:13:59.870 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4307000000000001e-05 2020-01-16T00:13:59.870 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.870 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.870 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.870 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556422+0000", 2020-01-16T00:13:59.870 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9620000000000005e-06 2020-01-16T00:13:59.871 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.871 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.871 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.871 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556432+0000", 2020-01-16T00:13:59.871 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5979999999999999e-06 2020-01-16T00:13:59.871 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.871 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.871 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.872 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556436+0000", 2020-01-16T00:13:59.872 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021059469999999999 2020-01-16T00:13:59.872 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.872 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.872 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.872 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.558542+0000", 2020-01-16T00:13:59.872 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000194458 2020-01-16T00:13:59.872 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.873 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.873 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.873 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.558737+0000", 2020-01-16T00:13:59.873 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00032867900000000002 2020-01-16T00:13:59.874 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.874 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.874 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:59.874 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.559065+0000", 2020-01-16T00:13:59.875 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052547499999999999 2020-01-16T00:13:59.875 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.875 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.875 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:59.875 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.564320+0000", 2020-01-16T00:13:59.875 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001565213 2020-01-16T00:13:59.876 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.876 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.876 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.876 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.565885+0000", 2020-01-16T00:13:59.877 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00076994300000000004 2020-01-16T00:13:59.877 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.877 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.877 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.877 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566655+0000", 2020-01-16T00:13:59.878 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7533000000000005e-05 2020-01-16T00:13:59.878 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.878 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.878 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:59.878 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566723+0000", 2020-01-16T00:13:59.878 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1616999999999997e-05 2020-01-16T00:13:59.878 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.879 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.879 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:59.879 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566774+0000", 2020-01-16T00:13:59.879 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010396311 2020-01-16T00:13:59.879 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.879 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.879 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.879 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.880 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.880 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:72 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 56832~1408000] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e37)", 2020-01-16T00:13:59.880 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.578753+0000", 2020-01-16T00:13:59.880 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 47.200302581999999, 2020-01-16T00:13:59.880 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011826918, 2020-01-16T00:13:59.880 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.880 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:59.881 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.881 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.881 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.881 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578753+0000", 2020-01-16T00:13:59.881 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.881 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.881 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.881 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.882 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578753+0000", 2020-01-16T00:13:59.882 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:59.882 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.882 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.882 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.882 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578751+0000", 2020-01-16T00:13:59.882 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00042457699999999998 2020-01-16T00:13:59.882 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.883 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.883 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.883 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579175+0000", 2020-01-16T00:13:59.883 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5404999999999999e-05 2020-01-16T00:13:59.883 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.883 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.883 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.883 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579191+0000", 2020-01-16T00:13:59.884 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6149999999999996e-06 2020-01-16T00:13:59.884 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.884 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.884 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.884 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579196+0000", 2020-01-16T00:13:59.884 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000104794 2020-01-16T00:13:59.884 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.885 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.885 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.885 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579301+0000", 2020-01-16T00:13:59.885 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020947199999999999 2020-01-16T00:13:59.885 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.886 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.886 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.886 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579511+0000", 2020-01-16T00:13:59.886 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034328400000000002 2020-01-16T00:13:59.886 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.887 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.887 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:59.888 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579854+0000", 2020-01-16T00:13:59.888 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.005078117 2020-01-16T00:13:59.888 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.888 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.888 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:59.888 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.584932+0000", 2020-01-16T00:13:59.888 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0014702160000000001 2020-01-16T00:13:59.889 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.889 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.889 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.889 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.586402+0000", 2020-01-16T00:13:59.889 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040623589999999998 2020-01-16T00:13:59.889 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.889 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.889 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.890 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590465+0000", 2020-01-16T00:13:59.890 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4478000000000002e-05 2020-01-16T00:13:59.890 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.890 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.890 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:59.890 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590529+0000", 2020-01-16T00:13:59.890 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1121000000000001e-05 2020-01-16T00:13:59.891 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.891 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.891 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:59.891 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590580+0000", 2020-01-16T00:13:59.891 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011826918 2020-01-16T00:13:59.891 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.891 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.891 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.892 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.892 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.892 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:73 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 1922048~118272] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e37)", 2020-01-16T00:13:59.892 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:13:59.892 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 47.199849784999998, 2020-01-16T00:13:59.892 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011750764, 2020-01-16T00:13:59.892 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.893 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:59.893 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.893 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.893 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.893 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:13:59.893 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.893 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.893 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.894 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.894 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:13:59.894 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:59.894 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.894 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.894 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.894 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:13:59.894 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2780000000000001e-05 2020-01-16T00:13:59.895 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.895 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.895 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.895 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579238+0000", 2020-01-16T00:13:59.895 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4341999999999999e-05 2020-01-16T00:13:59.895 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.895 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.896 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.896 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579253+0000", 2020-01-16T00:13:59.896 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1920000000000001e-06 2020-01-16T00:13:59.896 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.896 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.896 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.896 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579256+0000", 2020-01-16T00:13:59.896 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003224902 2020-01-16T00:13:59.897 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.897 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.897 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.897 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.582481+0000", 2020-01-16T00:13:59.897 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000180729 2020-01-16T00:13:59.897 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.897 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.898 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.898 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.582662+0000", 2020-01-16T00:13:59.898 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00025106100000000001 2020-01-16T00:13:59.898 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.898 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.898 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:59.898 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.582913+0000", 2020-01-16T00:13:59.898 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020814449999999999 2020-01-16T00:13:59.899 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.899 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.899 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:59.899 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.584994+0000", 2020-01-16T00:13:59.899 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001717243 2020-01-16T00:13:59.899 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.899 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.899 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.900 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.586711+0000", 2020-01-16T00:13:59.900 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041349869999999997 2020-01-16T00:13:59.900 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.900 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.900 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.900 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590846+0000", 2020-01-16T00:13:59.900 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3565000000000006e-05 2020-01-16T00:13:59.901 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.901 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.901 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:59.901 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590910+0000", 2020-01-16T00:13:59.901 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7002999999999997e-05 2020-01-16T00:13:59.901 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.901 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.902 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:59.902 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590957+0000", 2020-01-16T00:13:59.902 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011750764 2020-01-16T00:13:59.902 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.902 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.902 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.902 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.902 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.903 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:89 2.1 2:9b8617ce:::smithi12313934-38 low_tier:head [write 69632~773120] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e37)", 2020-01-16T00:13:59.903 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.593701+0000", 2020-01-16T00:13:59.903 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 47.185355158999997, 2020-01-16T00:13:59.903 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010026929, 2020-01-16T00:13:59.903 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.903 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:59.903 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.904 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.904 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.904 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593701+0000", 2020-01-16T00:13:59.904 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.904 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.904 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.904 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.904 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593701+0000", 2020-01-16T00:13:59.905 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:59.905 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.905 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.905 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.905 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593699+0000", 2020-01-16T00:13:59.905 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000237564 2020-01-16T00:13:59.905 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.906 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.906 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.906 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593936+0000", 2020-01-16T00:13:59.906 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2666e-05 2020-01-16T00:13:59.906 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.906 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.906 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.906 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593949+0000", 2020-01-16T00:13:59.907 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0089999999999998e-06 2020-01-16T00:13:59.907 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.907 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.907 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.907 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593954+0000", 2020-01-16T00:13:59.907 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5813000000000003e-05 2020-01-16T00:13:59.907 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.908 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.908 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.908 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.594000+0000", 2020-01-16T00:13:59.908 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010155299999999999 2020-01-16T00:13:59.908 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.908 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.908 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.908 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.594101+0000", 2020-01-16T00:13:59.909 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000203807 2020-01-16T00:13:59.909 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.909 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.909 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:59.909 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.594305+0000", 2020-01-16T00:13:59.909 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021655390000000002 2020-01-16T00:13:59.909 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.909 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.910 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.910 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.596471+0000", 2020-01-16T00:13:59.910 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0070645320000000001 2020-01-16T00:13:59.910 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.910 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.910 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.910 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603535+0000", 2020-01-16T00:13:59.911 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7577999999999998e-05 2020-01-16T00:13:59.911 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.911 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.911 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:59.911 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603573+0000", 2020-01-16T00:13:59.911 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9933999999999997e-05 2020-01-16T00:13:59.911 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.911 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.912 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:59.912 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603653+0000", 2020-01-16T00:13:59.912 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4993000000000005e-05 2020-01-16T00:13:59.912 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.912 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.912 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:59.913 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603728+0000", 2020-01-16T00:13:59.913 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010026929 2020-01-16T00:13:59.913 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.913 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.913 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.913 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.913 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.914 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:66 2.3 2:c3b82d87:::smithi12313934-44 low_tier:head [write 169472~818176] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e44)", 2020-01-16T00:13:59.914 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:25.278583+0000", 2020-01-16T00:13:59.914 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 34.500472762000001, 2020-01-16T00:13:59.914 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010294862, 2020-01-16T00:13:59.914 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.914 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:59.914 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.914 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.915 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.915 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.278583+0000", 2020-01-16T00:13:59.915 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.915 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.915 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.915 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.915 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.278583+0000", 2020-01-16T00:13:59.915 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:59.916 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.916 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.916 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.916 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.278581+0000", 2020-01-16T00:13:59.916 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00136721 2020-01-16T00:13:59.916 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.916 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.916 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.917 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.279948+0000", 2020-01-16T00:13:59.917 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6109999999999999e-05 2020-01-16T00:13:59.917 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.917 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.917 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.917 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.279974+0000", 2020-01-16T00:13:59.917 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.102e-05 2020-01-16T00:13:59.918 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.918 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.918 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.918 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.279985+0000", 2020-01-16T00:13:59.918 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6035e-05 2020-01-16T00:13:59.918 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.918 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.918 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.919 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.280061+0000", 2020-01-16T00:13:59.919 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000176243 2020-01-16T00:13:59.919 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.919 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.919 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.919 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.280238+0000", 2020-01-16T00:13:59.919 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019866099999999999 2020-01-16T00:13:59.920 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.920 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.920 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 7", 2020-01-16T00:13:59.920 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.280436+0000", 2020-01-16T00:13:59.920 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0049761830000000003 2020-01-16T00:13:59.920 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.920 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.920 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.921 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.285412+0000", 2020-01-16T00:13:59.921 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033347339999999998 2020-01-16T00:13:59.921 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.921 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.921 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:59.921 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.288747+0000", 2020-01-16T00:13:59.921 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9156000000000001e-05 2020-01-16T00:13:59.922 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.922 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.922 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:59.922 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.288806+0000", 2020-01-16T00:13:59.922 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1643000000000005e-05 2020-01-16T00:13:59.922 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.922 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.922 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:59.923 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.288878+0000", 2020-01-16T00:13:59.923 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010294862 2020-01-16T00:13:59.923 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.923 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.923 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.923 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.924 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.924 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:77 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 56832~1408000] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e45)", 2020-01-16T00:13:59.924 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.584566+0000", 2020-01-16T00:13:59.924 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 30.194490111, 2020-01-16T00:13:59.924 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017037775000000002, 2020-01-16T00:13:59.924 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.924 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:59.925 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.925 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.925 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.925 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584566+0000", 2020-01-16T00:13:59.925 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.925 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.925 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.926 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.926 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584566+0000", 2020-01-16T00:13:59.926 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:13:59.926 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.926 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.926 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.926 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584564+0000", 2020-01-16T00:13:59.927 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043024300000000002 2020-01-16T00:13:59.927 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.927 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.927 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.927 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584994+0000", 2020-01-16T00:13:59.927 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9485999999999999e-05 2020-01-16T00:13:59.927 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.927 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.928 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.928 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585014+0000", 2020-01-16T00:13:59.928 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8909999999999995e-06 2020-01-16T00:13:59.928 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.928 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.928 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.928 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585022+0000", 2020-01-16T00:13:59.928 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039114459999999998 2020-01-16T00:13:59.929 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.929 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.929 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.929 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588933+0000", 2020-01-16T00:13:59.929 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011178700000000001 2020-01-16T00:13:59.929 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.929 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.930 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.930 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589045+0000", 2020-01-16T00:13:59.930 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016967300000000001 2020-01-16T00:13:59.930 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.930 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.930 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:59.931 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589214+0000", 2020-01-16T00:13:59.931 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012142356 2020-01-16T00:13:59.931 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.931 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.931 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.931 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601357+0000", 2020-01-16T00:13:59.931 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.1466999999999997e-05 2020-01-16T00:13:59.931 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.932 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.932 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.932 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601448+0000", 2020-01-16T00:13:59.932 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4817000000000001e-05 2020-01-16T00:13:59.932 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.932 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.932 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:59.932 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601493+0000", 2020-01-16T00:13:59.934 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9734000000000001e-05 2020-01-16T00:13:59.935 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.935 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.935 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:59.935 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601553+0000", 2020-01-16T00:13:59.935 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0677999999999998e-05 2020-01-16T00:13:59.935 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.936 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.936 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:59.938 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601604+0000", 2020-01-16T00:13:59.938 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017037775000000002 2020-01-16T00:13:59.939 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.939 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.939 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.939 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.939 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.939 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:78 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 1922048~118272] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e45)", 2020-01-16T00:13:59.939 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.585030+0000", 2020-01-16T00:13:59.940 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 30.194025389, 2020-01-16T00:13:59.940 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016699964000000001, 2020-01-16T00:13:59.940 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.940 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:59.940 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.940 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.940 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.940 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585030+0000", 2020-01-16T00:13:59.941 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.942 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.942 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.943 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.943 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585030+0000", 2020-01-16T00:13:59.943 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:59.943 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.943 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.943 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.943 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585030+0000", 2020-01-16T00:13:59.943 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7085000000000001e-05 2020-01-16T00:13:59.944 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.944 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.944 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.944 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585077+0000", 2020-01-16T00:13:59.944 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5380000000000008e-06 2020-01-16T00:13:59.944 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.944 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.945 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.945 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585086+0000", 2020-01-16T00:13:59.945 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5410000000000002e-06 2020-01-16T00:13:59.945 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.945 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.945 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.945 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585091+0000", 2020-01-16T00:13:59.945 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0088486620000000002 2020-01-16T00:13:59.946 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.946 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.946 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.946 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.593940+0000", 2020-01-16T00:13:59.946 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112594 2020-01-16T00:13:59.946 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.946 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.947 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.947 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.594052+0000", 2020-01-16T00:13:59.947 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019893999999999999 2020-01-16T00:13:59.947 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.947 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.947 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:59.947 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.594251+0000", 2020-01-16T00:13:59.947 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0071632730000000004 2020-01-16T00:13:59.948 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.948 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.948 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.948 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601414+0000", 2020-01-16T00:13:59.948 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4672999999999996e-05 2020-01-16T00:13:59.948 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.948 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.948 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.949 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601479+0000", 2020-01-16T00:13:59.949 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000158453 2020-01-16T00:13:59.949 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.949 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.949 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:59.949 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601638+0000", 2020-01-16T00:13:59.949 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8605e-05 2020-01-16T00:13:59.950 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.950 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.950 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:59.950 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601686+0000", 2020-01-16T00:13:59.950 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4283999999999998e-05 2020-01-16T00:13:59.950 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.950 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.950 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:59.951 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601730+0000", 2020-01-16T00:13:59.951 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016699964000000001 2020-01-16T00:13:59.951 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.951 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.951 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.951 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.951 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.951 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:126 2.1 2:a4df40e2:::smithi12313934-42 low_tier:head [write 70656~727552] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e45)", 2020-01-16T00:13:59.952 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.627078+0000", 2020-01-16T00:13:59.952 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 30.151977732999999, 2020-01-16T00:13:59.952 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010098995, 2020-01-16T00:13:59.952 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.952 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:59.952 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.952 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.953 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627078+0000", 2020-01-16T00:13:59.953 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.953 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.953 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.953 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627078+0000", 2020-01-16T00:13:59.954 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:59.954 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.954 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.954 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.954 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627076+0000", 2020-01-16T00:13:59.954 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019321599999999999 2020-01-16T00:13:59.954 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.954 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.955 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.955 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627269+0000", 2020-01-16T00:13:59.955 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8471999999999999e-05 2020-01-16T00:13:59.955 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.955 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.955 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.955 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627288+0000", 2020-01-16T00:13:59.956 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3199999999999996e-06 2020-01-16T00:13:59.956 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.956 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.956 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.956 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627294+0000", 2020-01-16T00:13:59.956 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3341000000000002e-05 2020-01-16T00:13:59.956 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.956 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.957 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.957 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627357+0000", 2020-01-16T00:13:59.957 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010985 2020-01-16T00:13:59.957 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.957 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.957 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.957 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627467+0000", 2020-01-16T00:13:59.958 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022914499999999999 2020-01-16T00:13:59.958 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.958 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.958 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:59.958 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627696+0000", 2020-01-16T00:13:59.958 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0055120000000000004 2020-01-16T00:13:59.958 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.958 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.959 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:59.959 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.633208+0000", 2020-01-16T00:13:59.959 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0019968569999999999 2020-01-16T00:13:59.959 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.959 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.959 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.959 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.635205+0000", 2020-01-16T00:13:59.959 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018113160000000001 2020-01-16T00:13:59.960 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.960 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.960 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.960 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.637016+0000", 2020-01-16T00:13:59.960 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9515e-05 2020-01-16T00:13:59.960 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.960 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.961 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:59.961 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.637096+0000", 2020-01-16T00:13:59.961 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1213999999999999e-05 2020-01-16T00:13:59.961 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.961 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.961 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:59.961 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.637177+0000", 2020-01-16T00:13:59.961 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010098995 2020-01-16T00:13:59.962 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.962 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.962 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.962 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.962 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.962 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:127 2.1 2:a4df40e2:::smithi12313934-42 low_tier:head [write 1364992~1145344] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e45)", 2020-01-16T00:13:59.962 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.627308+0000", 2020-01-16T00:13:59.963 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 30.151747937, 2020-01-16T00:13:59.963 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011068883999999999, 2020-01-16T00:13:59.963 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.963 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:59.963 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.963 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.963 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.963 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627308+0000", 2020-01-16T00:13:59.964 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.964 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.964 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.964 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.964 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627308+0000", 2020-01-16T00:13:59.964 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:59.964 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.964 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.965 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.965 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627307+0000", 2020-01-16T00:13:59.965 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000278597 2020-01-16T00:13:59.965 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.965 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.965 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.965 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627586+0000", 2020-01-16T00:13:59.966 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9169999999999997e-06 2020-01-16T00:13:59.966 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.966 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.966 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.966 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627591+0000", 2020-01-16T00:13:59.966 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0479999999999999e-06 2020-01-16T00:13:59.966 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.966 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.967 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.967 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627594+0000", 2020-01-16T00:13:59.967 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030987269999999999 2020-01-16T00:13:59.967 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.967 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.967 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.967 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.630693+0000", 2020-01-16T00:13:59.968 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000103688 2020-01-16T00:13:59.968 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.968 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.968 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.968 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.630796+0000", 2020-01-16T00:13:59.968 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000144606 2020-01-16T00:13:59.968 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.968 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.969 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:59.969 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.630941+0000", 2020-01-16T00:13:59.969 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039264269999999997 2020-01-16T00:13:59.969 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.969 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.969 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:59.969 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.634867+0000", 2020-01-16T00:13:59.970 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00038581200000000003 2020-01-16T00:13:59.970 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.970 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.970 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.970 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.635253+0000", 2020-01-16T00:13:59.970 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030568879999999998 2020-01-16T00:13:59.970 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.970 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.971 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.971 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.638310+0000", 2020-01-16T00:13:59.971 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8390999999999999e-05 2020-01-16T00:13:59.971 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.971 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.971 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:59.971 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.638348+0000", 2020-01-16T00:13:59.972 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8402999999999999e-05 2020-01-16T00:13:59.972 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.972 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.972 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:59.972 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.638377+0000", 2020-01-16T00:13:59.972 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011068883999999999 2020-01-16T00:13:59.972 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.972 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.973 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.973 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.973 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.973 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.6.13:30 2.3 2:fc791323:::smithi12313934-2 low_tier:head [write 476160~1240064] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e48)", 2020-01-16T00:13:59.973 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:39.466268+0000", 2020-01-16T00:13:59.973 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 20.312787632999999, 2020-01-16T00:13:59.973 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010593112, 2020-01-16T00:13:59.973 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.974 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:59.974 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.974 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.974 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.974 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.466268+0000", 2020-01-16T00:13:59.974 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.974 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.975 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.975 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.975 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.466268+0000", 2020-01-16T00:13:59.975 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:13:59.975 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.975 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.975 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.975 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.466266+0000", 2020-01-16T00:13:59.976 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012489210000000001 2020-01-16T00:13:59.976 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.976 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.976 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.976 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467515+0000", 2020-01-16T00:13:59.976 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8411999999999999e-05 2020-01-16T00:13:59.976 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.977 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.977 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.977 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467533+0000", 2020-01-16T00:13:59.977 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9319999999999996e-06 2020-01-16T00:13:59.977 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.977 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.977 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.977 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467541+0000", 2020-01-16T00:13:59.978 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5289999999999993e-05 2020-01-16T00:13:59.978 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.978 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.978 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.978 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467607+0000", 2020-01-16T00:13:59.978 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010962 2020-01-16T00:13:59.978 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.979 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.979 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.979 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467716+0000", 2020-01-16T00:13:59.979 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020883699999999999 2020-01-16T00:13:59.979 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.979 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.979 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 7", 2020-01-16T00:13:59.979 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467925+0000", 2020-01-16T00:13:59.980 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063872829999999997 2020-01-16T00:13:59.980 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.980 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.980 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:59.980 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.474312+0000", 2020-01-16T00:13:59.980 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0023165769999999998 2020-01-16T00:13:59.980 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.981 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.981 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.981 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.476629+0000", 2020-01-16T00:13:59.981 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8543999999999999e-05 2020-01-16T00:13:59.981 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.981 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.981 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:59.981 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.476707+0000", 2020-01-16T00:13:59.982 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000153914 2020-01-16T00:13:59.982 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.982 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.982 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:59.982 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.476861+0000", 2020-01-16T00:13:59.982 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010593112 2020-01-16T00:13:59.982 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.982 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.983 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.983 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.983 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.983 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:155 2.3 2:c9873fc7:::smithi12313934-30 low_tier:head [write 1542656~711168] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e50)", 2020-01-16T00:13:59.983 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:43.769943+0000", 2020-01-16T00:13:59.983 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 16.009112707, 2020-01-16T00:13:59.983 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010710127999999999, 2020-01-16T00:13:59.984 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.984 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:59.984 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.984 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.984 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.984 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.769943+0000", 2020-01-16T00:13:59.984 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.984 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.985 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.985 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.985 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.769943+0000", 2020-01-16T00:13:59.985 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:13:59.985 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.985 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.985 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.986 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.769942+0000", 2020-01-16T00:13:59.986 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017878100000000001 2020-01-16T00:13:59.986 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.986 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.986 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.986 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.770121+0000", 2020-01-16T00:13:59.986 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2980000000000003e-06 2020-01-16T00:13:59.986 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.987 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.987 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.987 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.770129+0000", 2020-01-16T00:13:59.987 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9129999999999999e-06 2020-01-16T00:13:59.987 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.987 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.987 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.987 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.770131+0000", 2020-01-16T00:13:59.988 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0053233519999999999 2020-01-16T00:13:59.988 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.988 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.988 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.988 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.775455+0000", 2020-01-16T00:13:59.988 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012385099999999999 2020-01-16T00:13:59.988 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.988 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.989 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.989 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.775579+0000", 2020-01-16T00:13:59.989 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000211554 2020-01-16T00:13:59.989 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.989 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.989 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 7", 2020-01-16T00:13:59.989 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.775790+0000", 2020-01-16T00:13:59.990 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029614210000000001 2020-01-16T00:13:59.990 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.990 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.990 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:13:59.990 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.778752+0000", 2020-01-16T00:13:59.990 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001798928 2020-01-16T00:13:59.990 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.990 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.991 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:13:59.991 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.780551+0000", 2020-01-16T00:13:59.991 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9413999999999999e-05 2020-01-16T00:13:59.991 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.991 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.991 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:13:59.991 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.780590+0000", 2020-01-16T00:13:59.991 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3275999999999995e-05 2020-01-16T00:13:59.992 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.992 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.992 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:13:59.992 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.780653+0000", 2020-01-16T00:13:59.992 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010710127999999999 2020-01-16T00:13:59.992 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.992 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:13:59.992 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:13:59.993 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.993 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.993 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:170 2.1 2:a935ad67:::smithi12313934-50 low_tier:head [write 1613312~498688] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e55)", 2020-01-16T00:13:59.993 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:58.196603+0000", 2020-01-16T00:13:59.993 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 1.5824530080000001, 2020-01-16T00:13:59.993 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0099186129999999997, 2020-01-16T00:13:59.993 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:13:59.994 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:13:59.994 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:13:59.994 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.994 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:13:59.994 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196603+0000", 2020-01-16T00:13:59.994 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:13:59.994 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.994 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.995 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:13:59.995 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196603+0000", 2020-01-16T00:13:59.995 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:13:59.995 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.995 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.995 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:13:59.995 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196602+0000", 2020-01-16T00:13:59.996 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000129679 2020-01-16T00:13:59.996 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.996 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.996 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:13:59.996 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196732+0000", 2020-01-16T00:13:59.996 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6699999999999994e-06 2020-01-16T00:13:59.996 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.996 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.997 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:13:59.997 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196739+0000", 2020-01-16T00:13:59.997 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8080000000000004e-06 2020-01-16T00:13:59.997 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.997 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.997 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:13:59.997 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196744+0000", 2020-01-16T00:13:59.997 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0015686980000000001 2020-01-16T00:13:59.998 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.998 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.998 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:13:59.998 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.198313+0000", 2020-01-16T00:13:59.998 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000120653 2020-01-16T00:13:59.998 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.998 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.999 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:13:59.999 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.198434+0000", 2020-01-16T00:13:59.999 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000183845 2020-01-16T00:13:59.999 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:13:59.999 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:13:59.999 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:13:59.999 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.198617+0000", 2020-01-16T00:13:59.999 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0074501710000000002 2020-01-16T00:14:00.000 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:00.000 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:00.000 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:00.000 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.206068+0000", 2020-01-16T00:14:00.000 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00031512800000000002 2020-01-16T00:14:00.000 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:00.000 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:00.000 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:00.001 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.206383+0000", 2020-01-16T00:14:00.001 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2570000000000007e-06 2020-01-16T00:14:00.001 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:00.001 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:00.001 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:00.001 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.206391+0000", 2020-01-16T00:14:00.001 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5562000000000006e-05 2020-01-16T00:14:00.001 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:00.002 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:00.002 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:00.002 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.206467+0000", 2020-01-16T00:14:00.002 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4911000000000003e-05 2020-01-16T00:14:00.002 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:00.002 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:00.002 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:00.003 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.206521+0000", 2020-01-16T00:14:00.003 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0099186129999999997 2020-01-16T00:14:00.003 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:00.003 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:00.003 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:00.003 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:00.003 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:00.003 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:171 2.1 2:a935ad67:::smithi12313934-50 low_tier:head [write 2422272~237568] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e55)", 2020-01-16T00:14:00.004 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:58.196749+0000", 2020-01-16T00:14:00.004 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 1.582306422, 2020-01-16T00:14:00.004 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010677759, 2020-01-16T00:14:00.004 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:00.004 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:00.004 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:00.004 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:00.005 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:00.005 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196749+0000", 2020-01-16T00:14:00.005 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:00.005 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:00.005 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:00.005 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:00.005 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196749+0000", 2020-01-16T00:14:00.005 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:00.006 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:00.006 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:00.006 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:00.006 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196749+0000", 2020-01-16T00:14:00.006 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6604000000000001e-05 2020-01-16T00:14:00.006 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:00.006 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:00.007 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:00.007 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196815+0000", 2020-01-16T00:14:00.007 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4900000000000006e-06 2020-01-16T00:14:00.007 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:00.007 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:00.007 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:00.007 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196825+0000", 2020-01-16T00:14:00.007 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3989999999999997e-06 2020-01-16T00:14:00.008 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:00.008 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:00.008 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:00.008 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196829+0000", 2020-01-16T00:14:00.008 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0076943869999999996 2020-01-16T00:14:00.008 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:00.008 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:00.008 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:00.009 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.204524+0000", 2020-01-16T00:14:00.009 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000124745 2020-01-16T00:14:00.009 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:00.009 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:00.009 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:00.009 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.204649+0000", 2020-01-16T00:14:00.009 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018801700000000001 2020-01-16T00:14:00.010 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:00.010 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:00.010 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:00.010 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.204837+0000", 2020-01-16T00:14:00.010 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002354765 2020-01-16T00:14:00.010 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:00.010 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:00.010 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:00.011 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.207191+0000", 2020-01-16T00:14:00.011 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.6249999999999996e-06 2020-01-16T00:14:00.011 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:00.011 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:00.011 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:00.011 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.207200+0000", 2020-01-16T00:14:00.011 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000162246 2020-01-16T00:14:00.012 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:00.012 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:00.012 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:00.012 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.207362+0000", 2020-01-16T00:14:00.012 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5117999999999998e-05 2020-01-16T00:14:00.012 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:00.012 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:00.012 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:00.013 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.207397+0000", 2020-01-16T00:14:00.013 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9912000000000001e-05 2020-01-16T00:14:00.013 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:00.013 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:00.013 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:00.013 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.207427+0000", 2020-01-16T00:14:00.014 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010677759 2020-01-16T00:14:00.014 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:00.014 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:00.014 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:00.014 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:00.014 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:00.014 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:00.015 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2020-01-16T00:14:00.019 INFO:tasks.rados.rados.0.smithi123.stdout:1603: finishing write tid 1 to smithi12313934-13 2020-01-16T00:14:00.019 INFO:tasks.rados.rados.0.smithi123.stdout:1601: finishing write tid 3 to smithi12313934-45 2020-01-16T00:14:00.019 INFO:tasks.rados.rados.0.smithi123.stdout:1601: finishing write tid 5 to smithi12313934-45 2020-01-16T00:14:00.019 INFO:tasks.rados.rados.0.smithi123.stdout:1601: finishing write tid 6 to smithi12313934-45 2020-01-16T00:14:00.020 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 45 v 169 (ObjNum 559 snap 0 seq_num 559) dirty exists 2020-01-16T00:14:00.020 INFO:tasks.rados.rados.0.smithi123.stdout:1601: left oid 45 (ObjNum 559 snap 0 seq_num 559) 2020-01-16T00:14:00.020 INFO:tasks.rados.rados.0.smithi123.stdout:1603: finishing write tid 2 to smithi12313934-13 2020-01-16T00:14:00.020 INFO:tasks.rados.rados.0.smithi123.stdout:1600: done (15 left) 2020-01-16T00:14:00.020 INFO:tasks.rados.rados.0.smithi123.stdout:1601: done (14 left) 2020-01-16T00:14:00.020 INFO:tasks.rados.rados.0.smithi123.stdout:1602: done (13 left) 2020-01-16T00:14:00.020 INFO:tasks.rados.rados.0.smithi123.stdout:1615: done (12 left) 2020-01-16T00:14:00.020 INFO:tasks.rados.rados.0.smithi123.stdout:1616: write (excl) oid 6 current snap is 0 2020-01-16T00:14:00.021 INFO:tasks.rados.rados.0.smithi123.stdout:1616: seq_num 568 ranges {746376=551554,2054831=649524,3315201=231175} 2020-01-16T00:14:00.021 INFO:tasks.rados.rados.0.smithi123.stdout:1616: writing smithi12313934-6 from 746376 to 1297930 tid 1 2020-01-16T00:14:00.021 INFO:tasks.rados.rados.0.smithi123.stdout:1616: writing smithi12313934-6 from 2054831 to 2704355 tid 2 2020-01-16T00:14:00.021 INFO:tasks.rados.rados.0.smithi123.stdout:1616: writing smithi12313934-6 from 3315201 to 3546376 tid 3 2020-01-16T00:14:00.021 INFO:tasks.rados.rados.0.smithi123.stdout:1617: read oid 19 snap -1 2020-01-16T00:14:00.021 INFO:tasks.rados.rados.0.smithi123.stdout:1617: expect (ObjNum 554 snap 0 seq_num 554) 2020-01-16T00:14:00.021 INFO:tasks.rados.rados.0.smithi123.stdout:1618: write oid 35 current snap is 0 2020-01-16T00:14:00.022 INFO:tasks.rados.rados.0.smithi123.stdout:1618: seq_num 569 ranges {763183=626803,1996960=793182,3226271=336912} 2020-01-16T00:14:00.022 INFO:tasks.rados.rados.0.smithi123.stdout:1618: writing smithi12313934-35 from 763183 to 1389986 tid 1 2020-01-16T00:14:00.022 INFO:tasks.rados.rados.0.smithi123.stdout:1618: writing smithi12313934-35 from 1996960 to 2790142 tid 2 2020-01-16T00:14:00.022 INFO:tasks.rados.rados.0.smithi123.stdout:1618: writing smithi12313934-35 from 3226271 to 3563183 tid 3 2020-01-16T00:14:00.022 INFO:tasks.rados.rados.0.smithi123.stdout:1619: read oid 50 snap -1 2020-01-16T00:14:00.022 INFO:tasks.rados.rados.0.smithi123.stdout:1619: expect (ObjNum 553 snap 0 seq_num 553) 2020-01-16T00:14:00.022 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:00.022 INFO:tasks.rados.rados.0.smithi123.stdout:1603: finishing write tid 3 to smithi12313934-13 2020-01-16T00:14:00.023 INFO:tasks.rados.rados.0.smithi123.stdout:1603: finishing write tid 5 to smithi12313934-13 2020-01-16T00:14:00.023 INFO:tasks.rados.rados.0.smithi123.stdout:1603: finishing write tid 6 to smithi12313934-13 2020-01-16T00:14:00.023 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 113 (ObjNum 560 snap 0 seq_num 560) dirty exists 2020-01-16T00:14:00.023 INFO:tasks.rados.rados.0.smithi123.stdout:1603: left oid 13 (ObjNum 560 snap 0 seq_num 560) 2020-01-16T00:14:00.023 INFO:tasks.rados.rados.0.smithi123.stdout:1604: expect (ObjNum 539 snap 0 seq_num 539) 2020-01-16T00:14:00.023 INFO:tasks.rados.rados.0.smithi123.stdout:1606: finishing write tid 1 to smithi12313934-38 2020-01-16T00:14:00.023 INFO:tasks.rados.rados.0.smithi123.stdout:1606: finishing write tid 2 to smithi12313934-38 2020-01-16T00:14:00.024 INFO:tasks.rados.rados.0.smithi123.stdout:1606: finishing write tid 3 to smithi12313934-38 2020-01-16T00:14:00.024 INFO:tasks.rados.rados.0.smithi123.stdout:1606: finishing write tid 5 to smithi12313934-38 2020-01-16T00:14:00.024 INFO:tasks.rados.rados.0.smithi123.stdout:1606: finishing write tid 6 to smithi12313934-38 2020-01-16T00:14:00.024 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 38 v 191 (ObjNum 561 snap 0 seq_num 561) dirty exists 2020-01-16T00:14:00.024 INFO:tasks.rados.rados.0.smithi123.stdout:1606: left oid 38 (ObjNum 561 snap 0 seq_num 561) 2020-01-16T00:14:00.024 INFO:tasks.rados.rados.0.smithi123.stdout:1607: finishing write tid 1 to smithi12313934-27 2020-01-16T00:14:00.024 INFO:tasks.rados.rados.0.smithi123.stdout:1607: finishing write tid 2 to smithi12313934-27 2020-01-16T00:14:00.024 INFO:tasks.rados.rados.0.smithi123.stdout:1607: finishing write tid 3 to smithi12313934-27 2020-01-16T00:14:00.025 INFO:tasks.rados.rados.0.smithi123.stdout:1607: finishing write tid 5 to smithi12313934-27 2020-01-16T00:14:00.025 INFO:tasks.rados.rados.0.smithi123.stdout:1607: finishing write tid 6 to smithi12313934-27 2020-01-16T00:14:00.025 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 27 v 160 (ObjNum 562 snap 0 seq_num 562) dirty exists 2020-01-16T00:14:00.025 INFO:tasks.rados.rados.0.smithi123.stdout:1607: left oid 27 (ObjNum 562 snap 0 seq_num 562) 2020-01-16T00:14:00.025 INFO:tasks.rados.rados.0.smithi123.stdout:1608: expect (ObjNum 535 snap 0 seq_num 535) 2020-01-16T00:14:00.025 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:14:00.025 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:14:00.026 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:14:00.026 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:14:00.026 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:14:00.026 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:14:00.026 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:14:00.026 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-01-16T00:14:00.026 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-01-16T00:14:00.027 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-01-16T00:14:00.027 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-01-16T00:14:00.027 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-01-16T00:14:00.027 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENXIO: problem getting command descriptions from osd.5 2020-01-16T00:14:00.027 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: problem getting command descriptions from osd.5 2020-01-16T00:14:00.027 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-01-16T00:14:00.027 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-01-16T00:14:00.027 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-01-16T00:14:00.028 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-01-16T00:14:00.028 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:00.014+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:00.028 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-01-16T00:14:00.116 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:00.114+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:00.145 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:00.155 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-01-16T00:14:00.217 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:00.214+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:00.237 INFO:tasks.rados.rados.0.smithi123.stdout:1609: expect (ObjNum 532 snap 0 seq_num 532) 2020-01-16T00:14:00.274 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:00.295 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-01-16T00:14:00.318 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:00.314+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:00.395 INFO:tasks.rados.rados.0.smithi123.stdout:1610: finishing write tid 1 to smithi12313934-23 2020-01-16T00:14:00.395 INFO:tasks.rados.rados.0.smithi123.stdout:1610: finishing write tid 2 to smithi12313934-23 2020-01-16T00:14:00.395 INFO:tasks.rados.rados.0.smithi123.stdout:1610: finishing write tid 3 to smithi12313934-23 2020-01-16T00:14:00.396 INFO:tasks.rados.rados.0.smithi123.stdout:1610: finishing write tid 5 to smithi12313934-23 2020-01-16T00:14:00.396 INFO:tasks.rados.rados.0.smithi123.stdout:1610: finishing write tid 6 to smithi12313934-23 2020-01-16T00:14:00.396 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 23 v 178 (ObjNum 563 snap 0 seq_num 563) dirty exists 2020-01-16T00:14:00.396 INFO:tasks.rados.rados.0.smithi123.stdout:1610: left oid 23 (ObjNum 563 snap 0 seq_num 563) 2020-01-16T00:14:00.397 INFO:tasks.rados.rados.0.smithi123.stdout:1611: finishing write tid 1 to smithi12313934-49 2020-01-16T00:14:00.397 INFO:tasks.rados.rados.0.smithi123.stdout:1611: finishing write tid 2 to smithi12313934-49 2020-01-16T00:14:00.397 INFO:tasks.rados.rados.0.smithi123.stdout:1611: finishing write tid 3 to smithi12313934-49 2020-01-16T00:14:00.397 INFO:tasks.rados.rados.0.smithi123.stdout:1611: finishing write tid 5 to smithi12313934-49 2020-01-16T00:14:00.397 INFO:tasks.rados.rados.0.smithi123.stdout:1603: done (15 left) 2020-01-16T00:14:00.398 INFO:tasks.rados.rados.0.smithi123.stdout:1604: done (14 left) 2020-01-16T00:14:00.398 INFO:tasks.rados.rados.0.smithi123.stdout:1605: done (13 left) 2020-01-16T00:14:00.398 INFO:tasks.rados.rados.0.smithi123.stdout:1606: done (12 left) 2020-01-16T00:14:00.398 INFO:tasks.rados.rados.0.smithi123.stdout:1607: done (11 left) 2020-01-16T00:14:00.399 INFO:tasks.rados.rados.0.smithi123.stdout:1608: done (10 left) 2020-01-16T00:14:00.399 INFO:tasks.rados.rados.0.smithi123.stdout:1609: done (9 left) 2020-01-16T00:14:00.399 INFO:tasks.rados.rados.0.smithi123.stdout:1610: done (8 left) 2020-01-16T00:14:00.399 INFO:tasks.rados.rados.0.smithi123.stdout:1620: write oid 29 current snap is 0 2020-01-16T00:14:00.399 INFO:tasks.rados.rados.0.smithi123.stdout:1620: seq_num 570 ranges {779990=702052,2055442=653193,3253694=326296} 2020-01-16T00:14:00.403 INFO:tasks.rados.rados.0.smithi123.stdout:1620: writing smithi12313934-29 from 779990 to 1482042 tid 1 2020-01-16T00:14:00.411 INFO:tasks.rados.rados.0.smithi123.stdout:1620: writing smithi12313934-29 from 2055442 to 2708635 tid 2 2020-01-16T00:14:00.418 INFO:tasks.rados.rados.0.smithi123.stdout:1620: writing smithi12313934-29 from 3253694 to 3579990 tid 3 2020-01-16T00:14:00.418 INFO:tasks.rados.rados.0.smithi123.stdout:1621: read oid 40 snap -1 2020-01-16T00:14:00.418 INFO:tasks.rados.rados.0.smithi123.stdout:1621: expect deleted 2020-01-16T00:14:00.419 INFO:tasks.rados.rados.0.smithi123.stdout:1622: read oid 26 snap -1 2020-01-16T00:14:00.419 INFO:tasks.rados.rados.0.smithi123.stdout:1622: expect (ObjNum 556 snap 0 seq_num 556) 2020-01-16T00:14:00.419 INFO:tasks.rados.rados.0.smithi123.stdout:1623: read oid 41 snap -1 2020-01-16T00:14:00.419 INFO:tasks.rados.rados.0.smithi123.stdout:1623: expect (ObjNum 441 snap 0 seq_num 441) 2020-01-16T00:14:00.419 INFO:tasks.rados.rados.0.smithi123.stdout:1624: read oid 14 snap -1 2020-01-16T00:14:00.419 INFO:tasks.rados.rados.0.smithi123.stdout:1624: expect (ObjNum 551 snap 0 seq_num 551) 2020-01-16T00:14:00.419 INFO:tasks.rados.rados.0.smithi123.stdout:1625: write oid 25 current snap is 0 2020-01-16T00:14:00.419 INFO:tasks.rados.rados.0.smithi123.stdout:1625: seq_num 571 ranges {796797=493654,1830277=796851,3397470=199327} 2020-01-16T00:14:00.421 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:00.422 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:00.418+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:00.424 INFO:tasks.rados.rados.0.smithi123.stdout:1625: writing smithi12313934-25 from 796797 to 1290451 tid 1 2020-01-16T00:14:00.432 INFO:tasks.rados.rados.0.smithi123.stdout:1625: writing smithi12313934-25 from 1830277 to 2627128 tid 2 2020-01-16T00:14:00.438 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-01-16T00:14:00.439 INFO:tasks.rados.rados.0.smithi123.stdout:1625: writing smithi12313934-25 from 3397470 to 3596797 tid 3 2020-01-16T00:14:00.440 INFO:tasks.rados.rados.0.smithi123.stdout:1626: read oid 24 snap -1 2020-01-16T00:14:00.440 INFO:tasks.rados.rados.0.smithi123.stdout:1626: expect (ObjNum 510 snap 0 seq_num 510) 2020-01-16T00:14:00.440 INFO:tasks.rados.rados.0.smithi123.stdout:1627: write (excl) oid 9 current snap is 0 2020-01-16T00:14:00.440 INFO:tasks.rados.rados.0.smithi123.stdout:1627: seq_num 572 ranges {413604=568903,1488759=656862,2624893=591709} 2020-01-16T00:14:00.445 INFO:tasks.rados.rados.0.smithi123.stdout:1627: writing smithi12313934-9 from 413604 to 982507 tid 1 2020-01-16T00:14:00.451 INFO:tasks.rados.rados.0.smithi123.stdout:1627: writing smithi12313934-9 from 1488759 to 2145621 tid 2 2020-01-16T00:14:00.458 INFO:tasks.rados.rados.0.smithi123.stdout:1627: writing smithi12313934-9 from 2624893 to 3216602 tid 3 2020-01-16T00:14:00.458 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:00.458 INFO:tasks.rados.rados.0.smithi123.stdout:1611: finishing write tid 6 to smithi12313934-49 2020-01-16T00:14:00.459 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 49 v 173 (ObjNum 564 snap 0 seq_num 564) dirty exists 2020-01-16T00:14:00.459 INFO:tasks.rados.rados.0.smithi123.stdout:1611: left oid 49 (ObjNum 564 snap 0 seq_num 564) 2020-01-16T00:14:00.459 INFO:tasks.rados.rados.0.smithi123.stdout:1612: finishing write tid 1 to smithi12313934-8 2020-01-16T00:14:00.459 INFO:tasks.rados.rados.0.smithi123.stdout:1612: finishing write tid 2 to smithi12313934-8 2020-01-16T00:14:00.459 INFO:tasks.rados.rados.0.smithi123.stdout:1612: finishing write tid 3 to smithi12313934-8 2020-01-16T00:14:00.459 INFO:tasks.rados.rados.0.smithi123.stdout:1612: finishing write tid 5 to smithi12313934-8 2020-01-16T00:14:00.459 INFO:tasks.rados.rados.0.smithi123.stdout:1611: done (15 left) 2020-01-16T00:14:00.460 INFO:tasks.rados.rados.0.smithi123.stdout:1628: write (excl) oid 5 current snap is 0 2020-01-16T00:14:00.460 INFO:tasks.rados.rados.0.smithi123.stdout:1628: seq_num 573 ranges {430411=644152,1830888=400520,2935963=402981} 2020-01-16T00:14:00.463 INFO:tasks.rados.rados.0.smithi123.stdout:1628: writing smithi12313934-5 from 430411 to 1074563 tid 1 2020-01-16T00:14:00.473 INFO:tasks.rados.rados.0.smithi123.stdout:1628: writing smithi12313934-5 from 1830888 to 2231408 tid 2 2020-01-16T00:14:00.475 INFO:tasks.rados.rados.0.smithi123.stdout:1628: writing smithi12313934-5 from 2935963 to 3338944 tid 3 2020-01-16T00:14:00.476 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:00.476 INFO:tasks.rados.rados.0.smithi123.stdout:1612: finishing write tid 6 to smithi12313934-8 2020-01-16T00:14:00.476 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 8 v 120 (ObjNum 565 snap 0 seq_num 565) dirty exists 2020-01-16T00:14:00.476 INFO:tasks.rados.rados.0.smithi123.stdout:1612: left oid 8 (ObjNum 565 snap 0 seq_num 565) 2020-01-16T00:14:00.476 INFO:tasks.rados.rados.0.smithi123.stdout:1612: done (15 left) 2020-01-16T00:14:00.476 INFO:tasks.rados.rados.0.smithi123.stdout:1629: write oid 33 current snap is 0 2020-01-16T00:14:00.476 INFO:tasks.rados.rados.0.smithi123.stdout:1629: seq_num 574 ranges {447218=719401,1889370=660531,2963386=614253} 2020-01-16T00:14:00.482 INFO:tasks.rados.rados.0.smithi123.stdout:1629: writing smithi12313934-33 from 447218 to 1166619 tid 1 2020-01-16T00:14:00.490 INFO:tasks.rados.rados.0.smithi123.stdout:1629: writing smithi12313934-33 from 1889370 to 2549901 tid 2 2020-01-16T00:14:00.496 INFO:tasks.rados.rados.0.smithi123.stdout:1629: writing smithi12313934-33 from 2963386 to 3577639 tid 3 2020-01-16T00:14:00.496 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:00.496 INFO:tasks.rados.rados.0.smithi123.stdout:1613: finishing write tid 1 to smithi12313934-18 2020-01-16T00:14:00.497 INFO:tasks.rados.rados.0.smithi123.stdout:1613: finishing write tid 2 to smithi12313934-18 2020-01-16T00:14:00.497 INFO:tasks.rados.rados.0.smithi123.stdout:1613: finishing write tid 3 to smithi12313934-18 2020-01-16T00:14:00.497 INFO:tasks.rados.rados.0.smithi123.stdout:1613: finishing write tid 5 to smithi12313934-18 2020-01-16T00:14:00.497 INFO:tasks.rados.rados.0.smithi123.stdout:1613: finishing write tid 6 to smithi12313934-18 2020-01-16T00:14:00.497 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 18 v 301 (ObjNum 566 snap 0 seq_num 566) dirty exists 2020-01-16T00:14:00.497 INFO:tasks.rados.rados.0.smithi123.stdout:1613: left oid 18 (ObjNum 566 snap 0 seq_num 566) 2020-01-16T00:14:00.497 INFO:tasks.rados.rados.0.smithi123.stdout:1614: expect (ObjNum 543 snap 0 seq_num 543) 2020-01-16T00:14:00.523 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:00.518+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:00.536 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:00.546 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-01-16T00:14:00.624 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:00.620+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:00.644 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:00.651 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-01-16T00:14:00.667 INFO:tasks.rados.rados.0.smithi123.stdout:1616: finishing write tid 1 to smithi12313934-6 2020-01-16T00:14:00.668 INFO:tasks.rados.rados.0.smithi123.stdout:1616: finishing write tid 2 to smithi12313934-6 2020-01-16T00:14:00.668 INFO:tasks.rados.rados.0.smithi123.stdout:1616: finishing write tid 3 to smithi12313934-6 2020-01-16T00:14:00.668 INFO:tasks.rados.rados.0.smithi123.stdout:1616: finishing write tid 5 to smithi12313934-6 2020-01-16T00:14:00.668 INFO:tasks.rados.rados.0.smithi123.stdout:1616: finishing write tid 6 to smithi12313934-6 2020-01-16T00:14:00.668 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 6 v 165 (ObjNum 568 snap 0 seq_num 568) dirty exists 2020-01-16T00:14:00.669 INFO:tasks.rados.rados.0.smithi123.stdout:1616: left oid 6 (ObjNum 568 snap 0 seq_num 568) 2020-01-16T00:14:00.669 INFO:tasks.rados.rados.0.smithi123.stdout:1618: finishing write tid 1 to smithi12313934-35 2020-01-16T00:14:00.669 INFO:tasks.rados.rados.0.smithi123.stdout:1617: expect (ObjNum 554 snap 0 seq_num 554) 2020-01-16T00:14:00.740 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:00.736+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:00.804 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:00.814 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-01-16T00:14:00.824 INFO:teuthology.orchestra.run.smithi123.stderr:noscrub is unset 2020-01-16T00:14:00.858 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:00.856+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:00.930 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:00.924+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:00.968 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:00.979 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-01-16T00:14:00.981 INFO:tasks.rados.rados.0.smithi123.stdout:1618: finishing write tid 2 to smithi12313934-35 2020-01-16T00:14:00.981 INFO:tasks.rados.rados.0.smithi123.stdout:1618: finishing write tid 3 to smithi12313934-35 2020-01-16T00:14:00.981 INFO:tasks.rados.rados.0.smithi123.stdout:1618: finishing write tid 5 to smithi12313934-35 2020-01-16T00:14:00.981 INFO:tasks.rados.rados.0.smithi123.stdout:1618: finishing write tid 6 to smithi12313934-35 2020-01-16T00:14:00.982 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 35 v 182 (ObjNum 569 snap 0 seq_num 569) dirty exists 2020-01-16T00:14:00.982 INFO:tasks.rados.rados.0.smithi123.stdout:1618: left oid 35 (ObjNum 569 snap 0 seq_num 569) 2020-01-16T00:14:00.982 INFO:tasks.rados.rados.0.smithi123.stdout:1619: expect (ObjNum 553 snap 0 seq_num 553) 2020-01-16T00:14:01.070 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:01.068+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:01.131 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:01.130+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:01.139 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:01.148 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-01-16T00:14:01.215 INFO:tasks.rados.rados.0.smithi123.stdout:1620: finishing write tid 1 to smithi12313934-29 2020-01-16T00:14:01.216 INFO:tasks.rados.rados.0.smithi123.stdout:1620: finishing write tid 2 to smithi12313934-29 2020-01-16T00:14:01.216 INFO:tasks.rados.rados.0.smithi123.stdout:1622: expect (ObjNum 556 snap 0 seq_num 556) 2020-01-16T00:14:01.232 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:01.230+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:01.304 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:01.313 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-01-16T00:14:01.359 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:01.358+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:01.420 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:01.429 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-01-16T00:14:01.463 INFO:tasks.rados.rados.0.smithi123.stdout:1620: finishing write tid 3 to smithi12313934-29 2020-01-16T00:14:01.464 INFO:tasks.rados.rados.0.smithi123.stdout:1620: finishing write tid 5 to smithi12313934-29 2020-01-16T00:14:01.464 INFO:tasks.rados.rados.0.smithi123.stdout:1620: finishing write tid 6 to smithi12313934-29 2020-01-16T00:14:01.464 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 29 v 175 (ObjNum 570 snap 0 seq_num 570) dirty exists 2020-01-16T00:14:01.464 INFO:tasks.rados.rados.0.smithi123.stdout:1620: left oid 29 (ObjNum 570 snap 0 seq_num 570) 2020-01-16T00:14:01.464 INFO:tasks.rados.rados.0.smithi123.stdout:1625: finishing write tid 1 to smithi12313934-25 2020-01-16T00:14:01.464 INFO:tasks.rados.rados.0.smithi123.stdout:1624: expect (ObjNum 551 snap 0 seq_num 551) 2020-01-16T00:14:01.465 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:01.462+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:01.535 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:01.532+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:01.569 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:01.579 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-01-16T00:14:01.636 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:01.632+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:01.681 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:01.690 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-01-16T00:14:01.730 INFO:tasks.rados.rados.0.smithi123.stdout:1623: expect (ObjNum 441 snap 0 seq_num 441) 2020-01-16T00:14:01.742 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:01.732+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:01.791 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:01.801 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-01-16T00:14:01.843 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:01.842+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:01.939 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:01.938+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:01.941 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:01.945 INFO:tasks.rados.rados.0.smithi123.stdout:1625: finishing write tid 2 to smithi12313934-25 2020-01-16T00:14:01.945 INFO:tasks.rados.rados.0.smithi123.stdout:1625: finishing write tid 3 to smithi12313934-25 2020-01-16T00:14:01.946 INFO:tasks.rados.rados.0.smithi123.stdout:1625: finishing write tid 5 to smithi12313934-25 2020-01-16T00:14:01.946 INFO:tasks.rados.rados.0.smithi123.stdout:1625: finishing write tid 6 to smithi12313934-25 2020-01-16T00:14:01.946 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 25 v 249 (ObjNum 571 snap 0 seq_num 571) dirty exists 2020-01-16T00:14:01.946 INFO:tasks.rados.rados.0.smithi123.stdout:1625: left oid 25 (ObjNum 571 snap 0 seq_num 571) 2020-01-16T00:14:01.946 INFO:tasks.rados.rados.0.smithi123.stdout:1626: expect (ObjNum 510 snap 0 seq_num 510) 2020-01-16T00:14:01.955 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-01-16T00:14:02.026 INFO:tasks.rados.rados.0.smithi123.stdout:1627: finishing write tid 1 to smithi12313934-9 2020-01-16T00:14:02.026 INFO:tasks.rados.rados.0.smithi123.stdout:1627: finishing write tid 2 to smithi12313934-9 2020-01-16T00:14:02.026 INFO:tasks.rados.rados.0.smithi123.stdout:1627: finishing write tid 3 to smithi12313934-9 2020-01-16T00:14:02.027 INFO:tasks.rados.rados.0.smithi123.stdout:1627: finishing write tid 5 to smithi12313934-9 2020-01-16T00:14:02.027 INFO:tasks.rados.rados.0.smithi123.stdout:1627: finishing write tid 6 to smithi12313934-9 2020-01-16T00:14:02.027 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 9 v 164 (ObjNum 572 snap 0 seq_num 572) dirty exists 2020-01-16T00:14:02.027 INFO:tasks.rados.rados.0.smithi123.stdout:1627: left oid 9 (ObjNum 572 snap 0 seq_num 572) 2020-01-16T00:14:02.027 INFO:tasks.rados.rados.0.smithi123.stdout:1628: finishing write tid 1 to smithi12313934-5 2020-01-16T00:14:02.027 INFO:tasks.rados.rados.0.smithi123.stdout:1628: finishing write tid 2 to smithi12313934-5 2020-01-16T00:14:02.028 INFO:tasks.rados.rados.0.smithi123.stdout:1628: finishing write tid 3 to smithi12313934-5 2020-01-16T00:14:02.028 INFO:tasks.rados.rados.0.smithi123.stdout:1628: finishing write tid 5 to smithi12313934-5 2020-01-16T00:14:02.028 INFO:tasks.rados.rados.0.smithi123.stdout:1628: finishing write tid 6 to smithi12313934-5 2020-01-16T00:14:02.028 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 5 v 253 (ObjNum 573 snap 0 seq_num 573) dirty exists 2020-01-16T00:14:02.028 INFO:tasks.rados.rados.0.smithi123.stdout:1628: left oid 5 (ObjNum 573 snap 0 seq_num 573) 2020-01-16T00:14:02.028 INFO:tasks.rados.rados.0.smithi123.stdout:1629: finishing write tid 1 to smithi12313934-33 2020-01-16T00:14:02.028 INFO:tasks.rados.rados.0.smithi123.stdout:1613: done (15 left) 2020-01-16T00:14:02.029 INFO:tasks.rados.rados.0.smithi123.stdout:1614: done (14 left) 2020-01-16T00:14:02.029 INFO:tasks.rados.rados.0.smithi123.stdout:1616: done (13 left) 2020-01-16T00:14:02.029 INFO:tasks.rados.rados.0.smithi123.stdout:1617: done (12 left) 2020-01-16T00:14:02.029 INFO:tasks.rados.rados.0.smithi123.stdout:1618: done (11 left) 2020-01-16T00:14:02.029 INFO:tasks.rados.rados.0.smithi123.stdout:1619: done (10 left) 2020-01-16T00:14:02.029 INFO:tasks.rados.rados.0.smithi123.stdout:1620: done (9 left) 2020-01-16T00:14:02.029 INFO:tasks.rados.rados.0.smithi123.stdout:1621: done (8 left) 2020-01-16T00:14:02.029 INFO:tasks.rados.rados.0.smithi123.stdout:1622: done (7 left) 2020-01-16T00:14:02.030 INFO:tasks.rados.rados.0.smithi123.stdout:1623: done (6 left) 2020-01-16T00:14:02.030 INFO:tasks.rados.rados.0.smithi123.stdout:1624: done (5 left) 2020-01-16T00:14:02.030 INFO:tasks.rados.rados.0.smithi123.stdout:1625: done (4 left) 2020-01-16T00:14:02.030 INFO:tasks.rados.rados.0.smithi123.stdout:1626: done (3 left) 2020-01-16T00:14:02.030 INFO:tasks.rados.rados.0.smithi123.stdout:1627: done (2 left) 2020-01-16T00:14:02.030 INFO:tasks.rados.rados.0.smithi123.stdout:1628: done (1 left) 2020-01-16T00:14:02.030 INFO:tasks.rados.rados.0.smithi123.stdout:1630: write oid 22 current snap is 0 2020-01-16T00:14:02.031 INFO:tasks.rados.rados.0.smithi123.stdout:1630: seq_num 575 ranges {464025=794650,1947852=404189,2990809=425525} 2020-01-16T00:14:02.032 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-01-16T00:14:02.039 INFO:tasks.rados.rados.0.smithi123.stdout:1630: writing smithi12313934-22 from 464025 to 1258675 tid 1 2020-01-16T00:14:02.040 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:02.038+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:02.048 INFO:tasks.rados.rados.0.smithi123.stdout:1630: writing smithi12313934-22 from 1947852 to 2352041 tid 2 2020-01-16T00:14:02.054 INFO:tasks.rados.rados.0.smithi123.stdout:1630: writing smithi12313934-22 from 2990809 to 3416334 tid 3 2020-01-16T00:14:02.054 INFO:tasks.rados.rados.0.smithi123.stdout:1631: read oid 47 snap -1 2020-01-16T00:14:02.054 INFO:tasks.rados.rados.0.smithi123.stdout:1631: expect (ObjNum 550 snap 0 seq_num 550) 2020-01-16T00:14:02.054 INFO:tasks.rados.rados.0.smithi123.stdout:1632: write oid 21 current snap is 0 2020-01-16T00:14:02.054 INFO:tasks.rados.rados.0.smithi123.stdout:1632: seq_num 576 ranges {480832=469899,1606334=664200,3018232=662600} 2020-01-16T00:14:02.059 INFO:tasks.rados.rados.0.smithi123.stdout:1632: writing smithi12313934-21 from 480832 to 950731 tid 1 2020-01-16T00:14:02.067 INFO:tasks.rados.rados.0.smithi123.stdout:1632: writing smithi12313934-21 from 1606334 to 2270534 tid 2 2020-01-16T00:14:02.075 INFO:tasks.rados.rados.0.smithi123.stdout:1632: writing smithi12313934-21 from 3018232 to 3680832 tid 3 2020-01-16T00:14:02.076 INFO:tasks.rados.rados.0.smithi123.stdout:1633: write oid 43 current snap is 0 2020-01-16T00:14:02.076 INFO:tasks.rados.rados.0.smithi123.stdout:1633: seq_num 577 ranges {497639=545148,1548463=407858,2529302=564422,3528143=169496} 2020-01-16T00:14:02.082 INFO:tasks.rados.rados.0.smithi123.stdout:1633: writing smithi12313934-43 from 497639 to 1042787 tid 1 2020-01-16T00:14:02.088 INFO:tasks.rados.rados.0.smithi123.stdout:1633: writing smithi12313934-43 from 1548463 to 1956321 tid 2 2020-01-16T00:14:02.094 INFO:tasks.rados.rados.0.smithi123.stdout:1633: writing smithi12313934-43 from 2529302 to 3093724 tid 3 2020-01-16T00:14:02.097 INFO:tasks.rados.rados.0.smithi123.stdout:1633: writing smithi12313934-43 from 3528143 to 3697639 tid 4 2020-01-16T00:14:02.099 INFO:tasks.rados.rados.0.smithi123.stdout:1634: write oid 17 current snap is 0 2020-01-16T00:14:02.099 INFO:tasks.rados.rados.0.smithi123.stdout:1634: seq_num 578 ranges {514446=736750,1723298=667869,3073078=641368} 2020-01-16T00:14:02.104 INFO:tasks.rados.rados.0.smithi123.stdout:1634: writing smithi12313934-17 from 514446 to 1251196 tid 1 2020-01-16T00:14:02.111 INFO:tasks.rados.rados.0.smithi123.stdout:1634: writing smithi12313934-17 from 1723298 to 2391167 tid 2 2020-01-16T00:14:02.115 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:02.119 INFO:tasks.rados.rados.0.smithi123.stdout:1634: writing smithi12313934-17 from 3073078 to 3714446 tid 3 2020-01-16T00:14:02.119 INFO:tasks.rados.rados.0.smithi123.stdout:1635: read oid 15 snap -1 2020-01-16T00:14:02.119 INFO:tasks.rados.rados.0.smithi123.stdout:1635: expect (ObjNum 543 snap 0 seq_num 543) 2020-01-16T00:14:02.122 INFO:tasks.rados.rados.0.smithi123.stdout:1636: read oid 39 snap -1 2020-01-16T00:14:02.122 INFO:tasks.rados.rados.0.smithi123.stdout:1636: expect (ObjNum 535 snap 0 seq_num 535) 2020-01-16T00:14:02.123 INFO:tasks.rados.rados.0.smithi123.stdout:1637: write (excl) oid 11 current snap is 0 2020-01-16T00:14:02.123 INFO:tasks.rados.rados.0.smithi123.stdout:1637: seq_num 579 ranges {531253=411999,1381780=411527,2300501=586966,3376974=354279} 2020-01-16T00:14:02.125 INFO:tasks.rados.rados.0.smithi123.stdout:1637: writing smithi12313934-11 from 531253 to 943252 tid 1 2020-01-16T00:14:02.128 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-01-16T00:14:02.131 INFO:tasks.rados.rados.0.smithi123.stdout:1637: writing smithi12313934-11 from 1381780 to 1793307 tid 2 2020-01-16T00:14:02.137 INFO:tasks.rados.rados.0.smithi123.stdout:1637: writing smithi12313934-11 from 2300501 to 2887467 tid 3 2020-01-16T00:14:02.140 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:02.136+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:02.142 INFO:tasks.rados.rados.0.smithi123.stdout:1637: writing smithi12313934-11 from 3376974 to 3731253 tid 4 2020-01-16T00:14:02.142 INFO:tasks.rados.rados.0.smithi123.stdout:1638: read oid 37 snap -1 2020-01-16T00:14:02.143 INFO:tasks.rados.rados.0.smithi123.stdout:1638: expect (ObjNum 502 snap 0 seq_num 502) 2020-01-16T00:14:02.143 INFO:tasks.rados.rados.0.smithi123.stdout:1639: write oid 34 current snap is 0 2020-01-16T00:14:02.143 INFO:tasks.rados.rados.0.smithi123.stdout:1639: seq_num 580 ranges {548060=487248,1440262=671538,2844277=798238} 2020-01-16T00:14:02.149 INFO:tasks.rados.rados.0.smithi123.stdout:1639: writing smithi12313934-34 from 548060 to 1035308 tid 1 2020-01-16T00:14:02.155 INFO:tasks.rados.rados.0.smithi123.stdout:1639: writing smithi12313934-34 from 1440262 to 2111800 tid 2 2020-01-16T00:14:02.164 INFO:tasks.rados.rados.0.smithi123.stdout:1639: writing smithi12313934-34 from 2844277 to 3642515 tid 3 2020-01-16T00:14:02.164 INFO:tasks.rados.rados.0.smithi123.stdout:1640: write (excl) oid 16 current snap is 0 2020-01-16T00:14:02.164 INFO:tasks.rados.rados.0.smithi123.stdout:1640: seq_num 581 ranges {564867=562497,1898744=415196,2755347=725863} 2020-01-16T00:14:02.170 INFO:tasks.rados.rados.0.smithi123.stdout:1640: writing smithi12313934-16 from 564867 to 1127364 tid 1 2020-01-16T00:14:02.176 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:14:02.177 INFO:tasks.rados.rados.0.smithi123.stdout:1640: writing smithi12313934-16 from 1898744 to 2313940 tid 2 2020-01-16T00:14:02.185 INFO:tasks.rados.rados.0.smithi123.stdout:1640: writing smithi12313934-16 from 2755347 to 3481210 tid 3 2020-01-16T00:14:02.185 INFO:tasks.rados.rados.0.smithi123.stdout:1641: read oid 40 snap -1 2020-01-16T00:14:02.185 INFO:tasks.rados.rados.0.smithi123.stdout:1641: expect deleted 2020-01-16T00:14:02.185 INFO:tasks.rados.rados.0.smithi123.stdout:1642: read oid 38 snap -1 2020-01-16T00:14:02.185 INFO:tasks.rados.rados.0.smithi123.stdout:1642: expect (ObjNum 561 snap 0 seq_num 561) 2020-01-16T00:14:02.186 INFO:tasks.rados.rados.0.smithi123.stdout:1643: read oid 32 snap -1 2020-01-16T00:14:02.186 INFO:tasks.rados.rados.0.smithi123.stdout:1643: expect (ObjNum 513 snap 0 seq_num 513) 2020-01-16T00:14:02.186 INFO:tasks.rados.rados.0.smithi123.stdout:1644: write oid 5 current snap is 0 2020-01-16T00:14:02.186 INFO:tasks.rados.rados.0.smithi123.stdout:1644: seq_num 582 ranges {581674=637746,1840873=558854,3066417=537135} 2020-01-16T00:14:02.191 INFO:tasks.rados.rados.0.smithi123.stdout:1644: writing smithi12313934-5 from 581674 to 1219420 tid 1 2020-01-16T00:14:02.197 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:14:02.197 INFO:tasks.rados.rados.0.smithi123.stdout:1644: writing smithi12313934-5 from 1840873 to 2399727 tid 2 2020-01-16T00:14:02.208 INFO:tasks.rados.rados.0.smithi123.stdout:1644: writing smithi12313934-5 from 3066417 to 3603552 tid 3 2020-01-16T00:14:02.208 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:02.208 INFO:tasks.rados.rados.0.smithi123.stdout:1629: finishing write tid 2 to smithi12313934-33 2020-01-16T00:14:02.209 INFO:tasks.rados.rados.0.smithi123.stdout:1629: finishing write tid 3 to smithi12313934-33 2020-01-16T00:14:02.209 INFO:tasks.rados.rados.0.smithi123.stdout:1629: finishing write tid 5 to smithi12313934-33 2020-01-16T00:14:02.209 INFO:tasks.rados.rados.0.smithi123.stdout:1629: finishing write tid 6 to smithi12313934-33 2020-01-16T00:14:02.209 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 33 v 307 (ObjNum 574 snap 0 seq_num 574) dirty exists 2020-01-16T00:14:02.209 INFO:tasks.rados.rados.0.smithi123.stdout:1629: left oid 33 (ObjNum 574 snap 0 seq_num 574) 2020-01-16T00:14:02.209 INFO:tasks.rados.rados.0.smithi123.stdout:1630: finishing write tid 1 to smithi12313934-22 2020-01-16T00:14:02.209 INFO:tasks.rados.rados.0.smithi123.stdout:1630: finishing write tid 2 to smithi12313934-22 2020-01-16T00:14:02.209 INFO:tasks.rados.rados.0.smithi123.stdout:1629: done (15 left) 2020-01-16T00:14:02.210 INFO:tasks.rados.rados.0.smithi123.stdout:1645: read oid 19 snap -1 2020-01-16T00:14:02.210 INFO:tasks.rados.rados.0.smithi123.stdout:1645: expect (ObjNum 554 snap 0 seq_num 554) 2020-01-16T00:14:02.210 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:02.210 INFO:tasks.rados.rados.0.smithi123.stdout:1630: finishing write tid 3 to smithi12313934-22 2020-01-16T00:14:02.210 INFO:tasks.rados.rados.0.smithi123.stdout:1631: expect (ObjNum 550 snap 0 seq_num 550) 2020-01-16T00:14:02.218 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:14:02.229 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:02.239 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-01-16T00:14:02.242 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:14:02.242 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:02.238+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:02.262 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:14:02.288 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:14:02.309 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:14:02.310 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-01-16T00:14:02.311 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-01-16T00:14:02.311 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-01-16T00:14:02.311 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-01-16T00:14:02.312 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-01-16T00:14:02.312 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENXIO: problem getting command descriptions from osd.5 2020-01-16T00:14:02.312 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: problem getting command descriptions from osd.5 2020-01-16T00:14:02.312 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-01-16T00:14:02.312 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-01-16T00:14:02.320 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-01-16T00:14:02.321 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-01-16T00:14:02.339 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:02.339 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:14:02.340 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:14:02.340 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:14:02.340 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:02.343 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:02.340+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:02.348 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-01-16T00:14:02.411 INFO:tasks.rados.rados.0.smithi123.stdout:1630: finishing write tid 5 to smithi12313934-22 2020-01-16T00:14:02.412 INFO:tasks.rados.rados.0.smithi123.stdout:1630: finishing write tid 6 to smithi12313934-22 2020-01-16T00:14:02.412 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 22 v 37 (ObjNum 575 snap 0 seq_num 575) dirty exists 2020-01-16T00:14:02.412 INFO:tasks.rados.rados.0.smithi123.stdout:1630: left oid 22 (ObjNum 575 snap 0 seq_num 575) 2020-01-16T00:14:02.413 INFO:tasks.rados.rados.0.smithi123.stdout:1632: finishing write tid 1 to smithi12313934-21 2020-01-16T00:14:02.413 INFO:tasks.rados.rados.0.smithi123.stdout:1632: finishing write tid 2 to smithi12313934-21 2020-01-16T00:14:02.413 INFO:tasks.rados.rados.0.smithi123.stdout:1632: finishing write tid 3 to smithi12313934-21 2020-01-16T00:14:02.413 INFO:tasks.rados.rados.0.smithi123.stdout:1632: finishing write tid 5 to smithi12313934-21 2020-01-16T00:14:02.414 INFO:tasks.rados.rados.0.smithi123.stdout:1632: finishing write tid 6 to smithi12313934-21 2020-01-16T00:14:02.414 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 21 v 76 (ObjNum 576 snap 0 seq_num 576) dirty exists 2020-01-16T00:14:02.414 INFO:tasks.rados.rados.0.smithi123.stdout:1632: left oid 21 (ObjNum 576 snap 0 seq_num 576) 2020-01-16T00:14:02.414 INFO:tasks.rados.rados.0.smithi123.stdout:1633: finishing write tid 1 to smithi12313934-43 2020-01-16T00:14:02.414 INFO:tasks.rados.rados.0.smithi123.stdout:1633: finishing write tid 2 to smithi12313934-43 2020-01-16T00:14:02.415 INFO:tasks.rados.rados.0.smithi123.stdout:1633: finishing write tid 3 to smithi12313934-43 2020-01-16T00:14:02.415 INFO:tasks.rados.rados.0.smithi123.stdout:1633: finishing write tid 4 to smithi12313934-43 2020-01-16T00:14:02.415 INFO:tasks.rados.rados.0.smithi123.stdout:1633: finishing write tid 6 to smithi12313934-43 2020-01-16T00:14:02.415 INFO:tasks.rados.rados.0.smithi123.stdout:1633: finishing write tid 7 to smithi12313934-43 2020-01-16T00:14:02.416 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 43 v 264 (ObjNum 577 snap 0 seq_num 577) dirty exists 2020-01-16T00:14:02.416 INFO:tasks.rados.rados.0.smithi123.stdout:1633: left oid 43 (ObjNum 577 snap 0 seq_num 577) 2020-01-16T00:14:02.416 INFO:tasks.rados.rados.0.smithi123.stdout:1634: finishing write tid 1 to smithi12313934-17 2020-01-16T00:14:02.417 INFO:tasks.rados.rados.0.smithi123.stdout:1634: finishing write tid 2 to smithi12313934-17 2020-01-16T00:14:02.417 INFO:tasks.rados.rados.0.smithi123.stdout:1634: finishing write tid 3 to smithi12313934-17 2020-01-16T00:14:02.417 INFO:tasks.rados.rados.0.smithi123.stdout:1634: finishing write tid 5 to smithi12313934-17 2020-01-16T00:14:02.417 INFO:tasks.rados.rados.0.smithi123.stdout:1634: finishing write tid 6 to smithi12313934-17 2020-01-16T00:14:02.418 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 17 v 186 (ObjNum 578 snap 0 seq_num 578) dirty exists 2020-01-16T00:14:02.418 INFO:tasks.rados.rados.0.smithi123.stdout:1634: left oid 17 (ObjNum 578 snap 0 seq_num 578) 2020-01-16T00:14:02.418 INFO:tasks.rados.rados.0.smithi123.stdout:1630: done (15 left) 2020-01-16T00:14:02.418 INFO:tasks.rados.rados.0.smithi123.stdout:1631: done (14 left) 2020-01-16T00:14:02.419 INFO:tasks.rados.rados.0.smithi123.stdout:1632: done (13 left) 2020-01-16T00:14:02.419 INFO:tasks.rados.rados.0.smithi123.stdout:1633: done (12 left) 2020-01-16T00:14:02.419 INFO:tasks.rados.rados.0.smithi123.stdout:1634: done (11 left) 2020-01-16T00:14:02.419 INFO:tasks.rados.rados.0.smithi123.stdout:1646: delete oid 44 current snap is 0 2020-01-16T00:14:02.420 INFO:tasks.rados.rados.0.smithi123.stdout:1635: expect (ObjNum 543 snap 0 seq_num 543) 2020-01-16T00:14:02.444 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:02.442+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:02.497 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:02.537 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:14:02.537 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:14:02.537 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:14:02.537 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.537 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:923 3.8 3:113947f2:::smithi12313934-12:head [write 1564138~470341] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:14:02.537 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.869292+0000", 2020-01-16T00:14:02.538 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 84.626120962000002, 2020-01-16T00:14:02.538 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017661620999999999, 2020-01-16T00:14:02.538 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:02.538 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:02.538 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:02.538 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:02.538 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:02.538 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 923 2020-01-16T00:14:02.539 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.539 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:02.539 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.539 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:02.539 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869292+0000", 2020-01-16T00:14:02.539 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:02.539 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.539 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.540 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:02.540 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869292+0000", 2020-01-16T00:14:02.540 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:02.540 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.540 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.540 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:02.540 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869291+0000", 2020-01-16T00:14:02.541 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00049842899999999999 2020-01-16T00:14:02.541 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.541 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.541 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:02.541 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869789+0000", 2020-01-16T00:14:02.541 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1014999999999999e-05 2020-01-16T00:14:02.541 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.541 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.542 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:02.543 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869830+0000", 2020-01-16T00:14:02.543 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7880000000000007e-06 2020-01-16T00:14:02.545 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.545 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.545 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:02.546 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869838+0000", 2020-01-16T00:14:02.546 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8346999999999997e-05 2020-01-16T00:14:02.546 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.547 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.547 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.547 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869876+0000", 2020-01-16T00:14:02.547 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9726999999999999e-05 2020-01-16T00:14:02.547 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.547 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.548 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:02.548 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.869936+0000", 2020-01-16T00:14:02.548 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003434917 2020-01-16T00:14:02.549 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.549 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.549 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.549 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.873371+0000", 2020-01-16T00:14:02.549 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118808 2020-01-16T00:14:02.549 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.549 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.550 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:02.550 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.873490+0000", 2020-01-16T00:14:02.550 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036712910000000001 2020-01-16T00:14:02.550 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.550 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.550 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.550 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877161+0000", 2020-01-16T00:14:02.550 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9909999999999999e-05 2020-01-16T00:14:02.551 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.551 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.551 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:02.551 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.877241+0000", 2020-01-16T00:14:02.551 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031619970000000002 2020-01-16T00:14:02.551 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.551 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.551 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.552 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880403+0000", 2020-01-16T00:14:02.552 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013071099999999999 2020-01-16T00:14:02.552 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.552 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.552 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:02.552 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880534+0000", 2020-01-16T00:14:02.553 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00032823500000000002 2020-01-16T00:14:02.553 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.553 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.553 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 1,7", 2020-01-16T00:14:02.553 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.880862+0000", 2020-01-16T00:14:02.553 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00513862 2020-01-16T00:14:02.553 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.554 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.554 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:02.554 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886001+0000", 2020-01-16T00:14:02.554 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8869000000000001e-05 2020-01-16T00:14:02.554 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.554 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.554 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:02.555 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886030+0000", 2020-01-16T00:14:02.555 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00081600200000000003 2020-01-16T00:14:02.555 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.555 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.555 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:02.555 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886846+0000", 2020-01-16T00:14:02.555 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4095999999999998e-05 2020-01-16T00:14:02.555 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.556 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.556 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:02.556 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886900+0000", 2020-01-16T00:14:02.556 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.448e-05 2020-01-16T00:14:02.556 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.556 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.556 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:02.556 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.886954+0000", 2020-01-16T00:14:02.557 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017661620999999999 2020-01-16T00:14:02.557 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.557 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:02.557 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.557 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.557 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2226 3.7 3:e1be6d3b:::smithi12313934-24:head [setxattr _header (56) in=7b,truncate 2100908] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:14:02.558 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.980217+0000", 2020-01-16T00:14:02.558 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 56.515196297000003, 2020-01-16T00:14:02.558 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023113018999999999, 2020-01-16T00:14:02.558 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:02.558 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:02.558 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:02.558 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:02.558 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:02.559 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2226 2020-01-16T00:14:02.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.559 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:02.559 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.559 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:02.559 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980217+0000", 2020-01-16T00:14:02.559 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:02.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.560 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.560 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:02.560 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980217+0000", 2020-01-16T00:14:02.560 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:02.560 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.560 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.560 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:02.560 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980216+0000", 2020-01-16T00:14:02.561 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5049999999999998e-06 2020-01-16T00:14:02.561 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.561 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.561 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:02.561 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980222+0000", 2020-01-16T00:14:02.561 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5654999999999998e-05 2020-01-16T00:14:02.561 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.562 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.562 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:02.562 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980237+0000", 2020-01-16T00:14:02.562 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9740000000000004e-06 2020-01-16T00:14:02.562 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.562 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.562 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:02.562 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980241+0000", 2020-01-16T00:14:02.563 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002947776 2020-01-16T00:14:02.574 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.574 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.574 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.574 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.983189+0000", 2020-01-16T00:14:02.574 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0027099400000000001 2020-01-16T00:14:02.574 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.574 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.575 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:02.575 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985899+0000", 2020-01-16T00:14:02.575 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014275589999999999 2020-01-16T00:14:02.575 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.575 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.575 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.575 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.000175+0000", 2020-01-16T00:14:02.576 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011211 2020-01-16T00:14:02.576 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.576 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.576 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:02.576 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.000287+0000", 2020-01-16T00:14:02.576 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000229886 2020-01-16T00:14:02.576 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.576 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.577 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,7", 2020-01-16T00:14:02.577 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.000517+0000", 2020-01-16T00:14:02.577 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00095715399999999999 2020-01-16T00:14:02.577 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.577 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.577 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:02.577 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.001474+0000", 2020-01-16T00:14:02.578 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001341512 2020-01-16T00:14:02.578 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.578 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.578 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:02.578 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.002815+0000", 2020-01-16T00:14:02.578 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035789099999999999 2020-01-16T00:14:02.578 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.578 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.579 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:02.579 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003173+0000", 2020-01-16T00:14:02.579 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8571000000000005e-05 2020-01-16T00:14:02.579 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.579 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.579 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:02.579 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003242+0000", 2020-01-16T00:14:02.580 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8293000000000003e-05 2020-01-16T00:14:02.580 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.580 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.580 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:02.580 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003330+0000", 2020-01-16T00:14:02.580 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023113018999999999 2020-01-16T00:14:02.580 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.580 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:02.581 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.581 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.581 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.581 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2227 3.7 3:e1be6d3b:::smithi12313934-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e35)", 2020-01-16T00:14:02.581 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:14:02.581 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 56.515149043999998, 2020-01-16T00:14:02.581 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023303356000000001, 2020-01-16T00:14:02.581 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:02.582 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:02.582 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:02.582 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:02.582 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:02.582 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2227 2020-01-16T00:14:02.582 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.582 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:02.583 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.583 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:02.583 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:14:02.583 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:02.583 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.583 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.583 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:02.583 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:14:02.584 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:02.584 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.584 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.584 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:02.584 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:14:02.584 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3079999999999999e-06 2020-01-16T00:14:02.584 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.584 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.585 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:02.585 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980267+0000", 2020-01-16T00:14:02.585 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9290000000000001e-06 2020-01-16T00:14:02.585 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.585 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.585 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:02.585 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980274+0000", 2020-01-16T00:14:02.585 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7400000000000002e-06 2020-01-16T00:14:02.586 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.586 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.586 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:02.586 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980278+0000", 2020-01-16T00:14:02.586 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056624300000000004 2020-01-16T00:14:02.586 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.586 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.586 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.587 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985940+0000", 2020-01-16T00:14:02.587 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4261000000000002e-05 2020-01-16T00:14:02.587 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.587 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.587 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:02.587 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985985+0000", 2020-01-16T00:14:02.587 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015087676 2020-01-16T00:14:02.588 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.588 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.588 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.588 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.001072+0000", 2020-01-16T00:14:02.588 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011960600000000001 2020-01-16T00:14:02.588 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.588 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.588 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:02.589 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.001192+0000", 2020-01-16T00:14:02.589 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021718309999999999 2020-01-16T00:14:02.589 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.589 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.589 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.589 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003364+0000", 2020-01-16T00:14:02.589 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000102855 2020-01-16T00:14:02.589 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.590 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.590 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:02.590 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003467+0000", 2020-01-16T00:14:02.590 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010122499999999999 2020-01-16T00:14:02.590 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.590 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.590 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:02.591 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003568+0000", 2020-01-16T00:14:02.591 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023303356000000001 2020-01-16T00:14:02.591 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.591 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:02.591 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.591 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.591 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.591 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2233 3.2 3:4f8fe446:::smithi12313934-48:head [setxattr _header (56) in=7b,truncate 2117715] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:14:02.592 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.993103+0000", 2020-01-16T00:14:02.592 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 56.502310821999998, 2020-01-16T00:14:02.592 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017951763999999999, 2020-01-16T00:14:02.592 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:02.592 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:02.592 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:02.592 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:02.592 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:02.593 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2233 2020-01-16T00:14:02.593 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.593 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:02.593 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.593 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:02.593 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993103+0000", 2020-01-16T00:14:02.593 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:02.593 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.594 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.594 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:02.594 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993103+0000", 2020-01-16T00:14:02.594 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:02.594 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.594 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.594 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:02.594 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993102+0000", 2020-01-16T00:14:02.595 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9369999999999993e-06 2020-01-16T00:14:02.595 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.595 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.595 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:02.595 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993111+0000", 2020-01-16T00:14:02.595 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3340000000000002e-06 2020-01-16T00:14:02.595 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.596 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.596 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:02.596 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993119+0000", 2020-01-16T00:14:02.596 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1769999999999997e-06 2020-01-16T00:14:02.596 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.596 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.596 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:02.596 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993124+0000", 2020-01-16T00:14:02.597 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001601304 2020-01-16T00:14:02.597 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.597 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.597 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.597 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.994725+0000", 2020-01-16T00:14:02.597 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048777250000000003 2020-01-16T00:14:02.597 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.597 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.598 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:02.598 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999603+0000", 2020-01-16T00:14:02.598 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0099793880000000005 2020-01-16T00:14:02.598 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.598 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.598 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.598 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.009583+0000", 2020-01-16T00:14:02.598 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000108207 2020-01-16T00:14:02.599 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.599 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.599 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:02.599 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.009691+0000", 2020-01-16T00:14:02.599 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012862999999999999 2020-01-16T00:14:02.599 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.599 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.599 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:14:02.600 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.009819+0000", 2020-01-16T00:14:02.600 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00077947599999999998 2020-01-16T00:14:02.600 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.600 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.600 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:02.600 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010599+0000", 2020-01-16T00:14:02.600 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000221282 2020-01-16T00:14:02.600 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.601 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.601 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:02.601 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010820+0000", 2020-01-16T00:14:02.601 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7021999999999995e-05 2020-01-16T00:14:02.601 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.601 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.601 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:02.602 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010907+0000", 2020-01-16T00:14:02.602 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9815999999999997e-05 2020-01-16T00:14:02.602 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.602 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.602 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:02.602 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010977+0000", 2020-01-16T00:14:02.602 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7453999999999994e-05 2020-01-16T00:14:02.602 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.603 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.603 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:02.603 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.011054+0000", 2020-01-16T00:14:02.603 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017951763999999999 2020-01-16T00:14:02.603 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.603 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:02.603 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.603 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.604 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.604 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2234 3.2 3:4f8fe446:::smithi12313934-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e35)", 2020-01-16T00:14:02.604 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.993135+0000", 2020-01-16T00:14:02.604 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 56.502278369000003, 2020-01-16T00:14:02.604 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018067916999999999, 2020-01-16T00:14:02.604 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:02.604 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:02.605 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:02.605 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:02.605 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:02.605 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2234 2020-01-16T00:14:02.605 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.605 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:02.605 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.605 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:02.606 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993135+0000", 2020-01-16T00:14:02.606 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:02.606 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.606 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.606 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:02.606 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993135+0000", 2020-01-16T00:14:02.606 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:02.606 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.607 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.607 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:02.607 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993134+0000", 2020-01-16T00:14:02.607 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5630000000000002e-06 2020-01-16T00:14:02.607 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.607 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.607 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:02.607 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993139+0000", 2020-01-16T00:14:02.608 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1730000000000003e-06 2020-01-16T00:14:02.608 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.608 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.608 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:02.608 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993144+0000", 2020-01-16T00:14:02.608 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.937e-06 2020-01-16T00:14:02.608 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.608 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.609 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:02.609 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993147+0000", 2020-01-16T00:14:02.609 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0065164000000000003 2020-01-16T00:14:02.609 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.609 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.609 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.609 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999663+0000", 2020-01-16T00:14:02.610 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6215000000000002e-05 2020-01-16T00:14:02.610 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.610 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.610 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:02.610 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999729+0000", 2020-01-16T00:14:02.610 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010475887999999999 2020-01-16T00:14:02.610 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.610 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.611 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.611 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010205+0000", 2020-01-16T00:14:02.611 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119471 2020-01-16T00:14:02.611 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.611 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.611 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:02.611 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.010325+0000", 2020-01-16T00:14:02.611 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00075498799999999995 2020-01-16T00:14:02.612 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.612 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.612 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.612 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.011080+0000", 2020-01-16T00:14:02.612 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7904e-05 2020-01-16T00:14:02.612 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.612 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.612 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:02.613 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.011138+0000", 2020-01-16T00:14:02.613 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5273999999999999e-05 2020-01-16T00:14:02.613 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.613 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.613 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:02.613 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.011203+0000", 2020-01-16T00:14:02.613 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018067916999999999 2020-01-16T00:14:02.614 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.614 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:02.614 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.614 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.614 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.614 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2324 3.7 3:e1be6d3b:::smithi12313934-24:head [setxattr _header (56) in=7b,truncate 2336206] snapc 0=[] ondisk+write+known_if_redirected e36)", 2020-01-16T00:14:02.614 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.550846+0000", 2020-01-16T00:14:02.614 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 54.944567782999997, 2020-01-16T00:14:02.615 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018364742, 2020-01-16T00:14:02.615 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:02.615 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:02.615 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:02.615 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:02.615 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:02.615 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2324 2020-01-16T00:14:02.615 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.616 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:02.616 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.616 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:02.616 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550846+0000", 2020-01-16T00:14:02.616 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:02.616 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.617 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.617 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:02.617 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550846+0000", 2020-01-16T00:14:02.617 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:02.617 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.617 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.617 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:02.617 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550845+0000", 2020-01-16T00:14:02.618 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1829999999999999e-06 2020-01-16T00:14:02.618 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.618 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.618 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:02.618 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550851+0000", 2020-01-16T00:14:02.618 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1925e-05 2020-01-16T00:14:02.618 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.619 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.619 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:02.619 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550863+0000", 2020-01-16T00:14:02.619 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1139999999999997e-06 2020-01-16T00:14:02.619 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.619 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.619 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:02.619 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550868+0000", 2020-01-16T00:14:02.620 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001388423 2020-01-16T00:14:02.620 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.620 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.620 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.620 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.552256+0000", 2020-01-16T00:14:02.620 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0034246540000000001 2020-01-16T00:14:02.620 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.620 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.621 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:02.621 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555681+0000", 2020-01-16T00:14:02.621 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011882762999999999 2020-01-16T00:14:02.621 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.621 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.621 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.621 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.567564+0000", 2020-01-16T00:14:02.621 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015010800000000001 2020-01-16T00:14:02.622 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.622 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.622 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:02.622 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.567714+0000", 2020-01-16T00:14:02.622 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011527999999999999 2020-01-16T00:14:02.622 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.622 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.623 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,7", 2020-01-16T00:14:02.623 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.567829+0000", 2020-01-16T00:14:02.623 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00039888499999999997 2020-01-16T00:14:02.623 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.623 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.623 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:02.623 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568228+0000", 2020-01-16T00:14:02.623 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00054254599999999998 2020-01-16T00:14:02.624 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.624 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.624 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:02.624 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568771+0000", 2020-01-16T00:14:02.624 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035206400000000002 2020-01-16T00:14:02.624 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.624 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.624 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:02.625 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569123+0000", 2020-01-16T00:14:02.625 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1262e-05 2020-01-16T00:14:02.625 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.625 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.625 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:02.625 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569154+0000", 2020-01-16T00:14:02.625 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6499000000000002e-05 2020-01-16T00:14:02.625 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.626 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.626 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:02.626 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569210+0000", 2020-01-16T00:14:02.626 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018364742 2020-01-16T00:14:02.626 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.626 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:02.626 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.626 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.627 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.627 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2325 3.7 3:e1be6d3b:::smithi12313934-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e36)", 2020-01-16T00:14:02.627 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.550877+0000", 2020-01-16T00:14:02.627 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 54.944536716999998, 2020-01-16T00:14:02.627 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018565154, 2020-01-16T00:14:02.627 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:02.627 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:02.628 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:02.628 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:02.628 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:02.628 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2325 2020-01-16T00:14:02.628 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.628 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:02.628 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.628 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:02.629 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550877+0000", 2020-01-16T00:14:02.629 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:02.629 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.629 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.629 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:02.629 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550877+0000", 2020-01-16T00:14:02.629 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:02.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.630 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.630 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:02.630 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550876+0000", 2020-01-16T00:14:02.630 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1080000000000003e-06 2020-01-16T00:14:02.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.630 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.630 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:02.631 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550880+0000", 2020-01-16T00:14:02.631 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3269999999999997e-06 2020-01-16T00:14:02.631 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.631 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.631 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:02.631 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550886+0000", 2020-01-16T00:14:02.631 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.386e-06 2020-01-16T00:14:02.631 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.632 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.632 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:02.632 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550891+0000", 2020-01-16T00:14:02.632 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048231410000000004 2020-01-16T00:14:02.632 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.632 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.632 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.632 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555714+0000", 2020-01-16T00:14:02.633 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0071999999999999e-05 2020-01-16T00:14:02.633 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.633 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.633 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:02.633 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555744+0000", 2020-01-16T00:14:02.633 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012329005000000001 2020-01-16T00:14:02.633 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.634 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.634 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.634 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568073+0000", 2020-01-16T00:14:02.634 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9087999999999997e-05 2020-01-16T00:14:02.634 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.634 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.634 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:02.634 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568172+0000", 2020-01-16T00:14:02.635 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010718220000000001 2020-01-16T00:14:02.635 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.635 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.635 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.635 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569244+0000", 2020-01-16T00:14:02.635 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8199e-05 2020-01-16T00:14:02.635 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.635 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.636 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:02.636 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569342+0000", 2020-01-16T00:14:02.636 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9708000000000001e-05 2020-01-16T00:14:02.636 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.636 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.636 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:02.636 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569442+0000", 2020-01-16T00:14:02.637 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018565154 2020-01-16T00:14:02.637 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.637 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:02.637 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.637 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.637 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.637 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2800 3.2 3:42b7f20f:::smithi12313934-42:head [write 1395221~583781] snapc 0=[] ondisk+write+known_if_redirected e41)", 2020-01-16T00:14:02.637 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:17.256182+0000", 2020-01-16T00:14:02.638 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 45.239231177000001, 2020-01-16T00:14:02.638 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018509650999999998, 2020-01-16T00:14:02.638 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:02.638 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:02.638 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:02.638 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:02.638 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:02.638 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2800 2020-01-16T00:14:02.639 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.639 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:02.639 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.639 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:02.639 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256182+0000", 2020-01-16T00:14:02.639 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:02.639 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.639 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.640 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:02.640 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256182+0000", 2020-01-16T00:14:02.640 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:02.640 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.640 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.640 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:02.640 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256180+0000", 2020-01-16T00:14:02.641 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020514799999999999 2020-01-16T00:14:02.641 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.641 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.641 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:02.641 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256385+0000", 2020-01-16T00:14:02.641 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0288e-05 2020-01-16T00:14:02.641 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.641 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.642 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:02.642 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256406+0000", 2020-01-16T00:14:02.642 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7160000000000003e-06 2020-01-16T00:14:02.642 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.642 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.642 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:02.642 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256413+0000", 2020-01-16T00:14:02.642 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5293999999999998e-05 2020-01-16T00:14:02.643 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.643 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.643 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.643 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256449+0000", 2020-01-16T00:14:02.643 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001297737 2020-01-16T00:14:02.644 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.644 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.645 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:02.645 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.257746+0000", 2020-01-16T00:14:02.645 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0079866529999999998 2020-01-16T00:14:02.645 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.645 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.645 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.645 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.265733+0000", 2020-01-16T00:14:02.645 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4028000000000004e-05 2020-01-16T00:14:02.646 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.646 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.646 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:02.646 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.265797+0000", 2020-01-16T00:14:02.646 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011338899999999999 2020-01-16T00:14:02.646 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.646 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.647 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:02.647 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.265910+0000", 2020-01-16T00:14:02.647 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003372496 2020-01-16T00:14:02.647 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.647 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.647 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:02.648 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.269283+0000", 2020-01-16T00:14:02.648 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0053107959999999996 2020-01-16T00:14:02.648 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.648 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.648 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:02.648 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.274594+0000", 2020-01-16T00:14:02.648 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8000999999999998e-05 2020-01-16T00:14:02.649 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.649 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.649 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:02.649 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.274632+0000", 2020-01-16T00:14:02.649 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0265e-05 2020-01-16T00:14:02.649 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.649 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.649 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:02.650 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.274692+0000", 2020-01-16T00:14:02.650 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018509650999999998 2020-01-16T00:14:02.650 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.650 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:02.650 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.650 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.650 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.650 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2971 3.c 3:3025546b:::smithi12313934-15:head [write 1424565~609148] snapc 0=[] ondisk+write+known_if_redirected e42)", 2020-01-16T00:14:02.651 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:21.839711+0000", 2020-01-16T00:14:02.651 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 40.655702333999997, 2020-01-16T00:14:02.651 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017534376000000001, 2020-01-16T00:14:02.651 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:02.651 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:02.651 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:02.651 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:02.652 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:02.652 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2971 2020-01-16T00:14:02.652 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.652 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:02.652 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.652 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:02.652 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839711+0000", 2020-01-16T00:14:02.652 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:02.653 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.653 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.653 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:02.653 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839711+0000", 2020-01-16T00:14:02.653 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:02.653 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.653 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.654 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:02.654 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839710+0000", 2020-01-16T00:14:02.654 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017658600000000001 2020-01-16T00:14:02.654 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.654 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.654 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:02.654 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839886+0000", 2020-01-16T00:14:02.654 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0492000000000001e-05 2020-01-16T00:14:02.655 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.655 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.655 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:02.655 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839897+0000", 2020-01-16T00:14:02.655 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.019e-06 2020-01-16T00:14:02.655 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.655 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.655 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:02.656 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839901+0000", 2020-01-16T00:14:02.656 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5572000000000001e-05 2020-01-16T00:14:02.656 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.656 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.656 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.656 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839976+0000", 2020-01-16T00:14:02.656 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034616099999999999 2020-01-16T00:14:02.656 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.657 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.657 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:02.657 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.840323+0000", 2020-01-16T00:14:02.657 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.01246249 2020-01-16T00:14:02.657 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.657 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.657 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.657 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.852785+0000", 2020-01-16T00:14:02.658 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014474799999999999 2020-01-16T00:14:02.658 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.658 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.658 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:02.658 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.852930+0000", 2020-01-16T00:14:02.658 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000213907 2020-01-16T00:14:02.658 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.659 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.659 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:02.659 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.853144+0000", 2020-01-16T00:14:02.659 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0028426979999999998 2020-01-16T00:14:02.659 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.659 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.659 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:02.659 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.855986+0000", 2020-01-16T00:14:02.660 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010694280000000001 2020-01-16T00:14:02.660 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.660 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.660 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:02.660 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.857056+0000", 2020-01-16T00:14:02.660 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8935000000000003e-05 2020-01-16T00:14:02.660 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.660 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.661 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:02.661 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.857125+0000", 2020-01-16T00:14:02.661 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000120721 2020-01-16T00:14:02.661 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.661 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.661 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:02.661 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.857245+0000", 2020-01-16T00:14:02.662 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017534376000000001 2020-01-16T00:14:02.662 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.662 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:02.662 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.662 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.662 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.662 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2972 3.c 3:3025546b:::smithi12313934-15:head [setxattr _header (56) in=7b,truncate 2033713] snapc 0=[] ondisk+write+known_if_redirected e42)", 2020-01-16T00:14:02.662 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:21.839923+0000", 2020-01-16T00:14:02.663 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 40.655490069999999, 2020-01-16T00:14:02.663 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018367108, 2020-01-16T00:14:02.663 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:02.663 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:02.663 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:02.663 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:02.663 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:02.663 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2972 2020-01-16T00:14:02.664 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.664 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:02.664 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.664 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:02.664 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839923+0000", 2020-01-16T00:14:02.664 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:02.664 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.664 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.665 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:02.665 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839923+0000", 2020-01-16T00:14:02.665 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:02.665 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.665 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.665 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:02.665 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839923+0000", 2020-01-16T00:14:02.666 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0629999999999999e-06 2020-01-16T00:14:02.666 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.666 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.666 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:02.666 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839927+0000", 2020-01-16T00:14:02.666 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1980000000000003e-06 2020-01-16T00:14:02.666 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.666 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.667 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:02.667 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839931+0000", 2020-01-16T00:14:02.667 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5979999999999999e-06 2020-01-16T00:14:02.667 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.667 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.667 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:02.667 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839934+0000", 2020-01-16T00:14:02.667 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00042258500000000001 2020-01-16T00:14:02.668 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.668 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.668 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.668 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.840356+0000", 2020-01-16T00:14:02.668 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4919000000000002e-05 2020-01-16T00:14:02.668 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.668 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.668 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:02.669 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.840391+0000", 2020-01-16T00:14:02.670 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013783103 2020-01-16T00:14:02.670 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.670 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.670 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.670 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.854174+0000", 2020-01-16T00:14:02.670 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3455999999999995e-05 2020-01-16T00:14:02.671 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.671 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.671 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:02.671 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.854258+0000", 2020-01-16T00:14:02.671 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011060700000000001 2020-01-16T00:14:02.671 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.671 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.671 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:02.672 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.854368+0000", 2020-01-16T00:14:02.672 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020642500000000001 2020-01-16T00:14:02.672 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.672 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.672 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:02.672 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.856433+0000", 2020-01-16T00:14:02.673 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0016284820000000001 2020-01-16T00:14:02.673 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.673 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.673 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:02.673 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858061+0000", 2020-01-16T00:14:02.673 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0663999999999998e-05 2020-01-16T00:14:02.674 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.674 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.674 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:02.674 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858152+0000", 2020-01-16T00:14:02.674 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013882900000000001 2020-01-16T00:14:02.674 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.674 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.674 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:02.675 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858290+0000", 2020-01-16T00:14:02.675 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018367108 2020-01-16T00:14:02.675 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.675 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:02.675 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.675 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.675 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.676 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2973 3.c 3:3025546b:::smithi12313934-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e42)", 2020-01-16T00:14:02.676 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:21.839961+0000", 2020-01-16T00:14:02.676 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 40.655452576000002, 2020-01-16T00:14:02.676 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018691670000000001, 2020-01-16T00:14:02.676 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:02.676 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:02.676 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:02.677 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:02.677 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:02.677 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2973 2020-01-16T00:14:02.677 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.677 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:02.678 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.678 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:02.678 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839961+0000", 2020-01-16T00:14:02.678 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:02.678 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.678 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.678 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:02.679 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839961+0000", 2020-01-16T00:14:02.680 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:02.680 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.680 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.680 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:02.680 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839960+0000", 2020-01-16T00:14:02.680 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4060000000000001e-06 2020-01-16T00:14:02.681 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.681 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.681 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:02.681 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839964+0000", 2020-01-16T00:14:02.681 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5412000000000001e-05 2020-01-16T00:14:02.681 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.681 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.681 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:02.682 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839979+0000", 2020-01-16T00:14:02.682 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.444e-06 2020-01-16T00:14:02.682 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.682 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.682 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:02.682 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839983+0000", 2020-01-16T00:14:02.682 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000430028 2020-01-16T00:14:02.683 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.683 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.683 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.683 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.840413+0000", 2020-01-16T00:14:02.683 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9270999999999997e-05 2020-01-16T00:14:02.683 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.683 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.683 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:02.684 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.840452+0000", 2020-01-16T00:14:02.684 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014853975 2020-01-16T00:14:02.684 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.684 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.684 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.684 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.855306+0000", 2020-01-16T00:14:02.684 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011098400000000001 2020-01-16T00:14:02.684 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.685 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.685 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:02.685 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.855417+0000", 2020-01-16T00:14:02.685 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029108060000000002 2020-01-16T00:14:02.685 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.685 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.685 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.685 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858328+0000", 2020-01-16T00:14:02.686 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000144697 2020-01-16T00:14:02.686 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.686 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.686 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:02.686 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858472+0000", 2020-01-16T00:14:02.686 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018011899999999999 2020-01-16T00:14:02.686 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.687 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.687 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:02.687 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858653+0000", 2020-01-16T00:14:02.687 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018691670000000001 2020-01-16T00:14:02.687 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.687 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:02.687 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.687 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.688 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.688 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3295 3.2 3:42b7f20f:::smithi12313934-42:head [write 1982691~499457] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:14:02.688 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:30.744201+0000", 2020-01-16T00:14:02.688 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 31.751212299999999, 2020-01-16T00:14:02.688 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018346738000000001, 2020-01-16T00:14:02.688 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:02.688 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:02.689 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:02.689 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:02.689 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:02.689 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3295 2020-01-16T00:14:02.689 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.689 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:02.689 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.689 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:02.690 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744201+0000", 2020-01-16T00:14:02.690 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:02.690 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.690 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.690 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:02.690 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744201+0000", 2020-01-16T00:14:02.690 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:02.690 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.691 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.691 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:02.691 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744198+0000", 2020-01-16T00:14:02.691 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00030375699999999999 2020-01-16T00:14:02.691 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.691 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.691 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:02.691 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744502+0000", 2020-01-16T00:14:02.692 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0121e-05 2020-01-16T00:14:02.692 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.692 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.692 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:02.692 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744512+0000", 2020-01-16T00:14:02.692 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2293000000000001e-05 2020-01-16T00:14:02.692 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.692 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.693 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:02.693 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744524+0000", 2020-01-16T00:14:02.693 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000274837 2020-01-16T00:14:02.693 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.693 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.693 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.693 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744799+0000", 2020-01-16T00:14:02.694 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036951879999999999 2020-01-16T00:14:02.694 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.694 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.694 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:02.694 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748494+0000", 2020-01-16T00:14:02.694 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010091493 2020-01-16T00:14:02.694 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.694 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.695 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.695 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.758586+0000", 2020-01-16T00:14:02.695 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011674400000000001 2020-01-16T00:14:02.695 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.695 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.695 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:02.695 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.758703+0000", 2020-01-16T00:14:02.695 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021412699999999999 2020-01-16T00:14:02.696 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.696 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.696 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:02.696 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.758917+0000", 2020-01-16T00:14:02.696 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0022519340000000001 2020-01-16T00:14:02.696 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.696 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.696 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:02.697 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.761169+0000", 2020-01-16T00:14:02.697 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001240891 2020-01-16T00:14:02.697 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.697 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.697 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:02.697 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.762410+0000", 2020-01-16T00:14:02.697 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6942e-05 2020-01-16T00:14:02.698 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.698 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.698 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:02.698 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.762447+0000", 2020-01-16T00:14:02.698 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000101358 2020-01-16T00:14:02.698 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.698 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.699 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:02.700 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.762548+0000", 2020-01-16T00:14:02.700 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018346738000000001 2020-01-16T00:14:02.700 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.700 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:02.700 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.700 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.701 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.701 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3296 3.2 3:42b7f20f:::smithi12313934-42:head [setxattr _header (56) in=7b,truncate 2739607] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:14:02.701 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:30.744532+0000", 2020-01-16T00:14:02.702 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 31.750881567, 2020-01-16T00:14:02.702 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018472374, 2020-01-16T00:14:02.702 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:02.702 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:02.702 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:02.702 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:02.702 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:02.703 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3296 2020-01-16T00:14:02.703 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.703 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:02.703 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.703 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:02.703 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744532+0000", 2020-01-16T00:14:02.703 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:02.703 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.704 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.704 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:02.704 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744532+0000", 2020-01-16T00:14:02.704 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:02.704 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.704 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.704 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:02.704 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744531+0000", 2020-01-16T00:14:02.705 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7059999999999993e-06 2020-01-16T00:14:02.705 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.705 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.705 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:02.705 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744540+0000", 2020-01-16T00:14:02.705 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8589999999999998e-06 2020-01-16T00:14:02.705 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.705 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.706 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:02.706 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744544+0000", 2020-01-16T00:14:02.706 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7609999999999999e-06 2020-01-16T00:14:02.706 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.706 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.706 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:02.706 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744546+0000", 2020-01-16T00:14:02.707 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039933670000000003 2020-01-16T00:14:02.707 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.707 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.707 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.707 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748540+0000", 2020-01-16T00:14:02.707 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000110281 2020-01-16T00:14:02.707 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.707 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.708 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:02.708 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748650+0000", 2020-01-16T00:14:02.708 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.01160457 2020-01-16T00:14:02.708 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.708 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.708 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.708 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.760255+0000", 2020-01-16T00:14:02.709 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000124714 2020-01-16T00:14:02.709 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.709 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.709 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:02.709 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.760379+0000", 2020-01-16T00:14:02.709 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022099700000000001 2020-01-16T00:14:02.710 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.710 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.710 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:02.710 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.760600+0000", 2020-01-16T00:14:02.710 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00057843300000000005 2020-01-16T00:14:02.710 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.711 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.711 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:02.711 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.761179+0000", 2020-01-16T00:14:02.711 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00161477 2020-01-16T00:14:02.711 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.711 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.712 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:02.712 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.762794+0000", 2020-01-16T00:14:02.712 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4408999999999999e-05 2020-01-16T00:14:02.712 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.712 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.712 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:02.713 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.762858+0000", 2020-01-16T00:14:02.713 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014634199999999999 2020-01-16T00:14:02.713 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.713 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.713 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:02.713 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.763004+0000", 2020-01-16T00:14:02.714 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018472374 2020-01-16T00:14:02.714 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.714 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:02.714 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.714 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.714 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.714 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3297 3.2 3:42b7f20f:::smithi12313934-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e45)", 2020-01-16T00:14:02.714 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:30.744551+0000", 2020-01-16T00:14:02.715 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 31.750862597000001, 2020-01-16T00:14:02.715 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018660209000000001, 2020-01-16T00:14:02.715 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:02.715 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:02.715 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:02.715 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:02.715 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:02.715 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3297 2020-01-16T00:14:02.716 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.716 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:02.716 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.716 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:02.716 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744551+0000", 2020-01-16T00:14:02.716 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:02.716 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.717 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.717 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:02.717 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744551+0000", 2020-01-16T00:14:02.717 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:02.717 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.717 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.717 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:02.717 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744550+0000", 2020-01-16T00:14:02.718 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5279999999999999e-06 2020-01-16T00:14:02.718 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.718 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.718 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:02.718 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744554+0000", 2020-01-16T00:14:02.718 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8420000000000001e-06 2020-01-16T00:14:02.719 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.719 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.719 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:02.719 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744559+0000", 2020-01-16T00:14:02.719 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.965e-06 2020-01-16T00:14:02.719 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.719 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.719 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:02.720 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744562+0000", 2020-01-16T00:14:02.720 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041215380000000001 2020-01-16T00:14:02.720 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.720 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.720 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.720 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748684+0000", 2020-01-16T00:14:02.720 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8165999999999997e-05 2020-01-16T00:14:02.720 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.721 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.721 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:02.721 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748762+0000", 2020-01-16T00:14:02.721 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012136152000000001 2020-01-16T00:14:02.721 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.721 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.721 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.721 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.760898+0000", 2020-01-16T00:14:02.722 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000114833 2020-01-16T00:14:02.722 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.722 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.722 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:02.722 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.761013+0000", 2020-01-16T00:14:02.722 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002017015 2020-01-16T00:14:02.722 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.723 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.723 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.723 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.763030+0000", 2020-01-16T00:14:02.723 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5741000000000006e-05 2020-01-16T00:14:02.723 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.723 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.723 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:02.723 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.763096+0000", 2020-01-16T00:14:02.724 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011519900000000001 2020-01-16T00:14:02.724 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.724 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.724 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:02.724 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.763211+0000", 2020-01-16T00:14:02.724 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018660209000000001 2020-01-16T00:14:02.724 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.725 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:02.725 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.725 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.726 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.726 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3481 3.2 3:4f8fe446:::smithi12313934-48:head [setxattr _header (56) in=7b,truncate 3193396] snapc 0=[] ondisk+write+known_if_redirected e47)", 2020-01-16T00:14:02.726 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:36.349654+0000", 2020-01-16T00:14:02.726 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 26.145759143999999, 2020-01-16T00:14:02.726 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017529643000000001, 2020-01-16T00:14:02.726 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:02.726 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:02.727 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:02.727 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:02.727 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:02.727 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3481 2020-01-16T00:14:02.727 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.727 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:02.727 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.727 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:02.728 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349654+0000", 2020-01-16T00:14:02.728 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:02.728 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.728 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.728 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:02.728 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349654+0000", 2020-01-16T00:14:02.728 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:02.728 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.729 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.729 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:02.729 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349654+0000", 2020-01-16T00:14:02.729 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2519999999999996e-06 2020-01-16T00:14:02.729 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.729 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.730 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:02.730 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349660+0000", 2020-01-16T00:14:02.730 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2789999999999999e-06 2020-01-16T00:14:02.730 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.731 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.731 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:02.731 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349663+0000", 2020-01-16T00:14:02.731 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2479999999999999e-06 2020-01-16T00:14:02.731 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.731 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.731 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:02.731 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349666+0000", 2020-01-16T00:14:02.732 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0026359180000000001 2020-01-16T00:14:02.732 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.732 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.732 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.732 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.352301+0000", 2020-01-16T00:14:02.732 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0035532799999999998 2020-01-16T00:14:02.732 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.732 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.733 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:02.733 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.355855+0000", 2020-01-16T00:14:02.733 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0099904050000000008 2020-01-16T00:14:02.733 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.733 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.733 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.733 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.365845+0000", 2020-01-16T00:14:02.734 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012439300000000001 2020-01-16T00:14:02.734 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.734 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.734 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:02.734 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.365970+0000", 2020-01-16T00:14:02.734 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00026105699999999998 2020-01-16T00:14:02.734 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.734 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.735 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:02.735 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.366231+0000", 2020-01-16T00:14:02.735 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00056610199999999997 2020-01-16T00:14:02.735 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.735 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.735 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:02.735 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.366797+0000", 2020-01-16T00:14:02.735 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000171996 2020-01-16T00:14:02.736 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.736 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.736 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:02.736 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.366969+0000", 2020-01-16T00:14:02.736 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5282000000000003e-05 2020-01-16T00:14:02.736 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.736 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.737 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:02.737 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.367034+0000", 2020-01-16T00:14:02.737 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014998100000000001 2020-01-16T00:14:02.737 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.737 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.737 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:02.737 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.367184+0000", 2020-01-16T00:14:02.737 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017529643000000001 2020-01-16T00:14:02.738 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.738 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:02.738 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.738 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.738 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.739 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3482 3.2 3:4f8fe446:::smithi12313934-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e47)", 2020-01-16T00:14:02.739 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:36.349724+0000", 2020-01-16T00:14:02.739 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 26.145689507, 2020-01-16T00:14:02.739 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017836446999999998, 2020-01-16T00:14:02.739 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:02.739 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:02.739 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:02.739 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:02.740 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:02.740 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3482 2020-01-16T00:14:02.740 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.740 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:02.740 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.740 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:02.740 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349724+0000", 2020-01-16T00:14:02.741 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:02.741 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.741 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.741 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:02.741 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349724+0000", 2020-01-16T00:14:02.741 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:02.741 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.741 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.742 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:02.742 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349723+0000", 2020-01-16T00:14:02.742 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9709999999999998e-06 2020-01-16T00:14:02.742 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.742 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.742 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:02.742 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349727+0000", 2020-01-16T00:14:02.743 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5419999999999999e-06 2020-01-16T00:14:02.743 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.743 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.743 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:02.743 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349732+0000", 2020-01-16T00:14:02.743 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.481e-06 2020-01-16T00:14:02.743 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.743 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.744 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:02.744 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.349734+0000", 2020-01-16T00:14:02.744 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.006195028 2020-01-16T00:14:02.744 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.744 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.744 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.744 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.355929+0000", 2020-01-16T00:14:02.745 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1483999999999996e-05 2020-01-16T00:14:02.745 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.745 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.745 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:02.745 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.355991+0000", 2020-01-16T00:14:02.745 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010536415 2020-01-16T00:14:02.745 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.745 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.746 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.746 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.366527+0000", 2020-01-16T00:14:02.746 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8099e-05 2020-01-16T00:14:02.746 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.746 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.746 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:02.747 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.366595+0000", 2020-01-16T00:14:02.747 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00063624500000000002 2020-01-16T00:14:02.747 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.747 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.747 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.747 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.367231+0000", 2020-01-16T00:14:02.747 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000121196 2020-01-16T00:14:02.747 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.748 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.748 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:02.748 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.367353+0000", 2020-01-16T00:14:02.748 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020780500000000001 2020-01-16T00:14:02.748 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.748 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.748 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:02.749 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:36.367560+0000", 2020-01-16T00:14:02.749 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017836446999999998 2020-01-16T00:14:02.749 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.749 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:02.749 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.749 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.749 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.750 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3810 3.7 3:efab040c:::smithi12313934-30:head [write 1837406~542493] snapc 0=[] ondisk+write+known_if_redirected e50)", 2020-01-16T00:14:02.750 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:43.768525+0000", 2020-01-16T00:14:02.750 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 18.726888423999998, 2020-01-16T00:14:02.750 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018385839000000001, 2020-01-16T00:14:02.750 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:02.750 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:02.750 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:02.751 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:02.751 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:02.751 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3810 2020-01-16T00:14:02.751 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.751 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:02.751 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.751 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:02.752 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768525+0000", 2020-01-16T00:14:02.752 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:02.752 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.752 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.752 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:02.752 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768525+0000", 2020-01-16T00:14:02.752 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:02.753 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.753 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.753 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:02.753 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768523+0000", 2020-01-16T00:14:02.753 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000149512 2020-01-16T00:14:02.753 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.753 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.753 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:02.754 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768673+0000", 2020-01-16T00:14:02.754 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0732e-05 2020-01-16T00:14:02.754 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.754 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.754 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:02.754 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768684+0000", 2020-01-16T00:14:02.754 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3719999999999998e-06 2020-01-16T00:14:02.755 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.755 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.755 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:02.755 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768688+0000", 2020-01-16T00:14:02.755 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6513999999999994e-05 2020-01-16T00:14:02.755 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.755 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.755 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.756 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768754+0000", 2020-01-16T00:14:02.756 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00075867800000000004 2020-01-16T00:14:02.756 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.756 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.756 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:02.756 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.769513+0000", 2020-01-16T00:14:02.756 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012395692 2020-01-16T00:14:02.756 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.757 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.757 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.757 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.781909+0000", 2020-01-16T00:14:02.757 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118372 2020-01-16T00:14:02.757 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.757 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.757 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:02.757 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.782027+0000", 2020-01-16T00:14:02.758 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019909699999999999 2020-01-16T00:14:02.758 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.758 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.758 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,7", 2020-01-16T00:14:02.758 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.782226+0000", 2020-01-16T00:14:02.758 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0038778369999999999 2020-01-16T00:14:02.758 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.759 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.759 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:02.759 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786104+0000", 2020-01-16T00:14:02.759 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022391199999999999 2020-01-16T00:14:02.759 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.759 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.759 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:02.759 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786328+0000", 2020-01-16T00:14:02.760 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00048851299999999999 2020-01-16T00:14:02.760 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.760 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.760 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:02.760 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786817+0000", 2020-01-16T00:14:02.761 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3597999999999998e-05 2020-01-16T00:14:02.761 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.761 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.761 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:02.762 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786860+0000", 2020-01-16T00:14:02.762 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0741000000000001e-05 2020-01-16T00:14:02.762 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.762 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.762 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:02.762 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786911+0000", 2020-01-16T00:14:02.762 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018385839000000001 2020-01-16T00:14:02.763 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.763 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:02.763 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.763 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.763 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.763 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3863 3.2 3:4f8fe446:::smithi12313934-48:head [setxattr _header (56) in=7b,truncate 2151395] snapc 0=[] ondisk+write+known_if_redirected e50)", 2020-01-16T00:14:02.763 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:44.489284+0000", 2020-01-16T00:14:02.763 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 18.006129951999998, 2020-01-16T00:14:02.764 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.026323742000000001, 2020-01-16T00:14:02.764 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:02.764 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:02.764 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:02.764 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:02.764 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:02.764 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3863 2020-01-16T00:14:02.765 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.765 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:02.765 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.765 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:02.765 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489284+0000", 2020-01-16T00:14:02.765 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:02.765 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.765 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.766 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:02.766 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489284+0000", 2020-01-16T00:14:02.766 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:02.766 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.766 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.766 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:02.766 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489282+0000", 2020-01-16T00:14:02.766 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6660000000000002e-06 2020-01-16T00:14:02.767 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.767 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.767 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:02.767 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489289+0000", 2020-01-16T00:14:02.767 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3069999999999999e-06 2020-01-16T00:14:02.767 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.767 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.767 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:02.768 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489295+0000", 2020-01-16T00:14:02.768 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6090000000000003e-06 2020-01-16T00:14:02.768 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.768 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.768 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:02.768 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489300+0000", 2020-01-16T00:14:02.768 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002242464 2020-01-16T00:14:02.769 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.769 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.769 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.769 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491542+0000", 2020-01-16T00:14:02.769 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00059546200000000005 2020-01-16T00:14:02.769 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.769 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.769 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:02.770 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.492138+0000", 2020-01-16T00:14:02.770 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018547999999999999 2020-01-16T00:14:02.770 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.770 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.770 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.770 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.510686+0000", 2020-01-16T00:14:02.770 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018590100000000001 2020-01-16T00:14:02.770 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.771 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.771 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:02.771 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.510872+0000", 2020-01-16T00:14:02.771 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035115299999999999 2020-01-16T00:14:02.771 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.771 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.771 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:02.772 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.511223+0000", 2020-01-16T00:14:02.772 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003765493 2020-01-16T00:14:02.772 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.772 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.772 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:02.772 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.514988+0000", 2020-01-16T00:14:02.772 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000432022 2020-01-16T00:14:02.773 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.773 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.773 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:02.773 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515421+0000", 2020-01-16T00:14:02.773 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2874000000000002e-05 2020-01-16T00:14:02.774 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.774 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.774 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:02.774 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515473+0000", 2020-01-16T00:14:02.774 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013385 2020-01-16T00:14:02.774 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.774 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.775 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:02.775 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515607+0000", 2020-01-16T00:14:02.775 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.026323742000000001 2020-01-16T00:14:02.775 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.775 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:02.775 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.775 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.776 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.776 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3864 3.2 3:4f8fe446:::smithi12313934-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e50)", 2020-01-16T00:14:02.776 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:44.491131+0000", 2020-01-16T00:14:02.776 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 18.004282554, 2020-01-16T00:14:02.776 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024813806000000001, 2020-01-16T00:14:02.776 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:02.776 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:02.777 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:02.777 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:02.777 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:02.777 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3864 2020-01-16T00:14:02.778 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.778 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:02.778 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.778 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:02.778 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491131+0000", 2020-01-16T00:14:02.778 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:02.778 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.779 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.779 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:02.779 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491131+0000", 2020-01-16T00:14:02.779 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:02.779 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.779 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.779 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:02.780 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491128+0000", 2020-01-16T00:14:02.780 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2053e-05 2020-01-16T00:14:02.780 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.780 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.780 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:02.780 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491140+0000", 2020-01-16T00:14:02.780 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5869999999999999e-05 2020-01-16T00:14:02.781 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.781 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.781 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:02.781 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491156+0000", 2020-01-16T00:14:02.781 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2571e-05 2020-01-16T00:14:02.781 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.781 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.781 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:02.782 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491168+0000", 2020-01-16T00:14:02.782 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001031045 2020-01-16T00:14:02.782 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.782 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.782 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.782 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.492199+0000", 2020-01-16T00:14:02.782 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000170865 2020-01-16T00:14:02.783 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.783 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.783 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:02.783 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.492370+0000", 2020-01-16T00:14:02.783 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020180638000000001 2020-01-16T00:14:02.783 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.783 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.783 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.784 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.512551+0000", 2020-01-16T00:14:02.784 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012478599999999999 2020-01-16T00:14:02.784 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.784 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.784 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:02.784 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.512676+0000", 2020-01-16T00:14:02.784 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029628660000000002 2020-01-16T00:14:02.784 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.785 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.785 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.785 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515639+0000", 2020-01-16T00:14:02.785 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012250799999999999 2020-01-16T00:14:02.785 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.785 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.785 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:02.786 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515761+0000", 2020-01-16T00:14:02.786 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018356399999999999 2020-01-16T00:14:02.786 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.786 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.786 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:02.786 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515945+0000", 2020-01-16T00:14:02.786 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024813806000000001 2020-01-16T00:14:02.786 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.787 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:02.787 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.787 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.787 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.787 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3874 3.c 3:378f3722:::smithi12313934-50:head [write 568202~701253,stat] snapc 0=[] ondisk+write+known_if_redirected e50)", 2020-01-16T00:14:02.787 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:44.497041+0000", 2020-01-16T00:14:02.787 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 17.998372923000002, 2020-01-16T00:14:02.787 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019602525999999999, 2020-01-16T00:14:02.788 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:02.788 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:02.788 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:02.788 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:02.788 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:02.788 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3874 2020-01-16T00:14:02.788 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.789 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:02.789 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.789 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:02.789 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497041+0000", 2020-01-16T00:14:02.789 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:02.789 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.789 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.789 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:02.790 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497041+0000", 2020-01-16T00:14:02.790 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999962 2020-01-16T00:14:02.790 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.790 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.790 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:02.790 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497037+0000", 2020-01-16T00:14:02.790 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00027168899999999998 2020-01-16T00:14:02.790 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.791 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.791 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:02.791 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497308+0000", 2020-01-16T00:14:02.791 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4202999999999999e-05 2020-01-16T00:14:02.791 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.791 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.791 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:02.792 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497323+0000", 2020-01-16T00:14:02.792 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0280000000000003e-06 2020-01-16T00:14:02.792 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.792 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.792 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:02.792 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497330+0000", 2020-01-16T00:14:02.792 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010681352 2020-01-16T00:14:02.792 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.793 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.793 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:02.793 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.508011+0000", 2020-01-16T00:14:02.793 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5886999999999999e-05 2020-01-16T00:14:02.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.793 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.793 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:02.794 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.508097+0000", 2020-01-16T00:14:02.794 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023353600000000001 2020-01-16T00:14:02.794 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.794 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.794 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:02.794 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.508331+0000", 2020-01-16T00:14:02.794 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0075218339999999998 2020-01-16T00:14:02.794 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.795 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.795 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:02.795 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515852+0000", 2020-01-16T00:14:02.795 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00064178599999999998 2020-01-16T00:14:02.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.796 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.796 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:02.796 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.516494+0000", 2020-01-16T00:14:02.796 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6712999999999999e-05 2020-01-16T00:14:02.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.796 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.796 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:02.797 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.516531+0000", 2020-01-16T00:14:02.797 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112225 2020-01-16T00:14:02.797 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:02.797 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:02.797 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:02.797 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.516643+0000", 2020-01-16T00:14:02.797 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019602525999999999 2020-01-16T00:14:02.797 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.798 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:02.798 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.798 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:02.798 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:02.798 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:02.799 INFO:tasks.rados.rados.0.smithi123.stdout:1637: finishing write tid 1 to smithi12313934-11 2020-01-16T00:14:02.800 INFO:tasks.rados.rados.0.smithi123.stdout:1636: expect (ObjNum 535 snap 0 seq_num 535) 2020-01-16T00:14:02.800 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:02.798+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:02.800 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-01-16T00:14:02.801 INFO:tasks.rados.rados.0.smithi123.stdout:1635: done (11 left) 2020-01-16T00:14:02.801 INFO:tasks.rados.rados.0.smithi123.stdout:1636: done (10 left) 2020-01-16T00:14:02.801 INFO:tasks.rados.rados.0.smithi123.stdout:1646: done (9 left) 2020-01-16T00:14:02.802 INFO:tasks.rados.rados.0.smithi123.stdout:1647: read oid 8 snap -1 2020-01-16T00:14:02.802 INFO:tasks.rados.rados.0.smithi123.stdout:1647: expect (ObjNum 565 snap 0 seq_num 565) 2020-01-16T00:14:02.802 INFO:tasks.rados.rados.0.smithi123.stdout:1648: read oid 17 snap -1 2020-01-16T00:14:02.802 INFO:tasks.rados.rados.0.smithi123.stdout:1648: expect (ObjNum 578 snap 0 seq_num 578) 2020-01-16T00:14:02.802 INFO:tasks.rados.rados.0.smithi123.stdout:1649: write (excl) oid 50 current snap is 0 2020-01-16T00:14:02.802 INFO:tasks.rados.rados.0.smithi123.stdout:1649: seq_num 584 ranges {615288=788244,1957837=562523,3121263=559679} 2020-01-16T00:14:02.809 INFO:tasks.rados.rados.0.smithi123.stdout:1649: writing smithi12313934-50 from 615288 to 1403532 tid 1 2020-01-16T00:14:02.818 INFO:tasks.rados.rados.0.smithi123.stdout:1649: writing smithi12313934-50 from 1957837 to 2520360 tid 2 2020-01-16T00:14:02.825 INFO:tasks.rados.rados.0.smithi123.stdout:1649: writing smithi12313934-50 from 3121263 to 3680942 tid 3 2020-01-16T00:14:02.826 INFO:tasks.rados.rados.0.smithi123.stdout:1650: write oid 1 current snap is 0 2020-01-16T00:14:02.826 INFO:tasks.rados.rados.0.smithi123.stdout:1650: seq_num 585 ranges {632095=463493,1616319=422534,2748686=487304} 2020-01-16T00:14:02.831 INFO:tasks.rados.rados.0.smithi123.stdout:1650: writing smithi12313934-1 from 632095 to 1095588 tid 1 2020-01-16T00:14:02.835 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2020-01-16T00:14:02.837 INFO:tasks.rados.rados.0.smithi123.stdout:1650: writing smithi12313934-1 from 1616319 to 2038853 tid 2 2020-01-16T00:14:02.844 INFO:tasks.rados.rados.0.smithi123.stdout:1650: writing smithi12313934-1 from 2748686 to 3235990 tid 3 2020-01-16T00:14:02.844 INFO:tasks.rados.rados.0.smithi123.stdout:1651: write oid 46 current snap is 0 2020-01-16T00:14:02.844 INFO:tasks.rados.rados.0.smithi123.stdout:1651: seq_num 586 ranges {648902=655095,2074801=566192,3176109=672793} 2020-01-16T00:14:02.851 INFO:tasks.rados.rados.0.smithi123.stdout:1651: writing smithi12313934-46 from 648902 to 1303997 tid 1 2020-01-16T00:14:02.858 INFO:tasks.rados.rados.0.smithi123.stdout:1651: writing smithi12313934-46 from 2074801 to 2640993 tid 2 2020-01-16T00:14:02.865 INFO:tasks.rados.rados.0.smithi123.stdout:1651: writing smithi12313934-46 from 3176109 to 3848902 tid 3 2020-01-16T00:14:02.866 INFO:tasks.rados.rados.0.smithi123.stdout:1652: read oid 4 snap -1 2020-01-16T00:14:02.866 INFO:tasks.rados.rados.0.smithi123.stdout:1652: expect (ObjNum 557 snap 0 seq_num 557) 2020-01-16T00:14:02.866 INFO:tasks.rados.rados.0.smithi123.stdout:1653: write (excl) oid 48 current snap is 0 2020-01-16T00:14:02.866 INFO:tasks.rados.rados.0.smithi123.stdout:1653: seq_num 587 ranges {665709=730344,2133283=426203,3203532=509848} 2020-01-16T00:14:02.874 INFO:tasks.rados.rados.0.smithi123.stdout:1653: writing smithi12313934-48 from 665709 to 1396053 tid 1 2020-01-16T00:14:02.880 INFO:tasks.rados.rados.0.smithi123.stdout:1653: writing smithi12313934-48 from 2133283 to 2559486 tid 2 2020-01-16T00:14:02.887 INFO:tasks.rados.rados.0.smithi123.stdout:1653: writing smithi12313934-48 from 3203532 to 3713380 tid 3 2020-01-16T00:14:02.887 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:02.887 INFO:tasks.rados.rados.0.smithi123.stdout:1637: finishing write tid 2 to smithi12313934-11 2020-01-16T00:14:02.887 INFO:tasks.rados.rados.0.smithi123.stdout:1637: finishing write tid 3 to smithi12313934-11 2020-01-16T00:14:02.887 INFO:tasks.rados.rados.0.smithi123.stdout:1638: expect (ObjNum 502 snap 0 seq_num 502) 2020-01-16T00:14:02.900 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:02.896+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:02.934 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:02.934 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:14:02.934 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:14:02.934 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:02.943 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-01-16T00:14:03.001 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:02.996+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:03.036 INFO:tasks.rados.rados.0.smithi123.stdout:1637: finishing write tid 4 to smithi12313934-11 2020-01-16T00:14:03.036 INFO:tasks.rados.rados.0.smithi123.stdout:1637: finishing write tid 6 to smithi12313934-11 2020-01-16T00:14:03.036 INFO:tasks.rados.rados.0.smithi123.stdout:1638: done (15 left) 2020-01-16T00:14:03.036 INFO:tasks.rados.rados.0.smithi123.stdout:1654: write (excl) oid 2 current snap is 0 2020-01-16T00:14:03.036 INFO:tasks.rados.rados.0.smithi123.stdout:1654: seq_num 588 ranges {682516=405593,1791765=569861,2830955=721120} 2020-01-16T00:14:03.051 INFO:tasks.rados.rados.0.smithi123.stdout:1654: writing smithi12313934-2 from 682516 to 1088109 tid 1 2020-01-16T00:14:03.058 INFO:tasks.rados.rados.0.smithi123.stdout:1654: writing smithi12313934-2 from 1791765 to 2361626 tid 2 2020-01-16T00:14:03.066 INFO:tasks.rados.rados.0.smithi123.stdout:1654: writing smithi12313934-2 from 2830955 to 3552075 tid 3 2020-01-16T00:14:03.066 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:03.066 INFO:tasks.rados.rados.0.smithi123.stdout:1637: finishing write tid 7 to smithi12313934-11 2020-01-16T00:14:03.066 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 11 v 170 (ObjNum 579 snap 0 seq_num 579) dirty exists 2020-01-16T00:14:03.066 INFO:tasks.rados.rados.0.smithi123.stdout:1637: left oid 11 (ObjNum 579 snap 0 seq_num 579) 2020-01-16T00:14:03.066 INFO:tasks.rados.rados.0.smithi123.stdout:1639: finishing write tid 1 to smithi12313934-34 2020-01-16T00:14:03.067 INFO:tasks.rados.rados.0.smithi123.stdout:1639: finishing write tid 2 to smithi12313934-34 2020-01-16T00:14:03.067 INFO:tasks.rados.rados.0.smithi123.stdout:1639: finishing write tid 3 to smithi12313934-34 2020-01-16T00:14:03.067 INFO:tasks.rados.rados.0.smithi123.stdout:1639: finishing write tid 5 to smithi12313934-34 2020-01-16T00:14:03.067 INFO:tasks.rados.rados.0.smithi123.stdout:1639: finishing write tid 6 to smithi12313934-34 2020-01-16T00:14:03.067 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 34 v 80 (ObjNum 580 snap 0 seq_num 580) dirty exists 2020-01-16T00:14:03.067 INFO:tasks.rados.rados.0.smithi123.stdout:1639: left oid 34 (ObjNum 580 snap 0 seq_num 580) 2020-01-16T00:14:03.067 INFO:tasks.rados.rados.0.smithi123.stdout:1640: finishing write tid 1 to smithi12313934-16 2020-01-16T00:14:03.067 INFO:tasks.rados.rados.0.smithi123.stdout:1637: done (15 left) 2020-01-16T00:14:03.068 INFO:tasks.rados.rados.0.smithi123.stdout:1639: done (14 left) 2020-01-16T00:14:03.068 INFO:tasks.rados.rados.0.smithi123.stdout:1655: write (excl) oid 29 current snap is 0 2020-01-16T00:14:03.068 INFO:tasks.rados.rados.0.smithi123.stdout:1655: seq_num 589 ranges {699323=480842,1850247=429872,2858378=532392} 2020-01-16T00:14:03.072 INFO:tasks.rados.rados.0.smithi123.stdout:1655: writing smithi12313934-29 from 699323 to 1180165 tid 1 2020-01-16T00:14:03.078 INFO:tasks.rados.rados.0.smithi123.stdout:1655: writing smithi12313934-29 from 1850247 to 2280119 tid 2 2020-01-16T00:14:03.085 INFO:tasks.rados.rados.0.smithi123.stdout:1655: writing smithi12313934-29 from 2858378 to 3390770 tid 3 2020-01-16T00:14:03.085 INFO:tasks.rados.rados.0.smithi123.stdout:1656: delete oid 23 current snap is 0 2020-01-16T00:14:03.085 INFO:tasks.rados.rados.0.smithi123.stdout:1640: finishing write tid 2 to smithi12313934-16 2020-01-16T00:14:03.086 INFO:tasks.rados.rados.0.smithi123.stdout:1640: finishing write tid 3 to smithi12313934-16 2020-01-16T00:14:03.086 INFO:tasks.rados.rados.0.smithi123.stdout:1640: finishing write tid 5 to smithi12313934-16 2020-01-16T00:14:03.086 INFO:tasks.rados.rados.0.smithi123.stdout:1640: finishing write tid 6 to smithi12313934-16 2020-01-16T00:14:03.086 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 16 v 257 (ObjNum 581 snap 0 seq_num 581) dirty exists 2020-01-16T00:14:03.086 INFO:tasks.rados.rados.0.smithi123.stdout:1640: left oid 16 (ObjNum 581 snap 0 seq_num 581) 2020-01-16T00:14:03.086 INFO:tasks.rados.rados.0.smithi123.stdout:1643: expect (ObjNum 513 snap 0 seq_num 513) 2020-01-16T00:14:03.099 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:03.099 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:14:03.099 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:14:03.100 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:14:03.100 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:03.102 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:03.100+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:03.109 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-01-16T00:14:03.202 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:03.202 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:14:03.202 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:14:03.202 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:14:03.202 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.203 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.4.0:114 3.9 e24/20 3:9fc687b2:::smithi12313934-44:head v 24'74, mlcod=24'72)", 2020-01-16T00:14:03.203 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:14:03.203 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 84.873187707, 2020-01-16T00:14:03.203 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010399429, 2020-01-16T00:14:03.203 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:03.203 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:03.203 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:03.204 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.204 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:03.204 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:14:03.204 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:03.204 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.205 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.205 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:03.205 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:14:03.205 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:03.205 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.206 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.206 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:03.206 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325276+0000", 2020-01-16T00:14:03.206 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0046127069999999997 2020-01-16T00:14:03.206 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.206 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.207 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:03.207 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329889+0000", 2020-01-16T00:14:03.207 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3519999999999998e-05 2020-01-16T00:14:03.207 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.208 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.208 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:03.208 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329912+0000", 2020-01-16T00:14:03.208 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4070000000000004e-06 2020-01-16T00:14:03.209 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.209 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.209 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:03.209 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329920+0000", 2020-01-16T00:14:03.209 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8817000000000003e-05 2020-01-16T00:14:03.210 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.210 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.210 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:03.210 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329978+0000", 2020-01-16T00:14:03.211 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9721e-05 2020-01-16T00:14:03.211 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.211 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.212 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:03.212 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329998+0000", 2020-01-16T00:14:03.212 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056328139999999999 2020-01-16T00:14:03.212 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.213 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.213 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:03.213 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.335631+0000", 2020-01-16T00:14:03.213 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7170999999999999e-05 2020-01-16T00:14:03.213 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.214 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.214 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:03.215 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.335678+0000", 2020-01-16T00:14:03.215 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010399429 2020-01-16T00:14:03.215 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.215 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:03.216 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.216 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.216 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.217 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:24 2.1 2:8b431f80:::smithi12313934-40 low_tier:head [write 790528~688128] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:14:03.217 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.482303+0000", 2020-01-16T00:14:03.217 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 84.716163610999999, 2020-01-16T00:14:03.217 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010071853, 2020-01-16T00:14:03.217 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:03.217 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:03.218 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:03.218 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.218 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:03.218 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.482303+0000", 2020-01-16T00:14:03.218 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:03.218 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.219 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.219 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:03.219 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.482303+0000", 2020-01-16T00:14:03.219 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:03.219 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.219 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.220 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:03.220 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.482301+0000", 2020-01-16T00:14:03.220 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031173960000000001 2020-01-16T00:14:03.220 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.220 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.221 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:03.221 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485418+0000", 2020-01-16T00:14:03.221 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9300000000000001e-05 2020-01-16T00:14:03.221 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.221 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.221 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:03.221 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485448+0000", 2020-01-16T00:14:03.222 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2611e-05 2020-01-16T00:14:03.222 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.222 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.222 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:03.222 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485460+0000", 2020-01-16T00:14:03.222 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112487 2020-01-16T00:14:03.222 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.222 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.223 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:03.223 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485573+0000", 2020-01-16T00:14:03.223 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022169299999999999 2020-01-16T00:14:03.223 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.223 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.223 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:03.223 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.485794+0000", 2020-01-16T00:14:03.224 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00036287500000000002 2020-01-16T00:14:03.224 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.224 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.224 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:03.224 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.486157+0000", 2020-01-16T00:14:03.224 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040260310000000002 2020-01-16T00:14:03.225 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.225 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.225 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:03.225 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.490183+0000", 2020-01-16T00:14:03.225 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00081083000000000001 2020-01-16T00:14:03.226 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.226 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.226 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.226 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.490994+0000", 2020-01-16T00:14:03.226 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001287746 2020-01-16T00:14:03.226 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.226 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.226 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.227 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492282+0000", 2020-01-16T00:14:03.227 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8810999999999998e-05 2020-01-16T00:14:03.227 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.227 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.227 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:03.227 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492321+0000", 2020-01-16T00:14:03.227 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4095000000000003e-05 2020-01-16T00:14:03.227 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.228 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.228 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:03.228 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.492375+0000", 2020-01-16T00:14:03.228 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010071853 2020-01-16T00:14:03.228 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.228 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:03.228 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.228 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.229 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.229 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:55 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e30)", 2020-01-16T00:14:03.229 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.736036+0000", 2020-01-16T00:14:03.229 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 70.462430487999995, 2020-01-16T00:14:03.229 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010974536, 2020-01-16T00:14:03.229 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:03.229 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:03.230 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:03.230 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.230 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:03.230 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736036+0000", 2020-01-16T00:14:03.230 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:03.230 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.230 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.230 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:03.231 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736036+0000", 2020-01-16T00:14:03.231 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:03.231 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.231 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.231 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:03.231 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736034+0000", 2020-01-16T00:14:03.231 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043589699999999998 2020-01-16T00:14:03.231 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.232 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.232 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:03.232 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736469+0000", 2020-01-16T00:14:03.232 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9882e-05 2020-01-16T00:14:03.232 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.232 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.232 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:03.233 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736489+0000", 2020-01-16T00:14:03.233 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8339999999999999e-06 2020-01-16T00:14:03.233 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.233 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.233 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:03.233 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736497+0000", 2020-01-16T00:14:03.233 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5354999999999999e-05 2020-01-16T00:14:03.234 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.234 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.234 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:03.234 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736572+0000", 2020-01-16T00:14:03.234 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5228000000000004e-05 2020-01-16T00:14:03.234 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.235 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.235 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:03.235 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736668+0000", 2020-01-16T00:14:03.236 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017212999999999999 2020-01-16T00:14:03.236 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.236 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.236 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:03.236 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736840+0000", 2020-01-16T00:14:03.236 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0099057020000000006 2020-01-16T00:14:03.236 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.237 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.237 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.237 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746746+0000", 2020-01-16T00:14:03.237 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0586000000000005e-05 2020-01-16T00:14:03.237 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.237 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.237 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:03.238 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746816+0000", 2020-01-16T00:14:03.238 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2445000000000002e-05 2020-01-16T00:14:03.238 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.238 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.238 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.238 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746889+0000", 2020-01-16T00:14:03.238 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6839000000000002e-05 2020-01-16T00:14:03.238 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.239 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.239 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:03.239 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746955+0000", 2020-01-16T00:14:03.239 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5074000000000002e-05 2020-01-16T00:14:03.239 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.239 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.239 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:03.240 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.747011+0000", 2020-01-16T00:14:03.240 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010974536 2020-01-16T00:14:03.240 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.240 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:03.240 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.240 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.240 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.240 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:56 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e30)", 2020-01-16T00:14:03.241 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.736514+0000", 2020-01-16T00:14:03.241 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 70.461952652999997, 2020-01-16T00:14:03.241 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011660584999999999, 2020-01-16T00:14:03.241 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:03.241 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:03.241 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:03.242 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.242 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:03.242 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736514+0000", 2020-01-16T00:14:03.242 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:03.242 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.242 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.243 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:03.243 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736514+0000", 2020-01-16T00:14:03.243 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:03.243 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.243 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.243 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:03.243 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736513+0000", 2020-01-16T00:14:03.244 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9677000000000004e-05 2020-01-16T00:14:03.244 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.244 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.244 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:03.244 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736583+0000", 2020-01-16T00:14:03.244 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5269999999999998e-06 2020-01-16T00:14:03.245 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.245 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.245 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:03.245 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736591+0000", 2020-01-16T00:14:03.245 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5769999999999996e-06 2020-01-16T00:14:03.245 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.246 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.246 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:03.246 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736599+0000", 2020-01-16T00:14:03.246 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0053593310000000002 2020-01-16T00:14:03.246 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.246 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.247 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:03.247 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.741958+0000", 2020-01-16T00:14:03.247 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014164099999999999 2020-01-16T00:14:03.247 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.247 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.247 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:03.247 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.742100+0000", 2020-01-16T00:14:03.247 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020126000000000001 2020-01-16T00:14:03.248 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.248 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.248 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:03.248 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.742301+0000", 2020-01-16T00:14:03.248 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0045052859999999998 2020-01-16T00:14:03.248 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.248 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.248 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.249 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746806+0000", 2020-01-16T00:14:03.249 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00070802699999999996 2020-01-16T00:14:03.249 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.249 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.249 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:03.249 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.747514+0000", 2020-01-16T00:14:03.249 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00059247200000000003 2020-01-16T00:14:03.250 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.250 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.250 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.250 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748107+0000", 2020-01-16T00:14:03.250 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1870999999999999e-05 2020-01-16T00:14:03.250 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.250 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.250 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:03.251 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748149+0000", 2020-01-16T00:14:03.251 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5831000000000001e-05 2020-01-16T00:14:03.251 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.251 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.251 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:03.251 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748174+0000", 2020-01-16T00:14:03.251 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011660584999999999 2020-01-16T00:14:03.251 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.252 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:03.252 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.252 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.252 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.252 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:70 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e35)", 2020-01-16T00:14:03.252 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.985494+0000", 2020-01-16T00:14:03.252 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 57.212972856999997, 2020-01-16T00:14:03.253 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013746267, 2020-01-16T00:14:03.253 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:03.253 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:03.253 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:03.253 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.253 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:03.253 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985494+0000", 2020-01-16T00:14:03.253 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:03.254 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.254 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.254 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:03.254 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985494+0000", 2020-01-16T00:14:03.254 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:03.254 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.254 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.255 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:03.255 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985490+0000", 2020-01-16T00:14:03.255 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00077792099999999999 2020-01-16T00:14:03.255 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.255 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.255 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:03.255 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986268+0000", 2020-01-16T00:14:03.255 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9718000000000001e-05 2020-01-16T00:14:03.256 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.256 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.256 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:03.256 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986318+0000", 2020-01-16T00:14:03.256 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.4349999999999997e-06 2020-01-16T00:14:03.256 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.257 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.257 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:03.257 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986326+0000", 2020-01-16T00:14:03.257 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3298999999999999e-05 2020-01-16T00:14:03.257 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.257 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.257 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:03.258 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986380+0000", 2020-01-16T00:14:03.258 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4969999999999995e-05 2020-01-16T00:14:03.258 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.258 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.258 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:03.258 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986455+0000", 2020-01-16T00:14:03.258 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000110587 2020-01-16T00:14:03.258 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.259 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.259 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:03.259 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986565+0000", 2020-01-16T00:14:03.259 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063931600000000002 2020-01-16T00:14:03.259 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.259 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.259 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:03.259 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.992958+0000", 2020-01-16T00:14:03.260 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013008480000000001 2020-01-16T00:14:03.260 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.260 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.260 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.260 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.994259+0000", 2020-01-16T00:14:03.261 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048399979999999999 2020-01-16T00:14:03.261 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.261 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.261 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.261 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999099+0000", 2020-01-16T00:14:03.261 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1874999999999998e-05 2020-01-16T00:14:03.261 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.261 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.262 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:03.262 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999181+0000", 2020-01-16T00:14:03.262 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8865e-05 2020-01-16T00:14:03.262 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.262 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.262 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:03.262 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999240+0000", 2020-01-16T00:14:03.263 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013746267 2020-01-16T00:14:03.263 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.263 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:03.263 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.263 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.263 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.263 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:71 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e35)", 2020-01-16T00:14:03.264 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.986341+0000", 2020-01-16T00:14:03.264 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 57.212125495999999, 2020-01-16T00:14:03.264 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013252807, 2020-01-16T00:14:03.264 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:03.264 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:03.264 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:03.264 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.264 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:03.265 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986341+0000", 2020-01-16T00:14:03.265 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:03.265 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.265 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.265 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:03.265 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986341+0000", 2020-01-16T00:14:03.266 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:03.266 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.266 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.266 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:03.266 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986340+0000", 2020-01-16T00:14:03.266 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000103867 2020-01-16T00:14:03.267 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.267 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.267 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:03.267 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986444+0000", 2020-01-16T00:14:03.267 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0104e-05 2020-01-16T00:14:03.267 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.268 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.268 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:03.268 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986454+0000", 2020-01-16T00:14:03.268 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4220000000000002e-06 2020-01-16T00:14:03.269 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.269 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.269 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:03.269 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986458+0000", 2020-01-16T00:14:03.269 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033523070000000001 2020-01-16T00:14:03.270 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.270 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.270 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:03.270 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.989811+0000", 2020-01-16T00:14:03.271 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3296999999999999e-05 2020-01-16T00:14:03.271 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.271 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.271 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:03.271 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.989894+0000", 2020-01-16T00:14:03.272 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000117701 2020-01-16T00:14:03.272 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.272 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.272 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:03.272 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.990012+0000", 2020-01-16T00:14:03.273 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0032873490000000002 2020-01-16T00:14:03.273 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.273 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.273 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:03.274 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993299+0000", 2020-01-16T00:14:03.274 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002422619 2020-01-16T00:14:03.274 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.274 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.275 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.275 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.995722+0000", 2020-01-16T00:14:03.275 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0037839589999999999 2020-01-16T00:14:03.275 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.275 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.276 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.276 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999506+0000", 2020-01-16T00:14:03.276 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8559999999999997e-05 2020-01-16T00:14:03.277 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.277 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.277 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:03.277 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999544+0000", 2020-01-16T00:14:03.278 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9599999999999999e-05 2020-01-16T00:14:03.278 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.278 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.278 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:03.279 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999594+0000", 2020-01-16T00:14:03.279 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013252807 2020-01-16T00:14:03.279 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.279 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:03.279 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.280 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.280 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.280 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:77 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e36)", 2020-01-16T00:14:03.280 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.555854+0000", 2020-01-16T00:14:03.281 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 55.642612407000001, 2020-01-16T00:14:03.281 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010455002999999999, 2020-01-16T00:14:03.281 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:03.281 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:03.282 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:03.282 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.282 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:03.282 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555854+0000", 2020-01-16T00:14:03.282 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:03.283 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.283 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.283 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:03.283 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555854+0000", 2020-01-16T00:14:03.284 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:03.284 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.284 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.284 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:03.285 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555851+0000", 2020-01-16T00:14:03.285 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00048310200000000001 2020-01-16T00:14:03.285 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.285 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.285 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:03.286 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556334+0000", 2020-01-16T00:14:03.286 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2232e-05 2020-01-16T00:14:03.286 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.286 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.287 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:03.287 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556356+0000", 2020-01-16T00:14:03.287 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9750000000000007e-06 2020-01-16T00:14:03.287 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.287 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.288 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:03.288 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556364+0000", 2020-01-16T00:14:03.288 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8872999999999997e-05 2020-01-16T00:14:03.288 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.288 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.288 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:03.288 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556423+0000", 2020-01-16T00:14:03.288 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8687000000000001e-05 2020-01-16T00:14:03.289 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.289 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.289 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:03.289 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556502+0000", 2020-01-16T00:14:03.289 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013235300000000001 2020-01-16T00:14:03.289 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.289 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.289 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:03.290 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556634+0000", 2020-01-16T00:14:03.290 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0059770520000000001 2020-01-16T00:14:03.290 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.290 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.290 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:03.290 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.562611+0000", 2020-01-16T00:14:03.290 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029492820000000001 2020-01-16T00:14:03.291 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.291 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.291 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.291 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.565561+0000", 2020-01-16T00:14:03.291 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00060620300000000004 2020-01-16T00:14:03.291 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.291 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.292 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.292 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566167+0000", 2020-01-16T00:14:03.292 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3831000000000004e-05 2020-01-16T00:14:03.292 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.292 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.293 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:03.293 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566251+0000", 2020-01-16T00:14:03.293 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8328000000000001e-05 2020-01-16T00:14:03.293 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.294 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.294 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:03.294 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566309+0000", 2020-01-16T00:14:03.294 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010455002999999999 2020-01-16T00:14:03.294 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.294 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:03.294 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.295 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.295 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.295 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:78 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e36)", 2020-01-16T00:14:03.295 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.556378+0000", 2020-01-16T00:14:03.295 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 55.642088473999998, 2020-01-16T00:14:03.295 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010396311, 2020-01-16T00:14:03.295 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:03.295 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:03.296 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:03.296 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.296 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:03.296 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556378+0000", 2020-01-16T00:14:03.296 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:03.296 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.296 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.296 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:03.297 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556378+0000", 2020-01-16T00:14:03.297 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:03.297 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.297 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.298 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:03.298 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556377+0000", 2020-01-16T00:14:03.298 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4307000000000001e-05 2020-01-16T00:14:03.298 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.298 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.298 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:03.298 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556422+0000", 2020-01-16T00:14:03.299 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9620000000000005e-06 2020-01-16T00:14:03.299 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.299 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.299 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:03.299 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556432+0000", 2020-01-16T00:14:03.299 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5979999999999999e-06 2020-01-16T00:14:03.300 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.300 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.300 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:03.300 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556436+0000", 2020-01-16T00:14:03.300 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021059469999999999 2020-01-16T00:14:03.300 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.301 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.301 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:03.301 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.558542+0000", 2020-01-16T00:14:03.301 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000194458 2020-01-16T00:14:03.301 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.301 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.301 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:03.301 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.558737+0000", 2020-01-16T00:14:03.302 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00032867900000000002 2020-01-16T00:14:03.302 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.302 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.302 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:03.302 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.559065+0000", 2020-01-16T00:14:03.302 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052547499999999999 2020-01-16T00:14:03.302 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.303 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.303 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:03.303 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.564320+0000", 2020-01-16T00:14:03.303 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001565213 2020-01-16T00:14:03.303 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.303 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.303 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.303 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.565885+0000", 2020-01-16T00:14:03.303 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00076994300000000004 2020-01-16T00:14:03.304 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.304 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.304 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.304 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566655+0000", 2020-01-16T00:14:03.304 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7533000000000005e-05 2020-01-16T00:14:03.304 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.304 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.305 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:03.305 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566723+0000", 2020-01-16T00:14:03.305 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1616999999999997e-05 2020-01-16T00:14:03.305 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.305 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.305 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:03.306 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566774+0000", 2020-01-16T00:14:03.306 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010396311 2020-01-16T00:14:03.306 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.306 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:03.306 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.306 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.306 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.307 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:72 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 56832~1408000] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e37)", 2020-01-16T00:14:03.307 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.578753+0000", 2020-01-16T00:14:03.307 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 50.619713171000001, 2020-01-16T00:14:03.307 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011826918, 2020-01-16T00:14:03.307 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:03.308 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:03.308 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:03.308 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.308 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:03.309 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578753+0000", 2020-01-16T00:14:03.309 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:03.309 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.309 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.309 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:03.310 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578753+0000", 2020-01-16T00:14:03.310 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:03.310 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.310 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.311 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:03.311 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578751+0000", 2020-01-16T00:14:03.311 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00042457699999999998 2020-01-16T00:14:03.311 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.311 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.312 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:03.312 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579175+0000", 2020-01-16T00:14:03.312 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5404999999999999e-05 2020-01-16T00:14:03.313 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.314 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.314 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:03.314 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579191+0000", 2020-01-16T00:14:03.314 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6149999999999996e-06 2020-01-16T00:14:03.314 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.315 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.315 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:03.315 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579196+0000", 2020-01-16T00:14:03.315 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000104794 2020-01-16T00:14:03.315 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.315 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.315 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:03.316 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579301+0000", 2020-01-16T00:14:03.316 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020947199999999999 2020-01-16T00:14:03.316 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.316 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.316 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:03.316 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579511+0000", 2020-01-16T00:14:03.317 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034328400000000002 2020-01-16T00:14:03.317 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.317 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.317 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:03.317 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579854+0000", 2020-01-16T00:14:03.317 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.005078117 2020-01-16T00:14:03.318 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.318 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.318 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:03.318 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.584932+0000", 2020-01-16T00:14:03.318 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0014702160000000001 2020-01-16T00:14:03.318 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.319 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.319 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.319 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.586402+0000", 2020-01-16T00:14:03.319 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040623589999999998 2020-01-16T00:14:03.319 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.319 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.320 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.320 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590465+0000", 2020-01-16T00:14:03.320 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4478000000000002e-05 2020-01-16T00:14:03.320 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.320 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.321 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:03.321 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590529+0000", 2020-01-16T00:14:03.321 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1121000000000001e-05 2020-01-16T00:14:03.321 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.321 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.321 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:03.322 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590580+0000", 2020-01-16T00:14:03.322 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011826918 2020-01-16T00:14:03.322 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.322 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:03.322 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.322 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.322 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.323 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:73 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 1922048~118272] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e37)", 2020-01-16T00:14:03.323 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:14:03.323 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 50.619260374, 2020-01-16T00:14:03.323 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011750764, 2020-01-16T00:14:03.323 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:03.323 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:03.323 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:03.324 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.324 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:03.324 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:14:03.324 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:03.324 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.324 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.324 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:03.325 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:14:03.325 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:03.325 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.325 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.325 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:03.325 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:14:03.326 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2780000000000001e-05 2020-01-16T00:14:03.326 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.326 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.326 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:03.326 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579238+0000", 2020-01-16T00:14:03.326 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4341999999999999e-05 2020-01-16T00:14:03.326 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.327 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.327 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:03.327 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579253+0000", 2020-01-16T00:14:03.327 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1920000000000001e-06 2020-01-16T00:14:03.328 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.328 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.329 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:03.329 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579256+0000", 2020-01-16T00:14:03.329 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003224902 2020-01-16T00:14:03.329 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.329 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.329 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:03.329 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.582481+0000", 2020-01-16T00:14:03.330 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000180729 2020-01-16T00:14:03.330 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.330 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.330 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:03.330 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.582662+0000", 2020-01-16T00:14:03.330 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00025106100000000001 2020-01-16T00:14:03.330 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.331 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.331 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:03.331 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.582913+0000", 2020-01-16T00:14:03.331 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020814449999999999 2020-01-16T00:14:03.331 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.331 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.332 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:03.332 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.584994+0000", 2020-01-16T00:14:03.332 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001717243 2020-01-16T00:14:03.332 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.332 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.332 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.332 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.586711+0000", 2020-01-16T00:14:03.333 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041349869999999997 2020-01-16T00:14:03.333 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.333 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.333 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.333 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590846+0000", 2020-01-16T00:14:03.333 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3565000000000006e-05 2020-01-16T00:14:03.333 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.333 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.334 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:03.334 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590910+0000", 2020-01-16T00:14:03.334 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7002999999999997e-05 2020-01-16T00:14:03.334 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.334 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.334 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:03.334 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590957+0000", 2020-01-16T00:14:03.335 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011750764 2020-01-16T00:14:03.335 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.335 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:03.335 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.335 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.335 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.335 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:89 2.1 2:9b8617ce:::smithi12313934-38 low_tier:head [write 69632~773120] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e37)", 2020-01-16T00:14:03.336 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.593701+0000", 2020-01-16T00:14:03.336 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 50.604765747999998, 2020-01-16T00:14:03.336 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010026929, 2020-01-16T00:14:03.337 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:03.337 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:03.337 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:03.337 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.337 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:03.337 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593701+0000", 2020-01-16T00:14:03.338 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:03.338 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.338 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.338 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:03.339 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593701+0000", 2020-01-16T00:14:03.339 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:03.339 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.339 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.339 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:03.339 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593699+0000", 2020-01-16T00:14:03.339 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000237564 2020-01-16T00:14:03.340 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.340 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.340 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:03.340 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593936+0000", 2020-01-16T00:14:03.340 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2666e-05 2020-01-16T00:14:03.340 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.340 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.340 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:03.341 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593949+0000", 2020-01-16T00:14:03.341 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0089999999999998e-06 2020-01-16T00:14:03.341 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.341 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.341 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:03.341 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593954+0000", 2020-01-16T00:14:03.341 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5813000000000003e-05 2020-01-16T00:14:03.342 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.342 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.342 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:03.342 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.594000+0000", 2020-01-16T00:14:03.342 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010155299999999999 2020-01-16T00:14:03.342 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.342 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.342 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:03.343 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.594101+0000", 2020-01-16T00:14:03.343 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000203807 2020-01-16T00:14:03.343 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.343 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.343 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:03.343 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.594305+0000", 2020-01-16T00:14:03.343 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021655390000000002 2020-01-16T00:14:03.343 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.344 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.344 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.344 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.596471+0000", 2020-01-16T00:14:03.344 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0070645320000000001 2020-01-16T00:14:03.344 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.344 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.344 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.345 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603535+0000", 2020-01-16T00:14:03.345 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7577999999999998e-05 2020-01-16T00:14:03.345 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.345 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.345 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:03.345 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603573+0000", 2020-01-16T00:14:03.345 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9933999999999997e-05 2020-01-16T00:14:03.346 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.346 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.346 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:03.346 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603653+0000", 2020-01-16T00:14:03.346 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4993000000000005e-05 2020-01-16T00:14:03.346 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.347 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.347 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:03.347 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.603728+0000", 2020-01-16T00:14:03.347 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010026929 2020-01-16T00:14:03.347 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.347 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:03.347 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.347 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.348 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.348 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:66 2.3 2:c3b82d87:::smithi12313934-44 low_tier:head [write 169472~818176] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e44)", 2020-01-16T00:14:03.348 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:25.278583+0000", 2020-01-16T00:14:03.348 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 37.919883351000003, 2020-01-16T00:14:03.348 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010294862, 2020-01-16T00:14:03.348 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:03.348 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:03.349 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:03.349 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.349 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:03.349 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.278583+0000", 2020-01-16T00:14:03.350 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:03.350 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.350 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.350 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:03.350 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.278583+0000", 2020-01-16T00:14:03.350 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:03.350 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.350 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.351 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:03.351 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.278581+0000", 2020-01-16T00:14:03.351 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00136721 2020-01-16T00:14:03.351 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.351 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.351 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:03.351 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.279948+0000", 2020-01-16T00:14:03.352 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6109999999999999e-05 2020-01-16T00:14:03.352 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.352 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.352 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:03.352 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.279974+0000", 2020-01-16T00:14:03.352 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.102e-05 2020-01-16T00:14:03.352 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.353 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.353 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:03.353 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.279985+0000", 2020-01-16T00:14:03.353 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6035e-05 2020-01-16T00:14:03.353 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.353 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.353 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:03.354 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.280061+0000", 2020-01-16T00:14:03.354 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000176243 2020-01-16T00:14:03.354 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.354 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.354 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:03.355 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.280238+0000", 2020-01-16T00:14:03.355 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019866099999999999 2020-01-16T00:14:03.355 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.355 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.355 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 7", 2020-01-16T00:14:03.355 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.280436+0000", 2020-01-16T00:14:03.355 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0049761830000000003 2020-01-16T00:14:03.356 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.356 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.356 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.356 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.285412+0000", 2020-01-16T00:14:03.356 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033347339999999998 2020-01-16T00:14:03.356 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.356 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.356 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:03.357 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.288747+0000", 2020-01-16T00:14:03.357 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9156000000000001e-05 2020-01-16T00:14:03.357 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.357 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.357 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:03.357 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.288806+0000", 2020-01-16T00:14:03.357 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1643000000000005e-05 2020-01-16T00:14:03.358 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.358 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.358 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:03.358 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.288878+0000", 2020-01-16T00:14:03.358 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010294862 2020-01-16T00:14:03.358 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.358 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:03.358 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.359 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.359 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.359 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:77 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 56832~1408000] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e45)", 2020-01-16T00:14:03.360 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.584566+0000", 2020-01-16T00:14:03.361 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 33.613900700000002, 2020-01-16T00:14:03.361 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017037775000000002, 2020-01-16T00:14:03.361 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:03.361 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:03.361 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:03.361 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.362 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:03.362 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584566+0000", 2020-01-16T00:14:03.362 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:03.362 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.362 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.362 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:03.363 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584566+0000", 2020-01-16T00:14:03.363 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:03.363 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.363 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.363 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:03.363 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584564+0000", 2020-01-16T00:14:03.363 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043024300000000002 2020-01-16T00:14:03.363 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.364 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.364 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:03.364 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584994+0000", 2020-01-16T00:14:03.364 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9485999999999999e-05 2020-01-16T00:14:03.364 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.365 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.365 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:03.365 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585014+0000", 2020-01-16T00:14:03.365 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8909999999999995e-06 2020-01-16T00:14:03.365 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.365 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.365 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:03.366 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585022+0000", 2020-01-16T00:14:03.366 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039114459999999998 2020-01-16T00:14:03.366 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.366 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.366 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:03.366 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588933+0000", 2020-01-16T00:14:03.366 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011178700000000001 2020-01-16T00:14:03.367 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.367 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.367 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:03.367 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589045+0000", 2020-01-16T00:14:03.367 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016967300000000001 2020-01-16T00:14:03.367 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.367 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.368 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:03.368 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589214+0000", 2020-01-16T00:14:03.368 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012142356 2020-01-16T00:14:03.368 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.368 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.368 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.368 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601357+0000", 2020-01-16T00:14:03.368 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.1466999999999997e-05 2020-01-16T00:14:03.369 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.369 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.369 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.369 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601448+0000", 2020-01-16T00:14:03.369 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4817000000000001e-05 2020-01-16T00:14:03.369 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.369 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.370 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:03.370 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601493+0000", 2020-01-16T00:14:03.370 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9734000000000001e-05 2020-01-16T00:14:03.370 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.371 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.371 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:03.371 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601553+0000", 2020-01-16T00:14:03.371 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0677999999999998e-05 2020-01-16T00:14:03.371 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.371 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.371 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:03.371 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601604+0000", 2020-01-16T00:14:03.372 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017037775000000002 2020-01-16T00:14:03.372 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.372 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:03.372 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.372 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.372 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.372 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:78 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 1922048~118272] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e45)", 2020-01-16T00:14:03.373 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.585030+0000", 2020-01-16T00:14:03.373 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 33.613435977999998, 2020-01-16T00:14:03.373 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016699964000000001, 2020-01-16T00:14:03.373 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:03.373 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:03.373 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:03.373 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.373 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:03.374 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585030+0000", 2020-01-16T00:14:03.374 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:03.374 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.374 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.374 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:03.374 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585030+0000", 2020-01-16T00:14:03.374 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:03.374 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.375 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.375 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:03.375 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585030+0000", 2020-01-16T00:14:03.375 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7085000000000001e-05 2020-01-16T00:14:03.375 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.375 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.375 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:03.376 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585077+0000", 2020-01-16T00:14:03.376 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5380000000000008e-06 2020-01-16T00:14:03.376 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.376 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.376 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:03.376 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585086+0000", 2020-01-16T00:14:03.376 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5410000000000002e-06 2020-01-16T00:14:03.376 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.377 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.377 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:03.377 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585091+0000", 2020-01-16T00:14:03.377 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0088486620000000002 2020-01-16T00:14:03.377 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.377 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.377 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:03.378 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.593940+0000", 2020-01-16T00:14:03.378 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112594 2020-01-16T00:14:03.378 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.378 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.378 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:03.378 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.594052+0000", 2020-01-16T00:14:03.378 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019893999999999999 2020-01-16T00:14:03.378 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.379 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.379 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:03.379 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.594251+0000", 2020-01-16T00:14:03.379 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0071632730000000004 2020-01-16T00:14:03.379 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.379 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.379 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.380 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601414+0000", 2020-01-16T00:14:03.380 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4672999999999996e-05 2020-01-16T00:14:03.380 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.380 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.381 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.381 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601479+0000", 2020-01-16T00:14:03.381 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000158453 2020-01-16T00:14:03.381 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.381 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.381 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:03.381 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601638+0000", 2020-01-16T00:14:03.382 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8605e-05 2020-01-16T00:14:03.382 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.382 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.382 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:03.382 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601686+0000", 2020-01-16T00:14:03.382 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4283999999999998e-05 2020-01-16T00:14:03.383 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.383 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.383 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:03.383 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601730+0000", 2020-01-16T00:14:03.383 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016699964000000001 2020-01-16T00:14:03.383 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.383 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:03.383 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.384 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.384 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.384 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:126 2.1 2:a4df40e2:::smithi12313934-42 low_tier:head [write 70656~727552] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e45)", 2020-01-16T00:14:03.384 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.627078+0000", 2020-01-16T00:14:03.384 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 33.571388321999997, 2020-01-16T00:14:03.384 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010098995, 2020-01-16T00:14:03.384 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:03.385 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:03.385 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:03.385 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.385 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:03.385 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627078+0000", 2020-01-16T00:14:03.385 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:03.385 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.385 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.386 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:03.386 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627078+0000", 2020-01-16T00:14:03.386 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:03.386 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.386 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.386 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:03.386 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627076+0000", 2020-01-16T00:14:03.387 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019321599999999999 2020-01-16T00:14:03.387 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.388 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.389 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:03.389 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627269+0000", 2020-01-16T00:14:03.389 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8471999999999999e-05 2020-01-16T00:14:03.389 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.389 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.389 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:03.389 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627288+0000", 2020-01-16T00:14:03.390 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3199999999999996e-06 2020-01-16T00:14:03.390 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.390 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.390 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:03.390 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627294+0000", 2020-01-16T00:14:03.390 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3341000000000002e-05 2020-01-16T00:14:03.390 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.391 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.391 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:03.391 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627357+0000", 2020-01-16T00:14:03.391 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010985 2020-01-16T00:14:03.392 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.392 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.392 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:03.392 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627467+0000", 2020-01-16T00:14:03.392 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022914499999999999 2020-01-16T00:14:03.392 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.392 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.393 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:03.393 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627696+0000", 2020-01-16T00:14:03.393 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0055120000000000004 2020-01-16T00:14:03.393 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.393 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.393 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:03.394 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.633208+0000", 2020-01-16T00:14:03.394 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0019968569999999999 2020-01-16T00:14:03.394 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.394 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.394 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.394 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.635205+0000", 2020-01-16T00:14:03.395 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018113160000000001 2020-01-16T00:14:03.395 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.395 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.395 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.395 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.637016+0000", 2020-01-16T00:14:03.395 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9515e-05 2020-01-16T00:14:03.395 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.395 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.396 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:03.396 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.637096+0000", 2020-01-16T00:14:03.396 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1213999999999999e-05 2020-01-16T00:14:03.396 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.396 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.396 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:03.396 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.637177+0000", 2020-01-16T00:14:03.397 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010098995 2020-01-16T00:14:03.397 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.397 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:03.397 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.397 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.397 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.397 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:127 2.1 2:a4df40e2:::smithi12313934-42 low_tier:head [write 1364992~1145344] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e45)", 2020-01-16T00:14:03.398 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.627308+0000", 2020-01-16T00:14:03.398 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 33.571158525999998, 2020-01-16T00:14:03.398 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011068883999999999, 2020-01-16T00:14:03.398 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:03.398 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:03.398 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:03.398 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.398 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:03.399 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627308+0000", 2020-01-16T00:14:03.399 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:03.399 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.399 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.399 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:03.399 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627308+0000", 2020-01-16T00:14:03.399 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:03.400 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.400 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.400 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:03.400 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627307+0000", 2020-01-16T00:14:03.400 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000278597 2020-01-16T00:14:03.400 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.401 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.401 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:03.401 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627586+0000", 2020-01-16T00:14:03.402 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9169999999999997e-06 2020-01-16T00:14:03.402 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.402 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.402 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:03.402 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627591+0000", 2020-01-16T00:14:03.403 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0479999999999999e-06 2020-01-16T00:14:03.403 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.403 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.403 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:03.403 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627594+0000", 2020-01-16T00:14:03.403 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030987269999999999 2020-01-16T00:14:03.403 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.404 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.404 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:03.404 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.630693+0000", 2020-01-16T00:14:03.404 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000103688 2020-01-16T00:14:03.404 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.404 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.404 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:03.404 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.630796+0000", 2020-01-16T00:14:03.405 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000144606 2020-01-16T00:14:03.405 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.405 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.405 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:03.405 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.630941+0000", 2020-01-16T00:14:03.405 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039264269999999997 2020-01-16T00:14:03.405 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.406 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.406 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:03.406 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.634867+0000", 2020-01-16T00:14:03.406 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00038581200000000003 2020-01-16T00:14:03.406 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.406 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.406 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.407 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.635253+0000", 2020-01-16T00:14:03.407 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030568879999999998 2020-01-16T00:14:03.407 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.408 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.408 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.408 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.638310+0000", 2020-01-16T00:14:03.408 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8390999999999999e-05 2020-01-16T00:14:03.408 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.408 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.409 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:03.409 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.638348+0000", 2020-01-16T00:14:03.409 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8402999999999999e-05 2020-01-16T00:14:03.409 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.409 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.409 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:03.409 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.638377+0000", 2020-01-16T00:14:03.409 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011068883999999999 2020-01-16T00:14:03.410 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.410 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:03.410 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.410 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.410 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.410 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.6.13:30 2.3 2:fc791323:::smithi12313934-2 low_tier:head [write 476160~1240064] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e48)", 2020-01-16T00:14:03.410 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:39.466268+0000", 2020-01-16T00:14:03.410 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 23.732198222000001, 2020-01-16T00:14:03.411 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010593112, 2020-01-16T00:14:03.411 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:03.411 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:03.411 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:03.411 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.411 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:03.411 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.466268+0000", 2020-01-16T00:14:03.412 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:03.412 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.412 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.412 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:03.412 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.466268+0000", 2020-01-16T00:14:03.412 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:03.412 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.412 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.413 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:03.413 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.466266+0000", 2020-01-16T00:14:03.413 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012489210000000001 2020-01-16T00:14:03.413 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.413 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.413 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:03.414 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467515+0000", 2020-01-16T00:14:03.414 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8411999999999999e-05 2020-01-16T00:14:03.414 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.414 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.414 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:03.414 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467533+0000", 2020-01-16T00:14:03.415 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9319999999999996e-06 2020-01-16T00:14:03.415 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.415 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.415 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:03.415 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467541+0000", 2020-01-16T00:14:03.415 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5289999999999993e-05 2020-01-16T00:14:03.416 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.416 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.416 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:03.416 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467607+0000", 2020-01-16T00:14:03.416 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010962 2020-01-16T00:14:03.416 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.416 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.417 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:03.417 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467716+0000", 2020-01-16T00:14:03.417 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020883699999999999 2020-01-16T00:14:03.417 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.417 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.417 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 7", 2020-01-16T00:14:03.417 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467925+0000", 2020-01-16T00:14:03.418 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063872829999999997 2020-01-16T00:14:03.418 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.418 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.418 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:03.418 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.474312+0000", 2020-01-16T00:14:03.418 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0023165769999999998 2020-01-16T00:14:03.418 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.418 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.419 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.419 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.476629+0000", 2020-01-16T00:14:03.419 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8543999999999999e-05 2020-01-16T00:14:03.419 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.419 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.419 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:03.419 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.476707+0000", 2020-01-16T00:14:03.420 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000153914 2020-01-16T00:14:03.420 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.420 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.420 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:03.420 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.476861+0000", 2020-01-16T00:14:03.420 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010593112 2020-01-16T00:14:03.420 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.421 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:03.421 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.421 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.421 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.421 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:155 2.3 2:c9873fc7:::smithi12313934-30 low_tier:head [write 1542656~711168] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e50)", 2020-01-16T00:14:03.421 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:43.769943+0000", 2020-01-16T00:14:03.421 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 19.428523296000002, 2020-01-16T00:14:03.422 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010710127999999999, 2020-01-16T00:14:03.422 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:03.422 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:03.422 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:03.422 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.422 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:03.422 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.769943+0000", 2020-01-16T00:14:03.422 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:03.423 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.423 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.423 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:03.423 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.769943+0000", 2020-01-16T00:14:03.423 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:03.423 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.423 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.424 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:03.424 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.769942+0000", 2020-01-16T00:14:03.424 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017878100000000001 2020-01-16T00:14:03.424 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.424 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.424 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:03.424 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.770121+0000", 2020-01-16T00:14:03.424 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2980000000000003e-06 2020-01-16T00:14:03.425 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.425 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.425 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:03.425 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.770129+0000", 2020-01-16T00:14:03.425 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9129999999999999e-06 2020-01-16T00:14:03.426 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.426 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.426 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:03.426 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.770131+0000", 2020-01-16T00:14:03.426 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0053233519999999999 2020-01-16T00:14:03.426 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.427 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.427 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:03.427 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.775455+0000", 2020-01-16T00:14:03.427 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012385099999999999 2020-01-16T00:14:03.427 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.427 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.427 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:03.427 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.775579+0000", 2020-01-16T00:14:03.428 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000211554 2020-01-16T00:14:03.428 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.428 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.428 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 7", 2020-01-16T00:14:03.428 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.775790+0000", 2020-01-16T00:14:03.428 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029614210000000001 2020-01-16T00:14:03.428 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.429 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.429 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:03.429 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.778752+0000", 2020-01-16T00:14:03.429 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001798928 2020-01-16T00:14:03.429 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.429 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.429 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.429 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.780551+0000", 2020-01-16T00:14:03.430 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9413999999999999e-05 2020-01-16T00:14:03.430 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.430 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.430 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:03.430 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.780590+0000", 2020-01-16T00:14:03.430 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3275999999999995e-05 2020-01-16T00:14:03.430 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.431 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.431 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:03.431 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.780653+0000", 2020-01-16T00:14:03.431 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010710127999999999 2020-01-16T00:14:03.431 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.431 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:03.432 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.432 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.432 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.432 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:170 2.1 2:a935ad67:::smithi12313934-50 low_tier:head [write 1613312~498688] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e55)", 2020-01-16T00:14:03.432 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:58.196603+0000", 2020-01-16T00:14:03.432 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.0018635969999998, 2020-01-16T00:14:03.433 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0099186129999999997, 2020-01-16T00:14:03.433 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:03.433 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:03.433 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:03.433 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.433 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:03.433 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196603+0000", 2020-01-16T00:14:03.433 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:03.434 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.434 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.434 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:03.434 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196603+0000", 2020-01-16T00:14:03.434 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:03.434 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.434 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.435 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:03.435 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196602+0000", 2020-01-16T00:14:03.435 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000129679 2020-01-16T00:14:03.435 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.435 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.435 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:03.436 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196732+0000", 2020-01-16T00:14:03.436 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6699999999999994e-06 2020-01-16T00:14:03.436 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.436 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.436 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:03.436 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196739+0000", 2020-01-16T00:14:03.437 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8080000000000004e-06 2020-01-16T00:14:03.437 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.437 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.437 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:03.437 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196744+0000", 2020-01-16T00:14:03.437 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0015686980000000001 2020-01-16T00:14:03.437 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.438 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.438 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:03.438 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.198313+0000", 2020-01-16T00:14:03.438 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000120653 2020-01-16T00:14:03.438 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.438 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.438 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:03.439 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.198434+0000", 2020-01-16T00:14:03.439 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000183845 2020-01-16T00:14:03.439 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.439 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.439 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:03.439 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.198617+0000", 2020-01-16T00:14:03.439 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0074501710000000002 2020-01-16T00:14:03.440 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.440 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.440 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.441 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.206068+0000", 2020-01-16T00:14:03.441 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00031512800000000002 2020-01-16T00:14:03.441 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.441 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.441 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.441 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.206383+0000", 2020-01-16T00:14:03.441 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2570000000000007e-06 2020-01-16T00:14:03.441 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.442 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.442 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:03.442 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.206391+0000", 2020-01-16T00:14:03.442 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5562000000000006e-05 2020-01-16T00:14:03.442 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.442 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.442 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:03.443 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.206467+0000", 2020-01-16T00:14:03.443 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4911000000000003e-05 2020-01-16T00:14:03.443 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.443 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.443 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:03.443 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.206521+0000", 2020-01-16T00:14:03.443 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0099186129999999997 2020-01-16T00:14:03.443 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.444 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:03.444 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.444 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.444 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.444 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:171 2.1 2:a935ad67:::smithi12313934-50 low_tier:head [write 2422272~237568] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e55)", 2020-01-16T00:14:03.444 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:58.196749+0000", 2020-01-16T00:14:03.444 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.0017170110000002, 2020-01-16T00:14:03.445 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010677759, 2020-01-16T00:14:03.445 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:03.445 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:03.445 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:03.445 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.446 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:03.446 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196749+0000", 2020-01-16T00:14:03.446 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:03.446 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.446 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.446 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:03.447 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196749+0000", 2020-01-16T00:14:03.447 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:03.447 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.447 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.447 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:03.447 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196749+0000", 2020-01-16T00:14:03.447 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6604000000000001e-05 2020-01-16T00:14:03.448 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.448 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.448 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:03.448 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196815+0000", 2020-01-16T00:14:03.448 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4900000000000006e-06 2020-01-16T00:14:03.448 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.448 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.448 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:03.449 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196825+0000", 2020-01-16T00:14:03.449 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3989999999999997e-06 2020-01-16T00:14:03.449 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.449 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.449 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:03.450 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196829+0000", 2020-01-16T00:14:03.450 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0076943869999999996 2020-01-16T00:14:03.450 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.450 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.450 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:03.450 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.204524+0000", 2020-01-16T00:14:03.450 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000124745 2020-01-16T00:14:03.450 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.451 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.451 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:03.451 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.204649+0000", 2020-01-16T00:14:03.451 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018801700000000001 2020-01-16T00:14:03.451 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.451 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.451 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:03.452 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.204837+0000", 2020-01-16T00:14:03.452 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002354765 2020-01-16T00:14:03.452 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.452 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.452 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.452 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.207191+0000", 2020-01-16T00:14:03.452 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.6249999999999996e-06 2020-01-16T00:14:03.452 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.453 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.453 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:03.453 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.207200+0000", 2020-01-16T00:14:03.453 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000162246 2020-01-16T00:14:03.453 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.453 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.453 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.454 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.207362+0000", 2020-01-16T00:14:03.454 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5117999999999998e-05 2020-01-16T00:14:03.454 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.454 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.454 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:03.454 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.207397+0000", 2020-01-16T00:14:03.454 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9912000000000001e-05 2020-01-16T00:14:03.454 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:03.455 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:03.455 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:03.455 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.207427+0000", 2020-01-16T00:14:03.455 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010677759 2020-01-16T00:14:03.455 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.455 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:03.455 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.455 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:03.456 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:03.456 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:03.457 INFO:tasks.rados.rados.0.smithi123.stdout:1642: expect (ObjNum 561 snap 0 seq_num 561) 2020-01-16T00:14:03.458 INFO:tasks.rados.rados.0.smithi123.stdout:1644: finishing write tid 1 to smithi12313934-5 2020-01-16T00:14:03.458 INFO:tasks.rados.rados.0.smithi123.stdout:1644: finishing write tid 2 to smithi12313934-5 2020-01-16T00:14:03.458 INFO:tasks.rados.rados.0.smithi123.stdout:1644: finishing write tid 3 to smithi12313934-5 2020-01-16T00:14:03.458 INFO:tasks.rados.rados.0.smithi123.stdout:1644: finishing write tid 5 to smithi12313934-5 2020-01-16T00:14:03.458 INFO:tasks.rados.rados.0.smithi123.stdout:1644: finishing write tid 6 to smithi12313934-5 2020-01-16T00:14:03.458 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 5 v 261 (ObjNum 582 snap 0 seq_num 582) dirty exists 2020-01-16T00:14:03.459 INFO:tasks.rados.rados.0.smithi123.stdout:1644: left oid 5 (ObjNum 582 snap 0 seq_num 582) 2020-01-16T00:14:03.459 INFO:tasks.rados.rados.0.smithi123.stdout:1645: expect (ObjNum 554 snap 0 seq_num 554) 2020-01-16T00:14:03.459 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:03.452+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:03.459 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-01-16T00:14:03.558 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:03.554+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:03.576 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:03.576 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:14:03.576 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:14:03.577 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:03.586 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-01-16T00:14:03.658 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:03.654+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:03.664 INFO:tasks.rados.rados.0.smithi123.stdout:1647: expect (ObjNum 565 snap 0 seq_num 565) 2020-01-16T00:14:03.695 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:03.695 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:14:03.695 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:14:03.696 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:14:03.696 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:03.704 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-01-16T00:14:03.759 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:03.758+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:03.815 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:03.815 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:14:03.815 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:14:03.815 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:14:03.815 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.816 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:943 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:14:03.816 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:14:03.816 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 85.896432665000006, 2020-01-16T00:14:03.816 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018847004000000001, 2020-01-16T00:14:03.816 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:03.816 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:03.816 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:03.817 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:03.817 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:03.817 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 943 2020-01-16T00:14:03.817 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.817 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:03.817 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.817 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:03.817 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:14:03.818 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:03.818 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.818 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.818 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:03.818 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:14:03.818 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:03.818 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.819 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.819 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:03.819 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914475+0000", 2020-01-16T00:14:03.819 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.508e-05 2020-01-16T00:14:03.819 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.819 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.819 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:03.819 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914490+0000", 2020-01-16T00:14:03.820 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.857e-05 2020-01-16T00:14:03.820 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.820 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.820 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:03.820 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914509+0000", 2020-01-16T00:14:03.820 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.46e-06 2020-01-16T00:14:03.820 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.820 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.821 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:03.821 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914515+0000", 2020-01-16T00:14:03.821 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2612999999999999e-05 2020-01-16T00:14:03.821 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.821 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.821 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.821 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914568+0000", 2020-01-16T00:14:03.822 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028312299999999998 2020-01-16T00:14:03.822 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.822 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.822 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:03.822 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914851+0000", 2020-01-16T00:14:03.822 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012282533 2020-01-16T00:14:03.822 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.822 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.823 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.823 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927133+0000", 2020-01-16T00:14:03.823 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4839999999999995e-05 2020-01-16T00:14:03.823 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.823 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.823 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:03.823 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927208+0000", 2020-01-16T00:14:03.823 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0047425610000000002 2020-01-16T00:14:03.824 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.824 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.824 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.824 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.931951+0000", 2020-01-16T00:14:03.824 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3628999999999994e-05 2020-01-16T00:14:03.824 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.824 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.824 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:03.825 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.932045+0000", 2020-01-16T00:14:03.825 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012810180000000001 2020-01-16T00:14:03.825 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.825 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.825 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:03.825 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933326+0000", 2020-01-16T00:14:03.825 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018847004000000001 2020-01-16T00:14:03.826 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:03.826 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:03.826 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:03.826 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.826 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.826 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:944 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:14:03.826 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:14:03.826 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 85.896383108999999, 2020-01-16T00:14:03.827 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019086901999999999, 2020-01-16T00:14:03.827 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:03.827 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:03.827 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:03.827 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:03.827 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:03.827 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 944 2020-01-16T00:14:03.828 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.828 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:03.828 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.828 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:03.828 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:14:03.828 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:03.828 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.828 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.829 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:03.829 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:14:03.829 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:03.829 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.829 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.829 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:03.829 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914527+0000", 2020-01-16T00:14:03.829 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7509999999999999e-06 2020-01-16T00:14:03.830 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.830 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.830 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:03.830 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914532+0000", 2020-01-16T00:14:03.830 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5149999999999997e-06 2020-01-16T00:14:03.830 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.830 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.831 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:03.831 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914538+0000", 2020-01-16T00:14:03.831 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1890000000000007e-06 2020-01-16T00:14:03.831 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.831 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.831 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:03.831 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914546+0000", 2020-01-16T00:14:03.831 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000354951 2020-01-16T00:14:03.832 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.832 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.832 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.832 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914901+0000", 2020-01-16T00:14:03.832 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1594000000000002e-05 2020-01-16T00:14:03.832 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.832 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.832 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:03.833 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914942+0000", 2020-01-16T00:14:03.833 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012300070999999999 2020-01-16T00:14:03.833 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.833 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.833 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.833 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927242+0000", 2020-01-16T00:14:03.833 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4076999999999997e-05 2020-01-16T00:14:03.833 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.834 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.834 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:03.834 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927296+0000", 2020-01-16T00:14:03.834 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0060609180000000002 2020-01-16T00:14:03.834 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.834 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.834 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.835 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933357+0000", 2020-01-16T00:14:03.835 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4488999999999999e-05 2020-01-16T00:14:03.835 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.835 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.835 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:03.835 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933422+0000", 2020-01-16T00:14:03.835 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019318300000000001 2020-01-16T00:14:03.835 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.836 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.836 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:03.836 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933615+0000", 2020-01-16T00:14:03.836 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019086901999999999 2020-01-16T00:14:03.836 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:03.836 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:03.836 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:03.836 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.837 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.837 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:945 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:14:03.837 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:14:03.837 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 85.896355903, 2020-01-16T00:14:03.837 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019342986, 2020-01-16T00:14:03.837 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:03.837 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:03.837 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:03.838 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:03.838 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:03.838 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 945 2020-01-16T00:14:03.838 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.838 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:03.838 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.838 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:03.839 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:14:03.839 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:03.839 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.839 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.839 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:03.839 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:14:03.839 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:03.839 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.840 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.840 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:03.840 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:14:03.840 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4580000000000002e-06 2020-01-16T00:14:03.840 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.840 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.840 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:03.841 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914558+0000", 2020-01-16T00:14:03.841 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7829999999999998e-06 2020-01-16T00:14:03.841 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.841 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.841 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:03.841 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914563+0000", 2020-01-16T00:14:03.841 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0001e-05 2020-01-16T00:14:03.841 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.842 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.842 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:03.842 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914573+0000", 2020-01-16T00:14:03.842 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00040365600000000001 2020-01-16T00:14:03.842 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.842 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.842 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.842 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914976+0000", 2020-01-16T00:14:03.843 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1390000000000001e-05 2020-01-16T00:14:03.843 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.843 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.843 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:03.843 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.915028+0000", 2020-01-16T00:14:03.843 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012288843000000001 2020-01-16T00:14:03.843 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.844 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.844 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.844 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927317+0000", 2020-01-16T00:14:03.844 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7871000000000002e-05 2020-01-16T00:14:03.844 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.844 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.844 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:03.844 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927375+0000", 2020-01-16T00:14:03.845 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0062580589999999998 2020-01-16T00:14:03.845 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.845 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.845 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.845 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933633+0000", 2020-01-16T00:14:03.845 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7086999999999997e-05 2020-01-16T00:14:03.845 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.845 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.846 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:03.846 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933690+0000", 2020-01-16T00:14:03.846 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020855799999999999 2020-01-16T00:14:03.846 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.846 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.846 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:03.846 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933898+0000", 2020-01-16T00:14:03.846 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019342986 2020-01-16T00:14:03.847 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:03.847 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:03.847 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:03.847 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.847 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.847 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:981 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:14:03.847 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:14:03.847 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 85.495579011000004, 2020-01-16T00:14:03.848 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021222494000000001, 2020-01-16T00:14:03.848 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:03.848 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:03.848 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:03.848 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:03.848 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:03.848 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 981 2020-01-16T00:14:03.849 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.849 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:03.849 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.849 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:03.849 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:14:03.849 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:03.849 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.849 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.850 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:03.850 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:14:03.850 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999948 2020-01-16T00:14:03.850 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.850 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.850 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:03.850 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315327+0000", 2020-01-16T00:14:03.850 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.508e-05 2020-01-16T00:14:03.851 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.851 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.851 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:03.851 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315352+0000", 2020-01-16T00:14:03.851 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1183999999999998e-05 2020-01-16T00:14:03.851 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.851 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.851 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:03.852 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315383+0000", 2020-01-16T00:14:03.852 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7180000000000002e-05 2020-01-16T00:14:03.852 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.852 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.852 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:03.852 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315401+0000", 2020-01-16T00:14:03.852 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017290599999999999 2020-01-16T00:14:03.853 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.853 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.853 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.853 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315574+0000", 2020-01-16T00:14:03.853 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00047908100000000002 2020-01-16T00:14:03.853 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.853 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.853 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:03.854 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316053+0000", 2020-01-16T00:14:03.854 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011808947 2020-01-16T00:14:03.854 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.854 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.854 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.854 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.327862+0000", 2020-01-16T00:14:03.854 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010067499999999999 2020-01-16T00:14:03.854 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.855 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.855 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:03.855 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.327962+0000", 2020-01-16T00:14:03.855 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0074745139999999998 2020-01-16T00:14:03.855 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.855 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.855 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.856 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.335437+0000", 2020-01-16T00:14:03.856 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000103403 2020-01-16T00:14:03.856 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.856 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.856 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:03.856 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.335540+0000", 2020-01-16T00:14:03.856 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001014544 2020-01-16T00:14:03.856 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.857 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.857 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:03.857 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336555+0000", 2020-01-16T00:14:03.857 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021222494000000001 2020-01-16T00:14:03.857 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:03.857 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:03.857 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:03.857 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.858 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.858 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:982 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:14:03.858 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:14:03.858 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 85.495476487000005, 2020-01-16T00:14:03.858 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.022087721000000001, 2020-01-16T00:14:03.858 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:03.858 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:03.858 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:03.859 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:03.859 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:03.859 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 982 2020-01-16T00:14:03.859 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.859 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:03.859 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.859 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:03.859 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:14:03.860 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:03.860 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.860 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.860 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:03.860 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:14:03.860 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:03.860 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.861 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.861 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:03.861 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315433+0000", 2020-01-16T00:14:03.861 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1347e-05 2020-01-16T00:14:03.861 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.861 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.861 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:03.861 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315445+0000", 2020-01-16T00:14:03.862 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4525999999999999e-05 2020-01-16T00:14:03.862 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.862 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.862 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:03.862 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315459+0000", 2020-01-16T00:14:03.862 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.024e-05 2020-01-16T00:14:03.862 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.862 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.863 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:03.863 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315470+0000", 2020-01-16T00:14:03.863 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000736468 2020-01-16T00:14:03.863 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.863 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.863 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.863 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316206+0000", 2020-01-16T00:14:03.863 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000132967 2020-01-16T00:14:03.864 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.864 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.864 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:03.864 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316339+0000", 2020-01-16T00:14:03.864 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011701717 2020-01-16T00:14:03.864 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.864 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.864 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.865 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328041+0000", 2020-01-16T00:14:03.865 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000120339 2020-01-16T00:14:03.865 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.865 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.865 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:03.865 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328161+0000", 2020-01-16T00:14:03.865 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0084341260000000001 2020-01-16T00:14:03.866 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.866 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.866 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.866 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336595+0000", 2020-01-16T00:14:03.866 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5247999999999993e-05 2020-01-16T00:14:03.866 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.866 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.866 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:03.867 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336690+0000", 2020-01-16T00:14:03.867 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00083204000000000004 2020-01-16T00:14:03.867 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.867 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.867 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:03.867 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337522+0000", 2020-01-16T00:14:03.867 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.022087721000000001 2020-01-16T00:14:03.868 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:03.868 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:03.868 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:03.868 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.868 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.868 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:983 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:14:03.868 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:14:03.868 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 85.495356776999998, 2020-01-16T00:14:03.869 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023363387999999999, 2020-01-16T00:14:03.869 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:03.869 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:03.869 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:03.869 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:03.869 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:03.869 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 983 2020-01-16T00:14:03.869 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.870 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:03.870 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.870 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:03.870 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:14:03.870 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:03.870 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.870 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.870 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:03.871 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:14:03.871 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:03.871 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.871 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.871 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:03.871 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315553+0000", 2020-01-16T00:14:03.871 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1760000000000004e-06 2020-01-16T00:14:03.871 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.872 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.872 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:03.872 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315560+0000", 2020-01-16T00:14:03.872 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3587000000000001e-05 2020-01-16T00:14:03.872 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.872 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.872 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:03.873 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315574+0000", 2020-01-16T00:14:03.873 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7800000000000003e-06 2020-01-16T00:14:03.873 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.873 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.873 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:03.873 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315581+0000", 2020-01-16T00:14:03.873 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000823024 2020-01-16T00:14:03.873 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.874 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.874 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.874 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316404+0000", 2020-01-16T00:14:03.874 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2413e-05 2020-01-16T00:14:03.874 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.874 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.874 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:03.874 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316466+0000", 2020-01-16T00:14:03.875 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011738561999999999 2020-01-16T00:14:03.875 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.875 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.875 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.875 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328205+0000", 2020-01-16T00:14:03.875 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6783999999999997e-05 2020-01-16T00:14:03.875 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.875 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.876 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:03.876 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328262+0000", 2020-01-16T00:14:03.876 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0093022560000000001 2020-01-16T00:14:03.876 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.876 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.876 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.876 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337564+0000", 2020-01-16T00:14:03.877 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000183151 2020-01-16T00:14:03.877 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.877 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.877 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:03.877 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337747+0000", 2020-01-16T00:14:03.877 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011707449999999999 2020-01-16T00:14:03.877 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.877 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.878 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:03.878 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.338918+0000", 2020-01-16T00:14:03.878 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023363387999999999 2020-01-16T00:14:03.878 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:03.878 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:03.878 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:03.878 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.878 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.879 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:986 3.9 3:93dfe6f1:::smithi12313934-40:head [write 2941380~419987] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:14:03.879 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:14:03.879 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 85.330057001, 2020-01-16T00:14:03.879 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016978400000000001, 2020-01-16T00:14:03.879 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:03.879 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:03.879 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:03.879 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:03.880 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:03.880 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 986 2020-01-16T00:14:03.880 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.880 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:03.880 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.880 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:03.880 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:14:03.881 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:03.881 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.881 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.881 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:03.881 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:14:03.881 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:03.881 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.881 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.882 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:03.882 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480852+0000", 2020-01-16T00:14:03.882 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042503899999999999 2020-01-16T00:14:03.882 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.882 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.882 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:03.882 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481277+0000", 2020-01-16T00:14:03.882 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3097e-05 2020-01-16T00:14:03.883 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.883 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.883 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:03.883 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481290+0000", 2020-01-16T00:14:03.883 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8460000000000002e-06 2020-01-16T00:14:03.883 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.883 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.883 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:03.884 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481297+0000", 2020-01-16T00:14:03.884 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8443999999999999e-05 2020-01-16T00:14:03.884 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.884 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.884 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.884 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481346+0000", 2020-01-16T00:14:03.884 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022115399999999999 2020-01-16T00:14:03.884 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.885 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.885 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:03.885 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481567+0000", 2020-01-16T00:14:03.885 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012531502 2020-01-16T00:14:03.885 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.885 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.885 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.886 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494098+0000", 2020-01-16T00:14:03.886 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.4462999999999993e-05 2020-01-16T00:14:03.886 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.886 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.886 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:03.886 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494193+0000", 2020-01-16T00:14:03.886 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017600999999999999 2020-01-16T00:14:03.886 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.887 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.887 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:14:03.887 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494369+0000", 2020-01-16T00:14:03.887 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0031739619999999998 2020-01-16T00:14:03.887 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.887 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.887 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:03.887 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497543+0000", 2020-01-16T00:14:03.888 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000175107 2020-01-16T00:14:03.888 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.888 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.888 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.888 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497718+0000", 2020-01-16T00:14:03.888 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5207000000000003e-05 2020-01-16T00:14:03.888 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.888 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.889 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.889 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497763+0000", 2020-01-16T00:14:03.889 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7042999999999998e-05 2020-01-16T00:14:03.889 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.889 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.889 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:03.889 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497800+0000", 2020-01-16T00:14:03.890 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2628e-05 2020-01-16T00:14:03.890 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.890 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.890 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:03.890 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497833+0000", 2020-01-16T00:14:03.890 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016978400000000001 2020-01-16T00:14:03.890 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:03.890 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:03.891 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:03.891 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.891 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.891 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:988 3.9 3:93dfe6f1:::smithi12313934-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:14:03.891 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:14:03.891 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 85.329571537000007, 2020-01-16T00:14:03.891 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017221692, 2020-01-16T00:14:03.891 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:03.892 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:03.892 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:03.892 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:03.892 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:03.892 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 988 2020-01-16T00:14:03.892 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.892 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:03.893 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.893 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:03.893 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:14:03.893 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:03.893 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.893 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.893 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:03.893 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:14:03.894 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:03.894 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.894 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.894 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:03.894 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481339+0000", 2020-01-16T00:14:03.894 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3089999999999997e-06 2020-01-16T00:14:03.894 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.895 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.895 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:03.895 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481343+0000", 2020-01-16T00:14:03.895 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2230000000000004e-06 2020-01-16T00:14:03.895 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.895 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.895 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:03.895 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481350+0000", 2020-01-16T00:14:03.896 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.319e-06 2020-01-16T00:14:03.896 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.896 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.896 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:03.896 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481356+0000", 2020-01-16T00:14:03.896 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000321544 2020-01-16T00:14:03.896 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.897 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.897 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.897 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481678+0000", 2020-01-16T00:14:03.897 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6375999999999999e-05 2020-01-16T00:14:03.897 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.897 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.897 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:03.897 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481714+0000", 2020-01-16T00:14:03.898 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015557666 2020-01-16T00:14:03.898 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.898 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.898 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.898 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497272+0000", 2020-01-16T00:14:03.898 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012791300000000001 2020-01-16T00:14:03.898 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.898 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.899 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:03.899 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497399+0000", 2020-01-16T00:14:03.899 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00076944600000000002 2020-01-16T00:14:03.899 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.899 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.899 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.899 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498169+0000", 2020-01-16T00:14:03.900 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018306799999999999 2020-01-16T00:14:03.900 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.900 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.900 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:03.900 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498352+0000", 2020-01-16T00:14:03.900 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020935999999999999 2020-01-16T00:14:03.900 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.900 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.901 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:03.901 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498561+0000", 2020-01-16T00:14:03.901 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017221692 2020-01-16T00:14:03.901 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:03.901 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:03.901 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:03.901 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.901 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.902 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1239 3.3 3:c204aa4f:::smithi12313934-45:head [setxattr _header (56) in=7b,truncate 2000033] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:14:03.902 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:14:03.902 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 81.220497718000004, 2020-01-16T00:14:03.902 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019161661999999999, 2020-01-16T00:14:03.902 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:03.902 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:03.902 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:03.902 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:03.903 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:03.903 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1239 2020-01-16T00:14:03.903 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.903 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:03.903 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.903 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:03.903 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:14:03.904 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:03.904 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.904 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.904 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:03.904 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:14:03.904 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:03.904 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.904 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.905 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:03.905 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590411+0000", 2020-01-16T00:14:03.905 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3879999999999999e-05 2020-01-16T00:14:03.905 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.905 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.905 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:03.905 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590425+0000", 2020-01-16T00:14:03.905 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6542999999999998e-05 2020-01-16T00:14:03.906 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.906 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.906 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:03.906 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590471+0000", 2020-01-16T00:14:03.906 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8907999999999999e-05 2020-01-16T00:14:03.906 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.906 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.907 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:03.907 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590490+0000", 2020-01-16T00:14:03.907 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0045691960000000002 2020-01-16T00:14:03.907 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.907 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.907 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.907 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.595059+0000", 2020-01-16T00:14:03.907 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0035638940000000002 2020-01-16T00:14:03.908 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.908 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.908 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:03.908 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598623+0000", 2020-01-16T00:14:03.908 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0077558310000000004 2020-01-16T00:14:03.908 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.908 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.908 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.909 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606379+0000", 2020-01-16T00:14:03.909 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012171999999999999 2020-01-16T00:14:03.909 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.909 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.909 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:03.909 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606501+0000", 2020-01-16T00:14:03.909 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013144200000000001 2020-01-16T00:14:03.909 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.910 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.910 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2020-01-16T00:14:03.910 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606632+0000", 2020-01-16T00:14:03.910 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001288565 2020-01-16T00:14:03.910 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.910 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.910 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:03.911 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607921+0000", 2020-01-16T00:14:03.911 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033399299999999999 2020-01-16T00:14:03.911 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.911 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.911 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.911 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.608255+0000", 2020-01-16T00:14:03.911 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012280870000000001 2020-01-16T00:14:03.911 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.912 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.912 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.912 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609483+0000", 2020-01-16T00:14:03.912 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2286000000000001e-05 2020-01-16T00:14:03.912 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.912 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.912 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:03.912 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609525+0000", 2020-01-16T00:14:03.913 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0121999999999999e-05 2020-01-16T00:14:03.913 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.913 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.913 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:03.913 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609575+0000", 2020-01-16T00:14:03.913 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019161661999999999 2020-01-16T00:14:03.913 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:03.913 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:03.914 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:03.914 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.914 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.914 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1240 3.3 3:c204aa4f:::smithi12313934-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-01-16T00:14:03.914 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:14:03.914 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 81.220397528999996, 2020-01-16T00:14:03.914 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019268828000000002, 2020-01-16T00:14:03.915 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:03.915 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:03.915 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:03.915 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:03.915 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:03.916 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1240 2020-01-16T00:14:03.916 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.916 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:03.916 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.916 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:03.916 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:14:03.916 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:03.916 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.917 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.917 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:03.917 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:14:03.917 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:03.917 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.917 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.917 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:03.918 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590512+0000", 2020-01-16T00:14:03.918 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0525e-05 2020-01-16T00:14:03.918 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.918 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.918 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:03.918 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590522+0000", 2020-01-16T00:14:03.918 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6418000000000001e-05 2020-01-16T00:14:03.918 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.919 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.919 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:03.919 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590539+0000", 2020-01-16T00:14:03.919 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0679999999999999e-05 2020-01-16T00:14:03.919 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.919 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.919 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:03.919 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590549+0000", 2020-01-16T00:14:03.920 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0083626789999999996 2020-01-16T00:14:03.920 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.920 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.920 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.920 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598912+0000", 2020-01-16T00:14:03.920 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013344 2020-01-16T00:14:03.920 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.920 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.921 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:03.921 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599045+0000", 2020-01-16T00:14:03.921 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0084002269999999997 2020-01-16T00:14:03.921 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.921 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.921 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.921 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607446+0000", 2020-01-16T00:14:03.922 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000109873 2020-01-16T00:14:03.922 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.922 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.922 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:03.922 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607555+0000", 2020-01-16T00:14:03.922 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0020573309999999999 2020-01-16T00:14:03.922 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.922 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.923 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.923 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609613+0000", 2020-01-16T00:14:03.923 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6351e-05 2020-01-16T00:14:03.923 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.923 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.923 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:03.923 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609699+0000", 2020-01-16T00:14:03.923 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3374999999999994e-05 2020-01-16T00:14:03.924 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.924 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.924 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:03.924 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609782+0000", 2020-01-16T00:14:03.924 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019268828000000002 2020-01-16T00:14:03.924 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:03.924 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:03.924 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:03.925 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.925 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.925 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3124 3.9 3:9fc687b2:::smithi12313934-44:head [write 1903628~435462] snapc 0=[] ondisk+write+known_if_redirected e44)", 2020-01-16T00:14:03.925 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.276145+0000", 2020-01-16T00:14:03.925 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 38.534765808000003, 2020-01-16T00:14:03.925 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017361898000000001, 2020-01-16T00:14:03.925 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:03.926 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:03.926 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:03.926 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:03.926 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:03.926 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3124 2020-01-16T00:14:03.926 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.926 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:03.926 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.927 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:03.927 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276145+0000", 2020-01-16T00:14:03.927 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:03.927 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.927 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.927 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:03.927 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276145+0000", 2020-01-16T00:14:03.927 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:03.928 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.928 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.928 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:03.928 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276143+0000", 2020-01-16T00:14:03.928 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042136099999999998 2020-01-16T00:14:03.928 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.928 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.928 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:03.929 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276564+0000", 2020-01-16T00:14:03.929 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.349e-05 2020-01-16T00:14:03.929 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.929 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.929 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:03.929 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276578+0000", 2020-01-16T00:14:03.929 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2469999999999999e-06 2020-01-16T00:14:03.929 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.930 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.930 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:03.930 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276583+0000", 2020-01-16T00:14:03.930 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4400000000000001e-05 2020-01-16T00:14:03.930 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.930 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.930 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.930 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276637+0000", 2020-01-16T00:14:03.931 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0046361789999999998 2020-01-16T00:14:03.931 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.931 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.931 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:03.931 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281274+0000", 2020-01-16T00:14:03.931 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0090025520000000005 2020-01-16T00:14:03.931 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.931 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.932 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.932 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.290276+0000", 2020-01-16T00:14:03.932 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011584200000000001 2020-01-16T00:14:03.932 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.932 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.932 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:03.932 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.290392+0000", 2020-01-16T00:14:03.933 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00025563099999999998 2020-01-16T00:14:03.933 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.933 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.933 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:14:03.933 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.290648+0000", 2020-01-16T00:14:03.933 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0026741360000000001 2020-01-16T00:14:03.933 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.933 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.934 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:03.934 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293322+0000", 2020-01-16T00:14:03.934 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9764000000000002e-05 2020-01-16T00:14:03.934 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.934 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.934 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.934 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293421+0000", 2020-01-16T00:14:03.934 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4779000000000001e-05 2020-01-16T00:14:03.935 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.935 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.935 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.935 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293446+0000", 2020-01-16T00:14:03.935 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4628000000000001e-05 2020-01-16T00:14:03.935 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.935 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.935 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:03.936 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293481+0000", 2020-01-16T00:14:03.936 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.6444999999999998e-05 2020-01-16T00:14:03.936 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.936 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.936 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:03.936 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293507+0000", 2020-01-16T00:14:03.936 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017361898000000001 2020-01-16T00:14:03.936 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:03.937 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:03.937 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:03.937 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.937 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.937 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3125 3.9 3:9fc687b2:::smithi12313934-44:head [setxattr _header (56) in=7b,truncate 2386660] snapc 0=[] ondisk+write+known_if_redirected e44)", 2020-01-16T00:14:03.937 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.276594+0000", 2020-01-16T00:14:03.937 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 38.534317035999997, 2020-01-16T00:14:03.937 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017760560000000002, 2020-01-16T00:14:03.938 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:03.938 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:03.938 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:03.938 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:03.938 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:03.938 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3125 2020-01-16T00:14:03.938 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.939 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:03.939 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.939 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:03.939 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276594+0000", 2020-01-16T00:14:03.939 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:03.939 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.939 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.939 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:03.940 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276594+0000", 2020-01-16T00:14:03.940 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:03.940 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.940 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.940 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:03.940 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276593+0000", 2020-01-16T00:14:03.940 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7320000000000003e-06 2020-01-16T00:14:03.940 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.941 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.941 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:03.941 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276599+0000", 2020-01-16T00:14:03.941 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3302000000000001e-05 2020-01-16T00:14:03.941 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.941 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.941 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:03.942 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276612+0000", 2020-01-16T00:14:03.942 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1659999999999998e-06 2020-01-16T00:14:03.942 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.942 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.942 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:03.942 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276615+0000", 2020-01-16T00:14:03.942 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0047328450000000003 2020-01-16T00:14:03.942 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.943 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.943 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.943 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281348+0000", 2020-01-16T00:14:03.943 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3966000000000003e-05 2020-01-16T00:14:03.943 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.943 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.943 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:03.943 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281422+0000", 2020-01-16T00:14:03.944 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01082613 2020-01-16T00:14:03.944 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.944 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.944 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.944 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.292248+0000", 2020-01-16T00:14:03.944 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000148617 2020-01-16T00:14:03.944 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.944 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.945 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:03.945 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.292397+0000", 2020-01-16T00:14:03.945 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019936300000000001 2020-01-16T00:14:03.945 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.945 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.945 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:14:03.945 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.292596+0000", 2020-01-16T00:14:03.945 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011208870000000001 2020-01-16T00:14:03.946 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.946 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.946 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.946 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293717+0000", 2020-01-16T00:14:03.946 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00039491599999999998 2020-01-16T00:14:03.946 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.946 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.947 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:03.947 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294112+0000", 2020-01-16T00:14:03.947 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000129326 2020-01-16T00:14:03.947 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.947 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.947 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.947 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294241+0000", 2020-01-16T00:14:03.947 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5586000000000001e-05 2020-01-16T00:14:03.948 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.948 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.948 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:03.948 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294277+0000", 2020-01-16T00:14:03.948 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7893999999999997e-05 2020-01-16T00:14:03.948 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.948 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.948 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:03.949 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294355+0000", 2020-01-16T00:14:03.949 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017760560000000002 2020-01-16T00:14:03.949 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:03.949 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:03.949 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:03.949 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.949 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.949 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3126 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e44)", 2020-01-16T00:14:03.950 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.276634+0000", 2020-01-16T00:14:03.950 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 38.534277320999998, 2020-01-16T00:14:03.950 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017942583000000002, 2020-01-16T00:14:03.950 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:03.950 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:03.950 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:03.950 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:03.951 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:03.951 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3126 2020-01-16T00:14:03.951 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.951 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:03.951 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.951 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:03.951 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276634+0000", 2020-01-16T00:14:03.951 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:03.952 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.952 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.952 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:03.952 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276634+0000", 2020-01-16T00:14:03.952 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:03.952 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.952 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.952 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:03.953 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276633+0000", 2020-01-16T00:14:03.953 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0769999999999999e-06 2020-01-16T00:14:03.953 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.953 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.953 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:03.953 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276636+0000", 2020-01-16T00:14:03.953 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5390000000000001e-06 2020-01-16T00:14:03.953 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.954 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.954 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:03.954 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276641+0000", 2020-01-16T00:14:03.954 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9969999999999999e-06 2020-01-16T00:14:03.954 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.954 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.954 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:03.955 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276644+0000", 2020-01-16T00:14:03.955 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0048246970000000002 2020-01-16T00:14:03.955 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.955 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.955 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.955 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281469+0000", 2020-01-16T00:14:03.955 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8723999999999999e-05 2020-01-16T00:14:03.955 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.956 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.956 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:03.956 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281567+0000", 2020-01-16T00:14:03.956 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011633447 2020-01-16T00:14:03.956 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.956 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.956 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.956 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293201+0000", 2020-01-16T00:14:03.957 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6796000000000002e-05 2020-01-16T00:14:03.957 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.957 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.957 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:03.957 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293268+0000", 2020-01-16T00:14:03.957 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001125247 2020-01-16T00:14:03.957 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.957 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.958 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.958 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294393+0000", 2020-01-16T00:14:03.958 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.6082000000000006e-05 2020-01-16T00:14:03.958 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.958 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.958 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:03.958 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294489+0000", 2020-01-16T00:14:03.958 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7515000000000005e-05 2020-01-16T00:14:03.959 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.959 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.959 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:03.959 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294576+0000", 2020-01-16T00:14:03.959 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017942583000000002 2020-01-16T00:14:03.959 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:03.959 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:03.959 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:03.960 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.960 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.960 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3204 3.9 3:93dfe6f1:::smithi12313934-40:head [setxattr _header (56) in=7b,truncate 2554730] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:14:03.960 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.568544+0000", 2020-01-16T00:14:03.960 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 34.242367100999999, 2020-01-16T00:14:03.960 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.026260867, 2020-01-16T00:14:03.960 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:03.961 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:03.961 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:03.961 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:03.961 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:03.961 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3204 2020-01-16T00:14:03.961 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.961 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:03.961 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.962 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:03.962 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568544+0000", 2020-01-16T00:14:03.962 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:03.962 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.962 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.962 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:03.962 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568544+0000", 2020-01-16T00:14:03.962 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:03.963 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.963 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.963 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:03.963 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568543+0000", 2020-01-16T00:14:03.963 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0810000000000001e-06 2020-01-16T00:14:03.963 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.963 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.963 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:03.964 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568548+0000", 2020-01-16T00:14:03.964 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6950000000000002e-06 2020-01-16T00:14:03.964 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.964 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.964 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:03.964 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568552+0000", 2020-01-16T00:14:03.964 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.875e-06 2020-01-16T00:14:03.965 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.965 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.965 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:03.965 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568555+0000", 2020-01-16T00:14:03.965 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0033130849999999999 2020-01-16T00:14:03.965 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.965 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.965 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.966 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.571868+0000", 2020-01-16T00:14:03.966 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0026014609999999998 2020-01-16T00:14:03.966 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.966 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.966 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:03.966 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.574469+0000", 2020-01-16T00:14:03.966 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010261381999999999 2020-01-16T00:14:03.966 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.967 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.967 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.967 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.584731+0000", 2020-01-16T00:14:03.967 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019704 2020-01-16T00:14:03.967 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.967 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.967 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:03.967 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.584928+0000", 2020-01-16T00:14:03.968 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000280869 2020-01-16T00:14:03.968 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.968 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.968 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:14:03.968 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.585209+0000", 2020-01-16T00:14:03.968 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002273859 2020-01-16T00:14:03.968 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.969 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.969 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.969 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.587483+0000", 2020-01-16T00:14:03.969 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3356000000000001e-05 2020-01-16T00:14:03.969 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.969 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.969 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:03.969 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.587506+0000", 2020-01-16T00:14:03.970 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0071864939999999999 2020-01-16T00:14:03.970 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.970 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.970 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:03.970 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594692+0000", 2020-01-16T00:14:03.970 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5475000000000001e-05 2020-01-16T00:14:03.970 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.970 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.971 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:03.971 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594738+0000", 2020-01-16T00:14:03.971 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7070000000000007e-05 2020-01-16T00:14:03.971 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.971 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.971 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:03.971 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594805+0000", 2020-01-16T00:14:03.971 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.026260867 2020-01-16T00:14:03.972 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:03.972 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:03.972 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:03.972 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.972 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.972 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3205 3.9 3:93dfe6f1:::smithi12313934-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e45)", 2020-01-16T00:14:03.972 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.568599+0000", 2020-01-16T00:14:03.973 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 34.242312165999998, 2020-01-16T00:14:03.973 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.026406882, 2020-01-16T00:14:03.973 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:03.973 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:03.973 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:03.973 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:03.973 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:03.973 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3205 2020-01-16T00:14:03.974 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.974 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:03.974 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.974 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:03.974 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568599+0000", 2020-01-16T00:14:03.974 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:03.974 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.974 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.975 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:03.975 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568599+0000", 2020-01-16T00:14:03.975 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:03.975 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.975 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.975 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:03.975 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568598+0000", 2020-01-16T00:14:03.975 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3939999999999998e-06 2020-01-16T00:14:03.976 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.976 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.976 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:03.976 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568603+0000", 2020-01-16T00:14:03.976 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9391000000000001e-05 2020-01-16T00:14:03.976 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.976 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.977 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:03.977 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568622+0000", 2020-01-16T00:14:03.977 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9599999999999997e-06 2020-01-16T00:14:03.977 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.977 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.977 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:03.977 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568628+0000", 2020-01-16T00:14:03.977 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0058679700000000001 2020-01-16T00:14:03.978 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.978 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.978 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.978 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.574496+0000", 2020-01-16T00:14:03.978 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5528999999999998e-05 2020-01-16T00:14:03.978 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.978 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.978 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:03.979 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.574532+0000", 2020-01-16T00:14:03.979 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011986749 2020-01-16T00:14:03.979 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.979 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.979 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.979 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.586518+0000", 2020-01-16T00:14:03.979 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016522600000000001 2020-01-16T00:14:03.979 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.980 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.980 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:03.980 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.586684+0000", 2020-01-16T00:14:03.980 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0081693890000000009 2020-01-16T00:14:03.980 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.980 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.980 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.980 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594853+0000", 2020-01-16T00:14:03.981 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7082000000000004e-05 2020-01-16T00:14:03.981 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.981 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.981 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:03.981 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594940+0000", 2020-01-16T00:14:03.981 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5789000000000003e-05 2020-01-16T00:14:03.981 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.982 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.982 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:03.982 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.595006+0000", 2020-01-16T00:14:03.982 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.026406882 2020-01-16T00:14:03.982 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:03.982 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:03.982 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:03.982 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.983 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.983 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3219 3.3 3:c478bfe8:::smithi12313934-10:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:03.983 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.580957+0000", 2020-01-16T00:14:03.983 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 34.229954650000003, 2020-01-16T00:14:03.983 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016807971000000001, 2020-01-16T00:14:03.983 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:03.983 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:03.983 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:03.984 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:03.984 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:03.984 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3219 2020-01-16T00:14:03.984 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.984 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:03.984 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.984 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:03.984 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580957+0000", 2020-01-16T00:14:03.985 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:03.985 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.985 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.985 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:03.985 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580957+0000", 2020-01-16T00:14:03.985 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:03.985 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.985 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.986 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:03.986 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580955+0000", 2020-01-16T00:14:03.986 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5120000000000002e-06 2020-01-16T00:14:03.986 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.986 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.986 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:03.986 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580963+0000", 2020-01-16T00:14:03.986 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6799999999999993e-06 2020-01-16T00:14:03.987 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.987 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.987 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:03.987 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580971+0000", 2020-01-16T00:14:03.987 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3109999999999996e-06 2020-01-16T00:14:03.987 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.987 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.988 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:03.988 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580976+0000", 2020-01-16T00:14:03.988 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014306146 2020-01-16T00:14:03.988 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.988 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.988 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.988 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.595282+0000", 2020-01-16T00:14:03.988 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.1883000000000001e-05 2020-01-16T00:14:03.989 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.989 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.989 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:03.989 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.595374+0000", 2020-01-16T00:14:03.989 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023904989999999999 2020-01-16T00:14:03.989 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.989 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.989 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:03.990 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.597765+0000", 2020-01-16T00:14:03.990 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016807971000000001 2020-01-16T00:14:03.990 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:03.990 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:03.990 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:03.990 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.990 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.991 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3220 3.3 3:c478bfe8:::smithi12313934-10:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:03.991 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.580982+0000", 2020-01-16T00:14:03.991 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 34.229929601999999, 2020-01-16T00:14:03.991 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017920330000000002, 2020-01-16T00:14:03.991 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:03.991 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:03.991 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:03.991 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:03.992 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:03.992 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3220 2020-01-16T00:14:03.992 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.992 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:03.992 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.992 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:03.992 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580982+0000", 2020-01-16T00:14:03.992 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:03.993 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.993 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.993 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:03.993 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580982+0000", 2020-01-16T00:14:03.993 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:03.993 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.993 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.993 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:03.994 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580981+0000", 2020-01-16T00:14:03.994 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0380000000000001e-06 2020-01-16T00:14:03.994 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.994 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.994 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:03.994 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580985+0000", 2020-01-16T00:14:03.994 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8929000000000001e-05 2020-01-16T00:14:03.994 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.995 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.995 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:03.995 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581004+0000", 2020-01-16T00:14:03.995 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7339999999999999e-06 2020-01-16T00:14:03.995 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.995 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.995 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:03.996 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581009+0000", 2020-01-16T00:14:03.996 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01679812 2020-01-16T00:14:03.996 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.996 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.996 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:03.996 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.597807+0000", 2020-01-16T00:14:03.996 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013333400000000001 2020-01-16T00:14:03.996 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.997 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.997 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:03.997 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.597940+0000", 2020-01-16T00:14:03.997 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000961899 2020-01-16T00:14:03.997 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.997 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.997 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:03.997 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.598902+0000", 2020-01-16T00:14:03.998 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017920330000000002 2020-01-16T00:14:03.998 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:03.998 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:03.998 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:03.998 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:03.998 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:03.998 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3221 3.3 3:c478bfe8:::smithi12313934-10:head [read 0~2403467,checksum] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:03.998 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.581014+0000", 2020-01-16T00:14:03.999 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 34.229897252999997, 2020-01-16T00:14:03.999 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018784551999999999, 2020-01-16T00:14:03.999 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:04.000 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:04.001 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:04.001 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:04.001 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:04.001 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3221 2020-01-16T00:14:04.001 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.001 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:04.001 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.001 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:04.002 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581014+0000", 2020-01-16T00:14:04.002 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:04.002 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.002 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.002 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:04.002 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581014+0000", 2020-01-16T00:14:04.002 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:04.003 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.003 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.003 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:04.003 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581013+0000", 2020-01-16T00:14:04.003 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9390000000000002e-06 2020-01-16T00:14:04.003 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.003 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.003 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:04.004 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581019+0000", 2020-01-16T00:14:04.004 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1131000000000001e-05 2020-01-16T00:14:04.004 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.004 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.004 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:04.004 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581030+0000", 2020-01-16T00:14:04.004 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.55e-06 2020-01-16T00:14:04.004 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.005 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.005 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:04.005 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581037+0000", 2020-01-16T00:14:04.005 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017890321000000001 2020-01-16T00:14:04.005 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.005 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.005 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:04.005 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.598927+0000", 2020-01-16T00:14:04.006 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000105999 2020-01-16T00:14:04.006 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.006 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.006 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:04.006 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.599033+0000", 2020-01-16T00:14:04.006 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00076542400000000001 2020-01-16T00:14:04.006 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.006 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.007 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:04.007 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.599798+0000", 2020-01-16T00:14:04.007 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018784551999999999 2020-01-16T00:14:04.007 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.007 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:04.007 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.007 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.008 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.008 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4152 3.9 3:93dfe6f1:::smithi12313934-40:head [setxattr _header (56) in=7b,truncate 2806868] snapc 0=[] ondisk+write+known_if_redirected e53)", 2020-01-16T00:14:04.008 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:53.810986+0000", 2020-01-16T00:14:04.008 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 9.999924944, 2020-01-16T00:14:04.008 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016830279, 2020-01-16T00:14:04.008 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:04.008 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:04.008 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:04.009 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:04.009 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:04.009 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4152 2020-01-16T00:14:04.009 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.009 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:04.009 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.009 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:04.009 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.810986+0000", 2020-01-16T00:14:04.010 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:04.010 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.010 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.010 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:04.010 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.810986+0000", 2020-01-16T00:14:04.010 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:04.010 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.010 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.011 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:04.011 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.810985+0000", 2020-01-16T00:14:04.011 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.7529999999999996e-06 2020-01-16T00:14:04.011 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.011 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.011 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:04.011 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.810995+0000", 2020-01-16T00:14:04.011 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7907999999999999e-05 2020-01-16T00:14:04.012 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.012 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.012 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:04.012 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811013+0000", 2020-01-16T00:14:04.012 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5829999999999997e-06 2020-01-16T00:14:04.012 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.012 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.012 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:04.013 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811018+0000", 2020-01-16T00:14:04.013 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0029165229999999999 2020-01-16T00:14:04.013 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.013 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.013 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:04.013 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.813935+0000", 2020-01-16T00:14:04.013 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2583000000000001e-05 2020-01-16T00:14:04.014 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.014 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.014 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:04.014 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.813988+0000", 2020-01-16T00:14:04.014 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010581012000000001 2020-01-16T00:14:04.014 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.014 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.014 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:04.015 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.824569+0000", 2020-01-16T00:14:04.015 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013191900000000001 2020-01-16T00:14:04.015 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.015 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.015 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:04.015 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.824700+0000", 2020-01-16T00:14:04.015 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022298000000000001 2020-01-16T00:14:04.016 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.016 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.016 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:14:04.016 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.824923+0000", 2020-01-16T00:14:04.016 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010694260000000001 2020-01-16T00:14:04.016 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.016 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.016 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:04.017 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.825993+0000", 2020-01-16T00:14:04.017 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000116939 2020-01-16T00:14:04.017 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.017 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.017 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:04.017 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.826110+0000", 2020-01-16T00:14:04.017 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0015606400000000001 2020-01-16T00:14:04.017 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.018 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.018 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:04.018 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.827670+0000", 2020-01-16T00:14:04.018 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7318999999999997e-05 2020-01-16T00:14:04.018 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.018 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.018 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:04.018 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.827738+0000", 2020-01-16T00:14:04.019 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.873e-05 2020-01-16T00:14:04.019 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.019 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.019 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:04.019 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.827817+0000", 2020-01-16T00:14:04.019 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016830279 2020-01-16T00:14:04.019 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.019 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:04.020 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.020 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.020 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.020 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4153 3.9 3:93dfe6f1:::smithi12313934-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e53)", 2020-01-16T00:14:04.020 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:53.811034+0000", 2020-01-16T00:14:04.020 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 9.9998768560000002, 2020-01-16T00:14:04.020 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017095406, 2020-01-16T00:14:04.020 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:04.021 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:04.021 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:04.021 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:04.021 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:04.021 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4153 2020-01-16T00:14:04.021 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.021 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:04.022 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.022 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:04.022 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811034+0000", 2020-01-16T00:14:04.022 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:04.022 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.022 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.022 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:04.022 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811034+0000", 2020-01-16T00:14:04.023 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:04.023 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.023 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.023 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:04.023 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811034+0000", 2020-01-16T00:14:04.023 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9060000000000001e-06 2020-01-16T00:14:04.023 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.023 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.024 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:04.024 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811038+0000", 2020-01-16T00:14:04.024 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7710000000000001e-06 2020-01-16T00:14:04.024 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.024 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.024 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:04.024 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811045+0000", 2020-01-16T00:14:04.024 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5189999999999998e-06 2020-01-16T00:14:04.025 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.025 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.025 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:04.025 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811052+0000", 2020-01-16T00:14:04.025 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002953991 2020-01-16T00:14:04.025 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.025 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.025 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:04.026 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.814006+0000", 2020-01-16T00:14:04.026 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5726999999999999e-05 2020-01-16T00:14:04.026 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.026 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.026 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:04.026 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.814041+0000", 2020-01-16T00:14:04.026 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011650031999999999 2020-01-16T00:14:04.026 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.027 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.027 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:04.027 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.825691+0000", 2020-01-16T00:14:04.027 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0625999999999998e-05 2020-01-16T00:14:04.027 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.027 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.027 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:04.027 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.825762+0000", 2020-01-16T00:14:04.028 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0020982370000000002 2020-01-16T00:14:04.028 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.028 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.028 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:04.028 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.827860+0000", 2020-01-16T00:14:04.028 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012736299999999999 2020-01-16T00:14:04.028 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.028 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.029 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:04.029 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.827988+0000", 2020-01-16T00:14:04.029 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000142015 2020-01-16T00:14:04.029 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.029 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.029 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:04.029 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.828130+0000", 2020-01-16T00:14:04.030 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017095406 2020-01-16T00:14:04.030 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.030 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:04.030 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.030 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.030 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:04.030 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:04.032 INFO:teuthology.orchestra.run.smithi123.stderr:nodeep-scrub is unset 2020-01-16T00:14:04.032 INFO:tasks.rados.rados.0.smithi123.stdout:1648: expect (ObjNum 578 snap 0 seq_num 578) 2020-01-16T00:14:04.033 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:04.028+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:04.033 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-01-16T00:14:04.034 INFO:tasks.rados.rados.0.smithi123.stdout:1649: finishing write tid 1 to smithi12313934-50 2020-01-16T00:14:04.034 INFO:tasks.rados.rados.0.smithi123.stdout:1649: finishing write tid 2 to smithi12313934-50 2020-01-16T00:14:04.034 INFO:tasks.rados.rados.0.smithi123.stdout:1649: finishing write tid 3 to smithi12313934-50 2020-01-16T00:14:04.034 INFO:tasks.rados.rados.0.smithi123.stdout:1640: done (15 left) 2020-01-16T00:14:04.035 INFO:tasks.rados.rados.0.smithi123.stdout:1641: done (14 left) 2020-01-16T00:14:04.035 INFO:tasks.rados.rados.0.smithi123.stdout:1642: done (13 left) 2020-01-16T00:14:04.035 INFO:tasks.rados.rados.0.smithi123.stdout:1643: done (12 left) 2020-01-16T00:14:04.035 INFO:tasks.rados.rados.0.smithi123.stdout:1644: done (11 left) 2020-01-16T00:14:04.035 INFO:tasks.rados.rados.0.smithi123.stdout:1645: done (10 left) 2020-01-16T00:14:04.035 INFO:tasks.rados.rados.0.smithi123.stdout:1647: done (9 left) 2020-01-16T00:14:04.035 INFO:tasks.rados.rados.0.smithi123.stdout:1648: done (8 left) 2020-01-16T00:14:04.035 INFO:tasks.rados.rados.0.smithi123.stdout:1656: done (7 left) 2020-01-16T00:14:04.036 INFO:tasks.rados.rados.0.smithi123.stdout:1657: write oid 6 current snap is 0 2020-01-16T00:14:04.036 INFO:tasks.rados.rados.0.smithi123.stdout:1657: seq_num 591 ranges {732937=631340,1850858=433541,2796871=671289,3888195=44742} 2020-01-16T00:14:04.046 INFO:tasks.rados.rados.0.smithi123.stdout:1657: writing smithi12313934-6 from 732937 to 1364277 tid 1 2020-01-16T00:14:04.053 INFO:tasks.rados.rados.0.smithi123.stdout:1657: writing smithi12313934-6 from 1850858 to 2284399 tid 2 2020-01-16T00:14:04.060 INFO:tasks.rados.rados.0.smithi123.stdout:1657: writing smithi12313934-6 from 2796871 to 3468160 tid 3 2020-01-16T00:14:04.063 INFO:tasks.rados.rados.0.smithi123.stdout:1657: writing smithi12313934-6 from 3888195 to 3932937 tid 4 2020-01-16T00:14:04.063 INFO:tasks.rados.rados.0.smithi123.stdout:1658: read oid 38 snap -1 2020-01-16T00:14:04.063 INFO:tasks.rados.rados.0.smithi123.stdout:1658: expect (ObjNum 561 snap 0 seq_num 561) 2020-01-16T00:14:04.063 INFO:tasks.rados.rados.0.smithi123.stdout:1659: read oid 12 snap -1 2020-01-16T00:14:04.064 INFO:tasks.rados.rados.0.smithi123.stdout:1659: expect (ObjNum 517 snap 0 seq_num 517) 2020-01-16T00:14:04.064 INFO:tasks.rados.rados.0.smithi123.stdout:1660: read oid 9 snap -1 2020-01-16T00:14:04.065 INFO:tasks.rados.rados.0.smithi123.stdout:1660: expect (ObjNum 572 snap 0 seq_num 572) 2020-01-16T00:14:04.065 INFO:tasks.rados.rados.0.smithi123.stdout:1661: read oid 22 snap -1 2020-01-16T00:14:04.065 INFO:tasks.rados.rados.0.smithi123.stdout:1661: expect (ObjNum 575 snap 0 seq_num 575) 2020-01-16T00:14:04.065 INFO:tasks.rados.rados.0.smithi123.stdout:1662: read oid 35 snap -1 2020-01-16T00:14:04.065 INFO:tasks.rados.rados.0.smithi123.stdout:1662: expect (ObjNum 569 snap 0 seq_num 569) 2020-01-16T00:14:04.066 INFO:tasks.rados.rados.0.smithi123.stdout:1663: write (excl) oid 16 current snap is 0 2020-01-16T00:14:04.066 INFO:tasks.rados.rados.0.smithi123.stdout:1663: seq_num 592 ranges {749744=706589,1909340=577199,3224294=482561} 2020-01-16T00:14:04.072 INFO:tasks.rados.rados.0.smithi123.stdout:1663: writing smithi12313934-16 from 749744 to 1456333 tid 1 2020-01-16T00:14:04.078 INFO:tasks.rados.rados.0.smithi123.stdout:1663: writing smithi12313934-16 from 1909340 to 2486539 tid 2 2020-01-16T00:14:04.086 INFO:tasks.rados.rados.0.smithi123.stdout:1663: writing smithi12313934-16 from 3224294 to 3706855 tid 3 2020-01-16T00:14:04.086 INFO:tasks.rados.rados.0.smithi123.stdout:1664: read oid 32 snap -1 2020-01-16T00:14:04.086 INFO:tasks.rados.rados.0.smithi123.stdout:1664: expect (ObjNum 513 snap 0 seq_num 513) 2020-01-16T00:14:04.086 INFO:tasks.rados.rados.0.smithi123.stdout:1665: read oid 25 snap -1 2020-01-16T00:14:04.086 INFO:tasks.rados.rados.0.smithi123.stdout:1665: expect (ObjNum 571 snap 0 seq_num 571) 2020-01-16T00:14:04.087 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:04.087 INFO:tasks.rados.rados.0.smithi123.stdout:1649: finishing write tid 5 to smithi12313934-50 2020-01-16T00:14:04.087 INFO:tasks.rados.rados.0.smithi123.stdout:1649: finishing write tid 6 to smithi12313934-50 2020-01-16T00:14:04.087 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 50 v 203 (ObjNum 584 snap 0 seq_num 584) dirty exists 2020-01-16T00:14:04.087 INFO:tasks.rados.rados.0.smithi123.stdout:1649: left oid 50 (ObjNum 584 snap 0 seq_num 584) 2020-01-16T00:14:04.087 INFO:tasks.rados.rados.0.smithi123.stdout:1650: finishing write tid 1 to smithi12313934-1 2020-01-16T00:14:04.087 INFO:tasks.rados.rados.0.smithi123.stdout:1650: finishing write tid 2 to smithi12313934-1 2020-01-16T00:14:04.087 INFO:tasks.rados.rados.0.smithi123.stdout:1649: done (15 left) 2020-01-16T00:14:04.088 INFO:tasks.rados.rados.0.smithi123.stdout:1666: write (excl) oid 15 current snap is 0 2020-01-16T00:14:04.088 INFO:tasks.rados.rados.0.smithi123.stdout:1666: seq_num 593 ranges {766551=498191,1684175=720857,2851717=693833} 2020-01-16T00:14:04.093 INFO:tasks.rados.rados.0.smithi123.stdout:1666: writing smithi12313934-15 from 766551 to 1264742 tid 1 2020-01-16T00:14:04.100 INFO:tasks.rados.rados.0.smithi123.stdout:1666: writing smithi12313934-15 from 1684175 to 2405032 tid 2 2020-01-16T00:14:04.107 INFO:tasks.rados.rados.0.smithi123.stdout:1666: writing smithi12313934-15 from 2851717 to 3545550 tid 3 2020-01-16T00:14:04.107 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:04.107 INFO:tasks.rados.rados.0.smithi123.stdout:1650: finishing write tid 3 to smithi12313934-1 2020-01-16T00:14:04.107 INFO:tasks.rados.rados.0.smithi123.stdout:1650: finishing write tid 5 to smithi12313934-1 2020-01-16T00:14:04.107 INFO:tasks.rados.rados.0.smithi123.stdout:1650: finishing write tid 6 to smithi12313934-1 2020-01-16T00:14:04.107 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 1 v 311 (ObjNum 585 snap 0 seq_num 585) dirty exists 2020-01-16T00:14:04.107 INFO:tasks.rados.rados.0.smithi123.stdout:1650: left oid 1 (ObjNum 585 snap 0 seq_num 585) 2020-01-16T00:14:04.108 INFO:tasks.rados.rados.0.smithi123.stdout:1651: finishing write tid 1 to smithi12313934-46 2020-01-16T00:14:04.108 INFO:tasks.rados.rados.0.smithi123.stdout:1651: finishing write tid 2 to smithi12313934-46 2020-01-16T00:14:04.108 INFO:tasks.rados.rados.0.smithi123.stdout:1650: done (15 left) 2020-01-16T00:14:04.108 INFO:tasks.rados.rados.0.smithi123.stdout:1667: read oid 28 snap -1 2020-01-16T00:14:04.108 INFO:tasks.rados.rados.0.smithi123.stdout:1667: expect (ObjNum 506 snap 0 seq_num 506) 2020-01-16T00:14:04.108 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:04.108 INFO:tasks.rados.rados.0.smithi123.stdout:1651: finishing write tid 3 to smithi12313934-46 2020-01-16T00:14:04.108 INFO:tasks.rados.rados.0.smithi123.stdout:1651: finishing write tid 5 to smithi12313934-46 2020-01-16T00:14:04.109 INFO:tasks.rados.rados.0.smithi123.stdout:1652: expect (ObjNum 557 snap 0 seq_num 557) 2020-01-16T00:14:04.132 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:04.130+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:04.175 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:04.175 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:14:04.175 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:14:04.175 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:04.188 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-01-16T00:14:04.234 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:04.230+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:04.284 INFO:tasks.rados.rados.0.smithi123.stdout:1651: finishing write tid 6 to smithi12313934-46 2020-01-16T00:14:04.285 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 46 v 315 (ObjNum 586 snap 0 seq_num 586) dirty exists 2020-01-16T00:14:04.285 INFO:tasks.rados.rados.0.smithi123.stdout:1651: left oid 46 (ObjNum 586 snap 0 seq_num 586) 2020-01-16T00:14:04.285 INFO:tasks.rados.rados.0.smithi123.stdout:1653: finishing write tid 1 to smithi12313934-48 2020-01-16T00:14:04.285 INFO:tasks.rados.rados.0.smithi123.stdout:1653: finishing write tid 2 to smithi12313934-48 2020-01-16T00:14:04.286 INFO:tasks.rados.rados.0.smithi123.stdout:1653: finishing write tid 3 to smithi12313934-48 2020-01-16T00:14:04.286 INFO:tasks.rados.rados.0.smithi123.stdout:1653: finishing write tid 5 to smithi12313934-48 2020-01-16T00:14:04.286 INFO:tasks.rados.rados.0.smithi123.stdout:1653: finishing write tid 6 to smithi12313934-48 2020-01-16T00:14:04.286 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 48 v 196 (ObjNum 587 snap 0 seq_num 587) dirty exists 2020-01-16T00:14:04.286 INFO:tasks.rados.rados.0.smithi123.stdout:1653: left oid 48 (ObjNum 587 snap 0 seq_num 587) 2020-01-16T00:14:04.287 INFO:tasks.rados.rados.0.smithi123.stdout:1654: finishing write tid 1 to smithi12313934-2 2020-01-16T00:14:04.287 INFO:tasks.rados.rados.0.smithi123.stdout:1654: finishing write tid 2 to smithi12313934-2 2020-01-16T00:14:04.287 INFO:tasks.rados.rados.0.smithi123.stdout:1651: done (15 left) 2020-01-16T00:14:04.287 INFO:tasks.rados.rados.0.smithi123.stdout:1652: done (14 left) 2020-01-16T00:14:04.288 INFO:tasks.rados.rados.0.smithi123.stdout:1653: done (13 left) 2020-01-16T00:14:04.288 INFO:tasks.rados.rados.0.smithi123.stdout:1668: delete oid 7 current snap is 0 2020-01-16T00:14:04.288 INFO:tasks.rados.rados.0.smithi123.stdout:1654: finishing write tid 3 to smithi12313934-2 2020-01-16T00:14:04.288 INFO:tasks.rados.rados.0.smithi123.stdout:1654: finishing write tid 5 to smithi12313934-2 2020-01-16T00:14:04.289 INFO:tasks.rados.rados.0.smithi123.stdout:1654: finishing write tid 6 to smithi12313934-2 2020-01-16T00:14:04.289 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 2 v 230 (ObjNum 588 snap 0 seq_num 588) dirty exists 2020-01-16T00:14:04.289 INFO:tasks.rados.rados.0.smithi123.stdout:1654: left oid 2 (ObjNum 588 snap 0 seq_num 588) 2020-01-16T00:14:04.289 INFO:tasks.rados.rados.0.smithi123.stdout:1655: finishing write tid 1 to smithi12313934-29 2020-01-16T00:14:04.289 INFO:tasks.rados.rados.0.smithi123.stdout:1655: finishing write tid 2 to smithi12313934-29 2020-01-16T00:14:04.290 INFO:tasks.rados.rados.0.smithi123.stdout:1655: finishing write tid 3 to smithi12313934-29 2020-01-16T00:14:04.290 INFO:tasks.rados.rados.0.smithi123.stdout:1655: finishing write tid 5 to smithi12313934-29 2020-01-16T00:14:04.290 INFO:tasks.rados.rados.0.smithi123.stdout:1655: finishing write tid 6 to smithi12313934-29 2020-01-16T00:14:04.290 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 29 v 179 (ObjNum 589 snap 0 seq_num 589) dirty exists 2020-01-16T00:14:04.291 INFO:tasks.rados.rados.0.smithi123.stdout:1655: left oid 29 (ObjNum 589 snap 0 seq_num 589) 2020-01-16T00:14:04.291 INFO:tasks.rados.rados.0.smithi123.stdout:1657: finishing write tid 1 to smithi12313934-6 2020-01-16T00:14:04.291 INFO:tasks.rados.rados.0.smithi123.stdout:1658: expect (ObjNum 561 snap 0 seq_num 561) 2020-01-16T00:14:04.296 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:04.296 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:14:04.297 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:14:04.297 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:14:04.298 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:04.306 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-01-16T00:14:04.322 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-01-16T00:14:04.367 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:04.366+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:04.383 INFO:tasks.rados.rados.0.smithi123.stdout:1657: finishing write tid 2 to smithi12313934-6 2020-01-16T00:14:04.384 INFO:tasks.rados.rados.0.smithi123.stdout:1657: finishing write tid 3 to smithi12313934-6 2020-01-16T00:14:04.384 INFO:tasks.rados.rados.0.smithi123.stdout:1657: finishing write tid 4 to smithi12313934-6 2020-01-16T00:14:04.384 INFO:tasks.rados.rados.0.smithi123.stdout:1654: done (13 left) 2020-01-16T00:14:04.384 INFO:tasks.rados.rados.0.smithi123.stdout:1655: done (12 left) 2020-01-16T00:14:04.384 INFO:tasks.rados.rados.0.smithi123.stdout:1658: done (11 left) 2020-01-16T00:14:04.384 INFO:tasks.rados.rados.0.smithi123.stdout:1668: done (10 left) 2020-01-16T00:14:04.385 INFO:tasks.rados.rados.0.smithi123.stdout:1669: read oid 18 snap -1 2020-01-16T00:14:04.385 INFO:tasks.rados.rados.0.smithi123.stdout:1669: expect (ObjNum 566 snap 0 seq_num 566) 2020-01-16T00:14:04.385 INFO:tasks.rados.rados.0.smithi123.stdout:1670: write oid 50 current snap is 0 2020-01-16T00:14:04.385 INFO:tasks.rados.rados.0.smithi123.stdout:1670: seq_num 595 ranges {400165=648689,1684786=315379} 2020-01-16T00:14:04.397 INFO:tasks.rados.rados.0.smithi123.stdout:1670: writing smithi12313934-50 from 400165 to 1048854 tid 1 2020-01-16T00:14:04.403 INFO:tasks.rados.rados.0.smithi123.stdout:1670: writing smithi12313934-50 from 1684786 to 2000165 tid 2 2020-01-16T00:14:04.403 INFO:tasks.rados.rados.0.smithi123.stdout:1671: read oid 27 snap -1 2020-01-16T00:14:04.403 INFO:tasks.rados.rados.0.smithi123.stdout:1671: expect (ObjNum 562 snap 0 seq_num 562) 2020-01-16T00:14:04.403 INFO:tasks.rados.rados.0.smithi123.stdout:1657: finishing write tid 6 to smithi12313934-6 2020-01-16T00:14:04.404 INFO:tasks.rados.rados.0.smithi123.stdout:1672: read oid 26 snap -1 2020-01-16T00:14:04.404 INFO:tasks.rados.rados.0.smithi123.stdout:1672: expect (ObjNum 556 snap 0 seq_num 556) 2020-01-16T00:14:04.404 INFO:tasks.rados.rados.0.smithi123.stdout:1673: read oid 5 snap -1 2020-01-16T00:14:04.404 INFO:tasks.rados.rados.0.smithi123.stdout:1673: expect (ObjNum 582 snap 0 seq_num 582) 2020-01-16T00:14:04.404 INFO:tasks.rados.rados.0.smithi123.stdout:1674: read oid 20 snap -1 2020-01-16T00:14:04.404 INFO:tasks.rados.rados.0.smithi123.stdout:1674: expect deleted 2020-01-16T00:14:04.404 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:04.405 INFO:tasks.rados.rados.0.smithi123.stdout:1657: finishing write tid 7 to smithi12313934-6 2020-01-16T00:14:04.405 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 6 v 176 (ObjNum 591 snap 0 seq_num 591) dirty exists 2020-01-16T00:14:04.405 INFO:tasks.rados.rados.0.smithi123.stdout:1657: left oid 6 (ObjNum 591 snap 0 seq_num 591) 2020-01-16T00:14:04.405 INFO:tasks.rados.rados.0.smithi123.stdout:1662: expect (ObjNum 569 snap 0 seq_num 569) 2020-01-16T00:14:04.413 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:04.413 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:14:04.413 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:14:04.413 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:14:04.414 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.414 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.5.13:36 2.0 2:20e40cec:::smithi12313934-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 113152~1699840] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:14:04.414 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:14:04.414 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 86.577886324000005, 2020-01-16T00:14:04.414 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011736636, 2020-01-16T00:14:04.414 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:04.415 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:04.415 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:04.415 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.415 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:04.415 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:14:04.415 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:04.415 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.415 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.416 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:04.416 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:14:04.416 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999952 2020-01-16T00:14:04.416 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.416 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.416 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:04.416 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831934+0000", 2020-01-16T00:14:04.417 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001034325 2020-01-16T00:14:04.417 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.417 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.417 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:04.417 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.832968+0000", 2020-01-16T00:14:04.417 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.506e-05 2020-01-16T00:14:04.417 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.418 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.418 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:04.418 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833004+0000", 2020-01-16T00:14:04.418 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3406e-05 2020-01-16T00:14:04.418 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.418 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.418 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:04.419 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833017+0000", 2020-01-16T00:14:04.419 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015477000000000001 2020-01-16T00:14:04.419 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.419 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.419 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:04.419 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833172+0000", 2020-01-16T00:14:04.419 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033750700000000002 2020-01-16T00:14:04.419 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.420 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.420 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:04.420 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833509+0000", 2020-01-16T00:14:04.420 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00055861400000000003 2020-01-16T00:14:04.420 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.420 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.420 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:14:04.420 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.834068+0000", 2020-01-16T00:14:04.421 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.007474545 2020-01-16T00:14:04.421 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.421 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.421 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:04.421 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.841542+0000", 2020-01-16T00:14:04.422 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00056319499999999995 2020-01-16T00:14:04.422 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.422 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.422 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:04.422 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.842106+0000", 2020-01-16T00:14:04.422 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001371017 2020-01-16T00:14:04.422 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.423 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.423 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:04.423 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843477+0000", 2020-01-16T00:14:04.423 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000123938 2020-01-16T00:14:04.423 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.423 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.423 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:04.424 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843601+0000", 2020-01-16T00:14:04.424 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5012000000000006e-05 2020-01-16T00:14:04.424 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.424 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.424 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:04.424 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843676+0000", 2020-01-16T00:14:04.425 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011736636 2020-01-16T00:14:04.425 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.425 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:04.425 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.425 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.426 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.426 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1187 3.d 3:b1521a3e:::smithi12313934-25:head [write 1721042~423699] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:14:04.426 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.263828+0000", 2020-01-16T00:14:04.426 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 82.145997252000001, 2020-01-16T00:14:04.426 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011305709000000001, 2020-01-16T00:14:04.427 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:04.427 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:04.427 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:04.428 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:04.428 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:04.428 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1187 2020-01-16T00:14:04.428 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.428 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:04.428 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.429 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:04.429 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.263828+0000", 2020-01-16T00:14:04.429 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:04.429 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.429 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.430 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:04.431 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.263828+0000", 2020-01-16T00:14:04.431 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:14:04.431 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.431 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.432 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:04.432 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.263824+0000", 2020-01-16T00:14:04.432 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00048485899999999997 2020-01-16T00:14:04.432 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.432 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.433 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:04.433 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264309+0000", 2020-01-16T00:14:04.433 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5622999999999999e-05 2020-01-16T00:14:04.433 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.434 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.434 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:04.434 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264325+0000", 2020-01-16T00:14:04.434 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2530000000000004e-06 2020-01-16T00:14:04.434 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.434 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.435 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:04.435 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264330+0000", 2020-01-16T00:14:04.435 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000102252 2020-01-16T00:14:04.435 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.435 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.435 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:04.435 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264432+0000", 2020-01-16T00:14:04.436 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022361000000000001 2020-01-16T00:14:04.436 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.436 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.436 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:04.436 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264656+0000", 2020-01-16T00:14:04.436 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0028688849999999998 2020-01-16T00:14:04.436 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.437 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.437 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:04.437 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.267525+0000", 2020-01-16T00:14:04.437 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00023640900000000001 2020-01-16T00:14:04.437 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.437 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.437 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:04.437 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.267761+0000", 2020-01-16T00:14:04.438 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00035667000000000001 2020-01-16T00:14:04.438 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.438 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.439 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:14:04.439 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.268118+0000", 2020-01-16T00:14:04.439 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023898040000000001 2020-01-16T00:14:04.439 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.439 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.439 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:04.440 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.270507+0000", 2020-01-16T00:14:04.440 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014919690000000001 2020-01-16T00:14:04.440 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.440 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.440 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:04.440 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.271999+0000", 2020-01-16T00:14:04.441 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0029764510000000002 2020-01-16T00:14:04.441 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.441 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.441 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:04.441 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.274976+0000", 2020-01-16T00:14:04.441 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5409999999999997e-05 2020-01-16T00:14:04.441 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.441 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.442 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:04.442 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.275061+0000", 2020-01-16T00:14:04.443 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2452000000000004e-05 2020-01-16T00:14:04.443 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.443 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.443 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:04.443 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.275134+0000", 2020-01-16T00:14:04.444 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011305709000000001 2020-01-16T00:14:04.444 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.444 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:04.444 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.444 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.444 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.444 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1276 3.d 3:b1521a3e:::smithi12313934-25:head [read 0~3882384,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:14:04.445 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:14:04.445 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 81.335126754000001, 2020-01-16T00:14:04.445 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015839157999999999, 2020-01-16T00:14:04.445 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:04.445 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:04.445 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:04.446 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:04.446 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:04.446 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1276 2020-01-16T00:14:04.446 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.446 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:04.446 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.446 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:04.448 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:14:04.448 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:04.448 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.448 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.448 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:04.448 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:14:04.449 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:04.449 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.449 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.450 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:04.450 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:14:04.450 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5429999999999999e-06 2020-01-16T00:14:04.450 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.450 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.450 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:04.450 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074704+0000", 2020-01-16T00:14:04.451 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0839999999999992e-06 2020-01-16T00:14:04.451 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.451 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.451 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:04.451 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074712+0000", 2020-01-16T00:14:04.451 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.173e-06 2020-01-16T00:14:04.451 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.452 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.452 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:04.452 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074718+0000", 2020-01-16T00:14:04.452 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0083034700000000003 2020-01-16T00:14:04.452 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.452 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.453 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:04.453 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.083022+0000", 2020-01-16T00:14:04.453 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028456400000000001 2020-01-16T00:14:04.453 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.453 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.453 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:04.453 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.083306+0000", 2020-01-16T00:14:04.454 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0072312640000000003 2020-01-16T00:14:04.454 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.454 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.454 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:04.454 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090538+0000", 2020-01-16T00:14:04.454 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015839157999999999 2020-01-16T00:14:04.454 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.454 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:04.455 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.455 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.455 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.455 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1277 3.d 3:b1521a3e:::smithi12313934-25:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:14:04.455 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:14:04.455 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 81.335083163999997, 2020-01-16T00:14:04.455 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016646459999999998, 2020-01-16T00:14:04.455 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:04.456 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:04.456 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:04.456 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:04.456 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:04.456 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1277 2020-01-16T00:14:04.457 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.457 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:04.457 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.457 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:04.457 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:14:04.457 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:04.457 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.457 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.458 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:04.458 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:14:04.458 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:04.458 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.458 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.458 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:04.458 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074741+0000", 2020-01-16T00:14:04.458 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0776e-05 2020-01-16T00:14:04.459 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.459 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.459 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:04.459 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074752+0000", 2020-01-16T00:14:04.459 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7600000000000002e-06 2020-01-16T00:14:04.459 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.459 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.460 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:04.460 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074759+0000", 2020-01-16T00:14:04.460 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6840000000000001e-06 2020-01-16T00:14:04.460 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.460 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.461 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:04.461 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074765+0000", 2020-01-16T00:14:04.461 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015815216 2020-01-16T00:14:04.461 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.461 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.461 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:04.461 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090580+0000", 2020-01-16T00:14:04.461 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000182404 2020-01-16T00:14:04.462 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.462 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.462 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:04.462 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090763+0000", 2020-01-16T00:14:04.462 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00062579300000000003 2020-01-16T00:14:04.462 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.462 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.463 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:04.463 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091389+0000", 2020-01-16T00:14:04.463 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016646459999999998 2020-01-16T00:14:04.463 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.463 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:04.463 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.463 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.463 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.464 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1278 3.d 3:b1521a3e:::smithi12313934-25:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:14:04.464 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:14:04.464 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 81.335053450999993, 2020-01-16T00:14:04.464 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017994915, 2020-01-16T00:14:04.464 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:04.464 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:04.464 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:04.464 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:04.465 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:04.465 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1278 2020-01-16T00:14:04.465 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.465 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:04.465 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.465 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:04.465 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:14:04.466 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:04.466 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.466 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.466 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:04.466 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:14:04.467 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:04.467 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.467 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.467 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:04.467 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074771+0000", 2020-01-16T00:14:04.467 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4569999999999998e-06 2020-01-16T00:14:04.468 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.468 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.468 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:04.468 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074775+0000", 2020-01-16T00:14:04.468 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0330000000000003e-06 2020-01-16T00:14:04.468 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.468 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.468 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:04.469 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074782+0000", 2020-01-16T00:14:04.469 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9470000000000002e-06 2020-01-16T00:14:04.469 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.469 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.469 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:04.469 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074787+0000", 2020-01-16T00:14:04.469 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01666134 2020-01-16T00:14:04.470 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.470 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.470 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:04.470 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091449+0000", 2020-01-16T00:14:04.470 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000247731 2020-01-16T00:14:04.470 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.470 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.470 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:04.471 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091696+0000", 2020-01-16T00:14:04.471 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010703170000000001 2020-01-16T00:14:04.471 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.471 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.471 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:04.471 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.092767+0000", 2020-01-16T00:14:04.471 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017994915 2020-01-16T00:14:04.471 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.472 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:04.472 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.472 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.472 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.472 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2260 3.d 3:b76659c4:::smithi12313934-47:head [write 601750~535546] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:14:04.472 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:06.050577+0000", 2020-01-16T00:14:04.472 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 58.359248135000001, 2020-01-16T00:14:04.473 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012415381, 2020-01-16T00:14:04.473 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:04.473 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:04.473 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:04.473 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:04.473 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:04.474 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2260 2020-01-16T00:14:04.474 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.474 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:04.474 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.474 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:04.474 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.050577+0000", 2020-01-16T00:14:04.474 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:04.475 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.475 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.475 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:04.475 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.050577+0000", 2020-01-16T00:14:04.475 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:04.476 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.476 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.476 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:04.476 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.050574+0000", 2020-01-16T00:14:04.476 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001523487 2020-01-16T00:14:04.477 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.477 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.477 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:04.477 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052097+0000", 2020-01-16T00:14:04.477 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3697e-05 2020-01-16T00:14:04.477 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.478 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.478 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:04.478 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052121+0000", 2020-01-16T00:14:04.478 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0070000000000003e-06 2020-01-16T00:14:04.478 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.479 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.479 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:04.479 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052130+0000", 2020-01-16T00:14:04.479 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4559000000000003e-05 2020-01-16T00:14:04.480 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.480 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.480 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:04.480 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052194+0000", 2020-01-16T00:14:04.481 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000165878 2020-01-16T00:14:04.481 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.481 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.481 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:04.481 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052360+0000", 2020-01-16T00:14:04.482 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018825499999999999 2020-01-16T00:14:04.482 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.482 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.482 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:14:04.482 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052549+0000", 2020-01-16T00:14:04.482 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002761782 2020-01-16T00:14:04.482 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.483 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.483 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:04.483 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.055310+0000", 2020-01-16T00:14:04.483 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00060385500000000004 2020-01-16T00:14:04.483 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.483 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.483 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:04.484 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.055914+0000", 2020-01-16T00:14:04.484 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0069733809999999999 2020-01-16T00:14:04.484 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.484 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.484 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:04.484 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.062888+0000", 2020-01-16T00:14:04.485 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2884999999999999e-05 2020-01-16T00:14:04.485 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.485 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.485 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:04.485 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.062940+0000", 2020-01-16T00:14:04.486 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1996999999999997e-05 2020-01-16T00:14:04.486 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.486 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.486 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:04.486 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.062992+0000", 2020-01-16T00:14:04.487 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012415381 2020-01-16T00:14:04.487 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.487 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:04.487 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.487 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.487 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.487 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2295 3.d 3:b1521a3e:::smithi12313934-25:head [write 1763838~505140] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:14:04.488 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:06.660258+0000", 2020-01-16T00:14:04.488 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 57.749566907999998, 2020-01-16T00:14:04.488 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011681408000000001, 2020-01-16T00:14:04.488 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:04.488 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:04.488 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:04.489 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:04.489 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:04.489 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2295 2020-01-16T00:14:04.489 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.489 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:04.490 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.490 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:04.490 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660258+0000", 2020-01-16T00:14:04.490 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:04.490 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.490 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.490 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:04.491 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660258+0000", 2020-01-16T00:14:04.491 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:04.491 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.491 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.491 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:04.491 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660256+0000", 2020-01-16T00:14:04.491 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00066501699999999995 2020-01-16T00:14:04.492 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.492 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.492 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:04.492 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660921+0000", 2020-01-16T00:14:04.492 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7339000000000001e-05 2020-01-16T00:14:04.492 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.492 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.493 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:04.493 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660938+0000", 2020-01-16T00:14:04.493 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0219999999999993e-06 2020-01-16T00:14:04.493 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.493 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.493 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:04.493 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660947+0000", 2020-01-16T00:14:04.494 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00094237200000000002 2020-01-16T00:14:04.494 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.494 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.494 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:04.494 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.661890+0000", 2020-01-16T00:14:04.494 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000117649 2020-01-16T00:14:04.495 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.495 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.495 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:04.495 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662007+0000", 2020-01-16T00:14:04.495 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00060662599999999998 2020-01-16T00:14:04.495 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.495 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.496 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:04.496 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662614+0000", 2020-01-16T00:14:04.496 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028424899999999998 2020-01-16T00:14:04.496 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.496 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.496 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:04.496 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662898+0000", 2020-01-16T00:14:04.497 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00054904599999999997 2020-01-16T00:14:04.497 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.497 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.497 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:14:04.497 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.663447+0000", 2020-01-16T00:14:04.497 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0051479990000000003 2020-01-16T00:14:04.497 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.498 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.498 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:04.498 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.668595+0000", 2020-01-16T00:14:04.498 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00080881799999999999 2020-01-16T00:14:04.498 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.498 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.498 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:04.499 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.669404+0000", 2020-01-16T00:14:04.499 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0024495290000000002 2020-01-16T00:14:04.499 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.499 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.499 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:04.500 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671854+0000", 2020-01-16T00:14:04.500 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6635000000000003e-05 2020-01-16T00:14:04.500 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.500 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.500 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:04.500 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671900+0000", 2020-01-16T00:14:04.500 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9567999999999998e-05 2020-01-16T00:14:04.501 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.501 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.501 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:04.501 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671940+0000", 2020-01-16T00:14:04.501 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011681408000000001 2020-01-16T00:14:04.501 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.502 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:04.502 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.502 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.502 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.502 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.3.13:109 2.0 2:11fa79bc:::smithi12313934-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 331264~198656] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e42)", 2020-01-16T00:14:04.502 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:21.840075+0000", 2020-01-16T00:14:04.502 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 42.569750611000003, 2020-01-16T00:14:04.503 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011558677999999999, 2020-01-16T00:14:04.503 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:04.503 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:04.503 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:04.503 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.503 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:04.503 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840075+0000", 2020-01-16T00:14:04.504 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:04.504 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.504 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.504 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:04.504 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840075+0000", 2020-01-16T00:14:04.504 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:04.505 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.505 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.505 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:04.505 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840073+0000", 2020-01-16T00:14:04.505 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033576299999999999 2020-01-16T00:14:04.505 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.506 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.506 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:04.506 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840409+0000", 2020-01-16T00:14:04.506 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2082999999999998e-05 2020-01-16T00:14:04.506 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.506 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.506 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:04.507 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840431+0000", 2020-01-16T00:14:04.507 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4410000000000003e-06 2020-01-16T00:14:04.507 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.507 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.507 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:04.507 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840436+0000", 2020-01-16T00:14:04.507 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000101016 2020-01-16T00:14:04.508 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.508 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.508 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:04.508 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840537+0000", 2020-01-16T00:14:04.508 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000117639 2020-01-16T00:14:04.508 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.508 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.509 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:04.509 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840654+0000", 2020-01-16T00:14:04.509 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000196162 2020-01-16T00:14:04.509 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.509 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.509 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:14:04.510 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840850+0000", 2020-01-16T00:14:04.510 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003319092 2020-01-16T00:14:04.510 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.510 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.510 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:04.510 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.844169+0000", 2020-01-16T00:14:04.510 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011349229999999999 2020-01-16T00:14:04.511 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.511 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.511 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:04.511 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.845304+0000", 2020-01-16T00:14:04.511 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0062517670000000001 2020-01-16T00:14:04.511 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.512 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.512 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:04.512 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851556+0000", 2020-01-16T00:14:04.512 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4138000000000003e-05 2020-01-16T00:14:04.512 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.512 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.512 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:04.513 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851600+0000", 2020-01-16T00:14:04.513 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2993000000000001e-05 2020-01-16T00:14:04.513 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.513 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.513 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:04.513 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851633+0000", 2020-01-16T00:14:04.514 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011558677999999999 2020-01-16T00:14:04.514 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.514 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:04.514 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.514 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.514 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.514 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.3.13:110 2.0 2:11fa79bc:::smithi12313934-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 752640~650240] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e42)", 2020-01-16T00:14:04.515 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:21.840457+0000", 2020-01-16T00:14:04.515 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 42.569368318999999, 2020-01-16T00:14:04.515 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011329271, 2020-01-16T00:14:04.515 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:04.515 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:04.515 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:04.515 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.516 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:04.516 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840457+0000", 2020-01-16T00:14:04.516 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:04.516 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.516 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.516 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:04.516 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840457+0000", 2020-01-16T00:14:04.517 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:04.517 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.517 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.517 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:04.517 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840456+0000", 2020-01-16T00:14:04.517 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00062607899999999998 2020-01-16T00:14:04.518 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.518 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.518 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:04.518 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841082+0000", 2020-01-16T00:14:04.518 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1239999999999999e-06 2020-01-16T00:14:04.518 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.518 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.519 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:04.519 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841090+0000", 2020-01-16T00:14:04.519 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7129999999999999e-06 2020-01-16T00:14:04.519 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.519 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.519 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:04.520 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841093+0000", 2020-01-16T00:14:04.520 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00075997800000000002 2020-01-16T00:14:04.520 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.520 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.520 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:04.520 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841853+0000", 2020-01-16T00:14:04.520 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0690000000000003e-05 2020-01-16T00:14:04.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.521 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.521 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:04.521 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841924+0000", 2020-01-16T00:14:04.521 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000115753 2020-01-16T00:14:04.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.522 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.522 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:14:04.522 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.842040+0000", 2020-01-16T00:14:04.522 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0021358890000000002 2020-01-16T00:14:04.522 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.522 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.522 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:04.523 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.844176+0000", 2020-01-16T00:14:04.523 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0042541870000000004 2020-01-16T00:14:04.523 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.523 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.523 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:04.523 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.848430+0000", 2020-01-16T00:14:04.524 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0032476929999999998 2020-01-16T00:14:04.524 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.524 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.524 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:04.524 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851677+0000", 2020-01-16T00:14:04.524 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0370000000000001e-05 2020-01-16T00:14:04.524 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.525 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.525 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:04.525 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851728+0000", 2020-01-16T00:14:04.525 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.834e-05 2020-01-16T00:14:04.525 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.525 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.526 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:04.526 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851786+0000", 2020-01-16T00:14:04.526 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011329271 2020-01-16T00:14:04.526 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.526 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:04.526 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.527 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.527 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.527 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2987 3.d 3:b3dd5d45:::smithi12313934-16:head [write 467327~633480] snapc 0=[] ondisk+write+known_if_redirected e42)", 2020-01-16T00:14:04.527 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:22.257904+0000", 2020-01-16T00:14:04.527 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 42.151921025, 2020-01-16T00:14:04.527 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012187450000000001, 2020-01-16T00:14:04.527 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:04.528 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:04.528 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:04.528 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:04.528 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:04.528 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2987 2020-01-16T00:14:04.528 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.528 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:04.529 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.529 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:04.529 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.257904+0000", 2020-01-16T00:14:04.529 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:04.529 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.529 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.530 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:04.530 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.257904+0000", 2020-01-16T00:14:04.530 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999933 2020-01-16T00:14:04.530 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.530 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.530 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:04.530 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.257898+0000", 2020-01-16T00:14:04.531 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00097099800000000004 2020-01-16T00:14:04.531 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.531 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.531 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:04.531 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.258869+0000", 2020-01-16T00:14:04.531 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7970000000000003e-05 2020-01-16T00:14:04.532 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.532 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.532 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:04.532 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.258917+0000", 2020-01-16T00:14:04.532 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5982999999999999e-05 2020-01-16T00:14:04.532 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.532 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.533 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:04.533 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.258933+0000", 2020-01-16T00:14:04.533 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00026239999999999998 2020-01-16T00:14:04.533 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.533 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.533 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:04.534 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.259195+0000", 2020-01-16T00:14:04.534 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024907400000000001 2020-01-16T00:14:04.534 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.534 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.534 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:04.534 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.259444+0000", 2020-01-16T00:14:04.534 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00034824999999999998 2020-01-16T00:14:04.535 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.535 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.535 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:04.535 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.259792+0000", 2020-01-16T00:14:04.535 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003251538 2020-01-16T00:14:04.535 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.535 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.536 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:04.536 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.263044+0000", 2020-01-16T00:14:04.536 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0068502880000000004 2020-01-16T00:14:04.536 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.536 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.536 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:04.537 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.269894+0000", 2020-01-16T00:14:04.537 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0945000000000006e-05 2020-01-16T00:14:04.537 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.537 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.537 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:04.537 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.269965+0000", 2020-01-16T00:14:04.537 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012667399999999999 2020-01-16T00:14:04.538 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.538 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.538 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:04.538 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.270092+0000", 2020-01-16T00:14:04.538 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012187450000000001 2020-01-16T00:14:04.538 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.539 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:04.539 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.539 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.539 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.539 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3273 3.d 3:b76659c4:::smithi12313934-47:head [read 0~3647152,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:04.539 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.728775+0000", 2020-01-16T00:14:04.539 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 33.681050470999999, 2020-01-16T00:14:04.540 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012922950000000001, 2020-01-16T00:14:04.540 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:04.540 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:04.540 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:04.540 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:04.540 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:04.540 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3273 2020-01-16T00:14:04.541 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.541 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:04.541 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.541 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:04.541 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728775+0000", 2020-01-16T00:14:04.541 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:04.541 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.542 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.542 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:04.542 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728775+0000", 2020-01-16T00:14:04.542 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:04.542 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.542 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.542 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:04.543 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728771+0000", 2020-01-16T00:14:04.543 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.5009000000000001e-05 2020-01-16T00:14:04.543 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.543 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.543 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:04.543 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728796+0000", 2020-01-16T00:14:04.543 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2357999999999998e-05 2020-01-16T00:14:04.544 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.544 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.544 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:04.544 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728819+0000", 2020-01-16T00:14:04.544 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.4800000000000007e-06 2020-01-16T00:14:04.544 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.545 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.545 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:04.545 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728828+0000", 2020-01-16T00:14:04.545 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000124824 2020-01-16T00:14:04.545 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.545 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.545 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:04.546 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728953+0000", 2020-01-16T00:14:04.546 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00044904899999999998 2020-01-16T00:14:04.546 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.546 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.546 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:04.546 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729402+0000", 2020-01-16T00:14:04.547 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012295706999999999 2020-01-16T00:14:04.547 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.547 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.547 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:04.547 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.741698+0000", 2020-01-16T00:14:04.547 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012922950000000001 2020-01-16T00:14:04.547 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.548 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:04.548 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.548 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.548 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.548 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3275 3.d 3:b76659c4:::smithi12313934-47:head [read 0~3647152,checksum] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:04.548 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.728933+0000", 2020-01-16T00:14:04.548 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 33.680892198999999, 2020-01-16T00:14:04.549 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01204874, 2020-01-16T00:14:04.549 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:04.549 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:04.549 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:04.549 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:04.549 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:04.550 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3275 2020-01-16T00:14:04.550 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.550 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:04.550 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.550 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:04.550 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728933+0000", 2020-01-16T00:14:04.550 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:04.551 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.551 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.551 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:04.551 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728933+0000", 2020-01-16T00:14:04.551 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:04.551 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.552 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.552 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:04.552 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728932+0000", 2020-01-16T00:14:04.552 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3569999999999998e-06 2020-01-16T00:14:04.552 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.552 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.552 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:04.553 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728940+0000", 2020-01-16T00:14:04.553 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0457e-05 2020-01-16T00:14:04.553 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.553 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.553 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:04.553 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728951+0000", 2020-01-16T00:14:04.553 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6799999999999993e-06 2020-01-16T00:14:04.554 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.554 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.554 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:04.554 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728959+0000", 2020-01-16T00:14:04.554 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010076307 2020-01-16T00:14:04.554 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.554 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.555 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:04.555 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.739035+0000", 2020-01-16T00:14:04.555 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024455400000000002 2020-01-16T00:14:04.555 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.555 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.555 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:04.555 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.739279+0000", 2020-01-16T00:14:04.556 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0017023660000000001 2020-01-16T00:14:04.556 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.556 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.556 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:04.556 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.740982+0000", 2020-01-16T00:14:04.556 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01204874 2020-01-16T00:14:04.556 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.557 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:04.557 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.557 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.557 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.557 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3285 3.d 3:bd9fc72f:::smithi12313934-5:head [read 0~3764801,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:04.557 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.729068+0000", 2020-01-16T00:14:04.557 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 33.680757407000002, 2020-01-16T00:14:04.558 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018537010999999999, 2020-01-16T00:14:04.558 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:04.558 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:04.558 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:04.558 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:04.558 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:04.559 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3285 2020-01-16T00:14:04.559 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.559 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:04.559 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.559 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:04.559 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729068+0000", 2020-01-16T00:14:04.559 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:04.559 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.560 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.560 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:04.560 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729068+0000", 2020-01-16T00:14:04.560 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:04.560 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.560 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.561 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:04.561 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729067+0000", 2020-01-16T00:14:04.561 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3339999999999996e-06 2020-01-16T00:14:04.561 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.561 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.561 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:04.561 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729075+0000", 2020-01-16T00:14:04.562 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4187e-05 2020-01-16T00:14:04.562 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.562 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.562 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:04.562 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729089+0000", 2020-01-16T00:14:04.562 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5600000000000004e-06 2020-01-16T00:14:04.562 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.563 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.563 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:04.563 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729093+0000", 2020-01-16T00:14:04.563 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011944309 2020-01-16T00:14:04.563 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.563 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.563 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:04.564 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.741038+0000", 2020-01-16T00:14:04.564 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016625799999999999 2020-01-16T00:14:04.564 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.564 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.564 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:04.564 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.741204+0000", 2020-01-16T00:14:04.564 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0064010120000000002 2020-01-16T00:14:04.565 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.565 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.565 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:04.565 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.747605+0000", 2020-01-16T00:14:04.565 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018537010999999999 2020-01-16T00:14:04.565 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.565 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:04.566 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.566 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.566 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.566 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3286 3.d 3:bd9fc72f:::smithi12313934-5:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:04.566 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.729099+0000", 2020-01-16T00:14:04.566 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 33.680726737000001, 2020-01-16T00:14:04.566 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018983897999999999, 2020-01-16T00:14:04.567 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:04.567 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:04.567 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:04.567 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:04.567 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:04.567 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3286 2020-01-16T00:14:04.567 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.568 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:04.568 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.568 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:04.568 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729099+0000", 2020-01-16T00:14:04.568 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:04.568 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.568 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.569 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:04.569 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729099+0000", 2020-01-16T00:14:04.569 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:04.569 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.569 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.569 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:04.570 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729098+0000", 2020-01-16T00:14:04.570 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8500000000000004e-06 2020-01-16T00:14:04.570 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.570 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.570 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:04.570 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729102+0000", 2020-01-16T00:14:04.570 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7329999999999997e-06 2020-01-16T00:14:04.571 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.571 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.571 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:04.571 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729108+0000", 2020-01-16T00:14:04.571 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.764e-06 2020-01-16T00:14:04.571 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.571 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.572 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:04.572 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729118+0000", 2020-01-16T00:14:04.572 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018524286000000001 2020-01-16T00:14:04.572 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.572 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.572 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:04.572 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.747642+0000", 2020-01-16T00:14:04.573 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9487999999999993e-05 2020-01-16T00:14:04.573 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.573 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.573 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:04.573 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.747742+0000", 2020-01-16T00:14:04.573 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000340429 2020-01-16T00:14:04.574 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.574 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.574 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:04.574 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.748082+0000", 2020-01-16T00:14:04.574 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018983897999999999 2020-01-16T00:14:04.574 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.574 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:04.575 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.575 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.575 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.575 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3287 3.d 3:bd9fc72f:::smithi12313934-5:head [read 0~3764801,checksum] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:04.575 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.729124+0000", 2020-01-16T00:14:04.575 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 33.680701597999999, 2020-01-16T00:14:04.575 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020552323000000001, 2020-01-16T00:14:04.576 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:04.576 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:04.576 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:04.576 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:04.576 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:04.576 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3287 2020-01-16T00:14:04.577 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.577 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:04.577 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.577 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:04.577 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729124+0000", 2020-01-16T00:14:04.577 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:04.577 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.578 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.578 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:04.578 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729124+0000", 2020-01-16T00:14:04.578 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:04.578 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.578 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.579 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:04.579 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729123+0000", 2020-01-16T00:14:04.579 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8239999999999996e-06 2020-01-16T00:14:04.579 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.579 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.579 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:04.579 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729129+0000", 2020-01-16T00:14:04.579 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6909999999999997e-06 2020-01-16T00:14:04.580 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.580 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.580 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:04.580 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729136+0000", 2020-01-16T00:14:04.580 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3329999999999997e-06 2020-01-16T00:14:04.580 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.580 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.581 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:04.581 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729145+0000", 2020-01-16T00:14:04.581 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018965065999999999 2020-01-16T00:14:04.581 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.581 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.581 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:04.581 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.748110+0000", 2020-01-16T00:14:04.582 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015781700000000001 2020-01-16T00:14:04.582 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.582 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.582 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:04.582 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.748268+0000", 2020-01-16T00:14:04.582 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014083800000000001 2020-01-16T00:14:04.582 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.583 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.583 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:04.583 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.749676+0000", 2020-01-16T00:14:04.583 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020552323000000001 2020-01-16T00:14:04.583 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.583 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:04.583 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.584 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.584 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.584 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3409 3.d 3:b3dd5d45:::smithi12313934-16:head [write 591712~786646,stat] snapc 0=[] ondisk+write+known_if_redirected e47)", 2020-01-16T00:14:04.584 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:34.100681+0000", 2020-01-16T00:14:04.584 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 30.309144457999999, 2020-01-16T00:14:04.584 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012248446, 2020-01-16T00:14:04.584 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:04.585 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:04.585 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:04.585 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:04.585 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:04.585 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3409 2020-01-16T00:14:04.585 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.585 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:04.586 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.586 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:04.586 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.100681+0000", 2020-01-16T00:14:04.586 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:04.586 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.586 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.587 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:04.587 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.100681+0000", 2020-01-16T00:14:04.587 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:14:04.587 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.587 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.587 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:04.587 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.100677+0000", 2020-01-16T00:14:04.588 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000998828 2020-01-16T00:14:04.588 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.588 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.588 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:04.588 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101676+0000", 2020-01-16T00:14:04.588 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5846999999999997e-05 2020-01-16T00:14:04.588 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.589 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.589 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:04.589 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101712+0000", 2020-01-16T00:14:04.589 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.364e-05 2020-01-16T00:14:04.589 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.589 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.590 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:04.590 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101725+0000", 2020-01-16T00:14:04.590 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015902700000000001 2020-01-16T00:14:04.590 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.590 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.590 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:04.590 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101884+0000", 2020-01-16T00:14:04.591 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0486e-05 2020-01-16T00:14:04.591 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.591 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.591 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op must wait for map", 2020-01-16T00:14:04.591 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101905+0000", 2020-01-16T00:14:04.591 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0038869009999999999 2020-01-16T00:14:04.592 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.592 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.592 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:04.592 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.105792+0000", 2020-01-16T00:14:04.592 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030748800000000002 2020-01-16T00:14:04.592 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.593 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.593 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:04.593 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.106099+0000", 2020-01-16T00:14:04.593 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042069299999999998 2020-01-16T00:14:04.593 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.593 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.593 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:04.594 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.106520+0000", 2020-01-16T00:14:04.594 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0052864560000000001 2020-01-16T00:14:04.594 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.594 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.594 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:04.594 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.111806+0000", 2020-01-16T00:14:04.594 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00080835800000000001 2020-01-16T00:14:04.595 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.595 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.595 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:04.595 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.112615+0000", 2020-01-16T00:14:04.595 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1490999999999997e-05 2020-01-16T00:14:04.595 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.595 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.596 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:04.596 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.112686+0000", 2020-01-16T00:14:04.596 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000243174 2020-01-16T00:14:04.596 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.596 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.596 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:04.596 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.112929+0000", 2020-01-16T00:14:04.597 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012248446 2020-01-16T00:14:04.597 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.597 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:04.597 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.597 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.597 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.597 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.0.13:31 2.0 2:1afbc8ac:::smithi12313934-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 4096~737792] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e50)", 2020-01-16T00:14:04.598 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:44.061109+0000", 2020-01-16T00:14:04.598 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 20.348716234000001, 2020-01-16T00:14:04.598 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012186723, 2020-01-16T00:14:04.598 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:04.598 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:04.598 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:04.598 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.599 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:04.599 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.061109+0000", 2020-01-16T00:14:04.599 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:04.599 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.599 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.599 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:04.600 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.061109+0000", 2020-01-16T00:14:04.600 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:04.600 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.600 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.600 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:04.600 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.061106+0000", 2020-01-16T00:14:04.600 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012349870000000001 2020-01-16T00:14:04.601 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.601 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.601 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:04.601 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062341+0000", 2020-01-16T00:14:04.601 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0235999999999996e-05 2020-01-16T00:14:04.601 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.601 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.602 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:04.602 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062411+0000", 2020-01-16T00:14:04.602 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7078000000000001e-05 2020-01-16T00:14:04.602 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.602 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.602 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:04.602 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062428+0000", 2020-01-16T00:14:04.603 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2158999999999996e-05 2020-01-16T00:14:04.603 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.603 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.603 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:04.603 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062520+0000", 2020-01-16T00:14:04.603 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011203499999999999 2020-01-16T00:14:04.603 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.604 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.604 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:04.604 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062632+0000", 2020-01-16T00:14:04.604 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017238400000000001 2020-01-16T00:14:04.604 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.604 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.604 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:14:04.605 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062805+0000", 2020-01-16T00:14:04.605 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010264989 2020-01-16T00:14:04.605 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.605 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.605 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:04.605 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073070+0000", 2020-01-16T00:14:04.605 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0533000000000001e-05 2020-01-16T00:14:04.606 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.606 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.606 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:04.606 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073090+0000", 2020-01-16T00:14:04.606 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2309000000000002e-05 2020-01-16T00:14:04.606 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.606 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.607 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:04.607 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073163+0000", 2020-01-16T00:14:04.607 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3035999999999997e-05 2020-01-16T00:14:04.607 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.607 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.607 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:04.607 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073236+0000", 2020-01-16T00:14:04.608 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0056999999999998e-05 2020-01-16T00:14:04.608 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.608 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.608 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:04.608 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073296+0000", 2020-01-16T00:14:04.608 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012186723 2020-01-16T00:14:04.609 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.609 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:04.609 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.609 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.609 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.609 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.0.13:32 2.0 2:1afbc8ac:::smithi12313934-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 916992~596480] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e50)", 2020-01-16T00:14:04.610 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:44.062464+0000", 2020-01-16T00:14:04.610 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 20.347361449000001, 2020-01-16T00:14:04.610 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011316405999999999, 2020-01-16T00:14:04.610 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:04.610 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:04.610 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:04.610 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.611 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:04.611 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062464+0000", 2020-01-16T00:14:04.611 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:04.611 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.611 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.611 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:04.611 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062464+0000", 2020-01-16T00:14:04.612 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:04.612 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.612 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.612 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:04.612 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062462+0000", 2020-01-16T00:14:04.612 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00073330900000000002 2020-01-16T00:14:04.613 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.613 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.613 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:04.613 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063196+0000", 2020-01-16T00:14:04.613 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5330000000000001e-05 2020-01-16T00:14:04.613 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.613 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.614 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:04.614 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063211+0000", 2020-01-16T00:14:04.614 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5840000000000001e-06 2020-01-16T00:14:04.614 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.614 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.614 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:04.615 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063221+0000", 2020-01-16T00:14:04.615 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001786915 2020-01-16T00:14:04.615 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.615 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.617 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:04.617 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.065007+0000", 2020-01-16T00:14:04.617 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00021770000000000001 2020-01-16T00:14:04.617 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.617 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.617 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:04.618 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.065225+0000", 2020-01-16T00:14:04.618 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00032623 2020-01-16T00:14:04.618 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.618 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.618 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:14:04.618 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.065551+0000", 2020-01-16T00:14:04.618 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0078227499999999998 2020-01-16T00:14:04.619 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.619 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.619 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:04.619 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073374+0000", 2020-01-16T00:14:04.619 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.717e-05 2020-01-16T00:14:04.619 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.619 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.620 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:04.620 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073391+0000", 2020-01-16T00:14:04.620 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011493999999999999 2020-01-16T00:14:04.620 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.620 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.620 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:04.620 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073506+0000", 2020-01-16T00:14:04.621 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000159281 2020-01-16T00:14:04.621 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.621 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.621 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:04.621 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073666+0000", 2020-01-16T00:14:04.621 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000114675 2020-01-16T00:14:04.621 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.622 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.622 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:04.622 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073780+0000", 2020-01-16T00:14:04.622 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011316405999999999 2020-01-16T00:14:04.622 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.622 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:04.622 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.623 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.623 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.623 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.0.13:33 2.0 2:1afbc8ac:::smithi12313934-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 1541120~378368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e50)", 2020-01-16T00:14:04.623 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:44.063236+0000", 2020-01-16T00:14:04.623 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 20.346589569999999, 2020-01-16T00:14:04.623 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015290962999999999, 2020-01-16T00:14:04.623 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:04.624 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:04.624 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:04.624 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.624 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:04.624 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063236+0000", 2020-01-16T00:14:04.624 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:04.624 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.625 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.625 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:04.625 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063236+0000", 2020-01-16T00:14:04.625 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:04.625 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.625 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.626 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:04.626 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063234+0000", 2020-01-16T00:14:04.626 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000500702 2020-01-16T00:14:04.626 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.626 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.626 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:04.626 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063735+0000", 2020-01-16T00:14:04.627 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4734e-05 2020-01-16T00:14:04.627 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.627 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.627 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:04.627 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063750+0000", 2020-01-16T00:14:04.627 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.382e-05 2020-01-16T00:14:04.627 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.628 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.628 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:04.628 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063764+0000", 2020-01-16T00:14:04.628 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0059030799999999998 2020-01-16T00:14:04.628 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.628 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.628 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:04.629 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.069667+0000", 2020-01-16T00:14:04.629 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3961999999999994e-05 2020-01-16T00:14:04.629 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.629 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.629 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:04.629 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.069741+0000", 2020-01-16T00:14:04.629 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000114931 2020-01-16T00:14:04.630 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.630 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.630 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:14:04.630 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.069856+0000", 2020-01-16T00:14:04.630 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0039948270000000003 2020-01-16T00:14:04.630 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.631 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.631 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:04.631 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073850+0000", 2020-01-16T00:14:04.631 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0016459140000000001 2020-01-16T00:14:04.631 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.631 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.631 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:04.632 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.075496+0000", 2020-01-16T00:14:04.632 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002957997 2020-01-16T00:14:04.632 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.632 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.632 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:04.632 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.078454+0000", 2020-01-16T00:14:04.632 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9190999999999998e-05 2020-01-16T00:14:04.632 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.633 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.633 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:04.633 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.078493+0000", 2020-01-16T00:14:04.633 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3176999999999998e-05 2020-01-16T00:14:04.633 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.633 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.634 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:04.634 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.078527+0000", 2020-01-16T00:14:04.634 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015290962999999999 2020-01-16T00:14:04.634 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.634 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:04.634 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.634 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.635 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.635 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4641 3.d 3:b1521a3e:::smithi12313934-25:head [read 0~3596797,checksum] snapc 0=[] ondisk+read+known_if_redirected e57)", 2020-01-16T00:14:04.635 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:04.086675+0000", 2020-01-16T00:14:04.635 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.323150514, 2020-01-16T00:14:04.635 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011822397, 2020-01-16T00:14:04.635 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:04.635 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:04.636 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:04.636 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:04.636 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:04.636 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4641 2020-01-16T00:14:04.636 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.636 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:04.636 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.637 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:04.637 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.086675+0000", 2020-01-16T00:14:04.637 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:04.637 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.637 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.637 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:04.638 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.086675+0000", 2020-01-16T00:14:04.638 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:04.638 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.638 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.638 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:04.638 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.086674+0000", 2020-01-16T00:14:04.638 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0809999999999998e-06 2020-01-16T00:14:04.639 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.639 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.639 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:04.639 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.086678+0000", 2020-01-16T00:14:04.639 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4889999999999997e-06 2020-01-16T00:14:04.639 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.639 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.640 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:04.640 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.086682+0000", 2020-01-16T00:14:04.640 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9620000000000002e-06 2020-01-16T00:14:04.640 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.640 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.640 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:04.641 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.086685+0000", 2020-01-16T00:14:04.641 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010215452 2020-01-16T00:14:04.641 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.641 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.641 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:04.641 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.096900+0000", 2020-01-16T00:14:04.641 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00021274299999999999 2020-01-16T00:14:04.642 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.642 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.642 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:04.642 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.097113+0000", 2020-01-16T00:14:04.642 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0013839480000000001 2020-01-16T00:14:04.642 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:04.642 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:04.643 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:04.643 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.098497+0000", 2020-01-16T00:14:04.643 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011822397 2020-01-16T00:14:04.643 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.643 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:04.643 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.644 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:04.644 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:04.644 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:04.646 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:14:04.647 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:14:04.647 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:14:04.647 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:14:04.647 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:14:04.647 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:14:04.647 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:14:04.648 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-01-16T00:14:04.648 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-01-16T00:14:04.648 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-01-16T00:14:04.648 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-01-16T00:14:04.648 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-01-16T00:14:04.648 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENXIO: problem getting command descriptions from osd.5 2020-01-16T00:14:04.649 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: problem getting command descriptions from osd.5 2020-01-16T00:14:04.649 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-01-16T00:14:04.649 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-01-16T00:14:04.649 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-01-16T00:14:04.649 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-01-16T00:14:04.649 INFO:tasks.rados.rados.0.smithi123.stdout:1660: expect (ObjNum 572 snap 0 seq_num 572) 2020-01-16T00:14:04.650 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-01-16T00:14:04.650 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:04.640+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:04.745 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:04.740+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:04.802 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:04.815 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-01-16T00:14:04.843 INFO:tasks.rados.rados.0.smithi123.stdout:1659: expect (ObjNum 517 snap 0 seq_num 517) 2020-01-16T00:14:04.858 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:04.856+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:04.947 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:04.946+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:04.971 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:04.981 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-01-16T00:14:05.049 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:05.046+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:05.090 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:05.100 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-01-16T00:14:05.120 INFO:tasks.rados.rados.0.smithi123.stdout:1661: expect (ObjNum 575 snap 0 seq_num 575) 2020-01-16T00:14:05.149 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:05.146+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:05.205 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:05.214 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-01-16T00:14:05.251 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:05.248+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:05.276 INFO:tasks.rados.rados.0.smithi123.stdout:1663: finishing write tid 1 to smithi12313934-16 2020-01-16T00:14:05.277 INFO:tasks.rados.rados.0.smithi123.stdout:1663: finishing write tid 2 to smithi12313934-16 2020-01-16T00:14:05.277 INFO:tasks.rados.rados.0.smithi123.stdout:1664: expect (ObjNum 513 snap 0 seq_num 513) 2020-01-16T00:14:05.351 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:05.350+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:05.353 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:05.362 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-01-16T00:14:05.453 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:05.448+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:05.455 INFO:tasks.rados.rados.0.smithi123.stdout:1666: finishing write tid 1 to smithi12313934-15 2020-01-16T00:14:05.455 INFO:tasks.rados.rados.0.smithi123.stdout:1663: finishing write tid 3 to smithi12313934-16 2020-01-16T00:14:05.456 INFO:tasks.rados.rados.0.smithi123.stdout:1663: finishing write tid 5 to smithi12313934-16 2020-01-16T00:14:05.456 INFO:tasks.rados.rados.0.smithi123.stdout:1663: finishing write tid 6 to smithi12313934-16 2020-01-16T00:14:05.456 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 16 v 265 (ObjNum 592 snap 0 seq_num 592) dirty exists 2020-01-16T00:14:05.456 INFO:tasks.rados.rados.0.smithi123.stdout:1663: left oid 16 (ObjNum 592 snap 0 seq_num 592) 2020-01-16T00:14:05.456 INFO:tasks.rados.rados.0.smithi123.stdout:1665: expect (ObjNum 571 snap 0 seq_num 571) 2020-01-16T00:14:05.508 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:05.517 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-01-16T00:14:05.563 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:05.562+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:05.613 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:05.619 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-01-16T00:14:05.663 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:05.662+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:05.714 INFO:tasks.rados.rados.0.smithi123.stdout:1666: finishing write tid 2 to smithi12313934-15 2020-01-16T00:14:05.714 INFO:tasks.rados.rados.0.smithi123.stdout:1666: finishing write tid 3 to smithi12313934-15 2020-01-16T00:14:05.714 INFO:tasks.rados.rados.0.smithi123.stdout:1666: finishing write tid 5 to smithi12313934-15 2020-01-16T00:14:05.714 INFO:tasks.rados.rados.0.smithi123.stdout:1666: finishing write tid 6 to smithi12313934-15 2020-01-16T00:14:05.714 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 15 v 207 (ObjNum 593 snap 0 seq_num 593) dirty exists 2020-01-16T00:14:05.715 INFO:tasks.rados.rados.0.smithi123.stdout:1666: left oid 15 (ObjNum 593 snap 0 seq_num 593) 2020-01-16T00:14:05.715 INFO:tasks.rados.rados.0.smithi123.stdout:1667: expect (ObjNum 506 snap 0 seq_num 506) 2020-01-16T00:14:05.755 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:05.752+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:05.767 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:05.776 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-01-16T00:14:05.856 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:05.852+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:05.871 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:05.879 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-01-16T00:14:05.890 INFO:tasks.rados.rados.0.smithi123.stdout:1669: expect (ObjNum 566 snap 0 seq_num 566) 2020-01-16T00:14:05.957 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:05.954+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:05.979 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:05.988 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-01-16T00:14:06.033 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2020-01-16T00:14:06.058 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:06.056+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:06.156 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:06.159 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:06.154+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:06.168 INFO:tasks.rados.rados.0.smithi123.stdout:1672: expect (ObjNum 556 snap 0 seq_num 556) 2020-01-16T00:14:06.169 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-01-16T00:14:06.267 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:06.266+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:06.327 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:06.336 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-01-16T00:14:06.379 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:06.378+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:06.421 INFO:tasks.rados.rados.0.smithi123.stdout:1670: finishing write tid 1 to smithi12313934-50 2020-01-16T00:14:06.421 INFO:tasks.rados.rados.0.smithi123.stdout:1673: expect (ObjNum 582 snap 0 seq_num 582) 2020-01-16T00:14:06.470 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:06.466+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:06.478 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:06.487 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-01-16T00:14:06.575 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:06.574+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:06.629 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:06.635 INFO:tasks.rados.rados.0.smithi123.stdout:1671: expect (ObjNum 562 snap 0 seq_num 562) 2020-01-16T00:14:06.638 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-01-16T00:14:06.651 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-01-16T00:14:06.679 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:06.678+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:06.772 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:06.770+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:06.779 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:06.789 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-01-16T00:14:06.810 INFO:teuthology.orchestra.run.smithi123.stderr:noscrub is set 2020-01-16T00:14:06.814 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:14:06.842 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:14:06.869 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:14:06.874 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:06.872+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:06.881 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:14:06.902 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:14:06.903 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:06.903 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:14:06.903 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:14:06.903 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:06.912 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-01-16T00:14:06.928 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:14:06.943 INFO:tasks.rados.rados.0.smithi123.stdout:1670: finishing write tid 2 to smithi12313934-50 2020-01-16T00:14:06.943 INFO:tasks.rados.rados.0.smithi123.stdout:1670: finishing write tid 4 to smithi12313934-50 2020-01-16T00:14:06.943 INFO:tasks.rados.rados.0.smithi123.stdout:1670: finishing write tid 5 to smithi12313934-50 2020-01-16T00:14:06.943 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 50 v 211 (ObjNum 595 snap 0 seq_num 595) dirty exists 2020-01-16T00:14:06.943 INFO:tasks.rados.rados.0.smithi123.stdout:1670: left oid 50 (ObjNum 595 snap 0 seq_num 595) 2020-01-16T00:14:06.943 INFO:tasks.rados.rados.0.smithi123.stdout:1657: done (15 left) 2020-01-16T00:14:06.944 INFO:tasks.rados.rados.0.smithi123.stdout:1659: done (14 left) 2020-01-16T00:14:06.944 INFO:tasks.rados.rados.0.smithi123.stdout:1660: done (13 left) 2020-01-16T00:14:06.944 INFO:tasks.rados.rados.0.smithi123.stdout:1661: done (12 left) 2020-01-16T00:14:06.944 INFO:tasks.rados.rados.0.smithi123.stdout:1662: done (11 left) 2020-01-16T00:14:06.944 INFO:tasks.rados.rados.0.smithi123.stdout:1663: done (10 left) 2020-01-16T00:14:06.944 INFO:tasks.rados.rados.0.smithi123.stdout:1664: done (9 left) 2020-01-16T00:14:06.944 INFO:tasks.rados.rados.0.smithi123.stdout:1665: done (8 left) 2020-01-16T00:14:06.944 INFO:tasks.rados.rados.0.smithi123.stdout:1666: done (7 left) 2020-01-16T00:14:06.946 INFO:tasks.rados.rados.0.smithi123.stdout:1667: done (6 left) 2020-01-16T00:14:06.946 INFO:tasks.rados.rados.0.smithi123.stdout:1669: done (5 left) 2020-01-16T00:14:06.947 INFO:tasks.rados.rados.0.smithi123.stdout:1670: done (4 left) 2020-01-16T00:14:06.947 INFO:tasks.rados.rados.0.smithi123.stdout:1671: done (3 left) 2020-01-16T00:14:06.947 INFO:tasks.rados.rados.0.smithi123.stdout:1672: done (2 left) 2020-01-16T00:14:06.947 INFO:tasks.rados.rados.0.smithi123.stdout:1673: done (1 left) 2020-01-16T00:14:06.947 INFO:tasks.rados.rados.0.smithi123.stdout:1674: done (0 left) 2020-01-16T00:14:06.947 INFO:tasks.rados.rados.0.smithi123.stdout:1675: read oid 13 snap -1 2020-01-16T00:14:06.947 INFO:tasks.rados.rados.0.smithi123.stdout:1675: expect (ObjNum 560 snap 0 seq_num 560) 2020-01-16T00:14:06.948 INFO:tasks.rados.rados.0.smithi123.stdout:1676: read oid 38 snap -1 2020-01-16T00:14:06.948 INFO:tasks.rados.rados.0.smithi123.stdout:1676: expect (ObjNum 561 snap 0 seq_num 561) 2020-01-16T00:14:06.948 INFO:tasks.rados.rados.0.smithi123.stdout:1677: write oid 17 current snap is 0 2020-01-16T00:14:06.948 INFO:tasks.rados.rados.0.smithi123.stdout:1677: seq_num 596 ranges {416972=723938,1743268=273704} 2020-01-16T00:14:06.950 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:14:06.952 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-01-16T00:14:06.953 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-01-16T00:14:06.953 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-01-16T00:14:06.953 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-01-16T00:14:06.953 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-01-16T00:14:06.953 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENXIO: problem getting command descriptions from osd.5 2020-01-16T00:14:06.953 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: problem getting command descriptions from osd.5 2020-01-16T00:14:06.953 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-01-16T00:14:06.954 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-01-16T00:14:06.957 INFO:tasks.rados.rados.0.smithi123.stdout:1677: writing smithi12313934-17 from 416972 to 1140910 tid 1 2020-01-16T00:14:06.962 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-01-16T00:14:06.962 INFO:tasks.rados.rados.0.smithi123.stdout:1677: writing smithi12313934-17 from 1743268 to 2016972 tid 2 2020-01-16T00:14:06.962 INFO:tasks.rados.rados.0.smithi123.stdout:1678: write (excl) oid 35 current snap is 0 2020-01-16T00:14:06.963 INFO:tasks.rados.rados.0.smithi123.stdout:1678: seq_num 597 ranges {433779=799187,1801750=232029} 2020-01-16T00:14:06.963 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-01-16T00:14:06.968 INFO:tasks.rados.rados.0.smithi123.stdout:1678: writing smithi12313934-35 from 433779 to 1232966 tid 1 2020-01-16T00:14:06.973 INFO:tasks.rados.rados.0.smithi123.stdout:1678: writing smithi12313934-35 from 1801750 to 2033779 tid 2 2020-01-16T00:14:06.973 INFO:tasks.rados.rados.0.smithi123.stdout:1679: write oid 28 current snap is 0 2020-01-16T00:14:06.973 INFO:tasks.rados.rados.0.smithi123.stdout:1679: seq_num 598 ranges {450586=474436,1343879=588206} 2020-01-16T00:14:06.975 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:06.974+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:06.978 INFO:tasks.rados.rados.0.smithi123.stdout:1679: writing smithi12313934-28 from 450586 to 925022 tid 1 2020-01-16T00:14:06.984 INFO:tasks.rados.rados.0.smithi123.stdout:1679: writing smithi12313934-28 from 1343879 to 1932085 tid 2 2020-01-16T00:14:06.984 INFO:tasks.rados.rados.0.smithi123.stdout:1680: write (excl) oid 48 current snap is 0 2020-01-16T00:14:06.984 INFO:tasks.rados.rados.0.smithi123.stdout:1680: seq_num 599 ranges {467393=549685,1802361=265032} 2020-01-16T00:14:06.989 INFO:tasks.rados.rados.0.smithi123.stdout:1680: writing smithi12313934-48 from 467393 to 1017078 tid 1 2020-01-16T00:14:06.995 INFO:tasks.rados.rados.0.smithi123.stdout:1680: writing smithi12313934-48 from 1802361 to 2067393 tid 2 2020-01-16T00:14:06.996 INFO:tasks.rados.rados.0.smithi123.stdout:1681: read oid 10 snap -1 2020-01-16T00:14:06.996 INFO:tasks.rados.rados.0.smithi123.stdout:1681: expect deleted 2020-01-16T00:14:06.996 INFO:tasks.rados.rados.0.smithi123.stdout:1682: write (excl) oid 34 current snap is 0 2020-01-16T00:14:06.996 INFO:tasks.rados.rados.0.smithi123.stdout:1682: seq_num 600 ranges {484200=624934,1860843=223357} 2020-01-16T00:14:07.005 INFO:tasks.rados.rados.0.smithi123.stdout:1682: writing smithi12313934-34 from 484200 to 1109134 tid 1 2020-01-16T00:14:07.007 INFO:tasks.rados.rados.0.smithi123.stdout:1682: writing smithi12313934-34 from 1860843 to 2084200 tid 2 2020-01-16T00:14:07.007 INFO:tasks.rados.rados.0.smithi123.stdout:1683: read oid 15 snap -1 2020-01-16T00:14:07.007 INFO:tasks.rados.rados.0.smithi123.stdout:1683: expect (ObjNum 593 snap 0 seq_num 593) 2020-01-16T00:14:07.007 INFO:tasks.rados.rados.0.smithi123.stdout:1684: write (excl) oid 42 current snap is 0 2020-01-16T00:14:07.008 INFO:tasks.rados.rados.0.smithi123.stdout:1684: seq_num 601 ranges {501007=416536,1635678=465329} 2020-01-16T00:14:07.012 INFO:tasks.rados.rados.0.smithi123.stdout:1684: writing smithi12313934-42 from 501007 to 917543 tid 1 2020-01-16T00:14:07.021 INFO:tasks.rados.rados.0.smithi123.stdout:1684: writing smithi12313934-42 from 1635678 to 2101007 tid 2 2020-01-16T00:14:07.021 INFO:tasks.rados.rados.0.smithi123.stdout:1685: read oid 36 snap -1 2020-01-16T00:14:07.021 INFO:tasks.rados.rados.0.smithi123.stdout:1685: expect (ObjNum 487 snap 0 seq_num 487) 2020-01-16T00:14:07.021 INFO:tasks.rados.rados.0.smithi123.stdout:1686: read oid 50 snap -1 2020-01-16T00:14:07.021 INFO:tasks.rados.rados.0.smithi123.stdout:1686: expect (ObjNum 595 snap 0 seq_num 595) 2020-01-16T00:14:07.021 INFO:tasks.rados.rados.0.smithi123.stdout:1687: write oid 41 current snap is 0 2020-01-16T00:14:07.022 INFO:tasks.rados.rados.0.smithi123.stdout:1687: seq_num 602 ranges {517814=491785,1577807=540007} 2020-01-16T00:14:07.025 INFO:tasks.rados.rados.0.smithi123.stdout:1687: writing smithi12313934-41 from 517814 to 1009599 tid 1 2020-01-16T00:14:07.032 INFO:tasks.rados.rados.0.smithi123.stdout:1687: writing smithi12313934-41 from 1577807 to 2117814 tid 2 2020-01-16T00:14:07.032 INFO:tasks.rados.rados.0.smithi123.stdout:1688: read oid 8 snap -1 2020-01-16T00:14:07.032 INFO:tasks.rados.rados.0.smithi123.stdout:1688: expect (ObjNum 565 snap 0 seq_num 565) 2020-01-16T00:14:07.033 INFO:tasks.rados.rados.0.smithi123.stdout:1689: write (excl) oid 46 current snap is 0 2020-01-16T00:14:07.033 INFO:tasks.rados.rados.0.smithi123.stdout:1689: seq_num 603 ranges {534621=567034,1636289=498332} 2020-01-16T00:14:07.038 INFO:tasks.rados.rados.0.smithi123.stdout:1689: writing smithi12313934-46 from 534621 to 1101655 tid 1 2020-01-16T00:14:07.044 INFO:tasks.rados.rados.0.smithi123.stdout:1689: writing smithi12313934-46 from 1636289 to 2134621 tid 2 2020-01-16T00:14:07.045 INFO:tasks.rados.rados.0.smithi123.stdout:1690: write oid 18 current snap is 0 2020-01-16T00:14:07.045 INFO:tasks.rados.rados.0.smithi123.stdout:1690: seq_num 604 ranges {551428=642283,1694771=456657} 2020-01-16T00:14:07.052 INFO:tasks.rados.rados.0.smithi123.stdout:1690: writing smithi12313934-18 from 551428 to 1193711 tid 1 2020-01-16T00:14:07.057 INFO:tasks.rados.rados.0.smithi123.stdout:1690: writing smithi12313934-18 from 1694771 to 2151428 tid 2 2020-01-16T00:14:07.058 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:07.058 INFO:tasks.rados.rados.0.smithi123.stdout:1676: expect (ObjNum 561 snap 0 seq_num 561) 2020-01-16T00:14:07.072 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:07.072 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:14:07.072 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:14:07.072 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:14:07.072 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:07.077 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:07.074+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:07.081 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-01-16T00:14:07.174 INFO:tasks.rados.rados.0.smithi123.stdout:1675: expect (ObjNum 560 snap 0 seq_num 560) 2020-01-16T00:14:07.178 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:07.174+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:07.182 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:07.182 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:14:07.182 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:14:07.182 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:14:07.183 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.183 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:943 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:14:07.183 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:14:07.183 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 89.263541861999997, 2020-01-16T00:14:07.183 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018847004000000001, 2020-01-16T00:14:07.183 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:07.184 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:07.184 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:07.184 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:07.184 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:07.184 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 943 2020-01-16T00:14:07.184 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.184 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:07.185 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.185 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:07.185 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:14:07.185 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:07.185 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.185 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.186 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:07.186 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:14:07.186 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:07.186 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.186 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.186 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:07.187 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914475+0000", 2020-01-16T00:14:07.187 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.508e-05 2020-01-16T00:14:07.187 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.187 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.187 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:07.187 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914490+0000", 2020-01-16T00:14:07.188 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.857e-05 2020-01-16T00:14:07.188 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.188 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.188 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:07.188 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914509+0000", 2020-01-16T00:14:07.188 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.46e-06 2020-01-16T00:14:07.188 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.189 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.189 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:07.189 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914515+0000", 2020-01-16T00:14:07.189 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2612999999999999e-05 2020-01-16T00:14:07.189 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.189 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.189 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.189 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914568+0000", 2020-01-16T00:14:07.190 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028312299999999998 2020-01-16T00:14:07.190 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.190 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.190 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:07.190 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914851+0000", 2020-01-16T00:14:07.190 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012282533 2020-01-16T00:14:07.190 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.191 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.191 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.191 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927133+0000", 2020-01-16T00:14:07.191 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4839999999999995e-05 2020-01-16T00:14:07.191 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.191 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.191 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:07.191 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927208+0000", 2020-01-16T00:14:07.192 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0047425610000000002 2020-01-16T00:14:07.192 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.192 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.192 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.192 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.931951+0000", 2020-01-16T00:14:07.192 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3628999999999994e-05 2020-01-16T00:14:07.192 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.192 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.193 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:07.193 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.932045+0000", 2020-01-16T00:14:07.193 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012810180000000001 2020-01-16T00:14:07.193 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.193 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.193 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:07.193 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933326+0000", 2020-01-16T00:14:07.194 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018847004000000001 2020-01-16T00:14:07.194 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.194 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:07.194 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.194 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.194 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.194 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:944 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:14:07.194 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:14:07.195 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 89.263492306000003, 2020-01-16T00:14:07.195 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019086901999999999, 2020-01-16T00:14:07.195 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:07.195 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:07.195 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:07.195 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:07.195 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:07.195 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 944 2020-01-16T00:14:07.196 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.196 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:07.196 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.197 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:07.197 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:14:07.197 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:07.197 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.197 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.197 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:07.197 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:14:07.198 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:07.198 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.198 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.198 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:07.198 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914527+0000", 2020-01-16T00:14:07.198 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7509999999999999e-06 2020-01-16T00:14:07.198 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.198 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.199 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:07.199 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914532+0000", 2020-01-16T00:14:07.199 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5149999999999997e-06 2020-01-16T00:14:07.199 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.199 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.199 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:07.199 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914538+0000", 2020-01-16T00:14:07.200 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1890000000000007e-06 2020-01-16T00:14:07.200 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.200 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.200 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:07.200 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914546+0000", 2020-01-16T00:14:07.200 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000354951 2020-01-16T00:14:07.201 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.201 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.201 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.201 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914901+0000", 2020-01-16T00:14:07.201 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1594000000000002e-05 2020-01-16T00:14:07.201 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.201 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.201 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:07.202 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914942+0000", 2020-01-16T00:14:07.202 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012300070999999999 2020-01-16T00:14:07.202 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.202 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.202 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.202 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927242+0000", 2020-01-16T00:14:07.202 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4076999999999997e-05 2020-01-16T00:14:07.203 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.203 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.203 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:07.203 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927296+0000", 2020-01-16T00:14:07.203 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0060609180000000002 2020-01-16T00:14:07.203 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.203 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.203 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.204 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933357+0000", 2020-01-16T00:14:07.204 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4488999999999999e-05 2020-01-16T00:14:07.204 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.204 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.204 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:07.205 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933422+0000", 2020-01-16T00:14:07.205 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019318300000000001 2020-01-16T00:14:07.205 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.205 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.206 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:07.206 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933615+0000", 2020-01-16T00:14:07.206 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019086901999999999 2020-01-16T00:14:07.207 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.207 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:07.207 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.207 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.207 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.208 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:945 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:14:07.208 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:14:07.208 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 89.263465100000005, 2020-01-16T00:14:07.208 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019342986, 2020-01-16T00:14:07.209 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:07.209 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:07.209 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:07.210 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:07.210 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:07.210 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 945 2020-01-16T00:14:07.211 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.211 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:07.211 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.211 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:07.212 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:14:07.212 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:07.212 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.212 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.212 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:07.213 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:14:07.213 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:07.213 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.213 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.214 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:07.214 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:14:07.214 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4580000000000002e-06 2020-01-16T00:14:07.214 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.215 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.215 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:07.215 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914558+0000", 2020-01-16T00:14:07.215 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7829999999999998e-06 2020-01-16T00:14:07.216 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.216 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.216 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:07.216 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914563+0000", 2020-01-16T00:14:07.224 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0001e-05 2020-01-16T00:14:07.224 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.224 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.225 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:07.225 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914573+0000", 2020-01-16T00:14:07.225 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00040365600000000001 2020-01-16T00:14:07.225 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.226 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.226 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.226 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914976+0000", 2020-01-16T00:14:07.226 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1390000000000001e-05 2020-01-16T00:14:07.226 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.226 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.227 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:07.227 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.915028+0000", 2020-01-16T00:14:07.227 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012288843000000001 2020-01-16T00:14:07.227 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.227 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.227 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.228 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927317+0000", 2020-01-16T00:14:07.228 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7871000000000002e-05 2020-01-16T00:14:07.228 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.228 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.228 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:07.229 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927375+0000", 2020-01-16T00:14:07.229 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0062580589999999998 2020-01-16T00:14:07.229 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.229 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.229 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.229 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933633+0000", 2020-01-16T00:14:07.230 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7086999999999997e-05 2020-01-16T00:14:07.230 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.230 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.230 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:07.230 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933690+0000", 2020-01-16T00:14:07.231 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020855799999999999 2020-01-16T00:14:07.231 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.231 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.231 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:07.231 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933898+0000", 2020-01-16T00:14:07.231 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019342986 2020-01-16T00:14:07.232 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.232 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:07.232 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.232 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.232 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.232 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:981 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:14:07.233 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:14:07.233 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 88.862688207999994, 2020-01-16T00:14:07.233 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021222494000000001, 2020-01-16T00:14:07.233 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:07.233 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:07.233 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:07.233 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:07.233 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:07.234 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 981 2020-01-16T00:14:07.234 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.234 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:07.234 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.235 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:07.235 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:14:07.235 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:07.235 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.235 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.235 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:07.235 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:14:07.235 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999948 2020-01-16T00:14:07.236 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.236 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.236 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:07.236 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315327+0000", 2020-01-16T00:14:07.236 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.508e-05 2020-01-16T00:14:07.236 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.236 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.237 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:07.237 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315352+0000", 2020-01-16T00:14:07.237 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1183999999999998e-05 2020-01-16T00:14:07.237 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.237 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.237 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:07.237 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315383+0000", 2020-01-16T00:14:07.237 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7180000000000002e-05 2020-01-16T00:14:07.238 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.238 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.238 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:07.238 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315401+0000", 2020-01-16T00:14:07.238 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017290599999999999 2020-01-16T00:14:07.238 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.238 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.239 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.239 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315574+0000", 2020-01-16T00:14:07.239 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00047908100000000002 2020-01-16T00:14:07.239 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.239 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.239 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:07.239 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316053+0000", 2020-01-16T00:14:07.239 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011808947 2020-01-16T00:14:07.240 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.240 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.240 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.240 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.327862+0000", 2020-01-16T00:14:07.240 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010067499999999999 2020-01-16T00:14:07.240 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.240 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.240 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:07.241 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.327962+0000", 2020-01-16T00:14:07.241 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0074745139999999998 2020-01-16T00:14:07.241 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.241 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.241 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.241 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.335437+0000", 2020-01-16T00:14:07.241 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000103403 2020-01-16T00:14:07.242 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.242 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.242 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:07.242 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.335540+0000", 2020-01-16T00:14:07.242 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001014544 2020-01-16T00:14:07.242 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.242 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.243 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:07.243 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336555+0000", 2020-01-16T00:14:07.243 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021222494000000001 2020-01-16T00:14:07.243 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.243 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:07.243 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.243 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.243 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.244 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:982 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:14:07.244 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:14:07.244 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 88.862585683999995, 2020-01-16T00:14:07.244 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.022087721000000001, 2020-01-16T00:14:07.244 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:07.245 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:07.245 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:07.245 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:07.245 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:07.245 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 982 2020-01-16T00:14:07.245 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.245 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:07.245 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.246 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:07.246 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:14:07.246 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:07.246 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.246 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.246 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:07.246 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:14:07.247 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:07.247 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.247 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.247 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:07.247 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315433+0000", 2020-01-16T00:14:07.247 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1347e-05 2020-01-16T00:14:07.247 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.247 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.248 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:07.248 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315445+0000", 2020-01-16T00:14:07.248 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4525999999999999e-05 2020-01-16T00:14:07.248 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.248 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.248 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:07.249 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315459+0000", 2020-01-16T00:14:07.249 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.024e-05 2020-01-16T00:14:07.249 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.249 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.250 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:07.250 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315470+0000", 2020-01-16T00:14:07.250 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000736468 2020-01-16T00:14:07.250 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.250 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.251 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.251 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316206+0000", 2020-01-16T00:14:07.251 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000132967 2020-01-16T00:14:07.251 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.251 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.252 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:07.252 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316339+0000", 2020-01-16T00:14:07.252 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011701717 2020-01-16T00:14:07.252 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.252 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.253 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.253 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328041+0000", 2020-01-16T00:14:07.253 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000120339 2020-01-16T00:14:07.253 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.254 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.254 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:07.254 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328161+0000", 2020-01-16T00:14:07.254 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0084341260000000001 2020-01-16T00:14:07.255 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.255 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.255 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.255 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336595+0000", 2020-01-16T00:14:07.256 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5247999999999993e-05 2020-01-16T00:14:07.256 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.256 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.256 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:07.256 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336690+0000", 2020-01-16T00:14:07.256 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00083204000000000004 2020-01-16T00:14:07.256 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.256 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.257 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:07.257 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337522+0000", 2020-01-16T00:14:07.257 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.022087721000000001 2020-01-16T00:14:07.257 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.257 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:07.257 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.257 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.258 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.258 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:983 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:14:07.258 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:14:07.258 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 88.862465974000003, 2020-01-16T00:14:07.258 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023363387999999999, 2020-01-16T00:14:07.258 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:07.258 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:07.258 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:07.259 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:07.259 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:07.259 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 983 2020-01-16T00:14:07.259 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.259 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:07.259 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.259 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:07.259 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:14:07.260 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:07.260 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.260 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.260 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:07.260 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:14:07.260 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:07.260 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.261 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.261 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:07.261 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315553+0000", 2020-01-16T00:14:07.261 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1760000000000004e-06 2020-01-16T00:14:07.261 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.261 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.261 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:07.261 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315560+0000", 2020-01-16T00:14:07.262 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3587000000000001e-05 2020-01-16T00:14:07.262 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.262 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.262 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:07.262 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315574+0000", 2020-01-16T00:14:07.262 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7800000000000003e-06 2020-01-16T00:14:07.262 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.263 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.263 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:07.263 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315581+0000", 2020-01-16T00:14:07.263 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000823024 2020-01-16T00:14:07.263 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.263 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.263 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.263 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316404+0000", 2020-01-16T00:14:07.264 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2413e-05 2020-01-16T00:14:07.264 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.264 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.264 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:07.264 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316466+0000", 2020-01-16T00:14:07.264 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011738561999999999 2020-01-16T00:14:07.264 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.264 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.265 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.265 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328205+0000", 2020-01-16T00:14:07.265 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6783999999999997e-05 2020-01-16T00:14:07.265 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.266 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.266 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:07.266 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328262+0000", 2020-01-16T00:14:07.266 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0093022560000000001 2020-01-16T00:14:07.266 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.266 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.266 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.266 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337564+0000", 2020-01-16T00:14:07.267 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000183151 2020-01-16T00:14:07.267 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.267 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.267 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:07.267 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337747+0000", 2020-01-16T00:14:07.267 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011707449999999999 2020-01-16T00:14:07.267 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.267 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.268 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:07.268 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.338918+0000", 2020-01-16T00:14:07.268 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023363387999999999 2020-01-16T00:14:07.268 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.268 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:07.268 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.268 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.269 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.269 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:986 3.9 3:93dfe6f1:::smithi12313934-40:head [write 2941380~419987] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:14:07.269 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:14:07.269 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 88.697166198000005, 2020-01-16T00:14:07.269 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016978400000000001, 2020-01-16T00:14:07.269 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:07.269 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:07.269 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:07.270 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:07.270 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:07.270 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 986 2020-01-16T00:14:07.270 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.270 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:07.270 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.270 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:07.270 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:14:07.271 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:07.271 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.271 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.273 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:07.273 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:14:07.273 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:07.273 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.274 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.274 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:07.274 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480852+0000", 2020-01-16T00:14:07.274 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042503899999999999 2020-01-16T00:14:07.274 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.274 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.274 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:07.274 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481277+0000", 2020-01-16T00:14:07.275 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3097e-05 2020-01-16T00:14:07.275 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.275 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.275 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:07.275 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481290+0000", 2020-01-16T00:14:07.276 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8460000000000002e-06 2020-01-16T00:14:07.276 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.276 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.276 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:07.276 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481297+0000", 2020-01-16T00:14:07.276 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8443999999999999e-05 2020-01-16T00:14:07.276 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.277 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.277 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.277 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481346+0000", 2020-01-16T00:14:07.277 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022115399999999999 2020-01-16T00:14:07.277 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.277 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.277 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:07.277 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481567+0000", 2020-01-16T00:14:07.278 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012531502 2020-01-16T00:14:07.279 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.279 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.279 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.279 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494098+0000", 2020-01-16T00:14:07.279 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.4462999999999993e-05 2020-01-16T00:14:07.279 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.279 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.280 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:07.280 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494193+0000", 2020-01-16T00:14:07.280 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017600999999999999 2020-01-16T00:14:07.280 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.280 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.280 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:14:07.280 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494369+0000", 2020-01-16T00:14:07.280 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0031739619999999998 2020-01-16T00:14:07.281 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.281 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.281 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:07.281 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497543+0000", 2020-01-16T00:14:07.281 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000175107 2020-01-16T00:14:07.281 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.281 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.282 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:07.282 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497718+0000", 2020-01-16T00:14:07.282 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5207000000000003e-05 2020-01-16T00:14:07.282 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.282 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.282 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:07.282 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497763+0000", 2020-01-16T00:14:07.282 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7042999999999998e-05 2020-01-16T00:14:07.283 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.283 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.283 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:07.283 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497800+0000", 2020-01-16T00:14:07.283 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2628e-05 2020-01-16T00:14:07.283 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.283 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.284 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:07.284 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497833+0000", 2020-01-16T00:14:07.284 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016978400000000001 2020-01-16T00:14:07.284 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.284 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:07.284 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.284 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.284 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.285 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:988 3.9 3:93dfe6f1:::smithi12313934-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:14:07.285 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:14:07.285 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 88.696680733999997, 2020-01-16T00:14:07.285 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017221692, 2020-01-16T00:14:07.285 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:07.285 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:07.285 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:07.286 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:07.286 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:07.286 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 988 2020-01-16T00:14:07.286 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.286 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:07.286 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.286 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:07.287 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:14:07.287 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:07.287 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.287 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.287 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:07.287 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:14:07.287 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:07.288 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.288 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.288 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:07.288 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481339+0000", 2020-01-16T00:14:07.288 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3089999999999997e-06 2020-01-16T00:14:07.288 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.288 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.289 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:07.289 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481343+0000", 2020-01-16T00:14:07.289 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2230000000000004e-06 2020-01-16T00:14:07.289 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.289 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.289 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:07.290 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481350+0000", 2020-01-16T00:14:07.290 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.319e-06 2020-01-16T00:14:07.290 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.290 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.290 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:07.290 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481356+0000", 2020-01-16T00:14:07.290 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000321544 2020-01-16T00:14:07.290 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.291 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.291 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.291 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481678+0000", 2020-01-16T00:14:07.291 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6375999999999999e-05 2020-01-16T00:14:07.291 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.291 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.291 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:07.291 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481714+0000", 2020-01-16T00:14:07.292 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015557666 2020-01-16T00:14:07.292 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.292 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.292 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.292 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497272+0000", 2020-01-16T00:14:07.292 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012791300000000001 2020-01-16T00:14:07.292 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.292 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.293 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:07.293 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497399+0000", 2020-01-16T00:14:07.293 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00076944600000000002 2020-01-16T00:14:07.293 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.293 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.293 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.293 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498169+0000", 2020-01-16T00:14:07.293 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018306799999999999 2020-01-16T00:14:07.294 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.294 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.295 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:07.295 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498352+0000", 2020-01-16T00:14:07.295 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020935999999999999 2020-01-16T00:14:07.295 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.295 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.295 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:07.295 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498561+0000", 2020-01-16T00:14:07.295 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017221692 2020-01-16T00:14:07.296 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.296 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:07.296 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.296 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.296 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.296 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1239 3.3 3:c204aa4f:::smithi12313934-45:head [setxattr _header (56) in=7b,truncate 2000033] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:14:07.296 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:14:07.297 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 84.587606914999995, 2020-01-16T00:14:07.297 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019161661999999999, 2020-01-16T00:14:07.297 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:07.297 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:07.297 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:07.297 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:07.298 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:07.298 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1239 2020-01-16T00:14:07.298 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.298 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:07.298 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.298 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:07.298 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:14:07.299 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:07.299 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.299 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.299 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:07.299 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:14:07.299 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:07.299 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.299 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.300 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:07.300 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590411+0000", 2020-01-16T00:14:07.300 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3879999999999999e-05 2020-01-16T00:14:07.300 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.300 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.300 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:07.300 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590425+0000", 2020-01-16T00:14:07.300 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6542999999999998e-05 2020-01-16T00:14:07.301 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.301 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.301 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:07.301 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590471+0000", 2020-01-16T00:14:07.301 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8907999999999999e-05 2020-01-16T00:14:07.301 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.301 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.301 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:07.302 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590490+0000", 2020-01-16T00:14:07.302 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0045691960000000002 2020-01-16T00:14:07.302 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.302 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.302 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.302 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.595059+0000", 2020-01-16T00:14:07.302 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0035638940000000002 2020-01-16T00:14:07.302 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.303 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.303 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:07.303 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598623+0000", 2020-01-16T00:14:07.303 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0077558310000000004 2020-01-16T00:14:07.303 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.303 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.303 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.303 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606379+0000", 2020-01-16T00:14:07.304 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012171999999999999 2020-01-16T00:14:07.304 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.304 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.304 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:07.304 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606501+0000", 2020-01-16T00:14:07.304 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013144200000000001 2020-01-16T00:14:07.304 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.305 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.305 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2020-01-16T00:14:07.305 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606632+0000", 2020-01-16T00:14:07.305 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001288565 2020-01-16T00:14:07.305 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.305 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.305 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:07.306 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607921+0000", 2020-01-16T00:14:07.306 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033399299999999999 2020-01-16T00:14:07.306 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.306 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.306 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:07.306 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.608255+0000", 2020-01-16T00:14:07.306 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012280870000000001 2020-01-16T00:14:07.307 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.307 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.307 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:07.307 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609483+0000", 2020-01-16T00:14:07.307 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2286000000000001e-05 2020-01-16T00:14:07.307 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.308 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.308 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:07.308 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609525+0000", 2020-01-16T00:14:07.308 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0121999999999999e-05 2020-01-16T00:14:07.308 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.308 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.308 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:07.308 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609575+0000", 2020-01-16T00:14:07.309 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019161661999999999 2020-01-16T00:14:07.309 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.309 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:07.309 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.309 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.309 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.309 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1240 3.3 3:c204aa4f:::smithi12313934-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-01-16T00:14:07.310 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:14:07.310 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 84.587506726000001, 2020-01-16T00:14:07.310 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019268828000000002, 2020-01-16T00:14:07.310 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:07.310 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:07.310 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:07.310 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:07.311 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:07.311 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1240 2020-01-16T00:14:07.311 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.311 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:07.311 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.311 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:07.311 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:14:07.311 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:07.312 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.312 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.312 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:07.312 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:14:07.312 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:07.313 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.313 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.313 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:07.313 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590512+0000", 2020-01-16T00:14:07.313 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0525e-05 2020-01-16T00:14:07.313 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.313 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.314 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:07.314 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590522+0000", 2020-01-16T00:14:07.314 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6418000000000001e-05 2020-01-16T00:14:07.314 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.314 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.314 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:07.314 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590539+0000", 2020-01-16T00:14:07.314 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0679999999999999e-05 2020-01-16T00:14:07.315 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.315 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.315 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:07.315 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590549+0000", 2020-01-16T00:14:07.315 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0083626789999999996 2020-01-16T00:14:07.315 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.315 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.316 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.316 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598912+0000", 2020-01-16T00:14:07.316 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013344 2020-01-16T00:14:07.316 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.316 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.316 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:07.317 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599045+0000", 2020-01-16T00:14:07.317 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0084002269999999997 2020-01-16T00:14:07.317 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.317 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.317 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.317 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607446+0000", 2020-01-16T00:14:07.318 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000109873 2020-01-16T00:14:07.318 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.318 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.318 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:07.318 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607555+0000", 2020-01-16T00:14:07.318 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0020573309999999999 2020-01-16T00:14:07.319 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.319 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.319 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.319 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609613+0000", 2020-01-16T00:14:07.319 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6351e-05 2020-01-16T00:14:07.319 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.319 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.319 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:07.320 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609699+0000", 2020-01-16T00:14:07.320 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3374999999999994e-05 2020-01-16T00:14:07.320 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.320 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.320 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:07.320 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609782+0000", 2020-01-16T00:14:07.320 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019268828000000002 2020-01-16T00:14:07.321 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.321 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:07.321 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.321 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.321 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.321 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3124 3.9 3:9fc687b2:::smithi12313934-44:head [write 1903628~435462] snapc 0=[] ondisk+write+known_if_redirected e44)", 2020-01-16T00:14:07.321 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.276145+0000", 2020-01-16T00:14:07.321 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 41.901875005000001, 2020-01-16T00:14:07.322 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017361898000000001, 2020-01-16T00:14:07.322 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:07.322 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:07.322 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:07.322 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:07.322 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:07.322 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3124 2020-01-16T00:14:07.323 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.323 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:07.323 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.323 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:07.323 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276145+0000", 2020-01-16T00:14:07.323 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:07.323 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.323 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.324 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:07.324 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276145+0000", 2020-01-16T00:14:07.324 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:07.324 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.324 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.324 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:07.324 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276143+0000", 2020-01-16T00:14:07.324 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042136099999999998 2020-01-16T00:14:07.325 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.325 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.325 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:07.325 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276564+0000", 2020-01-16T00:14:07.325 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.349e-05 2020-01-16T00:14:07.325 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.325 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.326 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:07.326 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276578+0000", 2020-01-16T00:14:07.326 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2469999999999999e-06 2020-01-16T00:14:07.326 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.326 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.326 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:07.326 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276583+0000", 2020-01-16T00:14:07.326 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4400000000000001e-05 2020-01-16T00:14:07.327 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.327 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.327 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.327 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276637+0000", 2020-01-16T00:14:07.327 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0046361789999999998 2020-01-16T00:14:07.327 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.327 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.327 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:07.328 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281274+0000", 2020-01-16T00:14:07.328 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0090025520000000005 2020-01-16T00:14:07.328 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.328 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.328 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.328 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.290276+0000", 2020-01-16T00:14:07.329 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011584200000000001 2020-01-16T00:14:07.329 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.329 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.329 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:07.329 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.290392+0000", 2020-01-16T00:14:07.329 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00025563099999999998 2020-01-16T00:14:07.329 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.330 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.330 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:14:07.330 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.290648+0000", 2020-01-16T00:14:07.330 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0026741360000000001 2020-01-16T00:14:07.330 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.330 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.330 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:07.330 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293322+0000", 2020-01-16T00:14:07.331 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9764000000000002e-05 2020-01-16T00:14:07.331 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.331 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.331 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:07.331 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293421+0000", 2020-01-16T00:14:07.331 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4779000000000001e-05 2020-01-16T00:14:07.331 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.332 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.332 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:07.332 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293446+0000", 2020-01-16T00:14:07.332 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4628000000000001e-05 2020-01-16T00:14:07.332 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.332 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.332 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:07.332 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293481+0000", 2020-01-16T00:14:07.333 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.6444999999999998e-05 2020-01-16T00:14:07.333 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.333 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.333 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:07.333 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293507+0000", 2020-01-16T00:14:07.333 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017361898000000001 2020-01-16T00:14:07.333 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.333 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:07.334 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.334 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.334 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.334 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3125 3.9 3:9fc687b2:::smithi12313934-44:head [setxattr _header (56) in=7b,truncate 2386660] snapc 0=[] ondisk+write+known_if_redirected e44)", 2020-01-16T00:14:07.334 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.276594+0000", 2020-01-16T00:14:07.334 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 41.901426233000002, 2020-01-16T00:14:07.334 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017760560000000002, 2020-01-16T00:14:07.335 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:07.335 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:07.335 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:07.335 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:07.335 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:07.335 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3125 2020-01-16T00:14:07.335 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.335 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:07.336 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.336 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:07.336 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276594+0000", 2020-01-16T00:14:07.336 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:07.336 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.336 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.336 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:07.336 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276594+0000", 2020-01-16T00:14:07.337 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:07.337 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.337 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.337 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:07.337 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276593+0000", 2020-01-16T00:14:07.337 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7320000000000003e-06 2020-01-16T00:14:07.337 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.338 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.338 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:07.338 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276599+0000", 2020-01-16T00:14:07.338 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3302000000000001e-05 2020-01-16T00:14:07.338 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.338 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.339 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:07.339 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276612+0000", 2020-01-16T00:14:07.339 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1659999999999998e-06 2020-01-16T00:14:07.339 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.339 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.339 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:07.339 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276615+0000", 2020-01-16T00:14:07.340 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0047328450000000003 2020-01-16T00:14:07.340 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.340 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.340 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.340 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281348+0000", 2020-01-16T00:14:07.340 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3966000000000003e-05 2020-01-16T00:14:07.340 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.340 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.340 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:07.341 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281422+0000", 2020-01-16T00:14:07.341 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01082613 2020-01-16T00:14:07.341 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.341 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.341 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.341 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.292248+0000", 2020-01-16T00:14:07.341 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000148617 2020-01-16T00:14:07.341 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.342 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.342 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:07.342 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.292397+0000", 2020-01-16T00:14:07.342 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019936300000000001 2020-01-16T00:14:07.342 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.342 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.342 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:14:07.343 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.292596+0000", 2020-01-16T00:14:07.343 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011208870000000001 2020-01-16T00:14:07.343 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.343 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.343 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:07.343 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293717+0000", 2020-01-16T00:14:07.343 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00039491599999999998 2020-01-16T00:14:07.343 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.344 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.344 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:07.344 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294112+0000", 2020-01-16T00:14:07.344 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000129326 2020-01-16T00:14:07.344 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.344 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.344 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:07.344 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294241+0000", 2020-01-16T00:14:07.345 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5586000000000001e-05 2020-01-16T00:14:07.345 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.345 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.345 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:07.345 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294277+0000", 2020-01-16T00:14:07.345 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7893999999999997e-05 2020-01-16T00:14:07.345 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.346 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.346 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:07.346 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294355+0000", 2020-01-16T00:14:07.346 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017760560000000002 2020-01-16T00:14:07.346 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.346 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:07.346 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.347 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.347 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.347 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3126 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e44)", 2020-01-16T00:14:07.347 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.276634+0000", 2020-01-16T00:14:07.347 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 41.901386518000002, 2020-01-16T00:14:07.347 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017942583000000002, 2020-01-16T00:14:07.347 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:07.348 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:07.348 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:07.348 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:07.348 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:07.348 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3126 2020-01-16T00:14:07.348 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.349 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:07.349 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.349 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:07.349 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276634+0000", 2020-01-16T00:14:07.349 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:07.349 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.349 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.349 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:07.350 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276634+0000", 2020-01-16T00:14:07.350 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:07.350 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.350 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.350 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:07.350 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276633+0000", 2020-01-16T00:14:07.350 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0769999999999999e-06 2020-01-16T00:14:07.351 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.351 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.351 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:07.351 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276636+0000", 2020-01-16T00:14:07.351 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5390000000000001e-06 2020-01-16T00:14:07.351 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.351 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.351 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:07.352 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276641+0000", 2020-01-16T00:14:07.352 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9969999999999999e-06 2020-01-16T00:14:07.352 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.352 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.352 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:07.352 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276644+0000", 2020-01-16T00:14:07.352 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0048246970000000002 2020-01-16T00:14:07.352 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.353 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.353 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.353 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281469+0000", 2020-01-16T00:14:07.353 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8723999999999999e-05 2020-01-16T00:14:07.353 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.353 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.353 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:07.353 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281567+0000", 2020-01-16T00:14:07.354 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011633447 2020-01-16T00:14:07.354 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.354 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.354 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.354 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293201+0000", 2020-01-16T00:14:07.354 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6796000000000002e-05 2020-01-16T00:14:07.354 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.355 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.355 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:07.355 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293268+0000", 2020-01-16T00:14:07.355 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001125247 2020-01-16T00:14:07.355 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.355 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.355 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.355 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294393+0000", 2020-01-16T00:14:07.356 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.6082000000000006e-05 2020-01-16T00:14:07.356 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.356 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.356 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:07.356 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294489+0000", 2020-01-16T00:14:07.356 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7515000000000005e-05 2020-01-16T00:14:07.356 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.356 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.357 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:07.357 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294576+0000", 2020-01-16T00:14:07.358 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017942583000000002 2020-01-16T00:14:07.358 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.358 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:07.358 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.358 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.358 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.358 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3204 3.9 3:93dfe6f1:::smithi12313934-40:head [setxattr _header (56) in=7b,truncate 2554730] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:14:07.359 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.568544+0000", 2020-01-16T00:14:07.359 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 37.609476297999997, 2020-01-16T00:14:07.359 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.026260867, 2020-01-16T00:14:07.359 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:07.359 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:07.360 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:07.360 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:07.361 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:07.361 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3204 2020-01-16T00:14:07.361 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.361 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:07.361 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.361 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:07.361 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568544+0000", 2020-01-16T00:14:07.362 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:07.362 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.362 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.362 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:07.362 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568544+0000", 2020-01-16T00:14:07.362 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:07.362 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.363 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.363 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:07.363 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568543+0000", 2020-01-16T00:14:07.363 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0810000000000001e-06 2020-01-16T00:14:07.363 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.363 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.363 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:07.364 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568548+0000", 2020-01-16T00:14:07.364 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6950000000000002e-06 2020-01-16T00:14:07.364 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.364 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.365 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:07.365 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568552+0000", 2020-01-16T00:14:07.365 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.875e-06 2020-01-16T00:14:07.365 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.365 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.365 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:07.365 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568555+0000", 2020-01-16T00:14:07.366 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0033130849999999999 2020-01-16T00:14:07.366 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.366 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.366 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.367 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.571868+0000", 2020-01-16T00:14:07.367 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0026014609999999998 2020-01-16T00:14:07.367 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.367 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.367 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:07.368 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.574469+0000", 2020-01-16T00:14:07.368 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010261381999999999 2020-01-16T00:14:07.368 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.368 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.368 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.368 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.584731+0000", 2020-01-16T00:14:07.368 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019704 2020-01-16T00:14:07.369 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.369 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.369 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:07.369 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.584928+0000", 2020-01-16T00:14:07.369 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000280869 2020-01-16T00:14:07.369 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.369 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.370 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:14:07.370 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.585209+0000", 2020-01-16T00:14:07.370 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002273859 2020-01-16T00:14:07.370 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.370 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.370 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:07.371 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.587483+0000", 2020-01-16T00:14:07.371 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3356000000000001e-05 2020-01-16T00:14:07.371 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.371 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.371 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:07.371 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.587506+0000", 2020-01-16T00:14:07.371 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0071864939999999999 2020-01-16T00:14:07.372 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.372 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.372 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:07.372 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594692+0000", 2020-01-16T00:14:07.372 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5475000000000001e-05 2020-01-16T00:14:07.372 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.372 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.373 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:07.373 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594738+0000", 2020-01-16T00:14:07.373 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7070000000000007e-05 2020-01-16T00:14:07.373 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.373 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.373 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:07.373 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594805+0000", 2020-01-16T00:14:07.374 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.026260867 2020-01-16T00:14:07.374 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.374 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:07.374 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.374 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.374 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.374 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3205 3.9 3:93dfe6f1:::smithi12313934-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e45)", 2020-01-16T00:14:07.374 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.568599+0000", 2020-01-16T00:14:07.375 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 37.609421363000003, 2020-01-16T00:14:07.375 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.026406882, 2020-01-16T00:14:07.375 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:07.375 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:07.375 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:07.375 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:07.375 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:07.376 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3205 2020-01-16T00:14:07.376 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.376 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:07.376 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.376 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:07.376 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568599+0000", 2020-01-16T00:14:07.376 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:07.376 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.377 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.377 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:07.377 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568599+0000", 2020-01-16T00:14:07.377 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:07.377 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.377 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.377 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:07.378 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568598+0000", 2020-01-16T00:14:07.378 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3939999999999998e-06 2020-01-16T00:14:07.378 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.378 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.378 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:07.378 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568603+0000", 2020-01-16T00:14:07.378 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9391000000000001e-05 2020-01-16T00:14:07.379 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.379 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.379 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:07.379 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568622+0000", 2020-01-16T00:14:07.379 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9599999999999997e-06 2020-01-16T00:14:07.379 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.379 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.379 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:07.380 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568628+0000", 2020-01-16T00:14:07.380 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0058679700000000001 2020-01-16T00:14:07.380 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.380 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.380 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.381 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.574496+0000", 2020-01-16T00:14:07.381 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5528999999999998e-05 2020-01-16T00:14:07.381 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.381 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.381 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:07.381 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.574532+0000", 2020-01-16T00:14:07.381 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011986749 2020-01-16T00:14:07.381 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.382 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.382 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.382 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.586518+0000", 2020-01-16T00:14:07.382 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016522600000000001 2020-01-16T00:14:07.382 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.383 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.383 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:07.383 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.586684+0000", 2020-01-16T00:14:07.383 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0081693890000000009 2020-01-16T00:14:07.383 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.383 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.383 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.383 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594853+0000", 2020-01-16T00:14:07.384 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7082000000000004e-05 2020-01-16T00:14:07.384 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.384 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.384 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:07.384 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594940+0000", 2020-01-16T00:14:07.385 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5789000000000003e-05 2020-01-16T00:14:07.385 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.385 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.385 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:07.385 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.595006+0000", 2020-01-16T00:14:07.385 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.026406882 2020-01-16T00:14:07.385 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.386 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:07.386 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.386 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.386 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.386 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3219 3.3 3:c478bfe8:::smithi12313934-10:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:07.386 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.580957+0000", 2020-01-16T00:14:07.386 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 37.597063847000001, 2020-01-16T00:14:07.387 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016807971000000001, 2020-01-16T00:14:07.387 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:07.387 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:07.387 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:07.387 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:07.387 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:07.387 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3219 2020-01-16T00:14:07.388 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.388 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:07.388 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.388 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:07.388 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580957+0000", 2020-01-16T00:14:07.388 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:07.388 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.388 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.389 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:07.389 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580957+0000", 2020-01-16T00:14:07.389 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:07.389 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.389 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.389 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:07.389 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580955+0000", 2020-01-16T00:14:07.390 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5120000000000002e-06 2020-01-16T00:14:07.390 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.390 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.390 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:07.390 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580963+0000", 2020-01-16T00:14:07.390 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6799999999999993e-06 2020-01-16T00:14:07.391 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.391 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.391 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:07.391 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580971+0000", 2020-01-16T00:14:07.391 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3109999999999996e-06 2020-01-16T00:14:07.391 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.391 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.392 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:07.392 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580976+0000", 2020-01-16T00:14:07.392 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014306146 2020-01-16T00:14:07.392 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.392 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.392 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.392 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.595282+0000", 2020-01-16T00:14:07.393 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.1883000000000001e-05 2020-01-16T00:14:07.393 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.393 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.393 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:07.393 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.595374+0000", 2020-01-16T00:14:07.393 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023904989999999999 2020-01-16T00:14:07.393 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.394 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.394 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:07.394 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.597765+0000", 2020-01-16T00:14:07.394 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016807971000000001 2020-01-16T00:14:07.394 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.394 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:07.394 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.394 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.395 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.395 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3220 3.3 3:c478bfe8:::smithi12313934-10:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:07.395 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.580982+0000", 2020-01-16T00:14:07.395 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 37.597038799000003, 2020-01-16T00:14:07.395 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017920330000000002, 2020-01-16T00:14:07.395 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:07.395 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:07.395 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:07.396 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:07.396 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:07.396 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3220 2020-01-16T00:14:07.396 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.396 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:07.396 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.396 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:07.397 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580982+0000", 2020-01-16T00:14:07.397 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:07.397 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.397 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.397 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:07.397 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580982+0000", 2020-01-16T00:14:07.397 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:07.398 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.398 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.398 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:07.398 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580981+0000", 2020-01-16T00:14:07.398 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0380000000000001e-06 2020-01-16T00:14:07.398 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.398 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.398 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:07.399 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580985+0000", 2020-01-16T00:14:07.399 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8929000000000001e-05 2020-01-16T00:14:07.399 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.399 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.399 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:07.399 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581004+0000", 2020-01-16T00:14:07.399 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7339999999999999e-06 2020-01-16T00:14:07.400 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.400 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.400 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:07.400 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581009+0000", 2020-01-16T00:14:07.400 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01679812 2020-01-16T00:14:07.400 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.401 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.401 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.401 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.597807+0000", 2020-01-16T00:14:07.401 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013333400000000001 2020-01-16T00:14:07.401 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.401 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.401 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:07.402 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.597940+0000", 2020-01-16T00:14:07.402 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000961899 2020-01-16T00:14:07.402 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.402 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.402 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:07.402 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.598902+0000", 2020-01-16T00:14:07.402 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017920330000000002 2020-01-16T00:14:07.403 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.403 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:07.403 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.403 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.403 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.404 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3221 3.3 3:c478bfe8:::smithi12313934-10:head [read 0~2403467,checksum] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:07.404 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.581014+0000", 2020-01-16T00:14:07.404 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 37.597006450000002, 2020-01-16T00:14:07.404 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018784551999999999, 2020-01-16T00:14:07.404 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:07.404 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:07.405 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:07.405 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:07.405 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:07.406 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3221 2020-01-16T00:14:07.406 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.406 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:07.406 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.406 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:07.406 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581014+0000", 2020-01-16T00:14:07.406 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:07.407 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.407 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.407 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:07.407 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581014+0000", 2020-01-16T00:14:07.407 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:07.407 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.407 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.408 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:07.408 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581013+0000", 2020-01-16T00:14:07.408 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9390000000000002e-06 2020-01-16T00:14:07.408 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.408 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.408 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:07.408 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581019+0000", 2020-01-16T00:14:07.409 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1131000000000001e-05 2020-01-16T00:14:07.409 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.409 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.409 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:07.409 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581030+0000", 2020-01-16T00:14:07.409 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.55e-06 2020-01-16T00:14:07.409 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.409 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.410 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:07.410 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581037+0000", 2020-01-16T00:14:07.410 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017890321000000001 2020-01-16T00:14:07.410 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.410 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.410 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.411 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.598927+0000", 2020-01-16T00:14:07.411 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000105999 2020-01-16T00:14:07.411 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.411 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.411 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:07.411 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.599033+0000", 2020-01-16T00:14:07.411 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00076542400000000001 2020-01-16T00:14:07.411 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.412 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.412 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:07.412 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.599798+0000", 2020-01-16T00:14:07.412 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018784551999999999 2020-01-16T00:14:07.412 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.412 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:07.412 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.413 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.413 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.413 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4152 3.9 3:93dfe6f1:::smithi12313934-40:head [setxattr _header (56) in=7b,truncate 2806868] snapc 0=[] ondisk+write+known_if_redirected e53)", 2020-01-16T00:14:07.413 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:53.810986+0000", 2020-01-16T00:14:07.413 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 13.367034141, 2020-01-16T00:14:07.413 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016830279, 2020-01-16T00:14:07.413 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:07.413 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:07.414 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:07.414 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:07.414 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:07.414 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4152 2020-01-16T00:14:07.414 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.414 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:07.414 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.415 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:07.415 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.810986+0000", 2020-01-16T00:14:07.415 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:07.415 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.415 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.415 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:07.415 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.810986+0000", 2020-01-16T00:14:07.415 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:07.416 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.416 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.416 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:07.416 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.810985+0000", 2020-01-16T00:14:07.416 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.7529999999999996e-06 2020-01-16T00:14:07.416 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.416 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.416 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:07.417 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.810995+0000", 2020-01-16T00:14:07.417 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7907999999999999e-05 2020-01-16T00:14:07.417 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.417 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.417 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:07.417 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811013+0000", 2020-01-16T00:14:07.417 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5829999999999997e-06 2020-01-16T00:14:07.418 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.418 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.418 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:07.418 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811018+0000", 2020-01-16T00:14:07.418 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0029165229999999999 2020-01-16T00:14:07.418 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.418 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.418 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.419 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.813935+0000", 2020-01-16T00:14:07.419 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2583000000000001e-05 2020-01-16T00:14:07.419 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.419 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.419 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:07.419 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.813988+0000", 2020-01-16T00:14:07.419 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010581012000000001 2020-01-16T00:14:07.420 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.420 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.420 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.420 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.824569+0000", 2020-01-16T00:14:07.420 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013191900000000001 2020-01-16T00:14:07.420 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.420 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.420 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:07.421 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.824700+0000", 2020-01-16T00:14:07.421 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022298000000000001 2020-01-16T00:14:07.421 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.421 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.421 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:14:07.421 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.824923+0000", 2020-01-16T00:14:07.421 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010694260000000001 2020-01-16T00:14:07.422 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.422 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.422 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:07.422 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.825993+0000", 2020-01-16T00:14:07.422 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000116939 2020-01-16T00:14:07.422 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.422 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.422 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:07.423 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.826110+0000", 2020-01-16T00:14:07.423 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0015606400000000001 2020-01-16T00:14:07.423 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.423 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.423 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:07.423 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.827670+0000", 2020-01-16T00:14:07.423 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7318999999999997e-05 2020-01-16T00:14:07.424 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.424 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.424 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:07.424 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.827738+0000", 2020-01-16T00:14:07.424 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.873e-05 2020-01-16T00:14:07.424 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.424 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.424 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:07.425 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.827817+0000", 2020-01-16T00:14:07.425 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016830279 2020-01-16T00:14:07.425 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.425 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:07.425 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.425 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.425 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.426 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4153 3.9 3:93dfe6f1:::smithi12313934-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e53)", 2020-01-16T00:14:07.426 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:53.811034+0000", 2020-01-16T00:14:07.426 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 13.366986053, 2020-01-16T00:14:07.426 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017095406, 2020-01-16T00:14:07.426 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:07.426 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:07.426 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:07.426 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:07.427 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:07.427 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4153 2020-01-16T00:14:07.427 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.427 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:07.427 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.428 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:07.428 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811034+0000", 2020-01-16T00:14:07.428 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:07.428 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.428 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.428 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:07.428 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811034+0000", 2020-01-16T00:14:07.428 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:07.429 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.429 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.429 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:07.429 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811034+0000", 2020-01-16T00:14:07.429 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9060000000000001e-06 2020-01-16T00:14:07.429 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.429 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.430 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:07.430 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811038+0000", 2020-01-16T00:14:07.430 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7710000000000001e-06 2020-01-16T00:14:07.430 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.430 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.430 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:07.430 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811045+0000", 2020-01-16T00:14:07.430 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5189999999999998e-06 2020-01-16T00:14:07.431 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.431 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.431 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:07.431 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811052+0000", 2020-01-16T00:14:07.431 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002953991 2020-01-16T00:14:07.431 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.431 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.431 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.432 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.814006+0000", 2020-01-16T00:14:07.432 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5726999999999999e-05 2020-01-16T00:14:07.432 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.432 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.432 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:07.432 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.814041+0000", 2020-01-16T00:14:07.432 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011650031999999999 2020-01-16T00:14:07.432 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.433 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.433 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.433 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.825691+0000", 2020-01-16T00:14:07.433 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0625999999999998e-05 2020-01-16T00:14:07.433 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.433 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.433 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:07.434 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.825762+0000", 2020-01-16T00:14:07.434 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0020982370000000002 2020-01-16T00:14:07.434 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.434 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.434 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.434 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.827860+0000", 2020-01-16T00:14:07.434 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012736299999999999 2020-01-16T00:14:07.434 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.435 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.435 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:07.435 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.827988+0000", 2020-01-16T00:14:07.435 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000142015 2020-01-16T00:14:07.435 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.435 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.435 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:07.435 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.828130+0000", 2020-01-16T00:14:07.436 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017095406 2020-01-16T00:14:07.436 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.436 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:07.436 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.436 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.436 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:07.436 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:07.442 INFO:tasks.rados.rados.0.smithi123.stdout:1677: finishing write tid 1 to smithi12313934-17 2020-01-16T00:14:07.443 INFO:tasks.rados.rados.0.smithi123.stdout:1677: finishing write tid 2 to smithi12313934-17 2020-01-16T00:14:07.443 INFO:tasks.rados.rados.0.smithi123.stdout:1677: finishing write tid 4 to smithi12313934-17 2020-01-16T00:14:07.443 INFO:tasks.rados.rados.0.smithi123.stdout:1677: finishing write tid 5 to smithi12313934-17 2020-01-16T00:14:07.443 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 17 v 190 (ObjNum 596 snap 0 seq_num 596) dirty exists 2020-01-16T00:14:07.443 INFO:tasks.rados.rados.0.smithi123.stdout:1677: left oid 17 (ObjNum 596 snap 0 seq_num 596) 2020-01-16T00:14:07.443 INFO:tasks.rados.rados.0.smithi123.stdout:1678: finishing write tid 1 to smithi12313934-35 2020-01-16T00:14:07.443 INFO:tasks.rados.rados.0.smithi123.stdout:1678: finishing write tid 2 to smithi12313934-35 2020-01-16T00:14:07.444 INFO:tasks.rados.rados.0.smithi123.stdout:1678: finishing write tid 4 to smithi12313934-35 2020-01-16T00:14:07.444 INFO:tasks.rados.rados.0.smithi123.stdout:1678: finishing write tid 5 to smithi12313934-35 2020-01-16T00:14:07.444 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 35 v 193 (ObjNum 597 snap 0 seq_num 597) dirty exists 2020-01-16T00:14:07.444 INFO:tasks.rados.rados.0.smithi123.stdout:1678: left oid 35 (ObjNum 597 snap 0 seq_num 597) 2020-01-16T00:14:07.444 INFO:tasks.rados.rados.0.smithi123.stdout:1679: finishing write tid 1 to smithi12313934-28 2020-01-16T00:14:07.444 INFO:tasks.rados.rados.0.smithi123.stdout:1679: finishing write tid 2 to smithi12313934-28 2020-01-16T00:14:07.444 INFO:tasks.rados.rados.0.smithi123.stdout:1679: finishing write tid 4 to smithi12313934-28 2020-01-16T00:14:07.445 INFO:tasks.rados.rados.0.smithi123.stdout:1679: finishing write tid 5 to smithi12313934-28 2020-01-16T00:14:07.445 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 28 v 214 (ObjNum 598 snap 0 seq_num 598) dirty exists 2020-01-16T00:14:07.445 INFO:tasks.rados.rados.0.smithi123.stdout:1679: left oid 28 (ObjNum 598 snap 0 seq_num 598) 2020-01-16T00:14:07.445 INFO:tasks.rados.rados.0.smithi123.stdout:1680: finishing write tid 1 to smithi12313934-48 2020-01-16T00:14:07.445 INFO:tasks.rados.rados.0.smithi123.stdout:1675: done (15 left) 2020-01-16T00:14:07.446 INFO:tasks.rados.rados.0.smithi123.stdout:1676: done (14 left) 2020-01-16T00:14:07.446 INFO:tasks.rados.rados.0.smithi123.stdout:1677: done (13 left) 2020-01-16T00:14:07.446 INFO:tasks.rados.rados.0.smithi123.stdout:1678: done (12 left) 2020-01-16T00:14:07.446 INFO:tasks.rados.rados.0.smithi123.stdout:1679: done (11 left) 2020-01-16T00:14:07.446 INFO:tasks.rados.rados.0.smithi123.stdout:1681: done (10 left) 2020-01-16T00:14:07.446 INFO:tasks.rados.rados.0.smithi123.stdout:1691: read oid 20 snap -1 2020-01-16T00:14:07.446 INFO:tasks.rados.rados.0.smithi123.stdout:1691: expect deleted 2020-01-16T00:14:07.446 INFO:tasks.rados.rados.0.smithi123.stdout:1692: write (excl) oid 40 current snap is 0 2020-01-16T00:14:07.447 INFO:tasks.rados.rados.0.smithi123.stdout:1692: seq_num 605 ranges {568235=717532,1753253=414982} 2020-01-16T00:14:07.447 INFO:tasks.rados.rados.0.smithi123.stdout:1692: writing smithi12313934-40 from 568235 to 1285767 tid 1 2020-01-16T00:14:07.447 INFO:tasks.rados.rados.0.smithi123.stdout:1692: writing smithi12313934-40 from 1753253 to 2168235 tid 2 2020-01-16T00:14:07.447 INFO:tasks.rados.rados.0.smithi123.stdout:1693: delete oid 29 current snap is 0 2020-01-16T00:14:07.447 INFO:tasks.rados.rados.0.smithi123.stdout:1680: finishing write tid 2 to smithi12313934-48 2020-01-16T00:14:07.447 INFO:tasks.rados.rados.0.smithi123.stdout:1682: finishing write tid 1 to smithi12313934-34 2020-01-16T00:14:07.447 INFO:tasks.rados.rados.0.smithi123.stdout:1682: finishing write tid 2 to smithi12313934-34 2020-01-16T00:14:07.448 INFO:tasks.rados.rados.0.smithi123.stdout:1682: finishing write tid 4 to smithi12313934-34 2020-01-16T00:14:07.448 INFO:tasks.rados.rados.0.smithi123.stdout:1682: finishing write tid 5 to smithi12313934-34 2020-01-16T00:14:07.448 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 34 v 83 (ObjNum 600 snap 0 seq_num 600) dirty exists 2020-01-16T00:14:07.448 INFO:tasks.rados.rados.0.smithi123.stdout:1682: left oid 34 (ObjNum 600 snap 0 seq_num 600) 2020-01-16T00:14:07.448 INFO:tasks.rados.rados.0.smithi123.stdout:1683: expect (ObjNum 593 snap 0 seq_num 593) 2020-01-16T00:14:07.448 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:07.436+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:07.449 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-01-16T00:14:07.538 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:07.536+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:07.599 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:07.600 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:14:07.600 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:14:07.600 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:07.609 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-01-16T00:14:07.634 INFO:tasks.rados.rados.0.smithi123.stdout:1680: finishing write tid 4 to smithi12313934-48 2020-01-16T00:14:07.634 INFO:tasks.rados.rados.0.smithi123.stdout:1680: finishing write tid 5 to smithi12313934-48 2020-01-16T00:14:07.635 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 48 v 200 (ObjNum 599 snap 0 seq_num 599) dirty exists 2020-01-16T00:14:07.635 INFO:tasks.rados.rados.0.smithi123.stdout:1680: left oid 48 (ObjNum 599 snap 0 seq_num 599) 2020-01-16T00:14:07.635 INFO:tasks.rados.rados.0.smithi123.stdout:1684: finishing write tid 1 to smithi12313934-42 2020-01-16T00:14:07.635 INFO:tasks.rados.rados.0.smithi123.stdout:1686: expect (ObjNum 595 snap 0 seq_num 595) 2020-01-16T00:14:07.640 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:07.638+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:07.741 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:07.738+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:07.755 INFO:tasks.rados.rados.0.smithi123.stdout:1685: expect (ObjNum 487 snap 0 seq_num 487) 2020-01-16T00:14:07.765 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:07.765 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:14:07.766 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:14:07.766 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:14:07.766 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:07.775 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-01-16T00:14:07.842 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:07.838+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:07.870 INFO:tasks.rados.rados.0.smithi123.stdout:1687: finishing write tid 1 to smithi12313934-41 2020-01-16T00:14:07.871 INFO:tasks.rados.rados.0.smithi123.stdout:1687: finishing write tid 2 to smithi12313934-41 2020-01-16T00:14:07.871 INFO:tasks.rados.rados.0.smithi123.stdout:1688: expect (ObjNum 565 snap 0 seq_num 565) 2020-01-16T00:14:07.882 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:07.882 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:14:07.882 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:14:07.882 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:14:07.883 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.883 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.5.13:36 2.0 2:20e40cec:::smithi12313934-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 113152~1699840] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:14:07.883 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:14:07.883 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 90.048528617000002, 2020-01-16T00:14:07.883 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011736636, 2020-01-16T00:14:07.883 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:07.883 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:07.883 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:07.884 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.884 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:07.884 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:14:07.884 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:07.884 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.884 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.884 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:07.885 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:14:07.885 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999952 2020-01-16T00:14:07.885 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.885 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.885 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:07.885 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831934+0000", 2020-01-16T00:14:07.885 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001034325 2020-01-16T00:14:07.885 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.886 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.886 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:07.886 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.832968+0000", 2020-01-16T00:14:07.886 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.506e-05 2020-01-16T00:14:07.886 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.886 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.886 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:07.886 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833004+0000", 2020-01-16T00:14:07.887 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3406e-05 2020-01-16T00:14:07.887 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.887 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.888 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:07.889 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833017+0000", 2020-01-16T00:14:07.889 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015477000000000001 2020-01-16T00:14:07.889 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.889 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.889 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.889 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833172+0000", 2020-01-16T00:14:07.889 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033750700000000002 2020-01-16T00:14:07.890 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.890 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.890 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:07.890 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833509+0000", 2020-01-16T00:14:07.890 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00055861400000000003 2020-01-16T00:14:07.890 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.890 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.890 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:14:07.891 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.834068+0000", 2020-01-16T00:14:07.891 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.007474545 2020-01-16T00:14:07.891 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.891 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.891 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:07.891 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.841542+0000", 2020-01-16T00:14:07.891 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00056319499999999995 2020-01-16T00:14:07.891 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.892 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.892 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:07.892 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.842106+0000", 2020-01-16T00:14:07.892 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001371017 2020-01-16T00:14:07.892 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.892 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.892 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:07.892 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843477+0000", 2020-01-16T00:14:07.893 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000123938 2020-01-16T00:14:07.893 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.893 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.893 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:07.893 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843601+0000", 2020-01-16T00:14:07.893 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5012000000000006e-05 2020-01-16T00:14:07.893 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.894 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.894 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:07.894 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843676+0000", 2020-01-16T00:14:07.894 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011736636 2020-01-16T00:14:07.894 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.894 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:07.894 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.894 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.895 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.895 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1187 3.d 3:b1521a3e:::smithi12313934-25:head [write 1721042~423699] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:14:07.895 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.263828+0000", 2020-01-16T00:14:07.895 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 85.616639544999998, 2020-01-16T00:14:07.895 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011305709000000001, 2020-01-16T00:14:07.895 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:07.895 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:07.896 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:07.896 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:07.896 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:07.896 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1187 2020-01-16T00:14:07.896 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.896 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:07.897 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.897 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:07.897 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.263828+0000", 2020-01-16T00:14:07.897 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:07.897 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.897 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.897 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:07.897 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.263828+0000", 2020-01-16T00:14:07.898 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:14:07.898 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.898 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.898 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:07.898 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.263824+0000", 2020-01-16T00:14:07.898 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00048485899999999997 2020-01-16T00:14:07.898 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.899 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.899 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:07.899 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264309+0000", 2020-01-16T00:14:07.899 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5622999999999999e-05 2020-01-16T00:14:07.899 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.899 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.899 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:07.899 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264325+0000", 2020-01-16T00:14:07.900 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2530000000000004e-06 2020-01-16T00:14:07.900 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.900 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.900 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:07.900 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264330+0000", 2020-01-16T00:14:07.900 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000102252 2020-01-16T00:14:07.900 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.900 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.901 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.901 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264432+0000", 2020-01-16T00:14:07.901 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022361000000000001 2020-01-16T00:14:07.901 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.901 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.901 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:07.901 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.264656+0000", 2020-01-16T00:14:07.902 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0028688849999999998 2020-01-16T00:14:07.902 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.902 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.902 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.902 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.267525+0000", 2020-01-16T00:14:07.902 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00023640900000000001 2020-01-16T00:14:07.902 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.903 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.903 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:07.903 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.267761+0000", 2020-01-16T00:14:07.903 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00035667000000000001 2020-01-16T00:14:07.903 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.903 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.903 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:14:07.903 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.268118+0000", 2020-01-16T00:14:07.904 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023898040000000001 2020-01-16T00:14:07.904 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.904 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.904 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:07.904 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.270507+0000", 2020-01-16T00:14:07.904 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014919690000000001 2020-01-16T00:14:07.904 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.904 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.905 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:07.905 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.271999+0000", 2020-01-16T00:14:07.905 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0029764510000000002 2020-01-16T00:14:07.905 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.905 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.905 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:07.905 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.274976+0000", 2020-01-16T00:14:07.906 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5409999999999997e-05 2020-01-16T00:14:07.906 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.906 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.906 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:07.906 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.275061+0000", 2020-01-16T00:14:07.906 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2452000000000004e-05 2020-01-16T00:14:07.906 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.906 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.907 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:07.907 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.275134+0000", 2020-01-16T00:14:07.907 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011305709000000001 2020-01-16T00:14:07.907 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.907 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:07.907 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.907 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.907 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.908 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1276 3.d 3:b1521a3e:::smithi12313934-25:head [read 0~3882384,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:14:07.908 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:14:07.908 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 84.805769046999998, 2020-01-16T00:14:07.908 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015839157999999999, 2020-01-16T00:14:07.908 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:07.908 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:07.908 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:07.909 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:07.909 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:07.909 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1276 2020-01-16T00:14:07.909 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.909 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:07.909 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.909 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:07.910 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:14:07.910 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:07.910 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.910 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.910 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:07.911 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:14:07.911 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:07.911 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.911 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.911 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:07.911 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:14:07.911 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5429999999999999e-06 2020-01-16T00:14:07.911 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.912 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.912 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:07.912 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074704+0000", 2020-01-16T00:14:07.912 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0839999999999992e-06 2020-01-16T00:14:07.912 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.912 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.912 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:07.912 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074712+0000", 2020-01-16T00:14:07.913 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.173e-06 2020-01-16T00:14:07.913 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.913 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.913 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:07.913 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074718+0000", 2020-01-16T00:14:07.913 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0083034700000000003 2020-01-16T00:14:07.913 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.914 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.914 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.914 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.083022+0000", 2020-01-16T00:14:07.914 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028456400000000001 2020-01-16T00:14:07.914 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.914 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.914 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:07.914 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.083306+0000", 2020-01-16T00:14:07.915 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0072312640000000003 2020-01-16T00:14:07.915 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.915 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.915 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:07.915 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090538+0000", 2020-01-16T00:14:07.915 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015839157999999999 2020-01-16T00:14:07.915 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.915 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:07.916 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.916 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.916 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.916 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1277 3.d 3:b1521a3e:::smithi12313934-25:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:14:07.916 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:14:07.916 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 84.805725456999994, 2020-01-16T00:14:07.916 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016646459999999998, 2020-01-16T00:14:07.917 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:07.917 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:07.917 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:07.917 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:07.917 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:07.917 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1277 2020-01-16T00:14:07.917 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.917 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:07.918 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.918 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:07.918 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:14:07.918 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:07.918 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.918 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.918 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:07.918 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:14:07.919 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:07.919 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.919 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.919 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:07.919 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074741+0000", 2020-01-16T00:14:07.919 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0776e-05 2020-01-16T00:14:07.919 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.920 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.920 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:07.920 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074752+0000", 2020-01-16T00:14:07.920 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7600000000000002e-06 2020-01-16T00:14:07.920 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.920 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.920 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:07.920 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074759+0000", 2020-01-16T00:14:07.921 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6840000000000001e-06 2020-01-16T00:14:07.921 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.921 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.921 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:07.921 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074765+0000", 2020-01-16T00:14:07.921 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015815216 2020-01-16T00:14:07.921 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.921 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.922 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.922 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090580+0000", 2020-01-16T00:14:07.922 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000182404 2020-01-16T00:14:07.922 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.922 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.922 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:07.922 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090763+0000", 2020-01-16T00:14:07.922 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00062579300000000003 2020-01-16T00:14:07.923 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.923 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.923 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:07.923 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091389+0000", 2020-01-16T00:14:07.923 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016646459999999998 2020-01-16T00:14:07.923 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.923 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:07.924 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.924 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.924 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.924 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1278 3.d 3:b1521a3e:::smithi12313934-25:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:14:07.924 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:14:07.924 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 84.805695744000005, 2020-01-16T00:14:07.924 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017994915, 2020-01-16T00:14:07.924 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:07.925 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:07.925 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:07.925 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:07.925 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:07.925 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1278 2020-01-16T00:14:07.925 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.925 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:07.926 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.926 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:07.926 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:14:07.926 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:07.926 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.926 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.926 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:07.926 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:14:07.927 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:07.927 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.927 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.927 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:07.927 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074771+0000", 2020-01-16T00:14:07.927 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4569999999999998e-06 2020-01-16T00:14:07.927 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.927 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.928 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:07.928 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074775+0000", 2020-01-16T00:14:07.928 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0330000000000003e-06 2020-01-16T00:14:07.928 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.928 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.928 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:07.928 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074782+0000", 2020-01-16T00:14:07.929 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9470000000000002e-06 2020-01-16T00:14:07.929 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.929 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.929 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:07.929 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074787+0000", 2020-01-16T00:14:07.929 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01666134 2020-01-16T00:14:07.929 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.929 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.930 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.930 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091449+0000", 2020-01-16T00:14:07.930 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000247731 2020-01-16T00:14:07.930 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.930 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.930 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:07.930 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091696+0000", 2020-01-16T00:14:07.931 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010703170000000001 2020-01-16T00:14:07.931 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.931 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.931 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:07.931 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.092767+0000", 2020-01-16T00:14:07.931 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017994915 2020-01-16T00:14:07.931 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.931 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:07.932 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.932 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.932 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.932 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2260 3.d 3:b76659c4:::smithi12313934-47:head [write 601750~535546] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:14:07.932 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:06.050577+0000", 2020-01-16T00:14:07.932 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 61.829890427999999, 2020-01-16T00:14:07.932 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012415381, 2020-01-16T00:14:07.932 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:07.933 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:07.933 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:07.933 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:07.933 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:07.933 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2260 2020-01-16T00:14:07.933 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.933 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:07.934 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.934 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:07.934 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.050577+0000", 2020-01-16T00:14:07.934 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:07.934 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.934 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.934 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:07.934 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.050577+0000", 2020-01-16T00:14:07.935 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:07.935 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.935 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.935 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:07.935 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.050574+0000", 2020-01-16T00:14:07.935 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001523487 2020-01-16T00:14:07.935 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.936 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.936 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:07.936 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052097+0000", 2020-01-16T00:14:07.936 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3697e-05 2020-01-16T00:14:07.937 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.937 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.937 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:07.937 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052121+0000", 2020-01-16T00:14:07.937 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0070000000000003e-06 2020-01-16T00:14:07.937 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.938 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.939 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:07.939 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052130+0000", 2020-01-16T00:14:07.939 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4559000000000003e-05 2020-01-16T00:14:07.939 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.939 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.939 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.940 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052194+0000", 2020-01-16T00:14:07.940 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000165878 2020-01-16T00:14:07.940 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.940 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.940 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:07.940 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052360+0000", 2020-01-16T00:14:07.940 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018825499999999999 2020-01-16T00:14:07.941 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.941 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.941 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:14:07.941 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052549+0000", 2020-01-16T00:14:07.941 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002761782 2020-01-16T00:14:07.941 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.941 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.942 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:07.942 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.055310+0000", 2020-01-16T00:14:07.942 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00060385500000000004 2020-01-16T00:14:07.942 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.942 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.942 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:07.942 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.055914+0000", 2020-01-16T00:14:07.942 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0069733809999999999 2020-01-16T00:14:07.943 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.943 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.943 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:07.943 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.062888+0000", 2020-01-16T00:14:07.943 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2884999999999999e-05 2020-01-16T00:14:07.943 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.943 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.943 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:07.944 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.062940+0000", 2020-01-16T00:14:07.944 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1996999999999997e-05 2020-01-16T00:14:07.944 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.944 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.944 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:07.944 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.062992+0000", 2020-01-16T00:14:07.944 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012415381 2020-01-16T00:14:07.945 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.945 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:07.945 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.945 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.945 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.945 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2295 3.d 3:b1521a3e:::smithi12313934-25:head [write 1763838~505140] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:14:07.945 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:06.660258+0000", 2020-01-16T00:14:07.945 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 61.220209201000003, 2020-01-16T00:14:07.946 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011681408000000001, 2020-01-16T00:14:07.946 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:07.946 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:07.946 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:07.946 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:07.946 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:07.946 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2295 2020-01-16T00:14:07.946 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.947 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:07.947 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.947 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:07.947 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660258+0000", 2020-01-16T00:14:07.947 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:07.947 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.947 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.948 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:07.948 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660258+0000", 2020-01-16T00:14:07.948 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:07.948 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.948 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.948 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:07.948 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660256+0000", 2020-01-16T00:14:07.948 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00066501699999999995 2020-01-16T00:14:07.949 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.949 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.949 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:07.949 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660921+0000", 2020-01-16T00:14:07.949 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7339000000000001e-05 2020-01-16T00:14:07.949 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.949 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.950 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:07.950 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660938+0000", 2020-01-16T00:14:07.950 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0219999999999993e-06 2020-01-16T00:14:07.950 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.950 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.950 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:07.950 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660947+0000", 2020-01-16T00:14:07.950 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00094237200000000002 2020-01-16T00:14:07.951 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.951 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.951 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.951 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.661890+0000", 2020-01-16T00:14:07.951 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000117649 2020-01-16T00:14:07.951 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.951 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.951 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:07.952 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662007+0000", 2020-01-16T00:14:07.952 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00060662599999999998 2020-01-16T00:14:07.952 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.952 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.952 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.952 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662614+0000", 2020-01-16T00:14:07.952 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028424899999999998 2020-01-16T00:14:07.952 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.953 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.953 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:07.953 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662898+0000", 2020-01-16T00:14:07.953 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00054904599999999997 2020-01-16T00:14:07.953 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.953 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.953 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:14:07.954 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.663447+0000", 2020-01-16T00:14:07.954 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0051479990000000003 2020-01-16T00:14:07.954 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.954 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.954 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:07.954 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.668595+0000", 2020-01-16T00:14:07.954 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00080881799999999999 2020-01-16T00:14:07.954 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.955 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.955 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:07.955 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.669404+0000", 2020-01-16T00:14:07.955 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0024495290000000002 2020-01-16T00:14:07.955 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.955 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.955 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:07.955 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671854+0000", 2020-01-16T00:14:07.956 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6635000000000003e-05 2020-01-16T00:14:07.956 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.956 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.956 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:07.956 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671900+0000", 2020-01-16T00:14:07.956 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9567999999999998e-05 2020-01-16T00:14:07.956 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.956 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.957 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:07.957 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671940+0000", 2020-01-16T00:14:07.957 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011681408000000001 2020-01-16T00:14:07.957 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.957 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:07.957 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.957 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.958 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.958 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.3.13:109 2.0 2:11fa79bc:::smithi12313934-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 331264~198656] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e42)", 2020-01-16T00:14:07.958 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:21.840075+0000", 2020-01-16T00:14:07.958 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 46.040392904000001, 2020-01-16T00:14:07.958 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011558677999999999, 2020-01-16T00:14:07.958 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:07.958 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:07.958 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:07.959 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.959 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:07.959 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840075+0000", 2020-01-16T00:14:07.959 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:07.959 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.959 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.959 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:07.960 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840075+0000", 2020-01-16T00:14:07.960 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:07.960 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.960 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.960 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:07.960 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840073+0000", 2020-01-16T00:14:07.960 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033576299999999999 2020-01-16T00:14:07.960 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.961 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.961 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:07.961 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840409+0000", 2020-01-16T00:14:07.961 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2082999999999998e-05 2020-01-16T00:14:07.961 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.961 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.961 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:07.962 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840431+0000", 2020-01-16T00:14:07.962 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4410000000000003e-06 2020-01-16T00:14:07.962 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.962 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.962 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:07.962 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840436+0000", 2020-01-16T00:14:07.962 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000101016 2020-01-16T00:14:07.962 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.963 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.963 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.963 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840537+0000", 2020-01-16T00:14:07.963 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000117639 2020-01-16T00:14:07.963 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.963 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.963 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:07.964 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840654+0000", 2020-01-16T00:14:07.964 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000196162 2020-01-16T00:14:07.964 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.964 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.964 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:14:07.964 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840850+0000", 2020-01-16T00:14:07.964 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003319092 2020-01-16T00:14:07.964 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.965 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.965 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:07.965 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.844169+0000", 2020-01-16T00:14:07.965 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011349229999999999 2020-01-16T00:14:07.965 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.965 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.965 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:07.966 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.845304+0000", 2020-01-16T00:14:07.966 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0062517670000000001 2020-01-16T00:14:07.966 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.966 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.966 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:07.966 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851556+0000", 2020-01-16T00:14:07.966 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4138000000000003e-05 2020-01-16T00:14:07.966 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.967 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.967 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:07.967 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851600+0000", 2020-01-16T00:14:07.967 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2993000000000001e-05 2020-01-16T00:14:07.967 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.967 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.967 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:07.968 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851633+0000", 2020-01-16T00:14:07.968 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011558677999999999 2020-01-16T00:14:07.968 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.968 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:07.968 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.968 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.968 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.968 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.3.13:110 2.0 2:11fa79bc:::smithi12313934-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 752640~650240] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e42)", 2020-01-16T00:14:07.969 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:21.840457+0000", 2020-01-16T00:14:07.969 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 46.040010612000003, 2020-01-16T00:14:07.969 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011329271, 2020-01-16T00:14:07.969 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:07.969 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:07.969 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:07.969 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.970 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:07.970 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840457+0000", 2020-01-16T00:14:07.970 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:07.970 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.970 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.970 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:07.970 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840457+0000", 2020-01-16T00:14:07.970 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:07.971 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.971 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.971 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:07.971 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840456+0000", 2020-01-16T00:14:07.971 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00062607899999999998 2020-01-16T00:14:07.971 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.971 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.972 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:07.972 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841082+0000", 2020-01-16T00:14:07.972 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1239999999999999e-06 2020-01-16T00:14:07.972 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.972 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.972 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:07.972 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841090+0000", 2020-01-16T00:14:07.972 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7129999999999999e-06 2020-01-16T00:14:07.973 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.973 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.973 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:07.973 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841093+0000", 2020-01-16T00:14:07.973 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00075997800000000002 2020-01-16T00:14:07.973 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.973 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.973 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.974 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841853+0000", 2020-01-16T00:14:07.974 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0690000000000003e-05 2020-01-16T00:14:07.974 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.974 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.974 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:07.974 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841924+0000", 2020-01-16T00:14:07.974 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000115753 2020-01-16T00:14:07.974 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.975 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.975 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:14:07.975 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.842040+0000", 2020-01-16T00:14:07.975 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0021358890000000002 2020-01-16T00:14:07.975 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.975 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.975 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:07.976 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.844176+0000", 2020-01-16T00:14:07.976 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0042541870000000004 2020-01-16T00:14:07.976 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.976 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.976 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:07.976 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.848430+0000", 2020-01-16T00:14:07.976 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0032476929999999998 2020-01-16T00:14:07.976 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.977 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.977 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:07.977 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851677+0000", 2020-01-16T00:14:07.977 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0370000000000001e-05 2020-01-16T00:14:07.977 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.977 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.977 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:07.977 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851728+0000", 2020-01-16T00:14:07.978 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.834e-05 2020-01-16T00:14:07.978 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.978 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.978 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:07.978 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851786+0000", 2020-01-16T00:14:07.978 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011329271 2020-01-16T00:14:07.978 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.979 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:07.979 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.979 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.979 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.979 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2987 3.d 3:b3dd5d45:::smithi12313934-16:head [write 467327~633480] snapc 0=[] ondisk+write+known_if_redirected e42)", 2020-01-16T00:14:07.979 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:22.257904+0000", 2020-01-16T00:14:07.979 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 45.622563317999997, 2020-01-16T00:14:07.979 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012187450000000001, 2020-01-16T00:14:07.980 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:07.980 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:07.980 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:07.980 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:07.980 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:07.980 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2987 2020-01-16T00:14:07.980 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.980 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:07.981 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.981 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:07.981 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.257904+0000", 2020-01-16T00:14:07.981 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:07.981 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.981 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.981 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:07.982 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.257904+0000", 2020-01-16T00:14:07.982 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999933 2020-01-16T00:14:07.982 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.982 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.982 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:07.982 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.257898+0000", 2020-01-16T00:14:07.982 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00097099800000000004 2020-01-16T00:14:07.982 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.983 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.983 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:07.983 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.258869+0000", 2020-01-16T00:14:07.983 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7970000000000003e-05 2020-01-16T00:14:07.983 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.983 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.983 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:07.983 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.258917+0000", 2020-01-16T00:14:07.984 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5982999999999999e-05 2020-01-16T00:14:07.984 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.984 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.984 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:07.984 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.258933+0000", 2020-01-16T00:14:07.984 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00026239999999999998 2020-01-16T00:14:07.984 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.985 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.985 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.985 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.259195+0000", 2020-01-16T00:14:07.985 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024907400000000001 2020-01-16T00:14:07.985 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.985 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.985 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:07.985 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.259444+0000", 2020-01-16T00:14:07.986 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00034824999999999998 2020-01-16T00:14:07.986 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.986 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.986 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:07.986 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.259792+0000", 2020-01-16T00:14:07.986 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003251538 2020-01-16T00:14:07.986 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.987 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.987 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:07.987 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.263044+0000", 2020-01-16T00:14:07.987 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0068502880000000004 2020-01-16T00:14:07.987 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.987 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.987 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:07.987 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.269894+0000", 2020-01-16T00:14:07.988 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0945000000000006e-05 2020-01-16T00:14:07.988 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.988 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.988 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:07.988 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.269965+0000", 2020-01-16T00:14:07.988 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012667399999999999 2020-01-16T00:14:07.988 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.989 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.989 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:07.989 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.270092+0000", 2020-01-16T00:14:07.989 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012187450000000001 2020-01-16T00:14:07.989 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.989 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:07.989 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.989 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.990 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.990 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3273 3.d 3:b76659c4:::smithi12313934-47:head [read 0~3647152,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:07.990 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.728775+0000", 2020-01-16T00:14:07.990 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 37.151692764000003, 2020-01-16T00:14:07.990 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012922950000000001, 2020-01-16T00:14:07.990 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:07.990 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:07.991 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:07.991 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:07.991 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:07.991 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3273 2020-01-16T00:14:07.991 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.991 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:07.991 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.991 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:07.992 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728775+0000", 2020-01-16T00:14:07.992 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:07.992 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.992 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.992 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:07.992 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728775+0000", 2020-01-16T00:14:07.992 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:07.993 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.993 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.993 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:07.993 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728771+0000", 2020-01-16T00:14:07.993 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.5009000000000001e-05 2020-01-16T00:14:07.993 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.993 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.993 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:07.994 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728796+0000", 2020-01-16T00:14:07.994 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2357999999999998e-05 2020-01-16T00:14:07.994 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.994 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.994 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:07.994 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728819+0000", 2020-01-16T00:14:07.994 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.4800000000000007e-06 2020-01-16T00:14:07.995 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.995 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.995 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:07.995 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728828+0000", 2020-01-16T00:14:07.995 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000124824 2020-01-16T00:14:07.995 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.995 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.995 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:07.996 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728953+0000", 2020-01-16T00:14:07.996 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00044904899999999998 2020-01-16T00:14:07.996 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.996 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.996 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:07.996 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729402+0000", 2020-01-16T00:14:07.996 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012295706999999999 2020-01-16T00:14:07.997 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.997 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.997 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:07.997 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.741698+0000", 2020-01-16T00:14:07.997 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012922950000000001 2020-01-16T00:14:07.997 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.997 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:07.997 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:07.998 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.998 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.998 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3275 3.d 3:b76659c4:::smithi12313934-47:head [read 0~3647152,checksum] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:07.998 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.728933+0000", 2020-01-16T00:14:07.998 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 37.151534492000003, 2020-01-16T00:14:07.998 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01204874, 2020-01-16T00:14:07.998 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:07.998 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:07.999 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:07.999 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:07.999 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:07.999 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3275 2020-01-16T00:14:07.999 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:07.999 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:07.999 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:07.999 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:08.000 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728933+0000", 2020-01-16T00:14:08.000 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:08.000 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.000 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.000 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:08.000 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728933+0000", 2020-01-16T00:14:08.000 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:08.001 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.001 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.001 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:08.001 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728932+0000", 2020-01-16T00:14:08.001 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3569999999999998e-06 2020-01-16T00:14:08.001 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.001 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.001 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:08.002 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728940+0000", 2020-01-16T00:14:08.002 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0457e-05 2020-01-16T00:14:08.002 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.002 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.002 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:08.002 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728951+0000", 2020-01-16T00:14:08.002 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6799999999999993e-06 2020-01-16T00:14:08.003 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.003 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.003 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:08.003 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728959+0000", 2020-01-16T00:14:08.003 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010076307 2020-01-16T00:14:08.003 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.003 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.003 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.004 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.739035+0000", 2020-01-16T00:14:08.004 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024455400000000002 2020-01-16T00:14:08.004 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.004 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.004 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:08.004 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.739279+0000", 2020-01-16T00:14:08.004 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0017023660000000001 2020-01-16T00:14:08.005 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.005 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.005 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:08.005 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.740982+0000", 2020-01-16T00:14:08.005 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01204874 2020-01-16T00:14:08.005 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.005 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:08.005 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.006 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.006 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.006 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3285 3.d 3:bd9fc72f:::smithi12313934-5:head [read 0~3764801,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:08.006 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.729068+0000", 2020-01-16T00:14:08.006 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 37.151399699999999, 2020-01-16T00:14:08.006 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018537010999999999, 2020-01-16T00:14:08.006 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:08.006 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:08.007 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:08.007 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:08.007 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:08.007 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3285 2020-01-16T00:14:08.007 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.007 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:08.007 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.007 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:08.008 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729068+0000", 2020-01-16T00:14:08.008 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:08.008 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.008 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.008 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:08.008 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729068+0000", 2020-01-16T00:14:08.008 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:08.009 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.009 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.009 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:08.009 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729067+0000", 2020-01-16T00:14:08.009 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3339999999999996e-06 2020-01-16T00:14:08.009 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.009 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.010 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:08.010 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729075+0000", 2020-01-16T00:14:08.010 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4187e-05 2020-01-16T00:14:08.010 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.010 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.010 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:08.010 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729089+0000", 2020-01-16T00:14:08.010 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5600000000000004e-06 2020-01-16T00:14:08.011 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.011 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.011 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:08.011 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729093+0000", 2020-01-16T00:14:08.011 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011944309 2020-01-16T00:14:08.011 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.011 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.011 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.012 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.741038+0000", 2020-01-16T00:14:08.012 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016625799999999999 2020-01-16T00:14:08.012 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.012 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.012 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:08.012 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.741204+0000", 2020-01-16T00:14:08.012 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0064010120000000002 2020-01-16T00:14:08.013 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.013 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.013 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:08.013 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.747605+0000", 2020-01-16T00:14:08.013 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018537010999999999 2020-01-16T00:14:08.013 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.013 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:08.013 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.014 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.014 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.014 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3286 3.d 3:bd9fc72f:::smithi12313934-5:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:08.014 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.729099+0000", 2020-01-16T00:14:08.014 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 37.151369029999998, 2020-01-16T00:14:08.014 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018983897999999999, 2020-01-16T00:14:08.014 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:08.014 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:08.015 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:08.015 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:08.015 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:08.015 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3286 2020-01-16T00:14:08.015 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.015 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:08.015 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.016 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:08.016 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729099+0000", 2020-01-16T00:14:08.016 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:08.016 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.016 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.016 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:08.016 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729099+0000", 2020-01-16T00:14:08.016 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:08.017 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.017 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.017 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:08.017 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729098+0000", 2020-01-16T00:14:08.017 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8500000000000004e-06 2020-01-16T00:14:08.017 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.017 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.017 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:08.018 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729102+0000", 2020-01-16T00:14:08.018 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7329999999999997e-06 2020-01-16T00:14:08.018 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.018 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.018 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:08.018 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729108+0000", 2020-01-16T00:14:08.018 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.764e-06 2020-01-16T00:14:08.019 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.019 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.019 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:08.019 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729118+0000", 2020-01-16T00:14:08.019 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018524286000000001 2020-01-16T00:14:08.019 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.019 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.019 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.020 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.747642+0000", 2020-01-16T00:14:08.020 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9487999999999993e-05 2020-01-16T00:14:08.020 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.020 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.020 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:08.021 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.747742+0000", 2020-01-16T00:14:08.021 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000340429 2020-01-16T00:14:08.021 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.021 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.021 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:08.021 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.748082+0000", 2020-01-16T00:14:08.021 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018983897999999999 2020-01-16T00:14:08.021 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.022 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:08.022 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.022 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.022 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.022 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3287 3.d 3:bd9fc72f:::smithi12313934-5:head [read 0~3764801,checksum] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:08.022 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.729124+0000", 2020-01-16T00:14:08.022 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 37.151343891000003, 2020-01-16T00:14:08.022 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020552323000000001, 2020-01-16T00:14:08.023 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:08.023 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:08.023 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:08.023 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:08.023 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:08.023 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3287 2020-01-16T00:14:08.023 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.024 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:08.024 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.024 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:08.024 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729124+0000", 2020-01-16T00:14:08.024 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:08.024 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.024 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.024 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:08.025 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729124+0000", 2020-01-16T00:14:08.025 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:08.025 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.025 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.025 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:08.025 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729123+0000", 2020-01-16T00:14:08.025 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8239999999999996e-06 2020-01-16T00:14:08.025 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.026 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.026 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:08.026 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729129+0000", 2020-01-16T00:14:08.026 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6909999999999997e-06 2020-01-16T00:14:08.026 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.026 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.026 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:08.026 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729136+0000", 2020-01-16T00:14:08.027 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3329999999999997e-06 2020-01-16T00:14:08.027 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.027 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.027 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:08.027 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729145+0000", 2020-01-16T00:14:08.027 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018965065999999999 2020-01-16T00:14:08.027 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.027 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.028 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.028 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.748110+0000", 2020-01-16T00:14:08.028 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015781700000000001 2020-01-16T00:14:08.028 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.028 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.028 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:08.028 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.748268+0000", 2020-01-16T00:14:08.029 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014083800000000001 2020-01-16T00:14:08.029 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.029 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.029 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:08.029 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.749676+0000", 2020-01-16T00:14:08.029 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020552323000000001 2020-01-16T00:14:08.029 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.029 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:08.030 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.030 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.030 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.030 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3409 3.d 3:b3dd5d45:::smithi12313934-16:head [write 591712~786646,stat] snapc 0=[] ondisk+write+known_if_redirected e47)", 2020-01-16T00:14:08.030 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:34.100681+0000", 2020-01-16T00:14:08.030 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 33.779786751000003, 2020-01-16T00:14:08.031 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012248446, 2020-01-16T00:14:08.031 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:08.031 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:08.031 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:08.031 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:08.031 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:08.031 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3409 2020-01-16T00:14:08.032 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.032 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:08.032 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.032 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:08.032 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.100681+0000", 2020-01-16T00:14:08.032 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:08.032 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.032 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.033 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:08.033 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.100681+0000", 2020-01-16T00:14:08.033 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:14:08.033 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.033 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.033 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:08.034 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.100677+0000", 2020-01-16T00:14:08.034 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000998828 2020-01-16T00:14:08.034 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.034 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.034 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:08.034 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101676+0000", 2020-01-16T00:14:08.034 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5846999999999997e-05 2020-01-16T00:14:08.034 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.035 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.035 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:08.035 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101712+0000", 2020-01-16T00:14:08.035 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.364e-05 2020-01-16T00:14:08.035 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.035 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.035 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:08.035 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101725+0000", 2020-01-16T00:14:08.036 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015902700000000001 2020-01-16T00:14:08.036 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.036 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.036 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.036 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101884+0000", 2020-01-16T00:14:08.036 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0486e-05 2020-01-16T00:14:08.036 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.037 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.037 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op must wait for map", 2020-01-16T00:14:08.037 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101905+0000", 2020-01-16T00:14:08.037 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0038869009999999999 2020-01-16T00:14:08.037 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.037 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.037 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.037 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.105792+0000", 2020-01-16T00:14:08.038 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030748800000000002 2020-01-16T00:14:08.038 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.038 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.038 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:08.038 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.106099+0000", 2020-01-16T00:14:08.038 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042069299999999998 2020-01-16T00:14:08.038 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.038 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.039 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:08.039 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.106520+0000", 2020-01-16T00:14:08.039 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0052864560000000001 2020-01-16T00:14:08.039 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.039 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.039 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:08.039 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.111806+0000", 2020-01-16T00:14:08.040 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00080835800000000001 2020-01-16T00:14:08.040 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.040 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.040 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:08.040 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.112615+0000", 2020-01-16T00:14:08.040 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1490999999999997e-05 2020-01-16T00:14:08.040 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.040 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.041 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:08.041 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.112686+0000", 2020-01-16T00:14:08.041 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000243174 2020-01-16T00:14:08.041 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.041 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.041 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:08.041 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.112929+0000", 2020-01-16T00:14:08.041 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012248446 2020-01-16T00:14:08.042 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.042 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:08.042 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.042 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.042 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.042 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.0.13:31 2.0 2:1afbc8ac:::smithi12313934-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 4096~737792] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e50)", 2020-01-16T00:14:08.042 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:44.061109+0000", 2020-01-16T00:14:08.043 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 23.819358526999999, 2020-01-16T00:14:08.043 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012186723, 2020-01-16T00:14:08.043 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:08.043 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:08.043 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:08.043 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.043 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:08.043 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.061109+0000", 2020-01-16T00:14:08.044 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:08.044 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.044 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.044 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:08.044 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.061109+0000", 2020-01-16T00:14:08.044 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:08.044 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.045 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.045 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:08.045 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.061106+0000", 2020-01-16T00:14:08.045 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012349870000000001 2020-01-16T00:14:08.045 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.045 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.045 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:08.045 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062341+0000", 2020-01-16T00:14:08.046 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0235999999999996e-05 2020-01-16T00:14:08.046 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.046 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.046 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:08.046 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062411+0000", 2020-01-16T00:14:08.046 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7078000000000001e-05 2020-01-16T00:14:08.046 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.047 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.047 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:08.047 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062428+0000", 2020-01-16T00:14:08.047 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2158999999999996e-05 2020-01-16T00:14:08.047 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.047 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.047 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.047 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062520+0000", 2020-01-16T00:14:08.048 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011203499999999999 2020-01-16T00:14:08.048 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.048 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.048 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:08.048 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062632+0000", 2020-01-16T00:14:08.048 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017238400000000001 2020-01-16T00:14:08.048 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.048 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.049 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:14:08.049 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062805+0000", 2020-01-16T00:14:08.049 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010264989 2020-01-16T00:14:08.049 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.049 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.049 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:08.049 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073070+0000", 2020-01-16T00:14:08.050 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0533000000000001e-05 2020-01-16T00:14:08.050 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.050 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.050 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:08.050 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073090+0000", 2020-01-16T00:14:08.050 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2309000000000002e-05 2020-01-16T00:14:08.050 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.050 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.051 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:08.051 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073163+0000", 2020-01-16T00:14:08.051 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3035999999999997e-05 2020-01-16T00:14:08.051 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.051 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.051 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:08.051 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073236+0000", 2020-01-16T00:14:08.051 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0056999999999998e-05 2020-01-16T00:14:08.052 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.052 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.052 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:08.052 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073296+0000", 2020-01-16T00:14:08.052 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012186723 2020-01-16T00:14:08.052 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.052 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:08.052 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.053 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.053 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.053 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.0.13:32 2.0 2:1afbc8ac:::smithi12313934-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 916992~596480] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e50)", 2020-01-16T00:14:08.053 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:44.062464+0000", 2020-01-16T00:14:08.053 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 23.818003741999998, 2020-01-16T00:14:08.053 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011316405999999999, 2020-01-16T00:14:08.053 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:08.054 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:08.054 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:08.054 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.054 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:08.054 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062464+0000", 2020-01-16T00:14:08.054 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:08.054 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.054 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.055 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:08.055 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062464+0000", 2020-01-16T00:14:08.055 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:08.055 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.055 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.055 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:08.055 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062462+0000", 2020-01-16T00:14:08.055 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00073330900000000002 2020-01-16T00:14:08.056 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.056 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.056 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:08.056 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063196+0000", 2020-01-16T00:14:08.056 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5330000000000001e-05 2020-01-16T00:14:08.056 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.056 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.056 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:08.057 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063211+0000", 2020-01-16T00:14:08.057 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5840000000000001e-06 2020-01-16T00:14:08.057 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.057 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.057 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:08.057 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063221+0000", 2020-01-16T00:14:08.057 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001786915 2020-01-16T00:14:08.058 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.058 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.058 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.058 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.065007+0000", 2020-01-16T00:14:08.058 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00021770000000000001 2020-01-16T00:14:08.058 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.058 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.058 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:08.059 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.065225+0000", 2020-01-16T00:14:08.059 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00032623 2020-01-16T00:14:08.059 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.059 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.059 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:14:08.059 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.065551+0000", 2020-01-16T00:14:08.059 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0078227499999999998 2020-01-16T00:14:08.060 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.060 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.060 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:08.060 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073374+0000", 2020-01-16T00:14:08.060 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.717e-05 2020-01-16T00:14:08.060 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.060 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.060 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:08.061 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073391+0000", 2020-01-16T00:14:08.061 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011493999999999999 2020-01-16T00:14:08.061 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.061 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.061 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:08.061 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073506+0000", 2020-01-16T00:14:08.061 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000159281 2020-01-16T00:14:08.061 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.062 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.062 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:08.062 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073666+0000", 2020-01-16T00:14:08.062 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000114675 2020-01-16T00:14:08.062 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.062 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.062 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:08.063 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073780+0000", 2020-01-16T00:14:08.063 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011316405999999999 2020-01-16T00:14:08.063 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.063 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:08.063 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.063 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.063 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.063 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.0.13:33 2.0 2:1afbc8ac:::smithi12313934-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 1541120~378368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e50)", 2020-01-16T00:14:08.064 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:44.063236+0000", 2020-01-16T00:14:08.064 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 23.817231863, 2020-01-16T00:14:08.064 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015290962999999999, 2020-01-16T00:14:08.064 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:08.064 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:08.064 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:08.064 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.064 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:08.065 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063236+0000", 2020-01-16T00:14:08.065 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:08.065 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.065 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.065 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:08.065 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063236+0000", 2020-01-16T00:14:08.065 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:08.065 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.066 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.066 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:08.066 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063234+0000", 2020-01-16T00:14:08.066 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000500702 2020-01-16T00:14:08.066 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.066 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.066 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:08.067 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063735+0000", 2020-01-16T00:14:08.067 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4734e-05 2020-01-16T00:14:08.067 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.067 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.067 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:08.067 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063750+0000", 2020-01-16T00:14:08.067 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.382e-05 2020-01-16T00:14:08.067 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.068 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.068 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:08.068 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063764+0000", 2020-01-16T00:14:08.068 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0059030799999999998 2020-01-16T00:14:08.068 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.068 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.068 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.069 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.069667+0000", 2020-01-16T00:14:08.069 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3961999999999994e-05 2020-01-16T00:14:08.069 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.069 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.069 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:08.069 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.069741+0000", 2020-01-16T00:14:08.069 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000114931 2020-01-16T00:14:08.069 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.070 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.070 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:14:08.070 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.069856+0000", 2020-01-16T00:14:08.070 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0039948270000000003 2020-01-16T00:14:08.070 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.070 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.070 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:08.070 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073850+0000", 2020-01-16T00:14:08.071 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0016459140000000001 2020-01-16T00:14:08.071 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.071 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.071 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:08.071 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.075496+0000", 2020-01-16T00:14:08.071 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002957997 2020-01-16T00:14:08.071 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.071 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.072 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:08.072 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.078454+0000", 2020-01-16T00:14:08.072 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9190999999999998e-05 2020-01-16T00:14:08.072 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.072 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.072 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:08.072 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.078493+0000", 2020-01-16T00:14:08.073 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3176999999999998e-05 2020-01-16T00:14:08.073 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.073 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.073 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:08.073 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.078527+0000", 2020-01-16T00:14:08.073 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015290962999999999 2020-01-16T00:14:08.073 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.073 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:08.074 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.074 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.074 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.074 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4641 3.d 3:b1521a3e:::smithi12313934-25:head [read 0~3596797,checksum] snapc 0=[] ondisk+read+known_if_redirected e57)", 2020-01-16T00:14:08.074 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:04.086675+0000", 2020-01-16T00:14:08.074 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 3.793792807, 2020-01-16T00:14:08.074 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011822397, 2020-01-16T00:14:08.074 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:08.075 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:08.075 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:08.075 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:08.075 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:08.075 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4641 2020-01-16T00:14:08.075 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.075 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:08.075 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.076 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:08.076 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.086675+0000", 2020-01-16T00:14:08.076 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:08.076 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.076 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.076 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:08.076 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.086675+0000", 2020-01-16T00:14:08.077 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:08.077 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.077 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.077 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:08.077 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.086674+0000", 2020-01-16T00:14:08.077 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0809999999999998e-06 2020-01-16T00:14:08.077 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.077 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.078 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:08.078 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.086678+0000", 2020-01-16T00:14:08.078 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4889999999999997e-06 2020-01-16T00:14:08.078 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.078 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.078 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:08.078 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.086682+0000", 2020-01-16T00:14:08.078 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9620000000000002e-06 2020-01-16T00:14:08.079 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.079 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.079 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:08.079 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.086685+0000", 2020-01-16T00:14:08.079 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010215452 2020-01-16T00:14:08.079 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.079 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.079 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.080 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.096900+0000", 2020-01-16T00:14:08.080 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00021274299999999999 2020-01-16T00:14:08.080 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.080 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.080 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:08.080 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.097113+0000", 2020-01-16T00:14:08.080 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0013839480000000001 2020-01-16T00:14:08.080 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.081 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.081 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:08.081 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.098497+0000", 2020-01-16T00:14:08.081 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011822397 2020-01-16T00:14:08.081 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.081 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:08.081 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.082 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.082 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:08.082 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:08.083 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:08.082+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:08.083 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-01-16T00:14:08.135 INFO:tasks.rados.rados.0.smithi123.stdout:1687: finishing write tid 4 to smithi12313934-41 2020-01-16T00:14:08.135 INFO:tasks.rados.rados.0.smithi123.stdout:1684: finishing write tid 2 to smithi12313934-42 2020-01-16T00:14:08.136 INFO:tasks.rados.rados.0.smithi123.stdout:1687: finishing write tid 5 to smithi12313934-41 2020-01-16T00:14:08.136 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 41 v 234 (ObjNum 602 snap 0 seq_num 602) dirty exists 2020-01-16T00:14:08.136 INFO:tasks.rados.rados.0.smithi123.stdout:1687: left oid 41 (ObjNum 602 snap 0 seq_num 602) 2020-01-16T00:14:08.136 INFO:tasks.rados.rados.0.smithi123.stdout:1684: finishing write tid 4 to smithi12313934-42 2020-01-16T00:14:08.136 INFO:tasks.rados.rados.0.smithi123.stdout:1684: finishing write tid 5 to smithi12313934-42 2020-01-16T00:14:08.136 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 42 v 204 (ObjNum 601 snap 0 seq_num 601) dirty exists 2020-01-16T00:14:08.136 INFO:tasks.rados.rados.0.smithi123.stdout:1684: left oid 42 (ObjNum 601 snap 0 seq_num 601) 2020-01-16T00:14:08.137 INFO:tasks.rados.rados.0.smithi123.stdout:1689: finishing write tid 1 to smithi12313934-46 2020-01-16T00:14:08.137 INFO:tasks.rados.rados.0.smithi123.stdout:1689: finishing write tid 2 to smithi12313934-46 2020-01-16T00:14:08.137 INFO:tasks.rados.rados.0.smithi123.stdout:1680: done (12 left) 2020-01-16T00:14:08.137 INFO:tasks.rados.rados.0.smithi123.stdout:1682: done (11 left) 2020-01-16T00:14:08.137 INFO:tasks.rados.rados.0.smithi123.stdout:1683: done (10 left) 2020-01-16T00:14:08.137 INFO:tasks.rados.rados.0.smithi123.stdout:1684: done (9 left) 2020-01-16T00:14:08.137 INFO:tasks.rados.rados.0.smithi123.stdout:1685: done (8 left) 2020-01-16T00:14:08.137 INFO:tasks.rados.rados.0.smithi123.stdout:1686: done (7 left) 2020-01-16T00:14:08.138 INFO:tasks.rados.rados.0.smithi123.stdout:1687: done (6 left) 2020-01-16T00:14:08.138 INFO:tasks.rados.rados.0.smithi123.stdout:1688: done (5 left) 2020-01-16T00:14:08.138 INFO:tasks.rados.rados.0.smithi123.stdout:1693: done (4 left) 2020-01-16T00:14:08.138 INFO:tasks.rados.rados.0.smithi123.stdout:1694: read oid 47 snap -1 2020-01-16T00:14:08.138 INFO:tasks.rados.rados.0.smithi123.stdout:1694: expect (ObjNum 550 snap 0 seq_num 550) 2020-01-16T00:14:08.138 INFO:tasks.rados.rados.0.smithi123.stdout:1695: write (excl) oid 43 current snap is 0 2020-01-16T00:14:08.138 INFO:tasks.rados.rados.0.smithi123.stdout:1695: seq_num 607 ranges {601849=468030,1753864=447985} 2020-01-16T00:14:08.149 INFO:tasks.rados.rados.0.smithi123.stdout:1695: writing smithi12313934-43 from 601849 to 1069879 tid 1 2020-01-16T00:14:08.155 INFO:tasks.rados.rados.0.smithi123.stdout:1695: writing smithi12313934-43 from 1753864 to 2201849 tid 2 2020-01-16T00:14:08.155 INFO:tasks.rados.rados.0.smithi123.stdout:1696: delete oid 33 current snap is 0 2020-01-16T00:14:08.155 INFO:tasks.rados.rados.0.smithi123.stdout:1690: finishing write tid 1 to smithi12313934-18 2020-01-16T00:14:08.155 INFO:tasks.rados.rados.0.smithi123.stdout:1690: finishing write tid 2 to smithi12313934-18 2020-01-16T00:14:08.156 INFO:tasks.rados.rados.0.smithi123.stdout:1690: finishing write tid 4 to smithi12313934-18 2020-01-16T00:14:08.156 INFO:tasks.rados.rados.0.smithi123.stdout:1690: finishing write tid 5 to smithi12313934-18 2020-01-16T00:14:08.156 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 18 v 320 (ObjNum 604 snap 0 seq_num 604) dirty exists 2020-01-16T00:14:08.156 INFO:tasks.rados.rados.0.smithi123.stdout:1690: left oid 18 (ObjNum 604 snap 0 seq_num 604) 2020-01-16T00:14:08.156 INFO:tasks.rados.rados.0.smithi123.stdout:1689: finishing write tid 4 to smithi12313934-46 2020-01-16T00:14:08.156 INFO:tasks.rados.rados.0.smithi123.stdout:1689: finishing write tid 5 to smithi12313934-46 2020-01-16T00:14:08.156 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 46 v 322 (ObjNum 603 snap 0 seq_num 603) dirty exists 2020-01-16T00:14:08.157 INFO:tasks.rados.rados.0.smithi123.stdout:1689: left oid 46 (ObjNum 603 snap 0 seq_num 603) 2020-01-16T00:14:08.157 INFO:tasks.rados.rados.0.smithi123.stdout:1692: finishing write tid 1 to smithi12313934-40 2020-01-16T00:14:08.157 INFO:tasks.rados.rados.0.smithi123.stdout:1692: finishing write tid 2 to smithi12313934-40 2020-01-16T00:14:08.157 INFO:tasks.rados.rados.0.smithi123.stdout:1692: finishing write tid 4 to smithi12313934-40 2020-01-16T00:14:08.157 INFO:tasks.rados.rados.0.smithi123.stdout:1692: finishing write tid 5 to smithi12313934-40 2020-01-16T00:14:08.157 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 40 v 268 (ObjNum 605 snap 0 seq_num 605) dirty exists 2020-01-16T00:14:08.157 INFO:tasks.rados.rados.0.smithi123.stdout:1692: left oid 40 (ObjNum 605 snap 0 seq_num 605) 2020-01-16T00:14:08.158 INFO:tasks.rados.rados.0.smithi123.stdout:1694: expect (ObjNum 550 snap 0 seq_num 550) 2020-01-16T00:14:08.184 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:08.182+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:08.204 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:08.204 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:14:08.204 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:14:08.204 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:08.213 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-01-16T00:14:08.285 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:08.282+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:08.314 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:08.314 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:14:08.314 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:14:08.314 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:14:08.315 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:08.326 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-01-16T00:14:08.351 INFO:tasks.rados.rados.0.smithi123.stdout:1695: finishing write tid 1 to smithi12313934-43 2020-01-16T00:14:08.351 INFO:tasks.rados.rados.0.smithi123.stdout:1695: finishing write tid 2 to smithi12313934-43 2020-01-16T00:14:08.351 INFO:tasks.rados.rados.0.smithi123.stdout:1695: finishing write tid 4 to smithi12313934-43 2020-01-16T00:14:08.351 INFO:tasks.rados.rados.0.smithi123.stdout:1695: finishing write tid 5 to smithi12313934-43 2020-01-16T00:14:08.351 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 43 v 271 (ObjNum 607 snap 0 seq_num 607) dirty exists 2020-01-16T00:14:08.351 INFO:tasks.rados.rados.0.smithi123.stdout:1695: left oid 43 (ObjNum 607 snap 0 seq_num 607) 2020-01-16T00:14:08.352 INFO:tasks.rados.rados.0.smithi123.stdout:1689: done (6 left) 2020-01-16T00:14:08.352 INFO:tasks.rados.rados.0.smithi123.stdout:1690: done (5 left) 2020-01-16T00:14:08.352 INFO:tasks.rados.rados.0.smithi123.stdout:1691: done (4 left) 2020-01-16T00:14:08.352 INFO:tasks.rados.rados.0.smithi123.stdout:1692: done (3 left) 2020-01-16T00:14:08.352 INFO:tasks.rados.rados.0.smithi123.stdout:1694: done (2 left) 2020-01-16T00:14:08.352 INFO:tasks.rados.rados.0.smithi123.stdout:1695: done (1 left) 2020-01-16T00:14:08.352 INFO:tasks.rados.rados.0.smithi123.stdout:1696: done (0 left) 2020-01-16T00:14:08.352 INFO:tasks.rados.rados.0.smithi123.stdout:1697: write (excl) oid 34 current snap is 0 2020-01-16T00:14:08.353 INFO:tasks.rados.rados.0.smithi123.stdout:1697: seq_num 609 ranges {635463=734881,1987181=248282} 2020-01-16T00:14:08.369 INFO:tasks.rados.rados.0.smithi123.stdout:1697: writing smithi12313934-34 from 635463 to 1370344 tid 1 2020-01-16T00:14:08.374 INFO:tasks.rados.rados.0.smithi123.stdout:1697: writing smithi12313934-34 from 1987181 to 2235463 tid 2 2020-01-16T00:14:08.374 INFO:tasks.rados.rados.0.smithi123.stdout:1698: read oid 9 snap -1 2020-01-16T00:14:08.374 INFO:tasks.rados.rados.0.smithi123.stdout:1698: expect (ObjNum 572 snap 0 seq_num 572) 2020-01-16T00:14:08.374 INFO:tasks.rados.rados.0.smithi123.stdout:1699: read oid 30 snap -1 2020-01-16T00:14:08.374 INFO:tasks.rados.rados.0.smithi123.stdout:1699: expect (ObjNum 540 snap 0 seq_num 540) 2020-01-16T00:14:08.374 INFO:tasks.rados.rados.0.smithi123.stdout:1700: write (excl) oid 42 current snap is 0 2020-01-16T00:14:08.375 INFO:tasks.rados.rados.0.smithi123.stdout:1700: seq_num 610 ranges {652270=410130,1529310=493867} 2020-01-16T00:14:08.380 INFO:tasks.rados.rados.0.smithi123.stdout:1700: writing smithi12313934-42 from 652270 to 1062400 tid 1 2020-01-16T00:14:08.386 INFO:tasks.rados.rados.0.smithi123.stdout:1700: writing smithi12313934-42 from 1529310 to 2023177 tid 2 2020-01-16T00:14:08.386 INFO:tasks.rados.rados.0.smithi123.stdout:1701: write oid 19 current snap is 0 2020-01-16T00:14:08.386 INFO:tasks.rados.rados.0.smithi123.stdout:1701: seq_num 611 ranges {669077=485379,1587792=681285} 2020-01-16T00:14:08.386 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:08.382+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:08.392 INFO:tasks.rados.rados.0.smithi123.stdout:1701: writing smithi12313934-19 from 669077 to 1154456 tid 1 2020-01-16T00:14:08.399 INFO:tasks.rados.rados.0.smithi123.stdout:1701: writing smithi12313934-19 from 1587792 to 2269077 tid 2 2020-01-16T00:14:08.399 INFO:tasks.rados.rados.0.smithi123.stdout:1702: read oid 16 snap -1 2020-01-16T00:14:08.399 INFO:tasks.rados.rados.0.smithi123.stdout:1702: expect (ObjNum 592 snap 0 seq_num 592) 2020-01-16T00:14:08.399 INFO:tasks.rados.rados.0.smithi123.stdout:1703: read oid 48 snap -1 2020-01-16T00:14:08.400 INFO:tasks.rados.rados.0.smithi123.stdout:1703: expect (ObjNum 599 snap 0 seq_num 599) 2020-01-16T00:14:08.400 INFO:tasks.rados.rados.0.smithi123.stdout:1704: read oid 37 snap -1 2020-01-16T00:14:08.400 INFO:tasks.rados.rados.0.smithi123.stdout:1704: expect (ObjNum 502 snap 0 seq_num 502) 2020-01-16T00:14:08.400 INFO:tasks.rados.rados.0.smithi123.stdout:1705: read oid 12 snap -1 2020-01-16T00:14:08.400 INFO:tasks.rados.rados.0.smithi123.stdout:1705: expect (ObjNum 517 snap 0 seq_num 517) 2020-01-16T00:14:08.400 INFO:tasks.rados.rados.0.smithi123.stdout:1706: read oid 25 snap -1 2020-01-16T00:14:08.400 INFO:tasks.rados.rados.0.smithi123.stdout:1706: expect (ObjNum 571 snap 0 seq_num 571) 2020-01-16T00:14:08.401 INFO:tasks.rados.rados.0.smithi123.stdout:1707: delete oid 36 current snap is 0 2020-01-16T00:14:08.401 INFO:tasks.rados.rados.0.smithi123.stdout:1697: finishing write tid 1 to smithi12313934-34 2020-01-16T00:14:08.401 INFO:tasks.rados.rados.0.smithi123.stdout:1697: finishing write tid 2 to smithi12313934-34 2020-01-16T00:14:08.401 INFO:tasks.rados.rados.0.smithi123.stdout:1698: expect (ObjNum 572 snap 0 seq_num 572) 2020-01-16T00:14:08.446 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:08.446 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:14:08.447 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:14:08.447 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:14:08.447 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.447 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1654 3.a 3:5069665e:::smithi12313934-2:head [write 1614880~581041] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:14:08.447 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.714839+0000", 2020-01-16T00:14:08.447 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 75.728817751999998, 2020-01-16T00:14:08.447 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013588093000000001, 2020-01-16T00:14:08.448 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:08.448 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:08.448 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:08.448 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:08.448 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:08.448 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1654 2020-01-16T00:14:08.448 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.448 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:08.449 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.449 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:08.449 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.714839+0000", 2020-01-16T00:14:08.449 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:08.449 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.449 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.449 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:08.450 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.714839+0000", 2020-01-16T00:14:08.450 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:08.450 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.450 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.450 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:08.450 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.714836+0000", 2020-01-16T00:14:08.450 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00054657999999999996 2020-01-16T00:14:08.451 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.451 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.451 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:08.451 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.715382+0000", 2020-01-16T00:14:08.451 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7743000000000001e-05 2020-01-16T00:14:08.451 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.451 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.451 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:08.452 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.715420+0000", 2020-01-16T00:14:08.452 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.221e-05 2020-01-16T00:14:08.452 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.452 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.452 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:08.453 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.715432+0000", 2020-01-16T00:14:08.453 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2487000000000001e-05 2020-01-16T00:14:08.453 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.453 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.453 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.454 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.715485+0000", 2020-01-16T00:14:08.454 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000726284 2020-01-16T00:14:08.454 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.454 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.454 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:08.454 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.716211+0000", 2020-01-16T00:14:08.454 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0067930899999999999 2020-01-16T00:14:08.454 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.455 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.455 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.455 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.723004+0000", 2020-01-16T00:14:08.455 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012579400000000001 2020-01-16T00:14:08.456 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.456 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.456 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:08.456 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.723130+0000", 2020-01-16T00:14:08.456 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013465800000000001 2020-01-16T00:14:08.456 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.457 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.457 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:08.457 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.723265+0000", 2020-01-16T00:14:08.457 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023307250000000001 2020-01-16T00:14:08.457 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.457 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.457 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:08.457 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.725595+0000", 2020-01-16T00:14:08.458 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012582139999999999 2020-01-16T00:14:08.458 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.458 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.458 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:08.458 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.726853+0000", 2020-01-16T00:14:08.458 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014479110000000001 2020-01-16T00:14:08.459 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.459 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.459 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:08.459 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.728301+0000", 2020-01-16T00:14:08.459 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6866000000000007e-05 2020-01-16T00:14:08.459 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.459 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.459 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:08.460 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.728368+0000", 2020-01-16T00:14:08.460 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8740999999999999e-05 2020-01-16T00:14:08.460 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.460 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.460 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:08.460 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.728427+0000", 2020-01-16T00:14:08.460 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013588093000000001 2020-01-16T00:14:08.461 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.461 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:08.461 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.461 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.461 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.461 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1655 3.a 3:5069665e:::smithi12313934-2:head [write 2785791~71399] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:14:08.461 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.718297+0000", 2020-01-16T00:14:08.461 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 75.725359898999997, 2020-01-16T00:14:08.462 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013817354, 2020-01-16T00:14:08.462 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:08.462 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:08.462 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:08.462 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:08.462 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:08.462 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1655 2020-01-16T00:14:08.463 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.463 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:08.468 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.468 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:08.469 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718297+0000", 2020-01-16T00:14:08.469 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:08.469 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.469 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.469 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:08.469 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718297+0000", 2020-01-16T00:14:08.469 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:08.470 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.470 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.470 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:08.470 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718294+0000", 2020-01-16T00:14:08.470 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000103558 2020-01-16T00:14:08.470 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.470 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.470 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:08.471 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718397+0000", 2020-01-16T00:14:08.471 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9287e-05 2020-01-16T00:14:08.471 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.471 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.471 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:08.471 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718417+0000", 2020-01-16T00:14:08.471 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6680000000000001e-06 2020-01-16T00:14:08.472 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.472 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.472 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:08.472 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718424+0000", 2020-01-16T00:14:08.472 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2701999999999994e-05 2020-01-16T00:14:08.472 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.472 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.472 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.473 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718517+0000", 2020-01-16T00:14:08.473 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3893000000000006e-05 2020-01-16T00:14:08.473 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.473 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.473 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:08.473 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718581+0000", 2020-01-16T00:14:08.473 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0059630079999999997 2020-01-16T00:14:08.474 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.474 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.474 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.474 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.724544+0000", 2020-01-16T00:14:08.474 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000309108 2020-01-16T00:14:08.474 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.474 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.474 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:08.475 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.724853+0000", 2020-01-16T00:14:08.475 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.005469794 2020-01-16T00:14:08.475 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.475 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.475 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.475 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.730323+0000", 2020-01-16T00:14:08.475 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018506100000000001 2020-01-16T00:14:08.475 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.476 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.476 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:08.476 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.730508+0000", 2020-01-16T00:14:08.476 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016729099999999999 2020-01-16T00:14:08.476 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.476 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.476 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:08.476 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.730675+0000", 2020-01-16T00:14:08.477 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00094193300000000005 2020-01-16T00:14:08.477 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.477 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.477 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:08.477 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731617+0000", 2020-01-16T00:14:08.477 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000214613 2020-01-16T00:14:08.477 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.478 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.478 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:08.478 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731832+0000", 2020-01-16T00:14:08.478 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018326400000000001 2020-01-16T00:14:08.478 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.478 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.478 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:08.478 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732015+0000", 2020-01-16T00:14:08.479 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0392e-05 2020-01-16T00:14:08.479 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.479 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.479 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:08.479 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732075+0000", 2020-01-16T00:14:08.479 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8677999999999998e-05 2020-01-16T00:14:08.479 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.479 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.480 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:08.480 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732114+0000", 2020-01-16T00:14:08.480 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013817354 2020-01-16T00:14:08.480 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.480 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:08.480 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.480 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.481 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.481 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1656 3.a 3:5069665e:::smithi12313934-2:head [setxattr _header (55) in=7b,truncate 2857190] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:14:08.481 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.718453+0000", 2020-01-16T00:14:08.481 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 75.725203218000004, 2020-01-16T00:14:08.481 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014083462, 2020-01-16T00:14:08.481 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:08.481 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:08.481 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:08.482 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:08.482 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:08.482 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1656 2020-01-16T00:14:08.482 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.482 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:08.482 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.482 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:08.483 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718453+0000", 2020-01-16T00:14:08.483 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:08.483 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.483 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.483 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:08.483 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718453+0000", 2020-01-16T00:14:08.483 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999828 2020-01-16T00:14:08.483 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.484 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.484 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:08.484 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718436+0000", 2020-01-16T00:14:08.484 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4051999999999999e-05 2020-01-16T00:14:08.484 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.484 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.484 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:08.484 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718460+0000", 2020-01-16T00:14:08.485 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6380000000000004e-06 2020-01-16T00:14:08.485 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.485 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.485 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:08.485 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718468+0000", 2020-01-16T00:14:08.485 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6750000000000001e-06 2020-01-16T00:14:08.485 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.485 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.486 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:08.486 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718472+0000", 2020-01-16T00:14:08.486 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016218499999999999 2020-01-16T00:14:08.486 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.486 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.486 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.486 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718635+0000", 2020-01-16T00:14:08.487 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2765e-05 2020-01-16T00:14:08.487 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.487 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.487 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:08.487 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718697+0000", 2020-01-16T00:14:08.487 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0062239249999999999 2020-01-16T00:14:08.487 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.487 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.488 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.488 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.724921+0000", 2020-01-16T00:14:08.488 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1527999999999999e-05 2020-01-16T00:14:08.488 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.488 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.488 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:08.488 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.724973+0000", 2020-01-16T00:14:08.488 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0060569719999999999 2020-01-16T00:14:08.489 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.489 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.489 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.489 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731030+0000", 2020-01-16T00:14:08.489 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5859999999999996e-05 2020-01-16T00:14:08.489 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.489 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.490 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:08.490 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731096+0000", 2020-01-16T00:14:08.490 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000112018 2020-01-16T00:14:08.490 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.490 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.490 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:08.490 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731208+0000", 2020-01-16T00:14:08.490 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000945889 2020-01-16T00:14:08.491 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.491 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.491 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:08.491 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732154+0000", 2020-01-16T00:14:08.491 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0834000000000003e-05 2020-01-16T00:14:08.491 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.491 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.492 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:08.492 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732184+0000", 2020-01-16T00:14:08.492 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028587600000000001 2020-01-16T00:14:08.492 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.492 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.492 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:08.492 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732470+0000", 2020-01-16T00:14:08.492 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1953999999999999e-05 2020-01-16T00:14:08.493 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.493 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.493 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:08.493 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732502+0000", 2020-01-16T00:14:08.493 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4644999999999998e-05 2020-01-16T00:14:08.493 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.493 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.493 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:08.494 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732537+0000", 2020-01-16T00:14:08.494 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014083462 2020-01-16T00:14:08.494 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.494 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:08.494 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.494 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.494 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.495 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1657 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-01-16T00:14:08.495 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.718495+0000", 2020-01-16T00:14:08.495 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 75.725161553000007, 2020-01-16T00:14:08.495 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014299630000000001, 2020-01-16T00:14:08.495 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:08.495 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:08.495 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:08.495 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:08.496 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:08.496 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1657 2020-01-16T00:14:08.496 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.496 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:08.496 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.496 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:08.496 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718495+0000", 2020-01-16T00:14:08.496 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:08.497 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.497 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.497 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:08.497 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718495+0000", 2020-01-16T00:14:08.497 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:08.497 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.497 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.498 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:08.498 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718494+0000", 2020-01-16T00:14:08.498 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8860000000000003e-06 2020-01-16T00:14:08.498 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.498 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.498 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:08.498 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718499+0000", 2020-01-16T00:14:08.498 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9786000000000002e-05 2020-01-16T00:14:08.499 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.499 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.499 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:08.499 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718539+0000", 2020-01-16T00:14:08.499 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1869999999999999e-05 2020-01-16T00:14:08.499 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.499 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.499 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:08.500 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718561+0000", 2020-01-16T00:14:08.500 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000172206 2020-01-16T00:14:08.500 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.500 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.500 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.500 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718733+0000", 2020-01-16T00:14:08.500 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0662999999999998e-05 2020-01-16T00:14:08.501 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.501 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.501 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:08.501 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718793+0000", 2020-01-16T00:14:08.501 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.006217958 2020-01-16T00:14:08.501 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.501 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.501 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.502 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.725011+0000", 2020-01-16T00:14:08.502 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1625999999999996e-05 2020-01-16T00:14:08.502 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.502 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.502 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:08.502 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.725073+0000", 2020-01-16T00:14:08.502 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0063923119999999998 2020-01-16T00:14:08.502 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.503 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.503 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.503 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731465+0000", 2020-01-16T00:14:08.503 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2534000000000005e-05 2020-01-16T00:14:08.503 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.503 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.503 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:08.504 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731558+0000", 2020-01-16T00:14:08.504 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010238809999999999 2020-01-16T00:14:08.504 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.504 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.504 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.504 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732582+0000", 2020-01-16T00:14:08.504 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000100918 2020-01-16T00:14:08.504 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.505 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.505 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:08.505 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732683+0000", 2020-01-16T00:14:08.505 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000112064 2020-01-16T00:14:08.505 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.505 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.505 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:08.506 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732795+0000", 2020-01-16T00:14:08.506 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014299630000000001 2020-01-16T00:14:08.506 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.506 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:08.506 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.506 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.506 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.507 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1917 3.a 3:5069665e:::smithi12313934-2:head [write 2113302~674388] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:14:08.507 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:57.330712+0000", 2020-01-16T00:14:08.507 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 71.112945038000007, 2020-01-16T00:14:08.507 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012781043000000001, 2020-01-16T00:14:08.507 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:08.507 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:08.507 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:08.508 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:08.508 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:08.508 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1917 2020-01-16T00:14:08.508 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.508 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:08.508 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.508 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:08.508 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.330712+0000", 2020-01-16T00:14:08.509 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:08.509 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.509 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.509 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:08.509 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.330712+0000", 2020-01-16T00:14:08.509 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:08.509 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.510 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.510 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:08.510 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.330709+0000", 2020-01-16T00:14:08.510 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00071550499999999996 2020-01-16T00:14:08.510 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.510 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.510 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:08.510 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331424+0000", 2020-01-16T00:14:08.511 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3354e-05 2020-01-16T00:14:08.511 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.511 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.511 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:08.511 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331447+0000", 2020-01-16T00:14:08.511 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8200000000000008e-06 2020-01-16T00:14:08.511 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.511 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.512 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:08.512 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331457+0000", 2020-01-16T00:14:08.512 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5557999999999997e-05 2020-01-16T00:14:08.512 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.512 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.512 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.512 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331533+0000", 2020-01-16T00:14:08.512 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030925000000000001 2020-01-16T00:14:08.513 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.513 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.513 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:08.513 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331842+0000", 2020-01-16T00:14:08.513 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0078507860000000002 2020-01-16T00:14:08.513 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.513 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.514 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.514 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.339693+0000", 2020-01-16T00:14:08.514 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9321000000000007e-05 2020-01-16T00:14:08.514 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.514 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.514 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:08.514 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.339792+0000", 2020-01-16T00:14:08.514 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015147700000000001 2020-01-16T00:14:08.515 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.515 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.515 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:08.515 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.339944+0000", 2020-01-16T00:14:08.515 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002663066 2020-01-16T00:14:08.515 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.515 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.515 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:08.516 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.342607+0000", 2020-01-16T00:14:08.516 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0001552 2020-01-16T00:14:08.516 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.516 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.516 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:08.516 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.342762+0000", 2020-01-16T00:14:08.516 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00063636900000000002 2020-01-16T00:14:08.517 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.517 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.517 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:08.517 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343398+0000", 2020-01-16T00:14:08.517 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9687000000000001e-05 2020-01-16T00:14:08.517 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.518 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.518 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:08.518 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343438+0000", 2020-01-16T00:14:08.518 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4647e-05 2020-01-16T00:14:08.518 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.518 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.518 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:08.518 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343493+0000", 2020-01-16T00:14:08.519 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012781043000000001 2020-01-16T00:14:08.519 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.519 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:08.519 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.519 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.519 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.519 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2505 3.a 3:5069665e:::smithi12313934-2:head [write 1862260~479427] snapc 0=[] ondisk+write+known_if_redirected e37)", 2020-01-16T00:14:08.519 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:11.286548+0000", 2020-01-16T00:14:08.520 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 57.157108870999998, 2020-01-16T00:14:08.520 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013314061, 2020-01-16T00:14:08.520 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:08.520 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:08.520 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:08.520 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:08.520 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:08.520 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2505 2020-01-16T00:14:08.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.521 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:08.521 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.521 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:08.521 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.286548+0000", 2020-01-16T00:14:08.521 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:08.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.522 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.522 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:08.522 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.286548+0000", 2020-01-16T00:14:08.522 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999957 2020-01-16T00:14:08.522 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.522 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.522 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:08.522 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.286544+0000", 2020-01-16T00:14:08.523 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042043700000000001 2020-01-16T00:14:08.523 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.523 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.523 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:08.523 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.286964+0000", 2020-01-16T00:14:08.523 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5756999999999997e-05 2020-01-16T00:14:08.523 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.524 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.524 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:08.524 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.287000+0000", 2020-01-16T00:14:08.524 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8960000000000001e-05 2020-01-16T00:14:08.524 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.524 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.524 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:08.524 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.287019+0000", 2020-01-16T00:14:08.525 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.7219000000000005e-05 2020-01-16T00:14:08.525 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.525 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.525 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.525 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.287116+0000", 2020-01-16T00:14:08.525 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033208899999999998 2020-01-16T00:14:08.525 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.525 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.526 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:08.526 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.287448+0000", 2020-01-16T00:14:08.526 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0082732420000000001 2020-01-16T00:14:08.526 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.526 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.526 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.526 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.295721+0000", 2020-01-16T00:14:08.527 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017259999999999999 2020-01-16T00:14:08.527 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.527 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.527 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:08.527 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.295894+0000", 2020-01-16T00:14:08.527 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00029238000000000002 2020-01-16T00:14:08.527 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.527 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.528 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:08.528 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.296186+0000", 2020-01-16T00:14:08.528 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0034626409999999998 2020-01-16T00:14:08.528 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.528 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.528 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:08.528 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299649+0000", 2020-01-16T00:14:08.528 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2300000000000002e-06 2020-01-16T00:14:08.529 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.529 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.529 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:08.529 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299656+0000", 2020-01-16T00:14:08.529 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1561000000000001e-05 2020-01-16T00:14:08.529 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.529 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.530 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:08.530 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299738+0000", 2020-01-16T00:14:08.530 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8491000000000005e-05 2020-01-16T00:14:08.530 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.530 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.530 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:08.530 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299806+0000", 2020-01-16T00:14:08.530 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5541000000000002e-05 2020-01-16T00:14:08.531 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.531 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.531 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:08.531 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299862+0000", 2020-01-16T00:14:08.531 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013314061 2020-01-16T00:14:08.531 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.531 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:08.532 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.532 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.532 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.532 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2791 3.a 3:54f2c117:::smithi12313934-41:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e41)", 2020-01-16T00:14:08.532 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:17.213167+0000", 2020-01-16T00:14:08.532 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 51.230489859999999, 2020-01-16T00:14:08.532 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016024037000000001, 2020-01-16T00:14:08.532 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:08.533 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:08.533 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:08.533 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:08.533 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:08.533 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2791 2020-01-16T00:14:08.533 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.533 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:08.533 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.534 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:08.534 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213167+0000", 2020-01-16T00:14:08.534 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:08.534 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.534 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.534 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:08.534 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213167+0000", 2020-01-16T00:14:08.535 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:08.535 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.535 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.535 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:08.535 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213166+0000", 2020-01-16T00:14:08.535 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6030000000000004e-06 2020-01-16T00:14:08.535 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.535 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.536 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:08.536 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213172+0000", 2020-01-16T00:14:08.536 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0929999999999997e-06 2020-01-16T00:14:08.536 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.536 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.536 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:08.536 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213179+0000", 2020-01-16T00:14:08.536 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4299999999999997e-06 2020-01-16T00:14:08.537 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.537 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.537 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:08.537 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213184+0000", 2020-01-16T00:14:08.537 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012106695000000001 2020-01-16T00:14:08.537 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.537 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.538 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.538 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.225291+0000", 2020-01-16T00:14:08.538 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020863699999999999 2020-01-16T00:14:08.538 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.538 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.538 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:08.538 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.225499+0000", 2020-01-16T00:14:08.538 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0036913050000000002 2020-01-16T00:14:08.539 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.539 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.539 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:08.539 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.229191+0000", 2020-01-16T00:14:08.539 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016024037000000001 2020-01-16T00:14:08.539 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.540 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:08.540 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.540 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.541 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.541 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2792 3.a 3:54f2c117:::smithi12313934-41:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e41)", 2020-01-16T00:14:08.541 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:17.213190+0000", 2020-01-16T00:14:08.541 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 51.230466386000003, 2020-01-16T00:14:08.541 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017165949, 2020-01-16T00:14:08.541 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:08.541 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:08.541 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:08.542 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:08.542 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:08.542 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2792 2020-01-16T00:14:08.542 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.542 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:08.542 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.542 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:08.543 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213190+0000", 2020-01-16T00:14:08.543 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:08.543 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.543 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.543 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:08.543 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213190+0000", 2020-01-16T00:14:08.543 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:08.543 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.544 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.544 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:08.544 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213189+0000", 2020-01-16T00:14:08.544 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2699999999999998e-06 2020-01-16T00:14:08.544 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.544 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.544 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:08.544 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213194+0000", 2020-01-16T00:14:08.545 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8600000000000004e-06 2020-01-16T00:14:08.545 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.545 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.545 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:08.545 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213201+0000", 2020-01-16T00:14:08.545 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8500000000000002e-06 2020-01-16T00:14:08.545 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.546 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.546 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:08.546 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213205+0000", 2020-01-16T00:14:08.546 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016028924999999999 2020-01-16T00:14:08.546 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.546 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.546 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.546 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.229234+0000", 2020-01-16T00:14:08.547 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019773999999999999 2020-01-16T00:14:08.547 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.547 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.547 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:08.547 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.229432+0000", 2020-01-16T00:14:08.547 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00092414400000000001 2020-01-16T00:14:08.547 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.548 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.548 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:08.548 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.230356+0000", 2020-01-16T00:14:08.548 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017165949 2020-01-16T00:14:08.548 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.548 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:08.548 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.548 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.549 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.549 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2793 3.a 3:54f2c117:::smithi12313934-41:head [read 0~3210170,checksum] snapc 0=[] ondisk+read+known_if_redirected e41)", 2020-01-16T00:14:08.549 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:17.213211+0000", 2020-01-16T00:14:08.549 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 51.230445840000002, 2020-01-16T00:14:08.549 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019069413, 2020-01-16T00:14:08.549 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:08.549 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:08.549 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:08.550 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:08.550 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:08.550 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2793 2020-01-16T00:14:08.550 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.550 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:08.550 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.550 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:08.550 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213211+0000", 2020-01-16T00:14:08.551 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:08.551 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.551 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.551 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:08.551 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213211+0000", 2020-01-16T00:14:08.551 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:08.551 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.552 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.552 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:08.552 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213210+0000", 2020-01-16T00:14:08.552 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.66e-06 2020-01-16T00:14:08.552 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.552 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.552 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:08.553 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213216+0000", 2020-01-16T00:14:08.553 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.9469999999999997e-06 2020-01-16T00:14:08.553 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.553 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.553 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:08.553 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213223+0000", 2020-01-16T00:14:08.553 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9259999999999999e-06 2020-01-16T00:14:08.554 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.554 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.554 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:08.554 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213228+0000", 2020-01-16T00:14:08.554 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017171046999999998 2020-01-16T00:14:08.554 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.554 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.554 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.555 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.230399+0000", 2020-01-16T00:14:08.555 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000189932 2020-01-16T00:14:08.555 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.555 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.555 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:08.555 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.230589+0000", 2020-01-16T00:14:08.556 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0016916539999999999 2020-01-16T00:14:08.556 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.556 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.556 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:08.556 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.232280+0000", 2020-01-16T00:14:08.556 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019069413 2020-01-16T00:14:08.556 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.556 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:08.557 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.557 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.557 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.557 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3120 3.a 3:5069665e:::smithi12313934-2:head [write 1845146~524707] snapc 0=[] ondisk+write+known_if_redirected e44)", 2020-01-16T00:14:08.557 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.262852+0000", 2020-01-16T00:14:08.557 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 43.180804657000003, 2020-01-16T00:14:08.557 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014991175000000001, 2020-01-16T00:14:08.558 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:08.558 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:08.558 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:08.558 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:08.558 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:08.558 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3120 2020-01-16T00:14:08.558 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.558 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:08.559 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.559 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:08.559 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.262852+0000", 2020-01-16T00:14:08.559 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:08.559 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.559 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.559 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:08.560 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.262852+0000", 2020-01-16T00:14:08.560 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:08.560 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.560 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.560 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:08.560 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.262849+0000", 2020-01-16T00:14:08.560 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00058469900000000005 2020-01-16T00:14:08.560 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.561 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.561 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:08.561 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263434+0000", 2020-01-16T00:14:08.561 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.6650000000000001e-05 2020-01-16T00:14:08.561 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.561 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.561 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:08.562 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263461+0000", 2020-01-16T00:14:08.562 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3478e-05 2020-01-16T00:14:08.562 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.562 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.562 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:08.562 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263474+0000", 2020-01-16T00:14:08.563 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7824000000000002e-05 2020-01-16T00:14:08.563 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.563 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.563 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.564 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263522+0000", 2020-01-16T00:14:08.564 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001365464 2020-01-16T00:14:08.564 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.564 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.564 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:08.564 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.264888+0000", 2020-01-16T00:14:08.564 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0081368250000000003 2020-01-16T00:14:08.565 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.565 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.565 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.565 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.273024+0000", 2020-01-16T00:14:08.565 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6130000000000006e-05 2020-01-16T00:14:08.565 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.565 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.566 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:08.566 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.273090+0000", 2020-01-16T00:14:08.566 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000207507 2020-01-16T00:14:08.566 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.566 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.566 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:08.566 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.273298+0000", 2020-01-16T00:14:08.566 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0035649980000000002 2020-01-16T00:14:08.567 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.567 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.567 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:08.567 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276863+0000", 2020-01-16T00:14:08.567 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.1459999999999995e-05 2020-01-16T00:14:08.567 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.567 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.567 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:08.568 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276954+0000", 2020-01-16T00:14:08.568 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00076363899999999998 2020-01-16T00:14:08.568 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.568 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.568 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:08.568 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277718+0000", 2020-01-16T00:14:08.568 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0154999999999995e-05 2020-01-16T00:14:08.569 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.569 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.569 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:08.569 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277788+0000", 2020-01-16T00:14:08.569 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4914000000000003e-05 2020-01-16T00:14:08.569 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.569 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.569 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:08.570 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277843+0000", 2020-01-16T00:14:08.570 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014991175000000001 2020-01-16T00:14:08.570 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.570 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:08.570 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.570 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.570 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.570 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3121 3.a 3:5069665e:::smithi12313934-2:head [setxattr _header (55) in=7b,truncate 2369853] snapc 0=[] ondisk+write+known_if_redirected e44)", 2020-01-16T00:14:08.571 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.263494+0000", 2020-01-16T00:14:08.571 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 43.180162369000001, 2020-01-16T00:14:08.571 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014559274000000001, 2020-01-16T00:14:08.571 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:08.571 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:08.571 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:08.571 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:08.572 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:08.572 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3121 2020-01-16T00:14:08.572 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.572 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:08.572 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.572 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:08.572 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263494+0000", 2020-01-16T00:14:08.572 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:08.573 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.573 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.573 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:08.573 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263494+0000", 2020-01-16T00:14:08.573 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:08.573 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.573 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.573 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:08.574 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263493+0000", 2020-01-16T00:14:08.574 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3359999999999995e-06 2020-01-16T00:14:08.574 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.574 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.574 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:08.574 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263501+0000", 2020-01-16T00:14:08.574 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4100000000000002e-06 2020-01-16T00:14:08.574 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.575 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.575 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:08.575 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263509+0000", 2020-01-16T00:14:08.575 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3839999999999996e-06 2020-01-16T00:14:08.575 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.575 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.575 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:08.576 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263514+0000", 2020-01-16T00:14:08.576 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001430295 2020-01-16T00:14:08.576 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.576 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.576 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.576 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.264944+0000", 2020-01-16T00:14:08.576 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9710999999999999e-05 2020-01-16T00:14:08.576 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.577 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.577 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:08.577 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.264994+0000", 2020-01-16T00:14:08.577 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010036188999999999 2020-01-16T00:14:08.577 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.577 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.577 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.577 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.275030+0000", 2020-01-16T00:14:08.578 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1743000000000007e-05 2020-01-16T00:14:08.578 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.578 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.578 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:08.578 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.275112+0000", 2020-01-16T00:14:08.578 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013036100000000001 2020-01-16T00:14:08.578 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.579 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.579 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:08.579 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.275242+0000", 2020-01-16T00:14:08.579 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0016795549999999999 2020-01-16T00:14:08.579 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.579 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.579 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:08.579 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276922+0000", 2020-01-16T00:14:08.580 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018479400000000001 2020-01-16T00:14:08.580 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.580 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.580 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:08.580 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277107+0000", 2020-01-16T00:14:08.580 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00083585300000000001 2020-01-16T00:14:08.580 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.580 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.581 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:08.581 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277943+0000", 2020-01-16T00:14:08.581 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1719999999999999e-05 2020-01-16T00:14:08.581 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.581 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.581 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:08.581 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277994+0000", 2020-01-16T00:14:08.582 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9212999999999997e-05 2020-01-16T00:14:08.582 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.582 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.582 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:08.582 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.278054+0000", 2020-01-16T00:14:08.582 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014559274000000001 2020-01-16T00:14:08.582 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.582 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:08.583 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.583 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.583 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.583 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3122 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e44)", 2020-01-16T00:14:08.583 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.263525+0000", 2020-01-16T00:14:08.583 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 43.180131181, 2020-01-16T00:14:08.583 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014731833999999999, 2020-01-16T00:14:08.584 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:08.584 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:08.584 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:08.584 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:08.584 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:08.584 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3122 2020-01-16T00:14:08.584 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.584 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:08.585 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.585 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:08.585 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263525+0000", 2020-01-16T00:14:08.585 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:08.585 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.585 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.585 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:08.585 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263525+0000", 2020-01-16T00:14:08.586 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:08.586 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.586 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.586 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:08.586 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263525+0000", 2020-01-16T00:14:08.586 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5000000000000001e-06 2020-01-16T00:14:08.586 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.587 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.587 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:08.587 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263529+0000", 2020-01-16T00:14:08.587 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4669999999999998e-06 2020-01-16T00:14:08.587 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.587 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.587 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:08.587 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263537+0000", 2020-01-16T00:14:08.588 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0819999999999998e-06 2020-01-16T00:14:08.588 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.588 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.588 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:08.588 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263542+0000", 2020-01-16T00:14:08.588 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0015024960000000001 2020-01-16T00:14:08.588 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.588 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.589 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.589 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.265044+0000", 2020-01-16T00:14:08.589 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6650999999999994e-05 2020-01-16T00:14:08.590 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.590 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.590 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:08.590 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.265131+0000", 2020-01-16T00:14:08.590 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010495310000000001 2020-01-16T00:14:08.590 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.590 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.591 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.591 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.275626+0000", 2020-01-16T00:14:08.591 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9536000000000004e-05 2020-01-16T00:14:08.591 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.591 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.591 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:08.591 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.275726+0000", 2020-01-16T00:14:08.591 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023719489999999999 2020-01-16T00:14:08.592 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.592 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.592 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.592 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.278098+0000", 2020-01-16T00:14:08.592 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3718000000000001e-05 2020-01-16T00:14:08.592 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.592 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.592 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:08.593 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.278171+0000", 2020-01-16T00:14:08.593 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5859999999999994e-05 2020-01-16T00:14:08.593 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.593 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.593 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:08.593 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.278257+0000", 2020-01-16T00:14:08.593 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014731833999999999 2020-01-16T00:14:08.594 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.594 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:08.594 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.594 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.594 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.594 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3627 3.a 3:5069665e:::smithi12313934-2:head [write 1749580~611149] snapc 0=[] ondisk+write+known_if_redirected e48)", 2020-01-16T00:14:08.594 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:39.462779+0000", 2020-01-16T00:14:08.594 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 28.980877369000002, 2020-01-16T00:14:08.595 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020946985000000001, 2020-01-16T00:14:08.595 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:08.595 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:08.595 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:08.595 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:08.595 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:08.595 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3627 2020-01-16T00:14:08.595 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.596 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:08.596 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.596 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:08.596 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.462779+0000", 2020-01-16T00:14:08.596 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:08.596 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.596 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.597 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:08.597 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.462779+0000", 2020-01-16T00:14:08.597 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:08.597 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.597 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.597 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:08.597 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.462777+0000", 2020-01-16T00:14:08.597 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00064136 2020-01-16T00:14:08.598 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.598 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.598 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:08.598 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.463418+0000", 2020-01-16T00:14:08.598 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4607e-05 2020-01-16T00:14:08.598 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.598 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.598 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:08.599 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.463443+0000", 2020-01-16T00:14:08.599 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0346999999999999e-05 2020-01-16T00:14:08.599 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.599 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.599 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:08.599 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.463453+0000", 2020-01-16T00:14:08.599 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000100988 2020-01-16T00:14:08.599 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.600 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.600 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.600 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.463554+0000", 2020-01-16T00:14:08.600 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023059280000000001 2020-01-16T00:14:08.600 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.600 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.600 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:08.600 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.465860+0000", 2020-01-16T00:14:08.601 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011316961 2020-01-16T00:14:08.601 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.601 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.601 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.601 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.477177+0000", 2020-01-16T00:14:08.601 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000132402 2020-01-16T00:14:08.601 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.601 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.602 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:08.602 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.477309+0000", 2020-01-16T00:14:08.602 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000223606 2020-01-16T00:14:08.602 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.602 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.602 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:08.602 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.477533+0000", 2020-01-16T00:14:08.603 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0057105209999999997 2020-01-16T00:14:08.603 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.603 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.603 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:08.603 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483244+0000", 2020-01-16T00:14:08.603 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000193031 2020-01-16T00:14:08.603 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.603 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.604 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:08.604 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483437+0000", 2020-01-16T00:14:08.604 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0902999999999995e-05 2020-01-16T00:14:08.604 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.604 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.604 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:08.604 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483507+0000", 2020-01-16T00:14:08.604 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000129412 2020-01-16T00:14:08.605 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.605 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.605 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:08.605 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483637+0000", 2020-01-16T00:14:08.605 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.9345999999999995e-05 2020-01-16T00:14:08.605 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.605 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.605 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:08.606 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483726+0000", 2020-01-16T00:14:08.606 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020946985000000001 2020-01-16T00:14:08.606 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.606 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:08.606 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.606 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.606 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.606 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3628 3.a 3:5069665e:::smithi12313934-2:head [write 2853383~452450] snapc 0=[] ondisk+write+known_if_redirected e48)", 2020-01-16T00:14:08.607 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:39.468178+0000", 2020-01-16T00:14:08.607 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 28.97547866, 2020-01-16T00:14:08.607 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015705872999999999, 2020-01-16T00:14:08.607 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:08.607 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:08.607 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:08.607 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:08.607 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:08.608 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3628 2020-01-16T00:14:08.608 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.608 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:08.608 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.608 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:08.608 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468178+0000", 2020-01-16T00:14:08.608 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:08.609 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.609 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.609 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:08.609 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468178+0000", 2020-01-16T00:14:08.609 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:08.609 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.609 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.610 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:08.610 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468175+0000", 2020-01-16T00:14:08.610 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00065948000000000005 2020-01-16T00:14:08.610 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.610 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.610 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:08.610 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468834+0000", 2020-01-16T00:14:08.611 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3557000000000002e-05 2020-01-16T00:14:08.611 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.611 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.611 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:08.611 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468858+0000", 2020-01-16T00:14:08.611 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2931e-05 2020-01-16T00:14:08.611 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.611 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.612 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:08.612 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468871+0000", 2020-01-16T00:14:08.612 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000123113 2020-01-16T00:14:08.612 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.612 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.612 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.612 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468994+0000", 2020-01-16T00:14:08.613 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017615999999999999 2020-01-16T00:14:08.613 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.613 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.613 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:08.613 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.469170+0000", 2020-01-16T00:14:08.613 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010336908000000001 2020-01-16T00:14:08.613 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.613 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.614 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.614 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.479507+0000", 2020-01-16T00:14:08.614 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015856400000000001 2020-01-16T00:14:08.614 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.614 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.614 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:08.614 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.479665+0000", 2020-01-16T00:14:08.614 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030344600000000002 2020-01-16T00:14:08.615 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.615 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.615 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:08.615 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.479969+0000", 2020-01-16T00:14:08.615 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0037935389999999999 2020-01-16T00:14:08.615 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.615 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.615 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:08.616 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483762+0000", 2020-01-16T00:14:08.616 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3170999999999997e-05 2020-01-16T00:14:08.616 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.616 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.616 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:08.616 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483806+0000", 2020-01-16T00:14:08.616 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3821000000000001e-05 2020-01-16T00:14:08.616 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.617 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.617 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:08.617 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483829+0000", 2020-01-16T00:14:08.617 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0076000000000001e-05 2020-01-16T00:14:08.617 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.617 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.617 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:08.618 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483859+0000", 2020-01-16T00:14:08.618 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.438e-05 2020-01-16T00:14:08.618 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.618 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.618 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:08.618 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483884+0000", 2020-01-16T00:14:08.618 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015705872999999999 2020-01-16T00:14:08.618 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.619 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:08.619 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.619 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.619 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.619 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3629 3.a 3:5069665e:::smithi12313934-2:head [setxattr _header (55) in=7b,truncate 3529536] snapc 0=[] ondisk+write+known_if_redirected e48)", 2020-01-16T00:14:08.619 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:39.468888+0000", 2020-01-16T00:14:08.619 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 28.974769039000002, 2020-01-16T00:14:08.619 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015108511, 2020-01-16T00:14:08.620 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:08.620 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:08.620 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:08.620 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:08.620 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:08.620 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3629 2020-01-16T00:14:08.620 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.620 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:08.621 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.621 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:08.621 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468888+0000", 2020-01-16T00:14:08.621 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:08.621 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.621 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.621 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:08.621 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468888+0000", 2020-01-16T00:14:08.622 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:08.622 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.622 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.622 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:08.622 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468886+0000", 2020-01-16T00:14:08.622 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5599999999999996e-06 2020-01-16T00:14:08.622 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.622 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.623 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:08.623 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468894+0000", 2020-01-16T00:14:08.623 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5279999999999998e-06 2020-01-16T00:14:08.623 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.623 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.623 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:08.623 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468901+0000", 2020-01-16T00:14:08.623 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2290000000000002e-06 2020-01-16T00:14:08.624 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.624 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.624 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:08.624 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468907+0000", 2020-01-16T00:14:08.624 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030197500000000001 2020-01-16T00:14:08.624 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.624 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.625 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.625 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.469209+0000", 2020-01-16T00:14:08.625 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6659999999999997e-05 2020-01-16T00:14:08.625 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.626 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.626 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:08.626 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.469255+0000", 2020-01-16T00:14:08.626 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013117904999999999 2020-01-16T00:14:08.627 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.627 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.627 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.627 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.482373+0000", 2020-01-16T00:14:08.627 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1260000000000006e-05 2020-01-16T00:14:08.627 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.627 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.627 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:08.628 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.482434+0000", 2020-01-16T00:14:08.628 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014433 2020-01-16T00:14:08.628 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.628 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.628 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:08.628 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.482579+0000", 2020-01-16T00:14:08.628 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012198160000000001 2020-01-16T00:14:08.629 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.629 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.629 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:08.629 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483799+0000", 2020-01-16T00:14:08.629 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.188e-05 2020-01-16T00:14:08.629 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.629 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.629 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:08.630 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483810+0000", 2020-01-16T00:14:08.630 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000114863 2020-01-16T00:14:08.630 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.630 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.630 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:08.630 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483925+0000", 2020-01-16T00:14:08.630 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0861000000000001e-05 2020-01-16T00:14:08.630 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.631 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.631 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:08.631 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483956+0000", 2020-01-16T00:14:08.631 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9894999999999997e-05 2020-01-16T00:14:08.631 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.631 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.631 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:08.631 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483996+0000", 2020-01-16T00:14:08.632 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015108511 2020-01-16T00:14:08.632 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.632 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:08.632 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.632 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.632 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.632 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3630 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e48)", 2020-01-16T00:14:08.633 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:39.468967+0000", 2020-01-16T00:14:08.633 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 28.974689905000002, 2020-01-16T00:14:08.633 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015160579, 2020-01-16T00:14:08.633 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:08.633 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:08.633 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:08.633 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:08.633 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:08.634 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3630 2020-01-16T00:14:08.634 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.634 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:08.634 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.634 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:08.634 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468967+0000", 2020-01-16T00:14:08.634 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:08.634 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.635 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.635 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:08.635 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468967+0000", 2020-01-16T00:14:08.635 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:08.635 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.635 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.635 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:08.635 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468966+0000", 2020-01-16T00:14:08.636 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3430000000000003e-06 2020-01-16T00:14:08.636 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.636 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.636 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:08.636 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468971+0000", 2020-01-16T00:14:08.636 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6577000000000001e-05 2020-01-16T00:14:08.636 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.636 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.637 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:08.637 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468988+0000", 2020-01-16T00:14:08.637 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.378e-06 2020-01-16T00:14:08.637 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.637 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.637 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:08.637 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468993+0000", 2020-01-16T00:14:08.638 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000320601 2020-01-16T00:14:08.638 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.638 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.638 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.638 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.469314+0000", 2020-01-16T00:14:08.638 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013833699999999999 2020-01-16T00:14:08.638 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.638 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.639 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:08.639 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.469452+0000", 2020-01-16T00:14:08.639 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013447838 2020-01-16T00:14:08.639 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.639 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.639 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.639 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.482900+0000", 2020-01-16T00:14:08.639 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028006899999999998 2020-01-16T00:14:08.640 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.640 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.640 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:08.640 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483180+0000", 2020-01-16T00:14:08.640 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000833594 2020-01-16T00:14:08.640 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.640 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.640 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.641 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.484013+0000", 2020-01-16T00:14:08.641 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2601e-05 2020-01-16T00:14:08.641 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.641 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.641 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:08.641 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.484066+0000", 2020-01-16T00:14:08.641 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1303999999999994e-05 2020-01-16T00:14:08.641 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.642 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.642 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:08.642 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.484127+0000", 2020-01-16T00:14:08.642 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015160579 2020-01-16T00:14:08.642 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.642 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:08.642 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.642 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.643 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.643 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4047 3.a 3:5069665e:::smithi12313934-2:head [write 1935828~635840] snapc 0=[] ondisk+write+known_if_redirected e52)", 2020-01-16T00:14:08.643 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:50.302087+0000", 2020-01-16T00:14:08.643 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 18.141569634, 2020-01-16T00:14:08.643 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014518982, 2020-01-16T00:14:08.643 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:08.643 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:08.643 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:08.644 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:08.644 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:08.644 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4047 2020-01-16T00:14:08.644 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.644 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:08.644 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.644 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:08.645 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302087+0000", 2020-01-16T00:14:08.645 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:08.645 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.645 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.645 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:08.645 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302087+0000", 2020-01-16T00:14:08.645 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:08.645 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.646 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.646 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:08.646 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302085+0000", 2020-01-16T00:14:08.646 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00057147600000000002 2020-01-16T00:14:08.646 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.646 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.646 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:08.646 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302657+0000", 2020-01-16T00:14:08.647 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6427999999999999e-05 2020-01-16T00:14:08.647 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.647 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.647 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:08.647 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302673+0000", 2020-01-16T00:14:08.647 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.127e-06 2020-01-16T00:14:08.647 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.647 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.648 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:08.648 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302680+0000", 2020-01-16T00:14:08.648 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6378000000000006e-05 2020-01-16T00:14:08.648 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.648 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.649 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.649 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302756+0000", 2020-01-16T00:14:08.649 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00039898000000000001 2020-01-16T00:14:08.650 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.650 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.650 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:08.650 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.303155+0000", 2020-01-16T00:14:08.650 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0079666870000000001 2020-01-16T00:14:08.651 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.651 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.651 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.651 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.311122+0000", 2020-01-16T00:14:08.651 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019101400000000001 2020-01-16T00:14:08.651 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.652 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.652 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:08.652 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.311313+0000", 2020-01-16T00:14:08.652 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033332499999999998 2020-01-16T00:14:08.652 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.652 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.652 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:08.653 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.311646+0000", 2020-01-16T00:14:08.653 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0029841440000000002 2020-01-16T00:14:08.653 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.653 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.653 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:08.653 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.314630+0000", 2020-01-16T00:14:08.654 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00027996800000000001 2020-01-16T00:14:08.654 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.654 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.654 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:08.654 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.314910+0000", 2020-01-16T00:14:08.654 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0015637159999999999 2020-01-16T00:14:08.654 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.654 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.655 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:08.655 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.316474+0000", 2020-01-16T00:14:08.655 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5986000000000001e-05 2020-01-16T00:14:08.655 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.655 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.655 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:08.655 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.316550+0000", 2020-01-16T00:14:08.655 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6258000000000002e-05 2020-01-16T00:14:08.656 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.656 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.656 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:08.656 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.316606+0000", 2020-01-16T00:14:08.656 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014518982 2020-01-16T00:14:08.656 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.656 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:08.656 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.657 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.657 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.657 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4048 3.a 3:5069665e:::smithi12313934-2:head [setxattr _header (55) in=7b,truncate 2588377] snapc 0=[] ondisk+write+known_if_redirected e52)", 2020-01-16T00:14:08.657 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:50.302688+0000", 2020-01-16T00:14:08.657 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 18.140968522000001, 2020-01-16T00:14:08.657 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014049927, 2020-01-16T00:14:08.657 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:08.658 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:08.658 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:08.658 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:08.658 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:08.658 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4048 2020-01-16T00:14:08.658 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.658 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:08.659 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.659 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:08.659 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302688+0000", 2020-01-16T00:14:08.659 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:08.659 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.659 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.659 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:08.659 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302688+0000", 2020-01-16T00:14:08.660 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:08.660 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.660 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.660 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:08.660 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302687+0000", 2020-01-16T00:14:08.660 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6e-06 2020-01-16T00:14:08.660 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.661 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.661 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:08.661 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302692+0000", 2020-01-16T00:14:08.661 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1999999999999996e-06 2020-01-16T00:14:08.661 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.661 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.661 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:08.661 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302696+0000", 2020-01-16T00:14:08.662 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9330000000000001e-06 2020-01-16T00:14:08.662 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.662 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.662 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:08.662 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302699+0000", 2020-01-16T00:14:08.662 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00049592099999999997 2020-01-16T00:14:08.662 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.663 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.663 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.663 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.303195+0000", 2020-01-16T00:14:08.663 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0647000000000003e-05 2020-01-16T00:14:08.663 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.663 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.663 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:08.664 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.303266+0000", 2020-01-16T00:14:08.664 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010088420000000001 2020-01-16T00:14:08.664 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.664 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.664 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.664 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.313354+0000", 2020-01-16T00:14:08.664 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000113136 2020-01-16T00:14:08.665 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.665 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.665 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:08.665 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.313467+0000", 2020-01-16T00:14:08.665 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012389000000000001 2020-01-16T00:14:08.665 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.665 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.665 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:08.666 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.313591+0000", 2020-01-16T00:14:08.666 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014897409999999999 2020-01-16T00:14:08.666 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.666 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.666 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:08.667 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.315081+0000", 2020-01-16T00:14:08.667 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00023842700000000001 2020-01-16T00:14:08.667 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.667 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.667 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:08.667 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.315319+0000", 2020-01-16T00:14:08.668 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012958449999999999 2020-01-16T00:14:08.668 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.668 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.668 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:08.668 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.316615+0000", 2020-01-16T00:14:08.668 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4045000000000002e-05 2020-01-16T00:14:08.668 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.668 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.669 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:08.669 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.316669+0000", 2020-01-16T00:14:08.669 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8827999999999999e-05 2020-01-16T00:14:08.669 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.669 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.669 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:08.669 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.316738+0000", 2020-01-16T00:14:08.670 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014049927 2020-01-16T00:14:08.670 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.670 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:08.670 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.670 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.671 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.671 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4049 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e52)", 2020-01-16T00:14:08.671 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:50.302702+0000", 2020-01-16T00:14:08.671 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 18.140955000999998, 2020-01-16T00:14:08.672 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014215574999999999, 2020-01-16T00:14:08.672 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:08.672 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:08.672 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:08.672 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:08.672 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:08.672 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4049 2020-01-16T00:14:08.673 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.673 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:08.673 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.673 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:08.673 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302702+0000", 2020-01-16T00:14:08.673 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:08.673 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.674 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.674 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:08.674 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302702+0000", 2020-01-16T00:14:08.674 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:08.674 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.674 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.674 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:08.674 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302701+0000", 2020-01-16T00:14:08.675 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0020000000000001e-06 2020-01-16T00:14:08.675 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.675 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.675 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:08.675 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302703+0000", 2020-01-16T00:14:08.675 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7740000000000002e-06 2020-01-16T00:14:08.675 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.675 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.676 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:08.676 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302707+0000", 2020-01-16T00:14:08.676 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.512e-06 2020-01-16T00:14:08.676 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.676 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.676 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:08.676 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302711+0000", 2020-01-16T00:14:08.677 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00058642799999999995 2020-01-16T00:14:08.677 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.677 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.677 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.677 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.303297+0000", 2020-01-16T00:14:08.677 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5887000000000001e-05 2020-01-16T00:14:08.677 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.678 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.678 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:08.678 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.303363+0000", 2020-01-16T00:14:08.678 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010875398999999999 2020-01-16T00:14:08.678 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.678 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.678 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.678 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.314238+0000", 2020-01-16T00:14:08.679 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000187014 2020-01-16T00:14:08.679 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.679 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.679 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:08.679 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.314425+0000", 2020-01-16T00:14:08.679 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023405000000000001 2020-01-16T00:14:08.679 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.680 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.680 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.680 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.316766+0000", 2020-01-16T00:14:08.680 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0636000000000006e-05 2020-01-16T00:14:08.680 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.680 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.680 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:08.681 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.316836+0000", 2020-01-16T00:14:08.681 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0747999999999993e-05 2020-01-16T00:14:08.681 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.681 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.681 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:08.681 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.316917+0000", 2020-01-16T00:14:08.681 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014215574999999999 2020-01-16T00:14:08.681 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.682 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:08.682 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.682 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.682 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.682 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4600 3.a 3:5069665e:::smithi12313934-2:head [write 1791765~569861] snapc 0=[] ondisk+write+known_if_redirected e56)", 2020-01-16T00:14:08.682 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:03.059744+0000", 2020-01-16T00:14:08.682 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.3839124170000003, 2020-01-16T00:14:08.683 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017075275000000001, 2020-01-16T00:14:08.683 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:08.683 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:08.683 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:08.683 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:08.683 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:08.683 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4600 2020-01-16T00:14:08.684 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.684 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:08.684 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.684 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:08.684 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.059744+0000", 2020-01-16T00:14:08.684 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:08.684 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.684 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.685 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:08.685 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.059744+0000", 2020-01-16T00:14:08.685 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999843 2020-01-16T00:14:08.685 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.685 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.685 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:08.685 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.059728+0000", 2020-01-16T00:14:08.686 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00054167699999999996 2020-01-16T00:14:08.686 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.686 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.686 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:08.686 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.060270+0000", 2020-01-16T00:14:08.686 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6693999999999998e-05 2020-01-16T00:14:08.686 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.687 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.687 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:08.687 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.060307+0000", 2020-01-16T00:14:08.687 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8753000000000002e-05 2020-01-16T00:14:08.687 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.687 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.687 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:08.687 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.060326+0000", 2020-01-16T00:14:08.688 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000114177 2020-01-16T00:14:08.688 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.688 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.688 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.688 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.060440+0000", 2020-01-16T00:14:08.688 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001642382 2020-01-16T00:14:08.688 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.688 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.689 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:08.689 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.062082+0000", 2020-01-16T00:14:08.689 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0093492149999999993 2020-01-16T00:14:08.689 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.689 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.689 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:08.689 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.071431+0000", 2020-01-16T00:14:08.690 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013135600000000001 2020-01-16T00:14:08.690 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.690 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.690 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:08.690 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.071563+0000", 2020-01-16T00:14:08.690 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00032728299999999999 2020-01-16T00:14:08.690 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.691 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.691 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:08.691 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.071890+0000", 2020-01-16T00:14:08.691 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0043339759999999998 2020-01-16T00:14:08.691 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.691 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.692 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:08.692 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.076224+0000", 2020-01-16T00:14:08.692 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000343462 2020-01-16T00:14:08.692 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.692 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.692 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:08.692 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.076567+0000", 2020-01-16T00:14:08.692 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1668000000000006e-05 2020-01-16T00:14:08.693 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.693 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.693 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:08.693 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.076639+0000", 2020-01-16T00:14:08.693 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8572000000000004e-05 2020-01-16T00:14:08.693 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.693 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.694 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:08.694 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.076738+0000", 2020-01-16T00:14:08.694 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1816000000000004e-05 2020-01-16T00:14:08.694 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:08.694 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:08.694 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:08.694 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.076819+0000", 2020-01-16T00:14:08.694 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017075275000000001 2020-01-16T00:14:08.695 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.695 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:08.695 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.695 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:08.695 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:08.695 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:08.697 INFO:tasks.rados.rados.0.smithi123.stdout:1697: finishing write tid 4 to smithi12313934-34 2020-01-16T00:14:08.697 INFO:tasks.rados.rados.0.smithi123.stdout:1697: finishing write tid 5 to smithi12313934-34 2020-01-16T00:14:08.697 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 34 v 86 (ObjNum 609 snap 0 seq_num 609) dirty exists 2020-01-16T00:14:08.697 INFO:tasks.rados.rados.0.smithi123.stdout:1697: left oid 34 (ObjNum 609 snap 0 seq_num 609) 2020-01-16T00:14:08.697 INFO:tasks.rados.rados.0.smithi123.stdout:1699: expect (ObjNum 540 snap 0 seq_num 540) 2020-01-16T00:14:08.698 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-01-16T00:14:08.699 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:08.694+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:08.798 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:08.796+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:08.799 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:08.800 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:14:08.800 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:14:08.800 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:08.809 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-01-16T00:14:08.822 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2020-01-16T00:14:08.890 INFO:tasks.rados.rados.0.smithi123.stdout:1700: finishing write tid 1 to smithi12313934-42 2020-01-16T00:14:08.890 INFO:tasks.rados.rados.0.smithi123.stdout:1700: finishing write tid 2 to smithi12313934-42 2020-01-16T00:14:08.890 INFO:tasks.rados.rados.0.smithi123.stdout:1701: finishing write tid 1 to smithi12313934-19 2020-01-16T00:14:08.891 INFO:tasks.rados.rados.0.smithi123.stdout:1700: finishing write tid 4 to smithi12313934-42 2020-01-16T00:14:08.891 INFO:tasks.rados.rados.0.smithi123.stdout:1700: finishing write tid 5 to smithi12313934-42 2020-01-16T00:14:08.891 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 42 v 208 (ObjNum 610 snap 0 seq_num 610) dirty exists 2020-01-16T00:14:08.891 INFO:tasks.rados.rados.0.smithi123.stdout:1700: left oid 42 (ObjNum 610 snap 0 seq_num 610) 2020-01-16T00:14:08.891 INFO:tasks.rados.rados.0.smithi123.stdout:1704: expect (ObjNum 502 snap 0 seq_num 502) 2020-01-16T00:14:08.899 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:08.896+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:08.925 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:08.926 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:14:08.926 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:14:08.926 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:14:08.926 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:08.935 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-01-16T00:14:08.963 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-01-16T00:14:09.023 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:09.022+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:09.033 INFO:tasks.rados.rados.0.smithi123.stdout:1705: expect (ObjNum 517 snap 0 seq_num 517) 2020-01-16T00:14:09.084 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:09.084 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:14:09.084 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:14:09.085 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:14:09.085 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.085 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:939 3.4 3:2bdd9e88:::smithi12313934-4:head [write 1681102~474010] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:14:09.085 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.908392+0000", 2020-01-16T00:14:09.085 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 91.171370351999997, 2020-01-16T00:14:09.085 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019294697, 2020-01-16T00:14:09.085 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:09.086 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:09.086 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:09.086 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:09.086 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:09.086 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 939 2020-01-16T00:14:09.086 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.086 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:09.086 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.087 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:09.087 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908392+0000", 2020-01-16T00:14:09.087 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:09.087 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.087 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.087 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:09.087 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908392+0000", 2020-01-16T00:14:09.087 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:09.088 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.088 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.089 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:09.089 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908390+0000", 2020-01-16T00:14:09.089 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015440399999999999 2020-01-16T00:14:09.089 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.089 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.089 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:09.090 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908545+0000", 2020-01-16T00:14:09.090 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3926e-05 2020-01-16T00:14:09.090 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.090 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.090 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:09.090 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908559+0000", 2020-01-16T00:14:09.091 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2200000000000003e-06 2020-01-16T00:14:09.091 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.091 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.091 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:09.091 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908566+0000", 2020-01-16T00:14:09.091 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020594799999999998 2020-01-16T00:14:09.091 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.091 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.092 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.092 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910625+0000", 2020-01-16T00:14:09.092 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118672 2020-01-16T00:14:09.092 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.093 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.093 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:09.093 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910744+0000", 2020-01-16T00:14:09.093 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031940969999999999 2020-01-16T00:14:09.093 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.093 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.093 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.093 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.913938+0000", 2020-01-16T00:14:09.094 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000309621 2020-01-16T00:14:09.094 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.094 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.094 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:09.094 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.914248+0000", 2020-01-16T00:14:09.094 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.008818579 2020-01-16T00:14:09.094 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.094 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.095 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.095 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.923066+0000", 2020-01-16T00:14:09.095 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000108989 2020-01-16T00:14:09.095 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.095 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.095 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:09.095 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.923175+0000", 2020-01-16T00:14:09.095 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019128299999999999 2020-01-16T00:14:09.096 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.096 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.096 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:14:09.096 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.923367+0000", 2020-01-16T00:14:09.096 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0032083760000000002 2020-01-16T00:14:09.096 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.096 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.096 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:09.097 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926575+0000", 2020-01-16T00:14:09.097 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2475999999999999e-05 2020-01-16T00:14:09.097 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.097 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.097 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:09.097 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926607+0000", 2020-01-16T00:14:09.097 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00099399800000000006 2020-01-16T00:14:09.098 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.098 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.098 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:09.098 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927601+0000", 2020-01-16T00:14:09.098 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0958999999999999e-05 2020-01-16T00:14:09.098 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.099 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.099 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:09.099 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927642+0000", 2020-01-16T00:14:09.099 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4747999999999999e-05 2020-01-16T00:14:09.099 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.099 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.099 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:09.099 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927687+0000", 2020-01-16T00:14:09.100 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019294697 2020-01-16T00:14:09.100 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.100 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:09.100 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.100 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.100 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.100 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:940 3.4 3:2bdd9e88:::smithi12313934-4:head [write 2687912~589420] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:14:09.100 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.915144+0000", 2020-01-16T00:14:09.101 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 91.164618984000001, 2020-01-16T00:14:09.101 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016092538999999999, 2020-01-16T00:14:09.101 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:09.101 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:09.101 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:09.101 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:09.101 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:09.101 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 940 2020-01-16T00:14:09.102 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.102 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:09.102 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.102 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:09.102 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915144+0000", 2020-01-16T00:14:09.103 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:09.103 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.103 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.103 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:09.103 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915144+0000", 2020-01-16T00:14:09.103 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:09.103 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.103 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.104 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:09.104 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915141+0000", 2020-01-16T00:14:09.104 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016366300000000001 2020-01-16T00:14:09.104 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.104 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.104 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:09.104 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915305+0000", 2020-01-16T00:14:09.104 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4847e-05 2020-01-16T00:14:09.105 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.105 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.105 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:09.105 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915320+0000", 2020-01-16T00:14:09.105 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2599999999999999e-06 2020-01-16T00:14:09.105 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.105 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.105 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:09.106 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915327+0000", 2020-01-16T00:14:09.106 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4282000000000005e-05 2020-01-16T00:14:09.106 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.106 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.106 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.106 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915401+0000", 2020-01-16T00:14:09.106 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4554000000000006e-05 2020-01-16T00:14:09.106 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.107 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.107 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:09.107 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915466+0000", 2020-01-16T00:14:09.107 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0090821900000000004 2020-01-16T00:14:09.107 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.107 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.107 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.107 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.924548+0000", 2020-01-16T00:14:09.108 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000114696 2020-01-16T00:14:09.108 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.108 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.108 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:09.108 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.924663+0000", 2020-01-16T00:14:09.108 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000168651 2020-01-16T00:14:09.109 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.109 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.109 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:14:09.109 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.924832+0000", 2020-01-16T00:14:09.109 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018253430000000001 2020-01-16T00:14:09.109 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.109 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.109 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:09.110 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926657+0000", 2020-01-16T00:14:09.110 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9957999999999999e-05 2020-01-16T00:14:09.110 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.110 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.110 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:09.110 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926697+0000", 2020-01-16T00:14:09.110 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0044685250000000001 2020-01-16T00:14:09.110 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.111 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.111 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:09.111 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931165+0000", 2020-01-16T00:14:09.111 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9211000000000001e-05 2020-01-16T00:14:09.111 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.111 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.111 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:09.112 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931205+0000", 2020-01-16T00:14:09.112 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1699999999999998e-05 2020-01-16T00:14:09.112 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.112 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.112 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:09.112 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931236+0000", 2020-01-16T00:14:09.113 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016092538999999999 2020-01-16T00:14:09.113 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.113 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:09.113 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.113 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.113 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.113 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:941 3.4 3:2bdd9e88:::smithi12313934-4:head [setxattr _header (55) in=7b,truncate 3277332] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:14:09.114 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.915341+0000", 2020-01-16T00:14:09.114 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 91.164422165999994, 2020-01-16T00:14:09.114 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016603769000000001, 2020-01-16T00:14:09.114 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:09.114 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:09.114 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:09.114 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:09.114 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:09.115 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 941 2020-01-16T00:14:09.115 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.115 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:09.115 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.115 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:09.115 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915341+0000", 2020-01-16T00:14:09.116 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:09.116 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.116 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.116 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:09.116 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915341+0000", 2020-01-16T00:14:09.116 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:09.116 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.117 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.117 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:09.117 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915340+0000", 2020-01-16T00:14:09.117 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.896e-06 2020-01-16T00:14:09.117 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.117 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.117 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:09.118 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915346+0000", 2020-01-16T00:14:09.118 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2730000000000002e-06 2020-01-16T00:14:09.118 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.118 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.118 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:09.118 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915354+0000", 2020-01-16T00:14:09.118 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0616e-05 2020-01-16T00:14:09.118 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.119 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.119 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:09.119 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915374+0000", 2020-01-16T00:14:09.119 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000133487 2020-01-16T00:14:09.119 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.119 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.119 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.120 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915508+0000", 2020-01-16T00:14:09.120 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8971000000000003e-05 2020-01-16T00:14:09.120 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.120 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.120 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:09.121 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915557+0000", 2020-01-16T00:14:09.121 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010138787999999999 2020-01-16T00:14:09.121 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.122 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.122 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.123 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.925696+0000", 2020-01-16T00:14:09.123 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012825400000000001 2020-01-16T00:14:09.123 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.123 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.123 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:09.124 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.925824+0000", 2020-01-16T00:14:09.124 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017706500000000001 2020-01-16T00:14:09.124 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.124 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.124 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:14:09.124 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926001+0000", 2020-01-16T00:14:09.124 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00079988900000000003 2020-01-16T00:14:09.124 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.125 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.125 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:09.125 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926801+0000", 2020-01-16T00:14:09.125 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119324 2020-01-16T00:14:09.125 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.125 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.125 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:09.125 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926920+0000", 2020-01-16T00:14:09.126 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048879910000000004 2020-01-16T00:14:09.126 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.126 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.126 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:09.126 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931808+0000", 2020-01-16T00:14:09.126 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6999999999999997e-05 2020-01-16T00:14:09.126 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.126 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.127 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:09.127 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931855+0000", 2020-01-16T00:14:09.127 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9092999999999996e-05 2020-01-16T00:14:09.127 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.127 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.127 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:09.127 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931944+0000", 2020-01-16T00:14:09.127 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016603769000000001 2020-01-16T00:14:09.128 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.128 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:09.128 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.128 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.128 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.128 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:942 3.4 3:2bdd9e88:::smithi12313934-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:14:09.128 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.915389+0000", 2020-01-16T00:14:09.128 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 91.164373800000007, 2020-01-16T00:14:09.129 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016820563, 2020-01-16T00:14:09.129 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:09.129 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:09.129 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:09.129 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:09.129 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:09.130 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 942 2020-01-16T00:14:09.130 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.130 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:09.130 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.130 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:09.130 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915389+0000", 2020-01-16T00:14:09.130 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:09.130 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.131 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.131 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:09.131 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915389+0000", 2020-01-16T00:14:09.131 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:09.131 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.131 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.131 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:09.131 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915388+0000", 2020-01-16T00:14:09.132 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.683e-06 2020-01-16T00:14:09.132 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.132 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.132 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:09.132 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915391+0000", 2020-01-16T00:14:09.132 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4839999999999998e-06 2020-01-16T00:14:09.132 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.132 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.133 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:09.133 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915399+0000", 2020-01-16T00:14:09.133 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2259999999999999e-06 2020-01-16T00:14:09.133 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.133 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.133 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:09.133 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915403+0000", 2020-01-16T00:14:09.134 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00031301199999999999 2020-01-16T00:14:09.134 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.134 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.134 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.134 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915716+0000", 2020-01-16T00:14:09.134 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2079999999999998e-05 2020-01-16T00:14:09.135 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.135 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.135 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:09.135 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915748+0000", 2020-01-16T00:14:09.135 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010547076000000001 2020-01-16T00:14:09.135 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.135 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.136 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.136 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926295+0000", 2020-01-16T00:14:09.136 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8040000000000005e-05 2020-01-16T00:14:09.136 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.136 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.136 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:09.136 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926373+0000", 2020-01-16T00:14:09.137 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056061990000000001 2020-01-16T00:14:09.137 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.137 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.137 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.137 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931979+0000", 2020-01-16T00:14:09.137 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011540399999999999 2020-01-16T00:14:09.137 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.138 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.138 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:09.138 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932095+0000", 2020-01-16T00:14:09.138 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011482899999999999 2020-01-16T00:14:09.138 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.138 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.138 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:09.138 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932210+0000", 2020-01-16T00:14:09.139 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016820563 2020-01-16T00:14:09.139 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.139 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:09.139 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.139 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.139 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.140 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:957 3.0 3:00c146d5:::smithi12313934-18:head [write 510946~632952] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:14:09.140 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.033290+0000", 2020-01-16T00:14:09.140 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 91.046472962999999, 2020-01-16T00:14:09.140 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016764102999999999, 2020-01-16T00:14:09.140 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:09.140 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:09.140 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:09.141 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:09.141 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:09.141 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 957 2020-01-16T00:14:09.141 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.141 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:09.141 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.141 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:09.141 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033290+0000", 2020-01-16T00:14:09.142 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:09.142 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.143 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.143 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:09.143 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033290+0000", 2020-01-16T00:14:09.143 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999952 2020-01-16T00:14:09.143 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.143 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.143 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:09.143 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033285+0000", 2020-01-16T00:14:09.144 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000405917 2020-01-16T00:14:09.144 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.144 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.144 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:09.144 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033691+0000", 2020-01-16T00:14:09.144 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6585999999999997e-05 2020-01-16T00:14:09.144 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.145 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.145 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:09.145 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033738+0000", 2020-01-16T00:14:09.145 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8374000000000001e-05 2020-01-16T00:14:09.145 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.146 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.146 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:09.146 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033756+0000", 2020-01-16T00:14:09.146 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013002900000000001 2020-01-16T00:14:09.146 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.146 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.146 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.146 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033886+0000", 2020-01-16T00:14:09.147 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035882600000000002 2020-01-16T00:14:09.147 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.147 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.147 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:09.147 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.034245+0000", 2020-01-16T00:14:09.147 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0072798489999999997 2020-01-16T00:14:09.147 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.148 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.148 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.148 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.041525+0000", 2020-01-16T00:14:09.148 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000140488 2020-01-16T00:14:09.148 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.148 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.148 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:09.148 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.041665+0000", 2020-01-16T00:14:09.149 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000233766 2020-01-16T00:14:09.149 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.149 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.149 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:09.149 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.041899+0000", 2020-01-16T00:14:09.149 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0035766539999999999 2020-01-16T00:14:09.149 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.149 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.150 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:09.150 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.045475+0000", 2020-01-16T00:14:09.150 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013333150000000001 2020-01-16T00:14:09.150 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.150 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.150 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:09.150 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.046809+0000", 2020-01-16T00:14:09.151 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031447860000000001 2020-01-16T00:14:09.151 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.151 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.151 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:09.151 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.049954+0000", 2020-01-16T00:14:09.151 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2318000000000003e-05 2020-01-16T00:14:09.151 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.151 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.152 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:09.152 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.050006+0000", 2020-01-16T00:14:09.152 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7972000000000001e-05 2020-01-16T00:14:09.152 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.152 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.152 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:09.152 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.050054+0000", 2020-01-16T00:14:09.152 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016764102999999999 2020-01-16T00:14:09.153 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.153 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:09.153 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.153 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.153 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.153 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1639 3.0 3:00c146d5:::smithi12313934-18:head [write 423576~401598] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:14:09.153 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.774317+0000", 2020-01-16T00:14:09.153 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 77.305446223999994, 2020-01-16T00:14:09.154 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018305219000000001, 2020-01-16T00:14:09.154 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:09.154 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:09.154 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:09.154 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:09.154 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:09.154 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1639 2020-01-16T00:14:09.155 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.155 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:09.155 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.155 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:09.156 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774317+0000", 2020-01-16T00:14:09.156 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:09.156 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.156 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.156 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:09.157 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774317+0000", 2020-01-16T00:14:09.157 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:09.157 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.157 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.157 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:09.157 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774313+0000", 2020-01-16T00:14:09.157 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000111173 2020-01-16T00:14:09.157 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.158 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.158 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:09.158 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774425+0000", 2020-01-16T00:14:09.158 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7598999999999999e-05 2020-01-16T00:14:09.158 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.158 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.158 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:09.158 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774442+0000", 2020-01-16T00:14:09.159 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0830000000000004e-06 2020-01-16T00:14:09.159 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.159 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.159 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:09.159 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774450+0000", 2020-01-16T00:14:09.159 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0017886309999999999 2020-01-16T00:14:09.159 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.159 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.160 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.160 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.776239+0000", 2020-01-16T00:14:09.160 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3789e-05 2020-01-16T00:14:09.160 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.160 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.160 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op must wait for map", 2020-01-16T00:14:09.160 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.776253+0000", 2020-01-16T00:14:09.160 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095730729999999997 2020-01-16T00:14:09.161 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.161 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.161 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.161 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.785826+0000", 2020-01-16T00:14:09.161 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014220300000000001 2020-01-16T00:14:09.161 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.161 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.162 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:09.162 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.785968+0000", 2020-01-16T00:14:09.162 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023618900000000001 2020-01-16T00:14:09.162 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.162 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.162 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:09.162 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.786204+0000", 2020-01-16T00:14:09.162 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0062297389999999998 2020-01-16T00:14:09.163 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.163 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.163 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:09.163 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792434+0000", 2020-01-16T00:14:09.163 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9635999999999998e-05 2020-01-16T00:14:09.163 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.163 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.163 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:09.164 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792473+0000", 2020-01-16T00:14:09.164 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6456000000000002e-05 2020-01-16T00:14:09.164 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.164 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.164 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:09.164 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792540+0000", 2020-01-16T00:14:09.164 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9873999999999998e-05 2020-01-16T00:14:09.164 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.165 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.165 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:09.165 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792580+0000", 2020-01-16T00:14:09.165 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1924e-05 2020-01-16T00:14:09.165 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.165 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.165 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:09.165 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792622+0000", 2020-01-16T00:14:09.166 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018305219000000001 2020-01-16T00:14:09.166 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.166 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:09.166 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.166 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.167 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.167 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1890 3.0 3:0b1f384f:::smithi12313934-33:head [write 2797915~596476] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:14:09.167 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.232534+0000", 2020-01-16T00:14:09.167 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 71.847228252999997, 2020-01-16T00:14:09.167 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016106137, 2020-01-16T00:14:09.167 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:09.167 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:09.167 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:09.168 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:09.168 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:09.168 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1890 2020-01-16T00:14:09.168 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.168 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:09.168 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.168 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:09.169 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232534+0000", 2020-01-16T00:14:09.169 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:09.169 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.169 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.169 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:09.169 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232534+0000", 2020-01-16T00:14:09.169 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:09.170 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.170 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.170 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:09.170 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232532+0000", 2020-01-16T00:14:09.170 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00024232100000000001 2020-01-16T00:14:09.170 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.170 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.171 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:09.171 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232774+0000", 2020-01-16T00:14:09.171 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4601e-05 2020-01-16T00:14:09.171 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.171 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.171 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:09.171 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232789+0000", 2020-01-16T00:14:09.171 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8410000000000006e-06 2020-01-16T00:14:09.172 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.172 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.172 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:09.172 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232798+0000", 2020-01-16T00:14:09.172 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9756999999999997e-05 2020-01-16T00:14:09.172 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.172 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.172 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.173 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232857+0000", 2020-01-16T00:14:09.173 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4678000000000007e-05 2020-01-16T00:14:09.173 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.173 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.173 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:09.173 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232922+0000", 2020-01-16T00:14:09.173 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033861020000000002 2020-01-16T00:14:09.173 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.174 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.174 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.174 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.236308+0000", 2020-01-16T00:14:09.174 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000568996 2020-01-16T00:14:09.174 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.174 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.174 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:09.174 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.236877+0000", 2020-01-16T00:14:09.175 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0085444240000000001 2020-01-16T00:14:09.175 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.175 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.175 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.175 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.245422+0000", 2020-01-16T00:14:09.175 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5595999999999998e-05 2020-01-16T00:14:09.175 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.176 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.176 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:09.176 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.245507+0000", 2020-01-16T00:14:09.176 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016984300000000001 2020-01-16T00:14:09.176 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.176 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.177 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:09.177 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.245677+0000", 2020-01-16T00:14:09.177 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001649495 2020-01-16T00:14:09.177 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.177 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.177 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:09.177 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247326+0000", 2020-01-16T00:14:09.177 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023308399999999999 2020-01-16T00:14:09.178 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.178 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.178 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:09.178 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247560+0000", 2020-01-16T00:14:09.178 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00099501800000000007 2020-01-16T00:14:09.178 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.178 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.179 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:09.179 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248555+0000", 2020-01-16T00:14:09.179 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5936999999999998e-05 2020-01-16T00:14:09.179 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.179 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.179 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:09.179 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248591+0000", 2020-01-16T00:14:09.179 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0108999999999997e-05 2020-01-16T00:14:09.180 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.180 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.180 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:09.180 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248641+0000", 2020-01-16T00:14:09.180 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016106137 2020-01-16T00:14:09.180 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.180 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:09.180 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.181 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.181 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.181 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1892 3.0 3:0b1f384f:::smithi12313934-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e31)", 2020-01-16T00:14:09.181 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.232856+0000", 2020-01-16T00:14:09.181 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 71.846907227000003, 2020-01-16T00:14:09.181 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016171544, 2020-01-16T00:14:09.181 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:09.182 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:09.182 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:09.182 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:09.182 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:09.182 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1892 2020-01-16T00:14:09.182 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.182 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:09.182 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.183 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:09.183 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232856+0000", 2020-01-16T00:14:09.183 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:09.183 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.183 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.183 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:09.183 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232856+0000", 2020-01-16T00:14:09.183 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:09.184 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.184 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.184 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:09.184 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232855+0000", 2020-01-16T00:14:09.184 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8640000000000004e-06 2020-01-16T00:14:09.184 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.184 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.184 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:09.185 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232860+0000", 2020-01-16T00:14:09.185 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0000000000000001e-05 2020-01-16T00:14:09.185 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.185 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.185 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:09.185 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232870+0000", 2020-01-16T00:14:09.185 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7420000000000003e-06 2020-01-16T00:14:09.186 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.186 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.186 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:09.186 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232875+0000", 2020-01-16T00:14:09.186 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017933200000000001 2020-01-16T00:14:09.186 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.186 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.187 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.187 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.233055+0000", 2020-01-16T00:14:09.187 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6269e-05 2020-01-16T00:14:09.187 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.187 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.187 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:09.187 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.233101+0000", 2020-01-16T00:14:09.188 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039780090000000002 2020-01-16T00:14:09.188 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.188 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.188 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.188 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.237079+0000", 2020-01-16T00:14:09.188 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3677000000000001e-05 2020-01-16T00:14:09.189 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.189 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.189 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:09.189 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.237133+0000", 2020-01-16T00:14:09.189 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010036671 2020-01-16T00:14:09.189 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.189 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.190 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.190 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247169+0000", 2020-01-16T00:14:09.190 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010280699999999999 2020-01-16T00:14:09.190 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.190 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.190 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:09.190 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247272+0000", 2020-01-16T00:14:09.190 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001579708 2020-01-16T00:14:09.191 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.191 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.191 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.191 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248852+0000", 2020-01-16T00:14:09.191 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9704999999999996e-05 2020-01-16T00:14:09.191 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.191 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.191 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:09.192 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248942+0000", 2020-01-16T00:14:09.192 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5557999999999996e-05 2020-01-16T00:14:09.192 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.192 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.192 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:09.192 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.249027+0000", 2020-01-16T00:14:09.193 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016171544 2020-01-16T00:14:09.193 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.193 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:09.193 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.193 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.194 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.194 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1950 3.0 3:0b1f384f:::smithi12313934-33:head [write 1889359~541737] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-01-16T00:14:09.194 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.922819+0000", 2020-01-16T00:14:09.194 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 71.156943862000006, 2020-01-16T00:14:09.194 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017502846999999998, 2020-01-16T00:14:09.194 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:09.194 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:09.194 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:09.195 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:09.195 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:09.195 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1950 2020-01-16T00:14:09.195 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.195 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:09.195 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.195 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:09.195 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922819+0000", 2020-01-16T00:14:09.196 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:09.196 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.196 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.196 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:09.196 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922819+0000", 2020-01-16T00:14:09.196 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:09.196 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.197 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.197 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:09.197 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922818+0000", 2020-01-16T00:14:09.197 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000146408 2020-01-16T00:14:09.197 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.197 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.197 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:09.197 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922964+0000", 2020-01-16T00:14:09.198 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4720000000000001e-06 2020-01-16T00:14:09.198 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.198 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.198 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:09.198 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922974+0000", 2020-01-16T00:14:09.198 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0720000000000001e-06 2020-01-16T00:14:09.199 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.199 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.199 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:09.199 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922978+0000", 2020-01-16T00:14:09.199 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6152000000000001e-05 2020-01-16T00:14:09.199 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.199 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.199 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.200 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.923024+0000", 2020-01-16T00:14:09.200 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00087368199999999997 2020-01-16T00:14:09.200 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.200 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.200 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:09.200 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.923897+0000", 2020-01-16T00:14:09.200 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011944435999999999 2020-01-16T00:14:09.200 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.201 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.201 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.201 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.935842+0000", 2020-01-16T00:14:09.201 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000144146 2020-01-16T00:14:09.201 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.201 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.201 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:09.202 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.935986+0000", 2020-01-16T00:14:09.202 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019775500000000001 2020-01-16T00:14:09.203 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.203 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.203 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:09.203 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.936184+0000", 2020-01-16T00:14:09.203 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002539342 2020-01-16T00:14:09.204 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.204 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.204 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:09.204 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938723+0000", 2020-01-16T00:14:09.204 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014193699999999999 2020-01-16T00:14:09.204 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.204 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.204 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:09.205 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938865+0000", 2020-01-16T00:14:09.205 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013829020000000001 2020-01-16T00:14:09.205 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.205 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.205 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:09.205 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.940248+0000", 2020-01-16T00:14:09.205 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2558000000000001e-05 2020-01-16T00:14:09.206 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.206 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.206 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:09.206 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.940291+0000", 2020-01-16T00:14:09.206 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1217999999999998e-05 2020-01-16T00:14:09.206 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.207 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.207 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:09.207 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.940322+0000", 2020-01-16T00:14:09.207 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017502846999999998 2020-01-16T00:14:09.207 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.207 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:09.207 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.208 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.208 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.208 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1951 3.0 3:0b1f384f:::smithi12313934-33:head [write 2955792~541902] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-01-16T00:14:09.208 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.929019+0000", 2020-01-16T00:14:09.208 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 71.150743711999993, 2020-01-16T00:14:09.208 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021625710999999999, 2020-01-16T00:14:09.209 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:09.209 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:09.209 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:09.209 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:09.209 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:09.209 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1951 2020-01-16T00:14:09.210 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.210 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:09.210 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.210 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:09.210 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929019+0000", 2020-01-16T00:14:09.211 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:09.211 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.211 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.211 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:09.211 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929019+0000", 2020-01-16T00:14:09.211 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999989 2020-01-16T00:14:09.211 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.211 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.212 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:09.212 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929008+0000", 2020-01-16T00:14:09.212 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000212513 2020-01-16T00:14:09.212 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.212 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.212 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:09.212 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929221+0000", 2020-01-16T00:14:09.212 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3229e-05 2020-01-16T00:14:09.213 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.213 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.213 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:09.213 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929234+0000", 2020-01-16T00:14:09.213 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9170000000000003e-06 2020-01-16T00:14:09.213 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.213 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.213 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:09.214 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929240+0000", 2020-01-16T00:14:09.214 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4045000000000002e-05 2020-01-16T00:14:09.214 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.214 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.214 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.214 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929294+0000", 2020-01-16T00:14:09.214 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9797999999999996e-05 2020-01-16T00:14:09.214 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.215 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.215 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:09.215 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929384+0000", 2020-01-16T00:14:09.215 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083450729999999997 2020-01-16T00:14:09.215 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.215 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.215 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.215 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.937729+0000", 2020-01-16T00:14:09.216 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043251699999999999 2020-01-16T00:14:09.216 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.216 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.216 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:09.216 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938161+0000", 2020-01-16T00:14:09.216 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095115489999999994 2020-01-16T00:14:09.216 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.217 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.217 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.217 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.947673+0000", 2020-01-16T00:14:09.217 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012395099999999999 2020-01-16T00:14:09.217 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.217 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.217 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:09.217 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.947797+0000", 2020-01-16T00:14:09.218 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00024849200000000001 2020-01-16T00:14:09.218 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.218 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.218 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:09.218 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.948045+0000", 2020-01-16T00:14:09.218 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0025081050000000001 2020-01-16T00:14:09.218 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.218 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.219 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:09.219 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950553+0000", 2020-01-16T00:14:09.219 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9760000000000003e-06 2020-01-16T00:14:09.219 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.219 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.219 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:09.219 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950558+0000", 2020-01-16T00:14:09.220 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3286999999999999e-05 2020-01-16T00:14:09.220 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.220 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.220 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:09.220 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950581+0000", 2020-01-16T00:14:09.220 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2759000000000003e-05 2020-01-16T00:14:09.220 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.220 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.221 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:09.221 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950614+0000", 2020-01-16T00:14:09.221 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0525999999999999e-05 2020-01-16T00:14:09.221 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.221 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.221 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:09.221 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950645+0000", 2020-01-16T00:14:09.221 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021625710999999999 2020-01-16T00:14:09.222 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.222 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:09.222 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.222 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.222 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.222 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1952 3.0 3:0b1f384f:::smithi12313934-33:head [setxattr _header (56) in=7b,truncate 3563084] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-01-16T00:14:09.222 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:14:09.222 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 71.150514375, 2020-01-16T00:14:09.223 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021534685000000001, 2020-01-16T00:14:09.223 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:09.223 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:09.223 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:09.223 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:09.223 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:09.223 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1952 2020-01-16T00:14:09.224 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.224 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:09.224 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.224 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:09.224 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:14:09.224 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:09.224 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.224 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.225 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:09.225 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:14:09.225 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:09.225 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.225 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.225 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:09.225 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:14:09.225 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6670000000000004e-06 2020-01-16T00:14:09.226 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.226 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.226 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:09.226 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929252+0000", 2020-01-16T00:14:09.226 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9380000000000002e-06 2020-01-16T00:14:09.226 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.226 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.227 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:09.227 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929256+0000", 2020-01-16T00:14:09.227 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.712e-06 2020-01-16T00:14:09.227 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.227 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.227 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:09.228 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929264+0000", 2020-01-16T00:14:09.228 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018030300000000001 2020-01-16T00:14:09.228 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.228 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.228 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.228 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929444+0000", 2020-01-16T00:14:09.228 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7562000000000002e-05 2020-01-16T00:14:09.229 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.229 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.229 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:09.229 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929502+0000", 2020-01-16T00:14:09.229 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0087297859999999998 2020-01-16T00:14:09.229 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.229 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.230 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.230 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938232+0000", 2020-01-16T00:14:09.230 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0452999999999996e-05 2020-01-16T00:14:09.230 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.230 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.230 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:09.230 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938322+0000", 2020-01-16T00:14:09.231 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011061728 2020-01-16T00:14:09.231 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.231 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.231 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.231 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.949384+0000", 2020-01-16T00:14:09.231 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000117828 2020-01-16T00:14:09.231 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.231 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.232 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:09.232 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.949502+0000", 2020-01-16T00:14:09.232 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022468499999999999 2020-01-16T00:14:09.232 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.232 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.232 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:09.232 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.949726+0000", 2020-01-16T00:14:09.232 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00083532099999999998 2020-01-16T00:14:09.233 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.233 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.233 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:09.233 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950562+0000", 2020-01-16T00:14:09.233 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013017700000000001 2020-01-16T00:14:09.233 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.233 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.233 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:09.234 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950692+0000", 2020-01-16T00:14:09.234 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9813000000000001e-05 2020-01-16T00:14:09.234 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.234 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.234 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:09.234 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950722+0000", 2020-01-16T00:14:09.234 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0592000000000001e-05 2020-01-16T00:14:09.235 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.235 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.235 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:09.235 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950752+0000", 2020-01-16T00:14:09.235 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0920000000000002e-05 2020-01-16T00:14:09.235 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.235 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.235 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:09.236 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950783+0000", 2020-01-16T00:14:09.236 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021534685000000001 2020-01-16T00:14:09.236 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.236 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:09.236 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.236 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.236 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.236 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1953 3.0 3:0b1f384f:::smithi12313934-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e32)", 2020-01-16T00:14:09.237 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:14:09.237 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 71.150481384000003, 2020-01-16T00:14:09.237 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021667537000000001, 2020-01-16T00:14:09.237 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:09.237 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:09.237 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:09.237 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:09.237 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:09.238 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1953 2020-01-16T00:14:09.238 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.238 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:09.238 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.238 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:09.238 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:14:09.238 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:09.238 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.239 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.239 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:09.239 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:14:09.239 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:09.239 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.239 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.239 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:09.239 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:14:09.240 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7030000000000002e-06 2020-01-16T00:14:09.240 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.240 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.240 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:09.240 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929284+0000", 2020-01-16T00:14:09.240 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0909999999999993e-06 2020-01-16T00:14:09.240 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.241 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.241 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:09.241 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929292+0000", 2020-01-16T00:14:09.241 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8359999999999999e-06 2020-01-16T00:14:09.241 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.241 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.241 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:09.241 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929297+0000", 2020-01-16T00:14:09.242 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023538199999999999 2020-01-16T00:14:09.242 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.242 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.242 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.242 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929532+0000", 2020-01-16T00:14:09.242 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1173000000000004e-05 2020-01-16T00:14:09.242 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.242 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.243 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:09.243 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929603+0000", 2020-01-16T00:14:09.243 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0087813509999999997 2020-01-16T00:14:09.243 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.243 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.243 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.244 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938384+0000", 2020-01-16T00:14:09.244 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9752000000000006e-05 2020-01-16T00:14:09.244 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.244 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.244 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:09.244 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938454+0000", 2020-01-16T00:14:09.244 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011707851999999999 2020-01-16T00:14:09.244 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.245 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.245 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.245 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950162+0000", 2020-01-16T00:14:09.245 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011060099999999999 2020-01-16T00:14:09.245 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.245 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.245 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:09.246 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950273+0000", 2020-01-16T00:14:09.246 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00053989700000000001 2020-01-16T00:14:09.246 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.246 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.246 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.246 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950813+0000", 2020-01-16T00:14:09.246 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8128999999999999e-05 2020-01-16T00:14:09.246 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.247 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.247 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:09.247 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950871+0000", 2020-01-16T00:14:09.247 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8242000000000001e-05 2020-01-16T00:14:09.247 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.247 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.247 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:09.247 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950949+0000", 2020-01-16T00:14:09.248 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021667537000000001 2020-01-16T00:14:09.248 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.248 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:09.248 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.248 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.248 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.248 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2555 3.0 3:0b70bb20:::smithi12313934-46:head [setxattr _header (56) in=7b,truncate 2991679] snapc 0=[] ondisk+write+known_if_redirected e37)", 2020-01-16T00:14:09.248 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.573432+0000", 2020-01-16T00:14:09.249 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 56.506330732999999, 2020-01-16T00:14:09.249 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019893926999999999, 2020-01-16T00:14:09.249 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:09.249 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:09.249 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:09.249 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:09.249 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:09.250 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2555 2020-01-16T00:14:09.250 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.250 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:09.250 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.250 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:09.250 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573432+0000", 2020-01-16T00:14:09.250 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:09.250 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.251 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.251 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:09.251 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573432+0000", 2020-01-16T00:14:09.251 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:09.251 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.251 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.251 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:09.251 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573432+0000", 2020-01-16T00:14:09.252 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4290000000000001e-06 2020-01-16T00:14:09.252 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.252 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.252 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:09.252 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573435+0000", 2020-01-16T00:14:09.252 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1300000000000001e-06 2020-01-16T00:14:09.252 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.252 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.253 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:09.253 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573438+0000", 2020-01-16T00:14:09.253 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9399999999999996e-06 2020-01-16T00:14:09.253 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.253 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.253 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:09.253 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573445+0000", 2020-01-16T00:14:09.254 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013947269999999999 2020-01-16T00:14:09.254 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.254 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.254 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.254 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.574840+0000", 2020-01-16T00:14:09.254 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003850058 2020-01-16T00:14:09.254 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.254 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.255 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:09.255 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578690+0000", 2020-01-16T00:14:09.255 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012896489000000001 2020-01-16T00:14:09.255 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.255 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.255 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.255 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591586+0000", 2020-01-16T00:14:09.255 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4757000000000001e-05 2020-01-16T00:14:09.256 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.256 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.256 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:09.256 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591681+0000", 2020-01-16T00:14:09.256 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000191535 2020-01-16T00:14:09.256 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.256 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.256 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:09.257 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591873+0000", 2020-01-16T00:14:09.257 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00069888899999999996 2020-01-16T00:14:09.257 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.257 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.257 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:09.257 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.592571+0000", 2020-01-16T00:14:09.257 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00029525900000000003 2020-01-16T00:14:09.257 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.258 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.258 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:09.258 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.592867+0000", 2020-01-16T00:14:09.258 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035243500000000001 2020-01-16T00:14:09.258 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.258 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.258 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:09.259 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593219+0000", 2020-01-16T00:14:09.259 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0954e-05 2020-01-16T00:14:09.259 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.259 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.259 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:09.259 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593270+0000", 2020-01-16T00:14:09.259 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5813000000000002e-05 2020-01-16T00:14:09.259 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.260 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.260 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:09.260 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593326+0000", 2020-01-16T00:14:09.260 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019893926999999999 2020-01-16T00:14:09.260 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.260 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:09.260 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.260 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.261 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.261 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2556 3.0 3:0b70bb20:::smithi12313934-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e37)", 2020-01-16T00:14:09.261 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.573472+0000", 2020-01-16T00:14:09.261 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 56.506291163, 2020-01-16T00:14:09.261 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020070068999999999, 2020-01-16T00:14:09.261 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:09.261 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:09.261 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:09.262 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:09.262 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:09.262 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2556 2020-01-16T00:14:09.262 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.262 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:09.262 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.262 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:09.262 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573472+0000", 2020-01-16T00:14:09.263 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:09.263 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.263 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.263 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:09.263 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573472+0000", 2020-01-16T00:14:09.263 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:09.263 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.264 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.264 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:09.264 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573471+0000", 2020-01-16T00:14:09.264 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4449999999999999e-06 2020-01-16T00:14:09.264 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.264 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.264 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:09.264 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573479+0000", 2020-01-16T00:14:09.265 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9789999999999998e-06 2020-01-16T00:14:09.265 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.265 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.265 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:09.265 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573485+0000", 2020-01-16T00:14:09.265 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3270000000000001e-06 2020-01-16T00:14:09.265 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.265 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.266 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:09.266 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573488+0000", 2020-01-16T00:14:09.266 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052586849999999999 2020-01-16T00:14:09.266 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.266 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.266 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.266 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578747+0000", 2020-01-16T00:14:09.266 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5667000000000007e-05 2020-01-16T00:14:09.267 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.267 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.267 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:09.267 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578812+0000", 2020-01-16T00:14:09.267 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013599539000000001 2020-01-16T00:14:09.267 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.267 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.268 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.268 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.592412+0000", 2020-01-16T00:14:09.268 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6018000000000001e-05 2020-01-16T00:14:09.268 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.268 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.268 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:09.268 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.592508+0000", 2020-01-16T00:14:09.268 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00084582199999999998 2020-01-16T00:14:09.269 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.269 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.269 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.269 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593354+0000", 2020-01-16T00:14:09.269 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4984999999999999e-05 2020-01-16T00:14:09.269 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.269 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.270 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:09.270 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593449+0000", 2020-01-16T00:14:09.270 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3104999999999997e-05 2020-01-16T00:14:09.270 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.270 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.270 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:09.271 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593542+0000", 2020-01-16T00:14:09.271 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020070068999999999 2020-01-16T00:14:09.271 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.271 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:09.271 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.271 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.271 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.271 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3211 3.0 3:0b70bb20:::smithi12313934-46:head [setxattr _header (56) in=7b,truncate 2571537] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:14:09.272 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.581146+0000", 2020-01-16T00:14:09.272 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 39.498616593000001, 2020-01-16T00:14:09.272 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024318514999999999, 2020-01-16T00:14:09.272 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:09.272 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:09.272 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:09.272 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:09.272 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:09.273 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3211 2020-01-16T00:14:09.273 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.273 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:09.273 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.273 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:09.273 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581146+0000", 2020-01-16T00:14:09.273 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:09.274 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.274 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.274 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:09.274 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581146+0000", 2020-01-16T00:14:09.274 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:09.274 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.274 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.274 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:09.275 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581145+0000", 2020-01-16T00:14:09.275 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2990000000000002e-06 2020-01-16T00:14:09.275 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.275 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.275 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:09.275 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581151+0000", 2020-01-16T00:14:09.275 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.041e-06 2020-01-16T00:14:09.276 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.276 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.276 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:09.276 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581159+0000", 2020-01-16T00:14:09.276 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5859999999999998e-06 2020-01-16T00:14:09.276 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.276 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.276 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:09.277 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581163+0000", 2020-01-16T00:14:09.277 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020769790000000001 2020-01-16T00:14:09.277 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.277 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.277 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.277 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.583240+0000", 2020-01-16T00:14:09.277 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012019579999999999 2020-01-16T00:14:09.277 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.278 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.278 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:09.278 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584442+0000", 2020-01-16T00:14:09.278 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018359812 2020-01-16T00:14:09.278 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.278 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.278 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.279 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.602802+0000", 2020-01-16T00:14:09.279 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000133072 2020-01-16T00:14:09.279 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.279 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.279 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:09.279 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.602935+0000", 2020-01-16T00:14:09.279 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019007200000000001 2020-01-16T00:14:09.279 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.280 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.280 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:09.280 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.603125+0000", 2020-01-16T00:14:09.280 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001261751 2020-01-16T00:14:09.280 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.280 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.280 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:09.280 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.604387+0000", 2020-01-16T00:14:09.281 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000304987 2020-01-16T00:14:09.281 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.281 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.281 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:09.281 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.604692+0000", 2020-01-16T00:14:09.281 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00063369299999999995 2020-01-16T00:14:09.281 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.282 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.282 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:09.282 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605325+0000", 2020-01-16T00:14:09.282 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1470000000000005e-05 2020-01-16T00:14:09.282 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.282 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.282 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:09.282 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605397+0000", 2020-01-16T00:14:09.283 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7784e-05 2020-01-16T00:14:09.283 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.283 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.283 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:09.283 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605465+0000", 2020-01-16T00:14:09.283 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024318514999999999 2020-01-16T00:14:09.283 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.283 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:09.284 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.284 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.284 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.284 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3212 3.0 3:0b70bb20:::smithi12313934-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e45)", 2020-01-16T00:14:09.284 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.581171+0000", 2020-01-16T00:14:09.284 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 39.498591894, 2020-01-16T00:14:09.284 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024540131, 2020-01-16T00:14:09.284 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:09.285 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:09.285 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:09.285 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:09.285 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:09.285 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3212 2020-01-16T00:14:09.285 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.285 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:09.285 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.286 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:09.286 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581171+0000", 2020-01-16T00:14:09.286 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:09.286 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.286 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.286 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:09.286 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581171+0000", 2020-01-16T00:14:09.287 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:09.287 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.287 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.287 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:09.287 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581170+0000", 2020-01-16T00:14:09.287 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5259999999999996e-06 2020-01-16T00:14:09.288 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.288 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.288 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:09.288 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581178+0000", 2020-01-16T00:14:09.288 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.469e-06 2020-01-16T00:14:09.288 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.289 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.289 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:09.289 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581185+0000", 2020-01-16T00:14:09.289 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4460000000000003e-06 2020-01-16T00:14:09.289 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.289 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.289 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:09.290 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581190+0000", 2020-01-16T00:14:09.290 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003286696 2020-01-16T00:14:09.290 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.290 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.290 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.290 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584476+0000", 2020-01-16T00:14:09.290 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9681e-05 2020-01-16T00:14:09.290 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.291 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.291 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:09.291 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584536+0000", 2020-01-16T00:14:09.291 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019075267 2020-01-16T00:14:09.291 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.291 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.291 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.291 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.603611+0000", 2020-01-16T00:14:09.292 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5981000000000001e-05 2020-01-16T00:14:09.292 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.292 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.292 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:09.292 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.603707+0000", 2020-01-16T00:14:09.292 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0017922649999999999 2020-01-16T00:14:09.292 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.292 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.293 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.293 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605500+0000", 2020-01-16T00:14:09.293 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5396999999999995e-05 2020-01-16T00:14:09.293 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.293 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.293 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:09.293 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605595+0000", 2020-01-16T00:14:09.293 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000116038 2020-01-16T00:14:09.294 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.294 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.294 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:09.294 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605711+0000", 2020-01-16T00:14:09.294 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024540131 2020-01-16T00:14:09.294 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.294 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:09.295 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.295 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.295 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.295 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3435 3.0 3:0b70bb20:::smithi12313934-46:head [setxattr _header (56) in=7b,truncate 3075747] snapc 0=[] ondisk+write+known_if_redirected e47)", 2020-01-16T00:14:09.295 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:34.388891+0000", 2020-01-16T00:14:09.295 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 34.690871878000003, 2020-01-16T00:14:09.295 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016440475999999999, 2020-01-16T00:14:09.295 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:09.296 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:09.296 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:09.296 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:09.296 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:09.296 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3435 2020-01-16T00:14:09.296 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.296 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:09.297 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.297 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:09.297 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388891+0000", 2020-01-16T00:14:09.297 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:09.297 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.298 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.298 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:09.298 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388891+0000", 2020-01-16T00:14:09.298 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:09.298 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.298 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.299 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:09.299 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388890+0000", 2020-01-16T00:14:09.299 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.647e-06 2020-01-16T00:14:09.299 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.299 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.299 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:09.300 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388894+0000", 2020-01-16T00:14:09.300 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2839999999999997e-06 2020-01-16T00:14:09.300 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.300 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.300 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:09.300 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388899+0000", 2020-01-16T00:14:09.300 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3149999999999999e-06 2020-01-16T00:14:09.300 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.301 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.301 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:09.301 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388901+0000", 2020-01-16T00:14:09.301 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001768543 2020-01-16T00:14:09.301 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.301 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.301 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.301 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.390670+0000", 2020-01-16T00:14:09.302 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0034533680000000001 2020-01-16T00:14:09.302 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.302 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.302 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:09.302 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.394123+0000", 2020-01-16T00:14:09.302 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010189962 2020-01-16T00:14:09.302 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.303 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.303 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.303 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404313+0000", 2020-01-16T00:14:09.303 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000108646 2020-01-16T00:14:09.303 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.303 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.303 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:09.303 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404422+0000", 2020-01-16T00:14:09.304 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119133 2020-01-16T00:14:09.304 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.304 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.304 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:09.304 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404541+0000", 2020-01-16T00:14:09.304 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000433764 2020-01-16T00:14:09.304 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.304 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.305 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:09.305 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404975+0000", 2020-01-16T00:14:09.305 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000266013 2020-01-16T00:14:09.305 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.305 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.305 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:09.305 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405241+0000", 2020-01-16T00:14:09.306 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5187000000000001e-05 2020-01-16T00:14:09.306 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.306 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.306 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:09.306 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405266+0000", 2020-01-16T00:14:09.306 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0889000000000002e-05 2020-01-16T00:14:09.306 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.307 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.307 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:09.307 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405297+0000", 2020-01-16T00:14:09.307 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4354999999999999e-05 2020-01-16T00:14:09.307 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.307 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.308 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:09.308 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405331+0000", 2020-01-16T00:14:09.308 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016440475999999999 2020-01-16T00:14:09.308 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.308 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:09.308 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.308 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.308 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.309 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3436 3.0 3:0b70bb20:::smithi12313934-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e47)", 2020-01-16T00:14:09.309 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:34.388920+0000", 2020-01-16T00:14:09.309 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 34.690843088999998, 2020-01-16T00:14:09.309 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016552641, 2020-01-16T00:14:09.309 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:09.309 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:09.309 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:09.310 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:09.310 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:09.310 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3436 2020-01-16T00:14:09.310 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.310 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:09.310 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.311 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:09.311 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388920+0000", 2020-01-16T00:14:09.311 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:09.311 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.311 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.311 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:09.311 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388920+0000", 2020-01-16T00:14:09.311 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:09.312 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.312 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.312 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:09.312 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388919+0000", 2020-01-16T00:14:09.312 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.181e-06 2020-01-16T00:14:09.312 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.312 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.313 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:09.313 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388921+0000", 2020-01-16T00:14:09.313 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9940000000000003e-06 2020-01-16T00:14:09.313 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.313 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.313 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:09.313 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388928+0000", 2020-01-16T00:14:09.313 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3549999999999999e-06 2020-01-16T00:14:09.314 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.314 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.314 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:09.314 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388931+0000", 2020-01-16T00:14:09.314 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.005242483 2020-01-16T00:14:09.314 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.314 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.315 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.315 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.394173+0000", 2020-01-16T00:14:09.315 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7435999999999995e-05 2020-01-16T00:14:09.315 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.315 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.315 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:09.315 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.394251+0000", 2020-01-16T00:14:09.316 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010593286 2020-01-16T00:14:09.316 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.316 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.316 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.316 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404844+0000", 2020-01-16T00:14:09.316 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0620999999999999e-05 2020-01-16T00:14:09.316 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.316 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.317 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:09.317 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404925+0000", 2020-01-16T00:14:09.317 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00042800899999999997 2020-01-16T00:14:09.317 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.317 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.317 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.317 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405353+0000", 2020-01-16T00:14:09.318 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2250000000000003e-05 2020-01-16T00:14:09.318 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.318 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.318 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:09.318 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405405+0000", 2020-01-16T00:14:09.318 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7411999999999998e-05 2020-01-16T00:14:09.318 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.318 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.319 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:09.319 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405472+0000", 2020-01-16T00:14:09.319 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016552641 2020-01-16T00:14:09.319 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.319 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:09.319 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.319 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.319 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.320 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4721 3.0 3:0b70bb20:::smithi12313934-46:head [setxattr _header (56) in=7b,truncate 2134621] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-01-16T00:14:09.320 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.045819+0000", 2020-01-16T00:14:09.320 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 2.0339439349999999, 2020-01-16T00:14:09.320 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019902080999999999, 2020-01-16T00:14:09.320 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:09.320 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:09.320 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:09.320 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:09.321 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:09.321 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4721 2020-01-16T00:14:09.321 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.321 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:09.321 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.321 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:09.321 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045819+0000", 2020-01-16T00:14:09.322 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:09.322 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.322 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.322 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:09.322 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045819+0000", 2020-01-16T00:14:09.322 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:09.322 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.322 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.323 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:09.323 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045818+0000", 2020-01-16T00:14:09.323 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.428e-06 2020-01-16T00:14:09.323 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.323 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.323 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:09.323 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045822+0000", 2020-01-16T00:14:09.324 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5940000000000004e-06 2020-01-16T00:14:09.324 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.324 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.324 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:09.324 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045826+0000", 2020-01-16T00:14:09.324 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6619999999999998e-06 2020-01-16T00:14:09.324 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.325 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.325 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:09.325 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045830+0000", 2020-01-16T00:14:09.325 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001710676 2020-01-16T00:14:09.325 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.325 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.325 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.325 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.047541+0000", 2020-01-16T00:14:09.326 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001637009 2020-01-16T00:14:09.326 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.326 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.326 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:09.326 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049178+0000", 2020-01-16T00:14:09.326 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.01461612 2020-01-16T00:14:09.326 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.327 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.327 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.327 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.063794+0000", 2020-01-16T00:14:09.327 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4941999999999997e-05 2020-01-16T00:14:09.327 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.327 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.328 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:09.328 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.063849+0000", 2020-01-16T00:14:09.328 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011165 2020-01-16T00:14:09.328 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.328 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.328 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:09.328 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.063960+0000", 2020-01-16T00:14:09.329 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013431630000000001 2020-01-16T00:14:09.329 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.329 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.329 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:09.329 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.065303+0000", 2020-01-16T00:14:09.329 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021894699999999999 2020-01-16T00:14:09.330 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.330 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.330 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:09.330 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.065522+0000", 2020-01-16T00:14:09.330 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013033899999999999 2020-01-16T00:14:09.330 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.330 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.331 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:09.331 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.065653+0000", 2020-01-16T00:14:09.331 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3800000000000002e-05 2020-01-16T00:14:09.331 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.331 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.331 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:09.331 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.065687+0000", 2020-01-16T00:14:09.332 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4332999999999998e-05 2020-01-16T00:14:09.332 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.332 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.332 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:09.332 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.065721+0000", 2020-01-16T00:14:09.332 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019902080999999999 2020-01-16T00:14:09.332 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.332 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:09.333 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.333 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.333 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.333 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4722 3.0 3:0b70bb20:::smithi12313934-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e58)", 2020-01-16T00:14:09.333 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.045850+0000", 2020-01-16T00:14:09.333 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 2.033913176, 2020-01-16T00:14:09.333 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020039033000000001, 2020-01-16T00:14:09.334 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:09.334 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:09.334 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:09.334 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:09.335 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:09.335 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4722 2020-01-16T00:14:09.335 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.336 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:09.336 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.336 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:09.336 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045850+0000", 2020-01-16T00:14:09.336 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:09.336 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.336 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.336 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:09.337 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045850+0000", 2020-01-16T00:14:09.337 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:09.337 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.337 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.338 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:09.338 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045849+0000", 2020-01-16T00:14:09.338 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7349999999999999e-06 2020-01-16T00:14:09.338 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.338 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.338 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:09.339 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045856+0000", 2020-01-16T00:14:09.339 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1330000000000001e-06 2020-01-16T00:14:09.339 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.339 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.339 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:09.339 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045860+0000", 2020-01-16T00:14:09.339 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.565e-06 2020-01-16T00:14:09.340 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.340 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.340 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:09.340 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045863+0000", 2020-01-16T00:14:09.340 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033590019999999998 2020-01-16T00:14:09.340 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.341 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.341 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.341 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049222+0000", 2020-01-16T00:14:09.341 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8267e-05 2020-01-16T00:14:09.342 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.342 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.342 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:09.342 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049320+0000", 2020-01-16T00:14:09.342 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015128875999999999 2020-01-16T00:14:09.343 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.343 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.343 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.343 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.064449+0000", 2020-01-16T00:14:09.343 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4913000000000001e-05 2020-01-16T00:14:09.343 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.344 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.344 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:09.344 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.064504+0000", 2020-01-16T00:14:09.344 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012498100000000001 2020-01-16T00:14:09.344 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.344 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.344 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:09.345 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.065753+0000", 2020-01-16T00:14:09.345 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6299999999999999e-05 2020-01-16T00:14:09.345 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.345 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.345 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:09.345 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.065820+0000", 2020-01-16T00:14:09.345 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8801999999999996e-05 2020-01-16T00:14:09.345 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:09.346 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:09.346 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:09.346 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.065889+0000", 2020-01-16T00:14:09.346 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020039033000000001 2020-01-16T00:14:09.346 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.346 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:09.346 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.347 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:09.347 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:09.347 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:09.351 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:14:09.351 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:14:09.351 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:14:09.351 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:14:09.352 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:14:09.352 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:14:09.352 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:14:09.352 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-01-16T00:14:09.352 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-01-16T00:14:09.352 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-01-16T00:14:09.353 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-01-16T00:14:09.353 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-01-16T00:14:09.353 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENXIO: problem getting command descriptions from osd.5 2020-01-16T00:14:09.353 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: problem getting command descriptions from osd.5 2020-01-16T00:14:09.353 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-01-16T00:14:09.353 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-01-16T00:14:09.353 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-01-16T00:14:09.354 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-01-16T00:14:09.354 INFO:tasks.rados.rados.0.smithi123.stdout:1701: finishing write tid 2 to smithi12313934-19 2020-01-16T00:14:09.354 INFO:tasks.rados.rados.0.smithi123.stdout:1701: finishing write tid 4 to smithi12313934-19 2020-01-16T00:14:09.354 INFO:tasks.rados.rados.0.smithi123.stdout:1701: finishing write tid 5 to smithi12313934-19 2020-01-16T00:14:09.354 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 19 v 237 (ObjNum 611 snap 0 seq_num 611) dirty exists 2020-01-16T00:14:09.354 INFO:tasks.rados.rados.0.smithi123.stdout:1701: left oid 19 (ObjNum 611 snap 0 seq_num 611) 2020-01-16T00:14:09.354 INFO:tasks.rados.rados.0.smithi123.stdout:1697: done (10 left) 2020-01-16T00:14:09.355 INFO:tasks.rados.rados.0.smithi123.stdout:1698: done (9 left) 2020-01-16T00:14:09.355 INFO:tasks.rados.rados.0.smithi123.stdout:1699: done (8 left) 2020-01-16T00:14:09.355 INFO:tasks.rados.rados.0.smithi123.stdout:1700: done (7 left) 2020-01-16T00:14:09.355 INFO:tasks.rados.rados.0.smithi123.stdout:1701: done (6 left) 2020-01-16T00:14:09.355 INFO:tasks.rados.rados.0.smithi123.stdout:1704: done (5 left) 2020-01-16T00:14:09.355 INFO:tasks.rados.rados.0.smithi123.stdout:1705: done (4 left) 2020-01-16T00:14:09.355 INFO:tasks.rados.rados.0.smithi123.stdout:1707: done (3 left) 2020-01-16T00:14:09.355 INFO:tasks.rados.rados.0.smithi123.stdout:1708: read oid 8 snap -1 2020-01-16T00:14:09.356 INFO:tasks.rados.rados.0.smithi123.stdout:1708: expect (ObjNum 565 snap 0 seq_num 565) 2020-01-16T00:14:09.356 INFO:tasks.rados.rados.0.smithi123.stdout:1709: read oid 49 snap -1 2020-01-16T00:14:09.356 INFO:tasks.rados.rados.0.smithi123.stdout:1709: expect (ObjNum 564 snap 0 seq_num 564) 2020-01-16T00:14:09.356 INFO:tasks.rados.rados.0.smithi123.stdout:1710: read oid 23 snap -1 2020-01-16T00:14:09.356 INFO:tasks.rados.rados.0.smithi123.stdout:1710: expect deleted 2020-01-16T00:14:09.356 INFO:tasks.rados.rados.0.smithi123.stdout:1711: read oid 21 snap -1 2020-01-16T00:14:09.356 INFO:tasks.rados.rados.0.smithi123.stdout:1711: expect (ObjNum 576 snap 0 seq_num 576) 2020-01-16T00:14:09.357 INFO:tasks.rados.rados.0.smithi123.stdout:1712: write (excl) oid 28 current snap is 0 2020-01-16T00:14:09.357 INFO:tasks.rados.rados.0.smithi123.stdout:1712: seq_num 613 ranges {702691=635877,2104756=197935} 2020-01-16T00:14:09.357 INFO:tasks.rados.rados.0.smithi123.stdout:1712: writing smithi12313934-28 from 702691 to 1338568 tid 1 2020-01-16T00:14:09.357 INFO:tasks.rados.rados.0.smithi123.stdout:1712: writing smithi12313934-28 from 2104756 to 2302691 tid 2 2020-01-16T00:14:09.358 INFO:tasks.rados.rados.0.smithi123.stdout:1713: read oid 43 snap -1 2020-01-16T00:14:09.358 INFO:tasks.rados.rados.0.smithi123.stdout:1713: expect (ObjNum 607 snap 0 seq_num 607) 2020-01-16T00:14:09.358 INFO:tasks.rados.rados.0.smithi123.stdout:1714: read oid 15 snap -1 2020-01-16T00:14:09.358 INFO:tasks.rados.rados.0.smithi123.stdout:1714: expect (ObjNum 593 snap 0 seq_num 593) 2020-01-16T00:14:09.358 INFO:tasks.rados.rados.0.smithi123.stdout:1715: write oid 7 current snap is 0 2020-01-16T00:14:09.359 INFO:tasks.rados.rados.0.smithi123.stdout:1715: seq_num 614 ranges {719498=711126,2046885=272613} 2020-01-16T00:14:09.359 INFO:tasks.rados.rados.0.smithi123.stdout:1715: writing smithi12313934-7 from 719498 to 1430624 tid 1 2020-01-16T00:14:09.359 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:09.346+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:09.360 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-01-16T00:14:09.361 INFO:tasks.rados.rados.0.smithi123.stdout:1715: writing smithi12313934-7 from 2046885 to 2319498 tid 2 2020-01-16T00:14:09.361 INFO:tasks.rados.rados.0.smithi123.stdout:1716: write (excl) oid 27 current snap is 0 2020-01-16T00:14:09.362 INFO:tasks.rados.rados.0.smithi123.stdout:1716: seq_num 615 ranges {736305=786375,2105367=230938} 2020-01-16T00:14:09.362 INFO:tasks.rados.rados.0.smithi123.stdout:1716: writing smithi12313934-27 from 736305 to 1522680 tid 1 2020-01-16T00:14:09.368 INFO:tasks.rados.rados.0.smithi123.stdout:1716: writing smithi12313934-27 from 2105367 to 2336305 tid 2 2020-01-16T00:14:09.368 INFO:tasks.rados.rados.0.smithi123.stdout:1717: read oid 9 snap -1 2020-01-16T00:14:09.368 INFO:tasks.rados.rados.0.smithi123.stdout:1717: expect (ObjNum 572 snap 0 seq_num 572) 2020-01-16T00:14:09.368 INFO:tasks.rados.rados.0.smithi123.stdout:1718: read oid 31 snap -1 2020-01-16T00:14:09.369 INFO:tasks.rados.rados.0.smithi123.stdout:1718: expect (ObjNum 522 snap 0 seq_num 522) 2020-01-16T00:14:09.369 INFO:tasks.rados.rados.0.smithi123.stdout:1719: write oid 40 current snap is 0 2020-01-16T00:14:09.369 INFO:tasks.rados.rados.0.smithi123.stdout:1719: seq_num 616 ranges {753112=577977,1880202=472910} 2020-01-16T00:14:09.375 INFO:tasks.rados.rados.0.smithi123.stdout:1719: writing smithi12313934-40 from 753112 to 1331089 tid 1 2020-01-16T00:14:09.381 INFO:tasks.rados.rados.0.smithi123.stdout:1719: writing smithi12313934-40 from 1880202 to 2353112 tid 2 2020-01-16T00:14:09.382 INFO:tasks.rados.rados.0.smithi123.stdout:1720: read oid 26 snap -1 2020-01-16T00:14:09.382 INFO:tasks.rados.rados.0.smithi123.stdout:1720: expect (ObjNum 556 snap 0 seq_num 556) 2020-01-16T00:14:09.382 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:09.382 INFO:tasks.rados.rados.0.smithi123.stdout:1703: expect (ObjNum 599 snap 0 seq_num 599) 2020-01-16T00:14:09.448 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:09.446+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:09.455 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:09.464 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-01-16T00:14:09.550 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:09.548+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:09.579 INFO:tasks.rados.rados.0.smithi123.stdout:1702: expect (ObjNum 592 snap 0 seq_num 592) 2020-01-16T00:14:09.605 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:09.615 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-01-16T00:14:09.650 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:09.648+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:09.713 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:09.721 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-01-16T00:14:09.751 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:09.748+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:09.755 INFO:tasks.rados.rados.0.smithi123.stdout:1702: done (15 left) 2020-01-16T00:14:09.756 INFO:tasks.rados.rados.0.smithi123.stdout:1703: done (14 left) 2020-01-16T00:14:09.756 INFO:tasks.rados.rados.0.smithi123.stdout:1721: write (excl) oid 5 current snap is 0 2020-01-16T00:14:09.756 INFO:tasks.rados.rados.0.smithi123.stdout:1721: seq_num 617 ranges {769919=653226,1938684=431235} 2020-01-16T00:14:09.763 INFO:tasks.rados.rados.0.smithi123.stdout:1721: writing smithi12313934-5 from 769919 to 1423145 tid 1 2020-01-16T00:14:09.769 INFO:tasks.rados.rados.0.smithi123.stdout:1721: writing smithi12313934-5 from 1938684 to 2369919 tid 2 2020-01-16T00:14:09.769 INFO:tasks.rados.rados.0.smithi123.stdout:1722: read oid 17 snap -1 2020-01-16T00:14:09.770 INFO:tasks.rados.rados.0.smithi123.stdout:1722: expect (ObjNum 596 snap 0 seq_num 596) 2020-01-16T00:14:09.770 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:09.770 INFO:tasks.rados.rados.0.smithi123.stdout:1706: expect (ObjNum 571 snap 0 seq_num 571) 2020-01-16T00:14:09.814 INFO:teuthology.orchestra.run.smithi123.stderr:nodeep-scrub is set 2020-01-16T00:14:09.831 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:09.840 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-01-16T00:14:09.852 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:09.852+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:09.953 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:09.950+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:09.991 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:10.001 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-01-16T00:14:10.044 INFO:tasks.rados.rados.0.smithi123.stdout:1706: done (15 left) 2020-01-16T00:14:10.045 INFO:tasks.rados.rados.0.smithi123.stdout:1723: write (excl) oid 1 current snap is 0 2020-01-16T00:14:10.045 INFO:tasks.rados.rados.0.smithi123.stdout:1723: seq_num 618 ranges {786726=728475,2280813=105913} 2020-01-16T00:14:10.052 INFO:tasks.rados.rados.0.smithi123.stdout:1723: writing smithi12313934-1 from 786726 to 1515201 tid 1 2020-01-16T00:14:10.056 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:10.054+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:10.057 INFO:tasks.rados.rados.0.smithi123.stdout:1723: writing smithi12313934-1 from 2280813 to 2386726 tid 2 2020-01-16T00:14:10.058 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:10.058 INFO:tasks.rados.rados.0.smithi123.stdout:1708: expect (ObjNum 565 snap 0 seq_num 565) 2020-01-16T00:14:10.150 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:10.157 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-01-16T00:14:10.158 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:10.152+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:10.258 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:10.254+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:10.299 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:10.308 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-01-16T00:14:10.357 INFO:tasks.rados.rados.0.smithi123.stdout:1709: expect (ObjNum 564 snap 0 seq_num 564) 2020-01-16T00:14:10.359 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:10.356+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:10.445 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:10.453 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-01-16T00:14:10.460 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:10.456+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:10.561 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:10.560+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:10.592 INFO:tasks.rados.rados.0.smithi123.stdout:1712: finishing write tid 1 to smithi12313934-28 2020-01-16T00:14:10.592 INFO:tasks.rados.rados.0.smithi123.stdout:1711: expect (ObjNum 576 snap 0 seq_num 576) 2020-01-16T00:14:10.596 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:10.605 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-01-16T00:14:10.662 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:10.660+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:10.724 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:10.726 INFO:tasks.rados.rados.0.smithi123.stdout:1713: expect (ObjNum 607 snap 0 seq_num 607) 2020-01-16T00:14:10.734 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-01-16T00:14:10.762 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:10.758+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:10.833 INFO:tasks.rados.rados.0.smithi123.stdout:1712: finishing write tid 2 to smithi12313934-28 2020-01-16T00:14:10.833 INFO:tasks.rados.rados.0.smithi123.stdout:1712: finishing write tid 4 to smithi12313934-28 2020-01-16T00:14:10.834 INFO:tasks.rados.rados.0.smithi123.stdout:1712: finishing write tid 5 to smithi12313934-28 2020-01-16T00:14:10.834 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 28 v 217 (ObjNum 613 snap 0 seq_num 613) dirty exists 2020-01-16T00:14:10.834 INFO:tasks.rados.rados.0.smithi123.stdout:1712: left oid 28 (ObjNum 613 snap 0 seq_num 613) 2020-01-16T00:14:10.834 INFO:tasks.rados.rados.0.smithi123.stdout:1715: finishing write tid 1 to smithi12313934-7 2020-01-16T00:14:10.834 INFO:tasks.rados.rados.0.smithi123.stdout:1715: finishing write tid 2 to smithi12313934-7 2020-01-16T00:14:10.834 INFO:tasks.rados.rados.0.smithi123.stdout:1715: finishing write tid 4 to smithi12313934-7 2020-01-16T00:14:10.834 INFO:tasks.rados.rados.0.smithi123.stdout:1708: done (15 left) 2020-01-16T00:14:10.834 INFO:tasks.rados.rados.0.smithi123.stdout:1709: done (14 left) 2020-01-16T00:14:10.835 INFO:tasks.rados.rados.0.smithi123.stdout:1710: done (13 left) 2020-01-16T00:14:10.835 INFO:tasks.rados.rados.0.smithi123.stdout:1711: done (12 left) 2020-01-16T00:14:10.835 INFO:tasks.rados.rados.0.smithi123.stdout:1712: done (11 left) 2020-01-16T00:14:10.835 INFO:tasks.rados.rados.0.smithi123.stdout:1713: done (10 left) 2020-01-16T00:14:10.835 INFO:tasks.rados.rados.0.smithi123.stdout:1724: read oid 46 snap -1 2020-01-16T00:14:10.835 INFO:tasks.rados.rados.0.smithi123.stdout:1724: expect (ObjNum 603 snap 0 seq_num 603) 2020-01-16T00:14:10.835 INFO:tasks.rados.rados.0.smithi123.stdout:1725: write (excl) oid 4 current snap is 0 2020-01-16T00:14:10.835 INFO:tasks.rados.rados.0.smithi123.stdout:1725: seq_num 619 ranges {403533=403724,1539295=652201} 2020-01-16T00:14:10.842 INFO:tasks.rados.rados.0.smithi123.stdout:1725: writing smithi12313934-4 from 403533 to 807257 tid 1 2020-01-16T00:14:10.850 INFO:tasks.rados.rados.0.smithi123.stdout:1725: writing smithi12313934-4 from 1539295 to 2191496 tid 2 2020-01-16T00:14:10.850 INFO:tasks.rados.rados.0.smithi123.stdout:1726: read oid 30 snap -1 2020-01-16T00:14:10.850 INFO:tasks.rados.rados.0.smithi123.stdout:1726: expect (ObjNum 540 snap 0 seq_num 540) 2020-01-16T00:14:10.851 INFO:tasks.rados.rados.0.smithi123.stdout:1727: write (excl) oid 28 current snap is 0 2020-01-16T00:14:10.851 INFO:tasks.rados.rados.0.smithi123.stdout:1727: seq_num 620 ranges {420340=478973,1597777=512212} 2020-01-16T00:14:10.856 INFO:tasks.rados.rados.0.smithi123.stdout:1727: writing smithi12313934-28 from 420340 to 899313 tid 1 2020-01-16T00:14:10.863 INFO:tasks.rados.rados.0.smithi123.stdout:1727: writing smithi12313934-28 from 1597777 to 2109989 tid 2 2020-01-16T00:14:10.863 INFO:tasks.rados.rados.0.smithi123.stdout:1728: write oid 49 current snap is 0 2020-01-16T00:14:10.863 INFO:tasks.rados.rados.0.smithi123.stdout:1728: seq_num 621 ranges {437147=554222,1656259=655870} 2020-01-16T00:14:10.863 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:10.862+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:10.868 INFO:tasks.rados.rados.0.smithi123.stdout:1728: writing smithi12313934-49 from 437147 to 991369 tid 1 2020-01-16T00:14:10.875 INFO:tasks.rados.rados.0.smithi123.stdout:1728: writing smithi12313934-49 from 1656259 to 2312129 tid 2 2020-01-16T00:14:10.875 INFO:tasks.rados.rados.0.smithi123.stdout:1729: write (excl) oid 32 current snap is 0 2020-01-16T00:14:10.875 INFO:tasks.rados.rados.0.smithi123.stdout:1729: seq_num 622 ranges {453954=629471,1598388=515881} 2020-01-16T00:14:10.881 INFO:tasks.rados.rados.0.smithi123.stdout:1729: writing smithi12313934-32 from 453954 to 1083425 tid 1 2020-01-16T00:14:10.887 INFO:tasks.rados.rados.0.smithi123.stdout:1729: writing smithi12313934-32 from 1598388 to 2114269 tid 2 2020-01-16T00:14:10.887 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:10.888 INFO:tasks.rados.rados.0.smithi123.stdout:1715: finishing write tid 5 to smithi12313934-7 2020-01-16T00:14:10.888 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 7 v 240 (ObjNum 614 snap 0 seq_num 614) dirty exists 2020-01-16T00:14:10.888 INFO:tasks.rados.rados.0.smithi123.stdout:1715: left oid 7 (ObjNum 614 snap 0 seq_num 614) 2020-01-16T00:14:10.888 INFO:tasks.rados.rados.0.smithi123.stdout:1715: done (15 left) 2020-01-16T00:14:10.888 INFO:tasks.rados.rados.0.smithi123.stdout:1730: read oid 25 snap -1 2020-01-16T00:14:10.888 INFO:tasks.rados.rados.0.smithi123.stdout:1730: expect (ObjNum 571 snap 0 seq_num 571) 2020-01-16T00:14:10.888 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:10.889 INFO:tasks.rados.rados.0.smithi123.stdout:1714: expect (ObjNum 593 snap 0 seq_num 593) 2020-01-16T00:14:10.890 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:10.903 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-01-16T00:14:10.963 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:10.962+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:11.009 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:11.018 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-01-16T00:14:11.064 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:11.062+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:11.123 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:11.133 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-01-16T00:14:11.137 INFO:tasks.rados.rados.0.smithi123.stdout:1716: finishing write tid 1 to smithi12313934-27 2020-01-16T00:14:11.137 INFO:tasks.rados.rados.0.smithi123.stdout:1718: expect (ObjNum 522 snap 0 seq_num 522) 2020-01-16T00:14:11.165 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:11.162+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:11.237 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:11.247 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-01-16T00:14:11.266 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:11.262+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:11.330 INFO:tasks.rados.rados.0.smithi123.stdout:1717: expect (ObjNum 572 snap 0 seq_num 572) 2020-01-16T00:14:11.355 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-01-16T00:14:11.355 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:11.366 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-01-16T00:14:11.367 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:11.364+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:11.468 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:11.464+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:11.475 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:11.483 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:14:11.486 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-01-16T00:14:11.502 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:14:11.520 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:14:11.540 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:14:11.563 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:14:11.565 INFO:tasks.rados.rados.0.smithi123.stdout:1716: finishing write tid 2 to smithi12313934-27 2020-01-16T00:14:11.566 INFO:tasks.rados.rados.0.smithi123.stdout:1716: finishing write tid 4 to smithi12313934-27 2020-01-16T00:14:11.566 INFO:tasks.rados.rados.0.smithi123.stdout:1716: finishing write tid 5 to smithi12313934-27 2020-01-16T00:14:11.566 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 27 v 167 (ObjNum 615 snap 0 seq_num 615) dirty exists 2020-01-16T00:14:11.566 INFO:tasks.rados.rados.0.smithi123.stdout:1716: left oid 27 (ObjNum 615 snap 0 seq_num 615) 2020-01-16T00:14:11.566 INFO:tasks.rados.rados.0.smithi123.stdout:1719: finishing write tid 1 to smithi12313934-40 2020-01-16T00:14:11.567 INFO:tasks.rados.rados.0.smithi123.stdout:1719: finishing write tid 2 to smithi12313934-40 2020-01-16T00:14:11.567 INFO:tasks.rados.rados.0.smithi123.stdout:1719: finishing write tid 4 to smithi12313934-40 2020-01-16T00:14:11.567 INFO:tasks.rados.rados.0.smithi123.stdout:1719: finishing write tid 5 to smithi12313934-40 2020-01-16T00:14:11.567 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 40 v 274 (ObjNum 616 snap 0 seq_num 616) dirty exists 2020-01-16T00:14:11.567 INFO:tasks.rados.rados.0.smithi123.stdout:1719: left oid 40 (ObjNum 616 snap 0 seq_num 616) 2020-01-16T00:14:11.567 INFO:tasks.rados.rados.0.smithi123.stdout:1720: expect (ObjNum 556 snap 0 seq_num 556) 2020-01-16T00:14:11.569 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:11.566+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:11.585 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:14:11.615 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:14:11.617 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-01-16T00:14:11.617 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-01-16T00:14:11.617 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-01-16T00:14:11.618 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-01-16T00:14:11.618 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-01-16T00:14:11.618 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENXIO: problem getting command descriptions from osd.5 2020-01-16T00:14:11.618 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: problem getting command descriptions from osd.5 2020-01-16T00:14:11.618 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-01-16T00:14:11.618 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-01-16T00:14:11.626 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-01-16T00:14:11.626 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-01-16T00:14:11.635 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:11.636 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:14:11.636 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:14:11.636 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:14:11.636 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:11.645 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-01-16T00:14:11.690 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:11.688+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:11.751 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:11.751 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:14:11.751 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:14:11.751 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:14:11.752 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.752 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1654 3.a 3:5069665e:::smithi12313934-2:head [write 1614880~581041] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:14:11.752 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.714839+0000", 2020-01-16T00:14:11.752 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 79.034059455000005, 2020-01-16T00:14:11.752 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013588093000000001, 2020-01-16T00:14:11.752 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:11.752 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:11.752 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:11.753 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:11.753 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:11.753 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1654 2020-01-16T00:14:11.753 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.753 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:11.753 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.753 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:11.753 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.714839+0000", 2020-01-16T00:14:11.754 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:11.754 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.754 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.754 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:11.754 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.714839+0000", 2020-01-16T00:14:11.754 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:11.754 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.755 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.755 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:11.755 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.714836+0000", 2020-01-16T00:14:11.755 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00054657999999999996 2020-01-16T00:14:11.755 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.755 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.755 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:11.755 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.715382+0000", 2020-01-16T00:14:11.756 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7743000000000001e-05 2020-01-16T00:14:11.756 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.756 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.756 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:11.756 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.715420+0000", 2020-01-16T00:14:11.756 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.221e-05 2020-01-16T00:14:11.756 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.756 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.757 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:11.757 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.715432+0000", 2020-01-16T00:14:11.757 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2487000000000001e-05 2020-01-16T00:14:11.757 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.757 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.757 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.757 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.715485+0000", 2020-01-16T00:14:11.758 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000726284 2020-01-16T00:14:11.758 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.758 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.758 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:11.758 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.716211+0000", 2020-01-16T00:14:11.758 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0067930899999999999 2020-01-16T00:14:11.759 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.759 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.759 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.759 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.723004+0000", 2020-01-16T00:14:11.759 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012579400000000001 2020-01-16T00:14:11.759 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.759 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.760 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:11.760 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.723130+0000", 2020-01-16T00:14:11.760 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013465800000000001 2020-01-16T00:14:11.760 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.760 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.760 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:11.760 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.723265+0000", 2020-01-16T00:14:11.761 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023307250000000001 2020-01-16T00:14:11.761 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.761 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.761 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:11.761 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.725595+0000", 2020-01-16T00:14:11.761 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012582139999999999 2020-01-16T00:14:11.761 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.761 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.762 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:11.762 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.726853+0000", 2020-01-16T00:14:11.762 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014479110000000001 2020-01-16T00:14:11.762 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.762 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.762 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:11.762 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.728301+0000", 2020-01-16T00:14:11.762 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6866000000000007e-05 2020-01-16T00:14:11.763 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.763 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.763 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:11.763 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.728368+0000", 2020-01-16T00:14:11.763 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8740999999999999e-05 2020-01-16T00:14:11.763 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.763 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.764 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:11.764 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.728427+0000", 2020-01-16T00:14:11.764 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013588093000000001 2020-01-16T00:14:11.764 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:11.764 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:11.764 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:11.764 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.764 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.765 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1655 3.a 3:5069665e:::smithi12313934-2:head [write 2785791~71399] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:14:11.765 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.718297+0000", 2020-01-16T00:14:11.765 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 79.030601602000004, 2020-01-16T00:14:11.765 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013817354, 2020-01-16T00:14:11.765 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:11.765 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:11.766 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:11.766 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:11.766 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:11.766 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1655 2020-01-16T00:14:11.766 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.766 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:11.766 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.766 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:11.767 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718297+0000", 2020-01-16T00:14:11.767 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:11.767 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.767 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.767 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:11.767 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718297+0000", 2020-01-16T00:14:11.767 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:11.767 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.768 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.768 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:11.768 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718294+0000", 2020-01-16T00:14:11.768 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000103558 2020-01-16T00:14:11.768 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.769 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.769 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:11.769 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718397+0000", 2020-01-16T00:14:11.769 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9287e-05 2020-01-16T00:14:11.769 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.769 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.769 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:11.769 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718417+0000", 2020-01-16T00:14:11.770 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6680000000000001e-06 2020-01-16T00:14:11.770 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.770 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.770 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:11.770 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718424+0000", 2020-01-16T00:14:11.770 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2701999999999994e-05 2020-01-16T00:14:11.770 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.771 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.771 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.771 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718517+0000", 2020-01-16T00:14:11.771 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3893000000000006e-05 2020-01-16T00:14:11.771 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.771 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.771 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:11.771 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718581+0000", 2020-01-16T00:14:11.772 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0059630079999999997 2020-01-16T00:14:11.772 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.772 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.772 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.772 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.724544+0000", 2020-01-16T00:14:11.772 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000309108 2020-01-16T00:14:11.772 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.772 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.773 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:11.773 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.724853+0000", 2020-01-16T00:14:11.773 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.005469794 2020-01-16T00:14:11.773 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.773 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.773 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.773 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.730323+0000", 2020-01-16T00:14:11.773 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018506100000000001 2020-01-16T00:14:11.774 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.774 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.774 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:11.774 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.730508+0000", 2020-01-16T00:14:11.774 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016729099999999999 2020-01-16T00:14:11.774 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.774 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.774 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:11.775 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.730675+0000", 2020-01-16T00:14:11.775 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00094193300000000005 2020-01-16T00:14:11.775 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.775 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.775 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:11.775 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731617+0000", 2020-01-16T00:14:11.775 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000214613 2020-01-16T00:14:11.775 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.776 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.776 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:11.776 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731832+0000", 2020-01-16T00:14:11.776 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018326400000000001 2020-01-16T00:14:11.776 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.776 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.776 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:11.777 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732015+0000", 2020-01-16T00:14:11.777 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0392e-05 2020-01-16T00:14:11.777 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.777 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.777 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:11.777 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732075+0000", 2020-01-16T00:14:11.777 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8677999999999998e-05 2020-01-16T00:14:11.778 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.778 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.778 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:11.778 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732114+0000", 2020-01-16T00:14:11.778 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013817354 2020-01-16T00:14:11.778 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:11.779 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:11.779 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:11.779 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.779 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.779 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1656 3.a 3:5069665e:::smithi12313934-2:head [setxattr _header (55) in=7b,truncate 2857190] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:14:11.779 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.718453+0000", 2020-01-16T00:14:11.779 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 79.030444920999997, 2020-01-16T00:14:11.780 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014083462, 2020-01-16T00:14:11.780 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:11.780 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:11.780 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:11.780 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:11.780 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:11.780 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1656 2020-01-16T00:14:11.780 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.781 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:11.781 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.781 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:11.781 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718453+0000", 2020-01-16T00:14:11.781 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:11.781 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.781 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.781 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:11.782 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718453+0000", 2020-01-16T00:14:11.782 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999828 2020-01-16T00:14:11.782 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.782 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.782 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:11.782 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718436+0000", 2020-01-16T00:14:11.782 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4051999999999999e-05 2020-01-16T00:14:11.783 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.783 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.783 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:11.783 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718460+0000", 2020-01-16T00:14:11.783 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6380000000000004e-06 2020-01-16T00:14:11.783 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.784 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.784 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:11.784 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718468+0000", 2020-01-16T00:14:11.784 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6750000000000001e-06 2020-01-16T00:14:11.784 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.784 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.784 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:11.784 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718472+0000", 2020-01-16T00:14:11.785 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016218499999999999 2020-01-16T00:14:11.785 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.785 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.785 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.785 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718635+0000", 2020-01-16T00:14:11.785 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2765e-05 2020-01-16T00:14:11.785 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.786 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.786 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:11.786 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718697+0000", 2020-01-16T00:14:11.786 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0062239249999999999 2020-01-16T00:14:11.786 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.786 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.786 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.787 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.724921+0000", 2020-01-16T00:14:11.787 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1527999999999999e-05 2020-01-16T00:14:11.787 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.787 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.787 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:11.787 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.724973+0000", 2020-01-16T00:14:11.787 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0060569719999999999 2020-01-16T00:14:11.787 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.788 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.788 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.788 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731030+0000", 2020-01-16T00:14:11.788 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5859999999999996e-05 2020-01-16T00:14:11.788 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.788 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.789 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:11.789 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731096+0000", 2020-01-16T00:14:11.789 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000112018 2020-01-16T00:14:11.789 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.789 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.789 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:11.789 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731208+0000", 2020-01-16T00:14:11.790 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000945889 2020-01-16T00:14:11.790 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.790 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.790 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:11.790 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732154+0000", 2020-01-16T00:14:11.790 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0834000000000003e-05 2020-01-16T00:14:11.790 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.790 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.791 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:11.791 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732184+0000", 2020-01-16T00:14:11.791 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028587600000000001 2020-01-16T00:14:11.791 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.791 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.791 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:11.791 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732470+0000", 2020-01-16T00:14:11.791 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1953999999999999e-05 2020-01-16T00:14:11.792 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.792 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.792 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:11.792 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732502+0000", 2020-01-16T00:14:11.792 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4644999999999998e-05 2020-01-16T00:14:11.792 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.792 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.792 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:11.793 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732537+0000", 2020-01-16T00:14:11.793 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014083462 2020-01-16T00:14:11.793 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:11.793 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:11.793 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:11.793 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.793 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.793 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1657 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-01-16T00:14:11.794 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.718495+0000", 2020-01-16T00:14:11.794 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 79.030403256, 2020-01-16T00:14:11.794 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014299630000000001, 2020-01-16T00:14:11.794 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:11.794 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:11.794 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:11.794 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:11.794 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:11.795 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1657 2020-01-16T00:14:11.795 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.795 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:11.795 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.795 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:11.795 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718495+0000", 2020-01-16T00:14:11.795 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:11.796 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.796 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.796 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:11.796 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718495+0000", 2020-01-16T00:14:11.796 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:11.796 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.796 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.796 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:11.797 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718494+0000", 2020-01-16T00:14:11.797 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8860000000000003e-06 2020-01-16T00:14:11.797 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.797 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.797 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:11.797 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718499+0000", 2020-01-16T00:14:11.797 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9786000000000002e-05 2020-01-16T00:14:11.798 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.798 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.798 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:11.798 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718539+0000", 2020-01-16T00:14:11.798 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1869999999999999e-05 2020-01-16T00:14:11.798 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.798 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.798 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:11.799 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718561+0000", 2020-01-16T00:14:11.799 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000172206 2020-01-16T00:14:11.799 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.799 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.799 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.799 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718733+0000", 2020-01-16T00:14:11.800 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0662999999999998e-05 2020-01-16T00:14:11.800 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.800 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.800 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:11.800 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718793+0000", 2020-01-16T00:14:11.800 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.006217958 2020-01-16T00:14:11.800 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.800 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.801 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.801 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.725011+0000", 2020-01-16T00:14:11.801 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1625999999999996e-05 2020-01-16T00:14:11.801 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.801 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.801 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:11.801 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.725073+0000", 2020-01-16T00:14:11.802 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0063923119999999998 2020-01-16T00:14:11.802 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.802 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.802 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.802 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731465+0000", 2020-01-16T00:14:11.802 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2534000000000005e-05 2020-01-16T00:14:11.803 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.803 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.803 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:11.803 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731558+0000", 2020-01-16T00:14:11.803 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010238809999999999 2020-01-16T00:14:11.803 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.803 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.803 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.804 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732582+0000", 2020-01-16T00:14:11.804 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000100918 2020-01-16T00:14:11.804 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.804 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.804 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:11.804 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732683+0000", 2020-01-16T00:14:11.804 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000112064 2020-01-16T00:14:11.804 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.805 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.805 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:11.805 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732795+0000", 2020-01-16T00:14:11.805 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014299630000000001 2020-01-16T00:14:11.805 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:11.805 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:11.805 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:11.806 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.806 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.806 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1917 3.a 3:5069665e:::smithi12313934-2:head [write 2113302~674388] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:14:11.806 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:57.330712+0000", 2020-01-16T00:14:11.806 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 74.418186741, 2020-01-16T00:14:11.806 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012781043000000001, 2020-01-16T00:14:11.806 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:11.807 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:11.807 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:11.807 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:11.807 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:11.807 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1917 2020-01-16T00:14:11.807 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.807 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:11.807 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.808 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:11.808 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.330712+0000", 2020-01-16T00:14:11.808 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:11.808 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.808 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.808 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:11.808 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.330712+0000", 2020-01-16T00:14:11.808 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:11.809 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.809 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.809 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:11.809 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.330709+0000", 2020-01-16T00:14:11.809 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00071550499999999996 2020-01-16T00:14:11.810 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.810 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.810 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:11.810 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331424+0000", 2020-01-16T00:14:11.810 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3354e-05 2020-01-16T00:14:11.810 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.810 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.810 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:11.811 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331447+0000", 2020-01-16T00:14:11.811 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8200000000000008e-06 2020-01-16T00:14:11.811 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.811 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.811 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:11.811 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331457+0000", 2020-01-16T00:14:11.811 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5557999999999997e-05 2020-01-16T00:14:11.812 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.812 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.812 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.812 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331533+0000", 2020-01-16T00:14:11.812 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030925000000000001 2020-01-16T00:14:11.812 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.812 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.812 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:11.813 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.331842+0000", 2020-01-16T00:14:11.813 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0078507860000000002 2020-01-16T00:14:11.813 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.813 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.813 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.813 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.339693+0000", 2020-01-16T00:14:11.813 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9321000000000007e-05 2020-01-16T00:14:11.813 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.814 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.814 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:11.814 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.339792+0000", 2020-01-16T00:14:11.814 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015147700000000001 2020-01-16T00:14:11.814 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.814 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.814 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:11.815 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.339944+0000", 2020-01-16T00:14:11.815 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002663066 2020-01-16T00:14:11.815 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.815 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.815 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:11.815 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.342607+0000", 2020-01-16T00:14:11.816 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0001552 2020-01-16T00:14:11.816 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.816 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.816 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:11.816 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.342762+0000", 2020-01-16T00:14:11.816 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00063636900000000002 2020-01-16T00:14:11.817 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.817 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.817 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:11.817 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343398+0000", 2020-01-16T00:14:11.817 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9687000000000001e-05 2020-01-16T00:14:11.817 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.817 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.818 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:11.818 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343438+0000", 2020-01-16T00:14:11.818 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4647e-05 2020-01-16T00:14:11.818 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.818 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.818 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:11.818 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:57.343493+0000", 2020-01-16T00:14:11.819 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012781043000000001 2020-01-16T00:14:11.819 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:11.819 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:11.819 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:11.819 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.819 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.820 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2505 3.a 3:5069665e:::smithi12313934-2:head [write 1862260~479427] snapc 0=[] ondisk+write+known_if_redirected e37)", 2020-01-16T00:14:11.820 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:11.286548+0000", 2020-01-16T00:14:11.820 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 60.462350573999998, 2020-01-16T00:14:11.820 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013314061, 2020-01-16T00:14:11.820 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:11.820 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:11.820 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:11.821 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:11.821 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:11.821 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2505 2020-01-16T00:14:11.821 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.821 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:11.821 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.821 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:11.821 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.286548+0000", 2020-01-16T00:14:11.822 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:11.822 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.822 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.822 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:11.822 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.286548+0000", 2020-01-16T00:14:11.822 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999957 2020-01-16T00:14:11.822 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.823 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.823 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:11.823 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.286544+0000", 2020-01-16T00:14:11.823 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042043700000000001 2020-01-16T00:14:11.823 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.823 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.823 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:11.823 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.286964+0000", 2020-01-16T00:14:11.823 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5756999999999997e-05 2020-01-16T00:14:11.824 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.824 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.824 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:11.824 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.287000+0000", 2020-01-16T00:14:11.824 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8960000000000001e-05 2020-01-16T00:14:11.824 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.824 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.824 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:11.825 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.287019+0000", 2020-01-16T00:14:11.825 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.7219000000000005e-05 2020-01-16T00:14:11.825 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.825 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.825 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.825 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.287116+0000", 2020-01-16T00:14:11.825 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033208899999999998 2020-01-16T00:14:11.825 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.826 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.826 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:11.826 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.287448+0000", 2020-01-16T00:14:11.826 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0082732420000000001 2020-01-16T00:14:11.826 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.826 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.827 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.827 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.295721+0000", 2020-01-16T00:14:11.827 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017259999999999999 2020-01-16T00:14:11.827 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.827 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.827 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:11.827 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.295894+0000", 2020-01-16T00:14:11.827 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00029238000000000002 2020-01-16T00:14:11.828 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.828 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.828 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:11.828 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.296186+0000", 2020-01-16T00:14:11.828 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0034626409999999998 2020-01-16T00:14:11.828 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.829 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.829 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:11.829 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299649+0000", 2020-01-16T00:14:11.829 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2300000000000002e-06 2020-01-16T00:14:11.829 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.829 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.830 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:11.830 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299656+0000", 2020-01-16T00:14:11.830 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1561000000000001e-05 2020-01-16T00:14:11.830 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.830 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.830 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:11.831 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299738+0000", 2020-01-16T00:14:11.831 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8491000000000005e-05 2020-01-16T00:14:11.831 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.831 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.831 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:11.831 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299806+0000", 2020-01-16T00:14:11.831 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5541000000000002e-05 2020-01-16T00:14:11.831 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.832 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.832 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:11.832 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:11.299862+0000", 2020-01-16T00:14:11.832 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013314061 2020-01-16T00:14:11.832 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:11.832 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:11.832 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:11.832 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.833 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.833 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2791 3.a 3:54f2c117:::smithi12313934-41:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e41)", 2020-01-16T00:14:11.833 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:17.213167+0000", 2020-01-16T00:14:11.833 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 54.535731562999999, 2020-01-16T00:14:11.833 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016024037000000001, 2020-01-16T00:14:11.833 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:11.833 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:11.834 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:11.834 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:11.834 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:11.834 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2791 2020-01-16T00:14:11.834 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.834 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:11.834 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.834 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:11.835 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213167+0000", 2020-01-16T00:14:11.835 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:11.835 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.835 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.835 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:11.835 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213167+0000", 2020-01-16T00:14:11.835 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:11.836 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.836 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.836 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:11.836 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213166+0000", 2020-01-16T00:14:11.836 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6030000000000004e-06 2020-01-16T00:14:11.836 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.836 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.837 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:11.837 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213172+0000", 2020-01-16T00:14:11.837 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0929999999999997e-06 2020-01-16T00:14:11.837 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.837 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.837 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:11.838 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213179+0000", 2020-01-16T00:14:11.838 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4299999999999997e-06 2020-01-16T00:14:11.838 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.838 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.838 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:11.838 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213184+0000", 2020-01-16T00:14:11.838 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012106695000000001 2020-01-16T00:14:11.839 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.839 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.839 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.839 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.225291+0000", 2020-01-16T00:14:11.839 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020863699999999999 2020-01-16T00:14:11.839 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.839 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.839 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:11.840 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.225499+0000", 2020-01-16T00:14:11.840 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0036913050000000002 2020-01-16T00:14:11.840 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.840 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.840 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:11.840 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.229191+0000", 2020-01-16T00:14:11.841 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016024037000000001 2020-01-16T00:14:11.841 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:11.841 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:11.841 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:11.841 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.841 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.841 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2792 3.a 3:54f2c117:::smithi12313934-41:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e41)", 2020-01-16T00:14:11.842 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:17.213190+0000", 2020-01-16T00:14:11.842 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 54.535708089000003, 2020-01-16T00:14:11.842 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017165949, 2020-01-16T00:14:11.842 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:11.842 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:11.842 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:11.843 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:11.843 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:11.843 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2792 2020-01-16T00:14:11.843 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.843 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:11.844 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.844 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:11.844 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213190+0000", 2020-01-16T00:14:11.844 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:11.844 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.845 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.845 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:11.845 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213190+0000", 2020-01-16T00:14:11.845 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:11.845 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.846 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.846 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:11.846 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213189+0000", 2020-01-16T00:14:11.846 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2699999999999998e-06 2020-01-16T00:14:11.846 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.847 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.847 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:11.847 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213194+0000", 2020-01-16T00:14:11.847 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8600000000000004e-06 2020-01-16T00:14:11.847 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.848 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.848 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:11.848 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213201+0000", 2020-01-16T00:14:11.848 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8500000000000002e-06 2020-01-16T00:14:11.848 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.849 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.849 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:11.849 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213205+0000", 2020-01-16T00:14:11.849 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016028924999999999 2020-01-16T00:14:11.849 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.850 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.850 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.850 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.229234+0000", 2020-01-16T00:14:11.850 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019773999999999999 2020-01-16T00:14:11.851 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.851 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.851 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:11.851 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.229432+0000", 2020-01-16T00:14:11.851 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00092414400000000001 2020-01-16T00:14:11.852 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.852 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.852 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:11.852 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.230356+0000", 2020-01-16T00:14:11.852 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017165949 2020-01-16T00:14:11.853 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:11.853 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:11.853 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:11.853 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.853 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.853 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2793 3.a 3:54f2c117:::smithi12313934-41:head [read 0~3210170,checksum] snapc 0=[] ondisk+read+known_if_redirected e41)", 2020-01-16T00:14:11.853 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:17.213211+0000", 2020-01-16T00:14:11.854 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 54.535687543000002, 2020-01-16T00:14:11.854 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019069413, 2020-01-16T00:14:11.854 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:11.854 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:11.854 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:11.854 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:11.854 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:11.855 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2793 2020-01-16T00:14:11.855 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.855 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:11.855 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.855 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:11.855 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213211+0000", 2020-01-16T00:14:11.855 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:11.855 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.856 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.856 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:11.856 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213211+0000", 2020-01-16T00:14:11.856 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:11.856 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.856 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.856 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:11.856 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213210+0000", 2020-01-16T00:14:11.857 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.66e-06 2020-01-16T00:14:11.857 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.857 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.857 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:11.857 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213216+0000", 2020-01-16T00:14:11.857 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.9469999999999997e-06 2020-01-16T00:14:11.857 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.857 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.858 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:11.858 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213223+0000", 2020-01-16T00:14:11.858 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9259999999999999e-06 2020-01-16T00:14:11.858 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.858 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.858 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:11.858 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213228+0000", 2020-01-16T00:14:11.859 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017171046999999998 2020-01-16T00:14:11.859 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.859 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.859 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.859 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.230399+0000", 2020-01-16T00:14:11.859 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000189932 2020-01-16T00:14:11.859 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.860 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.860 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:11.860 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.230589+0000", 2020-01-16T00:14:11.860 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0016916539999999999 2020-01-16T00:14:11.860 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.860 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.861 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:11.861 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.232280+0000", 2020-01-16T00:14:11.861 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019069413 2020-01-16T00:14:11.861 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:11.861 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:11.861 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:11.861 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.861 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.862 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3120 3.a 3:5069665e:::smithi12313934-2:head [write 1845146~524707] snapc 0=[] ondisk+write+known_if_redirected e44)", 2020-01-16T00:14:11.862 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.262852+0000", 2020-01-16T00:14:11.862 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 46.486046360000003, 2020-01-16T00:14:11.862 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014991175000000001, 2020-01-16T00:14:11.862 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:11.862 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:11.862 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:11.863 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:11.863 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:11.863 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3120 2020-01-16T00:14:11.863 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.863 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:11.863 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.863 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:11.863 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.262852+0000", 2020-01-16T00:14:11.864 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:11.864 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.864 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.864 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:11.864 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.262852+0000", 2020-01-16T00:14:11.864 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:11.864 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.865 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.865 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:11.865 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.262849+0000", 2020-01-16T00:14:11.865 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00058469900000000005 2020-01-16T00:14:11.865 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.865 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.865 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:11.865 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263434+0000", 2020-01-16T00:14:11.866 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.6650000000000001e-05 2020-01-16T00:14:11.866 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.866 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.866 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:11.866 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263461+0000", 2020-01-16T00:14:11.866 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3478e-05 2020-01-16T00:14:11.866 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.867 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.867 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:11.867 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263474+0000", 2020-01-16T00:14:11.867 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7824000000000002e-05 2020-01-16T00:14:11.867 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.867 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.867 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.867 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263522+0000", 2020-01-16T00:14:11.868 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001365464 2020-01-16T00:14:11.868 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.868 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.868 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:11.868 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.264888+0000", 2020-01-16T00:14:11.868 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0081368250000000003 2020-01-16T00:14:11.868 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.869 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.869 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.869 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.273024+0000", 2020-01-16T00:14:11.869 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6130000000000006e-05 2020-01-16T00:14:11.869 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.869 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.869 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:11.869 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.273090+0000", 2020-01-16T00:14:11.870 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000207507 2020-01-16T00:14:11.870 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.870 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.870 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:11.870 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.273298+0000", 2020-01-16T00:14:11.870 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0035649980000000002 2020-01-16T00:14:11.870 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.871 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.871 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:11.871 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276863+0000", 2020-01-16T00:14:11.871 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.1459999999999995e-05 2020-01-16T00:14:11.871 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.871 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.872 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:11.872 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276954+0000", 2020-01-16T00:14:11.872 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00076363899999999998 2020-01-16T00:14:11.872 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.872 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.872 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:11.872 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277718+0000", 2020-01-16T00:14:11.872 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0154999999999995e-05 2020-01-16T00:14:11.873 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.873 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.873 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:11.873 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277788+0000", 2020-01-16T00:14:11.873 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4914000000000003e-05 2020-01-16T00:14:11.873 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.873 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.873 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:11.874 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277843+0000", 2020-01-16T00:14:11.874 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014991175000000001 2020-01-16T00:14:11.874 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:11.874 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:11.874 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:11.874 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.874 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.874 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3121 3.a 3:5069665e:::smithi12313934-2:head [setxattr _header (55) in=7b,truncate 2369853] snapc 0=[] ondisk+write+known_if_redirected e44)", 2020-01-16T00:14:11.875 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.263494+0000", 2020-01-16T00:14:11.875 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 46.485404072000001, 2020-01-16T00:14:11.875 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014559274000000001, 2020-01-16T00:14:11.875 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:11.875 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:11.875 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:11.875 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:11.875 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:11.876 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3121 2020-01-16T00:14:11.876 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.876 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:11.876 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.876 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:11.876 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263494+0000", 2020-01-16T00:14:11.876 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:11.876 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.877 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.877 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:11.877 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263494+0000", 2020-01-16T00:14:11.877 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:11.877 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.877 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.878 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:11.878 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263493+0000", 2020-01-16T00:14:11.878 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3359999999999995e-06 2020-01-16T00:14:11.878 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.878 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.878 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:11.879 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263501+0000", 2020-01-16T00:14:11.879 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4100000000000002e-06 2020-01-16T00:14:11.879 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.879 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.879 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:11.880 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263509+0000", 2020-01-16T00:14:11.880 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3839999999999996e-06 2020-01-16T00:14:11.880 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.880 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.880 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:11.880 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263514+0000", 2020-01-16T00:14:11.881 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001430295 2020-01-16T00:14:11.881 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.881 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.882 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.882 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.264944+0000", 2020-01-16T00:14:11.882 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9710999999999999e-05 2020-01-16T00:14:11.882 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.882 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.883 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:11.883 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.264994+0000", 2020-01-16T00:14:11.883 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010036188999999999 2020-01-16T00:14:11.883 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.883 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.883 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.883 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.275030+0000", 2020-01-16T00:14:11.883 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1743000000000007e-05 2020-01-16T00:14:11.884 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.884 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.884 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:11.884 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.275112+0000", 2020-01-16T00:14:11.884 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013036100000000001 2020-01-16T00:14:11.884 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.885 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.885 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:11.885 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.275242+0000", 2020-01-16T00:14:11.885 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0016795549999999999 2020-01-16T00:14:11.885 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.885 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.885 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:11.885 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276922+0000", 2020-01-16T00:14:11.886 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018479400000000001 2020-01-16T00:14:11.886 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.886 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.886 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:11.886 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277107+0000", 2020-01-16T00:14:11.886 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00083585300000000001 2020-01-16T00:14:11.886 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.887 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.887 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:11.887 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277943+0000", 2020-01-16T00:14:11.887 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1719999999999999e-05 2020-01-16T00:14:11.887 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.887 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.887 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:11.887 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277994+0000", 2020-01-16T00:14:11.888 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9212999999999997e-05 2020-01-16T00:14:11.888 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.888 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.888 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:11.888 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.278054+0000", 2020-01-16T00:14:11.888 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014559274000000001 2020-01-16T00:14:11.888 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:11.889 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:11.889 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:11.889 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.889 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.889 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3122 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e44)", 2020-01-16T00:14:11.889 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.263525+0000", 2020-01-16T00:14:11.889 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 46.485372884, 2020-01-16T00:14:11.889 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014731833999999999, 2020-01-16T00:14:11.890 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:11.890 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:11.890 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:11.890 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:11.890 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:11.890 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3122 2020-01-16T00:14:11.890 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.891 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:11.891 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.891 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:11.891 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263525+0000", 2020-01-16T00:14:11.891 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:11.891 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.892 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.892 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:11.892 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263525+0000", 2020-01-16T00:14:11.892 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:11.892 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.892 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.892 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:11.892 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263525+0000", 2020-01-16T00:14:11.893 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5000000000000001e-06 2020-01-16T00:14:11.893 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.893 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.893 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:11.893 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263529+0000", 2020-01-16T00:14:11.893 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4669999999999998e-06 2020-01-16T00:14:11.893 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.894 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.894 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:11.894 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263537+0000", 2020-01-16T00:14:11.894 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0819999999999998e-06 2020-01-16T00:14:11.894 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.894 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.894 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:11.894 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263542+0000", 2020-01-16T00:14:11.895 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0015024960000000001 2020-01-16T00:14:11.895 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.895 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.895 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.895 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.265044+0000", 2020-01-16T00:14:11.895 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6650999999999994e-05 2020-01-16T00:14:11.895 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.895 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.896 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:11.896 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.265131+0000", 2020-01-16T00:14:11.896 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010495310000000001 2020-01-16T00:14:11.896 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.896 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.896 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.896 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.275626+0000", 2020-01-16T00:14:11.897 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9536000000000004e-05 2020-01-16T00:14:11.897 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.897 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.897 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:11.897 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.275726+0000", 2020-01-16T00:14:11.897 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023719489999999999 2020-01-16T00:14:11.897 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.897 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.898 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.898 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.278098+0000", 2020-01-16T00:14:11.898 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3718000000000001e-05 2020-01-16T00:14:11.898 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.898 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.898 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:11.898 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.278171+0000", 2020-01-16T00:14:11.898 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5859999999999994e-05 2020-01-16T00:14:11.899 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.899 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.899 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:11.899 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.278257+0000", 2020-01-16T00:14:11.899 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014731833999999999 2020-01-16T00:14:11.899 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:11.899 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:11.899 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:11.900 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.900 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.900 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3627 3.a 3:5069665e:::smithi12313934-2:head [write 1749580~611149] snapc 0=[] ondisk+write+known_if_redirected e48)", 2020-01-16T00:14:11.900 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:39.462779+0000", 2020-01-16T00:14:11.900 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 32.286119071999998, 2020-01-16T00:14:11.900 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020946985000000001, 2020-01-16T00:14:11.900 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:11.901 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:11.901 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:11.901 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:11.901 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:11.901 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3627 2020-01-16T00:14:11.901 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.901 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:11.902 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.902 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:11.902 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.462779+0000", 2020-01-16T00:14:11.902 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:11.902 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.902 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.902 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:11.903 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.462779+0000", 2020-01-16T00:14:11.903 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:11.903 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.903 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.903 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:11.903 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.462777+0000", 2020-01-16T00:14:11.904 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00064136 2020-01-16T00:14:11.904 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.904 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.904 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:11.904 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.463418+0000", 2020-01-16T00:14:11.904 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4607e-05 2020-01-16T00:14:11.904 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.904 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.905 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:11.905 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.463443+0000", 2020-01-16T00:14:11.905 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0346999999999999e-05 2020-01-16T00:14:11.905 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.905 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.905 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:11.905 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.463453+0000", 2020-01-16T00:14:11.905 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000100988 2020-01-16T00:14:11.906 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.906 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.906 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.906 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.463554+0000", 2020-01-16T00:14:11.906 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023059280000000001 2020-01-16T00:14:11.906 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.906 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.907 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:11.907 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.465860+0000", 2020-01-16T00:14:11.907 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011316961 2020-01-16T00:14:11.907 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.907 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.907 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.907 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.477177+0000", 2020-01-16T00:14:11.907 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000132402 2020-01-16T00:14:11.908 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.908 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.908 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:11.908 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.477309+0000", 2020-01-16T00:14:11.908 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000223606 2020-01-16T00:14:11.908 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.908 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.908 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:11.909 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.477533+0000", 2020-01-16T00:14:11.909 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0057105209999999997 2020-01-16T00:14:11.909 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.909 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.909 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:11.909 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483244+0000", 2020-01-16T00:14:11.909 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000193031 2020-01-16T00:14:11.909 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.910 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.910 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:11.910 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483437+0000", 2020-01-16T00:14:11.910 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0902999999999995e-05 2020-01-16T00:14:11.910 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.910 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.910 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:11.910 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483507+0000", 2020-01-16T00:14:11.911 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000129412 2020-01-16T00:14:11.911 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.911 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.911 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:11.911 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483637+0000", 2020-01-16T00:14:11.911 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.9345999999999995e-05 2020-01-16T00:14:11.911 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.911 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.912 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:11.912 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483726+0000", 2020-01-16T00:14:11.912 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020946985000000001 2020-01-16T00:14:11.912 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:11.912 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:11.913 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:11.913 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.913 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.913 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3628 3.a 3:5069665e:::smithi12313934-2:head [write 2853383~452450] snapc 0=[] ondisk+write+known_if_redirected e48)", 2020-01-16T00:14:11.913 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:39.468178+0000", 2020-01-16T00:14:11.913 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 32.280720363, 2020-01-16T00:14:11.913 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015705872999999999, 2020-01-16T00:14:11.913 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:11.914 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:11.914 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:11.914 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:11.914 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:11.914 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3628 2020-01-16T00:14:11.914 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.914 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:11.915 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.915 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:11.915 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468178+0000", 2020-01-16T00:14:11.915 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:11.915 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.915 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.915 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:11.916 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468178+0000", 2020-01-16T00:14:11.916 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:11.916 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.916 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.916 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:11.916 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468175+0000", 2020-01-16T00:14:11.917 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00065948000000000005 2020-01-16T00:14:11.917 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.917 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.917 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:11.917 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468834+0000", 2020-01-16T00:14:11.917 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3557000000000002e-05 2020-01-16T00:14:11.917 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.918 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.918 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:11.918 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468858+0000", 2020-01-16T00:14:11.918 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2931e-05 2020-01-16T00:14:11.918 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.918 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.918 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:11.919 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468871+0000", 2020-01-16T00:14:11.919 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000123113 2020-01-16T00:14:11.919 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.919 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.919 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.919 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468994+0000", 2020-01-16T00:14:11.920 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017615999999999999 2020-01-16T00:14:11.920 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.920 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.920 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:11.920 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.469170+0000", 2020-01-16T00:14:11.920 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010336908000000001 2020-01-16T00:14:11.920 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.920 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.921 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.921 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.479507+0000", 2020-01-16T00:14:11.921 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015856400000000001 2020-01-16T00:14:11.921 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.921 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.921 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:11.921 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.479665+0000", 2020-01-16T00:14:11.922 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030344600000000002 2020-01-16T00:14:11.922 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.922 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.922 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:11.922 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.479969+0000", 2020-01-16T00:14:11.922 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0037935389999999999 2020-01-16T00:14:11.922 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.923 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.923 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:11.923 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483762+0000", 2020-01-16T00:14:11.923 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3170999999999997e-05 2020-01-16T00:14:11.923 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.923 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.924 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:11.924 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483806+0000", 2020-01-16T00:14:11.924 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3821000000000001e-05 2020-01-16T00:14:11.924 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.924 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.924 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:11.924 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483829+0000", 2020-01-16T00:14:11.924 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0076000000000001e-05 2020-01-16T00:14:11.925 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.925 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.925 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:11.925 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483859+0000", 2020-01-16T00:14:11.925 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.438e-05 2020-01-16T00:14:11.925 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.925 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.925 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:11.926 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483884+0000", 2020-01-16T00:14:11.926 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015705872999999999 2020-01-16T00:14:11.926 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:11.926 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:11.926 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:11.926 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.926 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.927 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3629 3.a 3:5069665e:::smithi12313934-2:head [setxattr _header (55) in=7b,truncate 3529536] snapc 0=[] ondisk+write+known_if_redirected e48)", 2020-01-16T00:14:11.927 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:39.468888+0000", 2020-01-16T00:14:11.927 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 32.280010742000002, 2020-01-16T00:14:11.927 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015108511, 2020-01-16T00:14:11.927 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:11.927 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:11.927 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:11.927 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:11.928 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:11.928 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3629 2020-01-16T00:14:11.928 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.928 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:11.928 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.928 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:11.928 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468888+0000", 2020-01-16T00:14:11.928 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:11.929 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.929 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.929 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:11.929 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468888+0000", 2020-01-16T00:14:11.929 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:11.929 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.929 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.929 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:11.930 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468886+0000", 2020-01-16T00:14:11.930 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5599999999999996e-06 2020-01-16T00:14:11.930 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.930 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.930 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:11.930 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468894+0000", 2020-01-16T00:14:11.930 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5279999999999998e-06 2020-01-16T00:14:11.930 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.931 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.931 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:11.931 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468901+0000", 2020-01-16T00:14:11.931 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2290000000000002e-06 2020-01-16T00:14:11.931 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.931 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.931 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:11.931 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468907+0000", 2020-01-16T00:14:11.932 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030197500000000001 2020-01-16T00:14:11.932 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.932 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.932 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.932 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.469209+0000", 2020-01-16T00:14:11.932 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6659999999999997e-05 2020-01-16T00:14:11.933 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.933 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.933 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:11.933 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.469255+0000", 2020-01-16T00:14:11.933 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013117904999999999 2020-01-16T00:14:11.933 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.933 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.934 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.934 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.482373+0000", 2020-01-16T00:14:11.934 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1260000000000006e-05 2020-01-16T00:14:11.934 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.934 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.934 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:11.934 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.482434+0000", 2020-01-16T00:14:11.934 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014433 2020-01-16T00:14:11.935 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.935 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.935 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:11.935 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.482579+0000", 2020-01-16T00:14:11.935 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012198160000000001 2020-01-16T00:14:11.935 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.935 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.935 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:11.936 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483799+0000", 2020-01-16T00:14:11.936 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.188e-05 2020-01-16T00:14:11.936 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.936 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.936 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:11.936 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483810+0000", 2020-01-16T00:14:11.936 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000114863 2020-01-16T00:14:11.937 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.937 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.937 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:11.937 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483925+0000", 2020-01-16T00:14:11.937 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0861000000000001e-05 2020-01-16T00:14:11.937 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.937 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.938 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:11.938 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483956+0000", 2020-01-16T00:14:11.938 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9894999999999997e-05 2020-01-16T00:14:11.938 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.938 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.938 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:11.939 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483996+0000", 2020-01-16T00:14:11.939 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015108511 2020-01-16T00:14:11.939 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:11.939 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:11.939 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:11.939 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.939 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.940 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3630 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e48)", 2020-01-16T00:14:11.940 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:39.468967+0000", 2020-01-16T00:14:11.940 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 32.279931607999998, 2020-01-16T00:14:11.940 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015160579, 2020-01-16T00:14:11.940 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:11.940 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:11.940 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:11.940 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:11.941 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:11.941 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3630 2020-01-16T00:14:11.941 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.941 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:11.941 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.941 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:11.941 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468967+0000", 2020-01-16T00:14:11.941 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:11.942 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.942 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.942 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:11.942 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468967+0000", 2020-01-16T00:14:11.942 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:11.942 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.943 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.943 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:11.943 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468966+0000", 2020-01-16T00:14:11.943 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3430000000000003e-06 2020-01-16T00:14:11.943 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.943 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.943 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:11.944 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468971+0000", 2020-01-16T00:14:11.944 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6577000000000001e-05 2020-01-16T00:14:11.944 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.944 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.944 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:11.944 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468988+0000", 2020-01-16T00:14:11.944 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.378e-06 2020-01-16T00:14:11.945 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.945 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.945 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:11.945 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468993+0000", 2020-01-16T00:14:11.945 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000320601 2020-01-16T00:14:11.945 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.945 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.945 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.946 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.469314+0000", 2020-01-16T00:14:11.946 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013833699999999999 2020-01-16T00:14:11.946 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.946 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.946 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:11.946 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.469452+0000", 2020-01-16T00:14:11.946 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013447838 2020-01-16T00:14:11.946 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.947 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.947 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.947 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.482900+0000", 2020-01-16T00:14:11.947 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028006899999999998 2020-01-16T00:14:11.947 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.947 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.947 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:11.948 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483180+0000", 2020-01-16T00:14:11.948 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000833594 2020-01-16T00:14:11.948 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.948 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.948 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.948 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.484013+0000", 2020-01-16T00:14:11.948 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2601e-05 2020-01-16T00:14:11.948 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.949 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.949 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:11.949 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.484066+0000", 2020-01-16T00:14:11.949 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1303999999999994e-05 2020-01-16T00:14:11.949 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.949 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.949 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:11.950 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.484127+0000", 2020-01-16T00:14:11.950 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015160579 2020-01-16T00:14:11.950 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:11.950 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:11.950 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:11.950 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.950 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.951 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4047 3.a 3:5069665e:::smithi12313934-2:head [write 1935828~635840] snapc 0=[] ondisk+write+known_if_redirected e52)", 2020-01-16T00:14:11.951 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:50.302087+0000", 2020-01-16T00:14:11.951 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 21.446811337, 2020-01-16T00:14:11.951 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014518982, 2020-01-16T00:14:11.951 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:11.951 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:11.951 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:11.952 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:11.952 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:11.952 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4047 2020-01-16T00:14:11.952 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.952 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:11.952 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.952 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:11.952 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302087+0000", 2020-01-16T00:14:11.953 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:11.953 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.953 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.953 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:11.953 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302087+0000", 2020-01-16T00:14:11.953 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:11.954 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.954 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.954 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:11.954 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302085+0000", 2020-01-16T00:14:11.954 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00057147600000000002 2020-01-16T00:14:11.954 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.954 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.954 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:11.955 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302657+0000", 2020-01-16T00:14:11.955 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6427999999999999e-05 2020-01-16T00:14:11.955 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.955 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.955 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:11.955 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302673+0000", 2020-01-16T00:14:11.955 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.127e-06 2020-01-16T00:14:11.956 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.956 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.956 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:11.956 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302680+0000", 2020-01-16T00:14:11.956 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6378000000000006e-05 2020-01-16T00:14:11.956 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.956 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.956 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.956 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302756+0000", 2020-01-16T00:14:11.957 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00039898000000000001 2020-01-16T00:14:11.957 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.957 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.957 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:11.957 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.303155+0000", 2020-01-16T00:14:11.957 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0079666870000000001 2020-01-16T00:14:11.957 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.957 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.958 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.958 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.311122+0000", 2020-01-16T00:14:11.958 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019101400000000001 2020-01-16T00:14:11.958 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.958 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.958 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:11.958 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.311313+0000", 2020-01-16T00:14:11.959 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033332499999999998 2020-01-16T00:14:11.959 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.959 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.959 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:11.959 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.311646+0000", 2020-01-16T00:14:11.959 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0029841440000000002 2020-01-16T00:14:11.959 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.959 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.960 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:11.960 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.314630+0000", 2020-01-16T00:14:11.960 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00027996800000000001 2020-01-16T00:14:11.960 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.960 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.960 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:11.961 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.314910+0000", 2020-01-16T00:14:11.961 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0015637159999999999 2020-01-16T00:14:11.961 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.961 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.961 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:11.961 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.316474+0000", 2020-01-16T00:14:11.961 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5986000000000001e-05 2020-01-16T00:14:11.962 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.962 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.962 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:11.962 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.316550+0000", 2020-01-16T00:14:11.962 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6258000000000002e-05 2020-01-16T00:14:11.962 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.962 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.963 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:11.963 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.316606+0000", 2020-01-16T00:14:11.963 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014518982 2020-01-16T00:14:11.963 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:11.963 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:11.963 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:11.963 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.964 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.964 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4048 3.a 3:5069665e:::smithi12313934-2:head [setxattr _header (55) in=7b,truncate 2588377] snapc 0=[] ondisk+write+known_if_redirected e52)", 2020-01-16T00:14:11.964 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:50.302688+0000", 2020-01-16T00:14:11.964 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 21.446210225000002, 2020-01-16T00:14:11.964 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014049927, 2020-01-16T00:14:11.964 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:11.964 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:11.965 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:11.965 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:11.965 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:11.965 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4048 2020-01-16T00:14:11.965 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.965 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:11.965 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.965 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:11.966 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302688+0000", 2020-01-16T00:14:11.966 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:11.966 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.966 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.966 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:11.966 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302688+0000", 2020-01-16T00:14:11.966 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:11.966 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.967 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.967 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:11.967 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302687+0000", 2020-01-16T00:14:11.967 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6e-06 2020-01-16T00:14:11.967 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.967 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.967 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:11.967 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302692+0000", 2020-01-16T00:14:11.968 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1999999999999996e-06 2020-01-16T00:14:11.968 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.968 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.968 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:11.968 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302696+0000", 2020-01-16T00:14:11.968 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9330000000000001e-06 2020-01-16T00:14:11.968 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.968 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.969 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:11.969 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302699+0000", 2020-01-16T00:14:11.969 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00049592099999999997 2020-01-16T00:14:11.969 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.969 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.969 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.969 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.303195+0000", 2020-01-16T00:14:11.969 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0647000000000003e-05 2020-01-16T00:14:11.970 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.970 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.970 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:11.970 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.303266+0000", 2020-01-16T00:14:11.970 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010088420000000001 2020-01-16T00:14:11.970 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.970 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.970 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.971 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.313354+0000", 2020-01-16T00:14:11.971 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000113136 2020-01-16T00:14:11.971 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.971 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.971 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:11.971 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.313467+0000", 2020-01-16T00:14:11.971 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012389000000000001 2020-01-16T00:14:11.971 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.972 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.972 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:11.972 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.313591+0000", 2020-01-16T00:14:11.972 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014897409999999999 2020-01-16T00:14:11.972 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.972 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.972 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:11.972 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.315081+0000", 2020-01-16T00:14:11.973 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00023842700000000001 2020-01-16T00:14:11.973 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.973 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.973 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:11.973 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.315319+0000", 2020-01-16T00:14:11.974 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012958449999999999 2020-01-16T00:14:11.974 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.974 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.974 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:11.974 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.316615+0000", 2020-01-16T00:14:11.975 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4045000000000002e-05 2020-01-16T00:14:11.975 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.975 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.975 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:11.975 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.316669+0000", 2020-01-16T00:14:11.976 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8827999999999999e-05 2020-01-16T00:14:11.976 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.976 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.976 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:11.976 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.316738+0000", 2020-01-16T00:14:11.977 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014049927 2020-01-16T00:14:11.977 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:11.977 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:11.977 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:11.977 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.977 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.978 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4049 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e52)", 2020-01-16T00:14:11.978 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:50.302702+0000", 2020-01-16T00:14:11.978 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 21.446196703999998, 2020-01-16T00:14:11.978 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014215574999999999, 2020-01-16T00:14:11.979 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:11.979 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:11.979 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:11.979 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:11.979 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:11.980 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4049 2020-01-16T00:14:11.980 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.980 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:11.980 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.980 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:11.981 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302702+0000", 2020-01-16T00:14:11.981 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:11.981 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.981 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.981 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:11.982 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302702+0000", 2020-01-16T00:14:11.982 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:11.982 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.982 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.982 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:11.982 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302701+0000", 2020-01-16T00:14:11.983 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0020000000000001e-06 2020-01-16T00:14:11.983 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.983 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.983 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:11.984 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302703+0000", 2020-01-16T00:14:11.984 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7740000000000002e-06 2020-01-16T00:14:11.984 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.984 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.984 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:11.985 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302707+0000", 2020-01-16T00:14:11.985 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.512e-06 2020-01-16T00:14:11.985 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.985 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.985 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:11.986 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302711+0000", 2020-01-16T00:14:11.986 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00058642799999999995 2020-01-16T00:14:11.986 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.986 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.986 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.986 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.303297+0000", 2020-01-16T00:14:11.986 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5887000000000001e-05 2020-01-16T00:14:11.987 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.987 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.987 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:11.987 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.303363+0000", 2020-01-16T00:14:11.987 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010875398999999999 2020-01-16T00:14:11.987 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.987 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.987 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.988 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.314238+0000", 2020-01-16T00:14:11.988 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000187014 2020-01-16T00:14:11.988 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.988 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.988 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:11.988 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.314425+0000", 2020-01-16T00:14:11.988 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023405000000000001 2020-01-16T00:14:11.988 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.989 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.989 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.989 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.316766+0000", 2020-01-16T00:14:11.989 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0636000000000006e-05 2020-01-16T00:14:11.989 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.989 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.989 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:11.990 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.316836+0000", 2020-01-16T00:14:11.990 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0747999999999993e-05 2020-01-16T00:14:11.990 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.990 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.990 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:11.990 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.316917+0000", 2020-01-16T00:14:11.990 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014215574999999999 2020-01-16T00:14:11.990 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:11.991 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:11.991 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:11.991 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.991 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.991 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4600 3.a 3:5069665e:::smithi12313934-2:head [write 1791765~569861] snapc 0=[] ondisk+write+known_if_redirected e56)", 2020-01-16T00:14:11.991 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:03.059744+0000", 2020-01-16T00:14:11.991 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 8.6891541199999995, 2020-01-16T00:14:11.992 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017075275000000001, 2020-01-16T00:14:11.992 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:11.992 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:11.992 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:11.992 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:11.992 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:11.992 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4600 2020-01-16T00:14:11.992 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.993 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:11.993 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.993 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:11.993 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.059744+0000", 2020-01-16T00:14:11.993 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:11.993 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.993 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.993 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:11.994 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.059744+0000", 2020-01-16T00:14:11.994 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999843 2020-01-16T00:14:11.994 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.994 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.994 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:11.994 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.059728+0000", 2020-01-16T00:14:11.995 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00054167699999999996 2020-01-16T00:14:11.995 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.995 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.995 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:11.995 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.060270+0000", 2020-01-16T00:14:11.995 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6693999999999998e-05 2020-01-16T00:14:11.995 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.996 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.996 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:11.996 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.060307+0000", 2020-01-16T00:14:11.996 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8753000000000002e-05 2020-01-16T00:14:11.996 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.996 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.996 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:11.996 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.060326+0000", 2020-01-16T00:14:11.997 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000114177 2020-01-16T00:14:11.997 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.997 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.997 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.997 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.060440+0000", 2020-01-16T00:14:11.997 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001642382 2020-01-16T00:14:11.997 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.997 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.998 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:11.998 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.062082+0000", 2020-01-16T00:14:11.998 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0093492149999999993 2020-01-16T00:14:11.998 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.998 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.998 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:11.998 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.071431+0000", 2020-01-16T00:14:11.998 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013135600000000001 2020-01-16T00:14:11.999 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.999 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.999 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:11.999 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.071563+0000", 2020-01-16T00:14:11.999 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00032728299999999999 2020-01-16T00:14:11.999 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:11.999 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:11.999 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:12.000 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.071890+0000", 2020-01-16T00:14:12.000 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0043339759999999998 2020-01-16T00:14:12.000 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:12.000 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:12.000 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:12.000 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.076224+0000", 2020-01-16T00:14:12.000 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000343462 2020-01-16T00:14:12.000 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:12.001 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:12.001 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:12.001 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.076567+0000", 2020-01-16T00:14:12.001 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1668000000000006e-05 2020-01-16T00:14:12.001 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:12.001 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:12.001 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:12.002 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.076639+0000", 2020-01-16T00:14:12.002 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8572000000000004e-05 2020-01-16T00:14:12.002 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:12.002 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:12.002 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:12.002 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.076738+0000", 2020-01-16T00:14:12.002 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1816000000000004e-05 2020-01-16T00:14:12.002 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:12.003 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:12.003 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:12.003 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.076819+0000", 2020-01-16T00:14:12.003 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017075275000000001 2020-01-16T00:14:12.003 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:12.003 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:12.003 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:12.003 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:12.004 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:12.004 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:12.005 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2020-01-16T00:14:12.008 INFO:tasks.rados.rados.0.smithi123.stdout:1721: finishing write tid 1 to smithi12313934-5 2020-01-16T00:14:12.008 INFO:tasks.rados.rados.0.smithi123.stdout:1721: finishing write tid 2 to smithi12313934-5 2020-01-16T00:14:12.009 INFO:tasks.rados.rados.0.smithi123.stdout:1722: expect (ObjNum 596 snap 0 seq_num 596) 2020-01-16T00:14:12.009 INFO:tasks.rados.rados.0.smithi123.stdout:1721: finishing write tid 4 to smithi12313934-5 2020-01-16T00:14:12.009 INFO:tasks.rados.rados.0.smithi123.stdout:1721: finishing write tid 5 to smithi12313934-5 2020-01-16T00:14:12.009 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 5 v 268 (ObjNum 617 snap 0 seq_num 617) dirty exists 2020-01-16T00:14:12.009 INFO:tasks.rados.rados.0.smithi123.stdout:1721: left oid 5 (ObjNum 617 snap 0 seq_num 617) 2020-01-16T00:14:12.009 INFO:tasks.rados.rados.0.smithi123.stdout:1723: finishing write tid 1 to smithi12313934-1 2020-01-16T00:14:12.009 INFO:tasks.rados.rados.0.smithi123.stdout:1723: finishing write tid 2 to smithi12313934-1 2020-01-16T00:14:12.009 INFO:tasks.rados.rados.0.smithi123.stdout:1723: finishing write tid 4 to smithi12313934-1 2020-01-16T00:14:12.010 INFO:tasks.rados.rados.0.smithi123.stdout:1714: done (15 left) 2020-01-16T00:14:12.010 INFO:tasks.rados.rados.0.smithi123.stdout:1716: done (14 left) 2020-01-16T00:14:12.010 INFO:tasks.rados.rados.0.smithi123.stdout:1717: done (13 left) 2020-01-16T00:14:12.010 INFO:tasks.rados.rados.0.smithi123.stdout:1718: done (12 left) 2020-01-16T00:14:12.010 INFO:tasks.rados.rados.0.smithi123.stdout:1719: done (11 left) 2020-01-16T00:14:12.010 INFO:tasks.rados.rados.0.smithi123.stdout:1720: done (10 left) 2020-01-16T00:14:12.010 INFO:tasks.rados.rados.0.smithi123.stdout:1721: done (9 left) 2020-01-16T00:14:12.010 INFO:tasks.rados.rados.0.smithi123.stdout:1722: done (8 left) 2020-01-16T00:14:12.011 INFO:tasks.rados.rados.0.smithi123.stdout:1731: read oid 10 snap -1 2020-01-16T00:14:12.011 INFO:tasks.rados.rados.0.smithi123.stdout:1731: expect deleted 2020-01-16T00:14:12.011 INFO:tasks.rados.rados.0.smithi123.stdout:1732: write (excl) oid 47 current snap is 0 2020-01-16T00:14:12.011 INFO:tasks.rados.rados.0.smithi123.stdout:1732: seq_num 623 ranges {470761=704720,1656870=659539} 2020-01-16T00:14:12.011 INFO:tasks.rados.rados.0.smithi123.stdout:1732: writing smithi12313934-47 from 470761 to 1175481 tid 1 2020-01-16T00:14:12.011 INFO:tasks.rados.rados.0.smithi123.stdout:1732: writing smithi12313934-47 from 1656870 to 2316409 tid 2 2020-01-16T00:14:12.011 INFO:tasks.rados.rados.0.smithi123.stdout:1733: read oid 27 snap -1 2020-01-16T00:14:12.011 INFO:tasks.rados.rados.0.smithi123.stdout:1733: expect (ObjNum 615 snap 0 seq_num 615) 2020-01-16T00:14:12.012 INFO:tasks.rados.rados.0.smithi123.stdout:1734: write (excl) oid 15 current snap is 0 2020-01-16T00:14:12.012 INFO:tasks.rados.rados.0.smithi123.stdout:1734: seq_num 624 ranges {487568=496322,1431705=519550} 2020-01-16T00:14:12.012 INFO:tasks.rados.rados.0.smithi123.stdout:1734: writing smithi12313934-15 from 487568 to 983890 tid 1 2020-01-16T00:14:12.012 INFO:tasks.rados.rados.0.smithi123.stdout:1734: writing smithi12313934-15 from 1431705 to 1951255 tid 2 2020-01-16T00:14:12.012 INFO:tasks.rados.rados.0.smithi123.stdout:1735: write oid 16 current snap is 0 2020-01-16T00:14:12.012 INFO:tasks.rados.rados.0.smithi123.stdout:1735: seq_num 625 ranges {504375=571571,1490187=663208} 2020-01-16T00:14:12.012 INFO:tasks.rados.rados.0.smithi123.stdout:1735: writing smithi12313934-16 from 504375 to 1075946 tid 1 2020-01-16T00:14:12.012 INFO:tasks.rados.rados.0.smithi123.stdout:1735: writing smithi12313934-16 from 1490187 to 2153395 tid 2 2020-01-16T00:14:12.013 INFO:tasks.rados.rados.0.smithi123.stdout:1736: read oid 22 snap -1 2020-01-16T00:14:12.013 INFO:tasks.rados.rados.0.smithi123.stdout:1736: expect (ObjNum 575 snap 0 seq_num 575) 2020-01-16T00:14:12.013 INFO:tasks.rados.rados.0.smithi123.stdout:1737: read oid 33 snap -1 2020-01-16T00:14:12.013 INFO:tasks.rados.rados.0.smithi123.stdout:1737: expect deleted 2020-01-16T00:14:12.013 INFO:tasks.rados.rados.0.smithi123.stdout:1738: read oid 36 snap -1 2020-01-16T00:14:12.014 INFO:tasks.rados.rados.0.smithi123.stdout:1738: expect deleted 2020-01-16T00:14:12.014 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:12.014 INFO:tasks.rados.rados.0.smithi123.stdout:1723: finishing write tid 5 to smithi12313934-1 2020-01-16T00:14:12.014 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 1 v 326 (ObjNum 618 snap 0 seq_num 618) dirty exists 2020-01-16T00:14:12.014 INFO:tasks.rados.rados.0.smithi123.stdout:1723: left oid 1 (ObjNum 618 snap 0 seq_num 618) 2020-01-16T00:14:12.015 INFO:tasks.rados.rados.0.smithi123.stdout:1723: done (15 left) 2020-01-16T00:14:12.015 INFO:tasks.rados.rados.0.smithi123.stdout:1739: read oid 43 snap -1 2020-01-16T00:14:12.015 INFO:tasks.rados.rados.0.smithi123.stdout:1739: expect (ObjNum 607 snap 0 seq_num 607) 2020-01-16T00:14:12.015 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:12.015 INFO:tasks.rados.rados.0.smithi123.stdout:1724: expect (ObjNum 603 snap 0 seq_num 603) 2020-01-16T00:14:12.015 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:12.002+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:12.016 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-01-16T00:14:12.106 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:12.104+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:12.114 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:12.114 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:14:12.115 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:14:12.115 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:12.123 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-01-16T00:14:12.127 INFO:tasks.rados.rados.0.smithi123.stdout:1725: finishing write tid 1 to smithi12313934-4 2020-01-16T00:14:12.127 INFO:tasks.rados.rados.0.smithi123.stdout:1725: finishing write tid 2 to smithi12313934-4 2020-01-16T00:14:12.128 INFO:tasks.rados.rados.0.smithi123.stdout:1725: finishing write tid 4 to smithi12313934-4 2020-01-16T00:14:12.128 INFO:tasks.rados.rados.0.smithi123.stdout:1725: finishing write tid 5 to smithi12313934-4 2020-01-16T00:14:12.128 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 4 v 196 (ObjNum 619 snap 0 seq_num 619) dirty exists 2020-01-16T00:14:12.128 INFO:tasks.rados.rados.0.smithi123.stdout:1725: left oid 4 (ObjNum 619 snap 0 seq_num 619) 2020-01-16T00:14:12.128 INFO:tasks.rados.rados.0.smithi123.stdout:1726: expect (ObjNum 540 snap 0 seq_num 540) 2020-01-16T00:14:12.207 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:12.204+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:12.236 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:12.236 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:14:12.237 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:14:12.237 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:14:12.237 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:12.245 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-01-16T00:14:12.335 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:12.334+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:12.391 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:12.392 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:14:12.392 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:14:12.392 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:14:12.392 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.392 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:939 3.4 3:2bdd9e88:::smithi12313934-4:head [write 1681102~474010] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:14:12.392 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.908392+0000", 2020-01-16T00:14:12.393 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 94.48172434, 2020-01-16T00:14:12.393 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019294697, 2020-01-16T00:14:12.393 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:12.393 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:12.393 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:12.393 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:12.393 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:12.394 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 939 2020-01-16T00:14:12.394 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.394 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:12.394 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.394 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:12.394 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908392+0000", 2020-01-16T00:14:12.394 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:12.394 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.395 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.395 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:12.395 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908392+0000", 2020-01-16T00:14:12.395 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:12.395 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.395 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.395 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:12.395 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908390+0000", 2020-01-16T00:14:12.396 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015440399999999999 2020-01-16T00:14:12.396 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.396 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.396 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:12.396 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908545+0000", 2020-01-16T00:14:12.396 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3926e-05 2020-01-16T00:14:12.396 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.396 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.397 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:12.397 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908559+0000", 2020-01-16T00:14:12.397 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2200000000000003e-06 2020-01-16T00:14:12.397 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.397 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.397 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:12.397 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908566+0000", 2020-01-16T00:14:12.398 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020594799999999998 2020-01-16T00:14:12.398 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.398 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.398 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.398 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910625+0000", 2020-01-16T00:14:12.398 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118672 2020-01-16T00:14:12.398 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.398 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.399 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:12.399 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910744+0000", 2020-01-16T00:14:12.399 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031940969999999999 2020-01-16T00:14:12.399 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.399 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.399 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.399 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.913938+0000", 2020-01-16T00:14:12.399 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000309621 2020-01-16T00:14:12.400 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.400 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.400 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:12.400 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.914248+0000", 2020-01-16T00:14:12.400 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.008818579 2020-01-16T00:14:12.400 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.400 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.401 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.401 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.923066+0000", 2020-01-16T00:14:12.401 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000108989 2020-01-16T00:14:12.401 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.401 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.401 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:12.401 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.923175+0000", 2020-01-16T00:14:12.401 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019128299999999999 2020-01-16T00:14:12.402 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.402 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.402 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:14:12.402 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.923367+0000", 2020-01-16T00:14:12.402 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0032083760000000002 2020-01-16T00:14:12.402 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.402 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.403 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:12.403 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926575+0000", 2020-01-16T00:14:12.403 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2475999999999999e-05 2020-01-16T00:14:12.403 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.403 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.403 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:12.403 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926607+0000", 2020-01-16T00:14:12.404 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00099399800000000006 2020-01-16T00:14:12.404 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.404 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.404 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:12.404 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927601+0000", 2020-01-16T00:14:12.404 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0958999999999999e-05 2020-01-16T00:14:12.404 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.404 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.405 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:12.405 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927642+0000", 2020-01-16T00:14:12.405 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4747999999999999e-05 2020-01-16T00:14:12.405 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.405 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.405 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:12.405 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927687+0000", 2020-01-16T00:14:12.405 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019294697 2020-01-16T00:14:12.406 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.406 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:12.406 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.406 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.406 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.406 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:940 3.4 3:2bdd9e88:::smithi12313934-4:head [write 2687912~589420] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:14:12.406 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.915144+0000", 2020-01-16T00:14:12.407 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 94.474972972000003, 2020-01-16T00:14:12.407 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016092538999999999, 2020-01-16T00:14:12.407 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:12.407 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:12.407 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:12.407 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:12.407 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:12.407 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 940 2020-01-16T00:14:12.408 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.408 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:12.408 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.408 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:12.408 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915144+0000", 2020-01-16T00:14:12.408 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:12.409 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.409 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.409 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:12.409 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915144+0000", 2020-01-16T00:14:12.409 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:12.409 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.409 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.410 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:12.410 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915141+0000", 2020-01-16T00:14:12.410 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016366300000000001 2020-01-16T00:14:12.410 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.410 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.410 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:12.410 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915305+0000", 2020-01-16T00:14:12.410 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4847e-05 2020-01-16T00:14:12.411 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.411 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.411 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:12.411 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915320+0000", 2020-01-16T00:14:12.411 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2599999999999999e-06 2020-01-16T00:14:12.411 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.411 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.412 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:12.412 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915327+0000", 2020-01-16T00:14:12.412 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4282000000000005e-05 2020-01-16T00:14:12.412 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.412 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.412 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.412 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915401+0000", 2020-01-16T00:14:12.412 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4554000000000006e-05 2020-01-16T00:14:12.413 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.413 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.413 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:12.413 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915466+0000", 2020-01-16T00:14:12.413 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0090821900000000004 2020-01-16T00:14:12.413 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.413 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.413 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.414 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.924548+0000", 2020-01-16T00:14:12.414 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000114696 2020-01-16T00:14:12.414 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.414 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.414 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:12.414 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.924663+0000", 2020-01-16T00:14:12.414 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000168651 2020-01-16T00:14:12.414 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.415 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.415 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:14:12.415 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.924832+0000", 2020-01-16T00:14:12.415 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018253430000000001 2020-01-16T00:14:12.415 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.415 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.415 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:12.415 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926657+0000", 2020-01-16T00:14:12.416 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9957999999999999e-05 2020-01-16T00:14:12.416 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.416 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.416 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:12.416 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926697+0000", 2020-01-16T00:14:12.416 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0044685250000000001 2020-01-16T00:14:12.416 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.416 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.417 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:12.417 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931165+0000", 2020-01-16T00:14:12.417 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9211000000000001e-05 2020-01-16T00:14:12.417 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.417 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.417 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:12.417 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931205+0000", 2020-01-16T00:14:12.418 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1699999999999998e-05 2020-01-16T00:14:12.418 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.418 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.418 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:12.418 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931236+0000", 2020-01-16T00:14:12.419 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016092538999999999 2020-01-16T00:14:12.419 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.419 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:12.419 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.419 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.419 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.419 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:941 3.4 3:2bdd9e88:::smithi12313934-4:head [setxattr _header (55) in=7b,truncate 3277332] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:14:12.420 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.915341+0000", 2020-01-16T00:14:12.420 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 94.474776153999997, 2020-01-16T00:14:12.420 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016603769000000001, 2020-01-16T00:14:12.420 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:12.420 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:12.420 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:12.420 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:12.421 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:12.421 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 941 2020-01-16T00:14:12.421 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.421 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:12.421 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.421 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:12.421 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915341+0000", 2020-01-16T00:14:12.421 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:12.422 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.422 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.422 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:12.422 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915341+0000", 2020-01-16T00:14:12.422 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:12.422 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.422 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.423 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:12.423 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915340+0000", 2020-01-16T00:14:12.423 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.896e-06 2020-01-16T00:14:12.423 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.423 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.423 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:12.424 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915346+0000", 2020-01-16T00:14:12.424 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2730000000000002e-06 2020-01-16T00:14:12.424 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.424 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.424 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:12.424 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915354+0000", 2020-01-16T00:14:12.424 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0616e-05 2020-01-16T00:14:12.424 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.425 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.425 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:12.425 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915374+0000", 2020-01-16T00:14:12.425 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000133487 2020-01-16T00:14:12.425 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.425 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.425 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.425 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915508+0000", 2020-01-16T00:14:12.426 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8971000000000003e-05 2020-01-16T00:14:12.426 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.426 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.426 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:12.426 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915557+0000", 2020-01-16T00:14:12.426 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010138787999999999 2020-01-16T00:14:12.426 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.427 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.427 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.427 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.925696+0000", 2020-01-16T00:14:12.427 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012825400000000001 2020-01-16T00:14:12.427 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.427 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.427 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:12.427 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.925824+0000", 2020-01-16T00:14:12.428 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017706500000000001 2020-01-16T00:14:12.428 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.428 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.428 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:14:12.428 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926001+0000", 2020-01-16T00:14:12.428 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00079988900000000003 2020-01-16T00:14:12.428 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.429 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.429 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:12.429 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926801+0000", 2020-01-16T00:14:12.429 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119324 2020-01-16T00:14:12.429 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.429 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.429 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:12.429 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926920+0000", 2020-01-16T00:14:12.430 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048879910000000004 2020-01-16T00:14:12.430 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.430 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.430 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:12.430 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931808+0000", 2020-01-16T00:14:12.430 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6999999999999997e-05 2020-01-16T00:14:12.431 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.431 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.431 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:12.431 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931855+0000", 2020-01-16T00:14:12.431 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9092999999999996e-05 2020-01-16T00:14:12.431 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.431 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.432 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:12.432 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931944+0000", 2020-01-16T00:14:12.432 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016603769000000001 2020-01-16T00:14:12.432 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.432 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:12.432 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.432 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.432 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.433 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:942 3.4 3:2bdd9e88:::smithi12313934-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:14:12.433 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.915389+0000", 2020-01-16T00:14:12.433 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 94.474727787999996, 2020-01-16T00:14:12.433 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016820563, 2020-01-16T00:14:12.433 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:12.433 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:12.433 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:12.433 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:12.434 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:12.434 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 942 2020-01-16T00:14:12.434 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.434 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:12.434 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.434 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:12.434 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915389+0000", 2020-01-16T00:14:12.435 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:12.435 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.435 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.435 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:12.435 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915389+0000", 2020-01-16T00:14:12.435 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:12.435 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.435 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.436 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:12.436 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915388+0000", 2020-01-16T00:14:12.436 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.683e-06 2020-01-16T00:14:12.436 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.436 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.436 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:12.436 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915391+0000", 2020-01-16T00:14:12.436 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4839999999999998e-06 2020-01-16T00:14:12.437 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.437 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.437 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:12.437 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915399+0000", 2020-01-16T00:14:12.437 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2259999999999999e-06 2020-01-16T00:14:12.437 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.437 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.437 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:12.438 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915403+0000", 2020-01-16T00:14:12.438 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00031301199999999999 2020-01-16T00:14:12.438 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.438 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.438 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.439 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915716+0000", 2020-01-16T00:14:12.439 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2079999999999998e-05 2020-01-16T00:14:12.439 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.439 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.439 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:12.440 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915748+0000", 2020-01-16T00:14:12.440 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010547076000000001 2020-01-16T00:14:12.440 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.440 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.440 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.441 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926295+0000", 2020-01-16T00:14:12.441 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8040000000000005e-05 2020-01-16T00:14:12.441 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.441 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.441 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:12.442 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926373+0000", 2020-01-16T00:14:12.442 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056061990000000001 2020-01-16T00:14:12.442 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.442 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.442 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.443 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931979+0000", 2020-01-16T00:14:12.443 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011540399999999999 2020-01-16T00:14:12.443 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.443 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.444 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:12.444 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932095+0000", 2020-01-16T00:14:12.444 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011482899999999999 2020-01-16T00:14:12.444 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.444 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.445 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:12.445 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932210+0000", 2020-01-16T00:14:12.445 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016820563 2020-01-16T00:14:12.445 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.445 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:12.446 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.446 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.446 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.446 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:957 3.0 3:00c146d5:::smithi12313934-18:head [write 510946~632952] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:14:12.446 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.033290+0000", 2020-01-16T00:14:12.447 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 94.356826951000002, 2020-01-16T00:14:12.447 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016764102999999999, 2020-01-16T00:14:12.447 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:12.447 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:12.447 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:12.448 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:12.448 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:12.448 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 957 2020-01-16T00:14:12.448 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.448 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:12.449 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.449 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:12.449 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033290+0000", 2020-01-16T00:14:12.449 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:12.449 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.450 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.450 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:12.450 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033290+0000", 2020-01-16T00:14:12.450 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999952 2020-01-16T00:14:12.450 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.451 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.451 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:12.451 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033285+0000", 2020-01-16T00:14:12.451 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000405917 2020-01-16T00:14:12.451 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.452 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.452 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:12.452 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033691+0000", 2020-01-16T00:14:12.452 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6585999999999997e-05 2020-01-16T00:14:12.452 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.452 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.453 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:12.453 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033738+0000", 2020-01-16T00:14:12.453 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8374000000000001e-05 2020-01-16T00:14:12.453 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.453 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.454 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:12.454 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033756+0000", 2020-01-16T00:14:12.454 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013002900000000001 2020-01-16T00:14:12.454 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.455 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.455 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.455 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033886+0000", 2020-01-16T00:14:12.455 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035882600000000002 2020-01-16T00:14:12.455 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.455 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.456 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:12.456 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.034245+0000", 2020-01-16T00:14:12.456 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0072798489999999997 2020-01-16T00:14:12.456 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.457 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.457 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.457 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.041525+0000", 2020-01-16T00:14:12.457 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000140488 2020-01-16T00:14:12.457 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.458 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.458 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:12.458 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.041665+0000", 2020-01-16T00:14:12.458 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000233766 2020-01-16T00:14:12.458 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.459 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.459 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:12.459 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.041899+0000", 2020-01-16T00:14:12.459 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0035766539999999999 2020-01-16T00:14:12.459 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.459 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.460 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:12.460 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.045475+0000", 2020-01-16T00:14:12.460 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013333150000000001 2020-01-16T00:14:12.460 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.460 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.461 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:12.461 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.046809+0000", 2020-01-16T00:14:12.461 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031447860000000001 2020-01-16T00:14:12.461 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.461 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.462 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:12.462 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.049954+0000", 2020-01-16T00:14:12.462 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2318000000000003e-05 2020-01-16T00:14:12.462 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.462 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.463 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:12.463 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.050006+0000", 2020-01-16T00:14:12.463 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7972000000000001e-05 2020-01-16T00:14:12.463 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.463 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.464 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:12.464 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.050054+0000", 2020-01-16T00:14:12.464 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016764102999999999 2020-01-16T00:14:12.464 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.464 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:12.465 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.465 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.465 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.465 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1639 3.0 3:00c146d5:::smithi12313934-18:head [write 423576~401598] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:14:12.465 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.774317+0000", 2020-01-16T00:14:12.466 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 80.615800211999996, 2020-01-16T00:14:12.466 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018305219000000001, 2020-01-16T00:14:12.466 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:12.466 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:12.466 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:12.467 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:12.467 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:12.467 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1639 2020-01-16T00:14:12.467 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.467 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:12.468 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.468 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:12.468 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774317+0000", 2020-01-16T00:14:12.468 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:12.468 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.469 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.469 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:12.469 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774317+0000", 2020-01-16T00:14:12.469 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:12.469 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.470 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.470 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:12.470 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774313+0000", 2020-01-16T00:14:12.470 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000111173 2020-01-16T00:14:12.471 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.471 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.471 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:12.471 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774425+0000", 2020-01-16T00:14:12.471 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7598999999999999e-05 2020-01-16T00:14:12.472 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.472 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.472 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:12.472 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774442+0000", 2020-01-16T00:14:12.472 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0830000000000004e-06 2020-01-16T00:14:12.472 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.473 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.473 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:12.473 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774450+0000", 2020-01-16T00:14:12.473 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0017886309999999999 2020-01-16T00:14:12.473 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.474 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.474 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.474 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.776239+0000", 2020-01-16T00:14:12.475 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3789e-05 2020-01-16T00:14:12.475 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.475 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.475 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op must wait for map", 2020-01-16T00:14:12.475 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.776253+0000", 2020-01-16T00:14:12.476 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095730729999999997 2020-01-16T00:14:12.476 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.476 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.476 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.476 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.785826+0000", 2020-01-16T00:14:12.477 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014220300000000001 2020-01-16T00:14:12.477 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.477 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.477 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:12.477 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.785968+0000", 2020-01-16T00:14:12.478 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023618900000000001 2020-01-16T00:14:12.478 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.478 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.478 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:12.478 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.786204+0000", 2020-01-16T00:14:12.479 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0062297389999999998 2020-01-16T00:14:12.479 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.479 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.479 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:12.480 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792434+0000", 2020-01-16T00:14:12.480 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9635999999999998e-05 2020-01-16T00:14:12.480 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.480 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.480 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:12.481 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792473+0000", 2020-01-16T00:14:12.481 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6456000000000002e-05 2020-01-16T00:14:12.481 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.481 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.481 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:12.482 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792540+0000", 2020-01-16T00:14:12.482 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9873999999999998e-05 2020-01-16T00:14:12.482 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.482 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.482 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:12.483 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792580+0000", 2020-01-16T00:14:12.483 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1924e-05 2020-01-16T00:14:12.483 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.483 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.483 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:12.483 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792622+0000", 2020-01-16T00:14:12.483 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018305219000000001 2020-01-16T00:14:12.483 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.484 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:12.484 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.484 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.484 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.484 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1890 3.0 3:0b1f384f:::smithi12313934-33:head [write 2797915~596476] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:14:12.485 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.232534+0000", 2020-01-16T00:14:12.485 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 75.157582241, 2020-01-16T00:14:12.485 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016106137, 2020-01-16T00:14:12.485 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:12.485 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:12.485 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:12.486 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:12.486 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:12.486 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1890 2020-01-16T00:14:12.486 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.486 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:12.487 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.487 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:12.487 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232534+0000", 2020-01-16T00:14:12.487 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:12.487 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.487 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.487 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:12.487 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232534+0000", 2020-01-16T00:14:12.488 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:12.488 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.488 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.488 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:12.488 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232532+0000", 2020-01-16T00:14:12.488 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00024232100000000001 2020-01-16T00:14:12.488 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.489 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.489 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:12.489 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232774+0000", 2020-01-16T00:14:12.489 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4601e-05 2020-01-16T00:14:12.489 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.489 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.489 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:12.490 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232789+0000", 2020-01-16T00:14:12.490 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8410000000000006e-06 2020-01-16T00:14:12.490 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.490 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.490 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:12.490 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232798+0000", 2020-01-16T00:14:12.490 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9756999999999997e-05 2020-01-16T00:14:12.491 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.491 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.491 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.491 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232857+0000", 2020-01-16T00:14:12.491 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4678000000000007e-05 2020-01-16T00:14:12.491 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.491 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.491 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:12.492 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232922+0000", 2020-01-16T00:14:12.492 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033861020000000002 2020-01-16T00:14:12.492 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.492 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.492 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.492 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.236308+0000", 2020-01-16T00:14:12.492 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000568996 2020-01-16T00:14:12.492 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.493 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.493 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:12.493 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.236877+0000", 2020-01-16T00:14:12.493 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0085444240000000001 2020-01-16T00:14:12.493 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.493 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.493 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.493 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.245422+0000", 2020-01-16T00:14:12.494 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5595999999999998e-05 2020-01-16T00:14:12.494 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.494 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.494 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:12.494 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.245507+0000", 2020-01-16T00:14:12.494 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016984300000000001 2020-01-16T00:14:12.494 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.495 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.495 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:12.495 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.245677+0000", 2020-01-16T00:14:12.495 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001649495 2020-01-16T00:14:12.495 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.495 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.495 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:12.495 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247326+0000", 2020-01-16T00:14:12.496 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023308399999999999 2020-01-16T00:14:12.496 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.496 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.496 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:12.496 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247560+0000", 2020-01-16T00:14:12.496 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00099501800000000007 2020-01-16T00:14:12.496 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.497 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.497 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:12.497 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248555+0000", 2020-01-16T00:14:12.497 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5936999999999998e-05 2020-01-16T00:14:12.497 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.497 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.497 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:12.497 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248591+0000", 2020-01-16T00:14:12.498 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0108999999999997e-05 2020-01-16T00:14:12.498 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.498 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.498 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:12.498 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248641+0000", 2020-01-16T00:14:12.498 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016106137 2020-01-16T00:14:12.498 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.499 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:12.499 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.499 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.499 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.499 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1892 3.0 3:0b1f384f:::smithi12313934-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e31)", 2020-01-16T00:14:12.499 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.232856+0000", 2020-01-16T00:14:12.499 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 75.157261215000005, 2020-01-16T00:14:12.499 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016171544, 2020-01-16T00:14:12.500 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:12.500 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:12.500 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:12.500 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:12.500 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:12.500 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1892 2020-01-16T00:14:12.500 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.501 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:12.501 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.501 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:12.501 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232856+0000", 2020-01-16T00:14:12.501 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:12.501 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.501 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.501 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:12.502 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232856+0000", 2020-01-16T00:14:12.502 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:12.502 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.502 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.502 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:12.502 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232855+0000", 2020-01-16T00:14:12.502 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8640000000000004e-06 2020-01-16T00:14:12.502 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.503 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.503 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:12.503 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232860+0000", 2020-01-16T00:14:12.503 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0000000000000001e-05 2020-01-16T00:14:12.503 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.503 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.503 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:12.503 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232870+0000", 2020-01-16T00:14:12.504 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7420000000000003e-06 2020-01-16T00:14:12.504 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.504 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.504 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:12.504 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232875+0000", 2020-01-16T00:14:12.504 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017933200000000001 2020-01-16T00:14:12.504 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.505 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.505 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.505 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.233055+0000", 2020-01-16T00:14:12.505 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6269e-05 2020-01-16T00:14:12.505 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.505 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.505 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:12.505 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.233101+0000", 2020-01-16T00:14:12.506 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039780090000000002 2020-01-16T00:14:12.506 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.506 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.506 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.506 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.237079+0000", 2020-01-16T00:14:12.506 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3677000000000001e-05 2020-01-16T00:14:12.506 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.506 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.507 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:12.507 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.237133+0000", 2020-01-16T00:14:12.507 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010036671 2020-01-16T00:14:12.507 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.507 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.507 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.507 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247169+0000", 2020-01-16T00:14:12.507 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010280699999999999 2020-01-16T00:14:12.508 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.508 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.508 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:12.508 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247272+0000", 2020-01-16T00:14:12.508 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001579708 2020-01-16T00:14:12.508 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.508 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.509 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.509 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248852+0000", 2020-01-16T00:14:12.509 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9704999999999996e-05 2020-01-16T00:14:12.509 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.509 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.509 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:12.510 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248942+0000", 2020-01-16T00:14:12.510 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5557999999999996e-05 2020-01-16T00:14:12.510 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.510 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.510 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:12.510 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.249027+0000", 2020-01-16T00:14:12.510 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016171544 2020-01-16T00:14:12.510 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.511 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:12.511 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.511 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.511 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.511 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1950 3.0 3:0b1f384f:::smithi12313934-33:head [write 1889359~541737] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-01-16T00:14:12.511 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.922819+0000", 2020-01-16T00:14:12.511 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 74.467297849999994, 2020-01-16T00:14:12.512 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017502846999999998, 2020-01-16T00:14:12.512 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:12.512 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:12.512 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:12.512 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:12.512 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:12.512 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1950 2020-01-16T00:14:12.512 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.513 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:12.513 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.513 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:12.513 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922819+0000", 2020-01-16T00:14:12.513 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:12.513 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.513 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.513 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:12.514 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922819+0000", 2020-01-16T00:14:12.514 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:12.514 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.514 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.514 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:12.514 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922818+0000", 2020-01-16T00:14:12.514 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000146408 2020-01-16T00:14:12.515 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.515 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.515 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:12.515 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922964+0000", 2020-01-16T00:14:12.515 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4720000000000001e-06 2020-01-16T00:14:12.515 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.515 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.515 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:12.516 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922974+0000", 2020-01-16T00:14:12.516 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0720000000000001e-06 2020-01-16T00:14:12.516 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.516 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.516 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:12.516 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922978+0000", 2020-01-16T00:14:12.516 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6152000000000001e-05 2020-01-16T00:14:12.516 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.517 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.517 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.517 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.923024+0000", 2020-01-16T00:14:12.517 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00087368199999999997 2020-01-16T00:14:12.517 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.517 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.517 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:12.518 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.923897+0000", 2020-01-16T00:14:12.518 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011944435999999999 2020-01-16T00:14:12.518 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.518 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.518 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.518 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.935842+0000", 2020-01-16T00:14:12.519 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000144146 2020-01-16T00:14:12.519 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.519 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.519 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:12.519 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.935986+0000", 2020-01-16T00:14:12.519 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019775500000000001 2020-01-16T00:14:12.519 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.519 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.520 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:12.520 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.936184+0000", 2020-01-16T00:14:12.520 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002539342 2020-01-16T00:14:12.520 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.520 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.520 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:12.520 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938723+0000", 2020-01-16T00:14:12.520 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014193699999999999 2020-01-16T00:14:12.521 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.521 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.521 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:12.521 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938865+0000", 2020-01-16T00:14:12.521 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013829020000000001 2020-01-16T00:14:12.521 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.521 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.522 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:12.522 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.940248+0000", 2020-01-16T00:14:12.522 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2558000000000001e-05 2020-01-16T00:14:12.522 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.522 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.522 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:12.522 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.940291+0000", 2020-01-16T00:14:12.523 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1217999999999998e-05 2020-01-16T00:14:12.523 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.523 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.523 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:12.523 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.940322+0000", 2020-01-16T00:14:12.523 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017502846999999998 2020-01-16T00:14:12.523 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.523 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:12.524 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.524 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.524 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.524 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1951 3.0 3:0b1f384f:::smithi12313934-33:head [write 2955792~541902] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-01-16T00:14:12.524 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.929019+0000", 2020-01-16T00:14:12.524 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 74.461097699999996, 2020-01-16T00:14:12.524 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021625710999999999, 2020-01-16T00:14:12.525 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:12.525 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:12.525 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:12.525 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:12.525 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:12.525 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1951 2020-01-16T00:14:12.525 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.526 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:12.526 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.526 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:12.526 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929019+0000", 2020-01-16T00:14:12.526 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:12.526 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.526 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.526 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:12.527 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929019+0000", 2020-01-16T00:14:12.527 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999989 2020-01-16T00:14:12.527 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.527 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.527 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:12.527 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929008+0000", 2020-01-16T00:14:12.527 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000212513 2020-01-16T00:14:12.528 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.528 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.528 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:12.528 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929221+0000", 2020-01-16T00:14:12.528 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3229e-05 2020-01-16T00:14:12.528 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.528 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.528 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:12.529 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929234+0000", 2020-01-16T00:14:12.529 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9170000000000003e-06 2020-01-16T00:14:12.529 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.529 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.529 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:12.529 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929240+0000", 2020-01-16T00:14:12.529 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4045000000000002e-05 2020-01-16T00:14:12.529 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.530 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.530 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.530 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929294+0000", 2020-01-16T00:14:12.530 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9797999999999996e-05 2020-01-16T00:14:12.530 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.530 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.530 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:12.531 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929384+0000", 2020-01-16T00:14:12.531 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083450729999999997 2020-01-16T00:14:12.531 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.531 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.531 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.531 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.937729+0000", 2020-01-16T00:14:12.531 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043251699999999999 2020-01-16T00:14:12.531 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.532 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.532 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:12.532 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938161+0000", 2020-01-16T00:14:12.532 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095115489999999994 2020-01-16T00:14:12.532 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.532 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.532 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.532 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.947673+0000", 2020-01-16T00:14:12.533 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012395099999999999 2020-01-16T00:14:12.533 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.533 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.533 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:12.533 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.947797+0000", 2020-01-16T00:14:12.533 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00024849200000000001 2020-01-16T00:14:12.533 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.533 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.534 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:12.534 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.948045+0000", 2020-01-16T00:14:12.534 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0025081050000000001 2020-01-16T00:14:12.534 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.534 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.534 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:12.534 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950553+0000", 2020-01-16T00:14:12.535 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9760000000000003e-06 2020-01-16T00:14:12.535 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.535 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.535 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:12.535 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950558+0000", 2020-01-16T00:14:12.535 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3286999999999999e-05 2020-01-16T00:14:12.535 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.535 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.536 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:12.536 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950581+0000", 2020-01-16T00:14:12.536 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2759000000000003e-05 2020-01-16T00:14:12.536 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.536 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.536 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:12.536 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950614+0000", 2020-01-16T00:14:12.537 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0525999999999999e-05 2020-01-16T00:14:12.537 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.537 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.537 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:12.537 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950645+0000", 2020-01-16T00:14:12.537 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021625710999999999 2020-01-16T00:14:12.537 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.537 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:12.538 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.538 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.538 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.538 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1952 3.0 3:0b1f384f:::smithi12313934-33:head [setxattr _header (56) in=7b,truncate 3563084] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-01-16T00:14:12.538 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:14:12.538 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 74.460868363000003, 2020-01-16T00:14:12.538 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021534685000000001, 2020-01-16T00:14:12.539 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:12.539 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:12.539 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:12.539 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:12.539 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:12.539 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1952 2020-01-16T00:14:12.539 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.539 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:12.540 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.540 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:12.540 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:14:12.540 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:12.540 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.540 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.540 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:12.541 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:14:12.541 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:12.541 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.541 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.541 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:12.541 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:14:12.541 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6670000000000004e-06 2020-01-16T00:14:12.541 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.542 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.542 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:12.542 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929252+0000", 2020-01-16T00:14:12.542 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9380000000000002e-06 2020-01-16T00:14:12.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.542 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.542 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929256+0000", 2020-01-16T00:14:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.712e-06 2020-01-16T00:14:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929264+0000", 2020-01-16T00:14:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018030300000000001 2020-01-16T00:14:12.543 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.544 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.544 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.544 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929444+0000", 2020-01-16T00:14:12.544 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7562000000000002e-05 2020-01-16T00:14:12.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.544 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.545 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:12.545 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929502+0000", 2020-01-16T00:14:12.545 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0087297859999999998 2020-01-16T00:14:12.545 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.545 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.545 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.545 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938232+0000", 2020-01-16T00:14:12.546 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0452999999999996e-05 2020-01-16T00:14:12.546 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.546 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.546 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:12.546 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938322+0000", 2020-01-16T00:14:12.546 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011061728 2020-01-16T00:14:12.546 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.546 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.547 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.547 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.949384+0000", 2020-01-16T00:14:12.547 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000117828 2020-01-16T00:14:12.547 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.547 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.547 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:12.547 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.949502+0000", 2020-01-16T00:14:12.547 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022468499999999999 2020-01-16T00:14:12.548 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.548 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.548 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:12.548 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.949726+0000", 2020-01-16T00:14:12.548 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00083532099999999998 2020-01-16T00:14:12.548 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.548 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.548 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:12.549 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950562+0000", 2020-01-16T00:14:12.549 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013017700000000001 2020-01-16T00:14:12.549 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.549 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.549 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:12.549 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950692+0000", 2020-01-16T00:14:12.549 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9813000000000001e-05 2020-01-16T00:14:12.550 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.550 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.550 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:12.550 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950722+0000", 2020-01-16T00:14:12.550 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0592000000000001e-05 2020-01-16T00:14:12.550 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.550 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.550 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:12.551 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950752+0000", 2020-01-16T00:14:12.551 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0920000000000002e-05 2020-01-16T00:14:12.551 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.551 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.551 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:12.551 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950783+0000", 2020-01-16T00:14:12.551 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021534685000000001 2020-01-16T00:14:12.551 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.552 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:12.552 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.552 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.552 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.552 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1953 3.0 3:0b1f384f:::smithi12313934-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e32)", 2020-01-16T00:14:12.552 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:14:12.552 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 74.460835372000005, 2020-01-16T00:14:12.552 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021667537000000001, 2020-01-16T00:14:12.553 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:12.553 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:12.553 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:12.553 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:12.553 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:12.553 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1953 2020-01-16T00:14:12.553 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.553 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:12.554 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.554 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:12.554 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:14:12.554 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:12.554 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.554 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.554 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:12.555 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:14:12.555 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:12.555 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.555 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.555 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:12.555 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:14:12.555 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7030000000000002e-06 2020-01-16T00:14:12.555 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.556 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.556 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:12.556 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929284+0000", 2020-01-16T00:14:12.556 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0909999999999993e-06 2020-01-16T00:14:12.556 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.556 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.556 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:12.556 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929292+0000", 2020-01-16T00:14:12.557 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8359999999999999e-06 2020-01-16T00:14:12.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.557 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.557 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:12.557 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929297+0000", 2020-01-16T00:14:12.557 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023538199999999999 2020-01-16T00:14:12.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.558 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.558 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.558 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929532+0000", 2020-01-16T00:14:12.558 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1173000000000004e-05 2020-01-16T00:14:12.558 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.558 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.558 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:12.558 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929603+0000", 2020-01-16T00:14:12.559 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0087813509999999997 2020-01-16T00:14:12.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.559 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.559 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.559 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938384+0000", 2020-01-16T00:14:12.559 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9752000000000006e-05 2020-01-16T00:14:12.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.560 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.560 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:12.560 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938454+0000", 2020-01-16T00:14:12.560 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011707851999999999 2020-01-16T00:14:12.560 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.560 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.560 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.560 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950162+0000", 2020-01-16T00:14:12.561 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011060099999999999 2020-01-16T00:14:12.561 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.561 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.561 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:12.561 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950273+0000", 2020-01-16T00:14:12.561 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00053989700000000001 2020-01-16T00:14:12.561 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.562 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.562 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.562 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950813+0000", 2020-01-16T00:14:12.562 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8128999999999999e-05 2020-01-16T00:14:12.562 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.562 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.562 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:12.562 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950871+0000", 2020-01-16T00:14:12.563 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8242000000000001e-05 2020-01-16T00:14:12.563 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.563 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.563 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:12.563 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950949+0000", 2020-01-16T00:14:12.563 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021667537000000001 2020-01-16T00:14:12.563 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.563 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:12.564 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.564 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.564 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2555 3.0 3:0b70bb20:::smithi12313934-46:head [setxattr _header (56) in=7b,truncate 2991679] snapc 0=[] ondisk+write+known_if_redirected e37)", 2020-01-16T00:14:12.564 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.573432+0000", 2020-01-16T00:14:12.564 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 59.816684721000001, 2020-01-16T00:14:12.564 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019893926999999999, 2020-01-16T00:14:12.564 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:12.565 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:12.565 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:12.565 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:12.565 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:12.565 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2555 2020-01-16T00:14:12.565 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.565 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:12.566 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.566 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:12.566 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573432+0000", 2020-01-16T00:14:12.566 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:12.566 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.566 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.566 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:12.566 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573432+0000", 2020-01-16T00:14:12.567 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:12.567 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.567 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.567 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:12.567 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573432+0000", 2020-01-16T00:14:12.567 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4290000000000001e-06 2020-01-16T00:14:12.567 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.567 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.568 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:12.568 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573435+0000", 2020-01-16T00:14:12.568 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1300000000000001e-06 2020-01-16T00:14:12.568 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.568 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.568 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:12.568 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573438+0000", 2020-01-16T00:14:12.568 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9399999999999996e-06 2020-01-16T00:14:12.569 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.569 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.569 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:12.569 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573445+0000", 2020-01-16T00:14:12.569 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013947269999999999 2020-01-16T00:14:12.569 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.569 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.570 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.570 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.574840+0000", 2020-01-16T00:14:12.570 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003850058 2020-01-16T00:14:12.570 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.570 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.570 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:12.570 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578690+0000", 2020-01-16T00:14:12.570 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012896489000000001 2020-01-16T00:14:12.571 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.571 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.571 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.571 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591586+0000", 2020-01-16T00:14:12.571 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4757000000000001e-05 2020-01-16T00:14:12.571 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.571 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.571 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:12.572 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591681+0000", 2020-01-16T00:14:12.572 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000191535 2020-01-16T00:14:12.572 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.572 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.572 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:12.572 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591873+0000", 2020-01-16T00:14:12.572 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00069888899999999996 2020-01-16T00:14:12.572 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.573 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.573 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:12.573 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.592571+0000", 2020-01-16T00:14:12.573 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00029525900000000003 2020-01-16T00:14:12.573 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.573 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.573 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:12.573 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.592867+0000", 2020-01-16T00:14:12.574 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035243500000000001 2020-01-16T00:14:12.574 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.574 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.574 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:12.574 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593219+0000", 2020-01-16T00:14:12.574 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0954e-05 2020-01-16T00:14:12.574 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.574 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.575 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:12.575 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593270+0000", 2020-01-16T00:14:12.575 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5813000000000002e-05 2020-01-16T00:14:12.575 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.575 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.575 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:12.575 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593326+0000", 2020-01-16T00:14:12.576 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019893926999999999 2020-01-16T00:14:12.576 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.576 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:12.576 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.576 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.576 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.576 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2556 3.0 3:0b70bb20:::smithi12313934-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e37)", 2020-01-16T00:14:12.576 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.573472+0000", 2020-01-16T00:14:12.577 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 59.816645151000003, 2020-01-16T00:14:12.577 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020070068999999999, 2020-01-16T00:14:12.577 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:12.577 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:12.577 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:12.577 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:12.577 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:12.577 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2556 2020-01-16T00:14:12.578 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.578 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:12.578 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.578 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:12.578 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573472+0000", 2020-01-16T00:14:12.578 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:12.578 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.578 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.579 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:12.579 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573472+0000", 2020-01-16T00:14:12.579 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:12.579 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.579 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.579 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:12.579 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573471+0000", 2020-01-16T00:14:12.579 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4449999999999999e-06 2020-01-16T00:14:12.580 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.580 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.580 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:12.580 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573479+0000", 2020-01-16T00:14:12.580 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9789999999999998e-06 2020-01-16T00:14:12.580 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.580 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.580 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:12.581 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573485+0000", 2020-01-16T00:14:12.581 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3270000000000001e-06 2020-01-16T00:14:12.581 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.581 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.581 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:12.581 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573488+0000", 2020-01-16T00:14:12.581 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052586849999999999 2020-01-16T00:14:12.582 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.582 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.582 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.582 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578747+0000", 2020-01-16T00:14:12.582 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5667000000000007e-05 2020-01-16T00:14:12.582 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.582 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.582 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:12.583 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578812+0000", 2020-01-16T00:14:12.583 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013599539000000001 2020-01-16T00:14:12.583 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.583 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.583 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.583 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.592412+0000", 2020-01-16T00:14:12.583 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6018000000000001e-05 2020-01-16T00:14:12.583 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.584 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.584 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:12.584 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.592508+0000", 2020-01-16T00:14:12.584 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00084582199999999998 2020-01-16T00:14:12.584 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.584 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.584 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.585 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593354+0000", 2020-01-16T00:14:12.585 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4984999999999999e-05 2020-01-16T00:14:12.585 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.585 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.585 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:12.585 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593449+0000", 2020-01-16T00:14:12.585 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3104999999999997e-05 2020-01-16T00:14:12.585 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.586 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.586 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:12.586 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593542+0000", 2020-01-16T00:14:12.586 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020070068999999999 2020-01-16T00:14:12.586 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.586 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:12.586 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.587 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.587 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.587 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3211 3.0 3:0b70bb20:::smithi12313934-46:head [setxattr _header (56) in=7b,truncate 2571537] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:14:12.587 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.581146+0000", 2020-01-16T00:14:12.587 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 42.808970580999997, 2020-01-16T00:14:12.587 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024318514999999999, 2020-01-16T00:14:12.587 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:12.587 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:12.588 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:12.588 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:12.588 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:12.588 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3211 2020-01-16T00:14:12.588 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.588 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:12.588 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.589 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:12.589 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581146+0000", 2020-01-16T00:14:12.589 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:12.589 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.589 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.589 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:12.589 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581146+0000", 2020-01-16T00:14:12.589 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:12.590 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.590 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.590 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:12.590 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581145+0000", 2020-01-16T00:14:12.590 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2990000000000002e-06 2020-01-16T00:14:12.590 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.591 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.591 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:12.591 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581151+0000", 2020-01-16T00:14:12.591 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.041e-06 2020-01-16T00:14:12.591 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.591 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.591 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:12.591 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581159+0000", 2020-01-16T00:14:12.592 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5859999999999998e-06 2020-01-16T00:14:12.592 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.592 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.592 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:12.592 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581163+0000", 2020-01-16T00:14:12.592 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020769790000000001 2020-01-16T00:14:12.592 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.592 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.593 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.593 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.583240+0000", 2020-01-16T00:14:12.593 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012019579999999999 2020-01-16T00:14:12.593 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.593 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.593 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:12.593 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584442+0000", 2020-01-16T00:14:12.593 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018359812 2020-01-16T00:14:12.594 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.594 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.594 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.594 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.602802+0000", 2020-01-16T00:14:12.594 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000133072 2020-01-16T00:14:12.594 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.594 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.595 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:12.595 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.602935+0000", 2020-01-16T00:14:12.595 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019007200000000001 2020-01-16T00:14:12.595 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.595 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.595 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:12.595 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.603125+0000", 2020-01-16T00:14:12.595 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001261751 2020-01-16T00:14:12.596 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.596 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.596 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:12.596 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.604387+0000", 2020-01-16T00:14:12.596 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000304987 2020-01-16T00:14:12.596 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.596 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.596 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:12.597 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.604692+0000", 2020-01-16T00:14:12.597 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00063369299999999995 2020-01-16T00:14:12.597 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.597 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.597 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:12.597 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605325+0000", 2020-01-16T00:14:12.597 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1470000000000005e-05 2020-01-16T00:14:12.598 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.598 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.598 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:12.598 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605397+0000", 2020-01-16T00:14:12.598 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7784e-05 2020-01-16T00:14:12.598 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.598 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.598 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:12.599 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605465+0000", 2020-01-16T00:14:12.599 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024318514999999999 2020-01-16T00:14:12.599 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.599 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:12.599 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.599 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.599 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.599 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3212 3.0 3:0b70bb20:::smithi12313934-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e45)", 2020-01-16T00:14:12.600 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.581171+0000", 2020-01-16T00:14:12.600 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 42.808945882000003, 2020-01-16T00:14:12.600 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024540131, 2020-01-16T00:14:12.600 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:12.600 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:12.600 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:12.600 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:12.600 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:12.601 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3212 2020-01-16T00:14:12.601 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.601 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:12.601 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.601 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:12.601 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581171+0000", 2020-01-16T00:14:12.601 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:12.601 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.602 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.602 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:12.602 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581171+0000", 2020-01-16T00:14:12.602 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:12.602 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.602 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.602 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:12.603 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581170+0000", 2020-01-16T00:14:12.603 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5259999999999996e-06 2020-01-16T00:14:12.603 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.603 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.603 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:12.603 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581178+0000", 2020-01-16T00:14:12.603 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.469e-06 2020-01-16T00:14:12.603 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.604 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.604 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:12.604 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581185+0000", 2020-01-16T00:14:12.604 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4460000000000003e-06 2020-01-16T00:14:12.604 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.604 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.604 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:12.604 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581190+0000", 2020-01-16T00:14:12.605 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003286696 2020-01-16T00:14:12.605 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.605 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.605 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.605 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584476+0000", 2020-01-16T00:14:12.605 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9681e-05 2020-01-16T00:14:12.605 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.605 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.606 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:12.606 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584536+0000", 2020-01-16T00:14:12.606 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019075267 2020-01-16T00:14:12.606 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.606 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.606 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.606 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.603611+0000", 2020-01-16T00:14:12.606 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5981000000000001e-05 2020-01-16T00:14:12.607 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.607 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.607 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:12.608 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.603707+0000", 2020-01-16T00:14:12.609 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0017922649999999999 2020-01-16T00:14:12.609 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.609 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.609 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.609 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605500+0000", 2020-01-16T00:14:12.609 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5396999999999995e-05 2020-01-16T00:14:12.609 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.610 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.610 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:12.610 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605595+0000", 2020-01-16T00:14:12.610 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000116038 2020-01-16T00:14:12.610 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.610 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.610 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:12.611 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605711+0000", 2020-01-16T00:14:12.611 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024540131 2020-01-16T00:14:12.611 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.611 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:12.611 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.611 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.611 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.612 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3435 3.0 3:0b70bb20:::smithi12313934-46:head [setxattr _header (56) in=7b,truncate 3075747] snapc 0=[] ondisk+write+known_if_redirected e47)", 2020-01-16T00:14:12.612 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:34.388891+0000", 2020-01-16T00:14:12.612 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 38.001225865999999, 2020-01-16T00:14:12.612 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016440475999999999, 2020-01-16T00:14:12.612 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:12.612 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:12.612 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:12.612 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:12.613 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:12.613 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3435 2020-01-16T00:14:12.613 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.613 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:12.613 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.613 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:12.613 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388891+0000", 2020-01-16T00:14:12.614 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:12.614 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.614 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.614 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:12.614 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388891+0000", 2020-01-16T00:14:12.614 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:12.614 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.614 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.615 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:12.615 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388890+0000", 2020-01-16T00:14:12.615 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.647e-06 2020-01-16T00:14:12.615 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.615 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.615 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:12.615 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388894+0000", 2020-01-16T00:14:12.616 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2839999999999997e-06 2020-01-16T00:14:12.616 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.616 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.616 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:12.616 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388899+0000", 2020-01-16T00:14:12.616 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3149999999999999e-06 2020-01-16T00:14:12.616 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.616 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.617 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:12.617 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388901+0000", 2020-01-16T00:14:12.617 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001768543 2020-01-16T00:14:12.617 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.617 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.617 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.617 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.390670+0000", 2020-01-16T00:14:12.617 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0034533680000000001 2020-01-16T00:14:12.618 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.618 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.618 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:12.618 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.394123+0000", 2020-01-16T00:14:12.618 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010189962 2020-01-16T00:14:12.618 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.618 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.619 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.619 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404313+0000", 2020-01-16T00:14:12.619 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000108646 2020-01-16T00:14:12.619 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.619 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.619 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:12.619 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404422+0000", 2020-01-16T00:14:12.620 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119133 2020-01-16T00:14:12.620 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.620 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.620 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:12.620 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404541+0000", 2020-01-16T00:14:12.620 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000433764 2020-01-16T00:14:12.620 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.620 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.621 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:12.621 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404975+0000", 2020-01-16T00:14:12.621 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000266013 2020-01-16T00:14:12.621 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.621 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.621 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:12.621 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405241+0000", 2020-01-16T00:14:12.622 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5187000000000001e-05 2020-01-16T00:14:12.622 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.622 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.623 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:12.623 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405266+0000", 2020-01-16T00:14:12.623 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0889000000000002e-05 2020-01-16T00:14:12.623 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.623 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.623 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:12.623 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405297+0000", 2020-01-16T00:14:12.623 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4354999999999999e-05 2020-01-16T00:14:12.624 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.624 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.624 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:12.624 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405331+0000", 2020-01-16T00:14:12.624 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016440475999999999 2020-01-16T00:14:12.624 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.624 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:12.624 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.625 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.625 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.625 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3436 3.0 3:0b70bb20:::smithi12313934-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e47)", 2020-01-16T00:14:12.625 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:34.388920+0000", 2020-01-16T00:14:12.625 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 38.001197077, 2020-01-16T00:14:12.625 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016552641, 2020-01-16T00:14:12.625 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:12.625 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:12.626 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:12.626 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:12.626 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:12.626 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3436 2020-01-16T00:14:12.626 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.626 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:12.626 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.627 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:12.627 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388920+0000", 2020-01-16T00:14:12.627 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:12.627 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.627 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.627 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:12.627 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388920+0000", 2020-01-16T00:14:12.627 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:12.628 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.628 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.628 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:12.628 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388919+0000", 2020-01-16T00:14:12.628 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.181e-06 2020-01-16T00:14:12.628 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.628 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.628 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:12.629 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388921+0000", 2020-01-16T00:14:12.629 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9940000000000003e-06 2020-01-16T00:14:12.629 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.629 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.629 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:12.629 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388928+0000", 2020-01-16T00:14:12.629 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3549999999999999e-06 2020-01-16T00:14:12.629 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.630 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.630 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:12.630 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388931+0000", 2020-01-16T00:14:12.630 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.005242483 2020-01-16T00:14:12.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.630 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.630 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.631 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.394173+0000", 2020-01-16T00:14:12.631 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7435999999999995e-05 2020-01-16T00:14:12.631 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.631 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.631 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:12.631 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.394251+0000", 2020-01-16T00:14:12.631 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010593286 2020-01-16T00:14:12.631 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.632 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.632 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.632 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404844+0000", 2020-01-16T00:14:12.632 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0620999999999999e-05 2020-01-16T00:14:12.632 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.632 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.632 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:12.632 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404925+0000", 2020-01-16T00:14:12.633 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00042800899999999997 2020-01-16T00:14:12.633 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.633 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.633 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.633 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405353+0000", 2020-01-16T00:14:12.633 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2250000000000003e-05 2020-01-16T00:14:12.633 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.633 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.634 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:12.634 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405405+0000", 2020-01-16T00:14:12.634 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7411999999999998e-05 2020-01-16T00:14:12.634 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.634 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.634 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:12.634 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405472+0000", 2020-01-16T00:14:12.634 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016552641 2020-01-16T00:14:12.635 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.635 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:12.635 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.635 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.635 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.635 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4721 3.0 3:0b70bb20:::smithi12313934-46:head [setxattr _header (56) in=7b,truncate 2134621] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-01-16T00:14:12.635 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.045819+0000", 2020-01-16T00:14:12.635 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.3442979230000001, 2020-01-16T00:14:12.636 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019902080999999999, 2020-01-16T00:14:12.636 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:12.636 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:12.636 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:12.636 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:12.636 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:12.636 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4721 2020-01-16T00:14:12.637 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.637 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:12.637 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.637 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:12.637 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045819+0000", 2020-01-16T00:14:12.637 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:12.637 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.637 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.638 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:12.638 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045819+0000", 2020-01-16T00:14:12.638 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:12.638 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.638 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.638 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:12.638 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045818+0000", 2020-01-16T00:14:12.638 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.428e-06 2020-01-16T00:14:12.639 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.639 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.639 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:12.639 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045822+0000", 2020-01-16T00:14:12.639 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5940000000000004e-06 2020-01-16T00:14:12.639 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.639 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.640 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:12.640 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045826+0000", 2020-01-16T00:14:12.640 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6619999999999998e-06 2020-01-16T00:14:12.640 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.640 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.640 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:12.640 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045830+0000", 2020-01-16T00:14:12.640 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001710676 2020-01-16T00:14:12.641 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.641 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.641 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.641 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.047541+0000", 2020-01-16T00:14:12.641 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001637009 2020-01-16T00:14:12.641 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.641 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.641 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:12.642 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049178+0000", 2020-01-16T00:14:12.642 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.01461612 2020-01-16T00:14:12.642 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.642 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.642 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.642 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.063794+0000", 2020-01-16T00:14:12.642 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4941999999999997e-05 2020-01-16T00:14:12.642 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.643 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.643 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:12.643 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.063849+0000", 2020-01-16T00:14:12.643 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011165 2020-01-16T00:14:12.643 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.643 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.643 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:12.644 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.063960+0000", 2020-01-16T00:14:12.644 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013431630000000001 2020-01-16T00:14:12.644 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.644 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.644 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:12.644 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.065303+0000", 2020-01-16T00:14:12.644 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021894699999999999 2020-01-16T00:14:12.644 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.645 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.645 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:12.645 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.065522+0000", 2020-01-16T00:14:12.645 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013033899999999999 2020-01-16T00:14:12.645 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.645 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.646 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:12.646 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.065653+0000", 2020-01-16T00:14:12.646 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3800000000000002e-05 2020-01-16T00:14:12.646 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.646 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.646 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:12.646 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.065687+0000", 2020-01-16T00:14:12.646 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4332999999999998e-05 2020-01-16T00:14:12.647 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.647 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.647 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:12.647 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.065721+0000", 2020-01-16T00:14:12.647 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019902080999999999 2020-01-16T00:14:12.647 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.647 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:12.647 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.648 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.648 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.648 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4722 3.0 3:0b70bb20:::smithi12313934-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e58)", 2020-01-16T00:14:12.648 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.045850+0000", 2020-01-16T00:14:12.648 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.3442671639999997, 2020-01-16T00:14:12.648 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020039033000000001, 2020-01-16T00:14:12.648 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:12.648 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:12.649 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:12.649 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:12.649 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:12.649 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4722 2020-01-16T00:14:12.649 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.649 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:12.649 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.650 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:12.650 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045850+0000", 2020-01-16T00:14:12.650 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:12.650 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.650 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.650 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:12.650 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045850+0000", 2020-01-16T00:14:12.650 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:12.651 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.651 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.651 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:12.651 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045849+0000", 2020-01-16T00:14:12.651 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7349999999999999e-06 2020-01-16T00:14:12.651 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.651 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.651 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:12.652 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045856+0000", 2020-01-16T00:14:12.652 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1330000000000001e-06 2020-01-16T00:14:12.652 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.652 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.652 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:12.652 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045860+0000", 2020-01-16T00:14:12.652 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.565e-06 2020-01-16T00:14:12.653 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.653 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.653 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:12.653 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045863+0000", 2020-01-16T00:14:12.653 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033590019999999998 2020-01-16T00:14:12.653 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.653 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.653 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.654 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049222+0000", 2020-01-16T00:14:12.654 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8267e-05 2020-01-16T00:14:12.654 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.654 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.654 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:12.654 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049320+0000", 2020-01-16T00:14:12.654 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015128875999999999 2020-01-16T00:14:12.654 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.655 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.655 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.655 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.064449+0000", 2020-01-16T00:14:12.655 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4913000000000001e-05 2020-01-16T00:14:12.655 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.655 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.655 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:12.655 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.064504+0000", 2020-01-16T00:14:12.656 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012498100000000001 2020-01-16T00:14:12.656 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.656 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.656 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:12.656 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.065753+0000", 2020-01-16T00:14:12.656 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6299999999999999e-05 2020-01-16T00:14:12.656 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.656 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.657 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:12.657 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.065820+0000", 2020-01-16T00:14:12.657 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8801999999999996e-05 2020-01-16T00:14:12.657 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:12.657 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:12.657 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:12.657 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.065889+0000", 2020-01-16T00:14:12.657 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020039033000000001 2020-01-16T00:14:12.658 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.658 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:12.658 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.658 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:12.658 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:12.658 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:12.662 INFO:tasks.rados.rados.0.smithi123.stdout:1727: finishing write tid 1 to smithi12313934-28 2020-01-16T00:14:12.662 INFO:tasks.rados.rados.0.smithi123.stdout:1727: finishing write tid 2 to smithi12313934-28 2020-01-16T00:14:12.662 INFO:tasks.rados.rados.0.smithi123.stdout:1727: finishing write tid 4 to smithi12313934-28 2020-01-16T00:14:12.663 INFO:tasks.rados.rados.0.smithi123.stdout:1727: finishing write tid 5 to smithi12313934-28 2020-01-16T00:14:12.663 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 28 v 220 (ObjNum 620 snap 0 seq_num 620) dirty exists 2020-01-16T00:14:12.663 INFO:tasks.rados.rados.0.smithi123.stdout:1727: left oid 28 (ObjNum 620 snap 0 seq_num 620) 2020-01-16T00:14:12.663 INFO:tasks.rados.rados.0.smithi123.stdout:1728: finishing write tid 1 to smithi12313934-49 2020-01-16T00:14:12.663 INFO:tasks.rados.rados.0.smithi123.stdout:1728: finishing write tid 2 to smithi12313934-49 2020-01-16T00:14:12.663 INFO:tasks.rados.rados.0.smithi123.stdout:1724: done (15 left) 2020-01-16T00:14:12.663 INFO:tasks.rados.rados.0.smithi123.stdout:1725: done (14 left) 2020-01-16T00:14:12.664 INFO:tasks.rados.rados.0.smithi123.stdout:1726: done (13 left) 2020-01-16T00:14:12.664 INFO:tasks.rados.rados.0.smithi123.stdout:1727: done (12 left) 2020-01-16T00:14:12.664 INFO:tasks.rados.rados.0.smithi123.stdout:1740: write (excl) oid 12 current snap is 0 2020-01-16T00:14:12.664 INFO:tasks.rados.rados.0.smithi123.stdout:1740: seq_num 626 ranges {521182=646820,1948669=523219} 2020-01-16T00:14:12.664 INFO:tasks.rados.rados.0.smithi123.stdout:1740: writing smithi12313934-12 from 521182 to 1168002 tid 1 2020-01-16T00:14:12.664 INFO:tasks.rados.rados.0.smithi123.stdout:1740: writing smithi12313934-12 from 1948669 to 2471888 tid 2 2020-01-16T00:14:12.664 INFO:tasks.rados.rados.0.smithi123.stdout:1741: write (excl) oid 17 current snap is 0 2020-01-16T00:14:12.664 INFO:tasks.rados.rados.0.smithi123.stdout:1741: seq_num 627 ranges {537989=722069,1890798=647191} 2020-01-16T00:14:12.665 INFO:tasks.rados.rados.0.smithi123.stdout:1741: writing smithi12313934-17 from 537989 to 1260058 tid 1 2020-01-16T00:14:12.665 INFO:tasks.rados.rados.0.smithi123.stdout:1741: writing smithi12313934-17 from 1890798 to 2537989 tid 2 2020-01-16T00:14:12.665 INFO:tasks.rados.rados.0.smithi123.stdout:1742: write oid 35 current snap is 0 2020-01-16T00:14:12.665 INFO:tasks.rados.rados.0.smithi123.stdout:1742: seq_num 628 ranges {554796=797318,1949280=526888} 2020-01-16T00:14:12.665 INFO:tasks.rados.rados.0.smithi123.stdout:1742: writing smithi12313934-35 from 554796 to 1352114 tid 1 2020-01-16T00:14:12.665 INFO:tasks.rados.rados.0.smithi123.stdout:1742: writing smithi12313934-35 from 1949280 to 2476168 tid 2 2020-01-16T00:14:12.665 INFO:tasks.rados.rados.0.smithi123.stdout:1743: write (excl) oid 6 current snap is 0 2020-01-16T00:14:12.665 INFO:tasks.rados.rados.0.smithi123.stdout:1743: seq_num 629 ranges {571603=472567,1607762=670546} 2020-01-16T00:14:12.666 INFO:tasks.rados.rados.0.smithi123.stdout:1743: writing smithi12313934-6 from 571603 to 1044170 tid 1 2020-01-16T00:14:12.666 INFO:tasks.rados.rados.0.smithi123.stdout:1743: writing smithi12313934-6 from 1607762 to 2278308 tid 2 2020-01-16T00:14:12.666 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:12.666 INFO:tasks.rados.rados.0.smithi123.stdout:1728: finishing write tid 4 to smithi12313934-49 2020-01-16T00:14:12.666 INFO:tasks.rados.rados.0.smithi123.stdout:1728: finishing write tid 5 to smithi12313934-49 2020-01-16T00:14:12.666 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 49 v 176 (ObjNum 621 snap 0 seq_num 621) dirty exists 2020-01-16T00:14:12.666 INFO:tasks.rados.rados.0.smithi123.stdout:1728: left oid 49 (ObjNum 621 snap 0 seq_num 621) 2020-01-16T00:14:12.666 INFO:tasks.rados.rados.0.smithi123.stdout:1729: finishing write tid 1 to smithi12313934-32 2020-01-16T00:14:12.667 INFO:tasks.rados.rados.0.smithi123.stdout:1729: finishing write tid 2 to smithi12313934-32 2020-01-16T00:14:12.667 INFO:tasks.rados.rados.0.smithi123.stdout:1729: finishing write tid 4 to smithi12313934-32 2020-01-16T00:14:12.667 INFO:tasks.rados.rados.0.smithi123.stdout:1728: done (15 left) 2020-01-16T00:14:12.667 INFO:tasks.rados.rados.0.smithi123.stdout:1744: write oid 24 current snap is 0 2020-01-16T00:14:12.667 INFO:tasks.rados.rados.0.smithi123.stdout:1744: seq_num 630 ranges {588410=547816,1666244=414204,2484603=103807} 2020-01-16T00:14:12.667 INFO:tasks.rados.rados.0.smithi123.stdout:1744: writing smithi12313934-24 from 588410 to 1136226 tid 1 2020-01-16T00:14:12.667 INFO:tasks.rados.rados.0.smithi123.stdout:1744: writing smithi12313934-24 from 1666244 to 2080448 tid 2 2020-01-16T00:14:12.668 INFO:tasks.rados.rados.0.smithi123.stdout:1744: writing smithi12313934-24 from 2484603 to 2588410 tid 3 2020-01-16T00:14:12.668 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:12.668 INFO:tasks.rados.rados.0.smithi123.stdout:1729: finishing write tid 5 to smithi12313934-32 2020-01-16T00:14:12.668 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 32 v 170 (ObjNum 622 snap 0 seq_num 622) dirty exists 2020-01-16T00:14:12.668 INFO:tasks.rados.rados.0.smithi123.stdout:1729: left oid 32 (ObjNum 622 snap 0 seq_num 622) 2020-01-16T00:14:12.668 INFO:tasks.rados.rados.0.smithi123.stdout:1730: expect (ObjNum 571 snap 0 seq_num 571) 2020-01-16T00:14:12.668 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:12.656+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:12.669 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-01-16T00:14:12.759 INFO:tasks.rados.rados.0.smithi123.stdout:1732: finishing write tid 1 to smithi12313934-47 2020-01-16T00:14:12.759 INFO:tasks.rados.rados.0.smithi123.stdout:1733: expect (ObjNum 615 snap 0 seq_num 615) 2020-01-16T00:14:12.761 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:12.758+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:12.810 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:12.811 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:14:12.811 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:14:12.813 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:12.820 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-01-16T00:14:12.849 INFO:teuthology.orchestra.run.smithi123.stderr:noscrub is unset 2020-01-16T00:14:12.903 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:12.902+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:12.942 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:12.942 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:14:12.942 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:14:12.942 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:14:12.943 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:12.951 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-01-16T00:14:12.984 INFO:tasks.rados.rados.0.smithi123.stdout:1732: finishing write tid 2 to smithi12313934-47 2020-01-16T00:14:12.985 INFO:tasks.rados.rados.0.smithi123.stdout:1732: finishing write tid 4 to smithi12313934-47 2020-01-16T00:14:12.985 INFO:tasks.rados.rados.0.smithi123.stdout:1732: finishing write tid 5 to smithi12313934-47 2020-01-16T00:14:12.985 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 47 v 271 (ObjNum 623 snap 0 seq_num 623) dirty exists 2020-01-16T00:14:12.985 INFO:tasks.rados.rados.0.smithi123.stdout:1732: left oid 47 (ObjNum 623 snap 0 seq_num 623) 2020-01-16T00:14:12.985 INFO:tasks.rados.rados.0.smithi123.stdout:1734: finishing write tid 1 to smithi12313934-15 2020-01-16T00:14:12.985 INFO:tasks.rados.rados.0.smithi123.stdout:1735: finishing write tid 1 to smithi12313934-16 2020-01-16T00:14:12.985 INFO:tasks.rados.rados.0.smithi123.stdout:1729: done (15 left) 2020-01-16T00:14:12.986 INFO:tasks.rados.rados.0.smithi123.stdout:1730: done (14 left) 2020-01-16T00:14:12.986 INFO:tasks.rados.rados.0.smithi123.stdout:1731: done (13 left) 2020-01-16T00:14:12.986 INFO:tasks.rados.rados.0.smithi123.stdout:1732: done (12 left) 2020-01-16T00:14:12.986 INFO:tasks.rados.rados.0.smithi123.stdout:1733: done (11 left) 2020-01-16T00:14:12.986 INFO:tasks.rados.rados.0.smithi123.stdout:1737: done (10 left) 2020-01-16T00:14:12.986 INFO:tasks.rados.rados.0.smithi123.stdout:1738: done (9 left) 2020-01-16T00:14:12.987 INFO:tasks.rados.rados.0.smithi123.stdout:1745: read oid 42 snap -1 2020-01-16T00:14:12.987 INFO:tasks.rados.rados.0.smithi123.stdout:1745: expect (ObjNum 610 snap 0 seq_num 610) 2020-01-16T00:14:12.987 INFO:tasks.rados.rados.0.smithi123.stdout:1746: write oid 37 current snap is 0 2020-01-16T00:14:12.987 INFO:tasks.rados.rados.0.smithi123.stdout:1746: seq_num 631 ranges {605217=739418,2124726=480491} 2020-01-16T00:14:12.988 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:12.982+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:13.002 INFO:tasks.rados.rados.0.smithi123.stdout:1746: writing smithi12313934-37 from 605217 to 1344635 tid 1 2020-01-16T00:14:13.009 INFO:tasks.rados.rados.0.smithi123.stdout:1746: writing smithi12313934-37 from 2124726 to 2605217 tid 2 2020-01-16T00:14:13.009 INFO:tasks.rados.rados.0.smithi123.stdout:1747: read oid 19 snap -1 2020-01-16T00:14:13.010 INFO:tasks.rados.rados.0.smithi123.stdout:1747: expect (ObjNum 611 snap 0 seq_num 611) 2020-01-16T00:14:13.010 INFO:tasks.rados.rados.0.smithi123.stdout:1748: write (excl) oid 46 current snap is 0 2020-01-16T00:14:13.010 INFO:tasks.rados.rados.0.smithi123.stdout:1748: seq_num 632 ranges {622024=414667,1783208=417873} 2020-01-16T00:14:13.015 INFO:tasks.rados.rados.0.smithi123.stdout:1748: writing smithi12313934-46 from 622024 to 1036691 tid 1 2020-01-16T00:14:13.022 INFO:tasks.rados.rados.0.smithi123.stdout:1748: writing smithi12313934-46 from 1783208 to 2201081 tid 2 2020-01-16T00:14:13.022 INFO:tasks.rados.rados.0.smithi123.stdout:1749: write oid 9 current snap is 0 2020-01-16T00:14:13.022 INFO:tasks.rados.rados.0.smithi123.stdout:1749: seq_num 633 ranges {638831=489916,1841690=677884} 2020-01-16T00:14:13.029 INFO:tasks.rados.rados.0.smithi123.stdout:1749: writing smithi12313934-9 from 638831 to 1128747 tid 1 2020-01-16T00:14:13.038 INFO:tasks.rados.rados.0.smithi123.stdout:1749: writing smithi12313934-9 from 1841690 to 2519574 tid 2 2020-01-16T00:14:13.038 INFO:tasks.rados.rados.0.smithi123.stdout:1750: write (excl) oid 18 current snap is 0 2020-01-16T00:14:13.038 INFO:tasks.rados.rados.0.smithi123.stdout:1750: seq_num 634 ranges {655638=565165,1900172=421542} 2020-01-16T00:14:13.045 INFO:tasks.rados.rados.0.smithi123.stdout:1750: writing smithi12313934-18 from 655638 to 1220803 tid 1 2020-01-16T00:14:13.051 INFO:tasks.rados.rados.0.smithi123.stdout:1750: writing smithi12313934-18 from 1900172 to 2321714 tid 2 2020-01-16T00:14:13.051 INFO:tasks.rados.rados.0.smithi123.stdout:1751: write oid 34 current snap is 0 2020-01-16T00:14:13.051 INFO:tasks.rados.rados.0.smithi123.stdout:1751: seq_num 635 ranges {672445=640414,1842301=681553} 2020-01-16T00:14:13.058 INFO:tasks.rados.rados.0.smithi123.stdout:1751: writing smithi12313934-34 from 672445 to 1312859 tid 1 2020-01-16T00:14:13.063 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:13.062+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:13.066 INFO:tasks.rados.rados.0.smithi123.stdout:1751: writing smithi12313934-34 from 1842301 to 2523854 tid 2 2020-01-16T00:14:13.066 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:13.066 INFO:tasks.rados.rados.0.smithi123.stdout:1734: finishing write tid 2 to smithi12313934-15 2020-01-16T00:14:13.066 INFO:tasks.rados.rados.0.smithi123.stdout:1734: finishing write tid 4 to smithi12313934-15 2020-01-16T00:14:13.066 INFO:tasks.rados.rados.0.smithi123.stdout:1734: finishing write tid 5 to smithi12313934-15 2020-01-16T00:14:13.066 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 15 v 224 (ObjNum 624 snap 0 seq_num 624) dirty exists 2020-01-16T00:14:13.067 INFO:tasks.rados.rados.0.smithi123.stdout:1734: left oid 15 (ObjNum 624 snap 0 seq_num 624) 2020-01-16T00:14:13.067 INFO:tasks.rados.rados.0.smithi123.stdout:1736: expect (ObjNum 575 snap 0 seq_num 575) 2020-01-16T00:14:13.099 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:13.100 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:14:13.100 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:14:13.100 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:14:13.100 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.100 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1649 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~2084068,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:14:13.100 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847683+0000", 2020-01-16T00:14:13.101 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 81.249454405999998, 2020-01-16T00:14:13.101 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010654691000000001, 2020-01-16T00:14:13.101 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.101 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:13.101 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:13.101 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:13.101 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:13.101 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1649 2020-01-16T00:14:13.102 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.102 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.102 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.102 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.102 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847683+0000", 2020-01-16T00:14:13.102 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.102 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.103 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.103 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.103 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847683+0000", 2020-01-16T00:14:13.103 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:13.103 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.103 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.103 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.103 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847682+0000", 2020-01-16T00:14:13.104 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8569999999999997e-06 2020-01-16T00:14:13.104 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.104 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.104 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.104 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847689+0000", 2020-01-16T00:14:13.104 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8090000000000004e-06 2020-01-16T00:14:13.104 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.104 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.105 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.105 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847696+0000", 2020-01-16T00:14:13.105 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2690000000000001e-06 2020-01-16T00:14:13.105 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.105 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.105 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.105 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847700+0000", 2020-01-16T00:14:13.106 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8189999999999993e-05 2020-01-16T00:14:13.106 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.106 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.106 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.106 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847798+0000", 2020-01-16T00:14:13.106 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7129999999999999e-06 2020-01-16T00:14:13.107 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.107 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.107 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:14:13.107 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847802+0000", 2020-01-16T00:14:13.107 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083913049999999999 2020-01-16T00:14:13.107 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.107 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.108 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.108 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.856193+0000", 2020-01-16T00:14:13.108 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034400300000000001 2020-01-16T00:14:13.108 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.108 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.108 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.108 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.856537+0000", 2020-01-16T00:14:13.108 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018006820000000001 2020-01-16T00:14:13.109 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.109 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.109 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.109 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858338+0000", 2020-01-16T00:14:13.109 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010654691000000001 2020-01-16T00:14:13.109 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.109 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.110 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.110 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.110 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.110 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1650 3.1 3:8de1d8a3:::smithi12313934-13:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:14:13.110 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847717+0000", 2020-01-16T00:14:13.110 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 81.249420452999999, 2020-01-16T00:14:13.110 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010913658, 2020-01-16T00:14:13.110 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.111 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:13.111 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:13.111 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:13.111 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:13.111 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1650 2020-01-16T00:14:13.111 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.111 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.112 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.112 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.112 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847717+0000", 2020-01-16T00:14:13.112 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.112 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.112 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.112 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.113 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847717+0000", 2020-01-16T00:14:13.113 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:13.113 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.113 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.113 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.113 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847716+0000", 2020-01-16T00:14:13.113 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.433e-06 2020-01-16T00:14:13.113 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.114 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.114 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.114 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847720+0000", 2020-01-16T00:14:13.114 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1800000000000004e-06 2020-01-16T00:14:13.114 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.114 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.114 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.115 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847725+0000", 2020-01-16T00:14:13.115 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6689999999999997e-06 2020-01-16T00:14:13.115 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.115 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.115 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.115 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847730+0000", 2020-01-16T00:14:13.116 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000102468 2020-01-16T00:14:13.116 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.116 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.116 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.116 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847832+0000", 2020-01-16T00:14:13.116 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3999999999999997e-06 2020-01-16T00:14:13.116 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.117 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.117 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:14:13.117 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847838+0000", 2020-01-16T00:14:13.117 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010519395000000001 2020-01-16T00:14:13.117 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.117 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.117 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.117 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858358+0000", 2020-01-16T00:14:13.118 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7600000000000002e-05 2020-01-16T00:14:13.118 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.118 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.118 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.118 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858445+0000", 2020-01-16T00:14:13.118 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018515900000000001 2020-01-16T00:14:13.118 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.119 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.119 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.119 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858631+0000", 2020-01-16T00:14:13.119 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010913658 2020-01-16T00:14:13.119 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.119 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.119 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.119 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.120 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.120 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1651 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~2084068,checksum] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:14:13.120 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847752+0000", 2020-01-16T00:14:13.120 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 81.249385837000005, 2020-01-16T00:14:13.120 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011734249, 2020-01-16T00:14:13.120 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.120 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:13.120 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:13.121 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:13.121 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:13.121 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1651 2020-01-16T00:14:13.121 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.121 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.121 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.122 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.122 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847752+0000", 2020-01-16T00:14:13.122 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.122 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.122 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.122 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.122 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847752+0000", 2020-01-16T00:14:13.122 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:13.123 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.123 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.123 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.123 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847751+0000", 2020-01-16T00:14:13.123 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1980000000000001e-06 2020-01-16T00:14:13.123 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.123 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.124 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.124 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847754+0000", 2020-01-16T00:14:13.124 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6679999999999999e-06 2020-01-16T00:14:13.124 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.124 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.124 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.124 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847758+0000", 2020-01-16T00:14:13.124 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2500000000000001e-06 2020-01-16T00:14:13.125 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.125 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.125 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.125 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847760+0000", 2020-01-16T00:14:13.125 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7614000000000004e-05 2020-01-16T00:14:13.125 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.125 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.125 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.126 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847858+0000", 2020-01-16T00:14:13.126 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7950000000000004e-06 2020-01-16T00:14:13.126 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.126 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.126 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:14:13.126 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847864+0000", 2020-01-16T00:14:13.126 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010790345 2020-01-16T00:14:13.127 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.127 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.127 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.127 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858654+0000", 2020-01-16T00:14:13.127 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000141466 2020-01-16T00:14:13.127 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.127 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.127 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.128 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858795+0000", 2020-01-16T00:14:13.128 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00069037899999999997 2020-01-16T00:14:13.128 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.128 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.128 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.128 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.859486+0000", 2020-01-16T00:14:13.128 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011734249 2020-01-16T00:14:13.128 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.129 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.129 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.129 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.129 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.129 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2108 3.1 3:8de1d8a3:::smithi12313934-13:head [write 2816700~410210] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:14:13.129 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.641861+0000", 2020-01-16T00:14:13.129 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 71.455276647000005, 2020-01-16T00:14:13.130 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018545604, 2020-01-16T00:14:13.130 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.130 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:13.130 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:13.130 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:13.130 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:13.130 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2108 2020-01-16T00:14:13.130 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.131 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.131 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.131 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.131 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.641861+0000", 2020-01-16T00:14:13.131 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.131 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.131 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.131 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.132 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.641861+0000", 2020-01-16T00:14:13.132 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:13.132 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.132 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.132 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.132 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.641859+0000", 2020-01-16T00:14:13.132 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018466399999999999 2020-01-16T00:14:13.133 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.133 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.133 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.133 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642043+0000", 2020-01-16T00:14:13.133 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2999999999999999e-05 2020-01-16T00:14:13.133 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.133 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.133 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.134 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642056+0000", 2020-01-16T00:14:13.134 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0730000000000005e-06 2020-01-16T00:14:13.134 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.134 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.134 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.134 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642064+0000", 2020-01-16T00:14:13.134 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9527999999999997e-05 2020-01-16T00:14:13.135 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.135 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.135 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.135 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642114+0000", 2020-01-16T00:14:13.135 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036616050000000001 2020-01-16T00:14:13.135 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.135 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.135 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:13.136 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645776+0000", 2020-01-16T00:14:13.136 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012222663999999999 2020-01-16T00:14:13.136 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.136 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.136 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.136 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.657998+0000", 2020-01-16T00:14:13.136 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014280299999999999 2020-01-16T00:14:13.136 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.137 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.137 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.137 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.658141+0000", 2020-01-16T00:14:13.137 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0001102 2020-01-16T00:14:13.137 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.137 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.137 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:14:13.138 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.658251+0000", 2020-01-16T00:14:13.138 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020421219999999999 2020-01-16T00:14:13.138 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.138 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.138 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:13.138 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660293+0000", 2020-01-16T00:14:13.138 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7399999999999999e-06 2020-01-16T00:14:13.138 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.139 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.139 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:13.139 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660303+0000", 2020-01-16T00:14:13.139 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9873999999999998e-05 2020-01-16T00:14:13.139 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.139 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.139 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:13.139 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660343+0000", 2020-01-16T00:14:13.140 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4448e-05 2020-01-16T00:14:13.140 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.140 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.140 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:13.140 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660377+0000", 2020-01-16T00:14:13.140 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8915999999999999e-05 2020-01-16T00:14:13.140 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.140 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.141 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.141 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660406+0000", 2020-01-16T00:14:13.141 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018545604 2020-01-16T00:14:13.141 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.141 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.141 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.141 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.142 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.142 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2109 3.1 3:8de1d8a3:::smithi12313934-13:head [setxattr _header (56) in=7b,truncate 3865610] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:14:13.142 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.642075+0000", 2020-01-16T00:14:13.142 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 71.455062073999997, 2020-01-16T00:14:13.142 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018477655999999999, 2020-01-16T00:14:13.142 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.142 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:13.142 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:13.143 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:13.143 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:13.143 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2109 2020-01-16T00:14:13.143 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.143 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.143 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.143 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.144 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642075+0000", 2020-01-16T00:14:13.144 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.144 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.144 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.144 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.144 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642075+0000", 2020-01-16T00:14:13.145 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:13.145 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.145 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.145 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.145 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642074+0000", 2020-01-16T00:14:13.145 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4500000000000003e-06 2020-01-16T00:14:13.145 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.146 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.146 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.146 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642080+0000", 2020-01-16T00:14:13.146 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7229999999999998e-06 2020-01-16T00:14:13.146 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.146 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.146 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.147 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642087+0000", 2020-01-16T00:14:13.147 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2390000000000004e-06 2020-01-16T00:14:13.147 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.147 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.147 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.147 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642091+0000", 2020-01-16T00:14:13.147 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0037396550000000001 2020-01-16T00:14:13.147 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.148 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.148 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.148 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645830+0000", 2020-01-16T00:14:13.148 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2200000000000002e-05 2020-01-16T00:14:13.148 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.148 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.148 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:13.148 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645883+0000", 2020-01-16T00:14:13.148 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013432934000000001 2020-01-16T00:14:13.149 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.149 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.149 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.149 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659316+0000", 2020-01-16T00:14:13.149 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000140006 2020-01-16T00:14:13.149 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.149 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.149 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.150 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659456+0000", 2020-01-16T00:14:13.150 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000184873 2020-01-16T00:14:13.150 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.150 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.150 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:14:13.150 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659640+0000", 2020-01-16T00:14:13.151 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00066954999999999998 2020-01-16T00:14:13.151 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.151 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.151 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:13.151 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660310+0000", 2020-01-16T00:14:13.151 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014847699999999999 2020-01-16T00:14:13.151 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.152 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.152 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:13.152 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660459+0000", 2020-01-16T00:14:13.152 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6869e-05 2020-01-16T00:14:13.152 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.152 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.152 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:13.152 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660485+0000", 2020-01-16T00:14:13.153 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8478000000000001e-05 2020-01-16T00:14:13.153 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.153 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.153 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:13.153 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660514+0000", 2020-01-16T00:14:13.153 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9153999999999998e-05 2020-01-16T00:14:13.153 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.154 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.154 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.154 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660553+0000", 2020-01-16T00:14:13.154 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018477655999999999 2020-01-16T00:14:13.154 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.154 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.154 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.154 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.155 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.155 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2110 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-01-16T00:14:13.155 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.642112+0000", 2020-01-16T00:14:13.155 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 71.455025436, 2020-01-16T00:14:13.155 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018620198000000001, 2020-01-16T00:14:13.155 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.155 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:13.155 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:13.156 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:13.156 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:13.156 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2110 2020-01-16T00:14:13.156 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.156 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.156 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.156 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.157 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642112+0000", 2020-01-16T00:14:13.157 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.157 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.157 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.157 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.157 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642112+0000", 2020-01-16T00:14:13.157 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:13.157 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.158 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.158 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.158 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642111+0000", 2020-01-16T00:14:13.158 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4450000000000001e-06 2020-01-16T00:14:13.158 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.158 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.158 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.159 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642117+0000", 2020-01-16T00:14:13.159 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1567e-05 2020-01-16T00:14:13.159 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.159 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.159 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.159 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642129+0000", 2020-01-16T00:14:13.159 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.754e-06 2020-01-16T00:14:13.159 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.160 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.160 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.160 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642136+0000", 2020-01-16T00:14:13.160 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040239719999999998 2020-01-16T00:14:13.160 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.160 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.160 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.161 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646160+0000", 2020-01-16T00:14:13.161 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5600999999999998e-05 2020-01-16T00:14:13.161 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.161 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.161 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:13.161 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646215+0000", 2020-01-16T00:14:13.161 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013733006000000001 2020-01-16T00:14:13.161 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.162 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.162 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.162 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659948+0000", 2020-01-16T00:14:13.162 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000106277 2020-01-16T00:14:13.162 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.162 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.162 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:13.163 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660054+0000", 2020-01-16T00:14:13.163 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000524327 2020-01-16T00:14:13.163 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.163 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.163 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.163 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660579+0000", 2020-01-16T00:14:13.163 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9841999999999999e-05 2020-01-16T00:14:13.163 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.164 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.164 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.164 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660659+0000", 2020-01-16T00:14:13.164 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3572999999999995e-05 2020-01-16T00:14:13.164 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.164 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.165 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.165 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660732+0000", 2020-01-16T00:14:13.165 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018620198000000001 2020-01-16T00:14:13.165 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.165 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.165 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.165 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.165 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.166 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:70 2.1 e35/19 2:91f85463:::smithi12313934-24 low_tier:head v 35'66, mlcod=34'64)", 2020-01-16T00:14:13.166 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.986684+0000", 2020-01-16T00:14:13.166 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 67.110453582000005, 2020-01-16T00:14:13.166 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012287378999999999, 2020-01-16T00:14:13.166 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.166 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:13.166 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.167 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.167 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.167 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986684+0000", 2020-01-16T00:14:13.167 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.167 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.167 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.167 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.167 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986684+0000", 2020-01-16T00:14:13.167 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:13.168 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.168 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.168 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.168 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986683+0000", 2020-01-16T00:14:13.168 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012884179999999999 2020-01-16T00:14:13.168 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.168 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.168 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.169 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.987971+0000", 2020-01-16T00:14:13.169 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011558200000000001 2020-01-16T00:14:13.169 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.169 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.169 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.169 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988087+0000", 2020-01-16T00:14:13.169 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0472000000000001e-05 2020-01-16T00:14:13.169 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.170 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.170 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.170 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988137+0000", 2020-01-16T00:14:13.170 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000107484 2020-01-16T00:14:13.170 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.170 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.171 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.171 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988245+0000", 2020-01-16T00:14:13.171 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4409999999999998e-05 2020-01-16T00:14:13.171 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.171 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.172 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.172 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988279+0000", 2020-01-16T00:14:13.172 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010641609999999999 2020-01-16T00:14:13.172 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.172 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.172 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:13.172 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.998921+0000", 2020-01-16T00:14:13.173 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0278000000000002e-05 2020-01-16T00:14:13.173 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.173 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.173 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.173 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.998971+0000", 2020-01-16T00:14:13.173 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012287378999999999 2020-01-16T00:14:13.173 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.173 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.174 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.174 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.174 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.174 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:109 2.0 e42/19 2:11fa79bc:::smithi12313934-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 42'67, mlcod=41'65)", 2020-01-16T00:14:13.174 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:21.841140+0000", 2020-01-16T00:14:13.174 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 51.255997514000001, 2020-01-16T00:14:13.174 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010439159999999999, 2020-01-16T00:14:13.174 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.175 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:13.175 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.175 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.175 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.175 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841140+0000", 2020-01-16T00:14:13.175 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.175 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.175 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.176 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.176 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841140+0000", 2020-01-16T00:14:13.176 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:13.176 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.176 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.176 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.176 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841137+0000", 2020-01-16T00:14:13.176 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00061049800000000001 2020-01-16T00:14:13.177 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.177 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.177 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.177 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841748+0000", 2020-01-16T00:14:13.177 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8703e-05 2020-01-16T00:14:13.177 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.177 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.177 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.178 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841767+0000", 2020-01-16T00:14:13.178 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5410000000000003e-06 2020-01-16T00:14:13.178 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.178 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.178 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.178 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841774+0000", 2020-01-16T00:14:13.178 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000100144 2020-01-16T00:14:13.178 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.179 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.179 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.179 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841874+0000", 2020-01-16T00:14:13.179 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9402999999999999e-05 2020-01-16T00:14:13.179 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.179 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.179 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.179 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841944+0000", 2020-01-16T00:14:13.180 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095901649999999995 2020-01-16T00:14:13.180 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.180 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.180 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:13.180 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.851534+0000", 2020-01-16T00:14:13.180 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5296000000000001e-05 2020-01-16T00:14:13.180 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.181 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.181 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.181 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.851579+0000", 2020-01-16T00:14:13.181 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010439159999999999 2020-01-16T00:14:13.181 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.181 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.181 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.181 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.182 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.182 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3228 3.1 3:8de1d8a3:::smithi12313934-13:head [write 588344~631611,stat] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:14:13.182 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.588539+0000", 2020-01-16T00:14:13.182 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 43.508598345000003, 2020-01-16T00:14:13.182 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012776884000000001, 2020-01-16T00:14:13.182 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.182 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:13.183 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:13.183 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:13.183 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:13.183 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3228 2020-01-16T00:14:13.183 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.183 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.183 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.183 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.184 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588539+0000", 2020-01-16T00:14:13.184 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.184 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.184 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.184 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.184 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588539+0000", 2020-01-16T00:14:13.184 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:13.184 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.185 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.185 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.185 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588536+0000", 2020-01-16T00:14:13.185 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021452 2020-01-16T00:14:13.185 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.185 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.185 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.186 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588750+0000", 2020-01-16T00:14:13.186 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.1432000000000001e-05 2020-01-16T00:14:13.186 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.186 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.186 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.186 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588772+0000", 2020-01-16T00:14:13.186 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1799999999999999e-06 2020-01-16T00:14:13.186 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.187 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.187 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.187 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588779+0000", 2020-01-16T00:14:13.187 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00095777700000000004 2020-01-16T00:14:13.187 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.187 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.187 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.187 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589737+0000", 2020-01-16T00:14:13.188 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013306599999999999 2020-01-16T00:14:13.188 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.188 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.188 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.188 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589870+0000", 2020-01-16T00:14:13.188 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020556 2020-01-16T00:14:13.188 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.189 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.189 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:14:13.189 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.590075+0000", 2020-01-16T00:14:13.189 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040226080000000004 2020-01-16T00:14:13.189 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.189 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.189 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:13.189 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.594098+0000", 2020-01-16T00:14:13.190 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.006043288 2020-01-16T00:14:13.190 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.190 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.190 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:13.190 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.600141+0000", 2020-01-16T00:14:13.190 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00089242900000000003 2020-01-16T00:14:13.190 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.191 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.191 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:13.191 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601034+0000", 2020-01-16T00:14:13.191 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6097e-05 2020-01-16T00:14:13.191 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.191 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.191 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:13.192 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601110+0000", 2020-01-16T00:14:13.192 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000206088 2020-01-16T00:14:13.192 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.192 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.192 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.192 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601316+0000", 2020-01-16T00:14:13.192 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012776884000000001 2020-01-16T00:14:13.192 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.193 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.193 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.193 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.193 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.193 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.1.13:77 2.1 e45/19 2:bafac7f5:::smithi12313934-46 low_tier:head v 45'100, mlcod=45'98)", 2020-01-16T00:14:13.193 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.589633+0000", 2020-01-16T00:14:13.193 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 43.507504122, 2020-01-16T00:14:13.194 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010560460000000001, 2020-01-16T00:14:13.194 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.194 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:13.194 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.194 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.194 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.194 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589633+0000", 2020-01-16T00:14:13.195 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.195 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.195 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.195 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.195 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589633+0000", 2020-01-16T00:14:13.195 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:13.196 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.196 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.196 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.196 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589630+0000", 2020-01-16T00:14:13.196 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00050499799999999999 2020-01-16T00:14:13.196 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.196 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.196 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.197 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.590135+0000", 2020-01-16T00:14:13.197 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0473e-05 2020-01-16T00:14:13.197 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.197 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.197 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.197 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.590146+0000", 2020-01-16T00:14:13.197 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9710000000000001e-06 2020-01-16T00:14:13.198 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.198 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.198 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.198 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.590152+0000", 2020-01-16T00:14:13.198 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00040412999999999999 2020-01-16T00:14:13.198 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.198 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.198 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.199 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.590556+0000", 2020-01-16T00:14:13.199 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6491000000000001e-05 2020-01-16T00:14:13.199 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.199 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.199 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.199 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.590583+0000", 2020-01-16T00:14:13.199 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095621360000000006 2020-01-16T00:14:13.199 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.200 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.200 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:13.200 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.600145+0000", 2020-01-16T00:14:13.200 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9072e-05 2020-01-16T00:14:13.200 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.200 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.200 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.201 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.600194+0000", 2020-01-16T00:14:13.201 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010560460000000001 2020-01-16T00:14:13.201 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.201 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.201 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.201 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.201 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.201 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3229 3.1 3:8de1d8a3:::smithi12313934-13:head [write 1689059~741123] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:14:13.202 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.596087+0000", 2020-01-16T00:14:13.202 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 43.501050284000002, 2020-01-16T00:14:13.202 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010699248, 2020-01-16T00:14:13.202 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.202 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:13.202 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:13.202 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:13.203 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:13.203 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3229 2020-01-16T00:14:13.203 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.203 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.203 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.203 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.203 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596087+0000", 2020-01-16T00:14:13.203 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.204 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.204 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.204 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.204 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596087+0000", 2020-01-16T00:14:13.204 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:13.204 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.204 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.204 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.205 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596084+0000", 2020-01-16T00:14:13.205 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00026243600000000001 2020-01-16T00:14:13.205 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.205 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.205 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.205 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596347+0000", 2020-01-16T00:14:13.205 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3594000000000001e-05 2020-01-16T00:14:13.206 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.206 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.206 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.206 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596370+0000", 2020-01-16T00:14:13.206 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5259999999999999e-06 2020-01-16T00:14:13.206 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.206 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.206 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.207 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596380+0000", 2020-01-16T00:14:13.207 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2298000000000005e-05 2020-01-16T00:14:13.207 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.207 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.207 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.207 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596452+0000", 2020-01-16T00:14:13.208 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000163702 2020-01-16T00:14:13.208 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.208 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.208 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:13.208 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596616+0000", 2020-01-16T00:14:13.208 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0047611729999999996 2020-01-16T00:14:13.208 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.209 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.209 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.209 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601377+0000", 2020-01-16T00:14:13.209 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014076900000000001 2020-01-16T00:14:13.209 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.209 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.209 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.209 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601518+0000", 2020-01-16T00:14:13.210 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021367599999999999 2020-01-16T00:14:13.210 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.210 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.210 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:14:13.210 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601731+0000", 2020-01-16T00:14:13.210 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0028479149999999999 2020-01-16T00:14:13.210 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.210 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.211 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:13.211 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.604579+0000", 2020-01-16T00:14:13.211 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001408133 2020-01-16T00:14:13.211 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.211 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.211 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:13.211 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605987+0000", 2020-01-16T00:14:13.212 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00072335599999999998 2020-01-16T00:14:13.212 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.212 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.212 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:13.212 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.606711+0000", 2020-01-16T00:14:13.212 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0289e-05 2020-01-16T00:14:13.212 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.213 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.213 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:13.213 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.606751+0000", 2020-01-16T00:14:13.213 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5298e-05 2020-01-16T00:14:13.213 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.213 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.213 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.213 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.606786+0000", 2020-01-16T00:14:13.214 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010699248 2020-01-16T00:14:13.214 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.214 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.214 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.214 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.214 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.214 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3230 3.1 3:8de1d8a3:::smithi12313934-13:head [setxattr _header (56) in=7b,truncate 2588344] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:14:13.215 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.596394+0000", 2020-01-16T00:14:13.215 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 43.500743325000002, 2020-01-16T00:14:13.215 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021205134, 2020-01-16T00:14:13.215 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.215 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:13.215 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:13.215 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:13.216 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:13.216 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3230 2020-01-16T00:14:13.216 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.216 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.216 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.216 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.217 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596394+0000", 2020-01-16T00:14:13.217 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.217 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.217 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.217 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.217 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596394+0000", 2020-01-16T00:14:13.217 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:13.218 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.218 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.218 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.218 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596393+0000", 2020-01-16T00:14:13.218 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7159999999999997e-06 2020-01-16T00:14:13.218 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.218 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.218 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.219 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596400+0000", 2020-01-16T00:14:13.219 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0074e-05 2020-01-16T00:14:13.220 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.221 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.221 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.221 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596410+0000", 2020-01-16T00:14:13.221 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1679999999999997e-06 2020-01-16T00:14:13.221 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.221 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.222 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.222 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596414+0000", 2020-01-16T00:14:13.222 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023609 2020-01-16T00:14:13.222 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.222 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.222 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.222 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596650+0000", 2020-01-16T00:14:13.223 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012316399999999999 2020-01-16T00:14:13.223 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.223 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.223 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:13.223 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596774+0000", 2020-01-16T00:14:13.223 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0067849670000000003 2020-01-16T00:14:13.223 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.224 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.224 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.224 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.603558+0000", 2020-01-16T00:14:13.224 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00092385999999999998 2020-01-16T00:14:13.224 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.224 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.224 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:13.224 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.604482+0000", 2020-01-16T00:14:13.225 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011729201 2020-01-16T00:14:13.225 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.225 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.225 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.225 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.616212+0000", 2020-01-16T00:14:13.225 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019515099999999999 2020-01-16T00:14:13.225 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.226 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.226 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.226 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.616407+0000", 2020-01-16T00:14:13.226 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000182843 2020-01-16T00:14:13.226 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.226 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.226 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:14:13.227 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.616590+0000", 2020-01-16T00:14:13.227 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000660572 2020-01-16T00:14:13.227 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.227 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.227 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:13.227 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617250+0000", 2020-01-16T00:14:13.227 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5333999999999997e-05 2020-01-16T00:14:13.227 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.228 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.228 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:13.228 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617285+0000", 2020-01-16T00:14:13.228 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023020899999999999 2020-01-16T00:14:13.228 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.228 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.228 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:13.228 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617516+0000", 2020-01-16T00:14:13.229 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4214000000000001e-05 2020-01-16T00:14:13.229 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.229 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.229 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:13.229 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617560+0000", 2020-01-16T00:14:13.229 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9366999999999998e-05 2020-01-16T00:14:13.229 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.230 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.230 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.230 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617599+0000", 2020-01-16T00:14:13.230 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021205134 2020-01-16T00:14:13.230 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.230 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.230 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.230 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.231 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.231 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3231 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e45)", 2020-01-16T00:14:13.231 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.596438+0000", 2020-01-16T00:14:13.231 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 43.500699314999999, 2020-01-16T00:14:13.231 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021455667000000001, 2020-01-16T00:14:13.231 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.231 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:13.232 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:13.232 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:13.232 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:13.232 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3231 2020-01-16T00:14:13.232 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.232 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.232 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.232 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.233 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596438+0000", 2020-01-16T00:14:13.233 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.233 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.233 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.233 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.233 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596438+0000", 2020-01-16T00:14:13.233 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:13.234 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.234 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.234 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.234 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596437+0000", 2020-01-16T00:14:13.234 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8500000000000004e-06 2020-01-16T00:14:13.234 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.234 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.234 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.235 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596441+0000", 2020-01-16T00:14:13.235 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9439999999999994e-06 2020-01-16T00:14:13.235 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.235 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.235 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.235 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596450+0000", 2020-01-16T00:14:13.235 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.267e-06 2020-01-16T00:14:13.236 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.236 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.236 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.236 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596455+0000", 2020-01-16T00:14:13.236 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034332399999999999 2020-01-16T00:14:13.236 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.236 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.236 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.237 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596798+0000", 2020-01-16T00:14:13.237 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011687400000000001 2020-01-16T00:14:13.237 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.237 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.237 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:13.237 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596915+0000", 2020-01-16T00:14:13.237 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0076057049999999999 2020-01-16T00:14:13.238 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.238 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.238 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.238 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.604520+0000", 2020-01-16T00:14:13.238 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8143999999999998e-05 2020-01-16T00:14:13.238 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.238 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.238 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:13.239 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.604569+0000", 2020-01-16T00:14:13.239 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012379960000000001 2020-01-16T00:14:13.239 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.239 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.239 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.239 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.616949+0000", 2020-01-16T00:14:13.239 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000120617 2020-01-16T00:14:13.239 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.240 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.240 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:13.240 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617069+0000", 2020-01-16T00:14:13.240 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00056715400000000005 2020-01-16T00:14:13.240 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.240 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.240 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.241 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617636+0000", 2020-01-16T00:14:13.241 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012844700000000001 2020-01-16T00:14:13.241 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.241 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.241 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.241 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617765+0000", 2020-01-16T00:14:13.241 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000129037 2020-01-16T00:14:13.241 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.242 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.242 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.242 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617894+0000", 2020-01-16T00:14:13.242 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021455667000000001 2020-01-16T00:14:13.242 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.242 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.242 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.242 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.243 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.243 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3822 3.1 3:8de1d8a3:::smithi12313934-13:head [setxattr _header (56) in=7b,truncate 2016939] snapc 0=[] ondisk+write+known_if_redirected e50)", 2020-01-16T00:14:13.243 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:44.059584+0000", 2020-01-16T00:14:13.243 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 29.037552998999999, 2020-01-16T00:14:13.243 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021977219999999999, 2020-01-16T00:14:13.243 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.243 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:13.243 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:13.244 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:13.244 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:13.244 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3822 2020-01-16T00:14:13.244 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.244 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.244 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.244 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.245 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059584+0000", 2020-01-16T00:14:13.245 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.245 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.245 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.245 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.245 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059584+0000", 2020-01-16T00:14:13.246 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:13.246 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.246 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.246 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.246 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059584+0000", 2020-01-16T00:14:13.246 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1329999999999998e-06 2020-01-16T00:14:13.247 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.247 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.247 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.247 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059589+0000", 2020-01-16T00:14:13.247 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8136000000000002e-05 2020-01-16T00:14:13.247 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.247 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.247 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.248 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059627+0000", 2020-01-16T00:14:13.248 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1409999999999998e-06 2020-01-16T00:14:13.248 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.248 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.248 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.248 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059631+0000", 2020-01-16T00:14:13.248 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0011314020000000001 2020-01-16T00:14:13.248 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.249 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.249 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.249 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.060763+0000", 2020-01-16T00:14:13.249 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00085078200000000001 2020-01-16T00:14:13.249 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.249 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.249 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:13.249 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.061613+0000", 2020-01-16T00:14:13.250 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017766311999999999 2020-01-16T00:14:13.250 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.250 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.250 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.250 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.079380+0000", 2020-01-16T00:14:13.250 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000133144 2020-01-16T00:14:13.250 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.251 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.251 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.251 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.079513+0000", 2020-01-16T00:14:13.251 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021271500000000001 2020-01-16T00:14:13.251 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.251 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.251 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:14:13.252 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.079725+0000", 2020-01-16T00:14:13.252 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001302582 2020-01-16T00:14:13.252 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.252 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.252 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:13.252 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081028+0000", 2020-01-16T00:14:13.252 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017197 2020-01-16T00:14:13.253 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.253 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.253 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:13.253 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081200+0000", 2020-01-16T00:14:13.253 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000202757 2020-01-16T00:14:13.253 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.253 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.253 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:13.254 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081403+0000", 2020-01-16T00:14:13.254 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1068999999999993e-05 2020-01-16T00:14:13.254 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.254 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.254 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:13.254 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081474+0000", 2020-01-16T00:14:13.254 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7800000000000006e-05 2020-01-16T00:14:13.254 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.255 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.255 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.255 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081562+0000", 2020-01-16T00:14:13.255 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021977219999999999 2020-01-16T00:14:13.255 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.255 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.255 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.255 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.256 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.256 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3823 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e50)", 2020-01-16T00:14:13.256 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:44.059639+0000", 2020-01-16T00:14:13.256 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 29.037498844000002, 2020-01-16T00:14:13.256 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022245396000000001, 2020-01-16T00:14:13.256 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.256 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:13.257 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:13.257 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:13.257 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:13.257 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3823 2020-01-16T00:14:13.257 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.257 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.257 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.258 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.258 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059639+0000", 2020-01-16T00:14:13.258 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.258 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.258 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.258 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.258 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059639+0000", 2020-01-16T00:14:13.258 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:13.259 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.259 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.259 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.259 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059638+0000", 2020-01-16T00:14:13.259 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.405e-06 2020-01-16T00:14:13.259 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.259 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.259 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.260 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059641+0000", 2020-01-16T00:14:13.260 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9770000000000003e-06 2020-01-16T00:14:13.260 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.260 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.260 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.260 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059648+0000", 2020-01-16T00:14:13.260 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1140000000000002e-06 2020-01-16T00:14:13.260 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.261 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.261 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.261 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059652+0000", 2020-01-16T00:14:13.261 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020063659999999999 2020-01-16T00:14:13.261 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.261 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.261 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.262 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.061658+0000", 2020-01-16T00:14:13.262 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3183999999999999e-05 2020-01-16T00:14:13.262 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.262 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.262 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:13.262 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.061691+0000", 2020-01-16T00:14:13.262 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018817654999999999 2020-01-16T00:14:13.262 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.263 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.263 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.263 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.080509+0000", 2020-01-16T00:14:13.263 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000104409 2020-01-16T00:14:13.263 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.263 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.263 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:13.263 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.080613+0000", 2020-01-16T00:14:13.264 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00098755999999999996 2020-01-16T00:14:13.264 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.264 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.264 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.264 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081601+0000", 2020-01-16T00:14:13.264 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000145766 2020-01-16T00:14:13.264 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.265 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.265 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.265 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081746+0000", 2020-01-16T00:14:13.265 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013751799999999999 2020-01-16T00:14:13.265 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.265 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.266 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.266 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081884+0000", 2020-01-16T00:14:13.266 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022245396000000001 2020-01-16T00:14:13.266 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.266 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.266 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.266 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.266 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.267 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4398 3.1 3:8de1d8a3:::smithi12313934-13:head [write 2812170~507386] snapc 0=[] ondisk+write+known_if_redirected e55)", 2020-01-16T00:14:13.267 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:58.336543+0000", 2020-01-16T00:14:13.267 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 14.760594411, 2020-01-16T00:14:13.267 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015697994, 2020-01-16T00:14:13.267 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.267 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:13.267 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:13.267 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:13.268 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:13.268 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4398 2020-01-16T00:14:13.268 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.268 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.268 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.268 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.268 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336543+0000", 2020-01-16T00:14:13.269 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.269 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.269 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.269 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.269 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336543+0000", 2020-01-16T00:14:13.269 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:13.269 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.269 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.270 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.270 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336542+0000", 2020-01-16T00:14:13.270 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014254 2020-01-16T00:14:13.270 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.270 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.270 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.270 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336684+0000", 2020-01-16T00:14:13.271 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2019e-05 2020-01-16T00:14:13.271 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.271 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.271 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.271 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336696+0000", 2020-01-16T00:14:13.271 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4269999999999996e-06 2020-01-16T00:14:13.271 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.272 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.272 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.272 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336702+0000", 2020-01-16T00:14:13.272 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8040000000000002e-05 2020-01-16T00:14:13.272 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.272 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.272 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.273 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336741+0000", 2020-01-16T00:14:13.273 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00030631400000000002 2020-01-16T00:14:13.273 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.273 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.273 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:13.273 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.337047+0000", 2020-01-16T00:14:13.273 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012249832 2020-01-16T00:14:13.273 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.274 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.274 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.274 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.349297+0000", 2020-01-16T00:14:13.274 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013870899999999999 2020-01-16T00:14:13.274 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.274 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.274 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.274 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.349435+0000", 2020-01-16T00:14:13.275 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021913600000000001 2020-01-16T00:14:13.275 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.275 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.275 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:14:13.275 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.349655+0000", 2020-01-16T00:14:13.275 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021982490000000002 2020-01-16T00:14:13.275 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.276 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.276 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:13.276 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351853+0000", 2020-01-16T00:14:13.276 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8295999999999996e-05 2020-01-16T00:14:13.276 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.276 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.276 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:13.276 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351941+0000", 2020-01-16T00:14:13.277 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021662 2020-01-16T00:14:13.277 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.277 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.277 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:13.277 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352158+0000", 2020-01-16T00:14:13.277 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6031000000000001e-05 2020-01-16T00:14:13.277 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.277 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.278 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:13.278 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352194+0000", 2020-01-16T00:14:13.278 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7290000000000003e-05 2020-01-16T00:14:13.278 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.278 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.278 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.278 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352241+0000", 2020-01-16T00:14:13.278 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015697994 2020-01-16T00:14:13.279 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.279 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.279 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.279 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.279 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.279 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4399 3.1 3:8de1d8a3:::smithi12313934-13:head [setxattr _header (56) in=7b,truncate 3411920] snapc 0=[] ondisk+write+known_if_redirected e55)", 2020-01-16T00:14:13.279 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:58.336714+0000", 2020-01-16T00:14:13.280 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 14.760423054, 2020-01-16T00:14:13.280 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016051421, 2020-01-16T00:14:13.280 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.280 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:13.280 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:13.280 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:13.280 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:13.281 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4399 2020-01-16T00:14:13.281 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.281 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.281 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.281 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.281 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336714+0000", 2020-01-16T00:14:13.281 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.282 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.282 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.282 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.282 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336714+0000", 2020-01-16T00:14:13.282 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:13.282 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.282 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.282 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.283 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336714+0000", 2020-01-16T00:14:13.283 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7540000000000003e-06 2020-01-16T00:14:13.283 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.283 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.283 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.283 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336719+0000", 2020-01-16T00:14:13.283 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2879999999999998e-06 2020-01-16T00:14:13.284 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.284 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.284 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.284 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336726+0000", 2020-01-16T00:14:13.284 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6399999999999996e-06 2020-01-16T00:14:13.284 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.284 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.284 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.285 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336730+0000", 2020-01-16T00:14:13.285 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00036299699999999999 2020-01-16T00:14:13.285 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.285 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.285 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.285 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.337093+0000", 2020-01-16T00:14:13.285 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1297999999999994e-05 2020-01-16T00:14:13.286 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.286 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.286 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:13.286 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.337165+0000", 2020-01-16T00:14:13.286 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013848911 2020-01-16T00:14:13.286 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.286 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.286 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.287 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351013+0000", 2020-01-16T00:14:13.287 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7732999999999994e-05 2020-01-16T00:14:13.287 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.287 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.287 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.287 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351111+0000", 2020-01-16T00:14:13.287 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000109089 2020-01-16T00:14:13.287 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.288 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.288 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:14:13.288 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351220+0000", 2020-01-16T00:14:13.288 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00063960199999999997 2020-01-16T00:14:13.288 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.288 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.288 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:13.289 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351860+0000", 2020-01-16T00:14:13.289 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00050205699999999998 2020-01-16T00:14:13.289 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.289 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.289 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:13.289 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352362+0000", 2020-01-16T00:14:13.289 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000282195 2020-01-16T00:14:13.290 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.290 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.290 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:13.290 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352644+0000", 2020-01-16T00:14:13.290 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3927000000000001e-05 2020-01-16T00:14:13.290 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.290 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.290 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:13.291 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352698+0000", 2020-01-16T00:14:13.291 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7770999999999999e-05 2020-01-16T00:14:13.291 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.291 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.291 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.291 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352766+0000", 2020-01-16T00:14:13.291 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016051421 2020-01-16T00:14:13.292 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.292 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.292 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.292 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.292 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.292 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4400 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e55)", 2020-01-16T00:14:13.292 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:58.336738+0000", 2020-01-16T00:14:13.292 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 14.760399325, 2020-01-16T00:14:13.293 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016207452000000001, 2020-01-16T00:14:13.293 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.293 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:13.293 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:13.293 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:13.293 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:13.293 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4400 2020-01-16T00:14:13.294 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.294 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.294 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.294 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.294 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336738+0000", 2020-01-16T00:14:13.294 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.294 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.294 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.295 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.295 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336738+0000", 2020-01-16T00:14:13.295 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:13.295 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.295 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.295 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.295 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336738+0000", 2020-01-16T00:14:13.295 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5200000000000002e-06 2020-01-16T00:14:13.296 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.296 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.296 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.296 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336741+0000", 2020-01-16T00:14:13.296 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1500000000000004e-06 2020-01-16T00:14:13.296 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.297 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.297 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.297 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336747+0000", 2020-01-16T00:14:13.297 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7500000000000001e-06 2020-01-16T00:14:13.297 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.297 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.297 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.298 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336751+0000", 2020-01-16T00:14:13.298 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043512299999999999 2020-01-16T00:14:13.298 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.298 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.298 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.298 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.337186+0000", 2020-01-16T00:14:13.298 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7048000000000002e-05 2020-01-16T00:14:13.298 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.299 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.299 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:13.299 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.337223+0000", 2020-01-16T00:14:13.299 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014258785 2020-01-16T00:14:13.299 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.299 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.299 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.299 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351482+0000", 2020-01-16T00:14:13.300 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012002000000000001 2020-01-16T00:14:13.300 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.300 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.300 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:13.300 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351602+0000", 2020-01-16T00:14:13.300 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001192404 2020-01-16T00:14:13.300 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.301 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.301 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.301 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352794+0000", 2020-01-16T00:14:13.301 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8901999999999998e-05 2020-01-16T00:14:13.301 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.301 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.301 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.301 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352873+0000", 2020-01-16T00:14:13.302 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2317999999999994e-05 2020-01-16T00:14:13.302 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.302 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.302 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.302 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352946+0000", 2020-01-16T00:14:13.302 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016207452000000001 2020-01-16T00:14:13.302 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.303 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.303 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.303 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.303 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.303 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:183 2.1 e58/19 2:a4df40e2:::smithi12313934-42 low_tier:head v 58'138, mlcod=57'136)", 2020-01-16T00:14:13.303 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.021967+0000", 2020-01-16T00:14:13.303 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 6.0751704560000004, 2020-01-16T00:14:13.303 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011697308999999999, 2020-01-16T00:14:13.304 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.304 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:13.304 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.304 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.304 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.304 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021967+0000", 2020-01-16T00:14:13.304 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.305 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.305 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.305 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.305 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021967+0000", 2020-01-16T00:14:13.305 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:13.305 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.305 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.305 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.306 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021966+0000", 2020-01-16T00:14:13.306 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020837899999999999 2020-01-16T00:14:13.306 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.306 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.306 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.306 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.022175+0000", 2020-01-16T00:14:13.306 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7650000000000005e-06 2020-01-16T00:14:13.307 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.307 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.307 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.307 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.022185+0000", 2020-01-16T00:14:13.307 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.759e-06 2020-01-16T00:14:13.307 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.307 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.308 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.308 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.022191+0000", 2020-01-16T00:14:13.308 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00559108 2020-01-16T00:14:13.308 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.308 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.308 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.308 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.027782+0000", 2020-01-16T00:14:13.309 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5761e-05 2020-01-16T00:14:13.309 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.309 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.309 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.309 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.027798+0000", 2020-01-16T00:14:13.309 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0058039989999999998 2020-01-16T00:14:13.309 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.310 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.310 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:13.310 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.033602+0000", 2020-01-16T00:14:13.310 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2147e-05 2020-01-16T00:14:13.310 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.310 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.310 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.310 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.033664+0000", 2020-01-16T00:14:13.311 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011697308999999999 2020-01-16T00:14:13.311 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.311 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.311 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.311 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.311 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.311 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.1.13:98 2.1 e58/19 2:bafac7f5:::smithi12313934-46 low_tier:head v 58'140, mlcod=58'138)", 2020-01-16T00:14:13.312 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.050024+0000", 2020-01-16T00:14:13.312 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 6.0471130979999996, 2020-01-16T00:14:13.312 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011307467, 2020-01-16T00:14:13.312 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.312 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:13.312 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.312 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.312 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.313 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.050024+0000", 2020-01-16T00:14:13.313 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.313 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.313 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.313 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.313 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.050024+0000", 2020-01-16T00:14:13.314 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:13.314 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.314 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.314 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.314 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.050022+0000", 2020-01-16T00:14:13.314 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000438839 2020-01-16T00:14:13.314 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.315 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.315 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.315 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.050461+0000", 2020-01-16T00:14:13.315 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9307999999999999e-05 2020-01-16T00:14:13.315 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.315 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.315 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.315 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.050480+0000", 2020-01-16T00:14:13.316 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7249999999999997e-06 2020-01-16T00:14:13.316 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.316 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.316 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.316 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.050488+0000", 2020-01-16T00:14:13.316 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00026342100000000001 2020-01-16T00:14:13.317 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.317 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.317 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.317 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.050751+0000", 2020-01-16T00:14:13.317 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8136000000000001e-05 2020-01-16T00:14:13.317 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.318 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.318 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.318 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.050769+0000", 2020-01-16T00:14:13.318 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010518582 2020-01-16T00:14:13.318 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.318 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.318 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:13.319 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.061288+0000", 2020-01-16T00:14:13.319 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4085000000000002e-05 2020-01-16T00:14:13.319 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.319 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.319 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.319 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.061332+0000", 2020-01-16T00:14:13.319 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011307467 2020-01-16T00:14:13.319 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.320 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.320 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.320 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.320 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.320 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:13.322 INFO:tasks.rados.rados.0.smithi123.stdout:1735: finishing write tid 2 to smithi12313934-16 2020-01-16T00:14:13.322 INFO:tasks.rados.rados.0.smithi123.stdout:1735: finishing write tid 4 to smithi12313934-16 2020-01-16T00:14:13.322 INFO:tasks.rados.rados.0.smithi123.stdout:1735: finishing write tid 5 to smithi12313934-16 2020-01-16T00:14:13.322 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 16 v 274 (ObjNum 625 snap 0 seq_num 625) dirty exists 2020-01-16T00:14:13.322 INFO:tasks.rados.rados.0.smithi123.stdout:1735: left oid 16 (ObjNum 625 snap 0 seq_num 625) 2020-01-16T00:14:13.322 INFO:tasks.rados.rados.0.smithi123.stdout:1739: expect (ObjNum 607 snap 0 seq_num 607) 2020-01-16T00:14:13.323 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:13.320+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:13.323 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-01-16T00:14:13.332 INFO:tasks.rados.rados.0.smithi123.stdout:1740: finishing write tid 1 to smithi12313934-12 2020-01-16T00:14:13.333 INFO:tasks.rados.rados.0.smithi123.stdout:1740: finishing write tid 2 to smithi12313934-12 2020-01-16T00:14:13.333 INFO:tasks.rados.rados.0.smithi123.stdout:1740: finishing write tid 4 to smithi12313934-12 2020-01-16T00:14:13.333 INFO:tasks.rados.rados.0.smithi123.stdout:1740: finishing write tid 5 to smithi12313934-12 2020-01-16T00:14:13.333 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 12 v 179 (ObjNum 626 snap 0 seq_num 626) dirty exists 2020-01-16T00:14:13.333 INFO:tasks.rados.rados.0.smithi123.stdout:1740: left oid 12 (ObjNum 626 snap 0 seq_num 626) 2020-01-16T00:14:13.333 INFO:tasks.rados.rados.0.smithi123.stdout:1741: finishing write tid 1 to smithi12313934-17 2020-01-16T00:14:13.334 INFO:tasks.rados.rados.0.smithi123.stdout:1741: finishing write tid 2 to smithi12313934-17 2020-01-16T00:14:13.334 INFO:tasks.rados.rados.0.smithi123.stdout:1741: finishing write tid 4 to smithi12313934-17 2020-01-16T00:14:13.334 INFO:tasks.rados.rados.0.smithi123.stdout:1741: finishing write tid 5 to smithi12313934-17 2020-01-16T00:14:13.334 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 17 v 199 (ObjNum 627 snap 0 seq_num 627) dirty exists 2020-01-16T00:14:13.334 INFO:tasks.rados.rados.0.smithi123.stdout:1741: left oid 17 (ObjNum 627 snap 0 seq_num 627) 2020-01-16T00:14:13.334 INFO:tasks.rados.rados.0.smithi123.stdout:1742: finishing write tid 1 to smithi12313934-35 2020-01-16T00:14:13.334 INFO:tasks.rados.rados.0.smithi123.stdout:1734: done (15 left) 2020-01-16T00:14:13.334 INFO:tasks.rados.rados.0.smithi123.stdout:1735: done (14 left) 2020-01-16T00:14:13.335 INFO:tasks.rados.rados.0.smithi123.stdout:1736: done (13 left) 2020-01-16T00:14:13.335 INFO:tasks.rados.rados.0.smithi123.stdout:1739: done (12 left) 2020-01-16T00:14:13.335 INFO:tasks.rados.rados.0.smithi123.stdout:1740: done (11 left) 2020-01-16T00:14:13.335 INFO:tasks.rados.rados.0.smithi123.stdout:1741: done (10 left) 2020-01-16T00:14:13.335 INFO:tasks.rados.rados.0.smithi123.stdout:1752: write oid 2 current snap is 0 2020-01-16T00:14:13.335 INFO:tasks.rados.rados.0.smithi123.stdout:1752: seq_num 636 ranges {689252=715663,1900783=425211} 2020-01-16T00:14:13.350 INFO:tasks.rados.rados.0.smithi123.stdout:1752: writing smithi12313934-2 from 689252 to 1404915 tid 1 2020-01-16T00:14:13.356 INFO:tasks.rados.rados.0.smithi123.stdout:1752: writing smithi12313934-2 from 1900783 to 2325994 tid 2 2020-01-16T00:14:13.356 INFO:tasks.rados.rados.0.smithi123.stdout:1753: read oid 13 snap -1 2020-01-16T00:14:13.356 INFO:tasks.rados.rados.0.smithi123.stdout:1753: expect (ObjNum 560 snap 0 seq_num 560) 2020-01-16T00:14:13.356 INFO:tasks.rados.rados.0.smithi123.stdout:1754: write oid 44 current snap is 0 2020-01-16T00:14:13.356 INFO:tasks.rados.rados.0.smithi123.stdout:1754: seq_num 637 ranges {706059=790912,1959265=685222} 2020-01-16T00:14:13.364 INFO:tasks.rados.rados.0.smithi123.stdout:1754: writing smithi12313934-44 from 706059 to 1496971 tid 1 2020-01-16T00:14:13.372 INFO:tasks.rados.rados.0.smithi123.stdout:1754: writing smithi12313934-44 from 1959265 to 2644487 tid 2 2020-01-16T00:14:13.372 INFO:tasks.rados.rados.0.smithi123.stdout:1755: read oid 43 snap -1 2020-01-16T00:14:13.372 INFO:tasks.rados.rados.0.smithi123.stdout:1755: expect (ObjNum 607 snap 0 seq_num 607) 2020-01-16T00:14:13.372 INFO:tasks.rados.rados.0.smithi123.stdout:1756: read oid 26 snap -1 2020-01-16T00:14:13.372 INFO:tasks.rados.rados.0.smithi123.stdout:1756: expect (ObjNum 556 snap 0 seq_num 556) 2020-01-16T00:14:13.373 INFO:tasks.rados.rados.0.smithi123.stdout:1757: write oid 40 current snap is 0 2020-01-16T00:14:13.373 INFO:tasks.rados.rados.0.smithi123.stdout:1742: finishing write tid 2 to smithi12313934-35 2020-01-16T00:14:13.373 INFO:tasks.rados.rados.0.smithi123.stdout:1757: seq_num 638 ranges {722866=466161,1617747=428880,2587634=135232} 2020-01-16T00:14:13.379 INFO:tasks.rados.rados.0.smithi123.stdout:1757: writing smithi12313934-40 from 722866 to 1189027 tid 1 2020-01-16T00:14:13.384 INFO:tasks.rados.rados.0.smithi123.stdout:1757: writing smithi12313934-40 from 1617747 to 2046627 tid 2 2020-01-16T00:14:13.388 INFO:tasks.rados.rados.0.smithi123.stdout:1757: writing smithi12313934-40 from 2587634 to 2722866 tid 3 2020-01-16T00:14:13.389 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:13.389 INFO:tasks.rados.rados.0.smithi123.stdout:1742: finishing write tid 4 to smithi12313934-35 2020-01-16T00:14:13.389 INFO:tasks.rados.rados.0.smithi123.stdout:1742: finishing write tid 5 to smithi12313934-35 2020-01-16T00:14:13.389 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 35 v 202 (ObjNum 628 snap 0 seq_num 628) dirty exists 2020-01-16T00:14:13.389 INFO:tasks.rados.rados.0.smithi123.stdout:1742: left oid 35 (ObjNum 628 snap 0 seq_num 628) 2020-01-16T00:14:13.389 INFO:tasks.rados.rados.0.smithi123.stdout:1743: finishing write tid 1 to smithi12313934-6 2020-01-16T00:14:13.390 INFO:tasks.rados.rados.0.smithi123.stdout:1743: finishing write tid 2 to smithi12313934-6 2020-01-16T00:14:13.390 INFO:tasks.rados.rados.0.smithi123.stdout:1743: finishing write tid 4 to smithi12313934-6 2020-01-16T00:14:13.390 INFO:tasks.rados.rados.0.smithi123.stdout:1743: finishing write tid 5 to smithi12313934-6 2020-01-16T00:14:13.390 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 6 v 183 (ObjNum 629 snap 0 seq_num 629) dirty exists 2020-01-16T00:14:13.390 INFO:tasks.rados.rados.0.smithi123.stdout:1743: left oid 6 (ObjNum 629 snap 0 seq_num 629) 2020-01-16T00:14:13.390 INFO:tasks.rados.rados.0.smithi123.stdout:1744: finishing write tid 1 to smithi12313934-24 2020-01-16T00:14:13.390 INFO:tasks.rados.rados.0.smithi123.stdout:1744: finishing write tid 2 to smithi12313934-24 2020-01-16T00:14:13.391 INFO:tasks.rados.rados.0.smithi123.stdout:1744: finishing write tid 3 to smithi12313934-24 2020-01-16T00:14:13.391 INFO:tasks.rados.rados.0.smithi123.stdout:1744: finishing write tid 5 to smithi12313934-24 2020-01-16T00:14:13.391 INFO:tasks.rados.rados.0.smithi123.stdout:1744: finishing write tid 6 to smithi12313934-24 2020-01-16T00:14:13.391 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 24 v 184 (ObjNum 630 snap 0 seq_num 630) dirty exists 2020-01-16T00:14:13.391 INFO:tasks.rados.rados.0.smithi123.stdout:1744: left oid 24 (ObjNum 630 snap 0 seq_num 630) 2020-01-16T00:14:13.391 INFO:tasks.rados.rados.0.smithi123.stdout:1742: done (15 left) 2020-01-16T00:14:13.391 INFO:tasks.rados.rados.0.smithi123.stdout:1743: done (14 left) 2020-01-16T00:14:13.391 INFO:tasks.rados.rados.0.smithi123.stdout:1744: done (13 left) 2020-01-16T00:14:13.392 INFO:tasks.rados.rados.0.smithi123.stdout:1758: read oid 1 snap -1 2020-01-16T00:14:13.392 INFO:tasks.rados.rados.0.smithi123.stdout:1758: expect (ObjNum 618 snap 0 seq_num 618) 2020-01-16T00:14:13.392 INFO:tasks.rados.rados.0.smithi123.stdout:1759: write oid 45 current snap is 0 2020-01-16T00:14:13.392 INFO:tasks.rados.rados.0.smithi123.stdout:1759: seq_num 639 ranges {739673=657763,2076229=663444} 2020-01-16T00:14:13.396 INFO:tasks.rados.rados.0.smithi123.stdout:1759: writing smithi12313934-45 from 739673 to 1397436 tid 1 2020-01-16T00:14:13.404 INFO:tasks.rados.rados.0.smithi123.stdout:1759: writing smithi12313934-45 from 2076229 to 2739673 tid 2 2020-01-16T00:14:13.404 INFO:tasks.rados.rados.0.smithi123.stdout:1760: read oid 50 snap -1 2020-01-16T00:14:13.404 INFO:tasks.rados.rados.0.smithi123.stdout:1760: expect (ObjNum 595 snap 0 seq_num 595) 2020-01-16T00:14:13.404 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:13.404 INFO:tasks.rados.rados.0.smithi123.stdout:1745: expect (ObjNum 610 snap 0 seq_num 610) 2020-01-16T00:14:13.422 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:13.418+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:13.447 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:13.448 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:14:13.448 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:14:13.448 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:13.456 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-01-16T00:14:13.523 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:13.520+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:13.599 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:13.600 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:14:13.600 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:14:13.600 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:14:13.600 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:13.601 INFO:tasks.rados.rados.0.smithi123.stdout:1746: finishing write tid 1 to smithi12313934-37 2020-01-16T00:14:13.601 INFO:tasks.rados.rados.0.smithi123.stdout:1746: finishing write tid 2 to smithi12313934-37 2020-01-16T00:14:13.601 INFO:tasks.rados.rados.0.smithi123.stdout:1746: finishing write tid 4 to smithi12313934-37 2020-01-16T00:14:13.601 INFO:tasks.rados.rados.0.smithi123.stdout:1746: finishing write tid 5 to smithi12313934-37 2020-01-16T00:14:13.602 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 37 v 329 (ObjNum 631 snap 0 seq_num 631) dirty exists 2020-01-16T00:14:13.602 INFO:tasks.rados.rados.0.smithi123.stdout:1746: left oid 37 (ObjNum 631 snap 0 seq_num 631) 2020-01-16T00:14:13.602 INFO:tasks.rados.rados.0.smithi123.stdout:1747: expect (ObjNum 611 snap 0 seq_num 611) 2020-01-16T00:14:13.608 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-01-16T00:14:13.627 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-01-16T00:14:13.655 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:13.654+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:13.711 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:13.711 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:14:13.711 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:14:13.711 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:14:13.711 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.712 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2226 3.7 3:e1be6d3b:::smithi12313934-24:head [setxattr _header (56) in=7b,truncate 2100908] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:14:13.712 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.980217+0000", 2020-01-16T00:14:13.712 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 67.727075170999996, 2020-01-16T00:14:13.712 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023113018999999999, 2020-01-16T00:14:13.712 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.712 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:13.712 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:13.712 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:13.713 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:13.713 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2226 2020-01-16T00:14:13.713 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.713 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.713 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.713 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.713 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980217+0000", 2020-01-16T00:14:13.714 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.714 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.714 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.714 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.714 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980217+0000", 2020-01-16T00:14:13.714 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:13.714 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.715 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.715 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.715 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980216+0000", 2020-01-16T00:14:13.715 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5049999999999998e-06 2020-01-16T00:14:13.716 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.716 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.716 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.716 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980222+0000", 2020-01-16T00:14:13.724 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5654999999999998e-05 2020-01-16T00:14:13.724 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.724 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.725 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.725 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980237+0000", 2020-01-16T00:14:13.725 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9740000000000004e-06 2020-01-16T00:14:13.725 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.725 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.725 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.725 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980241+0000", 2020-01-16T00:14:13.726 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002947776 2020-01-16T00:14:13.726 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.726 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.726 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.726 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.983189+0000", 2020-01-16T00:14:13.726 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0027099400000000001 2020-01-16T00:14:13.726 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.726 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.727 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:13.727 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985899+0000", 2020-01-16T00:14:13.727 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014275589999999999 2020-01-16T00:14:13.727 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.727 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.727 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.727 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.000175+0000", 2020-01-16T00:14:13.728 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011211 2020-01-16T00:14:13.728 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.728 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.728 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.728 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.000287+0000", 2020-01-16T00:14:13.728 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000229886 2020-01-16T00:14:13.728 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.728 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.729 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,7", 2020-01-16T00:14:13.729 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.000517+0000", 2020-01-16T00:14:13.729 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00095715399999999999 2020-01-16T00:14:13.729 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.729 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.729 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:13.729 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.001474+0000", 2020-01-16T00:14:13.729 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001341512 2020-01-16T00:14:13.730 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.730 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.730 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:13.730 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.002815+0000", 2020-01-16T00:14:13.730 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035789099999999999 2020-01-16T00:14:13.730 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.730 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.731 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:13.731 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003173+0000", 2020-01-16T00:14:13.731 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8571000000000005e-05 2020-01-16T00:14:13.731 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.731 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.731 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:13.731 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003242+0000", 2020-01-16T00:14:13.731 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8293000000000003e-05 2020-01-16T00:14:13.732 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.732 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.732 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.732 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003330+0000", 2020-01-16T00:14:13.732 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023113018999999999 2020-01-16T00:14:13.732 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.732 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.733 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.733 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.733 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.733 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2227 3.7 3:e1be6d3b:::smithi12313934-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e35)", 2020-01-16T00:14:13.733 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:14:13.737 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 67.727027918000005, 2020-01-16T00:14:13.737 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023303356000000001, 2020-01-16T00:14:13.738 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.738 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:13.738 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:13.739 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:13.739 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:13.739 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2227 2020-01-16T00:14:13.739 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.739 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.739 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.740 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.740 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:14:13.740 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.740 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.740 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.740 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.740 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:14:13.740 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:13.741 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.741 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.741 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.741 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:14:13.741 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3079999999999999e-06 2020-01-16T00:14:13.741 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.741 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.742 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.742 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980267+0000", 2020-01-16T00:14:13.742 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9290000000000001e-06 2020-01-16T00:14:13.742 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.742 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.742 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.742 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980274+0000", 2020-01-16T00:14:13.742 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7400000000000002e-06 2020-01-16T00:14:13.743 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.743 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.743 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.743 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980278+0000", 2020-01-16T00:14:13.743 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056624300000000004 2020-01-16T00:14:13.744 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.744 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.746 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.746 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985940+0000", 2020-01-16T00:14:13.746 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4261000000000002e-05 2020-01-16T00:14:13.746 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.746 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.746 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:13.746 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985985+0000", 2020-01-16T00:14:13.747 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015087676 2020-01-16T00:14:13.747 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.747 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.747 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.747 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.001072+0000", 2020-01-16T00:14:13.758 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011960600000000001 2020-01-16T00:14:13.758 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.758 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.758 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:13.758 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.001192+0000", 2020-01-16T00:14:13.759 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021718309999999999 2020-01-16T00:14:13.759 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.759 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.759 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.759 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003364+0000", 2020-01-16T00:14:13.759 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000102855 2020-01-16T00:14:13.759 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.759 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.760 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.760 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003467+0000", 2020-01-16T00:14:13.760 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010122499999999999 2020-01-16T00:14:13.760 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.760 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.760 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.760 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003568+0000", 2020-01-16T00:14:13.760 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023303356000000001 2020-01-16T00:14:13.761 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.761 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.761 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.761 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.761 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.761 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2325 3.7 3:e1be6d3b:::smithi12313934-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e36)", 2020-01-16T00:14:13.761 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.550877+0000", 2020-01-16T00:14:13.762 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 66.156415590999998, 2020-01-16T00:14:13.762 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018565154, 2020-01-16T00:14:13.762 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.762 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:13.763 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:13.763 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:13.763 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:13.763 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2325 2020-01-16T00:14:13.763 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.763 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.763 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.764 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.764 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550877+0000", 2020-01-16T00:14:13.764 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.764 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.765 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.765 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.765 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550877+0000", 2020-01-16T00:14:13.765 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:13.765 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.765 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.766 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.766 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550876+0000", 2020-01-16T00:14:13.766 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1080000000000003e-06 2020-01-16T00:14:13.766 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.766 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.766 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.767 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550880+0000", 2020-01-16T00:14:13.767 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3269999999999997e-06 2020-01-16T00:14:13.767 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.767 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.767 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.767 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550886+0000", 2020-01-16T00:14:13.767 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.386e-06 2020-01-16T00:14:13.768 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.768 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.768 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.768 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550891+0000", 2020-01-16T00:14:13.768 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048231410000000004 2020-01-16T00:14:13.768 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.769 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.769 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.769 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555714+0000", 2020-01-16T00:14:13.769 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0071999999999999e-05 2020-01-16T00:14:13.769 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.769 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.769 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:13.770 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555744+0000", 2020-01-16T00:14:13.770 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012329005000000001 2020-01-16T00:14:13.770 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.770 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.770 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.770 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568073+0000", 2020-01-16T00:14:13.771 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9087999999999997e-05 2020-01-16T00:14:13.771 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.771 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.771 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:13.771 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568172+0000", 2020-01-16T00:14:13.771 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010718220000000001 2020-01-16T00:14:13.771 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.772 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.772 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.772 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569244+0000", 2020-01-16T00:14:13.772 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8199e-05 2020-01-16T00:14:13.772 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.772 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.772 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.772 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569342+0000", 2020-01-16T00:14:13.773 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9708000000000001e-05 2020-01-16T00:14:13.773 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.773 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.773 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.773 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569442+0000", 2020-01-16T00:14:13.774 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018565154 2020-01-16T00:14:13.774 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.774 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.774 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.774 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.775 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.775 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2800 3.2 3:42b7f20f:::smithi12313934-42:head [write 1395221~583781] snapc 0=[] ondisk+write+known_if_redirected e41)", 2020-01-16T00:14:13.775 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:17.256182+0000", 2020-01-16T00:14:13.775 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 56.451110051000001, 2020-01-16T00:14:13.775 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018509650999999998, 2020-01-16T00:14:13.775 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.776 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:13.776 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:13.776 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:13.776 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:13.776 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2800 2020-01-16T00:14:13.776 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.776 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.777 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.777 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.777 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256182+0000", 2020-01-16T00:14:13.777 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.777 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.777 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.777 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.777 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256182+0000", 2020-01-16T00:14:13.778 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:13.778 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.778 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.778 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.778 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256180+0000", 2020-01-16T00:14:13.779 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020514799999999999 2020-01-16T00:14:13.779 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.779 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.779 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.779 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256385+0000", 2020-01-16T00:14:13.779 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0288e-05 2020-01-16T00:14:13.779 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.780 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.780 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.780 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256406+0000", 2020-01-16T00:14:13.780 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7160000000000003e-06 2020-01-16T00:14:13.780 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.780 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.780 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.780 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256413+0000", 2020-01-16T00:14:13.781 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5293999999999998e-05 2020-01-16T00:14:13.781 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.781 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.781 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.781 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256449+0000", 2020-01-16T00:14:13.781 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001297737 2020-01-16T00:14:13.781 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.782 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.782 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:13.782 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.257746+0000", 2020-01-16T00:14:13.782 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0079866529999999998 2020-01-16T00:14:13.782 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.782 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.782 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.782 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.265733+0000", 2020-01-16T00:14:13.783 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4028000000000004e-05 2020-01-16T00:14:13.783 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.783 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.783 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.783 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.265797+0000", 2020-01-16T00:14:13.783 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011338899999999999 2020-01-16T00:14:13.783 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.784 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.784 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:13.784 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.265910+0000", 2020-01-16T00:14:13.784 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003372496 2020-01-16T00:14:13.784 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.784 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.785 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:13.785 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.269283+0000", 2020-01-16T00:14:13.785 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0053107959999999996 2020-01-16T00:14:13.785 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.785 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.786 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:13.786 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.274594+0000", 2020-01-16T00:14:13.786 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8000999999999998e-05 2020-01-16T00:14:13.786 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.786 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.786 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:13.786 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.274632+0000", 2020-01-16T00:14:13.787 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0265e-05 2020-01-16T00:14:13.787 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.787 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.787 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.787 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.274692+0000", 2020-01-16T00:14:13.787 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018509650999999998 2020-01-16T00:14:13.787 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.787 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.788 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.788 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.788 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.788 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2973 3.c 3:3025546b:::smithi12313934-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e42)", 2020-01-16T00:14:13.788 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:21.839961+0000", 2020-01-16T00:14:13.788 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 51.867331450000002, 2020-01-16T00:14:13.788 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018691670000000001, 2020-01-16T00:14:13.789 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.789 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:13.789 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:13.789 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:13.789 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:13.789 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2973 2020-01-16T00:14:13.789 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.789 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.790 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.790 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.790 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839961+0000", 2020-01-16T00:14:13.790 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.790 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.790 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.790 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.791 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839961+0000", 2020-01-16T00:14:13.791 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:13.791 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.791 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.791 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.791 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839960+0000", 2020-01-16T00:14:13.791 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4060000000000001e-06 2020-01-16T00:14:13.791 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.792 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.792 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.792 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839964+0000", 2020-01-16T00:14:13.792 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5412000000000001e-05 2020-01-16T00:14:13.792 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.792 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.792 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.792 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839979+0000", 2020-01-16T00:14:13.793 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.444e-06 2020-01-16T00:14:13.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.793 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.793 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.793 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839983+0000", 2020-01-16T00:14:13.793 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000430028 2020-01-16T00:14:13.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.793 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.794 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.794 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.840413+0000", 2020-01-16T00:14:13.794 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9270999999999997e-05 2020-01-16T00:14:13.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.796 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.796 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:13.797 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.840452+0000", 2020-01-16T00:14:13.797 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014853975 2020-01-16T00:14:13.797 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.797 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.797 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.797 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.855306+0000", 2020-01-16T00:14:13.797 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011098400000000001 2020-01-16T00:14:13.798 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.798 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.798 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:13.798 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.855417+0000", 2020-01-16T00:14:13.798 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029108060000000002 2020-01-16T00:14:13.798 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.799 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.799 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.799 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858328+0000", 2020-01-16T00:14:13.799 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000144697 2020-01-16T00:14:13.799 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.799 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.799 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.800 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858472+0000", 2020-01-16T00:14:13.800 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018011899999999999 2020-01-16T00:14:13.800 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.800 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.800 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.800 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858653+0000", 2020-01-16T00:14:13.800 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018691670000000001 2020-01-16T00:14:13.801 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.801 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.801 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.801 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.801 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.801 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3296 3.2 3:42b7f20f:::smithi12313934-42:head [setxattr _header (56) in=7b,truncate 2739607] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:14:13.801 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:30.744532+0000", 2020-01-16T00:14:13.801 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 42.962760441, 2020-01-16T00:14:13.802 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018472374, 2020-01-16T00:14:13.802 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.802 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:13.802 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:13.802 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:13.802 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:13.802 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3296 2020-01-16T00:14:13.803 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.803 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.803 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.803 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.803 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744532+0000", 2020-01-16T00:14:13.803 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.803 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.803 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.803 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.804 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744532+0000", 2020-01-16T00:14:13.804 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:13.804 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.804 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.804 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.804 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744531+0000", 2020-01-16T00:14:13.804 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7059999999999993e-06 2020-01-16T00:14:13.805 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.805 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.805 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.805 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744540+0000", 2020-01-16T00:14:13.805 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8589999999999998e-06 2020-01-16T00:14:13.805 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.806 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.806 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.806 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744544+0000", 2020-01-16T00:14:13.807 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7609999999999999e-06 2020-01-16T00:14:13.807 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.807 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.807 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.807 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744546+0000", 2020-01-16T00:14:13.807 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039933670000000003 2020-01-16T00:14:13.807 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.808 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.808 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.808 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748540+0000", 2020-01-16T00:14:13.808 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000110281 2020-01-16T00:14:13.808 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.808 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.808 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:13.808 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748650+0000", 2020-01-16T00:14:13.809 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.01160457 2020-01-16T00:14:13.809 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.809 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.809 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.809 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.760255+0000", 2020-01-16T00:14:13.810 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000124714 2020-01-16T00:14:13.810 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.810 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.810 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.810 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.760379+0000", 2020-01-16T00:14:13.810 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022099700000000001 2020-01-16T00:14:13.810 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.811 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.811 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:13.811 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.760600+0000", 2020-01-16T00:14:13.811 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00057843300000000005 2020-01-16T00:14:13.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.811 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.811 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:13.811 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.761179+0000", 2020-01-16T00:14:13.812 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00161477 2020-01-16T00:14:13.812 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.812 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.812 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:13.812 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.762794+0000", 2020-01-16T00:14:13.812 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4408999999999999e-05 2020-01-16T00:14:13.812 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.812 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.813 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:13.813 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.762858+0000", 2020-01-16T00:14:13.813 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014634199999999999 2020-01-16T00:14:13.813 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.813 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.813 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.813 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.763004+0000", 2020-01-16T00:14:13.813 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018472374 2020-01-16T00:14:13.814 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.814 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.814 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.814 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.814 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.814 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3297 3.2 3:42b7f20f:::smithi12313934-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e45)", 2020-01-16T00:14:13.814 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:30.744551+0000", 2020-01-16T00:14:13.814 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 42.962741471000001, 2020-01-16T00:14:13.815 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018660209000000001, 2020-01-16T00:14:13.815 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.815 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:13.815 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:13.815 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:13.815 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:13.815 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3297 2020-01-16T00:14:13.816 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.816 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.816 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.816 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.816 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744551+0000", 2020-01-16T00:14:13.816 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.816 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.816 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.817 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.817 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744551+0000", 2020-01-16T00:14:13.817 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:13.817 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.817 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.818 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.818 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744550+0000", 2020-01-16T00:14:13.818 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5279999999999999e-06 2020-01-16T00:14:13.818 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.818 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.818 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.818 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744554+0000", 2020-01-16T00:14:13.819 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8420000000000001e-06 2020-01-16T00:14:13.819 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.819 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.819 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.819 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744559+0000", 2020-01-16T00:14:13.819 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.965e-06 2020-01-16T00:14:13.819 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.820 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.820 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.820 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744562+0000", 2020-01-16T00:14:13.820 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041215380000000001 2020-01-16T00:14:13.820 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.821 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.821 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.821 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748684+0000", 2020-01-16T00:14:13.821 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8165999999999997e-05 2020-01-16T00:14:13.821 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.821 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.821 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:13.822 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748762+0000", 2020-01-16T00:14:13.822 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012136152000000001 2020-01-16T00:14:13.822 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.822 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.822 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.822 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.760898+0000", 2020-01-16T00:14:13.822 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000114833 2020-01-16T00:14:13.823 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.823 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.823 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:13.823 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.761013+0000", 2020-01-16T00:14:13.823 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002017015 2020-01-16T00:14:13.823 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.823 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.823 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.824 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.763030+0000", 2020-01-16T00:14:13.824 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5741000000000006e-05 2020-01-16T00:14:13.824 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.824 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.824 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.824 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.763096+0000", 2020-01-16T00:14:13.825 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011519900000000001 2020-01-16T00:14:13.825 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.825 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.825 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.825 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.763211+0000", 2020-01-16T00:14:13.826 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018660209000000001 2020-01-16T00:14:13.826 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.826 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.826 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.826 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.826 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.826 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3810 3.7 3:efab040c:::smithi12313934-30:head [write 1837406~542493] snapc 0=[] ondisk+write+known_if_redirected e50)", 2020-01-16T00:14:13.826 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:43.768525+0000", 2020-01-16T00:14:13.827 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 29.938767297999998, 2020-01-16T00:14:13.827 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018385839000000001, 2020-01-16T00:14:13.827 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.827 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:13.827 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:13.828 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:13.828 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:13.828 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3810 2020-01-16T00:14:13.828 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.828 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.828 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.828 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.828 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768525+0000", 2020-01-16T00:14:13.829 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.829 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.829 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.829 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.829 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768525+0000", 2020-01-16T00:14:13.829 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:13.829 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.830 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.830 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.830 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768523+0000", 2020-01-16T00:14:13.830 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000149512 2020-01-16T00:14:13.830 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.830 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.830 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.831 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768673+0000", 2020-01-16T00:14:13.831 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0732e-05 2020-01-16T00:14:13.831 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.831 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.831 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.831 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768684+0000", 2020-01-16T00:14:13.831 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3719999999999998e-06 2020-01-16T00:14:13.831 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.832 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.832 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.832 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768688+0000", 2020-01-16T00:14:13.832 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6513999999999994e-05 2020-01-16T00:14:13.832 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.832 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.832 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.832 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768754+0000", 2020-01-16T00:14:13.833 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00075867800000000004 2020-01-16T00:14:13.833 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.833 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.833 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:13.833 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.769513+0000", 2020-01-16T00:14:13.833 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012395692 2020-01-16T00:14:13.833 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.834 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.834 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.834 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.781909+0000", 2020-01-16T00:14:13.834 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118372 2020-01-16T00:14:13.834 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.834 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.834 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.834 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.782027+0000", 2020-01-16T00:14:13.835 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019909699999999999 2020-01-16T00:14:13.835 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.835 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.836 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,7", 2020-01-16T00:14:13.837 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.782226+0000", 2020-01-16T00:14:13.837 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0038778369999999999 2020-01-16T00:14:13.837 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.837 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.837 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:13.838 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786104+0000", 2020-01-16T00:14:13.838 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022391199999999999 2020-01-16T00:14:13.838 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.838 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.838 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:13.838 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786328+0000", 2020-01-16T00:14:13.838 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00048851299999999999 2020-01-16T00:14:13.839 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.839 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.839 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:13.839 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786817+0000", 2020-01-16T00:14:13.839 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3597999999999998e-05 2020-01-16T00:14:13.839 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.840 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.840 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:13.840 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786860+0000", 2020-01-16T00:14:13.840 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0741000000000001e-05 2020-01-16T00:14:13.840 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.840 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.840 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.840 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786911+0000", 2020-01-16T00:14:13.841 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018385839000000001 2020-01-16T00:14:13.841 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.841 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.841 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.841 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.841 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.841 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3863 3.2 3:4f8fe446:::smithi12313934-48:head [setxattr _header (56) in=7b,truncate 2151395] snapc 0=[] ondisk+write+known_if_redirected e50)", 2020-01-16T00:14:13.842 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:44.489284+0000", 2020-01-16T00:14:13.842 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 29.218008825999998, 2020-01-16T00:14:13.842 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.026323742000000001, 2020-01-16T00:14:13.842 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.842 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:13.842 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:13.842 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:13.843 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:13.843 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3863 2020-01-16T00:14:13.843 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.843 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.843 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.843 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.843 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489284+0000", 2020-01-16T00:14:13.843 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.844 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.844 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.844 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.844 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489284+0000", 2020-01-16T00:14:13.844 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:13.844 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.844 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.845 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.845 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489282+0000", 2020-01-16T00:14:13.845 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6660000000000002e-06 2020-01-16T00:14:13.845 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.845 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.845 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.845 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489289+0000", 2020-01-16T00:14:13.846 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3069999999999999e-06 2020-01-16T00:14:13.846 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.846 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.846 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.846 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489295+0000", 2020-01-16T00:14:13.846 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6090000000000003e-06 2020-01-16T00:14:13.847 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.847 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.847 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.847 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489300+0000", 2020-01-16T00:14:13.847 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002242464 2020-01-16T00:14:13.848 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.848 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.848 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.848 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491542+0000", 2020-01-16T00:14:13.848 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00059546200000000005 2020-01-16T00:14:13.848 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.849 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.849 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:13.849 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.492138+0000", 2020-01-16T00:14:13.849 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018547999999999999 2020-01-16T00:14:13.849 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.849 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.849 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.850 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.510686+0000", 2020-01-16T00:14:13.850 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018590100000000001 2020-01-16T00:14:13.850 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.850 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.850 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.850 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.510872+0000", 2020-01-16T00:14:13.850 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035115299999999999 2020-01-16T00:14:13.850 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.851 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.851 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:13.851 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.511223+0000", 2020-01-16T00:14:13.851 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003765493 2020-01-16T00:14:13.851 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.851 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.851 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:13.852 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.514988+0000", 2020-01-16T00:14:13.852 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000432022 2020-01-16T00:14:13.852 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.852 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.852 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:13.852 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515421+0000", 2020-01-16T00:14:13.852 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2874000000000002e-05 2020-01-16T00:14:13.852 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.853 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.853 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:13.853 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515473+0000", 2020-01-16T00:14:13.853 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013385 2020-01-16T00:14:13.853 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.853 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.853 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.853 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515607+0000", 2020-01-16T00:14:13.854 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.026323742000000001 2020-01-16T00:14:13.854 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.854 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.854 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.854 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.854 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.854 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3864 3.2 3:4f8fe446:::smithi12313934-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e50)", 2020-01-16T00:14:13.854 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:44.491131+0000", 2020-01-16T00:14:13.855 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 29.216161427999999, 2020-01-16T00:14:13.855 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024813806000000001, 2020-01-16T00:14:13.855 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.855 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:13.855 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:13.855 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:13.855 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:13.855 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3864 2020-01-16T00:14:13.856 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.856 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.856 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.856 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.856 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491131+0000", 2020-01-16T00:14:13.856 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.856 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.857 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.857 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.857 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491131+0000", 2020-01-16T00:14:13.857 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:13.857 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.857 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.857 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.857 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491128+0000", 2020-01-16T00:14:13.858 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2053e-05 2020-01-16T00:14:13.858 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.858 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.858 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.858 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491140+0000", 2020-01-16T00:14:13.859 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5869999999999999e-05 2020-01-16T00:14:13.859 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.859 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.859 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.859 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491156+0000", 2020-01-16T00:14:13.859 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2571e-05 2020-01-16T00:14:13.859 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.859 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.860 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.860 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491168+0000", 2020-01-16T00:14:13.860 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001031045 2020-01-16T00:14:13.860 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.860 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.860 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.860 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.492199+0000", 2020-01-16T00:14:13.860 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000170865 2020-01-16T00:14:13.861 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.861 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.861 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:13.861 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.492370+0000", 2020-01-16T00:14:13.861 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020180638000000001 2020-01-16T00:14:13.861 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.861 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.862 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.862 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.512551+0000", 2020-01-16T00:14:13.862 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012478599999999999 2020-01-16T00:14:13.862 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.862 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.862 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:13.862 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.512676+0000", 2020-01-16T00:14:13.862 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029628660000000002 2020-01-16T00:14:13.863 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.863 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.863 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.863 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515639+0000", 2020-01-16T00:14:13.863 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012250799999999999 2020-01-16T00:14:13.863 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.863 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.864 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.864 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515761+0000", 2020-01-16T00:14:13.864 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018356399999999999 2020-01-16T00:14:13.864 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.864 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.864 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.864 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515945+0000", 2020-01-16T00:14:13.864 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024813806000000001 2020-01-16T00:14:13.865 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.865 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.865 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.865 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.865 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.865 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3874 3.c 3:378f3722:::smithi12313934-50:head [write 568202~701253,stat] snapc 0=[] ondisk+write+known_if_redirected e50)", 2020-01-16T00:14:13.866 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:44.497041+0000", 2020-01-16T00:14:13.866 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 29.210251797000002, 2020-01-16T00:14:13.866 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019602525999999999, 2020-01-16T00:14:13.866 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.866 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:13.866 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:13.867 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:13.867 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:13.867 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3874 2020-01-16T00:14:13.867 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.867 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.867 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.867 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.867 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497041+0000", 2020-01-16T00:14:13.868 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.868 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.868 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.868 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.868 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497041+0000", 2020-01-16T00:14:13.868 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999962 2020-01-16T00:14:13.868 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.869 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.869 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.869 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497037+0000", 2020-01-16T00:14:13.869 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00027168899999999998 2020-01-16T00:14:13.869 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.869 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.869 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.870 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497308+0000", 2020-01-16T00:14:13.870 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4202999999999999e-05 2020-01-16T00:14:13.870 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.870 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.870 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.870 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497323+0000", 2020-01-16T00:14:13.871 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0280000000000003e-06 2020-01-16T00:14:13.871 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.871 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.871 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.871 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497330+0000", 2020-01-16T00:14:13.871 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010681352 2020-01-16T00:14:13.871 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.872 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.872 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.872 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.508011+0000", 2020-01-16T00:14:13.872 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5886999999999999e-05 2020-01-16T00:14:13.872 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.872 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.872 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.872 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.508097+0000", 2020-01-16T00:14:13.873 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023353600000000001 2020-01-16T00:14:13.873 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.873 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.873 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:13.873 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.508331+0000", 2020-01-16T00:14:13.873 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0075218339999999998 2020-01-16T00:14:13.873 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.874 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.874 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:13.874 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515852+0000", 2020-01-16T00:14:13.874 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00064178599999999998 2020-01-16T00:14:13.874 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.874 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.874 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:13.874 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.516494+0000", 2020-01-16T00:14:13.875 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6712999999999999e-05 2020-01-16T00:14:13.875 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.875 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.875 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:13.875 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.516531+0000", 2020-01-16T00:14:13.875 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112225 2020-01-16T00:14:13.875 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.875 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.876 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.876 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.516643+0000", 2020-01-16T00:14:13.876 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019602525999999999 2020-01-16T00:14:13.876 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.876 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.876 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.876 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.877 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.877 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4655 3.c 3:378f3722:::smithi12313934-50:head [write 1684786~315379] snapc 0=[] ondisk+write+known_if_redirected e57)", 2020-01-16T00:14:13.877 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:04.403811+0000", 2020-01-16T00:14:13.877 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 9.3034813120000006, 2020-01-16T00:14:13.877 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022055851000000001, 2020-01-16T00:14:13.877 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.877 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:13.878 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:13.878 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:13.878 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:13.878 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4655 2020-01-16T00:14:13.878 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.878 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.879 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.879 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.879 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.403811+0000", 2020-01-16T00:14:13.879 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.879 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.879 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.879 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.880 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.403811+0000", 2020-01-16T00:14:13.880 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:13.880 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.880 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.880 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.880 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.403809+0000", 2020-01-16T00:14:13.880 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012429100000000001 2020-01-16T00:14:13.881 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.881 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.881 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.881 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.403934+0000", 2020-01-16T00:14:13.881 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3743e-05 2020-01-16T00:14:13.881 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.881 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.881 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.882 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.403947+0000", 2020-01-16T00:14:13.882 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.267e-06 2020-01-16T00:14:13.882 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.882 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.882 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.882 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.403955+0000", 2020-01-16T00:14:13.882 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1935999999999999e-05 2020-01-16T00:14:13.883 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.883 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.883 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.883 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404006+0000", 2020-01-16T00:14:13.883 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0060147769999999998 2020-01-16T00:14:13.883 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.883 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.883 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:13.884 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.410021+0000", 2020-01-16T00:14:13.884 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012889367000000001 2020-01-16T00:14:13.884 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.884 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.884 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.884 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.422911+0000", 2020-01-16T00:14:13.884 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012536700000000001 2020-01-16T00:14:13.884 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.885 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.885 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.885 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.423036+0000", 2020-01-16T00:14:13.885 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021238999999999999 2020-01-16T00:14:13.885 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.885 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.885 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:13.886 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.423248+0000", 2020-01-16T00:14:13.886 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0024175289999999999 2020-01-16T00:14:13.886 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.886 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.886 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:13.886 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.425666+0000", 2020-01-16T00:14:13.886 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7323e-05 2020-01-16T00:14:13.886 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.887 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.887 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:13.887 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.425723+0000", 2020-01-16T00:14:13.887 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0198999999999997e-05 2020-01-16T00:14:13.887 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.887 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.887 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:13.888 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.425783+0000", 2020-01-16T00:14:13.888 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3456999999999997e-05 2020-01-16T00:14:13.888 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.888 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.888 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.888 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.425867+0000", 2020-01-16T00:14:13.888 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022055851000000001 2020-01-16T00:14:13.888 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.889 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.889 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.889 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.889 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.889 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4656 3.c 3:378f3722:::smithi12313934-50:head [setxattr _header (56) in=7b,truncate 2000165] snapc 0=[] ondisk+write+known_if_redirected e57)", 2020-01-16T00:14:13.889 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:04.404295+0000", 2020-01-16T00:14:13.890 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 9.3029971729999996, 2020-01-16T00:14:13.890 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021950859, 2020-01-16T00:14:13.890 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.890 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:13.890 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:13.890 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:13.890 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:13.890 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4656 2020-01-16T00:14:13.891 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.891 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.891 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.891 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.891 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404295+0000", 2020-01-16T00:14:13.891 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.891 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.892 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.892 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.892 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404295+0000", 2020-01-16T00:14:13.892 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:13.892 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.892 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.892 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.892 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404294+0000", 2020-01-16T00:14:13.893 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6139999999999997e-06 2020-01-16T00:14:13.893 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.893 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.893 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.893 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404301+0000", 2020-01-16T00:14:13.893 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1420000000000001e-05 2020-01-16T00:14:13.893 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.893 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.894 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.894 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404312+0000", 2020-01-16T00:14:13.894 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4840000000000003e-06 2020-01-16T00:14:13.894 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.894 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.894 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.894 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404318+0000", 2020-01-16T00:14:13.895 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0058040970000000002 2020-01-16T00:14:13.895 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.895 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.895 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.895 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.410122+0000", 2020-01-16T00:14:13.895 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000168297 2020-01-16T00:14:13.895 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.895 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.896 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:13.896 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.410290+0000", 2020-01-16T00:14:13.896 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013892204999999999 2020-01-16T00:14:13.896 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.896 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.896 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.896 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.424182+0000", 2020-01-16T00:14:13.896 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6482000000000002e-05 2020-01-16T00:14:13.897 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.897 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.897 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.897 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.424279+0000", 2020-01-16T00:14:13.897 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000130025 2020-01-16T00:14:13.897 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.897 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.897 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:13.898 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.424409+0000", 2020-01-16T00:14:13.898 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00151793 2020-01-16T00:14:13.898 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.898 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.898 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:13.898 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.425927+0000", 2020-01-16T00:14:13.898 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020657800000000001 2020-01-16T00:14:13.899 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.899 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.899 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:13.900 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.426133+0000", 2020-01-16T00:14:13.900 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3606999999999999e-05 2020-01-16T00:14:13.900 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.900 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.900 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:13.900 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.426167+0000", 2020-01-16T00:14:13.900 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9117999999999997e-05 2020-01-16T00:14:13.900 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.901 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.901 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.901 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.426246+0000", 2020-01-16T00:14:13.901 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021950859 2020-01-16T00:14:13.901 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.901 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.901 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.902 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.902 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.902 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4657 3.c 3:378f3722:::smithi12313934-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e57)", 2020-01-16T00:14:13.902 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:04.404326+0000", 2020-01-16T00:14:13.902 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 9.302966026, 2020-01-16T00:14:13.902 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022101092999999999, 2020-01-16T00:14:13.902 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.902 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:13.903 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:13.903 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:13.903 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:13.903 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4657 2020-01-16T00:14:13.903 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.903 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.903 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.904 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.904 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404326+0000", 2020-01-16T00:14:13.904 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.904 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.904 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.904 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.904 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404326+0000", 2020-01-16T00:14:13.905 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:13.905 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.905 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.905 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.905 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404326+0000", 2020-01-16T00:14:13.905 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.771e-06 2020-01-16T00:14:13.905 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.906 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.906 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.906 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404329+0000", 2020-01-16T00:14:13.906 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9909999999999999e-06 2020-01-16T00:14:13.906 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.906 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.906 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.907 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404335+0000", 2020-01-16T00:14:13.907 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8870000000000002e-06 2020-01-16T00:14:13.907 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.907 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.907 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.907 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404339+0000", 2020-01-16T00:14:13.907 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0066816460000000003 2020-01-16T00:14:13.907 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.908 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.908 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.908 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.411021+0000", 2020-01-16T00:14:13.908 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3233000000000007e-05 2020-01-16T00:14:13.908 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.908 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.908 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:13.909 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.411114+0000", 2020-01-16T00:14:13.909 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014244643 2020-01-16T00:14:13.909 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.909 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.910 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.910 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.425359+0000", 2020-01-16T00:14:13.910 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012771200000000001 2020-01-16T00:14:13.910 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.910 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.910 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:13.910 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.425487+0000", 2020-01-16T00:14:13.910 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00078975699999999998 2020-01-16T00:14:13.911 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.911 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.911 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.911 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.426276+0000", 2020-01-16T00:14:13.911 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8829999999999997e-05 2020-01-16T00:14:13.911 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.911 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.911 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.912 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.426335+0000", 2020-01-16T00:14:13.912 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2219999999999995e-05 2020-01-16T00:14:13.912 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.912 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.912 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.912 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.426427+0000", 2020-01-16T00:14:13.912 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022101092999999999 2020-01-16T00:14:13.912 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.913 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.913 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.913 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.913 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.913 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4690 3.2 3:4f8fe446:::smithi12313934-48:head [setxattr _header (56) in=7b,truncate 2067393] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-01-16T00:14:13.913 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:06.996624+0000", 2020-01-16T00:14:13.913 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 6.7106687919999999, 2020-01-16T00:14:13.914 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021459626999999998, 2020-01-16T00:14:13.914 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.914 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:13.914 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:13.914 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:13.914 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:13.914 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4690 2020-01-16T00:14:13.915 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.915 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.915 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.915 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.915 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996624+0000", 2020-01-16T00:14:13.915 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.915 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.915 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.916 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.916 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996624+0000", 2020-01-16T00:14:13.916 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:13.916 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.916 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.916 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.916 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996623+0000", 2020-01-16T00:14:13.916 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.549e-06 2020-01-16T00:14:13.917 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.917 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.917 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.917 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996627+0000", 2020-01-16T00:14:13.917 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4501999999999999e-05 2020-01-16T00:14:13.917 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.917 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.918 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.918 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996642+0000", 2020-01-16T00:14:13.918 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1139999999999997e-06 2020-01-16T00:14:13.918 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.918 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.918 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.918 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996647+0000", 2020-01-16T00:14:13.918 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001305489 2020-01-16T00:14:13.919 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.919 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.919 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.919 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.997952+0000", 2020-01-16T00:14:13.919 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036299430000000001 2020-01-16T00:14:13.919 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.920 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.920 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:13.920 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.001582+0000", 2020-01-16T00:14:13.920 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011633755000000001 2020-01-16T00:14:13.920 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.920 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.920 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.921 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.013216+0000", 2020-01-16T00:14:13.921 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021714 2020-01-16T00:14:13.921 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.921 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.921 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.921 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.013433+0000", 2020-01-16T00:14:13.921 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00050686199999999996 2020-01-16T00:14:13.921 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.922 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.922 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:13.922 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.013940+0000", 2020-01-16T00:14:13.922 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039060330000000002 2020-01-16T00:14:13.922 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.922 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.922 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:13.922 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.017846+0000", 2020-01-16T00:14:13.923 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7609e-05 2020-01-16T00:14:13.923 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.923 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.923 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:13.923 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.017864+0000", 2020-01-16T00:14:13.923 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3856999999999994e-05 2020-01-16T00:14:13.923 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.924 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.924 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:13.924 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.017937+0000", 2020-01-16T00:14:13.924 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014570800000000001 2020-01-16T00:14:13.924 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.924 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.924 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.925 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.018083+0000", 2020-01-16T00:14:13.925 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021459626999999998 2020-01-16T00:14:13.925 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.925 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.925 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.925 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.925 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.925 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4691 3.2 3:4f8fe446:::smithi12313934-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e58)", 2020-01-16T00:14:13.926 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:06.996654+0000", 2020-01-16T00:14:13.926 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 6.7106386870000003, 2020-01-16T00:14:13.926 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021903913000000001, 2020-01-16T00:14:13.926 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.926 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:13.926 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:13.927 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:13.927 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:13.927 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4691 2020-01-16T00:14:13.927 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.927 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.927 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.927 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.927 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996654+0000", 2020-01-16T00:14:13.928 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.928 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.928 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.928 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.928 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996654+0000", 2020-01-16T00:14:13.928 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:13.928 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.929 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.929 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.929 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996653+0000", 2020-01-16T00:14:13.929 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0939999999999999e-06 2020-01-16T00:14:13.929 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.929 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.929 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.929 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996656+0000", 2020-01-16T00:14:13.930 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7609999999999996e-06 2020-01-16T00:14:13.930 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.930 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.930 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.930 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996665+0000", 2020-01-16T00:14:13.931 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3589999999999999e-06 2020-01-16T00:14:13.931 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.931 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.931 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.931 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996668+0000", 2020-01-16T00:14:13.931 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0049635909999999998 2020-01-16T00:14:13.931 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.931 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.932 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.932 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.001632+0000", 2020-01-16T00:14:13.932 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9422000000000001e-05 2020-01-16T00:14:13.932 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.932 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.932 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:13.932 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.001691+0000", 2020-01-16T00:14:13.932 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013157973 2020-01-16T00:14:13.933 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.933 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.933 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.933 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.014849+0000", 2020-01-16T00:14:13.933 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00027545199999999998 2020-01-16T00:14:13.933 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.933 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.934 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:13.934 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.015125+0000", 2020-01-16T00:14:13.934 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030160159999999998 2020-01-16T00:14:13.934 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.934 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.934 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.934 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.018141+0000", 2020-01-16T00:14:13.935 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015583 2020-01-16T00:14:13.935 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.935 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.935 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.935 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.018297+0000", 2020-01-16T00:14:13.935 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00026096000000000002 2020-01-16T00:14:13.935 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.935 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.936 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.936 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.018558+0000", 2020-01-16T00:14:13.936 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021903913000000001 2020-01-16T00:14:13.936 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.936 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.936 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.936 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.936 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.937 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4703 3.2 3:42b7f20f:::smithi12313934-42:head [write 1635678~465329] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-01-16T00:14:13.937 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.020171+0000", 2020-01-16T00:14:13.937 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 6.687121479, 2020-01-16T00:14:13.937 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020016459, 2020-01-16T00:14:13.937 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.937 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:13.937 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:13.937 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:13.938 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:13.938 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4703 2020-01-16T00:14:13.938 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.938 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.938 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.938 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.938 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020171+0000", 2020-01-16T00:14:13.939 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.939 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.939 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.939 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.939 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020171+0000", 2020-01-16T00:14:13.939 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:13.939 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.939 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.940 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.940 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020169+0000", 2020-01-16T00:14:13.940 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013985399999999999 2020-01-16T00:14:13.940 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.941 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.941 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.941 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020309+0000", 2020-01-16T00:14:13.941 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9029999999999999e-06 2020-01-16T00:14:13.941 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.941 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.941 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.941 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020319+0000", 2020-01-16T00:14:13.942 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9589999999999997e-06 2020-01-16T00:14:13.942 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.942 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.942 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.942 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020323+0000", 2020-01-16T00:14:13.942 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8680000000000002e-05 2020-01-16T00:14:13.942 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.942 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.943 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.943 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020362+0000", 2020-01-16T00:14:13.943 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029188460000000001 2020-01-16T00:14:13.943 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.943 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.943 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:13.943 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023281+0000", 2020-01-16T00:14:13.943 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011810522 2020-01-16T00:14:13.944 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.944 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.944 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.944 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.035091+0000", 2020-01-16T00:14:13.944 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000126148 2020-01-16T00:14:13.944 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.944 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.944 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.945 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.035217+0000", 2020-01-16T00:14:13.945 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019636799999999999 2020-01-16T00:14:13.945 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.945 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.945 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:13.945 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.035414+0000", 2020-01-16T00:14:13.945 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003550919 2020-01-16T00:14:13.945 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.946 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.946 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:13.946 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.038965+0000", 2020-01-16T00:14:13.946 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001007441 2020-01-16T00:14:13.946 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.946 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.946 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:13.946 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.039972+0000", 2020-01-16T00:14:13.947 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1954999999999998e-05 2020-01-16T00:14:13.947 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.947 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.947 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:13.947 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.040044+0000", 2020-01-16T00:14:13.947 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014325899999999999 2020-01-16T00:14:13.947 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.948 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.948 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.948 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.040187+0000", 2020-01-16T00:14:13.948 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020016459 2020-01-16T00:14:13.948 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.948 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.948 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.949 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.949 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.949 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4704 3.2 3:42b7f20f:::smithi12313934-42:head [setxattr _header (56) in=7b,truncate 2101007] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-01-16T00:14:13.949 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.020335+0000", 2020-01-16T00:14:13.949 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 6.6869576679999998, 2020-01-16T00:14:13.949 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023082354999999999, 2020-01-16T00:14:13.950 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.950 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:13.950 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:13.950 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:13.950 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:13.950 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4704 2020-01-16T00:14:13.950 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.951 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.951 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.951 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.951 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020335+0000", 2020-01-16T00:14:13.951 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.951 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.951 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.952 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.952 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020335+0000", 2020-01-16T00:14:13.952 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:13.952 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.952 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.952 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.952 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020334+0000", 2020-01-16T00:14:13.952 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7780000000000002e-06 2020-01-16T00:14:13.953 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.953 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.953 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020341+0000", 2020-01-16T00:14:13.953 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2039999999999996e-06 2020-01-16T00:14:13.953 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.953 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.954 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020346+0000", 2020-01-16T00:14:13.954 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8800000000000008e-06 2020-01-16T00:14:13.954 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.954 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.954 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.954 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020354+0000", 2020-01-16T00:14:13.954 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029530490000000001 2020-01-16T00:14:13.954 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.955 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.955 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.955 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023307+0000", 2020-01-16T00:14:13.955 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8681000000000003e-05 2020-01-16T00:14:13.955 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.955 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.955 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:13.955 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023365+0000", 2020-01-16T00:14:13.956 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013386927999999999 2020-01-16T00:14:13.956 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.956 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.956 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.956 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.036752+0000", 2020-01-16T00:14:13.956 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8906000000000007e-05 2020-01-16T00:14:13.956 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.957 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.957 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.957 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.036831+0000", 2020-01-16T00:14:13.957 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000142922 2020-01-16T00:14:13.957 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.957 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.957 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:13.957 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.036974+0000", 2020-01-16T00:14:13.958 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0038357280000000001 2020-01-16T00:14:13.958 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.958 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.958 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:13.958 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.040810+0000", 2020-01-16T00:14:13.958 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0024973949999999999 2020-01-16T00:14:13.958 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.958 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.959 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:13.959 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.043307+0000", 2020-01-16T00:14:13.959 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8513000000000002e-05 2020-01-16T00:14:13.959 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.959 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.959 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:13.959 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.043346+0000", 2020-01-16T00:14:13.960 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.127e-05 2020-01-16T00:14:13.960 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.960 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.960 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.960 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.043417+0000", 2020-01-16T00:14:13.960 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023082354999999999 2020-01-16T00:14:13.960 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.961 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.961 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.961 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.961 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.961 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4705 3.2 3:42b7f20f:::smithi12313934-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e58)", 2020-01-16T00:14:13.961 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.020362+0000", 2020-01-16T00:14:13.961 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 6.6869307329999996, 2020-01-16T00:14:13.962 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023262267999999999, 2020-01-16T00:14:13.962 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.962 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:13.962 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:13.962 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:13.962 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:13.962 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4705 2020-01-16T00:14:13.962 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.963 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.963 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.963 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.963 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020362+0000", 2020-01-16T00:14:13.963 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.963 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.963 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.963 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.964 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020362+0000", 2020-01-16T00:14:13.964 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:13.964 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.964 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.964 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.964 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020361+0000", 2020-01-16T00:14:13.964 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.365e-06 2020-01-16T00:14:13.965 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.965 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.965 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.965 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020364+0000", 2020-01-16T00:14:13.965 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5649999999999998e-06 2020-01-16T00:14:13.965 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.965 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.966 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.966 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020371+0000", 2020-01-16T00:14:13.966 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6349999999999999e-06 2020-01-16T00:14:13.966 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.966 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.966 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.966 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020375+0000", 2020-01-16T00:14:13.967 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030174920000000001 2020-01-16T00:14:13.967 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.967 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.967 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.967 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023392+0000", 2020-01-16T00:14:13.967 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4997000000000003e-05 2020-01-16T00:14:13.968 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.968 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.968 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:13.968 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023437+0000", 2020-01-16T00:14:13.968 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015293378999999999 2020-01-16T00:14:13.968 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.968 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.969 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.969 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.038730+0000", 2020-01-16T00:14:13.969 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000124206 2020-01-16T00:14:13.969 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.969 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.969 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:13.969 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.038855+0000", 2020-01-16T00:14:13.969 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0045951489999999998 2020-01-16T00:14:13.970 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.970 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.970 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.970 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.043450+0000", 2020-01-16T00:14:13.970 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1363000000000001e-05 2020-01-16T00:14:13.970 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.970 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.971 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.971 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.043521+0000", 2020-01-16T00:14:13.971 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000102737 2020-01-16T00:14:13.971 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.972 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.972 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.972 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.043624+0000", 2020-01-16T00:14:13.972 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023262267999999999 2020-01-16T00:14:13.972 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.972 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.972 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.972 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.973 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.973 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4869 3.c 3:3025546b:::smithi12313934-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-01-16T00:14:13.973 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:11.931564+0000", 2020-01-16T00:14:13.973 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 1.77572785, 2020-01-16T00:14:13.973 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018620945, 2020-01-16T00:14:13.973 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:13.973 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:13.974 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:13.974 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:13.974 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:13.974 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4869 2020-01-16T00:14:13.974 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.974 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:13.975 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.975 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:13.975 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.931564+0000", 2020-01-16T00:14:13.975 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:13.975 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.975 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.975 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:13.975 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.931564+0000", 2020-01-16T00:14:13.976 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:13.976 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.976 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.976 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:13.976 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.931563+0000", 2020-01-16T00:14:13.976 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8030000000000002e-06 2020-01-16T00:14:13.976 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.976 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.977 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:13.977 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.931569+0000", 2020-01-16T00:14:13.977 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3370000000000006e-06 2020-01-16T00:14:13.977 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.977 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.977 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:13.977 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.931579+0000", 2020-01-16T00:14:13.978 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9569999999999999e-06 2020-01-16T00:14:13.978 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.978 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.978 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:13.978 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.931585+0000", 2020-01-16T00:14:13.978 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021810430000000001 2020-01-16T00:14:13.978 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.978 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.979 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.979 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.933766+0000", 2020-01-16T00:14:13.979 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2802000000000001e-05 2020-01-16T00:14:13.979 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.979 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.979 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:13.979 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.933808+0000", 2020-01-16T00:14:13.980 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0099913469999999994 2020-01-16T00:14:13.980 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.980 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.980 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.980 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.943800+0000", 2020-01-16T00:14:13.980 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013257000000000001 2020-01-16T00:14:13.980 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.981 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.981 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:13.981 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.943932+0000", 2020-01-16T00:14:13.981 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0058405799999999997 2020-01-16T00:14:13.981 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.982 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.982 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:13.982 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.949773+0000", 2020-01-16T00:14:13.982 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000158493 2020-01-16T00:14:13.982 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.982 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.982 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:13.983 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.949931+0000", 2020-01-16T00:14:13.983 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000254083 2020-01-16T00:14:13.983 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:13.983 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:13.983 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:13.983 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.950185+0000", 2020-01-16T00:14:13.983 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018620945 2020-01-16T00:14:13.983 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.984 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.984 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.984 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:13.984 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:13.984 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:13.988 INFO:tasks.rados.rados.0.smithi123.stdout:1748: finishing write tid 1 to smithi12313934-46 2020-01-16T00:14:13.988 INFO:tasks.rados.rados.0.smithi123.stdout:1748: finishing write tid 2 to smithi12313934-46 2020-01-16T00:14:13.988 INFO:tasks.rados.rados.0.smithi123.stdout:1749: finishing write tid 1 to smithi12313934-9 2020-01-16T00:14:13.988 INFO:tasks.rados.rados.0.smithi123.stdout:1745: done (15 left) 2020-01-16T00:14:13.988 INFO:tasks.rados.rados.0.smithi123.stdout:1746: done (14 left) 2020-01-16T00:14:13.989 INFO:tasks.rados.rados.0.smithi123.stdout:1747: done (13 left) 2020-01-16T00:14:13.989 INFO:tasks.rados.rados.0.smithi123.stdout:1761: read oid 8 snap -1 2020-01-16T00:14:13.989 INFO:tasks.rados.rados.0.smithi123.stdout:1761: expect (ObjNum 565 snap 0 seq_num 565) 2020-01-16T00:14:13.989 INFO:tasks.rados.rados.0.smithi123.stdout:1762: delete oid 12 current snap is 0 2020-01-16T00:14:13.989 INFO:tasks.rados.rados.0.smithi123.stdout:1748: finishing write tid 4 to smithi12313934-46 2020-01-16T00:14:13.989 INFO:tasks.rados.rados.0.smithi123.stdout:1748: finishing write tid 5 to smithi12313934-46 2020-01-16T00:14:13.989 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 46 v 333 (ObjNum 632 snap 0 seq_num 632) dirty exists 2020-01-16T00:14:13.989 INFO:tasks.rados.rados.0.smithi123.stdout:1748: left oid 46 (ObjNum 632 snap 0 seq_num 632) 2020-01-16T00:14:13.990 INFO:tasks.rados.rados.0.smithi123.stdout:1749: finishing write tid 2 to smithi12313934-9 2020-01-16T00:14:13.990 INFO:tasks.rados.rados.0.smithi123.stdout:1749: finishing write tid 4 to smithi12313934-9 2020-01-16T00:14:13.990 INFO:tasks.rados.rados.0.smithi123.stdout:1749: finishing write tid 5 to smithi12313934-9 2020-01-16T00:14:13.990 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 9 v 173 (ObjNum 633 snap 0 seq_num 633) dirty exists 2020-01-16T00:14:13.990 INFO:tasks.rados.rados.0.smithi123.stdout:1749: left oid 9 (ObjNum 633 snap 0 seq_num 633) 2020-01-16T00:14:13.990 INFO:tasks.rados.rados.0.smithi123.stdout:1750: finishing write tid 1 to smithi12313934-18 2020-01-16T00:14:13.990 INFO:tasks.rados.rados.0.smithi123.stdout:1750: finishing write tid 2 to smithi12313934-18 2020-01-16T00:14:13.990 INFO:tasks.rados.rados.0.smithi123.stdout:1750: finishing write tid 4 to smithi12313934-18 2020-01-16T00:14:13.991 INFO:tasks.rados.rados.0.smithi123.stdout:1750: finishing write tid 5 to smithi12313934-18 2020-01-16T00:14:13.991 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 18 v 336 (ObjNum 634 snap 0 seq_num 634) dirty exists 2020-01-16T00:14:13.991 INFO:tasks.rados.rados.0.smithi123.stdout:1750: left oid 18 (ObjNum 634 snap 0 seq_num 634) 2020-01-16T00:14:13.991 INFO:tasks.rados.rados.0.smithi123.stdout:1751: finishing write tid 1 to smithi12313934-34 2020-01-16T00:14:13.991 INFO:tasks.rados.rados.0.smithi123.stdout:1751: finishing write tid 2 to smithi12313934-34 2020-01-16T00:14:13.991 INFO:tasks.rados.rados.0.smithi123.stdout:1751: finishing write tid 4 to smithi12313934-34 2020-01-16T00:14:13.991 INFO:tasks.rados.rados.0.smithi123.stdout:1751: finishing write tid 5 to smithi12313934-34 2020-01-16T00:14:13.992 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 34 v 89 (ObjNum 635 snap 0 seq_num 635) dirty exists 2020-01-16T00:14:13.992 INFO:tasks.rados.rados.0.smithi123.stdout:1751: left oid 34 (ObjNum 635 snap 0 seq_num 635) 2020-01-16T00:14:13.992 INFO:tasks.rados.rados.0.smithi123.stdout:1752: finishing write tid 1 to smithi12313934-2 2020-01-16T00:14:13.992 INFO:tasks.rados.rados.0.smithi123.stdout:1753: expect (ObjNum 560 snap 0 seq_num 560) 2020-01-16T00:14:13.992 INFO:tasks.rados.rados.0.smithi123.stdout:1754: finishing write tid 1 to smithi12313934-44 2020-01-16T00:14:13.992 INFO:tasks.rados.rados.0.smithi123.stdout:1752: finishing write tid 2 to smithi12313934-2 2020-01-16T00:14:13.992 INFO:tasks.rados.rados.0.smithi123.stdout:1752: finishing write tid 4 to smithi12313934-2 2020-01-16T00:14:13.993 INFO:tasks.rados.rados.0.smithi123.stdout:1752: finishing write tid 5 to smithi12313934-2 2020-01-16T00:14:13.993 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 2 v 244 (ObjNum 636 snap 0 seq_num 636) dirty exists 2020-01-16T00:14:13.993 INFO:tasks.rados.rados.0.smithi123.stdout:1752: left oid 2 (ObjNum 636 snap 0 seq_num 636) 2020-01-16T00:14:13.993 INFO:tasks.rados.rados.0.smithi123.stdout:1756: expect (ObjNum 556 snap 0 seq_num 556) 2020-01-16T00:14:13.993 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:14:13.993 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:14:13.993 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:14:13.994 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:14:13.994 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:14:13.994 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:14:13.994 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:14:13.994 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-01-16T00:14:13.994 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-01-16T00:14:13.994 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-01-16T00:14:13.995 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-01-16T00:14:13.995 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-01-16T00:14:13.995 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENXIO: problem getting command descriptions from osd.5 2020-01-16T00:14:13.995 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: problem getting command descriptions from osd.5 2020-01-16T00:14:13.995 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-01-16T00:14:13.995 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-01-16T00:14:13.995 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-01-16T00:14:13.996 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-01-16T00:14:13.996 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:13.984+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:13.996 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-01-16T00:14:14.086 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:14.084+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:14.101 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:14.110 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-01-16T00:14:14.188 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:14.186+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:14.214 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:14.222 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-01-16T00:14:14.227 INFO:tasks.rados.rados.0.smithi123.stdout:1755: expect (ObjNum 607 snap 0 seq_num 607) 2020-01-16T00:14:14.289 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:14.284+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:14.329 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:14.333 INFO:tasks.rados.rados.0.smithi123.stdout:1754: finishing write tid 2 to smithi12313934-44 2020-01-16T00:14:14.333 INFO:tasks.rados.rados.0.smithi123.stdout:1754: finishing write tid 4 to smithi12313934-44 2020-01-16T00:14:14.333 INFO:tasks.rados.rados.0.smithi123.stdout:1754: finishing write tid 5 to smithi12313934-44 2020-01-16T00:14:14.334 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 44 v 277 (ObjNum 637 snap 0 seq_num 637) dirty exists 2020-01-16T00:14:14.334 INFO:tasks.rados.rados.0.smithi123.stdout:1754: left oid 44 (ObjNum 637 snap 0 seq_num 637) 2020-01-16T00:14:14.334 INFO:tasks.rados.rados.0.smithi123.stdout:1757: finishing write tid 1 to smithi12313934-40 2020-01-16T00:14:14.334 INFO:tasks.rados.rados.0.smithi123.stdout:1748: done (14 left) 2020-01-16T00:14:14.334 INFO:tasks.rados.rados.0.smithi123.stdout:1749: done (13 left) 2020-01-16T00:14:14.334 INFO:tasks.rados.rados.0.smithi123.stdout:1750: done (12 left) 2020-01-16T00:14:14.334 INFO:tasks.rados.rados.0.smithi123.stdout:1751: done (11 left) 2020-01-16T00:14:14.335 INFO:tasks.rados.rados.0.smithi123.stdout:1752: done (10 left) 2020-01-16T00:14:14.335 INFO:tasks.rados.rados.0.smithi123.stdout:1753: done (9 left) 2020-01-16T00:14:14.335 INFO:tasks.rados.rados.0.smithi123.stdout:1754: done (8 left) 2020-01-16T00:14:14.335 INFO:tasks.rados.rados.0.smithi123.stdout:1755: done (7 left) 2020-01-16T00:14:14.335 INFO:tasks.rados.rados.0.smithi123.stdout:1756: done (6 left) 2020-01-16T00:14:14.335 INFO:tasks.rados.rados.0.smithi123.stdout:1762: done (5 left) 2020-01-16T00:14:14.335 INFO:tasks.rados.rados.0.smithi123.stdout:1763: write (excl) oid 5 current snap is 0 2020-01-16T00:14:14.336 INFO:tasks.rados.rados.0.smithi123.stdout:1763: seq_num 641 ranges {773287=408261,1793193=576207} 2020-01-16T00:14:14.343 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-01-16T00:14:14.343 INFO:tasks.rados.rados.0.smithi123.stdout:1763: writing smithi12313934-5 from 773287 to 1181548 tid 1 2020-01-16T00:14:14.347 INFO:tasks.rados.rados.0.smithi123.stdout:1763: writing smithi12313934-5 from 1793193 to 2369400 tid 2 2020-01-16T00:14:14.347 INFO:tasks.rados.rados.0.smithi123.stdout:1764: read oid 17 snap -1 2020-01-16T00:14:14.347 INFO:tasks.rados.rados.0.smithi123.stdout:1764: expect (ObjNum 627 snap 0 seq_num 627) 2020-01-16T00:14:14.348 INFO:tasks.rados.rados.0.smithi123.stdout:1765: read oid 28 snap -1 2020-01-16T00:14:14.348 INFO:tasks.rados.rados.0.smithi123.stdout:1765: expect (ObjNum 620 snap 0 seq_num 620) 2020-01-16T00:14:14.348 INFO:tasks.rados.rados.0.smithi123.stdout:1766: delete oid 4 current snap is 0 2020-01-16T00:14:14.348 INFO:tasks.rados.rados.0.smithi123.stdout:1757: finishing write tid 2 to smithi12313934-40 2020-01-16T00:14:14.348 INFO:tasks.rados.rados.0.smithi123.stdout:1757: finishing write tid 3 to smithi12313934-40 2020-01-16T00:14:14.348 INFO:tasks.rados.rados.0.smithi123.stdout:1757: finishing write tid 5 to smithi12313934-40 2020-01-16T00:14:14.348 INFO:tasks.rados.rados.0.smithi123.stdout:1757: finishing write tid 6 to smithi12313934-40 2020-01-16T00:14:14.348 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 40 v 281 (ObjNum 638 snap 0 seq_num 638) dirty exists 2020-01-16T00:14:14.349 INFO:tasks.rados.rados.0.smithi123.stdout:1757: left oid 40 (ObjNum 638 snap 0 seq_num 638) 2020-01-16T00:14:14.349 INFO:tasks.rados.rados.0.smithi123.stdout:1758: expect (ObjNum 618 snap 0 seq_num 618) 2020-01-16T00:14:14.389 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:14.386+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:14.454 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:14.463 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-01-16T00:14:14.490 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:14.486+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:14.547 INFO:tasks.rados.rados.0.smithi123.stdout:1759: finishing write tid 1 to smithi12313934-45 2020-01-16T00:14:14.547 INFO:tasks.rados.rados.0.smithi123.stdout:1759: finishing write tid 2 to smithi12313934-45 2020-01-16T00:14:14.547 INFO:tasks.rados.rados.0.smithi123.stdout:1760: expect (ObjNum 595 snap 0 seq_num 595) 2020-01-16T00:14:14.571 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:14.580 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-01-16T00:14:14.591 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:14.590+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:14.667 INFO:tasks.rados.rados.0.smithi123.stdout:1759: finishing write tid 4 to smithi12313934-45 2020-01-16T00:14:14.667 INFO:tasks.rados.rados.0.smithi123.stdout:1759: finishing write tid 5 to smithi12313934-45 2020-01-16T00:14:14.667 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 45 v 179 (ObjNum 639 snap 0 seq_num 639) dirty exists 2020-01-16T00:14:14.668 INFO:tasks.rados.rados.0.smithi123.stdout:1759: left oid 45 (ObjNum 639 snap 0 seq_num 639) 2020-01-16T00:14:14.668 INFO:tasks.rados.rados.0.smithi123.stdout:1761: expect (ObjNum 565 snap 0 seq_num 565) 2020-01-16T00:14:14.685 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:14.691 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:14.688+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:14.694 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-01-16T00:14:14.792 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:14.788+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:14.845 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:14.858 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-01-16T00:14:14.860 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2020-01-16T00:14:14.902 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:14.900+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:14.931 INFO:tasks.rados.rados.0.smithi123.stdout:1763: finishing write tid 1 to smithi12313934-5 2020-01-16T00:14:14.931 INFO:tasks.rados.rados.0.smithi123.stdout:1765: expect (ObjNum 620 snap 0 seq_num 620) 2020-01-16T00:14:14.994 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:14.990+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:15.010 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:15.020 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-01-16T00:14:15.061 INFO:tasks.rados.rados.0.smithi123.stdout:1764: expect (ObjNum 627 snap 0 seq_num 627) 2020-01-16T00:14:15.095 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:15.092+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:15.129 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:15.139 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-01-16T00:14:15.186 INFO:tasks.rados.rados.0.smithi123.stdout:1763: finishing write tid 2 to smithi12313934-5 2020-01-16T00:14:15.186 INFO:tasks.rados.rados.0.smithi123.stdout:1763: finishing write tid 4 to smithi12313934-5 2020-01-16T00:14:15.186 INFO:tasks.rados.rados.0.smithi123.stdout:1763: finishing write tid 5 to smithi12313934-5 2020-01-16T00:14:15.186 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 5 v 277 (ObjNum 641 snap 0 seq_num 641) dirty exists 2020-01-16T00:14:15.186 INFO:tasks.rados.rados.0.smithi123.stdout:1763: left oid 5 (ObjNum 641 snap 0 seq_num 641) 2020-01-16T00:14:15.187 INFO:tasks.rados.rados.0.smithi123.stdout:1757: done (8 left) 2020-01-16T00:14:15.187 INFO:tasks.rados.rados.0.smithi123.stdout:1758: done (7 left) 2020-01-16T00:14:15.187 INFO:tasks.rados.rados.0.smithi123.stdout:1759: done (6 left) 2020-01-16T00:14:15.187 INFO:tasks.rados.rados.0.smithi123.stdout:1760: done (5 left) 2020-01-16T00:14:15.187 INFO:tasks.rados.rados.0.smithi123.stdout:1761: done (4 left) 2020-01-16T00:14:15.187 INFO:tasks.rados.rados.0.smithi123.stdout:1763: done (3 left) 2020-01-16T00:14:15.187 INFO:tasks.rados.rados.0.smithi123.stdout:1764: done (2 left) 2020-01-16T00:14:15.188 INFO:tasks.rados.rados.0.smithi123.stdout:1765: done (1 left) 2020-01-16T00:14:15.188 INFO:tasks.rados.rados.0.smithi123.stdout:1766: done (0 left) 2020-01-16T00:14:15.188 INFO:tasks.rados.rados.0.smithi123.stdout:1767: write oid 42 current snap is 0 2020-01-16T00:14:15.188 INFO:tasks.rados.rados.0.smithi123.stdout:1767: seq_num 643 ranges {406901=558759,1393804=579876,2608396=198505} 2020-01-16T00:14:15.195 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:15.194+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:15.197 INFO:tasks.rados.rados.0.smithi123.stdout:1767: writing smithi12313934-42 from 406901 to 965660 tid 1 2020-01-16T00:14:15.204 INFO:tasks.rados.rados.0.smithi123.stdout:1767: writing smithi12313934-42 from 1393804 to 1973680 tid 2 2020-01-16T00:14:15.210 INFO:tasks.rados.rados.0.smithi123.stdout:1767: writing smithi12313934-42 from 2608396 to 2806901 tid 3 2020-01-16T00:14:15.210 INFO:tasks.rados.rados.0.smithi123.stdout:1768: write oid 19 current snap is 0 2020-01-16T00:14:15.210 INFO:tasks.rados.rados.0.smithi123.stdout:1768: seq_num 644 ranges {423708=634008,1852286=439887,2752172=71536} 2020-01-16T00:14:15.216 INFO:tasks.rados.rados.0.smithi123.stdout:1768: writing smithi12313934-19 from 423708 to 1057716 tid 1 2020-01-16T00:14:15.223 INFO:tasks.rados.rados.0.smithi123.stdout:1768: writing smithi12313934-19 from 1852286 to 2292173 tid 2 2020-01-16T00:14:15.227 INFO:tasks.rados.rados.0.smithi123.stdout:1768: writing smithi12313934-19 from 2752172 to 2823708 tid 3 2020-01-16T00:14:15.227 INFO:tasks.rados.rados.0.smithi123.stdout:1769: write (excl) oid 49 current snap is 0 2020-01-16T00:14:15.227 INFO:tasks.rados.rados.0.smithi123.stdout:1769: seq_num 645 ranges {440515=709257,1910768=583545} 2020-01-16T00:14:15.233 INFO:tasks.rados.rados.0.smithi123.stdout:1769: writing smithi12313934-49 from 440515 to 1149772 tid 1 2020-01-16T00:14:15.244 INFO:tasks.rados.rados.0.smithi123.stdout:1769: writing smithi12313934-49 from 1910768 to 2494313 tid 2 2020-01-16T00:14:15.244 INFO:tasks.rados.rados.0.smithi123.stdout:1770: read oid 5 snap -1 2020-01-16T00:14:15.244 INFO:tasks.rados.rados.0.smithi123.stdout:1770: expect (ObjNum 641 snap 0 seq_num 641) 2020-01-16T00:14:15.244 INFO:tasks.rados.rados.0.smithi123.stdout:1771: read oid 34 snap -1 2020-01-16T00:14:15.244 INFO:tasks.rados.rados.0.smithi123.stdout:1771: expect (ObjNum 635 snap 0 seq_num 635) 2020-01-16T00:14:15.245 INFO:tasks.rados.rados.0.smithi123.stdout:1772: write (excl) oid 44 current snap is 0 2020-01-16T00:14:15.245 INFO:tasks.rados.rados.0.smithi123.stdout:1772: seq_num 646 ranges {457322=784506,1969250=443556} 2020-01-16T00:14:15.250 INFO:tasks.rados.rados.0.smithi123.stdout:1772: writing smithi12313934-44 from 457322 to 1241828 tid 1 2020-01-16T00:14:15.255 INFO:tasks.rados.rados.0.smithi123.stdout:1772: writing smithi12313934-44 from 1969250 to 2412806 tid 2 2020-01-16T00:14:15.255 INFO:tasks.rados.rados.0.smithi123.stdout:1773: read oid 20 snap -1 2020-01-16T00:14:15.255 INFO:tasks.rados.rados.0.smithi123.stdout:1773: expect deleted 2020-01-16T00:14:15.255 INFO:tasks.rados.rados.0.smithi123.stdout:1774: delete oid 32 current snap is 0 2020-01-16T00:14:15.255 INFO:tasks.rados.rados.0.smithi123.stdout:1767: finishing write tid 1 to smithi12313934-42 2020-01-16T00:14:15.256 INFO:tasks.rados.rados.0.smithi123.stdout:1767: finishing write tid 2 to smithi12313934-42 2020-01-16T00:14:15.256 INFO:tasks.rados.rados.0.smithi123.stdout:1767: finishing write tid 3 to smithi12313934-42 2020-01-16T00:14:15.256 INFO:tasks.rados.rados.0.smithi123.stdout:1767: finishing write tid 5 to smithi12313934-42 2020-01-16T00:14:15.256 INFO:tasks.rados.rados.0.smithi123.stdout:1767: finishing write tid 6 to smithi12313934-42 2020-01-16T00:14:15.256 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 42 v 213 (ObjNum 643 snap 0 seq_num 643) dirty exists 2020-01-16T00:14:15.256 INFO:tasks.rados.rados.0.smithi123.stdout:1767: left oid 42 (ObjNum 643 snap 0 seq_num 643) 2020-01-16T00:14:15.256 INFO:tasks.rados.rados.0.smithi123.stdout:1768: finishing write tid 1 to smithi12313934-19 2020-01-16T00:14:15.256 INFO:tasks.rados.rados.0.smithi123.stdout:1768: finishing write tid 2 to smithi12313934-19 2020-01-16T00:14:15.257 INFO:tasks.rados.rados.0.smithi123.stdout:1768: finishing write tid 3 to smithi12313934-19 2020-01-16T00:14:15.257 INFO:tasks.rados.rados.0.smithi123.stdout:1768: finishing write tid 5 to smithi12313934-19 2020-01-16T00:14:15.257 INFO:tasks.rados.rados.0.smithi123.stdout:1768: finishing write tid 6 to smithi12313934-19 2020-01-16T00:14:15.257 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 19 v 248 (ObjNum 644 snap 0 seq_num 644) dirty exists 2020-01-16T00:14:15.257 INFO:tasks.rados.rados.0.smithi123.stdout:1768: left oid 19 (ObjNum 644 snap 0 seq_num 644) 2020-01-16T00:14:15.257 INFO:tasks.rados.rados.0.smithi123.stdout:1769: finishing write tid 1 to smithi12313934-49 2020-01-16T00:14:15.257 INFO:tasks.rados.rados.0.smithi123.stdout:1769: finishing write tid 2 to smithi12313934-49 2020-01-16T00:14:15.258 INFO:tasks.rados.rados.0.smithi123.stdout:1769: finishing write tid 4 to smithi12313934-49 2020-01-16T00:14:15.258 INFO:tasks.rados.rados.0.smithi123.stdout:1769: finishing write tid 5 to smithi12313934-49 2020-01-16T00:14:15.258 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 49 v 182 (ObjNum 645 snap 0 seq_num 645) dirty exists 2020-01-16T00:14:15.258 INFO:tasks.rados.rados.0.smithi123.stdout:1769: left oid 49 (ObjNum 645 snap 0 seq_num 645) 2020-01-16T00:14:15.259 INFO:tasks.rados.rados.0.smithi123.stdout:1770: expect (ObjNum 641 snap 0 seq_num 641) 2020-01-16T00:14:15.296 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:15.292+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:15.298 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:15.307 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-01-16T00:14:15.397 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:15.394+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:15.401 INFO:tasks.rados.rados.0.smithi123.stdout:1772: finishing write tid 1 to smithi12313934-44 2020-01-16T00:14:15.401 INFO:tasks.rados.rados.0.smithi123.stdout:1771: expect (ObjNum 635 snap 0 seq_num 635) 2020-01-16T00:14:15.456 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:15.466 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-01-16T00:14:15.497 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:15.494+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:15.560 INFO:tasks.rados.rados.0.smithi123.stdout:1772: finishing write tid 2 to smithi12313934-44 2020-01-16T00:14:15.560 INFO:tasks.rados.rados.0.smithi123.stdout:1772: finishing write tid 4 to smithi12313934-44 2020-01-16T00:14:15.560 INFO:tasks.rados.rados.0.smithi123.stdout:1772: finishing write tid 5 to smithi12313934-44 2020-01-16T00:14:15.560 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 44 v 284 (ObjNum 646 snap 0 seq_num 646) dirty exists 2020-01-16T00:14:15.561 INFO:tasks.rados.rados.0.smithi123.stdout:1772: left oid 44 (ObjNum 646 snap 0 seq_num 646) 2020-01-16T00:14:15.561 INFO:tasks.rados.rados.0.smithi123.stdout:1767: done (7 left) 2020-01-16T00:14:15.561 INFO:tasks.rados.rados.0.smithi123.stdout:1768: done (6 left) 2020-01-16T00:14:15.561 INFO:tasks.rados.rados.0.smithi123.stdout:1769: done (5 left) 2020-01-16T00:14:15.561 INFO:tasks.rados.rados.0.smithi123.stdout:1770: done (4 left) 2020-01-16T00:14:15.561 INFO:tasks.rados.rados.0.smithi123.stdout:1771: done (3 left) 2020-01-16T00:14:15.561 INFO:tasks.rados.rados.0.smithi123.stdout:1772: done (2 left) 2020-01-16T00:14:15.562 INFO:tasks.rados.rados.0.smithi123.stdout:1773: done (1 left) 2020-01-16T00:14:15.562 INFO:tasks.rados.rados.0.smithi123.stdout:1774: done (0 left) 2020-01-16T00:14:15.562 INFO:tasks.rados.rados.0.smithi123.stdout:1775: delete oid 17 current snap is 0 2020-01-16T00:14:15.564 INFO:tasks.rados.rados.0.smithi123.stdout:1775: done (0 left) 2020-01-16T00:14:15.564 INFO:tasks.rados.rados.0.smithi123.stdout:1776: read oid 1 snap -1 2020-01-16T00:14:15.564 INFO:tasks.rados.rados.0.smithi123.stdout:1776: expect (ObjNum 618 snap 0 seq_num 618) 2020-01-16T00:14:15.564 INFO:tasks.rados.rados.0.smithi123.stdout:1777: write oid 28 current snap is 0 2020-01-16T00:14:15.564 INFO:tasks.rados.rados.0.smithi123.stdout:1777: seq_num 649 ranges {507743=726606,1744696=590883,2772934=134809} 2020-01-16T00:14:15.575 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:15.578 INFO:tasks.rados.rados.0.smithi123.stdout:1777: writing smithi12313934-28 from 507743 to 1234349 tid 1 2020-01-16T00:14:15.585 INFO:tasks.rados.rados.0.smithi123.stdout:1777: writing smithi12313934-28 from 1744696 to 2335579 tid 2 2020-01-16T00:14:15.585 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-01-16T00:14:15.588 INFO:tasks.rados.rados.0.smithi123.stdout:1777: writing smithi12313934-28 from 2772934 to 2907743 tid 3 2020-01-16T00:14:15.588 INFO:tasks.rados.rados.0.smithi123.stdout:1778: write (excl) oid 37 current snap is 0 2020-01-16T00:14:15.588 INFO:tasks.rados.rados.0.smithi123.stdout:1778: seq_num 650 ranges {524550=401855,1403178=450894,2516710=407840} 2020-01-16T00:14:15.593 INFO:tasks.rados.rados.0.smithi123.stdout:1778: writing smithi12313934-37 from 524550 to 926405 tid 1 2020-01-16T00:14:15.598 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:15.594+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:15.598 INFO:tasks.rados.rados.0.smithi123.stdout:1778: writing smithi12313934-37 from 1403178 to 1854072 tid 2 2020-01-16T00:14:15.605 INFO:tasks.rados.rados.0.smithi123.stdout:1778: writing smithi12313934-37 from 2516710 to 2924550 tid 3 2020-01-16T00:14:15.605 INFO:tasks.rados.rados.0.smithi123.stdout:1779: read oid 26 snap -1 2020-01-16T00:14:15.605 INFO:tasks.rados.rados.0.smithi123.stdout:1779: expect (ObjNum 556 snap 0 seq_num 556) 2020-01-16T00:14:15.605 INFO:tasks.rados.rados.0.smithi123.stdout:1780: delete oid 44 current snap is 0 2020-01-16T00:14:15.605 INFO:tasks.rados.rados.0.smithi123.stdout:1776: expect (ObjNum 618 snap 0 seq_num 618) 2020-01-16T00:14:15.680 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:15.688 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-01-16T00:14:15.699 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:15.696+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:15.800 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:15.798+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:15.816 INFO:tasks.rados.rados.0.smithi123.stdout:1776: done (4 left) 2020-01-16T00:14:15.817 INFO:tasks.rados.rados.0.smithi123.stdout:1780: done (3 left) 2020-01-16T00:14:15.817 INFO:tasks.rados.rados.0.smithi123.stdout:1781: read oid 36 snap -1 2020-01-16T00:14:15.817 INFO:tasks.rados.rados.0.smithi123.stdout:1781: expect deleted 2020-01-16T00:14:15.817 INFO:tasks.rados.rados.0.smithi123.stdout:1777: finishing write tid 1 to smithi12313934-28 2020-01-16T00:14:15.817 INFO:tasks.rados.rados.0.smithi123.stdout:1777: finishing write tid 2 to smithi12313934-28 2020-01-16T00:14:15.817 INFO:tasks.rados.rados.0.smithi123.stdout:1777: finishing write tid 3 to smithi12313934-28 2020-01-16T00:14:15.817 INFO:tasks.rados.rados.0.smithi123.stdout:1777: finishing write tid 5 to smithi12313934-28 2020-01-16T00:14:15.818 INFO:tasks.rados.rados.0.smithi123.stdout:1777: finishing write tid 6 to smithi12313934-28 2020-01-16T00:14:15.818 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 28 v 228 (ObjNum 649 snap 0 seq_num 649) dirty exists 2020-01-16T00:14:15.818 INFO:tasks.rados.rados.0.smithi123.stdout:1777: left oid 28 (ObjNum 649 snap 0 seq_num 649) 2020-01-16T00:14:15.818 INFO:tasks.rados.rados.0.smithi123.stdout:1778: finishing write tid 1 to smithi12313934-37 2020-01-16T00:14:15.818 INFO:tasks.rados.rados.0.smithi123.stdout:1778: finishing write tid 2 to smithi12313934-37 2020-01-16T00:14:15.818 INFO:tasks.rados.rados.0.smithi123.stdout:1782: read oid 2 snap -1 2020-01-16T00:14:15.818 INFO:tasks.rados.rados.0.smithi123.stdout:1782: expect (ObjNum 636 snap 0 seq_num 636) 2020-01-16T00:14:15.818 INFO:tasks.rados.rados.0.smithi123.stdout:1777: done (4 left) 2020-01-16T00:14:15.819 INFO:tasks.rados.rados.0.smithi123.stdout:1783: write (excl) oid 28 current snap is 0 2020-01-16T00:14:15.819 INFO:tasks.rados.rados.0.smithi123.stdout:1783: seq_num 652 ranges {558164=552353,1803789=454563,2855203=102961} 2020-01-16T00:14:15.822 INFO:tasks.rados.rados.0.smithi123.stdout:1783: writing smithi12313934-28 from 558164 to 1110517 tid 1 2020-01-16T00:14:15.829 INFO:tasks.rados.rados.0.smithi123.stdout:1783: writing smithi12313934-28 from 1803789 to 2258352 tid 2 2020-01-16T00:14:15.834 INFO:tasks.rados.rados.0.smithi123.stdout:1783: writing smithi12313934-28 from 2855203 to 2958164 tid 3 2020-01-16T00:14:15.834 INFO:tasks.rados.rados.0.smithi123.stdout:1784: read oid 40 snap -1 2020-01-16T00:14:15.834 INFO:tasks.rados.rados.0.smithi123.stdout:1784: expect (ObjNum 638 snap 0 seq_num 638) 2020-01-16T00:14:15.834 INFO:tasks.rados.rados.0.smithi123.stdout:1785: write (excl) oid 3 current snap is 0 2020-01-16T00:14:15.834 INFO:tasks.rados.rados.0.smithi123.stdout:1785: seq_num 653 ranges {574971=627602,1862271=598221} 2020-01-16T00:14:15.834 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:15.840 INFO:tasks.rados.rados.0.smithi123.stdout:1785: writing smithi12313934-3 from 574971 to 1202573 tid 1 2020-01-16T00:14:15.843 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-01-16T00:14:15.848 INFO:tasks.rados.rados.0.smithi123.stdout:1785: writing smithi12313934-3 from 1862271 to 2460492 tid 2 2020-01-16T00:14:15.848 INFO:tasks.rados.rados.0.smithi123.stdout:1786: read oid 17 snap -1 2020-01-16T00:14:15.848 INFO:tasks.rados.rados.0.smithi123.stdout:1786: expect deleted 2020-01-16T00:14:15.848 INFO:tasks.rados.rados.0.smithi123.stdout:1787: write (excl) oid 49 current snap is 0 2020-01-16T00:14:15.848 INFO:tasks.rados.rados.0.smithi123.stdout:1787: seq_num 654 ranges {591778=419204,1637106=741879,2910049=81729} 2020-01-16T00:14:15.861 INFO:tasks.rados.rados.0.smithi123.stdout:1787: writing smithi12313934-49 from 591778 to 1010982 tid 1 2020-01-16T00:14:15.862 INFO:tasks.rados.rados.0.smithi123.stdout:1787: writing smithi12313934-49 from 1637106 to 2378985 tid 2 2020-01-16T00:14:15.866 INFO:tasks.rados.rados.0.smithi123.stdout:1787: writing smithi12313934-49 from 2910049 to 2991778 tid 3 2020-01-16T00:14:15.866 INFO:tasks.rados.rados.0.smithi123.stdout:1788: read oid 18 snap -1 2020-01-16T00:14:15.866 INFO:tasks.rados.rados.0.smithi123.stdout:1788: expect (ObjNum 634 snap 0 seq_num 634) 2020-01-16T00:14:15.867 INFO:tasks.rados.rados.0.smithi123.stdout:1789: read oid 10 snap -1 2020-01-16T00:14:15.867 INFO:tasks.rados.rados.0.smithi123.stdout:1789: expect deleted 2020-01-16T00:14:15.867 INFO:tasks.rados.rados.0.smithi123.stdout:1790: read oid 48 snap -1 2020-01-16T00:14:15.867 INFO:tasks.rados.rados.0.smithi123.stdout:1790: expect (ObjNum 599 snap 0 seq_num 599) 2020-01-16T00:14:15.867 INFO:tasks.rados.rados.0.smithi123.stdout:1791: write oid 12 current snap is 0 2020-01-16T00:14:15.867 INFO:tasks.rados.rados.0.smithi123.stdout:1791: seq_num 655 ranges {608585=494453,1579235=601890,2821119=187466} 2020-01-16T00:14:15.873 INFO:tasks.rados.rados.0.smithi123.stdout:1791: writing smithi12313934-12 from 608585 to 1103038 tid 1 2020-01-16T00:14:15.875 INFO:teuthology.orchestra.run.smithi123.stderr:nodeep-scrub is unset 2020-01-16T00:14:15.881 INFO:tasks.rados.rados.0.smithi123.stdout:1791: writing smithi12313934-12 from 1579235 to 2181125 tid 2 2020-01-16T00:14:15.885 INFO:tasks.rados.rados.0.smithi123.stdout:1791: writing smithi12313934-12 from 2821119 to 3008585 tid 3 2020-01-16T00:14:15.885 INFO:tasks.rados.rados.0.smithi123.stdout:1792: read oid 6 snap -1 2020-01-16T00:14:15.885 INFO:tasks.rados.rados.0.smithi123.stdout:1792: expect (ObjNum 629 snap 0 seq_num 629) 2020-01-16T00:14:15.885 INFO:tasks.rados.rados.0.smithi123.stdout:1793: write oid 31 current snap is 0 2020-01-16T00:14:15.885 INFO:tasks.rados.rados.0.smithi123.stdout:1793: seq_num 656 ranges {625392=569702,1637717=745548,2848542=176850} 2020-01-16T00:14:15.891 INFO:tasks.rados.rados.0.smithi123.stdout:1793: writing smithi12313934-31 from 625392 to 1195094 tid 1 2020-01-16T00:14:15.899 INFO:tasks.rados.rados.0.smithi123.stdout:1793: writing smithi12313934-31 from 1637717 to 2383265 tid 2 2020-01-16T00:14:15.902 INFO:tasks.rados.rados.0.smithi123.stdout:1793: writing smithi12313934-31 from 2848542 to 3025392 tid 3 2020-01-16T00:14:15.902 INFO:tasks.rados.rados.0.smithi123.stdout:1794: read oid 35 snap -1 2020-01-16T00:14:15.902 INFO:tasks.rados.rados.0.smithi123.stdout:1794: expect (ObjNum 628 snap 0 seq_num 628) 2020-01-16T00:14:15.903 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:15.903 INFO:tasks.rados.rados.0.smithi123.stdout:1778: finishing write tid 3 to smithi12313934-37 2020-01-16T00:14:15.903 INFO:tasks.rados.rados.0.smithi123.stdout:1778: finishing write tid 5 to smithi12313934-37 2020-01-16T00:14:15.903 INFO:tasks.rados.rados.0.smithi123.stdout:1778: finishing write tid 6 to smithi12313934-37 2020-01-16T00:14:15.903 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 37 v 340 (ObjNum 650 snap 0 seq_num 650) dirty exists 2020-01-16T00:14:15.904 INFO:tasks.rados.rados.0.smithi123.stdout:1778: left oid 37 (ObjNum 650 snap 0 seq_num 650) 2020-01-16T00:14:15.904 INFO:tasks.rados.rados.0.smithi123.stdout:1779: expect (ObjNum 556 snap 0 seq_num 556) 2020-01-16T00:14:15.904 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:15.900+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:15.958 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:15.966 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-01-16T00:14:15.999 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-01-16T00:14:16.007 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:16.006+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:16.104 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:16.102+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:16.111 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:16.126 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-01-16T00:14:16.132 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:14:16.157 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:14:16.170 INFO:tasks.rados.rados.0.smithi123.stdout:1778: done (15 left) 2020-01-16T00:14:16.170 INFO:tasks.rados.rados.0.smithi123.stdout:1779: done (14 left) 2020-01-16T00:14:16.170 INFO:tasks.rados.rados.0.smithi123.stdout:1795: write (excl) oid 32 current snap is 0 2020-01-16T00:14:16.170 INFO:tasks.rados.rados.0.smithi123.stdout:1795: seq_num 657 ranges {642199=644951,1696199=605559,2992318=49881} 2020-01-16T00:14:16.176 INFO:tasks.rados.rados.0.smithi123.stdout:1795: writing smithi12313934-32 from 642199 to 1287150 tid 1 2020-01-16T00:14:16.183 INFO:tasks.rados.rados.0.smithi123.stdout:1795: writing smithi12313934-32 from 1696199 to 2301758 tid 2 2020-01-16T00:14:16.187 INFO:tasks.rados.rados.0.smithi123.stdout:1795: writing smithi12313934-32 from 2992318 to 3042199 tid 3 2020-01-16T00:14:16.187 INFO:tasks.rados.rados.0.smithi123.stdout:1796: write (excl) oid 42 current snap is 0 2020-01-16T00:14:16.188 INFO:tasks.rados.rados.0.smithi123.stdout:1796: seq_num 658 ranges {659006=720200,2154681=749217} 2020-01-16T00:14:16.188 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:14:16.195 INFO:tasks.rados.rados.0.smithi123.stdout:1796: writing smithi12313934-42 from 659006 to 1379206 tid 1 2020-01-16T00:14:16.203 INFO:tasks.rados.rados.0.smithi123.stdout:1796: writing smithi12313934-42 from 2154681 to 2903898 tid 2 2020-01-16T00:14:16.204 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:16.204 INFO:tasks.rados.rados.0.smithi123.stdout:1782: expect (ObjNum 636 snap 0 seq_num 636) 2020-01-16T00:14:16.206 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:16.204+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:16.209 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:14:16.230 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:14:16.250 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:14:16.264 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:16.264 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:14:16.265 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:14:16.265 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:14:16.265 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:16.271 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:14:16.273 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-01-16T00:14:16.273 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-01-16T00:14:16.273 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-01-16T00:14:16.274 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-01-16T00:14:16.274 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-01-16T00:14:16.274 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-01-16T00:14:16.274 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENXIO: problem getting command descriptions from osd.5 2020-01-16T00:14:16.274 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: problem getting command descriptions from osd.5 2020-01-16T00:14:16.274 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-01-16T00:14:16.274 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-01-16T00:14:16.292 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-01-16T00:14:16.292 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-01-16T00:14:16.340 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:16.338+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:16.407 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:16.404+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:16.458 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:16.459 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:14:16.459 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:14:16.459 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:14:16.459 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.459 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1649 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~2084068,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:14:16.459 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847683+0000", 2020-01-16T00:14:16.460 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 84.607518975999994, 2020-01-16T00:14:16.460 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010654691000000001, 2020-01-16T00:14:16.460 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:16.460 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:16.460 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:16.460 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:16.460 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:16.460 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1649 2020-01-16T00:14:16.461 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.461 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:16.461 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.461 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:16.461 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847683+0000", 2020-01-16T00:14:16.461 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:16.461 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.462 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.462 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:16.462 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847683+0000", 2020-01-16T00:14:16.462 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:16.462 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.462 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.462 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:16.463 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847682+0000", 2020-01-16T00:14:16.463 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8569999999999997e-06 2020-01-16T00:14:16.463 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.463 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.463 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:16.463 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847689+0000", 2020-01-16T00:14:16.463 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8090000000000004e-06 2020-01-16T00:14:16.463 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.464 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.464 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:16.464 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847696+0000", 2020-01-16T00:14:16.464 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2690000000000001e-06 2020-01-16T00:14:16.464 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.464 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.464 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:16.464 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847700+0000", 2020-01-16T00:14:16.465 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8189999999999993e-05 2020-01-16T00:14:16.465 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.465 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.465 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.465 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847798+0000", 2020-01-16T00:14:16.465 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7129999999999999e-06 2020-01-16T00:14:16.465 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.465 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.466 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:14:16.466 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847802+0000", 2020-01-16T00:14:16.466 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083913049999999999 2020-01-16T00:14:16.466 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.466 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.466 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.466 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.856193+0000", 2020-01-16T00:14:16.466 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034400300000000001 2020-01-16T00:14:16.467 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.467 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.467 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:16.467 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.856537+0000", 2020-01-16T00:14:16.467 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018006820000000001 2020-01-16T00:14:16.467 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.467 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.468 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:16.468 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858338+0000", 2020-01-16T00:14:16.468 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010654691000000001 2020-01-16T00:14:16.468 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.468 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:16.468 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.468 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.468 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.469 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1650 3.1 3:8de1d8a3:::smithi12313934-13:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:14:16.469 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847717+0000", 2020-01-16T00:14:16.469 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 84.607485022999995, 2020-01-16T00:14:16.469 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010913658, 2020-01-16T00:14:16.469 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:16.469 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:16.469 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:16.469 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:16.470 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:16.470 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1650 2020-01-16T00:14:16.470 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.470 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:16.470 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.470 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:16.470 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847717+0000", 2020-01-16T00:14:16.470 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:16.471 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.471 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.471 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:16.471 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847717+0000", 2020-01-16T00:14:16.471 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:16.471 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.471 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.472 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:16.472 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847716+0000", 2020-01-16T00:14:16.472 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.433e-06 2020-01-16T00:14:16.472 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.472 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.472 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:16.472 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847720+0000", 2020-01-16T00:14:16.472 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1800000000000004e-06 2020-01-16T00:14:16.473 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.473 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.473 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:16.473 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847725+0000", 2020-01-16T00:14:16.473 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6689999999999997e-06 2020-01-16T00:14:16.473 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.473 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.473 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:16.474 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847730+0000", 2020-01-16T00:14:16.474 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000102468 2020-01-16T00:14:16.474 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.474 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.474 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.474 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847832+0000", 2020-01-16T00:14:16.474 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3999999999999997e-06 2020-01-16T00:14:16.474 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.475 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.475 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:14:16.475 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847838+0000", 2020-01-16T00:14:16.475 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010519395000000001 2020-01-16T00:14:16.475 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.475 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.475 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.475 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858358+0000", 2020-01-16T00:14:16.476 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7600000000000002e-05 2020-01-16T00:14:16.476 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.476 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.476 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:16.476 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858445+0000", 2020-01-16T00:14:16.476 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018515900000000001 2020-01-16T00:14:16.476 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.477 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.477 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:16.477 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858631+0000", 2020-01-16T00:14:16.477 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010913658 2020-01-16T00:14:16.477 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.477 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:16.477 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.477 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.478 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.478 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1651 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~2084068,checksum] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:14:16.478 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847752+0000", 2020-01-16T00:14:16.478 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 84.607450407000002, 2020-01-16T00:14:16.478 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011734249, 2020-01-16T00:14:16.478 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:16.478 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:16.478 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:16.479 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:16.479 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:16.479 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1651 2020-01-16T00:14:16.479 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.479 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:16.479 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.479 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:16.479 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847752+0000", 2020-01-16T00:14:16.480 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:16.480 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.480 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.480 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:16.480 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847752+0000", 2020-01-16T00:14:16.480 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:16.480 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.480 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.481 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:16.481 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847751+0000", 2020-01-16T00:14:16.481 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1980000000000001e-06 2020-01-16T00:14:16.481 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.481 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.481 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:16.481 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847754+0000", 2020-01-16T00:14:16.482 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6679999999999999e-06 2020-01-16T00:14:16.482 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.482 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.482 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:16.482 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847758+0000", 2020-01-16T00:14:16.482 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2500000000000001e-06 2020-01-16T00:14:16.482 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.482 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.483 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:16.483 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847760+0000", 2020-01-16T00:14:16.483 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7614000000000004e-05 2020-01-16T00:14:16.483 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.483 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.483 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.483 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847858+0000", 2020-01-16T00:14:16.483 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7950000000000004e-06 2020-01-16T00:14:16.484 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.484 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.484 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:14:16.484 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847864+0000", 2020-01-16T00:14:16.484 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010790345 2020-01-16T00:14:16.484 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.484 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.485 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.485 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858654+0000", 2020-01-16T00:14:16.485 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000141466 2020-01-16T00:14:16.485 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.485 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.485 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:16.485 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858795+0000", 2020-01-16T00:14:16.485 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00069037899999999997 2020-01-16T00:14:16.486 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.486 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.486 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:16.486 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.859486+0000", 2020-01-16T00:14:16.486 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011734249 2020-01-16T00:14:16.486 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.486 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:16.486 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.487 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.487 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.487 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2108 3.1 3:8de1d8a3:::smithi12313934-13:head [write 2816700~410210] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:14:16.487 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.641861+0000", 2020-01-16T00:14:16.487 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 74.813341217000001, 2020-01-16T00:14:16.487 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018545604, 2020-01-16T00:14:16.487 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:16.488 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:16.488 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:16.488 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:16.488 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:16.488 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2108 2020-01-16T00:14:16.488 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.489 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:16.489 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.489 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:16.492 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.641861+0000", 2020-01-16T00:14:16.493 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:16.493 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.493 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.493 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:16.493 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.641861+0000", 2020-01-16T00:14:16.494 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:16.494 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.494 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.494 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:16.494 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.641859+0000", 2020-01-16T00:14:16.494 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018466399999999999 2020-01-16T00:14:16.494 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.494 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.495 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:16.495 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642043+0000", 2020-01-16T00:14:16.495 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2999999999999999e-05 2020-01-16T00:14:16.495 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.495 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.495 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:16.495 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642056+0000", 2020-01-16T00:14:16.495 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0730000000000005e-06 2020-01-16T00:14:16.496 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.496 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.496 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:16.496 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642064+0000", 2020-01-16T00:14:16.496 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9527999999999997e-05 2020-01-16T00:14:16.496 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.496 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.497 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.497 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642114+0000", 2020-01-16T00:14:16.497 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036616050000000001 2020-01-16T00:14:16.497 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.497 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.497 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:16.497 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645776+0000", 2020-01-16T00:14:16.497 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012222663999999999 2020-01-16T00:14:16.498 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.498 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.498 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.498 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.657998+0000", 2020-01-16T00:14:16.498 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014280299999999999 2020-01-16T00:14:16.498 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.498 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.499 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:16.499 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.658141+0000", 2020-01-16T00:14:16.499 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0001102 2020-01-16T00:14:16.499 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.499 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.499 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:14:16.499 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.658251+0000", 2020-01-16T00:14:16.499 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020421219999999999 2020-01-16T00:14:16.500 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.500 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.500 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:16.500 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660293+0000", 2020-01-16T00:14:16.500 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7399999999999999e-06 2020-01-16T00:14:16.500 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.500 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.500 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:16.501 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660303+0000", 2020-01-16T00:14:16.501 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9873999999999998e-05 2020-01-16T00:14:16.501 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.501 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.501 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:16.501 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660343+0000", 2020-01-16T00:14:16.501 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4448e-05 2020-01-16T00:14:16.502 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.502 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.502 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:16.502 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660377+0000", 2020-01-16T00:14:16.502 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8915999999999999e-05 2020-01-16T00:14:16.502 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.502 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.503 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:16.503 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660406+0000", 2020-01-16T00:14:16.503 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018545604 2020-01-16T00:14:16.503 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.503 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:16.503 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.503 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.503 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.504 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2109 3.1 3:8de1d8a3:::smithi12313934-13:head [setxattr _header (56) in=7b,truncate 3865610] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:14:16.504 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.642075+0000", 2020-01-16T00:14:16.504 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 74.813126643999993, 2020-01-16T00:14:16.504 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018477655999999999, 2020-01-16T00:14:16.504 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:16.504 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:16.504 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:16.504 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:16.505 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:16.505 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2109 2020-01-16T00:14:16.505 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.505 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:16.505 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.505 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:16.505 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642075+0000", 2020-01-16T00:14:16.505 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:16.506 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.506 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.506 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:16.506 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642075+0000", 2020-01-16T00:14:16.506 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:16.506 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.506 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.506 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:16.507 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642074+0000", 2020-01-16T00:14:16.507 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4500000000000003e-06 2020-01-16T00:14:16.507 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.507 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.507 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:16.507 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642080+0000", 2020-01-16T00:14:16.507 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7229999999999998e-06 2020-01-16T00:14:16.507 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.508 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.508 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:16.508 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642087+0000", 2020-01-16T00:14:16.508 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2390000000000004e-06 2020-01-16T00:14:16.508 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.508 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.508 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:16.509 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642091+0000", 2020-01-16T00:14:16.509 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0037396550000000001 2020-01-16T00:14:16.509 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.509 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.509 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.509 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645830+0000", 2020-01-16T00:14:16.509 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2200000000000002e-05 2020-01-16T00:14:16.509 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.510 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.510 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:16.510 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645883+0000", 2020-01-16T00:14:16.510 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013432934000000001 2020-01-16T00:14:16.510 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.510 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.510 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.510 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659316+0000", 2020-01-16T00:14:16.511 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000140006 2020-01-16T00:14:16.511 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.511 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.511 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:16.511 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659456+0000", 2020-01-16T00:14:16.511 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000184873 2020-01-16T00:14:16.511 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.511 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.512 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:14:16.512 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659640+0000", 2020-01-16T00:14:16.512 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00066954999999999998 2020-01-16T00:14:16.512 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.512 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.512 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:16.512 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660310+0000", 2020-01-16T00:14:16.512 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014847699999999999 2020-01-16T00:14:16.513 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.513 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.513 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:16.513 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660459+0000", 2020-01-16T00:14:16.513 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6869e-05 2020-01-16T00:14:16.513 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.513 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.513 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:16.514 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660485+0000", 2020-01-16T00:14:16.514 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8478000000000001e-05 2020-01-16T00:14:16.514 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.514 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.514 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:16.514 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660514+0000", 2020-01-16T00:14:16.514 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9153999999999998e-05 2020-01-16T00:14:16.515 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.515 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.515 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:16.515 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660553+0000", 2020-01-16T00:14:16.515 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018477655999999999 2020-01-16T00:14:16.515 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.515 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:16.515 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.516 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.516 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.516 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2110 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-01-16T00:14:16.516 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.642112+0000", 2020-01-16T00:14:16.516 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 74.813090005999996, 2020-01-16T00:14:16.516 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018620198000000001, 2020-01-16T00:14:16.516 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:16.516 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:16.517 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:16.517 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:16.517 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:16.517 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2110 2020-01-16T00:14:16.517 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.517 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:16.517 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.517 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:16.518 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642112+0000", 2020-01-16T00:14:16.518 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:16.518 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.518 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.518 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:16.518 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642112+0000", 2020-01-16T00:14:16.518 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:16.519 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.519 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.519 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:16.519 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642111+0000", 2020-01-16T00:14:16.519 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4450000000000001e-06 2020-01-16T00:14:16.519 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.519 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.519 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:16.520 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642117+0000", 2020-01-16T00:14:16.520 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1567e-05 2020-01-16T00:14:16.520 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.520 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.520 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:16.520 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642129+0000", 2020-01-16T00:14:16.520 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.754e-06 2020-01-16T00:14:16.520 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.521 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.521 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:16.521 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642136+0000", 2020-01-16T00:14:16.521 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040239719999999998 2020-01-16T00:14:16.521 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.521 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.521 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.521 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646160+0000", 2020-01-16T00:14:16.522 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5600999999999998e-05 2020-01-16T00:14:16.522 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.522 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.522 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:16.522 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646215+0000", 2020-01-16T00:14:16.522 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013733006000000001 2020-01-16T00:14:16.522 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.522 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.523 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.523 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659948+0000", 2020-01-16T00:14:16.523 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000106277 2020-01-16T00:14:16.523 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.523 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.523 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:16.523 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660054+0000", 2020-01-16T00:14:16.523 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000524327 2020-01-16T00:14:16.524 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.524 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.524 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.524 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660579+0000", 2020-01-16T00:14:16.524 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9841999999999999e-05 2020-01-16T00:14:16.524 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.524 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.524 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:16.525 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660659+0000", 2020-01-16T00:14:16.525 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3572999999999995e-05 2020-01-16T00:14:16.525 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.525 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.525 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:16.525 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660732+0000", 2020-01-16T00:14:16.525 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018620198000000001 2020-01-16T00:14:16.525 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.526 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:16.526 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.526 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.526 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.526 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:70 2.1 e35/19 2:91f85463:::smithi12313934-24 low_tier:head v 35'66, mlcod=34'64)", 2020-01-16T00:14:16.526 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.986684+0000", 2020-01-16T00:14:16.526 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 70.468518152000001, 2020-01-16T00:14:16.527 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012287378999999999, 2020-01-16T00:14:16.527 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:16.527 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:16.527 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:16.527 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.527 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:16.527 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986684+0000", 2020-01-16T00:14:16.527 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:16.528 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.528 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.528 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:16.528 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986684+0000", 2020-01-16T00:14:16.528 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:16.528 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.528 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.528 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:16.529 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986683+0000", 2020-01-16T00:14:16.529 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012884179999999999 2020-01-16T00:14:16.529 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.529 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.529 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:16.529 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.987971+0000", 2020-01-16T00:14:16.529 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011558200000000001 2020-01-16T00:14:16.529 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.530 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.530 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:16.530 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988087+0000", 2020-01-16T00:14:16.530 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0472000000000001e-05 2020-01-16T00:14:16.530 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.530 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.530 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:16.530 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988137+0000", 2020-01-16T00:14:16.531 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000107484 2020-01-16T00:14:16.531 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.531 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.531 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.531 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988245+0000", 2020-01-16T00:14:16.531 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4409999999999998e-05 2020-01-16T00:14:16.531 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.531 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.532 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:16.532 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988279+0000", 2020-01-16T00:14:16.532 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010641609999999999 2020-01-16T00:14:16.532 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.532 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.532 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:16.532 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.998921+0000", 2020-01-16T00:14:16.532 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0278000000000002e-05 2020-01-16T00:14:16.533 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.533 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.533 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:16.533 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.998971+0000", 2020-01-16T00:14:16.533 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012287378999999999 2020-01-16T00:14:16.533 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.533 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:16.533 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.534 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.534 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.534 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:109 2.0 e42/19 2:11fa79bc:::smithi12313934-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head v 42'67, mlcod=41'65)", 2020-01-16T00:14:16.534 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:21.841140+0000", 2020-01-16T00:14:16.534 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 54.614062083999997, 2020-01-16T00:14:16.534 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010439159999999999, 2020-01-16T00:14:16.534 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:16.534 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:16.535 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:16.535 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.535 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:16.535 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841140+0000", 2020-01-16T00:14:16.535 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:16.535 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.536 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.536 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:16.536 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841140+0000", 2020-01-16T00:14:16.536 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:16.536 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.536 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.537 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:16.537 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841137+0000", 2020-01-16T00:14:16.537 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00061049800000000001 2020-01-16T00:14:16.537 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.537 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.537 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:16.537 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841748+0000", 2020-01-16T00:14:16.538 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8703e-05 2020-01-16T00:14:16.538 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.538 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.538 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:16.538 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841767+0000", 2020-01-16T00:14:16.538 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5410000000000003e-06 2020-01-16T00:14:16.538 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.538 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.539 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:16.539 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841774+0000", 2020-01-16T00:14:16.539 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000100144 2020-01-16T00:14:16.539 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.539 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.539 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.539 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841874+0000", 2020-01-16T00:14:16.539 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9402999999999999e-05 2020-01-16T00:14:16.540 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.540 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.540 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:16.540 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.841944+0000", 2020-01-16T00:14:16.540 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095901649999999995 2020-01-16T00:14:16.540 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.540 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.541 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:16.541 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.851534+0000", 2020-01-16T00:14:16.541 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5296000000000001e-05 2020-01-16T00:14:16.541 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.541 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.541 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:16.541 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.851579+0000", 2020-01-16T00:14:16.541 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010439159999999999 2020-01-16T00:14:16.542 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.542 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:16.542 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.542 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.542 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3228 3.1 3:8de1d8a3:::smithi12313934-13:head [write 588344~631611,stat] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:14:16.542 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.588539+0000", 2020-01-16T00:14:16.542 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 46.866662914999999, 2020-01-16T00:14:16.543 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012776884000000001, 2020-01-16T00:14:16.543 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:16.543 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:16.543 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:16.543 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:16.543 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:16.543 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3228 2020-01-16T00:14:16.543 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.544 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:16.544 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.544 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:16.544 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588539+0000", 2020-01-16T00:14:16.544 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:16.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.544 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.544 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:16.545 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588539+0000", 2020-01-16T00:14:16.545 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:16.545 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.545 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.545 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:16.545 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588536+0000", 2020-01-16T00:14:16.545 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021452 2020-01-16T00:14:16.546 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.546 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.546 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:16.546 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588750+0000", 2020-01-16T00:14:16.546 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.1432000000000001e-05 2020-01-16T00:14:16.546 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.546 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.546 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:16.547 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588772+0000", 2020-01-16T00:14:16.547 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1799999999999999e-06 2020-01-16T00:14:16.547 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.547 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.547 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:16.547 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588779+0000", 2020-01-16T00:14:16.547 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00095777700000000004 2020-01-16T00:14:16.547 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.548 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.548 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.548 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589737+0000", 2020-01-16T00:14:16.548 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013306599999999999 2020-01-16T00:14:16.548 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.548 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.548 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:16.549 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589870+0000", 2020-01-16T00:14:16.549 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020556 2020-01-16T00:14:16.549 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.550 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.550 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:14:16.550 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.590075+0000", 2020-01-16T00:14:16.550 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040226080000000004 2020-01-16T00:14:16.550 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.551 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.551 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:16.551 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.594098+0000", 2020-01-16T00:14:16.551 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.006043288 2020-01-16T00:14:16.551 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.551 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.551 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:16.551 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.600141+0000", 2020-01-16T00:14:16.552 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00089242900000000003 2020-01-16T00:14:16.552 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.552 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.552 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:16.552 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601034+0000", 2020-01-16T00:14:16.552 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6097e-05 2020-01-16T00:14:16.552 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.553 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.553 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:16.553 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601110+0000", 2020-01-16T00:14:16.553 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000206088 2020-01-16T00:14:16.553 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.553 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.554 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:16.554 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601316+0000", 2020-01-16T00:14:16.554 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012776884000000001 2020-01-16T00:14:16.554 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.554 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:16.554 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.554 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.554 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.555 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.1.13:77 2.1 e45/19 2:bafac7f5:::smithi12313934-46 low_tier:head v 45'100, mlcod=45'98)", 2020-01-16T00:14:16.555 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.589633+0000", 2020-01-16T00:14:16.555 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 46.865568691999997, 2020-01-16T00:14:16.555 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010560460000000001, 2020-01-16T00:14:16.555 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:16.555 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:16.555 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:16.555 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.556 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:16.556 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589633+0000", 2020-01-16T00:14:16.556 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:16.556 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.556 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.556 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:16.556 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589633+0000", 2020-01-16T00:14:16.557 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:16.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.557 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.557 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:16.557 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589630+0000", 2020-01-16T00:14:16.557 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00050499799999999999 2020-01-16T00:14:16.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.557 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.558 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:16.558 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.590135+0000", 2020-01-16T00:14:16.558 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0473e-05 2020-01-16T00:14:16.558 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.558 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.558 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:16.558 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.590146+0000", 2020-01-16T00:14:16.558 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9710000000000001e-06 2020-01-16T00:14:16.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.559 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.559 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:16.559 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.590152+0000", 2020-01-16T00:14:16.559 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00040412999999999999 2020-01-16T00:14:16.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.559 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.559 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.560 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.590556+0000", 2020-01-16T00:14:16.560 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6491000000000001e-05 2020-01-16T00:14:16.560 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.560 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.560 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:16.561 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.590583+0000", 2020-01-16T00:14:16.561 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095621360000000006 2020-01-16T00:14:16.561 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.561 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.561 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:16.561 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.600145+0000", 2020-01-16T00:14:16.561 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9072e-05 2020-01-16T00:14:16.561 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.562 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.562 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:16.562 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.600194+0000", 2020-01-16T00:14:16.562 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010560460000000001 2020-01-16T00:14:16.562 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.562 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:16.562 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.562 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.563 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.563 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3229 3.1 3:8de1d8a3:::smithi12313934-13:head [write 1689059~741123] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:14:16.563 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.596087+0000", 2020-01-16T00:14:16.563 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 46.859114853999998, 2020-01-16T00:14:16.563 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010699248, 2020-01-16T00:14:16.563 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:16.563 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:16.564 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:16.564 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:16.564 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:16.564 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3229 2020-01-16T00:14:16.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.564 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:16.564 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.564 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:16.565 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596087+0000", 2020-01-16T00:14:16.565 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:16.565 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.565 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.565 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:16.565 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596087+0000", 2020-01-16T00:14:16.565 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:16.565 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.566 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.566 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:16.566 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596084+0000", 2020-01-16T00:14:16.566 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00026243600000000001 2020-01-16T00:14:16.566 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.566 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.566 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:16.567 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596347+0000", 2020-01-16T00:14:16.567 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3594000000000001e-05 2020-01-16T00:14:16.567 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.567 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.567 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:16.567 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596370+0000", 2020-01-16T00:14:16.567 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5259999999999999e-06 2020-01-16T00:14:16.568 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.568 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.568 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:16.568 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596380+0000", 2020-01-16T00:14:16.568 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2298000000000005e-05 2020-01-16T00:14:16.568 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.569 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.569 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.569 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596452+0000", 2020-01-16T00:14:16.569 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000163702 2020-01-16T00:14:16.569 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.569 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.570 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:16.570 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596616+0000", 2020-01-16T00:14:16.570 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0047611729999999996 2020-01-16T00:14:16.570 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.570 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.571 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.571 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601377+0000", 2020-01-16T00:14:16.571 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014076900000000001 2020-01-16T00:14:16.571 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.571 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.571 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:16.571 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601518+0000", 2020-01-16T00:14:16.571 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021367599999999999 2020-01-16T00:14:16.572 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.572 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.572 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:14:16.572 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601731+0000", 2020-01-16T00:14:16.572 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0028479149999999999 2020-01-16T00:14:16.572 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.572 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.572 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:16.573 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.604579+0000", 2020-01-16T00:14:16.573 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001408133 2020-01-16T00:14:16.573 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.573 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.573 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:16.573 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605987+0000", 2020-01-16T00:14:16.573 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00072335599999999998 2020-01-16T00:14:16.573 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.574 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.574 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:16.574 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.606711+0000", 2020-01-16T00:14:16.574 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0289e-05 2020-01-16T00:14:16.574 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.574 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.574 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:16.575 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.606751+0000", 2020-01-16T00:14:16.575 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5298e-05 2020-01-16T00:14:16.575 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.575 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.575 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:16.575 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.606786+0000", 2020-01-16T00:14:16.575 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010699248 2020-01-16T00:14:16.575 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.576 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:16.576 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.576 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.576 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.576 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3230 3.1 3:8de1d8a3:::smithi12313934-13:head [setxattr _header (56) in=7b,truncate 2588344] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:14:16.576 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.596394+0000", 2020-01-16T00:14:16.577 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 46.858807894999998, 2020-01-16T00:14:16.577 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021205134, 2020-01-16T00:14:16.577 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:16.577 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:16.577 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:16.577 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:16.577 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:16.577 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3230 2020-01-16T00:14:16.578 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.578 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:16.578 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.578 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:16.578 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596394+0000", 2020-01-16T00:14:16.578 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:16.578 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.578 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.579 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:16.579 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596394+0000", 2020-01-16T00:14:16.579 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:16.579 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.579 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.580 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:16.580 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596393+0000", 2020-01-16T00:14:16.580 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7159999999999997e-06 2020-01-16T00:14:16.580 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.580 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.580 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:16.581 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596400+0000", 2020-01-16T00:14:16.581 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0074e-05 2020-01-16T00:14:16.581 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.581 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.581 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:16.581 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596410+0000", 2020-01-16T00:14:16.582 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1679999999999997e-06 2020-01-16T00:14:16.582 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.582 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.582 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:16.582 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596414+0000", 2020-01-16T00:14:16.582 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023609 2020-01-16T00:14:16.582 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.582 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.583 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.583 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596650+0000", 2020-01-16T00:14:16.583 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012316399999999999 2020-01-16T00:14:16.583 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.583 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.583 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:16.583 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596774+0000", 2020-01-16T00:14:16.583 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0067849670000000003 2020-01-16T00:14:16.584 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.584 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.584 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.584 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.603558+0000", 2020-01-16T00:14:16.584 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00092385999999999998 2020-01-16T00:14:16.584 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.584 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.584 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:16.585 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.604482+0000", 2020-01-16T00:14:16.585 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011729201 2020-01-16T00:14:16.585 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.585 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.585 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.585 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.616212+0000", 2020-01-16T00:14:16.585 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019515099999999999 2020-01-16T00:14:16.586 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.586 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.586 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:16.586 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.616407+0000", 2020-01-16T00:14:16.586 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000182843 2020-01-16T00:14:16.586 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.586 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.586 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:14:16.587 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.616590+0000", 2020-01-16T00:14:16.587 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000660572 2020-01-16T00:14:16.587 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.587 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.587 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:16.587 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617250+0000", 2020-01-16T00:14:16.587 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5333999999999997e-05 2020-01-16T00:14:16.587 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.588 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.588 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:16.588 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617285+0000", 2020-01-16T00:14:16.588 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023020899999999999 2020-01-16T00:14:16.588 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.588 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.588 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:16.588 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617516+0000", 2020-01-16T00:14:16.589 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4214000000000001e-05 2020-01-16T00:14:16.589 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.589 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.589 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:16.589 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617560+0000", 2020-01-16T00:14:16.589 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9366999999999998e-05 2020-01-16T00:14:16.589 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.590 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.590 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:16.590 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617599+0000", 2020-01-16T00:14:16.590 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021205134 2020-01-16T00:14:16.590 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.590 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:16.590 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.590 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.591 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.591 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3231 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e45)", 2020-01-16T00:14:16.591 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.596438+0000", 2020-01-16T00:14:16.591 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 46.858763885000002, 2020-01-16T00:14:16.591 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021455667000000001, 2020-01-16T00:14:16.591 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:16.592 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:16.592 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:16.592 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:16.592 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:16.592 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3231 2020-01-16T00:14:16.592 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.592 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:16.593 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.593 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:16.594 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596438+0000", 2020-01-16T00:14:16.594 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:16.594 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.594 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.594 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:16.595 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596438+0000", 2020-01-16T00:14:16.595 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:16.595 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.595 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.595 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:16.595 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596437+0000", 2020-01-16T00:14:16.596 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8500000000000004e-06 2020-01-16T00:14:16.596 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.596 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.596 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:16.596 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596441+0000", 2020-01-16T00:14:16.596 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9439999999999994e-06 2020-01-16T00:14:16.597 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.597 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.597 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:16.597 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596450+0000", 2020-01-16T00:14:16.597 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.267e-06 2020-01-16T00:14:16.597 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.597 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.598 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:16.598 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596455+0000", 2020-01-16T00:14:16.598 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034332399999999999 2020-01-16T00:14:16.598 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.598 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.598 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.598 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596798+0000", 2020-01-16T00:14:16.598 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011687400000000001 2020-01-16T00:14:16.599 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.599 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.599 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:16.599 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596915+0000", 2020-01-16T00:14:16.599 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0076057049999999999 2020-01-16T00:14:16.599 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.600 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.600 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.600 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.604520+0000", 2020-01-16T00:14:16.600 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8143999999999998e-05 2020-01-16T00:14:16.600 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.600 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.600 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:16.600 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.604569+0000", 2020-01-16T00:14:16.601 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012379960000000001 2020-01-16T00:14:16.601 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.601 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.601 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.601 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.616949+0000", 2020-01-16T00:14:16.601 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000120617 2020-01-16T00:14:16.602 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.602 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.602 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:16.602 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617069+0000", 2020-01-16T00:14:16.603 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00056715400000000005 2020-01-16T00:14:16.603 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.603 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.603 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.603 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617636+0000", 2020-01-16T00:14:16.603 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012844700000000001 2020-01-16T00:14:16.604 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.604 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.604 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:16.604 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617765+0000", 2020-01-16T00:14:16.604 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000129037 2020-01-16T00:14:16.604 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.604 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.605 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:16.605 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617894+0000", 2020-01-16T00:14:16.605 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021455667000000001 2020-01-16T00:14:16.605 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.605 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:16.605 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.605 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.606 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.606 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3822 3.1 3:8de1d8a3:::smithi12313934-13:head [setxattr _header (56) in=7b,truncate 2016939] snapc 0=[] ondisk+write+known_if_redirected e50)", 2020-01-16T00:14:16.606 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:44.059584+0000", 2020-01-16T00:14:16.606 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 32.395617569000002, 2020-01-16T00:14:16.606 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021977219999999999, 2020-01-16T00:14:16.606 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:16.606 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:16.607 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:16.607 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:16.607 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:16.607 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3822 2020-01-16T00:14:16.607 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.607 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:16.607 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.607 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:16.608 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059584+0000", 2020-01-16T00:14:16.608 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:16.608 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.608 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.608 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:16.608 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059584+0000", 2020-01-16T00:14:16.608 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:16.609 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.609 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.609 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:16.609 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059584+0000", 2020-01-16T00:14:16.610 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1329999999999998e-06 2020-01-16T00:14:16.610 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.610 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.610 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:16.610 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059589+0000", 2020-01-16T00:14:16.610 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8136000000000002e-05 2020-01-16T00:14:16.610 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.611 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.611 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:16.611 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059627+0000", 2020-01-16T00:14:16.611 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1409999999999998e-06 2020-01-16T00:14:16.611 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.611 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.611 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:16.611 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059631+0000", 2020-01-16T00:14:16.612 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0011314020000000001 2020-01-16T00:14:16.612 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.612 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.612 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.612 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.060763+0000", 2020-01-16T00:14:16.612 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00085078200000000001 2020-01-16T00:14:16.612 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.613 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.613 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:16.613 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.061613+0000", 2020-01-16T00:14:16.613 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017766311999999999 2020-01-16T00:14:16.613 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.613 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.613 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.613 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.079380+0000", 2020-01-16T00:14:16.614 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000133144 2020-01-16T00:14:16.614 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.614 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.614 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:16.614 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.079513+0000", 2020-01-16T00:14:16.614 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021271500000000001 2020-01-16T00:14:16.614 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.615 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.615 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:14:16.615 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.079725+0000", 2020-01-16T00:14:16.615 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001302582 2020-01-16T00:14:16.616 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.616 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.616 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:16.616 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081028+0000", 2020-01-16T00:14:16.616 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017197 2020-01-16T00:14:16.616 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.616 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.616 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:16.617 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081200+0000", 2020-01-16T00:14:16.617 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000202757 2020-01-16T00:14:16.617 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.617 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.617 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:16.617 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081403+0000", 2020-01-16T00:14:16.618 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1068999999999993e-05 2020-01-16T00:14:16.618 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.618 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.618 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:16.618 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081474+0000", 2020-01-16T00:14:16.618 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7800000000000006e-05 2020-01-16T00:14:16.618 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.619 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.619 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:16.619 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081562+0000", 2020-01-16T00:14:16.619 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021977219999999999 2020-01-16T00:14:16.619 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.619 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:16.620 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.620 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.620 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.620 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3823 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e50)", 2020-01-16T00:14:16.620 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:44.059639+0000", 2020-01-16T00:14:16.620 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 32.395563414000002, 2020-01-16T00:14:16.620 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022245396000000001, 2020-01-16T00:14:16.621 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:16.621 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:16.621 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:16.621 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:16.621 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:16.621 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3823 2020-01-16T00:14:16.622 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.622 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:16.622 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.622 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:16.622 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059639+0000", 2020-01-16T00:14:16.622 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:16.622 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.622 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.623 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:16.623 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059639+0000", 2020-01-16T00:14:16.623 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:16.623 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.623 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.623 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:16.623 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059638+0000", 2020-01-16T00:14:16.623 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.405e-06 2020-01-16T00:14:16.624 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.624 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.624 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:16.624 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059641+0000", 2020-01-16T00:14:16.625 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9770000000000003e-06 2020-01-16T00:14:16.625 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.625 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.625 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:16.626 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059648+0000", 2020-01-16T00:14:16.626 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1140000000000002e-06 2020-01-16T00:14:16.626 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.626 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.627 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:16.627 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059652+0000", 2020-01-16T00:14:16.627 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020063659999999999 2020-01-16T00:14:16.627 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.627 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.628 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.628 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.061658+0000", 2020-01-16T00:14:16.628 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3183999999999999e-05 2020-01-16T00:14:16.628 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.628 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.629 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:16.629 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.061691+0000", 2020-01-16T00:14:16.629 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018817654999999999 2020-01-16T00:14:16.629 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.629 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.630 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.630 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.080509+0000", 2020-01-16T00:14:16.630 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000104409 2020-01-16T00:14:16.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.630 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.631 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:16.631 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.080613+0000", 2020-01-16T00:14:16.631 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00098755999999999996 2020-01-16T00:14:16.631 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.631 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.632 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.632 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081601+0000", 2020-01-16T00:14:16.632 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000145766 2020-01-16T00:14:16.632 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.633 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.633 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:16.633 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081746+0000", 2020-01-16T00:14:16.633 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013751799999999999 2020-01-16T00:14:16.634 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.634 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.634 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:16.634 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081884+0000", 2020-01-16T00:14:16.635 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022245396000000001 2020-01-16T00:14:16.635 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.635 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:16.635 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.635 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.636 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.636 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4398 3.1 3:8de1d8a3:::smithi12313934-13:head [write 2812170~507386] snapc 0=[] ondisk+write+known_if_redirected e55)", 2020-01-16T00:14:16.636 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:58.336543+0000", 2020-01-16T00:14:16.636 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 18.118658980999999, 2020-01-16T00:14:16.636 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015697994, 2020-01-16T00:14:16.637 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:16.637 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:16.637 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:16.637 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:16.638 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:16.638 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4398 2020-01-16T00:14:16.638 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.638 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:16.639 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.639 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:16.639 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336543+0000", 2020-01-16T00:14:16.639 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:16.639 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.640 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.640 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:16.640 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336543+0000", 2020-01-16T00:14:16.640 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:16.641 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.641 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.641 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:16.641 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336542+0000", 2020-01-16T00:14:16.643 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014254 2020-01-16T00:14:16.643 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.643 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.643 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:16.643 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336684+0000", 2020-01-16T00:14:16.644 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2019e-05 2020-01-16T00:14:16.644 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.644 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.644 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:16.644 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336696+0000", 2020-01-16T00:14:16.644 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4269999999999996e-06 2020-01-16T00:14:16.644 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.645 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.645 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:16.645 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336702+0000", 2020-01-16T00:14:16.645 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8040000000000002e-05 2020-01-16T00:14:16.645 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.645 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.645 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.646 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336741+0000", 2020-01-16T00:14:16.646 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00030631400000000002 2020-01-16T00:14:16.646 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.646 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.646 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:16.646 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.337047+0000", 2020-01-16T00:14:16.646 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012249832 2020-01-16T00:14:16.646 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.647 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.647 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.647 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.349297+0000", 2020-01-16T00:14:16.647 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013870899999999999 2020-01-16T00:14:16.647 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.647 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.647 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:16.647 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.349435+0000", 2020-01-16T00:14:16.648 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021913600000000001 2020-01-16T00:14:16.648 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.648 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.648 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:14:16.648 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.349655+0000", 2020-01-16T00:14:16.648 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021982490000000002 2020-01-16T00:14:16.648 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.649 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.649 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:16.649 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351853+0000", 2020-01-16T00:14:16.649 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8295999999999996e-05 2020-01-16T00:14:16.649 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.649 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.649 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:16.649 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351941+0000", 2020-01-16T00:14:16.650 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021662 2020-01-16T00:14:16.650 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.650 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.650 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:16.650 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352158+0000", 2020-01-16T00:14:16.650 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6031000000000001e-05 2020-01-16T00:14:16.650 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.650 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.651 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:16.651 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352194+0000", 2020-01-16T00:14:16.651 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7290000000000003e-05 2020-01-16T00:14:16.651 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.651 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.651 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:16.651 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352241+0000", 2020-01-16T00:14:16.651 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015697994 2020-01-16T00:14:16.652 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.652 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:16.652 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.652 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.652 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.652 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4399 3.1 3:8de1d8a3:::smithi12313934-13:head [setxattr _header (56) in=7b,truncate 3411920] snapc 0=[] ondisk+write+known_if_redirected e55)", 2020-01-16T00:14:16.652 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:58.336714+0000", 2020-01-16T00:14:16.653 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 18.118487624, 2020-01-16T00:14:16.653 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016051421, 2020-01-16T00:14:16.653 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:16.653 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:16.653 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:16.654 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:16.654 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:16.654 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4399 2020-01-16T00:14:16.654 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.654 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:16.654 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.655 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:16.655 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336714+0000", 2020-01-16T00:14:16.655 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:16.655 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.655 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.655 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:16.655 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336714+0000", 2020-01-16T00:14:16.655 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:16.656 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.656 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.656 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:16.656 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336714+0000", 2020-01-16T00:14:16.656 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7540000000000003e-06 2020-01-16T00:14:16.656 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.656 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.656 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:16.657 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336719+0000", 2020-01-16T00:14:16.657 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2879999999999998e-06 2020-01-16T00:14:16.657 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.657 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.657 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:16.657 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336726+0000", 2020-01-16T00:14:16.658 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6399999999999996e-06 2020-01-16T00:14:16.658 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.658 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.658 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:16.658 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336730+0000", 2020-01-16T00:14:16.658 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00036299699999999999 2020-01-16T00:14:16.659 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.659 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.659 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.659 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.337093+0000", 2020-01-16T00:14:16.659 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1297999999999994e-05 2020-01-16T00:14:16.659 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.659 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.660 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:16.660 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.337165+0000", 2020-01-16T00:14:16.660 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013848911 2020-01-16T00:14:16.660 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.660 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.660 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.660 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351013+0000", 2020-01-16T00:14:16.660 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7732999999999994e-05 2020-01-16T00:14:16.661 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.661 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.661 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:16.661 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351111+0000", 2020-01-16T00:14:16.661 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000109089 2020-01-16T00:14:16.661 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.661 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.662 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:14:16.662 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351220+0000", 2020-01-16T00:14:16.662 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00063960199999999997 2020-01-16T00:14:16.662 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.662 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.662 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:16.662 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351860+0000", 2020-01-16T00:14:16.662 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00050205699999999998 2020-01-16T00:14:16.663 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.663 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.663 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:16.663 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352362+0000", 2020-01-16T00:14:16.663 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000282195 2020-01-16T00:14:16.664 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.664 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.664 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:16.664 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352644+0000", 2020-01-16T00:14:16.664 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3927000000000001e-05 2020-01-16T00:14:16.664 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.664 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.664 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:16.665 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352698+0000", 2020-01-16T00:14:16.665 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7770999999999999e-05 2020-01-16T00:14:16.665 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.665 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.665 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:16.665 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352766+0000", 2020-01-16T00:14:16.665 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016051421 2020-01-16T00:14:16.666 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.666 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:16.666 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.666 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.666 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.666 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4400 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e55)", 2020-01-16T00:14:16.666 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:58.336738+0000", 2020-01-16T00:14:16.666 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 18.118463895000001, 2020-01-16T00:14:16.667 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016207452000000001, 2020-01-16T00:14:16.667 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:16.667 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:16.667 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:16.667 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:16.667 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:16.667 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4400 2020-01-16T00:14:16.667 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.668 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:16.668 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.668 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:16.668 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336738+0000", 2020-01-16T00:14:16.668 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:16.668 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.668 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.669 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:16.669 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336738+0000", 2020-01-16T00:14:16.669 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:16.669 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.669 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.669 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:16.669 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336738+0000", 2020-01-16T00:14:16.670 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5200000000000002e-06 2020-01-16T00:14:16.670 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.670 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.670 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:16.670 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336741+0000", 2020-01-16T00:14:16.670 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1500000000000004e-06 2020-01-16T00:14:16.670 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.671 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.671 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:16.671 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336747+0000", 2020-01-16T00:14:16.671 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7500000000000001e-06 2020-01-16T00:14:16.672 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.672 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.672 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:16.672 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336751+0000", 2020-01-16T00:14:16.672 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043512299999999999 2020-01-16T00:14:16.673 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.673 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.673 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.673 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.337186+0000", 2020-01-16T00:14:16.673 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7048000000000002e-05 2020-01-16T00:14:16.673 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.673 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.674 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:16.674 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.337223+0000", 2020-01-16T00:14:16.674 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014258785 2020-01-16T00:14:16.674 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.674 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.674 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.674 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351482+0000", 2020-01-16T00:14:16.675 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012002000000000001 2020-01-16T00:14:16.675 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.675 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.675 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:16.675 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351602+0000", 2020-01-16T00:14:16.675 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001192404 2020-01-16T00:14:16.675 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.675 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.676 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.676 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352794+0000", 2020-01-16T00:14:16.676 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8901999999999998e-05 2020-01-16T00:14:16.676 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.676 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.676 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:16.676 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352873+0000", 2020-01-16T00:14:16.676 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2317999999999994e-05 2020-01-16T00:14:16.677 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.677 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.677 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:16.677 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352946+0000", 2020-01-16T00:14:16.677 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016207452000000001 2020-01-16T00:14:16.677 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.677 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:16.678 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.678 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.678 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.678 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:183 2.1 e58/19 2:a4df40e2:::smithi12313934-42 low_tier:head v 58'138, mlcod=57'136)", 2020-01-16T00:14:16.678 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.021967+0000", 2020-01-16T00:14:16.679 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 9.4332350260000002, 2020-01-16T00:14:16.679 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011697308999999999, 2020-01-16T00:14:16.679 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:16.679 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:16.679 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:16.679 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.679 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:16.679 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021967+0000", 2020-01-16T00:14:16.680 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:16.680 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.680 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.680 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:16.680 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021967+0000", 2020-01-16T00:14:16.680 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:16.680 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.680 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.681 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:16.681 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021966+0000", 2020-01-16T00:14:16.681 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020837899999999999 2020-01-16T00:14:16.681 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.681 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.681 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:16.681 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.022175+0000", 2020-01-16T00:14:16.682 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7650000000000005e-06 2020-01-16T00:14:16.683 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.683 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.683 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:16.683 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.022185+0000", 2020-01-16T00:14:16.683 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.759e-06 2020-01-16T00:14:16.683 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.684 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.684 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:16.684 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.022191+0000", 2020-01-16T00:14:16.684 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00559108 2020-01-16T00:14:16.684 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.684 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.684 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.684 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.027782+0000", 2020-01-16T00:14:16.685 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5761e-05 2020-01-16T00:14:16.685 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.685 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.685 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:16.685 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.027798+0000", 2020-01-16T00:14:16.685 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0058039989999999998 2020-01-16T00:14:16.685 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.686 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.686 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:16.686 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.033602+0000", 2020-01-16T00:14:16.686 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2147e-05 2020-01-16T00:14:16.686 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.686 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.687 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:16.687 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.033664+0000", 2020-01-16T00:14:16.688 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011697308999999999 2020-01-16T00:14:16.690 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.690 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:16.690 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.690 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.691 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.691 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.1.13:98 2.1 e58/19 2:bafac7f5:::smithi12313934-46 low_tier:head v 58'140, mlcod=58'138)", 2020-01-16T00:14:16.691 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.050024+0000", 2020-01-16T00:14:16.691 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 9.4051776680000003, 2020-01-16T00:14:16.692 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011307467, 2020-01-16T00:14:16.692 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:16.692 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:16.692 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:16.692 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.693 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:16.693 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.050024+0000", 2020-01-16T00:14:16.693 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:16.693 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.694 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.694 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:16.694 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.050024+0000", 2020-01-16T00:14:16.694 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:16.694 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.695 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.695 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:16.697 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.050022+0000", 2020-01-16T00:14:16.697 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000438839 2020-01-16T00:14:16.697 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.698 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.698 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:16.698 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.050461+0000", 2020-01-16T00:14:16.699 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9307999999999999e-05 2020-01-16T00:14:16.699 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.699 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.700 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:16.700 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.050480+0000", 2020-01-16T00:14:16.700 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7249999999999997e-06 2020-01-16T00:14:16.700 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.700 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.701 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:16.701 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.050488+0000", 2020-01-16T00:14:16.701 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00026342100000000001 2020-01-16T00:14:16.701 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.701 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.701 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:16.701 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.050751+0000", 2020-01-16T00:14:16.702 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8136000000000001e-05 2020-01-16T00:14:16.702 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.702 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.702 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:16.702 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.050769+0000", 2020-01-16T00:14:16.702 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010518582 2020-01-16T00:14:16.702 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.702 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.703 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:16.703 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.061288+0000", 2020-01-16T00:14:16.703 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4085000000000002e-05 2020-01-16T00:14:16.703 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:16.703 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:16.703 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:16.703 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.061332+0000", 2020-01-16T00:14:16.704 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011307467 2020-01-16T00:14:16.704 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.704 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:16.704 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.704 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:16.704 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:16.704 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:16.707 INFO:tasks.rados.rados.0.smithi123.stdout:1783: finishing write tid 1 to smithi12313934-28 2020-01-16T00:14:16.707 INFO:tasks.rados.rados.0.smithi123.stdout:1783: finishing write tid 2 to smithi12313934-28 2020-01-16T00:14:16.707 INFO:tasks.rados.rados.0.smithi123.stdout:1783: finishing write tid 3 to smithi12313934-28 2020-01-16T00:14:16.707 INFO:tasks.rados.rados.0.smithi123.stdout:1783: finishing write tid 5 to smithi12313934-28 2020-01-16T00:14:16.707 INFO:tasks.rados.rados.0.smithi123.stdout:1783: finishing write tid 6 to smithi12313934-28 2020-01-16T00:14:16.708 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 28 v 232 (ObjNum 652 snap 0 seq_num 652) dirty exists 2020-01-16T00:14:16.708 INFO:tasks.rados.rados.0.smithi123.stdout:1783: left oid 28 (ObjNum 652 snap 0 seq_num 652) 2020-01-16T00:14:16.709 INFO:tasks.rados.rados.0.smithi123.stdout:1784: expect (ObjNum 638 snap 0 seq_num 638) 2020-01-16T00:14:16.709 INFO:tasks.rados.rados.0.smithi123.stdout:1785: finishing write tid 1 to smithi12313934-3 2020-01-16T00:14:16.709 INFO:tasks.rados.rados.0.smithi123.stdout:1785: finishing write tid 2 to smithi12313934-3 2020-01-16T00:14:16.709 INFO:tasks.rados.rados.0.smithi123.stdout:1785: finishing write tid 4 to smithi12313934-3 2020-01-16T00:14:16.709 INFO:tasks.rados.rados.0.smithi123.stdout:1785: finishing write tid 5 to smithi12313934-3 2020-01-16T00:14:16.709 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 3 v 123 (ObjNum 653 snap 0 seq_num 653) dirty exists 2020-01-16T00:14:16.710 INFO:tasks.rados.rados.0.smithi123.stdout:1785: left oid 3 (ObjNum 653 snap 0 seq_num 653) 2020-01-16T00:14:16.710 INFO:tasks.rados.rados.0.smithi123.stdout:1787: finishing write tid 1 to smithi12313934-49 2020-01-16T00:14:16.710 INFO:tasks.rados.rados.0.smithi123.stdout:1788: expect (ObjNum 634 snap 0 seq_num 634) 2020-01-16T00:14:16.710 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:16.702+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:16.710 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-01-16T00:14:16.812 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:16.810+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:16.856 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:16.857 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:14:16.857 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:14:16.857 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:16.865 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-01-16T00:14:16.913 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:16.912+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:16.937 INFO:tasks.rados.rados.0.smithi123.stdout:1790: expect (ObjNum 599 snap 0 seq_num 599) 2020-01-16T00:14:17.007 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:17.008 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:14:17.008 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:14:17.008 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:14:17.009 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:17.014 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:17.010+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:17.016 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-01-16T00:14:17.114 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:17.114+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:17.142 INFO:tasks.rados.rados.0.smithi123.stdout:1787: finishing write tid 2 to smithi12313934-49 2020-01-16T00:14:17.142 INFO:tasks.rados.rados.0.smithi123.stdout:1787: finishing write tid 3 to smithi12313934-49 2020-01-16T00:14:17.142 INFO:tasks.rados.rados.0.smithi123.stdout:1791: finishing write tid 1 to smithi12313934-12 2020-01-16T00:14:17.142 INFO:tasks.rados.rados.0.smithi123.stdout:1787: finishing write tid 5 to smithi12313934-49 2020-01-16T00:14:17.143 INFO:tasks.rados.rados.0.smithi123.stdout:1781: done (15 left) 2020-01-16T00:14:17.143 INFO:tasks.rados.rados.0.smithi123.stdout:1782: done (14 left) 2020-01-16T00:14:17.143 INFO:tasks.rados.rados.0.smithi123.stdout:1783: done (13 left) 2020-01-16T00:14:17.143 INFO:tasks.rados.rados.0.smithi123.stdout:1784: done (12 left) 2020-01-16T00:14:17.143 INFO:tasks.rados.rados.0.smithi123.stdout:1785: done (11 left) 2020-01-16T00:14:17.144 INFO:tasks.rados.rados.0.smithi123.stdout:1786: done (10 left) 2020-01-16T00:14:17.144 INFO:tasks.rados.rados.0.smithi123.stdout:1788: done (9 left) 2020-01-16T00:14:17.144 INFO:tasks.rados.rados.0.smithi123.stdout:1789: done (8 left) 2020-01-16T00:14:17.144 INFO:tasks.rados.rados.0.smithi123.stdout:1790: done (7 left) 2020-01-16T00:14:17.144 INFO:tasks.rados.rados.0.smithi123.stdout:1797: write (excl) oid 29 current snap is 0 2020-01-16T00:14:17.144 INFO:tasks.rados.rados.0.smithi123.stdout:1797: seq_num 659 ranges {675813=795449,2096810=609228} 2020-01-16T00:14:17.150 INFO:tasks.rados.rados.0.smithi123.stdout:1797: writing smithi12313934-29 from 675813 to 1471262 tid 1 2020-01-16T00:14:17.159 INFO:tasks.rados.rados.0.smithi123.stdout:1797: writing smithi12313934-29 from 2096810 to 2706038 tid 2 2020-01-16T00:14:17.159 INFO:tasks.rados.rados.0.smithi123.stdout:1798: write oid 16 current snap is 0 2020-01-16T00:14:17.159 INFO:tasks.rados.rados.0.smithi123.stdout:1787: finishing write tid 6 to smithi12313934-49 2020-01-16T00:14:17.160 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 49 v 186 (ObjNum 654 snap 0 seq_num 654) dirty exists 2020-01-16T00:14:17.160 INFO:tasks.rados.rados.0.smithi123.stdout:1787: left oid 49 (ObjNum 654 snap 0 seq_num 654) 2020-01-16T00:14:17.160 INFO:tasks.rados.rados.0.smithi123.stdout:1798: seq_num 660 ranges {692620=470698,1755292=752886} 2020-01-16T00:14:17.165 INFO:tasks.rados.rados.0.smithi123.stdout:1798: writing smithi12313934-16 from 692620 to 1163318 tid 1 2020-01-16T00:14:17.175 INFO:tasks.rados.rados.0.smithi123.stdout:1798: writing smithi12313934-16 from 1755292 to 2508178 tid 2 2020-01-16T00:14:17.175 INFO:tasks.rados.rados.0.smithi123.stdout:1787: done (8 left) 2020-01-16T00:14:17.176 INFO:tasks.rados.rados.0.smithi123.stdout:1799: read oid 39 snap -1 2020-01-16T00:14:17.176 INFO:tasks.rados.rados.0.smithi123.stdout:1799: expect (ObjNum 535 snap 0 seq_num 535) 2020-01-16T00:14:17.176 INFO:tasks.rados.rados.0.smithi123.stdout:1800: read oid 40 snap -1 2020-01-16T00:14:17.176 INFO:tasks.rados.rados.0.smithi123.stdout:1800: expect (ObjNum 638 snap 0 seq_num 638) 2020-01-16T00:14:17.176 INFO:tasks.rados.rados.0.smithi123.stdout:1801: write (excl) oid 22 current snap is 0 2020-01-16T00:14:17.176 INFO:tasks.rados.rados.0.smithi123.stdout:1801: seq_num 661 ranges {709427=545947,1813774=612897,2985657=123770} 2020-01-16T00:14:17.176 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:17.177 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:14:17.177 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:14:17.177 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:14:17.177 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.178 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2226 3.7 3:e1be6d3b:::smithi12313934-24:head [setxattr _header (56) in=7b,truncate 2100908] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:14:17.178 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.980217+0000", 2020-01-16T00:14:17.178 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 71.192901847000002, 2020-01-16T00:14:17.178 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023113018999999999, 2020-01-16T00:14:17.178 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:17.178 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:17.179 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:17.179 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:17.179 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:17.179 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2226 2020-01-16T00:14:17.179 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.179 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:17.179 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.179 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:17.180 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980217+0000", 2020-01-16T00:14:17.180 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:17.180 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.180 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.180 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:17.180 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980217+0000", 2020-01-16T00:14:17.180 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:17.180 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.181 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.181 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:17.181 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980216+0000", 2020-01-16T00:14:17.181 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5049999999999998e-06 2020-01-16T00:14:17.181 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.181 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.182 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:17.182 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980222+0000", 2020-01-16T00:14:17.182 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5654999999999998e-05 2020-01-16T00:14:17.182 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.182 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.182 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:17.182 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980237+0000", 2020-01-16T00:14:17.182 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9740000000000004e-06 2020-01-16T00:14:17.183 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.183 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.183 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:17.183 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980241+0000", 2020-01-16T00:14:17.183 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002947776 2020-01-16T00:14:17.183 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.183 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.183 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.184 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.983189+0000", 2020-01-16T00:14:17.184 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0027099400000000001 2020-01-16T00:14:17.184 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.184 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.184 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:17.184 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985899+0000", 2020-01-16T00:14:17.184 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014275589999999999 2020-01-16T00:14:17.184 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.185 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.185 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.185 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.000175+0000", 2020-01-16T00:14:17.185 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011211 2020-01-16T00:14:17.185 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.185 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.185 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:17.186 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.000287+0000", 2020-01-16T00:14:17.186 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000229886 2020-01-16T00:14:17.186 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.186 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.186 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,7", 2020-01-16T00:14:17.186 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.000517+0000", 2020-01-16T00:14:17.186 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00095715399999999999 2020-01-16T00:14:17.186 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.187 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.187 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:17.187 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.001474+0000", 2020-01-16T00:14:17.187 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001341512 2020-01-16T00:14:17.187 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.187 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.187 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:17.190 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.002815+0000", 2020-01-16T00:14:17.190 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035789099999999999 2020-01-16T00:14:17.191 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.191 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.191 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:17.191 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003173+0000", 2020-01-16T00:14:17.191 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8571000000000005e-05 2020-01-16T00:14:17.191 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.191 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.191 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:17.192 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003242+0000", 2020-01-16T00:14:17.192 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8293000000000003e-05 2020-01-16T00:14:17.192 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.192 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.192 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:17.192 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003330+0000", 2020-01-16T00:14:17.192 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023113018999999999 2020-01-16T00:14:17.192 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.193 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:17.193 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.193 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.193 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.194 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2227 3.7 3:e1be6d3b:::smithi12313934-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e35)", 2020-01-16T00:14:17.194 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:14:17.194 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 71.192854593999996, 2020-01-16T00:14:17.194 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023303356000000001, 2020-01-16T00:14:17.194 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:17.195 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:17.195 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:17.195 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:17.195 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:17.195 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2227 2020-01-16T00:14:17.195 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.195 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:17.195 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.196 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:17.196 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:14:17.196 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:17.196 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.196 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.196 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:17.196 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:14:17.196 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:17.197 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.197 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.197 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:17.197 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:14:17.197 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3079999999999999e-06 2020-01-16T00:14:17.197 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.197 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.198 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:17.198 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980267+0000", 2020-01-16T00:14:17.199 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9290000000000001e-06 2020-01-16T00:14:17.199 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.199 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.199 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:17.200 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980274+0000", 2020-01-16T00:14:17.200 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7400000000000002e-06 2020-01-16T00:14:17.201 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.201 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.201 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:17.201 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980278+0000", 2020-01-16T00:14:17.202 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056624300000000004 2020-01-16T00:14:17.202 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.202 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.202 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.202 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985940+0000", 2020-01-16T00:14:17.202 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4261000000000002e-05 2020-01-16T00:14:17.202 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.203 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.203 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:17.203 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985985+0000", 2020-01-16T00:14:17.203 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015087676 2020-01-16T00:14:17.203 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.203 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.203 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.204 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.001072+0000", 2020-01-16T00:14:17.204 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011960600000000001 2020-01-16T00:14:17.204 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.204 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.204 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:17.204 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.001192+0000", 2020-01-16T00:14:17.204 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021718309999999999 2020-01-16T00:14:17.205 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.205 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.205 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.205 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003364+0000", 2020-01-16T00:14:17.205 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000102855 2020-01-16T00:14:17.205 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.205 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.205 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:17.206 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003467+0000", 2020-01-16T00:14:17.206 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010122499999999999 2020-01-16T00:14:17.206 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.207 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.207 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:17.207 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003568+0000", 2020-01-16T00:14:17.207 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023303356000000001 2020-01-16T00:14:17.207 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.207 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:17.207 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.207 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.208 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.208 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2325 3.7 3:e1be6d3b:::smithi12313934-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e36)", 2020-01-16T00:14:17.209 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.550877+0000", 2020-01-16T00:14:17.209 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 69.622242267000004, 2020-01-16T00:14:17.209 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018565154, 2020-01-16T00:14:17.209 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:17.209 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:17.209 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:17.209 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:17.210 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:17.210 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2325 2020-01-16T00:14:17.210 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.210 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:17.219 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.219 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:17.219 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550877+0000", 2020-01-16T00:14:17.219 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:17.219 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.219 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.220 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:17.220 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550877+0000", 2020-01-16T00:14:17.220 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:17.220 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.220 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.220 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:17.220 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550876+0000", 2020-01-16T00:14:17.220 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1080000000000003e-06 2020-01-16T00:14:17.221 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.221 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.221 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:17.221 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550880+0000", 2020-01-16T00:14:17.221 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3269999999999997e-06 2020-01-16T00:14:17.221 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.221 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.222 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:17.222 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550886+0000", 2020-01-16T00:14:17.222 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.386e-06 2020-01-16T00:14:17.222 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.222 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.222 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:17.223 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550891+0000", 2020-01-16T00:14:17.223 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048231410000000004 2020-01-16T00:14:17.223 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.223 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.223 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.223 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555714+0000", 2020-01-16T00:14:17.223 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0071999999999999e-05 2020-01-16T00:14:17.223 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.224 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.224 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:17.224 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555744+0000", 2020-01-16T00:14:17.224 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012329005000000001 2020-01-16T00:14:17.224 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.224 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.224 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.224 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568073+0000", 2020-01-16T00:14:17.225 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9087999999999997e-05 2020-01-16T00:14:17.225 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.225 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.225 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:17.225 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568172+0000", 2020-01-16T00:14:17.225 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010718220000000001 2020-01-16T00:14:17.225 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.226 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.226 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.226 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569244+0000", 2020-01-16T00:14:17.226 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8199e-05 2020-01-16T00:14:17.226 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.226 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.226 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:17.226 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569342+0000", 2020-01-16T00:14:17.227 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9708000000000001e-05 2020-01-16T00:14:17.227 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.227 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.230 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:17.230 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569442+0000", 2020-01-16T00:14:17.231 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018565154 2020-01-16T00:14:17.231 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.231 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:17.231 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.231 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.231 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.231 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2800 3.2 3:42b7f20f:::smithi12313934-42:head [write 1395221~583781] snapc 0=[] ondisk+write+known_if_redirected e41)", 2020-01-16T00:14:17.232 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:17.256182+0000", 2020-01-16T00:14:17.232 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 59.916936727, 2020-01-16T00:14:17.232 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018509650999999998, 2020-01-16T00:14:17.232 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:17.232 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:17.232 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:17.233 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:17.233 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:17.233 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2800 2020-01-16T00:14:17.233 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.233 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:17.233 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.234 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:17.234 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256182+0000", 2020-01-16T00:14:17.234 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:17.234 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.234 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.234 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:17.235 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256182+0000", 2020-01-16T00:14:17.235 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:17.235 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.235 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.235 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:17.235 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256180+0000", 2020-01-16T00:14:17.235 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020514799999999999 2020-01-16T00:14:17.236 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.236 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.236 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:17.236 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256385+0000", 2020-01-16T00:14:17.236 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0288e-05 2020-01-16T00:14:17.236 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.237 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.237 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:17.237 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256406+0000", 2020-01-16T00:14:17.237 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7160000000000003e-06 2020-01-16T00:14:17.237 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.237 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.238 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:17.238 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256413+0000", 2020-01-16T00:14:17.238 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5293999999999998e-05 2020-01-16T00:14:17.238 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.238 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.238 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.239 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256449+0000", 2020-01-16T00:14:17.239 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001297737 2020-01-16T00:14:17.239 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.239 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.239 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:17.239 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.257746+0000", 2020-01-16T00:14:17.239 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0079866529999999998 2020-01-16T00:14:17.240 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.240 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.240 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.240 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.265733+0000", 2020-01-16T00:14:17.241 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4028000000000004e-05 2020-01-16T00:14:17.241 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.241 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.241 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:17.241 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.265797+0000", 2020-01-16T00:14:17.241 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011338899999999999 2020-01-16T00:14:17.241 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.242 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.242 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:17.242 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.265910+0000", 2020-01-16T00:14:17.242 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003372496 2020-01-16T00:14:17.242 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.242 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.243 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:17.243 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.269283+0000", 2020-01-16T00:14:17.243 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0053107959999999996 2020-01-16T00:14:17.243 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.243 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.243 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:17.243 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.274594+0000", 2020-01-16T00:14:17.244 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8000999999999998e-05 2020-01-16T00:14:17.244 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.244 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.244 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:17.244 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.274632+0000", 2020-01-16T00:14:17.244 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0265e-05 2020-01-16T00:14:17.244 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.245 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.245 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:17.245 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.274692+0000", 2020-01-16T00:14:17.245 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018509650999999998 2020-01-16T00:14:17.245 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.245 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:17.245 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.246 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.246 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.246 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2973 3.c 3:3025546b:::smithi12313934-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e42)", 2020-01-16T00:14:17.246 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:21.839961+0000", 2020-01-16T00:14:17.246 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 55.333158126000001, 2020-01-16T00:14:17.246 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018691670000000001, 2020-01-16T00:14:17.246 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:17.247 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:17.247 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:17.247 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:17.247 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:17.247 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2973 2020-01-16T00:14:17.247 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.248 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:17.248 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.248 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:17.248 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839961+0000", 2020-01-16T00:14:17.248 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:17.248 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.249 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.249 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:17.249 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839961+0000", 2020-01-16T00:14:17.249 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:17.249 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.250 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.250 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:17.250 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839960+0000", 2020-01-16T00:14:17.250 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4060000000000001e-06 2020-01-16T00:14:17.250 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.250 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.251 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:17.251 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839964+0000", 2020-01-16T00:14:17.251 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5412000000000001e-05 2020-01-16T00:14:17.251 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.251 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.252 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:17.252 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839979+0000", 2020-01-16T00:14:17.252 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.444e-06 2020-01-16T00:14:17.252 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.252 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.252 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:17.253 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839983+0000", 2020-01-16T00:14:17.253 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000430028 2020-01-16T00:14:17.253 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.253 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.253 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.253 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.840413+0000", 2020-01-16T00:14:17.253 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9270999999999997e-05 2020-01-16T00:14:17.254 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.254 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.254 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:17.254 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.840452+0000", 2020-01-16T00:14:17.254 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014853975 2020-01-16T00:14:17.254 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.254 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.255 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.255 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.855306+0000", 2020-01-16T00:14:17.255 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011098400000000001 2020-01-16T00:14:17.255 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.255 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.255 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:17.256 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.855417+0000", 2020-01-16T00:14:17.256 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029108060000000002 2020-01-16T00:14:17.256 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.256 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.256 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.256 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858328+0000", 2020-01-16T00:14:17.256 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000144697 2020-01-16T00:14:17.257 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.257 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.257 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:17.257 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858472+0000", 2020-01-16T00:14:17.257 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018011899999999999 2020-01-16T00:14:17.257 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.257 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.258 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:17.258 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858653+0000", 2020-01-16T00:14:17.258 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018691670000000001 2020-01-16T00:14:17.258 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.258 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:17.258 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.259 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.259 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.259 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3296 3.2 3:42b7f20f:::smithi12313934-42:head [setxattr _header (56) in=7b,truncate 2739607] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:14:17.259 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:30.744532+0000", 2020-01-16T00:14:17.259 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 46.428587116999999, 2020-01-16T00:14:17.259 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018472374, 2020-01-16T00:14:17.259 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:17.260 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:17.260 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:17.260 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:17.260 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:17.260 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3296 2020-01-16T00:14:17.261 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.261 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:17.261 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.261 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:17.261 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744532+0000", 2020-01-16T00:14:17.262 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:17.262 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.262 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.262 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:17.262 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744532+0000", 2020-01-16T00:14:17.262 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:17.262 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.263 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.263 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:17.264 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744531+0000", 2020-01-16T00:14:17.264 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7059999999999993e-06 2020-01-16T00:14:17.264 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.264 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.264 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:17.264 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744540+0000", 2020-01-16T00:14:17.264 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8589999999999998e-06 2020-01-16T00:14:17.265 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.265 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.265 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:17.265 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744544+0000", 2020-01-16T00:14:17.265 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7609999999999999e-06 2020-01-16T00:14:17.265 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.265 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.266 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:17.266 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744546+0000", 2020-01-16T00:14:17.266 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039933670000000003 2020-01-16T00:14:17.266 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.266 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.266 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.266 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748540+0000", 2020-01-16T00:14:17.266 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000110281 2020-01-16T00:14:17.267 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.267 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.267 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:17.267 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748650+0000", 2020-01-16T00:14:17.267 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.01160457 2020-01-16T00:14:17.267 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.267 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.267 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.268 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.760255+0000", 2020-01-16T00:14:17.268 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000124714 2020-01-16T00:14:17.268 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.268 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.268 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:17.268 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.760379+0000", 2020-01-16T00:14:17.268 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022099700000000001 2020-01-16T00:14:17.268 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.269 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.269 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:17.269 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.760600+0000", 2020-01-16T00:14:17.269 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00057843300000000005 2020-01-16T00:14:17.269 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.269 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.269 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:17.270 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.761179+0000", 2020-01-16T00:14:17.270 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00161477 2020-01-16T00:14:17.270 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.270 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.270 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:17.270 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.762794+0000", 2020-01-16T00:14:17.270 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4408999999999999e-05 2020-01-16T00:14:17.271 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.271 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.271 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:17.271 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.762858+0000", 2020-01-16T00:14:17.271 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014634199999999999 2020-01-16T00:14:17.271 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.271 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.271 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:17.272 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.763004+0000", 2020-01-16T00:14:17.272 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018472374 2020-01-16T00:14:17.272 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.272 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:17.272 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.272 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.272 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.272 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3297 3.2 3:42b7f20f:::smithi12313934-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e45)", 2020-01-16T00:14:17.273 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:30.744551+0000", 2020-01-16T00:14:17.273 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 46.428568147, 2020-01-16T00:14:17.273 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018660209000000001, 2020-01-16T00:14:17.273 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:17.273 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:17.273 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:17.273 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:17.274 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:17.274 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3297 2020-01-16T00:14:17.274 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.274 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:17.274 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.274 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:17.274 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744551+0000", 2020-01-16T00:14:17.274 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:17.275 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.275 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.275 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:17.275 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744551+0000", 2020-01-16T00:14:17.275 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:17.275 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.275 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.275 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:17.276 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744550+0000", 2020-01-16T00:14:17.276 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5279999999999999e-06 2020-01-16T00:14:17.276 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.276 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.276 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:17.276 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744554+0000", 2020-01-16T00:14:17.276 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8420000000000001e-06 2020-01-16T00:14:17.277 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.277 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.277 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:17.277 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744559+0000", 2020-01-16T00:14:17.277 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.965e-06 2020-01-16T00:14:17.277 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.277 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.277 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:17.278 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744562+0000", 2020-01-16T00:14:17.278 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041215380000000001 2020-01-16T00:14:17.278 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.278 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.278 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.278 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748684+0000", 2020-01-16T00:14:17.278 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8165999999999997e-05 2020-01-16T00:14:17.278 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.279 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.279 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:17.279 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748762+0000", 2020-01-16T00:14:17.279 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012136152000000001 2020-01-16T00:14:17.279 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.279 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.279 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.280 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.760898+0000", 2020-01-16T00:14:17.280 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000114833 2020-01-16T00:14:17.280 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.280 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.280 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:17.280 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.761013+0000", 2020-01-16T00:14:17.280 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002017015 2020-01-16T00:14:17.280 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.281 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.281 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.281 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.763030+0000", 2020-01-16T00:14:17.281 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5741000000000006e-05 2020-01-16T00:14:17.281 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.281 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.281 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:17.282 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.763096+0000", 2020-01-16T00:14:17.282 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011519900000000001 2020-01-16T00:14:17.282 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.282 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.282 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:17.282 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.763211+0000", 2020-01-16T00:14:17.282 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018660209000000001 2020-01-16T00:14:17.282 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.283 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:17.283 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.283 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.283 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.283 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3810 3.7 3:efab040c:::smithi12313934-30:head [write 1837406~542493] snapc 0=[] ondisk+write+known_if_redirected e50)", 2020-01-16T00:14:17.283 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:43.768525+0000", 2020-01-16T00:14:17.283 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 33.404593974000001, 2020-01-16T00:14:17.283 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018385839000000001, 2020-01-16T00:14:17.284 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:17.284 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:17.284 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:17.284 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:17.284 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:17.284 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3810 2020-01-16T00:14:17.284 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.284 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:17.285 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.285 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:17.285 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768525+0000", 2020-01-16T00:14:17.285 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:17.285 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.285 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.285 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:17.286 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768525+0000", 2020-01-16T00:14:17.286 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:17.286 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.286 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.286 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:17.286 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768523+0000", 2020-01-16T00:14:17.286 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000149512 2020-01-16T00:14:17.286 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.287 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.287 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:17.287 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768673+0000", 2020-01-16T00:14:17.287 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0732e-05 2020-01-16T00:14:17.287 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.287 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.287 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:17.287 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768684+0000", 2020-01-16T00:14:17.288 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3719999999999998e-06 2020-01-16T00:14:17.288 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.288 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.288 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:17.288 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768688+0000", 2020-01-16T00:14:17.288 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6513999999999994e-05 2020-01-16T00:14:17.288 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.288 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.289 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.289 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768754+0000", 2020-01-16T00:14:17.289 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00075867800000000004 2020-01-16T00:14:17.289 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.289 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.289 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:17.289 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.769513+0000", 2020-01-16T00:14:17.290 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012395692 2020-01-16T00:14:17.290 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.290 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.290 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.290 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.781909+0000", 2020-01-16T00:14:17.290 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118372 2020-01-16T00:14:17.290 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.290 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.291 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:17.291 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.782027+0000", 2020-01-16T00:14:17.291 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019909699999999999 2020-01-16T00:14:17.291 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.291 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.291 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,7", 2020-01-16T00:14:17.291 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.782226+0000", 2020-01-16T00:14:17.291 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0038778369999999999 2020-01-16T00:14:17.292 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.292 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.292 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:17.292 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786104+0000", 2020-01-16T00:14:17.292 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022391199999999999 2020-01-16T00:14:17.292 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.292 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.292 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:17.293 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786328+0000", 2020-01-16T00:14:17.293 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00048851299999999999 2020-01-16T00:14:17.293 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.293 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.293 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:17.293 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786817+0000", 2020-01-16T00:14:17.293 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3597999999999998e-05 2020-01-16T00:14:17.294 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.294 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.294 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:17.294 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786860+0000", 2020-01-16T00:14:17.294 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0741000000000001e-05 2020-01-16T00:14:17.294 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.294 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.294 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:17.295 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786911+0000", 2020-01-16T00:14:17.295 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018385839000000001 2020-01-16T00:14:17.295 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.295 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:17.295 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.295 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.295 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.295 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3863 3.2 3:4f8fe446:::smithi12313934-48:head [setxattr _header (56) in=7b,truncate 2151395] snapc 0=[] ondisk+write+known_if_redirected e50)", 2020-01-16T00:14:17.296 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:44.489284+0000", 2020-01-16T00:14:17.296 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 32.683835502000001, 2020-01-16T00:14:17.296 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.026323742000000001, 2020-01-16T00:14:17.296 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:17.296 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:17.296 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:17.296 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:17.296 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:17.297 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3863 2020-01-16T00:14:17.297 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.297 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:17.297 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.297 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:17.297 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489284+0000", 2020-01-16T00:14:17.297 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:17.298 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.298 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.298 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:17.298 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489284+0000", 2020-01-16T00:14:17.298 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:17.298 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.298 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.298 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:17.299 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489282+0000", 2020-01-16T00:14:17.299 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6660000000000002e-06 2020-01-16T00:14:17.299 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.299 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.299 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:17.299 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489289+0000", 2020-01-16T00:14:17.299 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3069999999999999e-06 2020-01-16T00:14:17.299 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.300 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.300 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:17.300 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489295+0000", 2020-01-16T00:14:17.300 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6090000000000003e-06 2020-01-16T00:14:17.300 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.300 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.300 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:17.300 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489300+0000", 2020-01-16T00:14:17.301 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002242464 2020-01-16T00:14:17.301 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.301 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.301 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.301 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491542+0000", 2020-01-16T00:14:17.301 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00059546200000000005 2020-01-16T00:14:17.301 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.301 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.302 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:17.302 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.492138+0000", 2020-01-16T00:14:17.302 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018547999999999999 2020-01-16T00:14:17.302 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.302 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.302 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.302 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.510686+0000", 2020-01-16T00:14:17.303 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018590100000000001 2020-01-16T00:14:17.303 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.303 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.303 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:17.303 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.510872+0000", 2020-01-16T00:14:17.303 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035115299999999999 2020-01-16T00:14:17.303 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.304 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.304 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:17.304 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.511223+0000", 2020-01-16T00:14:17.304 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003765493 2020-01-16T00:14:17.304 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.305 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.305 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:17.305 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.514988+0000", 2020-01-16T00:14:17.305 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000432022 2020-01-16T00:14:17.305 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.305 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.305 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:17.306 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515421+0000", 2020-01-16T00:14:17.306 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2874000000000002e-05 2020-01-16T00:14:17.306 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.306 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.306 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:17.306 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515473+0000", 2020-01-16T00:14:17.306 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013385 2020-01-16T00:14:17.306 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.307 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.307 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:17.307 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515607+0000", 2020-01-16T00:14:17.307 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.026323742000000001 2020-01-16T00:14:17.307 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.307 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:17.307 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.307 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.308 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.308 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3864 3.2 3:4f8fe446:::smithi12313934-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e50)", 2020-01-16T00:14:17.308 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:44.491131+0000", 2020-01-16T00:14:17.308 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 32.681988103999998, 2020-01-16T00:14:17.308 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024813806000000001, 2020-01-16T00:14:17.308 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:17.308 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:17.308 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:17.309 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:17.309 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:17.309 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3864 2020-01-16T00:14:17.309 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.309 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:17.309 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.309 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:17.310 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491131+0000", 2020-01-16T00:14:17.310 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:17.310 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.310 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.310 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:17.310 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491131+0000", 2020-01-16T00:14:17.310 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:17.311 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.311 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.311 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:17.311 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491128+0000", 2020-01-16T00:14:17.312 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2053e-05 2020-01-16T00:14:17.313 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.313 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.313 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:17.313 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491140+0000", 2020-01-16T00:14:17.313 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5869999999999999e-05 2020-01-16T00:14:17.313 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.313 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.314 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:17.314 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491156+0000", 2020-01-16T00:14:17.314 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2571e-05 2020-01-16T00:14:17.314 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.314 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.314 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:17.314 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491168+0000", 2020-01-16T00:14:17.314 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001031045 2020-01-16T00:14:17.315 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.315 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.315 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.315 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.492199+0000", 2020-01-16T00:14:17.315 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000170865 2020-01-16T00:14:17.315 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.315 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.315 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:17.316 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.492370+0000", 2020-01-16T00:14:17.316 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020180638000000001 2020-01-16T00:14:17.316 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.316 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.316 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.316 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.512551+0000", 2020-01-16T00:14:17.316 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012478599999999999 2020-01-16T00:14:17.316 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.317 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.317 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:17.317 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.512676+0000", 2020-01-16T00:14:17.317 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029628660000000002 2020-01-16T00:14:17.317 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.317 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.317 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.318 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515639+0000", 2020-01-16T00:14:17.318 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012250799999999999 2020-01-16T00:14:17.318 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.318 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.318 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:17.318 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515761+0000", 2020-01-16T00:14:17.318 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018356399999999999 2020-01-16T00:14:17.318 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.319 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.319 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:17.319 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515945+0000", 2020-01-16T00:14:17.319 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024813806000000001 2020-01-16T00:14:17.319 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.319 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:17.319 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.319 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.320 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.320 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3874 3.c 3:378f3722:::smithi12313934-50:head [write 568202~701253,stat] snapc 0=[] ondisk+write+known_if_redirected e50)", 2020-01-16T00:14:17.320 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:44.497041+0000", 2020-01-16T00:14:17.320 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 32.676078472999997, 2020-01-16T00:14:17.320 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019602525999999999, 2020-01-16T00:14:17.320 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:17.320 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:17.320 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:17.321 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:17.321 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:17.321 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3874 2020-01-16T00:14:17.321 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.321 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:17.321 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.321 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:17.322 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497041+0000", 2020-01-16T00:14:17.322 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:17.322 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.322 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.322 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:17.322 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497041+0000", 2020-01-16T00:14:17.322 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999962 2020-01-16T00:14:17.322 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.323 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.323 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:17.323 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497037+0000", 2020-01-16T00:14:17.323 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00027168899999999998 2020-01-16T00:14:17.323 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.323 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.323 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:17.323 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497308+0000", 2020-01-16T00:14:17.324 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4202999999999999e-05 2020-01-16T00:14:17.324 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.324 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.324 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:17.324 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497323+0000", 2020-01-16T00:14:17.324 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0280000000000003e-06 2020-01-16T00:14:17.324 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.324 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.325 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:17.325 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497330+0000", 2020-01-16T00:14:17.325 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010681352 2020-01-16T00:14:17.325 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.325 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.325 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.325 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.508011+0000", 2020-01-16T00:14:17.325 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5886999999999999e-05 2020-01-16T00:14:17.326 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.326 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.326 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:17.326 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.508097+0000", 2020-01-16T00:14:17.326 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023353600000000001 2020-01-16T00:14:17.326 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.326 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.327 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:17.327 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.508331+0000", 2020-01-16T00:14:17.327 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0075218339999999998 2020-01-16T00:14:17.327 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.327 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.327 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:17.327 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515852+0000", 2020-01-16T00:14:17.327 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00064178599999999998 2020-01-16T00:14:17.328 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.328 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.328 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:17.328 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.516494+0000", 2020-01-16T00:14:17.328 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6712999999999999e-05 2020-01-16T00:14:17.328 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.328 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.328 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:17.329 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.516531+0000", 2020-01-16T00:14:17.329 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112225 2020-01-16T00:14:17.329 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.329 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.329 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:17.329 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.516643+0000", 2020-01-16T00:14:17.329 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019602525999999999 2020-01-16T00:14:17.329 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.330 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:17.330 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.330 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.330 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.330 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4655 3.c 3:378f3722:::smithi12313934-50:head [write 1684786~315379] snapc 0=[] ondisk+write+known_if_redirected e57)", 2020-01-16T00:14:17.330 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:04.403811+0000", 2020-01-16T00:14:17.330 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 12.769307988, 2020-01-16T00:14:17.331 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022055851000000001, 2020-01-16T00:14:17.331 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:17.331 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:17.331 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:17.331 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:17.331 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:17.331 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4655 2020-01-16T00:14:17.331 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.332 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:17.332 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.332 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:17.332 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.403811+0000", 2020-01-16T00:14:17.332 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:17.332 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.332 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.333 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:17.333 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.403811+0000", 2020-01-16T00:14:17.333 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:17.333 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.333 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.333 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:17.333 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.403809+0000", 2020-01-16T00:14:17.333 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012429100000000001 2020-01-16T00:14:17.334 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.334 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.334 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:17.334 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.403934+0000", 2020-01-16T00:14:17.334 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3743e-05 2020-01-16T00:14:17.334 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.334 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.334 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:17.335 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.403947+0000", 2020-01-16T00:14:17.335 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.267e-06 2020-01-16T00:14:17.335 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.335 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.335 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:17.335 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.403955+0000", 2020-01-16T00:14:17.335 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1935999999999999e-05 2020-01-16T00:14:17.336 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.336 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.336 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.336 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404006+0000", 2020-01-16T00:14:17.336 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0060147769999999998 2020-01-16T00:14:17.336 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.336 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.336 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:17.337 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.410021+0000", 2020-01-16T00:14:17.337 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012889367000000001 2020-01-16T00:14:17.337 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.337 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.337 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.337 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.422911+0000", 2020-01-16T00:14:17.337 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012536700000000001 2020-01-16T00:14:17.337 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.338 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.338 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:17.338 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.423036+0000", 2020-01-16T00:14:17.338 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021238999999999999 2020-01-16T00:14:17.338 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.338 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.338 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:17.338 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.423248+0000", 2020-01-16T00:14:17.339 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0024175289999999999 2020-01-16T00:14:17.339 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.339 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.339 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:17.339 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.425666+0000", 2020-01-16T00:14:17.339 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7323e-05 2020-01-16T00:14:17.339 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.340 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.340 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:17.340 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.425723+0000", 2020-01-16T00:14:17.340 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0198999999999997e-05 2020-01-16T00:14:17.340 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.340 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.340 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:17.340 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.425783+0000", 2020-01-16T00:14:17.341 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3456999999999997e-05 2020-01-16T00:14:17.341 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.341 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.341 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:17.341 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.425867+0000", 2020-01-16T00:14:17.341 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022055851000000001 2020-01-16T00:14:17.341 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.341 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:17.342 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.342 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.342 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.342 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4656 3.c 3:378f3722:::smithi12313934-50:head [setxattr _header (56) in=7b,truncate 2000165] snapc 0=[] ondisk+write+known_if_redirected e57)", 2020-01-16T00:14:17.342 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:04.404295+0000", 2020-01-16T00:14:17.342 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 12.768823849, 2020-01-16T00:14:17.342 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021950859, 2020-01-16T00:14:17.342 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:17.343 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:17.343 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:17.343 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:17.343 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:17.343 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4656 2020-01-16T00:14:17.343 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.343 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:17.344 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.344 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:17.344 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404295+0000", 2020-01-16T00:14:17.344 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:17.344 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.344 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.344 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:17.344 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404295+0000", 2020-01-16T00:14:17.345 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:17.345 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.345 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.345 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:17.345 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404294+0000", 2020-01-16T00:14:17.345 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6139999999999997e-06 2020-01-16T00:14:17.345 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.345 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.346 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:17.346 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404301+0000", 2020-01-16T00:14:17.346 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1420000000000001e-05 2020-01-16T00:14:17.346 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.346 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.346 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:17.346 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404312+0000", 2020-01-16T00:14:17.346 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4840000000000003e-06 2020-01-16T00:14:17.347 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.347 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.347 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:17.347 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404318+0000", 2020-01-16T00:14:17.347 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0058040970000000002 2020-01-16T00:14:17.347 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.347 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.348 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.348 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.410122+0000", 2020-01-16T00:14:17.348 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000168297 2020-01-16T00:14:17.348 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.348 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.348 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:17.348 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.410290+0000", 2020-01-16T00:14:17.348 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013892204999999999 2020-01-16T00:14:17.349 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.349 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.349 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.349 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.424182+0000", 2020-01-16T00:14:17.349 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6482000000000002e-05 2020-01-16T00:14:17.349 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.349 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.349 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:17.350 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.424279+0000", 2020-01-16T00:14:17.350 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000130025 2020-01-16T00:14:17.350 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.350 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.350 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:17.350 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.424409+0000", 2020-01-16T00:14:17.350 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00151793 2020-01-16T00:14:17.350 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.351 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.351 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:17.351 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.425927+0000", 2020-01-16T00:14:17.351 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020657800000000001 2020-01-16T00:14:17.351 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.351 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.351 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:17.352 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.426133+0000", 2020-01-16T00:14:17.352 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3606999999999999e-05 2020-01-16T00:14:17.352 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.352 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.352 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:17.352 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.426167+0000", 2020-01-16T00:14:17.352 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9117999999999997e-05 2020-01-16T00:14:17.352 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.353 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.353 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:17.353 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.426246+0000", 2020-01-16T00:14:17.353 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021950859 2020-01-16T00:14:17.353 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.353 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:17.353 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.353 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.354 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.354 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4657 3.c 3:378f3722:::smithi12313934-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e57)", 2020-01-16T00:14:17.354 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:04.404326+0000", 2020-01-16T00:14:17.354 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 12.768792702000001, 2020-01-16T00:14:17.354 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022101092999999999, 2020-01-16T00:14:17.354 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:17.354 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:17.354 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:17.355 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:17.355 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:17.355 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4657 2020-01-16T00:14:17.355 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.355 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:17.355 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.355 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:17.356 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404326+0000", 2020-01-16T00:14:17.356 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:17.356 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.356 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.356 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:17.356 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404326+0000", 2020-01-16T00:14:17.356 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:17.356 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.357 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.357 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:17.357 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404326+0000", 2020-01-16T00:14:17.357 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.771e-06 2020-01-16T00:14:17.357 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.357 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.357 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:17.357 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404329+0000", 2020-01-16T00:14:17.358 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9909999999999999e-06 2020-01-16T00:14:17.358 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.358 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.358 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:17.358 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404335+0000", 2020-01-16T00:14:17.358 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8870000000000002e-06 2020-01-16T00:14:17.358 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.359 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.359 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:17.359 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404339+0000", 2020-01-16T00:14:17.359 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0066816460000000003 2020-01-16T00:14:17.359 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.359 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.359 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.359 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.411021+0000", 2020-01-16T00:14:17.360 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3233000000000007e-05 2020-01-16T00:14:17.360 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.360 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.360 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:17.360 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.411114+0000", 2020-01-16T00:14:17.360 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014244643 2020-01-16T00:14:17.360 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.360 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.361 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.361 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.425359+0000", 2020-01-16T00:14:17.361 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012771200000000001 2020-01-16T00:14:17.361 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.361 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.361 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:17.361 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.425487+0000", 2020-01-16T00:14:17.361 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00078975699999999998 2020-01-16T00:14:17.362 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.362 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.362 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.362 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.426276+0000", 2020-01-16T00:14:17.362 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8829999999999997e-05 2020-01-16T00:14:17.362 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.362 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.363 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:17.363 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.426335+0000", 2020-01-16T00:14:17.363 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2219999999999995e-05 2020-01-16T00:14:17.363 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.363 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.363 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:17.363 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.426427+0000", 2020-01-16T00:14:17.363 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022101092999999999 2020-01-16T00:14:17.364 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.364 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:17.364 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.364 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.364 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.364 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4690 3.2 3:4f8fe446:::smithi12313934-48:head [setxattr _header (56) in=7b,truncate 2067393] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-01-16T00:14:17.364 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:06.996624+0000", 2020-01-16T00:14:17.364 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 10.176495468000001, 2020-01-16T00:14:17.365 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021459626999999998, 2020-01-16T00:14:17.365 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:17.365 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:17.365 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:17.365 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:17.365 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:17.365 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4690 2020-01-16T00:14:17.366 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.366 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:17.366 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.366 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:17.366 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996624+0000", 2020-01-16T00:14:17.366 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:17.366 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.366 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.367 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:17.367 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996624+0000", 2020-01-16T00:14:17.367 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:17.367 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.367 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.367 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:17.367 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996623+0000", 2020-01-16T00:14:17.367 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.549e-06 2020-01-16T00:14:17.368 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.368 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.368 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:17.368 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996627+0000", 2020-01-16T00:14:17.368 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4501999999999999e-05 2020-01-16T00:14:17.368 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.368 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.368 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:17.369 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996642+0000", 2020-01-16T00:14:17.369 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1139999999999997e-06 2020-01-16T00:14:17.369 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.369 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.369 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:17.369 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996647+0000", 2020-01-16T00:14:17.369 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001305489 2020-01-16T00:14:17.369 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.370 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.370 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.370 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.997952+0000", 2020-01-16T00:14:17.370 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036299430000000001 2020-01-16T00:14:17.370 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.370 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.370 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:17.371 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.001582+0000", 2020-01-16T00:14:17.371 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011633755000000001 2020-01-16T00:14:17.371 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.371 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.371 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.371 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.013216+0000", 2020-01-16T00:14:17.371 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021714 2020-01-16T00:14:17.371 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.372 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.372 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:17.372 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.013433+0000", 2020-01-16T00:14:17.372 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00050686199999999996 2020-01-16T00:14:17.372 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.372 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.372 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:17.372 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.013940+0000", 2020-01-16T00:14:17.373 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039060330000000002 2020-01-16T00:14:17.373 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.373 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.373 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:17.373 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.017846+0000", 2020-01-16T00:14:17.373 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7609e-05 2020-01-16T00:14:17.373 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.373 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.374 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:17.374 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.017864+0000", 2020-01-16T00:14:17.374 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3856999999999994e-05 2020-01-16T00:14:17.374 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.374 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.374 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:17.374 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.017937+0000", 2020-01-16T00:14:17.374 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014570800000000001 2020-01-16T00:14:17.375 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.375 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.375 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:17.375 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.018083+0000", 2020-01-16T00:14:17.375 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021459626999999998 2020-01-16T00:14:17.375 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.375 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:17.375 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.376 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.376 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.376 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4691 3.2 3:4f8fe446:::smithi12313934-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e58)", 2020-01-16T00:14:17.376 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:06.996654+0000", 2020-01-16T00:14:17.376 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 10.176465363, 2020-01-16T00:14:17.376 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021903913000000001, 2020-01-16T00:14:17.376 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:17.377 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:17.377 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:17.377 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:17.377 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:17.377 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4691 2020-01-16T00:14:17.377 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.377 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:17.377 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.378 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:17.378 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996654+0000", 2020-01-16T00:14:17.378 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:17.378 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.378 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.378 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:17.378 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996654+0000", 2020-01-16T00:14:17.378 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:17.379 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.379 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.379 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:17.379 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996653+0000", 2020-01-16T00:14:17.379 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0939999999999999e-06 2020-01-16T00:14:17.379 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.379 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.379 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:17.380 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996656+0000", 2020-01-16T00:14:17.380 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7609999999999996e-06 2020-01-16T00:14:17.380 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.380 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.380 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:17.380 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996665+0000", 2020-01-16T00:14:17.380 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3589999999999999e-06 2020-01-16T00:14:17.381 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.381 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.381 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:17.381 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996668+0000", 2020-01-16T00:14:17.381 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0049635909999999998 2020-01-16T00:14:17.381 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.381 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.381 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.382 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.001632+0000", 2020-01-16T00:14:17.382 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9422000000000001e-05 2020-01-16T00:14:17.382 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.382 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.382 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:17.382 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.001691+0000", 2020-01-16T00:14:17.382 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013157973 2020-01-16T00:14:17.382 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.383 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.383 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.383 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.014849+0000", 2020-01-16T00:14:17.383 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00027545199999999998 2020-01-16T00:14:17.383 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.383 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.383 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:17.383 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.015125+0000", 2020-01-16T00:14:17.384 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030160159999999998 2020-01-16T00:14:17.384 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.384 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.384 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.384 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.018141+0000", 2020-01-16T00:14:17.384 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015583 2020-01-16T00:14:17.384 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.384 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.385 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:17.385 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.018297+0000", 2020-01-16T00:14:17.385 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00026096000000000002 2020-01-16T00:14:17.385 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.385 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.385 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:17.385 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.018558+0000", 2020-01-16T00:14:17.386 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021903913000000001 2020-01-16T00:14:17.386 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.386 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:17.386 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.386 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.386 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.386 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4703 3.2 3:42b7f20f:::smithi12313934-42:head [write 1635678~465329] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-01-16T00:14:17.386 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.020171+0000", 2020-01-16T00:14:17.387 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 10.152948155000001, 2020-01-16T00:14:17.387 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020016459, 2020-01-16T00:14:17.387 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:17.387 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:17.387 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:17.387 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:17.387 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:17.387 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4703 2020-01-16T00:14:17.388 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.388 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:17.388 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.388 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:17.388 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020171+0000", 2020-01-16T00:14:17.388 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:17.388 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.388 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.389 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:17.389 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020171+0000", 2020-01-16T00:14:17.389 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:17.389 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.389 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.389 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:17.389 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020169+0000", 2020-01-16T00:14:17.389 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013985399999999999 2020-01-16T00:14:17.390 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.390 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.390 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:17.390 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020309+0000", 2020-01-16T00:14:17.390 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9029999999999999e-06 2020-01-16T00:14:17.390 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.390 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.391 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:17.391 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020319+0000", 2020-01-16T00:14:17.391 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9589999999999997e-06 2020-01-16T00:14:17.391 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.391 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.391 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:17.391 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020323+0000", 2020-01-16T00:14:17.391 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8680000000000002e-05 2020-01-16T00:14:17.392 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.392 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.392 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.392 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020362+0000", 2020-01-16T00:14:17.392 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029188460000000001 2020-01-16T00:14:17.392 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.392 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.392 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:17.393 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023281+0000", 2020-01-16T00:14:17.393 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011810522 2020-01-16T00:14:17.393 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.393 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.393 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.393 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.035091+0000", 2020-01-16T00:14:17.393 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000126148 2020-01-16T00:14:17.393 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.394 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.394 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:17.394 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.035217+0000", 2020-01-16T00:14:17.394 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019636799999999999 2020-01-16T00:14:17.394 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.394 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.394 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:17.394 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.035414+0000", 2020-01-16T00:14:17.395 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003550919 2020-01-16T00:14:17.395 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.395 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.395 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:17.395 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.038965+0000", 2020-01-16T00:14:17.395 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001007441 2020-01-16T00:14:17.395 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.396 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.396 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:17.396 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.039972+0000", 2020-01-16T00:14:17.396 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1954999999999998e-05 2020-01-16T00:14:17.396 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.396 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.396 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:17.396 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.040044+0000", 2020-01-16T00:14:17.397 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014325899999999999 2020-01-16T00:14:17.397 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.397 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.397 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:17.397 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.040187+0000", 2020-01-16T00:14:17.397 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020016459 2020-01-16T00:14:17.397 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.397 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:17.398 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.398 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.398 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.398 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4704 3.2 3:42b7f20f:::smithi12313934-42:head [setxattr _header (56) in=7b,truncate 2101007] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-01-16T00:14:17.398 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.020335+0000", 2020-01-16T00:14:17.398 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 10.152784344000001, 2020-01-16T00:14:17.398 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023082354999999999, 2020-01-16T00:14:17.398 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:17.399 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:17.399 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:17.399 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:17.399 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:17.399 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4704 2020-01-16T00:14:17.399 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.399 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:17.400 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.400 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:17.400 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020335+0000", 2020-01-16T00:14:17.400 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:17.400 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.400 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.400 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:17.400 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020335+0000", 2020-01-16T00:14:17.401 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:17.401 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.401 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.401 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:17.401 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020334+0000", 2020-01-16T00:14:17.401 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7780000000000002e-06 2020-01-16T00:14:17.401 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.401 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.402 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:17.402 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020341+0000", 2020-01-16T00:14:17.402 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2039999999999996e-06 2020-01-16T00:14:17.402 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.402 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.402 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:17.402 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020346+0000", 2020-01-16T00:14:17.402 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8800000000000008e-06 2020-01-16T00:14:17.403 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.403 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.403 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:17.403 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020354+0000", 2020-01-16T00:14:17.403 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029530490000000001 2020-01-16T00:14:17.403 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.403 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.403 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.404 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023307+0000", 2020-01-16T00:14:17.404 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8681000000000003e-05 2020-01-16T00:14:17.404 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.404 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.404 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:17.404 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023365+0000", 2020-01-16T00:14:17.404 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013386927999999999 2020-01-16T00:14:17.405 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.405 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.405 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.405 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.036752+0000", 2020-01-16T00:14:17.405 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8906000000000007e-05 2020-01-16T00:14:17.405 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.405 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.405 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:17.406 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.036831+0000", 2020-01-16T00:14:17.406 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000142922 2020-01-16T00:14:17.406 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.406 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.406 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:17.406 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.036974+0000", 2020-01-16T00:14:17.406 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0038357280000000001 2020-01-16T00:14:17.406 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.407 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.407 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:17.407 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.040810+0000", 2020-01-16T00:14:17.407 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0024973949999999999 2020-01-16T00:14:17.407 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.407 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.407 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:17.407 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.043307+0000", 2020-01-16T00:14:17.408 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8513000000000002e-05 2020-01-16T00:14:17.408 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.408 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.408 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:17.408 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.043346+0000", 2020-01-16T00:14:17.408 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.127e-05 2020-01-16T00:14:17.408 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.409 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.409 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:17.409 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.043417+0000", 2020-01-16T00:14:17.409 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023082354999999999 2020-01-16T00:14:17.409 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.409 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:17.409 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.409 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.410 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.410 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4705 3.2 3:42b7f20f:::smithi12313934-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e58)", 2020-01-16T00:14:17.410 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.020362+0000", 2020-01-16T00:14:17.410 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 10.152757408999999, 2020-01-16T00:14:17.410 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023262267999999999, 2020-01-16T00:14:17.410 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:17.410 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:17.410 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:17.411 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:17.411 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:17.411 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4705 2020-01-16T00:14:17.411 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.411 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:17.411 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.411 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:17.411 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020362+0000", 2020-01-16T00:14:17.412 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:17.412 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.412 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.412 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:17.412 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020362+0000", 2020-01-16T00:14:17.412 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:17.412 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.413 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.413 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:17.413 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020361+0000", 2020-01-16T00:14:17.413 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.365e-06 2020-01-16T00:14:17.413 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.413 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.413 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:17.413 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020364+0000", 2020-01-16T00:14:17.414 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5649999999999998e-06 2020-01-16T00:14:17.414 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.414 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.414 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:17.414 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020371+0000", 2020-01-16T00:14:17.414 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6349999999999999e-06 2020-01-16T00:14:17.414 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.414 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.415 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:17.415 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020375+0000", 2020-01-16T00:14:17.415 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030174920000000001 2020-01-16T00:14:17.415 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.415 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.415 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.415 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023392+0000", 2020-01-16T00:14:17.415 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4997000000000003e-05 2020-01-16T00:14:17.416 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.416 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.416 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:17.416 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023437+0000", 2020-01-16T00:14:17.416 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015293378999999999 2020-01-16T00:14:17.416 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.416 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.417 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.417 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.038730+0000", 2020-01-16T00:14:17.417 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000124206 2020-01-16T00:14:17.417 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.417 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.417 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:17.417 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.038855+0000", 2020-01-16T00:14:17.417 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0045951489999999998 2020-01-16T00:14:17.418 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.418 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.418 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.418 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.043450+0000", 2020-01-16T00:14:17.418 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1363000000000001e-05 2020-01-16T00:14:17.418 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.418 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.418 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:17.419 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.043521+0000", 2020-01-16T00:14:17.419 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000102737 2020-01-16T00:14:17.419 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.419 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.419 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:17.419 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.043624+0000", 2020-01-16T00:14:17.419 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023262267999999999 2020-01-16T00:14:17.419 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.420 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:17.420 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.420 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.420 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.420 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4869 3.c 3:3025546b:::smithi12313934-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-01-16T00:14:17.420 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:11.931564+0000", 2020-01-16T00:14:17.420 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 5.2415545259999998, 2020-01-16T00:14:17.421 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018620945, 2020-01-16T00:14:17.421 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:17.421 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:17.421 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:17.421 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:17.421 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:17.421 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4869 2020-01-16T00:14:17.421 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.422 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:17.422 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.422 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:17.422 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.931564+0000", 2020-01-16T00:14:17.422 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:17.422 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.422 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.422 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:17.423 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.931564+0000", 2020-01-16T00:14:17.423 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:17.423 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.423 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.423 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:17.423 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.931563+0000", 2020-01-16T00:14:17.423 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8030000000000002e-06 2020-01-16T00:14:17.423 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.424 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.424 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:17.424 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.931569+0000", 2020-01-16T00:14:17.424 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3370000000000006e-06 2020-01-16T00:14:17.424 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.424 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.424 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:17.424 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.931579+0000", 2020-01-16T00:14:17.425 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9569999999999999e-06 2020-01-16T00:14:17.425 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.425 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.425 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:17.425 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.931585+0000", 2020-01-16T00:14:17.425 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021810430000000001 2020-01-16T00:14:17.425 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.425 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.426 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.426 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.933766+0000", 2020-01-16T00:14:17.426 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2802000000000001e-05 2020-01-16T00:14:17.426 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.426 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.426 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:17.426 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.933808+0000", 2020-01-16T00:14:17.427 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0099913469999999994 2020-01-16T00:14:17.427 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.427 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.427 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.427 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.943800+0000", 2020-01-16T00:14:17.427 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013257000000000001 2020-01-16T00:14:17.427 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.427 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.428 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:17.428 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.943932+0000", 2020-01-16T00:14:17.428 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0058405799999999997 2020-01-16T00:14:17.428 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.428 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.428 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.428 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.949773+0000", 2020-01-16T00:14:17.428 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000158493 2020-01-16T00:14:17.429 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.429 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.429 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:17.429 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.949931+0000", 2020-01-16T00:14:17.429 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000254083 2020-01-16T00:14:17.429 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.429 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.429 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:17.430 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.950185+0000", 2020-01-16T00:14:17.430 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018620945 2020-01-16T00:14:17.430 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.430 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:17.430 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.430 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.430 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:17.430 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:17.434 INFO:tasks.rados.rados.0.smithi123.stdout:1801: writing smithi12313934-22 from 709427 to 1255374 tid 1 2020-01-16T00:14:17.434 INFO:tasks.rados.rados.0.smithi123.stdout:1801: writing smithi12313934-22 from 1813774 to 2426671 tid 2 2020-01-16T00:14:17.434 INFO:tasks.rados.rados.0.smithi123.stdout:1801: writing smithi12313934-22 from 2985657 to 3109427 tid 3 2020-01-16T00:14:17.434 INFO:tasks.rados.rados.0.smithi123.stdout:1802: write oid 14 current snap is 0 2020-01-16T00:14:17.434 INFO:tasks.rados.rados.0.smithi123.stdout:1802: seq_num 662 ranges {726234=737549,1988609=756555} 2020-01-16T00:14:17.434 INFO:tasks.rados.rados.0.smithi123.stdout:1802: writing smithi12313934-14 from 726234 to 1463783 tid 1 2020-01-16T00:14:17.435 INFO:tasks.rados.rados.0.smithi123.stdout:1802: writing smithi12313934-14 from 1988609 to 2745164 tid 2 2020-01-16T00:14:17.435 INFO:tasks.rados.rados.0.smithi123.stdout:1803: read oid 41 snap -1 2020-01-16T00:14:17.435 INFO:tasks.rados.rados.0.smithi123.stdout:1803: expect (ObjNum 602 snap 0 seq_num 602) 2020-01-16T00:14:17.435 INFO:tasks.rados.rados.0.smithi123.stdout:1804: write (excl) oid 10 current snap is 0 2020-01-16T00:14:17.435 INFO:tasks.rados.rados.0.smithi123.stdout:1804: seq_num 663 ranges {743041=412798,1930738=616566,3040503=102538} 2020-01-16T00:14:17.435 INFO:tasks.rados.rados.0.smithi123.stdout:1804: writing smithi12313934-10 from 743041 to 1155839 tid 1 2020-01-16T00:14:17.435 INFO:tasks.rados.rados.0.smithi123.stdout:1804: writing smithi12313934-10 from 1930738 to 2547304 tid 2 2020-01-16T00:14:17.436 INFO:tasks.rados.rados.0.smithi123.stdout:1804: writing smithi12313934-10 from 3040503 to 3143041 tid 3 2020-01-16T00:14:17.436 INFO:tasks.rados.rados.0.smithi123.stdout:1805: write (excl) oid 19 current snap is 0 2020-01-16T00:14:17.436 INFO:tasks.rados.rados.0.smithi123.stdout:1805: seq_num 664 ranges {759848=488047,1989220=760224} 2020-01-16T00:14:17.436 INFO:tasks.rados.rados.0.smithi123.stdout:1805: writing smithi12313934-19 from 759848 to 1247895 tid 1 2020-01-16T00:14:17.436 INFO:tasks.rados.rados.0.smithi123.stdout:1805: writing smithi12313934-19 from 1989220 to 2749444 tid 2 2020-01-16T00:14:17.436 INFO:tasks.rados.rados.0.smithi123.stdout:1806: write (excl) oid 46 current snap is 0 2020-01-16T00:14:17.436 INFO:tasks.rados.rados.0.smithi123.stdout:1806: seq_num 665 ranges {776655=563296,2047702=620235,3095349=81306} 2020-01-16T00:14:17.436 INFO:tasks.rados.rados.0.smithi123.stdout:1806: writing smithi12313934-46 from 776655 to 1339951 tid 1 2020-01-16T00:14:17.437 INFO:tasks.rados.rados.0.smithi123.stdout:1806: writing smithi12313934-46 from 2047702 to 2667937 tid 2 2020-01-16T00:14:17.437 INFO:tasks.rados.rados.0.smithi123.stdout:1806: writing smithi12313934-46 from 3095349 to 3176655 tid 3 2020-01-16T00:14:17.437 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:17.437 INFO:tasks.rados.rados.0.smithi123.stdout:1791: finishing write tid 2 to smithi12313934-12 2020-01-16T00:14:17.437 INFO:tasks.rados.rados.0.smithi123.stdout:1793: finishing write tid 1 to smithi12313934-31 2020-01-16T00:14:17.437 INFO:tasks.rados.rados.0.smithi123.stdout:1792: expect (ObjNum 629 snap 0 seq_num 629) 2020-01-16T00:14:17.437 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:17.428+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:17.438 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-01-16T00:14:17.513 INFO:tasks.rados.rados.0.smithi123.stdout:1791: finishing write tid 3 to smithi12313934-12 2020-01-16T00:14:17.513 INFO:tasks.rados.rados.0.smithi123.stdout:1791: finishing write tid 5 to smithi12313934-12 2020-01-16T00:14:17.514 INFO:tasks.rados.rados.0.smithi123.stdout:1791: finishing write tid 6 to smithi12313934-12 2020-01-16T00:14:17.514 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 12 v 188 (ObjNum 655 snap 0 seq_num 655) dirty exists 2020-01-16T00:14:17.514 INFO:tasks.rados.rados.0.smithi123.stdout:1791: left oid 12 (ObjNum 655 snap 0 seq_num 655) 2020-01-16T00:14:17.514 INFO:tasks.rados.rados.0.smithi123.stdout:1793: finishing write tid 2 to smithi12313934-31 2020-01-16T00:14:17.514 INFO:tasks.rados.rados.0.smithi123.stdout:1791: done (15 left) 2020-01-16T00:14:17.514 INFO:tasks.rados.rados.0.smithi123.stdout:1792: done (14 left) 2020-01-16T00:14:17.514 INFO:tasks.rados.rados.0.smithi123.stdout:1807: read oid 1 snap -1 2020-01-16T00:14:17.515 INFO:tasks.rados.rados.0.smithi123.stdout:1807: expect (ObjNum 618 snap 0 seq_num 618) 2020-01-16T00:14:17.515 INFO:tasks.rados.rados.0.smithi123.stdout:1808: read oid 43 snap -1 2020-01-16T00:14:17.515 INFO:tasks.rados.rados.0.smithi123.stdout:1808: expect (ObjNum 607 snap 0 seq_num 607) 2020-01-16T00:14:17.515 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:17.515 INFO:tasks.rados.rados.0.smithi123.stdout:1793: finishing write tid 3 to smithi12313934-31 2020-01-16T00:14:17.515 INFO:tasks.rados.rados.0.smithi123.stdout:1793: finishing write tid 5 to smithi12313934-31 2020-01-16T00:14:17.515 INFO:tasks.rados.rados.0.smithi123.stdout:1793: finishing write tid 6 to smithi12313934-31 2020-01-16T00:14:17.515 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 31 v 236 (ObjNum 656 snap 0 seq_num 656) dirty exists 2020-01-16T00:14:17.516 INFO:tasks.rados.rados.0.smithi123.stdout:1793: left oid 31 (ObjNum 656 snap 0 seq_num 656) 2020-01-16T00:14:17.516 INFO:tasks.rados.rados.0.smithi123.stdout:1794: expect (ObjNum 628 snap 0 seq_num 628) 2020-01-16T00:14:17.532 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:17.530+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:17.580 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:17.580 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:14:17.580 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:14:17.580 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:17.589 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-01-16T00:14:17.632 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:17.630+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:17.675 INFO:tasks.rados.rados.0.smithi123.stdout:1795: finishing write tid 1 to smithi12313934-32 2020-01-16T00:14:17.676 INFO:tasks.rados.rados.0.smithi123.stdout:1795: finishing write tid 2 to smithi12313934-32 2020-01-16T00:14:17.676 INFO:tasks.rados.rados.0.smithi123.stdout:1795: finishing write tid 3 to smithi12313934-32 2020-01-16T00:14:17.676 INFO:tasks.rados.rados.0.smithi123.stdout:1795: finishing write tid 5 to smithi12313934-32 2020-01-16T00:14:17.676 INFO:tasks.rados.rados.0.smithi123.stdout:1795: finishing write tid 6 to smithi12313934-32 2020-01-16T00:14:17.676 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 32 v 178 (ObjNum 657 snap 0 seq_num 657) dirty exists 2020-01-16T00:14:17.676 INFO:tasks.rados.rados.0.smithi123.stdout:1795: left oid 32 (ObjNum 657 snap 0 seq_num 657) 2020-01-16T00:14:17.676 INFO:tasks.rados.rados.0.smithi123.stdout:1796: finishing write tid 1 to smithi12313934-42 2020-01-16T00:14:17.677 INFO:tasks.rados.rados.0.smithi123.stdout:1796: finishing write tid 2 to smithi12313934-42 2020-01-16T00:14:17.677 INFO:tasks.rados.rados.0.smithi123.stdout:1796: finishing write tid 4 to smithi12313934-42 2020-01-16T00:14:17.677 INFO:tasks.rados.rados.0.smithi123.stdout:1796: finishing write tid 5 to smithi12313934-42 2020-01-16T00:14:17.677 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 42 v 217 (ObjNum 658 snap 0 seq_num 658) dirty exists 2020-01-16T00:14:17.677 INFO:tasks.rados.rados.0.smithi123.stdout:1796: left oid 42 (ObjNum 658 snap 0 seq_num 658) 2020-01-16T00:14:17.677 INFO:tasks.rados.rados.0.smithi123.stdout:1797: finishing write tid 1 to smithi12313934-29 2020-01-16T00:14:17.677 INFO:tasks.rados.rados.0.smithi123.stdout:1797: finishing write tid 2 to smithi12313934-29 2020-01-16T00:14:17.678 INFO:tasks.rados.rados.0.smithi123.stdout:1797: finishing write tid 4 to smithi12313934-29 2020-01-16T00:14:17.678 INFO:tasks.rados.rados.0.smithi123.stdout:1797: finishing write tid 5 to smithi12313934-29 2020-01-16T00:14:17.678 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 29 v 187 (ObjNum 659 snap 0 seq_num 659) dirty exists 2020-01-16T00:14:17.678 INFO:tasks.rados.rados.0.smithi123.stdout:1797: left oid 29 (ObjNum 659 snap 0 seq_num 659) 2020-01-16T00:14:17.678 INFO:tasks.rados.rados.0.smithi123.stdout:1798: finishing write tid 1 to smithi12313934-16 2020-01-16T00:14:17.678 INFO:tasks.rados.rados.0.smithi123.stdout:1798: finishing write tid 2 to smithi12313934-16 2020-01-16T00:14:17.678 INFO:tasks.rados.rados.0.smithi123.stdout:1798: finishing write tid 4 to smithi12313934-16 2020-01-16T00:14:17.678 INFO:tasks.rados.rados.0.smithi123.stdout:1798: finishing write tid 5 to smithi12313934-16 2020-01-16T00:14:17.679 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 16 v 280 (ObjNum 660 snap 0 seq_num 660) dirty exists 2020-01-16T00:14:17.679 INFO:tasks.rados.rados.0.smithi123.stdout:1798: left oid 16 (ObjNum 660 snap 0 seq_num 660) 2020-01-16T00:14:17.679 INFO:tasks.rados.rados.0.smithi123.stdout:1799: expect (ObjNum 535 snap 0 seq_num 535) 2020-01-16T00:14:17.731 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:17.731 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:14:17.732 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:14:17.732 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:14:17.732 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:17.734 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:17.732+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:17.740 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-01-16T00:14:17.835 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:17.832+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:17.841 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:17.841 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:14:17.841 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:14:17.841 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:14:17.841 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.841 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.4.0:114 3.9 e24/20 3:9fc687b2:::smithi12313934-44:head v 24'74, mlcod=24'72)", 2020-01-16T00:14:17.842 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:14:17.842 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 99.513635446999999, 2020-01-16T00:14:17.842 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010399429, 2020-01-16T00:14:17.842 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:17.842 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:17.842 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:17.842 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.842 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:17.843 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:14:17.843 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:17.843 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.843 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.843 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:17.843 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:14:17.843 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:17.844 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.844 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.844 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:17.844 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325276+0000", 2020-01-16T00:14:17.844 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0046127069999999997 2020-01-16T00:14:17.844 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.844 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.844 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:17.845 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329889+0000", 2020-01-16T00:14:17.845 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3519999999999998e-05 2020-01-16T00:14:17.845 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.845 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.845 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:17.845 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329912+0000", 2020-01-16T00:14:17.845 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4070000000000004e-06 2020-01-16T00:14:17.845 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.846 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.846 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:17.846 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329920+0000", 2020-01-16T00:14:17.846 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8817000000000003e-05 2020-01-16T00:14:17.846 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.846 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.846 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.846 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329978+0000", 2020-01-16T00:14:17.847 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9721e-05 2020-01-16T00:14:17.847 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.847 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.847 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:17.847 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329998+0000", 2020-01-16T00:14:17.847 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056328139999999999 2020-01-16T00:14:17.847 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.848 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.848 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:17.848 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.335631+0000", 2020-01-16T00:14:17.848 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7170999999999999e-05 2020-01-16T00:14:17.848 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.848 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.848 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:17.848 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.335678+0000", 2020-01-16T00:14:17.849 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010399429 2020-01-16T00:14:17.849 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.849 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:17.849 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.849 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.849 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.849 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:55 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e30)", 2020-01-16T00:14:17.849 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.736036+0000", 2020-01-16T00:14:17.850 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 85.102878227999994, 2020-01-16T00:14:17.850 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010974536, 2020-01-16T00:14:17.850 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:17.850 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:17.850 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:17.850 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.850 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:17.851 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736036+0000", 2020-01-16T00:14:17.851 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:17.851 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.851 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.851 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:17.851 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736036+0000", 2020-01-16T00:14:17.851 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:17.851 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.852 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.852 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:17.852 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736034+0000", 2020-01-16T00:14:17.852 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043589699999999998 2020-01-16T00:14:17.852 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.852 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.852 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:17.852 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736469+0000", 2020-01-16T00:14:17.853 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9882e-05 2020-01-16T00:14:17.853 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.853 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.853 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:17.853 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736489+0000", 2020-01-16T00:14:17.853 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8339999999999999e-06 2020-01-16T00:14:17.853 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.853 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.854 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:17.854 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736497+0000", 2020-01-16T00:14:17.854 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5354999999999999e-05 2020-01-16T00:14:17.854 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.854 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.854 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.854 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736572+0000", 2020-01-16T00:14:17.854 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5228000000000004e-05 2020-01-16T00:14:17.855 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.855 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.855 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:17.855 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736668+0000", 2020-01-16T00:14:17.855 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017212999999999999 2020-01-16T00:14:17.855 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.855 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.856 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:17.856 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736840+0000", 2020-01-16T00:14:17.856 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0099057020000000006 2020-01-16T00:14:17.856 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.856 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.856 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:17.856 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746746+0000", 2020-01-16T00:14:17.856 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0586000000000005e-05 2020-01-16T00:14:17.857 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.857 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.857 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:17.857 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746816+0000", 2020-01-16T00:14:17.857 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2445000000000002e-05 2020-01-16T00:14:17.857 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.857 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.858 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:17.858 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746889+0000", 2020-01-16T00:14:17.858 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6839000000000002e-05 2020-01-16T00:14:17.858 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.858 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.859 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:17.859 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746955+0000", 2020-01-16T00:14:17.859 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5074000000000002e-05 2020-01-16T00:14:17.859 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.859 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.859 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:17.859 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.747011+0000", 2020-01-16T00:14:17.859 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010974536 2020-01-16T00:14:17.860 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.860 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:17.860 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.860 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.860 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.860 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:56 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e30)", 2020-01-16T00:14:17.860 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.736514+0000", 2020-01-16T00:14:17.860 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 85.102400392999996, 2020-01-16T00:14:17.861 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011660584999999999, 2020-01-16T00:14:17.861 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:17.861 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:17.861 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:17.861 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.861 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:17.861 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736514+0000", 2020-01-16T00:14:17.861 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:17.862 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.862 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.862 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:17.862 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736514+0000", 2020-01-16T00:14:17.862 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:17.862 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.862 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.863 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:17.863 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736513+0000", 2020-01-16T00:14:17.863 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9677000000000004e-05 2020-01-16T00:14:17.863 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.863 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.863 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:17.863 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736583+0000", 2020-01-16T00:14:17.863 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5269999999999998e-06 2020-01-16T00:14:17.864 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.864 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.864 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:17.864 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736591+0000", 2020-01-16T00:14:17.864 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5769999999999996e-06 2020-01-16T00:14:17.864 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.864 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.865 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:17.865 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736599+0000", 2020-01-16T00:14:17.865 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0053593310000000002 2020-01-16T00:14:17.865 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.865 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.865 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.866 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.741958+0000", 2020-01-16T00:14:17.866 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014164099999999999 2020-01-16T00:14:17.866 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.866 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.866 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:17.867 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.742100+0000", 2020-01-16T00:14:17.867 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020126000000000001 2020-01-16T00:14:17.867 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.867 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.867 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:17.868 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.742301+0000", 2020-01-16T00:14:17.868 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0045052859999999998 2020-01-16T00:14:17.868 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.868 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.868 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:17.869 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746806+0000", 2020-01-16T00:14:17.869 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00070802699999999996 2020-01-16T00:14:17.869 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.869 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.869 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:17.870 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.747514+0000", 2020-01-16T00:14:17.870 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00059247200000000003 2020-01-16T00:14:17.870 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.870 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.870 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:17.870 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748107+0000", 2020-01-16T00:14:17.871 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1870999999999999e-05 2020-01-16T00:14:17.871 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.871 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.871 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:17.871 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748149+0000", 2020-01-16T00:14:17.871 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5831000000000001e-05 2020-01-16T00:14:17.872 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.872 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.872 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:17.872 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748174+0000", 2020-01-16T00:14:17.872 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011660584999999999 2020-01-16T00:14:17.872 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.873 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:17.873 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.873 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.873 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.873 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:70 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e35)", 2020-01-16T00:14:17.873 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.985494+0000", 2020-01-16T00:14:17.874 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 71.853420596999996, 2020-01-16T00:14:17.874 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013746267, 2020-01-16T00:14:17.874 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:17.874 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:17.874 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:17.874 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.875 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:17.875 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985494+0000", 2020-01-16T00:14:17.875 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:17.875 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.875 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.875 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:17.876 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985494+0000", 2020-01-16T00:14:17.876 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:17.876 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.876 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.876 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:17.877 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985490+0000", 2020-01-16T00:14:17.877 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00077792099999999999 2020-01-16T00:14:17.877 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.877 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.877 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:17.877 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986268+0000", 2020-01-16T00:14:17.878 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9718000000000001e-05 2020-01-16T00:14:17.878 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.878 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.878 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:17.878 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986318+0000", 2020-01-16T00:14:17.878 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.4349999999999997e-06 2020-01-16T00:14:17.879 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.879 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.879 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:17.879 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986326+0000", 2020-01-16T00:14:17.879 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3298999999999999e-05 2020-01-16T00:14:17.879 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.880 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.880 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.880 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986380+0000", 2020-01-16T00:14:17.880 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4969999999999995e-05 2020-01-16T00:14:17.880 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.880 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.881 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:17.881 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986455+0000", 2020-01-16T00:14:17.881 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000110587 2020-01-16T00:14:17.881 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.881 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.881 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:17.882 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986565+0000", 2020-01-16T00:14:17.882 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063931600000000002 2020-01-16T00:14:17.882 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.882 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.882 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:17.883 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.992958+0000", 2020-01-16T00:14:17.883 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013008480000000001 2020-01-16T00:14:17.883 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.883 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.883 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:17.883 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.994259+0000", 2020-01-16T00:14:17.884 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048399979999999999 2020-01-16T00:14:17.884 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.884 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.884 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:17.884 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999099+0000", 2020-01-16T00:14:17.884 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1874999999999998e-05 2020-01-16T00:14:17.885 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.885 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.885 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:17.885 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999181+0000", 2020-01-16T00:14:17.885 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8865e-05 2020-01-16T00:14:17.885 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.886 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.886 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:17.886 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999240+0000", 2020-01-16T00:14:17.886 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013746267 2020-01-16T00:14:17.886 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.887 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:17.887 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.887 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.887 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.887 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:71 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e35)", 2020-01-16T00:14:17.887 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.986341+0000", 2020-01-16T00:14:17.888 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 71.852573235999998, 2020-01-16T00:14:17.888 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013252807, 2020-01-16T00:14:17.888 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:17.888 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:17.888 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:17.888 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.889 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:17.889 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986341+0000", 2020-01-16T00:14:17.889 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:17.889 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.889 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.890 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:17.890 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986341+0000", 2020-01-16T00:14:17.890 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:17.890 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.890 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.890 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:17.891 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986340+0000", 2020-01-16T00:14:17.891 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000103867 2020-01-16T00:14:17.891 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.891 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.891 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:17.891 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986444+0000", 2020-01-16T00:14:17.892 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0104e-05 2020-01-16T00:14:17.892 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.892 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.892 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:17.892 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986454+0000", 2020-01-16T00:14:17.893 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4220000000000002e-06 2020-01-16T00:14:17.893 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.893 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.893 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:17.893 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986458+0000", 2020-01-16T00:14:17.893 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033523070000000001 2020-01-16T00:14:17.894 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.894 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.894 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.894 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.989811+0000", 2020-01-16T00:14:17.894 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3296999999999999e-05 2020-01-16T00:14:17.894 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.895 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.895 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:17.895 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.989894+0000", 2020-01-16T00:14:17.895 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000117701 2020-01-16T00:14:17.895 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.895 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.896 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:17.896 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.990012+0000", 2020-01-16T00:14:17.896 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0032873490000000002 2020-01-16T00:14:17.896 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.896 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.897 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:17.897 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993299+0000", 2020-01-16T00:14:17.897 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002422619 2020-01-16T00:14:17.897 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.897 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.897 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:17.898 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.995722+0000", 2020-01-16T00:14:17.898 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0037839589999999999 2020-01-16T00:14:17.898 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.898 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.898 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:17.898 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999506+0000", 2020-01-16T00:14:17.899 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8559999999999997e-05 2020-01-16T00:14:17.899 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.899 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.899 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:17.899 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999544+0000", 2020-01-16T00:14:17.899 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9599999999999999e-05 2020-01-16T00:14:17.900 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.900 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.900 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:17.900 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999594+0000", 2020-01-16T00:14:17.900 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013252807 2020-01-16T00:14:17.901 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.901 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:17.901 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.901 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.901 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.901 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:77 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e36)", 2020-01-16T00:14:17.902 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.555854+0000", 2020-01-16T00:14:17.902 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 70.283060147, 2020-01-16T00:14:17.902 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010455002999999999, 2020-01-16T00:14:17.902 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:17.902 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:17.902 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:17.903 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.903 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:17.903 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555854+0000", 2020-01-16T00:14:17.903 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:17.903 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.904 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.904 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:17.904 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555854+0000", 2020-01-16T00:14:17.904 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:17.904 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.904 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.904 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:17.905 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555851+0000", 2020-01-16T00:14:17.905 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00048310200000000001 2020-01-16T00:14:17.905 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.905 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.905 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:17.905 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556334+0000", 2020-01-16T00:14:17.905 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2232e-05 2020-01-16T00:14:17.906 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.906 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.906 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:17.906 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556356+0000", 2020-01-16T00:14:17.906 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9750000000000007e-06 2020-01-16T00:14:17.906 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.906 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.906 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:17.907 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556364+0000", 2020-01-16T00:14:17.907 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8872999999999997e-05 2020-01-16T00:14:17.907 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.907 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.907 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.907 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556423+0000", 2020-01-16T00:14:17.907 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8687000000000001e-05 2020-01-16T00:14:17.908 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.908 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.908 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:17.908 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556502+0000", 2020-01-16T00:14:17.908 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013235300000000001 2020-01-16T00:14:17.908 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.908 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.908 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:17.909 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556634+0000", 2020-01-16T00:14:17.909 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0059770520000000001 2020-01-16T00:14:17.909 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.909 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.909 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:17.909 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.562611+0000", 2020-01-16T00:14:17.909 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029492820000000001 2020-01-16T00:14:17.909 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.910 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.910 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:17.910 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.565561+0000", 2020-01-16T00:14:17.910 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00060620300000000004 2020-01-16T00:14:17.910 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.910 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.910 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:17.911 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566167+0000", 2020-01-16T00:14:17.911 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3831000000000004e-05 2020-01-16T00:14:17.911 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.911 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.911 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:17.911 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566251+0000", 2020-01-16T00:14:17.911 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8328000000000001e-05 2020-01-16T00:14:17.911 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.912 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.912 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:17.912 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566309+0000", 2020-01-16T00:14:17.912 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010455002999999999 2020-01-16T00:14:17.912 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.912 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:17.912 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.913 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.913 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.913 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:78 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e36)", 2020-01-16T00:14:17.913 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.556378+0000", 2020-01-16T00:14:17.913 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 70.282536214000004, 2020-01-16T00:14:17.913 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010396311, 2020-01-16T00:14:17.913 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:17.913 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:17.914 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:17.914 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.914 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:17.914 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556378+0000", 2020-01-16T00:14:17.914 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:17.914 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.914 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.914 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:17.915 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556378+0000", 2020-01-16T00:14:17.915 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:17.915 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.915 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.915 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:17.915 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556377+0000", 2020-01-16T00:14:17.915 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4307000000000001e-05 2020-01-16T00:14:17.916 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.916 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.916 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:17.916 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556422+0000", 2020-01-16T00:14:17.916 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9620000000000005e-06 2020-01-16T00:14:17.916 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.916 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.917 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:17.917 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556432+0000", 2020-01-16T00:14:17.917 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5979999999999999e-06 2020-01-16T00:14:17.917 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.917 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.917 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:17.917 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556436+0000", 2020-01-16T00:14:17.918 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021059469999999999 2020-01-16T00:14:17.918 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.918 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.918 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.918 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.558542+0000", 2020-01-16T00:14:17.918 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000194458 2020-01-16T00:14:17.918 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.918 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.919 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:17.919 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.558737+0000", 2020-01-16T00:14:17.919 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00032867900000000002 2020-01-16T00:14:17.919 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.919 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.919 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:17.919 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.559065+0000", 2020-01-16T00:14:17.919 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052547499999999999 2020-01-16T00:14:17.920 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.920 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.920 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:17.920 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.564320+0000", 2020-01-16T00:14:17.920 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001565213 2020-01-16T00:14:17.920 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.920 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.921 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:17.921 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.565885+0000", 2020-01-16T00:14:17.921 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00076994300000000004 2020-01-16T00:14:17.921 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.921 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.921 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:17.921 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566655+0000", 2020-01-16T00:14:17.921 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7533000000000005e-05 2020-01-16T00:14:17.922 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.922 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.922 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:17.922 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566723+0000", 2020-01-16T00:14:17.922 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1616999999999997e-05 2020-01-16T00:14:17.922 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.922 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.922 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:17.923 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566774+0000", 2020-01-16T00:14:17.923 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010396311 2020-01-16T00:14:17.923 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.923 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:17.923 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.923 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.923 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.923 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:72 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 56832~1408000] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e37)", 2020-01-16T00:14:17.924 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.578753+0000", 2020-01-16T00:14:17.924 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 65.260160911, 2020-01-16T00:14:17.924 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011826918, 2020-01-16T00:14:17.924 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:17.924 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:17.924 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:17.924 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.925 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:17.925 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578753+0000", 2020-01-16T00:14:17.925 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:17.925 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.925 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.925 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:17.925 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578753+0000", 2020-01-16T00:14:17.925 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:17.926 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.926 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.926 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:17.926 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578751+0000", 2020-01-16T00:14:17.926 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00042457699999999998 2020-01-16T00:14:17.926 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.926 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.927 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:17.927 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579175+0000", 2020-01-16T00:14:17.927 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5404999999999999e-05 2020-01-16T00:14:17.927 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.927 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.927 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:17.927 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579191+0000", 2020-01-16T00:14:17.927 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6149999999999996e-06 2020-01-16T00:14:17.928 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.928 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.928 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:17.928 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579196+0000", 2020-01-16T00:14:17.928 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000104794 2020-01-16T00:14:17.930 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.930 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.931 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.931 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579301+0000", 2020-01-16T00:14:17.931 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020947199999999999 2020-01-16T00:14:17.931 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.931 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.931 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:17.932 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579511+0000", 2020-01-16T00:14:17.932 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034328400000000002 2020-01-16T00:14:17.932 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.932 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.932 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:17.932 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579854+0000", 2020-01-16T00:14:17.932 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.005078117 2020-01-16T00:14:17.932 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.933 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.933 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:17.933 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.584932+0000", 2020-01-16T00:14:17.933 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0014702160000000001 2020-01-16T00:14:17.933 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.933 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.933 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:17.934 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.586402+0000", 2020-01-16T00:14:17.934 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040623589999999998 2020-01-16T00:14:17.934 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.934 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.934 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:17.934 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590465+0000", 2020-01-16T00:14:17.934 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4478000000000002e-05 2020-01-16T00:14:17.935 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.935 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.935 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:17.935 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590529+0000", 2020-01-16T00:14:17.935 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1121000000000001e-05 2020-01-16T00:14:17.935 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.935 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.936 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:17.936 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590580+0000", 2020-01-16T00:14:17.936 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011826918 2020-01-16T00:14:17.936 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.936 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:17.936 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.936 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.937 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.937 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:73 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 1922048~118272] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e37)", 2020-01-16T00:14:17.937 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:14:17.937 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 65.259708114000006, 2020-01-16T00:14:17.937 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011750764, 2020-01-16T00:14:17.937 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:17.937 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:17.938 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:17.938 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.938 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:17.938 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:14:17.938 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:17.938 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.938 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.939 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:17.939 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:14:17.939 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:17.939 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.939 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.939 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:17.939 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:14:17.939 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2780000000000001e-05 2020-01-16T00:14:17.940 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.940 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.940 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:17.940 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579238+0000", 2020-01-16T00:14:17.940 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4341999999999999e-05 2020-01-16T00:14:17.940 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.940 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.941 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:17.941 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579253+0000", 2020-01-16T00:14:17.941 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1920000000000001e-06 2020-01-16T00:14:17.941 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.941 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.941 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:17.941 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579256+0000", 2020-01-16T00:14:17.942 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003224902 2020-01-16T00:14:17.942 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.942 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.942 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.942 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.582481+0000", 2020-01-16T00:14:17.942 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000180729 2020-01-16T00:14:17.943 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.943 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.943 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:17.943 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.582662+0000", 2020-01-16T00:14:17.943 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00025106100000000001 2020-01-16T00:14:17.943 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.943 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.943 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:17.944 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.582913+0000", 2020-01-16T00:14:17.944 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020814449999999999 2020-01-16T00:14:17.944 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.944 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.944 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:17.944 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.584994+0000", 2020-01-16T00:14:17.945 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001717243 2020-01-16T00:14:17.945 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.945 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.945 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:17.945 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.586711+0000", 2020-01-16T00:14:17.945 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041349869999999997 2020-01-16T00:14:17.945 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.946 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.946 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:17.946 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590846+0000", 2020-01-16T00:14:17.946 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3565000000000006e-05 2020-01-16T00:14:17.946 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.946 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.946 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:17.947 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590910+0000", 2020-01-16T00:14:17.947 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7002999999999997e-05 2020-01-16T00:14:17.947 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.947 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.947 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:17.947 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590957+0000", 2020-01-16T00:14:17.947 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011750764 2020-01-16T00:14:17.948 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.948 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:17.948 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.948 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.948 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.948 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:66 2.3 2:c3b82d87:::smithi12313934-44 low_tier:head [write 169472~818176] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e44)", 2020-01-16T00:14:17.948 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:25.278583+0000", 2020-01-16T00:14:17.949 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 52.560331091000002, 2020-01-16T00:14:17.949 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010294862, 2020-01-16T00:14:17.949 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:17.949 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:17.949 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:17.949 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.949 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:17.950 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.278583+0000", 2020-01-16T00:14:17.950 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:17.950 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.950 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.950 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:17.950 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.278583+0000", 2020-01-16T00:14:17.950 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:17.950 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.951 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.951 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:17.951 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.278581+0000", 2020-01-16T00:14:17.951 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00136721 2020-01-16T00:14:17.951 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.951 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.951 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:17.952 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.279948+0000", 2020-01-16T00:14:17.952 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6109999999999999e-05 2020-01-16T00:14:17.952 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.952 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.952 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:17.952 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.279974+0000", 2020-01-16T00:14:17.952 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.102e-05 2020-01-16T00:14:17.952 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.953 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:17.953 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.279985+0000", 2020-01-16T00:14:17.953 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6035e-05 2020-01-16T00:14:17.953 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.953 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.954 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.280061+0000", 2020-01-16T00:14:17.954 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000176243 2020-01-16T00:14:17.954 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.954 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.954 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:17.954 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.280238+0000", 2020-01-16T00:14:17.954 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019866099999999999 2020-01-16T00:14:17.955 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.955 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.955 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 7", 2020-01-16T00:14:17.955 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.280436+0000", 2020-01-16T00:14:17.955 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0049761830000000003 2020-01-16T00:14:17.955 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.955 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.955 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:17.956 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.285412+0000", 2020-01-16T00:14:17.956 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033347339999999998 2020-01-16T00:14:17.956 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.956 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.956 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:17.956 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.288747+0000", 2020-01-16T00:14:17.956 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9156000000000001e-05 2020-01-16T00:14:17.957 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.957 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.957 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:17.957 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.288806+0000", 2020-01-16T00:14:17.957 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1643000000000005e-05 2020-01-16T00:14:17.957 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.957 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.957 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:17.958 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.288878+0000", 2020-01-16T00:14:17.958 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010294862 2020-01-16T00:14:17.958 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.958 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:17.958 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.958 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.958 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.959 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:77 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 56832~1408000] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e45)", 2020-01-16T00:14:17.959 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.584566+0000", 2020-01-16T00:14:17.959 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 48.254348440000001, 2020-01-16T00:14:17.959 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017037775000000002, 2020-01-16T00:14:17.959 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:17.959 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:17.959 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:17.959 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.960 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:17.960 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584566+0000", 2020-01-16T00:14:17.960 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:17.960 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.960 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.960 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:17.960 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584566+0000", 2020-01-16T00:14:17.961 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:17.961 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.961 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.961 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:17.961 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584564+0000", 2020-01-16T00:14:17.961 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043024300000000002 2020-01-16T00:14:17.961 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.961 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.962 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:17.962 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584994+0000", 2020-01-16T00:14:17.962 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9485999999999999e-05 2020-01-16T00:14:17.962 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.962 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.962 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:17.962 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585014+0000", 2020-01-16T00:14:17.963 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8909999999999995e-06 2020-01-16T00:14:17.963 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.963 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.963 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:17.963 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585022+0000", 2020-01-16T00:14:17.963 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039114459999999998 2020-01-16T00:14:17.963 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.963 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.964 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.964 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588933+0000", 2020-01-16T00:14:17.964 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011178700000000001 2020-01-16T00:14:17.964 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.964 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.964 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:17.964 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589045+0000", 2020-01-16T00:14:17.964 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016967300000000001 2020-01-16T00:14:17.965 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.965 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.965 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:17.965 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589214+0000", 2020-01-16T00:14:17.965 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012142356 2020-01-16T00:14:17.965 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.965 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.966 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:17.966 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601357+0000", 2020-01-16T00:14:17.966 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.1466999999999997e-05 2020-01-16T00:14:17.966 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.966 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.966 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:17.966 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601448+0000", 2020-01-16T00:14:17.966 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4817000000000001e-05 2020-01-16T00:14:17.967 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.967 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.967 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:17.967 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601493+0000", 2020-01-16T00:14:17.967 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9734000000000001e-05 2020-01-16T00:14:17.967 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.967 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.968 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:17.968 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601553+0000", 2020-01-16T00:14:17.968 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0677999999999998e-05 2020-01-16T00:14:17.968 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.968 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.968 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:17.968 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601604+0000", 2020-01-16T00:14:17.968 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017037775000000002 2020-01-16T00:14:17.969 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.969 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:17.969 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.969 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.969 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.969 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:78 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 1922048~118272] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e45)", 2020-01-16T00:14:17.969 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.585030+0000", 2020-01-16T00:14:17.970 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 48.253883717999997, 2020-01-16T00:14:17.970 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016699964000000001, 2020-01-16T00:14:17.970 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:17.970 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:17.970 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:17.970 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.970 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:17.970 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585030+0000", 2020-01-16T00:14:17.971 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:17.971 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.971 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.971 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:17.971 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585030+0000", 2020-01-16T00:14:17.971 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:17.971 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.972 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.972 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:17.972 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585030+0000", 2020-01-16T00:14:17.972 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7085000000000001e-05 2020-01-16T00:14:17.972 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.972 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.972 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:17.972 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585077+0000", 2020-01-16T00:14:17.973 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5380000000000008e-06 2020-01-16T00:14:17.973 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.973 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.973 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:17.973 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585086+0000", 2020-01-16T00:14:17.973 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5410000000000002e-06 2020-01-16T00:14:17.973 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.974 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.974 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:17.974 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585091+0000", 2020-01-16T00:14:17.974 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0088486620000000002 2020-01-16T00:14:17.974 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.974 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.974 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.975 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.593940+0000", 2020-01-16T00:14:17.975 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112594 2020-01-16T00:14:17.975 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.975 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.975 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:17.975 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.594052+0000", 2020-01-16T00:14:17.975 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019893999999999999 2020-01-16T00:14:17.975 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.976 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.976 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:17.976 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.594251+0000", 2020-01-16T00:14:17.976 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0071632730000000004 2020-01-16T00:14:17.976 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.976 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.976 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:17.976 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601414+0000", 2020-01-16T00:14:17.977 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4672999999999996e-05 2020-01-16T00:14:17.977 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.977 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.977 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:17.977 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601479+0000", 2020-01-16T00:14:17.977 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000158453 2020-01-16T00:14:17.977 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.978 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.978 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:17.978 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601638+0000", 2020-01-16T00:14:17.978 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8605e-05 2020-01-16T00:14:17.978 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.978 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.978 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:17.979 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601686+0000", 2020-01-16T00:14:17.979 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4283999999999998e-05 2020-01-16T00:14:17.979 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.979 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.979 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:17.979 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601730+0000", 2020-01-16T00:14:17.979 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016699964000000001 2020-01-16T00:14:17.979 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.980 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:17.980 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.980 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.980 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.980 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:127 2.1 2:a4df40e2:::smithi12313934-42 low_tier:head [write 1364992~1145344] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e45)", 2020-01-16T00:14:17.980 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.627308+0000", 2020-01-16T00:14:17.980 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 48.211606265999997, 2020-01-16T00:14:17.981 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011068883999999999, 2020-01-16T00:14:17.981 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:17.981 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:17.981 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:17.981 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.981 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:17.981 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627308+0000", 2020-01-16T00:14:17.982 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:17.982 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.982 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.982 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:17.982 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627308+0000", 2020-01-16T00:14:17.982 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:17.982 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.982 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.983 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:17.983 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627307+0000", 2020-01-16T00:14:17.983 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000278597 2020-01-16T00:14:17.983 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.983 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.983 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:17.983 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627586+0000", 2020-01-16T00:14:17.984 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9169999999999997e-06 2020-01-16T00:14:17.984 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.984 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.984 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:17.984 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627591+0000", 2020-01-16T00:14:17.984 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0479999999999999e-06 2020-01-16T00:14:17.984 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.984 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.985 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:17.985 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627594+0000", 2020-01-16T00:14:17.985 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030987269999999999 2020-01-16T00:14:17.985 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.985 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.985 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.985 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.630693+0000", 2020-01-16T00:14:17.986 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000103688 2020-01-16T00:14:17.986 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.986 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.986 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:17.986 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.630796+0000", 2020-01-16T00:14:17.986 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000144606 2020-01-16T00:14:17.986 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.987 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.987 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:17.987 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.630941+0000", 2020-01-16T00:14:17.987 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039264269999999997 2020-01-16T00:14:17.987 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.987 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.987 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:17.988 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.634867+0000", 2020-01-16T00:14:17.988 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00038581200000000003 2020-01-16T00:14:17.988 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.988 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.988 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:17.988 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.635253+0000", 2020-01-16T00:14:17.988 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030568879999999998 2020-01-16T00:14:17.988 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.989 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.989 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:17.989 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.638310+0000", 2020-01-16T00:14:17.989 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8390999999999999e-05 2020-01-16T00:14:17.989 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.989 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.989 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:17.990 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.638348+0000", 2020-01-16T00:14:17.990 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8402999999999999e-05 2020-01-16T00:14:17.990 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.990 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.990 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:17.990 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.638377+0000", 2020-01-16T00:14:17.990 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011068883999999999 2020-01-16T00:14:17.990 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.991 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:17.991 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:17.991 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.991 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.991 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.6.13:30 2.3 2:fc791323:::smithi12313934-2 low_tier:head [write 476160~1240064] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e48)", 2020-01-16T00:14:17.991 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:39.466268+0000", 2020-01-16T00:14:17.991 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 38.372645962, 2020-01-16T00:14:17.992 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010593112, 2020-01-16T00:14:17.992 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:17.992 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:17.992 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:17.992 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.992 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:17.992 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.466268+0000", 2020-01-16T00:14:17.992 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:17.993 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.993 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.993 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:17.993 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.466268+0000", 2020-01-16T00:14:17.993 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:17.993 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.993 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.994 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:17.994 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.466266+0000", 2020-01-16T00:14:17.994 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012489210000000001 2020-01-16T00:14:17.994 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.994 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.994 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:17.994 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467515+0000", 2020-01-16T00:14:17.994 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8411999999999999e-05 2020-01-16T00:14:17.995 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.995 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.995 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:17.995 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467533+0000", 2020-01-16T00:14:17.995 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9319999999999996e-06 2020-01-16T00:14:17.995 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.995 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.995 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:17.996 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467541+0000", 2020-01-16T00:14:17.996 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5289999999999993e-05 2020-01-16T00:14:17.996 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.996 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.996 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:17.996 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467607+0000", 2020-01-16T00:14:17.996 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010962 2020-01-16T00:14:17.996 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.997 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.997 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:17.997 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467716+0000", 2020-01-16T00:14:17.997 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020883699999999999 2020-01-16T00:14:17.997 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.997 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.997 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 7", 2020-01-16T00:14:17.998 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467925+0000", 2020-01-16T00:14:17.998 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063872829999999997 2020-01-16T00:14:17.998 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.998 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.998 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:17.998 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.474312+0000", 2020-01-16T00:14:17.998 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0023165769999999998 2020-01-16T00:14:17.999 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.999 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.999 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:17.999 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.476629+0000", 2020-01-16T00:14:17.999 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8543999999999999e-05 2020-01-16T00:14:17.999 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:17.999 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:17.999 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:18.000 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.476707+0000", 2020-01-16T00:14:18.000 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000153914 2020-01-16T00:14:18.000 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.000 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.000 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:18.000 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.476861+0000", 2020-01-16T00:14:18.000 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010593112 2020-01-16T00:14:18.001 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:18.001 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:18.001 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:18.001 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.001 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.001 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:155 2.3 2:c9873fc7:::smithi12313934-30 low_tier:head [write 1542656~711168] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e50)", 2020-01-16T00:14:18.001 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:43.769943+0000", 2020-01-16T00:14:18.001 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 34.068971036000001, 2020-01-16T00:14:18.002 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010710127999999999, 2020-01-16T00:14:18.002 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:18.002 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:18.002 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:18.002 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.002 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:18.002 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.769943+0000", 2020-01-16T00:14:18.003 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:18.003 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.003 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.003 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:18.003 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.769943+0000", 2020-01-16T00:14:18.003 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:18.003 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.003 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.004 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:18.004 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.769942+0000", 2020-01-16T00:14:18.004 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017878100000000001 2020-01-16T00:14:18.004 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.004 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.004 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:18.004 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.770121+0000", 2020-01-16T00:14:18.004 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2980000000000003e-06 2020-01-16T00:14:18.005 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.005 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.005 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:18.005 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.770129+0000", 2020-01-16T00:14:18.005 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9129999999999999e-06 2020-01-16T00:14:18.005 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.005 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.006 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:18.006 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.770131+0000", 2020-01-16T00:14:18.006 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0053233519999999999 2020-01-16T00:14:18.006 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.006 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.006 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:18.006 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.775455+0000", 2020-01-16T00:14:18.006 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012385099999999999 2020-01-16T00:14:18.007 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.007 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.007 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:18.007 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.775579+0000", 2020-01-16T00:14:18.007 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000211554 2020-01-16T00:14:18.007 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.007 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.007 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 7", 2020-01-16T00:14:18.008 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.775790+0000", 2020-01-16T00:14:18.008 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029614210000000001 2020-01-16T00:14:18.008 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.008 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.008 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:18.008 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.778752+0000", 2020-01-16T00:14:18.008 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001798928 2020-01-16T00:14:18.009 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.009 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.009 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:18.009 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.780551+0000", 2020-01-16T00:14:18.009 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9413999999999999e-05 2020-01-16T00:14:18.009 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.009 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.009 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:18.010 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.780590+0000", 2020-01-16T00:14:18.010 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3275999999999995e-05 2020-01-16T00:14:18.010 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.010 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.010 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:18.010 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.780653+0000", 2020-01-16T00:14:18.010 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010710127999999999 2020-01-16T00:14:18.011 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:18.011 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:18.011 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:18.011 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.011 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.011 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:171 2.1 2:a935ad67:::smithi12313934-50 low_tier:head [write 2422272~237568] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e55)", 2020-01-16T00:14:18.011 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:58.196749+0000", 2020-01-16T00:14:18.012 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 19.642164750999999, 2020-01-16T00:14:18.012 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010677759, 2020-01-16T00:14:18.012 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:18.012 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:18.012 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:18.012 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.012 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:18.012 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196749+0000", 2020-01-16T00:14:18.013 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:18.013 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.013 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.013 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:18.013 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196749+0000", 2020-01-16T00:14:18.013 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:18.013 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.014 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.014 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:18.014 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196749+0000", 2020-01-16T00:14:18.014 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6604000000000001e-05 2020-01-16T00:14:18.014 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.014 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.014 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:18.015 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196815+0000", 2020-01-16T00:14:18.015 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4900000000000006e-06 2020-01-16T00:14:18.015 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.015 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.015 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:18.015 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196825+0000", 2020-01-16T00:14:18.015 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3989999999999997e-06 2020-01-16T00:14:18.016 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.016 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.016 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:18.016 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196829+0000", 2020-01-16T00:14:18.016 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0076943869999999996 2020-01-16T00:14:18.016 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.016 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.016 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:18.017 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.204524+0000", 2020-01-16T00:14:18.017 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000124745 2020-01-16T00:14:18.017 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.017 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.017 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:18.017 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.204649+0000", 2020-01-16T00:14:18.017 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018801700000000001 2020-01-16T00:14:18.018 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.018 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.018 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:18.018 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.204837+0000", 2020-01-16T00:14:18.018 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002354765 2020-01-16T00:14:18.018 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.018 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.018 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:18.019 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.207191+0000", 2020-01-16T00:14:18.019 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.6249999999999996e-06 2020-01-16T00:14:18.019 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.019 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.019 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:18.019 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.207200+0000", 2020-01-16T00:14:18.019 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000162246 2020-01-16T00:14:18.019 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.020 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.020 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:18.020 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.207362+0000", 2020-01-16T00:14:18.020 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5117999999999998e-05 2020-01-16T00:14:18.020 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.020 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.020 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:18.021 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.207397+0000", 2020-01-16T00:14:18.021 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9912000000000001e-05 2020-01-16T00:14:18.021 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.021 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.021 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:18.021 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.207427+0000", 2020-01-16T00:14:18.021 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010677759 2020-01-16T00:14:18.021 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:18.022 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:18.022 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:18.022 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.022 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.022 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:183 2.1 2:a4df40e2:::smithi12313934-42 low_tier:head [write 70656~727552] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e58)", 2020-01-16T00:14:18.022 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.021222+0000", 2020-01-16T00:14:18.022 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 10.817692182, 2020-01-16T00:14:18.023 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012681151999999999, 2020-01-16T00:14:18.023 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:18.023 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:18.023 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:18.023 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.023 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:18.023 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021222+0000", 2020-01-16T00:14:18.023 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:18.024 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.024 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.024 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:18.024 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021222+0000", 2020-01-16T00:14:18.024 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:18.024 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.024 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.024 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:18.025 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021219+0000", 2020-01-16T00:14:18.025 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020912099999999999 2020-01-16T00:14:18.025 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.025 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.025 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:18.025 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021428+0000", 2020-01-16T00:14:18.025 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8595999999999999e-05 2020-01-16T00:14:18.026 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.026 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.026 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:18.026 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021447+0000", 2020-01-16T00:14:18.026 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2020000000000001e-06 2020-01-16T00:14:18.026 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.026 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.027 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:18.027 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021453+0000", 2020-01-16T00:14:18.027 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5297999999999999e-05 2020-01-16T00:14:18.027 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.027 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.027 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:18.027 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021499+0000", 2020-01-16T00:14:18.027 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2299999999999996e-05 2020-01-16T00:14:18.028 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.028 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.028 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:18.028 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021571+0000", 2020-01-16T00:14:18.028 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011000799999999999 2020-01-16T00:14:18.028 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.028 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.029 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:18.029 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021681+0000", 2020-01-16T00:14:18.029 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0065924879999999996 2020-01-16T00:14:18.029 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.029 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.029 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:18.029 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.028273+0000", 2020-01-16T00:14:18.030 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022212499999999999 2020-01-16T00:14:18.030 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.030 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.030 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:18.030 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.028495+0000", 2020-01-16T00:14:18.030 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0053014580000000002 2020-01-16T00:14:18.030 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.031 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.031 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:18.031 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.033797+0000", 2020-01-16T00:14:18.031 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6142e-05 2020-01-16T00:14:18.031 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.031 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.031 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:18.032 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.033843+0000", 2020-01-16T00:14:18.032 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0130999999999997e-05 2020-01-16T00:14:18.032 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.032 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.032 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:18.032 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.033903+0000", 2020-01-16T00:14:18.032 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012681151999999999 2020-01-16T00:14:18.032 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:18.033 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:18.033 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:18.033 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.033 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.033 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:184 2.1 2:a4df40e2:::smithi12313934-42 low_tier:head [write 1364992~1145344] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e58)", 2020-01-16T00:14:18.033 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.023539+0000", 2020-01-16T00:14:18.033 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 10.815375062999999, 2020-01-16T00:14:18.033 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011003893000000001, 2020-01-16T00:14:18.034 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:18.034 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:18.034 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:18.034 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.034 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:18.034 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023539+0000", 2020-01-16T00:14:18.034 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:18.035 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.035 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.035 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:18.035 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023539+0000", 2020-01-16T00:14:18.035 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:18.035 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.035 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.035 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:18.036 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023537+0000", 2020-01-16T00:14:18.036 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00033271700000000001 2020-01-16T00:14:18.036 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.036 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.036 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:18.036 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023870+0000", 2020-01-16T00:14:18.036 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5744999999999999e-05 2020-01-16T00:14:18.037 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.037 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.037 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:18.037 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023886+0000", 2020-01-16T00:14:18.037 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7389999999999994e-06 2020-01-16T00:14:18.037 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.037 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.038 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:18.038 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023895+0000", 2020-01-16T00:14:18.038 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6058999999999998e-05 2020-01-16T00:14:18.038 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.038 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.038 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:18.038 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023971+0000", 2020-01-16T00:14:18.038 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119207 2020-01-16T00:14:18.039 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.039 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.039 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:18.039 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.024091+0000", 2020-01-16T00:14:18.039 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000166074 2020-01-16T00:14:18.039 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.039 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.040 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:18.040 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.024257+0000", 2020-01-16T00:14:18.040 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0047290129999999998 2020-01-16T00:14:18.040 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.040 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.040 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:18.040 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.028986+0000", 2020-01-16T00:14:18.040 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00028545000000000003 2020-01-16T00:14:18.041 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.041 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.041 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:18.041 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.029271+0000", 2020-01-16T00:14:18.041 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052013049999999998 2020-01-16T00:14:18.041 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.041 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.042 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:18.042 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.034472+0000", 2020-01-16T00:14:18.042 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7738999999999999e-05 2020-01-16T00:14:18.042 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.042 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.042 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:18.043 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.034510+0000", 2020-01-16T00:14:18.043 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2913000000000001e-05 2020-01-16T00:14:18.043 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.043 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.043 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:18.043 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.034543+0000", 2020-01-16T00:14:18.043 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011003893000000001 2020-01-16T00:14:18.043 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:18.044 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:18.044 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:18.044 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.044 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.044 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:98 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 56832~1408000] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e58)", 2020-01-16T00:14:18.044 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.049295+0000", 2020-01-16T00:14:18.044 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 10.789619012999999, 2020-01-16T00:14:18.045 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012327324000000001, 2020-01-16T00:14:18.045 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:18.045 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:18.045 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:18.045 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.045 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:18.045 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049295+0000", 2020-01-16T00:14:18.046 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:18.046 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.046 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.046 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:18.046 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049295+0000", 2020-01-16T00:14:18.046 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:18.046 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.047 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.047 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:18.047 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049294+0000", 2020-01-16T00:14:18.047 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00036515000000000001 2020-01-16T00:14:18.047 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.047 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.047 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:18.048 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049659+0000", 2020-01-16T00:14:18.048 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4299e-05 2020-01-16T00:14:18.048 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.048 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.048 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:18.048 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049673+0000", 2020-01-16T00:14:18.048 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1250000000000001e-06 2020-01-16T00:14:18.048 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.049 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.049 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:18.049 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049678+0000", 2020-01-16T00:14:18.049 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9937e-05 2020-01-16T00:14:18.049 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.049 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.049 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:18.050 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049728+0000", 2020-01-16T00:14:18.050 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.1094999999999994e-05 2020-01-16T00:14:18.050 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.050 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.050 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:18.050 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049819+0000", 2020-01-16T00:14:18.050 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012621999999999999 2020-01-16T00:14:18.051 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.051 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.051 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:18.051 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049946+0000", 2020-01-16T00:14:18.051 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0081844340000000008 2020-01-16T00:14:18.051 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.051 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.052 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:18.052 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.058130+0000", 2020-01-16T00:14:18.052 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00058478000000000002 2020-01-16T00:14:18.052 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.052 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.052 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:18.052 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.058715+0000", 2020-01-16T00:14:18.053 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0027436349999999999 2020-01-16T00:14:18.053 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.053 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.053 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:18.053 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.061459+0000", 2020-01-16T00:14:18.053 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1755000000000007e-05 2020-01-16T00:14:18.053 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.053 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.054 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:18.054 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.061530+0000", 2020-01-16T00:14:18.054 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2236000000000003e-05 2020-01-16T00:14:18.054 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.054 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.054 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:18.054 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.061623+0000", 2020-01-16T00:14:18.055 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012327324000000001 2020-01-16T00:14:18.055 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:18.055 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:18.055 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:18.055 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.055 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.055 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:99 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 1922048~118272] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e58)", 2020-01-16T00:14:18.056 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.049687+0000", 2020-01-16T00:14:18.056 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 10.789227035, 2020-01-16T00:14:18.056 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012484960999999999, 2020-01-16T00:14:18.056 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:18.056 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:18.056 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:18.056 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.056 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:18.057 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049687+0000", 2020-01-16T00:14:18.057 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:18.057 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.057 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.057 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:18.057 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049687+0000", 2020-01-16T00:14:18.058 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:18.058 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.058 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.058 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:18.058 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049687+0000", 2020-01-16T00:14:18.058 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7373999999999999e-05 2020-01-16T00:14:18.058 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.058 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.059 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:18.059 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049714+0000", 2020-01-16T00:14:18.059 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2030999999999997e-05 2020-01-16T00:14:18.059 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.059 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.059 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:18.059 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049766+0000", 2020-01-16T00:14:18.059 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5559999999999999e-06 2020-01-16T00:14:18.060 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.060 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.060 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:18.060 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049770+0000", 2020-01-16T00:14:18.060 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0026935779999999999 2020-01-16T00:14:18.060 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.060 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.061 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:18.061 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.052463+0000", 2020-01-16T00:14:18.061 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5037999999999995e-05 2020-01-16T00:14:18.061 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.061 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.061 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:18.061 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.052538+0000", 2020-01-16T00:14:18.061 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000116791 2020-01-16T00:14:18.062 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.062 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.062 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:18.062 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.052655+0000", 2020-01-16T00:14:18.062 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063608579999999996 2020-01-16T00:14:18.062 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.062 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.063 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:18.063 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.059016+0000", 2020-01-16T00:14:18.063 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000131872 2020-01-16T00:14:18.063 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.063 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.063 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:18.063 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.059148+0000", 2020-01-16T00:14:18.063 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002907677 2020-01-16T00:14:18.064 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.064 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.064 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:18.064 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.062055+0000", 2020-01-16T00:14:18.064 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9624999999999999e-05 2020-01-16T00:14:18.064 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.064 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.065 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:18.065 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.062115+0000", 2020-01-16T00:14:18.065 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7096999999999999e-05 2020-01-16T00:14:18.065 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:18.065 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:18.065 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:18.065 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.062172+0000", 2020-01-16T00:14:18.065 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012484960999999999 2020-01-16T00:14:18.066 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:18.066 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:18.066 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:18.066 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:18.066 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:18.066 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:18.067 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2020-01-16T00:14:18.069 INFO:tasks.rados.rados.0.smithi123.stdout:1801: finishing write tid 1 to smithi12313934-22 2020-01-16T00:14:18.069 INFO:tasks.rados.rados.0.smithi123.stdout:1800: expect (ObjNum 638 snap 0 seq_num 638) 2020-01-16T00:14:18.069 INFO:tasks.rados.rados.0.smithi123.stdout:1801: finishing write tid 2 to smithi12313934-22 2020-01-16T00:14:18.069 INFO:tasks.rados.rados.0.smithi123.stdout:1801: finishing write tid 3 to smithi12313934-22 2020-01-16T00:14:18.069 INFO:tasks.rados.rados.0.smithi123.stdout:1801: finishing write tid 5 to smithi12313934-22 2020-01-16T00:14:18.069 INFO:tasks.rados.rados.0.smithi123.stdout:1801: finishing write tid 6 to smithi12313934-22 2020-01-16T00:14:18.069 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 22 v 41 (ObjNum 661 snap 0 seq_num 661) dirty exists 2020-01-16T00:14:18.070 INFO:tasks.rados.rados.0.smithi123.stdout:1801: left oid 22 (ObjNum 661 snap 0 seq_num 661) 2020-01-16T00:14:18.070 INFO:tasks.rados.rados.0.smithi123.stdout:1802: finishing write tid 1 to smithi12313934-14 2020-01-16T00:14:18.070 INFO:tasks.rados.rados.0.smithi123.stdout:1802: finishing write tid 2 to smithi12313934-14 2020-01-16T00:14:18.070 INFO:tasks.rados.rados.0.smithi123.stdout:1802: finishing write tid 4 to smithi12313934-14 2020-01-16T00:14:18.070 INFO:tasks.rados.rados.0.smithi123.stdout:1802: finishing write tid 5 to smithi12313934-14 2020-01-16T00:14:18.070 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 14 v 288 (ObjNum 662 snap 0 seq_num 662) dirty exists 2020-01-16T00:14:18.070 INFO:tasks.rados.rados.0.smithi123.stdout:1802: left oid 14 (ObjNum 662 snap 0 seq_num 662) 2020-01-16T00:14:18.071 INFO:tasks.rados.rados.0.smithi123.stdout:1803: expect (ObjNum 602 snap 0 seq_num 602) 2020-01-16T00:14:18.071 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:18.066+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:18.071 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-01-16T00:14:18.155 INFO:tasks.rados.rados.0.smithi123.stdout:1804: finishing write tid 1 to smithi12313934-10 2020-01-16T00:14:18.156 INFO:tasks.rados.rados.0.smithi123.stdout:1804: finishing write tid 2 to smithi12313934-10 2020-01-16T00:14:18.156 INFO:tasks.rados.rados.0.smithi123.stdout:1804: finishing write tid 3 to smithi12313934-10 2020-01-16T00:14:18.156 INFO:tasks.rados.rados.0.smithi123.stdout:1804: finishing write tid 5 to smithi12313934-10 2020-01-16T00:14:18.156 INFO:tasks.rados.rados.0.smithi123.stdout:1804: finishing write tid 6 to smithi12313934-10 2020-01-16T00:14:18.156 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 10 v 190 (ObjNum 663 snap 0 seq_num 663) dirty exists 2020-01-16T00:14:18.156 INFO:tasks.rados.rados.0.smithi123.stdout:1804: left oid 10 (ObjNum 663 snap 0 seq_num 663) 2020-01-16T00:14:18.156 INFO:tasks.rados.rados.0.smithi123.stdout:1805: finishing write tid 1 to smithi12313934-19 2020-01-16T00:14:18.157 INFO:tasks.rados.rados.0.smithi123.stdout:1805: finishing write tid 2 to smithi12313934-19 2020-01-16T00:14:18.157 INFO:tasks.rados.rados.0.smithi123.stdout:1805: finishing write tid 4 to smithi12313934-19 2020-01-16T00:14:18.157 INFO:tasks.rados.rados.0.smithi123.stdout:1793: done (15 left) 2020-01-16T00:14:18.157 INFO:tasks.rados.rados.0.smithi123.stdout:1794: done (14 left) 2020-01-16T00:14:18.157 INFO:tasks.rados.rados.0.smithi123.stdout:1795: done (13 left) 2020-01-16T00:14:18.157 INFO:tasks.rados.rados.0.smithi123.stdout:1796: done (12 left) 2020-01-16T00:14:18.157 INFO:tasks.rados.rados.0.smithi123.stdout:1797: done (11 left) 2020-01-16T00:14:18.158 INFO:tasks.rados.rados.0.smithi123.stdout:1798: done (10 left) 2020-01-16T00:14:18.158 INFO:tasks.rados.rados.0.smithi123.stdout:1799: done (9 left) 2020-01-16T00:14:18.158 INFO:tasks.rados.rados.0.smithi123.stdout:1800: done (8 left) 2020-01-16T00:14:18.158 INFO:tasks.rados.rados.0.smithi123.stdout:1801: done (7 left) 2020-01-16T00:14:18.158 INFO:tasks.rados.rados.0.smithi123.stdout:1802: done (6 left) 2020-01-16T00:14:18.158 INFO:tasks.rados.rados.0.smithi123.stdout:1803: done (5 left) 2020-01-16T00:14:18.158 INFO:tasks.rados.rados.0.smithi123.stdout:1804: done (4 left) 2020-01-16T00:14:18.158 INFO:tasks.rados.rados.0.smithi123.stdout:1809: read oid 45 snap -1 2020-01-16T00:14:18.159 INFO:tasks.rados.rados.0.smithi123.stdout:1809: expect (ObjNum 639 snap 0 seq_num 639) 2020-01-16T00:14:18.159 INFO:tasks.rados.rados.0.smithi123.stdout:1810: read oid 12 snap -1 2020-01-16T00:14:18.159 INFO:tasks.rados.rados.0.smithi123.stdout:1810: expect (ObjNum 655 snap 0 seq_num 655) 2020-01-16T00:14:18.159 INFO:tasks.rados.rados.0.smithi123.stdout:1811: read oid 16 snap -1 2020-01-16T00:14:18.159 INFO:tasks.rados.rados.0.smithi123.stdout:1811: expect (ObjNum 660 snap 0 seq_num 660) 2020-01-16T00:14:18.159 INFO:tasks.rados.rados.0.smithi123.stdout:1812: read oid 9 snap -1 2020-01-16T00:14:18.159 INFO:tasks.rados.rados.0.smithi123.stdout:1812: expect (ObjNum 633 snap 0 seq_num 633) 2020-01-16T00:14:18.160 INFO:tasks.rados.rados.0.smithi123.stdout:1813: read oid 50 snap -1 2020-01-16T00:14:18.160 INFO:tasks.rados.rados.0.smithi123.stdout:1813: expect (ObjNum 595 snap 0 seq_num 595) 2020-01-16T00:14:18.160 INFO:tasks.rados.rados.0.smithi123.stdout:1814: read oid 24 snap -1 2020-01-16T00:14:18.160 INFO:tasks.rados.rados.0.smithi123.stdout:1814: expect (ObjNum 630 snap 0 seq_num 630) 2020-01-16T00:14:18.160 INFO:tasks.rados.rados.0.smithi123.stdout:1815: write (excl) oid 42 current snap is 0 2020-01-16T00:14:18.160 INFO:tasks.rados.rados.0.smithi123.stdout:1815: seq_num 666 ranges {793462=638545,2106184=763893} 2020-01-16T00:14:18.169 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:18.164+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:18.175 INFO:tasks.rados.rados.0.smithi123.stdout:1815: writing smithi12313934-42 from 793462 to 1432007 tid 1 2020-01-16T00:14:18.183 INFO:tasks.rados.rados.0.smithi123.stdout:1815: writing smithi12313934-42 from 2106184 to 2870077 tid 2 2020-01-16T00:14:18.183 INFO:tasks.rados.rados.0.smithi123.stdout:1816: write (excl) oid 2 current snap is 0 2020-01-16T00:14:18.183 INFO:tasks.rados.rados.0.smithi123.stdout:1816: seq_num 667 ranges {410269=713794,1648313=507551,2917489=292780} 2020-01-16T00:14:18.189 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:18.190 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:14:18.190 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:14:18.190 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:18.190 INFO:tasks.rados.rados.0.smithi123.stdout:1816: writing smithi12313934-2 from 410269 to 1124063 tid 1 2020-01-16T00:14:18.196 INFO:tasks.rados.rados.0.smithi123.stdout:1816: writing smithi12313934-2 from 1648313 to 2155864 tid 2 2020-01-16T00:14:18.202 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-01-16T00:14:18.203 INFO:tasks.rados.rados.0.smithi123.stdout:1816: writing smithi12313934-2 from 2917489 to 3210269 tid 3 2020-01-16T00:14:18.203 INFO:tasks.rados.rados.0.smithi123.stdout:1817: write (excl) oid 38 current snap is 0 2020-01-16T00:14:18.203 INFO:tasks.rados.rados.0.smithi123.stdout:1817: seq_num 668 ranges {427076=789043,1706795=767562,2944912=282164} 2020-01-16T00:14:18.216 INFO:tasks.rados.rados.0.smithi123.stdout:1817: writing smithi12313934-38 from 427076 to 1216119 tid 1 2020-01-16T00:14:18.216 INFO:tasks.rados.rados.0.smithi123.stdout:1817: writing smithi12313934-38 from 1706795 to 2474357 tid 2 2020-01-16T00:14:18.221 INFO:tasks.rados.rados.0.smithi123.stdout:1817: writing smithi12313934-38 from 2944912 to 3227076 tid 3 2020-01-16T00:14:18.221 INFO:tasks.rados.rados.0.smithi123.stdout:1818: write oid 40 current snap is 0 2020-01-16T00:14:18.221 INFO:tasks.rados.rados.0.smithi123.stdout:1818: seq_num 669 ranges {443883=464292,1365277=511220,2572335=671548} 2020-01-16T00:14:18.226 INFO:tasks.rados.rados.0.smithi123.stdout:1818: writing smithi12313934-40 from 443883 to 908175 tid 1 2020-01-16T00:14:18.232 INFO:tasks.rados.rados.0.smithi123.stdout:1818: writing smithi12313934-40 from 1365277 to 1876497 tid 2 2020-01-16T00:14:18.241 INFO:tasks.rados.rados.0.smithi123.stdout:1818: writing smithi12313934-40 from 2572335 to 3243883 tid 3 2020-01-16T00:14:18.241 INFO:tasks.rados.rados.0.smithi123.stdout:1819: read oid 36 snap -1 2020-01-16T00:14:18.242 INFO:tasks.rados.rados.0.smithi123.stdout:1819: expect deleted 2020-01-16T00:14:18.242 INFO:tasks.rados.rados.0.smithi123.stdout:1820: read oid 3 snap -1 2020-01-16T00:14:18.242 INFO:tasks.rados.rados.0.smithi123.stdout:1820: expect (ObjNum 653 snap 0 seq_num 653) 2020-01-16T00:14:18.242 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:18.242 INFO:tasks.rados.rados.0.smithi123.stdout:1805: finishing write tid 5 to smithi12313934-19 2020-01-16T00:14:18.242 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 19 v 251 (ObjNum 664 snap 0 seq_num 664) dirty exists 2020-01-16T00:14:18.242 INFO:tasks.rados.rados.0.smithi123.stdout:1805: left oid 19 (ObjNum 664 snap 0 seq_num 664) 2020-01-16T00:14:18.242 INFO:tasks.rados.rados.0.smithi123.stdout:1805: done (15 left) 2020-01-16T00:14:18.243 INFO:tasks.rados.rados.0.smithi123.stdout:1821: read oid 35 snap -1 2020-01-16T00:14:18.243 INFO:tasks.rados.rados.0.smithi123.stdout:1821: expect (ObjNum 628 snap 0 seq_num 628) 2020-01-16T00:14:18.243 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:18.243 INFO:tasks.rados.rados.0.smithi123.stdout:1806: finishing write tid 1 to smithi12313934-46 2020-01-16T00:14:18.243 INFO:tasks.rados.rados.0.smithi123.stdout:1806: finishing write tid 2 to smithi12313934-46 2020-01-16T00:14:18.243 INFO:tasks.rados.rados.0.smithi123.stdout:1806: finishing write tid 3 to smithi12313934-46 2020-01-16T00:14:18.243 INFO:tasks.rados.rados.0.smithi123.stdout:1806: finishing write tid 5 to smithi12313934-46 2020-01-16T00:14:18.244 INFO:tasks.rados.rados.0.smithi123.stdout:1806: finishing write tid 6 to smithi12313934-46 2020-01-16T00:14:18.244 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 46 v 344 (ObjNum 665 snap 0 seq_num 665) dirty exists 2020-01-16T00:14:18.244 INFO:tasks.rados.rados.0.smithi123.stdout:1806: left oid 46 (ObjNum 665 snap 0 seq_num 665) 2020-01-16T00:14:18.244 INFO:tasks.rados.rados.0.smithi123.stdout:1806: done (15 left) 2020-01-16T00:14:18.244 INFO:tasks.rados.rados.0.smithi123.stdout:1822: read oid 7 snap -1 2020-01-16T00:14:18.245 INFO:tasks.rados.rados.0.smithi123.stdout:1822: expect (ObjNum 614 snap 0 seq_num 614) 2020-01-16T00:14:18.245 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:18.245 INFO:tasks.rados.rados.0.smithi123.stdout:1807: expect (ObjNum 618 snap 0 seq_num 618) 2020-01-16T00:14:18.270 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:18.266+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:18.293 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-01-16T00:14:18.343 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:18.343 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:14:18.344 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:14:18.344 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:14:18.344 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:18.352 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-01-16T00:14:18.371 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:18.370+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:18.433 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:14:18.452 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:14:18.454 INFO:tasks.rados.rados.0.smithi123.stdout:1808: expect (ObjNum 607 snap 0 seq_num 607) 2020-01-16T00:14:18.463 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:18.463 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:14:18.463 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:14:18.463 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:14:18.463 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.463 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:943 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:14:18.464 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:14:18.464 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 100.544562363, 2020-01-16T00:14:18.464 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018847004000000001, 2020-01-16T00:14:18.464 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:18.464 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:18.464 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:18.464 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:18.465 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:18.465 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 943 2020-01-16T00:14:18.465 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.465 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:18.465 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.465 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:18.465 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:14:18.466 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:18.466 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.466 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.466 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:18.466 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:14:18.467 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:18.467 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.467 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.467 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:18.467 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914475+0000", 2020-01-16T00:14:18.467 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.508e-05 2020-01-16T00:14:18.467 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.468 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.468 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:18.468 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914490+0000", 2020-01-16T00:14:18.468 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.857e-05 2020-01-16T00:14:18.468 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.468 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.468 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:18.468 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914509+0000", 2020-01-16T00:14:18.469 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.46e-06 2020-01-16T00:14:18.469 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.469 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.469 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:18.469 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914515+0000", 2020-01-16T00:14:18.469 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2612999999999999e-05 2020-01-16T00:14:18.469 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.470 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.470 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.470 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914568+0000", 2020-01-16T00:14:18.470 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028312299999999998 2020-01-16T00:14:18.470 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.470 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.470 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:18.470 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914851+0000", 2020-01-16T00:14:18.471 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012282533 2020-01-16T00:14:18.471 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.471 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.471 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.471 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927133+0000", 2020-01-16T00:14:18.471 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4839999999999995e-05 2020-01-16T00:14:18.471 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.478 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.478 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:18.479 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927208+0000", 2020-01-16T00:14:18.481 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0047425610000000002 2020-01-16T00:14:18.481 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.481 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.482 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.482 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.931951+0000", 2020-01-16T00:14:18.490 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3628999999999994e-05 2020-01-16T00:14:18.490 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.490 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.491 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:18.491 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.932045+0000", 2020-01-16T00:14:18.491 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012810180000000001 2020-01-16T00:14:18.491 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.491 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.491 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:18.491 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933326+0000", 2020-01-16T00:14:18.492 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018847004000000001 2020-01-16T00:14:18.492 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.492 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:18.492 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.493 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.493 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.493 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:944 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:14:18.494 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:14:18.494 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 100.544512807, 2020-01-16T00:14:18.494 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019086901999999999, 2020-01-16T00:14:18.494 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:18.494 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:18.494 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:18.494 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:18.495 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:18.495 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 944 2020-01-16T00:14:18.495 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.495 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:18.495 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.495 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:18.495 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:14:18.496 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:18.496 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.496 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.496 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:18.496 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:14:18.496 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:18.496 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.496 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.497 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:18.497 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914527+0000", 2020-01-16T00:14:18.497 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7509999999999999e-06 2020-01-16T00:14:18.497 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.497 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.497 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:18.497 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914532+0000", 2020-01-16T00:14:18.498 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5149999999999997e-06 2020-01-16T00:14:18.498 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.498 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.498 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:18.498 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914538+0000", 2020-01-16T00:14:18.498 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1890000000000007e-06 2020-01-16T00:14:18.498 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.498 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.499 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:18.499 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914546+0000", 2020-01-16T00:14:18.499 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000354951 2020-01-16T00:14:18.499 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.499 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.499 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.499 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914901+0000", 2020-01-16T00:14:18.499 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1594000000000002e-05 2020-01-16T00:14:18.500 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.500 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.500 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:18.500 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914942+0000", 2020-01-16T00:14:18.500 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012300070999999999 2020-01-16T00:14:18.500 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.500 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.501 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.501 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927242+0000", 2020-01-16T00:14:18.501 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4076999999999997e-05 2020-01-16T00:14:18.501 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.503 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.503 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:18.503 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927296+0000", 2020-01-16T00:14:18.503 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0060609180000000002 2020-01-16T00:14:18.503 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.503 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.504 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.504 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933357+0000", 2020-01-16T00:14:18.504 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4488999999999999e-05 2020-01-16T00:14:18.504 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.504 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.504 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:18.504 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933422+0000", 2020-01-16T00:14:18.504 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019318300000000001 2020-01-16T00:14:18.505 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.505 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.505 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:18.505 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933615+0000", 2020-01-16T00:14:18.505 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019086901999999999 2020-01-16T00:14:18.505 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.505 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:18.506 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.506 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.506 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.506 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:945 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:14:18.506 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:14:18.506 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 100.54448560100001, 2020-01-16T00:14:18.506 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019342986, 2020-01-16T00:14:18.506 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:18.507 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:18.507 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:18.507 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:18.507 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:18.507 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 945 2020-01-16T00:14:18.507 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.507 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:18.507 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.508 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:18.508 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:14:18.508 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:18.508 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.508 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.508 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:18.508 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:14:18.508 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:18.509 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.509 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.509 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:18.509 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:14:18.509 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4580000000000002e-06 2020-01-16T00:14:18.509 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.511 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.511 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:18.511 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914558+0000", 2020-01-16T00:14:18.512 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7829999999999998e-06 2020-01-16T00:14:18.512 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.512 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.512 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:18.512 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914563+0000", 2020-01-16T00:14:18.512 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0001e-05 2020-01-16T00:14:18.512 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.512 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.513 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:18.518 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914573+0000", 2020-01-16T00:14:18.518 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00040365600000000001 2020-01-16T00:14:18.518 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.518 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.518 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.518 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914976+0000", 2020-01-16T00:14:18.518 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1390000000000001e-05 2020-01-16T00:14:18.519 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.519 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.519 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:18.519 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.915028+0000", 2020-01-16T00:14:18.519 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012288843000000001 2020-01-16T00:14:18.519 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.519 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.520 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.520 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927317+0000", 2020-01-16T00:14:18.520 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7871000000000002e-05 2020-01-16T00:14:18.520 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.520 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.520 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:18.520 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927375+0000", 2020-01-16T00:14:18.520 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0062580589999999998 2020-01-16T00:14:18.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.521 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.521 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.521 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933633+0000", 2020-01-16T00:14:18.521 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7086999999999997e-05 2020-01-16T00:14:18.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.521 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.522 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:18.522 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933690+0000", 2020-01-16T00:14:18.522 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020855799999999999 2020-01-16T00:14:18.522 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.522 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.522 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:18.522 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933898+0000", 2020-01-16T00:14:18.522 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019342986 2020-01-16T00:14:18.523 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.523 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:18.523 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.523 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.523 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.523 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:981 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:14:18.523 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:14:18.524 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 100.14370870899999, 2020-01-16T00:14:18.524 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021222494000000001, 2020-01-16T00:14:18.524 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:18.524 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:18.524 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:18.524 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:18.524 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:18.524 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 981 2020-01-16T00:14:18.525 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.525 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:18.525 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.525 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:18.525 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:14:18.526 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:18.526 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.526 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.526 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:18.526 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:14:18.526 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999948 2020-01-16T00:14:18.526 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.527 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.527 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:18.527 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315327+0000", 2020-01-16T00:14:18.527 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.508e-05 2020-01-16T00:14:18.527 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.527 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.527 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:18.527 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315352+0000", 2020-01-16T00:14:18.528 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1183999999999998e-05 2020-01-16T00:14:18.528 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.528 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.528 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:18.528 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315383+0000", 2020-01-16T00:14:18.528 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7180000000000002e-05 2020-01-16T00:14:18.528 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.529 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.529 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:18.529 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315401+0000", 2020-01-16T00:14:18.529 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017290599999999999 2020-01-16T00:14:18.529 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.529 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.529 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.529 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315574+0000", 2020-01-16T00:14:18.530 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00047908100000000002 2020-01-16T00:14:18.530 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.530 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.530 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:18.530 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316053+0000", 2020-01-16T00:14:18.530 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011808947 2020-01-16T00:14:18.530 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.531 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.531 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.531 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.327862+0000", 2020-01-16T00:14:18.531 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010067499999999999 2020-01-16T00:14:18.531 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.531 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.531 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:18.531 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.327962+0000", 2020-01-16T00:14:18.532 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0074745139999999998 2020-01-16T00:14:18.532 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.532 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.532 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.532 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.335437+0000", 2020-01-16T00:14:18.532 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000103403 2020-01-16T00:14:18.532 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.532 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.533 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:18.533 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.335540+0000", 2020-01-16T00:14:18.533 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001014544 2020-01-16T00:14:18.533 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.533 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.533 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:18.533 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336555+0000", 2020-01-16T00:14:18.534 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021222494000000001 2020-01-16T00:14:18.534 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.534 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:18.534 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.534 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.534 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.534 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:982 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:14:18.534 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:14:18.535 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 100.143606185, 2020-01-16T00:14:18.535 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.022087721000000001, 2020-01-16T00:14:18.535 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:18.535 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:18.535 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:18.536 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:18.536 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:18.536 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 982 2020-01-16T00:14:18.536 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.536 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:18.536 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.536 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:18.537 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:14:18.537 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:18.537 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.537 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.537 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:18.537 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:14:18.537 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:18.537 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.538 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.538 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:18.538 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315433+0000", 2020-01-16T00:14:18.538 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1347e-05 2020-01-16T00:14:18.538 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.538 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.538 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:18.539 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315445+0000", 2020-01-16T00:14:18.539 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4525999999999999e-05 2020-01-16T00:14:18.539 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.541 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.541 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:18.541 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315459+0000", 2020-01-16T00:14:18.541 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.024e-05 2020-01-16T00:14:18.541 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.541 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.541 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:18.541 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315470+0000", 2020-01-16T00:14:18.542 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000736468 2020-01-16T00:14:18.542 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.542 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.542 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.542 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316206+0000", 2020-01-16T00:14:18.542 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000132967 2020-01-16T00:14:18.542 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.543 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.543 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:18.543 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316339+0000", 2020-01-16T00:14:18.543 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011701717 2020-01-16T00:14:18.543 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.543 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.543 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.543 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328041+0000", 2020-01-16T00:14:18.544 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000120339 2020-01-16T00:14:18.544 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.544 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.544 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:18.544 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328161+0000", 2020-01-16T00:14:18.544 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0084341260000000001 2020-01-16T00:14:18.544 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.545 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.545 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.545 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336595+0000", 2020-01-16T00:14:18.545 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5247999999999993e-05 2020-01-16T00:14:18.545 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.546 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.546 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:18.546 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336690+0000", 2020-01-16T00:14:18.546 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00083204000000000004 2020-01-16T00:14:18.546 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.546 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.547 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:18.547 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337522+0000", 2020-01-16T00:14:18.547 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.022087721000000001 2020-01-16T00:14:18.547 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.547 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:18.547 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.547 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.547 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.548 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:983 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:14:18.548 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:14:18.548 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 100.143486475, 2020-01-16T00:14:18.548 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023363387999999999, 2020-01-16T00:14:18.548 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:18.548 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:18.548 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:18.549 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:18.549 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:18.549 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 983 2020-01-16T00:14:18.549 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.549 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:18.549 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.549 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:18.549 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:14:18.550 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:18.550 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.550 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.550 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:18.550 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:14:18.550 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:18.550 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.552 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.552 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:18.552 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315553+0000", 2020-01-16T00:14:18.553 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1760000000000004e-06 2020-01-16T00:14:18.553 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.553 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.553 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:18.553 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315560+0000", 2020-01-16T00:14:18.553 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3587000000000001e-05 2020-01-16T00:14:18.553 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.554 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.554 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:18.554 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315574+0000", 2020-01-16T00:14:18.554 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7800000000000003e-06 2020-01-16T00:14:18.554 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.554 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.554 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:18.554 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315581+0000", 2020-01-16T00:14:18.555 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000823024 2020-01-16T00:14:18.555 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.555 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.555 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.555 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316404+0000", 2020-01-16T00:14:18.555 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2413e-05 2020-01-16T00:14:18.555 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.555 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.556 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:18.556 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316466+0000", 2020-01-16T00:14:18.556 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011738561999999999 2020-01-16T00:14:18.556 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.557 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.557 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.557 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328205+0000", 2020-01-16T00:14:18.557 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6783999999999997e-05 2020-01-16T00:14:18.557 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.558 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.558 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:18.558 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328262+0000", 2020-01-16T00:14:18.558 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0093022560000000001 2020-01-16T00:14:18.558 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.559 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.559 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.559 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337564+0000", 2020-01-16T00:14:18.559 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000183151 2020-01-16T00:14:18.560 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.560 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.560 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:18.560 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337747+0000", 2020-01-16T00:14:18.560 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011707449999999999 2020-01-16T00:14:18.560 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.561 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.561 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:18.561 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.338918+0000", 2020-01-16T00:14:18.561 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023363387999999999 2020-01-16T00:14:18.561 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.561 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:18.561 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.562 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.562 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.562 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:986 3.9 3:93dfe6f1:::smithi12313934-40:head [write 2941380~419987] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:14:18.562 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:14:18.562 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 99.978186699000005, 2020-01-16T00:14:18.562 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016978400000000001, 2020-01-16T00:14:18.562 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:18.562 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:18.563 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:18.563 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:18.563 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:18.563 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 986 2020-01-16T00:14:18.563 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.563 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:18.563 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.563 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:18.564 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:14:18.564 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:18.564 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.564 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.564 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:18.564 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:14:18.564 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:18.565 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.565 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.565 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:18.565 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480852+0000", 2020-01-16T00:14:18.565 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042503899999999999 2020-01-16T00:14:18.565 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.565 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.565 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:18.566 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481277+0000", 2020-01-16T00:14:18.566 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3097e-05 2020-01-16T00:14:18.566 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.566 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.566 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:18.567 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481290+0000", 2020-01-16T00:14:18.567 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8460000000000002e-06 2020-01-16T00:14:18.567 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.567 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.567 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:18.567 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481297+0000", 2020-01-16T00:14:18.567 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8443999999999999e-05 2020-01-16T00:14:18.567 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.568 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.568 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.568 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481346+0000", 2020-01-16T00:14:18.568 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022115399999999999 2020-01-16T00:14:18.568 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.568 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.568 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:18.568 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481567+0000", 2020-01-16T00:14:18.569 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012531502 2020-01-16T00:14:18.569 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.569 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.569 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.569 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494098+0000", 2020-01-16T00:14:18.569 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.4462999999999993e-05 2020-01-16T00:14:18.569 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.570 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.570 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:18.570 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494193+0000", 2020-01-16T00:14:18.570 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017600999999999999 2020-01-16T00:14:18.570 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.570 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.570 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:14:18.570 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494369+0000", 2020-01-16T00:14:18.571 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0031739619999999998 2020-01-16T00:14:18.571 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.571 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.571 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:18.571 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497543+0000", 2020-01-16T00:14:18.571 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000175107 2020-01-16T00:14:18.571 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.571 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.572 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:18.572 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497718+0000", 2020-01-16T00:14:18.572 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5207000000000003e-05 2020-01-16T00:14:18.572 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.572 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.572 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:18.572 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497763+0000", 2020-01-16T00:14:18.573 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7042999999999998e-05 2020-01-16T00:14:18.573 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.573 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.573 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:18.573 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497800+0000", 2020-01-16T00:14:18.573 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2628e-05 2020-01-16T00:14:18.573 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.573 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.574 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:18.574 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497833+0000", 2020-01-16T00:14:18.574 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016978400000000001 2020-01-16T00:14:18.574 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.574 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:18.574 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.574 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.575 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.575 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:988 3.9 3:93dfe6f1:::smithi12313934-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:14:18.575 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:14:18.575 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 99.977701234999998, 2020-01-16T00:14:18.575 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017221692, 2020-01-16T00:14:18.575 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:18.575 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:18.575 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:18.576 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:18.576 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:18.576 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 988 2020-01-16T00:14:18.576 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.576 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:18.576 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.577 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:18.577 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:14:18.577 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:18.577 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.577 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.577 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:18.578 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:14:18.578 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:18.578 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.578 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.578 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:18.578 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481339+0000", 2020-01-16T00:14:18.578 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3089999999999997e-06 2020-01-16T00:14:18.579 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.579 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.579 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:18.579 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481343+0000", 2020-01-16T00:14:18.579 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2230000000000004e-06 2020-01-16T00:14:18.579 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.580 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.580 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:18.580 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481350+0000", 2020-01-16T00:14:18.580 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.319e-06 2020-01-16T00:14:18.580 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.580 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.580 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:18.580 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481356+0000", 2020-01-16T00:14:18.581 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000321544 2020-01-16T00:14:18.581 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.581 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.581 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.581 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481678+0000", 2020-01-16T00:14:18.581 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6375999999999999e-05 2020-01-16T00:14:18.581 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.582 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.582 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:18.582 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481714+0000", 2020-01-16T00:14:18.582 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015557666 2020-01-16T00:14:18.582 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.582 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.582 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.582 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497272+0000", 2020-01-16T00:14:18.583 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012791300000000001 2020-01-16T00:14:18.583 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.583 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.583 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:18.583 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497399+0000", 2020-01-16T00:14:18.583 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00076944600000000002 2020-01-16T00:14:18.583 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.583 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.584 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.584 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498169+0000", 2020-01-16T00:14:18.584 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018306799999999999 2020-01-16T00:14:18.584 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.584 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.584 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:18.584 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498352+0000", 2020-01-16T00:14:18.585 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020935999999999999 2020-01-16T00:14:18.585 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.585 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.585 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:18.585 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498561+0000", 2020-01-16T00:14:18.585 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017221692 2020-01-16T00:14:18.585 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.585 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:18.586 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.586 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.587 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.587 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1239 3.3 3:c204aa4f:::smithi12313934-45:head [setxattr _header (56) in=7b,truncate 2000033] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:14:18.587 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:14:18.587 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 95.868627415999995, 2020-01-16T00:14:18.587 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019161661999999999, 2020-01-16T00:14:18.587 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:18.588 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:18.588 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:18.588 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:18.588 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:18.588 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1239 2020-01-16T00:14:18.588 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.588 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:18.589 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.589 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:18.589 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:14:18.589 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:18.589 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.590 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.590 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:18.590 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:14:18.590 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:18.590 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.590 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.590 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:18.591 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590411+0000", 2020-01-16T00:14:18.591 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3879999999999999e-05 2020-01-16T00:14:18.591 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.591 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.591 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:18.591 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590425+0000", 2020-01-16T00:14:18.591 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6542999999999998e-05 2020-01-16T00:14:18.591 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.592 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.592 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:18.592 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590471+0000", 2020-01-16T00:14:18.592 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8907999999999999e-05 2020-01-16T00:14:18.592 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.592 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.592 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:18.592 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590490+0000", 2020-01-16T00:14:18.593 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0045691960000000002 2020-01-16T00:14:18.593 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.593 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.593 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.593 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.595059+0000", 2020-01-16T00:14:18.593 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0035638940000000002 2020-01-16T00:14:18.593 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.594 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.594 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:18.594 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598623+0000", 2020-01-16T00:14:18.594 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0077558310000000004 2020-01-16T00:14:18.594 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.594 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.594 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.594 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606379+0000", 2020-01-16T00:14:18.595 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012171999999999999 2020-01-16T00:14:18.595 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.595 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.595 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:18.595 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606501+0000", 2020-01-16T00:14:18.595 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013144200000000001 2020-01-16T00:14:18.595 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.596 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.596 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2020-01-16T00:14:18.596 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606632+0000", 2020-01-16T00:14:18.596 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001288565 2020-01-16T00:14:18.596 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.596 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.596 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:18.597 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607921+0000", 2020-01-16T00:14:18.597 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033399299999999999 2020-01-16T00:14:18.597 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.597 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.597 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:18.597 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.608255+0000", 2020-01-16T00:14:18.598 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012280870000000001 2020-01-16T00:14:18.598 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.598 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.598 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:18.599 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609483+0000", 2020-01-16T00:14:18.600 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2286000000000001e-05 2020-01-16T00:14:18.600 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.600 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.600 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:18.600 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609525+0000", 2020-01-16T00:14:18.602 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0121999999999999e-05 2020-01-16T00:14:18.602 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.602 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.602 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:18.602 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609575+0000", 2020-01-16T00:14:18.602 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019161661999999999 2020-01-16T00:14:18.603 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.603 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:18.603 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.603 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.604 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.604 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1240 3.3 3:c204aa4f:::smithi12313934-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-01-16T00:14:18.604 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:14:18.604 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 95.868527227000001, 2020-01-16T00:14:18.604 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019268828000000002, 2020-01-16T00:14:18.605 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:18.605 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:18.605 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:18.606 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:18.606 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:18.606 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1240 2020-01-16T00:14:18.607 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.607 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:18.607 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.607 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:18.607 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:14:18.608 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:18.608 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.608 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.608 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:18.608 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:14:18.609 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:18.609 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.609 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.609 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:18.610 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590512+0000", 2020-01-16T00:14:18.610 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0525e-05 2020-01-16T00:14:18.610 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.610 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.610 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:18.611 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590522+0000", 2020-01-16T00:14:18.611 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6418000000000001e-05 2020-01-16T00:14:18.611 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.611 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.612 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:18.612 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590539+0000", 2020-01-16T00:14:18.612 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0679999999999999e-05 2020-01-16T00:14:18.612 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.613 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.613 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:18.613 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590549+0000", 2020-01-16T00:14:18.613 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0083626789999999996 2020-01-16T00:14:18.614 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.614 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.615 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.615 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598912+0000", 2020-01-16T00:14:18.615 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013344 2020-01-16T00:14:18.615 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.615 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.616 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:18.616 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599045+0000", 2020-01-16T00:14:18.616 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0084002269999999997 2020-01-16T00:14:18.616 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.617 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.617 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.617 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607446+0000", 2020-01-16T00:14:18.617 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000109873 2020-01-16T00:14:18.617 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.618 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.618 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:18.618 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607555+0000", 2020-01-16T00:14:18.618 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0020573309999999999 2020-01-16T00:14:18.619 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.619 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.619 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.620 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609613+0000", 2020-01-16T00:14:18.620 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6351e-05 2020-01-16T00:14:18.620 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.620 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.621 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:18.621 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609699+0000", 2020-01-16T00:14:18.621 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3374999999999994e-05 2020-01-16T00:14:18.621 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.622 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.622 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:18.622 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609782+0000", 2020-01-16T00:14:18.622 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019268828000000002 2020-01-16T00:14:18.623 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.623 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:18.623 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.623 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.624 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.624 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3124 3.9 3:9fc687b2:::smithi12313934-44:head [write 1903628~435462] snapc 0=[] ondisk+write+known_if_redirected e44)", 2020-01-16T00:14:18.624 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.276145+0000", 2020-01-16T00:14:18.624 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 53.182895506000001, 2020-01-16T00:14:18.624 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017361898000000001, 2020-01-16T00:14:18.625 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:18.625 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:18.625 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:18.625 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:18.626 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:18.626 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3124 2020-01-16T00:14:18.626 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.626 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:18.627 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.627 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:18.627 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276145+0000", 2020-01-16T00:14:18.627 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:18.627 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.628 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.628 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:18.628 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276145+0000", 2020-01-16T00:14:18.628 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:18.629 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.629 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.629 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:18.629 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276143+0000", 2020-01-16T00:14:18.630 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042136099999999998 2020-01-16T00:14:18.630 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.630 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.630 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:18.630 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276564+0000", 2020-01-16T00:14:18.631 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.349e-05 2020-01-16T00:14:18.631 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.631 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.631 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:18.634 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276578+0000", 2020-01-16T00:14:18.634 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2469999999999999e-06 2020-01-16T00:14:18.634 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.634 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.634 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:18.635 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276583+0000", 2020-01-16T00:14:18.635 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4400000000000001e-05 2020-01-16T00:14:18.635 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.635 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.635 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.635 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276637+0000", 2020-01-16T00:14:18.635 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0046361789999999998 2020-01-16T00:14:18.636 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.636 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.636 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:18.637 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281274+0000", 2020-01-16T00:14:18.637 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0090025520000000005 2020-01-16T00:14:18.637 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.637 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.637 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.637 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.290276+0000", 2020-01-16T00:14:18.638 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011584200000000001 2020-01-16T00:14:18.638 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.638 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.638 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:18.638 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.290392+0000", 2020-01-16T00:14:18.638 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00025563099999999998 2020-01-16T00:14:18.638 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.638 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.639 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:14:18.639 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.290648+0000", 2020-01-16T00:14:18.639 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0026741360000000001 2020-01-16T00:14:18.639 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.639 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.639 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:18.639 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293322+0000", 2020-01-16T00:14:18.640 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9764000000000002e-05 2020-01-16T00:14:18.640 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.640 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.640 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:18.640 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293421+0000", 2020-01-16T00:14:18.640 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4779000000000001e-05 2020-01-16T00:14:18.640 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.640 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.641 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:18.641 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293446+0000", 2020-01-16T00:14:18.641 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4628000000000001e-05 2020-01-16T00:14:18.641 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.641 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.641 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:18.642 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293481+0000", 2020-01-16T00:14:18.642 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.6444999999999998e-05 2020-01-16T00:14:18.642 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.642 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.642 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:18.642 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293507+0000", 2020-01-16T00:14:18.642 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017361898000000001 2020-01-16T00:14:18.642 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.643 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:18.643 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.643 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.643 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.643 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3125 3.9 3:9fc687b2:::smithi12313934-44:head [setxattr _header (56) in=7b,truncate 2386660] snapc 0=[] ondisk+write+known_if_redirected e44)", 2020-01-16T00:14:18.643 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.276594+0000", 2020-01-16T00:14:18.643 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 53.182446734000003, 2020-01-16T00:14:18.644 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017760560000000002, 2020-01-16T00:14:18.644 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:18.644 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:18.644 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:18.644 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:18.644 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:18.644 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3125 2020-01-16T00:14:18.645 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.645 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:18.645 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.645 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:18.645 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276594+0000", 2020-01-16T00:14:18.645 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:18.645 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.646 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.646 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:18.646 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276594+0000", 2020-01-16T00:14:18.646 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:18.646 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.646 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.646 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:18.647 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276593+0000", 2020-01-16T00:14:18.647 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7320000000000003e-06 2020-01-16T00:14:18.647 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.647 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.647 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:18.647 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276599+0000", 2020-01-16T00:14:18.647 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3302000000000001e-05 2020-01-16T00:14:18.647 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.648 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.648 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:18.648 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276612+0000", 2020-01-16T00:14:18.648 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1659999999999998e-06 2020-01-16T00:14:18.648 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.648 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.648 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:18.648 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276615+0000", 2020-01-16T00:14:18.649 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0047328450000000003 2020-01-16T00:14:18.649 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.649 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.649 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.649 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281348+0000", 2020-01-16T00:14:18.649 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3966000000000003e-05 2020-01-16T00:14:18.649 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.650 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.650 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:18.650 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281422+0000", 2020-01-16T00:14:18.650 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01082613 2020-01-16T00:14:18.650 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.650 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.650 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.650 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.292248+0000", 2020-01-16T00:14:18.651 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000148617 2020-01-16T00:14:18.651 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.651 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.651 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:18.651 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.292397+0000", 2020-01-16T00:14:18.651 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019936300000000001 2020-01-16T00:14:18.651 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.651 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.652 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:14:18.652 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.292596+0000", 2020-01-16T00:14:18.652 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011208870000000001 2020-01-16T00:14:18.652 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.652 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.652 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:18.652 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293717+0000", 2020-01-16T00:14:18.653 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00039491599999999998 2020-01-16T00:14:18.653 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.653 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.653 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:18.653 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294112+0000", 2020-01-16T00:14:18.653 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000129326 2020-01-16T00:14:18.653 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.653 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.654 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:18.654 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294241+0000", 2020-01-16T00:14:18.654 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5586000000000001e-05 2020-01-16T00:14:18.654 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.654 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.654 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:18.654 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294277+0000", 2020-01-16T00:14:18.654 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7893999999999997e-05 2020-01-16T00:14:18.655 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.655 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.655 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:18.655 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294355+0000", 2020-01-16T00:14:18.655 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017760560000000002 2020-01-16T00:14:18.655 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.655 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:18.656 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.656 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.656 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.656 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3126 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e44)", 2020-01-16T00:14:18.656 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.276634+0000", 2020-01-16T00:14:18.656 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 53.182407019000003, 2020-01-16T00:14:18.656 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017942583000000002, 2020-01-16T00:14:18.656 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:18.657 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:18.657 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:18.657 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:18.657 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:18.657 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3126 2020-01-16T00:14:18.657 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.657 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:18.658 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.658 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:18.658 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276634+0000", 2020-01-16T00:14:18.658 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:18.658 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.658 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.658 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:18.659 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276634+0000", 2020-01-16T00:14:18.659 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:18.659 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.659 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.659 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:18.659 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276633+0000", 2020-01-16T00:14:18.659 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0769999999999999e-06 2020-01-16T00:14:18.659 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.660 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.660 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:18.660 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276636+0000", 2020-01-16T00:14:18.660 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5390000000000001e-06 2020-01-16T00:14:18.660 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.660 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.660 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:18.660 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276641+0000", 2020-01-16T00:14:18.661 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9969999999999999e-06 2020-01-16T00:14:18.661 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.661 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.661 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:18.661 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276644+0000", 2020-01-16T00:14:18.661 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0048246970000000002 2020-01-16T00:14:18.661 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.661 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.662 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.662 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281469+0000", 2020-01-16T00:14:18.662 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8723999999999999e-05 2020-01-16T00:14:18.662 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.662 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.662 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:18.662 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281567+0000", 2020-01-16T00:14:18.663 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011633447 2020-01-16T00:14:18.663 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.663 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.663 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.663 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293201+0000", 2020-01-16T00:14:18.663 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6796000000000002e-05 2020-01-16T00:14:18.663 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.663 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.664 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:18.664 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293268+0000", 2020-01-16T00:14:18.664 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001125247 2020-01-16T00:14:18.664 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.664 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.664 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.664 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294393+0000", 2020-01-16T00:14:18.664 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.6082000000000006e-05 2020-01-16T00:14:18.665 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.665 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.665 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:18.665 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294489+0000", 2020-01-16T00:14:18.665 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7515000000000005e-05 2020-01-16T00:14:18.665 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.665 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.666 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:18.666 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294576+0000", 2020-01-16T00:14:18.666 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017942583000000002 2020-01-16T00:14:18.666 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.666 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:18.666 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.666 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.666 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.667 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3204 3.9 3:93dfe6f1:::smithi12313934-40:head [setxattr _header (56) in=7b,truncate 2554730] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:14:18.667 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.568544+0000", 2020-01-16T00:14:18.667 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 48.890496798999997, 2020-01-16T00:14:18.667 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.026260867, 2020-01-16T00:14:18.667 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:18.667 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:18.667 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:18.667 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:18.668 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:18.668 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3204 2020-01-16T00:14:18.668 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.668 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:18.668 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.668 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:18.668 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568544+0000", 2020-01-16T00:14:18.668 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:18.669 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.669 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.669 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:18.669 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568544+0000", 2020-01-16T00:14:18.669 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:18.669 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.669 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.670 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:18.670 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568543+0000", 2020-01-16T00:14:18.670 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0810000000000001e-06 2020-01-16T00:14:18.670 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.670 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.670 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:18.671 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568548+0000", 2020-01-16T00:14:18.671 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6950000000000002e-06 2020-01-16T00:14:18.671 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.671 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.671 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:18.671 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568552+0000", 2020-01-16T00:14:18.671 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.875e-06 2020-01-16T00:14:18.672 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.672 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.672 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:18.672 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568555+0000", 2020-01-16T00:14:18.672 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0033130849999999999 2020-01-16T00:14:18.672 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.672 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.673 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.673 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.571868+0000", 2020-01-16T00:14:18.673 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0026014609999999998 2020-01-16T00:14:18.673 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.673 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.673 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:18.673 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.574469+0000", 2020-01-16T00:14:18.673 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010261381999999999 2020-01-16T00:14:18.674 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.674 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.674 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.674 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.584731+0000", 2020-01-16T00:14:18.674 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019704 2020-01-16T00:14:18.674 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.674 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.674 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:18.675 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.584928+0000", 2020-01-16T00:14:18.675 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000280869 2020-01-16T00:14:18.675 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.675 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.675 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:14:18.675 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.585209+0000", 2020-01-16T00:14:18.675 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002273859 2020-01-16T00:14:18.676 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.676 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.676 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:18.676 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.587483+0000", 2020-01-16T00:14:18.676 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3356000000000001e-05 2020-01-16T00:14:18.676 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.676 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.676 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:18.677 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.587506+0000", 2020-01-16T00:14:18.677 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0071864939999999999 2020-01-16T00:14:18.677 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.677 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.677 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:18.677 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594692+0000", 2020-01-16T00:14:18.677 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5475000000000001e-05 2020-01-16T00:14:18.677 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.678 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.678 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:18.678 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594738+0000", 2020-01-16T00:14:18.678 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7070000000000007e-05 2020-01-16T00:14:18.678 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.678 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.678 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:18.679 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594805+0000", 2020-01-16T00:14:18.679 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.026260867 2020-01-16T00:14:18.679 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.679 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:18.679 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.679 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.679 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.679 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3205 3.9 3:93dfe6f1:::smithi12313934-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e45)", 2020-01-16T00:14:18.680 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.568599+0000", 2020-01-16T00:14:18.680 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 48.890441864000003, 2020-01-16T00:14:18.680 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.026406882, 2020-01-16T00:14:18.680 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:18.680 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:18.680 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:18.680 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:18.680 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:18.681 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3205 2020-01-16T00:14:18.681 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.681 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:18.681 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.681 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:18.681 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568599+0000", 2020-01-16T00:14:18.681 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:18.682 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.682 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.682 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:18.682 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568599+0000", 2020-01-16T00:14:18.682 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:18.682 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.682 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.682 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:18.683 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568598+0000", 2020-01-16T00:14:18.683 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3939999999999998e-06 2020-01-16T00:14:18.683 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.683 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.683 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:18.683 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568603+0000", 2020-01-16T00:14:18.683 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9391000000000001e-05 2020-01-16T00:14:18.683 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.684 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.684 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:18.684 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568622+0000", 2020-01-16T00:14:18.684 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9599999999999997e-06 2020-01-16T00:14:18.684 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.684 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.685 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:18.685 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568628+0000", 2020-01-16T00:14:18.685 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0058679700000000001 2020-01-16T00:14:18.685 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.685 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.685 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.685 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.574496+0000", 2020-01-16T00:14:18.686 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5528999999999998e-05 2020-01-16T00:14:18.686 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.686 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.686 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:18.686 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.574532+0000", 2020-01-16T00:14:18.686 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011986749 2020-01-16T00:14:18.686 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.686 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.687 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.687 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.586518+0000", 2020-01-16T00:14:18.687 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016522600000000001 2020-01-16T00:14:18.687 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.687 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.687 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:18.687 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.586684+0000", 2020-01-16T00:14:18.687 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0081693890000000009 2020-01-16T00:14:18.688 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.688 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.688 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.688 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594853+0000", 2020-01-16T00:14:18.688 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7082000000000004e-05 2020-01-16T00:14:18.688 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.688 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.688 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:18.689 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594940+0000", 2020-01-16T00:14:18.689 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5789000000000003e-05 2020-01-16T00:14:18.689 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.689 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.689 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:18.689 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.595006+0000", 2020-01-16T00:14:18.689 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.026406882 2020-01-16T00:14:18.690 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.690 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:18.690 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.690 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.690 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.690 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3219 3.3 3:c478bfe8:::smithi12313934-10:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:18.690 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.580957+0000", 2020-01-16T00:14:18.691 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 48.878084348000002, 2020-01-16T00:14:18.691 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016807971000000001, 2020-01-16T00:14:18.691 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:18.691 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:18.691 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:18.691 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:18.691 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:18.691 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3219 2020-01-16T00:14:18.692 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.692 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:18.692 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.692 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:18.692 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580957+0000", 2020-01-16T00:14:18.692 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:18.692 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.692 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.693 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:18.693 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580957+0000", 2020-01-16T00:14:18.693 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:18.693 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.693 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.693 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:18.693 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580955+0000", 2020-01-16T00:14:18.694 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5120000000000002e-06 2020-01-16T00:14:18.694 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.694 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.694 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:18.694 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580963+0000", 2020-01-16T00:14:18.694 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6799999999999993e-06 2020-01-16T00:14:18.694 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.694 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.695 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:18.695 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580971+0000", 2020-01-16T00:14:18.695 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3109999999999996e-06 2020-01-16T00:14:18.695 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.695 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.695 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:18.695 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580976+0000", 2020-01-16T00:14:18.695 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014306146 2020-01-16T00:14:18.696 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.696 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.696 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.696 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.595282+0000", 2020-01-16T00:14:18.696 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.1883000000000001e-05 2020-01-16T00:14:18.696 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.696 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.697 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:18.697 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.595374+0000", 2020-01-16T00:14:18.697 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023904989999999999 2020-01-16T00:14:18.697 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.697 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.697 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:18.697 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.597765+0000", 2020-01-16T00:14:18.697 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016807971000000001 2020-01-16T00:14:18.698 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.698 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:18.698 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.698 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.698 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.698 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3220 3.3 3:c478bfe8:::smithi12313934-10:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:18.698 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.580982+0000", 2020-01-16T00:14:18.699 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 48.878059299999997, 2020-01-16T00:14:18.699 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017920330000000002, 2020-01-16T00:14:18.699 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:18.699 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:18.699 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:18.699 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:18.699 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:18.699 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3220 2020-01-16T00:14:18.700 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.700 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:18.700 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.700 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:18.700 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580982+0000", 2020-01-16T00:14:18.700 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:18.700 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.700 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.701 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:18.701 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580982+0000", 2020-01-16T00:14:18.701 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:18.701 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.701 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.701 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:18.701 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580981+0000", 2020-01-16T00:14:18.702 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0380000000000001e-06 2020-01-16T00:14:18.702 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.702 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.702 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:18.702 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580985+0000", 2020-01-16T00:14:18.702 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8929000000000001e-05 2020-01-16T00:14:18.702 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.702 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.703 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:18.703 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581004+0000", 2020-01-16T00:14:18.703 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7339999999999999e-06 2020-01-16T00:14:18.703 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.703 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.703 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:18.703 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581009+0000", 2020-01-16T00:14:18.703 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01679812 2020-01-16T00:14:18.704 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.704 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.704 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.704 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.597807+0000", 2020-01-16T00:14:18.704 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013333400000000001 2020-01-16T00:14:18.704 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.704 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.704 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:18.705 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.597940+0000", 2020-01-16T00:14:18.705 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000961899 2020-01-16T00:14:18.705 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.705 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.705 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:18.705 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.598902+0000", 2020-01-16T00:14:18.705 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017920330000000002 2020-01-16T00:14:18.706 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.706 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:18.706 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.706 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.706 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.706 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3221 3.3 3:c478bfe8:::smithi12313934-10:head [read 0~2403467,checksum] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:18.706 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.581014+0000", 2020-01-16T00:14:18.706 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 48.878026951000002, 2020-01-16T00:14:18.707 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018784551999999999, 2020-01-16T00:14:18.707 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:18.707 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:18.707 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:18.707 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:18.707 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:18.707 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3221 2020-01-16T00:14:18.708 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.708 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:18.708 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.708 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:18.708 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581014+0000", 2020-01-16T00:14:18.708 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:18.708 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.708 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.709 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:18.709 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581014+0000", 2020-01-16T00:14:18.709 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:18.709 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.709 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.709 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:18.709 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581013+0000", 2020-01-16T00:14:18.710 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9390000000000002e-06 2020-01-16T00:14:18.710 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.710 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.710 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:18.710 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581019+0000", 2020-01-16T00:14:18.710 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1131000000000001e-05 2020-01-16T00:14:18.710 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.710 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.711 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:18.711 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581030+0000", 2020-01-16T00:14:18.711 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.55e-06 2020-01-16T00:14:18.711 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.711 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.711 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:18.711 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581037+0000", 2020-01-16T00:14:18.711 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017890321000000001 2020-01-16T00:14:18.712 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.712 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.712 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.712 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.598927+0000", 2020-01-16T00:14:18.712 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000105999 2020-01-16T00:14:18.712 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.712 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.713 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:18.713 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.599033+0000", 2020-01-16T00:14:18.713 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00076542400000000001 2020-01-16T00:14:18.713 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.713 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.713 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:18.713 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.599798+0000", 2020-01-16T00:14:18.713 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018784551999999999 2020-01-16T00:14:18.714 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.714 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:18.714 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.714 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.714 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.714 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4152 3.9 3:93dfe6f1:::smithi12313934-40:head [setxattr _header (56) in=7b,truncate 2806868] snapc 0=[] ondisk+write+known_if_redirected e53)", 2020-01-16T00:14:18.714 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:53.810986+0000", 2020-01-16T00:14:18.715 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 24.648054642000002, 2020-01-16T00:14:18.715 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016830279, 2020-01-16T00:14:18.715 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:18.715 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:18.715 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:18.715 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:18.716 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:18.716 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4152 2020-01-16T00:14:18.716 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.716 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:18.716 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.716 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:18.716 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.810986+0000", 2020-01-16T00:14:18.716 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:18.717 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.717 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.717 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:18.717 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.810986+0000", 2020-01-16T00:14:18.717 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:18.717 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.717 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.718 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:18.718 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.810985+0000", 2020-01-16T00:14:18.718 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.7529999999999996e-06 2020-01-16T00:14:18.718 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.718 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.718 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:18.718 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.810995+0000", 2020-01-16T00:14:18.718 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7907999999999999e-05 2020-01-16T00:14:18.719 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.719 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.719 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:18.719 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811013+0000", 2020-01-16T00:14:18.719 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5829999999999997e-06 2020-01-16T00:14:18.719 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.719 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.719 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:18.720 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811018+0000", 2020-01-16T00:14:18.720 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0029165229999999999 2020-01-16T00:14:18.720 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.720 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.720 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.720 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.813935+0000", 2020-01-16T00:14:18.720 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2583000000000001e-05 2020-01-16T00:14:18.721 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.721 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.721 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:18.721 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.813988+0000", 2020-01-16T00:14:18.721 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010581012000000001 2020-01-16T00:14:18.721 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.721 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.721 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.722 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.824569+0000", 2020-01-16T00:14:18.722 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013191900000000001 2020-01-16T00:14:18.722 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.722 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.722 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:18.722 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.824700+0000", 2020-01-16T00:14:18.722 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022298000000000001 2020-01-16T00:14:18.723 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.723 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.723 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:14:18.723 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.824923+0000", 2020-01-16T00:14:18.723 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010694260000000001 2020-01-16T00:14:18.723 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.723 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.723 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:18.724 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.825993+0000", 2020-01-16T00:14:18.724 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000116939 2020-01-16T00:14:18.724 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.724 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.724 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:18.724 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.826110+0000", 2020-01-16T00:14:18.724 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0015606400000000001 2020-01-16T00:14:18.724 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.725 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.725 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:18.725 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.827670+0000", 2020-01-16T00:14:18.725 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7318999999999997e-05 2020-01-16T00:14:18.725 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.725 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.725 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:18.726 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.827738+0000", 2020-01-16T00:14:18.726 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.873e-05 2020-01-16T00:14:18.726 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.726 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.726 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:18.726 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.827817+0000", 2020-01-16T00:14:18.726 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016830279 2020-01-16T00:14:18.726 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.727 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:18.727 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.727 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.727 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.728 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4153 3.9 3:93dfe6f1:::smithi12313934-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e53)", 2020-01-16T00:14:18.728 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:53.811034+0000", 2020-01-16T00:14:18.728 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 24.648006553999998, 2020-01-16T00:14:18.728 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017095406, 2020-01-16T00:14:18.728 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:18.728 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:18.728 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:18.729 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:18.729 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:18.729 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4153 2020-01-16T00:14:18.729 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.729 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:18.729 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.729 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:18.730 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811034+0000", 2020-01-16T00:14:18.730 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:18.730 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.730 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.730 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:18.730 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811034+0000", 2020-01-16T00:14:18.730 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:18.730 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.731 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.731 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:18.731 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811034+0000", 2020-01-16T00:14:18.731 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9060000000000001e-06 2020-01-16T00:14:18.731 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.731 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.731 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:18.731 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811038+0000", 2020-01-16T00:14:18.732 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7710000000000001e-06 2020-01-16T00:14:18.732 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.732 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.732 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:18.732 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811045+0000", 2020-01-16T00:14:18.732 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5189999999999998e-06 2020-01-16T00:14:18.732 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.733 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.733 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:18.733 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811052+0000", 2020-01-16T00:14:18.733 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002953991 2020-01-16T00:14:18.733 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.733 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.733 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.733 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.814006+0000", 2020-01-16T00:14:18.734 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5726999999999999e-05 2020-01-16T00:14:18.734 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.734 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.734 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:18.734 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.814041+0000", 2020-01-16T00:14:18.734 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011650031999999999 2020-01-16T00:14:18.734 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.734 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.735 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.735 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.825691+0000", 2020-01-16T00:14:18.735 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0625999999999998e-05 2020-01-16T00:14:18.735 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.735 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.735 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:18.735 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.825762+0000", 2020-01-16T00:14:18.736 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0020982370000000002 2020-01-16T00:14:18.736 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.736 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.736 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:18.736 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.827860+0000", 2020-01-16T00:14:18.736 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012736299999999999 2020-01-16T00:14:18.736 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.736 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.737 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:18.737 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.827988+0000", 2020-01-16T00:14:18.737 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000142015 2020-01-16T00:14:18.737 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:18.737 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:18.737 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:18.738 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.828130+0000", 2020-01-16T00:14:18.738 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017095406 2020-01-16T00:14:18.738 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.738 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:18.738 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.738 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:18.738 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:18.738 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:18.741 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:14:18.741 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:14:18.741 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:14:18.741 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:14:18.741 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:14:18.742 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-01-16T00:14:18.742 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-01-16T00:14:18.742 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-01-16T00:14:18.742 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-01-16T00:14:18.742 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-01-16T00:14:18.742 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENXIO: problem getting command descriptions from osd.5 2020-01-16T00:14:18.743 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: problem getting command descriptions from osd.5 2020-01-16T00:14:18.743 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-01-16T00:14:18.743 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-01-16T00:14:18.746 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-01-16T00:14:18.746 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-01-16T00:14:18.746 INFO:tasks.rados.rados.0.smithi123.stdout:1813: expect (ObjNum 595 snap 0 seq_num 595) 2020-01-16T00:14:18.746 INFO:tasks.rados.rados.0.smithi123.stdout:1814: expect (ObjNum 630 snap 0 seq_num 630) 2020-01-16T00:14:18.747 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:18.738+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:18.747 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-01-16T00:14:18.795 INFO:tasks.rados.rados.0.smithi123.stdout:1811: expect (ObjNum 660 snap 0 seq_num 660) 2020-01-16T00:14:18.849 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:18.848+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:18.878 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:18.885 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-01-16T00:14:18.936 INFO:tasks.rados.rados.0.smithi123.stdout:1812: expect (ObjNum 633 snap 0 seq_num 633) 2020-01-16T00:14:18.951 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:18.950+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:19.035 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:19.045 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-01-16T00:14:19.053 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:19.050+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:19.057 INFO:teuthology.orchestra.run.smithi123.stderr:noscrub is set 2020-01-16T00:14:19.084 INFO:tasks.rados.rados.0.smithi123.stdout:1810: expect (ObjNum 655 snap 0 seq_num 655) 2020-01-16T00:14:19.152 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:19.153 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:19.150+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:19.162 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-01-16T00:14:19.169 INFO:tasks.rados.rados.0.smithi123.stdout:1809: expect (ObjNum 639 snap 0 seq_num 639) 2020-01-16T00:14:19.253 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:19.252+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:19.269 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:19.279 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-01-16T00:14:19.369 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:19.364+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:19.370 INFO:tasks.rados.rados.0.smithi123.stdout:1815: finishing write tid 1 to smithi12313934-42 2020-01-16T00:14:19.370 INFO:tasks.rados.rados.0.smithi123.stdout:1815: finishing write tid 2 to smithi12313934-42 2020-01-16T00:14:19.370 INFO:tasks.rados.rados.0.smithi123.stdout:1816: finishing write tid 1 to smithi12313934-2 2020-01-16T00:14:19.370 INFO:tasks.rados.rados.0.smithi123.stdout:1815: finishing write tid 4 to smithi12313934-42 2020-01-16T00:14:19.371 INFO:tasks.rados.rados.0.smithi123.stdout:1815: finishing write tid 5 to smithi12313934-42 2020-01-16T00:14:19.371 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 42 v 221 (ObjNum 666 snap 0 seq_num 666) dirty exists 2020-01-16T00:14:19.371 INFO:tasks.rados.rados.0.smithi123.stdout:1815: left oid 42 (ObjNum 666 snap 0 seq_num 666) 2020-01-16T00:14:19.371 INFO:tasks.rados.rados.0.smithi123.stdout:1816: finishing write tid 2 to smithi12313934-2 2020-01-16T00:14:19.371 INFO:tasks.rados.rados.0.smithi123.stdout:1817: finishing write tid 1 to smithi12313934-38 2020-01-16T00:14:19.372 INFO:tasks.rados.rados.0.smithi123.stdout:1816: finishing write tid 3 to smithi12313934-2 2020-01-16T00:14:19.372 INFO:tasks.rados.rados.0.smithi123.stdout:1807: done (15 left) 2020-01-16T00:14:19.372 INFO:tasks.rados.rados.0.smithi123.stdout:1808: done (14 left) 2020-01-16T00:14:19.372 INFO:tasks.rados.rados.0.smithi123.stdout:1809: done (13 left) 2020-01-16T00:14:19.373 INFO:tasks.rados.rados.0.smithi123.stdout:1810: done (12 left) 2020-01-16T00:14:19.373 INFO:tasks.rados.rados.0.smithi123.stdout:1811: done (11 left) 2020-01-16T00:14:19.373 INFO:tasks.rados.rados.0.smithi123.stdout:1812: done (10 left) 2020-01-16T00:14:19.373 INFO:tasks.rados.rados.0.smithi123.stdout:1813: done (9 left) 2020-01-16T00:14:19.374 INFO:tasks.rados.rados.0.smithi123.stdout:1814: done (8 left) 2020-01-16T00:14:19.374 INFO:tasks.rados.rados.0.smithi123.stdout:1815: done (7 left) 2020-01-16T00:14:19.374 INFO:tasks.rados.rados.0.smithi123.stdout:1823: read oid 10 snap -1 2020-01-16T00:14:19.374 INFO:tasks.rados.rados.0.smithi123.stdout:1823: expect (ObjNum 663 snap 0 seq_num 663) 2020-01-16T00:14:19.374 INFO:tasks.rados.rados.0.smithi123.stdout:1824: read oid 50 snap -1 2020-01-16T00:14:19.375 INFO:tasks.rados.rados.0.smithi123.stdout:1824: expect (ObjNum 595 snap 0 seq_num 595) 2020-01-16T00:14:19.375 INFO:tasks.rados.rados.0.smithi123.stdout:1825: read oid 46 snap -1 2020-01-16T00:14:19.375 INFO:tasks.rados.rados.0.smithi123.stdout:1825: expect (ObjNum 665 snap 0 seq_num 665) 2020-01-16T00:14:19.375 INFO:tasks.rados.rados.0.smithi123.stdout:1826: read oid 24 snap -1 2020-01-16T00:14:19.376 INFO:tasks.rados.rados.0.smithi123.stdout:1826: expect (ObjNum 630 snap 0 seq_num 630) 2020-01-16T00:14:19.376 INFO:tasks.rados.rados.0.smithi123.stdout:1827: write oid 8 current snap is 0 2020-01-16T00:14:19.376 INFO:tasks.rados.rados.0.smithi123.stdout:1827: seq_num 670 ranges {460690=655894,1540112=771231,2716111=539778} 2020-01-16T00:14:19.377 INFO:tasks.rados.rados.0.smithi123.stdout:1827: writing smithi12313934-8 from 460690 to 1116584 tid 1 2020-01-16T00:14:19.385 INFO:tasks.rados.rados.0.smithi123.stdout:1827: writing smithi12313934-8 from 1540112 to 2311343 tid 2 2020-01-16T00:14:19.390 INFO:tasks.rados.rados.0.smithi123.stdout:1827: writing smithi12313934-8 from 2716111 to 3255889 tid 3 2020-01-16T00:14:19.390 INFO:tasks.rados.rados.0.smithi123.stdout:1828: read oid 20 snap -1 2020-01-16T00:14:19.391 INFO:tasks.rados.rados.0.smithi123.stdout:1828: expect deleted 2020-01-16T00:14:19.391 INFO:tasks.rados.rados.0.smithi123.stdout:1829: write oid 49 current snap is 0 2020-01-16T00:14:19.391 INFO:tasks.rados.rados.0.smithi123.stdout:1829: seq_num 671 ranges {477497=731143,1998594=514889,3143534=133963} 2020-01-16T00:14:19.397 INFO:tasks.rados.rados.0.smithi123.stdout:1829: writing smithi12313934-49 from 477497 to 1208640 tid 1 2020-01-16T00:14:19.404 INFO:tasks.rados.rados.0.smithi123.stdout:1829: writing smithi12313934-49 from 1998594 to 2513483 tid 2 2020-01-16T00:14:19.408 INFO:tasks.rados.rados.0.smithi123.stdout:1829: writing smithi12313934-49 from 3143534 to 3277497 tid 3 2020-01-16T00:14:19.408 INFO:tasks.rados.rados.0.smithi123.stdout:1830: write oid 9 current snap is 0 2020-01-16T00:14:19.409 INFO:tasks.rados.rados.0.smithi123.stdout:1830: seq_num 672 ranges {494304=406392,1540723=774900,2770957=523347} 2020-01-16T00:14:19.413 INFO:tasks.rados.rados.0.smithi123.stdout:1830: writing smithi12313934-9 from 494304 to 900696 tid 1 2020-01-16T00:14:19.421 INFO:tasks.rados.rados.0.smithi123.stdout:1830: writing smithi12313934-9 from 1540723 to 2315623 tid 2 2020-01-16T00:14:19.428 INFO:tasks.rados.rados.0.smithi123.stdout:1830: writing smithi12313934-9 from 2770957 to 3294304 tid 3 2020-01-16T00:14:19.428 INFO:tasks.rados.rados.0.smithi123.stdout:1831: delete oid 41 current snap is 0 2020-01-16T00:14:19.428 INFO:tasks.rados.rados.0.smithi123.stdout:1816: finishing write tid 5 to smithi12313934-2 2020-01-16T00:14:19.428 INFO:tasks.rados.rados.0.smithi123.stdout:1816: finishing write tid 6 to smithi12313934-2 2020-01-16T00:14:19.429 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 2 v 257 (ObjNum 667 snap 0 seq_num 667) dirty exists 2020-01-16T00:14:19.429 INFO:tasks.rados.rados.0.smithi123.stdout:1816: left oid 2 (ObjNum 667 snap 0 seq_num 667) 2020-01-16T00:14:19.429 INFO:tasks.rados.rados.0.smithi123.stdout:1817: finishing write tid 2 to smithi12313934-38 2020-01-16T00:14:19.429 INFO:tasks.rados.rados.0.smithi123.stdout:1817: finishing write tid 3 to smithi12313934-38 2020-01-16T00:14:19.429 INFO:tasks.rados.rados.0.smithi123.stdout:1817: finishing write tid 5 to smithi12313934-38 2020-01-16T00:14:19.430 INFO:tasks.rados.rados.0.smithi123.stdout:1817: finishing write tid 6 to smithi12313934-38 2020-01-16T00:14:19.430 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 38 v 225 (ObjNum 668 snap 0 seq_num 668) dirty exists 2020-01-16T00:14:19.430 INFO:tasks.rados.rados.0.smithi123.stdout:1817: left oid 38 (ObjNum 668 snap 0 seq_num 668) 2020-01-16T00:14:19.430 INFO:tasks.rados.rados.0.smithi123.stdout:1818: finishing write tid 1 to smithi12313934-40 2020-01-16T00:14:19.430 INFO:tasks.rados.rados.0.smithi123.stdout:1818: finishing write tid 2 to smithi12313934-40 2020-01-16T00:14:19.431 INFO:tasks.rados.rados.0.smithi123.stdout:1821: expect (ObjNum 628 snap 0 seq_num 628) 2020-01-16T00:14:19.440 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:19.447 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-01-16T00:14:19.490 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:19.488+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:19.556 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:19.552+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:19.598 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:19.601 INFO:tasks.rados.rados.0.smithi123.stdout:1818: finishing write tid 3 to smithi12313934-40 2020-01-16T00:14:19.601 INFO:tasks.rados.rados.0.smithi123.stdout:1818: finishing write tid 5 to smithi12313934-40 2020-01-16T00:14:19.601 INFO:tasks.rados.rados.0.smithi123.stdout:1818: finishing write tid 6 to smithi12313934-40 2020-01-16T00:14:19.602 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 40 v 292 (ObjNum 669 snap 0 seq_num 669) dirty exists 2020-01-16T00:14:19.602 INFO:tasks.rados.rados.0.smithi123.stdout:1818: left oid 40 (ObjNum 669 snap 0 seq_num 669) 2020-01-16T00:14:19.602 INFO:tasks.rados.rados.0.smithi123.stdout:1822: expect (ObjNum 614 snap 0 seq_num 614) 2020-01-16T00:14:19.608 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-01-16T00:14:19.657 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:19.654+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:19.674 INFO:tasks.rados.rados.0.smithi123.stdout:1820: expect (ObjNum 653 snap 0 seq_num 653) 2020-01-16T00:14:19.711 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:19.720 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-01-16T00:14:19.763 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:19.762+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:19.856 INFO:tasks.rados.rados.0.smithi123.stdout:1826: expect (ObjNum 630 snap 0 seq_num 630) 2020-01-16T00:14:19.859 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:19.856+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:19.862 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:19.871 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-01-16T00:14:19.960 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:19.958+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:19.968 INFO:tasks.rados.rados.0.smithi123.stdout:1824: expect (ObjNum 595 snap 0 seq_num 595) 2020-01-16T00:14:19.970 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:19.979 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-01-16T00:14:20.062 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:20.056+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:20.092 INFO:tasks.rados.rados.0.smithi123.stdout:1825: expect (ObjNum 665 snap 0 seq_num 665) 2020-01-16T00:14:20.123 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:20.131 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-01-16T00:14:20.176 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:20.174+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:20.223 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:20.231 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-01-16T00:14:20.275 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:20.274+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:20.326 INFO:tasks.rados.rados.0.smithi123.stdout:1823: expect (ObjNum 663 snap 0 seq_num 663) 2020-01-16T00:14:20.363 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:20.362+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:20.378 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:20.388 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-01-16T00:14:20.421 INFO:tasks.rados.rados.0.smithi123.stdout:1827: finishing write tid 1 to smithi12313934-8 2020-01-16T00:14:20.421 INFO:tasks.rados.rados.0.smithi123.stdout:1827: finishing write tid 2 to smithi12313934-8 2020-01-16T00:14:20.421 INFO:tasks.rados.rados.0.smithi123.stdout:1827: finishing write tid 3 to smithi12313934-8 2020-01-16T00:14:20.421 INFO:tasks.rados.rados.0.smithi123.stdout:1827: finishing write tid 5 to smithi12313934-8 2020-01-16T00:14:20.422 INFO:tasks.rados.rados.0.smithi123.stdout:1827: finishing write tid 6 to smithi12313934-8 2020-01-16T00:14:20.422 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 8 v 127 (ObjNum 670 snap 0 seq_num 670) dirty exists 2020-01-16T00:14:20.422 INFO:tasks.rados.rados.0.smithi123.stdout:1827: left oid 8 (ObjNum 670 snap 0 seq_num 670) 2020-01-16T00:14:20.422 INFO:tasks.rados.rados.0.smithi123.stdout:1829: finishing write tid 1 to smithi12313934-49 2020-01-16T00:14:20.422 INFO:tasks.rados.rados.0.smithi123.stdout:1829: finishing write tid 2 to smithi12313934-49 2020-01-16T00:14:20.422 INFO:tasks.rados.rados.0.smithi123.stdout:1829: finishing write tid 3 to smithi12313934-49 2020-01-16T00:14:20.423 INFO:tasks.rados.rados.0.smithi123.stdout:1829: finishing write tid 5 to smithi12313934-49 2020-01-16T00:14:20.423 INFO:tasks.rados.rados.0.smithi123.stdout:1829: finishing write tid 6 to smithi12313934-49 2020-01-16T00:14:20.423 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 49 v 194 (ObjNum 671 snap 0 seq_num 671) dirty exists 2020-01-16T00:14:20.423 INFO:tasks.rados.rados.0.smithi123.stdout:1829: left oid 49 (ObjNum 671 snap 0 seq_num 671) 2020-01-16T00:14:20.423 INFO:tasks.rados.rados.0.smithi123.stdout:1830: finishing write tid 1 to smithi12313934-9 2020-01-16T00:14:20.423 INFO:tasks.rados.rados.0.smithi123.stdout:1830: finishing write tid 2 to smithi12313934-9 2020-01-16T00:14:20.423 INFO:tasks.rados.rados.0.smithi123.stdout:1830: finishing write tid 3 to smithi12313934-9 2020-01-16T00:14:20.423 INFO:tasks.rados.rados.0.smithi123.stdout:1830: finishing write tid 5 to smithi12313934-9 2020-01-16T00:14:20.424 INFO:tasks.rados.rados.0.smithi123.stdout:1830: finishing write tid 6 to smithi12313934-9 2020-01-16T00:14:20.424 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 9 v 182 (ObjNum 672 snap 0 seq_num 672) dirty exists 2020-01-16T00:14:20.424 INFO:tasks.rados.rados.0.smithi123.stdout:1830: left oid 9 (ObjNum 672 snap 0 seq_num 672) 2020-01-16T00:14:20.424 INFO:tasks.rados.rados.0.smithi123.stdout:1816: done (15 left) 2020-01-16T00:14:20.424 INFO:tasks.rados.rados.0.smithi123.stdout:1817: done (14 left) 2020-01-16T00:14:20.424 INFO:tasks.rados.rados.0.smithi123.stdout:1818: done (13 left) 2020-01-16T00:14:20.424 INFO:tasks.rados.rados.0.smithi123.stdout:1819: done (12 left) 2020-01-16T00:14:20.424 INFO:tasks.rados.rados.0.smithi123.stdout:1820: done (11 left) 2020-01-16T00:14:20.425 INFO:tasks.rados.rados.0.smithi123.stdout:1821: done (10 left) 2020-01-16T00:14:20.425 INFO:tasks.rados.rados.0.smithi123.stdout:1822: done (9 left) 2020-01-16T00:14:20.425 INFO:tasks.rados.rados.0.smithi123.stdout:1823: done (8 left) 2020-01-16T00:14:20.425 INFO:tasks.rados.rados.0.smithi123.stdout:1824: done (7 left) 2020-01-16T00:14:20.425 INFO:tasks.rados.rados.0.smithi123.stdout:1825: done (6 left) 2020-01-16T00:14:20.425 INFO:tasks.rados.rados.0.smithi123.stdout:1826: done (5 left) 2020-01-16T00:14:20.426 INFO:tasks.rados.rados.0.smithi123.stdout:1827: done (4 left) 2020-01-16T00:14:20.426 INFO:tasks.rados.rados.0.smithi123.stdout:1828: done (3 left) 2020-01-16T00:14:20.426 INFO:tasks.rados.rados.0.smithi123.stdout:1829: done (2 left) 2020-01-16T00:14:20.426 INFO:tasks.rados.rados.0.smithi123.stdout:1830: done (1 left) 2020-01-16T00:14:20.427 INFO:tasks.rados.rados.0.smithi123.stdout:1831: done (0 left) 2020-01-16T00:14:20.427 INFO:tasks.rados.rados.0.smithi123.stdout:1832: read oid 40 snap -1 2020-01-16T00:14:20.429 INFO:tasks.rados.rados.0.smithi123.stdout:1832: expect (ObjNum 669 snap 0 seq_num 669) 2020-01-16T00:14:20.429 INFO:tasks.rados.rados.0.smithi123.stdout:1833: write (excl) oid 32 current snap is 0 2020-01-16T00:14:20.430 INFO:tasks.rados.rados.0.smithi123.stdout:1833: seq_num 674 ranges {527918=556890,1657687=778569,3225803=102115} 2020-01-16T00:14:20.433 INFO:tasks.rados.rados.0.smithi123.stdout:1833: writing smithi12313934-32 from 527918 to 1084808 tid 1 2020-01-16T00:14:20.441 INFO:tasks.rados.rados.0.smithi123.stdout:1833: writing smithi12313934-32 from 1657687 to 2436256 tid 2 2020-01-16T00:14:20.454 INFO:tasks.rados.rados.0.smithi123.stdout:1833: writing smithi12313934-32 from 3225803 to 3327918 tid 3 2020-01-16T00:14:20.454 INFO:tasks.rados.rados.0.smithi123.stdout:1834: read oid 21 snap -1 2020-01-16T00:14:20.454 INFO:tasks.rados.rados.0.smithi123.stdout:1834: expect (ObjNum 576 snap 0 seq_num 576) 2020-01-16T00:14:20.455 INFO:tasks.rados.rados.0.smithi123.stdout:1835: write (excl) oid 17 current snap is 0 2020-01-16T00:14:20.455 INFO:tasks.rados.rados.0.smithi123.stdout:1835: seq_num 675 ranges {544725=632139,1716169=522227,2736873=512491} 2020-01-16T00:14:20.455 INFO:tasks.rados.rados.0.smithi123.stdout:1835: writing smithi12313934-17 from 544725 to 1176864 tid 1 2020-01-16T00:14:20.460 INFO:tasks.rados.rados.0.smithi123.stdout:1835: writing smithi12313934-17 from 1716169 to 2238396 tid 2 2020-01-16T00:14:20.471 INFO:tasks.rados.rados.0.smithi123.stdout:1835: writing smithi12313934-17 from 2736873 to 3249364 tid 3 2020-01-16T00:14:20.471 INFO:tasks.rados.rados.0.smithi123.stdout:1836: read oid 9 snap -1 2020-01-16T00:14:20.471 INFO:tasks.rados.rados.0.smithi123.stdout:1836: expect (ObjNum 672 snap 0 seq_num 672) 2020-01-16T00:14:20.472 INFO:tasks.rados.rados.0.smithi123.stdout:1837: write (excl) oid 24 current snap is 0 2020-01-16T00:14:20.478 INFO:tasks.rados.rados.0.smithi123.stdout:1837: seq_num 676 ranges {561532=707388,2058298=782238} 2020-01-16T00:14:20.482 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:20.470+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:20.486 INFO:tasks.rados.rados.0.smithi123.stdout:1837: writing smithi12313934-24 from 561532 to 1268920 tid 1 2020-01-16T00:14:20.489 INFO:tasks.rados.rados.0.smithi123.stdout:1837: writing smithi12313934-24 from 2058298 to 2840536 tid 2 2020-01-16T00:14:20.489 INFO:tasks.rados.rados.0.smithi123.stdout:1838: write (excl) oid 30 current snap is 0 2020-01-16T00:14:20.490 INFO:tasks.rados.rados.0.smithi123.stdout:1838: seq_num 677 ranges {578339=498990,1833133=525896,2791719=586620} 2020-01-16T00:14:20.504 INFO:tasks.rados.rados.0.smithi123.stdout:1838: writing smithi12313934-30 from 578339 to 1077329 tid 1 2020-01-16T00:14:20.505 INFO:tasks.rados.rados.0.smithi123.stdout:1838: writing smithi12313934-30 from 1833133 to 2359029 tid 2 2020-01-16T00:14:20.506 INFO:tasks.rados.rados.0.smithi123.stdout:1838: writing smithi12313934-30 from 2791719 to 3378339 tid 3 2020-01-16T00:14:20.506 INFO:tasks.rados.rados.0.smithi123.stdout:1839: read oid 46 snap -1 2020-01-16T00:14:20.506 INFO:tasks.rados.rados.0.smithi123.stdout:1839: expect (ObjNum 665 snap 0 seq_num 665) 2020-01-16T00:14:20.507 INFO:tasks.rados.rados.0.smithi123.stdout:1840: write oid 6 current snap is 0 2020-01-16T00:14:20.507 INFO:tasks.rados.rados.0.smithi123.stdout:1840: seq_num 678 ranges {595146=574239,1891615=669554,3219142=176004} 2020-01-16T00:14:20.514 INFO:tasks.rados.rados.0.smithi123.stdout:1840: writing smithi12313934-6 from 595146 to 1169385 tid 1 2020-01-16T00:14:20.518 INFO:tasks.rados.rados.0.smithi123.stdout:1840: writing smithi12313934-6 from 1891615 to 2561169 tid 2 2020-01-16T00:14:20.524 INFO:tasks.rados.rados.0.smithi123.stdout:1840: writing smithi12313934-6 from 3219142 to 3395146 tid 3 2020-01-16T00:14:20.524 INFO:tasks.rados.rados.0.smithi123.stdout:1841: write oid 36 current snap is 0 2020-01-16T00:14:20.524 INFO:tasks.rados.rados.0.smithi123.stdout:1841: seq_num 679 ranges {611953=649488,1950097=529565,3246565=165388} 2020-01-16T00:14:20.530 INFO:tasks.rados.rados.0.smithi123.stdout:1841: writing smithi12313934-36 from 611953 to 1261441 tid 1 2020-01-16T00:14:20.538 INFO:tasks.rados.rados.0.smithi123.stdout:1841: writing smithi12313934-36 from 1950097 to 2479662 tid 2 2020-01-16T00:14:20.540 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:20.544 INFO:tasks.rados.rados.0.smithi123.stdout:1841: writing smithi12313934-36 from 3246565 to 3411953 tid 3 2020-01-16T00:14:20.544 INFO:tasks.rados.rados.0.smithi123.stdout:1842: read oid 16 snap -1 2020-01-16T00:14:20.544 INFO:tasks.rados.rados.0.smithi123.stdout:1842: expect (ObjNum 660 snap 0 seq_num 660) 2020-01-16T00:14:20.545 INFO:tasks.rados.rados.0.smithi123.stdout:1843: read oid 3 snap -1 2020-01-16T00:14:20.545 INFO:tasks.rados.rados.0.smithi123.stdout:1843: expect (ObjNum 653 snap 0 seq_num 653) 2020-01-16T00:14:20.545 INFO:tasks.rados.rados.0.smithi123.stdout:1844: write oid 22 current snap is 0 2020-01-16T00:14:20.545 INFO:tasks.rados.rados.0.smithi123.stdout:1844: seq_num 680 ranges {628760=724737,1892226=673223,3157635=271125} 2020-01-16T00:14:20.551 INFO:tasks.rados.rados.0.smithi123.stdout:1844: writing smithi12313934-22 from 628760 to 1353497 tid 1 2020-01-16T00:14:20.552 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-01-16T00:14:20.558 INFO:tasks.rados.rados.0.smithi123.stdout:1844: writing smithi12313934-22 from 1892226 to 2565449 tid 2 2020-01-16T00:14:20.564 INFO:tasks.rados.rados.0.smithi123.stdout:1844: writing smithi12313934-22 from 3157635 to 3428760 tid 3 2020-01-16T00:14:20.564 INFO:tasks.rados.rados.0.smithi123.stdout:1845: read oid 11 snap -1 2020-01-16T00:14:20.564 INFO:tasks.rados.rados.0.smithi123.stdout:1845: expect (ObjNum 579 snap 0 seq_num 579) 2020-01-16T00:14:20.564 INFO:tasks.rados.rados.0.smithi123.stdout:1846: read oid 2 snap -1 2020-01-16T00:14:20.564 INFO:tasks.rados.rados.0.smithi123.stdout:1846: expect (ObjNum 667 snap 0 seq_num 667) 2020-01-16T00:14:20.564 INFO:tasks.rados.rados.0.smithi123.stdout:1847: read oid 4 snap -1 2020-01-16T00:14:20.565 INFO:tasks.rados.rados.0.smithi123.stdout:1847: expect deleted 2020-01-16T00:14:20.565 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:20.565 INFO:tasks.rados.rados.0.smithi123.stdout:1832: expect (ObjNum 669 snap 0 seq_num 669) 2020-01-16T00:14:20.571 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:20.568+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:20.649 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:20.658 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-01-16T00:14:20.699 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:20.698+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:20.728 INFO:tasks.rados.rados.0.smithi123.stdout:1833: finishing write tid 1 to smithi12313934-32 2020-01-16T00:14:20.728 INFO:tasks.rados.rados.0.smithi123.stdout:1833: finishing write tid 2 to smithi12313934-32 2020-01-16T00:14:20.728 INFO:tasks.rados.rados.0.smithi123.stdout:1833: finishing write tid 3 to smithi12313934-32 2020-01-16T00:14:20.728 INFO:tasks.rados.rados.0.smithi123.stdout:1833: finishing write tid 5 to smithi12313934-32 2020-01-16T00:14:20.728 INFO:tasks.rados.rados.0.smithi123.stdout:1833: finishing write tid 6 to smithi12313934-32 2020-01-16T00:14:20.729 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 32 v 186 (ObjNum 674 snap 0 seq_num 674) dirty exists 2020-01-16T00:14:20.729 INFO:tasks.rados.rados.0.smithi123.stdout:1833: left oid 32 (ObjNum 674 snap 0 seq_num 674) 2020-01-16T00:14:20.729 INFO:tasks.rados.rados.0.smithi123.stdout:1835: finishing write tid 1 to smithi12313934-17 2020-01-16T00:14:20.729 INFO:tasks.rados.rados.0.smithi123.stdout:1835: finishing write tid 2 to smithi12313934-17 2020-01-16T00:14:20.729 INFO:tasks.rados.rados.0.smithi123.stdout:1834: expect (ObjNum 576 snap 0 seq_num 576) 2020-01-16T00:14:20.747 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-01-16T00:14:20.773 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:20.768+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:20.821 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:20.831 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-01-16T00:14:20.876 INFO:tasks.rados.rados.0.smithi123.stdout:1835: finishing write tid 3 to smithi12313934-17 2020-01-16T00:14:20.876 INFO:tasks.rados.rados.0.smithi123.stdout:1835: finishing write tid 5 to smithi12313934-17 2020-01-16T00:14:20.877 INFO:tasks.rados.rados.0.smithi123.stdout:1835: finishing write tid 6 to smithi12313934-17 2020-01-16T00:14:20.877 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 17 v 208 (ObjNum 675 snap 0 seq_num 675) dirty exists 2020-01-16T00:14:20.877 INFO:tasks.rados.rados.0.smithi123.stdout:1835: left oid 17 (ObjNum 675 snap 0 seq_num 675) 2020-01-16T00:14:20.877 INFO:tasks.rados.rados.0.smithi123.stdout:1836: expect (ObjNum 672 snap 0 seq_num 672) 2020-01-16T00:14:20.878 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:20.874+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:20.898 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:14:20.917 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:14:20.937 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:14:20.950 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:20.950 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:14:20.950 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:14:20.950 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:20.958 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:14:20.959 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-01-16T00:14:20.975 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:20.972+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:20.981 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:14:21.004 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:14:21.027 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:14:21.034 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-01-16T00:14:21.034 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-01-16T00:14:21.034 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-01-16T00:14:21.035 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-01-16T00:14:21.035 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-01-16T00:14:21.035 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENXIO: problem getting command descriptions from osd.5 2020-01-16T00:14:21.035 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: problem getting command descriptions from osd.5 2020-01-16T00:14:21.035 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-01-16T00:14:21.035 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-01-16T00:14:21.037 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-01-16T00:14:21.038 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-01-16T00:14:21.069 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2020-01-16T00:14:21.076 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:21.072+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:21.080 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:21.080 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:14:21.081 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:14:21.081 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:14:21.081 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:21.090 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-01-16T00:14:21.100 INFO:tasks.rados.rados.0.smithi123.stdout:1837: finishing write tid 1 to smithi12313934-24 2020-01-16T00:14:21.101 INFO:tasks.rados.rados.0.smithi123.stdout:1837: finishing write tid 2 to smithi12313934-24 2020-01-16T00:14:21.101 INFO:tasks.rados.rados.0.smithi123.stdout:1837: finishing write tid 4 to smithi12313934-24 2020-01-16T00:14:21.101 INFO:tasks.rados.rados.0.smithi123.stdout:1837: finishing write tid 5 to smithi12313934-24 2020-01-16T00:14:21.101 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 24 v 190 (ObjNum 676 snap 0 seq_num 676) dirty exists 2020-01-16T00:14:21.101 INFO:tasks.rados.rados.0.smithi123.stdout:1837: left oid 24 (ObjNum 676 snap 0 seq_num 676) 2020-01-16T00:14:21.101 INFO:tasks.rados.rados.0.smithi123.stdout:1838: finishing write tid 1 to smithi12313934-30 2020-01-16T00:14:21.101 INFO:tasks.rados.rados.0.smithi123.stdout:1832: done (15 left) 2020-01-16T00:14:21.102 INFO:tasks.rados.rados.0.smithi123.stdout:1833: done (14 left) 2020-01-16T00:14:21.102 INFO:tasks.rados.rados.0.smithi123.stdout:1834: done (13 left) 2020-01-16T00:14:21.102 INFO:tasks.rados.rados.0.smithi123.stdout:1835: done (12 left) 2020-01-16T00:14:21.102 INFO:tasks.rados.rados.0.smithi123.stdout:1836: done (11 left) 2020-01-16T00:14:21.102 INFO:tasks.rados.rados.0.smithi123.stdout:1837: done (10 left) 2020-01-16T00:14:21.102 INFO:tasks.rados.rados.0.smithi123.stdout:1848: read oid 33 snap -1 2020-01-16T00:14:21.102 INFO:tasks.rados.rados.0.smithi123.stdout:1848: expect deleted 2020-01-16T00:14:21.103 INFO:tasks.rados.rados.0.smithi123.stdout:1849: write oid 37 current snap is 0 2020-01-16T00:14:21.103 INFO:tasks.rados.rados.0.smithi123.stdout:1849: seq_num 681 ranges {645567=799986,1950708=533234,3185058=260509} 2020-01-16T00:14:21.106 INFO:tasks.rados.rados.0.smithi123.stdout:1849: writing smithi12313934-37 from 645567 to 1445553 tid 1 2020-01-16T00:14:21.112 INFO:tasks.rados.rados.0.smithi123.stdout:1849: writing smithi12313934-37 from 1950708 to 2483942 tid 2 2020-01-16T00:14:21.118 INFO:tasks.rados.rados.0.smithi123.stdout:1849: writing smithi12313934-37 from 3185058 to 3445567 tid 3 2020-01-16T00:14:21.118 INFO:tasks.rados.rados.0.smithi123.stdout:1850: read oid 24 snap -1 2020-01-16T00:14:21.119 INFO:tasks.rados.rados.0.smithi123.stdout:1850: expect (ObjNum 676 snap 0 seq_num 676) 2020-01-16T00:14:21.119 INFO:tasks.rados.rados.0.smithi123.stdout:1851: write (excl) oid 25 current snap is 0 2020-01-16T00:14:21.119 INFO:tasks.rados.rados.0.smithi123.stdout:1851: seq_num 682 ranges {662374=475235,1609190=676892,2812481=624101} 2020-01-16T00:14:21.124 INFO:tasks.rados.rados.0.smithi123.stdout:1851: writing smithi12313934-25 from 662374 to 1137609 tid 1 2020-01-16T00:14:21.131 INFO:tasks.rados.rados.0.smithi123.stdout:1851: writing smithi12313934-25 from 1609190 to 2286082 tid 2 2020-01-16T00:14:21.141 INFO:tasks.rados.rados.0.smithi123.stdout:1851: writing smithi12313934-25 from 2812481 to 3436582 tid 3 2020-01-16T00:14:21.142 INFO:tasks.rados.rados.0.smithi123.stdout:1852: write oid 43 current snap is 0 2020-01-16T00:14:21.142 INFO:tasks.rados.rados.0.smithi123.stdout:1852: seq_num 683 ranges {679181=550484,1667672=536903,2839904=435373} 2020-01-16T00:14:21.144 INFO:tasks.rados.rados.0.smithi123.stdout:1852: writing smithi12313934-43 from 679181 to 1229665 tid 1 2020-01-16T00:14:21.149 INFO:tasks.rados.rados.0.smithi123.stdout:1852: writing smithi12313934-43 from 1667672 to 2204575 tid 2 2020-01-16T00:14:21.156 INFO:tasks.rados.rados.0.smithi123.stdout:1852: writing smithi12313934-43 from 2839904 to 3275277 tid 3 2020-01-16T00:14:21.157 INFO:tasks.rados.rados.0.smithi123.stdout:1853: read oid 49 snap -1 2020-01-16T00:14:21.157 INFO:tasks.rados.rados.0.smithi123.stdout:1853: expect (ObjNum 671 snap 0 seq_num 671) 2020-01-16T00:14:21.157 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:21.157 INFO:tasks.rados.rados.0.smithi123.stdout:1839: expect (ObjNum 665 snap 0 seq_num 665) 2020-01-16T00:14:21.177 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:21.174+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:21.229 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:21.229 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:14:21.229 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:14:21.229 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:14:21.229 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.230 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.4.0:114 3.9 e24/20 3:9fc687b2:::smithi12313934-44:head v 24'74, mlcod=24'72)", 2020-01-16T00:14:21.230 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:14:21.230 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 102.901605007, 2020-01-16T00:14:21.230 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010399429, 2020-01-16T00:14:21.230 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:21.230 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:21.230 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:21.230 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.231 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:21.231 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:14:21.231 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:21.231 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.231 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.231 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:21.231 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:14:21.232 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:21.232 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.232 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.232 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:21.232 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325276+0000", 2020-01-16T00:14:21.232 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0046127069999999997 2020-01-16T00:14:21.232 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.232 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.233 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:21.233 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329889+0000", 2020-01-16T00:14:21.233 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3519999999999998e-05 2020-01-16T00:14:21.233 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.233 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.233 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:21.234 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329912+0000", 2020-01-16T00:14:21.235 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4070000000000004e-06 2020-01-16T00:14:21.235 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.238 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.238 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:21.238 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329920+0000", 2020-01-16T00:14:21.238 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8817000000000003e-05 2020-01-16T00:14:21.238 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.239 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.239 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:21.239 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329978+0000", 2020-01-16T00:14:21.239 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9721e-05 2020-01-16T00:14:21.239 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.239 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.239 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:21.240 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329998+0000", 2020-01-16T00:14:21.240 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056328139999999999 2020-01-16T00:14:21.240 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.240 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.240 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:21.240 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.335631+0000", 2020-01-16T00:14:21.241 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7170999999999999e-05 2020-01-16T00:14:21.241 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.241 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.241 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:21.241 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.335678+0000", 2020-01-16T00:14:21.241 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010399429 2020-01-16T00:14:21.241 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.242 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:21.242 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.242 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.242 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.242 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:55 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e30)", 2020-01-16T00:14:21.242 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.736036+0000", 2020-01-16T00:14:21.242 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 88.490847787999996, 2020-01-16T00:14:21.243 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010974536, 2020-01-16T00:14:21.243 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:21.243 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:21.243 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:21.243 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.243 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:21.243 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736036+0000", 2020-01-16T00:14:21.243 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:21.244 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.244 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.244 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:21.244 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736036+0000", 2020-01-16T00:14:21.244 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:21.244 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.244 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.245 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:21.245 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736034+0000", 2020-01-16T00:14:21.245 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043589699999999998 2020-01-16T00:14:21.245 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.245 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.245 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:21.245 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736469+0000", 2020-01-16T00:14:21.246 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9882e-05 2020-01-16T00:14:21.246 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.246 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.246 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:21.246 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736489+0000", 2020-01-16T00:14:21.246 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8339999999999999e-06 2020-01-16T00:14:21.246 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.247 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.247 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:21.247 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736497+0000", 2020-01-16T00:14:21.247 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5354999999999999e-05 2020-01-16T00:14:21.247 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.247 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.247 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:21.248 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736572+0000", 2020-01-16T00:14:21.248 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5228000000000004e-05 2020-01-16T00:14:21.248 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.248 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.248 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:21.248 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736668+0000", 2020-01-16T00:14:21.248 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017212999999999999 2020-01-16T00:14:21.249 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.249 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.249 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:21.249 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736840+0000", 2020-01-16T00:14:21.249 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0099057020000000006 2020-01-16T00:14:21.249 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.249 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.249 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.250 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746746+0000", 2020-01-16T00:14:21.250 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0586000000000005e-05 2020-01-16T00:14:21.250 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.250 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.250 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:21.250 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746816+0000", 2020-01-16T00:14:21.250 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2445000000000002e-05 2020-01-16T00:14:21.251 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.251 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.251 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.251 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746889+0000", 2020-01-16T00:14:21.251 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6839000000000002e-05 2020-01-16T00:14:21.251 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.251 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.252 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:21.252 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746955+0000", 2020-01-16T00:14:21.252 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5074000000000002e-05 2020-01-16T00:14:21.252 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.252 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.252 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:21.252 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.747011+0000", 2020-01-16T00:14:21.252 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010974536 2020-01-16T00:14:21.253 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.253 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:21.253 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.253 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.253 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.253 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:56 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e30)", 2020-01-16T00:14:21.253 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.736514+0000", 2020-01-16T00:14:21.254 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 88.490369952999998, 2020-01-16T00:14:21.254 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011660584999999999, 2020-01-16T00:14:21.254 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:21.254 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:21.254 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:21.255 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.255 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:21.255 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736514+0000", 2020-01-16T00:14:21.255 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:21.255 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.255 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.255 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:21.256 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736514+0000", 2020-01-16T00:14:21.256 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:21.256 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.256 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.256 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:21.256 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736513+0000", 2020-01-16T00:14:21.256 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9677000000000004e-05 2020-01-16T00:14:21.257 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.257 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.257 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:21.257 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736583+0000", 2020-01-16T00:14:21.257 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5269999999999998e-06 2020-01-16T00:14:21.257 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.258 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.258 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:21.258 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736591+0000", 2020-01-16T00:14:21.258 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5769999999999996e-06 2020-01-16T00:14:21.258 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.258 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.258 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:21.259 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736599+0000", 2020-01-16T00:14:21.259 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0053593310000000002 2020-01-16T00:14:21.259 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.259 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.259 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:21.259 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.741958+0000", 2020-01-16T00:14:21.259 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014164099999999999 2020-01-16T00:14:21.260 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.260 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.260 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:21.260 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.742100+0000", 2020-01-16T00:14:21.260 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020126000000000001 2020-01-16T00:14:21.260 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.260 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.261 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:21.261 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.742301+0000", 2020-01-16T00:14:21.261 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0045052859999999998 2020-01-16T00:14:21.261 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.261 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.261 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.262 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746806+0000", 2020-01-16T00:14:21.262 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00070802699999999996 2020-01-16T00:14:21.262 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.262 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.262 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:21.262 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.747514+0000", 2020-01-16T00:14:21.262 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00059247200000000003 2020-01-16T00:14:21.263 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.263 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.263 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.263 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748107+0000", 2020-01-16T00:14:21.263 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1870999999999999e-05 2020-01-16T00:14:21.263 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.263 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.264 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:21.264 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748149+0000", 2020-01-16T00:14:21.264 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5831000000000001e-05 2020-01-16T00:14:21.264 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.264 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.264 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:21.264 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748174+0000", 2020-01-16T00:14:21.264 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011660584999999999 2020-01-16T00:14:21.265 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.265 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:21.265 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.265 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.265 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.265 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:70 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e35)", 2020-01-16T00:14:21.265 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.985494+0000", 2020-01-16T00:14:21.266 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 75.241390156999998, 2020-01-16T00:14:21.266 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013746267, 2020-01-16T00:14:21.266 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:21.266 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:21.266 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:21.266 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.267 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:21.267 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985494+0000", 2020-01-16T00:14:21.267 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:21.267 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.267 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.267 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:21.267 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985494+0000", 2020-01-16T00:14:21.268 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:21.268 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.268 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.268 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:21.268 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985490+0000", 2020-01-16T00:14:21.268 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00077792099999999999 2020-01-16T00:14:21.268 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.269 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.269 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:21.269 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986268+0000", 2020-01-16T00:14:21.269 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9718000000000001e-05 2020-01-16T00:14:21.269 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.269 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.269 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:21.270 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986318+0000", 2020-01-16T00:14:21.270 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.4349999999999997e-06 2020-01-16T00:14:21.270 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.270 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.270 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:21.270 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986326+0000", 2020-01-16T00:14:21.270 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3298999999999999e-05 2020-01-16T00:14:21.271 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.271 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.271 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:21.271 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986380+0000", 2020-01-16T00:14:21.271 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4969999999999995e-05 2020-01-16T00:14:21.271 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.271 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.272 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:21.272 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986455+0000", 2020-01-16T00:14:21.272 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000110587 2020-01-16T00:14:21.272 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.272 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.272 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:21.272 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986565+0000", 2020-01-16T00:14:21.273 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063931600000000002 2020-01-16T00:14:21.273 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.273 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.273 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:21.273 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.992958+0000", 2020-01-16T00:14:21.273 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013008480000000001 2020-01-16T00:14:21.273 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.274 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.274 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.274 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.994259+0000", 2020-01-16T00:14:21.274 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048399979999999999 2020-01-16T00:14:21.274 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.274 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.274 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.274 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999099+0000", 2020-01-16T00:14:21.275 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1874999999999998e-05 2020-01-16T00:14:21.275 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.275 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.275 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:21.275 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999181+0000", 2020-01-16T00:14:21.275 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8865e-05 2020-01-16T00:14:21.275 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.276 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.276 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:21.276 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999240+0000", 2020-01-16T00:14:21.276 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013746267 2020-01-16T00:14:21.276 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.276 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:21.276 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.277 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.277 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.277 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:71 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e35)", 2020-01-16T00:14:21.277 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.986341+0000", 2020-01-16T00:14:21.277 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 75.240542796, 2020-01-16T00:14:21.277 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013252807, 2020-01-16T00:14:21.278 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:21.278 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:21.278 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:21.278 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.278 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:21.278 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986341+0000", 2020-01-16T00:14:21.278 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:21.279 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.279 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.279 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:21.279 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986341+0000", 2020-01-16T00:14:21.279 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:21.279 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.279 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.280 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:21.280 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986340+0000", 2020-01-16T00:14:21.280 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000103867 2020-01-16T00:14:21.280 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.280 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.280 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:21.280 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986444+0000", 2020-01-16T00:14:21.281 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0104e-05 2020-01-16T00:14:21.281 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.281 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.281 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:21.281 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986454+0000", 2020-01-16T00:14:21.281 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4220000000000002e-06 2020-01-16T00:14:21.281 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.282 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.282 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:21.282 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986458+0000", 2020-01-16T00:14:21.282 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033523070000000001 2020-01-16T00:14:21.282 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.282 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.282 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:21.283 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.989811+0000", 2020-01-16T00:14:21.283 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3296999999999999e-05 2020-01-16T00:14:21.283 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.283 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.283 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:21.283 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.989894+0000", 2020-01-16T00:14:21.283 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000117701 2020-01-16T00:14:21.283 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.284 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.284 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:21.284 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.990012+0000", 2020-01-16T00:14:21.284 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0032873490000000002 2020-01-16T00:14:21.284 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.284 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.284 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:21.285 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993299+0000", 2020-01-16T00:14:21.285 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002422619 2020-01-16T00:14:21.285 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.285 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.285 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.285 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.995722+0000", 2020-01-16T00:14:21.285 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0037839589999999999 2020-01-16T00:14:21.285 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.286 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.286 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.286 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999506+0000", 2020-01-16T00:14:21.286 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8559999999999997e-05 2020-01-16T00:14:21.286 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.286 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.286 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:21.287 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999544+0000", 2020-01-16T00:14:21.287 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9599999999999999e-05 2020-01-16T00:14:21.287 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.287 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.287 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:21.287 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999594+0000", 2020-01-16T00:14:21.287 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013252807 2020-01-16T00:14:21.288 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.288 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:21.288 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.288 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.288 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.288 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:77 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e36)", 2020-01-16T00:14:21.288 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.555854+0000", 2020-01-16T00:14:21.289 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 73.671029707000002, 2020-01-16T00:14:21.289 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010455002999999999, 2020-01-16T00:14:21.289 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:21.289 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:21.289 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:21.289 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.290 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:21.290 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555854+0000", 2020-01-16T00:14:21.290 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:21.290 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.290 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.290 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:21.290 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555854+0000", 2020-01-16T00:14:21.291 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:21.291 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.291 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.291 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:21.291 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555851+0000", 2020-01-16T00:14:21.291 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00048310200000000001 2020-01-16T00:14:21.291 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.292 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.292 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:21.292 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556334+0000", 2020-01-16T00:14:21.292 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2232e-05 2020-01-16T00:14:21.292 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.292 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.292 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:21.292 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556356+0000", 2020-01-16T00:14:21.293 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9750000000000007e-06 2020-01-16T00:14:21.293 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.293 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.293 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:21.293 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556364+0000", 2020-01-16T00:14:21.293 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8872999999999997e-05 2020-01-16T00:14:21.293 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.294 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.294 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:21.294 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556423+0000", 2020-01-16T00:14:21.294 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8687000000000001e-05 2020-01-16T00:14:21.294 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.294 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.295 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:21.295 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556502+0000", 2020-01-16T00:14:21.295 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013235300000000001 2020-01-16T00:14:21.295 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.295 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.295 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:21.295 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556634+0000", 2020-01-16T00:14:21.296 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0059770520000000001 2020-01-16T00:14:21.296 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.296 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.296 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:21.296 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.562611+0000", 2020-01-16T00:14:21.296 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029492820000000001 2020-01-16T00:14:21.296 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.297 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.297 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.297 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.565561+0000", 2020-01-16T00:14:21.297 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00060620300000000004 2020-01-16T00:14:21.297 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.297 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.298 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.298 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566167+0000", 2020-01-16T00:14:21.298 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3831000000000004e-05 2020-01-16T00:14:21.298 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.298 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.298 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:21.298 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566251+0000", 2020-01-16T00:14:21.299 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8328000000000001e-05 2020-01-16T00:14:21.299 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.299 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.299 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:21.299 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566309+0000", 2020-01-16T00:14:21.299 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010455002999999999 2020-01-16T00:14:21.299 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.300 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:21.300 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.300 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.300 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.300 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:78 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e36)", 2020-01-16T00:14:21.300 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.556378+0000", 2020-01-16T00:14:21.300 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 73.670505774000006, 2020-01-16T00:14:21.301 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010396311, 2020-01-16T00:14:21.301 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:21.301 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:21.301 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:21.301 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.301 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:21.301 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556378+0000", 2020-01-16T00:14:21.302 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:21.302 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.302 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.302 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:21.302 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556378+0000", 2020-01-16T00:14:21.302 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:21.302 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.303 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.303 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:21.303 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556377+0000", 2020-01-16T00:14:21.303 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4307000000000001e-05 2020-01-16T00:14:21.303 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.304 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.304 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:21.304 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556422+0000", 2020-01-16T00:14:21.304 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9620000000000005e-06 2020-01-16T00:14:21.304 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.304 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.304 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:21.304 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556432+0000", 2020-01-16T00:14:21.305 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5979999999999999e-06 2020-01-16T00:14:21.305 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.305 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.305 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:21.305 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556436+0000", 2020-01-16T00:14:21.305 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021059469999999999 2020-01-16T00:14:21.306 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.306 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.306 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:21.306 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.558542+0000", 2020-01-16T00:14:21.306 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000194458 2020-01-16T00:14:21.306 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.306 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.306 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:21.307 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.558737+0000", 2020-01-16T00:14:21.307 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00032867900000000002 2020-01-16T00:14:21.307 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.307 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.307 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:21.307 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.559065+0000", 2020-01-16T00:14:21.307 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052547499999999999 2020-01-16T00:14:21.308 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.308 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.308 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:21.308 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.564320+0000", 2020-01-16T00:14:21.308 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001565213 2020-01-16T00:14:21.308 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.308 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.309 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.309 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.565885+0000", 2020-01-16T00:14:21.309 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00076994300000000004 2020-01-16T00:14:21.309 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.309 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.309 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.309 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566655+0000", 2020-01-16T00:14:21.310 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7533000000000005e-05 2020-01-16T00:14:21.310 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.310 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.310 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:21.310 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566723+0000", 2020-01-16T00:14:21.310 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1616999999999997e-05 2020-01-16T00:14:21.310 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.311 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.311 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:21.311 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566774+0000", 2020-01-16T00:14:21.311 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010396311 2020-01-16T00:14:21.311 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.311 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:21.311 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.311 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.312 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.312 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:72 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 56832~1408000] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e37)", 2020-01-16T00:14:21.312 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.578753+0000", 2020-01-16T00:14:21.312 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 68.648130471000002, 2020-01-16T00:14:21.312 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011826918, 2020-01-16T00:14:21.312 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:21.312 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:21.313 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:21.313 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.313 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:21.313 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578753+0000", 2020-01-16T00:14:21.313 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:21.313 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.313 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.313 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:21.314 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578753+0000", 2020-01-16T00:14:21.314 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:21.314 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.314 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.314 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:21.314 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578751+0000", 2020-01-16T00:14:21.314 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00042457699999999998 2020-01-16T00:14:21.315 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.315 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.315 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:21.315 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579175+0000", 2020-01-16T00:14:21.315 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5404999999999999e-05 2020-01-16T00:14:21.315 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.315 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.316 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:21.316 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579191+0000", 2020-01-16T00:14:21.316 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6149999999999996e-06 2020-01-16T00:14:21.316 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.316 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.316 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:21.316 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579196+0000", 2020-01-16T00:14:21.316 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000104794 2020-01-16T00:14:21.317 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.317 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.317 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:21.317 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579301+0000", 2020-01-16T00:14:21.317 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020947199999999999 2020-01-16T00:14:21.317 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.317 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.318 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:21.318 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579511+0000", 2020-01-16T00:14:21.318 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034328400000000002 2020-01-16T00:14:21.318 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.318 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.318 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:21.318 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579854+0000", 2020-01-16T00:14:21.319 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.005078117 2020-01-16T00:14:21.319 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.319 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.319 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:21.319 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.584932+0000", 2020-01-16T00:14:21.319 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0014702160000000001 2020-01-16T00:14:21.319 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.320 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.320 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.320 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.586402+0000", 2020-01-16T00:14:21.320 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040623589999999998 2020-01-16T00:14:21.320 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.320 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.321 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.321 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590465+0000", 2020-01-16T00:14:21.321 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4478000000000002e-05 2020-01-16T00:14:21.321 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.322 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.322 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:21.322 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590529+0000", 2020-01-16T00:14:21.323 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1121000000000001e-05 2020-01-16T00:14:21.323 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.323 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.323 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:21.323 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590580+0000", 2020-01-16T00:14:21.323 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011826918 2020-01-16T00:14:21.323 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.324 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:21.324 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.324 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.324 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.324 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:73 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 1922048~118272] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e37)", 2020-01-16T00:14:21.324 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:14:21.324 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 68.647677673999993, 2020-01-16T00:14:21.325 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011750764, 2020-01-16T00:14:21.325 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:21.325 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:21.325 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:21.325 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.325 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:21.325 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:14:21.326 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:21.326 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.326 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.326 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:21.326 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:14:21.326 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:21.326 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.326 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.327 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:21.327 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:14:21.327 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2780000000000001e-05 2020-01-16T00:14:21.327 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.327 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.327 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:21.327 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579238+0000", 2020-01-16T00:14:21.328 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4341999999999999e-05 2020-01-16T00:14:21.328 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.328 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.328 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:21.328 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579253+0000", 2020-01-16T00:14:21.328 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1920000000000001e-06 2020-01-16T00:14:21.328 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.328 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.329 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:21.329 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579256+0000", 2020-01-16T00:14:21.329 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003224902 2020-01-16T00:14:21.329 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.329 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.329 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:21.329 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.582481+0000", 2020-01-16T00:14:21.330 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000180729 2020-01-16T00:14:21.330 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.330 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.330 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:21.330 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.582662+0000", 2020-01-16T00:14:21.330 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00025106100000000001 2020-01-16T00:14:21.330 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.330 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.331 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:21.331 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.582913+0000", 2020-01-16T00:14:21.331 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020814449999999999 2020-01-16T00:14:21.331 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.331 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.331 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:21.331 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.584994+0000", 2020-01-16T00:14:21.332 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001717243 2020-01-16T00:14:21.332 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.332 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.332 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.332 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.586711+0000", 2020-01-16T00:14:21.332 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041349869999999997 2020-01-16T00:14:21.332 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.332 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.333 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.333 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590846+0000", 2020-01-16T00:14:21.333 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3565000000000006e-05 2020-01-16T00:14:21.333 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.333 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.333 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:21.333 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590910+0000", 2020-01-16T00:14:21.334 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7002999999999997e-05 2020-01-16T00:14:21.334 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.334 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.334 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:21.334 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590957+0000", 2020-01-16T00:14:21.334 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011750764 2020-01-16T00:14:21.334 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.335 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:21.335 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.335 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.335 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.335 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.4.13:66 2.3 2:c3b82d87:::smithi12313934-44 low_tier:head [write 169472~818176] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e44)", 2020-01-16T00:14:21.335 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:25.278583+0000", 2020-01-16T00:14:21.335 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 55.948300650999997, 2020-01-16T00:14:21.335 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010294862, 2020-01-16T00:14:21.336 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:21.336 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:21.336 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:21.336 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.336 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:21.336 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.278583+0000", 2020-01-16T00:14:21.336 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:21.336 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.337 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.337 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:21.337 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.278583+0000", 2020-01-16T00:14:21.337 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:21.337 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.337 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.337 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:21.338 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.278581+0000", 2020-01-16T00:14:21.338 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00136721 2020-01-16T00:14:21.338 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.338 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.338 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:21.339 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.279948+0000", 2020-01-16T00:14:21.339 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6109999999999999e-05 2020-01-16T00:14:21.339 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.339 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.339 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:21.339 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.279974+0000", 2020-01-16T00:14:21.339 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.102e-05 2020-01-16T00:14:21.339 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.340 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.340 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:21.340 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.279985+0000", 2020-01-16T00:14:21.340 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6035e-05 2020-01-16T00:14:21.340 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.340 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.340 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:21.341 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.280061+0000", 2020-01-16T00:14:21.341 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000176243 2020-01-16T00:14:21.341 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.341 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.341 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:21.341 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.280238+0000", 2020-01-16T00:14:21.341 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019866099999999999 2020-01-16T00:14:21.342 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.342 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.342 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 7", 2020-01-16T00:14:21.342 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.280436+0000", 2020-01-16T00:14:21.342 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0049761830000000003 2020-01-16T00:14:21.342 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.342 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.343 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.343 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.285412+0000", 2020-01-16T00:14:21.343 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033347339999999998 2020-01-16T00:14:21.343 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.343 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.343 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:21.343 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.288747+0000", 2020-01-16T00:14:21.343 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9156000000000001e-05 2020-01-16T00:14:21.344 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.344 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.344 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:21.344 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.288806+0000", 2020-01-16T00:14:21.344 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1643000000000005e-05 2020-01-16T00:14:21.344 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.344 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.345 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:21.345 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:25.288878+0000", 2020-01-16T00:14:21.345 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010294862 2020-01-16T00:14:21.345 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.345 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:21.345 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.346 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.346 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.346 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:77 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 56832~1408000] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e45)", 2020-01-16T00:14:21.346 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.584566+0000", 2020-01-16T00:14:21.346 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 51.642318000000003, 2020-01-16T00:14:21.346 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017037775000000002, 2020-01-16T00:14:21.346 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:21.346 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:21.347 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:21.347 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.347 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:21.347 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584566+0000", 2020-01-16T00:14:21.347 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:21.347 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.347 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.348 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:21.348 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584566+0000", 2020-01-16T00:14:21.348 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:21.348 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.348 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.348 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:21.348 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584564+0000", 2020-01-16T00:14:21.348 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043024300000000002 2020-01-16T00:14:21.349 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.349 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.349 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:21.349 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584994+0000", 2020-01-16T00:14:21.349 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9485999999999999e-05 2020-01-16T00:14:21.349 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.349 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.350 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:21.350 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585014+0000", 2020-01-16T00:14:21.350 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8909999999999995e-06 2020-01-16T00:14:21.350 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.351 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.351 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:21.351 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585022+0000", 2020-01-16T00:14:21.351 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039114459999999998 2020-01-16T00:14:21.351 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.351 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.351 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:21.351 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588933+0000", 2020-01-16T00:14:21.352 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011178700000000001 2020-01-16T00:14:21.352 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.352 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.352 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:21.352 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589045+0000", 2020-01-16T00:14:21.352 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016967300000000001 2020-01-16T00:14:21.352 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.352 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.353 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:21.353 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589214+0000", 2020-01-16T00:14:21.353 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012142356 2020-01-16T00:14:21.353 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.353 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.353 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.353 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601357+0000", 2020-01-16T00:14:21.354 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.1466999999999997e-05 2020-01-16T00:14:21.354 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.354 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.354 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.354 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601448+0000", 2020-01-16T00:14:21.354 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4817000000000001e-05 2020-01-16T00:14:21.354 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.354 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.355 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:21.355 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601493+0000", 2020-01-16T00:14:21.355 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9734000000000001e-05 2020-01-16T00:14:21.355 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.355 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.355 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:21.355 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601553+0000", 2020-01-16T00:14:21.356 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0677999999999998e-05 2020-01-16T00:14:21.356 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.356 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.356 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:21.356 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601604+0000", 2020-01-16T00:14:21.356 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017037775000000002 2020-01-16T00:14:21.356 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.356 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:21.357 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.357 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.357 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.357 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:78 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 1922048~118272] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e45)", 2020-01-16T00:14:21.357 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.585030+0000", 2020-01-16T00:14:21.357 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 51.641853277999999, 2020-01-16T00:14:21.357 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016699964000000001, 2020-01-16T00:14:21.358 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:21.358 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:21.358 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:21.358 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.358 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:21.358 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585030+0000", 2020-01-16T00:14:21.358 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:21.358 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.359 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.359 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:21.359 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585030+0000", 2020-01-16T00:14:21.359 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:21.359 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.359 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.359 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:21.360 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585030+0000", 2020-01-16T00:14:21.360 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7085000000000001e-05 2020-01-16T00:14:21.360 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.360 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.360 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:21.360 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585077+0000", 2020-01-16T00:14:21.360 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5380000000000008e-06 2020-01-16T00:14:21.360 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.361 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.361 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:21.361 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585086+0000", 2020-01-16T00:14:21.361 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5410000000000002e-06 2020-01-16T00:14:21.361 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.361 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.361 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:21.362 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585091+0000", 2020-01-16T00:14:21.362 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0088486620000000002 2020-01-16T00:14:21.362 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.362 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.362 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:21.362 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.593940+0000", 2020-01-16T00:14:21.362 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112594 2020-01-16T00:14:21.362 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.363 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.363 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:21.363 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.594052+0000", 2020-01-16T00:14:21.363 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019893999999999999 2020-01-16T00:14:21.363 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.363 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.363 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:21.364 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.594251+0000", 2020-01-16T00:14:21.364 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0071632730000000004 2020-01-16T00:14:21.364 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.364 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.364 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.364 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601414+0000", 2020-01-16T00:14:21.364 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4672999999999996e-05 2020-01-16T00:14:21.364 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.365 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.365 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.365 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601479+0000", 2020-01-16T00:14:21.365 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000158453 2020-01-16T00:14:21.365 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.365 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.365 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:21.366 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601638+0000", 2020-01-16T00:14:21.366 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8605e-05 2020-01-16T00:14:21.366 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.366 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.366 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:21.366 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601686+0000", 2020-01-16T00:14:21.366 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4283999999999998e-05 2020-01-16T00:14:21.367 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.367 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.367 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:21.367 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601730+0000", 2020-01-16T00:14:21.367 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016699964000000001 2020-01-16T00:14:21.367 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.367 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:21.367 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.368 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.368 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.368 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:127 2.1 2:a4df40e2:::smithi12313934-42 low_tier:head [write 1364992~1145344] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e45)", 2020-01-16T00:14:21.368 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.627308+0000", 2020-01-16T00:14:21.368 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 51.599575825999999, 2020-01-16T00:14:21.368 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011068883999999999, 2020-01-16T00:14:21.368 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:21.369 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:21.369 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:21.369 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.370 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:21.370 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627308+0000", 2020-01-16T00:14:21.370 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:21.370 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.371 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.371 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:21.372 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627308+0000", 2020-01-16T00:14:21.372 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:21.372 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.372 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.372 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:21.372 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627307+0000", 2020-01-16T00:14:21.372 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000278597 2020-01-16T00:14:21.373 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.373 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.373 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:21.373 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627586+0000", 2020-01-16T00:14:21.373 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9169999999999997e-06 2020-01-16T00:14:21.373 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.374 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.374 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:21.374 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627591+0000", 2020-01-16T00:14:21.374 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0479999999999999e-06 2020-01-16T00:14:21.374 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.375 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.375 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:21.375 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627594+0000", 2020-01-16T00:14:21.375 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030987269999999999 2020-01-16T00:14:21.375 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.376 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.376 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:21.376 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.630693+0000", 2020-01-16T00:14:21.376 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000103688 2020-01-16T00:14:21.376 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.377 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.377 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:21.377 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.630796+0000", 2020-01-16T00:14:21.377 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000144606 2020-01-16T00:14:21.378 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.378 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.378 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:21.378 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.630941+0000", 2020-01-16T00:14:21.379 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039264269999999997 2020-01-16T00:14:21.379 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.379 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.379 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:21.379 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.634867+0000", 2020-01-16T00:14:21.380 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00038581200000000003 2020-01-16T00:14:21.380 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.380 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.380 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.380 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.635253+0000", 2020-01-16T00:14:21.381 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030568879999999998 2020-01-16T00:14:21.381 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.381 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.381 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.381 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.638310+0000", 2020-01-16T00:14:21.382 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8390999999999999e-05 2020-01-16T00:14:21.382 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.382 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.382 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:21.382 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.638348+0000", 2020-01-16T00:14:21.383 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8402999999999999e-05 2020-01-16T00:14:21.383 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.383 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.383 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:21.383 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.638377+0000", 2020-01-16T00:14:21.383 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011068883999999999 2020-01-16T00:14:21.384 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.384 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:21.384 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.384 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.384 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.384 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.6.13:30 2.3 2:fc791323:::smithi12313934-2 low_tier:head [write 476160~1240064] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e48)", 2020-01-16T00:14:21.385 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:39.466268+0000", 2020-01-16T00:14:21.385 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 41.760615522000002, 2020-01-16T00:14:21.385 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010593112, 2020-01-16T00:14:21.385 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:21.385 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:21.385 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:21.386 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.386 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:21.386 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.466268+0000", 2020-01-16T00:14:21.386 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:21.386 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.386 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.386 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:21.387 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.466268+0000", 2020-01-16T00:14:21.387 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:21.387 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.387 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.387 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:21.387 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.466266+0000", 2020-01-16T00:14:21.388 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012489210000000001 2020-01-16T00:14:21.388 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.388 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.388 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:21.388 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467515+0000", 2020-01-16T00:14:21.388 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8411999999999999e-05 2020-01-16T00:14:21.388 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.389 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.389 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:21.389 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467533+0000", 2020-01-16T00:14:21.389 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9319999999999996e-06 2020-01-16T00:14:21.390 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.390 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.390 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:21.390 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467541+0000", 2020-01-16T00:14:21.390 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5289999999999993e-05 2020-01-16T00:14:21.391 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.391 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.391 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:21.391 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467607+0000", 2020-01-16T00:14:21.392 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010962 2020-01-16T00:14:21.392 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.392 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.392 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:21.392 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467716+0000", 2020-01-16T00:14:21.393 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020883699999999999 2020-01-16T00:14:21.393 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.393 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.394 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 7", 2020-01-16T00:14:21.394 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467925+0000", 2020-01-16T00:14:21.394 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063872829999999997 2020-01-16T00:14:21.394 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.395 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.395 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:21.395 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.474312+0000", 2020-01-16T00:14:21.395 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0023165769999999998 2020-01-16T00:14:21.395 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.396 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.396 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.396 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.476629+0000", 2020-01-16T00:14:21.396 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8543999999999999e-05 2020-01-16T00:14:21.396 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.396 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.396 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:21.397 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.476707+0000", 2020-01-16T00:14:21.397 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000153914 2020-01-16T00:14:21.397 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.397 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.397 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:21.397 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.476861+0000", 2020-01-16T00:14:21.397 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010593112 2020-01-16T00:14:21.398 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.398 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:21.398 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.398 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.398 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.398 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:155 2.3 2:c9873fc7:::smithi12313934-30 low_tier:head [write 1542656~711168] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e50)", 2020-01-16T00:14:21.398 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:43.769943+0000", 2020-01-16T00:14:21.398 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 37.456940596000003, 2020-01-16T00:14:21.399 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010710127999999999, 2020-01-16T00:14:21.399 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:21.399 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:21.399 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:21.399 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.399 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:21.399 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.769943+0000", 2020-01-16T00:14:21.399 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:21.400 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.400 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.400 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:21.400 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.769943+0000", 2020-01-16T00:14:21.400 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:21.400 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.400 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.400 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:21.401 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.769942+0000", 2020-01-16T00:14:21.401 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017878100000000001 2020-01-16T00:14:21.401 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.401 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.401 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:21.401 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.770121+0000", 2020-01-16T00:14:21.401 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2980000000000003e-06 2020-01-16T00:14:21.402 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.402 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.402 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:21.402 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.770129+0000", 2020-01-16T00:14:21.403 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9129999999999999e-06 2020-01-16T00:14:21.403 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.403 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.404 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:21.404 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.770131+0000", 2020-01-16T00:14:21.404 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0053233519999999999 2020-01-16T00:14:21.404 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.405 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.405 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:21.405 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.775455+0000", 2020-01-16T00:14:21.405 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012385099999999999 2020-01-16T00:14:21.405 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.405 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.405 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:21.406 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.775579+0000", 2020-01-16T00:14:21.406 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000211554 2020-01-16T00:14:21.406 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.406 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.406 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 7", 2020-01-16T00:14:21.406 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.775790+0000", 2020-01-16T00:14:21.406 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029614210000000001 2020-01-16T00:14:21.407 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.407 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.408 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:21.409 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.778752+0000", 2020-01-16T00:14:21.409 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001798928 2020-01-16T00:14:21.409 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.409 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.409 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.409 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.780551+0000", 2020-01-16T00:14:21.410 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9413999999999999e-05 2020-01-16T00:14:21.410 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.410 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.410 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:21.410 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.780590+0000", 2020-01-16T00:14:21.410 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3275999999999995e-05 2020-01-16T00:14:21.410 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.411 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.411 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:21.411 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.780653+0000", 2020-01-16T00:14:21.411 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010710127999999999 2020-01-16T00:14:21.411 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.411 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:21.411 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.412 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.412 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.412 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:171 2.1 2:a935ad67:::smithi12313934-50 low_tier:head [write 2422272~237568] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e55)", 2020-01-16T00:14:21.412 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:58.196749+0000", 2020-01-16T00:14:21.412 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 23.030134311000001, 2020-01-16T00:14:21.412 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010677759, 2020-01-16T00:14:21.412 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:21.413 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:21.413 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:21.413 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.413 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:21.413 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196749+0000", 2020-01-16T00:14:21.413 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:21.413 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.413 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.414 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:21.414 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196749+0000", 2020-01-16T00:14:21.414 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:21.414 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.414 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.414 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:21.414 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196749+0000", 2020-01-16T00:14:21.415 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6604000000000001e-05 2020-01-16T00:14:21.415 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.415 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.415 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:21.415 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196815+0000", 2020-01-16T00:14:21.415 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4900000000000006e-06 2020-01-16T00:14:21.415 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.416 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.416 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:21.416 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196825+0000", 2020-01-16T00:14:21.416 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3989999999999997e-06 2020-01-16T00:14:21.416 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.416 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.416 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:21.417 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196829+0000", 2020-01-16T00:14:21.417 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0076943869999999996 2020-01-16T00:14:21.417 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.417 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.417 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:21.417 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.204524+0000", 2020-01-16T00:14:21.418 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000124745 2020-01-16T00:14:21.418 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.418 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.418 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:21.418 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.204649+0000", 2020-01-16T00:14:21.419 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018801700000000001 2020-01-16T00:14:21.419 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.419 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.419 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:21.419 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.204837+0000", 2020-01-16T00:14:21.419 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002354765 2020-01-16T00:14:21.420 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.420 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.420 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.420 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.207191+0000", 2020-01-16T00:14:21.420 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.6249999999999996e-06 2020-01-16T00:14:21.420 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.420 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.421 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:21.421 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.207200+0000", 2020-01-16T00:14:21.421 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000162246 2020-01-16T00:14:21.421 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.421 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.421 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.421 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.207362+0000", 2020-01-16T00:14:21.422 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5117999999999998e-05 2020-01-16T00:14:21.422 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.422 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.422 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:21.422 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.207397+0000", 2020-01-16T00:14:21.422 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9912000000000001e-05 2020-01-16T00:14:21.422 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.423 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.423 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:21.423 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.207427+0000", 2020-01-16T00:14:21.423 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010677759 2020-01-16T00:14:21.423 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.423 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:21.423 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.424 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.424 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.424 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:183 2.1 2:a4df40e2:::smithi12313934-42 low_tier:head [write 70656~727552] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e58)", 2020-01-16T00:14:21.424 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.021222+0000", 2020-01-16T00:14:21.424 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 14.205661742, 2020-01-16T00:14:21.424 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012681151999999999, 2020-01-16T00:14:21.424 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:21.424 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:21.425 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:21.425 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.425 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:21.425 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021222+0000", 2020-01-16T00:14:21.425 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:21.425 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.425 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.425 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:21.426 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021222+0000", 2020-01-16T00:14:21.426 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:21.426 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.426 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.426 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:21.426 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021219+0000", 2020-01-16T00:14:21.426 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020912099999999999 2020-01-16T00:14:21.427 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.427 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.427 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:21.427 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021428+0000", 2020-01-16T00:14:21.427 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8595999999999999e-05 2020-01-16T00:14:21.427 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.427 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.427 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:21.427 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021447+0000", 2020-01-16T00:14:21.428 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2020000000000001e-06 2020-01-16T00:14:21.428 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.428 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.428 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:21.428 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021453+0000", 2020-01-16T00:14:21.428 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5297999999999999e-05 2020-01-16T00:14:21.428 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.429 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.429 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:21.429 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021499+0000", 2020-01-16T00:14:21.429 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2299999999999996e-05 2020-01-16T00:14:21.429 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.429 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.429 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:21.430 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021571+0000", 2020-01-16T00:14:21.430 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011000799999999999 2020-01-16T00:14:21.430 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.430 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.430 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:21.430 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021681+0000", 2020-01-16T00:14:21.430 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0065924879999999996 2020-01-16T00:14:21.431 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.431 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.431 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.431 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.028273+0000", 2020-01-16T00:14:21.431 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022212499999999999 2020-01-16T00:14:21.431 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.431 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.432 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:21.432 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.028495+0000", 2020-01-16T00:14:21.432 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0053014580000000002 2020-01-16T00:14:21.432 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.432 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.432 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.432 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.033797+0000", 2020-01-16T00:14:21.433 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6142e-05 2020-01-16T00:14:21.433 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.433 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.433 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:21.433 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.033843+0000", 2020-01-16T00:14:21.433 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0130999999999997e-05 2020-01-16T00:14:21.433 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.434 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.434 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:21.434 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.033903+0000", 2020-01-16T00:14:21.434 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012681151999999999 2020-01-16T00:14:21.434 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.434 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:21.434 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.435 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.435 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.435 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:184 2.1 2:a4df40e2:::smithi12313934-42 low_tier:head [write 1364992~1145344] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e58)", 2020-01-16T00:14:21.435 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.023539+0000", 2020-01-16T00:14:21.435 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 14.203344623, 2020-01-16T00:14:21.435 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011003893000000001, 2020-01-16T00:14:21.435 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:21.436 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:21.436 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:21.436 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.436 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:21.436 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023539+0000", 2020-01-16T00:14:21.436 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:21.436 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.437 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.437 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:21.437 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023539+0000", 2020-01-16T00:14:21.437 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:21.437 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.437 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.437 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:21.438 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023537+0000", 2020-01-16T00:14:21.438 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00033271700000000001 2020-01-16T00:14:21.438 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.438 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.438 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:21.438 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023870+0000", 2020-01-16T00:14:21.438 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5744999999999999e-05 2020-01-16T00:14:21.438 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.439 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.439 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:21.439 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023886+0000", 2020-01-16T00:14:21.439 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7389999999999994e-06 2020-01-16T00:14:21.439 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.439 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.439 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:21.439 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023895+0000", 2020-01-16T00:14:21.440 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6058999999999998e-05 2020-01-16T00:14:21.440 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.440 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.440 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:21.440 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023971+0000", 2020-01-16T00:14:21.440 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119207 2020-01-16T00:14:21.440 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.441 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.441 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:21.441 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.024091+0000", 2020-01-16T00:14:21.441 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000166074 2020-01-16T00:14:21.441 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.441 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.441 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:21.441 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.024257+0000", 2020-01-16T00:14:21.442 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0047290129999999998 2020-01-16T00:14:21.442 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.442 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.442 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.442 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.028986+0000", 2020-01-16T00:14:21.442 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00028545000000000003 2020-01-16T00:14:21.442 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.443 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.443 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:21.443 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.029271+0000", 2020-01-16T00:14:21.443 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052013049999999998 2020-01-16T00:14:21.443 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.443 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.443 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.444 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.034472+0000", 2020-01-16T00:14:21.444 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7738999999999999e-05 2020-01-16T00:14:21.444 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.444 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.444 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:21.444 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.034510+0000", 2020-01-16T00:14:21.444 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2913000000000001e-05 2020-01-16T00:14:21.445 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.445 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.445 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:21.445 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.034543+0000", 2020-01-16T00:14:21.445 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011003893000000001 2020-01-16T00:14:21.445 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.445 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:21.446 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.446 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.446 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.446 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:98 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 56832~1408000] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e58)", 2020-01-16T00:14:21.446 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.049295+0000", 2020-01-16T00:14:21.454 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 14.177588573, 2020-01-16T00:14:21.454 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012327324000000001, 2020-01-16T00:14:21.454 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:21.454 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:21.454 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:21.454 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.455 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:21.455 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049295+0000", 2020-01-16T00:14:21.455 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:21.455 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.455 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.455 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:21.455 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049295+0000", 2020-01-16T00:14:21.456 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:21.456 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.456 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.456 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:21.456 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049294+0000", 2020-01-16T00:14:21.456 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00036515000000000001 2020-01-16T00:14:21.456 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.456 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.457 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:21.457 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049659+0000", 2020-01-16T00:14:21.457 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4299e-05 2020-01-16T00:14:21.457 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.457 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.457 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:21.457 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049673+0000", 2020-01-16T00:14:21.458 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1250000000000001e-06 2020-01-16T00:14:21.458 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.458 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.458 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:21.458 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049678+0000", 2020-01-16T00:14:21.458 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9937e-05 2020-01-16T00:14:21.458 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.459 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.459 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:21.459 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049728+0000", 2020-01-16T00:14:21.459 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.1094999999999994e-05 2020-01-16T00:14:21.459 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.459 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.460 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:21.460 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049819+0000", 2020-01-16T00:14:21.460 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012621999999999999 2020-01-16T00:14:21.460 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.460 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.460 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:21.461 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049946+0000", 2020-01-16T00:14:21.461 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0081844340000000008 2020-01-16T00:14:21.461 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.461 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.462 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:21.462 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.058130+0000", 2020-01-16T00:14:21.462 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00058478000000000002 2020-01-16T00:14:21.462 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.462 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.462 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.462 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.058715+0000", 2020-01-16T00:14:21.463 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0027436349999999999 2020-01-16T00:14:21.463 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.463 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.463 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.463 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.061459+0000", 2020-01-16T00:14:21.463 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1755000000000007e-05 2020-01-16T00:14:21.463 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.464 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.464 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:21.464 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.061530+0000", 2020-01-16T00:14:21.464 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2236000000000003e-05 2020-01-16T00:14:21.464 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.464 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.464 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:21.464 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.061623+0000", 2020-01-16T00:14:21.465 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012327324000000001 2020-01-16T00:14:21.465 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.465 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:21.465 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.465 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.465 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.465 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:99 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 1922048~118272] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e58)", 2020-01-16T00:14:21.466 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.049687+0000", 2020-01-16T00:14:21.466 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 14.177196595, 2020-01-16T00:14:21.466 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012484960999999999, 2020-01-16T00:14:21.466 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:21.466 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:21.466 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:21.466 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.467 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:21.467 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049687+0000", 2020-01-16T00:14:21.467 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:21.467 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.467 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.467 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:21.467 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049687+0000", 2020-01-16T00:14:21.467 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:21.468 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.468 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.468 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:21.468 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049687+0000", 2020-01-16T00:14:21.468 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7373999999999999e-05 2020-01-16T00:14:21.468 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.468 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.469 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:21.469 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049714+0000", 2020-01-16T00:14:21.469 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2030999999999997e-05 2020-01-16T00:14:21.469 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.469 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.469 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:21.470 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049766+0000", 2020-01-16T00:14:21.470 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5559999999999999e-06 2020-01-16T00:14:21.470 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.470 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.470 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:21.470 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049770+0000", 2020-01-16T00:14:21.470 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0026935779999999999 2020-01-16T00:14:21.470 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.471 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.471 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:21.471 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.052463+0000", 2020-01-16T00:14:21.471 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5037999999999995e-05 2020-01-16T00:14:21.471 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.472 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.472 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:21.472 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.052538+0000", 2020-01-16T00:14:21.472 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000116791 2020-01-16T00:14:21.472 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.473 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.473 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:21.473 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.052655+0000", 2020-01-16T00:14:21.473 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063608579999999996 2020-01-16T00:14:21.473 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.474 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.474 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:21.474 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.059016+0000", 2020-01-16T00:14:21.474 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000131872 2020-01-16T00:14:21.474 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.474 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.475 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.475 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.059148+0000", 2020-01-16T00:14:21.475 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002907677 2020-01-16T00:14:21.475 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.475 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.475 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.475 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.062055+0000", 2020-01-16T00:14:21.476 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9624999999999999e-05 2020-01-16T00:14:21.476 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.476 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.476 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:21.476 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.062115+0000", 2020-01-16T00:14:21.476 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7096999999999999e-05 2020-01-16T00:14:21.476 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:21.477 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:21.477 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:21.477 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.062172+0000", 2020-01-16T00:14:21.477 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012484960999999999 2020-01-16T00:14:21.477 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.477 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:21.477 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.477 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:21.478 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:21.478 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:21.480 INFO:tasks.rados.rados.0.smithi123.stdout:1838: finishing write tid 2 to smithi12313934-30 2020-01-16T00:14:21.481 INFO:tasks.rados.rados.0.smithi123.stdout:1838: finishing write tid 3 to smithi12313934-30 2020-01-16T00:14:21.481 INFO:tasks.rados.rados.0.smithi123.stdout:1838: finishing write tid 5 to smithi12313934-30 2020-01-16T00:14:21.481 INFO:tasks.rados.rados.0.smithi123.stdout:1840: finishing write tid 1 to smithi12313934-6 2020-01-16T00:14:21.481 INFO:tasks.rados.rados.0.smithi123.stdout:1838: finishing write tid 6 to smithi12313934-30 2020-01-16T00:14:21.481 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 30 v 195 (ObjNum 677 snap 0 seq_num 677) dirty exists 2020-01-16T00:14:21.481 INFO:tasks.rados.rados.0.smithi123.stdout:1838: left oid 30 (ObjNum 677 snap 0 seq_num 677) 2020-01-16T00:14:21.482 INFO:tasks.rados.rados.0.smithi123.stdout:1840: finishing write tid 2 to smithi12313934-6 2020-01-16T00:14:21.482 INFO:tasks.rados.rados.0.smithi123.stdout:1840: finishing write tid 3 to smithi12313934-6 2020-01-16T00:14:21.482 INFO:tasks.rados.rados.0.smithi123.stdout:1840: finishing write tid 5 to smithi12313934-6 2020-01-16T00:14:21.482 INFO:tasks.rados.rados.0.smithi123.stdout:1840: finishing write tid 6 to smithi12313934-6 2020-01-16T00:14:21.482 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 6 v 193 (ObjNum 678 snap 0 seq_num 678) dirty exists 2020-01-16T00:14:21.482 INFO:tasks.rados.rados.0.smithi123.stdout:1840: left oid 6 (ObjNum 678 snap 0 seq_num 678) 2020-01-16T00:14:21.482 INFO:tasks.rados.rados.0.smithi123.stdout:1841: finishing write tid 1 to smithi12313934-36 2020-01-16T00:14:21.482 INFO:tasks.rados.rados.0.smithi123.stdout:1841: finishing write tid 2 to smithi12313934-36 2020-01-16T00:14:21.483 INFO:tasks.rados.rados.0.smithi123.stdout:1838: done (15 left) 2020-01-16T00:14:21.483 INFO:tasks.rados.rados.0.smithi123.stdout:1839: done (14 left) 2020-01-16T00:14:21.483 INFO:tasks.rados.rados.0.smithi123.stdout:1840: done (13 left) 2020-01-16T00:14:21.483 INFO:tasks.rados.rados.0.smithi123.stdout:1854: read oid 31 snap -1 2020-01-16T00:14:21.483 INFO:tasks.rados.rados.0.smithi123.stdout:1854: expect (ObjNum 656 snap 0 seq_num 656) 2020-01-16T00:14:21.483 INFO:tasks.rados.rados.0.smithi123.stdout:1855: write oid 6 current snap is 0 2020-01-16T00:14:21.484 INFO:tasks.rados.rados.0.smithi123.stdout:1855: seq_num 684 ranges {695988=625733,2009801=680561,3150974=345014} 2020-01-16T00:14:21.484 INFO:tasks.rados.rados.0.smithi123.stdout:1855: writing smithi12313934-6 from 695988 to 1321721 tid 1 2020-01-16T00:14:21.484 INFO:tasks.rados.rados.0.smithi123.stdout:1855: writing smithi12313934-6 from 2009801 to 2690362 tid 2 2020-01-16T00:14:21.484 INFO:tasks.rados.rados.0.smithi123.stdout:1855: writing smithi12313934-6 from 3150974 to 3495988 tid 3 2020-01-16T00:14:21.484 INFO:tasks.rados.rados.0.smithi123.stdout:1856: write (excl) oid 7 current snap is 0 2020-01-16T00:14:21.485 INFO:tasks.rados.rados.0.smithi123.stdout:1856: seq_num 685 ranges {712795=417335,1784636=540572,2894750=457917} 2020-01-16T00:14:21.485 INFO:tasks.rados.rados.0.smithi123.stdout:1856: writing smithi12313934-7 from 712795 to 1130130 tid 1 2020-01-16T00:14:21.485 INFO:tasks.rados.rados.0.smithi123.stdout:1856: writing smithi12313934-7 from 1784636 to 2325208 tid 2 2020-01-16T00:14:21.485 INFO:tasks.rados.rados.0.smithi123.stdout:1856: writing smithi12313934-7 from 2894750 to 3352667 tid 3 2020-01-16T00:14:21.486 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:21.486 INFO:tasks.rados.rados.0.smithi123.stdout:1841: finishing write tid 3 to smithi12313934-36 2020-01-16T00:14:21.486 INFO:tasks.rados.rados.0.smithi123.stdout:1841: finishing write tid 5 to smithi12313934-36 2020-01-16T00:14:21.486 INFO:tasks.rados.rados.0.smithi123.stdout:1841: finishing write tid 6 to smithi12313934-36 2020-01-16T00:14:21.486 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 36 v 118 (ObjNum 679 snap 0 seq_num 679) dirty exists 2020-01-16T00:14:21.487 INFO:tasks.rados.rados.0.smithi123.stdout:1841: left oid 36 (ObjNum 679 snap 0 seq_num 679) 2020-01-16T00:14:21.487 INFO:tasks.rados.rados.0.smithi123.stdout:1843: expect (ObjNum 653 snap 0 seq_num 653) 2020-01-16T00:14:21.487 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:21.478+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:21.488 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-01-16T00:14:21.580 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:21.578+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:21.598 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:21.598 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:14:21.598 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:14:21.599 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:21.608 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-01-16T00:14:21.632 INFO:tasks.rados.rados.0.smithi123.stdout:1842: expect (ObjNum 660 snap 0 seq_num 660) 2020-01-16T00:14:21.681 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:21.680+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:21.710 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:21.710 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:14:21.710 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:14:21.710 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:14:21.710 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:21.719 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-01-16T00:14:21.774 INFO:tasks.rados.rados.0.smithi123.stdout:1844: finishing write tid 1 to smithi12313934-22 2020-01-16T00:14:21.774 INFO:tasks.rados.rados.0.smithi123.stdout:1841: done (15 left) 2020-01-16T00:14:21.775 INFO:tasks.rados.rados.0.smithi123.stdout:1842: done (14 left) 2020-01-16T00:14:21.775 INFO:tasks.rados.rados.0.smithi123.stdout:1843: done (13 left) 2020-01-16T00:14:21.775 INFO:tasks.rados.rados.0.smithi123.stdout:1857: read oid 32 snap -1 2020-01-16T00:14:21.775 INFO:tasks.rados.rados.0.smithi123.stdout:1857: expect (ObjNum 674 snap 0 seq_num 674) 2020-01-16T00:14:21.775 INFO:tasks.rados.rados.0.smithi123.stdout:1858: write (excl) oid 18 current snap is 0 2020-01-16T00:14:21.775 INFO:tasks.rados.rados.0.smithi123.stdout:1858: seq_num 686 ranges {729602=492584,1843118=684230,3322173=207429} 2020-01-16T00:14:21.781 INFO:tasks.rados.rados.0.smithi123.stdout:1858: writing smithi12313934-18 from 729602 to 1222186 tid 1 2020-01-16T00:14:21.781 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:21.778+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:21.790 INFO:tasks.rados.rados.0.smithi123.stdout:1858: writing smithi12313934-18 from 1843118 to 2527348 tid 2 2020-01-16T00:14:21.796 INFO:tasks.rados.rados.0.smithi123.stdout:1858: writing smithi12313934-18 from 3322173 to 3529602 tid 3 2020-01-16T00:14:21.796 INFO:tasks.rados.rados.0.smithi123.stdout:1859: read oid 36 snap -1 2020-01-16T00:14:21.796 INFO:tasks.rados.rados.0.smithi123.stdout:1859: expect (ObjNum 679 snap 0 seq_num 679) 2020-01-16T00:14:21.797 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:21.797 INFO:tasks.rados.rados.0.smithi123.stdout:1844: finishing write tid 2 to smithi12313934-22 2020-01-16T00:14:21.797 INFO:tasks.rados.rados.0.smithi123.stdout:1844: finishing write tid 3 to smithi12313934-22 2020-01-16T00:14:21.797 INFO:tasks.rados.rados.0.smithi123.stdout:1844: finishing write tid 5 to smithi12313934-22 2020-01-16T00:14:21.797 INFO:tasks.rados.rados.0.smithi123.stdout:1844: finishing write tid 6 to smithi12313934-22 2020-01-16T00:14:21.797 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 22 v 45 (ObjNum 680 snap 0 seq_num 680) dirty exists 2020-01-16T00:14:21.797 INFO:tasks.rados.rados.0.smithi123.stdout:1844: left oid 22 (ObjNum 680 snap 0 seq_num 680) 2020-01-16T00:14:21.798 INFO:tasks.rados.rados.0.smithi123.stdout:1845: expect (ObjNum 579 snap 0 seq_num 579) 2020-01-16T00:14:21.870 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:21.870 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:14:21.870 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:14:21.870 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:14:21.870 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.871 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:943 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:14:21.871 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:14:21.871 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 103.953922913, 2020-01-16T00:14:21.871 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018847004000000001, 2020-01-16T00:14:21.871 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:21.871 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:21.871 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:21.872 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:21.872 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:21.872 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 943 2020-01-16T00:14:21.872 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.872 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:21.872 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.872 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:21.872 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:14:21.873 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:21.873 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.873 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.873 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:21.873 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:14:21.873 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:21.873 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.874 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.874 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:21.874 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914475+0000", 2020-01-16T00:14:21.874 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.508e-05 2020-01-16T00:14:21.874 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.874 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.875 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:21.875 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914490+0000", 2020-01-16T00:14:21.875 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.857e-05 2020-01-16T00:14:21.875 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.875 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.875 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:21.875 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914509+0000", 2020-01-16T00:14:21.876 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.46e-06 2020-01-16T00:14:21.876 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.876 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.876 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:21.876 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914515+0000", 2020-01-16T00:14:21.876 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2612999999999999e-05 2020-01-16T00:14:21.876 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.876 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.877 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:21.877 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914568+0000", 2020-01-16T00:14:21.877 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028312299999999998 2020-01-16T00:14:21.877 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.877 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.877 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:21.877 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914851+0000", 2020-01-16T00:14:21.877 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012282533 2020-01-16T00:14:21.878 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.878 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.878 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:21.878 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927133+0000", 2020-01-16T00:14:21.878 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4839999999999995e-05 2020-01-16T00:14:21.878 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.878 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.878 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:21.879 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927208+0000", 2020-01-16T00:14:21.879 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0047425610000000002 2020-01-16T00:14:21.879 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.879 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.879 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:21.879 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.931951+0000", 2020-01-16T00:14:21.879 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3628999999999994e-05 2020-01-16T00:14:21.879 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.880 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.880 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:21.880 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.932045+0000", 2020-01-16T00:14:21.880 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012810180000000001 2020-01-16T00:14:21.880 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.880 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.880 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:21.881 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933326+0000", 2020-01-16T00:14:21.881 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018847004000000001 2020-01-16T00:14:21.881 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:21.881 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:21.881 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:21.881 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.881 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.881 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:944 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:14:21.883 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:14:21.883 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 103.95387335700001, 2020-01-16T00:14:21.883 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019086901999999999, 2020-01-16T00:14:21.883 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:21.884 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:21.884 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:21.884 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:21.884 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:21.884 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 944 2020-01-16T00:14:21.884 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.884 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:21.884 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.885 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:21.885 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:14:21.885 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:21.885 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.885 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.885 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:21.885 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:14:21.885 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:21.886 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.886 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.886 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:21.886 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914527+0000", 2020-01-16T00:14:21.886 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7509999999999999e-06 2020-01-16T00:14:21.886 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.886 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.887 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:21.887 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914532+0000", 2020-01-16T00:14:21.887 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5149999999999997e-06 2020-01-16T00:14:21.887 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.887 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.887 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:21.887 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914538+0000", 2020-01-16T00:14:21.887 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1890000000000007e-06 2020-01-16T00:14:21.888 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.888 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.888 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:21.888 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914546+0000", 2020-01-16T00:14:21.888 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000354951 2020-01-16T00:14:21.888 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.888 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.888 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:21.889 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914901+0000", 2020-01-16T00:14:21.889 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1594000000000002e-05 2020-01-16T00:14:21.889 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.889 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.889 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:21.889 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914942+0000", 2020-01-16T00:14:21.889 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012300070999999999 2020-01-16T00:14:21.889 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.890 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.890 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:21.890 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927242+0000", 2020-01-16T00:14:21.890 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4076999999999997e-05 2020-01-16T00:14:21.890 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.890 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.890 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:21.890 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927296+0000", 2020-01-16T00:14:21.891 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0060609180000000002 2020-01-16T00:14:21.891 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.891 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.891 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:21.891 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933357+0000", 2020-01-16T00:14:21.891 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4488999999999999e-05 2020-01-16T00:14:21.891 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.891 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.892 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:21.892 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933422+0000", 2020-01-16T00:14:21.892 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019318300000000001 2020-01-16T00:14:21.892 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.892 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.892 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:21.892 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933615+0000", 2020-01-16T00:14:21.893 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019086901999999999 2020-01-16T00:14:21.893 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:21.893 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:21.893 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:21.893 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.893 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.893 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:945 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:14:21.893 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:14:21.894 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 103.95384615099999, 2020-01-16T00:14:21.894 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019342986, 2020-01-16T00:14:21.894 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:21.894 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:21.894 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:21.894 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:21.894 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:21.894 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 945 2020-01-16T00:14:21.895 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.895 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:21.895 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.895 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:21.895 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:14:21.895 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:21.895 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.895 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.896 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:21.896 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:14:21.896 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:21.896 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.896 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.896 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:21.897 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:14:21.897 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4580000000000002e-06 2020-01-16T00:14:21.897 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.897 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.897 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:21.897 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914558+0000", 2020-01-16T00:14:21.897 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7829999999999998e-06 2020-01-16T00:14:21.898 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.898 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.898 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:21.898 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914563+0000", 2020-01-16T00:14:21.898 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0001e-05 2020-01-16T00:14:21.898 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.898 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.898 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:21.899 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914573+0000", 2020-01-16T00:14:21.899 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00040365600000000001 2020-01-16T00:14:21.899 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.899 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.899 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:21.899 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914976+0000", 2020-01-16T00:14:21.899 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1390000000000001e-05 2020-01-16T00:14:21.899 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.900 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.900 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:21.900 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.915028+0000", 2020-01-16T00:14:21.900 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012288843000000001 2020-01-16T00:14:21.900 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.900 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.900 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:21.900 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927317+0000", 2020-01-16T00:14:21.901 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7871000000000002e-05 2020-01-16T00:14:21.901 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.901 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.901 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:21.901 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927375+0000", 2020-01-16T00:14:21.901 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0062580589999999998 2020-01-16T00:14:21.901 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.901 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.902 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:21.902 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933633+0000", 2020-01-16T00:14:21.902 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7086999999999997e-05 2020-01-16T00:14:21.902 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.902 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.902 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:21.902 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933690+0000", 2020-01-16T00:14:21.903 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020855799999999999 2020-01-16T00:14:21.903 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.903 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.903 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:21.903 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933898+0000", 2020-01-16T00:14:21.903 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019342986 2020-01-16T00:14:21.903 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:21.903 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:21.904 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:21.904 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.904 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.904 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:981 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:14:21.904 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:14:21.904 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 103.553069259, 2020-01-16T00:14:21.904 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021222494000000001, 2020-01-16T00:14:21.904 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:21.905 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:21.905 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:21.905 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:21.905 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:21.905 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 981 2020-01-16T00:14:21.905 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.905 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:21.905 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.906 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:21.906 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:14:21.906 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:21.906 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.906 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.906 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:21.906 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:14:21.906 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999948 2020-01-16T00:14:21.907 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.907 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.907 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:21.907 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315327+0000", 2020-01-16T00:14:21.907 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.508e-05 2020-01-16T00:14:21.907 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.907 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.908 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:21.908 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315352+0000", 2020-01-16T00:14:21.908 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1183999999999998e-05 2020-01-16T00:14:21.908 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.908 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.908 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:21.908 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315383+0000", 2020-01-16T00:14:21.908 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7180000000000002e-05 2020-01-16T00:14:21.909 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.909 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.909 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:21.909 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315401+0000", 2020-01-16T00:14:21.909 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017290599999999999 2020-01-16T00:14:21.909 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.909 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.909 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:21.910 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315574+0000", 2020-01-16T00:14:21.910 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00047908100000000002 2020-01-16T00:14:21.910 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.910 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.910 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:21.910 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316053+0000", 2020-01-16T00:14:21.910 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011808947 2020-01-16T00:14:21.910 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.911 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.911 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:21.911 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.327862+0000", 2020-01-16T00:14:21.911 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010067499999999999 2020-01-16T00:14:21.911 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.911 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.911 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:21.912 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.327962+0000", 2020-01-16T00:14:21.912 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0074745139999999998 2020-01-16T00:14:21.912 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.912 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.912 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:21.912 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.335437+0000", 2020-01-16T00:14:21.912 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000103403 2020-01-16T00:14:21.912 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.913 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.913 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:21.913 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.335540+0000", 2020-01-16T00:14:21.913 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001014544 2020-01-16T00:14:21.913 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.913 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.913 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:21.913 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336555+0000", 2020-01-16T00:14:21.914 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021222494000000001 2020-01-16T00:14:21.914 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:21.914 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:21.914 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:21.914 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.914 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.914 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:982 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:14:21.914 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:14:21.915 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 103.552966735, 2020-01-16T00:14:21.915 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.022087721000000001, 2020-01-16T00:14:21.915 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:21.915 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:21.915 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:21.915 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:21.915 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:21.915 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 982 2020-01-16T00:14:21.916 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.916 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:21.916 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.916 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:21.916 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:14:21.916 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:21.916 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.916 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.917 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:21.917 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:14:21.917 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:21.917 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.917 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.917 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:21.917 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315433+0000", 2020-01-16T00:14:21.918 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1347e-05 2020-01-16T00:14:21.918 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.918 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.918 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:21.918 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315445+0000", 2020-01-16T00:14:21.918 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4525999999999999e-05 2020-01-16T00:14:21.918 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.918 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.919 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:21.919 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315459+0000", 2020-01-16T00:14:21.919 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.024e-05 2020-01-16T00:14:21.919 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.919 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.919 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:21.919 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315470+0000", 2020-01-16T00:14:21.919 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000736468 2020-01-16T00:14:21.920 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.920 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.920 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:21.920 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316206+0000", 2020-01-16T00:14:21.920 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000132967 2020-01-16T00:14:21.920 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.920 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.920 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:21.921 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316339+0000", 2020-01-16T00:14:21.921 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011701717 2020-01-16T00:14:21.921 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.921 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.921 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:21.921 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328041+0000", 2020-01-16T00:14:21.921 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000120339 2020-01-16T00:14:21.922 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.922 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.922 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:21.922 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328161+0000", 2020-01-16T00:14:21.922 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0084341260000000001 2020-01-16T00:14:21.922 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.922 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.922 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:21.923 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336595+0000", 2020-01-16T00:14:21.923 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5247999999999993e-05 2020-01-16T00:14:21.923 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.923 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.923 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:21.923 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336690+0000", 2020-01-16T00:14:21.923 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00083204000000000004 2020-01-16T00:14:21.923 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.924 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.924 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:21.924 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337522+0000", 2020-01-16T00:14:21.924 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.022087721000000001 2020-01-16T00:14:21.924 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:21.924 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:21.924 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:21.924 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.925 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.925 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:983 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:14:21.925 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:14:21.925 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 103.55284702500001, 2020-01-16T00:14:21.925 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023363387999999999, 2020-01-16T00:14:21.925 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:21.925 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:21.926 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:21.926 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:21.926 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:21.926 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 983 2020-01-16T00:14:21.926 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.926 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:21.926 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.926 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:21.927 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:14:21.927 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:21.927 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.927 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.927 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:21.927 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:14:21.927 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:21.928 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.928 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.928 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:21.928 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315553+0000", 2020-01-16T00:14:21.928 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1760000000000004e-06 2020-01-16T00:14:21.928 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.928 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.928 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:21.929 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315560+0000", 2020-01-16T00:14:21.929 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3587000000000001e-05 2020-01-16T00:14:21.929 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.929 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.929 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:21.929 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315574+0000", 2020-01-16T00:14:21.929 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7800000000000003e-06 2020-01-16T00:14:21.929 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.930 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.930 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:21.930 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315581+0000", 2020-01-16T00:14:21.930 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000823024 2020-01-16T00:14:21.930 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.930 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.930 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:21.930 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316404+0000", 2020-01-16T00:14:21.931 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2413e-05 2020-01-16T00:14:21.931 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.931 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.931 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:21.931 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316466+0000", 2020-01-16T00:14:21.931 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011738561999999999 2020-01-16T00:14:21.931 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.931 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.932 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:21.932 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328205+0000", 2020-01-16T00:14:21.932 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6783999999999997e-05 2020-01-16T00:14:21.932 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.932 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.932 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:21.932 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328262+0000", 2020-01-16T00:14:21.933 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0093022560000000001 2020-01-16T00:14:21.933 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.933 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.933 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:21.933 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337564+0000", 2020-01-16T00:14:21.933 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000183151 2020-01-16T00:14:21.933 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.933 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.934 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:21.934 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337747+0000", 2020-01-16T00:14:21.934 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011707449999999999 2020-01-16T00:14:21.934 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.934 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.934 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:21.934 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.338918+0000", 2020-01-16T00:14:21.934 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023363387999999999 2020-01-16T00:14:21.935 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:21.935 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:21.935 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:21.935 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.935 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.935 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:986 3.9 3:93dfe6f1:::smithi12313934-40:head [write 2941380~419987] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:14:21.935 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:14:21.935 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 103.38754724899999, 2020-01-16T00:14:21.936 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016978400000000001, 2020-01-16T00:14:21.936 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:21.936 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:21.936 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:21.936 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:21.936 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:21.936 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 986 2020-01-16T00:14:21.937 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.937 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:21.938 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.938 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:21.938 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:14:21.938 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:21.938 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.938 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.938 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:21.939 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:14:21.939 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:21.939 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.939 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.939 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:21.939 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480852+0000", 2020-01-16T00:14:21.939 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042503899999999999 2020-01-16T00:14:21.940 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.940 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.940 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:21.940 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481277+0000", 2020-01-16T00:14:21.940 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3097e-05 2020-01-16T00:14:21.940 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.940 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.940 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:21.941 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481290+0000", 2020-01-16T00:14:21.941 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8460000000000002e-06 2020-01-16T00:14:21.941 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.941 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.941 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:21.941 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481297+0000", 2020-01-16T00:14:21.941 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8443999999999999e-05 2020-01-16T00:14:21.941 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.942 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.942 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:21.942 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481346+0000", 2020-01-16T00:14:21.942 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022115399999999999 2020-01-16T00:14:21.942 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.942 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.942 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:21.942 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481567+0000", 2020-01-16T00:14:21.943 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012531502 2020-01-16T00:14:21.943 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.943 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.943 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:21.943 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494098+0000", 2020-01-16T00:14:21.943 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.4462999999999993e-05 2020-01-16T00:14:21.943 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.943 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.944 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:21.944 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494193+0000", 2020-01-16T00:14:21.944 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017600999999999999 2020-01-16T00:14:21.944 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.944 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.944 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:14:21.944 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494369+0000", 2020-01-16T00:14:21.944 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0031739619999999998 2020-01-16T00:14:21.945 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.945 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.945 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:21.945 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497543+0000", 2020-01-16T00:14:21.945 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000175107 2020-01-16T00:14:21.945 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.945 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.945 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.946 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497718+0000", 2020-01-16T00:14:21.946 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5207000000000003e-05 2020-01-16T00:14:21.946 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.946 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.946 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.946 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497763+0000", 2020-01-16T00:14:21.946 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7042999999999998e-05 2020-01-16T00:14:21.946 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.947 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.947 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:21.947 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497800+0000", 2020-01-16T00:14:21.947 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2628e-05 2020-01-16T00:14:21.947 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.947 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.947 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:21.948 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497833+0000", 2020-01-16T00:14:21.948 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016978400000000001 2020-01-16T00:14:21.948 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:21.948 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:21.948 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:21.948 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.948 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.948 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:988 3.9 3:93dfe6f1:::smithi12313934-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:14:21.949 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:14:21.949 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 103.387061785, 2020-01-16T00:14:21.949 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017221692, 2020-01-16T00:14:21.949 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:21.949 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:21.949 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:21.949 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:21.949 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:21.950 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 988 2020-01-16T00:14:21.950 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.950 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:21.950 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.950 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:21.950 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:14:21.950 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:21.950 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.951 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.951 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:21.951 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:14:21.951 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:21.951 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.951 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.951 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:21.951 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481339+0000", 2020-01-16T00:14:21.952 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3089999999999997e-06 2020-01-16T00:14:21.952 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.952 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.952 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:21.952 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481343+0000", 2020-01-16T00:14:21.952 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2230000000000004e-06 2020-01-16T00:14:21.952 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.953 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.953 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:21.953 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481350+0000", 2020-01-16T00:14:21.953 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.319e-06 2020-01-16T00:14:21.953 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.953 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.953 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:21.953 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481356+0000", 2020-01-16T00:14:21.954 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000321544 2020-01-16T00:14:21.954 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.954 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.954 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:21.954 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481678+0000", 2020-01-16T00:14:21.954 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6375999999999999e-05 2020-01-16T00:14:21.954 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.954 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.955 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:21.955 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481714+0000", 2020-01-16T00:14:21.955 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015557666 2020-01-16T00:14:21.955 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.955 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.955 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:21.955 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497272+0000", 2020-01-16T00:14:21.955 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012791300000000001 2020-01-16T00:14:21.956 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.956 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.956 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:21.956 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497399+0000", 2020-01-16T00:14:21.956 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00076944600000000002 2020-01-16T00:14:21.956 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.956 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.956 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:21.957 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498169+0000", 2020-01-16T00:14:21.957 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018306799999999999 2020-01-16T00:14:21.957 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.957 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.957 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:21.957 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498352+0000", 2020-01-16T00:14:21.957 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020935999999999999 2020-01-16T00:14:21.957 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.958 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.958 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:21.958 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498561+0000", 2020-01-16T00:14:21.958 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017221692 2020-01-16T00:14:21.958 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:21.958 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:21.958 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:21.958 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.959 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.959 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1239 3.3 3:c204aa4f:::smithi12313934-45:head [setxattr _header (56) in=7b,truncate 2000033] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:14:21.959 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:14:21.959 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 99.277987965999998, 2020-01-16T00:14:21.959 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019161661999999999, 2020-01-16T00:14:21.959 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:21.959 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:21.960 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:21.960 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:21.960 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:21.960 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1239 2020-01-16T00:14:21.960 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.960 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:21.960 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.961 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:21.961 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:14:21.961 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:21.961 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.961 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.961 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:21.961 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:14:21.962 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:21.962 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.962 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.962 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:21.962 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590411+0000", 2020-01-16T00:14:21.962 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3879999999999999e-05 2020-01-16T00:14:21.962 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.962 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.963 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:21.963 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590425+0000", 2020-01-16T00:14:21.963 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6542999999999998e-05 2020-01-16T00:14:21.963 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.963 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.963 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:21.963 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590471+0000", 2020-01-16T00:14:21.963 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8907999999999999e-05 2020-01-16T00:14:21.964 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.964 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.964 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:21.964 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590490+0000", 2020-01-16T00:14:21.964 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0045691960000000002 2020-01-16T00:14:21.964 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.964 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.964 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:21.965 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.595059+0000", 2020-01-16T00:14:21.965 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0035638940000000002 2020-01-16T00:14:21.965 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.965 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.965 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:21.965 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598623+0000", 2020-01-16T00:14:21.965 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0077558310000000004 2020-01-16T00:14:21.965 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.966 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.966 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:21.966 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606379+0000", 2020-01-16T00:14:21.966 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012171999999999999 2020-01-16T00:14:21.966 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.966 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.966 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:21.966 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606501+0000", 2020-01-16T00:14:21.967 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013144200000000001 2020-01-16T00:14:21.967 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.967 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.967 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2020-01-16T00:14:21.967 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606632+0000", 2020-01-16T00:14:21.967 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001288565 2020-01-16T00:14:21.967 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.968 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.968 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:21.968 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607921+0000", 2020-01-16T00:14:21.968 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033399299999999999 2020-01-16T00:14:21.968 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.968 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.968 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.968 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.608255+0000", 2020-01-16T00:14:21.969 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012280870000000001 2020-01-16T00:14:21.969 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.969 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.969 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.969 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609483+0000", 2020-01-16T00:14:21.969 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2286000000000001e-05 2020-01-16T00:14:21.969 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.969 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.970 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:21.970 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609525+0000", 2020-01-16T00:14:21.970 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0121999999999999e-05 2020-01-16T00:14:21.970 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.970 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.970 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:21.970 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609575+0000", 2020-01-16T00:14:21.970 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019161661999999999 2020-01-16T00:14:21.971 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:21.971 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:21.971 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:21.971 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.971 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.971 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1240 3.3 3:c204aa4f:::smithi12313934-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-01-16T00:14:21.971 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:14:21.971 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 99.277887777000004, 2020-01-16T00:14:21.972 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019268828000000002, 2020-01-16T00:14:21.972 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:21.972 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:21.972 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:21.972 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:21.972 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:21.972 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1240 2020-01-16T00:14:21.973 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.973 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:21.973 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.973 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:21.973 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:14:21.973 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:21.973 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.973 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.974 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:21.974 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:14:21.974 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:21.974 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.974 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.974 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:21.975 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590512+0000", 2020-01-16T00:14:21.975 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0525e-05 2020-01-16T00:14:21.975 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.975 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.975 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:21.975 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590522+0000", 2020-01-16T00:14:21.975 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6418000000000001e-05 2020-01-16T00:14:21.975 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.976 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.976 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:21.976 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590539+0000", 2020-01-16T00:14:21.976 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0679999999999999e-05 2020-01-16T00:14:21.976 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.976 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.976 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:21.976 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590549+0000", 2020-01-16T00:14:21.977 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0083626789999999996 2020-01-16T00:14:21.977 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.977 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.977 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:21.977 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598912+0000", 2020-01-16T00:14:21.977 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013344 2020-01-16T00:14:21.977 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.978 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.978 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:21.978 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599045+0000", 2020-01-16T00:14:21.978 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0084002269999999997 2020-01-16T00:14:21.978 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.978 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.978 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:21.978 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607446+0000", 2020-01-16T00:14:21.979 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000109873 2020-01-16T00:14:21.979 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.979 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.979 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:21.979 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607555+0000", 2020-01-16T00:14:21.979 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0020573309999999999 2020-01-16T00:14:21.979 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.979 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.980 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:21.980 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609613+0000", 2020-01-16T00:14:21.980 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6351e-05 2020-01-16T00:14:21.980 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.980 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.980 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:21.980 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609699+0000", 2020-01-16T00:14:21.980 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3374999999999994e-05 2020-01-16T00:14:21.981 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.981 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.981 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:21.981 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609782+0000", 2020-01-16T00:14:21.981 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019268828000000002 2020-01-16T00:14:21.981 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:21.981 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:21.981 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:21.982 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.982 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.982 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3124 3.9 3:9fc687b2:::smithi12313934-44:head [write 1903628~435462] snapc 0=[] ondisk+write+known_if_redirected e44)", 2020-01-16T00:14:21.982 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.276145+0000", 2020-01-16T00:14:21.982 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 56.592256055999997, 2020-01-16T00:14:21.982 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017361898000000001, 2020-01-16T00:14:21.982 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:21.982 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:21.983 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:21.983 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:21.983 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:21.983 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3124 2020-01-16T00:14:21.983 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.983 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:21.983 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.983 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:21.984 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276145+0000", 2020-01-16T00:14:21.984 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:21.984 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.984 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.984 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:21.984 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276145+0000", 2020-01-16T00:14:21.984 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:21.985 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.985 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.985 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:21.985 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276143+0000", 2020-01-16T00:14:21.985 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042136099999999998 2020-01-16T00:14:21.985 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.985 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.985 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:21.986 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276564+0000", 2020-01-16T00:14:21.986 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.349e-05 2020-01-16T00:14:21.986 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.986 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.986 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:21.986 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276578+0000", 2020-01-16T00:14:21.986 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2469999999999999e-06 2020-01-16T00:14:21.986 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.987 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.987 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:21.987 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276583+0000", 2020-01-16T00:14:21.987 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4400000000000001e-05 2020-01-16T00:14:21.987 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.987 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.987 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:21.987 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276637+0000", 2020-01-16T00:14:21.988 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0046361789999999998 2020-01-16T00:14:21.988 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.988 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.988 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:21.988 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281274+0000", 2020-01-16T00:14:21.988 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0090025520000000005 2020-01-16T00:14:21.988 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.988 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.989 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:21.989 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.290276+0000", 2020-01-16T00:14:21.989 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011584200000000001 2020-01-16T00:14:21.989 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.989 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.989 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:21.990 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.290392+0000", 2020-01-16T00:14:21.990 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00025563099999999998 2020-01-16T00:14:21.990 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.990 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.991 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:14:21.991 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.290648+0000", 2020-01-16T00:14:21.991 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0026741360000000001 2020-01-16T00:14:21.991 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.991 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.992 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:21.992 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293322+0000", 2020-01-16T00:14:21.993 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9764000000000002e-05 2020-01-16T00:14:21.993 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.993 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.993 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.993 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293421+0000", 2020-01-16T00:14:21.993 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4779000000000001e-05 2020-01-16T00:14:21.993 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.993 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.994 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:21.994 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293446+0000", 2020-01-16T00:14:21.994 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4628000000000001e-05 2020-01-16T00:14:21.994 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.994 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.994 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:21.994 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293481+0000", 2020-01-16T00:14:21.994 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.6444999999999998e-05 2020-01-16T00:14:21.995 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.995 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.995 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:21.995 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293507+0000", 2020-01-16T00:14:21.995 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017361898000000001 2020-01-16T00:14:21.995 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:21.995 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:21.996 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:21.996 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.996 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.996 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3125 3.9 3:9fc687b2:::smithi12313934-44:head [setxattr _header (56) in=7b,truncate 2386660] snapc 0=[] ondisk+write+known_if_redirected e44)", 2020-01-16T00:14:21.996 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.276594+0000", 2020-01-16T00:14:21.996 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 56.591807283999998, 2020-01-16T00:14:21.996 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017760560000000002, 2020-01-16T00:14:21.997 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:21.997 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:21.997 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:21.997 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:21.997 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:21.997 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3125 2020-01-16T00:14:21.997 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.998 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:21.998 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.998 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:21.998 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276594+0000", 2020-01-16T00:14:21.998 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:21.998 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.998 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.998 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:21.999 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276594+0000", 2020-01-16T00:14:21.999 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:21.999 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:21.999 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:21.999 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:21.999 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276593+0000", 2020-01-16T00:14:21.999 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7320000000000003e-06 2020-01-16T00:14:22.000 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.000 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.000 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:22.000 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276599+0000", 2020-01-16T00:14:22.000 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3302000000000001e-05 2020-01-16T00:14:22.000 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.000 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.000 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:22.001 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276612+0000", 2020-01-16T00:14:22.001 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1659999999999998e-06 2020-01-16T00:14:22.001 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.001 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.001 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:22.001 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276615+0000", 2020-01-16T00:14:22.001 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0047328450000000003 2020-01-16T00:14:22.001 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.002 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.002 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.002 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281348+0000", 2020-01-16T00:14:22.002 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3966000000000003e-05 2020-01-16T00:14:22.002 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.002 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.002 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:22.002 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281422+0000", 2020-01-16T00:14:22.003 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01082613 2020-01-16T00:14:22.003 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.003 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.003 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.003 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.292248+0000", 2020-01-16T00:14:22.003 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000148617 2020-01-16T00:14:22.003 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.003 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.004 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:22.004 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.292397+0000", 2020-01-16T00:14:22.004 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019936300000000001 2020-01-16T00:14:22.004 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.004 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.004 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:14:22.004 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.292596+0000", 2020-01-16T00:14:22.005 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011208870000000001 2020-01-16T00:14:22.005 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.005 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.005 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:22.005 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293717+0000", 2020-01-16T00:14:22.005 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00039491599999999998 2020-01-16T00:14:22.005 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.005 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.006 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:22.006 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294112+0000", 2020-01-16T00:14:22.006 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000129326 2020-01-16T00:14:22.006 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.006 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.006 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:22.006 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294241+0000", 2020-01-16T00:14:22.006 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5586000000000001e-05 2020-01-16T00:14:22.007 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.007 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.007 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:22.007 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294277+0000", 2020-01-16T00:14:22.007 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7893999999999997e-05 2020-01-16T00:14:22.007 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.007 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.008 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:22.008 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294355+0000", 2020-01-16T00:14:22.008 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017760560000000002 2020-01-16T00:14:22.008 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.008 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:22.008 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.008 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.008 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.009 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3126 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e44)", 2020-01-16T00:14:22.009 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.276634+0000", 2020-01-16T00:14:22.009 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 56.591767568999998, 2020-01-16T00:14:22.009 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017942583000000002, 2020-01-16T00:14:22.009 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:22.009 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:22.009 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:22.009 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:22.010 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:22.010 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3126 2020-01-16T00:14:22.010 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.010 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:22.010 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.010 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:22.010 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276634+0000", 2020-01-16T00:14:22.010 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:22.011 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.011 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.011 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:22.011 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276634+0000", 2020-01-16T00:14:22.011 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:22.011 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.011 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.011 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:22.012 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276633+0000", 2020-01-16T00:14:22.012 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0769999999999999e-06 2020-01-16T00:14:22.012 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.012 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.012 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:22.012 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276636+0000", 2020-01-16T00:14:22.012 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5390000000000001e-06 2020-01-16T00:14:22.013 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.013 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.013 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:22.013 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276641+0000", 2020-01-16T00:14:22.013 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9969999999999999e-06 2020-01-16T00:14:22.013 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.013 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.013 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:22.014 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276644+0000", 2020-01-16T00:14:22.014 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0048246970000000002 2020-01-16T00:14:22.014 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.014 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.014 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.014 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281469+0000", 2020-01-16T00:14:22.014 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8723999999999999e-05 2020-01-16T00:14:22.014 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.015 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.015 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:22.015 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281567+0000", 2020-01-16T00:14:22.015 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011633447 2020-01-16T00:14:22.015 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.015 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.015 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.015 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293201+0000", 2020-01-16T00:14:22.016 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6796000000000002e-05 2020-01-16T00:14:22.016 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.016 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.016 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:22.016 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293268+0000", 2020-01-16T00:14:22.016 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001125247 2020-01-16T00:14:22.016 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.016 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.017 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.017 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294393+0000", 2020-01-16T00:14:22.017 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.6082000000000006e-05 2020-01-16T00:14:22.017 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.017 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.017 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:22.017 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294489+0000", 2020-01-16T00:14:22.017 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7515000000000005e-05 2020-01-16T00:14:22.018 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.018 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.018 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:22.018 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294576+0000", 2020-01-16T00:14:22.018 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017942583000000002 2020-01-16T00:14:22.018 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.018 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:22.019 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.019 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.019 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.019 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3204 3.9 3:93dfe6f1:::smithi12313934-40:head [setxattr _header (56) in=7b,truncate 2554730] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:14:22.019 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.568544+0000", 2020-01-16T00:14:22.019 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 52.299857349, 2020-01-16T00:14:22.019 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.026260867, 2020-01-16T00:14:22.019 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:22.020 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:22.020 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:22.020 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:22.020 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:22.020 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3204 2020-01-16T00:14:22.020 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.020 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:22.020 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.021 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:22.021 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568544+0000", 2020-01-16T00:14:22.021 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:22.021 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.021 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.021 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:22.021 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568544+0000", 2020-01-16T00:14:22.021 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:22.022 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.022 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.022 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:22.022 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568543+0000", 2020-01-16T00:14:22.022 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0810000000000001e-06 2020-01-16T00:14:22.022 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.022 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.022 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:22.023 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568548+0000", 2020-01-16T00:14:22.023 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6950000000000002e-06 2020-01-16T00:14:22.023 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.023 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.023 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:22.023 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568552+0000", 2020-01-16T00:14:22.023 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.875e-06 2020-01-16T00:14:22.024 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.024 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.024 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:22.024 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568555+0000", 2020-01-16T00:14:22.024 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0033130849999999999 2020-01-16T00:14:22.024 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.024 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.024 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.025 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.571868+0000", 2020-01-16T00:14:22.025 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0026014609999999998 2020-01-16T00:14:22.025 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.025 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.025 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:22.025 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.574469+0000", 2020-01-16T00:14:22.025 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010261381999999999 2020-01-16T00:14:22.025 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.026 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.026 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.026 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.584731+0000", 2020-01-16T00:14:22.026 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019704 2020-01-16T00:14:22.026 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.026 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.026 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:22.026 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.584928+0000", 2020-01-16T00:14:22.027 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000280869 2020-01-16T00:14:22.027 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.027 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.027 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:14:22.027 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.585209+0000", 2020-01-16T00:14:22.029 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002273859 2020-01-16T00:14:22.029 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.030 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.030 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:22.030 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.587483+0000", 2020-01-16T00:14:22.030 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3356000000000001e-05 2020-01-16T00:14:22.030 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.030 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.030 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:22.030 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.587506+0000", 2020-01-16T00:14:22.031 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0071864939999999999 2020-01-16T00:14:22.031 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.031 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.031 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:22.031 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594692+0000", 2020-01-16T00:14:22.031 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5475000000000001e-05 2020-01-16T00:14:22.031 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.032 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.032 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:22.032 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594738+0000", 2020-01-16T00:14:22.032 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7070000000000007e-05 2020-01-16T00:14:22.032 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.032 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.032 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:22.032 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594805+0000", 2020-01-16T00:14:22.033 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.026260867 2020-01-16T00:14:22.033 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.033 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:22.033 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.033 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.033 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.033 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3205 3.9 3:93dfe6f1:::smithi12313934-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e45)", 2020-01-16T00:14:22.033 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.568599+0000", 2020-01-16T00:14:22.034 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 52.299802413999998, 2020-01-16T00:14:22.034 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.026406882, 2020-01-16T00:14:22.034 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:22.034 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:22.034 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:22.034 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:22.034 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:22.034 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3205 2020-01-16T00:14:22.035 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.035 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:22.035 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.035 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:22.035 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568599+0000", 2020-01-16T00:14:22.035 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:22.035 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.035 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.036 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:22.036 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568599+0000", 2020-01-16T00:14:22.036 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:22.036 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.036 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.036 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:22.036 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568598+0000", 2020-01-16T00:14:22.036 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3939999999999998e-06 2020-01-16T00:14:22.037 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.037 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.037 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:22.037 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568603+0000", 2020-01-16T00:14:22.037 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9391000000000001e-05 2020-01-16T00:14:22.037 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.037 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.038 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:22.038 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568622+0000", 2020-01-16T00:14:22.038 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9599999999999997e-06 2020-01-16T00:14:22.038 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.038 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.038 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:22.038 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568628+0000", 2020-01-16T00:14:22.038 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0058679700000000001 2020-01-16T00:14:22.039 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.039 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.039 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.039 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.574496+0000", 2020-01-16T00:14:22.039 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5528999999999998e-05 2020-01-16T00:14:22.039 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.039 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.040 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:22.040 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.574532+0000", 2020-01-16T00:14:22.040 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011986749 2020-01-16T00:14:22.040 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.040 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.040 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.040 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.586518+0000", 2020-01-16T00:14:22.040 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016522600000000001 2020-01-16T00:14:22.041 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.041 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.041 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:22.041 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.586684+0000", 2020-01-16T00:14:22.041 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0081693890000000009 2020-01-16T00:14:22.041 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.041 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.041 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.042 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594853+0000", 2020-01-16T00:14:22.042 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7082000000000004e-05 2020-01-16T00:14:22.042 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.042 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.042 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:22.042 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594940+0000", 2020-01-16T00:14:22.042 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5789000000000003e-05 2020-01-16T00:14:22.043 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.043 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.043 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:22.043 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.595006+0000", 2020-01-16T00:14:22.043 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.026406882 2020-01-16T00:14:22.043 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.043 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:22.043 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.044 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.044 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.044 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3219 3.3 3:c478bfe8:::smithi12313934-10:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:22.044 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.580957+0000", 2020-01-16T00:14:22.044 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 52.287444897999997, 2020-01-16T00:14:22.044 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016807971000000001, 2020-01-16T00:14:22.044 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:22.044 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:22.045 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:22.045 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:22.045 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:22.045 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3219 2020-01-16T00:14:22.045 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.045 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:22.045 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.046 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:22.046 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580957+0000", 2020-01-16T00:14:22.046 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:22.046 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.046 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.046 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:22.046 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580957+0000", 2020-01-16T00:14:22.046 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:22.047 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.047 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.047 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:22.047 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580955+0000", 2020-01-16T00:14:22.047 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5120000000000002e-06 2020-01-16T00:14:22.047 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.047 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.047 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:22.048 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580963+0000", 2020-01-16T00:14:22.048 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6799999999999993e-06 2020-01-16T00:14:22.048 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.048 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.048 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:22.048 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580971+0000", 2020-01-16T00:14:22.048 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3109999999999996e-06 2020-01-16T00:14:22.048 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.049 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.049 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:22.049 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580976+0000", 2020-01-16T00:14:22.049 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014306146 2020-01-16T00:14:22.049 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.049 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.049 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.050 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.595282+0000", 2020-01-16T00:14:22.050 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.1883000000000001e-05 2020-01-16T00:14:22.050 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.050 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.050 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:22.050 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.595374+0000", 2020-01-16T00:14:22.050 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023904989999999999 2020-01-16T00:14:22.050 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.051 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.051 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:22.051 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.597765+0000", 2020-01-16T00:14:22.051 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016807971000000001 2020-01-16T00:14:22.051 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.051 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:22.051 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.051 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.052 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.052 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3220 3.3 3:c478bfe8:::smithi12313934-10:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:22.052 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.580982+0000", 2020-01-16T00:14:22.052 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 52.287419849999999, 2020-01-16T00:14:22.052 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017920330000000002, 2020-01-16T00:14:22.052 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:22.052 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:22.052 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:22.053 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:22.053 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:22.053 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3220 2020-01-16T00:14:22.053 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.053 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:22.053 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.053 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:22.054 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580982+0000", 2020-01-16T00:14:22.054 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:22.054 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.054 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.054 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:22.054 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580982+0000", 2020-01-16T00:14:22.054 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:22.054 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.055 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.055 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:22.055 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580981+0000", 2020-01-16T00:14:22.055 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0380000000000001e-06 2020-01-16T00:14:22.055 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.055 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.055 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:22.055 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580985+0000", 2020-01-16T00:14:22.056 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8929000000000001e-05 2020-01-16T00:14:22.056 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.056 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.056 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:22.056 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581004+0000", 2020-01-16T00:14:22.056 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7339999999999999e-06 2020-01-16T00:14:22.056 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.056 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.057 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:22.057 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581009+0000", 2020-01-16T00:14:22.057 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01679812 2020-01-16T00:14:22.057 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.057 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.057 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.057 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.597807+0000", 2020-01-16T00:14:22.058 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013333400000000001 2020-01-16T00:14:22.058 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.058 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.058 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:22.058 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.597940+0000", 2020-01-16T00:14:22.058 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000961899 2020-01-16T00:14:22.058 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.058 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.059 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:22.059 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.598902+0000", 2020-01-16T00:14:22.059 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017920330000000002 2020-01-16T00:14:22.059 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.059 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:22.059 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.059 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.059 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.060 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3221 3.3 3:c478bfe8:::smithi12313934-10:head [read 0~2403467,checksum] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:22.060 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.581014+0000", 2020-01-16T00:14:22.060 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 52.287387500999998, 2020-01-16T00:14:22.060 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018784551999999999, 2020-01-16T00:14:22.060 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:22.060 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:22.061 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:22.061 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:22.061 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:22.061 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3221 2020-01-16T00:14:22.061 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.061 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:22.061 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.061 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:22.062 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581014+0000", 2020-01-16T00:14:22.062 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:22.062 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.062 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.062 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:22.062 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581014+0000", 2020-01-16T00:14:22.062 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:22.063 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.063 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.063 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:22.063 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581013+0000", 2020-01-16T00:14:22.063 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9390000000000002e-06 2020-01-16T00:14:22.063 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.063 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.063 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:22.064 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581019+0000", 2020-01-16T00:14:22.064 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1131000000000001e-05 2020-01-16T00:14:22.064 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.064 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.064 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:22.064 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581030+0000", 2020-01-16T00:14:22.064 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.55e-06 2020-01-16T00:14:22.064 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.065 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.065 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:22.065 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581037+0000", 2020-01-16T00:14:22.065 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017890321000000001 2020-01-16T00:14:22.065 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.065 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.065 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.065 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.598927+0000", 2020-01-16T00:14:22.066 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000105999 2020-01-16T00:14:22.066 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.066 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.066 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:22.066 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.599033+0000", 2020-01-16T00:14:22.066 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00076542400000000001 2020-01-16T00:14:22.066 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.066 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.067 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:22.067 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.599798+0000", 2020-01-16T00:14:22.067 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018784551999999999 2020-01-16T00:14:22.067 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.067 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:22.067 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.067 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.067 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.068 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4152 3.9 3:93dfe6f1:::smithi12313934-40:head [setxattr _header (56) in=7b,truncate 2806868] snapc 0=[] ondisk+write+known_if_redirected e53)", 2020-01-16T00:14:22.068 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:53.810986+0000", 2020-01-16T00:14:22.068 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 28.057415192000001, 2020-01-16T00:14:22.068 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016830279, 2020-01-16T00:14:22.068 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:22.068 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:22.068 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:22.069 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:22.069 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:22.069 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4152 2020-01-16T00:14:22.069 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.069 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:22.069 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.069 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:22.069 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.810986+0000", 2020-01-16T00:14:22.070 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:22.070 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.070 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.070 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:22.070 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.810986+0000", 2020-01-16T00:14:22.070 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:22.070 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.070 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.071 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:22.071 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.810985+0000", 2020-01-16T00:14:22.071 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.7529999999999996e-06 2020-01-16T00:14:22.071 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.071 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.071 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:22.071 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.810995+0000", 2020-01-16T00:14:22.071 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7907999999999999e-05 2020-01-16T00:14:22.072 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.072 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.072 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:22.072 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811013+0000", 2020-01-16T00:14:22.072 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5829999999999997e-06 2020-01-16T00:14:22.072 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.072 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.072 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:22.073 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811018+0000", 2020-01-16T00:14:22.073 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0029165229999999999 2020-01-16T00:14:22.073 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.073 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.073 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.073 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.813935+0000", 2020-01-16T00:14:22.073 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2583000000000001e-05 2020-01-16T00:14:22.073 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.074 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.074 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:22.074 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.813988+0000", 2020-01-16T00:14:22.074 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010581012000000001 2020-01-16T00:14:22.074 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.074 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.074 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.075 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.824569+0000", 2020-01-16T00:14:22.075 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013191900000000001 2020-01-16T00:14:22.075 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.075 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.075 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:22.075 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.824700+0000", 2020-01-16T00:14:22.075 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022298000000000001 2020-01-16T00:14:22.075 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.076 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.076 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:14:22.076 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.824923+0000", 2020-01-16T00:14:22.076 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010694260000000001 2020-01-16T00:14:22.076 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.076 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.076 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:22.076 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.825993+0000", 2020-01-16T00:14:22.077 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000116939 2020-01-16T00:14:22.077 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.077 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.077 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:22.077 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.826110+0000", 2020-01-16T00:14:22.077 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0015606400000000001 2020-01-16T00:14:22.077 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.077 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.078 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:22.078 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.827670+0000", 2020-01-16T00:14:22.078 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7318999999999997e-05 2020-01-16T00:14:22.078 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.078 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.078 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:22.078 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.827738+0000", 2020-01-16T00:14:22.078 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.873e-05 2020-01-16T00:14:22.079 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.079 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.079 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:22.079 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.827817+0000", 2020-01-16T00:14:22.079 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016830279 2020-01-16T00:14:22.079 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.079 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:22.079 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.080 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.080 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.080 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4153 3.9 3:93dfe6f1:::smithi12313934-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e53)", 2020-01-16T00:14:22.080 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:53.811034+0000", 2020-01-16T00:14:22.080 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 28.057367104000001, 2020-01-16T00:14:22.080 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017095406, 2020-01-16T00:14:22.080 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:22.080 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:22.081 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:22.081 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:22.081 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:22.081 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4153 2020-01-16T00:14:22.081 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.081 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:22.081 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.081 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:22.082 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811034+0000", 2020-01-16T00:14:22.082 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:22.082 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.082 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.082 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:22.082 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811034+0000", 2020-01-16T00:14:22.082 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:22.082 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.083 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.083 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:22.083 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811034+0000", 2020-01-16T00:14:22.083 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9060000000000001e-06 2020-01-16T00:14:22.083 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.083 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.083 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:22.083 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811038+0000", 2020-01-16T00:14:22.084 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7710000000000001e-06 2020-01-16T00:14:22.084 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.084 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.084 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:22.084 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811045+0000", 2020-01-16T00:14:22.084 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5189999999999998e-06 2020-01-16T00:14:22.084 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.084 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.085 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:22.085 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811052+0000", 2020-01-16T00:14:22.085 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002953991 2020-01-16T00:14:22.085 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.085 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.085 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.085 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.814006+0000", 2020-01-16T00:14:22.086 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5726999999999999e-05 2020-01-16T00:14:22.086 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.086 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.086 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:22.086 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.814041+0000", 2020-01-16T00:14:22.086 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011650031999999999 2020-01-16T00:14:22.086 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.086 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.087 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.087 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.825691+0000", 2020-01-16T00:14:22.087 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0625999999999998e-05 2020-01-16T00:14:22.087 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.087 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.087 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:22.087 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.825762+0000", 2020-01-16T00:14:22.087 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0020982370000000002 2020-01-16T00:14:22.088 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.088 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.088 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.088 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.827860+0000", 2020-01-16T00:14:22.088 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012736299999999999 2020-01-16T00:14:22.088 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.088 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.088 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:22.089 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.827988+0000", 2020-01-16T00:14:22.089 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000142015 2020-01-16T00:14:22.089 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.089 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.089 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:22.089 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.828130+0000", 2020-01-16T00:14:22.089 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017095406 2020-01-16T00:14:22.089 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.090 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:22.090 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.090 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.090 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:22.090 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:22.092 INFO:tasks.rados.rados.0.smithi123.stdout:1846: expect (ObjNum 667 snap 0 seq_num 667) 2020-01-16T00:14:22.092 INFO:teuthology.orchestra.run.smithi123.stderr:nodeep-scrub is set 2020-01-16T00:14:22.092 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-01-16T00:14:22.093 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:22.088+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:22.192 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:22.190+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:22.247 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:22.247 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:14:22.247 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:14:22.248 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:22.257 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-01-16T00:14:22.293 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:22.290+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:22.370 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:22.370 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:14:22.370 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:14:22.370 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:14:22.370 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:22.379 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-01-16T00:14:22.395 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:22.394+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:22.399 INFO:tasks.rados.rados.0.smithi123.stdout:1849: finishing write tid 1 to smithi12313934-37 2020-01-16T00:14:22.400 INFO:tasks.rados.rados.0.smithi123.stdout:1849: finishing write tid 2 to smithi12313934-37 2020-01-16T00:14:22.400 INFO:tasks.rados.rados.0.smithi123.stdout:1850: expect (ObjNum 676 snap 0 seq_num 676) 2020-01-16T00:14:22.487 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:22.487 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:14:22.487 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:14:22.487 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:14:22.487 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.488 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.5.13:36 2.0 2:20e40cec:::smithi12313934-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 113152~1699840] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:14:22.488 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:14:22.488 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 104.654044781, 2020-01-16T00:14:22.488 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011736636, 2020-01-16T00:14:22.488 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:22.488 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:22.488 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:22.489 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.489 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:22.489 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:14:22.489 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:22.489 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.489 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.489 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:22.490 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:14:22.490 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999952 2020-01-16T00:14:22.490 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.490 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.490 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:22.490 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831934+0000", 2020-01-16T00:14:22.490 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001034325 2020-01-16T00:14:22.491 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.491 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.491 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:22.491 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.832968+0000", 2020-01-16T00:14:22.491 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.506e-05 2020-01-16T00:14:22.491 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.491 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.491 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:22.492 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833004+0000", 2020-01-16T00:14:22.492 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3406e-05 2020-01-16T00:14:22.492 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.492 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.492 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:22.492 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833017+0000", 2020-01-16T00:14:22.492 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015477000000000001 2020-01-16T00:14:22.493 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.493 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.493 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.493 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833172+0000", 2020-01-16T00:14:22.493 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033750700000000002 2020-01-16T00:14:22.493 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.493 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.494 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:22.494 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833509+0000", 2020-01-16T00:14:22.494 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00055861400000000003 2020-01-16T00:14:22.494 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.494 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.494 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:14:22.495 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.834068+0000", 2020-01-16T00:14:22.495 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.007474545 2020-01-16T00:14:22.495 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.495 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.495 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:22.495 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.841542+0000", 2020-01-16T00:14:22.495 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00056319499999999995 2020-01-16T00:14:22.496 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.496 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.496 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:22.496 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.842106+0000", 2020-01-16T00:14:22.496 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001371017 2020-01-16T00:14:22.496 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.496 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.497 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:22.497 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843477+0000", 2020-01-16T00:14:22.497 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000123938 2020-01-16T00:14:22.497 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.497 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.497 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:22.497 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843601+0000", 2020-01-16T00:14:22.497 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5012000000000006e-05 2020-01-16T00:14:22.498 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.498 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.498 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:22.498 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843676+0000", 2020-01-16T00:14:22.498 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011736636 2020-01-16T00:14:22.498 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.498 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:22.499 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.499 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.499 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.499 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1276 3.d 3:b1521a3e:::smithi12313934-25:head [read 0~3882384,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:14:22.499 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:14:22.499 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 99.411285211000006, 2020-01-16T00:14:22.499 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015839157999999999, 2020-01-16T00:14:22.499 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:22.500 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:22.500 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:22.500 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:22.500 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:22.500 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1276 2020-01-16T00:14:22.500 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.500 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:22.501 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.501 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:22.501 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:14:22.501 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:22.501 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.501 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.501 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:22.502 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:14:22.502 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:22.502 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.502 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.502 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:22.502 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:14:22.502 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5429999999999999e-06 2020-01-16T00:14:22.502 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.503 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.503 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:22.503 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074704+0000", 2020-01-16T00:14:22.503 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0839999999999992e-06 2020-01-16T00:14:22.503 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.503 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.504 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:22.504 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074712+0000", 2020-01-16T00:14:22.504 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.173e-06 2020-01-16T00:14:22.504 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.504 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.504 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:22.504 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074718+0000", 2020-01-16T00:14:22.504 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0083034700000000003 2020-01-16T00:14:22.505 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.505 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.505 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.505 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.083022+0000", 2020-01-16T00:14:22.505 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028456400000000001 2020-01-16T00:14:22.505 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.505 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.506 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:22.506 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.083306+0000", 2020-01-16T00:14:22.506 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0072312640000000003 2020-01-16T00:14:22.506 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.506 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.506 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:22.506 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090538+0000", 2020-01-16T00:14:22.506 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015839157999999999 2020-01-16T00:14:22.507 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.507 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:22.507 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.507 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.507 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.507 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1277 3.d 3:b1521a3e:::smithi12313934-25:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:14:22.507 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:14:22.508 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 99.411241621000002, 2020-01-16T00:14:22.508 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016646459999999998, 2020-01-16T00:14:22.508 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:22.508 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:22.508 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:22.508 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:22.508 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:22.509 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1277 2020-01-16T00:14:22.509 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.509 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:22.509 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.509 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:22.509 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:14:22.509 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:22.509 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.510 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.510 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:22.510 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:14:22.510 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:22.510 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.510 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.510 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:22.510 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074741+0000", 2020-01-16T00:14:22.511 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0776e-05 2020-01-16T00:14:22.511 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.511 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.511 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:22.511 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074752+0000", 2020-01-16T00:14:22.511 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7600000000000002e-06 2020-01-16T00:14:22.511 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.511 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.512 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:22.512 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074759+0000", 2020-01-16T00:14:22.512 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6840000000000001e-06 2020-01-16T00:14:22.512 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.512 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.512 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:22.512 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074765+0000", 2020-01-16T00:14:22.513 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015815216 2020-01-16T00:14:22.513 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.513 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.513 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.513 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090580+0000", 2020-01-16T00:14:22.513 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000182404 2020-01-16T00:14:22.513 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.514 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.514 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:22.514 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090763+0000", 2020-01-16T00:14:22.514 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00062579300000000003 2020-01-16T00:14:22.514 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.515 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.515 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:22.515 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091389+0000", 2020-01-16T00:14:22.515 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016646459999999998 2020-01-16T00:14:22.516 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.516 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:22.516 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.516 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.516 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.517 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1278 3.d 3:b1521a3e:::smithi12313934-25:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:14:22.517 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:14:22.517 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 99.411211907999999, 2020-01-16T00:14:22.517 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017994915, 2020-01-16T00:14:22.517 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:22.518 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:22.518 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:22.518 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:22.518 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:22.518 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1278 2020-01-16T00:14:22.519 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.519 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:22.519 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.519 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:22.519 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:14:22.519 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:22.520 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.520 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.520 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:22.520 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:14:22.520 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:22.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.521 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.521 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:22.521 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074771+0000", 2020-01-16T00:14:22.521 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4569999999999998e-06 2020-01-16T00:14:22.522 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.522 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.522 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:22.522 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074775+0000", 2020-01-16T00:14:22.522 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0330000000000003e-06 2020-01-16T00:14:22.522 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.523 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.523 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:22.523 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074782+0000", 2020-01-16T00:14:22.523 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9470000000000002e-06 2020-01-16T00:14:22.523 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.523 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.523 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:22.524 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074787+0000", 2020-01-16T00:14:22.524 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01666134 2020-01-16T00:14:22.524 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.524 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.524 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.524 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091449+0000", 2020-01-16T00:14:22.524 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000247731 2020-01-16T00:14:22.524 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.525 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.525 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:22.525 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091696+0000", 2020-01-16T00:14:22.525 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010703170000000001 2020-01-16T00:14:22.525 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.525 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.525 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:22.525 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.092767+0000", 2020-01-16T00:14:22.526 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017994915 2020-01-16T00:14:22.526 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.526 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:22.527 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.527 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.527 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.527 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2260 3.d 3:b76659c4:::smithi12313934-47:head [write 601750~535546] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:14:22.527 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:06.050577+0000", 2020-01-16T00:14:22.527 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 76.435406592000007, 2020-01-16T00:14:22.527 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012415381, 2020-01-16T00:14:22.528 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:22.528 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:22.528 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:22.528 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:22.528 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:22.528 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2260 2020-01-16T00:14:22.529 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.529 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:22.529 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.529 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:22.529 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.050577+0000", 2020-01-16T00:14:22.529 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:22.529 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.530 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.530 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:22.530 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.050577+0000", 2020-01-16T00:14:22.530 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:22.530 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.530 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.530 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:22.530 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.050574+0000", 2020-01-16T00:14:22.531 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001523487 2020-01-16T00:14:22.531 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.531 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.531 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:22.531 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052097+0000", 2020-01-16T00:14:22.531 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3697e-05 2020-01-16T00:14:22.531 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.531 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.532 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:22.532 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052121+0000", 2020-01-16T00:14:22.532 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0070000000000003e-06 2020-01-16T00:14:22.532 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.532 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.532 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:22.532 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052130+0000", 2020-01-16T00:14:22.532 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4559000000000003e-05 2020-01-16T00:14:22.533 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.533 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.533 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.533 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052194+0000", 2020-01-16T00:14:22.533 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000165878 2020-01-16T00:14:22.533 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.533 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.534 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:22.534 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052360+0000", 2020-01-16T00:14:22.534 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018825499999999999 2020-01-16T00:14:22.534 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.534 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.534 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:14:22.534 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052549+0000", 2020-01-16T00:14:22.535 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002761782 2020-01-16T00:14:22.535 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.535 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.535 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:22.535 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.055310+0000", 2020-01-16T00:14:22.535 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00060385500000000004 2020-01-16T00:14:22.535 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.536 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.536 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:22.536 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.055914+0000", 2020-01-16T00:14:22.536 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0069733809999999999 2020-01-16T00:14:22.536 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.536 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.537 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:22.537 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.062888+0000", 2020-01-16T00:14:22.537 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2884999999999999e-05 2020-01-16T00:14:22.537 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.537 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.537 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:22.537 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.062940+0000", 2020-01-16T00:14:22.537 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1996999999999997e-05 2020-01-16T00:14:22.538 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.538 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.538 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:22.538 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.062992+0000", 2020-01-16T00:14:22.538 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012415381 2020-01-16T00:14:22.538 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.538 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:22.538 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.539 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.539 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.539 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2295 3.d 3:b1521a3e:::smithi12313934-25:head [write 1763838~505140] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:14:22.539 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:06.660258+0000", 2020-01-16T00:14:22.539 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 75.825725364999997, 2020-01-16T00:14:22.539 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011681408000000001, 2020-01-16T00:14:22.539 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:22.540 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:22.540 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:22.540 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:22.540 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:22.540 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2295 2020-01-16T00:14:22.540 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.541 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:22.541 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.541 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:22.541 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660258+0000", 2020-01-16T00:14:22.541 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:22.541 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.541 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.541 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:22.542 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660258+0000", 2020-01-16T00:14:22.542 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:22.542 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.542 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.542 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:22.542 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660256+0000", 2020-01-16T00:14:22.542 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00066501699999999995 2020-01-16T00:14:22.543 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.543 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.543 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:22.543 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660921+0000", 2020-01-16T00:14:22.543 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7339000000000001e-05 2020-01-16T00:14:22.543 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.543 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.543 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:22.544 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660938+0000", 2020-01-16T00:14:22.544 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0219999999999993e-06 2020-01-16T00:14:22.544 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.544 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.544 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:22.544 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660947+0000", 2020-01-16T00:14:22.544 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00094237200000000002 2020-01-16T00:14:22.545 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.545 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.545 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.545 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.661890+0000", 2020-01-16T00:14:22.545 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000117649 2020-01-16T00:14:22.545 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.545 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.545 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:22.546 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662007+0000", 2020-01-16T00:14:22.546 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00060662599999999998 2020-01-16T00:14:22.546 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.546 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.546 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.546 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662614+0000", 2020-01-16T00:14:22.546 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028424899999999998 2020-01-16T00:14:22.546 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.547 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.547 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:22.547 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662898+0000", 2020-01-16T00:14:22.547 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00054904599999999997 2020-01-16T00:14:22.547 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.547 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.548 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:14:22.548 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.663447+0000", 2020-01-16T00:14:22.548 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0051479990000000003 2020-01-16T00:14:22.548 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.548 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.549 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:22.549 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.668595+0000", 2020-01-16T00:14:22.549 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00080881799999999999 2020-01-16T00:14:22.549 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.549 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.550 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:22.550 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.669404+0000", 2020-01-16T00:14:22.550 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0024495290000000002 2020-01-16T00:14:22.550 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.551 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.551 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:22.551 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671854+0000", 2020-01-16T00:14:22.551 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6635000000000003e-05 2020-01-16T00:14:22.551 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.552 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.552 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:22.552 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671900+0000", 2020-01-16T00:14:22.552 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9567999999999998e-05 2020-01-16T00:14:22.553 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.553 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.553 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:22.553 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671940+0000", 2020-01-16T00:14:22.554 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011681408000000001 2020-01-16T00:14:22.554 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.554 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:22.554 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.554 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.554 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.555 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.3.13:109 2.0 2:11fa79bc:::smithi12313934-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 331264~198656] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e42)", 2020-01-16T00:14:22.555 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:21.840075+0000", 2020-01-16T00:14:22.555 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 60.645909068000002, 2020-01-16T00:14:22.555 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011558677999999999, 2020-01-16T00:14:22.555 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:22.555 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:22.555 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:22.555 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.556 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:22.556 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840075+0000", 2020-01-16T00:14:22.556 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:22.556 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.556 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.556 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:22.557 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840075+0000", 2020-01-16T00:14:22.557 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:22.557 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.557 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.557 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:22.558 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840073+0000", 2020-01-16T00:14:22.558 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033576299999999999 2020-01-16T00:14:22.558 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.558 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.558 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:22.558 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840409+0000", 2020-01-16T00:14:22.559 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2082999999999998e-05 2020-01-16T00:14:22.559 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.559 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.559 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:22.559 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840431+0000", 2020-01-16T00:14:22.560 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4410000000000003e-06 2020-01-16T00:14:22.560 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.560 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.560 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:22.561 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840436+0000", 2020-01-16T00:14:22.561 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000101016 2020-01-16T00:14:22.561 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.561 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.561 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.562 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840537+0000", 2020-01-16T00:14:22.562 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000117639 2020-01-16T00:14:22.562 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.562 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.562 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:22.563 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840654+0000", 2020-01-16T00:14:22.563 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000196162 2020-01-16T00:14:22.563 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.563 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.563 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:14:22.563 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840850+0000", 2020-01-16T00:14:22.564 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003319092 2020-01-16T00:14:22.564 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.564 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.564 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:22.564 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.844169+0000", 2020-01-16T00:14:22.564 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011349229999999999 2020-01-16T00:14:22.564 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.565 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.565 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:22.565 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.845304+0000", 2020-01-16T00:14:22.565 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0062517670000000001 2020-01-16T00:14:22.565 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.565 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.565 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:22.565 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851556+0000", 2020-01-16T00:14:22.566 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4138000000000003e-05 2020-01-16T00:14:22.566 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.566 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.566 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:22.566 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851600+0000", 2020-01-16T00:14:22.566 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2993000000000001e-05 2020-01-16T00:14:22.567 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.567 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.567 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:22.567 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851633+0000", 2020-01-16T00:14:22.567 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011558677999999999 2020-01-16T00:14:22.568 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.568 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:22.568 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.568 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.568 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.569 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.3.13:110 2.0 2:11fa79bc:::smithi12313934-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 752640~650240] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e42)", 2020-01-16T00:14:22.569 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:21.840457+0000", 2020-01-16T00:14:22.569 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 60.645526775999997, 2020-01-16T00:14:22.570 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011329271, 2020-01-16T00:14:22.570 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:22.570 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:22.570 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:22.570 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.570 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:22.570 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840457+0000", 2020-01-16T00:14:22.571 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:22.571 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.571 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.571 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:22.571 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840457+0000", 2020-01-16T00:14:22.571 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:22.571 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.571 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.572 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:22.572 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840456+0000", 2020-01-16T00:14:22.572 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00062607899999999998 2020-01-16T00:14:22.572 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.572 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.572 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:22.572 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841082+0000", 2020-01-16T00:14:22.572 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1239999999999999e-06 2020-01-16T00:14:22.573 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.573 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.573 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:22.573 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841090+0000", 2020-01-16T00:14:22.573 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7129999999999999e-06 2020-01-16T00:14:22.573 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.573 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.574 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:22.574 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841093+0000", 2020-01-16T00:14:22.574 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00075997800000000002 2020-01-16T00:14:22.574 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.574 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.574 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.574 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841853+0000", 2020-01-16T00:14:22.574 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0690000000000003e-05 2020-01-16T00:14:22.575 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.575 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.575 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:22.575 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841924+0000", 2020-01-16T00:14:22.575 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000115753 2020-01-16T00:14:22.575 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.575 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.575 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:14:22.576 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.842040+0000", 2020-01-16T00:14:22.576 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0021358890000000002 2020-01-16T00:14:22.576 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.576 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.576 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:22.576 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.844176+0000", 2020-01-16T00:14:22.576 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0042541870000000004 2020-01-16T00:14:22.576 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.577 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.577 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:22.577 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.848430+0000", 2020-01-16T00:14:22.577 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0032476929999999998 2020-01-16T00:14:22.577 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.577 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.577 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:22.577 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851677+0000", 2020-01-16T00:14:22.578 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0370000000000001e-05 2020-01-16T00:14:22.578 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.578 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.578 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:22.578 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851728+0000", 2020-01-16T00:14:22.578 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.834e-05 2020-01-16T00:14:22.578 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.579 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.579 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:22.579 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851786+0000", 2020-01-16T00:14:22.579 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011329271 2020-01-16T00:14:22.579 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.579 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:22.579 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.579 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.580 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.580 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2987 3.d 3:b3dd5d45:::smithi12313934-16:head [write 467327~633480] snapc 0=[] ondisk+write+known_if_redirected e42)", 2020-01-16T00:14:22.580 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:22.257904+0000", 2020-01-16T00:14:22.580 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 60.228079481999998, 2020-01-16T00:14:22.580 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012187450000000001, 2020-01-16T00:14:22.580 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:22.580 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:22.580 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:22.581 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:22.581 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:22.581 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2987 2020-01-16T00:14:22.581 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.581 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:22.581 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.581 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:22.581 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.257904+0000", 2020-01-16T00:14:22.582 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:22.582 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.582 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.582 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:22.582 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.257904+0000", 2020-01-16T00:14:22.582 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999933 2020-01-16T00:14:22.582 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.583 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.583 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:22.583 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.257898+0000", 2020-01-16T00:14:22.583 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00097099800000000004 2020-01-16T00:14:22.583 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.583 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.583 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:22.583 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.258869+0000", 2020-01-16T00:14:22.584 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7970000000000003e-05 2020-01-16T00:14:22.584 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.584 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.584 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:22.584 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.258917+0000", 2020-01-16T00:14:22.584 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5982999999999999e-05 2020-01-16T00:14:22.584 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.584 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.585 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:22.585 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.258933+0000", 2020-01-16T00:14:22.585 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00026239999999999998 2020-01-16T00:14:22.585 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.585 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.585 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.585 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.259195+0000", 2020-01-16T00:14:22.586 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024907400000000001 2020-01-16T00:14:22.586 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.586 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.586 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:22.586 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.259444+0000", 2020-01-16T00:14:22.586 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00034824999999999998 2020-01-16T00:14:22.586 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.586 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.587 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:22.587 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.259792+0000", 2020-01-16T00:14:22.587 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003251538 2020-01-16T00:14:22.587 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.588 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.588 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:22.588 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.263044+0000", 2020-01-16T00:14:22.588 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0068502880000000004 2020-01-16T00:14:22.589 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.589 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.589 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:22.589 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.269894+0000", 2020-01-16T00:14:22.589 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0945000000000006e-05 2020-01-16T00:14:22.589 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.590 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.590 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:22.590 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.269965+0000", 2020-01-16T00:14:22.590 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012667399999999999 2020-01-16T00:14:22.590 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.590 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.591 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:22.591 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.270092+0000", 2020-01-16T00:14:22.591 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012187450000000001 2020-01-16T00:14:22.591 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.592 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:22.592 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.592 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.592 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.593 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3273 3.d 3:b76659c4:::smithi12313934-47:head [read 0~3647152,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:22.593 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.728775+0000", 2020-01-16T00:14:22.593 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 51.757208927999997, 2020-01-16T00:14:22.593 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012922950000000001, 2020-01-16T00:14:22.594 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:22.594 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:22.594 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:22.594 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:22.595 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:22.595 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3273 2020-01-16T00:14:22.595 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.596 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:22.596 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.596 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:22.596 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728775+0000", 2020-01-16T00:14:22.596 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:22.597 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.597 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.597 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:22.597 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728775+0000", 2020-01-16T00:14:22.598 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:22.598 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.598 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.598 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:22.600 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728771+0000", 2020-01-16T00:14:22.600 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.5009000000000001e-05 2020-01-16T00:14:22.600 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.601 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.601 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:22.602 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728796+0000", 2020-01-16T00:14:22.602 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2357999999999998e-05 2020-01-16T00:14:22.602 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.602 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.603 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:22.603 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728819+0000", 2020-01-16T00:14:22.603 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.4800000000000007e-06 2020-01-16T00:14:22.603 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.603 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.604 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:22.604 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728828+0000", 2020-01-16T00:14:22.604 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000124824 2020-01-16T00:14:22.604 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.605 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.605 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.605 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728953+0000", 2020-01-16T00:14:22.605 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00044904899999999998 2020-01-16T00:14:22.606 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.606 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.606 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:22.606 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729402+0000", 2020-01-16T00:14:22.607 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012295706999999999 2020-01-16T00:14:22.607 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.607 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.607 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:22.607 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.741698+0000", 2020-01-16T00:14:22.608 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012922950000000001 2020-01-16T00:14:22.608 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.608 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:22.608 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.609 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.609 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.609 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3275 3.d 3:b76659c4:::smithi12313934-47:head [read 0~3647152,checksum] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:22.609 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.728933+0000", 2020-01-16T00:14:22.609 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 51.757050655999997, 2020-01-16T00:14:22.610 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01204874, 2020-01-16T00:14:22.610 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:22.610 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:22.610 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:22.611 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:22.611 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:22.611 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3275 2020-01-16T00:14:22.611 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.611 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:22.612 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.612 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:22.612 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728933+0000", 2020-01-16T00:14:22.612 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:22.613 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.613 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.613 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:22.613 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728933+0000", 2020-01-16T00:14:22.615 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:22.615 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.615 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.615 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:22.615 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728932+0000", 2020-01-16T00:14:22.616 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3569999999999998e-06 2020-01-16T00:14:22.616 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.616 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.616 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:22.616 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728940+0000", 2020-01-16T00:14:22.617 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0457e-05 2020-01-16T00:14:22.617 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.617 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.617 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:22.617 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728951+0000", 2020-01-16T00:14:22.618 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6799999999999993e-06 2020-01-16T00:14:22.618 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.618 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.618 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:22.618 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728959+0000", 2020-01-16T00:14:22.618 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010076307 2020-01-16T00:14:22.618 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.619 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.619 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.619 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.739035+0000", 2020-01-16T00:14:22.619 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024455400000000002 2020-01-16T00:14:22.619 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.619 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.619 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:22.620 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.739279+0000", 2020-01-16T00:14:22.620 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0017023660000000001 2020-01-16T00:14:22.620 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.620 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.620 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:22.620 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.740982+0000", 2020-01-16T00:14:22.620 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01204874 2020-01-16T00:14:22.620 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.621 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:22.621 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.621 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.621 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.621 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3285 3.d 3:bd9fc72f:::smithi12313934-5:head [read 0~3764801,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:22.621 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.729068+0000", 2020-01-16T00:14:22.621 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 51.756915864, 2020-01-16T00:14:22.622 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018537010999999999, 2020-01-16T00:14:22.622 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:22.622 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:22.622 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:22.623 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:22.623 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:22.623 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3285 2020-01-16T00:14:22.623 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.623 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:22.623 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.624 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:22.624 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729068+0000", 2020-01-16T00:14:22.624 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:22.624 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.624 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.624 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:22.624 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729068+0000", 2020-01-16T00:14:22.625 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:22.625 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.625 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.625 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:22.625 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729067+0000", 2020-01-16T00:14:22.625 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3339999999999996e-06 2020-01-16T00:14:22.626 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.626 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.626 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:22.626 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729075+0000", 2020-01-16T00:14:22.626 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4187e-05 2020-01-16T00:14:22.626 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.626 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.626 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:22.627 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729089+0000", 2020-01-16T00:14:22.627 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5600000000000004e-06 2020-01-16T00:14:22.627 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.627 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.627 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:22.627 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729093+0000", 2020-01-16T00:14:22.627 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011944309 2020-01-16T00:14:22.628 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.628 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.628 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.628 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.741038+0000", 2020-01-16T00:14:22.628 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016625799999999999 2020-01-16T00:14:22.628 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.628 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.628 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:22.629 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.741204+0000", 2020-01-16T00:14:22.629 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0064010120000000002 2020-01-16T00:14:22.629 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.629 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.629 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:22.629 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.747605+0000", 2020-01-16T00:14:22.629 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018537010999999999 2020-01-16T00:14:22.630 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.630 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:22.630 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.630 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.630 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.630 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3286 3.d 3:bd9fc72f:::smithi12313934-5:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:22.630 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.729099+0000", 2020-01-16T00:14:22.631 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 51.756885193999999, 2020-01-16T00:14:22.631 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018983897999999999, 2020-01-16T00:14:22.631 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:22.631 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:22.631 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:22.631 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:22.632 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:22.632 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3286 2020-01-16T00:14:22.632 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.632 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:22.632 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.632 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:22.632 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729099+0000", 2020-01-16T00:14:22.633 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:22.633 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.633 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.633 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:22.633 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729099+0000", 2020-01-16T00:14:22.633 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:22.633 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.634 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.634 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:22.634 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729098+0000", 2020-01-16T00:14:22.634 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8500000000000004e-06 2020-01-16T00:14:22.634 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.634 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.634 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:22.635 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729102+0000", 2020-01-16T00:14:22.635 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7329999999999997e-06 2020-01-16T00:14:22.635 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.635 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.635 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:22.635 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729108+0000", 2020-01-16T00:14:22.635 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.764e-06 2020-01-16T00:14:22.636 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.636 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.636 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:22.636 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729118+0000", 2020-01-16T00:14:22.636 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018524286000000001 2020-01-16T00:14:22.636 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.636 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.637 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.637 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.747642+0000", 2020-01-16T00:14:22.637 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9487999999999993e-05 2020-01-16T00:14:22.637 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.637 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.637 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:22.637 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.747742+0000", 2020-01-16T00:14:22.638 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000340429 2020-01-16T00:14:22.638 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.638 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.638 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:22.638 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.748082+0000", 2020-01-16T00:14:22.638 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018983897999999999 2020-01-16T00:14:22.638 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.638 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:22.639 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.639 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.639 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.639 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3287 3.d 3:bd9fc72f:::smithi12313934-5:head [read 0~3764801,checksum] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:22.639 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.729124+0000", 2020-01-16T00:14:22.639 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 51.756860054999997, 2020-01-16T00:14:22.639 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020552323000000001, 2020-01-16T00:14:22.640 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:22.640 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:22.640 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:22.640 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:22.640 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:22.640 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3287 2020-01-16T00:14:22.640 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.641 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:22.641 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.641 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:22.641 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729124+0000", 2020-01-16T00:14:22.641 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:22.641 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.642 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.642 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:22.642 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729124+0000", 2020-01-16T00:14:22.642 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:22.642 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.642 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.642 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:22.642 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729123+0000", 2020-01-16T00:14:22.643 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8239999999999996e-06 2020-01-16T00:14:22.643 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.643 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.643 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:22.643 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729129+0000", 2020-01-16T00:14:22.643 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6909999999999997e-06 2020-01-16T00:14:22.643 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.644 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.644 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:22.644 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729136+0000", 2020-01-16T00:14:22.644 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3329999999999997e-06 2020-01-16T00:14:22.644 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.644 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.644 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:22.645 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729145+0000", 2020-01-16T00:14:22.645 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018965065999999999 2020-01-16T00:14:22.645 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.645 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.645 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.645 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.748110+0000", 2020-01-16T00:14:22.645 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015781700000000001 2020-01-16T00:14:22.646 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.646 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.646 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:22.646 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.748268+0000", 2020-01-16T00:14:22.646 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014083800000000001 2020-01-16T00:14:22.646 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.646 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.647 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:22.647 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.749676+0000", 2020-01-16T00:14:22.647 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020552323000000001 2020-01-16T00:14:22.647 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.647 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:22.647 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.647 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.647 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.648 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3409 3.d 3:b3dd5d45:::smithi12313934-16:head [write 591712~786646,stat] snapc 0=[] ondisk+write+known_if_redirected e47)", 2020-01-16T00:14:22.648 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:34.100681+0000", 2020-01-16T00:14:22.648 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 48.385302914999997, 2020-01-16T00:14:22.648 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012248446, 2020-01-16T00:14:22.648 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:22.648 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:22.648 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:22.649 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:22.649 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:22.649 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3409 2020-01-16T00:14:22.649 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.649 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:22.649 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.649 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:22.650 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.100681+0000", 2020-01-16T00:14:22.650 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:22.650 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.650 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.650 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:22.650 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.100681+0000", 2020-01-16T00:14:22.651 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:14:22.651 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.651 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.651 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:22.651 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.100677+0000", 2020-01-16T00:14:22.651 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000998828 2020-01-16T00:14:22.651 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.652 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.652 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:22.652 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101676+0000", 2020-01-16T00:14:22.652 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5846999999999997e-05 2020-01-16T00:14:22.652 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.652 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.652 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:22.653 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101712+0000", 2020-01-16T00:14:22.653 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.364e-05 2020-01-16T00:14:22.653 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.653 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.653 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:22.653 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101725+0000", 2020-01-16T00:14:22.653 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015902700000000001 2020-01-16T00:14:22.654 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.654 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.654 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.654 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101884+0000", 2020-01-16T00:14:22.654 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0486e-05 2020-01-16T00:14:22.654 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.654 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.654 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op must wait for map", 2020-01-16T00:14:22.655 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101905+0000", 2020-01-16T00:14:22.655 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0038869009999999999 2020-01-16T00:14:22.655 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.655 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.655 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.655 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.105792+0000", 2020-01-16T00:14:22.656 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030748800000000002 2020-01-16T00:14:22.656 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.656 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.656 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:22.656 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.106099+0000", 2020-01-16T00:14:22.656 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042069299999999998 2020-01-16T00:14:22.657 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.657 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.657 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:22.657 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.106520+0000", 2020-01-16T00:14:22.657 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0052864560000000001 2020-01-16T00:14:22.657 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.657 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.658 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:22.658 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.111806+0000", 2020-01-16T00:14:22.658 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00080835800000000001 2020-01-16T00:14:22.658 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.659 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.659 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:22.659 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.112615+0000", 2020-01-16T00:14:22.659 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1490999999999997e-05 2020-01-16T00:14:22.659 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.660 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.660 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:22.660 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.112686+0000", 2020-01-16T00:14:22.660 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000243174 2020-01-16T00:14:22.660 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.660 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.660 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:22.661 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.112929+0000", 2020-01-16T00:14:22.661 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012248446 2020-01-16T00:14:22.661 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.661 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:22.661 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.661 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.662 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.662 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.0.13:31 2.0 2:1afbc8ac:::smithi12313934-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 4096~737792] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e50)", 2020-01-16T00:14:22.662 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:44.061109+0000", 2020-01-16T00:14:22.662 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 38.424874690999999, 2020-01-16T00:14:22.662 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012186723, 2020-01-16T00:14:22.662 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:22.662 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:22.663 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:22.663 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.663 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:22.663 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.061109+0000", 2020-01-16T00:14:22.663 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:22.663 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.663 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.664 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:22.664 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.061109+0000", 2020-01-16T00:14:22.664 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:22.664 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.664 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.664 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:22.664 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.061106+0000", 2020-01-16T00:14:22.665 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012349870000000001 2020-01-16T00:14:22.665 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.665 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.665 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:22.665 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062341+0000", 2020-01-16T00:14:22.665 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0235999999999996e-05 2020-01-16T00:14:22.665 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.666 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.666 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:22.666 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062411+0000", 2020-01-16T00:14:22.666 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7078000000000001e-05 2020-01-16T00:14:22.666 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.666 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.666 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:22.667 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062428+0000", 2020-01-16T00:14:22.667 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2158999999999996e-05 2020-01-16T00:14:22.667 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.667 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.667 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.667 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062520+0000", 2020-01-16T00:14:22.667 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011203499999999999 2020-01-16T00:14:22.668 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.668 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.668 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:22.668 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062632+0000", 2020-01-16T00:14:22.668 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017238400000000001 2020-01-16T00:14:22.669 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.669 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.669 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:14:22.669 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062805+0000", 2020-01-16T00:14:22.669 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010264989 2020-01-16T00:14:22.669 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.669 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.670 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:22.670 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073070+0000", 2020-01-16T00:14:22.670 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0533000000000001e-05 2020-01-16T00:14:22.670 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.670 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.670 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:22.670 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073090+0000", 2020-01-16T00:14:22.671 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2309000000000002e-05 2020-01-16T00:14:22.671 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.671 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.671 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:22.671 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073163+0000", 2020-01-16T00:14:22.671 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3035999999999997e-05 2020-01-16T00:14:22.671 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.671 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.672 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:22.672 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073236+0000", 2020-01-16T00:14:22.672 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0056999999999998e-05 2020-01-16T00:14:22.672 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.672 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.672 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:22.673 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073296+0000", 2020-01-16T00:14:22.673 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012186723 2020-01-16T00:14:22.673 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.673 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:22.673 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.674 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.674 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.674 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.0.13:32 2.0 2:1afbc8ac:::smithi12313934-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 916992~596480] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e50)", 2020-01-16T00:14:22.674 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:44.062464+0000", 2020-01-16T00:14:22.674 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 38.423519906000003, 2020-01-16T00:14:22.674 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011316405999999999, 2020-01-16T00:14:22.674 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:22.675 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:22.675 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:22.675 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.675 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:22.675 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062464+0000", 2020-01-16T00:14:22.675 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:22.675 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.676 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.676 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:22.676 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062464+0000", 2020-01-16T00:14:22.676 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:22.676 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.676 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.676 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:22.676 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062462+0000", 2020-01-16T00:14:22.677 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00073330900000000002 2020-01-16T00:14:22.677 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.677 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.677 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:22.677 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063196+0000", 2020-01-16T00:14:22.678 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5330000000000001e-05 2020-01-16T00:14:22.678 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.678 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.678 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:22.678 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063211+0000", 2020-01-16T00:14:22.678 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5840000000000001e-06 2020-01-16T00:14:22.678 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.678 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.679 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:22.679 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063221+0000", 2020-01-16T00:14:22.679 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001786915 2020-01-16T00:14:22.679 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.679 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.679 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.679 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.065007+0000", 2020-01-16T00:14:22.680 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00021770000000000001 2020-01-16T00:14:22.680 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.680 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.680 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:22.680 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.065225+0000", 2020-01-16T00:14:22.680 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00032623 2020-01-16T00:14:22.680 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.681 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.681 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:14:22.681 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.065551+0000", 2020-01-16T00:14:22.681 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0078227499999999998 2020-01-16T00:14:22.681 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.681 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.681 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:22.682 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073374+0000", 2020-01-16T00:14:22.682 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.717e-05 2020-01-16T00:14:22.682 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.682 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.682 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:22.682 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073391+0000", 2020-01-16T00:14:22.682 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011493999999999999 2020-01-16T00:14:22.683 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.683 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.683 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:22.683 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073506+0000", 2020-01-16T00:14:22.683 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000159281 2020-01-16T00:14:22.683 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.684 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.684 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:22.684 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073666+0000", 2020-01-16T00:14:22.684 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000114675 2020-01-16T00:14:22.684 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.684 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.684 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:22.685 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073780+0000", 2020-01-16T00:14:22.685 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011316405999999999 2020-01-16T00:14:22.685 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.685 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:22.685 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.685 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.685 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.685 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.0.13:33 2.0 2:1afbc8ac:::smithi12313934-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 1541120~378368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e50)", 2020-01-16T00:14:22.686 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:44.063236+0000", 2020-01-16T00:14:22.686 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 38.422748026999997, 2020-01-16T00:14:22.686 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015290962999999999, 2020-01-16T00:14:22.686 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:22.686 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:22.686 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:22.686 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.686 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:22.687 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063236+0000", 2020-01-16T00:14:22.687 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:22.687 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.687 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.687 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:22.687 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063236+0000", 2020-01-16T00:14:22.687 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:22.688 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.688 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.688 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:22.688 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063234+0000", 2020-01-16T00:14:22.688 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000500702 2020-01-16T00:14:22.688 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.688 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.689 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:22.689 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063735+0000", 2020-01-16T00:14:22.689 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4734e-05 2020-01-16T00:14:22.689 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.689 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.689 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:22.689 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063750+0000", 2020-01-16T00:14:22.690 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.382e-05 2020-01-16T00:14:22.690 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.690 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.690 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:22.690 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063764+0000", 2020-01-16T00:14:22.690 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0059030799999999998 2020-01-16T00:14:22.690 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.691 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.691 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.691 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.069667+0000", 2020-01-16T00:14:22.691 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3961999999999994e-05 2020-01-16T00:14:22.691 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.691 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.691 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:22.692 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.069741+0000", 2020-01-16T00:14:22.692 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000114931 2020-01-16T00:14:22.692 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.692 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.692 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:14:22.692 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.069856+0000", 2020-01-16T00:14:22.692 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0039948270000000003 2020-01-16T00:14:22.693 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.693 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.693 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:22.693 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073850+0000", 2020-01-16T00:14:22.693 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0016459140000000001 2020-01-16T00:14:22.693 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.693 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.693 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:22.694 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.075496+0000", 2020-01-16T00:14:22.694 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002957997 2020-01-16T00:14:22.694 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.694 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.694 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:22.694 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.078454+0000", 2020-01-16T00:14:22.694 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9190999999999998e-05 2020-01-16T00:14:22.695 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.695 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.695 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:22.695 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.078493+0000", 2020-01-16T00:14:22.695 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3176999999999998e-05 2020-01-16T00:14:22.696 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.696 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.696 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:22.696 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.078527+0000", 2020-01-16T00:14:22.696 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015290962999999999 2020-01-16T00:14:22.696 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.696 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:22.696 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.697 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.697 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.697 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4641 3.d 3:b1521a3e:::smithi12313934-25:head [read 0~3596797,checksum] snapc 0=[] ondisk+read+known_if_redirected e57)", 2020-01-16T00:14:22.697 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:04.086675+0000", 2020-01-16T00:14:22.697 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 18.399308971, 2020-01-16T00:14:22.697 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011822397, 2020-01-16T00:14:22.698 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:22.698 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:22.698 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:22.698 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:22.698 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:22.698 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4641 2020-01-16T00:14:22.698 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.698 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:22.699 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.699 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:22.699 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.086675+0000", 2020-01-16T00:14:22.699 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:22.699 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.699 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.699 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:22.700 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.086675+0000", 2020-01-16T00:14:22.700 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:22.700 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.700 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.700 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:22.700 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.086674+0000", 2020-01-16T00:14:22.700 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0809999999999998e-06 2020-01-16T00:14:22.700 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.701 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.701 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:22.701 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.086678+0000", 2020-01-16T00:14:22.701 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4889999999999997e-06 2020-01-16T00:14:22.701 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.701 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.702 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:22.702 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.086682+0000", 2020-01-16T00:14:22.702 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9620000000000002e-06 2020-01-16T00:14:22.702 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.702 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.702 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:22.702 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.086685+0000", 2020-01-16T00:14:22.702 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010215452 2020-01-16T00:14:22.703 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.703 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.703 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.703 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.096900+0000", 2020-01-16T00:14:22.703 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00021274299999999999 2020-01-16T00:14:22.703 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.703 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.704 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:22.704 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.097113+0000", 2020-01-16T00:14:22.704 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0013839480000000001 2020-01-16T00:14:22.704 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.704 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.704 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:22.704 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.098497+0000", 2020-01-16T00:14:22.705 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011822397 2020-01-16T00:14:22.705 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.705 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:22.705 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.705 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.705 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.705 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4737 3.d 3:b76659c4:::smithi12313934-47:head [read 0~3243850,checksum] snapc 0=[] ondisk+read+known_if_redirected e58)", 2020-01-16T00:14:22.706 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:08.134998+0000", 2020-01-16T00:14:22.706 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 14.350986097, 2020-01-16T00:14:22.706 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011528518, 2020-01-16T00:14:22.706 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:22.706 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:22.706 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:22.706 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:22.707 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:22.707 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4737 2020-01-16T00:14:22.707 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.707 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:22.707 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.707 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:22.708 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:08.134998+0000", 2020-01-16T00:14:22.708 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:22.708 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.708 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.708 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:22.708 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:08.134998+0000", 2020-01-16T00:14:22.708 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:22.709 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.709 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.709 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:22.709 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:08.134997+0000", 2020-01-16T00:14:22.709 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0459999999999999e-06 2020-01-16T00:14:22.709 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.710 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.710 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:22.710 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:08.135001+0000", 2020-01-16T00:14:22.710 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2469999999999999e-06 2020-01-16T00:14:22.710 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.710 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.710 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:22.711 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:08.135006+0000", 2020-01-16T00:14:22.711 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.065e-06 2020-01-16T00:14:22.711 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.711 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.711 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:22.712 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:08.135010+0000", 2020-01-16T00:14:22.712 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0095733439999999993 2020-01-16T00:14:22.712 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.713 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.713 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:22.713 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:08.144584+0000", 2020-01-16T00:14:22.714 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000272623 2020-01-16T00:14:22.714 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.714 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.714 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:22.714 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:08.144856+0000", 2020-01-16T00:14:22.714 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001669769 2020-01-16T00:14:22.714 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:22.715 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:22.715 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:22.715 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:08.146526+0000", 2020-01-16T00:14:22.715 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011528518 2020-01-16T00:14:22.715 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.715 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:22.715 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.716 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:22.716 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:22.716 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:22.719 INFO:tasks.rados.rados.0.smithi123.stdout:1849: finishing write tid 3 to smithi12313934-37 2020-01-16T00:14:22.720 INFO:tasks.rados.rados.0.smithi123.stdout:1849: finishing write tid 5 to smithi12313934-37 2020-01-16T00:14:22.720 INFO:tasks.rados.rados.0.smithi123.stdout:1849: finishing write tid 6 to smithi12313934-37 2020-01-16T00:14:22.720 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 37 v 348 (ObjNum 681 snap 0 seq_num 681) dirty exists 2020-01-16T00:14:22.720 INFO:tasks.rados.rados.0.smithi123.stdout:1849: left oid 37 (ObjNum 681 snap 0 seq_num 681) 2020-01-16T00:14:22.720 INFO:tasks.rados.rados.0.smithi123.stdout:1851: finishing write tid 1 to smithi12313934-25 2020-01-16T00:14:22.720 INFO:tasks.rados.rados.0.smithi123.stdout:1851: finishing write tid 2 to smithi12313934-25 2020-01-16T00:14:22.720 INFO:tasks.rados.rados.0.smithi123.stdout:1851: finishing write tid 3 to smithi12313934-25 2020-01-16T00:14:22.721 INFO:tasks.rados.rados.0.smithi123.stdout:1851: finishing write tid 5 to smithi12313934-25 2020-01-16T00:14:22.721 INFO:tasks.rados.rados.0.smithi123.stdout:1851: finishing write tid 6 to smithi12313934-25 2020-01-16T00:14:22.721 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 25 v 284 (ObjNum 682 snap 0 seq_num 682) dirty exists 2020-01-16T00:14:22.721 INFO:tasks.rados.rados.0.smithi123.stdout:1851: left oid 25 (ObjNum 682 snap 0 seq_num 682) 2020-01-16T00:14:22.721 INFO:tasks.rados.rados.0.smithi123.stdout:1852: finishing write tid 1 to smithi12313934-43 2020-01-16T00:14:22.721 INFO:tasks.rados.rados.0.smithi123.stdout:1852: finishing write tid 2 to smithi12313934-43 2020-01-16T00:14:22.722 INFO:tasks.rados.rados.0.smithi123.stdout:1852: finishing write tid 3 to smithi12313934-43 2020-01-16T00:14:22.722 INFO:tasks.rados.rados.0.smithi123.stdout:1844: done (15 left) 2020-01-16T00:14:22.722 INFO:tasks.rados.rados.0.smithi123.stdout:1845: done (14 left) 2020-01-16T00:14:22.722 INFO:tasks.rados.rados.0.smithi123.stdout:1846: done (13 left) 2020-01-16T00:14:22.722 INFO:tasks.rados.rados.0.smithi123.stdout:1847: done (12 left) 2020-01-16T00:14:22.722 INFO:tasks.rados.rados.0.smithi123.stdout:1848: done (11 left) 2020-01-16T00:14:22.722 INFO:tasks.rados.rados.0.smithi123.stdout:1849: done (10 left) 2020-01-16T00:14:22.722 INFO:tasks.rados.rados.0.smithi123.stdout:1850: done (9 left) 2020-01-16T00:14:22.723 INFO:tasks.rados.rados.0.smithi123.stdout:1851: done (8 left) 2020-01-16T00:14:22.723 INFO:tasks.rados.rados.0.smithi123.stdout:1860: write (excl) oid 46 current snap is 0 2020-01-16T00:14:22.723 INFO:tasks.rados.rados.0.smithi123.stdout:1860: seq_num 687 ranges {746409=567833,1901600=544241,3065949=480460} 2020-01-16T00:14:22.723 INFO:tasks.rados.rados.0.smithi123.stdout:1860: writing smithi12313934-46 from 746409 to 1314242 tid 1 2020-01-16T00:14:22.723 INFO:tasks.rados.rados.0.smithi123.stdout:1860: writing smithi12313934-46 from 1901600 to 2445841 tid 2 2020-01-16T00:14:22.723 INFO:tasks.rados.rados.0.smithi123.stdout:1860: writing smithi12313934-46 from 3065949 to 3546409 tid 3 2020-01-16T00:14:22.723 INFO:tasks.rados.rados.0.smithi123.stdout:1861: read oid 26 snap -1 2020-01-16T00:14:22.724 INFO:tasks.rados.rados.0.smithi123.stdout:1861: expect (ObjNum 556 snap 0 seq_num 556) 2020-01-16T00:14:22.724 INFO:tasks.rados.rados.0.smithi123.stdout:1862: delete oid 47 current snap is 0 2020-01-16T00:14:22.724 INFO:tasks.rados.rados.0.smithi123.stdout:1852: finishing write tid 5 to smithi12313934-43 2020-01-16T00:14:22.724 INFO:tasks.rados.rados.0.smithi123.stdout:1852: finishing write tid 6 to smithi12313934-43 2020-01-16T00:14:22.724 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 43 v 296 (ObjNum 683 snap 0 seq_num 683) dirty exists 2020-01-16T00:14:22.724 INFO:tasks.rados.rados.0.smithi123.stdout:1852: left oid 43 (ObjNum 683 snap 0 seq_num 683) 2020-01-16T00:14:22.724 INFO:tasks.rados.rados.0.smithi123.stdout:1853: expect (ObjNum 671 snap 0 seq_num 671) 2020-01-16T00:14:22.725 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:22.714+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:22.725 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-01-16T00:14:22.818 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:22.814+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:22.843 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:22.844 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:14:22.844 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:14:22.844 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:22.856 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-01-16T00:14:22.868 INFO:tasks.rados.rados.0.smithi123.stdout:1854: expect (ObjNum 656 snap 0 seq_num 656) 2020-01-16T00:14:22.919 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:22.916+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:22.966 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:22.966 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:14:22.967 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:14:22.967 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:14:22.967 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:22.975 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-01-16T00:14:23.019 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:23.016+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:23.039 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-01-16T00:14:23.050 INFO:tasks.rados.rados.0.smithi123.stdout:1855: finishing write tid 1 to smithi12313934-6 2020-01-16T00:14:23.051 INFO:tasks.rados.rados.0.smithi123.stdout:1855: finishing write tid 2 to smithi12313934-6 2020-01-16T00:14:23.051 INFO:tasks.rados.rados.0.smithi123.stdout:1855: finishing write tid 3 to smithi12313934-6 2020-01-16T00:14:23.051 INFO:tasks.rados.rados.0.smithi123.stdout:1855: finishing write tid 5 to smithi12313934-6 2020-01-16T00:14:23.051 INFO:tasks.rados.rados.0.smithi123.stdout:1855: finishing write tid 6 to smithi12313934-6 2020-01-16T00:14:23.051 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 6 v 198 (ObjNum 684 snap 0 seq_num 684) dirty exists 2020-01-16T00:14:23.051 INFO:tasks.rados.rados.0.smithi123.stdout:1855: left oid 6 (ObjNum 684 snap 0 seq_num 684) 2020-01-16T00:14:23.052 INFO:tasks.rados.rados.0.smithi123.stdout:1856: finishing write tid 1 to smithi12313934-7 2020-01-16T00:14:23.052 INFO:tasks.rados.rados.0.smithi123.stdout:1856: finishing write tid 2 to smithi12313934-7 2020-01-16T00:14:23.052 INFO:tasks.rados.rados.0.smithi123.stdout:1852: done (10 left) 2020-01-16T00:14:23.053 INFO:tasks.rados.rados.0.smithi123.stdout:1853: done (9 left) 2020-01-16T00:14:23.053 INFO:tasks.rados.rados.0.smithi123.stdout:1854: done (8 left) 2020-01-16T00:14:23.053 INFO:tasks.rados.rados.0.smithi123.stdout:1855: done (7 left) 2020-01-16T00:14:23.054 INFO:tasks.rados.rados.0.smithi123.stdout:1862: done (6 left) 2020-01-16T00:14:23.054 INFO:tasks.rados.rados.0.smithi123.stdout:1863: write (excl) oid 20 current snap is 0 2020-01-16T00:14:23.054 INFO:tasks.rados.rados.0.smithi123.stdout:1863: seq_num 689 ranges {780023=718331,1902211=547910,3004442=575581} 2020-01-16T00:14:23.068 INFO:tasks.rados.rados.0.smithi123.stdout:1863: writing smithi12313934-20 from 780023 to 1498354 tid 1 2020-01-16T00:14:23.074 INFO:tasks.rados.rados.0.smithi123.stdout:1863: writing smithi12313934-20 from 1902211 to 2450121 tid 2 2020-01-16T00:14:23.079 INFO:tasks.rados.rados.0.smithi123.stdout:1863: writing smithi12313934-20 from 3004442 to 3580023 tid 3 2020-01-16T00:14:23.080 INFO:tasks.rados.rados.0.smithi123.stdout:1864: write (excl) oid 5 current snap is 0 2020-01-16T00:14:23.080 INFO:tasks.rados.rados.0.smithi123.stdout:1864: seq_num 690 ranges {796830=793580,2360693=691568} 2020-01-16T00:14:23.088 INFO:tasks.rados.rados.0.smithi123.stdout:1864: writing smithi12313934-5 from 796830 to 1590410 tid 1 2020-01-16T00:14:23.097 INFO:tasks.rados.rados.0.smithi123.stdout:1864: writing smithi12313934-5 from 2360693 to 3052261 tid 2 2020-01-16T00:14:23.097 INFO:tasks.rados.rados.0.smithi123.stdout:1865: write (excl) oid 33 current snap is 0 2020-01-16T00:14:23.097 INFO:tasks.rados.rados.0.smithi123.stdout:1865: seq_num 691 ranges {413637=468829,1619175=435226,2542935=758255} 2020-01-16T00:14:23.101 INFO:tasks.rados.rados.0.smithi123.stdout:1865: writing smithi12313934-33 from 413637 to 882466 tid 1 2020-01-16T00:14:23.108 INFO:tasks.rados.rados.0.smithi123.stdout:1865: writing smithi12313934-33 from 1619175 to 2054401 tid 2 2020-01-16T00:14:23.116 INFO:tasks.rados.rados.0.smithi123.stdout:1865: writing smithi12313934-33 from 2542935 to 3301190 tid 3 2020-01-16T00:14:23.116 INFO:tasks.rados.rados.0.smithi123.stdout:1866: read oid 4 snap -1 2020-01-16T00:14:23.116 INFO:tasks.rados.rados.0.smithi123.stdout:1866: expect deleted 2020-01-16T00:14:23.117 INFO:tasks.rados.rados.0.smithi123.stdout:1856: finishing write tid 3 to smithi12313934-7 2020-01-16T00:14:23.117 INFO:tasks.rados.rados.0.smithi123.stdout:1867: read oid 28 snap -1 2020-01-16T00:14:23.117 INFO:tasks.rados.rados.0.smithi123.stdout:1867: expect (ObjNum 652 snap 0 seq_num 652) 2020-01-16T00:14:23.117 INFO:tasks.rados.rados.0.smithi123.stdout:1868: write (excl) oid 15 current snap is 0 2020-01-16T00:14:23.117 INFO:tasks.rados.rados.0.smithi123.stdout:1868: seq_num 692 ranges {430444=660431,1677657=695237,2970358=569527} 2020-01-16T00:14:23.120 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:23.118+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:23.123 INFO:tasks.rados.rados.0.smithi123.stdout:1868: writing smithi12313934-15 from 430444 to 1090875 tid 1 2020-01-16T00:14:23.129 INFO:tasks.rados.rados.0.smithi123.stdout:1868: writing smithi12313934-15 from 1677657 to 2372894 tid 2 2020-01-16T00:14:23.136 INFO:tasks.rados.rados.0.smithi123.stdout:1868: writing smithi12313934-15 from 2970358 to 3539885 tid 3 2020-01-16T00:14:23.137 INFO:tasks.rados.rados.0.smithi123.stdout:1869: write (excl) oid 37 current snap is 0 2020-01-16T00:14:23.137 INFO:tasks.rados.rados.0.smithi123.stdout:1869: seq_num 693 ranges {447251=735680,1736139=438895,2597781=780799} 2020-01-16T00:14:23.139 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:23.139 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:14:23.139 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:14:23.139 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:14:23.139 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.140 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1657 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-01-16T00:14:23.140 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.718495+0000", 2020-01-16T00:14:23.140 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 90.416215488999995, 2020-01-16T00:14:23.140 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014299630000000001, 2020-01-16T00:14:23.140 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:23.140 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:23.140 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:23.141 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:23.141 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:23.141 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1657 2020-01-16T00:14:23.141 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.141 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:23.141 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.141 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:23.142 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718495+0000", 2020-01-16T00:14:23.142 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:23.142 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.142 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.142 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:23.142 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718495+0000", 2020-01-16T00:14:23.142 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:23.142 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.143 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.143 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:23.143 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718494+0000", 2020-01-16T00:14:23.143 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8860000000000003e-06 2020-01-16T00:14:23.143 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.143 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.143 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:23.143 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718499+0000", 2020-01-16T00:14:23.144 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9786000000000002e-05 2020-01-16T00:14:23.144 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.144 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.144 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:23.144 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718539+0000", 2020-01-16T00:14:23.144 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1869999999999999e-05 2020-01-16T00:14:23.144 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.144 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.145 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:23.145 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718561+0000", 2020-01-16T00:14:23.145 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000172206 2020-01-16T00:14:23.145 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.145 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.145 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.145 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718733+0000", 2020-01-16T00:14:23.146 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0662999999999998e-05 2020-01-16T00:14:23.146 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.146 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.146 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:23.146 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718793+0000", 2020-01-16T00:14:23.146 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.006217958 2020-01-16T00:14:23.147 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.147 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.147 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.147 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.725011+0000", 2020-01-16T00:14:23.147 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1625999999999996e-05 2020-01-16T00:14:23.147 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.147 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.147 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:23.148 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.725073+0000", 2020-01-16T00:14:23.148 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0063923119999999998 2020-01-16T00:14:23.148 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.148 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.148 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.148 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731465+0000", 2020-01-16T00:14:23.148 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2534000000000005e-05 2020-01-16T00:14:23.149 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.149 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.149 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:23.149 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731558+0000", 2020-01-16T00:14:23.149 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010238809999999999 2020-01-16T00:14:23.149 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.149 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.149 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.150 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732582+0000", 2020-01-16T00:14:23.150 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000100918 2020-01-16T00:14:23.150 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.150 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.150 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:23.150 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732683+0000", 2020-01-16T00:14:23.150 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000112064 2020-01-16T00:14:23.150 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.151 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.151 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:23.151 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732795+0000", 2020-01-16T00:14:23.151 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014299630000000001 2020-01-16T00:14:23.151 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.151 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:23.151 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.151 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.152 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.152 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2791 3.a 3:54f2c117:::smithi12313934-41:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e41)", 2020-01-16T00:14:23.152 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:17.213167+0000", 2020-01-16T00:14:23.152 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 65.921543795999995, 2020-01-16T00:14:23.152 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016024037000000001, 2020-01-16T00:14:23.152 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:23.152 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:23.152 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:23.153 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:23.153 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:23.153 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2791 2020-01-16T00:14:23.153 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.153 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:23.153 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.153 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:23.153 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213167+0000", 2020-01-16T00:14:23.154 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:23.154 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.154 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.154 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:23.154 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213167+0000", 2020-01-16T00:14:23.154 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:23.154 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.154 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.155 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:23.155 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213166+0000", 2020-01-16T00:14:23.155 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6030000000000004e-06 2020-01-16T00:14:23.155 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.155 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.155 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:23.155 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213172+0000", 2020-01-16T00:14:23.156 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0929999999999997e-06 2020-01-16T00:14:23.156 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.156 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.156 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:23.156 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213179+0000", 2020-01-16T00:14:23.156 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4299999999999997e-06 2020-01-16T00:14:23.156 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.156 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.157 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:23.157 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213184+0000", 2020-01-16T00:14:23.157 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012106695000000001 2020-01-16T00:14:23.157 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.157 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.157 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.157 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.225291+0000", 2020-01-16T00:14:23.157 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020863699999999999 2020-01-16T00:14:23.158 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.158 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.158 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:23.158 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.225499+0000", 2020-01-16T00:14:23.158 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0036913050000000002 2020-01-16T00:14:23.158 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.158 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.158 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:23.159 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.229191+0000", 2020-01-16T00:14:23.159 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016024037000000001 2020-01-16T00:14:23.159 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.159 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:23.159 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.159 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.159 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.159 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2792 3.a 3:54f2c117:::smithi12313934-41:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e41)", 2020-01-16T00:14:23.160 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:17.213190+0000", 2020-01-16T00:14:23.160 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 65.921520322000006, 2020-01-16T00:14:23.160 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017165949, 2020-01-16T00:14:23.160 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:23.160 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:23.160 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:23.160 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:23.160 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:23.161 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2792 2020-01-16T00:14:23.161 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.161 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:23.161 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.161 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:23.161 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213190+0000", 2020-01-16T00:14:23.161 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:23.161 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.162 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.162 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:23.162 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213190+0000", 2020-01-16T00:14:23.162 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:23.162 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.162 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.162 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:23.162 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213189+0000", 2020-01-16T00:14:23.163 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2699999999999998e-06 2020-01-16T00:14:23.163 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.163 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.163 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:23.163 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213194+0000", 2020-01-16T00:14:23.163 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8600000000000004e-06 2020-01-16T00:14:23.163 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.163 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.164 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:23.164 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213201+0000", 2020-01-16T00:14:23.164 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8500000000000002e-06 2020-01-16T00:14:23.164 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.164 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.164 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:23.164 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213205+0000", 2020-01-16T00:14:23.165 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016028924999999999 2020-01-16T00:14:23.165 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.165 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.165 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.165 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.229234+0000", 2020-01-16T00:14:23.165 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019773999999999999 2020-01-16T00:14:23.165 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.166 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.166 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:23.166 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.229432+0000", 2020-01-16T00:14:23.166 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00092414400000000001 2020-01-16T00:14:23.166 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.166 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.166 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:23.167 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.230356+0000", 2020-01-16T00:14:23.167 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017165949 2020-01-16T00:14:23.167 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.167 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:23.167 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.167 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.167 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.167 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2793 3.a 3:54f2c117:::smithi12313934-41:head [read 0~3210170,checksum] snapc 0=[] ondisk+read+known_if_redirected e41)", 2020-01-16T00:14:23.168 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:17.213211+0000", 2020-01-16T00:14:23.168 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 65.921499776000005, 2020-01-16T00:14:23.168 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019069413, 2020-01-16T00:14:23.168 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:23.168 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:23.168 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:23.168 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:23.169 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:23.169 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2793 2020-01-16T00:14:23.169 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.169 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:23.169 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.169 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:23.169 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213211+0000", 2020-01-16T00:14:23.169 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:23.170 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.170 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.170 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:23.170 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213211+0000", 2020-01-16T00:14:23.170 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:23.170 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.170 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.170 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:23.171 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213210+0000", 2020-01-16T00:14:23.171 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.66e-06 2020-01-16T00:14:23.171 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.171 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.171 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:23.171 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213216+0000", 2020-01-16T00:14:23.171 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.9469999999999997e-06 2020-01-16T00:14:23.172 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.172 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.172 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:23.172 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213223+0000", 2020-01-16T00:14:23.172 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9259999999999999e-06 2020-01-16T00:14:23.172 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.172 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.172 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:23.173 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213228+0000", 2020-01-16T00:14:23.173 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017171046999999998 2020-01-16T00:14:23.173 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.174 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.174 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.174 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.230399+0000", 2020-01-16T00:14:23.174 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000189932 2020-01-16T00:14:23.174 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.174 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.174 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:23.174 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.230589+0000", 2020-01-16T00:14:23.175 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0016916539999999999 2020-01-16T00:14:23.175 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.175 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.175 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:23.175 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.232280+0000", 2020-01-16T00:14:23.175 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019069413 2020-01-16T00:14:23.175 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.175 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:23.176 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.176 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.176 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.176 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3120 3.a 3:5069665e:::smithi12313934-2:head [write 1845146~524707] snapc 0=[] ondisk+write+known_if_redirected e44)", 2020-01-16T00:14:23.176 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.262852+0000", 2020-01-16T00:14:23.176 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 57.871858592999999, 2020-01-16T00:14:23.176 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014991175000000001, 2020-01-16T00:14:23.176 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:23.177 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:23.177 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:23.177 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:23.177 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:23.177 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3120 2020-01-16T00:14:23.177 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.177 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:23.178 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.178 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:23.178 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.262852+0000", 2020-01-16T00:14:23.178 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:23.178 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.178 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.178 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:23.178 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.262852+0000", 2020-01-16T00:14:23.179 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:23.179 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.179 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.179 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:23.179 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.262849+0000", 2020-01-16T00:14:23.179 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00058469900000000005 2020-01-16T00:14:23.179 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.179 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.180 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:23.180 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263434+0000", 2020-01-16T00:14:23.180 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.6650000000000001e-05 2020-01-16T00:14:23.180 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.180 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.180 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:23.180 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263461+0000", 2020-01-16T00:14:23.180 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3478e-05 2020-01-16T00:14:23.181 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.181 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.181 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:23.181 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263474+0000", 2020-01-16T00:14:23.181 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7824000000000002e-05 2020-01-16T00:14:23.181 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.181 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.181 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.182 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263522+0000", 2020-01-16T00:14:23.182 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001365464 2020-01-16T00:14:23.182 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.182 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.182 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:23.182 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.264888+0000", 2020-01-16T00:14:23.182 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0081368250000000003 2020-01-16T00:14:23.183 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.183 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.183 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.183 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.273024+0000", 2020-01-16T00:14:23.183 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6130000000000006e-05 2020-01-16T00:14:23.183 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.183 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.184 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:23.184 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.273090+0000", 2020-01-16T00:14:23.184 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000207507 2020-01-16T00:14:23.184 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.184 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.184 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:23.184 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.273298+0000", 2020-01-16T00:14:23.184 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0035649980000000002 2020-01-16T00:14:23.185 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.185 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.185 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:23.185 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276863+0000", 2020-01-16T00:14:23.185 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.1459999999999995e-05 2020-01-16T00:14:23.185 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.185 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.185 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:23.186 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276954+0000", 2020-01-16T00:14:23.186 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00076363899999999998 2020-01-16T00:14:23.186 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.186 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.186 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:23.186 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277718+0000", 2020-01-16T00:14:23.186 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0154999999999995e-05 2020-01-16T00:14:23.186 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.187 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.187 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:23.187 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277788+0000", 2020-01-16T00:14:23.187 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4914000000000003e-05 2020-01-16T00:14:23.187 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.187 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.187 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:23.188 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277843+0000", 2020-01-16T00:14:23.188 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014991175000000001 2020-01-16T00:14:23.188 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.188 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:23.188 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.188 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.188 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.189 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3121 3.a 3:5069665e:::smithi12313934-2:head [setxattr _header (55) in=7b,truncate 2369853] snapc 0=[] ondisk+write+known_if_redirected e44)", 2020-01-16T00:14:23.189 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.263494+0000", 2020-01-16T00:14:23.189 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 57.871216304999997, 2020-01-16T00:14:23.189 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014559274000000001, 2020-01-16T00:14:23.189 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:23.189 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:23.189 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:23.190 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:23.190 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:23.190 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3121 2020-01-16T00:14:23.190 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.190 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:23.190 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.190 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:23.190 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263494+0000", 2020-01-16T00:14:23.191 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:23.191 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.191 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.191 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:23.191 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263494+0000", 2020-01-16T00:14:23.191 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:23.191 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.192 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.192 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:23.192 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263493+0000", 2020-01-16T00:14:23.192 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3359999999999995e-06 2020-01-16T00:14:23.192 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.192 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.192 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:23.192 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263501+0000", 2020-01-16T00:14:23.193 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4100000000000002e-06 2020-01-16T00:14:23.193 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.193 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.193 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:23.193 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263509+0000", 2020-01-16T00:14:23.193 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3839999999999996e-06 2020-01-16T00:14:23.193 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.193 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.194 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:23.194 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263514+0000", 2020-01-16T00:14:23.194 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001430295 2020-01-16T00:14:23.194 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.194 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.194 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.194 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.264944+0000", 2020-01-16T00:14:23.195 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9710999999999999e-05 2020-01-16T00:14:23.195 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.195 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.195 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:23.195 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.264994+0000", 2020-01-16T00:14:23.195 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010036188999999999 2020-01-16T00:14:23.195 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.196 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.196 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.196 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.275030+0000", 2020-01-16T00:14:23.196 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1743000000000007e-05 2020-01-16T00:14:23.196 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.196 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.196 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:23.197 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.275112+0000", 2020-01-16T00:14:23.197 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013036100000000001 2020-01-16T00:14:23.197 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.197 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.197 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:23.198 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.275242+0000", 2020-01-16T00:14:23.198 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0016795549999999999 2020-01-16T00:14:23.198 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.198 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.198 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:23.198 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276922+0000", 2020-01-16T00:14:23.198 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018479400000000001 2020-01-16T00:14:23.199 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.199 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.199 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:23.199 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277107+0000", 2020-01-16T00:14:23.199 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00083585300000000001 2020-01-16T00:14:23.199 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.199 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.200 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:23.200 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277943+0000", 2020-01-16T00:14:23.201 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1719999999999999e-05 2020-01-16T00:14:23.201 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.201 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.201 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:23.201 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277994+0000", 2020-01-16T00:14:23.202 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9212999999999997e-05 2020-01-16T00:14:23.202 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.202 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.202 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:23.202 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.278054+0000", 2020-01-16T00:14:23.202 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014559274000000001 2020-01-16T00:14:23.202 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.202 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:23.203 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.203 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.203 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.203 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3122 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e44)", 2020-01-16T00:14:23.203 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.263525+0000", 2020-01-16T00:14:23.203 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 57.871185117000003, 2020-01-16T00:14:23.203 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014731833999999999, 2020-01-16T00:14:23.203 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:23.204 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:23.204 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:23.204 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:23.204 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:23.204 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3122 2020-01-16T00:14:23.204 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.204 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:23.205 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.205 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:23.205 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263525+0000", 2020-01-16T00:14:23.205 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:23.205 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.205 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.205 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:23.205 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263525+0000", 2020-01-16T00:14:23.206 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:23.206 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.206 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.206 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:23.206 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263525+0000", 2020-01-16T00:14:23.207 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5000000000000001e-06 2020-01-16T00:14:23.207 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.207 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.207 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:23.207 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263529+0000", 2020-01-16T00:14:23.207 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4669999999999998e-06 2020-01-16T00:14:23.207 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.207 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.208 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:23.208 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263537+0000", 2020-01-16T00:14:23.208 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0819999999999998e-06 2020-01-16T00:14:23.208 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.208 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.208 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:23.208 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263542+0000", 2020-01-16T00:14:23.208 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0015024960000000001 2020-01-16T00:14:23.209 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.209 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.209 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.209 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.265044+0000", 2020-01-16T00:14:23.209 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6650999999999994e-05 2020-01-16T00:14:23.209 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.209 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.210 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:23.210 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.265131+0000", 2020-01-16T00:14:23.210 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010495310000000001 2020-01-16T00:14:23.210 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.210 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.211 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.211 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.275626+0000", 2020-01-16T00:14:23.211 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9536000000000004e-05 2020-01-16T00:14:23.211 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.211 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.211 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:23.211 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.275726+0000", 2020-01-16T00:14:23.211 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023719489999999999 2020-01-16T00:14:23.212 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.212 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.212 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.212 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.278098+0000", 2020-01-16T00:14:23.212 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3718000000000001e-05 2020-01-16T00:14:23.212 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.212 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.212 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:23.213 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.278171+0000", 2020-01-16T00:14:23.213 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5859999999999994e-05 2020-01-16T00:14:23.213 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.213 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.213 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:23.213 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.278257+0000", 2020-01-16T00:14:23.213 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014731833999999999 2020-01-16T00:14:23.214 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.214 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:23.214 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.214 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.214 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.214 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3627 3.a 3:5069665e:::smithi12313934-2:head [write 1749580~611149] snapc 0=[] ondisk+write+known_if_redirected e48)", 2020-01-16T00:14:23.214 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:39.462779+0000", 2020-01-16T00:14:23.214 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 43.671931305000001, 2020-01-16T00:14:23.215 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020946985000000001, 2020-01-16T00:14:23.215 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:23.215 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:23.215 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:23.215 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:23.215 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:23.215 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3627 2020-01-16T00:14:23.215 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.216 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:23.216 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.216 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:23.216 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.462779+0000", 2020-01-16T00:14:23.216 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:23.216 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.216 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.216 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:23.217 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.462779+0000", 2020-01-16T00:14:23.217 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:23.217 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.217 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.217 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:23.217 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.462777+0000", 2020-01-16T00:14:23.217 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00064136 2020-01-16T00:14:23.218 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.218 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.218 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:23.218 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.463418+0000", 2020-01-16T00:14:23.218 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4607e-05 2020-01-16T00:14:23.218 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.218 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.218 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:23.219 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.463443+0000", 2020-01-16T00:14:23.219 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0346999999999999e-05 2020-01-16T00:14:23.219 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.219 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.219 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:23.219 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.463453+0000", 2020-01-16T00:14:23.219 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000100988 2020-01-16T00:14:23.219 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.220 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.220 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.220 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.463554+0000", 2020-01-16T00:14:23.220 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023059280000000001 2020-01-16T00:14:23.220 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.220 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.220 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:23.220 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.465860+0000", 2020-01-16T00:14:23.221 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011316961 2020-01-16T00:14:23.221 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.221 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.221 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.221 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.477177+0000", 2020-01-16T00:14:23.221 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000132402 2020-01-16T00:14:23.221 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.221 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.222 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:23.222 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.477309+0000", 2020-01-16T00:14:23.222 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000223606 2020-01-16T00:14:23.222 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.222 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.222 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:23.222 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.477533+0000", 2020-01-16T00:14:23.222 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0057105209999999997 2020-01-16T00:14:23.223 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.223 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.223 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:23.223 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483244+0000", 2020-01-16T00:14:23.223 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000193031 2020-01-16T00:14:23.223 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.223 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.224 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:23.224 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483437+0000", 2020-01-16T00:14:23.224 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0902999999999995e-05 2020-01-16T00:14:23.224 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.224 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.224 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:23.225 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483507+0000", 2020-01-16T00:14:23.225 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000129412 2020-01-16T00:14:23.225 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.225 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.225 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:23.225 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483637+0000", 2020-01-16T00:14:23.225 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.9345999999999995e-05 2020-01-16T00:14:23.226 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.226 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.226 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:23.226 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483726+0000", 2020-01-16T00:14:23.226 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020946985000000001 2020-01-16T00:14:23.226 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.226 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:23.226 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.227 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.227 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.227 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3628 3.a 3:5069665e:::smithi12313934-2:head [write 2853383~452450] snapc 0=[] ondisk+write+known_if_redirected e48)", 2020-01-16T00:14:23.227 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:39.468178+0000", 2020-01-16T00:14:23.227 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 43.666532596000003, 2020-01-16T00:14:23.227 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015705872999999999, 2020-01-16T00:14:23.227 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:23.228 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:23.228 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:23.228 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:23.228 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:23.228 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3628 2020-01-16T00:14:23.228 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.228 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:23.228 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.229 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:23.229 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468178+0000", 2020-01-16T00:14:23.229 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:23.229 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.229 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.229 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:23.229 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468178+0000", 2020-01-16T00:14:23.229 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:23.230 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.230 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.230 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:23.230 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468175+0000", 2020-01-16T00:14:23.230 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00065948000000000005 2020-01-16T00:14:23.230 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.230 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.231 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:23.231 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468834+0000", 2020-01-16T00:14:23.231 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3557000000000002e-05 2020-01-16T00:14:23.231 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.231 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.231 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:23.231 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468858+0000", 2020-01-16T00:14:23.232 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2931e-05 2020-01-16T00:14:23.232 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.232 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.232 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:23.232 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468871+0000", 2020-01-16T00:14:23.232 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000123113 2020-01-16T00:14:23.232 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.232 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.233 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.233 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468994+0000", 2020-01-16T00:14:23.233 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017615999999999999 2020-01-16T00:14:23.233 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.233 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.233 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:23.233 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.469170+0000", 2020-01-16T00:14:23.234 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010336908000000001 2020-01-16T00:14:23.234 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.234 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.234 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.234 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.479507+0000", 2020-01-16T00:14:23.234 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015856400000000001 2020-01-16T00:14:23.234 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.234 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.235 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:23.235 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.479665+0000", 2020-01-16T00:14:23.235 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030344600000000002 2020-01-16T00:14:23.235 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.235 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.235 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:23.235 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.479969+0000", 2020-01-16T00:14:23.235 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0037935389999999999 2020-01-16T00:14:23.236 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.236 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.236 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:23.236 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483762+0000", 2020-01-16T00:14:23.236 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3170999999999997e-05 2020-01-16T00:14:23.236 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.236 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.236 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:23.237 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483806+0000", 2020-01-16T00:14:23.237 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3821000000000001e-05 2020-01-16T00:14:23.237 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.237 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.237 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:23.237 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483829+0000", 2020-01-16T00:14:23.237 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0076000000000001e-05 2020-01-16T00:14:23.237 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.238 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.238 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:23.238 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483859+0000", 2020-01-16T00:14:23.238 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.438e-05 2020-01-16T00:14:23.238 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.238 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.238 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:23.238 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483884+0000", 2020-01-16T00:14:23.239 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015705872999999999 2020-01-16T00:14:23.239 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.239 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:23.239 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.239 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.239 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.239 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3629 3.a 3:5069665e:::smithi12313934-2:head [setxattr _header (55) in=7b,truncate 3529536] snapc 0=[] ondisk+write+known_if_redirected e48)", 2020-01-16T00:14:23.240 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:39.468888+0000", 2020-01-16T00:14:23.240 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 43.665822974999998, 2020-01-16T00:14:23.240 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015108511, 2020-01-16T00:14:23.240 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:23.240 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:23.240 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:23.240 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:23.240 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:23.241 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3629 2020-01-16T00:14:23.241 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.241 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:23.241 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.241 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:23.241 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468888+0000", 2020-01-16T00:14:23.241 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:23.241 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.242 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.242 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:23.242 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468888+0000", 2020-01-16T00:14:23.242 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:23.242 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.243 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.243 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:23.243 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468886+0000", 2020-01-16T00:14:23.243 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5599999999999996e-06 2020-01-16T00:14:23.243 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.243 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.243 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:23.244 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468894+0000", 2020-01-16T00:14:23.244 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5279999999999998e-06 2020-01-16T00:14:23.244 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.244 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.244 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:23.244 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468901+0000", 2020-01-16T00:14:23.244 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2290000000000002e-06 2020-01-16T00:14:23.244 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.245 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.245 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:23.245 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468907+0000", 2020-01-16T00:14:23.245 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030197500000000001 2020-01-16T00:14:23.245 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.245 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.245 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.246 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.469209+0000", 2020-01-16T00:14:23.246 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6659999999999997e-05 2020-01-16T00:14:23.246 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.246 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.246 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:23.247 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.469255+0000", 2020-01-16T00:14:23.247 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013117904999999999 2020-01-16T00:14:23.247 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.247 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.247 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.247 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.482373+0000", 2020-01-16T00:14:23.247 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1260000000000006e-05 2020-01-16T00:14:23.247 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.248 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.248 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:23.248 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.482434+0000", 2020-01-16T00:14:23.248 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014433 2020-01-16T00:14:23.248 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.248 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.248 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:23.248 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.482579+0000", 2020-01-16T00:14:23.249 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012198160000000001 2020-01-16T00:14:23.249 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.249 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.249 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:23.249 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483799+0000", 2020-01-16T00:14:23.249 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.188e-05 2020-01-16T00:14:23.249 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.250 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.250 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:23.250 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483810+0000", 2020-01-16T00:14:23.251 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000114863 2020-01-16T00:14:23.251 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.251 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.251 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:23.252 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483925+0000", 2020-01-16T00:14:23.252 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0861000000000001e-05 2020-01-16T00:14:23.252 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.252 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.253 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:23.253 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483956+0000", 2020-01-16T00:14:23.253 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9894999999999997e-05 2020-01-16T00:14:23.253 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.254 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.254 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:23.254 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483996+0000", 2020-01-16T00:14:23.254 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015108511 2020-01-16T00:14:23.255 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.255 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:23.255 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.255 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.255 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.256 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3630 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e48)", 2020-01-16T00:14:23.256 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:39.468967+0000", 2020-01-16T00:14:23.256 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 43.665743841000001, 2020-01-16T00:14:23.256 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015160579, 2020-01-16T00:14:23.257 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:23.257 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:23.257 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:23.257 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:23.257 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:23.258 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3630 2020-01-16T00:14:23.258 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.258 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:23.258 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.259 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:23.259 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468967+0000", 2020-01-16T00:14:23.259 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:23.259 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.260 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.260 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:23.260 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468967+0000", 2020-01-16T00:14:23.260 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:23.260 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.261 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.261 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:23.261 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468966+0000", 2020-01-16T00:14:23.261 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3430000000000003e-06 2020-01-16T00:14:23.262 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.262 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.262 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:23.262 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468971+0000", 2020-01-16T00:14:23.263 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6577000000000001e-05 2020-01-16T00:14:23.263 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.263 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.263 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:23.263 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468988+0000", 2020-01-16T00:14:23.264 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.378e-06 2020-01-16T00:14:23.264 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.264 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.264 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:23.265 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468993+0000", 2020-01-16T00:14:23.265 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000320601 2020-01-16T00:14:23.265 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.265 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.266 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.266 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.469314+0000", 2020-01-16T00:14:23.266 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013833699999999999 2020-01-16T00:14:23.266 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.266 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.267 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:23.267 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.469452+0000", 2020-01-16T00:14:23.267 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013447838 2020-01-16T00:14:23.267 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.268 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.268 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.268 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.482900+0000", 2020-01-16T00:14:23.268 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028006899999999998 2020-01-16T00:14:23.269 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.269 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.269 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:23.269 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483180+0000", 2020-01-16T00:14:23.269 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000833594 2020-01-16T00:14:23.270 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.270 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.270 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.270 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.484013+0000", 2020-01-16T00:14:23.271 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2601e-05 2020-01-16T00:14:23.271 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.271 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.271 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:23.272 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.484066+0000", 2020-01-16T00:14:23.272 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1303999999999994e-05 2020-01-16T00:14:23.272 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.272 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.272 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:23.273 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.484127+0000", 2020-01-16T00:14:23.273 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015160579 2020-01-16T00:14:23.273 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.273 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:23.274 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.274 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.274 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.274 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4047 3.a 3:5069665e:::smithi12313934-2:head [write 1935828~635840] snapc 0=[] ondisk+write+known_if_redirected e52)", 2020-01-16T00:14:23.275 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:50.302087+0000", 2020-01-16T00:14:23.275 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 32.832623570000003, 2020-01-16T00:14:23.275 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014518982, 2020-01-16T00:14:23.275 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:23.275 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:23.276 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:23.276 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:23.276 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:23.276 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4047 2020-01-16T00:14:23.277 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.277 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:23.277 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.277 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:23.277 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302087+0000", 2020-01-16T00:14:23.278 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:23.278 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.278 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.278 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:23.279 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302087+0000", 2020-01-16T00:14:23.279 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:23.279 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.279 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.279 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:23.280 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302085+0000", 2020-01-16T00:14:23.280 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00057147600000000002 2020-01-16T00:14:23.280 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.280 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.280 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:23.280 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302657+0000", 2020-01-16T00:14:23.281 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6427999999999999e-05 2020-01-16T00:14:23.281 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.281 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.281 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:23.281 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302673+0000", 2020-01-16T00:14:23.282 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.127e-06 2020-01-16T00:14:23.282 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.282 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.282 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:23.282 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302680+0000", 2020-01-16T00:14:23.282 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6378000000000006e-05 2020-01-16T00:14:23.283 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.283 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.283 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.283 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302756+0000", 2020-01-16T00:14:23.283 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00039898000000000001 2020-01-16T00:14:23.284 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.284 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.284 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:23.284 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.303155+0000", 2020-01-16T00:14:23.284 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0079666870000000001 2020-01-16T00:14:23.284 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.284 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.285 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.285 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.311122+0000", 2020-01-16T00:14:23.285 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019101400000000001 2020-01-16T00:14:23.285 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.285 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.285 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:23.285 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.311313+0000", 2020-01-16T00:14:23.286 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033332499999999998 2020-01-16T00:14:23.286 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.286 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.286 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:23.286 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.311646+0000", 2020-01-16T00:14:23.286 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0029841440000000002 2020-01-16T00:14:23.286 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.286 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.287 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:23.287 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.314630+0000", 2020-01-16T00:14:23.287 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00027996800000000001 2020-01-16T00:14:23.287 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.287 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.288 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:23.288 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.314910+0000", 2020-01-16T00:14:23.288 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0015637159999999999 2020-01-16T00:14:23.288 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.289 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.289 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:23.289 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.316474+0000", 2020-01-16T00:14:23.289 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5986000000000001e-05 2020-01-16T00:14:23.290 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.290 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.290 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:23.290 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.316550+0000", 2020-01-16T00:14:23.290 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6258000000000002e-05 2020-01-16T00:14:23.290 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.290 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.291 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:23.291 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.316606+0000", 2020-01-16T00:14:23.291 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014518982 2020-01-16T00:14:23.291 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.292 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:23.292 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.292 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.292 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.292 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4600 3.a 3:5069665e:::smithi12313934-2:head [write 1791765~569861] snapc 0=[] ondisk+write+known_if_redirected e56)", 2020-01-16T00:14:23.292 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:03.059744+0000", 2020-01-16T00:14:23.292 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 20.074966353000001, 2020-01-16T00:14:23.293 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017075275000000001, 2020-01-16T00:14:23.293 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:23.293 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:23.293 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:23.293 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:23.293 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:23.293 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4600 2020-01-16T00:14:23.294 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.294 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:23.294 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.294 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:23.294 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.059744+0000", 2020-01-16T00:14:23.294 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:23.294 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.295 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.295 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:23.295 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.059744+0000", 2020-01-16T00:14:23.295 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999843 2020-01-16T00:14:23.295 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.295 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.295 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:23.295 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.059728+0000", 2020-01-16T00:14:23.296 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00054167699999999996 2020-01-16T00:14:23.296 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.296 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.296 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:23.296 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.060270+0000", 2020-01-16T00:14:23.296 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6693999999999998e-05 2020-01-16T00:14:23.296 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.297 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.297 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:23.297 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.060307+0000", 2020-01-16T00:14:23.297 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8753000000000002e-05 2020-01-16T00:14:23.297 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.297 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.297 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:23.297 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.060326+0000", 2020-01-16T00:14:23.298 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000114177 2020-01-16T00:14:23.298 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.298 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.298 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.298 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.060440+0000", 2020-01-16T00:14:23.298 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001642382 2020-01-16T00:14:23.298 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.299 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.299 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:23.299 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.062082+0000", 2020-01-16T00:14:23.299 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0093492149999999993 2020-01-16T00:14:23.299 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.299 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.299 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.299 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.071431+0000", 2020-01-16T00:14:23.300 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013135600000000001 2020-01-16T00:14:23.300 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.300 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.300 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:23.300 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.071563+0000", 2020-01-16T00:14:23.300 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00032728299999999999 2020-01-16T00:14:23.300 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.301 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.301 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:23.301 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.071890+0000", 2020-01-16T00:14:23.301 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0043339759999999998 2020-01-16T00:14:23.301 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.301 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.301 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:23.301 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.076224+0000", 2020-01-16T00:14:23.302 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000343462 2020-01-16T00:14:23.302 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.302 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.302 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:23.302 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.076567+0000", 2020-01-16T00:14:23.302 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1668000000000006e-05 2020-01-16T00:14:23.302 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.303 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.303 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:23.303 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.076639+0000", 2020-01-16T00:14:23.303 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8572000000000004e-05 2020-01-16T00:14:23.303 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.303 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.304 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:23.304 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.076738+0000", 2020-01-16T00:14:23.304 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1816000000000004e-05 2020-01-16T00:14:23.304 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.304 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.304 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:23.304 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.076819+0000", 2020-01-16T00:14:23.305 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017075275000000001 2020-01-16T00:14:23.305 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.305 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:23.305 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.305 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.305 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.305 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4934 3.a 3:5069665e:::smithi12313934-2:head [write 1900783~425211] snapc 0=[] ondisk+write+known_if_redirected e60)", 2020-01-16T00:14:23.305 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:13.356857+0000", 2020-01-16T00:14:23.306 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 9.7778530739999994, 2020-01-16T00:14:23.306 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020085438000000001, 2020-01-16T00:14:23.306 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:23.306 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:23.306 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:23.306 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:23.306 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:23.307 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4934 2020-01-16T00:14:23.307 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.307 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:23.307 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.307 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:23.308 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.356857+0000", 2020-01-16T00:14:23.308 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:23.308 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.308 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.308 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:23.308 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.356857+0000", 2020-01-16T00:14:23.309 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:14:23.309 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.309 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.309 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:23.309 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.356854+0000", 2020-01-16T00:14:23.309 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000450741 2020-01-16T00:14:23.310 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.310 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.310 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:23.310 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357305+0000", 2020-01-16T00:14:23.310 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1829999999999998e-05 2020-01-16T00:14:23.310 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.310 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.311 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:23.311 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357336+0000", 2020-01-16T00:14:23.311 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.305e-05 2020-01-16T00:14:23.311 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.311 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.311 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:23.311 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357349+0000", 2020-01-16T00:14:23.312 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000131087 2020-01-16T00:14:23.312 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.312 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.312 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.312 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357480+0000", 2020-01-16T00:14:23.312 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0031475409999999998 2020-01-16T00:14:23.312 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.313 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.313 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:23.313 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.360628+0000", 2020-01-16T00:14:23.313 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011820447 2020-01-16T00:14:23.313 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.313 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.313 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.314 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.372448+0000", 2020-01-16T00:14:23.314 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000154093 2020-01-16T00:14:23.314 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.314 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.314 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:23.314 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.372603+0000", 2020-01-16T00:14:23.314 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00035365400000000001 2020-01-16T00:14:23.315 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.315 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.315 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:23.315 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.372956+0000", 2020-01-16T00:14:23.315 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0035908590000000001 2020-01-16T00:14:23.315 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.315 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.315 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:23.316 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.376547+0000", 2020-01-16T00:14:23.316 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3077e-05 2020-01-16T00:14:23.316 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.316 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.316 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:23.316 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.376560+0000", 2020-01-16T00:14:23.317 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024608200000000001 2020-01-16T00:14:23.317 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.317 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.317 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:23.317 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.376806+0000", 2020-01-16T00:14:23.318 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3510999999999995e-05 2020-01-16T00:14:23.318 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.318 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.318 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:23.319 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.376880+0000", 2020-01-16T00:14:23.319 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3170000000000007e-05 2020-01-16T00:14:23.319 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.319 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.319 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:23.320 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.376943+0000", 2020-01-16T00:14:23.320 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020085438000000001 2020-01-16T00:14:23.320 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.320 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:23.321 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.321 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.321 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.321 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4935 3.a 3:5069665e:::smithi12313934-2:head [setxattr _header (55) in=7b,truncate 2689252] snapc 0=[] ondisk+write+known_if_redirected e60)", 2020-01-16T00:14:23.321 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:13.357371+0000", 2020-01-16T00:14:23.322 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 9.7773394669999991, 2020-01-16T00:14:23.322 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021339295000000001, 2020-01-16T00:14:23.322 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:23.322 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:23.323 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:23.323 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:23.323 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:23.323 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4935 2020-01-16T00:14:23.324 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.324 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:23.324 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.324 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:23.324 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357371+0000", 2020-01-16T00:14:23.325 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:23.325 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.325 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.325 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:23.326 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357371+0000", 2020-01-16T00:14:23.326 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:23.326 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.326 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.327 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:23.327 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357369+0000", 2020-01-16T00:14:23.327 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2839999999999995e-06 2020-01-16T00:14:23.327 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.327 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.328 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:23.329 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357379+0000", 2020-01-16T00:14:23.329 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4363e-05 2020-01-16T00:14:23.329 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.329 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.329 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:23.329 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357403+0000", 2020-01-16T00:14:23.329 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7800000000000003e-06 2020-01-16T00:14:23.330 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.330 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.330 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:23.330 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357410+0000", 2020-01-16T00:14:23.330 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003365771 2020-01-16T00:14:23.330 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.330 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.331 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.331 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.360776+0000", 2020-01-16T00:14:23.331 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00021243299999999999 2020-01-16T00:14:23.331 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.331 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.331 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:23.331 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.360988+0000", 2020-01-16T00:14:23.331 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014015972 2020-01-16T00:14:23.332 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.332 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.332 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.332 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.375004+0000", 2020-01-16T00:14:23.332 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000143306 2020-01-16T00:14:23.332 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.332 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.333 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:23.333 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.375147+0000", 2020-01-16T00:14:23.333 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019620299999999999 2020-01-16T00:14:23.333 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.333 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.333 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:23.333 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.375344+0000", 2020-01-16T00:14:23.333 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001860483 2020-01-16T00:14:23.334 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.334 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.334 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:23.334 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.377204+0000", 2020-01-16T00:14:23.334 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0013554870000000001 2020-01-16T00:14:23.334 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.334 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.335 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:23.335 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.378560+0000", 2020-01-16T00:14:23.335 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8186000000000002e-05 2020-01-16T00:14:23.335 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.335 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.335 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:23.335 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.378608+0000", 2020-01-16T00:14:23.335 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8071000000000001e-05 2020-01-16T00:14:23.336 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.336 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.336 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:23.336 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.378656+0000", 2020-01-16T00:14:23.336 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4595999999999997e-05 2020-01-16T00:14:23.336 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.336 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.336 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:23.337 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.378710+0000", 2020-01-16T00:14:23.337 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021339295000000001 2020-01-16T00:14:23.337 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.337 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:23.337 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.337 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.338 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.338 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4936 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e60)", 2020-01-16T00:14:23.338 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:13.357465+0000", 2020-01-16T00:14:23.338 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 9.7772456969999997, 2020-01-16T00:14:23.339 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021436350999999999, 2020-01-16T00:14:23.339 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:23.340 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:23.340 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:23.340 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:23.340 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:23.341 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4936 2020-01-16T00:14:23.341 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.341 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:23.341 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.341 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:23.342 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357465+0000", 2020-01-16T00:14:23.342 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:23.342 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.342 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.342 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:23.342 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357465+0000", 2020-01-16T00:14:23.342 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:23.343 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.343 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.343 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:23.343 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357464+0000", 2020-01-16T00:14:23.343 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.704e-06 2020-01-16T00:14:23.343 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.343 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.343 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:23.344 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357472+0000", 2020-01-16T00:14:23.344 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6500000000000001e-05 2020-01-16T00:14:23.344 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.344 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.344 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:23.344 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357489+0000", 2020-01-16T00:14:23.344 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.026e-05 2020-01-16T00:14:23.345 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.346 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.346 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:23.346 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357499+0000", 2020-01-16T00:14:23.346 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0042232679999999996 2020-01-16T00:14:23.346 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.346 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.346 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.347 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.361722+0000", 2020-01-16T00:14:23.347 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015945999999999999 2020-01-16T00:14:23.347 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.347 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.347 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:23.347 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.361882+0000", 2020-01-16T00:14:23.347 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014213089999999999 2020-01-16T00:14:23.348 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.348 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.348 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.348 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.376095+0000", 2020-01-16T00:14:23.348 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013706999999999999 2020-01-16T00:14:23.348 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.349 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.349 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:23.349 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.376232+0000", 2020-01-16T00:14:23.349 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002519566 2020-01-16T00:14:23.349 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.349 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.349 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.349 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.378751+0000", 2020-01-16T00:14:23.350 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3343000000000006e-05 2020-01-16T00:14:23.350 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.350 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.350 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:23.350 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.378825+0000", 2020-01-16T00:14:23.351 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.64e-05 2020-01-16T00:14:23.351 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.351 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.351 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:23.351 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.378901+0000", 2020-01-16T00:14:23.351 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021436350999999999 2020-01-16T00:14:23.352 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.352 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:23.352 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.352 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.353 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.353 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:5155 3.a 3:5069665e:::smithi12313934-2:head [write 1648313~507551] snapc 0=[] ondisk+write+known_if_redirected e63)", 2020-01-16T00:14:23.353 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:18.197762+0000", 2020-01-16T00:14:23.354 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 4.9369481249999998, 2020-01-16T00:14:23.354 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014375384999999999, 2020-01-16T00:14:23.354 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:23.354 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:23.354 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:23.355 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:23.355 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:23.355 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 5155 2020-01-16T00:14:23.355 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.355 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:23.356 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.356 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:23.356 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.197762+0000", 2020-01-16T00:14:23.356 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:23.356 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.356 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.357 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:23.357 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.197762+0000", 2020-01-16T00:14:23.357 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:23.357 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.357 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.358 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:23.358 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.197760+0000", 2020-01-16T00:14:23.358 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00049261900000000002 2020-01-16T00:14:23.358 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.358 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.358 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:23.359 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.198253+0000", 2020-01-16T00:14:23.359 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8258e-05 2020-01-16T00:14:23.359 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.359 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.359 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:23.359 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.198271+0000", 2020-01-16T00:14:23.360 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0740000000000004e-06 2020-01-16T00:14:23.360 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.360 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.360 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:23.360 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.198278+0000", 2020-01-16T00:14:23.360 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001581121 2020-01-16T00:14:23.360 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.360 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.361 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.361 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.199859+0000", 2020-01-16T00:14:23.361 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00064528100000000004 2020-01-16T00:14:23.361 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.361 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.361 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:23.362 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.200504+0000", 2020-01-16T00:14:23.362 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0079495580000000007 2020-01-16T00:14:23.362 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.362 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.362 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.362 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.208454+0000", 2020-01-16T00:14:23.362 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.781e-05 2020-01-16T00:14:23.363 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.363 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.363 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:23.363 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.208552+0000", 2020-01-16T00:14:23.363 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014363900000000001 2020-01-16T00:14:23.363 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.363 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.364 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:23.364 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.208695+0000", 2020-01-16T00:14:23.364 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002629036 2020-01-16T00:14:23.364 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.364 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.365 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:23.365 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.211324+0000", 2020-01-16T00:14:23.365 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022371700000000001 2020-01-16T00:14:23.365 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.366 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.366 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:23.366 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.211548+0000", 2020-01-16T00:14:23.366 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00049117300000000002 2020-01-16T00:14:23.366 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.367 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.367 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:23.367 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.212039+0000", 2020-01-16T00:14:23.367 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7794999999999998e-05 2020-01-16T00:14:23.367 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.368 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.368 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:23.368 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.212097+0000", 2020-01-16T00:14:23.368 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0648000000000001e-05 2020-01-16T00:14:23.368 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.369 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.369 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:23.369 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.212138+0000", 2020-01-16T00:14:23.369 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014375384999999999 2020-01-16T00:14:23.370 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.370 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:23.370 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.370 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.371 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.371 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:5156 3.a 3:5069665e:::smithi12313934-2:head [write 2917489~292780] snapc 0=[] ondisk+write+known_if_redirected e63)", 2020-01-16T00:14:23.371 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:18.203801+0000", 2020-01-16T00:14:23.371 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 4.9309097749999999, 2020-01-16T00:14:23.372 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015806831, 2020-01-16T00:14:23.372 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:23.372 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:23.372 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:23.372 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:23.373 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:23.373 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 5156 2020-01-16T00:14:23.373 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.373 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:23.374 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.374 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:23.374 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.203801+0000", 2020-01-16T00:14:23.374 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:23.374 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.374 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.375 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:23.375 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.203801+0000", 2020-01-16T00:14:23.375 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:23.375 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.375 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.375 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:23.375 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.203798+0000", 2020-01-16T00:14:23.376 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00051419400000000004 2020-01-16T00:14:23.376 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.376 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.376 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:23.376 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204312+0000", 2020-01-16T00:14:23.376 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3977000000000001e-05 2020-01-16T00:14:23.376 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.376 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.377 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:23.377 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204336+0000", 2020-01-16T00:14:23.377 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4731e-05 2020-01-16T00:14:23.377 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.377 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.378 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:23.378 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204350+0000", 2020-01-16T00:14:23.378 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.522e-05 2020-01-16T00:14:23.378 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.378 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.378 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.378 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204446+0000", 2020-01-16T00:14:23.378 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010202 2020-01-16T00:14:23.379 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.379 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.379 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:23.379 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204548+0000", 2020-01-16T00:14:23.379 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.005158124 2020-01-16T00:14:23.379 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.380 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.380 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.380 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.209706+0000", 2020-01-16T00:14:23.380 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001294309 2020-01-16T00:14:23.380 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.380 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.380 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:23.381 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.211000+0000", 2020-01-16T00:14:23.381 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0065338569999999997 2020-01-16T00:14:23.381 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.381 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.381 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.381 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.217534+0000", 2020-01-16T00:14:23.381 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5905e-05 2020-01-16T00:14:23.382 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.382 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.382 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:23.382 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.217600+0000", 2020-01-16T00:14:23.382 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000107805 2020-01-16T00:14:23.382 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.382 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.383 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:23.383 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.217708+0000", 2020-01-16T00:14:23.383 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012035570000000001 2020-01-16T00:14:23.383 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.383 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.383 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:23.383 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.218911+0000", 2020-01-16T00:14:23.384 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00027311300000000001 2020-01-16T00:14:23.384 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.384 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.384 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:23.384 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.219184+0000", 2020-01-16T00:14:23.384 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00031979399999999998 2020-01-16T00:14:23.384 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.384 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.385 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:23.385 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.219504+0000", 2020-01-16T00:14:23.385 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2009999999999998e-05 2020-01-16T00:14:23.385 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.385 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.385 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:23.386 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.219566+0000", 2020-01-16T00:14:23.386 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1406000000000003e-05 2020-01-16T00:14:23.386 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.386 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.386 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:23.386 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.219608+0000", 2020-01-16T00:14:23.387 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015806831 2020-01-16T00:14:23.387 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.387 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:23.387 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.387 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.387 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.387 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:5157 3.a 3:5069665e:::smithi12313934-2:head [setxattr _header (55) in=7b,truncate 3210269] snapc 0=[] ondisk+write+known_if_redirected e63)", 2020-01-16T00:14:23.388 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:18.204374+0000", 2020-01-16T00:14:23.388 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 4.9303362609999999, 2020-01-16T00:14:23.388 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015400502, 2020-01-16T00:14:23.388 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:23.388 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:23.388 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:23.388 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:23.389 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:23.389 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 5157 2020-01-16T00:14:23.389 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.389 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:23.389 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.389 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:23.389 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204374+0000", 2020-01-16T00:14:23.390 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:23.390 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.390 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.390 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:23.390 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204374+0000", 2020-01-16T00:14:23.390 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:23.390 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.391 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.391 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:23.391 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204373+0000", 2020-01-16T00:14:23.391 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.4870000000000002e-06 2020-01-16T00:14:23.391 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.391 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.392 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:23.392 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204381+0000", 2020-01-16T00:14:23.392 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4402e-05 2020-01-16T00:14:23.392 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.392 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.392 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:23.392 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204396+0000", 2020-01-16T00:14:23.393 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6640000000000001e-06 2020-01-16T00:14:23.393 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.393 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.393 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:23.393 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204403+0000", 2020-01-16T00:14:23.393 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00021196100000000001 2020-01-16T00:14:23.393 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.394 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.394 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.394 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204614+0000", 2020-01-16T00:14:23.394 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1501999999999994e-05 2020-01-16T00:14:23.394 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.394 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.394 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:23.395 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204686+0000", 2020-01-16T00:14:23.395 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0063990649999999998 2020-01-16T00:14:23.395 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.395 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.395 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.395 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.211085+0000", 2020-01-16T00:14:23.395 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1210000000000003e-05 2020-01-16T00:14:23.396 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.396 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.396 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:23.396 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.211166+0000", 2020-01-16T00:14:23.396 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0071190699999999999 2020-01-16T00:14:23.396 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.396 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.397 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.397 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.218285+0000", 2020-01-16T00:14:23.397 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8831999999999994e-05 2020-01-16T00:14:23.397 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.397 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.397 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:23.397 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.218354+0000", 2020-01-16T00:14:23.398 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013551599999999999 2020-01-16T00:14:23.398 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.398 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.398 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:23.398 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.218490+0000", 2020-01-16T00:14:23.398 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00057678000000000004 2020-01-16T00:14:23.399 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.399 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.399 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:23.399 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.219066+0000", 2020-01-16T00:14:23.399 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022067000000000001 2020-01-16T00:14:23.399 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.399 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.399 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:23.400 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.219287+0000", 2020-01-16T00:14:23.400 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000401868 2020-01-16T00:14:23.400 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.400 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.400 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:23.400 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.219689+0000", 2020-01-16T00:14:23.400 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0040000000000003e-05 2020-01-16T00:14:23.401 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.401 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.401 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:23.401 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.219729+0000", 2020-01-16T00:14:23.401 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5762e-05 2020-01-16T00:14:23.401 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.401 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.402 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:23.402 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.219775+0000", 2020-01-16T00:14:23.402 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015400502 2020-01-16T00:14:23.402 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.402 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:23.402 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.402 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.403 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.403 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:5158 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e63)", 2020-01-16T00:14:23.403 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:18.204424+0000", 2020-01-16T00:14:23.403 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 4.9302865489999999, 2020-01-16T00:14:23.403 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015513872999999999, 2020-01-16T00:14:23.403 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:23.403 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:23.404 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:23.404 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:23.404 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:23.404 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 5158 2020-01-16T00:14:23.404 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.404 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:23.404 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.405 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:23.405 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204424+0000", 2020-01-16T00:14:23.405 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:23.405 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.405 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.405 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:23.405 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204424+0000", 2020-01-16T00:14:23.406 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:23.406 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.406 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.406 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:23.406 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204423+0000", 2020-01-16T00:14:23.406 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4739999999999996e-06 2020-01-16T00:14:23.406 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.407 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.407 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:23.407 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204429+0000", 2020-01-16T00:14:23.407 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9245999999999998e-05 2020-01-16T00:14:23.407 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.407 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.407 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:23.408 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204448+0000", 2020-01-16T00:14:23.408 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.4540000000000007e-06 2020-01-16T00:14:23.408 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.408 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.408 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:23.408 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204456+0000", 2020-01-16T00:14:23.408 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00027379599999999998 2020-01-16T00:14:23.409 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.409 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.409 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.409 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204730+0000", 2020-01-16T00:14:23.409 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2149000000000004e-05 2020-01-16T00:14:23.409 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.410 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.410 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:23.410 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204792+0000", 2020-01-16T00:14:23.410 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0064440110000000004 2020-01-16T00:14:23.410 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.410 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.410 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.411 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.211236+0000", 2020-01-16T00:14:23.411 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2772000000000001e-05 2020-01-16T00:14:23.411 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.411 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.411 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:23.411 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.211299+0000", 2020-01-16T00:14:23.411 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0074481130000000001 2020-01-16T00:14:23.412 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.412 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.412 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.412 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.218747+0000", 2020-01-16T00:14:23.412 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011178100000000001 2020-01-16T00:14:23.412 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.412 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.413 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:23.413 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.218859+0000", 2020-01-16T00:14:23.413 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00093467799999999998 2020-01-16T00:14:23.413 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.413 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.413 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:23.413 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.219794+0000", 2020-01-16T00:14:23.414 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2992999999999998e-05 2020-01-16T00:14:23.414 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.414 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.414 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:23.414 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.219857+0000", 2020-01-16T00:14:23.414 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1285000000000005e-05 2020-01-16T00:14:23.415 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:23.415 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:23.415 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:23.415 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.219938+0000", 2020-01-16T00:14:23.415 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015513872999999999 2020-01-16T00:14:23.415 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.415 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:23.416 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.416 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:23.416 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:23.416 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:23.420 INFO:tasks.rados.rados.0.smithi123.stdout:1869: writing smithi12313934-37 from 447251 to 1182931 tid 1 2020-01-16T00:14:23.420 INFO:tasks.rados.rados.0.smithi123.stdout:1869: writing smithi12313934-37 from 1736139 to 2175034 tid 2 2020-01-16T00:14:23.420 INFO:tasks.rados.rados.0.smithi123.stdout:1869: writing smithi12313934-37 from 2597781 to 3378580 tid 3 2020-01-16T00:14:23.420 INFO:tasks.rados.rados.0.smithi123.stdout:1870: write (excl) oid 13 current snap is 0 2020-01-16T00:14:23.420 INFO:tasks.rados.rados.0.smithi123.stdout:1870: seq_num 694 ranges {464058=410929,1394621=698906,2625204=592071,3656107=7951} 2020-01-16T00:14:23.420 INFO:tasks.rados.rados.0.smithi123.stdout:1870: writing smithi12313934-13 from 464058 to 874987 tid 1 2020-01-16T00:14:23.420 INFO:tasks.rados.rados.0.smithi123.stdout:1870: writing smithi12313934-13 from 1394621 to 2093527 tid 2 2020-01-16T00:14:23.421 INFO:tasks.rados.rados.0.smithi123.stdout:1870: writing smithi12313934-13 from 2625204 to 3217275 tid 3 2020-01-16T00:14:23.421 INFO:tasks.rados.rados.0.smithi123.stdout:1870: writing smithi12313934-13 from 3656107 to 3664058 tid 4 2020-01-16T00:14:23.421 INFO:tasks.rados.rados.0.smithi123.stdout:1871: delete oid 47 current snap is 0 2020-01-16T00:14:23.421 INFO:tasks.rados.rados.0.smithi123.stdout:1856: finishing write tid 5 to smithi12313934-7 2020-01-16T00:14:23.421 INFO:tasks.rados.rados.0.smithi123.stdout:1856: finishing write tid 6 to smithi12313934-7 2020-01-16T00:14:23.421 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 7 v 262 (ObjNum 685 snap 0 seq_num 685) dirty exists 2020-01-16T00:14:23.421 INFO:tasks.rados.rados.0.smithi123.stdout:1856: left oid 7 (ObjNum 685 snap 0 seq_num 685) 2020-01-16T00:14:23.422 INFO:tasks.rados.rados.0.smithi123.stdout:1857: expect (ObjNum 674 snap 0 seq_num 674) 2020-01-16T00:14:23.422 INFO:tasks.rados.rados.0.smithi123.stdout:1858: finishing write tid 1 to smithi12313934-18 2020-01-16T00:14:23.422 INFO:tasks.rados.rados.0.smithi123.stdout:1858: finishing write tid 2 to smithi12313934-18 2020-01-16T00:14:23.422 INFO:tasks.rados.rados.0.smithi123.stdout:1858: finishing write tid 3 to smithi12313934-18 2020-01-16T00:14:23.422 INFO:tasks.rados.rados.0.smithi123.stdout:1858: finishing write tid 5 to smithi12313934-18 2020-01-16T00:14:23.422 INFO:tasks.rados.rados.0.smithi123.stdout:1858: finishing write tid 6 to smithi12313934-18 2020-01-16T00:14:23.422 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 18 v 352 (ObjNum 686 snap 0 seq_num 686) dirty exists 2020-01-16T00:14:23.422 INFO:tasks.rados.rados.0.smithi123.stdout:1858: left oid 18 (ObjNum 686 snap 0 seq_num 686) 2020-01-16T00:14:23.423 INFO:tasks.rados.rados.0.smithi123.stdout:1859: expect (ObjNum 679 snap 0 seq_num 679) 2020-01-16T00:14:23.423 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:14:23.423 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:14:23.423 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:14:23.423 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:14:23.423 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:14:23.424 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:14:23.424 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:14:23.424 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-01-16T00:14:23.424 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-01-16T00:14:23.424 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-01-16T00:14:23.424 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-01-16T00:14:23.424 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-01-16T00:14:23.425 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENXIO: problem getting command descriptions from osd.5 2020-01-16T00:14:23.425 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: problem getting command descriptions from osd.5 2020-01-16T00:14:23.425 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-01-16T00:14:23.425 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-01-16T00:14:23.425 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-01-16T00:14:23.425 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-01-16T00:14:23.426 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:23.416+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:23.426 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-01-16T00:14:23.441 INFO:tasks.rados.rados.0.smithi123.stdout:1860: finishing write tid 1 to smithi12313934-46 2020-01-16T00:14:23.442 INFO:tasks.rados.rados.0.smithi123.stdout:1860: finishing write tid 2 to smithi12313934-46 2020-01-16T00:14:23.442 INFO:tasks.rados.rados.0.smithi123.stdout:1861: expect (ObjNum 556 snap 0 seq_num 556) 2020-01-16T00:14:23.518 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:23.516+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:23.547 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:23.557 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-01-16T00:14:23.619 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:23.616+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:23.654 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:23.663 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-01-16T00:14:23.694 INFO:tasks.rados.rados.0.smithi123.stdout:1860: finishing write tid 3 to smithi12313934-46 2020-01-16T00:14:23.694 INFO:tasks.rados.rados.0.smithi123.stdout:1860: finishing write tid 5 to smithi12313934-46 2020-01-16T00:14:23.695 INFO:tasks.rados.rados.0.smithi123.stdout:1860: finishing write tid 6 to smithi12313934-46 2020-01-16T00:14:23.695 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 46 v 357 (ObjNum 687 snap 0 seq_num 687) dirty exists 2020-01-16T00:14:23.695 INFO:tasks.rados.rados.0.smithi123.stdout:1860: left oid 46 (ObjNum 687 snap 0 seq_num 687) 2020-01-16T00:14:23.695 INFO:tasks.rados.rados.0.smithi123.stdout:1863: finishing write tid 1 to smithi12313934-20 2020-01-16T00:14:23.695 INFO:tasks.rados.rados.0.smithi123.stdout:1863: finishing write tid 2 to smithi12313934-20 2020-01-16T00:14:23.696 INFO:tasks.rados.rados.0.smithi123.stdout:1863: finishing write tid 3 to smithi12313934-20 2020-01-16T00:14:23.696 INFO:tasks.rados.rados.0.smithi123.stdout:1863: finishing write tid 5 to smithi12313934-20 2020-01-16T00:14:23.696 INFO:tasks.rados.rados.0.smithi123.stdout:1863: finishing write tid 6 to smithi12313934-20 2020-01-16T00:14:23.696 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 20 v 289 (ObjNum 689 snap 0 seq_num 689) dirty exists 2020-01-16T00:14:23.696 INFO:tasks.rados.rados.0.smithi123.stdout:1863: left oid 20 (ObjNum 689 snap 0 seq_num 689) 2020-01-16T00:14:23.696 INFO:tasks.rados.rados.0.smithi123.stdout:1864: finishing write tid 1 to smithi12313934-5 2020-01-16T00:14:23.696 INFO:tasks.rados.rados.0.smithi123.stdout:1864: finishing write tid 2 to smithi12313934-5 2020-01-16T00:14:23.696 INFO:tasks.rados.rados.0.smithi123.stdout:1864: finishing write tid 4 to smithi12313934-5 2020-01-16T00:14:23.697 INFO:tasks.rados.rados.0.smithi123.stdout:1864: finishing write tid 5 to smithi12313934-5 2020-01-16T00:14:23.697 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 5 v 292 (ObjNum 690 snap 0 seq_num 690) dirty exists 2020-01-16T00:14:23.697 INFO:tasks.rados.rados.0.smithi123.stdout:1864: left oid 5 (ObjNum 690 snap 0 seq_num 690) 2020-01-16T00:14:23.697 INFO:tasks.rados.rados.0.smithi123.stdout:1865: finishing write tid 1 to smithi12313934-33 2020-01-16T00:14:23.697 INFO:tasks.rados.rados.0.smithi123.stdout:1865: finishing write tid 2 to smithi12313934-33 2020-01-16T00:14:23.697 INFO:tasks.rados.rados.0.smithi123.stdout:1856: done (14 left) 2020-01-16T00:14:23.697 INFO:tasks.rados.rados.0.smithi123.stdout:1857: done (13 left) 2020-01-16T00:14:23.698 INFO:tasks.rados.rados.0.smithi123.stdout:1858: done (12 left) 2020-01-16T00:14:23.698 INFO:tasks.rados.rados.0.smithi123.stdout:1859: done (11 left) 2020-01-16T00:14:23.698 INFO:tasks.rados.rados.0.smithi123.stdout:1860: done (10 left) 2020-01-16T00:14:23.698 INFO:tasks.rados.rados.0.smithi123.stdout:1861: done (9 left) 2020-01-16T00:14:23.698 INFO:tasks.rados.rados.0.smithi123.stdout:1863: done (8 left) 2020-01-16T00:14:23.698 INFO:tasks.rados.rados.0.smithi123.stdout:1864: done (7 left) 2020-01-16T00:14:23.698 INFO:tasks.rados.rados.0.smithi123.stdout:1871: done (6 left) 2020-01-16T00:14:23.698 INFO:tasks.rados.rados.0.smithi123.stdout:1872: read oid 6 snap -1 2020-01-16T00:14:23.699 INFO:tasks.rados.rados.0.smithi123.stdout:1872: expect (ObjNum 684 snap 0 seq_num 684) 2020-01-16T00:14:23.699 INFO:tasks.rados.rados.0.smithi123.stdout:1873: read oid 38 snap -1 2020-01-16T00:14:23.699 INFO:tasks.rados.rados.0.smithi123.stdout:1873: expect (ObjNum 668 snap 0 seq_num 668) 2020-01-16T00:14:23.699 INFO:tasks.rados.rados.0.smithi123.stdout:1874: read oid 48 snap -1 2020-01-16T00:14:23.702 INFO:tasks.rados.rados.0.smithi123.stdout:1874: expect (ObjNum 599 snap 0 seq_num 599) 2020-01-16T00:14:23.702 INFO:tasks.rados.rados.0.smithi123.stdout:1875: read oid 21 snap -1 2020-01-16T00:14:23.702 INFO:tasks.rados.rados.0.smithi123.stdout:1875: expect (ObjNum 576 snap 0 seq_num 576) 2020-01-16T00:14:23.702 INFO:tasks.rados.rados.0.smithi123.stdout:1876: write (excl) oid 31 current snap is 0 2020-01-16T00:14:23.702 INFO:tasks.rados.rados.0.smithi123.stdout:1876: seq_num 696 ranges {497672=561427,1795232=702575,2963697=730968} 2020-01-16T00:14:23.706 INFO:tasks.rados.rados.0.smithi123.stdout:1876: writing smithi12313934-31 from 497672 to 1059099 tid 1 2020-01-16T00:14:23.714 INFO:tasks.rados.rados.0.smithi123.stdout:1876: writing smithi12313934-31 from 1795232 to 2497807 tid 2 2020-01-16T00:14:23.720 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:23.718+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:23.721 INFO:tasks.rados.rados.0.smithi123.stdout:1876: writing smithi12313934-31 from 2963697 to 3694665 tid 3 2020-01-16T00:14:23.721 INFO:tasks.rados.rados.0.smithi123.stdout:1877: write oid 3 current snap is 0 2020-01-16T00:14:23.722 INFO:tasks.rados.rados.0.smithi123.stdout:1877: seq_num 697 ranges {514479=636676,1853714=446233,2991120=542240} 2020-01-16T00:14:23.728 INFO:tasks.rados.rados.0.smithi123.stdout:1877: writing smithi12313934-3 from 514479 to 1151155 tid 1 2020-01-16T00:14:23.734 INFO:tasks.rados.rados.0.smithi123.stdout:1877: writing smithi12313934-3 from 1853714 to 2299947 tid 2 2020-01-16T00:14:23.742 INFO:tasks.rados.rados.0.smithi123.stdout:1877: writing smithi12313934-3 from 2991120 to 3533360 tid 3 2020-01-16T00:14:23.742 INFO:tasks.rados.rados.0.smithi123.stdout:1878: write oid 10 current snap is 0 2020-01-16T00:14:23.742 INFO:tasks.rados.rados.0.smithi123.stdout:1878: seq_num 698 ranges {531286=711925,1912196=706244,3018543=712743} 2020-01-16T00:14:23.748 INFO:tasks.rados.rados.0.smithi123.stdout:1878: writing smithi12313934-10 from 531286 to 1243211 tid 1 2020-01-16T00:14:23.756 INFO:tasks.rados.rados.0.smithi123.stdout:1878: writing smithi12313934-10 from 1912196 to 2618440 tid 2 2020-01-16T00:14:23.763 INFO:tasks.rados.rados.0.smithi123.stdout:1878: writing smithi12313934-10 from 3018543 to 3731286 tid 3 2020-01-16T00:14:23.763 INFO:tasks.rados.rados.0.smithi123.stdout:1879: write (excl) oid 19 current snap is 0 2020-01-16T00:14:23.764 INFO:tasks.rados.rados.0.smithi123.stdout:1879: seq_num 699 ranges {548093=787174,1970678=449902,3045966=681137} 2020-01-16T00:14:23.770 INFO:tasks.rados.rados.0.smithi123.stdout:1879: writing smithi12313934-19 from 548093 to 1335267 tid 1 2020-01-16T00:14:23.773 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:23.777 INFO:tasks.rados.rados.0.smithi123.stdout:1879: writing smithi12313934-19 from 1970678 to 2420580 tid 2 2020-01-16T00:14:23.783 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-01-16T00:14:23.784 INFO:tasks.rados.rados.0.smithi123.stdout:1879: writing smithi12313934-19 from 3045966 to 3727103 tid 3 2020-01-16T00:14:23.785 INFO:tasks.rados.rados.0.smithi123.stdout:1880: write (excl) oid 7 current snap is 0 2020-01-16T00:14:23.785 INFO:tasks.rados.rados.0.smithi123.stdout:1880: seq_num 700 ranges {564900=578776,1629160=709913,3073389=492409} 2020-01-16T00:14:23.795 INFO:tasks.rados.rados.0.smithi123.stdout:1880: writing smithi12313934-7 from 564900 to 1143676 tid 1 2020-01-16T00:14:23.798 INFO:tasks.rados.rados.0.smithi123.stdout:1880: writing smithi12313934-7 from 1629160 to 2339073 tid 2 2020-01-16T00:14:23.805 INFO:tasks.rados.rados.0.smithi123.stdout:1880: writing smithi12313934-7 from 3073389 to 3565798 tid 3 2020-01-16T00:14:23.805 INFO:tasks.rados.rados.0.smithi123.stdout:1881: write oid 18 current snap is 0 2020-01-16T00:14:23.805 INFO:tasks.rados.rados.0.smithi123.stdout:1881: seq_num 701 ranges {581707=654025,1687642=453571,2700812=703681} 2020-01-16T00:14:23.811 INFO:tasks.rados.rados.0.smithi123.stdout:1881: writing smithi12313934-18 from 581707 to 1235732 tid 1 2020-01-16T00:14:23.817 INFO:tasks.rados.rados.0.smithi123.stdout:1881: writing smithi12313934-18 from 1687642 to 2141213 tid 2 2020-01-16T00:14:23.820 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:23.818+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:23.824 INFO:tasks.rados.rados.0.smithi123.stdout:1881: writing smithi12313934-18 from 2700812 to 3404493 tid 3 2020-01-16T00:14:23.825 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:23.825 INFO:tasks.rados.rados.0.smithi123.stdout:1865: finishing write tid 3 to smithi12313934-33 2020-01-16T00:14:23.825 INFO:tasks.rados.rados.0.smithi123.stdout:1865: finishing write tid 5 to smithi12313934-33 2020-01-16T00:14:23.825 INFO:tasks.rados.rados.0.smithi123.stdout:1865: finishing write tid 6 to smithi12313934-33 2020-01-16T00:14:23.825 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 33 v 361 (ObjNum 691 snap 0 seq_num 691) dirty exists 2020-01-16T00:14:23.825 INFO:tasks.rados.rados.0.smithi123.stdout:1865: left oid 33 (ObjNum 691 snap 0 seq_num 691) 2020-01-16T00:14:23.825 INFO:tasks.rados.rados.0.smithi123.stdout:1867: expect (ObjNum 652 snap 0 seq_num 652) 2020-01-16T00:14:23.893 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:23.903 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-01-16T00:14:23.947 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:23.946+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:24.028 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:24.026+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:24.045 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:24.053 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-01-16T00:14:24.078 INFO:tasks.rados.rados.0.smithi123.stdout:1868: finishing write tid 1 to smithi12313934-15 2020-01-16T00:14:24.078 INFO:tasks.rados.rados.0.smithi123.stdout:1868: finishing write tid 2 to smithi12313934-15 2020-01-16T00:14:24.079 INFO:tasks.rados.rados.0.smithi123.stdout:1868: finishing write tid 3 to smithi12313934-15 2020-01-16T00:14:24.079 INFO:tasks.rados.rados.0.smithi123.stdout:1868: finishing write tid 5 to smithi12313934-15 2020-01-16T00:14:24.079 INFO:tasks.rados.rados.0.smithi123.stdout:1868: finishing write tid 6 to smithi12313934-15 2020-01-16T00:14:24.080 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 15 v 241 (ObjNum 692 snap 0 seq_num 692) dirty exists 2020-01-16T00:14:24.080 INFO:tasks.rados.rados.0.smithi123.stdout:1868: left oid 15 (ObjNum 692 snap 0 seq_num 692) 2020-01-16T00:14:24.080 INFO:tasks.rados.rados.0.smithi123.stdout:1869: finishing write tid 1 to smithi12313934-37 2020-01-16T00:14:24.080 INFO:tasks.rados.rados.0.smithi123.stdout:1869: finishing write tid 2 to smithi12313934-37 2020-01-16T00:14:24.081 INFO:tasks.rados.rados.0.smithi123.stdout:1869: finishing write tid 3 to smithi12313934-37 2020-01-16T00:14:24.081 INFO:tasks.rados.rados.0.smithi123.stdout:1869: finishing write tid 5 to smithi12313934-37 2020-01-16T00:14:24.081 INFO:tasks.rados.rados.0.smithi123.stdout:1869: finishing write tid 6 to smithi12313934-37 2020-01-16T00:14:24.082 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 37 v 365 (ObjNum 693 snap 0 seq_num 693) dirty exists 2020-01-16T00:14:24.082 INFO:tasks.rados.rados.0.smithi123.stdout:1869: left oid 37 (ObjNum 693 snap 0 seq_num 693) 2020-01-16T00:14:24.082 INFO:tasks.rados.rados.0.smithi123.stdout:1870: finishing write tid 1 to smithi12313934-13 2020-01-16T00:14:24.083 INFO:tasks.rados.rados.0.smithi123.stdout:1870: finishing write tid 2 to smithi12313934-13 2020-01-16T00:14:24.083 INFO:tasks.rados.rados.0.smithi123.stdout:1865: done (15 left) 2020-01-16T00:14:24.083 INFO:tasks.rados.rados.0.smithi123.stdout:1866: done (14 left) 2020-01-16T00:14:24.083 INFO:tasks.rados.rados.0.smithi123.stdout:1867: done (13 left) 2020-01-16T00:14:24.084 INFO:tasks.rados.rados.0.smithi123.stdout:1868: done (12 left) 2020-01-16T00:14:24.084 INFO:tasks.rados.rados.0.smithi123.stdout:1869: done (11 left) 2020-01-16T00:14:24.084 INFO:tasks.rados.rados.0.smithi123.stdout:1882: read oid 12 snap -1 2020-01-16T00:14:24.084 INFO:tasks.rados.rados.0.smithi123.stdout:1882: expect (ObjNum 655 snap 0 seq_num 655) 2020-01-16T00:14:24.094 INFO:tasks.rados.rados.0.smithi123.stdout:1883: read oid 41 snap -1 2020-01-16T00:14:24.095 INFO:tasks.rados.rados.0.smithi123.stdout:1883: expect deleted 2020-01-16T00:14:24.095 INFO:tasks.rados.rados.0.smithi123.stdout:1884: write (excl) oid 22 current snap is 0 2020-01-16T00:14:24.096 INFO:tasks.rados.rados.0.smithi123.stdout:1884: seq_num 702 ranges {598514=729274,1746124=597229,3128235=514953} 2020-01-16T00:14:24.097 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2020-01-16T00:14:24.100 INFO:tasks.rados.rados.0.smithi123.stdout:1884: writing smithi12313934-22 from 598514 to 1327788 tid 1 2020-01-16T00:14:24.101 INFO:tasks.rados.rados.0.smithi123.stdout:1884: writing smithi12313934-22 from 1746124 to 2343353 tid 2 2020-01-16T00:14:24.101 INFO:tasks.rados.rados.0.smithi123.stdout:1884: writing smithi12313934-22 from 3128235 to 3643188 tid 3 2020-01-16T00:14:24.101 INFO:tasks.rados.rados.0.smithi123.stdout:1885: read oid 28 snap -1 2020-01-16T00:14:24.102 INFO:tasks.rados.rados.0.smithi123.stdout:1885: expect (ObjNum 652 snap 0 seq_num 652) 2020-01-16T00:14:24.102 INFO:tasks.rados.rados.0.smithi123.stdout:1870: finishing write tid 3 to smithi12313934-13 2020-01-16T00:14:24.102 INFO:tasks.rados.rados.0.smithi123.stdout:1870: finishing write tid 4 to smithi12313934-13 2020-01-16T00:14:24.102 INFO:tasks.rados.rados.0.smithi123.stdout:1870: finishing write tid 6 to smithi12313934-13 2020-01-16T00:14:24.103 INFO:tasks.rados.rados.0.smithi123.stdout:1870: finishing write tid 7 to smithi12313934-13 2020-01-16T00:14:24.103 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 124 (ObjNum 694 snap 0 seq_num 694) dirty exists 2020-01-16T00:14:24.103 INFO:tasks.rados.rados.0.smithi123.stdout:1870: left oid 13 (ObjNum 694 snap 0 seq_num 694) 2020-01-16T00:14:24.104 INFO:tasks.rados.rados.0.smithi123.stdout:1886: read oid 9 snap -1 2020-01-16T00:14:24.104 INFO:tasks.rados.rados.0.smithi123.stdout:1886: expect (ObjNum 672 snap 0 seq_num 672) 2020-01-16T00:14:24.104 INFO:tasks.rados.rados.0.smithi123.stdout:1870: done (15 left) 2020-01-16T00:14:24.104 INFO:tasks.rados.rados.0.smithi123.stdout:1887: read oid 44 snap -1 2020-01-16T00:14:24.105 INFO:tasks.rados.rados.0.smithi123.stdout:1887: expect deleted 2020-01-16T00:14:24.105 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:24.105 INFO:tasks.rados.rados.0.smithi123.stdout:1873: expect (ObjNum 668 snap 0 seq_num 668) 2020-01-16T00:14:24.155 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:24.154+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:24.209 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:24.219 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-01-16T00:14:24.233 INFO:tasks.rados.rados.0.smithi123.stdout:1875: expect (ObjNum 576 snap 0 seq_num 576) 2020-01-16T00:14:24.234 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:24.230+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:24.332 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:24.328+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:24.356 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:24.364 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-01-16T00:14:24.373 INFO:tasks.rados.rados.0.smithi123.stdout:1872: expect (ObjNum 684 snap 0 seq_num 684) 2020-01-16T00:14:24.432 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:24.430+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:24.461 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:24.470 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-01-16T00:14:24.555 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:24.554+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:24.611 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:24.619 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-01-16T00:14:24.637 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:24.636+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:24.655 INFO:tasks.rados.rados.0.smithi123.stdout:1874: expect (ObjNum 599 snap 0 seq_num 599) 2020-01-16T00:14:24.738 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:24.736+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:24.754 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:24.763 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-01-16T00:14:24.839 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:24.836+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:24.840 INFO:tasks.rados.rados.0.smithi123.stdout:1876: finishing write tid 1 to smithi12313934-31 2020-01-16T00:14:24.841 INFO:tasks.rados.rados.0.smithi123.stdout:1876: finishing write tid 2 to smithi12313934-31 2020-01-16T00:14:24.841 INFO:tasks.rados.rados.0.smithi123.stdout:1876: finishing write tid 3 to smithi12313934-31 2020-01-16T00:14:24.841 INFO:tasks.rados.rados.0.smithi123.stdout:1876: finishing write tid 5 to smithi12313934-31 2020-01-16T00:14:24.841 INFO:tasks.rados.rados.0.smithi123.stdout:1876: finishing write tid 6 to smithi12313934-31 2020-01-16T00:14:24.841 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 31 v 245 (ObjNum 696 snap 0 seq_num 696) dirty exists 2020-01-16T00:14:24.841 INFO:tasks.rados.rados.0.smithi123.stdout:1876: left oid 31 (ObjNum 696 snap 0 seq_num 696) 2020-01-16T00:14:24.842 INFO:tasks.rados.rados.0.smithi123.stdout:1877: finishing write tid 1 to smithi12313934-3 2020-01-16T00:14:24.842 INFO:tasks.rados.rados.0.smithi123.stdout:1877: finishing write tid 2 to smithi12313934-3 2020-01-16T00:14:24.842 INFO:tasks.rados.rados.0.smithi123.stdout:1877: finishing write tid 3 to smithi12313934-3 2020-01-16T00:14:24.842 INFO:tasks.rados.rados.0.smithi123.stdout:1872: done (15 left) 2020-01-16T00:14:24.842 INFO:tasks.rados.rados.0.smithi123.stdout:1873: done (14 left) 2020-01-16T00:14:24.842 INFO:tasks.rados.rados.0.smithi123.stdout:1874: done (13 left) 2020-01-16T00:14:24.842 INFO:tasks.rados.rados.0.smithi123.stdout:1875: done (12 left) 2020-01-16T00:14:24.842 INFO:tasks.rados.rados.0.smithi123.stdout:1876: done (11 left) 2020-01-16T00:14:24.843 INFO:tasks.rados.rados.0.smithi123.stdout:1888: read oid 6 snap -1 2020-01-16T00:14:24.843 INFO:tasks.rados.rados.0.smithi123.stdout:1888: expect (ObjNum 684 snap 0 seq_num 684) 2020-01-16T00:14:24.843 INFO:tasks.rados.rados.0.smithi123.stdout:1889: write (excl) oid 47 current snap is 0 2020-01-16T00:14:24.843 INFO:tasks.rados.rados.0.smithi123.stdout:1889: seq_num 703 ranges {615321=404523,1804606=457240,2755658=726225} 2020-01-16T00:14:24.852 INFO:tasks.rados.rados.0.smithi123.stdout:1889: writing smithi12313934-47 from 615321 to 1019844 tid 1 2020-01-16T00:14:24.862 INFO:tasks.rados.rados.0.smithi123.stdout:1889: writing smithi12313934-47 from 1804606 to 2261846 tid 2 2020-01-16T00:14:24.863 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:24.869 INFO:tasks.rados.rados.0.smithi123.stdout:1889: writing smithi12313934-47 from 2755658 to 3481883 tid 3 2020-01-16T00:14:24.869 INFO:tasks.rados.rados.0.smithi123.stdout:1890: write oid 36 current snap is 0 2020-01-16T00:14:24.869 INFO:tasks.rados.rados.0.smithi123.stdout:1890: seq_num 704 ranges {632128=479772,1746735=600898,3066728=653850} 2020-01-16T00:14:24.872 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-01-16T00:14:24.875 INFO:tasks.rados.rados.0.smithi123.stdout:1890: writing smithi12313934-36 from 632128 to 1111900 tid 1 2020-01-16T00:14:24.882 INFO:tasks.rados.rados.0.smithi123.stdout:1890: writing smithi12313934-36 from 1746735 to 2347633 tid 2 2020-01-16T00:14:24.893 INFO:tasks.rados.rados.0.smithi123.stdout:1890: writing smithi12313934-36 from 3066728 to 3720578 tid 3 2020-01-16T00:14:24.893 INFO:tasks.rados.rados.0.smithi123.stdout:1891: read oid 25 snap -1 2020-01-16T00:14:24.893 INFO:tasks.rados.rados.0.smithi123.stdout:1891: expect (ObjNum 682 snap 0 seq_num 682) 2020-01-16T00:14:24.893 INFO:tasks.rados.rados.0.smithi123.stdout:1892: read oid 37 snap -1 2020-01-16T00:14:24.894 INFO:tasks.rados.rados.0.smithi123.stdout:1892: expect (ObjNum 693 snap 0 seq_num 693) 2020-01-16T00:14:24.894 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:24.894 INFO:tasks.rados.rados.0.smithi123.stdout:1877: finishing write tid 5 to smithi12313934-3 2020-01-16T00:14:24.894 INFO:tasks.rados.rados.0.smithi123.stdout:1877: finishing write tid 6 to smithi12313934-3 2020-01-16T00:14:24.895 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 3 v 131 (ObjNum 697 snap 0 seq_num 697) dirty exists 2020-01-16T00:14:24.895 INFO:tasks.rados.rados.0.smithi123.stdout:1877: left oid 3 (ObjNum 697 snap 0 seq_num 697) 2020-01-16T00:14:24.895 INFO:tasks.rados.rados.0.smithi123.stdout:1878: finishing write tid 1 to smithi12313934-10 2020-01-16T00:14:24.895 INFO:tasks.rados.rados.0.smithi123.stdout:1877: done (15 left) 2020-01-16T00:14:24.896 INFO:tasks.rados.rados.0.smithi123.stdout:1893: read oid 17 snap -1 2020-01-16T00:14:24.896 INFO:tasks.rados.rados.0.smithi123.stdout:1893: expect (ObjNum 675 snap 0 seq_num 675) 2020-01-16T00:14:24.896 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:24.896 INFO:tasks.rados.rados.0.smithi123.stdout:1878: finishing write tid 2 to smithi12313934-10 2020-01-16T00:14:24.897 INFO:tasks.rados.rados.0.smithi123.stdout:1878: finishing write tid 3 to smithi12313934-10 2020-01-16T00:14:24.897 INFO:tasks.rados.rados.0.smithi123.stdout:1878: finishing write tid 5 to smithi12313934-10 2020-01-16T00:14:24.897 INFO:tasks.rados.rados.0.smithi123.stdout:1878: finishing write tid 6 to smithi12313934-10 2020-01-16T00:14:24.897 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 10 v 198 (ObjNum 698 snap 0 seq_num 698) dirty exists 2020-01-16T00:14:24.898 INFO:tasks.rados.rados.0.smithi123.stdout:1878: left oid 10 (ObjNum 698 snap 0 seq_num 698) 2020-01-16T00:14:24.898 INFO:tasks.rados.rados.0.smithi123.stdout:1879: finishing write tid 1 to smithi12313934-19 2020-01-16T00:14:24.898 INFO:tasks.rados.rados.0.smithi123.stdout:1879: finishing write tid 2 to smithi12313934-19 2020-01-16T00:14:24.898 INFO:tasks.rados.rados.0.smithi123.stdout:1878: done (15 left) 2020-01-16T00:14:24.899 INFO:tasks.rados.rados.0.smithi123.stdout:1894: write (excl) oid 42 current snap is 0 2020-01-16T00:14:24.899 INFO:tasks.rados.rados.0.smithi123.stdout:1894: seq_num 705 ranges {648935=555021,1805217=460909,2694151=465122,3817442=31493} 2020-01-16T00:14:24.900 INFO:tasks.rados.rados.0.smithi123.stdout:1894: writing smithi12313934-42 from 648935 to 1203956 tid 1 2020-01-16T00:14:24.904 INFO:tasks.rados.rados.0.smithi123.stdout:1894: writing smithi12313934-42 from 1805217 to 2266126 tid 2 2020-01-16T00:14:24.910 INFO:tasks.rados.rados.0.smithi123.stdout:1894: writing smithi12313934-42 from 2694151 to 3159273 tid 3 2020-01-16T00:14:24.914 INFO:tasks.rados.rados.0.smithi123.stdout:1894: writing smithi12313934-42 from 3817442 to 3848935 tid 4 2020-01-16T00:14:24.914 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:24.914 INFO:tasks.rados.rados.0.smithi123.stdout:1879: finishing write tid 3 to smithi12313934-19 2020-01-16T00:14:24.915 INFO:tasks.rados.rados.0.smithi123.stdout:1879: finishing write tid 5 to smithi12313934-19 2020-01-16T00:14:24.915 INFO:tasks.rados.rados.0.smithi123.stdout:1879: finishing write tid 6 to smithi12313934-19 2020-01-16T00:14:24.915 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 19 v 266 (ObjNum 699 snap 0 seq_num 699) dirty exists 2020-01-16T00:14:24.915 INFO:tasks.rados.rados.0.smithi123.stdout:1879: left oid 19 (ObjNum 699 snap 0 seq_num 699) 2020-01-16T00:14:24.915 INFO:tasks.rados.rados.0.smithi123.stdout:1880: finishing write tid 1 to smithi12313934-7 2020-01-16T00:14:24.915 INFO:tasks.rados.rados.0.smithi123.stdout:1879: done (15 left) 2020-01-16T00:14:24.915 INFO:tasks.rados.rados.0.smithi123.stdout:1895: write (excl) oid 30 current snap is 0 2020-01-16T00:14:24.916 INFO:tasks.rados.rados.0.smithi123.stdout:1895: seq_num 706 ranges {665742=630270,1863699=604567,3121574=676394} 2020-01-16T00:14:24.922 INFO:tasks.rados.rados.0.smithi123.stdout:1895: writing smithi12313934-30 from 665742 to 1296012 tid 1 2020-01-16T00:14:24.929 INFO:tasks.rados.rados.0.smithi123.stdout:1895: writing smithi12313934-30 from 1863699 to 2468266 tid 2 2020-01-16T00:14:24.936 INFO:tasks.rados.rados.0.smithi123.stdout:1895: writing smithi12313934-30 from 3121574 to 3797968 tid 3 2020-01-16T00:14:24.937 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:24.937 INFO:tasks.rados.rados.0.smithi123.stdout:1880: finishing write tid 2 to smithi12313934-7 2020-01-16T00:14:24.937 INFO:tasks.rados.rados.0.smithi123.stdout:1880: finishing write tid 3 to smithi12313934-7 2020-01-16T00:14:24.937 INFO:tasks.rados.rados.0.smithi123.stdout:1880: finishing write tid 5 to smithi12313934-7 2020-01-16T00:14:24.938 INFO:tasks.rados.rados.0.smithi123.stdout:1880: finishing write tid 6 to smithi12313934-7 2020-01-16T00:14:24.938 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 7 v 270 (ObjNum 700 snap 0 seq_num 700) dirty exists 2020-01-16T00:14:24.938 INFO:tasks.rados.rados.0.smithi123.stdout:1880: left oid 7 (ObjNum 700 snap 0 seq_num 700) 2020-01-16T00:14:24.938 INFO:tasks.rados.rados.0.smithi123.stdout:1881: finishing write tid 1 to smithi12313934-18 2020-01-16T00:14:24.939 INFO:tasks.rados.rados.0.smithi123.stdout:1881: finishing write tid 2 to smithi12313934-18 2020-01-16T00:14:24.939 INFO:tasks.rados.rados.0.smithi123.stdout:1881: finishing write tid 3 to smithi12313934-18 2020-01-16T00:14:24.939 INFO:tasks.rados.rados.0.smithi123.stdout:1881: finishing write tid 5 to smithi12313934-18 2020-01-16T00:14:24.939 INFO:tasks.rados.rados.0.smithi123.stdout:1881: finishing write tid 6 to smithi12313934-18 2020-01-16T00:14:24.939 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 18 v 369 (ObjNum 701 snap 0 seq_num 701) dirty exists 2020-01-16T00:14:24.939 INFO:tasks.rados.rados.0.smithi123.stdout:1881: left oid 18 (ObjNum 701 snap 0 seq_num 701) 2020-01-16T00:14:24.939 INFO:tasks.rados.rados.0.smithi123.stdout:1880: done (15 left) 2020-01-16T00:14:24.940 INFO:tasks.rados.rados.0.smithi123.stdout:1881: done (14 left) 2020-01-16T00:14:24.940 INFO:tasks.rados.rados.0.smithi123.stdout:1896: write oid 5 current snap is 0 2020-01-16T00:14:24.940 INFO:tasks.rados.rados.0.smithi123.stdout:1896: seq_num 707 ranges {682549=705519,1922181=464578,3148997=487666} 2020-01-16T00:14:24.942 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:24.942+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:24.944 INFO:tasks.rados.rados.0.smithi123.stdout:1896: writing smithi12313934-5 from 682549 to 1388068 tid 1 2020-01-16T00:14:24.949 INFO:tasks.rados.rados.0.smithi123.stdout:1896: writing smithi12313934-5 from 1922181 to 2386759 tid 2 2020-01-16T00:14:24.957 INFO:tasks.rados.rados.0.smithi123.stdout:1896: writing smithi12313934-5 from 3148997 to 3636663 tid 3 2020-01-16T00:14:24.957 INFO:tasks.rados.rados.0.smithi123.stdout:1897: read oid 35 snap -1 2020-01-16T00:14:24.958 INFO:tasks.rados.rados.0.smithi123.stdout:1897: expect (ObjNum 628 snap 0 seq_num 628) 2020-01-16T00:14:24.958 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:24.958 INFO:tasks.rados.rados.0.smithi123.stdout:1882: expect (ObjNum 655 snap 0 seq_num 655) 2020-01-16T00:14:24.976 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:24.986 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-01-16T00:14:25.043 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:25.040+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:25.052 INFO:tasks.rados.rados.0.smithi123.stdout:1884: finishing write tid 1 to smithi12313934-22 2020-01-16T00:14:25.052 INFO:tasks.rados.rados.0.smithi123.stdout:1884: finishing write tid 2 to smithi12313934-22 2020-01-16T00:14:25.053 INFO:tasks.rados.rados.0.smithi123.stdout:1882: done (15 left) 2020-01-16T00:14:25.053 INFO:tasks.rados.rados.0.smithi123.stdout:1883: done (14 left) 2020-01-16T00:14:25.053 INFO:tasks.rados.rados.0.smithi123.stdout:1898: write (excl) oid 13 current snap is 0 2020-01-16T00:14:25.053 INFO:tasks.rados.rados.0.smithi123.stdout:1898: seq_num 708 ranges {699356=497121,1980663=608236,3176420=415291} 2020-01-16T00:14:25.063 INFO:tasks.rados.rados.0.smithi123.stdout:1898: writing smithi12313934-13 from 699356 to 1196477 tid 1 2020-01-16T00:14:25.074 INFO:tasks.rados.rados.0.smithi123.stdout:1898: writing smithi12313934-13 from 1980663 to 2588899 tid 2 2020-01-16T00:14:25.079 INFO:tasks.rados.rados.0.smithi123.stdout:1898: writing smithi12313934-13 from 3176420 to 3591711 tid 3 2020-01-16T00:14:25.080 INFO:tasks.rados.rados.0.smithi123.stdout:1899: write (excl) oid 21 current snap is 0 2020-01-16T00:14:25.080 INFO:tasks.rados.rados.0.smithi123.stdout:1899: seq_num 709 ranges {716163=572370,2039145=468247,3203843=626563} 2020-01-16T00:14:25.087 INFO:tasks.rados.rados.0.smithi123.stdout:1899: writing smithi12313934-21 from 716163 to 1288533 tid 1 2020-01-16T00:14:25.094 INFO:teuthology.orchestra.run.smithi123.stderr:noscrub is unset 2020-01-16T00:14:25.095 INFO:tasks.rados.rados.0.smithi123.stdout:1899: writing smithi12313934-21 from 2039145 to 2507392 tid 2 2020-01-16T00:14:25.100 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:25.102 INFO:tasks.rados.rados.0.smithi123.stdout:1899: writing smithi12313934-21 from 3203843 to 3830406 tid 3 2020-01-16T00:14:25.102 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:25.102 INFO:tasks.rados.rados.0.smithi123.stdout:1887: done (15 left) 2020-01-16T00:14:25.102 INFO:tasks.rados.rados.0.smithi123.stdout:1900: write (excl) oid 44 current snap is 0 2020-01-16T00:14:25.103 INFO:tasks.rados.rados.0.smithi123.stdout:1900: seq_num 710 ranges {732970=647619,2097627=611905,3231266=437835} 2020-01-16T00:14:25.109 INFO:tasks.rados.rados.0.smithi123.stdout:1900: writing smithi12313934-44 from 732970 to 1380589 tid 1 2020-01-16T00:14:25.111 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-01-16T00:14:25.117 INFO:tasks.rados.rados.0.smithi123.stdout:1900: writing smithi12313934-44 from 2097627 to 2709532 tid 2 2020-01-16T00:14:25.123 INFO:tasks.rados.rados.0.smithi123.stdout:1900: writing smithi12313934-44 from 3231266 to 3669101 tid 3 2020-01-16T00:14:25.123 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:25.123 INFO:tasks.rados.rados.0.smithi123.stdout:1884: finishing write tid 3 to smithi12313934-22 2020-01-16T00:14:25.123 INFO:tasks.rados.rados.0.smithi123.stdout:1884: finishing write tid 5 to smithi12313934-22 2020-01-16T00:14:25.124 INFO:tasks.rados.rados.0.smithi123.stdout:1884: finishing write tid 6 to smithi12313934-22 2020-01-16T00:14:25.124 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 22 v 49 (ObjNum 702 snap 0 seq_num 702) dirty exists 2020-01-16T00:14:25.124 INFO:tasks.rados.rados.0.smithi123.stdout:1884: left oid 22 (ObjNum 702 snap 0 seq_num 702) 2020-01-16T00:14:25.124 INFO:tasks.rados.rados.0.smithi123.stdout:1885: expect (ObjNum 652 snap 0 seq_num 652) 2020-01-16T00:14:25.154 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:25.152+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:25.244 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:25.242+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:25.273 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:25.282 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-01-16T00:14:25.345 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:25.340+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:25.354 INFO:tasks.rados.rados.0.smithi123.stdout:1886: expect (ObjNum 672 snap 0 seq_num 672) 2020-01-16T00:14:25.385 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:25.394 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-01-16T00:14:25.426 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-01-16T00:14:25.471 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:25.470+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:25.550 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:25.546+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:25.551 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:25.557 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:14:25.558 INFO:tasks.rados.rados.0.smithi123.stdout:1888: expect (ObjNum 684 snap 0 seq_num 684) 2020-01-16T00:14:25.560 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-01-16T00:14:25.577 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:14:25.594 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:14:25.611 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:14:25.629 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:14:25.651 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:14:25.651 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:25.650+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:25.671 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:14:25.673 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-01-16T00:14:25.673 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-01-16T00:14:25.673 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-01-16T00:14:25.674 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-01-16T00:14:25.674 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-01-16T00:14:25.674 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENXIO: problem getting command descriptions from osd.5 2020-01-16T00:14:25.674 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: problem getting command descriptions from osd.5 2020-01-16T00:14:25.674 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-01-16T00:14:25.674 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-01-16T00:14:25.675 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:25.675 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:14:25.675 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:14:25.675 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:14:25.675 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:25.683 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-01-16T00:14:25.684 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-01-16T00:14:25.684 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-01-16T00:14:25.752 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:25.750+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:25.795 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:25.796 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:14:25.796 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:14:25.796 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:14:25.796 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.796 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.5.13:36 2.0 2:20e40cec:::smithi12313934-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 113152~1699840] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:14:25.796 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:14:25.797 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 107.96043219400001, 2020-01-16T00:14:25.797 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011736636, 2020-01-16T00:14:25.797 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:25.797 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:25.797 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:25.797 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.797 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:25.798 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:14:25.798 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:25.798 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.798 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.798 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:25.798 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:14:25.798 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999952 2020-01-16T00:14:25.799 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.799 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.799 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:25.799 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831934+0000", 2020-01-16T00:14:25.799 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001034325 2020-01-16T00:14:25.799 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.799 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.799 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:25.800 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.832968+0000", 2020-01-16T00:14:25.800 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.506e-05 2020-01-16T00:14:25.800 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.800 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.800 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:25.800 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833004+0000", 2020-01-16T00:14:25.800 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3406e-05 2020-01-16T00:14:25.801 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.801 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.801 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:25.801 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833017+0000", 2020-01-16T00:14:25.801 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015477000000000001 2020-01-16T00:14:25.801 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.802 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.802 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:25.802 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833172+0000", 2020-01-16T00:14:25.802 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033750700000000002 2020-01-16T00:14:25.803 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.803 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.803 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:25.803 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833509+0000", 2020-01-16T00:14:25.803 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00055861400000000003 2020-01-16T00:14:25.804 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.804 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.804 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:14:25.804 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.834068+0000", 2020-01-16T00:14:25.804 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.007474545 2020-01-16T00:14:25.804 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.804 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.805 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:25.805 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.841542+0000", 2020-01-16T00:14:25.805 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00056319499999999995 2020-01-16T00:14:25.805 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.806 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.806 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:25.806 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.842106+0000", 2020-01-16T00:14:25.806 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001371017 2020-01-16T00:14:25.806 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.806 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.806 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:25.807 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843477+0000", 2020-01-16T00:14:25.807 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000123938 2020-01-16T00:14:25.807 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.807 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.807 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:25.807 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843601+0000", 2020-01-16T00:14:25.807 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5012000000000006e-05 2020-01-16T00:14:25.808 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.808 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.808 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:25.808 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843676+0000", 2020-01-16T00:14:25.808 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011736636 2020-01-16T00:14:25.808 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.808 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:25.809 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.809 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.809 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.809 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1276 3.d 3:b1521a3e:::smithi12313934-25:head [read 0~3882384,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:14:25.809 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:14:25.809 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 102.717672624, 2020-01-16T00:14:25.809 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015839157999999999, 2020-01-16T00:14:25.809 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:25.810 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:25.810 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:25.810 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:25.810 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:25.810 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1276 2020-01-16T00:14:25.810 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.810 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:25.811 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.811 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:25.811 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:14:25.811 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:25.811 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.811 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.811 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:25.812 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:14:25.812 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:25.812 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.812 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.812 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:25.812 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:14:25.812 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5429999999999999e-06 2020-01-16T00:14:25.813 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.813 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.813 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:25.813 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074704+0000", 2020-01-16T00:14:25.813 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0839999999999992e-06 2020-01-16T00:14:25.813 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.813 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.814 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:25.814 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074712+0000", 2020-01-16T00:14:25.814 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.173e-06 2020-01-16T00:14:25.814 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.815 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.815 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:25.815 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074718+0000", 2020-01-16T00:14:25.815 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0083034700000000003 2020-01-16T00:14:25.815 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.816 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.816 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:25.816 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.083022+0000", 2020-01-16T00:14:25.816 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028456400000000001 2020-01-16T00:14:25.816 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.816 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.816 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:25.817 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.083306+0000", 2020-01-16T00:14:25.817 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0072312640000000003 2020-01-16T00:14:25.817 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.817 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.817 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:25.817 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090538+0000", 2020-01-16T00:14:25.817 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015839157999999999 2020-01-16T00:14:25.818 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.818 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:25.818 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.818 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.818 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.818 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1277 3.d 3:b1521a3e:::smithi12313934-25:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:14:25.818 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:14:25.819 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 102.717629034, 2020-01-16T00:14:25.819 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016646459999999998, 2020-01-16T00:14:25.819 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:25.819 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:25.819 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:25.819 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:25.819 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:25.819 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1277 2020-01-16T00:14:25.820 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.820 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:25.820 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.820 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:25.820 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:14:25.820 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:25.820 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.820 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.821 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:25.821 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:14:25.821 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:25.821 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.821 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.821 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:25.821 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074741+0000", 2020-01-16T00:14:25.823 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0776e-05 2020-01-16T00:14:25.823 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.823 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.823 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:25.824 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074752+0000", 2020-01-16T00:14:25.824 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7600000000000002e-06 2020-01-16T00:14:25.824 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.824 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.824 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:25.824 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074759+0000", 2020-01-16T00:14:25.824 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6840000000000001e-06 2020-01-16T00:14:25.824 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.825 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.825 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:25.825 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074765+0000", 2020-01-16T00:14:25.825 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015815216 2020-01-16T00:14:25.825 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.825 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.825 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:25.826 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090580+0000", 2020-01-16T00:14:25.826 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000182404 2020-01-16T00:14:25.826 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.826 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.826 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:25.826 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090763+0000", 2020-01-16T00:14:25.826 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00062579300000000003 2020-01-16T00:14:25.826 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.827 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.827 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:25.827 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091389+0000", 2020-01-16T00:14:25.827 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016646459999999998 2020-01-16T00:14:25.827 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.827 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:25.827 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.827 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.828 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.828 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1278 3.d 3:b1521a3e:::smithi12313934-25:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:14:25.828 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:14:25.828 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 102.71759932099999, 2020-01-16T00:14:25.828 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017994915, 2020-01-16T00:14:25.828 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:25.828 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:25.829 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:25.829 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:25.829 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:25.829 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1278 2020-01-16T00:14:25.829 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.829 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:25.830 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.830 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:25.830 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:14:25.830 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:25.830 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.830 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.830 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:25.830 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:14:25.831 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:25.831 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.831 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.831 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:25.831 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074771+0000", 2020-01-16T00:14:25.831 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4569999999999998e-06 2020-01-16T00:14:25.831 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.831 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.832 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:25.832 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074775+0000", 2020-01-16T00:14:25.832 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0330000000000003e-06 2020-01-16T00:14:25.832 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.832 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.832 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:25.833 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074782+0000", 2020-01-16T00:14:25.833 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9470000000000002e-06 2020-01-16T00:14:25.833 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.833 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.833 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:25.833 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074787+0000", 2020-01-16T00:14:25.833 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01666134 2020-01-16T00:14:25.834 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.834 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.834 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:25.834 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091449+0000", 2020-01-16T00:14:25.834 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000247731 2020-01-16T00:14:25.834 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.834 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.835 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:25.835 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091696+0000", 2020-01-16T00:14:25.835 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010703170000000001 2020-01-16T00:14:25.835 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.835 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.835 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:25.835 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.092767+0000", 2020-01-16T00:14:25.836 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017994915 2020-01-16T00:14:25.836 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.836 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:25.836 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.836 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.836 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.836 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2260 3.d 3:b76659c4:::smithi12313934-47:head [write 601750~535546] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:14:25.837 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:06.050577+0000", 2020-01-16T00:14:25.837 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 79.741794005000003, 2020-01-16T00:14:25.837 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012415381, 2020-01-16T00:14:25.837 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:25.837 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:25.837 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:25.837 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:25.838 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:25.838 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2260 2020-01-16T00:14:25.838 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.838 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:25.838 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.838 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:25.838 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.050577+0000", 2020-01-16T00:14:25.838 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:25.839 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.839 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.839 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:25.839 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.050577+0000", 2020-01-16T00:14:25.839 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:25.839 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.839 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.839 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:25.840 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.050574+0000", 2020-01-16T00:14:25.840 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001523487 2020-01-16T00:14:25.840 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.840 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.840 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:25.840 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052097+0000", 2020-01-16T00:14:25.840 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3697e-05 2020-01-16T00:14:25.840 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.841 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.841 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:25.841 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052121+0000", 2020-01-16T00:14:25.841 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0070000000000003e-06 2020-01-16T00:14:25.841 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.841 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.841 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:25.841 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052130+0000", 2020-01-16T00:14:25.842 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4559000000000003e-05 2020-01-16T00:14:25.842 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.842 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.842 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:25.842 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052194+0000", 2020-01-16T00:14:25.842 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000165878 2020-01-16T00:14:25.842 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.842 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.843 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:25.843 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052360+0000", 2020-01-16T00:14:25.843 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018825499999999999 2020-01-16T00:14:25.843 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.843 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.843 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:14:25.844 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052549+0000", 2020-01-16T00:14:25.844 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002761782 2020-01-16T00:14:25.844 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.844 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.844 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:25.844 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.055310+0000", 2020-01-16T00:14:25.844 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00060385500000000004 2020-01-16T00:14:25.844 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.845 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.845 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:25.845 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.055914+0000", 2020-01-16T00:14:25.845 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0069733809999999999 2020-01-16T00:14:25.845 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.845 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.845 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:25.846 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.062888+0000", 2020-01-16T00:14:25.846 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2884999999999999e-05 2020-01-16T00:14:25.846 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.846 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.846 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:25.846 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.062940+0000", 2020-01-16T00:14:25.846 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1996999999999997e-05 2020-01-16T00:14:25.847 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.847 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.847 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:25.847 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.062992+0000", 2020-01-16T00:14:25.847 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012415381 2020-01-16T00:14:25.847 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.848 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:25.848 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.848 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.848 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.848 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2295 3.d 3:b1521a3e:::smithi12313934-25:head [write 1763838~505140] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:14:25.848 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:06.660258+0000", 2020-01-16T00:14:25.849 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 79.132112778000007, 2020-01-16T00:14:25.849 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011681408000000001, 2020-01-16T00:14:25.849 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:25.849 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:25.849 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:25.849 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:25.849 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:25.850 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2295 2020-01-16T00:14:25.850 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.850 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:25.850 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.850 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:25.850 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660258+0000", 2020-01-16T00:14:25.850 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:25.851 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.851 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.851 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:25.851 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660258+0000", 2020-01-16T00:14:25.851 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:25.851 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.851 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.851 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:25.852 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660256+0000", 2020-01-16T00:14:25.852 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00066501699999999995 2020-01-16T00:14:25.852 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.852 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.852 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:25.852 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660921+0000", 2020-01-16T00:14:25.852 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7339000000000001e-05 2020-01-16T00:14:25.852 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.853 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.853 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:25.853 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660938+0000", 2020-01-16T00:14:25.853 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0219999999999993e-06 2020-01-16T00:14:25.853 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.853 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.854 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:25.854 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660947+0000", 2020-01-16T00:14:25.854 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00094237200000000002 2020-01-16T00:14:25.854 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.854 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.854 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:25.854 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.661890+0000", 2020-01-16T00:14:25.854 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000117649 2020-01-16T00:14:25.855 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.855 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.855 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:25.855 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662007+0000", 2020-01-16T00:14:25.855 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00060662599999999998 2020-01-16T00:14:25.855 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.855 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.855 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:25.856 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662614+0000", 2020-01-16T00:14:25.856 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028424899999999998 2020-01-16T00:14:25.856 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.856 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.856 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:25.856 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662898+0000", 2020-01-16T00:14:25.856 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00054904599999999997 2020-01-16T00:14:25.856 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.857 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.857 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:14:25.857 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.663447+0000", 2020-01-16T00:14:25.857 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0051479990000000003 2020-01-16T00:14:25.857 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.857 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.857 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:25.857 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.668595+0000", 2020-01-16T00:14:25.858 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00080881799999999999 2020-01-16T00:14:25.858 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.858 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.858 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:25.858 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.669404+0000", 2020-01-16T00:14:25.858 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0024495290000000002 2020-01-16T00:14:25.858 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.859 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.859 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:25.859 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671854+0000", 2020-01-16T00:14:25.859 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6635000000000003e-05 2020-01-16T00:14:25.859 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.859 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.859 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:25.859 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671900+0000", 2020-01-16T00:14:25.860 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9567999999999998e-05 2020-01-16T00:14:25.860 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.860 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.860 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:25.860 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671940+0000", 2020-01-16T00:14:25.860 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011681408000000001 2020-01-16T00:14:25.860 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.860 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:25.861 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.861 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.861 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.861 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.3.13:109 2.0 2:11fa79bc:::smithi12313934-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 331264~198656] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e42)", 2020-01-16T00:14:25.861 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:21.840075+0000", 2020-01-16T00:14:25.861 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 63.952296480999998, 2020-01-16T00:14:25.861 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011558677999999999, 2020-01-16T00:14:25.862 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:25.862 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:25.862 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:25.862 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.862 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:25.862 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840075+0000", 2020-01-16T00:14:25.862 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:25.862 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.862 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.863 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:25.863 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840075+0000", 2020-01-16T00:14:25.863 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:25.863 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.863 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.863 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:25.864 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840073+0000", 2020-01-16T00:14:25.864 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033576299999999999 2020-01-16T00:14:25.864 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.864 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.864 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:25.864 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840409+0000", 2020-01-16T00:14:25.864 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2082999999999998e-05 2020-01-16T00:14:25.864 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.865 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.865 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:25.865 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840431+0000", 2020-01-16T00:14:25.865 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4410000000000003e-06 2020-01-16T00:14:25.865 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.865 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.865 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:25.865 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840436+0000", 2020-01-16T00:14:25.866 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000101016 2020-01-16T00:14:25.866 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.866 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.866 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:25.866 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840537+0000", 2020-01-16T00:14:25.866 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000117639 2020-01-16T00:14:25.866 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.867 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.867 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:25.867 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840654+0000", 2020-01-16T00:14:25.867 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000196162 2020-01-16T00:14:25.867 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.867 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.867 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:14:25.867 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840850+0000", 2020-01-16T00:14:25.868 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003319092 2020-01-16T00:14:25.868 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.868 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.868 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:25.868 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.844169+0000", 2020-01-16T00:14:25.868 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011349229999999999 2020-01-16T00:14:25.868 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.868 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.869 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:25.869 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.845304+0000", 2020-01-16T00:14:25.869 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0062517670000000001 2020-01-16T00:14:25.869 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.869 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.869 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:25.869 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851556+0000", 2020-01-16T00:14:25.869 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4138000000000003e-05 2020-01-16T00:14:25.870 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.870 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.870 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:25.870 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851600+0000", 2020-01-16T00:14:25.870 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2993000000000001e-05 2020-01-16T00:14:25.870 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.870 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.870 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:25.871 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851633+0000", 2020-01-16T00:14:25.871 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011558677999999999 2020-01-16T00:14:25.871 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.871 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:25.871 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.871 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.871 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.871 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.3.13:110 2.0 2:11fa79bc:::smithi12313934-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 752640~650240] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e42)", 2020-01-16T00:14:25.872 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:21.840457+0000", 2020-01-16T00:14:25.872 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 63.951914189, 2020-01-16T00:14:25.872 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011329271, 2020-01-16T00:14:25.872 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:25.872 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:25.872 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:25.872 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.872 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:25.873 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840457+0000", 2020-01-16T00:14:25.873 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:25.873 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.873 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.873 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:25.873 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840457+0000", 2020-01-16T00:14:25.874 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:25.874 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.874 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.874 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:25.874 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840456+0000", 2020-01-16T00:14:25.874 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00062607899999999998 2020-01-16T00:14:25.874 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.874 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.875 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:25.875 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841082+0000", 2020-01-16T00:14:25.875 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1239999999999999e-06 2020-01-16T00:14:25.875 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.875 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.875 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:25.875 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841090+0000", 2020-01-16T00:14:25.875 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7129999999999999e-06 2020-01-16T00:14:25.876 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.876 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.876 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:25.876 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841093+0000", 2020-01-16T00:14:25.876 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00075997800000000002 2020-01-16T00:14:25.876 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.876 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.876 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:25.877 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841853+0000", 2020-01-16T00:14:25.877 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0690000000000003e-05 2020-01-16T00:14:25.877 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.877 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.877 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:25.877 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.841924+0000", 2020-01-16T00:14:25.877 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000115753 2020-01-16T00:14:25.878 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.878 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.878 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:14:25.878 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.842040+0000", 2020-01-16T00:14:25.878 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0021358890000000002 2020-01-16T00:14:25.878 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.878 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.878 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:25.879 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.844176+0000", 2020-01-16T00:14:25.879 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0042541870000000004 2020-01-16T00:14:25.879 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.879 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.879 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:25.879 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.848430+0000", 2020-01-16T00:14:25.879 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0032476929999999998 2020-01-16T00:14:25.879 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.880 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.880 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:25.880 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851677+0000", 2020-01-16T00:14:25.880 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0370000000000001e-05 2020-01-16T00:14:25.880 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.880 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.880 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:25.880 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851728+0000", 2020-01-16T00:14:25.881 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.834e-05 2020-01-16T00:14:25.881 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.881 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.881 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:25.881 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851786+0000", 2020-01-16T00:14:25.881 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011329271 2020-01-16T00:14:25.881 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.881 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:25.882 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.882 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.882 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.882 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2987 3.d 3:b3dd5d45:::smithi12313934-16:head [write 467327~633480] snapc 0=[] ondisk+write+known_if_redirected e42)", 2020-01-16T00:14:25.882 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:22.257904+0000", 2020-01-16T00:14:25.882 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 63.534466895000001, 2020-01-16T00:14:25.882 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012187450000000001, 2020-01-16T00:14:25.882 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:25.883 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:25.883 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:25.883 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:25.883 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:25.883 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2987 2020-01-16T00:14:25.883 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.883 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:25.884 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.884 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:25.884 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.257904+0000", 2020-01-16T00:14:25.884 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:25.884 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.884 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.884 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:25.885 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.257904+0000", 2020-01-16T00:14:25.885 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999933 2020-01-16T00:14:25.885 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.885 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.885 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:25.885 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.257898+0000", 2020-01-16T00:14:25.885 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00097099800000000004 2020-01-16T00:14:25.885 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.886 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.886 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:25.886 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.258869+0000", 2020-01-16T00:14:25.886 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7970000000000003e-05 2020-01-16T00:14:25.886 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.886 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.886 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:25.886 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.258917+0000", 2020-01-16T00:14:25.887 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5982999999999999e-05 2020-01-16T00:14:25.887 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.887 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.887 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:25.887 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.258933+0000", 2020-01-16T00:14:25.887 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00026239999999999998 2020-01-16T00:14:25.887 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.887 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.888 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:25.888 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.259195+0000", 2020-01-16T00:14:25.888 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024907400000000001 2020-01-16T00:14:25.888 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.888 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.888 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:25.888 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.259444+0000", 2020-01-16T00:14:25.888 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00034824999999999998 2020-01-16T00:14:25.889 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.889 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.889 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:25.889 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.259792+0000", 2020-01-16T00:14:25.889 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003251538 2020-01-16T00:14:25.889 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.889 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.889 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:25.890 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.263044+0000", 2020-01-16T00:14:25.890 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0068502880000000004 2020-01-16T00:14:25.890 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.890 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.890 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:25.890 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.269894+0000", 2020-01-16T00:14:25.890 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0945000000000006e-05 2020-01-16T00:14:25.890 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.891 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.891 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:25.891 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.269965+0000", 2020-01-16T00:14:25.891 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012667399999999999 2020-01-16T00:14:25.891 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.891 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.891 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:25.891 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.270092+0000", 2020-01-16T00:14:25.892 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012187450000000001 2020-01-16T00:14:25.892 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.892 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:25.892 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.892 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.892 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.892 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3273 3.d 3:b76659c4:::smithi12313934-47:head [read 0~3647152,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:25.892 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.728775+0000", 2020-01-16T00:14:25.893 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 55.063596341, 2020-01-16T00:14:25.893 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012922950000000001, 2020-01-16T00:14:25.893 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:25.893 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:25.893 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:25.893 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:25.893 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:25.893 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3273 2020-01-16T00:14:25.894 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.894 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:25.894 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.894 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:25.894 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728775+0000", 2020-01-16T00:14:25.894 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:25.895 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.895 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.895 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:25.895 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728775+0000", 2020-01-16T00:14:25.895 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:25.895 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.895 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.895 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:25.896 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728771+0000", 2020-01-16T00:14:25.896 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.5009000000000001e-05 2020-01-16T00:14:25.896 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.896 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.896 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:25.896 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728796+0000", 2020-01-16T00:14:25.896 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2357999999999998e-05 2020-01-16T00:14:25.896 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.897 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.897 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:25.897 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728819+0000", 2020-01-16T00:14:25.897 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.4800000000000007e-06 2020-01-16T00:14:25.897 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.897 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.897 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:25.897 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728828+0000", 2020-01-16T00:14:25.898 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000124824 2020-01-16T00:14:25.898 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.898 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.898 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:25.898 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728953+0000", 2020-01-16T00:14:25.898 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00044904899999999998 2020-01-16T00:14:25.898 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.898 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.899 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:25.899 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729402+0000", 2020-01-16T00:14:25.899 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012295706999999999 2020-01-16T00:14:25.899 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.899 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.899 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:25.899 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.741698+0000", 2020-01-16T00:14:25.899 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012922950000000001 2020-01-16T00:14:25.900 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.900 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:25.900 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.900 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.900 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.900 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3275 3.d 3:b76659c4:::smithi12313934-47:head [read 0~3647152,checksum] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:25.900 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.728933+0000", 2020-01-16T00:14:25.901 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 55.063438069, 2020-01-16T00:14:25.901 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01204874, 2020-01-16T00:14:25.901 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:25.901 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:25.901 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:25.901 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:25.901 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:25.901 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3275 2020-01-16T00:14:25.902 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.902 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:25.902 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.902 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:25.902 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728933+0000", 2020-01-16T00:14:25.902 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:25.902 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.902 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.903 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:25.903 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728933+0000", 2020-01-16T00:14:25.903 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:25.903 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.903 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.903 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:25.903 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728932+0000", 2020-01-16T00:14:25.903 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3569999999999998e-06 2020-01-16T00:14:25.904 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.904 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.904 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:25.904 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728940+0000", 2020-01-16T00:14:25.904 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0457e-05 2020-01-16T00:14:25.904 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.905 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.905 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:25.905 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728951+0000", 2020-01-16T00:14:25.905 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6799999999999993e-06 2020-01-16T00:14:25.905 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.905 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.905 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:25.905 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728959+0000", 2020-01-16T00:14:25.906 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010076307 2020-01-16T00:14:25.906 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.906 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.906 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:25.906 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.739035+0000", 2020-01-16T00:14:25.906 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024455400000000002 2020-01-16T00:14:25.906 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.906 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.907 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:25.907 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.739279+0000", 2020-01-16T00:14:25.907 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0017023660000000001 2020-01-16T00:14:25.907 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.907 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.907 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:25.907 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.740982+0000", 2020-01-16T00:14:25.908 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01204874 2020-01-16T00:14:25.908 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.908 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:25.908 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.908 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.908 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.908 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3285 3.d 3:bd9fc72f:::smithi12313934-5:head [read 0~3764801,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:25.908 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.729068+0000", 2020-01-16T00:14:25.909 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 55.063303277000003, 2020-01-16T00:14:25.909 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018537010999999999, 2020-01-16T00:14:25.909 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:25.909 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:25.909 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:25.909 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:25.909 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:25.909 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3285 2020-01-16T00:14:25.910 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.910 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:25.910 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.910 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:25.910 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729068+0000", 2020-01-16T00:14:25.910 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:25.910 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.910 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.911 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:25.911 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729068+0000", 2020-01-16T00:14:25.911 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:25.913 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.913 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.913 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:25.913 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729067+0000", 2020-01-16T00:14:25.914 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3339999999999996e-06 2020-01-16T00:14:25.914 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.914 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.914 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:25.914 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729075+0000", 2020-01-16T00:14:25.915 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4187e-05 2020-01-16T00:14:25.915 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.915 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.915 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:25.915 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729089+0000", 2020-01-16T00:14:25.915 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5600000000000004e-06 2020-01-16T00:14:25.915 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.915 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.916 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:25.916 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729093+0000", 2020-01-16T00:14:25.916 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011944309 2020-01-16T00:14:25.916 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.916 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.916 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:25.916 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.741038+0000", 2020-01-16T00:14:25.917 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016625799999999999 2020-01-16T00:14:25.917 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.917 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.917 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:25.917 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.741204+0000", 2020-01-16T00:14:25.917 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0064010120000000002 2020-01-16T00:14:25.917 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.917 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.918 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:25.918 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.747605+0000", 2020-01-16T00:14:25.918 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018537010999999999 2020-01-16T00:14:25.918 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.918 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:25.918 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.918 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.918 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.919 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3286 3.d 3:bd9fc72f:::smithi12313934-5:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:25.919 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.729099+0000", 2020-01-16T00:14:25.919 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 55.063272607000002, 2020-01-16T00:14:25.919 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018983897999999999, 2020-01-16T00:14:25.919 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:25.919 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:25.919 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:25.919 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:25.920 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:25.920 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3286 2020-01-16T00:14:25.920 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.920 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:25.920 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.920 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:25.920 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729099+0000", 2020-01-16T00:14:25.920 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:25.921 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.921 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.921 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:25.921 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729099+0000", 2020-01-16T00:14:25.921 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:25.921 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.921 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.922 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:25.922 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729098+0000", 2020-01-16T00:14:25.922 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8500000000000004e-06 2020-01-16T00:14:25.922 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.922 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.922 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:25.922 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729102+0000", 2020-01-16T00:14:25.922 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7329999999999997e-06 2020-01-16T00:14:25.923 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.923 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.923 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:25.923 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729108+0000", 2020-01-16T00:14:25.923 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.764e-06 2020-01-16T00:14:25.923 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.923 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.923 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:25.924 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729118+0000", 2020-01-16T00:14:25.924 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018524286000000001 2020-01-16T00:14:25.924 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.924 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.924 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:25.924 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.747642+0000", 2020-01-16T00:14:25.924 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9487999999999993e-05 2020-01-16T00:14:25.925 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.925 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.925 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:25.925 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.747742+0000", 2020-01-16T00:14:25.925 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000340429 2020-01-16T00:14:25.925 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.925 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.926 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:25.926 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.748082+0000", 2020-01-16T00:14:25.926 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018983897999999999 2020-01-16T00:14:25.926 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.926 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:25.926 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.926 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.926 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.927 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3287 3.d 3:bd9fc72f:::smithi12313934-5:head [read 0~3764801,checksum] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:25.927 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.729124+0000", 2020-01-16T00:14:25.927 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 55.063247468, 2020-01-16T00:14:25.927 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020552323000000001, 2020-01-16T00:14:25.927 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:25.927 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:25.927 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:25.928 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:25.928 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:25.928 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3287 2020-01-16T00:14:25.928 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.928 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:25.928 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.928 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:25.929 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729124+0000", 2020-01-16T00:14:25.929 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:25.929 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.929 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.929 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:25.929 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729124+0000", 2020-01-16T00:14:25.929 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:25.929 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.930 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.930 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:25.930 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729123+0000", 2020-01-16T00:14:25.930 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8239999999999996e-06 2020-01-16T00:14:25.930 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.930 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.930 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:25.930 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729129+0000", 2020-01-16T00:14:25.931 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6909999999999997e-06 2020-01-16T00:14:25.931 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.931 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.931 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:25.931 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729136+0000", 2020-01-16T00:14:25.931 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3329999999999997e-06 2020-01-16T00:14:25.931 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.932 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.932 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:25.932 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729145+0000", 2020-01-16T00:14:25.932 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018965065999999999 2020-01-16T00:14:25.932 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.932 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.932 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:25.932 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.748110+0000", 2020-01-16T00:14:25.933 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015781700000000001 2020-01-16T00:14:25.933 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.933 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.933 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:25.933 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.748268+0000", 2020-01-16T00:14:25.933 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014083800000000001 2020-01-16T00:14:25.933 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.933 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.934 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:25.934 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.749676+0000", 2020-01-16T00:14:25.934 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020552323000000001 2020-01-16T00:14:25.934 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.934 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:25.934 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.934 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.934 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.935 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3409 3.d 3:b3dd5d45:::smithi12313934-16:head [write 591712~786646,stat] snapc 0=[] ondisk+write+known_if_redirected e47)", 2020-01-16T00:14:25.935 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:34.100681+0000", 2020-01-16T00:14:25.935 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 51.691690328, 2020-01-16T00:14:25.935 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012248446, 2020-01-16T00:14:25.935 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:25.936 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:25.936 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:25.936 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:25.936 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:25.936 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3409 2020-01-16T00:14:25.936 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.936 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:25.936 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.937 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:25.937 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.100681+0000", 2020-01-16T00:14:25.937 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:25.937 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.937 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.937 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:25.937 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.100681+0000", 2020-01-16T00:14:25.937 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:14:25.938 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.938 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.938 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:25.938 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.100677+0000", 2020-01-16T00:14:25.938 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000998828 2020-01-16T00:14:25.938 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.938 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.938 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:25.939 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101676+0000", 2020-01-16T00:14:25.939 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5846999999999997e-05 2020-01-16T00:14:25.939 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.939 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.939 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:25.939 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101712+0000", 2020-01-16T00:14:25.939 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.364e-05 2020-01-16T00:14:25.939 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.940 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.940 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:25.940 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101725+0000", 2020-01-16T00:14:25.940 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015902700000000001 2020-01-16T00:14:25.940 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.940 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.940 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:25.940 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101884+0000", 2020-01-16T00:14:25.941 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0486e-05 2020-01-16T00:14:25.941 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.941 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.941 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op must wait for map", 2020-01-16T00:14:25.941 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101905+0000", 2020-01-16T00:14:25.941 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0038869009999999999 2020-01-16T00:14:25.941 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.941 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.942 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:25.942 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.105792+0000", 2020-01-16T00:14:25.942 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030748800000000002 2020-01-16T00:14:25.942 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.942 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.942 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:25.942 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.106099+0000", 2020-01-16T00:14:25.943 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042069299999999998 2020-01-16T00:14:25.943 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.943 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.943 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:25.943 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.106520+0000", 2020-01-16T00:14:25.943 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0052864560000000001 2020-01-16T00:14:25.943 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.943 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.944 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:25.944 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.111806+0000", 2020-01-16T00:14:25.944 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00080835800000000001 2020-01-16T00:14:25.944 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.944 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.944 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:25.944 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.112615+0000", 2020-01-16T00:14:25.944 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1490999999999997e-05 2020-01-16T00:14:25.945 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.945 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.945 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:25.945 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.112686+0000", 2020-01-16T00:14:25.945 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000243174 2020-01-16T00:14:25.945 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.946 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.946 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:25.946 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.112929+0000", 2020-01-16T00:14:25.946 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012248446 2020-01-16T00:14:25.946 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.946 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:25.946 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.946 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.947 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.947 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.0.13:31 2.0 2:1afbc8ac:::smithi12313934-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 4096~737792] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e50)", 2020-01-16T00:14:25.947 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:44.061109+0000", 2020-01-16T00:14:25.947 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 41.731262104000002, 2020-01-16T00:14:25.947 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012186723, 2020-01-16T00:14:25.947 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:25.947 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:25.948 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:25.948 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.948 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:25.948 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.061109+0000", 2020-01-16T00:14:25.948 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:25.948 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.948 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.948 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:25.949 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.061109+0000", 2020-01-16T00:14:25.949 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:25.949 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.949 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.949 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:25.949 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.061106+0000", 2020-01-16T00:14:25.949 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012349870000000001 2020-01-16T00:14:25.949 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.950 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.950 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:25.950 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062341+0000", 2020-01-16T00:14:25.950 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0235999999999996e-05 2020-01-16T00:14:25.950 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.950 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.950 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:25.950 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062411+0000", 2020-01-16T00:14:25.951 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7078000000000001e-05 2020-01-16T00:14:25.951 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.951 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.951 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:25.951 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062428+0000", 2020-01-16T00:14:25.951 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2158999999999996e-05 2020-01-16T00:14:25.951 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.952 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.952 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:25.952 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062520+0000", 2020-01-16T00:14:25.952 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011203499999999999 2020-01-16T00:14:25.952 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.952 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.952 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:25.952 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062632+0000", 2020-01-16T00:14:25.953 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017238400000000001 2020-01-16T00:14:25.953 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.953 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.953 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:14:25.953 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062805+0000", 2020-01-16T00:14:25.953 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010264989 2020-01-16T00:14:25.953 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.953 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.954 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:25.954 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073070+0000", 2020-01-16T00:14:25.954 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0533000000000001e-05 2020-01-16T00:14:25.954 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.954 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.954 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:25.954 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073090+0000", 2020-01-16T00:14:25.954 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2309000000000002e-05 2020-01-16T00:14:25.955 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.955 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.955 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:25.955 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073163+0000", 2020-01-16T00:14:25.955 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3035999999999997e-05 2020-01-16T00:14:25.955 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.956 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.956 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:25.956 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073236+0000", 2020-01-16T00:14:25.956 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0056999999999998e-05 2020-01-16T00:14:25.956 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.956 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.956 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:25.957 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073296+0000", 2020-01-16T00:14:25.957 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012186723 2020-01-16T00:14:25.957 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.957 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:25.957 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.957 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.957 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.957 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.0.13:32 2.0 2:1afbc8ac:::smithi12313934-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 916992~596480] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e50)", 2020-01-16T00:14:25.958 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:44.062464+0000", 2020-01-16T00:14:25.958 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 41.729907318999999, 2020-01-16T00:14:25.958 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011316405999999999, 2020-01-16T00:14:25.958 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:25.958 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:25.958 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:25.958 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.958 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:25.959 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062464+0000", 2020-01-16T00:14:25.959 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:25.959 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.959 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.959 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:25.959 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062464+0000", 2020-01-16T00:14:25.959 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:25.959 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.960 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.960 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:25.960 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062462+0000", 2020-01-16T00:14:25.960 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00073330900000000002 2020-01-16T00:14:25.960 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.960 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.960 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:25.960 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063196+0000", 2020-01-16T00:14:25.961 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5330000000000001e-05 2020-01-16T00:14:25.961 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.961 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.961 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:25.961 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063211+0000", 2020-01-16T00:14:25.961 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5840000000000001e-06 2020-01-16T00:14:25.961 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.962 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.962 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:25.962 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063221+0000", 2020-01-16T00:14:25.962 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001786915 2020-01-16T00:14:25.962 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.962 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.962 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:25.962 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.065007+0000", 2020-01-16T00:14:25.963 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00021770000000000001 2020-01-16T00:14:25.963 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.963 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.963 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:25.963 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.065225+0000", 2020-01-16T00:14:25.963 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00032623 2020-01-16T00:14:25.963 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.963 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.964 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:14:25.964 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.065551+0000", 2020-01-16T00:14:25.964 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0078227499999999998 2020-01-16T00:14:25.964 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.964 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.964 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:25.964 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073374+0000", 2020-01-16T00:14:25.964 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.717e-05 2020-01-16T00:14:25.965 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.965 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.965 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:25.965 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073391+0000", 2020-01-16T00:14:25.965 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011493999999999999 2020-01-16T00:14:25.965 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.965 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.966 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:25.966 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073506+0000", 2020-01-16T00:14:25.966 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000159281 2020-01-16T00:14:25.966 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.966 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.966 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:25.967 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073666+0000", 2020-01-16T00:14:25.967 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000114675 2020-01-16T00:14:25.967 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.967 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.967 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:25.967 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073780+0000", 2020-01-16T00:14:25.967 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011316405999999999 2020-01-16T00:14:25.967 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.968 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:25.968 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.968 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.968 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.968 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.0.13:33 2.0 2:1afbc8ac:::smithi12313934-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 1541120~378368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e50)", 2020-01-16T00:14:25.968 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:44.063236+0000", 2020-01-16T00:14:25.968 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 41.72913544, 2020-01-16T00:14:25.968 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015290962999999999, 2020-01-16T00:14:25.969 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:25.969 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:25.969 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:25.969 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.969 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:25.969 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063236+0000", 2020-01-16T00:14:25.969 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:25.969 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.970 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.970 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:25.970 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063236+0000", 2020-01-16T00:14:25.970 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:25.970 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.970 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.970 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:25.970 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063234+0000", 2020-01-16T00:14:25.971 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000500702 2020-01-16T00:14:25.971 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.971 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.971 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:25.971 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063735+0000", 2020-01-16T00:14:25.971 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4734e-05 2020-01-16T00:14:25.971 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.971 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.972 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:25.972 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063750+0000", 2020-01-16T00:14:25.972 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.382e-05 2020-01-16T00:14:25.972 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.972 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.972 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:25.972 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063764+0000", 2020-01-16T00:14:25.972 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0059030799999999998 2020-01-16T00:14:25.973 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.973 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.973 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:25.973 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.069667+0000", 2020-01-16T00:14:25.973 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3961999999999994e-05 2020-01-16T00:14:25.973 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.973 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.973 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:25.974 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.069741+0000", 2020-01-16T00:14:25.974 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000114931 2020-01-16T00:14:25.974 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.974 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.974 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:14:25.974 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.069856+0000", 2020-01-16T00:14:25.974 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0039948270000000003 2020-01-16T00:14:25.975 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.975 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.975 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:25.975 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073850+0000", 2020-01-16T00:14:25.975 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0016459140000000001 2020-01-16T00:14:25.975 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.975 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.975 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:25.976 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.075496+0000", 2020-01-16T00:14:25.976 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002957997 2020-01-16T00:14:25.976 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.976 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.976 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:25.976 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.078454+0000", 2020-01-16T00:14:25.977 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9190999999999998e-05 2020-01-16T00:14:25.977 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.977 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.977 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:25.977 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.078493+0000", 2020-01-16T00:14:25.977 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3176999999999998e-05 2020-01-16T00:14:25.977 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.978 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.978 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:25.978 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.078527+0000", 2020-01-16T00:14:25.978 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015290962999999999 2020-01-16T00:14:25.978 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.978 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:25.978 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.978 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.979 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.979 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4641 3.d 3:b1521a3e:::smithi12313934-25:head [read 0~3596797,checksum] snapc 0=[] ondisk+read+known_if_redirected e57)", 2020-01-16T00:14:25.979 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:04.086675+0000", 2020-01-16T00:14:25.979 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 21.705696383999999, 2020-01-16T00:14:25.979 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011822397, 2020-01-16T00:14:25.979 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:25.979 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:25.979 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:25.980 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:25.980 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:25.980 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4641 2020-01-16T00:14:25.980 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.980 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:25.980 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.980 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:25.980 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.086675+0000", 2020-01-16T00:14:25.981 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:25.981 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.981 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.981 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:25.981 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.086675+0000", 2020-01-16T00:14:25.981 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:25.981 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.982 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.982 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:25.982 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.086674+0000", 2020-01-16T00:14:25.982 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0809999999999998e-06 2020-01-16T00:14:25.982 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.982 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.982 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:25.982 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.086678+0000", 2020-01-16T00:14:25.983 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4889999999999997e-06 2020-01-16T00:14:25.983 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.983 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.983 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:25.983 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.086682+0000", 2020-01-16T00:14:25.983 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9620000000000002e-06 2020-01-16T00:14:25.983 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.983 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.984 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:25.984 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.086685+0000", 2020-01-16T00:14:25.984 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010215452 2020-01-16T00:14:25.984 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.984 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.984 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:25.984 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.096900+0000", 2020-01-16T00:14:25.984 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00021274299999999999 2020-01-16T00:14:25.985 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.985 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.985 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:25.985 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.097113+0000", 2020-01-16T00:14:25.985 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0013839480000000001 2020-01-16T00:14:25.985 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.985 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.986 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:25.986 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.098497+0000", 2020-01-16T00:14:25.986 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011822397 2020-01-16T00:14:25.986 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.986 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:25.986 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.987 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.987 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.987 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4737 3.d 3:b76659c4:::smithi12313934-47:head [read 0~3243850,checksum] snapc 0=[] ondisk+read+known_if_redirected e58)", 2020-01-16T00:14:25.987 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:08.134998+0000", 2020-01-16T00:14:25.987 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 17.657373509999999, 2020-01-16T00:14:25.987 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011528518, 2020-01-16T00:14:25.987 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:25.987 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:25.988 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:25.988 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:25.988 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:25.988 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4737 2020-01-16T00:14:25.988 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.988 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:25.988 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.988 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:25.989 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:08.134998+0000", 2020-01-16T00:14:25.989 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:25.989 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.989 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.989 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:25.989 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:08.134998+0000", 2020-01-16T00:14:25.989 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:25.990 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.990 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.990 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:25.990 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:08.134997+0000", 2020-01-16T00:14:25.990 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0459999999999999e-06 2020-01-16T00:14:25.990 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.990 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.990 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:25.991 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:08.135001+0000", 2020-01-16T00:14:25.991 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2469999999999999e-06 2020-01-16T00:14:25.991 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.991 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.991 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:25.991 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:08.135006+0000", 2020-01-16T00:14:25.991 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.065e-06 2020-01-16T00:14:25.991 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.992 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.992 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:25.992 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:08.135010+0000", 2020-01-16T00:14:25.992 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0095733439999999993 2020-01-16T00:14:25.992 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.992 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.992 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:25.992 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:08.144584+0000", 2020-01-16T00:14:25.993 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000272623 2020-01-16T00:14:25.993 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.993 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.993 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:25.993 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:08.144856+0000", 2020-01-16T00:14:25.993 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001669769 2020-01-16T00:14:25.993 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:25.993 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:25.994 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:25.994 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:08.146526+0000", 2020-01-16T00:14:25.994 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011528518 2020-01-16T00:14:25.994 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.994 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:25.994 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.994 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:25.994 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:25.995 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:25.997 INFO:tasks.rados.rados.0.smithi123.stdout:1889: finishing write tid 1 to smithi12313934-47 2020-01-16T00:14:25.997 INFO:tasks.rados.rados.0.smithi123.stdout:1889: finishing write tid 2 to smithi12313934-47 2020-01-16T00:14:25.997 INFO:tasks.rados.rados.0.smithi123.stdout:1889: finishing write tid 3 to smithi12313934-47 2020-01-16T00:14:25.998 INFO:tasks.rados.rados.0.smithi123.stdout:1889: finishing write tid 5 to smithi12313934-47 2020-01-16T00:14:25.998 INFO:tasks.rados.rados.0.smithi123.stdout:1889: finishing write tid 6 to smithi12313934-47 2020-01-16T00:14:25.998 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 47 v 297 (ObjNum 703 snap 0 seq_num 703) dirty exists 2020-01-16T00:14:25.998 INFO:tasks.rados.rados.0.smithi123.stdout:1889: left oid 47 (ObjNum 703 snap 0 seq_num 703) 2020-01-16T00:14:25.998 INFO:tasks.rados.rados.0.smithi123.stdout:1890: finishing write tid 1 to smithi12313934-36 2020-01-16T00:14:25.998 INFO:tasks.rados.rados.0.smithi123.stdout:1890: finishing write tid 2 to smithi12313934-36 2020-01-16T00:14:25.998 INFO:tasks.rados.rados.0.smithi123.stdout:1890: finishing write tid 3 to smithi12313934-36 2020-01-16T00:14:25.999 INFO:tasks.rados.rados.0.smithi123.stdout:1890: finishing write tid 5 to smithi12313934-36 2020-01-16T00:14:25.999 INFO:tasks.rados.rados.0.smithi123.stdout:1884: done (15 left) 2020-01-16T00:14:25.999 INFO:tasks.rados.rados.0.smithi123.stdout:1885: done (14 left) 2020-01-16T00:14:25.999 INFO:tasks.rados.rados.0.smithi123.stdout:1886: done (13 left) 2020-01-16T00:14:25.999 INFO:tasks.rados.rados.0.smithi123.stdout:1888: done (12 left) 2020-01-16T00:14:25.999 INFO:tasks.rados.rados.0.smithi123.stdout:1889: done (11 left) 2020-01-16T00:14:25.999 INFO:tasks.rados.rados.0.smithi123.stdout:1901: write oid 22 current snap is 0 2020-01-16T00:14:25.999 INFO:tasks.rados.rados.0.smithi123.stdout:1901: seq_num 711 ranges {749777=722868,2156109=471916,3258689=649107} 2020-01-16T00:14:26.000 INFO:tasks.rados.rados.0.smithi123.stdout:1901: writing smithi12313934-22 from 749777 to 1472645 tid 1 2020-01-16T00:14:26.000 INFO:tasks.rados.rados.0.smithi123.stdout:1901: writing smithi12313934-22 from 2156109 to 2628025 tid 2 2020-01-16T00:14:26.000 INFO:tasks.rados.rados.0.smithi123.stdout:1901: writing smithi12313934-22 from 3258689 to 3907796 tid 3 2020-01-16T00:14:26.000 INFO:tasks.rados.rados.0.smithi123.stdout:1902: read oid 12 snap -1 2020-01-16T00:14:26.000 INFO:tasks.rados.rados.0.smithi123.stdout:1902: expect (ObjNum 655 snap 0 seq_num 655) 2020-01-16T00:14:26.000 INFO:tasks.rados.rados.0.smithi123.stdout:1903: delete oid 23 current snap is 0 2020-01-16T00:14:26.000 INFO:tasks.rados.rados.0.smithi123.stdout:1890: finishing write tid 6 to smithi12313934-36 2020-01-16T00:14:26.000 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 36 v 128 (ObjNum 704 snap 0 seq_num 704) dirty exists 2020-01-16T00:14:26.001 INFO:tasks.rados.rados.0.smithi123.stdout:1890: left oid 36 (ObjNum 704 snap 0 seq_num 704) 2020-01-16T00:14:26.001 INFO:tasks.rados.rados.0.smithi123.stdout:1893: expect (ObjNum 675 snap 0 seq_num 675) 2020-01-16T00:14:26.001 INFO:tasks.rados.rados.0.smithi123.stdout:1892: expect (ObjNum 693 snap 0 seq_num 693) 2020-01-16T00:14:26.001 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:25.994+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:26.001 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-01-16T00:14:26.097 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:26.094+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:26.121 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:26.121 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:14:26.121 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:14:26.121 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:26.131 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-01-16T00:14:26.198 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:26.196+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:26.221 INFO:tasks.rados.rados.0.smithi123.stdout:1894: finishing write tid 1 to smithi12313934-42 2020-01-16T00:14:26.222 INFO:tasks.rados.rados.0.smithi123.stdout:1891: expect (ObjNum 682 snap 0 seq_num 682) 2020-01-16T00:14:26.242 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:26.242 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:14:26.242 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:14:26.242 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:14:26.243 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:26.252 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-01-16T00:14:26.338 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:26.336+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:26.399 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:26.396+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:26.409 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:26.410 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:14:26.410 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:14:26.410 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:14:26.410 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.411 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1657 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-01-16T00:14:26.411 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.718495+0000", 2020-01-16T00:14:26.411 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 93.687689798999997, 2020-01-16T00:14:26.411 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014299630000000001, 2020-01-16T00:14:26.411 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:26.411 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:26.411 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:26.412 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:26.412 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:26.412 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1657 2020-01-16T00:14:26.412 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.412 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:26.412 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.412 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:26.413 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718495+0000", 2020-01-16T00:14:26.413 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:26.413 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.413 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.413 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:26.413 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718495+0000", 2020-01-16T00:14:26.413 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:26.414 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.414 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.414 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:26.414 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718494+0000", 2020-01-16T00:14:26.414 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8860000000000003e-06 2020-01-16T00:14:26.414 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.415 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.415 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:26.415 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718499+0000", 2020-01-16T00:14:26.415 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9786000000000002e-05 2020-01-16T00:14:26.415 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.415 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.415 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:26.416 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718539+0000", 2020-01-16T00:14:26.416 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1869999999999999e-05 2020-01-16T00:14:26.416 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.416 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.416 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:26.416 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718561+0000", 2020-01-16T00:14:26.416 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000172206 2020-01-16T00:14:26.416 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.417 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.417 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.417 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718733+0000", 2020-01-16T00:14:26.417 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0662999999999998e-05 2020-01-16T00:14:26.417 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.417 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.417 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:26.418 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718793+0000", 2020-01-16T00:14:26.418 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.006217958 2020-01-16T00:14:26.418 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.418 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.418 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.418 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.725011+0000", 2020-01-16T00:14:26.418 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1625999999999996e-05 2020-01-16T00:14:26.419 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.419 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.419 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:26.419 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.725073+0000", 2020-01-16T00:14:26.419 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0063923119999999998 2020-01-16T00:14:26.419 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.419 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.419 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.420 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731465+0000", 2020-01-16T00:14:26.420 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2534000000000005e-05 2020-01-16T00:14:26.420 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.420 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.420 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:26.420 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731558+0000", 2020-01-16T00:14:26.420 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010238809999999999 2020-01-16T00:14:26.421 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.421 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.421 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.421 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732582+0000", 2020-01-16T00:14:26.421 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000100918 2020-01-16T00:14:26.421 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.422 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.422 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:26.422 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732683+0000", 2020-01-16T00:14:26.422 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000112064 2020-01-16T00:14:26.422 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.422 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.423 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:26.423 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732795+0000", 2020-01-16T00:14:26.423 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014299630000000001 2020-01-16T00:14:26.423 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.423 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:26.424 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.424 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.424 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.425 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2791 3.a 3:54f2c117:::smithi12313934-41:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e41)", 2020-01-16T00:14:26.425 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:17.213167+0000", 2020-01-16T00:14:26.425 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 69.193018105999997, 2020-01-16T00:14:26.425 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016024037000000001, 2020-01-16T00:14:26.425 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:26.426 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:26.426 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:26.426 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:26.426 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:26.426 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2791 2020-01-16T00:14:26.427 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.427 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:26.427 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.427 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:26.427 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213167+0000", 2020-01-16T00:14:26.427 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:26.428 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.428 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.428 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:26.428 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213167+0000", 2020-01-16T00:14:26.428 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:26.429 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.429 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.429 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:26.429 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213166+0000", 2020-01-16T00:14:26.430 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6030000000000004e-06 2020-01-16T00:14:26.430 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.430 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.430 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:26.431 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213172+0000", 2020-01-16T00:14:26.431 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0929999999999997e-06 2020-01-16T00:14:26.431 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.431 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.431 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:26.432 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213179+0000", 2020-01-16T00:14:26.432 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4299999999999997e-06 2020-01-16T00:14:26.432 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.432 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.432 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:26.433 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213184+0000", 2020-01-16T00:14:26.433 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012106695000000001 2020-01-16T00:14:26.433 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.433 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.434 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.434 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.225291+0000", 2020-01-16T00:14:26.434 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020863699999999999 2020-01-16T00:14:26.434 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.434 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.434 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:26.435 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.225499+0000", 2020-01-16T00:14:26.435 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0036913050000000002 2020-01-16T00:14:26.435 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.435 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.435 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:26.435 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.229191+0000", 2020-01-16T00:14:26.435 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016024037000000001 2020-01-16T00:14:26.436 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.436 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:26.436 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.436 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.436 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.436 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2792 3.a 3:54f2c117:::smithi12313934-41:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e41)", 2020-01-16T00:14:26.436 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:17.213190+0000", 2020-01-16T00:14:26.437 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 69.192994631999994, 2020-01-16T00:14:26.437 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017165949, 2020-01-16T00:14:26.437 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:26.437 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:26.437 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:26.437 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:26.437 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:26.438 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2792 2020-01-16T00:14:26.438 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.438 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:26.438 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.438 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:26.438 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213190+0000", 2020-01-16T00:14:26.438 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:26.439 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.439 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.439 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:26.439 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213190+0000", 2020-01-16T00:14:26.439 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:26.439 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.439 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.439 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:26.440 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213189+0000", 2020-01-16T00:14:26.440 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2699999999999998e-06 2020-01-16T00:14:26.440 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.440 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.440 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:26.440 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213194+0000", 2020-01-16T00:14:26.440 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8600000000000004e-06 2020-01-16T00:14:26.441 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.441 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.441 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:26.441 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213201+0000", 2020-01-16T00:14:26.441 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8500000000000002e-06 2020-01-16T00:14:26.441 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.441 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.442 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:26.442 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213205+0000", 2020-01-16T00:14:26.442 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016028924999999999 2020-01-16T00:14:26.442 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.442 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.442 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.442 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.229234+0000", 2020-01-16T00:14:26.442 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019773999999999999 2020-01-16T00:14:26.443 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.443 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.443 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:26.443 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.229432+0000", 2020-01-16T00:14:26.443 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00092414400000000001 2020-01-16T00:14:26.443 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.443 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.444 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:26.444 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.230356+0000", 2020-01-16T00:14:26.444 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017165949 2020-01-16T00:14:26.444 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.444 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:26.444 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.444 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.445 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.445 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2793 3.a 3:54f2c117:::smithi12313934-41:head [read 0~3210170,checksum] snapc 0=[] ondisk+read+known_if_redirected e41)", 2020-01-16T00:14:26.445 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:17.213211+0000", 2020-01-16T00:14:26.445 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 69.192974086000007, 2020-01-16T00:14:26.445 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019069413, 2020-01-16T00:14:26.446 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:26.446 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:26.446 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:26.446 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:26.446 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:26.446 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2793 2020-01-16T00:14:26.446 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.446 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:26.447 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.447 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:26.447 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213211+0000", 2020-01-16T00:14:26.447 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:26.447 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.447 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.447 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:26.448 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213211+0000", 2020-01-16T00:14:26.448 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:26.448 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.448 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.448 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:26.448 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213210+0000", 2020-01-16T00:14:26.448 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.66e-06 2020-01-16T00:14:26.448 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.449 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.449 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:26.449 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213216+0000", 2020-01-16T00:14:26.449 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.9469999999999997e-06 2020-01-16T00:14:26.449 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.449 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.449 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:26.450 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213223+0000", 2020-01-16T00:14:26.450 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9259999999999999e-06 2020-01-16T00:14:26.450 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.450 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.450 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:26.450 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213228+0000", 2020-01-16T00:14:26.450 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017171046999999998 2020-01-16T00:14:26.450 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.451 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.451 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.451 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.230399+0000", 2020-01-16T00:14:26.451 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000189932 2020-01-16T00:14:26.451 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.452 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.452 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:26.452 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.230589+0000", 2020-01-16T00:14:26.452 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0016916539999999999 2020-01-16T00:14:26.452 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.452 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.452 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:26.452 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.232280+0000", 2020-01-16T00:14:26.453 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019069413 2020-01-16T00:14:26.453 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.453 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:26.453 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.453 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.453 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.453 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3120 3.a 3:5069665e:::smithi12313934-2:head [write 1845146~524707] snapc 0=[] ondisk+write+known_if_redirected e44)", 2020-01-16T00:14:26.454 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.262852+0000", 2020-01-16T00:14:26.454 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 61.143332903000001, 2020-01-16T00:14:26.454 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014991175000000001, 2020-01-16T00:14:26.454 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:26.454 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:26.454 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:26.454 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:26.454 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:26.455 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3120 2020-01-16T00:14:26.455 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.455 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:26.455 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.455 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:26.456 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.262852+0000", 2020-01-16T00:14:26.456 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:26.456 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.456 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.456 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:26.456 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.262852+0000", 2020-01-16T00:14:26.456 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:26.457 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.457 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.457 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:26.457 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.262849+0000", 2020-01-16T00:14:26.457 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00058469900000000005 2020-01-16T00:14:26.457 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.457 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.458 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:26.458 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263434+0000", 2020-01-16T00:14:26.458 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.6650000000000001e-05 2020-01-16T00:14:26.458 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.458 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.458 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:26.458 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263461+0000", 2020-01-16T00:14:26.458 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3478e-05 2020-01-16T00:14:26.459 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.459 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.459 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:26.459 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263474+0000", 2020-01-16T00:14:26.459 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7824000000000002e-05 2020-01-16T00:14:26.459 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.459 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.460 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.460 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263522+0000", 2020-01-16T00:14:26.460 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001365464 2020-01-16T00:14:26.460 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.460 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.460 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:26.460 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.264888+0000", 2020-01-16T00:14:26.460 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0081368250000000003 2020-01-16T00:14:26.461 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.461 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.461 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.461 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.273024+0000", 2020-01-16T00:14:26.461 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6130000000000006e-05 2020-01-16T00:14:26.461 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.461 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.462 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:26.462 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.273090+0000", 2020-01-16T00:14:26.462 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000207507 2020-01-16T00:14:26.462 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.462 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.462 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:26.462 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.273298+0000", 2020-01-16T00:14:26.463 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0035649980000000002 2020-01-16T00:14:26.463 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.463 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.463 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:26.463 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276863+0000", 2020-01-16T00:14:26.463 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.1459999999999995e-05 2020-01-16T00:14:26.463 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.464 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.464 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:26.464 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276954+0000", 2020-01-16T00:14:26.464 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00076363899999999998 2020-01-16T00:14:26.464 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.464 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.464 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:26.464 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277718+0000", 2020-01-16T00:14:26.465 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0154999999999995e-05 2020-01-16T00:14:26.465 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.465 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.465 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:26.465 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277788+0000", 2020-01-16T00:14:26.465 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4914000000000003e-05 2020-01-16T00:14:26.466 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.466 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.466 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:26.466 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277843+0000", 2020-01-16T00:14:26.466 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014991175000000001 2020-01-16T00:14:26.466 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.466 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:26.467 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.467 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.467 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.467 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3121 3.a 3:5069665e:::smithi12313934-2:head [setxattr _header (55) in=7b,truncate 2369853] snapc 0=[] ondisk+write+known_if_redirected e44)", 2020-01-16T00:14:26.467 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.263494+0000", 2020-01-16T00:14:26.467 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 61.142690614999999, 2020-01-16T00:14:26.467 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014559274000000001, 2020-01-16T00:14:26.468 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:26.468 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:26.468 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:26.468 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:26.468 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:26.468 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3121 2020-01-16T00:14:26.468 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.468 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:26.469 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.469 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:26.469 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263494+0000", 2020-01-16T00:14:26.469 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:26.469 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.469 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.470 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:26.470 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263494+0000", 2020-01-16T00:14:26.470 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:26.470 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.470 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.470 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:26.470 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263493+0000", 2020-01-16T00:14:26.471 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3359999999999995e-06 2020-01-16T00:14:26.471 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.471 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.471 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:26.471 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263501+0000", 2020-01-16T00:14:26.471 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4100000000000002e-06 2020-01-16T00:14:26.471 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.471 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.472 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:26.472 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263509+0000", 2020-01-16T00:14:26.472 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3839999999999996e-06 2020-01-16T00:14:26.472 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.472 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.472 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:26.472 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263514+0000", 2020-01-16T00:14:26.472 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001430295 2020-01-16T00:14:26.473 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.473 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.473 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.473 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.264944+0000", 2020-01-16T00:14:26.473 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9710999999999999e-05 2020-01-16T00:14:26.473 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.473 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.474 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:26.474 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.264994+0000", 2020-01-16T00:14:26.474 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010036188999999999 2020-01-16T00:14:26.474 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.474 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.474 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.474 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.275030+0000", 2020-01-16T00:14:26.474 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1743000000000007e-05 2020-01-16T00:14:26.475 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.475 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.475 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:26.475 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.275112+0000", 2020-01-16T00:14:26.475 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013036100000000001 2020-01-16T00:14:26.475 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.475 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.476 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:26.476 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.275242+0000", 2020-01-16T00:14:26.476 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0016795549999999999 2020-01-16T00:14:26.476 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.476 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.476 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:26.477 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276922+0000", 2020-01-16T00:14:26.477 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018479400000000001 2020-01-16T00:14:26.477 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.477 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.477 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:26.477 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277107+0000", 2020-01-16T00:14:26.478 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00083585300000000001 2020-01-16T00:14:26.478 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.478 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.478 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:26.478 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277943+0000", 2020-01-16T00:14:26.478 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1719999999999999e-05 2020-01-16T00:14:26.478 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.479 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.479 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:26.479 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277994+0000", 2020-01-16T00:14:26.479 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9212999999999997e-05 2020-01-16T00:14:26.479 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.480 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.480 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:26.480 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.278054+0000", 2020-01-16T00:14:26.480 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014559274000000001 2020-01-16T00:14:26.480 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.480 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:26.480 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.481 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.481 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.481 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3122 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e44)", 2020-01-16T00:14:26.481 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.263525+0000", 2020-01-16T00:14:26.481 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 61.142659426999998, 2020-01-16T00:14:26.481 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014731833999999999, 2020-01-16T00:14:26.481 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:26.482 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:26.482 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:26.482 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:26.482 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:26.482 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3122 2020-01-16T00:14:26.482 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.482 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:26.483 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.483 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:26.483 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263525+0000", 2020-01-16T00:14:26.483 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:26.483 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.483 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.483 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:26.483 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263525+0000", 2020-01-16T00:14:26.484 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:26.484 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.484 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.484 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:26.484 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263525+0000", 2020-01-16T00:14:26.484 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5000000000000001e-06 2020-01-16T00:14:26.484 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.485 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.485 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:26.485 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263529+0000", 2020-01-16T00:14:26.485 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4669999999999998e-06 2020-01-16T00:14:26.485 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.485 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.485 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:26.486 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263537+0000", 2020-01-16T00:14:26.486 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0819999999999998e-06 2020-01-16T00:14:26.486 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.486 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.486 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:26.487 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263542+0000", 2020-01-16T00:14:26.487 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0015024960000000001 2020-01-16T00:14:26.487 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.487 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.487 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.487 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.265044+0000", 2020-01-16T00:14:26.487 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6650999999999994e-05 2020-01-16T00:14:26.487 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.488 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.488 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:26.488 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.265131+0000", 2020-01-16T00:14:26.488 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010495310000000001 2020-01-16T00:14:26.488 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.488 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.488 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.488 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.275626+0000", 2020-01-16T00:14:26.489 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9536000000000004e-05 2020-01-16T00:14:26.489 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.489 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.489 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:26.489 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.275726+0000", 2020-01-16T00:14:26.489 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023719489999999999 2020-01-16T00:14:26.489 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.490 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.490 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.490 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.278098+0000", 2020-01-16T00:14:26.490 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3718000000000001e-05 2020-01-16T00:14:26.490 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.490 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.490 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:26.490 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.278171+0000", 2020-01-16T00:14:26.491 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5859999999999994e-05 2020-01-16T00:14:26.491 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.491 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.491 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:26.491 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.278257+0000", 2020-01-16T00:14:26.491 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014731833999999999 2020-01-16T00:14:26.491 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.491 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:26.492 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.492 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.492 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.492 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3627 3.a 3:5069665e:::smithi12313934-2:head [write 1749580~611149] snapc 0=[] ondisk+write+known_if_redirected e48)", 2020-01-16T00:14:26.492 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:39.462779+0000", 2020-01-16T00:14:26.492 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 46.943405615000003, 2020-01-16T00:14:26.492 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020946985000000001, 2020-01-16T00:14:26.493 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:26.493 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:26.493 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:26.493 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:26.493 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:26.493 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3627 2020-01-16T00:14:26.493 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.493 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:26.494 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.494 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:26.494 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.462779+0000", 2020-01-16T00:14:26.494 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:26.494 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.494 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.494 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:26.495 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.462779+0000", 2020-01-16T00:14:26.495 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:26.495 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.495 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.495 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:26.495 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.462777+0000", 2020-01-16T00:14:26.495 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00064136 2020-01-16T00:14:26.495 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.496 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.496 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:26.496 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.463418+0000", 2020-01-16T00:14:26.496 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4607e-05 2020-01-16T00:14:26.496 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.496 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.497 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:26.497 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.463443+0000", 2020-01-16T00:14:26.497 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0346999999999999e-05 2020-01-16T00:14:26.497 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.497 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.497 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:26.497 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.463453+0000", 2020-01-16T00:14:26.498 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000100988 2020-01-16T00:14:26.498 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.498 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.498 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.498 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.463554+0000", 2020-01-16T00:14:26.498 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023059280000000001 2020-01-16T00:14:26.498 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.498 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.499 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:26.499 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.465860+0000", 2020-01-16T00:14:26.499 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011316961 2020-01-16T00:14:26.499 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.499 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.499 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.499 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.477177+0000", 2020-01-16T00:14:26.499 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000132402 2020-01-16T00:14:26.500 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.500 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.500 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:26.500 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.477309+0000", 2020-01-16T00:14:26.500 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000223606 2020-01-16T00:14:26.500 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.500 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.501 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:26.501 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.477533+0000", 2020-01-16T00:14:26.501 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0057105209999999997 2020-01-16T00:14:26.501 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.501 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.501 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:26.502 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483244+0000", 2020-01-16T00:14:26.502 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000193031 2020-01-16T00:14:26.502 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.502 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.502 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:26.502 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483437+0000", 2020-01-16T00:14:26.502 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0902999999999995e-05 2020-01-16T00:14:26.503 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.503 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.503 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:26.503 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483507+0000", 2020-01-16T00:14:26.503 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000129412 2020-01-16T00:14:26.503 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.503 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.504 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:26.504 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483637+0000", 2020-01-16T00:14:26.504 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.9345999999999995e-05 2020-01-16T00:14:26.504 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.504 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.504 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:26.504 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483726+0000", 2020-01-16T00:14:26.504 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020946985000000001 2020-01-16T00:14:26.505 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.505 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:26.505 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.505 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.505 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.505 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3628 3.a 3:5069665e:::smithi12313934-2:head [write 2853383~452450] snapc 0=[] ondisk+write+known_if_redirected e48)", 2020-01-16T00:14:26.505 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:39.468178+0000", 2020-01-16T00:14:26.506 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 46.938006905999998, 2020-01-16T00:14:26.506 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015705872999999999, 2020-01-16T00:14:26.506 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:26.506 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:26.506 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:26.507 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:26.507 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:26.507 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3628 2020-01-16T00:14:26.507 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.507 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:26.507 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.507 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:26.507 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468178+0000", 2020-01-16T00:14:26.508 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:26.508 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.508 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.508 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:26.508 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468178+0000", 2020-01-16T00:14:26.508 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:26.508 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.509 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.509 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:26.509 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468175+0000", 2020-01-16T00:14:26.509 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00065948000000000005 2020-01-16T00:14:26.509 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.509 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.509 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:26.509 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468834+0000", 2020-01-16T00:14:26.510 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3557000000000002e-05 2020-01-16T00:14:26.510 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.510 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.510 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:26.510 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468858+0000", 2020-01-16T00:14:26.510 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2931e-05 2020-01-16T00:14:26.510 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.511 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.511 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:26.511 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468871+0000", 2020-01-16T00:14:26.511 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000123113 2020-01-16T00:14:26.511 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.511 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.511 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.511 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468994+0000", 2020-01-16T00:14:26.512 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017615999999999999 2020-01-16T00:14:26.512 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.512 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.512 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:26.512 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.469170+0000", 2020-01-16T00:14:26.512 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010336908000000001 2020-01-16T00:14:26.512 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.513 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.513 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.513 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.479507+0000", 2020-01-16T00:14:26.513 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015856400000000001 2020-01-16T00:14:26.513 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.513 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.513 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:26.513 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.479665+0000", 2020-01-16T00:14:26.514 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030344600000000002 2020-01-16T00:14:26.514 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.514 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.514 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:26.514 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.479969+0000", 2020-01-16T00:14:26.514 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0037935389999999999 2020-01-16T00:14:26.514 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.514 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.515 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:26.515 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483762+0000", 2020-01-16T00:14:26.515 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3170999999999997e-05 2020-01-16T00:14:26.515 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.515 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.515 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:26.515 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483806+0000", 2020-01-16T00:14:26.516 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3821000000000001e-05 2020-01-16T00:14:26.516 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.516 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.516 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:26.516 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483829+0000", 2020-01-16T00:14:26.516 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0076000000000001e-05 2020-01-16T00:14:26.516 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.516 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.517 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:26.517 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483859+0000", 2020-01-16T00:14:26.517 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.438e-05 2020-01-16T00:14:26.517 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.517 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.518 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:26.518 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483884+0000", 2020-01-16T00:14:26.518 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015705872999999999 2020-01-16T00:14:26.518 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.518 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:26.518 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.518 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.518 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.519 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3629 3.a 3:5069665e:::smithi12313934-2:head [setxattr _header (55) in=7b,truncate 3529536] snapc 0=[] ondisk+write+known_if_redirected e48)", 2020-01-16T00:14:26.519 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:39.468888+0000", 2020-01-16T00:14:26.519 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 46.937297285, 2020-01-16T00:14:26.519 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015108511, 2020-01-16T00:14:26.519 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:26.519 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:26.519 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:26.520 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:26.520 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:26.520 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3629 2020-01-16T00:14:26.520 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.520 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:26.520 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.520 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:26.520 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468888+0000", 2020-01-16T00:14:26.521 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:26.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.521 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.521 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:26.521 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468888+0000", 2020-01-16T00:14:26.521 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:26.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.522 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.522 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:26.522 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468886+0000", 2020-01-16T00:14:26.522 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5599999999999996e-06 2020-01-16T00:14:26.522 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.522 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.523 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:26.523 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468894+0000", 2020-01-16T00:14:26.523 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5279999999999998e-06 2020-01-16T00:14:26.523 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.523 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.523 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:26.524 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468901+0000", 2020-01-16T00:14:26.524 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2290000000000002e-06 2020-01-16T00:14:26.524 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.524 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.524 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:26.524 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468907+0000", 2020-01-16T00:14:26.524 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030197500000000001 2020-01-16T00:14:26.525 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.525 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.525 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.525 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.469209+0000", 2020-01-16T00:14:26.525 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6659999999999997e-05 2020-01-16T00:14:26.525 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.525 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.525 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:26.526 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.469255+0000", 2020-01-16T00:14:26.526 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013117904999999999 2020-01-16T00:14:26.526 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.526 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.526 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.526 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.482373+0000", 2020-01-16T00:14:26.526 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1260000000000006e-05 2020-01-16T00:14:26.527 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.527 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.529 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:26.529 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.482434+0000", 2020-01-16T00:14:26.529 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014433 2020-01-16T00:14:26.529 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.530 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.531 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:26.531 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.482579+0000", 2020-01-16T00:14:26.531 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012198160000000001 2020-01-16T00:14:26.532 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.532 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.533 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:26.533 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483799+0000", 2020-01-16T00:14:26.533 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.188e-05 2020-01-16T00:14:26.533 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.533 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.534 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:26.534 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483810+0000", 2020-01-16T00:14:26.534 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000114863 2020-01-16T00:14:26.534 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.534 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.535 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:26.535 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483925+0000", 2020-01-16T00:14:26.535 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0861000000000001e-05 2020-01-16T00:14:26.535 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.535 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.536 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:26.536 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483956+0000", 2020-01-16T00:14:26.536 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9894999999999997e-05 2020-01-16T00:14:26.536 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.536 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.537 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:26.537 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483996+0000", 2020-01-16T00:14:26.537 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015108511 2020-01-16T00:14:26.537 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.538 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:26.538 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.538 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.538 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.538 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3630 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e48)", 2020-01-16T00:14:26.539 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:39.468967+0000", 2020-01-16T00:14:26.539 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 46.937218151000003, 2020-01-16T00:14:26.539 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015160579, 2020-01-16T00:14:26.539 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:26.540 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:26.540 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:26.540 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:26.540 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:26.540 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3630 2020-01-16T00:14:26.541 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.541 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:26.541 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.541 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:26.541 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468967+0000", 2020-01-16T00:14:26.542 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:26.542 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.542 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.542 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:26.542 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468967+0000", 2020-01-16T00:14:26.543 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:26.543 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.543 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.543 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:26.543 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468966+0000", 2020-01-16T00:14:26.544 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3430000000000003e-06 2020-01-16T00:14:26.544 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.544 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.544 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:26.545 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468971+0000", 2020-01-16T00:14:26.545 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6577000000000001e-05 2020-01-16T00:14:26.545 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.545 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.545 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:26.545 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468988+0000", 2020-01-16T00:14:26.545 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.378e-06 2020-01-16T00:14:26.546 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.546 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.546 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:26.546 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468993+0000", 2020-01-16T00:14:26.546 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000320601 2020-01-16T00:14:26.546 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.546 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.546 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.547 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.469314+0000", 2020-01-16T00:14:26.547 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013833699999999999 2020-01-16T00:14:26.547 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.547 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.547 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:26.547 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.469452+0000", 2020-01-16T00:14:26.547 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013447838 2020-01-16T00:14:26.547 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.548 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.548 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.548 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.482900+0000", 2020-01-16T00:14:26.548 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028006899999999998 2020-01-16T00:14:26.548 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.548 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.548 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:26.549 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483180+0000", 2020-01-16T00:14:26.549 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000833594 2020-01-16T00:14:26.549 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.549 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.549 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.550 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.484013+0000", 2020-01-16T00:14:26.550 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2601e-05 2020-01-16T00:14:26.550 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.550 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.550 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:26.550 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.484066+0000", 2020-01-16T00:14:26.550 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1303999999999994e-05 2020-01-16T00:14:26.550 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.551 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.551 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:26.551 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.484127+0000", 2020-01-16T00:14:26.551 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015160579 2020-01-16T00:14:26.551 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.551 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:26.552 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.552 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.552 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.552 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4047 3.a 3:5069665e:::smithi12313934-2:head [write 1935828~635840] snapc 0=[] ondisk+write+known_if_redirected e52)", 2020-01-16T00:14:26.552 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:50.302087+0000", 2020-01-16T00:14:26.552 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 36.104097879999998, 2020-01-16T00:14:26.552 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014518982, 2020-01-16T00:14:26.553 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:26.553 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:26.553 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:26.553 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:26.553 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:26.553 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4047 2020-01-16T00:14:26.553 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.553 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:26.554 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.554 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:26.554 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302087+0000", 2020-01-16T00:14:26.554 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:26.554 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.554 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.554 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:26.555 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302087+0000", 2020-01-16T00:14:26.555 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:26.555 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.555 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.555 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:26.555 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302085+0000", 2020-01-16T00:14:26.555 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00057147600000000002 2020-01-16T00:14:26.555 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.556 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.556 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:26.556 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302657+0000", 2020-01-16T00:14:26.556 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6427999999999999e-05 2020-01-16T00:14:26.556 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.556 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.556 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:26.556 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302673+0000", 2020-01-16T00:14:26.557 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.127e-06 2020-01-16T00:14:26.557 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.557 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.557 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:26.557 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302680+0000", 2020-01-16T00:14:26.557 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6378000000000006e-05 2020-01-16T00:14:26.557 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.558 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.558 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.558 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302756+0000", 2020-01-16T00:14:26.558 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00039898000000000001 2020-01-16T00:14:26.558 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.558 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.558 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:26.559 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.303155+0000", 2020-01-16T00:14:26.559 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0079666870000000001 2020-01-16T00:14:26.559 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.559 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.559 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.559 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.311122+0000", 2020-01-16T00:14:26.560 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019101400000000001 2020-01-16T00:14:26.560 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.560 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.560 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:26.560 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.311313+0000", 2020-01-16T00:14:26.560 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033332499999999998 2020-01-16T00:14:26.560 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.560 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.561 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:26.561 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.311646+0000", 2020-01-16T00:14:26.561 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0029841440000000002 2020-01-16T00:14:26.561 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.561 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.561 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:26.561 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.314630+0000", 2020-01-16T00:14:26.562 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00027996800000000001 2020-01-16T00:14:26.562 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.562 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.562 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:26.562 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.314910+0000", 2020-01-16T00:14:26.562 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0015637159999999999 2020-01-16T00:14:26.562 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.563 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.563 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:26.563 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.316474+0000", 2020-01-16T00:14:26.563 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5986000000000001e-05 2020-01-16T00:14:26.563 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.563 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.563 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:26.563 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.316550+0000", 2020-01-16T00:14:26.564 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6258000000000002e-05 2020-01-16T00:14:26.564 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.564 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.564 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:26.564 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.316606+0000", 2020-01-16T00:14:26.564 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014518982 2020-01-16T00:14:26.564 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.564 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:26.565 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.565 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.565 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.565 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4600 3.a 3:5069665e:::smithi12313934-2:head [write 1791765~569861] snapc 0=[] ondisk+write+known_if_redirected e56)", 2020-01-16T00:14:26.565 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:03.059744+0000", 2020-01-16T00:14:26.565 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 23.346440662999999, 2020-01-16T00:14:26.565 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017075275000000001, 2020-01-16T00:14:26.566 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:26.566 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:26.566 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:26.566 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:26.566 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:26.566 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4600 2020-01-16T00:14:26.566 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.566 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:26.567 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.567 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:26.567 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.059744+0000", 2020-01-16T00:14:26.567 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:26.567 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.567 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.567 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:26.568 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.059744+0000", 2020-01-16T00:14:26.568 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999843 2020-01-16T00:14:26.568 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.568 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.568 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:26.568 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.059728+0000", 2020-01-16T00:14:26.568 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00054167699999999996 2020-01-16T00:14:26.568 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.569 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.569 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:26.569 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.060270+0000", 2020-01-16T00:14:26.569 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6693999999999998e-05 2020-01-16T00:14:26.569 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.569 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.569 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:26.570 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.060307+0000", 2020-01-16T00:14:26.570 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8753000000000002e-05 2020-01-16T00:14:26.570 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.570 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.570 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:26.570 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.060326+0000", 2020-01-16T00:14:26.571 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000114177 2020-01-16T00:14:26.571 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.571 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.571 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.571 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.060440+0000", 2020-01-16T00:14:26.571 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001642382 2020-01-16T00:14:26.571 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.571 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.572 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:26.572 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.062082+0000", 2020-01-16T00:14:26.572 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0093492149999999993 2020-01-16T00:14:26.572 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.572 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.572 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.572 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.071431+0000", 2020-01-16T00:14:26.573 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013135600000000001 2020-01-16T00:14:26.573 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.573 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.573 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:26.573 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.071563+0000", 2020-01-16T00:14:26.573 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00032728299999999999 2020-01-16T00:14:26.573 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.573 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.574 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:26.574 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.071890+0000", 2020-01-16T00:14:26.574 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0043339759999999998 2020-01-16T00:14:26.574 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.575 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.575 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:26.575 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.076224+0000", 2020-01-16T00:14:26.575 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000343462 2020-01-16T00:14:26.575 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.575 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.576 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:26.576 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.076567+0000", 2020-01-16T00:14:26.576 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1668000000000006e-05 2020-01-16T00:14:26.576 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.576 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.576 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:26.577 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.076639+0000", 2020-01-16T00:14:26.577 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8572000000000004e-05 2020-01-16T00:14:26.577 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.577 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.577 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:26.577 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.076738+0000", 2020-01-16T00:14:26.577 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1816000000000004e-05 2020-01-16T00:14:26.577 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.578 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.578 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:26.578 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.076819+0000", 2020-01-16T00:14:26.578 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017075275000000001 2020-01-16T00:14:26.578 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.578 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:26.578 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.579 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.579 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.579 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4934 3.a 3:5069665e:::smithi12313934-2:head [write 1900783~425211] snapc 0=[] ondisk+write+known_if_redirected e60)", 2020-01-16T00:14:26.579 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:13.356857+0000", 2020-01-16T00:14:26.579 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 13.049327384, 2020-01-16T00:14:26.579 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020085438000000001, 2020-01-16T00:14:26.579 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:26.580 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:26.580 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:26.580 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:26.580 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:26.580 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4934 2020-01-16T00:14:26.581 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.581 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:26.581 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.581 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:26.581 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.356857+0000", 2020-01-16T00:14:26.581 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:26.581 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.582 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.582 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:26.582 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.356857+0000", 2020-01-16T00:14:26.582 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:14:26.582 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.582 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.582 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:26.582 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.356854+0000", 2020-01-16T00:14:26.583 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000450741 2020-01-16T00:14:26.583 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.583 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.583 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:26.583 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357305+0000", 2020-01-16T00:14:26.583 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1829999999999998e-05 2020-01-16T00:14:26.583 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.584 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.584 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:26.584 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357336+0000", 2020-01-16T00:14:26.584 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.305e-05 2020-01-16T00:14:26.584 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.584 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.584 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:26.585 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357349+0000", 2020-01-16T00:14:26.585 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000131087 2020-01-16T00:14:26.585 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.585 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.585 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.586 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357480+0000", 2020-01-16T00:14:26.586 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0031475409999999998 2020-01-16T00:14:26.586 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.586 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.587 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:26.587 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.360628+0000", 2020-01-16T00:14:26.587 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011820447 2020-01-16T00:14:26.587 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.587 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.587 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.588 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.372448+0000", 2020-01-16T00:14:26.588 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000154093 2020-01-16T00:14:26.588 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.588 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.588 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:26.588 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.372603+0000", 2020-01-16T00:14:26.588 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00035365400000000001 2020-01-16T00:14:26.588 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.589 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.589 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:26.589 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.372956+0000", 2020-01-16T00:14:26.589 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0035908590000000001 2020-01-16T00:14:26.589 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.589 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.589 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:26.590 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.376547+0000", 2020-01-16T00:14:26.590 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3077e-05 2020-01-16T00:14:26.590 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.590 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.590 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:26.590 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.376560+0000", 2020-01-16T00:14:26.591 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024608200000000001 2020-01-16T00:14:26.591 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.591 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.591 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:26.591 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.376806+0000", 2020-01-16T00:14:26.591 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3510999999999995e-05 2020-01-16T00:14:26.591 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.592 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.592 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:26.592 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.376880+0000", 2020-01-16T00:14:26.592 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3170000000000007e-05 2020-01-16T00:14:26.592 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.592 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.592 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:26.592 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.376943+0000", 2020-01-16T00:14:26.593 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020085438000000001 2020-01-16T00:14:26.593 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.593 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:26.593 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.593 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.593 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.593 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4935 3.a 3:5069665e:::smithi12313934-2:head [setxattr _header (55) in=7b,truncate 2689252] snapc 0=[] ondisk+write+known_if_redirected e60)", 2020-01-16T00:14:26.594 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:13.357371+0000", 2020-01-16T00:14:26.594 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 13.048813776999999, 2020-01-16T00:14:26.594 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021339295000000001, 2020-01-16T00:14:26.594 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:26.594 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:26.594 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:26.594 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:26.595 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:26.595 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4935 2020-01-16T00:14:26.595 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.595 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:26.595 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.595 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:26.595 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357371+0000", 2020-01-16T00:14:26.595 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:26.596 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.596 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.596 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:26.596 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357371+0000", 2020-01-16T00:14:26.596 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:26.597 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.597 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.597 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:26.597 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357369+0000", 2020-01-16T00:14:26.597 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2839999999999995e-06 2020-01-16T00:14:26.598 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.598 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.598 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:26.598 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357379+0000", 2020-01-16T00:14:26.598 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4363e-05 2020-01-16T00:14:26.599 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.599 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.599 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:26.599 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357403+0000", 2020-01-16T00:14:26.599 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7800000000000003e-06 2020-01-16T00:14:26.599 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.599 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.600 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:26.600 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357410+0000", 2020-01-16T00:14:26.600 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003365771 2020-01-16T00:14:26.600 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.600 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.600 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.601 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.360776+0000", 2020-01-16T00:14:26.601 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00021243299999999999 2020-01-16T00:14:26.601 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.601 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.601 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:26.601 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.360988+0000", 2020-01-16T00:14:26.601 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014015972 2020-01-16T00:14:26.602 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.602 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.602 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.602 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.375004+0000", 2020-01-16T00:14:26.602 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000143306 2020-01-16T00:14:26.602 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.602 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.602 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:26.603 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.375147+0000", 2020-01-16T00:14:26.603 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019620299999999999 2020-01-16T00:14:26.603 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.603 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.603 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:26.603 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.375344+0000", 2020-01-16T00:14:26.603 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001860483 2020-01-16T00:14:26.604 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.604 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.604 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:26.604 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.377204+0000", 2020-01-16T00:14:26.604 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0013554870000000001 2020-01-16T00:14:26.604 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.604 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.605 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:26.605 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.378560+0000", 2020-01-16T00:14:26.605 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8186000000000002e-05 2020-01-16T00:14:26.605 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.605 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.605 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:26.605 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.378608+0000", 2020-01-16T00:14:26.606 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8071000000000001e-05 2020-01-16T00:14:26.606 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.606 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.606 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:26.606 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.378656+0000", 2020-01-16T00:14:26.606 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4595999999999997e-05 2020-01-16T00:14:26.606 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.607 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.607 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:26.607 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.378710+0000", 2020-01-16T00:14:26.607 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021339295000000001 2020-01-16T00:14:26.607 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.607 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:26.607 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.607 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.608 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.608 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4936 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e60)", 2020-01-16T00:14:26.608 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:13.357465+0000", 2020-01-16T00:14:26.608 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 13.048720007, 2020-01-16T00:14:26.608 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021436350999999999, 2020-01-16T00:14:26.608 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:26.608 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:26.609 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:26.609 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:26.609 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:26.609 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4936 2020-01-16T00:14:26.609 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.609 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:26.610 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.610 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:26.610 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357465+0000", 2020-01-16T00:14:26.610 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:26.610 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.610 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.610 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:26.611 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357465+0000", 2020-01-16T00:14:26.611 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:26.611 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.611 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.611 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:26.612 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357464+0000", 2020-01-16T00:14:26.612 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.704e-06 2020-01-16T00:14:26.612 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.612 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.612 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:26.612 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357472+0000", 2020-01-16T00:14:26.612 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6500000000000001e-05 2020-01-16T00:14:26.613 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.613 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.613 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:26.613 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357489+0000", 2020-01-16T00:14:26.613 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.026e-05 2020-01-16T00:14:26.613 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.613 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.614 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:26.614 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357499+0000", 2020-01-16T00:14:26.614 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0042232679999999996 2020-01-16T00:14:26.614 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.614 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.614 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.614 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.361722+0000", 2020-01-16T00:14:26.614 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015945999999999999 2020-01-16T00:14:26.615 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.615 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.615 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:26.615 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.361882+0000", 2020-01-16T00:14:26.615 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014213089999999999 2020-01-16T00:14:26.615 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.615 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.616 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.616 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.376095+0000", 2020-01-16T00:14:26.616 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013706999999999999 2020-01-16T00:14:26.616 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.616 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.616 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:26.616 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.376232+0000", 2020-01-16T00:14:26.616 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002519566 2020-01-16T00:14:26.617 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.617 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.617 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.617 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.378751+0000", 2020-01-16T00:14:26.617 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3343000000000006e-05 2020-01-16T00:14:26.617 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.618 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.618 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:26.618 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.378825+0000", 2020-01-16T00:14:26.618 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.64e-05 2020-01-16T00:14:26.618 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.618 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.619 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:26.619 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.378901+0000", 2020-01-16T00:14:26.619 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021436350999999999 2020-01-16T00:14:26.619 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.619 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:26.619 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.620 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.620 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.620 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:5155 3.a 3:5069665e:::smithi12313934-2:head [write 1648313~507551] snapc 0=[] ondisk+write+known_if_redirected e63)", 2020-01-16T00:14:26.620 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:18.197762+0000", 2020-01-16T00:14:26.620 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 8.2084224349999992, 2020-01-16T00:14:26.620 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014375384999999999, 2020-01-16T00:14:26.620 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:26.621 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:26.621 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:26.621 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:26.621 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:26.622 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 5155 2020-01-16T00:14:26.622 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.622 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:26.622 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.623 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:26.623 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.197762+0000", 2020-01-16T00:14:26.623 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:26.623 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.623 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.623 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:26.624 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.197762+0000", 2020-01-16T00:14:26.624 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:26.624 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.624 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.624 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:26.625 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.197760+0000", 2020-01-16T00:14:26.625 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00049261900000000002 2020-01-16T00:14:26.625 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.625 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.625 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:26.625 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.198253+0000", 2020-01-16T00:14:26.625 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8258e-05 2020-01-16T00:14:26.626 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.626 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.626 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:26.626 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.198271+0000", 2020-01-16T00:14:26.626 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0740000000000004e-06 2020-01-16T00:14:26.626 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.626 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.626 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:26.627 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.198278+0000", 2020-01-16T00:14:26.627 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001581121 2020-01-16T00:14:26.627 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.627 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.627 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.627 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.199859+0000", 2020-01-16T00:14:26.627 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00064528100000000004 2020-01-16T00:14:26.627 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.628 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.628 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:26.628 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.200504+0000", 2020-01-16T00:14:26.628 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0079495580000000007 2020-01-16T00:14:26.628 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.628 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.628 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.629 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.208454+0000", 2020-01-16T00:14:26.629 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.781e-05 2020-01-16T00:14:26.629 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.629 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.629 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:26.629 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.208552+0000", 2020-01-16T00:14:26.630 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014363900000000001 2020-01-16T00:14:26.630 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.630 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.630 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:26.630 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.208695+0000", 2020-01-16T00:14:26.631 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002629036 2020-01-16T00:14:26.631 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.631 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.631 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:26.631 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.211324+0000", 2020-01-16T00:14:26.631 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022371700000000001 2020-01-16T00:14:26.631 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.631 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.632 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:26.632 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.211548+0000", 2020-01-16T00:14:26.632 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00049117300000000002 2020-01-16T00:14:26.632 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.632 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.632 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:26.632 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.212039+0000", 2020-01-16T00:14:26.632 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7794999999999998e-05 2020-01-16T00:14:26.633 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.633 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.633 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:26.633 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.212097+0000", 2020-01-16T00:14:26.633 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0648000000000001e-05 2020-01-16T00:14:26.633 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.633 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.634 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:26.634 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.212138+0000", 2020-01-16T00:14:26.634 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014375384999999999 2020-01-16T00:14:26.634 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.634 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:26.634 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.634 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.634 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.635 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:5156 3.a 3:5069665e:::smithi12313934-2:head [write 2917489~292780] snapc 0=[] ondisk+write+known_if_redirected e63)", 2020-01-16T00:14:26.635 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:18.203801+0000", 2020-01-16T00:14:26.635 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 8.2023840850000003, 2020-01-16T00:14:26.635 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015806831, 2020-01-16T00:14:26.635 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:26.635 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:26.635 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:26.636 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:26.636 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:26.636 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 5156 2020-01-16T00:14:26.636 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.636 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:26.636 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.636 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:26.636 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.203801+0000", 2020-01-16T00:14:26.637 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:26.637 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.637 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.637 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:26.637 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.203801+0000", 2020-01-16T00:14:26.637 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:26.637 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.637 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.638 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:26.638 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.203798+0000", 2020-01-16T00:14:26.638 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00051419400000000004 2020-01-16T00:14:26.638 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.638 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.638 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:26.638 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204312+0000", 2020-01-16T00:14:26.639 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3977000000000001e-05 2020-01-16T00:14:26.639 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.639 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.639 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:26.639 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204336+0000", 2020-01-16T00:14:26.639 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4731e-05 2020-01-16T00:14:26.639 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.639 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.640 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:26.640 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204350+0000", 2020-01-16T00:14:26.640 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.522e-05 2020-01-16T00:14:26.640 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.640 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.640 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.640 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204446+0000", 2020-01-16T00:14:26.641 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010202 2020-01-16T00:14:26.641 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.641 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.641 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:26.642 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204548+0000", 2020-01-16T00:14:26.642 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.005158124 2020-01-16T00:14:26.642 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.642 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.643 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.643 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.209706+0000", 2020-01-16T00:14:26.643 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001294309 2020-01-16T00:14:26.643 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.643 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.644 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:26.644 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.211000+0000", 2020-01-16T00:14:26.644 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0065338569999999997 2020-01-16T00:14:26.644 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.644 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.644 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.645 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.217534+0000", 2020-01-16T00:14:26.645 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5905e-05 2020-01-16T00:14:26.645 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.645 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.645 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:26.645 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.217600+0000", 2020-01-16T00:14:26.645 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000107805 2020-01-16T00:14:26.645 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.646 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.646 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:26.646 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.217708+0000", 2020-01-16T00:14:26.646 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012035570000000001 2020-01-16T00:14:26.646 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.646 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.646 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:26.647 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.218911+0000", 2020-01-16T00:14:26.647 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00027311300000000001 2020-01-16T00:14:26.647 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.647 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.647 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:26.647 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.219184+0000", 2020-01-16T00:14:26.647 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00031979399999999998 2020-01-16T00:14:26.647 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.648 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.648 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:26.648 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.219504+0000", 2020-01-16T00:14:26.648 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2009999999999998e-05 2020-01-16T00:14:26.648 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.648 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.648 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:26.648 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.219566+0000", 2020-01-16T00:14:26.649 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1406000000000003e-05 2020-01-16T00:14:26.649 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.649 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.649 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:26.649 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.219608+0000", 2020-01-16T00:14:26.649 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015806831 2020-01-16T00:14:26.649 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.650 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:26.650 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.650 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.650 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.650 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:5157 3.a 3:5069665e:::smithi12313934-2:head [setxattr _header (55) in=7b,truncate 3210269] snapc 0=[] ondisk+write+known_if_redirected e63)", 2020-01-16T00:14:26.650 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:18.204374+0000", 2020-01-16T00:14:26.650 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 8.2018105709999993, 2020-01-16T00:14:26.651 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015400502, 2020-01-16T00:14:26.651 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:26.651 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:26.651 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:26.651 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:26.652 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:26.652 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 5157 2020-01-16T00:14:26.652 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.652 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:26.652 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.652 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:26.653 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204374+0000", 2020-01-16T00:14:26.653 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:26.653 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.653 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.653 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:26.653 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204374+0000", 2020-01-16T00:14:26.653 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:26.653 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.654 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.654 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:26.654 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204373+0000", 2020-01-16T00:14:26.654 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.4870000000000002e-06 2020-01-16T00:14:26.654 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.654 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.654 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:26.655 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204381+0000", 2020-01-16T00:14:26.655 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4402e-05 2020-01-16T00:14:26.655 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.655 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.655 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:26.655 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204396+0000", 2020-01-16T00:14:26.655 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6640000000000001e-06 2020-01-16T00:14:26.655 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.656 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.656 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:26.656 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204403+0000", 2020-01-16T00:14:26.656 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00021196100000000001 2020-01-16T00:14:26.656 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.656 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.657 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.657 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204614+0000", 2020-01-16T00:14:26.657 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1501999999999994e-05 2020-01-16T00:14:26.657 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.657 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.657 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:26.657 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204686+0000", 2020-01-16T00:14:26.657 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0063990649999999998 2020-01-16T00:14:26.658 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.658 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.658 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.658 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.211085+0000", 2020-01-16T00:14:26.658 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1210000000000003e-05 2020-01-16T00:14:26.658 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.658 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.659 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:26.659 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.211166+0000", 2020-01-16T00:14:26.659 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0071190699999999999 2020-01-16T00:14:26.659 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.659 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.659 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.659 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.218285+0000", 2020-01-16T00:14:26.659 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8831999999999994e-05 2020-01-16T00:14:26.660 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.660 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.660 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:26.660 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.218354+0000", 2020-01-16T00:14:26.660 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013551599999999999 2020-01-16T00:14:26.660 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.660 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.660 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:26.661 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.218490+0000", 2020-01-16T00:14:26.661 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00057678000000000004 2020-01-16T00:14:26.661 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.661 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.661 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:26.661 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.219066+0000", 2020-01-16T00:14:26.661 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022067000000000001 2020-01-16T00:14:26.662 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.662 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.662 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:26.662 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.219287+0000", 2020-01-16T00:14:26.662 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000401868 2020-01-16T00:14:26.662 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.663 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.663 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:26.663 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.219689+0000", 2020-01-16T00:14:26.663 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0040000000000003e-05 2020-01-16T00:14:26.663 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.664 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.664 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:26.664 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.219729+0000", 2020-01-16T00:14:26.664 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5762e-05 2020-01-16T00:14:26.664 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.664 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.665 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:26.665 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.219775+0000", 2020-01-16T00:14:26.665 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015400502 2020-01-16T00:14:26.665 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.665 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:26.665 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.666 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.666 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.666 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:5158 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e63)", 2020-01-16T00:14:26.666 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:18.204424+0000", 2020-01-16T00:14:26.667 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 8.2017608590000002, 2020-01-16T00:14:26.667 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015513872999999999, 2020-01-16T00:14:26.667 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:26.667 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:26.667 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:26.667 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:26.668 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:26.668 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 5158 2020-01-16T00:14:26.668 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.668 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:26.668 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.668 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:26.668 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204424+0000", 2020-01-16T00:14:26.668 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:26.669 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.669 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.669 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:26.669 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204424+0000", 2020-01-16T00:14:26.669 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:26.669 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.669 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.670 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:26.670 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204423+0000", 2020-01-16T00:14:26.670 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4739999999999996e-06 2020-01-16T00:14:26.670 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.670 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.670 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:26.670 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204429+0000", 2020-01-16T00:14:26.671 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9245999999999998e-05 2020-01-16T00:14:26.671 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.671 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.671 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:26.671 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204448+0000", 2020-01-16T00:14:26.671 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.4540000000000007e-06 2020-01-16T00:14:26.671 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.671 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.672 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:26.672 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204456+0000", 2020-01-16T00:14:26.672 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00027379599999999998 2020-01-16T00:14:26.672 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.672 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.672 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.673 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204730+0000", 2020-01-16T00:14:26.673 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2149000000000004e-05 2020-01-16T00:14:26.673 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.673 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.673 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:26.673 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204792+0000", 2020-01-16T00:14:26.673 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0064440110000000004 2020-01-16T00:14:26.674 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.674 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.674 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.674 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.211236+0000", 2020-01-16T00:14:26.674 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2772000000000001e-05 2020-01-16T00:14:26.674 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.674 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.675 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:26.675 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.211299+0000", 2020-01-16T00:14:26.675 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0074481130000000001 2020-01-16T00:14:26.675 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.676 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.676 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.676 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.218747+0000", 2020-01-16T00:14:26.676 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011178100000000001 2020-01-16T00:14:26.676 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.677 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.677 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:26.677 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.218859+0000", 2020-01-16T00:14:26.677 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00093467799999999998 2020-01-16T00:14:26.677 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.678 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.678 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:26.678 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.219794+0000", 2020-01-16T00:14:26.678 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2992999999999998e-05 2020-01-16T00:14:26.678 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.678 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.679 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:26.679 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.219857+0000", 2020-01-16T00:14:26.679 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1285000000000005e-05 2020-01-16T00:14:26.679 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:26.679 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:26.679 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:26.680 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.219938+0000", 2020-01-16T00:14:26.680 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015513872999999999 2020-01-16T00:14:26.680 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.680 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:26.681 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.681 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:26.681 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:26.681 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:26.688 INFO:tasks.rados.rados.0.smithi123.stdout:1894: finishing write tid 2 to smithi12313934-42 2020-01-16T00:14:26.688 INFO:tasks.rados.rados.0.smithi123.stdout:1894: finishing write tid 3 to smithi12313934-42 2020-01-16T00:14:26.688 INFO:tasks.rados.rados.0.smithi123.stdout:1894: finishing write tid 4 to smithi12313934-42 2020-01-16T00:14:26.689 INFO:tasks.rados.rados.0.smithi123.stdout:1894: finishing write tid 6 to smithi12313934-42 2020-01-16T00:14:26.689 INFO:tasks.rados.rados.0.smithi123.stdout:1894: finishing write tid 7 to smithi12313934-42 2020-01-16T00:14:26.689 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 42 v 231 (ObjNum 705 snap 0 seq_num 705) dirty exists 2020-01-16T00:14:26.689 INFO:tasks.rados.rados.0.smithi123.stdout:1894: left oid 42 (ObjNum 705 snap 0 seq_num 705) 2020-01-16T00:14:26.689 INFO:tasks.rados.rados.0.smithi123.stdout:1895: finishing write tid 1 to smithi12313934-30 2020-01-16T00:14:26.690 INFO:tasks.rados.rados.0.smithi123.stdout:1895: finishing write tid 2 to smithi12313934-30 2020-01-16T00:14:26.690 INFO:tasks.rados.rados.0.smithi123.stdout:1890: done (13 left) 2020-01-16T00:14:26.690 INFO:tasks.rados.rados.0.smithi123.stdout:1891: done (12 left) 2020-01-16T00:14:26.691 INFO:tasks.rados.rados.0.smithi123.stdout:1892: done (11 left) 2020-01-16T00:14:26.691 INFO:tasks.rados.rados.0.smithi123.stdout:1893: done (10 left) 2020-01-16T00:14:26.691 INFO:tasks.rados.rados.0.smithi123.stdout:1894: done (9 left) 2020-01-16T00:14:26.691 INFO:tasks.rados.rados.0.smithi123.stdout:1903: done (8 left) 2020-01-16T00:14:26.691 INFO:tasks.rados.rados.0.smithi123.stdout:1904: write oid 20 current snap is 0 2020-01-16T00:14:26.692 INFO:tasks.rados.rados.0.smithi123.stdout:1904: seq_num 713 ranges {783391=473366,1756720=475585,2797182=788004} 2020-01-16T00:14:26.692 INFO:tasks.rados.rados.0.smithi123.stdout:1904: writing smithi12313934-20 from 783391 to 1256757 tid 1 2020-01-16T00:14:26.692 INFO:tasks.rados.rados.0.smithi123.stdout:1904: writing smithi12313934-20 from 1756720 to 2232305 tid 2 2020-01-16T00:14:26.692 INFO:tasks.rados.rados.0.smithi123.stdout:1904: writing smithi12313934-20 from 2797182 to 3585186 tid 3 2020-01-16T00:14:26.692 INFO:tasks.rados.rados.0.smithi123.stdout:1905: read oid 33 snap -1 2020-01-16T00:14:26.692 INFO:tasks.rados.rados.0.smithi123.stdout:1905: expect (ObjNum 691 snap 0 seq_num 691) 2020-01-16T00:14:26.693 INFO:tasks.rados.rados.0.smithi123.stdout:1906: write oid 47 current snap is 0 2020-01-16T00:14:26.693 INFO:tasks.rados.rados.0.smithi123.stdout:1906: seq_num 714 ranges {400198=548615,1415202=584996} 2020-01-16T00:14:26.693 INFO:tasks.rados.rados.0.smithi123.stdout:1906: writing smithi12313934-47 from 400198 to 948813 tid 1 2020-01-16T00:14:26.693 INFO:tasks.rados.rados.0.smithi123.stdout:1906: writing smithi12313934-47 from 1415202 to 2000198 tid 2 2020-01-16T00:14:26.694 INFO:tasks.rados.rados.0.smithi123.stdout:1907: write (excl) oid 40 current snap is 0 2020-01-16T00:14:26.694 INFO:tasks.rados.rados.0.smithi123.stdout:1907: seq_num 715 ranges {417005=740217,1590037=426968} 2020-01-16T00:14:26.694 INFO:tasks.rados.rados.0.smithi123.stdout:1907: writing smithi12313934-40 from 417005 to 1157222 tid 1 2020-01-16T00:14:26.694 INFO:tasks.rados.rados.0.smithi123.stdout:1907: writing smithi12313934-40 from 1590037 to 2017005 tid 2 2020-01-16T00:14:26.694 INFO:tasks.rados.rados.0.smithi123.stdout:1908: read oid 29 snap -1 2020-01-16T00:14:26.694 INFO:tasks.rados.rados.0.smithi123.stdout:1908: expect (ObjNum 659 snap 0 seq_num 659) 2020-01-16T00:14:26.694 INFO:tasks.rados.rados.0.smithi123.stdout:1909: read oid 16 snap -1 2020-01-16T00:14:26.694 INFO:tasks.rados.rados.0.smithi123.stdout:1909: expect (ObjNum 660 snap 0 seq_num 660) 2020-01-16T00:14:26.695 INFO:tasks.rados.rados.0.smithi123.stdout:1910: write (excl) oid 9 current snap is 0 2020-01-16T00:14:26.695 INFO:tasks.rados.rados.0.smithi123.stdout:1910: seq_num 716 ranges {433812=415466,1648519=385293} 2020-01-16T00:14:26.695 INFO:tasks.rados.rados.0.smithi123.stdout:1910: writing smithi12313934-9 from 433812 to 849278 tid 1 2020-01-16T00:14:26.695 INFO:tasks.rados.rados.0.smithi123.stdout:1910: writing smithi12313934-9 from 1648519 to 2033812 tid 2 2020-01-16T00:14:26.695 INFO:tasks.rados.rados.0.smithi123.stdout:1911: read oid 46 snap -1 2020-01-16T00:14:26.695 INFO:tasks.rados.rados.0.smithi123.stdout:1911: expect (ObjNum 687 snap 0 seq_num 687) 2020-01-16T00:14:26.696 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:26.696 INFO:tasks.rados.rados.0.smithi123.stdout:1895: finishing write tid 3 to smithi12313934-30 2020-01-16T00:14:26.696 INFO:tasks.rados.rados.0.smithi123.stdout:1895: finishing write tid 5 to smithi12313934-30 2020-01-16T00:14:26.696 INFO:tasks.rados.rados.0.smithi123.stdout:1895: finishing write tid 6 to smithi12313934-30 2020-01-16T00:14:26.696 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 30 v 199 (ObjNum 706 snap 0 seq_num 706) dirty exists 2020-01-16T00:14:26.697 INFO:tasks.rados.rados.0.smithi123.stdout:1895: left oid 30 (ObjNum 706 snap 0 seq_num 706) 2020-01-16T00:14:26.697 INFO:tasks.rados.rados.0.smithi123.stdout:1896: finishing write tid 1 to smithi12313934-5 2020-01-16T00:14:26.697 INFO:tasks.rados.rados.0.smithi123.stdout:1895: done (15 left) 2020-01-16T00:14:26.697 INFO:tasks.rados.rados.0.smithi123.stdout:1912: write (excl) oid 39 current snap is 0 2020-01-16T00:14:26.697 INFO:tasks.rados.rados.0.smithi123.stdout:1912: seq_num 717 ranges {450619=490715,1590648=459971} 2020-01-16T00:14:26.697 INFO:tasks.rados.rados.0.smithi123.stdout:1912: writing smithi12313934-39 from 450619 to 941334 tid 1 2020-01-16T00:14:26.697 INFO:tasks.rados.rados.0.smithi123.stdout:1912: writing smithi12313934-39 from 1590648 to 2050619 tid 2 2020-01-16T00:14:26.698 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:26.698 INFO:tasks.rados.rados.0.smithi123.stdout:1896: finishing write tid 2 to smithi12313934-5 2020-01-16T00:14:26.698 INFO:tasks.rados.rados.0.smithi123.stdout:1897: expect (ObjNum 628 snap 0 seq_num 628) 2020-01-16T00:14:26.698 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:26.680+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:26.699 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-01-16T00:14:26.758 INFO:tasks.rados.rados.0.smithi123.stdout:1896: finishing write tid 3 to smithi12313934-5 2020-01-16T00:14:26.758 INFO:tasks.rados.rados.0.smithi123.stdout:1896: finishing write tid 5 to smithi12313934-5 2020-01-16T00:14:26.758 INFO:tasks.rados.rados.0.smithi123.stdout:1896: finishing write tid 6 to smithi12313934-5 2020-01-16T00:14:26.758 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 5 v 301 (ObjNum 707 snap 0 seq_num 707) dirty exists 2020-01-16T00:14:26.759 INFO:tasks.rados.rados.0.smithi123.stdout:1896: left oid 5 (ObjNum 707 snap 0 seq_num 707) 2020-01-16T00:14:26.759 INFO:tasks.rados.rados.0.smithi123.stdout:1896: done (15 left) 2020-01-16T00:14:26.759 INFO:tasks.rados.rados.0.smithi123.stdout:1897: done (14 left) 2020-01-16T00:14:26.759 INFO:tasks.rados.rados.0.smithi123.stdout:1913: write (excl) oid 38 current snap is 0 2020-01-16T00:14:26.759 INFO:tasks.rados.rados.0.smithi123.stdout:1913: seq_num 718 ranges {467426=565964,1649130=418296} 2020-01-16T00:14:26.772 INFO:tasks.rados.rados.0.smithi123.stdout:1913: writing smithi12313934-38 from 467426 to 1033390 tid 1 2020-01-16T00:14:26.778 INFO:tasks.rados.rados.0.smithi123.stdout:1913: writing smithi12313934-38 from 1649130 to 2067426 tid 2 2020-01-16T00:14:26.778 INFO:tasks.rados.rados.0.smithi123.stdout:1914: write oid 25 current snap is 0 2020-01-16T00:14:26.778 INFO:tasks.rados.rados.0.smithi123.stdout:1914: seq_num 719 ranges {484233=641213,1707612=376621} 2020-01-16T00:14:26.784 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:26.782+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:26.785 INFO:tasks.rados.rados.0.smithi123.stdout:1914: writing smithi12313934-25 from 484233 to 1125446 tid 1 2020-01-16T00:14:26.791 INFO:tasks.rados.rados.0.smithi123.stdout:1914: writing smithi12313934-25 from 1707612 to 2084233 tid 2 2020-01-16T00:14:26.791 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:26.791 INFO:tasks.rados.rados.0.smithi123.stdout:1898: finishing write tid 1 to smithi12313934-13 2020-01-16T00:14:26.791 INFO:tasks.rados.rados.0.smithi123.stdout:1898: finishing write tid 2 to smithi12313934-13 2020-01-16T00:14:26.791 INFO:tasks.rados.rados.0.smithi123.stdout:1898: finishing write tid 3 to smithi12313934-13 2020-01-16T00:14:26.792 INFO:tasks.rados.rados.0.smithi123.stdout:1898: finishing write tid 5 to smithi12313934-13 2020-01-16T00:14:26.792 INFO:tasks.rados.rados.0.smithi123.stdout:1898: finishing write tid 6 to smithi12313934-13 2020-01-16T00:14:26.792 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 132 (ObjNum 708 snap 0 seq_num 708) dirty exists 2020-01-16T00:14:26.792 INFO:tasks.rados.rados.0.smithi123.stdout:1898: left oid 13 (ObjNum 708 snap 0 seq_num 708) 2020-01-16T00:14:26.793 INFO:tasks.rados.rados.0.smithi123.stdout:1899: finishing write tid 1 to smithi12313934-21 2020-01-16T00:14:26.793 INFO:tasks.rados.rados.0.smithi123.stdout:1898: done (15 left) 2020-01-16T00:14:26.793 INFO:tasks.rados.rados.0.smithi123.stdout:1915: write oid 6 current snap is 0 2020-01-16T00:14:26.794 INFO:tasks.rados.rados.0.smithi123.stdout:1915: seq_num 720 ranges {501040=716462,1766094=334946} 2020-01-16T00:14:26.797 INFO:tasks.rados.rados.0.smithi123.stdout:1915: writing smithi12313934-6 from 501040 to 1217502 tid 1 2020-01-16T00:14:26.803 INFO:tasks.rados.rados.0.smithi123.stdout:1915: writing smithi12313934-6 from 1766094 to 2101040 tid 2 2020-01-16T00:14:26.803 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:26.803 INFO:tasks.rados.rados.0.smithi123.stdout:1899: finishing write tid 2 to smithi12313934-21 2020-01-16T00:14:26.803 INFO:tasks.rados.rados.0.smithi123.stdout:1899: finishing write tid 3 to smithi12313934-21 2020-01-16T00:14:26.803 INFO:tasks.rados.rados.0.smithi123.stdout:1899: finishing write tid 5 to smithi12313934-21 2020-01-16T00:14:26.803 INFO:tasks.rados.rados.0.smithi123.stdout:1899: finishing write tid 6 to smithi12313934-21 2020-01-16T00:14:26.803 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 21 v 93 (ObjNum 709 snap 0 seq_num 709) dirty exists 2020-01-16T00:14:26.804 INFO:tasks.rados.rados.0.smithi123.stdout:1899: left oid 21 (ObjNum 709 snap 0 seq_num 709) 2020-01-16T00:14:26.804 INFO:tasks.rados.rados.0.smithi123.stdout:1900: finishing write tid 1 to smithi12313934-44 2020-01-16T00:14:26.804 INFO:tasks.rados.rados.0.smithi123.stdout:1900: finishing write tid 2 to smithi12313934-44 2020-01-16T00:14:26.804 INFO:tasks.rados.rados.0.smithi123.stdout:1900: finishing write tid 3 to smithi12313934-44 2020-01-16T00:14:26.804 INFO:tasks.rados.rados.0.smithi123.stdout:1899: done (15 left) 2020-01-16T00:14:26.805 INFO:tasks.rados.rados.0.smithi123.stdout:1916: write (excl) oid 5 current snap is 0 2020-01-16T00:14:26.805 INFO:tasks.rados.rados.0.smithi123.stdout:1916: seq_num 721 ranges {517847=791711,2108223=9624} 2020-01-16T00:14:26.806 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:26.806 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:14:26.806 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:14:26.806 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:26.810 INFO:tasks.rados.rados.0.smithi123.stdout:1916: writing smithi12313934-5 from 517847 to 1309558 tid 1 2020-01-16T00:14:26.816 INFO:tasks.rados.rados.0.smithi123.stdout:1916: writing smithi12313934-5 from 2108223 to 2117847 tid 2 2020-01-16T00:14:26.816 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:26.816 INFO:tasks.rados.rados.0.smithi123.stdout:1900: finishing write tid 5 to smithi12313934-44 2020-01-16T00:14:26.817 INFO:tasks.rados.rados.0.smithi123.stdout:1900: finishing write tid 6 to smithi12313934-44 2020-01-16T00:14:26.817 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 44 v 300 (ObjNum 710 snap 0 seq_num 710) dirty exists 2020-01-16T00:14:26.817 INFO:tasks.rados.rados.0.smithi123.stdout:1900: left oid 44 (ObjNum 710 snap 0 seq_num 710) 2020-01-16T00:14:26.817 INFO:tasks.rados.rados.0.smithi123.stdout:1901: finishing write tid 1 to smithi12313934-22 2020-01-16T00:14:26.817 INFO:tasks.rados.rados.0.smithi123.stdout:1900: done (15 left) 2020-01-16T00:14:26.817 INFO:tasks.rados.rados.0.smithi123.stdout:1917: read oid 10 snap -1 2020-01-16T00:14:26.818 INFO:tasks.rados.rados.0.smithi123.stdout:1917: expect (ObjNum 698 snap 0 seq_num 698) 2020-01-16T00:14:26.818 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:26.818 INFO:tasks.rados.rados.0.smithi123.stdout:1901: finishing write tid 2 to smithi12313934-22 2020-01-16T00:14:26.818 INFO:tasks.rados.rados.0.smithi123.stdout:1901: finishing write tid 3 to smithi12313934-22 2020-01-16T00:14:26.818 INFO:tasks.rados.rados.0.smithi123.stdout:1901: finishing write tid 5 to smithi12313934-22 2020-01-16T00:14:26.818 INFO:tasks.rados.rados.0.smithi123.stdout:1901: finishing write tid 6 to smithi12313934-22 2020-01-16T00:14:26.818 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 22 v 53 (ObjNum 711 snap 0 seq_num 711) dirty exists 2020-01-16T00:14:26.819 INFO:tasks.rados.rados.0.smithi123.stdout:1901: left oid 22 (ObjNum 711 snap 0 seq_num 711) 2020-01-16T00:14:26.819 INFO:tasks.rados.rados.0.smithi123.stdout:1902: expect (ObjNum 655 snap 0 seq_num 655) 2020-01-16T00:14:26.819 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-01-16T00:14:26.886 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:26.884+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:26.906 INFO:tasks.rados.rados.0.smithi123.stdout:1904: finishing write tid 1 to smithi12313934-20 2020-01-16T00:14:26.906 INFO:tasks.rados.rados.0.smithi123.stdout:1904: finishing write tid 2 to smithi12313934-20 2020-01-16T00:14:26.907 INFO:tasks.rados.rados.0.smithi123.stdout:1901: done (15 left) 2020-01-16T00:14:26.907 INFO:tasks.rados.rados.0.smithi123.stdout:1902: done (14 left) 2020-01-16T00:14:26.907 INFO:tasks.rados.rados.0.smithi123.stdout:1918: read oid 11 snap -1 2020-01-16T00:14:26.907 INFO:tasks.rados.rados.0.smithi123.stdout:1918: expect (ObjNum 579 snap 0 seq_num 579) 2020-01-16T00:14:26.907 INFO:tasks.rados.rados.0.smithi123.stdout:1919: read oid 23 snap -1 2020-01-16T00:14:26.908 INFO:tasks.rados.rados.0.smithi123.stdout:1919: expect deleted 2020-01-16T00:14:26.908 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:26.908 INFO:tasks.rados.rados.0.smithi123.stdout:1904: finishing write tid 3 to smithi12313934-20 2020-01-16T00:14:26.908 INFO:tasks.rados.rados.0.smithi123.stdout:1904: finishing write tid 5 to smithi12313934-20 2020-01-16T00:14:26.909 INFO:tasks.rados.rados.0.smithi123.stdout:1904: finishing write tid 6 to smithi12313934-20 2020-01-16T00:14:26.909 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 20 v 305 (ObjNum 713 snap 0 seq_num 713) dirty exists 2020-01-16T00:14:26.909 INFO:tasks.rados.rados.0.smithi123.stdout:1904: left oid 20 (ObjNum 713 snap 0 seq_num 713) 2020-01-16T00:14:26.909 INFO:tasks.rados.rados.0.smithi123.stdout:1905: expect (ObjNum 691 snap 0 seq_num 691) 2020-01-16T00:14:26.962 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:26.962 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:14:26.963 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:14:26.963 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:14:26.963 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:26.971 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-01-16T00:14:27.013 INFO:tasks.rados.rados.0.smithi123.stdout:1906: finishing write tid 1 to smithi12313934-47 2020-01-16T00:14:27.013 INFO:tasks.rados.rados.0.smithi123.stdout:1906: finishing write tid 2 to smithi12313934-47 2020-01-16T00:14:27.013 INFO:tasks.rados.rados.0.smithi123.stdout:1904: done (15 left) 2020-01-16T00:14:27.014 INFO:tasks.rados.rados.0.smithi123.stdout:1905: done (14 left) 2020-01-16T00:14:27.014 INFO:tasks.rados.rados.0.smithi123.stdout:1920: read oid 2 snap -1 2020-01-16T00:14:27.014 INFO:tasks.rados.rados.0.smithi123.stdout:1920: expect (ObjNum 667 snap 0 seq_num 667) 2020-01-16T00:14:27.014 INFO:tasks.rados.rados.0.smithi123.stdout:1921: read oid 33 snap -1 2020-01-16T00:14:27.014 INFO:tasks.rados.rados.0.smithi123.stdout:1921: expect (ObjNum 691 snap 0 seq_num 691) 2020-01-16T00:14:27.015 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:27.015 INFO:tasks.rados.rados.0.smithi123.stdout:1906: finishing write tid 4 to smithi12313934-47 2020-01-16T00:14:27.015 INFO:tasks.rados.rados.0.smithi123.stdout:1906: finishing write tid 5 to smithi12313934-47 2020-01-16T00:14:27.015 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 47 v 308 (ObjNum 714 snap 0 seq_num 714) dirty exists 2020-01-16T00:14:27.015 INFO:tasks.rados.rados.0.smithi123.stdout:1906: left oid 47 (ObjNum 714 snap 0 seq_num 714) 2020-01-16T00:14:27.015 INFO:tasks.rados.rados.0.smithi123.stdout:1907: finishing write tid 1 to smithi12313934-40 2020-01-16T00:14:27.016 INFO:tasks.rados.rados.0.smithi123.stdout:1906: done (15 left) 2020-01-16T00:14:27.016 INFO:tasks.rados.rados.0.smithi123.stdout:1922: write (excl) oid 28 current snap is 0 2020-01-16T00:14:27.016 INFO:tasks.rados.rados.0.smithi123.stdout:1922: seq_num 722 ranges {534654=466960,1766705=367949} 2020-01-16T00:14:27.016 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:27.010+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:27.018 INFO:tasks.rados.rados.0.smithi123.stdout:1922: writing smithi12313934-28 from 534654 to 1001614 tid 1 2020-01-16T00:14:27.025 INFO:tasks.rados.rados.0.smithi123.stdout:1922: writing smithi12313934-28 from 1766705 to 2134654 tid 2 2020-01-16T00:14:27.025 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:27.026 INFO:tasks.rados.rados.0.smithi123.stdout:1908: expect (ObjNum 659 snap 0 seq_num 659) 2020-01-16T00:14:27.082 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:27.082 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:14:27.083 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:14:27.084 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:14:27.084 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.084 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:939 3.4 3:2bdd9e88:::smithi12313934-4:head [write 1681102~474010] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:14:27.084 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.908392+0000", 2020-01-16T00:14:27.084 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 109.168979276, 2020-01-16T00:14:27.085 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019294697, 2020-01-16T00:14:27.085 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.085 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:27.085 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:27.086 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:27.086 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:27.086 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 939 2020-01-16T00:14:27.086 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.087 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.087 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.087 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.087 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908392+0000", 2020-01-16T00:14:27.088 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.088 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.088 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.088 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.088 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908392+0000", 2020-01-16T00:14:27.089 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:27.089 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.089 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.089 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.090 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908390+0000", 2020-01-16T00:14:27.090 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015440399999999999 2020-01-16T00:14:27.090 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.090 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.091 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.091 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908545+0000", 2020-01-16T00:14:27.091 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3926e-05 2020-01-16T00:14:27.091 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.092 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.092 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.092 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908559+0000", 2020-01-16T00:14:27.092 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2200000000000003e-06 2020-01-16T00:14:27.093 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.093 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.094 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.094 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908566+0000", 2020-01-16T00:14:27.095 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020594799999999998 2020-01-16T00:14:27.095 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.095 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.095 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.095 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910625+0000", 2020-01-16T00:14:27.095 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118672 2020-01-16T00:14:27.095 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.095 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.096 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:27.096 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910744+0000", 2020-01-16T00:14:27.096 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031940969999999999 2020-01-16T00:14:27.096 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.096 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.096 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.096 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.913938+0000", 2020-01-16T00:14:27.097 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000309621 2020-01-16T00:14:27.097 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.097 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.097 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:27.097 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.914248+0000", 2020-01-16T00:14:27.097 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.008818579 2020-01-16T00:14:27.097 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.097 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.098 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.098 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.923066+0000", 2020-01-16T00:14:27.098 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000108989 2020-01-16T00:14:27.098 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.098 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.098 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.098 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.923175+0000", 2020-01-16T00:14:27.099 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019128299999999999 2020-01-16T00:14:27.099 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.099 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.099 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:14:27.099 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.923367+0000", 2020-01-16T00:14:27.099 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0032083760000000002 2020-01-16T00:14:27.099 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.100 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.100 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:27.100 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926575+0000", 2020-01-16T00:14:27.100 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2475999999999999e-05 2020-01-16T00:14:27.100 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.100 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.101 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.101 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926607+0000", 2020-01-16T00:14:27.101 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00099399800000000006 2020-01-16T00:14:27.101 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.101 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.101 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.101 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927601+0000", 2020-01-16T00:14:27.102 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0958999999999999e-05 2020-01-16T00:14:27.102 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.102 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.102 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:27.102 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927642+0000", 2020-01-16T00:14:27.102 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4747999999999999e-05 2020-01-16T00:14:27.102 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.102 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.103 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.103 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927687+0000", 2020-01-16T00:14:27.103 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019294697 2020-01-16T00:14:27.103 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.103 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.103 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.103 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.104 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.104 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:940 3.4 3:2bdd9e88:::smithi12313934-4:head [write 2687912~589420] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:14:27.104 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.915144+0000", 2020-01-16T00:14:27.105 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 109.16222790800001, 2020-01-16T00:14:27.105 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016092538999999999, 2020-01-16T00:14:27.105 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.105 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:27.106 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:27.106 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:27.106 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:27.106 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 940 2020-01-16T00:14:27.106 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.106 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.106 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.107 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.107 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915144+0000", 2020-01-16T00:14:27.107 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.107 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.107 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.107 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.108 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915144+0000", 2020-01-16T00:14:27.108 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:27.108 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.108 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.108 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.108 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915141+0000", 2020-01-16T00:14:27.109 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016366300000000001 2020-01-16T00:14:27.109 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.109 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.109 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.109 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915305+0000", 2020-01-16T00:14:27.109 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4847e-05 2020-01-16T00:14:27.109 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.110 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.110 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.110 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915320+0000", 2020-01-16T00:14:27.110 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2599999999999999e-06 2020-01-16T00:14:27.110 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.110 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.110 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.111 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915327+0000", 2020-01-16T00:14:27.111 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4282000000000005e-05 2020-01-16T00:14:27.111 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.111 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.111 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.111 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915401+0000", 2020-01-16T00:14:27.111 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4554000000000006e-05 2020-01-16T00:14:27.111 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.112 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.112 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:27.112 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915466+0000", 2020-01-16T00:14:27.112 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0090821900000000004 2020-01-16T00:14:27.112 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.112 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.112 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.113 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.924548+0000", 2020-01-16T00:14:27.113 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000114696 2020-01-16T00:14:27.113 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.113 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.113 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.113 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.924663+0000", 2020-01-16T00:14:27.113 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000168651 2020-01-16T00:14:27.114 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.114 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.114 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:14:27.114 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.924832+0000", 2020-01-16T00:14:27.114 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018253430000000001 2020-01-16T00:14:27.115 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.115 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.115 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:27.115 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926657+0000", 2020-01-16T00:14:27.116 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9957999999999999e-05 2020-01-16T00:14:27.116 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.116 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.116 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.117 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926697+0000", 2020-01-16T00:14:27.117 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0044685250000000001 2020-01-16T00:14:27.117 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.117 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.117 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.117 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931165+0000", 2020-01-16T00:14:27.118 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9211000000000001e-05 2020-01-16T00:14:27.118 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.118 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.118 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:27.118 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931205+0000", 2020-01-16T00:14:27.118 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1699999999999998e-05 2020-01-16T00:14:27.118 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.119 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.119 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.119 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931236+0000", 2020-01-16T00:14:27.119 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016092538999999999 2020-01-16T00:14:27.119 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.119 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.119 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.120 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.120 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.120 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:941 3.4 3:2bdd9e88:::smithi12313934-4:head [setxattr _header (55) in=7b,truncate 3277332] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:14:27.120 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.915341+0000", 2020-01-16T00:14:27.120 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 109.16203109, 2020-01-16T00:14:27.120 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016603769000000001, 2020-01-16T00:14:27.120 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.120 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:27.121 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:27.121 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:27.121 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:27.121 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 941 2020-01-16T00:14:27.121 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.121 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.121 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.122 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.122 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915341+0000", 2020-01-16T00:14:27.122 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.122 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.122 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.122 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.123 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915341+0000", 2020-01-16T00:14:27.123 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:27.123 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.123 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.123 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.123 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915340+0000", 2020-01-16T00:14:27.124 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.896e-06 2020-01-16T00:14:27.124 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.124 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.124 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.124 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915346+0000", 2020-01-16T00:14:27.125 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2730000000000002e-06 2020-01-16T00:14:27.125 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.125 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.125 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.125 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915354+0000", 2020-01-16T00:14:27.126 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0616e-05 2020-01-16T00:14:27.126 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.126 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.126 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.126 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915374+0000", 2020-01-16T00:14:27.126 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000133487 2020-01-16T00:14:27.126 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.127 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.127 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.127 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915508+0000", 2020-01-16T00:14:27.127 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8971000000000003e-05 2020-01-16T00:14:27.127 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.128 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.128 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:27.128 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915557+0000", 2020-01-16T00:14:27.128 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010138787999999999 2020-01-16T00:14:27.128 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.129 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.129 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.129 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.925696+0000", 2020-01-16T00:14:27.129 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012825400000000001 2020-01-16T00:14:27.129 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.129 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.130 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.130 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.925824+0000", 2020-01-16T00:14:27.130 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017706500000000001 2020-01-16T00:14:27.130 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.130 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.130 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:14:27.130 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926001+0000", 2020-01-16T00:14:27.131 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00079988900000000003 2020-01-16T00:14:27.131 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.131 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.131 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.131 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926801+0000", 2020-01-16T00:14:27.131 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119324 2020-01-16T00:14:27.131 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.132 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.132 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:27.132 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926920+0000", 2020-01-16T00:14:27.132 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048879910000000004 2020-01-16T00:14:27.132 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.132 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.132 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.132 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931808+0000", 2020-01-16T00:14:27.133 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6999999999999997e-05 2020-01-16T00:14:27.133 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.133 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.133 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:27.133 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931855+0000", 2020-01-16T00:14:27.133 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9092999999999996e-05 2020-01-16T00:14:27.133 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.134 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.134 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.134 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931944+0000", 2020-01-16T00:14:27.134 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016603769000000001 2020-01-16T00:14:27.134 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.134 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.134 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.134 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.135 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.135 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:942 3.4 3:2bdd9e88:::smithi12313934-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:14:27.135 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.915389+0000", 2020-01-16T00:14:27.136 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 109.161982724, 2020-01-16T00:14:27.136 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016820563, 2020-01-16T00:14:27.136 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.136 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:27.136 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:27.136 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:27.136 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:27.136 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 942 2020-01-16T00:14:27.137 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.137 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.137 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.137 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.137 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915389+0000", 2020-01-16T00:14:27.137 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.138 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.138 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.138 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.138 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915389+0000", 2020-01-16T00:14:27.138 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:27.139 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.139 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.139 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.139 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915388+0000", 2020-01-16T00:14:27.140 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.683e-06 2020-01-16T00:14:27.140 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.140 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.140 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.140 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915391+0000", 2020-01-16T00:14:27.141 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4839999999999998e-06 2020-01-16T00:14:27.141 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.141 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.141 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.141 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915399+0000", 2020-01-16T00:14:27.142 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2259999999999999e-06 2020-01-16T00:14:27.142 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.142 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.142 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.143 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915403+0000", 2020-01-16T00:14:27.143 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00031301199999999999 2020-01-16T00:14:27.143 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.143 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.143 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.144 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915716+0000", 2020-01-16T00:14:27.144 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2079999999999998e-05 2020-01-16T00:14:27.144 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.144 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.145 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:27.145 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915748+0000", 2020-01-16T00:14:27.145 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010547076000000001 2020-01-16T00:14:27.146 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.146 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.146 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.146 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926295+0000", 2020-01-16T00:14:27.147 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8040000000000005e-05 2020-01-16T00:14:27.147 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.147 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.147 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:27.147 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926373+0000", 2020-01-16T00:14:27.147 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056061990000000001 2020-01-16T00:14:27.148 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.148 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.148 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.148 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931979+0000", 2020-01-16T00:14:27.148 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011540399999999999 2020-01-16T00:14:27.148 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.149 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.149 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.149 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932095+0000", 2020-01-16T00:14:27.149 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011482899999999999 2020-01-16T00:14:27.149 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.149 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.150 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.150 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932210+0000", 2020-01-16T00:14:27.150 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016820563 2020-01-16T00:14:27.150 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.150 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.150 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.150 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.151 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.151 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:957 3.0 3:00c146d5:::smithi12313934-18:head [write 510946~632952] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:14:27.151 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.033290+0000", 2020-01-16T00:14:27.151 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 109.044081887, 2020-01-16T00:14:27.151 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016764102999999999, 2020-01-16T00:14:27.151 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.151 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:27.151 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:27.152 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:27.152 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:27.152 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 957 2020-01-16T00:14:27.152 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.152 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.152 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.152 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.153 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033290+0000", 2020-01-16T00:14:27.153 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.153 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.153 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.153 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.153 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033290+0000", 2020-01-16T00:14:27.153 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999952 2020-01-16T00:14:27.154 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.154 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.154 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.154 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033285+0000", 2020-01-16T00:14:27.154 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000405917 2020-01-16T00:14:27.154 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.154 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.154 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.155 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033691+0000", 2020-01-16T00:14:27.155 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6585999999999997e-05 2020-01-16T00:14:27.155 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.155 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.155 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.155 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033738+0000", 2020-01-16T00:14:27.156 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8374000000000001e-05 2020-01-16T00:14:27.156 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.157 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.157 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.157 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033756+0000", 2020-01-16T00:14:27.157 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013002900000000001 2020-01-16T00:14:27.158 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.158 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.158 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.158 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033886+0000", 2020-01-16T00:14:27.158 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035882600000000002 2020-01-16T00:14:27.158 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.158 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.158 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:27.159 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.034245+0000", 2020-01-16T00:14:27.159 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0072798489999999997 2020-01-16T00:14:27.159 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.159 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.159 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.159 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.041525+0000", 2020-01-16T00:14:27.159 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000140488 2020-01-16T00:14:27.159 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.160 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.160 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.160 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.041665+0000", 2020-01-16T00:14:27.160 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000233766 2020-01-16T00:14:27.160 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.160 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.160 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:27.161 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.041899+0000", 2020-01-16T00:14:27.161 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0035766539999999999 2020-01-16T00:14:27.161 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.161 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.161 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:27.161 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.045475+0000", 2020-01-16T00:14:27.161 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013333150000000001 2020-01-16T00:14:27.161 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.162 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.162 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.162 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.046809+0000", 2020-01-16T00:14:27.162 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031447860000000001 2020-01-16T00:14:27.162 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.162 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.162 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.162 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.049954+0000", 2020-01-16T00:14:27.163 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2318000000000003e-05 2020-01-16T00:14:27.163 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.163 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.163 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:27.163 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.050006+0000", 2020-01-16T00:14:27.163 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7972000000000001e-05 2020-01-16T00:14:27.163 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.164 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.164 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.164 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.050054+0000", 2020-01-16T00:14:27.164 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016764102999999999 2020-01-16T00:14:27.164 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.164 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.164 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.164 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.165 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.165 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1639 3.0 3:00c146d5:::smithi12313934-18:head [write 423576~401598] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:14:27.165 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.774317+0000", 2020-01-16T00:14:27.165 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 95.303055147999999, 2020-01-16T00:14:27.165 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018305219000000001, 2020-01-16T00:14:27.165 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.165 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:27.165 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:27.166 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:27.166 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:27.166 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1639 2020-01-16T00:14:27.166 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.166 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.166 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.166 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.166 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774317+0000", 2020-01-16T00:14:27.167 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.167 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.167 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.167 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.167 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774317+0000", 2020-01-16T00:14:27.167 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:27.167 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.168 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.168 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.168 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774313+0000", 2020-01-16T00:14:27.168 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000111173 2020-01-16T00:14:27.168 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.168 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.168 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.169 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774425+0000", 2020-01-16T00:14:27.169 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7598999999999999e-05 2020-01-16T00:14:27.169 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.169 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.169 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.169 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774442+0000", 2020-01-16T00:14:27.169 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0830000000000004e-06 2020-01-16T00:14:27.170 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.170 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.170 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.170 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774450+0000", 2020-01-16T00:14:27.170 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0017886309999999999 2020-01-16T00:14:27.170 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.170 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.171 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.171 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.776239+0000", 2020-01-16T00:14:27.171 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3789e-05 2020-01-16T00:14:27.171 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.171 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.171 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op must wait for map", 2020-01-16T00:14:27.171 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.776253+0000", 2020-01-16T00:14:27.171 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095730729999999997 2020-01-16T00:14:27.172 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.172 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.172 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.172 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.785826+0000", 2020-01-16T00:14:27.172 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014220300000000001 2020-01-16T00:14:27.172 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.172 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.172 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.173 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.785968+0000", 2020-01-16T00:14:27.173 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023618900000000001 2020-01-16T00:14:27.173 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.173 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.173 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:27.173 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.786204+0000", 2020-01-16T00:14:27.173 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0062297389999999998 2020-01-16T00:14:27.174 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.174 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.174 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:27.174 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792434+0000", 2020-01-16T00:14:27.174 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9635999999999998e-05 2020-01-16T00:14:27.174 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.174 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.174 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.175 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792473+0000", 2020-01-16T00:14:27.175 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6456000000000002e-05 2020-01-16T00:14:27.175 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.175 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.175 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.175 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792540+0000", 2020-01-16T00:14:27.175 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9873999999999998e-05 2020-01-16T00:14:27.175 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.176 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.176 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:27.176 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792580+0000", 2020-01-16T00:14:27.176 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1924e-05 2020-01-16T00:14:27.176 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.176 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.176 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.176 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792622+0000", 2020-01-16T00:14:27.177 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018305219000000001 2020-01-16T00:14:27.177 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.177 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.177 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.177 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.177 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.177 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1890 3.0 3:0b1f384f:::smithi12313934-33:head [write 2797915~596476] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:14:27.178 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.232534+0000", 2020-01-16T00:14:27.178 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 89.844837177000002, 2020-01-16T00:14:27.178 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016106137, 2020-01-16T00:14:27.178 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.178 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:27.178 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:27.178 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:27.179 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:27.179 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1890 2020-01-16T00:14:27.179 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.179 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.179 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.179 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.179 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232534+0000", 2020-01-16T00:14:27.179 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.180 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.180 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.180 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.180 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232534+0000", 2020-01-16T00:14:27.180 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:27.180 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.180 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.180 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.181 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232532+0000", 2020-01-16T00:14:27.181 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00024232100000000001 2020-01-16T00:14:27.181 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.181 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.181 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.181 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232774+0000", 2020-01-16T00:14:27.181 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4601e-05 2020-01-16T00:14:27.181 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.182 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.182 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.182 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232789+0000", 2020-01-16T00:14:27.182 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8410000000000006e-06 2020-01-16T00:14:27.182 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.182 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.182 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.183 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232798+0000", 2020-01-16T00:14:27.183 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9756999999999997e-05 2020-01-16T00:14:27.183 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.183 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.183 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.183 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232857+0000", 2020-01-16T00:14:27.183 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4678000000000007e-05 2020-01-16T00:14:27.183 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.184 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.184 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:27.184 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232922+0000", 2020-01-16T00:14:27.184 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033861020000000002 2020-01-16T00:14:27.184 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.184 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.184 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.184 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.236308+0000", 2020-01-16T00:14:27.185 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000568996 2020-01-16T00:14:27.185 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.185 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.185 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:27.185 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.236877+0000", 2020-01-16T00:14:27.185 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0085444240000000001 2020-01-16T00:14:27.185 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.185 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.186 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.186 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.245422+0000", 2020-01-16T00:14:27.186 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5595999999999998e-05 2020-01-16T00:14:27.186 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.186 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.186 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.186 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.245507+0000", 2020-01-16T00:14:27.187 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016984300000000001 2020-01-16T00:14:27.187 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.187 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.187 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:27.187 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.245677+0000", 2020-01-16T00:14:27.187 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001649495 2020-01-16T00:14:27.187 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.187 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.188 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:27.188 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247326+0000", 2020-01-16T00:14:27.188 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023308399999999999 2020-01-16T00:14:27.188 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.188 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.188 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.188 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247560+0000", 2020-01-16T00:14:27.188 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00099501800000000007 2020-01-16T00:14:27.189 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.189 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.189 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.189 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248555+0000", 2020-01-16T00:14:27.189 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5936999999999998e-05 2020-01-16T00:14:27.189 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.189 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.189 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:27.190 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248591+0000", 2020-01-16T00:14:27.190 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0108999999999997e-05 2020-01-16T00:14:27.190 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.190 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.190 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.190 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248641+0000", 2020-01-16T00:14:27.190 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016106137 2020-01-16T00:14:27.191 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.191 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.191 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.191 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.191 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.191 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1892 3.0 3:0b1f384f:::smithi12313934-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e31)", 2020-01-16T00:14:27.191 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.232856+0000", 2020-01-16T00:14:27.191 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 89.844516150999993, 2020-01-16T00:14:27.192 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016171544, 2020-01-16T00:14:27.192 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.192 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:27.192 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:27.192 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:27.192 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:27.192 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1892 2020-01-16T00:14:27.192 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.193 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.193 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.193 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.193 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232856+0000", 2020-01-16T00:14:27.193 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.193 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.193 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.194 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.194 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232856+0000", 2020-01-16T00:14:27.194 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:27.194 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.194 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.194 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.194 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232855+0000", 2020-01-16T00:14:27.194 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8640000000000004e-06 2020-01-16T00:14:27.195 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.195 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.195 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.195 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232860+0000", 2020-01-16T00:14:27.195 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0000000000000001e-05 2020-01-16T00:14:27.195 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.196 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.196 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.196 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232870+0000", 2020-01-16T00:14:27.196 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7420000000000003e-06 2020-01-16T00:14:27.196 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.196 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.196 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.196 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232875+0000", 2020-01-16T00:14:27.197 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017933200000000001 2020-01-16T00:14:27.197 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.197 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.197 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.197 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.233055+0000", 2020-01-16T00:14:27.197 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6269e-05 2020-01-16T00:14:27.197 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.197 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.198 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:27.198 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.233101+0000", 2020-01-16T00:14:27.198 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039780090000000002 2020-01-16T00:14:27.198 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.198 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.198 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.198 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.237079+0000", 2020-01-16T00:14:27.199 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3677000000000001e-05 2020-01-16T00:14:27.199 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.199 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.199 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:27.199 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.237133+0000", 2020-01-16T00:14:27.199 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010036671 2020-01-16T00:14:27.199 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.199 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.200 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.200 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247169+0000", 2020-01-16T00:14:27.200 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010280699999999999 2020-01-16T00:14:27.200 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.200 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.200 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:27.200 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247272+0000", 2020-01-16T00:14:27.200 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001579708 2020-01-16T00:14:27.201 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.201 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.201 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.201 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248852+0000", 2020-01-16T00:14:27.201 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9704999999999996e-05 2020-01-16T00:14:27.201 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.201 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.202 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.202 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248942+0000", 2020-01-16T00:14:27.202 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5557999999999996e-05 2020-01-16T00:14:27.202 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.202 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.203 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.203 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.249027+0000", 2020-01-16T00:14:27.203 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016171544 2020-01-16T00:14:27.203 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.203 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.203 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.203 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.203 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.204 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1950 3.0 3:0b1f384f:::smithi12313934-33:head [write 1889359~541737] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-01-16T00:14:27.204 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.922819+0000", 2020-01-16T00:14:27.204 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 89.154552785999996, 2020-01-16T00:14:27.204 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017502846999999998, 2020-01-16T00:14:27.204 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.204 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:27.204 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:27.204 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:27.205 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:27.205 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1950 2020-01-16T00:14:27.205 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.205 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.205 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.205 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.205 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922819+0000", 2020-01-16T00:14:27.206 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.206 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.206 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.206 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.206 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922819+0000", 2020-01-16T00:14:27.206 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:27.206 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.206 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.207 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.207 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922818+0000", 2020-01-16T00:14:27.207 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000146408 2020-01-16T00:14:27.207 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.207 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.207 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.207 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922964+0000", 2020-01-16T00:14:27.207 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4720000000000001e-06 2020-01-16T00:14:27.208 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.208 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.208 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.208 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922974+0000", 2020-01-16T00:14:27.208 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0720000000000001e-06 2020-01-16T00:14:27.208 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.208 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.209 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.209 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922978+0000", 2020-01-16T00:14:27.209 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6152000000000001e-05 2020-01-16T00:14:27.209 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.209 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.209 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.209 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.923024+0000", 2020-01-16T00:14:27.209 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00087368199999999997 2020-01-16T00:14:27.210 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.210 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.210 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:27.210 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.923897+0000", 2020-01-16T00:14:27.210 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011944435999999999 2020-01-16T00:14:27.210 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.210 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.210 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.211 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.935842+0000", 2020-01-16T00:14:27.211 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000144146 2020-01-16T00:14:27.211 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.211 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.211 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.211 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.935986+0000", 2020-01-16T00:14:27.211 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019775500000000001 2020-01-16T00:14:27.212 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.212 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.212 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:27.212 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.936184+0000", 2020-01-16T00:14:27.212 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002539342 2020-01-16T00:14:27.212 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.212 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.212 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.213 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938723+0000", 2020-01-16T00:14:27.213 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014193699999999999 2020-01-16T00:14:27.213 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.213 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.213 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.213 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938865+0000", 2020-01-16T00:14:27.213 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013829020000000001 2020-01-16T00:14:27.213 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.214 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.214 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:27.214 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.940248+0000", 2020-01-16T00:14:27.214 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2558000000000001e-05 2020-01-16T00:14:27.214 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.214 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.214 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:27.214 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.940291+0000", 2020-01-16T00:14:27.215 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1217999999999998e-05 2020-01-16T00:14:27.215 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.215 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.215 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.215 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.940322+0000", 2020-01-16T00:14:27.215 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017502846999999998 2020-01-16T00:14:27.215 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.216 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.216 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.216 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.216 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.216 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1951 3.0 3:0b1f384f:::smithi12313934-33:head [write 2955792~541902] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-01-16T00:14:27.216 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.929019+0000", 2020-01-16T00:14:27.216 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 89.148352635999998, 2020-01-16T00:14:27.216 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021625710999999999, 2020-01-16T00:14:27.217 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.217 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:27.217 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:27.217 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:27.217 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:27.217 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1951 2020-01-16T00:14:27.217 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.217 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.218 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.218 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.218 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929019+0000", 2020-01-16T00:14:27.218 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.218 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.218 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.218 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.219 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929019+0000", 2020-01-16T00:14:27.219 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999989 2020-01-16T00:14:27.219 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.219 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.219 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.219 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929008+0000", 2020-01-16T00:14:27.219 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000212513 2020-01-16T00:14:27.219 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.220 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.220 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.220 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929221+0000", 2020-01-16T00:14:27.220 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3229e-05 2020-01-16T00:14:27.220 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.220 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.220 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.220 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929234+0000", 2020-01-16T00:14:27.221 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9170000000000003e-06 2020-01-16T00:14:27.221 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.221 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.221 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.221 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929240+0000", 2020-01-16T00:14:27.221 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4045000000000002e-05 2020-01-16T00:14:27.221 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.222 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.222 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.222 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929294+0000", 2020-01-16T00:14:27.222 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9797999999999996e-05 2020-01-16T00:14:27.222 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.222 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.222 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:27.222 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929384+0000", 2020-01-16T00:14:27.223 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083450729999999997 2020-01-16T00:14:27.223 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.223 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.223 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.223 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.937729+0000", 2020-01-16T00:14:27.223 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043251699999999999 2020-01-16T00:14:27.223 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.223 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.224 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:27.224 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938161+0000", 2020-01-16T00:14:27.224 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095115489999999994 2020-01-16T00:14:27.224 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.224 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.224 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.225 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.947673+0000", 2020-01-16T00:14:27.225 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012395099999999999 2020-01-16T00:14:27.225 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.225 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.225 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.225 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.947797+0000", 2020-01-16T00:14:27.225 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00024849200000000001 2020-01-16T00:14:27.225 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.226 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.226 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:27.226 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.948045+0000", 2020-01-16T00:14:27.226 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0025081050000000001 2020-01-16T00:14:27.226 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.226 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.226 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.227 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950553+0000", 2020-01-16T00:14:27.227 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9760000000000003e-06 2020-01-16T00:14:27.227 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.227 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.227 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:27.227 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950558+0000", 2020-01-16T00:14:27.227 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3286999999999999e-05 2020-01-16T00:14:27.227 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.228 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.228 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.228 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950581+0000", 2020-01-16T00:14:27.228 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2759000000000003e-05 2020-01-16T00:14:27.228 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.228 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.228 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:27.228 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950614+0000", 2020-01-16T00:14:27.229 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0525999999999999e-05 2020-01-16T00:14:27.229 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.229 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.229 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.229 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950645+0000", 2020-01-16T00:14:27.229 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021625710999999999 2020-01-16T00:14:27.229 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.230 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.230 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.230 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.230 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.230 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1952 3.0 3:0b1f384f:::smithi12313934-33:head [setxattr _header (56) in=7b,truncate 3563084] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-01-16T00:14:27.230 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:14:27.230 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 89.148123299000005, 2020-01-16T00:14:27.230 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021534685000000001, 2020-01-16T00:14:27.231 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.231 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:27.231 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:27.231 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:27.231 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:27.231 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1952 2020-01-16T00:14:27.231 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.231 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.232 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.232 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.232 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:14:27.232 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.232 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.232 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.232 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.233 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:14:27.233 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:27.233 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.233 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.233 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.233 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:14:27.233 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6670000000000004e-06 2020-01-16T00:14:27.233 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.234 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.234 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.234 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929252+0000", 2020-01-16T00:14:27.234 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9380000000000002e-06 2020-01-16T00:14:27.234 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.234 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.234 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.235 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929256+0000", 2020-01-16T00:14:27.235 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.712e-06 2020-01-16T00:14:27.235 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.235 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.235 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.235 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929264+0000", 2020-01-16T00:14:27.235 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018030300000000001 2020-01-16T00:14:27.235 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.236 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.236 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.236 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929444+0000", 2020-01-16T00:14:27.236 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7562000000000002e-05 2020-01-16T00:14:27.236 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.236 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.236 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:27.237 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929502+0000", 2020-01-16T00:14:27.237 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0087297859999999998 2020-01-16T00:14:27.237 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.237 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.237 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.237 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938232+0000", 2020-01-16T00:14:27.237 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0452999999999996e-05 2020-01-16T00:14:27.237 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.238 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.238 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:27.238 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938322+0000", 2020-01-16T00:14:27.238 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011061728 2020-01-16T00:14:27.238 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.238 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.238 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.238 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.949384+0000", 2020-01-16T00:14:27.239 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000117828 2020-01-16T00:14:27.239 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.239 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.239 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.239 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.949502+0000", 2020-01-16T00:14:27.239 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022468499999999999 2020-01-16T00:14:27.239 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.240 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.240 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:27.240 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.949726+0000", 2020-01-16T00:14:27.240 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00083532099999999998 2020-01-16T00:14:27.240 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.240 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.240 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:27.240 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950562+0000", 2020-01-16T00:14:27.241 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013017700000000001 2020-01-16T00:14:27.241 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.241 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.241 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.241 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950692+0000", 2020-01-16T00:14:27.241 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9813000000000001e-05 2020-01-16T00:14:27.241 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.241 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.242 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.242 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950722+0000", 2020-01-16T00:14:27.242 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0592000000000001e-05 2020-01-16T00:14:27.242 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.242 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.242 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:27.242 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950752+0000", 2020-01-16T00:14:27.242 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0920000000000002e-05 2020-01-16T00:14:27.243 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.243 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.243 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.243 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950783+0000", 2020-01-16T00:14:27.243 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021534685000000001 2020-01-16T00:14:27.243 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.243 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.244 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.244 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.244 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.244 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1953 3.0 3:0b1f384f:::smithi12313934-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e32)", 2020-01-16T00:14:27.244 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:14:27.244 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 89.148090307999993, 2020-01-16T00:14:27.244 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021667537000000001, 2020-01-16T00:14:27.244 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.245 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:27.245 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:27.245 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:27.245 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:27.245 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1953 2020-01-16T00:14:27.245 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.245 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.246 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.246 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.246 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:14:27.246 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.246 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.246 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.246 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.246 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:14:27.247 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:27.247 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.247 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.247 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.247 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:14:27.247 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7030000000000002e-06 2020-01-16T00:14:27.247 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.247 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.248 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.248 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929284+0000", 2020-01-16T00:14:27.248 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0909999999999993e-06 2020-01-16T00:14:27.248 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.248 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.248 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.248 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929292+0000", 2020-01-16T00:14:27.249 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8359999999999999e-06 2020-01-16T00:14:27.249 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.249 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.249 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.249 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929297+0000", 2020-01-16T00:14:27.249 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023538199999999999 2020-01-16T00:14:27.249 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.251 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.251 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.251 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929532+0000", 2020-01-16T00:14:27.251 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1173000000000004e-05 2020-01-16T00:14:27.251 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.251 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.252 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:27.252 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929603+0000", 2020-01-16T00:14:27.252 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0087813509999999997 2020-01-16T00:14:27.252 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.252 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.252 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.252 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938384+0000", 2020-01-16T00:14:27.253 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9752000000000006e-05 2020-01-16T00:14:27.253 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.253 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.253 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:27.253 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938454+0000", 2020-01-16T00:14:27.253 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011707851999999999 2020-01-16T00:14:27.253 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.253 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.254 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.254 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950162+0000", 2020-01-16T00:14:27.254 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011060099999999999 2020-01-16T00:14:27.254 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.254 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.254 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:27.254 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950273+0000", 2020-01-16T00:14:27.255 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00053989700000000001 2020-01-16T00:14:27.255 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.255 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.255 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.255 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950813+0000", 2020-01-16T00:14:27.255 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8128999999999999e-05 2020-01-16T00:14:27.255 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.255 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.256 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.256 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950871+0000", 2020-01-16T00:14:27.256 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8242000000000001e-05 2020-01-16T00:14:27.256 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.256 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.256 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.256 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950949+0000", 2020-01-16T00:14:27.256 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021667537000000001 2020-01-16T00:14:27.257 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.257 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.257 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.257 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.257 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.257 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2555 3.0 3:0b70bb20:::smithi12313934-46:head [setxattr _header (56) in=7b,truncate 2991679] snapc 0=[] ondisk+write+known_if_redirected e37)", 2020-01-16T00:14:27.257 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.573432+0000", 2020-01-16T00:14:27.258 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 74.503939657000004, 2020-01-16T00:14:27.258 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019893926999999999, 2020-01-16T00:14:27.258 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.258 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:27.258 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:27.258 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:27.258 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:27.258 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2555 2020-01-16T00:14:27.259 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.259 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.259 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.259 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.259 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573432+0000", 2020-01-16T00:14:27.259 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.259 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.259 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.260 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.260 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573432+0000", 2020-01-16T00:14:27.260 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:27.260 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.260 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.260 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.260 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573432+0000", 2020-01-16T00:14:27.261 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4290000000000001e-06 2020-01-16T00:14:27.261 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.261 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.261 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.261 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573435+0000", 2020-01-16T00:14:27.261 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1300000000000001e-06 2020-01-16T00:14:27.261 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.261 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.262 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.262 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573438+0000", 2020-01-16T00:14:27.262 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9399999999999996e-06 2020-01-16T00:14:27.262 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.262 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.262 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.262 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573445+0000", 2020-01-16T00:14:27.262 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013947269999999999 2020-01-16T00:14:27.263 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.263 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.263 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.263 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.574840+0000", 2020-01-16T00:14:27.263 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003850058 2020-01-16T00:14:27.263 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.263 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.263 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:27.264 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578690+0000", 2020-01-16T00:14:27.264 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012896489000000001 2020-01-16T00:14:27.264 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.264 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.264 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.264 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591586+0000", 2020-01-16T00:14:27.264 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4757000000000001e-05 2020-01-16T00:14:27.265 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.265 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.265 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.265 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591681+0000", 2020-01-16T00:14:27.265 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000191535 2020-01-16T00:14:27.265 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.265 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.265 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:27.266 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591873+0000", 2020-01-16T00:14:27.266 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00069888899999999996 2020-01-16T00:14:27.266 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.266 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.266 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:27.266 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.592571+0000", 2020-01-16T00:14:27.266 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00029525900000000003 2020-01-16T00:14:27.266 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.267 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.267 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.267 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.592867+0000", 2020-01-16T00:14:27.267 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035243500000000001 2020-01-16T00:14:27.267 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.267 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.267 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.268 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593219+0000", 2020-01-16T00:14:27.268 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0954e-05 2020-01-16T00:14:27.268 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.268 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.268 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:27.268 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593270+0000", 2020-01-16T00:14:27.268 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5813000000000002e-05 2020-01-16T00:14:27.268 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.269 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.269 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.269 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593326+0000", 2020-01-16T00:14:27.269 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019893926999999999 2020-01-16T00:14:27.269 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.269 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.269 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.270 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.270 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.270 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2556 3.0 3:0b70bb20:::smithi12313934-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e37)", 2020-01-16T00:14:27.270 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.573472+0000", 2020-01-16T00:14:27.270 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 74.503900087000005, 2020-01-16T00:14:27.270 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020070068999999999, 2020-01-16T00:14:27.270 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.271 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:27.271 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:27.271 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:27.271 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:27.271 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2556 2020-01-16T00:14:27.271 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.271 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.271 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.272 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.272 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573472+0000", 2020-01-16T00:14:27.272 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.272 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.272 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.272 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.272 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573472+0000", 2020-01-16T00:14:27.273 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:27.273 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.273 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.273 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.273 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573471+0000", 2020-01-16T00:14:27.273 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4449999999999999e-06 2020-01-16T00:14:27.273 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.273 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.274 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.274 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573479+0000", 2020-01-16T00:14:27.274 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9789999999999998e-06 2020-01-16T00:14:27.274 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.274 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.274 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.274 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573485+0000", 2020-01-16T00:14:27.274 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3270000000000001e-06 2020-01-16T00:14:27.275 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.275 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.275 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.275 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573488+0000", 2020-01-16T00:14:27.275 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052586849999999999 2020-01-16T00:14:27.275 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.275 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.275 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.276 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578747+0000", 2020-01-16T00:14:27.276 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5667000000000007e-05 2020-01-16T00:14:27.276 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.276 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.276 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:27.276 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578812+0000", 2020-01-16T00:14:27.276 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013599539000000001 2020-01-16T00:14:27.277 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.277 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.277 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.277 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.592412+0000", 2020-01-16T00:14:27.277 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6018000000000001e-05 2020-01-16T00:14:27.277 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.277 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.277 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:27.278 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.592508+0000", 2020-01-16T00:14:27.278 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00084582199999999998 2020-01-16T00:14:27.278 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.278 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.278 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.278 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593354+0000", 2020-01-16T00:14:27.278 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4984999999999999e-05 2020-01-16T00:14:27.279 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.279 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.279 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.279 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593449+0000", 2020-01-16T00:14:27.279 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3104999999999997e-05 2020-01-16T00:14:27.279 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.279 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.279 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.280 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593542+0000", 2020-01-16T00:14:27.280 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020070068999999999 2020-01-16T00:14:27.280 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.280 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.280 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.280 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.280 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.280 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3211 3.0 3:0b70bb20:::smithi12313934-46:head [setxattr _header (56) in=7b,truncate 2571537] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:14:27.281 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.581146+0000", 2020-01-16T00:14:27.281 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 57.496225516999999, 2020-01-16T00:14:27.281 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024318514999999999, 2020-01-16T00:14:27.281 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.281 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:27.281 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:27.281 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:27.282 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:27.282 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3211 2020-01-16T00:14:27.282 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.282 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.282 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.282 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.282 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581146+0000", 2020-01-16T00:14:27.282 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.283 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.283 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.283 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.283 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581146+0000", 2020-01-16T00:14:27.283 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:27.283 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.283 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.283 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.284 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581145+0000", 2020-01-16T00:14:27.284 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2990000000000002e-06 2020-01-16T00:14:27.284 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.284 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.284 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.284 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581151+0000", 2020-01-16T00:14:27.284 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.041e-06 2020-01-16T00:14:27.284 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.285 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.285 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.285 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581159+0000", 2020-01-16T00:14:27.285 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5859999999999998e-06 2020-01-16T00:14:27.285 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.285 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.285 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.286 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581163+0000", 2020-01-16T00:14:27.286 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020769790000000001 2020-01-16T00:14:27.286 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.286 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.286 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.286 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.583240+0000", 2020-01-16T00:14:27.286 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012019579999999999 2020-01-16T00:14:27.286 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.287 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.287 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:27.287 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584442+0000", 2020-01-16T00:14:27.287 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018359812 2020-01-16T00:14:27.287 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.287 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.287 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.287 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.602802+0000", 2020-01-16T00:14:27.288 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000133072 2020-01-16T00:14:27.288 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.288 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.288 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.288 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.602935+0000", 2020-01-16T00:14:27.288 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019007200000000001 2020-01-16T00:14:27.288 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.288 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.289 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:27.289 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.603125+0000", 2020-01-16T00:14:27.289 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001261751 2020-01-16T00:14:27.289 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.289 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.289 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.289 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.604387+0000", 2020-01-16T00:14:27.290 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000304987 2020-01-16T00:14:27.290 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.290 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.290 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:27.290 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.604692+0000", 2020-01-16T00:14:27.290 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00063369299999999995 2020-01-16T00:14:27.290 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.290 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.291 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.291 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605325+0000", 2020-01-16T00:14:27.291 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1470000000000005e-05 2020-01-16T00:14:27.291 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.291 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.291 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:27.291 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605397+0000", 2020-01-16T00:14:27.291 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7784e-05 2020-01-16T00:14:27.292 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.292 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.292 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.292 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605465+0000", 2020-01-16T00:14:27.292 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024318514999999999 2020-01-16T00:14:27.292 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.292 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.293 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.293 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.293 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.293 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3212 3.0 3:0b70bb20:::smithi12313934-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e45)", 2020-01-16T00:14:27.293 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.581171+0000", 2020-01-16T00:14:27.293 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 57.496200817999998, 2020-01-16T00:14:27.293 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024540131, 2020-01-16T00:14:27.294 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.294 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:27.294 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:27.294 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:27.294 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:27.294 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3212 2020-01-16T00:14:27.294 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.294 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.295 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.295 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.295 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581171+0000", 2020-01-16T00:14:27.295 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.295 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.295 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.295 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.296 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581171+0000", 2020-01-16T00:14:27.296 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:27.296 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.296 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.296 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.296 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581170+0000", 2020-01-16T00:14:27.296 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5259999999999996e-06 2020-01-16T00:14:27.297 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.297 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.297 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.297 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581178+0000", 2020-01-16T00:14:27.297 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.469e-06 2020-01-16T00:14:27.297 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.297 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.298 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.298 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581185+0000", 2020-01-16T00:14:27.298 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4460000000000003e-06 2020-01-16T00:14:27.298 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.298 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.298 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.298 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581190+0000", 2020-01-16T00:14:27.298 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003286696 2020-01-16T00:14:27.299 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.299 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.299 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.299 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584476+0000", 2020-01-16T00:14:27.299 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9681e-05 2020-01-16T00:14:27.299 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.299 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.299 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:27.300 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584536+0000", 2020-01-16T00:14:27.300 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019075267 2020-01-16T00:14:27.300 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.300 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.300 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.300 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.603611+0000", 2020-01-16T00:14:27.300 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5981000000000001e-05 2020-01-16T00:14:27.301 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.301 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.301 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:27.301 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.603707+0000", 2020-01-16T00:14:27.301 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0017922649999999999 2020-01-16T00:14:27.301 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.301 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.301 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.302 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605500+0000", 2020-01-16T00:14:27.302 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5396999999999995e-05 2020-01-16T00:14:27.302 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.302 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.302 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.302 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605595+0000", 2020-01-16T00:14:27.303 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000116038 2020-01-16T00:14:27.303 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.303 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.303 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.303 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605711+0000", 2020-01-16T00:14:27.303 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024540131 2020-01-16T00:14:27.303 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.304 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.304 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.304 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.304 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.304 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3435 3.0 3:0b70bb20:::smithi12313934-46:head [setxattr _header (56) in=7b,truncate 3075747] snapc 0=[] ondisk+write+known_if_redirected e47)", 2020-01-16T00:14:27.304 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:34.388891+0000", 2020-01-16T00:14:27.304 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 52.688480802000001, 2020-01-16T00:14:27.305 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016440475999999999, 2020-01-16T00:14:27.305 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.305 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:27.305 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:27.305 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:27.305 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:27.305 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3435 2020-01-16T00:14:27.305 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.306 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.306 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.306 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.306 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388891+0000", 2020-01-16T00:14:27.306 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.306 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.306 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.307 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.307 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388891+0000", 2020-01-16T00:14:27.307 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:27.307 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.307 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.307 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.307 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388890+0000", 2020-01-16T00:14:27.307 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.647e-06 2020-01-16T00:14:27.308 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.308 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.308 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.308 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388894+0000", 2020-01-16T00:14:27.308 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2839999999999997e-06 2020-01-16T00:14:27.308 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.308 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.308 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.309 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388899+0000", 2020-01-16T00:14:27.309 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3149999999999999e-06 2020-01-16T00:14:27.309 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.309 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.309 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.309 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388901+0000", 2020-01-16T00:14:27.309 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001768543 2020-01-16T00:14:27.310 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.310 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.310 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.310 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.390670+0000", 2020-01-16T00:14:27.310 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0034533680000000001 2020-01-16T00:14:27.310 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.310 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.310 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:27.311 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.394123+0000", 2020-01-16T00:14:27.311 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010189962 2020-01-16T00:14:27.311 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.311 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.311 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.311 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404313+0000", 2020-01-16T00:14:27.311 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000108646 2020-01-16T00:14:27.311 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.312 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.312 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.312 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404422+0000", 2020-01-16T00:14:27.312 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119133 2020-01-16T00:14:27.312 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.312 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.312 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:27.312 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404541+0000", 2020-01-16T00:14:27.313 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000433764 2020-01-16T00:14:27.313 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.313 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.313 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:27.313 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404975+0000", 2020-01-16T00:14:27.313 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000266013 2020-01-16T00:14:27.313 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.314 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.314 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.314 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405241+0000", 2020-01-16T00:14:27.314 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5187000000000001e-05 2020-01-16T00:14:27.314 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.314 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.314 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.314 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405266+0000", 2020-01-16T00:14:27.315 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0889000000000002e-05 2020-01-16T00:14:27.315 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.315 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.315 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:27.315 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405297+0000", 2020-01-16T00:14:27.315 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4354999999999999e-05 2020-01-16T00:14:27.315 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.315 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.316 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.316 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405331+0000", 2020-01-16T00:14:27.316 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016440475999999999 2020-01-16T00:14:27.316 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.316 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.316 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.316 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.317 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.317 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3436 3.0 3:0b70bb20:::smithi12313934-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e47)", 2020-01-16T00:14:27.317 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:34.388920+0000", 2020-01-16T00:14:27.317 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 52.688452013000003, 2020-01-16T00:14:27.317 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016552641, 2020-01-16T00:14:27.317 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.317 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:27.317 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:27.318 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:27.318 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:27.318 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3436 2020-01-16T00:14:27.318 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.318 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.318 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.318 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.318 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388920+0000", 2020-01-16T00:14:27.319 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.319 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.319 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.319 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.319 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388920+0000", 2020-01-16T00:14:27.319 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:27.319 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.320 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.320 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.320 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388919+0000", 2020-01-16T00:14:27.320 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.181e-06 2020-01-16T00:14:27.320 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.320 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.320 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.320 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388921+0000", 2020-01-16T00:14:27.321 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9940000000000003e-06 2020-01-16T00:14:27.321 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.321 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.321 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.321 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388928+0000", 2020-01-16T00:14:27.321 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3549999999999999e-06 2020-01-16T00:14:27.321 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.321 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.322 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.322 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388931+0000", 2020-01-16T00:14:27.322 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.005242483 2020-01-16T00:14:27.322 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.322 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.322 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.322 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.394173+0000", 2020-01-16T00:14:27.323 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7435999999999995e-05 2020-01-16T00:14:27.323 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.323 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.323 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:27.323 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.394251+0000", 2020-01-16T00:14:27.323 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010593286 2020-01-16T00:14:27.323 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.323 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.324 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.324 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404844+0000", 2020-01-16T00:14:27.324 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0620999999999999e-05 2020-01-16T00:14:27.324 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.324 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.324 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:27.324 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404925+0000", 2020-01-16T00:14:27.324 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00042800899999999997 2020-01-16T00:14:27.325 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.325 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.325 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.325 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405353+0000", 2020-01-16T00:14:27.325 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2250000000000003e-05 2020-01-16T00:14:27.325 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.326 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.326 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.326 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405405+0000", 2020-01-16T00:14:27.326 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7411999999999998e-05 2020-01-16T00:14:27.326 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.326 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.326 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.326 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405472+0000", 2020-01-16T00:14:27.327 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016552641 2020-01-16T00:14:27.327 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.327 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.327 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.327 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.327 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.327 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4721 3.0 3:0b70bb20:::smithi12313934-46:head [setxattr _header (56) in=7b,truncate 2134621] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-01-16T00:14:27.328 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.045819+0000", 2020-01-16T00:14:27.328 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 20.031552859000001, 2020-01-16T00:14:27.328 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019902080999999999, 2020-01-16T00:14:27.328 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.328 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:27.328 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:27.328 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:27.328 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:27.329 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4721 2020-01-16T00:14:27.329 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.329 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.329 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.329 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.329 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045819+0000", 2020-01-16T00:14:27.329 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.330 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.330 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.330 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.330 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045819+0000", 2020-01-16T00:14:27.330 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:27.330 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.330 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.330 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.331 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045818+0000", 2020-01-16T00:14:27.331 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.428e-06 2020-01-16T00:14:27.331 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.331 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.331 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.331 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045822+0000", 2020-01-16T00:14:27.331 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5940000000000004e-06 2020-01-16T00:14:27.331 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.332 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.332 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.332 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045826+0000", 2020-01-16T00:14:27.332 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6619999999999998e-06 2020-01-16T00:14:27.332 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.332 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.332 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.332 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045830+0000", 2020-01-16T00:14:27.333 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001710676 2020-01-16T00:14:27.333 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.333 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.333 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.333 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.047541+0000", 2020-01-16T00:14:27.333 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001637009 2020-01-16T00:14:27.333 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.334 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.334 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:27.334 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049178+0000", 2020-01-16T00:14:27.334 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.01461612 2020-01-16T00:14:27.334 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.334 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.334 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.334 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.063794+0000", 2020-01-16T00:14:27.335 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4941999999999997e-05 2020-01-16T00:14:27.335 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.335 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.335 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.335 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.063849+0000", 2020-01-16T00:14:27.335 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011165 2020-01-16T00:14:27.335 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.335 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.336 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:27.336 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.063960+0000", 2020-01-16T00:14:27.336 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013431630000000001 2020-01-16T00:14:27.336 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.336 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.336 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:27.336 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.065303+0000", 2020-01-16T00:14:27.336 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021894699999999999 2020-01-16T00:14:27.337 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.337 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.337 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.337 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.065522+0000", 2020-01-16T00:14:27.337 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013033899999999999 2020-01-16T00:14:27.337 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.337 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.338 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.338 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.065653+0000", 2020-01-16T00:14:27.338 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3800000000000002e-05 2020-01-16T00:14:27.338 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.338 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.338 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:27.338 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.065687+0000", 2020-01-16T00:14:27.338 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4332999999999998e-05 2020-01-16T00:14:27.339 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.339 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.339 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.339 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.065721+0000", 2020-01-16T00:14:27.339 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019902080999999999 2020-01-16T00:14:27.339 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.339 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.339 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.340 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.340 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.340 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4722 3.0 3:0b70bb20:::smithi12313934-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e58)", 2020-01-16T00:14:27.340 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.045850+0000", 2020-01-16T00:14:27.340 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 20.0315221, 2020-01-16T00:14:27.340 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020039033000000001, 2020-01-16T00:14:27.340 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.340 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:27.341 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:27.341 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:27.341 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:27.341 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4722 2020-01-16T00:14:27.341 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.341 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.341 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.342 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.342 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045850+0000", 2020-01-16T00:14:27.342 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.342 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.342 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.342 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.342 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045850+0000", 2020-01-16T00:14:27.342 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:27.343 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.343 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.343 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.343 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045849+0000", 2020-01-16T00:14:27.343 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7349999999999999e-06 2020-01-16T00:14:27.343 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.343 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.344 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.344 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045856+0000", 2020-01-16T00:14:27.344 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1330000000000001e-06 2020-01-16T00:14:27.344 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.344 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.344 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.344 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045860+0000", 2020-01-16T00:14:27.344 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.565e-06 2020-01-16T00:14:27.345 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.345 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.345 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.345 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045863+0000", 2020-01-16T00:14:27.345 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033590019999999998 2020-01-16T00:14:27.345 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.345 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.345 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.346 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049222+0000", 2020-01-16T00:14:27.346 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8267e-05 2020-01-16T00:14:27.346 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.346 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.346 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:27.346 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049320+0000", 2020-01-16T00:14:27.346 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015128875999999999 2020-01-16T00:14:27.346 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.347 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.347 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.347 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.064449+0000", 2020-01-16T00:14:27.347 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4913000000000001e-05 2020-01-16T00:14:27.347 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.347 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.347 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:27.347 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.064504+0000", 2020-01-16T00:14:27.348 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012498100000000001 2020-01-16T00:14:27.348 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.348 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.348 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.348 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.065753+0000", 2020-01-16T00:14:27.348 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6299999999999999e-05 2020-01-16T00:14:27.348 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.349 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.349 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.349 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.065820+0000", 2020-01-16T00:14:27.349 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8801999999999996e-05 2020-01-16T00:14:27.349 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.349 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.349 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.349 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.065889+0000", 2020-01-16T00:14:27.350 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020039033000000001 2020-01-16T00:14:27.350 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.350 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.350 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.350 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.350 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.350 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:27.351 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2020-01-16T00:14:27.352 INFO:tasks.rados.rados.0.smithi123.stdout:1910: finishing write tid 1 to smithi12313934-9 2020-01-16T00:14:27.353 INFO:tasks.rados.rados.0.smithi123.stdout:1907: finishing write tid 2 to smithi12313934-40 2020-01-16T00:14:27.353 INFO:tasks.rados.rados.0.smithi123.stdout:1907: finishing write tid 4 to smithi12313934-40 2020-01-16T00:14:27.353 INFO:tasks.rados.rados.0.smithi123.stdout:1907: finishing write tid 5 to smithi12313934-40 2020-01-16T00:14:27.353 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 40 v 303 (ObjNum 715 snap 0 seq_num 715) dirty exists 2020-01-16T00:14:27.353 INFO:tasks.rados.rados.0.smithi123.stdout:1907: left oid 40 (ObjNum 715 snap 0 seq_num 715) 2020-01-16T00:14:27.353 INFO:tasks.rados.rados.0.smithi123.stdout:1909: expect (ObjNum 660 snap 0 seq_num 660) 2020-01-16T00:14:27.353 INFO:tasks.rados.rados.0.smithi123.stdout:1910: finishing write tid 2 to smithi12313934-9 2020-01-16T00:14:27.354 INFO:tasks.rados.rados.0.smithi123.stdout:1910: finishing write tid 4 to smithi12313934-9 2020-01-16T00:14:27.354 INFO:tasks.rados.rados.0.smithi123.stdout:1910: finishing write tid 5 to smithi12313934-9 2020-01-16T00:14:27.354 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 9 v 189 (ObjNum 716 snap 0 seq_num 716) dirty exists 2020-01-16T00:14:27.354 INFO:tasks.rados.rados.0.smithi123.stdout:1910: left oid 9 (ObjNum 716 snap 0 seq_num 716) 2020-01-16T00:14:27.354 INFO:tasks.rados.rados.0.smithi123.stdout:1912: finishing write tid 1 to smithi12313934-39 2020-01-16T00:14:27.354 INFO:tasks.rados.rados.0.smithi123.stdout:1911: expect (ObjNum 687 snap 0 seq_num 687) 2020-01-16T00:14:27.354 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:27.348+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:27.355 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-01-16T00:14:27.452 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:27.452+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:27.493 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:27.493 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:14:27.494 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:14:27.494 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:27.502 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-01-16T00:14:27.554 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:27.550+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:27.581 INFO:tasks.rados.rados.0.smithi123.stdout:1912: finishing write tid 2 to smithi12313934-39 2020-01-16T00:14:27.582 INFO:tasks.rados.rados.0.smithi123.stdout:1912: finishing write tid 4 to smithi12313934-39 2020-01-16T00:14:27.582 INFO:tasks.rados.rados.0.smithi123.stdout:1912: finishing write tid 5 to smithi12313934-39 2020-01-16T00:14:27.582 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 39 v 306 (ObjNum 717 snap 0 seq_num 717) dirty exists 2020-01-16T00:14:27.582 INFO:tasks.rados.rados.0.smithi123.stdout:1912: left oid 39 (ObjNum 717 snap 0 seq_num 717) 2020-01-16T00:14:27.582 INFO:tasks.rados.rados.0.smithi123.stdout:1913: finishing write tid 1 to smithi12313934-38 2020-01-16T00:14:27.582 INFO:tasks.rados.rados.0.smithi123.stdout:1913: finishing write tid 2 to smithi12313934-38 2020-01-16T00:14:27.582 INFO:tasks.rados.rados.0.smithi123.stdout:1913: finishing write tid 4 to smithi12313934-38 2020-01-16T00:14:27.582 INFO:tasks.rados.rados.0.smithi123.stdout:1907: done (15 left) 2020-01-16T00:14:27.583 INFO:tasks.rados.rados.0.smithi123.stdout:1908: done (14 left) 2020-01-16T00:14:27.583 INFO:tasks.rados.rados.0.smithi123.stdout:1909: done (13 left) 2020-01-16T00:14:27.583 INFO:tasks.rados.rados.0.smithi123.stdout:1910: done (12 left) 2020-01-16T00:14:27.583 INFO:tasks.rados.rados.0.smithi123.stdout:1911: done (11 left) 2020-01-16T00:14:27.583 INFO:tasks.rados.rados.0.smithi123.stdout:1912: done (10 left) 2020-01-16T00:14:27.583 INFO:tasks.rados.rados.0.smithi123.stdout:1923: write (excl) oid 46 current snap is 0 2020-01-16T00:14:27.583 INFO:tasks.rados.rados.0.smithi123.stdout:1923: seq_num 723 ranges {551461=658562,1941540=209921} 2020-01-16T00:14:27.592 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:27.593 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:14:27.593 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:14:27.593 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:14:27.593 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:27.596 INFO:tasks.rados.rados.0.smithi123.stdout:1923: writing smithi12313934-46 from 551461 to 1210023 tid 1 2020-01-16T00:14:27.603 INFO:tasks.rados.rados.0.smithi123.stdout:1923: writing smithi12313934-46 from 1941540 to 2151461 tid 2 2020-01-16T00:14:27.604 INFO:tasks.rados.rados.0.smithi123.stdout:1924: write (excl) oid 13 current snap is 0 2020-01-16T00:14:27.604 INFO:tasks.rados.rados.0.smithi123.stdout:1924: seq_num 724 ranges {568268=733811,2000022=168246} 2020-01-16T00:14:27.604 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-01-16T00:14:27.610 INFO:tasks.rados.rados.0.smithi123.stdout:1924: writing smithi12313934-13 from 568268 to 1302079 tid 1 2020-01-16T00:14:27.615 INFO:tasks.rados.rados.0.smithi123.stdout:1924: writing smithi12313934-13 from 2000022 to 2168268 tid 2 2020-01-16T00:14:27.615 INFO:tasks.rados.rados.0.smithi123.stdout:1925: read oid 14 snap -1 2020-01-16T00:14:27.616 INFO:tasks.rados.rados.0.smithi123.stdout:1925: expect (ObjNum 662 snap 0 seq_num 662) 2020-01-16T00:14:27.616 INFO:tasks.rados.rados.0.smithi123.stdout:1926: write (excl) oid 48 current snap is 0 2020-01-16T00:14:27.616 INFO:tasks.rados.rados.0.smithi123.stdout:1926: seq_num 725 ranges {585075=409060,1542151=642924} 2020-01-16T00:14:27.621 INFO:tasks.rados.rados.0.smithi123.stdout:1926: writing smithi12313934-48 from 585075 to 994135 tid 1 2020-01-16T00:14:27.629 INFO:tasks.rados.rados.0.smithi123.stdout:1926: writing smithi12313934-48 from 1542151 to 2185075 tid 2 2020-01-16T00:14:27.629 INFO:tasks.rados.rados.0.smithi123.stdout:1927: read oid 39 snap -1 2020-01-16T00:14:27.629 INFO:tasks.rados.rados.0.smithi123.stdout:1927: expect (ObjNum 717 snap 0 seq_num 717) 2020-01-16T00:14:27.629 INFO:tasks.rados.rados.0.smithi123.stdout:1928: delete oid 24 current snap is 0 2020-01-16T00:14:27.630 INFO:tasks.rados.rados.0.smithi123.stdout:1913: finishing write tid 5 to smithi12313934-38 2020-01-16T00:14:27.630 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 38 v 234 (ObjNum 718 snap 0 seq_num 718) dirty exists 2020-01-16T00:14:27.630 INFO:tasks.rados.rados.0.smithi123.stdout:1913: left oid 38 (ObjNum 718 snap 0 seq_num 718) 2020-01-16T00:14:27.630 INFO:tasks.rados.rados.0.smithi123.stdout:1914: finishing write tid 1 to smithi12313934-25 2020-01-16T00:14:27.630 INFO:tasks.rados.rados.0.smithi123.stdout:1914: finishing write tid 2 to smithi12313934-25 2020-01-16T00:14:27.630 INFO:tasks.rados.rados.0.smithi123.stdout:1914: finishing write tid 4 to smithi12313934-25 2020-01-16T00:14:27.630 INFO:tasks.rados.rados.0.smithi123.stdout:1914: finishing write tid 5 to smithi12313934-25 2020-01-16T00:14:27.631 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 25 v 311 (ObjNum 719 snap 0 seq_num 719) dirty exists 2020-01-16T00:14:27.631 INFO:tasks.rados.rados.0.smithi123.stdout:1914: left oid 25 (ObjNum 719 snap 0 seq_num 719) 2020-01-16T00:14:27.631 INFO:tasks.rados.rados.0.smithi123.stdout:1915: finishing write tid 1 to smithi12313934-6 2020-01-16T00:14:27.631 INFO:tasks.rados.rados.0.smithi123.stdout:1915: finishing write tid 2 to smithi12313934-6 2020-01-16T00:14:27.631 INFO:tasks.rados.rados.0.smithi123.stdout:1916: finishing write tid 1 to smithi12313934-5 2020-01-16T00:14:27.631 INFO:tasks.rados.rados.0.smithi123.stdout:1915: finishing write tid 4 to smithi12313934-6 2020-01-16T00:14:27.631 INFO:tasks.rados.rados.0.smithi123.stdout:1915: finishing write tid 5 to smithi12313934-6 2020-01-16T00:14:27.631 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 6 v 202 (ObjNum 720 snap 0 seq_num 720) dirty exists 2020-01-16T00:14:27.632 INFO:tasks.rados.rados.0.smithi123.stdout:1915: left oid 6 (ObjNum 720 snap 0 seq_num 720) 2020-01-16T00:14:27.632 INFO:tasks.rados.rados.0.smithi123.stdout:1916: finishing write tid 2 to smithi12313934-5 2020-01-16T00:14:27.632 INFO:tasks.rados.rados.0.smithi123.stdout:1916: finishing write tid 4 to smithi12313934-5 2020-01-16T00:14:27.632 INFO:tasks.rados.rados.0.smithi123.stdout:1916: finishing write tid 5 to smithi12313934-5 2020-01-16T00:14:27.632 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 5 v 314 (ObjNum 721 snap 0 seq_num 721) dirty exists 2020-01-16T00:14:27.632 INFO:tasks.rados.rados.0.smithi123.stdout:1916: left oid 5 (ObjNum 721 snap 0 seq_num 721) 2020-01-16T00:14:27.632 INFO:tasks.rados.rados.0.smithi123.stdout:1917: expect (ObjNum 698 snap 0 seq_num 698) 2020-01-16T00:14:27.655 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:27.654+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:27.685 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-01-16T00:14:27.710 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:27.711 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:14:27.711 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:14:27.711 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:14:27.711 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.711 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1650 3.1 3:8de1d8a3:::smithi12313934-13:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:14:27.711 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847717+0000", 2020-01-16T00:14:27.711 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 95.860439157000002, 2020-01-16T00:14:27.712 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010913658, 2020-01-16T00:14:27.712 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.712 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:27.712 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:27.712 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:27.712 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:27.712 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1650 2020-01-16T00:14:27.712 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.713 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.713 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.713 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.713 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847717+0000", 2020-01-16T00:14:27.713 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.713 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.713 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.714 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.714 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847717+0000", 2020-01-16T00:14:27.714 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:27.714 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.714 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.714 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.714 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847716+0000", 2020-01-16T00:14:27.715 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.433e-06 2020-01-16T00:14:27.715 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.715 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.715 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.715 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847720+0000", 2020-01-16T00:14:27.715 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1800000000000004e-06 2020-01-16T00:14:27.715 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.715 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.716 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.716 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847725+0000", 2020-01-16T00:14:27.716 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6689999999999997e-06 2020-01-16T00:14:27.716 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.716 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.716 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.716 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847730+0000", 2020-01-16T00:14:27.716 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000102468 2020-01-16T00:14:27.717 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.717 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.717 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.717 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847832+0000", 2020-01-16T00:14:27.717 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3999999999999997e-06 2020-01-16T00:14:27.717 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.717 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.718 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:14:27.718 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847838+0000", 2020-01-16T00:14:27.718 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010519395000000001 2020-01-16T00:14:27.718 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.718 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.718 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.718 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858358+0000", 2020-01-16T00:14:27.718 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7600000000000002e-05 2020-01-16T00:14:27.719 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.719 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.719 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.719 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858445+0000", 2020-01-16T00:14:27.719 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018515900000000001 2020-01-16T00:14:27.719 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.719 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.720 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.720 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858631+0000", 2020-01-16T00:14:27.720 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010913658 2020-01-16T00:14:27.720 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.720 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.720 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.720 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.720 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.721 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1651 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~2084068,checksum] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:14:27.721 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847752+0000", 2020-01-16T00:14:27.721 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 95.860404540999994, 2020-01-16T00:14:27.721 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011734249, 2020-01-16T00:14:27.721 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.735 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:27.735 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:27.736 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:27.736 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:27.736 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1651 2020-01-16T00:14:27.736 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.736 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.736 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.736 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.737 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847752+0000", 2020-01-16T00:14:27.737 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.737 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.737 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.737 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.737 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847752+0000", 2020-01-16T00:14:27.737 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:27.738 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.738 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.738 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.738 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847751+0000", 2020-01-16T00:14:27.738 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1980000000000001e-06 2020-01-16T00:14:27.738 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.738 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.738 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.739 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847754+0000", 2020-01-16T00:14:27.739 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6679999999999999e-06 2020-01-16T00:14:27.739 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.739 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.739 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.739 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847758+0000", 2020-01-16T00:14:27.739 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2500000000000001e-06 2020-01-16T00:14:27.739 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.740 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.740 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.740 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847760+0000", 2020-01-16T00:14:27.740 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7614000000000004e-05 2020-01-16T00:14:27.740 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.740 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.740 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.741 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847858+0000", 2020-01-16T00:14:27.741 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7950000000000004e-06 2020-01-16T00:14:27.741 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.741 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.741 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:14:27.741 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847864+0000", 2020-01-16T00:14:27.741 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010790345 2020-01-16T00:14:27.741 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.742 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.742 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.742 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858654+0000", 2020-01-16T00:14:27.742 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000141466 2020-01-16T00:14:27.742 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.742 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.742 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.742 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858795+0000", 2020-01-16T00:14:27.743 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00069037899999999997 2020-01-16T00:14:27.743 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.743 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.743 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.743 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.859486+0000", 2020-01-16T00:14:27.743 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011734249 2020-01-16T00:14:27.743 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.744 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.744 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.744 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.744 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.744 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2108 3.1 3:8de1d8a3:::smithi12313934-13:head [write 2816700~410210] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:14:27.744 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.641861+0000", 2020-01-16T00:14:27.744 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 86.066295350999994, 2020-01-16T00:14:27.744 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018545604, 2020-01-16T00:14:27.745 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.745 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:27.745 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:27.745 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:27.745 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:27.745 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2108 2020-01-16T00:14:27.745 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.746 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.746 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.746 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.747 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.641861+0000", 2020-01-16T00:14:27.747 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.747 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.747 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.747 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.747 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.641861+0000", 2020-01-16T00:14:27.747 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:27.748 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.748 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.748 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.748 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.641859+0000", 2020-01-16T00:14:27.748 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018466399999999999 2020-01-16T00:14:27.748 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.748 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.748 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.749 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642043+0000", 2020-01-16T00:14:27.749 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2999999999999999e-05 2020-01-16T00:14:27.749 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.749 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.749 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.749 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642056+0000", 2020-01-16T00:14:27.749 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0730000000000005e-06 2020-01-16T00:14:27.749 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.750 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.750 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.750 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642064+0000", 2020-01-16T00:14:27.750 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9527999999999997e-05 2020-01-16T00:14:27.750 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.750 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.750 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.751 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642114+0000", 2020-01-16T00:14:27.751 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036616050000000001 2020-01-16T00:14:27.751 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.751 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.751 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:27.751 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645776+0000", 2020-01-16T00:14:27.751 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012222663999999999 2020-01-16T00:14:27.751 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.752 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.752 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.752 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.657998+0000", 2020-01-16T00:14:27.752 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014280299999999999 2020-01-16T00:14:27.752 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.752 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.752 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.753 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.658141+0000", 2020-01-16T00:14:27.753 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0001102 2020-01-16T00:14:27.753 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.753 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.753 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:14:27.753 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.658251+0000", 2020-01-16T00:14:27.753 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020421219999999999 2020-01-16T00:14:27.753 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.754 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.754 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.754 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660293+0000", 2020-01-16T00:14:27.754 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7399999999999999e-06 2020-01-16T00:14:27.754 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.754 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.754 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:27.754 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660303+0000", 2020-01-16T00:14:27.755 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9873999999999998e-05 2020-01-16T00:14:27.755 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.755 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.756 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.757 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660343+0000", 2020-01-16T00:14:27.757 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4448e-05 2020-01-16T00:14:27.757 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.757 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.757 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:27.757 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660377+0000", 2020-01-16T00:14:27.757 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8915999999999999e-05 2020-01-16T00:14:27.757 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.758 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.758 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.758 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660406+0000", 2020-01-16T00:14:27.758 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018545604 2020-01-16T00:14:27.758 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.758 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.758 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.758 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.759 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.759 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2109 3.1 3:8de1d8a3:::smithi12313934-13:head [setxattr _header (56) in=7b,truncate 3865610] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:14:27.759 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.642075+0000", 2020-01-16T00:14:27.759 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 86.066080778, 2020-01-16T00:14:27.759 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018477655999999999, 2020-01-16T00:14:27.759 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.759 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:27.760 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:27.760 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:27.760 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:27.760 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2109 2020-01-16T00:14:27.760 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.760 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.760 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.760 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.761 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642075+0000", 2020-01-16T00:14:27.761 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.761 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.761 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.761 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.761 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642075+0000", 2020-01-16T00:14:27.761 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:27.762 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.762 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.762 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.762 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642074+0000", 2020-01-16T00:14:27.762 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4500000000000003e-06 2020-01-16T00:14:27.762 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.762 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.762 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.763 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642080+0000", 2020-01-16T00:14:27.763 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7229999999999998e-06 2020-01-16T00:14:27.763 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.763 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.763 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.763 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642087+0000", 2020-01-16T00:14:27.763 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2390000000000004e-06 2020-01-16T00:14:27.763 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.764 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.764 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.764 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642091+0000", 2020-01-16T00:14:27.764 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0037396550000000001 2020-01-16T00:14:27.764 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.764 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.764 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.764 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645830+0000", 2020-01-16T00:14:27.765 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2200000000000002e-05 2020-01-16T00:14:27.765 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.765 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.765 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:27.765 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645883+0000", 2020-01-16T00:14:27.765 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013432934000000001 2020-01-16T00:14:27.765 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.766 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.766 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.766 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659316+0000", 2020-01-16T00:14:27.766 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000140006 2020-01-16T00:14:27.766 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.766 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.766 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.766 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659456+0000", 2020-01-16T00:14:27.767 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000184873 2020-01-16T00:14:27.767 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.767 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.767 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:14:27.767 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659640+0000", 2020-01-16T00:14:27.767 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00066954999999999998 2020-01-16T00:14:27.767 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.768 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.768 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:27.768 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660310+0000", 2020-01-16T00:14:27.768 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014847699999999999 2020-01-16T00:14:27.768 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.768 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.768 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.768 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660459+0000", 2020-01-16T00:14:27.769 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6869e-05 2020-01-16T00:14:27.769 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.769 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.769 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.769 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660485+0000", 2020-01-16T00:14:27.769 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8478000000000001e-05 2020-01-16T00:14:27.769 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.769 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.770 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:27.770 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660514+0000", 2020-01-16T00:14:27.770 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9153999999999998e-05 2020-01-16T00:14:27.770 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.770 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.770 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.770 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660553+0000", 2020-01-16T00:14:27.770 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018477655999999999 2020-01-16T00:14:27.771 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.771 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.771 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.771 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.771 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.771 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2110 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-01-16T00:14:27.771 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.642112+0000", 2020-01-16T00:14:27.772 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 86.066044140000002, 2020-01-16T00:14:27.772 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018620198000000001, 2020-01-16T00:14:27.772 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.772 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:27.772 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:27.772 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:27.772 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:27.772 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2110 2020-01-16T00:14:27.773 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.773 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.773 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.773 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.773 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642112+0000", 2020-01-16T00:14:27.773 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.773 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.774 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.774 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.774 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642112+0000", 2020-01-16T00:14:27.774 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:27.774 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.774 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.774 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.775 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642111+0000", 2020-01-16T00:14:27.775 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4450000000000001e-06 2020-01-16T00:14:27.775 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.775 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.775 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.775 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642117+0000", 2020-01-16T00:14:27.775 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1567e-05 2020-01-16T00:14:27.775 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.776 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.776 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.776 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642129+0000", 2020-01-16T00:14:27.776 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.754e-06 2020-01-16T00:14:27.776 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.776 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.776 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.777 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642136+0000", 2020-01-16T00:14:27.777 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040239719999999998 2020-01-16T00:14:27.777 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.777 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.777 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.777 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646160+0000", 2020-01-16T00:14:27.777 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5600999999999998e-05 2020-01-16T00:14:27.777 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.778 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.778 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:27.778 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646215+0000", 2020-01-16T00:14:27.778 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013733006000000001 2020-01-16T00:14:27.778 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.778 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.778 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.779 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659948+0000", 2020-01-16T00:14:27.779 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000106277 2020-01-16T00:14:27.779 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.780 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.780 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:27.780 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660054+0000", 2020-01-16T00:14:27.780 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000524327 2020-01-16T00:14:27.780 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.780 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.780 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.780 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660579+0000", 2020-01-16T00:14:27.781 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9841999999999999e-05 2020-01-16T00:14:27.781 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.781 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.781 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.781 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660659+0000", 2020-01-16T00:14:27.781 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3572999999999995e-05 2020-01-16T00:14:27.781 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.782 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.782 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.782 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660732+0000", 2020-01-16T00:14:27.782 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018620198000000001 2020-01-16T00:14:27.782 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.782 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.782 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.782 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.783 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.783 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:70 2.1 e35/19 2:91f85463:::smithi12313934-24 low_tier:head v 35'66, mlcod=34'64)", 2020-01-16T00:14:27.783 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.986684+0000", 2020-01-16T00:14:27.783 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 81.721472285999994, 2020-01-16T00:14:27.783 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012287378999999999, 2020-01-16T00:14:27.783 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.783 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:27.783 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.784 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.784 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.784 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986684+0000", 2020-01-16T00:14:27.784 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.784 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.784 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.784 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.784 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986684+0000", 2020-01-16T00:14:27.785 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:27.785 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.785 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.785 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.785 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986683+0000", 2020-01-16T00:14:27.785 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012884179999999999 2020-01-16T00:14:27.785 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.786 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.786 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.786 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.987971+0000", 2020-01-16T00:14:27.786 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011558200000000001 2020-01-16T00:14:27.786 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.786 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.786 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.786 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988087+0000", 2020-01-16T00:14:27.787 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0472000000000001e-05 2020-01-16T00:14:27.787 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.787 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.787 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.787 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988137+0000", 2020-01-16T00:14:27.787 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000107484 2020-01-16T00:14:27.787 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.788 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.788 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.788 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988245+0000", 2020-01-16T00:14:27.788 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4409999999999998e-05 2020-01-16T00:14:27.788 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.788 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.788 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.788 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988279+0000", 2020-01-16T00:14:27.789 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010641609999999999 2020-01-16T00:14:27.789 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.789 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.789 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:27.789 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.998921+0000", 2020-01-16T00:14:27.789 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0278000000000002e-05 2020-01-16T00:14:27.789 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.789 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.790 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.790 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.998971+0000", 2020-01-16T00:14:27.790 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012287378999999999 2020-01-16T00:14:27.790 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.790 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.790 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.790 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.790 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.791 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3228 3.1 3:8de1d8a3:::smithi12313934-13:head [write 588344~631611,stat] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:14:27.791 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.588539+0000", 2020-01-16T00:14:27.791 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 58.119617048999999, 2020-01-16T00:14:27.791 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012776884000000001, 2020-01-16T00:14:27.791 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.791 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:27.791 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:27.792 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:27.792 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:27.792 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3228 2020-01-16T00:14:27.792 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.792 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.792 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.792 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.792 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588539+0000", 2020-01-16T00:14:27.793 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.793 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.793 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.793 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588539+0000", 2020-01-16T00:14:27.793 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:27.793 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.794 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.794 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.794 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588536+0000", 2020-01-16T00:14:27.794 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021452 2020-01-16T00:14:27.794 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.794 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.794 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.794 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588750+0000", 2020-01-16T00:14:27.795 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.1432000000000001e-05 2020-01-16T00:14:27.795 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.795 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.795 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.795 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588772+0000", 2020-01-16T00:14:27.795 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1799999999999999e-06 2020-01-16T00:14:27.795 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.795 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.796 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.796 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588779+0000", 2020-01-16T00:14:27.796 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00095777700000000004 2020-01-16T00:14:27.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.796 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.796 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.796 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589737+0000", 2020-01-16T00:14:27.796 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013306599999999999 2020-01-16T00:14:27.797 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.797 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.797 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.797 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589870+0000", 2020-01-16T00:14:27.797 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020556 2020-01-16T00:14:27.797 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.797 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.798 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:14:27.798 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.590075+0000", 2020-01-16T00:14:27.798 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040226080000000004 2020-01-16T00:14:27.798 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.798 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.798 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.798 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.594098+0000", 2020-01-16T00:14:27.798 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.006043288 2020-01-16T00:14:27.799 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.799 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.799 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:27.799 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.600141+0000", 2020-01-16T00:14:27.799 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00089242900000000003 2020-01-16T00:14:27.799 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.799 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.799 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.800 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601034+0000", 2020-01-16T00:14:27.800 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6097e-05 2020-01-16T00:14:27.800 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.800 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.801 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:27.801 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601110+0000", 2020-01-16T00:14:27.801 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000206088 2020-01-16T00:14:27.801 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.801 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.801 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.801 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601316+0000", 2020-01-16T00:14:27.801 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012776884000000001 2020-01-16T00:14:27.802 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.802 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.802 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.802 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.802 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.802 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3230 3.1 3:8de1d8a3:::smithi12313934-13:head [setxattr _header (56) in=7b,truncate 2588344] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:14:27.802 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.596394+0000", 2020-01-16T00:14:27.803 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 58.111762028999998, 2020-01-16T00:14:27.803 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021205134, 2020-01-16T00:14:27.803 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.803 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:27.803 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:27.803 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:27.803 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:27.804 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3230 2020-01-16T00:14:27.804 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.804 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.804 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.804 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.804 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596394+0000", 2020-01-16T00:14:27.804 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.804 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.805 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.805 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.805 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596394+0000", 2020-01-16T00:14:27.805 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:27.805 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.806 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.806 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.806 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596393+0000", 2020-01-16T00:14:27.806 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7159999999999997e-06 2020-01-16T00:14:27.806 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.806 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.806 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.806 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596400+0000", 2020-01-16T00:14:27.807 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0074e-05 2020-01-16T00:14:27.807 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.807 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.807 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.807 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596410+0000", 2020-01-16T00:14:27.807 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1679999999999997e-06 2020-01-16T00:14:27.807 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.808 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.808 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.808 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596414+0000", 2020-01-16T00:14:27.808 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023609 2020-01-16T00:14:27.808 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.808 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.808 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.808 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596650+0000", 2020-01-16T00:14:27.809 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012316399999999999 2020-01-16T00:14:27.809 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.809 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.809 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:27.809 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596774+0000", 2020-01-16T00:14:27.809 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0067849670000000003 2020-01-16T00:14:27.809 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.810 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.810 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.810 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.603558+0000", 2020-01-16T00:14:27.810 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00092385999999999998 2020-01-16T00:14:27.810 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.810 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.810 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:27.810 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.604482+0000", 2020-01-16T00:14:27.811 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011729201 2020-01-16T00:14:27.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.811 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.811 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.811 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.616212+0000", 2020-01-16T00:14:27.811 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019515099999999999 2020-01-16T00:14:27.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.811 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.812 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.812 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.616407+0000", 2020-01-16T00:14:27.812 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000182843 2020-01-16T00:14:27.812 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.812 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.812 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:14:27.812 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.616590+0000", 2020-01-16T00:14:27.812 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000660572 2020-01-16T00:14:27.813 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.813 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.813 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:27.813 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617250+0000", 2020-01-16T00:14:27.813 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5333999999999997e-05 2020-01-16T00:14:27.813 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.813 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.814 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.814 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617285+0000", 2020-01-16T00:14:27.814 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023020899999999999 2020-01-16T00:14:27.814 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.814 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.814 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.814 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617516+0000", 2020-01-16T00:14:27.814 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4214000000000001e-05 2020-01-16T00:14:27.815 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.815 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.815 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:27.815 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617560+0000", 2020-01-16T00:14:27.815 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9366999999999998e-05 2020-01-16T00:14:27.815 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.815 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.815 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.816 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617599+0000", 2020-01-16T00:14:27.816 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021205134 2020-01-16T00:14:27.816 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.816 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.816 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.816 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.816 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.817 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3231 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e45)", 2020-01-16T00:14:27.817 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.596438+0000", 2020-01-16T00:14:27.817 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 58.111718019000001, 2020-01-16T00:14:27.817 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021455667000000001, 2020-01-16T00:14:27.817 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.817 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:27.817 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:27.817 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:27.818 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:27.818 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3231 2020-01-16T00:14:27.818 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.818 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.818 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.818 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.818 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596438+0000", 2020-01-16T00:14:27.818 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.819 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.819 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.819 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.819 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596438+0000", 2020-01-16T00:14:27.819 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:27.819 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.819 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.820 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.820 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596437+0000", 2020-01-16T00:14:27.820 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8500000000000004e-06 2020-01-16T00:14:27.820 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.820 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.820 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.820 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596441+0000", 2020-01-16T00:14:27.820 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9439999999999994e-06 2020-01-16T00:14:27.821 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.821 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.821 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.821 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596450+0000", 2020-01-16T00:14:27.821 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.267e-06 2020-01-16T00:14:27.821 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.821 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.821 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.822 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596455+0000", 2020-01-16T00:14:27.822 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034332399999999999 2020-01-16T00:14:27.822 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.822 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.822 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.822 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596798+0000", 2020-01-16T00:14:27.822 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011687400000000001 2020-01-16T00:14:27.822 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.823 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.823 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:27.823 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596915+0000", 2020-01-16T00:14:27.823 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0076057049999999999 2020-01-16T00:14:27.823 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.823 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.823 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.824 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.604520+0000", 2020-01-16T00:14:27.824 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8143999999999998e-05 2020-01-16T00:14:27.824 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.824 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.824 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:27.824 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.604569+0000", 2020-01-16T00:14:27.824 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012379960000000001 2020-01-16T00:14:27.824 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.825 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.825 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.825 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.616949+0000", 2020-01-16T00:14:27.825 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000120617 2020-01-16T00:14:27.825 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.825 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.825 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:27.825 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617069+0000", 2020-01-16T00:14:27.826 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00056715400000000005 2020-01-16T00:14:27.826 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.826 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.826 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.826 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617636+0000", 2020-01-16T00:14:27.826 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012844700000000001 2020-01-16T00:14:27.826 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.826 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.827 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.827 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617765+0000", 2020-01-16T00:14:27.827 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000129037 2020-01-16T00:14:27.827 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.827 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.827 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.827 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617894+0000", 2020-01-16T00:14:27.827 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021455667000000001 2020-01-16T00:14:27.828 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.828 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.828 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.828 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.828 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.828 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3822 3.1 3:8de1d8a3:::smithi12313934-13:head [setxattr _header (56) in=7b,truncate 2016939] snapc 0=[] ondisk+write+known_if_redirected e50)", 2020-01-16T00:14:27.828 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:44.059584+0000", 2020-01-16T00:14:27.829 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 43.648571703000002, 2020-01-16T00:14:27.829 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021977219999999999, 2020-01-16T00:14:27.829 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.829 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:27.829 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:27.829 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:27.829 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:27.830 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3822 2020-01-16T00:14:27.830 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.830 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.830 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.830 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.830 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059584+0000", 2020-01-16T00:14:27.830 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.831 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.831 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.831 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.831 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059584+0000", 2020-01-16T00:14:27.831 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:27.831 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.831 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.832 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.832 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059584+0000", 2020-01-16T00:14:27.832 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1329999999999998e-06 2020-01-16T00:14:27.832 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.832 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.832 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.832 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059589+0000", 2020-01-16T00:14:27.832 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8136000000000002e-05 2020-01-16T00:14:27.833 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.833 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.833 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.833 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059627+0000", 2020-01-16T00:14:27.833 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1409999999999998e-06 2020-01-16T00:14:27.833 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.833 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.834 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.834 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059631+0000", 2020-01-16T00:14:27.834 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0011314020000000001 2020-01-16T00:14:27.834 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.834 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.834 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.834 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.060763+0000", 2020-01-16T00:14:27.834 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00085078200000000001 2020-01-16T00:14:27.835 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.835 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.835 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:27.835 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.061613+0000", 2020-01-16T00:14:27.835 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017766311999999999 2020-01-16T00:14:27.835 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.835 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.835 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.836 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.079380+0000", 2020-01-16T00:14:27.836 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000133144 2020-01-16T00:14:27.836 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.836 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.836 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.836 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.079513+0000", 2020-01-16T00:14:27.836 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021271500000000001 2020-01-16T00:14:27.836 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.837 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.837 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:14:27.837 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.079725+0000", 2020-01-16T00:14:27.837 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001302582 2020-01-16T00:14:27.837 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.837 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.837 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:27.838 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081028+0000", 2020-01-16T00:14:27.838 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017197 2020-01-16T00:14:27.838 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.838 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.838 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.838 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081200+0000", 2020-01-16T00:14:27.838 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000202757 2020-01-16T00:14:27.838 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.839 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.839 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.839 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081403+0000", 2020-01-16T00:14:27.839 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1068999999999993e-05 2020-01-16T00:14:27.839 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.839 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.839 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:27.839 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081474+0000", 2020-01-16T00:14:27.840 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7800000000000006e-05 2020-01-16T00:14:27.840 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.840 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.840 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.840 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081562+0000", 2020-01-16T00:14:27.840 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021977219999999999 2020-01-16T00:14:27.840 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.841 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.841 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.841 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.841 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.841 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3823 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e50)", 2020-01-16T00:14:27.841 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:44.059639+0000", 2020-01-16T00:14:27.841 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 43.648517548000001, 2020-01-16T00:14:27.841 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022245396000000001, 2020-01-16T00:14:27.842 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.842 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:27.842 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:27.842 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:27.842 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:27.842 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3823 2020-01-16T00:14:27.842 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.843 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.843 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.843 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.843 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059639+0000", 2020-01-16T00:14:27.843 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.843 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.843 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.843 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.844 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059639+0000", 2020-01-16T00:14:27.844 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:27.844 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.844 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.844 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.844 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059638+0000", 2020-01-16T00:14:27.844 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.405e-06 2020-01-16T00:14:27.844 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.845 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.845 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.845 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059641+0000", 2020-01-16T00:14:27.845 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9770000000000003e-06 2020-01-16T00:14:27.845 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.845 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.845 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.846 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059648+0000", 2020-01-16T00:14:27.846 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1140000000000002e-06 2020-01-16T00:14:27.846 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.846 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.846 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.846 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059652+0000", 2020-01-16T00:14:27.846 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020063659999999999 2020-01-16T00:14:27.846 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.847 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.847 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.847 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.061658+0000", 2020-01-16T00:14:27.847 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3183999999999999e-05 2020-01-16T00:14:27.847 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.847 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.848 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:27.848 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.061691+0000", 2020-01-16T00:14:27.848 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018817654999999999 2020-01-16T00:14:27.848 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.848 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.848 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.848 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.080509+0000", 2020-01-16T00:14:27.848 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000104409 2020-01-16T00:14:27.849 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.849 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.849 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:27.849 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.080613+0000", 2020-01-16T00:14:27.849 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00098755999999999996 2020-01-16T00:14:27.849 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.849 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.850 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.850 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081601+0000", 2020-01-16T00:14:27.850 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000145766 2020-01-16T00:14:27.850 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.850 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.850 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.850 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081746+0000", 2020-01-16T00:14:27.850 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013751799999999999 2020-01-16T00:14:27.851 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.851 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.851 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.851 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081884+0000", 2020-01-16T00:14:27.851 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022245396000000001 2020-01-16T00:14:27.851 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.851 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.851 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.852 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.852 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.852 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4398 3.1 3:8de1d8a3:::smithi12313934-13:head [write 2812170~507386] snapc 0=[] ondisk+write+known_if_redirected e55)", 2020-01-16T00:14:27.852 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:58.336543+0000", 2020-01-16T00:14:27.852 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 29.371613114999999, 2020-01-16T00:14:27.852 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015697994, 2020-01-16T00:14:27.852 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.852 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:27.853 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:27.853 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:27.853 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:27.853 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4398 2020-01-16T00:14:27.853 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.853 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.853 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.854 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.854 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336543+0000", 2020-01-16T00:14:27.854 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.854 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.854 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.854 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.854 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336543+0000", 2020-01-16T00:14:27.854 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:27.855 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.855 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.855 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.855 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336542+0000", 2020-01-16T00:14:27.855 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014254 2020-01-16T00:14:27.855 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.855 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.856 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.856 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336684+0000", 2020-01-16T00:14:27.856 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2019e-05 2020-01-16T00:14:27.856 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.856 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.856 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.856 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336696+0000", 2020-01-16T00:14:27.856 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4269999999999996e-06 2020-01-16T00:14:27.857 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.857 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.857 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.857 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336702+0000", 2020-01-16T00:14:27.857 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8040000000000002e-05 2020-01-16T00:14:27.857 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.857 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.857 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.858 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336741+0000", 2020-01-16T00:14:27.858 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00030631400000000002 2020-01-16T00:14:27.858 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.858 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.858 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:27.858 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.337047+0000", 2020-01-16T00:14:27.858 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012249832 2020-01-16T00:14:27.859 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.859 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.859 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.859 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.349297+0000", 2020-01-16T00:14:27.859 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013870899999999999 2020-01-16T00:14:27.859 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.859 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.859 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.860 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.349435+0000", 2020-01-16T00:14:27.860 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021913600000000001 2020-01-16T00:14:27.860 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.860 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.860 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:14:27.860 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.349655+0000", 2020-01-16T00:14:27.860 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021982490000000002 2020-01-16T00:14:27.860 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.861 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.861 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:27.861 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351853+0000", 2020-01-16T00:14:27.861 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8295999999999996e-05 2020-01-16T00:14:27.861 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.861 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.861 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.862 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351941+0000", 2020-01-16T00:14:27.862 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021662 2020-01-16T00:14:27.862 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.862 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.862 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.862 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352158+0000", 2020-01-16T00:14:27.862 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6031000000000001e-05 2020-01-16T00:14:27.862 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.863 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.863 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:27.863 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352194+0000", 2020-01-16T00:14:27.863 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7290000000000003e-05 2020-01-16T00:14:27.863 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.863 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.863 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.863 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352241+0000", 2020-01-16T00:14:27.864 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015697994 2020-01-16T00:14:27.864 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.864 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.864 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.864 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.864 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.864 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4399 3.1 3:8de1d8a3:::smithi12313934-13:head [setxattr _header (56) in=7b,truncate 3411920] snapc 0=[] ondisk+write+known_if_redirected e55)", 2020-01-16T00:14:27.864 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:58.336714+0000", 2020-01-16T00:14:27.865 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 29.371441758, 2020-01-16T00:14:27.865 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016051421, 2020-01-16T00:14:27.865 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.865 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:27.865 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:27.865 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:27.865 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:27.866 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4399 2020-01-16T00:14:27.866 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.866 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.866 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.866 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.866 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336714+0000", 2020-01-16T00:14:27.866 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.866 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.867 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.867 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.867 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336714+0000", 2020-01-16T00:14:27.867 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:27.867 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.867 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.867 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.867 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336714+0000", 2020-01-16T00:14:27.868 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7540000000000003e-06 2020-01-16T00:14:27.868 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.868 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.868 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.868 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336719+0000", 2020-01-16T00:14:27.868 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2879999999999998e-06 2020-01-16T00:14:27.868 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.868 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.869 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.869 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336726+0000", 2020-01-16T00:14:27.869 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6399999999999996e-06 2020-01-16T00:14:27.869 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.869 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.869 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.869 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336730+0000", 2020-01-16T00:14:27.869 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00036299699999999999 2020-01-16T00:14:27.870 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.870 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.870 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.870 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.337093+0000", 2020-01-16T00:14:27.870 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1297999999999994e-05 2020-01-16T00:14:27.870 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.870 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.871 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:27.871 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.337165+0000", 2020-01-16T00:14:27.871 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013848911 2020-01-16T00:14:27.871 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.871 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.871 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.871 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351013+0000", 2020-01-16T00:14:27.871 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7732999999999994e-05 2020-01-16T00:14:27.872 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.872 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.872 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.872 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351111+0000", 2020-01-16T00:14:27.872 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000109089 2020-01-16T00:14:27.872 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.872 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.872 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:14:27.873 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351220+0000", 2020-01-16T00:14:27.873 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00063960199999999997 2020-01-16T00:14:27.873 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.873 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.873 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:27.873 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351860+0000", 2020-01-16T00:14:27.873 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00050205699999999998 2020-01-16T00:14:27.873 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.874 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.874 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.874 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352362+0000", 2020-01-16T00:14:27.874 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000282195 2020-01-16T00:14:27.874 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.874 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.874 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.875 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352644+0000", 2020-01-16T00:14:27.875 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3927000000000001e-05 2020-01-16T00:14:27.875 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.875 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.875 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:27.875 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352698+0000", 2020-01-16T00:14:27.875 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7770999999999999e-05 2020-01-16T00:14:27.875 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.876 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.876 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.876 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352766+0000", 2020-01-16T00:14:27.876 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016051421 2020-01-16T00:14:27.876 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.876 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.876 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.876 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.877 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.877 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4400 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e55)", 2020-01-16T00:14:27.877 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:58.336738+0000", 2020-01-16T00:14:27.877 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 29.371418029000001, 2020-01-16T00:14:27.877 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016207452000000001, 2020-01-16T00:14:27.877 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.877 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:27.878 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:27.878 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:27.878 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:27.879 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4400 2020-01-16T00:14:27.879 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.879 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.879 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.879 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.879 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336738+0000", 2020-01-16T00:14:27.880 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.880 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.880 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.880 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.880 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336738+0000", 2020-01-16T00:14:27.880 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:27.880 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.880 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.881 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.881 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336738+0000", 2020-01-16T00:14:27.881 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5200000000000002e-06 2020-01-16T00:14:27.881 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.881 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.881 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.881 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336741+0000", 2020-01-16T00:14:27.882 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1500000000000004e-06 2020-01-16T00:14:27.882 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.882 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.882 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.882 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336747+0000", 2020-01-16T00:14:27.882 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7500000000000001e-06 2020-01-16T00:14:27.882 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.882 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.883 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.883 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336751+0000", 2020-01-16T00:14:27.883 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043512299999999999 2020-01-16T00:14:27.883 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.883 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.883 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.883 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.337186+0000", 2020-01-16T00:14:27.884 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7048000000000002e-05 2020-01-16T00:14:27.884 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.884 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.884 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:27.884 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.337223+0000", 2020-01-16T00:14:27.884 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014258785 2020-01-16T00:14:27.884 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.884 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.885 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.885 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351482+0000", 2020-01-16T00:14:27.885 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012002000000000001 2020-01-16T00:14:27.885 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.885 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.885 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:27.885 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351602+0000", 2020-01-16T00:14:27.885 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001192404 2020-01-16T00:14:27.886 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.886 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.886 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.886 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352794+0000", 2020-01-16T00:14:27.886 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8901999999999998e-05 2020-01-16T00:14:27.886 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.886 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.887 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.887 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352873+0000", 2020-01-16T00:14:27.887 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2317999999999994e-05 2020-01-16T00:14:27.887 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.887 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.887 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.887 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352946+0000", 2020-01-16T00:14:27.887 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016207452000000001 2020-01-16T00:14:27.888 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.888 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.888 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.888 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.888 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.888 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:183 2.1 e58/19 2:a4df40e2:::smithi12313934-42 low_tier:head v 58'138, mlcod=57'136)", 2020-01-16T00:14:27.888 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.021967+0000", 2020-01-16T00:14:27.888 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 20.686189160000001, 2020-01-16T00:14:27.889 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011697308999999999, 2020-01-16T00:14:27.889 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.889 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:27.889 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.889 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.889 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.890 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021967+0000", 2020-01-16T00:14:27.890 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.890 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.890 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.891 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.891 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021967+0000", 2020-01-16T00:14:27.891 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:27.891 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.891 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.891 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.892 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021966+0000", 2020-01-16T00:14:27.892 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020837899999999999 2020-01-16T00:14:27.892 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.892 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.892 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.892 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.022175+0000", 2020-01-16T00:14:27.893 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7650000000000005e-06 2020-01-16T00:14:27.893 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.893 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.893 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.893 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.022185+0000", 2020-01-16T00:14:27.893 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.759e-06 2020-01-16T00:14:27.893 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.894 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.894 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.894 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.022191+0000", 2020-01-16T00:14:27.894 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00559108 2020-01-16T00:14:27.894 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.894 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.894 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.895 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.027782+0000", 2020-01-16T00:14:27.895 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5761e-05 2020-01-16T00:14:27.895 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.895 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.895 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.895 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.027798+0000", 2020-01-16T00:14:27.895 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0058039989999999998 2020-01-16T00:14:27.895 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.896 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.896 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:27.896 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.033602+0000", 2020-01-16T00:14:27.896 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2147e-05 2020-01-16T00:14:27.896 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.896 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.896 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.896 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.033664+0000", 2020-01-16T00:14:27.897 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011697308999999999 2020-01-16T00:14:27.897 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.897 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.897 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.897 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.897 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.897 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.1.13:98 2.1 e58/19 2:bafac7f5:::smithi12313934-46 low_tier:head v 58'140, mlcod=58'138)", 2020-01-16T00:14:27.898 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.050024+0000", 2020-01-16T00:14:27.898 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 20.658131802, 2020-01-16T00:14:27.898 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011307467, 2020-01-16T00:14:27.898 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.898 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:27.898 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.898 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.898 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.899 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.050024+0000", 2020-01-16T00:14:27.899 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.899 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.899 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.899 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.899 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.050024+0000", 2020-01-16T00:14:27.899 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:27.900 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.900 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.900 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.900 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.050022+0000", 2020-01-16T00:14:27.900 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000438839 2020-01-16T00:14:27.900 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.900 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.901 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.901 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.050461+0000", 2020-01-16T00:14:27.901 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9307999999999999e-05 2020-01-16T00:14:27.901 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.901 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.901 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.901 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.050480+0000", 2020-01-16T00:14:27.902 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7249999999999997e-06 2020-01-16T00:14:27.902 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.902 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.902 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.902 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.050488+0000", 2020-01-16T00:14:27.902 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00026342100000000001 2020-01-16T00:14:27.902 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.902 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.903 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.903 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.050751+0000", 2020-01-16T00:14:27.903 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8136000000000001e-05 2020-01-16T00:14:27.903 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.903 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.903 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.903 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.050769+0000", 2020-01-16T00:14:27.904 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010518582 2020-01-16T00:14:27.904 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.904 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.904 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:27.904 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.061288+0000", 2020-01-16T00:14:27.904 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4085000000000002e-05 2020-01-16T00:14:27.905 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.905 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.905 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.905 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.061332+0000", 2020-01-16T00:14:27.905 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011307467 2020-01-16T00:14:27.905 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.905 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.906 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.906 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.906 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.906 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:5566 3.1 3:8de1d8a3:::smithi12313934-13:head [write 568268~733811,stat] snapc 0=[] ondisk+write+known_if_redirected e66)", 2020-01-16T00:14:27.906 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:27.611601+0000", 2020-01-16T00:14:27.906 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.096555009999999997, 2020-01-16T00:14:27.906 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014368947, 2020-01-16T00:14:27.907 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.907 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:27.907 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:27.907 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:27.907 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:27.907 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 5566 2020-01-16T00:14:27.907 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.907 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.908 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.908 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.908 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.611601+0000", 2020-01-16T00:14:27.908 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.908 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.908 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.908 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.909 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.611601+0000", 2020-01-16T00:14:27.909 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:27.909 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.909 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.909 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.909 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.611598+0000", 2020-01-16T00:14:27.909 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022899199999999999 2020-01-16T00:14:27.910 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.910 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.910 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.910 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.611827+0000", 2020-01-16T00:14:27.910 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7127000000000001e-05 2020-01-16T00:14:27.910 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.910 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.911 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.911 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.611844+0000", 2020-01-16T00:14:27.911 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3100000000000001e-06 2020-01-16T00:14:27.911 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.912 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.912 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.912 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.611853+0000", 2020-01-16T00:14:27.912 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3891e-05 2020-01-16T00:14:27.912 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.912 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.912 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.913 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.611937+0000", 2020-01-16T00:14:27.913 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017038599999999999 2020-01-16T00:14:27.913 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.913 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.913 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.913 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.612107+0000", 2020-01-16T00:14:27.913 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00027321700000000002 2020-01-16T00:14:27.914 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.914 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.914 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:14:27.914 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.612380+0000", 2020-01-16T00:14:27.914 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0034700379999999999 2020-01-16T00:14:27.914 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.914 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.914 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.915 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.615850+0000", 2020-01-16T00:14:27.915 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00031966699999999999 2020-01-16T00:14:27.915 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.915 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.915 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:27.915 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.616170+0000", 2020-01-16T00:14:27.915 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0096376469999999992 2020-01-16T00:14:27.916 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.916 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.916 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.916 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.625807+0000", 2020-01-16T00:14:27.916 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2891e-05 2020-01-16T00:14:27.916 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.916 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.916 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:27.917 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.625850+0000", 2020-01-16T00:14:27.917 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119683 2020-01-16T00:14:27.917 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.917 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.917 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.917 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.625970+0000", 2020-01-16T00:14:27.917 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014368947 2020-01-16T00:14:27.918 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.918 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.918 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.918 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.918 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.918 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:5567 3.1 3:8de1d8a3:::smithi12313934-13:head [write 2000022~168246] snapc 0=[] ondisk+write+known_if_redirected e66)", 2020-01-16T00:14:27.919 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:27.616881+0000", 2020-01-16T00:14:27.919 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.091274689000000006, 2020-01-16T00:14:27.919 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013554490000000001, 2020-01-16T00:14:27.919 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.919 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:27.920 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:27.920 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:27.920 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:27.920 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 5567 2020-01-16T00:14:27.920 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.921 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.921 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.921 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.921 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.616881+0000", 2020-01-16T00:14:27.921 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.922 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.922 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.922 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.922 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.616881+0000", 2020-01-16T00:14:27.923 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:27.923 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.923 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.923 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.923 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.616878+0000", 2020-01-16T00:14:27.924 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3263999999999996e-05 2020-01-16T00:14:27.924 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.924 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.924 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.924 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.616941+0000", 2020-01-16T00:14:27.925 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8110000000000001e-05 2020-01-16T00:14:27.925 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.925 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.925 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.926 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.616959+0000", 2020-01-16T00:14:27.926 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9469999999999997e-06 2020-01-16T00:14:27.926 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.926 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.927 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.927 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.616966+0000", 2020-01-16T00:14:27.927 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8065000000000001e-05 2020-01-16T00:14:27.927 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.927 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.927 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.927 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.617015+0000", 2020-01-16T00:14:27.927 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013213400000000001 2020-01-16T00:14:27.928 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.928 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.928 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:27.928 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.617147+0000", 2020-01-16T00:14:27.928 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0088762949999999993 2020-01-16T00:14:27.928 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.928 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.929 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.929 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.626023+0000", 2020-01-16T00:14:27.929 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018361500000000001 2020-01-16T00:14:27.929 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.929 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.929 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.930 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.626207+0000", 2020-01-16T00:14:27.930 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000209936 2020-01-16T00:14:27.930 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.930 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.930 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:14:27.930 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.626416+0000", 2020-01-16T00:14:27.930 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0035327610000000001 2020-01-16T00:14:27.931 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.931 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.931 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.931 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.629949+0000", 2020-01-16T00:14:27.931 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00033993400000000002 2020-01-16T00:14:27.931 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.931 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.931 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.932 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.630289+0000", 2020-01-16T00:14:27.932 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0885e-05 2020-01-16T00:14:27.932 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.932 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.932 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:27.932 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.630300+0000", 2020-01-16T00:14:27.933 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3071000000000007e-05 2020-01-16T00:14:27.933 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.933 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.933 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:27.933 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.630363+0000", 2020-01-16T00:14:27.933 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2787999999999995e-05 2020-01-16T00:14:27.933 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.934 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.934 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.934 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.630436+0000", 2020-01-16T00:14:27.934 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013554490000000001 2020-01-16T00:14:27.934 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.934 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.934 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.934 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.935 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.935 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:5568 3.1 3:8de1d8a3:::smithi12313934-13:head [setxattr _header (56) in=7b,truncate 2168268] snapc 0=[] ondisk+write+known_if_redirected e66)", 2020-01-16T00:14:27.935 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:27.616979+0000", 2020-01-16T00:14:27.935 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.091176955000000004, 2020-01-16T00:14:27.935 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018308560000000001, 2020-01-16T00:14:27.935 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.935 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:27.936 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:27.936 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:27.936 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:27.936 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 5568 2020-01-16T00:14:27.936 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.936 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.936 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.936 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.937 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.616979+0000", 2020-01-16T00:14:27.937 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.937 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.937 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.937 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.937 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.616979+0000", 2020-01-16T00:14:27.937 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:27.938 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.938 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.938 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.938 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.616978+0000", 2020-01-16T00:14:27.938 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4280000000000001e-06 2020-01-16T00:14:27.938 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.938 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.939 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.939 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.616985+0000", 2020-01-16T00:14:27.939 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7630000000000003e-06 2020-01-16T00:14:27.939 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.939 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.940 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.940 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.616992+0000", 2020-01-16T00:14:27.940 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6639999999999997e-06 2020-01-16T00:14:27.940 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.940 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.941 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.941 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.616996+0000", 2020-01-16T00:14:27.941 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020064400000000001 2020-01-16T00:14:27.941 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.942 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.942 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.942 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.617197+0000", 2020-01-16T00:14:27.942 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000149071 2020-01-16T00:14:27.943 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.943 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.943 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:27.943 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.617346+0000", 2020-01-16T00:14:27.944 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010707899 2020-01-16T00:14:27.944 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.944 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.944 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.944 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.628054+0000", 2020-01-16T00:14:27.945 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013353 2020-01-16T00:14:27.945 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.945 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.945 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.946 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.628187+0000", 2020-01-16T00:14:27.946 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000184432 2020-01-16T00:14:27.946 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.946 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.947 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:14:27.947 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.628372+0000", 2020-01-16T00:14:27.947 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0022270200000000001 2020-01-16T00:14:27.947 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.947 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.948 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.948 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.630599+0000", 2020-01-16T00:14:27.948 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000115078 2020-01-16T00:14:27.948 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.949 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.949 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:27.949 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.630714+0000", 2020-01-16T00:14:27.949 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0043920080000000002 2020-01-16T00:14:27.949 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.950 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.950 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:27.950 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.635106+0000", 2020-01-16T00:14:27.950 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8524999999999998e-05 2020-01-16T00:14:27.950 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.951 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.951 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:27.951 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.635184+0000", 2020-01-16T00:14:27.951 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010336300000000001 2020-01-16T00:14:27.952 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.952 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.952 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.952 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.635288+0000", 2020-01-16T00:14:27.952 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018308560000000001 2020-01-16T00:14:27.953 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.953 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.953 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.953 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.954 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.954 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:5569 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e66)", 2020-01-16T00:14:27.954 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:27.617006+0000", 2020-01-16T00:14:27.954 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 0.091150473999999995, 2020-01-16T00:14:27.954 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018588490999999999, 2020-01-16T00:14:27.955 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:27.955 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:27.955 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:27.955 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:27.956 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:27.956 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 5569 2020-01-16T00:14:27.956 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.956 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:27.957 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.957 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:27.957 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.617006+0000", 2020-01-16T00:14:27.957 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:27.957 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.958 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.958 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:27.958 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.617006+0000", 2020-01-16T00:14:27.958 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:27.959 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.959 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.959 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:27.959 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.617005+0000", 2020-01-16T00:14:27.959 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8569999999999997e-06 2020-01-16T00:14:27.959 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.960 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.960 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:27.960 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.617012+0000", 2020-01-16T00:14:27.960 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.669e-06 2020-01-16T00:14:27.960 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.960 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.960 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:27.961 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.617019+0000", 2020-01-16T00:14:27.961 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.34e-06 2020-01-16T00:14:27.961 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.961 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.961 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:27.961 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.617023+0000", 2020-01-16T00:14:27.961 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035691299999999999 2020-01-16T00:14:27.961 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.962 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.962 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.962 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.617380+0000", 2020-01-16T00:14:27.962 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000137615 2020-01-16T00:14:27.962 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.962 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.962 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:27.963 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.617517+0000", 2020-01-16T00:14:27.963 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.01195509 2020-01-16T00:14:27.963 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.963 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.963 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.963 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.629473+0000", 2020-01-16T00:14:27.963 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012814200000000001 2020-01-16T00:14:27.963 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.964 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.964 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:27.964 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.629601+0000", 2020-01-16T00:14:27.964 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0057381439999999997 2020-01-16T00:14:27.964 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.964 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.964 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:27.965 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.635339+0000", 2020-01-16T00:14:27.965 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012998600000000001 2020-01-16T00:14:27.965 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.965 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.965 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:27.965 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.635469+0000", 2020-01-16T00:14:27.965 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012535000000000001 2020-01-16T00:14:27.966 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:27.966 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:27.966 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:27.966 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.635594+0000", 2020-01-16T00:14:27.966 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018588490999999999 2020-01-16T00:14:27.966 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.966 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.966 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.967 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:27.967 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:27.967 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:27.970 INFO:tasks.rados.rados.0.smithi123.stdout:1913: done (15 left) 2020-01-16T00:14:27.971 INFO:tasks.rados.rados.0.smithi123.stdout:1914: done (14 left) 2020-01-16T00:14:27.971 INFO:tasks.rados.rados.0.smithi123.stdout:1915: done (13 left) 2020-01-16T00:14:27.971 INFO:tasks.rados.rados.0.smithi123.stdout:1916: done (12 left) 2020-01-16T00:14:27.971 INFO:tasks.rados.rados.0.smithi123.stdout:1917: done (11 left) 2020-01-16T00:14:27.971 INFO:tasks.rados.rados.0.smithi123.stdout:1928: done (10 left) 2020-01-16T00:14:27.971 INFO:tasks.rados.rados.0.smithi123.stdout:1929: read oid 42 snap -1 2020-01-16T00:14:27.971 INFO:tasks.rados.rados.0.smithi123.stdout:1929: expect (ObjNum 705 snap 0 seq_num 705) 2020-01-16T00:14:27.972 INFO:tasks.rados.rados.0.smithi123.stdout:1930: read oid 10 snap -1 2020-01-16T00:14:27.972 INFO:tasks.rados.rados.0.smithi123.stdout:1930: expect (ObjNum 698 snap 0 seq_num 698) 2020-01-16T00:14:27.972 INFO:tasks.rados.rados.0.smithi123.stdout:1931: read oid 20 snap -1 2020-01-16T00:14:27.972 INFO:tasks.rados.rados.0.smithi123.stdout:1931: expect (ObjNum 713 snap 0 seq_num 713) 2020-01-16T00:14:27.972 INFO:tasks.rados.rados.0.smithi123.stdout:1932: write oid 26 current snap is 0 2020-01-16T00:14:27.972 INFO:tasks.rados.rados.0.smithi123.stdout:1932: seq_num 727 ranges {618689=559558,1659115=559574} 2020-01-16T00:14:27.972 INFO:tasks.rados.rados.0.smithi123.stdout:1932: writing smithi12313934-26 from 618689 to 1178247 tid 1 2020-01-16T00:14:27.973 INFO:tasks.rados.rados.0.smithi123.stdout:1932: writing smithi12313934-26 from 1659115 to 2218689 tid 2 2020-01-16T00:14:27.973 INFO:tasks.rados.rados.0.smithi123.stdout:1933: read oid 6 snap -1 2020-01-16T00:14:27.973 INFO:tasks.rados.rados.0.smithi123.stdout:1933: expect (ObjNum 720 snap 0 seq_num 720) 2020-01-16T00:14:27.973 INFO:tasks.rados.rados.0.smithi123.stdout:1934: write oid 49 current snap is 0 2020-01-16T00:14:27.973 INFO:tasks.rados.rados.0.smithi123.stdout:1934: seq_num 728 ranges {635496=634807,1717597=517899} 2020-01-16T00:14:27.973 INFO:tasks.rados.rados.0.smithi123.stdout:1934: writing smithi12313934-49 from 635496 to 1270303 tid 1 2020-01-16T00:14:27.974 INFO:tasks.rados.rados.0.smithi123.stdout:1934: writing smithi12313934-49 from 1717597 to 2235496 tid 2 2020-01-16T00:14:27.974 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:27.974 INFO:tasks.rados.rados.0.smithi123.stdout:1918: expect (ObjNum 579 snap 0 seq_num 579) 2020-01-16T00:14:27.974 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:14:27.974 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:14:27.974 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:14:27.974 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:14:27.975 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:14:27.975 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:14:27.975 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:14:27.975 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-01-16T00:14:27.975 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-01-16T00:14:27.975 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-01-16T00:14:27.975 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-01-16T00:14:27.975 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-01-16T00:14:27.976 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENXIO: problem getting command descriptions from osd.5 2020-01-16T00:14:27.976 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: problem getting command descriptions from osd.5 2020-01-16T00:14:27.976 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-01-16T00:14:27.976 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-01-16T00:14:27.976 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-01-16T00:14:27.976 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-01-16T00:14:27.977 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:27.966+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:27.977 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-01-16T00:14:28.057 INFO:tasks.rados.rados.0.smithi123.stdout:1921: expect (ObjNum 691 snap 0 seq_num 691) 2020-01-16T00:14:28.069 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:28.066+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:28.101 INFO:teuthology.orchestra.run.smithi123.stderr:nodeep-scrub is unset 2020-01-16T00:14:28.116 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:28.125 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-01-16T00:14:28.168 INFO:tasks.rados.rados.0.smithi123.stdout:1920: expect (ObjNum 667 snap 0 seq_num 667) 2020-01-16T00:14:28.171 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:28.168+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:28.266 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:28.272 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:28.270+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:28.275 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-01-16T00:14:28.373 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:28.368+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:28.418 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:28.428 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-01-16T00:14:28.474 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:28.472+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:28.514 INFO:tasks.rados.rados.0.smithi123.stdout:1922: finishing write tid 1 to smithi12313934-28 2020-01-16T00:14:28.514 INFO:tasks.rados.rados.0.smithi123.stdout:1922: finishing write tid 2 to smithi12313934-28 2020-01-16T00:14:28.515 INFO:tasks.rados.rados.0.smithi123.stdout:1922: finishing write tid 4 to smithi12313934-28 2020-01-16T00:14:28.515 INFO:tasks.rados.rados.0.smithi123.stdout:1918: done (15 left) 2020-01-16T00:14:28.515 INFO:tasks.rados.rados.0.smithi123.stdout:1919: done (14 left) 2020-01-16T00:14:28.515 INFO:tasks.rados.rados.0.smithi123.stdout:1920: done (13 left) 2020-01-16T00:14:28.515 INFO:tasks.rados.rados.0.smithi123.stdout:1921: done (12 left) 2020-01-16T00:14:28.515 INFO:tasks.rados.rados.0.smithi123.stdout:1935: read oid 12 snap -1 2020-01-16T00:14:28.515 INFO:tasks.rados.rados.0.smithi123.stdout:1935: expect (ObjNum 655 snap 0 seq_num 655) 2020-01-16T00:14:28.515 INFO:tasks.rados.rados.0.smithi123.stdout:1936: write oid 1 current snap is 0 2020-01-16T00:14:28.516 INFO:tasks.rados.rados.0.smithi123.stdout:1936: seq_num 729 ranges {652303=710056,2059726=192577} 2020-01-16T00:14:28.534 INFO:tasks.rados.rados.0.smithi123.stdout:1936: writing smithi12313934-1 from 652303 to 1362359 tid 1 2020-01-16T00:14:28.534 INFO:tasks.rados.rados.0.smithi123.stdout:1936: writing smithi12313934-1 from 2059726 to 2252303 tid 2 2020-01-16T00:14:28.534 INFO:tasks.rados.rados.0.smithi123.stdout:1937: write (excl) oid 19 current snap is 0 2020-01-16T00:14:28.534 INFO:tasks.rados.rados.0.smithi123.stdout:1937: seq_num 730 ranges {669110=501658,1834561=434549} 2020-01-16T00:14:28.534 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:28.539 INFO:tasks.rados.rados.0.smithi123.stdout:1937: writing smithi12313934-19 from 669110 to 1170768 tid 1 2020-01-16T00:14:28.539 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-01-16T00:14:28.545 INFO:tasks.rados.rados.0.smithi123.stdout:1937: writing smithi12313934-19 from 1834561 to 2269110 tid 2 2020-01-16T00:14:28.545 INFO:tasks.rados.rados.0.smithi123.stdout:1938: write oid 27 current snap is 0 2020-01-16T00:14:28.545 INFO:tasks.rados.rados.0.smithi123.stdout:1938: seq_num 731 ranges {685917=576907,1893043=392874} 2020-01-16T00:14:28.552 INFO:tasks.rados.rados.0.smithi123.stdout:1938: writing smithi12313934-27 from 685917 to 1262824 tid 1 2020-01-16T00:14:28.560 INFO:tasks.rados.rados.0.smithi123.stdout:1938: writing smithi12313934-27 from 1893043 to 2285917 tid 2 2020-01-16T00:14:28.560 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:28.560 INFO:tasks.rados.rados.0.smithi123.stdout:1922: finishing write tid 5 to smithi12313934-28 2020-01-16T00:14:28.561 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 28 v 248 (ObjNum 722 snap 0 seq_num 722) dirty exists 2020-01-16T00:14:28.561 INFO:tasks.rados.rados.0.smithi123.stdout:1922: left oid 28 (ObjNum 722 snap 0 seq_num 722) 2020-01-16T00:14:28.561 INFO:tasks.rados.rados.0.smithi123.stdout:1922: done (15 left) 2020-01-16T00:14:28.561 INFO:tasks.rados.rados.0.smithi123.stdout:1939: read oid 21 snap -1 2020-01-16T00:14:28.561 INFO:tasks.rados.rados.0.smithi123.stdout:1939: expect (ObjNum 709 snap 0 seq_num 709) 2020-01-16T00:14:28.561 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:28.561 INFO:tasks.rados.rados.0.smithi123.stdout:1923: finishing write tid 1 to smithi12313934-46 2020-01-16T00:14:28.562 INFO:tasks.rados.rados.0.smithi123.stdout:1923: finishing write tid 2 to smithi12313934-46 2020-01-16T00:14:28.562 INFO:tasks.rados.rados.0.smithi123.stdout:1923: finishing write tid 4 to smithi12313934-46 2020-01-16T00:14:28.562 INFO:tasks.rados.rados.0.smithi123.stdout:1923: finishing write tid 5 to smithi12313934-46 2020-01-16T00:14:28.562 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 46 v 373 (ObjNum 723 snap 0 seq_num 723) dirty exists 2020-01-16T00:14:28.562 INFO:tasks.rados.rados.0.smithi123.stdout:1923: left oid 46 (ObjNum 723 snap 0 seq_num 723) 2020-01-16T00:14:28.562 INFO:tasks.rados.rados.0.smithi123.stdout:1924: finishing write tid 1 to smithi12313934-13 2020-01-16T00:14:28.562 INFO:tasks.rados.rados.0.smithi123.stdout:1926: finishing write tid 1 to smithi12313934-48 2020-01-16T00:14:28.562 INFO:tasks.rados.rados.0.smithi123.stdout:1924: finishing write tid 2 to smithi12313934-13 2020-01-16T00:14:28.563 INFO:tasks.rados.rados.0.smithi123.stdout:1925: expect (ObjNum 662 snap 0 seq_num 662) 2020-01-16T00:14:28.575 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:28.574+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:28.631 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:28.640 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-01-16T00:14:28.683 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:28.682+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:28.769 INFO:tasks.rados.rados.0.smithi123.stdout:1924: finishing write tid 4 to smithi12313934-13 2020-01-16T00:14:28.770 INFO:tasks.rados.rados.0.smithi123.stdout:1924: finishing write tid 5 to smithi12313934-13 2020-01-16T00:14:28.770 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 135 (ObjNum 724 snap 0 seq_num 724) dirty exists 2020-01-16T00:14:28.770 INFO:tasks.rados.rados.0.smithi123.stdout:1924: left oid 13 (ObjNum 724 snap 0 seq_num 724) 2020-01-16T00:14:28.770 INFO:tasks.rados.rados.0.smithi123.stdout:1923: done (15 left) 2020-01-16T00:14:28.770 INFO:tasks.rados.rados.0.smithi123.stdout:1924: done (14 left) 2020-01-16T00:14:28.770 INFO:tasks.rados.rados.0.smithi123.stdout:1925: done (13 left) 2020-01-16T00:14:28.770 INFO:tasks.rados.rados.0.smithi123.stdout:1940: delete oid 32 current snap is 0 2020-01-16T00:14:28.771 INFO:tasks.rados.rados.0.smithi123.stdout:1926: finishing write tid 2 to smithi12313934-48 2020-01-16T00:14:28.771 INFO:tasks.rados.rados.0.smithi123.stdout:1927: expect (ObjNum 717 snap 0 seq_num 717) 2020-01-16T00:14:28.778 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:28.776+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:28.780 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:28.787 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-01-16T00:14:28.878 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:28.876+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:28.904 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:28.910 INFO:tasks.rados.rados.0.smithi123.stdout:1926: finishing write tid 4 to smithi12313934-48 2020-01-16T00:14:28.910 INFO:tasks.rados.rados.0.smithi123.stdout:1926: finishing write tid 5 to smithi12313934-48 2020-01-16T00:14:28.911 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 48 v 238 (ObjNum 725 snap 0 seq_num 725) dirty exists 2020-01-16T00:14:28.911 INFO:tasks.rados.rados.0.smithi123.stdout:1926: left oid 48 (ObjNum 725 snap 0 seq_num 725) 2020-01-16T00:14:28.911 INFO:tasks.rados.rados.0.smithi123.stdout:1931: expect (ObjNum 713 snap 0 seq_num 713) 2020-01-16T00:14:28.914 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-01-16T00:14:29.001 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:29.000+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:29.066 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:29.075 INFO:tasks.rados.rados.0.smithi123.stdout:1929: expect (ObjNum 705 snap 0 seq_num 705) 2020-01-16T00:14:29.076 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-01-16T00:14:29.079 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:29.078+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:29.183 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:29.180+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:29.228 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:29.238 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-01-16T00:14:29.285 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:29.282+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:29.297 INFO:tasks.rados.rados.0.smithi123.stdout:1932: finishing write tid 1 to smithi12313934-26 2020-01-16T00:14:29.298 INFO:tasks.rados.rados.0.smithi123.stdout:1932: finishing write tid 2 to smithi12313934-26 2020-01-16T00:14:29.298 INFO:tasks.rados.rados.0.smithi123.stdout:1930: expect (ObjNum 698 snap 0 seq_num 698) 2020-01-16T00:14:29.385 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:29.382+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:29.391 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:29.398 INFO:tasks.rados.rados.0.smithi123.stdout:1933: expect (ObjNum 720 snap 0 seq_num 720) 2020-01-16T00:14:29.400 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-01-16T00:14:29.502 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:29.502+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:29.503 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:29.512 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-01-16T00:14:29.604 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:29.600+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:29.604 INFO:tasks.rados.rados.0.smithi123.stdout:1934: finishing write tid 1 to smithi12313934-49 2020-01-16T00:14:29.604 INFO:tasks.rados.rados.0.smithi123.stdout:1932: finishing write tid 4 to smithi12313934-26 2020-01-16T00:14:29.605 INFO:tasks.rados.rados.0.smithi123.stdout:1932: finishing write tid 5 to smithi12313934-26 2020-01-16T00:14:29.605 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 26 v 377 (ObjNum 727 snap 0 seq_num 727) dirty exists 2020-01-16T00:14:29.605 INFO:tasks.rados.rados.0.smithi123.stdout:1932: left oid 26 (ObjNum 727 snap 0 seq_num 727) 2020-01-16T00:14:29.605 INFO:tasks.rados.rados.0.smithi123.stdout:1934: finishing write tid 2 to smithi12313934-49 2020-01-16T00:14:29.605 INFO:tasks.rados.rados.0.smithi123.stdout:1934: finishing write tid 4 to smithi12313934-49 2020-01-16T00:14:29.605 INFO:tasks.rados.rados.0.smithi123.stdout:1934: finishing write tid 5 to smithi12313934-49 2020-01-16T00:14:29.605 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 49 v 201 (ObjNum 728 snap 0 seq_num 728) dirty exists 2020-01-16T00:14:29.606 INFO:tasks.rados.rados.0.smithi123.stdout:1934: left oid 49 (ObjNum 728 snap 0 seq_num 728) 2020-01-16T00:14:29.606 INFO:tasks.rados.rados.0.smithi123.stdout:1935: expect (ObjNum 655 snap 0 seq_num 655) 2020-01-16T00:14:29.617 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:29.628 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-01-16T00:14:29.690 INFO:tasks.rados.rados.0.smithi123.stdout:1936: finishing write tid 1 to smithi12313934-1 2020-01-16T00:14:29.690 INFO:tasks.rados.rados.0.smithi123.stdout:1936: finishing write tid 2 to smithi12313934-1 2020-01-16T00:14:29.690 INFO:tasks.rados.rados.0.smithi123.stdout:1936: finishing write tid 4 to smithi12313934-1 2020-01-16T00:14:29.690 INFO:tasks.rados.rados.0.smithi123.stdout:1936: finishing write tid 5 to smithi12313934-1 2020-01-16T00:14:29.691 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 1 v 380 (ObjNum 729 snap 0 seq_num 729) dirty exists 2020-01-16T00:14:29.691 INFO:tasks.rados.rados.0.smithi123.stdout:1936: left oid 1 (ObjNum 729 snap 0 seq_num 729) 2020-01-16T00:14:29.691 INFO:tasks.rados.rados.0.smithi123.stdout:1937: finishing write tid 1 to smithi12313934-19 2020-01-16T00:14:29.691 INFO:tasks.rados.rados.0.smithi123.stdout:1937: finishing write tid 2 to smithi12313934-19 2020-01-16T00:14:29.691 INFO:tasks.rados.rados.0.smithi123.stdout:1937: finishing write tid 4 to smithi12313934-19 2020-01-16T00:14:29.691 INFO:tasks.rados.rados.0.smithi123.stdout:1937: finishing write tid 5 to smithi12313934-19 2020-01-16T00:14:29.691 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 19 v 273 (ObjNum 730 snap 0 seq_num 730) dirty exists 2020-01-16T00:14:29.691 INFO:tasks.rados.rados.0.smithi123.stdout:1937: left oid 19 (ObjNum 730 snap 0 seq_num 730) 2020-01-16T00:14:29.692 INFO:tasks.rados.rados.0.smithi123.stdout:1938: finishing write tid 1 to smithi12313934-27 2020-01-16T00:14:29.692 INFO:tasks.rados.rados.0.smithi123.stdout:1938: finishing write tid 2 to smithi12313934-27 2020-01-16T00:14:29.692 INFO:tasks.rados.rados.0.smithi123.stdout:1926: done (13 left) 2020-01-16T00:14:29.692 INFO:tasks.rados.rados.0.smithi123.stdout:1927: done (12 left) 2020-01-16T00:14:29.692 INFO:tasks.rados.rados.0.smithi123.stdout:1929: done (11 left) 2020-01-16T00:14:29.692 INFO:tasks.rados.rados.0.smithi123.stdout:1930: done (10 left) 2020-01-16T00:14:29.692 INFO:tasks.rados.rados.0.smithi123.stdout:1931: done (9 left) 2020-01-16T00:14:29.692 INFO:tasks.rados.rados.0.smithi123.stdout:1932: done (8 left) 2020-01-16T00:14:29.693 INFO:tasks.rados.rados.0.smithi123.stdout:1933: done (7 left) 2020-01-16T00:14:29.693 INFO:tasks.rados.rados.0.smithi123.stdout:1934: done (6 left) 2020-01-16T00:14:29.693 INFO:tasks.rados.rados.0.smithi123.stdout:1935: done (5 left) 2020-01-16T00:14:29.693 INFO:tasks.rados.rados.0.smithi123.stdout:1936: done (4 left) 2020-01-16T00:14:29.693 INFO:tasks.rados.rados.0.smithi123.stdout:1937: done (3 left) 2020-01-16T00:14:29.693 INFO:tasks.rados.rados.0.smithi123.stdout:1940: done (2 left) 2020-01-16T00:14:29.693 INFO:tasks.rados.rados.0.smithi123.stdout:1941: write (excl) oid 50 current snap is 0 2020-01-16T00:14:29.694 INFO:tasks.rados.rados.0.smithi123.stdout:1938: finishing write tid 4 to smithi12313934-27 2020-01-16T00:14:29.694 INFO:tasks.rados.rados.0.smithi123.stdout:1938: finishing write tid 5 to smithi12313934-27 2020-01-16T00:14:29.694 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 27 v 192 (ObjNum 731 snap 0 seq_num 731) dirty exists 2020-01-16T00:14:29.694 INFO:tasks.rados.rados.0.smithi123.stdout:1938: left oid 27 (ObjNum 731 snap 0 seq_num 731) 2020-01-16T00:14:29.694 INFO:tasks.rados.rados.0.smithi123.stdout:1939: expect (ObjNum 709 snap 0 seq_num 709) 2020-01-16T00:14:29.705 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:29.702+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:29.779 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:29.789 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-01-16T00:14:29.806 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:29.802+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:29.890 INFO:tasks.rados.rados.0.smithi123.stdout:1941: seq_num 733 ranges {719531=727405,1893654=425877} 2020-01-16T00:14:29.896 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:29.906 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-01-16T00:14:29.908 INFO:tasks.rados.rados.0.smithi123.stdout:1941: writing smithi12313934-50 from 719531 to 1446936 tid 1 2020-01-16T00:14:29.908 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:29.906+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:29.915 INFO:tasks.rados.rados.0.smithi123.stdout:1941: writing smithi12313934-50 from 1893654 to 2319531 tid 2 2020-01-16T00:14:29.916 INFO:tasks.rados.rados.0.smithi123.stdout:1938: done (2 left) 2020-01-16T00:14:29.916 INFO:tasks.rados.rados.0.smithi123.stdout:1939: done (1 left) 2020-01-16T00:14:29.916 INFO:tasks.rados.rados.0.smithi123.stdout:1942: write oid 34 current snap is 0 2020-01-16T00:14:29.916 INFO:tasks.rados.rados.0.smithi123.stdout:1942: seq_num 734 ranges {736338=402654,1552136=539580} 2020-01-16T00:14:29.919 INFO:tasks.rados.rados.0.smithi123.stdout:1942: writing smithi12313934-34 from 736338 to 1138992 tid 1 2020-01-16T00:14:29.926 INFO:tasks.rados.rados.0.smithi123.stdout:1942: writing smithi12313934-34 from 1552136 to 2091716 tid 2 2020-01-16T00:14:29.926 INFO:tasks.rados.rados.0.smithi123.stdout:1943: write (excl) oid 35 current snap is 0 2020-01-16T00:14:29.926 INFO:tasks.rados.rados.0.smithi123.stdout:1943: seq_num 735 ranges {753145=477903,2010618=342527} 2020-01-16T00:14:29.932 INFO:tasks.rados.rados.0.smithi123.stdout:1943: writing smithi12313934-35 from 753145 to 1231048 tid 1 2020-01-16T00:14:29.938 INFO:tasks.rados.rados.0.smithi123.stdout:1943: writing smithi12313934-35 from 2010618 to 2353145 tid 2 2020-01-16T00:14:29.938 INFO:tasks.rados.rados.0.smithi123.stdout:1944: write oid 7 current snap is 0 2020-01-16T00:14:29.938 INFO:tasks.rados.rados.0.smithi123.stdout:1944: seq_num 736 ranges {769952=553152,2069100=300852} 2020-01-16T00:14:29.945 INFO:tasks.rados.rados.0.smithi123.stdout:1944: writing smithi12313934-7 from 769952 to 1323104 tid 1 2020-01-16T00:14:29.952 INFO:tasks.rados.rados.0.smithi123.stdout:1944: writing smithi12313934-7 from 2069100 to 2369952 tid 2 2020-01-16T00:14:29.952 INFO:tasks.rados.rados.0.smithi123.stdout:1945: read oid 25 snap -1 2020-01-16T00:14:29.953 INFO:tasks.rados.rados.0.smithi123.stdout:1945: expect (ObjNum 719 snap 0 seq_num 719) 2020-01-16T00:14:29.953 INFO:tasks.rados.rados.0.smithi123.stdout:1941: finishing write tid 1 to smithi12313934-50 2020-01-16T00:14:29.953 INFO:tasks.rados.rados.0.smithi123.stdout:1946: read oid 11 snap -1 2020-01-16T00:14:29.953 INFO:tasks.rados.rados.0.smithi123.stdout:1946: expect (ObjNum 579 snap 0 seq_num 579) 2020-01-16T00:14:29.953 INFO:tasks.rados.rados.0.smithi123.stdout:1947: read oid 38 snap -1 2020-01-16T00:14:29.953 INFO:tasks.rados.rados.0.smithi123.stdout:1947: expect (ObjNum 718 snap 0 seq_num 718) 2020-01-16T00:14:29.953 INFO:tasks.rados.rados.0.smithi123.stdout:1948: read oid 32 snap -1 2020-01-16T00:14:29.954 INFO:tasks.rados.rados.0.smithi123.stdout:1948: expect deleted 2020-01-16T00:14:29.954 INFO:tasks.rados.rados.0.smithi123.stdout:1949: read oid 37 snap -1 2020-01-16T00:14:29.954 INFO:tasks.rados.rados.0.smithi123.stdout:1949: expect (ObjNum 693 snap 0 seq_num 693) 2020-01-16T00:14:29.954 INFO:tasks.rados.rados.0.smithi123.stdout:1950: read oid 13 snap -1 2020-01-16T00:14:29.954 INFO:tasks.rados.rados.0.smithi123.stdout:1950: expect (ObjNum 724 snap 0 seq_num 724) 2020-01-16T00:14:29.954 INFO:tasks.rados.rados.0.smithi123.stdout:1951: read oid 41 snap -1 2020-01-16T00:14:29.955 INFO:tasks.rados.rados.0.smithi123.stdout:1951: expect deleted 2020-01-16T00:14:29.955 INFO:tasks.rados.rados.0.smithi123.stdout:1952: write oid 39 current snap is 0 2020-01-16T00:14:29.955 INFO:tasks.rados.rados.0.smithi123.stdout:1952: seq_num 737 ranges {786759=628401,2011229=375530} 2020-01-16T00:14:29.960 INFO:tasks.rados.rados.0.smithi123.stdout:1952: writing smithi12313934-39 from 786759 to 1415160 tid 1 2020-01-16T00:14:29.967 INFO:tasks.rados.rados.0.smithi123.stdout:1952: writing smithi12313934-39 from 2011229 to 2386759 tid 2 2020-01-16T00:14:29.967 INFO:tasks.rados.rados.0.smithi123.stdout:1953: write oid 33 current snap is 0 2020-01-16T00:14:29.967 INFO:tasks.rados.rados.0.smithi123.stdout:1953: seq_num 738 ranges {403566=420003,1386064=546918} 2020-01-16T00:14:29.971 INFO:tasks.rados.rados.0.smithi123.stdout:1953: writing smithi12313934-33 from 403566 to 823569 tid 1 2020-01-16T00:14:29.977 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-01-16T00:14:29.978 INFO:tasks.rados.rados.0.smithi123.stdout:1953: writing smithi12313934-33 from 1386064 to 1932982 tid 2 2020-01-16T00:14:29.978 INFO:tasks.rados.rados.0.smithi123.stdout:1954: write (excl) oid 5 current snap is 0 2020-01-16T00:14:29.978 INFO:tasks.rados.rados.0.smithi123.stdout:1954: seq_num 739 ranges {420373=495252,1444546=690576} 2020-01-16T00:14:29.983 INFO:tasks.rados.rados.0.smithi123.stdout:1954: writing smithi12313934-5 from 420373 to 915625 tid 1 2020-01-16T00:14:29.990 INFO:tasks.rados.rados.0.smithi123.stdout:1954: writing smithi12313934-5 from 1444546 to 2135122 tid 2 2020-01-16T00:14:29.991 INFO:tasks.rados.rados.0.smithi123.stdout:1955: write (excl) oid 12 current snap is 0 2020-01-16T00:14:29.991 INFO:tasks.rados.rados.0.smithi123.stdout:1955: seq_num 740 ranges {437180=570501,1503028=550587} 2020-01-16T00:14:29.996 INFO:tasks.rados.rados.0.smithi123.stdout:1955: writing smithi12313934-12 from 437180 to 1007681 tid 1 2020-01-16T00:14:30.002 INFO:tasks.rados.rados.0.smithi123.stdout:1955: writing smithi12313934-12 from 1503028 to 2053615 tid 2 2020-01-16T00:14:30.002 INFO:tasks.rados.rados.0.smithi123.stdout:1956: write (excl) oid 8 current snap is 0 2020-01-16T00:14:30.003 INFO:tasks.rados.rados.0.smithi123.stdout:1956: seq_num 741 ranges {453987=645750,1845157=608830} 2020-01-16T00:14:30.008 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:30.006+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:30.008 INFO:tasks.rados.rados.0.smithi123.stdout:1956: writing smithi12313934-8 from 453987 to 1099737 tid 1 2020-01-16T00:14:30.018 INFO:tasks.rados.rados.0.smithi123.stdout:1956: writing smithi12313934-8 from 1845157 to 2453987 tid 2 2020-01-16T00:14:30.018 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:30.018 INFO:tasks.rados.rados.0.smithi123.stdout:1941: finishing write tid 2 to smithi12313934-50 2020-01-16T00:14:30.018 INFO:tasks.rados.rados.0.smithi123.stdout:1941: finishing write tid 4 to smithi12313934-50 2020-01-16T00:14:30.019 INFO:tasks.rados.rados.0.smithi123.stdout:1941: finishing write tid 5 to smithi12313934-50 2020-01-16T00:14:30.019 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 50 v 251 (ObjNum 733 snap 0 seq_num 733) dirty exists 2020-01-16T00:14:30.019 INFO:tasks.rados.rados.0.smithi123.stdout:1941: left oid 50 (ObjNum 733 snap 0 seq_num 733) 2020-01-16T00:14:30.019 INFO:tasks.rados.rados.0.smithi123.stdout:1942: finishing write tid 1 to smithi12313934-34 2020-01-16T00:14:30.019 INFO:tasks.rados.rados.0.smithi123.stdout:1941: done (15 left) 2020-01-16T00:14:30.019 INFO:tasks.rados.rados.0.smithi123.stdout:1957: read oid 2 snap -1 2020-01-16T00:14:30.020 INFO:tasks.rados.rados.0.smithi123.stdout:1957: expect (ObjNum 667 snap 0 seq_num 667) 2020-01-16T00:14:30.020 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:30.020 INFO:tasks.rados.rados.0.smithi123.stdout:1942: finishing write tid 2 to smithi12313934-34 2020-01-16T00:14:30.021 INFO:tasks.rados.rados.0.smithi123.stdout:1942: finishing write tid 4 to smithi12313934-34 2020-01-16T00:14:30.021 INFO:tasks.rados.rados.0.smithi123.stdout:1942: finishing write tid 5 to smithi12313934-34 2020-01-16T00:14:30.021 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 34 v 96 (ObjNum 734 snap 0 seq_num 734) dirty exists 2020-01-16T00:14:30.021 INFO:tasks.rados.rados.0.smithi123.stdout:1942: left oid 34 (ObjNum 734 snap 0 seq_num 734) 2020-01-16T00:14:30.021 INFO:tasks.rados.rados.0.smithi123.stdout:1943: finishing write tid 1 to smithi12313934-35 2020-01-16T00:14:30.021 INFO:tasks.rados.rados.0.smithi123.stdout:1942: done (15 left) 2020-01-16T00:14:30.022 INFO:tasks.rados.rados.0.smithi123.stdout:1958: read oid 24 snap -1 2020-01-16T00:14:30.022 INFO:tasks.rados.rados.0.smithi123.stdout:1958: expect deleted 2020-01-16T00:14:30.022 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:30.022 INFO:tasks.rados.rados.0.smithi123.stdout:1943: finishing write tid 2 to smithi12313934-35 2020-01-16T00:14:30.022 INFO:tasks.rados.rados.0.smithi123.stdout:1943: finishing write tid 4 to smithi12313934-35 2020-01-16T00:14:30.022 INFO:tasks.rados.rados.0.smithi123.stdout:1943: finishing write tid 5 to smithi12313934-35 2020-01-16T00:14:30.022 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 35 v 212 (ObjNum 735 snap 0 seq_num 735) dirty exists 2020-01-16T00:14:30.023 INFO:tasks.rados.rados.0.smithi123.stdout:1943: left oid 35 (ObjNum 735 snap 0 seq_num 735) 2020-01-16T00:14:30.023 INFO:tasks.rados.rados.0.smithi123.stdout:1944: finishing write tid 1 to smithi12313934-7 2020-01-16T00:14:30.023 INFO:tasks.rados.rados.0.smithi123.stdout:1943: done (15 left) 2020-01-16T00:14:30.023 INFO:tasks.rados.rados.0.smithi123.stdout:1959: write (excl) oid 19 current snap is 0 2020-01-16T00:14:30.023 INFO:tasks.rados.rados.0.smithi123.stdout:1959: seq_num 742 ranges {470794=720999,1903639=554256} 2020-01-16T00:14:30.024 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:30.024 INFO:tasks.rados.rados.0.smithi123.stdout:1959: writing smithi12313934-19 from 470794 to 1191793 tid 1 2020-01-16T00:14:30.031 INFO:tasks.rados.rados.0.smithi123.stdout:1959: writing smithi12313934-19 from 1903639 to 2457895 tid 2 2020-01-16T00:14:30.032 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:30.032 INFO:tasks.rados.rados.0.smithi123.stdout:1944: finishing write tid 2 to smithi12313934-7 2020-01-16T00:14:30.032 INFO:tasks.rados.rados.0.smithi123.stdout:1944: finishing write tid 4 to smithi12313934-7 2020-01-16T00:14:30.032 INFO:tasks.rados.rados.0.smithi123.stdout:1944: finishing write tid 5 to smithi12313934-7 2020-01-16T00:14:30.033 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 7 v 276 (ObjNum 736 snap 0 seq_num 736) dirty exists 2020-01-16T00:14:30.033 INFO:tasks.rados.rados.0.smithi123.stdout:1944: left oid 7 (ObjNum 736 snap 0 seq_num 736) 2020-01-16T00:14:30.033 INFO:tasks.rados.rados.0.smithi123.stdout:1947: expect (ObjNum 718 snap 0 seq_num 718) 2020-01-16T00:14:30.034 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-01-16T00:14:30.105 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:14:30.109 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:30.106+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:30.114 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2020-01-16T00:14:30.129 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:14:30.159 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:14:30.164 INFO:tasks.rados.rados.0.smithi123.stdout:1945: expect (ObjNum 719 snap 0 seq_num 719) 2020-01-16T00:14:30.176 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:30.176 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:14:30.176 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:14:30.177 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:30.188 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-01-16T00:14:30.193 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:14:30.222 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:14:30.222 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:30.222+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:30.251 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:14:30.282 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:14:30.284 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-01-16T00:14:30.284 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-01-16T00:14:30.284 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-01-16T00:14:30.285 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-01-16T00:14:30.285 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-01-16T00:14:30.285 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENXIO: problem getting command descriptions from osd.5 2020-01-16T00:14:30.285 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: problem getting command descriptions from osd.5 2020-01-16T00:14:30.285 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-01-16T00:14:30.286 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-01-16T00:14:30.294 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:30.294 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:14:30.294 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:14:30.294 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:14:30.294 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:30.296 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-01-16T00:14:30.296 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-01-16T00:14:30.302 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-01-16T00:14:30.333 INFO:tasks.rados.rados.0.smithi123.stdout:1950: expect (ObjNum 724 snap 0 seq_num 724) 2020-01-16T00:14:30.333 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:30.330+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:30.412 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:30.410+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:30.438 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:30.438 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:14:30.438 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:14:30.439 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:14:30.439 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.439 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:939 3.4 3:2bdd9e88:::smithi12313934-4:head [write 1681102~474010] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:14:30.439 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.908392+0000", 2020-01-16T00:14:30.439 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 112.526858232, 2020-01-16T00:14:30.440 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019294697, 2020-01-16T00:14:30.440 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:30.440 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:30.440 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:30.440 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:30.441 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:30.441 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 939 2020-01-16T00:14:30.441 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.441 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:30.441 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.442 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:30.442 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908392+0000", 2020-01-16T00:14:30.442 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:30.442 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.443 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.443 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:30.443 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908392+0000", 2020-01-16T00:14:30.443 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:30.443 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.444 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.444 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:30.444 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908390+0000", 2020-01-16T00:14:30.444 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015440399999999999 2020-01-16T00:14:30.445 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.445 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.445 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:30.445 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908545+0000", 2020-01-16T00:14:30.445 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3926e-05 2020-01-16T00:14:30.446 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.446 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.446 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:30.446 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908559+0000", 2020-01-16T00:14:30.446 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2200000000000003e-06 2020-01-16T00:14:30.447 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.447 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.447 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:30.447 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.908566+0000", 2020-01-16T00:14:30.447 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020594799999999998 2020-01-16T00:14:30.448 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.448 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.448 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.448 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910625+0000", 2020-01-16T00:14:30.448 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118672 2020-01-16T00:14:30.449 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.449 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.449 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:30.449 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.910744+0000", 2020-01-16T00:14:30.449 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031940969999999999 2020-01-16T00:14:30.450 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.450 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.450 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.450 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.913938+0000", 2020-01-16T00:14:30.451 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000309621 2020-01-16T00:14:30.451 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.451 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.451 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:30.451 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.914248+0000", 2020-01-16T00:14:30.451 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.008818579 2020-01-16T00:14:30.451 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.452 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.452 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.452 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.923066+0000", 2020-01-16T00:14:30.452 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000108989 2020-01-16T00:14:30.452 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.452 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.452 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:30.453 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.923175+0000", 2020-01-16T00:14:30.453 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019128299999999999 2020-01-16T00:14:30.453 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.453 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.453 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:14:30.453 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.923367+0000", 2020-01-16T00:14:30.453 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0032083760000000002 2020-01-16T00:14:30.453 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.454 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.454 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:30.454 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926575+0000", 2020-01-16T00:14:30.454 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2475999999999999e-05 2020-01-16T00:14:30.454 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.454 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.454 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:30.454 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926607+0000", 2020-01-16T00:14:30.455 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00099399800000000006 2020-01-16T00:14:30.455 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.455 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.455 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:30.455 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927601+0000", 2020-01-16T00:14:30.455 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0958999999999999e-05 2020-01-16T00:14:30.455 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.456 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.456 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:30.456 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927642+0000", 2020-01-16T00:14:30.456 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4747999999999999e-05 2020-01-16T00:14:30.456 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.456 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.456 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:30.456 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.927687+0000", 2020-01-16T00:14:30.457 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019294697 2020-01-16T00:14:30.457 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.457 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:30.457 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.457 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.457 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.457 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:940 3.4 3:2bdd9e88:::smithi12313934-4:head [write 2687912~589420] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:14:30.457 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.915144+0000", 2020-01-16T00:14:30.458 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 112.520106864, 2020-01-16T00:14:30.458 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016092538999999999, 2020-01-16T00:14:30.458 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:30.458 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:30.458 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:30.458 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:30.458 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:30.459 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 940 2020-01-16T00:14:30.459 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.459 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:30.459 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.459 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:30.459 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915144+0000", 2020-01-16T00:14:30.459 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:30.459 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.460 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.460 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:30.460 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915144+0000", 2020-01-16T00:14:30.460 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:30.460 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.460 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.460 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:30.460 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915141+0000", 2020-01-16T00:14:30.461 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016366300000000001 2020-01-16T00:14:30.461 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.461 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.461 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:30.461 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915305+0000", 2020-01-16T00:14:30.461 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4847e-05 2020-01-16T00:14:30.462 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.462 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.462 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:30.462 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915320+0000", 2020-01-16T00:14:30.463 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2599999999999999e-06 2020-01-16T00:14:30.463 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.463 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.463 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:30.464 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915327+0000", 2020-01-16T00:14:30.464 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4282000000000005e-05 2020-01-16T00:14:30.464 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.464 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.465 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.465 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915401+0000", 2020-01-16T00:14:30.465 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4554000000000006e-05 2020-01-16T00:14:30.465 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.466 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.466 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:30.466 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915466+0000", 2020-01-16T00:14:30.466 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0090821900000000004 2020-01-16T00:14:30.466 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.467 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.467 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.467 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.924548+0000", 2020-01-16T00:14:30.467 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000114696 2020-01-16T00:14:30.468 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.468 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.468 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:30.468 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.924663+0000", 2020-01-16T00:14:30.469 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000168651 2020-01-16T00:14:30.469 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.469 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.469 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:14:30.469 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.924832+0000", 2020-01-16T00:14:30.470 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0018253430000000001 2020-01-16T00:14:30.470 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.470 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.470 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:30.471 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926657+0000", 2020-01-16T00:14:30.471 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9957999999999999e-05 2020-01-16T00:14:30.471 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.471 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.472 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:30.472 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926697+0000", 2020-01-16T00:14:30.472 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0044685250000000001 2020-01-16T00:14:30.472 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.473 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.473 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:30.473 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931165+0000", 2020-01-16T00:14:30.473 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9211000000000001e-05 2020-01-16T00:14:30.473 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.474 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.474 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:30.474 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931205+0000", 2020-01-16T00:14:30.474 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1699999999999998e-05 2020-01-16T00:14:30.474 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.474 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.475 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:30.475 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931236+0000", 2020-01-16T00:14:30.475 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016092538999999999 2020-01-16T00:14:30.475 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.475 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:30.475 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.475 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.475 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.476 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:941 3.4 3:2bdd9e88:::smithi12313934-4:head [setxattr _header (55) in=7b,truncate 3277332] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:14:30.476 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.915341+0000", 2020-01-16T00:14:30.476 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 112.51991004600001, 2020-01-16T00:14:30.476 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016603769000000001, 2020-01-16T00:14:30.476 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:30.476 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:30.476 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:30.477 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:30.477 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:30.477 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 941 2020-01-16T00:14:30.477 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.477 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:30.477 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.478 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:30.478 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915341+0000", 2020-01-16T00:14:30.478 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:30.478 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.478 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.478 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:30.479 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915341+0000", 2020-01-16T00:14:30.479 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:30.479 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.479 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.479 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:30.479 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915340+0000", 2020-01-16T00:14:30.479 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.896e-06 2020-01-16T00:14:30.480 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.480 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.480 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:30.480 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915346+0000", 2020-01-16T00:14:30.480 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.2730000000000002e-06 2020-01-16T00:14:30.480 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.480 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.480 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:30.481 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915354+0000", 2020-01-16T00:14:30.481 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0616e-05 2020-01-16T00:14:30.481 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.481 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.481 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:30.481 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915374+0000", 2020-01-16T00:14:30.481 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000133487 2020-01-16T00:14:30.482 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.482 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.482 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.482 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915508+0000", 2020-01-16T00:14:30.482 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8971000000000003e-05 2020-01-16T00:14:30.483 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.483 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.483 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:30.483 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915557+0000", 2020-01-16T00:14:30.483 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010138787999999999 2020-01-16T00:14:30.484 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.484 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.484 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.484 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.925696+0000", 2020-01-16T00:14:30.484 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012825400000000001 2020-01-16T00:14:30.485 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.485 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.485 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:30.485 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.925824+0000", 2020-01-16T00:14:30.485 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017706500000000001 2020-01-16T00:14:30.486 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.486 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.486 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,5", 2020-01-16T00:14:30.486 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926001+0000", 2020-01-16T00:14:30.487 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00079988900000000003 2020-01-16T00:14:30.487 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.487 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.487 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:30.487 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926801+0000", 2020-01-16T00:14:30.488 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119324 2020-01-16T00:14:30.488 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.488 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.488 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:30.488 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926920+0000", 2020-01-16T00:14:30.488 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048879910000000004 2020-01-16T00:14:30.488 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.489 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.489 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:30.489 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931808+0000", 2020-01-16T00:14:30.489 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6999999999999997e-05 2020-01-16T00:14:30.489 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.489 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.489 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:30.490 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931855+0000", 2020-01-16T00:14:30.490 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9092999999999996e-05 2020-01-16T00:14:30.490 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.490 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.490 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:30.490 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931944+0000", 2020-01-16T00:14:30.490 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016603769000000001 2020-01-16T00:14:30.490 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.491 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:30.491 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.491 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.491 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.491 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:942 3.4 3:2bdd9e88:::smithi12313934-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:14:30.491 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:37.915389+0000", 2020-01-16T00:14:30.491 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 112.51986168000001, 2020-01-16T00:14:30.492 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016820563, 2020-01-16T00:14:30.492 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:30.492 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:30.492 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:30.492 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:30.492 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:30.492 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 942 2020-01-16T00:14:30.492 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.493 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:30.493 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.493 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:30.493 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915389+0000", 2020-01-16T00:14:30.493 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:30.494 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.494 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.494 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:30.494 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915389+0000", 2020-01-16T00:14:30.494 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:30.494 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.494 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.494 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:30.495 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915388+0000", 2020-01-16T00:14:30.495 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.683e-06 2020-01-16T00:14:30.495 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.495 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.495 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:30.495 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915391+0000", 2020-01-16T00:14:30.495 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4839999999999998e-06 2020-01-16T00:14:30.495 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.496 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.496 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:30.496 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915399+0000", 2020-01-16T00:14:30.496 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2259999999999999e-06 2020-01-16T00:14:30.496 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.496 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.496 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:30.496 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915403+0000", 2020-01-16T00:14:30.497 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00031301199999999999 2020-01-16T00:14:30.497 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.497 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.497 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.497 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915716+0000", 2020-01-16T00:14:30.497 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2079999999999998e-05 2020-01-16T00:14:30.498 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.498 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.498 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:30.498 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.915748+0000", 2020-01-16T00:14:30.498 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010547076000000001 2020-01-16T00:14:30.499 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.499 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.499 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.499 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926295+0000", 2020-01-16T00:14:30.499 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8040000000000005e-05 2020-01-16T00:14:30.499 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.499 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.500 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:30.500 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.926373+0000", 2020-01-16T00:14:30.500 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056061990000000001 2020-01-16T00:14:30.500 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.500 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.500 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.500 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.931979+0000", 2020-01-16T00:14:30.501 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011540399999999999 2020-01-16T00:14:30.501 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.501 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.501 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:30.501 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932095+0000", 2020-01-16T00:14:30.502 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011482899999999999 2020-01-16T00:14:30.502 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.502 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.502 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:30.502 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:37.932210+0000", 2020-01-16T00:14:30.503 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016820563 2020-01-16T00:14:30.503 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.503 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:30.503 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.503 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.503 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.503 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:957 3.0 3:00c146d5:::smithi12313934-18:head [write 510946~632952] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:14:30.504 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.033290+0000", 2020-01-16T00:14:30.504 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 112.401960843, 2020-01-16T00:14:30.504 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016764102999999999, 2020-01-16T00:14:30.504 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:30.504 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:30.504 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:30.504 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:30.504 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:30.505 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 957 2020-01-16T00:14:30.505 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.505 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:30.505 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.505 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:30.505 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033290+0000", 2020-01-16T00:14:30.505 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:30.506 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.506 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.506 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:30.506 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033290+0000", 2020-01-16T00:14:30.506 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999952 2020-01-16T00:14:30.506 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.506 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.506 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:30.507 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033285+0000", 2020-01-16T00:14:30.507 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000405917 2020-01-16T00:14:30.507 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.507 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.507 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:30.507 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033691+0000", 2020-01-16T00:14:30.507 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6585999999999997e-05 2020-01-16T00:14:30.507 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.508 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.508 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:30.508 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033738+0000", 2020-01-16T00:14:30.508 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8374000000000001e-05 2020-01-16T00:14:30.508 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.508 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.509 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:30.509 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033756+0000", 2020-01-16T00:14:30.509 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013002900000000001 2020-01-16T00:14:30.509 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.509 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.509 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.509 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.033886+0000", 2020-01-16T00:14:30.510 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035882600000000002 2020-01-16T00:14:30.510 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.510 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.510 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:30.510 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.034245+0000", 2020-01-16T00:14:30.510 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0072798489999999997 2020-01-16T00:14:30.510 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.510 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.511 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.511 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.041525+0000", 2020-01-16T00:14:30.511 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000140488 2020-01-16T00:14:30.511 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.511 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.511 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:30.511 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.041665+0000", 2020-01-16T00:14:30.512 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000233766 2020-01-16T00:14:30.512 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.512 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.512 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:30.512 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.041899+0000", 2020-01-16T00:14:30.512 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0035766539999999999 2020-01-16T00:14:30.512 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.512 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.513 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:30.513 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.045475+0000", 2020-01-16T00:14:30.513 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013333150000000001 2020-01-16T00:14:30.513 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.513 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.513 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:30.513 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.046809+0000", 2020-01-16T00:14:30.514 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031447860000000001 2020-01-16T00:14:30.514 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.514 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.514 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:30.514 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.049954+0000", 2020-01-16T00:14:30.515 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2318000000000003e-05 2020-01-16T00:14:30.515 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.515 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.515 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:30.515 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.050006+0000", 2020-01-16T00:14:30.515 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7972000000000001e-05 2020-01-16T00:14:30.515 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.516 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.516 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:30.516 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.050054+0000", 2020-01-16T00:14:30.516 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016764102999999999 2020-01-16T00:14:30.516 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.516 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:30.516 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.517 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.517 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.517 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1639 3.0 3:00c146d5:::smithi12313934-18:head [write 423576~401598] snapc 0=[] ondisk+write+known_if_redirected e30)", 2020-01-16T00:14:30.517 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.774317+0000", 2020-01-16T00:14:30.517 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 98.660934104000006, 2020-01-16T00:14:30.517 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018305219000000001, 2020-01-16T00:14:30.517 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:30.517 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:30.518 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:30.518 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:30.518 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:30.518 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1639 2020-01-16T00:14:30.518 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.518 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:30.519 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.519 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:30.519 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774317+0000", 2020-01-16T00:14:30.519 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:30.519 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.519 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.520 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:30.520 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774317+0000", 2020-01-16T00:14:30.520 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:30.520 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.520 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.520 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:30.520 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774313+0000", 2020-01-16T00:14:30.521 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000111173 2020-01-16T00:14:30.521 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.521 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.521 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:30.521 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774425+0000", 2020-01-16T00:14:30.521 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7598999999999999e-05 2020-01-16T00:14:30.521 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.521 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.522 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:30.522 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774442+0000", 2020-01-16T00:14:30.522 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0830000000000004e-06 2020-01-16T00:14:30.522 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.522 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.522 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:30.522 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.774450+0000", 2020-01-16T00:14:30.523 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0017886309999999999 2020-01-16T00:14:30.523 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.523 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.523 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.524 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.776239+0000", 2020-01-16T00:14:30.524 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3789e-05 2020-01-16T00:14:30.524 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.524 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.524 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op must wait for map", 2020-01-16T00:14:30.525 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.776253+0000", 2020-01-16T00:14:30.525 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095730729999999997 2020-01-16T00:14:30.525 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.525 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.525 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.526 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.785826+0000", 2020-01-16T00:14:30.526 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014220300000000001 2020-01-16T00:14:30.526 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.526 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.526 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:30.526 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.785968+0000", 2020-01-16T00:14:30.526 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023618900000000001 2020-01-16T00:14:30.527 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.527 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.527 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:30.527 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.786204+0000", 2020-01-16T00:14:30.527 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0062297389999999998 2020-01-16T00:14:30.527 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.528 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.528 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:30.528 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792434+0000", 2020-01-16T00:14:30.528 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9635999999999998e-05 2020-01-16T00:14:30.528 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.528 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.528 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:30.529 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792473+0000", 2020-01-16T00:14:30.529 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6456000000000002e-05 2020-01-16T00:14:30.529 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.529 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.530 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:30.530 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792540+0000", 2020-01-16T00:14:30.530 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9873999999999998e-05 2020-01-16T00:14:30.530 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.530 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.530 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:30.531 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792580+0000", 2020-01-16T00:14:30.531 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1924e-05 2020-01-16T00:14:30.531 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.531 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.531 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:30.531 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.792622+0000", 2020-01-16T00:14:30.531 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018305219000000001 2020-01-16T00:14:30.531 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.532 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:30.532 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.532 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.532 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.532 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1890 3.0 3:0b1f384f:::smithi12313934-33:head [write 2797915~596476] snapc 0=[] ondisk+write+known_if_redirected e31)", 2020-01-16T00:14:30.532 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.232534+0000", 2020-01-16T00:14:30.532 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 93.202716132999996, 2020-01-16T00:14:30.533 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016106137, 2020-01-16T00:14:30.533 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:30.533 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:30.533 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:30.533 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:30.533 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:30.533 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1890 2020-01-16T00:14:30.533 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.534 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:30.534 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.534 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:30.534 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232534+0000", 2020-01-16T00:14:30.534 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:30.534 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.534 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.534 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:30.535 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232534+0000", 2020-01-16T00:14:30.535 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:30.535 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.535 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.535 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:30.535 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232532+0000", 2020-01-16T00:14:30.535 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00024232100000000001 2020-01-16T00:14:30.536 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.536 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.536 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:30.536 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232774+0000", 2020-01-16T00:14:30.536 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4601e-05 2020-01-16T00:14:30.536 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.536 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.536 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:30.537 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232789+0000", 2020-01-16T00:14:30.537 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8410000000000006e-06 2020-01-16T00:14:30.537 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.537 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.537 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:30.537 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232798+0000", 2020-01-16T00:14:30.537 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9756999999999997e-05 2020-01-16T00:14:30.537 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.538 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.538 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.538 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232857+0000", 2020-01-16T00:14:30.538 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4678000000000007e-05 2020-01-16T00:14:30.538 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.538 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.538 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:30.538 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232922+0000", 2020-01-16T00:14:30.539 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033861020000000002 2020-01-16T00:14:30.539 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.539 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.539 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.539 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.236308+0000", 2020-01-16T00:14:30.539 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000568996 2020-01-16T00:14:30.539 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.539 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.540 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:30.540 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.236877+0000", 2020-01-16T00:14:30.540 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0085444240000000001 2020-01-16T00:14:30.540 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.540 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.540 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.541 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.245422+0000", 2020-01-16T00:14:30.541 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5595999999999998e-05 2020-01-16T00:14:30.541 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.541 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.541 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:30.542 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.245507+0000", 2020-01-16T00:14:30.542 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016984300000000001 2020-01-16T00:14:30.542 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.542 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.542 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:30.542 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.245677+0000", 2020-01-16T00:14:30.542 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001649495 2020-01-16T00:14:30.543 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.543 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.543 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:30.543 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247326+0000", 2020-01-16T00:14:30.543 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023308399999999999 2020-01-16T00:14:30.543 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.543 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.543 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:30.544 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247560+0000", 2020-01-16T00:14:30.544 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00099501800000000007 2020-01-16T00:14:30.544 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.544 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.544 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:30.544 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248555+0000", 2020-01-16T00:14:30.545 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5936999999999998e-05 2020-01-16T00:14:30.545 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.545 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.545 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:30.545 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248591+0000", 2020-01-16T00:14:30.545 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0108999999999997e-05 2020-01-16T00:14:30.545 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.545 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.546 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:30.546 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248641+0000", 2020-01-16T00:14:30.546 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016106137 2020-01-16T00:14:30.546 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.546 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:30.546 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.547 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.547 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.547 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1892 3.0 3:0b1f384f:::smithi12313934-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e31)", 2020-01-16T00:14:30.547 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.232856+0000", 2020-01-16T00:14:30.547 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 93.202395107000001, 2020-01-16T00:14:30.547 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016171544, 2020-01-16T00:14:30.548 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:30.548 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:30.548 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:30.548 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:30.548 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:30.548 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1892 2020-01-16T00:14:30.548 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.548 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:30.549 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.549 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:30.549 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232856+0000", 2020-01-16T00:14:30.549 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:30.549 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.549 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.549 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:30.549 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232856+0000", 2020-01-16T00:14:30.550 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:30.550 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.550 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.550 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:30.550 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232855+0000", 2020-01-16T00:14:30.550 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8640000000000004e-06 2020-01-16T00:14:30.550 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.551 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.551 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:30.551 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232860+0000", 2020-01-16T00:14:30.551 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0000000000000001e-05 2020-01-16T00:14:30.551 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.551 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.551 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:30.552 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232870+0000", 2020-01-16T00:14:30.552 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7420000000000003e-06 2020-01-16T00:14:30.552 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.552 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.552 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:30.552 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.232875+0000", 2020-01-16T00:14:30.552 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017933200000000001 2020-01-16T00:14:30.553 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.553 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.553 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.553 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.233055+0000", 2020-01-16T00:14:30.553 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6269e-05 2020-01-16T00:14:30.553 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.553 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.553 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:30.554 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.233101+0000", 2020-01-16T00:14:30.554 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039780090000000002 2020-01-16T00:14:30.554 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.554 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.554 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.554 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.237079+0000", 2020-01-16T00:14:30.554 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3677000000000001e-05 2020-01-16T00:14:30.555 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.555 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.555 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:30.555 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.237133+0000", 2020-01-16T00:14:30.555 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010036671 2020-01-16T00:14:30.555 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.555 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.555 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.556 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247169+0000", 2020-01-16T00:14:30.556 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010280699999999999 2020-01-16T00:14:30.556 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.556 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.556 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:30.556 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.247272+0000", 2020-01-16T00:14:30.556 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001579708 2020-01-16T00:14:30.556 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.557 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.557 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.557 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248852+0000", 2020-01-16T00:14:30.557 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9704999999999996e-05 2020-01-16T00:14:30.557 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.558 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.558 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:30.558 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.248942+0000", 2020-01-16T00:14:30.558 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5557999999999996e-05 2020-01-16T00:14:30.558 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.558 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.558 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:30.559 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.249027+0000", 2020-01-16T00:14:30.559 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016171544 2020-01-16T00:14:30.559 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.559 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:30.559 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.559 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.559 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.559 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1950 3.0 3:0b1f384f:::smithi12313934-33:head [write 1889359~541737] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-01-16T00:14:30.560 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.922819+0000", 2020-01-16T00:14:30.560 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 92.512431742000004, 2020-01-16T00:14:30.560 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017502846999999998, 2020-01-16T00:14:30.560 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:30.560 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:30.560 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:30.560 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:30.560 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:30.561 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1950 2020-01-16T00:14:30.561 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.561 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:30.561 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.561 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:30.562 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922819+0000", 2020-01-16T00:14:30.562 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:30.562 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.562 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.562 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:30.562 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922819+0000", 2020-01-16T00:14:30.563 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:30.563 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.563 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.563 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:30.563 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922818+0000", 2020-01-16T00:14:30.563 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000146408 2020-01-16T00:14:30.563 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.564 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.564 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:30.564 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922964+0000", 2020-01-16T00:14:30.564 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4720000000000001e-06 2020-01-16T00:14:30.564 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.564 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.564 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:30.565 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922974+0000", 2020-01-16T00:14:30.565 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0720000000000001e-06 2020-01-16T00:14:30.565 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.565 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.565 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:30.565 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.922978+0000", 2020-01-16T00:14:30.565 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6152000000000001e-05 2020-01-16T00:14:30.566 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.566 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.566 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.566 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.923024+0000", 2020-01-16T00:14:30.566 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00087368199999999997 2020-01-16T00:14:30.566 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.566 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.566 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:30.567 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.923897+0000", 2020-01-16T00:14:30.567 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011944435999999999 2020-01-16T00:14:30.567 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.567 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.567 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.567 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.935842+0000", 2020-01-16T00:14:30.567 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000144146 2020-01-16T00:14:30.567 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.568 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.568 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:30.568 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.935986+0000", 2020-01-16T00:14:30.568 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019775500000000001 2020-01-16T00:14:30.568 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.569 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.569 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:30.569 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.936184+0000", 2020-01-16T00:14:30.569 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002539342 2020-01-16T00:14:30.569 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.570 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.570 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:30.570 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938723+0000", 2020-01-16T00:14:30.570 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014193699999999999 2020-01-16T00:14:30.570 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.571 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.571 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:30.571 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938865+0000", 2020-01-16T00:14:30.571 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013829020000000001 2020-01-16T00:14:30.571 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.571 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.572 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:30.572 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.940248+0000", 2020-01-16T00:14:30.572 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2558000000000001e-05 2020-01-16T00:14:30.572 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.572 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.572 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:30.572 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.940291+0000", 2020-01-16T00:14:30.573 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1217999999999998e-05 2020-01-16T00:14:30.573 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.573 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.573 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:30.573 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.940322+0000", 2020-01-16T00:14:30.573 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017502846999999998 2020-01-16T00:14:30.573 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.574 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:30.574 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.574 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.574 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.574 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1951 3.0 3:0b1f384f:::smithi12313934-33:head [write 2955792~541902] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-01-16T00:14:30.575 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.929019+0000", 2020-01-16T00:14:30.575 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 92.506231592000006, 2020-01-16T00:14:30.575 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021625710999999999, 2020-01-16T00:14:30.575 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:30.575 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:30.575 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:30.576 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:30.576 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:30.576 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1951 2020-01-16T00:14:30.576 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.576 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:30.576 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.576 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:30.576 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929019+0000", 2020-01-16T00:14:30.577 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:30.577 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.577 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.577 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:30.577 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929019+0000", 2020-01-16T00:14:30.577 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999989 2020-01-16T00:14:30.577 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.578 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.578 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:30.578 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929008+0000", 2020-01-16T00:14:30.578 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000212513 2020-01-16T00:14:30.578 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.578 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.578 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:30.579 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929221+0000", 2020-01-16T00:14:30.579 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3229e-05 2020-01-16T00:14:30.579 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.579 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.579 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:30.579 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929234+0000", 2020-01-16T00:14:30.579 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9170000000000003e-06 2020-01-16T00:14:30.579 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.580 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.580 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:30.580 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929240+0000", 2020-01-16T00:14:30.580 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4045000000000002e-05 2020-01-16T00:14:30.580 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.580 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.580 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.581 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929294+0000", 2020-01-16T00:14:30.581 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9797999999999996e-05 2020-01-16T00:14:30.581 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.581 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.581 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:30.581 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929384+0000", 2020-01-16T00:14:30.581 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0083450729999999997 2020-01-16T00:14:30.581 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.582 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.582 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.582 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.937729+0000", 2020-01-16T00:14:30.582 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043251699999999999 2020-01-16T00:14:30.582 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.582 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.583 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:30.583 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938161+0000", 2020-01-16T00:14:30.583 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0095115489999999994 2020-01-16T00:14:30.583 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.583 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.583 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.583 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.947673+0000", 2020-01-16T00:14:30.584 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012395099999999999 2020-01-16T00:14:30.584 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.584 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.584 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:30.584 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.947797+0000", 2020-01-16T00:14:30.584 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00024849200000000001 2020-01-16T00:14:30.584 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.584 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.585 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:30.585 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.948045+0000", 2020-01-16T00:14:30.585 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0025081050000000001 2020-01-16T00:14:30.585 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.585 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.585 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:30.585 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950553+0000", 2020-01-16T00:14:30.586 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9760000000000003e-06 2020-01-16T00:14:30.586 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.586 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.586 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:30.586 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950558+0000", 2020-01-16T00:14:30.586 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3286999999999999e-05 2020-01-16T00:14:30.586 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.586 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.587 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:30.587 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950581+0000", 2020-01-16T00:14:30.587 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2759000000000003e-05 2020-01-16T00:14:30.587 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.587 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.587 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:30.587 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950614+0000", 2020-01-16T00:14:30.587 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0525999999999999e-05 2020-01-16T00:14:30.588 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.588 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.588 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:30.588 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950645+0000", 2020-01-16T00:14:30.588 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021625710999999999 2020-01-16T00:14:30.588 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.588 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:30.589 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.589 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.589 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.589 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1952 3.0 3:0b1f384f:::smithi12313934-33:head [setxattr _header (56) in=7b,truncate 3563084] snapc 0=[] ondisk+write+known_if_redirected e32)", 2020-01-16T00:14:30.589 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:14:30.589 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 92.506002254999999, 2020-01-16T00:14:30.589 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021534685000000001, 2020-01-16T00:14:30.589 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:30.590 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:30.590 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:30.590 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:30.590 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:30.590 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1952 2020-01-16T00:14:30.590 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.591 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:30.591 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.591 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:30.591 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:14:30.591 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:30.591 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.591 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.591 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:30.592 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:14:30.592 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:30.592 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.592 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.592 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:30.592 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929248+0000", 2020-01-16T00:14:30.593 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6670000000000004e-06 2020-01-16T00:14:30.593 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.593 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.593 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:30.594 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929252+0000", 2020-01-16T00:14:30.594 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9380000000000002e-06 2020-01-16T00:14:30.594 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.594 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.594 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:30.594 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929256+0000", 2020-01-16T00:14:30.594 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.712e-06 2020-01-16T00:14:30.595 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.595 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.595 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:30.595 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929264+0000", 2020-01-16T00:14:30.595 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018030300000000001 2020-01-16T00:14:30.595 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.595 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.595 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.596 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929444+0000", 2020-01-16T00:14:30.596 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7562000000000002e-05 2020-01-16T00:14:30.596 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.596 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.596 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:30.596 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929502+0000", 2020-01-16T00:14:30.596 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0087297859999999998 2020-01-16T00:14:30.596 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.597 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.597 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.597 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938232+0000", 2020-01-16T00:14:30.597 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.0452999999999996e-05 2020-01-16T00:14:30.597 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.597 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.597 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:30.598 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938322+0000", 2020-01-16T00:14:30.598 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011061728 2020-01-16T00:14:30.598 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.598 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.598 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.598 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.949384+0000", 2020-01-16T00:14:30.598 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000117828 2020-01-16T00:14:30.599 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.599 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.599 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:30.599 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.949502+0000", 2020-01-16T00:14:30.599 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022468499999999999 2020-01-16T00:14:30.600 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.600 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.600 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:30.600 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.949726+0000", 2020-01-16T00:14:30.600 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00083532099999999998 2020-01-16T00:14:30.601 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.601 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.601 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:30.601 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950562+0000", 2020-01-16T00:14:30.601 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013017700000000001 2020-01-16T00:14:30.601 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.601 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.602 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:30.602 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950692+0000", 2020-01-16T00:14:30.602 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9813000000000001e-05 2020-01-16T00:14:30.602 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.602 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.602 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:30.602 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950722+0000", 2020-01-16T00:14:30.603 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0592000000000001e-05 2020-01-16T00:14:30.603 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.603 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.603 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:30.603 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950752+0000", 2020-01-16T00:14:30.604 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0920000000000002e-05 2020-01-16T00:14:30.604 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.604 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.604 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:30.604 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950783+0000", 2020-01-16T00:14:30.604 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021534685000000001 2020-01-16T00:14:30.604 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.605 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:30.605 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.605 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.605 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.605 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1953 3.0 3:0b1f384f:::smithi12313934-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e32)", 2020-01-16T00:14:30.605 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:14:30.606 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 92.505969264000001, 2020-01-16T00:14:30.606 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021667537000000001, 2020-01-16T00:14:30.606 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:30.606 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:30.606 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:30.606 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:30.606 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:30.607 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1953 2020-01-16T00:14:30.607 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.607 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:30.607 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.607 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:30.607 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:14:30.607 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:30.607 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.608 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.608 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:30.608 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:14:30.608 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:30.608 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.608 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.608 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:30.608 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929281+0000", 2020-01-16T00:14:30.609 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7030000000000002e-06 2020-01-16T00:14:30.609 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.609 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.609 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:30.609 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929284+0000", 2020-01-16T00:14:30.609 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0909999999999993e-06 2020-01-16T00:14:30.609 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.610 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.610 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:30.610 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929292+0000", 2020-01-16T00:14:30.610 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8359999999999999e-06 2020-01-16T00:14:30.610 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.610 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.610 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:30.610 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929297+0000", 2020-01-16T00:14:30.611 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023538199999999999 2020-01-16T00:14:30.611 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.611 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.611 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.611 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929532+0000", 2020-01-16T00:14:30.611 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1173000000000004e-05 2020-01-16T00:14:30.611 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.611 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.612 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:30.612 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.929603+0000", 2020-01-16T00:14:30.612 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0087813509999999997 2020-01-16T00:14:30.612 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.612 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.612 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.613 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938384+0000", 2020-01-16T00:14:30.613 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9752000000000006e-05 2020-01-16T00:14:30.613 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.613 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.614 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:30.614 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.938454+0000", 2020-01-16T00:14:30.614 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011707851999999999 2020-01-16T00:14:30.614 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.614 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.614 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.614 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950162+0000", 2020-01-16T00:14:30.615 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011060099999999999 2020-01-16T00:14:30.615 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.615 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.615 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:30.615 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950273+0000", 2020-01-16T00:14:30.615 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00053989700000000001 2020-01-16T00:14:30.615 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.615 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.616 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.616 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950813+0000", 2020-01-16T00:14:30.616 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8128999999999999e-05 2020-01-16T00:14:30.616 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.616 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.616 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:30.616 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950871+0000", 2020-01-16T00:14:30.616 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8242000000000001e-05 2020-01-16T00:14:30.617 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.617 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.617 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:30.617 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:57.950949+0000", 2020-01-16T00:14:30.617 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021667537000000001 2020-01-16T00:14:30.617 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.617 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:30.618 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.618 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.618 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.618 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2555 3.0 3:0b70bb20:::smithi12313934-46:head [setxattr _header (56) in=7b,truncate 2991679] snapc 0=[] ondisk+write+known_if_redirected e37)", 2020-01-16T00:14:30.620 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.573432+0000", 2020-01-16T00:14:30.621 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 77.861818612999997, 2020-01-16T00:14:30.621 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019893926999999999, 2020-01-16T00:14:30.621 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:30.621 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:30.621 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:30.621 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:30.621 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:30.622 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2555 2020-01-16T00:14:30.622 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.622 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:30.622 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.622 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:30.622 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573432+0000", 2020-01-16T00:14:30.622 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:30.623 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.623 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.623 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:30.623 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573432+0000", 2020-01-16T00:14:30.623 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:30.623 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.623 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.623 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:30.624 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573432+0000", 2020-01-16T00:14:30.624 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4290000000000001e-06 2020-01-16T00:14:30.624 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.624 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.624 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:30.624 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573435+0000", 2020-01-16T00:14:30.624 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1300000000000001e-06 2020-01-16T00:14:30.624 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.625 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.625 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:30.625 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573438+0000", 2020-01-16T00:14:30.625 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9399999999999996e-06 2020-01-16T00:14:30.625 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.625 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.625 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:30.625 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573445+0000", 2020-01-16T00:14:30.626 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013947269999999999 2020-01-16T00:14:30.626 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.626 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.626 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.626 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.574840+0000", 2020-01-16T00:14:30.626 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003850058 2020-01-16T00:14:30.627 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.627 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.627 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:30.627 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578690+0000", 2020-01-16T00:14:30.627 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012896489000000001 2020-01-16T00:14:30.627 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.627 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.627 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.628 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591586+0000", 2020-01-16T00:14:30.628 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4757000000000001e-05 2020-01-16T00:14:30.628 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.628 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.628 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:30.628 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591681+0000", 2020-01-16T00:14:30.628 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000191535 2020-01-16T00:14:30.628 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.629 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.629 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:30.629 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.591873+0000", 2020-01-16T00:14:30.629 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00069888899999999996 2020-01-16T00:14:30.629 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.629 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.630 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:30.630 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.592571+0000", 2020-01-16T00:14:30.630 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00029525900000000003 2020-01-16T00:14:30.630 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.630 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.630 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:30.630 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.592867+0000", 2020-01-16T00:14:30.631 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035243500000000001 2020-01-16T00:14:30.631 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.631 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.631 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:30.631 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593219+0000", 2020-01-16T00:14:30.631 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0954e-05 2020-01-16T00:14:30.631 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.632 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.632 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:30.632 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593270+0000", 2020-01-16T00:14:30.632 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5813000000000002e-05 2020-01-16T00:14:30.632 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.632 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.632 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:30.633 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593326+0000", 2020-01-16T00:14:30.633 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019893926999999999 2020-01-16T00:14:30.633 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.633 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:30.633 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.633 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.633 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.634 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2556 3.0 3:0b70bb20:::smithi12313934-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e37)", 2020-01-16T00:14:30.634 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.573472+0000", 2020-01-16T00:14:30.634 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 77.861779042999999, 2020-01-16T00:14:30.634 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020070068999999999, 2020-01-16T00:14:30.634 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:30.634 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:30.634 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:30.634 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:30.635 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:30.635 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2556 2020-01-16T00:14:30.635 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.635 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:30.635 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.635 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:30.635 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573472+0000", 2020-01-16T00:14:30.635 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:30.636 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.636 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.636 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:30.636 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573472+0000", 2020-01-16T00:14:30.636 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:30.636 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.636 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.637 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:30.637 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573471+0000", 2020-01-16T00:14:30.637 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4449999999999999e-06 2020-01-16T00:14:30.637 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.637 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.637 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:30.637 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573479+0000", 2020-01-16T00:14:30.638 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9789999999999998e-06 2020-01-16T00:14:30.638 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.638 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.638 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:30.638 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573485+0000", 2020-01-16T00:14:30.638 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3270000000000001e-06 2020-01-16T00:14:30.638 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.638 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.639 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:30.639 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.573488+0000", 2020-01-16T00:14:30.639 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052586849999999999 2020-01-16T00:14:30.639 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.639 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.639 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.639 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578747+0000", 2020-01-16T00:14:30.640 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5667000000000007e-05 2020-01-16T00:14:30.640 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.640 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.640 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:30.640 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578812+0000", 2020-01-16T00:14:30.640 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013599539000000001 2020-01-16T00:14:30.640 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.640 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.641 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.641 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.592412+0000", 2020-01-16T00:14:30.641 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6018000000000001e-05 2020-01-16T00:14:30.641 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.641 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.641 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:30.642 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.592508+0000", 2020-01-16T00:14:30.642 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00084582199999999998 2020-01-16T00:14:30.642 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.642 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.642 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.642 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593354+0000", 2020-01-16T00:14:30.642 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4984999999999999e-05 2020-01-16T00:14:30.642 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.643 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.643 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:30.643 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593449+0000", 2020-01-16T00:14:30.643 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3104999999999997e-05 2020-01-16T00:14:30.643 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.643 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.643 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:30.644 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.593542+0000", 2020-01-16T00:14:30.644 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020070068999999999 2020-01-16T00:14:30.644 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.644 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:30.644 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.644 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.644 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.644 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3211 3.0 3:0b70bb20:::smithi12313934-46:head [setxattr _header (56) in=7b,truncate 2571537] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:14:30.645 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.581146+0000", 2020-01-16T00:14:30.645 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 60.854104473, 2020-01-16T00:14:30.645 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024318514999999999, 2020-01-16T00:14:30.645 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:30.645 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:30.645 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:30.645 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:30.645 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:30.646 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3211 2020-01-16T00:14:30.646 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.646 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:30.646 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.646 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:30.646 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581146+0000", 2020-01-16T00:14:30.646 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:30.647 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.647 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.647 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:30.647 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581146+0000", 2020-01-16T00:14:30.647 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:30.647 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.648 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.648 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:30.648 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581145+0000", 2020-01-16T00:14:30.648 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2990000000000002e-06 2020-01-16T00:14:30.648 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.649 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.649 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:30.649 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581151+0000", 2020-01-16T00:14:30.649 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.041e-06 2020-01-16T00:14:30.649 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.649 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.649 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:30.650 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581159+0000", 2020-01-16T00:14:30.650 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5859999999999998e-06 2020-01-16T00:14:30.650 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.650 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.650 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:30.650 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581163+0000", 2020-01-16T00:14:30.650 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020769790000000001 2020-01-16T00:14:30.650 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.651 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.651 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.651 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.583240+0000", 2020-01-16T00:14:30.651 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012019579999999999 2020-01-16T00:14:30.651 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.651 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.651 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:30.651 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584442+0000", 2020-01-16T00:14:30.652 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018359812 2020-01-16T00:14:30.652 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.652 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.652 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.652 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.602802+0000", 2020-01-16T00:14:30.652 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000133072 2020-01-16T00:14:30.653 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.653 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.653 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:30.653 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.602935+0000", 2020-01-16T00:14:30.654 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019007200000000001 2020-01-16T00:14:30.654 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.654 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.654 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:30.654 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.603125+0000", 2020-01-16T00:14:30.654 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001261751 2020-01-16T00:14:30.655 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.655 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.655 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:30.655 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.604387+0000", 2020-01-16T00:14:30.655 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000304987 2020-01-16T00:14:30.655 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.655 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.656 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:30.656 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.604692+0000", 2020-01-16T00:14:30.656 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00063369299999999995 2020-01-16T00:14:30.656 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.656 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.656 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:30.656 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605325+0000", 2020-01-16T00:14:30.656 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1470000000000005e-05 2020-01-16T00:14:30.657 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.657 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.657 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:30.657 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605397+0000", 2020-01-16T00:14:30.657 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7784e-05 2020-01-16T00:14:30.657 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.658 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.658 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:30.658 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605465+0000", 2020-01-16T00:14:30.658 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024318514999999999 2020-01-16T00:14:30.658 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.659 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:30.659 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.659 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.659 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.659 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3212 3.0 3:0b70bb20:::smithi12313934-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e45)", 2020-01-16T00:14:30.659 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.581171+0000", 2020-01-16T00:14:30.659 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 60.854079773999999, 2020-01-16T00:14:30.659 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024540131, 2020-01-16T00:14:30.660 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:30.660 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:30.660 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:30.660 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:30.660 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:30.660 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3212 2020-01-16T00:14:30.660 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.660 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:30.661 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.661 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:30.661 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581171+0000", 2020-01-16T00:14:30.661 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:30.661 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.661 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.661 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:30.661 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581171+0000", 2020-01-16T00:14:30.662 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:30.662 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.662 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.662 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:30.662 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581170+0000", 2020-01-16T00:14:30.662 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5259999999999996e-06 2020-01-16T00:14:30.663 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.663 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.663 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:30.663 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581178+0000", 2020-01-16T00:14:30.663 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.469e-06 2020-01-16T00:14:30.663 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.663 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.663 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:30.664 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581185+0000", 2020-01-16T00:14:30.664 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4460000000000003e-06 2020-01-16T00:14:30.664 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.664 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.664 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:30.664 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.581190+0000", 2020-01-16T00:14:30.664 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003286696 2020-01-16T00:14:30.664 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.665 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.665 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.665 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584476+0000", 2020-01-16T00:14:30.665 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9681e-05 2020-01-16T00:14:30.665 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.665 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.665 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:30.665 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584536+0000", 2020-01-16T00:14:30.666 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019075267 2020-01-16T00:14:30.666 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.666 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.666 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.666 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.603611+0000", 2020-01-16T00:14:30.666 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5981000000000001e-05 2020-01-16T00:14:30.666 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.667 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.667 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:30.667 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.603707+0000", 2020-01-16T00:14:30.667 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0017922649999999999 2020-01-16T00:14:30.667 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.667 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.667 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.668 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605500+0000", 2020-01-16T00:14:30.668 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5396999999999995e-05 2020-01-16T00:14:30.668 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.668 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.668 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:30.668 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605595+0000", 2020-01-16T00:14:30.668 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000116038 2020-01-16T00:14:30.669 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.669 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.669 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:30.669 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.605711+0000", 2020-01-16T00:14:30.669 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024540131 2020-01-16T00:14:30.669 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.669 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:30.670 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.670 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.670 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.670 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3435 3.0 3:0b70bb20:::smithi12313934-46:head [setxattr _header (56) in=7b,truncate 3075747] snapc 0=[] ondisk+write+known_if_redirected e47)", 2020-01-16T00:14:30.670 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:34.388891+0000", 2020-01-16T00:14:30.670 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 56.046359758000001, 2020-01-16T00:14:30.670 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016440475999999999, 2020-01-16T00:14:30.671 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:30.671 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:30.671 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:30.671 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:30.671 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:30.671 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3435 2020-01-16T00:14:30.671 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.671 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:30.672 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.672 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:30.672 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388891+0000", 2020-01-16T00:14:30.672 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:30.672 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.672 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.673 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:30.673 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388891+0000", 2020-01-16T00:14:30.673 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:30.673 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.673 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.673 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:30.673 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388890+0000", 2020-01-16T00:14:30.673 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.647e-06 2020-01-16T00:14:30.674 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.674 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.674 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:30.674 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388894+0000", 2020-01-16T00:14:30.674 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2839999999999997e-06 2020-01-16T00:14:30.674 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.674 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.675 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:30.675 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388899+0000", 2020-01-16T00:14:30.675 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3149999999999999e-06 2020-01-16T00:14:30.675 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.675 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.675 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:30.675 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388901+0000", 2020-01-16T00:14:30.675 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001768543 2020-01-16T00:14:30.676 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.676 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.676 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.676 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.390670+0000", 2020-01-16T00:14:30.676 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0034533680000000001 2020-01-16T00:14:30.676 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.676 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.677 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:30.677 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.394123+0000", 2020-01-16T00:14:30.677 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010189962 2020-01-16T00:14:30.677 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.677 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.678 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.678 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404313+0000", 2020-01-16T00:14:30.678 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000108646 2020-01-16T00:14:30.678 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.678 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.678 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:30.678 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404422+0000", 2020-01-16T00:14:30.679 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119133 2020-01-16T00:14:30.679 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.679 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.679 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:30.679 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404541+0000", 2020-01-16T00:14:30.679 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000433764 2020-01-16T00:14:30.679 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.680 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.680 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:30.680 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404975+0000", 2020-01-16T00:14:30.680 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000266013 2020-01-16T00:14:30.680 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.680 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.680 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:30.680 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405241+0000", 2020-01-16T00:14:30.681 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5187000000000001e-05 2020-01-16T00:14:30.681 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.681 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.681 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:30.681 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405266+0000", 2020-01-16T00:14:30.681 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0889000000000002e-05 2020-01-16T00:14:30.681 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.682 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.682 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:30.682 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405297+0000", 2020-01-16T00:14:30.682 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4354999999999999e-05 2020-01-16T00:14:30.682 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.682 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.682 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:30.682 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405331+0000", 2020-01-16T00:14:30.683 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016440475999999999 2020-01-16T00:14:30.683 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.683 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:30.683 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.683 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.683 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.684 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3436 3.0 3:0b70bb20:::smithi12313934-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e47)", 2020-01-16T00:14:30.684 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:34.388920+0000", 2020-01-16T00:14:30.684 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 56.046330969000003, 2020-01-16T00:14:30.684 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016552641, 2020-01-16T00:14:30.684 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:30.684 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:30.685 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:30.685 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:30.685 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:30.685 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3436 2020-01-16T00:14:30.685 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.685 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:30.685 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.685 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:30.686 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388920+0000", 2020-01-16T00:14:30.686 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:30.686 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.686 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.686 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:30.686 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388920+0000", 2020-01-16T00:14:30.686 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:30.687 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.687 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.687 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:30.687 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388919+0000", 2020-01-16T00:14:30.687 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.181e-06 2020-01-16T00:14:30.687 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.687 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.688 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:30.688 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388921+0000", 2020-01-16T00:14:30.688 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9940000000000003e-06 2020-01-16T00:14:30.688 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.688 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.688 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:30.689 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388928+0000", 2020-01-16T00:14:30.689 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3549999999999999e-06 2020-01-16T00:14:30.689 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.689 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.689 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:30.689 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.388931+0000", 2020-01-16T00:14:30.689 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.005242483 2020-01-16T00:14:30.689 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.690 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.690 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.690 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.394173+0000", 2020-01-16T00:14:30.690 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7435999999999995e-05 2020-01-16T00:14:30.690 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.690 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.690 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:30.691 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.394251+0000", 2020-01-16T00:14:30.691 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010593286 2020-01-16T00:14:30.691 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.691 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.691 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.691 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404844+0000", 2020-01-16T00:14:30.691 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0620999999999999e-05 2020-01-16T00:14:30.691 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.692 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.692 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:30.692 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.404925+0000", 2020-01-16T00:14:30.692 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00042800899999999997 2020-01-16T00:14:30.692 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.692 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.692 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.692 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405353+0000", 2020-01-16T00:14:30.693 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2250000000000003e-05 2020-01-16T00:14:30.693 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.693 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.693 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:30.693 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405405+0000", 2020-01-16T00:14:30.693 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7411999999999998e-05 2020-01-16T00:14:30.694 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.694 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.694 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:30.694 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:34.405472+0000", 2020-01-16T00:14:30.694 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016552641 2020-01-16T00:14:30.694 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.694 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:30.695 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.695 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.695 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.695 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4721 3.0 3:0b70bb20:::smithi12313934-46:head [setxattr _header (56) in=7b,truncate 2134621] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-01-16T00:14:30.695 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.045819+0000", 2020-01-16T00:14:30.695 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 23.389431814999998, 2020-01-16T00:14:30.695 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019902080999999999, 2020-01-16T00:14:30.696 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:30.696 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:30.696 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:30.696 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:30.696 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:30.696 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4721 2020-01-16T00:14:30.696 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.697 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:30.697 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.697 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:30.697 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045819+0000", 2020-01-16T00:14:30.697 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:30.697 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.697 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.698 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:30.698 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045819+0000", 2020-01-16T00:14:30.698 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:30.698 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.698 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.698 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:30.699 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045818+0000", 2020-01-16T00:14:30.699 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.428e-06 2020-01-16T00:14:30.699 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.699 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.699 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:30.699 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045822+0000", 2020-01-16T00:14:30.699 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5940000000000004e-06 2020-01-16T00:14:30.700 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.700 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.700 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:30.700 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045826+0000", 2020-01-16T00:14:30.700 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6619999999999998e-06 2020-01-16T00:14:30.700 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.700 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.700 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:30.701 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045830+0000", 2020-01-16T00:14:30.701 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001710676 2020-01-16T00:14:30.701 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.701 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.701 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.701 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.047541+0000", 2020-01-16T00:14:30.701 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001637009 2020-01-16T00:14:30.701 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.702 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.702 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:30.702 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049178+0000", 2020-01-16T00:14:30.702 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.01461612 2020-01-16T00:14:30.702 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.702 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.702 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.702 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.063794+0000", 2020-01-16T00:14:30.703 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4941999999999997e-05 2020-01-16T00:14:30.703 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.703 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.703 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:30.703 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.063849+0000", 2020-01-16T00:14:30.703 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011165 2020-01-16T00:14:30.703 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.704 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.704 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 2,6", 2020-01-16T00:14:30.704 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.063960+0000", 2020-01-16T00:14:30.704 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013431630000000001 2020-01-16T00:14:30.704 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.704 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.704 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:30.705 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.065303+0000", 2020-01-16T00:14:30.705 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021894699999999999 2020-01-16T00:14:30.705 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.705 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.705 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:30.705 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.065522+0000", 2020-01-16T00:14:30.705 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013033899999999999 2020-01-16T00:14:30.705 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.706 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.706 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:30.706 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.065653+0000", 2020-01-16T00:14:30.706 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3800000000000002e-05 2020-01-16T00:14:30.706 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.706 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.706 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:30.707 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.065687+0000", 2020-01-16T00:14:30.707 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4332999999999998e-05 2020-01-16T00:14:30.707 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.707 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.707 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:30.707 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.065721+0000", 2020-01-16T00:14:30.707 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019902080999999999 2020-01-16T00:14:30.707 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.708 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:30.708 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.708 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.708 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.708 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4722 3.0 3:0b70bb20:::smithi12313934-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e58)", 2020-01-16T00:14:30.708 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.045850+0000", 2020-01-16T00:14:30.708 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 23.389401056000001, 2020-01-16T00:14:30.709 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020039033000000001, 2020-01-16T00:14:30.709 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:30.709 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:30.709 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:30.709 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:30.710 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:30.710 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4722 2020-01-16T00:14:30.710 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.710 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:30.710 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.710 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:30.710 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045850+0000", 2020-01-16T00:14:30.710 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:30.711 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.711 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.711 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:30.711 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045850+0000", 2020-01-16T00:14:30.711 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:30.711 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.712 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.712 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:30.712 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045849+0000", 2020-01-16T00:14:30.712 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7349999999999999e-06 2020-01-16T00:14:30.712 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.712 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.712 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:30.712 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045856+0000", 2020-01-16T00:14:30.713 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1330000000000001e-06 2020-01-16T00:14:30.713 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.713 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.713 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:30.713 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045860+0000", 2020-01-16T00:14:30.713 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.565e-06 2020-01-16T00:14:30.714 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.714 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.714 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:30.714 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.045863+0000", 2020-01-16T00:14:30.714 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033590019999999998 2020-01-16T00:14:30.714 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.715 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.715 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.715 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049222+0000", 2020-01-16T00:14:30.715 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8267e-05 2020-01-16T00:14:30.715 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.715 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.715 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:30.715 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049320+0000", 2020-01-16T00:14:30.716 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015128875999999999 2020-01-16T00:14:30.716 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.716 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.716 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.716 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.064449+0000", 2020-01-16T00:14:30.716 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4913000000000001e-05 2020-01-16T00:14:30.716 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.716 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.717 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:30.717 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.064504+0000", 2020-01-16T00:14:30.717 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012498100000000001 2020-01-16T00:14:30.717 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.717 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.717 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:30.717 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.065753+0000", 2020-01-16T00:14:30.718 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6299999999999999e-05 2020-01-16T00:14:30.718 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.718 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.718 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:30.718 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.065820+0000", 2020-01-16T00:14:30.718 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8801999999999996e-05 2020-01-16T00:14:30.718 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:30.718 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:30.719 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:30.719 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.065889+0000", 2020-01-16T00:14:30.719 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020039033000000001 2020-01-16T00:14:30.719 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.719 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:30.720 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.720 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:30.720 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:30.720 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:30.721 INFO:tasks.rados.rados.0.smithi123.stdout:1946: expect (ObjNum 579 snap 0 seq_num 579) 2020-01-16T00:14:30.722 INFO:tasks.rados.rados.0.smithi123.stdout:1949: expect (ObjNum 693 snap 0 seq_num 693) 2020-01-16T00:14:30.722 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:30.720+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:30.722 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-01-16T00:14:30.823 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:30.822+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:30.872 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:30.872 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:14:30.872 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:14:30.873 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:30.881 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-01-16T00:14:30.923 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:30.920+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:30.948 INFO:tasks.rados.rados.0.smithi123.stdout:1952: finishing write tid 1 to smithi12313934-39 2020-01-16T00:14:30.949 INFO:tasks.rados.rados.0.smithi123.stdout:1952: finishing write tid 2 to smithi12313934-39 2020-01-16T00:14:30.949 INFO:tasks.rados.rados.0.smithi123.stdout:1952: finishing write tid 4 to smithi12313934-39 2020-01-16T00:14:30.949 INFO:tasks.rados.rados.0.smithi123.stdout:1952: finishing write tid 5 to smithi12313934-39 2020-01-16T00:14:30.949 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 39 v 309 (ObjNum 737 snap 0 seq_num 737) dirty exists 2020-01-16T00:14:30.949 INFO:tasks.rados.rados.0.smithi123.stdout:1952: left oid 39 (ObjNum 737 snap 0 seq_num 737) 2020-01-16T00:14:30.949 INFO:tasks.rados.rados.0.smithi123.stdout:1953: finishing write tid 1 to smithi12313934-33 2020-01-16T00:14:30.950 INFO:tasks.rados.rados.0.smithi123.stdout:1953: finishing write tid 2 to smithi12313934-33 2020-01-16T00:14:30.950 INFO:tasks.rados.rados.0.smithi123.stdout:1953: finishing write tid 4 to smithi12313934-33 2020-01-16T00:14:30.950 INFO:tasks.rados.rados.0.smithi123.stdout:1953: finishing write tid 5 to smithi12313934-33 2020-01-16T00:14:30.950 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 33 v 383 (ObjNum 738 snap 0 seq_num 738) dirty exists 2020-01-16T00:14:30.950 INFO:tasks.rados.rados.0.smithi123.stdout:1953: left oid 33 (ObjNum 738 snap 0 seq_num 738) 2020-01-16T00:14:30.950 INFO:tasks.rados.rados.0.smithi123.stdout:1954: finishing write tid 1 to smithi12313934-5 2020-01-16T00:14:30.950 INFO:tasks.rados.rados.0.smithi123.stdout:1954: finishing write tid 2 to smithi12313934-5 2020-01-16T00:14:30.950 INFO:tasks.rados.rados.0.smithi123.stdout:1954: finishing write tid 4 to smithi12313934-5 2020-01-16T00:14:30.951 INFO:tasks.rados.rados.0.smithi123.stdout:1954: finishing write tid 5 to smithi12313934-5 2020-01-16T00:14:30.951 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 5 v 317 (ObjNum 739 snap 0 seq_num 739) dirty exists 2020-01-16T00:14:30.951 INFO:tasks.rados.rados.0.smithi123.stdout:1954: left oid 5 (ObjNum 739 snap 0 seq_num 739) 2020-01-16T00:14:30.951 INFO:tasks.rados.rados.0.smithi123.stdout:1955: finishing write tid 1 to smithi12313934-12 2020-01-16T00:14:30.951 INFO:tasks.rados.rados.0.smithi123.stdout:1955: finishing write tid 2 to smithi12313934-12 2020-01-16T00:14:30.951 INFO:tasks.rados.rados.0.smithi123.stdout:1955: finishing write tid 4 to smithi12313934-12 2020-01-16T00:14:30.951 INFO:tasks.rados.rados.0.smithi123.stdout:1955: finishing write tid 5 to smithi12313934-12 2020-01-16T00:14:30.951 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 12 v 205 (ObjNum 740 snap 0 seq_num 740) dirty exists 2020-01-16T00:14:30.952 INFO:tasks.rados.rados.0.smithi123.stdout:1955: left oid 12 (ObjNum 740 snap 0 seq_num 740) 2020-01-16T00:14:30.952 INFO:tasks.rados.rados.0.smithi123.stdout:1956: finishing write tid 1 to smithi12313934-8 2020-01-16T00:14:30.952 INFO:tasks.rados.rados.0.smithi123.stdout:1944: done (15 left) 2020-01-16T00:14:30.952 INFO:tasks.rados.rados.0.smithi123.stdout:1945: done (14 left) 2020-01-16T00:14:30.952 INFO:tasks.rados.rados.0.smithi123.stdout:1946: done (13 left) 2020-01-16T00:14:30.952 INFO:tasks.rados.rados.0.smithi123.stdout:1947: done (12 left) 2020-01-16T00:14:30.952 INFO:tasks.rados.rados.0.smithi123.stdout:1948: done (11 left) 2020-01-16T00:14:30.952 INFO:tasks.rados.rados.0.smithi123.stdout:1949: done (10 left) 2020-01-16T00:14:30.953 INFO:tasks.rados.rados.0.smithi123.stdout:1950: done (9 left) 2020-01-16T00:14:30.953 INFO:tasks.rados.rados.0.smithi123.stdout:1951: done (8 left) 2020-01-16T00:14:30.953 INFO:tasks.rados.rados.0.smithi123.stdout:1952: done (7 left) 2020-01-16T00:14:30.953 INFO:tasks.rados.rados.0.smithi123.stdout:1953: done (6 left) 2020-01-16T00:14:30.953 INFO:tasks.rados.rados.0.smithi123.stdout:1954: done (5 left) 2020-01-16T00:14:30.953 INFO:tasks.rados.rados.0.smithi123.stdout:1955: done (4 left) 2020-01-16T00:14:30.953 INFO:tasks.rados.rados.0.smithi123.stdout:1960: read oid 10 snap -1 2020-01-16T00:14:30.953 INFO:tasks.rados.rados.0.smithi123.stdout:1960: expect (ObjNum 698 snap 0 seq_num 698) 2020-01-16T00:14:30.954 INFO:tasks.rados.rados.0.smithi123.stdout:1961: read oid 42 snap -1 2020-01-16T00:14:30.954 INFO:tasks.rados.rados.0.smithi123.stdout:1961: expect (ObjNum 705 snap 0 seq_num 705) 2020-01-16T00:14:30.954 INFO:tasks.rados.rados.0.smithi123.stdout:1962: write oid 34 current snap is 0 2020-01-16T00:14:30.954 INFO:tasks.rados.rados.0.smithi123.stdout:1962: seq_num 743 ranges {487601=796248,1962121=525480} 2020-01-16T00:14:30.958 INFO:tasks.rados.rados.0.smithi123.stdout:1962: writing smithi12313934-34 from 487601 to 1283849 tid 1 2020-01-16T00:14:30.966 INFO:tasks.rados.rados.0.smithi123.stdout:1962: writing smithi12313934-34 from 1962121 to 2487601 tid 2 2020-01-16T00:14:30.966 INFO:tasks.rados.rados.0.smithi123.stdout:1963: write (excl) oid 50 current snap is 0 2020-01-16T00:14:30.966 INFO:tasks.rados.rados.0.smithi123.stdout:1963: seq_num 744 ranges {504408=471497,1620603=557925} 2020-01-16T00:14:30.972 INFO:tasks.rados.rados.0.smithi123.stdout:1963: writing smithi12313934-50 from 504408 to 975905 tid 1 2020-01-16T00:14:30.980 INFO:tasks.rados.rados.0.smithi123.stdout:1963: writing smithi12313934-50 from 1620603 to 2178528 tid 2 2020-01-16T00:14:30.981 INFO:tasks.rados.rados.0.smithi123.stdout:1964: write (excl) oid 21 current snap is 0 2020-01-16T00:14:30.981 INFO:tasks.rados.rados.0.smithi123.stdout:1964: seq_num 745 ranges {521215=546746,1562732=701583} 2020-01-16T00:14:30.987 INFO:tasks.rados.rados.0.smithi123.stdout:1964: writing smithi12313934-21 from 521215 to 1067961 tid 1 2020-01-16T00:14:30.997 INFO:tasks.rados.rados.0.smithi123.stdout:1964: writing smithi12313934-21 from 1562732 to 2264315 tid 2 2020-01-16T00:14:30.997 INFO:tasks.rados.rados.0.smithi123.stdout:1965: read oid 17 snap -1 2020-01-16T00:14:30.997 INFO:tasks.rados.rados.0.smithi123.stdout:1965: expect (ObjNum 675 snap 0 seq_num 675) 2020-01-16T00:14:30.997 INFO:tasks.rados.rados.0.smithi123.stdout:1966: read oid 1 snap -1 2020-01-16T00:14:30.998 INFO:tasks.rados.rados.0.smithi123.stdout:1966: expect (ObjNum 729 snap 0 seq_num 729) 2020-01-16T00:14:30.998 INFO:tasks.rados.rados.0.smithi123.stdout:1967: write (excl) oid 47 current snap is 0 2020-01-16T00:14:30.998 INFO:tasks.rados.rados.0.smithi123.stdout:1967: seq_num 746 ranges {538022=738348,1737567=561594} 2020-01-16T00:14:31.004 INFO:tasks.rados.rados.0.smithi123.stdout:1967: writing smithi12313934-47 from 538022 to 1276370 tid 1 2020-01-16T00:14:31.011 INFO:tasks.rados.rados.0.smithi123.stdout:1967: writing smithi12313934-47 from 1737567 to 2299161 tid 2 2020-01-16T00:14:31.011 INFO:tasks.rados.rados.0.smithi123.stdout:1968: read oid 29 snap -1 2020-01-16T00:14:31.011 INFO:tasks.rados.rados.0.smithi123.stdout:1968: expect (ObjNum 659 snap 0 seq_num 659) 2020-01-16T00:14:31.011 INFO:tasks.rados.rados.0.smithi123.stdout:1969: write (excl) oid 14 current snap is 0 2020-01-16T00:14:31.011 INFO:tasks.rados.rados.0.smithi123.stdout:1969: seq_num 747 ranges {554829=413597,1396049=705252} 2020-01-16T00:14:31.017 INFO:tasks.rados.rados.0.smithi123.stdout:1969: writing smithi12313934-14 from 554829 to 968426 tid 1 2020-01-16T00:14:31.024 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:31.020+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:31.024 INFO:tasks.rados.rados.0.smithi123.stdout:1969: writing smithi12313934-14 from 1396049 to 2101301 tid 2 2020-01-16T00:14:31.024 INFO:tasks.rados.rados.0.smithi123.stdout:1970: write oid 32 current snap is 0 2020-01-16T00:14:31.025 INFO:tasks.rados.rados.0.smithi123.stdout:1970: seq_num 748 ranges {571636=488846,1854531=565263} 2020-01-16T00:14:31.030 INFO:tasks.rados.rados.0.smithi123.stdout:1970: writing smithi12313934-32 from 571636 to 1060482 tid 1 2020-01-16T00:14:31.038 INFO:tasks.rados.rados.0.smithi123.stdout:1970: writing smithi12313934-32 from 1854531 to 2419794 tid 2 2020-01-16T00:14:31.038 INFO:tasks.rados.rados.0.smithi123.stdout:1971: write oid 30 current snap is 0 2020-01-16T00:14:31.038 INFO:tasks.rados.rados.0.smithi123.stdout:1971: seq_num 749 ranges {588443=564095,1796660=708921} 2020-01-16T00:14:31.044 INFO:tasks.rados.rados.0.smithi123.stdout:1971: writing smithi12313934-30 from 588443 to 1152538 tid 1 2020-01-16T00:14:31.052 INFO:tasks.rados.rados.0.smithi123.stdout:1971: writing smithi12313934-30 from 1796660 to 2505581 tid 2 2020-01-16T00:14:31.052 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:31.052 INFO:tasks.rados.rados.0.smithi123.stdout:1958: done (15 left) 2020-01-16T00:14:31.053 INFO:tasks.rados.rados.0.smithi123.stdout:1972: read oid 41 snap -1 2020-01-16T00:14:31.053 INFO:tasks.rados.rados.0.smithi123.stdout:1972: expect deleted 2020-01-16T00:14:31.053 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:31.053 INFO:tasks.rados.rados.0.smithi123.stdout:1956: finishing write tid 2 to smithi12313934-8 2020-01-16T00:14:31.053 INFO:tasks.rados.rados.0.smithi123.stdout:1956: finishing write tid 4 to smithi12313934-8 2020-01-16T00:14:31.053 INFO:tasks.rados.rados.0.smithi123.stdout:1956: finishing write tid 5 to smithi12313934-8 2020-01-16T00:14:31.054 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 8 v 134 (ObjNum 741 snap 0 seq_num 741) dirty exists 2020-01-16T00:14:31.054 INFO:tasks.rados.rados.0.smithi123.stdout:1956: left oid 8 (ObjNum 741 snap 0 seq_num 741) 2020-01-16T00:14:31.054 INFO:tasks.rados.rados.0.smithi123.stdout:1956: done (15 left) 2020-01-16T00:14:31.054 INFO:tasks.rados.rados.0.smithi123.stdout:1973: write oid 39 current snap is 0 2020-01-16T00:14:31.054 INFO:tasks.rados.rados.0.smithi123.stdout:1973: seq_num 750 ranges {605250=639344,1855142=452579} 2020-01-16T00:14:31.055 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:31.055 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:14:31.055 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:14:31.056 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:14:31.056 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:31.059 INFO:tasks.rados.rados.0.smithi123.stdout:1973: writing smithi12313934-39 from 605250 to 1244594 tid 1 2020-01-16T00:14:31.065 INFO:tasks.rados.rados.0.smithi123.stdout:1973: writing smithi12313934-39 from 1855142 to 2307721 tid 2 2020-01-16T00:14:31.066 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:31.066 INFO:tasks.rados.rados.0.smithi123.stdout:1957: expect (ObjNum 667 snap 0 seq_num 667) 2020-01-16T00:14:31.067 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-01-16T00:14:31.111 INFO:teuthology.orchestra.run.smithi123.stderr:noscrub is set 2020-01-16T00:14:31.124 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:31.120+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:31.213 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:31.213 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:14:31.213 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:14:31.214 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:14:31.214 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.214 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1650 3.1 3:8de1d8a3:::smithi12313934-13:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:14:31.214 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847717+0000", 2020-01-16T00:14:31.215 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 99.360961341999996, 2020-01-16T00:14:31.215 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010913658, 2020-01-16T00:14:31.215 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:31.215 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:31.216 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:31.216 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:31.216 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:31.216 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1650 2020-01-16T00:14:31.216 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.217 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:31.217 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.217 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:31.217 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847717+0000", 2020-01-16T00:14:31.218 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:31.218 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.218 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.218 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:31.219 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847717+0000", 2020-01-16T00:14:31.219 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:31.219 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.219 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.219 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:31.220 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847716+0000", 2020-01-16T00:14:31.220 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.433e-06 2020-01-16T00:14:31.220 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.220 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.221 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:31.221 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847720+0000", 2020-01-16T00:14:31.221 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1800000000000004e-06 2020-01-16T00:14:31.221 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.222 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.222 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:31.222 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847725+0000", 2020-01-16T00:14:31.222 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6689999999999997e-06 2020-01-16T00:14:31.223 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.223 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.223 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:31.224 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847730+0000", 2020-01-16T00:14:31.224 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000102468 2020-01-16T00:14:31.224 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.224 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.224 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.225 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847832+0000", 2020-01-16T00:14:31.225 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3999999999999997e-06 2020-01-16T00:14:31.225 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.225 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.226 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:14:31.226 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847838+0000", 2020-01-16T00:14:31.226 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010519395000000001 2020-01-16T00:14:31.226 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.227 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.227 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.228 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858358+0000", 2020-01-16T00:14:31.228 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7600000000000002e-05 2020-01-16T00:14:31.228 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.228 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.229 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:31.230 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858445+0000", 2020-01-16T00:14:31.230 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018515900000000001 2020-01-16T00:14:31.230 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.230 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.231 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:31.231 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858631+0000", 2020-01-16T00:14:31.231 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010913658 2020-01-16T00:14:31.231 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.231 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:31.232 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.232 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.232 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.232 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:1651 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~2084068,checksum] snapc 0=[] ondisk+read+known_if_redirected e30)", 2020-01-16T00:14:31.233 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:51.847752+0000", 2020-01-16T00:14:31.233 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 99.360926726000002, 2020-01-16T00:14:31.233 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011734249, 2020-01-16T00:14:31.234 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:31.234 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:31.234 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:31.234 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:31.234 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:31.235 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 1651 2020-01-16T00:14:31.235 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.235 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:31.235 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.236 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:31.236 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847752+0000", 2020-01-16T00:14:31.236 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:31.236 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.236 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.237 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:31.237 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847752+0000", 2020-01-16T00:14:31.237 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:31.237 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.238 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.238 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:31.238 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847751+0000", 2020-01-16T00:14:31.238 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1980000000000001e-06 2020-01-16T00:14:31.238 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.239 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.239 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:31.239 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847754+0000", 2020-01-16T00:14:31.239 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6679999999999999e-06 2020-01-16T00:14:31.240 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.240 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.240 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:31.240 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847758+0000", 2020-01-16T00:14:31.240 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2500000000000001e-06 2020-01-16T00:14:31.241 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.241 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.241 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:31.241 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847760+0000", 2020-01-16T00:14:31.242 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7614000000000004e-05 2020-01-16T00:14:31.242 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.242 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.242 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.242 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847858+0000", 2020-01-16T00:14:31.243 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7950000000000004e-06 2020-01-16T00:14:31.243 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.243 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.243 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting_for_map not empty", 2020-01-16T00:14:31.244 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.847864+0000", 2020-01-16T00:14:31.244 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010790345 2020-01-16T00:14:31.244 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.244 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.245 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.245 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858654+0000", 2020-01-16T00:14:31.245 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000141466 2020-01-16T00:14:31.245 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.246 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.246 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:31.246 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.858795+0000", 2020-01-16T00:14:31.246 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00069037899999999997 2020-01-16T00:14:31.246 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.247 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.247 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:31.247 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:51.859486+0000", 2020-01-16T00:14:31.247 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011734249 2020-01-16T00:14:31.248 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.248 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:31.248 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.248 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.248 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.249 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2108 3.1 3:8de1d8a3:::smithi12313934-13:head [write 2816700~410210] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:14:31.249 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.641861+0000", 2020-01-16T00:14:31.249 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 89.566817536000002, 2020-01-16T00:14:31.249 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018545604, 2020-01-16T00:14:31.250 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:31.250 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:31.250 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:31.250 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:31.250 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:31.251 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2108 2020-01-16T00:14:31.251 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.251 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:31.251 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.252 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:31.252 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.641861+0000", 2020-01-16T00:14:31.252 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:31.252 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.252 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.253 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:31.253 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.641861+0000", 2020-01-16T00:14:31.253 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:31.253 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.254 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.254 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:31.254 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.641859+0000", 2020-01-16T00:14:31.255 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018466399999999999 2020-01-16T00:14:31.255 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.255 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.255 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:31.256 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642043+0000", 2020-01-16T00:14:31.256 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2999999999999999e-05 2020-01-16T00:14:31.256 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.256 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.256 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:31.257 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642056+0000", 2020-01-16T00:14:31.257 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.0730000000000005e-06 2020-01-16T00:14:31.257 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.257 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.258 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:31.258 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642064+0000", 2020-01-16T00:14:31.258 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9527999999999997e-05 2020-01-16T00:14:31.258 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.259 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.259 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.259 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642114+0000", 2020-01-16T00:14:31.259 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036616050000000001 2020-01-16T00:14:31.260 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.260 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.260 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:31.260 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645776+0000", 2020-01-16T00:14:31.260 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012222663999999999 2020-01-16T00:14:31.261 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.261 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.261 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.262 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.657998+0000", 2020-01-16T00:14:31.262 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014280299999999999 2020-01-16T00:14:31.262 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.263 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.263 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:31.263 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.658141+0000", 2020-01-16T00:14:31.263 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0001102 2020-01-16T00:14:31.263 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.264 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.264 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:14:31.264 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.658251+0000", 2020-01-16T00:14:31.265 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020421219999999999 2020-01-16T00:14:31.265 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.265 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.265 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:31.266 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660293+0000", 2020-01-16T00:14:31.266 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7399999999999999e-06 2020-01-16T00:14:31.266 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.266 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.267 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:31.267 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660303+0000", 2020-01-16T00:14:31.267 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9873999999999998e-05 2020-01-16T00:14:31.267 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.267 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.268 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:31.268 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660343+0000", 2020-01-16T00:14:31.268 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4448e-05 2020-01-16T00:14:31.268 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.269 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.269 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:31.269 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660377+0000", 2020-01-16T00:14:31.269 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8915999999999999e-05 2020-01-16T00:14:31.270 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.270 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.270 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:31.270 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660406+0000", 2020-01-16T00:14:31.271 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018545604 2020-01-16T00:14:31.271 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.271 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:31.271 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.271 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.272 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.272 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2109 3.1 3:8de1d8a3:::smithi12313934-13:head [setxattr _header (56) in=7b,truncate 3865610] snapc 0=[] ondisk+write+known_if_redirected e33)", 2020-01-16T00:14:31.272 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.642075+0000", 2020-01-16T00:14:31.272 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 89.566602962999994, 2020-01-16T00:14:31.273 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018477655999999999, 2020-01-16T00:14:31.273 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:31.273 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:31.273 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:31.273 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:31.274 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:31.274 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2109 2020-01-16T00:14:31.274 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.274 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:31.275 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.275 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:31.275 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642075+0000", 2020-01-16T00:14:31.276 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:31.276 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.276 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.276 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:31.277 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642075+0000", 2020-01-16T00:14:31.277 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:31.277 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.277 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.277 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:31.278 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642074+0000", 2020-01-16T00:14:31.278 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4500000000000003e-06 2020-01-16T00:14:31.278 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.278 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.279 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:31.279 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642080+0000", 2020-01-16T00:14:31.279 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7229999999999998e-06 2020-01-16T00:14:31.279 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.279 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.280 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:31.280 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642087+0000", 2020-01-16T00:14:31.280 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2390000000000004e-06 2020-01-16T00:14:31.280 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.281 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.281 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:31.281 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642091+0000", 2020-01-16T00:14:31.281 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0037396550000000001 2020-01-16T00:14:31.281 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.282 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.282 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.282 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645830+0000", 2020-01-16T00:14:31.282 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2200000000000002e-05 2020-01-16T00:14:31.283 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.283 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.283 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:31.283 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.645883+0000", 2020-01-16T00:14:31.283 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013432934000000001 2020-01-16T00:14:31.284 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.284 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.284 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.284 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659316+0000", 2020-01-16T00:14:31.284 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000140006 2020-01-16T00:14:31.285 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.285 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.285 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:31.285 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659456+0000", 2020-01-16T00:14:31.285 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000184873 2020-01-16T00:14:31.285 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.286 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.286 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:14:31.286 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659640+0000", 2020-01-16T00:14:31.286 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00066954999999999998 2020-01-16T00:14:31.286 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.286 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.286 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:31.286 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660310+0000", 2020-01-16T00:14:31.287 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014847699999999999 2020-01-16T00:14:31.287 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.287 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.287 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:31.287 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660459+0000", 2020-01-16T00:14:31.287 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.6869e-05 2020-01-16T00:14:31.287 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.288 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.288 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:31.288 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660485+0000", 2020-01-16T00:14:31.288 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8478000000000001e-05 2020-01-16T00:14:31.288 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.288 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.288 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:31.288 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660514+0000", 2020-01-16T00:14:31.289 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9153999999999998e-05 2020-01-16T00:14:31.289 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.289 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.289 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:31.289 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660553+0000", 2020-01-16T00:14:31.289 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018477655999999999 2020-01-16T00:14:31.289 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.289 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:31.290 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.290 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.290 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.290 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2110 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e33)", 2020-01-16T00:14:31.290 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:01.642112+0000", 2020-01-16T00:14:31.290 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 89.566566324999997, 2020-01-16T00:14:31.290 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018620198000000001, 2020-01-16T00:14:31.290 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:31.291 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:31.291 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:31.291 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:31.291 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:31.291 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2110 2020-01-16T00:14:31.291 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.291 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:31.291 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.292 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:31.292 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642112+0000", 2020-01-16T00:14:31.292 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:31.292 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.292 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.292 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:31.292 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642112+0000", 2020-01-16T00:14:31.292 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:31.293 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.293 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.293 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:31.293 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642111+0000", 2020-01-16T00:14:31.293 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4450000000000001e-06 2020-01-16T00:14:31.293 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.293 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.293 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:31.294 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642117+0000", 2020-01-16T00:14:31.294 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1567e-05 2020-01-16T00:14:31.294 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.294 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.294 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:31.294 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642129+0000", 2020-01-16T00:14:31.294 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.754e-06 2020-01-16T00:14:31.295 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.295 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.295 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:31.295 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.642136+0000", 2020-01-16T00:14:31.295 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040239719999999998 2020-01-16T00:14:31.295 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.295 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.296 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.296 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646160+0000", 2020-01-16T00:14:31.296 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5600999999999998e-05 2020-01-16T00:14:31.296 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.296 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.296 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:31.296 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.646215+0000", 2020-01-16T00:14:31.297 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013733006000000001 2020-01-16T00:14:31.297 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.297 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.297 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.297 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.659948+0000", 2020-01-16T00:14:31.297 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000106277 2020-01-16T00:14:31.297 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.297 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.298 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:31.298 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660054+0000", 2020-01-16T00:14:31.298 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000524327 2020-01-16T00:14:31.298 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.298 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.298 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.298 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660579+0000", 2020-01-16T00:14:31.298 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9841999999999999e-05 2020-01-16T00:14:31.299 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.299 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.299 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:31.299 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660659+0000", 2020-01-16T00:14:31.299 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3572999999999995e-05 2020-01-16T00:14:31.299 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.299 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.299 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:31.300 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:01.660732+0000", 2020-01-16T00:14:31.300 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018620198000000001 2020-01-16T00:14:31.300 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.300 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:31.300 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.300 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.300 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.301 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:70 2.1 e35/19 2:91f85463:::smithi12313934-24 low_tier:head v 35'66, mlcod=34'64)", 2020-01-16T00:14:31.301 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.986684+0000", 2020-01-16T00:14:31.301 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 85.221994471000002, 2020-01-16T00:14:31.301 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012287378999999999, 2020-01-16T00:14:31.301 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:31.301 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:31.301 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:31.301 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.302 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:31.302 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986684+0000", 2020-01-16T00:14:31.302 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:31.302 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.302 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.302 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:31.303 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986684+0000", 2020-01-16T00:14:31.303 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:31.303 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.303 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.303 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:31.303 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986683+0000", 2020-01-16T00:14:31.303 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012884179999999999 2020-01-16T00:14:31.303 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.304 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.304 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:31.304 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.987971+0000", 2020-01-16T00:14:31.304 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011558200000000001 2020-01-16T00:14:31.304 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.304 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.304 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:31.304 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988087+0000", 2020-01-16T00:14:31.305 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0472000000000001e-05 2020-01-16T00:14:31.305 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.305 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.305 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:31.305 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988137+0000", 2020-01-16T00:14:31.305 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000107484 2020-01-16T00:14:31.305 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.306 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.306 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.306 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988245+0000", 2020-01-16T00:14:31.306 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4409999999999998e-05 2020-01-16T00:14:31.306 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.306 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.306 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:31.307 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.988279+0000", 2020-01-16T00:14:31.307 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010641609999999999 2020-01-16T00:14:31.307 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.307 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.307 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:31.307 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.998921+0000", 2020-01-16T00:14:31.307 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0278000000000002e-05 2020-01-16T00:14:31.308 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.308 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.308 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:31.308 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.998971+0000", 2020-01-16T00:14:31.308 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012287378999999999 2020-01-16T00:14:31.308 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.308 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:31.308 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.309 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.309 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.309 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3228 3.1 3:8de1d8a3:::smithi12313934-13:head [write 588344~631611,stat] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:14:31.309 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.588539+0000", 2020-01-16T00:14:31.309 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 61.620139234, 2020-01-16T00:14:31.309 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012776884000000001, 2020-01-16T00:14:31.309 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:31.309 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:31.310 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:31.310 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:31.310 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:31.310 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3228 2020-01-16T00:14:31.310 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.310 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:31.310 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.310 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:31.311 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588539+0000", 2020-01-16T00:14:31.311 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:31.311 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.311 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.311 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:31.311 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588539+0000", 2020-01-16T00:14:31.311 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:31.312 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.312 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.312 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:31.312 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588536+0000", 2020-01-16T00:14:31.312 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021452 2020-01-16T00:14:31.312 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.312 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.312 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:31.313 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588750+0000", 2020-01-16T00:14:31.313 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.1432000000000001e-05 2020-01-16T00:14:31.313 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.313 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.313 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:31.313 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588772+0000", 2020-01-16T00:14:31.314 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1799999999999999e-06 2020-01-16T00:14:31.314 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.314 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.314 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:31.314 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588779+0000", 2020-01-16T00:14:31.314 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00095777700000000004 2020-01-16T00:14:31.314 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.314 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.315 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.315 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589737+0000", 2020-01-16T00:14:31.315 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013306599999999999 2020-01-16T00:14:31.315 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.315 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.315 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:31.315 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589870+0000", 2020-01-16T00:14:31.315 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020556 2020-01-16T00:14:31.316 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.316 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.316 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:14:31.316 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.590075+0000", 2020-01-16T00:14:31.316 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040226080000000004 2020-01-16T00:14:31.316 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.317 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.317 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:31.317 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.594098+0000", 2020-01-16T00:14:31.317 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.006043288 2020-01-16T00:14:31.317 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.317 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.317 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:31.318 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.600141+0000", 2020-01-16T00:14:31.318 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00089242900000000003 2020-01-16T00:14:31.318 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.318 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.318 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:31.318 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601034+0000", 2020-01-16T00:14:31.318 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6097e-05 2020-01-16T00:14:31.318 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.319 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.319 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:31.319 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601110+0000", 2020-01-16T00:14:31.319 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000206088 2020-01-16T00:14:31.319 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.319 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.319 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:31.319 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601316+0000", 2020-01-16T00:14:31.320 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012776884000000001 2020-01-16T00:14:31.320 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.320 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:31.320 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.320 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.320 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.320 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3230 3.1 3:8de1d8a3:::smithi12313934-13:head [setxattr _header (56) in=7b,truncate 2588344] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:14:31.320 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.596394+0000", 2020-01-16T00:14:31.321 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 61.612284213999999, 2020-01-16T00:14:31.321 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021205134, 2020-01-16T00:14:31.321 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:31.321 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:31.321 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:31.321 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:31.321 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:31.322 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3230 2020-01-16T00:14:31.322 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.322 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:31.322 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.322 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:31.322 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596394+0000", 2020-01-16T00:14:31.322 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:31.322 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.323 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.323 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:31.323 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596394+0000", 2020-01-16T00:14:31.323 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:31.323 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.323 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.323 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:31.323 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596393+0000", 2020-01-16T00:14:31.324 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7159999999999997e-06 2020-01-16T00:14:31.324 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.324 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.324 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:31.324 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596400+0000", 2020-01-16T00:14:31.324 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0074e-05 2020-01-16T00:14:31.324 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.324 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.325 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:31.325 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596410+0000", 2020-01-16T00:14:31.325 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1679999999999997e-06 2020-01-16T00:14:31.325 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.325 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.325 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:31.325 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596414+0000", 2020-01-16T00:14:31.325 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023609 2020-01-16T00:14:31.326 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.326 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.326 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.326 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596650+0000", 2020-01-16T00:14:31.326 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012316399999999999 2020-01-16T00:14:31.326 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.327 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.327 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:31.327 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596774+0000", 2020-01-16T00:14:31.327 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0067849670000000003 2020-01-16T00:14:31.327 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.327 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.327 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.328 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.603558+0000", 2020-01-16T00:14:31.328 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00092385999999999998 2020-01-16T00:14:31.328 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.328 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.328 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:31.328 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.604482+0000", 2020-01-16T00:14:31.328 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011729201 2020-01-16T00:14:31.328 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.329 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.329 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.329 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.616212+0000", 2020-01-16T00:14:31.329 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019515099999999999 2020-01-16T00:14:31.329 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.329 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.329 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:31.329 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.616407+0000", 2020-01-16T00:14:31.330 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000182843 2020-01-16T00:14:31.330 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.330 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.330 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:14:31.330 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.616590+0000", 2020-01-16T00:14:31.330 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000660572 2020-01-16T00:14:31.331 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.331 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.331 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:31.331 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617250+0000", 2020-01-16T00:14:31.331 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5333999999999997e-05 2020-01-16T00:14:31.331 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.331 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.331 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:31.332 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617285+0000", 2020-01-16T00:14:31.332 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023020899999999999 2020-01-16T00:14:31.332 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.332 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.332 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:31.332 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617516+0000", 2020-01-16T00:14:31.332 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4214000000000001e-05 2020-01-16T00:14:31.332 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.333 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.333 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:31.333 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617560+0000", 2020-01-16T00:14:31.333 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9366999999999998e-05 2020-01-16T00:14:31.333 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.333 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.333 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:31.333 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617599+0000", 2020-01-16T00:14:31.334 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021205134 2020-01-16T00:14:31.334 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.334 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:31.334 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.334 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.334 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.334 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3231 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e45)", 2020-01-16T00:14:31.334 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.596438+0000", 2020-01-16T00:14:31.335 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 61.612240204000003, 2020-01-16T00:14:31.335 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021455667000000001, 2020-01-16T00:14:31.335 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:31.335 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:31.335 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:31.335 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:31.335 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:31.335 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3231 2020-01-16T00:14:31.336 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.336 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:31.336 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.336 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:31.336 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596438+0000", 2020-01-16T00:14:31.336 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:31.337 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.337 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.337 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:31.337 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596438+0000", 2020-01-16T00:14:31.337 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:31.337 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.337 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.337 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:31.338 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596437+0000", 2020-01-16T00:14:31.338 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8500000000000004e-06 2020-01-16T00:14:31.338 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.338 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.338 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:31.338 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596441+0000", 2020-01-16T00:14:31.338 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.9439999999999994e-06 2020-01-16T00:14:31.339 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.339 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.339 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:31.339 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596450+0000", 2020-01-16T00:14:31.339 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.267e-06 2020-01-16T00:14:31.339 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.339 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.339 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:31.340 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596455+0000", 2020-01-16T00:14:31.340 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034332399999999999 2020-01-16T00:14:31.340 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.340 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.340 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.340 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596798+0000", 2020-01-16T00:14:31.340 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011687400000000001 2020-01-16T00:14:31.340 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.341 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.341 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:31.341 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.596915+0000", 2020-01-16T00:14:31.341 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0076057049999999999 2020-01-16T00:14:31.341 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.341 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.341 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.341 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.604520+0000", 2020-01-16T00:14:31.342 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8143999999999998e-05 2020-01-16T00:14:31.342 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.342 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.342 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:31.342 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.604569+0000", 2020-01-16T00:14:31.342 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012379960000000001 2020-01-16T00:14:31.342 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.342 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.343 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.343 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.616949+0000", 2020-01-16T00:14:31.343 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000120617 2020-01-16T00:14:31.343 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.343 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.343 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:31.343 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617069+0000", 2020-01-16T00:14:31.343 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00056715400000000005 2020-01-16T00:14:31.344 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.344 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.344 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.344 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617636+0000", 2020-01-16T00:14:31.344 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012844700000000001 2020-01-16T00:14:31.344 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.344 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.344 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:31.345 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617765+0000", 2020-01-16T00:14:31.345 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000129037 2020-01-16T00:14:31.345 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.345 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.345 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:31.345 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.617894+0000", 2020-01-16T00:14:31.345 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021455667000000001 2020-01-16T00:14:31.345 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.346 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:31.346 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.346 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.346 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.346 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3822 3.1 3:8de1d8a3:::smithi12313934-13:head [setxattr _header (56) in=7b,truncate 2016939] snapc 0=[] ondisk+write+known_if_redirected e50)", 2020-01-16T00:14:31.346 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:44.059584+0000", 2020-01-16T00:14:31.346 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 47.149093888000003, 2020-01-16T00:14:31.347 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021977219999999999, 2020-01-16T00:14:31.347 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:31.347 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:31.347 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:31.347 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:31.347 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:31.348 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3822 2020-01-16T00:14:31.348 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.348 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:31.348 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.348 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:31.348 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059584+0000", 2020-01-16T00:14:31.348 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:31.348 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.349 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.349 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:31.349 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059584+0000", 2020-01-16T00:14:31.349 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:31.349 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.349 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.349 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:31.350 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059584+0000", 2020-01-16T00:14:31.350 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1329999999999998e-06 2020-01-16T00:14:31.350 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.350 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.350 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:31.350 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059589+0000", 2020-01-16T00:14:31.350 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8136000000000002e-05 2020-01-16T00:14:31.350 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.351 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.351 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:31.351 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059627+0000", 2020-01-16T00:14:31.351 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1409999999999998e-06 2020-01-16T00:14:31.351 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.351 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.351 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:31.351 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059631+0000", 2020-01-16T00:14:31.352 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0011314020000000001 2020-01-16T00:14:31.352 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.352 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.352 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.352 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.060763+0000", 2020-01-16T00:14:31.352 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00085078200000000001 2020-01-16T00:14:31.352 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.353 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.353 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:31.353 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.061613+0000", 2020-01-16T00:14:31.353 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017766311999999999 2020-01-16T00:14:31.353 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.353 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.353 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.353 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.079380+0000", 2020-01-16T00:14:31.354 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000133144 2020-01-16T00:14:31.354 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.354 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.354 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:31.354 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.079513+0000", 2020-01-16T00:14:31.354 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021271500000000001 2020-01-16T00:14:31.354 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.354 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.355 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:14:31.355 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.079725+0000", 2020-01-16T00:14:31.355 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001302582 2020-01-16T00:14:31.355 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.355 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.355 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:31.355 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081028+0000", 2020-01-16T00:14:31.356 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017197 2020-01-16T00:14:31.356 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.356 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.356 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:31.356 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081200+0000", 2020-01-16T00:14:31.356 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000202757 2020-01-16T00:14:31.356 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.356 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.357 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:31.357 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081403+0000", 2020-01-16T00:14:31.357 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1068999999999993e-05 2020-01-16T00:14:31.357 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.357 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.357 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:31.358 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081474+0000", 2020-01-16T00:14:31.358 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7800000000000006e-05 2020-01-16T00:14:31.358 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.358 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.358 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:31.358 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081562+0000", 2020-01-16T00:14:31.359 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021977219999999999 2020-01-16T00:14:31.359 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.359 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:31.359 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.359 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.359 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.359 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3823 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e50)", 2020-01-16T00:14:31.359 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:44.059639+0000", 2020-01-16T00:14:31.360 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 47.149039733000002, 2020-01-16T00:14:31.360 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022245396000000001, 2020-01-16T00:14:31.360 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:31.360 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:31.360 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:31.360 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:31.360 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:31.360 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3823 2020-01-16T00:14:31.361 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.361 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:31.361 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.361 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:31.361 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059639+0000", 2020-01-16T00:14:31.361 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:31.361 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.362 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.362 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:31.362 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059639+0000", 2020-01-16T00:14:31.362 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:31.362 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.362 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.363 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:31.363 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059638+0000", 2020-01-16T00:14:31.363 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.405e-06 2020-01-16T00:14:31.363 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.363 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.363 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:31.363 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059641+0000", 2020-01-16T00:14:31.364 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9770000000000003e-06 2020-01-16T00:14:31.364 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.364 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.364 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:31.364 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059648+0000", 2020-01-16T00:14:31.364 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1140000000000002e-06 2020-01-16T00:14:31.364 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.364 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.365 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:31.365 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.059652+0000", 2020-01-16T00:14:31.365 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020063659999999999 2020-01-16T00:14:31.365 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.365 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.365 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.365 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.061658+0000", 2020-01-16T00:14:31.365 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3183999999999999e-05 2020-01-16T00:14:31.366 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.366 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.366 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:31.366 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.061691+0000", 2020-01-16T00:14:31.366 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018817654999999999 2020-01-16T00:14:31.366 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.366 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.366 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.367 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.080509+0000", 2020-01-16T00:14:31.367 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000104409 2020-01-16T00:14:31.367 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.367 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.367 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:31.367 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.080613+0000", 2020-01-16T00:14:31.368 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00098755999999999996 2020-01-16T00:14:31.368 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.368 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.368 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.368 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081601+0000", 2020-01-16T00:14:31.368 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000145766 2020-01-16T00:14:31.368 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.368 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.369 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:31.369 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081746+0000", 2020-01-16T00:14:31.369 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013751799999999999 2020-01-16T00:14:31.370 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.370 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.370 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:31.370 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.081884+0000", 2020-01-16T00:14:31.370 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022245396000000001 2020-01-16T00:14:31.370 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.370 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:31.371 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.371 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.371 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.371 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4398 3.1 3:8de1d8a3:::smithi12313934-13:head [write 2812170~507386] snapc 0=[] ondisk+write+known_if_redirected e55)", 2020-01-16T00:14:31.371 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:58.336543+0000", 2020-01-16T00:14:31.371 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 32.872135299999997, 2020-01-16T00:14:31.371 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015697994, 2020-01-16T00:14:31.371 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:31.372 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:31.372 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:31.372 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:31.372 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:31.372 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4398 2020-01-16T00:14:31.372 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.372 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:31.372 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.373 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:31.373 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336543+0000", 2020-01-16T00:14:31.373 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:31.373 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.373 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.373 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:31.373 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336543+0000", 2020-01-16T00:14:31.373 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:31.374 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.374 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.374 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:31.374 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336542+0000", 2020-01-16T00:14:31.374 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014254 2020-01-16T00:14:31.374 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.374 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.374 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:31.375 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336684+0000", 2020-01-16T00:14:31.375 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2019e-05 2020-01-16T00:14:31.375 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.375 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.375 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:31.375 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336696+0000", 2020-01-16T00:14:31.375 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4269999999999996e-06 2020-01-16T00:14:31.376 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.376 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.376 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:31.376 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336702+0000", 2020-01-16T00:14:31.376 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8040000000000002e-05 2020-01-16T00:14:31.376 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.376 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.376 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.377 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336741+0000", 2020-01-16T00:14:31.377 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00030631400000000002 2020-01-16T00:14:31.377 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.377 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.377 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:31.377 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.337047+0000", 2020-01-16T00:14:31.378 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012249832 2020-01-16T00:14:31.378 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.378 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.378 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.378 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.349297+0000", 2020-01-16T00:14:31.378 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013870899999999999 2020-01-16T00:14:31.378 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.378 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.379 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:31.379 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.349435+0000", 2020-01-16T00:14:31.379 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021913600000000001 2020-01-16T00:14:31.379 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.379 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.380 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:14:31.380 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.349655+0000", 2020-01-16T00:14:31.380 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021982490000000002 2020-01-16T00:14:31.380 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.380 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.380 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:31.380 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351853+0000", 2020-01-16T00:14:31.380 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8295999999999996e-05 2020-01-16T00:14:31.381 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.381 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.381 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:31.381 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351941+0000", 2020-01-16T00:14:31.381 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021662 2020-01-16T00:14:31.381 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.381 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.381 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:31.382 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352158+0000", 2020-01-16T00:14:31.382 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6031000000000001e-05 2020-01-16T00:14:31.382 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.382 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.382 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:31.382 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352194+0000", 2020-01-16T00:14:31.382 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7290000000000003e-05 2020-01-16T00:14:31.382 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.383 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.383 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:31.383 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352241+0000", 2020-01-16T00:14:31.383 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015697994 2020-01-16T00:14:31.383 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.383 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:31.383 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.383 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.384 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.384 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4399 3.1 3:8de1d8a3:::smithi12313934-13:head [setxattr _header (56) in=7b,truncate 3411920] snapc 0=[] ondisk+write+known_if_redirected e55)", 2020-01-16T00:14:31.384 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:58.336714+0000", 2020-01-16T00:14:31.384 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 32.871963942999997, 2020-01-16T00:14:31.384 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016051421, 2020-01-16T00:14:31.384 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:31.384 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:31.384 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:31.385 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:31.385 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:31.385 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4399 2020-01-16T00:14:31.385 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.385 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:31.385 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.385 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:31.385 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336714+0000", 2020-01-16T00:14:31.386 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:31.386 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.386 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.386 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:31.386 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336714+0000", 2020-01-16T00:14:31.386 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:31.386 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.387 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.387 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:31.387 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336714+0000", 2020-01-16T00:14:31.387 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7540000000000003e-06 2020-01-16T00:14:31.387 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.387 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.387 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:31.387 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336719+0000", 2020-01-16T00:14:31.388 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2879999999999998e-06 2020-01-16T00:14:31.388 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.388 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.388 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:31.388 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336726+0000", 2020-01-16T00:14:31.388 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6399999999999996e-06 2020-01-16T00:14:31.389 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.389 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.389 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:31.389 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336730+0000", 2020-01-16T00:14:31.389 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00036299699999999999 2020-01-16T00:14:31.389 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.389 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.389 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.390 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.337093+0000", 2020-01-16T00:14:31.390 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1297999999999994e-05 2020-01-16T00:14:31.390 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.390 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.390 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:31.390 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.337165+0000", 2020-01-16T00:14:31.390 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013848911 2020-01-16T00:14:31.391 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.391 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.391 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.391 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351013+0000", 2020-01-16T00:14:31.391 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7732999999999994e-05 2020-01-16T00:14:31.391 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.391 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.391 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:31.392 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351111+0000", 2020-01-16T00:14:31.392 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000109089 2020-01-16T00:14:31.392 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.392 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.392 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:14:31.392 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351220+0000", 2020-01-16T00:14:31.392 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00063960199999999997 2020-01-16T00:14:31.392 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.393 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.393 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:31.393 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351860+0000", 2020-01-16T00:14:31.393 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00050205699999999998 2020-01-16T00:14:31.393 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.393 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.393 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:31.393 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352362+0000", 2020-01-16T00:14:31.394 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000282195 2020-01-16T00:14:31.394 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.394 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.394 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:31.394 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352644+0000", 2020-01-16T00:14:31.394 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3927000000000001e-05 2020-01-16T00:14:31.394 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.394 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.395 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:31.395 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352698+0000", 2020-01-16T00:14:31.395 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7770999999999999e-05 2020-01-16T00:14:31.395 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.395 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.395 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:31.395 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352766+0000", 2020-01-16T00:14:31.395 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016051421 2020-01-16T00:14:31.396 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.396 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:31.396 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.396 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.396 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.396 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4400 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e55)", 2020-01-16T00:14:31.396 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:58.336738+0000", 2020-01-16T00:14:31.396 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 32.871940213999999, 2020-01-16T00:14:31.397 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016207452000000001, 2020-01-16T00:14:31.397 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:31.397 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:31.397 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:31.397 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:31.397 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:31.397 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4400 2020-01-16T00:14:31.397 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.398 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:31.398 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.398 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:31.398 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336738+0000", 2020-01-16T00:14:31.398 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:31.399 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.399 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.399 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:31.399 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336738+0000", 2020-01-16T00:14:31.399 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:31.399 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.399 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.399 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:31.400 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336738+0000", 2020-01-16T00:14:31.400 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5200000000000002e-06 2020-01-16T00:14:31.400 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.400 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.400 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:31.400 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336741+0000", 2020-01-16T00:14:31.400 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.1500000000000004e-06 2020-01-16T00:14:31.400 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.401 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.401 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:31.401 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336747+0000", 2020-01-16T00:14:31.401 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7500000000000001e-06 2020-01-16T00:14:31.401 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.401 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.401 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:31.401 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.336751+0000", 2020-01-16T00:14:31.402 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043512299999999999 2020-01-16T00:14:31.402 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.402 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.402 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.402 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.337186+0000", 2020-01-16T00:14:31.402 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7048000000000002e-05 2020-01-16T00:14:31.402 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.402 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.403 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:31.403 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.337223+0000", 2020-01-16T00:14:31.403 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014258785 2020-01-16T00:14:31.403 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.403 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.403 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.403 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351482+0000", 2020-01-16T00:14:31.403 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012002000000000001 2020-01-16T00:14:31.404 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.404 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.404 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:31.404 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.351602+0000", 2020-01-16T00:14:31.404 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001192404 2020-01-16T00:14:31.405 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.405 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.405 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.405 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352794+0000", 2020-01-16T00:14:31.406 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8901999999999998e-05 2020-01-16T00:14:31.406 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.407 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.407 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:31.407 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352873+0000", 2020-01-16T00:14:31.407 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2317999999999994e-05 2020-01-16T00:14:31.407 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.407 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.407 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:31.408 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.352946+0000", 2020-01-16T00:14:31.408 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016207452000000001 2020-01-16T00:14:31.408 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.408 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:31.408 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.408 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.408 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.409 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.3.13:183 2.1 e58/19 2:a4df40e2:::smithi12313934-42 low_tier:head v 58'138, mlcod=57'136)", 2020-01-16T00:14:31.409 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.021967+0000", 2020-01-16T00:14:31.409 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 24.186711344999999, 2020-01-16T00:14:31.409 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011697308999999999, 2020-01-16T00:14:31.409 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:31.409 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:31.409 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:31.410 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.410 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:31.410 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021967+0000", 2020-01-16T00:14:31.410 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:31.410 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.410 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.410 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:31.411 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021967+0000", 2020-01-16T00:14:31.411 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:31.411 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.411 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.411 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:31.411 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021966+0000", 2020-01-16T00:14:31.412 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020837899999999999 2020-01-16T00:14:31.412 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.412 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.412 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:31.412 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.022175+0000", 2020-01-16T00:14:31.412 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7650000000000005e-06 2020-01-16T00:14:31.413 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.413 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.413 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:31.413 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.022185+0000", 2020-01-16T00:14:31.413 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.759e-06 2020-01-16T00:14:31.413 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.414 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.414 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:31.414 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.022191+0000", 2020-01-16T00:14:31.414 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00559108 2020-01-16T00:14:31.414 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.414 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.414 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.415 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.027782+0000", 2020-01-16T00:14:31.415 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5761e-05 2020-01-16T00:14:31.415 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.415 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.415 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:31.415 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.027798+0000", 2020-01-16T00:14:31.415 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0058039989999999998 2020-01-16T00:14:31.415 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.416 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.416 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:31.416 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.033602+0000", 2020-01-16T00:14:31.416 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2147e-05 2020-01-16T00:14:31.416 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.416 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.416 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:31.416 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.033664+0000", 2020-01-16T00:14:31.417 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011697308999999999 2020-01-16T00:14:31.417 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.417 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:31.417 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.417 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.417 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.417 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.1.13:98 2.1 e58/19 2:bafac7f5:::smithi12313934-46 low_tier:head v 58'140, mlcod=58'138)", 2020-01-16T00:14:31.418 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.050024+0000", 2020-01-16T00:14:31.418 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 24.158653987000001, 2020-01-16T00:14:31.418 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011307467, 2020-01-16T00:14:31.418 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:31.418 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:31.418 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:31.418 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.419 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:31.419 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.050024+0000", 2020-01-16T00:14:31.419 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:31.419 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.419 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.419 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:31.419 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.050024+0000", 2020-01-16T00:14:31.420 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:31.420 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.420 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.420 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:31.420 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.050022+0000", 2020-01-16T00:14:31.420 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000438839 2020-01-16T00:14:31.421 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.421 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.421 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:31.421 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.050461+0000", 2020-01-16T00:14:31.421 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9307999999999999e-05 2020-01-16T00:14:31.421 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.422 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.422 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:31.422 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.050480+0000", 2020-01-16T00:14:31.422 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7249999999999997e-06 2020-01-16T00:14:31.423 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.423 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.423 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:31.423 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.050488+0000", 2020-01-16T00:14:31.423 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00026342100000000001 2020-01-16T00:14:31.423 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.424 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.424 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.424 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.050751+0000", 2020-01-16T00:14:31.424 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8136000000000001e-05 2020-01-16T00:14:31.424 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.424 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.424 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:31.425 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.050769+0000", 2020-01-16T00:14:31.425 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010518582 2020-01-16T00:14:31.425 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.425 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.425 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:31.425 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.061288+0000", 2020-01-16T00:14:31.425 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4085000000000002e-05 2020-01-16T00:14:31.425 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.426 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.426 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:31.426 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.061332+0000", 2020-01-16T00:14:31.426 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011307467 2020-01-16T00:14:31.426 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.426 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:31.426 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.427 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.427 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.427 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:5566 3.1 3:8de1d8a3:::smithi12313934-13:head [write 568268~733811,stat] snapc 0=[] ondisk+write+known_if_redirected e66)", 2020-01-16T00:14:31.427 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:27.611601+0000", 2020-01-16T00:14:31.427 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 3.5970771949999998, 2020-01-16T00:14:31.427 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014368947, 2020-01-16T00:14:31.427 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:31.427 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:31.428 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:31.428 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:31.428 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:31.428 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 5566 2020-01-16T00:14:31.428 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.428 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:31.428 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.429 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:31.429 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.611601+0000", 2020-01-16T00:14:31.429 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:31.429 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.429 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.429 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:31.429 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.611601+0000", 2020-01-16T00:14:31.429 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:31.430 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.430 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.430 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:31.430 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.611598+0000", 2020-01-16T00:14:31.430 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022899199999999999 2020-01-16T00:14:31.430 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.430 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.431 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:31.431 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.611827+0000", 2020-01-16T00:14:31.431 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7127000000000001e-05 2020-01-16T00:14:31.431 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.431 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.431 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:31.431 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.611844+0000", 2020-01-16T00:14:31.432 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3100000000000001e-06 2020-01-16T00:14:31.432 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.432 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.432 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:31.432 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.611853+0000", 2020-01-16T00:14:31.432 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3891e-05 2020-01-16T00:14:31.433 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.433 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.433 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.433 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.611937+0000", 2020-01-16T00:14:31.433 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017038599999999999 2020-01-16T00:14:31.433 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.434 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.434 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:31.434 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.612107+0000", 2020-01-16T00:14:31.434 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00027321700000000002 2020-01-16T00:14:31.434 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.434 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.434 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:14:31.434 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.612380+0000", 2020-01-16T00:14:31.435 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0034700379999999999 2020-01-16T00:14:31.435 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.435 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.435 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:31.435 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.615850+0000", 2020-01-16T00:14:31.435 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00031966699999999999 2020-01-16T00:14:31.435 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.436 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.436 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:31.436 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.616170+0000", 2020-01-16T00:14:31.436 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0096376469999999992 2020-01-16T00:14:31.436 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.436 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.436 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:31.436 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.625807+0000", 2020-01-16T00:14:31.437 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2891e-05 2020-01-16T00:14:31.437 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.437 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.437 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:31.437 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.625850+0000", 2020-01-16T00:14:31.437 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119683 2020-01-16T00:14:31.437 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.438 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.438 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:31.438 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.625970+0000", 2020-01-16T00:14:31.438 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014368947 2020-01-16T00:14:31.438 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.439 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:31.439 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.439 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.439 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.439 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:5567 3.1 3:8de1d8a3:::smithi12313934-13:head [write 2000022~168246] snapc 0=[] ondisk+write+known_if_redirected e66)", 2020-01-16T00:14:31.439 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:27.616881+0000", 2020-01-16T00:14:31.439 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 3.5917968739999999, 2020-01-16T00:14:31.440 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013554490000000001, 2020-01-16T00:14:31.440 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:31.440 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:31.440 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:31.440 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:31.440 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:31.440 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 5567 2020-01-16T00:14:31.441 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.441 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:31.441 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.441 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:31.441 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.616881+0000", 2020-01-16T00:14:31.441 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:31.441 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.442 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.442 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:31.442 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.616881+0000", 2020-01-16T00:14:31.442 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:31.442 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.442 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.442 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:31.442 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.616878+0000", 2020-01-16T00:14:31.443 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3263999999999996e-05 2020-01-16T00:14:31.443 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.443 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.443 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:31.443 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.616941+0000", 2020-01-16T00:14:31.443 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8110000000000001e-05 2020-01-16T00:14:31.444 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.444 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.444 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:31.444 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.616959+0000", 2020-01-16T00:14:31.444 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9469999999999997e-06 2020-01-16T00:14:31.444 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.444 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.444 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:31.445 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.616966+0000", 2020-01-16T00:14:31.445 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8065000000000001e-05 2020-01-16T00:14:31.445 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.445 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.445 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.445 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.617015+0000", 2020-01-16T00:14:31.445 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013213400000000001 2020-01-16T00:14:31.446 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.446 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.446 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:31.446 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.617147+0000", 2020-01-16T00:14:31.446 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0088762949999999993 2020-01-16T00:14:31.446 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.446 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.446 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.447 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.626023+0000", 2020-01-16T00:14:31.447 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018361500000000001 2020-01-16T00:14:31.447 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.447 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.447 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:31.447 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.626207+0000", 2020-01-16T00:14:31.447 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000209936 2020-01-16T00:14:31.447 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.448 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.448 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:14:31.448 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.626416+0000", 2020-01-16T00:14:31.448 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0035327610000000001 2020-01-16T00:14:31.448 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.448 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.448 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:31.449 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.629949+0000", 2020-01-16T00:14:31.449 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00033993400000000002 2020-01-16T00:14:31.449 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.449 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.449 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:31.449 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.630289+0000", 2020-01-16T00:14:31.449 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0885e-05 2020-01-16T00:14:31.449 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.450 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.450 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:31.450 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.630300+0000", 2020-01-16T00:14:31.450 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3071000000000007e-05 2020-01-16T00:14:31.450 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.450 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.450 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:31.451 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.630363+0000", 2020-01-16T00:14:31.451 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2787999999999995e-05 2020-01-16T00:14:31.451 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.451 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.451 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:31.451 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.630436+0000", 2020-01-16T00:14:31.451 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013554490000000001 2020-01-16T00:14:31.452 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.452 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:31.452 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.452 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.452 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.453 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:5568 3.1 3:8de1d8a3:::smithi12313934-13:head [setxattr _header (56) in=7b,truncate 2168268] snapc 0=[] ondisk+write+known_if_redirected e66)", 2020-01-16T00:14:31.453 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:27.616979+0000", 2020-01-16T00:14:31.453 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 3.5916991399999998, 2020-01-16T00:14:31.453 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018308560000000001, 2020-01-16T00:14:31.453 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:31.453 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:31.453 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:31.454 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:31.454 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:31.454 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 5568 2020-01-16T00:14:31.454 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.454 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:31.454 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.454 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:31.455 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.616979+0000", 2020-01-16T00:14:31.455 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:31.455 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.455 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.455 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:31.455 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.616979+0000", 2020-01-16T00:14:31.455 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:31.456 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.456 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.456 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:31.456 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.616978+0000", 2020-01-16T00:14:31.456 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4280000000000001e-06 2020-01-16T00:14:31.456 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.456 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.456 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:31.457 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.616985+0000", 2020-01-16T00:14:31.457 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7630000000000003e-06 2020-01-16T00:14:31.457 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.457 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.457 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:31.457 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.616992+0000", 2020-01-16T00:14:31.457 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6639999999999997e-06 2020-01-16T00:14:31.457 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.458 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.458 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:31.458 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.616996+0000", 2020-01-16T00:14:31.458 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020064400000000001 2020-01-16T00:14:31.458 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.458 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.458 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.459 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.617197+0000", 2020-01-16T00:14:31.459 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000149071 2020-01-16T00:14:31.459 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.459 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.459 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:31.459 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.617346+0000", 2020-01-16T00:14:31.459 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010707899 2020-01-16T00:14:31.460 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.460 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.460 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.460 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.628054+0000", 2020-01-16T00:14:31.460 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013353 2020-01-16T00:14:31.460 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.461 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.461 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:31.461 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.628187+0000", 2020-01-16T00:14:31.461 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000184432 2020-01-16T00:14:31.461 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.461 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.461 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 3,4", 2020-01-16T00:14:31.462 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.628372+0000", 2020-01-16T00:14:31.462 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0022270200000000001 2020-01-16T00:14:31.462 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.462 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.462 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:31.462 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.630599+0000", 2020-01-16T00:14:31.462 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000115078 2020-01-16T00:14:31.462 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.463 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.463 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:31.463 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.630714+0000", 2020-01-16T00:14:31.463 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0043920080000000002 2020-01-16T00:14:31.463 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.463 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.463 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:31.463 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.635106+0000", 2020-01-16T00:14:31.464 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8524999999999998e-05 2020-01-16T00:14:31.464 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.464 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.464 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:31.464 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.635184+0000", 2020-01-16T00:14:31.464 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010336300000000001 2020-01-16T00:14:31.465 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.465 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.465 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:31.465 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.635288+0000", 2020-01-16T00:14:31.465 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018308560000000001 2020-01-16T00:14:31.465 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.465 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:31.465 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.466 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.466 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.466 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:5569 3.1 3:8de1d8a3:::smithi12313934-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e66)", 2020-01-16T00:14:31.466 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:27.617006+0000", 2020-01-16T00:14:31.466 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 3.5916726589999999, 2020-01-16T00:14:31.466 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018588490999999999, 2020-01-16T00:14:31.466 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:31.467 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:31.467 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:31.467 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:31.467 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:31.467 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 5569 2020-01-16T00:14:31.467 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.467 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:31.468 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.468 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:31.468 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.617006+0000", 2020-01-16T00:14:31.468 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:31.468 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.468 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.468 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:31.468 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.617006+0000", 2020-01-16T00:14:31.469 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:31.469 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.469 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.469 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:31.469 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.617005+0000", 2020-01-16T00:14:31.469 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8569999999999997e-06 2020-01-16T00:14:31.470 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.470 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.470 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:31.470 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.617012+0000", 2020-01-16T00:14:31.470 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.669e-06 2020-01-16T00:14:31.470 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.470 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.471 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:31.471 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.617019+0000", 2020-01-16T00:14:31.471 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.34e-06 2020-01-16T00:14:31.471 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.471 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.471 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:31.471 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.617023+0000", 2020-01-16T00:14:31.471 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035691299999999999 2020-01-16T00:14:31.472 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.472 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.472 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.472 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.617380+0000", 2020-01-16T00:14:31.472 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000137615 2020-01-16T00:14:31.472 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.472 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.472 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:31.473 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.617517+0000", 2020-01-16T00:14:31.473 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.01195509 2020-01-16T00:14:31.473 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.473 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.473 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.473 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.629473+0000", 2020-01-16T00:14:31.473 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012814200000000001 2020-01-16T00:14:31.474 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.474 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.474 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:31.474 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.629601+0000", 2020-01-16T00:14:31.474 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0057381439999999997 2020-01-16T00:14:31.474 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.475 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.475 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.475 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.635339+0000", 2020-01-16T00:14:31.475 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012998600000000001 2020-01-16T00:14:31.475 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.475 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.475 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:31.476 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.635469+0000", 2020-01-16T00:14:31.476 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012535000000000001 2020-01-16T00:14:31.476 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.476 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.476 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:31.476 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:27.635594+0000", 2020-01-16T00:14:31.476 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018588490999999999 2020-01-16T00:14:31.477 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.477 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:31.477 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.477 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.477 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:31.477 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:31.479 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-01-16T00:14:31.480 INFO:tasks.rados.rados.0.smithi123.stdout:1959: finishing write tid 1 to smithi12313934-19 2020-01-16T00:14:31.480 INFO:tasks.rados.rados.0.smithi123.stdout:1959: finishing write tid 2 to smithi12313934-19 2020-01-16T00:14:31.480 INFO:tasks.rados.rados.0.smithi123.stdout:1959: finishing write tid 4 to smithi12313934-19 2020-01-16T00:14:31.480 INFO:tasks.rados.rados.0.smithi123.stdout:1959: finishing write tid 5 to smithi12313934-19 2020-01-16T00:14:31.480 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 19 v 279 (ObjNum 742 snap 0 seq_num 742) dirty exists 2020-01-16T00:14:31.480 INFO:tasks.rados.rados.0.smithi123.stdout:1959: left oid 19 (ObjNum 742 snap 0 seq_num 742) 2020-01-16T00:14:31.481 INFO:tasks.rados.rados.0.smithi123.stdout:1957: done (15 left) 2020-01-16T00:14:31.481 INFO:tasks.rados.rados.0.smithi123.stdout:1959: done (14 left) 2020-01-16T00:14:31.481 INFO:tasks.rados.rados.0.smithi123.stdout:1974: read oid 6 snap -1 2020-01-16T00:14:31.481 INFO:tasks.rados.rados.0.smithi123.stdout:1974: expect (ObjNum 720 snap 0 seq_num 720) 2020-01-16T00:14:31.481 INFO:tasks.rados.rados.0.smithi123.stdout:1975: read oid 2 snap -1 2020-01-16T00:14:31.481 INFO:tasks.rados.rados.0.smithi123.stdout:1975: expect (ObjNum 667 snap 0 seq_num 667) 2020-01-16T00:14:31.482 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:31.482 INFO:tasks.rados.rados.0.smithi123.stdout:1961: expect (ObjNum 705 snap 0 seq_num 705) 2020-01-16T00:14:31.482 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:31.478+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:31.579 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:31.576+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:31.625 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:31.626 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:14:31.626 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:14:31.626 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:31.635 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-01-16T00:14:31.658 INFO:tasks.rados.rados.0.smithi123.stdout:1960: expect (ObjNum 698 snap 0 seq_num 698) 2020-01-16T00:14:31.680 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:31.678+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:31.729 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:31.729 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:14:31.730 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:14:31.730 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:14:31.730 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:31.738 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-01-16T00:14:31.761 INFO:tasks.rados.rados.0.smithi123.stdout:1962: finishing write tid 1 to smithi12313934-34 2020-01-16T00:14:31.761 INFO:tasks.rados.rados.0.smithi123.stdout:1962: finishing write tid 2 to smithi12313934-34 2020-01-16T00:14:31.761 INFO:tasks.rados.rados.0.smithi123.stdout:1962: finishing write tid 4 to smithi12313934-34 2020-01-16T00:14:31.761 INFO:tasks.rados.rados.0.smithi123.stdout:1962: finishing write tid 5 to smithi12313934-34 2020-01-16T00:14:31.762 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 34 v 99 (ObjNum 743 snap 0 seq_num 743) dirty exists 2020-01-16T00:14:31.762 INFO:tasks.rados.rados.0.smithi123.stdout:1962: left oid 34 (ObjNum 743 snap 0 seq_num 743) 2020-01-16T00:14:31.762 INFO:tasks.rados.rados.0.smithi123.stdout:1960: done (15 left) 2020-01-16T00:14:31.762 INFO:tasks.rados.rados.0.smithi123.stdout:1961: done (14 left) 2020-01-16T00:14:31.762 INFO:tasks.rados.rados.0.smithi123.stdout:1962: done (13 left) 2020-01-16T00:14:31.762 INFO:tasks.rados.rados.0.smithi123.stdout:1976: read oid 26 snap -1 2020-01-16T00:14:31.762 INFO:tasks.rados.rados.0.smithi123.stdout:1976: expect (ObjNum 727 snap 0 seq_num 727) 2020-01-16T00:14:31.762 INFO:tasks.rados.rados.0.smithi123.stdout:1977: write oid 27 current snap is 0 2020-01-16T00:14:31.763 INFO:tasks.rados.rados.0.smithi123.stdout:1977: seq_num 751 ranges {622057=714593,1913624=708433} 2020-01-16T00:14:31.777 INFO:tasks.rados.rados.0.smithi123.stdout:1977: writing smithi12313934-27 from 622057 to 1336650 tid 1 2020-01-16T00:14:31.781 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:31.776+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:31.786 INFO:tasks.rados.rados.0.smithi123.stdout:1977: writing smithi12313934-27 from 1913624 to 2622057 tid 2 2020-01-16T00:14:31.786 INFO:tasks.rados.rados.0.smithi123.stdout:1978: write oid 11 current snap is 0 2020-01-16T00:14:31.786 INFO:tasks.rados.rados.0.smithi123.stdout:1978: seq_num 752 ranges {638864=789842,1972106=456248} 2020-01-16T00:14:31.795 INFO:tasks.rados.rados.0.smithi123.stdout:1978: writing smithi12313934-11 from 638864 to 1428706 tid 1 2020-01-16T00:14:31.802 INFO:tasks.rados.rados.0.smithi123.stdout:1978: writing smithi12313934-11 from 1972106 to 2428354 tid 2 2020-01-16T00:14:31.802 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:31.802 INFO:tasks.rados.rados.0.smithi123.stdout:1963: finishing write tid 1 to smithi12313934-50 2020-01-16T00:14:31.802 INFO:tasks.rados.rados.0.smithi123.stdout:1963: finishing write tid 2 to smithi12313934-50 2020-01-16T00:14:31.802 INFO:tasks.rados.rados.0.smithi123.stdout:1963: finishing write tid 4 to smithi12313934-50 2020-01-16T00:14:31.802 INFO:tasks.rados.rados.0.smithi123.stdout:1963: finishing write tid 5 to smithi12313934-50 2020-01-16T00:14:31.802 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 50 v 254 (ObjNum 744 snap 0 seq_num 744) dirty exists 2020-01-16T00:14:31.803 INFO:tasks.rados.rados.0.smithi123.stdout:1963: left oid 50 (ObjNum 744 snap 0 seq_num 744) 2020-01-16T00:14:31.803 INFO:tasks.rados.rados.0.smithi123.stdout:1964: finishing write tid 1 to smithi12313934-21 2020-01-16T00:14:31.803 INFO:tasks.rados.rados.0.smithi123.stdout:1966: expect (ObjNum 729 snap 0 seq_num 729) 2020-01-16T00:14:31.877 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:31.877 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:14:31.877 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:14:31.878 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:14:31.878 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.878 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2226 3.7 3:e1be6d3b:::smithi12313934-24:head [setxattr _header (56) in=7b,truncate 2100908] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:14:31.878 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.980217+0000", 2020-01-16T00:14:31.878 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 85.891861746000004, 2020-01-16T00:14:31.878 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023113018999999999, 2020-01-16T00:14:31.878 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:31.879 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:31.879 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:31.879 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:31.879 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:31.879 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2226 2020-01-16T00:14:31.879 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.879 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:31.879 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.880 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:31.880 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980217+0000", 2020-01-16T00:14:31.880 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:31.880 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.880 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.880 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:31.880 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980217+0000", 2020-01-16T00:14:31.880 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:31.881 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.881 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.881 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:31.881 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980216+0000", 2020-01-16T00:14:31.881 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5049999999999998e-06 2020-01-16T00:14:31.881 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.881 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.882 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:31.882 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980222+0000", 2020-01-16T00:14:31.882 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5654999999999998e-05 2020-01-16T00:14:31.882 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.882 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.882 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:31.882 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980237+0000", 2020-01-16T00:14:31.883 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9740000000000004e-06 2020-01-16T00:14:31.883 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.883 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.883 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:31.883 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980241+0000", 2020-01-16T00:14:31.883 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002947776 2020-01-16T00:14:31.884 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.884 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.884 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.884 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.983189+0000", 2020-01-16T00:14:31.884 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0027099400000000001 2020-01-16T00:14:31.884 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.884 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.884 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:31.885 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985899+0000", 2020-01-16T00:14:31.885 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014275589999999999 2020-01-16T00:14:31.885 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.885 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.886 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.886 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.000175+0000", 2020-01-16T00:14:31.886 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011211 2020-01-16T00:14:31.886 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.886 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.886 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:31.888 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.000287+0000", 2020-01-16T00:14:31.888 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000229886 2020-01-16T00:14:31.888 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.889 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.889 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,7", 2020-01-16T00:14:31.889 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.000517+0000", 2020-01-16T00:14:31.889 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00095715399999999999 2020-01-16T00:14:31.889 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.889 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.889 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:31.890 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.001474+0000", 2020-01-16T00:14:31.890 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001341512 2020-01-16T00:14:31.890 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.890 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.890 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:31.890 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.002815+0000", 2020-01-16T00:14:31.891 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035789099999999999 2020-01-16T00:14:31.891 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.891 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.891 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:31.891 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003173+0000", 2020-01-16T00:14:31.891 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8571000000000005e-05 2020-01-16T00:14:31.891 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.891 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.892 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:31.892 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003242+0000", 2020-01-16T00:14:31.892 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8293000000000003e-05 2020-01-16T00:14:31.892 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.893 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.893 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:31.893 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003330+0000", 2020-01-16T00:14:31.893 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023113018999999999 2020-01-16T00:14:31.893 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.894 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:31.894 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.894 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.894 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.894 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2227 3.7 3:e1be6d3b:::smithi12313934-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e35)", 2020-01-16T00:14:31.894 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:14:31.894 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 85.891814492999998, 2020-01-16T00:14:31.894 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023303356000000001, 2020-01-16T00:14:31.895 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:31.895 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:31.895 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:31.895 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:31.895 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:31.895 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2227 2020-01-16T00:14:31.895 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.896 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:31.896 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.896 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:31.896 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:14:31.896 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:31.896 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.896 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.896 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:31.897 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:14:31.897 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:31.897 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.897 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.897 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:31.897 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:14:31.897 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3079999999999999e-06 2020-01-16T00:14:31.897 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.898 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.898 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:31.898 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980267+0000", 2020-01-16T00:14:31.898 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9290000000000001e-06 2020-01-16T00:14:31.898 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.898 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.898 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:31.898 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980274+0000", 2020-01-16T00:14:31.899 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7400000000000002e-06 2020-01-16T00:14:31.899 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.899 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.899 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:31.899 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980278+0000", 2020-01-16T00:14:31.899 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056624300000000004 2020-01-16T00:14:31.899 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.899 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.900 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.900 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985940+0000", 2020-01-16T00:14:31.900 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4261000000000002e-05 2020-01-16T00:14:31.900 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.900 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.900 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:31.900 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985985+0000", 2020-01-16T00:14:31.900 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015087676 2020-01-16T00:14:31.901 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.901 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.901 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.901 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.001072+0000", 2020-01-16T00:14:31.901 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011960600000000001 2020-01-16T00:14:31.901 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.901 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.901 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:31.902 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.001192+0000", 2020-01-16T00:14:31.902 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021718309999999999 2020-01-16T00:14:31.902 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.902 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.902 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.902 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003364+0000", 2020-01-16T00:14:31.902 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000102855 2020-01-16T00:14:31.903 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.903 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.903 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:31.903 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003467+0000", 2020-01-16T00:14:31.903 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010122499999999999 2020-01-16T00:14:31.903 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.904 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.904 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:31.904 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003568+0000", 2020-01-16T00:14:31.904 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023303356000000001 2020-01-16T00:14:31.904 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.904 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:31.904 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.904 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.905 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.905 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2325 3.7 3:e1be6d3b:::smithi12313934-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e36)", 2020-01-16T00:14:31.905 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.550877+0000", 2020-01-16T00:14:31.905 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 84.321202166000006, 2020-01-16T00:14:31.905 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018565154, 2020-01-16T00:14:31.906 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:31.906 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:31.906 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:31.906 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:31.906 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:31.906 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2325 2020-01-16T00:14:31.906 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.907 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:31.907 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.907 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:31.907 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550877+0000", 2020-01-16T00:14:31.907 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:31.907 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.907 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.908 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:31.908 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550877+0000", 2020-01-16T00:14:31.908 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:31.908 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.908 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.908 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:31.908 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550876+0000", 2020-01-16T00:14:31.908 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1080000000000003e-06 2020-01-16T00:14:31.909 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.909 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.909 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:31.909 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550880+0000", 2020-01-16T00:14:31.909 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3269999999999997e-06 2020-01-16T00:14:31.909 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.909 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.910 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:31.910 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550886+0000", 2020-01-16T00:14:31.910 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.386e-06 2020-01-16T00:14:31.910 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.910 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.910 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:31.910 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550891+0000", 2020-01-16T00:14:31.911 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048231410000000004 2020-01-16T00:14:31.911 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.911 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.911 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.911 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555714+0000", 2020-01-16T00:14:31.911 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0071999999999999e-05 2020-01-16T00:14:31.911 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.911 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.912 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:31.912 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555744+0000", 2020-01-16T00:14:31.912 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012329005000000001 2020-01-16T00:14:31.912 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.912 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.912 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.912 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568073+0000", 2020-01-16T00:14:31.912 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9087999999999997e-05 2020-01-16T00:14:31.913 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.913 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.913 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:31.913 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568172+0000", 2020-01-16T00:14:31.913 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010718220000000001 2020-01-16T00:14:31.913 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.914 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.914 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.914 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569244+0000", 2020-01-16T00:14:31.914 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8199e-05 2020-01-16T00:14:31.914 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.914 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.915 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:31.915 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569342+0000", 2020-01-16T00:14:31.915 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9708000000000001e-05 2020-01-16T00:14:31.915 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.915 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.915 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:31.915 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569442+0000", 2020-01-16T00:14:31.916 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018565154 2020-01-16T00:14:31.916 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.916 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:31.916 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.916 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.916 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.916 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2800 3.2 3:42b7f20f:::smithi12313934-42:head [write 1395221~583781] snapc 0=[] ondisk+write+known_if_redirected e41)", 2020-01-16T00:14:31.917 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:17.256182+0000", 2020-01-16T00:14:31.917 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 74.615896625999994, 2020-01-16T00:14:31.917 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018509650999999998, 2020-01-16T00:14:31.917 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:31.917 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:31.917 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:31.917 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:31.917 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:31.918 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2800 2020-01-16T00:14:31.918 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.918 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:31.918 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.918 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:31.918 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256182+0000", 2020-01-16T00:14:31.918 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:31.919 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.923 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.923 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:31.923 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256182+0000", 2020-01-16T00:14:31.923 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:31.923 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.923 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.923 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:31.924 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256180+0000", 2020-01-16T00:14:31.926 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020514799999999999 2020-01-16T00:14:31.926 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.926 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.926 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:31.927 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256385+0000", 2020-01-16T00:14:31.927 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0288e-05 2020-01-16T00:14:31.927 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.927 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.927 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:31.927 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256406+0000", 2020-01-16T00:14:31.928 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7160000000000003e-06 2020-01-16T00:14:31.928 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.928 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.928 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:31.928 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256413+0000", 2020-01-16T00:14:31.928 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5293999999999998e-05 2020-01-16T00:14:31.928 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.929 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.929 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.929 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256449+0000", 2020-01-16T00:14:31.929 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001297737 2020-01-16T00:14:31.929 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.929 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.930 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:31.930 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.257746+0000", 2020-01-16T00:14:31.930 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0079866529999999998 2020-01-16T00:14:31.930 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.930 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.930 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.931 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.265733+0000", 2020-01-16T00:14:31.931 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4028000000000004e-05 2020-01-16T00:14:31.931 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.931 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.931 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:31.931 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.265797+0000", 2020-01-16T00:14:31.931 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011338899999999999 2020-01-16T00:14:31.932 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.932 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.932 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:31.932 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.265910+0000", 2020-01-16T00:14:31.932 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003372496 2020-01-16T00:14:31.932 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.932 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.932 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:31.933 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.269283+0000", 2020-01-16T00:14:31.933 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0053107959999999996 2020-01-16T00:14:31.933 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.933 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.933 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:31.933 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.274594+0000", 2020-01-16T00:14:31.933 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8000999999999998e-05 2020-01-16T00:14:31.933 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.934 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.934 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:31.934 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.274632+0000", 2020-01-16T00:14:31.934 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0265e-05 2020-01-16T00:14:31.934 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.934 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.935 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:31.935 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.274692+0000", 2020-01-16T00:14:31.935 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018509650999999998 2020-01-16T00:14:31.935 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.935 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:31.935 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.935 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.936 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.936 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2973 3.c 3:3025546b:::smithi12313934-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e42)", 2020-01-16T00:14:31.936 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:21.839961+0000", 2020-01-16T00:14:31.936 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 70.032118025000003, 2020-01-16T00:14:31.936 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018691670000000001, 2020-01-16T00:14:31.936 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:31.936 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:31.936 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:31.937 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:31.937 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:31.937 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2973 2020-01-16T00:14:31.937 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.937 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:31.937 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.937 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:31.937 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839961+0000", 2020-01-16T00:14:31.938 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:31.938 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.938 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.938 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:31.938 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839961+0000", 2020-01-16T00:14:31.938 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:31.938 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.939 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.939 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:31.939 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839960+0000", 2020-01-16T00:14:31.939 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4060000000000001e-06 2020-01-16T00:14:31.939 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.939 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.939 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:31.939 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839964+0000", 2020-01-16T00:14:31.940 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5412000000000001e-05 2020-01-16T00:14:31.940 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.940 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.940 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:31.940 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839979+0000", 2020-01-16T00:14:31.940 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.444e-06 2020-01-16T00:14:31.940 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.941 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.941 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:31.941 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839983+0000", 2020-01-16T00:14:31.941 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000430028 2020-01-16T00:14:31.941 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.941 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.941 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.941 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.840413+0000", 2020-01-16T00:14:31.942 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9270999999999997e-05 2020-01-16T00:14:31.942 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.942 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.942 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:31.942 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.840452+0000", 2020-01-16T00:14:31.942 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014853975 2020-01-16T00:14:31.942 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.943 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.943 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.943 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.855306+0000", 2020-01-16T00:14:31.943 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011098400000000001 2020-01-16T00:14:31.943 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.944 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.944 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:31.944 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.855417+0000", 2020-01-16T00:14:31.944 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029108060000000002 2020-01-16T00:14:31.945 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.945 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.945 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.945 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858328+0000", 2020-01-16T00:14:31.945 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000144697 2020-01-16T00:14:31.945 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.946 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.946 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:31.946 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858472+0000", 2020-01-16T00:14:31.946 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018011899999999999 2020-01-16T00:14:31.946 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.946 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.946 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:31.947 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858653+0000", 2020-01-16T00:14:31.947 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018691670000000001 2020-01-16T00:14:31.947 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.947 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:31.947 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.947 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.947 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.948 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3296 3.2 3:42b7f20f:::smithi12313934-42:head [setxattr _header (56) in=7b,truncate 2739607] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:14:31.948 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:30.744532+0000", 2020-01-16T00:14:31.948 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 61.127547016000001, 2020-01-16T00:14:31.948 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018472374, 2020-01-16T00:14:31.948 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:31.948 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:31.948 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:31.948 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:31.949 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:31.949 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3296 2020-01-16T00:14:31.949 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.949 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:31.949 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.949 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:31.949 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744532+0000", 2020-01-16T00:14:31.950 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:31.950 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.950 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.950 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:31.950 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744532+0000", 2020-01-16T00:14:31.950 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:31.950 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.950 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.951 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:31.951 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744531+0000", 2020-01-16T00:14:31.951 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7059999999999993e-06 2020-01-16T00:14:31.951 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.951 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.951 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:31.951 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744540+0000", 2020-01-16T00:14:31.952 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8589999999999998e-06 2020-01-16T00:14:31.952 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.952 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.952 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:31.952 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744544+0000", 2020-01-16T00:14:31.952 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7609999999999999e-06 2020-01-16T00:14:31.952 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.952 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:31.953 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744546+0000", 2020-01-16T00:14:31.953 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039933670000000003 2020-01-16T00:14:31.953 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.953 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.953 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748540+0000", 2020-01-16T00:14:31.953 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000110281 2020-01-16T00:14:31.954 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.954 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.954 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:31.954 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748650+0000", 2020-01-16T00:14:31.954 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.01160457 2020-01-16T00:14:31.954 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.954 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.955 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.955 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.760255+0000", 2020-01-16T00:14:31.955 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000124714 2020-01-16T00:14:31.955 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.955 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.955 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:31.956 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.760379+0000", 2020-01-16T00:14:31.956 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022099700000000001 2020-01-16T00:14:31.956 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.956 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.956 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:31.956 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.760600+0000", 2020-01-16T00:14:31.956 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00057843300000000005 2020-01-16T00:14:31.956 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.957 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.957 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:31.957 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.761179+0000", 2020-01-16T00:14:31.957 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00161477 2020-01-16T00:14:31.957 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.957 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.957 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:31.958 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.762794+0000", 2020-01-16T00:14:31.958 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4408999999999999e-05 2020-01-16T00:14:31.958 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.958 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.958 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:31.958 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.762858+0000", 2020-01-16T00:14:31.958 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014634199999999999 2020-01-16T00:14:31.958 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.959 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.959 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:31.959 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.763004+0000", 2020-01-16T00:14:31.959 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018472374 2020-01-16T00:14:31.959 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.959 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:31.959 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.959 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.960 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.960 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3297 3.2 3:42b7f20f:::smithi12313934-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e45)", 2020-01-16T00:14:31.961 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:30.744551+0000", 2020-01-16T00:14:31.961 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 61.127528046000002, 2020-01-16T00:14:31.961 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018660209000000001, 2020-01-16T00:14:31.961 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:31.961 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:31.961 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:31.961 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:31.962 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:31.962 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3297 2020-01-16T00:14:31.962 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.962 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:31.962 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.962 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:31.962 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744551+0000", 2020-01-16T00:14:31.962 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:31.963 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.963 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.963 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:31.963 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744551+0000", 2020-01-16T00:14:31.963 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:31.963 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.963 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.964 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:31.964 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744550+0000", 2020-01-16T00:14:31.964 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5279999999999999e-06 2020-01-16T00:14:31.964 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.964 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.964 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:31.965 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744554+0000", 2020-01-16T00:14:31.965 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8420000000000001e-06 2020-01-16T00:14:31.965 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.965 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.965 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:31.966 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744559+0000", 2020-01-16T00:14:31.966 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.965e-06 2020-01-16T00:14:31.966 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.966 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.967 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:31.967 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744562+0000", 2020-01-16T00:14:31.967 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041215380000000001 2020-01-16T00:14:31.967 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.967 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.968 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.968 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748684+0000", 2020-01-16T00:14:31.968 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8165999999999997e-05 2020-01-16T00:14:31.968 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.968 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.969 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:31.969 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748762+0000", 2020-01-16T00:14:31.969 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012136152000000001 2020-01-16T00:14:31.969 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.969 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.970 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.970 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.760898+0000", 2020-01-16T00:14:31.970 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000114833 2020-01-16T00:14:31.971 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.971 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.971 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:31.971 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.761013+0000", 2020-01-16T00:14:31.971 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002017015 2020-01-16T00:14:31.972 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.972 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.972 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.972 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.763030+0000", 2020-01-16T00:14:31.973 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5741000000000006e-05 2020-01-16T00:14:31.973 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.973 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.973 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:31.973 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.763096+0000", 2020-01-16T00:14:31.973 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011519900000000001 2020-01-16T00:14:31.974 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.974 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.974 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:31.974 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.763211+0000", 2020-01-16T00:14:31.975 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018660209000000001 2020-01-16T00:14:31.975 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.975 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:31.975 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.976 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.977 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.977 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3810 3.7 3:efab040c:::smithi12313934-30:head [write 1837406~542493] snapc 0=[] ondisk+write+known_if_redirected e50)", 2020-01-16T00:14:31.977 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:43.768525+0000", 2020-01-16T00:14:31.977 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 48.103553873000003, 2020-01-16T00:14:31.977 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018385839000000001, 2020-01-16T00:14:31.977 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:31.977 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:31.978 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:31.978 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:31.978 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:31.978 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3810 2020-01-16T00:14:31.978 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.978 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:31.978 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.979 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:31.979 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768525+0000", 2020-01-16T00:14:31.979 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:31.979 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.979 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.979 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:31.979 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768525+0000", 2020-01-16T00:14:31.979 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:31.980 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.980 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.980 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:31.980 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768523+0000", 2020-01-16T00:14:31.980 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000149512 2020-01-16T00:14:31.980 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.980 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.980 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:31.981 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768673+0000", 2020-01-16T00:14:31.981 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0732e-05 2020-01-16T00:14:31.981 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.981 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.981 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:31.981 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768684+0000", 2020-01-16T00:14:31.981 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3719999999999998e-06 2020-01-16T00:14:31.982 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.982 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.982 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:31.982 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768688+0000", 2020-01-16T00:14:31.982 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6513999999999994e-05 2020-01-16T00:14:31.982 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.982 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.983 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.983 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768754+0000", 2020-01-16T00:14:31.983 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00075867800000000004 2020-01-16T00:14:31.983 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.983 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.983 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:31.983 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.769513+0000", 2020-01-16T00:14:31.983 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012395692 2020-01-16T00:14:31.984 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.984 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.984 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.984 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.781909+0000", 2020-01-16T00:14:31.984 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118372 2020-01-16T00:14:31.984 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.984 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.985 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:31.985 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.782027+0000", 2020-01-16T00:14:31.985 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019909699999999999 2020-01-16T00:14:31.985 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.985 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.985 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,7", 2020-01-16T00:14:31.985 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.782226+0000", 2020-01-16T00:14:31.986 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0038778369999999999 2020-01-16T00:14:31.986 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.986 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.986 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:31.986 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786104+0000", 2020-01-16T00:14:31.986 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022391199999999999 2020-01-16T00:14:31.987 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.987 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.987 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:31.987 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786328+0000", 2020-01-16T00:14:31.987 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00048851299999999999 2020-01-16T00:14:31.987 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.988 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.988 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:31.988 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786817+0000", 2020-01-16T00:14:31.988 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3597999999999998e-05 2020-01-16T00:14:31.988 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.988 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.988 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:31.988 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786860+0000", 2020-01-16T00:14:31.989 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0741000000000001e-05 2020-01-16T00:14:31.989 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.989 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.989 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:31.989 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786911+0000", 2020-01-16T00:14:31.989 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018385839000000001 2020-01-16T00:14:31.989 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.990 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:31.990 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:31.990 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.990 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.990 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3863 3.2 3:4f8fe446:::smithi12313934-48:head [setxattr _header (56) in=7b,truncate 2151395] snapc 0=[] ondisk+write+known_if_redirected e50)", 2020-01-16T00:14:31.990 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:44.489284+0000", 2020-01-16T00:14:31.990 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 47.382795401000003, 2020-01-16T00:14:31.991 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.026323742000000001, 2020-01-16T00:14:31.991 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:31.991 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:31.991 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:31.991 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:31.991 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:31.992 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3863 2020-01-16T00:14:31.992 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.992 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:31.992 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.992 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:31.992 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489284+0000", 2020-01-16T00:14:31.992 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:31.992 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.993 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.993 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:31.993 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489284+0000", 2020-01-16T00:14:31.993 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:31.993 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.993 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.993 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:31.994 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489282+0000", 2020-01-16T00:14:31.994 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6660000000000002e-06 2020-01-16T00:14:31.994 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.994 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.994 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:31.994 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489289+0000", 2020-01-16T00:14:31.994 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3069999999999999e-06 2020-01-16T00:14:31.994 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.995 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.995 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:31.995 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489295+0000", 2020-01-16T00:14:31.995 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6090000000000003e-06 2020-01-16T00:14:31.995 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.995 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.995 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:31.996 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489300+0000", 2020-01-16T00:14:31.996 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002242464 2020-01-16T00:14:31.996 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.996 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.996 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.996 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491542+0000", 2020-01-16T00:14:31.996 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00059546200000000005 2020-01-16T00:14:31.997 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.997 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.997 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:31.997 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.492138+0000", 2020-01-16T00:14:31.997 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018547999999999999 2020-01-16T00:14:31.997 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.998 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.998 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:31.998 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.510686+0000", 2020-01-16T00:14:31.998 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018590100000000001 2020-01-16T00:14:31.998 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.998 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.998 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:31.998 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.510872+0000", 2020-01-16T00:14:31.999 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035115299999999999 2020-01-16T00:14:31.999 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:31.999 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:31.999 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:31.999 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.511223+0000", 2020-01-16T00:14:31.999 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003765493 2020-01-16T00:14:31.999 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.000 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.000 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:32.000 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.514988+0000", 2020-01-16T00:14:32.000 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000432022 2020-01-16T00:14:32.000 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.000 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.001 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:32.001 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515421+0000", 2020-01-16T00:14:32.001 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2874000000000002e-05 2020-01-16T00:14:32.001 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.002 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.002 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:32.002 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515473+0000", 2020-01-16T00:14:32.002 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013385 2020-01-16T00:14:32.002 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.002 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.003 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:32.003 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515607+0000", 2020-01-16T00:14:32.003 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.026323742000000001 2020-01-16T00:14:32.003 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:32.003 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:32.003 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:32.003 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.004 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.004 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3864 3.2 3:4f8fe446:::smithi12313934-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e50)", 2020-01-16T00:14:32.004 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:44.491131+0000", 2020-01-16T00:14:32.004 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 47.380948003, 2020-01-16T00:14:32.004 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024813806000000001, 2020-01-16T00:14:32.004 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:32.004 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:32.005 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:32.005 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:32.005 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:32.005 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3864 2020-01-16T00:14:32.005 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.005 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:32.005 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.006 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:32.006 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491131+0000", 2020-01-16T00:14:32.006 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:32.006 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.006 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.006 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:32.007 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491131+0000", 2020-01-16T00:14:32.007 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:32.007 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.007 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.007 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:32.007 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491128+0000", 2020-01-16T00:14:32.007 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2053e-05 2020-01-16T00:14:32.007 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.008 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.008 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:32.008 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491140+0000", 2020-01-16T00:14:32.008 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5869999999999999e-05 2020-01-16T00:14:32.008 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.008 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.008 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:32.009 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491156+0000", 2020-01-16T00:14:32.009 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2571e-05 2020-01-16T00:14:32.009 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.009 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.009 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:32.009 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491168+0000", 2020-01-16T00:14:32.009 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001031045 2020-01-16T00:14:32.010 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.010 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.010 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:32.010 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.492199+0000", 2020-01-16T00:14:32.010 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000170865 2020-01-16T00:14:32.011 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.011 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.011 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:32.011 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.492370+0000", 2020-01-16T00:14:32.012 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020180638000000001 2020-01-16T00:14:32.012 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.013 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.013 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:32.013 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.512551+0000", 2020-01-16T00:14:32.013 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012478599999999999 2020-01-16T00:14:32.014 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.014 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.014 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:32.014 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.512676+0000", 2020-01-16T00:14:32.015 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029628660000000002 2020-01-16T00:14:32.015 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.015 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.015 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:32.016 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515639+0000", 2020-01-16T00:14:32.016 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012250799999999999 2020-01-16T00:14:32.016 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.016 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.016 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:32.017 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515761+0000", 2020-01-16T00:14:32.017 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018356399999999999 2020-01-16T00:14:32.017 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.017 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.018 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:32.018 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515945+0000", 2020-01-16T00:14:32.018 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024813806000000001 2020-01-16T00:14:32.018 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:32.019 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:32.019 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:32.019 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.019 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.019 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3874 3.c 3:378f3722:::smithi12313934-50:head [write 568202~701253,stat] snapc 0=[] ondisk+write+known_if_redirected e50)", 2020-01-16T00:14:32.020 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:44.497041+0000", 2020-01-16T00:14:32.020 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 47.375038371999999, 2020-01-16T00:14:32.020 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019602525999999999, 2020-01-16T00:14:32.020 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:32.021 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:32.021 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:32.021 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:32.021 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:32.021 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3874 2020-01-16T00:14:32.022 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.022 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:32.022 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.022 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:32.023 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497041+0000", 2020-01-16T00:14:32.023 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:32.023 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.023 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.024 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:32.024 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497041+0000", 2020-01-16T00:14:32.024 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999962 2020-01-16T00:14:32.024 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.025 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.025 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:32.025 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497037+0000", 2020-01-16T00:14:32.025 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00027168899999999998 2020-01-16T00:14:32.026 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.026 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.026 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:32.026 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497308+0000", 2020-01-16T00:14:32.027 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4202999999999999e-05 2020-01-16T00:14:32.027 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.027 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.028 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:32.028 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497323+0000", 2020-01-16T00:14:32.028 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0280000000000003e-06 2020-01-16T00:14:32.028 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.028 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.029 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:32.029 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497330+0000", 2020-01-16T00:14:32.029 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010681352 2020-01-16T00:14:32.029 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.030 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.030 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:32.030 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.508011+0000", 2020-01-16T00:14:32.030 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5886999999999999e-05 2020-01-16T00:14:32.031 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.031 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.031 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:32.031 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.508097+0000", 2020-01-16T00:14:32.032 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023353600000000001 2020-01-16T00:14:32.032 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.032 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.032 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:32.032 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.508331+0000", 2020-01-16T00:14:32.033 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0075218339999999998 2020-01-16T00:14:32.033 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.033 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.033 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:32.034 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515852+0000", 2020-01-16T00:14:32.034 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00064178599999999998 2020-01-16T00:14:32.034 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.034 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.035 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:32.035 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.516494+0000", 2020-01-16T00:14:32.035 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6712999999999999e-05 2020-01-16T00:14:32.035 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.036 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.036 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:32.036 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.516531+0000", 2020-01-16T00:14:32.036 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112225 2020-01-16T00:14:32.036 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.037 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.037 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:32.037 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.516643+0000", 2020-01-16T00:14:32.037 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019602525999999999 2020-01-16T00:14:32.039 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:32.039 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:32.039 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:32.039 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.039 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.039 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4655 3.c 3:378f3722:::smithi12313934-50:head [write 1684786~315379] snapc 0=[] ondisk+write+known_if_redirected e57)", 2020-01-16T00:14:32.040 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:04.403811+0000", 2020-01-16T00:14:32.040 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 27.468267887, 2020-01-16T00:14:32.040 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022055851000000001, 2020-01-16T00:14:32.040 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:32.040 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:32.040 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:32.040 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:32.041 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:32.041 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4655 2020-01-16T00:14:32.041 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.041 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:32.041 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.041 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:32.041 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.403811+0000", 2020-01-16T00:14:32.042 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:32.042 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.042 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.042 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:32.042 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.403811+0000", 2020-01-16T00:14:32.042 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:32.042 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.043 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.043 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:32.043 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.403809+0000", 2020-01-16T00:14:32.043 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012429100000000001 2020-01-16T00:14:32.043 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.044 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.044 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:32.044 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.403934+0000", 2020-01-16T00:14:32.044 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3743e-05 2020-01-16T00:14:32.044 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.044 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.044 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:32.045 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.403947+0000", 2020-01-16T00:14:32.045 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.267e-06 2020-01-16T00:14:32.045 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.045 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.045 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:32.045 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.403955+0000", 2020-01-16T00:14:32.045 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1935999999999999e-05 2020-01-16T00:14:32.046 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.046 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.046 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:32.046 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404006+0000", 2020-01-16T00:14:32.046 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0060147769999999998 2020-01-16T00:14:32.046 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.046 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.046 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:32.047 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.410021+0000", 2020-01-16T00:14:32.047 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012889367000000001 2020-01-16T00:14:32.047 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.047 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.047 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:32.047 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.422911+0000", 2020-01-16T00:14:32.047 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012536700000000001 2020-01-16T00:14:32.048 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.048 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.048 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:32.048 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.423036+0000", 2020-01-16T00:14:32.048 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021238999999999999 2020-01-16T00:14:32.048 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.048 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.048 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:32.049 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.423248+0000", 2020-01-16T00:14:32.049 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0024175289999999999 2020-01-16T00:14:32.049 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.049 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.049 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:32.049 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.425666+0000", 2020-01-16T00:14:32.049 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7323e-05 2020-01-16T00:14:32.050 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.050 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.051 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:32.051 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.425723+0000", 2020-01-16T00:14:32.051 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0198999999999997e-05 2020-01-16T00:14:32.051 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.051 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.051 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:32.051 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.425783+0000", 2020-01-16T00:14:32.052 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3456999999999997e-05 2020-01-16T00:14:32.052 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.052 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.052 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:32.052 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.425867+0000", 2020-01-16T00:14:32.052 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022055851000000001 2020-01-16T00:14:32.052 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:32.052 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:32.053 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:32.053 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.053 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.053 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4656 3.c 3:378f3722:::smithi12313934-50:head [setxattr _header (56) in=7b,truncate 2000165] snapc 0=[] ondisk+write+known_if_redirected e57)", 2020-01-16T00:14:32.053 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:04.404295+0000", 2020-01-16T00:14:32.053 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 27.467783747999999, 2020-01-16T00:14:32.053 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021950859, 2020-01-16T00:14:32.054 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:32.054 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:32.054 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:32.054 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:32.054 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:32.054 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4656 2020-01-16T00:14:32.054 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.054 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:32.055 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.055 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:32.055 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404295+0000", 2020-01-16T00:14:32.055 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:32.055 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.055 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.055 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:32.055 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404295+0000", 2020-01-16T00:14:32.056 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:32.056 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.056 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.056 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:32.056 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404294+0000", 2020-01-16T00:14:32.056 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6139999999999997e-06 2020-01-16T00:14:32.056 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.057 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.057 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:32.057 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404301+0000", 2020-01-16T00:14:32.057 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1420000000000001e-05 2020-01-16T00:14:32.057 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.057 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.057 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:32.057 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404312+0000", 2020-01-16T00:14:32.058 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4840000000000003e-06 2020-01-16T00:14:32.058 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.058 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.058 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:32.058 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404318+0000", 2020-01-16T00:14:32.058 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0058040970000000002 2020-01-16T00:14:32.058 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.058 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.059 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:32.059 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.410122+0000", 2020-01-16T00:14:32.059 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000168297 2020-01-16T00:14:32.059 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.059 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.059 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:32.059 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.410290+0000", 2020-01-16T00:14:32.059 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013892204999999999 2020-01-16T00:14:32.060 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.060 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.060 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:32.060 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.424182+0000", 2020-01-16T00:14:32.060 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6482000000000002e-05 2020-01-16T00:14:32.060 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.060 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.060 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:32.061 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.424279+0000", 2020-01-16T00:14:32.061 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000130025 2020-01-16T00:14:32.061 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.061 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.061 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:32.061 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.424409+0000", 2020-01-16T00:14:32.061 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00151793 2020-01-16T00:14:32.062 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.062 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.062 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:32.062 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.425927+0000", 2020-01-16T00:14:32.062 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020657800000000001 2020-01-16T00:14:32.062 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.062 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.062 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:32.063 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.426133+0000", 2020-01-16T00:14:32.063 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3606999999999999e-05 2020-01-16T00:14:32.063 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.063 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.063 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:32.063 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.426167+0000", 2020-01-16T00:14:32.063 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9117999999999997e-05 2020-01-16T00:14:32.063 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.064 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.064 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:32.064 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.426246+0000", 2020-01-16T00:14:32.064 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021950859 2020-01-16T00:14:32.064 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:32.064 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:32.064 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:32.064 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.065 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.065 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4657 3.c 3:378f3722:::smithi12313934-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e57)", 2020-01-16T00:14:32.065 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:04.404326+0000", 2020-01-16T00:14:32.065 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 27.467752601000001, 2020-01-16T00:14:32.065 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022101092999999999, 2020-01-16T00:14:32.065 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:32.065 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:32.065 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:32.066 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:32.066 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:32.066 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4657 2020-01-16T00:14:32.066 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.066 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:32.066 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.066 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:32.066 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404326+0000", 2020-01-16T00:14:32.067 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:32.067 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.067 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.067 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:32.067 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404326+0000", 2020-01-16T00:14:32.067 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:32.067 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.067 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.068 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:32.068 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404326+0000", 2020-01-16T00:14:32.068 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.771e-06 2020-01-16T00:14:32.068 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.068 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.068 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:32.068 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404329+0000", 2020-01-16T00:14:32.068 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9909999999999999e-06 2020-01-16T00:14:32.069 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.069 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.069 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:32.069 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404335+0000", 2020-01-16T00:14:32.069 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8870000000000002e-06 2020-01-16T00:14:32.069 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.069 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.070 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:32.070 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404339+0000", 2020-01-16T00:14:32.070 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0066816460000000003 2020-01-16T00:14:32.070 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.070 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.070 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:32.070 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.411021+0000", 2020-01-16T00:14:32.070 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3233000000000007e-05 2020-01-16T00:14:32.071 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.071 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.071 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:32.071 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.411114+0000", 2020-01-16T00:14:32.071 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014244643 2020-01-16T00:14:32.071 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.071 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.071 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:32.072 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.425359+0000", 2020-01-16T00:14:32.072 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012771200000000001 2020-01-16T00:14:32.072 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.072 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.072 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:32.072 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.425487+0000", 2020-01-16T00:14:32.072 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00078975699999999998 2020-01-16T00:14:32.072 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.073 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.073 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:32.073 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.426276+0000", 2020-01-16T00:14:32.073 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8829999999999997e-05 2020-01-16T00:14:32.073 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.073 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.073 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:32.073 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.426335+0000", 2020-01-16T00:14:32.074 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2219999999999995e-05 2020-01-16T00:14:32.074 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.074 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.074 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:32.074 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.426427+0000", 2020-01-16T00:14:32.074 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022101092999999999 2020-01-16T00:14:32.074 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:32.075 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:32.075 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:32.075 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.075 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.075 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4690 3.2 3:4f8fe446:::smithi12313934-48:head [setxattr _header (56) in=7b,truncate 2067393] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-01-16T00:14:32.075 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:06.996624+0000", 2020-01-16T00:14:32.075 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 24.875455367000001, 2020-01-16T00:14:32.075 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021459626999999998, 2020-01-16T00:14:32.076 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:32.076 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:32.076 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:32.076 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:32.076 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:32.076 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4690 2020-01-16T00:14:32.076 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.076 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:32.077 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.077 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:32.077 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996624+0000", 2020-01-16T00:14:32.077 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:32.077 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.077 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.077 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:32.078 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996624+0000", 2020-01-16T00:14:32.078 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:32.078 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.078 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.078 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:32.078 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996623+0000", 2020-01-16T00:14:32.078 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.549e-06 2020-01-16T00:14:32.079 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.079 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.079 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:32.079 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996627+0000", 2020-01-16T00:14:32.079 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4501999999999999e-05 2020-01-16T00:14:32.079 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.079 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.079 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:32.080 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996642+0000", 2020-01-16T00:14:32.080 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1139999999999997e-06 2020-01-16T00:14:32.080 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.080 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.080 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:32.080 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996647+0000", 2020-01-16T00:14:32.080 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001305489 2020-01-16T00:14:32.080 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.081 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.082 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:32.082 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.997952+0000", 2020-01-16T00:14:32.082 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036299430000000001 2020-01-16T00:14:32.082 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.082 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.082 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:32.083 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.001582+0000", 2020-01-16T00:14:32.083 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011633755000000001 2020-01-16T00:14:32.083 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.083 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.083 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:32.083 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.013216+0000", 2020-01-16T00:14:32.083 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021714 2020-01-16T00:14:32.083 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.084 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.084 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:32.084 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.013433+0000", 2020-01-16T00:14:32.084 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00050686199999999996 2020-01-16T00:14:32.084 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.084 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.084 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:32.084 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.013940+0000", 2020-01-16T00:14:32.085 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039060330000000002 2020-01-16T00:14:32.085 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.085 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.085 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:32.085 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.017846+0000", 2020-01-16T00:14:32.085 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7609e-05 2020-01-16T00:14:32.085 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.085 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.086 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:32.086 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.017864+0000", 2020-01-16T00:14:32.086 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3856999999999994e-05 2020-01-16T00:14:32.086 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.086 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.086 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:32.087 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.017937+0000", 2020-01-16T00:14:32.087 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014570800000000001 2020-01-16T00:14:32.087 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.087 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.087 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:32.087 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.018083+0000", 2020-01-16T00:14:32.087 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021459626999999998 2020-01-16T00:14:32.087 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:32.088 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:32.088 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:32.088 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.088 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.088 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4691 3.2 3:4f8fe446:::smithi12313934-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e58)", 2020-01-16T00:14:32.088 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:06.996654+0000", 2020-01-16T00:14:32.088 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 24.875425262, 2020-01-16T00:14:32.088 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021903913000000001, 2020-01-16T00:14:32.089 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:32.089 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:32.089 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:32.089 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:32.089 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:32.089 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4691 2020-01-16T00:14:32.089 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.090 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:32.090 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.090 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:32.090 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996654+0000", 2020-01-16T00:14:32.090 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:32.090 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.090 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.090 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:32.091 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996654+0000", 2020-01-16T00:14:32.091 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:32.091 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.091 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.091 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:32.091 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996653+0000", 2020-01-16T00:14:32.092 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0939999999999999e-06 2020-01-16T00:14:32.092 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.092 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.092 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:32.092 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996656+0000", 2020-01-16T00:14:32.092 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7609999999999996e-06 2020-01-16T00:14:32.092 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.093 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.093 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:32.093 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996665+0000", 2020-01-16T00:14:32.093 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3589999999999999e-06 2020-01-16T00:14:32.093 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.093 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.093 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:32.094 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996668+0000", 2020-01-16T00:14:32.094 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0049635909999999998 2020-01-16T00:14:32.094 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.094 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.094 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:32.094 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.001632+0000", 2020-01-16T00:14:32.094 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9422000000000001e-05 2020-01-16T00:14:32.095 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.095 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.095 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:32.095 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.001691+0000", 2020-01-16T00:14:32.095 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013157973 2020-01-16T00:14:32.095 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.095 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.095 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:32.096 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.014849+0000", 2020-01-16T00:14:32.096 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00027545199999999998 2020-01-16T00:14:32.096 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.096 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.096 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:32.096 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.015125+0000", 2020-01-16T00:14:32.096 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030160159999999998 2020-01-16T00:14:32.096 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.098 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.098 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:32.098 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.018141+0000", 2020-01-16T00:14:32.098 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015583 2020-01-16T00:14:32.098 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.098 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.099 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:32.099 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.018297+0000", 2020-01-16T00:14:32.099 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00026096000000000002 2020-01-16T00:14:32.099 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.099 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.099 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:32.099 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.018558+0000", 2020-01-16T00:14:32.099 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021903913000000001 2020-01-16T00:14:32.099 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:32.100 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:32.100 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:32.100 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.100 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.100 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4703 3.2 3:42b7f20f:::smithi12313934-42:head [write 1635678~465329] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-01-16T00:14:32.100 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.020171+0000", 2020-01-16T00:14:32.100 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 24.851908053999999, 2020-01-16T00:14:32.100 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020016459, 2020-01-16T00:14:32.101 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:32.101 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:32.101 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:32.101 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:32.101 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:32.101 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4703 2020-01-16T00:14:32.101 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.101 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:32.102 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.102 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:32.102 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020171+0000", 2020-01-16T00:14:32.102 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:32.102 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.102 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.103 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:32.103 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020171+0000", 2020-01-16T00:14:32.103 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:32.103 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.103 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.103 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:32.103 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020169+0000", 2020-01-16T00:14:32.103 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013985399999999999 2020-01-16T00:14:32.104 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.104 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.104 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:32.104 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020309+0000", 2020-01-16T00:14:32.104 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9029999999999999e-06 2020-01-16T00:14:32.104 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.104 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.104 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:32.104 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020319+0000", 2020-01-16T00:14:32.105 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9589999999999997e-06 2020-01-16T00:14:32.105 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.105 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.105 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:32.105 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020323+0000", 2020-01-16T00:14:32.105 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8680000000000002e-05 2020-01-16T00:14:32.105 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.106 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.106 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:32.106 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020362+0000", 2020-01-16T00:14:32.106 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029188460000000001 2020-01-16T00:14:32.106 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.106 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.106 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:32.106 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023281+0000", 2020-01-16T00:14:32.107 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011810522 2020-01-16T00:14:32.107 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.107 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.107 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:32.107 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.035091+0000", 2020-01-16T00:14:32.107 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000126148 2020-01-16T00:14:32.107 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.107 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.108 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:32.108 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.035217+0000", 2020-01-16T00:14:32.108 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019636799999999999 2020-01-16T00:14:32.108 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.108 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.108 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:32.108 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.035414+0000", 2020-01-16T00:14:32.108 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003550919 2020-01-16T00:14:32.108 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.109 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.109 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:32.109 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.038965+0000", 2020-01-16T00:14:32.109 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001007441 2020-01-16T00:14:32.109 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.109 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.109 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:32.109 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.039972+0000", 2020-01-16T00:14:32.110 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1954999999999998e-05 2020-01-16T00:14:32.110 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.110 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.110 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:32.110 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.040044+0000", 2020-01-16T00:14:32.110 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014325899999999999 2020-01-16T00:14:32.110 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.111 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.111 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:32.111 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.040187+0000", 2020-01-16T00:14:32.111 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020016459 2020-01-16T00:14:32.111 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:32.111 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:32.111 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:32.111 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.112 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.112 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4704 3.2 3:42b7f20f:::smithi12313934-42:head [setxattr _header (56) in=7b,truncate 2101007] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-01-16T00:14:32.112 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.020335+0000", 2020-01-16T00:14:32.112 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 24.851744242999999, 2020-01-16T00:14:32.113 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023082354999999999, 2020-01-16T00:14:32.113 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:32.113 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:32.113 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:32.114 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:32.114 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:32.114 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4704 2020-01-16T00:14:32.114 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.114 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:32.115 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.115 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:32.115 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020335+0000", 2020-01-16T00:14:32.116 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:32.116 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.116 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.116 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:32.117 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020335+0000", 2020-01-16T00:14:32.117 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:32.117 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.117 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.117 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:32.117 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020334+0000", 2020-01-16T00:14:32.118 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7780000000000002e-06 2020-01-16T00:14:32.118 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.118 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.118 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:32.118 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020341+0000", 2020-01-16T00:14:32.119 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2039999999999996e-06 2020-01-16T00:14:32.119 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.119 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.119 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:32.119 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020346+0000", 2020-01-16T00:14:32.119 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8800000000000008e-06 2020-01-16T00:14:32.120 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.120 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.120 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:32.120 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020354+0000", 2020-01-16T00:14:32.120 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029530490000000001 2020-01-16T00:14:32.121 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.121 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.121 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:32.121 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023307+0000", 2020-01-16T00:14:32.122 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8681000000000003e-05 2020-01-16T00:14:32.122 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.122 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.122 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:32.122 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023365+0000", 2020-01-16T00:14:32.122 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013386927999999999 2020-01-16T00:14:32.123 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.123 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.123 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:32.123 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.036752+0000", 2020-01-16T00:14:32.123 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8906000000000007e-05 2020-01-16T00:14:32.123 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.123 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.123 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:32.124 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.036831+0000", 2020-01-16T00:14:32.124 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000142922 2020-01-16T00:14:32.124 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.124 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.124 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:32.124 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.036974+0000", 2020-01-16T00:14:32.124 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0038357280000000001 2020-01-16T00:14:32.124 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.125 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.125 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:32.125 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.040810+0000", 2020-01-16T00:14:32.125 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0024973949999999999 2020-01-16T00:14:32.125 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.125 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.125 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:32.125 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.043307+0000", 2020-01-16T00:14:32.126 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8513000000000002e-05 2020-01-16T00:14:32.126 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.126 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.126 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:32.127 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.043346+0000", 2020-01-16T00:14:32.127 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.127e-05 2020-01-16T00:14:32.127 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.127 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.127 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:32.127 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.043417+0000", 2020-01-16T00:14:32.127 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023082354999999999 2020-01-16T00:14:32.127 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:32.128 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:32.128 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:32.128 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.128 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.128 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4705 3.2 3:42b7f20f:::smithi12313934-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e58)", 2020-01-16T00:14:32.128 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.020362+0000", 2020-01-16T00:14:32.128 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 24.851717308000001, 2020-01-16T00:14:32.129 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023262267999999999, 2020-01-16T00:14:32.129 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:32.129 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:32.129 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:32.129 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:32.129 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:32.129 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4705 2020-01-16T00:14:32.129 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.130 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:32.130 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.130 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:32.130 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020362+0000", 2020-01-16T00:14:32.130 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:32.130 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.131 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.131 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:32.131 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020362+0000", 2020-01-16T00:14:32.131 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:32.132 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.132 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.132 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:32.132 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020361+0000", 2020-01-16T00:14:32.133 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.365e-06 2020-01-16T00:14:32.133 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.133 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.133 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:32.133 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020364+0000", 2020-01-16T00:14:32.134 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5649999999999998e-06 2020-01-16T00:14:32.134 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.135 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.135 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:32.135 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020371+0000", 2020-01-16T00:14:32.135 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6349999999999999e-06 2020-01-16T00:14:32.135 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.135 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.135 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:32.136 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020375+0000", 2020-01-16T00:14:32.136 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030174920000000001 2020-01-16T00:14:32.136 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.136 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.136 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:32.136 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023392+0000", 2020-01-16T00:14:32.136 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4997000000000003e-05 2020-01-16T00:14:32.136 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.137 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.137 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:32.137 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023437+0000", 2020-01-16T00:14:32.137 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015293378999999999 2020-01-16T00:14:32.137 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.137 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.137 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:32.138 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.038730+0000", 2020-01-16T00:14:32.138 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000124206 2020-01-16T00:14:32.138 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.138 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.138 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:32.138 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.038855+0000", 2020-01-16T00:14:32.138 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0045951489999999998 2020-01-16T00:14:32.138 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.139 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.139 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:32.139 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.043450+0000", 2020-01-16T00:14:32.139 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1363000000000001e-05 2020-01-16T00:14:32.139 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.139 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.139 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:32.140 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.043521+0000", 2020-01-16T00:14:32.140 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000102737 2020-01-16T00:14:32.140 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.140 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.140 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:32.140 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.043624+0000", 2020-01-16T00:14:32.140 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023262267999999999 2020-01-16T00:14:32.141 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:32.141 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:32.141 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:32.141 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.141 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.141 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4869 3.c 3:3025546b:::smithi12313934-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-01-16T00:14:32.141 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:11.931564+0000", 2020-01-16T00:14:32.142 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 19.940514425, 2020-01-16T00:14:32.142 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018620945, 2020-01-16T00:14:32.142 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:32.142 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:32.142 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:32.142 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:32.142 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:32.143 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4869 2020-01-16T00:14:32.143 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.143 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:32.143 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.143 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:32.143 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.931564+0000", 2020-01-16T00:14:32.143 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:32.143 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.144 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.144 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:32.144 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.931564+0000", 2020-01-16T00:14:32.144 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:32.144 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.144 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.144 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:32.144 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.931563+0000", 2020-01-16T00:14:32.145 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8030000000000002e-06 2020-01-16T00:14:32.145 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.145 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.145 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:32.145 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.931569+0000", 2020-01-16T00:14:32.145 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3370000000000006e-06 2020-01-16T00:14:32.145 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.146 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.146 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:32.146 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.931579+0000", 2020-01-16T00:14:32.146 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9569999999999999e-06 2020-01-16T00:14:32.146 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.146 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.146 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:32.146 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.931585+0000", 2020-01-16T00:14:32.147 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021810430000000001 2020-01-16T00:14:32.147 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.147 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.147 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:32.147 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.933766+0000", 2020-01-16T00:14:32.147 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2802000000000001e-05 2020-01-16T00:14:32.147 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.147 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.148 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:32.148 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.933808+0000", 2020-01-16T00:14:32.148 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0099913469999999994 2020-01-16T00:14:32.148 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.148 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.148 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:32.148 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.943800+0000", 2020-01-16T00:14:32.149 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013257000000000001 2020-01-16T00:14:32.149 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.149 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.149 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:32.149 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.943932+0000", 2020-01-16T00:14:32.149 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0058405799999999997 2020-01-16T00:14:32.149 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.150 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.150 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:32.150 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.949773+0000", 2020-01-16T00:14:32.150 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000158493 2020-01-16T00:14:32.150 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.151 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.151 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:32.151 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.949931+0000", 2020-01-16T00:14:32.151 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000254083 2020-01-16T00:14:32.151 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:32.151 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:32.151 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:32.152 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.950185+0000", 2020-01-16T00:14:32.152 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018620945 2020-01-16T00:14:32.152 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:32.152 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:32.152 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:32.152 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:32.152 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:32.152 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:32.154 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-01-16T00:14:32.154 INFO:tasks.rados.rados.0.smithi123.stdout:1965: expect (ObjNum 675 snap 0 seq_num 675) 2020-01-16T00:14:32.155 INFO:tasks.rados.rados.0.smithi123.stdout:1964: finishing write tid 2 to smithi12313934-21 2020-01-16T00:14:32.155 INFO:tasks.rados.rados.0.smithi123.stdout:1964: finishing write tid 4 to smithi12313934-21 2020-01-16T00:14:32.155 INFO:tasks.rados.rados.0.smithi123.stdout:1964: finishing write tid 5 to smithi12313934-21 2020-01-16T00:14:32.155 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 21 v 102 (ObjNum 745 snap 0 seq_num 745) dirty exists 2020-01-16T00:14:32.155 INFO:tasks.rados.rados.0.smithi123.stdout:1964: left oid 21 (ObjNum 745 snap 0 seq_num 745) 2020-01-16T00:14:32.155 INFO:tasks.rados.rados.0.smithi123.stdout:1967: finishing write tid 1 to smithi12313934-47 2020-01-16T00:14:32.155 INFO:tasks.rados.rados.0.smithi123.stdout:1967: finishing write tid 2 to smithi12313934-47 2020-01-16T00:14:32.156 INFO:tasks.rados.rados.0.smithi123.stdout:1967: finishing write tid 4 to smithi12313934-47 2020-01-16T00:14:32.156 INFO:tasks.rados.rados.0.smithi123.stdout:1968: expect (ObjNum 659 snap 0 seq_num 659) 2020-01-16T00:14:32.156 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:32.152+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:32.166 INFO:tasks.rados.rados.0.smithi123.stdout:1967: finishing write tid 5 to smithi12313934-47 2020-01-16T00:14:32.166 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 47 v 320 (ObjNum 746 snap 0 seq_num 746) dirty exists 2020-01-16T00:14:32.166 INFO:tasks.rados.rados.0.smithi123.stdout:1967: left oid 47 (ObjNum 746 snap 0 seq_num 746) 2020-01-16T00:14:32.166 INFO:tasks.rados.rados.0.smithi123.stdout:1969: finishing write tid 1 to smithi12313934-14 2020-01-16T00:14:32.166 INFO:tasks.rados.rados.0.smithi123.stdout:1969: finishing write tid 2 to smithi12313934-14 2020-01-16T00:14:32.167 INFO:tasks.rados.rados.0.smithi123.stdout:1969: finishing write tid 4 to smithi12313934-14 2020-01-16T00:14:32.167 INFO:tasks.rados.rados.0.smithi123.stdout:1969: finishing write tid 5 to smithi12313934-14 2020-01-16T00:14:32.167 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 14 v 312 (ObjNum 747 snap 0 seq_num 747) dirty exists 2020-01-16T00:14:32.167 INFO:tasks.rados.rados.0.smithi123.stdout:1969: left oid 14 (ObjNum 747 snap 0 seq_num 747) 2020-01-16T00:14:32.167 INFO:tasks.rados.rados.0.smithi123.stdout:1970: finishing write tid 1 to smithi12313934-32 2020-01-16T00:14:32.167 INFO:tasks.rados.rados.0.smithi123.stdout:1970: finishing write tid 2 to smithi12313934-32 2020-01-16T00:14:32.167 INFO:tasks.rados.rados.0.smithi123.stdout:1963: done (15 left) 2020-01-16T00:14:32.167 INFO:tasks.rados.rados.0.smithi123.stdout:1964: done (14 left) 2020-01-16T00:14:32.168 INFO:tasks.rados.rados.0.smithi123.stdout:1965: done (13 left) 2020-01-16T00:14:32.168 INFO:tasks.rados.rados.0.smithi123.stdout:1966: done (12 left) 2020-01-16T00:14:32.168 INFO:tasks.rados.rados.0.smithi123.stdout:1967: done (11 left) 2020-01-16T00:14:32.168 INFO:tasks.rados.rados.0.smithi123.stdout:1968: done (10 left) 2020-01-16T00:14:32.168 INFO:tasks.rados.rados.0.smithi123.stdout:1969: done (9 left) 2020-01-16T00:14:32.168 INFO:tasks.rados.rados.0.smithi123.stdout:1979: read oid 20 snap -1 2020-01-16T00:14:32.168 INFO:tasks.rados.rados.0.smithi123.stdout:1979: expect (ObjNum 713 snap 0 seq_num 713) 2020-01-16T00:14:32.168 INFO:tasks.rados.rados.0.smithi123.stdout:1980: write oid 13 current snap is 0 2020-01-16T00:14:32.169 INFO:tasks.rados.rados.0.smithi123.stdout:1980: seq_num 753 ranges {655671=581444,2030588=625083} 2020-01-16T00:14:32.178 INFO:tasks.rados.rados.0.smithi123.stdout:1980: writing smithi12313934-13 from 655671 to 1237115 tid 1 2020-01-16T00:14:32.187 INFO:tasks.rados.rados.0.smithi123.stdout:1980: writing smithi12313934-13 from 2030588 to 2655671 tid 2 2020-01-16T00:14:32.187 INFO:tasks.rados.rados.0.smithi123.stdout:1981: read oid 33 snap -1 2020-01-16T00:14:32.187 INFO:tasks.rados.rados.0.smithi123.stdout:1981: expect (ObjNum 738 snap 0 seq_num 738) 2020-01-16T00:14:32.187 INFO:tasks.rados.rados.0.smithi123.stdout:1982: read oid 1 snap -1 2020-01-16T00:14:32.187 INFO:tasks.rados.rados.0.smithi123.stdout:1982: expect (ObjNum 729 snap 0 seq_num 729) 2020-01-16T00:14:32.187 INFO:tasks.rados.rados.0.smithi123.stdout:1983: write (excl) oid 31 current snap is 0 2020-01-16T00:14:32.187 INFO:tasks.rados.rados.0.smithi123.stdout:1983: seq_num 754 ranges {672478=656693,2089070=459917} 2020-01-16T00:14:32.195 INFO:tasks.rados.rados.0.smithi123.stdout:1983: writing smithi12313934-31 from 672478 to 1329171 tid 1 2020-01-16T00:14:32.203 INFO:tasks.rados.rados.0.smithi123.stdout:1983: writing smithi12313934-31 from 2089070 to 2548987 tid 2 2020-01-16T00:14:32.203 INFO:tasks.rados.rados.0.smithi123.stdout:1984: read oid 43 snap -1 2020-01-16T00:14:32.203 INFO:tasks.rados.rados.0.smithi123.stdout:1984: expect (ObjNum 683 snap 0 seq_num 683) 2020-01-16T00:14:32.203 INFO:tasks.rados.rados.0.smithi123.stdout:1985: write oid 35 current snap is 0 2020-01-16T00:14:32.203 INFO:tasks.rados.rados.0.smithi123.stdout:1985: seq_num 755 ranges {689285=731942,2147552=541733} 2020-01-16T00:14:32.212 INFO:tasks.rados.rados.0.smithi123.stdout:1985: writing smithi12313934-35 from 689285 to 1421227 tid 1 2020-01-16T00:14:32.220 INFO:tasks.rados.rados.0.smithi123.stdout:1985: writing smithi12313934-35 from 2147552 to 2689285 tid 2 2020-01-16T00:14:32.221 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:32.221 INFO:tasks.rados.rados.0.smithi123.stdout:1970: finishing write tid 4 to smithi12313934-32 2020-01-16T00:14:32.221 INFO:tasks.rados.rados.0.smithi123.stdout:1970: finishing write tid 5 to smithi12313934-32 2020-01-16T00:14:32.221 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 32 v 196 (ObjNum 748 snap 0 seq_num 748) dirty exists 2020-01-16T00:14:32.221 INFO:tasks.rados.rados.0.smithi123.stdout:1970: left oid 32 (ObjNum 748 snap 0 seq_num 748) 2020-01-16T00:14:32.221 INFO:tasks.rados.rados.0.smithi123.stdout:1971: finishing write tid 1 to smithi12313934-30 2020-01-16T00:14:32.221 INFO:tasks.rados.rados.0.smithi123.stdout:1970: done (15 left) 2020-01-16T00:14:32.222 INFO:tasks.rados.rados.0.smithi123.stdout:1986: read oid 22 snap -1 2020-01-16T00:14:32.222 INFO:tasks.rados.rados.0.smithi123.stdout:1986: expect (ObjNum 711 snap 0 seq_num 711) 2020-01-16T00:14:32.222 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:32.222 INFO:tasks.rados.rados.0.smithi123.stdout:1971: finishing write tid 2 to smithi12313934-30 2020-01-16T00:14:32.222 INFO:tasks.rados.rados.0.smithi123.stdout:1973: finishing write tid 1 to smithi12313934-39 2020-01-16T00:14:32.223 INFO:tasks.rados.rados.0.smithi123.stdout:1972: done (15 left) 2020-01-16T00:14:32.223 INFO:tasks.rados.rados.0.smithi123.stdout:1987: write (excl) oid 40 current snap is 0 2020-01-16T00:14:32.223 INFO:tasks.rados.rados.0.smithi123.stdout:1987: seq_num 756 ranges {706092=407191,1806034=463586} 2020-01-16T00:14:32.226 INFO:tasks.rados.rados.0.smithi123.stdout:1987: writing smithi12313934-40 from 706092 to 1113283 tid 1 2020-01-16T00:14:32.234 INFO:tasks.rados.rados.0.smithi123.stdout:1987: writing smithi12313934-40 from 1806034 to 2269620 tid 2 2020-01-16T00:14:32.234 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:32.234 INFO:tasks.rados.rados.0.smithi123.stdout:1971: finishing write tid 4 to smithi12313934-30 2020-01-16T00:14:32.234 INFO:tasks.rados.rados.0.smithi123.stdout:1971: finishing write tid 5 to smithi12313934-30 2020-01-16T00:14:32.234 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 30 v 204 (ObjNum 749 snap 0 seq_num 749) dirty exists 2020-01-16T00:14:32.235 INFO:tasks.rados.rados.0.smithi123.stdout:1971: left oid 30 (ObjNum 749 snap 0 seq_num 749) 2020-01-16T00:14:32.235 INFO:tasks.rados.rados.0.smithi123.stdout:1973: finishing write tid 2 to smithi12313934-39 2020-01-16T00:14:32.235 INFO:tasks.rados.rados.0.smithi123.stdout:1971: done (15 left) 2020-01-16T00:14:32.235 INFO:tasks.rados.rados.0.smithi123.stdout:1988: read oid 19 snap -1 2020-01-16T00:14:32.235 INFO:tasks.rados.rados.0.smithi123.stdout:1988: expect (ObjNum 742 snap 0 seq_num 742) 2020-01-16T00:14:32.235 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:32.235 INFO:tasks.rados.rados.0.smithi123.stdout:1973: finishing write tid 4 to smithi12313934-39 2020-01-16T00:14:32.235 INFO:tasks.rados.rados.0.smithi123.stdout:1973: finishing write tid 5 to smithi12313934-39 2020-01-16T00:14:32.236 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 39 v 315 (ObjNum 750 snap 0 seq_num 750) dirty exists 2020-01-16T00:14:32.236 INFO:tasks.rados.rados.0.smithi123.stdout:1973: left oid 39 (ObjNum 750 snap 0 seq_num 750) 2020-01-16T00:14:32.236 INFO:tasks.rados.rados.0.smithi123.stdout:1973: done (15 left) 2020-01-16T00:14:32.236 INFO:tasks.rados.rados.0.smithi123.stdout:1989: write oid 8 current snap is 0 2020-01-16T00:14:32.236 INFO:tasks.rados.rados.0.smithi123.stdout:1989: seq_num 757 ranges {722899=482440,1748163=723597} 2020-01-16T00:14:32.240 INFO:tasks.rados.rados.0.smithi123.stdout:1989: writing smithi12313934-8 from 722899 to 1205339 tid 1 2020-01-16T00:14:32.247 INFO:tasks.rados.rados.0.smithi123.stdout:1989: writing smithi12313934-8 from 1748163 to 2471760 tid 2 2020-01-16T00:14:32.248 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:32.248 INFO:tasks.rados.rados.0.smithi123.stdout:1974: expect (ObjNum 720 snap 0 seq_num 720) 2020-01-16T00:14:32.254 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:32.250+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:32.277 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:32.277 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:14:32.277 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:14:32.277 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:32.286 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-01-16T00:14:32.297 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-01-16T00:14:32.374 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:32.374+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:32.427 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:32.427 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:14:32.428 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:14:32.428 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:14:32.428 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:32.436 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-01-16T00:14:32.471 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:14:32.471 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:32.470+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:32.486 INFO:tasks.rados.rados.0.smithi123.stdout:1975: expect (ObjNum 667 snap 0 seq_num 667) 2020-01-16T00:14:32.493 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:14:32.512 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:14:32.532 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:14:32.554 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:14:32.556 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:32.554+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:32.568 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:32.577 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-01-16T00:14:32.578 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:14:32.608 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:14:32.609 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-01-16T00:14:32.609 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-01-16T00:14:32.610 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-01-16T00:14:32.610 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-01-16T00:14:32.610 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-01-16T00:14:32.610 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENXIO: problem getting command descriptions from osd.5 2020-01-16T00:14:32.610 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: problem getting command descriptions from osd.5 2020-01-16T00:14:32.610 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-01-16T00:14:32.610 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-01-16T00:14:32.618 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-01-16T00:14:32.618 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-01-16T00:14:32.658 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:32.656+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:32.692 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:32.702 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-01-16T00:14:32.759 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:32.758+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:32.846 INFO:tasks.rados.rados.0.smithi123.stdout:1976: expect (ObjNum 727 snap 0 seq_num 727) 2020-01-16T00:14:32.854 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:32.860 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:32.858+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:32.864 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-01-16T00:14:32.960 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:32.958+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:32.966 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:32.975 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-01-16T00:14:33.016 INFO:tasks.rados.rados.0.smithi123.stdout:1977: finishing write tid 1 to smithi12313934-27 2020-01-16T00:14:33.017 INFO:tasks.rados.rados.0.smithi123.stdout:1977: finishing write tid 2 to smithi12313934-27 2020-01-16T00:14:33.017 INFO:tasks.rados.rados.0.smithi123.stdout:1977: finishing write tid 4 to smithi12313934-27 2020-01-16T00:14:33.017 INFO:tasks.rados.rados.0.smithi123.stdout:1977: finishing write tid 5 to smithi12313934-27 2020-01-16T00:14:33.017 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 27 v 199 (ObjNum 751 snap 0 seq_num 751) dirty exists 2020-01-16T00:14:33.017 INFO:tasks.rados.rados.0.smithi123.stdout:1977: left oid 27 (ObjNum 751 snap 0 seq_num 751) 2020-01-16T00:14:33.017 INFO:tasks.rados.rados.0.smithi123.stdout:1978: finishing write tid 1 to smithi12313934-11 2020-01-16T00:14:33.018 INFO:tasks.rados.rados.0.smithi123.stdout:1978: finishing write tid 2 to smithi12313934-11 2020-01-16T00:14:33.018 INFO:tasks.rados.rados.0.smithi123.stdout:1978: finishing write tid 4 to smithi12313934-11 2020-01-16T00:14:33.018 INFO:tasks.rados.rados.0.smithi123.stdout:1978: finishing write tid 5 to smithi12313934-11 2020-01-16T00:14:33.018 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 11 v 208 (ObjNum 752 snap 0 seq_num 752) dirty exists 2020-01-16T00:14:33.018 INFO:tasks.rados.rados.0.smithi123.stdout:1978: left oid 11 (ObjNum 752 snap 0 seq_num 752) 2020-01-16T00:14:33.018 INFO:tasks.rados.rados.0.smithi123.stdout:1979: expect (ObjNum 713 snap 0 seq_num 713) 2020-01-16T00:14:33.061 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:33.056+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:33.082 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:33.091 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-01-16T00:14:33.121 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2020-01-16T00:14:33.166 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:33.166+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:33.184 INFO:tasks.rados.rados.0.smithi123.stdout:1980: finishing write tid 1 to smithi12313934-13 2020-01-16T00:14:33.184 INFO:tasks.rados.rados.0.smithi123.stdout:1981: expect (ObjNum 738 snap 0 seq_num 738) 2020-01-16T00:14:33.244 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:33.254 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-01-16T00:14:33.262 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:33.258+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:33.286 INFO:tasks.rados.rados.0.smithi123.stdout:1980: finishing write tid 2 to smithi12313934-13 2020-01-16T00:14:33.286 INFO:tasks.rados.rados.0.smithi123.stdout:1980: finishing write tid 4 to smithi12313934-13 2020-01-16T00:14:33.286 INFO:tasks.rados.rados.0.smithi123.stdout:1980: finishing write tid 5 to smithi12313934-13 2020-01-16T00:14:33.287 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 138 (ObjNum 753 snap 0 seq_num 753) dirty exists 2020-01-16T00:14:33.287 INFO:tasks.rados.rados.0.smithi123.stdout:1980: left oid 13 (ObjNum 753 snap 0 seq_num 753) 2020-01-16T00:14:33.287 INFO:tasks.rados.rados.0.smithi123.stdout:1982: expect (ObjNum 729 snap 0 seq_num 729) 2020-01-16T00:14:33.358 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:33.362 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:33.360+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:33.368 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-01-16T00:14:33.464 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:33.460+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:33.465 INFO:tasks.rados.rados.0.smithi123.stdout:1983: finishing write tid 1 to smithi12313934-31 2020-01-16T00:14:33.465 INFO:tasks.rados.rados.0.smithi123.stdout:1983: finishing write tid 2 to smithi12313934-31 2020-01-16T00:14:33.465 INFO:tasks.rados.rados.0.smithi123.stdout:1983: finishing write tid 4 to smithi12313934-31 2020-01-16T00:14:33.465 INFO:tasks.rados.rados.0.smithi123.stdout:1983: finishing write tid 5 to smithi12313934-31 2020-01-16T00:14:33.465 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 31 v 257 (ObjNum 754 snap 0 seq_num 754) dirty exists 2020-01-16T00:14:33.466 INFO:tasks.rados.rados.0.smithi123.stdout:1983: left oid 31 (ObjNum 754 snap 0 seq_num 754) 2020-01-16T00:14:33.466 INFO:tasks.rados.rados.0.smithi123.stdout:1984: expect (ObjNum 683 snap 0 seq_num 683) 2020-01-16T00:14:33.518 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:33.527 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-01-16T00:14:33.565 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:33.562+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:33.658 INFO:tasks.rados.rados.0.smithi123.stdout:1985: finishing write tid 1 to smithi12313934-35 2020-01-16T00:14:33.658 INFO:tasks.rados.rados.0.smithi123.stdout:1985: finishing write tid 2 to smithi12313934-35 2020-01-16T00:14:33.658 INFO:tasks.rados.rados.0.smithi123.stdout:1985: finishing write tid 4 to smithi12313934-35 2020-01-16T00:14:33.658 INFO:tasks.rados.rados.0.smithi123.stdout:1985: finishing write tid 5 to smithi12313934-35 2020-01-16T00:14:33.659 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 35 v 215 (ObjNum 755 snap 0 seq_num 755) dirty exists 2020-01-16T00:14:33.659 INFO:tasks.rados.rados.0.smithi123.stdout:1985: left oid 35 (ObjNum 755 snap 0 seq_num 755) 2020-01-16T00:14:33.659 INFO:tasks.rados.rados.0.smithi123.stdout:1987: finishing write tid 1 to smithi12313934-40 2020-01-16T00:14:33.659 INFO:tasks.rados.rados.0.smithi123.stdout:1986: expect (ObjNum 711 snap 0 seq_num 711) 2020-01-16T00:14:33.665 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:33.662+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:33.680 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:33.689 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-01-16T00:14:33.766 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:33.764+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:33.796 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:33.805 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-01-16T00:14:33.867 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:33.866+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:33.913 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:33.919 INFO:tasks.rados.rados.0.smithi123.stdout:1987: finishing write tid 2 to smithi12313934-40 2020-01-16T00:14:33.919 INFO:tasks.rados.rados.0.smithi123.stdout:1987: finishing write tid 4 to smithi12313934-40 2020-01-16T00:14:33.919 INFO:tasks.rados.rados.0.smithi123.stdout:1974: done (15 left) 2020-01-16T00:14:33.919 INFO:tasks.rados.rados.0.smithi123.stdout:1975: done (14 left) 2020-01-16T00:14:33.919 INFO:tasks.rados.rados.0.smithi123.stdout:1976: done (13 left) 2020-01-16T00:14:33.919 INFO:tasks.rados.rados.0.smithi123.stdout:1977: done (12 left) 2020-01-16T00:14:33.920 INFO:tasks.rados.rados.0.smithi123.stdout:1978: done (11 left) 2020-01-16T00:14:33.920 INFO:tasks.rados.rados.0.smithi123.stdout:1979: done (10 left) 2020-01-16T00:14:33.920 INFO:tasks.rados.rados.0.smithi123.stdout:1980: done (9 left) 2020-01-16T00:14:33.920 INFO:tasks.rados.rados.0.smithi123.stdout:1981: done (8 left) 2020-01-16T00:14:33.920 INFO:tasks.rados.rados.0.smithi123.stdout:1982: done (7 left) 2020-01-16T00:14:33.920 INFO:tasks.rados.rados.0.smithi123.stdout:1983: done (6 left) 2020-01-16T00:14:33.920 INFO:tasks.rados.rados.0.smithi123.stdout:1984: done (5 left) 2020-01-16T00:14:33.920 INFO:tasks.rados.rados.0.smithi123.stdout:1985: done (4 left) 2020-01-16T00:14:33.921 INFO:tasks.rados.rados.0.smithi123.stdout:1986: done (3 left) 2020-01-16T00:14:33.921 INFO:tasks.rados.rados.0.smithi123.stdout:1990: write (excl) oid 35 current snap is 0 2020-01-16T00:14:33.921 INFO:tasks.rados.rados.0.smithi123.stdout:1990: seq_num 758 ranges {739706=557689,1806645=467255} 2020-01-16T00:14:33.922 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-01-16T00:14:33.936 INFO:tasks.rados.rados.0.smithi123.stdout:1990: writing smithi12313934-35 from 739706 to 1297395 tid 1 2020-01-16T00:14:33.942 INFO:tasks.rados.rados.0.smithi123.stdout:1990: writing smithi12313934-35 from 1806645 to 2273900 tid 2 2020-01-16T00:14:33.943 INFO:tasks.rados.rados.0.smithi123.stdout:1991: read oid 30 snap -1 2020-01-16T00:14:33.943 INFO:tasks.rados.rados.0.smithi123.stdout:1991: expect (ObjNum 749 snap 0 seq_num 749) 2020-01-16T00:14:33.943 INFO:tasks.rados.rados.0.smithi123.stdout:1992: read oid 28 snap -1 2020-01-16T00:14:33.943 INFO:tasks.rados.rados.0.smithi123.stdout:1992: expect (ObjNum 722 snap 0 seq_num 722) 2020-01-16T00:14:33.943 INFO:tasks.rados.rados.0.smithi123.stdout:1993: read oid 18 snap -1 2020-01-16T00:14:33.943 INFO:tasks.rados.rados.0.smithi123.stdout:1993: expect (ObjNum 701 snap 0 seq_num 701) 2020-01-16T00:14:33.944 INFO:tasks.rados.rados.0.smithi123.stdout:1994: write (excl) oid 12 current snap is 0 2020-01-16T00:14:33.944 INFO:tasks.rados.rados.0.smithi123.stdout:1994: seq_num 759 ranges {756513=632938,1865127=727266} 2020-01-16T00:14:33.950 INFO:tasks.rados.rados.0.smithi123.stdout:1994: writing smithi12313934-12 from 756513 to 1389451 tid 1 2020-01-16T00:14:33.958 INFO:tasks.rados.rados.0.smithi123.stdout:1994: writing smithi12313934-12 from 1865127 to 2592393 tid 2 2020-01-16T00:14:33.958 INFO:tasks.rados.rados.0.smithi123.stdout:1995: delete oid 37 current snap is 0 2020-01-16T00:14:33.958 INFO:tasks.rados.rados.0.smithi123.stdout:1987: finishing write tid 5 to smithi12313934-40 2020-01-16T00:14:33.958 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 40 v 318 (ObjNum 756 snap 0 seq_num 756) dirty exists 2020-01-16T00:14:33.958 INFO:tasks.rados.rados.0.smithi123.stdout:1987: left oid 40 (ObjNum 756 snap 0 seq_num 756) 2020-01-16T00:14:33.959 INFO:tasks.rados.rados.0.smithi123.stdout:1988: expect (ObjNum 742 snap 0 seq_num 742) 2020-01-16T00:14:33.968 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:33.964+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:34.035 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:34.045 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-01-16T00:14:34.068 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:34.064+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:34.131 INFO:teuthology.orchestra.run.smithi123.stderr:nodeep-scrub is set 2020-01-16T00:14:34.169 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:34.168+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:34.192 INFO:tasks.rados.rados.0.smithi123.stdout:1989: finishing write tid 1 to smithi12313934-8 2020-01-16T00:14:34.193 INFO:tasks.rados.rados.0.smithi123.stdout:1989: finishing write tid 2 to smithi12313934-8 2020-01-16T00:14:34.193 INFO:tasks.rados.rados.0.smithi123.stdout:1989: finishing write tid 4 to smithi12313934-8 2020-01-16T00:14:34.193 INFO:tasks.rados.rados.0.smithi123.stdout:1989: finishing write tid 5 to smithi12313934-8 2020-01-16T00:14:34.193 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 8 v 137 (ObjNum 757 snap 0 seq_num 757) dirty exists 2020-01-16T00:14:34.193 INFO:tasks.rados.rados.0.smithi123.stdout:1989: left oid 8 (ObjNum 757 snap 0 seq_num 757) 2020-01-16T00:14:34.193 INFO:tasks.rados.rados.0.smithi123.stdout:1987: done (8 left) 2020-01-16T00:14:34.193 INFO:tasks.rados.rados.0.smithi123.stdout:1988: done (7 left) 2020-01-16T00:14:34.194 INFO:tasks.rados.rados.0.smithi123.stdout:1989: done (6 left) 2020-01-16T00:14:34.194 INFO:tasks.rados.rados.0.smithi123.stdout:1995: done (5 left) 2020-01-16T00:14:34.194 INFO:tasks.rados.rados.0.smithi123.stdout:1996: write (excl) oid 9 current snap is 0 2020-01-16T00:14:34.194 INFO:tasks.rados.rados.0.smithi123.stdout:1996: seq_num 761 ranges {790127=499789,1698444=730935} 2020-01-16T00:14:34.198 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:34.199 INFO:tasks.rados.rados.0.smithi123.stdout:1996: writing smithi12313934-9 from 790127 to 1289916 tid 1 2020-01-16T00:14:34.207 INFO:tasks.rados.rados.0.smithi123.stdout:1996: writing smithi12313934-9 from 1698444 to 2429379 tid 2 2020-01-16T00:14:34.207 INFO:tasks.rados.rados.0.smithi123.stdout:1997: write oid 47 current snap is 0 2020-01-16T00:14:34.207 INFO:tasks.rados.rados.0.smithi123.stdout:1997: seq_num 762 ranges {406934=575038,1640573=474593,2642791=164143} 2020-01-16T00:14:34.208 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-01-16T00:14:34.213 INFO:tasks.rados.rados.0.smithi123.stdout:1997: writing smithi12313934-47 from 406934 to 981972 tid 1 2020-01-16T00:14:34.220 INFO:tasks.rados.rados.0.smithi123.stdout:1997: writing smithi12313934-47 from 1640573 to 2115166 tid 2 2020-01-16T00:14:34.224 INFO:tasks.rados.rados.0.smithi123.stdout:1997: writing smithi12313934-47 from 2642791 to 2806934 tid 3 2020-01-16T00:14:34.224 INFO:tasks.rados.rados.0.smithi123.stdout:1998: read oid 5 snap -1 2020-01-16T00:14:34.224 INFO:tasks.rados.rados.0.smithi123.stdout:1998: expect (ObjNum 739 snap 0 seq_num 739) 2020-01-16T00:14:34.224 INFO:tasks.rados.rados.0.smithi123.stdout:1999: write oid 43 current snap is 0 2020-01-16T00:14:34.225 INFO:tasks.rados.rados.0.smithi123.stdout:1999: seq_num 763 ranges {423741=650287,1699055=618251} 2020-01-16T00:14:34.230 INFO:tasks.rados.rados.0.smithi123.stdout:1999: writing smithi12313934-43 from 423741 to 1074028 tid 1 2020-01-16T00:14:34.237 INFO:tasks.rados.rados.0.smithi123.stdout:1999: writing smithi12313934-43 from 1699055 to 2317306 tid 2 2020-01-16T00:14:34.238 INFO:tasks.rados.rados.0.smithi123.stdout:2000: write (excl) oid 48 current snap is 0 2020-01-16T00:14:34.238 INFO:tasks.rados.rados.0.smithi123.stdout:2000: seq_num 764 ranges {440548=725536,1757537=478262,2813990=26558} 2020-01-16T00:14:34.244 INFO:tasks.rados.rados.0.smithi123.stdout:2000: writing smithi12313934-48 from 440548 to 1166084 tid 1 2020-01-16T00:14:34.250 INFO:tasks.rados.rados.0.smithi123.stdout:2000: writing smithi12313934-48 from 1757537 to 2235799 tid 2 2020-01-16T00:14:34.253 INFO:tasks.rados.rados.0.smithi123.stdout:2000: writing smithi12313934-48 from 2813990 to 2840548 tid 3 2020-01-16T00:14:34.253 INFO:tasks.rados.rados.0.smithi123.stdout:2001: write oid 22 current snap is 0 2020-01-16T00:14:34.254 INFO:tasks.rados.rados.0.smithi123.stdout:2001: seq_num 765 ranges {457355=400785,1416019=621920,2725060=132295} 2020-01-16T00:14:34.258 INFO:tasks.rados.rados.0.smithi123.stdout:2001: writing smithi12313934-22 from 457355 to 858140 tid 1 2020-01-16T00:14:34.265 INFO:tasks.rados.rados.0.smithi123.stdout:2001: writing smithi12313934-22 from 1416019 to 2037939 tid 2 2020-01-16T00:14:34.269 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:34.266+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:34.270 INFO:tasks.rados.rados.0.smithi123.stdout:2001: writing smithi12313934-22 from 2725060 to 2857355 tid 3 2020-01-16T00:14:34.270 INFO:tasks.rados.rados.0.smithi123.stdout:2002: write (excl) oid 40 current snap is 0 2020-01-16T00:14:34.270 INFO:tasks.rados.rados.0.smithi123.stdout:2002: seq_num 766 ranges {474162=476034,1358148=481931,2352483=465303} 2020-01-16T00:14:34.275 INFO:tasks.rados.rados.0.smithi123.stdout:2002: writing smithi12313934-40 from 474162 to 950196 tid 1 2020-01-16T00:14:34.280 INFO:tasks.rados.rados.0.smithi123.stdout:2002: writing smithi12313934-40 from 1358148 to 1840079 tid 2 2020-01-16T00:14:34.286 INFO:tasks.rados.rados.0.smithi123.stdout:2002: writing smithi12313934-40 from 2352483 to 2817786 tid 3 2020-01-16T00:14:34.286 INFO:tasks.rados.rados.0.smithi123.stdout:2003: write oid 29 current snap is 0 2020-01-16T00:14:34.286 INFO:tasks.rados.rados.0.smithi123.stdout:2003: seq_num 767 ranges {490969=551283,1816630=625589} 2020-01-16T00:14:34.291 INFO:tasks.rados.rados.0.smithi123.stdout:2003: writing smithi12313934-29 from 490969 to 1042252 tid 1 2020-01-16T00:14:34.299 INFO:tasks.rados.rados.0.smithi123.stdout:2003: writing smithi12313934-29 from 1816630 to 2442219 tid 2 2020-01-16T00:14:34.300 INFO:tasks.rados.rados.0.smithi123.stdout:2004: write oid 38 current snap is 0 2020-01-16T00:14:34.300 INFO:tasks.rados.rados.0.smithi123.stdout:2004: seq_num 768 ranges {507776=742885,1991465=485600} 2020-01-16T00:14:34.306 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:34.306 INFO:tasks.rados.rados.0.smithi123.stdout:2004: writing smithi12313934-38 from 507776 to 1250661 tid 1 2020-01-16T00:14:34.313 INFO:tasks.rados.rados.0.smithi123.stdout:2004: writing smithi12313934-38 from 1991465 to 2477065 tid 2 2020-01-16T00:14:34.314 INFO:tasks.rados.rados.0.smithi123.stdout:2005: write oid 33 current snap is 0 2020-01-16T00:14:34.314 INFO:tasks.rados.rados.0.smithi123.stdout:2005: seq_num 769 ranges {524583=418134,1649947=629258,2834752=89831} 2020-01-16T00:14:34.315 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-01-16T00:14:34.319 INFO:tasks.rados.rados.0.smithi123.stdout:2005: writing smithi12313934-33 from 524583 to 942717 tid 1 2020-01-16T00:14:34.326 INFO:tasks.rados.rados.0.smithi123.stdout:2005: writing smithi12313934-33 from 1649947 to 2279205 tid 2 2020-01-16T00:14:34.330 INFO:tasks.rados.rados.0.smithi123.stdout:2005: writing smithi12313934-33 from 2834752 to 2924583 tid 3 2020-01-16T00:14:34.330 INFO:tasks.rados.rados.0.smithi123.stdout:2006: read oid 6 snap -1 2020-01-16T00:14:34.330 INFO:tasks.rados.rados.0.smithi123.stdout:2006: expect (ObjNum 720 snap 0 seq_num 720) 2020-01-16T00:14:34.331 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:34.331 INFO:tasks.rados.rados.0.smithi123.stdout:1990: finishing write tid 1 to smithi12313934-35 2020-01-16T00:14:34.331 INFO:tasks.rados.rados.0.smithi123.stdout:1990: finishing write tid 2 to smithi12313934-35 2020-01-16T00:14:34.331 INFO:tasks.rados.rados.0.smithi123.stdout:1990: finishing write tid 4 to smithi12313934-35 2020-01-16T00:14:34.331 INFO:tasks.rados.rados.0.smithi123.stdout:1990: finishing write tid 5 to smithi12313934-35 2020-01-16T00:14:34.331 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 35 v 218 (ObjNum 758 snap 0 seq_num 758) dirty exists 2020-01-16T00:14:34.331 INFO:tasks.rados.rados.0.smithi123.stdout:1990: left oid 35 (ObjNum 758 snap 0 seq_num 758) 2020-01-16T00:14:34.332 INFO:tasks.rados.rados.0.smithi123.stdout:1992: expect (ObjNum 722 snap 0 seq_num 722) 2020-01-16T00:14:34.370 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:34.366+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:34.411 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:34.420 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-01-16T00:14:34.506 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:34.506+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:34.511 INFO:tasks.rados.rados.0.smithi123.stdout:1991: expect (ObjNum 749 snap 0 seq_num 749) 2020-01-16T00:14:34.564 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:34.572 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:34.570+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:34.572 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-01-16T00:14:34.619 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-01-16T00:14:34.673 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:34.668+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:34.714 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:34.723 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-01-16T00:14:34.773 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:34.770+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:34.792 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:14:34.807 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:14:34.817 INFO:tasks.rados.rados.0.smithi123.stdout:1993: expect (ObjNum 701 snap 0 seq_num 701) 2020-01-16T00:14:34.825 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:34.827 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:14:34.834 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-01-16T00:14:34.844 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:14:34.865 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:14:34.874 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:34.872+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:34.886 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:14:34.907 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:14:34.909 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-01-16T00:14:34.909 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-01-16T00:14:34.909 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-01-16T00:14:34.909 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-01-16T00:14:34.910 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-01-16T00:14:34.910 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENXIO: problem getting command descriptions from osd.5 2020-01-16T00:14:34.910 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: problem getting command descriptions from osd.5 2020-01-16T00:14:34.910 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-01-16T00:14:34.910 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-01-16T00:14:34.918 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-01-16T00:14:34.919 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-01-16T00:14:34.946 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:34.946 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:14:34.946 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:14:34.946 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:14:34.946 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:34.947 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2226 3.7 3:e1be6d3b:::smithi12313934-24:head [setxattr _header (56) in=7b,truncate 2100908] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:14:34.947 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.980217+0000", 2020-01-16T00:14:34.947 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 88.963784415000006, 2020-01-16T00:14:34.947 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023113018999999999, 2020-01-16T00:14:34.948 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:34.948 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:34.948 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:34.948 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:34.948 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:34.948 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2226 2020-01-16T00:14:34.948 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:34.948 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:34.949 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:34.949 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:34.949 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980217+0000", 2020-01-16T00:14:34.949 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:34.949 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:34.949 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:34.949 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:34.950 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980217+0000", 2020-01-16T00:14:34.950 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:34.950 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:34.950 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:34.950 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:34.950 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980216+0000", 2020-01-16T00:14:34.950 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5049999999999998e-06 2020-01-16T00:14:34.951 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:34.951 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:34.951 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:34.951 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980222+0000", 2020-01-16T00:14:34.951 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5654999999999998e-05 2020-01-16T00:14:34.951 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:34.951 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:34.952 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:34.952 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980237+0000", 2020-01-16T00:14:34.952 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9740000000000004e-06 2020-01-16T00:14:34.952 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:34.952 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:34.952 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:34.953 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980241+0000", 2020-01-16T00:14:34.953 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002947776 2020-01-16T00:14:34.953 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:34.953 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:34.953 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:34.953 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.983189+0000", 2020-01-16T00:14:34.953 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0027099400000000001 2020-01-16T00:14:34.954 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:34.954 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:34.954 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:34.954 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985899+0000", 2020-01-16T00:14:34.954 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014275589999999999 2020-01-16T00:14:34.954 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:34.954 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:34.955 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:34.955 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.000175+0000", 2020-01-16T00:14:34.974 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011211 2020-01-16T00:14:34.974 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:34.975 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:34.975 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:34.975 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.000287+0000", 2020-01-16T00:14:34.975 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000229886 2020-01-16T00:14:34.975 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:34.975 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:34.975 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,7", 2020-01-16T00:14:34.976 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.000517+0000", 2020-01-16T00:14:34.976 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00095715399999999999 2020-01-16T00:14:34.976 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:34.976 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:34.976 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:34.976 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.001474+0000", 2020-01-16T00:14:34.977 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001341512 2020-01-16T00:14:34.977 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:34.977 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:34.978 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:34.978 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.002815+0000", 2020-01-16T00:14:34.978 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035789099999999999 2020-01-16T00:14:34.978 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:34.979 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:34.979 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:34.979 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003173+0000", 2020-01-16T00:14:34.979 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.8571000000000005e-05 2020-01-16T00:14:34.979 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:34.980 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:34.980 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:34.980 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003242+0000", 2020-01-16T00:14:34.980 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.8293000000000003e-05 2020-01-16T00:14:34.980 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:34.980 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:34.981 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:34.981 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003330+0000", 2020-01-16T00:14:34.981 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023113018999999999 2020-01-16T00:14:34.981 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:34.981 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:34.981 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:34.982 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:34.982 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:34.982 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2227 3.7 3:e1be6d3b:::smithi12313934-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e35)", 2020-01-16T00:14:34.982 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:14:34.982 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 88.963737162000001, 2020-01-16T00:14:34.982 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023303356000000001, 2020-01-16T00:14:34.982 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:34.983 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:34.983 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:34.983 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:34.983 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:34.983 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2227 2020-01-16T00:14:34.983 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:34.983 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:34.984 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:34.984 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:34.984 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:14:34.984 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:34.984 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:34.984 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:34.984 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:34.985 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:14:34.985 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:34.985 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:34.985 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:34.985 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:34.986 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980264+0000", 2020-01-16T00:14:34.986 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3079999999999999e-06 2020-01-16T00:14:34.986 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:34.986 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:34.986 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:34.986 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980267+0000", 2020-01-16T00:14:34.987 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9290000000000001e-06 2020-01-16T00:14:34.987 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:34.987 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:34.987 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:34.987 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980274+0000", 2020-01-16T00:14:34.988 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7400000000000002e-06 2020-01-16T00:14:34.988 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:34.988 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:34.988 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:34.988 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.980278+0000", 2020-01-16T00:14:34.988 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056624300000000004 2020-01-16T00:14:34.988 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:34.989 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:34.989 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:34.989 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985940+0000", 2020-01-16T00:14:34.989 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4261000000000002e-05 2020-01-16T00:14:34.989 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:34.989 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:34.990 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:34.990 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985985+0000", 2020-01-16T00:14:34.990 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015087676 2020-01-16T00:14:34.990 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:34.990 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:34.990 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:34.991 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.001072+0000", 2020-01-16T00:14:34.991 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011960600000000001 2020-01-16T00:14:34.991 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:34.991 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:34.991 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:34.991 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.001192+0000", 2020-01-16T00:14:34.991 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021718309999999999 2020-01-16T00:14:34.992 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:34.992 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:34.992 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:34.992 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003364+0000", 2020-01-16T00:14:34.992 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000102855 2020-01-16T00:14:34.992 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:34.993 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:34.993 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:34.993 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003467+0000", 2020-01-16T00:14:34.993 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010122499999999999 2020-01-16T00:14:34.993 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:34.994 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:34.994 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:34.994 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:06.003568+0000", 2020-01-16T00:14:34.994 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023303356000000001 2020-01-16T00:14:34.994 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:34.994 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:34.995 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:34.995 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:34.995 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:34.995 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2325 3.7 3:e1be6d3b:::smithi12313934-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e36)", 2020-01-16T00:14:34.995 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.550877+0000", 2020-01-16T00:14:34.995 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 87.393124834999995, 2020-01-16T00:14:34.996 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018565154, 2020-01-16T00:14:34.996 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:34.996 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:34.996 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:34.996 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:34.996 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:34.997 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2325 2020-01-16T00:14:34.997 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:34.997 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:34.997 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:34.997 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:34.997 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550877+0000", 2020-01-16T00:14:34.998 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:34.998 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:34.998 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:34.998 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:34.998 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550877+0000", 2020-01-16T00:14:34.999 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:34.999 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:34.999 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:34.999 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:34.999 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550876+0000", 2020-01-16T00:14:34.999 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1080000000000003e-06 2020-01-16T00:14:35.000 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.000 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.000 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:35.000 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550880+0000", 2020-01-16T00:14:35.000 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3269999999999997e-06 2020-01-16T00:14:35.000 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.001 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.001 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:35.001 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550886+0000", 2020-01-16T00:14:35.001 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.386e-06 2020-01-16T00:14:35.001 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.001 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.002 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:35.002 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.550891+0000", 2020-01-16T00:14:35.002 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048231410000000004 2020-01-16T00:14:35.002 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.002 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.002 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.003 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555714+0000", 2020-01-16T00:14:35.003 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0071999999999999e-05 2020-01-16T00:14:35.003 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.003 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.003 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:35.003 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555744+0000", 2020-01-16T00:14:35.004 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012329005000000001 2020-01-16T00:14:35.004 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.004 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.004 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.004 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568073+0000", 2020-01-16T00:14:35.004 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9087999999999997e-05 2020-01-16T00:14:35.004 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.005 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.005 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:35.005 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.568172+0000", 2020-01-16T00:14:35.005 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0010718220000000001 2020-01-16T00:14:35.005 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.005 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.006 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.006 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569244+0000", 2020-01-16T00:14:35.006 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.8199e-05 2020-01-16T00:14:35.006 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.006 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.006 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:35.007 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569342+0000", 2020-01-16T00:14:35.007 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9708000000000001e-05 2020-01-16T00:14:35.007 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.007 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.007 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:35.007 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.569442+0000", 2020-01-16T00:14:35.008 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018565154 2020-01-16T00:14:35.008 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.008 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.008 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.008 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.008 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.008 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2800 3.2 3:42b7f20f:::smithi12313934-42:head [write 1395221~583781] snapc 0=[] ondisk+write+known_if_redirected e41)", 2020-01-16T00:14:35.009 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:17.256182+0000", 2020-01-16T00:14:35.009 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 77.687819294999997, 2020-01-16T00:14:35.009 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018509650999999998, 2020-01-16T00:14:35.009 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:35.009 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:35.009 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:35.009 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:35.010 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:35.010 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2800 2020-01-16T00:14:35.010 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.010 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:35.010 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.010 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:35.011 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256182+0000", 2020-01-16T00:14:35.011 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:35.011 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.011 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.011 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:35.011 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256182+0000", 2020-01-16T00:14:35.012 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:35.012 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.012 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.012 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:35.012 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256180+0000", 2020-01-16T00:14:35.012 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020514799999999999 2020-01-16T00:14:35.012 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.013 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.013 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:35.013 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256385+0000", 2020-01-16T00:14:35.013 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.0288e-05 2020-01-16T00:14:35.013 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.013 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.014 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:35.014 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256406+0000", 2020-01-16T00:14:35.014 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7160000000000003e-06 2020-01-16T00:14:35.014 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.014 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.014 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:35.015 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256413+0000", 2020-01-16T00:14:35.015 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5293999999999998e-05 2020-01-16T00:14:35.015 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.015 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.015 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.015 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.256449+0000", 2020-01-16T00:14:35.015 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001297737 2020-01-16T00:14:35.016 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.016 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.016 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:35.016 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.257746+0000", 2020-01-16T00:14:35.016 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0079866529999999998 2020-01-16T00:14:35.016 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.017 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.017 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.017 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.265733+0000", 2020-01-16T00:14:35.017 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4028000000000004e-05 2020-01-16T00:14:35.017 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.017 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.018 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:35.018 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.265797+0000", 2020-01-16T00:14:35.018 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011338899999999999 2020-01-16T00:14:35.018 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.018 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.018 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:35.019 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.265910+0000", 2020-01-16T00:14:35.019 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003372496 2020-01-16T00:14:35.019 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.019 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.019 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:35.020 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.269283+0000", 2020-01-16T00:14:35.020 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0053107959999999996 2020-01-16T00:14:35.020 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.020 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.020 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.020 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.274594+0000", 2020-01-16T00:14:35.021 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8000999999999998e-05 2020-01-16T00:14:35.021 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.021 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.021 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:35.021 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.274632+0000", 2020-01-16T00:14:35.021 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0265e-05 2020-01-16T00:14:35.021 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.022 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.022 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:35.022 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:17.274692+0000", 2020-01-16T00:14:35.022 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018509650999999998 2020-01-16T00:14:35.022 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.022 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.023 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.023 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.023 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.023 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:2973 3.c 3:3025546b:::smithi12313934-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e42)", 2020-01-16T00:14:35.023 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:21.839961+0000", 2020-01-16T00:14:35.023 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 73.104040694000005, 2020-01-16T00:14:35.024 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018691670000000001, 2020-01-16T00:14:35.024 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:35.024 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:35.024 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:35.024 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:35.024 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:35.024 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 2973 2020-01-16T00:14:35.025 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.025 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:35.025 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.025 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:35.025 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839961+0000", 2020-01-16T00:14:35.025 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:35.026 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.026 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.026 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:35.026 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839961+0000", 2020-01-16T00:14:35.026 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:35.026 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.027 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.027 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:35.027 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839960+0000", 2020-01-16T00:14:35.027 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.4060000000000001e-06 2020-01-16T00:14:35.027 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.027 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.028 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:35.028 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839964+0000", 2020-01-16T00:14:35.028 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5412000000000001e-05 2020-01-16T00:14:35.028 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.028 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.028 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:35.028 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839979+0000", 2020-01-16T00:14:35.029 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.444e-06 2020-01-16T00:14:35.029 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.029 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.029 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:35.029 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.839983+0000", 2020-01-16T00:14:35.029 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000430028 2020-01-16T00:14:35.030 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.030 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.030 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.030 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.840413+0000", 2020-01-16T00:14:35.030 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9270999999999997e-05 2020-01-16T00:14:35.030 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.030 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.031 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:35.031 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.840452+0000", 2020-01-16T00:14:35.031 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014853975 2020-01-16T00:14:35.031 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.031 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.031 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.031 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.855306+0000", 2020-01-16T00:14:35.032 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011098400000000001 2020-01-16T00:14:35.032 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.032 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.032 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:35.032 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.855417+0000", 2020-01-16T00:14:35.032 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029108060000000002 2020-01-16T00:14:35.032 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.033 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.033 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.033 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858328+0000", 2020-01-16T00:14:35.033 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000144697 2020-01-16T00:14:35.033 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.033 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.034 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:35.034 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858472+0000", 2020-01-16T00:14:35.034 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018011899999999999 2020-01-16T00:14:35.034 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.034 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.034 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:35.034 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:21.858653+0000", 2020-01-16T00:14:35.035 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018691670000000001 2020-01-16T00:14:35.035 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.035 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.035 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.035 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.035 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.036 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3296 3.2 3:42b7f20f:::smithi12313934-42:head [setxattr _header (56) in=7b,truncate 2739607] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:14:35.036 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:30.744532+0000", 2020-01-16T00:14:35.036 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 64.199469684999997, 2020-01-16T00:14:35.036 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018472374, 2020-01-16T00:14:35.036 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:35.036 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:35.036 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:35.037 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:35.037 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:35.037 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3296 2020-01-16T00:14:35.037 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.037 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:35.037 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.038 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:35.038 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744532+0000", 2020-01-16T00:14:35.038 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:35.038 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.038 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.038 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:35.039 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744532+0000", 2020-01-16T00:14:35.039 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:35.039 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.039 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.039 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:35.039 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744531+0000", 2020-01-16T00:14:35.039 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7059999999999993e-06 2020-01-16T00:14:35.040 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.040 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.040 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:35.040 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744540+0000", 2020-01-16T00:14:35.040 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8589999999999998e-06 2020-01-16T00:14:35.040 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.040 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.041 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:35.041 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744544+0000", 2020-01-16T00:14:35.041 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7609999999999999e-06 2020-01-16T00:14:35.041 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.041 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.041 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:35.042 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744546+0000", 2020-01-16T00:14:35.042 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039933670000000003 2020-01-16T00:14:35.042 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.042 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.042 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.042 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748540+0000", 2020-01-16T00:14:35.043 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000110281 2020-01-16T00:14:35.043 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.043 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.043 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:35.043 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748650+0000", 2020-01-16T00:14:35.043 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.01160457 2020-01-16T00:14:35.043 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.044 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.044 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.044 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.760255+0000", 2020-01-16T00:14:35.044 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000124714 2020-01-16T00:14:35.044 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.044 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.044 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:35.045 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.760379+0000", 2020-01-16T00:14:35.045 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022099700000000001 2020-01-16T00:14:35.045 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.045 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.045 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:35.045 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.760600+0000", 2020-01-16T00:14:35.045 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00057843300000000005 2020-01-16T00:14:35.046 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.046 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.046 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:35.046 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.761179+0000", 2020-01-16T00:14:35.046 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00161477 2020-01-16T00:14:35.046 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.047 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.047 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.047 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.762794+0000", 2020-01-16T00:14:35.047 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4408999999999999e-05 2020-01-16T00:14:35.047 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.047 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.048 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:35.048 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.762858+0000", 2020-01-16T00:14:35.048 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014634199999999999 2020-01-16T00:14:35.048 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.048 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.048 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:35.048 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.763004+0000", 2020-01-16T00:14:35.049 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018472374 2020-01-16T00:14:35.049 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.049 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.049 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.049 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.049 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.050 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3297 3.2 3:42b7f20f:::smithi12313934-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e45)", 2020-01-16T00:14:35.050 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:30.744551+0000", 2020-01-16T00:14:35.050 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 64.199450714999998, 2020-01-16T00:14:35.050 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018660209000000001, 2020-01-16T00:14:35.050 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:35.050 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:35.051 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:35.051 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:35.051 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:35.051 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3297 2020-01-16T00:14:35.051 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.052 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:35.052 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.052 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:35.052 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744551+0000", 2020-01-16T00:14:35.052 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:35.052 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.052 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.053 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:35.053 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744551+0000", 2020-01-16T00:14:35.053 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:35.053 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.053 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.053 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:35.053 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744550+0000", 2020-01-16T00:14:35.054 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5279999999999999e-06 2020-01-16T00:14:35.054 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.054 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.054 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:35.054 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744554+0000", 2020-01-16T00:14:35.054 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8420000000000001e-06 2020-01-16T00:14:35.055 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.055 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.055 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:35.055 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744559+0000", 2020-01-16T00:14:35.055 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.965e-06 2020-01-16T00:14:35.055 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.056 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.056 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:35.056 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.744562+0000", 2020-01-16T00:14:35.056 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041215380000000001 2020-01-16T00:14:35.056 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.057 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.057 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.057 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748684+0000", 2020-01-16T00:14:35.057 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8165999999999997e-05 2020-01-16T00:14:35.057 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.057 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.057 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:35.058 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.748762+0000", 2020-01-16T00:14:35.058 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012136152000000001 2020-01-16T00:14:35.058 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.058 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.058 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.058 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.760898+0000", 2020-01-16T00:14:35.059 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000114833 2020-01-16T00:14:35.059 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.059 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.059 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:35.059 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.761013+0000", 2020-01-16T00:14:35.060 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002017015 2020-01-16T00:14:35.060 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.060 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.060 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.060 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.763030+0000", 2020-01-16T00:14:35.060 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5741000000000006e-05 2020-01-16T00:14:35.060 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.061 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.061 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:35.061 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.763096+0000", 2020-01-16T00:14:35.061 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011519900000000001 2020-01-16T00:14:35.061 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.061 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.062 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:35.062 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:30.763211+0000", 2020-01-16T00:14:35.062 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018660209000000001 2020-01-16T00:14:35.062 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.062 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.062 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.063 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.063 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.063 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3810 3.7 3:efab040c:::smithi12313934-30:head [write 1837406~542493] snapc 0=[] ondisk+write+known_if_redirected e50)", 2020-01-16T00:14:35.063 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:43.768525+0000", 2020-01-16T00:14:35.063 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 51.175476541999998, 2020-01-16T00:14:35.063 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018385839000000001, 2020-01-16T00:14:35.063 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:35.064 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:35.064 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:35.064 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:35.064 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:35.064 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3810 2020-01-16T00:14:35.064 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.064 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:35.065 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.065 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:35.065 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768525+0000", 2020-01-16T00:14:35.065 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:35.065 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.065 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.066 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:35.066 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768525+0000", 2020-01-16T00:14:35.066 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:35.066 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.066 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.066 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:35.067 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768523+0000", 2020-01-16T00:14:35.067 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000149512 2020-01-16T00:14:35.067 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.067 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.067 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:35.067 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768673+0000", 2020-01-16T00:14:35.067 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0732e-05 2020-01-16T00:14:35.068 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.068 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.068 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:35.068 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768684+0000", 2020-01-16T00:14:35.068 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3719999999999998e-06 2020-01-16T00:14:35.068 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.069 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.069 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:35.069 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768688+0000", 2020-01-16T00:14:35.069 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6513999999999994e-05 2020-01-16T00:14:35.069 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.070 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.070 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.070 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.768754+0000", 2020-01-16T00:14:35.070 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00075867800000000004 2020-01-16T00:14:35.070 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.070 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.071 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:35.071 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.769513+0000", 2020-01-16T00:14:35.071 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012395692 2020-01-16T00:14:35.071 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.071 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.071 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.071 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.781909+0000", 2020-01-16T00:14:35.072 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118372 2020-01-16T00:14:35.072 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.072 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.072 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:35.072 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.782027+0000", 2020-01-16T00:14:35.072 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019909699999999999 2020-01-16T00:14:35.072 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.073 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.073 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,7", 2020-01-16T00:14:35.073 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.782226+0000", 2020-01-16T00:14:35.073 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0038778369999999999 2020-01-16T00:14:35.073 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.074 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.074 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:35.074 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786104+0000", 2020-01-16T00:14:35.074 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022391199999999999 2020-01-16T00:14:35.074 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.074 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.075 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.075 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786328+0000", 2020-01-16T00:14:35.075 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00048851299999999999 2020-01-16T00:14:35.075 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.075 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.076 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.076 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786817+0000", 2020-01-16T00:14:35.076 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3597999999999998e-05 2020-01-16T00:14:35.076 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.076 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.076 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:35.077 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786860+0000", 2020-01-16T00:14:35.077 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0741000000000001e-05 2020-01-16T00:14:35.077 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.077 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.078 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:35.078 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.786911+0000", 2020-01-16T00:14:35.078 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018385839000000001 2020-01-16T00:14:35.078 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.078 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.078 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.079 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.079 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.079 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3863 3.2 3:4f8fe446:::smithi12313934-48:head [setxattr _header (56) in=7b,truncate 2151395] snapc 0=[] ondisk+write+known_if_redirected e50)", 2020-01-16T00:14:35.079 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:44.489284+0000", 2020-01-16T00:14:35.079 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 50.454718069999998, 2020-01-16T00:14:35.079 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.026323742000000001, 2020-01-16T00:14:35.079 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:35.080 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:35.080 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:35.080 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:35.080 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:35.080 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3863 2020-01-16T00:14:35.080 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.081 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:35.081 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.081 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:35.081 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489284+0000", 2020-01-16T00:14:35.081 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:35.081 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.082 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.082 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:35.082 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489284+0000", 2020-01-16T00:14:35.082 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:35.082 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.082 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.083 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:35.083 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489282+0000", 2020-01-16T00:14:35.083 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6660000000000002e-06 2020-01-16T00:14:35.083 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.083 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.083 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:35.083 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489289+0000", 2020-01-16T00:14:35.084 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3069999999999999e-06 2020-01-16T00:14:35.084 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.084 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.084 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:35.084 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489295+0000", 2020-01-16T00:14:35.084 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6090000000000003e-06 2020-01-16T00:14:35.084 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.085 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.085 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:35.085 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.489300+0000", 2020-01-16T00:14:35.085 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002242464 2020-01-16T00:14:35.085 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.085 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.086 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.086 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491542+0000", 2020-01-16T00:14:35.086 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00059546200000000005 2020-01-16T00:14:35.086 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.086 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.086 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:35.086 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.492138+0000", 2020-01-16T00:14:35.087 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018547999999999999 2020-01-16T00:14:35.087 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.087 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.087 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.087 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.510686+0000", 2020-01-16T00:14:35.087 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018590100000000001 2020-01-16T00:14:35.087 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.088 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.088 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:35.088 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.510872+0000", 2020-01-16T00:14:35.088 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00035115299999999999 2020-01-16T00:14:35.088 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.088 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.089 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:35.089 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.511223+0000", 2020-01-16T00:14:35.089 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003765493 2020-01-16T00:14:35.089 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.089 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.089 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.090 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.514988+0000", 2020-01-16T00:14:35.090 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000432022 2020-01-16T00:14:35.090 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.090 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.090 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:35.090 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515421+0000", 2020-01-16T00:14:35.090 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2874000000000002e-05 2020-01-16T00:14:35.091 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.091 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.091 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:35.091 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515473+0000", 2020-01-16T00:14:35.091 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013385 2020-01-16T00:14:35.091 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.092 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.092 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:35.092 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515607+0000", 2020-01-16T00:14:35.092 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.026323742000000001 2020-01-16T00:14:35.092 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.092 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.092 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.093 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.093 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.093 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3864 3.2 3:4f8fe446:::smithi12313934-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e50)", 2020-01-16T00:14:35.093 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:44.491131+0000", 2020-01-16T00:14:35.093 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 50.452870672000003, 2020-01-16T00:14:35.093 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024813806000000001, 2020-01-16T00:14:35.093 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:35.094 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:35.094 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:35.094 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:35.094 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:35.094 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3864 2020-01-16T00:14:35.095 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.095 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:35.095 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.095 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:35.095 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491131+0000", 2020-01-16T00:14:35.095 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:35.096 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.096 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.096 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:35.096 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491131+0000", 2020-01-16T00:14:35.096 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:35.096 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.097 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.097 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:35.097 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491128+0000", 2020-01-16T00:14:35.097 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2053e-05 2020-01-16T00:14:35.097 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.097 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.098 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:35.098 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491140+0000", 2020-01-16T00:14:35.098 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5869999999999999e-05 2020-01-16T00:14:35.098 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.098 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.098 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:35.098 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491156+0000", 2020-01-16T00:14:35.099 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.2571e-05 2020-01-16T00:14:35.099 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.099 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.099 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:35.099 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.491168+0000", 2020-01-16T00:14:35.100 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001031045 2020-01-16T00:14:35.100 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.100 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.100 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.100 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.492199+0000", 2020-01-16T00:14:35.100 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000170865 2020-01-16T00:14:35.100 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.101 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.101 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:35.101 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.492370+0000", 2020-01-16T00:14:35.101 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020180638000000001 2020-01-16T00:14:35.101 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.102 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.102 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.102 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.512551+0000", 2020-01-16T00:14:35.102 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012478599999999999 2020-01-16T00:14:35.102 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.102 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.102 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:35.103 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.512676+0000", 2020-01-16T00:14:35.103 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029628660000000002 2020-01-16T00:14:35.103 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.103 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.103 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.103 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515639+0000", 2020-01-16T00:14:35.103 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012250799999999999 2020-01-16T00:14:35.104 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.104 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.104 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:35.104 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515761+0000", 2020-01-16T00:14:35.104 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018356399999999999 2020-01-16T00:14:35.104 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.105 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.105 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:35.105 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515945+0000", 2020-01-16T00:14:35.105 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.024813806000000001 2020-01-16T00:14:35.105 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.105 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.105 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.106 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.106 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.106 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:3874 3.c 3:378f3722:::smithi12313934-50:head [write 568202~701253,stat] snapc 0=[] ondisk+write+known_if_redirected e50)", 2020-01-16T00:14:35.106 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:44.497041+0000", 2020-01-16T00:14:35.106 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 50.446961041000002, 2020-01-16T00:14:35.106 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019602525999999999, 2020-01-16T00:14:35.107 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:35.107 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:35.107 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:35.107 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:35.107 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:35.108 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 3874 2020-01-16T00:14:35.108 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.108 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:35.108 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.108 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:35.108 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497041+0000", 2020-01-16T00:14:35.108 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:35.109 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.109 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.109 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:35.109 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497041+0000", 2020-01-16T00:14:35.109 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999962 2020-01-16T00:14:35.109 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.110 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.110 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:35.110 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497037+0000", 2020-01-16T00:14:35.110 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00027168899999999998 2020-01-16T00:14:35.110 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.110 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.110 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:35.111 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497308+0000", 2020-01-16T00:14:35.111 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4202999999999999e-05 2020-01-16T00:14:35.111 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.111 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.111 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:35.111 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497323+0000", 2020-01-16T00:14:35.111 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0280000000000003e-06 2020-01-16T00:14:35.112 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.112 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.112 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:35.112 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.497330+0000", 2020-01-16T00:14:35.112 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010681352 2020-01-16T00:14:35.112 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.112 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.113 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.113 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.508011+0000", 2020-01-16T00:14:35.113 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5886999999999999e-05 2020-01-16T00:14:35.113 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.113 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.113 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:35.114 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.508097+0000", 2020-01-16T00:14:35.114 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00023353600000000001 2020-01-16T00:14:35.114 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.114 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.114 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:35.114 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.508331+0000", 2020-01-16T00:14:35.114 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0075218339999999998 2020-01-16T00:14:35.115 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.115 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.115 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:35.115 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.515852+0000", 2020-01-16T00:14:35.115 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00064178599999999998 2020-01-16T00:14:35.115 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.115 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.116 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.116 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.516494+0000", 2020-01-16T00:14:35.116 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6712999999999999e-05 2020-01-16T00:14:35.116 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.116 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.116 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:35.116 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.516531+0000", 2020-01-16T00:14:35.117 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112225 2020-01-16T00:14:35.117 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.117 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.117 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:35.117 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:44.516643+0000", 2020-01-16T00:14:35.117 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.019602525999999999 2020-01-16T00:14:35.118 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.118 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.118 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.118 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.118 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.118 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4655 3.c 3:378f3722:::smithi12313934-50:head [write 1684786~315379] snapc 0=[] ondisk+write+known_if_redirected e57)", 2020-01-16T00:14:35.119 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:04.403811+0000", 2020-01-16T00:14:35.119 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 30.540190555999999, 2020-01-16T00:14:35.119 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022055851000000001, 2020-01-16T00:14:35.119 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:35.119 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:35.119 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:35.119 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:35.120 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:35.120 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4655 2020-01-16T00:14:35.120 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.120 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:35.121 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.121 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:35.121 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.403811+0000", 2020-01-16T00:14:35.121 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:35.121 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.121 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.121 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:35.122 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.403811+0000", 2020-01-16T00:14:35.122 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:35.122 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.122 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.122 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:35.122 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.403809+0000", 2020-01-16T00:14:35.122 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012429100000000001 2020-01-16T00:14:35.123 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.123 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.123 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:35.123 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.403934+0000", 2020-01-16T00:14:35.123 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.3743e-05 2020-01-16T00:14:35.123 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.123 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.124 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:35.124 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.403947+0000", 2020-01-16T00:14:35.124 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.267e-06 2020-01-16T00:14:35.124 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.124 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.124 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:35.124 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.403955+0000", 2020-01-16T00:14:35.125 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1935999999999999e-05 2020-01-16T00:14:35.125 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.125 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.125 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.125 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404006+0000", 2020-01-16T00:14:35.125 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0060147769999999998 2020-01-16T00:14:35.125 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.126 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.126 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:35.126 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.410021+0000", 2020-01-16T00:14:35.126 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012889367000000001 2020-01-16T00:14:35.126 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.126 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.127 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.127 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.422911+0000", 2020-01-16T00:14:35.127 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012536700000000001 2020-01-16T00:14:35.127 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.127 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.127 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:35.127 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.423036+0000", 2020-01-16T00:14:35.128 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021238999999999999 2020-01-16T00:14:35.128 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.128 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.128 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:35.128 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.423248+0000", 2020-01-16T00:14:35.128 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0024175289999999999 2020-01-16T00:14:35.128 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.129 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.129 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:35.129 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.425666+0000", 2020-01-16T00:14:35.129 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7323e-05 2020-01-16T00:14:35.129 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.129 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.130 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.130 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.425723+0000", 2020-01-16T00:14:35.130 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0198999999999997e-05 2020-01-16T00:14:35.131 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.131 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.131 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:35.131 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.425783+0000", 2020-01-16T00:14:35.131 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3456999999999997e-05 2020-01-16T00:14:35.131 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.131 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.132 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:35.132 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.425867+0000", 2020-01-16T00:14:35.132 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022055851000000001 2020-01-16T00:14:35.132 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.132 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.132 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.133 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.133 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.133 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4656 3.c 3:378f3722:::smithi12313934-50:head [setxattr _header (56) in=7b,truncate 2000165] snapc 0=[] ondisk+write+known_if_redirected e57)", 2020-01-16T00:14:35.133 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:04.404295+0000", 2020-01-16T00:14:35.133 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 30.539706417000001, 2020-01-16T00:14:35.133 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021950859, 2020-01-16T00:14:35.134 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:35.134 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:35.134 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:35.134 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:35.134 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:35.134 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4656 2020-01-16T00:14:35.134 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.135 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:35.135 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.135 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:35.135 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404295+0000", 2020-01-16T00:14:35.135 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:35.135 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.135 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.136 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:35.136 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404295+0000", 2020-01-16T00:14:35.136 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:35.136 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.136 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.136 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:35.136 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404294+0000", 2020-01-16T00:14:35.137 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6139999999999997e-06 2020-01-16T00:14:35.137 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.137 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.137 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:35.137 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404301+0000", 2020-01-16T00:14:35.137 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.1420000000000001e-05 2020-01-16T00:14:35.138 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.138 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.138 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:35.138 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404312+0000", 2020-01-16T00:14:35.138 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.4840000000000003e-06 2020-01-16T00:14:35.138 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.138 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.139 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:35.139 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404318+0000", 2020-01-16T00:14:35.139 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0058040970000000002 2020-01-16T00:14:35.139 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.139 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.139 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.139 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.410122+0000", 2020-01-16T00:14:35.140 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000168297 2020-01-16T00:14:35.140 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.140 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.140 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:35.140 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.410290+0000", 2020-01-16T00:14:35.140 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013892204999999999 2020-01-16T00:14:35.141 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.141 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.141 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.141 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.424182+0000", 2020-01-16T00:14:35.141 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.6482000000000002e-05 2020-01-16T00:14:35.141 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.142 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.142 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:35.142 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.424279+0000", 2020-01-16T00:14:35.142 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000130025 2020-01-16T00:14:35.142 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.142 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.143 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:35.143 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.424409+0000", 2020-01-16T00:14:35.143 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00151793 2020-01-16T00:14:35.143 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.143 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.143 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:35.143 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.425927+0000", 2020-01-16T00:14:35.144 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020657800000000001 2020-01-16T00:14:35.144 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.144 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.144 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.144 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.426133+0000", 2020-01-16T00:14:35.144 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3606999999999999e-05 2020-01-16T00:14:35.144 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.145 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.145 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:35.145 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.426167+0000", 2020-01-16T00:14:35.145 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9117999999999997e-05 2020-01-16T00:14:35.146 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.146 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.146 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:35.146 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.426246+0000", 2020-01-16T00:14:35.146 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021950859 2020-01-16T00:14:35.146 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.147 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.147 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.147 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.147 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.147 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4657 3.c 3:378f3722:::smithi12313934-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e57)", 2020-01-16T00:14:35.147 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:04.404326+0000", 2020-01-16T00:14:35.147 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 30.53967527, 2020-01-16T00:14:35.148 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022101092999999999, 2020-01-16T00:14:35.148 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:35.148 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:35.148 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:35.148 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:35.148 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:35.148 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4657 2020-01-16T00:14:35.149 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.149 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:35.149 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.149 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:35.149 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404326+0000", 2020-01-16T00:14:35.149 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:35.149 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.150 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.150 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:35.150 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404326+0000", 2020-01-16T00:14:35.150 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:35.150 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.150 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.150 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:35.151 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404326+0000", 2020-01-16T00:14:35.151 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.771e-06 2020-01-16T00:14:35.151 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.151 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.151 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:35.152 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404329+0000", 2020-01-16T00:14:35.152 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9909999999999999e-06 2020-01-16T00:14:35.152 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.152 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.152 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:35.153 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404335+0000", 2020-01-16T00:14:35.153 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8870000000000002e-06 2020-01-16T00:14:35.153 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.153 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.153 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:35.154 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.404339+0000", 2020-01-16T00:14:35.154 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0066816460000000003 2020-01-16T00:14:35.154 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.154 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.154 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.154 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.411021+0000", 2020-01-16T00:14:35.154 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3233000000000007e-05 2020-01-16T00:14:35.155 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.155 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.155 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:35.155 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.411114+0000", 2020-01-16T00:14:35.155 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.014244643 2020-01-16T00:14:35.155 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.155 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.155 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.156 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.425359+0000", 2020-01-16T00:14:35.156 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012771200000000001 2020-01-16T00:14:35.156 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.156 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.156 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:35.156 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.425487+0000", 2020-01-16T00:14:35.156 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00078975699999999998 2020-01-16T00:14:35.156 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.157 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.157 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.157 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.426276+0000", 2020-01-16T00:14:35.157 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8829999999999997e-05 2020-01-16T00:14:35.157 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.157 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.157 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:35.157 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.426335+0000", 2020-01-16T00:14:35.158 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2219999999999995e-05 2020-01-16T00:14:35.158 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.158 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.158 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:35.158 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:04.426427+0000", 2020-01-16T00:14:35.158 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.022101092999999999 2020-01-16T00:14:35.158 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.158 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.159 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.159 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.159 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.159 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4690 3.2 3:4f8fe446:::smithi12313934-48:head [setxattr _header (56) in=7b,truncate 2067393] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-01-16T00:14:35.159 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:06.996624+0000", 2020-01-16T00:14:35.160 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 27.947378036, 2020-01-16T00:14:35.160 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021459626999999998, 2020-01-16T00:14:35.160 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:35.160 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:35.160 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:35.160 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:35.160 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:35.161 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4690 2020-01-16T00:14:35.161 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.161 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:35.161 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.161 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:35.161 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996624+0000", 2020-01-16T00:14:35.161 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:35.162 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.162 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.162 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:35.162 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996624+0000", 2020-01-16T00:14:35.162 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:35.162 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.162 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.163 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:35.163 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996623+0000", 2020-01-16T00:14:35.163 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.549e-06 2020-01-16T00:14:35.163 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.163 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.163 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:35.163 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996627+0000", 2020-01-16T00:14:35.163 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4501999999999999e-05 2020-01-16T00:14:35.164 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.164 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.164 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:35.164 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996642+0000", 2020-01-16T00:14:35.164 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1139999999999997e-06 2020-01-16T00:14:35.164 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.164 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.164 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:35.165 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996647+0000", 2020-01-16T00:14:35.165 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001305489 2020-01-16T00:14:35.165 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.165 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.165 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.165 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.997952+0000", 2020-01-16T00:14:35.165 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0036299430000000001 2020-01-16T00:14:35.166 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.166 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.166 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:35.166 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.001582+0000", 2020-01-16T00:14:35.166 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011633755000000001 2020-01-16T00:14:35.166 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.166 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.166 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.167 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.013216+0000", 2020-01-16T00:14:35.167 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00021714 2020-01-16T00:14:35.167 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.167 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.168 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:35.168 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.013433+0000", 2020-01-16T00:14:35.168 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00050686199999999996 2020-01-16T00:14:35.168 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.168 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.168 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:35.168 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.013940+0000", 2020-01-16T00:14:35.168 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039060330000000002 2020-01-16T00:14:35.169 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.169 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.169 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.169 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.017846+0000", 2020-01-16T00:14:35.169 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.7609e-05 2020-01-16T00:14:35.169 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.170 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.170 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:35.170 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.017864+0000", 2020-01-16T00:14:35.170 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.3856999999999994e-05 2020-01-16T00:14:35.170 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.170 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.171 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:35.171 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.017937+0000", 2020-01-16T00:14:35.171 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014570800000000001 2020-01-16T00:14:35.171 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.171 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.171 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:35.171 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.018083+0000", 2020-01-16T00:14:35.171 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021459626999999998 2020-01-16T00:14:35.172 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.172 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.172 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.172 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.172 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.172 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4691 3.2 3:4f8fe446:::smithi12313934-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e58)", 2020-01-16T00:14:35.172 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:06.996654+0000", 2020-01-16T00:14:35.172 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 27.947347930999999, 2020-01-16T00:14:35.173 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021903913000000001, 2020-01-16T00:14:35.173 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:35.173 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:35.173 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:35.173 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:35.173 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:35.173 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4691 2020-01-16T00:14:35.174 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.174 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:35.174 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.174 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:35.174 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996654+0000", 2020-01-16T00:14:35.174 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:35.174 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.174 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.175 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:35.175 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996654+0000", 2020-01-16T00:14:35.175 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:35.175 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.175 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.175 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:35.175 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996653+0000", 2020-01-16T00:14:35.176 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0939999999999999e-06 2020-01-16T00:14:35.176 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.176 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.176 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:35.176 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996656+0000", 2020-01-16T00:14:35.176 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.7609999999999996e-06 2020-01-16T00:14:35.176 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.176 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.177 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:35.177 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996665+0000", 2020-01-16T00:14:35.177 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.3589999999999999e-06 2020-01-16T00:14:35.177 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.177 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.177 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:35.177 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:06.996668+0000", 2020-01-16T00:14:35.178 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0049635909999999998 2020-01-16T00:14:35.178 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.178 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.178 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.178 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.001632+0000", 2020-01-16T00:14:35.178 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9422000000000001e-05 2020-01-16T00:14:35.178 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.178 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.179 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:35.179 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.001691+0000", 2020-01-16T00:14:35.179 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013157973 2020-01-16T00:14:35.180 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.180 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.180 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.180 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.014849+0000", 2020-01-16T00:14:35.180 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00027545199999999998 2020-01-16T00:14:35.180 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.181 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.181 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:35.181 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.015125+0000", 2020-01-16T00:14:35.181 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030160159999999998 2020-01-16T00:14:35.181 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.181 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.181 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.181 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.018141+0000", 2020-01-16T00:14:35.182 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00015583 2020-01-16T00:14:35.182 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.182 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.182 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:35.182 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.018297+0000", 2020-01-16T00:14:35.182 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00026096000000000002 2020-01-16T00:14:35.182 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.183 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.183 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:35.183 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.018558+0000", 2020-01-16T00:14:35.183 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.021903913000000001 2020-01-16T00:14:35.183 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.183 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.183 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.184 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.184 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.184 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4703 3.2 3:42b7f20f:::smithi12313934-42:head [write 1635678~465329] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-01-16T00:14:35.184 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.020171+0000", 2020-01-16T00:14:35.184 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 27.923830722999998, 2020-01-16T00:14:35.184 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020016459, 2020-01-16T00:14:35.184 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:35.184 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:35.185 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:35.185 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:35.185 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:35.185 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4703 2020-01-16T00:14:35.185 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.185 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:35.185 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.186 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:35.186 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020171+0000", 2020-01-16T00:14:35.186 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:35.186 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.186 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.186 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:35.186 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020171+0000", 2020-01-16T00:14:35.186 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:35.187 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.187 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.187 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:35.188 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020169+0000", 2020-01-16T00:14:35.188 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013985399999999999 2020-01-16T00:14:35.188 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.188 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.188 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:35.188 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020309+0000", 2020-01-16T00:14:35.188 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9029999999999999e-06 2020-01-16T00:14:35.189 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.189 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.189 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:35.189 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020319+0000", 2020-01-16T00:14:35.189 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9589999999999997e-06 2020-01-16T00:14:35.190 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.190 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.190 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:35.190 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020323+0000", 2020-01-16T00:14:35.190 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8680000000000002e-05 2020-01-16T00:14:35.190 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.190 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.190 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.191 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020362+0000", 2020-01-16T00:14:35.191 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029188460000000001 2020-01-16T00:14:35.191 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.191 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.191 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:35.191 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023281+0000", 2020-01-16T00:14:35.191 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011810522 2020-01-16T00:14:35.192 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.192 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.192 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.192 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.035091+0000", 2020-01-16T00:14:35.192 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000126148 2020-01-16T00:14:35.192 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.192 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.192 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:35.193 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.035217+0000", 2020-01-16T00:14:35.193 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019636799999999999 2020-01-16T00:14:35.193 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.193 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.193 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:35.193 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.035414+0000", 2020-01-16T00:14:35.193 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003550919 2020-01-16T00:14:35.194 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.194 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.194 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:35.194 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.038965+0000", 2020-01-16T00:14:35.194 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001007441 2020-01-16T00:14:35.194 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.194 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.195 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.195 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.039972+0000", 2020-01-16T00:14:35.195 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1954999999999998e-05 2020-01-16T00:14:35.195 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.195 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.195 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:35.196 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.040044+0000", 2020-01-16T00:14:35.196 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014325899999999999 2020-01-16T00:14:35.196 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.196 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.196 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:35.196 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.040187+0000", 2020-01-16T00:14:35.196 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.020016459 2020-01-16T00:14:35.197 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.197 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.197 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.197 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.197 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.197 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4704 3.2 3:42b7f20f:::smithi12313934-42:head [setxattr _header (56) in=7b,truncate 2101007] snapc 0=[] ondisk+write+known_if_redirected e58)", 2020-01-16T00:14:35.197 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.020335+0000", 2020-01-16T00:14:35.198 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 27.923666912000002, 2020-01-16T00:14:35.198 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023082354999999999, 2020-01-16T00:14:35.198 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:35.198 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:35.198 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:35.198 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:35.198 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:35.199 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4704 2020-01-16T00:14:35.199 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.199 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:35.199 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.199 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:35.199 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020335+0000", 2020-01-16T00:14:35.199 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:35.200 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.200 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.200 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:35.200 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020335+0000", 2020-01-16T00:14:35.200 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:35.200 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.201 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.201 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:35.201 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020334+0000", 2020-01-16T00:14:35.201 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.7780000000000002e-06 2020-01-16T00:14:35.201 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.201 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.201 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:35.202 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020341+0000", 2020-01-16T00:14:35.202 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2039999999999996e-06 2020-01-16T00:14:35.202 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.202 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.202 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:35.202 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020346+0000", 2020-01-16T00:14:35.202 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8800000000000008e-06 2020-01-16T00:14:35.203 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.203 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.203 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:35.203 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020354+0000", 2020-01-16T00:14:35.203 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029530490000000001 2020-01-16T00:14:35.203 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.203 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.203 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.204 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023307+0000", 2020-01-16T00:14:35.204 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8681000000000003e-05 2020-01-16T00:14:35.204 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.204 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.204 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:35.204 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023365+0000", 2020-01-16T00:14:35.204 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013386927999999999 2020-01-16T00:14:35.205 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.205 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.205 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.205 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.036752+0000", 2020-01-16T00:14:35.205 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8906000000000007e-05 2020-01-16T00:14:35.205 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.205 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.206 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:35.206 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.036831+0000", 2020-01-16T00:14:35.206 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000142922 2020-01-16T00:14:35.206 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.206 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.206 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:35.206 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.036974+0000", 2020-01-16T00:14:35.207 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0038357280000000001 2020-01-16T00:14:35.207 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.207 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.207 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:35.207 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.040810+0000", 2020-01-16T00:14:35.207 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0024973949999999999 2020-01-16T00:14:35.207 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.208 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.208 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.208 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.043307+0000", 2020-01-16T00:14:35.208 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8513000000000002e-05 2020-01-16T00:14:35.208 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.208 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.208 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:35.208 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.043346+0000", 2020-01-16T00:14:35.209 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.127e-05 2020-01-16T00:14:35.209 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.209 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.209 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:35.209 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.043417+0000", 2020-01-16T00:14:35.209 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023082354999999999 2020-01-16T00:14:35.209 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.209 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.210 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.210 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.210 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.210 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4705 3.2 3:42b7f20f:::smithi12313934-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e58)", 2020-01-16T00:14:35.210 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.020362+0000", 2020-01-16T00:14:35.210 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 27.923639977000001, 2020-01-16T00:14:35.210 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023262267999999999, 2020-01-16T00:14:35.211 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:35.211 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:35.211 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:35.211 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:35.211 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:35.211 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4705 2020-01-16T00:14:35.211 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.211 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:35.212 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.212 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:35.212 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020362+0000", 2020-01-16T00:14:35.212 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:35.212 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.212 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.212 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:35.212 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020362+0000", 2020-01-16T00:14:35.213 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:35.213 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.213 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.213 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:35.213 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020361+0000", 2020-01-16T00:14:35.213 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.365e-06 2020-01-16T00:14:35.213 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.214 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.214 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:35.214 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020364+0000", 2020-01-16T00:14:35.214 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5649999999999998e-06 2020-01-16T00:14:35.214 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.214 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.214 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:35.214 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020371+0000", 2020-01-16T00:14:35.215 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.6349999999999999e-06 2020-01-16T00:14:35.215 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.215 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.215 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:35.215 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.020375+0000", 2020-01-16T00:14:35.215 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030174920000000001 2020-01-16T00:14:35.215 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.216 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.216 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.216 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023392+0000", 2020-01-16T00:14:35.216 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4997000000000003e-05 2020-01-16T00:14:35.216 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.216 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.216 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:35.216 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023437+0000", 2020-01-16T00:14:35.217 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.015293378999999999 2020-01-16T00:14:35.217 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.217 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.217 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.217 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.038730+0000", 2020-01-16T00:14:35.217 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000124206 2020-01-16T00:14:35.217 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.218 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.218 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:35.218 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.038855+0000", 2020-01-16T00:14:35.218 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0045951489999999998 2020-01-16T00:14:35.218 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.218 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.218 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.218 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.043450+0000", 2020-01-16T00:14:35.219 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1363000000000001e-05 2020-01-16T00:14:35.219 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.219 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.219 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:35.219 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.043521+0000", 2020-01-16T00:14:35.219 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000102737 2020-01-16T00:14:35.219 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.220 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.220 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:35.220 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.043624+0000", 2020-01-16T00:14:35.220 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.023262267999999999 2020-01-16T00:14:35.220 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.220 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.220 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.220 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.221 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.221 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(client.4488.0:4869 3.c 3:3025546b:::smithi12313934-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e59)", 2020-01-16T00:14:35.221 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:11.931564+0000", 2020-01-16T00:14:35.221 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 23.012437093999999, 2020-01-16T00:14:35.221 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018620945, 2020-01-16T00:14:35.222 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:35.222 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "started", 2020-01-16T00:14:35.222 INFO:teuthology.orchestra.run.smithi123.stdout: "client_info": { 2020-01-16T00:14:35.222 INFO:teuthology.orchestra.run.smithi123.stdout: "client": "client.4488", 2020-01-16T00:14:35.222 INFO:teuthology.orchestra.run.smithi123.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:35.222 INFO:teuthology.orchestra.run.smithi123.stdout: "tid": 4869 2020-01-16T00:14:35.222 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.223 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:35.223 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.223 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:35.223 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.931564+0000", 2020-01-16T00:14:35.223 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:35.223 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.223 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.223 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:35.224 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.931564+0000", 2020-01-16T00:14:35.224 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:35.224 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.224 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.224 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:35.224 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.931563+0000", 2020-01-16T00:14:35.224 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8030000000000002e-06 2020-01-16T00:14:35.225 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.225 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.225 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:35.225 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.931569+0000", 2020-01-16T00:14:35.225 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.3370000000000006e-06 2020-01-16T00:14:35.225 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.225 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.226 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:35.226 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.931579+0000", 2020-01-16T00:14:35.226 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9569999999999999e-06 2020-01-16T00:14:35.226 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.226 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.226 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:35.226 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.931585+0000", 2020-01-16T00:14:35.226 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0021810430000000001 2020-01-16T00:14:35.227 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.227 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.227 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.227 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.933766+0000", 2020-01-16T00:14:35.227 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.2802000000000001e-05 2020-01-16T00:14:35.227 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.227 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.228 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:35.228 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.933808+0000", 2020-01-16T00:14:35.228 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0099913469999999994 2020-01-16T00:14:35.228 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.228 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.228 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.228 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.943800+0000", 2020-01-16T00:14:35.228 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013257000000000001 2020-01-16T00:14:35.229 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.229 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.229 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:35.229 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.943932+0000", 2020-01-16T00:14:35.229 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0058405799999999997 2020-01-16T00:14:35.229 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.229 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.229 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.230 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.949773+0000", 2020-01-16T00:14:35.230 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000158493 2020-01-16T00:14:35.230 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.230 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.230 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:35.230 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.949931+0000", 2020-01-16T00:14:35.230 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000254083 2020-01-16T00:14:35.231 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.231 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.231 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:35.231 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:11.950185+0000", 2020-01-16T00:14:35.231 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.018620945 2020-01-16T00:14:35.231 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.231 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.231 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.232 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.232 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.232 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:35.233 INFO:tasks.rados.rados.0.smithi123.stdout:1994: finishing write tid 1 to smithi12313934-12 2020-01-16T00:14:35.234 INFO:tasks.rados.rados.0.smithi123.stdout:1994: finishing write tid 2 to smithi12313934-12 2020-01-16T00:14:35.234 INFO:tasks.rados.rados.0.smithi123.stdout:1994: finishing write tid 4 to smithi12313934-12 2020-01-16T00:14:35.234 INFO:tasks.rados.rados.0.smithi123.stdout:1994: finishing write tid 5 to smithi12313934-12 2020-01-16T00:14:35.234 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 12 v 211 (ObjNum 759 snap 0 seq_num 759) dirty exists 2020-01-16T00:14:35.234 INFO:tasks.rados.rados.0.smithi123.stdout:1994: left oid 12 (ObjNum 759 snap 0 seq_num 759) 2020-01-16T00:14:35.234 INFO:tasks.rados.rados.0.smithi123.stdout:1996: finishing write tid 1 to smithi12313934-9 2020-01-16T00:14:35.235 INFO:tasks.rados.rados.0.smithi123.stdout:1996: finishing write tid 2 to smithi12313934-9 2020-01-16T00:14:35.235 INFO:tasks.rados.rados.0.smithi123.stdout:1996: finishing write tid 4 to smithi12313934-9 2020-01-16T00:14:35.235 INFO:tasks.rados.rados.0.smithi123.stdout:1996: finishing write tid 5 to smithi12313934-9 2020-01-16T00:14:35.235 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 9 v 202 (ObjNum 761 snap 0 seq_num 761) dirty exists 2020-01-16T00:14:35.235 INFO:tasks.rados.rados.0.smithi123.stdout:1996: left oid 9 (ObjNum 761 snap 0 seq_num 761) 2020-01-16T00:14:35.235 INFO:tasks.rados.rados.0.smithi123.stdout:1997: finishing write tid 1 to smithi12313934-47 2020-01-16T00:14:35.235 INFO:tasks.rados.rados.0.smithi123.stdout:1990: done (15 left) 2020-01-16T00:14:35.236 INFO:tasks.rados.rados.0.smithi123.stdout:1991: done (14 left) 2020-01-16T00:14:35.236 INFO:tasks.rados.rados.0.smithi123.stdout:1992: done (13 left) 2020-01-16T00:14:35.236 INFO:tasks.rados.rados.0.smithi123.stdout:1993: done (12 left) 2020-01-16T00:14:35.236 INFO:tasks.rados.rados.0.smithi123.stdout:1994: done (11 left) 2020-01-16T00:14:35.236 INFO:tasks.rados.rados.0.smithi123.stdout:1996: done (10 left) 2020-01-16T00:14:35.236 INFO:tasks.rados.rados.0.smithi123.stdout:2007: delete oid 24 current snap is 0 2020-01-16T00:14:35.236 INFO:tasks.rados.rados.0.smithi123.stdout:1997: finishing write tid 2 to smithi12313934-47 2020-01-16T00:14:35.237 INFO:tasks.rados.rados.0.smithi123.stdout:1997: finishing write tid 3 to smithi12313934-47 2020-01-16T00:14:35.237 INFO:tasks.rados.rados.0.smithi123.stdout:1997: finishing write tid 5 to smithi12313934-47 2020-01-16T00:14:35.237 INFO:tasks.rados.rados.0.smithi123.stdout:1997: finishing write tid 6 to smithi12313934-47 2020-01-16T00:14:35.237 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 47 v 324 (ObjNum 762 snap 0 seq_num 762) dirty exists 2020-01-16T00:14:35.237 INFO:tasks.rados.rados.0.smithi123.stdout:1997: left oid 47 (ObjNum 762 snap 0 seq_num 762) 2020-01-16T00:14:35.237 INFO:tasks.rados.rados.0.smithi123.stdout:1998: expect (ObjNum 739 snap 0 seq_num 739) 2020-01-16T00:14:35.237 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:35.230+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:35.238 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-01-16T00:14:35.334 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:35.332+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:35.337 INFO:tasks.rados.rados.0.smithi123.stdout:1999: finishing write tid 1 to smithi12313934-43 2020-01-16T00:14:35.337 INFO:tasks.rados.rados.0.smithi123.stdout:1999: finishing write tid 2 to smithi12313934-43 2020-01-16T00:14:35.337 INFO:tasks.rados.rados.0.smithi123.stdout:1999: finishing write tid 4 to smithi12313934-43 2020-01-16T00:14:35.337 INFO:tasks.rados.rados.0.smithi123.stdout:1999: finishing write tid 5 to smithi12313934-43 2020-01-16T00:14:35.337 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 43 v 321 (ObjNum 763 snap 0 seq_num 763) dirty exists 2020-01-16T00:14:35.338 INFO:tasks.rados.rados.0.smithi123.stdout:1999: left oid 43 (ObjNum 763 snap 0 seq_num 763) 2020-01-16T00:14:35.338 INFO:tasks.rados.rados.0.smithi123.stdout:2000: finishing write tid 1 to smithi12313934-48 2020-01-16T00:14:35.338 INFO:tasks.rados.rados.0.smithi123.stdout:2000: finishing write tid 2 to smithi12313934-48 2020-01-16T00:14:35.338 INFO:tasks.rados.rados.0.smithi123.stdout:2001: finishing write tid 1 to smithi12313934-22 2020-01-16T00:14:35.338 INFO:tasks.rados.rados.0.smithi123.stdout:2000: finishing write tid 3 to smithi12313934-48 2020-01-16T00:14:35.338 INFO:tasks.rados.rados.0.smithi123.stdout:2000: finishing write tid 5 to smithi12313934-48 2020-01-16T00:14:35.338 INFO:tasks.rados.rados.0.smithi123.stdout:2000: finishing write tid 6 to smithi12313934-48 2020-01-16T00:14:35.339 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 48 v 243 (ObjNum 764 snap 0 seq_num 764) dirty exists 2020-01-16T00:14:35.339 INFO:tasks.rados.rados.0.smithi123.stdout:2000: left oid 48 (ObjNum 764 snap 0 seq_num 764) 2020-01-16T00:14:35.339 INFO:tasks.rados.rados.0.smithi123.stdout:2001: finishing write tid 2 to smithi12313934-22 2020-01-16T00:14:35.339 INFO:tasks.rados.rados.0.smithi123.stdout:2001: finishing write tid 3 to smithi12313934-22 2020-01-16T00:14:35.339 INFO:tasks.rados.rados.0.smithi123.stdout:1997: done (10 left) 2020-01-16T00:14:35.339 INFO:tasks.rados.rados.0.smithi123.stdout:1998: done (9 left) 2020-01-16T00:14:35.339 INFO:tasks.rados.rados.0.smithi123.stdout:1999: done (8 left) 2020-01-16T00:14:35.340 INFO:tasks.rados.rados.0.smithi123.stdout:2000: done (7 left) 2020-01-16T00:14:35.340 INFO:tasks.rados.rados.0.smithi123.stdout:2007: done (6 left) 2020-01-16T00:14:35.340 INFO:tasks.rados.rados.0.smithi123.stdout:2008: read oid 14 snap -1 2020-01-16T00:14:35.340 INFO:tasks.rados.rados.0.smithi123.stdout:2008: expect (ObjNum 747 snap 0 seq_num 747) 2020-01-16T00:14:35.340 INFO:tasks.rados.rados.0.smithi123.stdout:2009: write (excl) oid 3 current snap is 0 2020-01-16T00:14:35.340 INFO:tasks.rados.rados.0.smithi123.stdout:2009: seq_num 771 ranges {558197=568632,1650558=632927,2773245=184952} 2020-01-16T00:14:35.351 INFO:tasks.rados.rados.0.smithi123.stdout:2009: writing smithi12313934-3 from 558197 to 1126829 tid 1 2020-01-16T00:14:35.358 INFO:tasks.rados.rados.0.smithi123.stdout:2009: writing smithi12313934-3 from 1650558 to 2283485 tid 2 2020-01-16T00:14:35.363 INFO:tasks.rados.rados.0.smithi123.stdout:2009: writing smithi12313934-3 from 2773245 to 2958197 tid 3 2020-01-16T00:14:35.363 INFO:tasks.rados.rados.0.smithi123.stdout:2010: write oid 19 current snap is 0 2020-01-16T00:14:35.363 INFO:tasks.rados.rados.0.smithi123.stdout:2010: seq_num 772 ranges {575004=643881,1709040=492938,2917021=57983} 2020-01-16T00:14:35.369 INFO:tasks.rados.rados.0.smithi123.stdout:2010: writing smithi12313934-19 from 575004 to 1218885 tid 1 2020-01-16T00:14:35.375 INFO:tasks.rados.rados.0.smithi123.stdout:2010: writing smithi12313934-19 from 1709040 to 2201978 tid 2 2020-01-16T00:14:35.380 INFO:tasks.rados.rados.0.smithi123.stdout:2010: writing smithi12313934-19 from 2917021 to 2975004 tid 3 2020-01-16T00:14:35.380 INFO:tasks.rados.rados.0.smithi123.stdout:2011: write oid 10 current snap is 0 2020-01-16T00:14:35.380 INFO:tasks.rados.rados.0.smithi123.stdout:2011: seq_num 773 ranges {591811=719130,1767522=636596,2828091=163720} 2020-01-16T00:14:35.386 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:35.386 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:14:35.387 INFO:teuthology.orchestra.run.smithi123.stdout: "num_ops": 0 2020-01-16T00:14:35.387 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:35.387 INFO:tasks.rados.rados.0.smithi123.stdout:2011: writing smithi12313934-10 from 591811 to 1310941 tid 1 2020-01-16T00:14:35.393 INFO:tasks.rados.rados.0.smithi123.stdout:2011: writing smithi12313934-10 from 1767522 to 2404118 tid 2 2020-01-16T00:14:35.396 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-01-16T00:14:35.398 INFO:tasks.rados.rados.0.smithi123.stdout:2011: writing smithi12313934-10 from 2828091 to 2991811 tid 3 2020-01-16T00:14:35.398 INFO:tasks.rados.rados.0.smithi123.stdout:2012: read oid 34 snap -1 2020-01-16T00:14:35.399 INFO:tasks.rados.rados.0.smithi123.stdout:2012: expect (ObjNum 743 snap 0 seq_num 743) 2020-01-16T00:14:35.399 INFO:tasks.rados.rados.0.smithi123.stdout:2013: delete oid 13 current snap is 0 2020-01-16T00:14:35.399 INFO:tasks.rados.rados.0.smithi123.stdout:2001: finishing write tid 5 to smithi12313934-22 2020-01-16T00:14:35.399 INFO:tasks.rados.rados.0.smithi123.stdout:2001: finishing write tid 6 to smithi12313934-22 2020-01-16T00:14:35.400 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 22 v 57 (ObjNum 765 snap 0 seq_num 765) dirty exists 2020-01-16T00:14:35.400 INFO:tasks.rados.rados.0.smithi123.stdout:2001: left oid 22 (ObjNum 765 snap 0 seq_num 765) 2020-01-16T00:14:35.400 INFO:tasks.rados.rados.0.smithi123.stdout:2002: finishing write tid 1 to smithi12313934-40 2020-01-16T00:14:35.400 INFO:tasks.rados.rados.0.smithi123.stdout:2002: finishing write tid 2 to smithi12313934-40 2020-01-16T00:14:35.400 INFO:tasks.rados.rados.0.smithi123.stdout:2002: finishing write tid 3 to smithi12313934-40 2020-01-16T00:14:35.400 INFO:tasks.rados.rados.0.smithi123.stdout:2002: finishing write tid 5 to smithi12313934-40 2020-01-16T00:14:35.401 INFO:tasks.rados.rados.0.smithi123.stdout:2002: finishing write tid 6 to smithi12313934-40 2020-01-16T00:14:35.401 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 40 v 325 (ObjNum 766 snap 0 seq_num 766) dirty exists 2020-01-16T00:14:35.401 INFO:tasks.rados.rados.0.smithi123.stdout:2002: left oid 40 (ObjNum 766 snap 0 seq_num 766) 2020-01-16T00:14:35.401 INFO:tasks.rados.rados.0.smithi123.stdout:2003: finishing write tid 1 to smithi12313934-29 2020-01-16T00:14:35.401 INFO:tasks.rados.rados.0.smithi123.stdout:2003: finishing write tid 2 to smithi12313934-29 2020-01-16T00:14:35.402 INFO:tasks.rados.rados.0.smithi123.stdout:2003: finishing write tid 4 to smithi12313934-29 2020-01-16T00:14:35.402 INFO:tasks.rados.rados.0.smithi123.stdout:2003: finishing write tid 5 to smithi12313934-29 2020-01-16T00:14:35.402 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 29 v 207 (ObjNum 767 snap 0 seq_num 767) dirty exists 2020-01-16T00:14:35.402 INFO:tasks.rados.rados.0.smithi123.stdout:2003: left oid 29 (ObjNum 767 snap 0 seq_num 767) 2020-01-16T00:14:35.402 INFO:tasks.rados.rados.0.smithi123.stdout:2004: finishing write tid 1 to smithi12313934-38 2020-01-16T00:14:35.402 INFO:tasks.rados.rados.0.smithi123.stdout:2004: finishing write tid 2 to smithi12313934-38 2020-01-16T00:14:35.402 INFO:tasks.rados.rados.0.smithi123.stdout:2004: finishing write tid 4 to smithi12313934-38 2020-01-16T00:14:35.403 INFO:tasks.rados.rados.0.smithi123.stdout:2004: finishing write tid 5 to smithi12313934-38 2020-01-16T00:14:35.403 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 38 v 246 (ObjNum 768 snap 0 seq_num 768) dirty exists 2020-01-16T00:14:35.403 INFO:tasks.rados.rados.0.smithi123.stdout:2004: left oid 38 (ObjNum 768 snap 0 seq_num 768) 2020-01-16T00:14:35.403 INFO:tasks.rados.rados.0.smithi123.stdout:2005: finishing write tid 1 to smithi12313934-33 2020-01-16T00:14:35.403 INFO:tasks.rados.rados.0.smithi123.stdout:2005: finishing write tid 2 to smithi12313934-33 2020-01-16T00:14:35.403 INFO:tasks.rados.rados.0.smithi123.stdout:2005: finishing write tid 3 to smithi12313934-33 2020-01-16T00:14:35.403 INFO:tasks.rados.rados.0.smithi123.stdout:2005: finishing write tid 5 to smithi12313934-33 2020-01-16T00:14:35.404 INFO:tasks.rados.rados.0.smithi123.stdout:2005: finishing write tid 6 to smithi12313934-33 2020-01-16T00:14:35.404 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 33 v 388 (ObjNum 769 snap 0 seq_num 769) dirty exists 2020-01-16T00:14:35.404 INFO:tasks.rados.rados.0.smithi123.stdout:2005: left oid 33 (ObjNum 769 snap 0 seq_num 769) 2020-01-16T00:14:35.404 INFO:tasks.rados.rados.0.smithi123.stdout:2006: expect (ObjNum 720 snap 0 seq_num 720) 2020-01-16T00:14:35.446 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:35.446+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:35.489 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:35.490 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [], 2020-01-16T00:14:35.490 INFO:teuthology.orchestra.run.smithi123.stdout: "complaint_time": 30, 2020-01-16T00:14:35.490 INFO:teuthology.orchestra.run.smithi123.stdout: "num_blocked_ops": 0 2020-01-16T00:14:35.490 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:35.498 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-01-16T00:14:35.537 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:35.532+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:35.608 INFO:tasks.rados.rados.0.smithi123.stdout:2008: expect (ObjNum 747 snap 0 seq_num 747) 2020-01-16T00:14:35.636 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:35.632+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:35.647 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2020-01-16T00:14:35.648 INFO:teuthology.orchestra.run.smithi123.stdout: "size": 20, 2020-01-16T00:14:35.648 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 600, 2020-01-16T00:14:35.648 INFO:teuthology.orchestra.run.smithi123.stdout: "ops": [ 2020-01-16T00:14:35.648 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.648 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_repop(osd.4.0:114 3.9 e24/20 3:9fc687b2:::smithi12313934-44:head v 24'74, mlcod=24'72)", 2020-01-16T00:14:35.648 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:14:35.648 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 117.32164741, 2020-01-16T00:14:35.649 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010399429, 2020-01-16T00:14:35.649 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:35.649 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:35.649 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:35.649 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.649 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:35.649 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:14:35.649 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:35.650 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.650 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.650 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:35.650 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325279+0000", 2020-01-16T00:14:35.650 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:35.650 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.650 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.651 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:35.651 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.325276+0000", 2020-01-16T00:14:35.651 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0046127069999999997 2020-01-16T00:14:35.651 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.651 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.651 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:35.651 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329889+0000", 2020-01-16T00:14:35.651 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.3519999999999998e-05 2020-01-16T00:14:35.652 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.652 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.652 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:35.652 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329912+0000", 2020-01-16T00:14:35.652 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4070000000000004e-06 2020-01-16T00:14:35.652 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.652 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.652 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:35.653 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329920+0000", 2020-01-16T00:14:35.653 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8817000000000003e-05 2020-01-16T00:14:35.653 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.653 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.653 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.653 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329978+0000", 2020-01-16T00:14:35.653 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9721e-05 2020-01-16T00:14:35.654 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.654 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.654 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:35.654 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.329998+0000", 2020-01-16T00:14:35.654 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0056328139999999999 2020-01-16T00:14:35.654 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.654 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.655 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:35.655 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.335631+0000", 2020-01-16T00:14:35.655 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7170999999999999e-05 2020-01-16T00:14:35.655 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.655 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.655 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:35.655 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:38.335678+0000", 2020-01-16T00:14:35.655 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010399429 2020-01-16T00:14:35.656 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.656 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.656 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.656 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.656 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.656 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:55 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e30)", 2020-01-16T00:14:35.656 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.736036+0000", 2020-01-16T00:14:35.656 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 102.91089019099999, 2020-01-16T00:14:35.657 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010974536, 2020-01-16T00:14:35.657 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:35.657 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:35.657 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:35.657 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.657 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:35.657 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736036+0000", 2020-01-16T00:14:35.658 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:35.658 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.658 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.658 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:35.658 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736036+0000", 2020-01-16T00:14:35.658 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:35.658 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.658 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.659 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:35.659 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736034+0000", 2020-01-16T00:14:35.659 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043589699999999998 2020-01-16T00:14:35.659 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.659 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.659 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:35.659 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736469+0000", 2020-01-16T00:14:35.659 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9882e-05 2020-01-16T00:14:35.660 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.660 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.660 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:35.660 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736489+0000", 2020-01-16T00:14:35.660 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8339999999999999e-06 2020-01-16T00:14:35.660 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.660 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.661 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:35.661 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736497+0000", 2020-01-16T00:14:35.661 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5354999999999999e-05 2020-01-16T00:14:35.661 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.661 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.661 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.661 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736572+0000", 2020-01-16T00:14:35.661 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5228000000000004e-05 2020-01-16T00:14:35.662 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.662 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.662 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:35.662 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736668+0000", 2020-01-16T00:14:35.662 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017212999999999999 2020-01-16T00:14:35.662 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.662 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.663 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:35.663 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736840+0000", 2020-01-16T00:14:35.663 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0099057020000000006 2020-01-16T00:14:35.663 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.663 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.663 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.663 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746746+0000", 2020-01-16T00:14:35.663 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0586000000000005e-05 2020-01-16T00:14:35.664 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.664 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.664 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:35.664 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746816+0000", 2020-01-16T00:14:35.664 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2445000000000002e-05 2020-01-16T00:14:35.664 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.664 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.665 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.665 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746889+0000", 2020-01-16T00:14:35.665 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6839000000000002e-05 2020-01-16T00:14:35.665 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.665 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.665 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:35.665 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746955+0000", 2020-01-16T00:14:35.665 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.5074000000000002e-05 2020-01-16T00:14:35.666 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.666 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.666 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:35.666 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.747011+0000", 2020-01-16T00:14:35.666 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010974536 2020-01-16T00:14:35.666 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.666 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.666 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.667 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.667 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.667 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:56 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e30)", 2020-01-16T00:14:35.667 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:12:52.736514+0000", 2020-01-16T00:14:35.667 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 102.91041235599999, 2020-01-16T00:14:35.667 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011660584999999999, 2020-01-16T00:14:35.667 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:35.668 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:35.668 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:35.668 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.668 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:35.668 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736514+0000", 2020-01-16T00:14:35.668 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:35.668 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.668 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.669 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:35.669 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736514+0000", 2020-01-16T00:14:35.669 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:35.669 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.669 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.669 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:35.669 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736513+0000", 2020-01-16T00:14:35.669 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.9677000000000004e-05 2020-01-16T00:14:35.670 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.670 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.670 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:35.670 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736583+0000", 2020-01-16T00:14:35.670 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.5269999999999998e-06 2020-01-16T00:14:35.670 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.670 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.671 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:35.671 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736591+0000", 2020-01-16T00:14:35.671 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5769999999999996e-06 2020-01-16T00:14:35.671 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.671 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.671 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:35.671 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.736599+0000", 2020-01-16T00:14:35.671 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0053593310000000002 2020-01-16T00:14:35.672 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.672 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.672 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.672 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.741958+0000", 2020-01-16T00:14:35.672 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00014164099999999999 2020-01-16T00:14:35.672 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.672 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.672 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:35.673 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.742100+0000", 2020-01-16T00:14:35.673 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020126000000000001 2020-01-16T00:14:35.673 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.673 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.673 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:35.673 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.742301+0000", 2020-01-16T00:14:35.673 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0045052859999999998 2020-01-16T00:14:35.674 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.674 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.674 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.674 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.746806+0000", 2020-01-16T00:14:35.674 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00070802699999999996 2020-01-16T00:14:35.674 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.674 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.674 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:35.675 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.747514+0000", 2020-01-16T00:14:35.675 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00059247200000000003 2020-01-16T00:14:35.675 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.675 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.675 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.675 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748107+0000", 2020-01-16T00:14:35.675 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.1870999999999999e-05 2020-01-16T00:14:35.675 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.676 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.676 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:35.676 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748149+0000", 2020-01-16T00:14:35.676 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.5831000000000001e-05 2020-01-16T00:14:35.676 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.676 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.676 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:35.676 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:12:52.748174+0000", 2020-01-16T00:14:35.677 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011660584999999999 2020-01-16T00:14:35.677 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.677 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.677 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.677 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.677 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.677 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:70 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e35)", 2020-01-16T00:14:35.677 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.985494+0000", 2020-01-16T00:14:35.678 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 89.661432559999994, 2020-01-16T00:14:35.678 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013746267, 2020-01-16T00:14:35.678 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:35.678 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:35.678 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:35.678 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.678 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:35.678 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985494+0000", 2020-01-16T00:14:35.679 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:35.679 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.679 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.679 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:35.679 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985494+0000", 2020-01-16T00:14:35.679 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:35.679 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.679 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.680 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:35.680 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.985490+0000", 2020-01-16T00:14:35.680 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00077792099999999999 2020-01-16T00:14:35.680 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.680 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.680 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:35.680 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986268+0000", 2020-01-16T00:14:35.681 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9718000000000001e-05 2020-01-16T00:14:35.681 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.681 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.681 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:35.681 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986318+0000", 2020-01-16T00:14:35.681 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.4349999999999997e-06 2020-01-16T00:14:35.681 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.681 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.682 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:35.682 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986326+0000", 2020-01-16T00:14:35.682 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.3298999999999999e-05 2020-01-16T00:14:35.682 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.682 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.682 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.682 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986380+0000", 2020-01-16T00:14:35.682 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.4969999999999995e-05 2020-01-16T00:14:35.683 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.683 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.683 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:35.683 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986455+0000", 2020-01-16T00:14:35.683 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000110587 2020-01-16T00:14:35.683 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.683 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.683 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:35.684 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986565+0000", 2020-01-16T00:14:35.684 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063931600000000002 2020-01-16T00:14:35.684 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.684 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.684 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:35.684 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.992958+0000", 2020-01-16T00:14:35.684 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0013008480000000001 2020-01-16T00:14:35.684 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.685 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.685 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.685 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.994259+0000", 2020-01-16T00:14:35.685 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0048399979999999999 2020-01-16T00:14:35.685 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.685 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.685 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.686 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999099+0000", 2020-01-16T00:14:35.686 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.1874999999999998e-05 2020-01-16T00:14:35.686 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.686 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.686 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:35.686 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999181+0000", 2020-01-16T00:14:35.686 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8865e-05 2020-01-16T00:14:35.686 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.687 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.687 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:35.687 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999240+0000", 2020-01-16T00:14:35.687 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013746267 2020-01-16T00:14:35.687 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.687 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.687 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.687 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.687 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.688 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:71 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 2011648~186368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e35)", 2020-01-16T00:14:35.688 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:05.986341+0000", 2020-01-16T00:14:35.688 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 89.660585198999996, 2020-01-16T00:14:35.688 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013252807, 2020-01-16T00:14:35.688 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:35.688 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:35.688 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:35.689 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.689 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:35.689 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986341+0000", 2020-01-16T00:14:35.689 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:35.689 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.689 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.689 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:35.689 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986341+0000", 2020-01-16T00:14:35.690 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:35.690 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.690 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.690 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:35.690 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986340+0000", 2020-01-16T00:14:35.690 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000103867 2020-01-16T00:14:35.690 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.690 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.691 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:35.691 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986444+0000", 2020-01-16T00:14:35.691 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.0104e-05 2020-01-16T00:14:35.691 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.691 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.691 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:35.691 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986454+0000", 2020-01-16T00:14:35.691 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4220000000000002e-06 2020-01-16T00:14:35.692 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.692 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.692 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:35.692 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.986458+0000", 2020-01-16T00:14:35.692 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0033523070000000001 2020-01-16T00:14:35.692 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.692 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.692 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.693 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.989811+0000", 2020-01-16T00:14:35.693 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3296999999999999e-05 2020-01-16T00:14:35.693 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.693 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.693 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:35.693 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.989894+0000", 2020-01-16T00:14:35.693 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000117701 2020-01-16T00:14:35.693 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.694 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.694 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:35.694 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.990012+0000", 2020-01-16T00:14:35.694 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0032873490000000002 2020-01-16T00:14:35.694 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.694 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.694 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:35.694 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.993299+0000", 2020-01-16T00:14:35.695 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002422619 2020-01-16T00:14:35.695 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.695 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.695 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.695 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.995722+0000", 2020-01-16T00:14:35.695 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0037839589999999999 2020-01-16T00:14:35.695 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.696 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.696 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.696 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999506+0000", 2020-01-16T00:14:35.696 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8559999999999997e-05 2020-01-16T00:14:35.696 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.696 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.696 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:35.696 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999544+0000", 2020-01-16T00:14:35.697 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9599999999999999e-05 2020-01-16T00:14:35.697 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.697 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.697 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:35.697 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:05.999594+0000", 2020-01-16T00:14:35.697 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.013252807 2020-01-16T00:14:35.697 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.698 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.698 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.698 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.698 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.698 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:77 2.1 2:91f85463:::smithi12313934-24 low_tier:head [write 30720~1585664] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e36)", 2020-01-16T00:14:35.698 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:07.555854+0000", 2020-01-16T00:14:35.698 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 88.091072109999999, 2020-01-16T00:14:35.698 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010455002999999999, 2020-01-16T00:14:35.699 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:35.699 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:35.699 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:35.699 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.699 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:35.699 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555854+0000", 2020-01-16T00:14:35.699 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:35.699 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.700 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.700 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:35.700 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555854+0000", 2020-01-16T00:14:35.700 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:35.700 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.700 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.700 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:35.701 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.555851+0000", 2020-01-16T00:14:35.701 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00048310200000000001 2020-01-16T00:14:35.701 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.701 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.701 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:35.701 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556334+0000", 2020-01-16T00:14:35.701 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.2232e-05 2020-01-16T00:14:35.701 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.702 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.702 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:35.702 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556356+0000", 2020-01-16T00:14:35.702 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9750000000000007e-06 2020-01-16T00:14:35.702 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.702 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.702 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:35.703 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556364+0000", 2020-01-16T00:14:35.703 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8872999999999997e-05 2020-01-16T00:14:35.703 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.703 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.703 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.703 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556423+0000", 2020-01-16T00:14:35.703 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8687000000000001e-05 2020-01-16T00:14:35.703 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.703 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.704 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:35.704 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556502+0000", 2020-01-16T00:14:35.704 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00013235300000000001 2020-01-16T00:14:35.704 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.704 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.704 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:35.704 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.556634+0000", 2020-01-16T00:14:35.705 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0059770520000000001 2020-01-16T00:14:35.705 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.705 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.705 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:35.705 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.562611+0000", 2020-01-16T00:14:35.705 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029492820000000001 2020-01-16T00:14:35.705 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.705 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.706 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.706 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.565561+0000", 2020-01-16T00:14:35.706 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00060620300000000004 2020-01-16T00:14:35.706 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.706 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.706 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.706 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566167+0000", 2020-01-16T00:14:35.707 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.3831000000000004e-05 2020-01-16T00:14:35.707 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.707 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.707 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:35.707 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566251+0000", 2020-01-16T00:14:35.707 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.8328000000000001e-05 2020-01-16T00:14:35.707 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.707 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.708 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:35.708 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:07.566309+0000", 2020-01-16T00:14:35.708 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010455002999999999 2020-01-16T00:14:35.708 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.708 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.708 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.708 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.708 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.709 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:72 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 56832~1408000] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e37)", 2020-01-16T00:14:35.709 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.578753+0000", 2020-01-16T00:14:35.709 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 83.068172873999998, 2020-01-16T00:14:35.709 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011826918, 2020-01-16T00:14:35.709 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:35.709 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:35.709 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:35.710 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.710 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:35.710 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578753+0000", 2020-01-16T00:14:35.710 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:35.710 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.710 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.710 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:35.710 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578753+0000", 2020-01-16T00:14:35.711 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:35.711 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.711 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.711 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:35.711 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.578751+0000", 2020-01-16T00:14:35.711 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00042457699999999998 2020-01-16T00:14:35.711 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.711 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.712 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:35.712 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579175+0000", 2020-01-16T00:14:35.712 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5404999999999999e-05 2020-01-16T00:14:35.712 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.712 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.712 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:35.712 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579191+0000", 2020-01-16T00:14:35.713 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.6149999999999996e-06 2020-01-16T00:14:35.713 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.713 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.713 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:35.713 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579196+0000", 2020-01-16T00:14:35.713 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000104794 2020-01-16T00:14:35.713 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.713 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.714 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.714 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579301+0000", 2020-01-16T00:14:35.714 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020947199999999999 2020-01-16T00:14:35.714 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.714 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.714 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:35.714 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579511+0000", 2020-01-16T00:14:35.714 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00034328400000000002 2020-01-16T00:14:35.715 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.715 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.715 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:35.715 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579854+0000", 2020-01-16T00:14:35.715 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.005078117 2020-01-16T00:14:35.715 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.715 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.715 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:35.716 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.584932+0000", 2020-01-16T00:14:35.716 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0014702160000000001 2020-01-16T00:14:35.716 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.716 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.716 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.716 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.586402+0000", 2020-01-16T00:14:35.716 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0040623589999999998 2020-01-16T00:14:35.716 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.717 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.717 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.717 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590465+0000", 2020-01-16T00:14:35.717 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4478000000000002e-05 2020-01-16T00:14:35.717 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.717 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.717 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:35.718 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590529+0000", 2020-01-16T00:14:35.718 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1121000000000001e-05 2020-01-16T00:14:35.718 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.718 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.718 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:35.718 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590580+0000", 2020-01-16T00:14:35.718 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011826918 2020-01-16T00:14:35.719 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.719 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.719 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.719 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.719 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.719 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:73 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 1922048~118272] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e37)", 2020-01-16T00:14:35.719 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:14:35.720 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 83.067720077000004, 2020-01-16T00:14:35.720 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011750764, 2020-01-16T00:14:35.720 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:35.720 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:35.720 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:35.720 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.720 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:35.720 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:14:35.721 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:35.721 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.721 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.721 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:35.721 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:14:35.721 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:35.721 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.721 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.722 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:35.722 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579206+0000", 2020-01-16T00:14:35.722 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2780000000000001e-05 2020-01-16T00:14:35.722 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.722 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.722 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:35.722 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579238+0000", 2020-01-16T00:14:35.722 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4341999999999999e-05 2020-01-16T00:14:35.723 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.723 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.723 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:35.723 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579253+0000", 2020-01-16T00:14:35.723 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.1920000000000001e-06 2020-01-16T00:14:35.723 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.723 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.724 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:35.724 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.579256+0000", 2020-01-16T00:14:35.724 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003224902 2020-01-16T00:14:35.724 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.724 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.724 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.724 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.582481+0000", 2020-01-16T00:14:35.724 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000180729 2020-01-16T00:14:35.725 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.725 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.725 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:35.725 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.582662+0000", 2020-01-16T00:14:35.725 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00025106100000000001 2020-01-16T00:14:35.725 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.725 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.726 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:35.726 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.582913+0000", 2020-01-16T00:14:35.726 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0020814449999999999 2020-01-16T00:14:35.726 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.726 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.726 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:35.726 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.584994+0000", 2020-01-16T00:14:35.726 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001717243 2020-01-16T00:14:35.727 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.727 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.727 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.727 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.586711+0000", 2020-01-16T00:14:35.727 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0041349869999999997 2020-01-16T00:14:35.727 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.727 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.727 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.728 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590846+0000", 2020-01-16T00:14:35.728 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3565000000000006e-05 2020-01-16T00:14:35.728 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.728 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.728 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:35.728 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590910+0000", 2020-01-16T00:14:35.728 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7002999999999997e-05 2020-01-16T00:14:35.728 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.729 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.729 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:35.729 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:12.590957+0000", 2020-01-16T00:14:35.729 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011750764 2020-01-16T00:14:35.729 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.729 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.729 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.730 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.730 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.730 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:77 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 56832~1408000] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e45)", 2020-01-16T00:14:35.730 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.584566+0000", 2020-01-16T00:14:35.730 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 66.062360403, 2020-01-16T00:14:35.730 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017037775000000002, 2020-01-16T00:14:35.730 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:35.730 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:35.731 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:35.731 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.731 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:35.731 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584566+0000", 2020-01-16T00:14:35.731 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:35.731 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.731 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.731 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:35.732 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584566+0000", 2020-01-16T00:14:35.732 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:35.732 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.732 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.732 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:35.732 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584564+0000", 2020-01-16T00:14:35.732 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00043024300000000002 2020-01-16T00:14:35.733 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.733 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.733 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:35.733 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.584994+0000", 2020-01-16T00:14:35.733 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.9485999999999999e-05 2020-01-16T00:14:35.733 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.733 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.733 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:35.734 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585014+0000", 2020-01-16T00:14:35.734 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8909999999999995e-06 2020-01-16T00:14:35.734 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.734 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.734 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:35.734 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585022+0000", 2020-01-16T00:14:35.734 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039114459999999998 2020-01-16T00:14:35.734 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.735 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.735 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.735 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.588933+0000", 2020-01-16T00:14:35.735 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011178700000000001 2020-01-16T00:14:35.735 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.735 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.735 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:35.735 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589045+0000", 2020-01-16T00:14:35.736 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00016967300000000001 2020-01-16T00:14:35.736 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.736 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.736 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:35.736 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.589214+0000", 2020-01-16T00:14:35.736 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012142356 2020-01-16T00:14:35.736 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.737 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.737 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.737 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601357+0000", 2020-01-16T00:14:35.737 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.1466999999999997e-05 2020-01-16T00:14:35.737 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.737 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.738 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.738 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601448+0000", 2020-01-16T00:14:35.738 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4817000000000001e-05 2020-01-16T00:14:35.738 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.738 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.738 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:35.738 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601493+0000", 2020-01-16T00:14:35.738 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9734000000000001e-05 2020-01-16T00:14:35.739 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.739 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.739 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:35.739 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601553+0000", 2020-01-16T00:14:35.741 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.0677999999999998e-05 2020-01-16T00:14:35.741 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.741 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.741 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:35.741 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601604+0000", 2020-01-16T00:14:35.741 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.017037775000000002 2020-01-16T00:14:35.741 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.742 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.742 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.742 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.742 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.742 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:78 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 1922048~118272] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e45)", 2020-01-16T00:14:35.742 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.585030+0000", 2020-01-16T00:14:35.742 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 66.061895680999996, 2020-01-16T00:14:35.742 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016699964000000001, 2020-01-16T00:14:35.743 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:35.743 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:35.743 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:35.743 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.743 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:35.743 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585030+0000", 2020-01-16T00:14:35.743 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:35.743 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.744 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.744 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:35.744 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585030+0000", 2020-01-16T00:14:35.744 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:35.744 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.744 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.744 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:35.744 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585030+0000", 2020-01-16T00:14:35.745 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.7085000000000001e-05 2020-01-16T00:14:35.745 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.745 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.745 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:35.745 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585077+0000", 2020-01-16T00:14:35.745 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.5380000000000008e-06 2020-01-16T00:14:35.745 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.746 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.746 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:35.746 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585086+0000", 2020-01-16T00:14:35.746 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5410000000000002e-06 2020-01-16T00:14:35.746 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.746 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.746 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:35.746 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.585091+0000", 2020-01-16T00:14:35.747 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0088486620000000002 2020-01-16T00:14:35.747 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.747 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.747 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.747 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.593940+0000", 2020-01-16T00:14:35.747 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000112594 2020-01-16T00:14:35.747 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.747 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.748 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:35.748 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.594052+0000", 2020-01-16T00:14:35.748 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00019893999999999999 2020-01-16T00:14:35.748 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.748 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.748 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:35.748 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.594251+0000", 2020-01-16T00:14:35.748 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0071632730000000004 2020-01-16T00:14:35.749 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.749 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.749 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.749 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601414+0000", 2020-01-16T00:14:35.749 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.4672999999999996e-05 2020-01-16T00:14:35.749 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.749 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.750 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.750 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601479+0000", 2020-01-16T00:14:35.750 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000158453 2020-01-16T00:14:35.750 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.750 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.750 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:35.750 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601638+0000", 2020-01-16T00:14:35.750 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.8605e-05 2020-01-16T00:14:35.751 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.751 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.751 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:35.751 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601686+0000", 2020-01-16T00:14:35.751 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4283999999999998e-05 2020-01-16T00:14:35.751 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.751 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.751 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:35.752 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.601730+0000", 2020-01-16T00:14:35.752 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.016699964000000001 2020-01-16T00:14:35.752 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.752 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.752 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.752 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.752 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.752 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:127 2.1 2:a4df40e2:::smithi12313934-42 low_tier:head [write 1364992~1145344] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e45)", 2020-01-16T00:14:35.753 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:29.627308+0000", 2020-01-16T00:14:35.753 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 66.019618229000002, 2020-01-16T00:14:35.753 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011068883999999999, 2020-01-16T00:14:35.753 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:35.753 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:35.753 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:35.753 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.754 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:35.754 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627308+0000", 2020-01-16T00:14:35.754 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:35.754 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.754 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.754 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:35.754 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627308+0000", 2020-01-16T00:14:35.754 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:35.755 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.755 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.755 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:35.755 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627307+0000", 2020-01-16T00:14:35.755 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000278597 2020-01-16T00:14:35.755 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.755 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.755 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:35.756 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627586+0000", 2020-01-16T00:14:35.756 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9169999999999997e-06 2020-01-16T00:14:35.756 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.756 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.756 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:35.756 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627591+0000", 2020-01-16T00:14:35.756 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.0479999999999999e-06 2020-01-16T00:14:35.756 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.757 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.757 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:35.757 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.627594+0000", 2020-01-16T00:14:35.757 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030987269999999999 2020-01-16T00:14:35.757 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.757 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.757 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.758 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.630693+0000", 2020-01-16T00:14:35.758 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000103688 2020-01-16T00:14:35.758 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.758 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.758 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:35.758 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.630796+0000", 2020-01-16T00:14:35.758 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000144606 2020-01-16T00:14:35.759 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.759 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.759 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:35.759 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.630941+0000", 2020-01-16T00:14:35.759 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0039264269999999997 2020-01-16T00:14:35.759 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.759 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.759 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:35.760 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.634867+0000", 2020-01-16T00:14:35.760 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00038581200000000003 2020-01-16T00:14:35.760 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.760 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.760 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.760 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.635253+0000", 2020-01-16T00:14:35.760 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0030568879999999998 2020-01-16T00:14:35.760 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.761 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.761 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.761 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.638310+0000", 2020-01-16T00:14:35.761 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.8390999999999999e-05 2020-01-16T00:14:35.761 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.761 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.761 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:35.762 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.638348+0000", 2020-01-16T00:14:35.762 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.8402999999999999e-05 2020-01-16T00:14:35.762 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.762 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.762 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:35.762 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:29.638377+0000", 2020-01-16T00:14:35.762 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011068883999999999 2020-01-16T00:14:35.763 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.763 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.763 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.763 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.763 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.763 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.6.13:30 2.3 2:fc791323:::smithi12313934-2 low_tier:head [write 476160~1240064] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e48)", 2020-01-16T00:14:35.763 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:39.466268+0000", 2020-01-16T00:14:35.763 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 56.180657924999998, 2020-01-16T00:14:35.764 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010593112, 2020-01-16T00:14:35.764 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:35.764 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:35.764 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:35.764 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.764 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:35.764 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.466268+0000", 2020-01-16T00:14:35.764 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:35.765 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.765 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.765 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:35.765 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.466268+0000", 2020-01-16T00:14:35.765 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:35.765 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.765 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.765 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:35.766 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.466266+0000", 2020-01-16T00:14:35.766 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0012489210000000001 2020-01-16T00:14:35.766 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.766 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.766 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:35.766 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467515+0000", 2020-01-16T00:14:35.766 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8411999999999999e-05 2020-01-16T00:14:35.767 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.767 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.767 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:35.767 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467533+0000", 2020-01-16T00:14:35.767 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.9319999999999996e-06 2020-01-16T00:14:35.767 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.767 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.767 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:35.768 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467541+0000", 2020-01-16T00:14:35.768 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5289999999999993e-05 2020-01-16T00:14:35.768 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.768 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.768 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.768 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467607+0000", 2020-01-16T00:14:35.768 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00010962 2020-01-16T00:14:35.769 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.769 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.769 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:35.769 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467716+0000", 2020-01-16T00:14:35.769 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020883699999999999 2020-01-16T00:14:35.769 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.769 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.769 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 7", 2020-01-16T00:14:35.770 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.467925+0000", 2020-01-16T00:14:35.770 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063872829999999997 2020-01-16T00:14:35.770 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.770 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.770 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:35.770 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.474312+0000", 2020-01-16T00:14:35.770 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0023165769999999998 2020-01-16T00:14:35.770 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.771 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.771 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.771 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.476629+0000", 2020-01-16T00:14:35.771 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.8543999999999999e-05 2020-01-16T00:14:35.771 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.771 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.771 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:35.771 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.476707+0000", 2020-01-16T00:14:35.772 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000153914 2020-01-16T00:14:35.772 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.772 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.772 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:35.772 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:39.476861+0000", 2020-01-16T00:14:35.772 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010593112 2020-01-16T00:14:35.772 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.772 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.773 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.773 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.773 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.773 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:155 2.3 2:c9873fc7:::smithi12313934-30 low_tier:head [write 1542656~711168] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e50)", 2020-01-16T00:14:35.773 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:43.769943+0000", 2020-01-16T00:14:35.773 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 51.876982998999999, 2020-01-16T00:14:35.773 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010710127999999999, 2020-01-16T00:14:35.774 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:35.774 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:35.774 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:35.774 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.774 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:35.774 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.769943+0000", 2020-01-16T00:14:35.774 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:35.774 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.775 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.775 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:35.775 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.769943+0000", 2020-01-16T00:14:35.775 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:35.775 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.775 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.775 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:35.775 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.769942+0000", 2020-01-16T00:14:35.776 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00017878100000000001 2020-01-16T00:14:35.776 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.776 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.776 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:35.776 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.770121+0000", 2020-01-16T00:14:35.776 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2980000000000003e-06 2020-01-16T00:14:35.776 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.776 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.777 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:35.777 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.770129+0000", 2020-01-16T00:14:35.777 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9129999999999999e-06 2020-01-16T00:14:35.777 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.777 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.777 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:35.777 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.770131+0000", 2020-01-16T00:14:35.777 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0053233519999999999 2020-01-16T00:14:35.778 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.778 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.778 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.778 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.775455+0000", 2020-01-16T00:14:35.778 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012385099999999999 2020-01-16T00:14:35.778 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.778 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.779 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:35.779 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.775579+0000", 2020-01-16T00:14:35.779 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000211554 2020-01-16T00:14:35.779 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.779 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.779 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 7", 2020-01-16T00:14:35.779 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.775790+0000", 2020-01-16T00:14:35.779 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0029614210000000001 2020-01-16T00:14:35.779 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.780 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.780 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:35.780 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.778752+0000", 2020-01-16T00:14:35.780 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.001798928 2020-01-16T00:14:35.780 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.780 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.780 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.780 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.780551+0000", 2020-01-16T00:14:35.781 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.9413999999999999e-05 2020-01-16T00:14:35.781 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.781 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.781 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:35.781 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.780590+0000", 2020-01-16T00:14:35.781 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.3275999999999995e-05 2020-01-16T00:14:35.781 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.781 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.782 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:35.782 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:43.780653+0000", 2020-01-16T00:14:35.782 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010710127999999999 2020-01-16T00:14:35.782 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.782 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.782 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.782 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.783 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.783 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:171 2.1 2:a935ad67:::smithi12313934-50 low_tier:head [write 2422272~237568] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e55)", 2020-01-16T00:14:35.783 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:13:58.196749+0000", 2020-01-16T00:14:35.783 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 37.450176714000001, 2020-01-16T00:14:35.783 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010677759, 2020-01-16T00:14:35.783 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:35.783 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:35.783 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:35.784 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.784 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:35.784 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196749+0000", 2020-01-16T00:14:35.784 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:35.784 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.784 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.784 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:35.784 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196749+0000", 2020-01-16T00:14:35.784 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:35.785 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.785 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.785 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:35.785 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196749+0000", 2020-01-16T00:14:35.785 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.6604000000000001e-05 2020-01-16T00:14:35.785 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.785 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.785 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:35.785 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196815+0000", 2020-01-16T00:14:35.786 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.4900000000000006e-06 2020-01-16T00:14:35.786 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.786 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.786 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:35.786 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196825+0000", 2020-01-16T00:14:35.786 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3989999999999997e-06 2020-01-16T00:14:35.786 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.787 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.787 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:35.787 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.196829+0000", 2020-01-16T00:14:35.787 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0076943869999999996 2020-01-16T00:14:35.787 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.787 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.787 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.787 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.204524+0000", 2020-01-16T00:14:35.788 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000124745 2020-01-16T00:14:35.788 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.788 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.788 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:35.788 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.204649+0000", 2020-01-16T00:14:35.788 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00018801700000000001 2020-01-16T00:14:35.788 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.788 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.789 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:35.789 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.204837+0000", 2020-01-16T00:14:35.789 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002354765 2020-01-16T00:14:35.789 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.789 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.789 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.789 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.207191+0000", 2020-01-16T00:14:35.789 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 8.6249999999999996e-06 2020-01-16T00:14:35.789 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.790 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.790 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:35.790 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.207200+0000", 2020-01-16T00:14:35.790 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000162246 2020-01-16T00:14:35.790 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.790 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.790 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.791 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.207362+0000", 2020-01-16T00:14:35.791 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5117999999999998e-05 2020-01-16T00:14:35.791 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.791 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.791 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:35.791 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.207397+0000", 2020-01-16T00:14:35.791 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.9912000000000001e-05 2020-01-16T00:14:35.791 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.792 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.792 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:35.792 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:13:58.207427+0000", 2020-01-16T00:14:35.792 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010677759 2020-01-16T00:14:35.792 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.792 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.792 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.792 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.793 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.793 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:183 2.1 2:a4df40e2:::smithi12313934-42 low_tier:head [write 70656~727552] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e58)", 2020-01-16T00:14:35.793 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.021222+0000", 2020-01-16T00:14:35.793 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 28.625704145, 2020-01-16T00:14:35.793 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012681151999999999, 2020-01-16T00:14:35.793 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:35.793 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:35.794 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:35.794 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.794 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:35.794 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021222+0000", 2020-01-16T00:14:35.794 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:35.794 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.794 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.794 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:35.795 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021222+0000", 2020-01-16T00:14:35.795 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:35.795 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.795 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.795 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:35.795 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021219+0000", 2020-01-16T00:14:35.795 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020912099999999999 2020-01-16T00:14:35.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.796 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.796 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:35.796 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021428+0000", 2020-01-16T00:14:35.796 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.8595999999999999e-05 2020-01-16T00:14:35.796 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.796 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.796 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:35.797 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021447+0000", 2020-01-16T00:14:35.797 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.2020000000000001e-06 2020-01-16T00:14:35.797 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.797 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.797 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:35.797 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021453+0000", 2020-01-16T00:14:35.797 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.5297999999999999e-05 2020-01-16T00:14:35.798 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.798 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.798 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.798 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021499+0000", 2020-01-16T00:14:35.798 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.2299999999999996e-05 2020-01-16T00:14:35.798 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.798 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.798 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:35.799 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021571+0000", 2020-01-16T00:14:35.799 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00011000799999999999 2020-01-16T00:14:35.799 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.799 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.799 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:35.799 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.021681+0000", 2020-01-16T00:14:35.799 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0065924879999999996 2020-01-16T00:14:35.800 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.800 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.800 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.800 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.028273+0000", 2020-01-16T00:14:35.800 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00022212499999999999 2020-01-16T00:14:35.800 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.800 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.800 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:35.801 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.028495+0000", 2020-01-16T00:14:35.801 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0053014580000000002 2020-01-16T00:14:35.801 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.801 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.801 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.801 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.033797+0000", 2020-01-16T00:14:35.801 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.6142e-05 2020-01-16T00:14:35.801 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.802 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.802 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:35.802 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.033843+0000", 2020-01-16T00:14:35.802 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.0130999999999997e-05 2020-01-16T00:14:35.802 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.802 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.802 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:35.802 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.033903+0000", 2020-01-16T00:14:35.803 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012681151999999999 2020-01-16T00:14:35.803 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.803 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.803 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.803 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.803 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.803 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.3.13:184 2.1 2:a4df40e2:::smithi12313934-42 low_tier:head [write 1364992~1145344] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e58)", 2020-01-16T00:14:35.804 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.023539+0000", 2020-01-16T00:14:35.804 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 28.623387026, 2020-01-16T00:14:35.804 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011003893000000001, 2020-01-16T00:14:35.804 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:35.804 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:35.804 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:35.804 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.804 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:35.805 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023539+0000", 2020-01-16T00:14:35.805 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:35.805 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.805 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.805 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:35.805 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023539+0000", 2020-01-16T00:14:35.805 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:35.806 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.806 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.806 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:35.806 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023537+0000", 2020-01-16T00:14:35.806 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00033271700000000001 2020-01-16T00:14:35.806 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.806 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.806 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:35.807 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023870+0000", 2020-01-16T00:14:35.807 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5744999999999999e-05 2020-01-16T00:14:35.807 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.807 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.807 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:35.807 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023886+0000", 2020-01-16T00:14:35.807 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.7389999999999994e-06 2020-01-16T00:14:35.808 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.808 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.808 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:35.808 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023895+0000", 2020-01-16T00:14:35.808 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.6058999999999998e-05 2020-01-16T00:14:35.808 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.808 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.808 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.809 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.023971+0000", 2020-01-16T00:14:35.809 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000119207 2020-01-16T00:14:35.809 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.809 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.809 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:35.809 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.024091+0000", 2020-01-16T00:14:35.809 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000166074 2020-01-16T00:14:35.809 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.810 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.810 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:35.810 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.024257+0000", 2020-01-16T00:14:35.810 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0047290129999999998 2020-01-16T00:14:35.810 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.810 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.810 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.811 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.028986+0000", 2020-01-16T00:14:35.811 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00028545000000000003 2020-01-16T00:14:35.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.811 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.811 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:35.811 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.029271+0000", 2020-01-16T00:14:35.811 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0052013049999999998 2020-01-16T00:14:35.811 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.812 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.812 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.812 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.034472+0000", 2020-01-16T00:14:35.812 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.7738999999999999e-05 2020-01-16T00:14:35.812 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.812 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.812 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:35.812 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.034510+0000", 2020-01-16T00:14:35.813 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2913000000000001e-05 2020-01-16T00:14:35.813 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.813 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.813 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:35.813 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.034543+0000", 2020-01-16T00:14:35.813 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.011003893000000001 2020-01-16T00:14:35.813 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.813 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.814 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.814 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.814 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.814 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:98 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 56832~1408000] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e58)", 2020-01-16T00:14:35.814 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.049295+0000", 2020-01-16T00:14:35.814 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 28.597630976000001, 2020-01-16T00:14:35.814 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012327324000000001, 2020-01-16T00:14:35.815 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:35.815 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:35.815 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:35.815 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.815 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:35.815 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049295+0000", 2020-01-16T00:14:35.815 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:35.815 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.816 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.816 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:35.816 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049295+0000", 2020-01-16T00:14:35.816 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:35.816 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.816 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.816 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:35.816 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049294+0000", 2020-01-16T00:14:35.817 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00036515000000000001 2020-01-16T00:14:35.817 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.817 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.817 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:35.817 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049659+0000", 2020-01-16T00:14:35.817 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.4299e-05 2020-01-16T00:14:35.817 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.817 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.818 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:35.818 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049673+0000", 2020-01-16T00:14:35.818 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.1250000000000001e-06 2020-01-16T00:14:35.818 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.818 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.818 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:35.818 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049678+0000", 2020-01-16T00:14:35.818 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.9937e-05 2020-01-16T00:14:35.819 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.819 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.819 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.819 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049728+0000", 2020-01-16T00:14:35.819 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.1094999999999994e-05 2020-01-16T00:14:35.819 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.819 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.819 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:35.820 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049819+0000", 2020-01-16T00:14:35.820 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00012621999999999999 2020-01-16T00:14:35.820 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.820 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.820 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:35.820 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049946+0000", 2020-01-16T00:14:35.820 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0081844340000000008 2020-01-16T00:14:35.820 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.821 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.821 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:35.821 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.058130+0000", 2020-01-16T00:14:35.821 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00058478000000000002 2020-01-16T00:14:35.821 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.821 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.821 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.821 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.058715+0000", 2020-01-16T00:14:35.822 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0027436349999999999 2020-01-16T00:14:35.822 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.822 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.822 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.822 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.061459+0000", 2020-01-16T00:14:35.822 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.1755000000000007e-05 2020-01-16T00:14:35.822 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.822 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.823 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:35.823 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.061530+0000", 2020-01-16T00:14:35.823 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.2236000000000003e-05 2020-01-16T00:14:35.823 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.823 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.823 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:35.823 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.061623+0000", 2020-01-16T00:14:35.824 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012327324000000001 2020-01-16T00:14:35.824 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.824 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.824 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.824 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.824 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.824 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:99 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 1922048~118272] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e58)", 2020-01-16T00:14:35.824 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:07.049687+0000", 2020-01-16T00:14:35.825 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 28.597238998000002, 2020-01-16T00:14:35.825 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012484960999999999, 2020-01-16T00:14:35.825 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:35.825 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:35.825 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:35.825 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.825 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:35.825 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049687+0000", 2020-01-16T00:14:35.826 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:35.826 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.826 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.826 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:35.826 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049687+0000", 2020-01-16T00:14:35.826 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:35.826 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.826 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.827 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:35.827 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049687+0000", 2020-01-16T00:14:35.827 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.7373999999999999e-05 2020-01-16T00:14:35.827 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.827 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.827 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:35.827 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049714+0000", 2020-01-16T00:14:35.827 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.2030999999999997e-05 2020-01-16T00:14:35.828 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.828 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.828 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:35.828 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049766+0000", 2020-01-16T00:14:35.828 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.5559999999999999e-06 2020-01-16T00:14:35.828 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.828 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.828 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:35.829 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.049770+0000", 2020-01-16T00:14:35.829 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0026935779999999999 2020-01-16T00:14:35.829 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.829 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.829 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.829 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.052463+0000", 2020-01-16T00:14:35.829 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.5037999999999995e-05 2020-01-16T00:14:35.830 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.830 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.830 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:35.830 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.052538+0000", 2020-01-16T00:14:35.830 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000116791 2020-01-16T00:14:35.830 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.830 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.830 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:35.831 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.052655+0000", 2020-01-16T00:14:35.831 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0063608579999999996 2020-01-16T00:14:35.831 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.831 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.831 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:35.831 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.059016+0000", 2020-01-16T00:14:35.831 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000131872 2020-01-16T00:14:35.831 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.832 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.832 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.832 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.059148+0000", 2020-01-16T00:14:35.832 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.002907677 2020-01-16T00:14:35.832 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.832 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.832 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.832 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.062055+0000", 2020-01-16T00:14:35.833 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.9624999999999999e-05 2020-01-16T00:14:35.833 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.833 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.833 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:35.833 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.062115+0000", 2020-01-16T00:14:35.833 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 5.7096999999999999e-05 2020-01-16T00:14:35.833 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.833 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.834 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:35.834 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:07.062172+0000", 2020-01-16T00:14:35.834 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.012484960999999999 2020-01-16T00:14:35.834 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.834 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.834 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.834 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.834 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.835 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:102 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 56832~1408000] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e65)", 2020-01-16T00:14:35.835 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:22.565060+0000", 2020-01-16T00:14:35.835 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 13.081866019, 2020-01-16T00:14:35.835 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010989515, 2020-01-16T00:14:35.835 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:35.835 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:35.835 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:35.836 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.836 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:35.836 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:22.565060+0000", 2020-01-16T00:14:35.836 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:35.836 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.836 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.836 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:35.836 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:22.565060+0000", 2020-01-16T00:14:35.837 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:35.837 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.837 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.837 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:35.837 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:22.565058+0000", 2020-01-16T00:14:35.837 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000485959 2020-01-16T00:14:35.837 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.837 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.838 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:35.838 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:22.565544+0000", 2020-01-16T00:14:35.838 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 1.5903e-05 2020-01-16T00:14:35.838 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.838 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.838 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:35.839 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:22.565560+0000", 2020-01-16T00:14:35.839 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5849999999999996e-06 2020-01-16T00:14:35.839 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.839 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.839 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:35.839 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:22.565567+0000", 2020-01-16T00:14:35.839 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.3266999999999998e-05 2020-01-16T00:14:35.839 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.840 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.840 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.840 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:22.565610+0000", 2020-01-16T00:14:35.840 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 6.5847999999999997e-05 2020-01-16T00:14:35.840 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.840 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.840 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:35.840 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:22.565676+0000", 2020-01-16T00:14:35.841 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000102602 2020-01-16T00:14:35.841 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.841 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.841 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:35.841 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:22.565779+0000", 2020-01-16T00:14:35.841 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0072662159999999998 2020-01-16T00:14:35.841 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.841 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.842 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:35.842 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:22.573045+0000", 2020-01-16T00:14:35.842 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0028547690000000001 2020-01-16T00:14:35.842 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.842 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.842 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.842 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:22.575900+0000", 2020-01-16T00:14:35.842 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.7398999999999995e-05 2020-01-16T00:14:35.843 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.843 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.843 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.843 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:22.575977+0000", 2020-01-16T00:14:35.843 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.0142999999999998e-05 2020-01-16T00:14:35.843 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.843 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.843 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:35.844 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:22.576017+0000", 2020-01-16T00:14:35.844 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2554e-05 2020-01-16T00:14:35.844 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.844 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.844 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:35.844 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:22.576050+0000", 2020-01-16T00:14:35.844 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010989515 2020-01-16T00:14:35.845 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.845 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.845 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.845 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.845 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.845 INFO:teuthology.orchestra.run.smithi123.stdout: "description": "osd_op(osd.1.13:103 2.1 2:bafac7f5:::smithi12313934-46 low_tier:head [write 1922048~118272] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e65)", 2020-01-16T00:14:35.845 INFO:teuthology.orchestra.run.smithi123.stdout: "initiated_at": "2020-01-16T00:14:22.565586+0000", 2020-01-16T00:14:35.845 INFO:teuthology.orchestra.run.smithi123.stdout: "age": 13.081340428000001, 2020-01-16T00:14:35.846 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010829599000000001, 2020-01-16T00:14:35.846 INFO:teuthology.orchestra.run.smithi123.stdout: "type_data": { 2020-01-16T00:14:35.846 INFO:teuthology.orchestra.run.smithi123.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:35.846 INFO:teuthology.orchestra.run.smithi123.stdout: "events": [ 2020-01-16T00:14:35.846 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.846 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "initiated", 2020-01-16T00:14:35.846 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:22.565586+0000", 2020-01-16T00:14:35.847 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0 2020-01-16T00:14:35.847 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.847 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.847 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "throttled", 2020-01-16T00:14:35.847 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:22.565586+0000", 2020-01-16T00:14:35.847 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4294967295.999999 2020-01-16T00:14:35.847 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.847 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.848 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "header_read", 2020-01-16T00:14:35.848 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:22.565585+0000", 2020-01-16T00:14:35.848 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2199000000000001e-05 2020-01-16T00:14:35.848 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.848 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.848 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "all_read", 2020-01-16T00:14:35.848 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:22.565617+0000", 2020-01-16T00:14:35.848 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 7.0809999999999996e-06 2020-01-16T00:14:35.849 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.849 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.849 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "dispatched", 2020-01-16T00:14:35.849 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:22.565624+0000", 2020-01-16T00:14:35.849 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 4.4889999999999997e-06 2020-01-16T00:14:35.849 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.849 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.849 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "queued_for_pg", 2020-01-16T00:14:35.850 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:22.565629+0000", 2020-01-16T00:14:35.850 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0031442010000000001 2020-01-16T00:14:35.850 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.850 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.850 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "reached_pg", 2020-01-16T00:14:35.850 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:22.568773+0000", 2020-01-16T00:14:35.850 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.000118705 2020-01-16T00:14:35.851 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.851 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.851 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "started", 2020-01-16T00:14:35.851 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:22.568892+0000", 2020-01-16T00:14:35.851 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.00020234300000000001 2020-01-16T00:14:35.851 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.851 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.851 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "waiting for subops from 0,3", 2020-01-16T00:14:35.852 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:22.569094+0000", 2020-01-16T00:14:35.852 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.003960786 2020-01-16T00:14:35.852 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.852 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.852 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "op_commit", 2020-01-16T00:14:35.852 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:22.573055+0000", 2020-01-16T00:14:35.852 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.0032031160000000002 2020-01-16T00:14:35.852 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.853 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.853 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.853 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:22.576258+0000", 2020-01-16T00:14:35.853 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 9.9885999999999999e-05 2020-01-16T00:14:35.853 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.853 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.853 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:35.853 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:22.576358+0000", 2020-01-16T00:14:35.854 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 3.2607000000000001e-05 2020-01-16T00:14:35.854 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.854 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.854 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "commit_sent", 2020-01-16T00:14:35.854 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:22.576390+0000", 2020-01-16T00:14:35.854 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 2.4958000000000001e-05 2020-01-16T00:14:35.854 INFO:teuthology.orchestra.run.smithi123.stdout: }, 2020-01-16T00:14:35.854 INFO:teuthology.orchestra.run.smithi123.stdout: { 2020-01-16T00:14:35.855 INFO:teuthology.orchestra.run.smithi123.stdout: "event": "done", 2020-01-16T00:14:35.855 INFO:teuthology.orchestra.run.smithi123.stdout: "time": "2020-01-16T00:14:22.576415+0000", 2020-01-16T00:14:35.855 INFO:teuthology.orchestra.run.smithi123.stdout: "duration": 0.010829599000000001 2020-01-16T00:14:35.855 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.855 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.855 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.855 INFO:teuthology.orchestra.run.smithi123.stdout: } 2020-01-16T00:14:35.855 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2020-01-16T00:14:35.856 INFO:teuthology.orchestra.run.smithi123.stdout:} 2020-01-16T00:14:35.860 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-01-16T00:14:35.860 INFO:tasks.rados.rados.0.smithi123.stdout:2009: finishing write tid 1 to smithi12313934-3 2020-01-16T00:14:35.860 INFO:tasks.rados.rados.0.smithi123.stdout:2009: finishing write tid 2 to smithi12313934-3 2020-01-16T00:14:35.860 INFO:tasks.rados.rados.0.smithi123.stdout:2009: finishing write tid 3 to smithi12313934-3 2020-01-16T00:14:35.861 INFO:tasks.rados.rados.0.smithi123.stdout:2009: finishing write tid 5 to smithi12313934-3 2020-01-16T00:14:35.861 INFO:tasks.rados.rados.0.smithi123.stdout:2009: finishing write tid 6 to smithi12313934-3 2020-01-16T00:14:35.861 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 3 v 141 (ObjNum 771 snap 0 seq_num 771) dirty exists 2020-01-16T00:14:35.861 INFO:tasks.rados.rados.0.smithi123.stdout:2009: left oid 3 (ObjNum 771 snap 0 seq_num 771) 2020-01-16T00:14:35.861 INFO:tasks.rados.rados.0.smithi123.stdout:2010: finishing write tid 1 to smithi12313934-19 2020-01-16T00:14:35.861 INFO:tasks.rados.rados.0.smithi123.stdout:2010: finishing write tid 2 to smithi12313934-19 2020-01-16T00:14:35.861 INFO:tasks.rados.rados.0.smithi123.stdout:2001: done (11 left) 2020-01-16T00:14:35.861 INFO:tasks.rados.rados.0.smithi123.stdout:2002: done (10 left) 2020-01-16T00:14:35.862 INFO:tasks.rados.rados.0.smithi123.stdout:2003: done (9 left) 2020-01-16T00:14:35.862 INFO:tasks.rados.rados.0.smithi123.stdout:2004: done (8 left) 2020-01-16T00:14:35.862 INFO:tasks.rados.rados.0.smithi123.stdout:2005: done (7 left) 2020-01-16T00:14:35.862 INFO:tasks.rados.rados.0.smithi123.stdout:2006: done (6 left) 2020-01-16T00:14:35.862 INFO:tasks.rados.rados.0.smithi123.stdout:2008: done (5 left) 2020-01-16T00:14:35.862 INFO:tasks.rados.rados.0.smithi123.stdout:2009: done (4 left) 2020-01-16T00:14:35.862 INFO:tasks.rados.rados.0.smithi123.stdout:2013: done (3 left) 2020-01-16T00:14:35.862 INFO:tasks.rados.rados.0.smithi123.stdout:2014: write (excl) oid 44 current snap is 0 2020-01-16T00:14:35.863 INFO:tasks.rados.rados.0.smithi123.stdout:2014: seq_num 775 ranges {625425=469628,1768133=640265} 2020-01-16T00:14:35.863 INFO:tasks.rados.rados.0.smithi123.stdout:2014: writing smithi12313934-44 from 625425 to 1095053 tid 1 2020-01-16T00:14:35.863 INFO:tasks.rados.rados.0.smithi123.stdout:2014: writing smithi12313934-44 from 1768133 to 2408398 tid 2 2020-01-16T00:14:35.863 INFO:tasks.rados.rados.0.smithi123.stdout:2015: write oid 47 current snap is 0 2020-01-16T00:14:35.863 INFO:tasks.rados.rados.0.smithi123.stdout:2015: seq_num 776 ranges {642232=661230,1942968=783923} 2020-01-16T00:14:35.863 INFO:tasks.rados.rados.0.smithi123.stdout:2015: writing smithi12313934-47 from 642232 to 1303462 tid 1 2020-01-16T00:14:35.863 INFO:tasks.rados.rados.0.smithi123.stdout:2015: writing smithi12313934-47 from 1942968 to 2726891 tid 2 2020-01-16T00:14:35.863 INFO:tasks.rados.rados.0.smithi123.stdout:2016: write (excl) oid 9 current snap is 0 2020-01-16T00:14:35.864 INFO:tasks.rados.rados.0.smithi123.stdout:2016: seq_num 777 ranges {659039=736479,2001450=643934} 2020-01-16T00:14:35.864 INFO:tasks.rados.rados.0.smithi123.stdout:2016: writing smithi12313934-9 from 659039 to 1395518 tid 1 2020-01-16T00:14:35.864 INFO:tasks.rados.rados.0.smithi123.stdout:2016: writing smithi12313934-9 from 2001450 to 2645384 tid 2 2020-01-16T00:14:35.864 INFO:tasks.rados.rados.0.smithi123.stdout:2017: write (excl) oid 6 current snap is 0 2020-01-16T00:14:35.864 INFO:tasks.rados.rados.0.smithi123.stdout:2017: seq_num 778 ranges {675846=411728,1543579=787592,2848853=226993} 2020-01-16T00:14:35.864 INFO:tasks.rados.rados.0.smithi123.stdout:2017: writing smithi12313934-6 from 675846 to 1087574 tid 1 2020-01-16T00:14:35.864 INFO:tasks.rados.rados.0.smithi123.stdout:2017: writing smithi12313934-6 from 1543579 to 2331171 tid 2 2020-01-16T00:14:35.865 INFO:tasks.rados.rados.0.smithi123.stdout:2017: writing smithi12313934-6 from 2848853 to 3075846 tid 3 2020-01-16T00:14:35.865 INFO:tasks.rados.rados.0.smithi123.stdout:2018: write (excl) oid 39 current snap is 0 2020-01-16T00:14:35.865 INFO:tasks.rados.rados.0.smithi123.stdout:2018: seq_num 779 ranges {692653=486977,1602061=647603,2992629=100024} 2020-01-16T00:14:35.865 INFO:tasks.rados.rados.0.smithi123.stdout:2018: writing smithi12313934-39 from 692653 to 1179630 tid 1 2020-01-16T00:14:35.865 INFO:tasks.rados.rados.0.smithi123.stdout:2018: writing smithi12313934-39 from 1602061 to 2249664 tid 2 2020-01-16T00:14:35.865 INFO:tasks.rados.rados.0.smithi123.stdout:2018: writing smithi12313934-39 from 2992629 to 3092653 tid 3 2020-01-16T00:14:35.865 INFO:tasks.rados.rados.0.smithi123.stdout:2019: write (excl) oid 49 current snap is 0 2020-01-16T00:14:35.865 INFO:tasks.rados.rados.0.smithi123.stdout:2010: finishing write tid 3 to smithi12313934-19 2020-01-16T00:14:35.866 INFO:tasks.rados.rados.0.smithi123.stdout:2010: finishing write tid 5 to smithi12313934-19 2020-01-16T00:14:35.866 INFO:tasks.rados.rados.0.smithi123.stdout:2010: finishing write tid 6 to smithi12313934-19 2020-01-16T00:14:35.866 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 19 v 283 (ObjNum 772 snap 0 seq_num 772) dirty exists 2020-01-16T00:14:35.866 INFO:tasks.rados.rados.0.smithi123.stdout:2010: left oid 19 (ObjNum 772 snap 0 seq_num 772) 2020-01-16T00:14:35.866 INFO:tasks.rados.rados.0.smithi123.stdout:2011: finishing write tid 1 to smithi12313934-10 2020-01-16T00:14:35.866 INFO:tasks.rados.rados.0.smithi123.stdout:2011: finishing write tid 2 to smithi12313934-10 2020-01-16T00:14:35.866 INFO:tasks.rados.rados.0.smithi123.stdout:2019: seq_num 780 ranges {709460=562226,2060543=791261} 2020-01-16T00:14:35.867 INFO:tasks.rados.rados.0.smithi123.stdout:2019: writing smithi12313934-49 from 709460 to 1271686 tid 1 2020-01-16T00:14:35.867 INFO:tasks.rados.rados.0.smithi123.stdout:2019: writing smithi12313934-49 from 2060543 to 2851804 tid 2 2020-01-16T00:14:35.867 INFO:tasks.rados.rados.0.smithi123.stdout:2010: done (8 left) 2020-01-16T00:14:35.867 INFO:tasks.rados.rados.0.smithi123.stdout:2020: read oid 7 snap -1 2020-01-16T00:14:35.867 INFO:tasks.rados.rados.0.smithi123.stdout:2020: expect (ObjNum 736 snap 0 seq_num 736) 2020-01-16T00:14:35.867 INFO:tasks.rados.rados.0.smithi123.stdout:2021: delete oid 4 current snap is 0 2020-01-16T00:14:35.867 INFO:tasks.rados.rados.0.smithi123.stdout:2011: finishing write tid 3 to smithi12313934-10 2020-01-16T00:14:35.867 INFO:tasks.rados.rados.0.smithi123.stdout:2011: finishing write tid 5 to smithi12313934-10 2020-01-16T00:14:35.868 INFO:tasks.rados.rados.0.smithi123.stdout:2011: finishing write tid 6 to smithi12313934-10 2020-01-16T00:14:35.868 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 10 v 205 (ObjNum 773 snap 0 seq_num 773) dirty exists 2020-01-16T00:14:35.868 INFO:tasks.rados.rados.0.smithi123.stdout:2011: left oid 10 (ObjNum 773 snap 0 seq_num 773) 2020-01-16T00:14:35.868 INFO:tasks.rados.rados.0.smithi123.stdout:2012: expect (ObjNum 743 snap 0 seq_num 743) 2020-01-16T00:14:35.868 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:35.854+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:35.957 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:35.954+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:35.981 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:35.981 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:14:35.982 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:14:35.982 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:35.991 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-01-16T00:14:36.025 INFO:tasks.rados.rados.0.smithi123.stdout:2014: finishing write tid 1 to smithi12313934-44 2020-01-16T00:14:36.025 INFO:tasks.rados.rados.0.smithi123.stdout:2014: finishing write tid 2 to smithi12313934-44 2020-01-16T00:14:36.025 INFO:tasks.rados.rados.0.smithi123.stdout:2014: finishing write tid 4 to smithi12313934-44 2020-01-16T00:14:36.026 INFO:tasks.rados.rados.0.smithi123.stdout:2014: finishing write tid 5 to smithi12313934-44 2020-01-16T00:14:36.026 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 44 v 328 (ObjNum 775 snap 0 seq_num 775) dirty exists 2020-01-16T00:14:36.026 INFO:tasks.rados.rados.0.smithi123.stdout:2014: left oid 44 (ObjNum 775 snap 0 seq_num 775) 2020-01-16T00:14:36.026 INFO:tasks.rados.rados.0.smithi123.stdout:2015: finishing write tid 1 to smithi12313934-47 2020-01-16T00:14:36.026 INFO:tasks.rados.rados.0.smithi123.stdout:2015: finishing write tid 2 to smithi12313934-47 2020-01-16T00:14:36.026 INFO:tasks.rados.rados.0.smithi123.stdout:2015: finishing write tid 4 to smithi12313934-47 2020-01-16T00:14:36.026 INFO:tasks.rados.rados.0.smithi123.stdout:2015: finishing write tid 5 to smithi12313934-47 2020-01-16T00:14:36.027 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 47 v 327 (ObjNum 776 snap 0 seq_num 776) dirty exists 2020-01-16T00:14:36.027 INFO:tasks.rados.rados.0.smithi123.stdout:2015: left oid 47 (ObjNum 776 snap 0 seq_num 776) 2020-01-16T00:14:36.027 INFO:tasks.rados.rados.0.smithi123.stdout:2016: finishing write tid 1 to smithi12313934-9 2020-01-16T00:14:36.027 INFO:tasks.rados.rados.0.smithi123.stdout:2016: finishing write tid 2 to smithi12313934-9 2020-01-16T00:14:36.027 INFO:tasks.rados.rados.0.smithi123.stdout:2016: finishing write tid 4 to smithi12313934-9 2020-01-16T00:14:36.027 INFO:tasks.rados.rados.0.smithi123.stdout:2016: finishing write tid 5 to smithi12313934-9 2020-01-16T00:14:36.027 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 9 v 205 (ObjNum 777 snap 0 seq_num 777) dirty exists 2020-01-16T00:14:36.027 INFO:tasks.rados.rados.0.smithi123.stdout:2016: left oid 9 (ObjNum 777 snap 0 seq_num 777) 2020-01-16T00:14:36.028 INFO:tasks.rados.rados.0.smithi123.stdout:2017: finishing write tid 1 to smithi12313934-6 2020-01-16T00:14:36.028 INFO:tasks.rados.rados.0.smithi123.stdout:2017: finishing write tid 2 to smithi12313934-6 2020-01-16T00:14:36.028 INFO:tasks.rados.rados.0.smithi123.stdout:2017: finishing write tid 3 to smithi12313934-6 2020-01-16T00:14:36.028 INFO:tasks.rados.rados.0.smithi123.stdout:2017: finishing write tid 5 to smithi12313934-6 2020-01-16T00:14:36.028 INFO:tasks.rados.rados.0.smithi123.stdout:2011: done (9 left) 2020-01-16T00:14:36.028 INFO:tasks.rados.rados.0.smithi123.stdout:2012: done (8 left) 2020-01-16T00:14:36.028 INFO:tasks.rados.rados.0.smithi123.stdout:2014: done (7 left) 2020-01-16T00:14:36.028 INFO:tasks.rados.rados.0.smithi123.stdout:2015: done (6 left) 2020-01-16T00:14:36.029 INFO:tasks.rados.rados.0.smithi123.stdout:2016: done (5 left) 2020-01-16T00:14:36.029 INFO:tasks.rados.rados.0.smithi123.stdout:2021: done (4 left) 2020-01-16T00:14:36.029 INFO:tasks.rados.rados.0.smithi123.stdout:2022: read oid 36 snap -1 2020-01-16T00:14:36.029 INFO:tasks.rados.rados.0.smithi123.stdout:2022: expect (ObjNum 704 snap 0 seq_num 704) 2020-01-16T00:14:36.029 INFO:tasks.rados.rados.0.smithi123.stdout:2023: write (excl) oid 17 current snap is 0 2020-01-16T00:14:36.029 INFO:tasks.rados.rados.0.smithi123.stdout:2023: seq_num 782 ranges {743074=712724,2061154=794930} 2020-01-16T00:14:36.042 INFO:tasks.rados.rados.0.smithi123.stdout:2023: writing smithi12313934-17 from 743074 to 1455798 tid 1 2020-01-16T00:14:36.050 INFO:tasks.rados.rados.0.smithi123.stdout:2023: writing smithi12313934-17 from 2061154 to 2856084 tid 2 2020-01-16T00:14:36.051 INFO:tasks.rados.rados.0.smithi123.stdout:2024: read oid 31 snap -1 2020-01-16T00:14:36.051 INFO:tasks.rados.rados.0.smithi123.stdout:2024: expect (ObjNum 754 snap 0 seq_num 754) 2020-01-16T00:14:36.051 INFO:tasks.rados.rados.0.smithi123.stdout:2017: finishing write tid 6 to smithi12313934-6 2020-01-16T00:14:36.051 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 6 v 216 (ObjNum 778 snap 0 seq_num 778) dirty exists 2020-01-16T00:14:36.051 INFO:tasks.rados.rados.0.smithi123.stdout:2017: left oid 6 (ObjNum 778 snap 0 seq_num 778) 2020-01-16T00:14:36.051 INFO:tasks.rados.rados.0.smithi123.stdout:2018: finishing write tid 1 to smithi12313934-39 2020-01-16T00:14:36.051 INFO:tasks.rados.rados.0.smithi123.stdout:2018: finishing write tid 2 to smithi12313934-39 2020-01-16T00:14:36.052 INFO:tasks.rados.rados.0.smithi123.stdout:2018: finishing write tid 3 to smithi12313934-39 2020-01-16T00:14:36.052 INFO:tasks.rados.rados.0.smithi123.stdout:2025: read oid 3 snap -1 2020-01-16T00:14:36.052 INFO:tasks.rados.rados.0.smithi123.stdout:2025: expect (ObjNum 771 snap 0 seq_num 771) 2020-01-16T00:14:36.052 INFO:tasks.rados.rados.0.smithi123.stdout:2017: done (7 left) 2020-01-16T00:14:36.052 INFO:tasks.rados.rados.0.smithi123.stdout:2026: read oid 32 snap -1 2020-01-16T00:14:36.052 INFO:tasks.rados.rados.0.smithi123.stdout:2026: expect (ObjNum 748 snap 0 seq_num 748) 2020-01-16T00:14:36.052 INFO:tasks.rados.rados.0.smithi123.stdout:2027: write (excl) oid 18 current snap is 0 2020-01-16T00:14:36.052 INFO:tasks.rados.rados.0.smithi123.stdout:2027: seq_num 783 ranges {759881=787973,2119636=654941} 2020-01-16T00:14:36.058 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:36.054+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:36.059 INFO:tasks.rados.rados.0.smithi123.stdout:2027: writing smithi12313934-18 from 759881 to 1547854 tid 1 2020-01-16T00:14:36.067 INFO:tasks.rados.rados.0.smithi123.stdout:2027: writing smithi12313934-18 from 2119636 to 2774577 tid 2 2020-01-16T00:14:36.067 INFO:tasks.rados.rados.0.smithi123.stdout:2028: write oid 45 current snap is 0 2020-01-16T00:14:36.067 INFO:tasks.rados.rados.0.smithi123.stdout:2028: seq_num 784 ranges {776688=579575,1894471=798599} 2020-01-16T00:14:36.075 INFO:tasks.rados.rados.0.smithi123.stdout:2028: writing smithi12313934-45 from 776688 to 1356263 tid 1 2020-01-16T00:14:36.082 INFO:tasks.rados.rados.0.smithi123.stdout:2028: writing smithi12313934-45 from 1894471 to 2693070 tid 2 2020-01-16T00:14:36.082 INFO:tasks.rados.rados.0.smithi123.stdout:2029: write oid 42 current snap is 0 2020-01-16T00:14:36.083 INFO:tasks.rados.rados.0.smithi123.stdout:2029: seq_num 785 ranges {793495=654824,1952953=658610,3157167=36328} 2020-01-16T00:14:36.090 INFO:tasks.rados.rados.0.smithi123.stdout:2029: writing smithi12313934-42 from 793495 to 1448319 tid 1 2020-01-16T00:14:36.097 INFO:tasks.rados.rados.0.smithi123.stdout:2029: writing smithi12313934-42 from 1952953 to 2611563 tid 2 2020-01-16T00:14:36.101 INFO:tasks.rados.rados.0.smithi123.stdout:2029: writing smithi12313934-42 from 3157167 to 3193495 tid 3 2020-01-16T00:14:36.101 INFO:tasks.rados.rados.0.smithi123.stdout:2030: read oid 10 snap -1 2020-01-16T00:14:36.101 INFO:tasks.rados.rados.0.smithi123.stdout:2030: expect (ObjNum 773 snap 0 seq_num 773) 2020-01-16T00:14:36.101 INFO:tasks.rados.rados.0.smithi123.stdout:2031: read oid 38 snap -1 2020-01-16T00:14:36.101 INFO:tasks.rados.rados.0.smithi123.stdout:2031: expect (ObjNum 768 snap 0 seq_num 768) 2020-01-16T00:14:36.102 INFO:tasks.rados.rados.0.smithi123.stdout:2032: delete oid 19 current snap is 0 2020-01-16T00:14:36.102 INFO:tasks.rados.rados.0.smithi123.stdout:2018: finishing write tid 5 to smithi12313934-39 2020-01-16T00:14:36.102 INFO:tasks.rados.rados.0.smithi123.stdout:2018: finishing write tid 6 to smithi12313934-39 2020-01-16T00:14:36.103 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 39 v 332 (ObjNum 779 snap 0 seq_num 779) dirty exists 2020-01-16T00:14:36.103 INFO:tasks.rados.rados.0.smithi123.stdout:2018: left oid 39 (ObjNum 779 snap 0 seq_num 779) 2020-01-16T00:14:36.103 INFO:tasks.rados.rados.0.smithi123.stdout:2019: finishing write tid 1 to smithi12313934-49 2020-01-16T00:14:36.103 INFO:tasks.rados.rados.0.smithi123.stdout:2019: finishing write tid 2 to smithi12313934-49 2020-01-16T00:14:36.103 INFO:tasks.rados.rados.0.smithi123.stdout:2019: finishing write tid 4 to smithi12313934-49 2020-01-16T00:14:36.103 INFO:tasks.rados.rados.0.smithi123.stdout:2019: finishing write tid 5 to smithi12313934-49 2020-01-16T00:14:36.103 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 49 v 208 (ObjNum 780 snap 0 seq_num 780) dirty exists 2020-01-16T00:14:36.103 INFO:tasks.rados.rados.0.smithi123.stdout:2019: left oid 49 (ObjNum 780 snap 0 seq_num 780) 2020-01-16T00:14:36.104 INFO:tasks.rados.rados.0.smithi123.stdout:2020: expect (ObjNum 736 snap 0 seq_num 736) 2020-01-16T00:14:36.105 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:36.105 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:14:36.105 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:14:36.105 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:14:36.105 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:36.115 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-01-16T00:14:36.145 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2020-01-16T00:14:36.161 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:36.160+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:36.215 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:36.215 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:14:36.215 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:14:36.215 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:14:36.216 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.216 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:943 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:14:36.216 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:14:36.216 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 118.298775073, 2020-01-16T00:14:36.216 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018847004000000001, 2020-01-16T00:14:36.216 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:36.216 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:36.217 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:36.217 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:36.217 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:36.217 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 943 2020-01-16T00:14:36.217 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.217 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:36.218 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.218 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:36.218 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:14:36.218 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:36.218 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.218 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.218 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:36.218 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914479+0000", 2020-01-16T00:14:36.219 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:36.219 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.219 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.219 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:36.219 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914475+0000", 2020-01-16T00:14:36.219 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.508e-05 2020-01-16T00:14:36.220 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.220 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.220 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:36.220 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914490+0000", 2020-01-16T00:14:36.220 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.857e-05 2020-01-16T00:14:36.220 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.220 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.220 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:36.221 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914509+0000", 2020-01-16T00:14:36.221 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.46e-06 2020-01-16T00:14:36.221 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.221 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.221 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:36.221 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914515+0000", 2020-01-16T00:14:36.221 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2612999999999999e-05 2020-01-16T00:14:36.221 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.222 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.222 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.222 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914568+0000", 2020-01-16T00:14:36.222 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028312299999999998 2020-01-16T00:14:36.222 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.223 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.223 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:36.223 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914851+0000", 2020-01-16T00:14:36.223 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012282533 2020-01-16T00:14:36.223 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.223 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.223 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.223 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927133+0000", 2020-01-16T00:14:36.224 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4839999999999995e-05 2020-01-16T00:14:36.224 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.224 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.224 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:36.224 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927208+0000", 2020-01-16T00:14:36.224 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0047425610000000002 2020-01-16T00:14:36.224 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.225 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.225 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.225 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.931951+0000", 2020-01-16T00:14:36.225 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3628999999999994e-05 2020-01-16T00:14:36.225 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.225 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.225 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:36.226 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.932045+0000", 2020-01-16T00:14:36.226 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012810180000000001 2020-01-16T00:14:36.226 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.226 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.226 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:36.226 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933326+0000", 2020-01-16T00:14:36.226 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018847004000000001 2020-01-16T00:14:36.226 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.227 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:36.227 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.227 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.227 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.227 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:944 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:14:36.227 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:14:36.228 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 118.29872551699999, 2020-01-16T00:14:36.228 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019086901999999999, 2020-01-16T00:14:36.228 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:36.228 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:36.228 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:36.228 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:36.229 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:36.229 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 944 2020-01-16T00:14:36.229 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.229 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:36.229 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.229 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:36.229 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:14:36.230 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:36.230 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.230 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.230 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:36.230 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914528+0000", 2020-01-16T00:14:36.230 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:36.230 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.230 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.231 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:36.231 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914527+0000", 2020-01-16T00:14:36.231 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7509999999999999e-06 2020-01-16T00:14:36.231 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.231 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.231 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:36.231 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914532+0000", 2020-01-16T00:14:36.231 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5149999999999997e-06 2020-01-16T00:14:36.232 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.232 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.232 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:36.232 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914538+0000", 2020-01-16T00:14:36.232 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1890000000000007e-06 2020-01-16T00:14:36.232 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.232 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.233 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:36.233 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914546+0000", 2020-01-16T00:14:36.233 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000354951 2020-01-16T00:14:36.233 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.233 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.233 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.233 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914901+0000", 2020-01-16T00:14:36.234 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1594000000000002e-05 2020-01-16T00:14:36.234 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.234 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.234 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:36.234 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914942+0000", 2020-01-16T00:14:36.234 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012300070999999999 2020-01-16T00:14:36.234 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.234 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.235 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.235 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927242+0000", 2020-01-16T00:14:36.235 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4076999999999997e-05 2020-01-16T00:14:36.235 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.235 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.235 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:36.235 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927296+0000", 2020-01-16T00:14:36.236 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0060609180000000002 2020-01-16T00:14:36.236 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.236 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.236 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.236 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933357+0000", 2020-01-16T00:14:36.236 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4488999999999999e-05 2020-01-16T00:14:36.236 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.236 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.237 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:36.237 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933422+0000", 2020-01-16T00:14:36.237 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019318300000000001 2020-01-16T00:14:36.237 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.237 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.237 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:36.237 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933615+0000", 2020-01-16T00:14:36.238 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019086901999999999 2020-01-16T00:14:36.238 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.238 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:36.239 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.239 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.239 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.239 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:945 3.9 3:9218e7f1:::smithi12313934-14:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:14:36.239 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:14:36.239 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 118.298698311, 2020-01-16T00:14:36.239 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019342986, 2020-01-16T00:14:36.239 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:36.240 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:36.240 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:36.240 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:36.240 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:36.240 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 945 2020-01-16T00:14:36.240 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.240 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:36.241 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.241 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:36.241 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:14:36.241 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:36.241 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.241 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.241 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:36.241 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:14:36.242 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:36.242 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.242 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.242 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:36.242 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914555+0000", 2020-01-16T00:14:36.243 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4580000000000002e-06 2020-01-16T00:14:36.243 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.243 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.243 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:36.243 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914558+0000", 2020-01-16T00:14:36.243 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7829999999999998e-06 2020-01-16T00:14:36.244 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.244 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.244 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:36.244 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914563+0000", 2020-01-16T00:14:36.244 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0001e-05 2020-01-16T00:14:36.244 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.244 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.244 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:36.245 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914573+0000", 2020-01-16T00:14:36.245 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00040365600000000001 2020-01-16T00:14:36.245 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.245 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.245 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.245 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.914976+0000", 2020-01-16T00:14:36.245 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1390000000000001e-05 2020-01-16T00:14:36.246 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.246 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.246 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:36.246 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.915028+0000", 2020-01-16T00:14:36.246 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012288843000000001 2020-01-16T00:14:36.246 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.246 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.247 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.247 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927317+0000", 2020-01-16T00:14:36.247 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7871000000000002e-05 2020-01-16T00:14:36.247 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.247 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.247 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:36.247 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.927375+0000", 2020-01-16T00:14:36.247 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0062580589999999998 2020-01-16T00:14:36.248 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.248 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.248 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.248 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933633+0000", 2020-01-16T00:14:36.248 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7086999999999997e-05 2020-01-16T00:14:36.248 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.249 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.249 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:36.249 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933690+0000", 2020-01-16T00:14:36.249 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020855799999999999 2020-01-16T00:14:36.249 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.249 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.249 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:36.250 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.933898+0000", 2020-01-16T00:14:36.250 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019342986 2020-01-16T00:14:36.250 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.250 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:36.250 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.250 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.250 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.251 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:981 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:14:36.251 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:14:36.251 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 117.897921419, 2020-01-16T00:14:36.251 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021222494000000001, 2020-01-16T00:14:36.251 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:36.251 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:36.251 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:36.252 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:36.252 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:36.252 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 981 2020-01-16T00:14:36.252 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.252 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:36.252 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.252 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:36.253 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:14:36.253 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:36.253 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.253 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.253 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:36.253 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315332+0000", 2020-01-16T00:14:36.254 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999948 2020-01-16T00:14:36.254 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.254 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.254 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:36.254 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315327+0000", 2020-01-16T00:14:36.254 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.508e-05 2020-01-16T00:14:36.254 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.255 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.255 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:36.255 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315352+0000", 2020-01-16T00:14:36.255 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1183999999999998e-05 2020-01-16T00:14:36.255 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.255 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.255 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:36.256 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315383+0000", 2020-01-16T00:14:36.256 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7180000000000002e-05 2020-01-16T00:14:36.256 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.256 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.256 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:36.256 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315401+0000", 2020-01-16T00:14:36.256 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017290599999999999 2020-01-16T00:14:36.257 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.257 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.257 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.257 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315574+0000", 2020-01-16T00:14:36.257 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00047908100000000002 2020-01-16T00:14:36.257 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.257 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.258 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:36.258 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316053+0000", 2020-01-16T00:14:36.258 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011808947 2020-01-16T00:14:36.258 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.258 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.258 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.258 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.327862+0000", 2020-01-16T00:14:36.259 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010067499999999999 2020-01-16T00:14:36.259 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.259 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.259 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:36.259 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.327962+0000", 2020-01-16T00:14:36.259 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0074745139999999998 2020-01-16T00:14:36.259 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.259 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.264 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.264 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.335437+0000", 2020-01-16T00:14:36.265 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000103403 2020-01-16T00:14:36.265 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.265 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.265 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:36.265 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.335540+0000", 2020-01-16T00:14:36.265 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001014544 2020-01-16T00:14:36.265 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.266 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.266 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:36.266 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336555+0000", 2020-01-16T00:14:36.266 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021222494000000001 2020-01-16T00:14:36.266 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.266 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:36.266 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.266 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.267 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.267 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:982 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:14:36.267 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:14:36.267 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 117.897818895, 2020-01-16T00:14:36.268 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.022087721000000001, 2020-01-16T00:14:36.269 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:36.269 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:36.269 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:36.269 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:36.269 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:36.269 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 982 2020-01-16T00:14:36.270 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.270 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:36.270 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.270 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:36.270 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:14:36.270 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:36.270 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.270 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.271 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:36.271 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315435+0000", 2020-01-16T00:14:36.271 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:36.271 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.271 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.271 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:36.271 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315433+0000", 2020-01-16T00:14:36.272 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1347e-05 2020-01-16T00:14:36.272 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.272 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.272 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:36.272 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315445+0000", 2020-01-16T00:14:36.272 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4525999999999999e-05 2020-01-16T00:14:36.272 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.272 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.273 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:36.273 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315459+0000", 2020-01-16T00:14:36.273 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.024e-05 2020-01-16T00:14:36.273 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.273 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.273 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:36.273 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315470+0000", 2020-01-16T00:14:36.274 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000736468 2020-01-16T00:14:36.274 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.274 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.274 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.274 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316206+0000", 2020-01-16T00:14:36.274 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000132967 2020-01-16T00:14:36.274 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.274 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.275 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:36.275 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316339+0000", 2020-01-16T00:14:36.275 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011701717 2020-01-16T00:14:36.275 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.275 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.275 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.275 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328041+0000", 2020-01-16T00:14:36.276 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000120339 2020-01-16T00:14:36.276 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.276 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.276 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:36.276 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328161+0000", 2020-01-16T00:14:36.276 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0084341260000000001 2020-01-16T00:14:36.276 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.276 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.277 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.277 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336595+0000", 2020-01-16T00:14:36.277 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5247999999999993e-05 2020-01-16T00:14:36.277 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.277 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.277 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:36.277 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.336690+0000", 2020-01-16T00:14:36.278 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00083204000000000004 2020-01-16T00:14:36.278 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.278 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.278 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:36.278 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337522+0000", 2020-01-16T00:14:36.278 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.022087721000000001 2020-01-16T00:14:36.278 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.279 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:36.279 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.279 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.279 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.279 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:983 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~2722701,checksum] snapc 0=[] ondisk+read+known_if_redirected e24)", 2020-01-16T00:14:36.279 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:14:36.279 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 117.89769918499999, 2020-01-16T00:14:36.279 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023363387999999999, 2020-01-16T00:14:36.280 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:36.280 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:36.280 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:36.280 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:36.281 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:36.281 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 983 2020-01-16T00:14:36.281 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.281 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:36.281 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.282 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:36.282 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:14:36.282 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:36.282 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.282 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.282 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:36.282 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315554+0000", 2020-01-16T00:14:36.282 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:36.283 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.283 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.283 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:36.283 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315553+0000", 2020-01-16T00:14:36.283 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1760000000000004e-06 2020-01-16T00:14:36.283 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.283 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.284 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:36.284 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315560+0000", 2020-01-16T00:14:36.284 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3587000000000001e-05 2020-01-16T00:14:36.284 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.284 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.284 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:36.284 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315574+0000", 2020-01-16T00:14:36.285 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7800000000000003e-06 2020-01-16T00:14:36.285 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.285 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.285 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:36.285 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.315581+0000", 2020-01-16T00:14:36.285 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000823024 2020-01-16T00:14:36.285 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.285 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.286 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.286 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316404+0000", 2020-01-16T00:14:36.286 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2413e-05 2020-01-16T00:14:36.286 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.286 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.286 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:36.286 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.316466+0000", 2020-01-16T00:14:36.287 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011738561999999999 2020-01-16T00:14:36.287 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.287 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.287 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.287 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328205+0000", 2020-01-16T00:14:36.287 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6783999999999997e-05 2020-01-16T00:14:36.287 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.287 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.288 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:36.288 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.328262+0000", 2020-01-16T00:14:36.288 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0093022560000000001 2020-01-16T00:14:36.288 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.288 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.288 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.288 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337564+0000", 2020-01-16T00:14:36.289 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000183151 2020-01-16T00:14:36.289 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.289 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.289 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:36.289 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.337747+0000", 2020-01-16T00:14:36.289 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011707449999999999 2020-01-16T00:14:36.289 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.289 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.290 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:36.290 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.338918+0000", 2020-01-16T00:14:36.290 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023363387999999999 2020-01-16T00:14:36.290 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.290 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:36.290 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.290 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.291 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.291 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:986 3.9 3:93dfe6f1:::smithi12313934-40:head [write 2941380~419987] snapc 0=[] ondisk+write+known_if_redirected e24)", 2020-01-16T00:14:36.291 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:14:36.291 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 117.732399409, 2020-01-16T00:14:36.291 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016978400000000001, 2020-01-16T00:14:36.291 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:36.291 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:36.291 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:36.292 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:36.292 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:36.292 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 986 2020-01-16T00:14:36.292 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.292 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:36.292 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.292 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:36.293 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:14:36.293 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:36.293 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.293 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.293 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:36.293 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480854+0000", 2020-01-16T00:14:36.293 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:36.294 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.294 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.294 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:36.294 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.480852+0000", 2020-01-16T00:14:36.294 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042503899999999999 2020-01-16T00:14:36.294 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.294 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.294 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:36.295 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481277+0000", 2020-01-16T00:14:36.295 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3097e-05 2020-01-16T00:14:36.295 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.295 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.295 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:36.295 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481290+0000", 2020-01-16T00:14:36.295 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8460000000000002e-06 2020-01-16T00:14:36.296 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.296 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.296 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:36.296 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481297+0000", 2020-01-16T00:14:36.296 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8443999999999999e-05 2020-01-16T00:14:36.296 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.296 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.296 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.297 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481346+0000", 2020-01-16T00:14:36.297 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022115399999999999 2020-01-16T00:14:36.297 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.297 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.297 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:36.297 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481567+0000", 2020-01-16T00:14:36.297 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012531502 2020-01-16T00:14:36.298 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.298 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.298 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.298 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494098+0000", 2020-01-16T00:14:36.298 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.4462999999999993e-05 2020-01-16T00:14:36.298 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.298 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.299 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:36.299 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494193+0000", 2020-01-16T00:14:36.299 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017600999999999999 2020-01-16T00:14:36.299 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.299 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.299 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:14:36.299 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.494369+0000", 2020-01-16T00:14:36.299 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0031739619999999998 2020-01-16T00:14:36.300 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.300 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.300 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:36.300 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497543+0000", 2020-01-16T00:14:36.300 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000175107 2020-01-16T00:14:36.300 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.300 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.301 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:36.301 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497718+0000", 2020-01-16T00:14:36.301 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5207000000000003e-05 2020-01-16T00:14:36.301 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.301 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.301 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:36.301 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497763+0000", 2020-01-16T00:14:36.301 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7042999999999998e-05 2020-01-16T00:14:36.302 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.302 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.302 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:36.302 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497800+0000", 2020-01-16T00:14:36.302 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2628e-05 2020-01-16T00:14:36.302 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.303 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.303 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:36.303 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497833+0000", 2020-01-16T00:14:36.303 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016978400000000001 2020-01-16T00:14:36.303 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.303 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:36.303 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.303 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.304 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.304 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:988 3.9 3:93dfe6f1:::smithi12313934-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e24)", 2020-01-16T00:14:36.304 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:14:36.304 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 117.731913945, 2020-01-16T00:14:36.305 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017221692, 2020-01-16T00:14:36.305 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:36.305 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:36.305 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:36.305 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:36.305 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:36.306 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 988 2020-01-16T00:14:36.306 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.306 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:36.306 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.306 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:36.306 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:14:36.306 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:36.307 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.307 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.307 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:36.307 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481340+0000", 2020-01-16T00:14:36.307 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:36.307 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.307 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.308 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:36.308 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481339+0000", 2020-01-16T00:14:36.308 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3089999999999997e-06 2020-01-16T00:14:36.308 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.308 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.308 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:36.308 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481343+0000", 2020-01-16T00:14:36.308 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2230000000000004e-06 2020-01-16T00:14:36.309 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.309 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.309 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:36.309 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481350+0000", 2020-01-16T00:14:36.309 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.319e-06 2020-01-16T00:14:36.309 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.310 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.310 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:36.310 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481356+0000", 2020-01-16T00:14:36.310 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000321544 2020-01-16T00:14:36.310 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.310 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.310 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.311 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481678+0000", 2020-01-16T00:14:36.311 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6375999999999999e-05 2020-01-16T00:14:36.311 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.311 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.311 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:36.311 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.481714+0000", 2020-01-16T00:14:36.311 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015557666 2020-01-16T00:14:36.312 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.312 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.312 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.312 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497272+0000", 2020-01-16T00:14:36.312 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012791300000000001 2020-01-16T00:14:36.312 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.312 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.312 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:36.313 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.497399+0000", 2020-01-16T00:14:36.313 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00076944600000000002 2020-01-16T00:14:36.314 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.314 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.314 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.314 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498169+0000", 2020-01-16T00:14:36.314 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018306799999999999 2020-01-16T00:14:36.314 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.314 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.315 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:36.315 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498352+0000", 2020-01-16T00:14:36.315 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020935999999999999 2020-01-16T00:14:36.315 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.315 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.315 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:36.315 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:38.498561+0000", 2020-01-16T00:14:36.316 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017221692 2020-01-16T00:14:36.316 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.316 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:36.316 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.316 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.316 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.316 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1239 3.3 3:c204aa4f:::smithi12313934-45:head [setxattr _header (56) in=7b,truncate 2000033] snapc 0=[] ondisk+write+known_if_redirected e26)", 2020-01-16T00:14:36.317 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:14:36.317 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 113.622840126, 2020-01-16T00:14:36.317 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019161661999999999, 2020-01-16T00:14:36.317 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:36.317 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:36.317 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:36.317 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:36.317 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:36.318 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1239 2020-01-16T00:14:36.318 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.318 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:36.318 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.318 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:36.318 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:14:36.318 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:36.319 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.319 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.319 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:36.319 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590413+0000", 2020-01-16T00:14:36.319 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:36.319 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.319 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.319 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:36.320 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590411+0000", 2020-01-16T00:14:36.320 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3879999999999999e-05 2020-01-16T00:14:36.320 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.320 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.320 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:36.320 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590425+0000", 2020-01-16T00:14:36.320 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6542999999999998e-05 2020-01-16T00:14:36.321 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.321 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.321 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:36.321 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590471+0000", 2020-01-16T00:14:36.321 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8907999999999999e-05 2020-01-16T00:14:36.322 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.322 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.322 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:36.322 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590490+0000", 2020-01-16T00:14:36.322 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0045691960000000002 2020-01-16T00:14:36.322 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.322 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.322 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.323 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.595059+0000", 2020-01-16T00:14:36.323 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0035638940000000002 2020-01-16T00:14:36.323 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.323 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.323 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:36.323 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598623+0000", 2020-01-16T00:14:36.323 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0077558310000000004 2020-01-16T00:14:36.324 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.324 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.324 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.324 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606379+0000", 2020-01-16T00:14:36.324 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012171999999999999 2020-01-16T00:14:36.324 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.324 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.324 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:36.325 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606501+0000", 2020-01-16T00:14:36.325 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013144200000000001 2020-01-16T00:14:36.325 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.325 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.325 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2020-01-16T00:14:36.325 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.606632+0000", 2020-01-16T00:14:36.325 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001288565 2020-01-16T00:14:36.326 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.326 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.326 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:36.326 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607921+0000", 2020-01-16T00:14:36.326 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033399299999999999 2020-01-16T00:14:36.326 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.326 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.326 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:36.327 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.608255+0000", 2020-01-16T00:14:36.327 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012280870000000001 2020-01-16T00:14:36.327 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.327 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.327 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:36.328 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609483+0000", 2020-01-16T00:14:36.328 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2286000000000001e-05 2020-01-16T00:14:36.328 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.328 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.328 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:36.328 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609525+0000", 2020-01-16T00:14:36.328 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0121999999999999e-05 2020-01-16T00:14:36.328 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.329 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.329 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:36.329 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609575+0000", 2020-01-16T00:14:36.329 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019161661999999999 2020-01-16T00:14:36.329 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.329 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:36.329 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.330 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.330 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.330 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1240 3.3 3:c204aa4f:::smithi12313934-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e26)", 2020-01-16T00:14:36.330 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:14:36.330 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 113.62273993700001, 2020-01-16T00:14:36.331 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019268828000000002, 2020-01-16T00:14:36.331 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:36.331 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:36.331 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:36.331 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:36.331 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:36.332 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1240 2020-01-16T00:14:36.332 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.332 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:36.332 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.332 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:36.332 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:14:36.332 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:36.333 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.333 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.333 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:36.333 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590514+0000", 2020-01-16T00:14:36.333 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2020-01-16T00:14:36.333 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.333 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.334 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:36.334 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590512+0000", 2020-01-16T00:14:36.334 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0525e-05 2020-01-16T00:14:36.334 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.334 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.335 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:36.335 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590522+0000", 2020-01-16T00:14:36.335 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6418000000000001e-05 2020-01-16T00:14:36.335 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.335 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.335 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:36.335 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590539+0000", 2020-01-16T00:14:36.335 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0679999999999999e-05 2020-01-16T00:14:36.336 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.336 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.336 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:36.336 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.590549+0000", 2020-01-16T00:14:36.336 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0083626789999999996 2020-01-16T00:14:36.336 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.336 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.337 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.337 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.598912+0000", 2020-01-16T00:14:36.337 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013344 2020-01-16T00:14:36.337 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.337 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.337 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:36.337 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.599045+0000", 2020-01-16T00:14:36.337 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0084002269999999997 2020-01-16T00:14:36.338 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.338 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.338 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.338 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607446+0000", 2020-01-16T00:14:36.338 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000109873 2020-01-16T00:14:36.338 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.338 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.339 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:36.339 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.607555+0000", 2020-01-16T00:14:36.339 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0020573309999999999 2020-01-16T00:14:36.339 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.339 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.339 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.339 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609613+0000", 2020-01-16T00:14:36.339 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6351e-05 2020-01-16T00:14:36.340 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.340 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.340 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:36.340 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609699+0000", 2020-01-16T00:14:36.340 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3374999999999994e-05 2020-01-16T00:14:36.340 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.341 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.341 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:36.341 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:42.609782+0000", 2020-01-16T00:14:36.341 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019268828000000002 2020-01-16T00:14:36.341 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.341 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:36.342 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.342 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.342 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.342 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3124 3.9 3:9fc687b2:::smithi12313934-44:head [write 1903628~435462] snapc 0=[] ondisk+write+known_if_redirected e44)", 2020-01-16T00:14:36.342 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.276145+0000", 2020-01-16T00:14:36.343 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 70.937108215999999, 2020-01-16T00:14:36.343 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017361898000000001, 2020-01-16T00:14:36.343 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:36.343 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:36.343 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:36.343 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:36.343 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:36.344 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3124 2020-01-16T00:14:36.344 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.344 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:36.344 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.344 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:36.344 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276145+0000", 2020-01-16T00:14:36.344 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:36.344 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.345 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.345 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:36.345 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276145+0000", 2020-01-16T00:14:36.345 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:36.345 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.345 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.345 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:36.346 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276143+0000", 2020-01-16T00:14:36.346 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042136099999999998 2020-01-16T00:14:36.346 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.346 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.346 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:36.346 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276564+0000", 2020-01-16T00:14:36.346 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.349e-05 2020-01-16T00:14:36.346 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.347 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.347 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:36.347 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276578+0000", 2020-01-16T00:14:36.347 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2469999999999999e-06 2020-01-16T00:14:36.347 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.347 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.347 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:36.348 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276583+0000", 2020-01-16T00:14:36.348 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4400000000000001e-05 2020-01-16T00:14:36.348 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.348 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.348 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.348 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276637+0000", 2020-01-16T00:14:36.348 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0046361789999999998 2020-01-16T00:14:36.348 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.349 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.349 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:36.349 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281274+0000", 2020-01-16T00:14:36.349 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0090025520000000005 2020-01-16T00:14:36.349 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.349 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.349 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.350 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.290276+0000", 2020-01-16T00:14:36.350 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011584200000000001 2020-01-16T00:14:36.350 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.350 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.350 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:36.350 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.290392+0000", 2020-01-16T00:14:36.350 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00025563099999999998 2020-01-16T00:14:36.350 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.351 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.351 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:14:36.351 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.290648+0000", 2020-01-16T00:14:36.351 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0026741360000000001 2020-01-16T00:14:36.351 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.351 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.352 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:36.352 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293322+0000", 2020-01-16T00:14:36.352 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9764000000000002e-05 2020-01-16T00:14:36.352 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.352 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.353 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:36.353 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293421+0000", 2020-01-16T00:14:36.353 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4779000000000001e-05 2020-01-16T00:14:36.353 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.353 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.353 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:36.353 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293446+0000", 2020-01-16T00:14:36.353 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4628000000000001e-05 2020-01-16T00:14:36.354 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.354 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.354 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:36.354 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293481+0000", 2020-01-16T00:14:36.354 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.6444999999999998e-05 2020-01-16T00:14:36.354 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.354 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.355 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:36.355 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293507+0000", 2020-01-16T00:14:36.355 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017361898000000001 2020-01-16T00:14:36.355 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.355 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:36.355 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.355 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.355 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.356 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3125 3.9 3:9fc687b2:::smithi12313934-44:head [setxattr _header (56) in=7b,truncate 2386660] snapc 0=[] ondisk+write+known_if_redirected e44)", 2020-01-16T00:14:36.356 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.276594+0000", 2020-01-16T00:14:36.356 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 70.936659444, 2020-01-16T00:14:36.356 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017760560000000002, 2020-01-16T00:14:36.356 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:36.356 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:36.356 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:36.356 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:36.357 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:36.357 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3125 2020-01-16T00:14:36.357 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.357 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:36.357 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.357 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:36.357 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276594+0000", 2020-01-16T00:14:36.358 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:36.358 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.358 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.358 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:36.358 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276594+0000", 2020-01-16T00:14:36.358 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:36.358 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.359 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.359 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:36.359 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276593+0000", 2020-01-16T00:14:36.359 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7320000000000003e-06 2020-01-16T00:14:36.359 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.359 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.359 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:36.359 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276599+0000", 2020-01-16T00:14:36.360 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3302000000000001e-05 2020-01-16T00:14:36.360 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.360 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.360 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:36.360 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276612+0000", 2020-01-16T00:14:36.360 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1659999999999998e-06 2020-01-16T00:14:36.360 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.360 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.361 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:36.361 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276615+0000", 2020-01-16T00:14:36.361 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0047328450000000003 2020-01-16T00:14:36.361 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.361 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.361 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.362 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281348+0000", 2020-01-16T00:14:36.362 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3966000000000003e-05 2020-01-16T00:14:36.362 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.362 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.362 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:36.362 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281422+0000", 2020-01-16T00:14:36.363 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01082613 2020-01-16T00:14:36.363 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.363 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.363 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.363 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.292248+0000", 2020-01-16T00:14:36.363 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000148617 2020-01-16T00:14:36.363 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.364 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.364 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:36.364 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.292397+0000", 2020-01-16T00:14:36.364 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019936300000000001 2020-01-16T00:14:36.364 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.364 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.364 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:14:36.364 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.292596+0000", 2020-01-16T00:14:36.365 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011208870000000001 2020-01-16T00:14:36.365 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.365 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.365 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:36.365 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293717+0000", 2020-01-16T00:14:36.365 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00039491599999999998 2020-01-16T00:14:36.365 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.365 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.366 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:36.366 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294112+0000", 2020-01-16T00:14:36.366 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000129326 2020-01-16T00:14:36.366 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.366 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.366 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:36.366 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294241+0000", 2020-01-16T00:14:36.367 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5586000000000001e-05 2020-01-16T00:14:36.367 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.367 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.367 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:36.367 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294277+0000", 2020-01-16T00:14:36.367 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7893999999999997e-05 2020-01-16T00:14:36.367 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.368 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.368 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:36.368 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294355+0000", 2020-01-16T00:14:36.368 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017760560000000002 2020-01-16T00:14:36.368 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.369 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:36.369 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.369 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.369 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.369 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3126 3.9 3:9fc687b2:::smithi12313934-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e44)", 2020-01-16T00:14:36.369 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.276634+0000", 2020-01-16T00:14:36.369 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 70.936619729, 2020-01-16T00:14:36.370 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017942583000000002, 2020-01-16T00:14:36.370 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:36.370 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:36.370 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:36.370 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:36.370 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:36.370 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3126 2020-01-16T00:14:36.370 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.371 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:36.371 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.371 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:36.371 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276634+0000", 2020-01-16T00:14:36.371 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:36.371 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.372 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.372 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:36.372 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276634+0000", 2020-01-16T00:14:36.372 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:36.372 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.372 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.373 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:36.373 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276633+0000", 2020-01-16T00:14:36.373 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0769999999999999e-06 2020-01-16T00:14:36.373 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.373 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.373 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:36.374 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276636+0000", 2020-01-16T00:14:36.374 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5390000000000001e-06 2020-01-16T00:14:36.374 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.374 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.374 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:36.374 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276641+0000", 2020-01-16T00:14:36.374 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9969999999999999e-06 2020-01-16T00:14:36.374 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.375 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.375 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:36.375 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276644+0000", 2020-01-16T00:14:36.375 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0048246970000000002 2020-01-16T00:14:36.375 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.375 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.375 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.376 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281469+0000", 2020-01-16T00:14:36.376 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8723999999999999e-05 2020-01-16T00:14:36.376 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.376 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.376 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:36.376 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.281567+0000", 2020-01-16T00:14:36.376 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011633447 2020-01-16T00:14:36.376 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.377 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.377 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.377 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293201+0000", 2020-01-16T00:14:36.377 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6796000000000002e-05 2020-01-16T00:14:36.377 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.377 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.377 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:36.378 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.293268+0000", 2020-01-16T00:14:36.378 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001125247 2020-01-16T00:14:36.378 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.378 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.379 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.379 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294393+0000", 2020-01-16T00:14:36.379 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.6082000000000006e-05 2020-01-16T00:14:36.379 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.379 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.379 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:36.379 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294489+0000", 2020-01-16T00:14:36.379 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7515000000000005e-05 2020-01-16T00:14:36.380 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.380 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.380 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:36.380 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.294576+0000", 2020-01-16T00:14:36.380 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017942583000000002 2020-01-16T00:14:36.380 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.380 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:36.381 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.381 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.381 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.381 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3204 3.9 3:93dfe6f1:::smithi12313934-40:head [setxattr _header (56) in=7b,truncate 2554730] snapc 0=[] ondisk+write+known_if_redirected e45)", 2020-01-16T00:14:36.381 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.568544+0000", 2020-01-16T00:14:36.381 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 66.644709508999995, 2020-01-16T00:14:36.381 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.026260867, 2020-01-16T00:14:36.381 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:36.382 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:36.382 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:36.383 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:36.383 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:36.383 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3204 2020-01-16T00:14:36.383 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.383 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:36.383 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.383 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:36.384 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568544+0000", 2020-01-16T00:14:36.384 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:36.384 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.384 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.384 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:36.384 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568544+0000", 2020-01-16T00:14:36.384 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:36.384 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.385 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.385 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:36.385 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568543+0000", 2020-01-16T00:14:36.385 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0810000000000001e-06 2020-01-16T00:14:36.385 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.385 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.386 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:36.386 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568548+0000", 2020-01-16T00:14:36.386 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6950000000000002e-06 2020-01-16T00:14:36.386 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.386 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.386 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:36.386 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568552+0000", 2020-01-16T00:14:36.387 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.875e-06 2020-01-16T00:14:36.387 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.387 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.387 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:36.387 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568555+0000", 2020-01-16T00:14:36.387 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0033130849999999999 2020-01-16T00:14:36.387 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.388 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.388 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.388 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.571868+0000", 2020-01-16T00:14:36.388 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0026014609999999998 2020-01-16T00:14:36.388 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.388 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.388 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:36.388 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.574469+0000", 2020-01-16T00:14:36.389 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010261381999999999 2020-01-16T00:14:36.389 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.389 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.389 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.389 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.584731+0000", 2020-01-16T00:14:36.389 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019704 2020-01-16T00:14:36.389 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.390 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.390 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:36.390 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.584928+0000", 2020-01-16T00:14:36.390 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000280869 2020-01-16T00:14:36.390 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.390 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.390 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:14:36.391 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.585209+0000", 2020-01-16T00:14:36.391 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002273859 2020-01-16T00:14:36.391 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.391 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.391 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:36.391 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.587483+0000", 2020-01-16T00:14:36.391 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3356000000000001e-05 2020-01-16T00:14:36.391 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.392 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.392 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:36.392 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.587506+0000", 2020-01-16T00:14:36.392 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0071864939999999999 2020-01-16T00:14:36.392 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.392 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.392 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:36.393 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594692+0000", 2020-01-16T00:14:36.393 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5475000000000001e-05 2020-01-16T00:14:36.393 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.393 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.393 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:36.393 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594738+0000", 2020-01-16T00:14:36.393 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7070000000000007e-05 2020-01-16T00:14:36.394 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.394 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.394 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:36.394 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594805+0000", 2020-01-16T00:14:36.394 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.026260867 2020-01-16T00:14:36.394 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.394 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:36.395 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.395 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.395 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.395 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3205 3.9 3:93dfe6f1:::smithi12313934-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e45)", 2020-01-16T00:14:36.395 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.568599+0000", 2020-01-16T00:14:36.395 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 66.644654574, 2020-01-16T00:14:36.395 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.026406882, 2020-01-16T00:14:36.396 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:36.396 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:36.396 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:36.396 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:36.396 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:36.396 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3205 2020-01-16T00:14:36.397 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.397 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:36.397 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.397 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:36.397 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568599+0000", 2020-01-16T00:14:36.397 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:36.397 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.397 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.398 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:36.398 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568599+0000", 2020-01-16T00:14:36.398 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:36.398 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.398 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.398 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:36.398 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568598+0000", 2020-01-16T00:14:36.399 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3939999999999998e-06 2020-01-16T00:14:36.399 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.399 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.399 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:36.399 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568603+0000", 2020-01-16T00:14:36.399 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9391000000000001e-05 2020-01-16T00:14:36.399 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.400 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.400 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:36.400 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568622+0000", 2020-01-16T00:14:36.400 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9599999999999997e-06 2020-01-16T00:14:36.400 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.400 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.400 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:36.400 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.568628+0000", 2020-01-16T00:14:36.401 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0058679700000000001 2020-01-16T00:14:36.401 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.401 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.401 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.401 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.574496+0000", 2020-01-16T00:14:36.401 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5528999999999998e-05 2020-01-16T00:14:36.401 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.402 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.402 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:36.402 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.574532+0000", 2020-01-16T00:14:36.402 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011986749 2020-01-16T00:14:36.402 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.402 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.402 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.403 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.586518+0000", 2020-01-16T00:14:36.403 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016522600000000001 2020-01-16T00:14:36.403 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.403 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.403 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:36.403 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.586684+0000", 2020-01-16T00:14:36.403 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0081693890000000009 2020-01-16T00:14:36.404 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.404 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.404 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.404 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594853+0000", 2020-01-16T00:14:36.404 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7082000000000004e-05 2020-01-16T00:14:36.404 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.404 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.404 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:36.405 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.594940+0000", 2020-01-16T00:14:36.405 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5789000000000003e-05 2020-01-16T00:14:36.405 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.405 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.405 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:36.405 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.595006+0000", 2020-01-16T00:14:36.405 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.026406882 2020-01-16T00:14:36.406 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.406 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:36.406 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.406 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.406 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.406 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3219 3.3 3:c478bfe8:::smithi12313934-10:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:36.406 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.580957+0000", 2020-01-16T00:14:36.407 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 66.632297058000006, 2020-01-16T00:14:36.407 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016807971000000001, 2020-01-16T00:14:36.407 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:36.407 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:36.407 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:36.407 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:36.407 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:36.408 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3219 2020-01-16T00:14:36.408 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.408 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:36.408 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.408 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:36.408 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580957+0000", 2020-01-16T00:14:36.408 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:36.408 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.409 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.409 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:36.409 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580957+0000", 2020-01-16T00:14:36.409 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:36.409 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.409 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.409 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:36.409 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580955+0000", 2020-01-16T00:14:36.410 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5120000000000002e-06 2020-01-16T00:14:36.410 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.410 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.410 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:36.410 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580963+0000", 2020-01-16T00:14:36.410 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6799999999999993e-06 2020-01-16T00:14:36.410 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.411 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.411 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:36.411 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580971+0000", 2020-01-16T00:14:36.411 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3109999999999996e-06 2020-01-16T00:14:36.411 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.411 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.411 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:36.412 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580976+0000", 2020-01-16T00:14:36.412 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014306146 2020-01-16T00:14:36.412 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.412 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.412 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.412 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.595282+0000", 2020-01-16T00:14:36.412 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.1883000000000001e-05 2020-01-16T00:14:36.412 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.413 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.413 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:36.413 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.595374+0000", 2020-01-16T00:14:36.413 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023904989999999999 2020-01-16T00:14:36.413 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.413 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.413 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:36.414 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.597765+0000", 2020-01-16T00:14:36.414 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016807971000000001 2020-01-16T00:14:36.414 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.414 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:36.414 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.414 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.414 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.414 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3220 3.3 3:c478bfe8:::smithi12313934-10:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:36.415 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.580982+0000", 2020-01-16T00:14:36.415 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 66.632272009999994, 2020-01-16T00:14:36.415 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017920330000000002, 2020-01-16T00:14:36.415 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:36.415 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:36.415 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:36.415 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:36.416 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:36.416 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3220 2020-01-16T00:14:36.416 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.416 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:36.416 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.416 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:36.416 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580982+0000", 2020-01-16T00:14:36.416 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:36.417 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.417 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.417 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:36.417 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580982+0000", 2020-01-16T00:14:36.417 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:36.417 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.417 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.418 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:36.418 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580981+0000", 2020-01-16T00:14:36.418 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0380000000000001e-06 2020-01-16T00:14:36.418 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.418 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.418 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:36.418 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.580985+0000", 2020-01-16T00:14:36.419 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8929000000000001e-05 2020-01-16T00:14:36.419 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.419 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.419 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:36.419 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581004+0000", 2020-01-16T00:14:36.419 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7339999999999999e-06 2020-01-16T00:14:36.419 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.419 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.420 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:36.420 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581009+0000", 2020-01-16T00:14:36.420 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01679812 2020-01-16T00:14:36.420 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.420 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.420 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.420 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.597807+0000", 2020-01-16T00:14:36.420 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013333400000000001 2020-01-16T00:14:36.421 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.421 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.421 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:36.421 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.597940+0000", 2020-01-16T00:14:36.421 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000961899 2020-01-16T00:14:36.421 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.421 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.422 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:36.422 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.598902+0000", 2020-01-16T00:14:36.422 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017920330000000002 2020-01-16T00:14:36.422 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.422 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:36.422 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.422 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.422 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.423 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3221 3.3 3:c478bfe8:::smithi12313934-10:head [read 0~2403467,checksum] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:36.423 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:29.581014+0000", 2020-01-16T00:14:36.423 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 66.632239661, 2020-01-16T00:14:36.423 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018784551999999999, 2020-01-16T00:14:36.423 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:36.423 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:36.423 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:36.424 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:36.424 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:36.424 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3221 2020-01-16T00:14:36.424 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.424 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:36.424 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.424 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:36.424 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581014+0000", 2020-01-16T00:14:36.425 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:36.425 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.425 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.425 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:36.425 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581014+0000", 2020-01-16T00:14:36.425 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:36.425 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.426 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.426 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:36.426 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581013+0000", 2020-01-16T00:14:36.426 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9390000000000002e-06 2020-01-16T00:14:36.426 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.426 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.426 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:36.426 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581019+0000", 2020-01-16T00:14:36.427 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1131000000000001e-05 2020-01-16T00:14:36.427 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.427 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.427 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:36.427 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581030+0000", 2020-01-16T00:14:36.427 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.55e-06 2020-01-16T00:14:36.427 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.427 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.428 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:36.428 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.581037+0000", 2020-01-16T00:14:36.428 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017890321000000001 2020-01-16T00:14:36.428 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.428 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.428 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.428 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.598927+0000", 2020-01-16T00:14:36.428 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000105999 2020-01-16T00:14:36.429 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.429 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.429 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:36.429 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.599033+0000", 2020-01-16T00:14:36.429 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00076542400000000001 2020-01-16T00:14:36.429 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.429 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.430 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:36.430 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:29.599798+0000", 2020-01-16T00:14:36.430 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018784551999999999 2020-01-16T00:14:36.430 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.430 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:36.430 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.430 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.431 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.431 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4152 3.9 3:93dfe6f1:::smithi12313934-40:head [setxattr _header (56) in=7b,truncate 2806868] snapc 0=[] ondisk+write+known_if_redirected e53)", 2020-01-16T00:14:36.431 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:53.810986+0000", 2020-01-16T00:14:36.431 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 42.402267352000003, 2020-01-16T00:14:36.431 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016830279, 2020-01-16T00:14:36.431 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:36.431 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:36.431 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:36.432 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:36.432 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:36.432 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4152 2020-01-16T00:14:36.432 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.432 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:36.432 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.432 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:36.433 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.810986+0000", 2020-01-16T00:14:36.433 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:36.433 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.433 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.433 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:36.433 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.810986+0000", 2020-01-16T00:14:36.433 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:36.434 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.434 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.434 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:36.434 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.810985+0000", 2020-01-16T00:14:36.434 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.7529999999999996e-06 2020-01-16T00:14:36.434 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.434 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.435 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:36.435 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.810995+0000", 2020-01-16T00:14:36.435 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7907999999999999e-05 2020-01-16T00:14:36.435 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.435 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.435 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:36.435 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811013+0000", 2020-01-16T00:14:36.436 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5829999999999997e-06 2020-01-16T00:14:36.436 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.436 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.436 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:36.436 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811018+0000", 2020-01-16T00:14:36.436 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0029165229999999999 2020-01-16T00:14:36.436 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.436 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.437 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.437 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.813935+0000", 2020-01-16T00:14:36.437 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2583000000000001e-05 2020-01-16T00:14:36.437 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.437 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.437 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:36.437 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.813988+0000", 2020-01-16T00:14:36.438 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010581012000000001 2020-01-16T00:14:36.438 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.438 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.438 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.438 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.824569+0000", 2020-01-16T00:14:36.438 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013191900000000001 2020-01-16T00:14:36.438 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.439 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.439 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:36.439 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.824700+0000", 2020-01-16T00:14:36.439 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022298000000000001 2020-01-16T00:14:36.439 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.439 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.439 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,7", 2020-01-16T00:14:36.439 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.824923+0000", 2020-01-16T00:14:36.440 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010694260000000001 2020-01-16T00:14:36.440 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.440 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.440 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:36.440 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.825993+0000", 2020-01-16T00:14:36.440 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000116939 2020-01-16T00:14:36.440 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.441 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.441 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:36.441 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.826110+0000", 2020-01-16T00:14:36.441 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0015606400000000001 2020-01-16T00:14:36.441 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.441 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.441 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:36.441 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.827670+0000", 2020-01-16T00:14:36.442 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7318999999999997e-05 2020-01-16T00:14:36.442 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.442 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.442 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:36.442 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.827738+0000", 2020-01-16T00:14:36.442 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.873e-05 2020-01-16T00:14:36.442 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.443 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.443 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:36.443 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.827817+0000", 2020-01-16T00:14:36.443 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016830279 2020-01-16T00:14:36.443 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.443 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:36.443 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.443 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.444 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.444 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4153 3.9 3:93dfe6f1:::smithi12313934-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e53)", 2020-01-16T00:14:36.444 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:53.811034+0000", 2020-01-16T00:14:36.444 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 42.402219264000003, 2020-01-16T00:14:36.444 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017095406, 2020-01-16T00:14:36.444 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:36.444 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:36.445 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:36.445 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:36.445 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:36.445 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4153 2020-01-16T00:14:36.445 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.445 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:36.445 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.446 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:36.446 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811034+0000", 2020-01-16T00:14:36.446 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:36.446 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.446 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.446 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:36.446 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811034+0000", 2020-01-16T00:14:36.446 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:36.447 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.447 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.447 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:36.447 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811034+0000", 2020-01-16T00:14:36.447 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9060000000000001e-06 2020-01-16T00:14:36.447 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.447 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.448 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:36.448 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811038+0000", 2020-01-16T00:14:36.448 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7710000000000001e-06 2020-01-16T00:14:36.448 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.448 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.448 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:36.448 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811045+0000", 2020-01-16T00:14:36.448 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5189999999999998e-06 2020-01-16T00:14:36.449 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.449 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.449 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:36.449 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.811052+0000", 2020-01-16T00:14:36.449 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002953991 2020-01-16T00:14:36.449 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.449 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.450 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.450 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.814006+0000", 2020-01-16T00:14:36.450 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5726999999999999e-05 2020-01-16T00:14:36.450 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.450 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.450 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:36.450 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.814041+0000", 2020-01-16T00:14:36.451 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011650031999999999 2020-01-16T00:14:36.451 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.451 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.451 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.451 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.825691+0000", 2020-01-16T00:14:36.451 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0625999999999998e-05 2020-01-16T00:14:36.451 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.451 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.452 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:36.452 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.825762+0000", 2020-01-16T00:14:36.452 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0020982370000000002 2020-01-16T00:14:36.452 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.452 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.452 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.452 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.827860+0000", 2020-01-16T00:14:36.452 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012736299999999999 2020-01-16T00:14:36.452 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.453 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.453 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:36.453 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.827988+0000", 2020-01-16T00:14:36.453 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000142015 2020-01-16T00:14:36.453 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.453 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.453 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:36.454 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:53.828130+0000", 2020-01-16T00:14:36.454 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017095406 2020-01-16T00:14:36.454 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.454 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:36.454 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.454 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.454 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:36.454 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:36.457 INFO:tasks.rados.rados.0.smithi123.stdout:2018: done (13 left) 2020-01-16T00:14:36.457 INFO:tasks.rados.rados.0.smithi123.stdout:2019: done (12 left) 2020-01-16T00:14:36.457 INFO:tasks.rados.rados.0.smithi123.stdout:2020: done (11 left) 2020-01-16T00:14:36.458 INFO:tasks.rados.rados.0.smithi123.stdout:2032: done (10 left) 2020-01-16T00:14:36.458 INFO:tasks.rados.rados.0.smithi123.stdout:2033: read oid 1 snap -1 2020-01-16T00:14:36.458 INFO:tasks.rados.rados.0.smithi123.stdout:2033: expect (ObjNum 729 snap 0 seq_num 729) 2020-01-16T00:14:36.458 INFO:tasks.rados.rados.0.smithi123.stdout:2034: read oid 19 snap -1 2020-01-16T00:14:36.458 INFO:tasks.rados.rados.0.smithi123.stdout:2034: expect deleted 2020-01-16T00:14:36.458 INFO:tasks.rados.rados.0.smithi123.stdout:2035: write (excl) oid 30 current snap is 0 2020-01-16T00:14:36.458 INFO:tasks.rados.rados.0.smithi123.stdout:2035: seq_num 787 ranges {427109=405322,1553564=545926,2579307=647802} 2020-01-16T00:14:36.459 INFO:tasks.rados.rados.0.smithi123.stdout:2035: writing smithi12313934-30 from 427109 to 832431 tid 1 2020-01-16T00:14:36.459 INFO:tasks.rados.rados.0.smithi123.stdout:2035: writing smithi12313934-30 from 1553564 to 2099490 tid 2 2020-01-16T00:14:36.459 INFO:tasks.rados.rados.0.smithi123.stdout:2035: writing smithi12313934-30 from 2579307 to 3227109 tid 3 2020-01-16T00:14:36.459 INFO:tasks.rados.rados.0.smithi123.stdout:2036: read oid 39 snap -1 2020-01-16T00:14:36.459 INFO:tasks.rados.rados.0.smithi123.stdout:2036: expect (ObjNum 779 snap 0 seq_num 779) 2020-01-16T00:14:36.459 INFO:tasks.rados.rados.0.smithi123.stdout:2037: read oid 43 snap -1 2020-01-16T00:14:36.459 INFO:tasks.rados.rados.0.smithi123.stdout:2037: expect (ObjNum 763 snap 0 seq_num 763) 2020-01-16T00:14:36.459 INFO:tasks.rados.rados.0.smithi123.stdout:2038: delete oid 7 current snap is 0 2020-01-16T00:14:36.460 INFO:tasks.rados.rados.0.smithi123.stdout:2022: expect (ObjNum 704 snap 0 seq_num 704) 2020-01-16T00:14:36.460 INFO:tasks.rados.rados.0.smithi123.stdout:2023: finishing write tid 1 to smithi12313934-17 2020-01-16T00:14:36.460 INFO:tasks.rados.rados.0.smithi123.stdout:2023: finishing write tid 2 to smithi12313934-17 2020-01-16T00:14:36.460 INFO:tasks.rados.rados.0.smithi123.stdout:2023: finishing write tid 4 to smithi12313934-17 2020-01-16T00:14:36.460 INFO:tasks.rados.rados.0.smithi123.stdout:2023: finishing write tid 5 to smithi12313934-17 2020-01-16T00:14:36.460 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 17 v 222 (ObjNum 782 snap 0 seq_num 782) dirty exists 2020-01-16T00:14:36.460 INFO:tasks.rados.rados.0.smithi123.stdout:2023: left oid 17 (ObjNum 782 snap 0 seq_num 782) 2020-01-16T00:14:36.461 INFO:tasks.rados.rados.0.smithi123.stdout:2024: expect (ObjNum 754 snap 0 seq_num 754) 2020-01-16T00:14:36.461 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:36.454+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:36.461 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-01-16T00:14:36.557 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:36.554+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:36.618 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:36.618 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:14:36.619 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:14:36.619 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:36.628 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-01-16T00:14:36.642 INFO:tasks.rados.rados.0.smithi123.stdout:2025: expect (ObjNum 771 snap 0 seq_num 771) 2020-01-16T00:14:36.658 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:36.658+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:36.737 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:36.738 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:14:36.738 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:14:36.738 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:14:36.738 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:36.747 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-01-16T00:14:36.759 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:36.758+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:36.860 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:36.860 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:14:36.860 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:14:36.861 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:14:36.861 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.861 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.5.13:36 2.0 2:20e40cec:::smithi12313934-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 113152~1699840] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:14:36.861 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:14:36.861 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 119.025206755, 2020-01-16T00:14:36.861 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011736636, 2020-01-16T00:14:36.861 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:36.861 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:36.862 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:36.862 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.862 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:36.862 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:14:36.862 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:36.862 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.862 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.862 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:36.863 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:14:36.863 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999952 2020-01-16T00:14:36.863 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.863 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.863 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:36.863 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831934+0000", 2020-01-16T00:14:36.863 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001034325 2020-01-16T00:14:36.864 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.864 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.864 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:36.864 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.832968+0000", 2020-01-16T00:14:36.864 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.506e-05 2020-01-16T00:14:36.864 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.864 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.865 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:36.865 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833004+0000", 2020-01-16T00:14:36.865 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3406e-05 2020-01-16T00:14:36.865 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.865 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.865 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:36.865 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833017+0000", 2020-01-16T00:14:36.865 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015477000000000001 2020-01-16T00:14:36.866 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.866 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.866 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.866 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833172+0000", 2020-01-16T00:14:36.866 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033750700000000002 2020-01-16T00:14:36.866 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.866 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.867 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:36.867 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833509+0000", 2020-01-16T00:14:36.867 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00055861400000000003 2020-01-16T00:14:36.867 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.867 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.867 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:14:36.867 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.834068+0000", 2020-01-16T00:14:36.867 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.007474545 2020-01-16T00:14:36.868 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.868 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.868 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:36.868 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.841542+0000", 2020-01-16T00:14:36.868 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00056319499999999995 2020-01-16T00:14:36.868 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.868 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.869 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:36.869 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.842106+0000", 2020-01-16T00:14:36.869 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001371017 2020-01-16T00:14:36.869 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.869 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.869 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:36.869 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843477+0000", 2020-01-16T00:14:36.870 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000123938 2020-01-16T00:14:36.870 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.870 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.870 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:36.870 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843601+0000", 2020-01-16T00:14:36.870 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5012000000000006e-05 2020-01-16T00:14:36.871 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.871 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.871 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:36.871 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843676+0000", 2020-01-16T00:14:36.871 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011736636 2020-01-16T00:14:36.871 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.871 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:36.872 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.872 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.872 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.872 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1276 3.d 3:b1521a3e:::smithi12313934-25:head [read 0~3882384,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:14:36.872 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:14:36.872 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 113.782447185, 2020-01-16T00:14:36.872 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015839157999999999, 2020-01-16T00:14:36.873 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:36.873 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:36.873 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:36.873 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:36.873 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:36.873 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1276 2020-01-16T00:14:36.873 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.874 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:36.874 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.874 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:36.874 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:14:36.874 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:36.874 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.875 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.875 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:36.875 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:14:36.875 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:36.875 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.875 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.875 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:36.875 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:14:36.876 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5429999999999999e-06 2020-01-16T00:14:36.876 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.876 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.876 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:36.876 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074704+0000", 2020-01-16T00:14:36.876 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0839999999999992e-06 2020-01-16T00:14:36.876 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.876 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.877 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:36.877 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074712+0000", 2020-01-16T00:14:36.877 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.173e-06 2020-01-16T00:14:36.877 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.877 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.877 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:36.877 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074718+0000", 2020-01-16T00:14:36.878 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0083034700000000003 2020-01-16T00:14:36.878 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.878 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.878 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.878 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.083022+0000", 2020-01-16T00:14:36.878 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028456400000000001 2020-01-16T00:14:36.878 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.879 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.879 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:36.879 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.083306+0000", 2020-01-16T00:14:36.879 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0072312640000000003 2020-01-16T00:14:36.879 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.879 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.879 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:36.880 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090538+0000", 2020-01-16T00:14:36.880 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015839157999999999 2020-01-16T00:14:36.880 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.880 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:36.880 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.880 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.880 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.880 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1277 3.d 3:b1521a3e:::smithi12313934-25:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:14:36.881 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:14:36.881 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 113.78240359500001, 2020-01-16T00:14:36.881 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016646459999999998, 2020-01-16T00:14:36.881 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:36.881 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:36.881 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:36.881 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:36.881 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:36.882 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1277 2020-01-16T00:14:36.882 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.882 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:36.882 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.882 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:36.882 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:14:36.882 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:36.882 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.883 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.883 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:36.883 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:14:36.883 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:36.883 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.883 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.883 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:36.883 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074741+0000", 2020-01-16T00:14:36.884 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0776e-05 2020-01-16T00:14:36.884 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.884 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.884 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:36.884 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074752+0000", 2020-01-16T00:14:36.884 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7600000000000002e-06 2020-01-16T00:14:36.884 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.884 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.885 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:36.885 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074759+0000", 2020-01-16T00:14:36.885 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6840000000000001e-06 2020-01-16T00:14:36.885 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.885 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.885 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:36.885 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074765+0000", 2020-01-16T00:14:36.886 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015815216 2020-01-16T00:14:36.886 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.886 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.886 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.886 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090580+0000", 2020-01-16T00:14:36.886 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000182404 2020-01-16T00:14:36.886 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.886 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.887 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:36.887 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090763+0000", 2020-01-16T00:14:36.887 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00062579300000000003 2020-01-16T00:14:36.887 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.887 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.887 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:36.887 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091389+0000", 2020-01-16T00:14:36.887 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016646459999999998 2020-01-16T00:14:36.888 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.888 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:36.888 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.888 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.888 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.888 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1278 3.d 3:b1521a3e:::smithi12313934-25:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:14:36.888 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:14:36.888 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 113.782373882, 2020-01-16T00:14:36.889 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017994915, 2020-01-16T00:14:36.889 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:36.889 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:36.889 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:36.889 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:36.889 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:36.889 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1278 2020-01-16T00:14:36.890 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.890 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:36.890 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.890 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:36.890 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:14:36.890 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:36.890 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.890 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.891 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:36.891 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:14:36.891 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:36.891 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.891 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.891 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:36.891 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074771+0000", 2020-01-16T00:14:36.891 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4569999999999998e-06 2020-01-16T00:14:36.891 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.892 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.892 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:36.892 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074775+0000", 2020-01-16T00:14:36.892 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0330000000000003e-06 2020-01-16T00:14:36.892 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.892 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.892 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:36.892 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074782+0000", 2020-01-16T00:14:36.892 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9470000000000002e-06 2020-01-16T00:14:36.893 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.893 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.893 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:36.893 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074787+0000", 2020-01-16T00:14:36.893 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01666134 2020-01-16T00:14:36.893 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.893 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.893 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.894 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091449+0000", 2020-01-16T00:14:36.894 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000247731 2020-01-16T00:14:36.894 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.894 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.894 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:36.894 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091696+0000", 2020-01-16T00:14:36.894 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010703170000000001 2020-01-16T00:14:36.894 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.894 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.895 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:36.895 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.092767+0000", 2020-01-16T00:14:36.895 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017994915 2020-01-16T00:14:36.895 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.895 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:36.895 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.895 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.895 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.896 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2260 3.d 3:b76659c4:::smithi12313934-47:head [write 601750~535546] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:14:36.896 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:06.050577+0000", 2020-01-16T00:14:36.896 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 90.806568565999996, 2020-01-16T00:14:36.896 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012415381, 2020-01-16T00:14:36.896 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:36.896 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:36.896 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:36.896 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:36.896 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:36.897 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2260 2020-01-16T00:14:36.897 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.897 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:36.897 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.897 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:36.897 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.050577+0000", 2020-01-16T00:14:36.897 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:36.897 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.898 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.898 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:36.898 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.050577+0000", 2020-01-16T00:14:36.898 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:36.898 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.898 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.898 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:36.898 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.050574+0000", 2020-01-16T00:14:36.898 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001523487 2020-01-16T00:14:36.899 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.899 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.899 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:36.899 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052097+0000", 2020-01-16T00:14:36.899 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3697e-05 2020-01-16T00:14:36.899 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.899 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.899 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:36.900 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052121+0000", 2020-01-16T00:14:36.900 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0070000000000003e-06 2020-01-16T00:14:36.900 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.900 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.900 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:36.900 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052130+0000", 2020-01-16T00:14:36.900 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4559000000000003e-05 2020-01-16T00:14:36.900 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.901 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.901 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.901 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052194+0000", 2020-01-16T00:14:36.901 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000165878 2020-01-16T00:14:36.901 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.901 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.901 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:36.901 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052360+0000", 2020-01-16T00:14:36.902 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018825499999999999 2020-01-16T00:14:36.902 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.902 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.902 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:14:36.902 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052549+0000", 2020-01-16T00:14:36.902 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002761782 2020-01-16T00:14:36.902 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.902 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.903 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:36.903 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.055310+0000", 2020-01-16T00:14:36.903 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00060385500000000004 2020-01-16T00:14:36.903 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.903 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.903 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:36.903 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.055914+0000", 2020-01-16T00:14:36.904 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0069733809999999999 2020-01-16T00:14:36.904 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.904 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.904 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:36.904 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.062888+0000", 2020-01-16T00:14:36.904 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2884999999999999e-05 2020-01-16T00:14:36.904 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.904 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.905 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:36.905 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.062940+0000", 2020-01-16T00:14:36.905 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1996999999999997e-05 2020-01-16T00:14:36.905 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.905 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.905 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:36.905 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.062992+0000", 2020-01-16T00:14:36.905 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012415381 2020-01-16T00:14:36.906 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.906 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:36.906 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.906 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.906 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.906 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2295 3.d 3:b1521a3e:::smithi12313934-25:head [write 1763838~505140] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:14:36.906 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:06.660258+0000", 2020-01-16T00:14:36.907 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 90.196887339, 2020-01-16T00:14:36.907 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011681408000000001, 2020-01-16T00:14:36.907 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:36.907 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:36.907 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:36.907 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:36.907 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:36.907 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2295 2020-01-16T00:14:36.908 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.908 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:36.908 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.908 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:36.908 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660258+0000", 2020-01-16T00:14:36.908 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:36.908 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.908 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.908 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:36.909 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660258+0000", 2020-01-16T00:14:36.909 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:36.909 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.909 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.909 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:36.909 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660256+0000", 2020-01-16T00:14:36.909 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00066501699999999995 2020-01-16T00:14:36.910 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.910 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.910 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:36.910 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660921+0000", 2020-01-16T00:14:36.910 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7339000000000001e-05 2020-01-16T00:14:36.910 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.910 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.911 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:36.911 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660938+0000", 2020-01-16T00:14:36.911 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0219999999999993e-06 2020-01-16T00:14:36.911 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.911 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.911 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:36.911 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660947+0000", 2020-01-16T00:14:36.912 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00094237200000000002 2020-01-16T00:14:36.912 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.912 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.912 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.912 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.661890+0000", 2020-01-16T00:14:36.912 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000117649 2020-01-16T00:14:36.912 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.913 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.913 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:36.913 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662007+0000", 2020-01-16T00:14:36.913 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00060662599999999998 2020-01-16T00:14:36.913 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.913 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.913 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.913 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662614+0000", 2020-01-16T00:14:36.914 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028424899999999998 2020-01-16T00:14:36.914 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.914 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.914 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:36.914 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662898+0000", 2020-01-16T00:14:36.914 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00054904599999999997 2020-01-16T00:14:36.914 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.915 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.915 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:14:36.915 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.663447+0000", 2020-01-16T00:14:36.915 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0051479990000000003 2020-01-16T00:14:36.915 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.915 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.915 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:36.915 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.668595+0000", 2020-01-16T00:14:36.916 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00080881799999999999 2020-01-16T00:14:36.916 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.916 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.916 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:36.916 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.669404+0000", 2020-01-16T00:14:36.916 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0024495290000000002 2020-01-16T00:14:36.916 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.916 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.917 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:36.917 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671854+0000", 2020-01-16T00:14:36.917 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6635000000000003e-05 2020-01-16T00:14:36.917 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.917 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.917 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:36.917 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671900+0000", 2020-01-16T00:14:36.918 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9567999999999998e-05 2020-01-16T00:14:36.918 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.918 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.918 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:36.918 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671940+0000", 2020-01-16T00:14:36.918 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011681408000000001 2020-01-16T00:14:36.918 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.918 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:36.919 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.919 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.919 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.919 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.3.13:109 2.0 2:11fa79bc:::smithi12313934-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 331264~198656] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e42)", 2020-01-16T00:14:36.919 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:21.840075+0000", 2020-01-16T00:14:36.919 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 75.017071041999998, 2020-01-16T00:14:36.919 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011558677999999999, 2020-01-16T00:14:36.919 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:36.920 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:36.920 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:36.920 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.920 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:36.920 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840075+0000", 2020-01-16T00:14:36.920 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:36.920 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.920 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.921 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:36.921 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840075+0000", 2020-01-16T00:14:36.921 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:36.921 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.921 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.921 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:36.921 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840073+0000", 2020-01-16T00:14:36.921 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033576299999999999 2020-01-16T00:14:36.922 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.922 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.922 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:36.922 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840409+0000", 2020-01-16T00:14:36.922 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2082999999999998e-05 2020-01-16T00:14:36.922 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.922 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.922 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:36.923 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840431+0000", 2020-01-16T00:14:36.923 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4410000000000003e-06 2020-01-16T00:14:36.923 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.923 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.923 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:36.923 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840436+0000", 2020-01-16T00:14:36.923 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000101016 2020-01-16T00:14:36.923 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.924 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.924 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.924 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840537+0000", 2020-01-16T00:14:36.924 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000117639 2020-01-16T00:14:36.924 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.924 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.924 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:36.925 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840654+0000", 2020-01-16T00:14:36.925 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000196162 2020-01-16T00:14:36.925 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.925 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.925 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:14:36.925 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840850+0000", 2020-01-16T00:14:36.925 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003319092 2020-01-16T00:14:36.925 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.926 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.926 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:36.926 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.844169+0000", 2020-01-16T00:14:36.926 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011349229999999999 2020-01-16T00:14:36.926 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.926 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.926 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:36.927 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.845304+0000", 2020-01-16T00:14:36.927 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0062517670000000001 2020-01-16T00:14:36.927 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.927 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.927 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:36.927 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851556+0000", 2020-01-16T00:14:36.927 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4138000000000003e-05 2020-01-16T00:14:36.927 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.927 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.928 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:36.928 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851600+0000", 2020-01-16T00:14:36.928 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2993000000000001e-05 2020-01-16T00:14:36.928 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.928 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.928 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:36.928 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851633+0000", 2020-01-16T00:14:36.928 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011558677999999999 2020-01-16T00:14:36.929 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.929 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:36.929 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.929 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.929 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.929 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2987 3.d 3:b3dd5d45:::smithi12313934-16:head [write 467327~633480] snapc 0=[] ondisk+write+known_if_redirected e42)", 2020-01-16T00:14:36.929 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:22.257904+0000", 2020-01-16T00:14:36.929 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 74.599241456000001, 2020-01-16T00:14:36.930 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012187450000000001, 2020-01-16T00:14:36.930 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:36.930 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:36.930 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:36.930 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:36.930 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:36.930 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2987 2020-01-16T00:14:36.931 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.931 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:36.931 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.931 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:36.931 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.257904+0000", 2020-01-16T00:14:36.931 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:36.931 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.931 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.932 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:36.932 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.257904+0000", 2020-01-16T00:14:36.932 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999933 2020-01-16T00:14:36.932 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.932 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.932 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:36.932 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.257898+0000", 2020-01-16T00:14:36.932 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00097099800000000004 2020-01-16T00:14:36.933 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.933 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.933 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:36.933 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.258869+0000", 2020-01-16T00:14:36.933 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7970000000000003e-05 2020-01-16T00:14:36.933 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.933 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.934 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:36.934 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.258917+0000", 2020-01-16T00:14:36.934 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5982999999999999e-05 2020-01-16T00:14:36.934 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.934 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.934 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:36.934 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.258933+0000", 2020-01-16T00:14:36.934 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00026239999999999998 2020-01-16T00:14:36.935 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.935 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.935 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.935 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.259195+0000", 2020-01-16T00:14:36.935 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024907400000000001 2020-01-16T00:14:36.935 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.935 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.935 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:36.936 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.259444+0000", 2020-01-16T00:14:36.936 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00034824999999999998 2020-01-16T00:14:36.936 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.936 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.936 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:36.936 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.259792+0000", 2020-01-16T00:14:36.936 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003251538 2020-01-16T00:14:36.937 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.937 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.937 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:36.937 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.263044+0000", 2020-01-16T00:14:36.937 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0068502880000000004 2020-01-16T00:14:36.937 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.937 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.937 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:36.938 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.269894+0000", 2020-01-16T00:14:36.938 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0945000000000006e-05 2020-01-16T00:14:36.938 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.938 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.938 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:36.938 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.269965+0000", 2020-01-16T00:14:36.938 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012667399999999999 2020-01-16T00:14:36.939 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.939 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.939 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:36.939 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.270092+0000", 2020-01-16T00:14:36.939 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012187450000000001 2020-01-16T00:14:36.939 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.939 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:36.939 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.940 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.940 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.940 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3273 3.d 3:b76659c4:::smithi12313934-47:head [read 0~3647152,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:36.940 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.728775+0000", 2020-01-16T00:14:36.940 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 66.128370902, 2020-01-16T00:14:36.940 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012922950000000001, 2020-01-16T00:14:36.940 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:36.940 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:36.941 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:36.941 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:36.941 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:36.941 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3273 2020-01-16T00:14:36.941 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.941 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:36.941 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.942 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:36.942 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728775+0000", 2020-01-16T00:14:36.942 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:36.942 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.942 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.942 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:36.942 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728775+0000", 2020-01-16T00:14:36.942 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:36.943 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.943 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.943 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:36.943 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728771+0000", 2020-01-16T00:14:36.943 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.5009000000000001e-05 2020-01-16T00:14:36.943 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.943 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.944 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:36.944 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728796+0000", 2020-01-16T00:14:36.944 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2357999999999998e-05 2020-01-16T00:14:36.944 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.944 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.944 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:36.944 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728819+0000", 2020-01-16T00:14:36.944 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.4800000000000007e-06 2020-01-16T00:14:36.945 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.945 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.945 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:36.945 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728828+0000", 2020-01-16T00:14:36.945 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000124824 2020-01-16T00:14:36.945 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.945 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.946 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.946 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728953+0000", 2020-01-16T00:14:36.946 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00044904899999999998 2020-01-16T00:14:36.946 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.946 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.946 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:36.946 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729402+0000", 2020-01-16T00:14:36.947 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012295706999999999 2020-01-16T00:14:36.947 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.947 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.947 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:36.947 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.741698+0000", 2020-01-16T00:14:36.947 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012922950000000001 2020-01-16T00:14:36.947 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.948 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:36.948 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.948 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.948 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.948 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3275 3.d 3:b76659c4:::smithi12313934-47:head [read 0~3647152,checksum] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:36.948 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.728933+0000", 2020-01-16T00:14:36.948 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 66.128212629999993, 2020-01-16T00:14:36.949 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01204874, 2020-01-16T00:14:36.949 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:36.949 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:36.949 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:36.949 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:36.949 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:36.949 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3275 2020-01-16T00:14:36.950 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.950 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:36.950 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.950 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:36.950 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728933+0000", 2020-01-16T00:14:36.950 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:36.950 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.950 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.951 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:36.951 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728933+0000", 2020-01-16T00:14:36.951 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:36.951 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.951 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.951 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:36.951 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728932+0000", 2020-01-16T00:14:36.952 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3569999999999998e-06 2020-01-16T00:14:36.952 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.952 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.952 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:36.952 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728940+0000", 2020-01-16T00:14:36.952 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0457e-05 2020-01-16T00:14:36.952 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.952 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.953 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:36.953 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728951+0000", 2020-01-16T00:14:36.953 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6799999999999993e-06 2020-01-16T00:14:36.953 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.953 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.953 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:36.953 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728959+0000", 2020-01-16T00:14:36.953 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010076307 2020-01-16T00:14:36.954 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.954 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.954 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.954 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.739035+0000", 2020-01-16T00:14:36.954 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024455400000000002 2020-01-16T00:14:36.954 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.954 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.954 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:36.955 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.739279+0000", 2020-01-16T00:14:36.955 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0017023660000000001 2020-01-16T00:14:36.955 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.955 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.955 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:36.955 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.740982+0000", 2020-01-16T00:14:36.955 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01204874 2020-01-16T00:14:36.955 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.956 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:36.956 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.956 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.956 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.956 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3285 3.d 3:bd9fc72f:::smithi12313934-5:head [read 0~3764801,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:36.956 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.729068+0000", 2020-01-16T00:14:36.956 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 66.128077837999996, 2020-01-16T00:14:36.956 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018537010999999999, 2020-01-16T00:14:36.957 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:36.957 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:36.957 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:36.957 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:36.957 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:36.957 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3285 2020-01-16T00:14:36.957 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.958 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:36.958 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.958 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:36.958 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729068+0000", 2020-01-16T00:14:36.958 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:36.958 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.958 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.958 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:36.959 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729068+0000", 2020-01-16T00:14:36.959 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:36.959 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.959 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.959 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:36.959 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729067+0000", 2020-01-16T00:14:36.959 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3339999999999996e-06 2020-01-16T00:14:36.959 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.960 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.960 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:36.960 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729075+0000", 2020-01-16T00:14:36.960 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4187e-05 2020-01-16T00:14:36.960 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.960 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.960 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:36.960 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729089+0000", 2020-01-16T00:14:36.961 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5600000000000004e-06 2020-01-16T00:14:36.961 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.961 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.961 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:36.961 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729093+0000", 2020-01-16T00:14:36.961 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011944309 2020-01-16T00:14:36.961 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.961 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.962 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.962 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.741038+0000", 2020-01-16T00:14:36.962 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016625799999999999 2020-01-16T00:14:36.962 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.962 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.962 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:36.962 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.741204+0000", 2020-01-16T00:14:36.963 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0064010120000000002 2020-01-16T00:14:36.963 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.963 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.963 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:36.963 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.747605+0000", 2020-01-16T00:14:36.963 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018537010999999999 2020-01-16T00:14:36.963 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.963 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:36.963 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.964 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.964 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.964 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3286 3.d 3:bd9fc72f:::smithi12313934-5:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:36.964 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.729099+0000", 2020-01-16T00:14:36.964 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 66.128047167999995, 2020-01-16T00:14:36.964 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018983897999999999, 2020-01-16T00:14:36.964 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:36.964 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:36.965 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:36.965 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:36.965 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:36.965 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3286 2020-01-16T00:14:36.965 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.965 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:36.965 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.966 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:36.966 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729099+0000", 2020-01-16T00:14:36.966 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:36.966 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.966 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.966 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:36.966 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729099+0000", 2020-01-16T00:14:36.966 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:36.967 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.967 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.967 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:36.967 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729098+0000", 2020-01-16T00:14:36.967 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8500000000000004e-06 2020-01-16T00:14:36.967 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.967 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.968 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:36.968 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729102+0000", 2020-01-16T00:14:36.968 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7329999999999997e-06 2020-01-16T00:14:36.968 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.968 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.968 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:36.968 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729108+0000", 2020-01-16T00:14:36.968 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.764e-06 2020-01-16T00:14:36.969 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.969 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.969 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:36.969 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729118+0000", 2020-01-16T00:14:36.969 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018524286000000001 2020-01-16T00:14:36.969 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.969 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.969 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.970 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.747642+0000", 2020-01-16T00:14:36.970 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9487999999999993e-05 2020-01-16T00:14:36.970 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.970 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.970 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:36.970 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.747742+0000", 2020-01-16T00:14:36.970 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000340429 2020-01-16T00:14:36.970 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.971 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.971 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:36.971 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.748082+0000", 2020-01-16T00:14:36.971 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018983897999999999 2020-01-16T00:14:36.971 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.971 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:36.971 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.971 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.972 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.972 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3287 3.d 3:bd9fc72f:::smithi12313934-5:head [read 0~3764801,checksum] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:36.972 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.729124+0000", 2020-01-16T00:14:36.972 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 66.128022028999993, 2020-01-16T00:14:36.972 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020552323000000001, 2020-01-16T00:14:36.972 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:36.972 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:36.972 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:36.973 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:36.973 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:36.973 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3287 2020-01-16T00:14:36.973 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.973 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:36.973 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.973 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:36.973 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729124+0000", 2020-01-16T00:14:36.974 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:36.974 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.974 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.974 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:36.974 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729124+0000", 2020-01-16T00:14:36.974 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:36.974 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.974 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.975 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:36.975 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729123+0000", 2020-01-16T00:14:36.975 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8239999999999996e-06 2020-01-16T00:14:36.975 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.975 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.975 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:36.975 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729129+0000", 2020-01-16T00:14:36.975 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6909999999999997e-06 2020-01-16T00:14:36.976 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.976 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.976 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:36.976 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729136+0000", 2020-01-16T00:14:36.976 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3329999999999997e-06 2020-01-16T00:14:36.976 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.976 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.976 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:36.977 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729145+0000", 2020-01-16T00:14:36.977 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018965065999999999 2020-01-16T00:14:36.977 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.977 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.977 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.977 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.748110+0000", 2020-01-16T00:14:36.977 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015781700000000001 2020-01-16T00:14:36.978 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.978 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.978 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:36.978 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.748268+0000", 2020-01-16T00:14:36.978 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014083800000000001 2020-01-16T00:14:36.978 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.978 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.978 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:36.979 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.749676+0000", 2020-01-16T00:14:36.979 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020552323000000001 2020-01-16T00:14:36.979 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.979 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:36.979 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.979 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.979 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.979 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3409 3.d 3:b3dd5d45:::smithi12313934-16:head [write 591712~786646,stat] snapc 0=[] ondisk+write+known_if_redirected e47)", 2020-01-16T00:14:36.979 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:34.100681+0000", 2020-01-16T00:14:36.980 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 62.756464889, 2020-01-16T00:14:36.980 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012248446, 2020-01-16T00:14:36.980 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:36.980 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:36.980 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:36.980 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:36.980 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:36.981 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3409 2020-01-16T00:14:36.981 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.981 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:36.981 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.981 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:36.981 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.100681+0000", 2020-01-16T00:14:36.981 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:36.981 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.982 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.982 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:36.982 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.100681+0000", 2020-01-16T00:14:36.982 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:14:36.982 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.982 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.982 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:36.983 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.100677+0000", 2020-01-16T00:14:36.983 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000998828 2020-01-16T00:14:36.983 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.983 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.983 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:36.983 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101676+0000", 2020-01-16T00:14:36.983 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5846999999999997e-05 2020-01-16T00:14:36.983 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.984 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.984 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:36.984 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101712+0000", 2020-01-16T00:14:36.984 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.364e-05 2020-01-16T00:14:36.984 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.984 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.984 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:36.984 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101725+0000", 2020-01-16T00:14:36.985 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015902700000000001 2020-01-16T00:14:36.985 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.985 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.985 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.985 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101884+0000", 2020-01-16T00:14:36.985 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0486e-05 2020-01-16T00:14:36.985 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.986 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.986 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op must wait for map", 2020-01-16T00:14:36.986 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101905+0000", 2020-01-16T00:14:36.986 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0038869009999999999 2020-01-16T00:14:36.986 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.986 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.986 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.986 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.105792+0000", 2020-01-16T00:14:36.987 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030748800000000002 2020-01-16T00:14:36.987 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.987 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.987 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:36.987 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.106099+0000", 2020-01-16T00:14:36.987 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042069299999999998 2020-01-16T00:14:36.987 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.988 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.988 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:36.988 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.106520+0000", 2020-01-16T00:14:36.988 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0052864560000000001 2020-01-16T00:14:36.988 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.988 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.988 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:36.988 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.111806+0000", 2020-01-16T00:14:36.989 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00080835800000000001 2020-01-16T00:14:36.989 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.989 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.989 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:36.989 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.112615+0000", 2020-01-16T00:14:36.989 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1490999999999997e-05 2020-01-16T00:14:36.989 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.990 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.990 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:36.990 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.112686+0000", 2020-01-16T00:14:36.990 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000243174 2020-01-16T00:14:36.990 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.990 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.990 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:36.990 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.112929+0000", 2020-01-16T00:14:36.991 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012248446 2020-01-16T00:14:36.991 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.991 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:36.991 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:36.991 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.991 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.991 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.0.13:31 2.0 2:1afbc8ac:::smithi12313934-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 4096~737792] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e50)", 2020-01-16T00:14:36.992 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:44.061109+0000", 2020-01-16T00:14:36.992 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 52.796036665000003, 2020-01-16T00:14:36.992 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012186723, 2020-01-16T00:14:36.992 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:36.992 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:36.992 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:36.992 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.992 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:36.993 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.061109+0000", 2020-01-16T00:14:36.993 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:36.993 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.993 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.993 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:36.993 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.061109+0000", 2020-01-16T00:14:36.993 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:36.994 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.994 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.994 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:36.994 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.061106+0000", 2020-01-16T00:14:36.994 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012349870000000001 2020-01-16T00:14:36.994 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.994 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.995 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:36.995 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062341+0000", 2020-01-16T00:14:36.995 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0235999999999996e-05 2020-01-16T00:14:36.995 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.995 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.995 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:36.995 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062411+0000", 2020-01-16T00:14:36.996 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7078000000000001e-05 2020-01-16T00:14:36.996 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.996 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.996 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:36.996 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062428+0000", 2020-01-16T00:14:36.996 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2158999999999996e-05 2020-01-16T00:14:36.996 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.996 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.997 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:36.997 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062520+0000", 2020-01-16T00:14:36.997 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011203499999999999 2020-01-16T00:14:36.997 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.997 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.997 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:36.997 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062632+0000", 2020-01-16T00:14:36.998 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017238400000000001 2020-01-16T00:14:36.998 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.998 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.998 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:14:36.998 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062805+0000", 2020-01-16T00:14:36.998 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010264989 2020-01-16T00:14:36.998 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.999 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.999 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:36.999 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073070+0000", 2020-01-16T00:14:36.999 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0533000000000001e-05 2020-01-16T00:14:36.999 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:36.999 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:36.999 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:36.999 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073090+0000", 2020-01-16T00:14:37.000 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2309000000000002e-05 2020-01-16T00:14:37.000 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.000 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.000 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:37.000 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073163+0000", 2020-01-16T00:14:37.000 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3035999999999997e-05 2020-01-16T00:14:37.000 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.000 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.001 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:37.001 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073236+0000", 2020-01-16T00:14:37.001 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0056999999999998e-05 2020-01-16T00:14:37.001 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.001 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.001 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:37.001 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073296+0000", 2020-01-16T00:14:37.002 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012186723 2020-01-16T00:14:37.002 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:37.002 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:37.002 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:37.002 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.002 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.002 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.0.13:33 2.0 2:1afbc8ac:::smithi12313934-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 1541120~378368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e50)", 2020-01-16T00:14:37.002 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:44.063236+0000", 2020-01-16T00:14:37.003 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 52.793910001, 2020-01-16T00:14:37.003 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015290962999999999, 2020-01-16T00:14:37.003 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:37.003 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:37.003 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:37.003 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.003 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:37.003 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063236+0000", 2020-01-16T00:14:37.004 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:37.004 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.004 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.004 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:37.004 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063236+0000", 2020-01-16T00:14:37.004 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:37.004 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.004 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.005 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:37.005 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063234+0000", 2020-01-16T00:14:37.005 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000500702 2020-01-16T00:14:37.005 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.005 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.005 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:37.005 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063735+0000", 2020-01-16T00:14:37.005 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4734e-05 2020-01-16T00:14:37.006 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.006 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.006 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:37.006 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063750+0000", 2020-01-16T00:14:37.006 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.382e-05 2020-01-16T00:14:37.006 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.006 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.007 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:37.007 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063764+0000", 2020-01-16T00:14:37.007 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0059030799999999998 2020-01-16T00:14:37.007 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.007 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.007 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:37.007 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.069667+0000", 2020-01-16T00:14:37.007 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3961999999999994e-05 2020-01-16T00:14:37.008 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.008 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.008 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:37.008 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.069741+0000", 2020-01-16T00:14:37.008 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000114931 2020-01-16T00:14:37.008 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.008 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.008 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:14:37.009 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.069856+0000", 2020-01-16T00:14:37.009 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0039948270000000003 2020-01-16T00:14:37.009 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.009 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.009 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:37.009 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073850+0000", 2020-01-16T00:14:37.009 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0016459140000000001 2020-01-16T00:14:37.010 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.010 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.010 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:37.010 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.075496+0000", 2020-01-16T00:14:37.010 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002957997 2020-01-16T00:14:37.010 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.010 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.011 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:37.011 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.078454+0000", 2020-01-16T00:14:37.011 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9190999999999998e-05 2020-01-16T00:14:37.011 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.011 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.011 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:37.011 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.078493+0000", 2020-01-16T00:14:37.011 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3176999999999998e-05 2020-01-16T00:14:37.012 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.012 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.012 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:37.012 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.078527+0000", 2020-01-16T00:14:37.012 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015290962999999999 2020-01-16T00:14:37.012 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:37.012 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:37.012 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:37.013 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.013 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.013 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4641 3.d 3:b1521a3e:::smithi12313934-25:head [read 0~3596797,checksum] snapc 0=[] ondisk+read+known_if_redirected e57)", 2020-01-16T00:14:37.013 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:04.086675+0000", 2020-01-16T00:14:37.013 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 32.770470945, 2020-01-16T00:14:37.013 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011822397, 2020-01-16T00:14:37.013 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:37.013 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:37.014 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:37.014 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:37.014 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:37.014 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4641 2020-01-16T00:14:37.014 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.014 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:37.014 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.014 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:37.015 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.086675+0000", 2020-01-16T00:14:37.015 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:37.015 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.015 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.015 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:37.015 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.086675+0000", 2020-01-16T00:14:37.015 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:37.015 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.016 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.016 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:37.016 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.086674+0000", 2020-01-16T00:14:37.016 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0809999999999998e-06 2020-01-16T00:14:37.016 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.016 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.016 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:37.016 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.086678+0000", 2020-01-16T00:14:37.017 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4889999999999997e-06 2020-01-16T00:14:37.017 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.017 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.017 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:37.017 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.086682+0000", 2020-01-16T00:14:37.017 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9620000000000002e-06 2020-01-16T00:14:37.017 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.018 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.018 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:37.018 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.086685+0000", 2020-01-16T00:14:37.018 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010215452 2020-01-16T00:14:37.018 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.018 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.018 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:37.018 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.096900+0000", 2020-01-16T00:14:37.019 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00021274299999999999 2020-01-16T00:14:37.019 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.019 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.019 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:37.019 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.097113+0000", 2020-01-16T00:14:37.019 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0013839480000000001 2020-01-16T00:14:37.019 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.019 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.020 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:37.020 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.098497+0000", 2020-01-16T00:14:37.020 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011822397 2020-01-16T00:14:37.020 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:37.020 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:37.020 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:37.020 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.020 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.021 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4737 3.d 3:b76659c4:::smithi12313934-47:head [read 0~3243850,checksum] snapc 0=[] ondisk+read+known_if_redirected e58)", 2020-01-16T00:14:37.021 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:08.134998+0000", 2020-01-16T00:14:37.021 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 28.722148070999999, 2020-01-16T00:14:37.021 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011528518, 2020-01-16T00:14:37.021 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:37.021 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:37.021 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:37.021 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:37.021 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:37.022 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4737 2020-01-16T00:14:37.022 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.022 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:37.022 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.022 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:37.022 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:08.134998+0000", 2020-01-16T00:14:37.022 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:37.022 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.023 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.023 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:37.023 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:08.134998+0000", 2020-01-16T00:14:37.023 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:37.023 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.023 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.023 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:37.023 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:08.134997+0000", 2020-01-16T00:14:37.023 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0459999999999999e-06 2020-01-16T00:14:37.024 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.024 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.024 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:37.024 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:08.135001+0000", 2020-01-16T00:14:37.024 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2469999999999999e-06 2020-01-16T00:14:37.024 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.024 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.024 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:37.025 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:08.135006+0000", 2020-01-16T00:14:37.025 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.065e-06 2020-01-16T00:14:37.025 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.025 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.025 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:37.025 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:08.135010+0000", 2020-01-16T00:14:37.025 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0095733439999999993 2020-01-16T00:14:37.026 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.026 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.026 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:37.026 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:08.144584+0000", 2020-01-16T00:14:37.026 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000272623 2020-01-16T00:14:37.026 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.026 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.027 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:37.027 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:08.144856+0000", 2020-01-16T00:14:37.027 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001669769 2020-01-16T00:14:37.027 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.027 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.027 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:37.027 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:08.146526+0000", 2020-01-16T00:14:37.027 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011528518 2020-01-16T00:14:37.028 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:37.028 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:37.028 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:37.028 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.028 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.028 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:5704 3.e 3:7cfd7007:::smithi12313934-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e67)", 2020-01-16T00:14:37.028 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:30.998090+0000", 2020-01-16T00:14:37.028 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 5.8590557839999997, 2020-01-16T00:14:37.029 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011469234999999999, 2020-01-16T00:14:37.029 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:37.029 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:37.029 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:37.029 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:37.029 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:37.029 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 5704 2020-01-16T00:14:37.030 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.030 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:37.030 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.030 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:37.030 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:30.998090+0000", 2020-01-16T00:14:37.030 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:37.030 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.030 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.030 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:37.031 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:30.998090+0000", 2020-01-16T00:14:37.031 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:37.031 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.031 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.031 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:37.031 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:30.998089+0000", 2020-01-16T00:14:37.031 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1510000000000001e-06 2020-01-16T00:14:37.032 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.032 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.032 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:37.032 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:30.998095+0000", 2020-01-16T00:14:37.032 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.9919999999999996e-06 2020-01-16T00:14:37.032 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.032 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.032 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:37.033 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:30.998104+0000", 2020-01-16T00:14:37.033 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3429999999999998e-06 2020-01-16T00:14:37.033 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.033 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.033 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:37.033 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:30.998111+0000", 2020-01-16T00:14:37.033 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0075958309999999999 2020-01-16T00:14:37.033 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.034 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.034 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:37.034 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:31.005707+0000", 2020-01-16T00:14:37.034 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015471300000000001 2020-01-16T00:14:37.034 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.034 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.034 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:37.035 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:31.005862+0000", 2020-01-16T00:14:37.035 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0034355800000000001 2020-01-16T00:14:37.035 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.035 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.035 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:37.035 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:31.009297+0000", 2020-01-16T00:14:37.035 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011861100000000001 2020-01-16T00:14:37.035 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.036 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.036 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:37.036 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:31.009416+0000", 2020-01-16T00:14:37.036 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014325699999999999 2020-01-16T00:14:37.036 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.036 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.036 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:37.036 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:31.009559+0000", 2020-01-16T00:14:37.037 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011469234999999999 2020-01-16T00:14:37.037 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:37.037 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:37.037 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:37.037 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.037 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.037 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:5780 3.f 3:f94a5a88:::smithi12313934-22:head [read 0~3949777,checksum] snapc 0=[] ondisk+read+known_if_redirected e68)", 2020-01-16T00:14:37.037 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:32.221847+0000", 2020-01-16T00:14:37.038 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 4.6352983300000004, 2020-01-16T00:14:37.038 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012182213000000001, 2020-01-16T00:14:37.038 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:37.038 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:37.038 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:37.038 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:37.038 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:37.038 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 5780 2020-01-16T00:14:37.039 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.039 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:37.039 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.039 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:37.039 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:32.221847+0000", 2020-01-16T00:14:37.039 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:37.039 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.039 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.040 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:37.040 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:32.221847+0000", 2020-01-16T00:14:37.040 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:37.040 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.040 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.040 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:37.040 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:32.221846+0000", 2020-01-16T00:14:37.040 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5370000000000003e-06 2020-01-16T00:14:37.041 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.041 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.041 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:37.041 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:32.221853+0000", 2020-01-16T00:14:37.041 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3326e-05 2020-01-16T00:14:37.041 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.041 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.041 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:37.042 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:32.221876+0000", 2020-01-16T00:14:37.042 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1012000000000001e-05 2020-01-16T00:14:37.042 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.042 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.042 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:37.042 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:32.221887+0000", 2020-01-16T00:14:37.042 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0078782320000000006 2020-01-16T00:14:37.042 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.043 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.043 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:37.043 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:32.229765+0000", 2020-01-16T00:14:37.043 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00049060300000000005 2020-01-16T00:14:37.043 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.043 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.043 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:37.043 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:32.230256+0000", 2020-01-16T00:14:37.044 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0037737030000000002 2020-01-16T00:14:37.044 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:37.044 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:37.044 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:37.044 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:32.234030+0000", 2020-01-16T00:14:37.044 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012182213000000001 2020-01-16T00:14:37.044 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:37.044 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:37.044 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:37.045 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:37.045 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:37.045 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:37.045 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:36.858+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:37.045 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2020-01-16T00:14:37.050 INFO:tasks.rados.rados.0.smithi123.stdout:2026: expect (ObjNum 748 snap 0 seq_num 748) 2020-01-16T00:14:37.050 INFO:tasks.rados.rados.0.smithi123.stdout:2027: finishing write tid 1 to smithi12313934-18 2020-01-16T00:14:37.050 INFO:tasks.rados.rados.0.smithi123.stdout:2027: finishing write tid 2 to smithi12313934-18 2020-01-16T00:14:37.051 INFO:tasks.rados.rados.0.smithi123.stdout:2027: finishing write tid 4 to smithi12313934-18 2020-01-16T00:14:37.051 INFO:tasks.rados.rados.0.smithi123.stdout:2027: finishing write tid 5 to smithi12313934-18 2020-01-16T00:14:37.051 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 18 v 391 (ObjNum 783 snap 0 seq_num 783) dirty exists 2020-01-16T00:14:37.051 INFO:tasks.rados.rados.0.smithi123.stdout:2027: left oid 18 (ObjNum 783 snap 0 seq_num 783) 2020-01-16T00:14:37.051 INFO:tasks.rados.rados.0.smithi123.stdout:2028: finishing write tid 1 to smithi12313934-45 2020-01-16T00:14:37.051 INFO:tasks.rados.rados.0.smithi123.stdout:2028: finishing write tid 2 to smithi12313934-45 2020-01-16T00:14:37.051 INFO:tasks.rados.rados.0.smithi123.stdout:2028: finishing write tid 4 to smithi12313934-45 2020-01-16T00:14:37.051 INFO:tasks.rados.rados.0.smithi123.stdout:2028: finishing write tid 5 to smithi12313934-45 2020-01-16T00:14:37.052 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 45 v 211 (ObjNum 784 snap 0 seq_num 784) dirty exists 2020-01-16T00:14:37.052 INFO:tasks.rados.rados.0.smithi123.stdout:2028: left oid 45 (ObjNum 784 snap 0 seq_num 784) 2020-01-16T00:14:37.052 INFO:tasks.rados.rados.0.smithi123.stdout:2029: finishing write tid 1 to smithi12313934-42 2020-01-16T00:14:37.052 INFO:tasks.rados.rados.0.smithi123.stdout:2022: done (15 left) 2020-01-16T00:14:37.052 INFO:tasks.rados.rados.0.smithi123.stdout:2023: done (14 left) 2020-01-16T00:14:37.052 INFO:tasks.rados.rados.0.smithi123.stdout:2024: done (13 left) 2020-01-16T00:14:37.052 INFO:tasks.rados.rados.0.smithi123.stdout:2025: done (12 left) 2020-01-16T00:14:37.052 INFO:tasks.rados.rados.0.smithi123.stdout:2026: done (11 left) 2020-01-16T00:14:37.052 INFO:tasks.rados.rados.0.smithi123.stdout:2027: done (10 left) 2020-01-16T00:14:37.053 INFO:tasks.rados.rados.0.smithi123.stdout:2028: done (9 left) 2020-01-16T00:14:37.053 INFO:tasks.rados.rados.0.smithi123.stdout:2038: done (8 left) 2020-01-16T00:14:37.053 INFO:tasks.rados.rados.0.smithi123.stdout:2039: read oid 23 snap -1 2020-01-16T00:14:37.053 INFO:tasks.rados.rados.0.smithi123.stdout:2039: expect deleted 2020-01-16T00:14:37.053 INFO:tasks.rados.rados.0.smithi123.stdout:2040: write oid 15 current snap is 0 2020-01-16T00:14:37.053 INFO:tasks.rados.rados.0.smithi123.stdout:2040: seq_num 789 ranges {460723=555820,1670528=549595,2634153=626570} 2020-01-16T00:14:37.053 INFO:tasks.rados.rados.0.smithi123.stdout:2040: writing smithi12313934-15 from 460723 to 1016543 tid 1 2020-01-16T00:14:37.053 INFO:tasks.rados.rados.0.smithi123.stdout:2040: writing smithi12313934-15 from 1670528 to 2220123 tid 2 2020-01-16T00:14:37.054 INFO:tasks.rados.rados.0.smithi123.stdout:2040: writing smithi12313934-15 from 2634153 to 3260723 tid 3 2020-01-16T00:14:37.054 INFO:tasks.rados.rados.0.smithi123.stdout:2041: write (excl) oid 13 current snap is 0 2020-01-16T00:14:37.054 INFO:tasks.rados.rados.0.smithi123.stdout:2041: seq_num 790 ranges {477530=631069,1612657=409606,2545223=517596} 2020-01-16T00:14:37.054 INFO:tasks.rados.rados.0.smithi123.stdout:2041: writing smithi12313934-13 from 477530 to 1108599 tid 1 2020-01-16T00:14:37.054 INFO:tasks.rados.rados.0.smithi123.stdout:2041: writing smithi12313934-13 from 1612657 to 2022263 tid 2 2020-01-16T00:14:37.054 INFO:tasks.rados.rados.0.smithi123.stdout:2041: writing smithi12313934-13 from 2545223 to 3062819 tid 3 2020-01-16T00:14:37.054 INFO:tasks.rados.rados.0.smithi123.stdout:2042: read oid 49 snap -1 2020-01-16T00:14:37.054 INFO:tasks.rados.rados.0.smithi123.stdout:2042: expect (ObjNum 780 snap 0 seq_num 780) 2020-01-16T00:14:37.055 INFO:tasks.rados.rados.0.smithi123.stdout:2043: write (excl) oid 17 current snap is 0 2020-01-16T00:14:37.055 INFO:tasks.rados.rados.0.smithi123.stdout:2043: seq_num 791 ranges {494337=422671,1387492=553264,2688999=445221} 2020-01-16T00:14:37.055 INFO:tasks.rados.rados.0.smithi123.stdout:2043: writing smithi12313934-17 from 494337 to 917008 tid 1 2020-01-16T00:14:37.055 INFO:tasks.rados.rados.0.smithi123.stdout:2043: writing smithi12313934-17 from 1387492 to 1940756 tid 2 2020-01-16T00:14:37.055 INFO:tasks.rados.rados.0.smithi123.stdout:2043: writing smithi12313934-17 from 2688999 to 3134220 tid 3 2020-01-16T00:14:37.055 INFO:tasks.rados.rados.0.smithi123.stdout:2044: read oid 4 snap -1 2020-01-16T00:14:37.055 INFO:tasks.rados.rados.0.smithi123.stdout:2044: expect deleted 2020-01-16T00:14:37.055 INFO:tasks.rados.rados.0.smithi123.stdout:2045: write (excl) oid 48 current snap is 0 2020-01-16T00:14:37.055 INFO:tasks.rados.rados.0.smithi123.stdout:2045: seq_num 792 ranges {511144=497920,1445974=413275,2316422=656493} 2020-01-16T00:14:37.056 INFO:tasks.rados.rados.0.smithi123.stdout:2045: writing smithi12313934-48 from 511144 to 1009064 tid 1 2020-01-16T00:14:37.056 INFO:tasks.rados.rados.0.smithi123.stdout:2045: writing smithi12313934-48 from 1445974 to 1859249 tid 2 2020-01-16T00:14:37.056 INFO:tasks.rados.rados.0.smithi123.stdout:2045: writing smithi12313934-48 from 2316422 to 2972915 tid 3 2020-01-16T00:14:37.056 INFO:tasks.rados.rados.0.smithi123.stdout:2046: write (excl) oid 27 current snap is 0 2020-01-16T00:14:37.056 INFO:tasks.rados.rados.0.smithi123.stdout:2046: seq_num 793 ranges {527951=573169,1504456=556933,2743845=467765} 2020-01-16T00:14:37.056 INFO:tasks.rados.rados.0.smithi123.stdout:2046: writing smithi12313934-27 from 527951 to 1101120 tid 1 2020-01-16T00:14:37.056 INFO:tasks.rados.rados.0.smithi123.stdout:2046: writing smithi12313934-27 from 1504456 to 2061389 tid 2 2020-01-16T00:14:37.056 INFO:tasks.rados.rados.0.smithi123.stdout:2046: writing smithi12313934-27 from 2743845 to 3211610 tid 3 2020-01-16T00:14:37.056 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:37.057 INFO:tasks.rados.rados.0.smithi123.stdout:2031: expect (ObjNum 768 snap 0 seq_num 768) 2020-01-16T00:14:37.057 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:37.046+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:37.057 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-01-16T00:14:37.146 INFO:teuthology.orchestra.run.smithi123.stderr:noscrub is unset 2020-01-16T00:14:37.148 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:37.146+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:37.166 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:37.166 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:14:37.166 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:14:37.166 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:37.175 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-01-16T00:14:37.198 INFO:tasks.rados.rados.0.smithi123.stdout:2031: done (15 left) 2020-01-16T00:14:37.198 INFO:tasks.rados.rados.0.smithi123.stdout:2047: read oid 47 snap -1 2020-01-16T00:14:37.198 INFO:tasks.rados.rados.0.smithi123.stdout:2047: expect (ObjNum 776 snap 0 seq_num 776) 2020-01-16T00:14:37.198 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:37.198 INFO:tasks.rados.rados.0.smithi123.stdout:2030: expect (ObjNum 773 snap 0 seq_num 773) 2020-01-16T00:14:37.220 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:14:37.238 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:14:37.250 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:37.246+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:37.262 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:14:37.279 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:14:37.281 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:37.282 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:14:37.282 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:14:37.282 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:14:37.282 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:37.292 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-01-16T00:14:37.299 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:14:37.320 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:14:37.323 INFO:tasks.rados.rados.0.smithi123.stdout:2029: finishing write tid 2 to smithi12313934-42 2020-01-16T00:14:37.324 INFO:tasks.rados.rados.0.smithi123.stdout:2029: finishing write tid 3 to smithi12313934-42 2020-01-16T00:14:37.324 INFO:tasks.rados.rados.0.smithi123.stdout:2029: finishing write tid 5 to smithi12313934-42 2020-01-16T00:14:37.324 INFO:tasks.rados.rados.0.smithi123.stdout:2029: finishing write tid 6 to smithi12313934-42 2020-01-16T00:14:37.324 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 42 v 251 (ObjNum 785 snap 0 seq_num 785) dirty exists 2020-01-16T00:14:37.324 INFO:tasks.rados.rados.0.smithi123.stdout:2029: left oid 42 (ObjNum 785 snap 0 seq_num 785) 2020-01-16T00:14:37.324 INFO:tasks.rados.rados.0.smithi123.stdout:2029: done (15 left) 2020-01-16T00:14:37.325 INFO:tasks.rados.rados.0.smithi123.stdout:2030: done (14 left) 2020-01-16T00:14:37.325 INFO:tasks.rados.rados.0.smithi123.stdout:2048: write (excl) oid 42 current snap is 0 2020-01-16T00:14:37.325 INFO:tasks.rados.rados.0.smithi123.stdout:2048: seq_num 794 ranges {544758=648418,1846585=416944,2771268=573490} 2020-01-16T00:14:37.330 INFO:tasks.rados.rados.0.smithi123.stdout:2048: writing smithi12313934-42 from 544758 to 1193176 tid 1 2020-01-16T00:14:37.336 INFO:tasks.rados.rados.0.smithi123.stdout:2048: writing smithi12313934-42 from 1846585 to 2263529 tid 2 2020-01-16T00:14:37.341 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:14:37.342 INFO:tasks.rados.rados.0.smithi123.stdout:2048: writing smithi12313934-42 from 2771268 to 3344758 tid 3 2020-01-16T00:14:37.342 INFO:tasks.rados.rados.0.smithi123.stdout:2049: read oid 3 snap -1 2020-01-16T00:14:37.342 INFO:tasks.rados.rados.0.smithi123.stdout:2049: expect (ObjNum 771 snap 0 seq_num 771) 2020-01-16T00:14:37.343 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 16 2020-01-16T00:14:37.343 INFO:tasks.rados.rados.0.smithi123.stdout:2033: expect (ObjNum 729 snap 0 seq_num 729) 2020-01-16T00:14:37.343 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'false' 2020-01-16T00:14:37.344 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'false' 2020-01-16T00:14:37.344 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'false' 2020-01-16T00:14:37.344 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'false' 2020-01-16T00:14:37.344 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'false' 2020-01-16T00:14:37.344 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENXIO: problem getting command descriptions from osd.5 2020-01-16T00:14:37.344 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: problem getting command descriptions from osd.5 2020-01-16T00:14:37.344 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'false' 2020-01-16T00:14:37.345 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'false' 2020-01-16T00:14:37.351 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:37.350+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:37.357 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-01-16T00:14:37.357 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-01-16T00:14:37.402 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:37.411 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2020-01-16T00:14:37.454 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:37.454+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:37.510 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:37.519 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2020-01-16T00:14:37.536 INFO:tasks.rados.rados.0.smithi123.stdout:2035: finishing write tid 1 to smithi12313934-30 2020-01-16T00:14:37.536 INFO:tasks.rados.rados.0.smithi123.stdout:2035: finishing write tid 2 to smithi12313934-30 2020-01-16T00:14:37.536 INFO:tasks.rados.rados.0.smithi123.stdout:2035: finishing write tid 3 to smithi12313934-30 2020-01-16T00:14:37.536 INFO:tasks.rados.rados.0.smithi123.stdout:2035: finishing write tid 5 to smithi12313934-30 2020-01-16T00:14:37.536 INFO:tasks.rados.rados.0.smithi123.stdout:2035: finishing write tid 6 to smithi12313934-30 2020-01-16T00:14:37.536 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 30 v 212 (ObjNum 787 snap 0 seq_num 787) dirty exists 2020-01-16T00:14:37.537 INFO:tasks.rados.rados.0.smithi123.stdout:2035: left oid 30 (ObjNum 787 snap 0 seq_num 787) 2020-01-16T00:14:37.537 INFO:tasks.rados.rados.0.smithi123.stdout:2036: expect (ObjNum 779 snap 0 seq_num 779) 2020-01-16T00:14:37.582 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:37.582+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:37.640 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:37.649 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2020-01-16T00:14:37.690 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:37.690+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:37.754 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:37.752+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:37.778 INFO:tasks.rados.rados.0.smithi123.stdout:2037: expect (ObjNum 763 snap 0 seq_num 763) 2020-01-16T00:14:37.793 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:37.802 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2020-01-16T00:14:37.854 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:37.852+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:37.904 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:37.914 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2020-01-16T00:14:37.955 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:37.952+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:37.966 INFO:tasks.rados.rados.0.smithi123.stdout:2040: finishing write tid 1 to smithi12313934-15 2020-01-16T00:14:37.966 INFO:tasks.rados.rados.0.smithi123.stdout:2041: finishing write tid 1 to smithi12313934-13 2020-01-16T00:14:37.966 INFO:tasks.rados.rados.0.smithi123.stdout:2040: finishing write tid 2 to smithi12313934-15 2020-01-16T00:14:37.966 INFO:tasks.rados.rados.0.smithi123.stdout:2041: finishing write tid 2 to smithi12313934-13 2020-01-16T00:14:37.967 INFO:tasks.rados.rados.0.smithi123.stdout:2040: finishing write tid 3 to smithi12313934-15 2020-01-16T00:14:37.967 INFO:tasks.rados.rados.0.smithi123.stdout:2040: finishing write tid 5 to smithi12313934-15 2020-01-16T00:14:37.967 INFO:tasks.rados.rados.0.smithi123.stdout:2040: finishing write tid 6 to smithi12313934-15 2020-01-16T00:14:37.967 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 15 v 262 (ObjNum 789 snap 0 seq_num 789) dirty exists 2020-01-16T00:14:37.967 INFO:tasks.rados.rados.0.smithi123.stdout:2040: left oid 15 (ObjNum 789 snap 0 seq_num 789) 2020-01-16T00:14:37.967 INFO:tasks.rados.rados.0.smithi123.stdout:2041: finishing write tid 3 to smithi12313934-13 2020-01-16T00:14:37.967 INFO:tasks.rados.rados.0.smithi123.stdout:2041: finishing write tid 5 to smithi12313934-13 2020-01-16T00:14:37.967 INFO:tasks.rados.rados.0.smithi123.stdout:2041: finishing write tid 6 to smithi12313934-13 2020-01-16T00:14:37.968 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 13 v 143 (ObjNum 790 snap 0 seq_num 790) dirty exists 2020-01-16T00:14:37.968 INFO:tasks.rados.rados.0.smithi123.stdout:2041: left oid 13 (ObjNum 790 snap 0 seq_num 790) 2020-01-16T00:14:37.968 INFO:tasks.rados.rados.0.smithi123.stdout:2042: expect (ObjNum 780 snap 0 seq_num 780) 2020-01-16T00:14:38.015 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:38.024 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2020-01-16T00:14:38.056 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:38.052+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:38.128 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:38.138 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2020-01-16T00:14:38.182 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:38.182+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:38.232 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:38.235 INFO:tasks.rados.rados.0.smithi123.stdout:2043: finishing write tid 1 to smithi12313934-17 2020-01-16T00:14:38.235 INFO:tasks.rados.rados.0.smithi123.stdout:2043: finishing write tid 2 to smithi12313934-17 2020-01-16T00:14:38.235 INFO:tasks.rados.rados.0.smithi123.stdout:2043: finishing write tid 3 to smithi12313934-17 2020-01-16T00:14:38.236 INFO:tasks.rados.rados.0.smithi123.stdout:2043: finishing write tid 5 to smithi12313934-17 2020-01-16T00:14:38.236 INFO:tasks.rados.rados.0.smithi123.stdout:2043: finishing write tid 6 to smithi12313934-17 2020-01-16T00:14:38.236 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 17 v 226 (ObjNum 791 snap 0 seq_num 791) dirty exists 2020-01-16T00:14:38.236 INFO:tasks.rados.rados.0.smithi123.stdout:2043: left oid 17 (ObjNum 791 snap 0 seq_num 791) 2020-01-16T00:14:38.236 INFO:tasks.rados.rados.0.smithi123.stdout:2045: finishing write tid 1 to smithi12313934-48 2020-01-16T00:14:38.236 INFO:tasks.rados.rados.0.smithi123.stdout:2045: finishing write tid 2 to smithi12313934-48 2020-01-16T00:14:38.237 INFO:tasks.rados.rados.0.smithi123.stdout:2045: finishing write tid 3 to smithi12313934-48 2020-01-16T00:14:38.237 INFO:tasks.rados.rados.0.smithi123.stdout:2046: finishing write tid 1 to smithi12313934-27 2020-01-16T00:14:38.237 INFO:tasks.rados.rados.0.smithi123.stdout:2046: finishing write tid 2 to smithi12313934-27 2020-01-16T00:14:38.237 INFO:tasks.rados.rados.0.smithi123.stdout:2045: finishing write tid 5 to smithi12313934-48 2020-01-16T00:14:38.237 INFO:tasks.rados.rados.0.smithi123.stdout:2045: finishing write tid 6 to smithi12313934-48 2020-01-16T00:14:38.237 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 48 v 256 (ObjNum 792 snap 0 seq_num 792) dirty exists 2020-01-16T00:14:38.238 INFO:tasks.rados.rados.0.smithi123.stdout:2045: left oid 48 (ObjNum 792 snap 0 seq_num 792) 2020-01-16T00:14:38.238 INFO:tasks.rados.rados.0.smithi123.stdout:2046: finishing write tid 3 to smithi12313934-27 2020-01-16T00:14:38.238 INFO:tasks.rados.rados.0.smithi123.stdout:2046: finishing write tid 5 to smithi12313934-27 2020-01-16T00:14:38.238 INFO:tasks.rados.rados.0.smithi123.stdout:2033: done (15 left) 2020-01-16T00:14:38.238 INFO:tasks.rados.rados.0.smithi123.stdout:2034: done (14 left) 2020-01-16T00:14:38.238 INFO:tasks.rados.rados.0.smithi123.stdout:2035: done (13 left) 2020-01-16T00:14:38.238 INFO:tasks.rados.rados.0.smithi123.stdout:2036: done (12 left) 2020-01-16T00:14:38.239 INFO:tasks.rados.rados.0.smithi123.stdout:2037: done (11 left) 2020-01-16T00:14:38.239 INFO:tasks.rados.rados.0.smithi123.stdout:2039: done (10 left) 2020-01-16T00:14:38.239 INFO:tasks.rados.rados.0.smithi123.stdout:2040: done (9 left) 2020-01-16T00:14:38.239 INFO:tasks.rados.rados.0.smithi123.stdout:2041: done (8 left) 2020-01-16T00:14:38.239 INFO:tasks.rados.rados.0.smithi123.stdout:2042: done (7 left) 2020-01-16T00:14:38.239 INFO:tasks.rados.rados.0.smithi123.stdout:2043: done (6 left) 2020-01-16T00:14:38.239 INFO:tasks.rados.rados.0.smithi123.stdout:2044: done (5 left) 2020-01-16T00:14:38.240 INFO:tasks.rados.rados.0.smithi123.stdout:2045: done (4 left) 2020-01-16T00:14:38.240 INFO:tasks.rados.rados.0.smithi123.stdout: waiting on 4 2020-01-16T00:14:38.240 INFO:tasks.rados.rados.0.smithi123.stdout:2046: finishing write tid 6 to smithi12313934-27 2020-01-16T00:14:38.240 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 27 v 209 (ObjNum 793 snap 0 seq_num 793) dirty exists 2020-01-16T00:14:38.240 INFO:tasks.rados.rados.0.smithi123.stdout:2046: left oid 27 (ObjNum 793 snap 0 seq_num 793) 2020-01-16T00:14:38.240 INFO:tasks.rados.rados.0.smithi123.stdout:2047: expect (ObjNum 776 snap 0 seq_num 776) 2020-01-16T00:14:38.241 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2020-01-16T00:14:38.282 INFO:tasks.ceph.osd.0.smithi123.stderr:2020-01-16T00:14:38.282+0000 7f4692a2c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 11896) UID: 0 2020-01-16T00:14:38.360 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:38.356+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:38.379 INFO:tasks.rados.rados.0.smithi123.stdout:2048: finishing write tid 1 to smithi12313934-42 2020-01-16T00:14:38.379 INFO:tasks.rados.rados.0.smithi123.stdout:2049: expect (ObjNum 771 snap 0 seq_num 771) 2020-01-16T00:14:38.385 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:38.395 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2020-01-16T00:14:38.461 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:38.460+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:38.498 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:38.507 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2020-01-16T00:14:38.561 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:38.560+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:38.614 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:38.615 INFO:tasks.rados.rados.0.smithi123.stdout:2048: finishing write tid 2 to smithi12313934-42 2020-01-16T00:14:38.615 INFO:tasks.rados.rados.0.smithi123.stdout:2048: finishing write tid 3 to smithi12313934-42 2020-01-16T00:14:38.615 INFO:tasks.rados.rados.0.smithi123.stdout:2048: finishing write tid 5 to smithi12313934-42 2020-01-16T00:14:38.615 INFO:tasks.rados.rados.0.smithi123.stdout:2048: finishing write tid 6 to smithi12313934-42 2020-01-16T00:14:38.615 INFO:tasks.rados.rados.0.smithi123.stdout:update_object_version oid 42 v 261 (ObjNum 794 snap 0 seq_num 794) dirty exists 2020-01-16T00:14:38.615 INFO:tasks.rados.rados.0.smithi123.stdout:2048: left oid 42 (ObjNum 794 snap 0 seq_num 794) 2020-01-16T00:14:38.616 INFO:tasks.rados.rados.0.smithi123.stdout:2046: done (3 left) 2020-01-16T00:14:38.616 INFO:tasks.rados.rados.0.smithi123.stdout:2047: done (2 left) 2020-01-16T00:14:38.616 INFO:tasks.rados.rados.0.smithi123.stdout:2048: done (1 left) 2020-01-16T00:14:38.616 INFO:tasks.rados.rados.0.smithi123.stdout:2049: done (0 left) 2020-01-16T00:14:38.619 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2020-01-16T00:14:38.619 INFO:tasks.rados.rados.0.smithi123.stderr:0 errors. 2020-01-16T00:14:38.619 INFO:tasks.rados.rados.0.smithi123.stderr: 2020-01-16T00:14:38.623 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:14:38.662 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:38.660+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:38.726 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:38.736 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2020-01-16T00:14:38.782 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:38.782+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:38.837 INFO:teuthology.orchestra.run.smithi123.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:38.846 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2020-01-16T00:14:38.893 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:38.892+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:38.901 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:14:38.901 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":70,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:14:37.129338+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2020-01-16T00:12:31.098841+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:14.368773+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.895777+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.185848+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.667896+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:14.148239+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.986248+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.699155+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:14:38.918 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-01-16T00:14:38.961 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:38.966 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:38.966+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:38.971 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2020-01-16T00:14:39.066 INFO:tasks.ceph.osd.2.smithi123.stderr:2020-01-16T00:14:39.066+0000 7f6cfa513700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 11902) UID: 0 2020-01-16T00:14:39.120 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:39.129 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2020-01-16T00:14:39.156 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2020-01-16T00:14:39.160 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:14:39.161 INFO:teuthology.orchestra.run.smithi123.stderr:dumped all 2020-01-16T00:14:39.172 INFO:teuthology.orchestra.run.smithi123.stdout:{"pg_ready":true,"pg_map":{"version":135,"stamp":"2020-01-16T00:14:37.991755+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":248244769,"num_objects":95,"num_object_clones":0,"num_object_copies":285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95,"num_whiteouts":0,"num_read":6616,"num_read_kb":5563808,"num_write":3852,"num_write_kb":1378696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":18,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":864,"ondisk_log_size":864,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":84,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":9047492,"kb_used_data":630472,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":745927228,"statfs":{"total":773094113280,"available":763829481472,"internally_reserved":8589934592,"allocated":645603328,"data_stored":641020015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3663497,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":212,"num_read_kb":148355,"num_write":134,"num_write_kb":39047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.886960"},"pg_stats":[{"pgid":"3.f","version":"69'57","reported_seq":"175","reported_epoch":"69","state":"active+clean","last_fresh":"2020-01-16T00:14:34.274719+0000","last_change":"2020-01-16T00:13:40.789270+0000","last_active":"2020-01-16T00:14:34.274719+0000","last_peered":"2020-01-16T00:14:34.274719+0000","last_clean":"2020-01-16T00:14:34.274719+0000","last_became_active":"2020-01-16T00:12:32.829304+0000","last_became_peered":"2020-01-16T00:12:32.829304+0000","last_unstale":"2020-01-16T00:14:34.274719+0000","last_undegraded":"2020-01-16T00:14:34.274719+0000","last_fullsized":"2020-01-16T00:14:34.274719+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'33","last_scrub_stamp":"2020-01-16T00:13:40.789179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:13:40.789179+0000","log_size":57,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2857355,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":146,"num_read_kb":137936,"num_write":63,"num_write_kb":17193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":1,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"67'102","reported_seq":"242","reported_epoch":"68","state":"active+clean","last_fresh":"2020-01-16T00:14:31.800260+0000","last_change":"2020-01-16T00:14:14.019457+0000","last_active":"2020-01-16T00:14:31.800260+0000","last_peered":"2020-01-16T00:14:31.800260+0000","last_clean":"2020-01-16T00:14:31.800260+0000","last_became_active":"2020-01-16T00:12:32.829047+0000","last_became_peered":"2020-01-16T00:12:32.829047+0000","last_unstale":"2020-01-16T00:14:31.800260+0000","last_undegraded":"2020-01-16T00:14:31.800260+0000","last_fullsized":"2020-01-16T00:14:31.800260+0000","mapping_epoch":20,"log_start":"67'100","ondisk_log_start":"67'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'89","last_scrub_stamp":"2020-01-16T00:14:14.019244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:14:14.019244+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5008816,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":190,"num_read_kb":170488,"num_write":106,"num_write_kb":31868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"69'324","reported_seq":"645","reported_epoch":"69","state":"active+undersized+degraded","last_fresh":"2020-01-16T00:14:34.233196+0000","last_change":"2020-01-16T00:13:15.861286+0000","last_active":"2020-01-16T00:14:34.233196+0000","last_peered":"2020-01-16T00:14:34.233196+0000","last_clean":"2020-01-16T00:13:14.009235+0000","last_became_active":"2020-01-16T00:13:15.861286+0000","last_became_peered":"2020-01-16T00:13:15.861286+0000","last_unstale":"2020-01-16T00:14:34.233196+0000","last_undegraded":"2020-01-16T00:13:15.858843+0000","last_fullsized":"2020-01-16T00:13:15.858679+0000","mapping_epoch":40,"log_start":"65'300","ondisk_log_start":"65'300","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14387551,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":635,"num_read_kb":559048,"num_write":354,"num_write_kb":102793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":5,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"68'257","reported_seq":"547","reported_epoch":"68","state":"active+undersized+degraded","last_fresh":"2020-01-16T00:14:33.948435+0000","last_change":"2020-01-16T00:13:15.860978+0000","last_active":"2020-01-16T00:14:33.948435+0000","last_peered":"2020-01-16T00:14:33.948435+0000","last_clean":"2020-01-16T00:13:14.664367+0000","last_became_active":"2020-01-16T00:13:15.860978+0000","last_became_peered":"2020-01-16T00:13:15.860978+0000","last_unstale":"2020-01-16T00:14:33.948435+0000","last_undegraded":"2020-01-16T00:13:15.855215+0000","last_fullsized":"2020-01-16T00:13:15.855056+0000","mapping_epoch":40,"log_start":"56'200","ondisk_log_start":"56'200","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":57,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10941984,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":574,"num_read_kb":466446,"num_write":271,"num_write_kb":81185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":4,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"68'137","reported_seq":"305","reported_epoch":"68","state":"active+clean","last_fresh":"2020-01-16T00:14:32.254641+0000","last_change":"2020-01-16T00:13:49.818288+0000","last_active":"2020-01-16T00:14:32.254641+0000","last_peered":"2020-01-16T00:14:32.254641+0000","last_clean":"2020-01-16T00:14:32.254641+0000","last_became_active":"2020-01-16T00:12:32.833051+0000","last_became_peered":"2020-01-16T00:12:32.833051+0000","last_unstale":"2020-01-16T00:14:32.254641+0000","last_undegraded":"2020-01-16T00:14:32.254641+0000","last_fullsized":"2020-01-16T00:14:32.254641+0000","mapping_epoch":20,"log_start":"49'100","ondisk_log_start":"49'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'109","last_scrub_stamp":"2020-01-16T00:13:49.818225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:13:49.818225+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6437378,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":270,"num_read_kb":243793,"num_write":152,"num_write_kb":44853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"67'279","reported_seq":"540","reported_epoch":"68","state":"active+clean","last_fresh":"2020-01-16T00:14:32.243057+0000","last_change":"2020-01-16T00:13:49.779771+0000","last_active":"2020-01-16T00:14:32.243057+0000","last_peered":"2020-01-16T00:14:32.243057+0000","last_clean":"2020-01-16T00:14:32.243057+0000","last_became_active":"2020-01-16T00:12:32.832936+0000","last_became_peered":"2020-01-16T00:12:32.832936+0000","last_unstale":"2020-01-16T00:14:32.243057+0000","last_undegraded":"2020-01-16T00:14:32.243057+0000","last_fullsized":"2020-01-16T00:14:32.243057+0000","mapping_epoch":20,"log_start":"48'200","ondisk_log_start":"48'200","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'214","last_scrub_stamp":"2020-01-16T00:13:49.779648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:13:49.779648+0000","log_size":79,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8051015,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":468,"num_read_kb":370157,"num_write":282,"num_write_kb":81988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":4,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"69'325","reported_seq":"628","reported_epoch":"69","state":"active+clean","last_fresh":"2020-01-16T00:14:34.292223+0000","last_change":"2020-01-16T00:13:49.206039+0000","last_active":"2020-01-16T00:14:34.292223+0000","last_peered":"2020-01-16T00:14:34.292223+0000","last_clean":"2020-01-16T00:14:34.292223+0000","last_became_active":"2020-01-16T00:12:32.832272+0000","last_became_peered":"2020-01-16T00:12:32.832272+0000","last_unstale":"2020-01-16T00:14:34.292223+0000","last_undegraded":"2020-01-16T00:14:34.292223+0000","last_fullsized":"2020-01-16T00:14:34.292223+0000","mapping_epoch":20,"log_start":"66'300","ondisk_log_start":"66'300","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'231","last_scrub_stamp":"2020-01-16T00:13:49.205837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:13:49.205837+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14790952,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":579,"num_read_kb":507003,"num_write":345,"num_write_kb":101215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":5,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"68'211","reported_seq":"469","reported_epoch":"69","state":"active+clean","last_fresh":"2020-01-16T00:14:34.334361+0000","last_change":"2020-01-16T00:13:50.843445+0000","last_active":"2020-01-16T00:14:34.334361+0000","last_peered":"2020-01-16T00:14:34.334361+0000","last_clean":"2020-01-16T00:14:34.334361+0000","last_became_active":"2020-01-16T00:12:32.830301+0000","last_became_peered":"2020-01-16T00:12:32.830301+0000","last_unstale":"2020-01-16T00:14:34.334361+0000","last_undegraded":"2020-01-16T00:14:34.334361+0000","last_fullsized":"2020-01-16T00:14:34.334361+0000","mapping_epoch":20,"log_start":"66'200","ondisk_log_start":"66'200","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'153","last_scrub_stamp":"2020-01-16T00:13:50.843316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:13:50.843316+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7496417,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":506,"num_read_kb":461383,"num_write":210,"num_write_kb":60942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"69'207","reported_seq":"418","reported_epoch":"69","state":"active+clean","last_fresh":"2020-01-16T00:14:34.305222+0000","last_change":"2020-01-16T00:13:41.746443+0000","last_active":"2020-01-16T00:14:34.305222+0000","last_peered":"2020-01-16T00:14:34.305222+0000","last_clean":"2020-01-16T00:14:34.305222+0000","last_became_active":"2020-01-16T00:12:32.829571+0000","last_became_peered":"2020-01-16T00:12:32.829571+0000","last_unstale":"2020-01-16T00:14:34.305222+0000","last_undegraded":"2020-01-16T00:14:34.305222+0000","last_fullsized":"2020-01-16T00:14:34.305222+0000","mapping_epoch":20,"log_start":"67'201","ondisk_log_start":"67'201","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'150","last_scrub_stamp":"2020-01-16T00:13:41.746340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:13:41.746340+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5479412,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":359,"num_read_kb":290085,"num_write":199,"num_write_kb":60084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"65","reported_epoch":"68","state":"active+clean","last_fresh":"2020-01-16T00:14:32.097070+0000","last_change":"2020-01-16T00:13:50.158057+0000","last_active":"2020-01-16T00:14:32.097070+0000","last_peered":"2020-01-16T00:14:32.097070+0000","last_clean":"2020-01-16T00:14:32.097070+0000","last_became_active":"2020-01-16T00:12:20.030776+0000","last_became_peered":"2020-01-16T00:12:20.030776+0000","last_unstale":"2020-01-16T00:14:32.097070+0000","last_undegraded":"2020-01-16T00:14:32.097070+0000","last_fullsized":"2020-01-16T00:14:32.097070+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:13:50.157899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:13:50.157899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"60","reported_epoch":"68","state":"active+clean","last_fresh":"2020-01-16T00:14:31.927870+0000","last_change":"2020-01-16T00:13:56.070112+0000","last_active":"2020-01-16T00:14:31.927870+0000","last_peered":"2020-01-16T00:14:31.927870+0000","last_clean":"2020-01-16T00:14:31.927870+0000","last_became_active":"2020-01-16T00:12:32.831847+0000","last_became_peered":"2020-01-16T00:12:32.831847+0000","last_unstale":"2020-01-16T00:14:31.927870+0000","last_undegraded":"2020-01-16T00:14:31.927870+0000","last_fullsized":"2020-01-16T00:14:31.927870+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:13:56.070021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:13:56.070021+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"68","reported_epoch":"68","state":"active+undersized","last_fresh":"2020-01-16T00:14:31.155741+0000","last_change":"2020-01-16T00:13:15.872850+0000","last_active":"2020-01-16T00:14:31.155741+0000","last_peered":"2020-01-16T00:14:31.155741+0000","last_clean":"2020-01-16T00:13:13.833273+0000","last_became_active":"2020-01-16T00:13:15.872850+0000","last_became_peered":"2020-01-16T00:13:15.872850+0000","last_unstale":"2020-01-16T00:14:31.155741+0000","last_undegraded":"2020-01-16T00:14:31.155741+0000","last_fullsized":"2020-01-16T00:13:15.855475+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"65","reported_epoch":"68","state":"active+clean","last_fresh":"2020-01-16T00:14:31.106789+0000","last_change":"2020-01-16T00:13:25.750462+0000","last_active":"2020-01-16T00:14:31.106789+0000","last_peered":"2020-01-16T00:14:31.106789+0000","last_clean":"2020-01-16T00:14:31.106789+0000","last_became_active":"2020-01-16T00:12:20.031560+0000","last_became_peered":"2020-01-16T00:12:20.031560+0000","last_unstale":"2020-01-16T00:14:31.106789+0000","last_undegraded":"2020-01-16T00:14:31.106789+0000","last_fullsized":"2020-01-16T00:14:31.106789+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:13:25.750391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:13:25.750391+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"65'102","reported_seq":"206","reported_epoch":"68","state":"active+clean","last_fresh":"2020-01-16T00:14:31.800892+0000","last_change":"2020-01-16T00:13:37.746554+0000","last_active":"2020-01-16T00:14:31.800892+0000","last_peered":"2020-01-16T00:14:31.800892+0000","last_clean":"2020-01-16T00:14:31.800892+0000","last_became_active":"2020-01-16T00:12:31.816785+0000","last_became_peered":"2020-01-16T00:12:31.816785+0000","last_unstale":"2020-01-16T00:14:31.800892+0000","last_undegraded":"2020-01-16T00:14:31.800892+0000","last_fullsized":"2020-01-16T00:14:31.800892+0000","mapping_epoch":19,"log_start":"65'100","ondisk_log_start":"65'100","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'78","last_scrub_stamp":"2020-01-16T00:13:37.746305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:30.783643+0000","last_clean_scrub_stamp":"2020-01-16T00:13:37.746305+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33008486,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":43,"num_read_kb":17379,"num_write":102,"num_write_kb":80888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"68'138","reported_seq":"316","reported_epoch":"68","state":"active+clean","last_fresh":"2020-01-16T00:14:32.192792+0000","last_change":"2020-01-16T00:14:01.060683+0000","last_active":"2020-01-16T00:14:32.192792+0000","last_peered":"2020-01-16T00:14:32.192792+0000","last_clean":"2020-01-16T00:14:32.192792+0000","last_became_active":"2020-01-16T00:12:32.836523+0000","last_became_peered":"2020-01-16T00:12:32.836523+0000","last_unstale":"2020-01-16T00:14:32.192792+0000","last_undegraded":"2020-01-16T00:14:32.192792+0000","last_fullsized":"2020-01-16T00:14:32.192792+0000","mapping_epoch":20,"log_start":"51'100","ondisk_log_start":"51'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'113","last_scrub_stamp":"2020-01-16T00:14:01.060617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:14:01.060617+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6487799,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":288,"num_read_kb":217659,"num_write":139,"num_write_kb":39841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":2,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"65","reported_epoch":"68","state":"active+clean","last_fresh":"2020-01-16T00:14:31.800111+0000","last_change":"2020-01-16T00:13:54.835347+0000","last_active":"2020-01-16T00:14:31.800111+0000","last_peered":"2020-01-16T00:14:31.800111+0000","last_clean":"2020-01-16T00:14:31.800111+0000","last_became_active":"2020-01-16T00:12:20.030794+0000","last_became_peered":"2020-01-16T00:12:20.030794+0000","last_unstale":"2020-01-16T00:14:31.800111+0000","last_undegraded":"2020-01-16T00:14:31.800111+0000","last_fullsized":"2020-01-16T00:14:31.800111+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:13:54.835143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:13:54.835143+0000","last_clean_scrub_stamp":"2020-01-16T00:13:54.835143+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"66'161","reported_seq":"259","reported_epoch":"68","state":"active+clean","last_fresh":"2020-01-16T00:14:31.107957+0000","last_change":"2020-01-16T00:14:02.617577+0000","last_active":"2020-01-16T00:14:31.107957+0000","last_peered":"2020-01-16T00:14:31.107957+0000","last_clean":"2020-01-16T00:14:31.107957+0000","last_became_active":"2020-01-16T00:12:31.817077+0000","last_became_peered":"2020-01-16T00:12:31.817077+0000","last_unstale":"2020-01-16T00:14:31.107957+0000","last_undegraded":"2020-01-16T00:14:31.107957+0000","last_fullsized":"2020-01-16T00:14:31.107957+0000","mapping_epoch":19,"log_start":"45'100","ondisk_log_start":"45'100","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'133","last_scrub_stamp":"2020-01-16T00:14:02.617471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:30.783643+0000","last_clean_scrub_stamp":"2020-01-16T00:14:02.617471+0000","log_size":61,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37445501,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":37,"num_read_kb":19216,"num_write":161,"num_write_kb":130662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"69'388","reported_seq":"797","reported_epoch":"69","state":"active+clean","last_fresh":"2020-01-16T00:14:34.334243+0000","last_change":"2020-01-16T00:14:14.744391+0000","last_active":"2020-01-16T00:14:34.334243+0000","last_peered":"2020-01-16T00:14:34.334243+0000","last_clean":"2020-01-16T00:14:34.334243+0000","last_became_active":"2020-01-16T00:12:32.829870+0000","last_became_peered":"2020-01-16T00:12:32.829870+0000","last_unstale":"2020-01-16T00:14:34.334243+0000","last_undegraded":"2020-01-16T00:14:34.334243+0000","last_fullsized":"2020-01-16T00:14:34.334243+0000","mapping_epoch":20,"log_start":"55'300","ondisk_log_start":"55'300","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"60'336","last_scrub_stamp":"2020-01-16T00:14:14.744322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:14:14.744322+0000","log_size":88,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13328743,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":869,"num_read_kb":783449,"num_write":380,"num_write_kb":109440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":6,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"65","reported_epoch":"68","state":"active+clean","last_fresh":"2020-01-16T00:14:31.800817+0000","last_change":"2020-01-16T00:14:05.938096+0000","last_active":"2020-01-16T00:14:31.800817+0000","last_peered":"2020-01-16T00:14:31.800817+0000","last_clean":"2020-01-16T00:14:31.800817+0000","last_became_active":"2020-01-16T00:12:20.030792+0000","last_became_peered":"2020-01-16T00:12:20.030792+0000","last_unstale":"2020-01-16T00:14:31.800817+0000","last_undegraded":"2020-01-16T00:14:31.800817+0000","last_fullsized":"2020-01-16T00:14:31.800817+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:14:05.937903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:14:05.937903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"67'131","reported_seq":"226","reported_epoch":"68","state":"active+undersized+degraded","last_fresh":"2020-01-16T00:14:31.107545+0000","last_change":"2020-01-16T00:13:15.860069+0000","last_active":"2020-01-16T00:14:31.107545+0000","last_peered":"2020-01-16T00:14:31.107545+0000","last_clean":"2020-01-16T00:13:10.183906+0000","last_became_active":"2020-01-16T00:13:15.860069+0000","last_became_peered":"2020-01-16T00:13:15.860069+0000","last_unstale":"2020-01-16T00:14:31.107545+0000","last_undegraded":"2020-01-16T00:13:15.855705+0000","last_fullsized":"2020-01-16T00:13:15.855556+0000","mapping_epoch":40,"log_start":"48'100","ondisk_log_start":"48'100","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:30.783643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:30.783643+0000","last_clean_scrub_stamp":"2020-01-16T00:12:30.783643+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30319399,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":34,"num_read_kb":16408,"num_write":131,"num_write_kb":94025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":13}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"69'246","reported_seq":"490","reported_epoch":"69","state":"active+undersized+degraded","last_fresh":"2020-01-16T00:14:34.318798+0000","last_change":"2020-01-16T00:13:15.861809+0000","last_active":"2020-01-16T00:14:34.318798+0000","last_peered":"2020-01-16T00:14:34.318798+0000","last_clean":"2020-01-16T00:13:13.832713+0000","last_became_active":"2020-01-16T00:13:15.861809+0000","last_became_peered":"2020-01-16T00:13:15.861809+0000","last_unstale":"2020-01-16T00:14:34.318798+0000","last_undegraded":"2020-01-16T00:13:15.854275+0000","last_fullsized":"2020-01-16T00:13:15.853934+0000","mapping_epoch":40,"log_start":"58'201","ondisk_log_start":"58'201","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":45,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9597259,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":444,"num_read_kb":368663,"num_write":228,"num_write_kb":68817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"37","reported_epoch":"68","state":"active+undersized","last_fresh":"2020-01-16T00:14:31.927988+0000","last_change":"2020-01-16T00:13:15.862578+0000","last_active":"2020-01-16T00:14:31.927988+0000","last_peered":"2020-01-16T00:14:31.927988+0000","last_clean":"2020-01-16T00:12:19.002864+0000","last_became_active":"2020-01-16T00:13:15.862578+0000","last_became_peered":"2020-01-16T00:13:15.862578+0000","last_unstale":"2020-01-16T00:14:31.927988+0000","last_undegraded":"2020-01-16T00:14:31.927988+0000","last_fullsized":"2020-01-16T00:13:15.857926+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:12:19.002864+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"69'81","reported_seq":"163","reported_epoch":"69","state":"active+undersized+degraded","last_fresh":"2020-01-16T00:14:34.263513+0000","last_change":"2020-01-16T00:13:15.860372+0000","last_active":"2020-01-16T00:14:34.263513+0000","last_peered":"2020-01-16T00:14:34.263513+0000","last_clean":"2020-01-16T00:13:13.829689+0000","last_became_active":"2020-01-16T00:13:15.860372+0000","last_became_peered":"2020-01-16T00:13:15.860372+0000","last_unstale":"2020-01-16T00:14:34.263513+0000","last_undegraded":"2020-01-16T00:13:15.856263+0000","last_fullsized":"2020-01-16T00:13:15.856136+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:30.783643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:30.783643+0000","last_clean_scrub_stamp":"2020-01-16T00:12:30.783643+0000","log_size":81,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19831996,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":21,"num_read_kb":10085,"num_write":81,"num_write_kb":80545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":8}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"66'201","reported_seq":"422","reported_epoch":"68","state":"active+clean","last_fresh":"2020-01-16T00:14:32.097316+0000","last_change":"2020-01-16T00:14:01.236081+0000","last_active":"2020-01-16T00:14:32.097316+0000","last_peered":"2020-01-16T00:14:32.097316+0000","last_clean":"2020-01-16T00:14:32.097316+0000","last_became_active":"2020-01-16T00:12:32.831577+0000","last_became_peered":"2020-01-16T00:12:32.831577+0000","last_unstale":"2020-01-16T00:14:32.097316+0000","last_undegraded":"2020-01-16T00:14:32.097316+0000","last_fullsized":"2020-01-16T00:14:32.097316+0000","mapping_epoch":20,"log_start":"36'100","ondisk_log_start":"36'100","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"55'173","last_scrub_stamp":"2020-01-16T00:14:01.235876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:14:01.235876+0000","log_size":101,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8706455,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":388,"num_read_kb":355962,"num_write":209,"num_write_kb":63292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"68'218","reported_seq":"448","reported_epoch":"68","state":"active+undersized+degraded","last_fresh":"2020-01-16T00:14:33.950268+0000","last_change":"2020-01-16T00:13:15.859104+0000","last_active":"2020-01-16T00:14:33.950268+0000","last_peered":"2020-01-16T00:14:33.950268+0000","last_clean":"2020-01-16T00:13:14.618463+0000","last_became_active":"2020-01-16T00:13:15.859104+0000","last_became_peered":"2020-01-16T00:13:15.859104+0000","last_unstale":"2020-01-16T00:14:33.950268+0000","last_undegraded":"2020-01-16T00:13:15.856688+0000","last_fullsized":"2020-01-16T00:13:15.856554+0000","mapping_epoch":40,"log_start":"59'200","ondisk_log_start":"59'200","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6084431,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":404,"num_read_kb":303633,"num_write":226,"num_write_kb":66572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":4,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"65","reported_epoch":"68","state":"active+clean","last_fresh":"2020-01-16T00:14:31.107699+0000","last_change":"2020-01-16T00:13:27.507502+0000","last_active":"2020-01-16T00:14:31.107699+0000","last_peered":"2020-01-16T00:14:31.107699+0000","last_clean":"2020-01-16T00:14:31.107699+0000","last_became_active":"2020-01-16T00:12:20.031576+0000","last_became_peered":"2020-01-16T00:12:20.031576+0000","last_unstale":"2020-01-16T00:14:31.107699+0000","last_undegraded":"2020-01-16T00:14:31.107699+0000","last_fullsized":"2020-01-16T00:14:31.107699+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:13:27.507325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:13:27.507325+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"69'202","reported_seq":"413","reported_epoch":"69","state":"active+undersized+degraded","last_fresh":"2020-01-16T00:14:34.212139+0000","last_change":"2020-01-16T00:13:15.857318+0000","last_active":"2020-01-16T00:14:34.212139+0000","last_peered":"2020-01-16T00:14:34.212139+0000","last_clean":"2020-01-16T00:13:13.275843+0000","last_became_active":"2020-01-16T00:13:15.857318+0000","last_became_peered":"2020-01-16T00:13:15.857318+0000","last_unstale":"2020-01-16T00:14:34.212139+0000","last_undegraded":"2020-01-16T00:13:15.854334+0000","last_fullsized":"2020-01-16T00:13:15.854171+0000","mapping_epoch":40,"log_start":"37'101","ondisk_log_start":"37'101","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:31.792816+0000","last_clean_scrub_stamp":"2020-01-16T00:12:31.792816+0000","log_size":101,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7983820,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":361,"num_read_kb":265015,"num_write":213,"num_write_kb":62493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":3,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"65","reported_epoch":"68","state":"active+clean","last_fresh":"2020-01-16T00:14:31.107885+0000","last_change":"2020-01-16T00:13:25.472079+0000","last_active":"2020-01-16T00:14:31.107885+0000","last_peered":"2020-01-16T00:14:31.107885+0000","last_clean":"2020-01-16T00:14:31.107885+0000","last_became_active":"2020-01-16T00:12:20.033645+0000","last_became_peered":"2020-01-16T00:12:20.033645+0000","last_unstale":"2020-01-16T00:14:31.107885+0000","last_undegraded":"2020-01-16T00:14:31.107885+0000","last_fullsized":"2020-01-16T00:14:31.107885+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-01-16T00:13:25.471988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-01-16T00:12:19.002864+0000","last_clean_scrub_stamp":"2020-01-16T00:13:25.471988+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":127639387,"num_objects":45,"num_object_clones":0,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":6481,"num_read_kb":5500720,"num_write":3377,"num_write_kb":992576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":50,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":18,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":279031808,"data_stored":277860848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":689,"ondisk_log_size":689,"up":43,"acting":43,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":22,"acting":22,"num_store_stats":8},{"poolid":2,"num_pg":4,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":135,"num_read_kb":63088,"num_write":475,"num_write_kb":386120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":362127360,"data_stored":361816146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":175,"ondisk_log_size":175,"up":10,"acting":10,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574878,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1149256,"kb_used_data":97812,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93222584,"statfs":{"total":96636764160,"available":95459926016,"internally_reserved":1073741824,"allocated":100159488,"data_stored":99543228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574879,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1117776,"kb_used_data":65916,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93254064,"statfs":{"total":96636764160,"available":95492161536,"internally_reserved":1073741824,"allocated":67497984,"data_stored":66894003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574878,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1137404,"kb_used_data":81904,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93234436,"statfs":{"total":96636764160,"available":95472062464,"internally_reserved":1073741824,"allocated":83869696,"data_stored":83285095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574878,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1142436,"kb_used_data":91532,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93229404,"statfs":{"total":96636764160,"available":95466909696,"internally_reserved":1073741824,"allocated":93728768,"data_stored":93179291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574878,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1145632,"kb_used_data":95380,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93226208,"statfs":{"total":96636764160,"available":95463636992,"internally_reserved":1073741824,"allocated":97669120,"data_stored":97066110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574878,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1132772,"kb_used_data":76940,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93239068,"statfs":{"total":96636764160,"available":95476805632,"internally_reserved":1073741824,"allocated":78786560,"data_stored":78202390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574881,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1090512,"kb_used_data":38408,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93281328,"statfs":{"total":96636764160,"available":95520079872,"internally_reserved":1073741824,"allocated":39329792,"data_stored":38763450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574861,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1131704,"kb_used_data":82580,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93240136,"statfs":{"total":96636764160,"available":95477899264,"internally_reserved":1073741824,"allocated":84561920,"data_stored":84086448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}]}} 2020-01-16T00:14:39.175 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2020-01-16T00:14:39.175 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2020-01-16T00:14:39.176 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:39.172+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:39.233 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:39.242 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2020-01-16T00:14:39.268 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:39.266+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:39.358 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2020-01-16T00:14:39.392 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:39.401 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2020-01-16T00:14:39.402 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:39.402+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:39.469 INFO:tasks.ceph.osd.4.smithi145.stderr:2020-01-16T00:14:39.468+0000 7f4bbf603700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 11839) UID: 0 2020-01-16T00:14:39.482 INFO:teuthology.orchestra.run.smithi123.stdout:osd.0: {} 2020-01-16T00:14:39.502 INFO:teuthology.orchestra.run.smithi123.stdout:osd.1: {} 2020-01-16T00:14:39.509 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2020-01-16T00:14:39.519 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2020-01-16T00:14:39.522 INFO:teuthology.orchestra.run.smithi123.stdout:osd.2: {} 2020-01-16T00:14:39.540 INFO:teuthology.orchestra.run.smithi123.stdout:osd.3: {} 2020-01-16T00:14:39.561 INFO:teuthology.orchestra.run.smithi123.stdout:osd.4: {} 2020-01-16T00:14:39.575 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:39.574+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:39.586 INFO:teuthology.orchestra.run.smithi123.stdout:osd.6: {} 2020-01-16T00:14:39.609 INFO:teuthology.orchestra.run.smithi123.stdout:osd.7: {} 2020-01-16T00:14:39.612 INFO:teuthology.orchestra.run.smithi123.stderr:osd.0: osd_enable_op_tracker = 'true' 2020-01-16T00:14:39.612 INFO:teuthology.orchestra.run.smithi123.stderr:osd.1: osd_enable_op_tracker = 'true' 2020-01-16T00:14:39.612 INFO:teuthology.orchestra.run.smithi123.stderr:osd.2: osd_enable_op_tracker = 'true' 2020-01-16T00:14:39.612 INFO:teuthology.orchestra.run.smithi123.stderr:osd.3: osd_enable_op_tracker = 'true' 2020-01-16T00:14:39.612 INFO:teuthology.orchestra.run.smithi123.stderr:osd.4: osd_enable_op_tracker = 'true' 2020-01-16T00:14:39.612 INFO:teuthology.orchestra.run.smithi123.stderr:Error ENXIO: problem getting command descriptions from osd.5 2020-01-16T00:14:39.613 INFO:teuthology.orchestra.run.smithi123.stderr:osd.5: problem getting command descriptions from osd.5 2020-01-16T00:14:39.613 INFO:teuthology.orchestra.run.smithi123.stderr:osd.6: osd_enable_op_tracker = 'true' 2020-01-16T00:14:39.613 INFO:teuthology.orchestra.run.smithi123.stderr:osd.7: osd_enable_op_tracker = 'true' 2020-01-16T00:14:39.622 DEBUG:teuthology.orchestra.run:got remote process result: 6 2020-01-16T00:14:39.622 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2020-01-16T00:14:39.675 INFO:tasks.ceph.osd.1.smithi123.stderr:2020-01-16T00:14:39.674+0000 7fba440de700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 11897) UID: 0 2020-01-16T00:14:39.677 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:39.677 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:14:39.677 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:14:39.677 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:14:39.678 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.678 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.5.13:36 2.0 2:20e40cec:::smithi12313934-27 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 113152~1699840] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e24)", 2020-01-16T00:14:39.678 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:14:39.678 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 121.841652985, 2020-01-16T00:14:39.678 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011736636, 2020-01-16T00:14:39.678 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:39.678 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:39.679 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:39.679 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.679 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:39.679 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:14:39.679 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:39.679 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.679 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.680 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:39.680 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831939+0000", 2020-01-16T00:14:39.680 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999952 2020-01-16T00:14:39.681 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.681 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.681 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:39.681 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.831934+0000", 2020-01-16T00:14:39.681 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001034325 2020-01-16T00:14:39.681 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.681 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.682 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:39.682 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.832968+0000", 2020-01-16T00:14:39.682 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.506e-05 2020-01-16T00:14:39.682 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.682 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.682 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:39.682 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833004+0000", 2020-01-16T00:14:39.683 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3406e-05 2020-01-16T00:14:39.683 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.683 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.683 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:39.683 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833017+0000", 2020-01-16T00:14:39.683 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015477000000000001 2020-01-16T00:14:39.683 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.683 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.684 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:39.684 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833172+0000", 2020-01-16T00:14:39.684 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033750700000000002 2020-01-16T00:14:39.684 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.684 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.684 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:39.684 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.833509+0000", 2020-01-16T00:14:39.685 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00055861400000000003 2020-01-16T00:14:39.685 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.685 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.685 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:14:39.685 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.834068+0000", 2020-01-16T00:14:39.685 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.007474545 2020-01-16T00:14:39.685 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.687 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.687 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:39.687 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.841542+0000", 2020-01-16T00:14:39.687 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00056319499999999995 2020-01-16T00:14:39.687 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.687 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.687 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:39.687 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.842106+0000", 2020-01-16T00:14:39.688 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001371017 2020-01-16T00:14:39.688 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.688 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.688 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:39.688 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843477+0000", 2020-01-16T00:14:39.688 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000123938 2020-01-16T00:14:39.688 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.689 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.689 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:39.689 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843601+0000", 2020-01-16T00:14:39.689 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5012000000000006e-05 2020-01-16T00:14:39.689 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.689 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.689 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:39.689 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:37.843676+0000", 2020-01-16T00:14:39.690 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011736636 2020-01-16T00:14:39.690 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.690 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:39.690 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.690 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.690 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.690 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1276 3.d 3:b1521a3e:::smithi12313934-25:head [read 0~3882384,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:14:39.691 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:14:39.691 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 116.59889341500001, 2020-01-16T00:14:39.691 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015839157999999999, 2020-01-16T00:14:39.691 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:39.691 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:39.691 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:39.691 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:39.691 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:39.692 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1276 2020-01-16T00:14:39.692 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.692 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:39.692 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.692 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:39.692 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:14:39.692 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:39.693 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.693 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.693 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:39.693 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:14:39.693 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:39.693 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.693 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.693 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:39.694 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074698+0000", 2020-01-16T00:14:39.694 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5429999999999999e-06 2020-01-16T00:14:39.694 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.694 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.694 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:39.694 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074704+0000", 2020-01-16T00:14:39.694 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0839999999999992e-06 2020-01-16T00:14:39.695 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.695 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.695 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:39.695 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074712+0000", 2020-01-16T00:14:39.695 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.173e-06 2020-01-16T00:14:39.695 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.695 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.695 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:39.696 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074718+0000", 2020-01-16T00:14:39.696 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0083034700000000003 2020-01-16T00:14:39.696 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.696 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.696 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:39.696 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.083022+0000", 2020-01-16T00:14:39.696 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028456400000000001 2020-01-16T00:14:39.696 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.697 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.697 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:39.697 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.083306+0000", 2020-01-16T00:14:39.697 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0072312640000000003 2020-01-16T00:14:39.697 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.697 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.697 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:39.698 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090538+0000", 2020-01-16T00:14:39.698 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015839157999999999 2020-01-16T00:14:39.698 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.698 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:39.698 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.698 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.698 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.699 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1277 3.d 3:b1521a3e:::smithi12313934-25:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:14:39.699 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:14:39.699 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 116.598849825, 2020-01-16T00:14:39.699 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016646459999999998, 2020-01-16T00:14:39.699 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:39.699 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:39.699 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:39.699 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:39.700 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:39.700 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1277 2020-01-16T00:14:39.700 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.700 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:39.700 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.700 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:39.700 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:14:39.701 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:39.701 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.701 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.701 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:39.701 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074742+0000", 2020-01-16T00:14:39.701 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:39.702 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.702 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.702 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:39.702 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074741+0000", 2020-01-16T00:14:39.702 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0776e-05 2020-01-16T00:14:39.702 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.702 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.702 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:39.703 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074752+0000", 2020-01-16T00:14:39.703 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7600000000000002e-06 2020-01-16T00:14:39.703 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.703 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.703 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:39.703 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074759+0000", 2020-01-16T00:14:39.703 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6840000000000001e-06 2020-01-16T00:14:39.704 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.704 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.704 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:39.704 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074765+0000", 2020-01-16T00:14:39.704 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015815216 2020-01-16T00:14:39.704 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.704 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.705 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:39.705 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090580+0000", 2020-01-16T00:14:39.705 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000182404 2020-01-16T00:14:39.705 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.705 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.705 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:39.706 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.090763+0000", 2020-01-16T00:14:39.706 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00062579300000000003 2020-01-16T00:14:39.706 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.706 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.706 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:39.706 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091389+0000", 2020-01-16T00:14:39.706 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016646459999999998 2020-01-16T00:14:39.706 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.707 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:39.707 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.707 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.707 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.707 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1278 3.d 3:b1521a3e:::smithi12313934-25:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e26)", 2020-01-16T00:14:39.707 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:14:39.707 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 116.598820112, 2020-01-16T00:14:39.708 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017994915, 2020-01-16T00:14:39.708 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:39.708 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:39.708 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:39.708 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:39.708 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:39.708 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1278 2020-01-16T00:14:39.708 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.709 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:39.709 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.709 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:39.709 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:14:39.709 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:39.709 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.709 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.710 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:39.710 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074772+0000", 2020-01-16T00:14:39.710 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:39.710 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.710 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.710 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:39.710 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074771+0000", 2020-01-16T00:14:39.710 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4569999999999998e-06 2020-01-16T00:14:39.711 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.711 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.711 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:39.711 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074775+0000", 2020-01-16T00:14:39.711 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0330000000000003e-06 2020-01-16T00:14:39.711 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.711 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.712 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:39.712 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074782+0000", 2020-01-16T00:14:39.712 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9470000000000002e-06 2020-01-16T00:14:39.712 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.712 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.712 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:39.712 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.074787+0000", 2020-01-16T00:14:39.712 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01666134 2020-01-16T00:14:39.713 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.713 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.713 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:39.713 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091449+0000", 2020-01-16T00:14:39.713 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000247731 2020-01-16T00:14:39.713 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.714 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.714 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:39.714 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.091696+0000", 2020-01-16T00:14:39.714 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010703170000000001 2020-01-16T00:14:39.714 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.714 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.714 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:39.715 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:43.092767+0000", 2020-01-16T00:14:39.715 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017994915 2020-01-16T00:14:39.715 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.715 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:39.715 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.715 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.716 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.716 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2260 3.d 3:b76659c4:::smithi12313934-47:head [write 601750~535546] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:14:39.716 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:06.050577+0000", 2020-01-16T00:14:39.716 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 93.623014796000007, 2020-01-16T00:14:39.716 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012415381, 2020-01-16T00:14:39.716 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:39.716 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:39.717 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:39.717 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:39.717 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:39.717 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2260 2020-01-16T00:14:39.717 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.717 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:39.717 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.717 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:39.718 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.050577+0000", 2020-01-16T00:14:39.718 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:39.718 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.718 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.718 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:39.718 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.050577+0000", 2020-01-16T00:14:39.718 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:39.719 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.719 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.719 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:39.719 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.050574+0000", 2020-01-16T00:14:39.719 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001523487 2020-01-16T00:14:39.719 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.719 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.719 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:39.720 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052097+0000", 2020-01-16T00:14:39.720 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3697e-05 2020-01-16T00:14:39.720 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.720 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.720 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:39.720 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052121+0000", 2020-01-16T00:14:39.720 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0070000000000003e-06 2020-01-16T00:14:39.720 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.721 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.721 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:39.721 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052130+0000", 2020-01-16T00:14:39.721 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4559000000000003e-05 2020-01-16T00:14:39.721 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.721 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.722 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:39.722 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052194+0000", 2020-01-16T00:14:39.723 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000165878 2020-01-16T00:14:39.723 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.723 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.723 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:39.724 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052360+0000", 2020-01-16T00:14:39.724 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018825499999999999 2020-01-16T00:14:39.724 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.724 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.724 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:14:39.725 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.052549+0000", 2020-01-16T00:14:39.725 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002761782 2020-01-16T00:14:39.725 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.725 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.725 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:39.725 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.055310+0000", 2020-01-16T00:14:39.725 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00060385500000000004 2020-01-16T00:14:39.726 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.726 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.726 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:39.726 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.055914+0000", 2020-01-16T00:14:39.726 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0069733809999999999 2020-01-16T00:14:39.726 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.726 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.726 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:39.727 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.062888+0000", 2020-01-16T00:14:39.727 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2884999999999999e-05 2020-01-16T00:14:39.727 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.727 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.727 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:39.727 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.062940+0000", 2020-01-16T00:14:39.727 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1996999999999997e-05 2020-01-16T00:14:39.728 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.728 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.728 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:39.728 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.062992+0000", 2020-01-16T00:14:39.728 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012415381 2020-01-16T00:14:39.728 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.728 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:39.728 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.729 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.729 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.729 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2295 3.d 3:b1521a3e:::smithi12313934-25:head [write 1763838~505140] snapc 0=[] ondisk+write+known_if_redirected e35)", 2020-01-16T00:14:39.729 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:06.660258+0000", 2020-01-16T00:14:39.729 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 93.013333568999997, 2020-01-16T00:14:39.729 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011681408000000001, 2020-01-16T00:14:39.729 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:39.729 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:39.730 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:39.730 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:39.730 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:39.730 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2295 2020-01-16T00:14:39.730 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.730 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:39.730 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.731 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:39.731 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660258+0000", 2020-01-16T00:14:39.731 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:39.731 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.731 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.731 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:39.731 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660258+0000", 2020-01-16T00:14:39.731 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:39.732 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.732 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.732 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:39.732 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660256+0000", 2020-01-16T00:14:39.732 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00066501699999999995 2020-01-16T00:14:39.732 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.732 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.733 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:39.733 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660921+0000", 2020-01-16T00:14:39.733 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7339000000000001e-05 2020-01-16T00:14:39.733 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.733 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.733 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:39.734 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660938+0000", 2020-01-16T00:14:39.734 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0219999999999993e-06 2020-01-16T00:14:39.734 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.734 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.734 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:39.734 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.660947+0000", 2020-01-16T00:14:39.734 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00094237200000000002 2020-01-16T00:14:39.734 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.735 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.735 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:39.735 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.661890+0000", 2020-01-16T00:14:39.735 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000117649 2020-01-16T00:14:39.735 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.735 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.735 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:39.736 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662007+0000", 2020-01-16T00:14:39.736 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00060662599999999998 2020-01-16T00:14:39.736 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.736 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.736 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:39.736 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662614+0000", 2020-01-16T00:14:39.736 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028424899999999998 2020-01-16T00:14:39.736 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.737 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.737 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:39.737 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.662898+0000", 2020-01-16T00:14:39.737 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00054904599999999997 2020-01-16T00:14:39.737 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.737 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.737 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 5,6", 2020-01-16T00:14:39.738 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.663447+0000", 2020-01-16T00:14:39.738 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0051479990000000003 2020-01-16T00:14:39.738 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.738 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.738 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:39.738 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.668595+0000", 2020-01-16T00:14:39.738 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00080881799999999999 2020-01-16T00:14:39.738 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.739 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.739 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:39.739 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.669404+0000", 2020-01-16T00:14:39.739 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0024495290000000002 2020-01-16T00:14:39.739 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.739 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.739 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:39.739 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671854+0000", 2020-01-16T00:14:39.740 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6635000000000003e-05 2020-01-16T00:14:39.740 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.740 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.740 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:39.740 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671900+0000", 2020-01-16T00:14:39.740 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9567999999999998e-05 2020-01-16T00:14:39.740 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.741 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.741 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:39.741 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:06.671940+0000", 2020-01-16T00:14:39.741 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011681408000000001 2020-01-16T00:14:39.741 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.741 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:39.741 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.741 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.742 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.742 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.3.13:109 2.0 2:11fa79bc:::smithi12313934-15 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 331264~198656] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e42)", 2020-01-16T00:14:39.742 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:21.840075+0000", 2020-01-16T00:14:39.742 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 77.833517271999995, 2020-01-16T00:14:39.742 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011558677999999999, 2020-01-16T00:14:39.742 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:39.742 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:39.743 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:39.743 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.743 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:39.743 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840075+0000", 2020-01-16T00:14:39.743 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:39.743 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.743 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.744 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:39.744 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840075+0000", 2020-01-16T00:14:39.744 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:39.744 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.744 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.744 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:39.744 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840073+0000", 2020-01-16T00:14:39.744 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033576299999999999 2020-01-16T00:14:39.745 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.745 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.745 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:39.745 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840409+0000", 2020-01-16T00:14:39.745 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2082999999999998e-05 2020-01-16T00:14:39.745 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.745 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.746 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:39.746 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840431+0000", 2020-01-16T00:14:39.746 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4410000000000003e-06 2020-01-16T00:14:39.746 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.746 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.746 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:39.746 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840436+0000", 2020-01-16T00:14:39.747 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000101016 2020-01-16T00:14:39.747 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.747 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.747 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:39.747 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840537+0000", 2020-01-16T00:14:39.747 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000117639 2020-01-16T00:14:39.748 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.748 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.748 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:39.748 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840654+0000", 2020-01-16T00:14:39.748 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000196162 2020-01-16T00:14:39.748 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.748 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.749 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:14:39.749 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.840850+0000", 2020-01-16T00:14:39.749 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003319092 2020-01-16T00:14:39.749 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.749 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.749 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:39.749 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.844169+0000", 2020-01-16T00:14:39.750 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011349229999999999 2020-01-16T00:14:39.750 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.750 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.750 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:39.750 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.845304+0000", 2020-01-16T00:14:39.750 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0062517670000000001 2020-01-16T00:14:39.750 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.751 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.751 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:39.751 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851556+0000", 2020-01-16T00:14:39.751 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4138000000000003e-05 2020-01-16T00:14:39.751 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.751 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.751 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:39.751 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851600+0000", 2020-01-16T00:14:39.752 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2993000000000001e-05 2020-01-16T00:14:39.752 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.752 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.752 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:39.752 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:21.851633+0000", 2020-01-16T00:14:39.752 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011558677999999999 2020-01-16T00:14:39.752 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.752 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:39.753 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.753 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.753 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.753 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2987 3.d 3:b3dd5d45:::smithi12313934-16:head [write 467327~633480] snapc 0=[] ondisk+write+known_if_redirected e42)", 2020-01-16T00:14:39.753 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:22.257904+0000", 2020-01-16T00:14:39.753 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 77.415687685999998, 2020-01-16T00:14:39.753 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012187450000000001, 2020-01-16T00:14:39.754 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:39.754 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:39.754 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:39.754 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:39.754 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:39.754 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2987 2020-01-16T00:14:39.754 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.754 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:39.755 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.755 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:39.755 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.257904+0000", 2020-01-16T00:14:39.755 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:39.755 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.755 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.755 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:39.756 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.257904+0000", 2020-01-16T00:14:39.756 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999933 2020-01-16T00:14:39.756 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.756 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.756 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:39.756 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.257898+0000", 2020-01-16T00:14:39.756 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00097099800000000004 2020-01-16T00:14:39.756 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.757 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.757 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:39.757 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.258869+0000", 2020-01-16T00:14:39.757 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7970000000000003e-05 2020-01-16T00:14:39.757 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.757 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.757 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:39.758 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.258917+0000", 2020-01-16T00:14:39.758 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5982999999999999e-05 2020-01-16T00:14:39.758 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.758 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.758 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:39.758 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.258933+0000", 2020-01-16T00:14:39.758 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00026239999999999998 2020-01-16T00:14:39.758 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.759 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.759 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:39.759 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.259195+0000", 2020-01-16T00:14:39.759 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024907400000000001 2020-01-16T00:14:39.759 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.759 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.759 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:39.759 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.259444+0000", 2020-01-16T00:14:39.760 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00034824999999999998 2020-01-16T00:14:39.760 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.760 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.760 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:39.760 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.259792+0000", 2020-01-16T00:14:39.760 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003251538 2020-01-16T00:14:39.760 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.761 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.761 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:39.761 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.263044+0000", 2020-01-16T00:14:39.761 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0068502880000000004 2020-01-16T00:14:39.761 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.761 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.761 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:39.762 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.269894+0000", 2020-01-16T00:14:39.762 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0945000000000006e-05 2020-01-16T00:14:39.762 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.762 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.762 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:39.762 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.269965+0000", 2020-01-16T00:14:39.762 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012667399999999999 2020-01-16T00:14:39.762 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.763 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.763 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:39.763 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:22.270092+0000", 2020-01-16T00:14:39.763 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012187450000000001 2020-01-16T00:14:39.763 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.763 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:39.763 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.763 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.764 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.764 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3273 3.d 3:b76659c4:::smithi12313934-47:head [read 0~3647152,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:39.764 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.728775+0000", 2020-01-16T00:14:39.764 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 68.944817131999997, 2020-01-16T00:14:39.764 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012922950000000001, 2020-01-16T00:14:39.764 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:39.764 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:39.765 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:39.765 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:39.765 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:39.765 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3273 2020-01-16T00:14:39.765 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.765 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:39.765 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.765 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:39.766 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728775+0000", 2020-01-16T00:14:39.766 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:39.766 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.766 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.766 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:39.766 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728775+0000", 2020-01-16T00:14:39.766 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:39.767 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.767 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.767 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:39.767 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728771+0000", 2020-01-16T00:14:39.767 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.5009000000000001e-05 2020-01-16T00:14:39.767 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.767 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.767 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:39.768 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728796+0000", 2020-01-16T00:14:39.768 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2357999999999998e-05 2020-01-16T00:14:39.768 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.768 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.768 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:39.768 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728819+0000", 2020-01-16T00:14:39.768 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.4800000000000007e-06 2020-01-16T00:14:39.769 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.769 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.769 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:39.769 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728828+0000", 2020-01-16T00:14:39.769 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000124824 2020-01-16T00:14:39.769 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.770 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.770 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:39.770 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728953+0000", 2020-01-16T00:14:39.770 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00044904899999999998 2020-01-16T00:14:39.770 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.770 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.770 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:39.770 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729402+0000", 2020-01-16T00:14:39.771 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012295706999999999 2020-01-16T00:14:39.771 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.771 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.771 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:39.771 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.741698+0000", 2020-01-16T00:14:39.771 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012922950000000001 2020-01-16T00:14:39.771 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.772 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:39.772 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.772 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.772 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.772 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3275 3.d 3:b76659c4:::smithi12313934-47:head [read 0~3647152,checksum] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:39.772 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.728933+0000", 2020-01-16T00:14:39.772 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 68.944658860000004, 2020-01-16T00:14:39.772 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01204874, 2020-01-16T00:14:39.773 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:39.773 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:39.773 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:39.773 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:39.773 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:39.773 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3275 2020-01-16T00:14:39.773 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.774 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:39.774 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.774 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:39.774 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728933+0000", 2020-01-16T00:14:39.774 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:39.774 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.774 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.774 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:39.775 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728933+0000", 2020-01-16T00:14:39.775 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:39.775 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.775 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.775 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:39.775 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728932+0000", 2020-01-16T00:14:39.775 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3569999999999998e-06 2020-01-16T00:14:39.776 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.776 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.776 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:39.776 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728940+0000", 2020-01-16T00:14:39.776 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0457e-05 2020-01-16T00:14:39.776 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.776 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.776 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:39.777 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728951+0000", 2020-01-16T00:14:39.777 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6799999999999993e-06 2020-01-16T00:14:39.777 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.777 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.777 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:39.777 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.728959+0000", 2020-01-16T00:14:39.777 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010076307 2020-01-16T00:14:39.777 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.778 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.778 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:39.778 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.739035+0000", 2020-01-16T00:14:39.778 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024455400000000002 2020-01-16T00:14:39.778 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.778 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.778 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:39.779 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.739279+0000", 2020-01-16T00:14:39.779 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0017023660000000001 2020-01-16T00:14:39.779 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.779 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.779 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:39.779 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.740982+0000", 2020-01-16T00:14:39.779 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01204874 2020-01-16T00:14:39.779 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.780 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:39.780 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.780 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.780 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.780 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3285 3.d 3:bd9fc72f:::smithi12313934-5:head [read 0~3764801,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:39.780 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.729068+0000", 2020-01-16T00:14:39.780 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 68.944524068000007, 2020-01-16T00:14:39.780 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018537010999999999, 2020-01-16T00:14:39.781 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:39.781 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:39.781 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:39.781 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:39.781 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:39.781 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3285 2020-01-16T00:14:39.781 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.782 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:39.782 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.782 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:39.782 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729068+0000", 2020-01-16T00:14:39.782 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:39.782 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.782 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.782 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:39.783 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729068+0000", 2020-01-16T00:14:39.783 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:39.783 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.783 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.783 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:39.783 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729067+0000", 2020-01-16T00:14:39.783 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3339999999999996e-06 2020-01-16T00:14:39.784 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.784 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.784 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:39.784 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729075+0000", 2020-01-16T00:14:39.784 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4187e-05 2020-01-16T00:14:39.784 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.784 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.784 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:39.785 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729089+0000", 2020-01-16T00:14:39.785 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5600000000000004e-06 2020-01-16T00:14:39.785 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.785 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.785 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:39.785 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729093+0000", 2020-01-16T00:14:39.785 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011944309 2020-01-16T00:14:39.786 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.786 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.786 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:39.786 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.741038+0000", 2020-01-16T00:14:39.786 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016625799999999999 2020-01-16T00:14:39.786 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.786 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.786 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:39.787 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.741204+0000", 2020-01-16T00:14:39.787 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0064010120000000002 2020-01-16T00:14:39.787 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.787 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.787 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:39.787 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.747605+0000", 2020-01-16T00:14:39.788 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018537010999999999 2020-01-16T00:14:39.788 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.788 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:39.788 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.788 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.788 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.788 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3286 3.d 3:bd9fc72f:::smithi12313934-5:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:39.788 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.729099+0000", 2020-01-16T00:14:39.789 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 68.944493398000006, 2020-01-16T00:14:39.789 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018983897999999999, 2020-01-16T00:14:39.789 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:39.789 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:39.789 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:39.789 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:39.789 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:39.790 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3286 2020-01-16T00:14:39.790 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.790 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:39.790 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.790 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:39.790 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729099+0000", 2020-01-16T00:14:39.790 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:39.791 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.791 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.791 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:39.791 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729099+0000", 2020-01-16T00:14:39.791 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:39.791 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.791 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.791 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:39.792 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729098+0000", 2020-01-16T00:14:39.792 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8500000000000004e-06 2020-01-16T00:14:39.792 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.792 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.792 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:39.792 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729102+0000", 2020-01-16T00:14:39.792 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7329999999999997e-06 2020-01-16T00:14:39.793 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.793 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.793 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:39.793 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729108+0000", 2020-01-16T00:14:39.793 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.764e-06 2020-01-16T00:14:39.793 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.793 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.793 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:39.794 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729118+0000", 2020-01-16T00:14:39.794 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018524286000000001 2020-01-16T00:14:39.794 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.794 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.794 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:39.794 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.747642+0000", 2020-01-16T00:14:39.794 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9487999999999993e-05 2020-01-16T00:14:39.795 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.795 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.795 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:39.795 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.747742+0000", 2020-01-16T00:14:39.795 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000340429 2020-01-16T00:14:39.795 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.795 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.795 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:39.796 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.748082+0000", 2020-01-16T00:14:39.796 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018983897999999999 2020-01-16T00:14:39.796 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.796 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:39.796 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.796 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.796 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.797 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3287 3.d 3:bd9fc72f:::smithi12313934-5:head [read 0~3764801,checksum] snapc 0=[] ondisk+read+known_if_redirected e45)", 2020-01-16T00:14:39.797 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:30.729124+0000", 2020-01-16T00:14:39.797 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 68.944468259000004, 2020-01-16T00:14:39.797 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020552323000000001, 2020-01-16T00:14:39.797 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:39.797 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:39.797 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:39.798 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:39.798 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:39.798 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3287 2020-01-16T00:14:39.798 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.798 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:39.798 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.798 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:39.798 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729124+0000", 2020-01-16T00:14:39.799 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:39.799 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.799 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.799 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:39.799 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729124+0000", 2020-01-16T00:14:39.799 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:39.799 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.800 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.800 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:39.800 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729123+0000", 2020-01-16T00:14:39.800 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8239999999999996e-06 2020-01-16T00:14:39.800 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.800 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.800 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:39.800 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729129+0000", 2020-01-16T00:14:39.801 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6909999999999997e-06 2020-01-16T00:14:39.801 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.801 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.801 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:39.801 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729136+0000", 2020-01-16T00:14:39.801 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3329999999999997e-06 2020-01-16T00:14:39.801 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.802 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.802 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:39.802 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.729145+0000", 2020-01-16T00:14:39.802 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018965065999999999 2020-01-16T00:14:39.802 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.803 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.803 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:39.803 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.748110+0000", 2020-01-16T00:14:39.803 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015781700000000001 2020-01-16T00:14:39.803 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.803 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.803 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:39.804 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.748268+0000", 2020-01-16T00:14:39.804 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014083800000000001 2020-01-16T00:14:39.804 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.804 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.804 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:39.804 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:30.749676+0000", 2020-01-16T00:14:39.804 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020552323000000001 2020-01-16T00:14:39.805 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.805 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:39.805 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.805 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.805 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.805 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3409 3.d 3:b3dd5d45:::smithi12313934-16:head [write 591712~786646,stat] snapc 0=[] ondisk+write+known_if_redirected e47)", 2020-01-16T00:14:39.805 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:34.100681+0000", 2020-01-16T00:14:39.806 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 65.572911118999997, 2020-01-16T00:14:39.806 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012248446, 2020-01-16T00:14:39.806 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:39.806 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:39.806 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:39.806 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:39.806 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:39.807 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3409 2020-01-16T00:14:39.807 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.807 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:39.807 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.807 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:39.807 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.100681+0000", 2020-01-16T00:14:39.807 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:39.807 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.808 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.808 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:39.808 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.100681+0000", 2020-01-16T00:14:39.808 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:14:39.808 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.808 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.808 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:39.808 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.100677+0000", 2020-01-16T00:14:39.809 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000998828 2020-01-16T00:14:39.809 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.809 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.809 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:39.809 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101676+0000", 2020-01-16T00:14:39.809 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5846999999999997e-05 2020-01-16T00:14:39.809 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.810 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.810 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:39.810 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101712+0000", 2020-01-16T00:14:39.810 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.364e-05 2020-01-16T00:14:39.810 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.810 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.810 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:39.810 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101725+0000", 2020-01-16T00:14:39.811 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015902700000000001 2020-01-16T00:14:39.811 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.811 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.811 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:39.811 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101884+0000", 2020-01-16T00:14:39.811 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0486e-05 2020-01-16T00:14:39.811 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.812 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.812 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op must wait for map", 2020-01-16T00:14:39.812 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.101905+0000", 2020-01-16T00:14:39.812 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0038869009999999999 2020-01-16T00:14:39.812 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.812 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.812 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:39.812 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.105792+0000", 2020-01-16T00:14:39.813 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030748800000000002 2020-01-16T00:14:39.813 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.813 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.813 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:39.813 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.106099+0000", 2020-01-16T00:14:39.813 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00042069299999999998 2020-01-16T00:14:39.813 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.813 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.814 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 6", 2020-01-16T00:14:39.814 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.106520+0000", 2020-01-16T00:14:39.814 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0052864560000000001 2020-01-16T00:14:39.814 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.814 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.814 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:39.814 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.111806+0000", 2020-01-16T00:14:39.815 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00080835800000000001 2020-01-16T00:14:39.815 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.815 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.815 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:39.815 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.112615+0000", 2020-01-16T00:14:39.815 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1490999999999997e-05 2020-01-16T00:14:39.816 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.816 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.816 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:39.816 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.112686+0000", 2020-01-16T00:14:39.816 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000243174 2020-01-16T00:14:39.816 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.816 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.817 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:39.817 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:34.112929+0000", 2020-01-16T00:14:39.817 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012248446 2020-01-16T00:14:39.817 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.817 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:39.817 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.817 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.817 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.818 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.0.13:31 2.0 2:1afbc8ac:::smithi12313934-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 4096~737792] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e50)", 2020-01-16T00:14:39.818 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:44.061109+0000", 2020-01-16T00:14:39.818 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 55.612482894999999, 2020-01-16T00:14:39.818 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012186723, 2020-01-16T00:14:39.818 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:39.818 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:39.818 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:39.818 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.819 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:39.819 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.061109+0000", 2020-01-16T00:14:39.819 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:39.819 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.819 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.819 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:39.819 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.061109+0000", 2020-01-16T00:14:39.820 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2020-01-16T00:14:39.820 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.820 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.820 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:39.820 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.061106+0000", 2020-01-16T00:14:39.820 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012349870000000001 2020-01-16T00:14:39.820 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.820 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.821 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:39.821 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062341+0000", 2020-01-16T00:14:39.821 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0235999999999996e-05 2020-01-16T00:14:39.821 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.821 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.821 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:39.821 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062411+0000", 2020-01-16T00:14:39.822 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7078000000000001e-05 2020-01-16T00:14:39.822 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.822 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.822 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:39.822 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062428+0000", 2020-01-16T00:14:39.822 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2158999999999996e-05 2020-01-16T00:14:39.822 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.822 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.823 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:39.823 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062520+0000", 2020-01-16T00:14:39.823 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011203499999999999 2020-01-16T00:14:39.823 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.823 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.823 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:39.823 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062632+0000", 2020-01-16T00:14:39.824 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017238400000000001 2020-01-16T00:14:39.824 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.824 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.824 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:14:39.824 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.062805+0000", 2020-01-16T00:14:39.824 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010264989 2020-01-16T00:14:39.824 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.825 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.825 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:39.825 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073070+0000", 2020-01-16T00:14:39.825 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0533000000000001e-05 2020-01-16T00:14:39.825 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.825 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.825 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:39.826 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073090+0000", 2020-01-16T00:14:39.826 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2309000000000002e-05 2020-01-16T00:14:39.826 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.826 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.826 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:39.826 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073163+0000", 2020-01-16T00:14:39.826 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3035999999999997e-05 2020-01-16T00:14:39.826 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.827 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.827 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:39.827 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073236+0000", 2020-01-16T00:14:39.827 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0056999999999998e-05 2020-01-16T00:14:39.827 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.827 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.827 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:39.827 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073296+0000", 2020-01-16T00:14:39.828 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012186723 2020-01-16T00:14:39.828 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.828 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:39.828 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.828 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.828 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.828 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(osd.0.13:33 2.0 2:1afbc8ac:::smithi12313934-13 low_tier mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm:head [write 1541120~378368] snapc 0=[] ondisk+write+rwordered+ignore_cache+ignore_overlay+known_if_redirected e50)", 2020-01-16T00:14:39.828 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:44.063236+0000", 2020-01-16T00:14:39.829 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 55.610356230999997, 2020-01-16T00:14:39.829 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015290962999999999, 2020-01-16T00:14:39.829 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:39.829 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:39.829 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:39.829 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.829 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:39.830 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063236+0000", 2020-01-16T00:14:39.830 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:39.830 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.830 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.830 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:39.830 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063236+0000", 2020-01-16T00:14:39.830 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:39.830 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.831 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.831 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:39.831 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063234+0000", 2020-01-16T00:14:39.831 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000500702 2020-01-16T00:14:39.831 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.831 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.831 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:39.831 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063735+0000", 2020-01-16T00:14:39.832 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4734e-05 2020-01-16T00:14:39.832 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.832 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.832 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:39.832 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063750+0000", 2020-01-16T00:14:39.832 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.382e-05 2020-01-16T00:14:39.832 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.833 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.833 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:39.833 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.063764+0000", 2020-01-16T00:14:39.833 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0059030799999999998 2020-01-16T00:14:39.833 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.833 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.833 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:39.834 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.069667+0000", 2020-01-16T00:14:39.834 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3961999999999994e-05 2020-01-16T00:14:39.834 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.834 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.834 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:39.834 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.069741+0000", 2020-01-16T00:14:39.834 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000114931 2020-01-16T00:14:39.835 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.835 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.835 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,1", 2020-01-16T00:14:39.835 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.069856+0000", 2020-01-16T00:14:39.835 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0039948270000000003 2020-01-16T00:14:39.835 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.835 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.836 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:39.836 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.073850+0000", 2020-01-16T00:14:39.836 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0016459140000000001 2020-01-16T00:14:39.836 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.836 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.836 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:39.836 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.075496+0000", 2020-01-16T00:14:39.836 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002957997 2020-01-16T00:14:39.837 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.837 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.837 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:39.837 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.078454+0000", 2020-01-16T00:14:39.837 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9190999999999998e-05 2020-01-16T00:14:39.837 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.837 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.838 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:39.838 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.078493+0000", 2020-01-16T00:14:39.838 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3176999999999998e-05 2020-01-16T00:14:39.838 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.838 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.838 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:39.838 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:44.078527+0000", 2020-01-16T00:14:39.838 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015290962999999999 2020-01-16T00:14:39.839 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.839 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:39.839 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.840 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.841 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.841 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4641 3.d 3:b1521a3e:::smithi12313934-25:head [read 0~3596797,checksum] snapc 0=[] ondisk+read+known_if_redirected e57)", 2020-01-16T00:14:39.841 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:04.086675+0000", 2020-01-16T00:14:39.841 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 35.586917175000004, 2020-01-16T00:14:39.841 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011822397, 2020-01-16T00:14:39.841 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:39.841 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:39.841 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:39.842 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:39.842 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:39.842 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4641 2020-01-16T00:14:39.842 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.842 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:39.842 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.842 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:39.842 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.086675+0000", 2020-01-16T00:14:39.843 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:39.843 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.843 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.843 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:39.843 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.086675+0000", 2020-01-16T00:14:39.843 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:39.843 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.844 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.844 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:39.844 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.086674+0000", 2020-01-16T00:14:39.844 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0809999999999998e-06 2020-01-16T00:14:39.844 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.844 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.844 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:39.844 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.086678+0000", 2020-01-16T00:14:39.845 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4889999999999997e-06 2020-01-16T00:14:39.845 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.845 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.845 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:39.845 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.086682+0000", 2020-01-16T00:14:39.845 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9620000000000002e-06 2020-01-16T00:14:39.845 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.846 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.846 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:39.846 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.086685+0000", 2020-01-16T00:14:39.846 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010215452 2020-01-16T00:14:39.846 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.846 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.846 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:39.846 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.096900+0000", 2020-01-16T00:14:39.847 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00021274299999999999 2020-01-16T00:14:39.847 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.847 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.847 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:39.847 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.097113+0000", 2020-01-16T00:14:39.847 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0013839480000000001 2020-01-16T00:14:39.847 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.848 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.848 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:39.848 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:04.098497+0000", 2020-01-16T00:14:39.848 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011822397 2020-01-16T00:14:39.848 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.848 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:39.849 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.849 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.849 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.849 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4737 3.d 3:b76659c4:::smithi12313934-47:head [read 0~3243850,checksum] snapc 0=[] ondisk+read+known_if_redirected e58)", 2020-01-16T00:14:39.849 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:08.134998+0000", 2020-01-16T00:14:39.849 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 31.538594301, 2020-01-16T00:14:39.849 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011528518, 2020-01-16T00:14:39.849 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:39.850 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:39.850 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:39.850 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:39.850 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:39.850 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4737 2020-01-16T00:14:39.850 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.850 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:39.851 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.851 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:39.851 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:08.134998+0000", 2020-01-16T00:14:39.851 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:39.851 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.851 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.851 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:39.851 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:08.134998+0000", 2020-01-16T00:14:39.852 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2020-01-16T00:14:39.852 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.852 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.852 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:39.852 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:08.134997+0000", 2020-01-16T00:14:39.852 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0459999999999999e-06 2020-01-16T00:14:39.852 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.852 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.853 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:39.853 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:08.135001+0000", 2020-01-16T00:14:39.853 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2469999999999999e-06 2020-01-16T00:14:39.853 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.853 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.853 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:39.853 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:08.135006+0000", 2020-01-16T00:14:39.854 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.065e-06 2020-01-16T00:14:39.854 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.854 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.854 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:39.854 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:08.135010+0000", 2020-01-16T00:14:39.854 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0095733439999999993 2020-01-16T00:14:39.854 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.854 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.855 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:39.855 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:08.144584+0000", 2020-01-16T00:14:39.855 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000272623 2020-01-16T00:14:39.855 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.855 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.855 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:39.855 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:08.144856+0000", 2020-01-16T00:14:39.855 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001669769 2020-01-16T00:14:39.856 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.856 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.856 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:39.856 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:08.146526+0000", 2020-01-16T00:14:39.856 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011528518 2020-01-16T00:14:39.856 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.856 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:39.857 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.857 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.857 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.857 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:5704 3.e 3:7cfd7007:::smithi12313934-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e67)", 2020-01-16T00:14:39.857 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:30.998090+0000", 2020-01-16T00:14:39.857 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 8.6755020139999992, 2020-01-16T00:14:39.857 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011469234999999999, 2020-01-16T00:14:39.857 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:39.858 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:39.858 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:39.858 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:39.858 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:39.858 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 5704 2020-01-16T00:14:39.858 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.858 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:39.859 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.859 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:39.859 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:30.998090+0000", 2020-01-16T00:14:39.859 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:39.859 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.859 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.859 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:39.859 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:30.998090+0000", 2020-01-16T00:14:39.860 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:39.860 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.860 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.860 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:39.860 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:30.998089+0000", 2020-01-16T00:14:39.860 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1510000000000001e-06 2020-01-16T00:14:39.860 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.860 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.861 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:39.861 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:30.998095+0000", 2020-01-16T00:14:39.861 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.9919999999999996e-06 2020-01-16T00:14:39.861 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.861 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.861 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:39.861 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:30.998104+0000", 2020-01-16T00:14:39.862 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3429999999999998e-06 2020-01-16T00:14:39.862 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.862 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.862 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:39.862 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:30.998111+0000", 2020-01-16T00:14:39.862 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0075958309999999999 2020-01-16T00:14:39.862 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.862 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.863 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:39.863 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:31.005707+0000", 2020-01-16T00:14:39.863 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015471300000000001 2020-01-16T00:14:39.863 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.863 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.863 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:39.863 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:31.005862+0000", 2020-01-16T00:14:39.863 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0034355800000000001 2020-01-16T00:14:39.864 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.864 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.864 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:39.864 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:31.009297+0000", 2020-01-16T00:14:39.864 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011861100000000001 2020-01-16T00:14:39.864 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.864 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.865 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:39.865 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:31.009416+0000", 2020-01-16T00:14:39.865 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014325699999999999 2020-01-16T00:14:39.865 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.865 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.865 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:39.865 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:31.009559+0000", 2020-01-16T00:14:39.866 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011469234999999999 2020-01-16T00:14:39.866 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.866 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:39.866 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.866 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.866 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.866 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:5780 3.f 3:f94a5a88:::smithi12313934-22:head [read 0~3949777,checksum] snapc 0=[] ondisk+read+known_if_redirected e68)", 2020-01-16T00:14:39.866 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:32.221847+0000", 2020-01-16T00:14:39.867 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 7.4517445599999999, 2020-01-16T00:14:39.867 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012182213000000001, 2020-01-16T00:14:39.867 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:39.867 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:39.867 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:39.867 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:39.867 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:39.867 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 5780 2020-01-16T00:14:39.868 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.868 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:39.868 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.868 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:39.868 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:32.221847+0000", 2020-01-16T00:14:39.868 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:39.868 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.869 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.869 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:39.869 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:32.221847+0000", 2020-01-16T00:14:39.869 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:39.869 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.869 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.869 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:39.869 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:32.221846+0000", 2020-01-16T00:14:39.870 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5370000000000003e-06 2020-01-16T00:14:39.870 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.870 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.870 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:39.870 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:32.221853+0000", 2020-01-16T00:14:39.870 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3326e-05 2020-01-16T00:14:39.871 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.871 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.871 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:39.871 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:32.221876+0000", 2020-01-16T00:14:39.871 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1012000000000001e-05 2020-01-16T00:14:39.871 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.871 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.872 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:39.872 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:32.221887+0000", 2020-01-16T00:14:39.872 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0078782320000000006 2020-01-16T00:14:39.872 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.872 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.872 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:39.872 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:32.229765+0000", 2020-01-16T00:14:39.872 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00049060300000000005 2020-01-16T00:14:39.873 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.873 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.873 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:39.873 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:32.230256+0000", 2020-01-16T00:14:39.873 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0037737030000000002 2020-01-16T00:14:39.873 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:39.873 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:39.873 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:39.874 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:32.234030+0000", 2020-01-16T00:14:39.874 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012182213000000001 2020-01-16T00:14:39.874 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.874 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:39.874 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.874 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:39.874 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:39.875 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:39.876 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2020-01-16T00:14:39.877 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:39.876+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:39.977 INFO:tasks.ceph.osd.7.smithi145.stderr:2020-01-16T00:14:39.976+0000 7f98524b8700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 11848) UID: 0 2020-01-16T00:14:40.031 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:40.031 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:14:40.031 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2020-01-16T00:14:40.032 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:40.041 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2020-01-16T00:14:40.085 INFO:tasks.ceph.osd.6.smithi145.stderr:2020-01-16T00:14:40.084+0000 7f7cf9e5f700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 11846) UID: 0 2020-01-16T00:14:40.173 INFO:teuthology.orchestra.run.smithi123.stderr:nodeep-scrub is unset 2020-01-16T00:14:40.177 INFO:teuthology.orchestra.run.smithi123.stderr:pool 'unique_pool_0' removed 2020-01-16T00:14:40.178 INFO:tasks.ceph.osd.3.smithi123.stderr:2020-01-16T00:14:40.178+0000 7fd8ba8c2700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 11900) UID: 0 2020-01-16T00:14:40.188 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2020-01-16T00:14:40.199 INFO:tasks.thrashosds:joining thrashosds 2020-01-16T00:14:40.199 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2020-01-16T00:14:40.203 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:40.203 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2020-01-16T00:14:40.203 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2020-01-16T00:14:40.203 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2020-01-16T00:14:40.203 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:40.212 INFO:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2020-01-16T00:14:40.278 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2020-01-16T00:14:40.316 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2020-01-16T00:14:40.316 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2020-01-16T00:14:40.316 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2020-01-16T00:14:40.317 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2020-01-16T00:14:40.317 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.317 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:1657 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e30)", 2020-01-16T00:14:40.317 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:12:52.718495+0000", 2020-01-16T00:14:40.317 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 107.59440949099999, 2020-01-16T00:14:40.317 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014299630000000001, 2020-01-16T00:14:40.317 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:40.318 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:40.318 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:40.318 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:40.318 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:40.318 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1657 2020-01-16T00:14:40.318 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.319 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:40.319 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.319 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:40.319 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718495+0000", 2020-01-16T00:14:40.319 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:40.319 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.320 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.320 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:40.320 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718495+0000", 2020-01-16T00:14:40.320 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:40.320 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.321 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.321 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:40.321 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718494+0000", 2020-01-16T00:14:40.321 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8860000000000003e-06 2020-01-16T00:14:40.321 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.321 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.322 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:40.322 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718499+0000", 2020-01-16T00:14:40.322 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9786000000000002e-05 2020-01-16T00:14:40.322 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.322 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.322 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:40.323 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718539+0000", 2020-01-16T00:14:40.323 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1869999999999999e-05 2020-01-16T00:14:40.323 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.323 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.323 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:40.323 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718561+0000", 2020-01-16T00:14:40.323 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000172206 2020-01-16T00:14:40.324 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.324 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.324 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.324 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718733+0000", 2020-01-16T00:14:40.324 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0662999999999998e-05 2020-01-16T00:14:40.324 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.324 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.324 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:40.325 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.718793+0000", 2020-01-16T00:14:40.325 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.006217958 2020-01-16T00:14:40.325 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.325 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.325 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.325 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.725011+0000", 2020-01-16T00:14:40.325 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1625999999999996e-05 2020-01-16T00:14:40.326 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.326 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.326 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:40.326 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.725073+0000", 2020-01-16T00:14:40.326 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0063923119999999998 2020-01-16T00:14:40.326 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.326 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.326 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.327 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731465+0000", 2020-01-16T00:14:40.327 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2534000000000005e-05 2020-01-16T00:14:40.327 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.327 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.327 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:40.327 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.731558+0000", 2020-01-16T00:14:40.327 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010238809999999999 2020-01-16T00:14:40.327 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.328 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.328 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.328 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732582+0000", 2020-01-16T00:14:40.328 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000100918 2020-01-16T00:14:40.328 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.328 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.328 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:40.328 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732683+0000", 2020-01-16T00:14:40.329 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000112064 2020-01-16T00:14:40.329 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.329 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.329 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:40.329 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:12:52.732795+0000", 2020-01-16T00:14:40.329 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014299630000000001 2020-01-16T00:14:40.329 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.329 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:40.330 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.330 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.330 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.330 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2791 3.a 3:54f2c117:::smithi12313934-41:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e41)", 2020-01-16T00:14:40.330 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:17.213167+0000", 2020-01-16T00:14:40.330 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 83.099737798000007, 2020-01-16T00:14:40.330 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016024037000000001, 2020-01-16T00:14:40.331 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:40.331 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:40.331 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:40.331 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:40.331 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:40.331 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2791 2020-01-16T00:14:40.331 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.331 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:40.332 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.332 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:40.332 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213167+0000", 2020-01-16T00:14:40.332 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:40.332 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.332 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.332 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:40.332 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213167+0000", 2020-01-16T00:14:40.333 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:40.333 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.333 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.333 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:40.333 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213166+0000", 2020-01-16T00:14:40.333 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6030000000000004e-06 2020-01-16T00:14:40.333 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.333 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.334 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:40.334 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213172+0000", 2020-01-16T00:14:40.334 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0929999999999997e-06 2020-01-16T00:14:40.334 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.334 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.334 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:40.334 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213179+0000", 2020-01-16T00:14:40.335 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4299999999999997e-06 2020-01-16T00:14:40.335 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.335 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.335 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:40.335 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213184+0000", 2020-01-16T00:14:40.336 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012106695000000001 2020-01-16T00:14:40.336 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.336 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.336 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.337 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.225291+0000", 2020-01-16T00:14:40.337 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020863699999999999 2020-01-16T00:14:40.337 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.337 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.338 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:40.338 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.225499+0000", 2020-01-16T00:14:40.338 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0036913050000000002 2020-01-16T00:14:40.338 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.338 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.339 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:40.339 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.229191+0000", 2020-01-16T00:14:40.339 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016024037000000001 2020-01-16T00:14:40.339 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.339 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:40.340 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.340 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.340 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.340 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2792 3.a 3:54f2c117:::smithi12313934-41:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e41)", 2020-01-16T00:14:40.340 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:17.213190+0000", 2020-01-16T00:14:40.341 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 83.099714324000004, 2020-01-16T00:14:40.341 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017165949, 2020-01-16T00:14:40.341 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:40.341 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:40.342 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:40.342 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:40.342 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:40.342 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2792 2020-01-16T00:14:40.342 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.343 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:40.343 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.343 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:40.343 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213190+0000", 2020-01-16T00:14:40.343 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:40.344 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.344 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.344 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:40.344 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213190+0000", 2020-01-16T00:14:40.344 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:40.345 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.345 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.345 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:40.345 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213189+0000", 2020-01-16T00:14:40.345 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2699999999999998e-06 2020-01-16T00:14:40.346 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.346 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.346 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:40.346 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213194+0000", 2020-01-16T00:14:40.346 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8600000000000004e-06 2020-01-16T00:14:40.347 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.347 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.347 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:40.347 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213201+0000", 2020-01-16T00:14:40.347 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8500000000000002e-06 2020-01-16T00:14:40.347 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.347 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.348 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:40.348 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213205+0000", 2020-01-16T00:14:40.348 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016028924999999999 2020-01-16T00:14:40.348 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.348 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.348 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.348 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.229234+0000", 2020-01-16T00:14:40.348 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019773999999999999 2020-01-16T00:14:40.349 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.349 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.349 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:40.349 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.229432+0000", 2020-01-16T00:14:40.349 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00092414400000000001 2020-01-16T00:14:40.349 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.349 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.350 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:40.350 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.230356+0000", 2020-01-16T00:14:40.350 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017165949 2020-01-16T00:14:40.350 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.350 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:40.350 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.350 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.350 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.351 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:2793 3.a 3:54f2c117:::smithi12313934-41:head [read 0~3210170,checksum] snapc 0=[] ondisk+read+known_if_redirected e41)", 2020-01-16T00:14:40.351 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:17.213211+0000", 2020-01-16T00:14:40.351 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 83.099693778000002, 2020-01-16T00:14:40.351 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019069413, 2020-01-16T00:14:40.352 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:40.352 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:40.352 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:40.352 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:40.352 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:40.352 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2793 2020-01-16T00:14:40.353 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.353 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:40.353 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.353 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:40.353 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213211+0000", 2020-01-16T00:14:40.353 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:40.353 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.353 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.354 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:40.354 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213211+0000", 2020-01-16T00:14:40.354 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:40.354 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.354 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.354 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:40.354 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213210+0000", 2020-01-16T00:14:40.355 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.66e-06 2020-01-16T00:14:40.355 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.355 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.355 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:40.355 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213216+0000", 2020-01-16T00:14:40.355 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.9469999999999997e-06 2020-01-16T00:14:40.355 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.356 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.356 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:40.356 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213223+0000", 2020-01-16T00:14:40.356 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9259999999999999e-06 2020-01-16T00:14:40.356 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.356 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.356 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:40.357 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.213228+0000", 2020-01-16T00:14:40.357 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017171046999999998 2020-01-16T00:14:40.357 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.357 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.357 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.357 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.230399+0000", 2020-01-16T00:14:40.357 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000189932 2020-01-16T00:14:40.358 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.358 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.358 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:40.358 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.230589+0000", 2020-01-16T00:14:40.358 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0016916539999999999 2020-01-16T00:14:40.358 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.358 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.358 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:40.359 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:17.232280+0000", 2020-01-16T00:14:40.359 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019069413 2020-01-16T00:14:40.359 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.359 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:40.359 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.359 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.359 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.359 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3120 3.a 3:5069665e:::smithi12313934-2:head [write 1845146~524707] snapc 0=[] ondisk+write+known_if_redirected e44)", 2020-01-16T00:14:40.360 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.262852+0000", 2020-01-16T00:14:40.360 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 75.050052594999997, 2020-01-16T00:14:40.360 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014991175000000001, 2020-01-16T00:14:40.360 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:40.360 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:40.360 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:40.360 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:40.360 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:40.361 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3120 2020-01-16T00:14:40.361 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.361 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:40.361 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.361 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:40.362 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.262852+0000", 2020-01-16T00:14:40.362 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:40.363 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.363 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.363 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:40.363 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.262852+0000", 2020-01-16T00:14:40.363 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:40.363 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.364 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.364 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:40.364 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.262849+0000", 2020-01-16T00:14:40.364 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00058469900000000005 2020-01-16T00:14:40.364 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.364 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.365 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:40.365 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263434+0000", 2020-01-16T00:14:40.365 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.6650000000000001e-05 2020-01-16T00:14:40.365 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.365 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.365 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:40.365 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263461+0000", 2020-01-16T00:14:40.365 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3478e-05 2020-01-16T00:14:40.366 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.366 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.366 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:40.366 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263474+0000", 2020-01-16T00:14:40.366 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7824000000000002e-05 2020-01-16T00:14:40.366 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.366 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.366 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.367 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263522+0000", 2020-01-16T00:14:40.367 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001365464 2020-01-16T00:14:40.367 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.367 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.367 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:40.367 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.264888+0000", 2020-01-16T00:14:40.367 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0081368250000000003 2020-01-16T00:14:40.367 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.368 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.368 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.368 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.273024+0000", 2020-01-16T00:14:40.368 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6130000000000006e-05 2020-01-16T00:14:40.368 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.368 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.368 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:40.368 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.273090+0000", 2020-01-16T00:14:40.369 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000207507 2020-01-16T00:14:40.369 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.369 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.369 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:40.369 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.273298+0000", 2020-01-16T00:14:40.369 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0035649980000000002 2020-01-16T00:14:40.369 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.370 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.370 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:40.370 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276863+0000", 2020-01-16T00:14:40.370 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.1459999999999995e-05 2020-01-16T00:14:40.370 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.371 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.371 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:40.371 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276954+0000", 2020-01-16T00:14:40.371 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00076363899999999998 2020-01-16T00:14:40.371 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.371 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.371 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:40.371 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277718+0000", 2020-01-16T00:14:40.372 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0154999999999995e-05 2020-01-16T00:14:40.372 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.372 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.372 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:40.372 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277788+0000", 2020-01-16T00:14:40.372 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4914000000000003e-05 2020-01-16T00:14:40.372 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.372 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.373 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:40.373 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277843+0000", 2020-01-16T00:14:40.373 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014991175000000001 2020-01-16T00:14:40.373 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.373 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:40.373 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.373 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.373 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.374 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3121 3.a 3:5069665e:::smithi12313934-2:head [setxattr _header (55) in=7b,truncate 2369853] snapc 0=[] ondisk+write+known_if_redirected e44)", 2020-01-16T00:14:40.374 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.263494+0000", 2020-01-16T00:14:40.374 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 75.049410307000002, 2020-01-16T00:14:40.374 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014559274000000001, 2020-01-16T00:14:40.374 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:40.374 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:40.374 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:40.375 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:40.375 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:40.375 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3121 2020-01-16T00:14:40.375 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.375 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:40.376 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.376 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:40.376 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263494+0000", 2020-01-16T00:14:40.376 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:40.376 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.376 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.376 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:40.376 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263494+0000", 2020-01-16T00:14:40.377 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:40.377 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.377 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.377 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:40.377 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263493+0000", 2020-01-16T00:14:40.377 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3359999999999995e-06 2020-01-16T00:14:40.377 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.377 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.378 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:40.378 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263501+0000", 2020-01-16T00:14:40.378 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4100000000000002e-06 2020-01-16T00:14:40.378 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.378 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.378 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:40.378 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263509+0000", 2020-01-16T00:14:40.378 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3839999999999996e-06 2020-01-16T00:14:40.379 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.379 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.379 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:40.379 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263514+0000", 2020-01-16T00:14:40.379 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001430295 2020-01-16T00:14:40.379 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.379 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.379 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.380 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.264944+0000", 2020-01-16T00:14:40.380 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9710999999999999e-05 2020-01-16T00:14:40.380 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.380 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.380 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:40.380 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.264994+0000", 2020-01-16T00:14:40.380 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010036188999999999 2020-01-16T00:14:40.381 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.381 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.381 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.381 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.275030+0000", 2020-01-16T00:14:40.381 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1743000000000007e-05 2020-01-16T00:14:40.381 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.381 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.381 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:40.382 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.275112+0000", 2020-01-16T00:14:40.382 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013036100000000001 2020-01-16T00:14:40.382 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.382 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.382 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:40.382 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.275242+0000", 2020-01-16T00:14:40.382 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0016795549999999999 2020-01-16T00:14:40.382 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.383 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.383 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:40.383 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.276922+0000", 2020-01-16T00:14:40.383 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018479400000000001 2020-01-16T00:14:40.383 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.383 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.383 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:40.383 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277107+0000", 2020-01-16T00:14:40.384 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00083585300000000001 2020-01-16T00:14:40.384 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.384 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.384 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:40.384 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277943+0000", 2020-01-16T00:14:40.384 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1719999999999999e-05 2020-01-16T00:14:40.384 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.384 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.385 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:40.385 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.277994+0000", 2020-01-16T00:14:40.385 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9212999999999997e-05 2020-01-16T00:14:40.385 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.385 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.385 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:40.385 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.278054+0000", 2020-01-16T00:14:40.385 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014559274000000001 2020-01-16T00:14:40.386 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.386 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:40.386 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.386 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.386 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.386 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3122 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e44)", 2020-01-16T00:14:40.386 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:25.263525+0000", 2020-01-16T00:14:40.386 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 75.049379118999994, 2020-01-16T00:14:40.387 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014731833999999999, 2020-01-16T00:14:40.387 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:40.387 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:40.387 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:40.387 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:40.387 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:40.387 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3122 2020-01-16T00:14:40.387 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.388 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:40.388 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.388 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:40.388 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263525+0000", 2020-01-16T00:14:40.388 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:40.388 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.388 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.388 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:40.389 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263525+0000", 2020-01-16T00:14:40.389 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:40.389 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.389 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.389 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:40.389 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263525+0000", 2020-01-16T00:14:40.389 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5000000000000001e-06 2020-01-16T00:14:40.390 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.390 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.390 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:40.390 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263529+0000", 2020-01-16T00:14:40.390 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4669999999999998e-06 2020-01-16T00:14:40.390 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.390 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.390 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:40.391 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263537+0000", 2020-01-16T00:14:40.391 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0819999999999998e-06 2020-01-16T00:14:40.391 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.391 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.391 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:40.391 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.263542+0000", 2020-01-16T00:14:40.392 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0015024960000000001 2020-01-16T00:14:40.392 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.392 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.392 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.392 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.265044+0000", 2020-01-16T00:14:40.392 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6650999999999994e-05 2020-01-16T00:14:40.392 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.392 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.393 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:40.393 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.265131+0000", 2020-01-16T00:14:40.393 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010495310000000001 2020-01-16T00:14:40.393 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.393 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.393 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.393 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.275626+0000", 2020-01-16T00:14:40.393 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9536000000000004e-05 2020-01-16T00:14:40.394 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.394 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.394 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:40.394 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.275726+0000", 2020-01-16T00:14:40.394 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023719489999999999 2020-01-16T00:14:40.394 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.394 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.395 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.395 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.278098+0000", 2020-01-16T00:14:40.395 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3718000000000001e-05 2020-01-16T00:14:40.395 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.395 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.395 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:40.395 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.278171+0000", 2020-01-16T00:14:40.395 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5859999999999994e-05 2020-01-16T00:14:40.395 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.396 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.396 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:40.396 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:25.278257+0000", 2020-01-16T00:14:40.396 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014731833999999999 2020-01-16T00:14:40.396 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.396 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:40.396 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.396 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.397 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.397 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3627 3.a 3:5069665e:::smithi12313934-2:head [write 1749580~611149] snapc 0=[] ondisk+write+known_if_redirected e48)", 2020-01-16T00:14:40.397 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:39.462779+0000", 2020-01-16T00:14:40.397 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 60.850125306999999, 2020-01-16T00:14:40.397 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020946985000000001, 2020-01-16T00:14:40.397 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:40.397 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:40.398 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:40.398 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:40.398 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:40.398 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3627 2020-01-16T00:14:40.398 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.398 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:40.398 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.398 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:40.399 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.462779+0000", 2020-01-16T00:14:40.399 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:40.399 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.399 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.399 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:40.399 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.462779+0000", 2020-01-16T00:14:40.399 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:40.399 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.400 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.400 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:40.400 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.462777+0000", 2020-01-16T00:14:40.400 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00064136 2020-01-16T00:14:40.400 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.400 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.400 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:40.400 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.463418+0000", 2020-01-16T00:14:40.401 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4607e-05 2020-01-16T00:14:40.401 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.401 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.401 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:40.401 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.463443+0000", 2020-01-16T00:14:40.401 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0346999999999999e-05 2020-01-16T00:14:40.401 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.402 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.402 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:40.402 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.463453+0000", 2020-01-16T00:14:40.402 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000100988 2020-01-16T00:14:40.402 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.402 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.402 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.402 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.463554+0000", 2020-01-16T00:14:40.403 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023059280000000001 2020-01-16T00:14:40.403 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.403 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.403 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:40.403 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.465860+0000", 2020-01-16T00:14:40.403 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011316961 2020-01-16T00:14:40.403 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.403 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.404 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.404 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.477177+0000", 2020-01-16T00:14:40.404 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000132402 2020-01-16T00:14:40.404 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.404 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.404 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:40.404 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.477309+0000", 2020-01-16T00:14:40.404 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000223606 2020-01-16T00:14:40.405 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.405 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.405 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:40.405 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.477533+0000", 2020-01-16T00:14:40.405 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0057105209999999997 2020-01-16T00:14:40.405 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.405 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.406 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:40.406 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483244+0000", 2020-01-16T00:14:40.406 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000193031 2020-01-16T00:14:40.406 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.406 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.406 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:40.406 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483437+0000", 2020-01-16T00:14:40.406 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0902999999999995e-05 2020-01-16T00:14:40.407 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.407 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.407 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:40.407 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483507+0000", 2020-01-16T00:14:40.407 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000129412 2020-01-16T00:14:40.407 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.407 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.407 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:40.408 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483637+0000", 2020-01-16T00:14:40.408 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.9345999999999995e-05 2020-01-16T00:14:40.408 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.408 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.408 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:40.408 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483726+0000", 2020-01-16T00:14:40.408 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020946985000000001 2020-01-16T00:14:40.409 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.409 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:40.409 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.409 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.409 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.409 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3628 3.a 3:5069665e:::smithi12313934-2:head [write 2853383~452450] snapc 0=[] ondisk+write+known_if_redirected e48)", 2020-01-16T00:14:40.409 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:39.468178+0000", 2020-01-16T00:14:40.410 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 60.844726598000001, 2020-01-16T00:14:40.410 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015705872999999999, 2020-01-16T00:14:40.410 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:40.410 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:40.410 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:40.410 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:40.410 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:40.410 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3628 2020-01-16T00:14:40.411 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.411 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:40.411 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.411 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:40.411 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468178+0000", 2020-01-16T00:14:40.411 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:40.411 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.411 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.412 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:40.412 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468178+0000", 2020-01-16T00:14:40.412 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:40.412 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.412 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.412 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:40.412 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468175+0000", 2020-01-16T00:14:40.412 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00065948000000000005 2020-01-16T00:14:40.413 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.413 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.413 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:40.413 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468834+0000", 2020-01-16T00:14:40.413 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3557000000000002e-05 2020-01-16T00:14:40.413 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.413 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.413 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:40.414 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468858+0000", 2020-01-16T00:14:40.414 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2931e-05 2020-01-16T00:14:40.414 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.414 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.414 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:40.414 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468871+0000", 2020-01-16T00:14:40.414 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000123113 2020-01-16T00:14:40.414 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.415 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.415 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.415 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468994+0000", 2020-01-16T00:14:40.415 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017615999999999999 2020-01-16T00:14:40.415 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.415 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.415 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:40.415 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.469170+0000", 2020-01-16T00:14:40.416 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010336908000000001 2020-01-16T00:14:40.416 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.416 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.416 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.416 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.479507+0000", 2020-01-16T00:14:40.416 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015856400000000001 2020-01-16T00:14:40.416 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.416 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.417 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:40.417 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.479665+0000", 2020-01-16T00:14:40.417 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030344600000000002 2020-01-16T00:14:40.417 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.417 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.417 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:40.417 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.479969+0000", 2020-01-16T00:14:40.417 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0037935389999999999 2020-01-16T00:14:40.418 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.418 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.418 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:40.418 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483762+0000", 2020-01-16T00:14:40.418 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3170999999999997e-05 2020-01-16T00:14:40.418 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.418 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.419 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:40.419 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483806+0000", 2020-01-16T00:14:40.419 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3821000000000001e-05 2020-01-16T00:14:40.419 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.419 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.419 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:40.419 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483829+0000", 2020-01-16T00:14:40.420 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0076000000000001e-05 2020-01-16T00:14:40.420 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.420 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.420 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:40.420 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483859+0000", 2020-01-16T00:14:40.420 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.438e-05 2020-01-16T00:14:40.421 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.421 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.421 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:40.421 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483884+0000", 2020-01-16T00:14:40.421 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015705872999999999 2020-01-16T00:14:40.421 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.421 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:40.421 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.422 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.422 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.422 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3629 3.a 3:5069665e:::smithi12313934-2:head [setxattr _header (55) in=7b,truncate 3529536] snapc 0=[] ondisk+write+known_if_redirected e48)", 2020-01-16T00:14:40.422 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:39.468888+0000", 2020-01-16T00:14:40.422 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 60.844016977000003, 2020-01-16T00:14:40.422 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015108511, 2020-01-16T00:14:40.422 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:40.423 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:40.423 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:40.423 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:40.423 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:40.423 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3629 2020-01-16T00:14:40.423 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.423 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:40.423 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.424 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:40.424 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468888+0000", 2020-01-16T00:14:40.424 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:40.424 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.424 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.424 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:40.424 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468888+0000", 2020-01-16T00:14:40.425 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:40.425 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.425 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.425 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:40.425 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468886+0000", 2020-01-16T00:14:40.425 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5599999999999996e-06 2020-01-16T00:14:40.425 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.425 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.426 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:40.426 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468894+0000", 2020-01-16T00:14:40.426 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5279999999999998e-06 2020-01-16T00:14:40.426 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.426 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.426 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:40.426 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468901+0000", 2020-01-16T00:14:40.427 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2290000000000002e-06 2020-01-16T00:14:40.427 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.427 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.427 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:40.427 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468907+0000", 2020-01-16T00:14:40.427 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030197500000000001 2020-01-16T00:14:40.427 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.427 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.428 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.428 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.469209+0000", 2020-01-16T00:14:40.428 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6659999999999997e-05 2020-01-16T00:14:40.428 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.428 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.428 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:40.428 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.469255+0000", 2020-01-16T00:14:40.428 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013117904999999999 2020-01-16T00:14:40.429 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.429 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.429 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.429 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.482373+0000", 2020-01-16T00:14:40.429 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1260000000000006e-05 2020-01-16T00:14:40.429 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.429 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.429 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:40.430 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.482434+0000", 2020-01-16T00:14:40.430 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014433 2020-01-16T00:14:40.430 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.430 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.430 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:40.430 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.482579+0000", 2020-01-16T00:14:40.430 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012198160000000001 2020-01-16T00:14:40.430 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.431 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.431 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:40.431 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483799+0000", 2020-01-16T00:14:40.431 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.188e-05 2020-01-16T00:14:40.431 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.431 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.431 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:40.431 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483810+0000", 2020-01-16T00:14:40.432 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000114863 2020-01-16T00:14:40.432 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.432 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.432 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:40.432 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483925+0000", 2020-01-16T00:14:40.432 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0861000000000001e-05 2020-01-16T00:14:40.432 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.432 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.433 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:40.433 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483956+0000", 2020-01-16T00:14:40.433 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9894999999999997e-05 2020-01-16T00:14:40.433 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.433 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.433 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:40.433 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483996+0000", 2020-01-16T00:14:40.433 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015108511 2020-01-16T00:14:40.434 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.434 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:40.434 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.434 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.434 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.434 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:3630 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e48)", 2020-01-16T00:14:40.434 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:39.468967+0000", 2020-01-16T00:14:40.434 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 60.843937842999999, 2020-01-16T00:14:40.435 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015160579, 2020-01-16T00:14:40.435 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:40.435 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:40.435 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:40.435 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:40.435 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:40.435 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3630 2020-01-16T00:14:40.435 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.436 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:40.436 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.436 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:40.436 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468967+0000", 2020-01-16T00:14:40.436 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:40.436 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.436 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.436 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:40.437 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468967+0000", 2020-01-16T00:14:40.437 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:40.437 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.437 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.437 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:40.437 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468966+0000", 2020-01-16T00:14:40.437 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3430000000000003e-06 2020-01-16T00:14:40.438 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.438 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.438 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:40.438 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468971+0000", 2020-01-16T00:14:40.438 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6577000000000001e-05 2020-01-16T00:14:40.438 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.438 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.439 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:40.439 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468988+0000", 2020-01-16T00:14:40.439 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.378e-06 2020-01-16T00:14:40.439 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.439 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.439 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:40.439 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.468993+0000", 2020-01-16T00:14:40.439 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000320601 2020-01-16T00:14:40.440 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.440 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.440 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.440 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.469314+0000", 2020-01-16T00:14:40.440 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013833699999999999 2020-01-16T00:14:40.440 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.440 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.440 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:40.441 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.469452+0000", 2020-01-16T00:14:40.441 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013447838 2020-01-16T00:14:40.441 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.441 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.441 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.441 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.482900+0000", 2020-01-16T00:14:40.441 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028006899999999998 2020-01-16T00:14:40.441 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.442 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.442 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:40.442 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.483180+0000", 2020-01-16T00:14:40.442 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000833594 2020-01-16T00:14:40.442 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.442 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.442 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.442 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.484013+0000", 2020-01-16T00:14:40.443 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2601e-05 2020-01-16T00:14:40.443 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.443 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.443 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:40.443 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.484066+0000", 2020-01-16T00:14:40.443 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1303999999999994e-05 2020-01-16T00:14:40.443 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.443 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.444 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:40.444 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:39.484127+0000", 2020-01-16T00:14:40.444 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015160579 2020-01-16T00:14:40.444 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.444 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:40.444 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.444 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.444 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.445 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4047 3.a 3:5069665e:::smithi12313934-2:head [write 1935828~635840] snapc 0=[] ondisk+write+known_if_redirected e52)", 2020-01-16T00:14:40.445 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:13:50.302087+0000", 2020-01-16T00:14:40.445 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 50.010817572000001, 2020-01-16T00:14:40.445 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014518982, 2020-01-16T00:14:40.445 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:40.445 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:40.445 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:40.446 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:40.446 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:40.446 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4047 2020-01-16T00:14:40.446 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.446 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:40.446 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.446 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:40.446 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302087+0000", 2020-01-16T00:14:40.447 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:40.447 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.447 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.447 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:40.447 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302087+0000", 2020-01-16T00:14:40.447 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:40.447 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.447 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.448 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:40.448 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302085+0000", 2020-01-16T00:14:40.448 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00057147600000000002 2020-01-16T00:14:40.448 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.448 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.448 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:40.448 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302657+0000", 2020-01-16T00:14:40.448 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6427999999999999e-05 2020-01-16T00:14:40.449 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.449 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.449 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:40.449 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302673+0000", 2020-01-16T00:14:40.449 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.127e-06 2020-01-16T00:14:40.449 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.449 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.449 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:40.450 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302680+0000", 2020-01-16T00:14:40.450 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6378000000000006e-05 2020-01-16T00:14:40.450 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.450 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.450 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.450 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.302756+0000", 2020-01-16T00:14:40.450 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00039898000000000001 2020-01-16T00:14:40.451 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.451 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.451 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:40.451 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.303155+0000", 2020-01-16T00:14:40.453 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0079666870000000001 2020-01-16T00:14:40.453 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.453 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.453 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.453 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.311122+0000", 2020-01-16T00:14:40.453 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019101400000000001 2020-01-16T00:14:40.453 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.454 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.454 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:40.454 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.311313+0000", 2020-01-16T00:14:40.454 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033332499999999998 2020-01-16T00:14:40.454 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.454 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.454 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:40.454 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.311646+0000", 2020-01-16T00:14:40.455 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0029841440000000002 2020-01-16T00:14:40.455 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.455 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.455 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:40.455 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.314630+0000", 2020-01-16T00:14:40.455 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00027996800000000001 2020-01-16T00:14:40.455 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.455 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.456 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:40.456 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.314910+0000", 2020-01-16T00:14:40.456 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0015637159999999999 2020-01-16T00:14:40.456 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.456 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.456 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:40.456 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.316474+0000", 2020-01-16T00:14:40.456 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5986000000000001e-05 2020-01-16T00:14:40.457 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.457 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.457 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:40.457 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.316550+0000", 2020-01-16T00:14:40.457 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6258000000000002e-05 2020-01-16T00:14:40.457 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.457 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.457 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:40.458 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:13:50.316606+0000", 2020-01-16T00:14:40.458 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014518982 2020-01-16T00:14:40.458 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.458 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:40.458 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.458 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.458 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.459 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4600 3.a 3:5069665e:::smithi12313934-2:head [write 1791765~569861] snapc 0=[] ondisk+write+known_if_redirected e56)", 2020-01-16T00:14:40.459 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:03.059744+0000", 2020-01-16T00:14:40.459 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 37.253160354999999, 2020-01-16T00:14:40.459 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017075275000000001, 2020-01-16T00:14:40.459 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:40.459 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:40.459 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:40.459 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:40.460 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:40.460 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4600 2020-01-16T00:14:40.460 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.460 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:40.460 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.460 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:40.460 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.059744+0000", 2020-01-16T00:14:40.460 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:40.461 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.461 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.461 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:40.461 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.059744+0000", 2020-01-16T00:14:40.461 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999843 2020-01-16T00:14:40.461 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.461 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.461 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:40.462 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.059728+0000", 2020-01-16T00:14:40.462 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00054167699999999996 2020-01-16T00:14:40.462 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.462 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.462 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:40.462 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.060270+0000", 2020-01-16T00:14:40.462 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6693999999999998e-05 2020-01-16T00:14:40.463 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.463 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.463 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:40.463 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.060307+0000", 2020-01-16T00:14:40.463 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8753000000000002e-05 2020-01-16T00:14:40.463 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.463 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.463 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:40.464 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.060326+0000", 2020-01-16T00:14:40.464 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000114177 2020-01-16T00:14:40.464 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.464 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.464 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.464 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.060440+0000", 2020-01-16T00:14:40.464 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001642382 2020-01-16T00:14:40.464 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.465 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.465 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:40.465 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.062082+0000", 2020-01-16T00:14:40.465 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0093492149999999993 2020-01-16T00:14:40.465 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.465 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.465 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.466 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.071431+0000", 2020-01-16T00:14:40.466 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013135600000000001 2020-01-16T00:14:40.466 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.466 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.466 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:40.466 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.071563+0000", 2020-01-16T00:14:40.466 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00032728299999999999 2020-01-16T00:14:40.466 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.467 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.467 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:40.467 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.071890+0000", 2020-01-16T00:14:40.467 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0043339759999999998 2020-01-16T00:14:40.467 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.467 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.467 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:40.467 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.076224+0000", 2020-01-16T00:14:40.468 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000343462 2020-01-16T00:14:40.468 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.468 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.468 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:40.468 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.076567+0000", 2020-01-16T00:14:40.468 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1668000000000006e-05 2020-01-16T00:14:40.468 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.468 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.469 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:40.469 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.076639+0000", 2020-01-16T00:14:40.469 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8572000000000004e-05 2020-01-16T00:14:40.469 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.469 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.469 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:40.469 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.076738+0000", 2020-01-16T00:14:40.469 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1816000000000004e-05 2020-01-16T00:14:40.470 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.470 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.470 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:40.470 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:03.076819+0000", 2020-01-16T00:14:40.470 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017075275000000001 2020-01-16T00:14:40.470 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.470 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:40.470 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.471 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.471 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.471 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4934 3.a 3:5069665e:::smithi12313934-2:head [write 1900783~425211] snapc 0=[] ondisk+write+known_if_redirected e60)", 2020-01-16T00:14:40.471 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:13.356857+0000", 2020-01-16T00:14:40.471 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 26.956047076000001, 2020-01-16T00:14:40.471 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020085438000000001, 2020-01-16T00:14:40.471 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:40.471 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:40.472 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:40.472 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:40.472 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:40.472 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4934 2020-01-16T00:14:40.472 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.472 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:40.472 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.472 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:40.473 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.356857+0000", 2020-01-16T00:14:40.473 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:40.473 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.473 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.473 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:40.473 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.356857+0000", 2020-01-16T00:14:40.473 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2020-01-16T00:14:40.473 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.474 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.474 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:40.474 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.356854+0000", 2020-01-16T00:14:40.474 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000450741 2020-01-16T00:14:40.474 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.474 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.474 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:40.474 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357305+0000", 2020-01-16T00:14:40.475 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1829999999999998e-05 2020-01-16T00:14:40.475 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.475 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.475 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:40.475 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357336+0000", 2020-01-16T00:14:40.475 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.305e-05 2020-01-16T00:14:40.475 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.476 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.476 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:40.476 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357349+0000", 2020-01-16T00:14:40.476 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000131087 2020-01-16T00:14:40.476 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.476 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.477 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.477 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357480+0000", 2020-01-16T00:14:40.477 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0031475409999999998 2020-01-16T00:14:40.477 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.477 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.477 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:40.477 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.360628+0000", 2020-01-16T00:14:40.477 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011820447 2020-01-16T00:14:40.478 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.478 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.478 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.478 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.372448+0000", 2020-01-16T00:14:40.478 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000154093 2020-01-16T00:14:40.478 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.478 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.478 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:40.479 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.372603+0000", 2020-01-16T00:14:40.479 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00035365400000000001 2020-01-16T00:14:40.479 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.479 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.479 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:40.479 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.372956+0000", 2020-01-16T00:14:40.479 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0035908590000000001 2020-01-16T00:14:40.479 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.480 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.480 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:40.480 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.376547+0000", 2020-01-16T00:14:40.480 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3077e-05 2020-01-16T00:14:40.480 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.480 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.480 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:40.480 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.376560+0000", 2020-01-16T00:14:40.481 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024608200000000001 2020-01-16T00:14:40.481 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.481 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.481 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:40.481 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.376806+0000", 2020-01-16T00:14:40.481 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3510999999999995e-05 2020-01-16T00:14:40.481 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.481 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.482 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:40.482 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.376880+0000", 2020-01-16T00:14:40.482 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3170000000000007e-05 2020-01-16T00:14:40.482 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.482 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.482 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:40.482 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.376943+0000", 2020-01-16T00:14:40.483 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020085438000000001 2020-01-16T00:14:40.483 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.483 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:40.483 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.483 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.483 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.483 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4935 3.a 3:5069665e:::smithi12313934-2:head [setxattr _header (55) in=7b,truncate 2689252] snapc 0=[] ondisk+write+known_if_redirected e60)", 2020-01-16T00:14:40.483 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:13.357371+0000", 2020-01-16T00:14:40.484 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 26.955533468999999, 2020-01-16T00:14:40.484 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021339295000000001, 2020-01-16T00:14:40.484 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:40.484 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:40.484 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:40.484 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:40.484 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:40.484 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4935 2020-01-16T00:14:40.485 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.485 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:40.485 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.485 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:40.485 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357371+0000", 2020-01-16T00:14:40.485 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:40.485 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.486 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.486 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:40.486 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357371+0000", 2020-01-16T00:14:40.486 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:40.486 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.486 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.486 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:40.486 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357369+0000", 2020-01-16T00:14:40.487 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2839999999999995e-06 2020-01-16T00:14:40.487 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.487 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.487 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:40.487 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357379+0000", 2020-01-16T00:14:40.487 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4363e-05 2020-01-16T00:14:40.487 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.487 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.488 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:40.488 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357403+0000", 2020-01-16T00:14:40.488 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7800000000000003e-06 2020-01-16T00:14:40.488 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.488 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.488 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:40.488 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357410+0000", 2020-01-16T00:14:40.488 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003365771 2020-01-16T00:14:40.489 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.489 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.489 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.489 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.360776+0000", 2020-01-16T00:14:40.489 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00021243299999999999 2020-01-16T00:14:40.489 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.489 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.489 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:40.490 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.360988+0000", 2020-01-16T00:14:40.490 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014015972 2020-01-16T00:14:40.490 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.490 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.490 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.490 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.375004+0000", 2020-01-16T00:14:40.490 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000143306 2020-01-16T00:14:40.490 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.491 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.491 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:40.491 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.375147+0000", 2020-01-16T00:14:40.491 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019620299999999999 2020-01-16T00:14:40.491 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.491 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.491 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:40.491 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.375344+0000", 2020-01-16T00:14:40.492 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001860483 2020-01-16T00:14:40.492 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.492 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.492 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:40.492 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.377204+0000", 2020-01-16T00:14:40.492 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0013554870000000001 2020-01-16T00:14:40.493 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.493 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.493 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:40.493 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.378560+0000", 2020-01-16T00:14:40.493 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8186000000000002e-05 2020-01-16T00:14:40.493 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.493 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.493 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:40.494 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.378608+0000", 2020-01-16T00:14:40.494 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8071000000000001e-05 2020-01-16T00:14:40.494 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.494 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.494 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:40.494 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.378656+0000", 2020-01-16T00:14:40.494 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4595999999999997e-05 2020-01-16T00:14:40.494 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.495 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.495 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:40.495 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.378710+0000", 2020-01-16T00:14:40.495 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021339295000000001 2020-01-16T00:14:40.495 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.495 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:40.495 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.495 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.496 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.496 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:4936 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e60)", 2020-01-16T00:14:40.496 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:13.357465+0000", 2020-01-16T00:14:40.496 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 26.955439698999999, 2020-01-16T00:14:40.496 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021436350999999999, 2020-01-16T00:14:40.496 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:40.496 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:40.496 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:40.497 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:40.497 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:40.497 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4936 2020-01-16T00:14:40.497 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.497 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:40.497 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.497 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:40.497 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357465+0000", 2020-01-16T00:14:40.498 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:40.498 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.498 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.498 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:40.498 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357465+0000", 2020-01-16T00:14:40.498 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:40.498 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.498 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.499 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:40.499 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357464+0000", 2020-01-16T00:14:40.499 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.704e-06 2020-01-16T00:14:40.499 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.499 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.499 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:40.499 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357472+0000", 2020-01-16T00:14:40.499 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6500000000000001e-05 2020-01-16T00:14:40.500 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.500 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.500 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:40.500 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357489+0000", 2020-01-16T00:14:40.500 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.026e-05 2020-01-16T00:14:40.500 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.500 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.501 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:40.501 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.357499+0000", 2020-01-16T00:14:40.501 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0042232679999999996 2020-01-16T00:14:40.501 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.501 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.501 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.501 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.361722+0000", 2020-01-16T00:14:40.501 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015945999999999999 2020-01-16T00:14:40.502 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.502 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.502 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:40.502 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.361882+0000", 2020-01-16T00:14:40.502 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014213089999999999 2020-01-16T00:14:40.502 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.502 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.502 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.503 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.376095+0000", 2020-01-16T00:14:40.503 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013706999999999999 2020-01-16T00:14:40.503 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.503 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.503 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:40.503 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.376232+0000", 2020-01-16T00:14:40.503 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002519566 2020-01-16T00:14:40.503 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.504 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.504 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.504 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.378751+0000", 2020-01-16T00:14:40.504 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3343000000000006e-05 2020-01-16T00:14:40.504 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.504 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.504 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:40.504 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.378825+0000", 2020-01-16T00:14:40.505 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.64e-05 2020-01-16T00:14:40.505 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.505 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.505 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:40.505 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:13.378901+0000", 2020-01-16T00:14:40.505 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021436350999999999 2020-01-16T00:14:40.505 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.505 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:40.506 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.506 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.506 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.506 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:5155 3.a 3:5069665e:::smithi12313934-2:head [write 1648313~507551] snapc 0=[] ondisk+write+known_if_redirected e63)", 2020-01-16T00:14:40.506 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:18.197762+0000", 2020-01-16T00:14:40.506 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 22.115142126999999, 2020-01-16T00:14:40.506 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014375384999999999, 2020-01-16T00:14:40.506 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:40.507 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:40.507 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:40.507 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:40.507 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:40.507 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 5155 2020-01-16T00:14:40.507 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.507 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:40.508 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.508 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:40.508 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.197762+0000", 2020-01-16T00:14:40.508 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:40.508 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.508 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.508 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:40.508 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.197762+0000", 2020-01-16T00:14:40.509 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2020-01-16T00:14:40.509 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.509 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.509 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:40.509 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.197760+0000", 2020-01-16T00:14:40.509 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00049261900000000002 2020-01-16T00:14:40.509 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.510 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.510 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:40.510 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.198253+0000", 2020-01-16T00:14:40.510 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8258e-05 2020-01-16T00:14:40.510 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.510 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.510 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:40.511 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.198271+0000", 2020-01-16T00:14:40.511 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0740000000000004e-06 2020-01-16T00:14:40.511 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.511 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.511 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:40.511 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.198278+0000", 2020-01-16T00:14:40.511 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001581121 2020-01-16T00:14:40.511 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.512 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.512 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.512 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.199859+0000", 2020-01-16T00:14:40.512 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00064528100000000004 2020-01-16T00:14:40.512 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.512 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.512 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:40.512 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.200504+0000", 2020-01-16T00:14:40.513 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0079495580000000007 2020-01-16T00:14:40.513 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.513 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.513 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.513 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.208454+0000", 2020-01-16T00:14:40.513 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.781e-05 2020-01-16T00:14:40.513 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.513 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.514 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:40.514 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.208552+0000", 2020-01-16T00:14:40.514 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014363900000000001 2020-01-16T00:14:40.514 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.514 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.514 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:40.514 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.208695+0000", 2020-01-16T00:14:40.515 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002629036 2020-01-16T00:14:40.515 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.515 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.515 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:40.515 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.211324+0000", 2020-01-16T00:14:40.515 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022371700000000001 2020-01-16T00:14:40.515 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.515 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.516 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:40.516 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.211548+0000", 2020-01-16T00:14:40.516 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00049117300000000002 2020-01-16T00:14:40.516 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.516 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.516 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:40.516 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.212039+0000", 2020-01-16T00:14:40.516 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7794999999999998e-05 2020-01-16T00:14:40.517 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.517 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.517 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:40.517 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.212097+0000", 2020-01-16T00:14:40.517 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0648000000000001e-05 2020-01-16T00:14:40.517 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.517 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.517 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:40.518 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.212138+0000", 2020-01-16T00:14:40.518 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014375384999999999 2020-01-16T00:14:40.518 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.518 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:40.518 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.518 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.518 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.519 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:5156 3.a 3:5069665e:::smithi12313934-2:head [write 2917489~292780] snapc 0=[] ondisk+write+known_if_redirected e63)", 2020-01-16T00:14:40.519 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:18.203801+0000", 2020-01-16T00:14:40.519 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 22.109103777000001, 2020-01-16T00:14:40.519 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015806831, 2020-01-16T00:14:40.519 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:40.519 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:40.519 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:40.520 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:40.520 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:40.520 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 5156 2020-01-16T00:14:40.520 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.520 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:40.521 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.521 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:40.521 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.203801+0000", 2020-01-16T00:14:40.521 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:40.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.521 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.521 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:40.521 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.203801+0000", 2020-01-16T00:14:40.522 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999967 2020-01-16T00:14:40.522 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.522 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.522 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:40.522 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.203798+0000", 2020-01-16T00:14:40.522 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00051419400000000004 2020-01-16T00:14:40.522 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.522 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.523 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:40.523 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204312+0000", 2020-01-16T00:14:40.523 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3977000000000001e-05 2020-01-16T00:14:40.523 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.523 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.523 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:40.524 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204336+0000", 2020-01-16T00:14:40.524 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4731e-05 2020-01-16T00:14:40.524 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.524 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.524 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:40.524 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204350+0000", 2020-01-16T00:14:40.524 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.522e-05 2020-01-16T00:14:40.525 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.525 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.525 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.525 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204446+0000", 2020-01-16T00:14:40.525 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010202 2020-01-16T00:14:40.525 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.525 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.526 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:40.526 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204548+0000", 2020-01-16T00:14:40.526 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.005158124 2020-01-16T00:14:40.526 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.526 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.526 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.526 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.209706+0000", 2020-01-16T00:14:40.526 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001294309 2020-01-16T00:14:40.527 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.527 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.527 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:40.527 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.211000+0000", 2020-01-16T00:14:40.527 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0065338569999999997 2020-01-16T00:14:40.527 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.527 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.527 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.528 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.217534+0000", 2020-01-16T00:14:40.528 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5905e-05 2020-01-16T00:14:40.528 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.528 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.528 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:40.528 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.217600+0000", 2020-01-16T00:14:40.528 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000107805 2020-01-16T00:14:40.529 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.529 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.529 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:40.529 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.217708+0000", 2020-01-16T00:14:40.529 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012035570000000001 2020-01-16T00:14:40.529 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.529 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.529 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:40.530 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.218911+0000", 2020-01-16T00:14:40.530 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00027311300000000001 2020-01-16T00:14:40.530 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.530 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.530 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:40.530 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.219184+0000", 2020-01-16T00:14:40.530 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00031979399999999998 2020-01-16T00:14:40.530 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.531 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.531 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:40.531 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.219504+0000", 2020-01-16T00:14:40.531 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2009999999999998e-05 2020-01-16T00:14:40.531 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.531 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.531 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:40.532 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.219566+0000", 2020-01-16T00:14:40.532 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1406000000000003e-05 2020-01-16T00:14:40.532 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.532 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.532 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:40.532 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.219608+0000", 2020-01-16T00:14:40.532 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015806831 2020-01-16T00:14:40.532 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.533 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:40.533 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.533 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.533 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.533 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:5157 3.a 3:5069665e:::smithi12313934-2:head [setxattr _header (55) in=7b,truncate 3210269] snapc 0=[] ondisk+write+known_if_redirected e63)", 2020-01-16T00:14:40.533 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:18.204374+0000", 2020-01-16T00:14:40.533 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 22.108530262999999, 2020-01-16T00:14:40.534 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015400502, 2020-01-16T00:14:40.534 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:40.534 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2020-01-16T00:14:40.534 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:40.534 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:40.534 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:40.534 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 5157 2020-01-16T00:14:40.534 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.535 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:40.535 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.535 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:40.535 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204374+0000", 2020-01-16T00:14:40.535 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:40.535 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.535 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.535 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:40.536 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204374+0000", 2020-01-16T00:14:40.536 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2020-01-16T00:14:40.536 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.536 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.536 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:40.536 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204373+0000", 2020-01-16T00:14:40.536 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.4870000000000002e-06 2020-01-16T00:14:40.536 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.537 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.537 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:40.537 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204381+0000", 2020-01-16T00:14:40.537 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4402e-05 2020-01-16T00:14:40.537 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.537 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.537 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:40.538 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204396+0000", 2020-01-16T00:14:40.538 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6640000000000001e-06 2020-01-16T00:14:40.538 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.538 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.538 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:40.538 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204403+0000", 2020-01-16T00:14:40.538 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00021196100000000001 2020-01-16T00:14:40.539 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.539 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.539 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.539 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204614+0000", 2020-01-16T00:14:40.539 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1501999999999994e-05 2020-01-16T00:14:40.539 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.539 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.540 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:40.540 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204686+0000", 2020-01-16T00:14:40.540 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0063990649999999998 2020-01-16T00:14:40.540 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.540 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.540 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.540 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.211085+0000", 2020-01-16T00:14:40.541 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1210000000000003e-05 2020-01-16T00:14:40.541 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.541 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.541 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:40.541 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.211166+0000", 2020-01-16T00:14:40.541 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0071190699999999999 2020-01-16T00:14:40.541 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.541 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.542 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.542 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.218285+0000", 2020-01-16T00:14:40.542 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8831999999999994e-05 2020-01-16T00:14:40.542 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.542 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.542 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:40.542 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.218354+0000", 2020-01-16T00:14:40.543 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013551599999999999 2020-01-16T00:14:40.543 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.543 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.543 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2020-01-16T00:14:40.543 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.218490+0000", 2020-01-16T00:14:40.543 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00057678000000000004 2020-01-16T00:14:40.543 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.543 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.544 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2020-01-16T00:14:40.544 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.219066+0000", 2020-01-16T00:14:40.544 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022067000000000001 2020-01-16T00:14:40.544 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.544 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.544 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:40.544 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.219287+0000", 2020-01-16T00:14:40.544 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000401868 2020-01-16T00:14:40.545 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.545 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.545 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2020-01-16T00:14:40.545 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.219689+0000", 2020-01-16T00:14:40.545 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0040000000000003e-05 2020-01-16T00:14:40.545 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.545 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.545 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2020-01-16T00:14:40.546 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.219729+0000", 2020-01-16T00:14:40.546 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5762e-05 2020-01-16T00:14:40.546 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.546 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.546 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:40.546 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.219775+0000", 2020-01-16T00:14:40.546 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015400502 2020-01-16T00:14:40.547 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.547 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:40.547 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.547 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.547 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.547 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.4488.0:5158 3.a 3:5069665e:::smithi12313934-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e63)", 2020-01-16T00:14:40.547 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2020-01-16T00:14:18.204424+0000", 2020-01-16T00:14:40.547 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 22.108480551, 2020-01-16T00:14:40.548 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015513872999999999, 2020-01-16T00:14:40.548 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2020-01-16T00:14:40.548 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2020-01-16T00:14:40.548 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2020-01-16T00:14:40.548 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.4488", 2020-01-16T00:14:40.548 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "v1:172.21.15.123:0/1615684717", 2020-01-16T00:14:40.548 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 5158 2020-01-16T00:14:40.548 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.549 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2020-01-16T00:14:40.549 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.549 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2020-01-16T00:14:40.549 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204424+0000", 2020-01-16T00:14:40.549 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2020-01-16T00:14:40.549 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.549 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.550 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2020-01-16T00:14:40.550 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204424+0000", 2020-01-16T00:14:40.550 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2020-01-16T00:14:40.550 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.550 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.550 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2020-01-16T00:14:40.550 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204423+0000", 2020-01-16T00:14:40.550 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4739999999999996e-06 2020-01-16T00:14:40.551 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.551 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.551 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2020-01-16T00:14:40.551 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204429+0000", 2020-01-16T00:14:40.551 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9245999999999998e-05 2020-01-16T00:14:40.551 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.551 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.551 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2020-01-16T00:14:40.552 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204448+0000", 2020-01-16T00:14:40.552 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.4540000000000007e-06 2020-01-16T00:14:40.552 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.552 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.552 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2020-01-16T00:14:40.552 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204456+0000", 2020-01-16T00:14:40.552 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00027379599999999998 2020-01-16T00:14:40.552 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.553 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.553 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.553 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204730+0000", 2020-01-16T00:14:40.553 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2149000000000004e-05 2020-01-16T00:14:40.553 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.553 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.554 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:40.554 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.204792+0000", 2020-01-16T00:14:40.554 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0064440110000000004 2020-01-16T00:14:40.554 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.554 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.554 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.554 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.211236+0000", 2020-01-16T00:14:40.554 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2772000000000001e-05 2020-01-16T00:14:40.555 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.555 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.555 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for blocked object", 2020-01-16T00:14:40.555 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.211299+0000", 2020-01-16T00:14:40.555 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0074481130000000001 2020-01-16T00:14:40.555 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.555 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.555 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.556 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.218747+0000", 2020-01-16T00:14:40.556 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011178100000000001 2020-01-16T00:14:40.556 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.556 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.556 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for rw locks", 2020-01-16T00:14:40.556 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.218859+0000", 2020-01-16T00:14:40.556 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00093467799999999998 2020-01-16T00:14:40.556 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.557 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.557 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2020-01-16T00:14:40.557 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.219794+0000", 2020-01-16T00:14:40.557 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2992999999999998e-05 2020-01-16T00:14:40.557 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.557 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.557 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2020-01-16T00:14:40.557 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.219857+0000", 2020-01-16T00:14:40.558 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1285000000000005e-05 2020-01-16T00:14:40.558 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2020-01-16T00:14:40.558 INFO:teuthology.orchestra.run.smithi145.stdout: { 2020-01-16T00:14:40.558 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2020-01-16T00:14:40.558 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2020-01-16T00:14:18.219938+0000", 2020-01-16T00:14:40.558 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015513872999999999 2020-01-16T00:14:40.558 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.559 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:40.559 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.559 INFO:teuthology.orchestra.run.smithi145.stdout: } 2020-01-16T00:14:40.559 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2020-01-16T00:14:40.559 INFO:teuthology.orchestra.run.smithi145.stdout:} 2020-01-16T00:14:41.623 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2020-01-16T00:14:41.624 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2020-01-16T00:14:42.183 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2020-01-16T00:14:43.186 INFO:teuthology.orchestra.run.smithi123.stderr:noscrub is unset 2020-01-16T00:14:43.200 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2020-01-16T00:14:44.194 INFO:teuthology.orchestra.run.smithi123.stderr:nodeep-scrub is unset 2020-01-16T00:14:44.208 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2020-01-16T00:14:44.208 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:14:44.454 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:14:44.454 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":73,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:14:44.178042+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:14.368773+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.895777+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.185848+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.667896+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:14.148239+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.986248+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.699155+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:14:47.468 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:14:47.719 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:14:47.719 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":73,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:14:44.178042+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:14.368773+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.895777+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.185848+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.667896+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:14.148239+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.986248+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.699155+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:14:50.733 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:14:50.984 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:14:50.984 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":73,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:14:44.178042+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:14.368773+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.895777+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.185848+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.667896+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:14.148239+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.986248+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.699155+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:14:53.998 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:14:54.248 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:14:54.249 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":73,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:14:44.178042+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:14.368773+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.895777+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.185848+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.667896+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:14.148239+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.986248+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.699155+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:14:57.273 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:14:57.525 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:14:57.526 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":73,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:14:44.178042+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:14.368773+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.895777+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.185848+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.667896+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:14.148239+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.986248+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.699155+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:15:00.540 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:15:00.801 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:15:00.802 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":73,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:14:44.178042+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:14.368773+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.895777+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.185848+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.667896+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:14.148239+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.986248+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.699155+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:15:03.820 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:15:04.071 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:15:04.072 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":73,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:14:44.178042+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:14.368773+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.895777+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.185848+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.667896+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:14.148239+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.986248+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.699155+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:15:07.086 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:15:07.345 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:15:07.345 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":73,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:14:44.178042+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:14.368773+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.895777+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.185848+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.667896+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:14.148239+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.986248+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.699155+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:15:10.359 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:15:10.614 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:15:10.614 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":73,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:14:44.178042+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:14.368773+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.895777+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.185848+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.667896+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:14.148239+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.986248+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.699155+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:15:13.633 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:15:13.888 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:15:13.889 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":74,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:15:13.376685+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:14.368773+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.895777+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:13.325525+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.667896+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:14.148239+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.986248+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:14:13.699155+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:15:16.902 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:15:17.156 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:15:17.157 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":76,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:15:15.389329+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.772542+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.060382+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:13.325525+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.668298+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.514703+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.434417+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.636174+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:15:20.170 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:15:20.421 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:15:20.422 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":76,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:15:15.389329+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.772542+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.060382+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:13.325525+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.668298+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.514703+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.434417+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.636174+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:15:23.443 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:15:23.697 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:15:23.697 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":76,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:15:15.389329+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.772542+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.060382+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:13.325525+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.668298+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.514703+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.434417+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.636174+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:15:26.711 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:15:26.968 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:15:26.968 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":76,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:15:15.389329+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.772542+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.060382+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:13.325525+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.668298+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.514703+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.434417+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.636174+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:15:29.982 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:15:30.239 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:15:30.240 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":76,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:15:15.389329+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.772542+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.060382+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:13.325525+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.668298+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.514703+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.434417+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.636174+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:15:33.255 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:15:33.508 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:15:33.508 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":76,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:15:15.389329+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.772542+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.060382+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:13.325525+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.668298+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.514703+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.434417+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.636174+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:15:36.523 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:15:36.778 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:15:36.779 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":76,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:15:15.389329+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.772542+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.060382+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:13.325525+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.668298+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.514703+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.434417+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.636174+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:15:39.797 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:15:40.058 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:15:40.059 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":76,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:15:15.389329+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.772542+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.060382+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:13.325525+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.668298+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.514703+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.434417+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.636174+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:15:43.073 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:15:43.328 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:15:43.329 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":76,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:15:15.389329+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.772542+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.060382+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:13.325525+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.668298+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.514703+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.434417+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.636174+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:15:46.347 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:15:46.599 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:15:46.599 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":76,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:15:15.389329+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.772542+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.060382+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:13.325525+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.668298+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.514703+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.434417+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.636174+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:15:49.620 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:15:49.874 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:15:49.875 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":76,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:15:15.389329+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.772542+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.060382+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:13.325525+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.668298+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.514703+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.434417+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.636174+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:15:52.896 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:15:53.154 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:15:53.154 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":76,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:15:15.389329+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.772542+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.060382+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:13.325525+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.668298+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.514703+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.434417+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.636174+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:15:56.170 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:15:56.425 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:15:56.425 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":76,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:15:15.389329+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.772542+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.060382+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:13.325525+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.668298+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.514703+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.434417+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.636174+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:15:59.440 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:15:59.686 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:15:59.687 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":76,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:15:15.389329+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.772542+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.060382+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:13.325525+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.668298+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.514703+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.434417+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.636174+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:16:02.700 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:16:02.959 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:16:02.959 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":76,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:15:15.389329+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.772542+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.060382+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:13.325525+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.668298+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.514703+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.434417+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.636174+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:16:05.973 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:16:06.226 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:16:06.226 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":76,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:15:15.389329+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.772542+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.060382+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:13.325525+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.668298+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.514703+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.434417+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.636174+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:16:09.248 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:16:09.503 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:16:09.503 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":76,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:15:15.389329+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.772542+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.060382+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:13.325525+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.668298+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.514703+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.434417+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.636174+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:16:12.518 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:16:12.762 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:16:12.763 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":76,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:15:15.389329+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.772542+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.060382+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:13.325525+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.668298+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.514703+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.434417+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.636174+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:16:15.776 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:16:16.029 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:16:16.029 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":78,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:16:15.091738+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.975476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.936467+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:13.392573+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.668298+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:15:14.514703+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.472979+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.882885+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:16:19.046 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:16:19.306 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:16:19.306 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":79,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:16:16.100870+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.975476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.936467+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:13.392573+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:15.539735+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:15.497325+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.472979+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.882885+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:16:22.320 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:16:22.566 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:16:22.567 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":79,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:16:16.100870+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.975476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.936467+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:13.392573+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:15.539735+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:15.497325+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.472979+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.882885+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:16:25.586 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:16:25.855 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:16:25.856 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":79,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:16:16.100870+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.975476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.936467+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:13.392573+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:15.539735+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:15.497325+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.472979+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.882885+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:16:28.873 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:16:29.125 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:16:29.126 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":79,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:16:16.100870+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.975476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.936467+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:13.392573+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:15.539735+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:15.497325+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.472979+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.882885+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:16:32.141 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:16:32.401 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:16:32.401 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":79,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:16:16.100870+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.975476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.936467+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:13.392573+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:15.539735+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:15.497325+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.472979+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.882885+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:16:35.415 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:16:35.671 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:16:35.672 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":79,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:16:16.100870+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.975476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.936467+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:13.392573+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:15.539735+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:15.497325+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.472979+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.882885+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:16:38.685 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:16:38.936 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:16:38.936 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":79,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:16:16.100870+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.975476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.936467+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:13.392573+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:15.539735+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:15.497325+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.472979+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.882885+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:16:41.950 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:16:42.207 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:16:42.208 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":79,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:16:16.100870+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.975476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.936467+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:13.392573+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:15.539735+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:15.497325+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.472979+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.882885+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:16:45.222 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:16:45.475 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:16:45.475 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":79,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:16:16.100870+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.975476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.936467+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:13.392573+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:15.539735+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:15.497325+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.472979+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.882885+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:16:48.490 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:16:48.743 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:16:48.744 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":79,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:16:16.100870+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.975476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.936467+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:13.392573+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:15.539735+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:15.497325+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.472979+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.882885+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:16:51.764 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:16:52.026 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:16:52.027 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":79,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:16:16.100870+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.975476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.936467+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:13.392573+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:15.539735+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:15.497325+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.472979+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.882885+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:16:55.040 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:16:55.300 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:16:55.301 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":79,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:16:16.100870+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.975476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.936467+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:13.392573+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:15.539735+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:15.497325+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.472979+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.882885+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:16:58.314 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:16:58.562 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:16:58.562 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":79,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:16:16.100870+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.975476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.936467+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:13.392573+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:15.539735+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:15.497325+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.472979+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.882885+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:17:01.577 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:17:01.829 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:17:01.830 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":79,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:16:16.100870+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.975476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.936467+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:13.392573+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:15.539735+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:15.497325+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.472979+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.882885+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:17:04.856 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:17:05.111 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:17:05.112 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":79,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:16:16.100870+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.975476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.936467+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:13.392573+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:15.539735+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:15.497325+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.472979+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.882885+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:17:08.125 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:17:08.378 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:17:08.379 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":79,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:16:16.100870+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.975476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.936467+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:13.392573+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:15.539735+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:15.497325+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.472979+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.882885+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:17:11.392 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:17:11.651 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:17:11.651 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":79,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:16:16.100870+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.975476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.936467+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:13.392573+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:15.539735+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:15.497325+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.472979+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.882885+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:17:14.665 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:17:14.926 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:17:14.926 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":81,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:17:14.857532+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.975476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.936467+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:13.574626+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:15.539735+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:15.497325+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:14.554468+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:16:14.882885+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:17:17.940 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:17:18.197 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:17:18.198 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":83,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:17:16.879972+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.098362+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.366842+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:13.574626+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.748982+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:16.416894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:14.554468+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.662120+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:17:21.212 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:17:21.469 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:17:21.469 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":83,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:17:16.879972+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.098362+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.366842+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:13.574626+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.748982+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:16.416894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:14.554468+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.662120+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:17:24.489 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:17:24.742 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:17:24.742 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":83,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:17:16.879972+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.098362+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.366842+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:13.574626+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.748982+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:16.416894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:14.554468+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.662120+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:17:27.756 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:17:28.002 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:17:28.003 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":83,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:17:16.879972+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.098362+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.366842+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:13.574626+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.748982+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:16.416894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:14.554468+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.662120+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:17:31.017 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:17:31.274 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:17:31.275 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":83,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:17:16.879972+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.098362+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.366842+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:13.574626+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.748982+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:16.416894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:14.554468+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.662120+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:17:34.295 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:17:34.545 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:17:34.545 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":83,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:17:16.879972+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.098362+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.366842+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:13.574626+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.748982+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:16.416894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:14.554468+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.662120+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:17:37.559 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:17:37.809 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:17:37.810 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":83,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:17:16.879972+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.098362+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.366842+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:13.574626+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.748982+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:16.416894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:14.554468+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.662120+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:17:40.829 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:17:41.086 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:17:41.086 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":83,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:17:16.879972+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.098362+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.366842+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:13.574626+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.748982+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:16.416894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:14.554468+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.662120+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:17:44.099 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:17:44.353 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:17:44.354 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":83,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:17:16.879972+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.098362+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.366842+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:13.574626+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.748982+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:16.416894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:14.554468+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.662120+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:17:47.367 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:17:47.623 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:17:47.623 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":83,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:17:16.879972+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.098362+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.366842+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:13.574626+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.748982+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:16.416894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:14.554468+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.662120+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:17:50.637 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:17:50.890 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:17:50.890 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":83,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:17:16.879972+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.098362+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.366842+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:13.574626+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.748982+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:16.416894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:14.554468+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.662120+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:17:53.905 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:17:54.167 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:17:54.167 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":83,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:17:16.879972+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.098362+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.366842+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:13.574626+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.748982+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:16.416894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:14.554468+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.662120+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:17:57.181 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:17:57.432 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:17:57.432 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":83,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:17:16.879972+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.098362+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.366842+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:13.574626+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.748982+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:16.416894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:14.554468+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.662120+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:18:00.446 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:18:00.696 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:18:00.697 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":83,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:17:16.879972+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.098362+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.366842+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:13.574626+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.748982+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:16.416894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:14.554468+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.662120+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:18:03.710 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:18:03.960 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:18:03.961 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":83,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:17:16.879972+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.098362+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.366842+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:13.574626+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.748982+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:16.416894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:14.554468+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.662120+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:18:06.980 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:18:07.239 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:18:07.239 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":83,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:17:16.879972+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.098362+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.366842+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:13.574626+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.748982+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:16.416894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:14.554468+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.662120+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:18:10.253 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:18:10.503 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:18:10.504 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":83,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:17:16.879972+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.098362+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.366842+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:13.574626+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.748982+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:16.416894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:14.554468+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.662120+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:18:13.518 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:18:13.767 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:18:13.767 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":83,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:17:16.879972+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.098362+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.366842+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:13.574626+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.748982+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:16.416894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:14.554468+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:15.662120+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:18:16.782 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:18:17.036 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:18:17.037 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":85,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:18:16.870786+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:15.981982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.091252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:14.488620+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.676401+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:17:16.416894+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:14.602327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.067116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:18:20.051 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:18:20.308 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:18:20.309 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":86,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:18:17.879512+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:15.981982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.091252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:14.488620+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.676401+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:17.423411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:14.602327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.067116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:18:23.323 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:18:23.581 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:18:23.582 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":86,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:18:17.879512+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:15.981982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.091252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:14.488620+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.676401+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:17.423411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:14.602327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.067116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:18:26.598 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:18:26.857 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:18:26.858 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":86,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:18:17.879512+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:15.981982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.091252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:14.488620+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.676401+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:17.423411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:14.602327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.067116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:18:29.871 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:18:30.124 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:18:30.124 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":86,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:18:17.879512+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:15.981982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.091252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:14.488620+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.676401+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:17.423411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:14.602327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.067116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:18:33.145 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:18:33.401 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:18:33.401 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":86,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:18:17.879512+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:15.981982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.091252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:14.488620+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.676401+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:17.423411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:14.602327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.067116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:18:36.423 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:18:36.680 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:18:36.680 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":86,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:18:17.879512+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:15.981982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.091252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:14.488620+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.676401+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:17.423411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:14.602327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.067116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:18:39.694 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:18:39.946 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:18:39.946 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":86,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:18:17.879512+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:15.981982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.091252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:14.488620+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.676401+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:17.423411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:14.602327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.067116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:18:42.966 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:18:43.228 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:18:43.228 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":86,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:18:17.879512+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:15.981982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.091252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:14.488620+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.676401+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:17.423411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:14.602327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.067116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:18:46.249 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:18:46.505 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:18:46.506 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":86,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:18:17.879512+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:15.981982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.091252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:14.488620+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.676401+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:17.423411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:14.602327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.067116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:18:49.520 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:18:49.773 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:18:49.773 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":86,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:18:17.879512+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:15.981982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.091252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:14.488620+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.676401+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:17.423411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:14.602327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.067116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:18:52.788 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:18:53.037 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:18:53.037 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":86,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:18:17.879512+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:15.981982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.091252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:14.488620+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.676401+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:17.423411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:14.602327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.067116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:18:56.053 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:18:56.306 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:18:56.306 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":86,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:18:17.879512+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:15.981982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.091252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:14.488620+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.676401+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:17.423411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:14.602327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.067116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:18:59.321 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:18:59.576 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:18:59.576 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":86,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:18:17.879512+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:15.981982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.091252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:14.488620+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.676401+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:17.423411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:14.602327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.067116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:19:02.591 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:19:02.850 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:19:02.851 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":86,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:18:17.879512+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:15.981982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.091252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:14.488620+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.676401+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:17.423411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:14.602327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.067116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:19:05.865 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:19:06.111 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:19:06.112 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":86,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:18:17.879512+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:15.981982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.091252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:14.488620+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.676401+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:17.423411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:14.602327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.067116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:19:09.125 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:19:09.381 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:19:09.382 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":86,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:18:17.879512+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:15.981982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.091252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:14.488620+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.676401+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:17.423411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:14.602327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.067116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:19:12.396 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:19:12.650 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:19:12.651 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":86,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:18:17.879512+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:15.981982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.091252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:14.488620+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.676401+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:17.423411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:14.602327+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.067116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:19:15.671 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:19:15.925 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:19:15.926 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":87,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:19:15.145892+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:15.981982+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.091252+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:14.488620+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.676401+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:17.423411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:15.000034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:18:16.067116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:19:18.939 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:19:19.187 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:19:19.187 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":89,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:19:17.891985+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.137232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.952622+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:15.520966+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:17.097476+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:17.481122+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:15.000034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.977325+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:19:22.201 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:19:22.457 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:19:22.458 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":89,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:19:17.891985+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.137232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.952622+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:15.520966+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:17.097476+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:17.481122+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:15.000034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.977325+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:19:25.471 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:19:25.726 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:19:25.727 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":89,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:19:17.891985+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.137232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.952622+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:15.520966+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:17.097476+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:17.481122+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:15.000034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.977325+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:19:28.742 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:19:28.996 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:19:28.996 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":89,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:19:17.891985+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.137232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.952622+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:15.520966+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:17.097476+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:17.481122+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:15.000034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.977325+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:19:32.011 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:19:32.268 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:19:32.268 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":89,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:19:17.891985+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.137232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.952622+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:15.520966+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:17.097476+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:17.481122+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:15.000034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.977325+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:19:35.289 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:19:35.537 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:19:35.537 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":89,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:19:17.891985+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.137232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.952622+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:15.520966+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:17.097476+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:17.481122+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:15.000034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.977325+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:19:38.557 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:19:38.812 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:19:38.812 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":89,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:19:17.891985+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.137232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.952622+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:15.520966+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:17.097476+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:17.481122+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:15.000034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.977325+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:19:41.832 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:19:42.092 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:19:42.093 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":89,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:19:17.891985+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.137232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.952622+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:15.520966+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:17.097476+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:17.481122+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:15.000034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.977325+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:19:45.107 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:19:45.354 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:19:45.354 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":89,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:19:17.891985+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.137232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.952622+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:15.520966+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:17.097476+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:17.481122+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:15.000034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.977325+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:19:48.375 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:19:48.633 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:19:48.633 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":89,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:19:17.891985+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.137232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.952622+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:15.520966+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:17.097476+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:17.481122+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:15.000034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.977325+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:19:51.647 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:19:51.903 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:19:51.903 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":89,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:19:17.891985+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.137232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.952622+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:15.520966+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:17.097476+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:17.481122+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:15.000034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.977325+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:19:54.917 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:19:55.172 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:19:55.172 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":89,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:19:17.891985+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.137232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.952622+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:15.520966+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:17.097476+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:17.481122+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:15.000034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.977325+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:19:58.192 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:19:58.443 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:19:58.443 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":89,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:19:17.891985+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.137232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.952622+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:15.520966+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:17.097476+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:17.481122+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:15.000034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.977325+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:20:01.463 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:20:01.712 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:20:01.712 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":89,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:19:17.891985+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.137232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.952622+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:15.520966+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:17.097476+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:17.481122+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:15.000034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.977325+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:20:04.726 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:20:04.982 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:20:04.983 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":89,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:19:17.891985+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.137232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.952622+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:15.520966+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:17.097476+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:17.481122+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:15.000034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.977325+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:20:07.997 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:20:08.257 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:20:08.258 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":89,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:19:17.891985+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.137232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.952622+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:15.520966+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:17.097476+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:17.481122+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:15.000034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.977325+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:20:11.272 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:20:11.529 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:20:11.529 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":89,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:19:17.891985+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.137232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.952622+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:15.520966+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:17.097476+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:17.481122+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:15.000034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.977325+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:20:14.550 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:20:14.810 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:20:14.810 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":89,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:19:17.891985+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.137232+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.952622+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:15.520966+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:17.097476+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:17.481122+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:15.000034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:16.977325+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:20:17.828 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:20:18.085 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:20:18.085 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":91,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:20:17.173538+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:16.204007+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.116385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:15.753141+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:17.097476+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:19:17.481122+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:15.233577+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.153524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:20:21.099 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:20:21.347 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:20:21.347 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":92,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:20:18.176779+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:16.204007+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.116385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:15.753141+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.486796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.841199+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:15.233577+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.153524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:20:24.361 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:20:24.609 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:20:24.610 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":92,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:20:18.176779+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:16.204007+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.116385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:15.753141+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.486796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.841199+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:15.233577+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.153524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:20:27.624 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:20:27.876 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:20:27.877 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":92,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:20:18.176779+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:16.204007+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.116385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:15.753141+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.486796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.841199+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:15.233577+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.153524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:20:30.890 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:20:31.168 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:20:31.169 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":92,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:20:18.176779+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:16.204007+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.116385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:15.753141+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.486796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.841199+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:15.233577+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.153524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:20:34.183 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:20:34.434 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:20:34.434 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":92,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:20:18.176779+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:16.204007+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.116385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:15.753141+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.486796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.841199+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:15.233577+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.153524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:20:37.448 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:20:37.707 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:20:37.708 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":92,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:20:18.176779+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:16.204007+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.116385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:15.753141+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.486796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.841199+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:15.233577+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.153524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:20:40.721 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:20:40.978 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:20:40.979 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":92,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:20:18.176779+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:16.204007+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.116385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:15.753141+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.486796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.841199+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:15.233577+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.153524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:20:43.992 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:20:44.247 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:20:44.247 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":92,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:20:18.176779+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:16.204007+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.116385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:15.753141+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.486796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.841199+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:15.233577+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.153524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:20:47.261 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:20:47.511 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:20:47.512 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":92,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:20:18.176779+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:16.204007+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.116385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:15.753141+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.486796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.841199+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:15.233577+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.153524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:20:50.526 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:20:50.772 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:20:50.772 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":92,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:20:18.176779+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:16.204007+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.116385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:15.753141+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.486796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.841199+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:15.233577+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.153524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:20:53.792 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:20:54.049 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:20:54.049 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":92,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:20:18.176779+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:16.204007+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.116385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:15.753141+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.486796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.841199+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:15.233577+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.153524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:20:57.063 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:20:57.311 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:20:57.311 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":92,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:20:18.176779+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:16.204007+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.116385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:15.753141+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.486796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.841199+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:15.233577+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.153524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:21:00.330 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:21:00.589 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:21:00.589 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":92,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:20:18.176779+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:16.204007+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.116385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:15.753141+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.486796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.841199+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:15.233577+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.153524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:21:03.603 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:21:03.863 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:21:03.863 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":92,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:20:18.176779+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:16.204007+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.116385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:15.753141+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.486796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.841199+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:15.233577+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.153524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:21:06.884 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:21:07.136 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:21:07.136 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":92,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:20:18.176779+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:16.204007+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.116385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:15.753141+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.486796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.841199+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:15.233577+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.153524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:21:10.157 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:21:10.414 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:21:10.414 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":92,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:20:18.176779+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:16.204007+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.116385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:15.753141+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.486796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.841199+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:15.233577+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.153524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:21:13.426 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:21:13.673 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:21:13.673 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":92,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:20:18.176779+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:16.204007+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.116385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:15.753141+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.486796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.841199+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:15.233577+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.153524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:21:16.687 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:21:16.942 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:21:16.942 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":94,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:21:16.930013+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:16.561152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.116385+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:16.415070+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.486796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.841199+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:15.421282+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:20:17.153524+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:21:19.957 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:21:20.209 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:21:20.209 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":96,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:21:18.945911+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:16.561152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.210658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:16.415070+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.688143+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:18.634793+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:15.421282+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.157951+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:21:23.223 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:21:23.479 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:21:23.479 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":96,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:21:18.945911+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:16.561152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.210658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:16.415070+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.688143+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:18.634793+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:15.421282+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.157951+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:21:26.493 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:21:26.751 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:21:26.751 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":96,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:21:18.945911+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:16.561152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.210658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:16.415070+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.688143+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:18.634793+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:15.421282+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.157951+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:21:29.776 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:21:30.027 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:21:30.027 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":96,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:21:18.945911+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:16.561152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.210658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:16.415070+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.688143+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:18.634793+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:15.421282+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.157951+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:21:33.047 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:21:33.302 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:21:33.302 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":96,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:21:18.945911+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:16.561152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.210658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:16.415070+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.688143+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:18.634793+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:15.421282+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.157951+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:21:36.317 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:21:36.566 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:21:36.567 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":96,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:21:18.945911+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:16.561152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.210658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:16.415070+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.688143+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:18.634793+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:15.421282+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.157951+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:21:39.580 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:21:39.831 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:21:39.832 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":96,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:21:18.945911+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:16.561152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.210658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:16.415070+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.688143+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:18.634793+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:15.421282+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.157951+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:21:42.846 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:21:43.097 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:21:43.097 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":96,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:21:18.945911+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:16.561152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.210658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:16.415070+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.688143+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:18.634793+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:15.421282+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.157951+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:21:46.116 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:21:46.369 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:21:46.369 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":96,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:21:18.945911+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:16.561152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.210658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:16.415070+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.688143+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:18.634793+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:15.421282+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.157951+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:21:49.384 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:21:49.633 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:21:49.634 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":96,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:21:18.945911+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:16.561152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.210658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:16.415070+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.688143+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:18.634793+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:15.421282+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.157951+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:21:52.648 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:21:52.907 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:21:52.908 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":96,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:21:18.945911+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:16.561152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.210658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:16.415070+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.688143+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:18.634793+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:15.421282+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.157951+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:21:55.922 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:21:56.176 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:21:56.177 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":96,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:21:18.945911+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:16.561152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.210658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:16.415070+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.688143+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:18.634793+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:15.421282+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.157951+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:21:59.196 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:21:59.447 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:21:59.448 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":96,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:21:18.945911+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:16.561152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.210658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:16.415070+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.688143+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:18.634793+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:15.421282+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.157951+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:22:02.469 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:22:02.726 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:22:02.726 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":96,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:21:18.945911+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:16.561152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.210658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:16.415070+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.688143+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:18.634793+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:15.421282+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.157951+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:22:05.747 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:22:06.026 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:22:06.026 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":96,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:21:18.945911+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:16.561152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.210658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:16.415070+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.688143+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:18.634793+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:15.421282+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.157951+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:22:09.040 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:22:09.296 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:22:09.297 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":96,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:21:18.945911+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:16.561152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.210658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:16.415070+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.688143+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:18.634793+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:15.421282+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.157951+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:22:12.311 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:22:12.562 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:22:12.563 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":96,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:21:18.945911+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:16.561152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.210658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:16.415070+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.688143+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:18.634793+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:15.421282+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.157951+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:22:15.582 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:22:15.825 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:22:15.825 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":97,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:22:15.676157+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:16.561152+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.210658+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:16.415070+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.688143+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:18.634793+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:15.624788+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:17.157951+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:22:18.839 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:22:19.100 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:22:19.100 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":99,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:22:18.955397+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.071370+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.440140+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.201686+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:18.647651+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:21:18.634793+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:15.624788+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.864365+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:22:22.114 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:22:22.364 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:22:22.364 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":101,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:22:20.948699+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.071370+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.440140+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.201686+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:18.647651+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:19.348194+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:15.624788+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.864365+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:22:25.378 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:22:25.630 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:22:25.631 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":101,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:22:20.948699+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.071370+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.440140+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.201686+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:18.647651+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:19.348194+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:15.624788+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.864365+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:22:28.645 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:22:28.894 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:22:28.894 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":102,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:22:25.951000+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.071370+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.440140+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.201686+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:18.647651+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:19.348194+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:15.624788+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.864365+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:22:31.912 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:22:32.171 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:22:32.172 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":103,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:22:30.953738+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.071370+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.440140+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.201686+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:18.647651+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:19.348194+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:15.624788+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.864365+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:22:35.184 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:22:35.435 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:22:35.435 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":103,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:22:30.953738+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.071370+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.440140+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.201686+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:18.647651+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:19.348194+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:15.624788+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.864365+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:22:38.450 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:22:38.707 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:22:38.708 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":104,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:22:35.955955+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.071370+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.440140+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.201686+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:18.647651+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:19.348194+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:15.624788+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.864365+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:22:41.723 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:22:41.977 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:22:41.978 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":104,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:22:35.955955+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.071370+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.440140+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.201686+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:18.647651+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:19.348194+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:15.624788+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.864365+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:22:44.992 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:22:45.249 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:22:45.250 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":104,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:22:35.955955+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.071370+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.440140+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.201686+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:18.647651+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:19.348194+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:15.624788+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.864365+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:22:48.263 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:22:48.523 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:22:48.523 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":104,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:22:35.955955+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.071370+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.440140+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.201686+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:18.647651+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:19.348194+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:15.624788+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.864365+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:22:51.537 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:22:51.794 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:22:51.795 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":104,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:22:35.955955+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.071370+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.440140+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.201686+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:18.647651+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:19.348194+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:15.624788+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.864365+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:22:54.809 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:22:55.068 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:22:55.068 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":104,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:22:35.955955+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.071370+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.440140+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.201686+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:18.647651+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:19.348194+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:15.624788+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.864365+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:22:58.096 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:22:58.363 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:22:58.363 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":104,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:22:35.955955+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.071370+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.440140+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.201686+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:18.647651+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:19.348194+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:15.624788+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.864365+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:23:01.377 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:23:01.629 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:23:01.629 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":104,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:22:35.955955+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.071370+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.440140+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.201686+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:18.647651+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:19.348194+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:15.624788+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.864365+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:23:04.643 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:23:04.896 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:23:04.896 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":104,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:22:35.955955+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.071370+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.440140+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.201686+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:18.647651+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:19.348194+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:15.624788+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.864365+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:23:07.911 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:23:08.168 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:23:08.169 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":104,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:22:35.955955+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.071370+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.440140+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.201686+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:18.647651+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:19.348194+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:15.624788+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.864365+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:23:11.191 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:23:11.445 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:23:11.446 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":104,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:22:35.955955+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.071370+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.440140+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.201686+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:18.647651+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:19.348194+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:15.624788+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.864365+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:23:14.462 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:23:14.713 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:23:14.714 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":104,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:22:35.955955+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.071370+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.440140+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.201686+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:18.647651+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:19.348194+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:15.624788+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.864365+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:23:17.727 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:23:17.980 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:23:17.980 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":105,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:23:16.236873+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.071370+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.440140+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.201686+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:18.647651+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:19.348194+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:15.655423+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:22:17.864365+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:23:20.995 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:23:21.252 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:23:21.252 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":108,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:23:20.002452+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:18.015023+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.483047+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.351288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:19.544206+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:19.496517+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:15.655423+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.869498+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:23:24.266 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:23:24.520 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:23:24.520 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":108,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:23:20.002452+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:18.015023+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.483047+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.351288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:19.544206+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:19.496517+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:15.655423+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.869498+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:23:27.533 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:23:27.795 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:23:27.796 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":108,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:23:20.002452+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:18.015023+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.483047+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.351288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:19.544206+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:19.496517+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:15.655423+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.869498+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:23:30.823 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:23:31.073 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:23:31.073 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":108,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:23:20.002452+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:18.015023+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.483047+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.351288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:19.544206+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:19.496517+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:15.655423+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.869498+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:23:34.095 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:23:34.349 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:23:34.349 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":108,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:23:20.002452+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:18.015023+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.483047+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.351288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:19.544206+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:19.496517+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:15.655423+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.869498+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:23:37.363 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:23:37.621 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:23:37.622 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":108,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:23:20.002452+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:18.015023+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.483047+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.351288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:19.544206+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:19.496517+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:15.655423+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.869498+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:23:40.639 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:23:40.894 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:23:40.895 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":108,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:23:20.002452+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:18.015023+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.483047+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.351288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:19.544206+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:19.496517+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:15.655423+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.869498+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:23:43.914 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:23:44.164 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:23:44.165 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":108,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:23:20.002452+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:18.015023+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.483047+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.351288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:19.544206+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:19.496517+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:15.655423+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.869498+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:23:47.178 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:23:47.436 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:23:47.437 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":108,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:23:20.002452+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:18.015023+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.483047+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.351288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:19.544206+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:19.496517+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:15.655423+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.869498+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:23:50.450 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:23:50.706 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:23:50.706 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":108,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:23:20.002452+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:18.015023+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.483047+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.351288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:19.544206+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:19.496517+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:15.655423+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.869498+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:23:53.726 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:23:53.975 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:23:53.976 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":108,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:23:20.002452+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:18.015023+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.483047+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.351288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:19.544206+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:19.496517+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:15.655423+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.869498+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:23:56.990 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:23:57.252 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:23:57.253 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":108,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:23:20.002452+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:18.015023+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.483047+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.351288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:19.544206+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:19.496517+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:15.655423+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.869498+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:24:00.262 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:24:00.526 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:24:00.527 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":108,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:23:20.002452+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:18.015023+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.483047+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.351288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:19.544206+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:19.496517+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:15.655423+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.869498+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:24:03.540 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:24:03.796 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:24:03.796 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":108,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:23:20.002452+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:18.015023+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.483047+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.351288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:19.544206+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:19.496517+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:15.655423+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.869498+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:24:06.810 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:24:07.067 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:24:07.067 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":108,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:23:20.002452+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:18.015023+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.483047+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.351288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:19.544206+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:19.496517+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:15.655423+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.869498+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:24:10.081 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:24:10.337 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:24:10.337 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":108,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:23:20.002452+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:18.015023+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.483047+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.351288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:19.544206+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:19.496517+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:15.655423+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.869498+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:24:13.353 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:24:13.604 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:24:13.604 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":108,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:23:20.002452+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:18.015023+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.483047+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.351288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:19.544206+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:19.496517+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:15.655423+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.869498+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:24:16.619 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:24:16.871 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:24:16.871 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":108,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:23:20.002452+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:18.015023+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.483047+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.351288+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:19.544206+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:19.496517+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:15.655423+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:17.869498+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:24:19.885 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:24:20.141 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:24:20.142 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":112,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:24:20.028132+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:18.736580+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.541121+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.995801+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:19.995706+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:23:19.496517+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:16.689378+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.976207+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:24:23.156 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:24:23.412 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:24:23.412 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":113,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:24:21.040331+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:18.736580+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.541121+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.995801+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:19.995706+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:20.236979+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:16.689378+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.976207+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:24:26.426 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:24:26.679 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:24:26.679 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":114,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:24:25.995092+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:18.736580+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.541121+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.995801+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:19.995706+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:20.236979+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:16.689378+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.976207+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:24:29.693 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:24:29.946 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:24:29.946 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":114,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:24:25.995092+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:18.736580+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.541121+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.995801+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:19.995706+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:20.236979+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:16.689378+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.976207+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:24:32.961 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:24:33.218 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:24:33.218 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":114,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:24:25.995092+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:18.736580+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.541121+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.995801+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:19.995706+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:20.236979+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:16.689378+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.976207+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:24:36.232 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:24:36.488 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:24:36.488 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":114,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:24:25.995092+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:18.736580+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.541121+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.995801+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:19.995706+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:20.236979+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:16.689378+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.976207+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:24:39.508 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:24:39.761 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:24:39.762 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":114,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:24:25.995092+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:18.736580+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.541121+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.995801+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:19.995706+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:20.236979+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:16.689378+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.976207+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:24:42.776 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:24:43.033 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:24:43.033 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":114,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:24:25.995092+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:18.736580+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.541121+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.995801+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:19.995706+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:20.236979+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:16.689378+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.976207+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:24:46.047 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:24:46.282 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:24:46.282 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":114,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:24:25.995092+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:18.736580+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.541121+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.995801+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:19.995706+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:20.236979+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:16.689378+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.976207+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:24:49.295 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:24:49.546 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:24:49.546 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":114,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:24:25.995092+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:18.736580+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.541121+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.995801+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:19.995706+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:20.236979+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:16.689378+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.976207+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:24:52.560 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:24:52.822 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:24:52.822 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":114,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:24:25.995092+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:18.736580+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.541121+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.995801+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:19.995706+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:20.236979+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:16.689378+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.976207+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:24:55.840 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:24:56.100 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:24:56.100 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":114,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:24:25.995092+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:18.736580+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.541121+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.995801+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:19.995706+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:20.236979+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:16.689378+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.976207+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:24:59.114 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:24:59.366 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:24:59.366 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":114,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:24:25.995092+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:18.736580+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.541121+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.995801+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:19.995706+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:20.236979+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:16.689378+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.976207+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:25:02.386 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:25:02.638 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:25:02.638 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":114,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:24:25.995092+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:18.736580+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.541121+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.995801+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:19.995706+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:20.236979+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:16.689378+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.976207+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:25:05.656 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:25:05.910 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:25:05.911 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":114,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:24:25.995092+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:18.736580+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.541121+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.995801+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:19.995706+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:20.236979+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:16.689378+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.976207+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:25:08.925 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:25:09.177 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:25:09.177 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":114,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:24:25.995092+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:18.736580+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.541121+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.995801+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:19.995706+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:20.236979+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:16.689378+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.976207+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:25:12.191 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:25:12.449 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:25:12.449 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":114,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:24:25.995092+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:18.736580+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.541121+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.995801+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:19.995706+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:20.236979+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:16.689378+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.976207+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:25:15.463 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:25:15.718 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:25:15.718 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":114,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:24:25.995092+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:18.736580+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.541121+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.995801+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:19.995706+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:20.236979+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:16.689378+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.976207+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:25:18.732 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:25:18.982 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:25:18.982 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":115,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:25:18.024053+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:18.736580+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:17.719143+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.995801+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:19.995706+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:20.236979+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:17.349649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:24:17.976207+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:25:21.996 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:25:22.251 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:25:22.252 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":118,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:25:21.014471+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:19.430159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:17.719143+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:18.208469+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:20.407882+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:20.348716+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:17.349649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:18.705704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:25:25.268 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:25:25.521 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:25:25.521 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":118,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:25:21.014471+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:19.430159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:17.719143+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:18.208469+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:20.407882+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:20.348716+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:17.349649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:18.705704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:25:28.542 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:25:28.805 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:25:28.805 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":118,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:25:21.014471+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:19.430159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:17.719143+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:18.208469+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:20.407882+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:20.348716+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:17.349649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:18.705704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:25:31.820 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:25:32.073 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:25:32.073 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":118,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:25:21.014471+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:19.430159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:17.719143+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:18.208469+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:20.407882+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:20.348716+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:17.349649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:18.705704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:25:35.087 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:25:35.336 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:25:35.336 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":118,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:25:21.014471+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:19.430159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:17.719143+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:18.208469+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:20.407882+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:20.348716+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:17.349649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:18.705704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:25:38.350 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:25:38.601 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:25:38.602 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":118,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:25:21.014471+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:19.430159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:17.719143+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:18.208469+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:20.407882+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:20.348716+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:17.349649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:18.705704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:25:41.615 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:25:41.897 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:25:41.898 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":118,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:25:21.014471+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:19.430159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:17.719143+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:18.208469+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:20.407882+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:20.348716+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:17.349649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:18.705704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:25:44.914 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:25:45.162 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:25:45.162 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":118,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:25:21.014471+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:19.430159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:17.719143+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:18.208469+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:20.407882+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:20.348716+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:17.349649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:18.705704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:25:48.177 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:25:48.427 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:25:48.427 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":118,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:25:21.014471+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:19.430159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:17.719143+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:18.208469+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:20.407882+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:20.348716+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:17.349649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:18.705704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:25:51.441 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:25:51.700 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:25:51.701 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":118,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:25:21.014471+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:19.430159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:17.719143+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:18.208469+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:20.407882+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:20.348716+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:17.349649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:18.705704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:25:54.714 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:25:54.969 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:25:54.969 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":118,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:25:21.014471+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:19.430159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:17.719143+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:18.208469+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:20.407882+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:20.348716+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:17.349649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:18.705704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:25:57.982 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:25:58.234 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:25:58.234 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":118,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:25:21.014471+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:19.430159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:17.719143+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:18.208469+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:20.407882+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:20.348716+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:17.349649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:18.705704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:26:01.254 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:26:01.506 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:26:01.506 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":118,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:25:21.014471+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:19.430159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:17.719143+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:18.208469+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:20.407882+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:20.348716+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:17.349649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:18.705704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:26:04.520 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:26:04.770 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:26:04.771 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":118,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:25:21.014471+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:19.430159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:17.719143+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:18.208469+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:20.407882+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:20.348716+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:17.349649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:18.705704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:26:07.786 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:26:08.046 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:26:08.046 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":118,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:25:21.014471+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:19.430159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:17.719143+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:18.208469+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:20.407882+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:20.348716+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:17.349649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:18.705704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:26:11.071 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:26:11.321 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:26:11.322 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":118,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:25:21.014471+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:19.430159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:17.719143+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:18.208469+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:20.407882+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:20.348716+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:17.349649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:18.705704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:26:14.336 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:26:14.589 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:26:14.590 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":118,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:25:21.014471+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:19.430159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:17.719143+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:18.208469+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:20.407882+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:20.348716+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:17.349649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:18.705704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:26:17.604 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:26:17.859 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:26:17.859 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":118,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:25:21.014471+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:19.430159+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:17.719143+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:18.208469+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:20.407882+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:20.348716+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:17.349649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:18.705704+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:26:20.874 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:26:21.131 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:26:21.132 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":122,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:26:21.035465+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:20.244302+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:18.622585+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:18.608632+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:20.541066+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:25:20.348716+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:17.734777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:19.522294+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:26:24.145 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:26:24.396 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:26:24.396 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":123,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:26:22.083506+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:20.244302+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:18.622585+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:18.608632+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:20.541066+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:21.295233+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:17.734777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:19.522294+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:26:27.410 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:26:27.661 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:26:27.661 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":124,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:26:26.037479+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:20.244302+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:18.622585+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:18.608632+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:20.541066+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:21.295233+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:17.734777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:19.522294+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:26:30.674 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:26:30.926 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:26:30.926 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":124,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:26:26.037479+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:20.244302+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:18.622585+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:18.608632+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:20.541066+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:21.295233+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:17.734777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:19.522294+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:26:33.938 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:26:34.191 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:26:34.191 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":124,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:26:26.037479+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:20.244302+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:18.622585+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:18.608632+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:20.541066+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:21.295233+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:17.734777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:19.522294+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:26:37.206 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:26:37.458 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:26:37.458 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":124,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:26:26.037479+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:20.244302+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:18.622585+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:18.608632+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:20.541066+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:21.295233+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:17.734777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:19.522294+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:26:40.472 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:26:40.722 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:26:40.722 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":124,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:26:26.037479+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:20.244302+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:18.622585+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:18.608632+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:20.541066+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:21.295233+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:17.734777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:19.522294+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:26:43.736 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:26:44.009 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:26:44.010 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":124,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:26:26.037479+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:20.244302+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:18.622585+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:18.608632+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:20.541066+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:21.295233+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:17.734777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:19.522294+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:26:47.025 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:26:47.277 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:26:47.277 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":124,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:26:26.037479+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:20.244302+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:18.622585+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:18.608632+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:20.541066+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:21.295233+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:17.734777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:19.522294+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:26:50.295 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:26:50.550 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:26:50.551 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":124,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:26:26.037479+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:20.244302+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:18.622585+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:18.608632+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:20.541066+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:21.295233+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:17.734777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:19.522294+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:26:53.564 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:26:53.821 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:26:53.821 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":124,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:26:26.037479+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:20.244302+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:18.622585+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:18.608632+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:20.541066+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:21.295233+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:17.734777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:19.522294+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:26:56.837 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:26:57.093 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:26:57.094 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":124,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:26:26.037479+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:20.244302+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:18.622585+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:18.608632+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:20.541066+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:21.295233+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:17.734777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:19.522294+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:27:00.107 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:27:00.360 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:27:00.360 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":124,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:26:26.037479+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:20.244302+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:18.622585+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:18.608632+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:20.541066+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:21.295233+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:17.734777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:19.522294+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:27:03.374 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:27:03.627 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:27:03.628 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":124,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:26:26.037479+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:20.244302+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:18.622585+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:18.608632+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:20.541066+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:21.295233+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:17.734777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:19.522294+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:27:06.642 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:27:06.898 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:27:06.899 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":124,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:26:26.037479+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:20.244302+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:18.622585+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:18.608632+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:20.541066+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:21.295233+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:17.734777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:19.522294+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:27:09.912 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:27:10.166 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:27:10.166 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":124,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:26:26.037479+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:20.244302+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:18.622585+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:18.608632+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:20.541066+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:21.295233+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:17.734777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:19.522294+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:27:13.179 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:27:13.431 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:27:13.432 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":124,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:26:26.037479+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:20.244302+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:18.622585+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:18.608632+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:20.541066+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:21.295233+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:17.734777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:19.522294+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:27:16.446 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:27:16.698 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:27:16.699 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":124,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:26:26.037479+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:20.244302+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:18.622585+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:18.608632+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:20.541066+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:21.295233+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:17.734777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:19.522294+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:27:19.717 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:27:19.972 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:27:19.972 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":126,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:27:19.069688+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:20.244302+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:18.801341+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:18.703320+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:20.541066+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:21.295233+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:17.865125+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:26:19.522294+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:27:22.986 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:27:23.241 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:27:23.242 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":129,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:27:23.105970+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:21.188734+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:18.801341+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:18.703320+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:20.632263+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:22.298626+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:17.865125+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:20.489279+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:27:26.255 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:27:26.511 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:27:26.512 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":130,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:27:26.058653+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:21.188734+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:18.801341+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:18.703320+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:20.632263+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:22.298626+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:17.865125+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:20.489279+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:27:29.529 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:27:29.785 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:27:29.785 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":130,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:27:26.058653+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:21.188734+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:18.801341+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:18.703320+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:20.632263+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:22.298626+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:17.865125+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:20.489279+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:27:32.799 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:27:33.054 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:27:33.055 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":130,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:27:26.058653+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:21.188734+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:18.801341+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:18.703320+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:20.632263+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:22.298626+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:17.865125+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:20.489279+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:27:36.069 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:27:36.323 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:27:36.323 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":130,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:27:26.058653+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:21.188734+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:18.801341+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:18.703320+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:20.632263+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:22.298626+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:17.865125+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:20.489279+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:27:39.344 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:27:39.590 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:27:39.591 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":130,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:27:26.058653+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:21.188734+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:18.801341+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:18.703320+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:20.632263+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:22.298626+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:17.865125+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:20.489279+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:27:42.611 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:27:42.867 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:27:42.868 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":130,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:27:26.058653+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:21.188734+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:18.801341+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:18.703320+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:20.632263+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:22.298626+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:17.865125+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:20.489279+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:27:45.888 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:27:46.142 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:27:46.143 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":130,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:27:26.058653+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:21.188734+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:18.801341+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:18.703320+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:20.632263+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:22.298626+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:17.865125+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:20.489279+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:27:49.161 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:27:49.414 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:27:49.414 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":130,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:27:26.058653+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:21.188734+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:18.801341+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:18.703320+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:20.632263+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:22.298626+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:17.865125+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:20.489279+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:27:52.428 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:27:52.685 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:27:52.686 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":130,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:27:26.058653+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:21.188734+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:18.801341+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:18.703320+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:20.632263+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:22.298626+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:17.865125+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:20.489279+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:27:55.699 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:27:55.949 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:27:55.950 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":130,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:27:26.058653+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:21.188734+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:18.801341+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:18.703320+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:20.632263+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:22.298626+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:17.865125+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:20.489279+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:27:58.964 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:27:59.234 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:27:59.235 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":130,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:27:26.058653+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:21.188734+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:18.801341+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:18.703320+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:20.632263+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:22.298626+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:17.865125+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:20.489279+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:28:02.250 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:28:02.508 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:28:02.508 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":130,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:27:26.058653+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:21.188734+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:18.801341+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:18.703320+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:20.632263+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:22.298626+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:17.865125+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:20.489279+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:28:05.522 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:28:05.767 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:28:05.768 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":130,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:27:26.058653+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:21.188734+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:18.801341+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:18.703320+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:20.632263+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:22.298626+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:17.865125+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:20.489279+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:28:08.788 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:28:09.042 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:28:09.043 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":130,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:27:26.058653+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:21.188734+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:18.801341+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:18.703320+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:20.632263+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:22.298626+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:17.865125+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:20.489279+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:28:12.061 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:28:12.311 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:28:12.311 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":130,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:27:26.058653+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:21.188734+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:18.801341+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:18.703320+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:20.632263+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:22.298626+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:17.865125+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:20.489279+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:28:15.326 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:28:15.578 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:28:15.578 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":130,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:27:26.058653+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:21.188734+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:18.801341+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:18.703320+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:20.632263+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:22.298626+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:17.865125+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:20.489279+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:28:18.592 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:28:18.846 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:28:18.846 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":130,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:27:26.058653+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:21.188734+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:18.801341+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:18.703320+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:20.632263+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:22.298626+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:17.865125+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:20.489279+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:28:21.864 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:28:22.117 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:28:22.117 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":133,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:28:22.090624+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:22.024724+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:19.395137+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:19.696201+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:21.256322+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:27:22.298626+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:18.709152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:21.147953+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:28:25.132 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:28:25.387 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:28:25.387 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":134,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:28:23.374958+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:22.024724+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:19.395137+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:19.696201+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:21.256322+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:22.679968+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:18.709152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:21.147953+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:28:28.400 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:28:28.647 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:28:28.647 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":134,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:28:23.374958+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:22.024724+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:19.395137+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:19.696201+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:21.256322+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:22.679968+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:18.709152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:21.147953+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:28:31.660 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:28:31.939 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:28:31.939 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":134,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:28:23.374958+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:22.024724+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:19.395137+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:19.696201+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:21.256322+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:22.679968+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:18.709152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:21.147953+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:28:34.957 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:28:35.214 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:28:35.214 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":134,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:28:23.374958+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:22.024724+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:19.395137+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:19.696201+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:21.256322+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:22.679968+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:18.709152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:21.147953+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:28:38.228 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:28:38.484 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:28:38.484 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":134,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:28:23.374958+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:22.024724+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:19.395137+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:19.696201+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:21.256322+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:22.679968+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:18.709152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:21.147953+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:28:41.498 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:28:41.751 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:28:41.751 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":134,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:28:23.374958+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:22.024724+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:19.395137+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:19.696201+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:21.256322+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:22.679968+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:18.709152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:21.147953+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:28:44.765 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:28:45.026 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:28:45.027 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":134,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:28:23.374958+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:22.024724+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:19.395137+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:19.696201+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:21.256322+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:22.679968+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:18.709152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:21.147953+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:28:48.046 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:28:48.295 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:28:48.295 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":134,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:28:23.374958+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:22.024724+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:19.395137+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:19.696201+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:21.256322+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:22.679968+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:18.709152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:21.147953+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:28:51.309 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:28:51.566 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:28:51.567 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":134,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:28:23.374958+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:22.024724+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:19.395137+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:19.696201+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:21.256322+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:22.679968+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:18.709152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:21.147953+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:28:54.581 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:28:54.837 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:28:54.837 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":134,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:28:23.374958+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:22.024724+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:19.395137+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:19.696201+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:21.256322+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:22.679968+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:18.709152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:21.147953+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:28:57.851 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:28:58.126 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:28:58.126 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":134,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:28:23.374958+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:22.024724+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:19.395137+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:19.696201+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:21.256322+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:22.679968+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:18.709152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:21.147953+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:29:01.132 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:29:01.380 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:29:01.380 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":134,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:28:23.374958+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:22.024724+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:19.395137+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:19.696201+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:21.256322+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:22.679968+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:18.709152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:21.147953+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:29:04.394 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:29:04.650 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:29:04.650 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":134,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:28:23.374958+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:22.024724+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:19.395137+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:19.696201+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:21.256322+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:22.679968+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:18.709152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:21.147953+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:29:07.668 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:29:07.930 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:29:07.930 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":134,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:28:23.374958+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:22.024724+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:19.395137+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:19.696201+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:21.256322+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:22.679968+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:18.709152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:21.147953+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:29:10.944 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:29:11.204 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:29:11.204 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":134,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:28:23.374958+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:22.024724+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:19.395137+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:19.696201+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:21.256322+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:22.679968+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:18.709152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:21.147953+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:29:14.217 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:29:14.473 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:29:14.474 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":134,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:28:23.374958+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:22.024724+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:19.395137+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:19.696201+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:21.256322+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:22.679968+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:18.709152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:21.147953+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:29:17.488 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:29:17.744 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:29:17.745 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":134,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:28:23.374958+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:22.024724+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:19.395137+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:19.696201+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:21.256322+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:22.679968+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:18.709152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:21.147953+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:29:20.765 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:29:21.013 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:29:21.013 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":135,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:29:20.118865+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:22.024724+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.400213+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.899315+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:21.256322+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:22.679968+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.493152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:28:21.147953+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:29:24.027 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:29:24.282 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:29:24.282 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":138,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:29:23.138448+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:22.853508+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.400213+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.899315+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:22.239819+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:23.026148+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.493152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:21.422362+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:29:27.296 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:29:27.562 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:29:27.562 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":138,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:29:23.138448+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:22.853508+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.400213+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.899315+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:22.239819+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:23.026148+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.493152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:21.422362+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:29:30.576 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:29:30.827 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:29:30.827 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":138,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:29:23.138448+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:22.853508+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.400213+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.899315+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:22.239819+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:23.026148+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.493152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:21.422362+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:29:33.840 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:29:34.092 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:29:34.093 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":138,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:29:23.138448+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:22.853508+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.400213+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.899315+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:22.239819+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:23.026148+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.493152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:21.422362+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:29:37.112 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:29:37.370 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:29:37.370 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":138,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:29:23.138448+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:22.853508+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.400213+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.899315+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:22.239819+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:23.026148+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.493152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:21.422362+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:29:40.386 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:29:40.638 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:29:40.638 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":138,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:29:23.138448+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:22.853508+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.400213+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.899315+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:22.239819+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:23.026148+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.493152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:21.422362+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:29:43.647 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:29:43.898 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:29:43.898 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":138,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:29:23.138448+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:22.853508+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.400213+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.899315+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:22.239819+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:23.026148+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.493152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:21.422362+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:29:46.916 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:29:47.172 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:29:47.172 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":138,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:29:23.138448+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:22.853508+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.400213+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.899315+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:22.239819+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:23.026148+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.493152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:21.422362+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:29:50.187 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:29:50.442 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:29:50.442 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":138,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:29:23.138448+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:22.853508+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.400213+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.899315+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:22.239819+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:23.026148+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.493152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:21.422362+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:29:53.456 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:29:53.724 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:29:53.725 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":138,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:29:23.138448+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:22.853508+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.400213+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.899315+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:22.239819+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:23.026148+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.493152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:21.422362+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:29:56.733 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:29:56.989 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:29:56.990 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":138,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:29:23.138448+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:22.853508+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.400213+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.899315+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:22.239819+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:23.026148+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.493152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:21.422362+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:30:00.006 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:30:00.254 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:30:00.254 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":138,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:29:23.138448+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:22.853508+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.400213+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.899315+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:22.239819+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:23.026148+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.493152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:21.422362+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:30:03.268 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:30:03.524 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:30:03.527 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":138,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:29:23.138448+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:22.853508+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.400213+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.899315+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:22.239819+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:23.026148+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.493152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:21.422362+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:30:06.539 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:30:06.800 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:30:06.800 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":138,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:29:23.138448+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:22.853508+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.400213+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.899315+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:22.239819+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:23.026148+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.493152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:21.422362+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:30:09.814 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:30:10.071 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:30:10.071 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":138,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:29:23.138448+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:22.853508+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.400213+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.899315+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:22.239819+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:23.026148+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.493152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:21.422362+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:30:13.086 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:30:13.346 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:30:13.346 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":138,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:29:23.138448+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:22.853508+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.400213+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.899315+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:22.239819+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:23.026148+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.493152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:21.422362+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:30:16.359 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:30:16.619 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:30:16.620 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":138,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:29:23.138448+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:22.853508+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.400213+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.899315+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:22.239819+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:23.026148+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.493152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:21.422362+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:30:19.630 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:30:19.914 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:30:19.915 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":138,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:29:23.138448+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:22.853508+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.400213+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.899315+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:22.239819+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:23.026148+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:19.493152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:21.422362+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:30:22.923 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:30:23.180 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:30:23.180 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":141,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:30:22.957966+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:22.853508+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.115643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.678797+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:22.915453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:29:23.026148+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.252476+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:22.315341+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:30:26.195 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:30:26.452 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:30:26.452 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":142,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:30:23.967646+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:23.554934+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.115643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.678797+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:22.915453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:23.070978+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.252476+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:22.315341+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:30:29.466 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:30:29.716 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:30:29.716 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":142,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:30:23.967646+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:23.554934+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.115643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.678797+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:22.915453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:23.070978+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.252476+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:22.315341+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:30:32.730 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:30:32.988 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:30:32.989 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":142,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:30:23.967646+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:23.554934+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.115643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.678797+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:22.915453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:23.070978+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.252476+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:22.315341+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:30:36.007 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:30:36.270 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:30:36.270 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":142,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:30:23.967646+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:23.554934+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.115643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.678797+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:22.915453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:23.070978+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.252476+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:22.315341+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:30:39.290 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:30:39.547 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:30:39.547 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":142,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:30:23.967646+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:23.554934+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.115643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.678797+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:22.915453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:23.070978+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.252476+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:22.315341+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:30:42.561 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:30:42.817 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:30:42.817 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":142,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:30:23.967646+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:23.554934+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.115643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.678797+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:22.915453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:23.070978+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.252476+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:22.315341+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:30:45.830 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:30:46.088 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:30:46.088 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":142,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:30:23.967646+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:23.554934+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.115643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.678797+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:22.915453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:23.070978+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.252476+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:22.315341+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:30:49.102 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:30:49.356 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:30:49.357 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":142,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:30:23.967646+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:23.554934+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.115643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.678797+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:22.915453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:23.070978+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.252476+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:22.315341+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:30:52.380 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:30:52.631 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:30:52.631 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":142,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:30:23.967646+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:23.554934+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.115643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.678797+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:22.915453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:23.070978+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.252476+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:22.315341+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:30:55.645 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:30:55.894 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:30:55.895 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":142,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:30:23.967646+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:23.554934+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.115643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.678797+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:22.915453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:23.070978+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.252476+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:22.315341+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:30:58.909 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:30:59.168 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:30:59.168 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":142,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:30:23.967646+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:23.554934+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.115643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.678797+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:22.915453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:23.070978+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.252476+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:22.315341+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:31:02.182 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:31:02.435 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:31:02.435 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":142,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:30:23.967646+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:23.554934+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.115643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.678797+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:22.915453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:23.070978+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.252476+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:22.315341+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:31:05.450 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:31:05.700 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:31:05.700 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":142,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:30:23.967646+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:23.554934+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.115643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.678797+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:22.915453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:23.070978+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.252476+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:22.315341+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:31:08.713 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:31:08.958 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:31:08.958 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":142,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:30:23.967646+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:23.554934+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.115643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.678797+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:22.915453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:23.070978+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.252476+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:22.315341+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:31:11.972 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:31:12.229 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:31:12.229 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":142,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:30:23.967646+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:23.554934+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.115643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.678797+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:22.915453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:23.070978+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.252476+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:22.315341+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:31:15.248 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:31:15.502 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:31:15.503 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":142,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:30:23.967646+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:23.554934+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.115643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.678797+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:22.915453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:23.070978+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.252476+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:22.315341+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:31:18.516 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:31:18.771 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:31:18.772 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":142,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:30:23.967646+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:23.554934+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.115643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.678797+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:22.915453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:23.070978+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.252476+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:22.315341+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:31:21.792 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:31:22.050 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:31:22.050 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":144,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:31:21.153479+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:23.554934+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:20.180112+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:20.678797+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:22.915453+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:23.070978+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:21.129202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:30:22.315341+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:31:25.070 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:31:25.323 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:31:25.324 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":147,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:31:24.456769+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.968697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:20.180112+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:21.574002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:22.916880+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.781947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:21.129202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.328874+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:31:28.338 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:31:28.593 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:31:28.593 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":148,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:31:26.156879+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.968697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:20.180112+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:21.574002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:22.916880+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.781947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:21.129202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.328874+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:31:31.612 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:31:31.870 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:31:31.871 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":148,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:31:26.156879+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.968697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:20.180112+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:21.574002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:22.916880+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.781947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:21.129202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.328874+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:31:34.885 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:31:35.142 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:31:35.143 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":148,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:31:26.156879+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.968697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:20.180112+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:21.574002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:22.916880+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.781947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:21.129202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.328874+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:31:38.157 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:31:38.413 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:31:38.414 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":148,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:31:26.156879+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.968697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:20.180112+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:21.574002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:22.916880+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.781947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:21.129202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.328874+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:31:41.429 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:31:41.683 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:31:41.683 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":148,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:31:26.156879+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.968697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:20.180112+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:21.574002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:22.916880+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.781947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:21.129202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.328874+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:31:44.697 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:31:44.951 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:31:44.951 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":148,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:31:26.156879+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.968697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:20.180112+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:21.574002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:22.916880+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.781947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:21.129202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.328874+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:31:47.966 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:31:48.218 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:31:48.218 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":148,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:31:26.156879+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.968697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:20.180112+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:21.574002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:22.916880+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.781947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:21.129202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.328874+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:31:51.231 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:31:51.488 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:31:51.488 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":148,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:31:26.156879+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.968697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:20.180112+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:21.574002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:22.916880+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.781947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:21.129202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.328874+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:31:54.503 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:31:54.757 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:31:54.757 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":148,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:31:26.156879+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.968697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:20.180112+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:21.574002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:22.916880+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.781947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:21.129202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.328874+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:31:57.771 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:31:58.026 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:31:58.027 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":148,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:31:26.156879+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.968697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:20.180112+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:21.574002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:22.916880+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.781947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:21.129202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.328874+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:32:01.041 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:32:01.292 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:32:01.292 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":148,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:31:26.156879+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.968697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:20.180112+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:21.574002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:22.916880+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.781947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:21.129202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.328874+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:32:04.305 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:32:04.563 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:32:04.563 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":148,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:31:26.156879+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.968697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:20.180112+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:21.574002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:22.916880+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.781947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:21.129202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.328874+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:32:07.583 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:32:07.834 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:32:07.834 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":148,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:31:26.156879+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.968697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:20.180112+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:21.574002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:22.916880+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.781947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:21.129202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.328874+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:32:10.848 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:32:11.100 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:32:11.100 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":148,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:31:26.156879+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.968697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:20.180112+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:21.574002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:22.916880+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.781947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:21.129202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.328874+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:32:14.114 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:32:14.371 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:32:14.371 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":148,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:31:26.156879+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.968697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:20.180112+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:21.574002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:22.916880+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.781947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:21.129202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.328874+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:32:17.386 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:32:17.639 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:32:17.640 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":148,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:31:26.156879+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.968697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:20.180112+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:21.574002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:22.916880+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.781947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:21.129202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.328874+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:32:20.654 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:32:20.906 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:32:20.907 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":149,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:32:20.344728+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.968697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:20.293436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:21.574002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:22.916880+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.781947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:21.129202+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.328874+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:32:23.920 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:32:24.182 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:32:24.183 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":151,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:32:22.356329+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.968697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:20.293436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:21.841587+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:22.916880+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.781947+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:21.635923+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:31:23.328874+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:32:27.197 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:32:27.449 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:32:27.449 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":154,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:32:26.181329+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.581586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:20.293436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:21.841587+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:23.775569+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.664060+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:21.635923+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.136424+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:32:30.463 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:32:30.715 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:32:30.715 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":154,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:32:26.181329+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.581586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:20.293436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:21.841587+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:23.775569+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.664060+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:21.635923+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.136424+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:32:33.729 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:32:33.984 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:32:33.985 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":154,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:32:26.181329+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.581586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:20.293436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:21.841587+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:23.775569+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.664060+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:21.635923+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.136424+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:32:36.998 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:32:37.255 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:32:37.255 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":154,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:32:26.181329+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.581586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:20.293436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:21.841587+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:23.775569+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.664060+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:21.635923+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.136424+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:32:40.269 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:32:40.521 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:32:40.522 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":154,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:32:26.181329+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.581586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:20.293436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:21.841587+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:23.775569+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.664060+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:21.635923+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.136424+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:32:43.536 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:32:43.790 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:32:43.790 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":154,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:32:26.181329+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.581586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:20.293436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:21.841587+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:23.775569+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.664060+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:21.635923+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.136424+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:32:46.803 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:32:47.056 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:32:47.056 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":154,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:32:26.181329+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.581586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:20.293436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:21.841587+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:23.775569+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.664060+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:21.635923+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.136424+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:32:50.076 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:32:50.317 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:32:50.318 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":154,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:32:26.181329+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.581586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:20.293436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:21.841587+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:23.775569+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.664060+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:21.635923+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.136424+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:32:53.338 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:32:53.595 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:32:53.596 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":154,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:32:26.181329+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.581586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:20.293436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:21.841587+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:23.775569+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.664060+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:21.635923+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.136424+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:32:56.610 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:32:56.866 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:32:56.867 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":154,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:32:26.181329+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.581586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:20.293436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:21.841587+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:23.775569+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.664060+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:21.635923+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.136424+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:32:59.881 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:33:00.131 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:33:00.131 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":154,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:32:26.181329+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.581586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:20.293436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:21.841587+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:23.775569+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.664060+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:21.635923+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.136424+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:33:03.146 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:33:03.399 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:33:03.400 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":154,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:32:26.181329+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.581586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:20.293436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:21.841587+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:23.775569+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.664060+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:21.635923+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.136424+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:33:06.414 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:33:06.676 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:33:06.677 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":154,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:32:26.181329+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.581586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:20.293436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:21.841587+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:23.775569+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.664060+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:21.635923+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.136424+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:33:09.699 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:33:09.956 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:33:09.957 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":154,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:32:26.181329+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.581586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:20.293436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:21.841587+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:23.775569+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.664060+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:21.635923+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.136424+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:33:12.971 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:33:13.225 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:33:13.226 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":154,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:32:26.181329+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.581586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:20.293436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:21.841587+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:23.775569+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.664060+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:21.635923+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.136424+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:33:16.238 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:33:16.481 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:33:16.481 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":154,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:32:26.181329+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.581586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:20.293436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:21.841587+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:23.775569+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.664060+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:21.635923+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.136424+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:33:19.498 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:33:19.749 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:33:19.750 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":154,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:32:26.181329+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.581586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:20.293436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:21.841587+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:23.775569+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.664060+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:21.635923+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.136424+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:33:22.767 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:33:23.020 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:33:23.021 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":157,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:33:22.221829+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.581586+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:20.423738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:21.841587+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:23.775569+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.664060+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:21.703570+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:32:24.136424+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:33:26.041 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:33:26.298 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:33:26.299 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":161,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:33:26.257374+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:25.434739+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:20.423738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:22.830985+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:23.896805+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:24.924740+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:21.703570+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:25.021919+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:33:29.317 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:33:29.574 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:33:29.575 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":161,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:33:26.257374+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:25.434739+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:20.423738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:22.830985+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:23.896805+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:24.924740+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:21.703570+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:25.021919+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:33:32.589 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:33:32.840 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:33:32.841 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":162,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:33:31.208601+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:25.434739+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:20.423738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:22.830985+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:23.896805+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:24.924740+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:21.703570+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:25.021919+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:33:35.854 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:33:36.118 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:33:36.119 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":162,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:33:31.208601+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:25.434739+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:20.423738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:22.830985+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:23.896805+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:24.924740+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:21.703570+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:25.021919+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:33:39.144 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:33:39.400 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:33:39.400 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":162,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:33:31.208601+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:25.434739+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:20.423738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:22.830985+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:23.896805+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:24.924740+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:21.703570+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:25.021919+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:33:42.414 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:33:42.665 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:33:42.665 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":162,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:33:31.208601+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:25.434739+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:20.423738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:22.830985+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:23.896805+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:24.924740+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:21.703570+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:25.021919+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:33:45.680 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:33:45.934 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:33:45.935 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":162,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:33:31.208601+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:25.434739+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:20.423738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:22.830985+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:23.896805+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:24.924740+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:21.703570+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:25.021919+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:33:48.946 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:33:49.197 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:33:49.197 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":162,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:33:31.208601+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:25.434739+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:20.423738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:22.830985+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:23.896805+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:24.924740+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:21.703570+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:25.021919+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:33:52.211 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:33:52.459 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:33:52.460 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":162,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:33:31.208601+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:25.434739+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:20.423738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:22.830985+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:23.896805+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:24.924740+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:21.703570+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:25.021919+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:33:55.474 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:33:55.724 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:33:55.725 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":162,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:33:31.208601+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:25.434739+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:20.423738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:22.830985+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:23.896805+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:24.924740+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:21.703570+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:25.021919+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:33:58.739 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:33:58.987 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:33:58.988 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":162,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:33:31.208601+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:25.434739+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:20.423738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:22.830985+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:23.896805+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:24.924740+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:21.703570+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:25.021919+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:34:02.001 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:34:02.253 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:34:02.254 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":162,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:33:31.208601+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:25.434739+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:20.423738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:22.830985+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:23.896805+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:24.924740+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:21.703570+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:25.021919+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:34:05.268 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:34:05.528 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:34:05.528 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":162,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:33:31.208601+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:25.434739+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:20.423738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:22.830985+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:23.896805+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:24.924740+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:21.703570+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:25.021919+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:34:08.541 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:34:08.797 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:34:08.798 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":162,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:33:31.208601+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:25.434739+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:20.423738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:22.830985+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:23.896805+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:24.924740+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:21.703570+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:25.021919+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:34:11.812 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:34:12.057 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:34:12.057 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":162,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:33:31.208601+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:25.434739+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:20.423738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:22.830985+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:23.896805+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:24.924740+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:21.703570+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:25.021919+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:34:15.071 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:34:15.326 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:34:15.327 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":162,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:33:31.208601+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:25.434739+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:20.423738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:22.830985+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:23.896805+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:24.924740+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:21.703570+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:25.021919+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:34:18.341 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:34:18.600 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:34:18.600 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":162,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:33:31.208601+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:25.434739+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:20.423738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:22.830985+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:23.896805+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:24.924740+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:21.703570+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:25.021919+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:34:21.621 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:34:21.881 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:34:21.882 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":163,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:34:21.488793+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:25.434739+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:21.221524+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:22.830985+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:23.896805+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:24.924740+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:21.703570+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:25.021919+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:34:24.899 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:34:25.155 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:34:25.155 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":165,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:34:24.249924+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:25.434739+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:21.221524+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:23.365647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:24.041927+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:24.924740+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:22.598493+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:33:25.021919+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:34:28.168 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:34:28.423 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:34:28.424 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":167,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:34:26.266822+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:25.619278+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:21.221524+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:23.365647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:24.041927+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:24.991728+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:22.598493+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:25.577232+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:34:31.438 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:34:31.690 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:34:31.690 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":168,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:34:31.233035+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:25.619278+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:21.221524+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:23.365647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:24.041927+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:24.991728+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:22.598493+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:25.577232+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:34:34.705 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:34:34.952 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:34:34.952 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":168,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:34:31.233035+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:25.619278+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:21.221524+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:23.365647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:24.041927+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:24.991728+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:22.598493+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:25.577232+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:34:37.966 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:34:38.230 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:34:38.230 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":168,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:34:31.233035+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:25.619278+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:21.221524+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:23.365647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:24.041927+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:24.991728+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:22.598493+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:25.577232+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:34:41.244 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:34:41.489 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:34:41.489 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":168,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:34:31.233035+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:25.619278+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:21.221524+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:23.365647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:24.041927+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:24.991728+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:22.598493+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:25.577232+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:34:44.503 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:34:44.759 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:34:44.760 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":168,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:34:31.233035+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:25.619278+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:21.221524+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:23.365647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:24.041927+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:24.991728+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:22.598493+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:25.577232+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:34:47.780 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:34:48.035 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:34:48.036 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":168,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:34:31.233035+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:25.619278+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:21.221524+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:23.365647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:24.041927+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:24.991728+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:22.598493+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:25.577232+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:34:51.050 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:34:51.299 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:34:51.299 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":168,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:34:31.233035+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:25.619278+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:21.221524+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:23.365647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:24.041927+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:24.991728+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:22.598493+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:25.577232+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:34:54.317 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:34:54.573 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:34:54.574 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":168,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:34:31.233035+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:25.619278+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:21.221524+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:23.365647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:24.041927+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:24.991728+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:22.598493+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:25.577232+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:34:57.587 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:34:57.843 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:34:57.843 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":168,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:34:31.233035+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:25.619278+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:21.221524+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:23.365647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:24.041927+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:24.991728+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:22.598493+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:25.577232+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:35:00.862 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:35:01.123 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:35:01.123 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":168,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:34:31.233035+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:25.619278+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:21.221524+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:23.365647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:24.041927+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:24.991728+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:22.598493+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:25.577232+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:35:04.138 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:35:04.392 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:35:04.392 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":168,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:34:31.233035+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:25.619278+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:21.221524+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:23.365647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:24.041927+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:24.991728+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:22.598493+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:25.577232+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:35:07.405 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:35:07.651 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:35:07.651 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":168,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:34:31.233035+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:25.619278+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:21.221524+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:23.365647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:24.041927+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:24.991728+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:22.598493+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:25.577232+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:35:10.665 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:35:10.919 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:35:10.919 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":168,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:34:31.233035+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:25.619278+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:21.221524+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:23.365647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:24.041927+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:24.991728+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:22.598493+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:25.577232+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:35:13.933 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:35:14.186 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:35:14.187 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":168,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:34:31.233035+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:25.619278+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:21.221524+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:23.365647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:24.041927+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:24.991728+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:22.598493+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:25.577232+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:35:17.201 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:35:17.457 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:35:17.458 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":168,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:34:31.233035+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:25.619278+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:21.221524+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:23.365647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:24.041927+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:24.991728+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:22.598493+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:25.577232+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:35:20.472 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:35:20.728 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:35:20.728 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":168,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:34:31.233035+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:25.619278+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:21.221524+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:23.365647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:24.041927+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:24.991728+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:22.598493+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:25.577232+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:35:23.742 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:35:24.000 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:35:24.000 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":170,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:35:23.512142+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:25.619278+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:21.260068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:23.365647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:24.041927+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:24.991728+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:22.632932+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:34:25.577232+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:35:27.015 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:35:27.268 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:35:27.269 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":173,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:35:26.535319+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.708195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:21.260068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:24.289309+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:24.821576+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.449685+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:22.632932+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.849404+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:35:30.286 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:35:30.546 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:35:30.547 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":173,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:35:26.535319+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.708195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:21.260068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:24.289309+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:24.821576+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.449685+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:22.632932+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.849404+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:35:33.565 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:35:33.819 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:35:33.819 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":174,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:35:31.256349+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.708195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:21.260068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:24.289309+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:24.821576+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.449685+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:22.632932+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.849404+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:35:36.833 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:35:37.087 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:35:37.088 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":174,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:35:31.256349+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.708195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:21.260068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:24.289309+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:24.821576+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.449685+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:22.632932+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.849404+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:35:40.101 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:35:40.356 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:35:40.356 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":174,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:35:31.256349+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.708195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:21.260068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:24.289309+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:24.821576+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.449685+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:22.632932+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.849404+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:35:43.376 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:35:43.642 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:35:43.642 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":174,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:35:31.256349+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.708195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:21.260068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:24.289309+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:24.821576+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.449685+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:22.632932+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.849404+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:35:46.660 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:35:46.914 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:35:46.915 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":174,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:35:31.256349+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.708195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:21.260068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:24.289309+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:24.821576+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.449685+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:22.632932+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.849404+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:35:49.934 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:35:50.189 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:35:50.190 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":174,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:35:31.256349+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.708195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:21.260068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:24.289309+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:24.821576+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.449685+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:22.632932+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.849404+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:35:53.204 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:35:53.458 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:35:53.459 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":174,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:35:31.256349+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.708195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:21.260068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:24.289309+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:24.821576+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.449685+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:22.632932+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.849404+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:35:56.484 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:35:56.740 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:35:56.740 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":174,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:35:31.256349+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.708195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:21.260068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:24.289309+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:24.821576+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.449685+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:22.632932+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.849404+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:35:59.754 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:36:00.012 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:36:00.012 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":174,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:35:31.256349+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.708195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:21.260068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:24.289309+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:24.821576+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.449685+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:22.632932+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.849404+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:36:03.027 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:36:03.284 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:36:03.284 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":174,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:35:31.256349+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.708195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:21.260068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:24.289309+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:24.821576+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.449685+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:22.632932+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.849404+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:36:06.310 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:36:06.556 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:36:06.556 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":174,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:35:31.256349+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.708195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:21.260068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:24.289309+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:24.821576+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.449685+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:22.632932+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.849404+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:36:09.570 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:36:09.828 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:36:09.828 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":174,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:35:31.256349+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.708195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:21.260068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:24.289309+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:24.821576+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.449685+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:22.632932+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.849404+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:36:12.843 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:36:13.095 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:36:13.095 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":174,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:35:31.256349+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.708195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:21.260068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:24.289309+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:24.821576+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.449685+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:22.632932+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.849404+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:36:16.109 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:36:16.372 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:36:16.372 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":174,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:35:31.256349+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.708195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:21.260068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:24.289309+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:24.821576+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.449685+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:22.632932+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.849404+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:36:19.388 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:36:19.643 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:36:19.643 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":174,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:35:31.256349+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.708195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:21.260068+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:24.289309+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:24.821576+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.449685+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:22.632932+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.849404+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:36:22.657 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:36:22.914 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:36:22.915 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":175,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:36:22.277892+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.708195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:21.416651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:24.289309+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:24.821576+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.449685+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:22.632932+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.849404+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:36:25.930 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:36:26.186 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:36:26.186 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":177,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:36:25.298451+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.708195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:21.416651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.201069+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:24.821576+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.449685+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:23.303160+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:35:25.849404+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:36:29.200 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:36:29.458 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:36:29.459 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":179,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:36:27.308650+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:26.660559+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:21.416651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.201069+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.478752+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:26.400031+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:23.303160+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.942715+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:36:32.473 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:36:32.723 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:36:32.723 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":180,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:36:31.280377+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:26.660559+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:21.416651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.201069+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.478752+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:26.400031+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:23.303160+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.942715+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:36:35.736 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:36:35.993 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:36:35.994 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":180,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:36:31.280377+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:26.660559+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:21.416651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.201069+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.478752+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:26.400031+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:23.303160+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.942715+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:36:39.014 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:36:39.268 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:36:39.268 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":180,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:36:31.280377+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:26.660559+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:21.416651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.201069+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.478752+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:26.400031+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:23.303160+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.942715+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:36:42.283 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:36:42.544 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:36:42.544 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":180,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:36:31.280377+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:26.660559+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:21.416651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.201069+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.478752+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:26.400031+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:23.303160+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.942715+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:36:45.563 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:36:45.818 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:36:45.818 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":180,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:36:31.280377+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:26.660559+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:21.416651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.201069+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.478752+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:26.400031+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:23.303160+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.942715+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:36:48.833 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:36:49.091 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:36:49.091 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":180,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:36:31.280377+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:26.660559+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:21.416651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.201069+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.478752+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:26.400031+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:23.303160+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.942715+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:36:52.104 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:36:52.358 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:36:52.359 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":180,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:36:31.280377+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:26.660559+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:21.416651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.201069+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.478752+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:26.400031+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:23.303160+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.942715+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:36:55.373 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:36:55.627 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:36:55.627 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":180,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:36:31.280377+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:26.660559+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:21.416651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.201069+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.478752+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:26.400031+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:23.303160+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.942715+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:36:58.640 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:36:58.891 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:36:58.892 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":180,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:36:31.280377+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:26.660559+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:21.416651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.201069+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.478752+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:26.400031+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:23.303160+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.942715+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:37:01.912 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:37:02.169 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:37:02.169 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":180,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:36:31.280377+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:26.660559+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:21.416651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.201069+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.478752+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:26.400031+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:23.303160+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.942715+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:37:05.183 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:37:05.434 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:37:05.434 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":180,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:36:31.280377+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:26.660559+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:21.416651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.201069+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.478752+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:26.400031+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:23.303160+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.942715+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:37:08.449 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:37:08.702 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:37:08.702 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":180,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:36:31.280377+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:26.660559+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:21.416651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.201069+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.478752+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:26.400031+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:23.303160+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.942715+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:37:11.716 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:37:11.989 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:37:11.990 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":180,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:36:31.280377+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:26.660559+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:21.416651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.201069+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.478752+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:26.400031+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:23.303160+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.942715+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:37:15.006 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:37:15.258 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:37:15.258 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":180,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:36:31.280377+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:26.660559+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:21.416651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.201069+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.478752+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:26.400031+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:23.303160+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.942715+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:37:18.279 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:37:18.538 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:37:18.538 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":180,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:36:31.280377+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:26.660559+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:21.416651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.201069+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.478752+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:26.400031+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:23.303160+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.942715+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:37:21.558 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:37:21.809 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:37:21.809 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":180,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:36:31.280377+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:26.660559+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:21.416651+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.201069+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.478752+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:26.400031+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:23.303160+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.942715+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:37:24.823 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:37:25.076 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:37:25.077 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":182,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:37:24.323562+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:26.660559+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:22.252251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.201069+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.478752+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:26.400031+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:23.609583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:36:25.942715+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:37:28.099 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:37:28.345 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:37:28.345 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":184,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:37:27.346382+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.687180+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:22.252251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.041664+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.447311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.410673+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:23.609583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.866123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:37:31.359 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:37:31.611 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:37:31.611 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":184,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:37:27.346382+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.687180+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:22.252251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.041664+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.447311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.410673+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:23.609583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.866123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:37:34.629 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:37:34.886 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:37:34.886 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":184,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:37:27.346382+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.687180+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:22.252251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.041664+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.447311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.410673+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:23.609583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.866123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:37:37.899 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:37:38.147 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:37:38.148 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":184,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:37:27.346382+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.687180+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:22.252251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.041664+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.447311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.410673+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:23.609583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.866123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:37:41.161 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:37:41.413 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:37:41.414 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":184,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:37:27.346382+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.687180+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:22.252251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.041664+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.447311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.410673+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:23.609583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.866123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:37:44.432 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:37:44.687 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:37:44.688 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":184,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:37:27.346382+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.687180+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:22.252251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.041664+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.447311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.410673+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:23.609583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.866123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:37:47.707 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:37:47.962 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:37:47.962 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":184,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:37:27.346382+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.687180+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:22.252251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.041664+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.447311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.410673+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:23.609583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.866123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:37:50.977 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:37:51.233 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:37:51.233 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":184,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:37:27.346382+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.687180+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:22.252251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.041664+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.447311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.410673+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:23.609583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.866123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:37:54.252 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:37:54.502 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:37:54.502 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":184,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:37:27.346382+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.687180+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:22.252251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.041664+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.447311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.410673+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:23.609583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.866123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:37:57.521 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:37:57.777 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:37:57.777 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":184,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:37:27.346382+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.687180+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:22.252251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.041664+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.447311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.410673+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:23.609583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.866123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:38:00.795 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:38:01.047 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:38:01.048 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":184,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:37:27.346382+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.687180+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:22.252251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.041664+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.447311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.410673+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:23.609583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.866123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:38:04.067 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:38:04.323 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:38:04.324 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":184,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:37:27.346382+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.687180+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:22.252251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.041664+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.447311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.410673+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:23.609583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.866123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:38:07.338 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:38:07.586 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:38:07.587 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":184,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:37:27.346382+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.687180+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:22.252251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.041664+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.447311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.410673+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:23.609583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.866123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:38:10.600 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:38:10.853 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:38:10.853 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":184,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:37:27.346382+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.687180+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:22.252251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.041664+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.447311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.410673+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:23.609583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.866123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:38:13.871 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:38:14.123 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:38:14.124 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":184,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:37:27.346382+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.687180+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:22.252251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.041664+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.447311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.410673+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:23.609583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.866123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:38:17.137 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:38:17.406 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:38:17.407 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":184,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:37:27.346382+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.687180+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:22.252251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.041664+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.447311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.410673+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:23.609583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.866123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:38:20.417 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:38:20.683 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:38:20.683 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":184,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:37:27.346382+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.687180+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:22.252251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.041664+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.447311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.410673+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:23.609583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.866123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:38:23.711 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:38:23.970 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:38:23.970 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":185,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:38:23.583779+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.687180+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:22.777031+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.041664+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.447311+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.410673+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:23.609583+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.866123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:38:26.985 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:38:27.242 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:38:27.242 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":187,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:38:26.598163+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.687180+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:22.777031+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.275344+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.566755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.410673+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:23.649268+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:37:26.866123+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:38:30.263 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:38:30.524 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:38:30.524 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":188,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:38:27.609026+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:27.304985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:22.777031+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.275344+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.566755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.624316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:23.649268+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:27.149711+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:38:33.544 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:38:33.806 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:38:33.806 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":188,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:38:27.609026+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:27.304985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:22.777031+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.275344+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.566755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.624316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:23.649268+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:27.149711+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:38:36.818 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:38:37.069 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:38:37.070 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":188,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:38:27.609026+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:27.304985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:22.777031+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.275344+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.566755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.624316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:23.649268+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:27.149711+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:38:40.083 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:38:40.340 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:38:40.340 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":188,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:38:27.609026+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:27.304985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:22.777031+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.275344+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.566755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.624316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:23.649268+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:27.149711+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:38:43.354 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:38:43.600 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:38:43.600 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":188,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:38:27.609026+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:27.304985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:22.777031+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.275344+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.566755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.624316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:23.649268+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:27.149711+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:38:46.614 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:38:46.869 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:38:46.870 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":188,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:38:27.609026+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:27.304985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:22.777031+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.275344+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.566755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.624316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:23.649268+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:27.149711+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:38:49.882 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:38:50.156 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:38:50.156 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":188,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:38:27.609026+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:27.304985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:22.777031+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.275344+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.566755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.624316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:23.649268+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:27.149711+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:38:53.170 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:38:53.433 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:38:53.433 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":188,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:38:27.609026+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:27.304985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:22.777031+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.275344+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.566755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.624316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:23.649268+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:27.149711+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:38:56.448 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:38:56.693 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:38:56.694 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":188,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:38:27.609026+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:27.304985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:22.777031+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.275344+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.566755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.624316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:23.649268+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:27.149711+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:38:59.715 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:38:59.966 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:38:59.966 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":188,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:38:27.609026+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:27.304985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:22.777031+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.275344+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.566755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.624316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:23.649268+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:27.149711+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:39:02.978 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:39:03.230 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:39:03.231 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":188,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:38:27.609026+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:27.304985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:22.777031+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.275344+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.566755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.624316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:23.649268+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:27.149711+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:39:06.247 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:39:06.499 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:39:06.499 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":188,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:38:27.609026+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:27.304985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:22.777031+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.275344+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.566755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.624316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:23.649268+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:27.149711+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:39:09.513 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:39:09.767 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:39:09.767 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":188,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:38:27.609026+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:27.304985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:22.777031+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.275344+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.566755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.624316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:23.649268+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:27.149711+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:39:12.781 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:39:13.034 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:39:13.034 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":188,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:38:27.609026+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:27.304985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:22.777031+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.275344+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.566755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.624316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:23.649268+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:27.149711+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:39:16.054 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:39:16.309 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:39:16.310 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":188,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:38:27.609026+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:27.304985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:22.777031+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.275344+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.566755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.624316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:23.649268+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:27.149711+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:39:19.322 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:39:19.579 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:39:19.580 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":188,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:38:27.609026+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:27.304985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:22.777031+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.275344+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.566755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.624316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:23.649268+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:27.149711+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:39:22.600 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:39:22.854 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:39:22.854 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":188,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:38:27.609026+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:27.304985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:22.777031+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.275344+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.566755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.624316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:23.649268+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:27.149711+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:39:25.867 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:39:26.126 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:39:26.126 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":190,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:39:24.607053+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:27.304985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:22.856363+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.275344+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.566755+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:26.624316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:24.332026+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:38:27.149711+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:39:29.141 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:39:29.394 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:39:29.394 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":192,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:39:28.623851+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:28.306063+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:22.856363+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.278188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.439301+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:26.647080+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:24.332026+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.945850+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:39:32.409 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:39:32.660 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:39:32.661 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":192,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:39:28.623851+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:28.306063+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:22.856363+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.278188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.439301+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:26.647080+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:24.332026+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.945850+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:39:35.686 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:39:35.940 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:39:35.941 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":192,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:39:28.623851+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:28.306063+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:22.856363+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.278188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.439301+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:26.647080+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:24.332026+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.945850+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:39:38.955 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:39:39.206 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:39:39.206 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":192,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:39:28.623851+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:28.306063+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:22.856363+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.278188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.439301+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:26.647080+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:24.332026+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.945850+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:39:42.220 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:39:42.474 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:39:42.474 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":192,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:39:28.623851+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:28.306063+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:22.856363+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.278188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.439301+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:26.647080+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:24.332026+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.945850+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:39:45.488 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:39:45.739 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:39:45.739 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":192,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:39:28.623851+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:28.306063+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:22.856363+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.278188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.439301+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:26.647080+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:24.332026+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.945850+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:39:48.753 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:39:49.009 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:39:49.009 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":192,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:39:28.623851+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:28.306063+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:22.856363+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.278188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.439301+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:26.647080+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:24.332026+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.945850+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:39:52.024 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:39:52.286 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:39:52.287 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":192,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:39:28.623851+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:28.306063+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:22.856363+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.278188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.439301+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:26.647080+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:24.332026+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.945850+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:39:55.301 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:39:55.549 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:39:55.550 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":192,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:39:28.623851+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:28.306063+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:22.856363+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.278188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.439301+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:26.647080+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:24.332026+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.945850+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:39:58.569 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:39:58.820 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:39:58.820 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":192,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:39:28.623851+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:28.306063+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:22.856363+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.278188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.439301+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:26.647080+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:24.332026+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.945850+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:40:01.835 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:40:02.086 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:40:02.086 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":192,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:39:28.623851+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:28.306063+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:22.856363+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.278188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.439301+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:26.647080+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:24.332026+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.945850+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:40:05.107 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:40:05.355 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:40:05.356 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":192,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:39:28.623851+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:28.306063+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:22.856363+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.278188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.439301+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:26.647080+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:24.332026+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.945850+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:40:08.369 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:40:08.627 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:40:08.628 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":192,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:39:28.623851+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:28.306063+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:22.856363+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.278188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.439301+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:26.647080+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:24.332026+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.945850+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:40:11.646 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:40:11.891 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:40:11.892 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":192,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:39:28.623851+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:28.306063+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:22.856363+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.278188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.439301+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:26.647080+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:24.332026+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.945850+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:40:14.904 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:40:15.158 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:40:15.158 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":192,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:39:28.623851+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:28.306063+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:22.856363+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.278188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.439301+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:26.647080+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:24.332026+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.945850+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:40:18.172 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:40:18.430 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:40:18.430 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":192,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:39:28.623851+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:28.306063+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:22.856363+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.278188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.439301+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:26.647080+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:24.332026+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.945850+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:40:21.444 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:40:21.702 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:40:21.702 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":192,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:39:28.623851+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:28.306063+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:22.856363+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.278188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.439301+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:26.647080+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:24.332026+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.945850+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:40:24.717 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:40:24.963 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:40:24.963 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":193,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:40:24.382343+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:28.306063+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:23.602216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.278188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.439301+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:26.647080+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:24.333722+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.945850+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:40:27.976 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:40:28.235 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:40:28.236 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":194,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:40:26.376555+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:28.306063+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:23.602216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.278188+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.439301+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:26.647080+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:24.333722+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:39:27.945850+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:40:31.250 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:40:31.535 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:40:31.535 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":196,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:40:29.408094+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:29.053405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:23.602216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.597264+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.812208+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.495363+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:24.333722+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:28.060150+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:40:34.550 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:40:34.811 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:40:34.811 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":196,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:40:29.408094+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:29.053405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:23.602216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.597264+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.812208+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.495363+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:24.333722+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:28.060150+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:40:37.825 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:40:38.085 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:40:38.085 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":196,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:40:29.408094+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:29.053405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:23.602216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.597264+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.812208+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.495363+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:24.333722+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:28.060150+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:40:41.099 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:40:41.356 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:40:41.356 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":196,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:40:29.408094+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:29.053405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:23.602216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.597264+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.812208+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.495363+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:24.333722+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:28.060150+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:40:44.371 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:40:44.623 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:40:44.623 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":196,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:40:29.408094+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:29.053405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:23.602216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.597264+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.812208+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.495363+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:24.333722+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:28.060150+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:40:47.641 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:40:47.896 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:40:47.897 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":196,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:40:29.408094+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:29.053405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:23.602216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.597264+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.812208+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.495363+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:24.333722+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:28.060150+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:40:50.913 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:40:51.179 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:40:51.180 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":196,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:40:29.408094+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:29.053405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:23.602216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.597264+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.812208+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.495363+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:24.333722+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:28.060150+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:40:54.184 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:40:54.446 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:40:54.447 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":196,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:40:29.408094+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:29.053405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:23.602216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.597264+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.812208+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.495363+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:24.333722+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:28.060150+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:40:57.460 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:40:57.712 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:40:57.713 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":196,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:40:29.408094+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:29.053405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:23.602216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.597264+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.812208+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.495363+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:24.333722+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:28.060150+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:41:00.727 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:41:00.978 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:41:00.978 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":196,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:40:29.408094+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:29.053405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:23.602216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.597264+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.812208+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.495363+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:24.333722+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:28.060150+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:41:03.992 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:41:04.240 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:41:04.240 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":196,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:40:29.408094+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:29.053405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:23.602216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.597264+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.812208+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.495363+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:24.333722+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:28.060150+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:41:07.253 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:41:07.505 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:41:07.506 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":196,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:40:29.408094+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:29.053405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:23.602216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.597264+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.812208+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.495363+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:24.333722+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:28.060150+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:41:10.521 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:41:10.780 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:41:10.780 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":196,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:40:29.408094+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:29.053405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:23.602216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.597264+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.812208+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.495363+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:24.333722+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:28.060150+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:41:13.800 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:41:14.059 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:41:14.060 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":196,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:40:29.408094+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:29.053405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:23.602216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.597264+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.812208+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.495363+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:24.333722+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:28.060150+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:41:17.077 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:41:17.422 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:41:17.422 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":196,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:40:29.408094+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:29.053405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:23.602216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.597264+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.812208+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.495363+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:24.333722+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:28.060150+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:41:20.424 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:41:20.682 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:41:20.683 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":196,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:40:29.408094+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:29.053405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:23.602216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.597264+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.812208+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.495363+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:24.333722+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:28.060150+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:41:23.696 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:41:23.947 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:41:23.947 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":196,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:40:29.408094+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:29.053405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:23.602216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.597264+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.812208+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.495363+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:24.333722+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:28.060150+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:41:26.961 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:41:27.213 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:41:27.214 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":198,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:41:26.010552+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:29.053405+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:24.431856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.597264+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.812208+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:27.495363+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:25.056591+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:40:28.060150+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:41:30.227 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:41:30.488 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:41:30.488 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":201,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:41:30.422154+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:29.977909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:24.431856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.477042+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.003570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.420638+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:25.056591+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.777031+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:41:33.507 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:41:33.759 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:41:33.759 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":202,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:41:31.403265+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:29.977909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:24.431856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.477042+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.003570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.420638+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:25.056591+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.777031+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:41:36.774 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:41:37.031 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:41:37.032 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":202,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:41:31.403265+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:29.977909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:24.431856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.477042+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.003570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.420638+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:25.056591+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.777031+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:41:40.045 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:41:40.298 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:41:40.298 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":202,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:41:31.403265+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:29.977909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:24.431856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.477042+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.003570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.420638+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:25.056591+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.777031+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:41:43.312 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:41:43.570 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:41:43.570 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":202,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:41:31.403265+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:29.977909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:24.431856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.477042+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.003570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.420638+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:25.056591+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.777031+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:41:46.584 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:41:46.840 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:41:46.841 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":202,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:41:31.403265+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:29.977909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:24.431856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.477042+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.003570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.420638+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:25.056591+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.777031+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:41:49.855 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:41:50.113 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:41:50.114 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":202,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:41:31.403265+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:29.977909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:24.431856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.477042+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.003570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.420638+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:25.056591+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.777031+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:41:53.129 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:41:53.391 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:41:53.391 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":202,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:41:31.403265+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:29.977909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:24.431856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.477042+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.003570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.420638+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:25.056591+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.777031+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:41:56.407 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:41:56.655 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:41:56.655 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":202,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:41:31.403265+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:29.977909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:24.431856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.477042+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.003570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.420638+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:25.056591+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.777031+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:41:59.669 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:41:59.916 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:41:59.917 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":202,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:41:31.403265+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:29.977909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:24.431856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.477042+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.003570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.420638+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:25.056591+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.777031+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:42:02.931 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:42:03.187 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:42:03.188 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":202,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:41:31.403265+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:29.977909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:24.431856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.477042+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.003570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.420638+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:25.056591+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.777031+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:42:06.201 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:42:06.459 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:42:06.459 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":202,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:41:31.403265+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:29.977909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:24.431856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.477042+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.003570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.420638+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:25.056591+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.777031+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:42:09.474 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:42:09.745 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:42:09.746 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":202,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:41:31.403265+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:29.977909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:24.431856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.477042+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.003570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.420638+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:25.056591+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.777031+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:42:12.759 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:42:13.011 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:42:13.011 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":202,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:41:31.403265+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:29.977909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:24.431856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.477042+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.003570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.420638+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:25.056591+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.777031+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:42:16.030 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:42:16.286 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:42:16.287 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":202,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:41:31.403265+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:29.977909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:24.431856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.477042+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.003570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.420638+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:25.056591+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.777031+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:42:19.302 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:42:19.560 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:42:19.561 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":202,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:41:31.403265+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:29.977909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:24.431856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.477042+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.003570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.420638+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:25.056591+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.777031+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:42:22.574 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:42:22.829 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:42:22.830 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":202,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:41:31.403265+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:29.977909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:24.431856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.477042+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.003570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.420638+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:25.056591+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.777031+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:42:25.843 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:42:26.101 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:42:26.101 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":203,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:42:25.663947+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:29.977909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:25.378814+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.477042+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.003570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.420638+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:25.056591+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.777031+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:42:29.115 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:42:29.371 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:42:29.371 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":204,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:42:26.427809+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:29.977909+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:25.378814+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.477042+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.003570+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.420638+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:26.039097+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:41:28.777031+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:42:32.386 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:42:32.638 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:42:32.638 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":206,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:42:31.430144+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:30.894979+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:25.378814+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.775775+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.820001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:29.411067+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:26.039097+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.912242+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:42:35.654 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:42:35.905 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:42:35.905 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":206,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:42:31.430144+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:30.894979+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:25.378814+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.775775+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.820001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:29.411067+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:26.039097+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.912242+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:42:38.929 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:42:39.183 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:42:39.183 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":206,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:42:31.430144+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:30.894979+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:25.378814+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.775775+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.820001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:29.411067+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:26.039097+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.912242+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:42:42.198 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:42:42.457 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:42:42.458 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":206,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:42:31.430144+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:30.894979+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:25.378814+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.775775+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.820001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:29.411067+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:26.039097+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.912242+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:42:45.472 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:42:45.741 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:42:45.741 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":206,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:42:31.430144+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:30.894979+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:25.378814+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.775775+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.820001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:29.411067+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:26.039097+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.912242+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:42:48.750 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:42:49.006 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:42:49.006 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":206,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:42:31.430144+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:30.894979+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:25.378814+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.775775+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.820001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:29.411067+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:26.039097+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.912242+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:42:52.019 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:42:52.274 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:42:52.275 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":206,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:42:31.430144+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:30.894979+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:25.378814+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.775775+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.820001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:29.411067+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:26.039097+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.912242+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:42:55.289 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:42:55.546 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:42:55.546 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":206,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:42:31.430144+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:30.894979+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:25.378814+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.775775+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.820001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:29.411067+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:26.039097+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.912242+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:42:58.570 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:42:58.825 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:42:58.825 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":206,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:42:31.430144+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:30.894979+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:25.378814+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.775775+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.820001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:29.411067+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:26.039097+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.912242+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:43:01.838 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:43:02.094 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:43:02.095 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":206,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:42:31.430144+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:30.894979+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:25.378814+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.775775+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.820001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:29.411067+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:26.039097+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.912242+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:43:05.104 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:43:05.358 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:43:05.358 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":206,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:42:31.430144+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:30.894979+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:25.378814+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.775775+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.820001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:29.411067+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:26.039097+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.912242+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:43:08.374 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:43:08.629 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:43:08.630 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":206,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:42:31.430144+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:30.894979+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:25.378814+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.775775+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.820001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:29.411067+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:26.039097+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.912242+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:43:11.644 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:43:11.894 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:43:11.895 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":206,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:42:31.430144+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:30.894979+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:25.378814+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.775775+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.820001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:29.411067+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:26.039097+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.912242+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:43:14.914 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:43:15.171 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:43:15.171 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":206,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:42:31.430144+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:30.894979+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:25.378814+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.775775+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.820001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:29.411067+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:26.039097+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.912242+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:43:18.185 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:43:18.440 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:43:18.440 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":206,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:42:31.430144+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:30.894979+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:25.378814+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.775775+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.820001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:29.411067+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:26.039097+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.912242+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:43:21.462 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:43:21.718 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:43:21.718 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":206,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:42:31.430144+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:30.894979+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:25.378814+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.775775+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.820001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:29.411067+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:26.039097+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.912242+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:43:24.733 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:43:24.993 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:43:24.993 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":206,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:42:31.430144+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:30.894979+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:25.378814+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.775775+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.820001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:29.411067+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:26.039097+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.912242+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:43:28.007 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:43:28.264 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:43:28.264 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":208,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:43:27.708726+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:30.894979+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:26.248864+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.775775+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.820001+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:29.411067+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:26.991240+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:42:28.912242+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:43:31.280 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:43:31.540 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:43:31.540 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":210,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:43:31.455014+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:30.915454+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:26.248864+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.610750+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:28.983243+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.446495+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:26.991240+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.076770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:43:34.559 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:43:34.820 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:43:34.821 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":210,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:43:31.455014+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:30.915454+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:26.248864+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.610750+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:28.983243+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.446495+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:26.991240+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.076770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:43:37.844 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:43:38.097 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:43:38.097 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":210,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:43:31.455014+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:30.915454+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:26.248864+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.610750+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:28.983243+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.446495+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:26.991240+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.076770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:43:41.117 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:43:41.371 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:43:41.372 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":210,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:43:31.455014+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:30.915454+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:26.248864+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.610750+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:28.983243+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.446495+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:26.991240+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.076770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:43:44.389 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:43:44.640 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:43:44.640 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":210,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:43:31.455014+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:30.915454+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:26.248864+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.610750+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:28.983243+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.446495+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:26.991240+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.076770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:43:47.661 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:43:47.915 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:43:47.915 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":210,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:43:31.455014+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:30.915454+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:26.248864+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.610750+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:28.983243+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.446495+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:26.991240+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.076770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:43:50.930 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:43:51.193 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:43:51.194 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":210,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:43:31.455014+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:30.915454+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:26.248864+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.610750+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:28.983243+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.446495+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:26.991240+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.076770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:43:54.208 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:43:54.482 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:43:54.482 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":210,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:43:31.455014+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:30.915454+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:26.248864+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.610750+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:28.983243+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.446495+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:26.991240+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.076770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:43:57.501 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:43:57.750 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:43:57.750 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":210,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:43:31.455014+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:30.915454+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:26.248864+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.610750+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:28.983243+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.446495+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:26.991240+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.076770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:44:00.772 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:44:01.029 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:44:01.029 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":210,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:43:31.455014+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:30.915454+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:26.248864+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.610750+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:28.983243+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.446495+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:26.991240+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.076770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:44:04.044 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:44:04.299 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:44:04.299 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":210,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:43:31.455014+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:30.915454+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:26.248864+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.610750+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:28.983243+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.446495+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:26.991240+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.076770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:44:07.324 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:44:07.618 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:44:07.618 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":210,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:43:31.455014+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:30.915454+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:26.248864+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.610750+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:28.983243+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.446495+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:26.991240+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.076770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:44:10.636 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:44:10.905 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:44:10.905 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":210,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:43:31.455014+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:30.915454+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:26.248864+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.610750+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:28.983243+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.446495+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:26.991240+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.076770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:44:13.920 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:44:14.170 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:44:14.170 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":210,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:43:31.455014+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:30.915454+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:26.248864+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.610750+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:28.983243+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.446495+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:26.991240+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.076770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:44:17.185 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:44:17.439 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:44:17.439 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":210,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:43:31.455014+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:30.915454+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:26.248864+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.610750+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:28.983243+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.446495+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:26.991240+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.076770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:44:20.460 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:44:20.714 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:44:20.714 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":210,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:43:31.455014+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:30.915454+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:26.248864+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.610750+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:28.983243+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.446495+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:26.991240+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.076770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:44:23.729 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:44:23.977 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:44:23.978 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":210,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:43:31.455014+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:30.915454+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:26.248864+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.610750+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:28.983243+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.446495+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:26.991240+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.076770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:44:26.991 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:44:27.239 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:44:27.239 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":210,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:43:31.455014+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:30.915454+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:26.248864+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.610750+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:28.983243+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.446495+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:26.991240+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.076770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:44:30.254 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:44:30.505 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:44:30.505 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":213,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:44:29.500327+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:30.915454+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:26.983748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.610750+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:28.983243+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.477234+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:27.853807+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:43:29.076770+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:44:33.518 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:44:33.774 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:44:33.774 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":215,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:44:32.525459+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:31.855869+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:26.983748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.643746+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.822285+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.477234+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:27.853807+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.724628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:44:36.788 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:44:37.039 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:44:37.040 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":216,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:44:36.482009+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:31.855869+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:26.983748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.643746+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.822285+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.477234+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:27.853807+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.724628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:44:40.053 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:44:40.308 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:44:40.309 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":216,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:44:36.482009+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:31.855869+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:26.983748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.643746+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.822285+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.477234+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:27.853807+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.724628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:44:43.327 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:44:43.578 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:44:43.578 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":216,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:44:36.482009+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:31.855869+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:26.983748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.643746+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.822285+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.477234+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:27.853807+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.724628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:44:46.592 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:44:46.847 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:44:46.847 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":216,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:44:36.482009+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:31.855869+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:26.983748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.643746+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.822285+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.477234+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:27.853807+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.724628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:44:49.862 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:44:50.121 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:44:50.121 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":216,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:44:36.482009+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:31.855869+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:26.983748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.643746+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.822285+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.477234+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:27.853807+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.724628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:44:53.136 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:44:53.402 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:44:53.402 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":216,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:44:36.482009+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:31.855869+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:26.983748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.643746+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.822285+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.477234+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:27.853807+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.724628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:44:56.416 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:44:56.675 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:44:56.675 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":216,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:44:36.482009+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:31.855869+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:26.983748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.643746+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.822285+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.477234+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:27.853807+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.724628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:44:59.695 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:44:59.950 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:44:59.951 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":216,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:44:36.482009+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:31.855869+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:26.983748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.643746+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.822285+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.477234+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:27.853807+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.724628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:45:02.964 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:45:03.225 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:45:03.226 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":216,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:44:36.482009+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:31.855869+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:26.983748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.643746+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.822285+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.477234+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:27.853807+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.724628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:45:06.245 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:45:06.506 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:45:06.506 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":216,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:44:36.482009+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:31.855869+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:26.983748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.643746+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.822285+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.477234+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:27.853807+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.724628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:45:09.526 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:45:09.779 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:45:09.779 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":216,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:44:36.482009+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:31.855869+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:26.983748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.643746+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.822285+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.477234+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:27.853807+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.724628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:45:12.794 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:45:13.052 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:45:13.052 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":216,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:44:36.482009+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:31.855869+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:26.983748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.643746+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.822285+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.477234+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:27.853807+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.724628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:45:16.067 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:45:16.325 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:45:16.325 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":216,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:44:36.482009+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:31.855869+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:26.983748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.643746+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.822285+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.477234+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:27.853807+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.724628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:45:19.347 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:45:19.598 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:45:19.598 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":216,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:44:36.482009+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:31.855869+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:26.983748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.643746+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.822285+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.477234+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:27.853807+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.724628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:45:22.613 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:45:22.882 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:45:22.882 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":216,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:44:36.482009+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:31.855869+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:26.983748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.643746+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.822285+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.477234+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:27.853807+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.724628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:45:25.888 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:45:26.152 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:45:26.152 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":216,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:44:36.482009+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:31.855869+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:26.983748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.643746+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.822285+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.477234+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:27.853807+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.724628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:45:29.160 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:45:29.415 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:45:29.415 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":218,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:45:28.515880+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:31.855869+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:45:27.263503+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.643746+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.822285+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.477234+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:45:28.167833+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:29.724628+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:45:32.429 INFO:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-01-16T00:45:32.682 INFO:teuthology.orchestra.run.smithi123.stdout: 2020-01-16T00:45:32.682 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":220,"fsid":"864e7191-e3db-4201-b88a-a4873d1eb7fe","created":"2020-01-16T00:12:06.009499+0000","modified":"2020-01-16T00:45:30.537481+0000","last_up_change":"2020-01-16T00:13:14.826532+0000","last_in_change":"2020-01-16T00:12:13.968282+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-01-16T00:12:18.200931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"low_tier","create_time":"2020-01-16T00:12:30.019477+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"4b8d1e40-95e3-4adc-9929-cb7e6cc8b766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6809","nonce":11906}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6810","nonce":11906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6812","nonce":11906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6811","nonce":11906}]},"public_addr":"172.21.15.123:6809/11906","cluster_addr":"172.21.15.123:6810/11906","heartbeat_back_addr":"172.21.15.123:6812/11906","heartbeat_front_addr":"172.21.15.123:6811/11906","state":["exists","up"]},{"osd":1,"uuid":"2e373d15-6348-4caa-aa9f-b46037376a3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6805","nonce":11905}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6806","nonce":11905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6808","nonce":11905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6807","nonce":11905}]},"public_addr":"172.21.15.123:6805/11905","cluster_addr":"172.21.15.123:6806/11905","heartbeat_back_addr":"172.21.15.123:6808/11905","heartbeat_front_addr":"172.21.15.123:6807/11905","state":["exists","up"]},{"osd":2,"uuid":"5736ff2c-076a-4d41-8e11-06e2fd1f5f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6801","nonce":11907}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6802","nonce":11907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6804","nonce":11907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6803","nonce":11907}]},"public_addr":"172.21.15.123:6801/11907","cluster_addr":"172.21.15.123:6802/11907","heartbeat_back_addr":"172.21.15.123:6804/11907","heartbeat_front_addr":"172.21.15.123:6803/11907","state":["exists","up"]},{"osd":3,"uuid":"554f53a8-c5b1-43c9-a7d6-93629d84f1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6813","nonce":11917}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6814","nonce":11917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6816","nonce":11917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.123:6815","nonce":11917}]},"public_addr":"172.21.15.123:6813/11917","cluster_addr":"172.21.15.123:6814/11917","heartbeat_back_addr":"172.21.15.123:6816/11917","heartbeat_front_addr":"172.21.15.123:6815/11917","state":["exists","up"]},{"osd":4,"uuid":"77945419-c04b-4071-ab87-256d77640dc0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6800","nonce":11851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6801","nonce":11851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6803","nonce":11851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6802","nonce":11851}]},"public_addr":"172.21.15.145:6800/11851","cluster_addr":"172.21.15.145:6801/11851","heartbeat_back_addr":"172.21.15.145:6803/11851","heartbeat_front_addr":"172.21.15.145:6802/11851","state":["exists","up"]},{"osd":5,"uuid":"d5c073a5-8f8e-44fa-ab21-7ef6dbf5a2b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":40,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6804","nonce":11860}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6805","nonce":11860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6807","nonce":11860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6806","nonce":11860}]},"public_addr":"172.21.15.145:6804/11860","cluster_addr":"172.21.15.145:6805/11860","heartbeat_back_addr":"172.21.15.145:6807/11860","heartbeat_front_addr":"172.21.15.145:6806/11860","state":["exists"]},{"osd":6,"uuid":"e9d342cd-5c2d-40a0-b8ea-24588f280389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6812","nonce":11867}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6813","nonce":11867}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6815","nonce":11867}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6814","nonce":11867}]},"public_addr":"172.21.15.145:6812/11867","cluster_addr":"172.21.15.145:6813/11867","heartbeat_back_addr":"172.21.15.145:6815/11867","heartbeat_front_addr":"172.21.15.145:6814/11867","state":["exists","up"]},{"osd":7,"uuid":"6a301b66-384a-40d8-adc8-96fff49e982e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6808","nonce":11868}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6809","nonce":11868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6811","nonce":11868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.145:6810","nonce":11868}]},"public_addr":"172.21.15.145:6808/11868","cluster_addr":"172.21.15.145:6809/11868","heartbeat_back_addr":"172.21.15.145:6811/11868","heartbeat_front_addr":"172.21.15.145:6810/11868","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:44:31.855869+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:45:27.263503+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:45:29.728455+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:45:29.969309+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:45:29.496629+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:13:13.453529+0000","dead_epoch":40},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:45:28.167833+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-01-16T00:45:29.858111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-01-16T00:45:35.453 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-01-16T00:45:35.454 DEBUG:teuthology.task.console_log:Killing console logger for smithi145 2020-01-16T00:45:35.454 DEBUG:teuthology.task.console_log:Killing console logger for smithi123 2020-01-16T00:45:35.454 DEBUG:teuthology.task.console_log:Killing console logger for smithi145 2020-01-16T00:45:35.455 DEBUG:teuthology.task.console_log:Killing console logger for smithi123 2020-01-16T00:45:35.455 DEBUG:teuthology.exit:Finished running handlers